- How to do DMA with ISA (and LPC) devices.
DMA-attributes.txt
- listing of the various possible attributes a DMA region can have
-DocBook/
- - directory with DocBook templates etc. for kernel documentation.
EDID/
- directory with info on customizing EDID for broken gfx/displays.
IPMI.txt
- basic info on the Intel IOMMU virtualization support.
Makefile
- It's not of interest for those who aren't touching the build system.
-Makefile.sphinx
- - It's not of interest for those who aren't touching the build system.
PCI/
- info related to PCI drivers.
RCU/
- full colour GIF image of Linux logo (penguin - Tux).
logo.txt
- info on creator of above logo & site to get additional images from.
+lsm.txt
+ - Linux Security Modules: General Security Hooks for Linux
lzo.txt
- kernel LZO decompressor input formats
m68k/
Indicates the maximum USB speed supported by this port.
Users:
-What: /sys/class/udc/<udc>/maximum_speed
-Date: June 2011
-KernelVersion: 3.1
-Contact: Felipe Balbi <balbi@kernel.org>
-Description:
- Indicates the maximum USB speed supported by this port.
-Users:
-
What: /sys/class/udc/<udc>/soft_connect
Date: June 2011
KernelVersion: 3.1
'configured', and 'suspended'; however not all USB Device
Controllers support reporting all states.
Users:
+
+What: /sys/class/udc/<udc>/function
+Date: June 2017
+KernelVersion: 4.13
+Contact: Felipe Balbi <balbi@kernel.org>
+Description:
+ Prints out name of currently running USB Gadget Driver.
+Users:
--- /dev/null
+What: /sys/bus/platform/drivers/aspeed-vuart/*/lpc_address
+Date: April 2017
+Contact: Jeremy Kerr <jk@ozlabs.org>
+Description: Configures which IO port the host side of the UART
+ will appear on the host <-> BMC LPC bus.
+Users: OpenBMC. Proposed changes should be mailed to
+ openbmc@lists.ozlabs.org
+
+What: /sys/bus/platform/drivers/aspeed-vuart*/sirq
+Date: April 2017
+Contact: Jeremy Kerr <jk@ozlabs.org>
+Description: Configures which interrupt number the host side of
+ the UART will appear on the host <-> BMC LPC bus.
+Users: OpenBMC. Proposed changes should be mailed to
+ openbmc@lists.ozlabs.org
What: /config/usb-gadget/gadget/functions/uac1.name
-Date: Sep 2014
-KernelVersion: 3.18
+Date: June 2017
+KernelVersion: 4.14
Description:
The attributes:
- audio_buf_size - audio buffer size
- fn_cap - capture pcm device file name
- fn_cntl - control device file name
- fn_play - playback pcm device file name
- req_buf_size - ISO OUT endpoint request buffer size
- req_count - ISO OUT endpoint request count
+ c_chmask - capture channel mask
+ c_srate - capture sampling rate
+ c_ssize - capture sample size (bytes)
+ p_chmask - playback channel mask
+ p_srate - playback sampling rate
+ p_ssize - playback sample size (bytes)
+ req_number - the number of pre-allocated request
+ for both capture and playback
--- /dev/null
+What: /config/usb-gadget/gadget/functions/uac1_legacy.name
+Date: Sep 2014
+KernelVersion: 3.18
+Description:
+ The attributes:
+
+ audio_buf_size - audio buffer size
+ fn_cap - capture pcm device file name
+ fn_cntl - control device file name
+ fn_play - playback pcm device file name
+ req_buf_size - ISO OUT endpoint request buffer size
+ req_count - ISO OUT endpoint request count
--- /dev/null
+What: /sys/bus/platform/devices/fsi-master/rescan
+Date: May 2017
+KernelVersion: 4.12
+Contact: cbostic@linux.vnet.ibm.com
+Description:
+ Initiates a FSI master scan for all connected slave devices
+ on its links.
+
+What: /sys/bus/platform/devices/fsi-master/break
+Date: May 2017
+KernelVersion: 4.12
+Contact: cbostic@linux.vnet.ibm.com
+Description:
+ Sends an FSI BREAK command on a master's communication
+ link to any connnected slaves. A BREAK resets connected
+ device's logic and preps it to receive further commands
+ from the master.
+
+What: /sys/bus/platform/devices/fsi-master/slave@00:00/term
+Date: May 2017
+KernelVersion: 4.12
+Contact: cbostic@linux.vnet.ibm.com
+Description:
+ Sends an FSI terminate command from the master to its
+ connected slave. A terminate resets the slave's state machines
+ that control access to the internally connected engines. In
+ addition the slave freezes its internal error register for
+ debugging purposes. This command is also needed to abort any
+ ongoing operation in case of an expired 'Master Time Out'
+ timer.
+
+What: /sys/bus/platform/devices/fsi-master/slave@00:00/raw
+Date: May 2017
+KernelVersion: 4.12
+Contact: cbostic@linux.vnet.ibm.com
+Description:
+ Provides a means of reading/writing a 32 bit value from/to a
+ specified FSI bus address.
guarantees that the hardware fifo is flushed to the device
buffer.
+What: /sys/bus/iio/devices/iio:device*/buffer/hwfifo_timeout
+KernelVersion: 4.12
+Contact: linux-iio@vger.kernel.org
+Description:
+ A read/write property to provide capability to delay reporting of
+ samples till a timeout is reached. This allows host processors to
+ sleep, while the sensor is storing samples in its internal fifo.
+ The maximum timeout in seconds can be specified by setting
+ hwfifo_timeout.The current delay can be read by reading
+ hwfifo_timeout. A value of 0 means that there is no timeout.
+
What: /sys/bus/iio/devices/iio:deviceX/buffer/hwfifo_watermark
KernelVersion: 4.2
Contact: linux-iio@vger.kernel.org
Reading returns either '1' or '0'. '1' means that the
battery level supplied to sensor is below 2.25V.
This ABI is available for tsys02d, htu21, ms8607
- This ABI is available for htu21, ms8607
- "OC2REF" : OC2REF signal is used as trigger output.
- "OC3REF" : OC3REF signal is used as trigger output.
- "OC4REF" : OC4REF signal is used as trigger output.
+ Additional modes (on TRGO2 only):
+ - "OC5REF" : OC5REF signal is used as trigger output.
+ - "OC6REF" : OC6REF signal is used as trigger output.
+ - "compare_pulse_OC4REF":
+ OC4REF rising or falling edges generate pulses.
+ - "compare_pulse_OC6REF":
+ OC6REF rising or falling edges generate pulses.
+ - "compare_pulse_OC4REF_r_or_OC6REF_r":
+ OC4REF or OC6REF rising edges generate pulses.
+ - "compare_pulse_OC4REF_r_or_OC6REF_f":
+ OC4REF rising or OC6REF falling edges generate pulses.
+ - "compare_pulse_OC5REF_r_or_OC6REF_r":
+ OC5REF or OC6REF rising edges generate pulses.
+ - "compare_pulse_OC5REF_r_or_OC6REF_f":
+ OC5REF rising or OC6REF falling edges generate pulses.
+
+ +-----------+ +-------------+ +---------+
+ | Prescaler +-> | Counter | +-> | Master | TRGO(2)
+ +-----------+ +--+--------+-+ |-> | Control +-->
+ | | || +---------+
+ +--v--------+-+ OCxREF || +---------+
+ | Chx compare +----------> | Output | ChX
+ +-----------+-+ | | Control +-->
+ . | | +---------+
+ . | | .
+ +-----------v-+ OC6REF | .
+ | Ch6 compare +---------+>
+ +-------------+
+
+ Example with: "compare_pulse_OC4REF_r_or_OC6REF_r":
+
+ X
+ X X
+ X . . X
+ X . . X
+ X . . X
+ count X . . . . X
+ . . . .
+ . . . .
+ +---------------+
+ OC4REF | . . |
+ +-+ . . +-+
+ . +---+ .
+ OC6REF . | | .
+ +-------+ +-------+
+ +-+ +-+
+ TRGO2 | | | |
+ +-+ +---+ +---------+
What: /sys/bus/iio/devices/triggerX/master_mode
KernelVersion: 4.11
Counting is enabled on rising edge of the connected
trigger, and remains enabled for the duration of this
selected mode.
+
+What: /sys/bus/iio/devices/iio:deviceX/in_count_trigger_mode_available
+KernelVersion: 4.13
+Contact: benjamin.gaignard@st.com
+Description:
+ Reading returns the list possible trigger modes.
+
+What: /sys/bus/iio/devices/iio:deviceX/in_count0_trigger_mode
+KernelVersion: 4.13
+Contact: benjamin.gaignard@st.com
+Description:
+ Configure the device counter trigger mode
+ counting direction is set by in_count0_count_direction
+ attribute and the counter is clocked by the connected trigger
+ rising edges.
--- /dev/null
+What: /sys/bus/thunderbolt/devices/.../domainX/security
+Date: Sep 2017
+KernelVersion: 4.13
+Contact: thunderbolt-software@lists.01.org
+Description: This attribute holds current Thunderbolt security level
+ set by the system BIOS. Possible values are:
+
+ none: All devices are automatically authorized
+ user: Devices are only authorized based on writing
+ appropriate value to the authorized attribute
+ secure: Require devices that support secure connect at
+ minimum. User needs to authorize each device.
+ dponly: Automatically tunnel Display port (and USB). No
+ PCIe tunnels are created.
+
+What: /sys/bus/thunderbolt/devices/.../authorized
+Date: Sep 2017
+KernelVersion: 4.13
+Contact: thunderbolt-software@lists.01.org
+Description: This attribute is used to authorize Thunderbolt devices
+ after they have been connected. If the device is not
+ authorized, no devices such as PCIe and Display port are
+ available to the system.
+
+ Contents of this attribute will be 0 when the device is not
+ yet authorized.
+
+ Possible values are supported:
+ 1: The device will be authorized and connected
+
+ When key attribute contains 32 byte hex string the possible
+ values are:
+ 1: The 32 byte hex string is added to the device NVM and
+ the device is authorized.
+ 2: Send a challenge based on the 32 byte hex string. If the
+ challenge response from device is valid, the device is
+ authorized. In case of failure errno will be ENOKEY if
+ the device did not contain a key at all, and
+ EKEYREJECTED if the challenge response did not match.
+
+What: /sys/bus/thunderbolt/devices/.../key
+Date: Sep 2017
+KernelVersion: 4.13
+Contact: thunderbolt-software@lists.01.org
+Description: When a devices supports Thunderbolt secure connect it will
+ have this attribute. Writing 32 byte hex string changes
+ authorization to use the secure connection method instead.
+
+What: /sys/bus/thunderbolt/devices/.../device
+Date: Sep 2017
+KernelVersion: 4.13
+Contact: thunderbolt-software@lists.01.org
+Description: This attribute contains id of this device extracted from
+ the device DROM.
+
+What: /sys/bus/thunderbolt/devices/.../device_name
+Date: Sep 2017
+KernelVersion: 4.13
+Contact: thunderbolt-software@lists.01.org
+Description: This attribute contains name of this device extracted from
+ the device DROM.
+
+What: /sys/bus/thunderbolt/devices/.../vendor
+Date: Sep 2017
+KernelVersion: 4.13
+Contact: thunderbolt-software@lists.01.org
+Description: This attribute contains vendor id of this device extracted
+ from the device DROM.
+
+What: /sys/bus/thunderbolt/devices/.../vendor_name
+Date: Sep 2017
+KernelVersion: 4.13
+Contact: thunderbolt-software@lists.01.org
+Description: This attribute contains vendor name of this device extracted
+ from the device DROM.
+
+What: /sys/bus/thunderbolt/devices/.../unique_id
+Date: Sep 2017
+KernelVersion: 4.13
+Contact: thunderbolt-software@lists.01.org
+Description: This attribute contains unique_id string of this device.
+ This is either read from hardware registers (UUID on
+ newer hardware) or based on UID from the device DROM.
+ Can be used to uniquely identify particular device.
+
+What: /sys/bus/thunderbolt/devices/.../nvm_version
+Date: Sep 2017
+KernelVersion: 4.13
+Contact: thunderbolt-software@lists.01.org
+Description: If the device has upgradeable firmware the version
+ number is available here. Format: %x.%x, major.minor.
+ If the device is in safe mode reading the file returns
+ -ENODATA instead as the NVM version is not available.
+
+What: /sys/bus/thunderbolt/devices/.../nvm_authenticate
+Date: Sep 2017
+KernelVersion: 4.13
+Contact: thunderbolt-software@lists.01.org
+Description: When new NVM image is written to the non-active NVM
+ area (through non_activeX NVMem device), the
+ authentication procedure is started by writing 1 to
+ this file. If everything goes well, the device is
+ restarted with the new NVM firmware. If the image
+ verification fails an error code is returned instead.
+
+ When read holds status of the last authentication
+ operation if an error occurred during the process. This
+ is directly the status value from the DMA configuration
+ based mailbox before the device is power cycled. Writing
+ 0 here clears the status.
--- /dev/null
+What: /sys/class/mux/
+Date: April 2017
+KernelVersion: 4.13
+Contact: Peter Rosin <peda@axentia.se>
+Description:
+ The mux/ class sub-directory belongs to the Generic MUX
+ Framework and provides a sysfs interface for using MUX
+ controllers.
+
+What: /sys/class/mux/muxchipN/
+Date: April 2017
+KernelVersion: 4.13
+Contact: Peter Rosin <peda@axentia.se>
+Description:
+ A /sys/class/mux/muxchipN directory is created for each
+ probed MUX chip where N is a simple enumeration.
Valid values: source, sink
+What: /sys/class/typec/<port>/port_type
+Date: May 2017
+Contact: Badhri Jagan Sridharan <Badhri@google.com>
+Description:
+ Indicates the type of the port. This attribute can be used for
+ requesting a change in the port type. Port type change is
+ supported as a synchronous operation, so write(2) to the
+ attribute will not return until the operation has finished.
+
+ Valid values:
+ - source (The port will behave as source only DFP port)
+ - sink (The port will behave as sink only UFP port)
+ - dual (The port will behave as dual-role-data and
+ dual-role-power port)
+
What: /sys/class/typec/<port>/vconn_source
Date: April 2017
Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
--- /dev/null
+What: /sys/.../uevent
+Date: May 2017
+KernelVersion: 4.13
+Contact: Linux kernel mailing list <linux-kernel@vger.kernel.org>
+Description:
+ Enable passing additional variables for synthetic uevents that
+ are generated by writing /sys/.../uevent file.
+
+ Recognized extended format is ACTION [UUID [KEY=VALUE ...].
+
+ The ACTION is compulsory - it is the name of the uevent action
+ ("add", "change", "remove"). There is no change compared to
+ previous functionality here. The rest of the extended format
+ is optional.
+
+ You need to pass UUID first before any KEY=VALUE pairs.
+ The UUID must be in "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
+ format where 'x' is a hex digit. The UUID is considered to be
+ a transaction identifier so it's possible to use the same UUID
+ value for one or more synthetic uevents in which case we
+ logically group these uevents together for any userspace
+ listeners. The UUID value appears in uevent as
+ "SYNTH_UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" environment
+ variable.
+
+ If UUID is not passed in, the generated synthetic uevent gains
+ "SYNTH_UUID=0" environment variable automatically.
+
+ The KEY=VALUE pairs can contain alphanumeric characters only.
+ It's possible to define zero or more pairs - each pair is then
+ delimited by a space character ' '. Each pair appears in
+ synthetic uevent as "SYNTH_ARG_KEY=VALUE". That means the KEY
+ name gains "SYNTH_ARG_" prefix to avoid possible collisions
+ with existing variables.
+
+ Example of valid sequence written to the uevent file:
+
+ add fe4d7c9d-b8c6-4a70-9ef1-3d8a58d18eed A=1 B=abc
+
+ This generates synthetic uevent including these variables:
+
+ ACTION=add
+ SYNTH_ARG_A=1
+ SYNTH_ARG_B=abc
+ SYNTH_UUID=fe4d7c9d-b8c6-4a70-9ef1-3d8a58d18eed
+Users:
+ udev, userspace tools generating synthetic uevents
boot with 'dma_debug_entries=<your_desired_number>' to overwrite the
architectural default.
-void debug_dmap_mapping_error(struct device *dev, dma_addr_t dma_addr);
+void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr);
dma-debug interface debug_dma_mapping_error() to debug drivers that fail
to check DMA mapping errors on addresses returned by dma_map_single() and
+++ /dev/null
-*.xml
-*.ps
-*.pdf
-*.html
-*.9.gz
-*.9
-*.aux
-*.dvi
-*.log
-*.out
-*.png
-*.gif
-*.svg
-*.proc
-*.db
-media-indices.tmpl
-media-entities.tmpl
+++ /dev/null
-###
-# This makefile is used to generate the kernel documentation,
-# primarily based on in-line comments in various source files.
-# See Documentation/kernel-doc-nano-HOWTO.txt for instruction in how
-# to document the SRC - and how to read it.
-# To add a new book the only step required is to add the book to the
-# list of DOCBOOKS.
-
-DOCBOOKS := z8530book.xml \
- kernel-hacking.xml kernel-locking.xml \
- networking.xml \
- filesystems.xml lsm.xml kgdb.xml \
- libata.xml mtdnand.xml librs.xml rapidio.xml \
- s390-drivers.xml scsi.xml \
- sh.xml w1.xml
-
-ifeq ($(DOCBOOKS),)
-
-# Skip DocBook build if the user explicitly requested no DOCBOOKS.
-.DEFAULT:
- @echo " SKIP DocBook $@ target (DOCBOOKS=\"\" specified)."
-else
-ifneq ($(SPHINXDIRS),)
-
-# Skip DocBook build if the user explicitly requested a sphinx dir
-.DEFAULT:
- @echo " SKIP DocBook $@ target (SPHINXDIRS specified)."
-else
-
-
-###
-# The build process is as follows (targets):
-# (xmldocs) [by docproc]
-# file.tmpl --> file.xml +--> file.ps (psdocs) [by db2ps or xmlto]
-# +--> file.pdf (pdfdocs) [by db2pdf or xmlto]
-# +--> DIR=file (htmldocs) [by xmlto]
-# +--> man/ (mandocs) [by xmlto]
-
-
-# for PDF and PS output you can choose between xmlto and docbook-utils tools
-PDF_METHOD = $(prefer-db2x)
-PS_METHOD = $(prefer-db2x)
-
-
-targets += $(DOCBOOKS)
-BOOKS := $(addprefix $(obj)/,$(DOCBOOKS))
-xmldocs: $(BOOKS)
-sgmldocs: xmldocs
-
-PS := $(patsubst %.xml, %.ps, $(BOOKS))
-psdocs: $(PS)
-
-PDF := $(patsubst %.xml, %.pdf, $(BOOKS))
-pdfdocs: $(PDF)
-
-HTML := $(sort $(patsubst %.xml, %.html, $(BOOKS)))
-htmldocs: $(HTML)
- $(call cmd,build_main_index)
-
-MAN := $(patsubst %.xml, %.9, $(BOOKS))
-mandocs: $(MAN)
- find $(obj)/man -name '*.9' | xargs gzip -nf
-
-# Default location for installed man pages
-export INSTALL_MAN_PATH = $(objtree)/usr
-
-installmandocs: mandocs
- mkdir -p $(INSTALL_MAN_PATH)/man/man9/
- find $(obj)/man -name '*.9.gz' -printf '%h %f\n' | \
- sort -k 2 -k 1 | uniq -f 1 | sed -e 's: :/:' | \
- xargs install -m 644 -t $(INSTALL_MAN_PATH)/man/man9/
-
-# no-op for the DocBook toolchain
-epubdocs:
-latexdocs:
-linkcheckdocs:
-
-###
-#External programs used
-KERNELDOCXMLREF = $(srctree)/scripts/kernel-doc-xml-ref
-KERNELDOC = $(srctree)/scripts/kernel-doc
-DOCPROC = $(objtree)/scripts/docproc
-CHECK_LC_CTYPE = $(objtree)/scripts/check-lc_ctype
-
-# Use a fixed encoding - UTF-8 if the C library has support built-in
-# or ASCII if not
-LC_CTYPE := $(call try-run, LC_CTYPE=C.UTF-8 $(CHECK_LC_CTYPE),C.UTF-8,C)
-export LC_CTYPE
-
-XMLTOFLAGS = -m $(srctree)/$(src)/stylesheet.xsl
-XMLTOFLAGS += --skip-validation
-
-###
-# DOCPROC is used for two purposes:
-# 1) To generate a dependency list for a .tmpl file
-# 2) To preprocess a .tmpl file and call kernel-doc with
-# appropriate parameters.
-# The following rules are used to generate the .xml documentation
-# required to generate the final targets. (ps, pdf, html).
-quiet_cmd_docproc = DOCPROC $@
- cmd_docproc = SRCTREE=$(srctree)/ $(DOCPROC) doc $< >$@
-define rule_docproc
- set -e; \
- $(if $($(quiet)cmd_$(1)),echo ' $($(quiet)cmd_$(1))';) \
- $(cmd_$(1)); \
- ( \
- echo 'cmd_$@ := $(cmd_$(1))'; \
- echo $@: `SRCTREE=$(srctree) $(DOCPROC) depend $<`; \
- ) > $(dir $@).$(notdir $@).cmd
-endef
-
-%.xml: %.tmpl $(KERNELDOC) $(DOCPROC) $(KERNELDOCXMLREF) FORCE
- $(call if_changed_rule,docproc)
-
-# Tell kbuild to always build the programs
-always := $(hostprogs-y)
-
-notfoundtemplate = echo "*** You have to install docbook-utils or xmlto ***"; \
- exit 1
-db2xtemplate = db2TYPE -o $(dir $@) $<
-xmltotemplate = xmlto TYPE $(XMLTOFLAGS) -o $(dir $@) $<
-
-# determine which methods are available
-ifeq ($(shell which db2ps >/dev/null 2>&1 && echo found),found)
- use-db2x = db2x
- prefer-db2x = db2x
-else
- use-db2x = notfound
- prefer-db2x = $(use-xmlto)
-endif
-ifeq ($(shell which xmlto >/dev/null 2>&1 && echo found),found)
- use-xmlto = xmlto
- prefer-xmlto = xmlto
-else
- use-xmlto = notfound
- prefer-xmlto = $(use-db2x)
-endif
-
-# the commands, generated from the chosen template
-quiet_cmd_db2ps = PS $@
- cmd_db2ps = $(subst TYPE,ps, $($(PS_METHOD)template))
-%.ps : %.xml
- $(call cmd,db2ps)
-
-quiet_cmd_db2pdf = PDF $@
- cmd_db2pdf = $(subst TYPE,pdf, $($(PDF_METHOD)template))
-%.pdf : %.xml
- $(call cmd,db2pdf)
-
-
-index = index.html
-main_idx = $(obj)/$(index)
-quiet_cmd_build_main_index = HTML $(main_idx)
- cmd_build_main_index = rm -rf $(main_idx); \
- echo '<h1>Linux Kernel HTML Documentation</h1>' >> $(main_idx) && \
- echo '<h2>Kernel Version: $(KERNELVERSION)</h2>' >> $(main_idx) && \
- cat $(HTML) >> $(main_idx)
-
-quiet_cmd_db2html = HTML $@
- cmd_db2html = xmlto html $(XMLTOFLAGS) -o $(patsubst %.html,%,$@) $< && \
- echo '<a HREF="$(patsubst %.html,%,$(notdir $@))/index.html"> \
- $(patsubst %.html,%,$(notdir $@))</a><p>' > $@
-
-###
-# Rules to create an aux XML and .db, and use them to re-process the DocBook XML
-# to fill internal hyperlinks
- gen_aux_xml = :
- quiet_gen_aux_xml = echo ' XMLREF $@'
-silent_gen_aux_xml = :
-%.aux.xml: %.xml
- @$($(quiet)gen_aux_xml)
- @rm -rf $@
- @(cat $< | egrep "^<refentry id" | egrep -o "\".*\"" | cut -f 2 -d \" > $<.db)
- @$(KERNELDOCXMLREF) -db $<.db $< > $@
-.PRECIOUS: %.aux.xml
-
-%.html: %.aux.xml
- @(which xmlto > /dev/null 2>&1) || \
- (echo "*** You need to install xmlto ***"; \
- exit 1)
- @rm -rf $@ $(patsubst %.html,%,$@)
- $(call cmd,db2html)
- @if [ ! -z "$(PNG-$(basename $(notdir $@)))" ]; then \
- cp $(PNG-$(basename $(notdir $@))) $(patsubst %.html,%,$@); fi
-
-quiet_cmd_db2man = MAN $@
- cmd_db2man = if grep -q refentry $<; then xmlto man $(XMLTOFLAGS) -o $(obj)/man/$(*F) $< ; fi
-%.9 : %.xml
- @(which xmlto > /dev/null 2>&1) || \
- (echo "*** You need to install xmlto ***"; \
- exit 1)
- $(Q)mkdir -p $(obj)/man/$(*F)
- $(call cmd,db2man)
- @touch $@
-
-###
-# Rules to generate postscripts and PNG images from .fig format files
-quiet_cmd_fig2eps = FIG2EPS $@
- cmd_fig2eps = fig2dev -Leps $< $@
-
-%.eps: %.fig
- @(which fig2dev > /dev/null 2>&1) || \
- (echo "*** You need to install transfig ***"; \
- exit 1)
- $(call cmd,fig2eps)
-
-quiet_cmd_fig2png = FIG2PNG $@
- cmd_fig2png = fig2dev -Lpng $< $@
-
-%.png: %.fig
- @(which fig2dev > /dev/null 2>&1) || \
- (echo "*** You need to install transfig ***"; \
- exit 1)
- $(call cmd,fig2png)
-
-###
-# Rule to convert a .c file to inline XML documentation
- gen_xml = :
- quiet_gen_xml = echo ' GEN $@'
-silent_gen_xml = :
-%.xml: %.c
- @$($(quiet)gen_xml)
- @( \
- echo "<programlisting>"; \
- expand --tabs=8 < $< | \
- sed -e "s/&/\\&/g" \
- -e "s/</\\</g" \
- -e "s/>/\\>/g"; \
- echo "</programlisting>") > $@
-
-endif # DOCBOOKS=""
-endif # SPHINDIR=...
-
-###
-# Help targets as used by the top-level makefile
-dochelp:
- @echo ' Linux kernel internal documentation in different formats (DocBook):'
- @echo ' htmldocs - HTML'
- @echo ' pdfdocs - PDF'
- @echo ' psdocs - Postscript'
- @echo ' xmldocs - XML DocBook'
- @echo ' mandocs - man pages'
- @echo ' installmandocs - install man pages generated by mandocs to INSTALL_MAN_PATH'; \
- echo ' (default: $(INSTALL_MAN_PATH))'; \
- echo ''
- @echo ' cleandocs - clean all generated DocBook files'
- @echo
- @echo ' make DOCBOOKS="s1.xml s2.xml" [target] Generate only docs s1.xml s2.xml'
- @echo ' valid values for DOCBOOKS are: $(DOCBOOKS)'
- @echo
- @echo " make DOCBOOKS=\"\" [target] Don't generate docs from Docbook"
- @echo ' This is useful to generate only the ReST docs (Sphinx)'
-
-
-###
-# Temporary files left by various tools
-clean-files := $(DOCBOOKS) \
- $(patsubst %.xml, %.dvi, $(DOCBOOKS)) \
- $(patsubst %.xml, %.aux, $(DOCBOOKS)) \
- $(patsubst %.xml, %.tex, $(DOCBOOKS)) \
- $(patsubst %.xml, %.log, $(DOCBOOKS)) \
- $(patsubst %.xml, %.out, $(DOCBOOKS)) \
- $(patsubst %.xml, %.ps, $(DOCBOOKS)) \
- $(patsubst %.xml, %.pdf, $(DOCBOOKS)) \
- $(patsubst %.xml, %.html, $(DOCBOOKS)) \
- $(patsubst %.xml, %.9, $(DOCBOOKS)) \
- $(patsubst %.xml, %.aux.xml, $(DOCBOOKS)) \
- $(patsubst %.xml, %.xml.db, $(DOCBOOKS)) \
- $(patsubst %.xml, %.xml, $(DOCBOOKS)) \
- $(patsubst %.xml, .%.xml.cmd, $(DOCBOOKS)) \
- $(index)
-
-clean-dirs := $(patsubst %.xml,%,$(DOCBOOKS)) man
-
-cleandocs:
- $(Q)rm -f $(call objectify, $(clean-files))
- $(Q)rm -rf $(call objectify, $(clean-dirs))
-
-# Declare the contents of the .PHONY variable as phony. We keep that
-# information in a variable so we can use it in if_changed and friends.
-
-.PHONY: $(PHONY)
+++ /dev/null
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
-
-<book id="Linux-filesystems-API">
- <bookinfo>
- <title>Linux Filesystems API</title>
-
- <legalnotice>
- <para>
- This documentation is free software; you can redistribute
- it and/or modify it under the terms of the GNU General Public
- License as published by the Free Software Foundation; either
- version 2 of the License, or (at your option) any later
- version.
- </para>
-
- <para>
- This program is distributed in the hope that it will be
- useful, but WITHOUT ANY WARRANTY; without even the implied
- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- See the GNU General Public License for more details.
- </para>
-
- <para>
- You should have received a copy of the GNU General Public
- License along with this program; if not, write to the Free
- Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
- MA 02111-1307 USA
- </para>
-
- <para>
- For more details see the file COPYING in the source
- distribution of Linux.
- </para>
- </legalnotice>
- </bookinfo>
-
-<toc></toc>
-
- <chapter id="vfs">
- <title>The Linux VFS</title>
- <sect1 id="the_filesystem_types"><title>The Filesystem types</title>
-!Iinclude/linux/fs.h
- </sect1>
- <sect1 id="the_directory_cache"><title>The Directory Cache</title>
-!Efs/dcache.c
-!Iinclude/linux/dcache.h
- </sect1>
- <sect1 id="inode_handling"><title>Inode Handling</title>
-!Efs/inode.c
-!Efs/bad_inode.c
- </sect1>
- <sect1 id="registration_and_superblocks"><title>Registration and Superblocks</title>
-!Efs/super.c
- </sect1>
- <sect1 id="file_locks"><title>File Locks</title>
-!Efs/locks.c
-!Ifs/locks.c
- </sect1>
- <sect1 id="other_functions"><title>Other Functions</title>
-!Efs/mpage.c
-!Efs/namei.c
-!Efs/buffer.c
-!Eblock/bio.c
-!Efs/seq_file.c
-!Efs/filesystems.c
-!Efs/fs-writeback.c
-!Efs/block_dev.c
- </sect1>
- </chapter>
-
- <chapter id="proc">
- <title>The proc filesystem</title>
-
- <sect1 id="sysctl_interface"><title>sysctl interface</title>
-!Ekernel/sysctl.c
- </sect1>
-
- <sect1 id="proc_filesystem_interface"><title>proc filesystem interface</title>
-!Ifs/proc/base.c
- </sect1>
- </chapter>
-
- <chapter id="fs_events">
- <title>Events based on file descriptors</title>
-!Efs/eventfd.c
- </chapter>
-
- <chapter id="sysfs">
- <title>The Filesystem for Exporting Kernel Objects</title>
-!Efs/sysfs/file.c
-!Efs/sysfs/symlink.c
- </chapter>
-
- <chapter id="debugfs">
- <title>The debugfs filesystem</title>
-
- <sect1 id="debugfs_interface"><title>debugfs interface</title>
-!Efs/debugfs/inode.c
-!Efs/debugfs/file.c
- </sect1>
- </chapter>
-
- <chapter id="LinuxJDBAPI">
- <chapterinfo>
- <title>The Linux Journalling API</title>
-
- <authorgroup>
- <author>
- <firstname>Roger</firstname>
- <surname>Gammans</surname>
- <affiliation>
- <address>
- <email>rgammans@computer-surgery.co.uk</email>
- </address>
- </affiliation>
- </author>
- </authorgroup>
-
- <authorgroup>
- <author>
- <firstname>Stephen</firstname>
- <surname>Tweedie</surname>
- <affiliation>
- <address>
- <email>sct@redhat.com</email>
- </address>
- </affiliation>
- </author>
- </authorgroup>
-
- <copyright>
- <year>2002</year>
- <holder>Roger Gammans</holder>
- </copyright>
- </chapterinfo>
-
- <title>The Linux Journalling API</title>
-
- <sect1 id="journaling_overview">
- <title>Overview</title>
- <sect2 id="journaling_details">
- <title>Details</title>
-<para>
-The journalling layer is easy to use. You need to
-first of all create a journal_t data structure. There are
-two calls to do this dependent on how you decide to allocate the physical
-media on which the journal resides. The jbd2_journal_init_inode() call
-is for journals stored in filesystem inodes, or the jbd2_journal_init_dev()
-call can be used for journal stored on a raw device (in a continuous range
-of blocks). A journal_t is a typedef for a struct pointer, so when
-you are finally finished make sure you call jbd2_journal_destroy() on it
-to free up any used kernel memory.
-</para>
-
-<para>
-Once you have got your journal_t object you need to 'mount' or load the journal
-file. The journalling layer expects the space for the journal was already
-allocated and initialized properly by the userspace tools. When loading the
-journal you must call jbd2_journal_load() to process journal contents. If the
-client file system detects the journal contents does not need to be processed
-(or even need not have valid contents), it may call jbd2_journal_wipe() to
-clear the journal contents before calling jbd2_journal_load().
-</para>
-
-<para>
-Note that jbd2_journal_wipe(..,0) calls jbd2_journal_skip_recovery() for you if
-it detects any outstanding transactions in the journal and similarly
-jbd2_journal_load() will call jbd2_journal_recover() if necessary. I would
-advise reading ext4_load_journal() in fs/ext4/super.c for examples on this
-stage.
-</para>
-
-<para>
-Now you can go ahead and start modifying the underlying
-filesystem. Almost.
-</para>
-
-<para>
-
-You still need to actually journal your filesystem changes, this
-is done by wrapping them into transactions. Additionally you
-also need to wrap the modification of each of the buffers
-with calls to the journal layer, so it knows what the modifications
-you are actually making are. To do this use jbd2_journal_start() which
-returns a transaction handle.
-</para>
-
-<para>
-jbd2_journal_start()
-and its counterpart jbd2_journal_stop(), which indicates the end of a
-transaction are nestable calls, so you can reenter a transaction if necessary,
-but remember you must call jbd2_journal_stop() the same number of times as
-jbd2_journal_start() before the transaction is completed (or more accurately
-leaves the update phase). Ext4/VFS makes use of this feature to simplify
-handling of inode dirtying, quota support, etc.
-</para>
-
-<para>
-Inside each transaction you need to wrap the modifications to the
-individual buffers (blocks). Before you start to modify a buffer you
-need to call jbd2_journal_get_{create,write,undo}_access() as appropriate,
-this allows the journalling layer to copy the unmodified data if it
-needs to. After all the buffer may be part of a previously uncommitted
-transaction.
-At this point you are at last ready to modify a buffer, and once
-you are have done so you need to call jbd2_journal_dirty_{meta,}data().
-Or if you've asked for access to a buffer you now know is now longer
-required to be pushed back on the device you can call jbd2_journal_forget()
-in much the same way as you might have used bforget() in the past.
-</para>
-
-<para>
-A jbd2_journal_flush() may be called at any time to commit and checkpoint
-all your transactions.
-</para>
-
-<para>
-Then at umount time , in your put_super() you can then call jbd2_journal_destroy()
-to clean up your in-core journal object.
-</para>
-
-<para>
-Unfortunately there a couple of ways the journal layer can cause a deadlock.
-The first thing to note is that each task can only have
-a single outstanding transaction at any one time, remember nothing
-commits until the outermost jbd2_journal_stop(). This means
-you must complete the transaction at the end of each file/inode/address
-etc. operation you perform, so that the journalling system isn't re-entered
-on another journal. Since transactions can't be nested/batched
-across differing journals, and another filesystem other than
-yours (say ext4) may be modified in a later syscall.
-</para>
-
-<para>
-The second case to bear in mind is that jbd2_journal_start() can
-block if there isn't enough space in the journal for your transaction
-(based on the passed nblocks param) - when it blocks it merely(!) needs to
-wait for transactions to complete and be committed from other tasks,
-so essentially we are waiting for jbd2_journal_stop(). So to avoid
-deadlocks you must treat jbd2_journal_start/stop() as if they
-were semaphores and include them in your semaphore ordering rules to prevent
-deadlocks. Note that jbd2_journal_extend() has similar blocking behaviour to
-jbd2_journal_start() so you can deadlock here just as easily as on
-jbd2_journal_start().
-</para>
-
-<para>
-Try to reserve the right number of blocks the first time. ;-). This will
-be the maximum number of blocks you are going to touch in this transaction.
-I advise having a look at at least ext4_jbd.h to see the basis on which
-ext4 uses to make these decisions.
-</para>
-
-<para>
-Another wriggle to watch out for is your on-disk block allocation strategy.
-Why? Because, if you do a delete, you need to ensure you haven't reused any
-of the freed blocks until the transaction freeing these blocks commits. If you
-reused these blocks and crash happens, there is no way to restore the contents
-of the reallocated blocks at the end of the last fully committed transaction.
-
-One simple way of doing this is to mark blocks as free in internal in-memory
-block allocation structures only after the transaction freeing them commits.
-Ext4 uses journal commit callback for this purpose.
-</para>
-
-<para>
-With journal commit callbacks you can ask the journalling layer to call a
-callback function when the transaction is finally committed to disk, so that
-you can do some of your own management. You ask the journalling layer for
-calling the callback by simply setting journal->j_commit_callback function
-pointer and that function is called after each transaction commit. You can also
-use transaction->t_private_list for attaching entries to a transaction that
-need processing when the transaction commits.
-</para>
-
-<para>
-JBD2 also provides a way to block all transaction updates via
-jbd2_journal_{un,}lock_updates(). Ext4 uses this when it wants a window with a
-clean and stable fs for a moment. E.g.
-</para>
-
-<programlisting>
-
- jbd2_journal_lock_updates() //stop new stuff happening..
- jbd2_journal_flush() // checkpoint everything.
- ..do stuff on stable fs
- jbd2_journal_unlock_updates() // carry on with filesystem use.
-</programlisting>
-
-<para>
-The opportunities for abuse and DOS attacks with this should be obvious,
-if you allow unprivileged userspace to trigger codepaths containing these
-calls.
-</para>
-
- </sect2>
-
- <sect2 id="jbd_summary">
- <title>Summary</title>
-<para>
-Using the journal is a matter of wrapping the different context changes,
-being each mount, each modification (transaction) and each changed buffer
-to tell the journalling layer about them.
-</para>
-
- </sect2>
-
- </sect1>
-
- <sect1 id="data_types">
- <title>Data Types</title>
- <para>
- The journalling layer uses typedefs to 'hide' the concrete definitions
- of the structures used. As a client of the JBD2 layer you can
- just rely on the using the pointer as a magic cookie of some sort.
-
- Obviously the hiding is not enforced as this is 'C'.
- </para>
- <sect2 id="structures"><title>Structures</title>
-!Iinclude/linux/jbd2.h
- </sect2>
- </sect1>
-
- <sect1 id="functions">
- <title>Functions</title>
- <para>
- The functions here are split into two groups those that
- affect a journal as a whole, and those which are used to
- manage transactions
- </para>
- <sect2 id="journal_level"><title>Journal Level</title>
-!Efs/jbd2/journal.c
-!Ifs/jbd2/recovery.c
- </sect2>
- <sect2 id="transaction_level"><title>Transasction Level</title>
-!Efs/jbd2/transaction.c
- </sect2>
- </sect1>
- <sect1 id="see_also">
- <title>See also</title>
- <para>
- <citation>
- <ulink url="http://kernel.org/pub/linux/kernel/people/sct/ext3/journal-design.ps.gz">
- Journaling the Linux ext2fs Filesystem, LinuxExpo 98, Stephen Tweedie
- </ulink>
- </citation>
- </para>
- <para>
- <citation>
- <ulink url="http://olstrans.sourceforge.net/release/OLS2000-ext3/OLS2000-ext3.html">
- Ext3 Journalling FileSystem, OLS 2000, Dr. Stephen Tweedie
- </ulink>
- </citation>
- </para>
- </sect1>
-
- </chapter>
-
- <chapter id="splice">
- <title>splice API</title>
- <para>
- splice is a method for moving blocks of data around inside the
- kernel, without continually transferring them between the kernel
- and user space.
- </para>
-!Ffs/splice.c
- </chapter>
-
- <chapter id="pipes">
- <title>pipes API</title>
- <para>
- Pipe interfaces are all for in-kernel (builtin image) use.
- They are not exported for use by modules.
- </para>
-!Iinclude/linux/pipe_fs_i.h
-!Ffs/pipe.c
- </chapter>
-
-</book>
+++ /dev/null
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
-
-<book id="lk-hacking-guide">
- <bookinfo>
- <title>Unreliable Guide To Hacking The Linux Kernel</title>
-
- <authorgroup>
- <author>
- <firstname>Rusty</firstname>
- <surname>Russell</surname>
- <affiliation>
- <address>
- <email>rusty@rustcorp.com.au</email>
- </address>
- </affiliation>
- </author>
- </authorgroup>
-
- <copyright>
- <year>2005</year>
- <holder>Rusty Russell</holder>
- </copyright>
-
- <legalnotice>
- <para>
- This documentation is free software; you can redistribute
- it and/or modify it under the terms of the GNU General Public
- License as published by the Free Software Foundation; either
- version 2 of the License, or (at your option) any later
- version.
- </para>
-
- <para>
- This program is distributed in the hope that it will be
- useful, but WITHOUT ANY WARRANTY; without even the implied
- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- See the GNU General Public License for more details.
- </para>
-
- <para>
- You should have received a copy of the GNU General Public
- License along with this program; if not, write to the Free
- Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
- MA 02111-1307 USA
- </para>
-
- <para>
- For more details see the file COPYING in the source
- distribution of Linux.
- </para>
- </legalnotice>
-
- <releaseinfo>
- This is the first release of this document as part of the kernel tarball.
- </releaseinfo>
-
- </bookinfo>
-
- <toc></toc>
-
- <chapter id="introduction">
- <title>Introduction</title>
- <para>
- Welcome, gentle reader, to Rusty's Remarkably Unreliable Guide to Linux
- Kernel Hacking. This document describes the common routines and
- general requirements for kernel code: its goal is to serve as a
- primer for Linux kernel development for experienced C
- programmers. I avoid implementation details: that's what the
- code is for, and I ignore whole tracts of useful routines.
- </para>
- <para>
- Before you read this, please understand that I never wanted to
- write this document, being grossly under-qualified, but I always
- wanted to read it, and this was the only way. I hope it will
- grow into a compendium of best practice, common starting points
- and random information.
- </para>
- </chapter>
-
- <chapter id="basic-players">
- <title>The Players</title>
-
- <para>
- At any time each of the CPUs in a system can be:
- </para>
-
- <itemizedlist>
- <listitem>
- <para>
- not associated with any process, serving a hardware interrupt;
- </para>
- </listitem>
-
- <listitem>
- <para>
- not associated with any process, serving a softirq or tasklet;
- </para>
- </listitem>
-
- <listitem>
- <para>
- running in kernel space, associated with a process (user context);
- </para>
- </listitem>
-
- <listitem>
- <para>
- running a process in user space.
- </para>
- </listitem>
- </itemizedlist>
-
- <para>
- There is an ordering between these. The bottom two can preempt
- each other, but above that is a strict hierarchy: each can only be
- preempted by the ones above it. For example, while a softirq is
- running on a CPU, no other softirq will preempt it, but a hardware
- interrupt can. However, any other CPUs in the system execute
- independently.
- </para>
-
- <para>
- We'll see a number of ways that the user context can block
- interrupts, to become truly non-preemptable.
- </para>
-
- <sect1 id="basics-usercontext">
- <title>User Context</title>
-
- <para>
- User context is when you are coming in from a system call or other
- trap: like userspace, you can be preempted by more important tasks
- and by interrupts. You can sleep, by calling
- <function>schedule()</function>.
- </para>
-
- <note>
- <para>
- You are always in user context on module load and unload,
- and on operations on the block device layer.
- </para>
- </note>
-
- <para>
- In user context, the <varname>current</varname> pointer (indicating
- the task we are currently executing) is valid, and
- <function>in_interrupt()</function>
- (<filename>include/linux/interrupt.h</filename>) is <returnvalue>false
- </returnvalue>.
- </para>
-
- <caution>
- <para>
- Beware that if you have preemption or softirqs disabled
- (see below), <function>in_interrupt()</function> will return a
- false positive.
- </para>
- </caution>
- </sect1>
-
- <sect1 id="basics-hardirqs">
- <title>Hardware Interrupts (Hard IRQs)</title>
-
- <para>
- Timer ticks, <hardware>network cards</hardware> and
- <hardware>keyboard</hardware> are examples of real
- hardware which produce interrupts at any time. The kernel runs
- interrupt handlers, which services the hardware. The kernel
- guarantees that this handler is never re-entered: if the same
- interrupt arrives, it is queued (or dropped). Because it
- disables interrupts, this handler has to be fast: frequently it
- simply acknowledges the interrupt, marks a 'software interrupt'
- for execution and exits.
- </para>
-
- <para>
- You can tell you are in a hardware interrupt, because
- <function>in_irq()</function> returns <returnvalue>true</returnvalue>.
- </para>
- <caution>
- <para>
- Beware that this will return a false positive if interrupts are disabled
- (see below).
- </para>
- </caution>
- </sect1>
-
- <sect1 id="basics-softirqs">
- <title>Software Interrupt Context: Softirqs and Tasklets</title>
-
- <para>
- Whenever a system call is about to return to userspace, or a
- hardware interrupt handler exits, any 'software interrupts'
- which are marked pending (usually by hardware interrupts) are
- run (<filename>kernel/softirq.c</filename>).
- </para>
-
- <para>
- Much of the real interrupt handling work is done here. Early in
- the transition to <acronym>SMP</acronym>, there were only 'bottom
- halves' (BHs), which didn't take advantage of multiple CPUs. Shortly
- after we switched from wind-up computers made of match-sticks and snot,
- we abandoned this limitation and switched to 'softirqs'.
- </para>
-
- <para>
- <filename class="headerfile">include/linux/interrupt.h</filename> lists the
- different softirqs. A very important softirq is the
- timer softirq (<filename
- class="headerfile">include/linux/timer.h</filename>): you can
- register to have it call functions for you in a given length of
- time.
- </para>
-
- <para>
- Softirqs are often a pain to deal with, since the same softirq
- will run simultaneously on more than one CPU. For this reason,
- tasklets (<filename
- class="headerfile">include/linux/interrupt.h</filename>) are more
- often used: they are dynamically-registrable (meaning you can have
- as many as you want), and they also guarantee that any tasklet
- will only run on one CPU at any time, although different tasklets
- can run simultaneously.
- </para>
- <caution>
- <para>
- The name 'tasklet' is misleading: they have nothing to do with 'tasks',
- and probably more to do with some bad vodka Alexey Kuznetsov had at the
- time.
- </para>
- </caution>
-
- <para>
- You can tell you are in a softirq (or tasklet)
- using the <function>in_softirq()</function> macro
- (<filename class="headerfile">include/linux/interrupt.h</filename>).
- </para>
- <caution>
- <para>
- Beware that this will return a false positive if a bh lock (see below)
- is held.
- </para>
- </caution>
- </sect1>
- </chapter>
-
- <chapter id="basic-rules">
- <title>Some Basic Rules</title>
-
- <variablelist>
- <varlistentry>
- <term>No memory protection</term>
- <listitem>
- <para>
- If you corrupt memory, whether in user context or
- interrupt context, the whole machine will crash. Are you
- sure you can't do what you want in userspace?
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term>No floating point or <acronym>MMX</acronym></term>
- <listitem>
- <para>
- The <acronym>FPU</acronym> context is not saved; even in user
- context the <acronym>FPU</acronym> state probably won't
- correspond with the current process: you would mess with some
- user process' <acronym>FPU</acronym> state. If you really want
- to do this, you would have to explicitly save/restore the full
- <acronym>FPU</acronym> state (and avoid context switches). It
- is generally a bad idea; use fixed point arithmetic first.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term>A rigid stack limit</term>
- <listitem>
- <para>
- Depending on configuration options the kernel stack is about 3K to 6K for most 32-bit architectures: it's
- about 14K on most 64-bit archs, and often shared with interrupts
- so you can't use it all. Avoid deep recursion and huge local
- arrays on the stack (allocate them dynamically instead).
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term>The Linux kernel is portable</term>
- <listitem>
- <para>
- Let's keep it that way. Your code should be 64-bit clean,
- and endian-independent. You should also minimize CPU
- specific stuff, e.g. inline assembly should be cleanly
- encapsulated and minimized to ease porting. Generally it
- should be restricted to the architecture-dependent part of
- the kernel tree.
- </para>
- </listitem>
- </varlistentry>
- </variablelist>
- </chapter>
-
- <chapter id="ioctls">
- <title>ioctls: Not writing a new system call</title>
-
- <para>
- A system call generally looks like this
- </para>
-
- <programlisting>
-asmlinkage long sys_mycall(int arg)
-{
- return 0;
-}
- </programlisting>
-
- <para>
- First, in most cases you don't want to create a new system call.
- You create a character device and implement an appropriate ioctl
- for it. This is much more flexible than system calls, doesn't have
- to be entered in every architecture's
- <filename class="headerfile">include/asm/unistd.h</filename> and
- <filename>arch/kernel/entry.S</filename> file, and is much more
- likely to be accepted by Linus.
- </para>
-
- <para>
- If all your routine does is read or write some parameter, consider
- implementing a <function>sysfs</function> interface instead.
- </para>
-
- <para>
- Inside the ioctl you're in user context to a process. When a
- error occurs you return a negated errno (see
- <filename class="headerfile">include/linux/errno.h</filename>),
- otherwise you return <returnvalue>0</returnvalue>.
- </para>
-
- <para>
- After you slept you should check if a signal occurred: the
- Unix/Linux way of handling signals is to temporarily exit the
- system call with the <constant>-ERESTARTSYS</constant> error. The
- system call entry code will switch back to user context, process
- the signal handler and then your system call will be restarted
- (unless the user disabled that). So you should be prepared to
- process the restart, e.g. if you're in the middle of manipulating
- some data structure.
- </para>
-
- <programlisting>
-if (signal_pending(current))
- return -ERESTARTSYS;
- </programlisting>
-
- <para>
- If you're doing longer computations: first think userspace. If you
- <emphasis>really</emphasis> want to do it in kernel you should
- regularly check if you need to give up the CPU (remember there is
- cooperative multitasking per CPU). Idiom:
- </para>
-
- <programlisting>
-cond_resched(); /* Will sleep */
- </programlisting>
-
- <para>
- A short note on interface design: the UNIX system call motto is
- "Provide mechanism not policy".
- </para>
- </chapter>
-
- <chapter id="deadlock-recipes">
- <title>Recipes for Deadlock</title>
-
- <para>
- You cannot call any routines which may sleep, unless:
- </para>
- <itemizedlist>
- <listitem>
- <para>
- You are in user context.
- </para>
- </listitem>
-
- <listitem>
- <para>
- You do not own any spinlocks.
- </para>
- </listitem>
-
- <listitem>
- <para>
- You have interrupts enabled (actually, Andi Kleen says
- that the scheduling code will enable them for you, but
- that's probably not what you wanted).
- </para>
- </listitem>
- </itemizedlist>
-
- <para>
- Note that some functions may sleep implicitly: common ones are
- the user space access functions (*_user) and memory allocation
- functions without <symbol>GFP_ATOMIC</symbol>.
- </para>
-
- <para>
- You should always compile your kernel
- <symbol>CONFIG_DEBUG_ATOMIC_SLEEP</symbol> on, and it will warn
- you if you break these rules. If you <emphasis>do</emphasis> break
- the rules, you will eventually lock up your box.
- </para>
-
- <para>
- Really.
- </para>
- </chapter>
-
- <chapter id="common-routines">
- <title>Common Routines</title>
-
- <sect1 id="routines-printk">
- <title>
- <function>printk()</function>
- <filename class="headerfile">include/linux/kernel.h</filename>
- </title>
-
- <para>
- <function>printk()</function> feeds kernel messages to the
- console, dmesg, and the syslog daemon. It is useful for debugging
- and reporting errors, and can be used inside interrupt context,
- but use with caution: a machine which has its console flooded with
- printk messages is unusable. It uses a format string mostly
- compatible with ANSI C printf, and C string concatenation to give
- it a first "priority" argument:
- </para>
-
- <programlisting>
-printk(KERN_INFO "i = %u\n", i);
- </programlisting>
-
- <para>
- See <filename class="headerfile">include/linux/kernel.h</filename>;
- for other KERN_ values; these are interpreted by syslog as the
- level. Special case: for printing an IP address use
- </para>
-
- <programlisting>
-__be32 ipaddress;
-printk(KERN_INFO "my ip: %pI4\n", &ipaddress);
- </programlisting>
-
- <para>
- <function>printk()</function> internally uses a 1K buffer and does
- not catch overruns. Make sure that will be enough.
- </para>
-
- <note>
- <para>
- You will know when you are a real kernel hacker
- when you start typoing printf as printk in your user programs :)
- </para>
- </note>
-
- <!--- From the Lions book reader department -->
-
- <note>
- <para>
- Another sidenote: the original Unix Version 6 sources had a
- comment on top of its printf function: "Printf should not be
- used for chit-chat". You should follow that advice.
- </para>
- </note>
- </sect1>
-
- <sect1 id="routines-copy">
- <title>
- <function>copy_[to/from]_user()</function>
- /
- <function>get_user()</function>
- /
- <function>put_user()</function>
- <filename class="headerfile">include/linux/uaccess.h</filename>
- </title>
-
- <para>
- <emphasis>[SLEEPS]</emphasis>
- </para>
-
- <para>
- <function>put_user()</function> and <function>get_user()</function>
- are used to get and put single values (such as an int, char, or
- long) from and to userspace. A pointer into userspace should
- never be simply dereferenced: data should be copied using these
- routines. Both return <constant>-EFAULT</constant> or 0.
- </para>
- <para>
- <function>copy_to_user()</function> and
- <function>copy_from_user()</function> are more general: they copy
- an arbitrary amount of data to and from userspace.
- <caution>
- <para>
- Unlike <function>put_user()</function> and
- <function>get_user()</function>, they return the amount of
- uncopied data (ie. <returnvalue>0</returnvalue> still means
- success).
- </para>
- </caution>
- [Yes, this moronic interface makes me cringe. The flamewar comes up every year or so. --RR.]
- </para>
- <para>
- The functions may sleep implicitly. This should never be called
- outside user context (it makes no sense), with interrupts
- disabled, or a spinlock held.
- </para>
- </sect1>
-
- <sect1 id="routines-kmalloc">
- <title><function>kmalloc()</function>/<function>kfree()</function>
- <filename class="headerfile">include/linux/slab.h</filename></title>
-
- <para>
- <emphasis>[MAY SLEEP: SEE BELOW]</emphasis>
- </para>
-
- <para>
- These routines are used to dynamically request pointer-aligned
- chunks of memory, like malloc and free do in userspace, but
- <function>kmalloc()</function> takes an extra flag word.
- Important values:
- </para>
-
- <variablelist>
- <varlistentry>
- <term>
- <constant>
- GFP_KERNEL
- </constant>
- </term>
- <listitem>
- <para>
- May sleep and swap to free memory. Only allowed in user
- context, but is the most reliable way to allocate memory.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term>
- <constant>
- GFP_ATOMIC
- </constant>
- </term>
- <listitem>
- <para>
- Don't sleep. Less reliable than <constant>GFP_KERNEL</constant>,
- but may be called from interrupt context. You should
- <emphasis>really</emphasis> have a good out-of-memory
- error-handling strategy.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term>
- <constant>
- GFP_DMA
- </constant>
- </term>
- <listitem>
- <para>
- Allocate ISA DMA lower than 16MB. If you don't know what that
- is you don't need it. Very unreliable.
- </para>
- </listitem>
- </varlistentry>
- </variablelist>
-
- <para>
- If you see a <errorname>sleeping function called from invalid
- context</errorname> warning message, then maybe you called a
- sleeping allocation function from interrupt context without
- <constant>GFP_ATOMIC</constant>. You should really fix that.
- Run, don't walk.
- </para>
-
- <para>
- If you are allocating at least <constant>PAGE_SIZE</constant>
- (<filename class="headerfile">include/asm/page.h</filename>) bytes,
- consider using <function>__get_free_pages()</function>
-
- (<filename class="headerfile">include/linux/mm.h</filename>). It
- takes an order argument (0 for page sized, 1 for double page, 2
- for four pages etc.) and the same memory priority flag word as
- above.
- </para>
-
- <para>
- If you are allocating more than a page worth of bytes you can use
- <function>vmalloc()</function>. It'll allocate virtual memory in
- the kernel map. This block is not contiguous in physical memory,
- but the <acronym>MMU</acronym> makes it look like it is for you
- (so it'll only look contiguous to the CPUs, not to external device
- drivers). If you really need large physically contiguous memory
- for some weird device, you have a problem: it is poorly supported
- in Linux because after some time memory fragmentation in a running
- kernel makes it hard. The best way is to allocate the block early
- in the boot process via the <function>alloc_bootmem()</function>
- routine.
- </para>
-
- <para>
- Before inventing your own cache of often-used objects consider
- using a slab cache in
- <filename class="headerfile">include/linux/slab.h</filename>
- </para>
- </sect1>
-
- <sect1 id="routines-current">
- <title><function>current</function>
- <filename class="headerfile">include/asm/current.h</filename></title>
-
- <para>
- This global variable (really a macro) contains a pointer to
- the current task structure, so is only valid in user context.
- For example, when a process makes a system call, this will
- point to the task structure of the calling process. It is
- <emphasis>not NULL</emphasis> in interrupt context.
- </para>
- </sect1>
-
- <sect1 id="routines-udelay">
- <title><function>mdelay()</function>/<function>udelay()</function>
- <filename class="headerfile">include/asm/delay.h</filename>
- <filename class="headerfile">include/linux/delay.h</filename>
- </title>
-
- <para>
- The <function>udelay()</function> and <function>ndelay()</function> functions can be used for small pauses.
- Do not use large values with them as you risk
- overflow - the helper function <function>mdelay()</function> is useful
- here, or consider <function>msleep()</function>.
- </para>
- </sect1>
-
- <sect1 id="routines-endian">
- <title><function>cpu_to_be32()</function>/<function>be32_to_cpu()</function>/<function>cpu_to_le32()</function>/<function>le32_to_cpu()</function>
- <filename class="headerfile">include/asm/byteorder.h</filename>
- </title>
-
- <para>
- The <function>cpu_to_be32()</function> family (where the "32" can
- be replaced by 64 or 16, and the "be" can be replaced by "le") are
- the general way to do endian conversions in the kernel: they
- return the converted value. All variations supply the reverse as
- well: <function>be32_to_cpu()</function>, etc.
- </para>
-
- <para>
- There are two major variations of these functions: the pointer
- variation, such as <function>cpu_to_be32p()</function>, which take
- a pointer to the given type, and return the converted value. The
- other variation is the "in-situ" family, such as
- <function>cpu_to_be32s()</function>, which convert value referred
- to by the pointer, and return void.
- </para>
- </sect1>
-
- <sect1 id="routines-local-irqs">
- <title><function>local_irq_save()</function>/<function>local_irq_restore()</function>
- <filename class="headerfile">include/linux/irqflags.h</filename>
- </title>
-
- <para>
- These routines disable hard interrupts on the local CPU, and
- restore them. They are reentrant; saving the previous state in
- their one <varname>unsigned long flags</varname> argument. If you
- know that interrupts are enabled, you can simply use
- <function>local_irq_disable()</function> and
- <function>local_irq_enable()</function>.
- </para>
- </sect1>
-
- <sect1 id="routines-softirqs">
- <title><function>local_bh_disable()</function>/<function>local_bh_enable()</function>
- <filename class="headerfile">include/linux/interrupt.h</filename></title>
-
- <para>
- These routines disable soft interrupts on the local CPU, and
- restore them. They are reentrant; if soft interrupts were
- disabled before, they will still be disabled after this pair
- of functions has been called. They prevent softirqs and tasklets
- from running on the current CPU.
- </para>
- </sect1>
-
- <sect1 id="routines-processorids">
- <title><function>smp_processor_id</function>()
- <filename class="headerfile">include/asm/smp.h</filename></title>
-
- <para>
- <function>get_cpu()</function> disables preemption (so you won't
- suddenly get moved to another CPU) and returns the current
- processor number, between 0 and <symbol>NR_CPUS</symbol>. Note
- that the CPU numbers are not necessarily continuous. You return
- it again with <function>put_cpu()</function> when you are done.
- </para>
- <para>
- If you know you cannot be preempted by another task (ie. you are
- in interrupt context, or have preemption disabled) you can use
- smp_processor_id().
- </para>
- </sect1>
-
- <sect1 id="routines-init">
- <title><type>__init</type>/<type>__exit</type>/<type>__initdata</type>
- <filename class="headerfile">include/linux/init.h</filename></title>
-
- <para>
- After boot, the kernel frees up a special section; functions
- marked with <type>__init</type> and data structures marked with
- <type>__initdata</type> are dropped after boot is complete: similarly
- modules discard this memory after initialization. <type>__exit</type>
- is used to declare a function which is only required on exit: the
- function will be dropped if this file is not compiled as a module.
- See the header file for use. Note that it makes no sense for a function
- marked with <type>__init</type> to be exported to modules with
- <function>EXPORT_SYMBOL()</function> - this will break.
- </para>
-
- </sect1>
-
- <sect1 id="routines-init-again">
- <title><function>__initcall()</function>/<function>module_init()</function>
- <filename class="headerfile">include/linux/init.h</filename></title>
- <para>
- Many parts of the kernel are well served as a module
- (dynamically-loadable parts of the kernel). Using the
- <function>module_init()</function> and
- <function>module_exit()</function> macros it is easy to write code
- without #ifdefs which can operate both as a module or built into
- the kernel.
- </para>
-
- <para>
- The <function>module_init()</function> macro defines which
- function is to be called at module insertion time (if the file is
- compiled as a module), or at boot time: if the file is not
- compiled as a module the <function>module_init()</function> macro
- becomes equivalent to <function>__initcall()</function>, which
- through linker magic ensures that the function is called on boot.
- </para>
-
- <para>
- The function can return a negative error number to cause
- module loading to fail (unfortunately, this has no effect if
- the module is compiled into the kernel). This function is
- called in user context with interrupts enabled, so it can sleep.
- </para>
- </sect1>
-
- <sect1 id="routines-moduleexit">
- <title> <function>module_exit()</function>
- <filename class="headerfile">include/linux/init.h</filename> </title>
-
- <para>
- This macro defines the function to be called at module removal
- time (or never, in the case of the file compiled into the
- kernel). It will only be called if the module usage count has
- reached zero. This function can also sleep, but cannot fail:
- everything must be cleaned up by the time it returns.
- </para>
-
- <para>
- Note that this macro is optional: if it is not present, your
- module will not be removable (except for 'rmmod -f').
- </para>
- </sect1>
-
- <sect1 id="routines-module-use-counters">
- <title> <function>try_module_get()</function>/<function>module_put()</function>
- <filename class="headerfile">include/linux/module.h</filename></title>
-
- <para>
- These manipulate the module usage count, to protect against
- removal (a module also can't be removed if another module uses one
- of its exported symbols: see below). Before calling into module
- code, you should call <function>try_module_get()</function> on
- that module: if it fails, then the module is being removed and you
- should act as if it wasn't there. Otherwise, you can safely enter
- the module, and call <function>module_put()</function> when you're
- finished.
- </para>
-
- <para>
- Most registerable structures have an
- <structfield>owner</structfield> field, such as in the
- <structname>file_operations</structname> structure. Set this field
- to the macro <symbol>THIS_MODULE</symbol>.
- </para>
- </sect1>
-
- <!-- add info on new-style module refcounting here -->
- </chapter>
-
- <chapter id="queues">
- <title>Wait Queues
- <filename class="headerfile">include/linux/wait.h</filename>
- </title>
- <para>
- <emphasis>[SLEEPS]</emphasis>
- </para>
-
- <para>
- A wait queue is used to wait for someone to wake you up when a
- certain condition is true. They must be used carefully to ensure
- there is no race condition. You declare a
- <type>wait_queue_head_t</type>, and then processes which want to
- wait for that condition declare a <type>wait_queue_t</type>
- referring to themselves, and place that in the queue.
- </para>
-
- <sect1 id="queue-declaring">
- <title>Declaring</title>
-
- <para>
- You declare a <type>wait_queue_head_t</type> using the
- <function>DECLARE_WAIT_QUEUE_HEAD()</function> macro, or using the
- <function>init_waitqueue_head()</function> routine in your
- initialization code.
- </para>
- </sect1>
-
- <sect1 id="queue-waitqueue">
- <title>Queuing</title>
-
- <para>
- Placing yourself in the waitqueue is fairly complex, because you
- must put yourself in the queue before checking the condition.
- There is a macro to do this:
- <function>wait_event_interruptible()</function>
-
- <filename class="headerfile">include/linux/wait.h</filename> The
- first argument is the wait queue head, and the second is an
- expression which is evaluated; the macro returns
- <returnvalue>0</returnvalue> when this expression is true, or
- <returnvalue>-ERESTARTSYS</returnvalue> if a signal is received.
- The <function>wait_event()</function> version ignores signals.
- </para>
-
- </sect1>
-
- <sect1 id="queue-waking">
- <title>Waking Up Queued Tasks</title>
-
- <para>
- Call <function>wake_up()</function>
-
- <filename class="headerfile">include/linux/wait.h</filename>;,
- which will wake up every process in the queue. The exception is
- if one has <constant>TASK_EXCLUSIVE</constant> set, in which case
- the remainder of the queue will not be woken. There are other variants
- of this basic function available in the same header.
- </para>
- </sect1>
- </chapter>
-
- <chapter id="atomic-ops">
- <title>Atomic Operations</title>
-
- <para>
- Certain operations are guaranteed atomic on all platforms. The
- first class of operations work on <type>atomic_t</type>
-
- <filename class="headerfile">include/asm/atomic.h</filename>; this
- contains a signed integer (at least 32 bits long), and you must use
- these functions to manipulate or read atomic_t variables.
- <function>atomic_read()</function> and
- <function>atomic_set()</function> get and set the counter,
- <function>atomic_add()</function>,
- <function>atomic_sub()</function>,
- <function>atomic_inc()</function>,
- <function>atomic_dec()</function>, and
- <function>atomic_dec_and_test()</function> (returns
- <returnvalue>true</returnvalue> if it was decremented to zero).
- </para>
-
- <para>
- Yes. It returns <returnvalue>true</returnvalue> (i.e. != 0) if the
- atomic variable is zero.
- </para>
-
- <para>
- Note that these functions are slower than normal arithmetic, and
- so should not be used unnecessarily.
- </para>
-
- <para>
- The second class of atomic operations is atomic bit operations on an
- <type>unsigned long</type>, defined in
-
- <filename class="headerfile">include/linux/bitops.h</filename>. These
- operations generally take a pointer to the bit pattern, and a bit
- number: 0 is the least significant bit.
- <function>set_bit()</function>, <function>clear_bit()</function>
- and <function>change_bit()</function> set, clear, and flip the
- given bit. <function>test_and_set_bit()</function>,
- <function>test_and_clear_bit()</function> and
- <function>test_and_change_bit()</function> do the same thing,
- except return true if the bit was previously set; these are
- particularly useful for atomically setting flags.
- </para>
-
- <para>
- It is possible to call these operations with bit indices greater
- than BITS_PER_LONG. The resulting behavior is strange on big-endian
- platforms though so it is a good idea not to do this.
- </para>
- </chapter>
-
- <chapter id="symbols">
- <title>Symbols</title>
-
- <para>
- Within the kernel proper, the normal linking rules apply
- (ie. unless a symbol is declared to be file scope with the
- <type>static</type> keyword, it can be used anywhere in the
- kernel). However, for modules, a special exported symbol table is
- kept which limits the entry points to the kernel proper. Modules
- can also export symbols.
- </para>
-
- <sect1 id="sym-exportsymbols">
- <title><function>EXPORT_SYMBOL()</function>
- <filename class="headerfile">include/linux/export.h</filename></title>
-
- <para>
- This is the classic method of exporting a symbol: dynamically
- loaded modules will be able to use the symbol as normal.
- </para>
- </sect1>
-
- <sect1 id="sym-exportsymbols-gpl">
- <title><function>EXPORT_SYMBOL_GPL()</function>
- <filename class="headerfile">include/linux/export.h</filename></title>
-
- <para>
- Similar to <function>EXPORT_SYMBOL()</function> except that the
- symbols exported by <function>EXPORT_SYMBOL_GPL()</function> can
- only be seen by modules with a
- <function>MODULE_LICENSE()</function> that specifies a GPL
- compatible license. It implies that the function is considered
- an internal implementation issue, and not really an interface.
- Some maintainers and developers may however
- require EXPORT_SYMBOL_GPL() when adding any new APIs or functionality.
- </para>
- </sect1>
- </chapter>
-
- <chapter id="conventions">
- <title>Routines and Conventions</title>
-
- <sect1 id="conventions-doublelinkedlist">
- <title>Double-linked lists
- <filename class="headerfile">include/linux/list.h</filename></title>
-
- <para>
- There used to be three sets of linked-list routines in the kernel
- headers, but this one is the winner. If you don't have some
- particular pressing need for a single list, it's a good choice.
- </para>
-
- <para>
- In particular, <function>list_for_each_entry</function> is useful.
- </para>
- </sect1>
-
- <sect1 id="convention-returns">
- <title>Return Conventions</title>
-
- <para>
- For code called in user context, it's very common to defy C
- convention, and return <returnvalue>0</returnvalue> for success,
- and a negative error number
- (eg. <returnvalue>-EFAULT</returnvalue>) for failure. This can be
- unintuitive at first, but it's fairly widespread in the kernel.
- </para>
-
- <para>
- Using <function>ERR_PTR()</function>
-
- <filename class="headerfile">include/linux/err.h</filename>; to
- encode a negative error number into a pointer, and
- <function>IS_ERR()</function> and <function>PTR_ERR()</function>
- to get it back out again: avoids a separate pointer parameter for
- the error number. Icky, but in a good way.
- </para>
- </sect1>
-
- <sect1 id="conventions-borkedcompile">
- <title>Breaking Compilation</title>
-
- <para>
- Linus and the other developers sometimes change function or
- structure names in development kernels; this is not done just to
- keep everyone on their toes: it reflects a fundamental change
- (eg. can no longer be called with interrupts on, or does extra
- checks, or doesn't do checks which were caught before). Usually
- this is accompanied by a fairly complete note to the linux-kernel
- mailing list; search the archive. Simply doing a global replace
- on the file usually makes things <emphasis>worse</emphasis>.
- </para>
- </sect1>
-
- <sect1 id="conventions-initialising">
- <title>Initializing structure members</title>
-
- <para>
- The preferred method of initializing structures is to use
- designated initialisers, as defined by ISO C99, eg:
- </para>
- <programlisting>
-static struct block_device_operations opt_fops = {
- .open = opt_open,
- .release = opt_release,
- .ioctl = opt_ioctl,
- .check_media_change = opt_media_change,
-};
- </programlisting>
- <para>
- This makes it easy to grep for, and makes it clear which
- structure fields are set. You should do this because it looks
- cool.
- </para>
- </sect1>
-
- <sect1 id="conventions-gnu-extns">
- <title>GNU Extensions</title>
-
- <para>
- GNU Extensions are explicitly allowed in the Linux kernel.
- Note that some of the more complex ones are not very well
- supported, due to lack of general use, but the following are
- considered standard (see the GCC info page section "C
- Extensions" for more details - Yes, really the info page, the
- man page is only a short summary of the stuff in info).
- </para>
- <itemizedlist>
- <listitem>
- <para>
- Inline functions
- </para>
- </listitem>
- <listitem>
- <para>
- Statement expressions (ie. the ({ and }) constructs).
- </para>
- </listitem>
- <listitem>
- <para>
- Declaring attributes of a function / variable / type
- (__attribute__)
- </para>
- </listitem>
- <listitem>
- <para>
- typeof
- </para>
- </listitem>
- <listitem>
- <para>
- Zero length arrays
- </para>
- </listitem>
- <listitem>
- <para>
- Macro varargs
- </para>
- </listitem>
- <listitem>
- <para>
- Arithmetic on void pointers
- </para>
- </listitem>
- <listitem>
- <para>
- Non-Constant initializers
- </para>
- </listitem>
- <listitem>
- <para>
- Assembler Instructions (not outside arch/ and include/asm/)
- </para>
- </listitem>
- <listitem>
- <para>
- Function names as strings (__func__).
- </para>
- </listitem>
- <listitem>
- <para>
- __builtin_constant_p()
- </para>
- </listitem>
- </itemizedlist>
-
- <para>
- Be wary when using long long in the kernel, the code gcc generates for
- it is horrible and worse: division and multiplication does not work
- on i386 because the GCC runtime functions for it are missing from
- the kernel environment.
- </para>
-
- <!-- FIXME: add a note about ANSI aliasing cleanness -->
- </sect1>
-
- <sect1 id="conventions-cplusplus">
- <title>C++</title>
-
- <para>
- Using C++ in the kernel is usually a bad idea, because the
- kernel does not provide the necessary runtime environment
- and the include files are not tested for it. It is still
- possible, but not recommended. If you really want to do
- this, forget about exceptions at least.
- </para>
- </sect1>
-
- <sect1 id="conventions-ifdef">
- <title>#if</title>
-
- <para>
- It is generally considered cleaner to use macros in header files
- (or at the top of .c files) to abstract away functions rather than
- using `#if' pre-processor statements throughout the source code.
- </para>
- </sect1>
- </chapter>
-
- <chapter id="submitting">
- <title>Putting Your Stuff in the Kernel</title>
-
- <para>
- In order to get your stuff into shape for official inclusion, or
- even to make a neat patch, there's administrative work to be
- done:
- </para>
- <itemizedlist>
- <listitem>
- <para>
- Figure out whose pond you've been pissing in. Look at the top of
- the source files, inside the <filename>MAINTAINERS</filename>
- file, and last of all in the <filename>CREDITS</filename> file.
- You should coordinate with this person to make sure you're not
- duplicating effort, or trying something that's already been
- rejected.
- </para>
-
- <para>
- Make sure you put your name and EMail address at the top of
- any files you create or mangle significantly. This is the
- first place people will look when they find a bug, or when
- <emphasis>they</emphasis> want to make a change.
- </para>
- </listitem>
-
- <listitem>
- <para>
- Usually you want a configuration option for your kernel hack.
- Edit <filename>Kconfig</filename> in the appropriate directory.
- The Config language is simple to use by cut and paste, and there's
- complete documentation in
- <filename>Documentation/kbuild/kconfig-language.txt</filename>.
- </para>
-
- <para>
- In your description of the option, make sure you address both the
- expert user and the user who knows nothing about your feature. Mention
- incompatibilities and issues here. <emphasis> Definitely
- </emphasis> end your description with <quote> if in doubt, say N
- </quote> (or, occasionally, `Y'); this is for people who have no
- idea what you are talking about.
- </para>
- </listitem>
-
- <listitem>
- <para>
- Edit the <filename>Makefile</filename>: the CONFIG variables are
- exported here so you can usually just add a "obj-$(CONFIG_xxx) +=
- xxx.o" line. The syntax is documented in
- <filename>Documentation/kbuild/makefiles.txt</filename>.
- </para>
- </listitem>
-
- <listitem>
- <para>
- Put yourself in <filename>CREDITS</filename> if you've done
- something noteworthy, usually beyond a single file (your name
- should be at the top of the source files anyway).
- <filename>MAINTAINERS</filename> means you want to be consulted
- when changes are made to a subsystem, and hear about bugs; it
- implies a more-than-passing commitment to some part of the code.
- </para>
- </listitem>
-
- <listitem>
- <para>
- Finally, don't forget to read <filename>Documentation/process/submitting-patches.rst</filename>
- and possibly <filename>Documentation/process/submitting-drivers.rst</filename>.
- </para>
- </listitem>
- </itemizedlist>
- </chapter>
-
- <chapter id="cantrips">
- <title>Kernel Cantrips</title>
-
- <para>
- Some favorites from browsing the source. Feel free to add to this
- list.
- </para>
-
- <para>
- <filename>arch/x86/include/asm/delay.h:</filename>
- </para>
- <programlisting>
-#define ndelay(n) (__builtin_constant_p(n) ? \
- ((n) > 20000 ? __bad_ndelay() : __const_udelay((n) * 5ul)) : \
- __ndelay(n))
- </programlisting>
-
- <para>
- <filename>include/linux/fs.h</filename>:
- </para>
- <programlisting>
-/*
- * Kernel pointers have redundant information, so we can use a
- * scheme where we can return either an error code or a dentry
- * pointer with the same return value.
- *
- * This should be a per-architecture thing, to allow different
- * error and pointer decisions.
- */
- #define ERR_PTR(err) ((void *)((long)(err)))
- #define PTR_ERR(ptr) ((long)(ptr))
- #define IS_ERR(ptr) ((unsigned long)(ptr) > (unsigned long)(-1000))
-</programlisting>
-
- <para>
- <filename>arch/x86/include/asm/uaccess_32.h:</filename>
- </para>
-
- <programlisting>
-#define copy_to_user(to,from,n) \
- (__builtin_constant_p(n) ? \
- __constant_copy_to_user((to),(from),(n)) : \
- __generic_copy_to_user((to),(from),(n)))
- </programlisting>
-
- <para>
- <filename>arch/sparc/kernel/head.S:</filename>
- </para>
-
- <programlisting>
-/*
- * Sun people can't spell worth damn. "compatability" indeed.
- * At least we *know* we can't spell, and use a spell-checker.
- */
-
-/* Uh, actually Linus it is I who cannot spell. Too much murky
- * Sparc assembly will do this to ya.
- */
-C_LABEL(cputypvar):
- .asciz "compatibility"
-
-/* Tested on SS-5, SS-10. Probably someone at Sun applied a spell-checker. */
- .align 4
-C_LABEL(cputypvar_sun4m):
- .asciz "compatible"
- </programlisting>
-
- <para>
- <filename>arch/sparc/lib/checksum.S:</filename>
- </para>
-
- <programlisting>
- /* Sun, you just can't beat me, you just can't. Stop trying,
- * give up. I'm serious, I am going to kick the living shit
- * out of you, game over, lights out.
- */
- </programlisting>
- </chapter>
-
- <chapter id="credits">
- <title>Thanks</title>
-
- <para>
- Thanks to Andi Kleen for the idea, answering my questions, fixing
- my mistakes, filling content, etc. Philipp Rumpf for more spelling
- and clarity fixes, and some excellent non-obvious points. Werner
- Almesberger for giving me a great summary of
- <function>disable_irq()</function>, and Jes Sorensen and Andrea
- Arcangeli added caveats. Michael Elizabeth Chastain for checking
- and adding to the Configure section. <!-- Rusty insisted on this
- bit; I didn't do it! --> Telsa Gwynne for teaching me DocBook.
- </para>
- </chapter>
-</book>
-
+++ /dev/null
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
-
-<book id="LKLockingGuide">
- <bookinfo>
- <title>Unreliable Guide To Locking</title>
-
- <authorgroup>
- <author>
- <firstname>Rusty</firstname>
- <surname>Russell</surname>
- <affiliation>
- <address>
- <email>rusty@rustcorp.com.au</email>
- </address>
- </affiliation>
- </author>
- </authorgroup>
-
- <copyright>
- <year>2003</year>
- <holder>Rusty Russell</holder>
- </copyright>
-
- <legalnotice>
- <para>
- This documentation is free software; you can redistribute
- it and/or modify it under the terms of the GNU General Public
- License as published by the Free Software Foundation; either
- version 2 of the License, or (at your option) any later
- version.
- </para>
-
- <para>
- This program is distributed in the hope that it will be
- useful, but WITHOUT ANY WARRANTY; without even the implied
- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- See the GNU General Public License for more details.
- </para>
-
- <para>
- You should have received a copy of the GNU General Public
- License along with this program; if not, write to the Free
- Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
- MA 02111-1307 USA
- </para>
-
- <para>
- For more details see the file COPYING in the source
- distribution of Linux.
- </para>
- </legalnotice>
- </bookinfo>
-
- <toc></toc>
- <chapter id="intro">
- <title>Introduction</title>
- <para>
- Welcome, to Rusty's Remarkably Unreliable Guide to Kernel
- Locking issues. This document describes the locking systems in
- the Linux Kernel in 2.6.
- </para>
- <para>
- With the wide availability of HyperThreading, and <firstterm
- linkend="gloss-preemption">preemption </firstterm> in the Linux
- Kernel, everyone hacking on the kernel needs to know the
- fundamentals of concurrency and locking for
- <firstterm linkend="gloss-smp"><acronym>SMP</acronym></firstterm>.
- </para>
- </chapter>
-
- <chapter id="races">
- <title>The Problem With Concurrency</title>
- <para>
- (Skip this if you know what a Race Condition is).
- </para>
- <para>
- In a normal program, you can increment a counter like so:
- </para>
- <programlisting>
- very_important_count++;
- </programlisting>
-
- <para>
- This is what they would expect to happen:
- </para>
-
- <table>
- <title>Expected Results</title>
-
- <tgroup cols="2" align="left">
-
- <thead>
- <row>
- <entry>Instance 1</entry>
- <entry>Instance 2</entry>
- </row>
- </thead>
-
- <tbody>
- <row>
- <entry>read very_important_count (5)</entry>
- <entry></entry>
- </row>
- <row>
- <entry>add 1 (6)</entry>
- <entry></entry>
- </row>
- <row>
- <entry>write very_important_count (6)</entry>
- <entry></entry>
- </row>
- <row>
- <entry></entry>
- <entry>read very_important_count (6)</entry>
- </row>
- <row>
- <entry></entry>
- <entry>add 1 (7)</entry>
- </row>
- <row>
- <entry></entry>
- <entry>write very_important_count (7)</entry>
- </row>
- </tbody>
-
- </tgroup>
- </table>
-
- <para>
- This is what might happen:
- </para>
-
- <table>
- <title>Possible Results</title>
-
- <tgroup cols="2" align="left">
- <thead>
- <row>
- <entry>Instance 1</entry>
- <entry>Instance 2</entry>
- </row>
- </thead>
-
- <tbody>
- <row>
- <entry>read very_important_count (5)</entry>
- <entry></entry>
- </row>
- <row>
- <entry></entry>
- <entry>read very_important_count (5)</entry>
- </row>
- <row>
- <entry>add 1 (6)</entry>
- <entry></entry>
- </row>
- <row>
- <entry></entry>
- <entry>add 1 (6)</entry>
- </row>
- <row>
- <entry>write very_important_count (6)</entry>
- <entry></entry>
- </row>
- <row>
- <entry></entry>
- <entry>write very_important_count (6)</entry>
- </row>
- </tbody>
- </tgroup>
- </table>
-
- <sect1 id="race-condition">
- <title>Race Conditions and Critical Regions</title>
- <para>
- This overlap, where the result depends on the
- relative timing of multiple tasks, is called a <firstterm>race condition</firstterm>.
- The piece of code containing the concurrency issue is called a
- <firstterm>critical region</firstterm>. And especially since Linux starting running
- on SMP machines, they became one of the major issues in kernel
- design and implementation.
- </para>
- <para>
- Preemption can have the same effect, even if there is only one
- CPU: by preempting one task during the critical region, we have
- exactly the same race condition. In this case the thread which
- preempts might run the critical region itself.
- </para>
- <para>
- The solution is to recognize when these simultaneous accesses
- occur, and use locks to make sure that only one instance can
- enter the critical region at any time. There are many
- friendly primitives in the Linux kernel to help you do this.
- And then there are the unfriendly primitives, but I'll pretend
- they don't exist.
- </para>
- </sect1>
- </chapter>
-
- <chapter id="locks">
- <title>Locking in the Linux Kernel</title>
-
- <para>
- If I could give you one piece of advice: never sleep with anyone
- crazier than yourself. But if I had to give you advice on
- locking: <emphasis>keep it simple</emphasis>.
- </para>
-
- <para>
- Be reluctant to introduce new locks.
- </para>
-
- <para>
- Strangely enough, this last one is the exact reverse of my advice when
- you <emphasis>have</emphasis> slept with someone crazier than yourself.
- And you should think about getting a big dog.
- </para>
-
- <sect1 id="lock-intro">
- <title>Two Main Types of Kernel Locks: Spinlocks and Mutexes</title>
-
- <para>
- There are two main types of kernel locks. The fundamental type
- is the spinlock
- (<filename class="headerfile">include/asm/spinlock.h</filename>),
- which is a very simple single-holder lock: if you can't get the
- spinlock, you keep trying (spinning) until you can. Spinlocks are
- very small and fast, and can be used anywhere.
- </para>
- <para>
- The second type is a mutex
- (<filename class="headerfile">include/linux/mutex.h</filename>): it
- is like a spinlock, but you may block holding a mutex.
- If you can't lock a mutex, your task will suspend itself, and be woken
- up when the mutex is released. This means the CPU can do something
- else while you are waiting. There are many cases when you simply
- can't sleep (see <xref linkend="sleeping-things"/>), and so have to
- use a spinlock instead.
- </para>
- <para>
- Neither type of lock is recursive: see
- <xref linkend="deadlock"/>.
- </para>
- </sect1>
-
- <sect1 id="uniprocessor">
- <title>Locks and Uniprocessor Kernels</title>
-
- <para>
- For kernels compiled without <symbol>CONFIG_SMP</symbol>, and
- without <symbol>CONFIG_PREEMPT</symbol> spinlocks do not exist at
- all. This is an excellent design decision: when no-one else can
- run at the same time, there is no reason to have a lock.
- </para>
-
- <para>
- If the kernel is compiled without <symbol>CONFIG_SMP</symbol>,
- but <symbol>CONFIG_PREEMPT</symbol> is set, then spinlocks
- simply disable preemption, which is sufficient to prevent any
- races. For most purposes, we can think of preemption as
- equivalent to SMP, and not worry about it separately.
- </para>
-
- <para>
- You should always test your locking code with <symbol>CONFIG_SMP</symbol>
- and <symbol>CONFIG_PREEMPT</symbol> enabled, even if you don't have an SMP test box, because it
- will still catch some kinds of locking bugs.
- </para>
-
- <para>
- Mutexes still exist, because they are required for
- synchronization between <firstterm linkend="gloss-usercontext">user
- contexts</firstterm>, as we will see below.
- </para>
- </sect1>
-
- <sect1 id="usercontextlocking">
- <title>Locking Only In User Context</title>
-
- <para>
- If you have a data structure which is only ever accessed from
- user context, then you can use a simple mutex
- (<filename>include/linux/mutex.h</filename>) to protect it. This
- is the most trivial case: you initialize the mutex. Then you can
- call <function>mutex_lock_interruptible()</function> to grab the mutex,
- and <function>mutex_unlock()</function> to release it. There is also a
- <function>mutex_lock()</function>, which should be avoided, because it
- will not return if a signal is received.
- </para>
-
- <para>
- Example: <filename>net/netfilter/nf_sockopt.c</filename> allows
- registration of new <function>setsockopt()</function> and
- <function>getsockopt()</function> calls, with
- <function>nf_register_sockopt()</function>. Registration and
- de-registration are only done on module load and unload (and boot
- time, where there is no concurrency), and the list of registrations
- is only consulted for an unknown <function>setsockopt()</function>
- or <function>getsockopt()</function> system call. The
- <varname>nf_sockopt_mutex</varname> is perfect to protect this,
- especially since the setsockopt and getsockopt calls may well
- sleep.
- </para>
- </sect1>
-
- <sect1 id="lock-user-bh">
- <title>Locking Between User Context and Softirqs</title>
-
- <para>
- If a <firstterm linkend="gloss-softirq">softirq</firstterm> shares
- data with user context, you have two problems. Firstly, the current
- user context can be interrupted by a softirq, and secondly, the
- critical region could be entered from another CPU. This is where
- <function>spin_lock_bh()</function>
- (<filename class="headerfile">include/linux/spinlock.h</filename>) is
- used. It disables softirqs on that CPU, then grabs the lock.
- <function>spin_unlock_bh()</function> does the reverse. (The
- '_bh' suffix is a historical reference to "Bottom Halves", the
- old name for software interrupts. It should really be
- called spin_lock_softirq()' in a perfect world).
- </para>
-
- <para>
- Note that you can also use <function>spin_lock_irq()</function>
- or <function>spin_lock_irqsave()</function> here, which stop
- hardware interrupts as well: see <xref linkend="hardirq-context"/>.
- </para>
-
- <para>
- This works perfectly for <firstterm linkend="gloss-up"><acronym>UP
- </acronym></firstterm> as well: the spin lock vanishes, and this macro
- simply becomes <function>local_bh_disable()</function>
- (<filename class="headerfile">include/linux/interrupt.h</filename>), which
- protects you from the softirq being run.
- </para>
- </sect1>
-
- <sect1 id="lock-user-tasklet">
- <title>Locking Between User Context and Tasklets</title>
-
- <para>
- This is exactly the same as above, because <firstterm
- linkend="gloss-tasklet">tasklets</firstterm> are actually run
- from a softirq.
- </para>
- </sect1>
-
- <sect1 id="lock-user-timers">
- <title>Locking Between User Context and Timers</title>
-
- <para>
- This, too, is exactly the same as above, because <firstterm
- linkend="gloss-timers">timers</firstterm> are actually run from
- a softirq. From a locking point of view, tasklets and timers
- are identical.
- </para>
- </sect1>
-
- <sect1 id="lock-tasklets">
- <title>Locking Between Tasklets/Timers</title>
-
- <para>
- Sometimes a tasklet or timer might want to share data with
- another tasklet or timer.
- </para>
-
- <sect2 id="lock-tasklets-same">
- <title>The Same Tasklet/Timer</title>
- <para>
- Since a tasklet is never run on two CPUs at once, you don't
- need to worry about your tasklet being reentrant (running
- twice at once), even on SMP.
- </para>
- </sect2>
-
- <sect2 id="lock-tasklets-different">
- <title>Different Tasklets/Timers</title>
- <para>
- If another tasklet/timer wants
- to share data with your tasklet or timer , you will both need to use
- <function>spin_lock()</function> and
- <function>spin_unlock()</function> calls.
- <function>spin_lock_bh()</function> is
- unnecessary here, as you are already in a tasklet, and
- none will be run on the same CPU.
- </para>
- </sect2>
- </sect1>
-
- <sect1 id="lock-softirqs">
- <title>Locking Between Softirqs</title>
-
- <para>
- Often a softirq might
- want to share data with itself or a tasklet/timer.
- </para>
-
- <sect2 id="lock-softirqs-same">
- <title>The Same Softirq</title>
-
- <para>
- The same softirq can run on the other CPUs: you can use a
- per-CPU array (see <xref linkend="per-cpu"/>) for better
- performance. If you're going so far as to use a softirq,
- you probably care about scalable performance enough
- to justify the extra complexity.
- </para>
-
- <para>
- You'll need to use <function>spin_lock()</function> and
- <function>spin_unlock()</function> for shared data.
- </para>
- </sect2>
-
- <sect2 id="lock-softirqs-different">
- <title>Different Softirqs</title>
-
- <para>
- You'll need to use <function>spin_lock()</function> and
- <function>spin_unlock()</function> for shared data, whether it
- be a timer, tasklet, different softirq or the same or another
- softirq: any of them could be running on a different CPU.
- </para>
- </sect2>
- </sect1>
- </chapter>
-
- <chapter id="hardirq-context">
- <title>Hard IRQ Context</title>
-
- <para>
- Hardware interrupts usually communicate with a
- tasklet or softirq. Frequently this involves putting work in a
- queue, which the softirq will take out.
- </para>
-
- <sect1 id="hardirq-softirq">
- <title>Locking Between Hard IRQ and Softirqs/Tasklets</title>
-
- <para>
- If a hardware irq handler shares data with a softirq, you have
- two concerns. Firstly, the softirq processing can be
- interrupted by a hardware interrupt, and secondly, the
- critical region could be entered by a hardware interrupt on
- another CPU. This is where <function>spin_lock_irq()</function> is
- used. It is defined to disable interrupts on that cpu, then grab
- the lock. <function>spin_unlock_irq()</function> does the reverse.
- </para>
-
- <para>
- The irq handler does not to use
- <function>spin_lock_irq()</function>, because the softirq cannot
- run while the irq handler is running: it can use
- <function>spin_lock()</function>, which is slightly faster. The
- only exception would be if a different hardware irq handler uses
- the same lock: <function>spin_lock_irq()</function> will stop
- that from interrupting us.
- </para>
-
- <para>
- This works perfectly for UP as well: the spin lock vanishes,
- and this macro simply becomes <function>local_irq_disable()</function>
- (<filename class="headerfile">include/asm/smp.h</filename>), which
- protects you from the softirq/tasklet/BH being run.
- </para>
-
- <para>
- <function>spin_lock_irqsave()</function>
- (<filename>include/linux/spinlock.h</filename>) is a variant
- which saves whether interrupts were on or off in a flags word,
- which is passed to <function>spin_unlock_irqrestore()</function>. This
- means that the same code can be used inside an hard irq handler (where
- interrupts are already off) and in softirqs (where the irq
- disabling is required).
- </para>
-
- <para>
- Note that softirqs (and hence tasklets and timers) are run on
- return from hardware interrupts, so
- <function>spin_lock_irq()</function> also stops these. In that
- sense, <function>spin_lock_irqsave()</function> is the most
- general and powerful locking function.
- </para>
-
- </sect1>
- <sect1 id="hardirq-hardirq">
- <title>Locking Between Two Hard IRQ Handlers</title>
- <para>
- It is rare to have to share data between two IRQ handlers, but
- if you do, <function>spin_lock_irqsave()</function> should be
- used: it is architecture-specific whether all interrupts are
- disabled inside irq handlers themselves.
- </para>
- </sect1>
-
- </chapter>
-
- <chapter id="cheatsheet">
- <title>Cheat Sheet For Locking</title>
- <para>
- Pete Zaitcev gives the following summary:
- </para>
- <itemizedlist>
- <listitem>
- <para>
- If you are in a process context (any syscall) and want to
- lock other process out, use a mutex. You can take a mutex
- and sleep (<function>copy_from_user*(</function> or
- <function>kmalloc(x,GFP_KERNEL)</function>).
- </para>
- </listitem>
- <listitem>
- <para>
- Otherwise (== data can be touched in an interrupt), use
- <function>spin_lock_irqsave()</function> and
- <function>spin_unlock_irqrestore()</function>.
- </para>
- </listitem>
- <listitem>
- <para>
- Avoid holding spinlock for more than 5 lines of code and
- across any function call (except accessors like
- <function>readb</function>).
- </para>
- </listitem>
- </itemizedlist>
-
- <sect1 id="minimum-lock-reqirements">
- <title>Table of Minimum Requirements</title>
-
- <para> The following table lists the <emphasis>minimum</emphasis>
- locking requirements between various contexts. In some cases,
- the same context can only be running on one CPU at a time, so
- no locking is required for that context (eg. a particular
- thread can only run on one CPU at a time, but if it needs
- shares data with another thread, locking is required).
- </para>
- <para>
- Remember the advice above: you can always use
- <function>spin_lock_irqsave()</function>, which is a superset
- of all other spinlock primitives.
- </para>
-
- <table>
-<title>Table of Locking Requirements</title>
-<tgroup cols="11">
-<tbody>
-
-<row>
-<entry></entry>
-<entry>IRQ Handler A</entry>
-<entry>IRQ Handler B</entry>
-<entry>Softirq A</entry>
-<entry>Softirq B</entry>
-<entry>Tasklet A</entry>
-<entry>Tasklet B</entry>
-<entry>Timer A</entry>
-<entry>Timer B</entry>
-<entry>User Context A</entry>
-<entry>User Context B</entry>
-</row>
-
-<row>
-<entry>IRQ Handler A</entry>
-<entry>None</entry>
-</row>
-
-<row>
-<entry>IRQ Handler B</entry>
-<entry>SLIS</entry>
-<entry>None</entry>
-</row>
-
-<row>
-<entry>Softirq A</entry>
-<entry>SLI</entry>
-<entry>SLI</entry>
-<entry>SL</entry>
-</row>
-
-<row>
-<entry>Softirq B</entry>
-<entry>SLI</entry>
-<entry>SLI</entry>
-<entry>SL</entry>
-<entry>SL</entry>
-</row>
-
-<row>
-<entry>Tasklet A</entry>
-<entry>SLI</entry>
-<entry>SLI</entry>
-<entry>SL</entry>
-<entry>SL</entry>
-<entry>None</entry>
-</row>
-
-<row>
-<entry>Tasklet B</entry>
-<entry>SLI</entry>
-<entry>SLI</entry>
-<entry>SL</entry>
-<entry>SL</entry>
-<entry>SL</entry>
-<entry>None</entry>
-</row>
-
-<row>
-<entry>Timer A</entry>
-<entry>SLI</entry>
-<entry>SLI</entry>
-<entry>SL</entry>
-<entry>SL</entry>
-<entry>SL</entry>
-<entry>SL</entry>
-<entry>None</entry>
-</row>
-
-<row>
-<entry>Timer B</entry>
-<entry>SLI</entry>
-<entry>SLI</entry>
-<entry>SL</entry>
-<entry>SL</entry>
-<entry>SL</entry>
-<entry>SL</entry>
-<entry>SL</entry>
-<entry>None</entry>
-</row>
-
-<row>
-<entry>User Context A</entry>
-<entry>SLI</entry>
-<entry>SLI</entry>
-<entry>SLBH</entry>
-<entry>SLBH</entry>
-<entry>SLBH</entry>
-<entry>SLBH</entry>
-<entry>SLBH</entry>
-<entry>SLBH</entry>
-<entry>None</entry>
-</row>
-
-<row>
-<entry>User Context B</entry>
-<entry>SLI</entry>
-<entry>SLI</entry>
-<entry>SLBH</entry>
-<entry>SLBH</entry>
-<entry>SLBH</entry>
-<entry>SLBH</entry>
-<entry>SLBH</entry>
-<entry>SLBH</entry>
-<entry>MLI</entry>
-<entry>None</entry>
-</row>
-
-</tbody>
-</tgroup>
-</table>
-
- <table>
-<title>Legend for Locking Requirements Table</title>
-<tgroup cols="2">
-<tbody>
-
-<row>
-<entry>SLIS</entry>
-<entry>spin_lock_irqsave</entry>
-</row>
-<row>
-<entry>SLI</entry>
-<entry>spin_lock_irq</entry>
-</row>
-<row>
-<entry>SL</entry>
-<entry>spin_lock</entry>
-</row>
-<row>
-<entry>SLBH</entry>
-<entry>spin_lock_bh</entry>
-</row>
-<row>
-<entry>MLI</entry>
-<entry>mutex_lock_interruptible</entry>
-</row>
-
-</tbody>
-</tgroup>
-</table>
-
-</sect1>
-</chapter>
-
-<chapter id="trylock-functions">
- <title>The trylock Functions</title>
- <para>
- There are functions that try to acquire a lock only once and immediately
- return a value telling about success or failure to acquire the lock.
- They can be used if you need no access to the data protected with the lock
- when some other thread is holding the lock. You should acquire the lock
- later if you then need access to the data protected with the lock.
- </para>
-
- <para>
- <function>spin_trylock()</function> does not spin but returns non-zero if
- it acquires the spinlock on the first try or 0 if not. This function can
- be used in all contexts like <function>spin_lock</function>: you must have
- disabled the contexts that might interrupt you and acquire the spin lock.
- </para>
-
- <para>
- <function>mutex_trylock()</function> does not suspend your task
- but returns non-zero if it could lock the mutex on the first try
- or 0 if not. This function cannot be safely used in hardware or software
- interrupt contexts despite not sleeping.
- </para>
-</chapter>
-
- <chapter id="Examples">
- <title>Common Examples</title>
- <para>
-Let's step through a simple example: a cache of number to name
-mappings. The cache keeps a count of how often each of the objects is
-used, and when it gets full, throws out the least used one.
-
- </para>
-
- <sect1 id="examples-usercontext">
- <title>All In User Context</title>
- <para>
-For our first example, we assume that all operations are in user
-context (ie. from system calls), so we can sleep. This means we can
-use a mutex to protect the cache and all the objects within
-it. Here's the code:
- </para>
-
- <programlisting>
-#include <linux/list.h>
-#include <linux/slab.h>
-#include <linux/string.h>
-#include <linux/mutex.h>
-#include <asm/errno.h>
-
-struct object
-{
- struct list_head list;
- int id;
- char name[32];
- int popularity;
-};
-
-/* Protects the cache, cache_num, and the objects within it */
-static DEFINE_MUTEX(cache_lock);
-static LIST_HEAD(cache);
-static unsigned int cache_num = 0;
-#define MAX_CACHE_SIZE 10
-
-/* Must be holding cache_lock */
-static struct object *__cache_find(int id)
-{
- struct object *i;
-
- list_for_each_entry(i, &cache, list)
- if (i->id == id) {
- i->popularity++;
- return i;
- }
- return NULL;
-}
-
-/* Must be holding cache_lock */
-static void __cache_delete(struct object *obj)
-{
- BUG_ON(!obj);
- list_del(&obj->list);
- kfree(obj);
- cache_num--;
-}
-
-/* Must be holding cache_lock */
-static void __cache_add(struct object *obj)
-{
- list_add(&obj->list, &cache);
- if (++cache_num > MAX_CACHE_SIZE) {
- struct object *i, *outcast = NULL;
- list_for_each_entry(i, &cache, list) {
- if (!outcast || i->popularity < outcast->popularity)
- outcast = i;
- }
- __cache_delete(outcast);
- }
-}
-
-int cache_add(int id, const char *name)
-{
- struct object *obj;
-
- if ((obj = kmalloc(sizeof(*obj), GFP_KERNEL)) == NULL)
- return -ENOMEM;
-
- strlcpy(obj->name, name, sizeof(obj->name));
- obj->id = id;
- obj->popularity = 0;
-
- mutex_lock(&cache_lock);
- __cache_add(obj);
- mutex_unlock(&cache_lock);
- return 0;
-}
-
-void cache_delete(int id)
-{
- mutex_lock(&cache_lock);
- __cache_delete(__cache_find(id));
- mutex_unlock(&cache_lock);
-}
-
-int cache_find(int id, char *name)
-{
- struct object *obj;
- int ret = -ENOENT;
-
- mutex_lock(&cache_lock);
- obj = __cache_find(id);
- if (obj) {
- ret = 0;
- strcpy(name, obj->name);
- }
- mutex_unlock(&cache_lock);
- return ret;
-}
-</programlisting>
-
- <para>
-Note that we always make sure we have the cache_lock when we add,
-delete, or look up the cache: both the cache infrastructure itself and
-the contents of the objects are protected by the lock. In this case
-it's easy, since we copy the data for the user, and never let them
-access the objects directly.
- </para>
- <para>
-There is a slight (and common) optimization here: in
-<function>cache_add</function> we set up the fields of the object
-before grabbing the lock. This is safe, as no-one else can access it
-until we put it in cache.
- </para>
- </sect1>
-
- <sect1 id="examples-interrupt">
- <title>Accessing From Interrupt Context</title>
- <para>
-Now consider the case where <function>cache_find</function> can be
-called from interrupt context: either a hardware interrupt or a
-softirq. An example would be a timer which deletes object from the
-cache.
- </para>
- <para>
-The change is shown below, in standard patch format: the
-<symbol>-</symbol> are lines which are taken away, and the
-<symbol>+</symbol> are lines which are added.
- </para>
-<programlisting>
---- cache.c.usercontext 2003-12-09 13:58:54.000000000 +1100
-+++ cache.c.interrupt 2003-12-09 14:07:49.000000000 +1100
-@@ -12,7 +12,7 @@
- int popularity;
- };
-
--static DEFINE_MUTEX(cache_lock);
-+static DEFINE_SPINLOCK(cache_lock);
- static LIST_HEAD(cache);
- static unsigned int cache_num = 0;
- #define MAX_CACHE_SIZE 10
-@@ -55,6 +55,7 @@
- int cache_add(int id, const char *name)
- {
- struct object *obj;
-+ unsigned long flags;
-
- if ((obj = kmalloc(sizeof(*obj), GFP_KERNEL)) == NULL)
- return -ENOMEM;
-@@ -63,30 +64,33 @@
- obj->id = id;
- obj->popularity = 0;
-
-- mutex_lock(&cache_lock);
-+ spin_lock_irqsave(&cache_lock, flags);
- __cache_add(obj);
-- mutex_unlock(&cache_lock);
-+ spin_unlock_irqrestore(&cache_lock, flags);
- return 0;
- }
-
- void cache_delete(int id)
- {
-- mutex_lock(&cache_lock);
-+ unsigned long flags;
-+
-+ spin_lock_irqsave(&cache_lock, flags);
- __cache_delete(__cache_find(id));
-- mutex_unlock(&cache_lock);
-+ spin_unlock_irqrestore(&cache_lock, flags);
- }
-
- int cache_find(int id, char *name)
- {
- struct object *obj;
- int ret = -ENOENT;
-+ unsigned long flags;
-
-- mutex_lock(&cache_lock);
-+ spin_lock_irqsave(&cache_lock, flags);
- obj = __cache_find(id);
- if (obj) {
- ret = 0;
- strcpy(name, obj->name);
- }
-- mutex_unlock(&cache_lock);
-+ spin_unlock_irqrestore(&cache_lock, flags);
- return ret;
- }
-</programlisting>
-
- <para>
-Note that the <function>spin_lock_irqsave</function> will turn off
-interrupts if they are on, otherwise does nothing (if we are already
-in an interrupt handler), hence these functions are safe to call from
-any context.
- </para>
- <para>
-Unfortunately, <function>cache_add</function> calls
-<function>kmalloc</function> with the <symbol>GFP_KERNEL</symbol>
-flag, which is only legal in user context. I have assumed that
-<function>cache_add</function> is still only called in user context,
-otherwise this should become a parameter to
-<function>cache_add</function>.
- </para>
- </sect1>
- <sect1 id="examples-refcnt">
- <title>Exposing Objects Outside This File</title>
- <para>
-If our objects contained more information, it might not be sufficient
-to copy the information in and out: other parts of the code might want
-to keep pointers to these objects, for example, rather than looking up
-the id every time. This produces two problems.
- </para>
- <para>
-The first problem is that we use the <symbol>cache_lock</symbol> to
-protect objects: we'd need to make this non-static so the rest of the
-code can use it. This makes locking trickier, as it is no longer all
-in one place.
- </para>
- <para>
-The second problem is the lifetime problem: if another structure keeps
-a pointer to an object, it presumably expects that pointer to remain
-valid. Unfortunately, this is only guaranteed while you hold the
-lock, otherwise someone might call <function>cache_delete</function>
-and even worse, add another object, re-using the same address.
- </para>
- <para>
-As there is only one lock, you can't hold it forever: no-one else would
-get any work done.
- </para>
- <para>
-The solution to this problem is to use a reference count: everyone who
-has a pointer to the object increases it when they first get the
-object, and drops the reference count when they're finished with it.
-Whoever drops it to zero knows it is unused, and can actually delete it.
- </para>
- <para>
-Here is the code:
- </para>
-
-<programlisting>
---- cache.c.interrupt 2003-12-09 14:25:43.000000000 +1100
-+++ cache.c.refcnt 2003-12-09 14:33:05.000000000 +1100
-@@ -7,6 +7,7 @@
- struct object
- {
- struct list_head list;
-+ unsigned int refcnt;
- int id;
- char name[32];
- int popularity;
-@@ -17,6 +18,35 @@
- static unsigned int cache_num = 0;
- #define MAX_CACHE_SIZE 10
-
-+static void __object_put(struct object *obj)
-+{
-+ if (--obj->refcnt == 0)
-+ kfree(obj);
-+}
-+
-+static void __object_get(struct object *obj)
-+{
-+ obj->refcnt++;
-+}
-+
-+void object_put(struct object *obj)
-+{
-+ unsigned long flags;
-+
-+ spin_lock_irqsave(&cache_lock, flags);
-+ __object_put(obj);
-+ spin_unlock_irqrestore(&cache_lock, flags);
-+}
-+
-+void object_get(struct object *obj)
-+{
-+ unsigned long flags;
-+
-+ spin_lock_irqsave(&cache_lock, flags);
-+ __object_get(obj);
-+ spin_unlock_irqrestore(&cache_lock, flags);
-+}
-+
- /* Must be holding cache_lock */
- static struct object *__cache_find(int id)
- {
-@@ -35,6 +65,7 @@
- {
- BUG_ON(!obj);
- list_del(&obj->list);
-+ __object_put(obj);
- cache_num--;
- }
-
-@@ -63,6 +94,7 @@
- strlcpy(obj->name, name, sizeof(obj->name));
- obj->id = id;
- obj->popularity = 0;
-+ obj->refcnt = 1; /* The cache holds a reference */
-
- spin_lock_irqsave(&cache_lock, flags);
- __cache_add(obj);
-@@ -79,18 +111,15 @@
- spin_unlock_irqrestore(&cache_lock, flags);
- }
-
--int cache_find(int id, char *name)
-+struct object *cache_find(int id)
- {
- struct object *obj;
-- int ret = -ENOENT;
- unsigned long flags;
-
- spin_lock_irqsave(&cache_lock, flags);
- obj = __cache_find(id);
-- if (obj) {
-- ret = 0;
-- strcpy(name, obj->name);
-- }
-+ if (obj)
-+ __object_get(obj);
- spin_unlock_irqrestore(&cache_lock, flags);
-- return ret;
-+ return obj;
- }
-</programlisting>
-
-<para>
-We encapsulate the reference counting in the standard 'get' and 'put'
-functions. Now we can return the object itself from
-<function>cache_find</function> which has the advantage that the user
-can now sleep holding the object (eg. to
-<function>copy_to_user</function> to name to userspace).
-</para>
-<para>
-The other point to note is that I said a reference should be held for
-every pointer to the object: thus the reference count is 1 when first
-inserted into the cache. In some versions the framework does not hold
-a reference count, but they are more complicated.
-</para>
-
- <sect2 id="examples-refcnt-atomic">
- <title>Using Atomic Operations For The Reference Count</title>
-<para>
-In practice, <type>atomic_t</type> would usually be used for
-<structfield>refcnt</structfield>. There are a number of atomic
-operations defined in
-
-<filename class="headerfile">include/asm/atomic.h</filename>: these are
-guaranteed to be seen atomically from all CPUs in the system, so no
-lock is required. In this case, it is simpler than using spinlocks,
-although for anything non-trivial using spinlocks is clearer. The
-<function>atomic_inc</function> and
-<function>atomic_dec_and_test</function> are used instead of the
-standard increment and decrement operators, and the lock is no longer
-used to protect the reference count itself.
-</para>
-
-<programlisting>
---- cache.c.refcnt 2003-12-09 15:00:35.000000000 +1100
-+++ cache.c.refcnt-atomic 2003-12-11 15:49:42.000000000 +1100
-@@ -7,7 +7,7 @@
- struct object
- {
- struct list_head list;
-- unsigned int refcnt;
-+ atomic_t refcnt;
- int id;
- char name[32];
- int popularity;
-@@ -18,33 +18,15 @@
- static unsigned int cache_num = 0;
- #define MAX_CACHE_SIZE 10
-
--static void __object_put(struct object *obj)
--{
-- if (--obj->refcnt == 0)
-- kfree(obj);
--}
--
--static void __object_get(struct object *obj)
--{
-- obj->refcnt++;
--}
--
- void object_put(struct object *obj)
- {
-- unsigned long flags;
--
-- spin_lock_irqsave(&cache_lock, flags);
-- __object_put(obj);
-- spin_unlock_irqrestore(&cache_lock, flags);
-+ if (atomic_dec_and_test(&obj->refcnt))
-+ kfree(obj);
- }
-
- void object_get(struct object *obj)
- {
-- unsigned long flags;
--
-- spin_lock_irqsave(&cache_lock, flags);
-- __object_get(obj);
-- spin_unlock_irqrestore(&cache_lock, flags);
-+ atomic_inc(&obj->refcnt);
- }
-
- /* Must be holding cache_lock */
-@@ -65,7 +47,7 @@
- {
- BUG_ON(!obj);
- list_del(&obj->list);
-- __object_put(obj);
-+ object_put(obj);
- cache_num--;
- }
-
-@@ -94,7 +76,7 @@
- strlcpy(obj->name, name, sizeof(obj->name));
- obj->id = id;
- obj->popularity = 0;
-- obj->refcnt = 1; /* The cache holds a reference */
-+ atomic_set(&obj->refcnt, 1); /* The cache holds a reference */
-
- spin_lock_irqsave(&cache_lock, flags);
- __cache_add(obj);
-@@ -119,7 +101,7 @@
- spin_lock_irqsave(&cache_lock, flags);
- obj = __cache_find(id);
- if (obj)
-- __object_get(obj);
-+ object_get(obj);
- spin_unlock_irqrestore(&cache_lock, flags);
- return obj;
- }
-</programlisting>
-</sect2>
-</sect1>
-
- <sect1 id="examples-lock-per-obj">
- <title>Protecting The Objects Themselves</title>
- <para>
-In these examples, we assumed that the objects (except the reference
-counts) never changed once they are created. If we wanted to allow
-the name to change, there are three possibilities:
- </para>
- <itemizedlist>
- <listitem>
- <para>
-You can make <symbol>cache_lock</symbol> non-static, and tell people
-to grab that lock before changing the name in any object.
- </para>
- </listitem>
- <listitem>
- <para>
-You can provide a <function>cache_obj_rename</function> which grabs
-this lock and changes the name for the caller, and tell everyone to
-use that function.
- </para>
- </listitem>
- <listitem>
- <para>
-You can make the <symbol>cache_lock</symbol> protect only the cache
-itself, and use another lock to protect the name.
- </para>
- </listitem>
- </itemizedlist>
-
- <para>
-Theoretically, you can make the locks as fine-grained as one lock for
-every field, for every object. In practice, the most common variants
-are:
-</para>
- <itemizedlist>
- <listitem>
- <para>
-One lock which protects the infrastructure (the <symbol>cache</symbol>
-list in this example) and all the objects. This is what we have done
-so far.
- </para>
- </listitem>
- <listitem>
- <para>
-One lock which protects the infrastructure (including the list
-pointers inside the objects), and one lock inside the object which
-protects the rest of that object.
- </para>
- </listitem>
- <listitem>
- <para>
-Multiple locks to protect the infrastructure (eg. one lock per hash
-chain), possibly with a separate per-object lock.
- </para>
- </listitem>
- </itemizedlist>
-
-<para>
-Here is the "lock-per-object" implementation:
-</para>
-<programlisting>
---- cache.c.refcnt-atomic 2003-12-11 15:50:54.000000000 +1100
-+++ cache.c.perobjectlock 2003-12-11 17:15:03.000000000 +1100
-@@ -6,11 +6,17 @@
-
- struct object
- {
-+ /* These two protected by cache_lock. */
- struct list_head list;
-+ int popularity;
-+
- atomic_t refcnt;
-+
-+ /* Doesn't change once created. */
- int id;
-+
-+ spinlock_t lock; /* Protects the name */
- char name[32];
-- int popularity;
- };
-
- static DEFINE_SPINLOCK(cache_lock);
-@@ -77,6 +84,7 @@
- obj->id = id;
- obj->popularity = 0;
- atomic_set(&obj->refcnt, 1); /* The cache holds a reference */
-+ spin_lock_init(&obj->lock);
-
- spin_lock_irqsave(&cache_lock, flags);
- __cache_add(obj);
-</programlisting>
-
-<para>
-Note that I decide that the <structfield>popularity</structfield>
-count should be protected by the <symbol>cache_lock</symbol> rather
-than the per-object lock: this is because it (like the
-<structname>struct list_head</structname> inside the object) is
-logically part of the infrastructure. This way, I don't need to grab
-the lock of every object in <function>__cache_add</function> when
-seeking the least popular.
-</para>
-
-<para>
-I also decided that the <structfield>id</structfield> member is
-unchangeable, so I don't need to grab each object lock in
-<function>__cache_find()</function> to examine the
-<structfield>id</structfield>: the object lock is only used by a
-caller who wants to read or write the <structfield>name</structfield>
-field.
-</para>
-
-<para>
-Note also that I added a comment describing what data was protected by
-which locks. This is extremely important, as it describes the runtime
-behavior of the code, and can be hard to gain from just reading. And
-as Alan Cox says, <quote>Lock data, not code</quote>.
-</para>
-</sect1>
-</chapter>
-
- <chapter id="common-problems">
- <title>Common Problems</title>
- <sect1 id="deadlock">
- <title>Deadlock: Simple and Advanced</title>
-
- <para>
- There is a coding bug where a piece of code tries to grab a
- spinlock twice: it will spin forever, waiting for the lock to
- be released (spinlocks, rwlocks and mutexes are not
- recursive in Linux). This is trivial to diagnose: not a
- stay-up-five-nights-talk-to-fluffy-code-bunnies kind of
- problem.
- </para>
-
- <para>
- For a slightly more complex case, imagine you have a region
- shared by a softirq and user context. If you use a
- <function>spin_lock()</function> call to protect it, it is
- possible that the user context will be interrupted by the softirq
- while it holds the lock, and the softirq will then spin
- forever trying to get the same lock.
- </para>
-
- <para>
- Both of these are called deadlock, and as shown above, it can
- occur even with a single CPU (although not on UP compiles,
- since spinlocks vanish on kernel compiles with
- <symbol>CONFIG_SMP</symbol>=n. You'll still get data corruption
- in the second example).
- </para>
-
- <para>
- This complete lockup is easy to diagnose: on SMP boxes the
- watchdog timer or compiling with <symbol>DEBUG_SPINLOCK</symbol> set
- (<filename>include/linux/spinlock.h</filename>) will show this up
- immediately when it happens.
- </para>
-
- <para>
- A more complex problem is the so-called 'deadly embrace',
- involving two or more locks. Say you have a hash table: each
- entry in the table is a spinlock, and a chain of hashed
- objects. Inside a softirq handler, you sometimes want to
- alter an object from one place in the hash to another: you
- grab the spinlock of the old hash chain and the spinlock of
- the new hash chain, and delete the object from the old one,
- and insert it in the new one.
- </para>
-
- <para>
- There are two problems here. First, if your code ever
- tries to move the object to the same chain, it will deadlock
- with itself as it tries to lock it twice. Secondly, if the
- same softirq on another CPU is trying to move another object
- in the reverse direction, the following could happen:
- </para>
-
- <table>
- <title>Consequences</title>
-
- <tgroup cols="2" align="left">
-
- <thead>
- <row>
- <entry>CPU 1</entry>
- <entry>CPU 2</entry>
- </row>
- </thead>
-
- <tbody>
- <row>
- <entry>Grab lock A -> OK</entry>
- <entry>Grab lock B -> OK</entry>
- </row>
- <row>
- <entry>Grab lock B -> spin</entry>
- <entry>Grab lock A -> spin</entry>
- </row>
- </tbody>
- </tgroup>
- </table>
-
- <para>
- The two CPUs will spin forever, waiting for the other to give up
- their lock. It will look, smell, and feel like a crash.
- </para>
- </sect1>
-
- <sect1 id="techs-deadlock-prevent">
- <title>Preventing Deadlock</title>
-
- <para>
- Textbooks will tell you that if you always lock in the same
- order, you will never get this kind of deadlock. Practice
- will tell you that this approach doesn't scale: when I
- create a new lock, I don't understand enough of the kernel
- to figure out where in the 5000 lock hierarchy it will fit.
- </para>
-
- <para>
- The best locks are encapsulated: they never get exposed in
- headers, and are never held around calls to non-trivial
- functions outside the same file. You can read through this
- code and see that it will never deadlock, because it never
- tries to grab another lock while it has that one. People
- using your code don't even need to know you are using a
- lock.
- </para>
-
- <para>
- A classic problem here is when you provide callbacks or
- hooks: if you call these with the lock held, you risk simple
- deadlock, or a deadly embrace (who knows what the callback
- will do?). Remember, the other programmers are out to get
- you, so don't do this.
- </para>
-
- <sect2 id="techs-deadlock-overprevent">
- <title>Overzealous Prevention Of Deadlocks</title>
-
- <para>
- Deadlocks are problematic, but not as bad as data
- corruption. Code which grabs a read lock, searches a list,
- fails to find what it wants, drops the read lock, grabs a
- write lock and inserts the object has a race condition.
- </para>
-
- <para>
- If you don't see why, please stay the fuck away from my code.
- </para>
- </sect2>
- </sect1>
-
- <sect1 id="racing-timers">
- <title>Racing Timers: A Kernel Pastime</title>
-
- <para>
- Timers can produce their own special problems with races.
- Consider a collection of objects (list, hash, etc) where each
- object has a timer which is due to destroy it.
- </para>
-
- <para>
- If you want to destroy the entire collection (say on module
- removal), you might do the following:
- </para>
-
- <programlisting>
- /* THIS CODE BAD BAD BAD BAD: IF IT WAS ANY WORSE IT WOULD USE
- HUNGARIAN NOTATION */
- spin_lock_bh(&list_lock);
-
- while (list) {
- struct foo *next = list->next;
- del_timer(&list->timer);
- kfree(list);
- list = next;
- }
-
- spin_unlock_bh(&list_lock);
- </programlisting>
-
- <para>
- Sooner or later, this will crash on SMP, because a timer can
- have just gone off before the <function>spin_lock_bh()</function>,
- and it will only get the lock after we
- <function>spin_unlock_bh()</function>, and then try to free
- the element (which has already been freed!).
- </para>
-
- <para>
- This can be avoided by checking the result of
- <function>del_timer()</function>: if it returns
- <returnvalue>1</returnvalue>, the timer has been deleted.
- If <returnvalue>0</returnvalue>, it means (in this
- case) that it is currently running, so we can do:
- </para>
-
- <programlisting>
- retry:
- spin_lock_bh(&list_lock);
-
- while (list) {
- struct foo *next = list->next;
- if (!del_timer(&list->timer)) {
- /* Give timer a chance to delete this */
- spin_unlock_bh(&list_lock);
- goto retry;
- }
- kfree(list);
- list = next;
- }
-
- spin_unlock_bh(&list_lock);
- </programlisting>
-
- <para>
- Another common problem is deleting timers which restart
- themselves (by calling <function>add_timer()</function> at the end
- of their timer function). Because this is a fairly common case
- which is prone to races, you should use <function>del_timer_sync()</function>
- (<filename class="headerfile">include/linux/timer.h</filename>)
- to handle this case. It returns the number of times the timer
- had to be deleted before we finally stopped it from adding itself back
- in.
- </para>
- </sect1>
-
- </chapter>
-
- <chapter id="Efficiency">
- <title>Locking Speed</title>
-
- <para>
-There are three main things to worry about when considering speed of
-some code which does locking. First is concurrency: how many things
-are going to be waiting while someone else is holding a lock. Second
-is the time taken to actually acquire and release an uncontended lock.
-Third is using fewer, or smarter locks. I'm assuming that the lock is
-used fairly often: otherwise, you wouldn't be concerned about
-efficiency.
-</para>
- <para>
-Concurrency depends on how long the lock is usually held: you should
-hold the lock for as long as needed, but no longer. In the cache
-example, we always create the object without the lock held, and then
-grab the lock only when we are ready to insert it in the list.
-</para>
- <para>
-Acquisition times depend on how much damage the lock operations do to
-the pipeline (pipeline stalls) and how likely it is that this CPU was
-the last one to grab the lock (ie. is the lock cache-hot for this
-CPU): on a machine with more CPUs, this likelihood drops fast.
-Consider a 700MHz Intel Pentium III: an instruction takes about 0.7ns,
-an atomic increment takes about 58ns, a lock which is cache-hot on
-this CPU takes 160ns, and a cacheline transfer from another CPU takes
-an additional 170 to 360ns. (These figures from Paul McKenney's
-<ulink url="http://www.linuxjournal.com/article.php?sid=6993"> Linux
-Journal RCU article</ulink>).
-</para>
- <para>
-These two aims conflict: holding a lock for a short time might be done
-by splitting locks into parts (such as in our final per-object-lock
-example), but this increases the number of lock acquisitions, and the
-results are often slower than having a single lock. This is another
-reason to advocate locking simplicity.
-</para>
- <para>
-The third concern is addressed below: there are some methods to reduce
-the amount of locking which needs to be done.
-</para>
-
- <sect1 id="efficiency-rwlocks">
- <title>Read/Write Lock Variants</title>
-
- <para>
- Both spinlocks and mutexes have read/write variants:
- <type>rwlock_t</type> and <structname>struct rw_semaphore</structname>.
- These divide users into two classes: the readers and the writers. If
- you are only reading the data, you can get a read lock, but to write to
- the data you need the write lock. Many people can hold a read lock,
- but a writer must be sole holder.
- </para>
-
- <para>
- If your code divides neatly along reader/writer lines (as our
- cache code does), and the lock is held by readers for
- significant lengths of time, using these locks can help. They
- are slightly slower than the normal locks though, so in practice
- <type>rwlock_t</type> is not usually worthwhile.
- </para>
- </sect1>
-
- <sect1 id="efficiency-read-copy-update">
- <title>Avoiding Locks: Read Copy Update</title>
-
- <para>
- There is a special method of read/write locking called Read Copy
- Update. Using RCU, the readers can avoid taking a lock
- altogether: as we expect our cache to be read more often than
- updated (otherwise the cache is a waste of time), it is a
- candidate for this optimization.
- </para>
-
- <para>
- How do we get rid of read locks? Getting rid of read locks
- means that writers may be changing the list underneath the
- readers. That is actually quite simple: we can read a linked
- list while an element is being added if the writer adds the
- element very carefully. For example, adding
- <symbol>new</symbol> to a single linked list called
- <symbol>list</symbol>:
- </para>
-
- <programlisting>
- new->next = list->next;
- wmb();
- list->next = new;
- </programlisting>
-
- <para>
- The <function>wmb()</function> is a write memory barrier. It
- ensures that the first operation (setting the new element's
- <symbol>next</symbol> pointer) is complete and will be seen by
- all CPUs, before the second operation is (putting the new
- element into the list). This is important, since modern
- compilers and modern CPUs can both reorder instructions unless
- told otherwise: we want a reader to either not see the new
- element at all, or see the new element with the
- <symbol>next</symbol> pointer correctly pointing at the rest of
- the list.
- </para>
- <para>
- Fortunately, there is a function to do this for standard
- <structname>struct list_head</structname> lists:
- <function>list_add_rcu()</function>
- (<filename>include/linux/list.h</filename>).
- </para>
- <para>
- Removing an element from the list is even simpler: we replace
- the pointer to the old element with a pointer to its successor,
- and readers will either see it, or skip over it.
- </para>
- <programlisting>
- list->next = old->next;
- </programlisting>
- <para>
- There is <function>list_del_rcu()</function>
- (<filename>include/linux/list.h</filename>) which does this (the
- normal version poisons the old object, which we don't want).
- </para>
- <para>
- The reader must also be careful: some CPUs can look through the
- <symbol>next</symbol> pointer to start reading the contents of
- the next element early, but don't realize that the pre-fetched
- contents is wrong when the <symbol>next</symbol> pointer changes
- underneath them. Once again, there is a
- <function>list_for_each_entry_rcu()</function>
- (<filename>include/linux/list.h</filename>) to help you. Of
- course, writers can just use
- <function>list_for_each_entry()</function>, since there cannot
- be two simultaneous writers.
- </para>
- <para>
- Our final dilemma is this: when can we actually destroy the
- removed element? Remember, a reader might be stepping through
- this element in the list right now: if we free this element and
- the <symbol>next</symbol> pointer changes, the reader will jump
- off into garbage and crash. We need to wait until we know that
- all the readers who were traversing the list when we deleted the
- element are finished. We use <function>call_rcu()</function> to
- register a callback which will actually destroy the object once
- all pre-existing readers are finished. Alternatively,
- <function>synchronize_rcu()</function> may be used to block until
- all pre-existing are finished.
- </para>
- <para>
- But how does Read Copy Update know when the readers are
- finished? The method is this: firstly, the readers always
- traverse the list inside
- <function>rcu_read_lock()</function>/<function>rcu_read_unlock()</function>
- pairs: these simply disable preemption so the reader won't go to
- sleep while reading the list.
- </para>
- <para>
- RCU then waits until every other CPU has slept at least once:
- since readers cannot sleep, we know that any readers which were
- traversing the list during the deletion are finished, and the
- callback is triggered. The real Read Copy Update code is a
- little more optimized than this, but this is the fundamental
- idea.
- </para>
-
-<programlisting>
---- cache.c.perobjectlock 2003-12-11 17:15:03.000000000 +1100
-+++ cache.c.rcupdate 2003-12-11 17:55:14.000000000 +1100
-@@ -1,15 +1,18 @@
- #include <linux/list.h>
- #include <linux/slab.h>
- #include <linux/string.h>
-+#include <linux/rcupdate.h>
- #include <linux/mutex.h>
- #include <asm/errno.h>
-
- struct object
- {
-- /* These two protected by cache_lock. */
-+ /* This is protected by RCU */
- struct list_head list;
- int popularity;
-
-+ struct rcu_head rcu;
-+
- atomic_t refcnt;
-
- /* Doesn't change once created. */
-@@ -40,7 +43,7 @@
- {
- struct object *i;
-
-- list_for_each_entry(i, &cache, list) {
-+ list_for_each_entry_rcu(i, &cache, list) {
- if (i->id == id) {
- i->popularity++;
- return i;
-@@ -49,19 +52,25 @@
- return NULL;
- }
-
-+/* Final discard done once we know no readers are looking. */
-+static void cache_delete_rcu(void *arg)
-+{
-+ object_put(arg);
-+}
-+
- /* Must be holding cache_lock */
- static void __cache_delete(struct object *obj)
- {
- BUG_ON(!obj);
-- list_del(&obj->list);
-- object_put(obj);
-+ list_del_rcu(&obj->list);
- cache_num--;
-+ call_rcu(&obj->rcu, cache_delete_rcu);
- }
-
- /* Must be holding cache_lock */
- static void __cache_add(struct object *obj)
- {
-- list_add(&obj->list, &cache);
-+ list_add_rcu(&obj->list, &cache);
- if (++cache_num > MAX_CACHE_SIZE) {
- struct object *i, *outcast = NULL;
- list_for_each_entry(i, &cache, list) {
-@@ -104,12 +114,11 @@
- struct object *cache_find(int id)
- {
- struct object *obj;
-- unsigned long flags;
-
-- spin_lock_irqsave(&cache_lock, flags);
-+ rcu_read_lock();
- obj = __cache_find(id);
- if (obj)
- object_get(obj);
-- spin_unlock_irqrestore(&cache_lock, flags);
-+ rcu_read_unlock();
- return obj;
- }
-</programlisting>
-
-<para>
-Note that the reader will alter the
-<structfield>popularity</structfield> member in
-<function>__cache_find()</function>, and now it doesn't hold a lock.
-One solution would be to make it an <type>atomic_t</type>, but for
-this usage, we don't really care about races: an approximate result is
-good enough, so I didn't change it.
-</para>
-
-<para>
-The result is that <function>cache_find()</function> requires no
-synchronization with any other functions, so is almost as fast on SMP
-as it would be on UP.
-</para>
-
-<para>
-There is a further optimization possible here: remember our original
-cache code, where there were no reference counts and the caller simply
-held the lock whenever using the object? This is still possible: if
-you hold the lock, no one can delete the object, so you don't need to
-get and put the reference count.
-</para>
-
-<para>
-Now, because the 'read lock' in RCU is simply disabling preemption, a
-caller which always has preemption disabled between calling
-<function>cache_find()</function> and
-<function>object_put()</function> does not need to actually get and
-put the reference count: we could expose
-<function>__cache_find()</function> by making it non-static, and
-such callers could simply call that.
-</para>
-<para>
-The benefit here is that the reference count is not written to: the
-object is not altered in any way, which is much faster on SMP
-machines due to caching.
-</para>
- </sect1>
-
- <sect1 id="per-cpu">
- <title>Per-CPU Data</title>
-
- <para>
- Another technique for avoiding locking which is used fairly
- widely is to duplicate information for each CPU. For example,
- if you wanted to keep a count of a common condition, you could
- use a spin lock and a single counter. Nice and simple.
- </para>
-
- <para>
- If that was too slow (it's usually not, but if you've got a
- really big machine to test on and can show that it is), you
- could instead use a counter for each CPU, then none of them need
- an exclusive lock. See <function>DEFINE_PER_CPU()</function>,
- <function>get_cpu_var()</function> and
- <function>put_cpu_var()</function>
- (<filename class="headerfile">include/linux/percpu.h</filename>).
- </para>
-
- <para>
- Of particular use for simple per-cpu counters is the
- <type>local_t</type> type, and the
- <function>cpu_local_inc()</function> and related functions,
- which are more efficient than simple code on some architectures
- (<filename class="headerfile">include/asm/local.h</filename>).
- </para>
-
- <para>
- Note that there is no simple, reliable way of getting an exact
- value of such a counter, without introducing more locks. This
- is not a problem for some uses.
- </para>
- </sect1>
-
- <sect1 id="mostly-hardirq">
- <title>Data Which Mostly Used By An IRQ Handler</title>
-
- <para>
- If data is always accessed from within the same IRQ handler, you
- don't need a lock at all: the kernel already guarantees that the
- irq handler will not run simultaneously on multiple CPUs.
- </para>
- <para>
- Manfred Spraul points out that you can still do this, even if
- the data is very occasionally accessed in user context or
- softirqs/tasklets. The irq handler doesn't use a lock, and
- all other accesses are done as so:
- </para>
-
-<programlisting>
- spin_lock(&lock);
- disable_irq(irq);
- ...
- enable_irq(irq);
- spin_unlock(&lock);
-</programlisting>
- <para>
- The <function>disable_irq()</function> prevents the irq handler
- from running (and waits for it to finish if it's currently
- running on other CPUs). The spinlock prevents any other
- accesses happening at the same time. Naturally, this is slower
- than just a <function>spin_lock_irq()</function> call, so it
- only makes sense if this type of access happens extremely
- rarely.
- </para>
- </sect1>
- </chapter>
-
- <chapter id="sleeping-things">
- <title>What Functions Are Safe To Call From Interrupts?</title>
-
- <para>
- Many functions in the kernel sleep (ie. call schedule())
- directly or indirectly: you can never call them while holding a
- spinlock, or with preemption disabled. This also means you need
- to be in user context: calling them from an interrupt is illegal.
- </para>
-
- <sect1 id="sleeping">
- <title>Some Functions Which Sleep</title>
-
- <para>
- The most common ones are listed below, but you usually have to
- read the code to find out if other calls are safe. If everyone
- else who calls it can sleep, you probably need to be able to
- sleep, too. In particular, registration and deregistration
- functions usually expect to be called from user context, and can
- sleep.
- </para>
-
- <itemizedlist>
- <listitem>
- <para>
- Accesses to
- <firstterm linkend="gloss-userspace">userspace</firstterm>:
- </para>
- <itemizedlist>
- <listitem>
- <para>
- <function>copy_from_user()</function>
- </para>
- </listitem>
- <listitem>
- <para>
- <function>copy_to_user()</function>
- </para>
- </listitem>
- <listitem>
- <para>
- <function>get_user()</function>
- </para>
- </listitem>
- <listitem>
- <para>
- <function>put_user()</function>
- </para>
- </listitem>
- </itemizedlist>
- </listitem>
-
- <listitem>
- <para>
- <function>kmalloc(GFP_KERNEL)</function>
- </para>
- </listitem>
-
- <listitem>
- <para>
- <function>mutex_lock_interruptible()</function> and
- <function>mutex_lock()</function>
- </para>
- <para>
- There is a <function>mutex_trylock()</function> which does not
- sleep. Still, it must not be used inside interrupt context since
- its implementation is not safe for that.
- <function>mutex_unlock()</function> will also never sleep.
- It cannot be used in interrupt context either since a mutex
- must be released by the same task that acquired it.
- </para>
- </listitem>
- </itemizedlist>
- </sect1>
-
- <sect1 id="dont-sleep">
- <title>Some Functions Which Don't Sleep</title>
-
- <para>
- Some functions are safe to call from any context, or holding
- almost any lock.
- </para>
-
- <itemizedlist>
- <listitem>
- <para>
- <function>printk()</function>
- </para>
- </listitem>
- <listitem>
- <para>
- <function>kfree()</function>
- </para>
- </listitem>
- <listitem>
- <para>
- <function>add_timer()</function> and <function>del_timer()</function>
- </para>
- </listitem>
- </itemizedlist>
- </sect1>
- </chapter>
-
- <chapter id="apiref-mutex">
- <title>Mutex API reference</title>
-!Iinclude/linux/mutex.h
-!Ekernel/locking/mutex.c
- </chapter>
-
- <chapter id="apiref-futex">
- <title>Futex API reference</title>
-!Ikernel/futex.c
- </chapter>
-
- <chapter id="references">
- <title>Further reading</title>
-
- <itemizedlist>
- <listitem>
- <para>
- <filename>Documentation/locking/spinlocks.txt</filename>:
- Linus Torvalds' spinlocking tutorial in the kernel sources.
- </para>
- </listitem>
-
- <listitem>
- <para>
- Unix Systems for Modern Architectures: Symmetric
- Multiprocessing and Caching for Kernel Programmers:
- </para>
-
- <para>
- Curt Schimmel's very good introduction to kernel level
- locking (not written for Linux, but nearly everything
- applies). The book is expensive, but really worth every
- penny to understand SMP locking. [ISBN: 0201633388]
- </para>
- </listitem>
- </itemizedlist>
- </chapter>
-
- <chapter id="thanks">
- <title>Thanks</title>
-
- <para>
- Thanks to Telsa Gwynne for DocBooking, neatening and adding
- style.
- </para>
-
- <para>
- Thanks to Martin Pool, Philipp Rumpf, Stephen Rothwell, Paul
- Mackerras, Ruedi Aschwanden, Alan Cox, Manfred Spraul, Tim
- Waugh, Pete Zaitcev, James Morris, Robert Love, Paul McKenney,
- John Ashby for proofreading, correcting, flaming, commenting.
- </para>
-
- <para>
- Thanks to the cabal for having no influence on this document.
- </para>
- </chapter>
-
- <glossary id="glossary">
- <title>Glossary</title>
-
- <glossentry id="gloss-preemption">
- <glossterm>preemption</glossterm>
- <glossdef>
- <para>
- Prior to 2.5, or when <symbol>CONFIG_PREEMPT</symbol> is
- unset, processes in user context inside the kernel would not
- preempt each other (ie. you had that CPU until you gave it up,
- except for interrupts). With the addition of
- <symbol>CONFIG_PREEMPT</symbol> in 2.5.4, this changed: when
- in user context, higher priority tasks can "cut in": spinlocks
- were changed to disable preemption, even on UP.
- </para>
- </glossdef>
- </glossentry>
-
- <glossentry id="gloss-bh">
- <glossterm>bh</glossterm>
- <glossdef>
- <para>
- Bottom Half: for historical reasons, functions with
- '_bh' in them often now refer to any software interrupt, e.g.
- <function>spin_lock_bh()</function> blocks any software interrupt
- on the current CPU. Bottom halves are deprecated, and will
- eventually be replaced by tasklets. Only one bottom half will be
- running at any time.
- </para>
- </glossdef>
- </glossentry>
-
- <glossentry id="gloss-hwinterrupt">
- <glossterm>Hardware Interrupt / Hardware IRQ</glossterm>
- <glossdef>
- <para>
- Hardware interrupt request. <function>in_irq()</function> returns
- <returnvalue>true</returnvalue> in a hardware interrupt handler.
- </para>
- </glossdef>
- </glossentry>
-
- <glossentry id="gloss-interruptcontext">
- <glossterm>Interrupt Context</glossterm>
- <glossdef>
- <para>
- Not user context: processing a hardware irq or software irq.
- Indicated by the <function>in_interrupt()</function> macro
- returning <returnvalue>true</returnvalue>.
- </para>
- </glossdef>
- </glossentry>
-
- <glossentry id="gloss-smp">
- <glossterm><acronym>SMP</acronym></glossterm>
- <glossdef>
- <para>
- Symmetric Multi-Processor: kernels compiled for multiple-CPU
- machines. (CONFIG_SMP=y).
- </para>
- </glossdef>
- </glossentry>
-
- <glossentry id="gloss-softirq">
- <glossterm>Software Interrupt / softirq</glossterm>
- <glossdef>
- <para>
- Software interrupt handler. <function>in_irq()</function> returns
- <returnvalue>false</returnvalue>; <function>in_softirq()</function>
- returns <returnvalue>true</returnvalue>. Tasklets and softirqs
- both fall into the category of 'software interrupts'.
- </para>
- <para>
- Strictly speaking a softirq is one of up to 32 enumerated software
- interrupts which can run on multiple CPUs at once.
- Sometimes used to refer to tasklets as
- well (ie. all software interrupts).
- </para>
- </glossdef>
- </glossentry>
-
- <glossentry id="gloss-tasklet">
- <glossterm>tasklet</glossterm>
- <glossdef>
- <para>
- A dynamically-registrable software interrupt,
- which is guaranteed to only run on one CPU at a time.
- </para>
- </glossdef>
- </glossentry>
-
- <glossentry id="gloss-timers">
- <glossterm>timer</glossterm>
- <glossdef>
- <para>
- A dynamically-registrable software interrupt, which is run at
- (or close to) a given time. When running, it is just like a
- tasklet (in fact, they are called from the TIMER_SOFTIRQ).
- </para>
- </glossdef>
- </glossentry>
-
- <glossentry id="gloss-up">
- <glossterm><acronym>UP</acronym></glossterm>
- <glossdef>
- <para>
- Uni-Processor: Non-SMP. (CONFIG_SMP=n).
- </para>
- </glossdef>
- </glossentry>
-
- <glossentry id="gloss-usercontext">
- <glossterm>User Context</glossterm>
- <glossdef>
- <para>
- The kernel executing on behalf of a particular process (ie. a
- system call or trap) or kernel thread. You can tell which
- process with the <symbol>current</symbol> macro.) Not to
- be confused with userspace. Can be interrupted by software or
- hardware interrupts.
- </para>
- </glossdef>
- </glossentry>
-
- <glossentry id="gloss-userspace">
- <glossterm>Userspace</glossterm>
- <glossdef>
- <para>
- A process executing its own code outside the kernel.
- </para>
- </glossdef>
- </glossentry>
-
- </glossary>
-</book>
-
+++ /dev/null
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
-
-<book id="kgdbOnLinux">
- <bookinfo>
- <title>Using kgdb, kdb and the kernel debugger internals</title>
-
- <authorgroup>
- <author>
- <firstname>Jason</firstname>
- <surname>Wessel</surname>
- <affiliation>
- <address>
- <email>jason.wessel@windriver.com</email>
- </address>
- </affiliation>
- </author>
- </authorgroup>
- <copyright>
- <year>2008,2010</year>
- <holder>Wind River Systems, Inc.</holder>
- </copyright>
- <copyright>
- <year>2004-2005</year>
- <holder>MontaVista Software, Inc.</holder>
- </copyright>
- <copyright>
- <year>2004</year>
- <holder>Amit S. Kale</holder>
- </copyright>
-
- <legalnotice>
- <para>
- This file is licensed under the terms of the GNU General Public License
- version 2. This program is licensed "as is" without any warranty of any
- kind, whether express or implied.
- </para>
-
- </legalnotice>
- </bookinfo>
-
-<toc></toc>
- <chapter id="Introduction">
- <title>Introduction</title>
- <para>
- The kernel has two different debugger front ends (kdb and kgdb)
- which interface to the debug core. It is possible to use either
- of the debugger front ends and dynamically transition between them
- if you configure the kernel properly at compile and runtime.
- </para>
- <para>
- Kdb is simplistic shell-style interface which you can use on a
- system console with a keyboard or serial console. You can use it
- to inspect memory, registers, process lists, dmesg, and even set
- breakpoints to stop in a certain location. Kdb is not a source
- level debugger, although you can set breakpoints and execute some
- basic kernel run control. Kdb is mainly aimed at doing some
- analysis to aid in development or diagnosing kernel problems. You
- can access some symbols by name in kernel built-ins or in kernel
- modules if the code was built
- with <symbol>CONFIG_KALLSYMS</symbol>.
- </para>
- <para>
- Kgdb is intended to be used as a source level debugger for the
- Linux kernel. It is used along with gdb to debug a Linux kernel.
- The expectation is that gdb can be used to "break in" to the
- kernel to inspect memory, variables and look through call stack
- information similar to the way an application developer would use
- gdb to debug an application. It is possible to place breakpoints
- in kernel code and perform some limited execution stepping.
- </para>
- <para>
- Two machines are required for using kgdb. One of these machines is
- a development machine and the other is the target machine. The
- kernel to be debugged runs on the target machine. The development
- machine runs an instance of gdb against the vmlinux file which
- contains the symbols (not a boot image such as bzImage, zImage,
- uImage...). In gdb the developer specifies the connection
- parameters and connects to kgdb. The type of connection a
- developer makes with gdb depends on the availability of kgdb I/O
- modules compiled as built-ins or loadable kernel modules in the test
- machine's kernel.
- </para>
- </chapter>
- <chapter id="CompilingAKernel">
- <title>Compiling a kernel</title>
- <para>
- <itemizedlist>
- <listitem><para>In order to enable compilation of kdb, you must first enable kgdb.</para></listitem>
- <listitem><para>The kgdb test compile options are described in the kgdb test suite chapter.</para></listitem>
- </itemizedlist>
- </para>
- <sect1 id="CompileKGDB">
- <title>Kernel config options for kgdb</title>
- <para>
- To enable <symbol>CONFIG_KGDB</symbol> you should look under
- "Kernel hacking" / "Kernel debugging" and select "KGDB: kernel debugger".
- </para>
- <para>
- While it is not a hard requirement that you have symbols in your
- vmlinux file, gdb tends not to be very useful without the symbolic
- data, so you will want to turn
- on <symbol>CONFIG_DEBUG_INFO</symbol> which is called "Compile the
- kernel with debug info" in the config menu.
- </para>
- <para>
- It is advised, but not required, that you turn on the
- <symbol>CONFIG_FRAME_POINTER</symbol> kernel option which is called "Compile the
- kernel with frame pointers" in the config menu. This option
- inserts code to into the compiled executable which saves the frame
- information in registers or on the stack at different points which
- allows a debugger such as gdb to more accurately construct
- stack back traces while debugging the kernel.
- </para>
- <para>
- If the architecture that you are using supports the kernel option
- CONFIG_STRICT_KERNEL_RWX, you should consider turning it off. This
- option will prevent the use of software breakpoints because it
- marks certain regions of the kernel's memory space as read-only.
- If kgdb supports it for the architecture you are using, you can
- use hardware breakpoints if you desire to run with the
- CONFIG_STRICT_KERNEL_RWX option turned on, else you need to turn off
- this option.
- </para>
- <para>
- Next you should choose one of more I/O drivers to interconnect
- debugging host and debugged target. Early boot debugging requires
- a KGDB I/O driver that supports early debugging and the driver
- must be built into the kernel directly. Kgdb I/O driver
- configuration takes place via kernel or module parameters which
- you can learn more about in the in the section that describes the
- parameter "kgdboc".
- </para>
- <para>Here is an example set of .config symbols to enable or
- disable for kgdb:
- <itemizedlist>
- <listitem><para># CONFIG_STRICT_KERNEL_RWX is not set</para></listitem>
- <listitem><para>CONFIG_FRAME_POINTER=y</para></listitem>
- <listitem><para>CONFIG_KGDB=y</para></listitem>
- <listitem><para>CONFIG_KGDB_SERIAL_CONSOLE=y</para></listitem>
- </itemizedlist>
- </para>
- </sect1>
- <sect1 id="CompileKDB">
- <title>Kernel config options for kdb</title>
- <para>Kdb is quite a bit more complex than the simple gdbstub
- sitting on top of the kernel's debug core. Kdb must implement a
- shell, and also adds some helper functions in other parts of the
- kernel, responsible for printing out interesting data such as what
- you would see if you ran "lsmod", or "ps". In order to build kdb
- into the kernel you follow the same steps as you would for kgdb.
- </para>
- <para>The main config option for kdb
- is <symbol>CONFIG_KGDB_KDB</symbol> which is called "KGDB_KDB:
- include kdb frontend for kgdb" in the config menu. In theory you
- would have already also selected an I/O driver such as the
- CONFIG_KGDB_SERIAL_CONSOLE interface if you plan on using kdb on a
- serial port, when you were configuring kgdb.
- </para>
- <para>If you want to use a PS/2-style keyboard with kdb, you would
- select CONFIG_KDB_KEYBOARD which is called "KGDB_KDB: keyboard as
- input device" in the config menu. The CONFIG_KDB_KEYBOARD option
- is not used for anything in the gdb interface to kgdb. The
- CONFIG_KDB_KEYBOARD option only works with kdb.
- </para>
- <para>Here is an example set of .config symbols to enable/disable kdb:
- <itemizedlist>
- <listitem><para># CONFIG_STRICT_KERNEL_RWX is not set</para></listitem>
- <listitem><para>CONFIG_FRAME_POINTER=y</para></listitem>
- <listitem><para>CONFIG_KGDB=y</para></listitem>
- <listitem><para>CONFIG_KGDB_SERIAL_CONSOLE=y</para></listitem>
- <listitem><para>CONFIG_KGDB_KDB=y</para></listitem>
- <listitem><para>CONFIG_KDB_KEYBOARD=y</para></listitem>
- </itemizedlist>
- </para>
- </sect1>
- </chapter>
- <chapter id="kgdbKernelArgs">
- <title>Kernel Debugger Boot Arguments</title>
- <para>This section describes the various runtime kernel
- parameters that affect the configuration of the kernel debugger.
- The following chapter covers using kdb and kgdb as well as
- providing some examples of the configuration parameters.</para>
- <sect1 id="kgdboc">
- <title>Kernel parameter: kgdboc</title>
- <para>The kgdboc driver was originally an abbreviation meant to
- stand for "kgdb over console". Today it is the primary mechanism
- to configure how to communicate from gdb to kgdb as well as the
- devices you want to use to interact with the kdb shell.
- </para>
- <para>For kgdb/gdb, kgdboc is designed to work with a single serial
- port. It is intended to cover the circumstance where you want to
- use a serial console as your primary console as well as using it to
- perform kernel debugging. It is also possible to use kgdb on a
- serial port which is not designated as a system console. Kgdboc
- may be configured as a kernel built-in or a kernel loadable module.
- You can only make use of <constant>kgdbwait</constant> and early
- debugging if you build kgdboc into the kernel as a built-in.
- </para>
- <para>Optionally you can elect to activate kms (Kernel Mode
- Setting) integration. When you use kms with kgdboc and you have a
- video driver that has atomic mode setting hooks, it is possible to
- enter the debugger on the graphics console. When the kernel
- execution is resumed, the previous graphics mode will be restored.
- This integration can serve as a useful tool to aid in diagnosing
- crashes or doing analysis of memory with kdb while allowing the
- full graphics console applications to run.
- </para>
- <sect2 id="kgdbocArgs">
- <title>kgdboc arguments</title>
- <para>Usage: <constant>kgdboc=[kms][[,]kbd][[,]serial_device][,baud]</constant></para>
- <para>The order listed above must be observed if you use any of the
- optional configurations together.
- </para>
- <para>Abbreviations:
- <itemizedlist>
- <listitem><para>kms = Kernel Mode Setting</para></listitem>
- <listitem><para>kbd = Keyboard</para></listitem>
- </itemizedlist>
- </para>
- <para>You can configure kgdboc to use the keyboard, and/or a serial
- device depending on if you are using kdb and/or kgdb, in one of the
- following scenarios. The order listed above must be observed if
- you use any of the optional configurations together. Using kms +
- only gdb is generally not a useful combination.</para>
- <sect3 id="kgdbocArgs1">
- <title>Using loadable module or built-in</title>
- <para>
- <orderedlist>
- <listitem><para>As a kernel built-in:</para>
- <para>Use the kernel boot argument: <constant>kgdboc=<tty-device>,[baud]</constant></para></listitem>
- <listitem>
- <para>As a kernel loadable module:</para>
- <para>Use the command: <constant>modprobe kgdboc kgdboc=<tty-device>,[baud]</constant></para>
- <para>Here are two examples of how you might format the kgdboc
- string. The first is for an x86 target using the first serial port.
- The second example is for the ARM Versatile AB using the second
- serial port.
- <orderedlist>
- <listitem><para><constant>kgdboc=ttyS0,115200</constant></para></listitem>
- <listitem><para><constant>kgdboc=ttyAMA1,115200</constant></para></listitem>
- </orderedlist>
- </para>
- </listitem>
- </orderedlist></para>
- </sect3>
- <sect3 id="kgdbocArgs2">
- <title>Configure kgdboc at runtime with sysfs</title>
- <para>At run time you can enable or disable kgdboc by echoing a
- parameters into the sysfs. Here are two examples:</para>
- <orderedlist>
- <listitem><para>Enable kgdboc on ttyS0</para>
- <para><constant>echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc</constant></para></listitem>
- <listitem><para>Disable kgdboc</para>
- <para><constant>echo "" > /sys/module/kgdboc/parameters/kgdboc</constant></para></listitem>
- </orderedlist>
- <para>NOTE: You do not need to specify the baud if you are
- configuring the console on tty which is already configured or
- open.</para>
- </sect3>
- <sect3 id="kgdbocArgs3">
- <title>More examples</title>
- <para>You can configure kgdboc to use the keyboard, and/or a serial device
- depending on if you are using kdb and/or kgdb, in one of the
- following scenarios.
- <orderedlist>
- <listitem><para>kdb and kgdb over only a serial port</para>
- <para><constant>kgdboc=<serial_device>[,baud]</constant></para>
- <para>Example: <constant>kgdboc=ttyS0,115200</constant></para>
- </listitem>
- <listitem><para>kdb and kgdb with keyboard and a serial port</para>
- <para><constant>kgdboc=kbd,<serial_device>[,baud]</constant></para>
- <para>Example: <constant>kgdboc=kbd,ttyS0,115200</constant></para>
- </listitem>
- <listitem><para>kdb with a keyboard</para>
- <para><constant>kgdboc=kbd</constant></para>
- </listitem>
- <listitem><para>kdb with kernel mode setting</para>
- <para><constant>kgdboc=kms,kbd</constant></para>
- </listitem>
- <listitem><para>kdb with kernel mode setting and kgdb over a serial port</para>
- <para><constant>kgdboc=kms,kbd,ttyS0,115200</constant></para>
- </listitem>
- </orderedlist>
- </para>
- <para>NOTE: Kgdboc does not support interrupting the target via the
- gdb remote protocol. You must manually send a sysrq-g unless you
- have a proxy that splits console output to a terminal program.
- A console proxy has a separate TCP port for the debugger and a separate
- TCP port for the "human" console. The proxy can take care of sending
- the sysrq-g for you.
- </para>
- <para>When using kgdboc with no debugger proxy, you can end up
- connecting the debugger at one of two entry points. If an
- exception occurs after you have loaded kgdboc, a message should
- print on the console stating it is waiting for the debugger. In
- this case you disconnect your terminal program and then connect the
- debugger in its place. If you want to interrupt the target system
- and forcibly enter a debug session you have to issue a Sysrq
- sequence and then type the letter <constant>g</constant>. Then
- you disconnect the terminal session and connect gdb. Your options
- if you don't like this are to hack gdb to send the sysrq-g for you
- as well as on the initial connect, or to use a debugger proxy that
- allows an unmodified gdb to do the debugging.
- </para>
- </sect3>
- </sect2>
- </sect1>
- <sect1 id="kgdbwait">
- <title>Kernel parameter: kgdbwait</title>
- <para>
- The Kernel command line option <constant>kgdbwait</constant> makes
- kgdb wait for a debugger connection during booting of a kernel. You
- can only use this option if you compiled a kgdb I/O driver into the
- kernel and you specified the I/O driver configuration as a kernel
- command line option. The kgdbwait parameter should always follow the
- configuration parameter for the kgdb I/O driver in the kernel
- command line else the I/O driver will not be configured prior to
- asking the kernel to use it to wait.
- </para>
- <para>
- The kernel will stop and wait as early as the I/O driver and
- architecture allows when you use this option. If you build the
- kgdb I/O driver as a loadable kernel module kgdbwait will not do
- anything.
- </para>
- </sect1>
- <sect1 id="kgdbcon">
- <title>Kernel parameter: kgdbcon</title>
- <para> The kgdbcon feature allows you to see printk() messages
- inside gdb while gdb is connected to the kernel. Kdb does not make
- use of the kgdbcon feature.
- </para>
- <para>Kgdb supports using the gdb serial protocol to send console
- messages to the debugger when the debugger is connected and running.
- There are two ways to activate this feature.
- <orderedlist>
- <listitem><para>Activate with the kernel command line option:</para>
- <para><constant>kgdbcon</constant></para>
- </listitem>
- <listitem><para>Use sysfs before configuring an I/O driver</para>
- <para>
- <constant>echo 1 > /sys/module/kgdb/parameters/kgdb_use_con</constant>
- </para>
- <para>
- NOTE: If you do this after you configure the kgdb I/O driver, the
- setting will not take effect until the next point the I/O is
- reconfigured.
- </para>
- </listitem>
- </orderedlist>
- </para>
- <para>IMPORTANT NOTE: You cannot use kgdboc + kgdbcon on a tty that is an
- active system console. An example of incorrect usage is <constant>console=ttyS0,115200 kgdboc=ttyS0 kgdbcon</constant>
- </para>
- <para>It is possible to use this option with kgdboc on a tty that is not a system console.
- </para>
- </sect1>
- <sect1 id="kgdbreboot">
- <title>Run time parameter: kgdbreboot</title>
- <para> The kgdbreboot feature allows you to change how the debugger
- deals with the reboot notification. You have 3 choices for the
- behavior. The default behavior is always set to 0.</para>
- <orderedlist>
- <listitem><para>echo -1 > /sys/module/debug_core/parameters/kgdbreboot</para>
- <para>Ignore the reboot notification entirely.</para>
- </listitem>
- <listitem><para>echo 0 > /sys/module/debug_core/parameters/kgdbreboot</para>
- <para>Send the detach message to any attached debugger client.</para>
- </listitem>
- <listitem><para>echo 1 > /sys/module/debug_core/parameters/kgdbreboot</para>
- <para>Enter the debugger on reboot notify.</para>
- </listitem>
- </orderedlist>
- </sect1>
- </chapter>
- <chapter id="usingKDB">
- <title>Using kdb</title>
- <para>
- </para>
- <sect1 id="quickKDBserial">
- <title>Quick start for kdb on a serial port</title>
- <para>This is a quick example of how to use kdb.</para>
- <para><orderedlist>
- <listitem><para>Configure kgdboc at boot using kernel parameters:
- <itemizedlist>
- <listitem><para><constant>console=ttyS0,115200 kgdboc=ttyS0,115200</constant></para></listitem>
- </itemizedlist></para>
- <para>OR</para>
- <para>Configure kgdboc after the kernel has booted; assuming you are using a serial port console:
- <itemizedlist>
- <listitem><para><constant>echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc</constant></para></listitem>
- </itemizedlist>
- </para>
- </listitem>
- <listitem><para>Enter the kernel debugger manually or by waiting for an oops or fault. There are several ways you can enter the kernel debugger manually; all involve using the sysrq-g, which means you must have enabled CONFIG_MAGIC_SYSRQ=y in your kernel config.</para>
- <itemizedlist>
- <listitem><para>When logged in as root or with a super user session you can run:</para>
- <para><constant>echo g > /proc/sysrq-trigger</constant></para></listitem>
- <listitem><para>Example using minicom 2.2</para>
- <para>Press: <constant>Control-a</constant></para>
- <para>Press: <constant>f</constant></para>
- <para>Press: <constant>g</constant></para>
- </listitem>
- <listitem><para>When you have telneted to a terminal server that supports sending a remote break</para>
- <para>Press: <constant>Control-]</constant></para>
- <para>Type in:<constant>send break</constant></para>
- <para>Press: <constant>Enter</constant></para>
- <para>Press: <constant>g</constant></para>
- </listitem>
- </itemizedlist>
- </listitem>
- <listitem><para>From the kdb prompt you can run the "help" command to see a complete list of the commands that are available.</para>
- <para>Some useful commands in kdb include:
- <itemizedlist>
- <listitem><para>lsmod -- Shows where kernel modules are loaded</para></listitem>
- <listitem><para>ps -- Displays only the active processes</para></listitem>
- <listitem><para>ps A -- Shows all the processes</para></listitem>
- <listitem><para>summary -- Shows kernel version info and memory usage</para></listitem>
- <listitem><para>bt -- Get a backtrace of the current process using dump_stack()</para></listitem>
- <listitem><para>dmesg -- View the kernel syslog buffer</para></listitem>
- <listitem><para>go -- Continue the system</para></listitem>
- </itemizedlist>
- </para>
- </listitem>
- <listitem>
- <para>When you are done using kdb you need to consider rebooting the
- system or using the "go" command to resuming normal kernel
- execution. If you have paused the kernel for a lengthy period of
- time, applications that rely on timely networking or anything to do
- with real wall clock time could be adversely affected, so you
- should take this into consideration when using the kernel
- debugger.</para>
- </listitem>
- </orderedlist></para>
- </sect1>
- <sect1 id="quickKDBkeyboard">
- <title>Quick start for kdb using a keyboard connected console</title>
- <para>This is a quick example of how to use kdb with a keyboard.</para>
- <para><orderedlist>
- <listitem><para>Configure kgdboc at boot using kernel parameters:
- <itemizedlist>
- <listitem><para><constant>kgdboc=kbd</constant></para></listitem>
- </itemizedlist></para>
- <para>OR</para>
- <para>Configure kgdboc after the kernel has booted:
- <itemizedlist>
- <listitem><para><constant>echo kbd > /sys/module/kgdboc/parameters/kgdboc</constant></para></listitem>
- </itemizedlist>
- </para>
- </listitem>
- <listitem><para>Enter the kernel debugger manually or by waiting for an oops or fault. There are several ways you can enter the kernel debugger manually; all involve using the sysrq-g, which means you must have enabled CONFIG_MAGIC_SYSRQ=y in your kernel config.</para>
- <itemizedlist>
- <listitem><para>When logged in as root or with a super user session you can run:</para>
- <para><constant>echo g > /proc/sysrq-trigger</constant></para></listitem>
- <listitem><para>Example using a laptop keyboard</para>
- <para>Press and hold down: <constant>Alt</constant></para>
- <para>Press and hold down: <constant>Fn</constant></para>
- <para>Press and release the key with the label: <constant>SysRq</constant></para>
- <para>Release: <constant>Fn</constant></para>
- <para>Press and release: <constant>g</constant></para>
- <para>Release: <constant>Alt</constant></para>
- </listitem>
- <listitem><para>Example using a PS/2 101-key keyboard</para>
- <para>Press and hold down: <constant>Alt</constant></para>
- <para>Press and release the key with the label: <constant>SysRq</constant></para>
- <para>Press and release: <constant>g</constant></para>
- <para>Release: <constant>Alt</constant></para>
- </listitem>
- </itemizedlist>
- </listitem>
- <listitem>
- <para>Now type in a kdb command such as "help", "dmesg", "bt" or "go" to continue kernel execution.</para>
- </listitem>
- </orderedlist></para>
- </sect1>
- </chapter>
- <chapter id="EnableKGDB">
- <title>Using kgdb / gdb</title>
- <para>In order to use kgdb you must activate it by passing
- configuration information to one of the kgdb I/O drivers. If you
- do not pass any configuration information kgdb will not do anything
- at all. Kgdb will only actively hook up to the kernel trap hooks
- if a kgdb I/O driver is loaded and configured. If you unconfigure
- a kgdb I/O driver, kgdb will unregister all the kernel hook points.
- </para>
- <para> All kgdb I/O drivers can be reconfigured at run time, if
- <symbol>CONFIG_SYSFS</symbol> and <symbol>CONFIG_MODULES</symbol>
- are enabled, by echo'ing a new config string to
- <constant>/sys/module/<driver>/parameter/<option></constant>.
- The driver can be unconfigured by passing an empty string. You cannot
- change the configuration while the debugger is attached. Make sure
- to detach the debugger with the <constant>detach</constant> command
- prior to trying to unconfigure a kgdb I/O driver.
- </para>
- <sect1 id="ConnectingGDB">
- <title>Connecting with gdb to a serial port</title>
- <orderedlist>
- <listitem><para>Configure kgdboc</para>
- <para>Configure kgdboc at boot using kernel parameters:
- <itemizedlist>
- <listitem><para><constant>kgdboc=ttyS0,115200</constant></para></listitem>
- </itemizedlist></para>
- <para>OR</para>
- <para>Configure kgdboc after the kernel has booted:
- <itemizedlist>
- <listitem><para><constant>echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc</constant></para></listitem>
- </itemizedlist></para>
- </listitem>
- <listitem>
- <para>Stop kernel execution (break into the debugger)</para>
- <para>In order to connect to gdb via kgdboc, the kernel must
- first be stopped. There are several ways to stop the kernel which
- include using kgdbwait as a boot argument, via a sysrq-g, or running
- the kernel until it takes an exception where it waits for the
- debugger to attach.
- <itemizedlist>
- <listitem><para>When logged in as root or with a super user session you can run:</para>
- <para><constant>echo g > /proc/sysrq-trigger</constant></para></listitem>
- <listitem><para>Example using minicom 2.2</para>
- <para>Press: <constant>Control-a</constant></para>
- <para>Press: <constant>f</constant></para>
- <para>Press: <constant>g</constant></para>
- </listitem>
- <listitem><para>When you have telneted to a terminal server that supports sending a remote break</para>
- <para>Press: <constant>Control-]</constant></para>
- <para>Type in:<constant>send break</constant></para>
- <para>Press: <constant>Enter</constant></para>
- <para>Press: <constant>g</constant></para>
- </listitem>
- </itemizedlist>
- </para>
- </listitem>
- <listitem>
- <para>Connect from gdb</para>
- <para>
- Example (using a directly connected port):
- </para>
- <programlisting>
- % gdb ./vmlinux
- (gdb) set remotebaud 115200
- (gdb) target remote /dev/ttyS0
- </programlisting>
- <para>
- Example (kgdb to a terminal server on TCP port 2012):
- </para>
- <programlisting>
- % gdb ./vmlinux
- (gdb) target remote 192.168.2.2:2012
- </programlisting>
- <para>
- Once connected, you can debug a kernel the way you would debug an
- application program.
- </para>
- <para>
- If you are having problems connecting or something is going
- seriously wrong while debugging, it will most often be the case
- that you want to enable gdb to be verbose about its target
- communications. You do this prior to issuing the <constant>target
- remote</constant> command by typing in: <constant>set debug remote 1</constant>
- </para>
- </listitem>
- </orderedlist>
- <para>Remember if you continue in gdb, and need to "break in" again,
- you need to issue an other sysrq-g. It is easy to create a simple
- entry point by putting a breakpoint at <constant>sys_sync</constant>
- and then you can run "sync" from a shell or script to break into the
- debugger.</para>
- </sect1>
- </chapter>
- <chapter id="switchKdbKgdb">
- <title>kgdb and kdb interoperability</title>
- <para>It is possible to transition between kdb and kgdb dynamically.
- The debug core will remember which you used the last time and
- automatically start in the same mode.</para>
- <sect1>
- <title>Switching between kdb and kgdb</title>
- <sect2>
- <title>Switching from kgdb to kdb</title>
- <para>
- There are two ways to switch from kgdb to kdb: you can use gdb to
- issue a maintenance packet, or you can blindly type the command $3#33.
- Whenever the kernel debugger stops in kgdb mode it will print the
- message <constant>KGDB or $3#33 for KDB</constant>. It is important
- to note that you have to type the sequence correctly in one pass.
- You cannot type a backspace or delete because kgdb will interpret
- that as part of the debug stream.
- <orderedlist>
- <listitem><para>Change from kgdb to kdb by blindly typing:</para>
- <para><constant>$3#33</constant></para></listitem>
- <listitem><para>Change from kgdb to kdb with gdb</para>
- <para><constant>maintenance packet 3</constant></para>
- <para>NOTE: Now you must kill gdb. Typically you press control-z and
- issue the command: kill -9 %</para></listitem>
- </orderedlist>
- </para>
- </sect2>
- <sect2>
- <title>Change from kdb to kgdb</title>
- <para>There are two ways you can change from kdb to kgdb. You can
- manually enter kgdb mode by issuing the kgdb command from the kdb
- shell prompt, or you can connect gdb while the kdb shell prompt is
- active. The kdb shell looks for the typical first commands that gdb
- would issue with the gdb remote protocol and if it sees one of those
- commands it automatically changes into kgdb mode.</para>
- <orderedlist>
- <listitem><para>From kdb issue the command:</para>
- <para><constant>kgdb</constant></para>
- <para>Now disconnect your terminal program and connect gdb in its place</para></listitem>
- <listitem><para>At the kdb prompt, disconnect the terminal program and connect gdb in its place.</para></listitem>
- </orderedlist>
- </sect2>
- </sect1>
- <sect1>
- <title>Running kdb commands from gdb</title>
- <para>It is possible to run a limited set of kdb commands from gdb,
- using the gdb monitor command. You don't want to execute any of the
- run control or breakpoint operations, because it can disrupt the
- state of the kernel debugger. You should be using gdb for
- breakpoints and run control operations if you have gdb connected.
- The more useful commands to run are things like lsmod, dmesg, ps or
- possibly some of the memory information commands. To see all the kdb
- commands you can run <constant>monitor help</constant>.</para>
- <para>Example:
- <informalexample><programlisting>
-(gdb) monitor ps
-1 idle process (state I) and
-27 sleeping system daemon (state M) processes suppressed,
-use 'ps A' to see all.
-Task Addr Pid Parent [*] cpu State Thread Command
-
-0xc78291d0 1 0 0 0 S 0xc7829404 init
-0xc7954150 942 1 0 0 S 0xc7954384 dropbear
-0xc78789c0 944 1 0 0 S 0xc7878bf4 sh
-(gdb)
- </programlisting></informalexample>
- </para>
- </sect1>
- </chapter>
- <chapter id="KGDBTestSuite">
- <title>kgdb Test Suite</title>
- <para>
- When kgdb is enabled in the kernel config you can also elect to
- enable the config parameter KGDB_TESTS. Turning this on will
- enable a special kgdb I/O module which is designed to test the
- kgdb internal functions.
- </para>
- <para>
- The kgdb tests are mainly intended for developers to test the kgdb
- internals as well as a tool for developing a new kgdb architecture
- specific implementation. These tests are not really for end users
- of the Linux kernel. The primary source of documentation would be
- to look in the drivers/misc/kgdbts.c file.
- </para>
- <para>
- The kgdb test suite can also be configured at compile time to run
- the core set of tests by setting the kernel config parameter
- KGDB_TESTS_ON_BOOT. This particular option is aimed at automated
- regression testing and does not require modifying the kernel boot
- config arguments. If this is turned on, the kgdb test suite can
- be disabled by specifying "kgdbts=" as a kernel boot argument.
- </para>
- </chapter>
- <chapter id="CommonBackEndReq">
- <title>Kernel Debugger Internals</title>
- <sect1 id="kgdbArchitecture">
- <title>Architecture Specifics</title>
- <para>
- The kernel debugger is organized into a number of components:
- <orderedlist>
- <listitem><para>The debug core</para>
- <para>
- The debug core is found in kernel/debugger/debug_core.c. It contains:
- <itemizedlist>
- <listitem><para>A generic OS exception handler which includes
- sync'ing the processors into a stopped state on an multi-CPU
- system.</para></listitem>
- <listitem><para>The API to talk to the kgdb I/O drivers</para></listitem>
- <listitem><para>The API to make calls to the arch-specific kgdb implementation</para></listitem>
- <listitem><para>The logic to perform safe memory reads and writes to memory while using the debugger</para></listitem>
- <listitem><para>A full implementation for software breakpoints unless overridden by the arch</para></listitem>
- <listitem><para>The API to invoke either the kdb or kgdb frontend to the debug core.</para></listitem>
- <listitem><para>The structures and callback API for atomic kernel mode setting.</para>
- <para>NOTE: kgdboc is where the kms callbacks are invoked.</para></listitem>
- </itemizedlist>
- </para>
- </listitem>
- <listitem><para>kgdb arch-specific implementation</para>
- <para>
- This implementation is generally found in arch/*/kernel/kgdb.c.
- As an example, arch/x86/kernel/kgdb.c contains the specifics to
- implement HW breakpoint as well as the initialization to
- dynamically register and unregister for the trap handlers on
- this architecture. The arch-specific portion implements:
- <itemizedlist>
- <listitem><para>contains an arch-specific trap catcher which
- invokes kgdb_handle_exception() to start kgdb about doing its
- work</para></listitem>
- <listitem><para>translation to and from gdb specific packet format to pt_regs</para></listitem>
- <listitem><para>Registration and unregistration of architecture specific trap hooks</para></listitem>
- <listitem><para>Any special exception handling and cleanup</para></listitem>
- <listitem><para>NMI exception handling and cleanup</para></listitem>
- <listitem><para>(optional) HW breakpoints</para></listitem>
- </itemizedlist>
- </para>
- </listitem>
- <listitem><para>gdbstub frontend (aka kgdb)</para>
- <para>The gdbstub is located in kernel/debug/gdbstub.c. It contains:</para>
- <itemizedlist>
- <listitem><para>All the logic to implement the gdb serial protocol</para></listitem>
- </itemizedlist>
- </listitem>
- <listitem><para>kdb frontend</para>
- <para>The kdb debugger shell is broken down into a number of
- components. The kdb core is located in kernel/debug/kdb. There
- are a number of helper functions in some of the other kernel
- components to make it possible for kdb to examine and report
- information about the kernel without taking locks that could
- cause a kernel deadlock. The kdb core contains implements the following functionality.</para>
- <itemizedlist>
- <listitem><para>A simple shell</para></listitem>
- <listitem><para>The kdb core command set</para></listitem>
- <listitem><para>A registration API to register additional kdb shell commands.</para>
- <itemizedlist>
- <listitem><para>A good example of a self-contained kdb module
- is the "ftdump" command for dumping the ftrace buffer. See:
- kernel/trace/trace_kdb.c</para></listitem>
- <listitem><para>For an example of how to dynamically register
- a new kdb command you can build the kdb_hello.ko kernel module
- from samples/kdb/kdb_hello.c. To build this example you can
- set CONFIG_SAMPLES=y and CONFIG_SAMPLE_KDB=m in your kernel
- config. Later run "modprobe kdb_hello" and the next time you
- enter the kdb shell, you can run the "hello"
- command.</para></listitem>
- </itemizedlist></listitem>
- <listitem><para>The implementation for kdb_printf() which
- emits messages directly to I/O drivers, bypassing the kernel
- log.</para></listitem>
- <listitem><para>SW / HW breakpoint management for the kdb shell</para></listitem>
- </itemizedlist>
- </listitem>
- <listitem><para>kgdb I/O driver</para>
- <para>
- Each kgdb I/O driver has to provide an implementation for the following:
- <itemizedlist>
- <listitem><para>configuration via built-in or module</para></listitem>
- <listitem><para>dynamic configuration and kgdb hook registration calls</para></listitem>
- <listitem><para>read and write character interface</para></listitem>
- <listitem><para>A cleanup handler for unconfiguring from the kgdb core</para></listitem>
- <listitem><para>(optional) Early debug methodology</para></listitem>
- </itemizedlist>
- Any given kgdb I/O driver has to operate very closely with the
- hardware and must do it in such a way that does not enable
- interrupts or change other parts of the system context without
- completely restoring them. The kgdb core will repeatedly "poll"
- a kgdb I/O driver for characters when it needs input. The I/O
- driver is expected to return immediately if there is no data
- available. Doing so allows for the future possibility to touch
- watchdog hardware in such a way as to have a target system not
- reset when these are enabled.
- </para>
- </listitem>
- </orderedlist>
- </para>
- <para>
- If you are intent on adding kgdb architecture specific support
- for a new architecture, the architecture should define
- <constant>HAVE_ARCH_KGDB</constant> in the architecture specific
- Kconfig file. This will enable kgdb for the architecture, and
- at that point you must create an architecture specific kgdb
- implementation.
- </para>
- <para>
- There are a few flags which must be set on every architecture in
- their <asm/kgdb.h> file. These are:
- <itemizedlist>
- <listitem>
- <para>
- NUMREGBYTES: The size in bytes of all of the registers, so
- that we can ensure they will all fit into a packet.
- </para>
- </listitem>
- <listitem>
- <para>
- BUFMAX: The size in bytes of the buffer GDB will read into.
- This must be larger than NUMREGBYTES.
- </para>
- </listitem>
- <listitem>
- <para>
- CACHE_FLUSH_IS_SAFE: Set to 1 if it is always safe to call
- flush_cache_range or flush_icache_range. On some architectures,
- these functions may not be safe to call on SMP since we keep other
- CPUs in a holding pattern.
- </para>
- </listitem>
- </itemizedlist>
- </para>
- <para>
- There are also the following functions for the common backend,
- found in kernel/kgdb.c, that must be supplied by the
- architecture-specific backend unless marked as (optional), in
- which case a default function maybe used if the architecture
- does not need to provide a specific implementation.
- </para>
-!Iinclude/linux/kgdb.h
- </sect1>
- <sect1 id="kgdbocDesign">
- <title>kgdboc internals</title>
- <sect2>
- <title>kgdboc and uarts</title>
- <para>
- The kgdboc driver is actually a very thin driver that relies on the
- underlying low level to the hardware driver having "polling hooks"
- to which the tty driver is attached. In the initial
- implementation of kgdboc the serial_core was changed to expose a
- low level UART hook for doing polled mode reading and writing of a
- single character while in an atomic context. When kgdb makes an I/O
- request to the debugger, kgdboc invokes a callback in the serial
- core which in turn uses the callback in the UART driver.</para>
- <para>
- When using kgdboc with a UART, the UART driver must implement two callbacks in the <constant>struct uart_ops</constant>. Example from drivers/8250.c:<programlisting>
-#ifdef CONFIG_CONSOLE_POLL
- .poll_get_char = serial8250_get_poll_char,
- .poll_put_char = serial8250_put_poll_char,
-#endif
- </programlisting>
- Any implementation specifics around creating a polling driver use the
- <constant>#ifdef CONFIG_CONSOLE_POLL</constant>, as shown above.
- Keep in mind that polling hooks have to be implemented in such a way
- that they can be called from an atomic context and have to restore
- the state of the UART chip on return such that the system can return
- to normal when the debugger detaches. You need to be very careful
- with any kind of lock you consider, because failing here is most likely
- going to mean pressing the reset button.
- </para>
- </sect2>
- <sect2 id="kgdbocKbd">
- <title>kgdboc and keyboards</title>
- <para>The kgdboc driver contains logic to configure communications
- with an attached keyboard. The keyboard infrastructure is only
- compiled into the kernel when CONFIG_KDB_KEYBOARD=y is set in the
- kernel configuration.</para>
- <para>The core polled keyboard driver driver for PS/2 type keyboards
- is in drivers/char/kdb_keyboard.c. This driver is hooked into the
- debug core when kgdboc populates the callback in the array
- called <constant>kdb_poll_funcs[]</constant>. The
- kdb_get_kbd_char() is the top-level function which polls hardware
- for single character input.
- </para>
- </sect2>
- <sect2 id="kgdbocKms">
- <title>kgdboc and kms</title>
- <para>The kgdboc driver contains logic to request the graphics
- display to switch to a text context when you are using
- "kgdboc=kms,kbd", provided that you have a video driver which has a
- frame buffer console and atomic kernel mode setting support.</para>
- <para>
- Every time the kernel
- debugger is entered it calls kgdboc_pre_exp_handler() which in turn
- calls con_debug_enter() in the virtual console layer. On resuming kernel
- execution, the kernel debugger calls kgdboc_post_exp_handler() which
- in turn calls con_debug_leave().</para>
- <para>Any video driver that wants to be compatible with the kernel
- debugger and the atomic kms callbacks must implement the
- mode_set_base_atomic, fb_debug_enter and fb_debug_leave operations.
- For the fb_debug_enter and fb_debug_leave the option exists to use
- the generic drm fb helper functions or implement something custom for
- the hardware. The following example shows the initialization of the
- .mode_set_base_atomic operation in
- drivers/gpu/drm/i915/intel_display.c:
- <informalexample>
- <programlisting>
-static const struct drm_crtc_helper_funcs intel_helper_funcs = {
-[...]
- .mode_set_base_atomic = intel_pipe_set_base_atomic,
-[...]
-};
- </programlisting>
- </informalexample>
- </para>
- <para>Here is an example of how the i915 driver initializes the fb_debug_enter and fb_debug_leave functions to use the generic drm helpers in
- drivers/gpu/drm/i915/intel_fb.c:
- <informalexample>
- <programlisting>
-static struct fb_ops intelfb_ops = {
-[...]
- .fb_debug_enter = drm_fb_helper_debug_enter,
- .fb_debug_leave = drm_fb_helper_debug_leave,
-[...]
-};
- </programlisting>
- </informalexample>
- </para>
- </sect2>
- </sect1>
- </chapter>
- <chapter id="credits">
- <title>Credits</title>
- <para>
- The following people have contributed to this document:
- <orderedlist>
- <listitem><para>Amit Kale<email>amitkale@linsyssoft.com</email></para></listitem>
- <listitem><para>Tom Rini<email>trini@kernel.crashing.org</email></para></listitem>
- </orderedlist>
- In March 2008 this document was completely rewritten by:
- <itemizedlist>
- <listitem><para>Jason Wessel<email>jason.wessel@windriver.com</email></para></listitem>
- </itemizedlist>
- In Jan 2010 this document was updated to include kdb.
- <itemizedlist>
- <listitem><para>Jason Wessel<email>jason.wessel@windriver.com</email></para></listitem>
- </itemizedlist>
- </para>
- </chapter>
-</book>
-
+++ /dev/null
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
-
-<book id="libataDevGuide">
- <bookinfo>
- <title>libATA Developer's Guide</title>
-
- <authorgroup>
- <author>
- <firstname>Jeff</firstname>
- <surname>Garzik</surname>
- </author>
- </authorgroup>
-
- <copyright>
- <year>2003-2006</year>
- <holder>Jeff Garzik</holder>
- </copyright>
-
- <legalnotice>
- <para>
- The contents of this file are subject to the Open
- Software License version 1.1 that can be found at
- <ulink url="http://fedoraproject.org/wiki/Licensing:OSL1.1">http://fedoraproject.org/wiki/Licensing:OSL1.1</ulink>
- and is included herein by reference.
- </para>
-
- <para>
- Alternatively, the contents of this file may be used under the terms
- of the GNU General Public License version 2 (the "GPL") as distributed
- in the kernel source COPYING file, in which case the provisions of
- the GPL are applicable instead of the above. If you wish to allow
- the use of your version of this file only under the terms of the
- GPL and not to allow others to use your version of this file under
- the OSL, indicate your decision by deleting the provisions above and
- replace them with the notice and other provisions required by the GPL.
- If you do not delete the provisions above, a recipient may use your
- version of this file under either the OSL or the GPL.
- </para>
-
- </legalnotice>
- </bookinfo>
-
-<toc></toc>
-
- <chapter id="libataIntroduction">
- <title>Introduction</title>
- <para>
- libATA is a library used inside the Linux kernel to support ATA host
- controllers and devices. libATA provides an ATA driver API, class
- transports for ATA and ATAPI devices, and SCSI<->ATA translation
- for ATA devices according to the T10 SAT specification.
- </para>
- <para>
- This Guide documents the libATA driver API, library functions, library
- internals, and a couple sample ATA low-level drivers.
- </para>
- </chapter>
-
- <chapter id="libataDriverApi">
- <title>libata Driver API</title>
- <para>
- struct ata_port_operations is defined for every low-level libata
- hardware driver, and it controls how the low-level driver
- interfaces with the ATA and SCSI layers.
- </para>
- <para>
- FIS-based drivers will hook into the system with ->qc_prep() and
- ->qc_issue() high-level hooks. Hardware which behaves in a manner
- similar to PCI IDE hardware may utilize several generic helpers,
- defining at a bare minimum the bus I/O addresses of the ATA shadow
- register blocks.
- </para>
- <sect1>
- <title>struct ata_port_operations</title>
-
- <sect2><title>Disable ATA port</title>
- <programlisting>
-void (*port_disable) (struct ata_port *);
- </programlisting>
-
- <para>
- Called from ata_bus_probe() error path, as well as when
- unregistering from the SCSI module (rmmod, hot unplug).
- This function should do whatever needs to be done to take the
- port out of use. In most cases, ata_port_disable() can be used
- as this hook.
- </para>
- <para>
- Called from ata_bus_probe() on a failed probe.
- Called from ata_scsi_release().
- </para>
-
- </sect2>
-
- <sect2><title>Post-IDENTIFY device configuration</title>
- <programlisting>
-void (*dev_config) (struct ata_port *, struct ata_device *);
- </programlisting>
-
- <para>
- Called after IDENTIFY [PACKET] DEVICE is issued to each device
- found. Typically used to apply device-specific fixups prior to
- issue of SET FEATURES - XFER MODE, and prior to operation.
- </para>
- <para>
- This entry may be specified as NULL in ata_port_operations.
- </para>
-
- </sect2>
-
- <sect2><title>Set PIO/DMA mode</title>
- <programlisting>
-void (*set_piomode) (struct ata_port *, struct ata_device *);
-void (*set_dmamode) (struct ata_port *, struct ata_device *);
-void (*post_set_mode) (struct ata_port *);
-unsigned int (*mode_filter) (struct ata_port *, struct ata_device *, unsigned int);
- </programlisting>
-
- <para>
- Hooks called prior to the issue of SET FEATURES - XFER MODE
- command. The optional ->mode_filter() hook is called when libata
- has built a mask of the possible modes. This is passed to the
- ->mode_filter() function which should return a mask of valid modes
- after filtering those unsuitable due to hardware limits. It is not
- valid to use this interface to add modes.
- </para>
- <para>
- dev->pio_mode and dev->dma_mode are guaranteed to be valid when
- ->set_piomode() and when ->set_dmamode() is called. The timings for
- any other drive sharing the cable will also be valid at this point.
- That is the library records the decisions for the modes of each
- drive on a channel before it attempts to set any of them.
- </para>
- <para>
- ->post_set_mode() is
- called unconditionally, after the SET FEATURES - XFER MODE
- command completes successfully.
- </para>
-
- <para>
- ->set_piomode() is always called (if present), but
- ->set_dma_mode() is only called if DMA is possible.
- </para>
-
- </sect2>
-
- <sect2><title>Taskfile read/write</title>
- <programlisting>
-void (*sff_tf_load) (struct ata_port *ap, struct ata_taskfile *tf);
-void (*sff_tf_read) (struct ata_port *ap, struct ata_taskfile *tf);
- </programlisting>
-
- <para>
- ->tf_load() is called to load the given taskfile into hardware
- registers / DMA buffers. ->tf_read() is called to read the
- hardware registers / DMA buffers, to obtain the current set of
- taskfile register values.
- Most drivers for taskfile-based hardware (PIO or MMIO) use
- ata_sff_tf_load() and ata_sff_tf_read() for these hooks.
- </para>
-
- </sect2>
-
- <sect2><title>PIO data read/write</title>
- <programlisting>
-void (*sff_data_xfer) (struct ata_device *, unsigned char *, unsigned int, int);
- </programlisting>
-
- <para>
-All bmdma-style drivers must implement this hook. This is the low-level
-operation that actually copies the data bytes during a PIO data
-transfer.
-Typically the driver will choose one of ata_sff_data_xfer_noirq(),
-ata_sff_data_xfer(), or ata_sff_data_xfer32().
- </para>
-
- </sect2>
-
- <sect2><title>ATA command execute</title>
- <programlisting>
-void (*sff_exec_command)(struct ata_port *ap, struct ata_taskfile *tf);
- </programlisting>
-
- <para>
- causes an ATA command, previously loaded with
- ->tf_load(), to be initiated in hardware.
- Most drivers for taskfile-based hardware use ata_sff_exec_command()
- for this hook.
- </para>
-
- </sect2>
-
- <sect2><title>Per-cmd ATAPI DMA capabilities filter</title>
- <programlisting>
-int (*check_atapi_dma) (struct ata_queued_cmd *qc);
- </programlisting>
-
- <para>
-Allow low-level driver to filter ATA PACKET commands, returning a status
-indicating whether or not it is OK to use DMA for the supplied PACKET
-command.
- </para>
- <para>
- This hook may be specified as NULL, in which case libata will
- assume that atapi dma can be supported.
- </para>
-
- </sect2>
-
- <sect2><title>Read specific ATA shadow registers</title>
- <programlisting>
-u8 (*sff_check_status)(struct ata_port *ap);
-u8 (*sff_check_altstatus)(struct ata_port *ap);
- </programlisting>
-
- <para>
- Reads the Status/AltStatus ATA shadow register from
- hardware. On some hardware, reading the Status register has
- the side effect of clearing the interrupt condition.
- Most drivers for taskfile-based hardware use
- ata_sff_check_status() for this hook.
- </para>
-
- </sect2>
-
- <sect2><title>Write specific ATA shadow register</title>
- <programlisting>
-void (*sff_set_devctl)(struct ata_port *ap, u8 ctl);
- </programlisting>
-
- <para>
- Write the device control ATA shadow register to the hardware.
- Most drivers don't need to define this.
- </para>
-
- </sect2>
-
- <sect2><title>Select ATA device on bus</title>
- <programlisting>
-void (*sff_dev_select)(struct ata_port *ap, unsigned int device);
- </programlisting>
-
- <para>
- Issues the low-level hardware command(s) that causes one of N
- hardware devices to be considered 'selected' (active and
- available for use) on the ATA bus. This generally has no
- meaning on FIS-based devices.
- </para>
- <para>
- Most drivers for taskfile-based hardware use
- ata_sff_dev_select() for this hook.
- </para>
-
- </sect2>
-
- <sect2><title>Private tuning method</title>
- <programlisting>
-void (*set_mode) (struct ata_port *ap);
- </programlisting>
-
- <para>
- By default libata performs drive and controller tuning in
- accordance with the ATA timing rules and also applies blacklists
- and cable limits. Some controllers need special handling and have
- custom tuning rules, typically raid controllers that use ATA
- commands but do not actually do drive timing.
- </para>
-
- <warning>
- <para>
- This hook should not be used to replace the standard controller
- tuning logic when a controller has quirks. Replacing the default
- tuning logic in that case would bypass handling for drive and
- bridge quirks that may be important to data reliability. If a
- controller needs to filter the mode selection it should use the
- mode_filter hook instead.
- </para>
- </warning>
-
- </sect2>
-
- <sect2><title>Control PCI IDE BMDMA engine</title>
- <programlisting>
-void (*bmdma_setup) (struct ata_queued_cmd *qc);
-void (*bmdma_start) (struct ata_queued_cmd *qc);
-void (*bmdma_stop) (struct ata_port *ap);
-u8 (*bmdma_status) (struct ata_port *ap);
- </programlisting>
-
- <para>
-When setting up an IDE BMDMA transaction, these hooks arm
-(->bmdma_setup), fire (->bmdma_start), and halt (->bmdma_stop)
-the hardware's DMA engine. ->bmdma_status is used to read the standard
-PCI IDE DMA Status register.
- </para>
-
- <para>
-These hooks are typically either no-ops, or simply not implemented, in
-FIS-based drivers.
- </para>
- <para>
-Most legacy IDE drivers use ata_bmdma_setup() for the bmdma_setup()
-hook. ata_bmdma_setup() will write the pointer to the PRD table to
-the IDE PRD Table Address register, enable DMA in the DMA Command
-register, and call exec_command() to begin the transfer.
- </para>
- <para>
-Most legacy IDE drivers use ata_bmdma_start() for the bmdma_start()
-hook. ata_bmdma_start() will write the ATA_DMA_START flag to the DMA
-Command register.
- </para>
- <para>
-Many legacy IDE drivers use ata_bmdma_stop() for the bmdma_stop()
-hook. ata_bmdma_stop() clears the ATA_DMA_START flag in the DMA
-command register.
- </para>
- <para>
-Many legacy IDE drivers use ata_bmdma_status() as the bmdma_status() hook.
- </para>
-
- </sect2>
-
- <sect2><title>High-level taskfile hooks</title>
- <programlisting>
-void (*qc_prep) (struct ata_queued_cmd *qc);
-int (*qc_issue) (struct ata_queued_cmd *qc);
- </programlisting>
-
- <para>
- Higher-level hooks, these two hooks can potentially supercede
- several of the above taskfile/DMA engine hooks. ->qc_prep is
- called after the buffers have been DMA-mapped, and is typically
- used to populate the hardware's DMA scatter-gather table.
- Most drivers use the standard ata_qc_prep() helper function, but
- more advanced drivers roll their own.
- </para>
- <para>
- ->qc_issue is used to make a command active, once the hardware
- and S/G tables have been prepared. IDE BMDMA drivers use the
- helper function ata_qc_issue_prot() for taskfile protocol-based
- dispatch. More advanced drivers implement their own ->qc_issue.
- </para>
- <para>
- ata_qc_issue_prot() calls ->tf_load(), ->bmdma_setup(), and
- ->bmdma_start() as necessary to initiate a transfer.
- </para>
-
- </sect2>
-
- <sect2><title>Exception and probe handling (EH)</title>
- <programlisting>
-void (*eng_timeout) (struct ata_port *ap);
-void (*phy_reset) (struct ata_port *ap);
- </programlisting>
-
- <para>
-Deprecated. Use ->error_handler() instead.
- </para>
-
- <programlisting>
-void (*freeze) (struct ata_port *ap);
-void (*thaw) (struct ata_port *ap);
- </programlisting>
-
- <para>
-ata_port_freeze() is called when HSM violations or some other
-condition disrupts normal operation of the port. A frozen port
-is not allowed to perform any operation until the port is
-thawed, which usually follows a successful reset.
- </para>
-
- <para>
-The optional ->freeze() callback can be used for freezing the port
-hardware-wise (e.g. mask interrupt and stop DMA engine). If a
-port cannot be frozen hardware-wise, the interrupt handler
-must ack and clear interrupts unconditionally while the port
-is frozen.
- </para>
- <para>
-The optional ->thaw() callback is called to perform the opposite of ->freeze():
-prepare the port for normal operation once again. Unmask interrupts,
-start DMA engine, etc.
- </para>
-
- <programlisting>
-void (*error_handler) (struct ata_port *ap);
- </programlisting>
-
- <para>
-->error_handler() is a driver's hook into probe, hotplug, and recovery
-and other exceptional conditions. The primary responsibility of an
-implementation is to call ata_do_eh() or ata_bmdma_drive_eh() with a set
-of EH hooks as arguments:
- </para>
-
- <para>
-'prereset' hook (may be NULL) is called during an EH reset, before any other actions
-are taken.
- </para>
-
- <para>
-'postreset' hook (may be NULL) is called after the EH reset is performed. Based on
-existing conditions, severity of the problem, and hardware capabilities,
- </para>
-
- <para>
-Either 'softreset' (may be NULL) or 'hardreset' (may be NULL) will be
-called to perform the low-level EH reset.
- </para>
-
- <programlisting>
-void (*post_internal_cmd) (struct ata_queued_cmd *qc);
- </programlisting>
-
- <para>
-Perform any hardware-specific actions necessary to finish processing
-after executing a probe-time or EH-time command via ata_exec_internal().
- </para>
-
- </sect2>
-
- <sect2><title>Hardware interrupt handling</title>
- <programlisting>
-irqreturn_t (*irq_handler)(int, void *, struct pt_regs *);
-void (*irq_clear) (struct ata_port *);
- </programlisting>
-
- <para>
- ->irq_handler is the interrupt handling routine registered with
- the system, by libata. ->irq_clear is called during probe just
- before the interrupt handler is registered, to be sure hardware
- is quiet.
- </para>
- <para>
- The second argument, dev_instance, should be cast to a pointer
- to struct ata_host_set.
- </para>
- <para>
- Most legacy IDE drivers use ata_sff_interrupt() for the
- irq_handler hook, which scans all ports in the host_set,
- determines which queued command was active (if any), and calls
- ata_sff_host_intr(ap,qc).
- </para>
- <para>
- Most legacy IDE drivers use ata_sff_irq_clear() for the
- irq_clear() hook, which simply clears the interrupt and error
- flags in the DMA status register.
- </para>
-
- </sect2>
-
- <sect2><title>SATA phy read/write</title>
- <programlisting>
-int (*scr_read) (struct ata_port *ap, unsigned int sc_reg,
- u32 *val);
-int (*scr_write) (struct ata_port *ap, unsigned int sc_reg,
- u32 val);
- </programlisting>
-
- <para>
- Read and write standard SATA phy registers. Currently only used
- if ->phy_reset hook called the sata_phy_reset() helper function.
- sc_reg is one of SCR_STATUS, SCR_CONTROL, SCR_ERROR, or SCR_ACTIVE.
- </para>
-
- </sect2>
-
- <sect2><title>Init and shutdown</title>
- <programlisting>
-int (*port_start) (struct ata_port *ap);
-void (*port_stop) (struct ata_port *ap);
-void (*host_stop) (struct ata_host_set *host_set);
- </programlisting>
-
- <para>
- ->port_start() is called just after the data structures for each
- port are initialized. Typically this is used to alloc per-port
- DMA buffers / tables / rings, enable DMA engines, and similar
- tasks. Some drivers also use this entry point as a chance to
- allocate driver-private memory for ap->private_data.
- </para>
- <para>
- Many drivers use ata_port_start() as this hook or call
- it from their own port_start() hooks. ata_port_start()
- allocates space for a legacy IDE PRD table and returns.
- </para>
- <para>
- ->port_stop() is called after ->host_stop(). Its sole function
- is to release DMA/memory resources, now that they are no longer
- actively being used. Many drivers also free driver-private
- data from port at this time.
- </para>
- <para>
- ->host_stop() is called after all ->port_stop() calls
-have completed. The hook must finalize hardware shutdown, release DMA
-and other resources, etc.
- This hook may be specified as NULL, in which case it is not called.
- </para>
-
- </sect2>
-
- </sect1>
- </chapter>
-
- <chapter id="libataEH">
- <title>Error handling</title>
-
- <para>
- This chapter describes how errors are handled under libata.
- Readers are advised to read SCSI EH
- (Documentation/scsi/scsi_eh.txt) and ATA exceptions doc first.
- </para>
-
- <sect1><title>Origins of commands</title>
- <para>
- In libata, a command is represented with struct ata_queued_cmd
- or qc. qc's are preallocated during port initialization and
- repetitively used for command executions. Currently only one
- qc is allocated per port but yet-to-be-merged NCQ branch
- allocates one for each tag and maps each qc to NCQ tag 1-to-1.
- </para>
- <para>
- libata commands can originate from two sources - libata itself
- and SCSI midlayer. libata internal commands are used for
- initialization and error handling. All normal blk requests
- and commands for SCSI emulation are passed as SCSI commands
- through queuecommand callback of SCSI host template.
- </para>
- </sect1>
-
- <sect1><title>How commands are issued</title>
-
- <variablelist>
-
- <varlistentry><term>Internal commands</term>
- <listitem>
- <para>
- First, qc is allocated and initialized using
- ata_qc_new_init(). Although ata_qc_new_init() doesn't
- implement any wait or retry mechanism when qc is not
- available, internal commands are currently issued only during
- initialization and error recovery, so no other command is
- active and allocation is guaranteed to succeed.
- </para>
- <para>
- Once allocated qc's taskfile is initialized for the command to
- be executed. qc currently has two mechanisms to notify
- completion. One is via qc->complete_fn() callback and the
- other is completion qc->waiting. qc->complete_fn() callback
- is the asynchronous path used by normal SCSI translated
- commands and qc->waiting is the synchronous (issuer sleeps in
- process context) path used by internal commands.
- </para>
- <para>
- Once initialization is complete, host_set lock is acquired
- and the qc is issued.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry><term>SCSI commands</term>
- <listitem>
- <para>
- All libata drivers use ata_scsi_queuecmd() as
- hostt->queuecommand callback. scmds can either be simulated
- or translated. No qc is involved in processing a simulated
- scmd. The result is computed right away and the scmd is
- completed.
- </para>
- <para>
- For a translated scmd, ata_qc_new_init() is invoked to
- allocate a qc and the scmd is translated into the qc. SCSI
- midlayer's completion notification function pointer is stored
- into qc->scsidone.
- </para>
- <para>
- qc->complete_fn() callback is used for completion
- notification. ATA commands use ata_scsi_qc_complete() while
- ATAPI commands use atapi_qc_complete(). Both functions end up
- calling qc->scsidone to notify upper layer when the qc is
- finished. After translation is completed, the qc is issued
- with ata_qc_issue().
- </para>
- <para>
- Note that SCSI midlayer invokes hostt->queuecommand while
- holding host_set lock, so all above occur while holding
- host_set lock.
- </para>
- </listitem>
- </varlistentry>
-
- </variablelist>
- </sect1>
-
- <sect1><title>How commands are processed</title>
- <para>
- Depending on which protocol and which controller are used,
- commands are processed differently. For the purpose of
- discussion, a controller which uses taskfile interface and all
- standard callbacks is assumed.
- </para>
- <para>
- Currently 6 ATA command protocols are used. They can be
- sorted into the following four categories according to how
- they are processed.
- </para>
-
- <variablelist>
- <varlistentry><term>ATA NO DATA or DMA</term>
- <listitem>
- <para>
- ATA_PROT_NODATA and ATA_PROT_DMA fall into this category.
- These types of commands don't require any software
- intervention once issued. Device will raise interrupt on
- completion.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry><term>ATA PIO</term>
- <listitem>
- <para>
- ATA_PROT_PIO is in this category. libata currently
- implements PIO with polling. ATA_NIEN bit is set to turn
- off interrupt and pio_task on ata_wq performs polling and
- IO.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry><term>ATAPI NODATA or DMA</term>
- <listitem>
- <para>
- ATA_PROT_ATAPI_NODATA and ATA_PROT_ATAPI_DMA are in this
- category. packet_task is used to poll BSY bit after
- issuing PACKET command. Once BSY is turned off by the
- device, packet_task transfers CDB and hands off processing
- to interrupt handler.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry><term>ATAPI PIO</term>
- <listitem>
- <para>
- ATA_PROT_ATAPI is in this category. ATA_NIEN bit is set
- and, as in ATAPI NODATA or DMA, packet_task submits cdb.
- However, after submitting cdb, further processing (data
- transfer) is handed off to pio_task.
- </para>
- </listitem>
- </varlistentry>
- </variablelist>
- </sect1>
-
- <sect1><title>How commands are completed</title>
- <para>
- Once issued, all qc's are either completed with
- ata_qc_complete() or time out. For commands which are handled
- by interrupts, ata_host_intr() invokes ata_qc_complete(), and,
- for PIO tasks, pio_task invokes ata_qc_complete(). In error
- cases, packet_task may also complete commands.
- </para>
- <para>
- ata_qc_complete() does the following.
- </para>
-
- <orderedlist>
-
- <listitem>
- <para>
- DMA memory is unmapped.
- </para>
- </listitem>
-
- <listitem>
- <para>
- ATA_QCFLAG_ACTIVE is cleared from qc->flags.
- </para>
- </listitem>
-
- <listitem>
- <para>
- qc->complete_fn() callback is invoked. If the return value of
- the callback is not zero. Completion is short circuited and
- ata_qc_complete() returns.
- </para>
- </listitem>
-
- <listitem>
- <para>
- __ata_qc_complete() is called, which does
- <orderedlist>
-
- <listitem>
- <para>
- qc->flags is cleared to zero.
- </para>
- </listitem>
-
- <listitem>
- <para>
- ap->active_tag and qc->tag are poisoned.
- </para>
- </listitem>
-
- <listitem>
- <para>
- qc->waiting is cleared & completed (in that order).
- </para>
- </listitem>
-
- <listitem>
- <para>
- qc is deallocated by clearing appropriate bit in ap->qactive.
- </para>
- </listitem>
-
- </orderedlist>
- </para>
- </listitem>
-
- </orderedlist>
-
- <para>
- So, it basically notifies upper layer and deallocates qc. One
- exception is short-circuit path in #3 which is used by
- atapi_qc_complete().
- </para>
- <para>
- For all non-ATAPI commands, whether it fails or not, almost
- the same code path is taken and very little error handling
- takes place. A qc is completed with success status if it
- succeeded, with failed status otherwise.
- </para>
- <para>
- However, failed ATAPI commands require more handling as
- REQUEST SENSE is needed to acquire sense data. If an ATAPI
- command fails, ata_qc_complete() is invoked with error status,
- which in turn invokes atapi_qc_complete() via
- qc->complete_fn() callback.
- </para>
- <para>
- This makes atapi_qc_complete() set scmd->result to
- SAM_STAT_CHECK_CONDITION, complete the scmd and return 1. As
- the sense data is empty but scmd->result is CHECK CONDITION,
- SCSI midlayer will invoke EH for the scmd, and returning 1
- makes ata_qc_complete() to return without deallocating the qc.
- This leads us to ata_scsi_error() with partially completed qc.
- </para>
-
- </sect1>
-
- <sect1><title>ata_scsi_error()</title>
- <para>
- ata_scsi_error() is the current transportt->eh_strategy_handler()
- for libata. As discussed above, this will be entered in two
- cases - timeout and ATAPI error completion. This function
- calls low level libata driver's eng_timeout() callback, the
- standard callback for which is ata_eng_timeout(). It checks
- if a qc is active and calls ata_qc_timeout() on the qc if so.
- Actual error handling occurs in ata_qc_timeout().
- </para>
- <para>
- If EH is invoked for timeout, ata_qc_timeout() stops BMDMA and
- completes the qc. Note that as we're currently in EH, we
- cannot call scsi_done. As described in SCSI EH doc, a
- recovered scmd should be either retried with
- scsi_queue_insert() or finished with scsi_finish_command().
- Here, we override qc->scsidone with scsi_finish_command() and
- calls ata_qc_complete().
- </para>
- <para>
- If EH is invoked due to a failed ATAPI qc, the qc here is
- completed but not deallocated. The purpose of this
- half-completion is to use the qc as place holder to make EH
- code reach this place. This is a bit hackish, but it works.
- </para>
- <para>
- Once control reaches here, the qc is deallocated by invoking
- __ata_qc_complete() explicitly. Then, internal qc for REQUEST
- SENSE is issued. Once sense data is acquired, scmd is
- finished by directly invoking scsi_finish_command() on the
- scmd. Note that as we already have completed and deallocated
- the qc which was associated with the scmd, we don't need
- to/cannot call ata_qc_complete() again.
- </para>
-
- </sect1>
-
- <sect1><title>Problems with the current EH</title>
-
- <itemizedlist>
-
- <listitem>
- <para>
- Error representation is too crude. Currently any and all
- error conditions are represented with ATA STATUS and ERROR
- registers. Errors which aren't ATA device errors are treated
- as ATA device errors by setting ATA_ERR bit. Better error
- descriptor which can properly represent ATA and other
- errors/exceptions is needed.
- </para>
- </listitem>
-
- <listitem>
- <para>
- When handling timeouts, no action is taken to make device
- forget about the timed out command and ready for new commands.
- </para>
- </listitem>
-
- <listitem>
- <para>
- EH handling via ata_scsi_error() is not properly protected
- from usual command processing. On EH entrance, the device is
- not in quiescent state. Timed out commands may succeed or
- fail any time. pio_task and atapi_task may still be running.
- </para>
- </listitem>
-
- <listitem>
- <para>
- Too weak error recovery. Devices / controllers causing HSM
- mismatch errors and other errors quite often require reset to
- return to known state. Also, advanced error handling is
- necessary to support features like NCQ and hotplug.
- </para>
- </listitem>
-
- <listitem>
- <para>
- ATA errors are directly handled in the interrupt handler and
- PIO errors in pio_task. This is problematic for advanced
- error handling for the following reasons.
- </para>
- <para>
- First, advanced error handling often requires context and
- internal qc execution.
- </para>
- <para>
- Second, even a simple failure (say, CRC error) needs
- information gathering and could trigger complex error handling
- (say, resetting & reconfiguring). Having multiple code
- paths to gather information, enter EH and trigger actions
- makes life painful.
- </para>
- <para>
- Third, scattered EH code makes implementing low level drivers
- difficult. Low level drivers override libata callbacks. If
- EH is scattered over several places, each affected callbacks
- should perform its part of error handling. This can be error
- prone and painful.
- </para>
- </listitem>
-
- </itemizedlist>
- </sect1>
- </chapter>
-
- <chapter id="libataExt">
- <title>libata Library</title>
-!Edrivers/ata/libata-core.c
- </chapter>
-
- <chapter id="libataInt">
- <title>libata Core Internals</title>
-!Idrivers/ata/libata-core.c
- </chapter>
-
- <chapter id="libataScsiInt">
- <title>libata SCSI translation/emulation</title>
-!Edrivers/ata/libata-scsi.c
-!Idrivers/ata/libata-scsi.c
- </chapter>
-
- <chapter id="ataExceptions">
- <title>ATA errors and exceptions</title>
-
- <para>
- This chapter tries to identify what error/exception conditions exist
- for ATA/ATAPI devices and describe how they should be handled in
- implementation-neutral way.
- </para>
-
- <para>
- The term 'error' is used to describe conditions where either an
- explicit error condition is reported from device or a command has
- timed out.
- </para>
-
- <para>
- The term 'exception' is either used to describe exceptional
- conditions which are not errors (say, power or hotplug events), or
- to describe both errors and non-error exceptional conditions. Where
- explicit distinction between error and exception is necessary, the
- term 'non-error exception' is used.
- </para>
-
- <sect1 id="excat">
- <title>Exception categories</title>
- <para>
- Exceptions are described primarily with respect to legacy
- taskfile + bus master IDE interface. If a controller provides
- other better mechanism for error reporting, mapping those into
- categories described below shouldn't be difficult.
- </para>
-
- <para>
- In the following sections, two recovery actions - reset and
- reconfiguring transport - are mentioned. These are described
- further in <xref linkend="exrec"/>.
- </para>
-
- <sect2 id="excatHSMviolation">
- <title>HSM violation</title>
- <para>
- This error is indicated when STATUS value doesn't match HSM
- requirement during issuing or execution any ATA/ATAPI command.
- </para>
-
- <itemizedlist>
- <title>Examples</title>
-
- <listitem>
- <para>
- ATA_STATUS doesn't contain !BSY && DRDY && !DRQ while trying
- to issue a command.
- </para>
- </listitem>
-
- <listitem>
- <para>
- !BSY && !DRQ during PIO data transfer.
- </para>
- </listitem>
-
- <listitem>
- <para>
- DRQ on command completion.
- </para>
- </listitem>
-
- <listitem>
- <para>
- !BSY && ERR after CDB transfer starts but before the
- last byte of CDB is transferred. ATA/ATAPI standard states
- that "The device shall not terminate the PACKET command
- with an error before the last byte of the command packet has
- been written" in the error outputs description of PACKET
- command and the state diagram doesn't include such
- transitions.
- </para>
- </listitem>
-
- </itemizedlist>
-
- <para>
- In these cases, HSM is violated and not much information
- regarding the error can be acquired from STATUS or ERROR
- register. IOW, this error can be anything - driver bug,
- faulty device, controller and/or cable.
- </para>
-
- <para>
- As HSM is violated, reset is necessary to restore known state.
- Reconfiguring transport for lower speed might be helpful too
- as transmission errors sometimes cause this kind of errors.
- </para>
- </sect2>
-
- <sect2 id="excatDevErr">
- <title>ATA/ATAPI device error (non-NCQ / non-CHECK CONDITION)</title>
-
- <para>
- These are errors detected and reported by ATA/ATAPI devices
- indicating device problems. For this type of errors, STATUS
- and ERROR register values are valid and describe error
- condition. Note that some of ATA bus errors are detected by
- ATA/ATAPI devices and reported using the same mechanism as
- device errors. Those cases are described later in this
- section.
- </para>
-
- <para>
- For ATA commands, this type of errors are indicated by !BSY
- && ERR during command execution and on completion.
- </para>
-
- <para>For ATAPI commands,</para>
-
- <itemizedlist>
-
- <listitem>
- <para>
- !BSY && ERR && ABRT right after issuing PACKET
- indicates that PACKET command is not supported and falls in
- this category.
- </para>
- </listitem>
-
- <listitem>
- <para>
- !BSY && ERR(==CHK) && !ABRT after the last
- byte of CDB is transferred indicates CHECK CONDITION and
- doesn't fall in this category.
- </para>
- </listitem>
-
- <listitem>
- <para>
- !BSY && ERR(==CHK) && ABRT after the last byte
- of CDB is transferred *probably* indicates CHECK CONDITION and
- doesn't fall in this category.
- </para>
- </listitem>
-
- </itemizedlist>
-
- <para>
- Of errors detected as above, the following are not ATA/ATAPI
- device errors but ATA bus errors and should be handled
- according to <xref linkend="excatATAbusErr"/>.
- </para>
-
- <variablelist>
-
- <varlistentry>
- <term>CRC error during data transfer</term>
- <listitem>
- <para>
- This is indicated by ICRC bit in the ERROR register and
- means that corruption occurred during data transfer. Up to
- ATA/ATAPI-7, the standard specifies that this bit is only
- applicable to UDMA transfers but ATA/ATAPI-8 draft revision
- 1f says that the bit may be applicable to multiword DMA and
- PIO.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term>ABRT error during data transfer or on completion</term>
- <listitem>
- <para>
- Up to ATA/ATAPI-7, the standard specifies that ABRT could be
- set on ICRC errors and on cases where a device is not able
- to complete a command. Combined with the fact that MWDMA
- and PIO transfer errors aren't allowed to use ICRC bit up to
- ATA/ATAPI-7, it seems to imply that ABRT bit alone could
- indicate transfer errors.
- </para>
- <para>
- However, ATA/ATAPI-8 draft revision 1f removes the part
- that ICRC errors can turn on ABRT. So, this is kind of
- gray area. Some heuristics are needed here.
- </para>
- </listitem>
- </varlistentry>
-
- </variablelist>
-
- <para>
- ATA/ATAPI device errors can be further categorized as follows.
- </para>
-
- <variablelist>
-
- <varlistentry>
- <term>Media errors</term>
- <listitem>
- <para>
- This is indicated by UNC bit in the ERROR register. ATA
- devices reports UNC error only after certain number of
- retries cannot recover the data, so there's nothing much
- else to do other than notifying upper layer.
- </para>
- <para>
- READ and WRITE commands report CHS or LBA of the first
- failed sector but ATA/ATAPI standard specifies that the
- amount of transferred data on error completion is
- indeterminate, so we cannot assume that sectors preceding
- the failed sector have been transferred and thus cannot
- complete those sectors successfully as SCSI does.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term>Media changed / media change requested error</term>
- <listitem>
- <para>
- <<TODO: fill here>>
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry><term>Address error</term>
- <listitem>
- <para>
- This is indicated by IDNF bit in the ERROR register.
- Report to upper layer.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry><term>Other errors</term>
- <listitem>
- <para>
- This can be invalid command or parameter indicated by ABRT
- ERROR bit or some other error condition. Note that ABRT
- bit can indicate a lot of things including ICRC and Address
- errors. Heuristics needed.
- </para>
- </listitem>
- </varlistentry>
-
- </variablelist>
-
- <para>
- Depending on commands, not all STATUS/ERROR bits are
- applicable. These non-applicable bits are marked with
- "na" in the output descriptions but up to ATA/ATAPI-7
- no definition of "na" can be found. However,
- ATA/ATAPI-8 draft revision 1f describes "N/A" as
- follows.
- </para>
-
- <blockquote>
- <variablelist>
- <varlistentry><term>3.2.3.3a N/A</term>
- <listitem>
- <para>
- A keyword the indicates a field has no defined value in
- this standard and should not be checked by the host or
- device. N/A fields should be cleared to zero.
- </para>
- </listitem>
- </varlistentry>
- </variablelist>
- </blockquote>
-
- <para>
- So, it seems reasonable to assume that "na" bits are
- cleared to zero by devices and thus need no explicit masking.
- </para>
-
- </sect2>
-
- <sect2 id="excatATAPIcc">
- <title>ATAPI device CHECK CONDITION</title>
-
- <para>
- ATAPI device CHECK CONDITION error is indicated by set CHK bit
- (ERR bit) in the STATUS register after the last byte of CDB is
- transferred for a PACKET command. For this kind of errors,
- sense data should be acquired to gather information regarding
- the errors. REQUEST SENSE packet command should be used to
- acquire sense data.
- </para>
-
- <para>
- Once sense data is acquired, this type of errors can be
- handled similarly to other SCSI errors. Note that sense data
- may indicate ATA bus error (e.g. Sense Key 04h HARDWARE ERROR
- && ASC/ASCQ 47h/00h SCSI PARITY ERROR). In such
- cases, the error should be considered as an ATA bus error and
- handled according to <xref linkend="excatATAbusErr"/>.
- </para>
-
- </sect2>
-
- <sect2 id="excatNCQerr">
- <title>ATA device error (NCQ)</title>
-
- <para>
- NCQ command error is indicated by cleared BSY and set ERR bit
- during NCQ command phase (one or more NCQ commands
- outstanding). Although STATUS and ERROR registers will
- contain valid values describing the error, READ LOG EXT is
- required to clear the error condition, determine which command
- has failed and acquire more information.
- </para>
-
- <para>
- READ LOG EXT Log Page 10h reports which tag has failed and
- taskfile register values describing the error. With this
- information the failed command can be handled as a normal ATA
- command error as in <xref linkend="excatDevErr"/> and all
- other in-flight commands must be retried. Note that this
- retry should not be counted - it's likely that commands
- retried this way would have completed normally if it were not
- for the failed command.
- </para>
-
- <para>
- Note that ATA bus errors can be reported as ATA device NCQ
- errors. This should be handled as described in <xref
- linkend="excatATAbusErr"/>.
- </para>
-
- <para>
- If READ LOG EXT Log Page 10h fails or reports NQ, we're
- thoroughly screwed. This condition should be treated
- according to <xref linkend="excatHSMviolation"/>.
- </para>
-
- </sect2>
-
- <sect2 id="excatATAbusErr">
- <title>ATA bus error</title>
-
- <para>
- ATA bus error means that data corruption occurred during
- transmission over ATA bus (SATA or PATA). This type of errors
- can be indicated by
- </para>
-
- <itemizedlist>
-
- <listitem>
- <para>
- ICRC or ABRT error as described in <xref linkend="excatDevErr"/>.
- </para>
- </listitem>
-
- <listitem>
- <para>
- Controller-specific error completion with error information
- indicating transmission error.
- </para>
- </listitem>
-
- <listitem>
- <para>
- On some controllers, command timeout. In this case, there may
- be a mechanism to determine that the timeout is due to
- transmission error.
- </para>
- </listitem>
-
- <listitem>
- <para>
- Unknown/random errors, timeouts and all sorts of weirdities.
- </para>
- </listitem>
-
- </itemizedlist>
-
- <para>
- As described above, transmission errors can cause wide variety
- of symptoms ranging from device ICRC error to random device
- lockup, and, for many cases, there is no way to tell if an
- error condition is due to transmission error or not;
- therefore, it's necessary to employ some kind of heuristic
- when dealing with errors and timeouts. For example,
- encountering repetitive ABRT errors for known supported
- command is likely to indicate ATA bus error.
- </para>
-
- <para>
- Once it's determined that ATA bus errors have possibly
- occurred, lowering ATA bus transmission speed is one of
- actions which may alleviate the problem. See <xref
- linkend="exrecReconf"/> for more information.
- </para>
-
- </sect2>
-
- <sect2 id="excatPCIbusErr">
- <title>PCI bus error</title>
-
- <para>
- Data corruption or other failures during transmission over PCI
- (or other system bus). For standard BMDMA, this is indicated
- by Error bit in the BMDMA Status register. This type of
- errors must be logged as it indicates something is very wrong
- with the system. Resetting host controller is recommended.
- </para>
-
- </sect2>
-
- <sect2 id="excatLateCompletion">
- <title>Late completion</title>
-
- <para>
- This occurs when timeout occurs and the timeout handler finds
- out that the timed out command has completed successfully or
- with error. This is usually caused by lost interrupts. This
- type of errors must be logged. Resetting host controller is
- recommended.
- </para>
-
- </sect2>
-
- <sect2 id="excatUnknown">
- <title>Unknown error (timeout)</title>
-
- <para>
- This is when timeout occurs and the command is still
- processing or the host and device are in unknown state. When
- this occurs, HSM could be in any valid or invalid state. To
- bring the device to known state and make it forget about the
- timed out command, resetting is necessary. The timed out
- command may be retried.
- </para>
-
- <para>
- Timeouts can also be caused by transmission errors. Refer to
- <xref linkend="excatATAbusErr"/> for more details.
- </para>
-
- </sect2>
-
- <sect2 id="excatHoplugPM">
- <title>Hotplug and power management exceptions</title>
-
- <para>
- <<TODO: fill here>>
- </para>
-
- </sect2>
-
- </sect1>
-
- <sect1 id="exrec">
- <title>EH recovery actions</title>
-
- <para>
- This section discusses several important recovery actions.
- </para>
-
- <sect2 id="exrecClr">
- <title>Clearing error condition</title>
-
- <para>
- Many controllers require its error registers to be cleared by
- error handler. Different controllers may have different
- requirements.
- </para>
-
- <para>
- For SATA, it's strongly recommended to clear at least SError
- register during error handling.
- </para>
- </sect2>
-
- <sect2 id="exrecRst">
- <title>Reset</title>
-
- <para>
- During EH, resetting is necessary in the following cases.
- </para>
-
- <itemizedlist>
-
- <listitem>
- <para>
- HSM is in unknown or invalid state
- </para>
- </listitem>
-
- <listitem>
- <para>
- HBA is in unknown or invalid state
- </para>
- </listitem>
-
- <listitem>
- <para>
- EH needs to make HBA/device forget about in-flight commands
- </para>
- </listitem>
-
- <listitem>
- <para>
- HBA/device behaves weirdly
- </para>
- </listitem>
-
- </itemizedlist>
-
- <para>
- Resetting during EH might be a good idea regardless of error
- condition to improve EH robustness. Whether to reset both or
- either one of HBA and device depends on situation but the
- following scheme is recommended.
- </para>
-
- <itemizedlist>
-
- <listitem>
- <para>
- When it's known that HBA is in ready state but ATA/ATAPI
- device is in unknown state, reset only device.
- </para>
- </listitem>
-
- <listitem>
- <para>
- If HBA is in unknown state, reset both HBA and device.
- </para>
- </listitem>
-
- </itemizedlist>
-
- <para>
- HBA resetting is implementation specific. For a controller
- complying to taskfile/BMDMA PCI IDE, stopping active DMA
- transaction may be sufficient iff BMDMA state is the only HBA
- context. But even mostly taskfile/BMDMA PCI IDE complying
- controllers may have implementation specific requirements and
- mechanism to reset themselves. This must be addressed by
- specific drivers.
- </para>
-
- <para>
- OTOH, ATA/ATAPI standard describes in detail ways to reset
- ATA/ATAPI devices.
- </para>
-
- <variablelist>
-
- <varlistentry><term>PATA hardware reset</term>
- <listitem>
- <para>
- This is hardware initiated device reset signalled with
- asserted PATA RESET- signal. There is no standard way to
- initiate hardware reset from software although some
- hardware provides registers that allow driver to directly
- tweak the RESET- signal.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry><term>Software reset</term>
- <listitem>
- <para>
- This is achieved by turning CONTROL SRST bit on for at
- least 5us. Both PATA and SATA support it but, in case of
- SATA, this may require controller-specific support as the
- second Register FIS to clear SRST should be transmitted
- while BSY bit is still set. Note that on PATA, this resets
- both master and slave devices on a channel.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry><term>EXECUTE DEVICE DIAGNOSTIC command</term>
- <listitem>
- <para>
- Although ATA/ATAPI standard doesn't describe exactly, EDD
- implies some level of resetting, possibly similar level
- with software reset. Host-side EDD protocol can be handled
- with normal command processing and most SATA controllers
- should be able to handle EDD's just like other commands.
- As in software reset, EDD affects both devices on a PATA
- bus.
- </para>
- <para>
- Although EDD does reset devices, this doesn't suit error
- handling as EDD cannot be issued while BSY is set and it's
- unclear how it will act when device is in unknown/weird
- state.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry><term>ATAPI DEVICE RESET command</term>
- <listitem>
- <para>
- This is very similar to software reset except that reset
- can be restricted to the selected device without affecting
- the other device sharing the cable.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry><term>SATA phy reset</term>
- <listitem>
- <para>
- This is the preferred way of resetting a SATA device. In
- effect, it's identical to PATA hardware reset. Note that
- this can be done with the standard SCR Control register.
- As such, it's usually easier to implement than software
- reset.
- </para>
- </listitem>
- </varlistentry>
-
- </variablelist>
-
- <para>
- One more thing to consider when resetting devices is that
- resetting clears certain configuration parameters and they
- need to be set to their previous or newly adjusted values
- after reset.
- </para>
-
- <para>
- Parameters affected are.
- </para>
-
- <itemizedlist>
-
- <listitem>
- <para>
- CHS set up with INITIALIZE DEVICE PARAMETERS (seldom used)
- </para>
- </listitem>
-
- <listitem>
- <para>
- Parameters set with SET FEATURES including transfer mode setting
- </para>
- </listitem>
-
- <listitem>
- <para>
- Block count set with SET MULTIPLE MODE
- </para>
- </listitem>
-
- <listitem>
- <para>
- Other parameters (SET MAX, MEDIA LOCK...)
- </para>
- </listitem>
-
- </itemizedlist>
-
- <para>
- ATA/ATAPI standard specifies that some parameters must be
- maintained across hardware or software reset, but doesn't
- strictly specify all of them. Always reconfiguring needed
- parameters after reset is required for robustness. Note that
- this also applies when resuming from deep sleep (power-off).
- </para>
-
- <para>
- Also, ATA/ATAPI standard requires that IDENTIFY DEVICE /
- IDENTIFY PACKET DEVICE is issued after any configuration
- parameter is updated or a hardware reset and the result used
- for further operation. OS driver is required to implement
- revalidation mechanism to support this.
- </para>
-
- </sect2>
-
- <sect2 id="exrecReconf">
- <title>Reconfigure transport</title>
-
- <para>
- For both PATA and SATA, a lot of corners are cut for cheap
- connectors, cables or controllers and it's quite common to see
- high transmission error rate. This can be mitigated by
- lowering transmission speed.
- </para>
-
- <para>
- The following is a possible scheme Jeff Garzik suggested.
- </para>
-
- <blockquote>
- <para>
- If more than $N (3?) transmission errors happen in 15 minutes,
- </para>
- <itemizedlist>
- <listitem>
- <para>
- if SATA, decrease SATA PHY speed. if speed cannot be decreased,
- </para>
- </listitem>
- <listitem>
- <para>
- decrease UDMA xfer speed. if at UDMA0, switch to PIO4,
- </para>
- </listitem>
- <listitem>
- <para>
- decrease PIO xfer speed. if at PIO3, complain, but continue
- </para>
- </listitem>
- </itemizedlist>
- </blockquote>
-
- </sect2>
-
- </sect1>
-
- </chapter>
-
- <chapter id="PiixInt">
- <title>ata_piix Internals</title>
-!Idrivers/ata/ata_piix.c
- </chapter>
-
- <chapter id="SILInt">
- <title>sata_sil Internals</title>
-!Idrivers/ata/sata_sil.c
- </chapter>
-
- <chapter id="libataThanks">
- <title>Thanks</title>
- <para>
- The bulk of the ATA knowledge comes thanks to long conversations with
- Andre Hedrick (www.linux-ide.org), and long hours pondering the ATA
- and SCSI specifications.
- </para>
- <para>
- Thanks to Alan Cox for pointing out similarities
- between SATA and SCSI, and in general for motivation to hack on
- libata.
- </para>
- <para>
- libata's device detection
- method, ata_pio_devchk, and in general all the early probing was
- based on extensive study of Hale Landis's probe/reset code in his
- ATADRVR driver (www.ata-atapi.com).
- </para>
- </chapter>
-
-</book>
+++ /dev/null
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
-
-<book id="Reed-Solomon-Library-Guide">
- <bookinfo>
- <title>Reed-Solomon Library Programming Interface</title>
-
- <authorgroup>
- <author>
- <firstname>Thomas</firstname>
- <surname>Gleixner</surname>
- <affiliation>
- <address>
- <email>tglx@linutronix.de</email>
- </address>
- </affiliation>
- </author>
- </authorgroup>
-
- <copyright>
- <year>2004</year>
- <holder>Thomas Gleixner</holder>
- </copyright>
-
- <legalnotice>
- <para>
- This documentation is free software; you can redistribute
- it and/or modify it under the terms of the GNU General Public
- License version 2 as published by the Free Software Foundation.
- </para>
-
- <para>
- This program is distributed in the hope that it will be
- useful, but WITHOUT ANY WARRANTY; without even the implied
- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- See the GNU General Public License for more details.
- </para>
-
- <para>
- You should have received a copy of the GNU General Public
- License along with this program; if not, write to the Free
- Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
- MA 02111-1307 USA
- </para>
-
- <para>
- For more details see the file COPYING in the source
- distribution of Linux.
- </para>
- </legalnotice>
- </bookinfo>
-
-<toc></toc>
-
- <chapter id="intro">
- <title>Introduction</title>
- <para>
- The generic Reed-Solomon Library provides encoding, decoding
- and error correction functions.
- </para>
- <para>
- Reed-Solomon codes are used in communication and storage
- applications to ensure data integrity.
- </para>
- <para>
- This documentation is provided for developers who want to utilize
- the functions provided by the library.
- </para>
- </chapter>
-
- <chapter id="bugs">
- <title>Known Bugs And Assumptions</title>
- <para>
- None.
- </para>
- </chapter>
-
- <chapter id="usage">
- <title>Usage</title>
- <para>
- This chapter provides examples of how to use the library.
- </para>
- <sect1>
- <title>Initializing</title>
- <para>
- The init function init_rs returns a pointer to an
- rs decoder structure, which holds the necessary
- information for encoding, decoding and error correction
- with the given polynomial. It either uses an existing
- matching decoder or creates a new one. On creation all
- the lookup tables for fast en/decoding are created.
- The function may take a while, so make sure not to
- call it in critical code paths.
- </para>
- <programlisting>
-/* the Reed Solomon control structure */
-static struct rs_control *rs_decoder;
-
-/* Symbolsize is 10 (bits)
- * Primitive polynomial is x^10+x^3+1
- * first consecutive root is 0
- * primitive element to generate roots = 1
- * generator polynomial degree (number of roots) = 6
- */
-rs_decoder = init_rs (10, 0x409, 0, 1, 6);
- </programlisting>
- </sect1>
- <sect1>
- <title>Encoding</title>
- <para>
- The encoder calculates the Reed-Solomon code over
- the given data length and stores the result in
- the parity buffer. Note that the parity buffer must
- be initialized before calling the encoder.
- </para>
- <para>
- The expanded data can be inverted on the fly by
- providing a non-zero inversion mask. The expanded data is
- XOR'ed with the mask. This is used e.g. for FLASH
- ECC, where the all 0xFF is inverted to an all 0x00.
- The Reed-Solomon code for all 0x00 is all 0x00. The
- code is inverted before storing to FLASH so it is 0xFF
- too. This prevents that reading from an erased FLASH
- results in ECC errors.
- </para>
- <para>
- The databytes are expanded to the given symbol size
- on the fly. There is no support for encoding continuous
- bitstreams with a symbol size != 8 at the moment. If
- it is necessary it should be not a big deal to implement
- such functionality.
- </para>
- <programlisting>
-/* Parity buffer. Size = number of roots */
-uint16_t par[6];
-/* Initialize the parity buffer */
-memset(par, 0, sizeof(par));
-/* Encode 512 byte in data8. Store parity in buffer par */
-encode_rs8 (rs_decoder, data8, 512, par, 0);
- </programlisting>
- </sect1>
- <sect1>
- <title>Decoding</title>
- <para>
- The decoder calculates the syndrome over
- the given data length and the received parity symbols
- and corrects errors in the data.
- </para>
- <para>
- If a syndrome is available from a hardware decoder
- then the syndrome calculation is skipped.
- </para>
- <para>
- The correction of the data buffer can be suppressed
- by providing a correction pattern buffer and an error
- location buffer to the decoder. The decoder stores the
- calculated error location and the correction bitmask
- in the given buffers. This is useful for hardware
- decoders which use a weird bit ordering scheme.
- </para>
- <para>
- The databytes are expanded to the given symbol size
- on the fly. There is no support for decoding continuous
- bitstreams with a symbolsize != 8 at the moment. If
- it is necessary it should be not a big deal to implement
- such functionality.
- </para>
-
- <sect2>
- <title>
- Decoding with syndrome calculation, direct data correction
- </title>
- <programlisting>
-/* Parity buffer. Size = number of roots */
-uint16_t par[6];
-uint8_t data[512];
-int numerr;
-/* Receive data */
-.....
-/* Receive parity */
-.....
-/* Decode 512 byte in data8.*/
-numerr = decode_rs8 (rs_decoder, data8, par, 512, NULL, 0, NULL, 0, NULL);
- </programlisting>
- </sect2>
-
- <sect2>
- <title>
- Decoding with syndrome given by hardware decoder, direct data correction
- </title>
- <programlisting>
-/* Parity buffer. Size = number of roots */
-uint16_t par[6], syn[6];
-uint8_t data[512];
-int numerr;
-/* Receive data */
-.....
-/* Receive parity */
-.....
-/* Get syndrome from hardware decoder */
-.....
-/* Decode 512 byte in data8.*/
-numerr = decode_rs8 (rs_decoder, data8, par, 512, syn, 0, NULL, 0, NULL);
- </programlisting>
- </sect2>
-
- <sect2>
- <title>
- Decoding with syndrome given by hardware decoder, no direct data correction.
- </title>
- <para>
- Note: It's not necessary to give data and received parity to the decoder.
- </para>
- <programlisting>
-/* Parity buffer. Size = number of roots */
-uint16_t par[6], syn[6], corr[8];
-uint8_t data[512];
-int numerr, errpos[8];
-/* Receive data */
-.....
-/* Receive parity */
-.....
-/* Get syndrome from hardware decoder */
-.....
-/* Decode 512 byte in data8.*/
-numerr = decode_rs8 (rs_decoder, NULL, NULL, 512, syn, 0, errpos, 0, corr);
-for (i = 0; i < numerr; i++) {
- do_error_correction_in_your_buffer(errpos[i], corr[i]);
-}
- </programlisting>
- </sect2>
- </sect1>
- <sect1>
- <title>Cleanup</title>
- <para>
- The function free_rs frees the allocated resources,
- if the caller is the last user of the decoder.
- </para>
- <programlisting>
-/* Release resources */
-free_rs(rs_decoder);
- </programlisting>
- </sect1>
-
- </chapter>
-
- <chapter id="structs">
- <title>Structures</title>
- <para>
- This chapter contains the autogenerated documentation of the structures which are
- used in the Reed-Solomon Library and are relevant for a developer.
- </para>
-!Iinclude/linux/rslib.h
- </chapter>
-
- <chapter id="pubfunctions">
- <title>Public Functions Provided</title>
- <para>
- This chapter contains the autogenerated documentation of the Reed-Solomon functions
- which are exported.
- </para>
-!Elib/reed_solomon/reed_solomon.c
- </chapter>
-
- <chapter id="credits">
- <title>Credits</title>
- <para>
- The library code for encoding and decoding was written by Phil Karn.
- </para>
- <programlisting>
- Copyright 2002, Phil Karn, KA9Q
- May be used under the terms of the GNU General Public License (GPL)
- </programlisting>
- <para>
- The wrapper functions and interfaces are written by Thomas Gleixner.
- </para>
- <para>
- Many users have provided bugfixes, improvements and helping hands for testing.
- Thanks a lot.
- </para>
- <para>
- The following people have contributed to this document:
- </para>
- <para>
- Thomas Gleixner<email>tglx@linutronix.de</email>
- </para>
- </chapter>
-</book>
+++ /dev/null
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
-
-<article class="whitepaper" id="LinuxSecurityModule" lang="en">
- <articleinfo>
- <title>Linux Security Modules: General Security Hooks for Linux</title>
- <authorgroup>
- <author>
- <firstname>Stephen</firstname>
- <surname>Smalley</surname>
- <affiliation>
- <orgname>NAI Labs</orgname>
- <address><email>ssmalley@nai.com</email></address>
- </affiliation>
- </author>
- <author>
- <firstname>Timothy</firstname>
- <surname>Fraser</surname>
- <affiliation>
- <orgname>NAI Labs</orgname>
- <address><email>tfraser@nai.com</email></address>
- </affiliation>
- </author>
- <author>
- <firstname>Chris</firstname>
- <surname>Vance</surname>
- <affiliation>
- <orgname>NAI Labs</orgname>
- <address><email>cvance@nai.com</email></address>
- </affiliation>
- </author>
- </authorgroup>
- </articleinfo>
-
-<sect1 id="Introduction"><title>Introduction</title>
-
-<para>
-In March 2001, the National Security Agency (NSA) gave a presentation
-about Security-Enhanced Linux (SELinux) at the 2.5 Linux Kernel
-Summit. SELinux is an implementation of flexible and fine-grained
-nondiscretionary access controls in the Linux kernel, originally
-implemented as its own particular kernel patch. Several other
-security projects (e.g. RSBAC, Medusa) have also developed flexible
-access control architectures for the Linux kernel, and various
-projects have developed particular access control models for Linux
-(e.g. LIDS, DTE, SubDomain). Each project has developed and
-maintained its own kernel patch to support its security needs.
-</para>
-
-<para>
-In response to the NSA presentation, Linus Torvalds made a set of
-remarks that described a security framework he would be willing to
-consider for inclusion in the mainstream Linux kernel. He described a
-general framework that would provide a set of security hooks to
-control operations on kernel objects and a set of opaque security
-fields in kernel data structures for maintaining security attributes.
-This framework could then be used by loadable kernel modules to
-implement any desired model of security. Linus also suggested the
-possibility of migrating the Linux capabilities code into such a
-module.
-</para>
-
-<para>
-The Linux Security Modules (LSM) project was started by WireX to
-develop such a framework. LSM is a joint development effort by
-several security projects, including Immunix, SELinux, SGI and Janus,
-and several individuals, including Greg Kroah-Hartman and James
-Morris, to develop a Linux kernel patch that implements this
-framework. The patch is currently tracking the 2.4 series and is
-targeted for integration into the 2.5 development series. This
-technical report provides an overview of the framework and the example
-capabilities security module provided by the LSM kernel patch.
-</para>
-
-</sect1>
-
-<sect1 id="framework"><title>LSM Framework</title>
-
-<para>
-The LSM kernel patch provides a general kernel framework to support
-security modules. In particular, the LSM framework is primarily
-focused on supporting access control modules, although future
-development is likely to address other security needs such as
-auditing. By itself, the framework does not provide any additional
-security; it merely provides the infrastructure to support security
-modules. The LSM kernel patch also moves most of the capabilities
-logic into an optional security module, with the system defaulting
-to the traditional superuser logic. This capabilities module
-is discussed further in <xref linkend="cap"/>.
-</para>
-
-<para>
-The LSM kernel patch adds security fields to kernel data structures
-and inserts calls to hook functions at critical points in the kernel
-code to manage the security fields and to perform access control. It
-also adds functions for registering and unregistering security
-modules, and adds a general <function>security</function> system call
-to support new system calls for security-aware applications.
-</para>
-
-<para>
-The LSM security fields are simply <type>void*</type> pointers. For
-process and program execution security information, security fields
-were added to <structname>struct task_struct</structname> and
-<structname>struct linux_binprm</structname>. For filesystem security
-information, a security field was added to
-<structname>struct super_block</structname>. For pipe, file, and socket
-security information, security fields were added to
-<structname>struct inode</structname> and
-<structname>struct file</structname>. For packet and network device security
-information, security fields were added to
-<structname>struct sk_buff</structname> and
-<structname>struct net_device</structname>. For System V IPC security
-information, security fields were added to
-<structname>struct kern_ipc_perm</structname> and
-<structname>struct msg_msg</structname>; additionally, the definitions
-for <structname>struct msg_msg</structname>, <structname>struct
-msg_queue</structname>, and <structname>struct
-shmid_kernel</structname> were moved to header files
-(<filename>include/linux/msg.h</filename> and
-<filename>include/linux/shm.h</filename> as appropriate) to allow
-the security modules to use these definitions.
-</para>
-
-<para>
-Each LSM hook is a function pointer in a global table,
-security_ops. This table is a
-<structname>security_operations</structname> structure as defined by
-<filename>include/linux/security.h</filename>. Detailed documentation
-for each hook is included in this header file. At present, this
-structure consists of a collection of substructures that group related
-hooks based on the kernel object (e.g. task, inode, file, sk_buff,
-etc) as well as some top-level hook function pointers for system
-operations. This structure is likely to be flattened in the future
-for performance. The placement of the hook calls in the kernel code
-is described by the "called:" lines in the per-hook documentation in
-the header file. The hook calls can also be easily found in the
-kernel code by looking for the string "security_ops->".
-
-</para>
-
-<para>
-Linus mentioned per-process security hooks in his original remarks as a
-possible alternative to global security hooks. However, if LSM were
-to start from the perspective of per-process hooks, then the base
-framework would have to deal with how to handle operations that
-involve multiple processes (e.g. kill), since each process might have
-its own hook for controlling the operation. This would require a
-general mechanism for composing hooks in the base framework.
-Additionally, LSM would still need global hooks for operations that
-have no process context (e.g. network input operations).
-Consequently, LSM provides global security hooks, but a security
-module is free to implement per-process hooks (where that makes sense)
-by storing a security_ops table in each process' security field and
-then invoking these per-process hooks from the global hooks.
-The problem of composition is thus deferred to the module.
-</para>
-
-<para>
-The global security_ops table is initialized to a set of hook
-functions provided by a dummy security module that provides
-traditional superuser logic. A <function>register_security</function>
-function (in <filename>security/security.c</filename>) is provided to
-allow a security module to set security_ops to refer to its own hook
-functions, and an <function>unregister_security</function> function is
-provided to revert security_ops to the dummy module hooks. This
-mechanism is used to set the primary security module, which is
-responsible for making the final decision for each hook.
-</para>
-
-<para>
-LSM also provides a simple mechanism for stacking additional security
-modules with the primary security module. It defines
-<function>register_security</function> and
-<function>unregister_security</function> hooks in the
-<structname>security_operations</structname> structure and provides
-<function>mod_reg_security</function> and
-<function>mod_unreg_security</function> functions that invoke these
-hooks after performing some sanity checking. A security module can
-call these functions in order to stack with other modules. However,
-the actual details of how this stacking is handled are deferred to the
-module, which can implement these hooks in any way it wishes
-(including always returning an error if it does not wish to support
-stacking). In this manner, LSM again defers the problem of
-composition to the module.
-</para>
-
-<para>
-Although the LSM hooks are organized into substructures based on
-kernel object, all of the hooks can be viewed as falling into two
-major categories: hooks that are used to manage the security fields
-and hooks that are used to perform access control. Examples of the
-first category of hooks include the
-<function>alloc_security</function> and
-<function>free_security</function> hooks defined for each kernel data
-structure that has a security field. These hooks are used to allocate
-and free security structures for kernel objects. The first category
-of hooks also includes hooks that set information in the security
-field after allocation, such as the <function>post_lookup</function>
-hook in <structname>struct inode_security_ops</structname>. This hook
-is used to set security information for inodes after successful lookup
-operations. An example of the second category of hooks is the
-<function>permission</function> hook in
-<structname>struct inode_security_ops</structname>. This hook checks
-permission when accessing an inode.
-</para>
-
-</sect1>
-
-<sect1 id="cap"><title>LSM Capabilities Module</title>
-
-<para>
-The LSM kernel patch moves most of the existing POSIX.1e capabilities
-logic into an optional security module stored in the file
-<filename>security/capability.c</filename>. This change allows
-users who do not want to use capabilities to omit this code entirely
-from their kernel, instead using the dummy module for traditional
-superuser logic or any other module that they desire. This change
-also allows the developers of the capabilities logic to maintain and
-enhance their code more freely, without needing to integrate patches
-back into the base kernel.
-</para>
-
-<para>
-In addition to moving the capabilities logic, the LSM kernel patch
-could move the capability-related fields from the kernel data
-structures into the new security fields managed by the security
-modules. However, at present, the LSM kernel patch leaves the
-capability fields in the kernel data structures. In his original
-remarks, Linus suggested that this might be preferable so that other
-security modules can be easily stacked with the capabilities module
-without needing to chain multiple security structures on the security field.
-It also avoids imposing extra overhead on the capabilities module
-to manage the security fields. However, the LSM framework could
-certainly support such a move if it is determined to be desirable,
-with only a few additional changes described below.
-</para>
-
-<para>
-At present, the capabilities logic for computing process capabilities
-on <function>execve</function> and <function>set*uid</function>,
-checking capabilities for a particular process, saving and checking
-capabilities for netlink messages, and handling the
-<function>capget</function> and <function>capset</function> system
-calls have been moved into the capabilities module. There are still a
-few locations in the base kernel where capability-related fields are
-directly examined or modified, but the current version of the LSM
-patch does allow a security module to completely replace the
-assignment and testing of capabilities. These few locations would
-need to be changed if the capability-related fields were moved into
-the security field. The following is a list of known locations that
-still perform such direct examination or modification of
-capability-related fields:
-<itemizedlist>
-<listitem><para><filename>fs/open.c</filename>:<function>sys_access</function></para></listitem>
-<listitem><para><filename>fs/lockd/host.c</filename>:<function>nlm_bind_host</function></para></listitem>
-<listitem><para><filename>fs/nfsd/auth.c</filename>:<function>nfsd_setuser</function></para></listitem>
-<listitem><para><filename>fs/proc/array.c</filename>:<function>task_cap</function></para></listitem>
-</itemizedlist>
-</para>
-
-</sect1>
-
-</article>
+++ /dev/null
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
-
-<book id="MTD-NAND-Guide">
- <bookinfo>
- <title>MTD NAND Driver Programming Interface</title>
-
- <authorgroup>
- <author>
- <firstname>Thomas</firstname>
- <surname>Gleixner</surname>
- <affiliation>
- <address>
- <email>tglx@linutronix.de</email>
- </address>
- </affiliation>
- </author>
- </authorgroup>
-
- <copyright>
- <year>2004</year>
- <holder>Thomas Gleixner</holder>
- </copyright>
-
- <legalnotice>
- <para>
- This documentation is free software; you can redistribute
- it and/or modify it under the terms of the GNU General Public
- License version 2 as published by the Free Software Foundation.
- </para>
-
- <para>
- This program is distributed in the hope that it will be
- useful, but WITHOUT ANY WARRANTY; without even the implied
- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- See the GNU General Public License for more details.
- </para>
-
- <para>
- You should have received a copy of the GNU General Public
- License along with this program; if not, write to the Free
- Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
- MA 02111-1307 USA
- </para>
-
- <para>
- For more details see the file COPYING in the source
- distribution of Linux.
- </para>
- </legalnotice>
- </bookinfo>
-
-<toc></toc>
-
- <chapter id="intro">
- <title>Introduction</title>
- <para>
- The generic NAND driver supports almost all NAND and AG-AND based
- chips and connects them to the Memory Technology Devices (MTD)
- subsystem of the Linux Kernel.
- </para>
- <para>
- This documentation is provided for developers who want to implement
- board drivers or filesystem drivers suitable for NAND devices.
- </para>
- </chapter>
-
- <chapter id="bugs">
- <title>Known Bugs And Assumptions</title>
- <para>
- None.
- </para>
- </chapter>
-
- <chapter id="dochints">
- <title>Documentation hints</title>
- <para>
- The function and structure docs are autogenerated. Each function and
- struct member has a short description which is marked with an [XXX] identifier.
- The following chapters explain the meaning of those identifiers.
- </para>
- <sect1 id="Function_identifiers_XXX">
- <title>Function identifiers [XXX]</title>
- <para>
- The functions are marked with [XXX] identifiers in the short
- comment. The identifiers explain the usage and scope of the
- functions. Following identifiers are used:
- </para>
- <itemizedlist>
- <listitem><para>
- [MTD Interface]</para><para>
- These functions provide the interface to the MTD kernel API.
- They are not replaceable and provide functionality
- which is complete hardware independent.
- </para></listitem>
- <listitem><para>
- [NAND Interface]</para><para>
- These functions are exported and provide the interface to the NAND kernel API.
- </para></listitem>
- <listitem><para>
- [GENERIC]</para><para>
- Generic functions are not replaceable and provide functionality
- which is complete hardware independent.
- </para></listitem>
- <listitem><para>
- [DEFAULT]</para><para>
- Default functions provide hardware related functionality which is suitable
- for most of the implementations. These functions can be replaced by the
- board driver if necessary. Those functions are called via pointers in the
- NAND chip description structure. The board driver can set the functions which
- should be replaced by board dependent functions before calling nand_scan().
- If the function pointer is NULL on entry to nand_scan() then the pointer
- is set to the default function which is suitable for the detected chip type.
- </para></listitem>
- </itemizedlist>
- </sect1>
- <sect1 id="Struct_member_identifiers_XXX">
- <title>Struct member identifiers [XXX]</title>
- <para>
- The struct members are marked with [XXX] identifiers in the
- comment. The identifiers explain the usage and scope of the
- members. Following identifiers are used:
- </para>
- <itemizedlist>
- <listitem><para>
- [INTERN]</para><para>
- These members are for NAND driver internal use only and must not be
- modified. Most of these values are calculated from the chip geometry
- information which is evaluated during nand_scan().
- </para></listitem>
- <listitem><para>
- [REPLACEABLE]</para><para>
- Replaceable members hold hardware related functions which can be
- provided by the board driver. The board driver can set the functions which
- should be replaced by board dependent functions before calling nand_scan().
- If the function pointer is NULL on entry to nand_scan() then the pointer
- is set to the default function which is suitable for the detected chip type.
- </para></listitem>
- <listitem><para>
- [BOARDSPECIFIC]</para><para>
- Board specific members hold hardware related information which must
- be provided by the board driver. The board driver must set the function
- pointers and datafields before calling nand_scan().
- </para></listitem>
- <listitem><para>
- [OPTIONAL]</para><para>
- Optional members can hold information relevant for the board driver. The
- generic NAND driver code does not use this information.
- </para></listitem>
- </itemizedlist>
- </sect1>
- </chapter>
-
- <chapter id="basicboarddriver">
- <title>Basic board driver</title>
- <para>
- For most boards it will be sufficient to provide just the
- basic functions and fill out some really board dependent
- members in the nand chip description structure.
- </para>
- <sect1 id="Basic_defines">
- <title>Basic defines</title>
- <para>
- At least you have to provide a nand_chip structure
- and a storage for the ioremap'ed chip address.
- You can allocate the nand_chip structure using
- kmalloc or you can allocate it statically.
- The NAND chip structure embeds an mtd structure
- which will be registered to the MTD subsystem.
- You can extract a pointer to the mtd structure
- from a nand_chip pointer using the nand_to_mtd()
- helper.
- </para>
- <para>
- Kmalloc based example
- </para>
- <programlisting>
-static struct mtd_info *board_mtd;
-static void __iomem *baseaddr;
- </programlisting>
- <para>
- Static example
- </para>
- <programlisting>
-static struct nand_chip board_chip;
-static void __iomem *baseaddr;
- </programlisting>
- </sect1>
- <sect1 id="Partition_defines">
- <title>Partition defines</title>
- <para>
- If you want to divide your device into partitions, then
- define a partitioning scheme suitable to your board.
- </para>
- <programlisting>
-#define NUM_PARTITIONS 2
-static struct mtd_partition partition_info[] = {
- { .name = "Flash partition 1",
- .offset = 0,
- .size = 8 * 1024 * 1024 },
- { .name = "Flash partition 2",
- .offset = MTDPART_OFS_NEXT,
- .size = MTDPART_SIZ_FULL },
-};
- </programlisting>
- </sect1>
- <sect1 id="Hardware_control_functions">
- <title>Hardware control function</title>
- <para>
- The hardware control function provides access to the
- control pins of the NAND chip(s).
- The access can be done by GPIO pins or by address lines.
- If you use address lines, make sure that the timing
- requirements are met.
- </para>
- <para>
- <emphasis>GPIO based example</emphasis>
- </para>
- <programlisting>
-static void board_hwcontrol(struct mtd_info *mtd, int cmd)
-{
- switch(cmd){
- case NAND_CTL_SETCLE: /* Set CLE pin high */ break;
- case NAND_CTL_CLRCLE: /* Set CLE pin low */ break;
- case NAND_CTL_SETALE: /* Set ALE pin high */ break;
- case NAND_CTL_CLRALE: /* Set ALE pin low */ break;
- case NAND_CTL_SETNCE: /* Set nCE pin low */ break;
- case NAND_CTL_CLRNCE: /* Set nCE pin high */ break;
- }
-}
- </programlisting>
- <para>
- <emphasis>Address lines based example.</emphasis> It's assumed that the
- nCE pin is driven by a chip select decoder.
- </para>
- <programlisting>
-static void board_hwcontrol(struct mtd_info *mtd, int cmd)
-{
- struct nand_chip *this = mtd_to_nand(mtd);
- switch(cmd){
- case NAND_CTL_SETCLE: this->IO_ADDR_W |= CLE_ADRR_BIT; break;
- case NAND_CTL_CLRCLE: this->IO_ADDR_W &= ~CLE_ADRR_BIT; break;
- case NAND_CTL_SETALE: this->IO_ADDR_W |= ALE_ADRR_BIT; break;
- case NAND_CTL_CLRALE: this->IO_ADDR_W &= ~ALE_ADRR_BIT; break;
- }
-}
- </programlisting>
- </sect1>
- <sect1 id="Device_ready_function">
- <title>Device ready function</title>
- <para>
- If the hardware interface has the ready busy pin of the NAND chip connected to a
- GPIO or other accessible I/O pin, this function is used to read back the state of the
- pin. The function has no arguments and should return 0, if the device is busy (R/B pin
- is low) and 1, if the device is ready (R/B pin is high).
- If the hardware interface does not give access to the ready busy pin, then
- the function must not be defined and the function pointer this->dev_ready is set to NULL.
- </para>
- </sect1>
- <sect1 id="Init_function">
- <title>Init function</title>
- <para>
- The init function allocates memory and sets up all the board
- specific parameters and function pointers. When everything
- is set up nand_scan() is called. This function tries to
- detect and identify then chip. If a chip is found all the
- internal data fields are initialized accordingly.
- The structure(s) have to be zeroed out first and then filled with the necessary
- information about the device.
- </para>
- <programlisting>
-static int __init board_init (void)
-{
- struct nand_chip *this;
- int err = 0;
-
- /* Allocate memory for MTD device structure and private data */
- this = kzalloc(sizeof(struct nand_chip), GFP_KERNEL);
- if (!this) {
- printk ("Unable to allocate NAND MTD device structure.\n");
- err = -ENOMEM;
- goto out;
- }
-
- board_mtd = nand_to_mtd(this);
-
- /* map physical address */
- baseaddr = ioremap(CHIP_PHYSICAL_ADDRESS, 1024);
- if (!baseaddr) {
- printk("Ioremap to access NAND chip failed\n");
- err = -EIO;
- goto out_mtd;
- }
-
- /* Set address of NAND IO lines */
- this->IO_ADDR_R = baseaddr;
- this->IO_ADDR_W = baseaddr;
- /* Reference hardware control function */
- this->hwcontrol = board_hwcontrol;
- /* Set command delay time, see datasheet for correct value */
- this->chip_delay = CHIP_DEPENDEND_COMMAND_DELAY;
- /* Assign the device ready function, if available */
- this->dev_ready = board_dev_ready;
- this->eccmode = NAND_ECC_SOFT;
-
- /* Scan to find existence of the device */
- if (nand_scan (board_mtd, 1)) {
- err = -ENXIO;
- goto out_ior;
- }
-
- add_mtd_partitions(board_mtd, partition_info, NUM_PARTITIONS);
- goto out;
-
-out_ior:
- iounmap(baseaddr);
-out_mtd:
- kfree (this);
-out:
- return err;
-}
-module_init(board_init);
- </programlisting>
- </sect1>
- <sect1 id="Exit_function">
- <title>Exit function</title>
- <para>
- The exit function is only necessary if the driver is
- compiled as a module. It releases all resources which
- are held by the chip driver and unregisters the partitions
- in the MTD layer.
- </para>
- <programlisting>
-#ifdef MODULE
-static void __exit board_cleanup (void)
-{
- /* Release resources, unregister device */
- nand_release (board_mtd);
-
- /* unmap physical address */
- iounmap(baseaddr);
-
- /* Free the MTD device structure */
- kfree (mtd_to_nand(board_mtd));
-}
-module_exit(board_cleanup);
-#endif
- </programlisting>
- </sect1>
- </chapter>
-
- <chapter id="boarddriversadvanced">
- <title>Advanced board driver functions</title>
- <para>
- This chapter describes the advanced functionality of the NAND
- driver. For a list of functions which can be overridden by the board
- driver see the documentation of the nand_chip structure.
- </para>
- <sect1 id="Multiple_chip_control">
- <title>Multiple chip control</title>
- <para>
- The nand driver can control chip arrays. Therefore the
- board driver must provide an own select_chip function. This
- function must (de)select the requested chip.
- The function pointer in the nand_chip structure must
- be set before calling nand_scan(). The maxchip parameter
- of nand_scan() defines the maximum number of chips to
- scan for. Make sure that the select_chip function can
- handle the requested number of chips.
- </para>
- <para>
- The nand driver concatenates the chips to one virtual
- chip and provides this virtual chip to the MTD layer.
- </para>
- <para>
- <emphasis>Note: The driver can only handle linear chip arrays
- of equally sized chips. There is no support for
- parallel arrays which extend the buswidth.</emphasis>
- </para>
- <para>
- <emphasis>GPIO based example</emphasis>
- </para>
- <programlisting>
-static void board_select_chip (struct mtd_info *mtd, int chip)
-{
- /* Deselect all chips, set all nCE pins high */
- GPIO(BOARD_NAND_NCE) |= 0xff;
- if (chip >= 0)
- GPIO(BOARD_NAND_NCE) &= ~ (1 << chip);
-}
- </programlisting>
- <para>
- <emphasis>Address lines based example.</emphasis>
- Its assumed that the nCE pins are connected to an
- address decoder.
- </para>
- <programlisting>
-static void board_select_chip (struct mtd_info *mtd, int chip)
-{
- struct nand_chip *this = mtd_to_nand(mtd);
-
- /* Deselect all chips */
- this->IO_ADDR_R &= ~BOARD_NAND_ADDR_MASK;
- this->IO_ADDR_W &= ~BOARD_NAND_ADDR_MASK;
- switch (chip) {
- case 0:
- this->IO_ADDR_R |= BOARD_NAND_ADDR_CHIP0;
- this->IO_ADDR_W |= BOARD_NAND_ADDR_CHIP0;
- break;
- ....
- case n:
- this->IO_ADDR_R |= BOARD_NAND_ADDR_CHIPn;
- this->IO_ADDR_W |= BOARD_NAND_ADDR_CHIPn;
- break;
- }
-}
- </programlisting>
- </sect1>
- <sect1 id="Hardware_ECC_support">
- <title>Hardware ECC support</title>
- <sect2 id="Functions_and_constants">
- <title>Functions and constants</title>
- <para>
- The nand driver supports three different types of
- hardware ECC.
- <itemizedlist>
- <listitem><para>NAND_ECC_HW3_256</para><para>
- Hardware ECC generator providing 3 bytes ECC per
- 256 byte.
- </para> </listitem>
- <listitem><para>NAND_ECC_HW3_512</para><para>
- Hardware ECC generator providing 3 bytes ECC per
- 512 byte.
- </para> </listitem>
- <listitem><para>NAND_ECC_HW6_512</para><para>
- Hardware ECC generator providing 6 bytes ECC per
- 512 byte.
- </para> </listitem>
- <listitem><para>NAND_ECC_HW8_512</para><para>
- Hardware ECC generator providing 6 bytes ECC per
- 512 byte.
- </para> </listitem>
- </itemizedlist>
- If your hardware generator has a different functionality
- add it at the appropriate place in nand_base.c
- </para>
- <para>
- The board driver must provide following functions:
- <itemizedlist>
- <listitem><para>enable_hwecc</para><para>
- This function is called before reading / writing to
- the chip. Reset or initialize the hardware generator
- in this function. The function is called with an
- argument which let you distinguish between read
- and write operations.
- </para> </listitem>
- <listitem><para>calculate_ecc</para><para>
- This function is called after read / write from / to
- the chip. Transfer the ECC from the hardware to
- the buffer. If the option NAND_HWECC_SYNDROME is set
- then the function is only called on write. See below.
- </para> </listitem>
- <listitem><para>correct_data</para><para>
- In case of an ECC error this function is called for
- error detection and correction. Return 1 respectively 2
- in case the error can be corrected. If the error is
- not correctable return -1. If your hardware generator
- matches the default algorithm of the nand_ecc software
- generator then use the correction function provided
- by nand_ecc instead of implementing duplicated code.
- </para> </listitem>
- </itemizedlist>
- </para>
- </sect2>
- <sect2 id="Hardware_ECC_with_syndrome_calculation">
- <title>Hardware ECC with syndrome calculation</title>
- <para>
- Many hardware ECC implementations provide Reed-Solomon
- codes and calculate an error syndrome on read. The syndrome
- must be converted to a standard Reed-Solomon syndrome
- before calling the error correction code in the generic
- Reed-Solomon library.
- </para>
- <para>
- The ECC bytes must be placed immediately after the data
- bytes in order to make the syndrome generator work. This
- is contrary to the usual layout used by software ECC. The
- separation of data and out of band area is not longer
- possible. The nand driver code handles this layout and
- the remaining free bytes in the oob area are managed by
- the autoplacement code. Provide a matching oob-layout
- in this case. See rts_from4.c and diskonchip.c for
- implementation reference. In those cases we must also
- use bad block tables on FLASH, because the ECC layout is
- interfering with the bad block marker positions.
- See bad block table support for details.
- </para>
- </sect2>
- </sect1>
- <sect1 id="Bad_Block_table_support">
- <title>Bad block table support</title>
- <para>
- Most NAND chips mark the bad blocks at a defined
- position in the spare area. Those blocks must
- not be erased under any circumstances as the bad
- block information would be lost.
- It is possible to check the bad block mark each
- time when the blocks are accessed by reading the
- spare area of the first page in the block. This
- is time consuming so a bad block table is used.
- </para>
- <para>
- The nand driver supports various types of bad block
- tables.
- <itemizedlist>
- <listitem><para>Per device</para><para>
- The bad block table contains all bad block information
- of the device which can consist of multiple chips.
- </para> </listitem>
- <listitem><para>Per chip</para><para>
- A bad block table is used per chip and contains the
- bad block information for this particular chip.
- </para> </listitem>
- <listitem><para>Fixed offset</para><para>
- The bad block table is located at a fixed offset
- in the chip (device). This applies to various
- DiskOnChip devices.
- </para> </listitem>
- <listitem><para>Automatic placed</para><para>
- The bad block table is automatically placed and
- detected either at the end or at the beginning
- of a chip (device)
- </para> </listitem>
- <listitem><para>Mirrored tables</para><para>
- The bad block table is mirrored on the chip (device) to
- allow updates of the bad block table without data loss.
- </para> </listitem>
- </itemizedlist>
- </para>
- <para>
- nand_scan() calls the function nand_default_bbt().
- nand_default_bbt() selects appropriate default
- bad block table descriptors depending on the chip information
- which was retrieved by nand_scan().
- </para>
- <para>
- The standard policy is scanning the device for bad
- blocks and build a ram based bad block table which
- allows faster access than always checking the
- bad block information on the flash chip itself.
- </para>
- <sect2 id="Flash_based_tables">
- <title>Flash based tables</title>
- <para>
- It may be desired or necessary to keep a bad block table in FLASH.
- For AG-AND chips this is mandatory, as they have no factory marked
- bad blocks. They have factory marked good blocks. The marker pattern
- is erased when the block is erased to be reused. So in case of
- powerloss before writing the pattern back to the chip this block
- would be lost and added to the bad blocks. Therefore we scan the
- chip(s) when we detect them the first time for good blocks and
- store this information in a bad block table before erasing any
- of the blocks.
- </para>
- <para>
- The blocks in which the tables are stored are protected against
- accidental access by marking them bad in the memory bad block
- table. The bad block table management functions are allowed
- to circumvent this protection.
- </para>
- <para>
- The simplest way to activate the FLASH based bad block table support
- is to set the option NAND_BBT_USE_FLASH in the bbt_option field of
- the nand chip structure before calling nand_scan(). For AG-AND
- chips is this done by default.
- This activates the default FLASH based bad block table functionality
- of the NAND driver. The default bad block table options are
- <itemizedlist>
- <listitem><para>Store bad block table per chip</para></listitem>
- <listitem><para>Use 2 bits per block</para></listitem>
- <listitem><para>Automatic placement at the end of the chip</para></listitem>
- <listitem><para>Use mirrored tables with version numbers</para></listitem>
- <listitem><para>Reserve 4 blocks at the end of the chip</para></listitem>
- </itemizedlist>
- </para>
- </sect2>
- <sect2 id="User_defined_tables">
- <title>User defined tables</title>
- <para>
- User defined tables are created by filling out a
- nand_bbt_descr structure and storing the pointer in the
- nand_chip structure member bbt_td before calling nand_scan().
- If a mirror table is necessary a second structure must be
- created and a pointer to this structure must be stored
- in bbt_md inside the nand_chip structure. If the bbt_md
- member is set to NULL then only the main table is used
- and no scan for the mirrored table is performed.
- </para>
- <para>
- The most important field in the nand_bbt_descr structure
- is the options field. The options define most of the
- table properties. Use the predefined constants from
- nand.h to define the options.
- <itemizedlist>
- <listitem><para>Number of bits per block</para>
- <para>The supported number of bits is 1, 2, 4, 8.</para></listitem>
- <listitem><para>Table per chip</para>
- <para>Setting the constant NAND_BBT_PERCHIP selects that
- a bad block table is managed for each chip in a chip array.
- If this option is not set then a per device bad block table
- is used.</para></listitem>
- <listitem><para>Table location is absolute</para>
- <para>Use the option constant NAND_BBT_ABSPAGE and
- define the absolute page number where the bad block
- table starts in the field pages. If you have selected bad block
- tables per chip and you have a multi chip array then the start page
- must be given for each chip in the chip array. Note: there is no scan
- for a table ident pattern performed, so the fields
- pattern, veroffs, offs, len can be left uninitialized</para></listitem>
- <listitem><para>Table location is automatically detected</para>
- <para>The table can either be located in the first or the last good
- blocks of the chip (device). Set NAND_BBT_LASTBLOCK to place
- the bad block table at the end of the chip (device). The
- bad block tables are marked and identified by a pattern which
- is stored in the spare area of the first page in the block which
- holds the bad block table. Store a pointer to the pattern
- in the pattern field. Further the length of the pattern has to be
- stored in len and the offset in the spare area must be given
- in the offs member of the nand_bbt_descr structure. For mirrored
- bad block tables different patterns are mandatory.</para></listitem>
- <listitem><para>Table creation</para>
- <para>Set the option NAND_BBT_CREATE to enable the table creation
- if no table can be found during the scan. Usually this is done only
- once if a new chip is found. </para></listitem>
- <listitem><para>Table write support</para>
- <para>Set the option NAND_BBT_WRITE to enable the table write support.
- This allows the update of the bad block table(s) in case a block has
- to be marked bad due to wear. The MTD interface function block_markbad
- is calling the update function of the bad block table. If the write
- support is enabled then the table is updated on FLASH.</para>
- <para>
- Note: Write support should only be enabled for mirrored tables with
- version control.
- </para></listitem>
- <listitem><para>Table version control</para>
- <para>Set the option NAND_BBT_VERSION to enable the table version control.
- It's highly recommended to enable this for mirrored tables with write
- support. It makes sure that the risk of losing the bad block
- table information is reduced to the loss of the information about the
- one worn out block which should be marked bad. The version is stored in
- 4 consecutive bytes in the spare area of the device. The position of
- the version number is defined by the member veroffs in the bad block table
- descriptor.</para></listitem>
- <listitem><para>Save block contents on write</para>
- <para>
- In case that the block which holds the bad block table does contain
- other useful information, set the option NAND_BBT_SAVECONTENT. When
- the bad block table is written then the whole block is read the bad
- block table is updated and the block is erased and everything is
- written back. If this option is not set only the bad block table
- is written and everything else in the block is ignored and erased.
- </para></listitem>
- <listitem><para>Number of reserved blocks</para>
- <para>
- For automatic placement some blocks must be reserved for
- bad block table storage. The number of reserved blocks is defined
- in the maxblocks member of the bad block table description structure.
- Reserving 4 blocks for mirrored tables should be a reasonable number.
- This also limits the number of blocks which are scanned for the bad
- block table ident pattern.
- </para></listitem>
- </itemizedlist>
- </para>
- </sect2>
- </sect1>
- <sect1 id="Spare_area_placement">
- <title>Spare area (auto)placement</title>
- <para>
- The nand driver implements different possibilities for
- placement of filesystem data in the spare area,
- <itemizedlist>
- <listitem><para>Placement defined by fs driver</para></listitem>
- <listitem><para>Automatic placement</para></listitem>
- </itemizedlist>
- The default placement function is automatic placement. The
- nand driver has built in default placement schemes for the
- various chiptypes. If due to hardware ECC functionality the
- default placement does not fit then the board driver can
- provide a own placement scheme.
- </para>
- <para>
- File system drivers can provide a own placement scheme which
- is used instead of the default placement scheme.
- </para>
- <para>
- Placement schemes are defined by a nand_oobinfo structure
- <programlisting>
-struct nand_oobinfo {
- int useecc;
- int eccbytes;
- int eccpos[24];
- int oobfree[8][2];
-};
- </programlisting>
- <itemizedlist>
- <listitem><para>useecc</para><para>
- The useecc member controls the ecc and placement function. The header
- file include/mtd/mtd-abi.h contains constants to select ecc and
- placement. MTD_NANDECC_OFF switches off the ecc complete. This is
- not recommended and available for testing and diagnosis only.
- MTD_NANDECC_PLACE selects caller defined placement, MTD_NANDECC_AUTOPLACE
- selects automatic placement.
- </para></listitem>
- <listitem><para>eccbytes</para><para>
- The eccbytes member defines the number of ecc bytes per page.
- </para></listitem>
- <listitem><para>eccpos</para><para>
- The eccpos array holds the byte offsets in the spare area where
- the ecc codes are placed.
- </para></listitem>
- <listitem><para>oobfree</para><para>
- The oobfree array defines the areas in the spare area which can be
- used for automatic placement. The information is given in the format
- {offset, size}. offset defines the start of the usable area, size the
- length in bytes. More than one area can be defined. The list is terminated
- by an {0, 0} entry.
- </para></listitem>
- </itemizedlist>
- </para>
- <sect2 id="Placement_defined_by_fs_driver">
- <title>Placement defined by fs driver</title>
- <para>
- The calling function provides a pointer to a nand_oobinfo
- structure which defines the ecc placement. For writes the
- caller must provide a spare area buffer along with the
- data buffer. The spare area buffer size is (number of pages) *
- (size of spare area). For reads the buffer size is
- (number of pages) * ((size of spare area) + (number of ecc
- steps per page) * sizeof (int)). The driver stores the
- result of the ecc check for each tuple in the spare buffer.
- The storage sequence is
- </para>
- <para>
- <spare data page 0><ecc result 0>...<ecc result n>
- </para>
- <para>
- ...
- </para>
- <para>
- <spare data page n><ecc result 0>...<ecc result n>
- </para>
- <para>
- This is a legacy mode used by YAFFS1.
- </para>
- <para>
- If the spare area buffer is NULL then only the ECC placement is
- done according to the given scheme in the nand_oobinfo structure.
- </para>
- </sect2>
- <sect2 id="Automatic_placement">
- <title>Automatic placement</title>
- <para>
- Automatic placement uses the built in defaults to place the
- ecc bytes in the spare area. If filesystem data have to be stored /
- read into the spare area then the calling function must provide a
- buffer. The buffer size per page is determined by the oobfree array in
- the nand_oobinfo structure.
- </para>
- <para>
- If the spare area buffer is NULL then only the ECC placement is
- done according to the default builtin scheme.
- </para>
- </sect2>
- </sect1>
- <sect1 id="Spare_area_autoplacement_default">
- <title>Spare area autoplacement default schemes</title>
- <sect2 id="pagesize_256">
- <title>256 byte pagesize</title>
-<informaltable><tgroup cols="3"><tbody>
-<row>
-<entry>Offset</entry>
-<entry>Content</entry>
-<entry>Comment</entry>
-</row>
-<row>
-<entry>0x00</entry>
-<entry>ECC byte 0</entry>
-<entry>Error correction code byte 0</entry>
-</row>
-<row>
-<entry>0x01</entry>
-<entry>ECC byte 1</entry>
-<entry>Error correction code byte 1</entry>
-</row>
-<row>
-<entry>0x02</entry>
-<entry>ECC byte 2</entry>
-<entry>Error correction code byte 2</entry>
-</row>
-<row>
-<entry>0x03</entry>
-<entry>Autoplace 0</entry>
-<entry></entry>
-</row>
-<row>
-<entry>0x04</entry>
-<entry>Autoplace 1</entry>
-<entry></entry>
-</row>
-<row>
-<entry>0x05</entry>
-<entry>Bad block marker</entry>
-<entry>If any bit in this byte is zero, then this block is bad.
-This applies only to the first page in a block. In the remaining
-pages this byte is reserved</entry>
-</row>
-<row>
-<entry>0x06</entry>
-<entry>Autoplace 2</entry>
-<entry></entry>
-</row>
-<row>
-<entry>0x07</entry>
-<entry>Autoplace 3</entry>
-<entry></entry>
-</row>
-</tbody></tgroup></informaltable>
- </sect2>
- <sect2 id="pagesize_512">
- <title>512 byte pagesize</title>
-<informaltable><tgroup cols="3"><tbody>
-<row>
-<entry>Offset</entry>
-<entry>Content</entry>
-<entry>Comment</entry>
-</row>
-<row>
-<entry>0x00</entry>
-<entry>ECC byte 0</entry>
-<entry>Error correction code byte 0 of the lower 256 Byte data in
-this page</entry>
-</row>
-<row>
-<entry>0x01</entry>
-<entry>ECC byte 1</entry>
-<entry>Error correction code byte 1 of the lower 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x02</entry>
-<entry>ECC byte 2</entry>
-<entry>Error correction code byte 2 of the lower 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x03</entry>
-<entry>ECC byte 3</entry>
-<entry>Error correction code byte 0 of the upper 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x04</entry>
-<entry>reserved</entry>
-<entry>reserved</entry>
-</row>
-<row>
-<entry>0x05</entry>
-<entry>Bad block marker</entry>
-<entry>If any bit in this byte is zero, then this block is bad.
-This applies only to the first page in a block. In the remaining
-pages this byte is reserved</entry>
-</row>
-<row>
-<entry>0x06</entry>
-<entry>ECC byte 4</entry>
-<entry>Error correction code byte 1 of the upper 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x07</entry>
-<entry>ECC byte 5</entry>
-<entry>Error correction code byte 2 of the upper 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x08 - 0x0F</entry>
-<entry>Autoplace 0 - 7</entry>
-<entry></entry>
-</row>
-</tbody></tgroup></informaltable>
- </sect2>
- <sect2 id="pagesize_2048">
- <title>2048 byte pagesize</title>
-<informaltable><tgroup cols="3"><tbody>
-<row>
-<entry>Offset</entry>
-<entry>Content</entry>
-<entry>Comment</entry>
-</row>
-<row>
-<entry>0x00</entry>
-<entry>Bad block marker</entry>
-<entry>If any bit in this byte is zero, then this block is bad.
-This applies only to the first page in a block. In the remaining
-pages this byte is reserved</entry>
-</row>
-<row>
-<entry>0x01</entry>
-<entry>Reserved</entry>
-<entry>Reserved</entry>
-</row>
-<row>
-<entry>0x02-0x27</entry>
-<entry>Autoplace 0 - 37</entry>
-<entry></entry>
-</row>
-<row>
-<entry>0x28</entry>
-<entry>ECC byte 0</entry>
-<entry>Error correction code byte 0 of the first 256 Byte data in
-this page</entry>
-</row>
-<row>
-<entry>0x29</entry>
-<entry>ECC byte 1</entry>
-<entry>Error correction code byte 1 of the first 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x2A</entry>
-<entry>ECC byte 2</entry>
-<entry>Error correction code byte 2 of the first 256 Bytes data in
-this page</entry>
-</row>
-<row>
-<entry>0x2B</entry>
-<entry>ECC byte 3</entry>
-<entry>Error correction code byte 0 of the second 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x2C</entry>
-<entry>ECC byte 4</entry>
-<entry>Error correction code byte 1 of the second 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x2D</entry>
-<entry>ECC byte 5</entry>
-<entry>Error correction code byte 2 of the second 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x2E</entry>
-<entry>ECC byte 6</entry>
-<entry>Error correction code byte 0 of the third 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x2F</entry>
-<entry>ECC byte 7</entry>
-<entry>Error correction code byte 1 of the third 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x30</entry>
-<entry>ECC byte 8</entry>
-<entry>Error correction code byte 2 of the third 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x31</entry>
-<entry>ECC byte 9</entry>
-<entry>Error correction code byte 0 of the fourth 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x32</entry>
-<entry>ECC byte 10</entry>
-<entry>Error correction code byte 1 of the fourth 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x33</entry>
-<entry>ECC byte 11</entry>
-<entry>Error correction code byte 2 of the fourth 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x34</entry>
-<entry>ECC byte 12</entry>
-<entry>Error correction code byte 0 of the fifth 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x35</entry>
-<entry>ECC byte 13</entry>
-<entry>Error correction code byte 1 of the fifth 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x36</entry>
-<entry>ECC byte 14</entry>
-<entry>Error correction code byte 2 of the fifth 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x37</entry>
-<entry>ECC byte 15</entry>
-<entry>Error correction code byte 0 of the sixt 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x38</entry>
-<entry>ECC byte 16</entry>
-<entry>Error correction code byte 1 of the sixt 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x39</entry>
-<entry>ECC byte 17</entry>
-<entry>Error correction code byte 2 of the sixt 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x3A</entry>
-<entry>ECC byte 18</entry>
-<entry>Error correction code byte 0 of the seventh 256 Bytes of
-data in this page</entry>
-</row>
-<row>
-<entry>0x3B</entry>
-<entry>ECC byte 19</entry>
-<entry>Error correction code byte 1 of the seventh 256 Bytes of
-data in this page</entry>
-</row>
-<row>
-<entry>0x3C</entry>
-<entry>ECC byte 20</entry>
-<entry>Error correction code byte 2 of the seventh 256 Bytes of
-data in this page</entry>
-</row>
-<row>
-<entry>0x3D</entry>
-<entry>ECC byte 21</entry>
-<entry>Error correction code byte 0 of the eighth 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x3E</entry>
-<entry>ECC byte 22</entry>
-<entry>Error correction code byte 1 of the eighth 256 Bytes of data
-in this page</entry>
-</row>
-<row>
-<entry>0x3F</entry>
-<entry>ECC byte 23</entry>
-<entry>Error correction code byte 2 of the eighth 256 Bytes of data
-in this page</entry>
-</row>
-</tbody></tgroup></informaltable>
- </sect2>
- </sect1>
- </chapter>
-
- <chapter id="filesystems">
- <title>Filesystem support</title>
- <para>
- The NAND driver provides all necessary functions for a
- filesystem via the MTD interface.
- </para>
- <para>
- Filesystems must be aware of the NAND peculiarities and
- restrictions. One major restrictions of NAND Flash is, that you cannot
- write as often as you want to a page. The consecutive writes to a page,
- before erasing it again, are restricted to 1-3 writes, depending on the
- manufacturers specifications. This applies similar to the spare area.
- </para>
- <para>
- Therefore NAND aware filesystems must either write in page size chunks
- or hold a writebuffer to collect smaller writes until they sum up to
- pagesize. Available NAND aware filesystems: JFFS2, YAFFS.
- </para>
- <para>
- The spare area usage to store filesystem data is controlled by
- the spare area placement functionality which is described in one
- of the earlier chapters.
- </para>
- </chapter>
- <chapter id="tools">
- <title>Tools</title>
- <para>
- The MTD project provides a couple of helpful tools to handle NAND Flash.
- <itemizedlist>
- <listitem><para>flasherase, flasheraseall: Erase and format FLASH partitions</para></listitem>
- <listitem><para>nandwrite: write filesystem images to NAND FLASH</para></listitem>
- <listitem><para>nanddump: dump the contents of a NAND FLASH partitions</para></listitem>
- </itemizedlist>
- </para>
- <para>
- These tools are aware of the NAND restrictions. Please use those tools
- instead of complaining about errors which are caused by non NAND aware
- access methods.
- </para>
- </chapter>
-
- <chapter id="defines">
- <title>Constants</title>
- <para>
- This chapter describes the constants which might be relevant for a driver developer.
- </para>
- <sect1 id="Chip_option_constants">
- <title>Chip option constants</title>
- <sect2 id="Constants_for_chip_id_table">
- <title>Constants for chip id table</title>
- <para>
- These constants are defined in nand.h. They are ored together to describe
- the chip functionality.
- <programlisting>
-/* Buswitdh is 16 bit */
-#define NAND_BUSWIDTH_16 0x00000002
-/* Device supports partial programming without padding */
-#define NAND_NO_PADDING 0x00000004
-/* Chip has cache program function */
-#define NAND_CACHEPRG 0x00000008
-/* Chip has copy back function */
-#define NAND_COPYBACK 0x00000010
-/* AND Chip which has 4 banks and a confusing page / block
- * assignment. See Renesas datasheet for further information */
-#define NAND_IS_AND 0x00000020
-/* Chip has a array of 4 pages which can be read without
- * additional ready /busy waits */
-#define NAND_4PAGE_ARRAY 0x00000040
- </programlisting>
- </para>
- </sect2>
- <sect2 id="Constants_for_runtime_options">
- <title>Constants for runtime options</title>
- <para>
- These constants are defined in nand.h. They are ored together to describe
- the functionality.
- <programlisting>
-/* The hw ecc generator provides a syndrome instead a ecc value on read
- * This can only work if we have the ecc bytes directly behind the
- * data bytes. Applies for DOC and AG-AND Renesas HW Reed Solomon generators */
-#define NAND_HWECC_SYNDROME 0x00020000
- </programlisting>
- </para>
- </sect2>
- </sect1>
-
- <sect1 id="EEC_selection_constants">
- <title>ECC selection constants</title>
- <para>
- Use these constants to select the ECC algorithm.
- <programlisting>
-/* No ECC. Usage is not recommended ! */
-#define NAND_ECC_NONE 0
-/* Software ECC 3 byte ECC per 256 Byte data */
-#define NAND_ECC_SOFT 1
-/* Hardware ECC 3 byte ECC per 256 Byte data */
-#define NAND_ECC_HW3_256 2
-/* Hardware ECC 3 byte ECC per 512 Byte data */
-#define NAND_ECC_HW3_512 3
-/* Hardware ECC 6 byte ECC per 512 Byte data */
-#define NAND_ECC_HW6_512 4
-/* Hardware ECC 6 byte ECC per 512 Byte data */
-#define NAND_ECC_HW8_512 6
- </programlisting>
- </para>
- </sect1>
-
- <sect1 id="Hardware_control_related_constants">
- <title>Hardware control related constants</title>
- <para>
- These constants describe the requested hardware access function when
- the boardspecific hardware control function is called
- <programlisting>
-/* Select the chip by setting nCE to low */
-#define NAND_CTL_SETNCE 1
-/* Deselect the chip by setting nCE to high */
-#define NAND_CTL_CLRNCE 2
-/* Select the command latch by setting CLE to high */
-#define NAND_CTL_SETCLE 3
-/* Deselect the command latch by setting CLE to low */
-#define NAND_CTL_CLRCLE 4
-/* Select the address latch by setting ALE to high */
-#define NAND_CTL_SETALE 5
-/* Deselect the address latch by setting ALE to low */
-#define NAND_CTL_CLRALE 6
-/* Set write protection by setting WP to high. Not used! */
-#define NAND_CTL_SETWP 7
-/* Clear write protection by setting WP to low. Not used! */
-#define NAND_CTL_CLRWP 8
- </programlisting>
- </para>
- </sect1>
-
- <sect1 id="Bad_block_table_constants">
- <title>Bad block table related constants</title>
- <para>
- These constants describe the options used for bad block
- table descriptors.
- <programlisting>
-/* Options for the bad block table descriptors */
-
-/* The number of bits used per block in the bbt on the device */
-#define NAND_BBT_NRBITS_MSK 0x0000000F
-#define NAND_BBT_1BIT 0x00000001
-#define NAND_BBT_2BIT 0x00000002
-#define NAND_BBT_4BIT 0x00000004
-#define NAND_BBT_8BIT 0x00000008
-/* The bad block table is in the last good block of the device */
-#define NAND_BBT_LASTBLOCK 0x00000010
-/* The bbt is at the given page, else we must scan for the bbt */
-#define NAND_BBT_ABSPAGE 0x00000020
-/* bbt is stored per chip on multichip devices */
-#define NAND_BBT_PERCHIP 0x00000080
-/* bbt has a version counter at offset veroffs */
-#define NAND_BBT_VERSION 0x00000100
-/* Create a bbt if none axists */
-#define NAND_BBT_CREATE 0x00000200
-/* Write bbt if necessary */
-#define NAND_BBT_WRITE 0x00001000
-/* Read and write back block contents when writing bbt */
-#define NAND_BBT_SAVECONTENT 0x00002000
- </programlisting>
- </para>
- </sect1>
-
- </chapter>
-
- <chapter id="structs">
- <title>Structures</title>
- <para>
- This chapter contains the autogenerated documentation of the structures which are
- used in the NAND driver and might be relevant for a driver developer. Each
- struct member has a short description which is marked with an [XXX] identifier.
- See the chapter "Documentation hints" for an explanation.
- </para>
-!Iinclude/linux/mtd/nand.h
- </chapter>
-
- <chapter id="pubfunctions">
- <title>Public Functions Provided</title>
- <para>
- This chapter contains the autogenerated documentation of the NAND kernel API functions
- which are exported. Each function has a short description which is marked with an [XXX] identifier.
- See the chapter "Documentation hints" for an explanation.
- </para>
-!Edrivers/mtd/nand/nand_base.c
-!Edrivers/mtd/nand/nand_bbt.c
-!Edrivers/mtd/nand/nand_ecc.c
- </chapter>
-
- <chapter id="intfunctions">
- <title>Internal Functions Provided</title>
- <para>
- This chapter contains the autogenerated documentation of the NAND driver internal functions.
- Each function has a short description which is marked with an [XXX] identifier.
- See the chapter "Documentation hints" for an explanation.
- The functions marked with [DEFAULT] might be relevant for a board driver developer.
- </para>
-!Idrivers/mtd/nand/nand_base.c
-!Idrivers/mtd/nand/nand_bbt.c
-<!-- No internal functions for kernel-doc:
-X!Idrivers/mtd/nand/nand_ecc.c
--->
- </chapter>
-
- <chapter id="credits">
- <title>Credits</title>
- <para>
- The following people have contributed to the NAND driver:
- <orderedlist>
- <listitem><para>Steven J. Hill<email>sjhill@realitydiluted.com</email></para></listitem>
- <listitem><para>David Woodhouse<email>dwmw2@infradead.org</email></para></listitem>
- <listitem><para>Thomas Gleixner<email>tglx@linutronix.de</email></para></listitem>
- </orderedlist>
- A lot of users have provided bugfixes, improvements and helping hands for testing.
- Thanks a lot.
- </para>
- <para>
- The following people have contributed to this document:
- <orderedlist>
- <listitem><para>Thomas Gleixner<email>tglx@linutronix.de</email></para></listitem>
- </orderedlist>
- </para>
- </chapter>
-</book>
+++ /dev/null
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
-
-<book id="LinuxNetworking">
- <bookinfo>
- <title>Linux Networking and Network Devices APIs</title>
-
- <legalnotice>
- <para>
- This documentation is free software; you can redistribute
- it and/or modify it under the terms of the GNU General Public
- License as published by the Free Software Foundation; either
- version 2 of the License, or (at your option) any later
- version.
- </para>
-
- <para>
- This program is distributed in the hope that it will be
- useful, but WITHOUT ANY WARRANTY; without even the implied
- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- See the GNU General Public License for more details.
- </para>
-
- <para>
- You should have received a copy of the GNU General Public
- License along with this program; if not, write to the Free
- Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
- MA 02111-1307 USA
- </para>
-
- <para>
- For more details see the file COPYING in the source
- distribution of Linux.
- </para>
- </legalnotice>
- </bookinfo>
-
-<toc></toc>
-
- <chapter id="netcore">
- <title>Linux Networking</title>
- <sect1><title>Networking Base Types</title>
-!Iinclude/linux/net.h
- </sect1>
- <sect1><title>Socket Buffer Functions</title>
-!Iinclude/linux/skbuff.h
-!Iinclude/net/sock.h
-!Enet/socket.c
-!Enet/core/skbuff.c
-!Enet/core/sock.c
-!Enet/core/datagram.c
-!Enet/core/stream.c
- </sect1>
- <sect1><title>Socket Filter</title>
-!Enet/core/filter.c
- </sect1>
- <sect1><title>Generic Network Statistics</title>
-!Iinclude/uapi/linux/gen_stats.h
-!Enet/core/gen_stats.c
-!Enet/core/gen_estimator.c
- </sect1>
- <sect1><title>SUN RPC subsystem</title>
-<!-- The !D functionality is not perfect, garbage has to be protected by comments
-!Dnet/sunrpc/sunrpc_syms.c
--->
-!Enet/sunrpc/xdr.c
-!Enet/sunrpc/svc_xprt.c
-!Enet/sunrpc/xprt.c
-!Enet/sunrpc/sched.c
-!Enet/sunrpc/socklib.c
-!Enet/sunrpc/stats.c
-!Enet/sunrpc/rpc_pipe.c
-!Enet/sunrpc/rpcb_clnt.c
-!Enet/sunrpc/clnt.c
- </sect1>
- <sect1><title>WiMAX</title>
-!Enet/wimax/op-msg.c
-!Enet/wimax/op-reset.c
-!Enet/wimax/op-rfkill.c
-!Enet/wimax/stack.c
-!Iinclude/net/wimax.h
-!Iinclude/uapi/linux/wimax.h
- </sect1>
- </chapter>
-
- <chapter id="netdev">
- <title>Network device support</title>
- <sect1><title>Driver Support</title>
-!Enet/core/dev.c
-!Enet/ethernet/eth.c
-!Enet/sched/sch_generic.c
-!Iinclude/linux/etherdevice.h
-!Iinclude/linux/netdevice.h
- </sect1>
- <sect1><title>PHY Support</title>
-!Edrivers/net/phy/phy.c
-!Idrivers/net/phy/phy.c
-!Edrivers/net/phy/phy_device.c
-!Idrivers/net/phy/phy_device.c
-!Edrivers/net/phy/mdio_bus.c
-!Idrivers/net/phy/mdio_bus.c
- </sect1>
-<!-- FIXME: Removed for now since no structured comments in source
- <sect1><title>Wireless</title>
-X!Enet/core/wireless.c
- </sect1>
--->
- </chapter>
-
-</book>
+++ /dev/null
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" [
- <!ENTITY rapidio SYSTEM "rapidio.xml">
- ]>
-
-<book id="RapidIO-Guide">
- <bookinfo>
- <title>RapidIO Subsystem Guide</title>
-
- <authorgroup>
- <author>
- <firstname>Matt</firstname>
- <surname>Porter</surname>
- <affiliation>
- <address>
- <email>mporter@kernel.crashing.org</email>
- <email>mporter@mvista.com</email>
- </address>
- </affiliation>
- </author>
- </authorgroup>
-
- <copyright>
- <year>2005</year>
- <holder>MontaVista Software, Inc.</holder>
- </copyright>
-
- <legalnotice>
- <para>
- This documentation is free software; you can redistribute
- it and/or modify it under the terms of the GNU General Public
- License version 2 as published by the Free Software Foundation.
- </para>
-
- <para>
- This program is distributed in the hope that it will be
- useful, but WITHOUT ANY WARRANTY; without even the implied
- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- See the GNU General Public License for more details.
- </para>
-
- <para>
- You should have received a copy of the GNU General Public
- License along with this program; if not, write to the Free
- Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
- MA 02111-1307 USA
- </para>
-
- <para>
- For more details see the file COPYING in the source
- distribution of Linux.
- </para>
- </legalnotice>
- </bookinfo>
-
-<toc></toc>
-
- <chapter id="intro">
- <title>Introduction</title>
- <para>
- RapidIO is a high speed switched fabric interconnect with
- features aimed at the embedded market. RapidIO provides
- support for memory-mapped I/O as well as message-based
- transactions over the switched fabric network. RapidIO has
- a standardized discovery mechanism not unlike the PCI bus
- standard that allows simple detection of devices in a
- network.
- </para>
- <para>
- This documentation is provided for developers intending
- to support RapidIO on new architectures, write new drivers,
- or to understand the subsystem internals.
- </para>
- </chapter>
-
- <chapter id="bugs">
- <title>Known Bugs and Limitations</title>
-
- <sect1 id="known_bugs">
- <title>Bugs</title>
- <para>None. ;)</para>
- </sect1>
- <sect1 id="Limitations">
- <title>Limitations</title>
- <para>
- <orderedlist>
- <listitem><para>Access/management of RapidIO memory regions is not supported</para></listitem>
- <listitem><para>Multiple host enumeration is not supported</para></listitem>
- </orderedlist>
- </para>
- </sect1>
- </chapter>
-
- <chapter id="drivers">
- <title>RapidIO driver interface</title>
- <para>
- Drivers are provided a set of calls in order
- to interface with the subsystem to gather info
- on devices, request/map memory region resources,
- and manage mailboxes/doorbells.
- </para>
- <sect1 id="Functions">
- <title>Functions</title>
-!Iinclude/linux/rio_drv.h
-!Edrivers/rapidio/rio-driver.c
-!Edrivers/rapidio/rio.c
- </sect1>
- </chapter>
-
- <chapter id="internals">
- <title>Internals</title>
-
- <para>
- This chapter contains the autogenerated documentation of the RapidIO
- subsystem.
- </para>
-
- <sect1 id="Structures"><title>Structures</title>
-!Iinclude/linux/rio.h
- </sect1>
- <sect1 id="Enumeration_and_Discovery"><title>Enumeration and Discovery</title>
-!Idrivers/rapidio/rio-scan.c
- </sect1>
- <sect1 id="Driver_functionality"><title>Driver functionality</title>
-!Idrivers/rapidio/rio.c
-!Idrivers/rapidio/rio-access.c
- </sect1>
- <sect1 id="Device_model_support"><title>Device model support</title>
-!Idrivers/rapidio/rio-driver.c
- </sect1>
- <sect1 id="PPC32_support"><title>PPC32 support</title>
-!Iarch/powerpc/sysdev/fsl_rio.c
- </sect1>
- </chapter>
-
- <chapter id="credits">
- <title>Credits</title>
- <para>
- The following people have contributed to the RapidIO
- subsystem directly or indirectly:
- <orderedlist>
- <listitem><para>Matt Porter<email>mporter@kernel.crashing.org</email></para></listitem>
- <listitem><para>Randy Vinson<email>rvinson@mvista.com</email></para></listitem>
- <listitem><para>Dan Malek<email>dan@embeddedalley.com</email></para></listitem>
- </orderedlist>
- </para>
- <para>
- The following people have contributed to this document:
- <orderedlist>
- <listitem><para>Matt Porter<email>mporter@kernel.crashing.org</email></para></listitem>
- </orderedlist>
- </para>
- </chapter>
-</book>
+++ /dev/null
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
-
-<book id="s390drivers">
- <bookinfo>
- <title>Writing s390 channel device drivers</title>
-
- <authorgroup>
- <author>
- <firstname>Cornelia</firstname>
- <surname>Huck</surname>
- <affiliation>
- <address>
- <email>cornelia.huck@de.ibm.com</email>
- </address>
- </affiliation>
- </author>
- </authorgroup>
-
- <copyright>
- <year>2007</year>
- <holder>IBM Corp.</holder>
- </copyright>
-
- <legalnotice>
- <para>
- This documentation is free software; you can redistribute
- it and/or modify it under the terms of the GNU General Public
- License as published by the Free Software Foundation; either
- version 2 of the License, or (at your option) any later
- version.
- </para>
-
- <para>
- This program is distributed in the hope that it will be
- useful, but WITHOUT ANY WARRANTY; without even the implied
- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- See the GNU General Public License for more details.
- </para>
-
- <para>
- You should have received a copy of the GNU General Public
- License along with this program; if not, write to the Free
- Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
- MA 02111-1307 USA
- </para>
-
- <para>
- For more details see the file COPYING in the source
- distribution of Linux.
- </para>
- </legalnotice>
- </bookinfo>
-
-<toc></toc>
-
- <chapter id="intro">
- <title>Introduction</title>
- <para>
- This document describes the interfaces available for device drivers that
- drive s390 based channel attached I/O devices. This includes interfaces for
- interaction with the hardware and interfaces for interacting with the
- common driver core. Those interfaces are provided by the s390 common I/O
- layer.
- </para>
- <para>
- The document assumes a familarity with the technical terms associated
- with the s390 channel I/O architecture. For a description of this
- architecture, please refer to the "z/Architecture: Principles of
- Operation", IBM publication no. SA22-7832.
- </para>
- <para>
- While most I/O devices on a s390 system are typically driven through the
- channel I/O mechanism described here, there are various other methods
- (like the diag interface). These are out of the scope of this document.
- </para>
- <para>
- Some additional information can also be found in the kernel source
- under Documentation/s390/driver-model.txt.
- </para>
- </chapter>
- <chapter id="ccw">
- <title>The ccw bus</title>
- <para>
- The ccw bus typically contains the majority of devices available to
- a s390 system. Named after the channel command word (ccw), the basic
- command structure used to address its devices, the ccw bus contains
- so-called channel attached devices. They are addressed via I/O
- subchannels, visible on the css bus. A device driver for
- channel-attached devices, however, will never interact with the
- subchannel directly, but only via the I/O device on the ccw bus,
- the ccw device.
- </para>
- <sect1 id="channelIO">
- <title>I/O functions for channel-attached devices</title>
- <para>
- Some hardware structures have been translated into C structures for use
- by the common I/O layer and device drivers. For more information on
- the hardware structures represented here, please consult the Principles
- of Operation.
- </para>
-!Iarch/s390/include/asm/cio.h
- </sect1>
- <sect1 id="ccwdev">
- <title>ccw devices</title>
- <para>
- Devices that want to initiate channel I/O need to attach to the ccw bus.
- Interaction with the driver core is done via the common I/O layer, which
- provides the abstractions of ccw devices and ccw device drivers.
- </para>
- <para>
- The functions that initiate or terminate channel I/O all act upon a
- ccw device structure. Device drivers must not bypass those functions
- or strange side effects may happen.
- </para>
-!Iarch/s390/include/asm/ccwdev.h
-!Edrivers/s390/cio/device.c
-!Edrivers/s390/cio/device_ops.c
- </sect1>
- <sect1 id="cmf">
- <title>The channel-measurement facility</title>
- <para>
- The channel-measurement facility provides a means to collect
- measurement data which is made available by the channel subsystem
- for each channel attached device.
- </para>
-!Iarch/s390/include/asm/cmb.h
-!Edrivers/s390/cio/cmf.c
- </sect1>
- </chapter>
-
- <chapter id="ccwgroup">
- <title>The ccwgroup bus</title>
- <para>
- The ccwgroup bus only contains artificial devices, created by the user.
- Many networking devices (e.g. qeth) are in fact composed of several
- ccw devices (like read, write and data channel for qeth). The
- ccwgroup bus provides a mechanism to create a meta-device which
- contains those ccw devices as slave devices and can be associated
- with the netdevice.
- </para>
- <sect1 id="ccwgroupdevices">
- <title>ccw group devices</title>
-!Iarch/s390/include/asm/ccwgroup.h
-!Edrivers/s390/cio/ccwgroup.c
- </sect1>
- </chapter>
-
- <chapter id="genericinterfaces">
- <title>Generic interfaces</title>
- <para>
- Some interfaces are available to other drivers that do not necessarily
- have anything to do with the busses described above, but still are
- indirectly using basic infrastructure in the common I/O layer.
- One example is the support for adapter interrupts.
- </para>
-!Edrivers/s390/cio/airq.c
- </chapter>
-
-</book>
+++ /dev/null
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
-
-<book id="scsimid">
- <bookinfo>
- <title>SCSI Interfaces Guide</title>
-
- <authorgroup>
- <author>
- <firstname>James</firstname>
- <surname>Bottomley</surname>
- <affiliation>
- <address>
- <email>James.Bottomley@hansenpartnership.com</email>
- </address>
- </affiliation>
- </author>
-
- <author>
- <firstname>Rob</firstname>
- <surname>Landley</surname>
- <affiliation>
- <address>
- <email>rob@landley.net</email>
- </address>
- </affiliation>
- </author>
-
- </authorgroup>
-
- <copyright>
- <year>2007</year>
- <holder>Linux Foundation</holder>
- </copyright>
-
- <legalnotice>
- <para>
- This documentation is free software; you can redistribute
- it and/or modify it under the terms of the GNU General Public
- License version 2.
- </para>
-
- <para>
- This program is distributed in the hope that it will be
- useful, but WITHOUT ANY WARRANTY; without even the implied
- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- For more details see the file COPYING in the source
- distribution of Linux.
- </para>
- </legalnotice>
- </bookinfo>
-
- <toc></toc>
-
- <chapter id="intro">
- <title>Introduction</title>
- <sect1 id="protocol_vs_bus">
- <title>Protocol vs bus</title>
- <para>
- Once upon a time, the Small Computer Systems Interface defined both
- a parallel I/O bus and a data protocol to connect a wide variety of
- peripherals (disk drives, tape drives, modems, printers, scanners,
- optical drives, test equipment, and medical devices) to a host
- computer.
- </para>
- <para>
- Although the old parallel (fast/wide/ultra) SCSI bus has largely
- fallen out of use, the SCSI command set is more widely used than ever
- to communicate with devices over a number of different busses.
- </para>
- <para>
- The <ulink url='http://www.t10.org/scsi-3.htm'>SCSI protocol</ulink>
- is a big-endian peer-to-peer packet based protocol. SCSI commands
- are 6, 10, 12, or 16 bytes long, often followed by an associated data
- payload.
- </para>
- <para>
- SCSI commands can be transported over just about any kind of bus, and
- are the default protocol for storage devices attached to USB, SATA,
- SAS, Fibre Channel, FireWire, and ATAPI devices. SCSI packets are
- also commonly exchanged over Infiniband,
- <ulink url='http://i2o.shadowconnect.com/faq.php'>I20</ulink>, TCP/IP
- (<ulink url='https://en.wikipedia.org/wiki/ISCSI'>iSCSI</ulink>), even
- <ulink url='http://cyberelk.net/tim/parport/parscsi.html'>Parallel
- ports</ulink>.
- </para>
- </sect1>
- <sect1 id="subsystem_design">
- <title>Design of the Linux SCSI subsystem</title>
- <para>
- The SCSI subsystem uses a three layer design, with upper, mid, and low
- layers. Every operation involving the SCSI subsystem (such as reading
- a sector from a disk) uses one driver at each of the 3 levels: one
- upper layer driver, one lower layer driver, and the SCSI midlayer.
- </para>
- <para>
- The SCSI upper layer provides the interface between userspace and the
- kernel, in the form of block and char device nodes for I/O and
- ioctl(). The SCSI lower layer contains drivers for specific hardware
- devices.
- </para>
- <para>
- In between is the SCSI mid-layer, analogous to a network routing
- layer such as the IPv4 stack. The SCSI mid-layer routes a packet
- based data protocol between the upper layer's /dev nodes and the
- corresponding devices in the lower layer. It manages command queues,
- provides error handling and power management functions, and responds
- to ioctl() requests.
- </para>
- </sect1>
- </chapter>
-
- <chapter id="upper_layer">
- <title>SCSI upper layer</title>
- <para>
- The upper layer supports the user-kernel interface by providing
- device nodes.
- </para>
- <sect1 id="sd">
- <title>sd (SCSI Disk)</title>
- <para>sd (sd_mod.o)</para>
-<!-- !Idrivers/scsi/sd.c -->
- </sect1>
- <sect1 id="sr">
- <title>sr (SCSI CD-ROM)</title>
- <para>sr (sr_mod.o)</para>
- </sect1>
- <sect1 id="st">
- <title>st (SCSI Tape)</title>
- <para>st (st.o)</para>
- </sect1>
- <sect1 id="sg">
- <title>sg (SCSI Generic)</title>
- <para>sg (sg.o)</para>
- </sect1>
- <sect1 id="ch">
- <title>ch (SCSI Media Changer)</title>
- <para>ch (ch.c)</para>
- </sect1>
- </chapter>
-
- <chapter id="mid_layer">
- <title>SCSI mid layer</title>
-
- <sect1 id="midlayer_implementation">
- <title>SCSI midlayer implementation</title>
- <sect2 id="scsi_device.h">
- <title>include/scsi/scsi_device.h</title>
- <para>
- </para>
-!Iinclude/scsi/scsi_device.h
- </sect2>
-
- <sect2 id="scsi.c">
- <title>drivers/scsi/scsi.c</title>
- <para>Main file for the SCSI midlayer.</para>
-!Edrivers/scsi/scsi.c
- </sect2>
- <sect2 id="scsicam.c">
- <title>drivers/scsi/scsicam.c</title>
- <para>
- <ulink url='http://www.t10.org/ftp/t10/drafts/cam/cam-r12b.pdf'>SCSI
- Common Access Method</ulink> support functions, for use with
- HDIO_GETGEO, etc.
- </para>
-!Edrivers/scsi/scsicam.c
- </sect2>
- <sect2 id="scsi_error.c">
- <title>drivers/scsi/scsi_error.c</title>
- <para>Common SCSI error/timeout handling routines.</para>
-!Edrivers/scsi/scsi_error.c
- </sect2>
- <sect2 id="scsi_devinfo.c">
- <title>drivers/scsi/scsi_devinfo.c</title>
- <para>
- Manage scsi_dev_info_list, which tracks blacklisted and whitelisted
- devices.
- </para>
-!Idrivers/scsi/scsi_devinfo.c
- </sect2>
- <sect2 id="scsi_ioctl.c">
- <title>drivers/scsi/scsi_ioctl.c</title>
- <para>
- Handle ioctl() calls for SCSI devices.
- </para>
-!Edrivers/scsi/scsi_ioctl.c
- </sect2>
- <sect2 id="scsi_lib.c">
- <title>drivers/scsi/scsi_lib.c</title>
- <para>
- SCSI queuing library.
- </para>
-!Edrivers/scsi/scsi_lib.c
- </sect2>
- <sect2 id="scsi_lib_dma.c">
- <title>drivers/scsi/scsi_lib_dma.c</title>
- <para>
- SCSI library functions depending on DMA
- (map and unmap scatter-gather lists).
- </para>
-!Edrivers/scsi/scsi_lib_dma.c
- </sect2>
- <sect2 id="scsi_module.c">
- <title>drivers/scsi/scsi_module.c</title>
- <para>
- The file drivers/scsi/scsi_module.c contains legacy support for
- old-style host templates. It should never be used by any new driver.
- </para>
- </sect2>
- <sect2 id="scsi_proc.c">
- <title>drivers/scsi/scsi_proc.c</title>
- <para>
- The functions in this file provide an interface between
- the PROC file system and the SCSI device drivers
- It is mainly used for debugging, statistics and to pass
- information directly to the lowlevel driver.
-
- I.E. plumbing to manage /proc/scsi/*
- </para>
-!Idrivers/scsi/scsi_proc.c
- </sect2>
- <sect2 id="scsi_netlink.c">
- <title>drivers/scsi/scsi_netlink.c</title>
- <para>
- Infrastructure to provide async events from transports to userspace
- via netlink, using a single NETLINK_SCSITRANSPORT protocol for all
- transports.
-
- See <ulink url='http://marc.info/?l=linux-scsi&m=115507374832500&w=2'>the
- original patch submission</ulink> for more details.
- </para>
-!Idrivers/scsi/scsi_netlink.c
- </sect2>
- <sect2 id="scsi_scan.c">
- <title>drivers/scsi/scsi_scan.c</title>
- <para>
- Scan a host to determine which (if any) devices are attached.
-
- The general scanning/probing algorithm is as follows, exceptions are
- made to it depending on device specific flags, compilation options,
- and global variable (boot or module load time) settings.
-
- A specific LUN is scanned via an INQUIRY command; if the LUN has a
- device attached, a scsi_device is allocated and setup for it.
-
- For every id of every channel on the given host, start by scanning
- LUN 0. Skip hosts that don't respond at all to a scan of LUN 0.
- Otherwise, if LUN 0 has a device attached, allocate and setup a
- scsi_device for it. If target is SCSI-3 or up, issue a REPORT LUN,
- and scan all of the LUNs returned by the REPORT LUN; else,
- sequentially scan LUNs up until some maximum is reached, or a LUN is
- seen that cannot have a device attached to it.
- </para>
-!Idrivers/scsi/scsi_scan.c
- </sect2>
- <sect2 id="scsi_sysctl.c">
- <title>drivers/scsi/scsi_sysctl.c</title>
- <para>
- Set up the sysctl entry: "/dev/scsi/logging_level"
- (DEV_SCSI_LOGGING_LEVEL) which sets/returns scsi_logging_level.
- </para>
- </sect2>
- <sect2 id="scsi_sysfs.c">
- <title>drivers/scsi/scsi_sysfs.c</title>
- <para>
- SCSI sysfs interface routines.
- </para>
-!Edrivers/scsi/scsi_sysfs.c
- </sect2>
- <sect2 id="hosts.c">
- <title>drivers/scsi/hosts.c</title>
- <para>
- mid to lowlevel SCSI driver interface
- </para>
-!Edrivers/scsi/hosts.c
- </sect2>
- <sect2 id="constants.c">
- <title>drivers/scsi/constants.c</title>
- <para>
- mid to lowlevel SCSI driver interface
- </para>
-!Edrivers/scsi/constants.c
- </sect2>
- </sect1>
-
- <sect1 id="Transport_classes">
- <title>Transport classes</title>
- <para>
- Transport classes are service libraries for drivers in the SCSI
- lower layer, which expose transport attributes in sysfs.
- </para>
- <sect2 id="Fibre_Channel_transport">
- <title>Fibre Channel transport</title>
- <para>
- The file drivers/scsi/scsi_transport_fc.c defines transport attributes
- for Fibre Channel.
- </para>
-!Edrivers/scsi/scsi_transport_fc.c
- </sect2>
- <sect2 id="iSCSI_transport">
- <title>iSCSI transport class</title>
- <para>
- The file drivers/scsi/scsi_transport_iscsi.c defines transport
- attributes for the iSCSI class, which sends SCSI packets over TCP/IP
- connections.
- </para>
-!Edrivers/scsi/scsi_transport_iscsi.c
- </sect2>
- <sect2 id="SAS_transport">
- <title>Serial Attached SCSI (SAS) transport class</title>
- <para>
- The file drivers/scsi/scsi_transport_sas.c defines transport
- attributes for Serial Attached SCSI, a variant of SATA aimed at
- large high-end systems.
- </para>
- <para>
- The SAS transport class contains common code to deal with SAS HBAs,
- an aproximated representation of SAS topologies in the driver model,
- and various sysfs attributes to expose these topologies and management
- interfaces to userspace.
- </para>
- <para>
- In addition to the basic SCSI core objects this transport class
- introduces two additional intermediate objects: The SAS PHY
- as represented by struct sas_phy defines an "outgoing" PHY on
- a SAS HBA or Expander, and the SAS remote PHY represented by
- struct sas_rphy defines an "incoming" PHY on a SAS Expander or
- end device. Note that this is purely a software concept, the
- underlying hardware for a PHY and a remote PHY is the exactly
- the same.
- </para>
- <para>
- There is no concept of a SAS port in this code, users can see
- what PHYs form a wide port based on the port_identifier attribute,
- which is the same for all PHYs in a port.
- </para>
-!Edrivers/scsi/scsi_transport_sas.c
- </sect2>
- <sect2 id="SATA_transport">
- <title>SATA transport class</title>
- <para>
- The SATA transport is handled by libata, which has its own book of
- documentation in this directory.
- </para>
- </sect2>
- <sect2 id="SPI_transport">
- <title>Parallel SCSI (SPI) transport class</title>
- <para>
- The file drivers/scsi/scsi_transport_spi.c defines transport
- attributes for traditional (fast/wide/ultra) SCSI busses.
- </para>
-!Edrivers/scsi/scsi_transport_spi.c
- </sect2>
- <sect2 id="SRP_transport">
- <title>SCSI RDMA (SRP) transport class</title>
- <para>
- The file drivers/scsi/scsi_transport_srp.c defines transport
- attributes for SCSI over Remote Direct Memory Access.
- </para>
-!Edrivers/scsi/scsi_transport_srp.c
- </sect2>
- </sect1>
-
- </chapter>
-
- <chapter id="lower_layer">
- <title>SCSI lower layer</title>
- <sect1 id="hba_drivers">
- <title>Host Bus Adapter transport types</title>
- <para>
- Many modern device controllers use the SCSI command set as a protocol to
- communicate with their devices through many different types of physical
- connections.
- </para>
- <para>
- In SCSI language a bus capable of carrying SCSI commands is
- called a "transport", and a controller connecting to such a bus is
- called a "host bus adapter" (HBA).
- </para>
- <sect2 id="scsi_debug.c">
- <title>Debug transport</title>
- <para>
- The file drivers/scsi/scsi_debug.c simulates a host adapter with a
- variable number of disks (or disk like devices) attached, sharing a
- common amount of RAM. Does a lot of checking to make sure that we are
- not getting blocks mixed up, and panics the kernel if anything out of
- the ordinary is seen.
- </para>
- <para>
- To be more realistic, the simulated devices have the transport
- attributes of SAS disks.
- </para>
- <para>
- For documentation see
- <ulink url='http://sg.danny.cz/sg/sdebug26.html'>http://sg.danny.cz/sg/sdebug26.html</ulink>
- </para>
-<!-- !Edrivers/scsi/scsi_debug.c -->
- </sect2>
- <sect2 id="todo">
- <title>todo</title>
- <para>Parallel (fast/wide/ultra) SCSI, USB, SATA,
- SAS, Fibre Channel, FireWire, ATAPI devices, Infiniband,
- I20, iSCSI, Parallel ports, netlink...
- </para>
- </sect2>
- </sect1>
- </chapter>
-</book>
+++ /dev/null
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
-
-<book id="sh-drivers">
- <bookinfo>
- <title>SuperH Interfaces Guide</title>
-
- <authorgroup>
- <author>
- <firstname>Paul</firstname>
- <surname>Mundt</surname>
- <affiliation>
- <address>
- <email>lethal@linux-sh.org</email>
- </address>
- </affiliation>
- </author>
- </authorgroup>
-
- <copyright>
- <year>2008-2010</year>
- <holder>Paul Mundt</holder>
- </copyright>
- <copyright>
- <year>2008-2010</year>
- <holder>Renesas Technology Corp.</holder>
- </copyright>
- <copyright>
- <year>2010</year>
- <holder>Renesas Electronics Corp.</holder>
- </copyright>
-
- <legalnotice>
- <para>
- This documentation is free software; you can redistribute
- it and/or modify it under the terms of the GNU General Public
- License version 2 as published by the Free Software Foundation.
- </para>
-
- <para>
- This program is distributed in the hope that it will be
- useful, but WITHOUT ANY WARRANTY; without even the implied
- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- See the GNU General Public License for more details.
- </para>
-
- <para>
- You should have received a copy of the GNU General Public
- License along with this program; if not, write to the Free
- Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
- MA 02111-1307 USA
- </para>
-
- <para>
- For more details see the file COPYING in the source
- distribution of Linux.
- </para>
- </legalnotice>
- </bookinfo>
-
-<toc></toc>
-
- <chapter id="mm">
- <title>Memory Management</title>
- <sect1 id="sh4">
- <title>SH-4</title>
- <sect2 id="sq">
- <title>Store Queue API</title>
-!Earch/sh/kernel/cpu/sh4/sq.c
- </sect2>
- </sect1>
- <sect1 id="sh5">
- <title>SH-5</title>
- <sect2 id="tlb">
- <title>TLB Interfaces</title>
-!Iarch/sh/mm/tlb-sh5.c
-!Iarch/sh/include/asm/tlb_64.h
- </sect2>
- </sect1>
- </chapter>
- <chapter id="mach">
- <title>Machine Specific Interfaces</title>
- <sect1 id="dreamcast">
- <title>mach-dreamcast</title>
-!Iarch/sh/boards/mach-dreamcast/rtc.c
- </sect1>
- <sect1 id="x3proto">
- <title>mach-x3proto</title>
-!Earch/sh/boards/mach-x3proto/ilsel.c
- </sect1>
- </chapter>
- <chapter id="busses">
- <title>Busses</title>
- <sect1 id="superhyway">
- <title>SuperHyway</title>
-!Edrivers/sh/superhyway/superhyway.c
- </sect1>
-
- <sect1 id="maple">
- <title>Maple</title>
-!Edrivers/sh/maple/maple.c
- </sect1>
- </chapter>
-</book>
+++ /dev/null
-<?xml version="1.0" encoding="UTF-8"?>
-<stylesheet xmlns="http://www.w3.org/1999/XSL/Transform" version="1.0">
-<param name="chunk.quietly">1</param>
-<param name="funcsynopsis.style">ansi</param>
-<param name="funcsynopsis.tabular.threshold">80</param>
-<param name="callout.graphics">0</param>
-<!-- <param name="paper.type">A4</param> -->
-<param name="generate.consistent.ids">1</param>
-<param name="generate.section.toc.level">2</param>
-<param name="use.id.as.filename">1</param>
-</stylesheet>
+++ /dev/null
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
-
-<book id="w1id">
- <bookinfo>
- <title>W1: Dallas' 1-wire bus</title>
-
- <authorgroup>
- <author>
- <firstname>David</firstname>
- <surname>Fries</surname>
- <affiliation>
- <address>
- <email>David@Fries.net</email>
- </address>
- </affiliation>
- </author>
-
- </authorgroup>
-
- <copyright>
- <year>2013</year>
- <!--
- <holder></holder>
- -->
- </copyright>
-
- <legalnotice>
- <para>
- This documentation is free software; you can redistribute
- it and/or modify it under the terms of the GNU General Public
- License version 2.
- </para>
-
- <para>
- This program is distributed in the hope that it will be
- useful, but WITHOUT ANY WARRANTY; without even the implied
- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- For more details see the file COPYING in the source
- distribution of Linux.
- </para>
- </legalnotice>
- </bookinfo>
-
- <toc></toc>
-
- <chapter id="w1_internal">
- <title>W1 API internal to the kernel</title>
-
- <sect1 id="w1_internal_api">
- <title>W1 API internal to the kernel</title>
- <sect2 id="w1.h">
- <title>drivers/w1/w1.h</title>
- <para>W1 core functions.</para>
-!Idrivers/w1/w1.h
- </sect2>
-
- <sect2 id="w1.c">
- <title>drivers/w1/w1.c</title>
- <para>W1 core functions.</para>
-!Idrivers/w1/w1.c
- </sect2>
-
- <sect2 id="w1_family.h">
- <title>drivers/w1/w1_family.h</title>
- <para>Allows registering device family operations.</para>
-!Idrivers/w1/w1_family.h
- </sect2>
-
- <sect2 id="w1_family.c">
- <title>drivers/w1/w1_family.c</title>
- <para>Allows registering device family operations.</para>
-!Edrivers/w1/w1_family.c
- </sect2>
-
- <sect2 id="w1_int.c">
- <title>drivers/w1/w1_int.c</title>
- <para>W1 internal initialization for master devices.</para>
-!Edrivers/w1/w1_int.c
- </sect2>
-
- <sect2 id="w1_netlink.h">
- <title>drivers/w1/w1_netlink.h</title>
- <para>W1 external netlink API structures and commands.</para>
-!Idrivers/w1/w1_netlink.h
- </sect2>
-
- <sect2 id="w1_io.c">
- <title>drivers/w1/w1_io.c</title>
- <para>W1 input/output.</para>
-!Edrivers/w1/w1_io.c
-!Idrivers/w1/w1_io.c
- </sect2>
-
- </sect1>
-
-
- </chapter>
-
-</book>
+++ /dev/null
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
-
-<book id="Z85230Guide">
- <bookinfo>
- <title>Z8530 Programming Guide</title>
-
- <authorgroup>
- <author>
- <firstname>Alan</firstname>
- <surname>Cox</surname>
- <affiliation>
- <address>
- <email>alan@lxorguk.ukuu.org.uk</email>
- </address>
- </affiliation>
- </author>
- </authorgroup>
-
- <copyright>
- <year>2000</year>
- <holder>Alan Cox</holder>
- </copyright>
-
- <legalnotice>
- <para>
- This documentation is free software; you can redistribute
- it and/or modify it under the terms of the GNU General Public
- License as published by the Free Software Foundation; either
- version 2 of the License, or (at your option) any later
- version.
- </para>
-
- <para>
- This program is distributed in the hope that it will be
- useful, but WITHOUT ANY WARRANTY; without even the implied
- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- See the GNU General Public License for more details.
- </para>
-
- <para>
- You should have received a copy of the GNU General Public
- License along with this program; if not, write to the Free
- Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
- MA 02111-1307 USA
- </para>
-
- <para>
- For more details see the file COPYING in the source
- distribution of Linux.
- </para>
- </legalnotice>
- </bookinfo>
-
-<toc></toc>
-
- <chapter id="intro">
- <title>Introduction</title>
- <para>
- The Z85x30 family synchronous/asynchronous controller chips are
- used on a large number of cheap network interface cards. The
- kernel provides a core interface layer that is designed to make
- it easy to provide WAN services using this chip.
- </para>
- <para>
- The current driver only support synchronous operation. Merging the
- asynchronous driver support into this code to allow any Z85x30
- device to be used as both a tty interface and as a synchronous
- controller is a project for Linux post the 2.4 release
- </para>
- </chapter>
-
- <chapter id="Driver_Modes">
- <title>Driver Modes</title>
- <para>
- The Z85230 driver layer can drive Z8530, Z85C30 and Z85230 devices
- in three different modes. Each mode can be applied to an individual
- channel on the chip (each chip has two channels).
- </para>
- <para>
- The PIO synchronous mode supports the most common Z8530 wiring. Here
- the chip is interface to the I/O and interrupt facilities of the
- host machine but not to the DMA subsystem. When running PIO the
- Z8530 has extremely tight timing requirements. Doing high speeds,
- even with a Z85230 will be tricky. Typically you should expect to
- achieve at best 9600 baud with a Z8C530 and 64Kbits with a Z85230.
- </para>
- <para>
- The DMA mode supports the chip when it is configured to use dual DMA
- channels on an ISA bus. The better cards tend to support this mode
- of operation for a single channel. With DMA running the Z85230 tops
- out when it starts to hit ISA DMA constraints at about 512Kbits. It
- is worth noting here that many PC machines hang or crash when the
- chip is driven fast enough to hold the ISA bus solid.
- </para>
- <para>
- Transmit DMA mode uses a single DMA channel. The DMA channel is used
- for transmission as the transmit FIFO is smaller than the receive
- FIFO. it gives better performance than pure PIO mode but is nowhere
- near as ideal as pure DMA mode.
- </para>
- </chapter>
-
- <chapter id="Using_the_Z85230_driver">
- <title>Using the Z85230 driver</title>
- <para>
- The Z85230 driver provides the back end interface to your board. To
- configure a Z8530 interface you need to detect the board and to
- identify its ports and interrupt resources. It is also your problem
- to verify the resources are available.
- </para>
- <para>
- Having identified the chip you need to fill in a struct z8530_dev,
- which describes each chip. This object must exist until you finally
- shutdown the board. Firstly zero the active field. This ensures
- nothing goes off without you intending it. The irq field should
- be set to the interrupt number of the chip. (Each chip has a single
- interrupt source rather than each channel). You are responsible
- for allocating the interrupt line. The interrupt handler should be
- set to <function>z8530_interrupt</function>. The device id should
- be set to the z8530_dev structure pointer. Whether the interrupt can
- be shared or not is board dependent, and up to you to initialise.
- </para>
- <para>
- The structure holds two channel structures.
- Initialise chanA.ctrlio and chanA.dataio with the address of the
- control and data ports. You can or this with Z8530_PORT_SLEEP to
- indicate your interface needs the 5uS delay for chip settling done
- in software. The PORT_SLEEP option is architecture specific. Other
- flags may become available on future platforms, eg for MMIO.
- Initialise the chanA.irqs to &z8530_nop to start the chip up
- as disabled and discarding interrupt events. This ensures that
- stray interrupts will be mopped up and not hang the bus. Set
- chanA.dev to point to the device structure itself. The
- private and name field you may use as you wish. The private field
- is unused by the Z85230 layer. The name is used for error reporting
- and it may thus make sense to make it match the network name.
- </para>
- <para>
- Repeat the same operation with the B channel if your chip has
- both channels wired to something useful. This isn't always the
- case. If it is not wired then the I/O values do not matter, but
- you must initialise chanB.dev.
- </para>
- <para>
- If your board has DMA facilities then initialise the txdma and
- rxdma fields for the relevant channels. You must also allocate the
- ISA DMA channels and do any necessary board level initialisation
- to configure them. The low level driver will do the Z8530 and
- DMA controller programming but not board specific magic.
- </para>
- <para>
- Having initialised the device you can then call
- <function>z8530_init</function>. This will probe the chip and
- reset it into a known state. An identification sequence is then
- run to identify the chip type. If the checks fail to pass the
- function returns a non zero error code. Typically this indicates
- that the port given is not valid. After this call the
- type field of the z8530_dev structure is initialised to either
- Z8530, Z85C30 or Z85230 according to the chip found.
- </para>
- <para>
- Once you have called z8530_init you can also make use of the utility
- function <function>z8530_describe</function>. This provides a
- consistent reporting format for the Z8530 devices, and allows all
- the drivers to provide consistent reporting.
- </para>
- </chapter>
-
- <chapter id="Attaching_Network_Interfaces">
- <title>Attaching Network Interfaces</title>
- <para>
- If you wish to use the network interface facilities of the driver,
- then you need to attach a network device to each channel that is
- present and in use. In addition to use the generic HDLC
- you need to follow some additional plumbing rules. They may seem
- complex but a look at the example hostess_sv11 driver should
- reassure you.
- </para>
- <para>
- The network device used for each channel should be pointed to by
- the netdevice field of each channel. The hdlc-> priv field of the
- network device points to your private data - you will need to be
- able to find your private data from this.
- </para>
- <para>
- The way most drivers approach this particular problem is to
- create a structure holding the Z8530 device definition and
- put that into the private field of the network device. The
- network device fields of the channels then point back to the
- network devices.
- </para>
- <para>
- If you wish to use the generic HDLC then you need to register
- the HDLC device.
- </para>
- <para>
- Before you register your network device you will also need to
- provide suitable handlers for most of the network device callbacks.
- See the network device documentation for more details on this.
- </para>
- </chapter>
-
- <chapter id="Configuring_And_Activating_The_Port">
- <title>Configuring And Activating The Port</title>
- <para>
- The Z85230 driver provides helper functions and tables to load the
- port registers on the Z8530 chips. When programming the register
- settings for a channel be aware that the documentation recommends
- initialisation orders. Strange things happen when these are not
- followed.
- </para>
- <para>
- <function>z8530_channel_load</function> takes an array of
- pairs of initialisation values in an array of u8 type. The first
- value is the Z8530 register number. Add 16 to indicate the alternate
- register bank on the later chips. The array is terminated by a 255.
- </para>
- <para>
- The driver provides a pair of public tables. The
- z8530_hdlc_kilostream table is for the UK 'Kilostream' service and
- also happens to cover most other end host configurations. The
- z8530_hdlc_kilostream_85230 table is the same configuration using
- the enhancements of the 85230 chip. The configuration loaded is
- standard NRZ encoded synchronous data with HDLC bitstuffing. All
- of the timing is taken from the other end of the link.
- </para>
- <para>
- When writing your own tables be aware that the driver internally
- tracks register values. It may need to reload values. You should
- therefore be sure to set registers 1-7, 9-11, 14 and 15 in all
- configurations. Where the register settings depend on DMA selection
- the driver will update the bits itself when you open or close.
- Loading a new table with the interface open is not recommended.
- </para>
- <para>
- There are three standard configurations supported by the core
- code. In PIO mode the interface is programmed up to use
- interrupt driven PIO. This places high demands on the host processor
- to avoid latency. The driver is written to take account of latency
- issues but it cannot avoid latencies caused by other drivers,
- notably IDE in PIO mode. Because the drivers allocate buffers you
- must also prevent MTU changes while the port is open.
- </para>
- <para>
- Once the port is open it will call the rx_function of each channel
- whenever a completed packet arrived. This is invoked from
- interrupt context and passes you the channel and a network
- buffer (struct sk_buff) holding the data. The data includes
- the CRC bytes so most users will want to trim the last two
- bytes before processing the data. This function is very timing
- critical. When you wish to simply discard data the support
- code provides the function <function>z8530_null_rx</function>
- to discard the data.
- </para>
- <para>
- To active PIO mode sending and receiving the <function>
- z8530_sync_open</function> is called. This expects to be passed
- the network device and the channel. Typically this is called from
- your network device open callback. On a failure a non zero error
- status is returned. The <function>z8530_sync_close</function>
- function shuts down a PIO channel. This must be done before the
- channel is opened again and before the driver shuts down
- and unloads.
- </para>
- <para>
- The ideal mode of operation is dual channel DMA mode. Here the
- kernel driver will configure the board for DMA in both directions.
- The driver also handles ISA DMA issues such as controller
- programming and the memory range limit for you. This mode is
- activated by calling the <function>z8530_sync_dma_open</function>
- function. On failure a non zero error value is returned.
- Once this mode is activated it can be shut down by calling the
- <function>z8530_sync_dma_close</function>. You must call the close
- function matching the open mode you used.
- </para>
- <para>
- The final supported mode uses a single DMA channel to drive the
- transmit side. As the Z85C30 has a larger FIFO on the receive
- channel this tends to increase the maximum speed a little.
- This is activated by calling the <function>z8530_sync_txdma_open
- </function>. This returns a non zero error code on failure. The
- <function>z8530_sync_txdma_close</function> function closes down
- the Z8530 interface from this mode.
- </para>
- </chapter>
-
- <chapter id="Network_Layer_Functions">
- <title>Network Layer Functions</title>
- <para>
- The Z8530 layer provides functions to queue packets for
- transmission. The driver internally buffers the frame currently
- being transmitted and one further frame (in order to keep back
- to back transmission running). Any further buffering is up to
- the caller.
- </para>
- <para>
- The function <function>z8530_queue_xmit</function> takes a network
- buffer in sk_buff format and queues it for transmission. The
- caller must provide the entire packet with the exception of the
- bitstuffing and CRC. This is normally done by the caller via
- the generic HDLC interface layer. It returns 0 if the buffer has been
- queued and non zero values for queue full. If the function accepts
- the buffer it becomes property of the Z8530 layer and the caller
- should not free it.
- </para>
- <para>
- The function <function>z8530_get_stats</function> returns a pointer
- to an internally maintained per interface statistics block. This
- provides most of the interface code needed to implement the network
- layer get_stats callback.
- </para>
- </chapter>
-
- <chapter id="Porting_The_Z8530_Driver">
- <title>Porting The Z8530 Driver</title>
- <para>
- The Z8530 driver is written to be portable. In DMA mode it makes
- assumptions about the use of ISA DMA. These are probably warranted
- in most cases as the Z85230 in particular was designed to glue to PC
- type machines. The PIO mode makes no real assumptions.
- </para>
- <para>
- Should you need to retarget the Z8530 driver to another architecture
- the only code that should need changing are the port I/O functions.
- At the moment these assume PC I/O port accesses. This may not be
- appropriate for all platforms. Replacing
- <function>z8530_read_port</function> and <function>z8530_write_port
- </function> is intended to be all that is required to port this
- driver layer.
- </para>
- </chapter>
-
- <chapter id="bugs">
- <title>Known Bugs And Assumptions</title>
- <para>
- <variablelist>
- <varlistentry><term>Interrupt Locking</term>
- <listitem>
- <para>
- The locking in the driver is done via the global cli/sti lock. This
- makes for relatively poor SMP performance. Switching this to use a
- per device spin lock would probably materially improve performance.
- </para>
- </listitem></varlistentry>
-
- <varlistentry><term>Occasional Failures</term>
- <listitem>
- <para>
- We have reports of occasional failures when run for very long
- periods of time and the driver starts to receive junk frames. At
- the moment the cause of this is not clear.
- </para>
- </listitem></varlistentry>
- </variablelist>
-
- </para>
- </chapter>
-
- <chapter id="pubfunctions">
- <title>Public Functions Provided</title>
-!Edrivers/net/wan/z85230.c
- </chapter>
-
- <chapter id="intfunctions">
- <title>Internal Functions</title>
-!Idrivers/net/wan/z85230.c
- </chapter>
-
-</book>
4) No need to implement irq_domain_ops.map and irq_domain_ops.unmap,
they are unused with hierarchy irq_domain.
-Hierarchy irq_domain may also be used to support other architectures,
-such as ARM, ARM64 etc.
+Hierarchy irq_domain is in no way x86 specific, and is heavily used to
+support other architectures, such as ARM, ARM64 etc.
+
+=== Debugging ===
+
+If you switch on CONFIG_IRQ_DOMAIN_DEBUG (which depends on
+CONFIG_IRQ_DOMAIN and CONFIG_DEBUG_FS), you will find a new file in
+your debugfs mount point, called irq_domain_mapping. This file
+contains a live snapshot of all the IRQ domains in the system:
+
+ name mapped linear-max direct-max devtree-node
+ pl061 8 8 0 /smb/gpio@e0080000
+ pl061 8 8 0 /smb/gpio@e1050000
+ pMSI 0 0 0 /interrupt-controller@e1101000/v2m@e0080000
+ MSI 37 0 0 /interrupt-controller@e1101000/v2m@e0080000
+ GICv2m 37 0 0 /interrupt-controller@e1101000/v2m@e0080000
+ GICv2 448 448 0 /interrupt-controller@e1101000
+
+it also iterates over the interrupts to display their mapping in the
+domains, and makes the domain stacking visible:
+
+
+irq hwirq chip name chip data active type domain
+ 1 0x00019 GICv2 0xffff00000916bfd8 * LINEAR GICv2
+ 2 0x0001d GICv2 0xffff00000916bfd8 LINEAR GICv2
+ 3 0x0001e GICv2 0xffff00000916bfd8 * LINEAR GICv2
+ 4 0x0001b GICv2 0xffff00000916bfd8 * LINEAR GICv2
+ 5 0x0001a GICv2 0xffff00000916bfd8 LINEAR GICv2
+[...]
+ 96 0x81808 MSI 0x (null) RADIX MSI
+ 96+ 0x00063 GICv2m 0xffff8003ee116980 RADIX GICv2m
+ 96+ 0x00063 GICv2 0xffff00000916bfd8 LINEAR GICv2
+ 97 0x08800 MSI 0x (null) * RADIX MSI
+ 97+ 0x00064 GICv2m 0xffff8003ee116980 * RADIX GICv2m
+ 97+ 0x00064 GICv2 0xffff00000916bfd8 * LINEAR GICv2
+
+Here, interrupts 1-5 are only using a single domain, while 96 and 97
+are build out of a stack of three domain, each level performing a
+particular function.
+# -*- makefile -*-
+# Makefile for Sphinx documentation
+#
+
subdir-y :=
+
+# You can set these variables from the command line.
+SPHINXBUILD = sphinx-build
+SPHINXOPTS =
+SPHINXDIRS = .
+_SPHINXDIRS = $(patsubst $(srctree)/Documentation/%/conf.py,%,$(wildcard $(srctree)/Documentation/*/conf.py))
+SPHINX_CONF = conf.py
+PAPER =
+BUILDDIR = $(obj)/output
+PDFLATEX = xelatex
+LATEXOPTS = -interaction=batchmode
+
+# User-friendly check for sphinx-build
+HAVE_SPHINX := $(shell if which $(SPHINXBUILD) >/dev/null 2>&1; then echo 1; else echo 0; fi)
+
+ifeq ($(HAVE_SPHINX),0)
+
+.DEFAULT:
+ $(warning The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed and in PATH, or set the SPHINXBUILD make variable to point to the full path of the '$(SPHINXBUILD)' executable.)
+ @echo " SKIP Sphinx $@ target."
+
+else # HAVE_SPHINX
+
+# User-friendly check for pdflatex
+HAVE_PDFLATEX := $(shell if which $(PDFLATEX) >/dev/null 2>&1; then echo 1; else echo 0; fi)
+
+# Internal variables.
+PAPEROPT_a4 = -D latex_paper_size=a4
+PAPEROPT_letter = -D latex_paper_size=letter
+KERNELDOC = $(srctree)/scripts/kernel-doc
+KERNELDOC_CONF = -D kerneldoc_srctree=$(srctree) -D kerneldoc_bin=$(KERNELDOC)
+ALLSPHINXOPTS = $(KERNELDOC_CONF) $(PAPEROPT_$(PAPER)) $(SPHINXOPTS)
+# the i18n builder cannot share the environment and doctrees with the others
+I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
+
+# commands; the 'cmd' from scripts/Kbuild.include is not *loopable*
+loop_cmd = $(echo-cmd) $(cmd_$(1)) || exit;
+
+# $2 sphinx builder e.g. "html"
+# $3 name of the build subfolder / e.g. "media", used as:
+# * dest folder relative to $(BUILDDIR) and
+# * cache folder relative to $(BUILDDIR)/.doctrees
+# $4 dest subfolder e.g. "man" for man pages at media/man
+# $5 reST source folder relative to $(srctree)/$(src),
+# e.g. "media" for the linux-tv book-set at ./Documentation/media
+
+quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
+ cmd_sphinx = $(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=Documentation/media $2 && \
+ PYTHONDONTWRITEBYTECODE=1 \
+ BUILDDIR=$(abspath $(BUILDDIR)) SPHINX_CONF=$(abspath $(srctree)/$(src)/$5/$(SPHINX_CONF)) \
+ $(SPHINXBUILD) \
+ -b $2 \
+ -c $(abspath $(srctree)/$(src)) \
+ -d $(abspath $(BUILDDIR)/.doctrees/$3) \
+ -D version=$(KERNELVERSION) -D release=$(KERNELRELEASE) \
+ $(ALLSPHINXOPTS) \
+ $(abspath $(srctree)/$(src)/$5) \
+ $(abspath $(BUILDDIR)/$3/$4)
+
+htmldocs:
+ @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
+
+linkcheckdocs:
+ @$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,linkcheck,$(var),,$(var)))
+
+latexdocs:
+ @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$(var)))
+
+ifeq ($(HAVE_PDFLATEX),0)
+
+pdfdocs:
+ $(warning The '$(PDFLATEX)' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
+ @echo " SKIP Sphinx $@ target."
+
+else # HAVE_PDFLATEX
+
+pdfdocs: latexdocs
+ $(foreach var,$(SPHINXDIRS), $(MAKE) PDFLATEX=$(PDFLATEX) LATEXOPTS="$(LATEXOPTS)" -C $(BUILDDIR)/$(var)/latex || exit;)
+
+endif # HAVE_PDFLATEX
+
+epubdocs:
+ @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var)))
+
+xmldocs:
+ @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var)))
+
+endif # HAVE_SPHINX
+
+# The following targets are independent of HAVE_SPHINX, and the rules should
+# work or silently pass without Sphinx.
+
+# no-ops for the Sphinx toolchain
+sgmldocs:
+ @:
+psdocs:
+ @:
+mandocs:
+ @:
+installmandocs:
+ @:
+
+cleandocs:
+ $(Q)rm -rf $(BUILDDIR)
+ $(Q)$(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=Documentation/media clean
+
+dochelp:
+ @echo ' Linux kernel internal documentation in different formats from ReST:'
+ @echo ' htmldocs - HTML'
+ @echo ' latexdocs - LaTeX'
+ @echo ' pdfdocs - PDF'
+ @echo ' epubdocs - EPUB'
+ @echo ' xmldocs - XML'
+ @echo ' linkcheckdocs - check for broken external links (will connect to external hosts)'
+ @echo ' cleandocs - clean all generated files'
+ @echo
+ @echo ' make SPHINXDIRS="s1 s2" [target] Generate only docs of folder s1, s2'
+ @echo ' valid values for SPHINXDIRS are: $(_SPHINXDIRS)'
+ @echo
+ @echo ' make SPHINX_CONF={conf-file} [target] use *additional* sphinx-build'
+ @echo ' configuration. This is e.g. useful to build with nit-picking config.'
+++ /dev/null
-# -*- makefile -*-
-# Makefile for Sphinx documentation
-#
-
-# You can set these variables from the command line.
-SPHINXBUILD = sphinx-build
-SPHINXOPTS =
-SPHINXDIRS = .
-_SPHINXDIRS = $(patsubst $(srctree)/Documentation/%/conf.py,%,$(wildcard $(srctree)/Documentation/*/conf.py))
-SPHINX_CONF = conf.py
-PAPER =
-BUILDDIR = $(obj)/output
-PDFLATEX = xelatex
-LATEXOPTS = -interaction=batchmode
-
-# User-friendly check for sphinx-build
-HAVE_SPHINX := $(shell if which $(SPHINXBUILD) >/dev/null 2>&1; then echo 1; else echo 0; fi)
-
-ifeq ($(HAVE_SPHINX),0)
-
-.DEFAULT:
- $(warning The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed and in PATH, or set the SPHINXBUILD make variable to point to the full path of the '$(SPHINXBUILD)' executable.)
- @echo " SKIP Sphinx $@ target."
-
-else ifneq ($(DOCBOOKS),)
-
-# Skip Sphinx build if the user explicitly requested DOCBOOKS.
-.DEFAULT:
- @echo " SKIP Sphinx $@ target (DOCBOOKS specified)."
-
-else # HAVE_SPHINX
-
-# User-friendly check for pdflatex
-HAVE_PDFLATEX := $(shell if which $(PDFLATEX) >/dev/null 2>&1; then echo 1; else echo 0; fi)
-
-# Internal variables.
-PAPEROPT_a4 = -D latex_paper_size=a4
-PAPEROPT_letter = -D latex_paper_size=letter
-KERNELDOC = $(srctree)/scripts/kernel-doc
-KERNELDOC_CONF = -D kerneldoc_srctree=$(srctree) -D kerneldoc_bin=$(KERNELDOC)
-ALLSPHINXOPTS = $(KERNELDOC_CONF) $(PAPEROPT_$(PAPER)) $(SPHINXOPTS)
-# the i18n builder cannot share the environment and doctrees with the others
-I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
-
-# commands; the 'cmd' from scripts/Kbuild.include is not *loopable*
-loop_cmd = $(echo-cmd) $(cmd_$(1)) || exit;
-
-# $2 sphinx builder e.g. "html"
-# $3 name of the build subfolder / e.g. "media", used as:
-# * dest folder relative to $(BUILDDIR) and
-# * cache folder relative to $(BUILDDIR)/.doctrees
-# $4 dest subfolder e.g. "man" for man pages at media/man
-# $5 reST source folder relative to $(srctree)/$(src),
-# e.g. "media" for the linux-tv book-set at ./Documentation/media
-
-quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
- cmd_sphinx = $(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=Documentation/media $2 && \
- PYTHONDONTWRITEBYTECODE=1 \
- BUILDDIR=$(abspath $(BUILDDIR)) SPHINX_CONF=$(abspath $(srctree)/$(src)/$5/$(SPHINX_CONF)) \
- $(SPHINXBUILD) \
- -b $2 \
- -c $(abspath $(srctree)/$(src)) \
- -d $(abspath $(BUILDDIR)/.doctrees/$3) \
- -D version=$(KERNELVERSION) -D release=$(KERNELRELEASE) \
- $(ALLSPHINXOPTS) \
- $(abspath $(srctree)/$(src)/$5) \
- $(abspath $(BUILDDIR)/$3/$4)
-
-htmldocs:
- @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
-
-linkcheckdocs:
- @$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,linkcheck,$(var),,$(var)))
-
-latexdocs:
- @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$(var)))
-
-ifeq ($(HAVE_PDFLATEX),0)
-
-pdfdocs:
- $(warning The '$(PDFLATEX)' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
- @echo " SKIP Sphinx $@ target."
-
-else # HAVE_PDFLATEX
-
-pdfdocs: latexdocs
- $(foreach var,$(SPHINXDIRS), $(MAKE) PDFLATEX=$(PDFLATEX) LATEXOPTS="$(LATEXOPTS)" -C $(BUILDDIR)/$(var)/latex || exit;)
-
-endif # HAVE_PDFLATEX
-
-epubdocs:
- @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var)))
-
-xmldocs:
- @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var)))
-
-endif # HAVE_SPHINX
-
-# The following targets are independent of HAVE_SPHINX, and the rules should
-# work or silently pass without Sphinx.
-
-# no-ops for the Sphinx toolchain
-sgmldocs:
- @:
-psdocs:
- @:
-mandocs:
- @:
-installmandocs:
- @:
-
-cleandocs:
- $(Q)rm -rf $(BUILDDIR)
- $(Q)$(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=Documentation/media clean
-
-dochelp:
- @echo ' Linux kernel internal documentation in different formats (Sphinx):'
- @echo ' htmldocs - HTML'
- @echo ' latexdocs - LaTeX'
- @echo ' pdfdocs - PDF'
- @echo ' epubdocs - EPUB'
- @echo ' xmldocs - XML'
- @echo ' linkcheckdocs - check for broken external links (will connect to external hosts)'
- @echo ' cleandocs - clean all generated files'
- @echo
- @echo ' make SPHINXDIRS="s1 s2" [target] Generate only docs of folder s1, s2'
- @echo ' valid values for SPHINXDIRS are: $(_SPHINXDIRS)'
- @echo
- @echo ' make SPHINX_CONF={conf-file} [target] use *additional* sphinx-build'
- @echo ' configuration. This is e.g. useful to build with nit-picking config.'
a different interrupt, the driver will deadlock trying to recursively
acquire the spinlock. Such deadlocks can be avoided by using
spin_lock_irqsave() or spin_lock_irq() which disable local interrupts
-and acquire the lock (see Documentation/DocBook/kernel-locking).
+and acquire the lock (see Documentation/kernel-hacking/locking.rst).
4.5 How to tell whether MSI/MSI-X is enabled on a device
- RCU CPU stall warnings (module parameter rcu_cpu_stall_suppress)
torture.txt
- RCU Torture Test Operation (CONFIG_RCU_TORTURE_TEST)
-trace.txt
- - CONFIG_RCU_TRACE debugfs files and formats
UP.txt
- RCU on Uniprocessor Systems
whatisRCU.txt
For <tt>remove_gp_synchronous()</tt>, as long as all modifications
to <tt>gp</tt> are carried out while holding <tt>gp_lock</tt>,
the above optimizations are harmless.
- However,
- with <tt>CONFIG_SPARSE_RCU_POINTER=y</tt>,
- <tt>sparse</tt> will complain if you
+ However, <tt>sparse</tt> will complain if you
define <tt>gp</tt> with <tt>__rcu</tt> and then
access it without using
either <tt>rcu_access_pointer()</tt> or <tt>rcu_dereference()</tt>.
If the nesting is not visible to the compiler, as is the case with
mutually recursive functions each in its own translation unit,
stack overflow will result.
-If the nesting takes the form of loops, either the control variable
+If the nesting takes the form of loops, perhaps in the guise of tail
+recursion, either the control variable
will overflow or (in the Linux kernel) you will get an RCU CPU stall warning.
Nevertheless, this class of RCU implementations is one
of the most composable constructs in existence.
and <tt>rcu_dereference()</tt>, perhaps (incorrectly)
substituting a simple assignment.
To catch this sort of error, a given RCU-protected pointer may be
- tagged with <tt>__rcu</tt>, after which running sparse
- with <tt>CONFIG_SPARSE_RCU_POINTER=y</tt> will complain
- about simple-assignment accesses to that pointer.
+ tagged with <tt>__rcu</tt>, after which sparse
+ will complain about simple-assignment accesses to that pointer.
Arnd Bergmann made me aware of this requirement, and also
supplied the needed
<a href="https://lwn.net/Articles/376011/">patch series</a>.
some other synchronization mechanism, for example, reference
counting.
<li> In kernels built with <tt>CONFIG_RCU_TRACE=y</tt>, RCU-related
- information is provided via both debugfs and event tracing.
+ information is provided via event tracing.
<li> Open-coded use of <tt>rcu_assign_pointer()</tt> and
<tt>rcu_dereference()</tt> to create typical linked
data structures can be surprisingly error-prone.
<tt>nohz_full</tt> CPU running in userspace.
RCU must therefore track <tt>nohz_full</tt> userspace
execution.
-And in
-<a href="https://lwn.net/Articles/558284/"><tt>CONFIG_NO_HZ_FULL_SYSIDLE=y</tt></a>
-kernels, RCU must separately track idle CPUs on the one hand and
-CPUs that are either idle or executing in userspace on the other.
-In both cases, RCU must be able to sample state at two points in
+RCU must therefore be able to sample state at two points in
time, and be able to determine whether or not some other CPU spent
any time idle and/or executing in userspace.
to whether or not a CPU is online, which means that <tt>srcu_barrier()</tt>
need not exclude CPU-hotplug operations.
+<p>
+SRCU also differs from other RCU flavors in that SRCU's expedited and
+non-expedited grace periods are implemented by the same mechanism.
+This means that in the current SRCU implementation, expediting a
+future grace period has the side effect of expediting all prior
+grace periods that have not yet completed.
+(But please note that this is a property of the current implementation,
+not necessarily of future implementations.)
+In addition, if SRCU has been idle for longer than the interval
+specified by the <tt>srcutree.exp_holdoff</tt> kernel boot parameter
+(25 microseconds by default),
+and if a <tt>synchronize_srcu()</tt> invocation ends this idle period,
+that invocation will be automatically expedited.
+
<p>
As of v4.12, SRCU's callbacks are maintained per-CPU, eliminating
a locking bottleneck present in prior kernel versions.
read-side critical sections. It is the responsibility of the
RCU update-side primitives to deal with this.
-17. Use CONFIG_PROVE_RCU, CONFIG_DEBUG_OBJECTS_RCU_HEAD, and the
- __rcu sparse checks (enabled by CONFIG_SPARSE_RCU_POINTER) to
- validate your RCU code. These can help find problems as follows:
+17. Use CONFIG_PROVE_LOCKING, CONFIG_DEBUG_OBJECTS_RCU_HEAD, and the
+ __rcu sparse checks to validate your RCU code. These can help
+ find problems as follows:
- CONFIG_PROVE_RCU: check that accesses to RCU-protected data
+ CONFIG_PROVE_LOCKING: check that accesses to RCU-protected data
structures are carried out under the proper RCU
read-side critical section, while holding the right
combination of locks, or whatever other conditions
+++ /dev/null
-CONFIG_RCU_TRACE debugfs Files and Formats
-
-
-The rcutree and rcutiny implementations of RCU provide debugfs trace
-output that summarizes counters and state. This information is useful for
-debugging RCU itself, and can sometimes also help to debug abuses of RCU.
-The following sections describe the debugfs files and formats, first
-for rcutree and next for rcutiny.
-
-
-CONFIG_TREE_RCU and CONFIG_PREEMPT_RCU debugfs Files and Formats
-
-These implementations of RCU provide several debugfs directories under the
-top-level directory "rcu":
-
-rcu/rcu_bh
-rcu/rcu_preempt
-rcu/rcu_sched
-
-Each directory contains files for the corresponding flavor of RCU.
-Note that rcu/rcu_preempt is only present for CONFIG_PREEMPT_RCU.
-For CONFIG_TREE_RCU, the RCU flavor maps onto the RCU-sched flavor,
-so that activity for both appears in rcu/rcu_sched.
-
-In addition, the following file appears in the top-level directory:
-rcu/rcutorture. This file displays rcutorture test progress. The output
-of "cat rcu/rcutorture" looks as follows:
-
-rcutorture test sequence: 0 (test in progress)
-rcutorture update version number: 615
-
-The first line shows the number of rcutorture tests that have completed
-since boot. If a test is currently running, the "(test in progress)"
-string will appear as shown above. The second line shows the number of
-update cycles that the current test has started, or zero if there is
-no test in progress.
-
-
-Within each flavor directory (rcu/rcu_bh, rcu/rcu_sched, and possibly
-also rcu/rcu_preempt) the following files will be present:
-
-rcudata:
- Displays fields in struct rcu_data.
-rcuexp:
- Displays statistics for expedited grace periods.
-rcugp:
- Displays grace-period counters.
-rcuhier:
- Displays the struct rcu_node hierarchy.
-rcu_pending:
- Displays counts of the reasons rcu_pending() decided that RCU had
- work to do.
-rcuboost:
- Displays RCU boosting statistics. Only present if
- CONFIG_RCU_BOOST=y.
-
-The output of "cat rcu/rcu_preempt/rcudata" looks as follows:
-
- 0!c=30455 g=30456 cnq=1/0:1 dt=126535/140000000000000/0 df=2002 of=4 ql=0/0 qs=N... b=10 ci=74572 nci=0 co=1131 ca=716
- 1!c=30719 g=30720 cnq=1/0:0 dt=132007/140000000000000/0 df=1874 of=10 ql=0/0 qs=N... b=10 ci=123209 nci=0 co=685 ca=982
- 2!c=30150 g=30151 cnq=1/1:1 dt=138537/140000000000000/0 df=1707 of=8 ql=0/0 qs=N... b=10 ci=80132 nci=0 co=1328 ca=1458
- 3 c=31249 g=31250 cnq=1/1:0 dt=107255/140000000000000/0 df=1749 of=6 ql=0/450 qs=NRW. b=10 ci=151700 nci=0 co=509 ca=622
- 4!c=29502 g=29503 cnq=1/0:1 dt=83647/140000000000000/0 df=965 of=5 ql=0/0 qs=N... b=10 ci=65643 nci=0 co=1373 ca=1521
- 5 c=31201 g=31202 cnq=1/0:1 dt=70422/0/0 df=535 of=7 ql=0/0 qs=.... b=10 ci=58500 nci=0 co=764 ca=698
- 6!c=30253 g=30254 cnq=1/0:1 dt=95363/140000000000000/0 df=780 of=5 ql=0/0 qs=N... b=10 ci=100607 nci=0 co=1414 ca=1353
- 7 c=31178 g=31178 cnq=1/0:0 dt=91536/0/0 df=547 of=4 ql=0/0 qs=.... b=10 ci=109819 nci=0 co=1115 ca=969
-
-This file has one line per CPU, or eight for this 8-CPU system.
-The fields are as follows:
-
-o The number at the beginning of each line is the CPU number.
- CPUs numbers followed by an exclamation mark are offline,
- but have been online at least once since boot. There will be
- no output for CPUs that have never been online, which can be
- a good thing in the surprisingly common case where NR_CPUS is
- substantially larger than the number of actual CPUs.
-
-o "c" is the count of grace periods that this CPU believes have
- completed. Offlined CPUs and CPUs in dynticks idle mode may lag
- quite a ways behind, for example, CPU 4 under "rcu_sched" above,
- which has been offline through 16 RCU grace periods. It is not
- unusual to see offline CPUs lagging by thousands of grace periods.
- Note that although the grace-period number is an unsigned long,
- it is printed out as a signed long to allow more human-friendly
- representation near boot time.
-
-o "g" is the count of grace periods that this CPU believes have
- started. Again, offlined CPUs and CPUs in dynticks idle mode
- may lag behind. If the "c" and "g" values are equal, this CPU
- has already reported a quiescent state for the last RCU grace
- period that it is aware of, otherwise, the CPU believes that it
- owes RCU a quiescent state.
-
-o "pq" indicates that this CPU has passed through a quiescent state
- for the current grace period. It is possible for "pq" to be
- "1" and "c" different than "g", which indicates that although
- the CPU has passed through a quiescent state, either (1) this
- CPU has not yet reported that fact, (2) some other CPU has not
- yet reported for this grace period, or (3) both.
-
-o "qp" indicates that RCU still expects a quiescent state from
- this CPU. Offlined CPUs and CPUs in dyntick idle mode might
- well have qp=1, which is OK: RCU is still ignoring them.
-
-o "dt" is the current value of the dyntick counter that is incremented
- when entering or leaving idle, either due to a context switch or
- due to an interrupt. This number is even if the CPU is in idle
- from RCU's viewpoint and odd otherwise. The number after the
- first "/" is the interrupt nesting depth when in idle state,
- or a large number added to the interrupt-nesting depth when
- running a non-idle task. Some architectures do not accurately
- count interrupt nesting when running in non-idle kernel context,
- which can result in interesting anomalies such as negative
- interrupt-nesting levels. The number after the second "/"
- is the NMI nesting depth.
-
-o "df" is the number of times that some other CPU has forced a
- quiescent state on behalf of this CPU due to this CPU being in
- idle state.
-
-o "of" is the number of times that some other CPU has forced a
- quiescent state on behalf of this CPU due to this CPU being
- offline. In a perfect world, this might never happen, but it
- turns out that offlining and onlining a CPU can take several grace
- periods, and so there is likely to be an extended period of time
- when RCU believes that the CPU is online when it really is not.
- Please note that erring in the other direction (RCU believing a
- CPU is offline when it is really alive and kicking) is a fatal
- error, so it makes sense to err conservatively.
-
-o "ql" is the number of RCU callbacks currently residing on
- this CPU. The first number is the number of "lazy" callbacks
- that are known to RCU to only be freeing memory, and the number
- after the "/" is the total number of callbacks, lazy or not.
- These counters count callbacks regardless of what phase of
- grace-period processing that they are in (new, waiting for
- grace period to start, waiting for grace period to end, ready
- to invoke).
-
-o "qs" gives an indication of the state of the callback queue
- with four characters:
-
- "N" Indicates that there are callbacks queued that are not
- ready to be handled by the next grace period, and thus
- will be handled by the grace period following the next
- one.
-
- "R" Indicates that there are callbacks queued that are
- ready to be handled by the next grace period.
-
- "W" Indicates that there are callbacks queued that are
- waiting on the current grace period.
-
- "D" Indicates that there are callbacks queued that have
- already been handled by a prior grace period, and are
- thus waiting to be invoked. Note that callbacks in
- the process of being invoked are not counted here.
- Callbacks in the process of being invoked are those
- that have been removed from the rcu_data structures
- queues by rcu_do_batch(), but which have not yet been
- invoked.
-
- If there are no callbacks in a given one of the above states,
- the corresponding character is replaced by ".".
-
-o "b" is the batch limit for this CPU. If more than this number
- of RCU callbacks is ready to invoke, then the remainder will
- be deferred.
-
-o "ci" is the number of RCU callbacks that have been invoked for
- this CPU. Note that ci+nci+ql is the number of callbacks that have
- been registered in absence of CPU-hotplug activity.
-
-o "nci" is the number of RCU callbacks that have been offloaded from
- this CPU. This will always be zero unless the kernel was built
- with CONFIG_RCU_NOCB_CPU=y and the "rcu_nocbs=" kernel boot
- parameter was specified.
-
-o "co" is the number of RCU callbacks that have been orphaned due to
- this CPU going offline. These orphaned callbacks have been moved
- to an arbitrarily chosen online CPU.
-
-o "ca" is the number of RCU callbacks that have been adopted by this
- CPU due to other CPUs going offline. Note that ci+co-ca+ql is
- the number of RCU callbacks registered on this CPU.
-
-
-Kernels compiled with CONFIG_RCU_BOOST=y display the following from
-/debug/rcu/rcu_preempt/rcudata:
-
- 0!c=12865 g=12866 cnq=1/0:1 dt=83113/140000000000000/0 df=288 of=11 ql=0/0 qs=N... kt=0/O ktl=944 b=10 ci=60709 nci=0 co=748 ca=871
- 1 c=14407 g=14408 cnq=1/0:0 dt=100679/140000000000000/0 df=378 of=7 ql=0/119 qs=NRW. kt=0/W ktl=9b6 b=10 ci=109740 nci=0 co=589 ca=485
- 2 c=14407 g=14408 cnq=1/0:0 dt=105486/0/0 df=90 of=9 ql=0/89 qs=NRW. kt=0/W ktl=c0c b=10 ci=83113 nci=0 co=533 ca=490
- 3 c=14407 g=14408 cnq=1/0:0 dt=107138/0/0 df=142 of=8 ql=0/188 qs=NRW. kt=0/W ktl=b96 b=10 ci=121114 nci=0 co=426 ca=290
- 4 c=14405 g=14406 cnq=1/0:1 dt=50238/0/0 df=706 of=7 ql=0/0 qs=.... kt=0/W ktl=812 b=10 ci=34929 nci=0 co=643 ca=114
- 5!c=14168 g=14169 cnq=1/0:0 dt=45465/140000000000000/0 df=161 of=11 ql=0/0 qs=N... kt=0/O ktl=b4d b=10 ci=47712 nci=0 co=677 ca=722
- 6 c=14404 g=14405 cnq=1/0:0 dt=59454/0/0 df=94 of=6 ql=0/0 qs=.... kt=0/W ktl=e57 b=10 ci=55597 nci=0 co=701 ca=811
- 7 c=14407 g=14408 cnq=1/0:1 dt=68850/0/0 df=31 of=8 ql=0/0 qs=.... kt=0/W ktl=14bd b=10 ci=77475 nci=0 co=508 ca=1042
-
-This is similar to the output discussed above, but contains the following
-additional fields:
-
-o "kt" is the per-CPU kernel-thread state. The digit preceding
- the first slash is zero if there is no work pending and 1
- otherwise. The character between the first pair of slashes is
- as follows:
-
- "S" The kernel thread is stopped, in other words, all
- CPUs corresponding to this rcu_node structure are
- offline.
-
- "R" The kernel thread is running.
-
- "W" The kernel thread is waiting because there is no work
- for it to do.
-
- "O" The kernel thread is waiting because it has been
- forced off of its designated CPU or because its
- ->cpus_allowed mask permits it to run on other than
- its designated CPU.
-
- "Y" The kernel thread is yielding to avoid hogging CPU.
-
- "?" Unknown value, indicates a bug.
-
- The number after the final slash is the CPU that the kthread
- is actually running on.
-
- This field is displayed only for CONFIG_RCU_BOOST kernels.
-
-o "ktl" is the low-order 16 bits (in hexadecimal) of the count of
- the number of times that this CPU's per-CPU kthread has gone
- through its loop servicing invoke_rcu_cpu_kthread() requests.
-
- This field is displayed only for CONFIG_RCU_BOOST kernels.
-
-
-The output of "cat rcu/rcu_preempt/rcuexp" looks as follows:
-
-s=21872 wd1=0 wd2=0 wd3=5 enq=0 sc=21872
-
-These fields are as follows:
-
-o "s" is the sequence number, with an odd number indicating that
- an expedited grace period is in progress.
-
-o "wd1", "wd2", and "wd3" are the number of times that an attempt
- to start an expedited grace period found that someone else had
- completed an expedited grace period that satisfies the attempted
- request. "Our work is done."
-
-o "enq" is the number of quiescent states still outstanding.
-
-o "sc" is the number of times that the attempt to start a
- new expedited grace period succeeded.
-
-
-The output of "cat rcu/rcu_preempt/rcugp" looks as follows:
-
-completed=31249 gpnum=31250 age=1 max=18
-
-These fields are taken from the rcu_state structure, and are as follows:
-
-o "completed" is the number of grace periods that have completed.
- It is comparable to the "c" field from rcu/rcudata in that a
- CPU whose "c" field matches the value of "completed" is aware
- that the corresponding RCU grace period has completed.
-
-o "gpnum" is the number of grace periods that have started. It is
- similarly comparable to the "g" field from rcu/rcudata in that
- a CPU whose "g" field matches the value of "gpnum" is aware that
- the corresponding RCU grace period has started.
-
- If these two fields are equal, then there is no grace period
- in progress, in other words, RCU is idle. On the other hand,
- if the two fields differ (as they are above), then an RCU grace
- period is in progress.
-
-o "age" is the number of jiffies that the current grace period
- has extended for, or zero if there is no grace period currently
- in effect.
-
-o "max" is the age in jiffies of the longest-duration grace period
- thus far.
-
-The output of "cat rcu/rcu_preempt/rcuhier" looks as follows:
-
-c=14407 g=14408 s=0 jfq=2 j=c863 nfqs=12040/nfqsng=0(12040) fqlh=1051 oqlen=0/0
-3/3 ..>. 0:7 ^0
-e/e ..>. 0:3 ^0 d/d ..>. 4:7 ^1
-
-The fields are as follows:
-
-o "c" is exactly the same as "completed" under rcu/rcu_preempt/rcugp.
-
-o "g" is exactly the same as "gpnum" under rcu/rcu_preempt/rcugp.
-
-o "s" is the current state of the force_quiescent_state()
- state machine.
-
-o "jfq" is the number of jiffies remaining for this grace period
- before force_quiescent_state() is invoked to help push things
- along. Note that CPUs in idle mode throughout the grace period
- will not report on their own, but rather must be check by some
- other CPU via force_quiescent_state().
-
-o "j" is the low-order four hex digits of the jiffies counter.
- Yes, Paul did run into a number of problems that turned out to
- be due to the jiffies counter no longer counting. Why do you ask?
-
-o "nfqs" is the number of calls to force_quiescent_state() since
- boot.
-
-o "nfqsng" is the number of useless calls to force_quiescent_state(),
- where there wasn't actually a grace period active. This can
- no longer happen due to grace-period processing being pushed
- into a kthread. The number in parentheses is the difference
- between "nfqs" and "nfqsng", or the number of times that
- force_quiescent_state() actually did some real work.
-
-o "fqlh" is the number of calls to force_quiescent_state() that
- exited immediately (without even being counted in nfqs above)
- due to contention on ->fqslock.
-
-o Each element of the form "3/3 ..>. 0:7 ^0" represents one rcu_node
- structure. Each line represents one level of the hierarchy,
- from root to leaves. It is best to think of the rcu_data
- structures as forming yet another level after the leaves.
- Note that there might be either one, two, three, or even four
- levels of rcu_node structures, depending on the relationship
- between CONFIG_RCU_FANOUT, CONFIG_RCU_FANOUT_LEAF (possibly
- adjusted using the rcu_fanout_leaf kernel boot parameter), and
- CONFIG_NR_CPUS (possibly adjusted using the nr_cpu_ids count of
- possible CPUs for the booting hardware).
-
- o The numbers separated by the "/" are the qsmask followed
- by the qsmaskinit. The qsmask will have one bit
- set for each entity in the next lower level that has
- not yet checked in for the current grace period ("e"
- indicating CPUs 5, 6, and 7 in the example above).
- The qsmaskinit will have one bit for each entity that is
- currently expected to check in during each grace period.
- The value of qsmaskinit is assigned to that of qsmask
- at the beginning of each grace period.
-
- o The characters separated by the ">" indicate the state
- of the blocked-tasks lists. A "G" preceding the ">"
- indicates that at least one task blocked in an RCU
- read-side critical section blocks the current grace
- period, while a "E" preceding the ">" indicates that
- at least one task blocked in an RCU read-side critical
- section blocks the current expedited grace period.
- A "T" character following the ">" indicates that at
- least one task is blocked within an RCU read-side
- critical section, regardless of whether any current
- grace period (expedited or normal) is inconvenienced.
- A "." character appears if the corresponding condition
- does not hold, so that "..>." indicates that no tasks
- are blocked. In contrast, "GE>T" indicates maximal
- inconvenience from blocked tasks. CONFIG_TREE_RCU
- builds of the kernel will always show "..>.".
-
- o The numbers separated by the ":" are the range of CPUs
- served by this struct rcu_node. This can be helpful
- in working out how the hierarchy is wired together.
-
- For example, the example rcu_node structure shown above
- has "0:7", indicating that it covers CPUs 0 through 7.
-
- o The number after the "^" indicates the bit in the
- next higher level rcu_node structure that this rcu_node
- structure corresponds to. For example, the "d/d ..>. 4:7
- ^1" has a "1" in this position, indicating that it
- corresponds to the "1" bit in the "3" shown in the
- "3/3 ..>. 0:7 ^0" entry on the next level up.
-
-
-The output of "cat rcu/rcu_sched/rcu_pending" looks as follows:
-
- 0!np=26111 qsp=29 rpq=5386 cbr=1 cng=570 gpc=3674 gps=577 nn=15903 ndw=0
- 1!np=28913 qsp=35 rpq=6097 cbr=1 cng=448 gpc=3700 gps=554 nn=18113 ndw=0
- 2!np=32740 qsp=37 rpq=6202 cbr=0 cng=476 gpc=4627 gps=546 nn=20889 ndw=0
- 3 np=23679 qsp=22 rpq=5044 cbr=1 cng=415 gpc=3403 gps=347 nn=14469 ndw=0
- 4!np=30714 qsp=4 rpq=5574 cbr=0 cng=528 gpc=3931 gps=639 nn=20042 ndw=0
- 5 np=28910 qsp=2 rpq=5246 cbr=0 cng=428 gpc=4105 gps=709 nn=18422 ndw=0
- 6!np=38648 qsp=5 rpq=7076 cbr=0 cng=840 gpc=4072 gps=961 nn=25699 ndw=0
- 7 np=37275 qsp=2 rpq=6873 cbr=0 cng=868 gpc=3416 gps=971 nn=25147 ndw=0
-
-The fields are as follows:
-
-o The leading number is the CPU number, with "!" indicating
- an offline CPU.
-
-o "np" is the number of times that __rcu_pending() has been invoked
- for the corresponding flavor of RCU.
-
-o "qsp" is the number of times that the RCU was waiting for a
- quiescent state from this CPU.
-
-o "rpq" is the number of times that the CPU had passed through
- a quiescent state, but not yet reported it to RCU.
-
-o "cbr" is the number of times that this CPU had RCU callbacks
- that had passed through a grace period, and were thus ready
- to be invoked.
-
-o "cng" is the number of times that this CPU needed another
- grace period while RCU was idle.
-
-o "gpc" is the number of times that an old grace period had
- completed, but this CPU was not yet aware of it.
-
-o "gps" is the number of times that a new grace period had started,
- but this CPU was not yet aware of it.
-
-o "ndw" is the number of times that a wakeup of an rcuo
- callback-offload kthread had to be deferred in order to avoid
- deadlock.
-
-o "nn" is the number of times that this CPU needed nothing.
-
-
-The output of "cat rcu/rcuboost" looks as follows:
-
-0:3 tasks=.... kt=W ntb=0 neb=0 nnb=0 j=c864 bt=c894
- balk: nt=0 egt=4695 bt=0 nb=0 ny=56 nos=0
-4:7 tasks=.... kt=W ntb=0 neb=0 nnb=0 j=c864 bt=c894
- balk: nt=0 egt=6541 bt=0 nb=0 ny=126 nos=0
-
-This information is output only for rcu_preempt. Each two-line entry
-corresponds to a leaf rcu_node structure. The fields are as follows:
-
-o "n:m" is the CPU-number range for the corresponding two-line
- entry. In the sample output above, the first entry covers
- CPUs zero through three and the second entry covers CPUs four
- through seven.
-
-o "tasks=TNEB" gives the state of the various segments of the
- rnp->blocked_tasks list:
-
- "T" This indicates that there are some tasks that blocked
- while running on one of the corresponding CPUs while
- in an RCU read-side critical section.
-
- "N" This indicates that some of the blocked tasks are preventing
- the current normal (non-expedited) grace period from
- completing.
-
- "E" This indicates that some of the blocked tasks are preventing
- the current expedited grace period from completing.
-
- "B" This indicates that some of the blocked tasks are in
- need of RCU priority boosting.
-
- Each character is replaced with "." if the corresponding
- condition does not hold.
-
-o "kt" is the state of the RCU priority-boosting kernel
- thread associated with the corresponding rcu_node structure.
- The state can be one of the following:
-
- "S" The kernel thread is stopped, in other words, all
- CPUs corresponding to this rcu_node structure are
- offline.
-
- "R" The kernel thread is running.
-
- "W" The kernel thread is waiting because there is no work
- for it to do.
-
- "Y" The kernel thread is yielding to avoid hogging CPU.
-
- "?" Unknown value, indicates a bug.
-
-o "ntb" is the number of tasks boosted.
-
-o "neb" is the number of tasks boosted in order to complete an
- expedited grace period.
-
-o "nnb" is the number of tasks boosted in order to complete a
- normal (non-expedited) grace period. When boosting a task
- that was blocking both an expedited and a normal grace period,
- it is counted against the expedited total above.
-
-o "j" is the low-order 16 bits of the jiffies counter in
- hexadecimal.
-
-o "bt" is the low-order 16 bits of the value that the jiffies
- counter will have when we next start boosting, assuming that
- the current grace period does not end beforehand. This is
- also in hexadecimal.
-
-o "balk: nt" counts the number of times we didn't boost (in
- other words, we balked) even though it was time to boost because
- there were no blocked tasks to boost. This situation occurs
- when there is one blocked task on one rcu_node structure and
- none on some other rcu_node structure.
-
-o "egt" counts the number of times we balked because although
- there were blocked tasks, none of them were blocking the
- current grace period, whether expedited or otherwise.
-
-o "bt" counts the number of times we balked because boosting
- had already been initiated for the current grace period.
-
-o "nb" counts the number of times we balked because there
- was at least one task blocking the current non-expedited grace
- period that never had blocked. If it is already running, it
- just won't help to boost its priority!
-
-o "ny" counts the number of times we balked because it was
- not yet time to start boosting.
-
-o "nos" counts the number of times we balked for other
- reasons, e.g., the grace period ended first.
-
-
-CONFIG_TINY_RCU debugfs Files and Formats
-
-These implementations of RCU provides a single debugfs file under the
-top-level directory RCU, namely rcu/rcudata, which displays fields in
-rcu_bh_ctrlblk and rcu_sched_ctrlblk.
-
-The output of "cat rcu/rcudata" is as follows:
-
-rcu_sched: qlen: 0
-rcu_bh: qlen: 0
-
-This is split into rcu_sched and rcu_bh sections. The field is as
-follows:
-
-o "qlen" is the number of RCU callbacks currently waiting either
- for an RCU grace period or waiting to be invoked. This is the
- only field present for rcu_sched and rcu_bh, due to the
- short-circuiting of grace period in those two cases.
+=======
+LoadPin
+=======
+
LoadPin is a Linux Security Module that ensures all kernel-loaded files
(modules, firmware, etc) all originate from the same filesystem, with
the expectation that such a filesystem is backed by a read-only device
and/or unchangeable filesystem to enforce module and firmware loading
restrictions without needing to sign the files individually.
-The LSM is selectable at build-time with CONFIG_SECURITY_LOADPIN, and
+The LSM is selectable at build-time with ``CONFIG_SECURITY_LOADPIN``, and
can be controlled at boot-time with the kernel command line option
-"loadpin.enabled". By default, it is enabled, but can be disabled at
-boot ("loadpin.enabled=0").
+"``loadpin.enabled``". By default, it is enabled, but can be disabled at
+boot ("``loadpin.enabled=0``").
LoadPin starts pinning when it sees the first file loaded. If the
block device backing the filesystem is not read-only, a sysctl is
-created to toggle pinning: /proc/sys/kernel/loadpin/enabled. (Having
+created to toggle pinning: ``/proc/sys/kernel/loadpin/enabled``. (Having
a mutable filesystem means pinning is mutable too, but having the
sysctl allows for easy testing on systems with a mutable filesystem.)
+=======
+SELinux
+=======
+
If you want to use SELinux, chances are you will want
to use the distro-provided policies, or install the
latest reference policy release from
+
http://oss.tresys.com/projects/refpolicy
However, if you want to install a dummy policy for
-testing, you can do using 'mdp' provided under
+testing, you can do using ``mdp`` provided under
scripts/selinux. Note that this requires the selinux
userspace to be installed - in particular you will
need checkpolicy to compile a kernel, and setfiles and
fixfiles to label the filesystem.
1. Compile the kernel with selinux enabled.
- 2. Type 'make' to compile mdp.
+ 2. Type ``make`` to compile ``mdp``.
3. Make sure that you are not running with
SELinux enabled and a real policy. If
you are, reboot with selinux disabled
before continuing.
- 4. Run install_policy.sh:
+ 4. Run install_policy.sh::
+
cd scripts/selinux
sh install_policy.sh
Step 4 will create a new dummy policy valid for your
kernel, with a single selinux user, role, and type.
-It will compile the policy, will set your SELINUXTYPE to
-dummy in /etc/selinux/config, install the compiled policy
-as 'dummy', and relabel your filesystem.
+It will compile the policy, will set your ``SELINUXTYPE`` to
+``dummy`` in ``/etc/selinux/config``, install the compiled policy
+as ``dummy``, and relabel your filesystem.
+=====
+Smack
+=====
"Good for you, you've decided to clean the elevator!"
at hand.
Smack consists of three major components:
+
- The kernel
- Basic utilities, which are helpful but not required
- Configuration data
This should make and install on most modern distributions.
There are five commands included in smackutil:
-chsmack - display or set Smack extended attribute values
-smackctl - load the Smack access rules
-smackaccess - report if a process with one label has access
- to an object with another
+chsmack:
+ display or set Smack extended attribute values
+
+smackctl:
+ load the Smack access rules
+
+smackaccess:
+ report if a process with one label has access
+ to an object with another
These two commands are obsolete with the introduction of
the smackfs/load2 and smackfs/cipso2 interfaces.
-smackload - properly formats data for writing to smackfs/load
-smackcipso - properly formats data for writing to smackfs/cipso
+smackload:
+ properly formats data for writing to smackfs/load
+
+smackcipso:
+ properly formats data for writing to smackfs/cipso
In keeping with the intent of Smack, configuration data is
minimal and not strictly required. The most important
If smackutil is installed the startup script will take care
of this, but it can be manually as well.
-Add this line to /etc/fstab:
+Add this line to ``/etc/fstab``::
smackfs /sys/fs/smackfs smackfs defaults 0 0
-The /sys/fs/smackfs directory is created by the kernel.
+The ``/sys/fs/smackfs`` directory is created by the kernel.
Smack uses extended attributes (xattrs) to store labels on filesystem
objects. The attributes are stored in the extended attribute security
-name space. A process must have CAP_MAC_ADMIN to change any of these
+name space. A process must have ``CAP_MAC_ADMIN`` to change any of these
attributes.
The extended attributes that Smack uses are:
Used to make access control decisions. In almost all cases
the label given to a new filesystem object will be the label
of the process that created it.
+
SMACK64EXEC
The Smack label of a process that execs a program file with
this attribute set will run with this attribute's value.
+
SMACK64MMAP
Don't allow the file to be mmapped by a process whose Smack
label does not allow all of the access permitted to a process
with the label contained in this attribute. This is a very
specific use case for shared libraries.
+
SMACK64TRANSMUTE
Can only have the value "TRUE". If this attribute is present
on a directory when an object is created in the directory and
gets the label of the directory instead of the label of the
creating process. If the object being created is a directory
the SMACK64TRANSMUTE attribute is set as well.
+
SMACK64IPIN
This attribute is only available on file descriptors for sockets.
Use the Smack label in this attribute for access control
decisions on packets being delivered to this socket.
+
SMACK64IPOUT
This attribute is only available on file descriptors for sockets.
Use the Smack label in this attribute for access control
decisions on packets coming from this socket.
-There are multiple ways to set a Smack label on a file:
+There are multiple ways to set a Smack label on a file::
# attr -S -s SMACK64 -V "value" path
# chsmack -a value path
A process can see the Smack label it is running with by
-reading /proc/self/attr/current. A process with CAP_MAC_ADMIN
+reading ``/proc/self/attr/current``. A process with ``CAP_MAC_ADMIN``
can set the process Smack by writing there.
Most Smack configuration is accomplished by writing to files
in the smackfs filesystem. This pseudo-filesystem is mounted
-on /sys/fs/smackfs.
+on ``/sys/fs/smackfs``.
access
Provided for backward compatibility. The access2 interface
this file. The next read will indicate whether the access
would be permitted. The text will be either "1" indicating
access, or "0" indicating denial.
+
access2
This interface reports whether a subject with the specified
Smack label has a particular access to an object with a
this file. The next read will indicate whether the access
would be permitted. The text will be either "1" indicating
access, or "0" indicating denial.
+
ambient
This contains the Smack label applied to unlabeled network
packets.
+
change-rule
This interface allows modification of existing access control rules.
- The format accepted on write is:
+ The format accepted on write is::
+
"%s %s %s %s"
+
where the first string is the subject label, the second the
object label, the third the access to allow and the fourth the
access to deny. The access strings may contain only the characters
modified by enabling the permissions in the third string and disabling
those in the fourth string. If there is no such rule it will be
created using the access specified in the third and the fourth strings.
+
cipso
Provided for backward compatibility. The cipso2 interface
is preferred and should be used instead.
This interface allows a specific CIPSO header to be assigned
- to a Smack label. The format accepted on write is:
+ to a Smack label. The format accepted on write is::
+
"%24s%4d%4d"["%4d"]...
+
The first string is a fixed Smack label. The first number is
the level to use. The second number is the number of categories.
- The following numbers are the categories.
- "level-3-cats-5-19 3 2 5 19"
+ The following numbers are the categories::
+
+ "level-3-cats-5-19 3 2 5 19"
+
cipso2
This interface allows a specific CIPSO header to be assigned
- to a Smack label. The format accepted on write is:
- "%s%4d%4d"["%4d"]...
+ to a Smack label. The format accepted on write is::
+
+ "%s%4d%4d"["%4d"]...
+
The first string is a long Smack label. The first number is
the level to use. The second number is the number of categories.
- The following numbers are the categories.
- "level-3-cats-5-19 3 2 5 19"
+ The following numbers are the categories::
+
+ "level-3-cats-5-19 3 2 5 19"
+
direct
This contains the CIPSO level used for Smack direct label
representation in network packets.
+
doi
This contains the CIPSO domain of interpretation used in
network packets.
+
ipv6host
This interface allows specific IPv6 internet addresses to be
treated as single label hosts. Packets are sent to single
label hosts only from processes that have Smack write access
to the host label. All packets received from single label hosts
- are given the specified label. The format accepted on write is:
+ are given the specified label. The format accepted on write is::
+
"%h:%h:%h:%h:%h:%h:%h:%h label" or
"%h:%h:%h:%h:%h:%h:%h:%h/%d label".
+
The "::" address shortcut is not supported.
If label is "-DELETE" a matched entry will be deleted.
+
load
Provided for backward compatibility. The load2 interface
is preferred and should be used instead.
This interface allows access control rules in addition to
the system defined rules to be specified. The format accepted
- on write is:
+ on write is::
+
"%24s%24s%5s"
+
where the first string is the subject label, the second the
object label, and the third the requested access. The access
string may contain only the characters "rwxat-", and specifies
permissions that are not allowed. The string "r-x--" would
specify read and execute access. Labels are limited to 23
characters in length.
+
load2
This interface allows access control rules in addition to
the system defined rules to be specified. The format accepted
- on write is:
+ on write is::
+
"%s %s %s"
+
where the first string is the subject label, the second the
object label, and the third the requested access. The access
string may contain only the characters "rwxat-", and specifies
which sort of access is allowed. The "-" is a placeholder for
permissions that are not allowed. The string "r-x--" would
specify read and execute access.
+
load-self
Provided for backward compatibility. The load-self2 interface
is preferred and should be used instead.
otherwise be permitted, and are intended to provide additional
restrictions on the process. The format is the same as for
the load interface.
+
load-self2
This interface allows process specific access rules to be
defined. These rules are only consulted if access would
otherwise be permitted, and are intended to provide additional
restrictions on the process. The format is the same as for
the load2 interface.
+
logging
This contains the Smack logging state.
+
mapped
This contains the CIPSO level used for Smack mapped label
representation in network packets.
+
netlabel
This interface allows specific internet addresses to be
treated as single label hosts. Packets are sent to single
label hosts without CIPSO headers, but only from processes
that have Smack write access to the host label. All packets
received from single label hosts are given the specified
- label. The format accepted on write is:
+ label. The format accepted on write is::
+
"%d.%d.%d.%d label" or "%d.%d.%d.%d/%d label".
+
If the label specified is "-CIPSO" the address is treated
as a host that supports CIPSO headers.
+
onlycap
This contains labels processes must have for CAP_MAC_ADMIN
- and CAP_MAC_OVERRIDE to be effective. If this file is empty
+ and ``CAP_MAC_OVERRIDE`` to be effective. If this file is empty
these capabilities are effective at for processes with any
label. The values are set by writing the desired labels, separated
by spaces, to the file or cleared by writing "-" to the file.
+
ptrace
This is used to define the current ptrace policy
- 0 - default: this is the policy that relies on Smack access rules.
- For the PTRACE_READ a subject needs to have a read access on
- object. For the PTRACE_ATTACH a read-write access is required.
- 1 - exact: this is the policy that limits PTRACE_ATTACH. Attach is
+
+ 0 - default:
+ this is the policy that relies on Smack access rules.
+ For the ``PTRACE_READ`` a subject needs to have a read access on
+ object. For the ``PTRACE_ATTACH`` a read-write access is required.
+
+ 1 - exact:
+ this is the policy that limits ``PTRACE_ATTACH``. Attach is
only allowed when subject's and object's labels are equal.
- PTRACE_READ is not affected. Can be overridden with CAP_SYS_PTRACE.
- 2 - draconian: this policy behaves like the 'exact' above with an
- exception that it can't be overridden with CAP_SYS_PTRACE.
+ ``PTRACE_READ`` is not affected. Can be overridden with ``CAP_SYS_PTRACE``.
+
+ 2 - draconian:
+ this policy behaves like the 'exact' above with an
+ exception that it can't be overridden with ``CAP_SYS_PTRACE``.
+
revoke-subject
Writing a Smack label here sets the access to '-' for all access
rules with that subject label.
+
unconfined
- If the kernel is configured with CONFIG_SECURITY_SMACK_BRINGUP
- a process with CAP_MAC_ADMIN can write a label into this interface.
+ If the kernel is configured with ``CONFIG_SECURITY_SMACK_BRINGUP``
+ a process with ``CAP_MAC_ADMIN`` can write a label into this interface.
Thereafter, accesses that involve that label will be logged and
the access permitted if it wouldn't be otherwise. Note that this
is dangerous and can ruin the proper labeling of your system.
It should never be used in production.
+
relabel-self
This interface contains a list of labels to which the process can
- transition to, by writing to /proc/self/attr/current.
+ transition to, by writing to ``/proc/self/attr/current``.
Normally a process can change its own label to any legal value, but only
- if it has CAP_MAC_ADMIN. This interface allows a process without
- CAP_MAC_ADMIN to relabel itself to one of labels from predefined list.
- A process without CAP_MAC_ADMIN can change its label only once. When it
+ if it has ``CAP_MAC_ADMIN``. This interface allows a process without
+ ``CAP_MAC_ADMIN`` to relabel itself to one of labels from predefined list.
+ A process without ``CAP_MAC_ADMIN`` can change its label only once. When it
does, this list will be cleared.
The values are set by writing the desired labels, separated
by spaces, to the file or cleared by writing "-" to the file.
If you are using the smackload utility
-you can add access rules in /etc/smack/accesses. They take the form:
+you can add access rules in ``/etc/smack/accesses``. They take the form::
subjectlabel objectlabel access
Look for additional programs on http://schaufler-ca.com
-From the Smack Whitepaper:
-
-The Simplified Mandatory Access Control Kernel
+The Simplified Mandatory Access Control Kernel (Whitepaper)
+===========================================================
Casey Schaufler
casey@schaufler-ca.com
Mandatory Access Control
+------------------------
Computer systems employ a variety of schemes to constrain how information is
shared among the people and services using the machine. Some of these schemes
or programs that have access to pieces of data.
Bell & LaPadula
+---------------
From the middle of the 1980's until the turn of the century Mandatory Access
Control (MAC) was very closely associated with the Bell & LaPadula security
often sited as failing to address general needs.
Domain Type Enforcement
+-----------------------
Around the turn of the century Domain Type Enforcement (DTE) became popular.
This scheme organizes users, programs, and data into domains that are
disabled or used in limited ways in the majority of cases.
Smack
+-----
Smack is a Mandatory Access Control mechanism designed to provide useful MAC
while avoiding the pitfalls of its predecessors. The limitations of Bell &
modes already in use.
Smack Terminology
+-----------------
The jargon used to talk about Smack will be familiar to those who have dealt
with other MAC systems and shouldn't be too difficult for the uninitiated to
pick up. There are four terms that are used in a specific way and that are
especially important:
- Subject: A subject is an active entity on the computer system.
+ Subject:
+ A subject is an active entity on the computer system.
On Smack a subject is a task, which is in turn the basic unit
of execution.
- Object: An object is a passive entity on the computer system.
+ Object:
+ An object is a passive entity on the computer system.
On Smack files of all types, IPC, and tasks can be objects.
- Access: Any attempt by a subject to put information into or get
+ Access:
+ Any attempt by a subject to put information into or get
information from an object is an access.
- Label: Data that identifies the Mandatory Access Control
+ Label:
+ Data that identifies the Mandatory Access Control
characteristics of a subject or an object.
These definitions are consistent with the traditional use in the security
community. There are also some terms from Linux that are likely to crop up:
- Capability: A task that possesses a capability has permission to
+ Capability:
+ A task that possesses a capability has permission to
violate an aspect of the system security policy, as identified by
the specific capability. A task that possesses one or more
capabilities is a privileged task, whereas a task with no
capabilities is an unprivileged task.
- Privilege: A task that is allowed to violate the system security
+ Privilege:
+ A task that is allowed to violate the system security
policy is said to have privilege. As of this writing a task can
have privilege either by possessing capabilities or by having an
effective user of root.
Smack Basics
+------------
Smack is an extension to a Linux system. It enforces additional restrictions
on what subjects can access which objects, based on the labels attached to
each of the subject and the object.
Labels
+~~~~~~
Smack labels are ASCII character strings. They can be up to 255 characters
long, but keeping them to twenty-three characters is recommended.
(quote) and '"' (double-quote) characters.
Smack labels cannot begin with a '-'. This is reserved for special options.
-There are some predefined labels:
+There are some predefined labels::
_ Pronounced "floor", a single underscore character.
^ Pronounced "hat", a single circumflex character.
mechanism.
Access Rules
+~~~~~~~~~~~~
Smack uses the traditional access modes of Linux. These modes are read,
execute, write, and occasionally append. There are a few cases where the
access mode may not be obvious. These include:
- Signals: A signal is a write operation from the subject task to
+ Signals:
+ A signal is a write operation from the subject task to
the object task.
- Internet Domain IPC: Transmission of a packet is considered a
+
+ Internet Domain IPC:
+ Transmission of a packet is considered a
write operation from the source task to the destination task.
Smack restricts access based on the label attached to a subject and the label
7. Any other access is denied.
Smack Access Rules
+~~~~~~~~~~~~~~~~~~
With the isolation provided by Smack access separation is simple. There are
many interesting cases where limited access by subjects to objects with
mechanism for specifying rules allowing access between labels.
Access Rule Format
+~~~~~~~~~~~~~~~~~~
-The format of an access rule is:
+The format of an access rule is::
subject-label object-label access
Uppercase values for the specification letters are allowed as well.
Access mode specifications can be in any order. Examples of acceptable rules
-are:
+are::
TopSecret Secret rx
Secret Unclass R
New Old rRrRr
Closed Off -
-Examples of unacceptable rules are:
+Examples of unacceptable rules are::
Top Secret Secret rx
Ace Ace r
as "ar". A lone dash is used to specify that no access should be allowed.
Applying Access Rules
+~~~~~~~~~~~~~~~~~~~~~
The developers of Linux rarely define new sorts of things, usually importing
schemes and concepts from other systems. Most often, the other systems are
receiver. The receiver is not required to have read access to the sender.
Setting Access Rules
+~~~~~~~~~~~~~~~~~~~~
The configuration file /etc/smack/accesses contains the rules to be set at
system startup. The contents are written to the special file
specification.
Task Attribute
+~~~~~~~~~~~~~~
The Smack label of a process can be read from /proc/<pid>/attr/current. A
process can read its own Smack label from /proc/self/attr/current. A
/proc/self/attr/current but not the label of another process.
File Attribute
+~~~~~~~~~~~~~~
The Smack label of a filesystem object is stored as an extended attribute
named SMACK64 on the file. This attribute is in the security namespace. It can
only be changed by a process with privilege.
Privilege
+~~~~~~~~~
A process with CAP_MAC_OVERRIDE or CAP_MAC_ADMIN is privileged.
CAP_MAC_OVERRIDE allows the process access to objects it would
Smack data, including rules and attributes.
Smack Networking
+~~~~~~~~~~~~~~~~
As mentioned before, Smack enforces access control on network protocol
transmissions. Every packet sent by a Smack process is tagged with its Smack
the packet is dropped.
CIPSO Configuration
+~~~~~~~~~~~~~~~~~~~
It is normally unnecessary to specify the CIPSO configuration. The default
values used by the system handle all internal cases. Smack will compose CIPSO
The label and category set are mapped to a Smack label as defined in
/etc/smack/cipso.
-A Smack/CIPSO mapping has the form:
+A Smack/CIPSO mapping has the form::
smack level [category [category]*]
Smack does not expect the level or category sets to be related in any
particular way and does not assume or assign accesses based on them. Some
-examples of mappings:
+examples of mappings::
TopSecret 7
TS:A,B 7 1 2
/sys/fs/smackfs/direct.
Socket Attributes
+~~~~~~~~~~~~~~~~~
There are two attributes that are associated with sockets. These attributes
can only be set by privileged tasks, but any task can read them for their own
sockets.
- SMACK64IPIN: The Smack label of the task object. A privileged
+ SMACK64IPIN:
+ The Smack label of the task object. A privileged
program that will enforce policy may set this to the star label.
- SMACK64IPOUT: The Smack label transmitted with outgoing packets.
+ SMACK64IPOUT:
+ The Smack label transmitted with outgoing packets.
A privileged program may set this to match the label of another
task with which it hopes to communicate.
Smack Netlabel Exceptions
+~~~~~~~~~~~~~~~~~~~~~~~~~
You will often find that your labeled application has to talk to the outside,
unlabeled world. To do this there's a special file /sys/fs/smackfs/netlabel
-where you can add some exceptions in the form of :
-@IP1 LABEL1 or
-@IP2/MASK LABEL2
+where you can add some exceptions in the form of::
+
+ @IP1 LABEL1 or
+ @IP2/MASK LABEL2
It means that your application will have unlabeled access to @IP1 if it has
write access on LABEL1, and access to the subnet @IP2/MASK if it has write
Entries in the /sys/fs/smackfs/netlabel file are matched by longest mask
first, like in classless IPv4 routing.
-A special label '@' and an option '-CIPSO' can be used there :
-@ means Internet, any application with any label has access to it
--CIPSO means standard CIPSO networking
+A special label '@' and an option '-CIPSO' can be used there::
-If you don't know what CIPSO is and don't plan to use it, you can just do :
-echo 127.0.0.1 -CIPSO > /sys/fs/smackfs/netlabel
-echo 0.0.0.0/0 @ > /sys/fs/smackfs/netlabel
+ @ means Internet, any application with any label has access to it
+ -CIPSO means standard CIPSO networking
+
+If you don't know what CIPSO is and don't plan to use it, you can just do::
+
+ echo 127.0.0.1 -CIPSO > /sys/fs/smackfs/netlabel
+ echo 0.0.0.0/0 @ > /sys/fs/smackfs/netlabel
If you use CIPSO on your 192.168.0.0/16 local network and need also unlabeled
-Internet access, you can have :
-echo 127.0.0.1 -CIPSO > /sys/fs/smackfs/netlabel
-echo 192.168.0.0/16 -CIPSO > /sys/fs/smackfs/netlabel
-echo 0.0.0.0/0 @ > /sys/fs/smackfs/netlabel
+Internet access, you can have::
+ echo 127.0.0.1 -CIPSO > /sys/fs/smackfs/netlabel
+ echo 192.168.0.0/16 -CIPSO > /sys/fs/smackfs/netlabel
+ echo 0.0.0.0/0 @ > /sys/fs/smackfs/netlabel
Writing Applications for Smack
+------------------------------
There are three sorts of applications that will run on a Smack system. How an
application interacts with Smack will determine what it will have to do to
work properly under Smack.
Smack Ignorant Applications
+---------------------------
By far the majority of applications have no reason whatever to care about the
unique properties of Smack. Since invoking a program has no impact on the
whether the process has execute access to the program.
Smack Relevant Applications
+---------------------------
Some programs can be improved by teaching them about Smack, but do not make
any security decisions themselves. The utility ls(1) is one example of such a
program.
Smack Enforcing Applications
+----------------------------
These are special programs that not only know about Smack, but participate in
the enforcement of system policy. In most cases these are the programs that
to processes running with various labels.
File System Interfaces
+----------------------
Smack maintains labels on file system objects using extended attributes. The
Smack label of a file, directory, or other file system object can be obtained
-using getxattr(2).
+using getxattr(2)::
len = getxattr("/", "security.SMACK64", value, sizeof (value));
will put the Smack label of the root directory into value. A privileged
-process can set the Smack label of a file system object with setxattr(2).
+process can set the Smack label of a file system object with setxattr(2)::
len = strlen("Rubble");
rc = setxattr("/foo", "security.SMACK64", "Rubble", len, 0);
privilege.
Socket Interfaces
+-----------------
The socket attributes can be read using fgetxattr(2).
A privileged process can set the Smack label of outgoing packets with
-fsetxattr(2).
+fsetxattr(2)::
len = strlen("Rubble");
rc = fsetxattr(fd, "security.SMACK64IPOUT", "Rubble", len, 0);
will set the Smack label "Rubble" on packets going out from the socket if the
-program has appropriate privilege.
+program has appropriate privilege::
rc = fsetxattr(fd, "security.SMACK64IPIN, "*", strlen("*"), 0);
packets will be checked if the program has appropriate privilege.
Administration
+--------------
Smack supports some mount options:
- smackfsdef=label: specifies the label to give files that lack
+ smackfsdef=label:
+ specifies the label to give files that lack
the Smack label extended attribute.
- smackfsroot=label: specifies the label to assign the root of the
+ smackfsroot=label:
+ specifies the label to assign the root of the
file system if it lacks the Smack extended attribute.
- smackfshat=label: specifies a label that must have read access to
+ smackfshat=label:
+ specifies a label that must have read access to
all labels set on the filesystem. Not yet enforced.
- smackfsfloor=label: specifies a label to which all labels set on the
+ smackfsfloor=label:
+ specifies a label to which all labels set on the
filesystem must have read access. Not yet enforced.
These mount options apply to all file system types.
Smack auditing
+--------------
If you want Smack auditing of security events, you need to set CONFIG_AUDIT
in your kernel configuration.
By default, all denied events will be audited. You can change this behavior by
-writing a single character to the /sys/fs/smackfs/logging file :
-0 : no logging
-1 : log denied (default)
-2 : log accepted
-3 : log denied & accepted
+writing a single character to the /sys/fs/smackfs/logging file::
+
+ 0 : no logging
+ 1 : log denied (default)
+ 2 : log accepted
+ 3 : log denied & accepted
Events are logged as 'key=value' pairs, for each event you at least will get
the subject, the object, the rights requested, the action, the kernel function
audited.
Bringup Mode
+------------
Bringup mode provides logging features that can make application
configuration and system bringup easier. Configure the kernel with
+====
+Yama
+====
+
Yama is a Linux Security Module that collects system-wide DAC security
protections that are not handled by the core kernel itself. This is
-selectable at build-time with CONFIG_SECURITY_YAMA, and can be controlled
-at run-time through sysctls in /proc/sys/kernel/yama:
-
-- ptrace_scope
+selectable at build-time with ``CONFIG_SECURITY_YAMA``, and can be controlled
+at run-time through sysctls in ``/proc/sys/kernel/yama``:
-==============================================================
-
-ptrace_scope:
+ptrace_scope
+============
As Linux grows in popularity, it will become a larger target for
malware. One particularly troubling weakness of the Linux process
Since ptrace is not commonly used by non-developers and non-admins, system
builders should be allowed the option to disable this debugging system.
-For a solution, some applications use prctl(PR_SET_DUMPABLE, ...) to
+For a solution, some applications use ``prctl(PR_SET_DUMPABLE, ...)`` to
specifically disallow such ptrace attachment (e.g. ssh-agent), but many
do not. A more general solution is to only allow ptrace directly from a
parent to a child process (i.e. direct "gdb EXE" and "strace EXE" still
-work), or with CAP_SYS_PTRACE (i.e. "gdb --pid=PID", and "strace -p PID"
+work), or with ``CAP_SYS_PTRACE`` (i.e. "gdb --pid=PID", and "strace -p PID"
still work as root).
In mode 1, software that has defined application-specific relationships
between a debugging process and its inferior (crash handlers, etc),
-prctl(PR_SET_PTRACER, pid, ...) can be used. An inferior can declare which
-other process (and its descendants) are allowed to call PTRACE_ATTACH
+``prctl(PR_SET_PTRACER, pid, ...)`` can be used. An inferior can declare which
+other process (and its descendants) are allowed to call ``PTRACE_ATTACH``
against it. Only one such declared debugging process can exists for
each inferior at a time. For example, this is used by KDE, Chromium, and
Firefox's crash handlers, and by Wine for allowing only Wine processes
to ptrace each other. If a process wishes to entirely disable these ptrace
-restrictions, it can call prctl(PR_SET_PTRACER, PR_SET_PTRACER_ANY, ...)
+restrictions, it can call ``prctl(PR_SET_PTRACER, PR_SET_PTRACER_ANY, ...)``
so that any otherwise allowed process (even those in external pid namespaces)
may attach.
-The sysctl settings (writable only with CAP_SYS_PTRACE) are:
+The sysctl settings (writable only with ``CAP_SYS_PTRACE``) are:
-0 - classic ptrace permissions: a process can PTRACE_ATTACH to any other
+0 - classic ptrace permissions:
+ a process can ``PTRACE_ATTACH`` to any other
process running under the same uid, as long as it is dumpable (i.e.
did not transition uids, start privileged, or have called
- prctl(PR_SET_DUMPABLE...) already). Similarly, PTRACE_TRACEME is
+ ``prctl(PR_SET_DUMPABLE...)`` already). Similarly, ``PTRACE_TRACEME`` is
unchanged.
-1 - restricted ptrace: a process must have a predefined relationship
- with the inferior it wants to call PTRACE_ATTACH on. By default,
+1 - restricted ptrace:
+ a process must have a predefined relationship
+ with the inferior it wants to call ``PTRACE_ATTACH`` on. By default,
this relationship is that of only its descendants when the above
classic criteria is also met. To change the relationship, an
- inferior can call prctl(PR_SET_PTRACER, debugger, ...) to declare
- an allowed debugger PID to call PTRACE_ATTACH on the inferior.
- Using PTRACE_TRACEME is unchanged.
+ inferior can call ``prctl(PR_SET_PTRACER, debugger, ...)`` to declare
+ an allowed debugger PID to call ``PTRACE_ATTACH`` on the inferior.
+ Using ``PTRACE_TRACEME`` is unchanged.
-2 - admin-only attach: only processes with CAP_SYS_PTRACE may use ptrace
- with PTRACE_ATTACH, or through children calling PTRACE_TRACEME.
+2 - admin-only attach:
+ only processes with ``CAP_SYS_PTRACE`` may use ptrace
+ with ``PTRACE_ATTACH``, or through children calling ``PTRACE_TRACEME``.
-3 - no attach: no processes may use ptrace with PTRACE_ATTACH nor via
- PTRACE_TRACEME. Once set, this sysctl value cannot be changed.
+3 - no attach:
+ no processes may use ptrace with ``PTRACE_ATTACH`` nor via
+ ``PTRACE_TRACEME``. Once set, this sysctl value cannot be changed.
The original children-only logic was based on the restrictions in grsecurity.
-
-==============================================================
---- What is AppArmor? ---
+========
+AppArmor
+========
+
+What is AppArmor?
+=================
AppArmor is MAC style security extension for the Linux kernel. It implements
a task centered policy, with task "profiles" being created and loaded
them run in an unconfined state which is equivalent to standard Linux DAC
permissions.
---- How to enable/disable ---
+How to enable/disable
+=====================
+
+set ``CONFIG_SECURITY_APPARMOR=y``
-set CONFIG_SECURITY_APPARMOR=y
+If AppArmor should be selected as the default security module then set::
-If AppArmor should be selected as the default security module then
- set CONFIG_DEFAULT_SECURITY="apparmor"
- and CONFIG_SECURITY_APPARMOR_BOOTPARAM_VALUE=1
+ CONFIG_DEFAULT_SECURITY="apparmor"
+ CONFIG_SECURITY_APPARMOR_BOOTPARAM_VALUE=1
Build the kernel
If AppArmor is not the default security module it can be enabled by passing
-security=apparmor on the kernel's command line.
+``security=apparmor`` on the kernel's command line.
If AppArmor is the default security module it can be disabled by passing
-apparmor=0, security=XXXX (where XXX is valid security module), on the
-kernel's command line
+``apparmor=0, security=XXXX`` (where ``XXXX`` is valid security module), on the
+kernel's command line.
For AppArmor to enforce any restrictions beyond standard Linux DAC permissions
policy must be loaded into the kernel from user space (see the Documentation
and tools links).
---- Documentation ---
+Documentation
+=============
-Documentation can be found on the wiki.
+Documentation can be found on the wiki, linked below.
---- Links ---
+Links
+=====
Mailing List - apparmor@lists.ubuntu.com
+
Wiki - http://apparmor.wiki.kernel.org/
+
User space tools - https://launchpad.net/apparmor
+
Kernel module - git://git.kernel.org/pub/scm/linux/kernel/git/jj/apparmor-dev.git
-Linux Security Module framework
--------------------------------
+===========================
+Linux Security Module Usage
+===========================
The Linux Security Module (LSM) framework provides a mechanism for
various security checks to be hooked by new kernel extensions. The name
"module" is a bit of a misnomer since these extensions are not actually
loadable kernel modules. Instead, they are selectable at build-time via
CONFIG_DEFAULT_SECURITY and can be overridden at boot-time via the
-"security=..." kernel command line argument, in the case where multiple
+``"security=..."`` kernel command line argument, in the case where multiple
LSMs were built into a given kernel.
The primary users of the LSM interface are Mandatory Access Control
Without a specific LSM built into the kernel, the default LSM will be the
Linux capabilities system. Most LSMs choose to extend the capabilities
system, building their checks on top of the defined capability hooks.
-For more details on capabilities, see capabilities(7) in the Linux
+For more details on capabilities, see ``capabilities(7)`` in the Linux
man-pages project.
A list of the active security modules can be found by reading
-/sys/kernel/security/lsm. This is a comma separated list, and
+``/sys/kernel/security/lsm``. This is a comma separated list, and
will always include the capability module. The list reflects the
order in which checks are made. The capability module will always
be first, followed by any "minor" modules (e.g. Yama) and then
the one "major" module (e.g. SELinux) if there is one configured.
-Based on https://lkml.org/lkml/2007/10/26/215,
-a new LSM is accepted into the kernel when its intent (a description of
-what it tries to protect against and in what cases one would expect to
-use it) has been appropriately documented in Documentation/security/.
-This allows an LSM's code to be easily compared to its goals, and so
-that end users and distros can make a more informed decision about which
-LSMs suit their requirements.
+.. toctree::
+ :maxdepth: 1
-For extensive documentation on the available LSM hook interfaces, please
-see include/linux/security.h.
+ apparmor
+ LoadPin
+ SELinux
+ Smack
+ tomoyo
+ Yama
---- What is TOMOYO? ---
+======
+TOMOYO
+======
+
+What is TOMOYO?
+===============
TOMOYO is a name-based MAC extension (LSM module) for the Linux kernel.
LiveCD-based tutorials are available at
+
http://tomoyo.sourceforge.jp/1.7/1st-step/ubuntu10.04-live/
-http://tomoyo.sourceforge.jp/1.7/1st-step/centos5-live/ .
+http://tomoyo.sourceforge.jp/1.7/1st-step/centos5-live/
+
Though these tutorials use non-LSM version of TOMOYO, they are useful for you
to know what TOMOYO is.
---- How to enable TOMOYO? ---
+How to enable TOMOYO?
+=====================
-Build the kernel with CONFIG_SECURITY_TOMOYO=y and pass "security=tomoyo" on
+Build the kernel with ``CONFIG_SECURITY_TOMOYO=y`` and pass ``security=tomoyo`` on
kernel's command line.
Please see http://tomoyo.sourceforge.jp/2.3/ for details.
---- Where is documentation? ---
+Where is documentation?
+=======================
User <-> Kernel interface documentation is available at
http://tomoyo.sourceforge.jp/2.3/policy-reference.html .
Realities of Mainlining
http://sourceforge.jp/projects/tomoyo/docs/lfj2008.pdf
---- What is future plan? ---
+What is future plan?
+====================
We believe that inode based security and name based security are complementary
and both should be used together. But unfortunately, so far, we cannot enable
contains information about the problems, which may result by upgrading
your kernel.
- - The Documentation/DocBook/ subdirectory contains several guides for
- kernel developers and users. These guides can be rendered in a
- number of formats: PostScript (.ps), PDF, HTML, & man-pages, among others.
- After installation, ``make psdocs``, ``make pdfdocs``, ``make htmldocs``,
- or ``make mandocs`` will render the documentation in the requested format.
-
Installing the kernel source
----------------------------
237 = /dev/loop-control Loopback control device
238 = /dev/vhost-net Host kernel accelerator for virtio net
239 = /dev/uhid User-space I/O driver support for HID subsystem
+ 240 = /dev/userio Serio driver testing device
+ 241 = /dev/vhost-vsock Host kernel driver for virtio vsock
- 240-254 Reserved for local use
+ 242-254 Reserved for local use
255 Reserved for MISC_DYNAMIC_MINOR
11 char Raw keyboard device (Linux/SPARC only)
java
ras
pm/index
+ thunderbolt
+ LSM/index
.. only:: subproject and html
/proc/<pid>/coredump_filter.
See also Documentation/filesystems/proc.txt.
+ coresight_cpu_debug.enable
+ [ARM,ARM64]
+ Format: <bool>
+ Enable/disable the CPU sampling based debugging.
+ 0: default value, disable debugging
+ 1: enable debugging at boot time
+
cpuidle.off=1 [CPU_IDLE]
disable the cpuidle sub-system
See also Documentation/input/joystick-parport.txt
ddebug_query= [KNL,DYNAMIC_DEBUG] Enable debug messages at early boot
- time. See Documentation/dynamic-debug-howto.txt for
+ time. See
+ Documentation/admin-guide/dynamic-debug-howto.rst for
details. Deprecated, see dyndbg.
debug [KNL] Enable kernel debugging (events log level).
dyndbg[="val"] [KNL,DYNAMIC_DEBUG]
module.dyndbg[="val"]
Enable debug messages at boot time. See
- Documentation/dynamic-debug-howto.txt for details.
+ Documentation/admin-guide/dynamic-debug-howto.rst
+ for details.
nompx [X86] Disables Intel Memory Protection Extensions.
See Documentation/x86/intel_mpx.txt for more
must already be setup and configured. Options are not
yet supported.
+ owl,<addr>
+ Start an early, polled-mode console on a serial port
+ of an Actions Semi SoC, such as S500 or S900, at the
+ specified address. The serial port must already be
+ setup and configured. Options are not yet supported.
+
smh Use ARM semihosting calls for early console.
s3c2410,<addr>
memmap=nn[KMG]@ss[KMG]
[KNL] Force usage of a specific region of memory.
Region of memory to be used is from ss to ss+nn.
+ If @ss[KMG] is omitted, it is equivalent to mem=nn[KMG],
+ which limits max address to nn[KMG].
+ Multiple different regions can be specified,
+ comma delimited.
+ Example:
+ memmap=100M@2G,100M#3G,1G!1024G
memmap=nn[KMG]#ss[KMG]
[KNL,ACPI] Mark specific memory as ACPI data.
memmap=64K$0x18690000
or
memmap=0x10000$0x18690000
+ Some bootloaders may need an escape character before '$',
+ like Grub2, otherwise '$' and the following number
+ will be eaten.
memmap=nn[KMG]!ss[KMG]
[KNL,X86] Mark specific memory as protected.
rcutree.gp_cleanup_delay= [KNL]
Set the number of jiffies to delay each step of
- RCU grace-period cleanup. This only has effect
- when CONFIG_RCU_TORTURE_TEST_SLOW_CLEANUP is set.
+ RCU grace-period cleanup.
rcutree.gp_init_delay= [KNL]
Set the number of jiffies to delay each step of
- RCU grace-period initialization. This only has
- effect when CONFIG_RCU_TORTURE_TEST_SLOW_INIT
- is set.
+ RCU grace-period initialization.
rcutree.gp_preinit_delay= [KNL]
Set the number of jiffies to delay each step of
RCU grace-period pre-initialization, that is,
the propagation of recent CPU-hotplug changes up
- the rcu_node combining tree. This only has effect
- when CONFIG_RCU_TORTURE_TEST_SLOW_PREINIT is set.
+ the rcu_node combining tree.
rcutree.rcu_fanout_exact= [KNL]
Disable autobalancing of the rcu_node combining
This wake_up() will be accompanied by a
WARN_ONCE() splat and an ftrace_dump().
+ rcuperf.gp_async= [KNL]
+ Measure performance of asynchronous
+ grace-period primitives such as call_rcu().
+
+ rcuperf.gp_async_max= [KNL]
+ Specify the maximum number of outstanding
+ callbacks per writer thread. When a writer
+ thread exceeds this limit, it invokes the
+ corresponding flavor of rcu_barrier() to allow
+ previously posted callbacks to drain.
+
rcuperf.gp_exp= [KNL]
Measure performance of expedited synchronous
grace-period primitives.
rcuperf.perf_runnable= [BOOT]
Start rcuperf running at boot time.
+ rcuperf.perf_type= [KNL]
+ Specify the RCU implementation to test.
+
rcuperf.shutdown= [KNL]
Shut the system down after performance tests
complete. This is useful for hands-off automated
testing.
- rcuperf.perf_type= [KNL]
- Specify the RCU implementation to test.
-
rcuperf.verbose= [KNL]
Enable additional printk() statements.
+ rcuperf.writer_holdoff= [KNL]
+ Write-side holdoff between grace periods,
+ in microseconds. The default of zero says
+ no holdoff.
+
rcutorture.cbflood_inter_holdoff= [KNL]
Set holdoff time (jiffies) between successive
callback-flood tests.
spia_pedr=
spia_peddr=
+ srcutree.counter_wrap_check [KNL]
+ Specifies how frequently to check for
+ grace-period sequence counter wrap for the
+ srcu_data structure's ->srcu_gp_seq_needed field.
+ The greater the number of bits set in this kernel
+ parameter, the less frequently counter wrap will
+ be checked for. Note that the bottom two bits
+ are ignored.
+
srcutree.exp_holdoff [KNL]
Specifies how many nanoseconds must elapse
since the end of the last SRCU grace period for
``scaling_cur_freq``
Current frequency of all of the CPUs belonging to this policy (in kHz).
- For the majority of scaling drivers, this is the frequency of the last
- P-state requested by the driver from the hardware using the scaling
+ In the majority of cases, this is the frequency of the last P-state
+ requested by the scaling driver from the hardware using the scaling
interface provided by it, which may or may not reflect the frequency
the CPU is actually running at (due to hardware design and other
limitations).
- Some scaling drivers (e.g. |intel_pstate|) attempt to provide
- information more precisely reflecting the current CPU frequency through
- this attribute, but that still may not be the exact current CPU
- frequency as seen by the hardware at the moment.
+ Some architectures (e.g. ``x86``) may attempt to provide information
+ more precisely reflecting the current CPU frequency through this
+ attribute, but that still may not be the exact current CPU frequency as
+ seen by the hardware at the moment.
``scaling_driver``
The scaling driver currently in use.
the processor model and platform configuration.
It selects the maximum P-state it is allowed to use, subject to limits set via
-``sysfs``, every time the P-state selection computations are carried out by the
-driver's utilization update callback for the given CPU (that does not happen
-more often than every 10 ms), but the hardware configuration will not be changed
-if the new P-state is the same as the current one.
+``sysfs``, every time the driver configuration for the given CPU is updated
+(e.g. via ``sysfs``).
This is the default P-state selection algorithm if the
:c:macro:`CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE` kernel configuration option
controllers. The following example will assume 2 channels:
+------------+-----------------------+
- | Chip | Channels |
- | Select +-----------+-----------+
- | rows | ``ch0`` | ``ch1`` |
+ | CS Rows | Channels |
+ +------------+-----------+-----------+
+ | | ``ch0`` | ``ch1`` |
+============+===========+===========+
| ``csrow0`` | DIMM_A0 | DIMM_B0 |
+------------+ | |
The structure of the message is:
+---------------------------------------+-------------+
- | Content + Example |
+ | Content | Example |
+=======================================+=============+
| The memory controller | MC0 |
+---------------------------------------+-------------+
+---------------------------------------+-------------+
| The error syndrome | 0xb741 |
+---------------------------------------+-------------+
- | Memory row | row 0 +
+ | Memory row | row 0 |
+---------------------------------------+-------------+
| Memory channel | channel 1 |
+---------------------------------------+-------------+
--- /dev/null
+=============
+ Thunderbolt
+=============
+The interface presented here is not meant for end users. Instead there
+should be a userspace tool that handles all the low-level details, keeps
+database of the authorized devices and prompts user for new connections.
+
+More details about the sysfs interface for Thunderbolt devices can be
+found in ``Documentation/ABI/testing/sysfs-bus-thunderbolt``.
+
+Those users who just want to connect any device without any sort of
+manual work, can add following line to
+``/etc/udev/rules.d/99-local.rules``::
+
+ ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1"
+
+This will authorize all devices automatically when they appear. However,
+keep in mind that this bypasses the security levels and makes the system
+vulnerable to DMA attacks.
+
+Security levels and how to use them
+-----------------------------------
+Starting from Intel Falcon Ridge Thunderbolt controller there are 4
+security levels available. The reason for these is the fact that the
+connected devices can be DMA masters and thus read contents of the host
+memory without CPU and OS knowing about it. There are ways to prevent
+this by setting up an IOMMU but it is not always available for various
+reasons.
+
+The security levels are as follows:
+
+ none
+ All devices are automatically connected by the firmware. No user
+ approval is needed. In BIOS settings this is typically called
+ *Legacy mode*.
+
+ user
+ User is asked whether the device is allowed to be connected.
+ Based on the device identification information available through
+ ``/sys/bus/thunderbolt/devices``. user then can do the decision.
+ In BIOS settings this is typically called *Unique ID*.
+
+ secure
+ User is asked whether the device is allowed to be connected. In
+ addition to UUID the device (if it supports secure connect) is sent
+ a challenge that should match the expected one based on a random key
+ written to ``key`` sysfs attribute. In BIOS settings this is
+ typically called *One time saved key*.
+
+ dponly
+ The firmware automatically creates tunnels for Display Port and
+ USB. No PCIe tunneling is done. In BIOS settings this is
+ typically called *Display Port Only*.
+
+The current security level can be read from
+``/sys/bus/thunderbolt/devices/domainX/security`` where ``domainX`` is
+the Thunderbolt domain the host controller manages. There is typically
+one domain per Thunderbolt host controller.
+
+If the security level reads as ``user`` or ``secure`` the connected
+device must be authorized by the user before PCIe tunnels are created
+(e.g the PCIe device appears).
+
+Each Thunderbolt device plugged in will appear in sysfs under
+``/sys/bus/thunderbolt/devices``. The device directory carries
+information that can be used to identify the particular device,
+including its name and UUID.
+
+Authorizing devices when security level is ``user`` or ``secure``
+-----------------------------------------------------------------
+When a device is plugged in it will appear in sysfs as follows::
+
+ /sys/bus/thunderbolt/devices/0-1/authorized - 0
+ /sys/bus/thunderbolt/devices/0-1/device - 0x8004
+ /sys/bus/thunderbolt/devices/0-1/device_name - Thunderbolt to FireWire Adapter
+ /sys/bus/thunderbolt/devices/0-1/vendor - 0x1
+ /sys/bus/thunderbolt/devices/0-1/vendor_name - Apple, Inc.
+ /sys/bus/thunderbolt/devices/0-1/unique_id - e0376f00-0300-0100-ffff-ffffffffffff
+
+The ``authorized`` attribute reads 0 which means no PCIe tunnels are
+created yet. The user can authorize the device by simply::
+
+ # echo 1 > /sys/bus/thunderbolt/devices/0-1/authorized
+
+This will create the PCIe tunnels and the device is now connected.
+
+If the device supports secure connect, and the domain security level is
+set to ``secure``, it has an additional attribute ``key`` which can hold
+a random 32 byte value used for authorization and challenging the device in
+future connects::
+
+ /sys/bus/thunderbolt/devices/0-3/authorized - 0
+ /sys/bus/thunderbolt/devices/0-3/device - 0x305
+ /sys/bus/thunderbolt/devices/0-3/device_name - AKiTiO Thunder3 PCIe Box
+ /sys/bus/thunderbolt/devices/0-3/key -
+ /sys/bus/thunderbolt/devices/0-3/vendor - 0x41
+ /sys/bus/thunderbolt/devices/0-3/vendor_name - inXtron
+ /sys/bus/thunderbolt/devices/0-3/unique_id - dc010000-0000-8508-a22d-32ca6421cb16
+
+Notice the key is empty by default.
+
+If the user does not want to use secure connect it can just ``echo 1``
+to the ``authorized`` attribute and the PCIe tunnels will be created in
+the same way than in ``user`` security level.
+
+If the user wants to use secure connect, the first time the device is
+plugged a key needs to be created and send to the device::
+
+ # key=$(openssl rand -hex 32)
+ # echo $key > /sys/bus/thunderbolt/devices/0-3/key
+ # echo 1 > /sys/bus/thunderbolt/devices/0-3/authorized
+
+Now the device is connected (PCIe tunnels are created) and in addition
+the key is stored on the device NVM.
+
+Next time the device is plugged in the user can verify (challenge) the
+device using the same key::
+
+ # echo $key > /sys/bus/thunderbolt/devices/0-3/key
+ # echo 2 > /sys/bus/thunderbolt/devices/0-3/authorized
+
+If the challenge the device returns back matches the one we expect based
+on the key, the device is connected and the PCIe tunnels are created.
+However, if the challenge failed no tunnels are created and error is
+returned to the user.
+
+If the user still wants to connect the device it can either approve
+the device without a key or write new key and write 1 to the
+``authorized`` file to get the new key stored on the device NVM.
+
+Upgrading NVM on Thunderbolt device or host
+-------------------------------------------
+Since most of the functionality is handled in a firmware running on a
+host controller or a device, it is important that the firmware can be
+upgraded to the latest where possible bugs in it have been fixed.
+Typically OEMs provide this firmware from their support site.
+
+There is also a central site which has links where to download firmwares
+for some machines:
+
+ `Thunderbolt Updates <https://thunderbolttechnology.net/updates>`_
+
+Before you upgrade firmware on a device or host, please make sure it is
+the suitable. Failing to do that may render the device (or host) in a
+state where it cannot be used properly anymore without special tools!
+
+Host NVM upgrade on Apple Macs is not supported.
+
+Once the NVM image has been downloaded, you need to plug in a
+Thunderbolt device so that the host controller appears. It does not
+matter which device is connected (unless you are upgrading NVM on a
+device - then you need to connect that particular device).
+
+Note OEM-specific method to power the controller up ("force power") may
+be available for your system in which case there is no need to plug in a
+Thunderbolt device.
+
+After that we can write the firmware to the non-active parts of the NVM
+of the host or device. As an example here is how Intel NUC6i7KYK (Skull
+Canyon) Thunderbolt controller NVM is upgraded::
+
+ # dd if=KYK_TBT_FW_0018.bin of=/sys/bus/thunderbolt/devices/0-0/nvm_non_active0/nvmem
+
+Once the operation completes we can trigger NVM authentication and
+upgrade process as follows::
+
+ # echo 1 > /sys/bus/thunderbolt/devices/0-0/nvm_authenticate
+
+If no errors are returned, the host controller shortly disappears. Once
+it comes back the driver notices it and initiates a full power cycle.
+After a while the host controller appears again and this time it should
+be fully functional.
+
+We can verify that the new NVM firmware is active by running following
+commands::
+
+ # cat /sys/bus/thunderbolt/devices/0-0/nvm_authenticate
+ 0x0
+ # cat /sys/bus/thunderbolt/devices/0-0/nvm_version
+ 18.0
+
+If ``nvm_authenticate`` contains anything else than 0x0 it is the error
+code from the last authentication cycle, which means the authentication
+of the NVM image failed.
+
+Note names of the NVMem devices ``nvm_activeN`` and ``nvm_non_activeN``
+depends on the order they are registered in the NVMem subsystem. N in
+the name is the identifier added by the NVMem subsystem.
+
+Upgrading NVM when host controller is in safe mode
+--------------------------------------------------
+If the existing NVM is not properly authenticated (or is missing) the
+host controller goes into safe mode which means that only available
+functionality is flashing new NVM image. When in this mode the reading
+``nvm_version`` fails with ``ENODATA`` and the device identification
+information is missing.
+
+To recover from this mode, one needs to flash a valid NVM image to the
+host host controller in the same way it is done in the previous chapter.
i/o is issued (since the bio may otherwise get freed in case i/o completion
happens in the meantime).
-The bio_clone() routine may be used to duplicate a bio, where the clone
+The bio_clone_fast() routine may be used to duplicate a bio, where the clone
shares the bio_vec_list with the original bio (i.e. both point to the
same bio_vec_list). This would typically be used for splitting i/o requests
in lvm or md.
# Additional stuff for the LaTeX preamble.
'preamble': '''
- % Adjust margins
- \\usepackage[margin=0.5in, top=1in, bottom=1in]{geometry}
+ \\usepackage{ifthen}
% Allow generate some pages in landscape
\\usepackage{lscape}
\\definecolor{NoteColor}{RGB}{204,255,255}
\\definecolor{WarningColor}{RGB}{255,204,204}
\\definecolor{AttentionColor}{RGB}{255,255,204}
+ \\definecolor{ImportantColor}{RGB}{192,255,204}
\\definecolor{OtherColor}{RGB}{204,204,204}
\\newlength{\\mynoticelength}
\\makeatletter\\newenvironment{coloredbox}[1]{%
\\ifthenelse%
{\\equal{\\py@noticetype}{attention}}%
{\\colorbox{AttentionColor}{\\usebox{\\@tempboxa}}}%
- {\\colorbox{OtherColor}{\\usebox{\\@tempboxa}}}%
+ {%
+ \\ifthenelse%
+ {\\equal{\\py@noticetype}{important}}%
+ {\\colorbox{ImportantColor}{\\usebox{\\@tempboxa}}}%
+ {\\colorbox{OtherColor}{\\usebox{\\@tempboxa}}}%
+ }%
}%
}%
}\\makeatother
if major == 1 and minor > 3:
latex_elements['preamble'] += '\\renewcommand*{\\DUrole}[2]{ #2 }\n'
+if major == 1 and minor <= 4:
+ latex_elements['preamble'] += '\\usepackage[margin=0.5in, top=1in, bottom=1in]{geometry}'
+elif major == 1 and (minor > 5 or (minor == 5 and patch >= 3)):
+ latex_elements['sphinxsetup'] = 'hmargin=0.5in, vmargin=0.5in'
+
+
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
+# Sorted in alphabetical order
latex_documents = [
- ('doc-guide/index', 'kernel-doc-guide.tex', 'Linux Kernel Documentation Guide',
- 'The kernel development community', 'manual'),
('admin-guide/index', 'linux-user.tex', 'Linux Kernel User Documentation',
'The kernel development community', 'manual'),
('core-api/index', 'core-api.tex', 'The kernel core API manual',
'The kernel development community', 'manual'),
- ('driver-api/index', 'driver-api.tex', 'The kernel driver API manual',
+ ('crypto/index', 'crypto-api.tex', 'Linux Kernel Crypto API manual',
'The kernel development community', 'manual'),
- ('input/index', 'linux-input.tex', 'The Linux input driver subsystem',
+ ('dev-tools/index', 'dev-tools.tex', 'Development tools for the Kernel',
'The kernel development community', 'manual'),
- ('kernel-documentation', 'kernel-documentation.tex', 'The Linux Kernel Documentation',
+ ('doc-guide/index', 'kernel-doc-guide.tex', 'Linux Kernel Documentation Guide',
'The kernel development community', 'manual'),
- ('process/index', 'development-process.tex', 'Linux Kernel Development Documentation',
+ ('driver-api/index', 'driver-api.tex', 'The kernel driver API manual',
+ 'The kernel development community', 'manual'),
+ ('filesystems/index', 'filesystems.tex', 'Linux Filesystems API',
'The kernel development community', 'manual'),
('gpu/index', 'gpu.tex', 'Linux GPU Driver Developer\'s Guide',
'The kernel development community', 'manual'),
+ ('input/index', 'linux-input.tex', 'The Linux input driver subsystem',
+ 'The kernel development community', 'manual'),
+ ('kernel-hacking/index', 'kernel-hacking.tex', 'Unreliable Guide To Hacking The Linux Kernel',
+ 'The kernel development community', 'manual'),
('media/index', 'media.tex', 'Linux Media Subsystem Documentation',
'The kernel development community', 'manual'),
+ ('networking/index', 'networking.tex', 'Linux Networking Documentation',
+ 'The kernel development community', 'manual'),
+ ('process/index', 'development-process.tex', 'Linux Kernel Development Documentation',
+ 'The kernel development community', 'manual'),
('security/index', 'security.tex', 'The kernel security subsystem manual',
'The kernel development community', 'manual'),
+ ('sh/index', 'sh.tex', 'SuperH architecture implementation manual',
+ 'The kernel development community', 'manual'),
+ ('sound/index', 'sound.tex', 'Linux Sound Subsystem Documentation',
+ 'The kernel development community', 'manual'),
+ ('userspace-api/index', 'userspace-api.tex', 'The Linux kernel user-space API guide',
+ 'The kernel development community', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
1. Objects are opaque pointers. The implementation does not care where they
point (if anywhere) or what they point to (if anything).
-.. note:: Pointers to objects _must_ be zero in the least significant bit.
+
+ .. note::
+
+ Pointers to objects _must_ be zero in the least significant bit.
2. Objects do not need to contain linkage blocks for use by the array. This
permits an object to be located in multiple arrays simultaneously.
void smp_mb__before_atomic(void);
void smp_mb__after_atomic(void);
+Preceding a non-value-returning read-modify-write atomic operation with
+smp_mb__before_atomic() and following it with smp_mb__after_atomic()
+provides the same full ordering that is provided by value-returning
+read-modify-write atomic operations.
+
For example, smp_mb__before_atomic() can be used like so::
obj->dead = 1;
workqueue
genericirq
flexible-arrays
+ librs
Interfaces for kernel debugging
===============================
--- /dev/null
+==========================================
+Reed-Solomon Library Programming Interface
+==========================================
+
+:Author: Thomas Gleixner
+
+Introduction
+============
+
+The generic Reed-Solomon Library provides encoding, decoding and error
+correction functions.
+
+Reed-Solomon codes are used in communication and storage applications to
+ensure data integrity.
+
+This documentation is provided for developers who want to utilize the
+functions provided by the library.
+
+Known Bugs And Assumptions
+==========================
+
+None.
+
+Usage
+=====
+
+This chapter provides examples of how to use the library.
+
+Initializing
+------------
+
+The init function init_rs returns a pointer to an rs decoder structure,
+which holds the necessary information for encoding, decoding and error
+correction with the given polynomial. It either uses an existing
+matching decoder or creates a new one. On creation all the lookup tables
+for fast en/decoding are created. The function may take a while, so make
+sure not to call it in critical code paths.
+
+::
+
+ /* the Reed Solomon control structure */
+ static struct rs_control *rs_decoder;
+
+ /* Symbolsize is 10 (bits)
+ * Primitive polynomial is x^10+x^3+1
+ * first consecutive root is 0
+ * primitive element to generate roots = 1
+ * generator polynomial degree (number of roots) = 6
+ */
+ rs_decoder = init_rs (10, 0x409, 0, 1, 6);
+
+
+Encoding
+--------
+
+The encoder calculates the Reed-Solomon code over the given data length
+and stores the result in the parity buffer. Note that the parity buffer
+must be initialized before calling the encoder.
+
+The expanded data can be inverted on the fly by providing a non-zero
+inversion mask. The expanded data is XOR'ed with the mask. This is used
+e.g. for FLASH ECC, where the all 0xFF is inverted to an all 0x00. The
+Reed-Solomon code for all 0x00 is all 0x00. The code is inverted before
+storing to FLASH so it is 0xFF too. This prevents that reading from an
+erased FLASH results in ECC errors.
+
+The databytes are expanded to the given symbol size on the fly. There is
+no support for encoding continuous bitstreams with a symbol size != 8 at
+the moment. If it is necessary it should be not a big deal to implement
+such functionality.
+
+::
+
+ /* Parity buffer. Size = number of roots */
+ uint16_t par[6];
+ /* Initialize the parity buffer */
+ memset(par, 0, sizeof(par));
+ /* Encode 512 byte in data8. Store parity in buffer par */
+ encode_rs8 (rs_decoder, data8, 512, par, 0);
+
+
+Decoding
+--------
+
+The decoder calculates the syndrome over the given data length and the
+received parity symbols and corrects errors in the data.
+
+If a syndrome is available from a hardware decoder then the syndrome
+calculation is skipped.
+
+The correction of the data buffer can be suppressed by providing a
+correction pattern buffer and an error location buffer to the decoder.
+The decoder stores the calculated error location and the correction
+bitmask in the given buffers. This is useful for hardware decoders which
+use a weird bit ordering scheme.
+
+The databytes are expanded to the given symbol size on the fly. There is
+no support for decoding continuous bitstreams with a symbolsize != 8 at
+the moment. If it is necessary it should be not a big deal to implement
+such functionality.
+
+Decoding with syndrome calculation, direct data correction
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ /* Parity buffer. Size = number of roots */
+ uint16_t par[6];
+ uint8_t data[512];
+ int numerr;
+ /* Receive data */
+ .....
+ /* Receive parity */
+ .....
+ /* Decode 512 byte in data8.*/
+ numerr = decode_rs8 (rs_decoder, data8, par, 512, NULL, 0, NULL, 0, NULL);
+
+
+Decoding with syndrome given by hardware decoder, direct data correction
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ /* Parity buffer. Size = number of roots */
+ uint16_t par[6], syn[6];
+ uint8_t data[512];
+ int numerr;
+ /* Receive data */
+ .....
+ /* Receive parity */
+ .....
+ /* Get syndrome from hardware decoder */
+ .....
+ /* Decode 512 byte in data8.*/
+ numerr = decode_rs8 (rs_decoder, data8, par, 512, syn, 0, NULL, 0, NULL);
+
+
+Decoding with syndrome given by hardware decoder, no direct data correction.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Note: It's not necessary to give data and received parity to the
+decoder.
+
+::
+
+ /* Parity buffer. Size = number of roots */
+ uint16_t par[6], syn[6], corr[8];
+ uint8_t data[512];
+ int numerr, errpos[8];
+ /* Receive data */
+ .....
+ /* Receive parity */
+ .....
+ /* Get syndrome from hardware decoder */
+ .....
+ /* Decode 512 byte in data8.*/
+ numerr = decode_rs8 (rs_decoder, NULL, NULL, 512, syn, 0, errpos, 0, corr);
+ for (i = 0; i < numerr; i++) {
+ do_error_correction_in_your_buffer(errpos[i], corr[i]);
+ }
+
+
+Cleanup
+-------
+
+The function free_rs frees the allocated resources, if the caller is
+the last user of the decoder.
+
+::
+
+ /* Release resources */
+ free_rs(rs_decoder);
+
+
+Structures
+==========
+
+This chapter contains the autogenerated documentation of the structures
+which are used in the Reed-Solomon Library and are relevant for a
+developer.
+
+.. kernel-doc:: include/linux/rslib.h
+ :internal:
+
+Public Functions Provided
+=========================
+
+This chapter contains the autogenerated documentation of the
+Reed-Solomon functions which are exported.
+
+.. kernel-doc:: lib/reed_solomon/reed_solomon.c
+ :export:
+
+Credits
+=======
+
+The library code for encoding and decoding was written by Phil Karn.
+
+::
+
+ Copyright 2002, Phil Karn, KA9Q
+ May be used under the terms of the GNU General Public License (GPL)
+
+
+The wrapper functions and interfaces are written by Thomas Gleixner.
+
+Many users have provided bugfixes, improvements and helping hands for
+testing. Thanks a lot.
+
+The following people have contributed to this document:
+
+Thomas Gleixner\ tglx@linutronix.de
The caller passes a pointer to the following struct with all of the fields
cleared, except for data, datalen and quotalen [see
- Documentation/security/keys.txt].
+ Documentation/security/keys/core.rst].
struct key_preparsed_payload {
char *description;
--- /dev/null
+# -*- coding: utf-8; mode: python -*-
+
+project = 'Linux Kernel Crypto API'
+
+tags.add("subproject")
+
+latex_documents = [
+ ('index', 'crypto-api.tex', 'Linux Kernel Crypto API manual',
+ 'The kernel development community', 'manual'),
+]
kmemleak
kmemcheck
gdb-kernel-debugging
+ kgdb
.. only:: subproject and html
--- /dev/null
+=================================================
+Using kgdb, kdb and the kernel debugger internals
+=================================================
+
+:Author: Jason Wessel
+
+Introduction
+============
+
+The kernel has two different debugger front ends (kdb and kgdb) which
+interface to the debug core. It is possible to use either of the
+debugger front ends and dynamically transition between them if you
+configure the kernel properly at compile and runtime.
+
+Kdb is simplistic shell-style interface which you can use on a system
+console with a keyboard or serial console. You can use it to inspect
+memory, registers, process lists, dmesg, and even set breakpoints to
+stop in a certain location. Kdb is not a source level debugger, although
+you can set breakpoints and execute some basic kernel run control. Kdb
+is mainly aimed at doing some analysis to aid in development or
+diagnosing kernel problems. You can access some symbols by name in
+kernel built-ins or in kernel modules if the code was built with
+``CONFIG_KALLSYMS``.
+
+Kgdb is intended to be used as a source level debugger for the Linux
+kernel. It is used along with gdb to debug a Linux kernel. The
+expectation is that gdb can be used to "break in" to the kernel to
+inspect memory, variables and look through call stack information
+similar to the way an application developer would use gdb to debug an
+application. It is possible to place breakpoints in kernel code and
+perform some limited execution stepping.
+
+Two machines are required for using kgdb. One of these machines is a
+development machine and the other is the target machine. The kernel to
+be debugged runs on the target machine. The development machine runs an
+instance of gdb against the vmlinux file which contains the symbols (not
+a boot image such as bzImage, zImage, uImage...). In gdb the developer
+specifies the connection parameters and connects to kgdb. The type of
+connection a developer makes with gdb depends on the availability of
+kgdb I/O modules compiled as built-ins or loadable kernel modules in the
+test machine's kernel.
+
+Compiling a kernel
+==================
+
+- In order to enable compilation of kdb, you must first enable kgdb.
+
+- The kgdb test compile options are described in the kgdb test suite
+ chapter.
+
+Kernel config options for kgdb
+------------------------------
+
+To enable ``CONFIG_KGDB`` you should look under
+:menuselection:`Kernel hacking --> Kernel debugging` and select
+:menuselection:`KGDB: kernel debugger`.
+
+While it is not a hard requirement that you have symbols in your vmlinux
+file, gdb tends not to be very useful without the symbolic data, so you
+will want to turn on ``CONFIG_DEBUG_INFO`` which is called
+:menuselection:`Compile the kernel with debug info` in the config menu.
+
+It is advised, but not required, that you turn on the
+``CONFIG_FRAME_POINTER`` kernel option which is called :menuselection:`Compile
+the kernel with frame pointers` in the config menu. This option inserts code
+to into the compiled executable which saves the frame information in
+registers or on the stack at different points which allows a debugger
+such as gdb to more accurately construct stack back traces while
+debugging the kernel.
+
+If the architecture that you are using supports the kernel option
+``CONFIG_STRICT_KERNEL_RWX``, you should consider turning it off. This
+option will prevent the use of software breakpoints because it marks
+certain regions of the kernel's memory space as read-only. If kgdb
+supports it for the architecture you are using, you can use hardware
+breakpoints if you desire to run with the ``CONFIG_STRICT_KERNEL_RWX``
+option turned on, else you need to turn off this option.
+
+Next you should choose one of more I/O drivers to interconnect debugging
+host and debugged target. Early boot debugging requires a KGDB I/O
+driver that supports early debugging and the driver must be built into
+the kernel directly. Kgdb I/O driver configuration takes place via
+kernel or module parameters which you can learn more about in the in the
+section that describes the parameter kgdboc.
+
+Here is an example set of ``.config`` symbols to enable or disable for kgdb::
+
+ # CONFIG_STRICT_KERNEL_RWX is not set
+ CONFIG_FRAME_POINTER=y
+ CONFIG_KGDB=y
+ CONFIG_KGDB_SERIAL_CONSOLE=y
+
+Kernel config options for kdb
+-----------------------------
+
+Kdb is quite a bit more complex than the simple gdbstub sitting on top
+of the kernel's debug core. Kdb must implement a shell, and also adds
+some helper functions in other parts of the kernel, responsible for
+printing out interesting data such as what you would see if you ran
+``lsmod``, or ``ps``. In order to build kdb into the kernel you follow the
+same steps as you would for kgdb.
+
+The main config option for kdb is ``CONFIG_KGDB_KDB`` which is called
+:menuselection:`KGDB_KDB: include kdb frontend for kgdb` in the config menu.
+In theory you would have already also selected an I/O driver such as the
+``CONFIG_KGDB_SERIAL_CONSOLE`` interface if you plan on using kdb on a
+serial port, when you were configuring kgdb.
+
+If you want to use a PS/2-style keyboard with kdb, you would select
+``CONFIG_KDB_KEYBOARD`` which is called :menuselection:`KGDB_KDB: keyboard as
+input device` in the config menu. The ``CONFIG_KDB_KEYBOARD`` option is not
+used for anything in the gdb interface to kgdb. The ``CONFIG_KDB_KEYBOARD``
+option only works with kdb.
+
+Here is an example set of ``.config`` symbols to enable/disable kdb::
+
+ # CONFIG_STRICT_KERNEL_RWX is not set
+ CONFIG_FRAME_POINTER=y
+ CONFIG_KGDB=y
+ CONFIG_KGDB_SERIAL_CONSOLE=y
+ CONFIG_KGDB_KDB=y
+ CONFIG_KDB_KEYBOARD=y
+
+Kernel Debugger Boot Arguments
+==============================
+
+This section describes the various runtime kernel parameters that affect
+the configuration of the kernel debugger. The following chapter covers
+using kdb and kgdb as well as providing some examples of the
+configuration parameters.
+
+Kernel parameter: kgdboc
+------------------------
+
+The kgdboc driver was originally an abbreviation meant to stand for
+"kgdb over console". Today it is the primary mechanism to configure how
+to communicate from gdb to kgdb as well as the devices you want to use
+to interact with the kdb shell.
+
+For kgdb/gdb, kgdboc is designed to work with a single serial port. It
+is intended to cover the circumstance where you want to use a serial
+console as your primary console as well as using it to perform kernel
+debugging. It is also possible to use kgdb on a serial port which is not
+designated as a system console. Kgdboc may be configured as a kernel
+built-in or a kernel loadable module. You can only make use of
+``kgdbwait`` and early debugging if you build kgdboc into the kernel as
+a built-in.
+
+Optionally you can elect to activate kms (Kernel Mode Setting)
+integration. When you use kms with kgdboc and you have a video driver
+that has atomic mode setting hooks, it is possible to enter the debugger
+on the graphics console. When the kernel execution is resumed, the
+previous graphics mode will be restored. This integration can serve as a
+useful tool to aid in diagnosing crashes or doing analysis of memory
+with kdb while allowing the full graphics console applications to run.
+
+kgdboc arguments
+~~~~~~~~~~~~~~~~
+
+Usage::
+
+ kgdboc=[kms][[,]kbd][[,]serial_device][,baud]
+
+The order listed above must be observed if you use any of the optional
+configurations together.
+
+Abbreviations:
+
+- kms = Kernel Mode Setting
+
+- kbd = Keyboard
+
+You can configure kgdboc to use the keyboard, and/or a serial device
+depending on if you are using kdb and/or kgdb, in one of the following
+scenarios. The order listed above must be observed if you use any of the
+optional configurations together. Using kms + only gdb is generally not
+a useful combination.
+
+Using loadable module or built-in
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+1. As a kernel built-in:
+
+ Use the kernel boot argument::
+
+ kgdboc=<tty-device>,[baud]
+
+2. As a kernel loadable module:
+
+ Use the command::
+
+ modprobe kgdboc kgdboc=<tty-device>,[baud]
+
+ Here are two examples of how you might format the kgdboc string. The
+ first is for an x86 target using the first serial port. The second
+ example is for the ARM Versatile AB using the second serial port.
+
+ 1. ``kgdboc=ttyS0,115200``
+
+ 2. ``kgdboc=ttyAMA1,115200``
+
+Configure kgdboc at runtime with sysfs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+At run time you can enable or disable kgdboc by echoing a parameters
+into the sysfs. Here are two examples:
+
+1. Enable kgdboc on ttyS0::
+
+ echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc
+
+2. Disable kgdboc::
+
+ echo "" > /sys/module/kgdboc/parameters/kgdboc
+
+.. note::
+
+ You do not need to specify the baud if you are configuring the
+ console on tty which is already configured or open.
+
+More examples
+^^^^^^^^^^^^^
+
+You can configure kgdboc to use the keyboard, and/or a serial device
+depending on if you are using kdb and/or kgdb, in one of the following
+scenarios.
+
+1. kdb and kgdb over only a serial port::
+
+ kgdboc=<serial_device>[,baud]
+
+ Example::
+
+ kgdboc=ttyS0,115200
+
+2. kdb and kgdb with keyboard and a serial port::
+
+ kgdboc=kbd,<serial_device>[,baud]
+
+ Example::
+
+ kgdboc=kbd,ttyS0,115200
+
+3. kdb with a keyboard::
+
+ kgdboc=kbd
+
+4. kdb with kernel mode setting::
+
+ kgdboc=kms,kbd
+
+5. kdb with kernel mode setting and kgdb over a serial port::
+
+ kgdboc=kms,kbd,ttyS0,115200
+
+.. note::
+
+ Kgdboc does not support interrupting the target via the gdb remote
+ protocol. You must manually send a :kbd:`SysRq-G` unless you have a proxy
+ that splits console output to a terminal program. A console proxy has a
+ separate TCP port for the debugger and a separate TCP port for the
+ "human" console. The proxy can take care of sending the :kbd:`SysRq-G`
+ for you.
+
+When using kgdboc with no debugger proxy, you can end up connecting the
+debugger at one of two entry points. If an exception occurs after you
+have loaded kgdboc, a message should print on the console stating it is
+waiting for the debugger. In this case you disconnect your terminal
+program and then connect the debugger in its place. If you want to
+interrupt the target system and forcibly enter a debug session you have
+to issue a :kbd:`Sysrq` sequence and then type the letter :kbd:`g`. Then you
+disconnect the terminal session and connect gdb. Your options if you
+don't like this are to hack gdb to send the :kbd:`SysRq-G` for you as well as
+on the initial connect, or to use a debugger proxy that allows an
+unmodified gdb to do the debugging.
+
+Kernel parameter: ``kgdbwait``
+------------------------------
+
+The Kernel command line option ``kgdbwait`` makes kgdb wait for a
+debugger connection during booting of a kernel. You can only use this
+option if you compiled a kgdb I/O driver into the kernel and you
+specified the I/O driver configuration as a kernel command line option.
+The kgdbwait parameter should always follow the configuration parameter
+for the kgdb I/O driver in the kernel command line else the I/O driver
+will not be configured prior to asking the kernel to use it to wait.
+
+The kernel will stop and wait as early as the I/O driver and
+architecture allows when you use this option. If you build the kgdb I/O
+driver as a loadable kernel module kgdbwait will not do anything.
+
+Kernel parameter: ``kgdbcon``
+-----------------------------
+
+The ``kgdbcon`` feature allows you to see :c:func:`printk` messages inside gdb
+while gdb is connected to the kernel. Kdb does not make use of the kgdbcon
+feature.
+
+Kgdb supports using the gdb serial protocol to send console messages to
+the debugger when the debugger is connected and running. There are two
+ways to activate this feature.
+
+1. Activate with the kernel command line option::
+
+ kgdbcon
+
+2. Use sysfs before configuring an I/O driver::
+
+ echo 1 > /sys/module/kgdb/parameters/kgdb_use_con
+
+.. note::
+
+ If you do this after you configure the kgdb I/O driver, the
+ setting will not take effect until the next point the I/O is
+ reconfigured.
+
+.. important::
+
+ You cannot use kgdboc + kgdbcon on a tty that is an
+ active system console. An example of incorrect usage is::
+
+ console=ttyS0,115200 kgdboc=ttyS0 kgdbcon
+
+It is possible to use this option with kgdboc on a tty that is not a
+system console.
+
+Run time parameter: ``kgdbreboot``
+----------------------------------
+
+The kgdbreboot feature allows you to change how the debugger deals with
+the reboot notification. You have 3 choices for the behavior. The
+default behavior is always set to 0.
+
+.. tabularcolumns:: |p{0.4cm}|p{11.5cm}|p{5.6cm}|
+
+.. flat-table::
+ :widths: 1 10 8
+
+ * - 1
+ - ``echo -1 > /sys/module/debug_core/parameters/kgdbreboot``
+ - Ignore the reboot notification entirely.
+
+ * - 2
+ - ``echo 0 > /sys/module/debug_core/parameters/kgdbreboot``
+ - Send the detach message to any attached debugger client.
+
+ * - 3
+ - ``echo 1 > /sys/module/debug_core/parameters/kgdbreboot``
+ - Enter the debugger on reboot notify.
+
+Using kdb
+=========
+
+Quick start for kdb on a serial port
+------------------------------------
+
+This is a quick example of how to use kdb.
+
+1. Configure kgdboc at boot using kernel parameters::
+
+ console=ttyS0,115200 kgdboc=ttyS0,115200
+
+ OR
+
+ Configure kgdboc after the kernel has booted; assuming you are using
+ a serial port console::
+
+ echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc
+
+2. Enter the kernel debugger manually or by waiting for an oops or
+ fault. There are several ways you can enter the kernel debugger
+ manually; all involve using the :kbd:`SysRq-G`, which means you must have
+ enabled ``CONFIG_MAGIC_SysRq=y`` in your kernel config.
+
+ - When logged in as root or with a super user session you can run::
+
+ echo g > /proc/sysrq-trigger
+
+ - Example using minicom 2.2
+
+ Press: :kbd:`CTRL-A` :kbd:`f` :kbd:`g`
+
+ - When you have telneted to a terminal server that supports sending
+ a remote break
+
+ Press: :kbd:`CTRL-]`
+
+ Type in: ``send break``
+
+ Press: :kbd:`Enter` :kbd:`g`
+
+3. From the kdb prompt you can run the ``help`` command to see a complete
+ list of the commands that are available.
+
+ Some useful commands in kdb include:
+
+ =========== =================================================================
+ ``lsmod`` Shows where kernel modules are loaded
+ ``ps`` Displays only the active processes
+ ``ps A`` Shows all the processes
+ ``summary`` Shows kernel version info and memory usage
+ ``bt`` Get a backtrace of the current process using :c:func:`dump_stack`
+ ``dmesg`` View the kernel syslog buffer
+ ``go`` Continue the system
+ =========== =================================================================
+
+4. When you are done using kdb you need to consider rebooting the system
+ or using the ``go`` command to resuming normal kernel execution. If you
+ have paused the kernel for a lengthy period of time, applications
+ that rely on timely networking or anything to do with real wall clock
+ time could be adversely affected, so you should take this into
+ consideration when using the kernel debugger.
+
+Quick start for kdb using a keyboard connected console
+------------------------------------------------------
+
+This is a quick example of how to use kdb with a keyboard.
+
+1. Configure kgdboc at boot using kernel parameters::
+
+ kgdboc=kbd
+
+ OR
+
+ Configure kgdboc after the kernel has booted::
+
+ echo kbd > /sys/module/kgdboc/parameters/kgdboc
+
+2. Enter the kernel debugger manually or by waiting for an oops or
+ fault. There are several ways you can enter the kernel debugger
+ manually; all involve using the :kbd:`SysRq-G`, which means you must have
+ enabled ``CONFIG_MAGIC_SysRq=y`` in your kernel config.
+
+ - When logged in as root or with a super user session you can run::
+
+ echo g > /proc/sysrq-trigger
+
+ - Example using a laptop keyboard:
+
+ Press and hold down: :kbd:`Alt`
+
+ Press and hold down: :kbd:`Fn`
+
+ Press and release the key with the label: :kbd:`SysRq`
+
+ Release: :kbd:`Fn`
+
+ Press and release: :kbd:`g`
+
+ Release: :kbd:`Alt`
+
+ - Example using a PS/2 101-key keyboard
+
+ Press and hold down: :kbd:`Alt`
+
+ Press and release the key with the label: :kbd:`SysRq`
+
+ Press and release: :kbd:`g`
+
+ Release: :kbd:`Alt`
+
+3. Now type in a kdb command such as ``help``, ``dmesg``, ``bt`` or ``go`` to
+ continue kernel execution.
+
+Using kgdb / gdb
+================
+
+In order to use kgdb you must activate it by passing configuration
+information to one of the kgdb I/O drivers. If you do not pass any
+configuration information kgdb will not do anything at all. Kgdb will
+only actively hook up to the kernel trap hooks if a kgdb I/O driver is
+loaded and configured. If you unconfigure a kgdb I/O driver, kgdb will
+unregister all the kernel hook points.
+
+All kgdb I/O drivers can be reconfigured at run time, if
+``CONFIG_SYSFS`` and ``CONFIG_MODULES`` are enabled, by echo'ing a new
+config string to ``/sys/module/<driver>/parameter/<option>``. The driver
+can be unconfigured by passing an empty string. You cannot change the
+configuration while the debugger is attached. Make sure to detach the
+debugger with the ``detach`` command prior to trying to unconfigure a
+kgdb I/O driver.
+
+Connecting with gdb to a serial port
+------------------------------------
+
+1. Configure kgdboc
+
+ Configure kgdboc at boot using kernel parameters::
+
+ kgdboc=ttyS0,115200
+
+ OR
+
+ Configure kgdboc after the kernel has booted::
+
+ echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc
+
+2. Stop kernel execution (break into the debugger)
+
+ In order to connect to gdb via kgdboc, the kernel must first be
+ stopped. There are several ways to stop the kernel which include
+ using kgdbwait as a boot argument, via a :kbd:`SysRq-G`, or running the
+ kernel until it takes an exception where it waits for the debugger to
+ attach.
+
+ - When logged in as root or with a super user session you can run::
+
+ echo g > /proc/sysrq-trigger
+
+ - Example using minicom 2.2
+
+ Press: :kbd:`CTRL-A` :kbd:`f` :kbd:`g`
+
+ - When you have telneted to a terminal server that supports sending
+ a remote break
+
+ Press: :kbd:`CTRL-]`
+
+ Type in: ``send break``
+
+ Press: :kbd:`Enter` :kbd:`g`
+
+3. Connect from gdb
+
+ Example (using a directly connected port)::
+
+ % gdb ./vmlinux
+ (gdb) set remotebaud 115200
+ (gdb) target remote /dev/ttyS0
+
+
+ Example (kgdb to a terminal server on TCP port 2012)::
+
+ % gdb ./vmlinux
+ (gdb) target remote 192.168.2.2:2012
+
+
+ Once connected, you can debug a kernel the way you would debug an
+ application program.
+
+ If you are having problems connecting or something is going seriously
+ wrong while debugging, it will most often be the case that you want
+ to enable gdb to be verbose about its target communications. You do
+ this prior to issuing the ``target remote`` command by typing in::
+
+ set debug remote 1
+
+Remember if you continue in gdb, and need to "break in" again, you need
+to issue an other :kbd:`SysRq-G`. It is easy to create a simple entry point by
+putting a breakpoint at ``sys_sync`` and then you can run ``sync`` from a
+shell or script to break into the debugger.
+
+kgdb and kdb interoperability
+=============================
+
+It is possible to transition between kdb and kgdb dynamically. The debug
+core will remember which you used the last time and automatically start
+in the same mode.
+
+Switching between kdb and kgdb
+------------------------------
+
+Switching from kgdb to kdb
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are two ways to switch from kgdb to kdb: you can use gdb to issue
+a maintenance packet, or you can blindly type the command ``$3#33``.
+Whenever the kernel debugger stops in kgdb mode it will print the
+message ``KGDB or $3#33 for KDB``. It is important to note that you have
+to type the sequence correctly in one pass. You cannot type a backspace
+or delete because kgdb will interpret that as part of the debug stream.
+
+1. Change from kgdb to kdb by blindly typing::
+
+ $3#33
+
+2. Change from kgdb to kdb with gdb::
+
+ maintenance packet 3
+
+ .. note::
+
+ Now you must kill gdb. Typically you press :kbd:`CTRL-Z` and issue
+ the command::
+
+ kill -9 %
+
+Change from kdb to kgdb
+~~~~~~~~~~~~~~~~~~~~~~~
+
+There are two ways you can change from kdb to kgdb. You can manually
+enter kgdb mode by issuing the kgdb command from the kdb shell prompt,
+or you can connect gdb while the kdb shell prompt is active. The kdb
+shell looks for the typical first commands that gdb would issue with the
+gdb remote protocol and if it sees one of those commands it
+automatically changes into kgdb mode.
+
+1. From kdb issue the command::
+
+ kgdb
+
+ Now disconnect your terminal program and connect gdb in its place
+
+2. At the kdb prompt, disconnect the terminal program and connect gdb in
+ its place.
+
+Running kdb commands from gdb
+-----------------------------
+
+It is possible to run a limited set of kdb commands from gdb, using the
+gdb monitor command. You don't want to execute any of the run control or
+breakpoint operations, because it can disrupt the state of the kernel
+debugger. You should be using gdb for breakpoints and run control
+operations if you have gdb connected. The more useful commands to run
+are things like lsmod, dmesg, ps or possibly some of the memory
+information commands. To see all the kdb commands you can run
+``monitor help``.
+
+Example::
+
+ (gdb) monitor ps
+ 1 idle process (state I) and
+ 27 sleeping system daemon (state M) processes suppressed,
+ use 'ps A' to see all.
+ Task Addr Pid Parent [*] cpu State Thread Command
+
+ 0xc78291d0 1 0 0 0 S 0xc7829404 init
+ 0xc7954150 942 1 0 0 S 0xc7954384 dropbear
+ 0xc78789c0 944 1 0 0 S 0xc7878bf4 sh
+ (gdb)
+
+kgdb Test Suite
+===============
+
+When kgdb is enabled in the kernel config you can also elect to enable
+the config parameter ``KGDB_TESTS``. Turning this on will enable a special
+kgdb I/O module which is designed to test the kgdb internal functions.
+
+The kgdb tests are mainly intended for developers to test the kgdb
+internals as well as a tool for developing a new kgdb architecture
+specific implementation. These tests are not really for end users of the
+Linux kernel. The primary source of documentation would be to look in
+the ``drivers/misc/kgdbts.c`` file.
+
+The kgdb test suite can also be configured at compile time to run the
+core set of tests by setting the kernel config parameter
+``KGDB_TESTS_ON_BOOT``. This particular option is aimed at automated
+regression testing and does not require modifying the kernel boot config
+arguments. If this is turned on, the kgdb test suite can be disabled by
+specifying ``kgdbts=`` as a kernel boot argument.
+
+Kernel Debugger Internals
+=========================
+
+Architecture Specifics
+----------------------
+
+The kernel debugger is organized into a number of components:
+
+1. The debug core
+
+ The debug core is found in ``kernel/debugger/debug_core.c``. It
+ contains:
+
+ - A generic OS exception handler which includes sync'ing the
+ processors into a stopped state on an multi-CPU system.
+
+ - The API to talk to the kgdb I/O drivers
+
+ - The API to make calls to the arch-specific kgdb implementation
+
+ - The logic to perform safe memory reads and writes to memory while
+ using the debugger
+
+ - A full implementation for software breakpoints unless overridden
+ by the arch
+
+ - The API to invoke either the kdb or kgdb frontend to the debug
+ core.
+
+ - The structures and callback API for atomic kernel mode setting.
+
+ .. note:: kgdboc is where the kms callbacks are invoked.
+
+2. kgdb arch-specific implementation
+
+ This implementation is generally found in ``arch/*/kernel/kgdb.c``. As
+ an example, ``arch/x86/kernel/kgdb.c`` contains the specifics to
+ implement HW breakpoint as well as the initialization to dynamically
+ register and unregister for the trap handlers on this architecture.
+ The arch-specific portion implements:
+
+ - contains an arch-specific trap catcher which invokes
+ :c:func:`kgdb_handle_exception` to start kgdb about doing its work
+
+ - translation to and from gdb specific packet format to :c:type:`pt_regs`
+
+ - Registration and unregistration of architecture specific trap
+ hooks
+
+ - Any special exception handling and cleanup
+
+ - NMI exception handling and cleanup
+
+ - (optional) HW breakpoints
+
+3. gdbstub frontend (aka kgdb)
+
+ The gdbstub is located in ``kernel/debug/gdbstub.c``. It contains:
+
+ - All the logic to implement the gdb serial protocol
+
+4. kdb frontend
+
+ The kdb debugger shell is broken down into a number of components.
+ The kdb core is located in kernel/debug/kdb. There are a number of
+ helper functions in some of the other kernel components to make it
+ possible for kdb to examine and report information about the kernel
+ without taking locks that could cause a kernel deadlock. The kdb core
+ contains implements the following functionality.
+
+ - A simple shell
+
+ - The kdb core command set
+
+ - A registration API to register additional kdb shell commands.
+
+ - A good example of a self-contained kdb module is the ``ftdump``
+ command for dumping the ftrace buffer. See:
+ ``kernel/trace/trace_kdb.c``
+
+ - For an example of how to dynamically register a new kdb command
+ you can build the kdb_hello.ko kernel module from
+ ``samples/kdb/kdb_hello.c``. To build this example you can set
+ ``CONFIG_SAMPLES=y`` and ``CONFIG_SAMPLE_KDB=m`` in your kernel
+ config. Later run ``modprobe kdb_hello`` and the next time you
+ enter the kdb shell, you can run the ``hello`` command.
+
+ - The implementation for :c:func:`kdb_printf` which emits messages directly
+ to I/O drivers, bypassing the kernel log.
+
+ - SW / HW breakpoint management for the kdb shell
+
+5. kgdb I/O driver
+
+ Each kgdb I/O driver has to provide an implementation for the
+ following:
+
+ - configuration via built-in or module
+
+ - dynamic configuration and kgdb hook registration calls
+
+ - read and write character interface
+
+ - A cleanup handler for unconfiguring from the kgdb core
+
+ - (optional) Early debug methodology
+
+ Any given kgdb I/O driver has to operate very closely with the
+ hardware and must do it in such a way that does not enable interrupts
+ or change other parts of the system context without completely
+ restoring them. The kgdb core will repeatedly "poll" a kgdb I/O
+ driver for characters when it needs input. The I/O driver is expected
+ to return immediately if there is no data available. Doing so allows
+ for the future possibility to touch watchdog hardware in such a way
+ as to have a target system not reset when these are enabled.
+
+If you are intent on adding kgdb architecture specific support for a new
+architecture, the architecture should define ``HAVE_ARCH_KGDB`` in the
+architecture specific Kconfig file. This will enable kgdb for the
+architecture, and at that point you must create an architecture specific
+kgdb implementation.
+
+There are a few flags which must be set on every architecture in their
+``asm/kgdb.h`` file. These are:
+
+- ``NUMREGBYTES``:
+ The size in bytes of all of the registers, so that we
+ can ensure they will all fit into a packet.
+
+- ``BUFMAX``:
+ The size in bytes of the buffer GDB will read into. This must
+ be larger than NUMREGBYTES.
+
+- ``CACHE_FLUSH_IS_SAFE``:
+ Set to 1 if it is always safe to call
+ flush_cache_range or flush_icache_range. On some architectures,
+ these functions may not be safe to call on SMP since we keep other
+ CPUs in a holding pattern.
+
+There are also the following functions for the common backend, found in
+``kernel/kgdb.c``, that must be supplied by the architecture-specific
+backend unless marked as (optional), in which case a default function
+maybe used if the architecture does not need to provide a specific
+implementation.
+
+.. kernel-doc:: include/linux/kgdb.h
+ :internal:
+
+kgdboc internals
+----------------
+
+kgdboc and uarts
+~~~~~~~~~~~~~~~~
+
+The kgdboc driver is actually a very thin driver that relies on the
+underlying low level to the hardware driver having "polling hooks" to
+which the tty driver is attached. In the initial implementation of
+kgdboc the serial_core was changed to expose a low level UART hook for
+doing polled mode reading and writing of a single character while in an
+atomic context. When kgdb makes an I/O request to the debugger, kgdboc
+invokes a callback in the serial core which in turn uses the callback in
+the UART driver.
+
+When using kgdboc with a UART, the UART driver must implement two
+callbacks in the :c:type:`struct uart_ops <uart_ops>`.
+Example from ``drivers/8250.c``::
+
+
+ #ifdef CONFIG_CONSOLE_POLL
+ .poll_get_char = serial8250_get_poll_char,
+ .poll_put_char = serial8250_put_poll_char,
+ #endif
+
+
+Any implementation specifics around creating a polling driver use the
+``#ifdef CONFIG_CONSOLE_POLL``, as shown above. Keep in mind that
+polling hooks have to be implemented in such a way that they can be
+called from an atomic context and have to restore the state of the UART
+chip on return such that the system can return to normal when the
+debugger detaches. You need to be very careful with any kind of lock you
+consider, because failing here is most likely going to mean pressing the
+reset button.
+
+kgdboc and keyboards
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The kgdboc driver contains logic to configure communications with an
+attached keyboard. The keyboard infrastructure is only compiled into the
+kernel when ``CONFIG_KDB_KEYBOARD=y`` is set in the kernel configuration.
+
+The core polled keyboard driver driver for PS/2 type keyboards is in
+``drivers/char/kdb_keyboard.c``. This driver is hooked into the debug core
+when kgdboc populates the callback in the array called
+:c:type:`kdb_poll_funcs[]`. The :c:func:`kdb_get_kbd_char` is the top-level
+function which polls hardware for single character input.
+
+kgdboc and kms
+~~~~~~~~~~~~~~~~~~
+
+The kgdboc driver contains logic to request the graphics display to
+switch to a text context when you are using ``kgdboc=kms,kbd``, provided
+that you have a video driver which has a frame buffer console and atomic
+kernel mode setting support.
+
+Every time the kernel debugger is entered it calls
+:c:func:`kgdboc_pre_exp_handler` which in turn calls :c:func:`con_debug_enter`
+in the virtual console layer. On resuming kernel execution, the kernel
+debugger calls :c:func:`kgdboc_post_exp_handler` which in turn calls
+:c:func:`con_debug_leave`.
+
+Any video driver that wants to be compatible with the kernel debugger
+and the atomic kms callbacks must implement the ``mode_set_base_atomic``,
+``fb_debug_enter`` and ``fb_debug_leave operations``. For the
+``fb_debug_enter`` and ``fb_debug_leave`` the option exists to use the
+generic drm fb helper functions or implement something custom for the
+hardware. The following example shows the initialization of the
+.mode_set_base_atomic operation in
+drivers/gpu/drm/i915/intel_display.c::
+
+
+ static const struct drm_crtc_helper_funcs intel_helper_funcs = {
+ [...]
+ .mode_set_base_atomic = intel_pipe_set_base_atomic,
+ [...]
+ };
+
+
+Here is an example of how the i915 driver initializes the
+fb_debug_enter and fb_debug_leave functions to use the generic drm
+helpers in ``drivers/gpu/drm/i915/intel_fb.c``::
+
+
+ static struct fb_ops intelfb_ops = {
+ [...]
+ .fb_debug_enter = drm_fb_helper_debug_enter,
+ .fb_debug_leave = drm_fb_helper_debug_leave,
+ [...]
+ };
+
+
+Credits
+=======
+
+The following people have contributed to this document:
+
+1. Amit Kale <amitkale@linsyssoft.com>
+
+2. Tom Rini <trini@kernel.crashing.org>
+
+In March 2008 this document was completely rewritten by:
+
+- Jason Wessel <jason.wessel@windriver.com>
+
+In Jan 2010 this document was updated to include kdb.
+
+- Jason Wessel <jason.wessel@windriver.com>
The optional make variable CF can be used to pass arguments to sparse. The
build system passes -Wbitwise to sparse automatically.
-
-Checking RCU annotations
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-RCU annotations are not checked by default. To enable RCU annotation
-checks, include -DCONFIG_SPARSE_RCU_POINTER in your CF flags.
--- /dev/null
+* CoreSight CPU Debug Component:
+
+CoreSight CPU debug component are compliant with the ARMv8 architecture
+reference manual (ARM DDI 0487A.k) Chapter 'Part H: External debug'. The
+external debug module is mainly used for two modes: self-hosted debug and
+external debug, and it can be accessed from mmio region from Coresight
+and eventually the debug module connects with CPU for debugging. And the
+debug module provides sample-based profiling extension, which can be used
+to sample CPU program counter, secure state and exception level, etc;
+usually every CPU has one dedicated debug module to be connected.
+
+Required properties:
+
+- compatible : should be "arm,coresight-cpu-debug"; supplemented with
+ "arm,primecell" since this driver is using the AMBA bus
+ interface.
+
+- reg : physical base address and length of the register set.
+
+- clocks : the clock associated to this component.
+
+- clock-names : the name of the clock referenced by the code. Since we are
+ using the AMBA framework, the name of the clock providing
+ the interconnect should be "apb_pclk" and the clock is
+ mandatory. The interface between the debug logic and the
+ processor core is clocked by the internal CPU clock, so it
+ is enabled with CPU clock by default.
+
+- cpu : the CPU phandle the debug module is affined to. When omitted
+ the module is considered to belong to CPU0.
+
+Optional properties:
+
+- power-domains: a phandle to the debug power domain. We use "power-domains"
+ binding to turn on the debug logic if it has own dedicated
+ power domain and if necessary to use "cpuidle.off=1" or
+ "nohlt" in the kernel command line or sysfs node to
+ constrain idle states to ensure registers in the CPU power
+ domain are accessible.
+
+Example:
+
+ debug@f6590000 {
+ compatible = "arm,coresight-cpu-debug","arm,primecell";
+ reg = <0 0xf6590000 0 0x1000>;
+ clocks = <&sys_ctrl HI6220_DAPB_CLK>;
+ clock-names = "apb_pclk";
+ cpu = <&cpu0>;
+ };
Usage: Optional
Value type: <u32>
Definition:
- # u32 value representing CPU capacity [3] in
+ # u32 value representing CPU capacity [4] in
DMIPS/MHz, relative to highest capacity-dmips-mhz
in the system.
[2] arm/msm/qcom,kpss-acc.txt
[3] ARM Linux kernel documentation - idle states bindings
Documentation/devicetree/bindings/arm/idle-states.txt
-[3] ARM Linux kernel documentation - cpu capacity bindings
+[4] ARM Linux kernel documentation - cpu capacity bindings
Documentation/devicetree/bindings/arm/cpu-capacity.txt
- #clock-cells : must contain 1
- #reset-cells : must contain 1
-For the PRCM CCUs on H3/A64, one more clock is needed:
+For the PRCM CCUs on H3/A64, two more clocks are needed:
+- "pll-periph": the SoC's peripheral PLL from the main CCU
- "iosc": the SoC's internal frequency oscillator
Example for generic CCU:
r_ccu: clock@01f01400 {
compatible = "allwinner,sun50i-a64-r-ccu";
reg = <0x01f01400 0x100>;
- clocks = <&osc24M>, <&osc32k>, <&iosc>;
- clock-names = "hosc", "losc", "iosc";
+ clocks = <&osc24M>, <&osc32k>, <&iosc>, <&ccu CLK_PLL_PERIPH0>;
+ clock-names = "hosc", "losc", "iosc", "pll-periph";
#clock-cells = <1>;
#reset-cells = <1>;
};
--- /dev/null
+Device-tree bindings for gpio-based FSI master driver
+-----------------------------------------------------
+
+Required properties:
+ - compatible = "fsi-master-gpio";
+ - clock-gpios = <gpio-descriptor>; : GPIO for FSI clock
+ - data-gpios = <gpio-descriptor>; : GPIO for FSI data signal
+
+Optional properties:
+ - enable-gpios = <gpio-descriptor>; : GPIO for enable signal
+ - trans-gpios = <gpio-descriptor>; : GPIO for voltage translator enable
+ - mux-gpios = <gpio-descriptor>; : GPIO for pin multiplexing with other
+ functions (eg, external FSI masters)
+
+Examples:
+
+ fsi-master {
+ compatible = "fsi-master-gpio", "fsi-master";
+ clock-gpios = <&gpio 0>;
+ data-gpios = <&gpio 1>;
+ enable-gpios = <&gpio 2>;
+ trans-gpios = <&gpio 3>;
+ mux-gpios = <&gpio 4>;
+ }
Optional properties:
In order to use the GPIO lines in PWM mode, some additional optional
-properties are required. Only Armada 370 and XP support these properties.
+properties are required.
-- compatible: Must contain "marvell,armada-370-xp-gpio"
+- compatible: Must contain "marvell,armada-370-gpio"
- reg: an additional register set is needed, for the GPIO Blink
Counter on/off registers.
};
gpio1: gpio@18140 {
- compatible = "marvell,armada-370-xp-gpio";
+ compatible = "marvell,armada-370-gpio";
reg = <0x18140 0x40>, <0x181c8 0x08>;
reg-names = "gpio", "pwm";
ngpios = <17>;
--- /dev/null
+General Purpose I2C Bus Mux
+
+This binding describes an I2C bus multiplexer that uses a mux controller
+from the mux subsystem to route the I2C signals.
+
+ .-----. .-----.
+ | dev | | dev |
+ .------------. '-----' '-----'
+ | SoC | | |
+ | | .--------+--------'
+ | .------. | .------+ child bus A, on MUX value set to 0
+ | | I2C |-|--| Mux |
+ | '------' | '--+---+ child bus B, on MUX value set to 1
+ | .------. | | '----------+--------+--------.
+ | | MUX- | | | | | |
+ | | Ctrl |-|-----+ .-----. .-----. .-----.
+ | '------' | | dev | | dev | | dev |
+ '------------' '-----' '-----' '-----'
+
+Required properties:
+- compatible: i2c-mux
+- i2c-parent: The phandle of the I2C bus that this multiplexer's master-side
+ port is connected to.
+- mux-controls: The phandle of the mux controller to use for operating the
+ mux.
+* Standard I2C mux properties. See i2c-mux.txt in this directory.
+* I2C child bus nodes. See i2c-mux.txt in this directory. The sub-bus number
+ is also the mux-controller state described in ../mux/mux-controller.txt
+
+Optional properties:
+- mux-locked: If present, explicitly allow unrelated I2C transactions on the
+ parent I2C adapter at these times:
+ + during setup of the multiplexer
+ + between setup of the multiplexer and the child bus I2C transaction
+ + between the child bus I2C transaction and releasing of the multiplexer
+ + during releasing of the multiplexer
+ However, I2C transactions to devices behind all I2C multiplexers connected
+ to the same parent adapter that this multiplexer is connected to are blocked
+ for the full duration of the complete multiplexed I2C transaction (i.e.
+ including the times covered by the above list).
+ If mux-locked is not present, the multiplexer is assumed to be parent-locked.
+ This means that no unrelated I2C transactions are allowed on the parent I2C
+ adapter for the complete multiplexed I2C transaction.
+ The properties of mux-locked and parent-locked multiplexers are discussed
+ in more detail in Documentation/i2c/i2c-topology.
+
+For each i2c child node, an I2C child bus will be created. They will
+be numbered based on their order in the device tree.
+
+Whenever an access is made to a device on a child bus, the value set
+in the relevant node's reg property will be set as the state in the
+mux controller.
+
+Example:
+ mux: mux-controller {
+ compatible = "gpio-mux";
+ #mux-control-cells = <0>;
+
+ mux-gpios = <&pioA 0 GPIO_ACTIVE_HIGH>,
+ <&pioA 1 GPIO_ACTIVE_HIGH>;
+ };
+
+ i2c-mux {
+ compatible = "i2c-mux";
+ mux-locked;
+ i2c-parent = <&i2c1>;
+
+ mux-controls = <&mux>;
+
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ i2c@1 {
+ reg = <1>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ ssd1307: oled@3c {
+ compatible = "solomon,ssd1307fb-i2c";
+ reg = <0x3c>;
+ pwms = <&pwm 4 3000>;
+ reset-gpios = <&gpio2 7 1>;
+ reset-active-low;
+ };
+ };
+
+ i2c@3 {
+ reg = <3>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ pca9555: pca9555@20 {
+ compatible = "nxp,pca9555";
+ gpio-controller;
+ #gpio-cells = <2>;
+ reg = <0x20>;
+ };
+ };
+ };
Required properties:
- compatible: depending on the SoC this should be one of:
+ - "amlogic,meson8-saradc" for Meson8
+ - "amlogic,meson8b-saradc" for Meson8b
- "amlogic,meson-gxbb-saradc" for GXBB
- "amlogic,meson-gxl-saradc" for GXL
- "amlogic,meson-gxm-saradc" for GXM
-* Renesas RCar GyroADC device driver
+* Renesas R-Car GyroADC device driver
The GyroADC block is a reduced SPI block with up to 8 chipselect lines,
which supports the SPI protocol of a selected few SPI ADCs. The SPI ADCs
- clocks: References to all the clocks specified in the clock-names
property as specified in
Documentation/devicetree/bindings/clock/clock-bindings.txt.
-- clock-names: Shall contain "fck" and "if". The "fck" is the GyroADC block
- clock, the "if" is the interface clock.
+- clock-names: Shall contain "fck". The "fck" is the GyroADC block clock.
- power-domains: Must contain a reference to the PM domain, if available.
- #address-cells: Should be <1> (setting for the subnodes) for all ADCs
except for "fujitsu,mb88101a". Should be <0> (setting for
adc@e6e54000 {
compatible = "renesas,r8a7791-gyroadc", "renesas,rcar-gyroadc";
reg = <0 0xe6e54000 0 64>;
- clocks = <&mstp9_clks R8A7791_CLK_GYROADC>, <&clk_65m>;
- clock-names = "fck", "if";
+ clocks = <&mstp9_clks R8A7791_CLK_GYROADC>;
+ clock-names = "fck";
power-domains = <&sysc R8A7791_PD_ALWAYS_ON>;
pinctrl-0 = <&adc_pins>;
Contents of a stm32 adc root node:
-----------------------------------
Required properties:
-- compatible: Should be "st,stm32f4-adc-core".
+- compatible: Should be one of:
+ "st,stm32f4-adc-core"
+ "st,stm32h7-adc-core"
- reg: Offset and length of the ADC block register set.
- interrupts: Must contain the interrupt for ADC block.
-- clocks: Clock for the analog circuitry (common to all ADCs).
-- clock-names: Must be "adc".
+- clocks: Core can use up to two clocks, depending on part used:
+ - "adc" clock: for the analog circuitry, common to all ADCs.
+ It's required on stm32f4.
+ It's optional on stm32h7.
+ - "bus" clock: for registers access, common to all ADCs.
+ It's not present on stm32f4.
+ It's required on stm32h7.
+- clock-names: Must be "adc" and/or "bus" depending on part used.
- interrupt-controller: Identifies the controller node as interrupt-parent
- vref-supply: Phandle to the vref input analog reference voltage.
- #interrupt-cells = <1>;
ADC instance available on the machine.
Required properties:
-- compatible: Should be "st,stm32f4-adc".
+- compatible: Should be one of:
+ "st,stm32f4-adc"
+ "st,stm32h7-adc"
- reg: Offset of ADC instance in ADC block (e.g. may be 0x0, 0x100, 0x200).
-- clocks: Input clock private to this ADC instance.
+- clocks: Input clock private to this ADC instance. It's required only on
+ stm32f4, that has per instance clock input for registers access.
- interrupt-parent: Phandle to the parent interrupt controller.
- interrupts: IRQ Line for the ADC (e.g. may be 0 for adc@0, 1 for adc@100 or
2 for adc@200).
- st,adc-channels: List of single-ended channels muxed for this ADC.
- It can have up to 16 channels, numbered from 0 to 15 (resp. for in0..in15).
+ It can have up to 16 channels on stm32f4 or 20 channels on stm32h7, numbered
+ from 0 to 15 or 19 (resp. for in0..in15 or in0..in19).
- #io-channel-cells = <1>: See the IIO bindings section "IIO consumers" in
Documentation/devicetree/bindings/iio/iio-bindings.txt
See ../../dma/dma.txt for details.
- dma-names: Must be "rx" when dmas property is being used.
- assigned-resolution-bits: Resolution (bits) to use for conversions. Must
- match device available resolutions (e.g. can be 6, 8, 10 or 12 on stm32f4).
+ match device available resolutions:
+ * can be 6, 8, 10 or 12 on stm32f4
+ * can be 8, 10, 12, 14 or 16 on stm32h7
Default is maximum resolution if unset.
Example:
--- /dev/null
+* Texas Instruments' ADC084S021
+
+Required properties:
+ - compatible : Must be "ti,adc084s021"
+ - reg : SPI chip select number for the device
+ - vref-supply : The regulator supply for ADC reference voltage
+ - spi-cpol : Per spi-bus bindings
+ - spi-cpha : Per spi-bus bindings
+ - spi-max-frequency : Per spi-bus bindings
+
+Example:
+adc@0 {
+ compatible = "ti,adc084s021";
+ reg = <0>;
+ vref-supply = <&adc_vref>;
+ spi-cpol;
+ spi-cpha;
+ spi-max-frequency = <16000000>;
+};
--- /dev/null
+* Texas Instruments' ADC108S102 and ADC128S102 ADC chip
+
+Required properties:
+ - compatible: Should be "ti,adc108s102"
+ - reg: spi chip select number for the device
+ - vref-supply: The regulator supply for ADC reference voltage
+
+Recommended properties:
+ - spi-max-frequency: Definition as per
+ Documentation/devicetree/bindings/spi/spi-bus.txt
+
+Example:
+adc@0 {
+ compatible = "ti,adc108s102";
+ reg = <0>;
+ vref-supply = <&vdd_supply>;
+ spi-max-frequency = <1000000>;
+};
"data ready" (valid values: 1 or 2).
- interrupt-parent: should be the phandle for the interrupt controller
- interrupts: interrupt mapping for IRQ. It should be configured with
- flags IRQ_TYPE_LEVEL_HIGH or IRQ_TYPE_EDGE_RISING.
+ flags IRQ_TYPE_LEVEL_HIGH, IRQ_TYPE_EDGE_RISING, IRQ_TYPE_LEVEL_LOW or
+ IRQ_TYPE_EDGE_FALLING.
Refer to interrupt-controller/interrupts.txt for generic interrupt
client node bindings.
--- /dev/null
+I/O channel multiplexer bindings
+
+If a multiplexer is used to select which hardware signal is fed to
+e.g. an ADC channel, these bindings describe that situation.
+
+Required properties:
+- compatible : "io-channel-mux"
+- io-channels : Channel node of the parent channel that has multiplexed
+ input.
+- io-channel-names : Should be "parent".
+- #address-cells = <1>;
+- #size-cells = <0>;
+- mux-controls : Mux controller node to use for operating the mux
+- channels : List of strings, labeling the mux controller states.
+
+For each non-empty string in the channels property, an io-channel will
+be created. The number of this io-channel is the same as the index into
+the list of strings in the channels property, and also matches the mux
+controller state. The mux controller state is described in
+../mux/mux-controller.txt
+
+Example:
+ mux: mux-controller {
+ compatible = "mux-gpio";
+ #mux-control-cells = <0>;
+
+ mux-gpios = <&pioA 0 GPIO_ACTIVE_HIGH>,
+ <&pioA 1 GPIO_ACTIVE_HIGH>;
+ };
+
+ adc-mux {
+ compatible = "io-channel-mux";
+ io-channels = <&adc 0>;
+ io-channel-names = "parent";
+
+ mux-controls = <&mux>;
+
+ channels = "sync", "in", "system-regulator";
+ };
Required properties:
-- compatible : should be "allwinner,sun7i-a20-sc-nmi" or
- "allwinner,sun6i-a31-sc-nmi" or "allwinner,sun9i-a80-nmi"
+- compatible : should be one of the following:
+ - "allwinner,sun7i-a20-sc-nmi"
+ - "allwinner,sun6i-a31-sc-nmi" (deprecated)
+ - "allwinner,sun6i-a31-r-intc"
+ - "allwinner,sun9i-a80-nmi"
- reg : Specifies base physical address and size of the registers.
- interrupt-controller : Identifies the node as an interrupt controller
- #interrupt-cells : Specifies the number of cells needed to encode an
--- /dev/null
+Device tree configuration for the I2C Interrupt Controller on the AST24XX and
+AST25XX SoCs.
+
+Required Properties:
+- #address-cells : should be 1
+- #size-cells : should be 1
+- #interrupt-cells : should be 1
+- compatible : should be "aspeed,ast2400-i2c-ic"
+ or "aspeed,ast2500-i2c-ic"
+- reg : address start and range of controller
+- interrupts : interrupt number
+- interrupt-controller : denotes that the controller receives and fires
+ new interrupts for child busses
+
+Example:
+
+i2c_ic: interrupt-controller@0 {
+ #address-cells = <1>;
+ #size-cells = <1>;
+ #interrupt-cells = <1>;
+ compatible = "aspeed,ast2400-i2c-ic";
+ reg = <0x0 0x40>;
+ interrupts = <12>;
+ interrupt-controller;
+};
Aspeed Vectored Interrupt Controller
-These bindings are for the Aspeed AST2400 interrupt controller register layout.
-The SoC has an legacy register layout, but this driver does not support that
-mode of operation.
+These bindings are for the Aspeed interrupt controller. The AST2400 and
+AST2500 SoC families include a legacy register layout before a re-designed
+layout, but the bindings do not prescribe the use of one or the other.
Required properties:
-- compatible : should be "aspeed,ast2400-vic".
+- compatible : "aspeed,ast2400-vic"
+ "aspeed,ast2500-vic"
- interrupt-controller : Identifies the node as an interrupt controller
- #interrupt-cells : Specifies the number of cells needed to encode an
--- /dev/null
+Marvell GICP Controller
+-----------------------
+
+GICP is a Marvell extension of the GIC that allows to trigger GIC SPI
+interrupts by doing a memory transaction. It is used by the ICU
+located in the Marvell CP110 to turn wired interrupts inside the CP
+into GIC SPI interrupts.
+
+Required properties:
+
+- compatible: Must be "marvell,ap806-gicp"
+
+- reg: Must be the address and size of the GICP SPI registers
+
+- marvell,spi-ranges: tuples of GIC SPI interrupts ranges available
+ for this GICP
+
+- msi-controller: indicates that this is an MSI controller
+
+Example:
+
+gicp_spi: gicp-spi@3f0040 {
+ compatible = "marvell,ap806-gicp";
+ reg = <0x3f0040 0x10>;
+ marvell,spi-ranges = <64 64>, <288 64>;
+ msi-controller;
+};
--- /dev/null
+Marvell ICU Interrupt Controller
+--------------------------------
+
+The Marvell ICU (Interrupt Consolidation Unit) controller is
+responsible for collecting all wired-interrupt sources in the CP and
+communicating them to the GIC in the AP, the unit translates interrupt
+requests on input wires to MSG memory mapped transactions to the GIC.
+
+Required properties:
+
+- compatible: Should be "marvell,cp110-icu"
+
+- reg: Should contain ICU registers location and length.
+
+- #interrupt-cells: Specifies the number of cells needed to encode an
+ interrupt source. The value shall be 3.
+
+ The 1st cell is the group type of the ICU interrupt. Possible group
+ types are:
+
+ ICU_GRP_NSR (0x0) : Shared peripheral interrupt, non-secure
+ ICU_GRP_SR (0x1) : Shared peripheral interrupt, secure
+ ICU_GRP_SEI (0x4) : System error interrupt
+ ICU_GRP_REI (0x5) : RAM error interrupt
+
+ The 2nd cell is the index of the interrupt in the ICU unit.
+
+ The 3rd cell is the type of the interrupt. See arm,gic.txt for
+ details.
+
+- interrupt-controller: Identifies the node as an interrupt
+ controller.
+
+- msi-parent: Should point to the GICP controller, the GIC extension
+ that allows to trigger interrupts using MSG memory mapped
+ transactions.
+
+Example:
+
+icu: interrupt-controller@1e0000 {
+ compatible = "marvell,cp110-icu";
+ reg = <0x1e0000 0x10>;
+ #interrupt-cells = <3>;
+ interrupt-controller;
+ msi-parent = <&gicp>;
+};
+
+usb3h0: usb3@500000 {
+ interrupt-parent = <&icu>;
+ interrupts = <ICU_GRP_NSR 106 IRQ_TYPE_LEVEL_HIGH>;
+};
-Common leds properties.
+* Common leds properties.
LED and flash LED devices provide the same basic functionality as current
regulators, but extended with LED and flash LED specific features like
- panic-indicator : This property specifies that the LED should be used,
if at all possible, as a panic indicator.
+- trigger-sources : List of devices which should be used as a source triggering
+ this LED activity. Some LEDs can be related to a specific
+ device and should somehow indicate its state. E.g. USB 2.0
+ LED may react to device(s) in a USB 2.0 port(s).
+ Another common example is switch or router with multiple
+ Ethernet ports each of them having its own LED assigned
+ (assuming they are not hardwired). In such cases this
+ property should contain phandle(s) of related source
+ device(s).
+ In many cases LED can be related to more than one device
+ (e.g. one USB LED vs. multiple USB ports). Each source
+ should be represented by a node in the device tree and be
+ referenced by a phandle and a set of phandle arguments. A
+ length of arguments should be specified by the
+ #trigger-source-cells property in the source node.
+
Required properties for flash LED child nodes:
- flash-max-microamp : Maximum flash LED supply current in microamperes.
- flash-max-timeout-us : Maximum timeout in microseconds after which the flash
For controllers that have no configurable timeout the flash-max-timeout-us
property can be omitted.
-Examples:
+* Trigger source providers
+
+Each trigger source should be represented by a device tree node. It may be e.g.
+a USB port or an Ethernet device.
+
+Required properties for trigger source:
+- #trigger-source-cells : Number of cells in a source trigger. Typically 0 for
+ nodes of simple trigger sources (e.g. a specific USB
+ port).
+
+* Examples
gpio-leds {
compatible = "gpio-leds";
linux,default-trigger = "heartbeat";
gpios = <&gpio0 0 GPIO_ACTIVE_HIGH>;
};
+
+ usb {
+ gpios = <&gpio0 1 GPIO_ACTIVE_HIGH>;
+ trigger-sources = <&ohci_port1>, <&ehci_port1>;
+ };
};
max77693-led {
compatible = "st,stm32-timers";
reg = <0x40010000 0x400>;
clocks = <&rcc 0 160>;
- clock-names = "clk_int";
+ clock-names = "int";
pwm {
compatible = "st,stm32-pwm";
There should be 9 entries here, one for each gpio.
- ti,system-power-controller: Telling whether or not this pmic is controlling
the system power.
+- ti,sleep-enable: Enable SLEEP state.
+- ti,sleep-keep-therm: Keep thermal monitoring on in sleep state.
+- ti,sleep-keep-ck32k: Keep the 32KHz clock output on in sleep state.
+- ti,sleep-keep-hsclk: Keep high speed internal clock on in sleep state.
Regulator Optional properties:
- ti,regulator-ext-sleep-control: enable external sleep
- "rockchip,rk3288-dw-mshc": for Rockchip RK3288
- "rockchip,rv1108-dw-mshc", "rockchip,rk3288-dw-mshc": for Rockchip RV1108
- "rockchip,rk3036-dw-mshc", "rockchip,rk3288-dw-mshc": for Rockchip RK3036
+ - "rockchip,rk3328-dw-mshc", "rockchip,rk3288-dw-mshc": for Rockchip RK3328
- "rockchip,rk3368-dw-mshc", "rockchip,rk3288-dw-mshc": for Rockchip RK3368
- "rockchip,rk3399-dw-mshc", "rockchip,rk3288-dw-mshc": for Rockchip RK3399
probing, low speeds or in case where all phases work at tuning time.
If not specified 0 deg will be used.
+* rockchip,desired-num-phases: The desired number of times that the host
+ execute tuning when needed. If not specified, the host will do tuning
+ for 360 times, namely tuning for each degree.
+
Example:
rkdwmmc0@12200000 {
Optional properties:
ti,dual-volt: boolean, supports dual voltage cards
<supply-name>-supply: phandle to the regulator device tree node
-"supply-name" examples are "vmmc", "vmmc_aux" etc
+"supply-name" examples are "vmmc", "vmmc_aux"(deprecated)/"vqmmc" etc
ti,non-removable: non-removable slot (like eMMC)
ti,needs-special-reset: Requires a special softreset sequence
ti,needs-special-hs-handling: HSMMC IP needs special setting for handling High Speed
--- /dev/null
+Bindings for Analog Devices ADG792A/G Triple 4:1 Multiplexers
+
+Required properties:
+- compatible : "adi,adg792a" or "adi,adg792g"
+- #mux-control-cells : <0> if parallel (the three muxes are bound together
+ with a single mux controller controlling all three muxes), or <1> if
+ not (one mux controller for each mux).
+* Standard mux-controller bindings as described in mux-controller.txt
+
+Optional properties for ADG792G:
+- gpio-controller : if present, #gpio-cells below is required.
+- #gpio-cells : should be <2>
+ - First cell is the GPO line number, i.e. 0 or 1
+ - Second cell is used to specify active high (0)
+ or active low (1)
+
+Optional properties:
+- idle-state : if present, array of states that the mux controllers will have
+ when idle. The special state MUX_IDLE_AS_IS is the default and
+ MUX_IDLE_DISCONNECT is also supported.
+
+States 0 through 3 correspond to signals A through D in the datasheet.
+
+Example:
+
+ /*
+ * Three independent mux controllers (of which one is used).
+ * Mux 0 is disconnected when idle, mux 1 idles in the previously
+ * selected state and mux 2 idles with signal B.
+ */
+ &i2c0 {
+ mux: mux-controller@50 {
+ compatible = "adi,adg792a";
+ reg = <0x50>;
+ #mux-control-cells = <1>;
+
+ idle-state = <MUX_IDLE_DISCONNECT MUX_IDLE_AS_IS 1>;
+ };
+ };
+
+ adc-mux {
+ compatible = "io-channel-mux";
+ io-channels = <&adc 0>;
+ io-channel-names = "parent";
+
+ mux-controls = <&mux 2>;
+
+ channels = "sync-1", "", "out";
+ };
+
+
+ /*
+ * Three parallel muxes with one mux controller, useful e.g. if
+ * the adc is differential, thus needing two signals to be muxed
+ * simultaneously for correct operation.
+ */
+ &i2c0 {
+ pmux: mux-controller@50 {
+ compatible = "adi,adg792a";
+ reg = <0x50>;
+ #mux-control-cells = <0>;
+
+ idle-state = <1>;
+ };
+ };
+
+ diff-adc-mux {
+ compatible = "io-channel-mux";
+ io-channels = <&adc 0>;
+ io-channel-names = "parent";
+
+ mux-controls = <&pmux>;
+
+ channels = "sync-1", "", "out";
+ };
--- /dev/null
+GPIO-based multiplexer controller bindings
+
+Define what GPIO pins are used to control a multiplexer. Or several
+multiplexers, if the same pins control more than one multiplexer.
+
+Required properties:
+- compatible : "gpio-mux"
+- mux-gpios : list of gpios used to control the multiplexer, least
+ significant bit first.
+- #mux-control-cells : <0>
+* Standard mux-controller bindings as decribed in mux-controller.txt
+
+Optional properties:
+- idle-state : if present, the state the mux will have when idle. The
+ special state MUX_IDLE_AS_IS is the default.
+
+The multiplexer state is defined as the number represented by the
+multiplexer GPIO pins, where the first pin is the least significant
+bit. An active pin is a binary 1, an inactive pin is a binary 0.
+
+Example:
+
+ mux: mux-controller {
+ compatible = "gpio-mux";
+ #mux-control-cells = <0>;
+
+ mux-gpios = <&pioA 0 GPIO_ACTIVE_HIGH>,
+ <&pioA 1 GPIO_ACTIVE_HIGH>;
+ };
+
+ adc-mux {
+ compatible = "io-channel-mux";
+ io-channels = <&adc 0>;
+ io-channel-names = "parent";
+
+ mux-controls = <&mux>;
+
+ channels = "sync-1", "in", "out", "sync-2";
+ };
+
+ i2c-mux {
+ compatible = "i2c-mux";
+ i2c-parent = <&i2c1>;
+
+ mux-controls = <&mux>;
+
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ i2c@0 {
+ reg = <0>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ ssd1307: oled@3c {
+ /* ... */
+ };
+ };
+
+ i2c@3 {
+ reg = <3>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ pca9555: pca9555@20 {
+ /* ... */
+ };
+ };
+ };
--- /dev/null
+MMIO register bitfield-based multiplexer controller bindings
+
+Define register bitfields to be used to control multiplexers. The parent
+device tree node must be a syscon node to provide register access.
+
+Required properties:
+- compatible : "mmio-mux"
+- #mux-control-cells : <1>
+- mux-reg-masks : an array of register offset and pre-shifted bitfield mask
+ pairs, each describing a single mux control.
+* Standard mux-controller bindings as decribed in mux-controller.txt
+
+Optional properties:
+- idle-states : if present, the state the muxes will have when idle. The
+ special state MUX_IDLE_AS_IS is the default.
+
+The multiplexer state of each multiplexer is defined as the value of the
+bitfield described by the corresponding register offset and bitfield mask pair
+in the mux-reg-masks array, accessed through the parent syscon.
+
+Example:
+
+ syscon {
+ compatible = "syscon";
+
+ mux: mux-controller {
+ compatible = "mmio-mux";
+ #mux-control-cells = <1>;
+
+ mux-reg-masks = <0x3 0x30>, /* 0: reg 0x3, bits 5:4 */
+ <0x3 0x40>, /* 1: reg 0x3, bit 6 */
+ idle-states = <MUX_IDLE_AS_IS>, <0>;
+ };
+ };
+
+ video-mux {
+ compatible = "video-mux";
+ mux-controls = <&mux 0>;
+
+ ports {
+ /* inputs 0..3 */
+ port@0 {
+ reg = <0>;
+ };
+ port@1 {
+ reg = <1>;
+ };
+ port@2 {
+ reg = <2>;
+ };
+ port@3 {
+ reg = <3>;
+ };
+
+ /* output */
+ port@4 {
+ reg = <4>;
+ };
+ };
+ };
--- /dev/null
+Common multiplexer controller bindings
+======================================
+
+A multiplexer (or mux) controller will have one, or several, consumer devices
+that uses the mux controller. Thus, a mux controller can possibly control
+several parallel multiplexers. Presumably there will be at least one
+multiplexer needed by each consumer, but a single mux controller can of course
+control several multiplexers for a single consumer.
+
+A mux controller provides a number of states to its consumers, and the state
+space is a simple zero-based enumeration. I.e. 0-1 for a 2-way multiplexer,
+0-7 for an 8-way multiplexer, etc.
+
+
+Consumers
+---------
+
+Mux controller consumers should specify a list of mux controllers that they
+want to use with a property containing a 'mux-ctrl-list':
+
+ mux-ctrl-list ::= <single-mux-ctrl> [mux-ctrl-list]
+ single-mux-ctrl ::= <mux-ctrl-phandle> [mux-ctrl-specifier]
+ mux-ctrl-phandle : phandle to mux controller node
+ mux-ctrl-specifier : array of #mux-control-cells specifying the
+ given mux controller (controller specific)
+
+Mux controller properties should be named "mux-controls". The exact meaning of
+each mux controller property must be documented in the device tree binding for
+each consumer. An optional property "mux-control-names" may contain a list of
+strings to label each of the mux controllers listed in the "mux-controls"
+property.
+
+Drivers for devices that use more than a single mux controller can use the
+"mux-control-names" property to map the name of the requested mux controller
+to an index into the list given by the "mux-controls" property.
+
+mux-ctrl-specifier typically encodes the chip-relative mux controller number.
+If the mux controller chip only provides a single mux controller, the
+mux-ctrl-specifier can typically be left out.
+
+Example:
+
+ /* One consumer of a 2-way mux controller (one GPIO-line) */
+ mux: mux-controller {
+ compatible = "gpio-mux";
+ #mux-control-cells = <0>;
+
+ mux-gpios = <&pioA 0 GPIO_ACTIVE_HIGH>;
+ };
+
+ adc-mux {
+ compatible = "io-channel-mux";
+ io-channels = <&adc 0>;
+ io-channel-names = "parent";
+
+ mux-controls = <&mux>;
+ mux-control-names = "adc";
+
+ channels = "sync", "in";
+ };
+
+Note that in the example above, specifying the "mux-control-names" is redundant
+because there is only one mux controller in the list. However, if the driver
+for the consumer node in fact asks for a named mux controller, that name is of
+course still required.
+
+ /*
+ * Two consumers (one for an ADC line and one for an i2c bus) of
+ * parallel 4-way multiplexers controlled by the same two GPIO-lines.
+ */
+ mux: mux-controller {
+ compatible = "gpio-mux";
+ #mux-control-cells = <0>;
+
+ mux-gpios = <&pioA 0 GPIO_ACTIVE_HIGH>,
+ <&pioA 1 GPIO_ACTIVE_HIGH>;
+ };
+
+ adc-mux {
+ compatible = "io-channel-mux";
+ io-channels = <&adc 0>;
+ io-channel-names = "parent";
+
+ mux-controls = <&mux>;
+
+ channels = "sync-1", "in", "out", "sync-2";
+ };
+
+ i2c-mux {
+ compatible = "i2c-mux";
+ i2c-parent = <&i2c1>;
+
+ mux-controls = <&mux>;
+
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ i2c@0 {
+ reg = <0>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ ssd1307: oled@3c {
+ /* ... */
+ };
+ };
+
+ i2c@3 {
+ reg = <3>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ pca9555: pca9555@20 {
+ /* ... */
+ };
+ };
+ };
+
+
+Mux controller nodes
+--------------------
+
+Mux controller nodes must specify the number of cells used for the
+specifier using the '#mux-control-cells' property.
+
+Optionally, mux controller nodes can also specify the state the mux should
+have when it is idle. The idle-state property is used for this. If the
+idle-state is not present, the mux controller is typically left as is when
+it is idle. For multiplexer chips that expose several mux controllers, the
+idle-state property is an array with one idle state for each mux controller.
+
+The special value (-1) may be used to indicate that the mux should be left
+as is when it is idle. This is the default, but can still be useful for
+mux controller chips with more than one mux controller, particularly when
+there is a need to "step past" a mux controller and set some other idle
+state for a mux controller with a higher index.
+
+Some mux controllers have the ability to disconnect the input/output of the
+multiplexer. Using this disconnected high-impedance state as the idle state
+is indicated with idle state (-2).
+
+These constants are available in
+
+ #include <dt-bindings/mux/mux.h>
+
+as MUX_IDLE_AS_IS (-1) and MUX_IDLE_DISCONNECT (-2).
+
+An example mux controller node look like this (the adg972a chip is a triple
+4-way multiplexer):
+
+ mux: mux-controller@50 {
+ compatible = "adi,adg792a";
+ reg = <0x50>;
+ #mux-control-cells = <1>;
+
+ idle-state = <MUX_IDLE_DISCONNECT MUX_IDLE_AS_IS 2>;
+ };
"brcm,bcm6328-switch"
"brcm,bcm6368-switch" and the mandatory "brcm,bcm63xx-switch"
-See Documentation/devicetree/bindings/dsa/dsa.txt for a list of additional
+See Documentation/devicetree/bindings/net/dsa/dsa.txt for a list of additional
required and optional properties.
Examples:
of the device. On many systems this is wired high so the device goes
out of reset at power-on, but if it is under program control, this
optional GPIO can wake up in response to it.
+- vdd33a-supply, vddvario-supply : 3.3V analog and IO logic power supplies
Examples:
- compatible: Should be one of the following.
- "rockchip,rk3066a-efuse" - for RK3066a SoCs.
- "rockchip,rk3188-efuse" - for RK3188 SoCs.
+ - "rockchip,rk322x-efuse" - for RK322x SoCs.
- "rockchip,rk3288-efuse" - for RK3288 SoCs.
- "rockchip,rk3399-efuse" - for RK3399 SoCs.
- reg: Should contain the registers location and exact eFuse size
compatible = "operating-points-v2";
opp-shared;
- opp@1000000000 {
+ opp-1000000000 {
opp-hz = /bits/ 64 <1000000000>;
opp-microvolt = <975000 970000 985000>;
opp-microamp = <70000>;
clock-latency-ns = <300000>;
opp-suspend;
};
- opp@1100000000 {
+ opp-1100000000 {
opp-hz = /bits/ 64 <1100000000>;
opp-microvolt = <1000000 980000 1010000>;
opp-microamp = <80000>;
clock-latency-ns = <310000>;
};
- opp@1200000000 {
+ opp-1200000000 {
opp-hz = /bits/ 64 <1200000000>;
opp-microvolt = <1025000>;
clock-latency-ns = <290000>;
* independently.
*/
- opp@1000000000 {
+ opp-1000000000 {
opp-hz = /bits/ 64 <1000000000>;
opp-microvolt = <975000 970000 985000>;
opp-microamp = <70000>;
clock-latency-ns = <300000>;
opp-suspend;
};
- opp@1100000000 {
+ opp-1100000000 {
opp-hz = /bits/ 64 <1100000000>;
opp-microvolt = <1000000 980000 1010000>;
opp-microamp = <80000>;
clock-latency-ns = <310000>;
};
- opp@1200000000 {
+ opp-1200000000 {
opp-hz = /bits/ 64 <1200000000>;
opp-microvolt = <1025000>;
opp-microamp = <90000;
compatible = "operating-points-v2";
opp-shared;
- opp@1000000000 {
+ opp-1000000000 {
opp-hz = /bits/ 64 <1000000000>;
opp-microvolt = <975000 970000 985000>;
opp-microamp = <70000>;
clock-latency-ns = <300000>;
opp-suspend;
};
- opp@1100000000 {
+ opp-1100000000 {
opp-hz = /bits/ 64 <1100000000>;
opp-microvolt = <1000000 980000 1010000>;
opp-microamp = <80000>;
clock-latency-ns = <310000>;
};
- opp@1200000000 {
+ opp-1200000000 {
opp-hz = /bits/ 64 <1200000000>;
opp-microvolt = <1025000>;
opp-microamp = <90000>;
compatible = "operating-points-v2";
opp-shared;
- opp@1300000000 {
+ opp-1300000000 {
opp-hz = /bits/ 64 <1300000000>;
opp-microvolt = <1050000 1045000 1055000>;
opp-microamp = <95000>;
clock-latency-ns = <400000>;
opp-suspend;
};
- opp@1400000000 {
+ opp-1400000000 {
opp-hz = /bits/ 64 <1400000000>;
opp-microvolt = <1075000>;
opp-microamp = <100000>;
clock-latency-ns = <400000>;
};
- opp@1500000000 {
+ opp-1500000000 {
opp-hz = /bits/ 64 <1500000000>;
opp-microvolt = <1100000 1010000 1110000>;
opp-microamp = <95000>;
compatible = "operating-points-v2";
opp-shared;
- opp@1000000000 {
+ opp-1000000000 {
opp-hz = /bits/ 64 <1000000000>;
opp-microvolt = <970000>, /* Supply 0 */
<960000>, /* Supply 1 */
/* OR */
- opp@1000000000 {
+ opp-1000000000 {
opp-hz = /bits/ 64 <1000000000>;
opp-microvolt = <975000 970000 985000>, /* Supply 0 */
<965000 960000 975000>, /* Supply 1 */
/* OR */
- opp@1000000000 {
+ opp-1000000000 {
opp-hz = /bits/ 64 <1000000000>;
opp-microvolt = <975000 970000 985000>, /* Supply 0 */
<965000 960000 975000>, /* Supply 1 */
status = "okay";
opp-shared;
- opp@600000000 {
+ opp-600000000 {
/*
* Supports all substrate and process versions for 0xF
* cuts, i.e. only first four cuts.
...
};
- opp@800000000 {
+ opp-800000000 {
/*
* Supports:
* - cuts: only one, 6th cut (represented by 6th bit).
compatible = "operating-points-v2";
opp-shared;
- opp@1000000000 {
+ opp-1000000000 {
opp-hz = /bits/ 64 <1000000000>;
opp-microvolt-slow = <915000 900000 925000>;
opp-microvolt-fast = <975000 970000 985000>;
opp-microamp-fast = <71000>;
};
- opp@1200000000 {
+ opp-1200000000 {
opp-hz = /bits/ 64 <1200000000>;
opp-microvolt-slow = <915000 900000 925000>, /* Supply vcc0 */
<925000 910000 935000>; /* Supply vcc1 */
Required properties:
- compatible: one of: "brcm,ns-ax-usb3-phy", "brcm,ns-bx-usb3-phy".
-- reg: register mappings for DMP (Device Management Plugin) and ChipCommon B
- MMI.
-- reg-names: "dmp" and "ccb-mii"
+- reg: address of MDIO bus device
+- usb3-dmp-syscon: phandle to syscon with DMP (Device Management Plugin)
+ registers
+- #phy-cells: must be 0
Initialization of USB 3.0 PHY depends on Northstar version. There are currently
three known series: Ax, Bx and Cx.
Known C0: BCM47094 rev 0
Example:
- usb3-phy {
- compatible = "brcm,ns-ax-usb3-phy";
- reg = <0x18105000 0x1000>, <0x18003000 0x1000>;
- reg-names = "dmp", "ccb-mii";
- #phy-cells = <0>;
+ mdio: mdio@0 {
+ reg = <0x0>;
+ #size-cells = <1>;
+ #address-cells = <0>;
+
+ usb3-phy@10 {
+ compatible = "brcm,ns-ax-usb3-phy";
+ reg = <0x10>;
+ usb3-dmp-syscon = <&usb3_dmp>;
+ #phy-cells = <0>;
+ };
+ };
+
+ usb3_dmp: syscon@18105000 {
+ reg = <0x18105000 0x1000>;
};
--- /dev/null
+BROADCOM NORTHSTAR2 USB2 (DUAL ROLE DEVICE) PHY
+
+Required properties:
+ - compatible: brcm,ns2-drd-phy
+ - reg: offset and length of the NS2 PHY related registers.
+ - reg-names
+ The below registers must be provided.
+ icfg - for DRD ICFG configurations
+ rst-ctrl - for DRD IDM reset
+ crmu-ctrl - for CRMU core vdd, PHY and PHY PLL reset
+ usb2-strap - for port over current polarity reversal
+ - #phy-cells: Must be 0. No args required.
+ - vbus-gpios: vbus gpio binding
+ - id-gpios: id gpio binding
+
+Refer to phy/phy-bindings.txt for the generic PHY binding properties
+
+Example:
+ usbdrd_phy: phy@66000960 {
+ #phy-cells = <0>;
+ compatible = "brcm,ns2-drd-phy";
+ reg = <0x66000960 0x24>,
+ <0x67012800 0x4>,
+ <0x6501d148 0x4>,
+ <0x664d0700 0x4>;
+ reg-names = "icfg", "rst-ctrl",
+ "crmu-ctrl", "usb2-strap";
+ id-gpios = <&gpio_g 30 0>;
+ vbus-gpios = <&gpio_g 31 0>;
+ };
"brcm,iproc-ns2-sata-phy"
"brcm,iproc-nsp-sata-phy"
"brcm,phy-sata3"
+ "brcm,iproc-sr-sata-phy"
- address-cells: should be 1
- size-cells: should be 0
- reg: register ranges for the PHY PCB interface
- reg-names: should be "phy" and "phy-ctrl"
The "phy-ctrl" registers are only required for
- "brcm,iproc-ns2-sata-phy".
+ "brcm,iproc-ns2-sata-phy" and "brcm,iproc-sr-sata-phy".
Sub-nodes:
Each port's PHY should be represented as a sub-node.
Sub-nodes optional properties:
- brcm,enable-ssc: use spread spectrum clocking (SSC) on this port
- This property is not applicable for "brcm,iproc-ns2-sata-phy" and
- "brcm,iproc-nsp-sata-phy".
+ This property is not applicable for "brcm,iproc-ns2-sata-phy",
+ "brcm,iproc-nsp-sata-phy" and "brcm,iproc-sr-sata-phy".
Example:
sata-phy@f0458100 {
--- /dev/null
+* Amlogic Meson GXL and GXM USB2 PHY binding
+
+Required properties:
+- compatible: Should be "amlogic,meson-gxl-usb2-phy"
+- reg: The base address and length of the registers
+- #phys-cells: must be 0 (see phy-bindings.txt in this directory)
+
+Optional properties:
+- phy-supply: see phy-bindings.txt in this directory
+
+
+Example:
+ usb2_phy0: phy@78000 {
+ compatible = "amlogic,meson-gxl-usb2-phy";
+ #phy-cells = <0>;
+ reg = <0x0 0x78000 0x0 0x20>;
+ };
-* Amlogic Meson8b and GXBB USB2 PHY
+* Amlogic Meson8, Meson8b and GXBB USB2 PHY
Required properties:
- compatible: Depending on the platform this should be one of:
+ "amlogic,meson8-usb2-phy"
"amlogic,meson8b-usb2-phy"
"amlogic,meson-gxbb-usb2-phy"
- reg: The base address and length of the registers
--- /dev/null
+Motorola CPCAP PMIC USB PHY binding
+
+Required properties:
+compatible: Shall be either "motorola,cpcap-usb-phy" or
+ "motorola,mapphone-cpcap-usb-phy"
+#phy-cells: Shall be 0
+interrupts: CPCAP PMIC interrupts used by the USB PHY
+interrupt-names: Interrupt names
+io-channels: IIO ADC channels used by the USB PHY
+io-channel-names: IIO ADC channel names
+vusb-supply: Regulator for the PHY
+
+Optional properties:
+pinctrl: Optional alternate pin modes for the PHY
+pinctrl-names: Names for optional pin modes
+mode-gpios: Optional GPIOs for configuring alternate modes
+
+Example:
+cpcap_usb2_phy: phy {
+ compatible = "motorola,mapphone-cpcap-usb-phy";
+ pinctrl-0 = <&usb_gpio_mux_sel1 &usb_gpio_mux_sel2>;
+ pinctrl-1 = <&usb_ulpi_pins>;
+ pinctrl-2 = <&usb_utmi_pins>;
+ pinctrl-3 = <&uart3_pins>;
+ pinctrl-names = "default", "ulpi", "utmi", "uart";
+ #phy-cells = <0>;
+ interrupts-extended = <
+ &cpcap 15 0 &cpcap 14 0 &cpcap 28 0 &cpcap 19 0
+ &cpcap 18 0 &cpcap 17 0 &cpcap 16 0 &cpcap 49 0
+ &cpcap 48 1
+ >;
+ interrupt-names =
+ "id_ground", "id_float", "se0conn", "vbusvld",
+ "sessvld", "sessend", "se1", "dm", "dp";
+ mode-gpios = <&gpio2 28 GPIO_ACTIVE_HIGH
+ &gpio1 0 GPIO_ACTIVE_HIGH>;
+ io-channels = <&cpcap_adc 2>, <&cpcap_adc 7>;
+ io-channel-names = "vbus", "id";
+ vusb-supply = <&vusb>;
+};
Required properties (phy (parent) node):
- compatible : should be one of the listed compatibles:
+ * "rockchip,rk3228-usb2phy"
* "rockchip,rk3328-usb2phy"
* "rockchip,rk3366-usb2phy"
* "rockchip,rk3399-usb2phy"
--- /dev/null
+* Renesas R-Car generation 3 USB 3.0 PHY
+
+This file provides information on what the device node for the R-Car generation
+3 USB 3.0 PHY contains.
+If you want to enable spread spectrum clock (ssc), you should use USB_EXTAL
+instead of USB3_CLK. However, if you don't want to these features, you don't
+need this driver.
+
+Required properties:
+- compatible: "renesas,r8a7795-usb3-phy" if the device is a part of an R8A7795
+ SoC.
+ "renesas,r8a7796-usb3-phy" if the device is a part of an R8A7796
+ SoC.
+ "renesas,rcar-gen3-usb3-phy" for a generic R-Car Gen3 compatible
+ device.
+
+ When compatible with the generic version, nodes must list the
+ SoC-specific version corresponding to the platform first
+ followed by the generic version.
+
+- reg: offset and length of the USB 3.0 PHY register block.
+- clocks: A list of phandles and clock-specifier pairs.
+- clock-names: Name of the clocks.
+ - The funcional clock must be "usb3-if".
+ - The usb3's external clock must be "usb3s_clk".
+ - The usb2's external clock must be "usb_extal". If you want to use the ssc,
+ the clock-frequency must not be 0.
+- #phy-cells: see phy-bindings.txt in the same directory, must be <0>.
+
+Optional properties:
+- renesas,ssc-range: Enable/disable spread spectrum clock (ssc) by using
+ the following values as u32:
+ - 0 (or the property doesn't exist): disable the ssc
+ - 4980: enable the ssc as -4980 ppm
+ - 4492: enable the ssc as -4492 ppm
+ - 4003: enable the ssc as -4003 ppm
+
+Example (R-Car H3):
+
+ usb-phy@e65ee000 {
+ compatible = "renesas,r8a7795-usb3-phy",
+ "renesas,rcar-gen3-usb3-phy";
+ reg = <0 0xe65ee000 0 0x90>;
+ clocks = <&cpg CPG_MOD 328>, <&usb3s0_clk>, <&usb_extal>;
+ clock-names = "usb3-if", "usb3s_clk", "usb_extal";
+ };
Required properties:
- compatible: should be one of:
- "rockchip,rk3188-io-voltage-domain" for rk3188
+ - "rockchip,rk3228-io-voltage-domain" for rk3228
- "rockchip,rk3288-io-voltage-domain" for rk3288
- "rockchip,rk3328-io-voltage-domain" for rk3328
- "rockchip,rk3368-io-voltage-domain" for rk3368
- vccio1-supply: The supply connected to VCCIO1.
Sometimes also labeled VCCIO1 and VCCIO2.
+Possible supplies for rk3228:
+- vccio1-supply: The supply connected to VCCIO1.
+- vccio2-supply: The supply connected to VCCIO2.
+- vccio3-supply: The supply connected to VCCIO3.
+- vccio4-supply: The supply connected to VCCIO4.
+
Possible supplies for rk3288:
- audio-supply: The supply connected to APIO4_VDD.
- bb-supply: The supply connected to APIO5_VDD.
- regulator-settling-time-us: Settling time, in microseconds, for voltage
change if regulator have the constant time for any level voltage change.
This is useful when regulator have exponential voltage change.
+- regulator-settling-time-up-us: Settling time, in microseconds, for voltage
+ increase if the regulator needs a constant time to settle after voltage
+ increases of any level. This is useful for regulators with exponential
+ voltage changes.
+- regulator-settling-time-down-us: Settling time, in microseconds, for voltage
+ decrease if the regulator needs a constant time to settle after voltage
+ decreases of any level. This is useful for regulators with exponential
+ voltage changes.
- regulator-soft-start: Enable soft start so that voltage ramps slowly
- regulator-state-mem sub-root node for Suspend-to-RAM mode
: suspend to memory, the device goes to sleep, but all data stored in memory,
- "fsl,16550-FIFO64"
- "fsl,ns16550"
- "ti,da830-uart"
+ - "aspeed,ast2400-vuart"
+ - "aspeed,ast2500-vuart"
- "serial" if the port type is unknown.
- reg : offset and length of the register set for the device.
- interrupts : should contain uart interrupt.
property.
- tx-threshold: Specify the TX FIFO low water indication for parts with
programmable TX FIFO thresholds.
+- resets : phandle + reset specifier pairs
Note:
* fsl,ns16550:
--- /dev/null
+Actions Semi Owl UART
+
+Required properties:
+- compatible : "actions,s500-uart", "actions,owl-uart" for S500
+ "actions,s900-uart", "actions,owl-uart" for S900
+- reg : Offset and length of the register set for the device.
+- interrupts : Should contain UART interrupt.
+
+
+Example:
+
+ uart3: serial@b0126000 {
+ compatible = "actions,s500-uart", "actions,owl-uart";
+ reg = <0xb0126000 0x1000>;
+ interrupts = <GIC_SPI 32 IRQ_TYPE_LEVEL_HIGH>;
+ };
--- /dev/null
+Amlogic Meson SoC UART Serial Interface
+=======================================
+
+The Amlogic Meson SoC UART Serial Interface is present on a large range
+of SoCs, and can be present either in the "Always-On" power domain or the
+"Everything-Else" power domain.
+
+The particularity of the "Always-On" Serial Interface is that the hardware
+is active since power-on and does not need any clock gating and is usable
+as very early serial console.
+
+Required properties:
+- compatible : compatible: value should be different for each SoC family as :
+ - Meson6 : "amlogic,meson6-uart"
+ - Meson8 : "amlogic,meson8-uart"
+ - Meson8b : "amlogic,meson8b-uart"
+ - GX (GXBB, GXL, GXM) : "amlogic,meson-gx-uart"
+ eventually followed by : "amlogic,meson-ao-uart" if this UART interface
+ is in the "Always-On" power domain.
+- reg : offset and length of the register set for the device.
+- interrupts : identifier to the device interrupt
+- clocks : a list of phandle + clock-specifier pairs, one for each
+ entry in clock names.
+- clocks-names :
+ * "xtal" for external xtal clock identifier
+ * "pclk" for the bus core clock, either the clk81 clock or the gate clock
+ * "baud" for the source of the baudrate generator, can be either the xtal
+ or the pclk.
+
+e.g.
+uart_A: serial@84c0 {
+ compatible = "amlogic,meson-gx-uart";
+ reg = <0x0 0x84c0 0x0 0x14>;
+ interrupts = <GIC_SPI 26 IRQ_TYPE_EDGE_RISING>;
+ /* Use xtal as baud rate clock source */
+ clocks = <&xtal>, <&clkc CLKID_UART0>, <&xtal>;
+ clock-names = "xtal", "pclk", "baud";
+};
- fsl,irda-mode : Indicate the uart supports irda mode
- fsl,dte-mode : Indicate the uart works in DTE mode. The uart works
in DCE mode by default.
+- fsl,dma-size : Indicate the size of the DMA buffer and its periods
Please check Documentation/devicetree/bindings/serial/serial.txt
for the complete list of generic properties.
interrupts = <31>;
uart-has-rtscts;
fsl,dte-mode;
+ fsl,dma-size = <1024 4>;
};
on Vybrid vf610 SoC with 8-bit register organization
- "fsl,ls1021a-lpuart" for lpuart compatible with the one integrated
on LS1021A SoC with 32-bit big-endian register organization
+ - "fsl,imx7ulp-lpuart" for lpuart compatible with the one integrated
+ on i.MX7ULP SoC with 32-bit little-endian register organization
- reg : Address and length of the register set for the device
- interrupts : Should contain uart interrupt
- clocks : phandle + clock specifier pairs, one for each entry in clock-names
specifiers, one for transmission, and one for
reception.
- dma-names : Must contain a list of two DMA names, "tx" and "rx".
+- spi-slave : Empty property indicating the SPI controller is used
+ in slave mode.
- renesas,dtdl : delay sync signal (setup) in transmit mode.
Must contain one of the following values:
0 (no bit delay)
SPI (Serial Peripheral Interface) busses
-SPI busses can be described with a node for the SPI master device
-and a set of child nodes for each SPI slave on the bus. For this
-discussion, it is assumed that the system's SPI controller is in
-SPI master mode. This binding does not describe SPI controllers
-in slave mode.
+SPI busses can be described with a node for the SPI controller device
+and a set of child nodes for each SPI slave on the bus. The system's SPI
+controller may be described for use in SPI master mode or in SPI slave mode,
+but not for both at the same time.
-The SPI master node requires the following properties:
+The SPI controller node requires the following properties:
+- compatible - Name of SPI bus controller following generic names
+ recommended practice.
+
+In master mode, the SPI controller node requires the following additional
+properties:
- #address-cells - number of cells required to define a chip select
address on the SPI bus.
- #size-cells - should be zero.
-- compatible - name of SPI bus controller following generic names
- recommended practice.
+
+In slave mode, the SPI controller node requires one additional property:
+- spi-slave - Empty property.
+
No other properties are required in the SPI bus node. It is assumed
that a driver for an SPI bus device will understand that it is an SPI bus.
However, the binding does not attempt to define the specific method for
chip selects. Individual drivers can define additional properties to
support describing the chip select layout.
-Optional properties:
+Optional properties (master mode only):
- cs-gpios - gpios chip select.
- num-cs - total number of chipselects.
cs2 : &gpio1 1 0
cs3 : &gpio1 2 0
-SPI slave nodes must be children of the SPI master node and can
-contain the following properties.
-- reg - (required) chip select address of device.
-- compatible - (required) name of SPI device following generic names
- recommended practice.
-- spi-max-frequency - (required) Maximum SPI clocking speed of device in Hz.
-- spi-cpol - (optional) Empty property indicating device requires
- inverse clock polarity (CPOL) mode.
-- spi-cpha - (optional) Empty property indicating device requires
- shifted clock phase (CPHA) mode.
-- spi-cs-high - (optional) Empty property indicating device requires
- chip select active high.
-- spi-3wire - (optional) Empty property indicating device requires
- 3-wire mode.
-- spi-lsb-first - (optional) Empty property indicating device requires
- LSB first mode.
-- spi-tx-bus-width - (optional) The bus width (number of data wires) that is
- used for MOSI. Defaults to 1 if not present.
-- spi-rx-bus-width - (optional) The bus width (number of data wires) that is
- used for MISO. Defaults to 1 if not present.
-- spi-rx-delay-us - (optional) Microsecond delay after a read transfer.
-- spi-tx-delay-us - (optional) Microsecond delay after a write transfer.
+
+SPI slave nodes must be children of the SPI controller node.
+
+In master mode, one or more slave nodes (up to the number of chip selects) can
+be present. Required properties are:
+- compatible - Name of SPI device following generic names recommended
+ practice.
+- reg - Chip select address of device.
+- spi-max-frequency - Maximum SPI clocking speed of device in Hz.
+
+In slave mode, the (single) slave node is optional.
+If present, it must be called "slave". Required properties are:
+- compatible - Name of SPI device following generic names recommended
+ practice.
+
+All slave nodes can contain the following optional properties:
+- spi-cpol - Empty property indicating device requires inverse clock
+ polarity (CPOL) mode.
+- spi-cpha - Empty property indicating device requires shifted clock
+ phase (CPHA) mode.
+- spi-cs-high - Empty property indicating device requires chip select
+ active high.
+- spi-3wire - Empty property indicating device requires 3-wire mode.
+- spi-lsb-first - Empty property indicating device requires LSB first mode.
+- spi-tx-bus-width - The bus width (number of data wires) that is used for MOSI.
+ Defaults to 1 if not present.
+- spi-rx-bus-width - The bus width (number of data wires) that is used for MISO.
+ Defaults to 1 if not present.
+- spi-rx-delay-us - Microsecond delay after a read transfer.
+- spi-tx-delay-us - Microsecond delay after a write transfer.
Some SPI controllers and devices support Dual and Quad SPI transfer mode.
It allows data in the SPI system to be transferred using 2 wires (DUAL) or 4
#address-cells = <1>;
#size-cells = <0>;
};
+
+* SPICC (SPI Communication Controller)
+
+The Meson SPICC is generic SPI controller for general purpose Full-Duplex
+communications with dedicated 16 words RX/TX PIO FIFOs.
+
+Required properties:
+ - compatible: should be "amlogic,meson-gx-spicc" on Amlogic GX SoCs.
+ - reg: physical base address and length of the controller registers
+ - interrupts: The interrupt specifier
+ - clock-names: Must contain "core"
+ - clocks: phandle of the input clock for the baud rate generator
+ - #address-cells: should be 1
+ - #size-cells: should be 0
+
+Optional properties:
+ - resets: phandle of the internal reset line
+
+See ../spi/spi-bus.txt for more details on SPI bus master and slave devices
+required and optional properties.
+
+Example :
+ spi@c1108d80 {
+ compatible = "amlogic,meson-gx-spicc";
+ reg = <0xc1108d80 0x80>;
+ interrupts = <GIC_SPI 112 IRQ_TYPE_LEVEL_HIGH>;
+ clock-names = "core";
+ clocks = <&clk81>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ };
Required properties:
- compatible: should be one of the following.
- mediatek,mt2701-spi: for mt2701 platforms
+ - mediatek,mt2712-spi: for mt2712 platforms
- mediatek,mt6589-spi: for mt6589 platforms
+ - mediatek,mt7622-spi: for mt7622 platforms
- mediatek,mt8135-spi: for mt8135 platforms
- mediatek,mt8173-spi: for mt8173 platforms
--- /dev/null
+STMicroelectronics STM32 SPI Controller
+
+The STM32 SPI controller is used to communicate with external devices using
+the Serial Peripheral Interface. It supports full-duplex, half-duplex and
+simplex synchronous serial communication with external devices. It supports
+from 4 to 32-bit data size. Although it can be configured as master or slave,
+only master is supported by the driver.
+
+Required properties:
+- compatible: Must be "st,stm32h7-spi".
+- reg: Offset and length of the device's register set.
+- interrupts: Must contain the interrupt id.
+- clocks: Must contain an entry for spiclk (which feeds the internal clock
+ generator).
+- #address-cells: Number of cells required to define a chip select address.
+- #size-cells: Should be zero.
+
+Optional properties:
+- resets: Must contain the phandle to the reset controller.
+- A pinctrl state named "default" may be defined to set pins in mode of
+ operation for SPI transfer.
+- dmas: DMA specifiers for tx and rx dma. DMA fifo mode must be used. See the
+ STM32 DMA bindings, Documentation/devicetree/bindings/dma/stm32-dma.txt.
+- dma-names: DMA request names should include "tx" and "rx" if present.
+- cs-gpios: list of GPIO chip selects. See the SPI bus bindings,
+ Documentation/devicetree/bindings/spi/spi-bus.txt
+
+
+Child nodes represent devices on the SPI bus
+ See ../spi/spi-bus.txt
+
+Optional properties:
+- st,spi-midi-ns: (Master Inter-Data Idleness) minimum time delay in
+ nanoseconds inserted between two consecutive data frames.
+
+
+Example:
+ spi2: spi@40003800 {
+ #address-cells = <1>;
+ #size-cells = <0>;
+ compatible = "st,stm32h7-spi";
+ reg = <0x40003800 0x400>;
+ interrupts = <36>;
+ clocks = <&rcc SPI2_CK>;
+ resets = <&rcc 1166>;
+ dmas = <&dmamux1 0 39 0x400 0x01>,
+ <&dmamux1 1 40 0x400 0x01>;
+ dma-names = "rx", "tx";
+ pinctrl-0 = <&spi2_pins_b>;
+ pinctrl-names = "default";
+ cs-gpios = <&gpioa 11 0>;
+
+ aardvark@0 {
+ compatible = "totalphase,aardvark";
+ reg = <0>;
+ spi-max-frequency = <4000000>;
+ st,spi-midi-ns = <4000>;
+ };
+ };
- compatible : Must be one of
"faraday,fttmr010"
- "cortina,gemini-timer"
+ "cortina,gemini-timer", "faraday,fttmr010"
+ "moxa,moxart-timer", "faraday,fttmr010"
+ "aspeed,ast2400-timer"
+ "aspeed,ast2500-timer"
+
- reg : Should contain registers location and length
- interrupts : Should contain the three timer interrupts usually with
flags for falling edge
+++ /dev/null
-MOXA ART timer
-
-Required properties:
-
-- compatible : Must be one of:
- - "moxa,moxart-timer"
- - "aspeed,ast2400-timer"
-- reg : Should contain registers location and length
-- interrupts : Should contain the timer interrupt number
-- clocks : Should contain phandle for the clock that drives the counter
-
-Example:
-
- timer: timer@98400000 {
- compatible = "moxa,moxart-timer";
- reg = <0x98400000 0x42>;
- interrupts = <19 1>;
- clocks = <&coreclk>;
- };
infineon,slb9635tt Infineon SLB9635 (Soft-) I2C TPM (old protocol, max 100khz)
infineon,slb9645tt Infineon SLB9645 I2C TPM (new protocol, max 400khz)
isil,isl29028 Intersil ISL29028 Ambient Light and Proximity Sensor
+isil,isl29030 Intersil ISL29030 Ambient Light and Proximity Sensor
maxim,ds1050 5 Bit Programmable, Pulse-Width Modulator
maxim,max1237 Low-Power, 4-/12-Channel, 2-Wire Serial, 12-Bit ADCs
maxim,max6625 9-Bit/12-Bit Temperature Sensors with I²C-Compatible Serial Interface
a free-running PHY clock.
- snps,dis-del-phy-power-chg-quirk: when set core will change PHY power
from P0 to P1/P2/P3 without delay.
+ - snps,dis-tx-ipgap-linecheck-quirk: when set, disable u2mac linestate check
+ during HS transmit.
- snps,is-utmi-l1-suspend: true when DWC3 asserts output signal
utmi_l1_suspend_n, false when asserts utmi_sleep_n
- snps,hird-threshold: HIRD threshold
--- /dev/null
+Broadcom IPROC USB Device controller.
+
+The device node is used for UDCs integrated into Broadcom's
+iProc family (Northstar2, Cygnus) of SoCs'. The UDC is based
+on Synopsys Designware Cores AHB Subsystem Device Controller
+IP.
+
+Required properties:
+ - compatible: Add the compatibility strings for supported platforms.
+ For Broadcom NS2 platform, add "brcm,ns2-udc","brcm,iproc-udc".
+ For Broadcom Cygnus platform, add "brcm,cygnus-udc", "brcm,iproc-udc".
+ - reg: Offset and length of UDC register set
+ - interrupts: description of interrupt line
+ - phys: phandle to phy node.
+
+Example:
+ udc_dwc: usb@664e0000 {
+ compatible = "brcm,ns2-udc", "brcm,iproc-udc";
+ reg = <0x664e0000 0x2000>;
+ interrupts = <GIC_SPI 424 IRQ_TYPE_LEVEL_HIGH>;
+ phys = <&usbdrd_phy>;
- big-endian-desc : boolean, set this for hcds with big-endian descriptors
- big-endian : boolean, for hcds with big-endian-regs + big-endian-desc
- no-big-frame-no : boolean, set if frame_no lives in bits [15:0] of HCCA
+- remote-wakeup-connected: remote wakeup is wired on the platform
- num-ports : u32, to override the detected port count
- clocks : a list of phandle + clock specifier pairs
- phys : phandle + phy specifier pair
+++ /dev/null
-DocBook XML [DEPRECATED]
-========================
-
-.. attention::
-
- This section describes the deprecated DocBook XML toolchain. Please do not
- create new DocBook XML template files. Please consider converting existing
- DocBook XML templates files to Sphinx/reStructuredText.
-
-Converting DocBook to Sphinx
-----------------------------
-
-Over time, we expect all of the documents under ``Documentation/DocBook`` to be
-converted to Sphinx and reStructuredText. For most DocBook XML documents, a good
-enough solution is to use the simple ``Documentation/sphinx/tmplcvt`` script,
-which uses ``pandoc`` under the hood. For example::
-
- $ cd Documentation/sphinx
- $ ./tmplcvt ../DocBook/in.tmpl ../out.rst
-
-Then edit the resulting rst files to fix any remaining issues, and add the
-document in the ``toctree`` in ``Documentation/index.rst``.
-
-Components of the kernel-doc system
------------------------------------
-
-Many places in the source tree have extractable documentation in the form of
-block comments above functions. The components of this system are:
-
-- ``scripts/kernel-doc``
-
- This is a perl script that hunts for the block comments and can mark them up
- directly into reStructuredText, DocBook, man, text, and HTML. (No, not
- texinfo.)
-
-- ``Documentation/DocBook/*.tmpl``
-
- These are XML template files, which are normal XML files with special
- place-holders for where the extracted documentation should go.
-
-- ``scripts/docproc.c``
-
- This is a program for converting XML template files into XML files. When a
- file is referenced it is searched for symbols exported (EXPORT_SYMBOL), to be
- able to distinguish between internal and external functions.
-
- It invokes kernel-doc, giving it the list of functions that are to be
- documented.
-
- Additionally it is used to scan the XML template files to locate all the files
- referenced herein. This is used to generate dependency information as used by
- make.
-
-- ``Makefile``
-
- The targets 'xmldocs', 'psdocs', 'pdfdocs', and 'htmldocs' are used to build
- DocBook XML files, PostScript files, PDF files, and html files in
- Documentation/DocBook. The older target 'sgmldocs' is equivalent to 'xmldocs'.
-
-- ``Documentation/DocBook/Makefile``
-
- This is where C files are associated with SGML templates.
-
-How to use kernel-doc comments in DocBook XML template files
-------------------------------------------------------------
-
-DocBook XML template files (\*.tmpl) are like normal XML files, except that they
-can contain escape sequences where extracted documentation should be inserted.
-
-``!E<filename>`` is replaced by the documentation, in ``<filename>``, for
-functions that are exported using ``EXPORT_SYMBOL``: the function list is
-collected from files listed in ``Documentation/DocBook/Makefile``.
-
-``!I<filename>`` is replaced by the documentation for functions that are **not**
-exported using ``EXPORT_SYMBOL``.
-
-``!D<filename>`` is used to name additional files to search for functions
-exported using ``EXPORT_SYMBOL``.
-
-``!F<filename> <function [functions...]>`` is replaced by the documentation, in
-``<filename>``, for the functions listed.
-
-``!P<filename> <section title>`` is replaced by the contents of the ``DOC:``
-section titled ``<section title>`` from ``<filename>``. Spaces are allowed in
-``<section title>``; do not quote the ``<section title>``.
-
-``!C<filename>`` is replaced by nothing, but makes the tools check that all DOC:
-sections and documented functions, symbols, etc. are used. This makes sense to
-use when you use ``!F`` or ``!P`` only and want to verify that all documentation
-is included.
sphinx.rst
kernel-doc.rst
parse-headers.rst
- docbook.rst
.. only:: subproject and html
``%CONST``
Name of a constant. (No cross-referencing, just formatting.)
+````literal````
+ A literal block that should be handled as-is. The output will use a
+ ``monospaced font``.
+
+ Useful if you need to use special characters that would otherwise have some
+ meaning either by kernel-doc script of by reStructuredText.
+
+ This is particularly useful if you need to use things like ``%ph`` inside
+ a function description.
+
``$ENVVAR``
Name of an environment variable. (No cross-referencing, just formatting.)
kernel-doc comments have some special structure and formatting, but beyond that
they are also treated as reStructuredText.
-There is also the deprecated DocBook toolchain to generate documentation from
-DocBook XML template files under ``Documentation/DocBook``. The DocBook files
-are to be converted to reStructuredText, and the toolchain is slated to be
-removed.
-
Finally, there are thousands of plain text documentation files scattered around
``Documentation``. Some of these will likely be converted to reStructuredText
over time, but the bulk of them will remain in plain text.
devlist.h*
devicetable-offsets.h
dnotify_test
-docproc
dslm
dtc
elf2ecoff
.. kernel-doc:: drivers/base/firmware_class.c
:functions: request_firmware_nowait
+Considerations for suspend and resume
+=====================================
+
+During suspend and resume only the built-in firmware and the firmware cache
+elements of the firmware API can be used. This is managed by fw_pm_notify().
+
+fw_pm_notify
+------------
+.. kernel-doc:: drivers/base/firmware_class.c
+ :functions: fw_pm_notify
+
request firmware API expected driver use
========================================
between masters, as well as to handshake and to synchronize clocks from
slower clients.
-The Linux I2C programming interfaces support only the master side of bus
-interactions, not the slave side. The programming interface is
+The Linux I2C programming interfaces support the master side of bus
+interactions and the slave side. The programming interface is
structured around two kinds of driver, and two kinds of device. An I2C
"Adapter Driver" abstracts the controller hardware; it binds to a
physical device (perhaps a PCI device or platform_device) and exposes a
I2C bus segment it manages. On each I2C bus segment will be I2C devices
represented by a :c:type:`struct i2c_client <i2c_client>`.
Those devices will be bound to a :c:type:`struct i2c_driver
-<i2c_driver>`, which should follow the standard Linux driver
-model. (At this writing, a legacy model is more widely used.) There are
-functions to perform various I2C protocol operations; at this writing
+<i2c_driver>`, which should follow the standard Linux driver model. There
+are functions to perform various I2C protocol operations; at this writing
all such functions are usable only from task context.
The System Management Bus (SMBus) is a sibling protocol. Most SMBus
i2c
hsi
edac
+ scsi
+ libata
+ mtdnand
miscellaneous
+ w1
+ rapidio
+ s390-drivers
vme
80211/index
uio-howto
--- /dev/null
+========================
+libATA Developer's Guide
+========================
+
+:Author: Jeff Garzik
+
+Introduction
+============
+
+libATA is a library used inside the Linux kernel to support ATA host
+controllers and devices. libATA provides an ATA driver API, class
+transports for ATA and ATAPI devices, and SCSI<->ATA translation for ATA
+devices according to the T10 SAT specification.
+
+This Guide documents the libATA driver API, library functions, library
+internals, and a couple sample ATA low-level drivers.
+
+libata Driver API
+=================
+
+:c:type:`struct ata_port_operations <ata_port_operations>`
+is defined for every low-level libata
+hardware driver, and it controls how the low-level driver interfaces
+with the ATA and SCSI layers.
+
+FIS-based drivers will hook into the system with ``->qc_prep()`` and
+``->qc_issue()`` high-level hooks. Hardware which behaves in a manner
+similar to PCI IDE hardware may utilize several generic helpers,
+defining at a bare minimum the bus I/O addresses of the ATA shadow
+register blocks.
+
+:c:type:`struct ata_port_operations <ata_port_operations>`
+----------------------------------------------------------
+
+Disable ATA port
+~~~~~~~~~~~~~~~~
+
+::
+
+ void (*port_disable) (struct ata_port *);
+
+
+Called from :c:func:`ata_bus_probe` error path, as well as when unregistering
+from the SCSI module (rmmod, hot unplug). This function should do
+whatever needs to be done to take the port out of use. In most cases,
+:c:func:`ata_port_disable` can be used as this hook.
+
+Called from :c:func:`ata_bus_probe` on a failed probe. Called from
+:c:func:`ata_scsi_release`.
+
+Post-IDENTIFY device configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ void (*dev_config) (struct ata_port *, struct ata_device *);
+
+
+Called after IDENTIFY [PACKET] DEVICE is issued to each device found.
+Typically used to apply device-specific fixups prior to issue of SET
+FEATURES - XFER MODE, and prior to operation.
+
+This entry may be specified as NULL in ata_port_operations.
+
+Set PIO/DMA mode
+~~~~~~~~~~~~~~~~
+
+::
+
+ void (*set_piomode) (struct ata_port *, struct ata_device *);
+ void (*set_dmamode) (struct ata_port *, struct ata_device *);
+ void (*post_set_mode) (struct ata_port *);
+ unsigned int (*mode_filter) (struct ata_port *, struct ata_device *, unsigned int);
+
+
+Hooks called prior to the issue of SET FEATURES - XFER MODE command. The
+optional ``->mode_filter()`` hook is called when libata has built a mask of
+the possible modes. This is passed to the ``->mode_filter()`` function
+which should return a mask of valid modes after filtering those
+unsuitable due to hardware limits. It is not valid to use this interface
+to add modes.
+
+``dev->pio_mode`` and ``dev->dma_mode`` are guaranteed to be valid when
+``->set_piomode()`` and when ``->set_dmamode()`` is called. The timings for
+any other drive sharing the cable will also be valid at this point. That
+is the library records the decisions for the modes of each drive on a
+channel before it attempts to set any of them.
+
+``->post_set_mode()`` is called unconditionally, after the SET FEATURES -
+XFER MODE command completes successfully.
+
+``->set_piomode()`` is always called (if present), but ``->set_dma_mode()``
+is only called if DMA is possible.
+
+Taskfile read/write
+~~~~~~~~~~~~~~~~~~~
+
+::
+
+ void (*sff_tf_load) (struct ata_port *ap, struct ata_taskfile *tf);
+ void (*sff_tf_read) (struct ata_port *ap, struct ata_taskfile *tf);
+
+
+``->tf_load()`` is called to load the given taskfile into hardware
+registers / DMA buffers. ``->tf_read()`` is called to read the hardware
+registers / DMA buffers, to obtain the current set of taskfile register
+values. Most drivers for taskfile-based hardware (PIO or MMIO) use
+:c:func:`ata_sff_tf_load` and :c:func:`ata_sff_tf_read` for these hooks.
+
+PIO data read/write
+~~~~~~~~~~~~~~~~~~~
+
+::
+
+ void (*sff_data_xfer) (struct ata_device *, unsigned char *, unsigned int, int);
+
+
+All bmdma-style drivers must implement this hook. This is the low-level
+operation that actually copies the data bytes during a PIO data
+transfer. Typically the driver will choose one of
+:c:func:`ata_sff_data_xfer_noirq`, :c:func:`ata_sff_data_xfer`, or
+:c:func:`ata_sff_data_xfer32`.
+
+ATA command execute
+~~~~~~~~~~~~~~~~~~~
+
+::
+
+ void (*sff_exec_command)(struct ata_port *ap, struct ata_taskfile *tf);
+
+
+causes an ATA command, previously loaded with ``->tf_load()``, to be
+initiated in hardware. Most drivers for taskfile-based hardware use
+:c:func:`ata_sff_exec_command` for this hook.
+
+Per-cmd ATAPI DMA capabilities filter
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ int (*check_atapi_dma) (struct ata_queued_cmd *qc);
+
+
+Allow low-level driver to filter ATA PACKET commands, returning a status
+indicating whether or not it is OK to use DMA for the supplied PACKET
+command.
+
+This hook may be specified as NULL, in which case libata will assume
+that atapi dma can be supported.
+
+Read specific ATA shadow registers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ u8 (*sff_check_status)(struct ata_port *ap);
+ u8 (*sff_check_altstatus)(struct ata_port *ap);
+
+
+Reads the Status/AltStatus ATA shadow register from hardware. On some
+hardware, reading the Status register has the side effect of clearing
+the interrupt condition. Most drivers for taskfile-based hardware use
+:c:func:`ata_sff_check_status` for this hook.
+
+Write specific ATA shadow register
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ void (*sff_set_devctl)(struct ata_port *ap, u8 ctl);
+
+
+Write the device control ATA shadow register to the hardware. Most
+drivers don't need to define this.
+
+Select ATA device on bus
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ void (*sff_dev_select)(struct ata_port *ap, unsigned int device);
+
+
+Issues the low-level hardware command(s) that causes one of N hardware
+devices to be considered 'selected' (active and available for use) on
+the ATA bus. This generally has no meaning on FIS-based devices.
+
+Most drivers for taskfile-based hardware use :c:func:`ata_sff_dev_select` for
+this hook.
+
+Private tuning method
+~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ void (*set_mode) (struct ata_port *ap);
+
+
+By default libata performs drive and controller tuning in accordance
+with the ATA timing rules and also applies blacklists and cable limits.
+Some controllers need special handling and have custom tuning rules,
+typically raid controllers that use ATA commands but do not actually do
+drive timing.
+
+ **Warning**
+
+ This hook should not be used to replace the standard controller
+ tuning logic when a controller has quirks. Replacing the default
+ tuning logic in that case would bypass handling for drive and bridge
+ quirks that may be important to data reliability. If a controller
+ needs to filter the mode selection it should use the mode_filter
+ hook instead.
+
+Control PCI IDE BMDMA engine
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ void (*bmdma_setup) (struct ata_queued_cmd *qc);
+ void (*bmdma_start) (struct ata_queued_cmd *qc);
+ void (*bmdma_stop) (struct ata_port *ap);
+ u8 (*bmdma_status) (struct ata_port *ap);
+
+
+When setting up an IDE BMDMA transaction, these hooks arm
+(``->bmdma_setup``), fire (``->bmdma_start``), and halt (``->bmdma_stop``) the
+hardware's DMA engine. ``->bmdma_status`` is used to read the standard PCI
+IDE DMA Status register.
+
+These hooks are typically either no-ops, or simply not implemented, in
+FIS-based drivers.
+
+Most legacy IDE drivers use :c:func:`ata_bmdma_setup` for the
+:c:func:`bmdma_setup` hook. :c:func:`ata_bmdma_setup` will write the pointer
+to the PRD table to the IDE PRD Table Address register, enable DMA in the DMA
+Command register, and call :c:func:`exec_command` to begin the transfer.
+
+Most legacy IDE drivers use :c:func:`ata_bmdma_start` for the
+:c:func:`bmdma_start` hook. :c:func:`ata_bmdma_start` will write the
+ATA_DMA_START flag to the DMA Command register.
+
+Many legacy IDE drivers use :c:func:`ata_bmdma_stop` for the
+:c:func:`bmdma_stop` hook. :c:func:`ata_bmdma_stop` clears the ATA_DMA_START
+flag in the DMA command register.
+
+Many legacy IDE drivers use :c:func:`ata_bmdma_status` as the
+:c:func:`bmdma_status` hook.
+
+High-level taskfile hooks
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ void (*qc_prep) (struct ata_queued_cmd *qc);
+ int (*qc_issue) (struct ata_queued_cmd *qc);
+
+
+Higher-level hooks, these two hooks can potentially supercede several of
+the above taskfile/DMA engine hooks. ``->qc_prep`` is called after the
+buffers have been DMA-mapped, and is typically used to populate the
+hardware's DMA scatter-gather table. Most drivers use the standard
+:c:func:`ata_qc_prep` helper function, but more advanced drivers roll their
+own.
+
+``->qc_issue`` is used to make a command active, once the hardware and S/G
+tables have been prepared. IDE BMDMA drivers use the helper function
+:c:func:`ata_qc_issue_prot` for taskfile protocol-based dispatch. More
+advanced drivers implement their own ``->qc_issue``.
+
+:c:func:`ata_qc_issue_prot` calls ``->tf_load()``, ``->bmdma_setup()``, and
+``->bmdma_start()`` as necessary to initiate a transfer.
+
+Exception and probe handling (EH)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ void (*eng_timeout) (struct ata_port *ap);
+ void (*phy_reset) (struct ata_port *ap);
+
+
+Deprecated. Use ``->error_handler()`` instead.
+
+::
+
+ void (*freeze) (struct ata_port *ap);
+ void (*thaw) (struct ata_port *ap);
+
+
+:c:func:`ata_port_freeze` is called when HSM violations or some other
+condition disrupts normal operation of the port. A frozen port is not
+allowed to perform any operation until the port is thawed, which usually
+follows a successful reset.
+
+The optional ``->freeze()`` callback can be used for freezing the port
+hardware-wise (e.g. mask interrupt and stop DMA engine). If a port
+cannot be frozen hardware-wise, the interrupt handler must ack and clear
+interrupts unconditionally while the port is frozen.
+
+The optional ``->thaw()`` callback is called to perform the opposite of
+``->freeze()``: prepare the port for normal operation once again. Unmask
+interrupts, start DMA engine, etc.
+
+::
+
+ void (*error_handler) (struct ata_port *ap);
+
+
+``->error_handler()`` is a driver's hook into probe, hotplug, and recovery
+and other exceptional conditions. The primary responsibility of an
+implementation is to call :c:func:`ata_do_eh` or :c:func:`ata_bmdma_drive_eh`
+with a set of EH hooks as arguments:
+
+'prereset' hook (may be NULL) is called during an EH reset, before any
+other actions are taken.
+
+'postreset' hook (may be NULL) is called after the EH reset is
+performed. Based on existing conditions, severity of the problem, and
+hardware capabilities,
+
+Either 'softreset' (may be NULL) or 'hardreset' (may be NULL) will be
+called to perform the low-level EH reset.
+
+::
+
+ void (*post_internal_cmd) (struct ata_queued_cmd *qc);
+
+
+Perform any hardware-specific actions necessary to finish processing
+after executing a probe-time or EH-time command via
+:c:func:`ata_exec_internal`.
+
+Hardware interrupt handling
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ irqreturn_t (*irq_handler)(int, void *, struct pt_regs *);
+ void (*irq_clear) (struct ata_port *);
+
+
+``->irq_handler`` is the interrupt handling routine registered with the
+system, by libata. ``->irq_clear`` is called during probe just before the
+interrupt handler is registered, to be sure hardware is quiet.
+
+The second argument, dev_instance, should be cast to a pointer to
+:c:type:`struct ata_host_set <ata_host_set>`.
+
+Most legacy IDE drivers use :c:func:`ata_sff_interrupt` for the irq_handler
+hook, which scans all ports in the host_set, determines which queued
+command was active (if any), and calls ata_sff_host_intr(ap,qc).
+
+Most legacy IDE drivers use :c:func:`ata_sff_irq_clear` for the
+:c:func:`irq_clear` hook, which simply clears the interrupt and error flags
+in the DMA status register.
+
+SATA phy read/write
+~~~~~~~~~~~~~~~~~~~
+
+::
+
+ int (*scr_read) (struct ata_port *ap, unsigned int sc_reg,
+ u32 *val);
+ int (*scr_write) (struct ata_port *ap, unsigned int sc_reg,
+ u32 val);
+
+
+Read and write standard SATA phy registers. Currently only used if
+``->phy_reset`` hook called the :c:func:`sata_phy_reset` helper function.
+sc_reg is one of SCR_STATUS, SCR_CONTROL, SCR_ERROR, or SCR_ACTIVE.
+
+Init and shutdown
+~~~~~~~~~~~~~~~~~
+
+::
+
+ int (*port_start) (struct ata_port *ap);
+ void (*port_stop) (struct ata_port *ap);
+ void (*host_stop) (struct ata_host_set *host_set);
+
+
+``->port_start()`` is called just after the data structures for each port
+are initialized. Typically this is used to alloc per-port DMA buffers /
+tables / rings, enable DMA engines, and similar tasks. Some drivers also
+use this entry point as a chance to allocate driver-private memory for
+``ap->private_data``.
+
+Many drivers use :c:func:`ata_port_start` as this hook or call it from their
+own :c:func:`port_start` hooks. :c:func:`ata_port_start` allocates space for
+a legacy IDE PRD table and returns.
+
+``->port_stop()`` is called after ``->host_stop()``. Its sole function is to
+release DMA/memory resources, now that they are no longer actively being
+used. Many drivers also free driver-private data from port at this time.
+
+``->host_stop()`` is called after all ``->port_stop()`` calls have completed.
+The hook must finalize hardware shutdown, release DMA and other
+resources, etc. This hook may be specified as NULL, in which case it is
+not called.
+
+Error handling
+==============
+
+This chapter describes how errors are handled under libata. Readers are
+advised to read SCSI EH (Documentation/scsi/scsi_eh.txt) and ATA
+exceptions doc first.
+
+Origins of commands
+-------------------
+
+In libata, a command is represented with
+:c:type:`struct ata_queued_cmd <ata_queued_cmd>` or qc.
+qc's are preallocated during port initialization and repetitively used
+for command executions. Currently only one qc is allocated per port but
+yet-to-be-merged NCQ branch allocates one for each tag and maps each qc
+to NCQ tag 1-to-1.
+
+libata commands can originate from two sources - libata itself and SCSI
+midlayer. libata internal commands are used for initialization and error
+handling. All normal blk requests and commands for SCSI emulation are
+passed as SCSI commands through queuecommand callback of SCSI host
+template.
+
+How commands are issued
+-----------------------
+
+Internal commands
+ First, qc is allocated and initialized using :c:func:`ata_qc_new_init`.
+ Although :c:func:`ata_qc_new_init` doesn't implement any wait or retry
+ mechanism when qc is not available, internal commands are currently
+ issued only during initialization and error recovery, so no other
+ command is active and allocation is guaranteed to succeed.
+
+ Once allocated qc's taskfile is initialized for the command to be
+ executed. qc currently has two mechanisms to notify completion. One
+ is via ``qc->complete_fn()`` callback and the other is completion
+ ``qc->waiting``. ``qc->complete_fn()`` callback is the asynchronous path
+ used by normal SCSI translated commands and ``qc->waiting`` is the
+ synchronous (issuer sleeps in process context) path used by internal
+ commands.
+
+ Once initialization is complete, host_set lock is acquired and the
+ qc is issued.
+
+SCSI commands
+ All libata drivers use :c:func:`ata_scsi_queuecmd` as
+ ``hostt->queuecommand`` callback. scmds can either be simulated or
+ translated. No qc is involved in processing a simulated scmd. The
+ result is computed right away and the scmd is completed.
+
+ For a translated scmd, :c:func:`ata_qc_new_init` is invoked to allocate a
+ qc and the scmd is translated into the qc. SCSI midlayer's
+ completion notification function pointer is stored into
+ ``qc->scsidone``.
+
+ ``qc->complete_fn()`` callback is used for completion notification. ATA
+ commands use :c:func:`ata_scsi_qc_complete` while ATAPI commands use
+ :c:func:`atapi_qc_complete`. Both functions end up calling ``qc->scsidone``
+ to notify upper layer when the qc is finished. After translation is
+ completed, the qc is issued with :c:func:`ata_qc_issue`.
+
+ Note that SCSI midlayer invokes hostt->queuecommand while holding
+ host_set lock, so all above occur while holding host_set lock.
+
+How commands are processed
+--------------------------
+
+Depending on which protocol and which controller are used, commands are
+processed differently. For the purpose of discussion, a controller which
+uses taskfile interface and all standard callbacks is assumed.
+
+Currently 6 ATA command protocols are used. They can be sorted into the
+following four categories according to how they are processed.
+
+ATA NO DATA or DMA
+ ATA_PROT_NODATA and ATA_PROT_DMA fall into this category. These
+ types of commands don't require any software intervention once
+ issued. Device will raise interrupt on completion.
+
+ATA PIO
+ ATA_PROT_PIO is in this category. libata currently implements PIO
+ with polling. ATA_NIEN bit is set to turn off interrupt and
+ pio_task on ata_wq performs polling and IO.
+
+ATAPI NODATA or DMA
+ ATA_PROT_ATAPI_NODATA and ATA_PROT_ATAPI_DMA are in this
+ category. packet_task is used to poll BSY bit after issuing PACKET
+ command. Once BSY is turned off by the device, packet_task
+ transfers CDB and hands off processing to interrupt handler.
+
+ATAPI PIO
+ ATA_PROT_ATAPI is in this category. ATA_NIEN bit is set and, as
+ in ATAPI NODATA or DMA, packet_task submits cdb. However, after
+ submitting cdb, further processing (data transfer) is handed off to
+ pio_task.
+
+How commands are completed
+--------------------------
+
+Once issued, all qc's are either completed with :c:func:`ata_qc_complete` or
+time out. For commands which are handled by interrupts,
+:c:func:`ata_host_intr` invokes :c:func:`ata_qc_complete`, and, for PIO tasks,
+pio_task invokes :c:func:`ata_qc_complete`. In error cases, packet_task may
+also complete commands.
+
+:c:func:`ata_qc_complete` does the following.
+
+1. DMA memory is unmapped.
+
+2. ATA_QCFLAG_ACTIVE is cleared from qc->flags.
+
+3. :c:func:`qc->complete_fn` callback is invoked. If the return value of the
+ callback is not zero. Completion is short circuited and
+ :c:func:`ata_qc_complete` returns.
+
+4. :c:func:`__ata_qc_complete` is called, which does
+
+ 1. ``qc->flags`` is cleared to zero.
+
+ 2. ``ap->active_tag`` and ``qc->tag`` are poisoned.
+
+ 3. ``qc->waiting`` is cleared & completed (in that order).
+
+ 4. qc is deallocated by clearing appropriate bit in ``ap->qactive``.
+
+So, it basically notifies upper layer and deallocates qc. One exception
+is short-circuit path in #3 which is used by :c:func:`atapi_qc_complete`.
+
+For all non-ATAPI commands, whether it fails or not, almost the same
+code path is taken and very little error handling takes place. A qc is
+completed with success status if it succeeded, with failed status
+otherwise.
+
+However, failed ATAPI commands require more handling as REQUEST SENSE is
+needed to acquire sense data. If an ATAPI command fails,
+:c:func:`ata_qc_complete` is invoked with error status, which in turn invokes
+:c:func:`atapi_qc_complete` via ``qc->complete_fn()`` callback.
+
+This makes :c:func:`atapi_qc_complete` set ``scmd->result`` to
+SAM_STAT_CHECK_CONDITION, complete the scmd and return 1. As the
+sense data is empty but ``scmd->result`` is CHECK CONDITION, SCSI midlayer
+will invoke EH for the scmd, and returning 1 makes :c:func:`ata_qc_complete`
+to return without deallocating the qc. This leads us to
+:c:func:`ata_scsi_error` with partially completed qc.
+
+:c:func:`ata_scsi_error`
+------------------------
+
+:c:func:`ata_scsi_error` is the current ``transportt->eh_strategy_handler()``
+for libata. As discussed above, this will be entered in two cases -
+timeout and ATAPI error completion. This function calls low level libata
+driver's :c:func:`eng_timeout` callback, the standard callback for which is
+:c:func:`ata_eng_timeout`. It checks if a qc is active and calls
+:c:func:`ata_qc_timeout` on the qc if so. Actual error handling occurs in
+:c:func:`ata_qc_timeout`.
+
+If EH is invoked for timeout, :c:func:`ata_qc_timeout` stops BMDMA and
+completes the qc. Note that as we're currently in EH, we cannot call
+scsi_done. As described in SCSI EH doc, a recovered scmd should be
+either retried with :c:func:`scsi_queue_insert` or finished with
+:c:func:`scsi_finish_command`. Here, we override ``qc->scsidone`` with
+:c:func:`scsi_finish_command` and calls :c:func:`ata_qc_complete`.
+
+If EH is invoked due to a failed ATAPI qc, the qc here is completed but
+not deallocated. The purpose of this half-completion is to use the qc as
+place holder to make EH code reach this place. This is a bit hackish,
+but it works.
+
+Once control reaches here, the qc is deallocated by invoking
+:c:func:`__ata_qc_complete` explicitly. Then, internal qc for REQUEST SENSE
+is issued. Once sense data is acquired, scmd is finished by directly
+invoking :c:func:`scsi_finish_command` on the scmd. Note that as we already
+have completed and deallocated the qc which was associated with the
+scmd, we don't need to/cannot call :c:func:`ata_qc_complete` again.
+
+Problems with the current EH
+----------------------------
+
+- Error representation is too crude. Currently any and all error
+ conditions are represented with ATA STATUS and ERROR registers.
+ Errors which aren't ATA device errors are treated as ATA device
+ errors by setting ATA_ERR bit. Better error descriptor which can
+ properly represent ATA and other errors/exceptions is needed.
+
+- When handling timeouts, no action is taken to make device forget
+ about the timed out command and ready for new commands.
+
+- EH handling via :c:func:`ata_scsi_error` is not properly protected from
+ usual command processing. On EH entrance, the device is not in
+ quiescent state. Timed out commands may succeed or fail any time.
+ pio_task and atapi_task may still be running.
+
+- Too weak error recovery. Devices / controllers causing HSM mismatch
+ errors and other errors quite often require reset to return to known
+ state. Also, advanced error handling is necessary to support features
+ like NCQ and hotplug.
+
+- ATA errors are directly handled in the interrupt handler and PIO
+ errors in pio_task. This is problematic for advanced error handling
+ for the following reasons.
+
+ First, advanced error handling often requires context and internal qc
+ execution.
+
+ Second, even a simple failure (say, CRC error) needs information
+ gathering and could trigger complex error handling (say, resetting &
+ reconfiguring). Having multiple code paths to gather information,
+ enter EH and trigger actions makes life painful.
+
+ Third, scattered EH code makes implementing low level drivers
+ difficult. Low level drivers override libata callbacks. If EH is
+ scattered over several places, each affected callbacks should perform
+ its part of error handling. This can be error prone and painful.
+
+libata Library
+==============
+
+.. kernel-doc:: drivers/ata/libata-core.c
+ :export:
+
+libata Core Internals
+=====================
+
+.. kernel-doc:: drivers/ata/libata-core.c
+ :internal:
+
+.. kernel-doc:: drivers/ata/libata-eh.c
+
+libata SCSI translation/emulation
+=================================
+
+.. kernel-doc:: drivers/ata/libata-scsi.c
+ :export:
+
+.. kernel-doc:: drivers/ata/libata-scsi.c
+ :internal:
+
+ATA errors and exceptions
+=========================
+
+This chapter tries to identify what error/exception conditions exist for
+ATA/ATAPI devices and describe how they should be handled in
+implementation-neutral way.
+
+The term 'error' is used to describe conditions where either an explicit
+error condition is reported from device or a command has timed out.
+
+The term 'exception' is either used to describe exceptional conditions
+which are not errors (say, power or hotplug events), or to describe both
+errors and non-error exceptional conditions. Where explicit distinction
+between error and exception is necessary, the term 'non-error exception'
+is used.
+
+Exception categories
+--------------------
+
+Exceptions are described primarily with respect to legacy taskfile + bus
+master IDE interface. If a controller provides other better mechanism
+for error reporting, mapping those into categories described below
+shouldn't be difficult.
+
+In the following sections, two recovery actions - reset and
+reconfiguring transport - are mentioned. These are described further in
+`EH recovery actions <#exrec>`__.
+
+HSM violation
+~~~~~~~~~~~~~
+
+This error is indicated when STATUS value doesn't match HSM requirement
+during issuing or execution any ATA/ATAPI command.
+
+- ATA_STATUS doesn't contain !BSY && DRDY && !DRQ while trying to
+ issue a command.
+
+- !BSY && !DRQ during PIO data transfer.
+
+- DRQ on command completion.
+
+- !BSY && ERR after CDB transfer starts but before the last byte of CDB
+ is transferred. ATA/ATAPI standard states that "The device shall not
+ terminate the PACKET command with an error before the last byte of
+ the command packet has been written" in the error outputs description
+ of PACKET command and the state diagram doesn't include such
+ transitions.
+
+In these cases, HSM is violated and not much information regarding the
+error can be acquired from STATUS or ERROR register. IOW, this error can
+be anything - driver bug, faulty device, controller and/or cable.
+
+As HSM is violated, reset is necessary to restore known state.
+Reconfiguring transport for lower speed might be helpful too as
+transmission errors sometimes cause this kind of errors.
+
+ATA/ATAPI device error (non-NCQ / non-CHECK CONDITION)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+These are errors detected and reported by ATA/ATAPI devices indicating
+device problems. For this type of errors, STATUS and ERROR register
+values are valid and describe error condition. Note that some of ATA bus
+errors are detected by ATA/ATAPI devices and reported using the same
+mechanism as device errors. Those cases are described later in this
+section.
+
+For ATA commands, this type of errors are indicated by !BSY && ERR
+during command execution and on completion.
+
+For ATAPI commands,
+
+- !BSY && ERR && ABRT right after issuing PACKET indicates that PACKET
+ command is not supported and falls in this category.
+
+- !BSY && ERR(==CHK) && !ABRT after the last byte of CDB is transferred
+ indicates CHECK CONDITION and doesn't fall in this category.
+
+- !BSY && ERR(==CHK) && ABRT after the last byte of CDB is transferred
+ \*probably\* indicates CHECK CONDITION and doesn't fall in this
+ category.
+
+Of errors detected as above, the following are not ATA/ATAPI device
+errors but ATA bus errors and should be handled according to
+`ATA bus error <#excatATAbusErr>`__.
+
+CRC error during data transfer
+ This is indicated by ICRC bit in the ERROR register and means that
+ corruption occurred during data transfer. Up to ATA/ATAPI-7, the
+ standard specifies that this bit is only applicable to UDMA
+ transfers but ATA/ATAPI-8 draft revision 1f says that the bit may be
+ applicable to multiword DMA and PIO.
+
+ABRT error during data transfer or on completion
+ Up to ATA/ATAPI-7, the standard specifies that ABRT could be set on
+ ICRC errors and on cases where a device is not able to complete a
+ command. Combined with the fact that MWDMA and PIO transfer errors
+ aren't allowed to use ICRC bit up to ATA/ATAPI-7, it seems to imply
+ that ABRT bit alone could indicate transfer errors.
+
+ However, ATA/ATAPI-8 draft revision 1f removes the part that ICRC
+ errors can turn on ABRT. So, this is kind of gray area. Some
+ heuristics are needed here.
+
+ATA/ATAPI device errors can be further categorized as follows.
+
+Media errors
+ This is indicated by UNC bit in the ERROR register. ATA devices
+ reports UNC error only after certain number of retries cannot
+ recover the data, so there's nothing much else to do other than
+ notifying upper layer.
+
+ READ and WRITE commands report CHS or LBA of the first failed sector
+ but ATA/ATAPI standard specifies that the amount of transferred data
+ on error completion is indeterminate, so we cannot assume that
+ sectors preceding the failed sector have been transferred and thus
+ cannot complete those sectors successfully as SCSI does.
+
+Media changed / media change requested error
+ <<TODO: fill here>>
+
+Address error
+ This is indicated by IDNF bit in the ERROR register. Report to upper
+ layer.
+
+Other errors
+ This can be invalid command or parameter indicated by ABRT ERROR bit
+ or some other error condition. Note that ABRT bit can indicate a lot
+ of things including ICRC and Address errors. Heuristics needed.
+
+Depending on commands, not all STATUS/ERROR bits are applicable. These
+non-applicable bits are marked with "na" in the output descriptions but
+up to ATA/ATAPI-7 no definition of "na" can be found. However,
+ATA/ATAPI-8 draft revision 1f describes "N/A" as follows.
+
+ 3.2.3.3a N/A
+ A keyword the indicates a field has no defined value in this
+ standard and should not be checked by the host or device. N/A
+ fields should be cleared to zero.
+
+So, it seems reasonable to assume that "na" bits are cleared to zero by
+devices and thus need no explicit masking.
+
+ATAPI device CHECK CONDITION
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ATAPI device CHECK CONDITION error is indicated by set CHK bit (ERR bit)
+in the STATUS register after the last byte of CDB is transferred for a
+PACKET command. For this kind of errors, sense data should be acquired
+to gather information regarding the errors. REQUEST SENSE packet command
+should be used to acquire sense data.
+
+Once sense data is acquired, this type of errors can be handled
+similarly to other SCSI errors. Note that sense data may indicate ATA
+bus error (e.g. Sense Key 04h HARDWARE ERROR && ASC/ASCQ 47h/00h SCSI
+PARITY ERROR). In such cases, the error should be considered as an ATA
+bus error and handled according to `ATA bus error <#excatATAbusErr>`__.
+
+ATA device error (NCQ)
+~~~~~~~~~~~~~~~~~~~~~~
+
+NCQ command error is indicated by cleared BSY and set ERR bit during NCQ
+command phase (one or more NCQ commands outstanding). Although STATUS
+and ERROR registers will contain valid values describing the error, READ
+LOG EXT is required to clear the error condition, determine which
+command has failed and acquire more information.
+
+READ LOG EXT Log Page 10h reports which tag has failed and taskfile
+register values describing the error. With this information the failed
+command can be handled as a normal ATA command error as in
+`ATA/ATAPI device error (non-NCQ / non-CHECK CONDITION) <#excatDevErr>`__
+and all other in-flight commands must be retried. Note that this retry
+should not be counted - it's likely that commands retried this way would
+have completed normally if it were not for the failed command.
+
+Note that ATA bus errors can be reported as ATA device NCQ errors. This
+should be handled as described in `ATA bus error <#excatATAbusErr>`__.
+
+If READ LOG EXT Log Page 10h fails or reports NQ, we're thoroughly
+screwed. This condition should be treated according to
+`HSM violation <#excatHSMviolation>`__.
+
+ATA bus error
+~~~~~~~~~~~~~
+
+ATA bus error means that data corruption occurred during transmission
+over ATA bus (SATA or PATA). This type of errors can be indicated by
+
+- ICRC or ABRT error as described in
+ `ATA/ATAPI device error (non-NCQ / non-CHECK CONDITION) <#excatDevErr>`__.
+
+- Controller-specific error completion with error information
+ indicating transmission error.
+
+- On some controllers, command timeout. In this case, there may be a
+ mechanism to determine that the timeout is due to transmission error.
+
+- Unknown/random errors, timeouts and all sorts of weirdities.
+
+As described above, transmission errors can cause wide variety of
+symptoms ranging from device ICRC error to random device lockup, and,
+for many cases, there is no way to tell if an error condition is due to
+transmission error or not; therefore, it's necessary to employ some kind
+of heuristic when dealing with errors and timeouts. For example,
+encountering repetitive ABRT errors for known supported command is
+likely to indicate ATA bus error.
+
+Once it's determined that ATA bus errors have possibly occurred,
+lowering ATA bus transmission speed is one of actions which may
+alleviate the problem. See `Reconfigure transport <#exrecReconf>`__ for
+more information.
+
+PCI bus error
+~~~~~~~~~~~~~
+
+Data corruption or other failures during transmission over PCI (or other
+system bus). For standard BMDMA, this is indicated by Error bit in the
+BMDMA Status register. This type of errors must be logged as it
+indicates something is very wrong with the system. Resetting host
+controller is recommended.
+
+Late completion
+~~~~~~~~~~~~~~~
+
+This occurs when timeout occurs and the timeout handler finds out that
+the timed out command has completed successfully or with error. This is
+usually caused by lost interrupts. This type of errors must be logged.
+Resetting host controller is recommended.
+
+Unknown error (timeout)
+~~~~~~~~~~~~~~~~~~~~~~~
+
+This is when timeout occurs and the command is still processing or the
+host and device are in unknown state. When this occurs, HSM could be in
+any valid or invalid state. To bring the device to known state and make
+it forget about the timed out command, resetting is necessary. The timed
+out command may be retried.
+
+Timeouts can also be caused by transmission errors. Refer to
+`ATA bus error <#excatATAbusErr>`__ for more details.
+
+Hotplug and power management exceptions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+<<TODO: fill here>>
+
+EH recovery actions
+-------------------
+
+This section discusses several important recovery actions.
+
+Clearing error condition
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Many controllers require its error registers to be cleared by error
+handler. Different controllers may have different requirements.
+
+For SATA, it's strongly recommended to clear at least SError register
+during error handling.
+
+Reset
+~~~~~
+
+During EH, resetting is necessary in the following cases.
+
+- HSM is in unknown or invalid state
+
+- HBA is in unknown or invalid state
+
+- EH needs to make HBA/device forget about in-flight commands
+
+- HBA/device behaves weirdly
+
+Resetting during EH might be a good idea regardless of error condition
+to improve EH robustness. Whether to reset both or either one of HBA and
+device depends on situation but the following scheme is recommended.
+
+- When it's known that HBA is in ready state but ATA/ATAPI device is in
+ unknown state, reset only device.
+
+- If HBA is in unknown state, reset both HBA and device.
+
+HBA resetting is implementation specific. For a controller complying to
+taskfile/BMDMA PCI IDE, stopping active DMA transaction may be
+sufficient iff BMDMA state is the only HBA context. But even mostly
+taskfile/BMDMA PCI IDE complying controllers may have implementation
+specific requirements and mechanism to reset themselves. This must be
+addressed by specific drivers.
+
+OTOH, ATA/ATAPI standard describes in detail ways to reset ATA/ATAPI
+devices.
+
+PATA hardware reset
+ This is hardware initiated device reset signalled with asserted PATA
+ RESET- signal. There is no standard way to initiate hardware reset
+ from software although some hardware provides registers that allow
+ driver to directly tweak the RESET- signal.
+
+Software reset
+ This is achieved by turning CONTROL SRST bit on for at least 5us.
+ Both PATA and SATA support it but, in case of SATA, this may require
+ controller-specific support as the second Register FIS to clear SRST
+ should be transmitted while BSY bit is still set. Note that on PATA,
+ this resets both master and slave devices on a channel.
+
+EXECUTE DEVICE DIAGNOSTIC command
+ Although ATA/ATAPI standard doesn't describe exactly, EDD implies
+ some level of resetting, possibly similar level with software reset.
+ Host-side EDD protocol can be handled with normal command processing
+ and most SATA controllers should be able to handle EDD's just like
+ other commands. As in software reset, EDD affects both devices on a
+ PATA bus.
+
+ Although EDD does reset devices, this doesn't suit error handling as
+ EDD cannot be issued while BSY is set and it's unclear how it will
+ act when device is in unknown/weird state.
+
+ATAPI DEVICE RESET command
+ This is very similar to software reset except that reset can be
+ restricted to the selected device without affecting the other device
+ sharing the cable.
+
+SATA phy reset
+ This is the preferred way of resetting a SATA device. In effect,
+ it's identical to PATA hardware reset. Note that this can be done
+ with the standard SCR Control register. As such, it's usually easier
+ to implement than software reset.
+
+One more thing to consider when resetting devices is that resetting
+clears certain configuration parameters and they need to be set to their
+previous or newly adjusted values after reset.
+
+Parameters affected are.
+
+- CHS set up with INITIALIZE DEVICE PARAMETERS (seldom used)
+
+- Parameters set with SET FEATURES including transfer mode setting
+
+- Block count set with SET MULTIPLE MODE
+
+- Other parameters (SET MAX, MEDIA LOCK...)
+
+ATA/ATAPI standard specifies that some parameters must be maintained
+across hardware or software reset, but doesn't strictly specify all of
+them. Always reconfiguring needed parameters after reset is required for
+robustness. Note that this also applies when resuming from deep sleep
+(power-off).
+
+Also, ATA/ATAPI standard requires that IDENTIFY DEVICE / IDENTIFY PACKET
+DEVICE is issued after any configuration parameter is updated or a
+hardware reset and the result used for further operation. OS driver is
+required to implement revalidation mechanism to support this.
+
+Reconfigure transport
+~~~~~~~~~~~~~~~~~~~~~
+
+For both PATA and SATA, a lot of corners are cut for cheap connectors,
+cables or controllers and it's quite common to see high transmission
+error rate. This can be mitigated by lowering transmission speed.
+
+The following is a possible scheme Jeff Garzik suggested.
+
+ If more than $N (3?) transmission errors happen in 15 minutes,
+
+ - if SATA, decrease SATA PHY speed. if speed cannot be decreased,
+
+ - decrease UDMA xfer speed. if at UDMA0, switch to PIO4,
+
+ - decrease PIO xfer speed. if at PIO3, complain, but continue
+
+ata_piix Internals
+===================
+
+.. kernel-doc:: drivers/ata/ata_piix.c
+ :internal:
+
+sata_sil Internals
+===================
+
+.. kernel-doc:: drivers/ata/sata_sil.c
+ :internal:
+
+Thanks
+======
+
+The bulk of the ATA knowledge comes thanks to long conversations with
+Andre Hedrick (www.linux-ide.org), and long hours pondering the ATA and
+SCSI specifications.
+
+Thanks to Alan Cox for pointing out similarities between SATA and SCSI,
+and in general for motivation to hack on libata.
+
+libata's device detection method, ata_pio_devchk, and in general all
+the early probing was based on extensive study of Hale Landis's
+probe/reset code in his ATADRVR driver (www.ata-atapi.com).
--- /dev/null
+=====================================
+MTD NAND Driver Programming Interface
+=====================================
+
+:Author: Thomas Gleixner
+
+Introduction
+============
+
+The generic NAND driver supports almost all NAND and AG-AND based chips
+and connects them to the Memory Technology Devices (MTD) subsystem of
+the Linux Kernel.
+
+This documentation is provided for developers who want to implement
+board drivers or filesystem drivers suitable for NAND devices.
+
+Known Bugs And Assumptions
+==========================
+
+None.
+
+Documentation hints
+===================
+
+The function and structure docs are autogenerated. Each function and
+struct member has a short description which is marked with an [XXX]
+identifier. The following chapters explain the meaning of those
+identifiers.
+
+Function identifiers [XXX]
+--------------------------
+
+The functions are marked with [XXX] identifiers in the short comment.
+The identifiers explain the usage and scope of the functions. Following
+identifiers are used:
+
+- [MTD Interface]
+
+ These functions provide the interface to the MTD kernel API. They are
+ not replaceable and provide functionality which is complete hardware
+ independent.
+
+- [NAND Interface]
+
+ These functions are exported and provide the interface to the NAND
+ kernel API.
+
+- [GENERIC]
+
+ Generic functions are not replaceable and provide functionality which
+ is complete hardware independent.
+
+- [DEFAULT]
+
+ Default functions provide hardware related functionality which is
+ suitable for most of the implementations. These functions can be
+ replaced by the board driver if necessary. Those functions are called
+ via pointers in the NAND chip description structure. The board driver
+ can set the functions which should be replaced by board dependent
+ functions before calling nand_scan(). If the function pointer is
+ NULL on entry to nand_scan() then the pointer is set to the default
+ function which is suitable for the detected chip type.
+
+Struct member identifiers [XXX]
+-------------------------------
+
+The struct members are marked with [XXX] identifiers in the comment. The
+identifiers explain the usage and scope of the members. Following
+identifiers are used:
+
+- [INTERN]
+
+ These members are for NAND driver internal use only and must not be
+ modified. Most of these values are calculated from the chip geometry
+ information which is evaluated during nand_scan().
+
+- [REPLACEABLE]
+
+ Replaceable members hold hardware related functions which can be
+ provided by the board driver. The board driver can set the functions
+ which should be replaced by board dependent functions before calling
+ nand_scan(). If the function pointer is NULL on entry to
+ nand_scan() then the pointer is set to the default function which is
+ suitable for the detected chip type.
+
+- [BOARDSPECIFIC]
+
+ Board specific members hold hardware related information which must
+ be provided by the board driver. The board driver must set the
+ function pointers and datafields before calling nand_scan().
+
+- [OPTIONAL]
+
+ Optional members can hold information relevant for the board driver.
+ The generic NAND driver code does not use this information.
+
+Basic board driver
+==================
+
+For most boards it will be sufficient to provide just the basic
+functions and fill out some really board dependent members in the nand
+chip description structure.
+
+Basic defines
+-------------
+
+At least you have to provide a nand_chip structure and a storage for
+the ioremap'ed chip address. You can allocate the nand_chip structure
+using kmalloc or you can allocate it statically. The NAND chip structure
+embeds an mtd structure which will be registered to the MTD subsystem.
+You can extract a pointer to the mtd structure from a nand_chip pointer
+using the nand_to_mtd() helper.
+
+Kmalloc based example
+
+::
+
+ static struct mtd_info *board_mtd;
+ static void __iomem *baseaddr;
+
+
+Static example
+
+::
+
+ static struct nand_chip board_chip;
+ static void __iomem *baseaddr;
+
+
+Partition defines
+-----------------
+
+If you want to divide your device into partitions, then define a
+partitioning scheme suitable to your board.
+
+::
+
+ #define NUM_PARTITIONS 2
+ static struct mtd_partition partition_info[] = {
+ { .name = "Flash partition 1",
+ .offset = 0,
+ .size = 8 * 1024 * 1024 },
+ { .name = "Flash partition 2",
+ .offset = MTDPART_OFS_NEXT,
+ .size = MTDPART_SIZ_FULL },
+ };
+
+
+Hardware control function
+-------------------------
+
+The hardware control function provides access to the control pins of the
+NAND chip(s). The access can be done by GPIO pins or by address lines.
+If you use address lines, make sure that the timing requirements are
+met.
+
+*GPIO based example*
+
+::
+
+ static void board_hwcontrol(struct mtd_info *mtd, int cmd)
+ {
+ switch(cmd){
+ case NAND_CTL_SETCLE: /* Set CLE pin high */ break;
+ case NAND_CTL_CLRCLE: /* Set CLE pin low */ break;
+ case NAND_CTL_SETALE: /* Set ALE pin high */ break;
+ case NAND_CTL_CLRALE: /* Set ALE pin low */ break;
+ case NAND_CTL_SETNCE: /* Set nCE pin low */ break;
+ case NAND_CTL_CLRNCE: /* Set nCE pin high */ break;
+ }
+ }
+
+
+*Address lines based example.* It's assumed that the nCE pin is driven
+by a chip select decoder.
+
+::
+
+ static void board_hwcontrol(struct mtd_info *mtd, int cmd)
+ {
+ struct nand_chip *this = mtd_to_nand(mtd);
+ switch(cmd){
+ case NAND_CTL_SETCLE: this->IO_ADDR_W |= CLE_ADRR_BIT; break;
+ case NAND_CTL_CLRCLE: this->IO_ADDR_W &= ~CLE_ADRR_BIT; break;
+ case NAND_CTL_SETALE: this->IO_ADDR_W |= ALE_ADRR_BIT; break;
+ case NAND_CTL_CLRALE: this->IO_ADDR_W &= ~ALE_ADRR_BIT; break;
+ }
+ }
+
+
+Device ready function
+---------------------
+
+If the hardware interface has the ready busy pin of the NAND chip
+connected to a GPIO or other accessible I/O pin, this function is used
+to read back the state of the pin. The function has no arguments and
+should return 0, if the device is busy (R/B pin is low) and 1, if the
+device is ready (R/B pin is high). If the hardware interface does not
+give access to the ready busy pin, then the function must not be defined
+and the function pointer this->dev_ready is set to NULL.
+
+Init function
+-------------
+
+The init function allocates memory and sets up all the board specific
+parameters and function pointers. When everything is set up nand_scan()
+is called. This function tries to detect and identify then chip. If a
+chip is found all the internal data fields are initialized accordingly.
+The structure(s) have to be zeroed out first and then filled with the
+necessary information about the device.
+
+::
+
+ static int __init board_init (void)
+ {
+ struct nand_chip *this;
+ int err = 0;
+
+ /* Allocate memory for MTD device structure and private data */
+ this = kzalloc(sizeof(struct nand_chip), GFP_KERNEL);
+ if (!this) {
+ printk ("Unable to allocate NAND MTD device structure.\n");
+ err = -ENOMEM;
+ goto out;
+ }
+
+ board_mtd = nand_to_mtd(this);
+
+ /* map physical address */
+ baseaddr = ioremap(CHIP_PHYSICAL_ADDRESS, 1024);
+ if (!baseaddr) {
+ printk("Ioremap to access NAND chip failed\n");
+ err = -EIO;
+ goto out_mtd;
+ }
+
+ /* Set address of NAND IO lines */
+ this->IO_ADDR_R = baseaddr;
+ this->IO_ADDR_W = baseaddr;
+ /* Reference hardware control function */
+ this->hwcontrol = board_hwcontrol;
+ /* Set command delay time, see datasheet for correct value */
+ this->chip_delay = CHIP_DEPENDEND_COMMAND_DELAY;
+ /* Assign the device ready function, if available */
+ this->dev_ready = board_dev_ready;
+ this->eccmode = NAND_ECC_SOFT;
+
+ /* Scan to find existence of the device */
+ if (nand_scan (board_mtd, 1)) {
+ err = -ENXIO;
+ goto out_ior;
+ }
+
+ add_mtd_partitions(board_mtd, partition_info, NUM_PARTITIONS);
+ goto out;
+
+ out_ior:
+ iounmap(baseaddr);
+ out_mtd:
+ kfree (this);
+ out:
+ return err;
+ }
+ module_init(board_init);
+
+
+Exit function
+-------------
+
+The exit function is only necessary if the driver is compiled as a
+module. It releases all resources which are held by the chip driver and
+unregisters the partitions in the MTD layer.
+
+::
+
+ #ifdef MODULE
+ static void __exit board_cleanup (void)
+ {
+ /* Release resources, unregister device */
+ nand_release (board_mtd);
+
+ /* unmap physical address */
+ iounmap(baseaddr);
+
+ /* Free the MTD device structure */
+ kfree (mtd_to_nand(board_mtd));
+ }
+ module_exit(board_cleanup);
+ #endif
+
+
+Advanced board driver functions
+===============================
+
+This chapter describes the advanced functionality of the NAND driver.
+For a list of functions which can be overridden by the board driver see
+the documentation of the nand_chip structure.
+
+Multiple chip control
+---------------------
+
+The nand driver can control chip arrays. Therefore the board driver must
+provide an own select_chip function. This function must (de)select the
+requested chip. The function pointer in the nand_chip structure must be
+set before calling nand_scan(). The maxchip parameter of nand_scan()
+defines the maximum number of chips to scan for. Make sure that the
+select_chip function can handle the requested number of chips.
+
+The nand driver concatenates the chips to one virtual chip and provides
+this virtual chip to the MTD layer.
+
+*Note: The driver can only handle linear chip arrays of equally sized
+chips. There is no support for parallel arrays which extend the
+buswidth.*
+
+*GPIO based example*
+
+::
+
+ static void board_select_chip (struct mtd_info *mtd, int chip)
+ {
+ /* Deselect all chips, set all nCE pins high */
+ GPIO(BOARD_NAND_NCE) |= 0xff;
+ if (chip >= 0)
+ GPIO(BOARD_NAND_NCE) &= ~ (1 << chip);
+ }
+
+
+*Address lines based example.* Its assumed that the nCE pins are
+connected to an address decoder.
+
+::
+
+ static void board_select_chip (struct mtd_info *mtd, int chip)
+ {
+ struct nand_chip *this = mtd_to_nand(mtd);
+
+ /* Deselect all chips */
+ this->IO_ADDR_R &= ~BOARD_NAND_ADDR_MASK;
+ this->IO_ADDR_W &= ~BOARD_NAND_ADDR_MASK;
+ switch (chip) {
+ case 0:
+ this->IO_ADDR_R |= BOARD_NAND_ADDR_CHIP0;
+ this->IO_ADDR_W |= BOARD_NAND_ADDR_CHIP0;
+ break;
+ ....
+ case n:
+ this->IO_ADDR_R |= BOARD_NAND_ADDR_CHIPn;
+ this->IO_ADDR_W |= BOARD_NAND_ADDR_CHIPn;
+ break;
+ }
+ }
+
+
+Hardware ECC support
+--------------------
+
+Functions and constants
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The nand driver supports three different types of hardware ECC.
+
+- NAND_ECC_HW3_256
+
+ Hardware ECC generator providing 3 bytes ECC per 256 byte.
+
+- NAND_ECC_HW3_512
+
+ Hardware ECC generator providing 3 bytes ECC per 512 byte.
+
+- NAND_ECC_HW6_512
+
+ Hardware ECC generator providing 6 bytes ECC per 512 byte.
+
+- NAND_ECC_HW8_512
+
+ Hardware ECC generator providing 6 bytes ECC per 512 byte.
+
+If your hardware generator has a different functionality add it at the
+appropriate place in nand_base.c
+
+The board driver must provide following functions:
+
+- enable_hwecc
+
+ This function is called before reading / writing to the chip. Reset
+ or initialize the hardware generator in this function. The function
+ is called with an argument which let you distinguish between read and
+ write operations.
+
+- calculate_ecc
+
+ This function is called after read / write from / to the chip.
+ Transfer the ECC from the hardware to the buffer. If the option
+ NAND_HWECC_SYNDROME is set then the function is only called on
+ write. See below.
+
+- correct_data
+
+ In case of an ECC error this function is called for error detection
+ and correction. Return 1 respectively 2 in case the error can be
+ corrected. If the error is not correctable return -1. If your
+ hardware generator matches the default algorithm of the nand_ecc
+ software generator then use the correction function provided by
+ nand_ecc instead of implementing duplicated code.
+
+Hardware ECC with syndrome calculation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Many hardware ECC implementations provide Reed-Solomon codes and
+calculate an error syndrome on read. The syndrome must be converted to a
+standard Reed-Solomon syndrome before calling the error correction code
+in the generic Reed-Solomon library.
+
+The ECC bytes must be placed immediately after the data bytes in order
+to make the syndrome generator work. This is contrary to the usual
+layout used by software ECC. The separation of data and out of band area
+is not longer possible. The nand driver code handles this layout and the
+remaining free bytes in the oob area are managed by the autoplacement
+code. Provide a matching oob-layout in this case. See rts_from4.c and
+diskonchip.c for implementation reference. In those cases we must also
+use bad block tables on FLASH, because the ECC layout is interfering
+with the bad block marker positions. See bad block table support for
+details.
+
+Bad block table support
+-----------------------
+
+Most NAND chips mark the bad blocks at a defined position in the spare
+area. Those blocks must not be erased under any circumstances as the bad
+block information would be lost. It is possible to check the bad block
+mark each time when the blocks are accessed by reading the spare area of
+the first page in the block. This is time consuming so a bad block table
+is used.
+
+The nand driver supports various types of bad block tables.
+
+- Per device
+
+ The bad block table contains all bad block information of the device
+ which can consist of multiple chips.
+
+- Per chip
+
+ A bad block table is used per chip and contains the bad block
+ information for this particular chip.
+
+- Fixed offset
+
+ The bad block table is located at a fixed offset in the chip
+ (device). This applies to various DiskOnChip devices.
+
+- Automatic placed
+
+ The bad block table is automatically placed and detected either at
+ the end or at the beginning of a chip (device)
+
+- Mirrored tables
+
+ The bad block table is mirrored on the chip (device) to allow updates
+ of the bad block table without data loss.
+
+nand_scan() calls the function nand_default_bbt().
+nand_default_bbt() selects appropriate default bad block table
+descriptors depending on the chip information which was retrieved by
+nand_scan().
+
+The standard policy is scanning the device for bad blocks and build a
+ram based bad block table which allows faster access than always
+checking the bad block information on the flash chip itself.
+
+Flash based tables
+~~~~~~~~~~~~~~~~~~
+
+It may be desired or necessary to keep a bad block table in FLASH. For
+AG-AND chips this is mandatory, as they have no factory marked bad
+blocks. They have factory marked good blocks. The marker pattern is
+erased when the block is erased to be reused. So in case of powerloss
+before writing the pattern back to the chip this block would be lost and
+added to the bad blocks. Therefore we scan the chip(s) when we detect
+them the first time for good blocks and store this information in a bad
+block table before erasing any of the blocks.
+
+The blocks in which the tables are stored are protected against
+accidental access by marking them bad in the memory bad block table. The
+bad block table management functions are allowed to circumvent this
+protection.
+
+The simplest way to activate the FLASH based bad block table support is
+to set the option NAND_BBT_USE_FLASH in the bbt_option field of the
+nand chip structure before calling nand_scan(). For AG-AND chips is
+this done by default. This activates the default FLASH based bad block
+table functionality of the NAND driver. The default bad block table
+options are
+
+- Store bad block table per chip
+
+- Use 2 bits per block
+
+- Automatic placement at the end of the chip
+
+- Use mirrored tables with version numbers
+
+- Reserve 4 blocks at the end of the chip
+
+User defined tables
+~~~~~~~~~~~~~~~~~~~
+
+User defined tables are created by filling out a nand_bbt_descr
+structure and storing the pointer in the nand_chip structure member
+bbt_td before calling nand_scan(). If a mirror table is necessary a
+second structure must be created and a pointer to this structure must be
+stored in bbt_md inside the nand_chip structure. If the bbt_md member
+is set to NULL then only the main table is used and no scan for the
+mirrored table is performed.
+
+The most important field in the nand_bbt_descr structure is the
+options field. The options define most of the table properties. Use the
+predefined constants from nand.h to define the options.
+
+- Number of bits per block
+
+ The supported number of bits is 1, 2, 4, 8.
+
+- Table per chip
+
+ Setting the constant NAND_BBT_PERCHIP selects that a bad block
+ table is managed for each chip in a chip array. If this option is not
+ set then a per device bad block table is used.
+
+- Table location is absolute
+
+ Use the option constant NAND_BBT_ABSPAGE and define the absolute
+ page number where the bad block table starts in the field pages. If
+ you have selected bad block tables per chip and you have a multi chip
+ array then the start page must be given for each chip in the chip
+ array. Note: there is no scan for a table ident pattern performed, so
+ the fields pattern, veroffs, offs, len can be left uninitialized
+
+- Table location is automatically detected
+
+ The table can either be located in the first or the last good blocks
+ of the chip (device). Set NAND_BBT_LASTBLOCK to place the bad block
+ table at the end of the chip (device). The bad block tables are
+ marked and identified by a pattern which is stored in the spare area
+ of the first page in the block which holds the bad block table. Store
+ a pointer to the pattern in the pattern field. Further the length of
+ the pattern has to be stored in len and the offset in the spare area
+ must be given in the offs member of the nand_bbt_descr structure.
+ For mirrored bad block tables different patterns are mandatory.
+
+- Table creation
+
+ Set the option NAND_BBT_CREATE to enable the table creation if no
+ table can be found during the scan. Usually this is done only once if
+ a new chip is found.
+
+- Table write support
+
+ Set the option NAND_BBT_WRITE to enable the table write support.
+ This allows the update of the bad block table(s) in case a block has
+ to be marked bad due to wear. The MTD interface function
+ block_markbad is calling the update function of the bad block table.
+ If the write support is enabled then the table is updated on FLASH.
+
+ Note: Write support should only be enabled for mirrored tables with
+ version control.
+
+- Table version control
+
+ Set the option NAND_BBT_VERSION to enable the table version
+ control. It's highly recommended to enable this for mirrored tables
+ with write support. It makes sure that the risk of losing the bad
+ block table information is reduced to the loss of the information
+ about the one worn out block which should be marked bad. The version
+ is stored in 4 consecutive bytes in the spare area of the device. The
+ position of the version number is defined by the member veroffs in
+ the bad block table descriptor.
+
+- Save block contents on write
+
+ In case that the block which holds the bad block table does contain
+ other useful information, set the option NAND_BBT_SAVECONTENT. When
+ the bad block table is written then the whole block is read the bad
+ block table is updated and the block is erased and everything is
+ written back. If this option is not set only the bad block table is
+ written and everything else in the block is ignored and erased.
+
+- Number of reserved blocks
+
+ For automatic placement some blocks must be reserved for bad block
+ table storage. The number of reserved blocks is defined in the
+ maxblocks member of the bad block table description structure.
+ Reserving 4 blocks for mirrored tables should be a reasonable number.
+ This also limits the number of blocks which are scanned for the bad
+ block table ident pattern.
+
+Spare area (auto)placement
+--------------------------
+
+The nand driver implements different possibilities for placement of
+filesystem data in the spare area,
+
+- Placement defined by fs driver
+
+- Automatic placement
+
+The default placement function is automatic placement. The nand driver
+has built in default placement schemes for the various chiptypes. If due
+to hardware ECC functionality the default placement does not fit then
+the board driver can provide a own placement scheme.
+
+File system drivers can provide a own placement scheme which is used
+instead of the default placement scheme.
+
+Placement schemes are defined by a nand_oobinfo structure
+
+::
+
+ struct nand_oobinfo {
+ int useecc;
+ int eccbytes;
+ int eccpos[24];
+ int oobfree[8][2];
+ };
+
+
+- useecc
+
+ The useecc member controls the ecc and placement function. The header
+ file include/mtd/mtd-abi.h contains constants to select ecc and
+ placement. MTD_NANDECC_OFF switches off the ecc complete. This is
+ not recommended and available for testing and diagnosis only.
+ MTD_NANDECC_PLACE selects caller defined placement,
+ MTD_NANDECC_AUTOPLACE selects automatic placement.
+
+- eccbytes
+
+ The eccbytes member defines the number of ecc bytes per page.
+
+- eccpos
+
+ The eccpos array holds the byte offsets in the spare area where the
+ ecc codes are placed.
+
+- oobfree
+
+ The oobfree array defines the areas in the spare area which can be
+ used for automatic placement. The information is given in the format
+ {offset, size}. offset defines the start of the usable area, size the
+ length in bytes. More than one area can be defined. The list is
+ terminated by an {0, 0} entry.
+
+Placement defined by fs driver
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The calling function provides a pointer to a nand_oobinfo structure
+which defines the ecc placement. For writes the caller must provide a
+spare area buffer along with the data buffer. The spare area buffer size
+is (number of pages) \* (size of spare area). For reads the buffer size
+is (number of pages) \* ((size of spare area) + (number of ecc steps per
+page) \* sizeof (int)). The driver stores the result of the ecc check
+for each tuple in the spare buffer. The storage sequence is::
+
+ <spare data page 0><ecc result 0>...<ecc result n>
+
+ ...
+
+ <spare data page n><ecc result 0>...<ecc result n>
+
+This is a legacy mode used by YAFFS1.
+
+If the spare area buffer is NULL then only the ECC placement is done
+according to the given scheme in the nand_oobinfo structure.
+
+Automatic placement
+~~~~~~~~~~~~~~~~~~~
+
+Automatic placement uses the built in defaults to place the ecc bytes in
+the spare area. If filesystem data have to be stored / read into the
+spare area then the calling function must provide a buffer. The buffer
+size per page is determined by the oobfree array in the nand_oobinfo
+structure.
+
+If the spare area buffer is NULL then only the ECC placement is done
+according to the default builtin scheme.
+
+Spare area autoplacement default schemes
+----------------------------------------
+
+256 byte pagesize
+~~~~~~~~~~~~~~~~~
+
+======== ================== ===================================================
+Offset Content Comment
+======== ================== ===================================================
+0x00 ECC byte 0 Error correction code byte 0
+0x01 ECC byte 1 Error correction code byte 1
+0x02 ECC byte 2 Error correction code byte 2
+0x03 Autoplace 0
+0x04 Autoplace 1
+0x05 Bad block marker If any bit in this byte is zero, then this
+ block is bad. This applies only to the first
+ page in a block. In the remaining pages this
+ byte is reserved
+0x06 Autoplace 2
+0x07 Autoplace 3
+======== ================== ===================================================
+
+512 byte pagesize
+~~~~~~~~~~~~~~~~~
+
+
+============= ================== ==============================================
+Offset Content Comment
+============= ================== ==============================================
+0x00 ECC byte 0 Error correction code byte 0 of the lower
+ 256 Byte data in this page
+0x01 ECC byte 1 Error correction code byte 1 of the lower
+ 256 Bytes of data in this page
+0x02 ECC byte 2 Error correction code byte 2 of the lower
+ 256 Bytes of data in this page
+0x03 ECC byte 3 Error correction code byte 0 of the upper
+ 256 Bytes of data in this page
+0x04 reserved reserved
+0x05 Bad block marker If any bit in this byte is zero, then this
+ block is bad. This applies only to the first
+ page in a block. In the remaining pages this
+ byte is reserved
+0x06 ECC byte 4 Error correction code byte 1 of the upper
+ 256 Bytes of data in this page
+0x07 ECC byte 5 Error correction code byte 2 of the upper
+ 256 Bytes of data in this page
+0x08 - 0x0F Autoplace 0 - 7
+============= ================== ==============================================
+
+2048 byte pagesize
+~~~~~~~~~~~~~~~~~~
+
+=========== ================== ================================================
+Offset Content Comment
+=========== ================== ================================================
+0x00 Bad block marker If any bit in this byte is zero, then this block
+ is bad. This applies only to the first page in a
+ block. In the remaining pages this byte is
+ reserved
+0x01 Reserved Reserved
+0x02-0x27 Autoplace 0 - 37
+0x28 ECC byte 0 Error correction code byte 0 of the first
+ 256 Byte data in this page
+0x29 ECC byte 1 Error correction code byte 1 of the first
+ 256 Bytes of data in this page
+0x2A ECC byte 2 Error correction code byte 2 of the first
+ 256 Bytes data in this page
+0x2B ECC byte 3 Error correction code byte 0 of the second
+ 256 Bytes of data in this page
+0x2C ECC byte 4 Error correction code byte 1 of the second
+ 256 Bytes of data in this page
+0x2D ECC byte 5 Error correction code byte 2 of the second
+ 256 Bytes of data in this page
+0x2E ECC byte 6 Error correction code byte 0 of the third
+ 256 Bytes of data in this page
+0x2F ECC byte 7 Error correction code byte 1 of the third
+ 256 Bytes of data in this page
+0x30 ECC byte 8 Error correction code byte 2 of the third
+ 256 Bytes of data in this page
+0x31 ECC byte 9 Error correction code byte 0 of the fourth
+ 256 Bytes of data in this page
+0x32 ECC byte 10 Error correction code byte 1 of the fourth
+ 256 Bytes of data in this page
+0x33 ECC byte 11 Error correction code byte 2 of the fourth
+ 256 Bytes of data in this page
+0x34 ECC byte 12 Error correction code byte 0 of the fifth
+ 256 Bytes of data in this page
+0x35 ECC byte 13 Error correction code byte 1 of the fifth
+ 256 Bytes of data in this page
+0x36 ECC byte 14 Error correction code byte 2 of the fifth
+ 256 Bytes of data in this page
+0x37 ECC byte 15 Error correction code byte 0 of the sixth
+ 256 Bytes of data in this page
+0x38 ECC byte 16 Error correction code byte 1 of the sixth
+ 256 Bytes of data in this page
+0x39 ECC byte 17 Error correction code byte 2 of the sixth
+ 256 Bytes of data in this page
+0x3A ECC byte 18 Error correction code byte 0 of the seventh
+ 256 Bytes of data in this page
+0x3B ECC byte 19 Error correction code byte 1 of the seventh
+ 256 Bytes of data in this page
+0x3C ECC byte 20 Error correction code byte 2 of the seventh
+ 256 Bytes of data in this page
+0x3D ECC byte 21 Error correction code byte 0 of the eighth
+ 256 Bytes of data in this page
+0x3E ECC byte 22 Error correction code byte 1 of the eighth
+ 256 Bytes of data in this page
+0x3F ECC byte 23 Error correction code byte 2 of the eighth
+ 256 Bytes of data in this page
+=========== ================== ================================================
+
+Filesystem support
+==================
+
+The NAND driver provides all necessary functions for a filesystem via
+the MTD interface.
+
+Filesystems must be aware of the NAND peculiarities and restrictions.
+One major restrictions of NAND Flash is, that you cannot write as often
+as you want to a page. The consecutive writes to a page, before erasing
+it again, are restricted to 1-3 writes, depending on the manufacturers
+specifications. This applies similar to the spare area.
+
+Therefore NAND aware filesystems must either write in page size chunks
+or hold a writebuffer to collect smaller writes until they sum up to
+pagesize. Available NAND aware filesystems: JFFS2, YAFFS.
+
+The spare area usage to store filesystem data is controlled by the spare
+area placement functionality which is described in one of the earlier
+chapters.
+
+Tools
+=====
+
+The MTD project provides a couple of helpful tools to handle NAND Flash.
+
+- flasherase, flasheraseall: Erase and format FLASH partitions
+
+- nandwrite: write filesystem images to NAND FLASH
+
+- nanddump: dump the contents of a NAND FLASH partitions
+
+These tools are aware of the NAND restrictions. Please use those tools
+instead of complaining about errors which are caused by non NAND aware
+access methods.
+
+Constants
+=========
+
+This chapter describes the constants which might be relevant for a
+driver developer.
+
+Chip option constants
+---------------------
+
+Constants for chip id table
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+These constants are defined in nand.h. They are OR-ed together to
+describe the chip functionality::
+
+ /* Buswitdh is 16 bit */
+ #define NAND_BUSWIDTH_16 0x00000002
+ /* Device supports partial programming without padding */
+ #define NAND_NO_PADDING 0x00000004
+ /* Chip has cache program function */
+ #define NAND_CACHEPRG 0x00000008
+ /* Chip has copy back function */
+ #define NAND_COPYBACK 0x00000010
+ /* AND Chip which has 4 banks and a confusing page / block
+ * assignment. See Renesas datasheet for further information */
+ #define NAND_IS_AND 0x00000020
+ /* Chip has a array of 4 pages which can be read without
+ * additional ready /busy waits */
+ #define NAND_4PAGE_ARRAY 0x00000040
+
+
+Constants for runtime options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+These constants are defined in nand.h. They are OR-ed together to
+describe the functionality::
+
+ /* The hw ecc generator provides a syndrome instead a ecc value on read
+ * This can only work if we have the ecc bytes directly behind the
+ * data bytes. Applies for DOC and AG-AND Renesas HW Reed Solomon generators */
+ #define NAND_HWECC_SYNDROME 0x00020000
+
+
+ECC selection constants
+-----------------------
+
+Use these constants to select the ECC algorithm::
+
+ /* No ECC. Usage is not recommended ! */
+ #define NAND_ECC_NONE 0
+ /* Software ECC 3 byte ECC per 256 Byte data */
+ #define NAND_ECC_SOFT 1
+ /* Hardware ECC 3 byte ECC per 256 Byte data */
+ #define NAND_ECC_HW3_256 2
+ /* Hardware ECC 3 byte ECC per 512 Byte data */
+ #define NAND_ECC_HW3_512 3
+ /* Hardware ECC 6 byte ECC per 512 Byte data */
+ #define NAND_ECC_HW6_512 4
+ /* Hardware ECC 6 byte ECC per 512 Byte data */
+ #define NAND_ECC_HW8_512 6
+
+
+Hardware control related constants
+----------------------------------
+
+These constants describe the requested hardware access function when the
+boardspecific hardware control function is called::
+
+ /* Select the chip by setting nCE to low */
+ #define NAND_CTL_SETNCE 1
+ /* Deselect the chip by setting nCE to high */
+ #define NAND_CTL_CLRNCE 2
+ /* Select the command latch by setting CLE to high */
+ #define NAND_CTL_SETCLE 3
+ /* Deselect the command latch by setting CLE to low */
+ #define NAND_CTL_CLRCLE 4
+ /* Select the address latch by setting ALE to high */
+ #define NAND_CTL_SETALE 5
+ /* Deselect the address latch by setting ALE to low */
+ #define NAND_CTL_CLRALE 6
+ /* Set write protection by setting WP to high. Not used! */
+ #define NAND_CTL_SETWP 7
+ /* Clear write protection by setting WP to low. Not used! */
+ #define NAND_CTL_CLRWP 8
+
+
+Bad block table related constants
+---------------------------------
+
+These constants describe the options used for bad block table
+descriptors::
+
+ /* Options for the bad block table descriptors */
+
+ /* The number of bits used per block in the bbt on the device */
+ #define NAND_BBT_NRBITS_MSK 0x0000000F
+ #define NAND_BBT_1BIT 0x00000001
+ #define NAND_BBT_2BIT 0x00000002
+ #define NAND_BBT_4BIT 0x00000004
+ #define NAND_BBT_8BIT 0x00000008
+ /* The bad block table is in the last good block of the device */
+ #define NAND_BBT_LASTBLOCK 0x00000010
+ /* The bbt is at the given page, else we must scan for the bbt */
+ #define NAND_BBT_ABSPAGE 0x00000020
+ /* bbt is stored per chip on multichip devices */
+ #define NAND_BBT_PERCHIP 0x00000080
+ /* bbt has a version counter at offset veroffs */
+ #define NAND_BBT_VERSION 0x00000100
+ /* Create a bbt if none axists */
+ #define NAND_BBT_CREATE 0x00000200
+ /* Write bbt if necessary */
+ #define NAND_BBT_WRITE 0x00001000
+ /* Read and write back block contents when writing bbt */
+ #define NAND_BBT_SAVECONTENT 0x00002000
+
+
+Structures
+==========
+
+This chapter contains the autogenerated documentation of the structures
+which are used in the NAND driver and might be relevant for a driver
+developer. Each struct member has a short description which is marked
+with an [XXX] identifier. See the chapter "Documentation hints" for an
+explanation.
+
+.. kernel-doc:: include/linux/mtd/nand.h
+ :internal:
+
+Public Functions Provided
+=========================
+
+This chapter contains the autogenerated documentation of the NAND kernel
+API functions which are exported. Each function has a short description
+which is marked with an [XXX] identifier. See the chapter "Documentation
+hints" for an explanation.
+
+.. kernel-doc:: drivers/mtd/nand/nand_base.c
+ :export:
+
+.. kernel-doc:: drivers/mtd/nand/nand_ecc.c
+ :export:
+
+Internal Functions Provided
+===========================
+
+This chapter contains the autogenerated documentation of the NAND driver
+internal functions. Each function has a short description which is
+marked with an [XXX] identifier. See the chapter "Documentation hints"
+for an explanation. The functions marked with [DEFAULT] might be
+relevant for a board driver developer.
+
+.. kernel-doc:: drivers/mtd/nand/nand_base.c
+ :internal:
+
+.. kernel-doc:: drivers/mtd/nand/nand_bbt.c
+ :internal:
+
+Credits
+=======
+
+The following people have contributed to the NAND driver:
+
+1. Steven J. Hill\ sjhill@realitydiluted.com
+
+2. David Woodhouse\ dwmw2@infradead.org
+
+3. Thomas Gleixner\ tglx@linutronix.de
+
+A lot of users have provided bugfixes, improvements and helping hands
+for testing. Thanks a lot.
+
+The following people have contributed to this document:
+
+1. Thomas Gleixner\ tglx@linutronix.de
--- /dev/null
+=======================
+RapidIO Subsystem Guide
+=======================
+
+:Author: Matt Porter
+
+Introduction
+============
+
+RapidIO is a high speed switched fabric interconnect with features aimed
+at the embedded market. RapidIO provides support for memory-mapped I/O
+as well as message-based transactions over the switched fabric network.
+RapidIO has a standardized discovery mechanism not unlike the PCI bus
+standard that allows simple detection of devices in a network.
+
+This documentation is provided for developers intending to support
+RapidIO on new architectures, write new drivers, or to understand the
+subsystem internals.
+
+Known Bugs and Limitations
+==========================
+
+Bugs
+----
+
+None. ;)
+
+Limitations
+-----------
+
+1. Access/management of RapidIO memory regions is not supported
+
+2. Multiple host enumeration is not supported
+
+RapidIO driver interface
+========================
+
+Drivers are provided a set of calls in order to interface with the
+subsystem to gather info on devices, request/map memory region
+resources, and manage mailboxes/doorbells.
+
+Functions
+---------
+
+.. kernel-doc:: include/linux/rio_drv.h
+ :internal:
+
+.. kernel-doc:: drivers/rapidio/rio-driver.c
+ :export:
+
+.. kernel-doc:: drivers/rapidio/rio.c
+ :export:
+
+Internals
+=========
+
+This chapter contains the autogenerated documentation of the RapidIO
+subsystem.
+
+Structures
+----------
+
+.. kernel-doc:: include/linux/rio.h
+ :internal:
+
+Enumeration and Discovery
+-------------------------
+
+.. kernel-doc:: drivers/rapidio/rio-scan.c
+ :internal:
+
+Driver functionality
+--------------------
+
+.. kernel-doc:: drivers/rapidio/rio.c
+ :internal:
+
+.. kernel-doc:: drivers/rapidio/rio-access.c
+ :internal:
+
+Device model support
+--------------------
+
+.. kernel-doc:: drivers/rapidio/rio-driver.c
+ :internal:
+
+PPC32 support
+-------------
+
+.. kernel-doc:: arch/powerpc/sysdev/fsl_rio.c
+ :internal:
+
+Credits
+=======
+
+The following people have contributed to the RapidIO subsystem directly
+or indirectly:
+
+1. Matt Porter\ mporter@kernel.crashing.org
+
+2. Randy Vinson\ rvinson@mvista.com
+
+3. Dan Malek\ dan@embeddedalley.com
+
+The following people have contributed to this document:
+
+1. Matt Porter\ mporter@kernel.crashing.org
--- /dev/null
+===================================
+Writing s390 channel device drivers
+===================================
+
+:Author: Cornelia Huck
+
+Introduction
+============
+
+This document describes the interfaces available for device drivers that
+drive s390 based channel attached I/O devices. This includes interfaces
+for interaction with the hardware and interfaces for interacting with
+the common driver core. Those interfaces are provided by the s390 common
+I/O layer.
+
+The document assumes a familarity with the technical terms associated
+with the s390 channel I/O architecture. For a description of this
+architecture, please refer to the "z/Architecture: Principles of
+Operation", IBM publication no. SA22-7832.
+
+While most I/O devices on a s390 system are typically driven through the
+channel I/O mechanism described here, there are various other methods
+(like the diag interface). These are out of the scope of this document.
+
+Some additional information can also be found in the kernel source under
+Documentation/s390/driver-model.txt.
+
+The ccw bus
+===========
+
+The ccw bus typically contains the majority of devices available to a
+s390 system. Named after the channel command word (ccw), the basic
+command structure used to address its devices, the ccw bus contains
+so-called channel attached devices. They are addressed via I/O
+subchannels, visible on the css bus. A device driver for
+channel-attached devices, however, will never interact with the
+subchannel directly, but only via the I/O device on the ccw bus, the ccw
+device.
+
+I/O functions for channel-attached devices
+------------------------------------------
+
+Some hardware structures have been translated into C structures for use
+by the common I/O layer and device drivers. For more information on the
+hardware structures represented here, please consult the Principles of
+Operation.
+
+.. kernel-doc:: arch/s390/include/asm/cio.h
+ :internal:
+
+ccw devices
+-----------
+
+Devices that want to initiate channel I/O need to attach to the ccw bus.
+Interaction with the driver core is done via the common I/O layer, which
+provides the abstractions of ccw devices and ccw device drivers.
+
+The functions that initiate or terminate channel I/O all act upon a ccw
+device structure. Device drivers must not bypass those functions or
+strange side effects may happen.
+
+.. kernel-doc:: arch/s390/include/asm/ccwdev.h
+ :internal:
+
+.. kernel-doc:: drivers/s390/cio/device.c
+ :export:
+
+.. kernel-doc:: drivers/s390/cio/device_ops.c
+ :export:
+
+The channel-measurement facility
+--------------------------------
+
+The channel-measurement facility provides a means to collect measurement
+data which is made available by the channel subsystem for each channel
+attached device.
+
+.. kernel-doc:: arch/s390/include/asm/cmb.h
+ :internal:
+
+.. kernel-doc:: drivers/s390/cio/cmf.c
+ :export:
+
+The ccwgroup bus
+================
+
+The ccwgroup bus only contains artificial devices, created by the user.
+Many networking devices (e.g. qeth) are in fact composed of several ccw
+devices (like read, write and data channel for qeth). The ccwgroup bus
+provides a mechanism to create a meta-device which contains those ccw
+devices as slave devices and can be associated with the netdevice.
+
+ccw group devices
+-----------------
+
+.. kernel-doc:: arch/s390/include/asm/ccwgroup.h
+ :internal:
+
+.. kernel-doc:: drivers/s390/cio/ccwgroup.c
+ :export:
+
+Generic interfaces
+==================
+
+Some interfaces are available to other drivers that do not necessarily
+have anything to do with the busses described above, but still are
+indirectly using basic infrastructure in the common I/O layer. One
+example is the support for adapter interrupts.
+
+.. kernel-doc:: drivers/s390/cio/airq.c
+ :export:
--- /dev/null
+=====================
+SCSI Interfaces Guide
+=====================
+
+:Author: James Bottomley
+:Author: Rob Landley
+
+Introduction
+============
+
+Protocol vs bus
+---------------
+
+Once upon a time, the Small Computer Systems Interface defined both a
+parallel I/O bus and a data protocol to connect a wide variety of
+peripherals (disk drives, tape drives, modems, printers, scanners,
+optical drives, test equipment, and medical devices) to a host computer.
+
+Although the old parallel (fast/wide/ultra) SCSI bus has largely fallen
+out of use, the SCSI command set is more widely used than ever to
+communicate with devices over a number of different busses.
+
+The `SCSI protocol <http://www.t10.org/scsi-3.htm>`__ is a big-endian
+peer-to-peer packet based protocol. SCSI commands are 6, 10, 12, or 16
+bytes long, often followed by an associated data payload.
+
+SCSI commands can be transported over just about any kind of bus, and
+are the default protocol for storage devices attached to USB, SATA, SAS,
+Fibre Channel, FireWire, and ATAPI devices. SCSI packets are also
+commonly exchanged over Infiniband,
+`I20 <http://i2o.shadowconnect.com/faq.php>`__, TCP/IP
+(`iSCSI <https://en.wikipedia.org/wiki/ISCSI>`__), even `Parallel
+ports <http://cyberelk.net/tim/parport/parscsi.html>`__.
+
+Design of the Linux SCSI subsystem
+----------------------------------
+
+The SCSI subsystem uses a three layer design, with upper, mid, and low
+layers. Every operation involving the SCSI subsystem (such as reading a
+sector from a disk) uses one driver at each of the 3 levels: one upper
+layer driver, one lower layer driver, and the SCSI midlayer.
+
+The SCSI upper layer provides the interface between userspace and the
+kernel, in the form of block and char device nodes for I/O and ioctl().
+The SCSI lower layer contains drivers for specific hardware devices.
+
+In between is the SCSI mid-layer, analogous to a network routing layer
+such as the IPv4 stack. The SCSI mid-layer routes a packet based data
+protocol between the upper layer's /dev nodes and the corresponding
+devices in the lower layer. It manages command queues, provides error
+handling and power management functions, and responds to ioctl()
+requests.
+
+SCSI upper layer
+================
+
+The upper layer supports the user-kernel interface by providing device
+nodes.
+
+sd (SCSI Disk)
+--------------
+
+sd (sd_mod.o)
+
+sr (SCSI CD-ROM)
+----------------
+
+sr (sr_mod.o)
+
+st (SCSI Tape)
+--------------
+
+st (st.o)
+
+sg (SCSI Generic)
+-----------------
+
+sg (sg.o)
+
+ch (SCSI Media Changer)
+-----------------------
+
+ch (ch.c)
+
+SCSI mid layer
+==============
+
+SCSI midlayer implementation
+----------------------------
+
+include/scsi/scsi_device.h
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. kernel-doc:: include/scsi/scsi_device.h
+ :internal:
+
+drivers/scsi/scsi.c
+~~~~~~~~~~~~~~~~~~~
+
+Main file for the SCSI midlayer.
+
+.. kernel-doc:: drivers/scsi/scsi.c
+ :export:
+
+drivers/scsi/scsicam.c
+~~~~~~~~~~~~~~~~~~~~~~
+
+`SCSI Common Access
+Method <http://www.t10.org/ftp/t10/drafts/cam/cam-r12b.pdf>`__ support
+functions, for use with HDIO_GETGEO, etc.
+
+.. kernel-doc:: drivers/scsi/scsicam.c
+ :export:
+
+drivers/scsi/scsi_error.c
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Common SCSI error/timeout handling routines.
+
+.. kernel-doc:: drivers/scsi/scsi_error.c
+ :export:
+
+drivers/scsi/scsi_devinfo.c
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Manage scsi_dev_info_list, which tracks blacklisted and whitelisted
+devices.
+
+.. kernel-doc:: drivers/scsi/scsi_devinfo.c
+ :internal:
+
+drivers/scsi/scsi_ioctl.c
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Handle ioctl() calls for SCSI devices.
+
+.. kernel-doc:: drivers/scsi/scsi_ioctl.c
+ :export:
+
+drivers/scsi/scsi_lib.c
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+SCSI queuing library.
+
+.. kernel-doc:: drivers/scsi/scsi_lib.c
+ :export:
+
+drivers/scsi/scsi_lib_dma.c
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+SCSI library functions depending on DMA (map and unmap scatter-gather
+lists).
+
+.. kernel-doc:: drivers/scsi/scsi_lib_dma.c
+ :export:
+
+drivers/scsi/scsi_module.c
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The file drivers/scsi/scsi_module.c contains legacy support for
+old-style host templates. It should never be used by any new driver.
+
+drivers/scsi/scsi_proc.c
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The functions in this file provide an interface between the PROC file
+system and the SCSI device drivers It is mainly used for debugging,
+statistics and to pass information directly to the lowlevel driver. I.E.
+plumbing to manage /proc/scsi/\*
+
+.. kernel-doc:: drivers/scsi/scsi_proc.c
+ :internal:
+
+drivers/scsi/scsi_netlink.c
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Infrastructure to provide async events from transports to userspace via
+netlink, using a single NETLINK_SCSITRANSPORT protocol for all
+transports. See `the original patch
+submission <http://marc.info/?l=linux-scsi&m=115507374832500&w=2>`__ for
+more details.
+
+.. kernel-doc:: drivers/scsi/scsi_netlink.c
+ :internal:
+
+drivers/scsi/scsi_scan.c
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Scan a host to determine which (if any) devices are attached. The
+general scanning/probing algorithm is as follows, exceptions are made to
+it depending on device specific flags, compilation options, and global
+variable (boot or module load time) settings. A specific LUN is scanned
+via an INQUIRY command; if the LUN has a device attached, a scsi_device
+is allocated and setup for it. For every id of every channel on the
+given host, start by scanning LUN 0. Skip hosts that don't respond at
+all to a scan of LUN 0. Otherwise, if LUN 0 has a device attached,
+allocate and setup a scsi_device for it. If target is SCSI-3 or up,
+issue a REPORT LUN, and scan all of the LUNs returned by the REPORT LUN;
+else, sequentially scan LUNs up until some maximum is reached, or a LUN
+is seen that cannot have a device attached to it.
+
+.. kernel-doc:: drivers/scsi/scsi_scan.c
+ :internal:
+
+drivers/scsi/scsi_sysctl.c
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Set up the sysctl entry: "/dev/scsi/logging_level"
+(DEV_SCSI_LOGGING_LEVEL) which sets/returns scsi_logging_level.
+
+drivers/scsi/scsi_sysfs.c
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+SCSI sysfs interface routines.
+
+.. kernel-doc:: drivers/scsi/scsi_sysfs.c
+ :export:
+
+drivers/scsi/hosts.c
+~~~~~~~~~~~~~~~~~~~~
+
+mid to lowlevel SCSI driver interface
+
+.. kernel-doc:: drivers/scsi/hosts.c
+ :export:
+
+drivers/scsi/constants.c
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+mid to lowlevel SCSI driver interface
+
+.. kernel-doc:: drivers/scsi/constants.c
+ :export:
+
+Transport classes
+-----------------
+
+Transport classes are service libraries for drivers in the SCSI lower
+layer, which expose transport attributes in sysfs.
+
+Fibre Channel transport
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The file drivers/scsi/scsi_transport_fc.c defines transport attributes
+for Fibre Channel.
+
+.. kernel-doc:: drivers/scsi/scsi_transport_fc.c
+ :export:
+
+iSCSI transport class
+~~~~~~~~~~~~~~~~~~~~~
+
+The file drivers/scsi/scsi_transport_iscsi.c defines transport
+attributes for the iSCSI class, which sends SCSI packets over TCP/IP
+connections.
+
+.. kernel-doc:: drivers/scsi/scsi_transport_iscsi.c
+ :export:
+
+Serial Attached SCSI (SAS) transport class
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The file drivers/scsi/scsi_transport_sas.c defines transport
+attributes for Serial Attached SCSI, a variant of SATA aimed at large
+high-end systems.
+
+The SAS transport class contains common code to deal with SAS HBAs, an
+aproximated representation of SAS topologies in the driver model, and
+various sysfs attributes to expose these topologies and management
+interfaces to userspace.
+
+In addition to the basic SCSI core objects this transport class
+introduces two additional intermediate objects: The SAS PHY as
+represented by struct sas_phy defines an "outgoing" PHY on a SAS HBA or
+Expander, and the SAS remote PHY represented by struct sas_rphy defines
+an "incoming" PHY on a SAS Expander or end device. Note that this is
+purely a software concept, the underlying hardware for a PHY and a
+remote PHY is the exactly the same.
+
+There is no concept of a SAS port in this code, users can see what PHYs
+form a wide port based on the port_identifier attribute, which is the
+same for all PHYs in a port.
+
+.. kernel-doc:: drivers/scsi/scsi_transport_sas.c
+ :export:
+
+SATA transport class
+~~~~~~~~~~~~~~~~~~~~
+
+The SATA transport is handled by libata, which has its own book of
+documentation in this directory.
+
+Parallel SCSI (SPI) transport class
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The file drivers/scsi/scsi_transport_spi.c defines transport
+attributes for traditional (fast/wide/ultra) SCSI busses.
+
+.. kernel-doc:: drivers/scsi/scsi_transport_spi.c
+ :export:
+
+SCSI RDMA (SRP) transport class
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The file drivers/scsi/scsi_transport_srp.c defines transport
+attributes for SCSI over Remote Direct Memory Access.
+
+.. kernel-doc:: drivers/scsi/scsi_transport_srp.c
+ :export:
+
+SCSI lower layer
+================
+
+Host Bus Adapter transport types
+--------------------------------
+
+Many modern device controllers use the SCSI command set as a protocol to
+communicate with their devices through many different types of physical
+connections.
+
+In SCSI language a bus capable of carrying SCSI commands is called a
+"transport", and a controller connecting to such a bus is called a "host
+bus adapter" (HBA).
+
+Debug transport
+~~~~~~~~~~~~~~~
+
+The file drivers/scsi/scsi_debug.c simulates a host adapter with a
+variable number of disks (or disk like devices) attached, sharing a
+common amount of RAM. Does a lot of checking to make sure that we are
+not getting blocks mixed up, and panics the kernel if anything out of
+the ordinary is seen.
+
+To be more realistic, the simulated devices have the transport
+attributes of SAS disks.
+
+For documentation see http://sg.danny.cz/sg/sdebug26.html
+
+todo
+~~~~
+
+Parallel (fast/wide/ultra) SCSI, USB, SATA, SAS, Fibre Channel,
+FireWire, ATAPI devices, Infiniband, I20, iSCSI, Parallel ports,
+netlink...
--- /dev/null
+===============================================================
+Synopsys DesignWare Core SuperSpeed USB 3.0 Controller
+===============================================================
+
+:Author: Felipe Balbi <felipe.balbi@linux.intel.com>
+:Date: April 2017
+
+Introduction
+============
+
+The *Synopsys DesignWare Core SuperSpeed USB 3.0 Controller*
+(hereinafter referred to as *DWC3*) is a USB SuperSpeed compliant
+controller which can be configured in one of 4 ways:
+
+ 1. Peripheral-only configuration
+ 2. Host-only configuration
+ 3. Dual-Role configuration
+ 4. Hub configuration
+
+Linux currently supports several versions of this controller. In all
+likelyhood, the version in your SoC is already supported. At the time
+of this writing, known tested versions range from 2.02a to 3.10a. As a
+rule of thumb, anything above 2.02a should work reliably well.
+
+Currently, we have many known users for this driver. In alphabetical
+order:
+
+ 1. Cavium
+ 2. Intel Corporation
+ 3. Qualcomm
+ 4. Rockchip
+ 5. ST
+ 6. Samsung
+ 7. Texas Instruments
+ 8. Xilinx
+
+Summary of Features
+======================
+
+For details about features supported by your version of DWC3, consult
+your IP team and/or *Synopsys DesignWare Core SuperSpeed USB 3.0
+Controller Databook*. Following is a list of features supported by the
+driver at the time of this writing:
+
+ 1. Up to 16 bidirectional endpoints (including the control
+ pipe - ep0)
+ 2. Flexible endpoint configuration
+ 3. Simultaneous IN and OUT transfer support
+ 4. Scatter-list support
+ 5. Up to 256 TRBs [#trb]_ per endpoint
+ 6. Support for all transfer types (*Control*, *Bulk*,
+ *Interrupt*, and *Isochronous*)
+ 7. SuperSpeed Bulk Streams
+ 8. Link Power Management
+ 9. Trace Events for debugging
+ 10. DebugFS [#debugfs]_ interface
+
+These features have all been exercised with many of the **in-tree**
+gadget drivers. We have verified both *ConfigFS* [#configfs]_ and
+legacy gadget drivers.
+
+Driver Design
+==============
+
+The DWC3 driver sits on the *drivers/usb/dwc3/* directory. All files
+related to this driver are in this one directory. This makes it easy
+for new-comers to read the code and understand how it behaves.
+
+Because of DWC3's configuration flexibility, the driver is a little
+complex in some places but it should be rather straightforward to
+understand.
+
+The biggest part of the driver refers to the Gadget API.
+
+Known Limitations
+===================
+
+Like any other HW, DWC3 has its own set of limitations. To avoid
+constant questions about such problems, we decided to document them
+here and have a single location to where we could point users.
+
+OUT Transfer Size Requirements
+---------------------------------
+
+According to Synopsys Databook, all OUT transfer TRBs [#trb]_ must
+have their *size* field set to a value which is integer divisible by
+the endpoint's *wMaxPacketSize*. This means that *e.g.* in order to
+receive a Mass Storage *CBW* [#cbw]_, req->length must either be set
+to a value that's divisible by *wMaxPacketSize* (1024 on SuperSpeed,
+512 on HighSpeed, etc), or DWC3 driver must add a Chained TRB pointing
+to a throw-away buffer for the remaining length. Without this, OUT
+transfers will **NOT** start.
+
+Note that as of this writing, this won't be a problem because DWC3 is
+fully capable of appending a chained TRB for the remaining length and
+completely hide this detail from the gadget driver. It's still worth
+mentioning because this seems to be the largest source of queries
+about DWC3 and *non-working transfers*.
+
+TRB Ring Size Limitation
+-------------------------
+
+We, currently, have a hard limit of 256 TRBs [#trb]_ per endpoint,
+with the last TRB being a Link TRB [#link_trb]_ pointing back to the
+first. This limit is arbitrary but it has the benefit of adding up to
+exactly 4096 bytes, or 1 Page.
+
+DWC3 driver will try its best to cope with more than 255 requests and,
+for the most part, it should work normally. However this is not
+something that has been exercised very frequently. If you experience
+any problems, see section **Reporting Bugs** below.
+
+Reporting Bugs
+================
+
+Whenever you encounter a problem with DWC3, first and foremost you
+should make sure that:
+
+ 1. You're running latest tag from `Linus' tree`_
+ 2. You can reproduce the error without any out-of-tree changes
+ to DWC3
+ 3. You have checked that it's not a fault on the host machine
+
+After all these are verified, then here's how to capture enough
+information so we can be of any help to you.
+
+Required Information
+---------------------
+
+DWC3 relies exclusively on Trace Events for debugging. Everything is
+exposed there, with some extra bits being exposed to DebugFS
+[#debugfs]_.
+
+In order to capture DWC3's Trace Events you should run the following
+commands **before** plugging the USB cable to a host machine:
+
+.. code-block:: sh
+
+ # mkdir -p /d
+ # mkdir -p /t
+ # mount -t debugfs none /d
+ # mount -t tracefs none /t
+ # echo 81920 > /t/buffer_size_kb
+ # echo 1 > /t/events/dwc3/enable
+
+After this is done, you can connect your USB cable and reproduce the
+problem. As soon as the fault is reproduced, make a copy of files
+``trace`` and ``regdump``, like so:
+
+.. code-block:: sh
+
+ # cp /t/trace /root/trace.txt
+ # cat /d/*dwc3*/regdump > /root/regdump.txt
+
+Make sure to compress ``trace.txt`` and ``regdump.txt`` in a tarball
+and email it to `me`_ with `linux-usb`_ in Cc. If you want to be extra
+sure that I'll help you, write your subject line in the following
+format:
+
+ **[BUG REPORT] usb: dwc3: Bug while doing XYZ**
+
+On the email body, make sure to detail what you doing, which gadget
+driver you were using, how to reproduce the problem, what SoC you're
+using, which OS (and its version) was running on the Host machine.
+
+With all this information, we should be able to understand what's
+going on and be helpful to you.
+
+Debugging
+===========
+
+First and foremost a disclaimer::
+
+ DISCLAIMER: The information available on DebugFS and/or TraceFS can
+ change at any time at any Major Linux Kernel Release. If writing
+ scripts, do **NOT** assume information to be available in the
+ current format.
+
+With that out of the way, let's carry on.
+
+If you're willing to debug your own problem, you deserve a round of
+applause :-)
+
+Anyway, there isn't much to say here other than Trace Events will be
+really helpful in figuring out issues with DWC3. Also, access to
+Synopsys Databook will be **really** valuable in this case.
+
+A USB Sniffer can be helpful at times but it's not entirely required,
+there's a lot that can be understood without looking at the wire.
+
+Feel free to email `me`_ and Cc `linux-usb`_ if you need any help.
+
+``DebugFS``
+-------------
+
+``DebugFS`` is very good for gathering snapshots of what's going on
+with DWC3 and/or any endpoint.
+
+On DWC3's ``DebugFS`` directory, you will find the following files and
+directories:
+
+``ep[0..15]{in,out}/``
+``link_state``
+``regdump``
+``testmode``
+
+``link_state``
+``````````````
+
+When read, ``link_state`` will print out one of ``U0``, ``U1``,
+``U2``, ``U3``, ``SS.Disabled``, ``RX.Detect``, ``SS.Inactive``,
+``Polling``, ``Recovery``, ``Hot Reset``, ``Compliance``,
+``Loopback``, ``Reset``, ``Resume`` or ``UNKNOWN link state``.
+
+This file can also be written to in order to force link to one of the
+states above.
+
+``regdump``
+`````````````
+
+File name is self-explanatory. When read, ``regdump`` will print out a
+register dump of DWC3. Note that this file can be grepped to find the
+information you want.
+
+``testmode``
+``````````````
+
+When read, ``testmode`` will print out a name of one of the specified
+USB 2.0 Testmodes (``test_j``, ``test_k``, ``test_se0_nak``,
+``test_packet``, ``test_force_enable``) or the string ``no test`` in
+case no tests are currently being executed.
+
+In order to start any of these test modes, the same strings can be
+written to the file and DWC3 will enter the requested test mode.
+
+
+``ep[0..15]{in,out}``
+``````````````````````
+
+For each endpoint we expose one directory following the naming
+convention ``ep$num$dir`` *(ep0in, ep0out, ep1in, ...)*. Inside each
+of these directories you will find the following files:
+
+``descriptor_fetch_queue``
+``event_queue``
+``rx_fifo_queue``
+``rx_info_queue``
+``rx_request_queue``
+``transfer_type``
+``trb_ring``
+``tx_fifo_queue``
+``tx_request_queue``
+
+With access to Synopsys Databook, you can decode the information on
+them.
+
+``transfer_type``
+~~~~~~~~~~~~~~~~~~
+
+When read, ``transfer_type`` will print out one of ``control``,
+``bulk``, ``interrupt`` or ``isochronous`` depending on what the
+endpoint descriptor says. If the endpoint hasn't been enabled yet, it
+will print ``--``.
+
+``trb_ring``
+~~~~~~~~~~~~~
+
+When read, ``trb_ring`` will print out details about all TRBs on the
+ring. It will also tell you where our enqueue and dequeue pointers are
+located in the ring:
+
+.. code-block:: sh
+
+ buffer_addr,size,type,ioc,isp_imi,csp,chn,lst,hwo
+ 000000002c754000,481,normal,1,0,1,0,0,0
+ 000000002c75c000,481,normal,1,0,1,0,0,0
+ 000000002c780000,481,normal,1,0,1,0,0,0
+ 000000002c788000,481,normal,1,0,1,0,0,0
+ 000000002c78c000,481,normal,1,0,1,0,0,0
+ 000000002c754000,481,normal,1,0,1,0,0,0
+ 000000002c75c000,481,normal,1,0,1,0,0,0
+ 000000002c784000,481,normal,1,0,1,0,0,0
+ 000000002c788000,481,normal,1,0,1,0,0,0
+ 000000002c78c000,481,normal,1,0,1,0,0,0
+ 000000002c790000,481,normal,1,0,1,0,0,0
+ 000000002c758000,481,normal,1,0,1,0,0,0
+ 000000002c780000,481,normal,1,0,1,0,0,0
+ 000000002c788000,481,normal,1,0,1,0,0,0
+ 000000002c790000,481,normal,1,0,1,0,0,0
+ 000000002c758000,481,normal,1,0,1,0,0,0
+ 000000002c780000,481,normal,1,0,1,0,0,0
+ 000000002c784000,481,normal,1,0,1,0,0,0
+ 000000002c788000,481,normal,1,0,1,0,0,0
+ 000000002c78c000,481,normal,1,0,1,0,0,0
+ 000000002c754000,481,normal,1,0,1,0,0,0
+ 000000002c758000,481,normal,1,0,1,0,0,0
+ 000000002c780000,481,normal,1,0,1,0,0,0
+ 000000002c784000,481,normal,1,0,1,0,0,0
+ 000000002c78c000,481,normal,1,0,1,0,0,0
+ 000000002c790000,481,normal,1,0,1,0,0,0
+ 000000002c758000,481,normal,1,0,1,0,0,0
+ 000000002c780000,481,normal,1,0,1,0,0,0
+ 000000002c788000,481,normal,1,0,1,0,0,0
+ 000000002c790000,481,normal,1,0,1,0,0,0
+ 000000002c758000,481,normal,1,0,1,0,0,0
+ 000000002c780000,481,normal,1,0,1,0,0,0
+ 000000002c788000,481,normal,1,0,1,0,0,0
+ 000000002c790000,481,normal,1,0,1,0,0,0
+ 000000002c758000,481,normal,1,0,1,0,0,0
+ 000000002c780000,481,normal,1,0,1,0,0,0
+ 000000002c788000,481,normal,1,0,1,0,0,0
+ 000000002c790000,481,normal,1,0,1,0,0,0
+ 000000002c758000,481,normal,1,0,1,0,0,0
+ 000000002c780000,481,normal,1,0,1,0,0,0
+ 000000002c788000,481,normal,1,0,1,0,0,0
+ 000000002c790000,481,normal,1,0,1,0,0,0
+ 000000002c758000,481,normal,1,0,1,0,0,0
+ 000000002c780000,481,normal,1,0,1,0,0,0
+ 000000002c788000,481,normal,1,0,1,0,0,0
+ 000000002c790000,481,normal,1,0,1,0,0,0
+ 000000002c758000,481,normal,1,0,1,0,0,0
+ 000000002c780000,481,normal,1,0,1,0,0,0
+ 000000002c788000,481,normal,1,0,1,0,0,0
+ 000000002c790000,481,normal,1,0,1,0,0,0
+ 000000002c758000,481,normal,1,0,1,0,0,0
+ 000000002c780000,481,normal,1,0,1,0,0,0
+ 000000002c788000,481,normal,1,0,1,0,0,0
+ 000000002c790000,481,normal,1,0,1,0,0,0
+ 000000002c758000,481,normal,1,0,1,0,0,0
+ 000000002c780000,481,normal,1,0,1,0,0,0
+ 000000002c78c000,481,normal,1,0,1,0,0,0
+ 000000002c784000,481,normal,1,0,1,0,0,0
+ 000000002c788000,481,normal,1,0,1,0,0,0
+ 000000002c78c000,481,normal,1,0,1,0,0,0
+ 000000002c754000,481,normal,1,0,1,0,0,0
+ 000000002c758000,481,normal,1,0,1,0,0,0
+ 000000002c780000,481,normal,1,0,1,0,0,0
+ 000000002c788000,481,normal,1,0,1,0,0,0
+ 000000002c790000,481,normal,1,0,1,0,0,0
+ 000000002c758000,481,normal,1,0,1,0,0,0
+ 000000002c780000,481,normal,1,0,1,0,0,0
+ 000000002c758000,481,normal,1,0,1,0,0,0
+ 000000002c780000,481,normal,1,0,1,0,0,0
+ 000000002c78c000,481,normal,1,0,1,0,0,0
+ 000000002c75c000,481,normal,1,0,1,0,0,0
+ 000000002c78c000,481,normal,1,0,1,0,0,0
+ 000000002c780000,481,normal,1,0,1,0,0,0
+ 000000002c754000,481,normal,1,0,1,0,0,0
+ 000000002c788000,481,normal,1,0,1,0,0,0
+ 000000002c754000,481,normal,1,0,1,0,0,0
+ 000000002c780000,481,normal,1,0,1,0,0,0
+ 000000002c788000,481,normal,1,0,1,0,0,0
+ 000000002c78c000,481,normal,1,0,1,0,0,0
+ 000000002c790000,481,normal,1,0,1,0,0,0
+ 000000002c754000,481,normal,1,0,1,0,0,0
+ 000000002c758000,481,normal,1,0,1,0,0,0
+ 000000002c75c000,481,normal,1,0,1,0,0,0
+ 000000002c780000,481,normal,1,0,1,0,0,0
+ 000000002c784000,481,normal,1,0,1,0,0,0
+ 000000002c788000,481,normal,1,0,1,0,0,0
+ 000000002c78c000,481,normal,1,0,1,0,0,0
+ 000000002c790000,481,normal,1,0,1,0,0,0
+ 000000002c754000,481,normal,1,0,1,0,0,0
+ 000000002c758000,481,normal,1,0,1,0,0,0
+ 000000002c75c000,512,normal,1,0,1,0,0,1 D
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0 E
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 0000000000000000,0,UNKNOWN,0,0,0,0,0,0
+ 00000000381ab000,0,link,0,0,0,0,0,1
+
+
+Trace Events
+-------------
+
+DWC3 also provides several trace events which help us gathering
+information about the behavior of the driver during runtime.
+
+In order to use these events, you must enable ``CONFIG_FTRACE`` in
+your kernel config.
+
+For details about how enable DWC3 events, see section **Reporting
+Bugs**.
+
+The following subsections will give details about each Event Class and
+each Event defined by DWC3.
+
+MMIO
+```````
+
+It is sometimes useful to look at every MMIO access when looking for
+bugs. Because of that, DWC3 offers two Trace Events (one for
+dwc3_readl() and one for dwc3_writel()). ``TP_printk`` follows::
+
+ TP_printk("addr %p value %08x", __entry->base + __entry->offset,
+ __entry->value)
+
+Interrupt Events
+````````````````
+
+Every IRQ event can be logged and decoded into a human readable
+string. Because every event will be different, we don't give an
+example other than the ``TP_printk`` format used::
+
+ TP_printk("event (%08x): %s", __entry->event,
+ dwc3_decode_event(__entry->event, __entry->ep0state))
+
+Control Request
+`````````````````
+
+Every USB Control Request can be logged to the trace buffer. The
+output format is::
+
+ TP_printk("%s", dwc3_decode_ctrl(__entry->bRequestType,
+ __entry->bRequest, __entry->wValue,
+ __entry->wIndex, __entry->wLength)
+ )
+
+Note that Standard Control Requests will be decoded into
+human-readable strings with their respective arguments. Class and
+Vendor requests will be printed out a sequence of 8 bytes in hex
+format.
+
+Lifetime of a ``struct usb_request``
+```````````````````````````````````````
+
+The entire lifetime of a ``struct usb_request`` can be tracked on the
+trace buffer. We have one event for each of allocation, free,
+queueing, dequeueing, and giveback. Output format is::
+
+ TP_printk("%s: req %p length %u/%u %s%s%s ==> %d",
+ __get_str(name), __entry->req, __entry->actual, __entry->length,
+ __entry->zero ? "Z" : "z",
+ __entry->short_not_ok ? "S" : "s",
+ __entry->no_interrupt ? "i" : "I",
+ __entry->status
+ )
+
+Generic Commands
+````````````````````
+
+We can log and decode every Generic Command with its completion
+code. Format is::
+
+ TP_printk("cmd '%s' [%x] param %08x --> status: %s",
+ dwc3_gadget_generic_cmd_string(__entry->cmd),
+ __entry->cmd, __entry->param,
+ dwc3_gadget_generic_cmd_status_string(__entry->status)
+ )
+
+Endpoint Commands
+````````````````````
+
+Endpoints commands can also be logged together with completion
+code. Format is::
+
+ TP_printk("%s: cmd '%s' [%d] params %08x %08x %08x --> status: %s",
+ __get_str(name), dwc3_gadget_ep_cmd_string(__entry->cmd),
+ __entry->cmd, __entry->param0,
+ __entry->param1, __entry->param2,
+ dwc3_ep_cmd_status_string(__entry->cmd_status)
+ )
+
+Lifetime of a ``TRB``
+``````````````````````
+
+A ``TRB`` Lifetime is simple. We are either preparing a ``TRB`` or
+completing it. With these two events, we can see how a ``TRB`` changes
+over time. Format is::
+
+ TP_printk("%s: %d/%d trb %p buf %08x%08x size %s%d ctrl %08x (%c%c%c%c:%c%c:%s)",
+ __get_str(name), __entry->queued, __entry->allocated,
+ __entry->trb, __entry->bph, __entry->bpl,
+ ({char *s;
+ int pcm = ((__entry->size >> 24) & 3) + 1;
+ switch (__entry->type) {
+ case USB_ENDPOINT_XFER_INT:
+ case USB_ENDPOINT_XFER_ISOC:
+ switch (pcm) {
+ case 1:
+ s = "1x ";
+ break;
+ case 2:
+ s = "2x ";
+ break;
+ case 3:
+ s = "3x ";
+ break;
+ }
+ default:
+ s = "";
+ } s; }),
+ DWC3_TRB_SIZE_LENGTH(__entry->size), __entry->ctrl,
+ __entry->ctrl & DWC3_TRB_CTRL_HWO ? 'H' : 'h',
+ __entry->ctrl & DWC3_TRB_CTRL_LST ? 'L' : 'l',
+ __entry->ctrl & DWC3_TRB_CTRL_CHN ? 'C' : 'c',
+ __entry->ctrl & DWC3_TRB_CTRL_CSP ? 'S' : 's',
+ __entry->ctrl & DWC3_TRB_CTRL_ISP_IMI ? 'S' : 's',
+ __entry->ctrl & DWC3_TRB_CTRL_IOC ? 'C' : 'c',
+ dwc3_trb_type_string(DWC3_TRBCTL_TYPE(__entry->ctrl))
+ )
+
+Lifetime of an Endpoint
+```````````````````````
+
+And endpoint's lifetime is summarized with enable and disable
+operations, both of which can be traced. Format is::
+
+ TP_printk("%s: mps %d/%d streams %d burst %d ring %d/%d flags %c:%c%c%c%c%c:%c:%c",
+ __get_str(name), __entry->maxpacket,
+ __entry->maxpacket_limit, __entry->max_streams,
+ __entry->maxburst, __entry->trb_enqueue,
+ __entry->trb_dequeue,
+ __entry->flags & DWC3_EP_ENABLED ? 'E' : 'e',
+ __entry->flags & DWC3_EP_STALL ? 'S' : 's',
+ __entry->flags & DWC3_EP_WEDGE ? 'W' : 'w',
+ __entry->flags & DWC3_EP_BUSY ? 'B' : 'b',
+ __entry->flags & DWC3_EP_PENDING_REQUEST ? 'P' : 'p',
+ __entry->flags & DWC3_EP_MISSED_ISOC ? 'M' : 'm',
+ __entry->flags & DWC3_EP_END_TRANSFER_PENDING ? 'E' : 'e',
+ __entry->direction ? '<' : '>'
+ )
+
+
+Structures, Methods and Definitions
+====================================
+
+.. kernel-doc:: drivers/usb/dwc3/core.h
+ :doc: main data structures
+ :internal:
+
+.. kernel-doc:: drivers/usb/dwc3/gadget.h
+ :doc: gadget-only helpers
+ :internal:
+
+.. kernel-doc:: drivers/usb/dwc3/gadget.c
+ :doc: gadget-side implementation
+ :internal:
+
+.. kernel-doc:: drivers/usb/dwc3/core.c
+ :doc: core driver (probe, PM, etc)
+ :internal:
+
+.. [#trb] Transfer Request Block
+.. [#link_trb] Transfer Request Block pointing to another Transfer
+ Request Block.
+.. [#debugfs] The Debug File System
+.. [#configfs] The Config File System
+.. [#cbw] Command Block Wrapper
+.. _Linus' tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/
+.. _me: felipe.balbi@linux.intel.com
+.. _linux-usb: linux-usb@vger.kernel.org
persist
error-codes
writing_usb_driver
+ dwc3
writing_musb_glue_layer
+ typec
+ usb3-debug-port
.. only:: subproject and html
--- /dev/null
+======================
+W1: Dallas' 1-wire bus
+======================
+
+:Author: David Fries
+
+W1 API internal to the kernel
+=============================
+
+W1 API internal to the kernel
+-----------------------------
+
+include/linux/w1.h
+~~~~~~~~~~~~~~~~~~
+
+W1 kernel API functions.
+
+.. kernel-doc:: include/linux/w1.h
+ :internal:
+
+drivers/w1/w1.c
+~~~~~~~~~~~~~~~
+
+W1 core functions.
+
+.. kernel-doc:: drivers/w1/w1.c
+ :internal:
+
+drivers/w1/w1_family.c
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Allows registering device family operations.
+
+.. kernel-doc:: drivers/w1/w1_family.c
+ :export:
+
+drivers/w1/w1_internal.h
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+W1 internal initialization for master devices.
+
+.. kernel-doc:: drivers/w1/w1_internal.h
+ :internal:
+
+drivers/w1/w1_int.c
+~~~~~~~~~~~~~~~~~~~~
+
+W1 internal initialization for master devices.
+
+.. kernel-doc:: drivers/w1/w1_int.c
+ :export:
+
+drivers/w1/w1_netlink.h
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+W1 external netlink API structures and commands.
+
+.. kernel-doc:: drivers/w1/w1_netlink.h
+ :internal:
+
+drivers/w1/w1_io.c
+~~~~~~~~~~~~~~~~~~~
+
+W1 input/output.
+
+.. kernel-doc:: drivers/w1/w1_io.c
+ :export:
+
+.. kernel-doc:: drivers/w1/w1_io.c
+ :internal:
devm_irq_alloc_desc_at()
devm_irq_alloc_desc_from()
devm_irq_alloc_descs_from()
+ devm_irq_alloc_generic_chip()
+ devm_irq_setup_generic_chip()
LED
devm_led_classdev_register()
devm_kzalloc()
MFD
- devm_mfd_add_devices()
+ devm_mfd_add_devices()
+
+MUX
+ devm_mux_chip_alloc()
+ devm_mux_chip_register()
+ devm_mux_control_get()
PER-CPU MEM
devm_alloc_percpu()
FOURCC definitions are located in the linux/videodev2.h header. However, and
despite starting with the V4L2_PIX_FMT_prefix, they are not restricted to V4L2
and don't require usage of the V4L2 subsystem. FOURCC documentation is
-available in Documentation/DocBook/v4l/pixfmt.xml.
+available in Documentation/media/uapi/v4l/pixfmt.rst.
To select a format, applications set the grayscale field to the desired FOURCC.
For YUV formats, they should also select the appropriate colorspace by setting
the colorspace field to one of the colorspaces listed in linux/videodev2.h and
-documented in Documentation/DocBook/v4l/colorspaces.xml.
+documented in Documentation/media/uapi/v4l/colorspaces.rst.
The red, green, blue and transp fields are not used with the FOURCC-based API.
For forward compatibility reasons applications must zero those fields, and
struct autofs_v5_packet {
int proto_version; /* Protocol version */
int type; /* Type of packet */
- autofs_wqt_t wait_queue_token;
+ autofs_wqt_t wait_queue_entry_token;
__u32 dev;
__u64 ino;
__u32 uid;
`O_DIRECT`) to _pipe2(2)_ so that a read from the pipe will return at
most one packet, and any unread portion of a packet will be discarded.
-The `wait_queue_token` is a unique number which can identify a
+The `wait_queue_entry_token` is a unique number which can identify a
particular request to be acknowledged. When a message is sent over
the pipe the affected dentry is marked as either "active" or
"expiring" and other accesses to it block until the message is
acknowledged using one of the ioctls below and the relevant
-`wait_queue_token`.
+`wait_queue_entry_token`.
Communicating with autofs: root directory ioctls
------------------------------------------------
The available ioctl commands are:
- **AUTOFS_IOC_READY**: a notification has been handled. The argument
- to the ioctl command is the "wait_queue_token" number
+ to the ioctl command is the "wait_queue_entry_token" number
corresponding to the notification being acknowledged.
- **AUTOFS_IOC_FAIL**: similar to above, but indicates failure with
the error code `ENOENT`.
struct autofs_packet_expire_multi {
int proto_version; /* Protocol version */
int type; /* Type of packet */
- autofs_wqt_t wait_queue_token;
+ autofs_wqt_t wait_queue_entry_token;
int len;
char name[NAME_MAX+1];
};
is required. This is filled in with the name of something
that can be unmounted or removed. If nothing can be expired,
- `errno` is set to `EAGAIN`. Even though a `wait_queue_token`
+ `errno` is set to `EAGAIN`. Even though a `wait_queue_entry_token`
is present in the structure, no "wait queue" is established
and no acknowledgment is needed.
- **AUTOFS_IOC_EXPIRE_MULTI**: This is similar to
--- /dev/null
+# -*- coding: utf-8; mode: python -*-
+
+project = "Linux Filesystems API"
+
+tags.add("subproject")
+
+latex_documents = [
+ ('index', 'filesystems.tex', project,
+ 'The kernel development community', 'manual'),
+]
--- /dev/null
+=====================
+Linux Filesystems API
+=====================
+
+The Linux VFS
+=============
+
+The Filesystem types
+--------------------
+
+.. kernel-doc:: include/linux/fs.h
+ :internal:
+
+The Directory Cache
+-------------------
+
+.. kernel-doc:: fs/dcache.c
+ :export:
+
+.. kernel-doc:: include/linux/dcache.h
+ :internal:
+
+Inode Handling
+--------------
+
+.. kernel-doc:: fs/inode.c
+ :export:
+
+.. kernel-doc:: fs/bad_inode.c
+ :export:
+
+Registration and Superblocks
+----------------------------
+
+.. kernel-doc:: fs/super.c
+ :export:
+
+File Locks
+----------
+
+.. kernel-doc:: fs/locks.c
+ :export:
+
+.. kernel-doc:: fs/locks.c
+ :internal:
+
+Other Functions
+---------------
+
+.. kernel-doc:: fs/mpage.c
+ :export:
+
+.. kernel-doc:: fs/namei.c
+ :export:
+
+.. kernel-doc:: fs/buffer.c
+ :export:
+
+.. kernel-doc:: block/bio.c
+ :export:
+
+.. kernel-doc:: fs/seq_file.c
+ :export:
+
+.. kernel-doc:: fs/filesystems.c
+ :export:
+
+.. kernel-doc:: fs/fs-writeback.c
+ :export:
+
+.. kernel-doc:: fs/block_dev.c
+ :export:
+
+The proc filesystem
+===================
+
+sysctl interface
+----------------
+
+.. kernel-doc:: kernel/sysctl.c
+ :export:
+
+proc filesystem interface
+-------------------------
+
+.. kernel-doc:: fs/proc/base.c
+ :internal:
+
+Events based on file descriptors
+================================
+
+.. kernel-doc:: fs/eventfd.c
+ :export:
+
+The Filesystem for Exporting Kernel Objects
+===========================================
+
+.. kernel-doc:: fs/sysfs/file.c
+ :export:
+
+.. kernel-doc:: fs/sysfs/symlink.c
+ :export:
+
+The debugfs filesystem
+======================
+
+debugfs interface
+-----------------
+
+.. kernel-doc:: fs/debugfs/inode.c
+ :export:
+
+.. kernel-doc:: fs/debugfs/file.c
+ :export:
+
+The Linux Journalling API
+=========================
+
+Overview
+--------
+
+Details
+~~~~~~~
+
+The journalling layer is easy to use. You need to first of all create a
+journal_t data structure. There are two calls to do this dependent on
+how you decide to allocate the physical media on which the journal
+resides. The :c:func:`jbd2_journal_init_inode` call is for journals stored in
+filesystem inodes, or the :c:func:`jbd2_journal_init_dev` call can be used
+for journal stored on a raw device (in a continuous range of blocks). A
+journal_t is a typedef for a struct pointer, so when you are finally
+finished make sure you call :c:func:`jbd2_journal_destroy` on it to free up
+any used kernel memory.
+
+Once you have got your journal_t object you need to 'mount' or load the
+journal file. The journalling layer expects the space for the journal
+was already allocated and initialized properly by the userspace tools.
+When loading the journal you must call :c:func:`jbd2_journal_load` to process
+journal contents. If the client file system detects the journal contents
+does not need to be processed (or even need not have valid contents), it
+may call :c:func:`jbd2_journal_wipe` to clear the journal contents before
+calling :c:func:`jbd2_journal_load`.
+
+Note that jbd2_journal_wipe(..,0) calls
+:c:func:`jbd2_journal_skip_recovery` for you if it detects any outstanding
+transactions in the journal and similarly :c:func:`jbd2_journal_load` will
+call :c:func:`jbd2_journal_recover` if necessary. I would advise reading
+:c:func:`ext4_load_journal` in fs/ext4/super.c for examples on this stage.
+
+Now you can go ahead and start modifying the underlying filesystem.
+Almost.
+
+You still need to actually journal your filesystem changes, this is done
+by wrapping them into transactions. Additionally you also need to wrap
+the modification of each of the buffers with calls to the journal layer,
+so it knows what the modifications you are actually making are. To do
+this use :c:func:`jbd2_journal_start` which returns a transaction handle.
+
+:c:func:`jbd2_journal_start` and its counterpart :c:func:`jbd2_journal_stop`,
+which indicates the end of a transaction are nestable calls, so you can
+reenter a transaction if necessary, but remember you must call
+:c:func:`jbd2_journal_stop` the same number of times as
+:c:func:`jbd2_journal_start` before the transaction is completed (or more
+accurately leaves the update phase). Ext4/VFS makes use of this feature to
+simplify handling of inode dirtying, quota support, etc.
+
+Inside each transaction you need to wrap the modifications to the
+individual buffers (blocks). Before you start to modify a buffer you
+need to call :c:func:`jbd2_journal_get_create_access()` /
+:c:func:`jbd2_journal_get_write_access()` /
+:c:func:`jbd2_journal_get_undo_access()` as appropriate, this allows the
+journalling layer to copy the unmodified
+data if it needs to. After all the buffer may be part of a previously
+uncommitted transaction. At this point you are at last ready to modify a
+buffer, and once you are have done so you need to call
+:c:func:`jbd2_journal_dirty_metadata`. Or if you've asked for access to a
+buffer you now know is now longer required to be pushed back on the
+device you can call :c:func:`jbd2_journal_forget` in much the same way as you
+might have used :c:func:`bforget` in the past.
+
+A :c:func:`jbd2_journal_flush` may be called at any time to commit and
+checkpoint all your transactions.
+
+Then at umount time , in your :c:func:`put_super` you can then call
+:c:func:`jbd2_journal_destroy` to clean up your in-core journal object.
+
+Unfortunately there a couple of ways the journal layer can cause a
+deadlock. The first thing to note is that each task can only have a
+single outstanding transaction at any one time, remember nothing commits
+until the outermost :c:func:`jbd2_journal_stop`. This means you must complete
+the transaction at the end of each file/inode/address etc. operation you
+perform, so that the journalling system isn't re-entered on another
+journal. Since transactions can't be nested/batched across differing
+journals, and another filesystem other than yours (say ext4) may be
+modified in a later syscall.
+
+The second case to bear in mind is that :c:func:`jbd2_journal_start` can block
+if there isn't enough space in the journal for your transaction (based
+on the passed nblocks param) - when it blocks it merely(!) needs to wait
+for transactions to complete and be committed from other tasks, so
+essentially we are waiting for :c:func:`jbd2_journal_stop`. So to avoid
+deadlocks you must treat :c:func:`jbd2_journal_start` /
+:c:func:`jbd2_journal_stop` as if they were semaphores and include them in
+your semaphore ordering rules to prevent
+deadlocks. Note that :c:func:`jbd2_journal_extend` has similar blocking
+behaviour to :c:func:`jbd2_journal_start` so you can deadlock here just as
+easily as on :c:func:`jbd2_journal_start`.
+
+Try to reserve the right number of blocks the first time. ;-). This will
+be the maximum number of blocks you are going to touch in this
+transaction. I advise having a look at at least ext4_jbd.h to see the
+basis on which ext4 uses to make these decisions.
+
+Another wriggle to watch out for is your on-disk block allocation
+strategy. Why? Because, if you do a delete, you need to ensure you
+haven't reused any of the freed blocks until the transaction freeing
+these blocks commits. If you reused these blocks and crash happens,
+there is no way to restore the contents of the reallocated blocks at the
+end of the last fully committed transaction. One simple way of doing
+this is to mark blocks as free in internal in-memory block allocation
+structures only after the transaction freeing them commits. Ext4 uses
+journal commit callback for this purpose.
+
+With journal commit callbacks you can ask the journalling layer to call
+a callback function when the transaction is finally committed to disk,
+so that you can do some of your own management. You ask the journalling
+layer for calling the callback by simply setting
+``journal->j_commit_callback`` function pointer and that function is
+called after each transaction commit. You can also use
+``transaction->t_private_list`` for attaching entries to a transaction
+that need processing when the transaction commits.
+
+JBD2 also provides a way to block all transaction updates via
+:c:func:`jbd2_journal_lock_updates()` /
+:c:func:`jbd2_journal_unlock_updates()`. Ext4 uses this when it wants a
+window with a clean and stable fs for a moment. E.g.
+
+::
+
+
+ jbd2_journal_lock_updates() //stop new stuff happening..
+ jbd2_journal_flush() // checkpoint everything.
+ ..do stuff on stable fs
+ jbd2_journal_unlock_updates() // carry on with filesystem use.
+
+The opportunities for abuse and DOS attacks with this should be obvious,
+if you allow unprivileged userspace to trigger codepaths containing
+these calls.
+
+Summary
+~~~~~~~
+
+Using the journal is a matter of wrapping the different context changes,
+being each mount, each modification (transaction) and each changed
+buffer to tell the journalling layer about them.
+
+Data Types
+----------
+
+The journalling layer uses typedefs to 'hide' the concrete definitions
+of the structures used. As a client of the JBD2 layer you can just rely
+on the using the pointer as a magic cookie of some sort. Obviously the
+hiding is not enforced as this is 'C'.
+
+Structures
+~~~~~~~~~~
+
+.. kernel-doc:: include/linux/jbd2.h
+ :internal:
+
+Functions
+---------
+
+The functions here are split into two groups those that affect a journal
+as a whole, and those which are used to manage transactions
+
+Journal Level
+~~~~~~~~~~~~~
+
+.. kernel-doc:: fs/jbd2/journal.c
+ :export:
+
+.. kernel-doc:: fs/jbd2/recovery.c
+ :internal:
+
+Transasction Level
+~~~~~~~~~~~~~~~~~~
+
+.. kernel-doc:: fs/jbd2/transaction.c
+
+See also
+--------
+
+`Journaling the Linux ext2fs Filesystem, LinuxExpo 98, Stephen
+Tweedie <http://kernel.org/pub/linux/kernel/people/sct/ext3/journal-design.ps.gz>`__
+
+`Ext3 Journalling FileSystem, OLS 2000, Dr. Stephen
+Tweedie <http://olstrans.sourceforge.net/release/OLS2000-ext3/OLS2000-ext3.html>`__
+
+splice API
+==========
+
+splice is a method for moving blocks of data around inside the kernel,
+without continually transferring them between the kernel and user space.
+
+.. kernel-doc:: fs/splice.c
+
+pipes API
+=========
+
+Pipe interfaces are all for in-kernel (builtin image) use. They are not
+exported for use by modules.
+
+.. kernel-doc:: include/linux/pipe_fs_i.h
+ :internal:
+
+.. kernel-doc:: fs/pipe.c
this case, /some/other/program will handle all uid lookups and
/usr/sbin/nfs.idmap will handle gid, user, and group lookups.
-See <file:Documentation/security/keys-request-key.txt> for more information
+See <file:Documentation/security/keys/request-key.rst> for more information
about the request-key function.
task would be to clean up interfaces like moving functions around between
files to better group them and improving the interfaces like dropping return
values for functions that never fail. Then write kerneldoc for all exported
-functions and an overview section and integrate it all into the drm DocBook.
+functions and an overview section and integrate it all into the drm book.
See https://dri.freedesktop.org/docs/drm/ for what's there already.
Platform Data
-------------
-In linux/i2c/ads1015.h platform data is defined, channel_data contains
+In linux/platform_data/ads1015.h platform data is defined, channel_data contains
configuration data for the used input combinations:
- pga is the programmable gain amplifier (values are full scale)
0: +/- 6.144 V
Fan speed may be set to maximum when the temperature sensor associated with
the PWM control exceeds temp#_max.
+At Tmin - hysteresis the PWM output can either be off (0% duty cycle) or at the
+minimum (i.e. auto_point1_pwm). This behaviour can be configured using the
+pwm[1-*]_stall_disable sysfs attribute. A value of 0 means the fans will shut
+off. A value of 1 means the fans will run at auto_point1_pwm.
+
+The responsiveness of the ADT747x to temperature changes can be configured.
+This allows smoothing of the fan speed transition. To set the transition time
+set the value in ms in the temp[1-*]_smoothing sysfs attribute.
+
Notes
-----
--- /dev/null
+Kernel driver ir35221
+=====================
+
+Supported chips:
+ * Infinion IR35221
+ Prefix: 'ir35221'
+ Addresses scanned: -
+ Datasheet: Datasheet is not publicly available.
+
+Author: Samuel Mendoza-Jonas <sam@mendozajonas.com>
+
+
+Description
+-----------
+
+IR35221 is a Digital DC-DC Multiphase Converter
+
+
+Usage Notes
+-----------
+
+This driver does not probe for PMBus devices. You will have to instantiate
+devices explicitly.
+
+Example: the following commands will load the driver for an IR35221
+at address 0x70 on I2C bus #4:
+
+# modprobe ir35221
+# echo ir35221 0x70 > /sys/bus/i2c/devices/i2c-4/new_device
+
+
+Sysfs attributes
+----------------
+
+curr1_label "iin"
+curr1_input Measured input current
+curr1_max Maximum current
+curr1_max_alarm Current high alarm
+
+curr[2-3]_label "iout[1-2]"
+curr[2-3]_input Measured output current
+curr[2-3]_crit Critical maximum current
+curr[2-3]_crit_alarm Current critical high alarm
+curr[2-3]_highest Highest output current
+curr[2-3]_lowest Lowest output current
+curr[2-3]_max Maximum current
+curr[2-3]_max_alarm Current high alarm
+
+in1_label "vin"
+in1_input Measured input voltage
+in1_crit Critical maximum input voltage
+in1_crit_alarm Input voltage critical high alarm
+in1_highest Highest input voltage
+in1_lowest Lowest input voltage
+in1_min Minimum input voltage
+in1_min_alarm Input voltage low alarm
+
+in[2-3]_label "vout[1-2]"
+in[2-3]_input Measured output voltage
+in[2-3]_lcrit Critical minimum output voltage
+in[2-3]_lcrit_alarm Output voltage critical low alarm
+in[2-3]_crit Critical maximum output voltage
+in[2-3]_crit_alarm Output voltage critical high alarm
+in[2-3]_highest Highest output voltage
+in[2-3]_lowest Lowest output voltage
+in[2-3]_max Maximum output voltage
+in[2-3]_max_alarm Output voltage high alarm
+in[2-3]_min Minimum output voltage
+in[2-3]_min_alarm Output voltage low alarm
+
+power1_label "pin"
+power1_input Measured input power
+power1_alarm Input power high alarm
+power1_max Input power limit
+
+power[2-3]_label "pout[1-2]"
+power[2-3]_input Measured output power
+power[2-3]_max Output power limit
+power[2-3]_max_alarm Output power high alarm
+
+temp[1-2]_input Measured temperature
+temp[1-2]_crit Critical high temperature
+temp[1-2]_crit_alarm Chip temperature critical high alarm
+temp[1-2]_highest Highest temperature
+temp[1-2]_lowest Lowest temperature
+temp[1-2]_max Maximum temperature
+temp[1-2]_max_alarm Chip temperature high alarm
the sensor reading.
The LTC4245 chip can be configured to sample all GPIO pins with two methods:
-1) platform data -- see include/linux/i2c/ltc4245.h
+1) platform data -- see include/linux/platform_data/ltc4245.h
2) OF device tree -- add the "ltc4245,use-extra-gpios" property to each chip
The default mode of operation is to sample a single GPIO pin.
PMBus driver platform data
==========================
-PMBus platform data is defined in include/linux/i2c/pmbus.h. Platform data
+PMBus platform data is defined in include/linux/pmbus.h. Platform data
currently only provides a flag field with a single bit used.
#define PMBUS_SKIP_STATUS_CHECK (1 << 0)
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
-Welcome to The Linux Kernel's documentation
-===========================================
+The Linux Kernel documentation
+==============================
This is the top level of the kernel's documentation tree. Kernel
documentation, like the kernel itself, is very much a work in progress;
process/index
dev-tools/index
doc-guide/index
+ kernel-hacking/index
Kernel API documentation
------------------------
driver-api/index
core-api/index
media/index
+ networking/index
input/index
gpu/index
security/index
sound/index
crypto/index
+ filesystems/index
+
+Architecture-specific documentation
+-----------------------------------
+
+These books provide programming details about architecture-specific
+implementation.
+
+.. toctree::
+ :maxdepth: 2
+
+ sh/index
Korean translations
-------------------
--- 7.5 mandatory-y
mandatory-y is essentially used by include/uapi/asm-generic/Kbuild.asm
- to define the minimun set of headers that must be exported in
+ to define the minimum set of headers that must be exported in
include/asm.
The convention is to list one subdir per line and
It is documented in this Documentation/kernel-doc-nano-HOWTO.txt file.
This style embeds the documentation within the source files, using
-a few simple conventions. The scripts/kernel-doc perl script, some
-SGML templates in Documentation/DocBook, and other tools understand
+a few simple conventions. The scripts/kernel-doc perl script, the
+Documentation/sphinx/kerneldoc.py Sphinx extension and other tools understand
these conventions, and are used to extract this embedded documentation
into various documents.
- scripts/kernel-doc
This is a perl script that hunts for the block comments and can mark
- them up directly into DocBook, man, text, and HTML. (No, not
+ them up directly into DocBook, ReST, man, text, and HTML. (No, not
texinfo.)
-- Documentation/DocBook/*.tmpl
-
- These are SGML template files, which are normal SGML files with
- special place-holders for where the extracted documentation should
- go.
-
- scripts/docproc.c
This is a program for converting SGML template files into SGML
- Makefile
- The targets 'xmldocs', 'psdocs', 'pdfdocs', and 'htmldocs' are used
- to build XML DocBook files, PostScript files, PDF files, and html files
- in Documentation/DocBook. The older target 'sgmldocs' is equivalent
- to 'xmldocs'.
-
-- Documentation/DocBook/Makefile
-
- This is where C files are associated with SGML templates.
-
+ The targets 'xmldocs', 'latexdocs', 'pdfdocs', 'epubdocs'and 'htmldocs'
+ are used to build XML DocBook files, LaTeX files, PDF files,
+ ePub files and html files in Documentation/.
How to extract the documentation
--------------------------------
If you just want to read the ready-made books on the various
-subsystems (see Documentation/DocBook/*.tmpl), just type 'make
-psdocs', or 'make pdfdocs', or 'make htmldocs', depending on your
-preference. If you would rather read a different format, you can type
-'make xmldocs' and then use DocBook tools to convert
-Documentation/DocBook/*.xml to a format of your choice (for example,
+subsystems, just type 'make epubdocs', or 'make pdfdocs', or 'make htmldocs',
+depending on your preference. If you would rather read a different format,
+you can type 'make xmldocs' and then use DocBook tools to convert
+Documentation/output/*.xml to a format of your choice (for example,
'db2html ...' if 'make htmldocs' was not defined).
If you want to see man pages instead, you can do this:
* hardware, software, or its subject(s).
*/
-DOC: sections are used in SGML templates files as indicated below.
-
-
-How to make new SGML template files
------------------------------------
-
-SGML template files (*.tmpl) are like normal SGML files, except that
-they can contain escape sequences where extracted documentation should
-be inserted.
-
-!E<filename> is replaced by the documentation, in <filename>, for
-functions that are exported using EXPORT_SYMBOL: the function list is
-collected from files listed in Documentation/DocBook/Makefile.
-
-!I<filename> is replaced by the documentation for functions that are
-_not_ exported using EXPORT_SYMBOL.
-
-!D<filename> is used to name additional files to search for functions
-exported using EXPORT_SYMBOL.
-
-!F<filename> <function [functions...]> is replaced by the
-documentation, in <filename>, for the functions listed.
-
-!P<filename> <section title> is replaced by the contents of the DOC:
-section titled <section title> from <filename>.
-Spaces are allowed in <section title>; do not quote the <section title>.
-
-!C<filename> is replaced by nothing, but makes the tools check that
-all DOC: sections and documented functions, symbols, etc. are used.
-This makes sense to use when you use !F/!P only and want to verify
-that all documentation is included.
+DOC: sections are used in ReST files.
Tim.
*/ <twaugh@redhat.com>
--- /dev/null
+# -*- coding: utf-8; mode: python -*-
+
+project = "Kernel Hacking Guides"
+
+tags.add("subproject")
+
+latex_documents = [
+ ('index', 'kernel-hacking.tex', project,
+ 'The kernel development community', 'manual'),
+]
--- /dev/null
+============================================
+Unreliable Guide To Hacking The Linux Kernel
+============================================
+
+:Author: Rusty Russell
+
+Introduction
+============
+
+Welcome, gentle reader, to Rusty's Remarkably Unreliable Guide to Linux
+Kernel Hacking. This document describes the common routines and general
+requirements for kernel code: its goal is to serve as a primer for Linux
+kernel development for experienced C programmers. I avoid implementation
+details: that's what the code is for, and I ignore whole tracts of
+useful routines.
+
+Before you read this, please understand that I never wanted to write
+this document, being grossly under-qualified, but I always wanted to
+read it, and this was the only way. I hope it will grow into a
+compendium of best practice, common starting points and random
+information.
+
+The Players
+===========
+
+At any time each of the CPUs in a system can be:
+
+- not associated with any process, serving a hardware interrupt;
+
+- not associated with any process, serving a softirq or tasklet;
+
+- running in kernel space, associated with a process (user context);
+
+- running a process in user space.
+
+There is an ordering between these. The bottom two can preempt each
+other, but above that is a strict hierarchy: each can only be preempted
+by the ones above it. For example, while a softirq is running on a CPU,
+no other softirq will preempt it, but a hardware interrupt can. However,
+any other CPUs in the system execute independently.
+
+We'll see a number of ways that the user context can block interrupts,
+to become truly non-preemptable.
+
+User Context
+------------
+
+User context is when you are coming in from a system call or other trap:
+like userspace, you can be preempted by more important tasks and by
+interrupts. You can sleep, by calling :c:func:`schedule()`.
+
+.. note::
+
+ You are always in user context on module load and unload, and on
+ operations on the block device layer.
+
+In user context, the ``current`` pointer (indicating the task we are
+currently executing) is valid, and :c:func:`in_interrupt()`
+(``include/linux/preempt.h``) is false.
+
+.. warning::
+
+ Beware that if you have preemption or softirqs disabled (see below),
+ :c:func:`in_interrupt()` will return a false positive.
+
+Hardware Interrupts (Hard IRQs)
+-------------------------------
+
+Timer ticks, network cards and keyboard are examples of real hardware
+which produce interrupts at any time. The kernel runs interrupt
+handlers, which services the hardware. The kernel guarantees that this
+handler is never re-entered: if the same interrupt arrives, it is queued
+(or dropped). Because it disables interrupts, this handler has to be
+fast: frequently it simply acknowledges the interrupt, marks a 'software
+interrupt' for execution and exits.
+
+You can tell you are in a hardware interrupt, because
+:c:func:`in_irq()` returns true.
+
+.. warning::
+
+ Beware that this will return a false positive if interrupts are
+ disabled (see below).
+
+Software Interrupt Context: Softirqs and Tasklets
+-------------------------------------------------
+
+Whenever a system call is about to return to userspace, or a hardware
+interrupt handler exits, any 'software interrupts' which are marked
+pending (usually by hardware interrupts) are run (``kernel/softirq.c``).
+
+Much of the real interrupt handling work is done here. Early in the
+transition to SMP, there were only 'bottom halves' (BHs), which didn't
+take advantage of multiple CPUs. Shortly after we switched from wind-up
+computers made of match-sticks and snot, we abandoned this limitation
+and switched to 'softirqs'.
+
+``include/linux/interrupt.h`` lists the different softirqs. A very
+important softirq is the timer softirq (``include/linux/timer.h``): you
+can register to have it call functions for you in a given length of
+time.
+
+Softirqs are often a pain to deal with, since the same softirq will run
+simultaneously on more than one CPU. For this reason, tasklets
+(``include/linux/interrupt.h``) are more often used: they are
+dynamically-registrable (meaning you can have as many as you want), and
+they also guarantee that any tasklet will only run on one CPU at any
+time, although different tasklets can run simultaneously.
+
+.. warning::
+
+ The name 'tasklet' is misleading: they have nothing to do with
+ 'tasks', and probably more to do with some bad vodka Alexey
+ Kuznetsov had at the time.
+
+You can tell you are in a softirq (or tasklet) using the
+:c:func:`in_softirq()` macro (``include/linux/preempt.h``).
+
+.. warning::
+
+ Beware that this will return a false positive if a
+ :ref:`botton half lock <local_bh_disable>` is held.
+
+Some Basic Rules
+================
+
+No memory protection
+ If you corrupt memory, whether in user context or interrupt context,
+ the whole machine will crash. Are you sure you can't do what you
+ want in userspace?
+
+No floating point or MMX
+ The FPU context is not saved; even in user context the FPU state
+ probably won't correspond with the current process: you would mess
+ with some user process' FPU state. If you really want to do this,
+ you would have to explicitly save/restore the full FPU state (and
+ avoid context switches). It is generally a bad idea; use fixed point
+ arithmetic first.
+
+A rigid stack limit
+ Depending on configuration options the kernel stack is about 3K to
+ 6K for most 32-bit architectures: it's about 14K on most 64-bit
+ archs, and often shared with interrupts so you can't use it all.
+ Avoid deep recursion and huge local arrays on the stack (allocate
+ them dynamically instead).
+
+The Linux kernel is portable
+ Let's keep it that way. Your code should be 64-bit clean, and
+ endian-independent. You should also minimize CPU specific stuff,
+ e.g. inline assembly should be cleanly encapsulated and minimized to
+ ease porting. Generally it should be restricted to the
+ architecture-dependent part of the kernel tree.
+
+ioctls: Not writing a new system call
+=====================================
+
+A system call generally looks like this::
+
+ asmlinkage long sys_mycall(int arg)
+ {
+ return 0;
+ }
+
+
+First, in most cases you don't want to create a new system call. You
+create a character device and implement an appropriate ioctl for it.
+This is much more flexible than system calls, doesn't have to be entered
+in every architecture's ``include/asm/unistd.h`` and
+``arch/kernel/entry.S`` file, and is much more likely to be accepted by
+Linus.
+
+If all your routine does is read or write some parameter, consider
+implementing a :c:func:`sysfs()` interface instead.
+
+Inside the ioctl you're in user context to a process. When a error
+occurs you return a negated errno (see
+``include/uapi/asm-generic/errno-base.h``,
+``include/uapi/asm-generic/errno.h`` and ``include/linux/errno.h``),
+otherwise you return 0.
+
+After you slept you should check if a signal occurred: the Unix/Linux
+way of handling signals is to temporarily exit the system call with the
+``-ERESTARTSYS`` error. The system call entry code will switch back to
+user context, process the signal handler and then your system call will
+be restarted (unless the user disabled that). So you should be prepared
+to process the restart, e.g. if you're in the middle of manipulating
+some data structure.
+
+::
+
+ if (signal_pending(current))
+ return -ERESTARTSYS;
+
+
+If you're doing longer computations: first think userspace. If you
+**really** want to do it in kernel you should regularly check if you need
+to give up the CPU (remember there is cooperative multitasking per CPU).
+Idiom::
+
+ cond_resched(); /* Will sleep */
+
+
+A short note on interface design: the UNIX system call motto is "Provide
+mechanism not policy".
+
+Recipes for Deadlock
+====================
+
+You cannot call any routines which may sleep, unless:
+
+- You are in user context.
+
+- You do not own any spinlocks.
+
+- You have interrupts enabled (actually, Andi Kleen says that the
+ scheduling code will enable them for you, but that's probably not
+ what you wanted).
+
+Note that some functions may sleep implicitly: common ones are the user
+space access functions (\*_user) and memory allocation functions
+without ``GFP_ATOMIC``.
+
+You should always compile your kernel ``CONFIG_DEBUG_ATOMIC_SLEEP`` on,
+and it will warn you if you break these rules. If you **do** break the
+rules, you will eventually lock up your box.
+
+Really.
+
+Common Routines
+===============
+
+:c:func:`printk()`
+------------------
+
+Defined in ``include/linux/printk.h``
+
+:c:func:`printk()` feeds kernel messages to the console, dmesg, and
+the syslog daemon. It is useful for debugging and reporting errors, and
+can be used inside interrupt context, but use with caution: a machine
+which has its console flooded with printk messages is unusable. It uses
+a format string mostly compatible with ANSI C printf, and C string
+concatenation to give it a first "priority" argument::
+
+ printk(KERN_INFO "i = %u\n", i);
+
+
+See ``include/linux/kern_levels.h``; for other ``KERN_`` values; these are
+interpreted by syslog as the level. Special case: for printing an IP
+address use::
+
+ __be32 ipaddress;
+ printk(KERN_INFO "my ip: %pI4\n", &ipaddress);
+
+
+:c:func:`printk()` internally uses a 1K buffer and does not catch
+overruns. Make sure that will be enough.
+
+.. note::
+
+ You will know when you are a real kernel hacker when you start
+ typoing printf as printk in your user programs :)
+
+.. note::
+
+ Another sidenote: the original Unix Version 6 sources had a comment
+ on top of its printf function: "Printf should not be used for
+ chit-chat". You should follow that advice.
+
+:c:func:`copy_to_user()` / :c:func:`copy_from_user()` / :c:func:`get_user()` / :c:func:`put_user()`
+---------------------------------------------------------------------------------------------------
+
+Defined in ``include/linux/uaccess.h`` / ``asm/uaccess.h``
+
+**[SLEEPS]**
+
+:c:func:`put_user()` and :c:func:`get_user()` are used to get
+and put single values (such as an int, char, or long) from and to
+userspace. A pointer into userspace should never be simply dereferenced:
+data should be copied using these routines. Both return ``-EFAULT`` or
+0.
+
+:c:func:`copy_to_user()` and :c:func:`copy_from_user()` are
+more general: they copy an arbitrary amount of data to and from
+userspace.
+
+.. warning::
+
+ Unlike :c:func:`put_user()` and :c:func:`get_user()`, they
+ return the amount of uncopied data (ie. 0 still means success).
+
+[Yes, this moronic interface makes me cringe. The flamewar comes up
+every year or so. --RR.]
+
+The functions may sleep implicitly. This should never be called outside
+user context (it makes no sense), with interrupts disabled, or a
+spinlock held.
+
+:c:func:`kmalloc()`/:c:func:`kfree()`
+-------------------------------------
+
+Defined in ``include/linux/slab.h``
+
+**[MAY SLEEP: SEE BELOW]**
+
+These routines are used to dynamically request pointer-aligned chunks of
+memory, like malloc and free do in userspace, but
+:c:func:`kmalloc()` takes an extra flag word. Important values:
+
+``GFP_KERNEL``
+ May sleep and swap to free memory. Only allowed in user context, but
+ is the most reliable way to allocate memory.
+
+``GFP_ATOMIC``
+ Don't sleep. Less reliable than ``GFP_KERNEL``, but may be called
+ from interrupt context. You should **really** have a good
+ out-of-memory error-handling strategy.
+
+``GFP_DMA``
+ Allocate ISA DMA lower than 16MB. If you don't know what that is you
+ don't need it. Very unreliable.
+
+If you see a sleeping function called from invalid context warning
+message, then maybe you called a sleeping allocation function from
+interrupt context without ``GFP_ATOMIC``. You should really fix that.
+Run, don't walk.
+
+If you are allocating at least ``PAGE_SIZE`` (``asm/page.h`` or
+``asm/page_types.h``) bytes, consider using :c:func:`__get_free_pages()`
+(``include/linux/gfp.h``). It takes an order argument (0 for page sized,
+1 for double page, 2 for four pages etc.) and the same memory priority
+flag word as above.
+
+If you are allocating more than a page worth of bytes you can use
+:c:func:`vmalloc()`. It'll allocate virtual memory in the kernel
+map. This block is not contiguous in physical memory, but the MMU makes
+it look like it is for you (so it'll only look contiguous to the CPUs,
+not to external device drivers). If you really need large physically
+contiguous memory for some weird device, you have a problem: it is
+poorly supported in Linux because after some time memory fragmentation
+in a running kernel makes it hard. The best way is to allocate the block
+early in the boot process via the :c:func:`alloc_bootmem()`
+routine.
+
+Before inventing your own cache of often-used objects consider using a
+slab cache in ``include/linux/slab.h``
+
+:c:func:`current()`
+-------------------
+
+Defined in ``include/asm/current.h``
+
+This global variable (really a macro) contains a pointer to the current
+task structure, so is only valid in user context. For example, when a
+process makes a system call, this will point to the task structure of
+the calling process. It is **not NULL** in interrupt context.
+
+:c:func:`mdelay()`/:c:func:`udelay()`
+-------------------------------------
+
+Defined in ``include/asm/delay.h`` / ``include/linux/delay.h``
+
+The :c:func:`udelay()` and :c:func:`ndelay()` functions can be
+used for small pauses. Do not use large values with them as you risk
+overflow - the helper function :c:func:`mdelay()` is useful here, or
+consider :c:func:`msleep()`.
+
+:c:func:`cpu_to_be32()`/:c:func:`be32_to_cpu()`/:c:func:`cpu_to_le32()`/:c:func:`le32_to_cpu()`
+-----------------------------------------------------------------------------------------------
+
+Defined in ``include/asm/byteorder.h``
+
+The :c:func:`cpu_to_be32()` family (where the "32" can be replaced
+by 64 or 16, and the "be" can be replaced by "le") are the general way
+to do endian conversions in the kernel: they return the converted value.
+All variations supply the reverse as well:
+:c:func:`be32_to_cpu()`, etc.
+
+There are two major variations of these functions: the pointer
+variation, such as :c:func:`cpu_to_be32p()`, which take a pointer
+to the given type, and return the converted value. The other variation
+is the "in-situ" family, such as :c:func:`cpu_to_be32s()`, which
+convert value referred to by the pointer, and return void.
+
+:c:func:`local_irq_save()`/:c:func:`local_irq_restore()`
+--------------------------------------------------------
+
+Defined in ``include/linux/irqflags.h``
+
+These routines disable hard interrupts on the local CPU, and restore
+them. They are reentrant; saving the previous state in their one
+``unsigned long flags`` argument. If you know that interrupts are
+enabled, you can simply use :c:func:`local_irq_disable()` and
+:c:func:`local_irq_enable()`.
+
+.. _local_bh_disable:
+
+:c:func:`local_bh_disable()`/:c:func:`local_bh_enable()`
+--------------------------------------------------------
+
+Defined in ``include/linux/bottom_half.h``
+
+
+These routines disable soft interrupts on the local CPU, and restore
+them. They are reentrant; if soft interrupts were disabled before, they
+will still be disabled after this pair of functions has been called.
+They prevent softirqs and tasklets from running on the current CPU.
+
+:c:func:`smp_processor_id()`
+----------------------------
+
+Defined in ``include/linux/smp.h``
+
+:c:func:`get_cpu()` disables preemption (so you won't suddenly get
+moved to another CPU) and returns the current processor number, between
+0 and ``NR_CPUS``. Note that the CPU numbers are not necessarily
+continuous. You return it again with :c:func:`put_cpu()` when you
+are done.
+
+If you know you cannot be preempted by another task (ie. you are in
+interrupt context, or have preemption disabled) you can use
+smp_processor_id().
+
+``__init``/``__exit``/``__initdata``
+------------------------------------
+
+Defined in ``include/linux/init.h``
+
+After boot, the kernel frees up a special section; functions marked with
+``__init`` and data structures marked with ``__initdata`` are dropped
+after boot is complete: similarly modules discard this memory after
+initialization. ``__exit`` is used to declare a function which is only
+required on exit: the function will be dropped if this file is not
+compiled as a module. See the header file for use. Note that it makes no
+sense for a function marked with ``__init`` to be exported to modules
+with :c:func:`EXPORT_SYMBOL()` or :c:func:`EXPORT_SYMBOL_GPL()`- this
+will break.
+
+:c:func:`__initcall()`/:c:func:`module_init()`
+----------------------------------------------
+
+Defined in ``include/linux/init.h`` / ``include/linux/module.h``
+
+Many parts of the kernel are well served as a module
+(dynamically-loadable parts of the kernel). Using the
+:c:func:`module_init()` and :c:func:`module_exit()` macros it
+is easy to write code without #ifdefs which can operate both as a module
+or built into the kernel.
+
+The :c:func:`module_init()` macro defines which function is to be
+called at module insertion time (if the file is compiled as a module),
+or at boot time: if the file is not compiled as a module the
+:c:func:`module_init()` macro becomes equivalent to
+:c:func:`__initcall()`, which through linker magic ensures that
+the function is called on boot.
+
+The function can return a negative error number to cause module loading
+to fail (unfortunately, this has no effect if the module is compiled
+into the kernel). This function is called in user context with
+interrupts enabled, so it can sleep.
+
+:c:func:`module_exit()`
+-----------------------
+
+
+Defined in ``include/linux/module.h``
+
+This macro defines the function to be called at module removal time (or
+never, in the case of the file compiled into the kernel). It will only
+be called if the module usage count has reached zero. This function can
+also sleep, but cannot fail: everything must be cleaned up by the time
+it returns.
+
+Note that this macro is optional: if it is not present, your module will
+not be removable (except for 'rmmod -f').
+
+:c:func:`try_module_get()`/:c:func:`module_put()`
+-------------------------------------------------
+
+Defined in ``include/linux/module.h``
+
+These manipulate the module usage count, to protect against removal (a
+module also can't be removed if another module uses one of its exported
+symbols: see below). Before calling into module code, you should call
+:c:func:`try_module_get()` on that module: if it fails, then the
+module is being removed and you should act as if it wasn't there.
+Otherwise, you can safely enter the module, and call
+:c:func:`module_put()` when you're finished.
+
+Most registerable structures have an owner field, such as in the
+:c:type:`struct file_operations <file_operations>` structure.
+Set this field to the macro ``THIS_MODULE``.
+
+Wait Queues ``include/linux/wait.h``
+====================================
+
+**[SLEEPS]**
+
+A wait queue is used to wait for someone to wake you up when a certain
+condition is true. They must be used carefully to ensure there is no
+race condition. You declare a :c:type:`wait_queue_head_t`, and then processes
+which want to wait for that condition declare a :c:type:`wait_queue_entry_t`
+referring to themselves, and place that in the queue.
+
+Declaring
+---------
+
+You declare a ``wait_queue_head_t`` using the
+:c:func:`DECLARE_WAIT_QUEUE_HEAD()` macro, or using the
+:c:func:`init_waitqueue_head()` routine in your initialization
+code.
+
+Queuing
+-------
+
+Placing yourself in the waitqueue is fairly complex, because you must
+put yourself in the queue before checking the condition. There is a
+macro to do this: :c:func:`wait_event_interruptible()`
+(``include/linux/wait.h``) The first argument is the wait queue head, and
+the second is an expression which is evaluated; the macro returns 0 when
+this expression is true, or ``-ERESTARTSYS`` if a signal is received. The
+:c:func:`wait_event()` version ignores signals.
+
+Waking Up Queued Tasks
+----------------------
+
+Call :c:func:`wake_up()` (``include/linux/wait.h``);, which will wake
+up every process in the queue. The exception is if one has
+``TASK_EXCLUSIVE`` set, in which case the remainder of the queue will
+not be woken. There are other variants of this basic function available
+in the same header.
+
+Atomic Operations
+=================
+
+Certain operations are guaranteed atomic on all platforms. The first
+class of operations work on :c:type:`atomic_t` (``include/asm/atomic.h``);
+this contains a signed integer (at least 32 bits long), and you must use
+these functions to manipulate or read :c:type:`atomic_t` variables.
+:c:func:`atomic_read()` and :c:func:`atomic_set()` get and set
+the counter, :c:func:`atomic_add()`, :c:func:`atomic_sub()`,
+:c:func:`atomic_inc()`, :c:func:`atomic_dec()`, and
+:c:func:`atomic_dec_and_test()` (returns true if it was
+decremented to zero).
+
+Yes. It returns true (i.e. != 0) if the atomic variable is zero.
+
+Note that these functions are slower than normal arithmetic, and so
+should not be used unnecessarily.
+
+The second class of atomic operations is atomic bit operations on an
+``unsigned long``, defined in ``include/linux/bitops.h``. These
+operations generally take a pointer to the bit pattern, and a bit
+number: 0 is the least significant bit. :c:func:`set_bit()`,
+:c:func:`clear_bit()` and :c:func:`change_bit()` set, clear,
+and flip the given bit. :c:func:`test_and_set_bit()`,
+:c:func:`test_and_clear_bit()` and
+:c:func:`test_and_change_bit()` do the same thing, except return
+true if the bit was previously set; these are particularly useful for
+atomically setting flags.
+
+It is possible to call these operations with bit indices greater than
+``BITS_PER_LONG``. The resulting behavior is strange on big-endian
+platforms though so it is a good idea not to do this.
+
+Symbols
+=======
+
+Within the kernel proper, the normal linking rules apply (ie. unless a
+symbol is declared to be file scope with the ``static`` keyword, it can
+be used anywhere in the kernel). However, for modules, a special
+exported symbol table is kept which limits the entry points to the
+kernel proper. Modules can also export symbols.
+
+:c:func:`EXPORT_SYMBOL()`
+-------------------------
+
+Defined in ``include/linux/export.h``
+
+This is the classic method of exporting a symbol: dynamically loaded
+modules will be able to use the symbol as normal.
+
+:c:func:`EXPORT_SYMBOL_GPL()`
+-----------------------------
+
+Defined in ``include/linux/export.h``
+
+Similar to :c:func:`EXPORT_SYMBOL()` except that the symbols
+exported by :c:func:`EXPORT_SYMBOL_GPL()` can only be seen by
+modules with a :c:func:`MODULE_LICENSE()` that specifies a GPL
+compatible license. It implies that the function is considered an
+internal implementation issue, and not really an interface. Some
+maintainers and developers may however require EXPORT_SYMBOL_GPL()
+when adding any new APIs or functionality.
+
+Routines and Conventions
+========================
+
+Double-linked lists ``include/linux/list.h``
+--------------------------------------------
+
+There used to be three sets of linked-list routines in the kernel
+headers, but this one is the winner. If you don't have some particular
+pressing need for a single list, it's a good choice.
+
+In particular, :c:func:`list_for_each_entry()` is useful.
+
+Return Conventions
+------------------
+
+For code called in user context, it's very common to defy C convention,
+and return 0 for success, and a negative error number (eg. ``-EFAULT``) for
+failure. This can be unintuitive at first, but it's fairly widespread in
+the kernel.
+
+Using :c:func:`ERR_PTR()` (``include/linux/err.h``) to encode a
+negative error number into a pointer, and :c:func:`IS_ERR()` and
+:c:func:`PTR_ERR()` to get it back out again: avoids a separate
+pointer parameter for the error number. Icky, but in a good way.
+
+Breaking Compilation
+--------------------
+
+Linus and the other developers sometimes change function or structure
+names in development kernels; this is not done just to keep everyone on
+their toes: it reflects a fundamental change (eg. can no longer be
+called with interrupts on, or does extra checks, or doesn't do checks
+which were caught before). Usually this is accompanied by a fairly
+complete note to the linux-kernel mailing list; search the archive.
+Simply doing a global replace on the file usually makes things **worse**.
+
+Initializing structure members
+------------------------------
+
+The preferred method of initializing structures is to use designated
+initialisers, as defined by ISO C99, eg::
+
+ static struct block_device_operations opt_fops = {
+ .open = opt_open,
+ .release = opt_release,
+ .ioctl = opt_ioctl,
+ .check_media_change = opt_media_change,
+ };
+
+
+This makes it easy to grep for, and makes it clear which structure
+fields are set. You should do this because it looks cool.
+
+GNU Extensions
+--------------
+
+GNU Extensions are explicitly allowed in the Linux kernel. Note that
+some of the more complex ones are not very well supported, due to lack
+of general use, but the following are considered standard (see the GCC
+info page section "C Extensions" for more details - Yes, really the info
+page, the man page is only a short summary of the stuff in info).
+
+- Inline functions
+
+- Statement expressions (ie. the ({ and }) constructs).
+
+- Declaring attributes of a function / variable / type
+ (__attribute__)
+
+- typeof
+
+- Zero length arrays
+
+- Macro varargs
+
+- Arithmetic on void pointers
+
+- Non-Constant initializers
+
+- Assembler Instructions (not outside arch/ and include/asm/)
+
+- Function names as strings (__func__).
+
+- __builtin_constant_p()
+
+Be wary when using long long in the kernel, the code gcc generates for
+it is horrible and worse: division and multiplication does not work on
+i386 because the GCC runtime functions for it are missing from the
+kernel environment.
+
+C++
+---
+
+Using C++ in the kernel is usually a bad idea, because the kernel does
+not provide the necessary runtime environment and the include files are
+not tested for it. It is still possible, but not recommended. If you
+really want to do this, forget about exceptions at least.
+
+NUMif
+-----
+
+It is generally considered cleaner to use macros in header files (or at
+the top of .c files) to abstract away functions rather than using \`#if'
+pre-processor statements throughout the source code.
+
+Putting Your Stuff in the Kernel
+================================
+
+In order to get your stuff into shape for official inclusion, or even to
+make a neat patch, there's administrative work to be done:
+
+- Figure out whose pond you've been pissing in. Look at the top of the
+ source files, inside the ``MAINTAINERS`` file, and last of all in the
+ ``CREDITS`` file. You should coordinate with this person to make sure
+ you're not duplicating effort, or trying something that's already
+ been rejected.
+
+ Make sure you put your name and EMail address at the top of any files
+ you create or mangle significantly. This is the first place people
+ will look when they find a bug, or when **they** want to make a change.
+
+- Usually you want a configuration option for your kernel hack. Edit
+ ``Kconfig`` in the appropriate directory. The Config language is
+ simple to use by cut and paste, and there's complete documentation in
+ ``Documentation/kbuild/kconfig-language.txt``.
+
+ In your description of the option, make sure you address both the
+ expert user and the user who knows nothing about your feature.
+ Mention incompatibilities and issues here. **Definitely** end your
+ description with “if in doubt, say N” (or, occasionally, \`Y'); this
+ is for people who have no idea what you are talking about.
+
+- Edit the ``Makefile``: the CONFIG variables are exported here so you
+ can usually just add a "obj-$(CONFIG_xxx) += xxx.o" line. The syntax
+ is documented in ``Documentation/kbuild/makefiles.txt``.
+
+- Put yourself in ``CREDITS`` if you've done something noteworthy,
+ usually beyond a single file (your name should be at the top of the
+ source files anyway). ``MAINTAINERS`` means you want to be consulted
+ when changes are made to a subsystem, and hear about bugs; it implies
+ a more-than-passing commitment to some part of the code.
+
+- Finally, don't forget to read
+ ``Documentation/process/submitting-patches.rst`` and possibly
+ ``Documentation/process/submitting-drivers.rst``.
+
+Kernel Cantrips
+===============
+
+Some favorites from browsing the source. Feel free to add to this list.
+
+``arch/x86/include/asm/delay.h``::
+
+ #define ndelay(n) (__builtin_constant_p(n) ? \
+ ((n) > 20000 ? __bad_ndelay() : __const_udelay((n) * 5ul)) : \
+ __ndelay(n))
+
+
+``include/linux/fs.h``::
+
+ /*
+ * Kernel pointers have redundant information, so we can use a
+ * scheme where we can return either an error code or a dentry
+ * pointer with the same return value.
+ *
+ * This should be a per-architecture thing, to allow different
+ * error and pointer decisions.
+ */
+ #define ERR_PTR(err) ((void *)((long)(err)))
+ #define PTR_ERR(ptr) ((long)(ptr))
+ #define IS_ERR(ptr) ((unsigned long)(ptr) > (unsigned long)(-1000))
+
+``arch/x86/include/asm/uaccess_32.h:``::
+
+ #define copy_to_user(to,from,n) \
+ (__builtin_constant_p(n) ? \
+ __constant_copy_to_user((to),(from),(n)) : \
+ __generic_copy_to_user((to),(from),(n)))
+
+
+``arch/sparc/kernel/head.S:``::
+
+ /*
+ * Sun people can't spell worth damn. "compatability" indeed.
+ * At least we *know* we can't spell, and use a spell-checker.
+ */
+
+ /* Uh, actually Linus it is I who cannot spell. Too much murky
+ * Sparc assembly will do this to ya.
+ */
+ C_LABEL(cputypvar):
+ .asciz "compatibility"
+
+ /* Tested on SS-5, SS-10. Probably someone at Sun applied a spell-checker. */
+ .align 4
+ C_LABEL(cputypvar_sun4m):
+ .asciz "compatible"
+
+
+``arch/sparc/lib/checksum.S:``::
+
+ /* Sun, you just can't beat me, you just can't. Stop trying,
+ * give up. I'm serious, I am going to kick the living shit
+ * out of you, game over, lights out.
+ */
+
+
+Thanks
+======
+
+Thanks to Andi Kleen for the idea, answering my questions, fixing my
+mistakes, filling content, etc. Philipp Rumpf for more spelling and
+clarity fixes, and some excellent non-obvious points. Werner Almesberger
+for giving me a great summary of :c:func:`disable_irq()`, and Jes
+Sorensen and Andrea Arcangeli added caveats. Michael Elizabeth Chastain
+for checking and adding to the Configure section. Telsa Gwynne for
+teaching me DocBook.
--- /dev/null
+=====================
+Kernel Hacking Guides
+=====================
+
+.. toctree::
+ :maxdepth: 2
+
+ hacking
+ locking
--- /dev/null
+===========================
+Unreliable Guide To Locking
+===========================
+
+:Author: Rusty Russell
+
+Introduction
+============
+
+Welcome, to Rusty's Remarkably Unreliable Guide to Kernel Locking
+issues. This document describes the locking systems in the Linux Kernel
+in 2.6.
+
+With the wide availability of HyperThreading, and preemption in the
+Linux Kernel, everyone hacking on the kernel needs to know the
+fundamentals of concurrency and locking for SMP.
+
+The Problem With Concurrency
+============================
+
+(Skip this if you know what a Race Condition is).
+
+In a normal program, you can increment a counter like so:
+
+::
+
+ very_important_count++;
+
+
+This is what they would expect to happen:
+
+
+.. table:: Expected Results
+
+ +------------------------------------+------------------------------------+
+ | Instance 1 | Instance 2 |
+ +====================================+====================================+
+ | read very_important_count (5) | |
+ +------------------------------------+------------------------------------+
+ | add 1 (6) | |
+ +------------------------------------+------------------------------------+
+ | write very_important_count (6) | |
+ +------------------------------------+------------------------------------+
+ | | read very_important_count (6) |
+ +------------------------------------+------------------------------------+
+ | | add 1 (7) |
+ +------------------------------------+------------------------------------+
+ | | write very_important_count (7) |
+ +------------------------------------+------------------------------------+
+
+This is what might happen:
+
+.. table:: Possible Results
+
+ +------------------------------------+------------------------------------+
+ | Instance 1 | Instance 2 |
+ +====================================+====================================+
+ | read very_important_count (5) | |
+ +------------------------------------+------------------------------------+
+ | | read very_important_count (5) |
+ +------------------------------------+------------------------------------+
+ | add 1 (6) | |
+ +------------------------------------+------------------------------------+
+ | | add 1 (6) |
+ +------------------------------------+------------------------------------+
+ | write very_important_count (6) | |
+ +------------------------------------+------------------------------------+
+ | | write very_important_count (6) |
+ +------------------------------------+------------------------------------+
+
+
+Race Conditions and Critical Regions
+------------------------------------
+
+This overlap, where the result depends on the relative timing of
+multiple tasks, is called a race condition. The piece of code containing
+the concurrency issue is called a critical region. And especially since
+Linux starting running on SMP machines, they became one of the major
+issues in kernel design and implementation.
+
+Preemption can have the same effect, even if there is only one CPU: by
+preempting one task during the critical region, we have exactly the same
+race condition. In this case the thread which preempts might run the
+critical region itself.
+
+The solution is to recognize when these simultaneous accesses occur, and
+use locks to make sure that only one instance can enter the critical
+region at any time. There are many friendly primitives in the Linux
+kernel to help you do this. And then there are the unfriendly
+primitives, but I'll pretend they don't exist.
+
+Locking in the Linux Kernel
+===========================
+
+If I could give you one piece of advice: never sleep with anyone crazier
+than yourself. But if I had to give you advice on locking: **keep it
+simple**.
+
+Be reluctant to introduce new locks.
+
+Strangely enough, this last one is the exact reverse of my advice when
+you **have** slept with someone crazier than yourself. And you should
+think about getting a big dog.
+
+Two Main Types of Kernel Locks: Spinlocks and Mutexes
+-----------------------------------------------------
+
+There are two main types of kernel locks. The fundamental type is the
+spinlock (``include/asm/spinlock.h``), which is a very simple
+single-holder lock: if you can't get the spinlock, you keep trying
+(spinning) until you can. Spinlocks are very small and fast, and can be
+used anywhere.
+
+The second type is a mutex (``include/linux/mutex.h``): it is like a
+spinlock, but you may block holding a mutex. If you can't lock a mutex,
+your task will suspend itself, and be woken up when the mutex is
+released. This means the CPU can do something else while you are
+waiting. There are many cases when you simply can't sleep (see
+`What Functions Are Safe To Call From Interrupts? <#sleeping-things>`__),
+and so have to use a spinlock instead.
+
+Neither type of lock is recursive: see
+`Deadlock: Simple and Advanced <#deadlock>`__.
+
+Locks and Uniprocessor Kernels
+------------------------------
+
+For kernels compiled without ``CONFIG_SMP``, and without
+``CONFIG_PREEMPT`` spinlocks do not exist at all. This is an excellent
+design decision: when no-one else can run at the same time, there is no
+reason to have a lock.
+
+If the kernel is compiled without ``CONFIG_SMP``, but ``CONFIG_PREEMPT``
+is set, then spinlocks simply disable preemption, which is sufficient to
+prevent any races. For most purposes, we can think of preemption as
+equivalent to SMP, and not worry about it separately.
+
+You should always test your locking code with ``CONFIG_SMP`` and
+``CONFIG_PREEMPT`` enabled, even if you don't have an SMP test box,
+because it will still catch some kinds of locking bugs.
+
+Mutexes still exist, because they are required for synchronization
+between user contexts, as we will see below.
+
+Locking Only In User Context
+----------------------------
+
+If you have a data structure which is only ever accessed from user
+context, then you can use a simple mutex (``include/linux/mutex.h``) to
+protect it. This is the most trivial case: you initialize the mutex.
+Then you can call :c:func:`mutex_lock_interruptible()` to grab the
+mutex, and :c:func:`mutex_unlock()` to release it. There is also a
+:c:func:`mutex_lock()`, which should be avoided, because it will
+not return if a signal is received.
+
+Example: ``net/netfilter/nf_sockopt.c`` allows registration of new
+:c:func:`setsockopt()` and :c:func:`getsockopt()` calls, with
+:c:func:`nf_register_sockopt()`. Registration and de-registration
+are only done on module load and unload (and boot time, where there is
+no concurrency), and the list of registrations is only consulted for an
+unknown :c:func:`setsockopt()` or :c:func:`getsockopt()` system
+call. The ``nf_sockopt_mutex`` is perfect to protect this, especially
+since the setsockopt and getsockopt calls may well sleep.
+
+Locking Between User Context and Softirqs
+-----------------------------------------
+
+If a softirq shares data with user context, you have two problems.
+Firstly, the current user context can be interrupted by a softirq, and
+secondly, the critical region could be entered from another CPU. This is
+where :c:func:`spin_lock_bh()` (``include/linux/spinlock.h``) is
+used. It disables softirqs on that CPU, then grabs the lock.
+:c:func:`spin_unlock_bh()` does the reverse. (The '_bh' suffix is
+a historical reference to "Bottom Halves", the old name for software
+interrupts. It should really be called spin_lock_softirq()' in a
+perfect world).
+
+Note that you can also use :c:func:`spin_lock_irq()` or
+:c:func:`spin_lock_irqsave()` here, which stop hardware interrupts
+as well: see `Hard IRQ Context <#hardirq-context>`__.
+
+This works perfectly for UP as well: the spin lock vanishes, and this
+macro simply becomes :c:func:`local_bh_disable()`
+(``include/linux/interrupt.h``), which protects you from the softirq
+being run.
+
+Locking Between User Context and Tasklets
+-----------------------------------------
+
+This is exactly the same as above, because tasklets are actually run
+from a softirq.
+
+Locking Between User Context and Timers
+---------------------------------------
+
+This, too, is exactly the same as above, because timers are actually run
+from a softirq. From a locking point of view, tasklets and timers are
+identical.
+
+Locking Between Tasklets/Timers
+-------------------------------
+
+Sometimes a tasklet or timer might want to share data with another
+tasklet or timer.
+
+The Same Tasklet/Timer
+~~~~~~~~~~~~~~~~~~~~~~
+
+Since a tasklet is never run on two CPUs at once, you don't need to
+worry about your tasklet being reentrant (running twice at once), even
+on SMP.
+
+Different Tasklets/Timers
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If another tasklet/timer wants to share data with your tasklet or timer
+, you will both need to use :c:func:`spin_lock()` and
+:c:func:`spin_unlock()` calls. :c:func:`spin_lock_bh()` is
+unnecessary here, as you are already in a tasklet, and none will be run
+on the same CPU.
+
+Locking Between Softirqs
+------------------------
+
+Often a softirq might want to share data with itself or a tasklet/timer.
+
+The Same Softirq
+~~~~~~~~~~~~~~~~
+
+The same softirq can run on the other CPUs: you can use a per-CPU array
+(see `Per-CPU Data <#per-cpu>`__) for better performance. If you're
+going so far as to use a softirq, you probably care about scalable
+performance enough to justify the extra complexity.
+
+You'll need to use :c:func:`spin_lock()` and
+:c:func:`spin_unlock()` for shared data.
+
+Different Softirqs
+~~~~~~~~~~~~~~~~~~
+
+You'll need to use :c:func:`spin_lock()` and
+:c:func:`spin_unlock()` for shared data, whether it be a timer,
+tasklet, different softirq or the same or another softirq: any of them
+could be running on a different CPU.
+
+Hard IRQ Context
+================
+
+Hardware interrupts usually communicate with a tasklet or softirq.
+Frequently this involves putting work in a queue, which the softirq will
+take out.
+
+Locking Between Hard IRQ and Softirqs/Tasklets
+----------------------------------------------
+
+If a hardware irq handler shares data with a softirq, you have two
+concerns. Firstly, the softirq processing can be interrupted by a
+hardware interrupt, and secondly, the critical region could be entered
+by a hardware interrupt on another CPU. This is where
+:c:func:`spin_lock_irq()` is used. It is defined to disable
+interrupts on that cpu, then grab the lock.
+:c:func:`spin_unlock_irq()` does the reverse.
+
+The irq handler does not to use :c:func:`spin_lock_irq()`, because
+the softirq cannot run while the irq handler is running: it can use
+:c:func:`spin_lock()`, which is slightly faster. The only exception
+would be if a different hardware irq handler uses the same lock:
+:c:func:`spin_lock_irq()` will stop that from interrupting us.
+
+This works perfectly for UP as well: the spin lock vanishes, and this
+macro simply becomes :c:func:`local_irq_disable()`
+(``include/asm/smp.h``), which protects you from the softirq/tasklet/BH
+being run.
+
+:c:func:`spin_lock_irqsave()` (``include/linux/spinlock.h``) is a
+variant which saves whether interrupts were on or off in a flags word,
+which is passed to :c:func:`spin_unlock_irqrestore()`. This means
+that the same code can be used inside an hard irq handler (where
+interrupts are already off) and in softirqs (where the irq disabling is
+required).
+
+Note that softirqs (and hence tasklets and timers) are run on return
+from hardware interrupts, so :c:func:`spin_lock_irq()` also stops
+these. In that sense, :c:func:`spin_lock_irqsave()` is the most
+general and powerful locking function.
+
+Locking Between Two Hard IRQ Handlers
+-------------------------------------
+
+It is rare to have to share data between two IRQ handlers, but if you
+do, :c:func:`spin_lock_irqsave()` should be used: it is
+architecture-specific whether all interrupts are disabled inside irq
+handlers themselves.
+
+Cheat Sheet For Locking
+=======================
+
+Pete Zaitcev gives the following summary:
+
+- If you are in a process context (any syscall) and want to lock other
+ process out, use a mutex. You can take a mutex and sleep
+ (``copy_from_user*(`` or ``kmalloc(x,GFP_KERNEL)``).
+
+- Otherwise (== data can be touched in an interrupt), use
+ :c:func:`spin_lock_irqsave()` and
+ :c:func:`spin_unlock_irqrestore()`.
+
+- Avoid holding spinlock for more than 5 lines of code and across any
+ function call (except accessors like :c:func:`readb()`).
+
+Table of Minimum Requirements
+-----------------------------
+
+The following table lists the **minimum** locking requirements between
+various contexts. In some cases, the same context can only be running on
+one CPU at a time, so no locking is required for that context (eg. a
+particular thread can only run on one CPU at a time, but if it needs
+shares data with another thread, locking is required).
+
+Remember the advice above: you can always use
+:c:func:`spin_lock_irqsave()`, which is a superset of all other
+spinlock primitives.
+
+============== ============= ============= ========= ========= ========= ========= ======= ======= ============== ==============
+. IRQ Handler A IRQ Handler B Softirq A Softirq B Tasklet A Tasklet B Timer A Timer B User Context A User Context B
+============== ============= ============= ========= ========= ========= ========= ======= ======= ============== ==============
+IRQ Handler A None
+IRQ Handler B SLIS None
+Softirq A SLI SLI SL
+Softirq B SLI SLI SL SL
+Tasklet A SLI SLI SL SL None
+Tasklet B SLI SLI SL SL SL None
+Timer A SLI SLI SL SL SL SL None
+Timer B SLI SLI SL SL SL SL SL None
+User Context A SLI SLI SLBH SLBH SLBH SLBH SLBH SLBH None
+User Context B SLI SLI SLBH SLBH SLBH SLBH SLBH SLBH MLI None
+============== ============= ============= ========= ========= ========= ========= ======= ======= ============== ==============
+
+Table: Table of Locking Requirements
+
++--------+----------------------------+
+| SLIS | spin_lock_irqsave |
++--------+----------------------------+
+| SLI | spin_lock_irq |
++--------+----------------------------+
+| SL | spin_lock |
++--------+----------------------------+
+| SLBH | spin_lock_bh |
++--------+----------------------------+
+| MLI | mutex_lock_interruptible |
++--------+----------------------------+
+
+Table: Legend for Locking Requirements Table
+
+The trylock Functions
+=====================
+
+There are functions that try to acquire a lock only once and immediately
+return a value telling about success or failure to acquire the lock.
+They can be used if you need no access to the data protected with the
+lock when some other thread is holding the lock. You should acquire the
+lock later if you then need access to the data protected with the lock.
+
+:c:func:`spin_trylock()` does not spin but returns non-zero if it
+acquires the spinlock on the first try or 0 if not. This function can be
+used in all contexts like :c:func:`spin_lock()`: you must have
+disabled the contexts that might interrupt you and acquire the spin
+lock.
+
+:c:func:`mutex_trylock()` does not suspend your task but returns
+non-zero if it could lock the mutex on the first try or 0 if not. This
+function cannot be safely used in hardware or software interrupt
+contexts despite not sleeping.
+
+Common Examples
+===============
+
+Let's step through a simple example: a cache of number to name mappings.
+The cache keeps a count of how often each of the objects is used, and
+when it gets full, throws out the least used one.
+
+All In User Context
+-------------------
+
+For our first example, we assume that all operations are in user context
+(ie. from system calls), so we can sleep. This means we can use a mutex
+to protect the cache and all the objects within it. Here's the code::
+
+ #include <linux/list.h>
+ #include <linux/slab.h>
+ #include <linux/string.h>
+ #include <linux/mutex.h>
+ #include <asm/errno.h>
+
+ struct object
+ {
+ struct list_head list;
+ int id;
+ char name[32];
+ int popularity;
+ };
+
+ /* Protects the cache, cache_num, and the objects within it */
+ static DEFINE_MUTEX(cache_lock);
+ static LIST_HEAD(cache);
+ static unsigned int cache_num = 0;
+ #define MAX_CACHE_SIZE 10
+
+ /* Must be holding cache_lock */
+ static struct object *__cache_find(int id)
+ {
+ struct object *i;
+
+ list_for_each_entry(i, &cache, list)
+ if (i->id == id) {
+ i->popularity++;
+ return i;
+ }
+ return NULL;
+ }
+
+ /* Must be holding cache_lock */
+ static void __cache_delete(struct object *obj)
+ {
+ BUG_ON(!obj);
+ list_del(&obj->list);
+ kfree(obj);
+ cache_num--;
+ }
+
+ /* Must be holding cache_lock */
+ static void __cache_add(struct object *obj)
+ {
+ list_add(&obj->list, &cache);
+ if (++cache_num > MAX_CACHE_SIZE) {
+ struct object *i, *outcast = NULL;
+ list_for_each_entry(i, &cache, list) {
+ if (!outcast || i->popularity < outcast->popularity)
+ outcast = i;
+ }
+ __cache_delete(outcast);
+ }
+ }
+
+ int cache_add(int id, const char *name)
+ {
+ struct object *obj;
+
+ if ((obj = kmalloc(sizeof(*obj), GFP_KERNEL)) == NULL)
+ return -ENOMEM;
+
+ strlcpy(obj->name, name, sizeof(obj->name));
+ obj->id = id;
+ obj->popularity = 0;
+
+ mutex_lock(&cache_lock);
+ __cache_add(obj);
+ mutex_unlock(&cache_lock);
+ return 0;
+ }
+
+ void cache_delete(int id)
+ {
+ mutex_lock(&cache_lock);
+ __cache_delete(__cache_find(id));
+ mutex_unlock(&cache_lock);
+ }
+
+ int cache_find(int id, char *name)
+ {
+ struct object *obj;
+ int ret = -ENOENT;
+
+ mutex_lock(&cache_lock);
+ obj = __cache_find(id);
+ if (obj) {
+ ret = 0;
+ strcpy(name, obj->name);
+ }
+ mutex_unlock(&cache_lock);
+ return ret;
+ }
+
+Note that we always make sure we have the cache_lock when we add,
+delete, or look up the cache: both the cache infrastructure itself and
+the contents of the objects are protected by the lock. In this case it's
+easy, since we copy the data for the user, and never let them access the
+objects directly.
+
+There is a slight (and common) optimization here: in
+:c:func:`cache_add()` we set up the fields of the object before
+grabbing the lock. This is safe, as no-one else can access it until we
+put it in cache.
+
+Accessing From Interrupt Context
+--------------------------------
+
+Now consider the case where :c:func:`cache_find()` can be called
+from interrupt context: either a hardware interrupt or a softirq. An
+example would be a timer which deletes object from the cache.
+
+The change is shown below, in standard patch format: the ``-`` are lines
+which are taken away, and the ``+`` are lines which are added.
+
+::
+
+ --- cache.c.usercontext 2003-12-09 13:58:54.000000000 +1100
+ +++ cache.c.interrupt 2003-12-09 14:07:49.000000000 +1100
+ @@ -12,7 +12,7 @@
+ int popularity;
+ };
+
+ -static DEFINE_MUTEX(cache_lock);
+ +static DEFINE_SPINLOCK(cache_lock);
+ static LIST_HEAD(cache);
+ static unsigned int cache_num = 0;
+ #define MAX_CACHE_SIZE 10
+ @@ -55,6 +55,7 @@
+ int cache_add(int id, const char *name)
+ {
+ struct object *obj;
+ + unsigned long flags;
+
+ if ((obj = kmalloc(sizeof(*obj), GFP_KERNEL)) == NULL)
+ return -ENOMEM;
+ @@ -63,30 +64,33 @@
+ obj->id = id;
+ obj->popularity = 0;
+
+ - mutex_lock(&cache_lock);
+ + spin_lock_irqsave(&cache_lock, flags);
+ __cache_add(obj);
+ - mutex_unlock(&cache_lock);
+ + spin_unlock_irqrestore(&cache_lock, flags);
+ return 0;
+ }
+
+ void cache_delete(int id)
+ {
+ - mutex_lock(&cache_lock);
+ + unsigned long flags;
+ +
+ + spin_lock_irqsave(&cache_lock, flags);
+ __cache_delete(__cache_find(id));
+ - mutex_unlock(&cache_lock);
+ + spin_unlock_irqrestore(&cache_lock, flags);
+ }
+
+ int cache_find(int id, char *name)
+ {
+ struct object *obj;
+ int ret = -ENOENT;
+ + unsigned long flags;
+
+ - mutex_lock(&cache_lock);
+ + spin_lock_irqsave(&cache_lock, flags);
+ obj = __cache_find(id);
+ if (obj) {
+ ret = 0;
+ strcpy(name, obj->name);
+ }
+ - mutex_unlock(&cache_lock);
+ + spin_unlock_irqrestore(&cache_lock, flags);
+ return ret;
+ }
+
+Note that the :c:func:`spin_lock_irqsave()` will turn off
+interrupts if they are on, otherwise does nothing (if we are already in
+an interrupt handler), hence these functions are safe to call from any
+context.
+
+Unfortunately, :c:func:`cache_add()` calls :c:func:`kmalloc()`
+with the ``GFP_KERNEL`` flag, which is only legal in user context. I
+have assumed that :c:func:`cache_add()` is still only called in
+user context, otherwise this should become a parameter to
+:c:func:`cache_add()`.
+
+Exposing Objects Outside This File
+----------------------------------
+
+If our objects contained more information, it might not be sufficient to
+copy the information in and out: other parts of the code might want to
+keep pointers to these objects, for example, rather than looking up the
+id every time. This produces two problems.
+
+The first problem is that we use the ``cache_lock`` to protect objects:
+we'd need to make this non-static so the rest of the code can use it.
+This makes locking trickier, as it is no longer all in one place.
+
+The second problem is the lifetime problem: if another structure keeps a
+pointer to an object, it presumably expects that pointer to remain
+valid. Unfortunately, this is only guaranteed while you hold the lock,
+otherwise someone might call :c:func:`cache_delete()` and even
+worse, add another object, re-using the same address.
+
+As there is only one lock, you can't hold it forever: no-one else would
+get any work done.
+
+The solution to this problem is to use a reference count: everyone who
+has a pointer to the object increases it when they first get the object,
+and drops the reference count when they're finished with it. Whoever
+drops it to zero knows it is unused, and can actually delete it.
+
+Here is the code::
+
+ --- cache.c.interrupt 2003-12-09 14:25:43.000000000 +1100
+ +++ cache.c.refcnt 2003-12-09 14:33:05.000000000 +1100
+ @@ -7,6 +7,7 @@
+ struct object
+ {
+ struct list_head list;
+ + unsigned int refcnt;
+ int id;
+ char name[32];
+ int popularity;
+ @@ -17,6 +18,35 @@
+ static unsigned int cache_num = 0;
+ #define MAX_CACHE_SIZE 10
+
+ +static void __object_put(struct object *obj)
+ +{
+ + if (--obj->refcnt == 0)
+ + kfree(obj);
+ +}
+ +
+ +static void __object_get(struct object *obj)
+ +{
+ + obj->refcnt++;
+ +}
+ +
+ +void object_put(struct object *obj)
+ +{
+ + unsigned long flags;
+ +
+ + spin_lock_irqsave(&cache_lock, flags);
+ + __object_put(obj);
+ + spin_unlock_irqrestore(&cache_lock, flags);
+ +}
+ +
+ +void object_get(struct object *obj)
+ +{
+ + unsigned long flags;
+ +
+ + spin_lock_irqsave(&cache_lock, flags);
+ + __object_get(obj);
+ + spin_unlock_irqrestore(&cache_lock, flags);
+ +}
+ +
+ /* Must be holding cache_lock */
+ static struct object *__cache_find(int id)
+ {
+ @@ -35,6 +65,7 @@
+ {
+ BUG_ON(!obj);
+ list_del(&obj->list);
+ + __object_put(obj);
+ cache_num--;
+ }
+
+ @@ -63,6 +94,7 @@
+ strlcpy(obj->name, name, sizeof(obj->name));
+ obj->id = id;
+ obj->popularity = 0;
+ + obj->refcnt = 1; /* The cache holds a reference */
+
+ spin_lock_irqsave(&cache_lock, flags);
+ __cache_add(obj);
+ @@ -79,18 +111,15 @@
+ spin_unlock_irqrestore(&cache_lock, flags);
+ }
+
+ -int cache_find(int id, char *name)
+ +struct object *cache_find(int id)
+ {
+ struct object *obj;
+ - int ret = -ENOENT;
+ unsigned long flags;
+
+ spin_lock_irqsave(&cache_lock, flags);
+ obj = __cache_find(id);
+ - if (obj) {
+ - ret = 0;
+ - strcpy(name, obj->name);
+ - }
+ + if (obj)
+ + __object_get(obj);
+ spin_unlock_irqrestore(&cache_lock, flags);
+ - return ret;
+ + return obj;
+ }
+
+We encapsulate the reference counting in the standard 'get' and 'put'
+functions. Now we can return the object itself from
+:c:func:`cache_find()` which has the advantage that the user can
+now sleep holding the object (eg. to :c:func:`copy_to_user()` to
+name to userspace).
+
+The other point to note is that I said a reference should be held for
+every pointer to the object: thus the reference count is 1 when first
+inserted into the cache. In some versions the framework does not hold a
+reference count, but they are more complicated.
+
+Using Atomic Operations For The Reference Count
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In practice, :c:type:`atomic_t` would usually be used for refcnt. There are a
+number of atomic operations defined in ``include/asm/atomic.h``: these
+are guaranteed to be seen atomically from all CPUs in the system, so no
+lock is required. In this case, it is simpler than using spinlocks,
+although for anything non-trivial using spinlocks is clearer. The
+:c:func:`atomic_inc()` and :c:func:`atomic_dec_and_test()`
+are used instead of the standard increment and decrement operators, and
+the lock is no longer used to protect the reference count itself.
+
+::
+
+ --- cache.c.refcnt 2003-12-09 15:00:35.000000000 +1100
+ +++ cache.c.refcnt-atomic 2003-12-11 15:49:42.000000000 +1100
+ @@ -7,7 +7,7 @@
+ struct object
+ {
+ struct list_head list;
+ - unsigned int refcnt;
+ + atomic_t refcnt;
+ int id;
+ char name[32];
+ int popularity;
+ @@ -18,33 +18,15 @@
+ static unsigned int cache_num = 0;
+ #define MAX_CACHE_SIZE 10
+
+ -static void __object_put(struct object *obj)
+ -{
+ - if (--obj->refcnt == 0)
+ - kfree(obj);
+ -}
+ -
+ -static void __object_get(struct object *obj)
+ -{
+ - obj->refcnt++;
+ -}
+ -
+ void object_put(struct object *obj)
+ {
+ - unsigned long flags;
+ -
+ - spin_lock_irqsave(&cache_lock, flags);
+ - __object_put(obj);
+ - spin_unlock_irqrestore(&cache_lock, flags);
+ + if (atomic_dec_and_test(&obj->refcnt))
+ + kfree(obj);
+ }
+
+ void object_get(struct object *obj)
+ {
+ - unsigned long flags;
+ -
+ - spin_lock_irqsave(&cache_lock, flags);
+ - __object_get(obj);
+ - spin_unlock_irqrestore(&cache_lock, flags);
+ + atomic_inc(&obj->refcnt);
+ }
+
+ /* Must be holding cache_lock */
+ @@ -65,7 +47,7 @@
+ {
+ BUG_ON(!obj);
+ list_del(&obj->list);
+ - __object_put(obj);
+ + object_put(obj);
+ cache_num--;
+ }
+
+ @@ -94,7 +76,7 @@
+ strlcpy(obj->name, name, sizeof(obj->name));
+ obj->id = id;
+ obj->popularity = 0;
+ - obj->refcnt = 1; /* The cache holds a reference */
+ + atomic_set(&obj->refcnt, 1); /* The cache holds a reference */
+
+ spin_lock_irqsave(&cache_lock, flags);
+ __cache_add(obj);
+ @@ -119,7 +101,7 @@
+ spin_lock_irqsave(&cache_lock, flags);
+ obj = __cache_find(id);
+ if (obj)
+ - __object_get(obj);
+ + object_get(obj);
+ spin_unlock_irqrestore(&cache_lock, flags);
+ return obj;
+ }
+
+Protecting The Objects Themselves
+---------------------------------
+
+In these examples, we assumed that the objects (except the reference
+counts) never changed once they are created. If we wanted to allow the
+name to change, there are three possibilities:
+
+- You can make ``cache_lock`` non-static, and tell people to grab that
+ lock before changing the name in any object.
+
+- You can provide a :c:func:`cache_obj_rename()` which grabs this
+ lock and changes the name for the caller, and tell everyone to use
+ that function.
+
+- You can make the ``cache_lock`` protect only the cache itself, and
+ use another lock to protect the name.
+
+Theoretically, you can make the locks as fine-grained as one lock for
+every field, for every object. In practice, the most common variants
+are:
+
+- One lock which protects the infrastructure (the ``cache`` list in
+ this example) and all the objects. This is what we have done so far.
+
+- One lock which protects the infrastructure (including the list
+ pointers inside the objects), and one lock inside the object which
+ protects the rest of that object.
+
+- Multiple locks to protect the infrastructure (eg. one lock per hash
+ chain), possibly with a separate per-object lock.
+
+Here is the "lock-per-object" implementation:
+
+::
+
+ --- cache.c.refcnt-atomic 2003-12-11 15:50:54.000000000 +1100
+ +++ cache.c.perobjectlock 2003-12-11 17:15:03.000000000 +1100
+ @@ -6,11 +6,17 @@
+
+ struct object
+ {
+ + /* These two protected by cache_lock. */
+ struct list_head list;
+ + int popularity;
+ +
+ atomic_t refcnt;
+ +
+ + /* Doesn't change once created. */
+ int id;
+ +
+ + spinlock_t lock; /* Protects the name */
+ char name[32];
+ - int popularity;
+ };
+
+ static DEFINE_SPINLOCK(cache_lock);
+ @@ -77,6 +84,7 @@
+ obj->id = id;
+ obj->popularity = 0;
+ atomic_set(&obj->refcnt, 1); /* The cache holds a reference */
+ + spin_lock_init(&obj->lock);
+
+ spin_lock_irqsave(&cache_lock, flags);
+ __cache_add(obj);
+
+Note that I decide that the popularity count should be protected by the
+``cache_lock`` rather than the per-object lock: this is because it (like
+the :c:type:`struct list_head <list_head>` inside the object)
+is logically part of the infrastructure. This way, I don't need to grab
+the lock of every object in :c:func:`__cache_add()` when seeking
+the least popular.
+
+I also decided that the id member is unchangeable, so I don't need to
+grab each object lock in :c:func:`__cache_find()` to examine the
+id: the object lock is only used by a caller who wants to read or write
+the name field.
+
+Note also that I added a comment describing what data was protected by
+which locks. This is extremely important, as it describes the runtime
+behavior of the code, and can be hard to gain from just reading. And as
+Alan Cox says, “Lock data, not code”.
+
+Common Problems
+===============
+
+Deadlock: Simple and Advanced
+-----------------------------
+
+There is a coding bug where a piece of code tries to grab a spinlock
+twice: it will spin forever, waiting for the lock to be released
+(spinlocks, rwlocks and mutexes are not recursive in Linux). This is
+trivial to diagnose: not a
+stay-up-five-nights-talk-to-fluffy-code-bunnies kind of problem.
+
+For a slightly more complex case, imagine you have a region shared by a
+softirq and user context. If you use a :c:func:`spin_lock()` call
+to protect it, it is possible that the user context will be interrupted
+by the softirq while it holds the lock, and the softirq will then spin
+forever trying to get the same lock.
+
+Both of these are called deadlock, and as shown above, it can occur even
+with a single CPU (although not on UP compiles, since spinlocks vanish
+on kernel compiles with ``CONFIG_SMP``\ =n. You'll still get data
+corruption in the second example).
+
+This complete lockup is easy to diagnose: on SMP boxes the watchdog
+timer or compiling with ``DEBUG_SPINLOCK`` set
+(``include/linux/spinlock.h``) will show this up immediately when it
+happens.
+
+A more complex problem is the so-called 'deadly embrace', involving two
+or more locks. Say you have a hash table: each entry in the table is a
+spinlock, and a chain of hashed objects. Inside a softirq handler, you
+sometimes want to alter an object from one place in the hash to another:
+you grab the spinlock of the old hash chain and the spinlock of the new
+hash chain, and delete the object from the old one, and insert it in the
+new one.
+
+There are two problems here. First, if your code ever tries to move the
+object to the same chain, it will deadlock with itself as it tries to
+lock it twice. Secondly, if the same softirq on another CPU is trying to
+move another object in the reverse direction, the following could
+happen:
+
++-----------------------+-----------------------+
+| CPU 1 | CPU 2 |
++=======================+=======================+
+| Grab lock A -> OK | Grab lock B -> OK |
++-----------------------+-----------------------+
+| Grab lock B -> spin | Grab lock A -> spin |
++-----------------------+-----------------------+
+
+Table: Consequences
+
+The two CPUs will spin forever, waiting for the other to give up their
+lock. It will look, smell, and feel like a crash.
+
+Preventing Deadlock
+-------------------
+
+Textbooks will tell you that if you always lock in the same order, you
+will never get this kind of deadlock. Practice will tell you that this
+approach doesn't scale: when I create a new lock, I don't understand
+enough of the kernel to figure out where in the 5000 lock hierarchy it
+will fit.
+
+The best locks are encapsulated: they never get exposed in headers, and
+are never held around calls to non-trivial functions outside the same
+file. You can read through this code and see that it will never
+deadlock, because it never tries to grab another lock while it has that
+one. People using your code don't even need to know you are using a
+lock.
+
+A classic problem here is when you provide callbacks or hooks: if you
+call these with the lock held, you risk simple deadlock, or a deadly
+embrace (who knows what the callback will do?). Remember, the other
+programmers are out to get you, so don't do this.
+
+Overzealous Prevention Of Deadlocks
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Deadlocks are problematic, but not as bad as data corruption. Code which
+grabs a read lock, searches a list, fails to find what it wants, drops
+the read lock, grabs a write lock and inserts the object has a race
+condition.
+
+If you don't see why, please stay the fuck away from my code.
+
+Racing Timers: A Kernel Pastime
+-------------------------------
+
+Timers can produce their own special problems with races. Consider a
+collection of objects (list, hash, etc) where each object has a timer
+which is due to destroy it.
+
+If you want to destroy the entire collection (say on module removal),
+you might do the following::
+
+ /* THIS CODE BAD BAD BAD BAD: IF IT WAS ANY WORSE IT WOULD USE
+ HUNGARIAN NOTATION */
+ spin_lock_bh(&list_lock);
+
+ while (list) {
+ struct foo *next = list->next;
+ del_timer(&list->timer);
+ kfree(list);
+ list = next;
+ }
+
+ spin_unlock_bh(&list_lock);
+
+
+Sooner or later, this will crash on SMP, because a timer can have just
+gone off before the :c:func:`spin_lock_bh()`, and it will only get
+the lock after we :c:func:`spin_unlock_bh()`, and then try to free
+the element (which has already been freed!).
+
+This can be avoided by checking the result of
+:c:func:`del_timer()`: if it returns 1, the timer has been deleted.
+If 0, it means (in this case) that it is currently running, so we can
+do::
+
+ retry:
+ spin_lock_bh(&list_lock);
+
+ while (list) {
+ struct foo *next = list->next;
+ if (!del_timer(&list->timer)) {
+ /* Give timer a chance to delete this */
+ spin_unlock_bh(&list_lock);
+ goto retry;
+ }
+ kfree(list);
+ list = next;
+ }
+
+ spin_unlock_bh(&list_lock);
+
+
+Another common problem is deleting timers which restart themselves (by
+calling :c:func:`add_timer()` at the end of their timer function).
+Because this is a fairly common case which is prone to races, you should
+use :c:func:`del_timer_sync()` (``include/linux/timer.h``) to
+handle this case. It returns the number of times the timer had to be
+deleted before we finally stopped it from adding itself back in.
+
+Locking Speed
+=============
+
+There are three main things to worry about when considering speed of
+some code which does locking. First is concurrency: how many things are
+going to be waiting while someone else is holding a lock. Second is the
+time taken to actually acquire and release an uncontended lock. Third is
+using fewer, or smarter locks. I'm assuming that the lock is used fairly
+often: otherwise, you wouldn't be concerned about efficiency.
+
+Concurrency depends on how long the lock is usually held: you should
+hold the lock for as long as needed, but no longer. In the cache
+example, we always create the object without the lock held, and then
+grab the lock only when we are ready to insert it in the list.
+
+Acquisition times depend on how much damage the lock operations do to
+the pipeline (pipeline stalls) and how likely it is that this CPU was
+the last one to grab the lock (ie. is the lock cache-hot for this CPU):
+on a machine with more CPUs, this likelihood drops fast. Consider a
+700MHz Intel Pentium III: an instruction takes about 0.7ns, an atomic
+increment takes about 58ns, a lock which is cache-hot on this CPU takes
+160ns, and a cacheline transfer from another CPU takes an additional 170
+to 360ns. (These figures from Paul McKenney's `Linux Journal RCU
+article <http://www.linuxjournal.com/article.php?sid=6993>`__).
+
+These two aims conflict: holding a lock for a short time might be done
+by splitting locks into parts (such as in our final per-object-lock
+example), but this increases the number of lock acquisitions, and the
+results are often slower than having a single lock. This is another
+reason to advocate locking simplicity.
+
+The third concern is addressed below: there are some methods to reduce
+the amount of locking which needs to be done.
+
+Read/Write Lock Variants
+------------------------
+
+Both spinlocks and mutexes have read/write variants: ``rwlock_t`` and
+:c:type:`struct rw_semaphore <rw_semaphore>`. These divide
+users into two classes: the readers and the writers. If you are only
+reading the data, you can get a read lock, but to write to the data you
+need the write lock. Many people can hold a read lock, but a writer must
+be sole holder.
+
+If your code divides neatly along reader/writer lines (as our cache code
+does), and the lock is held by readers for significant lengths of time,
+using these locks can help. They are slightly slower than the normal
+locks though, so in practice ``rwlock_t`` is not usually worthwhile.
+
+Avoiding Locks: Read Copy Update
+--------------------------------
+
+There is a special method of read/write locking called Read Copy Update.
+Using RCU, the readers can avoid taking a lock altogether: as we expect
+our cache to be read more often than updated (otherwise the cache is a
+waste of time), it is a candidate for this optimization.
+
+How do we get rid of read locks? Getting rid of read locks means that
+writers may be changing the list underneath the readers. That is
+actually quite simple: we can read a linked list while an element is
+being added if the writer adds the element very carefully. For example,
+adding ``new`` to a single linked list called ``list``::
+
+ new->next = list->next;
+ wmb();
+ list->next = new;
+
+
+The :c:func:`wmb()` is a write memory barrier. It ensures that the
+first operation (setting the new element's ``next`` pointer) is complete
+and will be seen by all CPUs, before the second operation is (putting
+the new element into the list). This is important, since modern
+compilers and modern CPUs can both reorder instructions unless told
+otherwise: we want a reader to either not see the new element at all, or
+see the new element with the ``next`` pointer correctly pointing at the
+rest of the list.
+
+Fortunately, there is a function to do this for standard
+:c:type:`struct list_head <list_head>` lists:
+:c:func:`list_add_rcu()` (``include/linux/list.h``).
+
+Removing an element from the list is even simpler: we replace the
+pointer to the old element with a pointer to its successor, and readers
+will either see it, or skip over it.
+
+::
+
+ list->next = old->next;
+
+
+There is :c:func:`list_del_rcu()` (``include/linux/list.h``) which
+does this (the normal version poisons the old object, which we don't
+want).
+
+The reader must also be careful: some CPUs can look through the ``next``
+pointer to start reading the contents of the next element early, but
+don't realize that the pre-fetched contents is wrong when the ``next``
+pointer changes underneath them. Once again, there is a
+:c:func:`list_for_each_entry_rcu()` (``include/linux/list.h``)
+to help you. Of course, writers can just use
+:c:func:`list_for_each_entry()`, since there cannot be two
+simultaneous writers.
+
+Our final dilemma is this: when can we actually destroy the removed
+element? Remember, a reader might be stepping through this element in
+the list right now: if we free this element and the ``next`` pointer
+changes, the reader will jump off into garbage and crash. We need to
+wait until we know that all the readers who were traversing the list
+when we deleted the element are finished. We use
+:c:func:`call_rcu()` to register a callback which will actually
+destroy the object once all pre-existing readers are finished.
+Alternatively, :c:func:`synchronize_rcu()` may be used to block
+until all pre-existing are finished.
+
+But how does Read Copy Update know when the readers are finished? The
+method is this: firstly, the readers always traverse the list inside
+:c:func:`rcu_read_lock()`/:c:func:`rcu_read_unlock()` pairs:
+these simply disable preemption so the reader won't go to sleep while
+reading the list.
+
+RCU then waits until every other CPU has slept at least once: since
+readers cannot sleep, we know that any readers which were traversing the
+list during the deletion are finished, and the callback is triggered.
+The real Read Copy Update code is a little more optimized than this, but
+this is the fundamental idea.
+
+::
+
+ --- cache.c.perobjectlock 2003-12-11 17:15:03.000000000 +1100
+ +++ cache.c.rcupdate 2003-12-11 17:55:14.000000000 +1100
+ @@ -1,15 +1,18 @@
+ #include <linux/list.h>
+ #include <linux/slab.h>
+ #include <linux/string.h>
+ +#include <linux/rcupdate.h>
+ #include <linux/mutex.h>
+ #include <asm/errno.h>
+
+ struct object
+ {
+ - /* These two protected by cache_lock. */
+ + /* This is protected by RCU */
+ struct list_head list;
+ int popularity;
+
+ + struct rcu_head rcu;
+ +
+ atomic_t refcnt;
+
+ /* Doesn't change once created. */
+ @@ -40,7 +43,7 @@
+ {
+ struct object *i;
+
+ - list_for_each_entry(i, &cache, list) {
+ + list_for_each_entry_rcu(i, &cache, list) {
+ if (i->id == id) {
+ i->popularity++;
+ return i;
+ @@ -49,19 +52,25 @@
+ return NULL;
+ }
+
+ +/* Final discard done once we know no readers are looking. */
+ +static void cache_delete_rcu(void *arg)
+ +{
+ + object_put(arg);
+ +}
+ +
+ /* Must be holding cache_lock */
+ static void __cache_delete(struct object *obj)
+ {
+ BUG_ON(!obj);
+ - list_del(&obj->list);
+ - object_put(obj);
+ + list_del_rcu(&obj->list);
+ cache_num--;
+ + call_rcu(&obj->rcu, cache_delete_rcu);
+ }
+
+ /* Must be holding cache_lock */
+ static void __cache_add(struct object *obj)
+ {
+ - list_add(&obj->list, &cache);
+ + list_add_rcu(&obj->list, &cache);
+ if (++cache_num > MAX_CACHE_SIZE) {
+ struct object *i, *outcast = NULL;
+ list_for_each_entry(i, &cache, list) {
+ @@ -104,12 +114,11 @@
+ struct object *cache_find(int id)
+ {
+ struct object *obj;
+ - unsigned long flags;
+
+ - spin_lock_irqsave(&cache_lock, flags);
+ + rcu_read_lock();
+ obj = __cache_find(id);
+ if (obj)
+ object_get(obj);
+ - spin_unlock_irqrestore(&cache_lock, flags);
+ + rcu_read_unlock();
+ return obj;
+ }
+
+Note that the reader will alter the popularity member in
+:c:func:`__cache_find()`, and now it doesn't hold a lock. One
+solution would be to make it an ``atomic_t``, but for this usage, we
+don't really care about races: an approximate result is good enough, so
+I didn't change it.
+
+The result is that :c:func:`cache_find()` requires no
+synchronization with any other functions, so is almost as fast on SMP as
+it would be on UP.
+
+There is a further optimization possible here: remember our original
+cache code, where there were no reference counts and the caller simply
+held the lock whenever using the object? This is still possible: if you
+hold the lock, no one can delete the object, so you don't need to get
+and put the reference count.
+
+Now, because the 'read lock' in RCU is simply disabling preemption, a
+caller which always has preemption disabled between calling
+:c:func:`cache_find()` and :c:func:`object_put()` does not
+need to actually get and put the reference count: we could expose
+:c:func:`__cache_find()` by making it non-static, and such
+callers could simply call that.
+
+The benefit here is that the reference count is not written to: the
+object is not altered in any way, which is much faster on SMP machines
+due to caching.
+
+Per-CPU Data
+------------
+
+Another technique for avoiding locking which is used fairly widely is to
+duplicate information for each CPU. For example, if you wanted to keep a
+count of a common condition, you could use a spin lock and a single
+counter. Nice and simple.
+
+If that was too slow (it's usually not, but if you've got a really big
+machine to test on and can show that it is), you could instead use a
+counter for each CPU, then none of them need an exclusive lock. See
+:c:func:`DEFINE_PER_CPU()`, :c:func:`get_cpu_var()` and
+:c:func:`put_cpu_var()` (``include/linux/percpu.h``).
+
+Of particular use for simple per-cpu counters is the ``local_t`` type,
+and the :c:func:`cpu_local_inc()` and related functions, which are
+more efficient than simple code on some architectures
+(``include/asm/local.h``).
+
+Note that there is no simple, reliable way of getting an exact value of
+such a counter, without introducing more locks. This is not a problem
+for some uses.
+
+Data Which Mostly Used By An IRQ Handler
+----------------------------------------
+
+If data is always accessed from within the same IRQ handler, you don't
+need a lock at all: the kernel already guarantees that the irq handler
+will not run simultaneously on multiple CPUs.
+
+Manfred Spraul points out that you can still do this, even if the data
+is very occasionally accessed in user context or softirqs/tasklets. The
+irq handler doesn't use a lock, and all other accesses are done as so::
+
+ spin_lock(&lock);
+ disable_irq(irq);
+ ...
+ enable_irq(irq);
+ spin_unlock(&lock);
+
+The :c:func:`disable_irq()` prevents the irq handler from running
+(and waits for it to finish if it's currently running on other CPUs).
+The spinlock prevents any other accesses happening at the same time.
+Naturally, this is slower than just a :c:func:`spin_lock_irq()`
+call, so it only makes sense if this type of access happens extremely
+rarely.
+
+What Functions Are Safe To Call From Interrupts?
+================================================
+
+Many functions in the kernel sleep (ie. call schedule()) directly or
+indirectly: you can never call them while holding a spinlock, or with
+preemption disabled. This also means you need to be in user context:
+calling them from an interrupt is illegal.
+
+Some Functions Which Sleep
+--------------------------
+
+The most common ones are listed below, but you usually have to read the
+code to find out if other calls are safe. If everyone else who calls it
+can sleep, you probably need to be able to sleep, too. In particular,
+registration and deregistration functions usually expect to be called
+from user context, and can sleep.
+
+- Accesses to userspace:
+
+ - :c:func:`copy_from_user()`
+
+ - :c:func:`copy_to_user()`
+
+ - :c:func:`get_user()`
+
+ - :c:func:`put_user()`
+
+- :c:func:`kmalloc(GFP_KERNEL) <kmalloc>`
+
+- :c:func:`mutex_lock_interruptible()` and
+ :c:func:`mutex_lock()`
+
+ There is a :c:func:`mutex_trylock()` which does not sleep.
+ Still, it must not be used inside interrupt context since its
+ implementation is not safe for that. :c:func:`mutex_unlock()`
+ will also never sleep. It cannot be used in interrupt context either
+ since a mutex must be released by the same task that acquired it.
+
+Some Functions Which Don't Sleep
+--------------------------------
+
+Some functions are safe to call from any context, or holding almost any
+lock.
+
+- :c:func:`printk()`
+
+- :c:func:`kfree()`
+
+- :c:func:`add_timer()` and :c:func:`del_timer()`
+
+Mutex API reference
+===================
+
+.. kernel-doc:: include/linux/mutex.h
+ :internal:
+
+.. kernel-doc:: kernel/locking/mutex.c
+ :export:
+
+Futex API reference
+===================
+
+.. kernel-doc:: kernel/futex.c
+ :internal:
+
+Further reading
+===============
+
+- ``Documentation/locking/spinlocks.txt``: Linus Torvalds' spinlocking
+ tutorial in the kernel sources.
+
+- Unix Systems for Modern Architectures: Symmetric Multiprocessing and
+ Caching for Kernel Programmers:
+
+ Curt Schimmel's very good introduction to kernel level locking (not
+ written for Linux, but nearly everything applies). The book is
+ expensive, but really worth every penny to understand SMP locking.
+ [ISBN: 0201633388]
+
+Thanks
+======
+
+Thanks to Telsa Gwynne for DocBooking, neatening and adding style.
+
+Thanks to Martin Pool, Philipp Rumpf, Stephen Rothwell, Paul Mackerras,
+Ruedi Aschwanden, Alan Cox, Manfred Spraul, Tim Waugh, Pete Zaitcev,
+James Morris, Robert Love, Paul McKenney, John Ashby for proofreading,
+correcting, flaming, commenting.
+
+Thanks to the cabal for having no influence on this document.
+
+Glossary
+========
+
+preemption
+ Prior to 2.5, or when ``CONFIG_PREEMPT`` is unset, processes in user
+ context inside the kernel would not preempt each other (ie. you had that
+ CPU until you gave it up, except for interrupts). With the addition of
+ ``CONFIG_PREEMPT`` in 2.5.4, this changed: when in user context, higher
+ priority tasks can "cut in": spinlocks were changed to disable
+ preemption, even on UP.
+
+bh
+ Bottom Half: for historical reasons, functions with '_bh' in them often
+ now refer to any software interrupt, e.g. :c:func:`spin_lock_bh()`
+ blocks any software interrupt on the current CPU. Bottom halves are
+ deprecated, and will eventually be replaced by tasklets. Only one bottom
+ half will be running at any time.
+
+Hardware Interrupt / Hardware IRQ
+ Hardware interrupt request. :c:func:`in_irq()` returns true in a
+ hardware interrupt handler.
+
+Interrupt Context
+ Not user context: processing a hardware irq or software irq. Indicated
+ by the :c:func:`in_interrupt()` macro returning true.
+
+SMP
+ Symmetric Multi-Processor: kernels compiled for multiple-CPU machines.
+ (``CONFIG_SMP=y``).
+
+Software Interrupt / softirq
+ Software interrupt handler. :c:func:`in_irq()` returns false;
+ :c:func:`in_softirq()` returns true. Tasklets and softirqs both
+ fall into the category of 'software interrupts'.
+
+ Strictly speaking a softirq is one of up to 32 enumerated software
+ interrupts which can run on multiple CPUs at once. Sometimes used to
+ refer to tasklets as well (ie. all software interrupts).
+
+tasklet
+ A dynamically-registrable software interrupt, which is guaranteed to
+ only run on one CPU at a time.
+
+timer
+ A dynamically-registrable software interrupt, which is run at (or close
+ to) a given time. When running, it is just like a tasklet (in fact, they
+ are called from the ``TIMER_SOFTIRQ``).
+
+UP
+ Uni-Processor: Non-SMP. (``CONFIG_SMP=n``).
+
+User Context
+ The kernel executing on behalf of a particular process (ie. a system
+ call or trap) or kernel thread. You can tell which process with the
+ ``current`` macro.) Not to be confused with userspace. Can be
+ interrupted by software or hardware interrupts.
+
+Userspace
+ A process executing its own code outside the kernel.
on that CPU. If a thread that expects to run on the de-jittered
CPU awakens, the scheduler will send an IPI that can result in
a subsequent SCHED_SOFTIRQ.
-2. Build with CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_ALL=y,
- CONFIG_NO_HZ_FULL=y, and, in addition, ensure that the CPU
- to be de-jittered is marked as an adaptive-ticks CPU using the
- "nohz_full=" boot parameter. This reduces the number of
- scheduler-clock interrupts that the de-jittered CPU receives,
- minimizing its chances of being selected to do the load balancing
- work that runs in SCHED_SOFTIRQ context.
+2. CONFIG_NO_HZ_FULL=y and ensure that the CPU to be de-jittered
+ is marked as an adaptive-ticks CPU using the "nohz_full="
+ boot parameter. This reduces the number of scheduler-clock
+ interrupts that the de-jittered CPU receives, minimizing its
+ chances of being selected to do the load balancing work that
+ runs in SCHED_SOFTIRQ context.
3. To the extent possible, keep the CPU out of the kernel when it
is non-idle, for example, by avoiding system calls and by
forcing both kernel threads and interrupts to execute elsewhere.
RCU_SOFTIRQ: Do at least one of the following:
1. Offload callbacks and keep the CPU in either dyntick-idle or
adaptive-ticks state by doing all of the following:
- a. Build with CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_ALL=y,
- CONFIG_NO_HZ_FULL=y, and, in addition ensure that the CPU
- to be de-jittered is marked as an adaptive-ticks CPU using
- the "nohz_full=" boot parameter. Bind the rcuo kthreads
- to housekeeping CPUs, which can tolerate OS jitter.
+ a. CONFIG_NO_HZ_FULL=y and ensure that the CPU to be
+ de-jittered is marked as an adaptive-ticks CPU using the
+ "nohz_full=" boot parameter. Bind the rcuo kthreads to
+ housekeeping CPUs, which can tolerate OS jitter.
b. To the extent possible, keep the CPU out of the kernel
when it is non-idle, for example, by avoiding system
calls and by forcing both kernel threads and interrupts
is feasible only if your workload never requires RCU priority
boosting, for example, if you ensure frequent idle time on all
CPUs that might execute within the kernel.
-3. Build with CONFIG_RCU_NOCB_CPU=y and CONFIG_RCU_NOCB_CPU_ALL=y,
- which offloads all RCU callbacks to kthreads that can be moved
- off of CPUs susceptible to OS jitter. This approach prevents the
- rcuc/%u kthreads from having any work to do, so that they are
- never awakened.
+3. Build with CONFIG_RCU_NOCB_CPU=y and boot with the rcu_nocbs=
+ boot parameter offloading RCU callbacks from all CPUs susceptible
+ to OS jitter. This approach prevents the rcuc/%u kthreads from
+ having any work to do, so that they are never awakened.
4. Ensure that the CPU never enters the kernel, and, in particular,
avoid initiating any CPU hotplug operations on this CPU. This is
another way of preventing any callbacks from being queued on the
--- /dev/null
+========================================================
+Linux Security Modules: General Security Hooks for Linux
+========================================================
+
+:Author: Stephen Smalley
+:Author: Timothy Fraser
+:Author: Chris Vance
+
+.. note::
+
+ The APIs described in this book are outdated.
+
+Introduction
+============
+
+In March 2001, the National Security Agency (NSA) gave a presentation
+about Security-Enhanced Linux (SELinux) at the 2.5 Linux Kernel Summit.
+SELinux is an implementation of flexible and fine-grained
+nondiscretionary access controls in the Linux kernel, originally
+implemented as its own particular kernel patch. Several other security
+projects (e.g. RSBAC, Medusa) have also developed flexible access
+control architectures for the Linux kernel, and various projects have
+developed particular access control models for Linux (e.g. LIDS, DTE,
+SubDomain). Each project has developed and maintained its own kernel
+patch to support its security needs.
+
+In response to the NSA presentation, Linus Torvalds made a set of
+remarks that described a security framework he would be willing to
+consider for inclusion in the mainstream Linux kernel. He described a
+general framework that would provide a set of security hooks to control
+operations on kernel objects and a set of opaque security fields in
+kernel data structures for maintaining security attributes. This
+framework could then be used by loadable kernel modules to implement any
+desired model of security. Linus also suggested the possibility of
+migrating the Linux capabilities code into such a module.
+
+The Linux Security Modules (LSM) project was started by WireX to develop
+such a framework. LSM is a joint development effort by several security
+projects, including Immunix, SELinux, SGI and Janus, and several
+individuals, including Greg Kroah-Hartman and James Morris, to develop a
+Linux kernel patch that implements this framework. The patch is
+currently tracking the 2.4 series and is targeted for integration into
+the 2.5 development series. This technical report provides an overview
+of the framework and the example capabilities security module provided
+by the LSM kernel patch.
+
+LSM Framework
+=============
+
+The LSM kernel patch provides a general kernel framework to support
+security modules. In particular, the LSM framework is primarily focused
+on supporting access control modules, although future development is
+likely to address other security needs such as auditing. By itself, the
+framework does not provide any additional security; it merely provides
+the infrastructure to support security modules. The LSM kernel patch
+also moves most of the capabilities logic into an optional security
+module, with the system defaulting to the traditional superuser logic.
+This capabilities module is discussed further in
+`LSM Capabilities Module <#cap>`__.
+
+The LSM kernel patch adds security fields to kernel data structures and
+inserts calls to hook functions at critical points in the kernel code to
+manage the security fields and to perform access control. It also adds
+functions for registering and unregistering security modules, and adds a
+general :c:func:`security()` system call to support new system calls
+for security-aware applications.
+
+The LSM security fields are simply ``void*`` pointers. For process and
+program execution security information, security fields were added to
+:c:type:`struct task_struct <task_struct>` and
+:c:type:`struct linux_binprm <linux_binprm>`. For filesystem
+security information, a security field was added to :c:type:`struct
+super_block <super_block>`. For pipe, file, and socket security
+information, security fields were added to :c:type:`struct inode
+<inode>` and :c:type:`struct file <file>`. For packet and
+network device security information, security fields were added to
+:c:type:`struct sk_buff <sk_buff>` and :c:type:`struct
+net_device <net_device>`. For System V IPC security information,
+security fields were added to :c:type:`struct kern_ipc_perm
+<kern_ipc_perm>` and :c:type:`struct msg_msg
+<msg_msg>`; additionally, the definitions for :c:type:`struct
+msg_msg <msg_msg>`, struct msg_queue, and struct shmid_kernel
+were moved to header files (``include/linux/msg.h`` and
+``include/linux/shm.h`` as appropriate) to allow the security modules to
+use these definitions.
+
+Each LSM hook is a function pointer in a global table, security_ops.
+This table is a :c:type:`struct security_operations
+<security_operations>` structure as defined by
+``include/linux/security.h``. Detailed documentation for each hook is
+included in this header file. At present, this structure consists of a
+collection of substructures that group related hooks based on the kernel
+object (e.g. task, inode, file, sk_buff, etc) as well as some top-level
+hook function pointers for system operations. This structure is likely
+to be flattened in the future for performance. The placement of the hook
+calls in the kernel code is described by the "called:" lines in the
+per-hook documentation in the header file. The hook calls can also be
+easily found in the kernel code by looking for the string
+"security_ops->".
+
+Linus mentioned per-process security hooks in his original remarks as a
+possible alternative to global security hooks. However, if LSM were to
+start from the perspective of per-process hooks, then the base framework
+would have to deal with how to handle operations that involve multiple
+processes (e.g. kill), since each process might have its own hook for
+controlling the operation. This would require a general mechanism for
+composing hooks in the base framework. Additionally, LSM would still
+need global hooks for operations that have no process context (e.g.
+network input operations). Consequently, LSM provides global security
+hooks, but a security module is free to implement per-process hooks
+(where that makes sense) by storing a security_ops table in each
+process' security field and then invoking these per-process hooks from
+the global hooks. The problem of composition is thus deferred to the
+module.
+
+The global security_ops table is initialized to a set of hook functions
+provided by a dummy security module that provides traditional superuser
+logic. A :c:func:`register_security()` function (in
+``security/security.c``) is provided to allow a security module to set
+security_ops to refer to its own hook functions, and an
+:c:func:`unregister_security()` function is provided to revert
+security_ops to the dummy module hooks. This mechanism is used to set
+the primary security module, which is responsible for making the final
+decision for each hook.
+
+LSM also provides a simple mechanism for stacking additional security
+modules with the primary security module. It defines
+:c:func:`register_security()` and
+:c:func:`unregister_security()` hooks in the :c:type:`struct
+security_operations <security_operations>` structure and
+provides :c:func:`mod_reg_security()` and
+:c:func:`mod_unreg_security()` functions that invoke these hooks
+after performing some sanity checking. A security module can call these
+functions in order to stack with other modules. However, the actual
+details of how this stacking is handled are deferred to the module,
+which can implement these hooks in any way it wishes (including always
+returning an error if it does not wish to support stacking). In this
+manner, LSM again defers the problem of composition to the module.
+
+Although the LSM hooks are organized into substructures based on kernel
+object, all of the hooks can be viewed as falling into two major
+categories: hooks that are used to manage the security fields and hooks
+that are used to perform access control. Examples of the first category
+of hooks include the :c:func:`alloc_security()` and
+:c:func:`free_security()` hooks defined for each kernel data
+structure that has a security field. These hooks are used to allocate
+and free security structures for kernel objects. The first category of
+hooks also includes hooks that set information in the security field
+after allocation, such as the :c:func:`post_lookup()` hook in
+:c:type:`struct inode_security_ops <inode_security_ops>`.
+This hook is used to set security information for inodes after
+successful lookup operations. An example of the second category of hooks
+is the :c:func:`permission()` hook in :c:type:`struct
+inode_security_ops <inode_security_ops>`. This hook checks
+permission when accessing an inode.
+
+LSM Capabilities Module
+=======================
+
+The LSM kernel patch moves most of the existing POSIX.1e capabilities
+logic into an optional security module stored in the file
+``security/capability.c``. This change allows users who do not want to
+use capabilities to omit this code entirely from their kernel, instead
+using the dummy module for traditional superuser logic or any other
+module that they desire. This change also allows the developers of the
+capabilities logic to maintain and enhance their code more freely,
+without needing to integrate patches back into the base kernel.
+
+In addition to moving the capabilities logic, the LSM kernel patch could
+move the capability-related fields from the kernel data structures into
+the new security fields managed by the security modules. However, at
+present, the LSM kernel patch leaves the capability fields in the kernel
+data structures. In his original remarks, Linus suggested that this
+might be preferable so that other security modules can be easily stacked
+with the capabilities module without needing to chain multiple security
+structures on the security field. It also avoids imposing extra overhead
+on the capabilities module to manage the security fields. However, the
+LSM framework could certainly support such a move if it is determined to
+be desirable, with only a few additional changes described below.
+
+At present, the capabilities logic for computing process capabilities on
+:c:func:`execve()` and :c:func:`set\*uid()`, checking
+capabilities for a particular process, saving and checking capabilities
+for netlink messages, and handling the :c:func:`capget()` and
+:c:func:`capset()` system calls have been moved into the
+capabilities module. There are still a few locations in the base kernel
+where capability-related fields are directly examined or modified, but
+the current version of the LSM patch does allow a security module to
+completely replace the assignment and testing of capabilities. These few
+locations would need to be changed if the capability-related fields were
+moved into the security field. The following is a list of known
+locations that still perform such direct examination or modification of
+capability-related fields:
+
+- ``fs/open.c``::c:func:`sys_access()`
+
+- ``fs/lockd/host.c``::c:func:`nlm_bind_host()`
+
+- ``fs/nfsd/auth.c``::c:func:`nfsd_setuser()`
+
+- ``fs/proc/array.c``::c:func:`task_cap()`
.. _sel-const-adjust:
-.. figure:: constraints.*
- :alt: constraints.pdf / constraints.svg
+.. kernel-figure:: constraints.svg
+ :alt: constraints.svg
:align: center
Size adjustments with constraint flags.
(2) to provide a guide as to how to use the barriers that are available.
Note that an architecture can provide more than the minimum requirement
-for any particular barrier, but if the architecure provides less than
+for any particular barrier, but if the architecture provides less than
that, that architecture is incorrect.
Note also that it is possible that a barrier may be a no-op for an
This means that ACQUIRE acts as a minimal "acquire" operation and
RELEASE acts as a minimal "release" operation.
-A subset of the atomic operations described in atomic_ops.txt have ACQUIRE
-and RELEASE variants in addition to fully-ordered and relaxed (no barrier
-semantics) definitions. For compound atomics performing both a load and a
-store, ACQUIRE semantics apply only to the load and RELEASE semantics apply
-only to the store portion of the operation.
+A subset of the atomic operations described in core-api/atomic_ops.rst have
+ACQUIRE and RELEASE variants in addition to fully-ordered and relaxed (no
+barrier semantics) definitions. For compound atomics performing both a load
+and a store, ACQUIRE semantics apply only to the load and RELEASE semantics
+apply only to the store portion of the operation.
Memory barriers are only required where there's a possibility of interaction
between two CPUs or between a CPU and a device. If it can be guaranteed that
--- /dev/null
+# -*- coding: utf-8; mode: python -*-
+
+project = "Linux Networking Documentation"
+
+tags.add("subproject")
+
+latex_documents = [
+ ('index', 'networking.tex', project,
+ 'The kernel development community', 'manual'),
+]
dns_query() returns a copy of the value attached to the key, or an error if
that is indicated instead.
-See <file:Documentation/security/keys-request-key.txt> for further
+See <file:Documentation/security/keys/request-key.rst> for further
information about request-key function.
--- /dev/null
+Linux Networking Documentation
+==============================
+
+Contents:
+
+.. toctree::
+ :maxdepth: 2
+
+ kapi
+ z8530book
+
+.. only:: subproject
+
+ Indices
+ =======
+
+ * :ref:`genindex`
+
--- /dev/null
+=========================================
+Linux Networking and Network Devices APIs
+=========================================
+
+Linux Networking
+================
+
+Networking Base Types
+---------------------
+
+.. kernel-doc:: include/linux/net.h
+ :internal:
+
+Socket Buffer Functions
+-----------------------
+
+.. kernel-doc:: include/linux/skbuff.h
+ :internal:
+
+.. kernel-doc:: include/net/sock.h
+ :internal:
+
+.. kernel-doc:: net/socket.c
+ :export:
+
+.. kernel-doc:: net/core/skbuff.c
+ :export:
+
+.. kernel-doc:: net/core/sock.c
+ :export:
+
+.. kernel-doc:: net/core/datagram.c
+ :export:
+
+.. kernel-doc:: net/core/stream.c
+ :export:
+
+Socket Filter
+-------------
+
+.. kernel-doc:: net/core/filter.c
+ :export:
+
+Generic Network Statistics
+--------------------------
+
+.. kernel-doc:: include/uapi/linux/gen_stats.h
+ :internal:
+
+.. kernel-doc:: net/core/gen_stats.c
+ :export:
+
+.. kernel-doc:: net/core/gen_estimator.c
+ :export:
+
+SUN RPC subsystem
+-----------------
+
+.. kernel-doc:: net/sunrpc/xdr.c
+ :export:
+
+.. kernel-doc:: net/sunrpc/svc_xprt.c
+ :export:
+
+.. kernel-doc:: net/sunrpc/xprt.c
+ :export:
+
+.. kernel-doc:: net/sunrpc/sched.c
+ :export:
+
+.. kernel-doc:: net/sunrpc/socklib.c
+ :export:
+
+.. kernel-doc:: net/sunrpc/stats.c
+ :export:
+
+.. kernel-doc:: net/sunrpc/rpc_pipe.c
+ :export:
+
+.. kernel-doc:: net/sunrpc/rpcb_clnt.c
+ :export:
+
+.. kernel-doc:: net/sunrpc/clnt.c
+ :export:
+
+WiMAX
+-----
+
+.. kernel-doc:: net/wimax/op-msg.c
+ :export:
+
+.. kernel-doc:: net/wimax/op-reset.c
+ :export:
+
+.. kernel-doc:: net/wimax/op-rfkill.c
+ :export:
+
+.. kernel-doc:: net/wimax/stack.c
+ :export:
+
+.. kernel-doc:: include/net/wimax.h
+ :internal:
+
+.. kernel-doc:: include/uapi/linux/wimax.h
+ :internal:
+
+Network device support
+======================
+
+Driver Support
+--------------
+
+.. kernel-doc:: net/core/dev.c
+ :export:
+
+.. kernel-doc:: net/ethernet/eth.c
+ :export:
+
+.. kernel-doc:: net/sched/sch_generic.c
+ :export:
+
+.. kernel-doc:: include/linux/etherdevice.h
+ :internal:
+
+.. kernel-doc:: include/linux/netdevice.h
+ :internal:
+
+PHY Support
+-----------
+
+.. kernel-doc:: drivers/net/phy/phy.c
+ :export:
+
+.. kernel-doc:: drivers/net/phy/phy.c
+ :internal:
+
+.. kernel-doc:: drivers/net/phy/phy_device.c
+ :export:
+
+.. kernel-doc:: drivers/net/phy/phy_device.c
+ :internal:
+
+.. kernel-doc:: drivers/net/phy/mdio_bus.c
+ :export:
+
+.. kernel-doc:: drivers/net/phy/mdio_bus.c
+ :internal:
--- /dev/null
+=======================
+Z8530 Programming Guide
+=======================
+
+:Author: Alan Cox
+
+Introduction
+============
+
+The Z85x30 family synchronous/asynchronous controller chips are used on
+a large number of cheap network interface cards. The kernel provides a
+core interface layer that is designed to make it easy to provide WAN
+services using this chip.
+
+The current driver only support synchronous operation. Merging the
+asynchronous driver support into this code to allow any Z85x30 device to
+be used as both a tty interface and as a synchronous controller is a
+project for Linux post the 2.4 release
+
+Driver Modes
+============
+
+The Z85230 driver layer can drive Z8530, Z85C30 and Z85230 devices in
+three different modes. Each mode can be applied to an individual channel
+on the chip (each chip has two channels).
+
+The PIO synchronous mode supports the most common Z8530 wiring. Here the
+chip is interface to the I/O and interrupt facilities of the host
+machine but not to the DMA subsystem. When running PIO the Z8530 has
+extremely tight timing requirements. Doing high speeds, even with a
+Z85230 will be tricky. Typically you should expect to achieve at best
+9600 baud with a Z8C530 and 64Kbits with a Z85230.
+
+The DMA mode supports the chip when it is configured to use dual DMA
+channels on an ISA bus. The better cards tend to support this mode of
+operation for a single channel. With DMA running the Z85230 tops out
+when it starts to hit ISA DMA constraints at about 512Kbits. It is worth
+noting here that many PC machines hang or crash when the chip is driven
+fast enough to hold the ISA bus solid.
+
+Transmit DMA mode uses a single DMA channel. The DMA channel is used for
+transmission as the transmit FIFO is smaller than the receive FIFO. it
+gives better performance than pure PIO mode but is nowhere near as ideal
+as pure DMA mode.
+
+Using the Z85230 driver
+=======================
+
+The Z85230 driver provides the back end interface to your board. To
+configure a Z8530 interface you need to detect the board and to identify
+its ports and interrupt resources. It is also your problem to verify the
+resources are available.
+
+Having identified the chip you need to fill in a struct z8530_dev,
+which describes each chip. This object must exist until you finally
+shutdown the board. Firstly zero the active field. This ensures nothing
+goes off without you intending it. The irq field should be set to the
+interrupt number of the chip. (Each chip has a single interrupt source
+rather than each channel). You are responsible for allocating the
+interrupt line. The interrupt handler should be set to
+:c:func:`z8530_interrupt()`. The device id should be set to the
+z8530_dev structure pointer. Whether the interrupt can be shared or not
+is board dependent, and up to you to initialise.
+
+The structure holds two channel structures. Initialise chanA.ctrlio and
+chanA.dataio with the address of the control and data ports. You can or
+this with Z8530_PORT_SLEEP to indicate your interface needs the 5uS
+delay for chip settling done in software. The PORT_SLEEP option is
+architecture specific. Other flags may become available on future
+platforms, eg for MMIO. Initialise the chanA.irqs to &z8530_nop to
+start the chip up as disabled and discarding interrupt events. This
+ensures that stray interrupts will be mopped up and not hang the bus.
+Set chanA.dev to point to the device structure itself. The private and
+name field you may use as you wish. The private field is unused by the
+Z85230 layer. The name is used for error reporting and it may thus make
+sense to make it match the network name.
+
+Repeat the same operation with the B channel if your chip has both
+channels wired to something useful. This isn't always the case. If it is
+not wired then the I/O values do not matter, but you must initialise
+chanB.dev.
+
+If your board has DMA facilities then initialise the txdma and rxdma
+fields for the relevant channels. You must also allocate the ISA DMA
+channels and do any necessary board level initialisation to configure
+them. The low level driver will do the Z8530 and DMA controller
+programming but not board specific magic.
+
+Having initialised the device you can then call
+:c:func:`z8530_init()`. This will probe the chip and reset it into
+a known state. An identification sequence is then run to identify the
+chip type. If the checks fail to pass the function returns a non zero
+error code. Typically this indicates that the port given is not valid.
+After this call the type field of the z8530_dev structure is
+initialised to either Z8530, Z85C30 or Z85230 according to the chip
+found.
+
+Once you have called z8530_init you can also make use of the utility
+function :c:func:`z8530_describe()`. This provides a consistent
+reporting format for the Z8530 devices, and allows all the drivers to
+provide consistent reporting.
+
+Attaching Network Interfaces
+============================
+
+If you wish to use the network interface facilities of the driver, then
+you need to attach a network device to each channel that is present and
+in use. In addition to use the generic HDLC you need to follow some
+additional plumbing rules. They may seem complex but a look at the
+example hostess_sv11 driver should reassure you.
+
+The network device used for each channel should be pointed to by the
+netdevice field of each channel. The hdlc-> priv field of the network
+device points to your private data - you will need to be able to find
+your private data from this.
+
+The way most drivers approach this particular problem is to create a
+structure holding the Z8530 device definition and put that into the
+private field of the network device. The network device fields of the
+channels then point back to the network devices.
+
+If you wish to use the generic HDLC then you need to register the HDLC
+device.
+
+Before you register your network device you will also need to provide
+suitable handlers for most of the network device callbacks. See the
+network device documentation for more details on this.
+
+Configuring And Activating The Port
+===================================
+
+The Z85230 driver provides helper functions and tables to load the port
+registers on the Z8530 chips. When programming the register settings for
+a channel be aware that the documentation recommends initialisation
+orders. Strange things happen when these are not followed.
+
+:c:func:`z8530_channel_load()` takes an array of pairs of
+initialisation values in an array of u8 type. The first value is the
+Z8530 register number. Add 16 to indicate the alternate register bank on
+the later chips. The array is terminated by a 255.
+
+The driver provides a pair of public tables. The z8530_hdlc_kilostream
+table is for the UK 'Kilostream' service and also happens to cover most
+other end host configurations. The z8530_hdlc_kilostream_85230 table
+is the same configuration using the enhancements of the 85230 chip. The
+configuration loaded is standard NRZ encoded synchronous data with HDLC
+bitstuffing. All of the timing is taken from the other end of the link.
+
+When writing your own tables be aware that the driver internally tracks
+register values. It may need to reload values. You should therefore be
+sure to set registers 1-7, 9-11, 14 and 15 in all configurations. Where
+the register settings depend on DMA selection the driver will update the
+bits itself when you open or close. Loading a new table with the
+interface open is not recommended.
+
+There are three standard configurations supported by the core code. In
+PIO mode the interface is programmed up to use interrupt driven PIO.
+This places high demands on the host processor to avoid latency. The
+driver is written to take account of latency issues but it cannot avoid
+latencies caused by other drivers, notably IDE in PIO mode. Because the
+drivers allocate buffers you must also prevent MTU changes while the
+port is open.
+
+Once the port is open it will call the rx_function of each channel
+whenever a completed packet arrived. This is invoked from interrupt
+context and passes you the channel and a network buffer (struct
+sk_buff) holding the data. The data includes the CRC bytes so most
+users will want to trim the last two bytes before processing the data.
+This function is very timing critical. When you wish to simply discard
+data the support code provides the function
+:c:func:`z8530_null_rx()` to discard the data.
+
+To active PIO mode sending and receiving the ``z8530_sync_open`` is called.
+This expects to be passed the network device and the channel. Typically
+this is called from your network device open callback. On a failure a
+non zero error status is returned.
+The :c:func:`z8530_sync_close()` function shuts down a PIO
+channel. This must be done before the channel is opened again and before
+the driver shuts down and unloads.
+
+The ideal mode of operation is dual channel DMA mode. Here the kernel
+driver will configure the board for DMA in both directions. The driver
+also handles ISA DMA issues such as controller programming and the
+memory range limit for you. This mode is activated by calling the
+:c:func:`z8530_sync_dma_open()` function. On failure a non zero
+error value is returned. Once this mode is activated it can be shut down
+by calling the :c:func:`z8530_sync_dma_close()`. You must call
+the close function matching the open mode you used.
+
+The final supported mode uses a single DMA channel to drive the transmit
+side. As the Z85C30 has a larger FIFO on the receive channel this tends
+to increase the maximum speed a little. This is activated by calling the
+``z8530_sync_txdma_open``. This returns a non zero error code on failure. The
+:c:func:`z8530_sync_txdma_close()` function closes down the Z8530
+interface from this mode.
+
+Network Layer Functions
+=======================
+
+The Z8530 layer provides functions to queue packets for transmission.
+The driver internally buffers the frame currently being transmitted and
+one further frame (in order to keep back to back transmission running).
+Any further buffering is up to the caller.
+
+The function :c:func:`z8530_queue_xmit()` takes a network buffer
+in sk_buff format and queues it for transmission. The caller must
+provide the entire packet with the exception of the bitstuffing and CRC.
+This is normally done by the caller via the generic HDLC interface
+layer. It returns 0 if the buffer has been queued and non zero values
+for queue full. If the function accepts the buffer it becomes property
+of the Z8530 layer and the caller should not free it.
+
+The function :c:func:`z8530_get_stats()` returns a pointer to an
+internally maintained per interface statistics block. This provides most
+of the interface code needed to implement the network layer get_stats
+callback.
+
+Porting The Z8530 Driver
+========================
+
+The Z8530 driver is written to be portable. In DMA mode it makes
+assumptions about the use of ISA DMA. These are probably warranted in
+most cases as the Z85230 in particular was designed to glue to PC type
+machines. The PIO mode makes no real assumptions.
+
+Should you need to retarget the Z8530 driver to another architecture the
+only code that should need changing are the port I/O functions. At the
+moment these assume PC I/O port accesses. This may not be appropriate
+for all platforms. Replacing :c:func:`z8530_read_port()` and
+``z8530_write_port`` is intended to be all that is required to port
+this driver layer.
+
+Known Bugs And Assumptions
+==========================
+
+Interrupt Locking
+ The locking in the driver is done via the global cli/sti lock. This
+ makes for relatively poor SMP performance. Switching this to use a
+ per device spin lock would probably materially improve performance.
+
+Occasional Failures
+ We have reports of occasional failures when run for very long
+ periods of time and the driver starts to receive junk frames. At the
+ moment the cause of this is not clear.
+
+Public Functions Provided
+=========================
+
+.. kernel-doc:: drivers/net/wan/z85230.c
+ :export:
+
+Internal Functions
+==================
+
+.. kernel-doc:: drivers/net/wan/z85230.c
+ :internal:
In particular, if the driver requires remote wakeup capability (i.e. hardware
mechanism allowing the device to request a change of its power state, such as
-PCI PME) for proper functioning and device_run_wake() returns 'false' for the
+PCI PME) for proper functioning and device_can_wakeup() returns 'false' for the
device, then ->runtime_suspend() should return -EBUSY. On the other hand, if
-device_run_wake() returns 'true' for the device and the device is put into a
+device_can_wakeup() returns 'true' for the device and the device is put into a
low-power state during the execution of the suspend callback, it is expected
that remote wakeup will be enabled for the device. Generally, remote wakeup
should be enabled for all input devices put into low-power states at run time.
being executed for that device and it is not practical to wait for the
suspend to complete; means "start a resume as soon as you've suspended"
- unsigned int run_wake;
- - set if the device is capable of generating runtime wake-up events
-
enum rpm_status runtime_status;
- the runtime PM status of the device; this field's initial value is
RPM_SUSPENDED, which means that each device is initially regarded by the
Linux documentation for functions is transitioning to inline
documentation via specially-formatted comments near their
-definitions in the source. These comments can be combined with the
-SGML templates in the Documentation/DocBook directory to make DocBook
-files, which can then be converted by DocBook stylesheets to PostScript,
-HTML, PDF files, and several other formats. In order to convert from
-DocBook format to a format of your choice, you'll need to install Jade as
-well as the desired DocBook stylesheets.
+definitions in the source. These comments can be combined with ReST
+files the Documentation/ directory to make enriched documentation, which can
+then be converted to PostScript, HTML, LaTex, ePUB and PDF files.
+In order to convert from ReST format to a format of your choice, you'll need
+Sphinx.
Util-linux
----------
functionalities required for ``XeLaTex`` to work. For PDF output you'll also
need ``convert(1)`` from ImageMagick (https://www.imagemagick.org).
-Other tools
------------
-
-In order to produce documentation from DocBook, you'll also need ``xmlto``.
-Please notice, however, that we're currently migrating all documents to use
-``Sphinx``.
Getting updated software
========================
- <http://sourceforge.net/projects/linuxquota/>
-DocBook Stylesheets
--------------------
-
-- <http://sourceforge.net/projects/docbook/files/docbook-dsssl/>
-
-XMLTO XSLT Frontend
--------------------
-
-- <http://cyberelk.net/tim/xmlto/>
Intel P6 microcode
------------------
When writing a single inline assembly statement containing multiple
instructions, put each instruction on a separate line in a separate quoted
-string, and end each string except the last with \n\t to properly indent the
-next instruction in the assembly output:
+string, and end each string except the last with ``\n\t`` to properly indent
+the next instruction in the assembly output:
.. code-block:: c
Run away from it.
+IBM Verse (Web GUI)
+*******************
+
+See Lotus Notes.
+
Mutt (TUI)
**********
make latexdocs
make epubdocs
-Currently, there are some documents written on DocBook that are in
-the process of conversion to ReST. Such documents will be created in the
-Documentation/DocBook/ directory and can be generated also as
-Postscript or man pages by running::
-
- make psdocs
- make mandocs
-
Becoming A Kernel Developer
---------------------------
Docs at the Linux Kernel tree
-----------------------------
-The DocBook books should be built with ``make {htmldocs | psdocs | pdfdocs}``.
The Sphinx books should be built with ``make {htmldocs | pdfdocs | epubdocs}``.
* Name: **linux/Documentation**
:Author: Many.
:Location: Documentation/
- :Keywords: text files, Sphinx, DocBook.
+ :Keywords: text files, Sphinx.
:Description: Documentation that comes with the kernel sources,
inside the Documentation directory. Some pages from this document
(including this document itself) have been moved there, and might
be more up to date than the web version.
- * Title: **The Kernel Hacking HOWTO**
-
- :Author: Various Talented People, and Rusty.
- :Location: Documentation/DocBook/kernel-hacking.tmpl
- :Keywords: HOWTO, kernel contexts, deadlock, locking, modules,
- symbols, return conventions.
- :Description: From the Introduction: "Please understand that I
- never wanted to write this document, being grossly underqualified,
- but I always wanted to read it, and this was the only way. I
- simply explain some best practices, and give reading entry-points
- into the kernel sources. I avoid implementation details: that's
- what the code is for, and I ignore whole tracts of useful
- routines. This document assumes familiarity with C, and an
- understanding of what the kernel is, and how it is used. It was
- originally written for the 2.3 kernels, but nearly all of it
- applies to 2.2 too; 2.0 is slightly different".
-
- * Title: **Linux Kernel Locking HOWTO**
-
- :Author: Various Talented People, and Rusty.
- :Location: Documentation/DocBook/kernel-locking.tmpl
- :Keywords: locks, locking, spinlock, semaphore, atomic, race
- condition, bottom halves, tasklets, softirqs.
- :Description: The title says it all: document describing the
- locking system in the Linux Kernel either in uniprocessor or SMP
- systems.
- :Notes: "It was originally written for the later (>2.3.47) 2.3
- kernels, but most of it applies to 2.2 too; 2.0 is slightly
- different". Freely redistributable under the conditions of the GNU
- General Public License.
-
On-line docs
------------
0. WARNING
1. Overview
2. Scheduling algorithm
+ 2.1 Main algorithm
+ 2.2 Bandwidth reclaiming
3. Scheduling Real-Time Tasks
3.1 Definitions
3.2 Schedulability Analysis for Uniprocessor Systems
2. Scheduling algorithm
==================
+2.1 Main algorithm
+------------------
+
SCHED_DEADLINE uses three parameters, named "runtime", "period", and
"deadline", to schedule tasks. A SCHED_DEADLINE task should receive
"runtime" microseconds of execution time every "period" microseconds, and
remaining runtime = remaining runtime + runtime
+2.2 Bandwidth reclaiming
+------------------------
+
+ Bandwidth reclaiming for deadline tasks is based on the GRUB (Greedy
+ Reclamation of Unused Bandwidth) algorithm [15, 16, 17] and it is enabled
+ when flag SCHED_FLAG_RECLAIM is set.
+
+ The following diagram illustrates the state names for tasks handled by GRUB:
+
+ ------------
+ (d) | Active |
+ ------------->| |
+ | | Contending |
+ | ------------
+ | A |
+ ---------- | |
+ | | | |
+ | Inactive | |(b) | (a)
+ | | | |
+ ---------- | |
+ A | V
+ | ------------
+ | | Active |
+ --------------| Non |
+ (c) | Contending |
+ ------------
+
+ A task can be in one of the following states:
+
+ - ActiveContending: if it is ready for execution (or executing);
+
+ - ActiveNonContending: if it just blocked and has not yet surpassed the 0-lag
+ time;
+
+ - Inactive: if it is blocked and has surpassed the 0-lag time.
+
+ State transitions:
+
+ (a) When a task blocks, it does not become immediately inactive since its
+ bandwidth cannot be immediately reclaimed without breaking the
+ real-time guarantees. It therefore enters a transitional state called
+ ActiveNonContending. The scheduler arms the "inactive timer" to fire at
+ the 0-lag time, when the task's bandwidth can be reclaimed without
+ breaking the real-time guarantees.
+
+ The 0-lag time for a task entering the ActiveNonContending state is
+ computed as
+
+ (runtime * dl_period)
+ deadline - ---------------------
+ dl_runtime
+
+ where runtime is the remaining runtime, while dl_runtime and dl_period
+ are the reservation parameters.
+
+ (b) If the task wakes up before the inactive timer fires, the task re-enters
+ the ActiveContending state and the "inactive timer" is canceled.
+ In addition, if the task wakes up on a different runqueue, then
+ the task's utilization must be removed from the previous runqueue's active
+ utilization and must be added to the new runqueue's active utilization.
+ In order to avoid races between a task waking up on a runqueue while the
+ "inactive timer" is running on a different CPU, the "dl_non_contending"
+ flag is used to indicate that a task is not on a runqueue but is active
+ (so, the flag is set when the task blocks and is cleared when the
+ "inactive timer" fires or when the task wakes up).
+
+ (c) When the "inactive timer" fires, the task enters the Inactive state and
+ its utilization is removed from the runqueue's active utilization.
+
+ (d) When an inactive task wakes up, it enters the ActiveContending state and
+ its utilization is added to the active utilization of the runqueue where
+ it has been enqueued.
+
+ For each runqueue, the algorithm GRUB keeps track of two different bandwidths:
+
+ - Active bandwidth (running_bw): this is the sum of the bandwidths of all
+ tasks in active state (i.e., ActiveContending or ActiveNonContending);
+
+ - Total bandwidth (this_bw): this is the sum of all tasks "belonging" to the
+ runqueue, including the tasks in Inactive state.
+
+
+ The algorithm reclaims the bandwidth of the tasks in Inactive state.
+ It does so by decrementing the runtime of the executing task Ti at a pace equal
+ to
+
+ dq = -max{ Ui, (1 - Uinact) } dt
+
+ where Uinact is the inactive utilization, computed as (this_bq - running_bw),
+ and Ui is the bandwidth of task Ti.
+
+
+ Let's now see a trivial example of two deadline tasks with runtime equal
+ to 4 and period equal to 8 (i.e., bandwidth equal to 0.5):
+
+ A Task T1
+ |
+ | |
+ | |
+ |-------- |----
+ | | V
+ |---|---|---|---|---|---|---|---|--------->t
+ 0 1 2 3 4 5 6 7 8
+
+
+ A Task T2
+ |
+ | |
+ | |
+ | ------------------------|
+ | | V
+ |---|---|---|---|---|---|---|---|--------->t
+ 0 1 2 3 4 5 6 7 8
+
+
+ A running_bw
+ |
+ 1 ----------------- ------
+ | | |
+ 0.5- -----------------
+ | |
+ |---|---|---|---|---|---|---|---|--------->t
+ 0 1 2 3 4 5 6 7 8
+
+
+ - Time t = 0:
+
+ Both tasks are ready for execution and therefore in ActiveContending state.
+ Suppose Task T1 is the first task to start execution.
+ Since there are no inactive tasks, its runtime is decreased as dq = -1 dt.
+
+ - Time t = 2:
+
+ Suppose that task T1 blocks
+ Task T1 therefore enters the ActiveNonContending state. Since its remaining
+ runtime is equal to 2, its 0-lag time is equal to t = 4.
+ Task T2 start execution, with runtime still decreased as dq = -1 dt since
+ there are no inactive tasks.
+
+ - Time t = 4:
+
+ This is the 0-lag time for Task T1. Since it didn't woken up in the
+ meantime, it enters the Inactive state. Its bandwidth is removed from
+ running_bw.
+ Task T2 continues its execution. However, its runtime is now decreased as
+ dq = - 0.5 dt because Uinact = 0.5.
+ Task T2 therefore reclaims the bandwidth unused by Task T1.
+
+ - Time t = 8:
+
+ Task T1 wakes up. It enters the ActiveContending state again, and the
+ running_bw is incremented.
+
+
3. Scheduling Real-Time Tasks
=============================
14 - J. Erickson, U. Devi and S. Baruah. Improved tardiness bounds for
Global EDF. Proceedings of the 22nd Euromicro Conference on
Real-Time Systems, 2010.
+ 15 - G. Lipari, S. Baruah, Greedy reclamation of unused bandwidth in
+ constant-bandwidth servers, 12th IEEE Euromicro Conference on Real-Time
+ Systems, 2000.
+ 16 - L. Abeni, J. Lelli, C. Scordino, L. Palopoli, Greedy CPU reclaiming for
+ SCHED DEADLINE. In Proceedings of the Real-Time Linux Workshop (RTLWS),
+ Dusseldorf, Germany, 2014.
+ 17 - L. Abeni, G. Lipari, A. Parri, Y. Sun, Multicore CPU reclaiming: parallel
+ or sequential?. In Proceedings of the 31st Annual ACM Symposium on Applied
+ Computing, 2016.
4. Bandwidth management
+++ /dev/null
-00-INDEX
- - this file.
-LSM.txt
- - description of the Linux Security Module framework.
-SELinux.txt
- - how to get started with the SELinux security enhancement.
-Smack.txt
- - documentation on the Smack Linux Security Module.
-Yama.txt
- - documentation on the Yama Linux Security Module.
-apparmor.txt
- - documentation on the AppArmor security extension.
-credentials.txt
- - documentation about credentials in Linux.
-keys-ecryptfs.txt
- - description of the encryption keys for the ecryptfs filesystem.
-keys-request-key.txt
- - description of the kernel key request service.
-keys-trusted-encrypted.txt
- - info on the Trusted and Encrypted keys in the kernel key ring service.
-keys.txt
- - description of the kernel key retention service.
-tomoyo.txt
- - documentation on the TOMOYO Linux Security Module.
-IMA-templates.txt
- - documentation on the template management mechanism for IMA.
- IMA Template Management Mechanism
+=================================
+IMA Template Management Mechanism
+=================================
-==== INTRODUCTION ====
+Introduction
+============
-The original 'ima' template is fixed length, containing the filedata hash
+The original ``ima`` template is fixed length, containing the filedata hash
and pathname. The filedata hash is limited to 20 bytes (md5/sha1).
The pathname is a null terminated string, limited to 255 characters.
To overcome these limitations and to add additional file metadata, it is
two functions, init() and show(), respectively to generate and display
measurement entries. Defining a new template descriptor requires
specifying the template format (a string of field identifiers separated
-by the '|' character) through the 'ima_template_fmt' kernel command line
+by the ``|`` character) through the ``ima_template_fmt`` kernel command line
parameter. At boot time, IMA initializes the chosen template descriptor
by translating the format into an array of template fields structures taken
from the set of the supported ones.
-After the initialization step, IMA will call ima_alloc_init_template()
+After the initialization step, IMA will call ``ima_alloc_init_template()``
(new function defined within the patches for the new template management
mechanism) to generate a new measurement entry by using the template
descriptor chosen through the kernel configuration or through the newly
-introduced 'ima_template' and 'ima_template_fmt' kernel command line parameters.
+introduced ``ima_template`` and ``ima_template_fmt`` kernel command line parameters.
It is during this phase that the advantages of the new architecture are
clearly shown: the latter function will not contain specific code to handle
-a given template but, instead, it simply calls the init() method of the template
+a given template but, instead, it simply calls the ``init()`` method of the template
fields associated to the chosen template descriptor and store the result
(pointer to allocated data and data length) in the measurement entry structure.
The same mechanism is employed to display measurements entries.
-The functions ima[_ascii]_measurements_show() retrieve, for each entry,
+The functions ``ima[_ascii]_measurements_show()`` retrieve, for each entry,
the template descriptor used to produce that entry and call the show()
method for each item of the array of template fields structures.
-==== SUPPORTED TEMPLATE FIELDS AND DESCRIPTORS ====
+Supported Template Fields and Descriptors
+=========================================
In the following, there is the list of supported template fields
-('<identifier>': description), that can be used to define new template
+``('<identifier>': description)``, that can be used to define new template
descriptors by adding their identifier to the format string
(support for more data types will be added later):
- 'd': the digest of the event (i.e. the digest of a measured file),
- calculated with the SHA1 or MD5 hash algorithm;
+ calculated with the SHA1 or MD5 hash algorithm;
- 'n': the name of the event (i.e. the file name), with size up to 255 bytes;
- 'd-ng': the digest of the event, calculated with an arbitrary hash
- algorithm (field format: [<hash algo>:]digest, where the digest
- prefix is shown only if the hash algorithm is not SHA1 or MD5);
+ algorithm (field format: [<hash algo>:]digest, where the digest
+ prefix is shown only if the hash algorithm is not SHA1 or MD5);
- 'n-ng': the name of the event, without size limitations;
- 'sig': the file signature.
Below, there is the list of defined template descriptors:
- - "ima": its format is 'd|n';
- - "ima-ng" (default): its format is 'd-ng|n-ng';
- - "ima-sig": its format is 'd-ng|n-ng|sig'.
+ - "ima": its format is ``d|n``;
+ - "ima-ng" (default): its format is ``d-ng|n-ng``;
+ - "ima-sig": its format is ``d-ng|n-ng|sig``.
-==== USE ====
+
+Use
+===
To specify the template descriptor to be used to generate measurement entries,
currently the following methods are supported:
- select a template descriptor among those supported in the kernel
- configuration ('ima-ng' is the default choice);
+ configuration (``ima-ng`` is the default choice);
- specify a template descriptor name from the kernel command line through
- the 'ima_template=' parameter;
+ the ``ima_template=`` parameter;
- register a new template descriptor with custom format through the kernel
- command line parameter 'ima_template_fmt='.
+ command line parameter ``ima_template_fmt=``.
--- /dev/null
+=================================
+Linux Security Module Development
+=================================
+
+Based on https://lkml.org/lkml/2007/10/26/215,
+a new LSM is accepted into the kernel when its intent (a description of
+what it tries to protect against and in what cases one would expect to
+use it) has been appropriately documented in ``Documentation/security/LSM``.
+This allows an LSM's code to be easily compared to its goals, and so
+that end users and distros can make a more informed decision about which
+LSMs suit their requirements.
+
+For extensive documentation on the available LSM hook interfaces, please
+see ``include/linux/lsm_hooks.h``.
+++ /dev/null
-project = "The kernel security subsystem manual"
-
-tags.add("subproject")
-
-latex_documents = [
- ('index', 'security.tex', project,
- 'The kernel development community', 'manual'),
-]
- ====================
- CREDENTIALS IN LINUX
- ====================
+====================
+Credentials in Linux
+====================
By: David Howells <dhowells@redhat.com>
-Contents:
-
- (*) Overview.
-
- (*) Types of credentials.
-
- (*) File markings.
-
- (*) Task credentials.
+.. contents:: :local:
- - Immutable credentials.
- - Accessing task credentials.
- - Accessing another task's credentials.
- - Altering credentials.
- - Managing credentials.
-
- (*) Open file credentials.
-
- (*) Overriding the VFS's use of credentials.
-
-
-========
-OVERVIEW
+Overview
========
There are several parts to the security check performed by Linux when one
object acts upon another:
- (1) Objects.
+ 1. Objects.
Objects are things in the system that may be acted upon directly by
userspace programs. Linux has a variety of actionable objects, including:
As a part of the description of all these objects there is a set of
credentials. What's in the set depends on the type of object.
- (2) Object ownership.
+ 2. Object ownership.
Amongst the credentials of most objects, there will be a subset that
indicates the ownership of that object. This is used for resource
In a standard UNIX filesystem, for instance, this will be defined by the
UID marked on the inode.
- (3) The objective context.
+ 3. The objective context.
Also amongst the credentials of those objects, there will be a subset that
indicates the 'objective context' of that object. This may or may not be
The objective context is used as part of the security calculation that is
carried out when an object is acted upon.
- (4) Subjects.
+ 4. Subjects.
A subject is an object that is acting upon another object.
Objects other than tasks may under some circumstances also be subjects.
For instance an open file may send SIGIO to a task using the UID and EUID
- given to it by a task that called fcntl(F_SETOWN) upon it. In this case,
+ given to it by a task that called ``fcntl(F_SETOWN)`` upon it. In this case,
the file struct will have a subjective context too.
- (5) The subjective context.
+ 5. The subjective context.
A subject has an additional interpretation of its credentials. A subset
of its credentials forms the 'subjective context'. The subjective context
from the real UID and GID that normally form the objective context of the
task.
- (6) Actions.
+ 6. Actions.
Linux has a number of actions available that a subject may perform upon an
object. The set of actions available depends on the nature of the subject
Actions include reading, writing, creating and deleting files; forking or
signalling and tracing tasks.
- (7) Rules, access control lists and security calculations.
+ 7. Rules, access control lists and security calculations.
When a subject acts upon an object, a security calculation is made. This
involves taking the subjective context, the objective context and the
There are two main sources of rules:
- (a) Discretionary access control (DAC):
+ a. Discretionary access control (DAC):
Sometimes the object will include sets of rules as part of its
description. This is an 'Access Control List' or 'ACL'. A Linux
A Linux file might also sport a POSIX ACL. This is a list of rules
that grants various permissions to arbitrary subjects.
- (b) Mandatory access control (MAC):
+ b. Mandatory access control (MAC):
The system as a whole may have one or more sets of rules that get
applied to all subjects and objects, regardless of their source.
that says that this action is either granted or denied.
-====================
-TYPES OF CREDENTIALS
+Types of Credentials
====================
The Linux kernel supports the following types of credentials:
- (1) Traditional UNIX credentials.
+ 1. Traditional UNIX credentials.
- Real User ID
- Real Group ID
+ - Real User ID
+ - Real Group ID
The UID and GID are carried by most, if not all, Linux objects, even if in
some cases it has to be invented (FAT or CIFS files for example, which are
derived from Windows). These (mostly) define the objective context of
that object, with tasks being slightly different in some cases.
- Effective, Saved and FS User ID
- Effective, Saved and FS Group ID
- Supplementary groups
+ - Effective, Saved and FS User ID
+ - Effective, Saved and FS Group ID
+ - Supplementary groups
These are additional credentials used by tasks only. Usually, an
EUID/EGID/GROUPS will be used as the subjective context, and real UID/GID
will be used as the objective. For tasks, it should be noted that this is
not always true.
- (2) Capabilities.
+ 2. Capabilities.
- Set of permitted capabilities
- Set of inheritable capabilities
- Set of effective capabilities
- Capability bounding set
+ - Set of permitted capabilities
+ - Set of inheritable capabilities
+ - Set of effective capabilities
+ - Capability bounding set
These are only carried by tasks. They indicate superior capabilities
granted piecemeal to a task that an ordinary task wouldn't otherwise have.
These are manipulated implicitly by changes to the traditional UNIX
- credentials, but can also be manipulated directly by the capset() system
- call.
+ credentials, but can also be manipulated directly by the ``capset()``
+ system call.
The permitted capabilities are those caps that the process might grant
- itself to its effective or permitted sets through capset(). This
+ itself to its effective or permitted sets through ``capset()``. This
inheritable set might also be so constrained.
The effective capabilities are the ones that a task is actually allowed to
make use of itself.
The inheritable capabilities are the ones that may get passed across
- execve().
+ ``execve()``.
The bounding set limits the capabilities that may be inherited across
- execve(), especially when a binary is executed that will execute as UID 0.
+ ``execve()``, especially when a binary is executed that will execute as
+ UID 0.
- (3) Secure management flags (securebits).
+ 3. Secure management flags (securebits).
These are only carried by tasks. These govern the way the above
credentials are manipulated and inherited over certain operations such as
execve(). They aren't used directly as objective or subjective
credentials.
- (4) Keys and keyrings.
+ 4. Keys and keyrings.
These are only carried by tasks. They carry and cache security tokens
that don't fit into the other standard UNIX credentials. They are for
For more information on using keys, see Documentation/security/keys.txt.
- (5) LSM
+ 5. LSM
The Linux Security Module allows extra controls to be placed over the
operations that a task may do. Currently Linux supports several LSM
rules (policies) that say what operations a task with one label may do to
an object with another label.
- (6) AF_KEY
+ 6. AF_KEY
This is a socket-based approach to credential management for networking
stacks [RFC 2367]. It isn't discussed by this document as it doesn't
to the server, regardless of who is actually doing a read or a write upon it.
-=============
-FILE MARKINGS
+File Markings
=============
Files on disk or obtained over the network may have annotations that form the
objective security context of that file. Depending on the type of filesystem,
this may include one or more of the following:
- (*) UNIX UID, GID, mode;
-
- (*) Windows user ID;
-
- (*) Access control list;
-
- (*) LSM security label;
-
- (*) UNIX exec privilege escalation bits (SUID/SGID);
-
- (*) File capabilities exec privilege escalation bits.
+ * UNIX UID, GID, mode;
+ * Windows user ID;
+ * Access control list;
+ * LSM security label;
+ * UNIX exec privilege escalation bits (SUID/SGID);
+ * File capabilities exec privilege escalation bits.
These are compared to the task's subjective security context, and certain
operations allowed or disallowed as a result. In the case of execve(), the
extra privileges, based on the annotations on the executable file.
-================
-TASK CREDENTIALS
+Task Credentials
================
In Linux, all of a task's credentials are held in (uid, gid) or through
Once a set of credentials has been prepared and committed, it may not be
changed, barring the following exceptions:
- (1) its reference count may be changed;
+ 1. its reference count may be changed;
- (2) the reference count on the group_info struct it points to may be changed;
+ 2. the reference count on the group_info struct it points to may be changed;
- (3) the reference count on the security data it points to may be changed;
+ 3. the reference count on the security data it points to may be changed;
- (4) the reference count on any keyrings it points to may be changed;
+ 4. the reference count on any keyrings it points to may be changed;
- (5) any keyrings it points to may be revoked, expired or have their security
- attributes changed; and
+ 5. any keyrings it points to may be revoked, expired or have their security
+ attributes changed; and
- (6) the contents of any keyrings to which it points may be changed (the whole
- point of keyrings being a shared set of credentials, modifiable by anyone
- with appropriate access).
+ 6. the contents of any keyrings to which it points may be changed (the whole
+ point of keyrings being a shared set of credentials, modifiable by anyone
+ with appropriate access).
To alter anything in the cred struct, the copy-and-replace principle must be
adhered to. First take a copy, then alter the copy and then use RCU to change
with this (see below).
A task may only alter its _own_ credentials; it is no longer permitted for a
-task to alter another's credentials. This means the capset() system call is no
-longer permitted to take any PID other than the one of the current process.
-Also keyctl_instantiate() and keyctl_negate() functions no longer permit
-attachment to process-specific keyrings in the requesting process as the
-instantiating process may need to create them.
+task to alter another's credentials. This means the ``capset()`` system call
+is no longer permitted to take any PID other than the one of the current
+process. Also ``keyctl_instantiate()`` and ``keyctl_negate()`` functions no
+longer permit attachment to process-specific keyrings in the requesting
+process as the instantiating process may need to create them.
-IMMUTABLE CREDENTIALS
+Immutable Credentials
---------------------
-Once a set of credentials has been made public (by calling commit_creds() for
-example), it must be considered immutable, barring two exceptions:
+Once a set of credentials has been made public (by calling ``commit_creds()``
+for example), it must be considered immutable, barring two exceptions:
- (1) The reference count may be altered.
+ 1. The reference count may be altered.
- (2) Whilst the keyring subscriptions of a set of credentials may not be
- changed, the keyrings subscribed to may have their contents altered.
+ 2. Whilst the keyring subscriptions of a set of credentials may not be
+ changed, the keyrings subscribed to may have their contents altered.
To catch accidental credential alteration at compile time, struct task_struct
has _const_ pointers to its credential sets, as does struct file. Furthermore,
-certain functions such as get_cred() and put_cred() operate on const pointers,
-thus rendering casts unnecessary, but require to temporarily ditch the const
-qualification to be able to alter the reference count.
+certain functions such as ``get_cred()`` and ``put_cred()`` operate on const
+pointers, thus rendering casts unnecessary, but require to temporarily ditch
+the const qualification to be able to alter the reference count.
-ACCESSING TASK CREDENTIALS
+Accessing Task Credentials
--------------------------
A task being able to alter only its own credentials permits the current process
to read or replace its own credentials without the need for any form of locking
-- which simplifies things greatly. It can just call:
+-- which simplifies things greatly. It can just call::
const struct cred *current_cred()
it afterwards.
There are convenience wrappers for retrieving specific aspects of a task's
-credentials (the value is simply returned in each case):
+credentials (the value is simply returned in each case)::
uid_t current_uid(void) Current's real UID
gid_t current_gid(void) Current's real GID
struct user_struct *current_user(void) Current's user account
There are also convenience wrappers for retrieving specific associated pairs of
-a task's credentials:
+a task's credentials::
void current_uid_gid(uid_t *, gid_t *);
void current_euid_egid(uid_t *, gid_t *);
In addition, there is a function for obtaining a reference on the current
-process's current set of credentials:
+process's current set of credentials::
const struct cred *get_current_cred(void);
and functions for getting references to one of the credentials that don't
-actually live in struct cred:
+actually live in struct cred::
struct user_struct *get_current_user(void);
struct group_info *get_current_groups(void);
which get references to the current process's user accounting structure and
supplementary groups list respectively.
-Once a reference has been obtained, it must be released with put_cred(),
-free_uid() or put_group_info() as appropriate.
+Once a reference has been obtained, it must be released with ``put_cred()``,
+``free_uid()`` or ``put_group_info()`` as appropriate.
-ACCESSING ANOTHER TASK'S CREDENTIALS
+Accessing Another Task's Credentials
------------------------------------
Whilst a task may access its own credentials without the need for locking, the
same is not true of a task wanting to access another task's credentials. It
-must use the RCU read lock and rcu_dereference().
+must use the RCU read lock and ``rcu_dereference()``.
-The rcu_dereference() is wrapped by:
+The ``rcu_dereference()`` is wrapped by::
const struct cred *__task_cred(struct task_struct *task);
-This should be used inside the RCU read lock, as in the following example:
+This should be used inside the RCU read lock, as in the following example::
void foo(struct task_struct *t, struct foo_data *f)
{
Should it be necessary to hold another task's credentials for a long period of
time, and possibly to sleep whilst doing so, then the caller should get a
-reference on them using:
+reference on them using::
const struct cred *get_task_cred(struct task_struct *task);
This does all the RCU magic inside of it. The caller must call put_cred() on
the credentials so obtained when they're finished with.
- [*] Note: The result of __task_cred() should not be passed directly to
- get_cred() as this may race with commit_cred().
+.. note::
+ The result of ``__task_cred()`` should not be passed directly to
+ ``get_cred()`` as this may race with ``commit_cred()``.
There are a couple of convenience functions to access bits of another task's
-credentials, hiding the RCU magic from the caller:
+credentials, hiding the RCU magic from the caller::
uid_t task_uid(task) Task's real UID
uid_t task_euid(task) Task's effective UID
-If the caller is holding the RCU read lock at the time anyway, then:
+If the caller is holding the RCU read lock at the time anyway, then::
__task_cred(task)->uid
__task_cred(task)->euid
should be used instead. Similarly, if multiple aspects of a task's credentials
-need to be accessed, RCU read lock should be used, __task_cred() called, the
-result stored in a temporary pointer and then the credential aspects called
+need to be accessed, RCU read lock should be used, ``__task_cred()`` called,
+the result stored in a temporary pointer and then the credential aspects called
from that before dropping the lock. This prevents the potentially expensive
RCU magic from being invoked multiple times.
Should some other single aspect of another task's credentials need to be
-accessed, then this can be used:
+accessed, then this can be used::
task_cred_xxx(task, member)
-where 'member' is a non-pointer member of the cred struct. For instance:
+where 'member' is a non-pointer member of the cred struct. For instance::
uid_t task_cred_xxx(task, suid);
disappear the moment the RCU read lock is dropped.
-ALTERING CREDENTIALS
+Altering Credentials
--------------------
As previously mentioned, a task may only alter its own credentials, and may not
locking to alter its own credentials.
To alter the current process's credentials, a function should first prepare a
-new set of credentials by calling:
+new set of credentials by calling::
struct cred *prepare_creds(void);
duplicate of the current process's credentials, returning with the mutex still
held if successful. It returns NULL if not successful (out of memory).
-The mutex prevents ptrace() from altering the ptrace state of a process whilst
-security checks on credentials construction and changing is taking place as
-the ptrace state may alter the outcome, particularly in the case of execve().
+The mutex prevents ``ptrace()`` from altering the ptrace state of a process
+whilst security checks on credentials construction and changing is taking place
+as the ptrace state may alter the outcome, particularly in the case of
+``execve()``.
The new credentials set should be altered appropriately, and any security
checks and hooks done. Both the current and the proposed sets of credentials
When the credential set is ready, it should be committed to the current process
-by calling:
+by calling::
int commit_creds(struct cred *new);
This will alter various aspects of the credentials and the process, giving the
-LSM a chance to do likewise, then it will use rcu_assign_pointer() to actually
-commit the new credentials to current->cred, it will release
-current->cred_replace_mutex to allow ptrace() to take place, and it will notify
-the scheduler and others of the changes.
+LSM a chance to do likewise, then it will use ``rcu_assign_pointer()`` to
+actually commit the new credentials to ``current->cred``, it will release
+``current->cred_replace_mutex`` to allow ``ptrace()`` to take place, and it
+will notify the scheduler and others of the changes.
This function is guaranteed to return 0, so that it can be tail-called at the
-end of such functions as sys_setresuid().
+end of such functions as ``sys_setresuid()``.
Note that this function consumes the caller's reference to the new credentials.
-The caller should _not_ call put_cred() on the new credentials afterwards.
+The caller should _not_ call ``put_cred()`` on the new credentials afterwards.
Furthermore, once this function has been called on a new set of credentials,
those credentials may _not_ be changed further.
-Should the security checks fail or some other error occur after prepare_creds()
-has been called, then the following function should be invoked:
+Should the security checks fail or some other error occur after
+``prepare_creds()`` has been called, then the following function should be
+invoked::
void abort_creds(struct cred *new);
-This releases the lock on current->cred_replace_mutex that prepare_creds() got
-and then releases the new credentials.
+This releases the lock on ``current->cred_replace_mutex`` that
+``prepare_creds()`` got and then releases the new credentials.
-A typical credentials alteration function would look something like this:
+A typical credentials alteration function would look something like this::
int alter_suid(uid_t suid)
{
}
-MANAGING CREDENTIALS
+Managing Credentials
--------------------
There are some functions to help manage credentials:
- (*) void put_cred(const struct cred *cred);
+ - ``void put_cred(const struct cred *cred);``
This releases a reference to the given set of credentials. If the
reference count reaches zero, the credentials will be scheduled for
destruction by the RCU system.
- (*) const struct cred *get_cred(const struct cred *cred);
+ - ``const struct cred *get_cred(const struct cred *cred);``
This gets a reference on a live set of credentials, returning a pointer to
that set of credentials.
- (*) struct cred *get_new_cred(struct cred *cred);
+ - ``struct cred *get_new_cred(struct cred *cred);``
This gets a reference on a set of credentials that is under construction
and is thus still mutable, returning a pointer to that set of credentials.
-=====================
-OPEN FILE CREDENTIALS
+Open File Credentials
=====================
When a new file is opened, a reference is obtained on the opening task's
-credentials and this is attached to the file struct as 'f_cred' in place of
-'f_uid' and 'f_gid'. Code that used to access file->f_uid and file->f_gid
-should now access file->f_cred->fsuid and file->f_cred->fsgid.
+credentials and this is attached to the file struct as ``f_cred`` in place of
+``f_uid`` and ``f_gid``. Code that used to access ``file->f_uid`` and
+``file->f_gid`` should now access ``file->f_cred->fsuid`` and
+``file->f_cred->fsgid``.
-It is safe to access f_cred without the use of RCU or locking because the
+It is safe to access ``f_cred`` without the use of RCU or locking because the
pointer will not change over the lifetime of the file struct, and nor will the
contents of the cred struct pointed to, barring the exceptions listed above
(see the Task Credentials section).
-=======================================
-OVERRIDING THE VFS'S USE OF CREDENTIALS
+Overriding the VFS's Use of Credentials
=======================================
Under some circumstances it is desirable to override the credentials used by
-the VFS, and that can be done by calling into such as vfs_mkdir() with a
+the VFS, and that can be done by calling into such as ``vfs_mkdir()`` with a
different set of credentials. This is done in the following places:
- (*) sys_faccessat().
-
- (*) do_coredump().
-
- (*) nfs4recover.c.
+ * ``sys_faccessat()``.
+ * ``do_coredump()``.
+ * nfs4recover.c.
======================
-Security documentation
+Security Documentation
======================
.. toctree::
+ :maxdepth: 1
+ credentials
+ IMA-templates
+ keys/index
+ LSM
+ self-protection
tpm/index
- ============================
- KERNEL KEY RETENTION SERVICE
- ============================
+============================
+Kernel Key Retention Service
+============================
This service allows cryptographic keys, authentication tokens, cross-domain
user mappings, and similar to be cached in the kernel for the use of
- Garbage collection
-============
-KEY OVERVIEW
+Key Overview
============
In this context, keys represent units of cryptographic data, authentication
- State.
- (*) Each key is issued a serial number of type key_serial_t that is unique for
+ * Each key is issued a serial number of type key_serial_t that is unique for
the lifetime of that key. All serial numbers are positive non-zero 32-bit
integers.
Userspace programs can use a key's serial numbers as a way to gain access
to it, subject to permission checking.
- (*) Each key is of a defined "type". Types must be registered inside the
+ * Each key is of a defined "type". Types must be registered inside the
kernel by a kernel service (such as a filesystem) before keys of that type
can be added or used. Userspace programs cannot define new types directly.
Should a type be removed from the system, all the keys of that type will
be invalidated.
- (*) Each key has a description. This should be a printable string. The key
+ * Each key has a description. This should be a printable string. The key
type provides an operation to perform a match between the description on a
key and a criterion string.
- (*) Each key has an owner user ID, a group ID and a permissions mask. These
+ * Each key has an owner user ID, a group ID and a permissions mask. These
are used to control what a process may do to a key from userspace, and
whether a kernel service will be able to find the key.
- (*) Each key can be set to expire at a specific time by the key type's
+ * Each key can be set to expire at a specific time by the key type's
instantiation function. Keys can also be immortal.
- (*) Each key can have a payload. This is a quantity of data that represent the
+ * Each key can have a payload. This is a quantity of data that represent the
actual "key". In the case of a keyring, this is a list of keys to which
the keyring links; in the case of a user-defined key, it's an arbitrary
blob of data.
permitted, another key type operation will be called to convert the key's
attached payload back into a blob of data.
- (*) Each key can be in one of a number of basic states:
+ * Each key can be in one of a number of basic states:
- (*) Uninstantiated. The key exists, but does not have any data attached.
+ * Uninstantiated. The key exists, but does not have any data attached.
Keys being requested from userspace will be in this state.
- (*) Instantiated. This is the normal state. The key is fully formed, and
+ * Instantiated. This is the normal state. The key is fully formed, and
has data attached.
- (*) Negative. This is a relatively short-lived state. The key acts as a
+ * Negative. This is a relatively short-lived state. The key acts as a
note saying that a previous call out to userspace failed, and acts as
a throttle on key lookups. A negative key can be updated to a normal
state.
- (*) Expired. Keys can have lifetimes set. If their lifetime is exceeded,
+ * Expired. Keys can have lifetimes set. If their lifetime is exceeded,
they traverse to this state. An expired key can be updated back to a
normal state.
- (*) Revoked. A key is put in this state by userspace action. It can't be
+ * Revoked. A key is put in this state by userspace action. It can't be
found or operated upon (apart from by unlinking it).
- (*) Dead. The key's type was unregistered, and so the key is now useless.
+ * Dead. The key's type was unregistered, and so the key is now useless.
Keys in the last three states are subject to garbage collection. See the
section on "Garbage collection".
-====================
-KEY SERVICE OVERVIEW
+Key Service Overview
====================
The key service provides a number of features besides keys:
- (*) The key service defines three special key types:
+ * The key service defines three special key types:
(+) "keyring"
be created and updated from userspace, but the payload is only
readable from kernel space.
- (*) Each process subscribes to three keyrings: a thread-specific keyring, a
+ * Each process subscribes to three keyrings: a thread-specific keyring, a
process-specific keyring, and a session-specific keyring.
The thread-specific keyring is discarded from the child when any sort of
The ownership of the thread keyring changes when the real UID and GID of
the thread changes.
- (*) Each user ID resident in the system holds two special keyrings: a user
+ * Each user ID resident in the system holds two special keyrings: a user
specific keyring and a default user session keyring. The default session
keyring is initialised with a link to the user-specific keyring.
If a process attempts to access its session key when it doesn't have one,
it will be subscribed to the default for its current UID.
- (*) Each user has two quotas against which the keys they own are tracked. One
+ * Each user has two quotas against which the keys they own are tracked. One
limits the total number of keys and keyrings, the other limits the total
amount of description and payload space that can be consumed.
If a system call that modifies a key or keyring in some way would put the
user over quota, the operation is refused and error EDQUOT is returned.
- (*) There's a system call interface by which userspace programs can create and
+ * There's a system call interface by which userspace programs can create and
manipulate keys and keyrings.
- (*) There's a kernel interface by which services can register types and search
+ * There's a kernel interface by which services can register types and search
for keys.
- (*) There's a way for the a search done from the kernel to call back to
+ * There's a way for the a search done from the kernel to call back to
userspace to request a key that can't be found in a process's keyrings.
- (*) An optional filesystem is available through which the key database can be
+ * An optional filesystem is available through which the key database can be
viewed and manipulated.
-======================
-KEY ACCESS PERMISSIONS
+Key Access Permissions
======================
Keys have an owner user ID, a group access ID, and a permissions mask. The mask
has up to eight bits each for possessor, user, group and other access. Only
six of each set of eight bits are defined. These permissions granted are:
- (*) View
+ * View
This permits a key or keyring's attributes to be viewed - including key
type and description.
- (*) Read
+ * Read
This permits a key's payload to be viewed or a keyring's list of linked
keys.
- (*) Write
+ * Write
This permits a key's payload to be instantiated or updated, or it allows a
link to be added to or removed from a keyring.
- (*) Search
+ * Search
This permits keyrings to be searched and keys to be found. Searches can
only recurse into nested keyrings that have search permission set.
- (*) Link
+ * Link
This permits a key or keyring to be linked to. To create a link from a
keyring to a key, a process must have Write permission on the keyring and
Link permission on the key.
- (*) Set Attribute
+ * Set Attribute
This permits a key's UID, GID and permissions mask to be changed.
the key or having the sysadmin capability is sufficient.
-===============
-SELINUX SUPPORT
+SELinux Support
===============
The security class "key" has been added to SELinux so that mandatory access
similarly.
-================
-NEW PROCFS FILES
+New ProcFS Files
================
Two files have been added to procfs by which an administrator can find out
about the status of the key service:
- (*) /proc/keys
+ * /proc/keys
This lists the keys that are currently viewable by the task reading the
file, giving information about their type, description and permissions.
security checks are still performed, and may further filter out keys that
the current process is not authorised to view.
- The contents of the file look like this:
+ The contents of the file look like this::
SERIAL FLAGS USAGE EXPY PERM UID GID TYPE DESCRIPTION: SUMMARY
00000001 I----- 39 perm 1f3f0000 0 0 keyring _uid_ses.0: 1/4
00000893 I--Q-N 1 35s 1f3f0000 0 0 user metal:silver: 0
00000894 I--Q-- 1 10h 003f0000 0 0 user metal:gold: 0
- The flags are:
+ The flags are::
I Instantiated
R Revoked
N Negative key
- (*) /proc/key-users
+ * /proc/key-users
This file lists the tracking data for each user that has at least one key
- on the system. Such data includes quota information and statistics:
+ on the system. Such data includes quota information and statistics::
[root@andromeda root]# cat /proc/key-users
0: 46 45/45 1/100 13/10000
32: 2 2/2 2/100 40/10000
38: 2 2/2 2/100 40/10000
- The format of each line is
+ The format of each line is::
+
<UID>: User ID to which this applies
<usage> Structure refcount
<inst>/<keys> Total number of keys and number instantiated
Four new sysctl files have been added also for the purpose of controlling the
quota limits on keys:
- (*) /proc/sys/kernel/keys/root_maxkeys
+ * /proc/sys/kernel/keys/root_maxkeys
/proc/sys/kernel/keys/root_maxbytes
These files hold the maximum number of keys that root may have and the
maximum total number of bytes of data that root may have stored in those
keys.
- (*) /proc/sys/kernel/keys/maxkeys
+ * /proc/sys/kernel/keys/maxkeys
/proc/sys/kernel/keys/maxbytes
These files hold the maximum number of keys that each non-root user may
the appropriate file.
-===============================
-USERSPACE SYSTEM CALL INTERFACE
+Userspace System Call Interface
===============================
Userspace can manipulate keys directly through three new syscalls: add_key,
When referring to a key directly, userspace programs should use the key's
serial number (a positive 32-bit integer). However, there are some special
values available for referring to special keys and keyrings that relate to the
-process making the call:
+process making the call::
CONSTANT VALUE KEY REFERENCED
============================== ====== ===========================
The main syscalls are:
- (*) Create a new key of given type, description and payload and add it to the
- nominated keyring:
+ * Create a new key of given type, description and payload and add it to the
+ nominated keyring::
key_serial_t add_key(const char *type, const char *desc,
const void *payload, size_t plen,
The ID of the new or updated key is returned if successful.
- (*) Search the process's keyrings for a key, potentially calling out to
- userspace to create it.
+ * Search the process's keyrings for a key, potentially calling out to
+ userspace to create it::
key_serial_t request_key(const char *type, const char *description,
const char *callout_info,
The keyctl syscall functions are:
- (*) Map a special key ID to a real key ID for this process:
+ * Map a special key ID to a real key ID for this process::
key_serial_t keyctl(KEYCTL_GET_KEYRING_ID, key_serial_t id,
int create);
non-zero; and the error ENOKEY will be returned if "create" is zero.
- (*) Replace the session keyring this process subscribes to with a new one:
+ * Replace the session keyring this process subscribes to with a new one::
key_serial_t keyctl(KEYCTL_JOIN_SESSION_KEYRING, const char *name);
The ID of the new session keyring is returned if successful.
- (*) Update the specified key:
+ * Update the specified key::
long keyctl(KEYCTL_UPDATE, key_serial_t key, const void *payload,
size_t plen);
add_key().
- (*) Revoke a key:
+ * Revoke a key::
long keyctl(KEYCTL_REVOKE, key_serial_t key);
be findable.
- (*) Change the ownership of a key:
+ * Change the ownership of a key::
long keyctl(KEYCTL_CHOWN, key_serial_t key, uid_t uid, gid_t gid);
its group list members.
- (*) Change the permissions mask on a key:
+ * Change the permissions mask on a key::
long keyctl(KEYCTL_SETPERM, key_serial_t key, key_perm_t perm);
error EINVAL will be returned.
- (*) Describe a key:
+ * Describe a key::
long keyctl(KEYCTL_DESCRIBE, key_serial_t key, char *buffer,
size_t buflen);
A process must have view permission on the key for this function to be
successful.
- If successful, a string is placed in the buffer in the following format:
+ If successful, a string is placed in the buffer in the following format::
<type>;<uid>;<gid>;<perm>;<description>
is hexadecimal. A NUL character is included at the end of the string if
the buffer is sufficiently big.
- This can be parsed with
+ This can be parsed with::
sscanf(buffer, "%[^;];%d;%d;%o;%s", type, &uid, &gid, &mode, desc);
- (*) Clear out a keyring:
+ * Clear out a keyring::
long keyctl(KEYCTL_CLEAR, key_serial_t keyring);
DNS resolver cache keyring is an example of this.
- (*) Link a key into a keyring:
+ * Link a key into a keyring::
long keyctl(KEYCTL_LINK, key_serial_t keyring, key_serial_t key);
added.
- (*) Unlink a key or keyring from another keyring:
+ * Unlink a key or keyring from another keyring::
long keyctl(KEYCTL_UNLINK, key_serial_t keyring, key_serial_t key);
is not present, error ENOENT will be the result.
- (*) Search a keyring tree for a key:
+ * Search a keyring tree for a key::
key_serial_t keyctl(KEYCTL_SEARCH, key_serial_t keyring,
const char *type, const char *description,
fails. On success, the resulting key ID will be returned.
- (*) Read the payload data from a key:
+ * Read the payload data from a key::
long keyctl(KEYCTL_READ, key_serial_t keyring, char *buffer,
size_t buflen);
available rather than the amount copied.
- (*) Instantiate a partially constructed key.
+ * Instantiate a partially constructed key::
long keyctl(KEYCTL_INSTANTIATE, key_serial_t key,
const void *payload, size_t plen,
array instead of a single buffer.
- (*) Negatively instantiate a partially constructed key.
+ * Negatively instantiate a partially constructed key::
long keyctl(KEYCTL_NEGATE, key_serial_t key,
unsigned timeout, key_serial_t keyring);
as rejecting the key with ENOKEY as the error code.
- (*) Set the default request-key destination keyring.
+ * Set the default request-key destination keyring::
long keyctl(KEYCTL_SET_REQKEY_KEYRING, int reqkey_defl);
This sets the default keyring to which implicitly requested keys will be
- attached for this thread. reqkey_defl should be one of these constants:
+ attached for this thread. reqkey_defl should be one of these constants::
CONSTANT VALUE NEW DEFAULT KEYRING
====================================== ====== =======================
there is one, otherwise the user default session keyring.
- (*) Set the timeout on a key.
+ * Set the timeout on a key::
long keyctl(KEYCTL_SET_TIMEOUT, key_serial_t key, unsigned timeout);
or expired keys.
- (*) Assume the authority granted to instantiate a key
+ * Assume the authority granted to instantiate a key::
long keyctl(KEYCTL_ASSUME_AUTHORITY, key_serial_t key);
The assumed authoritative key is inherited across fork and exec.
- (*) Get the LSM security context attached to a key.
+ * Get the LSM security context attached to a key::
long keyctl(KEYCTL_GET_SECURITY, key_serial_t key, char *buffer,
size_t buflen)
successful.
- (*) Install the calling process's session keyring on its parent.
+ * Install the calling process's session keyring on its parent::
long keyctl(KEYCTL_SESSION_TO_PARENT);
kernel and resumes executing userspace.
- (*) Invalidate a key.
+ * Invalidate a key::
long keyctl(KEYCTL_INVALIDATE, key_serial_t key);
A process must have search permission on the key for this function to be
successful.
- (*) Compute a Diffie-Hellman shared secret or public key
+ * Compute a Diffie-Hellman shared secret or public key::
- long keyctl(KEYCTL_DH_COMPUTE, struct keyctl_dh_params *params,
- char *buffer, size_t buflen,
- struct keyctl_kdf_params *kdf);
+ long keyctl(KEYCTL_DH_COMPUTE, struct keyctl_dh_params *params,
+ char *buffer, size_t buflen, struct keyctl_kdf_params *kdf);
- The params struct contains serial numbers for three keys:
+ The params struct contains serial numbers for three keys::
- The prime, p, known to both parties
- The local private key
- The base integer, which is either a shared generator or the
remote public key
- The value computed is:
+ The value computed is::
result = base ^ private (mod prime)
of the KDF is returned to the caller. The KDF is characterized with
struct keyctl_kdf_params as follows:
- - char *hashname specifies the NUL terminated string identifying
+ - ``char *hashname`` specifies the NUL terminated string identifying
the hash used from the kernel crypto API and applied for the KDF
operation. The KDF implemenation complies with SP800-56A as well
as with SP800-108 (the counter KDF).
- - char *otherinfo specifies the OtherInfo data as documented in
+ - ``char *otherinfo`` specifies the OtherInfo data as documented in
SP800-56A section 5.8.1.2. The length of the buffer is given with
otherinfolen. The format of OtherInfo is defined by the caller.
The otherinfo pointer may be NULL if no OtherInfo shall be used.
and either the buffer length or the OtherInfo length exceeds the
allowed length.
- (*) Restrict keyring linkage
+ * Restrict keyring linkage::
- long keyctl(KEYCTL_RESTRICT_KEYRING, key_serial_t keyring,
- const char *type, const char *restriction);
+ long keyctl(KEYCTL_RESTRICT_KEYRING, key_serial_t keyring,
+ const char *type, const char *restriction);
An existing keyring can restrict linkage of additional keys by evaluating
the contents of the key according to a restriction scheme.
To apply a keyring restriction the process must have Set Attribute
permission and the keyring must not be previously restricted.
-===============
-KERNEL SERVICES
+Kernel Services
===============
The kernel services for key management are fairly simple to deal with. They can
two different users opening the same file is left to the filesystem author to
solve.
-To access the key manager, the following header must be #included:
+To access the key manager, the following header must be #included::
<linux/key.h>
Specific key types should have a header file under include/keys/ that should be
-used to access that type. For keys of type "user", for example, that would be:
+used to access that type. For keys of type "user", for example, that would be::
<keys/user-type.h>
Note that there are two different types of pointers to keys that may be
encountered:
- (*) struct key *
+ * struct key *
This simply points to the key structure itself. Key structures will be at
least four-byte aligned.
- (*) key_ref_t
+ * key_ref_t
- This is equivalent to a struct key *, but the least significant bit is set
+ This is equivalent to a ``struct key *``, but the least significant bit is set
if the caller "possesses" the key. By "possession" it is meant that the
calling processes has a searchable link to the key from one of its
- keyrings. There are three functions for dealing with these:
+ keyrings. There are three functions for dealing with these::
key_ref_t make_key_ref(const struct key *key, bool possession);
prevent access vs modification races. See the section "Notes on accessing
payload contents" for more information.
-(*) To search for a key, call:
+ * To search for a key, call::
struct key *request_key(const struct key_type *type,
const char *description,
See also Documentation/security/keys-request-key.txt.
-(*) To search for a key, passing auxiliary data to the upcaller, call:
+ * To search for a key, passing auxiliary data to the upcaller, call::
struct key *request_key_with_auxdata(const struct key_type *type,
const char *description,
is a blob of length callout_len, if given (the length may be 0).
-(*) A key can be requested asynchronously by calling one of:
+ * A key can be requested asynchronously by calling one of::
struct key *request_key_async(const struct key_type *type,
const char *description,
const void *callout_info,
size_t callout_len);
- or:
+ or::
struct key *request_key_async_with_auxdata(const struct key_type *type,
const char *description,
These two functions return with the key potentially still under
construction. To wait for construction completion, the following should be
- called:
+ called::
int wait_for_key_construction(struct key *key, bool intr);
case error ERESTARTSYS will be returned.
-(*) When it is no longer required, the key should be released using:
+ * When it is no longer required, the key should be released using::
void key_put(struct key *key);
- Or:
+ Or::
void key_ref_put(key_ref_t key_ref);
the argument will not be parsed.
-(*) Extra references can be made to a key by calling one of the following
- functions:
+ * Extra references can be made to a key by calling one of the following
+ functions::
struct key *__key_get(struct key *key);
struct key *key_get(struct key *key);
then the key will not be dereferenced and no increment will take place.
-(*) A key's serial number can be obtained by calling:
+ * A key's serial number can be obtained by calling::
key_serial_t key_serial(struct key *key);
latter case without parsing the argument).
-(*) If a keyring was found in the search, this can be further searched by:
+ * If a keyring was found in the search, this can be further searched by::
key_ref_t keyring_search(key_ref_t keyring_ref,
const struct key_type *type,
reference pointer if successful.
-(*) A keyring can be created by:
+ * A keyring can be created by::
struct key *keyring_alloc(const char *description, uid_t uid, gid_t gid,
const struct cred *cred,
-EPERM to in this case.
-(*) To check the validity of a key, this function can be called:
+ * To check the validity of a key, this function can be called::
int validate_key(struct key *key);
returned (in the latter case without parsing the argument).
-(*) To register a key type, the following function should be called:
+ * To register a key type, the following function should be called::
int register_key_type(struct key_type *type);
present.
-(*) To unregister a key type, call:
+ * To unregister a key type, call::
void unregister_key_type(struct key_type *type);
Under some circumstances, it may be desirable to deal with a bundle of keys.
-The facility provides access to the keyring type for managing such a bundle:
+The facility provides access to the keyring type for managing such a bundle::
struct key_type key_type_keyring;
search a specific keyring, so using keyrings in this way is of limited utility.
-===================================
-NOTES ON ACCESSING PAYLOAD CONTENTS
+Notes On Accessing Payload Contents
===================================
The simplest payload is just data stored in key->payload directly. In this
key->payload.data[] array. One of the following ways must be selected to
access the data:
- (1) Unmodifiable key type.
+ 1) Unmodifiable key type.
If the key type does not have a modify method, then the key's payload can
be accessed without any form of locking, provided that it's known to be
instantiated (uninstantiated keys cannot be "found").
- (2) The key's semaphore.
+ 2) The key's semaphore.
The semaphore could be used to govern access to the payload and to control
the payload pointer. It must be write-locked for modifications and would
have to be read-locked for general access. The disadvantage of doing this
is that the accessor may be required to sleep.
- (3) RCU.
+ 3) RCU.
RCU must be used when the semaphore isn't already held; if the semaphore
is held then the contents can't change under you unexpectedly as the
semaphore must still be used to serialise modifications to the key. The
key management code takes care of this for the key type.
- However, this means using:
+ However, this means using::
rcu_read_lock() ... rcu_dereference() ... rcu_read_unlock()
- to read the pointer, and:
+ to read the pointer, and::
rcu_dereference() ... rcu_assign_pointer() ... call_rcu()
usage. This is called key->payload.rcu_data0. The following accessors
wrap the RCU calls to this element:
- (a) Set or change the first payload pointer:
+ a) Set or change the first payload pointer::
rcu_assign_keypointer(struct key *key, void *data);
- (b) Read the first payload pointer with the key semaphore held:
+ b) Read the first payload pointer with the key semaphore held::
[const] void *dereference_key_locked([const] struct key *key);
parameter. Static analysis will give an error if it things the lock
isn't held.
- (c) Read the first payload pointer with the RCU read lock held:
+ c) Read the first payload pointer with the RCU read lock held::
const void *dereference_key_rcu(const struct key *key);
-===================
-DEFINING A KEY TYPE
+Defining a Key Type
===================
A kernel service may want to define its own key type. For instance, an AFS
filesystem might want to define a Kerberos 5 ticket key type. To do this, it
author fills in a key_type struct and registers it with the system.
-Source files that implement key types should include the following header file:
+Source files that implement key types should include the following header file::
<linux/key-type.h>
The structure has a number of fields, some of which are mandatory:
- (*) const char *name
+ * ``const char *name``
The name of the key type. This is used to translate a key type name
supplied by userspace into a pointer to the structure.
- (*) size_t def_datalen
+ * ``size_t def_datalen``
This is optional - it supplies the default payload data length as
contributed to the quota. If the key type's payload is always or almost
always the same size, then this is a more efficient way to do things.
The data length (and quota) on a particular key can always be changed
- during instantiation or update by calling:
+ during instantiation or update by calling::
int key_payload_reserve(struct key *key, size_t datalen);
viable.
- (*) int (*vet_description)(const char *description);
+ * ``int (*vet_description)(const char *description);``
This optional method is called to vet a key description. If the key type
doesn't approve of the key description, it may return an error, otherwise
it should return 0.
- (*) int (*preparse)(struct key_preparsed_payload *prep);
+ * ``int (*preparse)(struct key_preparsed_payload *prep);``
This optional method permits the key type to attempt to parse payload
before a key is created (add key) or the key semaphore is taken (update or
- instantiate key). The structure pointed to by prep looks like:
+ instantiate key). The structure pointed to by prep looks like::
struct key_preparsed_payload {
char *description;
otherwise.
- (*) void (*free_preparse)(struct key_preparsed_payload *prep);
+ * ``void (*free_preparse)(struct key_preparsed_payload *prep);``
This method is only required if the preparse() method is provided,
otherwise it is unused. It cleans up anything attached to the description
successfully, even if instantiate() or update() succeed.
- (*) int (*instantiate)(struct key *key, struct key_preparsed_payload *prep);
+ * ``int (*instantiate)(struct key *key, struct key_preparsed_payload *prep);``
This method is called to attach a payload to a key during construction.
The payload attached need not bear any relation to the data passed to this
free_preparse method doesn't release the data.
- (*) int (*update)(struct key *key, const void *data, size_t datalen);
+ * ``int (*update)(struct key *key, const void *data, size_t datalen);``
If this type of key can be updated, then this method should be provided.
It is called to update a key's payload from the blob of data provided.
It is safe to sleep in this method.
- (*) int (*match_preparse)(struct key_match_data *match_data);
+ * ``int (*match_preparse)(struct key_match_data *match_data);``
This method is optional. It is called when a key search is about to be
- performed. It is given the following structure:
+ performed. It is given the following structure::
struct key_match_data {
bool (*cmp)(const struct key *key,
};
On entry, raw_data will be pointing to the criteria to be used in matching
- a key by the caller and should not be modified. (*cmp)() will be pointing
+ a key by the caller and should not be modified. ``(*cmp)()`` will be pointing
to the default matcher function (which does an exact description match
against raw_data) and lookup_type will be set to indicate a direct lookup.
The following lookup_type values are available:
- [*] KEYRING_SEARCH_LOOKUP_DIRECT - A direct lookup hashes the type and
+ * KEYRING_SEARCH_LOOKUP_DIRECT - A direct lookup hashes the type and
description to narrow down the search to a small number of keys.
- [*] KEYRING_SEARCH_LOOKUP_ITERATE - An iterative lookup walks all the
+ * KEYRING_SEARCH_LOOKUP_ITERATE - An iterative lookup walks all the
keys in the keyring until one is matched. This must be used for any
search that's not doing a simple direct match on the key description.
The method may set cmp to point to a function of its choice that does some
other form of match, may set lookup_type to KEYRING_SEARCH_LOOKUP_ITERATE
- and may attach something to the preparsed pointer for use by (*cmp)().
- (*cmp)() should return true if a key matches and false otherwise.
+ and may attach something to the preparsed pointer for use by ``(*cmp)()``.
+ ``(*cmp)()`` should return true if a key matches and false otherwise.
If preparsed is set, it may be necessary to use the match_free() method to
clean it up.
The method should return 0 if successful or a negative error code
otherwise.
- It is permitted to sleep in this method, but (*cmp)() may not sleep as
+ It is permitted to sleep in this method, but ``(*cmp)()`` may not sleep as
locks will be held over it.
If match_preparse() is not provided, keys of this type will be matched
exactly by their description.
- (*) void (*match_free)(struct key_match_data *match_data);
+ * ``void (*match_free)(struct key_match_data *match_data);``
This method is optional. If given, it called to clean up
match_data->preparsed after a successful call to match_preparse().
- (*) void (*revoke)(struct key *key);
+ * ``void (*revoke)(struct key *key);``
This method is optional. It is called to discard part of the payload
data upon a key being revoked. The caller will have the key semaphore
a deadlock against the key semaphore.
- (*) void (*destroy)(struct key *key);
+ * ``void (*destroy)(struct key *key);``
This method is optional. It is called to discard the payload data on a key
when it is being destroyed.
It is not safe to sleep in this method; the caller may hold spinlocks.
- (*) void (*describe)(const struct key *key, struct seq_file *p);
+ * ``void (*describe)(const struct key *key, struct seq_file *p);``
This method is optional. It is called during /proc/keys reading to
summarise a key's description and payload in text form.
caller.
- (*) long (*read)(const struct key *key, char __user *buffer, size_t buflen);
+ * ``long (*read)(const struct key *key, char __user *buffer, size_t buflen);``
This method is optional. It is called by KEYCTL_READ to translate the
key's payload into something a blob of data for userspace to deal with.
as might happen when the userspace buffer is accessed.
- (*) int (*request_key)(struct key_construction *cons, const char *op,
- void *aux);
+ * ``int (*request_key)(struct key_construction *cons, const char *op, void *aux);``
This method is optional. If provided, request_key() and friends will
invoke this function rather than upcalling to /sbin/request-key to operate
This method is permitted to return before the upcall is complete, but the
following function must be called under all circumstances to complete the
instantiation process, whether or not it succeeds, whether or not there's
- an error:
+ an error::
void complete_request_key(struct key_construction *cons, int error);
The key under construction and the authorisation key can be found in the
key_construction struct pointed to by cons:
- (*) struct key *key;
+ * ``struct key *key;``
The key under construction.
- (*) struct key *authkey;
+ * ``struct key *authkey;``
The authorisation key.
- (*) struct key_restriction *(*lookup_restriction)(const char *params);
+ * ``struct key_restriction *(*lookup_restriction)(const char *params);``
This optional method is used to enable userspace configuration of keyring
restrictions. The restriction parameter string (not including the key type
attempted key link operation. If there is no match, -EINVAL is returned.
-============================
-REQUEST-KEY CALLBACK SERVICE
+Request-Key Callback Service
============================
To create a new key, the kernel will attempt to execute the following command
-line:
+line::
/sbin/request-key create <key> <uid> <gid> \
<threadring> <processring> <sessionring> <callout_info>
keyrings from the process that caused the search to be issued. These are
included for two reasons:
- (1) There may be an authentication token in one of the keyrings that is
+ 1 There may be an authentication token in one of the keyrings that is
required to obtain the key, eg: a Kerberos Ticket-Granting Ticket.
- (2) The new key should probably be cached in one of these rings.
+ 2 The new key should probably be cached in one of these rings.
This program should set it UID and GID to those specified before attempting to
access any more keys. It may then look around for a user specific process to
Similarly, the kernel may attempt to update an expired or a soon to expire key
-by executing:
+by executing::
/sbin/request-key update <key> <uid> <gid> \
<threadring> <processring> <sessionring>
the rings are provided for reference.
-==================
-GARBAGE COLLECTION
+Garbage Collection
==================
Dead keys (for which the type has been removed) will be automatically unlinked
background garbage collector.
Similarly, revoked and expired keys will be garbage collected, but only after a
-certain amount of time has passed. This time is set as a number of seconds in:
+certain amount of time has passed. This time is set as a number of seconds in::
/proc/sys/kernel/keys/gc_delay
- Encrypted keys for the eCryptfs filesystem
+==========================================
+Encrypted keys for the eCryptfs filesystem
+==========================================
ECryptfs is a stacked filesystem which transparently encrypts and decrypts each
file using a randomly generated File Encryption Key (FEK).
threats of malicious software, because it is available in clear form only at
kernel level.
-Usage:
+Usage::
+
keyctl add encrypted name "new ecryptfs key-type:master-key-name keylen" ring
keyctl add encrypted name "load hex_blob" ring
keyctl update keyid "update key-type:master-key-name"
-name:= '<16 hexadecimal characters>'
-key-type:= 'trusted' | 'user'
-keylen:= 64
+Where::
+
+ name:= '<16 hexadecimal characters>'
+ key-type:= 'trusted' | 'user'
+ keylen:= 64
Example of encrypted key usage with the eCryptfs filesystem:
Create an encrypted key "1000100010001000" of length 64 bytes with format
-'ecryptfs' and save it using a previously loaded user key "test":
+'ecryptfs' and save it using a previously loaded user key "test"::
$ keyctl add encrypted 1000100010001000 "new ecryptfs user:test 64" @u
19184530
$ keyctl pipe 19184530 > ecryptfs.blob
Mount an eCryptfs filesystem using the created encrypted key "1000100010001000"
-into the '/secret' directory:
+into the '/secret' directory::
$ mount -i -t ecryptfs -oecryptfs_sig=1000100010001000,\
ecryptfs_cipher=aes,ecryptfs_key_bytes=32 /secret /secret
--- /dev/null
+===========
+Kernel Keys
+===========
+
+.. toctree::
+ :maxdepth: 1
+
+ core
+ ecryptfs
+ request-key
+ trusted-encrypted
- ===================
- KEY REQUEST SERVICE
- ===================
+===================
+Key Request Service
+===================
The key request service is part of the key retention service (refer to
Documentation/security/keys.txt). This document explains more fully how
the requesting algorithm works.
The process starts by either the kernel requesting a service by calling
-request_key*():
+``request_key*()``::
struct key *request_key(const struct key_type *type,
const char *description,
const char *callout_info);
-or:
+or::
struct key *request_key_with_auxdata(const struct key_type *type,
const char *description,
size_t callout_len,
void *aux);
-or:
+or::
struct key *request_key_async(const struct key_type *type,
const char *description,
const char *callout_info,
size_t callout_len);
-or:
+or::
struct key *request_key_async_with_auxdata(const struct key_type *type,
const char *description,
size_t callout_len,
void *aux);
-Or by userspace invoking the request_key system call:
+Or by userspace invoking the request_key system call::
key_serial_t request_key(const char *type,
const char *description,
forking and execution of /sbin/request-key.
-===========
-THE PROCESS
+The Process
===========
A request proceeds in the following manner:
- (1) Process A calls request_key() [the userspace syscall calls the kernel
+ 1) Process A calls request_key() [the userspace syscall calls the kernel
interface].
- (2) request_key() searches the process's subscribed keyrings to see if there's
+ 2) request_key() searches the process's subscribed keyrings to see if there's
a suitable key there. If there is, it returns the key. If there isn't,
and callout_info is not set, an error is returned. Otherwise the process
proceeds to the next step.
- (3) request_key() sees that A doesn't have the desired key yet, so it creates
+ 3) request_key() sees that A doesn't have the desired key yet, so it creates
two things:
- (a) An uninstantiated key U of requested type and description.
+ a) An uninstantiated key U of requested type and description.
- (b) An authorisation key V that refers to key U and notes that process A
+ b) An authorisation key V that refers to key U and notes that process A
is the context in which key U should be instantiated and secured, and
from which associated key requests may be satisfied.
- (4) request_key() then forks and executes /sbin/request-key with a new session
+ 4) request_key() then forks and executes /sbin/request-key with a new session
keyring that contains a link to auth key V.
- (5) /sbin/request-key assumes the authority associated with key U.
+ 5) /sbin/request-key assumes the authority associated with key U.
- (6) /sbin/request-key execs an appropriate program to perform the actual
+ 6) /sbin/request-key execs an appropriate program to perform the actual
instantiation.
- (7) The program may want to access another key from A's context (say a
+ 7) The program may want to access another key from A's context (say a
Kerberos TGT key). It just requests the appropriate key, and the keyring
search notes that the session keyring has auth key V in its bottom level.
UID, GID, groups and security info of process A as if it was process A,
and come up with key W.
- (8) The program then does what it must to get the data with which to
+ 8) The program then does what it must to get the data with which to
instantiate key U, using key W as a reference (perhaps it contacts a
Kerberos server using the TGT) and then instantiates key U.
- (9) Upon instantiating key U, auth key V is automatically revoked so that it
+ 9) Upon instantiating key U, auth key V is automatically revoked so that it
may not be used again.
-(10) The program then exits 0 and request_key() deletes key V and returns key
- U to the caller.
+ 10) The program then exits 0 and request_key() deletes key V and returns key
+ U to the caller.
This also extends further. If key W (step 7 above) didn't exist, key W would
be created uninstantiated, another auth key (X) would be created (as per step
of them, and (b) it requires the same UID/GID/Groups all the way through.
-====================================
-NEGATIVE INSTANTIATION AND REJECTION
+Negative Instantiation And Rejection
====================================
Rather than instantiating a key, it is possible for the possessor of an
instantiated for a short amount of time.
-====================
-THE SEARCH ALGORITHM
+The Search Algorithm
====================
A search of any particular keyring proceeds in the following fashion:
- (1) When the key management code searches for a key (keyring_search_aux) it
+ 1) When the key management code searches for a key (keyring_search_aux) it
firstly calls key_permission(SEARCH) on the keyring it's starting with,
if this denies permission, it doesn't search further.
- (2) It considers all the non-keyring keys within that keyring and, if any key
+ 2) It considers all the non-keyring keys within that keyring and, if any key
matches the criteria specified, calls key_permission(SEARCH) on it to see
if the key is allowed to be found. If it is, that key is returned; if
not, the search continues, and the error code is retained if of higher
priority than the one currently set.
- (3) It then considers all the keyring-type keys in the keyring it's currently
+ 3) It then considers all the keyring-type keys in the keyring it's currently
searching. It calls key_permission(SEARCH) on each keyring, and if this
grants permission, it recurses, executing steps (2) and (3) on that
keyring.
When search_process_keyrings() is invoked, it performs the following searches
until one succeeds:
- (1) If extant, the process's thread keyring is searched.
+ 1) If extant, the process's thread keyring is searched.
- (2) If extant, the process's process keyring is searched.
+ 2) If extant, the process's process keyring is searched.
- (3) The process's session keyring is searched.
+ 3) The process's session keyring is searched.
- (4) If the process has assumed the authority associated with a request_key()
+ 4) If the process has assumed the authority associated with a request_key()
authorisation key then:
- (a) If extant, the calling process's thread keyring is searched.
+ a) If extant, the calling process's thread keyring is searched.
- (b) If extant, the calling process's process keyring is searched.
+ b) If extant, the calling process's process keyring is searched.
- (c) The calling process's session keyring is searched.
+ c) The calling process's session keyring is searched.
The moment one succeeds, all pending errors are discarded and the found key is
returned.
Only if all these fail does the whole thing fail with the highest priority
error. Note that several errors may have come from LSM.
-The error priority is:
+The error priority is::
EKEYREVOKED > EKEYEXPIRED > ENOKEY
- Trusted and Encrypted Keys
+==========================
+Trusted and Encrypted Keys
+==========================
Trusted and Encrypted Keys are two new key types added to the existing kernel
key ring service. Both of these new types are variable length symmetric keys,
authorization value (20 zeros). This can be set at takeownership time with the
trouser's utility: "tpm_takeownership -u -z".
-Usage:
+Usage::
+
keyctl add trusted name "new keylen [options]" ring
keyctl add trusted name "load hex_blob [pcrlock=pcrnum]" ring
keyctl update key "update [options]"
key or a more complex structure. The format of the more complex structure is
application specific, which is identified by 'format'.
-Usage:
+Usage::
+
keyctl add encrypted name "new [format] key-type:master-key-name keylen"
ring
keyctl add encrypted name "load hex_blob" ring
keyctl update keyid "update key-type:master-key-name"
-format:= 'default | ecryptfs'
-key-type:= 'trusted' | 'user'
+Where::
+
+ format:= 'default | ecryptfs'
+ key-type:= 'trusted' | 'user'
Examples of trusted and encrypted key usage:
-Create and save a trusted key named "kmk" of length 32 bytes:
+Create and save a trusted key named "kmk" of length 32 bytes::
$ keyctl add trusted kmk "new 32" @u
440502848
$ keyctl pipe 440502848 > kmk.blob
-Load a trusted key from the saved blob:
+Load a trusted key from the saved blob::
$ keyctl add trusted kmk "load `cat kmk.blob`" @u
268728824
f1f8fff03ad0acb083725535636addb08d73dedb9832da198081e5deae84bfaf0409c22b
e4a8aea2b607ec96931e6f4d4fe563ba
-Reseal a trusted key under new pcr values:
+Reseal a trusted key under new pcr values::
$ keyctl update 268728824 "update pcrinfo=`cat pcr.blob`"
$ keyctl print 268728824
values, protects against boot and offline attacks. Create and save an
encrypted key "evm" using the above trusted key "kmk":
-option 1: omitting 'format'
+option 1: omitting 'format'::
+
$ keyctl add encrypted evm "new trusted:kmk 32" @u
159771175
-option 2: explicitly defining 'format' as 'default'
+option 2: explicitly defining 'format' as 'default'::
+
$ keyctl add encrypted evm "new default trusted:kmk 32" @u
159771175
$ keyctl pipe 159771175 > evm.blob
-Load an encrypted key "evm" from saved blob:
+Load an encrypted key "evm" from saved blob::
$ keyctl add encrypted evm "load `cat evm.blob`" @u
831684262
are anticipated. In particular the new format 'ecryptfs' has been defined in
in order to use encrypted keys to mount an eCryptfs filesystem. More details
about the usage can be found in the file
-'Documentation/security/keys-ecryptfs.txt'.
+``Documentation/security/keys-ecryptfs.txt``.
-# Kernel Self-Protection
+======================
+Kernel Self-Protection
+======================
Kernel self-protection is the design and implementation of systems and
structures within the Linux kernel to protect against security flaws in
and/or accepted.
-## Attack Surface Reduction
+Attack Surface Reduction
+========================
The most fundamental defense against security exploits is to reduce the
areas of the kernel that can be used to redirect execution. This ranges
APIs hard to use incorrectly, minimizing the areas of writable kernel
memory, etc.
-### Strict kernel memory permissions
+Strict kernel memory permissions
+--------------------------------
When all of kernel memory is writable, it becomes trivial for attacks
to redirect execution flow. To reduce the availability of these targets
the kernel needs to protect its memory with a tight set of permissions.
-#### Executable code and read-only data must not be writable
+Executable code and read-only data must not be writable
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Any areas of the kernel with executable memory must not be writable.
While this obviously includes the kernel text itself, we must consider
made writable during the update, and then returned to the original
permissions.)
-In support of this are CONFIG_STRICT_KERNEL_RWX and
-CONFIG_STRICT_MODULE_RWX, which seek to make sure that code is not
+In support of this are ``CONFIG_STRICT_KERNEL_RWX`` and
+``CONFIG_STRICT_MODULE_RWX``, which seek to make sure that code is not
writable, data is not executable, and read-only data is neither writable
nor executable.
Most architectures have these options on by default and not user selectable.
For some architectures like arm that wish to have these be selectable,
the architecture Kconfig can select ARCH_OPTIONAL_KERNEL_RWX to enable
-a Kconfig prompt. CONFIG_ARCH_OPTIONAL_KERNEL_RWX_DEFAULT determines
+a Kconfig prompt. ``CONFIG_ARCH_OPTIONAL_KERNEL_RWX_DEFAULT`` determines
the default setting when ARCH_OPTIONAL_KERNEL_RWX is enabled.
-#### Function pointers and sensitive variables must not be writable
+Function pointers and sensitive variables must not be writable
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Vast areas of kernel memory contain function pointers that are looked
up by the kernel and used to continue execution (e.g. descriptor/vector
of the kernel, gaining the protection of the kernel's strict memory
permissions as described above.
-For variables that are initialized once at __init time, these can
-be marked with the (new and under development) __ro_after_init
+For variables that are initialized once at ``__init`` time, these can
+be marked with the (new and under development) ``__ro_after_init``
attribute.
What remains are variables that are updated rarely (e.g. GDT). These
CPU thread performing the update would be given uninterruptible write
access to the memory.)
-#### Segregation of kernel memory from userspace memory
+Segregation of kernel memory from userspace memory
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The kernel must never execute userspace memory. The kernel must also never
access userspace memory without explicit expectation to do so. These
cannot be passed to trivially-controlled userspace memory, forcing
attacks to operate entirely in kernel memory.
-### Reduced access to syscalls
+Reduced access to syscalls
+--------------------------
One trivial way to eliminate many syscalls for 64-bit systems is building
-without CONFIG_COMPAT. However, this is rarely a feasible scenario.
+without ``CONFIG_COMPAT``. However, this is rarely a feasible scenario.
The "seccomp" system provides an opt-in feature made available to
userspace, which provides a way to reduce the number of kernel entry
restricted to the more regular set of normally available to unprivileged
userspace.
-### Restricting access to kernel modules
+Restricting access to kernel modules
+------------------------------------
The kernel should never allow an unprivileged user the ability to
load specific kernel modules, since that would provide a facility to
To protect against even privileged users, systems may need to either
disable module loading entirely (e.g. monolithic kernel builds or
modules_disabled sysctl), or provide signed modules (e.g.
-CONFIG_MODULE_SIG_FORCE, or dm-crypt with LoadPin), to keep from having
+``CONFIG_MODULE_SIG_FORCE``, or dm-crypt with LoadPin), to keep from having
root load arbitrary kernel code via the module loader interface.
-## Memory integrity
+Memory integrity
+================
There are many memory structures in the kernel that are regularly abused
to gain execution control during an attack, By far the most commonly
address stored on the stack is overwritten. Many other examples of this
kind of attack exist, and protections exist to defend against them.
-### Stack buffer overflow
+Stack buffer overflow
+---------------------
The classic stack buffer overflow involves writing past the expected end
of a variable stored on the stack, ultimately writing a controlled value
to the stack frame's stored return address. The most widely used defense
is the presence of a stack canary between the stack variables and the
-return address (CONFIG_CC_STACKPROTECTOR), which is verified just before
+return address (``CONFIG_CC_STACKPROTECTOR``), which is verified just before
the function returns. Other defenses include things like shadow stacks.
-### Stack depth overflow
+Stack depth overflow
+--------------------
A less well understood attack is using a bug that triggers the
kernel to consume stack memory with deep function calls or large stack
sensitive thread_info structure elsewhere, and adding a faulting memory
hole at the bottom of the stack to catch these overflows.
-### Heap memory integrity
+Heap memory integrity
+---------------------
The structures used to track heap free lists can be sanity-checked during
allocation and freeing to make sure they aren't being used to manipulate
other memory areas.
-### Counter integrity
+Counter integrity
+-----------------
Many places in the kernel use atomic counters to track object references
or perform similar lifetime management. When these counters can be made
to wrap (over or under) this traditionally exposes a use-after-free
flaw. By trapping atomic wrapping, this class of bug vanishes.
-### Size calculation overflow detection
+Size calculation overflow detection
+-----------------------------------
Similar to counter overflow, integer overflows (usually size calculations)
need to be detected at runtime to kill this class of bug, which
traditionally leads to being able to write past the end of kernel buffers.
-## Statistical defenses
+Probabilistic defenses
+======================
While many protections can be considered deterministic (e.g. read-only
memory cannot be written to), some protections provide only statistical
running system to overcome the defense. While not perfect, these do
provide meaningful defenses.
-### Canaries, blinding, and other secrets
+Canaries, blinding, and other secrets
+-------------------------------------
It should be noted that things like the stack canary discussed earlier
are technically statistical defenses, since they rely on a secret value,
different canary per stack) and high entropy (e.g. is the RNG actually
working?) in order to maximize their success.
-### Kernel Address Space Layout Randomization (KASLR)
+Kernel Address Space Layout Randomization (KASLR)
+-------------------------------------------------
Since the location of kernel memory is almost always instrumental in
mounting a successful attack, making the location non-deterministic
the value of information exposures higher, since they may be used to
discover desired memory locations.)
-#### Text and module base
+Text and module base
+~~~~~~~~~~~~~~~~~~~~
By relocating the physical and virtual base address of the kernel at
-boot-time (CONFIG_RANDOMIZE_BASE), attacks needing kernel code will be
+boot-time (``CONFIG_RANDOMIZE_BASE``), attacks needing kernel code will be
frustrated. Additionally, offsetting the module loading base address
means that even systems that load the same set of modules in the same
order every boot will not share a common base address with the rest of
the kernel text.
-#### Stack base
+Stack base
+~~~~~~~~~~
If the base address of the kernel stack is not the same between processes,
or even not the same between syscalls, targets on or beyond the stack
become more difficult to locate.
-#### Dynamic memory base
+Dynamic memory base
+~~~~~~~~~~~~~~~~~~~
Much of the kernel's dynamic memory (e.g. kmalloc, vmalloc, etc) ends up
being relatively deterministic in layout due to the order of early-boot
between boots, targeting them is frustrated, requiring an information
exposure specific to the region.
-#### Structure layout
+Structure layout
+~~~~~~~~~~~~~~~~
By performing a per-build randomization of the layout of sensitive
structures, attacks must either be tuned to known kernel builds or expose
them.
-## Preventing Information Exposures
+Preventing Information Exposures
+================================
Since the locations of sensitive structures are the primary target for
attacks, it is important to defend against exposure of both kernel memory
addresses and kernel memory contents (since they may contain kernel
addresses or other sensitive things like canary values).
-### Unique identifiers
+Unique identifiers
+------------------
Kernel memory addresses must never be used as identifiers exposed to
userspace. Instead, use an atomic counter, an idr, or similar unique
identifier.
-### Memory initialization
+Memory initialization
+---------------------
Memory copied to userspace must always be fully initialized. If not
explicitly memset(), this will require changes to the compiler to make
sure structure holes are cleared.
-### Memory poisoning
+Memory poisoning
+----------------
When releasing memory, it is best to poison the contents (clear stack on
syscall return, wipe heap memory on a free), to avoid reuse attacks that
variable attacks, stack content exposures, heap content exposures, and
use-after-free attacks.
-### Destination tracking
+Destination tracking
+--------------------
To help kill classes of bugs that result in kernel addresses being
written to userspace, the destination of writes needs to be tracked. If
-the buffer is destined for userspace (e.g. seq_file backed /proc files),
+the buffer is destined for userspace (e.g. seq_file backed ``/proc`` files),
it should automatically censor sensitive values.
6- first close all virtual ports before closing the physical port.
+Note that after closing the physical port the modem is still in multiplexing
+mode. This may prevent a successful re-opening of the port later. To avoid
+this situation either reset the modem if your hardware allows that or send
+a disconnect command frame manually before initializing the multiplexing mode
+for the second time. The byte sequence for the disconnect command frame is:
+0xf9, 0x03, 0xef, 0x03, 0xc3, 0x16, 0xf9.
+
Additional Documentation
------------------------
More practical details on the protocol and how it's supported by industrial
--- /dev/null
+# -*- coding: utf-8; mode: python -*-
+
+project = "SuperH architecture implementation manual"
+
+tags.add("subproject")
+
+latex_documents = [
+ ('index', 'sh.tex', project,
+ 'The kernel development community', 'manual'),
+]
--- /dev/null
+=======================
+SuperH Interfaces Guide
+=======================
+
+:Author: Paul Mundt
+
+Memory Management
+=================
+
+SH-4
+----
+
+Store Queue API
+~~~~~~~~~~~~~~~
+
+.. kernel-doc:: arch/sh/kernel/cpu/sh4/sq.c
+ :export:
+
+SH-5
+----
+
+TLB Interfaces
+~~~~~~~~~~~~~~
+
+.. kernel-doc:: arch/sh/mm/tlb-sh5.c
+ :internal:
+
+.. kernel-doc:: arch/sh/include/asm/tlb_64.h
+ :internal:
+
+Machine Specific Interfaces
+===========================
+
+mach-dreamcast
+--------------
+
+.. kernel-doc:: arch/sh/boards/mach-dreamcast/rtc.c
+ :internal:
+
+mach-x3proto
+------------
+
+.. kernel-doc:: arch/sh/boards/mach-x3proto/ilsel.c
+ :export:
+
+Busses
+======
+
+SuperHyway
+----------
+
+.. kernel-doc:: drivers/sh/superhyway/superhyway.c
+ :export:
+
+Maple
+-----
+
+.. kernel-doc:: drivers/sh/maple/maple.c
+ :export:
--- /dev/null
+# -*- coding: utf-8; mode: python -*-
+
+project = "Linux Sound Subsystem Documentation"
+
+tags.add("subproject")
+
+latex_documents = [
+ ('index', 'sound.tex', project,
+ 'The kernel development community', 'manual'),
+]
+++ /dev/null
-#
-# Pandoc doesn't grok <function> or <structname>, so convert them
-# ahead of time.
-#
-# Use the following escapes to pass through pandoc:
-# $bq = "`"
-# $lt = "<"
-# $gt = ">"
-#
-s%<function>\([^<(]\+\)()</function>%:c:func:$bq\1()$bq%g
-s%<function>\([^<(]\+\)</function>%:c:func:$bq\1()$bq%g
-s%<structname>struct *\([^<]\+\)</structname>%:c:type:$bqstruct \1 $lt\1$gt$bq%g
-s%struct <structname>\([^<]\+\)</structname>%:c:type:$bqstruct \1 $lt\1$gt$bq%g
-s%<structname>\([^<]\+\)</structname>%:c:type:$bqstruct \1 $lt\1$gt$bq%g
-#
-# Wrap docproc directives in para and code blocks.
-#
-s%^\(!.*\)$%<para><code>DOCPROC: \1</code></para>%
+++ /dev/null
-#
-# Unescape.
-#
-s/$bq/`/g
-s/$lt/</g
-s/$gt/>/g
-#
-# pandoc thinks that both "_" needs to be escaped. Remove the extra
-# backslashes.
-#
-s/\\_/_/g
-#
-# Unwrap docproc directives.
-#
-s/^``DOCPROC: !E\(.*\)``$/.. kernel-doc:: \1\n :export:/
-s/^``DOCPROC: !I\(.*\)``$/.. kernel-doc:: \1\n :internal:/
-s/^``DOCPROC: !F\([^ ]*\) \(.*\)``$/.. kernel-doc:: \1\n :functions: \2/
-s/^``DOCPROC: !P\([^ ]*\) \(.*\)``$/.. kernel-doc:: \1\n :doc: \2/
-s/^``DOCPROC: \(!.*\)``$/.. WARNING: DOCPROC directive not supported: \1/
-#
-# Trim trailing whitespace.
-#
-s/[[:space:]]*$//
+++ /dev/null
-#!/bin/bash
-#
-# Convert a template file into something like RST
-#
-# fix <function>
-# feed to pandoc
-# fix \_
-# title line?
-#
-set -eu
-
-if [ "$#" != "2" ]; then
- echo "$0 <docbook file> <rst file>"
- exit
-fi
-
-DIR=$(dirname $0)
-
-in=$1
-rst=$2
-tmp=$rst.tmp
-
-cp $in $tmp
-sed --in-place -f $DIR/convert_template.sed $tmp
-pandoc -s -S -f docbook -t rst -o $rst $tmp
-sed --in-place -f $DIR/post_convert.sed $rst
-rm $tmp
-echo "book writen to $rst"
(That data line is sometimes called MOMI or SISO.)
Microcontrollers often support both master and slave sides of the SPI
-protocol. This document (and Linux) currently only supports the master
-side of SPI interactions.
+protocol. This document (and Linux) supports both the master and slave
+sides of SPI interactions.
Who uses it? On what kinds of systems?
or monitor temperature and voltage levels during industrial processing.
And those might all be sharing the same controller driver.
-A "struct spi_device" encapsulates the master-side interface between
-those two types of driver. At this writing, Linux has no slave side
-programming interface.
+A "struct spi_device" encapsulates the controller-side interface between
+those two types of drivers.
There is a minimal core of SPI programming interfaces, focussing on
using the driver model to connect controller and protocol drivers using
/sys/bus/spi/drivers/D ... driver for one or more spi*.* devices
/sys/class/spi_master/spiB ... symlink (or actual device node) to
- a logical node which could hold class related state for the
- controller managing bus "B". All spiB.* devices share one
+ a logical node which could hold class related state for the SPI
+ master controller managing bus "B". All spiB.* devices share one
physical SPI bus segment, with SCLK, MOSI, and MISO.
+ /sys/devices/.../CTLR/slave ... virtual file for (un)registering the
+ slave device for an SPI slave controller.
+ Writing the driver name of an SPI slave handler to this file
+ registers the slave device; writing "(null)" unregisters the slave
+ device.
+ Reading from this file shows the name of the slave device ("(null)"
+ if not registered).
+
+ /sys/class/spi_slave/spiB ... symlink (or actual device node) to
+ a logical node which could hold class related state for the SPI
+ slave controller on bus "B". When registered, a single spiB.*
+ device is present here, possible sharing the physical SPI bus
+ segment with other SPI slave devices.
+
Note that the actual location of the controller's class state depends
on whether you enabled CONFIG_SYSFS_DEPRECATED or not. At this time,
the only class-specific state is the bus number ("B" in "spiB"), so
Another approach is to offload RCU callback processing to "rcuo" kthreads
using the CONFIG_RCU_NOCB_CPU=y Kconfig option. The specific CPUs to
-offload may be selected via several methods:
-
-1. One of three mutually exclusive Kconfig options specify a
- build-time default for the CPUs to offload:
-
- a. The CONFIG_RCU_NOCB_CPU_NONE=y Kconfig option results in
- no CPUs being offloaded.
-
- b. The CONFIG_RCU_NOCB_CPU_ZERO=y Kconfig option causes
- CPU 0 to be offloaded.
-
- c. The CONFIG_RCU_NOCB_CPU_ALL=y Kconfig option causes all
- CPUs to be offloaded. Note that the callbacks will be
- offloaded to "rcuo" kthreads, and that those kthreads
- will in fact run on some CPU. However, this approach
- gives fine-grained control on exactly which CPUs the
- callbacks run on, along with their scheduling priority
- (including the default of SCHED_OTHER), and it further
- allows this control to be varied dynamically at runtime.
-
-2. The "rcu_nocbs=" kernel boot parameter, which takes a comma-separated
- list of CPUs and CPU ranges, for example, "1,3-5" selects CPUs 1,
- 3, 4, and 5. The specified CPUs will be offloaded in addition to
- any CPUs specified as offloaded by CONFIG_RCU_NOCB_CPU_ZERO=y or
- CONFIG_RCU_NOCB_CPU_ALL=y. This means that the "rcu_nocbs=" boot
- parameter has no effect for kernels built with RCU_NOCB_CPU_ALL=y.
+offload may be selected using The "rcu_nocbs=" kernel boot parameter,
+which takes a comma-separated list of CPUs and CPU ranges, for example,
+"1,3-5" selects CPUs 1, 3, 4, and 5.
The offloaded CPUs will never queue RCU callbacks, and therefore RCU
never prevents offloaded CPUs from entering either dyntick-idle mode
--- /dev/null
+ Coresight CPU Debug Module
+ ==========================
+
+ Author: Leo Yan <leo.yan@linaro.org>
+ Date: April 5th, 2017
+
+Introduction
+------------
+
+Coresight CPU debug module is defined in ARMv8-a architecture reference manual
+(ARM DDI 0487A.k) Chapter 'Part H: External debug', the CPU can integrate
+debug module and it is mainly used for two modes: self-hosted debug and
+external debug. Usually the external debug mode is well known as the external
+debugger connects with SoC from JTAG port; on the other hand the program can
+explore debugging method which rely on self-hosted debug mode, this document
+is to focus on this part.
+
+The debug module provides sample-based profiling extension, which can be used
+to sample CPU program counter, secure state and exception level, etc; usually
+every CPU has one dedicated debug module to be connected. Based on self-hosted
+debug mechanism, Linux kernel can access these related registers from mmio
+region when the kernel panic happens. The callback notifier for kernel panic
+will dump related registers for every CPU; finally this is good for assistant
+analysis for panic.
+
+
+Implementation
+--------------
+
+- During driver registration, it uses EDDEVID and EDDEVID1 - two device ID
+ registers to decide if sample-based profiling is implemented or not. On some
+ platforms this hardware feature is fully or partially implemented; and if
+ this feature is not supported then registration will fail.
+
+- At the time this documentation was written, the debug driver mainly relies on
+ information gathered by the kernel panic callback notifier from three
+ sampling registers: EDPCSR, EDVIDSR and EDCIDSR: from EDPCSR we can get
+ program counter; EDVIDSR has information for secure state, exception level,
+ bit width, etc; EDCIDSR is context ID value which contains the sampled value
+ of CONTEXTIDR_EL1.
+
+- The driver supports a CPU running in either AArch64 or AArch32 mode. The
+ registers naming convention is a bit different between them, AArch64 uses
+ 'ED' for register prefix (ARM DDI 0487A.k, chapter H9.1) and AArch32 uses
+ 'DBG' as prefix (ARM DDI 0487A.k, chapter G5.1). The driver is unified to
+ use AArch64 naming convention.
+
+- ARMv8-a (ARM DDI 0487A.k) and ARMv7-a (ARM DDI 0406C.b) have different
+ register bits definition. So the driver consolidates two difference:
+
+ If PCSROffset=0b0000, on ARMv8-a the feature of EDPCSR is not implemented;
+ but ARMv7-a defines "PCSR samples are offset by a value that depends on the
+ instruction set state". For ARMv7-a, the driver checks furthermore if CPU
+ runs with ARM or thumb instruction set and calibrate PCSR value, the
+ detailed description for offset is in ARMv7-a ARM (ARM DDI 0406C.b) chapter
+ C11.11.34 "DBGPCSR, Program Counter Sampling Register".
+
+ If PCSROffset=0b0010, ARMv8-a defines "EDPCSR implemented, and samples have
+ no offset applied and do not sample the instruction set state in AArch32
+ state". So on ARMv8 if EDDEVID1.PCSROffset is 0b0010 and the CPU operates
+ in AArch32 state, EDPCSR is not sampled; when the CPU operates in AArch64
+ state EDPCSR is sampled and no offset are applied.
+
+
+Clock and power domain
+----------------------
+
+Before accessing debug registers, we should ensure the clock and power domain
+have been enabled properly. In ARMv8-a ARM (ARM DDI 0487A.k) chapter 'H9.1
+Debug registers', the debug registers are spread into two domains: the debug
+domain and the CPU domain.
+
+ +---------------+
+ | |
+ | |
+ +----------+--+ |
+ dbg_clock -->| |**| |<-- cpu_clock
+ | Debug |**| CPU |
+ dbg_power_domain -->| |**| |<-- cpu_power_domain
+ +----------+--+ |
+ | |
+ | |
+ +---------------+
+
+For debug domain, the user uses DT binding "clocks" and "power-domains" to
+specify the corresponding clock source and power supply for the debug logic.
+The driver calls the pm_runtime_{put|get} operations as needed to handle the
+debug power domain.
+
+For CPU domain, the different SoC designs have different power management
+schemes and finally this heavily impacts external debug module. So we can
+divide into below cases:
+
+- On systems with a sane power controller which can behave correctly with
+ respect to CPU power domain, the CPU power domain can be controlled by
+ register EDPRCR in driver. The driver firstly writes bit EDPRCR.COREPURQ
+ to power up the CPU, and then writes bit EDPRCR.CORENPDRQ for emulation
+ of CPU power down. As result, this can ensure the CPU power domain is
+ powered on properly during the period when access debug related registers;
+
+- Some designs will power down an entire cluster if all CPUs on the cluster
+ are powered down - including the parts of the debug registers that should
+ remain powered in the debug power domain. The bits in EDPRCR are not
+ respected in these cases, so these designs do not support debug over
+ power down in the way that the CoreSight / Debug designers anticipated.
+ This means that even checking EDPRSR has the potential to cause a bus hang
+ if the target register is unpowered.
+
+ In this case, accessing to the debug registers while they are not powered
+ is a recipe for disaster; so we need preventing CPU low power states at boot
+ time or when user enable module at the run time. Please see chapter
+ "How to use the module" for detailed usage info for this.
+
+
+Device Tree Bindings
+--------------------
+
+See Documentation/devicetree/bindings/arm/coresight-cpu-debug.txt for details.
+
+
+How to use the module
+---------------------
+
+If you want to enable debugging functionality at boot time, you can add
+"coresight_cpu_debug.enable=1" to the kernel command line parameter.
+
+The driver also can work as module, so can enable the debugging when insmod
+module:
+# insmod coresight_cpu_debug.ko debug=1
+
+When boot time or insmod module you have not enabled the debugging, the driver
+uses the debugfs file system to provide a knob to dynamically enable or disable
+debugging:
+
+To enable it, write a '1' into /sys/kernel/debug/coresight_cpu_debug/enable:
+# echo 1 > /sys/kernel/debug/coresight_cpu_debug/enable
+
+To disable it, write a '0' into /sys/kernel/debug/coresight_cpu_debug/enable:
+# echo 0 > /sys/kernel/debug/coresight_cpu_debug/enable
+
+As explained in chapter "Clock and power domain", if you are working on one
+platform which has idle states to power off debug logic and the power
+controller cannot work well for the request from EDPRCR, then you should
+firstly constraint CPU idle states before enable CPU debugging feature; so can
+ensure the accessing to debug logic.
+
+If you want to limit idle states at boot time, you can use "nohlt" or
+"cpuidle.off=1" in the kernel command line.
+
+At the runtime you can disable idle states with below methods:
+
+Set latency request to /dev/cpu_dma_latency to disable all CPUs specific idle
+states (if latency = 0uS then disable all idle states):
+# echo "what_ever_latency_you_need_in_uS" > /dev/cpu_dma_latency
+
+Disable specific CPU's specific idle state:
+# echo 1 > /sys/devices/system/cpu/cpu$cpu/cpuidle/state$state/disable
+
+
+Output format
+-------------
+
+Here is an example of the debugging output format:
+
+ARM external debug module:
+coresight-cpu-debug 850000.debug: CPU[0]:
+coresight-cpu-debug 850000.debug: EDPRSR: 00000001 (Power:On DLK:Unlock)
+coresight-cpu-debug 850000.debug: EDPCSR: [<ffff00000808e9bc>] handle_IPI+0x174/0x1d8
+coresight-cpu-debug 850000.debug: EDCIDSR: 00000000
+coresight-cpu-debug 850000.debug: EDVIDSR: 90000000 (State:Non-secure Mode:EL1/0 Width:64bits VMID:0)
+coresight-cpu-debug 852000.debug: CPU[1]:
+coresight-cpu-debug 852000.debug: EDPRSR: 00000001 (Power:On DLK:Unlock)
+coresight-cpu-debug 852000.debug: EDPCSR: [<ffff0000087fab34>] debug_notifier_call+0x23c/0x358
+coresight-cpu-debug 852000.debug: EDCIDSR: 00000000
+coresight-cpu-debug 852000.debug: EDVIDSR: 90000000 (State:Non-secure Mode:EL1/0 Width:64bits VMID:0)
<idle>-0 3dN.2 14us : sched_avg_update <-__cpu_load_update
<idle>-0 3dN.2 14us : _raw_spin_unlock <-cpu_load_update_nohz
<idle>-0 3dN.2 14us : sub_preempt_count <-_raw_spin_unlock
- <idle>-0 3dN.1 15us : calc_load_exit_idle <-tick_nohz_idle_exit
+ <idle>-0 3dN.1 15us : calc_load_nohz_stop <-tick_nohz_idle_exit
<idle>-0 3dN.1 15us : touch_softlockup_watchdog <-tick_nohz_idle_exit
<idle>-0 3dN.1 15us : hrtimer_cancel <-tick_nohz_idle_exit
<idle>-0 3dN.1 15us : hrtimer_try_to_cancel <-hrtimer_cancel
make latexdocs
make epubdocs
-現在、幾つかの DocBook形式で書かれたドキュメントは ReST形式に転換中で
-す。それらのドキュメントはDocumentation/DocBook ディレクトリに生成され、
-Postscript または man ページの形式を生成するには以下のようにします - ::
-
- make psdocs
- make mandocs
-
カーネル開発者になるには
------------------------
make latexdocs
make epubdocs
-현재, ReST 로의 변환이 진행중인, DocBook 으로 쓰인 문서들이 존재한다. 그런
-문서들은 Documentation/DocBook/ 디렉토리 안에 생성될 것이고 다음 커맨드를 통해
-Postscript 나 man page 로도 만들어질 수 있다::
-
- make psdocs
- make mandocs
-
커널 개발자가 되는 것
---------------------
선호되는 방법은 git(커널의 소스 관리 툴, 더 많은 정보들은
https://git-scm.com/ 에서 참조할 수 있다)를 사용하는 것이지만 순수한
패치파일의 형식으로 보내는 것도 무관하다.
- - 2주 후에 -rc1 커널이 배포되며 지금부터는 전체 커널의 안정성에 영향을
- 미칠수 있는 새로운 기능들을 포함하지 않는 패치들만이 추가될 수 있다.
- 완전히 새로운 드라이버(혹은 파일시스템)는 -rc1 이후에만 받아들여진다는
- 것을 기억해라. 왜냐하면 변경이 자체내에서만 발생하고 추가된 코드가
- 드라이버 외부의 다른 부분에는 영향을 주지 않으므로 그런 변경은
+ - 2주 후에 -rc1 커널이 릴리즈되며 여기서부터의 주안점은 새로운 커널을
+ 가능한한 안정되게 하는 것이다. 이 시점에서의 대부분의 패치들은
회귀(역자주: 이전에는 존재하지 않았지만 새로운 기능추가나 변경으로 인해
- 생겨난 버그)를 일으킬 만한 위험을 가지고 있지 않기 때문이다. -rc1이
- 배포된 이후에 git를 사용하여 패치들을 Linus에게 보낼수 있지만 패치들은
- 공식적인 메일링 리스트로 보내서 검토를 받을 필요가 있다.
+ 생겨난 버그)를 고쳐야 한다. 이전부터 존재한 버그는 회귀가 아니므로, 그런
+ 버그에 대한 수정사항은 중요한 경우에만 보내져야 한다. 완전히 새로운
+ 드라이버(혹은 파일시스템)는 -rc1 이후에만 받아들여진다는 것을 기억해라.
+ 왜냐하면 변경이 자체내에서만 발생하고 추가된 코드가 드라이버 외부의 다른
+ 부분에는 영향을 주지 않으므로 그런 변경은 회귀를 일으킬 만한 위험을 가지고
+ 있지 않기 때문이다. -rc1이 배포된 이후에 git를 사용하여 패치들을 Linus에게
+ 보낼수 있지만 패치들은 공식적인 메일링 리스트로 보내서 검토를 받을 필요가
+ 있다.
- 새로운 -rc는 Linus가 현재 git tree가 테스트 하기에 충분히 안정된 상태에
있다고 판단될 때마다 배포된다. 목표는 새로운 -rc 커널을 매주 배포하는
것이다.
버그 보고
---------
-https://bugzilla.kernel.org는 리눅스 커널 개발자들이 커널의 버그를 추적하는
+https://bugzilla.kernel.org 는 리눅스 커널 개발자들이 커널의 버그를 추적하는
곳이다. 사용자들은 발견한 모든 버그들을 보고하기 위하여 이 툴을 사용할 것을
권장한다. kernel bugzilla를 사용하는 자세한 방법은 다음을 참조하라.
위의 코드를 아래와 같이 바꿔버릴 수 있습니다:
q = READ_ONCE(a);
- WRITE_ONCE(b, 1);
+ WRITE_ONCE(b, 2);
do_something_else();
이렇게 되면, CPU 는 변수 'a' 로부터의 로드와 변수 'b' 로의 스토어 사이의 순서를
13. RNDIS function
14. SERIAL function
15. SOURCESINK function
-16. UAC1 function
+16. UAC1 function (legacy implementation)
17. UAC2 function
18. UVC function
19. PRINTER function
+20. UAC1 function (new API)
1. ACM function
host: test-usb (tools/usb/testusb.c)
-16. UAC1 function
+16. UAC1 function (legacy implementation)
=================
-The function is provided by usb_f_uac1.ko module.
+The function is provided by usb_f_uac1_legacy.ko module.
Function-specific configfs interface
------------------------------------
-The function name to use when creating the function directory is "uac1".
+The function name to use when creating the function directory
+is "uac1_legacy".
The uac1 function provides these attributes in its function directory:
audio_buf_size - audio buffer size
More advanced testing can be done with the prn_example
described in Documentation/usb/gadget-printer.txt.
+
+
+20. UAC1 function (virtual ALSA card, using u_audio API)
+=================
+
+The function is provided by usb_f_uac1.ko module.
+It will create a virtual ALSA card and the audio streams are simply
+sinked to and sourced from it.
+
+Function-specific configfs interface
+------------------------------------
+
+The function name to use when creating the function directory is "uac1".
+The uac1 function provides these attributes in its function directory:
+
+ c_chmask - capture channel mask
+ c_srate - capture sampling rate
+ c_ssize - capture sample size (bytes)
+ p_chmask - playback channel mask
+ p_srate - playback sampling rate
+ p_ssize - playback sample size (bytes)
+ req_number - the number of pre-allocated request for both capture
+ and playback
+
+The attributes have sane default values.
+
+Testing the UAC1 function
+-------------------------
+
+device: run the gadget
+host: aplay -l # should list our USB Audio Gadget
+
+This function does not require real hardware support, it just
+sends a stream of audio data to/from the host. In order to
+actually hear something at the device side, a command similar
+to this must be used at the device side:
+
+$ arecord -f dat -t wav -D hw:2,0 | aplay -D hw:0,0 &
+
+e.g.:
+
+$ arecord -f dat -t wav -D hw:CARD=UAC1Gadget,DEV=0 | \
+aplay -D default:CARD=OdroidU3
.. toctree::
:maxdepth: 2
+ no_new_privs
+ seccomp_filter
unshare
.. only:: subproject and html
+======================
+No New Privileges Flag
+======================
+
The execve system call can grant a newly-started program privileges that
its parent did not have. The most obvious examples are setuid/setgid
programs and file capabilities. To prevent the parent program from
careful to prevent the parent from doing anything that could subvert the
child. For example:
- - The dynamic loader handles LD_* environment variables differently if
+ - The dynamic loader handles ``LD_*`` environment variables differently if
a program is setuid.
- chroot is disallowed to unprivileged processes, since it would allow
- /etc/passwd to be replaced from the point of view of a process that
+ ``/etc/passwd`` to be replaced from the point of view of a process that
inherited chroot.
- The exec code has special handling for ptrace.
-These are all ad-hoc fixes. The no_new_privs bit (since Linux 3.5) is a
+These are all ad-hoc fixes. The ``no_new_privs`` bit (since Linux 3.5) is a
new, generic mechanism to make it safe for a process to modify its
execution environment in a manner that persists across execve. Any task
-can set no_new_privs. Once the bit is set, it is inherited across fork,
-clone, and execve and cannot be unset. With no_new_privs set, execve
+can set ``no_new_privs``. Once the bit is set, it is inherited across fork,
+clone, and execve and cannot be unset. With ``no_new_privs`` set, ``execve()``
promises not to grant the privilege to do anything that could not have
been done without the execve call. For example, the setuid and setgid
bits will no longer change the uid or gid; file capabilities will not
add to the permitted set, and LSMs will not relax constraints after
execve.
-To set no_new_privs, use prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0).
+To set ``no_new_privs``, use::
+
+ prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0);
Be careful, though: LSMs might also not tighten constraints on exec
-in no_new_privs mode. (This means that setting up a general-purpose
-service launcher to set no_new_privs before execing daemons may
+in ``no_new_privs`` mode. (This means that setting up a general-purpose
+service launcher to set ``no_new_privs`` before execing daemons may
interfere with LSM-based sandboxing.)
-Note that no_new_privs does not prevent privilege changes that do not
-involve execve. An appropriately privileged task can still call
-setuid(2) and receive SCM_RIGHTS datagrams.
+Note that ``no_new_privs`` does not prevent privilege changes that do not
+involve ``execve()``. An appropriately privileged task can still call
+``setuid(2)`` and receive SCM_RIGHTS datagrams.
-There are two main use cases for no_new_privs so far:
+There are two main use cases for ``no_new_privs`` so far:
- Filters installed for the seccomp mode 2 sandbox persist across
execve and can change the behavior of newly-executed programs.
Unprivileged users are therefore only allowed to install such filters
- if no_new_privs is set.
+ if ``no_new_privs`` is set.
- - By itself, no_new_privs can be used to reduce the attack surface
+ - By itself, ``no_new_privs`` can be used to reduce the attack surface
available to an unprivileged user. If everything running with a
- given uid has no_new_privs set, then that uid will be unable to
+ given uid has ``no_new_privs`` set, then that uid will be unable to
escalate its privileges by directly attacking setuid, setgid, and
fcap-using binaries; it will need to compromise something without the
- no_new_privs bit set first.
+ ``no_new_privs`` bit set first.
In the future, other potentially dangerous kernel features could become
-available to unprivileged tasks if no_new_privs is set. In principle,
-several options to unshare(2) and clone(2) would be safe when
-no_new_privs is set, and no_new_privs + chroot is considerable less
+available to unprivileged tasks if ``no_new_privs`` is set. In principle,
+several options to ``unshare(2)`` and ``clone(2)`` would be safe when
+``no_new_privs`` is set, and ``no_new_privs`` + ``chroot`` is considerable less
dangerous than chroot by itself.
- SECure COMPuting with filters
- =============================
+===========================================
+Seccomp BPF (SECure COMPuting with filters)
+===========================================
Introduction
-------------
+============
A large number of system calls are exposed to every userland process
with many of them going unused for the entire lifetime of the process.
call arguments directly.
What it isn't
--------------
+=============
System call filtering isn't a sandbox. It provides a clearly defined
mechanism for minimizing the exposed kernel surface. It is meant to be
construed, incorrectly, as a more complete sandboxing solution.
Usage
------
+=====
An additional seccomp mode is added and is enabled using the same
prctl(2) call as the strict seccomp. If the architecture has
-CONFIG_HAVE_ARCH_SECCOMP_FILTER, then filters may be added as below:
+``CONFIG_HAVE_ARCH_SECCOMP_FILTER``, then filters may be added as below:
-PR_SET_SECCOMP:
+``PR_SET_SECCOMP``:
Now takes an additional argument which specifies a new filter
using a BPF program.
The BPF program will be executed over struct seccomp_data
acceptable values to inform the kernel which action should be
taken.
- Usage:
+ Usage::
+
prctl(PR_SET_SECCOMP, SECCOMP_MODE_FILTER, prog);
The 'prog' argument is a pointer to a struct sock_fprog which
will contain the filter program. If the program is invalid, the
- call will return -1 and set errno to EINVAL.
+ call will return -1 and set errno to ``EINVAL``.
- If fork/clone and execve are allowed by @prog, any child
+ If ``fork``/``clone`` and ``execve`` are allowed by @prog, any child
processes will be constrained to the same filters and system
call ABI as the parent.
- Prior to use, the task must call prctl(PR_SET_NO_NEW_PRIVS, 1) or
- run with CAP_SYS_ADMIN privileges in its namespace. If these are not
- true, -EACCES will be returned. This requirement ensures that filter
+ Prior to use, the task must call ``prctl(PR_SET_NO_NEW_PRIVS, 1)`` or
+ run with ``CAP_SYS_ADMIN`` privileges in its namespace. If these are not
+ true, ``-EACCES`` will be returned. This requirement ensures that filter
programs cannot be applied to child processes with greater privileges
than the task that installed them.
- Additionally, if prctl(2) is allowed by the attached filter,
+ Additionally, if ``prctl(2)`` is allowed by the attached filter,
additional filters may be layered on which will increase evaluation
time, but allow for further decreasing the attack surface during
execution of a process.
The above call returns 0 on success and non-zero on error.
Return values
--------------
+=============
+
A seccomp filter may return any of the following values. If multiple
filters exist, the return value for the evaluation of a given system
call will always use the highest precedent value. (For example,
-SECCOMP_RET_KILL will always take precedence.)
+``SECCOMP_RET_KILL`` will always take precedence.)
In precedence order, they are:
-SECCOMP_RET_KILL:
+``SECCOMP_RET_KILL``:
Results in the task exiting immediately without executing the
- system call. The exit status of the task (status & 0x7f) will
- be SIGSYS, not SIGKILL.
+ system call. The exit status of the task (``status & 0x7f``) will
+ be ``SIGSYS``, not ``SIGKILL``.
-SECCOMP_RET_TRAP:
- Results in the kernel sending a SIGSYS signal to the triggering
- task without executing the system call. siginfo->si_call_addr
+``SECCOMP_RET_TRAP``:
+ Results in the kernel sending a ``SIGSYS`` signal to the triggering
+ task without executing the system call. ``siginfo->si_call_addr``
will show the address of the system call instruction, and
- siginfo->si_syscall and siginfo->si_arch will indicate which
+ ``siginfo->si_syscall`` and ``siginfo->si_arch`` will indicate which
syscall was attempted. The program counter will be as though
the syscall happened (i.e. it will not point to the syscall
instruction). The return value register will contain an arch-
dependent value -- if resuming execution, set it to something
sensible. (The architecture dependency is because replacing
- it with -ENOSYS could overwrite some useful information.)
+ it with ``-ENOSYS`` could overwrite some useful information.)
- The SECCOMP_RET_DATA portion of the return value will be passed
- as si_errno.
+ The ``SECCOMP_RET_DATA`` portion of the return value will be passed
+ as ``si_errno``.
- SIGSYS triggered by seccomp will have a si_code of SYS_SECCOMP.
+ ``SIGSYS`` triggered by seccomp will have a si_code of ``SYS_SECCOMP``.
-SECCOMP_RET_ERRNO:
+``SECCOMP_RET_ERRNO``:
Results in the lower 16-bits of the return value being passed
to userland as the errno without executing the system call.
-SECCOMP_RET_TRACE:
+``SECCOMP_RET_TRACE``:
When returned, this value will cause the kernel to attempt to
- notify a ptrace()-based tracer prior to executing the system
- call. If there is no tracer present, -ENOSYS is returned to
+ notify a ``ptrace()``-based tracer prior to executing the system
+ call. If there is no tracer present, ``-ENOSYS`` is returned to
userland and the system call is not executed.
- A tracer will be notified if it requests PTRACE_O_TRACESECCOMP
- using ptrace(PTRACE_SETOPTIONS). The tracer will be notified
- of a PTRACE_EVENT_SECCOMP and the SECCOMP_RET_DATA portion of
+ A tracer will be notified if it requests ``PTRACE_O_TRACESECCOM``P
+ using ``ptrace(PTRACE_SETOPTIONS)``. The tracer will be notified
+ of a ``PTRACE_EVENT_SECCOMP`` and the ``SECCOMP_RET_DATA`` portion of
the BPF program return value will be available to the tracer
- via PTRACE_GETEVENTMSG.
+ via ``PTRACE_GETEVENTMSG``.
The tracer can skip the system call by changing the syscall number
to -1. Alternatively, the tracer can change the system call
allow use of ptrace, even of other sandboxed processes, without
extreme care; ptracers can use this mechanism to escape.)
-SECCOMP_RET_ALLOW:
+``SECCOMP_RET_ALLOW``:
Results in the system call being executed.
If multiple filters exist, the return value for the evaluation of a
given system call will always use the highest precedent value.
-Precedence is only determined using the SECCOMP_RET_ACTION mask. When
+Precedence is only determined using the ``SECCOMP_RET_ACTION`` mask. When
multiple filters return values of the same precedence, only the
-SECCOMP_RET_DATA from the most recently installed filter will be
+``SECCOMP_RET_DATA`` from the most recently installed filter will be
returned.
Pitfalls
---------
+========
The biggest pitfall to avoid during use is filtering on system call
number without checking the architecture value. Why? On any
the filters may be abused. Always check the arch value!
Example
--------
+=======
-The samples/seccomp/ directory contains both an x86-specific example
+The ``samples/seccomp/`` directory contains both an x86-specific example
and a more generic example of a higher level macro interface for BPF
program generation.
Adding architecture support
------------------------
+===========================
-See arch/Kconfig for the authoritative requirements. In general, if an
+See ``arch/Kconfig`` for the authoritative requirements. In general, if an
architecture supports both ptrace_event and seccomp, it will be able to
-support seccomp filter with minor fixup: SIGSYS support and seccomp return
-value checking. Then it must just add CONFIG_HAVE_ARCH_SECCOMP_FILTER
+support seccomp filter with minor fixup: ``SIGSYS`` support and seccomp return
+value checking. Then it must just add ``CONFIG_HAVE_ARCH_SECCOMP_FILTER``
to its arch-specific Kconfig.
Caveats
--------
+=======
The vDSO can cause some system calls to run entirely in userspace,
leading to surprises when you run programs on different machines that
fall back to real syscalls. To minimize these surprises on x86, make
sure you test with
-/sys/devices/system/clocksource/clocksource0/current_clocksource set to
-something like acpi_pm.
+``/sys/devices/system/clocksource/clocksource0/current_clocksource`` set to
+something like ``acpi_pm``.
On x86-64, vsyscall emulation is enabled by default. (vsyscalls are
-legacy variants on vDSO calls.) Currently, emulated vsyscalls will honor seccomp, with a few oddities:
+legacy variants on vDSO calls.) Currently, emulated vsyscalls will
+honor seccomp, with a few oddities:
-- A return value of SECCOMP_RET_TRAP will set a si_call_addr pointing to
+- A return value of ``SECCOMP_RET_TRAP`` will set a ``si_call_addr`` pointing to
the vsyscall entry for the given call and not the address after the
'syscall' instruction. Any code which wants to restart the call
should be aware that (a) a ret instruction has been emulated and (b)
emulation security checks, making resuming the syscall mostly
pointless.
-- A return value of SECCOMP_RET_TRACE will signal the tracer as usual,
+- A return value of ``SECCOMP_RET_TRACE`` will signal the tracer as usual,
but the syscall may not be changed to another system call using the
orig_rax register. It may only be changed to -1 order to skip the
currently emulated call. Any other change MAY terminate the process.
rip or rsp. (Do not rely on other changes terminating the process.
They might work. For example, on some kernels, choosing a syscall
that only exists in future kernels will be correctly emulated (by
- returning -ENOSYS).
+ returning ``-ENOSYS``).
-To detect this quirky behavior, check for addr & ~0x0C00 ==
-0xFFFFFFFFFF600000. (For SECCOMP_RET_TRACE, use rip. For
-SECCOMP_RET_TRAP, use siginfo->si_call_addr.) Do not check any other
+To detect this quirky behavior, check for ``addr & ~0x0C00 ==
+0xFFFFFFFFFF600000``. (For ``SECCOMP_RET_TRACE``, use rip. For
+``SECCOMP_RET_TRAP``, use ``siginfo->si_call_addr``.) Do not check any other
condition: future kernels may improve vsyscall emulation and current
kernels in vsyscall=native mode will behave differently, but the
-instructions at 0xF...F600{0,4,8,C}00 will not be system calls in these
+instructions at ``0xF...F600{0,4,8,C}00`` will not be system calls in these
cases.
Note that modern systems are unlikely to use vsyscalls at all -- they
unshare() reverses sharing that was done using clone(2) system call,
so unshare() should have a similar interface as clone(2). That is,
-since flags in clone(int flags, void *stack) specifies what should
+since flags in clone(int flags, void \*stack) specifies what should
be shared, similar flags in unshare(int flags) should specify
what should be unshared. Unfortunately, this may appear to invert
the meaning of the flags from the way they are used in clone(2).
to broadcast MCEs.
mce=bootlog
Enable logging of machine checks left over from booting.
- Disabled by default on AMD because some BIOS leave bogus ones.
+ Disabled by default on AMD Fam10h and older because some BIOS
+ leave bogus ones.
If your BIOS doesn't do that it's a good idea to enable though
to make sure you log even machine check events that result
in a reboot. On Intel systems it is enabled by default.
S: Maintained
F: Documentation/hwmon/ads1015
F: drivers/hwmon/ads1015.c
-F: include/linux/i2c/ads1015.h
+F: include/linux/platform_data/ads1015.h
ADT746X FAN DRIVER
M: Colin Leroy <colin@colino.net>
S: Maintained
F: drivers/hwtracing/coresight/*
F: Documentation/trace/coresight.txt
+F: Documentation/trace/coresight-cpu-debug.txt
F: Documentation/devicetree/bindings/arm/coresight.txt
+F: Documentation/devicetree/bindings/arm/coresight-cpu-debug.txt
F: Documentation/ABI/testing/sysfs-bus-coresight-devices-*
F: tools/perf/arch/arm/util/pmu.c
F: tools/perf/arch/arm/util/auxtrace.c
F: drivers/media/rc/st_rc.c
F: drivers/media/platform/sti/c8sectpfe/
F: drivers/mmc/host/sdhci-st.c
-F: drivers/phy/phy-miphy28lp.c
-F: drivers/phy/phy-stih407-usb.c
+F: drivers/phy/st/phy-miphy28lp.c
+F: drivers/phy/st/phy-stih407-usb.c
F: drivers/pinctrl/pinctrl-st.c
F: drivers/remoteproc/st_remoteproc.c
F: drivers/remoteproc/st_slim_rproc.c
F: drivers/input/touchscreen/atmel_mxt_ts.c
F: include/linux/platform_data/atmel_mxt_ts.h
+ATOMIC INFRASTRUCTURE
+M: Will Deacon <will.deacon@arm.com>
+M: Peter Zijlstra <peterz@infradead.org>
+R: Boqun Feng <boqun.feng@gmail.com>
+L: linux-kernel@vger.kernel.org
+S: Maintained
+F: arch/*/include/asm/atomic*.h
+F: include/*/atomic*.h
+
ATTO EXPRESSSAS SAS/SATA RAID SCSI DRIVER
M: Bradley Grove <linuxdrivers@attotech.com>
L: linux-scsi@vger.kernel.org
C6X ARCHITECTURE
M: Mark Salter <msalter@redhat.com>
-M: Aurelien Jacquiot <a-jacquiot@ti.com>
+M: Aurelien Jacquiot <jacquiot.aurelien@gmail.com>
L: linux-c6x-dev@linux-c6x.org
W: http://www.linux-c6x.org/wiki/index.php/Main_Page
S: Maintained
S: Maintained
F: Documentation/crypto/
F: Documentation/devicetree/bindings/crypto/
-F: Documentation/DocBook/crypto-API.tmpl
F: arch/*/crypto/
F: crypto/
F: drivers/crypto/
L: linux-doc@vger.kernel.org
S: Maintained
F: Documentation/
-F: scripts/docproc.c
-F: scripts/kernel-doc*
+F: scripts/kernel-doc
X: Documentation/ABI/
X: Documentation/devicetree/
X: Documentation/acpi
F: Documentation/devicetree/bindings/iio/adc/envelope-detector.txt
F: drivers/iio/adc/envelope-detector.c
+IIO MULTIPLEXER
+M: Peter Rosin <peda@axentia.se>
+L: linux-iio@vger.kernel.org
+S: Maintained
+F: Documentation/devicetree/bindings/iio/multiplexer/iio-mux.txt
+F: drivers/iio/multiplexer/iio-mux.c
+
IIO SUBSYSTEM AND DRIVERS
M: Jonathan Cameron <jic23@kernel.org>
R: Hartmut Knaack <knaack.h@gmx.de>
M: David Howells <dhowells@redhat.com>
L: keyrings@vger.kernel.org
S: Maintained
-F: Documentation/security/keys.txt
+F: Documentation/security/keys/core.rst
F: include/linux/key.h
F: include/linux/key-type.h
F: include/linux/keyctl.h
L: linux-security-module@vger.kernel.org
L: keyrings@vger.kernel.org
S: Supported
-F: Documentation/security/keys-trusted-encrypted.txt
+F: Documentation/security/keys/trusted-encrypted.rst
F: include/keys/trusted-type.h
F: security/keys/trusted.c
F: security/keys/trusted.h
L: linux-security-module@vger.kernel.org
L: keyrings@vger.kernel.org
S: Supported
-F: Documentation/security/keys-trusted-encrypted.txt
+F: Documentation/security/keys/trusted-encrypted.rst
F: include/keys/encrypted-type.h
F: security/keys/encrypted-keys/
L: kgdb-bugreport@lists.sourceforge.net
T: git git://git.kernel.org/pub/scm/linux/kernel/git/jwessel/kgdb.git
S: Maintained
-F: Documentation/DocBook/kgdb.tmpl
+F: Documentation/dev-tools/kgdb.rst
F: drivers/misc/kgdbts.c
F: drivers/tty/serial/kgdboc.c
F: include/linux/kdb.h
F: drivers/ata/sata_promise.*
LIBLOCKDEP
-M: Sasha Levin <sasha.levin@oracle.com>
+M: Sasha Levin <alexander.levin@verizon.com>
S: Maintained
F: tools/lib/lockdep/
F: drivers/power/supply/max14577_charger.c
F: drivers/power/supply/max77693_charger.c
-MAXIM MAX77802 MULTIFUNCTION PMIC DEVICE DRIVERS
-M: Javier Martinez Canillas <javier@osg.samsung.com>
+MAXIM MAX77802 PMIC REGULATOR DEVICE DRIVER
+M: Javier Martinez Canillas <javier@dowhile0.org>
L: linux-kernel@vger.kernel.org
S: Supported
-F: drivers/*/*max77802*.c
+F: drivers/regulator/max77802-regulator.c
F: Documentation/devicetree/bindings/*/*max77802.txt
F: include/dt-bindings/*/*max77802.h
S: Supported
F: arch/microblaze/
-MICROCHIP / ATMEL AT91 / AT32 SERIAL DRIVER
+MICROCHIP / ATMEL AT91 SERIAL DRIVER
M: Richard Genoud <richard.genoud@gmail.com>
S: Maintained
F: drivers/tty/serial/atmel_serial.c
F: drivers/mmc/host/mmc_spi.c
F: include/linux/spi/mmc_spi.h
+MULTIPLEXER SUBSYSTEM
+M: Peter Rosin <peda@axentia.se>
+S: Maintained
+F: Documentation/ABI/testing/mux/sysfs-class-mux*
+F: Documentation/devicetree/bindings/mux/
+F: include/linux/dt-bindings/mux/
+F: include/linux/mux/
+F: drivers/mux/
+
MULTISOUND SOUND DRIVER
M: Andrew Veliath <andrewtv@usa.net>
S: Maintained
S: Maintained
F: Documentation/hwmon/pmbus
F: drivers/hwmon/pmbus/
-F: include/linux/i2c/pmbus.h
+F: include/linux/pmbus.h
PMC SIERRA MaxRAID DRIVER
L: linux-scsi@vger.kernel.org
L: linux-kernel@vger.kernel.org
S: Maintained
F: drivers/staging/fsl-mc/
+F: Documentation/devicetree/bindings/misc/fsl,qoriq-mc.txt
QT1010 MEDIA DRIVER
M: Antti Palosaari <crope@iki.fi>
S: Supported
F: drivers/iio/adc/rcar_gyro_adc.c
-RENESAS USB2 PHY DRIVER
+RENESAS USB PHY DRIVER
M: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
L: linux-renesas-soc@vger.kernel.org
S: Maintained
-F: drivers/phy/phy-rcar-gen3-usb2.c
+F: drivers/phy/renesas/phy-rcar-gen3-usb*.c
RESET CONTROLLER FRAMEWORK
M: Philipp Zabel <p.zabel@pengutronix.de>
F: arch/s390/
F: drivers/s390/
F: Documentation/s390/
-F: Documentation/DocBook/s390*
+F: Documentation/driver-api/s390-drivers.rst
S390 COMMON I/O LAYER
M: Sebastian Ott <sebott@linux.vnet.ibm.com>
S: Supported
F: Documentation/devicetree/bindings/phy/samsung-phy.txt
F: Documentation/phy/samsung-usb2.txt
-F: drivers/phy/phy-exynos4210-usb2.c
-F: drivers/phy/phy-exynos4x12-usb2.c
-F: drivers/phy/phy-exynos5250-usb2.c
-F: drivers/phy/phy-s5pv210-usb2.c
-F: drivers/phy/phy-samsung-usb2.c
-F: drivers/phy/phy-samsung-usb2.h
+F: drivers/phy/samsung/phy-exynos4210-usb2.c
+F: drivers/phy/samsung/phy-exynos4x12-usb2.c
+F: drivers/phy/samsung/phy-exynos5250-usb2.c
+F: drivers/phy/samsung/phy-s5pv210-usb2.c
+F: drivers/phy/samsung/phy-samsung-usb2.c
+F: drivers/phy/samsung/phy-samsung-usb2.h
SERIAL DRIVERS
M: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
THUNDERBOLT DRIVER
M: Andreas Noever <andreas.noever@gmail.com>
+M: Michael Jamet <michael.jamet@intel.com>
+M: Mika Westerberg <mika.westerberg@linux.intel.com>
+M: Yehezkel Bernat <yehezkel.bernat@intel.com>
S: Maintained
F: drivers/thunderbolt/
F: include/uapi/linux/seccomp.h
F: include/linux/seccomp.h
F: tools/testing/selftests/seccomp/*
+F: Documentation/userspace-api/seccomp_filter.rst
K: \bsecure_computing
K: \bTIF_SECCOMP\b
F: include/linux/selinux*
F: security/selinux/
F: scripts/selinux/
+F: Documentation/admin-guide/LSM/SELinux.rst
APPARMOR SECURITY MODULE
M: John Johansen <john.johansen@canonical.com>
T: git git://git.kernel.org/pub/scm/linux/kernel/git/jj/apparmor-dev.git
S: Supported
F: security/apparmor/
+F: Documentation/admin-guide/LSM/apparmor.rst
LOADPIN SECURITY MODULE
M: Kees Cook <keescook@chromium.org>
T: git git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git lsm/loadpin
S: Supported
F: security/loadpin/
+F: Documentation/admin-guide/LSM/LoadPin.rst
YAMA SECURITY MODULE
M: Kees Cook <keescook@chromium.org>
T: git git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git yama/tip
S: Supported
F: security/yama/
+F: Documentation/admin-guide/LSM/Yama.rst
SENSABLE PHANTOM
M: Jiri Slaby <jirislaby@gmail.com>
W: http://schaufler-ca.com
T: git git://github.com/cschaufler/smack-next
S: Maintained
-F: Documentation/security/Smack.txt
+F: Documentation/admin-guide/LSM/Smack.rst
F: security/smack/
DRIVERS FOR ADAPTIVE VOLTAGE SCALING (AVS)
L: linux-mmc@vger.kernel.org
S: Supported
F: drivers/mmc/host/tmio_mmc*
-F: drivers/mmc/host/sh_mobile_sdhi.c
+F: drivers/mmc/host/renesas_sdhi*
F: include/linux/mfd/tmio.h
TMP401 HARDWARE MONITOR DRIVER
T: git git://git.kernel.org/pub/scm/utils/util-linux/util-linux.git
S: Maintained
+UUID HELPERS
+M: Christoph Hellwig <hch@lst.de>
+R: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
+L: linux-kernel@vger.kernel.org
+T: git git://git.infradead.org/users/hch/uuid.git
+F: lib/uuid.c
+F: lib/test_uuid.c
+F: include/linux/uuid.h
+F: include/uapi/linux/uuid.h
+S: Maintained
+
UVESAFB DRIVER
M: Michal Januszewski <spock@gentoo.org>
L: linux-fbdev@vger.kernel.org
S: Maintained
F: Documentation/w1/
F: drivers/w1/
+F: include/linux/w1.h
W83791D HARDWARE MONITORING DRIVER
M: Marc Hulsman <m.hulsman@tudelft.nl>
VERSION = 4
PATCHLEVEL = 12
SUBLEVEL = 0
-EXTRAVERSION = -rc6
+EXTRAVERSION =
NAME = Fearless Coyote
# *DOCUMENTATION*
#
mrproper: rm-dirs := $(wildcard $(MRPROPER_DIRS))
mrproper: rm-files := $(wildcard $(MRPROPER_FILES))
-mrproper-dirs := $(addprefix _mrproper_,Documentation/DocBook scripts)
+mrproper-dirs := $(addprefix _mrproper_,scripts)
PHONY += $(mrproper-dirs) mrproper archmrproper
$(mrproper-dirs):
@$(MAKE) $(build)=$(package-dir) help
@echo ''
@echo 'Documentation targets:'
- @$(MAKE) -f $(srctree)/Documentation/Makefile.sphinx dochelp
- @echo ''
- @$(MAKE) -f $(srctree)/Documentation/DocBook/Makefile dochelp
+ @$(MAKE) -f $(srctree)/Documentation/Makefile dochelp
@echo ''
@echo 'Architecture specific targets ($(SRCARCH)):'
@$(if $(archhelp),$(archhelp),\
@echo ' make V=0|1 [targets] 0 => quiet build (default), 1 => verbose build'
@echo ' make V=2 [targets] 2 => give reason for rebuild of target'
@echo ' make O=dir [targets] Locate all output files in "dir", including .config'
- @echo ' make C=1 [targets] Check all c source with $$CHECK (sparse by default)'
+ @echo ' make C=1 [targets] Check re-compiled c source with $$CHECK (sparse by default)'
@echo ' make C=2 [targets] Force check of all c source with $$CHECK'
@echo ' make RECORDMCOUNT_WARN=1 [targets] Warn about ignored mcount sections'
@echo ' make W=n [targets] Enable extra gcc checks, n=1,2,3 where'
DOC_TARGETS := xmldocs sgmldocs psdocs latexdocs pdfdocs htmldocs mandocs installmandocs epubdocs cleandocs linkcheckdocs
PHONY += $(DOC_TARGETS)
$(DOC_TARGETS): scripts_basic FORCE
- $(Q)$(MAKE) $(build)=scripts build_docproc build_check-lc_ctype
- $(Q)$(MAKE) $(build)=Documentation -f $(srctree)/Documentation/Makefile.sphinx $@
- $(Q)$(MAKE) $(build)=Documentation/DocBook $@
+ $(Q)$(MAKE) $(build)=Documentation $@
else # KBUILD_EXTMOD
config ARCH_WANT_RELAX_ORDER
bool
+config REFCOUNT_FULL
+ bool "Perform full reference count validation at the expense of speed"
+ help
+ Enabling this switches the refcounting infrastructure from a fast
+ unchecked atomic_t implementation to a fully state checked
+ implementation, which can be (slightly) slower but provides protections
+ against various use-after-free conditions that can be used in
+ security flaw exploits.
+
source "kernel/gcov/Kconfig"
#define TIOCGPKT _IOR('T', 0x38, int) /* Get packet mode state */
#define TIOCGPTLCK _IOR('T', 0x39, int) /* Get Pty lock state */
#define TIOCGEXCL _IOR('T', 0x40, int) /* Get exclusive mode state */
+#define TIOCGPTPEER _IOR('T', 0x41, int) /* Safely open the slave */
#define TIOCSERCONFIG 0x5453
#define TIOCSERGWILD 0x5454
generic-y += resource.h
generic-y += sembuf.h
generic-y += shmbuf.h
-generic-y += siginfo.h
generic-y += socket.h
generic-y += sockios.h
generic-y += stat.h
#define TSK_K_BLINK(tsk) TSK_K_REG(tsk, 4)
#define TSK_K_FP(tsk) TSK_K_REG(tsk, 0)
-#define thread_saved_pc(tsk) TSK_K_BLINK(tsk)
-
extern void start_thread(struct pt_regs * regs, unsigned long pc,
unsigned long usp);
# UAPI Header export list
include include/uapi/asm-generic/Kbuild.asm
+
+generic-y += siginfo.h
void __init time_init(void)
{
of_clk_init(NULL);
- clocksource_probe();
+ timer_probe();
}
static int __init customize_machine(void)
select EDAC_SUPPORT
select EDAC_ATOMIC_SCRUB
select GENERIC_ALLOCATOR
+ select GENERIC_ARCH_TOPOLOGY if ARM_CPU_TOPOLOGY
select GENERIC_ATOMIC64 if (CPU_V7M || CPU_V6 || !CPU_32v6K || !AEABI)
select GENERIC_CLOCKEVENTS_BROADCAST if SMP
select GENERIC_CPU_AUTOPROBE
select ARM_HAS_SG_CHAIN
select ARM_PATCH_PHYS_VIRT
select AUTO_ZRELADDR
- select CLKSRC_OF
+ select TIMER_OF
select COMMON_CLK
select GENERIC_CLOCKEVENTS
select MIGHT_HAVE_PCI
depends on !MMU
select ARM_NVIC
select AUTO_ZRELADDR
- select CLKSRC_OF
+ select TIMER_OF
select COMMON_CLK
select CPU_V7M
select GENERIC_CLOCKEVENTS
select CLKDEV_LOOKUP
select CLKSRC_PXA
select CLKSRC_MMIO
- select CLKSRC_OF
+ select TIMER_OF
select CPU_XSCALE if !CPU_XSC3
select GENERIC_CLOCKEVENTS
select GPIO_PXA
select CLKDEV_LOOKUP
select CLKSRC_MMIO
select CLKSRC_PXA
- select CLKSRC_OF if OF
+ select TIMER_OF if OF
select CPU_FREQ
select CPU_SA1100
select GENERIC_CLOCKEVENTS
config HAVE_ARM_TWD
bool
- select CLKSRC_OF if OF
+ select TIMER_OF if OF
help
This options enables support for the ARM timer and watchdog unit
config VMSPLIT_3G
bool "3G/1G user/kernel split"
config VMSPLIT_3G_OPT
+ depends on !ARM_LPAE
bool "3G/1G user/kernel split (for full 1G low memory)"
config VMSPLIT_2G
bool "2G/2G user/kernel split"
config HAVE_ARCH_PFN_VALID
def_bool ARCH_HAS_HOLES_MEMORYMODEL || !SPARSEMEM
-config HAVE_GENERIC_RCU_GUP
+config HAVE_GENERIC_GUP
def_bool y
depends on ARM_LPAE
is only useful for kernels that may run on systems that have
UEFI firmware.
+config DMI
+ bool "Enable support for SMBIOS (DMI) tables"
+ depends on EFI
+ default y
+ help
+ This enables SMBIOS/DMI feature for systems.
+
+ This option is only useful on systems that have UEFI firmware.
+ However, even with this option, the resultant kernel should
+ continue to boot on existing non-UEFI platforms.
+
+ NOTE: This does *NOT* enable or encourage the use of DMI quirks,
+ i.e., the the practice of identifying the platform via DMI to
+ decide whether certain workarounds for buggy hardware and/or
+ firmware need to be enabled. This would require the DMI subsystem
+ to be enabled much earlier than we do on ARM, which is non-trivial.
+
endmenu
menu "CPU Power Management"
@ there.
.inst 'M' | ('Z' << 8) | (0x1310 << 16) @ tstne r0, #0x4d000
#else
- W(mov) r0, r0
+ AR_CLASS( mov r0, r0 )
+ M_CLASS( nop.w )
#endif
.endm
//interrupts = <16 17 18 35 36 37 38 39>;
interrupts = <16>;
clocks = <&clk_apb>;
+ clock-names = "PCLK";
};
wdt1: wdt@1e785000 {
//interrupts = <16 17 18 35 36 37 38 39>;
interrupts = <16>;
clocks = <&clk_apb>;
+ clock-names = "PCLK";
};
gpio = <&gpio1 16 GPIO_ACTIVE_HIGH>; /* gpio_16: WiFi nReset */
startup-delay-us = <10000>;
};
-
- /* Regulator to trigger the nReset signal of the Bluetooth module */
- w3cbw003c_bt_nreset: regulator-w3cbw003c-bt-nreset {
- compatible = "regulator-fixed";
- regulator-name = "regulator-w3cbw003c-bt-nreset";
- regulator-min-microvolt = <3300000>;
- regulator-max-microvolt = <3300000>;
- gpio = <&gpio6 4 GPIO_ACTIVE_HIGH>; /* gpio_164: BT nReset */
- startup-delay-us = <10000>;
- };
};
&omap3_pmx_core {
pinctrl-names = "default";
pinctrl-0 = <&mmc2_pins>;
vmmc-supply = <&w3cbw003c_npoweron>;
- vqmmc-supply = <&w3cbw003c_bt_nreset>;
vmmc_aux-supply = <&w3cbw003c_wifi_nreset>;
bus-width = <4>;
cap-sdio-irq;
generic-y += sembuf.h
generic-y += serial.h
generic-y += shmbuf.h
-generic-y += siginfo.h
generic-y += simd.h
generic-y += sizes.h
generic-y += socket.h
--- /dev/null
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef __ASM_DMI_H
+#define __ASM_DMI_H
+
+#include <linux/io.h>
+#include <linux/slab.h>
+
+#define dmi_early_remap(x, l) memremap(x, l, MEMREMAP_WB)
+#define dmi_early_unmap(x, l) memunmap(x)
+#define dmi_remap(x, l) memremap(x, l, MEMREMAP_WB)
+#define dmi_unmap(x) memunmap(x)
+#define dmi_alloc(l) kzalloc(l, GFP_KERNEL)
+
+#endif
genhdr-y += unistd-common.h
genhdr-y += unistd-oabi.h
genhdr-y += unistd-eabi.h
+
+generic-y += siginfo.h
* driven low on this core and there isn't an architected way to
* determine that.
*/
- get_online_cpus();
+ cpus_read_lock();
register_undef_hook(&debug_reg_hook);
/*
* assume that a halting debugger will leave the world in a nice state
* for us.
*/
- ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "arm/hw_breakpoint:online",
- dbg_reset_online, NULL);
+ ret = cpuhp_setup_state_cpuslocked(CPUHP_AP_ONLINE_DYN,
+ "arm/hw_breakpoint:online",
+ dbg_reset_online, NULL);
unregister_undef_hook(&debug_reg_hook);
if (WARN_ON(ret < 0) || !cpumask_empty(&debug_err_mask)) {
core_num_brps = 0;
core_num_wrps = 0;
if (ret > 0)
- cpuhp_remove_state_nocalls(ret);
- put_online_cpus();
+ cpuhp_remove_state_nocalls_cpuslocked(ret);
+ cpus_read_unlock();
return 0;
}
TRAP_HWBKPT, "watchpoint debug exception");
hook_ifault_code(FAULT_CODE_DEBUG, hw_breakpoint_pending, SIGTRAP,
TRAP_HWBKPT, "breakpoint debug exception");
- put_online_cpus();
+ cpus_read_unlock();
/* Register PM notifiers. */
pm_init();
.insn = insn,
};
- stop_machine(patch_text_stop_machine, &patch, NULL);
+ stop_machine_cpuslocked(patch_text_stop_machine, &patch, NULL);
}
if (arch >= CPU_ARCH_ARMv6) {
unsigned int cachetype = read_cpuid_cachetype();
- if ((arch == CPU_ARCH_ARMv7M) && !cachetype) {
+ if ((arch == CPU_ARCH_ARMv7M) && !(cachetype & 0xf000f)) {
cacheid = 0;
} else if ((cachetype & (7 << 29)) == 4 << 29) {
/* ARMv7 register format */
*/
static void ipi_cpu_stop(unsigned int cpu)
{
- if (system_state == SYSTEM_BOOTING ||
- system_state == SYSTEM_RUNNING) {
+ if (system_state <= SYSTEM_RUNNING) {
raw_spin_lock(&stop_lock);
pr_crit("CPU%u: stopping\n", cpu);
dump_stack();
WARN(err, "twd_local_timer_of_register failed (%d)\n", err);
return err;
}
-CLOCKSOURCE_OF_DECLARE(arm_twd_a9, "arm,cortex-a9-twd-timer", twd_local_timer_of_register);
-CLOCKSOURCE_OF_DECLARE(arm_twd_a5, "arm,cortex-a5-twd-timer", twd_local_timer_of_register);
-CLOCKSOURCE_OF_DECLARE(arm_twd_11mp, "arm,arm11mp-twd-timer", twd_local_timer_of_register);
+TIMER_OF_DECLARE(arm_twd_a9, "arm,cortex-a9-twd-timer", twd_local_timer_of_register);
+TIMER_OF_DECLARE(arm_twd_a5, "arm,cortex-a5-twd-timer", twd_local_timer_of_register);
+TIMER_OF_DECLARE(arm_twd_11mp, "arm,arm11mp-twd-timer", twd_local_timer_of_register);
#endif
#ifdef CONFIG_COMMON_CLK
of_clk_init(NULL);
#endif
- clocksource_probe();
+ timer_probe();
}
}
* for more details.
*/
+#include <linux/arch_topology.h>
#include <linux/cpu.h>
#include <linux/cpufreq.h>
#include <linux/cpumask.h>
* to run the rebalance_domains for all idle cores and the cpu_capacity can be
* updated during this sequence.
*/
-static DEFINE_PER_CPU(unsigned long, cpu_scale) = SCHED_CAPACITY_SCALE;
-static DEFINE_MUTEX(cpu_scale_mutex);
-
-unsigned long arch_scale_cpu_capacity(struct sched_domain *sd, int cpu)
-{
- return per_cpu(cpu_scale, cpu);
-}
-
-static void set_capacity_scale(unsigned int cpu, unsigned long capacity)
-{
- per_cpu(cpu_scale, cpu) = capacity;
-}
-
-#ifdef CONFIG_PROC_SYSCTL
-static ssize_t cpu_capacity_show(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct cpu *cpu = container_of(dev, struct cpu, dev);
-
- return sprintf(buf, "%lu\n",
- arch_scale_cpu_capacity(NULL, cpu->dev.id));
-}
-
-static ssize_t cpu_capacity_store(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct cpu *cpu = container_of(dev, struct cpu, dev);
- int this_cpu = cpu->dev.id, i;
- unsigned long new_capacity;
- ssize_t ret;
-
- if (count) {
- ret = kstrtoul(buf, 0, &new_capacity);
- if (ret)
- return ret;
- if (new_capacity > SCHED_CAPACITY_SCALE)
- return -EINVAL;
-
- mutex_lock(&cpu_scale_mutex);
- for_each_cpu(i, &cpu_topology[this_cpu].core_sibling)
- set_capacity_scale(i, new_capacity);
- mutex_unlock(&cpu_scale_mutex);
- }
-
- return count;
-}
-
-static DEVICE_ATTR_RW(cpu_capacity);
-
-static int register_cpu_capacity_sysctl(void)
-{
- int i;
- struct device *cpu;
-
- for_each_possible_cpu(i) {
- cpu = get_cpu_device(i);
- if (!cpu) {
- pr_err("%s: too early to get CPU%d device!\n",
- __func__, i);
- continue;
- }
- device_create_file(cpu, &dev_attr_cpu_capacity);
- }
-
- return 0;
-}
-subsys_initcall(register_cpu_capacity_sysctl);
-#endif
#ifdef CONFIG_OF
struct cpu_efficiency {
static unsigned long middle_capacity = 1;
static bool cap_from_dt = true;
-static u32 *raw_capacity;
-static bool cap_parsing_failed;
-static u32 capacity_scale;
-
-static int __init parse_cpu_capacity(struct device_node *cpu_node, int cpu)
-{
- int ret = 1;
- u32 cpu_capacity;
-
- if (cap_parsing_failed)
- return !ret;
-
- ret = of_property_read_u32(cpu_node,
- "capacity-dmips-mhz",
- &cpu_capacity);
- if (!ret) {
- if (!raw_capacity) {
- raw_capacity = kcalloc(num_possible_cpus(),
- sizeof(*raw_capacity),
- GFP_KERNEL);
- if (!raw_capacity) {
- pr_err("cpu_capacity: failed to allocate memory for raw capacities\n");
- cap_parsing_failed = true;
- return !ret;
- }
- }
- capacity_scale = max(cpu_capacity, capacity_scale);
- raw_capacity[cpu] = cpu_capacity;
- pr_debug("cpu_capacity: %s cpu_capacity=%u (raw)\n",
- cpu_node->full_name, raw_capacity[cpu]);
- } else {
- if (raw_capacity) {
- pr_err("cpu_capacity: missing %s raw capacity\n",
- cpu_node->full_name);
- pr_err("cpu_capacity: partial information: fallback to 1024 for all CPUs\n");
- }
- cap_parsing_failed = true;
- kfree(raw_capacity);
- }
-
- return !ret;
-}
-
-static void normalize_cpu_capacity(void)
-{
- u64 capacity;
- int cpu;
-
- if (!raw_capacity || cap_parsing_failed)
- return;
-
- pr_debug("cpu_capacity: capacity_scale=%u\n", capacity_scale);
- mutex_lock(&cpu_scale_mutex);
- for_each_possible_cpu(cpu) {
- capacity = (raw_capacity[cpu] << SCHED_CAPACITY_SHIFT)
- / capacity_scale;
- set_capacity_scale(cpu, capacity);
- pr_debug("cpu_capacity: CPU%d cpu_capacity=%lu\n",
- cpu, arch_scale_cpu_capacity(NULL, cpu));
- }
- mutex_unlock(&cpu_scale_mutex);
-}
-
-#ifdef CONFIG_CPU_FREQ
-static cpumask_var_t cpus_to_visit;
-static bool cap_parsing_done;
-static void parsing_done_workfn(struct work_struct *work);
-static DECLARE_WORK(parsing_done_work, parsing_done_workfn);
-
-static int
-init_cpu_capacity_callback(struct notifier_block *nb,
- unsigned long val,
- void *data)
-{
- struct cpufreq_policy *policy = data;
- int cpu;
-
- if (cap_parsing_failed || cap_parsing_done)
- return 0;
-
- switch (val) {
- case CPUFREQ_NOTIFY:
- pr_debug("cpu_capacity: init cpu capacity for CPUs [%*pbl] (to_visit=%*pbl)\n",
- cpumask_pr_args(policy->related_cpus),
- cpumask_pr_args(cpus_to_visit));
- cpumask_andnot(cpus_to_visit,
- cpus_to_visit,
- policy->related_cpus);
- for_each_cpu(cpu, policy->related_cpus) {
- raw_capacity[cpu] = arch_scale_cpu_capacity(NULL, cpu) *
- policy->cpuinfo.max_freq / 1000UL;
- capacity_scale = max(raw_capacity[cpu], capacity_scale);
- }
- if (cpumask_empty(cpus_to_visit)) {
- normalize_cpu_capacity();
- kfree(raw_capacity);
- pr_debug("cpu_capacity: parsing done\n");
- cap_parsing_done = true;
- schedule_work(&parsing_done_work);
- }
- }
- return 0;
-}
-
-static struct notifier_block init_cpu_capacity_notifier = {
- .notifier_call = init_cpu_capacity_callback,
-};
-
-static int __init register_cpufreq_notifier(void)
-{
- if (cap_parsing_failed)
- return -EINVAL;
-
- if (!alloc_cpumask_var(&cpus_to_visit, GFP_KERNEL)) {
- pr_err("cpu_capacity: failed to allocate memory for cpus_to_visit\n");
- return -ENOMEM;
- }
- cpumask_copy(cpus_to_visit, cpu_possible_mask);
-
- return cpufreq_register_notifier(&init_cpu_capacity_notifier,
- CPUFREQ_POLICY_NOTIFIER);
-}
-core_initcall(register_cpufreq_notifier);
-
-static void parsing_done_workfn(struct work_struct *work)
-{
- cpufreq_unregister_notifier(&init_cpu_capacity_notifier,
- CPUFREQ_POLICY_NOTIFIER);
-}
-
-#else
-static int __init free_raw_capacity(void)
-{
- kfree(raw_capacity);
-
- return 0;
-}
-core_initcall(free_raw_capacity);
-#endif
/*
* Iterate all CPUs' descriptor in DT and compute the efficiency
continue;
}
- if (parse_cpu_capacity(cn, cpu)) {
+ if (topology_parse_cpu_capacity(cn, cpu)) {
of_node_put(cn);
continue;
}
middle_capacity = ((max_capacity / 3)
>> (SCHED_CAPACITY_SHIFT-1)) + 1;
- if (cap_from_dt && !cap_parsing_failed)
- normalize_cpu_capacity();
+ if (cap_from_dt)
+ topology_normalize_cpu_scale();
}
/*
if (!cpu_capacity(cpu) || cap_from_dt)
return;
- set_capacity_scale(cpu, cpu_capacity(cpu) / middle_capacity);
+ topology_set_cpu_scale(cpu, cpu_capacity(cpu) / middle_capacity);
pr_info("CPU%u: update cpu_capacity %lu\n",
- cpu, arch_scale_cpu_capacity(NULL, cpu));
+ cpu, topology_get_cpu_scale(NULL, cpu));
}
#else
select SRAM
select WATCHDOG
select ASPEED_WATCHDOG
- select MOXART_TIMER
+ select FTTMR010_TIMER
select MFD_SYSCON
select PINCTRL
help
select ARM_ERRATA_411920 if ARCH_MULTI_V6
select ARM_TIMER_SP804
select HAVE_ARM_ARCH_TIMER if ARCH_MULTI_V7
- select CLKSRC_OF
+ select TIMER_OF
select BCM2835_TIMER
select PINCTRL
select PINCTRL_BCM2835
bool "Cirrus Logic EP721x/EP731x-based"
depends on ARCH_MULTI_V4T
select AUTO_ZRELADDR
- select CLKSRC_OF
+ select TIMER_OF
select CLPS711X_TIMER
select COMMON_CLK
select CPU_ARM720T
}
of_clk_init(NULL);
- clocksource_probe();
+ timer_probe();
};
static const char * const mediatek_board_dt_compat[] = {
select CPU_FA526
select ARM_DMA_MEM_BUFFERABLE
select FARADAY_FTINTC010
- select MOXART_TIMER
+ select FTTMR010_TIMER
select GPIOLIB
select PHYLIB if NETDEVICES
help
{ "omap_uart.2", "rx", SDMA_FILTER_PARAM(54) },
{ "omap_hsmmc.0", "tx", SDMA_FILTER_PARAM(61) },
{ "omap_hsmmc.0", "rx", SDMA_FILTER_PARAM(62) },
+
+ /* external DMA requests when tusb6010 is used */
+ { "musb-tusb", "dmareq0", SDMA_FILTER_PARAM(2) },
+ { "musb-tusb", "dmareq1", SDMA_FILTER_PARAM(3) },
+ { "musb-tusb", "dmareq2", SDMA_FILTER_PARAM(14) }, /* OMAP2420 only */
+ { "musb-tusb", "dmareq3", SDMA_FILTER_PARAM(15) }, /* OMAP2420 only */
+ { "musb-tusb", "dmareq4", SDMA_FILTER_PARAM(16) }, /* OMAP2420 only */
+ { "musb-tusb", "dmareq5", SDMA_FILTER_PARAM(64) }, /* OMAP2420 only */
+};
+
+static const struct dma_slave_map omap24xx_sdma_dt_map[] = {
+ /* external DMA requests when tusb6010 is used */
+ { "musb-hdrc.1.auto", "dmareq0", SDMA_FILTER_PARAM(2) },
+ { "musb-hdrc.1.auto", "dmareq1", SDMA_FILTER_PARAM(3) },
+ { "musb-hdrc.1.auto", "dmareq2", SDMA_FILTER_PARAM(14) }, /* OMAP2420 only */
+ { "musb-hdrc.1.auto", "dmareq3", SDMA_FILTER_PARAM(15) }, /* OMAP2420 only */
+ { "musb-hdrc.1.auto", "dmareq4", SDMA_FILTER_PARAM(16) }, /* OMAP2420 only */
+ { "musb-hdrc.1.auto", "dmareq5", SDMA_FILTER_PARAM(64) }, /* OMAP2420 only */
};
static const struct dma_slave_map omap3xxx_sdma_map[] = {
__func__);
return -ENODEV;
}
+ } else {
+ if (soc_is_omap24xx()) {
+ /* DMA slave map for drivers not yet converted to DT */
+ p.slave_map = omap24xx_sdma_dt_map;
+ p.slavecnt = ARRAY_SIZE(omap24xx_sdma_dt_map);
+ }
}
pdev = omap_device_build(name, 0, oh, &p, sizeof(p));
__omap_sync32k_timer_init(1, "timer_32k_ck", "ti,timer-alwon",
2, "timer_sys_ck", NULL, false);
- clocksource_probe();
+ timer_probe();
}
#if defined(CONFIG_ARCH_OMAP3) || defined(CONFIG_SOC_AM43XX)
__omap_sync32k_timer_init(12, "secure_32k_fck", "ti,timer-secure",
2, "timer_sys_ck", NULL, false);
- clocksource_probe();
+ timer_probe();
}
#endif /* CONFIG_ARCH_OMAP3 */
__omap_sync32k_timer_init(2, "timer_sys_ck", NULL,
1, "timer_sys_ck", "ti,timer-alwon", true);
if (of_have_populated_dt())
- clocksource_probe();
+ timer_probe();
}
#endif
void __init omap4_local_timer_init(void)
{
omap4_sync32k_timer_init();
- clocksource_probe();
+ timer_probe();
}
#endif
omap4_sync32k_timer_init();
realtime_counter_init();
- clocksource_probe();
+ timer_probe();
}
#endif /* CONFIG_SOC_OMAP5 || CONFIG_SOC_DRA7XX */
}
of_clk_init(NULL);
- clocksource_probe();
+ timer_probe();
}
static void __init rockchip_dt_init(void)
return ec;
}
-static ssize_t ecard_show_irq(struct device *dev, struct device_attribute *attr, char *buf)
+static ssize_t irq_show(struct device *dev, struct device_attribute *attr, char *buf)
{
struct expansion_card *ec = ECARD_DEV(dev);
return sprintf(buf, "%u\n", ec->irq);
}
+static DEVICE_ATTR_RO(irq);
-static ssize_t ecard_show_dma(struct device *dev, struct device_attribute *attr, char *buf)
+static ssize_t dma_show(struct device *dev, struct device_attribute *attr, char *buf)
{
struct expansion_card *ec = ECARD_DEV(dev);
return sprintf(buf, "%u\n", ec->dma);
}
+static DEVICE_ATTR_RO(dma);
-static ssize_t ecard_show_resources(struct device *dev, struct device_attribute *attr, char *buf)
+static ssize_t resource_show(struct device *dev, struct device_attribute *attr, char *buf)
{
struct expansion_card *ec = ECARD_DEV(dev);
char *str = buf;
return str - buf;
}
+static DEVICE_ATTR_RO(resource);
-static ssize_t ecard_show_vendor(struct device *dev, struct device_attribute *attr, char *buf)
+static ssize_t vendor_show(struct device *dev, struct device_attribute *attr, char *buf)
{
struct expansion_card *ec = ECARD_DEV(dev);
return sprintf(buf, "%u\n", ec->cid.manufacturer);
}
+static DEVICE_ATTR_RO(vendor);
-static ssize_t ecard_show_device(struct device *dev, struct device_attribute *attr, char *buf)
+static ssize_t device_show(struct device *dev, struct device_attribute *attr, char *buf)
{
struct expansion_card *ec = ECARD_DEV(dev);
return sprintf(buf, "%u\n", ec->cid.product);
}
+static DEVICE_ATTR_RO(device);
-static ssize_t ecard_show_type(struct device *dev, struct device_attribute *attr, char *buf)
+static ssize_t type_show(struct device *dev, struct device_attribute *attr, char *buf)
{
struct expansion_card *ec = ECARD_DEV(dev);
return sprintf(buf, "%s\n", ec->easi ? "EASI" : "IOC");
}
-
-static struct device_attribute ecard_dev_attrs[] = {
- __ATTR(device, S_IRUGO, ecard_show_device, NULL),
- __ATTR(dma, S_IRUGO, ecard_show_dma, NULL),
- __ATTR(irq, S_IRUGO, ecard_show_irq, NULL),
- __ATTR(resource, S_IRUGO, ecard_show_resources, NULL),
- __ATTR(type, S_IRUGO, ecard_show_type, NULL),
- __ATTR(vendor, S_IRUGO, ecard_show_vendor, NULL),
- __ATTR_NULL,
+static DEVICE_ATTR_RO(type);
+
+static struct attribute *ecard_dev_attrs[] = {
+ &dev_attr_device.attr,
+ &dev_attr_dma.attr,
+ &dev_attr_irq.attr,
+ &dev_attr_resource.attr,
+ &dev_attr_type.attr,
+ &dev_attr_vendor.attr,
+ NULL,
};
-
+ATTRIBUTE_GROUPS(ecard_dev);
int ecard_request_resources(struct expansion_card *ec)
{
struct bus_type ecard_bus_type = {
.name = "ecard",
- .dev_attrs = ecard_dev_attrs,
+ .dev_groups = ecard_dev_groups,
.match = ecard_match,
.probe = ecard_drv_probe,
.remove = ecard_drv_remove,
config MACH_S3C2416_DT
bool "Samsung S3C2416 machine using devicetree"
- select CLKSRC_OF
+ select TIMER_OF
select USE_OF
select PINCTRL
select PINCTRL_S3C24XX
config MACH_S3C64XX_DT
bool "Samsung S3C6400/S3C6410 machine using Device Tree"
- select CLKSRC_OF
+ select TIMER_OF
select CPU_S3C6400
select CPU_S3C6410
select PINCTRL
#endif /* CONFIG_ARM_ARCH_TIMER */
of_clk_init(NULL);
- clocksource_probe();
+ timer_probe();
}
struct memory_reserve_config {
clk_put(pclk);
spear_setup_of_timer();
- clocksource_probe();
+ timer_probe();
}
of_clk_init(NULL);
if (IS_ENABLED(CONFIG_RESET_CONTROLLER))
sun6i_reset_init();
- clocksource_probe();
+ timer_probe();
}
DT_MACHINE_START(SUN6I_DT, "Allwinner sun6i (A31) Family")
DT_MACHINE_START(U300_DT, "U300 S335/B335 (Device Tree)")
.map_io = u300_map_io,
.init_irq = u300_init_irq_dt,
- .init_time = clocksource_probe,
+ .init_time = timer_probe,
.init_machine = u300_init_machine_dt,
.restart = u300_restart,
.dt_compat = u300_board_compat,
{
zynq_clock_init();
of_clk_init(NULL);
- clocksource_probe();
+ timer_probe();
}
static struct map_desc zynq_cortex_a9_scu_map __initdata = {
high_memory = __va(arm_lowmem_limit - 1) + 1;
+ if (!memblock_limit)
+ memblock_limit = arm_lowmem_limit;
+
/*
* Round the memblock limit down to a pmd size. This
* helps to ensure that we will allocate memory from the
* last full pmd, which should be mapped.
*/
- if (memblock_limit)
- memblock_limit = round_down(memblock_limit, PMD_SIZE);
- if (!memblock_limit)
- memblock_limit = arm_lowmem_limit;
+ memblock_limit = round_down(memblock_limit, PMD_SIZE);
if (!IS_ENABLED(CONFIG_HIGHMEM) || cache_is_vipt_aliasing()) {
if (memblock_end_of_DRAM() > arm_lowmem_limit) {
.addr = addr,
.insn = insn,
};
- stop_machine(__kprobes_remove_breakpoint, &p, cpu_online_mask);
+ stop_machine_cpuslocked(__kprobes_remove_breakpoint, &p,
+ cpu_online_mask);
}
void __kprobes arch_disarm_kprobe(struct kprobe *p)
select EDAC_SUPPORT
select FRAME_POINTER
select GENERIC_ALLOCATOR
+ select GENERIC_ARCH_TOPOLOGY
select GENERIC_CLOCKEVENTS
select GENERIC_CLOCKEVENTS_BROADCAST
select GENERIC_CPU_AUTOPROBE
config ZONE_DMA
def_bool y
-config HAVE_GENERIC_RCU_GUP
+config HAVE_GENERIC_GUP
def_bool y
config ARCH_DMA_ADDR_T_64BIT
config ARCH_BCM2835
bool "Broadcom BCM2835 family"
- select CLKSRC_OF
+ select TIMER_OF
select GPIOLIB
select PINCTRL
select PINCTRL_BCM2835
select ARCH_HAS_RESET_CONTROLLER
select CLKDEV_LOOKUP
select CLKSRC_MMIO
- select CLKSRC_OF
+ select TIMER_OF
select GENERIC_CLOCKEVENTS
select GPIOLIB
select PINCTRL
};
};
};
+
+ debug@f6590000 {
+ compatible = "arm,coresight-cpu-debug","arm,primecell";
+ reg = <0 0xf6590000 0 0x1000>;
+ clocks = <&sys_ctrl HI6220_DAPB_CLK>;
+ clock-names = "apb_pclk";
+ cpu = <&cpu0>;
+ };
+
+ debug@f6592000 {
+ compatible = "arm,coresight-cpu-debug","arm,primecell";
+ reg = <0 0xf6592000 0 0x1000>;
+ clocks = <&sys_ctrl HI6220_DAPB_CLK>;
+ clock-names = "apb_pclk";
+ cpu = <&cpu1>;
+ };
+
+ debug@f6594000 {
+ compatible = "arm,coresight-cpu-debug","arm,primecell";
+ reg = <0 0xf6594000 0 0x1000>;
+ clocks = <&sys_ctrl HI6220_DAPB_CLK>;
+ clock-names = "apb_pclk";
+ cpu = <&cpu2>;
+ };
+
+ debug@f6596000 {
+ compatible = "arm,coresight-cpu-debug","arm,primecell";
+ reg = <0 0xf6596000 0 0x1000>;
+ clocks = <&sys_ctrl HI6220_DAPB_CLK>;
+ clock-names = "apb_pclk";
+ cpu = <&cpu3>;
+ };
+
+ debug@f65d0000 {
+ compatible = "arm,coresight-cpu-debug","arm,primecell";
+ reg = <0 0xf65d0000 0 0x1000>;
+ clocks = <&sys_ctrl HI6220_DAPB_CLK>;
+ clock-names = "apb_pclk";
+ cpu = <&cpu4>;
+ };
+
+ debug@f65d2000 {
+ compatible = "arm,coresight-cpu-debug","arm,primecell";
+ reg = <0 0xf65d2000 0 0x1000>;
+ clocks = <&sys_ctrl HI6220_DAPB_CLK>;
+ clock-names = "apb_pclk";
+ cpu = <&cpu5>;
+ };
+
+ debug@f65d4000 {
+ compatible = "arm,coresight-cpu-debug","arm,primecell";
+ reg = <0 0xf65d4000 0 0x1000>;
+ clocks = <&sys_ctrl HI6220_DAPB_CLK>;
+ clock-names = "apb_pclk";
+ cpu = <&cpu6>;
+ };
+
+ debug@f65d6000 {
+ compatible = "arm,coresight-cpu-debug","arm,primecell";
+ reg = <0 0xf65d6000 0 0x1000>;
+ clocks = <&sys_ctrl HI6220_DAPB_CLK>;
+ clock-names = "apb_pclk";
+ cpu = <&cpu7>;
+ };
};
};
};
};
+ debug@850000 {
+ compatible = "arm,coresight-cpu-debug","arm,primecell";
+ reg = <0x850000 0x1000>;
+ clocks = <&rpmcc RPM_QDSS_CLK>;
+ clock-names = "apb_pclk";
+ cpu = <&CPU0>;
+ };
+
+ debug@852000 {
+ compatible = "arm,coresight-cpu-debug","arm,primecell";
+ reg = <0x852000 0x1000>;
+ clocks = <&rpmcc RPM_QDSS_CLK>;
+ clock-names = "apb_pclk";
+ cpu = <&CPU1>;
+ };
+
+ debug@854000 {
+ compatible = "arm,coresight-cpu-debug","arm,primecell";
+ reg = <0x854000 0x1000>;
+ clocks = <&rpmcc RPM_QDSS_CLK>;
+ clock-names = "apb_pclk";
+ cpu = <&CPU2>;
+ };
+
+ debug@856000 {
+ compatible = "arm,coresight-cpu-debug","arm,primecell";
+ reg = <0x856000 0x1000>;
+ clocks = <&rpmcc RPM_QDSS_CLK>;
+ clock-names = "apb_pclk";
+ cpu = <&CPU3>;
+ };
+
etm@85c000 {
compatible = "arm,coresight-etm4x", "arm,primecell";
reg = <0x85c000 0x1000>;
bool aarch64_insn_hotpatch_safe(u32 old_insn, u32 new_insn);
int aarch64_insn_patch_text_nosync(void *addr, u32 insn);
-int aarch64_insn_patch_text_sync(void *addrs[], u32 insns[], int cnt);
int aarch64_insn_patch_text(void *addrs[], u32 insns[], int cnt);
s32 aarch64_insn_adrp_get_offset(u32 insn);
*
*/
-#include <linux/dmi.h>
#include <linux/efi.h>
#include <linux/init.h>
set_permissions, md);
}
-static int __init arm64_dmi_init(void)
-{
- /*
- * On arm64, DMI depends on UEFI, and dmi_scan_machine() needs to
- * be called early because dmi_id_init(), which is an arch_initcall
- * itself, depends on dmi_scan_machine() having been called already.
- */
- dmi_scan_machine();
- if (dmi_available)
- dmi_set_dump_stack_arch_desc();
- return 0;
-}
-core_initcall(arm64_dmi_init);
-
/*
* UpdateCapsule() depends on the system being shutdown via
* ResetSystem().
return ret;
}
+static
int __kprobes aarch64_insn_patch_text_sync(void *addrs[], u32 insns[], int cnt)
{
struct aarch64_insn_patch patch = {
if (cnt <= 0)
return -EINVAL;
- return stop_machine(aarch64_insn_patch_text_cb, &patch,
- cpu_online_mask);
+ return stop_machine_cpuslocked(aarch64_insn_patch_text_cb, &patch,
+ cpu_online_mask);
}
int __kprobes aarch64_insn_patch_text(void *addrs[], u32 insns[], int cnt)
cpumask_copy(&mask, cpu_online_mask);
cpumask_clear_cpu(smp_processor_id(), &mask);
- if (system_state == SYSTEM_BOOTING ||
- system_state == SYSTEM_RUNNING)
+ if (system_state <= SYSTEM_RUNNING)
pr_crit("SMP: stopping secondary CPUs\n");
smp_cross_call(&mask, IPI_CPU_STOP);
}
u32 arch_timer_rate;
of_clk_init(NULL);
- clocksource_probe();
+ timer_probe();
tick_setup_hrtimer_broadcast();
* for more details.
*/
-#include <linux/acpi.h>
+#include <linux/arch_topology.h>
#include <linux/cpu.h>
#include <linux/cpumask.h>
#include <linux/init.h>
#include <linux/sched/topology.h>
#include <linux/slab.h>
#include <linux/string.h>
-#include <linux/cpufreq.h>
#include <asm/cpu.h>
#include <asm/cputype.h>
#include <asm/topology.h>
-static DEFINE_PER_CPU(unsigned long, cpu_scale) = SCHED_CAPACITY_SCALE;
-static DEFINE_MUTEX(cpu_scale_mutex);
-
-unsigned long arch_scale_cpu_capacity(struct sched_domain *sd, int cpu)
-{
- return per_cpu(cpu_scale, cpu);
-}
-
-static void set_capacity_scale(unsigned int cpu, unsigned long capacity)
-{
- per_cpu(cpu_scale, cpu) = capacity;
-}
-
-static ssize_t cpu_capacity_show(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct cpu *cpu = container_of(dev, struct cpu, dev);
-
- return sprintf(buf, "%lu\n",
- arch_scale_cpu_capacity(NULL, cpu->dev.id));
-}
-
-static ssize_t cpu_capacity_store(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct cpu *cpu = container_of(dev, struct cpu, dev);
- int this_cpu = cpu->dev.id, i;
- unsigned long new_capacity;
- ssize_t ret;
-
- if (count) {
- ret = kstrtoul(buf, 0, &new_capacity);
- if (ret)
- return ret;
- if (new_capacity > SCHED_CAPACITY_SCALE)
- return -EINVAL;
-
- mutex_lock(&cpu_scale_mutex);
- for_each_cpu(i, &cpu_topology[this_cpu].core_sibling)
- set_capacity_scale(i, new_capacity);
- mutex_unlock(&cpu_scale_mutex);
- }
-
- return count;
-}
-
-static DEVICE_ATTR_RW(cpu_capacity);
-
-static int register_cpu_capacity_sysctl(void)
-{
- int i;
- struct device *cpu;
-
- for_each_possible_cpu(i) {
- cpu = get_cpu_device(i);
- if (!cpu) {
- pr_err("%s: too early to get CPU%d device!\n",
- __func__, i);
- continue;
- }
- device_create_file(cpu, &dev_attr_cpu_capacity);
- }
-
- return 0;
-}
-subsys_initcall(register_cpu_capacity_sysctl);
-
-static u32 capacity_scale;
-static u32 *raw_capacity;
-static bool cap_parsing_failed;
-
-static void __init parse_cpu_capacity(struct device_node *cpu_node, int cpu)
-{
- int ret;
- u32 cpu_capacity;
-
- if (cap_parsing_failed)
- return;
-
- ret = of_property_read_u32(cpu_node,
- "capacity-dmips-mhz",
- &cpu_capacity);
- if (!ret) {
- if (!raw_capacity) {
- raw_capacity = kcalloc(num_possible_cpus(),
- sizeof(*raw_capacity),
- GFP_KERNEL);
- if (!raw_capacity) {
- pr_err("cpu_capacity: failed to allocate memory for raw capacities\n");
- cap_parsing_failed = true;
- return;
- }
- }
- capacity_scale = max(cpu_capacity, capacity_scale);
- raw_capacity[cpu] = cpu_capacity;
- pr_debug("cpu_capacity: %s cpu_capacity=%u (raw)\n",
- cpu_node->full_name, raw_capacity[cpu]);
- } else {
- if (raw_capacity) {
- pr_err("cpu_capacity: missing %s raw capacity\n",
- cpu_node->full_name);
- pr_err("cpu_capacity: partial information: fallback to 1024 for all CPUs\n");
- }
- cap_parsing_failed = true;
- kfree(raw_capacity);
- }
-}
-
-static void normalize_cpu_capacity(void)
-{
- u64 capacity;
- int cpu;
-
- if (!raw_capacity || cap_parsing_failed)
- return;
-
- pr_debug("cpu_capacity: capacity_scale=%u\n", capacity_scale);
- mutex_lock(&cpu_scale_mutex);
- for_each_possible_cpu(cpu) {
- pr_debug("cpu_capacity: cpu=%d raw_capacity=%u\n",
- cpu, raw_capacity[cpu]);
- capacity = (raw_capacity[cpu] << SCHED_CAPACITY_SHIFT)
- / capacity_scale;
- set_capacity_scale(cpu, capacity);
- pr_debug("cpu_capacity: CPU%d cpu_capacity=%lu\n",
- cpu, arch_scale_cpu_capacity(NULL, cpu));
- }
- mutex_unlock(&cpu_scale_mutex);
-}
-
-#ifdef CONFIG_CPU_FREQ
-static cpumask_var_t cpus_to_visit;
-static bool cap_parsing_done;
-static void parsing_done_workfn(struct work_struct *work);
-static DECLARE_WORK(parsing_done_work, parsing_done_workfn);
-
-static int
-init_cpu_capacity_callback(struct notifier_block *nb,
- unsigned long val,
- void *data)
-{
- struct cpufreq_policy *policy = data;
- int cpu;
-
- if (cap_parsing_failed || cap_parsing_done)
- return 0;
-
- switch (val) {
- case CPUFREQ_NOTIFY:
- pr_debug("cpu_capacity: init cpu capacity for CPUs [%*pbl] (to_visit=%*pbl)\n",
- cpumask_pr_args(policy->related_cpus),
- cpumask_pr_args(cpus_to_visit));
- cpumask_andnot(cpus_to_visit,
- cpus_to_visit,
- policy->related_cpus);
- for_each_cpu(cpu, policy->related_cpus) {
- raw_capacity[cpu] = arch_scale_cpu_capacity(NULL, cpu) *
- policy->cpuinfo.max_freq / 1000UL;
- capacity_scale = max(raw_capacity[cpu], capacity_scale);
- }
- if (cpumask_empty(cpus_to_visit)) {
- normalize_cpu_capacity();
- kfree(raw_capacity);
- pr_debug("cpu_capacity: parsing done\n");
- cap_parsing_done = true;
- schedule_work(&parsing_done_work);
- }
- }
- return 0;
-}
-
-static struct notifier_block init_cpu_capacity_notifier = {
- .notifier_call = init_cpu_capacity_callback,
-};
-
-static int __init register_cpufreq_notifier(void)
-{
- /*
- * on ACPI-based systems we need to use the default cpu capacity
- * until we have the necessary code to parse the cpu capacity, so
- * skip registering cpufreq notifier.
- */
- if (!acpi_disabled || cap_parsing_failed)
- return -EINVAL;
-
- if (!alloc_cpumask_var(&cpus_to_visit, GFP_KERNEL)) {
- pr_err("cpu_capacity: failed to allocate memory for cpus_to_visit\n");
- return -ENOMEM;
- }
- cpumask_copy(cpus_to_visit, cpu_possible_mask);
-
- return cpufreq_register_notifier(&init_cpu_capacity_notifier,
- CPUFREQ_POLICY_NOTIFIER);
-}
-core_initcall(register_cpufreq_notifier);
-
-static void parsing_done_workfn(struct work_struct *work)
-{
- cpufreq_unregister_notifier(&init_cpu_capacity_notifier,
- CPUFREQ_POLICY_NOTIFIER);
-}
-
-#else
-static int __init free_raw_capacity(void)
-{
- kfree(raw_capacity);
-
- return 0;
-}
-core_initcall(free_raw_capacity);
-#endif
-
static int __init get_cpu_for_node(struct device_node *node)
{
struct device_node *cpu_node;
for_each_possible_cpu(cpu) {
if (of_get_cpu_node(cpu, NULL) == cpu_node) {
- parse_cpu_capacity(cpu_node, cpu);
+ topology_parse_cpu_capacity(cpu_node, cpu);
of_node_put(cpu_node);
return cpu;
}
* cluster with restricted subnodes.
*/
map = of_get_child_by_name(cn, "cpu-map");
- if (!map) {
- cap_parsing_failed = true;
+ if (!map)
goto out;
- }
ret = parse_cluster(map, 0);
if (ret != 0)
goto out_map;
- normalize_cpu_capacity();
+ topology_normalize_cpu_scale();
/*
* Check that all cores are in the topology; the SMP code will
if (!use_syscall) {
/* tkr_mono.cycle_last == tkr_raw.cycle_last */
vdso_data->cs_cycle_last = tk->tkr_mono.cycle_last;
- vdso_data->raw_time_sec = tk->raw_time.tv_sec;
- vdso_data->raw_time_nsec = tk->raw_time.tv_nsec;
+ vdso_data->raw_time_sec = tk->raw_sec;
+ vdso_data->raw_time_nsec = tk->tkr_raw.xtime_nsec;
vdso_data->xtime_clock_sec = tk->xtime_sec;
vdso_data->xtime_clock_nsec = tk->tkr_mono.xtime_nsec;
- /* tkr_raw.xtime_nsec == 0 */
vdso_data->cs_mono_mult = tk->tkr_mono.mult;
vdso_data->cs_raw_mult = tk->tkr_raw.mult;
/* tkr_mono.shift == tkr_raw.shift */
seqcnt_check fail=monotonic_raw
/* All computations are done with left-shifted nsecs. */
- lsl x14, x14, x12
get_nsec_per_sec res=x9
lsl x9, x9, x12
{
}
-/*
- * Return saved PC of a blocked thread.
- */
-#define thread_saved_pc(tsk) (tsk->thread.pc)
-
unsigned long get_wchan(struct task_struct *p);
#define KSTK_EIP(tsk) \
generic-y += serial.h
generic-y += shmbuf.h
generic-y += shmparam.h
-generic-y += siginfo.h
generic-y += signal.h
generic-y += socket.h
generic-y += sockios.h
#define copy_segments(tsk, mm) do { } while (0)
#define release_segments(mm) do { } while (0)
-/*
- * saved PC of a blocked thread.
- */
-#define thread_saved_pc(tsk) (task_pt_regs(tsk)->pc)
-
/*
* saved kernel SP and DP of a blocked thread.
*/
include include/uapi/asm-generic/Kbuild.asm
generic-y += kvm_para.h
+generic-y += siginfo.h
while(1) /* waiting for RETRIBUTION! */ ;
}
-/*
- * Return saved PC of a blocked thread.
- */
-unsigned long thread_saved_pc(struct task_struct *t)
-{
- return task_pt_regs(t)->irp;
-}
-
/* setup the child's kernel stack with a pt_regs and switch_stack on it.
* it will be un-nested during _resume and _ret_from_sys_call when the
* new thread is scheduled.
; /* Wait for reset. */
}
-/*
- * Return saved PC of a blocked thread.
- */
-unsigned long thread_saved_pc(struct task_struct *t)
-{
- return task_pt_regs(t)->erp;
-}
-
/*
* Setup the child's kernel stack with a pt_regs and call switch_stack() on it.
* It will be unnested during _resume and _ret_from_sys_call when the new thread
generic-y += sections.h
generic-y += sembuf.h
generic-y += shmbuf.h
-generic-y += siginfo.h
generic-y += socket.h
generic-y += sockios.h
generic-y += statfs.h
#define KSTK_ESP(tsk) ((tsk) == current ? rdusp() : (tsk)->thread.usp)
-extern unsigned long thread_saved_pc(struct task_struct *tsk);
-
/* Free all resources held by a thread. */
static inline void release_thread(struct task_struct *dead_task)
{
# UAPI Header export list
include include/uapi/asm-generic/Kbuild.asm
+
+generic-y += siginfo.h
#define release_segments(mm) do { } while (0)
#define forget_segments() do { } while (0)
-/*
- * Return saved PC of a blocked thread.
- */
-extern unsigned long thread_saved_pc(struct task_struct *tsk);
-
unsigned long get_wchan(struct task_struct *p);
#define KSTK_EIP(tsk) ((tsk)->thread.frame0->pc)
return 0;
}
-unsigned long thread_saved_pc(struct task_struct *tsk)
-{
- /* Check whether the thread is blocked in resume() */
- if (in_sched_functions(tsk->thread.pc))
- return ((unsigned long *)tsk->thread.fp)[2];
- else
- return tsk->thread.pc;
-}
-
int elf_check_arch(const struct elf32_hdr *hdr)
{
unsigned long hsr0 = __get_HSR(0);
select OF_IRQ
select OF_EARLY_FLATTREE
select HAVE_MEMBLOCK
- select CLKSRC_OF
+ select TIMER_OF
select H8300_TMR8
select HAVE_KERNEL_GZIP
select HAVE_KERNEL_LZO
generic-y += setup.h
generic-y += shmbuf.h
generic-y += shmparam.h
-generic-y += siginfo.h
generic-y += sizes.h
generic-y += socket.h
generic-y += sockios.h
{
}
-/*
- * Return saved PC of a blocked thread.
- */
-unsigned long thread_saved_pc(struct task_struct *tsk);
unsigned long get_wchan(struct task_struct *p);
#define KSTK_EIP(tsk) \
# UAPI Header export list
include include/uapi/asm-generic/Kbuild.asm
+
+generic-y += siginfo.h
return 0;
}
-unsigned long thread_saved_pc(struct task_struct *tsk)
-{
- return ((struct pt_regs *)tsk->thread.esp0)->pc;
-}
-
unsigned long get_wchan(struct task_struct *p)
{
unsigned long fp, pc;
void __init time_init(void)
{
of_clk_init(NULL);
- clocksource_probe();
+ timer_probe();
}
generic-y += serial.h
generic-y += shmbuf.h
generic-y += shmparam.h
-generic-y += siginfo.h
generic-y += sizes.h
generic-y += socket.h
generic-y += sockios.h
/* task_struct, defined elsewhere, is the "process descriptor" */
struct task_struct;
-/* this is defined in arch/process.c */
-extern unsigned long thread_saved_pc(struct task_struct *tsk);
-
extern void start_thread(struct pt_regs *, unsigned long, unsigned long);
/*
# UAPI Header export list
include include/uapi/asm-generic/Kbuild.asm
+
+generic-y += siginfo.h
local_irq_enable();
}
-/*
- * Return saved PC of a blocked thread
- */
-unsigned long thread_saved_pc(struct task_struct *tsk)
-{
- return 0;
-}
-
/*
* Copy architecture-specific thread state
*/
}
state->xmit.head = state->xmit.tail = 0;
-
- /*
- * Set up the tty->alt_speed kludge
- */
- if ((port->flags & ASYNC_SPD_MASK) == ASYNC_SPD_HI)
- tty->alt_speed = 57600;
- if ((port->flags & ASYNC_SPD_MASK) == ASYNC_SPD_VHI)
- tty->alt_speed = 115200;
- if ((port->flags & ASYNC_SPD_MASK) == ASYNC_SPD_SHI)
- tty->alt_speed = 230400;
- if ((port->flags & ASYNC_SPD_MASK) == ASYNC_SPD_WARP)
- tty->alt_speed = 460800;
-
errout:
local_irq_restore(flags);
return retval;
* following the barrier will arrive after all previous writes. For most
* ia64 platforms, this is a simple 'mf.a' instruction.
*
- * See Documentation/DocBook/deviceiobook.tmpl for more information.
+ * See Documentation/driver-api/device-io.rst for more information.
*/
static inline void ___ia64_mmiowb(void)
{
*unat = (*unat & ~mask) | (nat << bit);
}
-/*
- * Return saved PC of a blocked thread.
- * Note that the only way T can block is through a call to schedule() -> switch_to().
- */
-static inline unsigned long
-thread_saved_pc (struct task_struct *t)
-{
- struct unw_frame_info info;
- unsigned long ip;
-
- unw_init_from_blocked_task(&info, t);
- if (unw_unwind(&info) < 0)
- return 0;
- unw_get_ip(&info, &ip);
- return ip;
-}
-
/*
* Get the current instruction/program counter value.
*/
+++ /dev/null
-/*
- * Based on <asm-i386/siginfo.h>.
- *
- * Modified 1998-2002
- * David Mosberger-Tang <davidm@hpl.hp.com>, Hewlett-Packard Co
- */
-#ifndef _ASM_IA64_SIGINFO_H
-#define _ASM_IA64_SIGINFO_H
-
-#include <linux/string.h>
-#include <uapi/asm/siginfo.h>
-
-static inline void
-copy_siginfo (siginfo_t *to, siginfo_t *from)
-{
- if (from->si_code < 0)
- memcpy(to, from, sizeof(siginfo_t));
- else
- /* _sigchld is currently the largest know union member */
- memcpy(to, from, 4*sizeof(int) + sizeof(from->_sifields._sigchld));
-}
-
-#endif /* _ASM_IA64_SIGINFO_H */
#define __ARCH_SI_PREAMBLE_SIZE (4 * sizeof(int))
#define HAVE_ARCH_SIGINFO_T
-#define HAVE_ARCH_COPY_SIGINFO
#define HAVE_ARCH_COPY_SIGINFO_TO_USER
#include <asm-generic/siginfo.h>
/**
* __sn_mmiowb - I/O space memory barrier
*
- * See arch/ia64/include/asm/io.h and Documentation/DocBook/deviceiobook.tmpl
+ * See arch/ia64/include/asm/io.h and Documentation/driver-api/device-io.rst
* for details.
*
* On SN2, we wait for the PIO_WRITE_STATUS SHub register to clear.
extern void copy_segments(struct task_struct *p, struct mm_struct * mm);
extern void release_segments(struct mm_struct * mm);
-extern unsigned long thread_saved_pc(struct task_struct *);
-
/* Copy and release all segment info associated with a VM */
#define copy_segments(p, mm) do { } while (0)
#define release_segments(mm) do { } while (0)
# UAPI Header export list
include include/uapi/asm-generic/Kbuild.asm
+
+generic-y += siginfo.h
+++ /dev/null
-#ifndef _M32R_SIGINFO_H
-#define _M32R_SIGINFO_H
-
-#include <asm-generic/siginfo.h>
-
-#endif /* _M32R_SIGINFO_H */
#include <linux/err.h>
-/*
- * Return saved PC of a blocked thread.
- */
-unsigned long thread_saved_pc(struct task_struct *tsk)
-{
- return tsk->thread.lr;
-}
-
void (*pm_power_off)(void) = NULL;
EXPORT_SYMBOL(pm_power_off);
CONFIG_SYSV68_PARTITION=y
CONFIG_IOSCHED_DEADLINE=m
CONFIG_MQ_IOSCHED_DEADLINE=m
+CONFIG_MQ_IOSCHED_KYBER=m
+CONFIG_IOSCHED_BFQ=m
CONFIG_KEXEC=y
CONFIG_BOOTINFO_PROC=y
CONFIG_M68020=y
CONFIG_SCSI_ZORRO7XX=y
CONFIG_MD=y
CONFIG_MD_LINEAR=m
-CONFIG_MD_RAID0=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_UEVENT=y
CONFIG_DM_LOG_WRITES=m
+CONFIG_DM_INTEGRITY=m
CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
# CONFIG_NET_VENDOR_STMICRO is not set
# CONFIG_NET_VENDOR_VIA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
+# CONFIG_NET_VENDOR_SYNOPSYS is not set
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
# CONFIG_SECTION_MISMATCH_WARN_ONLY is not set
CONFIG_MAGIC_SYSRQ=y
CONFIG_WW_MUTEX_SELFTEST=m
+CONFIG_TEST_LIST_SORT=m
+CONFIG_TEST_SORT=m
CONFIG_ATOMIC64_SELFTEST=m
CONFIG_ASYNC_RAID6_TEST=m
CONFIG_TEST_HEXDUMP=m
CONFIG_TEST_STATIC_KEYS=m
CONFIG_EARLY_PRINTK=y
CONFIG_ENCRYPTED_KEYS=m
+CONFIG_HARDENED_USERCOPY=y
CONFIG_CRYPTO_RSA=m
CONFIG_CRYPTO_DH=m
CONFIG_CRYPTO_ECDH=m
CONFIG_SYSV68_PARTITION=y
CONFIG_IOSCHED_DEADLINE=m
CONFIG_MQ_IOSCHED_DEADLINE=m
+CONFIG_MQ_IOSCHED_KYBER=m
+CONFIG_IOSCHED_BFQ=m
CONFIG_KEXEC=y
CONFIG_BOOTINFO_PROC=y
CONFIG_M68020=y
CONFIG_ISCSI_BOOT_SYSFS=m
CONFIG_MD=y
CONFIG_MD_LINEAR=m
-CONFIG_MD_RAID0=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_UEVENT=y
CONFIG_DM_LOG_WRITES=m
+CONFIG_DM_INTEGRITY=m
CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
# CONFIG_NET_VENDOR_STMICRO is not set
# CONFIG_NET_VENDOR_VIA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
+# CONFIG_NET_VENDOR_SYNOPSYS is not set
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
# CONFIG_SECTION_MISMATCH_WARN_ONLY is not set
CONFIG_MAGIC_SYSRQ=y
CONFIG_WW_MUTEX_SELFTEST=m
+CONFIG_TEST_LIST_SORT=m
+CONFIG_TEST_SORT=m
CONFIG_ATOMIC64_SELFTEST=m
CONFIG_ASYNC_RAID6_TEST=m
CONFIG_TEST_HEXDUMP=m
CONFIG_TEST_STATIC_KEYS=m
CONFIG_EARLY_PRINTK=y
CONFIG_ENCRYPTED_KEYS=m
+CONFIG_HARDENED_USERCOPY=y
CONFIG_CRYPTO_RSA=m
CONFIG_CRYPTO_DH=m
CONFIG_CRYPTO_ECDH=m
CONFIG_SYSV68_PARTITION=y
CONFIG_IOSCHED_DEADLINE=m
CONFIG_MQ_IOSCHED_DEADLINE=m
+CONFIG_MQ_IOSCHED_KYBER=m
+CONFIG_IOSCHED_BFQ=m
CONFIG_KEXEC=y
CONFIG_BOOTINFO_PROC=y
CONFIG_M68020=y
CONFIG_ATARI_SCSI=y
CONFIG_MD=y
CONFIG_MD_LINEAR=m
-CONFIG_MD_RAID0=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_UEVENT=y
CONFIG_DM_LOG_WRITES=m
+CONFIG_DM_INTEGRITY=m
CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
# CONFIG_NET_VENDOR_STMICRO is not set
# CONFIG_NET_VENDOR_VIA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
+# CONFIG_NET_VENDOR_SYNOPSYS is not set
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
# CONFIG_SECTION_MISMATCH_WARN_ONLY is not set
CONFIG_MAGIC_SYSRQ=y
CONFIG_WW_MUTEX_SELFTEST=m
+CONFIG_TEST_LIST_SORT=m
+CONFIG_TEST_SORT=m
CONFIG_ATOMIC64_SELFTEST=m
CONFIG_ASYNC_RAID6_TEST=m
CONFIG_TEST_HEXDUMP=m
CONFIG_TEST_STATIC_KEYS=m
CONFIG_EARLY_PRINTK=y
CONFIG_ENCRYPTED_KEYS=m
+CONFIG_HARDENED_USERCOPY=y
CONFIG_CRYPTO_RSA=m
CONFIG_CRYPTO_DH=m
CONFIG_CRYPTO_ECDH=m
# CONFIG_EFI_PARTITION is not set
CONFIG_IOSCHED_DEADLINE=m
CONFIG_MQ_IOSCHED_DEADLINE=m
+CONFIG_MQ_IOSCHED_KYBER=m
+CONFIG_IOSCHED_BFQ=m
CONFIG_KEXEC=y
CONFIG_BOOTINFO_PROC=y
CONFIG_M68040=y
CONFIG_BVME6000_SCSI=y
CONFIG_MD=y
CONFIG_MD_LINEAR=m
-CONFIG_MD_RAID0=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_UEVENT=y
CONFIG_DM_LOG_WRITES=m
+CONFIG_DM_INTEGRITY=m
CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
# CONFIG_NET_VENDOR_STMICRO is not set
# CONFIG_NET_VENDOR_VIA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
+# CONFIG_NET_VENDOR_SYNOPSYS is not set
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
# CONFIG_SECTION_MISMATCH_WARN_ONLY is not set
CONFIG_MAGIC_SYSRQ=y
CONFIG_WW_MUTEX_SELFTEST=m
+CONFIG_TEST_LIST_SORT=m
+CONFIG_TEST_SORT=m
CONFIG_ATOMIC64_SELFTEST=m
CONFIG_ASYNC_RAID6_TEST=m
CONFIG_TEST_HEXDUMP=m
CONFIG_TEST_STATIC_KEYS=m
CONFIG_EARLY_PRINTK=y
CONFIG_ENCRYPTED_KEYS=m
+CONFIG_HARDENED_USERCOPY=y
CONFIG_CRYPTO_RSA=m
CONFIG_CRYPTO_DH=m
CONFIG_CRYPTO_ECDH=m
CONFIG_SYSV68_PARTITION=y
CONFIG_IOSCHED_DEADLINE=m
CONFIG_MQ_IOSCHED_DEADLINE=m
+CONFIG_MQ_IOSCHED_KYBER=m
+CONFIG_IOSCHED_BFQ=m
CONFIG_KEXEC=y
CONFIG_BOOTINFO_PROC=y
CONFIG_M68020=y
CONFIG_ISCSI_BOOT_SYSFS=m
CONFIG_MD=y
CONFIG_MD_LINEAR=m
-CONFIG_MD_RAID0=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_UEVENT=y
CONFIG_DM_LOG_WRITES=m
+CONFIG_DM_INTEGRITY=m
CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
# CONFIG_NET_VENDOR_STMICRO is not set
# CONFIG_NET_VENDOR_VIA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
+# CONFIG_NET_VENDOR_SYNOPSYS is not set
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
# CONFIG_SECTION_MISMATCH_WARN_ONLY is not set
CONFIG_MAGIC_SYSRQ=y
CONFIG_WW_MUTEX_SELFTEST=m
+CONFIG_TEST_LIST_SORT=m
+CONFIG_TEST_SORT=m
CONFIG_ATOMIC64_SELFTEST=m
CONFIG_ASYNC_RAID6_TEST=m
CONFIG_TEST_HEXDUMP=m
CONFIG_TEST_STATIC_KEYS=m
CONFIG_EARLY_PRINTK=y
CONFIG_ENCRYPTED_KEYS=m
+CONFIG_HARDENED_USERCOPY=y
CONFIG_CRYPTO_RSA=m
CONFIG_CRYPTO_DH=m
CONFIG_CRYPTO_ECDH=m
CONFIG_SYSV68_PARTITION=y
CONFIG_IOSCHED_DEADLINE=m
CONFIG_MQ_IOSCHED_DEADLINE=m
+CONFIG_MQ_IOSCHED_KYBER=m
+CONFIG_IOSCHED_BFQ=m
CONFIG_KEXEC=y
CONFIG_BOOTINFO_PROC=y
CONFIG_M68020=y
CONFIG_SCSI_MAC_ESP=y
CONFIG_MD=y
CONFIG_MD_LINEAR=m
-CONFIG_MD_RAID0=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_UEVENT=y
CONFIG_DM_LOG_WRITES=m
+CONFIG_DM_INTEGRITY=m
CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
# CONFIG_NET_VENDOR_STMICRO is not set
# CONFIG_NET_VENDOR_VIA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
+# CONFIG_NET_VENDOR_SYNOPSYS is not set
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
# CONFIG_SECTION_MISMATCH_WARN_ONLY is not set
CONFIG_MAGIC_SYSRQ=y
CONFIG_WW_MUTEX_SELFTEST=m
+CONFIG_TEST_LIST_SORT=m
+CONFIG_TEST_SORT=m
CONFIG_ATOMIC64_SELFTEST=m
CONFIG_ASYNC_RAID6_TEST=m
CONFIG_TEST_HEXDUMP=m
CONFIG_TEST_STATIC_KEYS=m
CONFIG_EARLY_PRINTK=y
CONFIG_ENCRYPTED_KEYS=m
+CONFIG_HARDENED_USERCOPY=y
CONFIG_CRYPTO_RSA=m
CONFIG_CRYPTO_DH=m
CONFIG_CRYPTO_ECDH=m
# CONFIG_EFI_PARTITION is not set
CONFIG_IOSCHED_DEADLINE=m
CONFIG_MQ_IOSCHED_DEADLINE=m
+CONFIG_MQ_IOSCHED_KYBER=m
+CONFIG_IOSCHED_BFQ=m
CONFIG_KEXEC=y
CONFIG_BOOTINFO_PROC=y
CONFIG_M68020=y
CONFIG_SUN3X_ESP=y
CONFIG_MD=y
CONFIG_MD_LINEAR=m
-CONFIG_MD_RAID0=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_UEVENT=y
CONFIG_DM_LOG_WRITES=m
+CONFIG_DM_INTEGRITY=m
CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
# CONFIG_NET_VENDOR_STMICRO is not set
# CONFIG_NET_VENDOR_VIA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
+# CONFIG_NET_VENDOR_SYNOPSYS is not set
CONFIG_PLIP=m
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
# CONFIG_SECTION_MISMATCH_WARN_ONLY is not set
CONFIG_MAGIC_SYSRQ=y
CONFIG_WW_MUTEX_SELFTEST=m
+CONFIG_TEST_LIST_SORT=m
+CONFIG_TEST_SORT=m
CONFIG_ATOMIC64_SELFTEST=m
CONFIG_ASYNC_RAID6_TEST=m
CONFIG_TEST_HEXDUMP=m
CONFIG_TEST_STATIC_KEYS=m
CONFIG_EARLY_PRINTK=y
CONFIG_ENCRYPTED_KEYS=m
+CONFIG_HARDENED_USERCOPY=y
CONFIG_CRYPTO_RSA=m
CONFIG_CRYPTO_DH=m
CONFIG_CRYPTO_ECDH=m
# CONFIG_EFI_PARTITION is not set
CONFIG_IOSCHED_DEADLINE=m
CONFIG_MQ_IOSCHED_DEADLINE=m
+CONFIG_MQ_IOSCHED_KYBER=m
+CONFIG_IOSCHED_BFQ=m
CONFIG_KEXEC=y
CONFIG_BOOTINFO_PROC=y
CONFIG_M68030=y
CONFIG_MVME147_SCSI=y
CONFIG_MD=y
CONFIG_MD_LINEAR=m
-CONFIG_MD_RAID0=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_UEVENT=y
CONFIG_DM_LOG_WRITES=m
+CONFIG_DM_INTEGRITY=m
CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
# CONFIG_NET_VENDOR_STMICRO is not set
# CONFIG_NET_VENDOR_VIA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
+# CONFIG_NET_VENDOR_SYNOPSYS is not set
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
# CONFIG_SECTION_MISMATCH_WARN_ONLY is not set
CONFIG_MAGIC_SYSRQ=y
CONFIG_WW_MUTEX_SELFTEST=m
+CONFIG_TEST_LIST_SORT=m
+CONFIG_TEST_SORT=m
CONFIG_ATOMIC64_SELFTEST=m
CONFIG_ASYNC_RAID6_TEST=m
CONFIG_TEST_HEXDUMP=m
CONFIG_TEST_STATIC_KEYS=m
CONFIG_EARLY_PRINTK=y
CONFIG_ENCRYPTED_KEYS=m
+CONFIG_HARDENED_USERCOPY=y
CONFIG_CRYPTO_RSA=m
CONFIG_CRYPTO_DH=m
CONFIG_CRYPTO_ECDH=m
# CONFIG_EFI_PARTITION is not set
CONFIG_IOSCHED_DEADLINE=m
CONFIG_MQ_IOSCHED_DEADLINE=m
+CONFIG_MQ_IOSCHED_KYBER=m
+CONFIG_IOSCHED_BFQ=m
CONFIG_KEXEC=y
CONFIG_BOOTINFO_PROC=y
CONFIG_M68040=y
CONFIG_MVME16x_SCSI=y
CONFIG_MD=y
CONFIG_MD_LINEAR=m
-CONFIG_MD_RAID0=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_UEVENT=y
CONFIG_DM_LOG_WRITES=m
+CONFIG_DM_INTEGRITY=m
CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
# CONFIG_NET_VENDOR_STMICRO is not set
# CONFIG_NET_VENDOR_VIA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
+# CONFIG_NET_VENDOR_SYNOPSYS is not set
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
# CONFIG_SECTION_MISMATCH_WARN_ONLY is not set
CONFIG_MAGIC_SYSRQ=y
CONFIG_WW_MUTEX_SELFTEST=m
+CONFIG_TEST_LIST_SORT=m
+CONFIG_TEST_SORT=m
CONFIG_ATOMIC64_SELFTEST=m
CONFIG_ASYNC_RAID6_TEST=m
CONFIG_TEST_HEXDUMP=m
CONFIG_TEST_STATIC_KEYS=m
CONFIG_EARLY_PRINTK=y
CONFIG_ENCRYPTED_KEYS=m
+CONFIG_HARDENED_USERCOPY=y
CONFIG_CRYPTO_RSA=m
CONFIG_CRYPTO_DH=m
CONFIG_CRYPTO_ECDH=m
CONFIG_SYSV68_PARTITION=y
CONFIG_IOSCHED_DEADLINE=m
CONFIG_MQ_IOSCHED_DEADLINE=m
+CONFIG_MQ_IOSCHED_KYBER=m
+CONFIG_IOSCHED_BFQ=m
CONFIG_KEXEC=y
CONFIG_BOOTINFO_PROC=y
CONFIG_M68040=y
CONFIG_ISCSI_BOOT_SYSFS=m
CONFIG_MD=y
CONFIG_MD_LINEAR=m
-CONFIG_MD_RAID0=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_UEVENT=y
CONFIG_DM_LOG_WRITES=m
+CONFIG_DM_INTEGRITY=m
CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
# CONFIG_NET_VENDOR_STMICRO is not set
# CONFIG_NET_VENDOR_VIA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
+# CONFIG_NET_VENDOR_SYNOPSYS is not set
CONFIG_PLIP=m
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
# CONFIG_SECTION_MISMATCH_WARN_ONLY is not set
CONFIG_MAGIC_SYSRQ=y
CONFIG_WW_MUTEX_SELFTEST=m
+CONFIG_TEST_LIST_SORT=m
+CONFIG_TEST_SORT=m
CONFIG_ATOMIC64_SELFTEST=m
CONFIG_ASYNC_RAID6_TEST=m
CONFIG_TEST_HEXDUMP=m
CONFIG_TEST_STATIC_KEYS=m
CONFIG_EARLY_PRINTK=y
CONFIG_ENCRYPTED_KEYS=m
+CONFIG_HARDENED_USERCOPY=y
CONFIG_CRYPTO_RSA=m
CONFIG_CRYPTO_DH=m
CONFIG_CRYPTO_ECDH=m
CONFIG_SYSV68_PARTITION=y
CONFIG_IOSCHED_DEADLINE=m
CONFIG_MQ_IOSCHED_DEADLINE=m
+CONFIG_MQ_IOSCHED_KYBER=m
+CONFIG_IOSCHED_BFQ=m
CONFIG_KEXEC=y
CONFIG_BOOTINFO_PROC=y
CONFIG_SUN3=y
CONFIG_SUN3_SCSI=y
CONFIG_MD=y
CONFIG_MD_LINEAR=m
-CONFIG_MD_RAID0=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_UEVENT=y
CONFIG_DM_LOG_WRITES=m
+CONFIG_DM_INTEGRITY=m
CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
# CONFIG_NET_VENDOR_SUN is not set
# CONFIG_NET_VENDOR_VIA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
+# CONFIG_NET_VENDOR_SYNOPSYS is not set
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
# CONFIG_SECTION_MISMATCH_WARN_ONLY is not set
CONFIG_MAGIC_SYSRQ=y
CONFIG_WW_MUTEX_SELFTEST=m
+CONFIG_TEST_LIST_SORT=m
+CONFIG_TEST_SORT=m
CONFIG_ATOMIC64_SELFTEST=m
CONFIG_ASYNC_RAID6_TEST=m
CONFIG_TEST_HEXDUMP=m
CONFIG_TEST_UDELAY=m
CONFIG_TEST_STATIC_KEYS=m
CONFIG_ENCRYPTED_KEYS=m
+CONFIG_HARDENED_USERCOPY=y
CONFIG_CRYPTO_RSA=m
CONFIG_CRYPTO_DH=m
CONFIG_CRYPTO_ECDH=m
CONFIG_SYSV68_PARTITION=y
CONFIG_IOSCHED_DEADLINE=m
CONFIG_MQ_IOSCHED_DEADLINE=m
+CONFIG_MQ_IOSCHED_KYBER=m
+CONFIG_IOSCHED_BFQ=m
CONFIG_KEXEC=y
CONFIG_BOOTINFO_PROC=y
CONFIG_SUN3X=y
CONFIG_SUN3X_ESP=y
CONFIG_MD=y
CONFIG_MD_LINEAR=m
-CONFIG_MD_RAID0=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_UEVENT=y
CONFIG_DM_LOG_WRITES=m
+CONFIG_DM_INTEGRITY=m
CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
# CONFIG_NET_VENDOR_STMICRO is not set
# CONFIG_NET_VENDOR_VIA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
+# CONFIG_NET_VENDOR_SYNOPSYS is not set
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
# CONFIG_SECTION_MISMATCH_WARN_ONLY is not set
CONFIG_MAGIC_SYSRQ=y
CONFIG_WW_MUTEX_SELFTEST=m
+CONFIG_TEST_LIST_SORT=m
+CONFIG_TEST_SORT=m
CONFIG_ATOMIC64_SELFTEST=m
CONFIG_ASYNC_RAID6_TEST=m
CONFIG_TEST_HEXDUMP=m
CONFIG_TEST_STATIC_KEYS=m
CONFIG_EARLY_PRINTK=y
CONFIG_ENCRYPTED_KEYS=m
+CONFIG_HARDENED_USERCOPY=y
CONFIG_CRYPTO_RSA=m
CONFIG_CRYPTO_DH=m
CONFIG_CRYPTO_ECDH=m
generic-y += resource.h
generic-y += sections.h
generic-y += shmparam.h
-generic-y += siginfo.h
generic-y += spinlock.h
generic-y += statfs.h
generic-y += termios.h
{
}
-extern unsigned long thread_saved_pc(struct task_struct *tsk);
-
unsigned long get_wchan(struct task_struct *p);
#define KSTK_EIP(tsk) \
#endif /* !CONFIG_CPU_HAS_NO_BITFIELDS */
-#ifndef __uClinux__
-extern void ptrace_signal_deliver(void);
-#define ptrace_signal_deliver ptrace_signal_deliver
-#endif /* __uClinux__ */
-
#endif /* _M68K_SIGNAL_H */
generic-y += msgbuf.h
generic-y += sembuf.h
generic-y += shmbuf.h
+generic-y += siginfo.h
generic-y += socket.h
generic-y += sockios.h
generic-y += termbits.h
asmlinkage void ret_from_fork(void);
asmlinkage void ret_from_kernel_thread(void);
-
-/*
- * Return saved PC from a blocked thread
- */
-unsigned long thread_saved_pc(struct task_struct *tsk)
-{
- struct switch_stack *sw = (struct switch_stack *)tsk->thread.ksp;
- /* Check whether the thread is blocked in resume() */
- if (in_sched_functions(sw->retpc))
- return ((unsigned long *)sw->a6)[1];
- else
- return sw->retpc;
-}
-
void arch_cpu_idle(void)
{
#if defined(MACH_ATARI_ONLY)
return 1;
}
-void ptrace_signal_deliver(void)
-{
- struct pt_regs *regs = signal_pt_regs();
- if (regs->orig_d0 < 0)
- return;
- switch (regs->d0) {
- case -ERESTARTNOHAND:
- case -ERESTARTSYS:
- case -ERESTARTNOINTR:
- regs->d0 = regs->orig_d0;
- regs->orig_d0 = -1;
- regs->pc -= 2;
- break;
- }
-}
-
static inline void push_cache (unsigned long vaddr)
{
/*
{
unsigned int cpu = smp_processor_id();
- if (system_state == SYSTEM_BOOTING ||
- system_state == SYSTEM_RUNNING) {
+ if (system_state <= SYSTEM_RUNNING) {
spin_lock(&stop_lock);
pr_crit("CPU%u: stopping\n", cpu);
dump_stack();
select ARCH_MIGHT_HAVE_PC_PARPORT
select ARCH_WANT_IPC_PARSE_VERSION
select BUILDTIME_EXTABLE_SORT
- select CLKSRC_OF
+ select TIMER_OF
select CLONE_BACKWARDS3
select COMMON_CLK
select GENERIC_ATOMIC64
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
-CONFIG_FHANDLE=y
CONFIG_AUDIT=y
-CONFIG_AUDIT_LOGINUID_IMMUTABLE=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_SYSFS_DEPRECATED=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_INET=y
-# CONFIG_INET_LRO is not set
# CONFIG_IPV6 is not set
+CONFIG_BRIDGE=m
CONFIG_MTD=y
-CONFIG_PROC_DEVICETREE=y
+CONFIG_MTD_CFI=y
+CONFIG_MTD_CFI_INTELEXT=y
+CONFIG_MTD_CFI_AMDSTD=y
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_SIZE=8192
CONFIG_NETDEVICES=y
# CONFIG_VT is not set
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
+CONFIG_SERIAL_OF_PLATFORM=y
CONFIG_SERIAL_UARTLITE=y
CONFIG_SERIAL_UARTLITE_CONSOLE=y
-CONFIG_SERIAL_OF_PLATFORM=y
# CONFIG_HW_RANDOM is not set
CONFIG_XILINX_HWICAP=y
CONFIG_I2C=y
CONFIG_FB_XILINX=y
# CONFIG_USB_SUPPORT is not set
CONFIG_UIO=y
-CONFIG_UIO_PDRV=y
CONFIG_UIO_PDRV_GENIRQ=y
CONFIG_UIO_DMEM_GENIRQ=y
CONFIG_EXT2_FS=y
CONFIG_CIFS=y
CONFIG_CIFS_STATS=y
CONFIG_CIFS_STATS2=y
-CONFIG_DETECT_HUNG_TASK=y
+CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_SLAB=y
+CONFIG_DETECT_HUNG_TASK=y
CONFIG_DEBUG_SPINLOCK=y
-CONFIG_DEBUG_INFO=y
CONFIG_KGDB=y
CONFIG_KGDB_TESTS=y
CONFIG_KGDB_KDB=y
CONFIG_EARLY_PRINTK=y
CONFIG_KEYS=y
CONFIG_ENCRYPTED_KEYS=y
-# CONFIG_CRYPTO_ANSI_CPRNG is not set
-CONFIG_EXPERIMENTAL=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
-CONFIG_FHANDLE=y
CONFIG_AUDIT=y
-CONFIG_AUDIT_LOGINUID_IMMUTABLE=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
CONFIG_IKCONFIG=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_INET=y
-# CONFIG_INET_LRO is not set
# CONFIG_IPV6 is not set
CONFIG_MTD=y
CONFIG_MTD_CMDLINE_PARTS=y
-CONFIG_MTD_CHAR=y
CONFIG_MTD_BLOCK=y
CONFIG_MTD_CFI=y
CONFIG_MTD_CFI_INTELEXT=y
CONFIG_MTD_CFI_AMDSTD=y
CONFIG_MTD_RAM=y
CONFIG_MTD_UCLINUX=y
-CONFIG_PROC_DEVICETREE=y
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_SIZE=8192
CONFIG_NETDEVICES=y
# CONFIG_VT is not set
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
+CONFIG_SERIAL_OF_PLATFORM=y
CONFIG_SERIAL_UARTLITE=y
CONFIG_SERIAL_UARTLITE_CONSOLE=y
-CONFIG_SERIAL_OF_PLATFORM=y
# CONFIG_HW_RANDOM is not set
CONFIG_XILINX_HWICAP=y
CONFIG_I2C=y
CONFIG_FB=y
CONFIG_FB_XILINX=y
# CONFIG_USB_SUPPORT is not set
-CONFIG_UIO=y
-CONFIG_UIO_PDRV=y
-CONFIG_UIO_PDRV_GENIRQ=y
-CONFIG_UIO_DMEM_GENIRQ=y
CONFIG_EXT2_FS=y
# CONFIG_DNOTIFY is not set
CONFIG_CRAMFS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V3_ACL=y
CONFIG_NLS=y
-CONFIG_DETECT_HUNG_TASK=y
+CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_SLAB=y
+CONFIG_DETECT_HUNG_TASK=y
CONFIG_DEBUG_SPINLOCK=y
-CONFIG_DEBUG_INFO=y
CONFIG_EARLY_PRINTK=y
CONFIG_KEYS=y
CONFIG_ENCRYPTED_KEYS=y
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_ARC4=y
CONFIG_CRYPTO_DES=y
-# CONFIG_CRYPTO_ANSI_CPRNG is not set
generic-y += barrier.h
+generic-y += bitops.h
+generic-y += bitsperlong.h
+generic-y += bug.h
+generic-y += bugs.h
generic-y += clkdev.h
generic-y += device.h
+generic-y += div64.h
+generic-y += emergency-restart.h
+generic-y += errno.h
generic-y += exec.h
generic-y += extable.h
+generic-y += fb.h
+generic-y += fcntl.h
+generic-y += hardirq.h
+generic-y += ioctl.h
+generic-y += ioctls.h
+generic-y += ipcbuf.h
+generic-y += irq_regs.h
generic-y += irq_work.h
+generic-y += kdebug.h
+generic-y += kmap_types.h
+generic-y += kprobes.h
+generic-y += linkage.h
+generic-y += local.h
+generic-y += local64.h
generic-y += mcs_spinlock.h
generic-y += mm-arch-hooks.h
+generic-y += mman.h
+generic-y += msgbuf.h
+generic-y += param.h
+generic-y += parport.h
+generic-y += percpu.h
+generic-y += poll.h
generic-y += preempt.h
+generic-y += resource.h
+generic-y += sembuf.h
+generic-y += serial.h
+generic-y += shmbuf.h
+generic-y += shmparam.h
+generic-y += siginfo.h
+generic-y += signal.h
+generic-y += socket.h
+generic-y += sockios.h
+generic-y += stat.h
+generic-y += statfs.h
+generic-y += swab.h
generic-y += syscalls.h
+generic-y += termbits.h
+generic-y += termios.h
+generic-y += topology.h
generic-y += trace_clock.h
+generic-y += ucontext.h
+generic-y += vga.h
generic-y += word-at-a-time.h
-generic-y += kprobes.h
+generic-y += xor.h
+++ /dev/null
-#include <asm-generic/bitops.h>
+++ /dev/null
-#include <asm-generic/bug.h>
+++ /dev/null
-#include <asm-generic/bugs.h>
+++ /dev/null
-#include <asm-generic/div64.h>
+++ /dev/null
-#include <asm-generic/emergency-restart.h>
+++ /dev/null
-#include <asm-generic/fb.h>
+++ /dev/null
-#include <asm-generic/hardirq.h>
+++ /dev/null
-#include <asm-generic/irq_regs.h>
+++ /dev/null
-#include <asm-generic/kdebug.h>
+++ /dev/null
-#ifndef _ASM_MICROBLAZE_KMAP_TYPES_H
-#define _ASM_MICROBLAZE_KMAP_TYPES_H
-
-#include <asm-generic/kmap_types.h>
-
-#endif /* _ASM_MICROBLAZE_KMAP_TYPES_H */
+++ /dev/null
-#include <asm-generic/linkage.h>
+++ /dev/null
-#include <asm-generic/local.h>
+++ /dev/null
-#include <asm-generic/local64.h>
+++ /dev/null
-#include <asm-generic/parport.h>
+++ /dev/null
-#include <asm-generic/percpu.h>
{
}
-extern unsigned long thread_saved_pc(struct task_struct *t);
-
extern unsigned long get_wchan(struct task_struct *p);
# define KSTK_EIP(tsk) (0)
{
}
-/* Return saved (kernel) PC of a blocked thread. */
-# define thread_saved_pc(tsk) \
- ((tsk)->thread.regs ? (tsk)->thread.regs->r15 : 0)
-
unsigned long get_wchan(struct task_struct *p);
/* The size allocated for kernel stacks. This _must_ be a power of two! */
+++ /dev/null
-#include <asm-generic/serial.h>
+++ /dev/null
-#include <asm-generic/shmparam.h>
+++ /dev/null
-#include <asm-generic/topology.h>
+++ /dev/null
-#include <asm-generic/ucontext.h>
#endif /* __ASSEMBLY__ */
-#define __NR_syscalls 398
+#define __NR_syscalls 399
#endif /* _ASM_MICROBLAZE_UNISTD_H */
+++ /dev/null
-#include <asm-generic/vga.h>
+++ /dev/null
-#include <asm-generic/xor.h>
include include/uapi/asm-generic/Kbuild.asm
generic-y += types.h
+generic-y += siginfo.h
+++ /dev/null
-#include <asm-generic/bitsperlong.h>
+++ /dev/null
-#include <asm-generic/errno.h>
+++ /dev/null
-#include <asm-generic/fcntl.h>
+++ /dev/null
-#include <asm-generic/ioctl.h>
+++ /dev/null
-#include <asm-generic/ioctls.h>
+++ /dev/null
-#include <asm-generic/ipcbuf.h>
+++ /dev/null
-#include <asm-generic/kvm_para.h>
+++ /dev/null
-#include <asm-generic/mman.h>
+++ /dev/null
-#include <asm-generic/msgbuf.h>
+++ /dev/null
-#include <asm-generic/param.h>
+++ /dev/null
-#include <asm-generic/poll.h>
+++ /dev/null
-#include <asm-generic/resource.h>
+++ /dev/null
-#include <asm-generic/sembuf.h>
+++ /dev/null
-#include <asm-generic/shmbuf.h>
+++ /dev/null
-#include <asm-generic/siginfo.h>
+++ /dev/null
-#include <asm-generic/signal.h>
+++ /dev/null
-#include <asm-generic/socket.h>
+++ /dev/null
-#include <asm-generic/sockios.h>
+++ /dev/null
-#include <asm-generic/stat.h>
+++ /dev/null
-#include <asm-generic/statfs.h>
+++ /dev/null
-#include <asm-generic/swab.h>
+++ /dev/null
-#include <asm-generic/termbits.h>
+++ /dev/null
-#include <asm-generic/termios.h>
#define __NR_pkey_mprotect 395
#define __NR_pkey_alloc 396
#define __NR_pkey_free 397
+#define __NR_statx 398
#endif /* _UAPI_ASM_MICROBLAZE_UNISTD_H */
if (attrs & DMA_ATTR_SKIP_CPU_SYNC)
continue;
- __dma_sync(page_to_phys(sg_page(sg)) + sg->offset,
- sg->length, direction);
+ __dma_sync(sg_phys(sg), sg->length, direction);
}
return nents;
mfs r11, rmsr; /* save MSR */ \
swi r11, r1, PT_MSR;
-#define RESTORE_REGS \
- lwi r11, r1, PT_MSR; \
- mts rmsr , r11; \
+#define RESTORE_REGS_GP \
lwi r2, r1, PT_R2; /* restore SDA */ \
lwi r3, r1, PT_R3; \
lwi r4, r1, PT_R4; \
lwi r30, r1, PT_R30; \
lwi r31, r1, PT_R31; /* Restore cur task reg */
+#define RESTORE_REGS \
+ lwi r11, r1, PT_MSR; \
+ mts rmsr , r11; \
+ RESTORE_REGS_GP
+
+#define RESTORE_REGS_RTBD \
+ lwi r11, r1, PT_MSR; \
+ andni r11, r11, MSR_EIP; /* clear EIP */ \
+ ori r11, r11, MSR_EE | MSR_BIP; /* set EE and BIP */ \
+ mts rmsr , r11; \
+ RESTORE_REGS_GP
+
#define SAVE_STATE \
swi r1, r0, TOPHYS(PER_CPU(ENTRY_SP)); /* save stack */ \
/* See if already in kernel mode.*/ \
swi CURRENT_TASK, r0, PER_CPU(CURRENT_SAVE); /* save current */
VM_OFF;
tophys(r1,r1);
- RESTORE_REGS;
+ RESTORE_REGS_RTBD;
addik r1, r1, PT_SIZE /* Clean up stack space. */
lwi r1, r1, PT_R1 - PT_SIZE;/* Restore user stack pointer. */
bri 6f;
2: set_bip; /* Ints masked for state restore */
VM_OFF;
tophys(r1,r1);
- RESTORE_REGS;
+ RESTORE_REGS_RTBD;
addik r1, r1, PT_SIZE /* Clean up stack space. */
tovirt(r1,r1);
6:
VM_OFF;
tophys(r1,r1);
- RESTORE_REGS;
+ RESTORE_REGS_RTBD;
addik r1, r1, PT_SIZE /* Clean up stack space. */
lwi r1, r1, PT_R1 - PT_SIZE; /* Restore user stack pointer. */
2: set_bip; /* Ints masked for state restore */
VM_OFF;
tophys(r1,r1);
- RESTORE_REGS;
+ RESTORE_REGS_RTBD;
addik r1, r1, PT_SIZE /* Clean up stack space. */
tovirt(r1,r1);
VM_OFF;
tophys(r1,r1);
/* MS: Restore all regs */
- RESTORE_REGS
+ RESTORE_REGS_RTBD
addik r1, r1, PT_SIZE /* Clean up stack space */
lwi r1, r1, PT_R1 - PT_SIZE; /* Restore user stack pointer */
DBTRAP_return_user: /* MS: Make global symbol for debugging */
2: VM_OFF;
tophys(r1,r1);
/* MS: Restore all regs */
- RESTORE_REGS
+ RESTORE_REGS_RTBD
lwi r14, r1, PT_R14;
lwi r16, r1, PT_PC;
addik r1, r1, PT_SIZE; /* MS: Clean up stack space */
return 0;
}
-#ifndef CONFIG_MMU
-/*
- * Return saved PC of a blocked thread.
- */
-unsigned long thread_saved_pc(struct task_struct *tsk)
-{
- struct cpu_context *ctx =
- &(((struct thread_info *)(tsk->stack))->cpu_context);
-
- /* Check whether the thread is blocked in resume() */
- if (in_sched_functions(ctx->r15))
- return (unsigned long)ctx->r15;
- else
- return ctx->r14;
-}
-#endif
-
unsigned long get_wchan(struct task_struct *p)
{
/* TBD (used by procfs) */
{
of_clk_init(NULL);
setup_cpuinfo_clk();
- clocksource_probe();
+ timer_probe();
}
#ifdef CONFIG_DEBUG_FS
.long sys_pkey_mprotect /* 395 */
.long sys_pkey_alloc
.long sys_pkey_free
+ .long sys_statx
clockevent_xilinx_timer.shift);
clockevent_xilinx_timer.max_delta_ns =
clockevent_delta2ns((u32)~0, &clockevent_xilinx_timer);
+ clockevent_xilinx_timer.max_delta_ticks = (u32)~0;
clockevent_xilinx_timer.min_delta_ns =
clockevent_delta2ns(1, &clockevent_xilinx_timer);
+ clockevent_xilinx_timer.min_delta_ticks = 1;
clockevent_xilinx_timer.cpumask = cpumask_of(0);
clockevents_register_device(&clockevent_xilinx_timer);
return 0;
}
-CLOCKSOURCE_OF_DECLARE(xilinx_timer, "xlnx,xps-timer-1.00.a",
+TIMER_OF_DECLARE(xilinx_timer, "xlnx,xps-timer-1.00.a",
xilinx_timer_init);
{
unsigned long vaddr = (unsigned long) kvaddr & PAGE_MASK;
int type;
+ unsigned int idx;
if (vaddr < __fix_to_virt(FIX_KMAP_END)) {
pagefault_enable();
}
type = kmap_atomic_idx();
-#ifdef CONFIG_DEBUG_HIGHMEM
- {
- unsigned int idx;
-
- idx = type + KM_TYPE_NR * smp_processor_id();
- BUG_ON(vaddr != __fix_to_virt(FIX_KMAP_BEGIN + idx));
- /*
- * force other mappings to Oops if they'll try to access
- * this pte without first remap it
- */
- pte_clear(&init_mm, vaddr, kmap_pte-idx);
- local_flush_tlb_page(NULL, vaddr);
- }
+ idx = type + KM_TYPE_NR * smp_processor_id();
+#ifdef CONFIG_DEBUG_HIGHMEM
+ BUG_ON(vaddr != __fix_to_virt(FIX_KMAP_BEGIN + idx));
#endif
+ /*
+ * force other mappings to Oops if they'll try to access
+ * this pte without first remap it
+ */
+ pte_clear(&init_mm, vaddr, kmap_pte-idx);
+ local_flush_tlb_page(NULL, vaddr);
+
kmap_atomic_idx_pop();
pagefault_enable();
preempt_enable();
}
}
- clocksource_probe();
+ timer_probe();
}
void __init arch_init_irq(void)
#define TIOCGPKT _IOR('T', 0x38, int) /* Get packet mode state */
#define TIOCGPTLCK _IOR('T', 0x39, int) /* Get Pty lock state */
#define TIOCGEXCL _IOR('T', 0x40, int) /* Get exclusive mode state */
+#define TIOCGPTPEER _IOR('T', 0x41, int) /* Safely open the slave */
/* I hope the range from 0x5480 on is free ... */
#define TIOCSCTTY 0x5480 /* become controlling tty */
#include <asm/asm.h>
#include <asm/asmmacro.h>
#include <asm/compiler.h>
+#include <asm/irqflags.h>
#include <asm/regdef.h>
#include <asm/mipsregs.h>
#include <asm/stackframe.h>
andi t0, a2, _TIF_NEED_RESCHED # a2 is preloaded with TI_FLAGS
beqz t0, work_notifysig
work_resched:
+ TRACE_IRQS_OFF
jal schedule
local_irq_disable # make sure need_resched and
beqz t0, work_pending # trace bit set?
local_irq_enable # could let syscall_trace_leave()
# call schedule() instead
+ TRACE_IRQS_ON
move a0, sp
jal syscall_trace_leave
b resume_userspace
beq t0, t1, dtb_found
#endif
li t1, -2
- beq a0, t1, dtb_found
move t2, a1
+ beq a0, t1, dtb_found
li t2, 0
dtb_found:
insn.word = 0; /* nop */
}
- get_online_cpus();
mutex_lock(&text_mutex);
if (IS_ENABLED(CONFIG_CPU_MICROMIPS)) {
insn_p->halfword[0] = insn.word >> 16;
(unsigned long)insn_p + sizeof(*insn_p));
mutex_unlock(&text_mutex);
- put_online_cpus();
}
#endif /* HAVE_JUMP_LABEL */
* state. Actually per-core rather than per-CPU.
*/
static DEFINE_PER_CPU_ALIGNED(u32*, ready_count);
-static DEFINE_PER_CPU_ALIGNED(void*, ready_count_alloc);
/* Indicates online CPUs coupled with the current CPU */
static DEFINE_PER_CPU_ALIGNED(cpumask_t, online_coupled);
{
enum cps_pm_state state;
unsigned core = cpu_data[cpu].core;
- unsigned dlinesz = cpu_data[cpu].dcache.linesz;
void *entry_fn, *core_rc;
for (state = CPS_PM_NC_WAIT; state < CPS_PM_STATE_COUNT; state++) {
}
if (!per_cpu(ready_count, core)) {
- core_rc = kmalloc(dlinesz * 2, GFP_KERNEL);
+ core_rc = kmalloc(sizeof(u32), GFP_KERNEL);
if (!core_rc) {
pr_err("Failed allocate core %u ready_count\n", core);
return -ENOMEM;
}
- per_cpu(ready_count_alloc, core) = core_rc;
-
- /* Ensure ready_count is aligned to a cacheline boundary */
- core_rc += dlinesz - 1;
- core_rc = (void *)((unsigned long)core_rc & ~(dlinesz - 1));
per_cpu(ready_count, core) = core_rc;
}
{
struct pt_regs regs;
mm_segment_t old_fs = get_fs();
+
+ regs.cp0_status = KSU_KERNEL;
if (sp) {
regs.regs[29] = (unsigned long)sp;
regs.regs[31] = 0;
int kvm_mips_host_tlb_inv(struct kvm_vcpu *vcpu, unsigned long va,
bool user, bool kernel)
{
- int idx_user, idx_kernel;
+ /*
+ * Initialize idx_user and idx_kernel to workaround bogus
+ * maybe-initialized warning when using GCC 6.
+ */
+ int idx_user = 0, idx_kernel = 0;
unsigned long flags, old_entryhi;
local_irq_save(flags);
return ieee754dp_nanxcpt(z);
case IEEE754_CLASS_DNORM:
DPDNORMZ;
- /* QNAN is handled separately below */
+ /* QNAN and ZERO cases are handled separately below */
}
switch (CLPAIR(xc, yc)) {
}
assert(rm & (DP_HIDDEN_BIT << 3));
+ if (zc == IEEE754_CLASS_ZERO)
+ return ieee754dp_format(rs, re, rm);
+
/* And now the addition */
assert(zm & DP_HIDDEN_BIT);
return ieee754sp_nanxcpt(z);
case IEEE754_CLASS_DNORM:
SPDNORMZ;
- /* QNAN is handled separately below */
+ /* QNAN and ZERO cases are handled separately below */
}
switch (CLPAIR(xc, yc)) {
}
assert(rm & (SP_HIDDEN_BIT << 3));
+ if (zc == IEEE754_CLASS_ZERO)
+ return ieee754sp_format(rs, re, rm);
+
/* And now the addition */
assert(zm & SP_HIDDEN_BIT);
* systems and only the R10000 and R12000 are used in such systems, the
* SGI IP28 Indigo² rsp. SGI IP32 aka O2.
*/
-static inline int cpu_needs_post_dma_flush(struct device *dev)
+static inline bool cpu_needs_post_dma_flush(struct device *dev)
{
- return !plat_device_is_coherent(dev) &&
- (boot_cpu_type() == CPU_R10000 ||
- boot_cpu_type() == CPU_R12000 ||
- boot_cpu_type() == CPU_BMIPS5000);
+ if (plat_device_is_coherent(dev))
+ return false;
+
+ switch (boot_cpu_type()) {
+ case CPU_R10000:
+ case CPU_R12000:
+ case CPU_BMIPS5000:
+ return true;
+
+ default:
+ /*
+ * Presence of MAARs suggests that the CPU supports
+ * speculatively prefetching data, and therefore requires
+ * the post-DMA flush/invalidate.
+ */
+ return cpu_has_maar;
+ }
}
static gfp_t massage_gfp_flags(const struct device *dev, gfp_t gfp)
(freq%1000000)*100/1000000);
#ifdef CONFIG_CLKSRC_MIPS_GIC
update_gic_frequency_dt();
- clocksource_probe();
+ timer_probe();
#endif
}
#endif
pr_info("CPU Clock: %ldMHz\n", rate / 1000000);
mips_hpt_frequency = rate / 2;
- clocksource_probe();
+ timer_probe();
}
struct clk *clk;
of_clk_init(NULL);
- clocksource_probe();
+ timer_probe();
np = of_get_cpu_node(0, NULL);
if (!np) {
bool
depends on SOC_RT305X || SOC_MT7620
default y
- select CLKSRC_OF
+ select TIMER_OF
select CLKSRC_MMIO
config RALINK_ILL_ACC
return 0;
}
-CLOCKSOURCE_OF_DECLARE(systick, "ralink,cevt-systick", ralink_systick_init);
+TIMER_OF_DECLARE(systick, "ralink,cevt-systick", ralink_systick_init);
pr_info("CPU Clock: %ldMHz\n", clk_get_rate(clk) / 1000000);
mips_hpt_frequency = clk_get_rate(clk) / 2;
clk_put(clk);
- clocksource_probe();
+ timer_probe();
}
ralink_of_remap();
of_clk_init(NULL);
- clocksource_probe();
+ timer_probe();
}
return (len >= PAGE_SIZE) ? (PAGE_SIZE - 1) : len;
}
+static DEVICE_ATTR_RO(modalias);
static ssize_t name_show(struct device *dev,
struct device_attribute *attr, char *buf)
giodev = to_gio_device(dev);
return sprintf(buf, "%s", giodev->name);
}
+static DEVICE_ATTR_RO(name);
static ssize_t id_show(struct device *dev,
struct device_attribute *attr, char *buf)
giodev = to_gio_device(dev);
return sprintf(buf, "%x", giodev->id.id);
}
+static DEVICE_ATTR_RO(id);
-static struct device_attribute gio_dev_attrs[] = {
- __ATTR_RO(modalias),
- __ATTR_RO(name),
- __ATTR_RO(id),
- __ATTR_NULL,
+static struct attribute *gio_dev_attrs[] = {
+ &dev_attr_modalias.attr,
+ &dev_attr_name.attr,
+ &dev_attr_id.attr,
+ NULL,
};
+ATTRIBUTE_GROUPS(gio_dev);
static int gio_device_uevent(struct device *dev, struct kobj_uevent_env *env)
{
static struct bus_type gio_bus_type = {
.name = "gio",
- .dev_attrs = gio_dev_attrs,
+ .dev_groups = gio_dev_groups,
.match = gio_bus_match,
.probe = gio_device_probe,
.remove = gio_device_remove,
struct clk *clk;
of_clk_init(NULL);
- clocksource_probe();
+ timer_probe();
np = of_get_cpu_node(0, NULL);
if (!np) {
/* Free all resources held by a thread. */
extern void release_thread(struct task_struct *);
-/*
- * Return saved PC of a blocked thread.
- */
-extern unsigned long thread_saved_pc(struct task_struct *tsk);
-
unsigned long get_wchan(struct task_struct *p);
#define task_pt_regs(task) ((task)->thread.uregs)
# UAPI Header export list
include include/uapi/asm-generic/Kbuild.asm
+
+generic-y += siginfo.h
+++ /dev/null
-#include <asm-generic/siginfo.h>
#include <asm/gdb-stub.h>
#include "internal.h"
-/*
- * return saved PC of a blocked thread.
- */
-unsigned long thread_saved_pc(struct task_struct *tsk)
-{
- return ((unsigned long *) tsk->thread.sp)[3];
-}
-
/*
* power off function, if any
*/
config NIOS2
def_bool y
- select CLKSRC_OF
+ select TIMER_OF
select GENERIC_ATOMIC64
select GENERIC_CLOCKEVENTS
select GENERIC_CPU_DEVICES
generic-y += sembuf.h
generic-y += serial.h
generic-y += shmbuf.h
-generic-y += siginfo.h
generic-y += signal.h
generic-y += socket.h
generic-y += sockios.h
{
}
-/* Return saved PC of a blocked thread. */
-#define thread_saved_pc(tsk) ((tsk)->thread.kregs->ea)
-
extern unsigned long get_wchan(struct task_struct *p);
#define task_pt_regs(p) \
include include/uapi/asm-generic/Kbuild.asm
generic-y += setup.h
+generic-y += siginfo.h
generic-y += ucontext.h
if (count < 2)
panic("%d timer is found, it needs 2 timers in system\n", count);
- clocksource_probe();
+ timer_probe();
}
-CLOCKSOURCE_OF_DECLARE(nios2_timer, ALTR_TIMER_COMPATIBLE, nios2_time_init);
+TIMER_OF_DECLARE(nios2_timer, ALTR_TIMER_COMPATIBLE, nios2_time_init);
generic-y += setup.h
generic-y += shmbuf.h
generic-y += shmparam.h
-generic-y += siginfo.h
generic-y += signal.h
generic-y += socket.h
generic-y += sockios.h
void release_thread(struct task_struct *);
unsigned long get_wchan(struct task_struct *p);
-/*
- * Return saved PC of a blocked thread. For now, this is the "user" PC
- */
-extern unsigned long thread_saved_pc(struct task_struct *t);
-
#define init_stack (init_thread_union.stack)
#define cpu_relax() barrier()
# UAPI Header export list
include include/uapi/asm-generic/Kbuild.asm
+
+generic-y += siginfo.h
show_registers(regs);
}
-unsigned long thread_saved_pc(struct task_struct *t)
-{
- return (unsigned long)user_regs(t->stack)->pc;
-}
-
void release_thread(struct task_struct *dead_task)
{
}
** flush/purge and allocate "regular" cacheable pages for everything.
*/
+#define DMA_ERROR_CODE (~(dma_addr_t)0)
+
#ifdef CONFIG_PA11
extern const struct dma_map_ops pcxl_dma_ops;
extern const struct dma_map_ops pcx_dma_ops;
break;
}
}
- BUG_ON(!dev->platform_data);
return dev->platform_data;
}
-
-#define GET_IOC(dev) (HBA_DATA(parisc_walk_tree(dev))->iommu)
-
+
+#define GET_IOC(dev) ({ \
+ void *__pdata = parisc_walk_tree(dev); \
+ __pdata ? HBA_DATA(__pdata)->iommu : NULL; \
+})
#ifdef CONFIG_IOMMU_CCIO
struct parisc_device;
unsigned char ret;
__asm__ __volatile__(
- " rsm 2,%0\n"
+ " rsm %3,%0\n"
" ldbx 0(%2),%1\n"
" mtsm %0\n"
- : "=&r" (flags), "=r" (ret) : "r" (addr) );
+ : "=&r" (flags), "=r" (ret) : "r" (addr), "i" (PSW_SM_D) );
return ret;
}
unsigned short ret;
__asm__ __volatile__(
- " rsm 2,%0\n"
+ " rsm %3,%0\n"
" ldhx 0(%2),%1\n"
" mtsm %0\n"
- : "=&r" (flags), "=r" (ret) : "r" (addr) );
+ : "=&r" (flags), "=r" (ret) : "r" (addr), "i" (PSW_SM_D) );
return ret;
}
{
long flags;
__asm__ __volatile__(
- " rsm 2,%0\n"
+ " rsm %3,%0\n"
" stbs %1,0(%2)\n"
" mtsm %0\n"
- : "=&r" (flags) : "r" (val), "r" (addr) );
+ : "=&r" (flags) : "r" (val), "r" (addr), "i" (PSW_SM_D) );
}
static inline void gsc_writew(unsigned short val, unsigned long addr)
{
long flags;
__asm__ __volatile__(
- " rsm 2,%0\n"
+ " rsm %3,%0\n"
" sths %1,0(%2)\n"
" mtsm %0\n"
- : "=&r" (flags) : "r" (val), "r" (addr) );
+ : "=&r" (flags) : "r" (val), "r" (addr), "i" (PSW_SM_D) );
}
static inline void gsc_writel(unsigned int val, unsigned long addr)
mtctl(__space_to_prot(context), 8);
}
-static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next, struct task_struct *tsk)
+static inline void switch_mm_irqs_off(struct mm_struct *prev,
+ struct mm_struct *next, struct task_struct *tsk)
{
-
if (prev != next) {
mtctl(__pa(next->pgd), 25);
load_context(next->context);
}
}
+static inline void switch_mm(struct mm_struct *prev,
+ struct mm_struct *next, struct task_struct *tsk)
+{
+ unsigned long flags;
+
+ local_irq_save(flags);
+ switch_mm_irqs_off(prev, next, tsk);
+ local_irq_restore(flags);
+}
+#define switch_mm_irqs_off switch_mm_irqs_off
+
#define deactivate_mm(tsk,mm) do { } while (0)
static inline void activate_mm(struct mm_struct *prev, struct mm_struct *next)
#if !defined(__ASSEMBLY__)
extern int pdc_type;
+extern unsigned long parisc_cell_num; /* cell number the CPU runs on (PAT) */
+extern unsigned long parisc_cell_loc; /* cell location of CPU (PAT) */
/* Values for pdc_type */
#define PDC_TYPE_ILLEGAL -1
#endif /* !CONFIG_PA20 */
+struct pdc_mem_retinfo { /* PDC_MEM/PDC_MEM_MEMINFO (return info) */
+ unsigned long pdt_size;
+ unsigned long pdt_entries;
+ unsigned long pdt_status;
+ unsigned long first_dbe_loc;
+ unsigned long good_mem;
+};
+
+struct pdc_mem_read_pdt { /* PDC_MEM/PDC_MEM_READ_PDT (return info) */
+ unsigned long pdt_entries;
+};
+
#ifdef CONFIG_64BIT
struct pdc_memory_table_raddr { /* PDC_MEM/PDC_MEM_TABLE (return info) */
unsigned long entries_returned;
int pdc_tod_read(struct pdc_tod *tod);
int pdc_tod_set(unsigned long sec, unsigned long usec);
+void pdc_pdt_init(void); /* in pdt.c */
+int pdc_mem_pdt_info(struct pdc_mem_retinfo *rinfo);
+int pdc_mem_pdt_read_entries(struct pdc_mem_read_pdt *rpdt_read,
+ unsigned long *pdt_entries_ptr);
#ifdef CONFIG_64BIT
int pdc_mem_mem_table(struct pdc_memory_table_raddr *r_addr,
struct pdc_memory_table *tbl, unsigned long entries);
#define PDC_PAT_MEM_CELL_CLEAR 6L /* Clear PDT For Cell */
#define PDC_PAT_MEM_CELL_READ 7L /* Read PDT entries For Cell */
#define PDC_PAT_MEM_CELL_RESET 8L /* Reset clear bit For Cell */
-#define PDC_PAT_MEM_SETGM 9L /* Set Golden Memory value */
-#define PDC_PAT_MEM_ADD_PAGE 10L /* ADDs a page to the cell */
-#define PDC_PAT_MEM_ADDRESS 11L /* Get Physical Location From */
+#define PDC_PAT_MEM_SETGM 9L /* Set Good Memory value */
+#define PDC_PAT_MEM_ADD_PAGE 10L /* ADDs a page to the cell */
+#define PDC_PAT_MEM_ADDRESS 11L /* Get Physical Location From */
/* Memory Address */
#define PDC_PAT_MEM_GET_TXT_SIZE 12L /* Get Formatted Text Size */
#define PDC_PAT_MEM_GET_PD_TXT 13L /* Get PD Formatted Text */
unsigned long cpu_loc;
};
+struct pdc_pat_mem_retinfo { /* PDC_PAT_MEM/PDC_PAT_MEM_PD_INFO (return info) */
+ unsigned int ke; /* bit 0: memory inside good memory? */
+ unsigned int current_pdt_entries:16;
+ unsigned int max_pdt_entries:16;
+ unsigned long Cs_bitmap;
+ unsigned long Ic_bitmap;
+ unsigned long good_mem;
+ unsigned long first_dbe_loc; /* first location of double bit error */
+ unsigned long clear_time; /* last PDT clear time (since Jan 1970) */
+};
+
+struct pdc_pat_mem_read_pd_retinfo { /* PDC_PAT_MEM/PDC_PAT_MEM_PD_READ */
+ unsigned long actual_count_bytes;
+ unsigned long pdt_entries;
+};
+
+
struct pdc_pat_pd_addr_map_entry {
unsigned char entry_type; /* 1 = Memory Descriptor Entry Type */
unsigned char reserve1[5];
extern int pdc_pat_pd_get_addr_map(unsigned long *actual_len, void *mem_addr, unsigned long count, unsigned long offset);
-
extern int pdc_pat_io_pci_cfg_read(unsigned long pci_addr, int pci_size, u32 *val);
extern int pdc_pat_io_pci_cfg_write(unsigned long pci_addr, int pci_size, u32 val);
-
-/* Flag to indicate this is a PAT box...don't use this unless you
-** really have to...it might go away some day.
-*/
-extern int pdc_pat; /* arch/parisc/kernel/inventory.c */
+extern int pdc_pat_mem_pdt_info(struct pdc_pat_mem_retinfo *rinfo);
+extern int pdc_pat_mem_read_cell_pdt(struct pdc_pat_mem_read_pd_retinfo *pret,
+ unsigned long *pdt_entries_ptr, unsigned long max_entries);
+extern int pdc_pat_mem_read_pd_pdt(struct pdc_pat_mem_read_pd_retinfo *pret,
+ unsigned long *pdt_entries_ptr, unsigned long count,
+ unsigned long offset);
#endif /* __ASSEMBLY__ */
#define pte_same(A,B) (pte_val(A) == pte_val(B))
+struct seq_file;
+extern void arch_report_meminfo(struct seq_file *m);
+
#endif /* !__ASSEMBLY__ */
unsigned long bh_count; /* number of times bh was invoked */
unsigned long fp_rev;
unsigned long fp_model;
+ unsigned long cpu_num; /* CPU number from PAT firmware */
+ unsigned long cpu_loc; /* CPU location from PAT firmware */
unsigned int state;
struct parisc_device *dev;
unsigned long loops_per_jiffy;
.flags = 0 \
}
-/*
- * Return saved PC of a blocked thread. This is used by ps mostly.
- */
-
struct task_struct;
-unsigned long thread_saved_pc(struct task_struct *t);
void show_trace(struct task_struct *task, unsigned long *stack);
/*
#define ASM_EXCEPTIONTABLE_ENTRY_EFAULT( fault_addr, except_addr )\
ASM_EXCEPTIONTABLE_ENTRY( fault_addr, except_addr + 1)
-/*
- * The page fault handler stores, in a per-cpu area, the following information
- * if a fixup routine is available.
- */
-struct exception_data {
- unsigned long fault_ip;
- unsigned long fault_gp;
- unsigned long fault_space;
- unsigned long fault_addr;
-};
-
/*
* load_sr2() preloads the space register %%sr2 - based on the value of
* get_fs() - with either a value of 0 to access kernel space (KERNEL_DS which
#define TIOCGPKT _IOR('T', 0x38, int) /* Get packet mode state */
#define TIOCGPTLCK _IOR('T', 0x39, int) /* Get Pty lock state */
#define TIOCGEXCL _IOR('T', 0x40, int) /* Get exclusive mode state */
+#define TIOCGPTPEER _IOR('T', 0x41, int) /* Safely open the slave */
#define FIONCLEX 0x5450 /* these numbers need to be adjusted. */
#define FIOCLEX 0x5451
#define PDC_TLB_SETUP 1 /* set up miss handling */
#define PDC_MEM 20 /* Manage memory */
-#define PDC_MEM_MEMINFO 0
-#define PDC_MEM_ADD_PAGE 1
-#define PDC_MEM_CLEAR_PDT 2
-#define PDC_MEM_READ_PDT 3
-#define PDC_MEM_RESET_CLEAR 4
-#define PDC_MEM_GOODMEM 5
+#define PDC_MEM_MEMINFO 0 /* Return PDT info */
+#define PDC_MEM_ADD_PAGE 1 /* Add page to PDT */
+#define PDC_MEM_CLEAR_PDT 2 /* Clear PDT */
+#define PDC_MEM_READ_PDT 3 /* Read PDT entry */
+#define PDC_MEM_RESET_CLEAR 4 /* Reset PDT clear flag */
+#define PDC_MEM_GOODMEM 5 /* Set good_mem value */
#define PDC_MEM_TABLE 128 /* Non contig mem map (sprockets) */
#define PDC_MEM_RETURN_ADDRESS_TABLE PDC_MEM_TABLE
#define PDC_MEM_GET_MEMORY_SYSTEM_TABLES_SIZE 131
extra-y := head.o vmlinux.lds
-obj-y := cache.o pacache.o setup.o traps.o time.o irq.o \
+obj-y := cache.o pacache.o setup.o pdt.o traps.o time.o irq.o \
pa7300lc.o syscall.o entry.o sys_parisc.o firmware.o \
ptrace.o hardware.o inventory.o drivers.o \
signal.o hpmc.o real2.o parisc_ksyms.o unaligned.o \
#else
DEFINE(HUGEPAGE_SIZE, PAGE_SIZE);
#endif
- BLANK();
- DEFINE(EXCDATA_IP, offsetof(struct exception_data, fault_ip));
- DEFINE(EXCDATA_GP, offsetof(struct exception_data, fault_gp));
- DEFINE(EXCDATA_SPACE, offsetof(struct exception_data, fault_space));
- DEFINE(EXCDATA_ADDR, offsetof(struct exception_data, fault_addr));
BLANK();
DEFINE(ASM_PDC_RESULT_SIZE, NUM_PDC_RESULT * sizeof(unsigned long));
BLANK();
{ \
struct parisc_device *padev = to_parisc_device(dev); \
return sprintf(buf, format_string, padev->field); \
-}
+} \
+static DEVICE_ATTR_RO(name);
#define pa_dev_attr_id(field, format) pa_dev_attr(field, id.field, format)
{
return make_modalias(dev, buf);
}
+static DEVICE_ATTR_RO(modalias);
-static struct device_attribute parisc_device_attrs[] = {
- __ATTR_RO(irq),
- __ATTR_RO(hw_type),
- __ATTR_RO(rev),
- __ATTR_RO(hversion),
- __ATTR_RO(sversion),
- __ATTR_RO(modalias),
- __ATTR_NULL,
+static struct attribute *parisc_device_attrs[] = {
+ &dev_attr_irq.attr,
+ &dev_attr_hw_type.attr,
+ &dev_attr_rev.attr,
+ &dev_attr_hversion.attr,
+ &dev_attr_sversion.attr,
+ &dev_attr_modalias.attr,
+ NULL,
};
+ATTRIBUTE_GROUPS(parisc_device);
struct bus_type parisc_bus_type = {
.name = "parisc",
.match = parisc_generic_match,
.uevent = parisc_uevent,
- .dev_attrs = parisc_device_attrs,
+ .dev_groups = parisc_device_groups,
.probe = parisc_driver_probe,
.remove = parisc_driver_remove,
};
}
EXPORT_SYMBOL(pdc_tod_read);
+int pdc_mem_pdt_info(struct pdc_mem_retinfo *rinfo)
+{
+ int retval;
+ unsigned long flags;
+
+ spin_lock_irqsave(&pdc_lock, flags);
+ retval = mem_pdc_call(PDC_MEM, PDC_MEM_MEMINFO, __pa(pdc_result), 0);
+ convert_to_wide(pdc_result);
+ memcpy(rinfo, pdc_result, sizeof(*rinfo));
+ spin_unlock_irqrestore(&pdc_lock, flags);
+
+ return retval;
+}
+
+int pdc_mem_pdt_read_entries(struct pdc_mem_read_pdt *pret,
+ unsigned long *pdt_entries_ptr)
+{
+ int retval;
+ unsigned long flags;
+
+ spin_lock_irqsave(&pdc_lock, flags);
+ retval = mem_pdc_call(PDC_MEM, PDC_MEM_READ_PDT, __pa(pdc_result),
+ __pa(pdc_result2));
+ if (retval == PDC_OK) {
+ convert_to_wide(pdc_result);
+ memcpy(pret, pdc_result, sizeof(*pret));
+ convert_to_wide(pdc_result2);
+ memcpy(pdt_entries_ptr, pdc_result2,
+ pret->pdt_entries * sizeof(*pdt_entries_ptr));
+ }
+ spin_unlock_irqrestore(&pdc_lock, flags);
+
+ return retval;
+}
+
/**
* pdc_tod_set - Set the Time-Of-Day clock.
* @sec: The number of seconds since epoch.
return retval;
}
+
+/**
+ * pdc_pat_mem_pdc_info - Retrieve information about page deallocation table
+ * @rinfo: memory pdt information
+ *
+ */
+int pdc_pat_mem_pdt_info(struct pdc_pat_mem_retinfo *rinfo)
+{
+ int retval;
+ unsigned long flags;
+
+ spin_lock_irqsave(&pdc_lock, flags);
+ retval = mem_pdc_call(PDC_PAT_MEM, PDC_PAT_MEM_PD_INFO,
+ __pa(&pdc_result));
+ if (retval == PDC_OK)
+ memcpy(rinfo, &pdc_result, sizeof(*rinfo));
+ spin_unlock_irqrestore(&pdc_lock, flags);
+
+ return retval;
+}
+
+/**
+ * pdc_pat_mem_read_cell_pdt - Read PDT entries from (old) PAT firmware
+ * @pret: array of PDT entries
+ * @pdt_entries_ptr: ptr to hold number of PDT entries
+ * @max_entries: maximum number of entries to be read
+ *
+ */
+int pdc_pat_mem_read_cell_pdt(struct pdc_pat_mem_read_pd_retinfo *pret,
+ unsigned long *pdt_entries_ptr, unsigned long max_entries)
+{
+ int retval;
+ unsigned long flags, entries;
+
+ spin_lock_irqsave(&pdc_lock, flags);
+ /* PDC_PAT_MEM_CELL_READ is available on early PAT machines only */
+ retval = mem_pdc_call(PDC_PAT_MEM, PDC_PAT_MEM_CELL_READ,
+ __pa(&pdc_result), parisc_cell_num, __pa(&pdc_result2));
+
+ if (retval == PDC_OK) {
+ /* build up return value as for PDC_PAT_MEM_PD_READ */
+ entries = min(pdc_result[0], max_entries);
+ pret->pdt_entries = entries;
+ pret->actual_count_bytes = entries * sizeof(unsigned long);
+ memcpy(pdt_entries_ptr, &pdc_result2, pret->actual_count_bytes);
+ }
+
+ spin_unlock_irqrestore(&pdc_lock, flags);
+ WARN_ON(retval == PDC_OK && pdc_result[0] > max_entries);
+
+ return retval;
+}
+/**
+ * pdc_pat_mem_read_pd_pdt - Read PDT entries from (newer) PAT firmware
+ * @pret: array of PDT entries
+ * @pdt_entries_ptr: ptr to hold number of PDT entries
+ *
+ */
+int pdc_pat_mem_read_pd_pdt(struct pdc_pat_mem_read_pd_retinfo *pret,
+ unsigned long *pdt_entries_ptr, unsigned long count,
+ unsigned long offset)
+{
+ int retval;
+ unsigned long flags;
+
+ spin_lock_irqsave(&pdc_lock, flags);
+ retval = mem_pdc_call(PDC_PAT_MEM, PDC_PAT_MEM_PD_READ,
+ __pa(&pret), __pa(pdt_entries_ptr),
+ count, offset);
+ spin_unlock_irqrestore(&pdc_lock, flags);
+
+ return retval;
+}
#endif /* CONFIG_64BIT */
#include <asm/assembly.h>
#include <asm/pdc.h>
+#include <asm/psw.h>
#include <linux/linkage.h>
#include <linux/init.h>
* So turn on the Q bit and turn off the M bit.
*/
- ldo 8(%r0),%r4 /* PSW Q on, PSW M off */
+ ldi PSW_SM_Q,%r4 /* PSW Q on, PSW M off */
mtctl %r4,ipsw
mtctl %r0,pcsq
mtctl %r0,pcsq
tovirt_r1 %r30 /* make sp virtual */
- rsm 8,%r0 /* Clear Q bit */
+ rsm PSW_SM_Q,%r0 /* Clear Q bit */
ldi 1,%r8 /* Set trap code to "1" for HPMC */
load32 PA(intr_save),%r1
be 0(%sr7,%r1)
int pdc_type __read_mostly = PDC_TYPE_ILLEGAL;
+/* cell number and location (PAT firmware only) */
+unsigned long parisc_cell_num __read_mostly;
+unsigned long parisc_cell_loc __read_mostly;
+
+
void __init setup_pdc(void)
{
long status;
if (status == PDC_OK) {
pdc_type = PDC_TYPE_PAT;
pr_cont("64 bit PAT.\n");
+ parisc_cell_num = cell_info.cell_num;
+ parisc_cell_loc = cell_info.cell_loc;
+ pr_info("PAT: Running on cell %lu and location %lu.\n",
+ parisc_cell_num, parisc_cell_loc);
return;
}
#endif
--- /dev/null
+/*
+ * Page Deallocation Table (PDT) support
+ *
+ * The Page Deallocation Table (PDT) holds a table with pointers to bad
+ * memory (broken RAM modules) which is maintained by firmware.
+ *
+ * Copyright 2017 by Helge Deller <deller@gmx.de>
+ *
+ * TODO:
+ * - check regularily for new bad memory
+ * - add userspace interface with procfs or sysfs
+ * - increase number of PDT entries dynamically
+ */
+
+#include <linux/memblock.h>
+#include <linux/seq_file.h>
+
+#include <asm/pdc.h>
+#include <asm/pdcpat.h>
+#include <asm/sections.h>
+#include <asm/pgtable.h>
+
+enum pdt_access_type {
+ PDT_NONE,
+ PDT_PDC,
+ PDT_PAT_NEW,
+ PDT_PAT_OLD
+};
+
+static enum pdt_access_type pdt_type;
+
+/* global PDT status information */
+static struct pdc_mem_retinfo pdt_status;
+
+#define MAX_PDT_TABLE_SIZE PAGE_SIZE
+#define MAX_PDT_ENTRIES (MAX_PDT_TABLE_SIZE / sizeof(unsigned long))
+static unsigned long pdt_entry[MAX_PDT_ENTRIES] __page_aligned_bss;
+
+
+/* report PDT entries via /proc/meminfo */
+void arch_report_meminfo(struct seq_file *m)
+{
+ if (pdt_type == PDT_NONE)
+ return;
+
+ seq_printf(m, "PDT_max_entries: %7lu\n",
+ pdt_status.pdt_size);
+ seq_printf(m, "PDT_cur_entries: %7lu\n",
+ pdt_status.pdt_entries);
+}
+
+/*
+ * pdc_pdt_init()
+ *
+ * Initialize kernel PDT structures, read initial PDT table from firmware,
+ * report all current PDT entries and mark bad memory with memblock_reserve()
+ * to avoid that the kernel will use broken memory areas.
+ *
+ */
+void __init pdc_pdt_init(void)
+{
+ int ret, i;
+ unsigned long entries;
+ struct pdc_mem_read_pdt pdt_read_ret;
+
+ if (is_pdc_pat()) {
+ struct pdc_pat_mem_retinfo pat_rinfo;
+
+ pdt_type = PDT_PAT_NEW;
+ ret = pdc_pat_mem_pdt_info(&pat_rinfo);
+ pdt_status.pdt_size = pat_rinfo.max_pdt_entries;
+ pdt_status.pdt_entries = pat_rinfo.current_pdt_entries;
+ pdt_status.pdt_status = 0;
+ pdt_status.first_dbe_loc = pat_rinfo.first_dbe_loc;
+ pdt_status.good_mem = pat_rinfo.good_mem;
+ } else {
+ pdt_type = PDT_PDC;
+ ret = pdc_mem_pdt_info(&pdt_status);
+ }
+
+ if (ret != PDC_OK) {
+ pdt_type = PDT_NONE;
+ pr_info("PDT: Firmware does not provide any page deallocation"
+ " information.\n");
+ return;
+ }
+
+ entries = pdt_status.pdt_entries;
+ WARN_ON(entries > MAX_PDT_ENTRIES);
+
+ pr_info("PDT: size %lu, entries %lu, status %lu, dbe_loc 0x%lx,"
+ " good_mem %lu\n",
+ pdt_status.pdt_size, pdt_status.pdt_entries,
+ pdt_status.pdt_status, pdt_status.first_dbe_loc,
+ pdt_status.good_mem);
+
+ if (entries == 0) {
+ pr_info("PDT: Firmware reports all memory OK.\n");
+ return;
+ }
+
+ if (pdt_status.first_dbe_loc &&
+ pdt_status.first_dbe_loc <= __pa((unsigned long)&_end))
+ pr_crit("CRITICAL: Bad memory inside kernel image memory area!\n");
+
+ pr_warn("PDT: Firmware reports %lu entries of faulty memory:\n",
+ entries);
+
+ if (pdt_type == PDT_PDC)
+ ret = pdc_mem_pdt_read_entries(&pdt_read_ret, pdt_entry);
+ else {
+#ifdef CONFIG_64BIT
+ struct pdc_pat_mem_read_pd_retinfo pat_pret;
+
+ ret = pdc_pat_mem_read_cell_pdt(&pat_pret, pdt_entry,
+ MAX_PDT_ENTRIES);
+ if (ret != PDC_OK) {
+ pdt_type = PDT_PAT_OLD;
+ ret = pdc_pat_mem_read_pd_pdt(&pat_pret, pdt_entry,
+ MAX_PDT_TABLE_SIZE, 0);
+ }
+#else
+ ret = PDC_BAD_PROC;
+#endif
+ }
+
+ if (ret != PDC_OK) {
+ pdt_type = PDT_NONE;
+ pr_debug("PDT type %d, retval = %d\n", pdt_type, ret);
+ return;
+ }
+
+ for (i = 0; i < pdt_status.pdt_entries; i++) {
+ if (i < 20)
+ pr_warn("PDT: BAD PAGE #%d at 0x%08lx (error_type = %lu)\n",
+ i,
+ pdt_entry[i] & PAGE_MASK,
+ pdt_entry[i] & 1);
+
+ /* mark memory page bad */
+ memblock_reserve(pdt_entry[i] & PAGE_MASK, PAGE_SIZE);
+ }
+}
return 0;
}
-unsigned long thread_saved_pc(struct task_struct *t)
-{
- return t->thread.regs.kpc;
-}
-
unsigned long
get_wchan(struct task_struct *p)
{
unsigned long txn_addr;
unsigned long cpuid;
struct cpuinfo_parisc *p;
- struct pdc_pat_cpu_num cpu_info __maybe_unused;
+ struct pdc_pat_cpu_num cpu_info = { };
#ifdef CONFIG_SMP
if (num_online_cpus() >= nr_cpu_ids) {
*/
cpuid = boot_cpu_data.cpu_count;
txn_addr = dev->hpa.start; /* for legacy PDC */
+ cpu_info.cpu_num = cpu_info.cpu_loc = cpuid;
#ifdef CONFIG_64BIT
if (is_pdc_pat()) {
p->hpa = dev->hpa.start; /* save CPU hpa */
p->cpuid = cpuid; /* save CPU id */
p->txn_addr = txn_addr; /* save CPU IRQ address */
+ p->cpu_num = cpu_info.cpu_num;
+ p->cpu_loc = cpu_info.cpu_loc;
#ifdef CONFIG_SMP
/*
** FIXME: review if any other initialization is clobbered
ENTRY_SAME(ni_syscall) /* 263: reserved for vserver */
ENTRY_SAME(add_key)
ENTRY_SAME(request_key) /* 265 */
- ENTRY_SAME(keyctl)
+ ENTRY_COMP(keyctl)
ENTRY_SAME(ioprio_set)
ENTRY_SAME(ioprio_get)
ENTRY_SAME(inotify_init)
static int __init init_cr16_clocksource(void)
{
/*
- * The cr16 interval timers are not syncronized across CPUs, so mark
- * them unstable and lower rating on SMP systems.
+ * The cr16 interval timers are not syncronized across CPUs on
+ * different sockets, so mark them unstable and lower rating on
+ * multi-socket SMP systems.
*/
if (num_online_cpus() > 1) {
- clocksource_cr16.flags = CLOCK_SOURCE_UNSTABLE;
- clocksource_cr16.rating = 0;
+ int cpu;
+ unsigned long cpu0_loc;
+ cpu0_loc = per_cpu(cpu_data, 0).cpu_loc;
+
+ for_each_online_cpu(cpu) {
+ if (cpu0_loc == per_cpu(cpu_data, cpu).cpu_loc)
+ continue;
+
+ clocksource_cr16.name = "cr16_unstable";
+ clocksource_cr16.flags = CLOCK_SOURCE_UNSTABLE;
+ clocksource_cr16.rating = 0;
+ break;
+ }
}
+ /* XXX: We may want to mark sched_clock stable here if cr16 clocks are
+ * in sync:
+ * (clocksource_cr16.flags == CLOCK_SOURCE_IS_CONTINUOUS) */
+
/* register at clocksource framework */
clocksource_register_hz(&clocksource_cr16,
100 * PAGE0->mem_10msec);
mtsp %r1,%sr1
.endm
- .macro fixup_branch lbl
- ldil L%\lbl, %r1
- ldo R%\lbl(%r1), %r1
- bv %r0(%r1)
- .endm
-
/*
* unsigned long lclear_user(void *to, unsigned long n)
*
$lclu_done:
bv %r0(%r2)
copy %r25,%r28
- .exit
-ENDPROC_CFI(lclear_user)
- .section .fixup,"ax"
-2: fixup_branch $lclu_done
- ldo 1(%r25),%r25
- .previous
+2: b $lclu_done
+ ldo 1(%r25),%r25
ASM_EXCEPTIONTABLE_ENTRY(1b,2b)
+ .exit
+ENDPROC_CFI(lclear_user)
+
+
.procend
/*
$lslen_nzero:
b $lslen_done
ldo 1(%r26),%r26 /* special case for N == 0 */
-ENDPROC_CFI(lstrnlen_user)
- .section .fixup,"ax"
-3: fixup_branch $lslen_done
+3: b $lslen_done
copy %r24,%r26 /* reset r26 so 0 is returned on fault */
- .previous
ASM_EXCEPTIONTABLE_ENTRY(1b,3b)
ASM_EXCEPTIONTABLE_ENTRY(2b,3b)
+ENDPROC_CFI(lstrnlen_user)
+
.procend
#define BITSSET 0x1c0 /* for identifying LDCW */
-DEFINE_PER_CPU(struct exception_data, exception_data);
-
int show_unhandled_signals = 1;
/*
fix = search_exception_tables(regs->iaoq[0]);
if (fix) {
- struct exception_data *d;
- d = this_cpu_ptr(&exception_data);
- d->fault_ip = regs->iaoq[0];
- d->fault_gp = regs->gr[27];
- d->fault_space = regs->isr;
- d->fault_addr = regs->ior;
-
/*
* Fix up get_user() and put_user().
* ASM_EXCEPTIONTABLE_ENTRY_EFAULT() sets the least-significant
/* zero target register for get_user() */
if (parisc_acctyp(0, regs->iir) == VM_READ) {
int treg = regs->iir & 0x1f;
+ BUG_ON(treg == 0);
regs->gr[treg] = 0;
}
}
case 15: /* Data TLB miss fault/Data page fault */
/* send SIGSEGV when outside of vma */
if (!vma ||
- address < vma->vm_start || address > vma->vm_end) {
+ address < vma->vm_start || address >= vma->vm_end) {
si.si_signo = SIGSEGV;
si.si_code = SEGV_MAPERR;
break;
request_resource(res, &data_resource);
}
request_resource(&sysram_resources[0], &pdcdata_resource);
+
+ /* Initialize Page Deallocation Table (PDT) and check for bad memory. */
+ pdc_pdt_init();
}
static int __init parisc_text_address(unsigned long vaddr)
select HAVE_FUNCTION_GRAPH_TRACER
select HAVE_FUNCTION_TRACER
select HAVE_GCC_PLUGINS
- select HAVE_GENERIC_RCU_GUP
+ select HAVE_GENERIC_GUP
select HAVE_HW_BREAKPOINT if PERF_EVENTS && (PPC_BOOK3S || PPC_8xx)
select HAVE_IDE
select HAVE_IOREMAP_PROT
extern int kprobe_fault_handler(struct pt_regs *regs, int trapnr);
extern int kprobe_handler(struct pt_regs *regs);
extern int kprobe_post_handler(struct pt_regs *regs);
+extern int is_current_kprobe_addr(unsigned long addr);
#ifdef CONFIG_KPROBES_ON_FTRACE
extern int skip_singlestep(struct kprobe *p, struct pt_regs *regs,
struct kprobe_ctlblk *kcb);
}
#endif
-/*
- * Return saved PC of a blocked thread. For now, this is the "user" PC
- */
-#define thread_saved_pc(tsk) \
- ((tsk)->thread.regs? (tsk)->thread.regs->nip: 0)
-
#define task_pt_regs(tsk) ((struct pt_regs *)(tsk)->thread.regs)
unsigned long get_wchan(struct task_struct *p);
extern int sysfs_add_device_to_node(struct device *dev, int nid);
extern void sysfs_remove_device_from_node(struct device *dev, int nid);
+extern int numa_update_cpu_topology(bool cpus_locked);
static inline int early_cpu_to_node(int cpu)
{
int nid)
{
}
+
+static inline int numa_update_cpu_topology(bool cpus_locked)
+{
+ return 0;
+}
#endif /* CONFIG_NUMA */
#if defined(CONFIG_NUMA) && defined(CONFIG_PPC_SPLPAR)
extern unsigned long __copy_tofrom_user(void __user *to,
const void __user *from, unsigned long size);
-#ifndef __powerpc64__
-
-#define INLINE_COPY_FROM_USER
-#define INLINE_COPY_TO_USER
-
-#else /* __powerpc64__ */
-
+#ifdef __powerpc64__
static inline unsigned long
raw_copy_in_user(void __user *to, const void __user *from, unsigned long n)
{
#define TIOCGPKT _IOR('T', 0x38, int) /* Get packet mode state */
#define TIOCGPTLCK _IOR('T', 0x39, int) /* Get Pty lock state */
#define TIOCGEXCL _IOR('T', 0x40, int) /* Get exclusive mode state */
+#define TIOCGPTPEER _IOR('T', 0x41, int) /* Safely open the slave */
#define TIOCSERCONFIG 0x5453
#define TIOCSERGWILD 0x5454
.balign IFETCH_ALIGN_BYTES
do_hash_page:
#ifdef CONFIG_PPC_STD_MMU_64
- andis. r0,r4,0xa410 /* weird error? */
+ andis. r0,r4,0xa450 /* weird error? */
bne- handle_page_fault /* if not, try to insert a HPTE */
- andis. r0,r4,DSISR_DABRMATCH@h
- bne- handle_dabr_fault
CURRENT_THREAD_INFO(r11, r1)
lwz r0,TI_PREEMPT(r11) /* If we're in an "NMI" */
andis. r0,r0,NMI_MASK@h /* (i.e. an irq when soft-disabled) */
/* Error */
blt- 13f
+
+ /* Reload DSISR into r4 for the DABR check below */
+ ld r4,_DSISR(r1)
#endif /* CONFIG_PPC_STD_MMU_64 */
/* Here we have a page fault that hash_page can't handle. */
handle_page_fault:
-11: ld r4,_DAR(r1)
+11: andis. r0,r4,DSISR_DABRMATCH@h
+ bne- handle_dabr_fault
+ ld r4,_DAR(r1)
ld r5,_DSISR(r1)
addi r3,r1,STACK_FRAME_OVERHEAD
bl do_page_fault
struct kretprobe_blackpoint kretprobe_blacklist[] = {{NULL, NULL}};
+int is_current_kprobe_addr(unsigned long addr)
+{
+ struct kprobe *p = kprobe_running();
+ return (p && (unsigned long)p->addr == addr) ? 1 : 0;
+}
+
bool arch_within_kprobe_blacklist(unsigned long addr)
{
return (addr >= (unsigned long)__kprobes_text_start &&
regs->gpr[2] = (unsigned long)(((func_descr_t *)jp->entry)->toc);
#endif
+ /*
+ * jprobes use jprobe_return() which skips the normal return
+ * path of the function, and this messes up the accounting of the
+ * function graph tracer.
+ *
+ * Pause function graph tracing while performing the jprobe function.
+ */
+ pause_graph_tracing();
+
return 1;
}
NOKPROBE_SYMBOL(setjmp_pre_handler);
* saved regs...
*/
memcpy(regs, &kcb->jprobe_saved_regs, sizeof(struct pt_regs));
+ /* It's OK to start function graph tracing again */
+ unpause_graph_tracing();
preempt_enable_no_resched();
return 1;
}
* the RTAS event.
*/
pseries_devicetree_update(-prrn_update_scope);
- arch_update_cpu_topology();
+ numa_update_cpu_topology(false);
}
static DECLARE_WORK(prrn_work, prrn_work_fn);
}
#endif
+/*
+ * Emergency stacks are used for a range of things, from asynchronous
+ * NMIs (system reset, machine check) to synchronous, process context.
+ * We set preempt_count to zero, even though that isn't necessarily correct. To
+ * get the right value we'd need to copy it from the previous thread_info, but
+ * doing that might fault causing more problems.
+ * TODO: what to do with accounting?
+ */
+static void emerg_stack_init_thread_info(struct thread_info *ti, int cpu)
+{
+ ti->task = NULL;
+ ti->cpu = cpu;
+ ti->preempt_count = 0;
+ ti->local_flags = 0;
+ ti->flags = 0;
+ klp_init_thread_info(ti);
+}
+
/*
* Stack space used when we detect a bad kernel stack pointer, and
* early in SMP boots before relocation is enabled. Exclusive emergency
* Since we use these as temporary stacks during secondary CPU
* bringup, we need to get at them in real mode. This means they
* must also be within the RMO region.
+ *
+ * The IRQ stacks allocated elsewhere in this file are zeroed and
+ * initialized in kernel/irq.c. These are initialized here in order
+ * to have emergency stacks available as early as possible.
*/
limit = min(safe_stack_limit(), ppc64_rma_size);
for_each_possible_cpu(i) {
struct thread_info *ti;
ti = __va(memblock_alloc_base(THREAD_SIZE, THREAD_SIZE, limit));
- klp_init_thread_info(ti);
+ memset(ti, 0, THREAD_SIZE);
+ emerg_stack_init_thread_info(ti, i);
paca[i].emergency_sp = (void *)ti + THREAD_SIZE;
#ifdef CONFIG_PPC_BOOK3S_64
/* emergency stack for NMI exception handling. */
ti = __va(memblock_alloc_base(THREAD_SIZE, THREAD_SIZE, limit));
- klp_init_thread_info(ti);
+ memset(ti, 0, THREAD_SIZE);
+ emerg_stack_init_thread_info(ti, i);
paca[i].nmi_emergency_sp = (void *)ti + THREAD_SIZE;
/* emergency stack for machine check exception handling. */
ti = __va(memblock_alloc_base(THREAD_SIZE, THREAD_SIZE, limit));
- klp_init_thread_info(ti);
+ memset(ti, 0, THREAD_SIZE);
+ emerg_stack_init_thread_info(ti, i);
paca[i].mc_emergency_sp = (void *)ti + THREAD_SIZE;
#endif
}
/* Special case - we inhibit secondary thread startup
* during boot if the user requests it.
*/
- if (system_state == SYSTEM_BOOTING && cpu_has_feature(CPU_FTR_SMT)) {
+ if (system_state < SYSTEM_RUNNING && cpu_has_feature(CPU_FTR_SMT)) {
if (!smt_enabled_at_boot && cpu_thread_in_core(nr) != 0)
return 0;
if (smt_enabled_at_boot
stdu r1,-SWITCH_FRAME_SIZE(r1)
/* Save all gprs to pt_regs */
- SAVE_8GPRS(0,r1)
- SAVE_8GPRS(8,r1)
- SAVE_8GPRS(16,r1)
- SAVE_8GPRS(24,r1)
+ SAVE_GPR(0, r1)
+ SAVE_10GPRS(2, r1)
+ SAVE_10GPRS(12, r1)
+ SAVE_10GPRS(22, r1)
+
+ /* Save previous stack pointer (r1) */
+ addi r8, r1, SWITCH_FRAME_SIZE
+ std r8, GPR1(r1)
/* Load special regs for save below */
mfmsr r8
bl ftrace_stub
nop
- /* Load ctr with the possibly modified NIP */
- ld r3, _NIP(r1)
- mtctr r3
+ /* Load the possibly modified NIP */
+ ld r15, _NIP(r1)
+
#ifdef CONFIG_LIVEPATCH
- cmpd r14,r3 /* has NIP been altered? */
+ cmpd r14, r15 /* has NIP been altered? */
+#endif
+
+#if defined(CONFIG_LIVEPATCH) && defined(CONFIG_KPROBES_ON_FTRACE)
+ /* NIP has not been altered, skip over further checks */
+ beq 1f
+
+ /* Check if there is an active kprobe on us */
+ subi r3, r14, 4
+ bl is_current_kprobe_addr
+ nop
+
+ /*
+ * If r3 == 1, then this is a kprobe/jprobe.
+ * else, this is livepatched function.
+ *
+ * The conditional branch for livepatch_handler below will use the
+ * result of this comparison. For kprobe/jprobe, we just need to branch to
+ * the new NIP, not call livepatch_handler. The branch below is bne, so we
+ * want CR0[EQ] to be true if this is a kprobe/jprobe. Which means we want
+ * CR0[EQ] = (r3 == 1).
+ */
+ cmpdi r3, 1
+1:
#endif
+ /* Load CTR with the possibly modified NIP */
+ mtctr r15
+
/* Restore gprs */
- REST_8GPRS(0,r1)
- REST_8GPRS(8,r1)
- REST_8GPRS(16,r1)
- REST_8GPRS(24,r1)
+ REST_GPR(0,r1)
+ REST_10GPRS(2,r1)
+ REST_10GPRS(12,r1)
+ REST_10GPRS(22,r1)
/* Restore possibly modified LR */
ld r0, _LINK(r1)
addi r1, r1, SWITCH_FRAME_SIZE
#ifdef CONFIG_LIVEPATCH
- /* Based on the cmpd above, if the NIP was altered handle livepatch */
+ /*
+ * Based on the cmpd or cmpdi above, if the NIP was altered and we're
+ * not on a kprobe/jprobe, then handle livepatch.
+ */
bne- livepatch_handler
#endif
r = set_vpa(vcpu, &vcpu->arch.dtl, addr, len);
break;
case KVM_REG_PPC_TB_OFFSET:
+ /*
+ * POWER9 DD1 has an erratum where writing TBU40 causes
+ * the timebase to lose ticks. So we don't let the
+ * timebase offset be changed on P9 DD1. (It is
+ * initialized to zero.)
+ */
+ if (cpu_has_feature(CPU_FTR_POWER9_DD1))
+ break;
/* round up to multiple of 2^24 */
vcpu->arch.vcore->tb_offset =
ALIGN(set_reg_val(id, *val), 1UL << 24);
{
int r;
int srcu_idx;
+ unsigned long ebb_regs[3] = {}; /* shut up GCC */
+ unsigned long user_tar = 0;
+ unsigned int user_vrsave;
if (!vcpu->arch.sane) {
run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
return -EINVAL;
}
+ /*
+ * Don't allow entry with a suspended transaction, because
+ * the guest entry/exit code will lose it.
+ * If the guest has TM enabled, save away their TM-related SPRs
+ * (they will get restored by the TM unavailable interrupt).
+ */
+#ifdef CONFIG_PPC_TRANSACTIONAL_MEM
+ if (cpu_has_feature(CPU_FTR_TM) && current->thread.regs &&
+ (current->thread.regs->msr & MSR_TM)) {
+ if (MSR_TM_ACTIVE(current->thread.regs->msr)) {
+ run->exit_reason = KVM_EXIT_FAIL_ENTRY;
+ run->fail_entry.hardware_entry_failure_reason = 0;
+ return -EINVAL;
+ }
+ current->thread.tm_tfhar = mfspr(SPRN_TFHAR);
+ current->thread.tm_tfiar = mfspr(SPRN_TFIAR);
+ current->thread.tm_texasr = mfspr(SPRN_TEXASR);
+ current->thread.regs->msr &= ~MSR_TM;
+ }
+#endif
+
kvmppc_core_prepare_to_enter(vcpu);
/* No need to go into the guest when all we'll do is come back out */
flush_all_to_thread(current);
+ /* Save userspace EBB and other register values */
+ if (cpu_has_feature(CPU_FTR_ARCH_207S)) {
+ ebb_regs[0] = mfspr(SPRN_EBBHR);
+ ebb_regs[1] = mfspr(SPRN_EBBRR);
+ ebb_regs[2] = mfspr(SPRN_BESCR);
+ user_tar = mfspr(SPRN_TAR);
+ }
+ user_vrsave = mfspr(SPRN_VRSAVE);
+
vcpu->arch.wqp = &vcpu->arch.vcore->wq;
vcpu->arch.pgdir = current->mm->pgd;
vcpu->arch.state = KVMPPC_VCPU_BUSY_IN_HOST;
}
} while (is_kvmppc_resume_guest(r));
+ /* Restore userspace EBB and other register values */
+ if (cpu_has_feature(CPU_FTR_ARCH_207S)) {
+ mtspr(SPRN_EBBHR, ebb_regs[0]);
+ mtspr(SPRN_EBBRR, ebb_regs[1]);
+ mtspr(SPRN_BESCR, ebb_regs[2]);
+ mtspr(SPRN_TAR, user_tar);
+ mtspr(SPRN_FSCR, current->thread.fscr);
+ }
+ mtspr(SPRN_VRSAVE, user_vrsave);
+
out:
vcpu->arch.state = KVMPPC_VCPU_NOTREADY;
atomic_dec(&vcpu->kvm->arch.vcpus_running);
return;
}
- get_online_cpus();
+ cpus_read_lock();
for (cpu = 0; cpu < nr_cpu_ids; cpu += threads_per_core) {
if (!cpu_online(cpu))
l_ops = (unsigned long) ops;
if (cmpxchg64((unsigned long *)&kvmppc_host_rm_ops_hv, 0, l_ops)) {
- put_online_cpus();
+ cpus_read_unlock();
kfree(ops->rm_core);
kfree(ops);
return;
}
- cpuhp_setup_state_nocalls(CPUHP_KVM_PPC_BOOK3S_PREPARE,
- "ppc/kvm_book3s:prepare",
- kvmppc_set_host_core,
- kvmppc_clear_host_core);
- put_online_cpus();
+ cpuhp_setup_state_nocalls_cpuslocked(CPUHP_KVM_PPC_BOOK3S_PREPARE,
+ "ppc/kvm_book3s:prepare",
+ kvmppc_set_host_core,
+ kvmppc_clear_host_core);
+ cpus_read_unlock();
}
void kvmppc_free_host_rm_ops(void)
* Put whatever is in the decrementer into the
* hypervisor decrementer.
*/
+BEGIN_FTR_SECTION
+ ld r5, HSTATE_KVM_VCORE(r13)
+ ld r6, VCORE_KVM(r5)
+ ld r9, KVM_HOST_LPCR(r6)
+ andis. r9, r9, LPCR_LD@h
+END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300)
mfspr r8,SPRN_DEC
mftb r7
- mtspr SPRN_HDEC,r8
+BEGIN_FTR_SECTION
+ /* On POWER9, don't sign-extend if host LPCR[LD] bit is set */
+ bne 32f
+END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300)
extsw r8,r8
+32: mtspr SPRN_HDEC,r8
add r8,r8,r7
std r8,HSTATE_DECEXP(r13)
#include <asm/opal.h>
#include <asm/xive-regs.h>
+/* Sign-extend HDEC if not on POWER9 */
+#define EXTEND_HDEC(reg) \
+BEGIN_FTR_SECTION; \
+ extsw reg, reg; \
+END_FTR_SECTION_IFCLR(CPU_FTR_ARCH_300)
+
#define VCPU_GPRS_TM(reg) (((reg) * ULONG_SIZE) + VCPU_GPR_TM)
/* Values in HSTATE_NAPPING(r13) */
#define NAPPING_CEDE 1
#define NAPPING_NOVCPU 2
+/* Stack frame offsets for kvmppc_hv_entry */
+#define SFS 144
+#define STACK_SLOT_TRAP (SFS-4)
+#define STACK_SLOT_TID (SFS-16)
+#define STACK_SLOT_PSSCR (SFS-24)
+#define STACK_SLOT_PID (SFS-32)
+#define STACK_SLOT_IAMR (SFS-40)
+#define STACK_SLOT_CIABR (SFS-48)
+#define STACK_SLOT_DAWR (SFS-56)
+#define STACK_SLOT_DAWRX (SFS-64)
+
/*
* Call kvmppc_hv_entry in real mode.
* Must be called with interrupts hard-disabled.
kvmppc_primary_no_guest:
/* We handle this much like a ceded vcpu */
/* put the HDEC into the DEC, since HDEC interrupts don't wake us */
+ /* HDEC may be larger than DEC for arch >= v3.00, but since the */
+ /* HDEC value came from DEC in the first place, it will fit */
mfspr r3, SPRN_HDEC
mtspr SPRN_DEC, r3
/*
/* See if our timeslice has expired (HDEC is negative) */
mfspr r0, SPRN_HDEC
+ EXTEND_HDEC(r0)
li r12, BOOK3S_INTERRUPT_HV_DECREMENTER
- cmpwi r0, 0
+ cmpdi r0, 0
blt kvm_novcpu_exit
/* Got an IPI but other vcpus aren't yet exiting, must be a latecomer */
bl kvmhv_accumulate_time
#endif
13: mr r3, r12
- stw r12, 112-4(r1)
+ stw r12, STACK_SLOT_TRAP(r1)
bl kvmhv_commence_exit
nop
- lwz r12, 112-4(r1)
+ lwz r12, STACK_SLOT_TRAP(r1)
b kvmhv_switch_to_host
/*
lbz r4, HSTATE_PTID(r13)
cmpwi r4, 0
bne 63f
- lis r6, 0x7fff
- ori r6, r6, 0xffff
+ LOAD_REG_ADDR(r6, decrementer_max)
+ ld r6, 0(r6)
mtspr SPRN_HDEC, r6
/* and set per-LPAR registers, if doing dynamic micro-threading */
ld r6, HSTATE_SPLIT_MODE(r13)
* *
*****************************************************************************/
-/* Stack frame offsets */
-#define STACK_SLOT_TID (112-16)
-#define STACK_SLOT_PSSCR (112-24)
-#define STACK_SLOT_PID (112-32)
-
.global kvmppc_hv_entry
kvmppc_hv_entry:
*/
mflr r0
std r0, PPC_LR_STKOFF(r1)
- stdu r1, -112(r1)
+ stdu r1, -SFS(r1)
/* Save R1 in the PACA */
std r1, HSTATE_HOST_R1(r13)
mfspr r5, SPRN_TIDR
mfspr r6, SPRN_PSSCR
mfspr r7, SPRN_PID
+ mfspr r8, SPRN_IAMR
std r5, STACK_SLOT_TID(r1)
std r6, STACK_SLOT_PSSCR(r1)
std r7, STACK_SLOT_PID(r1)
+ std r8, STACK_SLOT_IAMR(r1)
END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300)
+BEGIN_FTR_SECTION
+ mfspr r5, SPRN_CIABR
+ mfspr r6, SPRN_DAWR
+ mfspr r7, SPRN_DAWRX
+ std r5, STACK_SLOT_CIABR(r1)
+ std r6, STACK_SLOT_DAWR(r1)
+ std r7, STACK_SLOT_DAWRX(r1)
+END_FTR_SECTION_IFSET(CPU_FTR_ARCH_207S)
BEGIN_FTR_SECTION
/* Set partition DABR */
/* Check if HDEC expires soon */
mfspr r3, SPRN_HDEC
- cmpwi r3, 512 /* 1 microsecond */
+ EXTEND_HDEC(r3)
+ cmpdi r3, 512 /* 1 microsecond */
blt hdec_soon
#ifdef CONFIG_KVM_XICS
* set by the guest could disrupt the host.
*/
li r0, 0
- mtspr SPRN_IAMR, r0
- mtspr SPRN_CIABR, r0
- mtspr SPRN_DAWRX, r0
+ mtspr SPRN_PSPB, r0
mtspr SPRN_WORT, r0
BEGIN_FTR_SECTION
+ mtspr SPRN_IAMR, r0
mtspr SPRN_TCSCR, r0
/* Set MMCRS to 1<<31 to freeze and disable the SPMC counters */
li r0, 1
std r6,VCPU_UAMOR(r9)
li r6,0
mtspr SPRN_AMR,r6
+ mtspr SPRN_UAMOR, r6
/* Switch DSCR back to host value */
mfspr r8, SPRN_DSCR
ptesync
/* Restore host values of some registers */
+BEGIN_FTR_SECTION
+ ld r5, STACK_SLOT_CIABR(r1)
+ ld r6, STACK_SLOT_DAWR(r1)
+ ld r7, STACK_SLOT_DAWRX(r1)
+ mtspr SPRN_CIABR, r5
+ mtspr SPRN_DAWR, r6
+ mtspr SPRN_DAWRX, r7
+END_FTR_SECTION_IFSET(CPU_FTR_ARCH_207S)
BEGIN_FTR_SECTION
ld r5, STACK_SLOT_TID(r1)
ld r6, STACK_SLOT_PSSCR(r1)
ld r7, STACK_SLOT_PID(r1)
+ ld r8, STACK_SLOT_IAMR(r1)
mtspr SPRN_TIDR, r5
mtspr SPRN_PSSCR, r6
mtspr SPRN_PID, r7
+ mtspr SPRN_IAMR, r8
END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300)
BEGIN_FTR_SECTION
PPC_INVALIDATE_ERAT
li r0, KVM_GUEST_MODE_NONE
stb r0, HSTATE_IN_GUEST(r13)
- ld r0, 112+PPC_LR_STKOFF(r1)
- addi r1, r1, 112
+ ld r0, SFS+PPC_LR_STKOFF(r1)
+ addi r1, r1, SFS
mtlr r0
blr
mfspr r3, SPRN_DEC
mfspr r4, SPRN_HDEC
mftb r5
- cmpw r3, r4
+ extsw r3, r3
+ EXTEND_HDEC(r4)
+ cmpd r3, r4
ble 67f
mtspr SPRN_DEC, r4
67:
/* save expiry time of guest decrementer */
- extsw r3, r3
add r3, r3, r5
ld r4, HSTATE_KVM_VCPU(r13)
ld r5, HSTATE_KVM_VCORE(r13)
/*
* Update the node maps and sysfs entries for each cpu whose home node
* has changed. Returns 1 when the topology has changed, and 0 otherwise.
+ *
+ * cpus_locked says whether we already hold cpu_hotplug_lock.
*/
-int arch_update_cpu_topology(void)
+int numa_update_cpu_topology(bool cpus_locked)
{
unsigned int cpu, sibling, changed = 0;
struct topology_update_data *updates, *ud;
if (!cpumask_weight(&updated_cpus))
goto out;
- stop_machine(update_cpu_topology, &updates[0], &updated_cpus);
+ if (cpus_locked)
+ stop_machine_cpuslocked(update_cpu_topology, &updates[0],
+ &updated_cpus);
+ else
+ stop_machine(update_cpu_topology, &updates[0], &updated_cpus);
/*
* Update the numa-cpu lookup table with the new mappings, even for
* offline CPUs. It is best to perform this update from the stop-
* machine context.
*/
- stop_machine(update_lookup_table, &updates[0],
+ if (cpus_locked)
+ stop_machine_cpuslocked(update_lookup_table, &updates[0],
cpumask_of(raw_smp_processor_id()));
+ else
+ stop_machine(update_lookup_table, &updates[0],
+ cpumask_of(raw_smp_processor_id()));
for (ud = &updates[0]; ud; ud = ud->next) {
unregister_cpu_under_node(ud->cpu, ud->old_nid);
return changed;
}
+int arch_update_cpu_topology(void)
+{
+ lockdep_assert_cpus_held();
+ return numa_update_cpu_topology(true);
+}
+
static void topology_work_fn(struct work_struct *work)
{
rebuild_sched_domains();
struct pt_regs *regs_user_copy)
{
regs_user->regs = task_pt_regs(current);
- regs_user->abi = perf_reg_abi(current);
+ regs_user->abi = (regs_user->regs) ? perf_reg_abi(current) :
+ PERF_SAMPLE_REGS_ABI_NONE;
}
return mmio_atsd_reg;
}
-static int mmio_invalidate_pid(struct npu *npu, unsigned long pid)
+static int mmio_invalidate_pid(struct npu *npu, unsigned long pid, bool flush)
{
unsigned long launch;
/* PID */
launch |= pid << PPC_BITLSHIFT(38);
+ /* No flush */
+ launch |= !flush << PPC_BITLSHIFT(39);
+
/* Invalidating the entire process doesn't use a va */
return mmio_launch_invalidate(npu, launch, 0);
}
static int mmio_invalidate_va(struct npu *npu, unsigned long va,
- unsigned long pid)
+ unsigned long pid, bool flush)
{
unsigned long launch;
/* PID */
launch |= pid << PPC_BITLSHIFT(38);
+ /* No flush */
+ launch |= !flush << PPC_BITLSHIFT(39);
+
return mmio_launch_invalidate(npu, launch, va);
}
#define mn_to_npu_context(x) container_of(x, struct npu_context, mn)
+struct mmio_atsd_reg {
+ struct npu *npu;
+ int reg;
+};
+
+static void mmio_invalidate_wait(
+ struct mmio_atsd_reg mmio_atsd_reg[NV_MAX_NPUS], bool flush)
+{
+ struct npu *npu;
+ int i, reg;
+
+ /* Wait for all invalidations to complete */
+ for (i = 0; i <= max_npu2_index; i++) {
+ if (mmio_atsd_reg[i].reg < 0)
+ continue;
+
+ /* Wait for completion */
+ npu = mmio_atsd_reg[i].npu;
+ reg = mmio_atsd_reg[i].reg;
+ while (__raw_readq(npu->mmio_atsd_regs[reg] + XTS_ATSD_STAT))
+ cpu_relax();
+
+ put_mmio_atsd_reg(npu, reg);
+
+ /*
+ * The GPU requires two flush ATSDs to ensure all entries have
+ * been flushed. We use PID 0 as it will never be used for a
+ * process on the GPU.
+ */
+ if (flush)
+ mmio_invalidate_pid(npu, 0, true);
+ }
+}
+
/*
* Invalidate either a single address or an entire PID depending on
* the value of va.
*/
static void mmio_invalidate(struct npu_context *npu_context, int va,
- unsigned long address)
+ unsigned long address, bool flush)
{
- int i, j, reg;
+ int i, j;
struct npu *npu;
struct pnv_phb *nphb;
struct pci_dev *npdev;
- struct {
- struct npu *npu;
- int reg;
- } mmio_atsd_reg[NV_MAX_NPUS];
+ struct mmio_atsd_reg mmio_atsd_reg[NV_MAX_NPUS];
unsigned long pid = npu_context->mm->context.id;
/*
if (va)
mmio_atsd_reg[i].reg =
- mmio_invalidate_va(npu, address, pid);
+ mmio_invalidate_va(npu, address, pid,
+ flush);
else
mmio_atsd_reg[i].reg =
- mmio_invalidate_pid(npu, pid);
+ mmio_invalidate_pid(npu, pid, flush);
/*
* The NPU hardware forwards the shootdown to all GPUs
*/
flush_tlb_mm(npu_context->mm);
- /* Wait for all invalidations to complete */
- for (i = 0; i <= max_npu2_index; i++) {
- if (mmio_atsd_reg[i].reg < 0)
- continue;
-
- /* Wait for completion */
- npu = mmio_atsd_reg[i].npu;
- reg = mmio_atsd_reg[i].reg;
- while (__raw_readq(npu->mmio_atsd_regs[reg] + XTS_ATSD_STAT))
- cpu_relax();
- put_mmio_atsd_reg(npu, reg);
- }
+ mmio_invalidate_wait(mmio_atsd_reg, flush);
+ if (flush)
+ /* Wait for the flush to complete */
+ mmio_invalidate_wait(mmio_atsd_reg, false);
}
static void pnv_npu2_mn_release(struct mmu_notifier *mn,
* There should be no more translation requests for this PID, but we
* need to ensure any entries for it are removed from the TLB.
*/
- mmio_invalidate(npu_context, 0, 0);
+ mmio_invalidate(npu_context, 0, 0, true);
}
static void pnv_npu2_mn_change_pte(struct mmu_notifier *mn,
{
struct npu_context *npu_context = mn_to_npu_context(mn);
- mmio_invalidate(npu_context, 1, address);
+ mmio_invalidate(npu_context, 1, address, true);
}
static void pnv_npu2_mn_invalidate_page(struct mmu_notifier *mn,
{
struct npu_context *npu_context = mn_to_npu_context(mn);
- mmio_invalidate(npu_context, 1, address);
+ mmio_invalidate(npu_context, 1, address, true);
}
static void pnv_npu2_mn_invalidate_range(struct mmu_notifier *mn,
struct npu_context *npu_context = mn_to_npu_context(mn);
unsigned long address;
- for (address = start; address <= end; address += PAGE_SIZE)
- mmio_invalidate(npu_context, 1, address);
+ for (address = start; address < end; address += PAGE_SIZE)
+ mmio_invalidate(npu_context, 1, address, false);
+
+ /* Do the flush only on the final addess == end */
+ mmio_invalidate(npu_context, 1, address, true);
}
static const struct mmu_notifier_ops nv_nmmu_notifier_ops = {
/* No nvlink associated with this GPU device */
return ERR_PTR(-ENODEV);
- if (!mm) {
- /* kernel thread contexts are not supported */
+ if (!mm || mm->context.id == 0) {
+ /*
+ * Kernel thread contexts are not supported and context id 0 is
+ * reserved on the GPU.
+ */
return ERR_PTR(-EINVAL);
}
state->master = 0;
}
- get_online_cpus();
+ cpus_read_lock();
/* This cpu will update the globals before exiting stop machine */
this_cpu_ptr(&split_state)->master = 1;
/* Ensure state is consistent before we call the other cpus */
mb();
- stop_machine(cpu_update_split_mode, &new_mode, cpu_online_mask);
+ stop_machine_cpuslocked(cpu_update_split_mode, &new_mode,
+ cpu_online_mask);
- put_online_cpus();
+ cpus_read_unlock();
return 0;
}
return (len >= PAGE_SIZE) ? (PAGE_SIZE - 1) : len;
}
+static DEVICE_ATTR_RO(modalias);
-static struct device_attribute ps3_system_bus_dev_attrs[] = {
- __ATTR_RO(modalias),
- __ATTR_NULL,
+static struct attribute *ps3_system_bus_dev_attrs[] = {
+ &dev_attr_modalias.attr,
+ NULL,
};
+ATTRIBUTE_GROUPS(ps3_system_bus_dev);
struct bus_type ps3_system_bus_type = {
.name = "ps3_system_bus",
.probe = ps3_system_bus_probe,
.remove = ps3_system_bus_remove,
.shutdown = ps3_system_bus_shutdown,
- .dev_attrs = ps3_system_bus_dev_attrs,
+ .dev_groups = ps3_system_bus_dev_groups,
};
static int __init ps3_system_bus_init(void)
return sprintf(buf, "%s\n", "memory,cpu");
}
-static CLASS_ATTR(dlpar, S_IWUSR | S_IRUSR, dlpar_show, dlpar_store);
+static CLASS_ATTR_RW(dlpar);
static int __init pseries_dlpar_init(void)
{
ofdev = to_platform_device(dev);
return sprintf(buf, "%s\n", ofdev->dev.of_node->full_name);
}
+static DEVICE_ATTR_RO(devspec);
static ssize_t name_show(struct device *dev,
struct device_attribute *attr, char *buf)
ofdev = to_platform_device(dev);
return sprintf(buf, "%s\n", ofdev->dev.of_node->name);
}
+static DEVICE_ATTR_RO(name);
static ssize_t modalias_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
return of_device_modalias(dev, buf, PAGE_SIZE);
}
+static DEVICE_ATTR_RO(modalias);
-static struct device_attribute ibmebus_bus_device_attrs[] = {
- __ATTR_RO(devspec),
- __ATTR_RO(name),
- __ATTR_RO(modalias),
- __ATTR_NULL
+static struct attribute *ibmebus_bus_device_attrs[] = {
+ &dev_attr_devspec.attr,
+ &dev_attr_name.attr,
+ &dev_attr_modalias.attr,
+ NULL,
};
+ATTRIBUTE_GROUPS(ibmebus_bus_device);
struct bus_type ibmebus_bus_type = {
.name = "ibmebus",
.probe = ibmebus_bus_device_probe,
.remove = ibmebus_bus_device_remove,
.shutdown = ibmebus_bus_device_shutdown,
- .dev_attrs = ibmebus_bus_device_attrs,
+ .dev_groups = ibmebus_bus_device_groups,
};
EXPORT_SYMBOL(ibmebus_bus_type);
return;
}
-static ssize_t migrate_store(struct class *class, struct class_attribute *attr,
- const char *buf, size_t count)
+static ssize_t migration_store(struct class *class,
+ struct class_attribute *attr, const char *buf,
+ size_t count)
{
u64 streamid;
int rc;
*/
#define MIGRATION_API_VERSION 1
-static CLASS_ATTR(migration, S_IWUSR, NULL, migrate_store);
+static CLASS_ATTR_WO(migration);
static CLASS_ATTR_STRING(api_version, S_IRUGO, __stringify(MIGRATION_API_VERSION));
static int __init mobility_sysfs_init(void)
/* sysfs device functions and data structures for CMO */
#define viodev_cmo_rd_attr(name) \
-static ssize_t viodev_cmo_##name##_show(struct device *dev, \
+static ssize_t cmo_##name##_show(struct device *dev, \
struct device_attribute *attr, \
char *buf) \
{ \
return sprintf(buf, "%lu\n", to_vio_dev(dev)->cmo.name); \
}
-static ssize_t viodev_cmo_allocs_failed_show(struct device *dev,
+static ssize_t cmo_allocs_failed_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct vio_dev *viodev = to_vio_dev(dev);
return sprintf(buf, "%d\n", atomic_read(&viodev->cmo.allocs_failed));
}
-static ssize_t viodev_cmo_allocs_failed_reset(struct device *dev,
+static ssize_t cmo_allocs_failed_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t count)
{
struct vio_dev *viodev = to_vio_dev(dev);
return count;
}
-static ssize_t viodev_cmo_desired_set(struct device *dev,
+static ssize_t cmo_desired_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t count)
{
struct vio_dev *viodev = to_vio_dev(dev);
static ssize_t devspec_show(struct device *, struct device_attribute *, char *);
static ssize_t modalias_show(struct device *dev, struct device_attribute *attr,
char *buf);
-static struct device_attribute vio_cmo_dev_attrs[] = {
- __ATTR_RO(name),
- __ATTR_RO(devspec),
- __ATTR_RO(modalias),
- __ATTR(cmo_desired, S_IWUSR|S_IRUSR|S_IWGRP|S_IRGRP|S_IROTH,
- viodev_cmo_desired_show, viodev_cmo_desired_set),
- __ATTR(cmo_entitled, S_IRUGO, viodev_cmo_entitled_show, NULL),
- __ATTR(cmo_allocated, S_IRUGO, viodev_cmo_allocated_show, NULL),
- __ATTR(cmo_allocs_failed, S_IWUSR|S_IRUSR|S_IWGRP|S_IRGRP|S_IROTH,
- viodev_cmo_allocs_failed_show, viodev_cmo_allocs_failed_reset),
- __ATTR_NULL
+
+static struct device_attribute dev_attr_name;
+static struct device_attribute dev_attr_devspec;
+static struct device_attribute dev_attr_modalias;
+
+static DEVICE_ATTR_RO(cmo_entitled);
+static DEVICE_ATTR_RO(cmo_allocated);
+static DEVICE_ATTR_RW(cmo_desired);
+static DEVICE_ATTR_RW(cmo_allocs_failed);
+
+static struct attribute *vio_cmo_dev_attrs[] = {
+ &dev_attr_name.attr,
+ &dev_attr_devspec.attr,
+ &dev_attr_modalias.attr,
+ &dev_attr_cmo_entitled.attr,
+ &dev_attr_cmo_allocated.attr,
+ &dev_attr_cmo_desired.attr,
+ &dev_attr_cmo_allocs_failed.attr,
+ NULL,
};
+ATTRIBUTE_GROUPS(vio_cmo_dev);
/* sysfs bus functions and data structures for CMO */
#define viobus_cmo_rd_attr(name) \
-static ssize_t cmo_##name##_show(struct bus_type *bt, char *buf) \
+static ssize_t cmo_bus_##name##_show(struct bus_type *bt, char *buf) \
{ \
return sprintf(buf, "%lu\n", vio_cmo.name); \
} \
-static BUS_ATTR_RO(cmo_##name)
+static struct bus_attribute bus_attr_cmo_bus_##name = \
+ __ATTR(cmo_##name, S_IRUGO, cmo_bus_##name##_show, NULL)
#define viobus_cmo_pool_rd_attr(name, var) \
static ssize_t \
static BUS_ATTR_RW(cmo_high);
static struct attribute *vio_bus_attrs[] = {
- &bus_attr_cmo_entitled.attr,
- &bus_attr_cmo_spare.attr,
- &bus_attr_cmo_min.attr,
- &bus_attr_cmo_desired.attr,
- &bus_attr_cmo_curr.attr,
+ &bus_attr_cmo_bus_entitled.attr,
+ &bus_attr_cmo_bus_spare.attr,
+ &bus_attr_cmo_bus_min.attr,
+ &bus_attr_cmo_bus_desired.attr,
+ &bus_attr_cmo_bus_curr.attr,
&bus_attr_cmo_high.attr,
&bus_attr_cmo_reserve_size.attr,
&bus_attr_cmo_excess_size.attr,
static void vio_cmo_sysfs_init(void)
{
- vio_bus_type.dev_attrs = vio_cmo_dev_attrs;
+ vio_bus_type.dev_groups = vio_cmo_dev_groups;
vio_bus_type.bus_groups = vio_bus_groups;
}
#else /* CONFIG_PPC_SMLPAR */
{
return sprintf(buf, "%s\n", to_vio_dev(dev)->name);
}
+static DEVICE_ATTR_RO(name);
static ssize_t devspec_show(struct device *dev,
struct device_attribute *attr, char *buf)
return sprintf(buf, "%s\n", of_node_full_name(of_node));
}
+static DEVICE_ATTR_RO(devspec);
static ssize_t modalias_show(struct device *dev, struct device_attribute *attr,
char *buf)
return sprintf(buf, "vio:T%sS%s\n", vio_dev->type, cp);
}
+static DEVICE_ATTR_RO(modalias);
-static struct device_attribute vio_dev_attrs[] = {
- __ATTR_RO(name),
- __ATTR_RO(devspec),
- __ATTR_RO(modalias),
- __ATTR_NULL
+static struct attribute *vio_dev_attrs[] = {
+ &dev_attr_name.attr,
+ &dev_attr_devspec.attr,
+ &dev_attr_modalias.attr,
+ NULL,
};
+ATTRIBUTE_GROUPS(vio_dev);
void vio_unregister_device(struct vio_dev *viodev)
{
struct bus_type vio_bus_type = {
.name = "vio",
- .dev_attrs = vio_dev_attrs,
+ .dev_groups = vio_dev_groups,
.uevent = vio_hotplug,
.match = vio_bus_match,
.probe = vio_bus_probe,
config S390
def_bool y
+ select ARCH_BINFMT_ELF_STATE
select ARCH_HAS_DEVMEM_IS_ALLOWED
select ARCH_HAS_ELF_RANDOMIZE
select ARCH_HAS_GCOV_PROFILE_ALL
config PGTABLE_LEVELS
int
- default 4
+ default 5
source "init/Kconfig"
obj-$(CONFIG_CRYPTO_SHA256_S390) += sha256_s390.o sha_common.o
obj-$(CONFIG_CRYPTO_SHA512_S390) += sha512_s390.o sha_common.o
obj-$(CONFIG_CRYPTO_DES_S390) += des_s390.o
-obj-$(CONFIG_CRYPTO_AES_S390) += aes_s390.o paes_s390.o
+obj-$(CONFIG_CRYPTO_AES_S390) += aes_s390.o
+obj-$(CONFIG_CRYPTO_PAES_S390) += paes_s390.o
obj-$(CONFIG_S390_PRNG) += prng.o
obj-$(CONFIG_CRYPTO_GHASH_S390) += ghash_s390.o
obj-$(CONFIG_CRYPTO_CRC32_S390) += crc32-vx_s390.o
#include <linux/kernel.h>
#include <linux/atomic.h>
+#include <linux/random.h>
#include <linux/static_key.h>
#include <asm/cpacf.h>
generic-y += asm-offsets.h
generic-y += cacheflush.h
generic-y += clkdev.h
+generic-y += device.h
generic-y += dma-contiguous.h
generic-y += div64.h
generic-y += emergency-restart.h
generic-y += export.h
+generic-y += fb.h
generic-y += irq_regs.h
generic-y += irq_work.h
generic-y += kmap_types.h
+++ /dev/null
-/*
- * Arch specific extensions to struct device
- *
- * This file is released under the GPLv2
- */
-struct dev_archdata {
-};
-
-struct pdev_archdata {
-};
#include <linux/types.h>
#include <linux/device.h>
+#include <linux/blkdev.h>
struct arqb {
u64 data;
int (*probe) (struct scm_device *scmdev);
int (*remove) (struct scm_device *scmdev);
void (*notify) (struct scm_device *scmdev, enum scm_event event);
- void (*handler) (struct scm_device *scmdev, void *data, int error);
+ void (*handler) (struct scm_device *scmdev, void *data,
+ blk_status_t error);
};
int scm_driver_register(struct scm_driver *scmdrv);
void scm_driver_unregister(struct scm_driver *scmdrv);
int eadm_start_aob(struct aob *aob);
-void scm_irq_handler(struct aob *aob, int error);
+void scm_irq_handler(struct aob *aob, blk_status_t error);
#endif /* _ASM_S390_EADM_H */
#define ELF_DATA ELFDATA2MSB
#define ELF_ARCH EM_S390
+/* s390 specific phdr types */
+#define PT_S390_PGSTE 0x70000000
+
/*
* ELF register definitions..
*/
&& (x)->e_ident[EI_CLASS] == ELF_CLASS)
#define compat_start_thread start_thread31
+struct arch_elf_state {
+ int rc;
+};
+
+#define INIT_ARCH_ELF_STATE { .rc = 0 }
+
+#define arch_check_elf(ehdr, interp, interp_ehdr, state) (0)
+#ifdef CONFIG_PGSTE
+#define arch_elf_pt_proc(ehdr, phdr, elf, interp, state) \
+({ \
+ struct arch_elf_state *_state = state; \
+ if ((phdr)->p_type == PT_S390_PGSTE && \
+ !page_table_allocate_pgste && \
+ !test_thread_flag(TIF_PGSTE) && \
+ !current->mm->context.alloc_pgste) { \
+ set_thread_flag(TIF_PGSTE); \
+ set_pt_regs_flag(task_pt_regs(current), \
+ PIF_SYSCALL_RESTART); \
+ _state->rc = -EAGAIN; \
+ } \
+ _state->rc; \
+})
+#else
+#define arch_elf_pt_proc(ehdr, phdr, elf, interp, state) \
+({ \
+ (state)->rc; \
+})
+#endif
+
/* For SVR4/S390 the function pointer to be registered with `atexit` is
passed in R14. */
#define ELF_PLAT_INIT(_r, load_addr) \
+++ /dev/null
-#ifndef _ASM_FB_H_
-#define _ASM_FB_H_
-#include <linux/fb.h>
-
-#define fb_pgprotect(...) do {} while (0)
-
-static inline int fb_is_primary_device(struct fb_info *info)
-{
- return 0;
-}
-
-#endif /* _ASM_FB_H_ */
#define IO_SPACE_LIMIT 0
-#ifdef CONFIG_PCI
-
#define ioremap_nocache(addr, size) ioremap(addr, size)
#define ioremap_wc ioremap_nocache
#define ioremap_wt ioremap_nocache
{
}
+#ifdef CONFIG_PCI
+
/*
* s390 needs a private implementation of pci_iomap since ioremap with its
* offset parameter isn't sufficient. That's because BAR spaces are not
struct esca_entry cpu[KVM_S390_ESCA_CPU_SLOTS];
} __packed;
+/*
+ * This struct is used to store some machine check info from lowcore
+ * for machine checks that happen while the guest is running.
+ * This info in host's lowcore might be overwritten by a second machine
+ * check from host when host is in the machine check's high-level handling.
+ * The size is 24 bytes.
+ */
+struct mcck_volatile_info {
+ __u64 mcic;
+ __u64 failing_storage_address;
+ __u32 ext_damage_code;
+ __u32 reserved;
+};
+
#define CPUSTAT_STOPPED 0x80000000
#define CPUSTAT_WAIT 0x10000000
#define CPUSTAT_ECALL_PEND 0x08000000
struct sie_page {
struct kvm_s390_sie_block sie_block;
- __u8 reserved200[1024]; /* 0x0200 */
+ struct mcck_volatile_info mcck_info; /* 0x0200 */
+ __u8 reserved218[1000]; /* 0x0218 */
struct kvm_s390_itdb itdb; /* 0x0600 */
__u8 reserved700[2304]; /* 0x0700 */
} __packed;
mm->context.gmap_asce = 0;
mm->context.flush_mm = 0;
#ifdef CONFIG_PGSTE
- mm->context.alloc_pgste = page_table_allocate_pgste;
+ mm->context.alloc_pgste = page_table_allocate_pgste ||
+ test_thread_flag(TIF_PGSTE) ||
+ current->mm->context.alloc_pgste;
mm->context.has_pgste = 0;
mm->context.use_skey = 0;
mm->context.use_cmma = 0;
#include <linux/const.h>
#include <linux/types.h>
+#define MCIC_SUBCLASS_MASK (1ULL<<63 | 1ULL<<62 | 1ULL<<61 | \
+ 1ULL<<59 | 1ULL<<58 | 1ULL<<56 | \
+ 1ULL<<55 | 1ULL<<54 | 1ULL<<53 | \
+ 1ULL<<52 | 1ULL<<47 | 1ULL<<46 | \
+ 1ULL<<45 | 1ULL<<44)
#define MCCK_CODE_SYSTEM_DAMAGE _BITUL(63)
+#define MCCK_CODE_EXT_DAMAGE _BITUL(63 - 5)
+#define MCCK_CODE_CP _BITUL(63 - 9)
#define MCCK_CODE_CPU_TIMER_VALID _BITUL(63 - 46)
#define MCCK_CODE_PSW_MWP_VALID _BITUL(63 - 20)
#define MCCK_CODE_PSW_IA_VALID _BITUL(63 - 23)
typedef struct { unsigned long pte; } pte_t;
typedef struct { unsigned long pmd; } pmd_t;
typedef struct { unsigned long pud; } pud_t;
+typedef struct { unsigned long p4d; } p4d_t;
typedef struct { unsigned long pgd; } pgd_t;
typedef pte_t *pgtable_t;
#define pte_val(x) ((x).pte)
#define pmd_val(x) ((x).pmd)
#define pud_val(x) ((x).pud)
+#define p4d_val(x) ((x).p4d)
#define pgd_val(x) ((x).pgd)
#define __pgste(x) ((pgste_t) { (x) } )
#define __pte(x) ((pte_t) { (x) } )
#define __pmd(x) ((pmd_t) { (x) } )
#define __pud(x) ((pud_t) { (x) } )
+#define __p4d(x) ((p4d_t) { (x) } )
#define __pgd(x) ((pgd_t) { (x) } )
#define __pgprot(x) ((pgprot_t) { (x) } )
} __packed __aligned(128);
enum zpci_state {
- ZPCI_FN_STATE_RESERVED,
- ZPCI_FN_STATE_STANDBY,
- ZPCI_FN_STATE_CONFIGURED,
- ZPCI_FN_STATE_ONLINE,
- NR_ZPCI_FN_STATES,
+ ZPCI_FN_STATE_STANDBY = 0,
+ ZPCI_FN_STATE_CONFIGURED = 1,
+ ZPCI_FN_STATE_RESERVED = 2,
+ ZPCI_FN_STATE_ONLINE = 3,
};
struct zpci_bar_struct {
u64 msi_addr; /* MSI address */
unsigned int max_msi; /* maximum number of MSI's */
struct airq_iv *aibv; /* adapter interrupt bit vector */
- unsigned int aisb; /* number of the summary bit */
+ unsigned long aisb; /* number of the summary bit */
/* DMA stuff */
unsigned long *dma_table;
----------------------------------------------------------------------------- */
/* Base stuff */
int zpci_create_device(struct zpci_dev *);
+void zpci_remove_device(struct zpci_dev *zdev);
int zpci_enable_device(struct zpci_dev *);
int zpci_disable_device(struct zpci_dev *);
-void zpci_stop_device(struct zpci_dev *);
int zpci_register_ioat(struct zpci_dev *, u8, u64, u64, u64);
int zpci_unregister_ioat(struct zpci_dev *, u8);
+void zpci_remove_reserved_devices(void);
/* CLP */
int clp_scan_pci_devices(void);
int clp_add_pci_device(u32, u32, int);
int clp_enable_fh(struct zpci_dev *, u8);
int clp_disable_fh(struct zpci_dev *);
+int clp_get_state(u32 fid, enum zpci_state *state);
#ifdef CONFIG_PCI
/* Error handling and recovery */
u32 gd;
} __packed __aligned(8);
-int zpci_mod_fc(u64 req, struct zpci_fib *fib);
+u8 zpci_mod_fc(u64 req, struct zpci_fib *fib, u8 *status);
int zpci_refresh_trans(u64 fn, u64 addr, u64 range);
int zpci_load(u64 *data, u64 req, u64 offset);
int zpci_store(u64 data, u64 req, u64 offset);
return _SEGMENT_ENTRY_EMPTY;
if (mm->context.asce_limit <= (1UL << 42))
return _REGION3_ENTRY_EMPTY;
- return _REGION2_ENTRY_EMPTY;
+ if (mm->context.asce_limit <= (1UL << 53))
+ return _REGION2_ENTRY_EMPTY;
+ return _REGION1_ENTRY_EMPTY;
}
-int crst_table_upgrade(struct mm_struct *);
+int crst_table_upgrade(struct mm_struct *mm, unsigned long limit);
void crst_table_downgrade(struct mm_struct *);
+static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long address)
+{
+ unsigned long *table = crst_table_alloc(mm);
+
+ if (table)
+ crst_table_init(table, _REGION2_ENTRY_EMPTY);
+ return (p4d_t *) table;
+}
+#define p4d_free(mm, p4d) crst_table_free(mm, (unsigned long *) p4d)
+
static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address)
{
unsigned long *table = crst_table_alloc(mm);
crst_table_free(mm, (unsigned long *) pmd);
}
-static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, pud_t *pud)
+static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, p4d_t *p4d)
+{
+ pgd_val(*pgd) = _REGION1_ENTRY | __pa(p4d);
+}
+
+static inline void p4d_populate(struct mm_struct *mm, p4d_t *p4d, pud_t *pud)
{
- pgd_val(*pgd) = _REGION2_ENTRY | __pa(pud);
+ p4d_val(*p4d) = _REGION2_ENTRY | __pa(pud);
}
static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd)
* the S390 page table tree.
*/
#ifndef __ASSEMBLY__
-#include <asm-generic/5level-fixup.h>
#include <linux/sched.h>
#include <linux/mm_types.h>
#include <linux/page-flags.h>
*/
#define PMD_SHIFT 20
#define PUD_SHIFT 31
-#define PGDIR_SHIFT 42
+#define P4D_SHIFT 42
+#define PGDIR_SHIFT 53
#define PMD_SIZE (1UL << PMD_SHIFT)
#define PMD_MASK (~(PMD_SIZE-1))
#define PUD_SIZE (1UL << PUD_SHIFT)
#define PUD_MASK (~(PUD_SIZE-1))
+#define P4D_SIZE (1UL << P4D_SHIFT)
+#define P4D_MASK (~(P4D_SIZE-1))
#define PGDIR_SIZE (1UL << PGDIR_SHIFT)
#define PGDIR_MASK (~(PGDIR_SIZE-1))
#define PTRS_PER_PTE 256
#define PTRS_PER_PMD 2048
#define PTRS_PER_PUD 2048
+#define PTRS_PER_P4D 2048
#define PTRS_PER_PGD 2048
#define FIRST_USER_ADDRESS 0UL
printk("%s:%d: bad pmd %p.\n", __FILE__, __LINE__, (void *) pmd_val(e))
#define pud_ERROR(e) \
printk("%s:%d: bad pud %p.\n", __FILE__, __LINE__, (void *) pud_val(e))
+#define p4d_ERROR(e) \
+ printk("%s:%d: bad p4d %p.\n", __FILE__, __LINE__, (void *) p4d_val(e))
#define pgd_ERROR(e) \
printk("%s:%d: bad pgd %p.\n", __FILE__, __LINE__, (void *) pgd_val(e))
#define _REGION3_ENTRY_EMPTY (_REGION_ENTRY_TYPE_R3 | _REGION_ENTRY_INVALID)
#define _REGION3_ENTRY_ORIGIN_LARGE ~0x7fffffffUL /* large page address */
-#define _REGION3_ENTRY_ORIGIN ~0x7ffUL/* region third table origin */
-
#define _REGION3_ENTRY_DIRTY 0x2000 /* SW region dirty bit */
#define _REGION3_ENTRY_YOUNG 0x1000 /* SW region young bit */
#define _REGION3_ENTRY_LARGE 0x0400 /* RTTE-format control, large page */
#define _REGION3_ENTRY_SOFT_DIRTY 0x0000 /* SW region soft dirty bit */
#endif
-#define _REGION_ENTRY_BITS 0xfffffffffffff227UL
-#define _REGION_ENTRY_BITS_LARGE 0xffffffff8000fe27UL
+#define _REGION_ENTRY_BITS 0xfffffffffffff22fUL
+#define _REGION_ENTRY_BITS_LARGE 0xffffffff8000fe2fUL
/* Bits in the segment table entry */
#define _SEGMENT_ENTRY_BITS 0xfffffffffffffe33UL
}
/*
- * pgd/pmd/pte query functions
+ * pgd/p4d/pud/pmd/pte query functions
*/
+static inline int pgd_folded(pgd_t pgd)
+{
+ return (pgd_val(pgd) & _REGION_ENTRY_TYPE_MASK) < _REGION_ENTRY_TYPE_R1;
+}
+
static inline int pgd_present(pgd_t pgd)
{
- if ((pgd_val(pgd) & _REGION_ENTRY_TYPE_MASK) < _REGION_ENTRY_TYPE_R2)
+ if (pgd_folded(pgd))
return 1;
return (pgd_val(pgd) & _REGION_ENTRY_ORIGIN) != 0UL;
}
static inline int pgd_none(pgd_t pgd)
{
- if ((pgd_val(pgd) & _REGION_ENTRY_TYPE_MASK) < _REGION_ENTRY_TYPE_R2)
+ if (pgd_folded(pgd))
return 0;
return (pgd_val(pgd) & _REGION_ENTRY_INVALID) != 0UL;
}
return (pgd_val(pgd) & mask) != 0;
}
+static inline int p4d_folded(p4d_t p4d)
+{
+ return (p4d_val(p4d) & _REGION_ENTRY_TYPE_MASK) < _REGION_ENTRY_TYPE_R2;
+}
+
+static inline int p4d_present(p4d_t p4d)
+{
+ if (p4d_folded(p4d))
+ return 1;
+ return (p4d_val(p4d) & _REGION_ENTRY_ORIGIN) != 0UL;
+}
+
+static inline int p4d_none(p4d_t p4d)
+{
+ if (p4d_folded(p4d))
+ return 0;
+ return p4d_val(p4d) == _REGION2_ENTRY_EMPTY;
+}
+
+static inline unsigned long p4d_pfn(p4d_t p4d)
+{
+ unsigned long origin_mask;
+
+ origin_mask = _REGION_ENTRY_ORIGIN;
+ return (p4d_val(p4d) & origin_mask) >> PAGE_SHIFT;
+}
+
+static inline int pud_folded(pud_t pud)
+{
+ return (pud_val(pud) & _REGION_ENTRY_TYPE_MASK) < _REGION_ENTRY_TYPE_R3;
+}
+
static inline int pud_present(pud_t pud)
{
- if ((pud_val(pud) & _REGION_ENTRY_TYPE_MASK) < _REGION_ENTRY_TYPE_R3)
+ if (pud_folded(pud))
return 1;
return (pud_val(pud) & _REGION_ENTRY_ORIGIN) != 0UL;
}
static inline int pud_none(pud_t pud)
{
- if ((pud_val(pud) & _REGION_ENTRY_TYPE_MASK) < _REGION_ENTRY_TYPE_R3)
+ if (pud_folded(pud))
return 0;
return pud_val(pud) == _REGION3_ENTRY_EMPTY;
}
{
unsigned long origin_mask;
- origin_mask = _REGION3_ENTRY_ORIGIN;
+ origin_mask = _REGION_ENTRY_ORIGIN;
if (pud_large(pud))
origin_mask = _REGION3_ENTRY_ORIGIN_LARGE;
return (pud_val(pud) & origin_mask) >> PAGE_SHIFT;
return (pud_val(pud) & ~_REGION_ENTRY_BITS) != 0;
}
+static inline int p4d_bad(p4d_t p4d)
+{
+ if ((p4d_val(p4d) & _REGION_ENTRY_TYPE_MASK) < _REGION_ENTRY_TYPE_R2)
+ return pud_bad(__pud(p4d_val(p4d)));
+ return (p4d_val(p4d) & ~_REGION_ENTRY_BITS) != 0;
+}
+
static inline int pmd_present(pmd_t pmd)
{
return pmd_val(pmd) != _SEGMENT_ENTRY_EMPTY;
static inline void pgd_clear(pgd_t *pgd)
{
- if ((pgd_val(*pgd) & _REGION_ENTRY_TYPE_MASK) == _REGION_ENTRY_TYPE_R2)
- pgd_val(*pgd) = _REGION2_ENTRY_EMPTY;
+ if ((pgd_val(*pgd) & _REGION_ENTRY_TYPE_MASK) == _REGION_ENTRY_TYPE_R1)
+ pgd_val(*pgd) = _REGION1_ENTRY_EMPTY;
+}
+
+static inline void p4d_clear(p4d_t *p4d)
+{
+ if ((p4d_val(*p4d) & _REGION_ENTRY_TYPE_MASK) == _REGION_ENTRY_TYPE_R2)
+ p4d_val(*p4d) = _REGION2_ENTRY_EMPTY;
}
static inline void pud_clear(pud_t *pud)
}
#define pgd_index(address) (((address) >> PGDIR_SHIFT) & (PTRS_PER_PGD-1))
+#define p4d_index(address) (((address) >> P4D_SHIFT) & (PTRS_PER_P4D-1))
#define pud_index(address) (((address) >> PUD_SHIFT) & (PTRS_PER_PUD-1))
#define pmd_index(address) (((address) >> PMD_SHIFT) & (PTRS_PER_PMD-1))
#define pte_index(address) (((address) >> PAGE_SHIFT) & (PTRS_PER_PTE-1))
#define pmd_deref(pmd) (pmd_val(pmd) & _SEGMENT_ENTRY_ORIGIN)
#define pud_deref(pud) (pud_val(pud) & _REGION_ENTRY_ORIGIN)
+#define p4d_deref(pud) (p4d_val(pud) & _REGION_ENTRY_ORIGIN)
#define pgd_deref(pgd) (pgd_val(pgd) & _REGION_ENTRY_ORIGIN)
-static inline pud_t *pud_offset(pgd_t *pgd, unsigned long address)
+static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address)
{
- pud_t *pud = (pud_t *) pgd;
- if ((pgd_val(*pgd) & _REGION_ENTRY_TYPE_MASK) == _REGION_ENTRY_TYPE_R2)
- pud = (pud_t *) pgd_deref(*pgd);
- return pud + pud_index(address);
+ p4d_t *p4d = (p4d_t *) pgd;
+
+ if ((pgd_val(*pgd) & _REGION_ENTRY_TYPE_MASK) == _REGION_ENTRY_TYPE_R1)
+ p4d = (p4d_t *) pgd_deref(*pgd);
+ return p4d + p4d_index(address);
+}
+
+static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address)
+{
+ pud_t *pud = (pud_t *) p4d;
+
+ if ((p4d_val(*p4d) & _REGION_ENTRY_TYPE_MASK) == _REGION_ENTRY_TYPE_R2)
+ pud = (pud_t *) p4d_deref(*p4d);
+ return pud + pud_index(address);
}
static inline pmd_t *pmd_offset(pud_t *pud, unsigned long address)
{
pmd_t *pmd = (pmd_t *) pud;
+
if ((pud_val(*pud) & _REGION_ENTRY_TYPE_MASK) == _REGION_ENTRY_TYPE_R3)
pmd = (pmd_t *) pud_deref(*pud);
return pmd + pmd_index(address);
#define pmd_page(pmd) pfn_to_page(pmd_pfn(pmd))
#define pud_page(pud) pfn_to_page(pud_pfn(pud))
+#define p4d_page(pud) pfn_to_page(p4d_pfn(p4d))
/* Find an entry in the lowest level page table.. */
#define pte_offset(pmd, addr) ((pte_t *) pmd_deref(*(pmd)) + pte_index(addr))
#define CIF_FPU 4 /* restore FPU registers */
#define CIF_IGNORE_IRQ 5 /* ignore interrupt (for udelay) */
#define CIF_ENABLED_WAIT 6 /* in enabled wait state */
+#define CIF_MCCK_GUEST 7 /* machine check happening in guest */
#define _CIF_MCCK_PENDING _BITUL(CIF_MCCK_PENDING)
#define _CIF_ASCE_PRIMARY _BITUL(CIF_ASCE_PRIMARY)
#define _CIF_FPU _BITUL(CIF_FPU)
#define _CIF_IGNORE_IRQ _BITUL(CIF_IGNORE_IRQ)
#define _CIF_ENABLED_WAIT _BITUL(CIF_ENABLED_WAIT)
+#define _CIF_MCCK_GUEST _BITUL(CIF_MCCK_GUEST)
#ifndef __ASSEMBLY__
*/
#define TASK_SIZE_OF(tsk) (test_tsk_thread_flag(tsk, TIF_31BIT) ? \
- (1UL << 31) : (1UL << 53))
+ (1UL << 31) : -PAGE_SIZE)
#define TASK_UNMAPPED_BASE (test_thread_flag(TIF_31BIT) ? \
(1UL << 30) : (1UL << 41))
#define TASK_SIZE TASK_SIZE_OF(current)
-#define TASK_SIZE_MAX (1UL << 53)
+#define TASK_SIZE_MAX (-PAGE_SIZE)
#define STACK_TOP (test_thread_flag(TIF_31BIT) ? \
(1UL << 31) : (1UL << 42))
/* Free guarded storage control block for current */
void exit_thread_gs(void);
-/*
- * Return saved PC of a blocked thread.
- */
-extern unsigned long thread_saved_pc(struct task_struct *t);
-
unsigned long get_wchan(struct task_struct *p);
#define task_pt_regs(tsk) ((struct pt_regs *) \
(task_stack_page(tsk) + THREAD_SIZE) - 1)
#define PIF_SYSCALL 0 /* inside a system call */
#define PIF_PER_TRAP 1 /* deliver sigtrap on return to user */
+#define PIF_SYSCALL_RESTART 2 /* restart the current system call */
#define _PIF_SYSCALL _BITUL(PIF_SYSCALL)
#define _PIF_PER_TRAP _BITUL(PIF_PER_TRAP)
+#define _PIF_SYSCALL_RESTART _BITUL(PIF_SYSCALL_RESTART)
#ifndef __ASSEMBLY__
PSW_MASK_PSTATE | PSW_ASC_PRIMARY)
struct psw_bits {
- unsigned long : 1;
- unsigned long r : 1; /* PER-Mask */
- unsigned long : 3;
- unsigned long t : 1; /* DAT Mode */
- unsigned long i : 1; /* Input/Output Mask */
- unsigned long e : 1; /* External Mask */
- unsigned long key : 4; /* PSW Key */
- unsigned long : 1;
- unsigned long m : 1; /* Machine-Check Mask */
- unsigned long w : 1; /* Wait State */
- unsigned long p : 1; /* Problem State */
- unsigned long as : 2; /* Address Space Control */
- unsigned long cc : 2; /* Condition Code */
- unsigned long pm : 4; /* Program Mask */
- unsigned long ri : 1; /* Runtime Instrumentation */
- unsigned long : 6;
- unsigned long eaba : 2; /* Addressing Mode */
- unsigned long : 31;
- unsigned long ia : 64; /* Instruction Address */
+ unsigned long : 1;
+ unsigned long per : 1; /* PER-Mask */
+ unsigned long : 3;
+ unsigned long dat : 1; /* DAT Mode */
+ unsigned long io : 1; /* Input/Output Mask */
+ unsigned long ext : 1; /* External Mask */
+ unsigned long key : 4; /* PSW Key */
+ unsigned long : 1;
+ unsigned long mcheck : 1; /* Machine-Check Mask */
+ unsigned long wait : 1; /* Wait State */
+ unsigned long pstate : 1; /* Problem State */
+ unsigned long as : 2; /* Address Space Control */
+ unsigned long cc : 2; /* Condition Code */
+ unsigned long pm : 4; /* Program Mask */
+ unsigned long ri : 1; /* Runtime Instrumentation */
+ unsigned long : 6;
+ unsigned long eaba : 2; /* Addressing Mode */
+ unsigned long : 31;
+ unsigned long ia : 64; /* Instruction Address */
};
enum {
- PSW_AMODE_24BIT = 0,
- PSW_AMODE_31BIT = 1,
- PSW_AMODE_64BIT = 3
+ PSW_BITS_AMODE_24BIT = 0,
+ PSW_BITS_AMODE_31BIT = 1,
+ PSW_BITS_AMODE_64BIT = 3
};
enum {
- PSW_AS_PRIMARY = 0,
- PSW_AS_ACCREG = 1,
- PSW_AS_SECONDARY = 2,
- PSW_AS_HOME = 3
+ PSW_BITS_AS_PRIMARY = 0,
+ PSW_BITS_AS_ACCREG = 1,
+ PSW_BITS_AS_SECONDARY = 2,
+ PSW_BITS_AS_HOME = 3
};
#define psw_bits(__psw) (*({ \
int cc;
cc = ____pcpu_sigp(addr, order, parm, &_status);
- if (status && cc == 1)
+ if (status && cc == SIGP_CC_STATUS_STORED)
*status = _status;
return cc;
}
unsigned short cpus_shared;
char reserved_4[3];
unsigned char vsne;
- uuid_be uuid;
+ uuid_t uuid;
char reserved_5[160];
char ext_name[256];
};
char reserved_1[3];
unsigned char evmne;
unsigned int reserved_2;
- uuid_be uuid;
+ uuid_t uuid;
} vm[8];
char reserved_3[1504];
char ext_names[8][256];
#define TIF_UPROBE 3 /* breakpointed or single-stepping */
#define TIF_GUARDED_STORAGE 4 /* load guarded storage control block */
#define TIF_PATCH_PENDING 5 /* pending live patching update */
+#define TIF_PGSTE 6 /* New mm's will use 4K page tables */
#define TIF_31BIT 16 /* 32bit process */
#define TIF_MEMDIE 17 /* is terminating due to OOM killer */
tlb_remove_table(tlb, pmd);
}
+/*
+ * p4d_free_tlb frees a pud table and clears the CRSTE for the
+ * region second table entry from the tlb.
+ * If the mm uses a four level page table the single p4d is freed
+ * as the pgd. p4d_free_tlb checks the asce_limit against 8PB
+ * to avoid the double free of the p4d in this case.
+ */
+static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d,
+ unsigned long address)
+{
+ if (tlb->mm->context.asce_limit <= (1UL << 53))
+ return;
+ tlb_remove_table(tlb, p4d);
+}
+
/*
* pud_free_tlb frees a pud table and clears the CRSTE for the
* region third table entry from the tlb.
OFFSET(__SF_BACKCHAIN, stack_frame, back_chain);
OFFSET(__SF_GPRS, stack_frame, gprs);
OFFSET(__SF_EMPTY, stack_frame, empty1);
+ OFFSET(__SF_SIE_CONTROL, stack_frame, empty1[0]);
+ OFFSET(__SF_SIE_SAVEAREA, stack_frame, empty1[1]);
+ OFFSET(__SF_SIE_REASON, stack_frame, empty1[2]);
BLANK();
/* timeval/timezone offsets for use by vdso */
OFFSET(__VDSO_UPD_COUNT, vdso_data, tb_update_count);
return 0;
}
-static void show_trace(struct task_struct *task, unsigned long sp)
+void show_stack(struct task_struct *task, unsigned long *stack)
{
+ unsigned long sp = (unsigned long) stack;
+
if (!sp)
sp = task ? task->thread.ksp : current_stack_pointer();
printk("Call Trace:\n");
debug_show_held_locks(task);
}
-void show_stack(struct task_struct *task, unsigned long *sp)
-{
- unsigned long *stack;
- int i;
-
- stack = sp;
- if (!stack) {
- if (!task)
- stack = (unsigned long *)current_stack_pointer();
- else
- stack = (unsigned long *)task->thread.ksp;
- }
- printk(KERN_DEFAULT "Stack:\n");
- for (i = 0; i < 20; i++) {
- if (((addr_t) stack & (THREAD_SIZE-1)) == 0)
- break;
- if (i % 4 == 0)
- printk(KERN_DEFAULT " ");
- pr_cont("%016lx%c", *stack++, i % 4 == 3 ? '\n' : ' ');
- }
- show_trace(task, (unsigned long)sp);
-}
-
static void show_last_breaking_event(struct pt_regs *regs)
{
printk("Last Breaking-Event-Address:\n");
pr_cont(" (%pSR)", (void *)regs->psw.addr);
pr_cont("\n");
printk(" R:%x T:%x IO:%x EX:%x Key:%x M:%x W:%x "
- "P:%x AS:%x CC:%x PM:%x", psw->r, psw->t, psw->i, psw->e,
- psw->key, psw->m, psw->w, psw->p, psw->as, psw->cc, psw->pm);
+ "P:%x AS:%x CC:%x PM:%x", psw->per, psw->dat, psw->io, psw->ext,
+ psw->key, psw->mcheck, psw->wait, psw->pstate, psw->as, psw->cc, psw->pm);
pr_cont(" RI:%x EA:%x\n", psw->ri, psw->eaba);
printk("%s GPRS: %016lx %016lx %016lx %016lx\n", mode,
regs->gprs[0], regs->gprs[1], regs->gprs[2], regs->gprs[3]);
show_registers(regs);
/* Show stack backtrace if pt_regs is from kernel mode */
if (!user_mode(regs))
- show_trace(NULL, regs->gprs[15]);
+ show_stack(NULL, (unsigned long *) regs->gprs[15]);
show_last_breaking_event(regs);
}
_TIF_SYSCALL_TRACEPOINT)
_CIF_WORK = (_CIF_MCCK_PENDING | _CIF_ASCE_PRIMARY | \
_CIF_ASCE_SECONDARY | _CIF_FPU)
-_PIF_WORK = (_PIF_PER_TRAP)
+_PIF_WORK = (_PIF_PER_TRAP | _PIF_SYSCALL_RESTART)
#define BASED(name) name-cleanup_critical(%r13)
jnz .Lsie_skip
TSTMSK __LC_CPU_FLAGS,_CIF_FPU
jo .Lsie_skip # exit if fp/vx regs changed
+.Lsie_entry:
sie 0(%r14)
.Lsie_skip:
ni __SIE_PROG0C+3(%r14),0xfe # no longer in SIE
jo .Lsysc_mcck_pending
TSTMSK __TI_flags(%r12),_TIF_NEED_RESCHED
jo .Lsysc_reschedule
+ TSTMSK __PT_FLAGS(%r11),_PIF_SYSCALL_RESTART
+ jo .Lsysc_syscall_restart
#ifdef CONFIG_UPROBES
TSTMSK __TI_flags(%r12),_TIF_UPROBE
jo .Lsysc_uprobe_notify
jo .Lsysc_patch_pending # handle live patching just before
# signals and possible syscall restart
#endif
+ TSTMSK __PT_FLAGS(%r11),_PIF_SYSCALL_RESTART
+ jo .Lsysc_syscall_restart
TSTMSK __TI_flags(%r12),_TIF_SIGPENDING
jo .Lsysc_sigpending
TSTMSK __TI_flags(%r12),_TIF_NOTIFY_RESUME
larl %r14,.Lsysc_return
jg do_per_trap
+#
+# _PIF_SYSCALL_RESTART is set, repeat the current system call
+#
+.Lsysc_syscall_restart:
+ ni __PT_FLAGS+7(%r11),255-_PIF_SYSCALL_RESTART
+ lmg %r1,%r7,__PT_R1(%r11) # load svc arguments
+ lg %r2,__PT_ORIG_GPR2(%r11)
+ j .Lsysc_do_svc
+
#
# call tracehook_report_syscall_entry/tracehook_report_syscall_exit before
# and after the system call
oi __LC_CPU_FLAGS+7,_CIF_FPU
br %r14
.Lsave_fpu_regs_end:
-#if IS_ENABLED(CONFIG_KVM)
EXPORT_SYMBOL(save_fpu_regs)
-#endif
/*
* Load floating-point controls and floating-point or vector registers.
.quad .Lsie_done
.Lcleanup_sie:
- lg %r9,__SF_EMPTY(%r15) # get control block pointer
+ cghi %r11,__LC_SAVE_AREA_ASYNC #Is this in normal interrupt?
+ je 1f
+ slg %r9,BASED(.Lsie_crit_mcck_start)
+ clg %r9,BASED(.Lsie_crit_mcck_length)
+ jh 1f
+ oi __LC_CPU_FLAGS+7, _CIF_MCCK_GUEST
+1: lg %r9,__SF_EMPTY(%r15) # get control block pointer
ni __SIE_PROG0C+3(%r9),0xfe # no longer in SIE
lctlg %c1,%c1,__LC_USER_ASCE # load primary asce
larl %r9,sie_exit # skip forward to sie_exit
.quad .Lsie_gmap
.Lsie_critical_length:
.quad .Lsie_done - .Lsie_gmap
+.Lsie_crit_mcck_start:
+ .quad .Lsie_entry
+.Lsie_crit_mcck_length:
+ .quad .Lsie_skip - .Lsie_entry
#endif
.section .rodata, "a"
static void __ipl_run(void *unused)
{
- if (MACHINE_IS_LPAR && ipl_info.type == IPL_TYPE_CCW)
- diag308(DIAG308_LOAD_NORMAL_DUMP, NULL);
diag308(DIAG308_LOAD_CLEAR, NULL);
if (MACHINE_IS_VM)
__cpcmd("IPL", NULL, 0, NULL);
break;
case REIPL_METHOD_CCW_DIAG:
diag308(DIAG308_SET, reipl_block_ccw);
- if (MACHINE_IS_LPAR)
- diag308(DIAG308_LOAD_NORMAL_DUMP, NULL);
- else
- diag308(DIAG308_LOAD_CLEAR, NULL);
+ diag308(DIAG308_LOAD_CLEAR, NULL);
break;
case REIPL_METHOD_FCP_RW_DIAG:
diag308(DIAG308_SET, reipl_block_fcp);
args.entry = entry;
args.type = type;
- stop_machine(__sm_arch_jump_label_transform, &args, NULL);
+ stop_machine_cpuslocked(__sm_arch_jump_label_transform, &args, NULL);
}
void arch_jump_label_transform_static(struct jump_entry *entry,
{
struct swap_insn_args args = {.p = p, .arm_kprobe = 1};
- stop_machine(swap_instruction, &args, NULL);
+ stop_machine_cpuslocked(swap_instruction, &args, NULL);
}
NOKPROBE_SYMBOL(arch_arm_kprobe);
{
struct swap_insn_args args = {.p = p, .arm_kprobe = 0};
- stop_machine(swap_instruction, &args, NULL);
+ stop_machine_cpuslocked(swap_instruction, &args, NULL);
}
NOKPROBE_SYMBOL(arch_disarm_kprobe);
#include <asm/crw.h>
#include <asm/switch_to.h>
#include <asm/ctl_reg.h>
+#include <asm/asm-offsets.h>
+#include <linux/kvm_host.h>
struct mcck_struct {
unsigned int kill_task : 1;
return kill_task;
}
+/*
+ * Backup the guest's machine check info to its description block
+ */
+static void notrace s390_backup_mcck_info(struct pt_regs *regs)
+{
+ struct mcck_volatile_info *mcck_backup;
+ struct sie_page *sie_page;
+
+ /* r14 contains the sie block, which was set in sie64a */
+ struct kvm_s390_sie_block *sie_block =
+ (struct kvm_s390_sie_block *) regs->gprs[14];
+
+ if (sie_block == NULL)
+ /* Something's seriously wrong, stop system. */
+ s390_handle_damage();
+
+ sie_page = container_of(sie_block, struct sie_page, sie_block);
+ mcck_backup = &sie_page->mcck_info;
+ mcck_backup->mcic = S390_lowcore.mcck_interruption_code &
+ ~(MCCK_CODE_CP | MCCK_CODE_EXT_DAMAGE);
+ mcck_backup->ext_damage_code = S390_lowcore.external_damage_code;
+ mcck_backup->failing_storage_address
+ = S390_lowcore.failing_storage_address;
+}
+
#define MAX_IPD_COUNT 29
#define MAX_IPD_TIME (5 * 60 * USEC_PER_SEC) /* 5 minutes */
#define ED_STP_ISLAND 6 /* External damage STP island check */
#define ED_STP_SYNC 7 /* External damage STP sync check */
+#define MCCK_CODE_NO_GUEST (MCCK_CODE_CP | MCCK_CODE_EXT_DAMAGE)
+
/*
* machine check handler.
*/
struct mcck_struct *mcck;
unsigned long long tmp;
union mci mci;
+ unsigned long mcck_dam_code;
nmi_enter();
inc_irq_stat(NMI_NMI);
/* System damage -> stopping machine */
s390_handle_damage();
}
- if (mci.pd) {
+
+ /*
+ * Reinject the instruction processing damages' machine checks
+ * including Delayed Access Exception into the guest
+ * instead of damaging the host if they happen in the guest.
+ */
+ if (mci.pd && !test_cpu_flag(CIF_MCCK_GUEST)) {
if (mci.b) {
/* Processing backup -> verify if we can survive this */
u64 z_mcic, o_mcic, t_mcic;
mcck->mcck_code = mci.val;
set_cpu_flag(CIF_MCCK_PENDING);
}
+
+ /*
+ * Backup the machine check's info if it happens when the guest
+ * is running.
+ */
+ if (test_cpu_flag(CIF_MCCK_GUEST))
+ s390_backup_mcck_info(regs);
+
if (mci.cd) {
/* Timing facility damage */
s390_handle_damage();
if (mcck->stp_queue)
set_cpu_flag(CIF_MCCK_PENDING);
}
- if (mci.se)
- /* Storage error uncorrected */
- s390_handle_damage();
- if (mci.ke)
- /* Storage key-error uncorrected */
- s390_handle_damage();
- if (mci.ds && mci.fa)
- /* Storage degradation */
- s390_handle_damage();
+
+ /*
+ * Reinject storage related machine checks into the guest if they
+ * happen when the guest is running.
+ */
+ if (!test_cpu_flag(CIF_MCCK_GUEST)) {
+ if (mci.se)
+ /* Storage error uncorrected */
+ s390_handle_damage();
+ if (mci.ke)
+ /* Storage key-error uncorrected */
+ s390_handle_damage();
+ if (mci.ds && mci.fa)
+ /* Storage degradation */
+ s390_handle_damage();
+ }
if (mci.cp) {
/* Channel report word pending */
mcck->channel_report = 1;
mcck->warning = 1;
set_cpu_flag(CIF_MCCK_PENDING);
}
+
+ /*
+ * If there are only Channel Report Pending and External Damage
+ * machine checks, they will not be reinjected into the guest
+ * because they refer to host conditions only.
+ */
+ mcck_dam_code = (mci.val & MCIC_SUBCLASS_MASK);
+ if (test_cpu_flag(CIF_MCCK_GUEST) &&
+ (mcck_dam_code & MCCK_CODE_NO_GUEST) != mcck_dam_code) {
+ /* Set exit reason code for host's later handling */
+ *((long *)(regs->gprs[15] + __SF_SIE_REASON)) = -EINTR;
+ }
+ clear_cpu_flag(CIF_MCCK_GUEST);
nmi_exit();
}
regs.int_parm = CPU_MF_INT_SF_PRA;
sde_regs = (struct perf_sf_sde_regs *) ®s.int_parm_long;
- psw_bits(regs.psw).ia = sfr->basic.ia;
- psw_bits(regs.psw).t = sfr->basic.T;
- psw_bits(regs.psw).w = sfr->basic.W;
- psw_bits(regs.psw).p = sfr->basic.P;
- psw_bits(regs.psw).as = sfr->basic.AS;
+ psw_bits(regs.psw).ia = sfr->basic.ia;
+ psw_bits(regs.psw).dat = sfr->basic.T;
+ psw_bits(regs.psw).wait = sfr->basic.W;
+ psw_bits(regs.psw).per = sfr->basic.P;
+ psw_bits(regs.psw).as = sfr->basic.AS;
/*
* Use the hardware provided configuration level to decide if the
struct perf_pmu_events_attr *pmu_attr;
pmu_attr = container_of(attr, struct perf_pmu_events_attr, attr);
- return sprintf(page, "event=0x%04llx,name=%s\n",
- pmu_attr->id, attr->attr.name);
+ return sprintf(page, "event=0x%04llx\n", pmu_attr->id);
}
asmlinkage void ret_from_fork(void) asm ("ret_from_fork");
-/*
- * Return saved PC of a blocked thread. used in kernel/sched.
- * resume in entry.S does not create a new stack frame, it
- * just stores the registers %r6-%r15 to the frame given by
- * schedule. We want to return the address of the caller of
- * schedule, so we have to walk the backchain one time to
- * find the frame schedule() store its return address.
- */
-unsigned long thread_saved_pc(struct task_struct *tsk)
-{
- struct stack_frame *sf, *low, *high;
-
- if (!tsk || !task_stack_page(tsk))
- return 0;
- low = task_stack_page(tsk);
- high = (struct stack_frame *) task_pt_regs(tsk);
- sf = (struct stack_frame *) tsk->thread.ksp;
- if (sf <= low || sf > high)
- return 0;
- sf = (struct stack_frame *) sf->back_chain;
- if (sf <= low || sf > high)
- return 0;
- return sf->gprs[8];
-}
-
extern void kernel_thread_starter(void);
/*
return -ENODEV;
if (!data)
return -ENODATA;
+ if (target == current)
+ save_gs_cb(data);
return user_regset_copyout(&pos, &count, &kbuf, &ubuf,
data, 0, sizeof(struct gs_cb));
}
const void *kbuf, const void __user *ubuf)
{
struct gs_cb *data = target->thread.gs_cb;
+ int rc;
if (!MACHINE_HAS_GS)
return -ENODEV;
data = kzalloc(sizeof(*data), GFP_KERNEL);
if (!data)
return -ENOMEM;
+ data->gsd = 25;
target->thread.gs_cb = data;
+ if (target == current)
+ __ctl_set_bit(2, 4);
+ } else if (target == current) {
+ save_gs_cb(data);
}
- return user_regset_copyin(&pos, &count, &kbuf, &ubuf,
- data, 0, sizeof(struct gs_cb));
+ rc = user_regset_copyin(&pos, &count, &kbuf, &ubuf,
+ data, 0, sizeof(struct gs_cb));
+ if (target == current)
+ restore_gs_cb(data);
+ return rc;
}
static int s390_gs_bc_get(struct task_struct *target,
#include <linux/err.h>
#include <linux/spinlock.h>
#include <linux/kernel_stat.h>
+#include <linux/kmemleak.h>
#include <linux/delay.h>
#include <linux/interrupt.h>
#include <linux/irqflags.h>
kmem_cache_alloc(pcpu_mcesa_cache, GFP_KERNEL);
if (!mcesa_origin)
goto out;
+ /* The pointer is stored with mcesa_bits ORed in */
+ kmemleak_not_leak((void *) mcesa_origin);
mcesa_bits = MACHINE_HAS_GS ? 11 : 0;
}
} else {
static void print_uuid(struct seq_file *m, int i, struct sysinfo_3_2_2 *info)
{
- if (!memcmp(&info->vm[i].uuid, &NULL_UUID_BE, sizeof(uuid_be)))
+ if (uuid_is_null(&info->vm[i].uuid))
return;
seq_printf(m, "VM%02d UUID: %pUb\n", i, &info->vm[i].uuid);
}
goto out_unlock;
memset(&stp_sync, 0, sizeof(stp_sync));
- get_online_cpus();
+ cpus_read_lock();
atomic_set(&stp_sync.cpus, num_online_cpus() - 1);
- stop_machine(stp_sync_clock, &stp_sync, cpu_online_mask);
- put_online_cpus();
+ stop_machine_cpuslocked(stp_sync_clock, &stp_sync, cpu_online_mask);
+ cpus_read_unlock();
if (!check_sync_clock())
/*
#include <linux/mm.h>
#include <linux/slab.h>
#include <linux/uaccess.h>
+#include <linux/cpu.h>
#include <asm/fpu/api.h>
#include "entry.h"
int arch_uprobe_pre_xol(struct arch_uprobe *auprobe, struct pt_regs *regs)
{
- if (psw_bits(regs->psw).eaba == PSW_AMODE_24BIT)
+ if (psw_bits(regs->psw).eaba == PSW_BITS_AMODE_24BIT)
return -EINVAL;
- if (!is_compat_task() && psw_bits(regs->psw).eaba == PSW_AMODE_31BIT)
+ if (!is_compat_task() && psw_bits(regs->psw).eaba == PSW_BITS_AMODE_31BIT)
return -EINVAL;
clear_pt_regs_flag(regs, PIF_PER_TRAP);
- auprobe->saved_per = psw_bits(regs->psw).r;
+ auprobe->saved_per = psw_bits(regs->psw).per;
auprobe->saved_int_code = regs->int_code;
regs->int_code = UPROBE_TRAP_NR;
regs->psw.addr = current->utask->xol_vaddr;
clear_tsk_thread_flag(current, TIF_UPROBE_SINGLESTEP);
update_cr_regs(current);
- psw_bits(regs->psw).r = auprobe->saved_per;
+ psw_bits(regs->psw).per = auprobe->saved_per;
regs->int_code = auprobe->saved_int_code;
if (fixup & FIXUP_PSW_NORMAL)
bool arch_uprobe_skip_sstep(struct arch_uprobe *auprobe, struct pt_regs *regs)
{
- if ((psw_bits(regs->psw).eaba == PSW_AMODE_24BIT) ||
- ((psw_bits(regs->psw).eaba == PSW_AMODE_31BIT) &&
+ if ((psw_bits(regs->psw).eaba == PSW_BITS_AMODE_24BIT) ||
+ ((psw_bits(regs->psw).eaba == PSW_BITS_AMODE_31BIT) &&
!is_compat_task())) {
regs->psw.addr = __rewind_psw(regs->psw, UPROBE_SWBP_INSN_SIZE);
do_report_trap(regs, SIGILL, ILL_ILLADR, NULL);
*/
unsigned int __read_mostly vdso_enabled = 1;
+static int vdso_fault(const struct vm_special_mapping *sm,
+ struct vm_area_struct *vma, struct vm_fault *vmf)
+{
+ struct page **vdso_pagelist;
+ unsigned long vdso_pages;
+
+ vdso_pagelist = vdso64_pagelist;
+ vdso_pages = vdso64_pages;
+#ifdef CONFIG_COMPAT
+ if (is_compat_task()) {
+ vdso_pagelist = vdso32_pagelist;
+ vdso_pages = vdso32_pages;
+ }
+#endif
+
+ if (vmf->pgoff >= vdso_pages)
+ return VM_FAULT_SIGBUS;
+
+ vmf->page = vdso_pagelist[vmf->pgoff];
+ get_page(vmf->page);
+ return 0;
+}
+
+static int vdso_mremap(const struct vm_special_mapping *sm,
+ struct vm_area_struct *vma)
+{
+ unsigned long vdso_pages;
+
+ vdso_pages = vdso64_pages;
+#ifdef CONFIG_COMPAT
+ if (is_compat_task())
+ vdso_pages = vdso32_pages;
+#endif
+
+ if ((vdso_pages << PAGE_SHIFT) != vma->vm_end - vma->vm_start)
+ return -EINVAL;
+
+ if (WARN_ON_ONCE(current->mm != vma->vm_mm))
+ return -EFAULT;
+
+ current->mm->context.vdso_base = vma->vm_start;
+ return 0;
+}
+
+static const struct vm_special_mapping vdso_mapping = {
+ .name = "[vdso]",
+ .fault = vdso_fault,
+ .mremap = vdso_mremap,
+};
+
static int __init vdso_setup(char *s)
{
unsigned long val;
int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)
{
struct mm_struct *mm = current->mm;
- struct page **vdso_pagelist;
+ struct vm_area_struct *vma;
unsigned long vdso_pages;
unsigned long vdso_base;
int rc;
if (!uses_interp)
return 0;
- vdso_pagelist = vdso64_pagelist;
vdso_pages = vdso64_pages;
#ifdef CONFIG_COMPAT
- if (is_compat_task()) {
- vdso_pagelist = vdso32_pagelist;
+ if (is_compat_task())
vdso_pages = vdso32_pages;
- }
#endif
/*
* vDSO has a problem and was disabled, just don't "enable" it for
if (vdso_pages == 0)
return 0;
- current->mm->context.vdso_base = 0;
-
/*
* pick a base address for the vDSO in process space. We try to put
* it at vdso_base which is the "natural" base for it, but we might
goto out_up;
}
- /*
- * Put vDSO base into mm struct. We need to do this before calling
- * install_special_mapping or the perf counter mmap tracking code
- * will fail to recognise it as a vDSO (since arch_vma_name fails).
- */
- current->mm->context.vdso_base = vdso_base;
-
/*
* our vma flags don't have VM_WRITE so by default, the process
* isn't allowed to write those pages.
* It's fine to use that for setting breakpoints in the vDSO code
* pages though.
*/
- rc = install_special_mapping(mm, vdso_base, vdso_pages << PAGE_SHIFT,
- VM_READ|VM_EXEC|
- VM_MAYREAD|VM_MAYWRITE|VM_MAYEXEC,
- vdso_pagelist);
- if (rc)
- current->mm->context.vdso_base = 0;
+ vma = _install_special_mapping(mm, vdso_base, vdso_pages << PAGE_SHIFT,
+ VM_READ|VM_EXEC|
+ VM_MAYREAD|VM_MAYWRITE|VM_MAYEXEC,
+ &vdso_mapping);
+ if (IS_ERR(vma)) {
+ rc = PTR_ERR(vma);
+ goto out_up;
+ }
+
+ current->mm->context.vdso_base = vdso_base;
+ rc = 0;
+
out_up:
up_write(&mm->mmap_sem);
return rc;
}
-const char *arch_vma_name(struct vm_area_struct *vma)
-{
- if (vma->vm_mm && vma->vm_start == vma->vm_mm->context.vdso_base)
- return "[vdso]";
- return NULL;
-}
-
static int __init vdso_init(void)
{
int i;
return vtime;
}
-static void account_system_index_scaled(struct task_struct *p,
- u64 cputime, u64 scaled,
+static void account_system_index_scaled(struct task_struct *p, u64 cputime,
enum cpu_usage_stat index)
{
- p->stimescaled += cputime_to_nsecs(scaled);
+ p->stimescaled += cputime_to_nsecs(scale_vtime(cputime));
account_system_index_time(p, cputime_to_nsecs(cputime), index);
}
}
if (system)
- account_system_index_scaled(tsk, system, scale_vtime(system),
- CPUTIME_SYSTEM);
+ account_system_index_scaled(tsk, system, CPUTIME_SYSTEM);
if (hardirq)
- account_system_index_scaled(tsk, hardirq, scale_vtime(hardirq),
- CPUTIME_IRQ);
+ account_system_index_scaled(tsk, hardirq, CPUTIME_IRQ);
if (softirq)
- account_system_index_scaled(tsk, softirq, scale_vtime(softirq),
- CPUTIME_SOFTIRQ);
+ account_system_index_scaled(tsk, softirq, CPUTIME_SOFTIRQ);
steal = S390_lowcore.steal_timer;
if ((s64) steal > 0) {
int rc;
struct psw_bits psw = psw_bits(vcpu->arch.sie_block->gpsw);
- if (!psw.t) {
+ if (!psw.dat) {
asce->val = 0;
asce->r = 1;
return 0;
}
- if (mode == GACC_IFETCH)
- psw.as = psw.as == PSW_AS_HOME ? PSW_AS_HOME : PSW_AS_PRIMARY;
+ if ((mode == GACC_IFETCH) && (psw.as != PSW_BITS_AS_HOME))
+ psw.as = PSW_BITS_AS_PRIMARY;
switch (psw.as) {
- case PSW_AS_PRIMARY:
+ case PSW_BITS_AS_PRIMARY:
asce->val = vcpu->arch.sie_block->gcr[1];
return 0;
- case PSW_AS_SECONDARY:
+ case PSW_BITS_AS_SECONDARY:
asce->val = vcpu->arch.sie_block->gcr[7];
return 0;
- case PSW_AS_HOME:
+ case PSW_BITS_AS_HOME:
asce->val = vcpu->arch.sie_block->gcr[13];
return 0;
- case PSW_AS_ACCREG:
+ case PSW_BITS_AS_ACCREG:
rc = ar_translation(vcpu, asce, ar, mode);
if (rc > 0)
return trans_exc(vcpu, rc, ga, ar, mode, PROT_TYPE_ALC);
if (!ctlreg0.lap)
return 0;
- if (psw_bits(*psw).t && asce.p)
+ if (psw_bits(*psw).dat && asce.p)
return 0;
return 1;
}
return trans_exc(vcpu, PGM_PROTECTION, ga, ar, mode,
PROT_TYPE_LA);
ga &= PAGE_MASK;
- if (psw_bits(*psw).t) {
+ if (psw_bits(*psw).dat) {
rc = guest_translate(vcpu, ga, pages, asce, mode);
if (rc < 0)
return rc;
pages = vmalloc(nr_pages * sizeof(unsigned long));
if (!pages)
return -ENOMEM;
- need_ipte_lock = psw_bits(*psw).t && !asce.r;
+ need_ipte_lock = psw_bits(*psw).dat && !asce.r;
if (need_ipte_lock)
ipte_lock(vcpu);
rc = guest_page_range(vcpu, ga, ar, pages, nr_pages, asce, mode);
mode, PROT_TYPE_LA);
}
- if (psw_bits(*psw).t && !asce.r) { /* Use DAT? */
+ if (psw_bits(*psw).dat && !asce.r) { /* Use DAT? */
rc = guest_translate(vcpu, gva, gpa, asce, mode);
if (rc > 0)
return trans_exc(vcpu, rc, gva, 0, mode, PROT_TYPE_DAT);
ptr = asce.origin * 4096;
if (asce.r) {
*fake = 1;
+ ptr = 0;
asce.dt = ASCE_TYPE_REGION1;
}
switch (asce.dt) {
case ASCE_TYPE_REGION1:
- if (vaddr.rfx01 > asce.tl && !asce.r)
+ if (vaddr.rfx01 > asce.tl && !*fake)
return PGM_REGION_FIRST_TRANS;
break;
case ASCE_TYPE_REGION2:
union region1_table_entry rfte;
if (*fake) {
- /* offset in 16EB guest memory block */
- ptr = ptr + ((unsigned long) vaddr.rsx << 53UL);
+ ptr += (unsigned long) vaddr.rfx << 53;
rfte.val = ptr;
goto shadow_r2t;
}
union region2_table_entry rste;
if (*fake) {
- /* offset in 8PB guest memory block */
- ptr = ptr + ((unsigned long) vaddr.rtx << 42UL);
+ ptr += (unsigned long) vaddr.rsx << 42;
rste.val = ptr;
goto shadow_r3t;
}
union region3_table_entry rtte;
if (*fake) {
- /* offset in 4TB guest memory block */
- ptr = ptr + ((unsigned long) vaddr.sx << 31UL);
+ ptr += (unsigned long) vaddr.rtx << 31;
rtte.val = ptr;
goto shadow_sgt;
}
union segment_table_entry ste;
if (*fake) {
- /* offset in 2G guest memory block */
- ptr = ptr + ((unsigned long) vaddr.sx << 20UL);
+ ptr += (unsigned long) vaddr.sx << 20;
ste.val = ptr;
goto shadow_pgt;
}
{
psw_t *psw = &vcpu->arch.sie_block->gpsw;
- if (psw_bits(*psw).eaba == PSW_AMODE_64BIT)
+ if (psw_bits(*psw).eaba == PSW_BITS_AMODE_64BIT)
return ga;
- if (psw_bits(*psw).eaba == PSW_AMODE_31BIT)
+ if (psw_bits(*psw).eaba == PSW_BITS_AMODE_31BIT)
return ga & ((1UL << 31) - 1);
return ga & ((1UL << 24) - 1);
}
* instruction. Check primary and home space-switch-event
* controls. (theoretically home -> home produced no event)
*/
- if (((new_as == PSW_AS_HOME) ^ old_as_is_home(vcpu)) &&
- (pssec(vcpu) || hssec(vcpu)))
+ if (((new_as == PSW_BITS_AS_HOME) ^ old_as_is_home(vcpu)) &&
+ (pssec(vcpu) || hssec(vcpu)))
vcpu->arch.sie_block->iprcc = PGM_SPACE_SWITCH;
/*
* PT, PTI, PR, PC instruction operate on primary AS only. Check
* if the primary-space-switch-event control was or got set.
*/
- if (new_as == PSW_AS_PRIMARY && !old_as_is_home(vcpu) &&
+ if (new_as == PSW_BITS_AS_PRIMARY && !old_as_is_home(vcpu) &&
(pssec(vcpu) || old_ssec(vcpu)))
vcpu->arch.sie_block->iprcc = PGM_SPACE_SWITCH;
}
if (!vcpu)
goto out;
+ BUILD_BUG_ON(sizeof(struct sie_page) != 4096);
sie_page = (struct sie_page *) get_zeroed_page(GFP_KERNEL);
if (!sie_page)
goto out_free_cpu;
}
}
if (m3 & SSKE_MB) {
- if (psw_bits(vcpu->arch.sie_block->gpsw).eaba == PSW_AMODE_64BIT)
+ if (psw_bits(vcpu->arch.sie_block->gpsw).eaba == PSW_BITS_AMODE_64BIT)
vcpu->run->s.regs.gprs[reg2] &= ~PAGE_MASK;
else
vcpu->run->s.regs.gprs[reg2] &= ~0xfffff000UL;
static int handle_ipte_interlock(struct kvm_vcpu *vcpu)
{
vcpu->stat.instruction_ipte_interlock++;
- if (psw_bits(vcpu->arch.sie_block->gpsw).p)
+ if (psw_bits(vcpu->arch.sie_block->gpsw).pstate)
return kvm_s390_inject_program_int(vcpu, PGM_PRIVILEGED_OP);
wait_event(vcpu->kvm->arch.ipte_wq, !ipte_lock_held(vcpu));
kvm_s390_retry_instr(vcpu);
/* only support 2G frame size if EDAT2 is available and we are
not in 24-bit addressing mode */
if (!test_kvm_facility(vcpu->kvm, 78) ||
- psw_bits(vcpu->arch.sie_block->gpsw).eaba == PSW_AMODE_24BIT)
+ psw_bits(vcpu->arch.sie_block->gpsw).eaba == PSW_BITS_AMODE_24BIT)
return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION);
end = (start + (1UL << 31)) & ~((1UL << 31) - 1);
break;
start += PAGE_SIZE;
}
if (vcpu->run->s.regs.gprs[reg1] & PFMF_FSC) {
- if (psw_bits(vcpu->arch.sie_block->gpsw).eaba == PSW_AMODE_64BIT) {
+ if (psw_bits(vcpu->arch.sie_block->gpsw).eaba == PSW_BITS_AMODE_64BIT) {
vcpu->run->s.regs.gprs[reg2] = end;
} else {
vcpu->run->s.regs.gprs[reg2] &= ~0xffffffffUL;
}
static void walk_pud_level(struct seq_file *m, struct pg_state *st,
- pgd_t *pgd, unsigned long addr)
+ p4d_t *p4d, unsigned long addr)
{
unsigned int prot;
pud_t *pud;
for (i = 0; i < PTRS_PER_PUD && addr < max_addr; i++) {
st->current_address = addr;
- pud = pud_offset(pgd, addr);
+ pud = pud_offset(p4d, addr);
if (!pud_none(*pud))
if (pud_large(*pud)) {
prot = pud_val(*pud) &
}
}
+static void walk_p4d_level(struct seq_file *m, struct pg_state *st,
+ pgd_t *pgd, unsigned long addr)
+{
+ p4d_t *p4d;
+ int i;
+
+ for (i = 0; i < PTRS_PER_P4D && addr < max_addr; i++) {
+ st->current_address = addr;
+ p4d = p4d_offset(pgd, addr);
+ if (!p4d_none(*p4d))
+ walk_pud_level(m, st, p4d, addr);
+ else
+ note_page(m, st, _PAGE_INVALID, 2);
+ addr += P4D_SIZE;
+ }
+}
+
static void walk_pgd_level(struct seq_file *m)
{
unsigned long addr = 0;
st.current_address = addr;
pgd = pgd_offset_k(addr);
if (!pgd_none(*pgd))
- walk_pud_level(m, &st, pgd, addr);
+ walk_p4d_level(m, &st, pgd, addr);
else
note_page(m, &st, _PAGE_INVALID, 1);
addr += PGDIR_SIZE;
static void dump_pagetable(unsigned long asce, unsigned long address)
{
- unsigned long *table = __va(asce & PAGE_MASK);
+ unsigned long *table = __va(asce & _ASCE_ORIGIN);
pr_alert("AS:%016lx ", asce);
switch (asce & _ASCE_TYPE_MASK) {
struct radix_tree_iter iter;
unsigned long indices[16];
unsigned long index;
- void **slot;
+ void __rcu **slot;
int i, nr;
/* A radix tree is freed by deleting all of its entries */
struct radix_tree_iter iter;
unsigned long indices[16];
unsigned long index;
- void **slot;
+ void __rcu **slot;
int i, nr;
/* A radix tree is freed by deleting all of its entries */
unsigned long *table;
spinlock_t *ptl;
pgd_t *pgd;
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
int rc;
mm = gmap->mm;
pgd = pgd_offset(mm, vmaddr);
VM_BUG_ON(pgd_none(*pgd));
- pud = pud_offset(pgd, vmaddr);
+ p4d = p4d_offset(pgd, vmaddr);
+ VM_BUG_ON(p4d_none(*p4d));
+ pud = pud_offset(p4d, vmaddr);
VM_BUG_ON(pud_none(*pud));
/* large puds cannot yet be handled */
if (pud_large(*pud))
static inline void gmap_insert_rmap(struct gmap *sg, unsigned long vmaddr,
struct gmap_rmap *rmap)
{
- void **slot;
+ void __rcu **slot;
BUG_ON(!gmap_is_shadow(sg));
slot = radix_tree_lookup_slot(&sg->host_to_rmap, vmaddr >> PAGE_SHIFT);
return 1;
}
-static inline int gup_pud_range(pgd_t *pgdp, pgd_t pgd, unsigned long addr,
+static inline int gup_pud_range(p4d_t *p4dp, p4d_t p4d, unsigned long addr,
unsigned long end, int write, struct page **pages, int *nr)
{
unsigned long next;
pud_t *pudp, pud;
- pudp = (pud_t *) pgdp;
- if ((pgd_val(pgd) & _REGION_ENTRY_TYPE_MASK) == _REGION_ENTRY_TYPE_R2)
- pudp = (pud_t *) pgd_deref(pgd);
+ pudp = (pud_t *) p4dp;
+ if ((p4d_val(p4d) & _REGION_ENTRY_TYPE_MASK) == _REGION_ENTRY_TYPE_R2)
+ pudp = (pud_t *) p4d_deref(p4d);
pudp += pud_index(addr);
do {
pud = *pudp;
return 1;
}
+static inline int gup_p4d_range(pgd_t *pgdp, pgd_t pgd, unsigned long addr,
+ unsigned long end, int write, struct page **pages, int *nr)
+{
+ unsigned long next;
+ p4d_t *p4dp, p4d;
+
+ p4dp = (p4d_t *) pgdp;
+ if ((pgd_val(pgd) & _REGION_ENTRY_TYPE_MASK) == _REGION_ENTRY_TYPE_R1)
+ p4dp = (p4d_t *) pgd_deref(pgd);
+ p4dp += p4d_index(addr);
+ do {
+ p4d = *p4dp;
+ barrier();
+ next = p4d_addr_end(addr, end);
+ if (p4d_none(p4d))
+ return 0;
+ if (!gup_pud_range(p4dp, p4d, addr, next, write, pages, nr))
+ return 0;
+ } while (p4dp++, addr = next, addr != end);
+
+ return 1;
+}
+
/*
* Like get_user_pages_fast() except its IRQ-safe in that it won't fall
* back to the regular GUP.
next = pgd_addr_end(addr, end);
if (pgd_none(pgd))
break;
- if (!gup_pud_range(pgdp, pgd, addr, next, write, pages, &nr))
+ if (!gup_p4d_range(pgdp, pgd, addr, next, write, pages, &nr))
break;
} while (pgdp++, addr = next, addr != end);
local_irq_restore(flags);
unsigned long addr, unsigned long sz)
{
pgd_t *pgdp;
+ p4d_t *p4dp;
pud_t *pudp;
pmd_t *pmdp = NULL;
pgdp = pgd_offset(mm, addr);
- pudp = pud_alloc(mm, pgdp, addr);
- if (pudp) {
- if (sz == PUD_SIZE)
- return (pte_t *) pudp;
- else if (sz == PMD_SIZE)
- pmdp = pmd_alloc(mm, pudp, addr);
+ p4dp = p4d_alloc(mm, pgdp, addr);
+ if (p4dp) {
+ pudp = pud_alloc(mm, p4dp, addr);
+ if (pudp) {
+ if (sz == PUD_SIZE)
+ return (pte_t *) pudp;
+ else if (sz == PMD_SIZE)
+ pmdp = pmd_alloc(mm, pudp, addr);
+ }
}
return (pte_t *) pmdp;
}
pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr)
{
pgd_t *pgdp;
+ p4d_t *p4dp;
pud_t *pudp;
pmd_t *pmdp = NULL;
pgdp = pgd_offset(mm, addr);
if (pgd_present(*pgdp)) {
- pudp = pud_offset(pgdp, addr);
- if (pud_present(*pudp)) {
- if (pud_large(*pudp))
- return (pte_t *) pudp;
- pmdp = pmd_offset(pudp, addr);
+ p4dp = p4d_offset(pgdp, addr);
+ if (p4d_present(*p4dp)) {
+ pudp = pud_offset(p4dp, addr);
+ if (pud_present(*pudp)) {
+ if (pud_large(*pudp))
+ return (pte_t *) pudp;
+ pmdp = pmd_offset(pudp, addr);
+ }
}
}
return (pte_t *) pmdp;
{
unsigned long max_zone_pfns[MAX_NR_ZONES];
unsigned long pgd_type, asce_bits;
+ psw_t psw;
init_mm.pgd = swapper_pg_dir;
if (VMALLOC_END > (1UL << 42)) {
__ctl_load(S390_lowcore.kernel_asce, 1, 1);
__ctl_load(S390_lowcore.kernel_asce, 7, 7);
__ctl_load(S390_lowcore.kernel_asce, 13, 13);
- __arch_local_irq_stosm(0x04);
+ psw.mask = __extract_psw();
+ psw_bits(psw).dat = 1;
+ psw_bits(psw).as = PSW_BITS_AS_HOME;
+ __load_psw_mask(psw.mask);
sparse_memory_present_with_active_regions(MAX_NUMNODES);
sparse_init();
check_asce_limit:
if (addr + len > current->mm->context.asce_limit) {
- rc = crst_table_upgrade(mm);
+ rc = crst_table_upgrade(mm, addr + len);
if (rc)
return (unsigned long) rc;
}
check_asce_limit:
if (addr + len > current->mm->context.asce_limit) {
- rc = crst_table_upgrade(mm);
+ rc = crst_table_upgrade(mm, addr + len);
if (rc)
return (unsigned long) rc;
}
pgt_set((unsigned long *)pudp, pud_val(new), addr, CRDTE_DTT_REGION3);
}
-static int walk_pud_level(pgd_t *pgd, unsigned long addr, unsigned long end,
+static int walk_pud_level(p4d_t *p4d, unsigned long addr, unsigned long end,
unsigned long flags)
{
unsigned long next;
pud_t *pudp;
int rc = 0;
- pudp = pud_offset(pgd, addr);
+ pudp = pud_offset(p4d, addr);
do {
if (pud_none(*pudp))
return -EINVAL;
return rc;
}
+static int walk_p4d_level(pgd_t *pgd, unsigned long addr, unsigned long end,
+ unsigned long flags)
+{
+ unsigned long next;
+ p4d_t *p4dp;
+ int rc = 0;
+
+ p4dp = p4d_offset(pgd, addr);
+ do {
+ if (p4d_none(*p4dp))
+ return -EINVAL;
+ next = p4d_addr_end(addr, end);
+ rc = walk_pud_level(p4dp, addr, next, flags);
+ p4dp++;
+ addr = next;
+ cond_resched();
+ } while (addr < end && !rc);
+ return rc;
+}
+
static DEFINE_MUTEX(cpa_mutex);
static int change_page_attr(unsigned long addr, unsigned long end,
if (pgd_none(*pgdp))
break;
next = pgd_addr_end(addr, end);
- rc = walk_pud_level(pgdp, addr, next, flags);
+ rc = walk_p4d_level(pgdp, addr, next, flags);
if (rc)
break;
cond_resched();
unsigned long address;
int nr, i, j;
pgd_t *pgd;
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
pte_t *pte;
for (i = 0; i < numpages;) {
address = page_to_phys(page + i);
pgd = pgd_offset_k(address);
- pud = pud_offset(pgd, address);
+ p4d = p4d_offset(pgd, address);
+ pud = pud_offset(p4d, address);
pmd = pmd_offset(pud, address);
pte = pte_offset_kernel(pmd, address);
nr = (unsigned long)pte >> ilog2(sizeof(long));
__tlb_flush_local();
}
-int crst_table_upgrade(struct mm_struct *mm)
+int crst_table_upgrade(struct mm_struct *mm, unsigned long end)
{
unsigned long *table, *pgd;
+ int rc, notify;
- /* upgrade should only happen from 3 to 4 levels */
- BUG_ON(mm->context.asce_limit != (1UL << 42));
-
- table = crst_table_alloc(mm);
- if (!table)
+ /* upgrade should only happen from 3 to 4, 3 to 5, or 4 to 5 levels */
+ BUG_ON(mm->context.asce_limit < (1UL << 42));
+ if (end >= TASK_SIZE_MAX)
return -ENOMEM;
-
- spin_lock_bh(&mm->page_table_lock);
- pgd = (unsigned long *) mm->pgd;
- crst_table_init(table, _REGION2_ENTRY_EMPTY);
- pgd_populate(mm, (pgd_t *) table, (pud_t *) pgd);
- mm->pgd = (pgd_t *) table;
- mm->context.asce_limit = 1UL << 53;
- mm->context.asce = __pa(mm->pgd) | _ASCE_TABLE_LENGTH |
- _ASCE_USER_BITS | _ASCE_TYPE_REGION2;
- spin_unlock_bh(&mm->page_table_lock);
-
- on_each_cpu(__crst_table_upgrade, mm, 0);
- return 0;
+ rc = 0;
+ notify = 0;
+ while (mm->context.asce_limit < end) {
+ table = crst_table_alloc(mm);
+ if (!table) {
+ rc = -ENOMEM;
+ break;
+ }
+ spin_lock_bh(&mm->page_table_lock);
+ pgd = (unsigned long *) mm->pgd;
+ if (mm->context.asce_limit == (1UL << 42)) {
+ crst_table_init(table, _REGION2_ENTRY_EMPTY);
+ p4d_populate(mm, (p4d_t *) table, (pud_t *) pgd);
+ mm->pgd = (pgd_t *) table;
+ mm->context.asce_limit = 1UL << 53;
+ mm->context.asce = __pa(mm->pgd) | _ASCE_TABLE_LENGTH |
+ _ASCE_USER_BITS | _ASCE_TYPE_REGION2;
+ } else {
+ crst_table_init(table, _REGION1_ENTRY_EMPTY);
+ pgd_populate(mm, (pgd_t *) table, (p4d_t *) pgd);
+ mm->pgd = (pgd_t *) table;
+ mm->context.asce_limit = -PAGE_SIZE;
+ mm->context.asce = __pa(mm->pgd) | _ASCE_TABLE_LENGTH |
+ _ASCE_USER_BITS | _ASCE_TYPE_REGION1;
+ }
+ notify = 1;
+ spin_unlock_bh(&mm->page_table_lock);
+ }
+ if (notify)
+ on_each_cpu(__crst_table_upgrade, mm, 0);
+ return rc;
}
void crst_table_downgrade(struct mm_struct *mm)
struct page *page = pfn_to_page(__pa(table) >> PAGE_SHIFT);
switch (mask) {
- case 0: /* pmd or pud */
+ case 0: /* pmd, pud, or p4d */
free_pages((unsigned long) table, 2);
break;
case 1: /* lower 2K of a 4K page table */
{
spinlock_t *ptl;
pgd_t *pgd;
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
pgste_t pgste;
bool dirty;
pgd = pgd_offset(mm, addr);
- pud = pud_alloc(mm, pgd, addr);
+ p4d = p4d_alloc(mm, pgd, addr);
+ if (!p4d)
+ return false;
+ pud = pud_alloc(mm, p4d, addr);
if (!pud)
return false;
pmd = pmd_alloc(mm, pud, addr);
return (void *) memblock_alloc(size, size);
}
+static inline p4d_t *vmem_p4d_alloc(void)
+{
+ p4d_t *p4d = NULL;
+
+ p4d = vmem_alloc_pages(2);
+ if (!p4d)
+ return NULL;
+ clear_table((unsigned long *) p4d, _REGION2_ENTRY_EMPTY, PAGE_SIZE * 4);
+ return p4d;
+}
+
static inline pud_t *vmem_pud_alloc(void)
{
pud_t *pud = NULL;
unsigned long end = start + size;
unsigned long address = start;
pgd_t *pg_dir;
+ p4d_t *p4_dir;
pud_t *pu_dir;
pmd_t *pm_dir;
pte_t *pt_dir;
while (address < end) {
pg_dir = pgd_offset_k(address);
if (pgd_none(*pg_dir)) {
+ p4_dir = vmem_p4d_alloc();
+ if (!p4_dir)
+ goto out;
+ pgd_populate(&init_mm, pg_dir, p4_dir);
+ }
+ p4_dir = p4d_offset(pg_dir, address);
+ if (p4d_none(*p4_dir)) {
pu_dir = vmem_pud_alloc();
if (!pu_dir)
goto out;
- pgd_populate(&init_mm, pg_dir, pu_dir);
+ p4d_populate(&init_mm, p4_dir, pu_dir);
}
- pu_dir = pud_offset(pg_dir, address);
+ pu_dir = pud_offset(p4_dir, address);
if (MACHINE_HAS_EDAT2 && pud_none(*pu_dir) && address &&
!(address & ~PUD_MASK) && (address + PUD_SIZE <= end) &&
!debug_pagealloc_enabled()) {
unsigned long end = start + size;
unsigned long address = start;
pgd_t *pg_dir;
+ p4d_t *p4_dir;
pud_t *pu_dir;
pmd_t *pm_dir;
pte_t *pt_dir;
address += PGDIR_SIZE;
continue;
}
- pu_dir = pud_offset(pg_dir, address);
+ p4_dir = p4d_offset(pg_dir, address);
+ if (p4d_none(*p4_dir)) {
+ address += P4D_SIZE;
+ continue;
+ }
+ pu_dir = pud_offset(p4_dir, address);
if (pud_none(*pu_dir)) {
address += PUD_SIZE;
continue;
unsigned long pgt_prot, sgt_prot;
unsigned long address = start;
pgd_t *pg_dir;
+ p4d_t *p4_dir;
pud_t *pu_dir;
pmd_t *pm_dir;
pte_t *pt_dir;
for (address = start; address < end;) {
pg_dir = pgd_offset_k(address);
if (pgd_none(*pg_dir)) {
+ p4_dir = vmem_p4d_alloc();
+ if (!p4_dir)
+ goto out;
+ pgd_populate(&init_mm, pg_dir, p4_dir);
+ }
+
+ p4_dir = p4d_offset(pg_dir, address);
+ if (p4d_none(*p4_dir)) {
pu_dir = vmem_pud_alloc();
if (!pu_dir)
goto out;
- pgd_populate(&init_mm, pg_dir, pu_dir);
+ p4d_populate(&init_mm, p4_dir, pu_dir);
}
- pu_dir = pud_offset(pg_dir, address);
+ pu_dir = pud_offset(p4_dir, address);
if (pud_none(*pu_dir)) {
pm_dir = vmem_pmd_alloc();
if (!pm_dir)
return zdev;
}
+void zpci_remove_reserved_devices(void)
+{
+ struct zpci_dev *tmp, *zdev;
+ enum zpci_state state;
+ LIST_HEAD(remove);
+
+ spin_lock(&zpci_list_lock);
+ list_for_each_entry_safe(zdev, tmp, &zpci_list, entry) {
+ if (zdev->state == ZPCI_FN_STATE_STANDBY &&
+ !clp_get_state(zdev->fid, &state) &&
+ state == ZPCI_FN_STATE_RESERVED)
+ list_move_tail(&zdev->entry, &remove);
+ }
+ spin_unlock(&zpci_list_lock);
+
+ list_for_each_entry_safe(zdev, tmp, &remove, entry)
+ zpci_remove_device(zdev);
+}
+
static struct zpci_dev *get_zdev_by_bus(struct pci_bus *bus)
{
return (bus && bus->sysdata) ? (struct zpci_dev *) bus->sysdata : NULL;
{
u64 req = ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_REG_INT);
struct zpci_fib fib = {0};
+ u8 status;
fib.isc = PCI_ISC;
fib.sum = 1; /* enable summary notifications */
fib.aisb = (unsigned long) zpci_aisb_iv->vector + (zdev->aisb/64)*8;
fib.aisbo = zdev->aisb & 63;
- return zpci_mod_fc(req, &fib);
+ return zpci_mod_fc(req, &fib, &status) ? -EIO : 0;
}
-struct mod_pci_args {
- u64 base;
- u64 limit;
- u64 iota;
- u64 fmb_addr;
-};
-
-static int mod_pci(struct zpci_dev *zdev, int fn, u8 dmaas, struct mod_pci_args *args)
+/* Modify PCI: Unregister adapter interruptions */
+static int zpci_clear_airq(struct zpci_dev *zdev)
{
- u64 req = ZPCI_CREATE_REQ(zdev->fh, dmaas, fn);
+ u64 req = ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_DEREG_INT);
struct zpci_fib fib = {0};
+ u8 cc, status;
- fib.pba = args->base;
- fib.pal = args->limit;
- fib.iota = args->iota;
- fib.fmb_addr = args->fmb_addr;
+ cc = zpci_mod_fc(req, &fib, &status);
+ if (cc == 3 || (cc == 1 && status == 24))
+ /* Function already gone or IRQs already deregistered. */
+ cc = 0;
- return zpci_mod_fc(req, &fib);
+ return cc ? -EIO : 0;
}
/* Modify PCI: Register I/O address translation parameters */
int zpci_register_ioat(struct zpci_dev *zdev, u8 dmaas,
u64 base, u64 limit, u64 iota)
{
- struct mod_pci_args args = { base, limit, iota, 0 };
+ u64 req = ZPCI_CREATE_REQ(zdev->fh, dmaas, ZPCI_MOD_FC_REG_IOAT);
+ struct zpci_fib fib = {0};
+ u8 status;
WARN_ON_ONCE(iota & 0x3fff);
- args.iota |= ZPCI_IOTA_RTTO_FLAG;
- return mod_pci(zdev, ZPCI_MOD_FC_REG_IOAT, dmaas, &args);
+ fib.pba = base;
+ fib.pal = limit;
+ fib.iota = iota | ZPCI_IOTA_RTTO_FLAG;
+ return zpci_mod_fc(req, &fib, &status) ? -EIO : 0;
}
/* Modify PCI: Unregister I/O address translation parameters */
int zpci_unregister_ioat(struct zpci_dev *zdev, u8 dmaas)
{
- struct mod_pci_args args = { 0, 0, 0, 0 };
-
- return mod_pci(zdev, ZPCI_MOD_FC_DEREG_IOAT, dmaas, &args);
-}
-
-/* Modify PCI: Unregister adapter interruptions */
-static int zpci_clear_airq(struct zpci_dev *zdev)
-{
- struct mod_pci_args args = { 0, 0, 0, 0 };
+ u64 req = ZPCI_CREATE_REQ(zdev->fh, dmaas, ZPCI_MOD_FC_DEREG_IOAT);
+ struct zpci_fib fib = {0};
+ u8 cc, status;
- return mod_pci(zdev, ZPCI_MOD_FC_DEREG_INT, 0, &args);
+ cc = zpci_mod_fc(req, &fib, &status);
+ if (cc == 3) /* Function already gone. */
+ cc = 0;
+ return cc ? -EIO : 0;
}
/* Modify PCI: Set PCI function measurement parameters */
int zpci_fmb_enable_device(struct zpci_dev *zdev)
{
- struct mod_pci_args args = { 0, 0, 0, 0 };
+ u64 req = ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_SET_MEASURE);
+ struct zpci_fib fib = {0};
+ u8 cc, status;
if (zdev->fmb || sizeof(*zdev->fmb) < zdev->fmb_length)
return -EINVAL;
atomic64_set(&zdev->mapped_pages, 0);
atomic64_set(&zdev->unmapped_pages, 0);
- args.fmb_addr = virt_to_phys(zdev->fmb);
- return mod_pci(zdev, ZPCI_MOD_FC_SET_MEASURE, 0, &args);
+ fib.fmb_addr = virt_to_phys(zdev->fmb);
+ cc = zpci_mod_fc(req, &fib, &status);
+ if (cc) {
+ kmem_cache_free(zdev_fmb_cache, zdev->fmb);
+ zdev->fmb = NULL;
+ }
+ return cc ? -EIO : 0;
}
/* Modify PCI: Disable PCI function measurement */
int zpci_fmb_disable_device(struct zpci_dev *zdev)
{
- struct mod_pci_args args = { 0, 0, 0, 0 };
- int rc;
+ u64 req = ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_SET_MEASURE);
+ struct zpci_fib fib = {0};
+ u8 cc, status;
if (!zdev->fmb)
return -EINVAL;
/* Function measurement is disabled if fmb address is zero */
- rc = mod_pci(zdev, ZPCI_MOD_FC_SET_MEASURE, 0, &args);
+ cc = zpci_mod_fc(req, &fib, &status);
+ if (cc == 3) /* Function already gone. */
+ cc = 0;
- kmem_cache_free(zdev_fmb_cache, zdev->fmb);
- zdev->fmb = NULL;
- return rc;
+ if (!cc) {
+ kmem_cache_free(zdev_fmb_cache, zdev->fmb);
+ zdev->fmb = NULL;
+ }
+ return cc ? -EIO : 0;
}
static int zpci_cfg_load(struct zpci_dev *zdev, int offset, u32 *val, u8 len)
struct msi_msg msg;
int rc, irq;
+ zdev->aisb = -1UL;
if (type == PCI_CAP_ID_MSI && nvec > 1)
return 1;
msi_vecs = min_t(unsigned int, nvec, zdev->max_msi);
/* Allocate adapter summary indicator bit */
- rc = -EIO;
aisb = airq_iv_alloc_bit(zpci_aisb_iv);
if (aisb == -1UL)
- goto out;
+ return -EIO;
zdev->aisb = aisb;
/* Create adapter interrupt vector */
- rc = -ENOMEM;
zdev->aibv = airq_iv_create(msi_vecs, AIRQ_IV_DATA | AIRQ_IV_BITLOCK);
if (!zdev->aibv)
- goto out_si;
+ return -ENOMEM;
/* Wire up shortcut pointer */
zpci_aibv[aisb] = zdev->aibv;
rc = -EIO;
irq = irq_alloc_desc(0); /* Alloc irq on node 0 */
if (irq < 0)
- goto out_msi;
+ return -ENOMEM;
rc = irq_set_msi_desc(irq, msi);
if (rc)
- goto out_msi;
+ return rc;
irq_set_chip_and_handler(irq, &zpci_irq_chip,
handle_simple_irq);
msg.data = hwirq;
/* Enable adapter interrupts */
rc = zpci_set_airq(zdev);
if (rc)
- goto out_msi;
+ return rc;
return (msi_vecs == nvec) ? 0 : msi_vecs;
-
-out_msi:
- for_each_pci_msi_entry(msi, pdev) {
- if (hwirq-- == 0)
- break;
- irq_set_msi_desc(msi->irq, NULL);
- irq_free_desc(msi->irq);
- msi->msg.address_lo = 0;
- msi->msg.address_hi = 0;
- msi->msg.data = 0;
- msi->irq = 0;
- }
- zpci_aibv[aisb] = NULL;
- airq_iv_release(zdev->aibv);
-out_si:
- airq_iv_free_bit(zpci_aisb_iv, aisb);
-out:
- return rc;
}
void arch_teardown_msi_irqs(struct pci_dev *pdev)
/* Release MSI interrupts */
for_each_pci_msi_entry(msi, pdev) {
+ if (!msi->irq)
+ continue;
if (msi->msi_attrib.is_msix)
__pci_msix_desc_mask_irq(msi, 1);
else
msi->irq = 0;
}
- zpci_aibv[zdev->aisb] = NULL;
- airq_iv_release(zdev->aibv);
- airq_iv_free_bit(zpci_aisb_iv, zdev->aisb);
+ if (zdev->aisb != -1UL) {
+ zpci_aibv[zdev->aisb] = NULL;
+ airq_iv_free_bit(zpci_aisb_iv, zdev->aisb);
+ zdev->aisb = -1UL;
+ }
+ if (zdev->aibv) {
+ airq_iv_release(zdev->aibv);
+ zdev->aibv = NULL;
+ }
}
static void zpci_map_resources(struct pci_dev *pdev)
{
if (zpci_unique_uid) {
zdev->domain = (u16) zdev->uid;
+ if (zdev->domain >= ZPCI_NR_DEVICES)
+ return 0;
+
+ spin_lock(&zpci_domain_lock);
+ if (test_bit(zdev->domain, zpci_domain)) {
+ spin_unlock(&zpci_domain_lock);
+ return -EEXIST;
+ }
+ set_bit(zdev->domain, zpci_domain);
+ spin_unlock(&zpci_domain_lock);
return 0;
}
static void zpci_free_domain(struct zpci_dev *zdev)
{
- if (zpci_unique_uid)
+ if (zdev->domain >= ZPCI_NR_DEVICES)
return;
spin_lock(&zpci_domain_lock);
list_del(&zdev->entry);
spin_unlock(&zpci_list_lock);
+ zpci_dbg(3, "rem fid:%x\n", zdev->fid);
kfree(zdev);
}
return rc;
}
-void zpci_stop_device(struct zpci_dev *zdev)
+void zpci_remove_device(struct zpci_dev *zdev)
{
- zpci_dma_exit_device(zdev);
- /*
- * Note: SCLP disables fh via set-pci-fn so don't
- * do that here.
- */
+ if (!zdev->bus)
+ return;
+
+ pci_stop_root_bus(zdev->bus);
+ pci_remove_root_bus(zdev->bus);
}
-EXPORT_SYMBOL_GPL(zpci_stop_device);
int zpci_report_error(struct pci_dev *pdev,
struct zpci_report_error_header *report)
int clp_add_pci_device(u32 fid, u32 fh, int configured)
{
struct zpci_dev *zdev;
- int rc;
+ int rc = -ENOMEM;
zpci_dbg(3, "add fid:%x, fh:%x, c:%d\n", fid, fh, configured);
zdev = kzalloc(sizeof(*zdev), GFP_KERNEL);
if (!zdev)
- return -ENOMEM;
+ goto error;
zdev->fh = fh;
zdev->fid = fid;
return 0;
error:
+ zpci_dbg(0, "add fid:%x, rc:%d\n", fid, rc);
kfree(zdev);
return rc;
}
return rc;
}
-static int clp_list_pci(struct clp_req_rsp_list_pci *rrb,
- void (*cb)(struct clp_fh_list_entry *entry))
+static int clp_list_pci(struct clp_req_rsp_list_pci *rrb, void *data,
+ void (*cb)(struct clp_fh_list_entry *, void *))
{
u64 resume_token = 0;
int entries, i, rc;
resume_token = rrb->response.resume_token;
for (i = 0; i < entries; i++)
- cb(&rrb->response.fh_list[i]);
+ cb(&rrb->response.fh_list[i], data);
} while (resume_token);
out:
return rc;
}
-static void __clp_add(struct clp_fh_list_entry *entry)
-{
- if (!entry->vendor_id)
- return;
-
- clp_add_pci_device(entry->fid, entry->fh, entry->config_state);
-}
-
-static void __clp_rescan(struct clp_fh_list_entry *entry)
+static void __clp_add(struct clp_fh_list_entry *entry, void *data)
{
struct zpci_dev *zdev;
return;
zdev = get_zdev_by_fid(entry->fid);
- if (!zdev) {
+ if (!zdev)
clp_add_pci_device(entry->fid, entry->fh, entry->config_state);
- return;
- }
-
- if (!entry->config_state) {
- /*
- * The handle is already disabled, that means no iota/irq freeing via
- * the firmware interfaces anymore. Need to free resources manually
- * (DMA memory, debug, sysfs)...
- */
- zpci_stop_device(zdev);
- }
}
-static void __clp_update(struct clp_fh_list_entry *entry)
+static void __clp_update(struct clp_fh_list_entry *entry, void *data)
{
struct zpci_dev *zdev;
if (!rrb)
return -ENOMEM;
- rc = clp_list_pci(rrb, __clp_add);
+ rc = clp_list_pci(rrb, NULL, __clp_add);
clp_free_block(rrb);
return rc;
struct clp_req_rsp_list_pci *rrb;
int rc;
+ zpci_remove_reserved_devices();
+
rrb = clp_alloc_block(GFP_KERNEL);
if (!rrb)
return -ENOMEM;
- rc = clp_list_pci(rrb, __clp_rescan);
+ rc = clp_list_pci(rrb, NULL, __clp_add);
clp_free_block(rrb);
return rc;
if (!rrb)
return -ENOMEM;
- rc = clp_list_pci(rrb, __clp_update);
+ rc = clp_list_pci(rrb, NULL, __clp_update);
+
+ clp_free_block(rrb);
+ return rc;
+}
+
+struct clp_state_data {
+ u32 fid;
+ enum zpci_state state;
+};
+
+static void __clp_get_state(struct clp_fh_list_entry *entry, void *data)
+{
+ struct clp_state_data *sd = data;
+
+ if (entry->fid != sd->fid)
+ return;
+
+ sd->state = entry->config_state;
+}
+
+int clp_get_state(u32 fid, enum zpci_state *state)
+{
+ struct clp_req_rsp_list_pci *rrb;
+ struct clp_state_data sd = {fid, ZPCI_FN_STATE_RESERVED};
+ int rc;
+
+ rrb = clp_alloc_block(GFP_KERNEL);
+ if (!rrb)
+ return -ENOMEM;
+
+ rc = clp_list_pci(rrb, &sd, __clp_get_state);
+ if (!rc)
+ *state = sd.state;
clp_free_block(rrb);
return rc;
*/
WARN_ON(zdev->s390_domain);
- zpci_unregister_ioat(zdev, 0);
+ if (zpci_unregister_ioat(zdev, 0))
+ return;
+
dma_cleanup_tables(zdev->dma_table);
zdev->dma_table = NULL;
vfree(zdev->iommu_bitmap);
{
struct zpci_dev *zdev = get_zdev_by_fid(ccdf->fid);
struct pci_dev *pdev = NULL;
+ enum zpci_state state;
int ret;
if (zdev)
clp_add_pci_device(ccdf->fid, ccdf->fh, 0);
break;
case 0x0303: /* Deconfiguration requested */
+ if (!zdev)
+ break;
if (pdev)
pci_stop_and_remove_bus_device_locked(pdev);
zdev->state = ZPCI_FN_STATE_STANDBY;
break;
- case 0x0304: /* Configured -> Standby */
+ case 0x0304: /* Configured -> Standby|Reserved */
+ if (!zdev)
+ break;
if (pdev) {
/* Give the driver a hint that the function is
* already unusable. */
zdev->fh = ccdf->fh;
zpci_disable_device(zdev);
zdev->state = ZPCI_FN_STATE_STANDBY;
+ if (!clp_get_state(ccdf->fid, &state) &&
+ state == ZPCI_FN_STATE_RESERVED) {
+ zpci_remove_device(zdev);
+ }
break;
case 0x0306: /* 0x308 or 0x302 for multiple devices */
clp_rescan_pci_devices();
case 0x0308: /* Standby -> Reserved */
if (!zdev)
break;
- pci_stop_root_bus(zdev->bus);
- pci_remove_root_bus(zdev->bus);
+ zpci_remove_device(zdev);
break;
default:
break;
return cc;
}
-int zpci_mod_fc(u64 req, struct zpci_fib *fib)
+u8 zpci_mod_fc(u64 req, struct zpci_fib *fib, u8 *status)
{
- u8 cc, status;
+ u8 cc;
do {
- cc = __mpcifc(req, fib, &status);
+ cc = __mpcifc(req, fib, status);
if (cc == 2)
msleep(ZPCI_INSN_BUSY_DELAY);
} while (cc == 2);
if (cc)
- zpci_err_insn(cc, status, req, 0);
+ zpci_err_insn(cc, *status, req, 0);
- return (cc) ? -EIO : 0;
+ return cc;
}
/* Refresh PCI Translations */
18, /* long displacement facility */
#endif
#ifdef CONFIG_HAVE_MARCH_Z9_109_FEATURES
- 7, /* stfle */
- 17, /* message security assist */
21, /* extended-immediate facility */
25, /* store clock fast */
#endif
*/
extern void (*cpu_wait)(void);
-extern unsigned long thread_saved_pc(struct task_struct *tsk);
extern void start_thread(struct pt_regs *regs,
unsigned long pc, unsigned long sp);
extern unsigned long get_wchan(struct task_struct *p);
# UAPI Header export list
include include/uapi/asm-generic/Kbuild.asm
+
+generic-y += siginfo.h
+++ /dev/null
-#ifndef _ASM_SCORE_SIGINFO_H
-#define _ASM_SCORE_SIGINFO_H
-
-#include <asm-generic/siginfo.h>
-
-#endif /* _ASM_SCORE_SIGINFO_H */
return 1;
}
-unsigned long thread_saved_pc(struct task_struct *tsk)
-{
- return task_pt_regs(tsk)->cp0_epc;
-}
-
unsigned long get_wchan(struct task_struct *task)
{
if (!task || task == current || task->state == TASK_RUNNING)
bool "Board Described by Device Tree"
select OF
select OF_EARLY_FLATTREE
- select CLKSRC_OF
+ select TIMER_OF
select COMMON_CLK
select GENERIC_CALIBRATE_DELAY
help
static void __init sh_of_time_init(void)
{
pr_info("SH generic board support: scanning for clocksource devices\n");
- clocksource_probe();
+ timer_probe();
}
static void __init sh_of_setup(char **cmdline_p)
generic-y += sembuf.h
generic-y += serial.h
generic-y += shmbuf.h
-generic-y += siginfo.h
generic-y += sizes.h
generic-y += socket.h
generic-y += statfs.h
# UAPI Header export list
include include/uapi/asm-generic/Kbuild.asm
+
+generic-y += siginfo.h
#define TIOCGPKT _IOR('T', 0x38, int) /* Get packet mode state */
#define TIOCGPTLCK _IOR('T', 0x39, int) /* Get Pty lock state */
#define TIOCGEXCL _IOR('T', 0x40, int) /* Get exclusive mode state */
+#define TIOCGPTPEER _IOR('T', 0x41, int) /* Safely open the slave */
#define TIOCSERCONFIG _IO('T', 83) /* 0x5453 */
#define TIOCSERGWILD _IOR('T', 84, int) /* 0x5454 */
.current_ds = KERNEL_DS, \
}
-/* Return saved PC of a blocked thread. */
-unsigned long thread_saved_pc(struct task_struct *t);
-
/* Do necessary setup to start up a newly executed thread. */
static inline void start_thread(struct pt_regs * regs, unsigned long pc,
unsigned long sp)
#include <linux/types.h>
#include <asm/fpumacro.h>
-/* Return saved PC of a blocked thread. */
struct task_struct;
-unsigned long thread_saved_pc(struct task_struct *);
/* On Uniprocessor, even in RMO processes see TSO semantics */
#ifdef CONFIG_SMP
+++ /dev/null
-#ifndef __SPARC_SIGINFO_H
-#define __SPARC_SIGINFO_H
-
-#include <uapi/asm/siginfo.h>
-
-
-#ifdef CONFIG_COMPAT
-
-struct compat_siginfo;
-
-#endif /* CONFIG_COMPAT */
-
-#endif /* !(__SPARC_SIGINFO_H) */
#define TIOCGRS485 _IOR('T', 0x41, struct serial_rs485)
#define TIOCSRS485 _IOWR('T', 0x42, struct serial_rs485)
-/* Note that all the ioctls that are not available in Linux have a
+/* Note that all the ioctls that are not available in Linux have a
* double underscore on the front to: a) avoid some programs to
* think we support some ioctls under Linux (autoconfiguration stuff)
*/
#define TIOCGPTN _IOR('t', 134, unsigned int) /* Get Pty Number */
#define TIOCSPTLCK _IOW('t', 135, int) /* Lock/unlock PTY */
#define TIOCSIG _IOW('t', 136, int) /* Generate signal on Pty slave */
+#define TIOCGPTPEER _IOR('t', 137, int) /* Safely open the slave */
/* Little f */
#define FIOCLEX _IO('f', 1)
val = 0x01000000;
}
- get_online_cpus();
mutex_lock(&text_mutex);
*insn = val;
flushi(insn);
mutex_unlock(&text_mutex);
- put_online_cpus();
}
#endif
printk("\n");
}
-/*
- * Note: sparc64 has a pretty intricated thread_saved_pc, check it out.
- */
-unsigned long thread_saved_pc(struct task_struct *tsk)
-{
- return task_thread_info(tsk)->kpc;
-}
-
/*
* Free current thread data structures etc..
*/
#endif
-unsigned long thread_saved_pc(struct task_struct *tsk)
-{
- struct thread_info *ti = task_thread_info(tsk);
- unsigned long ret = 0xdeadbeefUL;
-
- if (ti && ti->ksp) {
- unsigned long *sp;
- sp = (unsigned long *)(ti->ksp + STACK_BIAS);
- if (((unsigned long)sp & (sizeof(long) - 1)) == 0UL &&
- sp[14]) {
- unsigned long *fp;
- fp = (unsigned long *)(sp[14] + STACK_BIAS);
- if (((unsigned long)fp & (sizeof(long) - 1)) == 0UL)
- ret = fp[15];
- }
- }
- return ret;
-}
-
/* Free current thread data structures etc.. */
void exit_thread(struct task_struct *tsk)
{
return sprintf(buf, "%s\n", str);
}
+static DEVICE_ATTR_RO(devspec);
static ssize_t type_show(struct device *dev,
struct device_attribute *attr, char *buf)
struct vio_dev *vdev = to_vio_dev(dev);
return sprintf(buf, "%s\n", vdev->type);
}
+static DEVICE_ATTR_RO(type);
static ssize_t modalias_show(struct device *dev, struct device_attribute *attr,
char *buf)
return sprintf(buf, "vio:T%sS%s\n", vdev->type, vdev->compat);
}
+static DEVICE_ATTR_RO(modalias);
-static struct device_attribute vio_dev_attrs[] = {
- __ATTR_RO(devspec),
- __ATTR_RO(type),
- __ATTR_RO(modalias),
- __ATTR_NULL
-};
+static struct attribute *vio_dev_attrs[] = {
+ &dev_attr_devspec.attr,
+ &dev_attr_type.attr,
+ &dev_attr_modalias.attr,
+ NULL,
+ };
+ATTRIBUTE_GROUPS(vio_dev);
static struct bus_type vio_bus_type = {
.name = "vio",
- .dev_attrs = vio_dev_attrs,
+ .dev_groups = vio_dev_groups,
.uevent = vio_hotplug,
.match = vio_bus_match,
.probe = vio_device_probe,
extern void prepare_exit_to_usermode(struct pt_regs *regs, u32 flags);
-
-/*
- * Return saved (kernel) PC of a blocked thread.
- * Only used in a printk() in kernel/sched/core.c, so don't work too hard.
- */
-#define thread_saved_pc(t) ((t)->thread.pc)
-
unsigned long get_wchan(struct task_struct *p);
/* Return initial ksp value for given task. */
void arch_jump_label_transform(struct jump_entry *e,
enum jump_label_type type)
{
- get_online_cpus();
mutex_lock(&text_mutex);
__jump_label_transform(e, type);
flush_icache_range(e->code, e->code + sizeof(tilegx_bundle_bits));
mutex_unlock(&text_mutex);
- put_online_cpus();
}
__init_or_module void arch_jump_label_transform_static(struct jump_entry *e,
* has an opportunity to return -EFAULT to the user if needed.
* The 64-bit routines just return a "long long" with the value,
* since they are only used from kernel space and don't expect to fault.
- * Support for 16-bit ops is included in the framework but we don't provide
- * any (x86_64 has an atomic_inc_short(), so we might want to some day).
+ * Support for 16-bit ops is included in the framework but we don't provide any.
*
* Note that the caller is advised to issue a suitable L1 or L2
* prefetch on the address being manipulated to avoid extra stalls.
for (count = 0; count < n/sizeof(struct io_thread_req *); count++) {
blk_end_request(
(*irq_req_buffer)[count]->req,
- 0,
+ BLK_STS_OK,
(*irq_req_buffer)[count]->length
);
kfree((*irq_req_buffer)[count]);
{
}
-extern unsigned long thread_saved_pc(struct task_struct *t);
-
static inline void mm_copy_segments(struct mm_struct *from_mm,
struct mm_struct *new_mm)
{
__attribute__((__section__(".data..init_irqstack"))) =
{ INIT_THREAD_INFO(init_task) };
-unsigned long thread_saved_pc(struct task_struct *task)
-{
- /* FIXME: Need to look up userspace_pid by cpu */
- return os_process_pc(userspace_pid[0]);
-}
-
/* Changed in setup_arch, which is called in early boot */
static char host_info[(__NEW_UTS_LEN + 1) * 5];
generic-y += setup.h
generic-y += shmbuf.h
generic-y += shmparam.h
-generic-y += siginfo.h
generic-y += signal.h
generic-y += sizes.h
generic-y += socket.h
include include/uapi/asm-generic/Kbuild.asm
generic-y += kvm_para.h
+generic-y += siginfo.h
select ARCH_USE_BUILTIN_BSWAP
select ARCH_USE_QUEUED_RWLOCKS
select ARCH_USE_QUEUED_SPINLOCKS
- select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH if SMP
+ select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH
select ARCH_WANT_FRAME_POINTERS
select ARCH_WANTS_DYNAMIC_TASK_STRUCT
select BUILDTIME_EXTABLE_SORT
select GENERIC_EARLY_IOREMAP
select GENERIC_FIND_FIRST_BIT
select GENERIC_IOMAP
+ select GENERIC_IRQ_EFFECTIVE_AFF_MASK if SMP
+ select GENERIC_IRQ_MIGRATION if SMP
select GENERIC_IRQ_PROBE
select GENERIC_IRQ_SHOW
select GENERIC_PENDING_IRQ if SMP
select HAVE_UNSTABLE_SCHED_CLOCK
select HAVE_USER_RETURN_NOTIFIER
select IRQ_FORCED_THREADING
+ select PCI_LOCKLESS_CONFIG
select PERF_EVENTS
select RTC_LIB
select RTC_MC146818_LIB
def_bool y
config X86_MCE_INJECT
- depends on X86_MCE && X86_LOCAL_APIC && X86_MCELOG_LEGACY
+ depends on X86_MCE && X86_LOCAL_APIC && DEBUG_FS
tristate "Machine check injector support"
---help---
Provide support for injecting machine checks for testing purposes.
bool
depends on STA2X11
+config HAVE_GENERIC_GUP
+ def_bool y
+
source "net/Kconfig"
source "drivers/Kconfig"
drivers-$(CONFIG_FB) += arch/x86/video/
-drivers-$(CONFIG_RAS) += arch/x86/ras/
-
####
# boot loader support. Several targets are kept for legacy purposes
return *((char *)(fs + addr));
}
#include "../cmdline.c"
-static unsigned long get_cmd_line_ptr(void)
+unsigned long get_cmd_line_ptr(void)
{
unsigned long cmd_line_ptr = boot_params->hdr.cmd_line_ptr;
memset((char *)gdt->address, 0x0, gdt->size);
desc = (struct desc_struct *)gdt->address;
- /* The first GDT is a dummy and the second is unused. */
- desc += 2;
+ /* The first GDT is a dummy. */
+ desc++;
+
+ if (IS_ENABLED(CONFIG_X86_64)) {
+ /* __KERNEL32_CS */
+ desc->limit0 = 0xffff;
+ desc->base0 = 0x0000;
+ desc->base1 = 0x0000;
+ desc->type = SEG_TYPE_CODE | SEG_TYPE_EXEC_READ;
+ desc->s = DESC_TYPE_CODE_DATA;
+ desc->dpl = 0;
+ desc->p = 1;
+ desc->limit = 0xf;
+ desc->avl = 0;
+ desc->l = 0;
+ desc->d = SEG_OP_SIZE_32BIT;
+ desc->g = SEG_GRANULARITY_4KB;
+ desc->base2 = 0x00;
+ desc++;
+ } else {
+ /* Second entry is unused on 32-bit */
+ desc++;
+ }
+ /* __KERNEL_CS */
desc->limit0 = 0xffff;
desc->base0 = 0x0000;
desc->base1 = 0x0000;
desc->p = 1;
desc->limit = 0xf;
desc->avl = 0;
- desc->l = 0;
- desc->d = SEG_OP_SIZE_32BIT;
+ if (IS_ENABLED(CONFIG_X86_64)) {
+ desc->l = 1;
+ desc->d = 0;
+ } else {
+ desc->l = 0;
+ desc->d = SEG_OP_SIZE_32BIT;
+ }
desc->g = SEG_GRANULARITY_4KB;
desc->base2 = 0x00;
-
desc++;
+
+ /* __KERNEL_DS */
desc->limit0 = 0xffff;
desc->base0 = 0x0000;
desc->base1 = 0x0000;
desc->d = SEG_OP_SIZE_32BIT;
desc->g = SEG_GRANULARITY_4KB;
desc->base2 = 0x00;
-
-#ifdef CONFIG_X86_64
- /* Task segment value */
desc++;
- desc->limit0 = 0x0000;
- desc->base0 = 0x0000;
- desc->base1 = 0x0000;
- desc->type = SEG_TYPE_TSS;
- desc->s = 0;
- desc->dpl = 0;
- desc->p = 1;
- desc->limit = 0x0;
- desc->avl = 0;
- desc->l = 0;
- desc->d = 0;
- desc->g = SEG_GRANULARITY_4KB;
- desc->base2 = 0x00;
-#endif /* CONFIG_X86_64 */
+
+ if (IS_ENABLED(CONFIG_X86_64)) {
+ /* Task segment value */
+ desc->limit0 = 0x0000;
+ desc->base0 = 0x0000;
+ desc->base1 = 0x0000;
+ desc->type = SEG_TYPE_TSS;
+ desc->s = 0;
+ desc->dpl = 0;
+ desc->p = 1;
+ desc->limit = 0x0;
+ desc->avl = 0;
+ desc->l = 0;
+ desc->d = 0;
+ desc->g = SEG_GRANULARITY_4KB;
+ desc->base2 = 0x00;
+ desc++;
+ }
asm volatile("cli");
asm volatile ("lgdt %0" : : "m" (*gdt));
/* Set up the stack */
leaq boot_stack_end(%rbx), %rsp
+#ifdef CONFIG_X86_5LEVEL
+ /* Check if 5-level paging has already enabled */
+ movq %cr4, %rax
+ testl $X86_CR4_LA57, %eax
+ jnz lvl5
+
+ /*
+ * At this point we are in long mode with 4-level paging enabled,
+ * but we want to enable 5-level paging.
+ *
+ * The problem is that we cannot do it directly. Setting LA57 in
+ * long mode would trigger #GP. So we need to switch off long mode
+ * first.
+ *
+ * NOTE: This is not going to work if bootloader put us above 4G
+ * limit.
+ *
+ * The first step is go into compatibility mode.
+ */
+
+ /* Clear additional page table */
+ leaq lvl5_pgtable(%rbx), %rdi
+ xorq %rax, %rax
+ movq $(PAGE_SIZE/8), %rcx
+ rep stosq
+
+ /*
+ * Setup current CR3 as the first and only entry in a new top level
+ * page table.
+ */
+ movq %cr3, %rdi
+ leaq 0x7 (%rdi), %rax
+ movq %rax, lvl5_pgtable(%rbx)
+
+ /* Switch to compatibility mode (CS.L = 0 CS.D = 1) via far return */
+ pushq $__KERNEL32_CS
+ leaq compatible_mode(%rip), %rax
+ pushq %rax
+ lretq
+lvl5:
+#endif
+
/* Zero EFLAGS */
pushq $0
popfq
jmp *%rax
.code32
+#ifdef CONFIG_X86_5LEVEL
+compatible_mode:
+ /* Setup data and stack segments */
+ movl $__KERNEL_DS, %eax
+ movl %eax, %ds
+ movl %eax, %ss
+
+ /* Disable paging */
+ movl %cr0, %eax
+ btrl $X86_CR0_PG_BIT, %eax
+ movl %eax, %cr0
+
+ /* Point CR3 to 5-level paging */
+ leal lvl5_pgtable(%ebx), %eax
+ movl %eax, %cr3
+
+ /* Enable PAE and LA57 mode */
+ movl %cr4, %eax
+ orl $(X86_CR4_PAE | X86_CR4_LA57), %eax
+ movl %eax, %cr4
+
+ /* Calculate address we are running at */
+ call 1f
+1: popl %edi
+ subl $1b, %edi
+
+ /* Prepare stack for far return to Long Mode */
+ pushl $__KERNEL_CS
+ leal lvl5(%edi), %eax
+ push %eax
+
+ /* Enable paging back */
+ movl $(X86_CR0_PG | X86_CR0_PE), %eax
+ movl %eax, %cr0
+
+ lret
+#endif
+
no_longmode:
/* This isn't an x86-64 CPU so hang */
1:
.word gdt_end - gdt
.long gdt
.word 0
- .quad 0x0000000000000000 /* NULL descriptor */
+ .quad 0x00cf9a000000ffff /* __KERNEL32_CS */
.quad 0x00af9a000000ffff /* __KERNEL_CS */
.quad 0x00cf92000000ffff /* __KERNEL_DS */
.quad 0x0080890000000000 /* TS descriptor */
.balign 4096
pgtable:
.fill BOOT_PGT_SIZE, 1, 0
+#ifdef CONFIG_X86_5LEVEL
+lvl5_pgtable:
+ .fill PAGE_SIZE, 1, 0
+#endif
* contain the entire properly aligned running kernel image.
*
*/
+
+/*
+ * isspace() in linux/ctype.h is expected by next_args() to filter
+ * out "space/lf/tab". While boot/ctype.h conflicts with linux/ctype.h,
+ * since isdigit() is implemented in both of them. Hence disable it
+ * here.
+ */
+#define BOOT_CTYPE_H
+
+/*
+ * _ctype[] in lib/ctype.c is needed by isspace() of linux/ctype.h.
+ * While both lib/ctype.c and lib/cmdline.c will bring EXPORT_SYMBOL
+ * which is meaningless and will cause compiling error in some cases.
+ * So do not include linux/export.h and define EXPORT_SYMBOL(sym)
+ * as empty.
+ */
+#define _LINUX_EXPORT_H
+#define EXPORT_SYMBOL(sym)
+
#include "misc.h"
#include "error.h"
-#include "../boot.h"
+#include "../string.h"
#include <generated/compile.h>
#include <linux/module.h>
#include <linux/uts.h>
#include <linux/utsname.h>
+#include <linux/ctype.h>
#include <generated/utsrelease.h>
+/* Macros used by the included decompressor code below. */
+#define STATIC
+#include <linux/decompress/mm.h>
+
+extern unsigned long get_cmd_line_ptr(void);
+
/* Simplified build-specific string for starting entropy. */
static const char build_str[] = UTS_RELEASE " (" LINUX_COMPILE_BY "@"
LINUX_COMPILE_HOST ") (" LINUX_COMPILER ") " UTS_VERSION;
static bool memmap_too_large;
+
+/* Store memory limit specified by "mem=nn[KMG]" or "memmap=nn[KMG]" */
+unsigned long long mem_limit = ULLONG_MAX;
+
+
enum mem_avoid_index {
MEM_AVOID_ZO_RANGE = 0,
MEM_AVOID_INITRD,
return true;
}
-/**
- * _memparse - Parse a string with mem suffixes into a number
- * @ptr: Where parse begins
- * @retptr: (output) Optional pointer to next char after parse completes
- *
- * Parses a string into a number. The number stored at @ptr is
- * potentially suffixed with K, M, G, T, P, E.
- */
-static unsigned long long _memparse(const char *ptr, char **retptr)
+char *skip_spaces(const char *str)
{
- char *endptr; /* Local pointer to end of parsed string */
-
- unsigned long long ret = simple_strtoull(ptr, &endptr, 0);
-
- switch (*endptr) {
- case 'E':
- case 'e':
- ret <<= 10;
- case 'P':
- case 'p':
- ret <<= 10;
- case 'T':
- case 't':
- ret <<= 10;
- case 'G':
- case 'g':
- ret <<= 10;
- case 'M':
- case 'm':
- ret <<= 10;
- case 'K':
- case 'k':
- ret <<= 10;
- endptr++;
- default:
- break;
- }
-
- if (retptr)
- *retptr = endptr;
-
- return ret;
+ while (isspace(*str))
+ ++str;
+ return (char *)str;
}
+#include "../../../../lib/ctype.c"
+#include "../../../../lib/cmdline.c"
static int
parse_memmap(char *p, unsigned long long *start, unsigned long long *size)
return -EINVAL;
oldp = p;
- *size = _memparse(p, &p);
+ *size = memparse(p, &p);
if (p == oldp)
return -EINVAL;
switch (*p) {
- case '@':
- /* Skip this region, usable */
- *start = 0;
- *size = 0;
- return 0;
case '#':
case '$':
case '!':
- *start = _memparse(p + 1, &p);
+ *start = memparse(p + 1, &p);
+ return 0;
+ case '@':
+ /* memmap=nn@ss specifies usable region, should be skipped */
+ *size = 0;
+ /* Fall through */
+ default:
+ /*
+ * If w/o offset, only size specified, memmap=nn[KMG] has the
+ * same behaviour as mem=nn[KMG]. It limits the max address
+ * system can use. Region above the limit should be avoided.
+ */
+ *start = 0;
return 0;
}
return -EINVAL;
}
-static void mem_avoid_memmap(void)
+static void mem_avoid_memmap(char *str)
{
- char arg[128];
+ static int i;
int rc;
- int i;
- char *str;
- /* See if we have any memmap areas */
- rc = cmdline_find_option("memmap", arg, sizeof(arg));
- if (rc <= 0)
+ if (i >= MAX_MEMMAP_REGIONS)
return;
- i = 0;
- str = arg;
while (str && (i < MAX_MEMMAP_REGIONS)) {
int rc;
unsigned long long start, size;
if (rc < 0)
break;
str = k;
- /* A usable region that should not be skipped */
- if (size == 0)
+
+ if (start == 0) {
+ /* Store the specified memory limit if size > 0 */
+ if (size > 0)
+ mem_limit = size;
+
continue;
+ }
mem_avoid[MEM_AVOID_MEMMAP_BEGIN + i].start = start;
mem_avoid[MEM_AVOID_MEMMAP_BEGIN + i].size = size;
memmap_too_large = true;
}
+static int handle_mem_memmap(void)
+{
+ char *args = (char *)get_cmd_line_ptr();
+ size_t len = strlen((char *)args);
+ char *tmp_cmdline;
+ char *param, *val;
+ u64 mem_size;
+
+ if (!strstr(args, "memmap=") && !strstr(args, "mem="))
+ return 0;
+
+ tmp_cmdline = malloc(len + 1);
+ if (!tmp_cmdline )
+ error("Failed to allocate space for tmp_cmdline");
+
+ memcpy(tmp_cmdline, args, len);
+ tmp_cmdline[len] = 0;
+ args = tmp_cmdline;
+
+ /* Chew leading spaces */
+ args = skip_spaces(args);
+
+ while (*args) {
+ args = next_arg(args, ¶m, &val);
+ /* Stop at -- */
+ if (!val && strcmp(param, "--") == 0) {
+ warn("Only '--' specified in cmdline");
+ free(tmp_cmdline);
+ return -1;
+ }
+
+ if (!strcmp(param, "memmap")) {
+ mem_avoid_memmap(val);
+ } else if (!strcmp(param, "mem")) {
+ char *p = val;
+
+ if (!strcmp(p, "nopentium"))
+ continue;
+ mem_size = memparse(p, &p);
+ if (mem_size == 0) {
+ free(tmp_cmdline);
+ return -EINVAL;
+ }
+ mem_limit = mem_size;
+ }
+ }
+
+ free(tmp_cmdline);
+ return 0;
+}
+
/*
* In theory, KASLR can put the kernel anywhere in the range of [16M, 64T).
* The mem_avoid array is used to store the ranges that need to be avoided
/* We don't need to set a mapping for setup_data. */
/* Mark the memmap regions we need to avoid */
- mem_avoid_memmap();
+ handle_mem_memmap();
#ifdef CONFIG_X86_VERBOSE_BOOTUP
/* Make sure video RAM can be used. */
{
struct mem_vector region, overlap;
struct slot_area slot_area;
- unsigned long start_orig;
+ unsigned long start_orig, end;
+ struct boot_e820_entry cur_entry;
/* Skip non-RAM entries. */
if (entry->type != E820_TYPE_RAM)
if (entry->addr + entry->size < minimum)
return;
- region.start = entry->addr;
- region.size = entry->size;
+ /* Ignore entries above memory limit */
+ end = min(entry->size + entry->addr, mem_limit);
+ if (entry->addr >= end)
+ return;
+ cur_entry.addr = entry->addr;
+ cur_entry.size = end - entry->addr;
+
+ region.start = cur_entry.addr;
+ region.size = cur_entry.size;
/* Give up if slot area array is full. */
while (slot_area_index < MAX_SLOT_AREA) {
region.start = ALIGN(region.start, CONFIG_PHYSICAL_ALIGN);
/* Did we raise the address above this e820 region? */
- if (region.start > entry->addr + entry->size)
+ if (region.start > cur_entry.addr + cur_entry.size)
return;
/* Reduce size by any delta from the original address. */
{
unsigned long random_addr, min_addr;
- /* By default, keep output position unchanged. */
- *virt_addr = *output;
-
if (cmdline_find_option_bool("nokaslr")) {
warn("KASLR disabled: 'nokaslr' on cmdline.");
return;
unsigned long output_len)
{
const unsigned long kernel_total_size = VO__end - VO__text;
- unsigned long virt_addr = (unsigned long)output;
+ unsigned long virt_addr = LOAD_PHYSICAL_ADDR;
/* Retain x86 boot parameters pointer passed from startup_32/64. */
boot_params = rmode;
#ifdef CONFIG_X86_64
if (heap > 0x3fffffffffffUL)
error("Destination address too large");
+ if (virt_addr + max(output_len, kernel_total_size) > KERNEL_IMAGE_SIZE)
+ error("Destination virtual address is beyond the kernel mapping area");
#else
if (heap > ((-__PAGE_OFFSET-(128<<20)-1) & 0x7fffffff))
error("Destination address too large");
#ifndef CONFIG_RELOCATABLE
if ((unsigned long)output != LOAD_PHYSICAL_ADDR)
error("Destination address does not match LOAD_PHYSICAL_ADDR");
- if ((unsigned long)output != virt_addr)
+ if (virt_addr != LOAD_PHYSICAL_ADDR)
error("Destination virtual address changed when not relocatable");
#endif
unsigned long output_size,
unsigned long *virt_addr)
{
- /* No change from existing output location. */
- *virt_addr = *output;
}
#endif
static struct alloc_pgt_data pgt_data;
/* The top level page table entry pointer. */
-static unsigned long level4p;
+static unsigned long top_level_pgt;
/*
* Mapping information structure passed to kernel_ident_mapping_init().
* If we came here via startup_32(), cr3 will be _pgtable already
* and we must append to the existing area instead of entirely
* overwriting it.
+ *
+ * With 5-level paging, we use '_pgtable' to allocate the p4d page table,
+ * the top-level page table is allocated separately.
+ *
+ * p4d_offset(top_level_pgt, 0) would cover both the 4- and 5-level
+ * cases. On 4-level paging it's equal to 'top_level_pgt'.
*/
- level4p = read_cr3();
- if (level4p == (unsigned long)_pgtable) {
+ top_level_pgt = read_cr3_pa();
+ if (p4d_offset((pgd_t *)top_level_pgt, 0) == (p4d_t *)_pgtable) {
debug_putstr("booted via startup_32()\n");
pgt_data.pgt_buf = _pgtable + BOOT_INIT_PGT_SIZE;
pgt_data.pgt_buf_size = BOOT_PGT_SIZE - BOOT_INIT_PGT_SIZE;
pgt_data.pgt_buf = _pgtable;
pgt_data.pgt_buf_size = BOOT_PGT_SIZE;
memset(pgt_data.pgt_buf, 0, pgt_data.pgt_buf_size);
- level4p = (unsigned long)alloc_pgt_page(&pgt_data);
+ top_level_pgt = (unsigned long)alloc_pgt_page(&pgt_data);
}
}
return;
/* Build the mapping. */
- kernel_ident_mapping_init(&mapping_info, (pgd_t *)level4p,
+ kernel_ident_mapping_init(&mapping_info, (pgd_t *)top_level_pgt,
start, end);
}
*/
void finalize_identity_maps(void)
{
- write_cr3(level4p);
+ write_cr3(top_level_pgt);
}
popw %es
retl
ENDPROC(copy_to_fs)
-
-#if 0 /* Not currently used, but can be enabled as needed */
-GLOBAL(copy_from_gs)
- pushw %ds
- pushw %gs
- popw %ds
- calll memcpy
- popw %ds
- retl
-ENDPROC(copy_from_gs)
-
-GLOBAL(copy_to_gs)
- pushw %es
- pushw %gs
- popw %es
- calll memcpy
- popw %es
- retl
-ENDPROC(copy_to_gs)
-#endif
return result;
}
+long simple_strtol(const char *cp, char **endp, unsigned int base)
+{
+ if (*cp == '-')
+ return -simple_strtoull(cp + 1, endp, base);
+
+ return simple_strtoull(cp, endp, base);
+}
+
/**
* strlen - Find the length of a string
* @s: The string to be sized
extern int strncmp(const char *cs, const char *ct, size_t count);
extern size_t strlen(const char *s);
extern char *strstr(const char *s1, const char *s2);
+extern char *strchr(const char *s, int c);
extern size_t strnlen(const char *s, size_t maxlen);
extern unsigned int atou(const char *s);
extern unsigned long long simple_strtoull(const char *cp, char **endp,
# Arch-specific CryptoAPI modules.
#
+OBJECT_FILES_NON_STANDARD := y
+
avx_supported := $(call as-instr,vpxor %xmm0$(comma)%xmm0$(comma)%xmm0,yes,no)
avx2_supported := $(call as-instr,vpgatherdd %ymm0$(comma)(%eax$(comma)%ymm1\
$(comma)4)$(comma)%ymm2,yes,no)
# Arch-specific CryptoAPI modules.
#
+OBJECT_FILES_NON_STANDARD := y
+
avx2_supported := $(call as-instr,vpgatherdd %ymm0$(comma)(%eax$(comma)%ymm1\
$(comma)4)$(comma)%ymm2,yes,no)
ifeq ($(avx2_supported),yes)
# Arch-specific CryptoAPI modules.
#
+OBJECT_FILES_NON_STANDARD := y
+
avx2_supported := $(call as-instr,vpgatherdd %ymm0$(comma)(%eax$(comma)%ymm1\
$(comma)4)$(comma)%ymm2,yes,no)
ifeq ($(avx2_supported),yes)
* If width of "canonical tail" ever becomes variable, this will need
* to be updated to remain correct on both old and new CPUs.
*
- * Change top 16 bits to be the sign-extension of 47th bit
+ * Change top bits to match most significant bit (47th or 56th bit
+ * depending on paging mode) in the address.
*/
shl $(64 - (__VIRTUAL_MASK_SHIFT+1)), %rcx
sar $(64 - (__VIRTUAL_MASK_SHIFT+1)), %rcx
return ret;
}
+static struct attribute_group x86_pmu_attr_group;
+
static int __init init_hw_perf_events(void)
{
struct x86_pmu_quirk *quirk;
x86_pmu_events_group.attrs = tmp;
}
+ if (x86_pmu.attrs) {
+ struct attribute **tmp;
+
+ tmp = merge_attr(x86_pmu_attr_group.attrs, x86_pmu.attrs);
+ if (!WARN_ON(!tmp))
+ x86_pmu_attr_group.attrs = tmp;
+ }
+
pr_info("... version: %d\n", x86_pmu.version);
pr_info("... bit width: %d\n", x86_pmu.cntval_bits);
pr_info("... generic registers: %d\n", x86_pmu.num_counters);
static void refresh_pce(void *ignored)
{
- if (current->active_mm)
- load_mm_cr4(current->active_mm);
+ load_mm_cr4(this_cpu_read(cpu_tlbstate.loaded_mm));
}
static void x86_pmu_event_mapped(struct perf_event *event)
if (x86_pmu.check_microcode)
x86_pmu.check_microcode();
}
-EXPORT_SYMBOL_GPL(perf_check_microcode);
static struct pmu pmu = {
.pmu_enable = x86_pmu_enable,
void arch_perf_update_userpage(struct perf_event *event,
struct perf_event_mmap_page *userpg, u64 now)
{
- struct cyc2ns_data *data;
+ struct cyc2ns_data data;
u64 offset;
userpg->cap_user_time = 0;
if (!using_native_sched_clock() || !sched_clock_stable())
return;
- data = cyc2ns_read_begin();
+ cyc2ns_read_begin(&data);
- offset = data->cyc2ns_offset + __sched_clock_offset;
+ offset = data.cyc2ns_offset + __sched_clock_offset;
/*
* Internal timekeeping for enabled/running/stopped times
* is always in the local_clock domain.
*/
userpg->cap_user_time = 1;
- userpg->time_mult = data->cyc2ns_mul;
- userpg->time_shift = data->cyc2ns_shift;
+ userpg->time_mult = data.cyc2ns_mul;
+ userpg->time_shift = data.cyc2ns_shift;
userpg->time_offset = offset - now;
/*
userpg->time_zero = offset;
}
- cyc2ns_read_end(data);
+ cyc2ns_read_end();
}
void
/* IRQs are off, so this synchronizes with smp_store_release */
ldt = lockless_dereference(current->active_mm->context.ldt);
- if (!ldt || idx > ldt->size)
+ if (!ldt || idx > ldt->nr_entries)
return 0;
desc = &ldt->entries[idx];
[ C(DTLB) ] = {
[ C(OP_READ) ] = {
[ C(RESULT_ACCESS) ] = 0x81d0, /* MEM_INST_RETIRED.ALL_LOADS */
- [ C(RESULT_MISS) ] = 0x608, /* DTLB_LOAD_MISSES.WALK_COMPLETED */
+ [ C(RESULT_MISS) ] = 0xe08, /* DTLB_LOAD_MISSES.WALK_COMPLETED */
},
[ C(OP_WRITE) ] = {
[ C(RESULT_ACCESS) ] = 0x82d0, /* MEM_INST_RETIRED.ALL_STORES */
- [ C(RESULT_MISS) ] = 0x649, /* DTLB_STORE_MISSES.WALK_COMPLETED */
+ [ C(RESULT_MISS) ] = 0xe49, /* DTLB_STORE_MISSES.WALK_COMPLETED */
},
[ C(OP_PREFETCH) ] = {
[ C(RESULT_ACCESS) ] = 0x0,
return -ENOMEM;
}
+static void flip_smm_bit(void *data)
+{
+ unsigned long set = *(unsigned long *)data;
+
+ if (set > 0) {
+ msr_set_bit(MSR_IA32_DEBUGCTLMSR,
+ DEBUGCTLMSR_FREEZE_IN_SMM_BIT);
+ } else {
+ msr_clear_bit(MSR_IA32_DEBUGCTLMSR,
+ DEBUGCTLMSR_FREEZE_IN_SMM_BIT);
+ }
+}
+
static void intel_pmu_cpu_starting(int cpu)
{
struct cpu_hw_events *cpuc = &per_cpu(cpu_hw_events, cpu);
cpuc->lbr_sel = NULL;
+ flip_smm_bit(&x86_pmu.attr_freeze_on_smi);
+
if (!cpuc->shared_regs)
return;
int pebs_broken = 0;
int cpu;
- get_online_cpus();
for_each_online_cpu(cpu) {
if ((pebs_broken = intel_snb_pebs_broken(cpu)))
break;
}
- put_online_cpus();
if (pebs_broken == x86_pmu.pebs_broken)
return;
static __init void intel_sandybridge_quirk(void)
{
x86_pmu.check_microcode = intel_snb_check_microcode;
+ cpus_read_lock();
intel_snb_check_microcode();
+ cpus_read_unlock();
}
static const struct { int id; char *name; } intel_arch_events_map[] __initconst = {
NULL
};
+static ssize_t freeze_on_smi_show(struct device *cdev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ return sprintf(buf, "%lu\n", x86_pmu.attr_freeze_on_smi);
+}
+
+static DEFINE_MUTEX(freeze_on_smi_mutex);
+
+static ssize_t freeze_on_smi_store(struct device *cdev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ unsigned long val;
+ ssize_t ret;
+
+ ret = kstrtoul(buf, 0, &val);
+ if (ret)
+ return ret;
+
+ if (val > 1)
+ return -EINVAL;
+
+ mutex_lock(&freeze_on_smi_mutex);
+
+ if (x86_pmu.attr_freeze_on_smi == val)
+ goto done;
+
+ x86_pmu.attr_freeze_on_smi = val;
+
+ get_online_cpus();
+ on_each_cpu(flip_smm_bit, &val, 1);
+ put_online_cpus();
+done:
+ mutex_unlock(&freeze_on_smi_mutex);
+
+ return count;
+}
+
+static DEVICE_ATTR_RW(freeze_on_smi);
+
+static struct attribute *intel_pmu_attrs[] = {
+ &dev_attr_freeze_on_smi.attr,
+ NULL,
+};
+
__init int intel_pmu_init(void)
{
union cpuid10_edx edx;
x86_pmu.max_pebs_events = min_t(unsigned, MAX_PEBS_EVENTS, x86_pmu.num_counters);
+
+ x86_pmu.attrs = intel_pmu_attrs;
/*
* Quirk: v2 perfmon does not report fixed-purpose events, so
* assume at least 3 events, when not running in a hypervisor:
lockup_detector_resume();
- get_online_cpus();
+ cpus_read_lock();
- for_each_online_cpu(c) {
+ for_each_online_cpu(c)
free_excl_cntrs(c);
- }
- put_online_cpus();
+ cpus_read_unlock();
pr_info("PMU erratum BJ122, BV98, HSD29 workaround disabled, HT off\n");
return 0;
}
*
* Also, check that the scales match on all cpus.
*/
- get_online_cpus();
+ cpus_read_lock();
for_each_online_cpu(cpu) {
struct cpuinfo_x86 *c = &cpu_data(cpu);
* Setup the hot cpu notifier once we are sure cqm
* is enabled to avoid notifier leak.
*/
- cpuhp_setup_state(CPUHP_AP_PERF_X86_CQM_STARTING,
- "perf/x86/cqm:starting",
- intel_cqm_cpu_starting, NULL);
- cpuhp_setup_state(CPUHP_AP_PERF_X86_CQM_ONLINE, "perf/x86/cqm:online",
- NULL, intel_cqm_cpu_exit);
-
+ cpuhp_setup_state_cpuslocked(CPUHP_AP_PERF_X86_CQM_STARTING,
+ "perf/x86/cqm:starting",
+ intel_cqm_cpu_starting, NULL);
+ cpuhp_setup_state_cpuslocked(CPUHP_AP_PERF_X86_CQM_ONLINE,
+ "perf/x86/cqm:online",
+ NULL, intel_cqm_cpu_exit);
out:
- put_online_cpus();
+ cpus_read_unlock();
if (ret) {
kfree(str);
LBR_FORMAT_MAX_KNOWN = LBR_FORMAT_TIME,
};
-static enum {
+static const enum {
LBR_EIP_FLAGS = 1,
LBR_TSX = 2,
} lbr_desc[LBR_FORMAT_MAX_KNOWN + 1] = {
/*
* If quirk is needed, ensure sign extension is 61 bits:
*/
-u64 lbr_from_signext_quirk_rd(u64 val)
+static u64 lbr_from_signext_quirk_rd(u64 val)
{
if (static_branch_unlikely(&lbr_from_quirk_key)) {
/*
pmu = type->pmus;
for (i = 0; i < type->num_boxes; i++, pmu++) {
box = pmu->boxes[pkg];
- if (!box && atomic_inc_return(&box->refcnt) == 1)
+ if (box && atomic_inc_return(&box->refcnt) == 1)
uncore_box_init(box);
}
}
ssize_t (*events_sysfs_show)(char *page, u64 config);
struct attribute **cpu_events;
+ unsigned long attr_freeze_on_smi;
+ struct attribute **attrs;
+
/*
* CPU Hotplug hooks
*/
#include <linux/ioport.h>
#include <linux/pci.h>
+#include <linux/refcount.h>
struct amd_nb_bus_dev_range {
u8 bus;
struct threshold_block *blocks;
/* initialized to the number of CPUs on the node sharing this bank */
- atomic_t cpus;
+ refcount_t cpus;
};
struct amd_northbridge {
#define x2apic_supported() (0)
#endif /* !CONFIG_X86_X2APIC */
+struct irq_data;
+
/*
* Copyright 2004 James Cleverdon, IBM.
* Subject to the GNU Public License, v.2
/* Can't be NULL on 64-bit */
unsigned long (*set_apic_id)(unsigned int id);
- int (*cpu_mask_to_apicid_and)(const struct cpumask *cpumask,
- const struct cpumask *andmask,
- unsigned int *apicid);
+ int (*cpu_mask_to_apicid)(const struct cpumask *cpumask,
+ struct irq_data *irqdata,
+ unsigned int *apicid);
/* ipi */
void (*send_IPI)(int cpu, int vector);
#endif
-static inline int
-flat_cpu_mask_to_apicid_and(const struct cpumask *cpumask,
- const struct cpumask *andmask,
- unsigned int *apicid)
-{
- unsigned long cpu_mask = cpumask_bits(cpumask)[0] &
- cpumask_bits(andmask)[0] &
- cpumask_bits(cpu_online_mask)[0] &
- APIC_ALL_CPUS;
-
- if (likely(cpu_mask)) {
- *apicid = (unsigned int)cpu_mask;
- return 0;
- } else {
- return -EINVAL;
- }
-}
-
-extern int
-default_cpu_mask_to_apicid_and(const struct cpumask *cpumask,
- const struct cpumask *andmask,
- unsigned int *apicid);
+extern int flat_cpu_mask_to_apicid(const struct cpumask *cpumask,
+ struct irq_data *irqdata,
+ unsigned int *apicid);
+extern int default_cpu_mask_to_apicid(const struct cpumask *cpumask,
+ struct irq_data *irqdata,
+ unsigned int *apicid);
static inline void
flat_vector_allocation_domain(int cpu, struct cpumask *retmask,
return c;
}
-/**
- * atomic_inc_short - increment of a short integer
- * @v: pointer to type int
- *
- * Atomically adds 1 to @v
- * Returns the new value of @u
- */
-static __always_inline short int atomic_inc_short(short int *v)
-{
- asm(LOCK_PREFIX "addw $1, %0" : "+m" (*v));
- return *v;
-}
-
#ifdef CONFIG_X86_32
# include <asm/atomic64_32.h>
#else
__kernel_fpu_begin(); \
\
if (efi_scratch.use_pgd) { \
- efi_scratch.prev_cr3 = read_cr3(); \
+ efi_scratch.prev_cr3 = __read_cr3(); \
write_cr3((unsigned long)efi_scratch.efi_pgt); \
__flush_tlb_all(); \
} \
#ifdef CONFIG_SMP
unsigned int irq_resched_count;
unsigned int irq_call_count;
- unsigned int irq_tlb_count;
#endif
+ unsigned int irq_tlb_count;
#ifdef CONFIG_X86_THERMAL_VECTOR
unsigned int irq_thermal_count;
#endif
#include <linux/cpumask.h>
extern int check_irq_vectors_for_cpu_disable(void);
extern void fixup_irqs(void);
-extern void irq_force_complete_move(struct irq_desc *desc);
#endif
#ifdef CONFIG_HAVE_KVM
irq_remapping_get_irq_domain(struct irq_alloc_info *info);
/* Create PCI MSI/MSIx irqdomain, use @parent as the parent irqdomain. */
-extern struct irq_domain *arch_create_msi_irq_domain(struct irq_domain *parent);
+extern struct irq_domain *
+arch_create_remap_msi_irq_domain(struct irq_domain *par, const char *n, int id);
/* Get parent irqdomain for interrupt remapping irqdomain */
static inline struct irq_domain *arch_get_ir_parent_domain(void)
bool perm_ok; /* do not check permissions if true */
bool ud; /* inject an #UD if host doesn't support insn */
+ bool tf; /* TF value before instruction (after for syscall/sysret) */
bool have_exception;
struct x86_exception exception;
DECLARE_PER_CPU(struct mce, injectm);
-extern void register_mce_write_callback(ssize_t (*)(struct file *filp,
- const char __user *ubuf,
- size_t usize, loff_t *off));
-
/* Disable CMCI/polling for MCA bank claimed by firmware */
extern void mce_disable_bank(int bank);
#endif
} mm_context_t;
-#ifdef CONFIG_SMP
void leave_mm(int cpu);
-#else
-static inline void leave_mm(int cpu)
-{
-}
-#endif
#endif /* _ASM_X86_MMU_H */
* allocations, but it's not worth trying to optimize.
*/
struct desc_struct *entries;
- unsigned int size;
+ unsigned int nr_entries;
};
/*
*/
if (unlikely(ldt))
- set_ldt(ldt->entries, ldt->size);
+ set_ldt(ldt->entries, ldt->nr_entries);
else
clear_LDT();
#else
clear_LDT();
#endif
+}
+
+static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next)
+{
+#ifdef CONFIG_MODIFY_LDT_SYSCALL
+ /*
+ * Load the LDT if either the old or new mm had an LDT.
+ *
+ * An mm will never go from having an LDT to not having an LDT. Two
+ * mms never share an LDT, so we don't gain anything by checking to
+ * see whether the LDT changed. There's also no guarantee that
+ * prev->context.ldt actually matches LDTR, but, if LDTR is non-NULL,
+ * then prev->context.ldt will also be non-NULL.
+ *
+ * If we really cared, we could optimize the case where prev == next
+ * and we're exiting lazy mode. Most of the time, if this happens,
+ * we don't actually need to reload LDTR, but modify_ldt() is mostly
+ * used by legacy code and emulators where we don't need this level of
+ * performance.
+ *
+ * This uses | instead of || because it generates better code.
+ */
+ if (unlikely((unsigned long)prev->context.ldt |
+ (unsigned long)next->context.ldt))
+ load_mm_ldt(next);
+#endif
DEBUG_LOCKS_WARN_ON(preemptible());
}
static inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk)
{
-#ifdef CONFIG_SMP
if (this_cpu_read(cpu_tlbstate.state) == TLBSTATE_OK)
this_cpu_write(cpu_tlbstate.state, TLBSTATE_LAZY);
-#endif
}
static inline int init_new_context(struct task_struct *tsk,
}
#endif
-static inline bool __pkru_allows_pkey(u16 pkey, bool write)
-{
- u32 pkru = read_pkru();
-
- if (!__pkru_allows_read(pkru, pkey))
- return false;
- if (write && !__pkru_allows_write(pkru, pkey))
- return false;
-
- return true;
-}
-
/*
* We only want to enforce protection keys on the current process
* because we effectively have no access to PKRU for other
return __pkru_allows_pkey(vma_pkey(vma), write);
}
+
+/*
+ * This can be used from process context to figure out what the value of
+ * CR3 is without needing to do a (slow) __read_cr3().
+ *
+ * It's intended to be used for code like KVM that sneakily changes CR3
+ * and needs to restore it. It needs to be used very carefully.
+ */
+static inline unsigned long __get_current_cr3_fast(void)
+{
+ unsigned long cr3 = __pa(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd);
+
+ /* For now, be very restrictive about when this can be called. */
+ VM_WARN_ON(in_nmi() || !in_atomic());
+
+ VM_BUG_ON(cr3 != __read_cr3());
+ return cr3;
+}
+
#endif /* _ASM_X86_MMU_CONTEXT_H */
#define _ASM_X86_MSHYPER_H
#include <linux/types.h>
-#include <linux/interrupt.h>
-#include <linux/clocksource.h>
+#include <linux/atomic.h>
#include <asm/hyperv.h>
/*
}
}
-#define hv_get_current_tick(tick) rdmsrl(HV_X64_MSR_TIME_REF_COUNT, tick)
#define hv_init_timer(timer, tick) wrmsrl(timer, tick)
#define hv_init_timer_config(config, val) wrmsrl(config, val)
#define DEBUGCTLMSR_BTS_OFF_OS (1UL << 9)
#define DEBUGCTLMSR_BTS_OFF_USR (1UL << 10)
#define DEBUGCTLMSR_FREEZE_LBRS_ON_PMI (1UL << 11)
+#define DEBUGCTLMSR_FREEZE_IN_SMM_BIT 14
+#define DEBUGCTLMSR_FREEZE_IN_SMM (1UL << DEBUGCTLMSR_FREEZE_IN_SMM_BIT)
#define MSR_PEBS_FRONTEND 0x000003f7
#define HWP_MIN_PERF(x) (x & 0xff)
#define HWP_MAX_PERF(x) ((x & 0xff) << 8)
#define HWP_DESIRED_PERF(x) ((x & 0xff) << 16)
-#define HWP_ENERGY_PERF_PREFERENCE(x) ((x & 0xff) << 24)
-#define HWP_ACTIVITY_WINDOW(x) ((x & 0xff3) << 32)
-#define HWP_PACKAGE_CONTROL(x) ((x & 0x1) << 42)
+#define HWP_ENERGY_PERF_PREFERENCE(x) (((unsigned long long) x & 0xff) << 24)
+#define HWP_EPP_PERFORMANCE 0x00
+#define HWP_EPP_BALANCE_PERFORMANCE 0x80
+#define HWP_EPP_BALANCE_POWERSAVE 0xC0
+#define HWP_EPP_POWERSAVE 0xFF
+#define HWP_ACTIVITY_WINDOW(x) ((unsigned long long)(x & 0xff3) << 32)
+#define HWP_PACKAGE_CONTROL(x) ((unsigned long long)(x & 0x1) << 42)
/* IA32_HWP_STATUS */
#define HWP_GUARANTEED_CHANGE(x) (x & 0x1)
#define MSR_MISC_PWR_MGMT 0x000001aa
#define MSR_IA32_ENERGY_PERF_BIAS 0x000001b0
-#define ENERGY_PERF_BIAS_PERFORMANCE 0
-#define ENERGY_PERF_BIAS_NORMAL 6
-#define ENERGY_PERF_BIAS_POWERSAVE 15
+#define ENERGY_PERF_BIAS_PERFORMANCE 0
+#define ENERGY_PERF_BIAS_BALANCE_PERFORMANCE 4
+#define ENERGY_PERF_BIAS_NORMAL 6
+#define ENERGY_PERF_BIAS_BALANCE_POWERSAVE 8
+#define ENERGY_PERF_BIAS_POWERSAVE 15
#define MSR_IA32_PACKAGE_THERM_STATUS 0x000001b1
PVOP_VCALL1(pv_mmu_ops.write_cr2, x);
}
-static inline unsigned long read_cr3(void)
+static inline unsigned long __read_cr3(void)
{
return PVOP_CALL0(unsigned long, pv_mmu_ops.read_cr3);
}
static inline void paravirt_write_msr(unsigned msr,
unsigned low, unsigned high)
{
- return PVOP_VCALL3(pv_cpu_ops.write_msr, msr, low, high);
+ PVOP_VCALL3(pv_cpu_ops.write_msr, msr, low, high);
}
static inline u64 paravirt_read_msr_safe(unsigned msr, int *err)
}
static inline void flush_tlb_others(const struct cpumask *cpumask,
- struct mm_struct *mm,
- unsigned long start,
- unsigned long end)
+ const struct flush_tlb_info *info)
{
- PVOP_VCALL4(pv_mmu_ops.flush_tlb_others, cpumask, mm, start, end);
+ PVOP_VCALL2(pv_mmu_ops.flush_tlb_others, cpumask, info);
}
static inline int paravirt_pgd_alloc(struct mm_struct *mm)
struct desc_struct;
struct task_struct;
struct cpumask;
+struct flush_tlb_info;
/*
* Wrapper type for pointers to code which uses the non-standard
void (*flush_tlb_kernel)(void);
void (*flush_tlb_single)(unsigned long addr);
void (*flush_tlb_others)(const struct cpumask *cpus,
- struct mm_struct *mm,
- unsigned long start,
- unsigned long end);
+ const struct flush_tlb_info *info);
/* Hooks for allocating and freeing a pagetable top-level */
int (*pgd_alloc)(struct mm_struct *mm);
extern unsigned int pcibios_assign_all_busses(void);
extern int pci_legacy_init(void);
-# ifdef CONFIG_ACPI
-# define x86_default_pci_init pci_acpi_init
-# else
-# define x86_default_pci_init pci_legacy_init
-# endif
#else
-# define pcibios_assign_all_busses() 0
-# define x86_default_pci_init NULL
+static inline int pcibios_assign_all_busses(void) { return 0; }
#endif
extern unsigned long pci_mem_start;
#define __pte_to_swp_entry(pte) ((swp_entry_t){ (pte).pte_high })
#define __swp_entry_to_pte(x) ((pte_t){ { .pte_high = (x).val } })
+#define gup_get_pte gup_get_pte
+/*
+ * WARNING: only to be used in the get_user_pages_fast() implementation.
+ *
+ * With get_user_pages_fast(), we walk down the pagetables without taking
+ * any locks. For this we would like to load the pointers atomically,
+ * but that is not possible (without expensive cmpxchg8b) on PAE. What
+ * we do have is the guarantee that a PTE will only either go from not
+ * present to present, or present to not present or both -- it will not
+ * switch to a completely different present page without a TLB flush in
+ * between; something that we are blocking by holding interrupts off.
+ *
+ * Setting ptes from not present to present goes:
+ *
+ * ptep->pte_high = h;
+ * smp_wmb();
+ * ptep->pte_low = l;
+ *
+ * And present to not present goes:
+ *
+ * ptep->pte_low = 0;
+ * smp_wmb();
+ * ptep->pte_high = 0;
+ *
+ * We must ensure here that the load of pte_low sees 'l' iff pte_high
+ * sees 'h'. We load pte_high *after* loading pte_low, which ensures we
+ * don't see an older value of pte_high. *Then* we recheck pte_low,
+ * which ensures that we haven't picked up a changed pte high. We might
+ * have gotten rubbish values from pte_low and pte_high, but we are
+ * guaranteed that pte_low will not have the present bit set *unless*
+ * it is 'l'. Because get_user_pages_fast() only operates on present ptes
+ * we're safe.
+ */
+static inline pte_t gup_get_pte(pte_t *ptep)
+{
+ pte_t pte;
+
+ do {
+ pte.pte_low = ptep->pte_low;
+ smp_rmb();
+ pte.pte_high = ptep->pte_high;
+ smp_rmb();
+ } while (unlikely(pte.pte_low != ptep->pte_low));
+
+ return pte;
+}
+
#endif /* _ASM_X86_PGTABLE_3LEVEL_H */
return 0;
}
#endif
+
+static inline int pgd_devmap(pgd_t pgd)
+{
+ return 0;
+}
#endif
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
static inline void __meminit init_trampoline_default(void)
{
/* Default trampoline pgd value */
- trampoline_pgd_entry = init_level4_pgt[pgd_index(__PAGE_OFFSET)];
+ trampoline_pgd_entry = init_top_pgt[pgd_index(__PAGE_OFFSET)];
}
# ifdef CONFIG_RANDOMIZE_MEMORY
void __meminit init_trampoline(void);
#endif
}
+static inline bool __pkru_allows_pkey(u16 pkey, bool write)
+{
+ u32 pkru = read_pkru();
+
+ if (!__pkru_allows_read(pkru, pkey))
+ return false;
+ if (write && !__pkru_allows_write(pkru, pkey))
+ return false;
+
+ return true;
+}
+
+/*
+ * 'pteval' can come from a PTE, PMD or PUD. We only check
+ * _PAGE_PRESENT, _PAGE_USER, and _PAGE_RW in here which are the
+ * same value on all 3 types.
+ */
+static inline bool __pte_access_permitted(unsigned long pteval, bool write)
+{
+ unsigned long need_pte_bits = _PAGE_PRESENT|_PAGE_USER;
+
+ if (write)
+ need_pte_bits |= _PAGE_RW;
+
+ if ((pteval & need_pte_bits) != need_pte_bits)
+ return 0;
+
+ return __pkru_allows_pkey(pte_flags_pkey(pteval), write);
+}
+
+#define pte_access_permitted pte_access_permitted
+static inline bool pte_access_permitted(pte_t pte, bool write)
+{
+ return __pte_access_permitted(pte_val(pte), write);
+}
+
+#define pmd_access_permitted pmd_access_permitted
+static inline bool pmd_access_permitted(pmd_t pmd, bool write)
+{
+ return __pte_access_permitted(pmd_val(pmd), write);
+}
+
+#define pud_access_permitted pud_access_permitted
+static inline bool pud_access_permitted(pud_t pud, bool write)
+{
+ return __pte_access_permitted(pud_val(pud), write);
+}
+
#include <asm-generic/pgtable.h>
#endif /* __ASSEMBLY__ */
#include <linux/bitops.h>
#include <linux/threads.h>
+extern p4d_t level4_kernel_pgt[512];
+extern p4d_t level4_ident_pgt[512];
extern pud_t level3_kernel_pgt[512];
extern pud_t level3_ident_pgt[512];
extern pmd_t level2_kernel_pgt[512];
extern pmd_t level2_fixmap_pgt[512];
extern pmd_t level2_ident_pgt[512];
extern pte_t level1_fixmap_pgt[512];
-extern pgd_t init_level4_pgt[];
+extern pgd_t init_top_pgt[];
-#define swapper_pg_dir init_level4_pgt
+#define swapper_pg_dir init_top_pgt
extern void paging_init(void);
extern void init_extra_mapping_uc(unsigned long phys, unsigned long size);
extern void init_extra_mapping_wb(unsigned long phys, unsigned long size);
-#endif /* !__ASSEMBLY__ */
+#define gup_fast_permitted gup_fast_permitted
+static inline bool gup_fast_permitted(unsigned long start, int nr_pages,
+ int write)
+{
+ unsigned long len, end;
+
+ len = (unsigned long)nr_pages << PAGE_SHIFT;
+ end = start + len;
+ if (end < start)
+ return false;
+ if (end >> __VIRTUAL_MASK_SHIFT)
+ return false;
+ return true;
+}
+#endif /* !__ASSEMBLY__ */
#endif /* _ASM_X86_PGTABLE_64_H */
#else
#define X86_VM_MASK 0 /* No VM86 support */
#endif
+
+/*
+ * CR3's layout varies depending on several things.
+ *
+ * If CR4.PCIDE is set (64-bit only), then CR3[11:0] is the address space ID.
+ * If PAE is enabled, then CR3[11:5] is part of the PDPT address
+ * (i.e. it's 32-byte aligned, not page-aligned) and CR3[4:0] is ignored.
+ * Otherwise (non-PAE, non-PCID), CR3[3] is PWT, CR3[4] is PCD, and
+ * CR3[2:0] and CR3[11:5] are ignored.
+ *
+ * In all cases, Linux puts zeros in the low ignored bits and in PWT and PCD.
+ *
+ * CR3[63] is always read as zero. If CR4.PCIDE is set, then CR3[63] may be
+ * written as 1 to prevent the write to CR3 from flushing the TLB.
+ *
+ * On systems with SME, one bit (in a variable position!) is stolen to indicate
+ * that the top-level paging structure is encrypted.
+ *
+ * All of the remaining bits indicate the physical address of the top-level
+ * paging structure.
+ *
+ * CR3_ADDR_MASK is the mask used by read_cr3_pa().
+ */
+#ifdef CONFIG_X86_64
+/* Mask off the address space ID bits. */
+#define CR3_ADDR_MASK 0x7FFFFFFFFFFFF000ull
+#define CR3_PCID_MASK 0xFFFull
+#else
+/*
+ * CR3_ADDR_MASK needs at least bits 31:5 set on PAE systems, and we save
+ * a tiny bit of code size by setting all the bits.
+ */
+#define CR3_ADDR_MASK 0xFFFFFFFFull
+#define CR3_PCID_MASK 0ull
+#endif
+
#endif /* _ASM_X86_PROCESSOR_FLAGS_H */
native_cpuid_reg(ecx)
native_cpuid_reg(edx)
+/*
+ * Friendlier CR3 helpers.
+ */
+static inline unsigned long read_cr3_pa(void)
+{
+ return __read_cr3() & CR3_ADDR_MASK;
+}
+
static inline void load_cr3(pgd_t *pgdir)
{
write_cr3(__pa(pgdir));
#endif /* CONFIG_X86_64 */
-extern unsigned long thread_saved_pc(struct task_struct *tsk);
-
extern void start_thread(struct pt_regs *regs, unsigned long new_ip,
unsigned long new_sp);
}
#endif /* CONFIG_X86_INTEL_MPX */
+#ifdef CONFIG_CPU_SUP_AMD
extern u16 amd_get_nb_id(int cpu);
extern u32 amd_get_nodes_per_socket(void);
+#else
+static inline u16 amd_get_nb_id(int cpu) { return 0; }
+static inline u32 amd_get_nodes_per_socket(void) { return 0; }
+#endif
static inline uint32_t hypervisor_cpuid_base(const char *sig, uint32_t leaves)
{
extern void reserve_standard_io_resources(void);
extern void i386_reserve_resources(void);
-extern void setup_default_timer_irq(void);
#ifdef CONFIG_X86_INTEL_MID
extern void x86_intel_mid_early_setup(void);
asm volatile("mov %0,%%cr2": : "r" (val), "m" (__force_order));
}
-static inline unsigned long native_read_cr3(void)
+static inline unsigned long __native_read_cr3(void)
{
unsigned long val;
asm volatile("mov %%cr3,%0\n\t" : "=r" (val), "=m" (__force_order));
native_write_cr2(x);
}
-static inline unsigned long read_cr3(void)
+/*
+ * Careful! CR3 contains more than just an address. You probably want
+ * read_cr3_pa() instead.
+ */
+static inline unsigned long __read_cr3(void)
{
- return native_read_cr3();
+ return __native_read_cr3();
}
static inline void write_cr3(unsigned long x)
set_debugreg((thread)->debugreg##register, register)
/* routines for saving/restoring kernel state */
-extern int acpi_save_state_mem(void);
-extern char core_restore_code;
-extern char restore_registers;
+extern char core_restore_code[];
+extern char restore_registers[];
#endif /* _ASM_X86_SUSPEND_64_H */
u32 cyc2ns_mul;
u32 cyc2ns_shift;
u64 cyc2ns_offset;
- u32 __count;
- /* u32 hole */
-}; /* 24 bytes -- do not grow */
+}; /* 16 bytes */
-extern struct cyc2ns_data *cyc2ns_read_begin(void);
-extern void cyc2ns_read_end(struct cyc2ns_data *);
+extern void cyc2ns_read_begin(struct cyc2ns_data *);
+extern void cyc2ns_read_end(void);
#endif /* _ASM_X86_TIMER_H */
--- /dev/null
+#ifndef _ARCH_X86_TLBBATCH_H
+#define _ARCH_X86_TLBBATCH_H
+
+#include <linux/cpumask.h>
+
+struct arch_tlbflush_unmap_batch {
+ /*
+ * Each bit set is a CPU that potentially has a TLB entry for one of
+ * the PFNs being flushed..
+ */
+ struct cpumask cpumask;
+};
+
+#endif /* _ARCH_X86_TLBBATCH_H */
#include <asm/processor.h>
#include <asm/cpufeature.h>
#include <asm/special_insns.h>
+#include <asm/smp.h>
static inline void __invpcid(unsigned long pcid, unsigned long addr,
unsigned long type)
#endif
struct tlb_state {
-#ifdef CONFIG_SMP
- struct mm_struct *active_mm;
+ /*
+ * cpu_tlbstate.loaded_mm should match CR3 whenever interrupts
+ * are on. This means that it may not match current->active_mm,
+ * which will contain the previous user mm when we're in lazy TLB
+ * mode even if we've already switched back to swapper_pg_dir.
+ */
+ struct mm_struct *loaded_mm;
int state;
-#endif
/*
* Access to this CR4 shadow and to H/W CR4 is protected by
* back:
*/
preempt_disable();
- native_write_cr3(native_read_cr3());
+ native_write_cr3(__native_read_cr3());
preempt_enable();
}
* - flush_tlb_page(vma, vmaddr) flushes one page
* - flush_tlb_range(vma, start, end) flushes a range of pages
* - flush_tlb_kernel_range(start, end) flushes a range of kernel pages
- * - flush_tlb_others(cpumask, mm, start, end) flushes TLBs on other cpus
+ * - flush_tlb_others(cpumask, info) flushes TLBs on other cpus
*
* ..but the i386 has somewhat limited tlb flushing capabilities,
* and page-granular flushes are available only on i486 and up.
*/
-
-#ifndef CONFIG_SMP
-
-/* "_up" is for UniProcessor.
- *
- * This is a helper for other header functions. *Not* intended to be called
- * directly. All global TLB flushes need to either call this, or to bump the
- * vm statistics themselves.
- */
-static inline void __flush_tlb_up(void)
-{
- count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ALL);
- __flush_tlb();
-}
-
-static inline void flush_tlb_all(void)
-{
- count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ALL);
- __flush_tlb_all();
-}
-
-static inline void local_flush_tlb(void)
-{
- __flush_tlb_up();
-}
-
-static inline void flush_tlb_mm(struct mm_struct *mm)
-{
- if (mm == current->active_mm)
- __flush_tlb_up();
-}
-
-static inline void flush_tlb_page(struct vm_area_struct *vma,
- unsigned long addr)
-{
- if (vma->vm_mm == current->active_mm)
- __flush_tlb_one(addr);
-}
-
-static inline void flush_tlb_range(struct vm_area_struct *vma,
- unsigned long start, unsigned long end)
-{
- if (vma->vm_mm == current->active_mm)
- __flush_tlb_up();
-}
-
-static inline void flush_tlb_mm_range(struct mm_struct *mm,
- unsigned long start, unsigned long end, unsigned long vmflag)
-{
- if (mm == current->active_mm)
- __flush_tlb_up();
-}
-
-static inline void native_flush_tlb_others(const struct cpumask *cpumask,
- struct mm_struct *mm,
- unsigned long start,
- unsigned long end)
-{
-}
-
-static inline void reset_lazy_tlbstate(void)
-{
-}
-
-static inline void flush_tlb_kernel_range(unsigned long start,
- unsigned long end)
-{
- flush_tlb_all();
-}
-
-#else /* SMP */
-
-#include <asm/smp.h>
+struct flush_tlb_info {
+ struct mm_struct *mm;
+ unsigned long start;
+ unsigned long end;
+};
#define local_flush_tlb() __flush_tlb()
flush_tlb_mm_range(vma->vm_mm, start, end, vma->vm_flags)
extern void flush_tlb_all(void);
-extern void flush_tlb_page(struct vm_area_struct *, unsigned long);
extern void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
unsigned long end, unsigned long vmflag);
extern void flush_tlb_kernel_range(unsigned long start, unsigned long end);
+static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long a)
+{
+ flush_tlb_mm_range(vma->vm_mm, a, a + PAGE_SIZE, VM_NONE);
+}
+
void native_flush_tlb_others(const struct cpumask *cpumask,
- struct mm_struct *mm,
- unsigned long start, unsigned long end);
+ const struct flush_tlb_info *info);
#define TLBSTATE_OK 1
#define TLBSTATE_LAZY 2
-static inline void reset_lazy_tlbstate(void)
+static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch,
+ struct mm_struct *mm)
{
- this_cpu_write(cpu_tlbstate.state, 0);
- this_cpu_write(cpu_tlbstate.active_mm, &init_mm);
+ cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm));
}
-#endif /* SMP */
+extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch);
#ifndef CONFIG_PARAVIRT
-#define flush_tlb_others(mask, mm, start, end) \
- native_flush_tlb_others(mask, mm, start, end)
+#define flush_tlb_others(mask, info) \
+ native_flush_tlb_others(mask, info)
#endif
#endif /* _ASM_X86_TLBFLUSH_H */
#ifndef _ASM_X86_UV_UV_H
#define _ASM_X86_UV_UV_H
+#include <asm/tlbflush.h>
+
enum uv_system_type {UV_NONE, UV_LEGACY_APIC, UV_X2APIC, UV_NON_UNIQUE_APIC};
struct cpumask;
extern void uv_nmi_init(void);
extern void uv_system_init(void);
extern const struct cpumask *uv_flush_tlb_others(const struct cpumask *cpumask,
- struct mm_struct *mm,
- unsigned long start,
- unsigned long end,
- unsigned int cpu);
+ const struct flush_tlb_info *info);
#else /* X86_UV */
static inline void uv_cpu_init(void) { }
static inline void uv_system_init(void) { }
static inline const struct cpumask *
-uv_flush_tlb_others(const struct cpumask *cpumask, struct mm_struct *mm,
- unsigned long start, unsigned long end, unsigned int cpu)
+uv_flush_tlb_others(const struct cpumask *cpumask,
+ const struct flush_tlb_info *info)
{ return cpumask; }
#endif /* X86_UV */
#define HV_X64_MSR_REFERENCE_TSC 0x40000021
/*
- * There is a single feature flag that signifies the presence of the MSR
- * that can be used to retrieve both the local APIC Timer frequency as
- * well as the TSC frequency.
+ * There is a single feature flag that signifies if the partition has access
+ * to MSRs with local APIC and TSC frequencies.
*/
-
-/* Local APIC timer frequency MSR (HV_X64_MSR_APIC_FREQUENCY) is available */
-#define HV_X64_MSR_APIC_FREQUENCY_AVAILABLE (1 << 11)
-
-/* TSC frequency MSR (HV_X64_MSR_TSC_FREQUENCY) is available */
-#define HV_X64_MSR_TSC_FREQUENCY_AVAILABLE (1 << 11)
+#define HV_X64_ACCESS_FREQUENCY_MSRS (1 << 11)
/*
* Basic SynIC MSRs (HV_X64_MSR_SCONTROL through HV_X64_MSR_EOM
*/
#define HV_X64_MSR_STAT_PAGES_AVAILABLE (1 << 8)
+/* Frequency MSRs available */
+#define HV_FEATURE_FREQUENCY_MSRS_AVAILABLE (1 << 8)
+
/* Crash MSR available */
#define HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE (1 << 10)
#define X86_CR4_OSFXSR _BITUL(X86_CR4_OSFXSR_BIT)
#define X86_CR4_OSXMMEXCPT_BIT 10 /* enable unmasked SSE exceptions */
#define X86_CR4_OSXMMEXCPT _BITUL(X86_CR4_OSXMMEXCPT_BIT)
+#define X86_CR4_LA57_BIT 12 /* enable 5-level page tables */
+#define X86_CR4_LA57 _BITUL(X86_CR4_LA57_BIT)
#define X86_CR4_VMXE_BIT 13 /* enable VMX virtualization */
#define X86_CR4_VMXE _BITUL(X86_CR4_VMXE_BIT)
#define X86_CR4_SMXE_BIT 14 /* enable safer mode (TXT) */
CFLAGS_REMOVE_kvmclock.o = -pg
CFLAGS_REMOVE_ftrace.o = -pg
CFLAGS_REMOVE_early_printk.o = -pg
+CFLAGS_REMOVE_head64.o = -pg
endif
KASAN_SANITIZE_head$(BITS).o := n
OBJECT_FILES_NON_STANDARD_relocate_kernel_$(BITS).o := y
OBJECT_FILES_NON_STANDARD_ftrace_$(BITS).o := y
OBJECT_FILES_NON_STANDARD_test_nx.o := y
+OBJECT_FILES_NON_STANDARD_paravirt_patch_$(BITS).o := y
# If instrumentation of this dir is enabled, boot hangs during first second.
# Probably could be more selective here, but note that files related to irqs,
+OBJECT_FILES_NON_STANDARD_wakeup_$(BITS).o := y
+
obj-$(CONFIG_ACPI) += boot.o
obj-$(CONFIG_ACPI_SLEEP) += sleep.o wakeup_$(BITS).o
obj-$(CONFIG_ACPI_APEI) += apei.o
{
struct cpuinfo_x86 *c = &boot_cpu_data;
- if (c->x86_vendor != X86_VENDOR_INTEL)
+ if (c->x86_vendor != X86_VENDOR_INTEL &&
+ c->x86_vendor != X86_VENDOR_AMD)
return -1;
cpu_cstate_entry = alloc_percpu(struct cstate_entry);
#include <asm/mce.h>
#include <asm/tsc.h>
#include <asm/hypervisor.h>
+#include <asm/cpu_device_id.h>
+#include <asm/intel-family.h>
unsigned int num_processors;
};
static DEFINE_PER_CPU(struct clock_event_device, lapic_events);
+#define DEADLINE_MODEL_MATCH_FUNC(model, func) \
+ { X86_VENDOR_INTEL, 6, model, X86_FEATURE_ANY, (unsigned long)&func }
+
+#define DEADLINE_MODEL_MATCH_REV(model, rev) \
+ { X86_VENDOR_INTEL, 6, model, X86_FEATURE_ANY, (unsigned long)rev }
+
+static u32 hsx_deadline_rev(void)
+{
+ switch (boot_cpu_data.x86_mask) {
+ case 0x02: return 0x3a; /* EP */
+ case 0x04: return 0x0f; /* EX */
+ }
+
+ return ~0U;
+}
+
+static u32 bdx_deadline_rev(void)
+{
+ switch (boot_cpu_data.x86_mask) {
+ case 0x02: return 0x00000011;
+ case 0x03: return 0x0700000e;
+ case 0x04: return 0x0f00000c;
+ case 0x05: return 0x0e000003;
+ }
+
+ return ~0U;
+}
+
+static const struct x86_cpu_id deadline_match[] = {
+ DEADLINE_MODEL_MATCH_FUNC( INTEL_FAM6_HASWELL_X, hsx_deadline_rev),
+ DEADLINE_MODEL_MATCH_REV ( INTEL_FAM6_BROADWELL_X, 0x0b000020),
+ DEADLINE_MODEL_MATCH_FUNC( INTEL_FAM6_BROADWELL_XEON_D, bdx_deadline_rev),
+ DEADLINE_MODEL_MATCH_REV ( INTEL_FAM6_SKYLAKE_X, 0x02000014),
+
+ DEADLINE_MODEL_MATCH_REV ( INTEL_FAM6_HASWELL_CORE, 0x22),
+ DEADLINE_MODEL_MATCH_REV ( INTEL_FAM6_HASWELL_ULT, 0x20),
+ DEADLINE_MODEL_MATCH_REV ( INTEL_FAM6_HASWELL_GT3E, 0x17),
+
+ DEADLINE_MODEL_MATCH_REV ( INTEL_FAM6_BROADWELL_CORE, 0x25),
+ DEADLINE_MODEL_MATCH_REV ( INTEL_FAM6_BROADWELL_GT3E, 0x17),
+
+ DEADLINE_MODEL_MATCH_REV ( INTEL_FAM6_SKYLAKE_MOBILE, 0xb2),
+ DEADLINE_MODEL_MATCH_REV ( INTEL_FAM6_SKYLAKE_DESKTOP, 0xb2),
+
+ DEADLINE_MODEL_MATCH_REV ( INTEL_FAM6_KABYLAKE_MOBILE, 0x52),
+ DEADLINE_MODEL_MATCH_REV ( INTEL_FAM6_KABYLAKE_DESKTOP, 0x52),
+
+ {},
+};
+
+static void apic_check_deadline_errata(void)
+{
+ const struct x86_cpu_id *m = x86_match_cpu(deadline_match);
+ u32 rev;
+
+ if (!m)
+ return;
+
+ /*
+ * Function pointers will have the MSB set due to address layout,
+ * immediate revisions will not.
+ */
+ if ((long)m->driver_data < 0)
+ rev = ((u32 (*)(void))(m->driver_data))();
+ else
+ rev = (u32)m->driver_data;
+
+ if (boot_cpu_data.microcode >= rev)
+ return;
+
+ setup_clear_cpu_cap(X86_FEATURE_TSC_DEADLINE_TIMER);
+ pr_err(FW_BUG "TSC_DEADLINE disabled due to Errata; "
+ "please update microcode to version: 0x%x (or later)\n", rev);
+}
+
/*
* Setup the local APIC timer for this CPU. Copy the initialized values
* of the boot CPU and register the clock event in the framework.
levt->cpumask = cpumask_of(smp_processor_id());
if (this_cpu_has(X86_FEATURE_TSC_DEADLINE_TIMER)) {
+ levt->name = "lapic-deadline";
levt->features &= ~(CLOCK_EVT_FEAT_PERIODIC |
CLOCK_EVT_FEAT_DUMMY);
levt->set_next_event = lapic_next_deadline;
{
unsigned int new_apicid;
+ apic_check_deadline_errata();
+
if (x2apic_mode) {
boot_cpu_physical_apicid = read_apic_id();
return;
apic_write(APIC_LDR, val);
}
-int default_cpu_mask_to_apicid_and(const struct cpumask *cpumask,
- const struct cpumask *andmask,
- unsigned int *apicid)
+int default_cpu_mask_to_apicid(const struct cpumask *mask,
+ struct irq_data *irqdata,
+ unsigned int *apicid)
{
- unsigned int cpu;
+ unsigned int cpu = cpumask_first(mask);
- for_each_cpu_and(cpu, cpumask, andmask) {
- if (cpumask_test_cpu(cpu, cpu_online_mask))
- break;
- }
+ if (cpu >= nr_cpu_ids)
+ return -EINVAL;
+ *apicid = per_cpu(x86_cpu_to_apicid, cpu);
+ irq_data_update_effective_affinity(irqdata, cpumask_of(cpu));
+ return 0;
+}
- if (likely(cpu < nr_cpu_ids)) {
- *apicid = per_cpu(x86_cpu_to_apicid, cpu);
- return 0;
- }
+int flat_cpu_mask_to_apicid(const struct cpumask *mask,
+ struct irq_data *irqdata,
+ unsigned int *apicid)
- return -EINVAL;
+{
+ struct cpumask *effmsk = irq_data_get_effective_affinity_mask(irqdata);
+ unsigned long cpu_mask = cpumask_bits(mask)[0] & APIC_ALL_CPUS;
+
+ if (!cpu_mask)
+ return -EINVAL;
+ *apicid = (unsigned int)cpu_mask;
+ cpumask_bits(effmsk)[0] = cpu_mask;
+ return 0;
}
/*
.get_apic_id = flat_get_apic_id,
.set_apic_id = set_apic_id,
- .cpu_mask_to_apicid_and = flat_cpu_mask_to_apicid_and,
+ .cpu_mask_to_apicid = flat_cpu_mask_to_apicid,
.send_IPI = default_send_IPI_single,
.send_IPI_mask = flat_send_IPI_mask,
.get_apic_id = flat_get_apic_id,
.set_apic_id = set_apic_id,
- .cpu_mask_to_apicid_and = default_cpu_mask_to_apicid_and,
+ .cpu_mask_to_apicid = default_cpu_mask_to_apicid,
.send_IPI = default_send_IPI_single_phys,
.send_IPI_mask = default_send_IPI_mask_sequence_phys,
.get_apic_id = noop_get_apic_id,
.set_apic_id = NULL,
- .cpu_mask_to_apicid_and = flat_cpu_mask_to_apicid_and,
+ .cpu_mask_to_apicid = flat_cpu_mask_to_apicid,
.send_IPI = noop_send_IPI,
.send_IPI_mask = noop_send_IPI_mask,
.get_apic_id = numachip1_get_apic_id,
.set_apic_id = numachip1_set_apic_id,
- .cpu_mask_to_apicid_and = default_cpu_mask_to_apicid_and,
+ .cpu_mask_to_apicid = default_cpu_mask_to_apicid,
.send_IPI = numachip_send_IPI_one,
.send_IPI_mask = numachip_send_IPI_mask,
.get_apic_id = numachip2_get_apic_id,
.set_apic_id = numachip2_set_apic_id,
- .cpu_mask_to_apicid_and = default_cpu_mask_to_apicid_and,
+ .cpu_mask_to_apicid = default_cpu_mask_to_apicid,
.send_IPI = numachip_send_IPI_one,
.send_IPI_mask = numachip_send_IPI_mask,
.get_apic_id = bigsmp_get_apic_id,
.set_apic_id = NULL,
- .cpu_mask_to_apicid_and = default_cpu_mask_to_apicid_and,
+ .cpu_mask_to_apicid = default_cpu_mask_to_apicid,
.send_IPI = default_send_IPI_single_phys,
.send_IPI_mask = default_send_IPI_mask_sequence_phys,
.deactivate = htirq_domain_deactivate,
};
-void arch_init_htirq_domain(struct irq_domain *parent)
+void __init arch_init_htirq_domain(struct irq_domain *parent)
{
+ struct fwnode_handle *fn;
+
if (disable_apic)
return;
- htirq_domain = irq_domain_add_tree(NULL, &htirq_domain_ops, NULL);
+ fn = irq_domain_alloc_named_fwnode("PCI-HT");
+ if (!fn)
+ goto warn;
+
+ htirq_domain = irq_domain_create_tree(fn, &htirq_domain_ops, NULL);
+ irq_domain_free_fwnode(fn);
if (!htirq_domain)
- pr_warn("failed to initialize irqdomain for HTIRQ.\n");
- else
- htirq_domain->parent = parent;
+ goto warn;
+
+ htirq_domain->parent = parent;
+ return;
+
+warn:
+ pr_warn("Failed to initialize irqdomain for HTIRQ.\n");
}
int arch_setup_ht_irq(int idx, int pos, struct pci_dev *dev,
static struct irq_chip ioapic_chip, ioapic_ir_chip;
-#ifdef CONFIG_X86_32
-static inline int IO_APIC_irq_trigger(int irq)
-{
- int apic, idx, pin;
-
- for_each_ioapic_pin(apic, pin) {
- idx = find_irq_entry(apic, pin, mp_INT);
- if ((idx != -1) && (irq == pin_2_irq(idx, apic, pin, 0)))
- return irq_trigger(idx);
- }
- /*
- * nonexistent IRQs are edge default
- */
- return 0;
-}
-#else
-static inline int IO_APIC_irq_trigger(int irq)
-{
- return 1;
-}
-#endif
-
static void __init setup_IO_APIC_irqs(void)
{
unsigned int ioapic, pin;
struct ioapic *ip = &ioapics[ioapic];
struct ioapic_domain_cfg *cfg = &ip->irqdomain_cfg;
struct mp_ioapic_gsi *gsi_cfg = mp_ioapic_gsi_routing(ioapic);
+ struct fwnode_handle *fn;
+ char *name = "IO-APIC";
if (cfg->type == IOAPIC_DOMAIN_INVALID)
return 0;
parent = irq_remapping_get_ir_irq_domain(&info);
if (!parent)
parent = x86_vector_domain;
+ else
+ name = "IO-APIC-IR";
+
+ /* Handle device tree enumerated APICs proper */
+ if (cfg->dev) {
+ fn = of_node_to_fwnode(cfg->dev);
+ } else {
+ fn = irq_domain_alloc_named_id_fwnode(name, ioapic);
+ if (!fn)
+ return -ENOMEM;
+ }
+
+ ip->irqdomain = irq_domain_create_linear(fn, hwirqs, cfg->ops,
+ (void *)(long)ioapic);
+
+ /* Release fw handle if it was allocated above */
+ if (!cfg->dev)
+ irq_domain_free_fwnode(fn);
- ip->irqdomain = irq_domain_add_linear(cfg->dev, hwirqs, cfg->ops,
- (void *)(long)ioapic);
if (!ip->irqdomain)
return -ENOMEM;
.handler_name = "edge",
};
-void arch_init_msi_domain(struct irq_domain *parent)
+void __init arch_init_msi_domain(struct irq_domain *parent)
{
+ struct fwnode_handle *fn;
+
if (disable_apic)
return;
- msi_default_domain = pci_msi_create_irq_domain(NULL,
- &pci_msi_domain_info, parent);
+ fn = irq_domain_alloc_named_fwnode("PCI-MSI");
+ if (fn) {
+ msi_default_domain =
+ pci_msi_create_irq_domain(fn, &pci_msi_domain_info,
+ parent);
+ irq_domain_free_fwnode(fn);
+ }
if (!msi_default_domain)
pr_warn("failed to initialize irqdomain for MSI/MSI-x.\n");
}
.handler_name = "edge",
};
-struct irq_domain *arch_create_msi_irq_domain(struct irq_domain *parent)
+struct irq_domain *arch_create_remap_msi_irq_domain(struct irq_domain *parent,
+ const char *name, int id)
{
- return pci_msi_create_irq_domain(NULL, &pci_msi_ir_domain_info, parent);
+ struct fwnode_handle *fn;
+ struct irq_domain *d;
+
+ fn = irq_domain_alloc_named_id_fwnode(name, id);
+ if (!fn)
+ return NULL;
+ d = pci_msi_create_irq_domain(fn, &pci_msi_ir_domain_info, parent);
+ irq_domain_free_fwnode(fn);
+ return d;
}
#endif
{
static struct irq_domain *dmar_domain;
static DEFINE_MUTEX(dmar_lock);
+ struct fwnode_handle *fn;
mutex_lock(&dmar_lock);
- if (dmar_domain == NULL)
- dmar_domain = msi_create_irq_domain(NULL, &dmar_msi_domain_info,
+ if (dmar_domain)
+ goto out;
+
+ fn = irq_domain_alloc_named_fwnode("DMAR-MSI");
+ if (fn) {
+ dmar_domain = msi_create_irq_domain(fn, &dmar_msi_domain_info,
x86_vector_domain);
+ irq_domain_free_fwnode(fn);
+ }
+out:
mutex_unlock(&dmar_lock);
-
return dmar_domain;
}
struct irq_domain *hpet_create_irq_domain(int hpet_id)
{
- struct irq_domain *parent;
- struct irq_alloc_info info;
struct msi_domain_info *domain_info;
+ struct irq_domain *parent, *d;
+ struct irq_alloc_info info;
+ struct fwnode_handle *fn;
if (x86_vector_domain == NULL)
return NULL;
else
hpet_msi_controller.name = "IR-HPET-MSI";
- return msi_create_irq_domain(NULL, domain_info, parent);
+ fn = irq_domain_alloc_named_id_fwnode(hpet_msi_controller.name,
+ hpet_id);
+ if (!fn) {
+ kfree(domain_info);
+ return NULL;
+ }
+
+ d = msi_create_irq_domain(fn, domain_info, parent);
+ irq_domain_free_fwnode(fn);
+ return d;
}
int hpet_assign_irq(struct irq_domain *domain, struct hpet_dev *dev,
.get_apic_id = default_get_apic_id,
.set_apic_id = NULL,
- .cpu_mask_to_apicid_and = flat_cpu_mask_to_apicid_and,
+ .cpu_mask_to_apicid = flat_cpu_mask_to_apicid,
.send_IPI = default_send_IPI_single,
.send_IPI_mask = default_send_IPI_mask_logical,
}
static int __assign_irq_vector(int irq, struct apic_chip_data *d,
- const struct cpumask *mask)
+ const struct cpumask *mask,
+ struct irq_data *irqdata)
{
/*
* NOTE! The local APIC isn't very good at handling
/*
* Clear the offline cpus from @vector_cpumask for searching
* and verify whether the result overlaps with @mask. If true,
- * then the call to apic->cpu_mask_to_apicid_and() will
+ * then the call to apic->cpu_mask_to_apicid() will
* succeed as well. If not, no point in trying to find a
* vector in this mask.
*/
* Cache destination APIC IDs into cfg->dest_apicid. This cannot fail
* as we already established, that mask & d->domain & cpu_online_mask
* is not empty.
+ *
+ * vector_searchmask is a subset of d->domain and has the offline
+ * cpus masked out.
*/
- BUG_ON(apic->cpu_mask_to_apicid_and(mask, d->domain,
- &d->cfg.dest_apicid));
+ cpumask_and(vector_searchmask, vector_searchmask, mask);
+ BUG_ON(apic->cpu_mask_to_apicid(vector_searchmask, irqdata,
+ &d->cfg.dest_apicid));
return 0;
}
static int assign_irq_vector(int irq, struct apic_chip_data *data,
- const struct cpumask *mask)
+ const struct cpumask *mask,
+ struct irq_data *irqdata)
{
int err;
unsigned long flags;
raw_spin_lock_irqsave(&vector_lock, flags);
- err = __assign_irq_vector(irq, data, mask);
+ err = __assign_irq_vector(irq, data, mask, irqdata);
raw_spin_unlock_irqrestore(&vector_lock, flags);
return err;
}
static int assign_irq_vector_policy(int irq, int node,
struct apic_chip_data *data,
- struct irq_alloc_info *info)
+ struct irq_alloc_info *info,
+ struct irq_data *irqdata)
{
if (info && info->mask)
- return assign_irq_vector(irq, data, info->mask);
+ return assign_irq_vector(irq, data, info->mask, irqdata);
if (node != NUMA_NO_NODE &&
- assign_irq_vector(irq, data, cpumask_of_node(node)) == 0)
+ assign_irq_vector(irq, data, cpumask_of_node(node), irqdata) == 0)
return 0;
- return assign_irq_vector(irq, data, apic->target_cpus());
+ return assign_irq_vector(irq, data, apic->target_cpus(), irqdata);
}
static void clear_irq_vector(int irq, struct apic_chip_data *data)
irq_data->chip = &lapic_controller;
irq_data->chip_data = data;
irq_data->hwirq = virq + i;
- err = assign_irq_vector_policy(virq + i, node, data, info);
+ err = assign_irq_vector_policy(virq + i, node, data, info,
+ irq_data);
if (err)
goto error;
+ /*
+ * If the apic destination mode is physical, then the
+ * effective affinity is restricted to a single target
+ * CPU. Mark the interrupt accordingly.
+ */
+ if (!apic->irq_dest_mode)
+ irqd_set_single_target(irq_data);
}
return 0;
}
#ifdef CONFIG_X86_IO_APIC
-static void init_legacy_irqs(void)
+static void __init init_legacy_irqs(void)
{
int i, node = cpu_to_node(0);
struct apic_chip_data *data;
}
}
#else
-static void init_legacy_irqs(void) { }
+static inline void init_legacy_irqs(void) { }
#endif
int __init arch_early_irq_init(void)
{
+ struct fwnode_handle *fn;
+
init_legacy_irqs();
- x86_vector_domain = irq_domain_add_tree(NULL, &x86_vector_domain_ops,
- NULL);
+ fn = irq_domain_alloc_named_fwnode("VECTOR");
+ BUG_ON(!fn);
+ x86_vector_domain = irq_domain_create_tree(fn, &x86_vector_domain_ops,
+ NULL);
BUG_ON(x86_vector_domain == NULL);
+ irq_domain_free_fwnode(fn);
irq_set_default_host(x86_vector_domain);
arch_init_msi_domain(x86_vector_domain);
if (!cpumask_intersects(dest, cpu_online_mask))
return -EINVAL;
- err = assign_irq_vector(irq, data, dest);
+ err = assign_irq_vector(irq, data, dest, irq_data);
return err ? err : IRQ_SET_MASK_OK;
}
static struct irq_chip lapic_controller = {
+ .name = "APIC",
.irq_ack = apic_ack_edge,
.irq_set_affinity = apic_set_affinity,
.irq_retrigger = apic_retrigger_irq,
#include <linux/kernel.h>
#include <linux/ctype.h>
#include <linux/dmar.h>
+#include <linux/irq.h>
#include <linux/cpu.h>
#include <asm/smp.h>
}
static int
-x2apic_cpu_mask_to_apicid_and(const struct cpumask *cpumask,
- const struct cpumask *andmask,
- unsigned int *apicid)
+x2apic_cpu_mask_to_apicid(const struct cpumask *mask, struct irq_data *irqdata,
+ unsigned int *apicid)
{
+ struct cpumask *effmsk = irq_data_get_effective_affinity_mask(irqdata);
+ unsigned int cpu;
u32 dest = 0;
u16 cluster;
- int i;
-
- for_each_cpu_and(i, cpumask, andmask) {
- if (!cpumask_test_cpu(i, cpu_online_mask))
- continue;
- dest = per_cpu(x86_cpu_to_logical_apicid, i);
- cluster = x2apic_cluster(i);
- break;
- }
- if (!dest)
+ cpu = cpumask_first(mask);
+ if (cpu >= nr_cpu_ids)
return -EINVAL;
- for_each_cpu_and(i, cpumask, andmask) {
- if (!cpumask_test_cpu(i, cpu_online_mask))
- continue;
- if (cluster != x2apic_cluster(i))
+ dest = per_cpu(x86_cpu_to_logical_apicid, cpu);
+ cluster = x2apic_cluster(cpu);
+
+ cpumask_clear(effmsk);
+ for_each_cpu(cpu, mask) {
+ if (cluster != x2apic_cluster(cpu))
continue;
- dest |= per_cpu(x86_cpu_to_logical_apicid, i);
+ dest |= per_cpu(x86_cpu_to_logical_apicid, cpu);
+ cpumask_set_cpu(cpu, effmsk);
}
*apicid = dest;
-
return 0;
}
.get_apic_id = x2apic_get_apic_id,
.set_apic_id = x2apic_set_apic_id,
- .cpu_mask_to_apicid_and = x2apic_cpu_mask_to_apicid_and,
+ .cpu_mask_to_apicid = x2apic_cpu_mask_to_apicid,
.send_IPI = x2apic_send_IPI,
.send_IPI_mask = x2apic_send_IPI_mask,
.get_apic_id = x2apic_get_apic_id,
.set_apic_id = x2apic_set_apic_id,
- .cpu_mask_to_apicid_and = default_cpu_mask_to_apicid_and,
+ .cpu_mask_to_apicid = default_cpu_mask_to_apicid,
.send_IPI = x2apic_send_IPI,
.send_IPI_mask = x2apic_send_IPI_mask,
}
static int
-uv_cpu_mask_to_apicid_and(const struct cpumask *cpumask,
- const struct cpumask *andmask,
- unsigned int *apicid)
+uv_cpu_mask_to_apicid(const struct cpumask *mask, struct irq_data *irqdata,
+ unsigned int *apicid)
{
- int unsigned cpu;
+ int ret = default_cpu_mask_to_apicid(mask, irqdata, apicid);
- /*
- * We're using fixed IRQ delivery, can only return one phys APIC ID.
- * May as well be the first.
- */
- for_each_cpu_and(cpu, cpumask, andmask) {
- if (cpumask_test_cpu(cpu, cpu_online_mask))
- break;
- }
-
- if (likely(cpu < nr_cpu_ids)) {
- *apicid = per_cpu(x86_cpu_to_apicid, cpu) | uv_apicid_hibits;
- return 0;
- }
+ if (!ret)
+ *apicid |= uv_apicid_hibits;
- return -EINVAL;
+ return ret;
}
static unsigned int x2apic_get_apic_id(unsigned long x)
.get_apic_id = x2apic_get_apic_id,
.set_apic_id = set_apic_id,
- .cpu_mask_to_apicid_and = uv_cpu_mask_to_apicid_and,
+ .cpu_mask_to_apicid = uv_cpu_mask_to_apicid,
.send_IPI = uv_send_IPI_one,
.send_IPI_mask = uv_send_IPI_mask,
obj-y += rdrand.o
obj-y += match.o
obj-y += bugs.o
+obj-$(CONFIG_CPU_FREQ) += aperfmperf.o
obj-$(CONFIG_PROC_FS) += proc.o
obj-$(CONFIG_X86_FEATURE_NAMES) += capflags.o powerflags.o
--- /dev/null
+/*
+ * x86 APERF/MPERF KHz calculation for
+ * /sys/.../cpufreq/scaling_cur_freq
+ *
+ * Copyright (C) 2017 Intel Corp.
+ * Author: Len Brown <len.brown@intel.com>
+ *
+ * This file is licensed under GPLv2.
+ */
+
+#include <linux/jiffies.h>
+#include <linux/math64.h>
+#include <linux/percpu.h>
+#include <linux/smp.h>
+
+struct aperfmperf_sample {
+ unsigned int khz;
+ unsigned long jiffies;
+ u64 aperf;
+ u64 mperf;
+};
+
+static DEFINE_PER_CPU(struct aperfmperf_sample, samples);
+
+/*
+ * aperfmperf_snapshot_khz()
+ * On the current CPU, snapshot APERF, MPERF, and jiffies
+ * unless we already did it within 10ms
+ * calculate kHz, save snapshot
+ */
+static void aperfmperf_snapshot_khz(void *dummy)
+{
+ u64 aperf, aperf_delta;
+ u64 mperf, mperf_delta;
+ struct aperfmperf_sample *s = this_cpu_ptr(&samples);
+
+ /* Don't bother re-computing within 10 ms */
+ if (time_before(jiffies, s->jiffies + HZ/100))
+ return;
+
+ rdmsrl(MSR_IA32_APERF, aperf);
+ rdmsrl(MSR_IA32_MPERF, mperf);
+
+ aperf_delta = aperf - s->aperf;
+ mperf_delta = mperf - s->mperf;
+
+ /*
+ * There is no architectural guarantee that MPERF
+ * increments faster than we can read it.
+ */
+ if (mperf_delta == 0)
+ return;
+
+ /*
+ * if (cpu_khz * aperf_delta) fits into ULLONG_MAX, then
+ * khz = (cpu_khz * aperf_delta) / mperf_delta
+ */
+ if (div64_u64(ULLONG_MAX, cpu_khz) > aperf_delta)
+ s->khz = div64_u64((cpu_khz * aperf_delta), mperf_delta);
+ else /* khz = aperf_delta / (mperf_delta / cpu_khz) */
+ s->khz = div64_u64(aperf_delta,
+ div64_u64(mperf_delta, cpu_khz));
+ s->jiffies = jiffies;
+ s->aperf = aperf;
+ s->mperf = mperf;
+}
+
+unsigned int arch_freq_get_on_cpu(int cpu)
+{
+ if (!cpu_khz)
+ return 0;
+
+ if (!static_cpu_has(X86_FEATURE_APERFMPERF))
+ return 0;
+
+ smp_call_function_single(cpu, aperfmperf_snapshot_khz, NULL, 1);
+
+ return per_cpu(samples.khz, cpu);
+}
dentry = kernfs_mount(fs_type, flags, rdt_root,
RDTGROUP_SUPER_MAGIC, NULL);
if (IS_ERR(dentry))
- goto out_cdp;
+ goto out_destroy;
static_branch_enable(&rdt_enable_key);
goto out;
+out_destroy:
+ kernfs_remove(kn_info);
out_cdp:
cdp_disable();
out:
#include "mce-internal.h"
+static BLOCKING_NOTIFIER_HEAD(mce_injector_chain);
+
static DEFINE_MUTEX(mce_chrdev_read_mutex);
static char mce_helper[128];
}
}
-static ssize_t (*mce_write)(struct file *filp, const char __user *ubuf,
- size_t usize, loff_t *off);
+void mce_register_injector_chain(struct notifier_block *nb)
+{
+ blocking_notifier_chain_register(&mce_injector_chain, nb);
+}
+EXPORT_SYMBOL_GPL(mce_register_injector_chain);
-void register_mce_write_callback(ssize_t (*fn)(struct file *filp,
- const char __user *ubuf,
- size_t usize, loff_t *off))
+void mce_unregister_injector_chain(struct notifier_block *nb)
{
- mce_write = fn;
+ blocking_notifier_chain_unregister(&mce_injector_chain, nb);
}
-EXPORT_SYMBOL_GPL(register_mce_write_callback);
+EXPORT_SYMBOL_GPL(mce_unregister_injector_chain);
static ssize_t mce_chrdev_write(struct file *filp, const char __user *ubuf,
size_t usize, loff_t *off)
{
- if (mce_write)
- return mce_write(filp, ubuf, usize, off);
- else
+ struct mce m;
+
+ if (!capable(CAP_SYS_ADMIN))
+ return -EPERM;
+ /*
+ * There are some cases where real MSR reads could slip
+ * through.
+ */
+ if (!boot_cpu_has(X86_FEATURE_MCE) || !boot_cpu_has(X86_FEATURE_MCA))
+ return -EIO;
+
+ if ((unsigned long)usize > sizeof(struct mce))
+ usize = sizeof(struct mce);
+ if (copy_from_user(&m, ubuf, usize))
+ return -EFAULT;
+
+ if (m.extcpu >= num_possible_cpus() || !cpu_online(m.extcpu))
return -EINVAL;
+
+ /*
+ * Need to give user space some time to set everything up,
+ * so do it a jiffie or two later everywhere.
+ */
+ schedule_timeout(2);
+
+ blocking_notifier_call_chain(&mce_injector_chain, 0, &m);
+
+ return usize;
}
static const struct file_operations mce_chrdev_ops = {
/* register character device /dev/mcelog */
err = misc_register(&mce_chrdev_device);
if (err) {
- pr_err("Unable to init device /dev/mcelog (rc: %d)\n", err);
+ if (err == -EBUSY)
+ /* Xen dom0 might have registered the device already. */
+ pr_info("Unable to init device /dev/mcelog, already registered");
+ else
+ pr_err("Unable to init device /dev/mcelog (rc: %d)\n", err);
+
return err;
}
+
mce_register_decode_chain(&dev_mcelog_nb);
return 0;
}
* Authors:
* Andi Kleen
* Ying Huang
+ *
+ * The AMD part (from mce_amd_inj.c): a simple MCE injection facility
+ * for testing different aspects of the RAS code. This driver should be
+ * built as module so that it can be loaded on production kernels for
+ * testing purposes.
+ *
+ * This file may be distributed under the terms of the GNU General Public
+ * License version 2.
+ *
+ * Copyright (c) 2010-17: Borislav Petkov <bp@alien8.de>
+ * Advanced Micro Devices Inc.
*/
-#include <linux/uaccess.h>
-#include <linux/module.h>
-#include <linux/timer.h>
+
+#include <linux/cpu.h>
+#include <linux/debugfs.h>
#include <linux/kernel.h>
-#include <linux/string.h>
-#include <linux/fs.h>
-#include <linux/preempt.h>
-#include <linux/smp.h>
+#include <linux/module.h>
#include <linux/notifier.h>
-#include <linux/kdebug.h>
-#include <linux/cpu.h>
-#include <linux/sched.h>
-#include <linux/gfp.h>
-#include <asm/mce.h>
+#include <linux/pci.h>
+#include <linux/uaccess.h>
+
+#include <asm/amd_nb.h>
#include <asm/apic.h>
+#include <asm/irq_vectors.h>
+#include <asm/mce.h>
#include <asm/nmi.h>
+#include <asm/smp.h>
+
+#include "mce-internal.h"
+
+/*
+ * Collect all the MCi_XXX settings
+ */
+static struct mce i_mce;
+static struct dentry *dfs_inj;
+
+static u8 n_banks;
+
+#define MAX_FLAG_OPT_SIZE 3
+#define NBCFG 0x44
+
+enum injection_type {
+ SW_INJ = 0, /* SW injection, simply decode the error */
+ HW_INJ, /* Trigger a #MC */
+ DFR_INT_INJ, /* Trigger Deferred error interrupt */
+ THR_INT_INJ, /* Trigger threshold interrupt */
+ N_INJ_TYPES,
+};
+
+static const char * const flags_options[] = {
+ [SW_INJ] = "sw",
+ [HW_INJ] = "hw",
+ [DFR_INT_INJ] = "df",
+ [THR_INT_INJ] = "th",
+ NULL
+};
+
+/* Set default injection to SW_INJ */
+static enum injection_type inj_type = SW_INJ;
+
+#define MCE_INJECT_SET(reg) \
+static int inj_##reg##_set(void *data, u64 val) \
+{ \
+ struct mce *m = (struct mce *)data; \
+ \
+ m->reg = val; \
+ return 0; \
+}
+
+MCE_INJECT_SET(status);
+MCE_INJECT_SET(misc);
+MCE_INJECT_SET(addr);
+MCE_INJECT_SET(synd);
+
+#define MCE_INJECT_GET(reg) \
+static int inj_##reg##_get(void *data, u64 *val) \
+{ \
+ struct mce *m = (struct mce *)data; \
+ \
+ *val = m->reg; \
+ return 0; \
+}
+
+MCE_INJECT_GET(status);
+MCE_INJECT_GET(misc);
+MCE_INJECT_GET(addr);
+MCE_INJECT_GET(synd);
+
+DEFINE_SIMPLE_ATTRIBUTE(status_fops, inj_status_get, inj_status_set, "%llx\n");
+DEFINE_SIMPLE_ATTRIBUTE(misc_fops, inj_misc_get, inj_misc_set, "%llx\n");
+DEFINE_SIMPLE_ATTRIBUTE(addr_fops, inj_addr_get, inj_addr_set, "%llx\n");
+DEFINE_SIMPLE_ATTRIBUTE(synd_fops, inj_synd_get, inj_synd_set, "%llx\n");
+
+static void setup_inj_struct(struct mce *m)
+{
+ memset(m, 0, sizeof(struct mce));
+
+ m->cpuvendor = boot_cpu_data.x86_vendor;
+}
/* Update fake mce registers on current CPU. */
static void inject_mce(struct mce *m)
return ret;
}
-static void raise_mce(struct mce *m)
+static void __maybe_unused raise_mce(struct mce *m)
{
int context = MCJ_CTX(m->inject_flags);
}
}
-/* Error injection interface */
-static ssize_t mce_write(struct file *filp, const char __user *ubuf,
- size_t usize, loff_t *off)
+static int mce_inject_raise(struct notifier_block *nb, unsigned long val,
+ void *data)
{
- struct mce m;
+ struct mce *m = (struct mce *)data;
- if (!capable(CAP_SYS_ADMIN))
- return -EPERM;
- /*
- * There are some cases where real MSR reads could slip
- * through.
- */
- if (!boot_cpu_has(X86_FEATURE_MCE) || !boot_cpu_has(X86_FEATURE_MCA))
- return -EIO;
+ if (!m)
+ return NOTIFY_DONE;
+
+ mutex_lock(&mce_inject_mutex);
+ raise_mce(m);
+ mutex_unlock(&mce_inject_mutex);
+
+ return NOTIFY_DONE;
+}
+
+static struct notifier_block inject_nb = {
+ .notifier_call = mce_inject_raise,
+};
+
+/*
+ * Caller needs to be make sure this cpu doesn't disappear
+ * from under us, i.e.: get_cpu/put_cpu.
+ */
+static int toggle_hw_mce_inject(unsigned int cpu, bool enable)
+{
+ u32 l, h;
+ int err;
+
+ err = rdmsr_on_cpu(cpu, MSR_K7_HWCR, &l, &h);
+ if (err) {
+ pr_err("%s: error reading HWCR\n", __func__);
+ return err;
+ }
+
+ enable ? (l |= BIT(18)) : (l &= ~BIT(18));
+
+ err = wrmsr_on_cpu(cpu, MSR_K7_HWCR, l, h);
+ if (err)
+ pr_err("%s: error writing HWCR\n", __func__);
- if ((unsigned long)usize > sizeof(struct mce))
- usize = sizeof(struct mce);
- if (copy_from_user(&m, ubuf, usize))
+ return err;
+}
+
+static int __set_inj(const char *buf)
+{
+ int i;
+
+ for (i = 0; i < N_INJ_TYPES; i++) {
+ if (!strncmp(flags_options[i], buf, strlen(flags_options[i]))) {
+ inj_type = i;
+ return 0;
+ }
+ }
+ return -EINVAL;
+}
+
+static ssize_t flags_read(struct file *filp, char __user *ubuf,
+ size_t cnt, loff_t *ppos)
+{
+ char buf[MAX_FLAG_OPT_SIZE];
+ int n;
+
+ n = sprintf(buf, "%s\n", flags_options[inj_type]);
+
+ return simple_read_from_buffer(ubuf, cnt, ppos, buf, n);
+}
+
+static ssize_t flags_write(struct file *filp, const char __user *ubuf,
+ size_t cnt, loff_t *ppos)
+{
+ char buf[MAX_FLAG_OPT_SIZE], *__buf;
+ int err;
+
+ if (cnt > MAX_FLAG_OPT_SIZE)
+ return -EINVAL;
+
+ if (copy_from_user(&buf, ubuf, cnt))
return -EFAULT;
- if (m.extcpu >= num_possible_cpus() || !cpu_online(m.extcpu))
+ buf[cnt - 1] = 0;
+
+ /* strip whitespace */
+ __buf = strstrip(buf);
+
+ err = __set_inj(__buf);
+ if (err) {
+ pr_err("%s: Invalid flags value: %s\n", __func__, __buf);
+ return err;
+ }
+
+ *ppos += cnt;
+
+ return cnt;
+}
+
+static const struct file_operations flags_fops = {
+ .read = flags_read,
+ .write = flags_write,
+ .llseek = generic_file_llseek,
+};
+
+/*
+ * On which CPU to inject?
+ */
+MCE_INJECT_GET(extcpu);
+
+static int inj_extcpu_set(void *data, u64 val)
+{
+ struct mce *m = (struct mce *)data;
+
+ if (val >= nr_cpu_ids || !cpu_online(val)) {
+ pr_err("%s: Invalid CPU: %llu\n", __func__, val);
return -EINVAL;
+ }
+ m->extcpu = val;
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(extcpu_fops, inj_extcpu_get, inj_extcpu_set, "%llu\n");
+
+static void trigger_mce(void *info)
+{
+ asm volatile("int $18");
+}
+
+static void trigger_dfr_int(void *info)
+{
+ asm volatile("int %0" :: "i" (DEFERRED_ERROR_VECTOR));
+}
+
+static void trigger_thr_int(void *info)
+{
+ asm volatile("int %0" :: "i" (THRESHOLD_APIC_VECTOR));
+}
+
+static u32 get_nbc_for_node(int node_id)
+{
+ struct cpuinfo_x86 *c = &boot_cpu_data;
+ u32 cores_per_node;
+
+ cores_per_node = (c->x86_max_cores * smp_num_siblings) / amd_get_nodes_per_socket();
+
+ return cores_per_node * node_id;
+}
+
+static void toggle_nb_mca_mst_cpu(u16 nid)
+{
+ struct amd_northbridge *nb;
+ struct pci_dev *F3;
+ u32 val;
+ int err;
+
+ nb = node_to_amd_nb(nid);
+ if (!nb)
+ return;
+
+ F3 = nb->misc;
+ if (!F3)
+ return;
+
+ err = pci_read_config_dword(F3, NBCFG, &val);
+ if (err) {
+ pr_err("%s: Error reading F%dx%03x.\n",
+ __func__, PCI_FUNC(F3->devfn), NBCFG);
+ return;
+ }
+
+ if (val & BIT(27))
+ return;
+
+ pr_err("%s: Set D18F3x44[NbMcaToMstCpuEn] which BIOS hasn't done.\n",
+ __func__);
+
+ val |= BIT(27);
+ err = pci_write_config_dword(F3, NBCFG, val);
+ if (err)
+ pr_err("%s: Error writing F%dx%03x.\n",
+ __func__, PCI_FUNC(F3->devfn), NBCFG);
+}
+
+static void prepare_msrs(void *info)
+{
+ struct mce m = *(struct mce *)info;
+ u8 b = m.bank;
+
+ wrmsrl(MSR_IA32_MCG_STATUS, m.mcgstatus);
+
+ if (boot_cpu_has(X86_FEATURE_SMCA)) {
+ if (m.inject_flags == DFR_INT_INJ) {
+ wrmsrl(MSR_AMD64_SMCA_MCx_DESTAT(b), m.status);
+ wrmsrl(MSR_AMD64_SMCA_MCx_DEADDR(b), m.addr);
+ } else {
+ wrmsrl(MSR_AMD64_SMCA_MCx_STATUS(b), m.status);
+ wrmsrl(MSR_AMD64_SMCA_MCx_ADDR(b), m.addr);
+ }
+
+ wrmsrl(MSR_AMD64_SMCA_MCx_MISC(b), m.misc);
+ wrmsrl(MSR_AMD64_SMCA_MCx_SYND(b), m.synd);
+ } else {
+ wrmsrl(MSR_IA32_MCx_STATUS(b), m.status);
+ wrmsrl(MSR_IA32_MCx_ADDR(b), m.addr);
+ wrmsrl(MSR_IA32_MCx_MISC(b), m.misc);
+ }
+}
+
+static void do_inject(void)
+{
+ u64 mcg_status = 0;
+ unsigned int cpu = i_mce.extcpu;
+ u8 b = i_mce.bank;
+
+ rdtscll(i_mce.tsc);
+
+ if (i_mce.misc)
+ i_mce.status |= MCI_STATUS_MISCV;
+
+ if (i_mce.synd)
+ i_mce.status |= MCI_STATUS_SYNDV;
+
+ if (inj_type == SW_INJ) {
+ mce_inject_log(&i_mce);
+ return;
+ }
+
+ /* prep MCE global settings for the injection */
+ mcg_status = MCG_STATUS_MCIP | MCG_STATUS_EIPV;
+
+ if (!(i_mce.status & MCI_STATUS_PCC))
+ mcg_status |= MCG_STATUS_RIPV;
/*
- * Need to give user space some time to set everything up,
- * so do it a jiffie or two later everywhere.
+ * Ensure necessary status bits for deferred errors:
+ * - MCx_STATUS[Deferred]: make sure it is a deferred error
+ * - MCx_STATUS[UC] cleared: deferred errors are _not_ UC
*/
- schedule_timeout(2);
+ if (inj_type == DFR_INT_INJ) {
+ i_mce.status |= MCI_STATUS_DEFERRED;
+ i_mce.status |= (i_mce.status & ~MCI_STATUS_UC);
+ }
- mutex_lock(&mce_inject_mutex);
- raise_mce(&m);
- mutex_unlock(&mce_inject_mutex);
- return usize;
+ /*
+ * For multi node CPUs, logging and reporting of bank 4 errors happens
+ * only on the node base core. Refer to D18F3x44[NbMcaToMstCpuEn] for
+ * Fam10h and later BKDGs.
+ */
+ if (static_cpu_has(X86_FEATURE_AMD_DCM) &&
+ b == 4 &&
+ boot_cpu_data.x86 < 0x17) {
+ toggle_nb_mca_mst_cpu(amd_get_nb_id(cpu));
+ cpu = get_nbc_for_node(amd_get_nb_id(cpu));
+ }
+
+ get_online_cpus();
+ if (!cpu_online(cpu))
+ goto err;
+
+ toggle_hw_mce_inject(cpu, true);
+
+ i_mce.mcgstatus = mcg_status;
+ i_mce.inject_flags = inj_type;
+ smp_call_function_single(cpu, prepare_msrs, &i_mce, 0);
+
+ toggle_hw_mce_inject(cpu, false);
+
+ switch (inj_type) {
+ case DFR_INT_INJ:
+ smp_call_function_single(cpu, trigger_dfr_int, NULL, 0);
+ break;
+ case THR_INT_INJ:
+ smp_call_function_single(cpu, trigger_thr_int, NULL, 0);
+ break;
+ default:
+ smp_call_function_single(cpu, trigger_mce, NULL, 0);
+ }
+
+err:
+ put_online_cpus();
+
+}
+
+/*
+ * This denotes into which bank we're injecting and triggers
+ * the injection, at the same time.
+ */
+static int inj_bank_set(void *data, u64 val)
+{
+ struct mce *m = (struct mce *)data;
+
+ if (val >= n_banks) {
+ pr_err("Non-existent MCE bank: %llu\n", val);
+ return -EINVAL;
+ }
+
+ m->bank = val;
+ do_inject();
+
+ return 0;
+}
+
+MCE_INJECT_GET(bank);
+
+DEFINE_SIMPLE_ATTRIBUTE(bank_fops, inj_bank_get, inj_bank_set, "%llu\n");
+
+static const char readme_msg[] =
+"Description of the files and their usages:\n"
+"\n"
+"Note1: i refers to the bank number below.\n"
+"Note2: See respective BKDGs for the exact bit definitions of the files below\n"
+"as they mirror the hardware registers.\n"
+"\n"
+"status:\t Set MCi_STATUS: the bits in that MSR control the error type and\n"
+"\t attributes of the error which caused the MCE.\n"
+"\n"
+"misc:\t Set MCi_MISC: provide auxiliary info about the error. It is mostly\n"
+"\t used for error thresholding purposes and its validity is indicated by\n"
+"\t MCi_STATUS[MiscV].\n"
+"\n"
+"synd:\t Set MCi_SYND: provide syndrome info about the error. Only valid on\n"
+"\t Scalable MCA systems, and its validity is indicated by MCi_STATUS[SyndV].\n"
+"\n"
+"addr:\t Error address value to be written to MCi_ADDR. Log address information\n"
+"\t associated with the error.\n"
+"\n"
+"cpu:\t The CPU to inject the error on.\n"
+"\n"
+"bank:\t Specify the bank you want to inject the error into: the number of\n"
+"\t banks in a processor varies and is family/model-specific, therefore, the\n"
+"\t supplied value is sanity-checked. Setting the bank value also triggers the\n"
+"\t injection.\n"
+"\n"
+"flags:\t Injection type to be performed. Writing to this file will trigger a\n"
+"\t real machine check, an APIC interrupt or invoke the error decoder routines\n"
+"\t for AMD processors.\n"
+"\n"
+"\t Allowed error injection types:\n"
+"\t - \"sw\": Software error injection. Decode error to a human-readable \n"
+"\t format only. Safe to use.\n"
+"\t - \"hw\": Hardware error injection. Causes the #MC exception handler to \n"
+"\t handle the error. Be warned: might cause system panic if MCi_STATUS[PCC] \n"
+"\t is set. Therefore, consider setting (debugfs_mountpoint)/mce/fake_panic \n"
+"\t before injecting.\n"
+"\t - \"df\": Trigger APIC interrupt for Deferred error. Causes deferred \n"
+"\t error APIC interrupt handler to handle the error if the feature is \n"
+"\t is present in hardware. \n"
+"\t - \"th\": Trigger APIC interrupt for Threshold errors. Causes threshold \n"
+"\t APIC interrupt handler to handle the error. \n"
+"\n";
+
+static ssize_t
+inj_readme_read(struct file *filp, char __user *ubuf,
+ size_t cnt, loff_t *ppos)
+{
+ return simple_read_from_buffer(ubuf, cnt, ppos,
+ readme_msg, strlen(readme_msg));
+}
+
+static const struct file_operations readme_fops = {
+ .read = inj_readme_read,
+};
+
+static struct dfs_node {
+ char *name;
+ struct dentry *d;
+ const struct file_operations *fops;
+ umode_t perm;
+} dfs_fls[] = {
+ { .name = "status", .fops = &status_fops, .perm = S_IRUSR | S_IWUSR },
+ { .name = "misc", .fops = &misc_fops, .perm = S_IRUSR | S_IWUSR },
+ { .name = "addr", .fops = &addr_fops, .perm = S_IRUSR | S_IWUSR },
+ { .name = "synd", .fops = &synd_fops, .perm = S_IRUSR | S_IWUSR },
+ { .name = "bank", .fops = &bank_fops, .perm = S_IRUSR | S_IWUSR },
+ { .name = "flags", .fops = &flags_fops, .perm = S_IRUSR | S_IWUSR },
+ { .name = "cpu", .fops = &extcpu_fops, .perm = S_IRUSR | S_IWUSR },
+ { .name = "README", .fops = &readme_fops, .perm = S_IRUSR | S_IRGRP | S_IROTH },
+};
+
+static int __init debugfs_init(void)
+{
+ unsigned int i;
+ u64 cap;
+
+ rdmsrl(MSR_IA32_MCG_CAP, cap);
+ n_banks = cap & MCG_BANKCNT_MASK;
+
+ dfs_inj = debugfs_create_dir("mce-inject", NULL);
+ if (!dfs_inj)
+ return -EINVAL;
+
+ for (i = 0; i < ARRAY_SIZE(dfs_fls); i++) {
+ dfs_fls[i].d = debugfs_create_file(dfs_fls[i].name,
+ dfs_fls[i].perm,
+ dfs_inj,
+ &i_mce,
+ dfs_fls[i].fops);
+
+ if (!dfs_fls[i].d)
+ goto err_dfs_add;
+ }
+
+ return 0;
+
+err_dfs_add:
+ while (i-- > 0)
+ debugfs_remove(dfs_fls[i].d);
+
+ debugfs_remove(dfs_inj);
+ dfs_inj = NULL;
+
+ return -ENODEV;
}
-static int inject_init(void)
+static int __init inject_init(void)
{
+ int err;
+
if (!alloc_cpumask_var(&mce_inject_cpumask, GFP_KERNEL))
return -ENOMEM;
+
+ err = debugfs_init();
+ if (err) {
+ free_cpumask_var(mce_inject_cpumask);
+ return err;
+ }
+
+ register_nmi_handler(NMI_LOCAL, mce_raise_notify, 0, "mce_notify");
+ mce_register_injector_chain(&inject_nb);
+
+ setup_inj_struct(&i_mce);
+
pr_info("Machine check injector initialized\n");
- register_mce_write_callback(mce_write);
- register_nmi_handler(NMI_LOCAL, mce_raise_notify, 0,
- "mce_notify");
+
return 0;
}
+static void __exit inject_exit(void)
+{
+
+ mce_unregister_injector_chain(&inject_nb);
+ unregister_nmi_handler(NMI_LOCAL, "mce_notify");
+
+ debugfs_remove_recursive(dfs_inj);
+ dfs_inj = NULL;
+
+ memset(&dfs_fls, 0, sizeof(dfs_fls));
+
+ free_cpumask_var(mce_inject_cpumask);
+}
+
module_init(inject_init);
-/*
- * Cannot tolerate unloading currently because we cannot
- * guarantee all openers of mce_chrdev will get a reference to us.
- */
+module_exit(inject_exit);
MODULE_LICENSE("GPL");
extern struct device_attribute dev_attr_trigger;
#ifdef CONFIG_X86_MCELOG_LEGACY
-extern void mce_work_trigger(void);
+void mce_work_trigger(void);
+void mce_register_injector_chain(struct notifier_block *nb);
+void mce_unregister_injector_chain(struct notifier_block *nb);
#else
static inline void mce_work_trigger(void) { }
+static inline void mce_register_injector_chain(struct notifier_block *nb) { }
+static inline void mce_unregister_injector_chain(struct notifier_block *nb) { }
#endif
{
bool error_seen = false;
struct mce m;
- int severity;
int i;
this_cpu_inc(mce_poll_count);
mce_read_aux(&m, i);
- severity = mce_severity(&m, mca_cfg.tolerant, NULL, false);
-
- if (severity == MCE_DEFERRED_SEVERITY && mce_is_memory_error(&m))
- if (m.status & MCI_STATUS_ADDRV)
- m.severity = severity;
+ m.severity = mce_severity(&m, mca_cfg.tolerant, NULL, false);
/*
* Don't get the IP here because it's unlikely to
*/
clear_bit(10, (unsigned long *)&mce_banks[4].ctl);
}
- if (c->x86 < 17 && cfg->bootlog < 0) {
+ if (c->x86 < 0x11 && cfg->bootlog < 0) {
/*
* Lots of broken BIOS around that don't clear them
* by default and leave crap in there. Don't log:
* mce=TOLERANCELEVEL[,monarchtimeout] (number, see above)
* monarchtimeout is how long to wait for other CPUs on machine
* check, or 0 to not wait
- * mce=bootlog Log MCEs from before booting. Disabled by default on AMD.
+ * mce=bootlog Log MCEs from before booting. Disabled by default on AMD Fam10h
+ and older.
* mce=nobootlog Don't log MCEs from before booting.
* mce=bios_cmci_threshold Don't program the CMCI threshold
* mce=recovery force enable memcpy_mcsafe()
static void vendor_disable_error_reporting(void)
{
/*
- * Don't clear on Intel CPUs. Some of these MSRs are socket-wide.
+ * Don't clear on Intel or AMD CPUs. Some of these MSRs are socket-wide.
* Disabling them for just a single offlined CPU is bad, since it will
* inhibit reporting for all shared resources on the socket like the
* last level cache (LLC), the integrated memory controller (iMC), etc.
*/
- if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL)
+ if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL ||
+ boot_cpu_data.x86_vendor == X86_VENDOR_AMD)
return;
mce_disable_error_reporting();
}
void (*deferred_error_int_vector)(void) = default_deferred_error_interrupt;
-static void get_smca_bank_info(unsigned int bank)
+static void smca_configure(unsigned int bank, unsigned int cpu)
{
- unsigned int i, hwid_mcatype, cpu = smp_processor_id();
+ unsigned int i, hwid_mcatype;
struct smca_hwid *s_hwid;
- u32 high, instance_id;
+ u32 high, low;
+ u32 smca_config = MSR_AMD64_SMCA_MCx_CONFIG(bank);
+
+ /* Set appropriate bits in MCA_CONFIG */
+ if (!rdmsr_safe(smca_config, &low, &high)) {
+ /*
+ * OS is required to set the MCAX bit to acknowledge that it is
+ * now using the new MSR ranges and new registers under each
+ * bank. It also means that the OS will configure deferred
+ * errors in the new MCx_CONFIG register. If the bit is not set,
+ * uncorrectable errors will cause a system panic.
+ *
+ * MCA_CONFIG[MCAX] is bit 32 (0 in the high portion of the MSR.)
+ */
+ high |= BIT(0);
+
+ /*
+ * SMCA sets the Deferred Error Interrupt type per bank.
+ *
+ * MCA_CONFIG[DeferredIntTypeSupported] is bit 5, and tells us
+ * if the DeferredIntType bit field is available.
+ *
+ * MCA_CONFIG[DeferredIntType] is bits [38:37] ([6:5] in the
+ * high portion of the MSR). OS should set this to 0x1 to enable
+ * APIC based interrupt. First, check that no interrupt has been
+ * set.
+ */
+ if ((low & BIT(5)) && !((high >> 5) & 0x3))
+ high |= BIT(5);
+
+ wrmsr(smca_config, low, high);
+ }
/* Collect bank_info using CPU 0 for now. */
if (cpu)
return;
- if (rdmsr_safe_on_cpu(cpu, MSR_AMD64_SMCA_MCx_IPID(bank), &instance_id, &high)) {
+ if (rdmsr_safe_on_cpu(cpu, MSR_AMD64_SMCA_MCx_IPID(bank), &low, &high)) {
pr_warn("Failed to read MCA_IPID for bank %d\n", bank);
return;
}
smca_get_name(s_hwid->bank_type));
smca_banks[bank].hwid = s_hwid;
- smca_banks[bank].id = instance_id;
+ smca_banks[bank].id = low;
smca_banks[bank].sysfs_id = s_hwid->count++;
break;
}
int offset, u32 misc_high)
{
unsigned int cpu = smp_processor_id();
- u32 smca_low, smca_high, smca_addr;
+ u32 smca_low, smca_high;
struct threshold_block b;
int new;
goto set_offset;
}
- smca_addr = MSR_AMD64_SMCA_MCx_CONFIG(bank);
-
- if (!rdmsr_safe(smca_addr, &smca_low, &smca_high)) {
- /*
- * OS is required to set the MCAX bit to acknowledge that it is
- * now using the new MSR ranges and new registers under each
- * bank. It also means that the OS will configure deferred
- * errors in the new MCx_CONFIG register. If the bit is not set,
- * uncorrectable errors will cause a system panic.
- *
- * MCA_CONFIG[MCAX] is bit 32 (0 in the high portion of the MSR.)
- */
- smca_high |= BIT(0);
-
- /*
- * SMCA logs Deferred Error information in MCA_DE{STAT,ADDR}
- * registers with the option of additionally logging to
- * MCA_{STATUS,ADDR} if MCA_CONFIG[LogDeferredInMcaStat] is set.
- *
- * This bit is usually set by BIOS to retain the old behavior
- * for OSes that don't use the new registers. Linux supports the
- * new registers so let's disable that additional logging here.
- *
- * MCA_CONFIG[LogDeferredInMcaStat] is bit 34 (bit 2 in the high
- * portion of the MSR).
- */
- smca_high &= ~BIT(2);
-
- /*
- * SMCA sets the Deferred Error Interrupt type per bank.
- *
- * MCA_CONFIG[DeferredIntTypeSupported] is bit 5, and tells us
- * if the DeferredIntType bit field is available.
- *
- * MCA_CONFIG[DeferredIntType] is bits [38:37] ([6:5] in the
- * high portion of the MSR). OS should set this to 0x1 to enable
- * APIC based interrupt. First, check that no interrupt has been
- * set.
- */
- if ((smca_low & BIT(5)) && !((smca_high >> 5) & 0x3))
- smca_high |= BIT(5);
-
- wrmsr(smca_addr, smca_low, smca_high);
- }
-
/* Gather LVT offset for thresholding: */
if (rdmsr_safe(MSR_CU_DEF_ERR, &smca_low, &smca_high))
goto out;
for (bank = 0; bank < mca_cfg.banks; ++bank) {
if (mce_flags.smca)
- get_smca_bank_info(bank);
+ smca_configure(bank, cpu);
for (block = 0; block < NR_BLOCKS; ++block) {
address = get_block_address(cpu, address, low, high, bank, block);
}
EXPORT_SYMBOL_GPL(umc_normaddr_to_sysaddr);
-static void
-__log_error(unsigned int bank, bool deferred_err, bool threshold_err, u64 misc)
+static void __log_error(unsigned int bank, u64 status, u64 addr, u64 misc)
{
- u32 msr_status = msr_ops.status(bank);
- u32 msr_addr = msr_ops.addr(bank);
struct mce m;
- u64 status;
-
- WARN_ON_ONCE(deferred_err && threshold_err);
-
- if (deferred_err && mce_flags.smca) {
- msr_status = MSR_AMD64_SMCA_MCx_DESTAT(bank);
- msr_addr = MSR_AMD64_SMCA_MCx_DEADDR(bank);
- }
-
- rdmsrl(msr_status, status);
-
- if (!(status & MCI_STATUS_VAL))
- return;
mce_setup(&m);
m.status = status;
+ m.misc = misc;
m.bank = bank;
m.tsc = rdtsc();
- if (threshold_err)
- m.misc = misc;
-
if (m.status & MCI_STATUS_ADDRV) {
- rdmsrl(msr_addr, m.addr);
+ m.addr = addr;
/*
* Extract [55:<lsb>] where lsb is the least significant
}
mce_log(&m);
-
- wrmsrl(msr_status, 0);
}
static inline void __smp_deferred_error_interrupt(void)
exiting_ack_irq();
}
-/* APIC interrupt handler for deferred errors */
-static void amd_deferred_error_interrupt(void)
+/*
+ * Returns true if the logged error is deferred. False, otherwise.
+ */
+static inline bool
+_log_error_bank(unsigned int bank, u32 msr_stat, u32 msr_addr, u64 misc)
{
- unsigned int bank;
- u32 msr_status;
- u64 status;
+ u64 status, addr = 0;
- for (bank = 0; bank < mca_cfg.banks; ++bank) {
- msr_status = (mce_flags.smca) ? MSR_AMD64_SMCA_MCx_DESTAT(bank)
- : msr_ops.status(bank);
+ rdmsrl(msr_stat, status);
+ if (!(status & MCI_STATUS_VAL))
+ return false;
- rdmsrl(msr_status, status);
+ if (status & MCI_STATUS_ADDRV)
+ rdmsrl(msr_addr, addr);
- if (!(status & MCI_STATUS_VAL) ||
- !(status & MCI_STATUS_DEFERRED))
- continue;
+ __log_error(bank, status, addr, misc);
- __log_error(bank, true, false, 0);
- break;
- }
+ wrmsrl(msr_stat, 0);
+
+ return status & MCI_STATUS_DEFERRED;
}
/*
- * APIC Interrupt Handler
+ * We have three scenarios for checking for Deferred errors:
+ *
+ * 1) Non-SMCA systems check MCA_STATUS and log error if found.
+ * 2) SMCA systems check MCA_STATUS. If error is found then log it and also
+ * clear MCA_DESTAT.
+ * 3) SMCA systems check MCA_DESTAT, if error was not found in MCA_STATUS, and
+ * log it.
*/
+static void log_error_deferred(unsigned int bank)
+{
+ bool defrd;
-/*
- * threshold interrupt handler will service THRESHOLD_APIC_VECTOR.
- * the interrupt goes off when error_count reaches threshold_limit.
- * the handler will simply log mcelog w/ software defined bank number.
- */
+ defrd = _log_error_bank(bank, msr_ops.status(bank),
+ msr_ops.addr(bank), 0);
-static void amd_threshold_interrupt(void)
+ if (!mce_flags.smca)
+ return;
+
+ /* Clear MCA_DESTAT if we logged the deferred error from MCA_STATUS. */
+ if (defrd) {
+ wrmsrl(MSR_AMD64_SMCA_MCx_DESTAT(bank), 0);
+ return;
+ }
+
+ /*
+ * Only deferred errors are logged in MCA_DE{STAT,ADDR} so just check
+ * for a valid error.
+ */
+ _log_error_bank(bank, MSR_AMD64_SMCA_MCx_DESTAT(bank),
+ MSR_AMD64_SMCA_MCx_DEADDR(bank), 0);
+}
+
+/* APIC interrupt handler for deferred errors */
+static void amd_deferred_error_interrupt(void)
{
- u32 low = 0, high = 0, address = 0;
- unsigned int bank, block, cpu = smp_processor_id();
- struct thresh_restart tr;
+ unsigned int bank;
- /* assume first bank caused it */
- for (bank = 0; bank < mca_cfg.banks; ++bank) {
- if (!(per_cpu(bank_map, cpu) & (1 << bank)))
- continue;
- for (block = 0; block < NR_BLOCKS; ++block) {
- address = get_block_address(cpu, address, low, high, bank, block);
- if (!address)
- break;
+ for (bank = 0; bank < mca_cfg.banks; ++bank)
+ log_error_deferred(bank);
+}
- if (rdmsr_safe(address, &low, &high))
- break;
+static void log_error_thresholding(unsigned int bank, u64 misc)
+{
+ _log_error_bank(bank, msr_ops.status(bank), msr_ops.addr(bank), misc);
+}
- if (!(high & MASK_VALID_HI)) {
- if (block)
- continue;
- else
- break;
- }
+static void log_and_reset_block(struct threshold_block *block)
+{
+ struct thresh_restart tr;
+ u32 low = 0, high = 0;
- if (!(high & MASK_CNTP_HI) ||
- (high & MASK_LOCKED_HI))
- continue;
+ if (!block)
+ return;
- /*
- * Log the machine check that caused the threshold
- * event.
- */
- if (high & MASK_OVERFLOW_HI)
- goto log;
- }
- }
- return;
+ if (rdmsr_safe(block->address, &low, &high))
+ return;
+
+ if (!(high & MASK_OVERFLOW_HI))
+ return;
-log:
- __log_error(bank, false, true, ((u64)high << 32) | low);
+ /* Log the MCE which caused the threshold event. */
+ log_error_thresholding(block->bank, ((u64)high << 32) | low);
/* Reset threshold block after logging error. */
memset(&tr, 0, sizeof(tr));
- tr.b = &per_cpu(threshold_banks, cpu)[bank]->blocks[block];
+ tr.b = block;
threshold_restart_bank(&tr);
}
+/*
+ * Threshold interrupt handler will service THRESHOLD_APIC_VECTOR. The interrupt
+ * goes off when error_count reaches threshold_limit.
+ */
+static void amd_threshold_interrupt(void)
+{
+ struct threshold_block *first_block = NULL, *block = NULL, *tmp = NULL;
+ unsigned int bank, cpu = smp_processor_id();
+
+ for (bank = 0; bank < mca_cfg.banks; ++bank) {
+ if (!(per_cpu(bank_map, cpu) & (1 << bank)))
+ continue;
+
+ first_block = per_cpu(threshold_banks, cpu)[bank]->blocks;
+ if (!first_block)
+ continue;
+
+ /*
+ * The first block is also the head of the list. Check it first
+ * before iterating over the rest.
+ */
+ log_and_reset_block(first_block);
+ list_for_each_entry_safe(block, tmp, &first_block->miscj, miscj)
+ log_and_reset_block(block);
+ }
+}
+
/*
* Sysfs Interface
*/
goto out;
per_cpu(threshold_banks, cpu)[bank] = b;
- atomic_inc(&b->cpus);
+ refcount_inc(&b->cpus);
err = __threshold_add_blocks(b);
per_cpu(threshold_banks, cpu)[bank] = b;
if (is_shared_bank(bank)) {
- atomic_set(&b->cpus, 1);
+ refcount_set(&b->cpus, 1);
/* nb is already initialized, see above */
if (nb) {
goto free_out;
if (is_shared_bank(bank)) {
- if (!atomic_dec_and_test(&b->cpus)) {
+ if (!refcount_dec_and_test(&b->cpus)) {
__threshold_remove_blocks(b);
per_cpu(threshold_banks, cpu)[bank] = NULL;
return;
#endif
}
-void __load_ucode_amd(unsigned int cpuid_1_eax, struct cpio_data *ret)
+static void __load_ucode_amd(unsigned int cpuid_1_eax, struct cpio_data *ret)
{
struct ucode_cpu_info *uci;
struct cpio_data cp;
return (struct cpio_data){ NULL, 0, "" };
if (initrd_start)
start = initrd_start;
+ } else {
+ /*
+ * The picture with physical addresses is a bit different: we
+ * need to get the *physical* address to which the ramdisk was
+ * relocated, i.e., relocated_ramdisk (not initrd_start) and
+ * since we're running from physical addresses, we need to access
+ * relocated_ramdisk through its *physical* address too.
+ */
+ u64 *rr = (u64 *)__pa_nodebug(&relocated_ramdisk);
+ if (*rr)
+ start = *rr;
}
return find_cpio_data(path, (void *)start, size, NULL);
static const char ucode_path[] = "kernel/x86/microcode/GenuineIntel.bin";
/* Current microcode patch used in early patching on the APs. */
-struct microcode_intel *intel_ucode_patch;
+static struct microcode_intel *intel_ucode_patch;
static inline bool cpu_signatures_match(unsigned int s1, unsigned int p1,
unsigned int s2, unsigned int p2)
static void save_microcode_patch(void *data, unsigned int size)
{
struct microcode_header_intel *mc_hdr, *mc_saved_hdr;
- struct ucode_patch *iter, *tmp, *p;
+ struct ucode_patch *iter, *tmp, *p = NULL;
bool prev_found = false;
unsigned int sig, pf;
else
list_add_tail(&p->plist, µcode_cache);
}
+
+ /*
+ * Save for early loading. On 32-bit, that needs to be a physical
+ * address as the APs are running from physical addresses, before
+ * paging has been enabled.
+ */
+ if (p) {
+ if (IS_ENABLED(CONFIG_X86_32))
+ intel_ucode_patch = (struct microcode_intel *)__pa_nodebug(p->data);
+ else
+ intel_ucode_patch = p->data;
+ }
}
static int microcode_sanity_check(void *mc, int print_err)
struct ucode_cpu_info uci;
struct cpio_data cp;
+ /*
+ * initrd is going away, clear patch ptr. We will scan the microcode one
+ * last time before jettisoning and save a patch, if found. Then we will
+ * update that pointer too, with a stable patch address to use when
+ * resuming the cores.
+ */
+ intel_ucode_patch = NULL;
+
if (!load_builtin_intel_microcode(&cp))
cp = find_microcode_in_initrd(ucode_path, false);
show_saved_mc();
- /* initrd is going away, clear patch ptr. */
- intel_ucode_patch = NULL;
-
return 0;
}
}
#endif
+static unsigned long hv_get_tsc_khz(void)
+{
+ unsigned long freq;
+
+ rdmsrl(HV_X64_MSR_TSC_FREQUENCY, freq);
+
+ return freq / 1000;
+}
+
static void __init ms_hyperv_init_platform(void)
{
int hv_host_info_eax;
hv_host_info_edx >> 24, hv_host_info_edx & 0xFFFFFF);
}
+ if (ms_hyperv.features & HV_X64_ACCESS_FREQUENCY_MSRS &&
+ ms_hyperv.misc_features & HV_FEATURE_FREQUENCY_MSRS_AVAILABLE) {
+ x86_platform.calibrate_tsc = hv_get_tsc_khz;
+ x86_platform.calibrate_cpu = hv_get_tsc_khz;
+ }
+
#ifdef CONFIG_X86_LOCAL_APIC
- if (ms_hyperv.features & HV_X64_MSR_APIC_FREQUENCY_AVAILABLE) {
+ if (ms_hyperv.features & HV_X64_ACCESS_FREQUENCY_MSRS &&
+ ms_hyperv.misc_features & HV_FEATURE_FREQUENCY_MSRS_AVAILABLE) {
/*
* Get the APIC frequency.
*/
if (!mtrr_enabled())
return;
- get_online_cpus();
first_cpu = cpumask_first(cpu_online_mask);
smp_call_function_single(first_cpu, mtrr_save_fixed_ranges, NULL, 1);
- put_online_cpus();
}
void set_mtrr_aps_delayed_init(void)
#include <linux/timex.h>
#include <linux/string.h>
#include <linux/seq_file.h>
-#include <linux/cpufreq.h>
/*
* Get CPU information for use by the procfs.
if (c->microcode)
seq_printf(m, "microcode\t: 0x%x\n", c->microcode);
- if (cpu_has(c, X86_FEATURE_TSC)) {
- unsigned int freq = cpufreq_quick_get(cpu);
-
- if (!freq)
- freq = cpu_khz;
+ if (cpu_has(c, X86_FEATURE_TSC))
seq_printf(m, "cpu MHz\t\t: %u.%03u\n",
- freq / 1000, (freq % 1000));
- }
+ cpu_khz / 1000, (cpu_khz % 1000));
/* Cache size */
if (c->x86_cache_size >= 0)
p4d_t *p4d;
/* Install the espfix pud into the kernel page directory */
- pgd = &init_level4_pgt[pgd_index(ESPFIX_BASE_ADDR)];
+ pgd = &init_top_pgt[pgd_index(ESPFIX_BASE_ADDR)];
p4d = p4d_alloc(&init_mm, pgd, ESPFIX_BASE_ADDR);
p4d_populate(&init_mm, p4d, espfix_pud_page);
/*
* Manage page tables very early on.
*/
-extern pgd_t early_level4_pgt[PTRS_PER_PGD];
+extern pgd_t early_top_pgt[PTRS_PER_PGD];
extern pmd_t early_dynamic_pgts[EARLY_DYNAMIC_PAGE_TABLES][PTRS_PER_PMD];
-static unsigned int __initdata next_early_pgt = 2;
+static unsigned int __initdata next_early_pgt;
pmdval_t early_pmd_flags = __PAGE_KERNEL_LARGE & ~(_PAGE_GLOBAL | _PAGE_NX);
+#define __head __section(.head.text)
+
+static void __head *fixup_pointer(void *ptr, unsigned long physaddr)
+{
+ return ptr - (void *)_text + (void *)physaddr;
+}
+
+void __head __startup_64(unsigned long physaddr)
+{
+ unsigned long load_delta, *p;
+ pgdval_t *pgd;
+ p4dval_t *p4d;
+ pudval_t *pud;
+ pmdval_t *pmd, pmd_entry;
+ int i;
+
+ /* Is the address too large? */
+ if (physaddr >> MAX_PHYSMEM_BITS)
+ for (;;);
+
+ /*
+ * Compute the delta between the address I am compiled to run at
+ * and the address I am actually running at.
+ */
+ load_delta = physaddr - (unsigned long)(_text - __START_KERNEL_map);
+
+ /* Is the address not 2M aligned? */
+ if (load_delta & ~PMD_PAGE_MASK)
+ for (;;);
+
+ /* Fixup the physical addresses in the page table */
+
+ pgd = fixup_pointer(&early_top_pgt, physaddr);
+ pgd[pgd_index(__START_KERNEL_map)] += load_delta;
+
+ if (IS_ENABLED(CONFIG_X86_5LEVEL)) {
+ p4d = fixup_pointer(&level4_kernel_pgt, physaddr);
+ p4d[511] += load_delta;
+ }
+
+ pud = fixup_pointer(&level3_kernel_pgt, physaddr);
+ pud[510] += load_delta;
+ pud[511] += load_delta;
+
+ pmd = fixup_pointer(level2_fixmap_pgt, physaddr);
+ pmd[506] += load_delta;
+
+ /*
+ * Set up the identity mapping for the switchover. These
+ * entries should *NOT* have the global bit set! This also
+ * creates a bunch of nonsense entries but that is fine --
+ * it avoids problems around wraparound.
+ */
+
+ pud = fixup_pointer(early_dynamic_pgts[next_early_pgt++], physaddr);
+ pmd = fixup_pointer(early_dynamic_pgts[next_early_pgt++], physaddr);
+
+ if (IS_ENABLED(CONFIG_X86_5LEVEL)) {
+ p4d = fixup_pointer(early_dynamic_pgts[next_early_pgt++], physaddr);
+
+ i = (physaddr >> PGDIR_SHIFT) % PTRS_PER_PGD;
+ pgd[i + 0] = (pgdval_t)p4d + _KERNPG_TABLE;
+ pgd[i + 1] = (pgdval_t)p4d + _KERNPG_TABLE;
+
+ i = (physaddr >> P4D_SHIFT) % PTRS_PER_P4D;
+ p4d[i + 0] = (pgdval_t)pud + _KERNPG_TABLE;
+ p4d[i + 1] = (pgdval_t)pud + _KERNPG_TABLE;
+ } else {
+ i = (physaddr >> PGDIR_SHIFT) % PTRS_PER_PGD;
+ pgd[i + 0] = (pgdval_t)pud + _KERNPG_TABLE;
+ pgd[i + 1] = (pgdval_t)pud + _KERNPG_TABLE;
+ }
+
+ i = (physaddr >> PUD_SHIFT) % PTRS_PER_PUD;
+ pud[i + 0] = (pudval_t)pmd + _KERNPG_TABLE;
+ pud[i + 1] = (pudval_t)pmd + _KERNPG_TABLE;
+
+ pmd_entry = __PAGE_KERNEL_LARGE_EXEC & ~_PAGE_GLOBAL;
+ pmd_entry += physaddr;
+
+ for (i = 0; i < DIV_ROUND_UP(_end - _text, PMD_SIZE); i++) {
+ int idx = i + (physaddr >> PMD_SHIFT) % PTRS_PER_PMD;
+ pmd[idx] = pmd_entry + i * PMD_SIZE;
+ }
+
+ /*
+ * Fixup the kernel text+data virtual addresses. Note that
+ * we might write invalid pmds, when the kernel is relocated
+ * cleanup_highmap() fixes this up along with the mappings
+ * beyond _end.
+ */
+
+ pmd = fixup_pointer(level2_kernel_pgt, physaddr);
+ for (i = 0; i < PTRS_PER_PMD; i++) {
+ if (pmd[i] & _PAGE_PRESENT)
+ pmd[i] += load_delta;
+ }
+
+ /* Fixup phys_base */
+ p = fixup_pointer(&phys_base, physaddr);
+ *p += load_delta;
+}
+
/* Wipe all early page tables except for the kernel symbol map */
static void __init reset_early_page_tables(void)
{
- memset(early_level4_pgt, 0, sizeof(pgd_t)*(PTRS_PER_PGD-1));
+ memset(early_top_pgt, 0, sizeof(pgd_t)*(PTRS_PER_PGD-1));
next_early_pgt = 0;
- write_cr3(__pa_nodebug(early_level4_pgt));
+ write_cr3(__pa_nodebug(early_top_pgt));
}
/* Create a new PMD entry */
{
unsigned long physaddr = address - __PAGE_OFFSET;
pgdval_t pgd, *pgd_p;
+ p4dval_t p4d, *p4d_p;
pudval_t pud, *pud_p;
pmdval_t pmd, *pmd_p;
/* Invalid address or early pgt is done ? */
- if (physaddr >= MAXMEM || read_cr3() != __pa_nodebug(early_level4_pgt))
+ if (physaddr >= MAXMEM || read_cr3_pa() != __pa_nodebug(early_top_pgt))
return -1;
again:
- pgd_p = &early_level4_pgt[pgd_index(address)].pgd;
+ pgd_p = &early_top_pgt[pgd_index(address)].pgd;
pgd = *pgd_p;
/*
* critical -- __PAGE_OFFSET would point us back into the dynamic
* range and we might end up looping forever...
*/
- if (pgd)
- pud_p = (pudval_t *)((pgd & PTE_PFN_MASK) + __START_KERNEL_map - phys_base);
+ if (!IS_ENABLED(CONFIG_X86_5LEVEL))
+ p4d_p = pgd_p;
+ else if (pgd)
+ p4d_p = (p4dval_t *)((pgd & PTE_PFN_MASK) + __START_KERNEL_map - phys_base);
+ else {
+ if (next_early_pgt >= EARLY_DYNAMIC_PAGE_TABLES) {
+ reset_early_page_tables();
+ goto again;
+ }
+
+ p4d_p = (p4dval_t *)early_dynamic_pgts[next_early_pgt++];
+ memset(p4d_p, 0, sizeof(*p4d_p) * PTRS_PER_P4D);
+ *pgd_p = (pgdval_t)p4d_p - __START_KERNEL_map + phys_base + _KERNPG_TABLE;
+ }
+ p4d_p += p4d_index(address);
+ p4d = *p4d_p;
+
+ if (p4d)
+ pud_p = (pudval_t *)((p4d & PTE_PFN_MASK) + __START_KERNEL_map - phys_base);
else {
if (next_early_pgt >= EARLY_DYNAMIC_PAGE_TABLES) {
reset_early_page_tables();
pud_p = (pudval_t *)early_dynamic_pgts[next_early_pgt++];
memset(pud_p, 0, sizeof(*pud_p) * PTRS_PER_PUD);
- *pgd_p = (pgdval_t)pud_p - __START_KERNEL_map + phys_base + _KERNPG_TABLE;
+ *p4d_p = (p4dval_t)pud_p - __START_KERNEL_map + phys_base + _KERNPG_TABLE;
}
pud_p += pud_index(address);
pud = *pud_p;
clear_bss();
- clear_page(init_level4_pgt);
+ clear_page(init_top_pgt);
kasan_early_init();
*/
load_ucode_bsp();
- /* set init_level4_pgt kernel high mapping*/
- init_level4_pgt[511] = early_level4_pgt[511];
+ /* set init_top_pgt kernel high mapping*/
+ init_top_pgt[511] = early_top_pgt[511];
x86_64_start_reservations(real_mode_data);
}
*
*/
+#define p4d_index(x) (((x) >> P4D_SHIFT) & (PTRS_PER_P4D-1))
#define pud_index(x) (((x) >> PUD_SHIFT) & (PTRS_PER_PUD-1))
-L4_PAGE_OFFSET = pgd_index(__PAGE_OFFSET_BASE)
-L4_START_KERNEL = pgd_index(__START_KERNEL_map)
+PGD_PAGE_OFFSET = pgd_index(__PAGE_OFFSET_BASE)
+PGD_START_KERNEL = pgd_index(__START_KERNEL_map)
L3_START_KERNEL = pud_index(__START_KERNEL_map)
.text
/* Sanitize CPU configuration */
call verify_cpu
- /*
- * Compute the delta between the address I am compiled to run at and the
- * address I am actually running at.
- */
- leaq _text(%rip), %rbp
- subq $_text - __START_KERNEL_map, %rbp
-
- /* Is the address not 2M aligned? */
- testl $~PMD_PAGE_MASK, %ebp
- jnz bad_address
-
- /*
- * Is the address too large?
- */
- leaq _text(%rip), %rax
- shrq $MAX_PHYSMEM_BITS, %rax
- jnz bad_address
-
- /*
- * Fixup the physical addresses in the page table
- */
- addq %rbp, early_level4_pgt + (L4_START_KERNEL*8)(%rip)
-
- addq %rbp, level3_kernel_pgt + (510*8)(%rip)
- addq %rbp, level3_kernel_pgt + (511*8)(%rip)
-
- addq %rbp, level2_fixmap_pgt + (506*8)(%rip)
-
- /*
- * Set up the identity mapping for the switchover. These
- * entries should *NOT* have the global bit set! This also
- * creates a bunch of nonsense entries but that is fine --
- * it avoids problems around wraparound.
- */
leaq _text(%rip), %rdi
- leaq early_level4_pgt(%rip), %rbx
-
- movq %rdi, %rax
- shrq $PGDIR_SHIFT, %rax
-
- leaq (PAGE_SIZE + _KERNPG_TABLE)(%rbx), %rdx
- movq %rdx, 0(%rbx,%rax,8)
- movq %rdx, 8(%rbx,%rax,8)
-
- addq $PAGE_SIZE, %rdx
- movq %rdi, %rax
- shrq $PUD_SHIFT, %rax
- andl $(PTRS_PER_PUD-1), %eax
- movq %rdx, PAGE_SIZE(%rbx,%rax,8)
- incl %eax
- andl $(PTRS_PER_PUD-1), %eax
- movq %rdx, PAGE_SIZE(%rbx,%rax,8)
-
- addq $PAGE_SIZE * 2, %rbx
- movq %rdi, %rax
- shrq $PMD_SHIFT, %rdi
- addq $(__PAGE_KERNEL_LARGE_EXEC & ~_PAGE_GLOBAL), %rax
- leaq (_end - 1)(%rip), %rcx
- shrq $PMD_SHIFT, %rcx
- subq %rdi, %rcx
- incl %ecx
-
-1:
- andq $(PTRS_PER_PMD - 1), %rdi
- movq %rax, (%rbx,%rdi,8)
- incq %rdi
- addq $PMD_SIZE, %rax
- decl %ecx
- jnz 1b
-
- test %rbp, %rbp
- jz .Lskip_fixup
+ pushq %rsi
+ call __startup_64
+ popq %rsi
- /*
- * Fixup the kernel text+data virtual addresses. Note that
- * we might write invalid pmds, when the kernel is relocated
- * cleanup_highmap() fixes this up along with the mappings
- * beyond _end.
- */
- leaq level2_kernel_pgt(%rip), %rdi
- leaq PAGE_SIZE(%rdi), %r8
- /* See if it is a valid page table entry */
-1: testb $_PAGE_PRESENT, 0(%rdi)
- jz 2f
- addq %rbp, 0(%rdi)
- /* Go to the next page */
-2: addq $8, %rdi
- cmp %r8, %rdi
- jne 1b
-
- /* Fixup phys_base */
- addq %rbp, phys_base(%rip)
-
-.Lskip_fixup:
- movq $(early_level4_pgt - __START_KERNEL_map), %rax
+ movq $(early_top_pgt - __START_KERNEL_map), %rax
jmp 1f
ENTRY(secondary_startup_64)
/*
/* Sanitize CPU configuration */
call verify_cpu
- movq $(init_level4_pgt - __START_KERNEL_map), %rax
+ movq $(init_top_pgt - __START_KERNEL_map), %rax
1:
- /* Enable PAE mode and PGE */
+ /* Enable PAE mode, PGE and LA57 */
movl $(X86_CR4_PAE | X86_CR4_PGE), %ecx
+#ifdef CONFIG_X86_5LEVEL
+ orl $X86_CR4_LA57, %ecx
+#endif
movq %rcx, %cr4
- /* Setup early boot stage 4 level pagetables. */
+ /* Setup early boot stage 4-/5-level pagetables. */
addq phys_base(%rip), %rax
movq %rax, %cr3
.endr
__INITDATA
-NEXT_PAGE(early_level4_pgt)
+NEXT_PAGE(early_top_pgt)
.fill 511,8,0
+#ifdef CONFIG_X86_5LEVEL
+ .quad level4_kernel_pgt - __START_KERNEL_map + _PAGE_TABLE
+#else
.quad level3_kernel_pgt - __START_KERNEL_map + _PAGE_TABLE
+#endif
NEXT_PAGE(early_dynamic_pgts)
.fill 512*EARLY_DYNAMIC_PAGE_TABLES,8,0
.data
#ifndef CONFIG_XEN
-NEXT_PAGE(init_level4_pgt)
+NEXT_PAGE(init_top_pgt)
.fill 512,8,0
#else
-NEXT_PAGE(init_level4_pgt)
+NEXT_PAGE(init_top_pgt)
.quad level3_ident_pgt - __START_KERNEL_map + _KERNPG_TABLE
- .org init_level4_pgt + L4_PAGE_OFFSET*8, 0
+ .org init_top_pgt + PGD_PAGE_OFFSET*8, 0
.quad level3_ident_pgt - __START_KERNEL_map + _KERNPG_TABLE
- .org init_level4_pgt + L4_START_KERNEL*8, 0
+ .org init_top_pgt + PGD_START_KERNEL*8, 0
/* (2^48-(2*1024*1024*1024))/(2^39) = 511 */
.quad level3_kernel_pgt - __START_KERNEL_map + _PAGE_TABLE
PMDS(0, __PAGE_KERNEL_IDENT_LARGE_EXEC, PTRS_PER_PMD)
#endif
+#ifdef CONFIG_X86_5LEVEL
+NEXT_PAGE(level4_kernel_pgt)
+ .fill 511,8,0
+ .quad level3_kernel_pgt - __START_KERNEL_map + _PAGE_TABLE
+#endif
+
NEXT_PAGE(level3_kernel_pgt)
.fill L3_START_KERNEL,8,0
/* (2^48-(2*1024*1024*1024)-((2^39)*511))/(2^30) = 510 */
* Start hpet with the boot cpu mask and make it
* global after the IO_APIC has been initialized.
*/
- hpet_clockevent.cpumask = cpumask_of(smp_processor_id());
+ hpet_clockevent.cpumask = cpumask_of(boot_cpu_data.cpu_index);
clockevents_config_and_register(&hpet_clockevent, hpet_freq,
HPET_MIN_PROG_DELTA, 0x7FFFFFFF);
global_clock_event = &hpet_clockevent;
/* A cpu has been removed from cpu_online_mask. Reset irq affinities. */
void fixup_irqs(void)
{
- unsigned int irq, vector;
- static int warned;
+ unsigned int irr, vector;
struct irq_desc *desc;
struct irq_data *data;
struct irq_chip *chip;
- int ret;
- for_each_irq_desc(irq, desc) {
- int break_affinity = 0;
- int set_affinity = 1;
- const struct cpumask *affinity;
-
- if (!desc)
- continue;
- if (irq == 2)
- continue;
-
- /* interrupt's are disabled at this point */
- raw_spin_lock(&desc->lock);
-
- data = irq_desc_get_irq_data(desc);
- affinity = irq_data_get_affinity_mask(data);
- if (!irq_has_action(irq) || irqd_is_per_cpu(data) ||
- cpumask_subset(affinity, cpu_online_mask)) {
- raw_spin_unlock(&desc->lock);
- continue;
- }
-
- /*
- * Complete the irq move. This cpu is going down and for
- * non intr-remapping case, we can't wait till this interrupt
- * arrives at this cpu before completing the irq move.
- */
- irq_force_complete_move(desc);
-
- if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
- break_affinity = 1;
- affinity = cpu_online_mask;
- }
-
- chip = irq_data_get_irq_chip(data);
- /*
- * The interrupt descriptor might have been cleaned up
- * already, but it is not yet removed from the radix tree
- */
- if (!chip) {
- raw_spin_unlock(&desc->lock);
- continue;
- }
-
- if (!irqd_can_move_in_process_context(data) && chip->irq_mask)
- chip->irq_mask(data);
-
- if (chip->irq_set_affinity) {
- ret = chip->irq_set_affinity(data, affinity, true);
- if (ret == -ENOSPC)
- pr_crit("IRQ %d set affinity failed because there are no available vectors. The device assigned to this IRQ is unstable.\n", irq);
- } else {
- if (!(warned++))
- set_affinity = 0;
- }
-
- /*
- * We unmask if the irq was not marked masked by the
- * core code. That respects the lazy irq disable
- * behaviour.
- */
- if (!irqd_can_move_in_process_context(data) &&
- !irqd_irq_masked(data) && chip->irq_unmask)
- chip->irq_unmask(data);
-
- raw_spin_unlock(&desc->lock);
-
- if (break_affinity && set_affinity)
- pr_notice("Broke affinity for irq %i\n", irq);
- else if (!set_affinity)
- pr_notice("Cannot set affinity for irq %i\n", irq);
- }
+ irq_migrate_all_off_this_cpu();
/*
* We can remove mdelay() and then send spuriuous interrupts to
* nothing else will touch it.
*/
for (vector = FIRST_EXTERNAL_VECTOR; vector < NR_VECTORS; vector++) {
- unsigned int irr;
-
if (IS_ERR_OR_NULL(__this_cpu_read(vector_irq[vector])))
continue;
void arch_jump_label_transform(struct jump_entry *entry,
enum jump_label_type type)
{
- get_online_cpus();
mutex_lock(&text_mutex);
__jump_label_transform(entry, type, NULL, 0);
mutex_unlock(&text_mutex);
- put_online_cpus();
}
static enum {
#include <linux/kdebug.h>
#include <linux/kallsyms.h>
#include <linux/ftrace.h>
+#include <linux/frame.h>
#include <asm/text-patching.h>
#include <asm/cacheflush.h>
}
asm (
+ "optprobe_template_func:\n"
".global optprobe_template_entry\n"
"optprobe_template_entry:\n"
#ifdef CONFIG_X86_64
" popf\n"
#endif
".global optprobe_template_end\n"
- "optprobe_template_end:\n");
+ "optprobe_template_end:\n"
+ ".type optprobe_template_func, @function\n"
+ ".size optprobe_template_func, .-optprobe_template_func\n");
+
+void optprobe_template_func(void);
+STACK_FRAME_NON_STANDARD(optprobe_template_func);
#define TMPL_MOVE_IDX \
((long)&optprobe_template_val - (long)&optprobe_template_entry)
#include <asm/syscalls.h>
/* context.lock is held for us, so we don't need any locking. */
-static void flush_ldt(void *current_mm)
+static void flush_ldt(void *__mm)
{
+ struct mm_struct *mm = __mm;
mm_context_t *pc;
- if (current->active_mm != current_mm)
+ if (this_cpu_read(cpu_tlbstate.loaded_mm) != mm)
return;
- pc = ¤t->active_mm->context;
- set_ldt(pc->ldt->entries, pc->ldt->size);
+ pc = &mm->context;
+ set_ldt(pc->ldt->entries, pc->ldt->nr_entries);
}
/* The caller must call finalize_ldt_struct on the result. LDT starts zeroed. */
-static struct ldt_struct *alloc_ldt_struct(unsigned int size)
+static struct ldt_struct *alloc_ldt_struct(unsigned int num_entries)
{
struct ldt_struct *new_ldt;
unsigned int alloc_size;
- if (size > LDT_ENTRIES)
+ if (num_entries > LDT_ENTRIES)
return NULL;
new_ldt = kmalloc(sizeof(struct ldt_struct), GFP_KERNEL);
return NULL;
BUILD_BUG_ON(LDT_ENTRY_SIZE != sizeof(struct desc_struct));
- alloc_size = size * LDT_ENTRY_SIZE;
+ alloc_size = num_entries * LDT_ENTRY_SIZE;
/*
* Xen is very picky: it requires a page-aligned LDT that has no
return NULL;
}
- new_ldt->size = size;
+ new_ldt->nr_entries = num_entries;
return new_ldt;
}
/* After calling this, the LDT is immutable. */
static void finalize_ldt_struct(struct ldt_struct *ldt)
{
- paravirt_alloc_ldt(ldt->entries, ldt->size);
+ paravirt_alloc_ldt(ldt->entries, ldt->nr_entries);
}
/* context.lock is held */
if (likely(!ldt))
return;
- paravirt_free_ldt(ldt->entries, ldt->size);
- if (ldt->size * LDT_ENTRY_SIZE > PAGE_SIZE)
+ paravirt_free_ldt(ldt->entries, ldt->nr_entries);
+ if (ldt->nr_entries * LDT_ENTRY_SIZE > PAGE_SIZE)
vfree_atomic(ldt->entries);
else
free_page((unsigned long)ldt->entries);
goto out_unlock;
}
- new_ldt = alloc_ldt_struct(old_mm->context.ldt->size);
+ new_ldt = alloc_ldt_struct(old_mm->context.ldt->nr_entries);
if (!new_ldt) {
retval = -ENOMEM;
goto out_unlock;
}
memcpy(new_ldt->entries, old_mm->context.ldt->entries,
- new_ldt->size * LDT_ENTRY_SIZE);
+ new_ldt->nr_entries * LDT_ENTRY_SIZE);
finalize_ldt_struct(new_ldt);
mm->context.ldt = new_ldt;
static int read_ldt(void __user *ptr, unsigned long bytecount)
{
- int retval;
- unsigned long size;
struct mm_struct *mm = current->mm;
+ unsigned long entries_size;
+ int retval;
mutex_lock(&mm->context.lock);
if (bytecount > LDT_ENTRY_SIZE * LDT_ENTRIES)
bytecount = LDT_ENTRY_SIZE * LDT_ENTRIES;
- size = mm->context.ldt->size * LDT_ENTRY_SIZE;
- if (size > bytecount)
- size = bytecount;
+ entries_size = mm->context.ldt->nr_entries * LDT_ENTRY_SIZE;
+ if (entries_size > bytecount)
+ entries_size = bytecount;
- if (copy_to_user(ptr, mm->context.ldt->entries, size)) {
+ if (copy_to_user(ptr, mm->context.ldt->entries, entries_size)) {
retval = -EFAULT;
goto out_unlock;
}
- if (size != bytecount) {
+ if (entries_size != bytecount) {
/* Zero-fill the rest and pretend we read bytecount bytes. */
- if (clear_user(ptr + size, bytecount - size)) {
+ if (clear_user(ptr + entries_size, bytecount - entries_size)) {
retval = -EFAULT;
goto out_unlock;
}
{
struct mm_struct *mm = current->mm;
struct ldt_struct *new_ldt, *old_ldt;
- unsigned int oldsize, newsize;
+ unsigned int old_nr_entries, new_nr_entries;
struct user_desc ldt_info;
struct desc_struct ldt;
int error;
mutex_lock(&mm->context.lock);
- old_ldt = mm->context.ldt;
- oldsize = old_ldt ? old_ldt->size : 0;
- newsize = max(ldt_info.entry_number + 1, oldsize);
+ old_ldt = mm->context.ldt;
+ old_nr_entries = old_ldt ? old_ldt->nr_entries : 0;
+ new_nr_entries = max(ldt_info.entry_number + 1, old_nr_entries);
error = -ENOMEM;
- new_ldt = alloc_ldt_struct(newsize);
+ new_ldt = alloc_ldt_struct(new_nr_entries);
if (!new_ldt)
goto out_unlock;
if (old_ldt)
- memcpy(new_ldt->entries, old_ldt->entries, oldsize * LDT_ENTRY_SIZE);
+ memcpy(new_ldt->entries, old_ldt->entries, old_nr_entries * LDT_ENTRY_SIZE);
+
new_ldt->entries[ldt_info.entry_number] = ldt;
finalize_ldt_struct(new_ldt);
void arch_crash_save_vmcoreinfo(void)
{
VMCOREINFO_NUMBER(phys_base);
- VMCOREINFO_SYMBOL(init_level4_pgt);
+ VMCOREINFO_SYMBOL(init_top_pgt);
#ifdef CONFIG_NUMA
VMCOREINFO_SYMBOL(node_data);
/* Don't wait longer than a second */
timeout = USEC_PER_SEC;
- while (!cpumask_empty(mask) && timeout--)
+ while (!cpumask_empty(mask) && --timeout)
udelay(1);
/* What happens if we timeout, do we still unregister?? */
.read_cr2 = native_read_cr2,
.write_cr2 = native_write_cr2,
- .read_cr3 = native_read_cr3,
+ .read_cr3 = __native_read_cr3,
.write_cr3 = native_write_cr3,
.flush_tlb_user = native_flush_tlb,
return randomize_page(mm->brk, 0x02000000);
}
-/*
- * Return saved PC of a blocked thread.
- * What is this good for? it will be always the scheduler or ret_from_fork.
- */
-unsigned long thread_saved_pc(struct task_struct *tsk)
-{
- struct inactive_task_frame *frame =
- (struct inactive_task_frame *) READ_ONCE(tsk->thread.sp);
- return READ_ONCE_NOCHECK(frame->ret_addr);
-}
-
/*
* Called from fs/proc with a reference on @p to find the function
* which called into schedule(). This needs to be done carefully
cr0 = read_cr0();
cr2 = read_cr2();
- cr3 = read_cr3();
+ cr3 = __read_cr3();
cr4 = __read_cr4();
printk(KERN_DEFAULT "CR0: %08lx CR2: %08lx CR3: %08lx CR4: %08lx\n",
cr0, cr2, cr3, cr4);
cr0 = read_cr0();
cr2 = read_cr2();
- cr3 = read_cr3();
+ cr3 = __read_cr3();
cr4 = __read_cr4();
printk(KERN_DEFAULT "FS: %016lx(%04x) GS:%016lx(%04x) knlGS:%016lx\n",
pr_warn("WARNING: dead process %s still has LDT? <%p/%d>\n",
dead_task->comm,
dead_task->mm->context.ldt->entries,
- dead_task->mm->context.ldt->size);
+ dead_task->mm->context.ldt->nr_entries);
BUG();
}
#endif
#include <linux/sched.h>
#include <linux/tboot.h>
#include <linux/delay.h>
+#include <linux/frame.h>
#include <acpi/reboot.h>
#include <asm/io.h>
#include <asm/apic.h>
#ifdef CONFIG_APM_MODULE
EXPORT_SYMBOL(machine_real_restart);
#endif
+STACK_FRAME_NON_STANDARD(machine_real_restart);
/*
* Some Apple MacBook and MacBookPro's needs reboot=p to be able to reboot
return 0;
}
- low_base = memblock_find_in_range(low_size, 1ULL << 32, low_size, CRASH_ALIGN);
+ low_base = memblock_find_in_range(0, 1ULL << 32, low_size, CRASH_ALIGN);
if (!low_base) {
pr_err("Cannot reserve %ldMB crashkernel low memory, please try smaller size.\n",
(unsigned long)(low_size >> 20));
if (cpu == 1)
printk(KERN_INFO "x86: Booting SMP configuration:\n");
- if (system_state == SYSTEM_BOOTING) {
+ if (system_state < SYSTEM_RUNNING) {
if (node != current_node) {
if (current_node > (-1))
pr_cont("\n");
void play_dead_common(void)
{
idle_task_exit();
- reset_lazy_tlbstate();
/* Ack it */
(void)cpu_report_death();
mutex_lock(&child->mm->context.lock);
if (unlikely(!child->mm->context.ldt ||
- seg >= child->mm->context.ldt->size))
+ seg >= child->mm->context.ldt->nr_entries))
addr = -1L; /* bogus selector, access would fault */
else {
desc = &child->mm->context.ldt->entries[seg];
if (!tboot_enabled())
return 0;
- if (!intel_iommu_tboot_noforce)
+ if (intel_iommu_tboot_noforce)
return 1;
if (no_iommu || swiotlb || dmar_disabled)
.name = "timer"
};
-void __init setup_default_timer_irq(void)
+static void __init setup_default_timer_irq(void)
{
if (!nr_legacy_irqs())
return;
static u64 art_to_tsc_offset;
struct clocksource *art_related_clocksource;
-/*
- * Use a ring-buffer like data structure, where a writer advances the head by
- * writing a new data entry and a reader advances the tail when it observes a
- * new entry.
- *
- * Writers are made to wait on readers until there's space to write a new
- * entry.
- *
- * This means that we can always use an {offset, mul} pair to compute a ns
- * value that is 'roughly' in the right direction, even if we're writing a new
- * {offset, mul} pair during the clock read.
- *
- * The down-side is that we can no longer guarantee strict monotonicity anymore
- * (assuming the TSC was that to begin with), because while we compute the
- * intersection point of the two clock slopes and make sure the time is
- * continuous at the point of switching; we can no longer guarantee a reader is
- * strictly before or after the switch point.
- *
- * It does mean a reader no longer needs to disable IRQs in order to avoid
- * CPU-Freq updates messing with his times, and similarly an NMI reader will
- * no longer run the risk of hitting half-written state.
- */
-
struct cyc2ns {
- struct cyc2ns_data data[2]; /* 0 + 2*24 = 48 */
- struct cyc2ns_data *head; /* 48 + 8 = 56 */
- struct cyc2ns_data *tail; /* 56 + 8 = 64 */
-}; /* exactly fits one cacheline */
-
-static DEFINE_PER_CPU_ALIGNED(struct cyc2ns, cyc2ns);
-
-struct cyc2ns_data *cyc2ns_read_begin(void)
-{
- struct cyc2ns_data *head;
-
- preempt_disable();
+ struct cyc2ns_data data[2]; /* 0 + 2*16 = 32 */
+ seqcount_t seq; /* 32 + 4 = 36 */
- head = this_cpu_read(cyc2ns.head);
- /*
- * Ensure we observe the entry when we observe the pointer to it.
- * matches the wmb from cyc2ns_write_end().
- */
- smp_read_barrier_depends();
- head->__count++;
- barrier();
-
- return head;
-}
+}; /* fits one cacheline */
-void cyc2ns_read_end(struct cyc2ns_data *head)
-{
- barrier();
- /*
- * If we're the outer most nested read; update the tail pointer
- * when we're done. This notifies possible pending writers
- * that we've observed the head pointer and that the other
- * entry is now free.
- */
- if (!--head->__count) {
- /*
- * x86-TSO does not reorder writes with older reads;
- * therefore once this write becomes visible to another
- * cpu, we must be finished reading the cyc2ns_data.
- *
- * matches with cyc2ns_write_begin().
- */
- this_cpu_write(cyc2ns.tail, head);
- }
- preempt_enable();
-}
+static DEFINE_PER_CPU_ALIGNED(struct cyc2ns, cyc2ns);
-/*
- * Begin writing a new @data entry for @cpu.
- *
- * Assumes some sort of write side lock; currently 'provided' by the assumption
- * that cpufreq will call its notifiers sequentially.
- */
-static struct cyc2ns_data *cyc2ns_write_begin(int cpu)
+void cyc2ns_read_begin(struct cyc2ns_data *data)
{
- struct cyc2ns *c2n = &per_cpu(cyc2ns, cpu);
- struct cyc2ns_data *data = c2n->data;
+ int seq, idx;
- if (data == c2n->head)
- data++;
+ preempt_disable_notrace();
- /* XXX send an IPI to @cpu in order to guarantee a read? */
+ do {
+ seq = this_cpu_read(cyc2ns.seq.sequence);
+ idx = seq & 1;
- /*
- * When we observe the tail write from cyc2ns_read_end(),
- * the cpu must be done with that entry and its safe
- * to start writing to it.
- */
- while (c2n->tail == data)
- cpu_relax();
+ data->cyc2ns_offset = this_cpu_read(cyc2ns.data[idx].cyc2ns_offset);
+ data->cyc2ns_mul = this_cpu_read(cyc2ns.data[idx].cyc2ns_mul);
+ data->cyc2ns_shift = this_cpu_read(cyc2ns.data[idx].cyc2ns_shift);
- return data;
+ } while (unlikely(seq != this_cpu_read(cyc2ns.seq.sequence)));
}
-static void cyc2ns_write_end(int cpu, struct cyc2ns_data *data)
+void cyc2ns_read_end(void)
{
- struct cyc2ns *c2n = &per_cpu(cyc2ns, cpu);
-
- /*
- * Ensure the @data writes are visible before we publish the
- * entry. Matches the data-depencency in cyc2ns_read_begin().
- */
- smp_wmb();
-
- ACCESS_ONCE(c2n->head) = data;
+ preempt_enable_notrace();
}
/*
data->cyc2ns_mul = 0;
data->cyc2ns_shift = 0;
data->cyc2ns_offset = 0;
- data->__count = 0;
}
static void cyc2ns_init(int cpu)
cyc2ns_data_init(&c2n->data[0]);
cyc2ns_data_init(&c2n->data[1]);
- c2n->head = c2n->data;
- c2n->tail = c2n->data;
+ seqcount_init(&c2n->seq);
}
static inline unsigned long long cycles_2_ns(unsigned long long cyc)
{
- struct cyc2ns_data *data, *tail;
+ struct cyc2ns_data data;
unsigned long long ns;
- /*
- * See cyc2ns_read_*() for details; replicated in order to avoid
- * an extra few instructions that came with the abstraction.
- * Notable, it allows us to only do the __count and tail update
- * dance when its actually needed.
- */
-
- preempt_disable_notrace();
- data = this_cpu_read(cyc2ns.head);
- tail = this_cpu_read(cyc2ns.tail);
-
- if (likely(data == tail)) {
- ns = data->cyc2ns_offset;
- ns += mul_u64_u32_shr(cyc, data->cyc2ns_mul, data->cyc2ns_shift);
- } else {
- data->__count++;
-
- barrier();
+ cyc2ns_read_begin(&data);
- ns = data->cyc2ns_offset;
- ns += mul_u64_u32_shr(cyc, data->cyc2ns_mul, data->cyc2ns_shift);
+ ns = data.cyc2ns_offset;
+ ns += mul_u64_u32_shr(cyc, data.cyc2ns_mul, data.cyc2ns_shift);
- barrier();
-
- if (!--data->__count)
- this_cpu_write(cyc2ns.tail, data);
- }
- preempt_enable_notrace();
+ cyc2ns_read_end();
return ns;
}
-static void set_cyc2ns_scale(unsigned long khz, int cpu)
+static void set_cyc2ns_scale(unsigned long khz, int cpu, unsigned long long tsc_now)
{
- unsigned long long tsc_now, ns_now;
- struct cyc2ns_data *data;
+ unsigned long long ns_now;
+ struct cyc2ns_data data;
+ struct cyc2ns *c2n;
unsigned long flags;
local_irq_save(flags);
if (!khz)
goto done;
- data = cyc2ns_write_begin(cpu);
-
- tsc_now = rdtsc();
ns_now = cycles_2_ns(tsc_now);
/*
* time function is continuous; see the comment near struct
* cyc2ns_data.
*/
- clocks_calc_mult_shift(&data->cyc2ns_mul, &data->cyc2ns_shift, khz,
+ clocks_calc_mult_shift(&data.cyc2ns_mul, &data.cyc2ns_shift, khz,
NSEC_PER_MSEC, 0);
/*
* conversion algorithm shifting a 32-bit value (now specifies a 64-bit
* value) - refer perf_event_mmap_page documentation in perf_event.h.
*/
- if (data->cyc2ns_shift == 32) {
- data->cyc2ns_shift = 31;
- data->cyc2ns_mul >>= 1;
+ if (data.cyc2ns_shift == 32) {
+ data.cyc2ns_shift = 31;
+ data.cyc2ns_mul >>= 1;
}
- data->cyc2ns_offset = ns_now -
- mul_u64_u32_shr(tsc_now, data->cyc2ns_mul, data->cyc2ns_shift);
+ data.cyc2ns_offset = ns_now -
+ mul_u64_u32_shr(tsc_now, data.cyc2ns_mul, data.cyc2ns_shift);
- cyc2ns_write_end(cpu, data);
+ c2n = per_cpu_ptr(&cyc2ns, cpu);
+
+ raw_write_seqcount_latch(&c2n->seq);
+ c2n->data[0] = data;
+ raw_write_seqcount_latch(&c2n->seq);
+ c2n->data[1] = data;
done:
- sched_clock_idle_wakeup_event(0);
+ sched_clock_idle_wakeup_event();
local_irq_restore(flags);
}
+
/*
* Scheduler clock - returns current time in nanosec units.
*/
tsc_clocksource_reliable = 1;
if (!strncmp(str, "noirqtime", 9))
no_sched_irq_time = 1;
+ if (!strcmp(str, "unstable"))
+ mark_tsc_unstable("boot parameter");
return 1;
}
}
#ifdef CONFIG_CPU_FREQ
-
/* Frequency scaling support. Adjust the TSC based timer when the cpu frequency
* changes.
*
if (!(freq->flags & CPUFREQ_CONST_LOOPS))
mark_tsc_unstable("cpufreq changes");
- set_cyc2ns_scale(tsc_khz, freq->cpu);
+ set_cyc2ns_scale(tsc_khz, freq->cpu, rdtsc());
}
return 0;
pr_info("Marking TSC unstable due to clocksource watchdog\n");
}
+static void tsc_cs_tick_stable(struct clocksource *cs)
+{
+ if (tsc_unstable)
+ return;
+
+ if (using_native_sched_clock())
+ sched_clock_tick_stable();
+}
+
/*
* .mask MUST be CLOCKSOURCE_MASK(64). See comment above read_tsc()
*/
.archdata = { .vclock_mode = VCLOCK_TSC },
.resume = tsc_resume,
.mark_unstable = tsc_cs_mark_unstable,
+ .tick_stable = tsc_cs_tick_stable,
};
void mark_tsc_unstable(char *reason)
static int hpet;
u64 tsc_stop, ref_stop, delta;
unsigned long freq;
+ int cpu;
/* Don't bother refining TSC on unstable systems */
if (check_tsc_unstable())
/* Inform the TSC deadline clockevent devices about the recalibration */
lapic_update_tsc_freq();
+ /* Update the sched_clock() rate to match the clocksource one */
+ for_each_possible_cpu(cpu)
+ set_cyc2ns_scale(tsc_khz, cpu, tsc_stop);
+
out:
if (boot_cpu_has(X86_FEATURE_ART))
art_related_clocksource = &clocksource_tsc;
void __init tsc_init(void)
{
- u64 lpj;
+ u64 lpj, cyc;
int cpu;
if (!boot_cpu_has(X86_FEATURE_TSC)) {
* speed as the bootup CPU. (cpufreq notifiers will fix this
* up if their speed diverges)
*/
+ cyc = rdtsc();
for_each_possible_cpu(cpu) {
cyc2ns_init(cpu);
- set_cyc2ns_scale(tsc_khz, cpu);
+ set_cyc2ns_scale(tsc_khz, cpu, cyc);
}
if (tsc_disabled > 0)
use_tsc_delay();
+ check_system_tsc_reliable();
+
if (unsynchronized_tsc())
mark_tsc_unstable("TSCs unsynchronized");
- check_system_tsc_reliable();
-
detect_art();
}
* non zero. We don't do that on non boot cpus because physical
* hotplug should have set the ADJUST register to a value > 0 so
* the TSC is in sync with the already running cpus.
- *
- * But we always force positive ADJUST values. Otherwise the TSC
- * deadline timer creates an interrupt storm. We also have to
- * prevent values > 0x7FFFFFFF as those wreckage the timer as well.
*/
- if ((bootcpu && bootval != 0) || (!bootcpu && bootval < 0) ||
- (bootval > 0x7FFFFFFF)) {
+ if (bootcpu && bootval != 0) {
pr_warn(FW_BUG "TSC ADJUST: CPU%u: %lld force to 0\n", cpu,
bootval);
wrmsrl(MSR_IA32_TSC_ADJUST, 0);
*/
cur->adjusted += cur_max_warp;
- /*
- * TSC deadline timer stops working or creates an interrupt storm
- * with adjust values < 0 and > x07ffffff.
- *
- * To allow adjust values > 0x7FFFFFFF we need to disable the
- * deadline timer and use the local APIC timer, but that requires
- * more intrusive changes and we do not have any useful information
- * from Intel about the underlying HW wreckage yet.
- */
- if (cur->adjusted < 0)
- cur->adjusted = 0;
- if (cur->adjusted > 0x7FFFFFFF)
- cur->adjusted = 0x7FFFFFFF;
-
pr_warn("TSC ADJUST compensate: CPU%u observed %lld warp. Adjust: %lld\n",
cpu, cur_max_warp, cur->adjusted);
ctxt->eflags &= ~(X86_EFLAGS_VM | X86_EFLAGS_IF);
}
+ ctxt->tf = (ctxt->eflags & X86_EFLAGS_TF) != 0;
return X86EMUL_CONTINUE;
}
#include <linux/slab.h>
#include <linux/amd-iommu.h>
#include <linux/hashtable.h>
+#include <linux/frame.h>
#include <asm/apic.h>
#include <asm/perf_event.h>
mark_all_clean(svm->vmcb);
}
+STACK_FRAME_NON_STANDARD(svm_vcpu_run);
static void svm_set_cr3(struct kvm_vcpu *vcpu, unsigned long root)
{
#include <linux/slab.h>
#include <linux/tboot.h>
#include <linux/hrtimer.h>
+#include <linux/frame.h>
#include "kvm_cache_regs.h"
#include "x86.h"
#include <asm/kexec.h>
#include <asm/apic.h>
#include <asm/irq_remapping.h>
+#include <asm/mmu_context.h>
#include "trace.h"
#include "pmu.h"
int gs_ldt_reload_needed;
int fs_reload_needed;
u64 msr_host_bndcfgs;
+ unsigned long vmcs_host_cr3; /* May not match real cr3 */
unsigned long vmcs_host_cr4; /* May not match real cr4 */
} host_state;
struct {
u32 low32, high32;
unsigned long tmpl;
struct desc_ptr dt;
- unsigned long cr0, cr4;
+ unsigned long cr0, cr3, cr4;
cr0 = read_cr0();
WARN_ON(cr0 & X86_CR0_TS);
vmcs_writel(HOST_CR0, cr0); /* 22.2.3 */
- vmcs_writel(HOST_CR3, read_cr3()); /* 22.2.3 FIXME: shadow tables */
+
+ /*
+ * Save the most likely value for this task's CR3 in the VMCS.
+ * We can't use __get_current_cr3_fast() because we're not atomic.
+ */
+ cr3 = __read_cr3();
+ vmcs_writel(HOST_CR3, cr3); /* 22.2.3 FIXME: shadow tables */
+ vmx->host_state.vmcs_host_cr3 = cr3;
/* Save the most likely value for this task's CR4 in the VMCS. */
cr4 = cr4_read_shadow();
);
}
}
+STACK_FRAME_NON_STANDARD(vmx_handle_external_intr);
static bool vmx_has_high_real_mode_segbase(void)
{
static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
{
struct vcpu_vmx *vmx = to_vmx(vcpu);
- unsigned long debugctlmsr, cr4;
+ unsigned long debugctlmsr, cr3, cr4;
/* Don't enter VMX if guest state is invalid, let the exit handler
start emulation until we arrive back to a valid state */
if (test_bit(VCPU_REGS_RIP, (unsigned long *)&vcpu->arch.regs_dirty))
vmcs_writel(GUEST_RIP, vcpu->arch.regs[VCPU_REGS_RIP]);
+ cr3 = __get_current_cr3_fast();
+ if (unlikely(cr3 != vmx->host_state.vmcs_host_cr3)) {
+ vmcs_writel(HOST_CR3, cr3);
+ vmx->host_state.vmcs_host_cr3 = cr3;
+ }
+
cr4 = cr4_read_shadow();
if (unlikely(cr4 != vmx->host_state.vmcs_host_cr4)) {
vmcs_writel(HOST_CR4, cr4);
vmx_recover_nmi_blocking(vmx);
vmx_complete_interrupts(vmx);
}
+STACK_FRAME_NON_STANDARD(vmx_vcpu_run);
static void vmx_switch_vmcs(struct kvm_vcpu *vcpu, struct loaded_vmcs *vmcs)
{
kvm_x86_ops->get_cs_db_l_bits(vcpu, &cs_db, &cs_l);
ctxt->eflags = kvm_get_rflags(vcpu);
+ ctxt->tf = (ctxt->eflags & X86_EFLAGS_TF) != 0;
+
ctxt->eip = kvm_rip_read(vcpu);
ctxt->mode = (!is_protmode(vcpu)) ? X86EMUL_MODE_REAL :
(ctxt->eflags & X86_EFLAGS_VM) ? X86EMUL_MODE_VM86 :
return dr6;
}
-static void kvm_vcpu_check_singlestep(struct kvm_vcpu *vcpu, unsigned long rflags, int *r)
+static void kvm_vcpu_do_singlestep(struct kvm_vcpu *vcpu, int *r)
{
struct kvm_run *kvm_run = vcpu->run;
- /*
- * rflags is the old, "raw" value of the flags. The new value has
- * not been saved yet.
- *
- * This is correct even for TF set by the guest, because "the
- * processor will not generate this exception after the instruction
- * that sets the TF flag".
- */
- if (unlikely(rflags & X86_EFLAGS_TF)) {
- if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP) {
- kvm_run->debug.arch.dr6 = DR6_BS | DR6_FIXED_1 |
- DR6_RTM;
- kvm_run->debug.arch.pc = vcpu->arch.singlestep_rip;
- kvm_run->debug.arch.exception = DB_VECTOR;
- kvm_run->exit_reason = KVM_EXIT_DEBUG;
- *r = EMULATE_USER_EXIT;
- } else {
- /*
- * "Certain debug exceptions may clear bit 0-3. The
- * remaining contents of the DR6 register are never
- * cleared by the processor".
- */
- vcpu->arch.dr6 &= ~15;
- vcpu->arch.dr6 |= DR6_BS | DR6_RTM;
- kvm_queue_exception(vcpu, DB_VECTOR);
- }
+ if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP) {
+ kvm_run->debug.arch.dr6 = DR6_BS | DR6_FIXED_1 | DR6_RTM;
+ kvm_run->debug.arch.pc = vcpu->arch.singlestep_rip;
+ kvm_run->debug.arch.exception = DB_VECTOR;
+ kvm_run->exit_reason = KVM_EXIT_DEBUG;
+ *r = EMULATE_USER_EXIT;
+ } else {
+ /*
+ * "Certain debug exceptions may clear bit 0-3. The
+ * remaining contents of the DR6 register are never
+ * cleared by the processor".
+ */
+ vcpu->arch.dr6 &= ~15;
+ vcpu->arch.dr6 |= DR6_BS | DR6_RTM;
+ kvm_queue_exception(vcpu, DB_VECTOR);
}
}
int r = EMULATE_DONE;
kvm_x86_ops->skip_emulated_instruction(vcpu);
- kvm_vcpu_check_singlestep(vcpu, rflags, &r);
+
+ /*
+ * rflags is the old, "raw" value of the flags. The new value has
+ * not been saved yet.
+ *
+ * This is correct even for TF set by the guest, because "the
+ * processor will not generate this exception after the instruction
+ * that sets the TF flag".
+ */
+ if (unlikely(rflags & X86_EFLAGS_TF))
+ kvm_vcpu_do_singlestep(vcpu, &r);
return r == EMULATE_DONE;
}
EXPORT_SYMBOL_GPL(kvm_skip_emulated_instruction);
toggle_interruptibility(vcpu, ctxt->interruptibility);
vcpu->arch.emulate_regs_need_sync_to_vcpu = false;
kvm_rip_write(vcpu, ctxt->eip);
- if (r == EMULATE_DONE)
- kvm_vcpu_check_singlestep(vcpu, rflags, &r);
+ if (r == EMULATE_DONE &&
+ (ctxt->tf || (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP)))
+ kvm_vcpu_do_singlestep(vcpu, &r);
if (!ctxt->have_exception ||
exception_type(ctxt->exception.vector) == EXCPT_TRAP)
__kvm_set_rflags(vcpu, ctxt->eflags);
movl %edx,%ecx
andl $63,%edx
shrl $6,%ecx
- jz 17f
+ jz .L_copy_short_string
1: movq (%rsi),%r8
2: movq 1*8(%rsi),%r9
3: movq 2*8(%rsi),%r10
leaq 64(%rdi),%rdi
decl %ecx
jnz 1b
-17: movl %edx,%ecx
+.L_copy_short_string:
+ movl %edx,%ecx
andl $7,%edx
shrl $3,%ecx
jz 20f
*/
ENTRY(copy_user_enhanced_fast_string)
ASM_STAC
+ cmpl $64,%edx
+ jb .L_copy_short_string /* less then 64 bytes, avoid the costly 'rep' */
movl %edx,%ecx
1: rep
movsb
.macro op_safe_regs op
ENTRY(\op\()_safe_regs)
pushq %rbx
- pushq %rbp
+ pushq %r12
movq %rdi, %r10 /* Save pointer */
xorl %r11d, %r11d /* Return value */
movl (%rdi), %eax
movl 4(%rdi), %ecx
movl 8(%rdi), %edx
movl 12(%rdi), %ebx
- movl 20(%rdi), %ebp
+ movl 20(%rdi), %r12d
movl 24(%rdi), %esi
movl 28(%rdi), %edi
1: \op
movl %ecx, 4(%r10)
movl %edx, 8(%r10)
movl %ebx, 12(%r10)
- movl %ebp, 20(%r10)
+ movl %r12d, 20(%r10)
movl %esi, 24(%r10)
movl %edi, 28(%r10)
- popq %rbp
+ popq %r12
popq %rbx
ret
3:
1: fxstor | RDGSBASE Ry (F3),(11B)
2: vldmxcsr Md (v1) | WRFSBASE Ry (F3),(11B)
3: vstmxcsr Md (v1) | WRGSBASE Ry (F3),(11B)
-4: XSAVE
+4: XSAVE | ptwrite Ey (F3),(11B)
5: XRSTOR | lfence (11B)
6: XSAVEOPT | clwb (66) | mfence (11B)
7: clflush | clflushopt (66) | sfence (11B)
#ifdef CONFIG_MODIFY_LDT_SYSCALL
seg >>= 3;
mutex_lock(¤t->mm->context.lock);
- if (current->mm->context.ldt && seg < current->mm->context.ldt->size)
+ if (current->mm->context.ldt && seg < current->mm->context.ldt->nr_entries)
ret = current->mm->context.ldt->entries[seg];
mutex_unlock(¤t->mm->context.lock);
#endif
KCOV_INSTRUMENT_tlb.o := n
obj-y := init.o init_$(BITS).o fault.o ioremap.o extable.o pageattr.o mmap.o \
- pat.o pgtable.o physaddr.o gup.o setup_nx.o tlb.o
+ pat.o pgtable.o physaddr.o setup_nx.o tlb.o
# Make sure __phys_addr has no stackprotector
nostackp := $(call cc-option, -fno-stack-protector)
bool checkwx)
{
#ifdef CONFIG_X86_64
- pgd_t *start = (pgd_t *) &init_level4_pgt;
+ pgd_t *start = (pgd_t *) &init_top_pgt;
#else
pgd_t *start = swapper_pg_dir;
#endif
* Do _not_ use "current" here. We might be inside
* an interrupt in the middle of a task switch..
*/
- pgd_paddr = read_cr3();
+ pgd_paddr = read_cr3_pa();
pmd_k = vmalloc_sync_one(__va(pgd_paddr), address);
if (!pmd_k)
return -1;
static void dump_pagetable(unsigned long address)
{
- pgd_t *base = __va(read_cr3());
+ pgd_t *base = __va(read_cr3_pa());
pgd_t *pgd = &base[pgd_index(address)];
p4d_t *p4d;
pud_t *pud;
* happen within a race in page table update. In the later
* case just flush:
*/
- pgd = (pgd_t *)__va(read_cr3()) + pgd_index(address);
+ pgd = (pgd_t *)__va(read_cr3_pa()) + pgd_index(address);
pgd_ref = pgd_offset_k(address);
if (pgd_none(*pgd_ref))
return -1;
static void dump_pagetable(unsigned long address)
{
- pgd_t *base = __va(read_cr3() & PHYSICAL_PAGE_MASK);
+ pgd_t *base = __va(read_cr3_pa());
pgd_t *pgd = base + pgd_index(address);
p4d_t *p4d;
pud_t *pud;
pgd_t *pgd;
pte_t *pte;
- pgd = __va(read_cr3() & PHYSICAL_PAGE_MASK);
+ pgd = __va(read_cr3_pa());
pgd += pgd_index(address);
pte = lookup_address_in_pgd(pgd, address, &level);
+++ /dev/null
-/*
- * Lockless get_user_pages_fast for x86
- *
- * Copyright (C) 2008 Nick Piggin
- * Copyright (C) 2008 Novell Inc.
- */
-#include <linux/sched.h>
-#include <linux/mm.h>
-#include <linux/vmstat.h>
-#include <linux/highmem.h>
-#include <linux/swap.h>
-#include <linux/memremap.h>
-
-#include <asm/mmu_context.h>
-#include <asm/pgtable.h>
-
-static inline pte_t gup_get_pte(pte_t *ptep)
-{
-#ifndef CONFIG_X86_PAE
- return READ_ONCE(*ptep);
-#else
- /*
- * With get_user_pages_fast, we walk down the pagetables without taking
- * any locks. For this we would like to load the pointers atomically,
- * but that is not possible (without expensive cmpxchg8b) on PAE. What
- * we do have is the guarantee that a pte will only either go from not
- * present to present, or present to not present or both -- it will not
- * switch to a completely different present page without a TLB flush in
- * between; something that we are blocking by holding interrupts off.
- *
- * Setting ptes from not present to present goes:
- * ptep->pte_high = h;
- * smp_wmb();
- * ptep->pte_low = l;
- *
- * And present to not present goes:
- * ptep->pte_low = 0;
- * smp_wmb();
- * ptep->pte_high = 0;
- *
- * We must ensure here that the load of pte_low sees l iff pte_high
- * sees h. We load pte_high *after* loading pte_low, which ensures we
- * don't see an older value of pte_high. *Then* we recheck pte_low,
- * which ensures that we haven't picked up a changed pte high. We might
- * have got rubbish values from pte_low and pte_high, but we are
- * guaranteed that pte_low will not have the present bit set *unless*
- * it is 'l'. And get_user_pages_fast only operates on present ptes, so
- * we're safe.
- *
- * gup_get_pte should not be used or copied outside gup.c without being
- * very careful -- it does not atomically load the pte or anything that
- * is likely to be useful for you.
- */
- pte_t pte;
-
-retry:
- pte.pte_low = ptep->pte_low;
- smp_rmb();
- pte.pte_high = ptep->pte_high;
- smp_rmb();
- if (unlikely(pte.pte_low != ptep->pte_low))
- goto retry;
-
- return pte;
-#endif
-}
-
-static void undo_dev_pagemap(int *nr, int nr_start, struct page **pages)
-{
- while ((*nr) - nr_start) {
- struct page *page = pages[--(*nr)];
-
- ClearPageReferenced(page);
- put_page(page);
- }
-}
-
-/*
- * 'pteval' can come from a pte, pmd, pud or p4d. We only check
- * _PAGE_PRESENT, _PAGE_USER, and _PAGE_RW in here which are the
- * same value on all 4 types.
- */
-static inline int pte_allows_gup(unsigned long pteval, int write)
-{
- unsigned long need_pte_bits = _PAGE_PRESENT|_PAGE_USER;
-
- if (write)
- need_pte_bits |= _PAGE_RW;
-
- if ((pteval & need_pte_bits) != need_pte_bits)
- return 0;
-
- /* Check memory protection keys permissions. */
- if (!__pkru_allows_pkey(pte_flags_pkey(pteval), write))
- return 0;
-
- return 1;
-}
-
-/*
- * The performance critical leaf functions are made noinline otherwise gcc
- * inlines everything into a single function which results in too much
- * register pressure.
- */
-static noinline int gup_pte_range(pmd_t pmd, unsigned long addr,
- unsigned long end, int write, struct page **pages, int *nr)
-{
- struct dev_pagemap *pgmap = NULL;
- int nr_start = *nr, ret = 0;
- pte_t *ptep, *ptem;
-
- /*
- * Keep the original mapped PTE value (ptem) around since we
- * might increment ptep off the end of the page when finishing
- * our loop iteration.
- */
- ptem = ptep = pte_offset_map(&pmd, addr);
- do {
- pte_t pte = gup_get_pte(ptep);
- struct page *page;
-
- /* Similar to the PMD case, NUMA hinting must take slow path */
- if (pte_protnone(pte))
- break;
-
- if (!pte_allows_gup(pte_val(pte), write))
- break;
-
- if (pte_devmap(pte)) {
- pgmap = get_dev_pagemap(pte_pfn(pte), pgmap);
- if (unlikely(!pgmap)) {
- undo_dev_pagemap(nr, nr_start, pages);
- break;
- }
- } else if (pte_special(pte))
- break;
-
- VM_BUG_ON(!pfn_valid(pte_pfn(pte)));
- page = pte_page(pte);
- get_page(page);
- put_dev_pagemap(pgmap);
- SetPageReferenced(page);
- pages[*nr] = page;
- (*nr)++;
-
- } while (ptep++, addr += PAGE_SIZE, addr != end);
- if (addr == end)
- ret = 1;
- pte_unmap(ptem);
-
- return ret;
-}
-
-static inline void get_head_page_multiple(struct page *page, int nr)
-{
- VM_BUG_ON_PAGE(page != compound_head(page), page);
- VM_BUG_ON_PAGE(page_count(page) == 0, page);
- page_ref_add(page, nr);
- SetPageReferenced(page);
-}
-
-static int __gup_device_huge(unsigned long pfn, unsigned long addr,
- unsigned long end, struct page **pages, int *nr)
-{
- int nr_start = *nr;
- struct dev_pagemap *pgmap = NULL;
-
- do {
- struct page *page = pfn_to_page(pfn);
-
- pgmap = get_dev_pagemap(pfn, pgmap);
- if (unlikely(!pgmap)) {
- undo_dev_pagemap(nr, nr_start, pages);
- return 0;
- }
- SetPageReferenced(page);
- pages[*nr] = page;
- get_page(page);
- put_dev_pagemap(pgmap);
- (*nr)++;
- pfn++;
- } while (addr += PAGE_SIZE, addr != end);
- return 1;
-}
-
-static int __gup_device_huge_pmd(pmd_t pmd, unsigned long addr,
- unsigned long end, struct page **pages, int *nr)
-{
- unsigned long fault_pfn;
-
- fault_pfn = pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
- return __gup_device_huge(fault_pfn, addr, end, pages, nr);
-}
-
-static int __gup_device_huge_pud(pud_t pud, unsigned long addr,
- unsigned long end, struct page **pages, int *nr)
-{
- unsigned long fault_pfn;
-
- fault_pfn = pud_pfn(pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT);
- return __gup_device_huge(fault_pfn, addr, end, pages, nr);
-}
-
-static noinline int gup_huge_pmd(pmd_t pmd, unsigned long addr,
- unsigned long end, int write, struct page **pages, int *nr)
-{
- struct page *head, *page;
- int refs;
-
- if (!pte_allows_gup(pmd_val(pmd), write))
- return 0;
-
- VM_BUG_ON(!pfn_valid(pmd_pfn(pmd)));
- if (pmd_devmap(pmd))
- return __gup_device_huge_pmd(pmd, addr, end, pages, nr);
-
- /* hugepages are never "special" */
- VM_BUG_ON(pmd_flags(pmd) & _PAGE_SPECIAL);
-
- refs = 0;
- head = pmd_page(pmd);
- page = head + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
- do {
- VM_BUG_ON_PAGE(compound_head(page) != head, page);
- pages[*nr] = page;
- (*nr)++;
- page++;
- refs++;
- } while (addr += PAGE_SIZE, addr != end);
- get_head_page_multiple(head, refs);
-
- return 1;
-}
-
-static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end,
- int write, struct page **pages, int *nr)
-{
- unsigned long next;
- pmd_t *pmdp;
-
- pmdp = pmd_offset(&pud, addr);
- do {
- pmd_t pmd = *pmdp;
-
- next = pmd_addr_end(addr, end);
- if (pmd_none(pmd))
- return 0;
- if (unlikely(pmd_large(pmd) || !pmd_present(pmd))) {
- /*
- * NUMA hinting faults need to be handled in the GUP
- * slowpath for accounting purposes and so that they
- * can be serialised against THP migration.
- */
- if (pmd_protnone(pmd))
- return 0;
- if (!gup_huge_pmd(pmd, addr, next, write, pages, nr))
- return 0;
- } else {
- if (!gup_pte_range(pmd, addr, next, write, pages, nr))
- return 0;
- }
- } while (pmdp++, addr = next, addr != end);
-
- return 1;
-}
-
-static noinline int gup_huge_pud(pud_t pud, unsigned long addr,
- unsigned long end, int write, struct page **pages, int *nr)
-{
- struct page *head, *page;
- int refs;
-
- if (!pte_allows_gup(pud_val(pud), write))
- return 0;
-
- VM_BUG_ON(!pfn_valid(pud_pfn(pud)));
- if (pud_devmap(pud))
- return __gup_device_huge_pud(pud, addr, end, pages, nr);
-
- /* hugepages are never "special" */
- VM_BUG_ON(pud_flags(pud) & _PAGE_SPECIAL);
-
- refs = 0;
- head = pud_page(pud);
- page = head + ((addr & ~PUD_MASK) >> PAGE_SHIFT);
- do {
- VM_BUG_ON_PAGE(compound_head(page) != head, page);
- pages[*nr] = page;
- (*nr)++;
- page++;
- refs++;
- } while (addr += PAGE_SIZE, addr != end);
- get_head_page_multiple(head, refs);
-
- return 1;
-}
-
-static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end,
- int write, struct page **pages, int *nr)
-{
- unsigned long next;
- pud_t *pudp;
-
- pudp = pud_offset(&p4d, addr);
- do {
- pud_t pud = *pudp;
-
- next = pud_addr_end(addr, end);
- if (pud_none(pud))
- return 0;
- if (unlikely(pud_large(pud))) {
- if (!gup_huge_pud(pud, addr, next, write, pages, nr))
- return 0;
- } else {
- if (!gup_pmd_range(pud, addr, next, write, pages, nr))
- return 0;
- }
- } while (pudp++, addr = next, addr != end);
-
- return 1;
-}
-
-static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end,
- int write, struct page **pages, int *nr)
-{
- unsigned long next;
- p4d_t *p4dp;
-
- p4dp = p4d_offset(&pgd, addr);
- do {
- p4d_t p4d = *p4dp;
-
- next = p4d_addr_end(addr, end);
- if (p4d_none(p4d))
- return 0;
- BUILD_BUG_ON(p4d_large(p4d));
- if (!gup_pud_range(p4d, addr, next, write, pages, nr))
- return 0;
- } while (p4dp++, addr = next, addr != end);
-
- return 1;
-}
-
-/*
- * Like get_user_pages_fast() except its IRQ-safe in that it won't fall
- * back to the regular GUP.
- */
-int __get_user_pages_fast(unsigned long start, int nr_pages, int write,
- struct page **pages)
-{
- struct mm_struct *mm = current->mm;
- unsigned long addr, len, end;
- unsigned long next;
- unsigned long flags;
- pgd_t *pgdp;
- int nr = 0;
-
- start &= PAGE_MASK;
- addr = start;
- len = (unsigned long) nr_pages << PAGE_SHIFT;
- end = start + len;
- if (unlikely(!access_ok(write ? VERIFY_WRITE : VERIFY_READ,
- (void __user *)start, len)))
- return 0;
-
- /*
- * XXX: batch / limit 'nr', to avoid large irq off latency
- * needs some instrumenting to determine the common sizes used by
- * important workloads (eg. DB2), and whether limiting the batch size
- * will decrease performance.
- *
- * It seems like we're in the clear for the moment. Direct-IO is
- * the main guy that batches up lots of get_user_pages, and even
- * they are limited to 64-at-a-time which is not so many.
- */
- /*
- * This doesn't prevent pagetable teardown, but does prevent
- * the pagetables and pages from being freed on x86.
- *
- * So long as we atomically load page table pointers versus teardown
- * (which we do on x86, with the above PAE exception), we can follow the
- * address down to the the page and take a ref on it.
- */
- local_irq_save(flags);
- pgdp = pgd_offset(mm, addr);
- do {
- pgd_t pgd = *pgdp;
-
- next = pgd_addr_end(addr, end);
- if (pgd_none(pgd))
- break;
- if (!gup_p4d_range(pgd, addr, next, write, pages, &nr))
- break;
- } while (pgdp++, addr = next, addr != end);
- local_irq_restore(flags);
-
- return nr;
-}
-
-/**
- * get_user_pages_fast() - pin user pages in memory
- * @start: starting user address
- * @nr_pages: number of pages from start to pin
- * @write: whether pages will be written to
- * @pages: array that receives pointers to the pages pinned.
- * Should be at least nr_pages long.
- *
- * Attempt to pin user pages in memory without taking mm->mmap_sem.
- * If not successful, it will fall back to taking the lock and
- * calling get_user_pages().
- *
- * Returns number of pages pinned. This may be fewer than the number
- * requested. If nr_pages is 0 or negative, returns 0. If no pages
- * were pinned, returns -errno.
- */
-int get_user_pages_fast(unsigned long start, int nr_pages, int write,
- struct page **pages)
-{
- struct mm_struct *mm = current->mm;
- unsigned long addr, len, end;
- unsigned long next;
- pgd_t *pgdp;
- int nr = 0;
-
- start &= PAGE_MASK;
- addr = start;
- len = (unsigned long) nr_pages << PAGE_SHIFT;
-
- end = start + len;
- if (end < start)
- goto slow_irqon;
-
-#ifdef CONFIG_X86_64
- if (end >> __VIRTUAL_MASK_SHIFT)
- goto slow_irqon;
-#endif
-
- /*
- * XXX: batch / limit 'nr', to avoid large irq off latency
- * needs some instrumenting to determine the common sizes used by
- * important workloads (eg. DB2), and whether limiting the batch size
- * will decrease performance.
- *
- * It seems like we're in the clear for the moment. Direct-IO is
- * the main guy that batches up lots of get_user_pages, and even
- * they are limited to 64-at-a-time which is not so many.
- */
- /*
- * This doesn't prevent pagetable teardown, but does prevent
- * the pagetables and pages from being freed on x86.
- *
- * So long as we atomically load page table pointers versus teardown
- * (which we do on x86, with the above PAE exception), we can follow the
- * address down to the the page and take a ref on it.
- */
- local_irq_disable();
- pgdp = pgd_offset(mm, addr);
- do {
- pgd_t pgd = *pgdp;
-
- next = pgd_addr_end(addr, end);
- if (pgd_none(pgd))
- goto slow;
- if (!gup_p4d_range(pgd, addr, next, write, pages, &nr))
- goto slow;
- } while (pgdp++, addr = next, addr != end);
- local_irq_enable();
-
- VM_BUG_ON(nr != (end - start) >> PAGE_SHIFT);
- return nr;
-
- {
- int ret;
-
-slow:
- local_irq_enable();
-slow_irqon:
- /* Try to get the remaining pages with get_user_pages */
- start += nr << PAGE_SHIFT;
- pages += nr;
-
- ret = get_user_pages_unlocked(start,
- (end - start) >> PAGE_SHIFT,
- pages, write ? FOLL_WRITE : 0);
-
- /* Have to be a bit careful with return values */
- if (nr > 0) {
- if (ret < 0)
- ret = nr;
- else
- ret += nr;
- }
-
- return ret;
- }
-}
}
DEFINE_PER_CPU_SHARED_ALIGNED(struct tlb_state, cpu_tlbstate) = {
-#ifdef CONFIG_SMP
- .active_mm = &init_mm,
+ .loaded_mm = &init_mm,
.state = 0,
-#endif
.cr4 = ~0UL, /* fail hard if we screw up cr4 shadow initialization */
};
EXPORT_SYMBOL_GPL(cpu_tlbstate);
* When memory was added make sure all the processes MM have
* suitable PGD entries in the local PGD level page.
*/
+#ifdef CONFIG_X86_5LEVEL
+void sync_global_pgds(unsigned long start, unsigned long end)
+{
+ unsigned long addr;
+
+ for (addr = start; addr <= end; addr = ALIGN(addr + 1, PGDIR_SIZE)) {
+ const pgd_t *pgd_ref = pgd_offset_k(addr);
+ struct page *page;
+
+ /* Check for overflow */
+ if (addr < start)
+ break;
+
+ if (pgd_none(*pgd_ref))
+ continue;
+
+ spin_lock(&pgd_lock);
+ list_for_each_entry(page, &pgd_list, lru) {
+ pgd_t *pgd;
+ spinlock_t *pgt_lock;
+
+ pgd = (pgd_t *)page_address(page) + pgd_index(addr);
+ /* the pgt_lock only for Xen */
+ pgt_lock = &pgd_page_get_mm(page)->page_table_lock;
+ spin_lock(pgt_lock);
+
+ if (!pgd_none(*pgd_ref) && !pgd_none(*pgd))
+ BUG_ON(pgd_page_vaddr(*pgd) != pgd_page_vaddr(*pgd_ref));
+
+ if (pgd_none(*pgd))
+ set_pgd(pgd, *pgd_ref);
+
+ spin_unlock(pgt_lock);
+ }
+ spin_unlock(&pgd_lock);
+ }
+}
+#else
void sync_global_pgds(unsigned long start, unsigned long end)
{
unsigned long addr;
spin_unlock(&pgd_lock);
}
}
+#endif
/*
* NOTE: This function is marked __ref because it calls __init function
return paddr_last;
}
+static unsigned long __meminit
+phys_p4d_init(p4d_t *p4d_page, unsigned long paddr, unsigned long paddr_end,
+ unsigned long page_size_mask)
+{
+ unsigned long paddr_next, paddr_last = paddr_end;
+ unsigned long vaddr = (unsigned long)__va(paddr);
+ int i = p4d_index(vaddr);
+
+ if (!IS_ENABLED(CONFIG_X86_5LEVEL))
+ return phys_pud_init((pud_t *) p4d_page, paddr, paddr_end, page_size_mask);
+
+ for (; i < PTRS_PER_P4D; i++, paddr = paddr_next) {
+ p4d_t *p4d;
+ pud_t *pud;
+
+ vaddr = (unsigned long)__va(paddr);
+ p4d = p4d_page + p4d_index(vaddr);
+ paddr_next = (paddr & P4D_MASK) + P4D_SIZE;
+
+ if (paddr >= paddr_end) {
+ if (!after_bootmem &&
+ !e820__mapped_any(paddr & P4D_MASK, paddr_next,
+ E820_TYPE_RAM) &&
+ !e820__mapped_any(paddr & P4D_MASK, paddr_next,
+ E820_TYPE_RESERVED_KERN))
+ set_p4d(p4d, __p4d(0));
+ continue;
+ }
+
+ if (!p4d_none(*p4d)) {
+ pud = pud_offset(p4d, 0);
+ paddr_last = phys_pud_init(pud, paddr,
+ paddr_end,
+ page_size_mask);
+ __flush_tlb_all();
+ continue;
+ }
+
+ pud = alloc_low_page();
+ paddr_last = phys_pud_init(pud, paddr, paddr_end,
+ page_size_mask);
+
+ spin_lock(&init_mm.page_table_lock);
+ p4d_populate(&init_mm, p4d, pud);
+ spin_unlock(&init_mm.page_table_lock);
+ }
+ __flush_tlb_all();
+
+ return paddr_last;
+}
+
/*
* Create page table mapping for the physical memory for specific physical
* addresses. The virtual and physical addresses have to be aligned on PMD level
for (; vaddr < vaddr_end; vaddr = vaddr_next) {
pgd_t *pgd = pgd_offset_k(vaddr);
p4d_t *p4d;
- pud_t *pud;
vaddr_next = (vaddr & PGDIR_MASK) + PGDIR_SIZE;
- BUILD_BUG_ON(pgd_none(*pgd));
- p4d = p4d_offset(pgd, vaddr);
- if (p4d_val(*p4d)) {
- pud = (pud_t *)p4d_page_vaddr(*p4d);
- paddr_last = phys_pud_init(pud, __pa(vaddr),
+ if (pgd_val(*pgd)) {
+ p4d = (p4d_t *)pgd_page_vaddr(*pgd);
+ paddr_last = phys_p4d_init(p4d, __pa(vaddr),
__pa(vaddr_end),
page_size_mask);
continue;
}
- pud = alloc_low_page();
- paddr_last = phys_pud_init(pud, __pa(vaddr), __pa(vaddr_end),
+ p4d = alloc_low_page();
+ paddr_last = phys_p4d_init(p4d, __pa(vaddr), __pa(vaddr_end),
page_size_mask);
spin_lock(&init_mm.page_table_lock);
- p4d_populate(&init_mm, p4d, pud);
+ if (IS_ENABLED(CONFIG_X86_5LEVEL))
+ pgd_populate(&init_mm, pgd, p4d);
+ else
+ p4d_populate(&init_mm, p4d_offset(pgd, vaddr), (pud_t *) p4d);
spin_unlock(&init_mm.page_table_lock);
pgd_changed = true;
}
pud_base = pud_offset(p4d, 0);
remove_pud_table(pud_base, addr, next, direct);
- free_pud_table(pud_base, p4d);
+ /*
+ * For 4-level page tables we do not want to free PUDs, but in the
+ * 5-level case we should free them. This code will have to change
+ * to adapt for boot-time switching between 4 and 5 level page tables.
+ */
+ if (CONFIG_PGTABLE_LEVELS == 5)
+ free_pud_table(pud_base, p4d);
}
if (direct)
static inline pmd_t * __init early_ioremap_pmd(unsigned long addr)
{
/* Don't assume we're using swapper_pg_dir at this point */
- pgd_t *base = __va(read_cr3());
+ pgd_t *base = __va(read_cr3_pa());
pgd_t *pgd = &base[pgd_index(addr)];
p4d_t *p4d = p4d_offset(pgd, addr);
pud_t *pud = pud_offset(p4d, addr);
#include <asm/tlbflush.h>
#include <asm/sections.h>
-extern pgd_t early_level4_pgt[PTRS_PER_PGD];
+extern pgd_t early_top_pgt[PTRS_PER_PGD];
extern struct range pfn_mapped[E820_MAX_ENTRIES];
static int __init map_range(struct range *range)
for (i = 0; CONFIG_PGTABLE_LEVELS >= 5 && i < PTRS_PER_P4D; i++)
kasan_zero_p4d[i] = __p4d(p4d_val);
- kasan_map_early_shadow(early_level4_pgt);
- kasan_map_early_shadow(init_level4_pgt);
+ kasan_map_early_shadow(early_top_pgt);
+ kasan_map_early_shadow(init_top_pgt);
}
void __init kasan_init(void)
register_die_notifier(&kasan_die_notifier);
#endif
- memcpy(early_level4_pgt, init_level4_pgt, sizeof(early_level4_pgt));
- load_cr3(early_level4_pgt);
+ memcpy(early_top_pgt, init_top_pgt, sizeof(early_top_pgt));
+ load_cr3(early_top_pgt);
__flush_tlb_all();
clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END);
kasan_populate_zero_shadow(kasan_mem_to_shadow((void *)MODULES_END),
(void *)KASAN_SHADOW_END);
- load_cr3(init_level4_pgt);
+ load_cr3(init_top_pgt);
__flush_tlb_all();
/*
*
* Entropy is generated using the KASLR early boot functions now shared in
* the lib directory (originally written by Kees Cook). Randomization is
- * done on PGD & PUD page table levels to increase possible addresses. The
- * physical memory mapping code was adapted to support PUD level virtual
- * addresses. This implementation on the best configuration provides 30,000
- * possible virtual addresses in average for each memory region. An additional
- * low memory page is used to ensure each CPU can start with a PGD aligned
- * virtual address (for realmode).
+ * done on PGD & P4D/PUD page table levels to increase possible addresses.
+ * The physical memory mapping code was adapted to support P4D/PUD level
+ * virtual addresses. This implementation on the best configuration provides
+ * 30,000 possible virtual addresses in average for each memory region.
+ * An additional low memory page is used to ensure each CPU can start with
+ * a PGD aligned virtual address (for realmode).
*
* The order of each memory region is not changed. The feature looks at
* the available space for the regions based on different configuration
unsigned long *base;
unsigned long size_tb;
} kaslr_regions[] = {
- { &page_offset_base, 64/* Maximum */ },
+ { &page_offset_base, 1 << (__PHYSICAL_MASK_SHIFT - TB_SHIFT) /* Maximum */ },
{ &vmalloc_base, VMALLOC_SIZE_TB },
{ &vmemmap_base, 1 },
};
*/
entropy = remain_entropy / (ARRAY_SIZE(kaslr_regions) - i);
prandom_bytes_state(&rand_state, &rand, sizeof(rand));
- entropy = (rand % (entropy + 1)) & PUD_MASK;
+ if (IS_ENABLED(CONFIG_X86_5LEVEL))
+ entropy = (rand % (entropy + 1)) & P4D_MASK;
+ else
+ entropy = (rand % (entropy + 1)) & PUD_MASK;
vaddr += entropy;
*kaslr_regions[i].base = vaddr;
* randomization alignment.
*/
vaddr += get_padding(&kaslr_regions[i]);
- vaddr = round_up(vaddr + 1, PUD_SIZE);
+ if (IS_ENABLED(CONFIG_X86_5LEVEL))
+ vaddr = round_up(vaddr + 1, P4D_SIZE);
+ else
+ vaddr = round_up(vaddr + 1, PUD_SIZE);
remain_entropy -= entropy;
}
}
-/*
- * Create PGD aligned trampoline table to allow real mode initialization
- * of additional CPUs. Consume only 1 low memory page.
- */
-void __meminit init_trampoline(void)
+static void __meminit init_trampoline_pud(void)
{
unsigned long paddr, paddr_next;
pgd_t *pgd;
pud_t *pud_page, *pud_page_tramp;
int i;
- if (!kaslr_memory_enabled()) {
- init_trampoline_default();
- return;
- }
-
pud_page_tramp = alloc_low_page();
paddr = 0;
set_pgd(&trampoline_pgd_entry,
__pgd(_KERNPG_TABLE | __pa(pud_page_tramp)));
}
+
+static void __meminit init_trampoline_p4d(void)
+{
+ unsigned long paddr, paddr_next;
+ pgd_t *pgd;
+ p4d_t *p4d_page, *p4d_page_tramp;
+ int i;
+
+ p4d_page_tramp = alloc_low_page();
+
+ paddr = 0;
+ pgd = pgd_offset_k((unsigned long)__va(paddr));
+ p4d_page = (p4d_t *) pgd_page_vaddr(*pgd);
+
+ for (i = p4d_index(paddr); i < PTRS_PER_P4D; i++, paddr = paddr_next) {
+ p4d_t *p4d, *p4d_tramp;
+ unsigned long vaddr = (unsigned long)__va(paddr);
+
+ p4d_tramp = p4d_page_tramp + p4d_index(paddr);
+ p4d = p4d_page + p4d_index(vaddr);
+ paddr_next = (paddr & P4D_MASK) + P4D_SIZE;
+
+ *p4d_tramp = *p4d;
+ }
+
+ set_pgd(&trampoline_pgd_entry,
+ __pgd(_KERNPG_TABLE | __pa(p4d_page_tramp)));
+}
+
+/*
+ * Create PGD aligned trampoline table to allow real mode initialization
+ * of additional CPUs. Consume only 1 low memory page.
+ */
+void __meminit init_trampoline(void)
+{
+
+ if (!kaslr_memory_enabled()) {
+ init_trampoline_default();
+ return;
+ }
+
+ if (IS_ENABLED(CONFIG_X86_5LEVEL))
+ init_trampoline_p4d();
+ else
+ init_trampoline_pud();
+}
if (current->personality & ADDR_COMPAT_LAYOUT)
return 1;
- if (rlimit(RLIMIT_STACK) == RLIM_INFINITY)
- return 1;
-
return sysctl_legacy_va_layout;
}
#include <linux/debugfs.h>
/*
- * Smarter SMP flushing macros.
+ * TLB flushing, formerly SMP-only
* c/o Linus Torvalds.
*
* These mean you can really definitely utterly forget about
* Implement flush IPI by CALL_FUNCTION_VECTOR, Alex Shi
*/
-#ifdef CONFIG_SMP
-
-struct flush_tlb_info {
- struct mm_struct *flush_mm;
- unsigned long flush_start;
- unsigned long flush_end;
-};
-
-/*
- * We cannot call mmdrop() because we are in interrupt context,
- * instead update mm->cpu_vm_mask.
- */
void leave_mm(int cpu)
{
- struct mm_struct *active_mm = this_cpu_read(cpu_tlbstate.active_mm);
+ struct mm_struct *loaded_mm = this_cpu_read(cpu_tlbstate.loaded_mm);
+
+ /*
+ * It's plausible that we're in lazy TLB mode while our mm is init_mm.
+ * If so, our callers still expect us to flush the TLB, but there
+ * aren't any user TLB entries in init_mm to worry about.
+ *
+ * This needs to happen before any other sanity checks due to
+ * intel_idle's shenanigans.
+ */
+ if (loaded_mm == &init_mm)
+ return;
+
if (this_cpu_read(cpu_tlbstate.state) == TLBSTATE_OK)
BUG();
- if (cpumask_test_cpu(cpu, mm_cpumask(active_mm))) {
- cpumask_clear_cpu(cpu, mm_cpumask(active_mm));
- load_cr3(swapper_pg_dir);
- /*
- * This gets called in the idle path where RCU
- * functions differently. Tracing normally
- * uses RCU, so we have to call the tracepoint
- * specially here.
- */
- trace_tlb_flush_rcuidle(TLB_FLUSH_ON_TASK_SWITCH, TLB_FLUSH_ALL);
- }
+
+ switch_mm(NULL, &init_mm, NULL);
}
EXPORT_SYMBOL_GPL(leave_mm);
-#endif /* CONFIG_SMP */
-
void switch_mm(struct mm_struct *prev, struct mm_struct *next,
struct task_struct *tsk)
{
struct task_struct *tsk)
{
unsigned cpu = smp_processor_id();
+ struct mm_struct *real_prev = this_cpu_read(cpu_tlbstate.loaded_mm);
- if (likely(prev != next)) {
- if (IS_ENABLED(CONFIG_VMAP_STACK)) {
- /*
- * If our current stack is in vmalloc space and isn't
- * mapped in the new pgd, we'll double-fault. Forcibly
- * map it.
- */
- unsigned int stack_pgd_index = pgd_index(current_stack_pointer());
-
- pgd_t *pgd = next->pgd + stack_pgd_index;
-
- if (unlikely(pgd_none(*pgd)))
- set_pgd(pgd, init_mm.pgd[stack_pgd_index]);
- }
+ /*
+ * NB: The scheduler will call us with prev == next when
+ * switching from lazy TLB mode to normal mode if active_mm
+ * isn't changing. When this happens, there is no guarantee
+ * that CR3 (and hence cpu_tlbstate.loaded_mm) matches next.
+ *
+ * NB: leave_mm() calls us with prev == NULL and tsk == NULL.
+ */
-#ifdef CONFIG_SMP
- this_cpu_write(cpu_tlbstate.state, TLBSTATE_OK);
- this_cpu_write(cpu_tlbstate.active_mm, next);
-#endif
+ this_cpu_write(cpu_tlbstate.state, TLBSTATE_OK);
- cpumask_set_cpu(cpu, mm_cpumask(next));
+ if (real_prev == next) {
+ /*
+ * There's nothing to do: we always keep the per-mm control
+ * regs in sync with cpu_tlbstate.loaded_mm. Just
+ * sanity-check mm_cpumask.
+ */
+ if (WARN_ON_ONCE(!cpumask_test_cpu(cpu, mm_cpumask(next))))
+ cpumask_set_cpu(cpu, mm_cpumask(next));
+ return;
+ }
+ if (IS_ENABLED(CONFIG_VMAP_STACK)) {
/*
- * Re-load page tables.
- *
- * This logic has an ordering constraint:
- *
- * CPU 0: Write to a PTE for 'next'
- * CPU 0: load bit 1 in mm_cpumask. if nonzero, send IPI.
- * CPU 1: set bit 1 in next's mm_cpumask
- * CPU 1: load from the PTE that CPU 0 writes (implicit)
- *
- * We need to prevent an outcome in which CPU 1 observes
- * the new PTE value and CPU 0 observes bit 1 clear in
- * mm_cpumask. (If that occurs, then the IPI will never
- * be sent, and CPU 0's TLB will contain a stale entry.)
- *
- * The bad outcome can occur if either CPU's load is
- * reordered before that CPU's store, so both CPUs must
- * execute full barriers to prevent this from happening.
- *
- * Thus, switch_mm needs a full barrier between the
- * store to mm_cpumask and any operation that could load
- * from next->pgd. TLB fills are special and can happen
- * due to instruction fetches or for no reason at all,
- * and neither LOCK nor MFENCE orders them.
- * Fortunately, load_cr3() is serializing and gives the
- * ordering guarantee we need.
- *
+ * If our current stack is in vmalloc space and isn't
+ * mapped in the new pgd, we'll double-fault. Forcibly
+ * map it.
*/
- load_cr3(next->pgd);
+ unsigned int stack_pgd_index = pgd_index(current_stack_pointer());
- trace_tlb_flush(TLB_FLUSH_ON_TASK_SWITCH, TLB_FLUSH_ALL);
+ pgd_t *pgd = next->pgd + stack_pgd_index;
- /* Stop flush ipis for the previous mm */
- cpumask_clear_cpu(cpu, mm_cpumask(prev));
+ if (unlikely(pgd_none(*pgd)))
+ set_pgd(pgd, init_mm.pgd[stack_pgd_index]);
+ }
- /* Load per-mm CR4 state */
- load_mm_cr4(next);
+ this_cpu_write(cpu_tlbstate.loaded_mm, next);
-#ifdef CONFIG_MODIFY_LDT_SYSCALL
- /*
- * Load the LDT, if the LDT is different.
- *
- * It's possible that prev->context.ldt doesn't match
- * the LDT register. This can happen if leave_mm(prev)
- * was called and then modify_ldt changed
- * prev->context.ldt but suppressed an IPI to this CPU.
- * In this case, prev->context.ldt != NULL, because we
- * never set context.ldt to NULL while the mm still
- * exists. That means that next->context.ldt !=
- * prev->context.ldt, because mms never share an LDT.
- */
- if (unlikely(prev->context.ldt != next->context.ldt))
- load_mm_ldt(next);
-#endif
+ WARN_ON_ONCE(cpumask_test_cpu(cpu, mm_cpumask(next)));
+ cpumask_set_cpu(cpu, mm_cpumask(next));
+
+ /*
+ * Re-load page tables.
+ *
+ * This logic has an ordering constraint:
+ *
+ * CPU 0: Write to a PTE for 'next'
+ * CPU 0: load bit 1 in mm_cpumask. if nonzero, send IPI.
+ * CPU 1: set bit 1 in next's mm_cpumask
+ * CPU 1: load from the PTE that CPU 0 writes (implicit)
+ *
+ * We need to prevent an outcome in which CPU 1 observes
+ * the new PTE value and CPU 0 observes bit 1 clear in
+ * mm_cpumask. (If that occurs, then the IPI will never
+ * be sent, and CPU 0's TLB will contain a stale entry.)
+ *
+ * The bad outcome can occur if either CPU's load is
+ * reordered before that CPU's store, so both CPUs must
+ * execute full barriers to prevent this from happening.
+ *
+ * Thus, switch_mm needs a full barrier between the
+ * store to mm_cpumask and any operation that could load
+ * from next->pgd. TLB fills are special and can happen
+ * due to instruction fetches or for no reason at all,
+ * and neither LOCK nor MFENCE orders them.
+ * Fortunately, load_cr3() is serializing and gives the
+ * ordering guarantee we need.
+ */
+ load_cr3(next->pgd);
+
+ /*
+ * This gets called via leave_mm() in the idle path where RCU
+ * functions differently. Tracing normally uses RCU, so we have to
+ * call the tracepoint specially here.
+ */
+ trace_tlb_flush_rcuidle(TLB_FLUSH_ON_TASK_SWITCH, TLB_FLUSH_ALL);
+
+ /* Stop flush ipis for the previous mm */
+ WARN_ON_ONCE(!cpumask_test_cpu(cpu, mm_cpumask(real_prev)) &&
+ real_prev != &init_mm);
+ cpumask_clear_cpu(cpu, mm_cpumask(real_prev));
+
+ /* Load per-mm CR4 and LDTR state */
+ load_mm_cr4(next);
+ switch_ldt(real_prev, next);
+}
+
+static void flush_tlb_func_common(const struct flush_tlb_info *f,
+ bool local, enum tlb_flush_reason reason)
+{
+ /* This code cannot presently handle being reentered. */
+ VM_WARN_ON(!irqs_disabled());
+
+ if (this_cpu_read(cpu_tlbstate.state) != TLBSTATE_OK) {
+ leave_mm(smp_processor_id());
+ return;
}
-#ifdef CONFIG_SMP
- else {
- this_cpu_write(cpu_tlbstate.state, TLBSTATE_OK);
- BUG_ON(this_cpu_read(cpu_tlbstate.active_mm) != next);
-
- if (!cpumask_test_cpu(cpu, mm_cpumask(next))) {
- /*
- * On established mms, the mm_cpumask is only changed
- * from irq context, from ptep_clear_flush() while in
- * lazy tlb mode, and here. Irqs are blocked during
- * schedule, protecting us from simultaneous changes.
- */
- cpumask_set_cpu(cpu, mm_cpumask(next));
- /*
- * We were in lazy tlb mode and leave_mm disabled
- * tlb flush IPI delivery. We must reload CR3
- * to make sure to use no freed page tables.
- *
- * As above, load_cr3() is serializing and orders TLB
- * fills with respect to the mm_cpumask write.
- */
- load_cr3(next->pgd);
- trace_tlb_flush(TLB_FLUSH_ON_TASK_SWITCH, TLB_FLUSH_ALL);
- load_mm_cr4(next);
- load_mm_ldt(next);
+ if (f->end == TLB_FLUSH_ALL) {
+ local_flush_tlb();
+ if (local)
+ count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ALL);
+ trace_tlb_flush(reason, TLB_FLUSH_ALL);
+ } else {
+ unsigned long addr;
+ unsigned long nr_pages = (f->end - f->start) >> PAGE_SHIFT;
+ addr = f->start;
+ while (addr < f->end) {
+ __flush_tlb_single(addr);
+ addr += PAGE_SIZE;
}
+ if (local)
+ count_vm_tlb_events(NR_TLB_LOCAL_FLUSH_ONE, nr_pages);
+ trace_tlb_flush(reason, nr_pages);
}
-#endif
}
-#ifdef CONFIG_SMP
+static void flush_tlb_func_local(void *info, enum tlb_flush_reason reason)
+{
+ const struct flush_tlb_info *f = info;
-/*
- * The flush IPI assumes that a thread switch happens in this order:
- * [cpu0: the cpu that switches]
- * 1) switch_mm() either 1a) or 1b)
- * 1a) thread switch to a different mm
- * 1a1) set cpu_tlbstate to TLBSTATE_OK
- * Now the tlb flush NMI handler flush_tlb_func won't call leave_mm
- * if cpu0 was in lazy tlb mode.
- * 1a2) update cpu active_mm
- * Now cpu0 accepts tlb flushes for the new mm.
- * 1a3) cpu_set(cpu, new_mm->cpu_vm_mask);
- * Now the other cpus will send tlb flush ipis.
- * 1a4) change cr3.
- * 1a5) cpu_clear(cpu, old_mm->cpu_vm_mask);
- * Stop ipi delivery for the old mm. This is not synchronized with
- * the other cpus, but flush_tlb_func ignore flush ipis for the wrong
- * mm, and in the worst case we perform a superfluous tlb flush.
- * 1b) thread switch without mm change
- * cpu active_mm is correct, cpu0 already handles flush ipis.
- * 1b1) set cpu_tlbstate to TLBSTATE_OK
- * 1b2) test_and_set the cpu bit in cpu_vm_mask.
- * Atomically set the bit [other cpus will start sending flush ipis],
- * and test the bit.
- * 1b3) if the bit was 0: leave_mm was called, flush the tlb.
- * 2) switch %%esp, ie current
- *
- * The interrupt must handle 2 special cases:
- * - cr3 is changed before %%esp, ie. it cannot use current->{active_,}mm.
- * - the cpu performs speculative tlb reads, i.e. even if the cpu only
- * runs in kernel space, the cpu could load tlb entries for user space
- * pages.
- *
- * The good news is that cpu_tlbstate is local to each cpu, no
- * write/read ordering problems.
- */
+ flush_tlb_func_common(f, true, reason);
+}
-/*
- * TLB flush funcation:
- * 1) Flush the tlb entries if the cpu uses the mm that's being flushed.
- * 2) Leave the mm if we are in the lazy tlb mode.
- */
-static void flush_tlb_func(void *info)
+static void flush_tlb_func_remote(void *info)
{
- struct flush_tlb_info *f = info;
+ const struct flush_tlb_info *f = info;
inc_irq_stat(irq_tlb_count);
- if (f->flush_mm && f->flush_mm != this_cpu_read(cpu_tlbstate.active_mm))
+ if (f->mm && f->mm != this_cpu_read(cpu_tlbstate.loaded_mm))
return;
count_vm_tlb_event(NR_TLB_REMOTE_FLUSH_RECEIVED);
- if (this_cpu_read(cpu_tlbstate.state) == TLBSTATE_OK) {
- if (f->flush_end == TLB_FLUSH_ALL) {
- local_flush_tlb();
- trace_tlb_flush(TLB_REMOTE_SHOOTDOWN, TLB_FLUSH_ALL);
- } else {
- unsigned long addr;
- unsigned long nr_pages =
- (f->flush_end - f->flush_start) / PAGE_SIZE;
- addr = f->flush_start;
- while (addr < f->flush_end) {
- __flush_tlb_single(addr);
- addr += PAGE_SIZE;
- }
- trace_tlb_flush(TLB_REMOTE_SHOOTDOWN, nr_pages);
- }
- } else
- leave_mm(smp_processor_id());
-
+ flush_tlb_func_common(f, false, TLB_REMOTE_SHOOTDOWN);
}
void native_flush_tlb_others(const struct cpumask *cpumask,
- struct mm_struct *mm, unsigned long start,
- unsigned long end)
+ const struct flush_tlb_info *info)
{
- struct flush_tlb_info info;
-
- info.flush_mm = mm;
- info.flush_start = start;
- info.flush_end = end;
-
count_vm_tlb_event(NR_TLB_REMOTE_FLUSH);
- if (end == TLB_FLUSH_ALL)
+ if (info->end == TLB_FLUSH_ALL)
trace_tlb_flush(TLB_REMOTE_SEND_IPI, TLB_FLUSH_ALL);
else
trace_tlb_flush(TLB_REMOTE_SEND_IPI,
- (end - start) >> PAGE_SHIFT);
+ (info->end - info->start) >> PAGE_SHIFT);
if (is_uv_system()) {
unsigned int cpu;
cpu = smp_processor_id();
- cpumask = uv_flush_tlb_others(cpumask, mm, start, end, cpu);
+ cpumask = uv_flush_tlb_others(cpumask, info);
if (cpumask)
- smp_call_function_many(cpumask, flush_tlb_func,
- &info, 1);
+ smp_call_function_many(cpumask, flush_tlb_func_remote,
+ (void *)info, 1);
return;
}
- smp_call_function_many(cpumask, flush_tlb_func, &info, 1);
+ smp_call_function_many(cpumask, flush_tlb_func_remote,
+ (void *)info, 1);
}
/*
void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
unsigned long end, unsigned long vmflag)
{
- unsigned long addr;
- /* do a global flush by default */
- unsigned long base_pages_to_flush = TLB_FLUSH_ALL;
-
- preempt_disable();
+ int cpu;
- if ((end != TLB_FLUSH_ALL) && !(vmflag & VM_HUGETLB))
- base_pages_to_flush = (end - start) >> PAGE_SHIFT;
- if (base_pages_to_flush > tlb_single_page_flush_ceiling)
- base_pages_to_flush = TLB_FLUSH_ALL;
+ struct flush_tlb_info info = {
+ .mm = mm,
+ };
- if (current->active_mm != mm) {
- /* Synchronize with switch_mm. */
- smp_mb();
+ cpu = get_cpu();
- goto out;
- }
-
- if (!current->mm) {
- leave_mm(smp_processor_id());
+ /* Synchronize with switch_mm. */
+ smp_mb();
- /* Synchronize with switch_mm. */
- smp_mb();
-
- goto out;
- }
-
- /*
- * Both branches below are implicit full barriers (MOV to CR or
- * INVLPG) that synchronize with switch_mm.
- */
- if (base_pages_to_flush == TLB_FLUSH_ALL) {
- count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ALL);
- local_flush_tlb();
+ /* Should we flush just the requested range? */
+ if ((end != TLB_FLUSH_ALL) &&
+ !(vmflag & VM_HUGETLB) &&
+ ((end - start) >> PAGE_SHIFT) <= tlb_single_page_flush_ceiling) {
+ info.start = start;
+ info.end = end;
} else {
- /* flush range by one by one 'invlpg' */
- for (addr = start; addr < end; addr += PAGE_SIZE) {
- count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ONE);
- __flush_tlb_single(addr);
- }
- }
- trace_tlb_flush(TLB_LOCAL_MM_SHOOTDOWN, base_pages_to_flush);
-out:
- if (base_pages_to_flush == TLB_FLUSH_ALL) {
- start = 0UL;
- end = TLB_FLUSH_ALL;
+ info.start = 0UL;
+ info.end = TLB_FLUSH_ALL;
}
- if (cpumask_any_but(mm_cpumask(mm), smp_processor_id()) < nr_cpu_ids)
- flush_tlb_others(mm_cpumask(mm), mm, start, end);
- preempt_enable();
-}
-void flush_tlb_page(struct vm_area_struct *vma, unsigned long start)
-{
- struct mm_struct *mm = vma->vm_mm;
-
- preempt_disable();
-
- if (current->active_mm == mm) {
- if (current->mm) {
- /*
- * Implicit full barrier (INVLPG) that synchronizes
- * with switch_mm.
- */
- __flush_tlb_one(start);
- } else {
- leave_mm(smp_processor_id());
-
- /* Synchronize with switch_mm. */
- smp_mb();
- }
+ if (mm == this_cpu_read(cpu_tlbstate.loaded_mm)) {
+ VM_WARN_ON(irqs_disabled());
+ local_irq_disable();
+ flush_tlb_func_local(&info, TLB_LOCAL_MM_SHOOTDOWN);
+ local_irq_enable();
}
- if (cpumask_any_but(mm_cpumask(mm), smp_processor_id()) < nr_cpu_ids)
- flush_tlb_others(mm_cpumask(mm), mm, start, start + PAGE_SIZE);
-
- preempt_enable();
+ if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids)
+ flush_tlb_others(mm_cpumask(mm), &info);
+ put_cpu();
}
+
static void do_flush_tlb_all(void *info)
{
count_vm_tlb_event(NR_TLB_REMOTE_FLUSH_RECEIVED);
unsigned long addr;
/* flush range by one by one 'invlpg' */
- for (addr = f->flush_start; addr < f->flush_end; addr += PAGE_SIZE)
+ for (addr = f->start; addr < f->end; addr += PAGE_SIZE)
__flush_tlb_single(addr);
}
/* Balance as user space task's flush, a bit conservative */
if (end == TLB_FLUSH_ALL ||
- (end - start) > tlb_single_page_flush_ceiling * PAGE_SIZE) {
+ (end - start) > tlb_single_page_flush_ceiling << PAGE_SHIFT) {
on_each_cpu(do_flush_tlb_all, NULL, 1);
} else {
struct flush_tlb_info info;
- info.flush_start = start;
- info.flush_end = end;
+ info.start = start;
+ info.end = end;
on_each_cpu(do_kernel_range_flush, &info, 1);
}
}
+void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
+{
+ struct flush_tlb_info info = {
+ .mm = NULL,
+ .start = 0UL,
+ .end = TLB_FLUSH_ALL,
+ };
+
+ int cpu = get_cpu();
+
+ if (cpumask_test_cpu(cpu, &batch->cpumask)) {
+ VM_WARN_ON(irqs_disabled());
+ local_irq_disable();
+ flush_tlb_func_local(&info, TLB_LOCAL_SHOOTDOWN);
+ local_irq_enable();
+ }
+
+ if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids)
+ flush_tlb_others(&batch->cpumask, &info);
+ cpumask_clear(&batch->cpumask);
+
+ put_cpu();
+}
+
static ssize_t tlbflush_read_file(struct file *file, char __user *user_buf,
size_t count, loff_t *ppos)
{
return 0;
}
late_initcall(create_tlb_single_page_flush_ceiling);
-
-#endif /* CONFIG_SMP */
#
# Arch-specific network modules
#
+OBJECT_FILES_NON_STANDARD_bpf_jit.o += y
+
obj-$(CONFIG_BPF_JIT) += bpf_jit.o bpf_jit_comp.o
{ PCI_DEVFN(device, func), offset, init_op, read_op, write_op,\
{0, SIZE_TO_MASK(size)} },
+/*
+ * All read/write functions are called with pci_config_lock held.
+ */
static void reg_init(struct sim_dev_reg *reg)
{
pci_direct_conf1.read(0, 1, reg->dev_func, reg->reg, 4,
static void reg_read(struct sim_dev_reg *reg, u32 *value)
{
- unsigned long flags;
-
- raw_spin_lock_irqsave(&pci_config_lock, flags);
*value = reg->sim_reg.value;
- raw_spin_unlock_irqrestore(&pci_config_lock, flags);
}
static void reg_write(struct sim_dev_reg *reg, u32 value)
{
- unsigned long flags;
-
- raw_spin_lock_irqsave(&pci_config_lock, flags);
reg->sim_reg.value = (value & reg->sim_reg.mask) |
(reg->sim_reg.value & ~reg->sim_reg.mask);
- raw_spin_unlock_irqrestore(&pci_config_lock, flags);
}
static void sata_reg_init(struct sim_dev_reg *reg)
static void reg_noirq_read(struct sim_dev_reg *reg, u32 *value)
{
- unsigned long flags;
-
- raw_spin_lock_irqsave(&pci_config_lock, flags);
/* force interrupt pin value to 0 */
*value = reg->sim_reg.value & 0xfff00ff;
- raw_spin_unlock_irqrestore(&pci_config_lock, flags);
}
static struct sim_dev_reg bus1_fixups[] = {
return retval;
}
-static int ce4100_conf_read(unsigned int seg, unsigned int bus,
- unsigned int devfn, int reg, int len, u32 *value)
+static int ce4100_bus1_read(unsigned int devfn, int reg, int len, u32 *value)
{
+ unsigned long flags;
int i;
- WARN_ON(seg);
- if (bus == 1) {
- for (i = 0; i < ARRAY_SIZE(bus1_fixups); i++) {
- if (bus1_fixups[i].dev_func == devfn &&
- bus1_fixups[i].reg == (reg & ~3) &&
- bus1_fixups[i].read) {
- bus1_fixups[i].read(&(bus1_fixups[i]),
- value);
- extract_bytes(value, reg, len);
- return 0;
- }
+ for (i = 0; i < ARRAY_SIZE(bus1_fixups); i++) {
+ if (bus1_fixups[i].dev_func == devfn &&
+ bus1_fixups[i].reg == (reg & ~3) &&
+ bus1_fixups[i].read) {
+
+ raw_spin_lock_irqsave(&pci_config_lock, flags);
+ bus1_fixups[i].read(&(bus1_fixups[i]), value);
+ raw_spin_unlock_irqrestore(&pci_config_lock, flags);
+ extract_bytes(value, reg, len);
+ return 0;
}
}
+ return -1;
+}
+
+static int ce4100_conf_read(unsigned int seg, unsigned int bus,
+ unsigned int devfn, int reg, int len, u32 *value)
+{
+ WARN_ON(seg);
+
+ if (bus == 1 && !ce4100_bus1_read(devfn, reg, len, value))
+ return 0;
if (bus == 0 && (PCI_DEVFN(1, 0) == devfn) &&
!bridge_read(devfn, reg, len, value))
return pci_direct_conf1.read(seg, bus, devfn, reg, len, value);
}
-static int ce4100_conf_write(unsigned int seg, unsigned int bus,
- unsigned int devfn, int reg, int len, u32 value)
+static int ce4100_bus1_write(unsigned int devfn, int reg, int len, u32 value)
{
+ unsigned long flags;
int i;
- WARN_ON(seg);
- if (bus == 1) {
- for (i = 0; i < ARRAY_SIZE(bus1_fixups); i++) {
- if (bus1_fixups[i].dev_func == devfn &&
- bus1_fixups[i].reg == (reg & ~3) &&
- bus1_fixups[i].write) {
- bus1_fixups[i].write(&(bus1_fixups[i]),
- value);
- return 0;
- }
+ for (i = 0; i < ARRAY_SIZE(bus1_fixups); i++) {
+ if (bus1_fixups[i].dev_func == devfn &&
+ bus1_fixups[i].reg == (reg & ~3) &&
+ bus1_fixups[i].write) {
+
+ raw_spin_lock_irqsave(&pci_config_lock, flags);
+ bus1_fixups[i].write(&(bus1_fixups[i]), value);
+ raw_spin_unlock_irqrestore(&pci_config_lock, flags);
+ return 0;
}
}
+ return -1;
+}
+
+static int ce4100_conf_write(unsigned int seg, unsigned int bus,
+ unsigned int devfn, int reg, int len, u32 value)
+{
+ WARN_ON(seg);
+
+ if (bus == 1 && !ce4100_bus1_write(devfn, reg, len, value))
+ return 0;
/* Discard writes to A/V bridge BAR. */
if (bus == 0 && PCI_DEVFN(1, 0) == devfn &&
}
static const struct pci_raw_ops ce4100_pci_conf = {
- .read = ce4100_conf_read,
- .write = ce4100_conf_write,
+ .read = ce4100_conf_read,
+ .write = ce4100_conf_write,
};
int __init ce4100_pci_init(void)
};
/*
- * This interrupt-safe spinlock protects all accesses to PCI
- * configuration space.
+ * This interrupt-safe spinlock protects all accesses to PCI configuration
+ * space, except for the mmconfig (ECAM) based operations.
*/
DEFINE_RAW_SPINLOCK(pci_config_lock);
int __init pci_legacy_init(void)
{
- if (!raw_pci_ops) {
- printk("PCI: System does not support PCI\n");
- return 0;
- }
+ if (!raw_pci_ops)
+ return 1;
- printk("PCI: Probing PCI hardware\n");
+ pr_info("PCI: Probing PCI hardware\n");
pcibios_scan_root(0);
return 0;
}
if (!raw_pci_read(0, busn, devfn, PCI_VENDOR_ID, 2, &l) &&
l != 0x0000 && l != 0xffff) {
DBG("Found device at %02x:%02x [%04x]\n", busn, devfn, l);
- printk(KERN_INFO "PCI: Discovered peer bus %02x\n", busn);
+ pr_info("PCI: Discovered peer bus %02x\n", busn);
pcibios_scan_root(busn);
return;
}
* The init function returns an non zero value when
* pci_legacy_init should be invoked.
*/
- if (x86_init.pci.init())
- pci_legacy_init();
+ if (x86_init.pci.init()) {
+ if (pci_legacy_init()) {
+ pr_info("PCI: System does not support PCI\n");
+ return -ENODEV;
+ }
+ }
pcibios_fixup_peer_bridges();
x86_init.pci.init_irq();
OBJECT_FILES_NON_STANDARD_efi_thunk_$(BITS).o := y
+OBJECT_FILES_NON_STANDARD_efi_stub_$(BITS).o := y
obj-$(CONFIG_EFI) += quirks.o efi.o efi_$(BITS).o efi_stub_$(BITS).o
obj-$(CONFIG_EARLY_PRINTK_EFI) += early_printk.o
* necessary relocation fixups for the new virtual addresses.
*/
efi_runtime_update_mappings();
- efi_dump_pagetable();
/* clean DUMMY object */
efi_delete_dummy_variable();
kexec_enter_virtual_mode();
else
__efi_enter_virtual_mode();
+
+ efi_dump_pagetable();
}
/*
}
void efi_sync_low_kernel_mappings(void) {}
-void __init efi_dump_pagetable(void) {}
+
+void __init efi_dump_pagetable(void)
+{
+#ifdef CONFIG_EFI_PGT_DUMP
+ ptdump_walk_pgd_level(NULL, swapper_pg_dir);
+#endif
+}
+
int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
{
return 0;
int n_pgds, i, j;
if (!efi_enabled(EFI_OLD_MEMMAP)) {
- save_pgd = (pgd_t *)read_cr3();
+ save_pgd = (pgd_t *)__read_cr3();
write_cr3((unsigned long)efi_scratch.efi_pgt);
goto out;
}
void __init efi_dump_pagetable(void)
{
#ifdef CONFIG_EFI_PGT_DUMP
- ptdump_walk_pgd_level(NULL, efi_pgd);
+ if (efi_enabled(EFI_OLD_MEMMAP))
+ ptdump_walk_pgd_level(NULL, swapper_pg_dir);
+ else
+ ptdump_walk_pgd_level(NULL, efi_pgd);
#endif
}
efi_sync_low_kernel_mappings();
local_irq_save(flags);
- efi_scratch.prev_cr3 = read_cr3();
+ efi_scratch.prev_cr3 = __read_cr3();
write_cr3((unsigned long)efi_scratch.efi_pgt);
__flush_tlb_all();
#include <asm/e820/api.h>
#include <asm/efi.h>
#include <asm/uv/uv.h>
+#include <asm/cpu_device_id.h>
#define EFI_MIN_RESERVE 5120
#define EFI_DUMMY_GUID \
EFI_GUID(0x4424ac57, 0xbe4b, 0x47dd, 0x9e, 0x97, 0xed, 0x50, 0xf0, 0x9f, 0x92, 0xa9)
+#define QUARK_CSH_SIGNATURE 0x5f435348 /* _CSH */
+#define QUARK_SECURITY_HEADER_SIZE 0x400
+
+/*
+ * Header prepended to the standard EFI capsule on Quark systems the are based
+ * on Intel firmware BSP.
+ * @csh_signature: Unique identifier to sanity check signed module
+ * presence ("_CSH").
+ * @version: Current version of CSH used. Should be one for Quark A0.
+ * @modulesize: Size of the entire module including the module header
+ * and payload.
+ * @security_version_number_index: Index of SVN to use for validation of signed
+ * module.
+ * @security_version_number: Used to prevent against roll back of modules.
+ * @rsvd_module_id: Currently unused for Clanton (Quark).
+ * @rsvd_module_vendor: Vendor Identifier. For Intel products value is
+ * 0x00008086.
+ * @rsvd_date: BCD representation of build date as yyyymmdd, where
+ * yyyy=4 digit year, mm=1-12, dd=1-31.
+ * @headersize: Total length of the header including including any
+ * padding optionally added by the signing tool.
+ * @hash_algo: What Hash is used in the module signing.
+ * @cryp_algo: What Crypto is used in the module signing.
+ * @keysize: Total length of the key data including including any
+ * padding optionally added by the signing tool.
+ * @signaturesize: Total length of the signature including including any
+ * padding optionally added by the signing tool.
+ * @rsvd_next_header: 32-bit pointer to the next Secure Boot Module in the
+ * chain, if there is a next header.
+ * @rsvd: Reserved, padding structure to required size.
+ *
+ * See also QuartSecurityHeader_t in
+ * Quark_EDKII_v1.2.1.1/QuarkPlatformPkg/Include/QuarkBootRom.h
+ * from https://downloadcenter.intel.com/download/23197/Intel-Quark-SoC-X1000-Board-Support-Package-BSP
+ */
+struct quark_security_header {
+ u32 csh_signature;
+ u32 version;
+ u32 modulesize;
+ u32 security_version_number_index;
+ u32 security_version_number;
+ u32 rsvd_module_id;
+ u32 rsvd_module_vendor;
+ u32 rsvd_date;
+ u32 headersize;
+ u32 hash_algo;
+ u32 cryp_algo;
+ u32 keysize;
+ u32 signaturesize;
+ u32 rsvd_next_header;
+ u32 rsvd[2];
+};
+
static efi_char16_t efi_dummy_name[6] = { 'D', 'U', 'M', 'M', 'Y', 0 };
static bool efi_no_storage_paranoia;
{
return acpi_gbl_reduced_hardware || acpi_no_s5;
}
+
+#ifdef CONFIG_EFI_CAPSULE_QUIRK_QUARK_CSH
+
+static int qrk_capsule_setup_info(struct capsule_info *cap_info, void **pkbuff,
+ size_t hdr_bytes)
+{
+ struct quark_security_header *csh = *pkbuff;
+
+ /* Only process data block that is larger than the security header */
+ if (hdr_bytes < sizeof(struct quark_security_header))
+ return 0;
+
+ if (csh->csh_signature != QUARK_CSH_SIGNATURE ||
+ csh->headersize != QUARK_SECURITY_HEADER_SIZE)
+ return 1;
+
+ /* Only process data block if EFI header is included */
+ if (hdr_bytes < QUARK_SECURITY_HEADER_SIZE +
+ sizeof(efi_capsule_header_t))
+ return 0;
+
+ pr_debug("Quark security header detected\n");
+
+ if (csh->rsvd_next_header != 0) {
+ pr_err("multiple Quark security headers not supported\n");
+ return -EINVAL;
+ }
+
+ *pkbuff += csh->headersize;
+ cap_info->total_size = csh->headersize;
+
+ /*
+ * Update the first page pointer to skip over the CSH header.
+ */
+ cap_info->pages[0] += csh->headersize;
+
+ return 1;
+}
+
+#define ICPU(family, model, quirk_handler) \
+ { X86_VENDOR_INTEL, family, model, X86_FEATURE_ANY, \
+ (unsigned long)&quirk_handler }
+
+static const struct x86_cpu_id efi_capsule_quirk_ids[] = {
+ ICPU(5, 9, qrk_capsule_setup_info), /* Intel Quark X1000 */
+ { }
+};
+
+int efi_capsule_setup_info(struct capsule_info *cap_info, void *kbuff,
+ size_t hdr_bytes)
+{
+ int (*quirk_handler)(struct capsule_info *, void **, size_t);
+ const struct x86_cpu_id *id;
+ int ret;
+
+ if (hdr_bytes < sizeof(efi_capsule_header_t))
+ return 0;
+
+ cap_info->total_size = 0;
+
+ id = x86_match_cpu(efi_capsule_quirk_ids);
+ if (id) {
+ /*
+ * The quirk handler is supposed to return
+ * - a value > 0 if the setup should continue, after advancing
+ * kbuff as needed
+ * - 0 if not enough hdr_bytes are available yet
+ * - a negative error code otherwise
+ */
+ quirk_handler = (typeof(quirk_handler))id->driver_data;
+ ret = quirk_handler(cap_info, &kbuff, hdr_bytes);
+ if (ret <= 0)
+ return ret;
+ }
+
+ memcpy(&cap_info->header, kbuff, sizeof(cap_info->header));
+
+ cap_info->total_size += cap_info->header.imagesize;
+
+ return __efi_capsule_setup_info(cap_info);
+}
+
+#endif
asmlinkage __visible int xo1_do_sleep(u8 sleep_state)
{
- void *pgd_addr = __va(read_cr3());
+ void *pgd_addr = __va(read_cr3_pa());
/* Program wakeup mask (using dword access to CS5536_PM1_EN) */
outl(wakeup_mask << 16, acpi_base + CS5536_PM1_STS);
*/
static inline unsigned long long cycles_2_ns(unsigned long long cyc)
{
- struct cyc2ns_data *data = cyc2ns_read_begin();
+ struct cyc2ns_data data;
unsigned long long ns;
- ns = mul_u64_u32_shr(cyc, data->cyc2ns_mul, data->cyc2ns_shift);
+ cyc2ns_read_begin(&data);
+ ns = mul_u64_u32_shr(cyc, data.cyc2ns_mul, data.cyc2ns_shift);
+ cyc2ns_read_end();
- cyc2ns_read_end(data);
return ns;
}
*/
static inline unsigned long long ns_2_cycles(unsigned long long ns)
{
- struct cyc2ns_data *data = cyc2ns_read_begin();
+ struct cyc2ns_data data;
unsigned long long cyc;
- cyc = (ns << data->cyc2ns_shift) / data->cyc2ns_mul;
+ cyc2ns_read_begin(&data);
+ cyc = (ns << data.cyc2ns_shift) / data.cyc2ns_mul;
+ cyc2ns_read_end();
- cyc2ns_read_end(data);
return cyc;
}
* done. The returned pointer is valid till preemption is re-enabled.
*/
const struct cpumask *uv_flush_tlb_others(const struct cpumask *cpumask,
- struct mm_struct *mm,
- unsigned long start,
- unsigned long end,
- unsigned int cpu)
+ const struct flush_tlb_info *info)
{
+ unsigned int cpu = smp_processor_id();
int locals = 0, remotes = 0, hubs = 0;
struct bau_desc *bau_desc;
struct cpumask *flush_mask;
record_send_statistics(stat, locals, hubs, remotes, bau_desc);
- if (!end || (end - start) <= PAGE_SIZE)
- address = start;
+ if (!info->end || (info->end - info->start) <= PAGE_SIZE)
+ address = info->start;
else
address = TLB_FLUSH_ALL;
{
static struct irq_domain *uv_domain;
static DEFINE_MUTEX(uv_lock);
+ struct fwnode_handle *fn;
mutex_lock(&uv_lock);
- if (uv_domain == NULL) {
- uv_domain = irq_domain_add_tree(NULL, &uv_domain_ops, NULL);
- if (uv_domain)
- uv_domain->parent = x86_vector_domain;
- }
+ if (uv_domain)
+ goto out;
+
+ fn = irq_domain_alloc_named_fwnode("UV-CORE");
+ if (!fn)
+ goto out;
+
+ uv_domain = irq_domain_create_tree(fn, &uv_domain_ops, NULL);
+ irq_domain_free_fwnode(fn);
+ if (uv_domain)
+ uv_domain->parent = x86_vector_domain;
+out:
mutex_unlock(&uv_lock);
return uv_domain;
+OBJECT_FILES_NON_STANDARD_hibernate_asm_$(BITS).o := y
+
# __restore_processor_state() restores %gs after S3 resume and so should not
# itself be stack-protected
nostackp := $(call cc-option, -fno-stack-protector)
*/
ctxt->cr0 = read_cr0();
ctxt->cr2 = read_cr2();
- ctxt->cr3 = read_cr3();
+ ctxt->cr3 = __read_cr3();
ctxt->cr4 = __read_cr4();
#ifdef CONFIG_X86_64
ctxt->cr8 = read_cr8();
if (!relocated_restore_code)
return -ENOMEM;
- memcpy((void *)relocated_restore_code, &core_restore_code, PAGE_SIZE);
+ memcpy((void *)relocated_restore_code, core_restore_code, PAGE_SIZE);
/* Make the page containing the relocated code executable */
- pgd = (pgd_t *)__va(read_cr3()) + pgd_index(relocated_restore_code);
+ pgd = (pgd_t *)__va(read_cr3_pa()) +
+ pgd_index(relocated_restore_code);
p4d = p4d_offset(pgd, relocated_restore_code);
if (p4d_large(*p4d)) {
set_p4d(p4d, __p4d(p4d_val(*p4d) & ~_PAGE_NX));
if (max_size < sizeof(struct restore_data_record))
return -EOVERFLOW;
- rdr->jump_address = (unsigned long)&restore_registers;
- rdr->jump_address_phys = __pa_symbol(&restore_registers);
+ rdr->jump_address = (unsigned long)restore_registers;
+ rdr->jump_address_phys = __pa_symbol(restore_registers);
rdr->cr3 = restore_cr3;
rdr->magic = RESTORE_MAGIC;
-config MCE_AMD_INJ
- tristate "Simple MCE injection interface for AMD processors"
- depends on RAS && X86_MCE && DEBUG_FS && AMD_NB
- default n
- help
- This is a simple debugfs interface to inject MCEs and test different
- aspects of the MCE handling code.
-
- WARNING: Do not even assume this interface is staying stable!
-
config RAS_CEC
bool "Correctable Errors Collector"
depends on X86_MCE && MEMORY_FAILURE && DEBUG_FS
Bear in mind that this is absolutely useless if your platform doesn't
have ECC DIMMs and doesn't have DRAM ECC checking enabled in the BIOS.
-
+++ /dev/null
-obj-$(CONFIG_MCE_AMD_INJ) += mce_amd_inj.o
-
+++ /dev/null
-/*
- * A simple MCE injection facility for testing different aspects of the RAS
- * code. This driver should be built as module so that it can be loaded
- * on production kernels for testing purposes.
- *
- * This file may be distributed under the terms of the GNU General Public
- * License version 2.
- *
- * Copyright (c) 2010-15: Borislav Petkov <bp@alien8.de>
- * Advanced Micro Devices Inc.
- */
-
-#include <linux/kobject.h>
-#include <linux/debugfs.h>
-#include <linux/device.h>
-#include <linux/module.h>
-#include <linux/cpu.h>
-#include <linux/string.h>
-#include <linux/uaccess.h>
-#include <linux/pci.h>
-
-#include <asm/mce.h>
-#include <asm/smp.h>
-#include <asm/amd_nb.h>
-#include <asm/irq_vectors.h>
-
-#include "../kernel/cpu/mcheck/mce-internal.h"
-
-/*
- * Collect all the MCi_XXX settings
- */
-static struct mce i_mce;
-static struct dentry *dfs_inj;
-
-static u8 n_banks;
-
-#define MAX_FLAG_OPT_SIZE 3
-#define NBCFG 0x44
-
-enum injection_type {
- SW_INJ = 0, /* SW injection, simply decode the error */
- HW_INJ, /* Trigger a #MC */
- DFR_INT_INJ, /* Trigger Deferred error interrupt */
- THR_INT_INJ, /* Trigger threshold interrupt */
- N_INJ_TYPES,
-};
-
-static const char * const flags_options[] = {
- [SW_INJ] = "sw",
- [HW_INJ] = "hw",
- [DFR_INT_INJ] = "df",
- [THR_INT_INJ] = "th",
- NULL
-};
-
-/* Set default injection to SW_INJ */
-static enum injection_type inj_type = SW_INJ;
-
-#define MCE_INJECT_SET(reg) \
-static int inj_##reg##_set(void *data, u64 val) \
-{ \
- struct mce *m = (struct mce *)data; \
- \
- m->reg = val; \
- return 0; \
-}
-
-MCE_INJECT_SET(status);
-MCE_INJECT_SET(misc);
-MCE_INJECT_SET(addr);
-MCE_INJECT_SET(synd);
-
-#define MCE_INJECT_GET(reg) \
-static int inj_##reg##_get(void *data, u64 *val) \
-{ \
- struct mce *m = (struct mce *)data; \
- \
- *val = m->reg; \
- return 0; \
-}
-
-MCE_INJECT_GET(status);
-MCE_INJECT_GET(misc);
-MCE_INJECT_GET(addr);
-MCE_INJECT_GET(synd);
-
-DEFINE_SIMPLE_ATTRIBUTE(status_fops, inj_status_get, inj_status_set, "%llx\n");
-DEFINE_SIMPLE_ATTRIBUTE(misc_fops, inj_misc_get, inj_misc_set, "%llx\n");
-DEFINE_SIMPLE_ATTRIBUTE(addr_fops, inj_addr_get, inj_addr_set, "%llx\n");
-DEFINE_SIMPLE_ATTRIBUTE(synd_fops, inj_synd_get, inj_synd_set, "%llx\n");
-
-/*
- * Caller needs to be make sure this cpu doesn't disappear
- * from under us, i.e.: get_cpu/put_cpu.
- */
-static int toggle_hw_mce_inject(unsigned int cpu, bool enable)
-{
- u32 l, h;
- int err;
-
- err = rdmsr_on_cpu(cpu, MSR_K7_HWCR, &l, &h);
- if (err) {
- pr_err("%s: error reading HWCR\n", __func__);
- return err;
- }
-
- enable ? (l |= BIT(18)) : (l &= ~BIT(18));
-
- err = wrmsr_on_cpu(cpu, MSR_K7_HWCR, l, h);
- if (err)
- pr_err("%s: error writing HWCR\n", __func__);
-
- return err;
-}
-
-static int __set_inj(const char *buf)
-{
- int i;
-
- for (i = 0; i < N_INJ_TYPES; i++) {
- if (!strncmp(flags_options[i], buf, strlen(flags_options[i]))) {
- inj_type = i;
- return 0;
- }
- }
- return -EINVAL;
-}
-
-static ssize_t flags_read(struct file *filp, char __user *ubuf,
- size_t cnt, loff_t *ppos)
-{
- char buf[MAX_FLAG_OPT_SIZE];
- int n;
-
- n = sprintf(buf, "%s\n", flags_options[inj_type]);
-
- return simple_read_from_buffer(ubuf, cnt, ppos, buf, n);
-}
-
-static ssize_t flags_write(struct file *filp, const char __user *ubuf,
- size_t cnt, loff_t *ppos)
-{
- char buf[MAX_FLAG_OPT_SIZE], *__buf;
- int err;
-
- if (cnt > MAX_FLAG_OPT_SIZE)
- return -EINVAL;
-
- if (copy_from_user(&buf, ubuf, cnt))
- return -EFAULT;
-
- buf[cnt - 1] = 0;
-
- /* strip whitespace */
- __buf = strstrip(buf);
-
- err = __set_inj(__buf);
- if (err) {
- pr_err("%s: Invalid flags value: %s\n", __func__, __buf);
- return err;
- }
-
- *ppos += cnt;
-
- return cnt;
-}
-
-static const struct file_operations flags_fops = {
- .read = flags_read,
- .write = flags_write,
- .llseek = generic_file_llseek,
-};
-
-/*
- * On which CPU to inject?
- */
-MCE_INJECT_GET(extcpu);
-
-static int inj_extcpu_set(void *data, u64 val)
-{
- struct mce *m = (struct mce *)data;
-
- if (val >= nr_cpu_ids || !cpu_online(val)) {
- pr_err("%s: Invalid CPU: %llu\n", __func__, val);
- return -EINVAL;
- }
- m->extcpu = val;
- return 0;
-}
-
-DEFINE_SIMPLE_ATTRIBUTE(extcpu_fops, inj_extcpu_get, inj_extcpu_set, "%llu\n");
-
-static void trigger_mce(void *info)
-{
- asm volatile("int $18");
-}
-
-static void trigger_dfr_int(void *info)
-{
- asm volatile("int %0" :: "i" (DEFERRED_ERROR_VECTOR));
-}
-
-static void trigger_thr_int(void *info)
-{
- asm volatile("int %0" :: "i" (THRESHOLD_APIC_VECTOR));
-}
-
-static u32 get_nbc_for_node(int node_id)
-{
- struct cpuinfo_x86 *c = &boot_cpu_data;
- u32 cores_per_node;
-
- cores_per_node = (c->x86_max_cores * smp_num_siblings) / amd_get_nodes_per_socket();
-
- return cores_per_node * node_id;
-}
-
-static void toggle_nb_mca_mst_cpu(u16 nid)
-{
- struct pci_dev *F3 = node_to_amd_nb(nid)->misc;
- u32 val;
- int err;
-
- if (!F3)
- return;
-
- err = pci_read_config_dword(F3, NBCFG, &val);
- if (err) {
- pr_err("%s: Error reading F%dx%03x.\n",
- __func__, PCI_FUNC(F3->devfn), NBCFG);
- return;
- }
-
- if (val & BIT(27))
- return;
-
- pr_err("%s: Set D18F3x44[NbMcaToMstCpuEn] which BIOS hasn't done.\n",
- __func__);
-
- val |= BIT(27);
- err = pci_write_config_dword(F3, NBCFG, val);
- if (err)
- pr_err("%s: Error writing F%dx%03x.\n",
- __func__, PCI_FUNC(F3->devfn), NBCFG);
-}
-
-static void prepare_msrs(void *info)
-{
- struct mce m = *(struct mce *)info;
- u8 b = m.bank;
-
- wrmsrl(MSR_IA32_MCG_STATUS, m.mcgstatus);
-
- if (boot_cpu_has(X86_FEATURE_SMCA)) {
- if (m.inject_flags == DFR_INT_INJ) {
- wrmsrl(MSR_AMD64_SMCA_MCx_DESTAT(b), m.status);
- wrmsrl(MSR_AMD64_SMCA_MCx_DEADDR(b), m.addr);
- } else {
- wrmsrl(MSR_AMD64_SMCA_MCx_STATUS(b), m.status);
- wrmsrl(MSR_AMD64_SMCA_MCx_ADDR(b), m.addr);
- }
-
- wrmsrl(MSR_AMD64_SMCA_MCx_MISC(b), m.misc);
- wrmsrl(MSR_AMD64_SMCA_MCx_SYND(b), m.synd);
- } else {
- wrmsrl(MSR_IA32_MCx_STATUS(b), m.status);
- wrmsrl(MSR_IA32_MCx_ADDR(b), m.addr);
- wrmsrl(MSR_IA32_MCx_MISC(b), m.misc);
- }
-}
-
-static void do_inject(void)
-{
- u64 mcg_status = 0;
- unsigned int cpu = i_mce.extcpu;
- u8 b = i_mce.bank;
-
- rdtscll(i_mce.tsc);
-
- if (i_mce.misc)
- i_mce.status |= MCI_STATUS_MISCV;
-
- if (i_mce.synd)
- i_mce.status |= MCI_STATUS_SYNDV;
-
- if (inj_type == SW_INJ) {
- mce_inject_log(&i_mce);
- return;
- }
-
- /* prep MCE global settings for the injection */
- mcg_status = MCG_STATUS_MCIP | MCG_STATUS_EIPV;
-
- if (!(i_mce.status & MCI_STATUS_PCC))
- mcg_status |= MCG_STATUS_RIPV;
-
- /*
- * Ensure necessary status bits for deferred errors:
- * - MCx_STATUS[Deferred]: make sure it is a deferred error
- * - MCx_STATUS[UC] cleared: deferred errors are _not_ UC
- */
- if (inj_type == DFR_INT_INJ) {
- i_mce.status |= MCI_STATUS_DEFERRED;
- i_mce.status |= (i_mce.status & ~MCI_STATUS_UC);
- }
-
- /*
- * For multi node CPUs, logging and reporting of bank 4 errors happens
- * only on the node base core. Refer to D18F3x44[NbMcaToMstCpuEn] for
- * Fam10h and later BKDGs.
- */
- if (static_cpu_has(X86_FEATURE_AMD_DCM) &&
- b == 4 &&
- boot_cpu_data.x86 < 0x17) {
- toggle_nb_mca_mst_cpu(amd_get_nb_id(cpu));
- cpu = get_nbc_for_node(amd_get_nb_id(cpu));
- }
-
- get_online_cpus();
- if (!cpu_online(cpu))
- goto err;
-
- toggle_hw_mce_inject(cpu, true);
-
- i_mce.mcgstatus = mcg_status;
- i_mce.inject_flags = inj_type;
- smp_call_function_single(cpu, prepare_msrs, &i_mce, 0);
-
- toggle_hw_mce_inject(cpu, false);
-
- switch (inj_type) {
- case DFR_INT_INJ:
- smp_call_function_single(cpu, trigger_dfr_int, NULL, 0);
- break;
- case THR_INT_INJ:
- smp_call_function_single(cpu, trigger_thr_int, NULL, 0);
- break;
- default:
- smp_call_function_single(cpu, trigger_mce, NULL, 0);
- }
-
-err:
- put_online_cpus();
-
-}
-
-/*
- * This denotes into which bank we're injecting and triggers
- * the injection, at the same time.
- */
-static int inj_bank_set(void *data, u64 val)
-{
- struct mce *m = (struct mce *)data;
-
- if (val >= n_banks) {
- pr_err("Non-existent MCE bank: %llu\n", val);
- return -EINVAL;
- }
-
- m->bank = val;
- do_inject();
-
- return 0;
-}
-
-MCE_INJECT_GET(bank);
-
-DEFINE_SIMPLE_ATTRIBUTE(bank_fops, inj_bank_get, inj_bank_set, "%llu\n");
-
-static const char readme_msg[] =
-"Description of the files and their usages:\n"
-"\n"
-"Note1: i refers to the bank number below.\n"
-"Note2: See respective BKDGs for the exact bit definitions of the files below\n"
-"as they mirror the hardware registers.\n"
-"\n"
-"status:\t Set MCi_STATUS: the bits in that MSR control the error type and\n"
-"\t attributes of the error which caused the MCE.\n"
-"\n"
-"misc:\t Set MCi_MISC: provide auxiliary info about the error. It is mostly\n"
-"\t used for error thresholding purposes and its validity is indicated by\n"
-"\t MCi_STATUS[MiscV].\n"
-"\n"
-"synd:\t Set MCi_SYND: provide syndrome info about the error. Only valid on\n"
-"\t Scalable MCA systems, and its validity is indicated by MCi_STATUS[SyndV].\n"
-"\n"
-"addr:\t Error address value to be written to MCi_ADDR. Log address information\n"
-"\t associated with the error.\n"
-"\n"
-"cpu:\t The CPU to inject the error on.\n"
-"\n"
-"bank:\t Specify the bank you want to inject the error into: the number of\n"
-"\t banks in a processor varies and is family/model-specific, therefore, the\n"
-"\t supplied value is sanity-checked. Setting the bank value also triggers the\n"
-"\t injection.\n"
-"\n"
-"flags:\t Injection type to be performed. Writing to this file will trigger a\n"
-"\t real machine check, an APIC interrupt or invoke the error decoder routines\n"
-"\t for AMD processors.\n"
-"\n"
-"\t Allowed error injection types:\n"
-"\t - \"sw\": Software error injection. Decode error to a human-readable \n"
-"\t format only. Safe to use.\n"
-"\t - \"hw\": Hardware error injection. Causes the #MC exception handler to \n"
-"\t handle the error. Be warned: might cause system panic if MCi_STATUS[PCC] \n"
-"\t is set. Therefore, consider setting (debugfs_mountpoint)/mce/fake_panic \n"
-"\t before injecting.\n"
-"\t - \"df\": Trigger APIC interrupt for Deferred error. Causes deferred \n"
-"\t error APIC interrupt handler to handle the error if the feature is \n"
-"\t is present in hardware. \n"
-"\t - \"th\": Trigger APIC interrupt for Threshold errors. Causes threshold \n"
-"\t APIC interrupt handler to handle the error. \n"
-"\n";
-
-static ssize_t
-inj_readme_read(struct file *filp, char __user *ubuf,
- size_t cnt, loff_t *ppos)
-{
- return simple_read_from_buffer(ubuf, cnt, ppos,
- readme_msg, strlen(readme_msg));
-}
-
-static const struct file_operations readme_fops = {
- .read = inj_readme_read,
-};
-
-static struct dfs_node {
- char *name;
- struct dentry *d;
- const struct file_operations *fops;
- umode_t perm;
-} dfs_fls[] = {
- { .name = "status", .fops = &status_fops, .perm = S_IRUSR | S_IWUSR },
- { .name = "misc", .fops = &misc_fops, .perm = S_IRUSR | S_IWUSR },
- { .name = "addr", .fops = &addr_fops, .perm = S_IRUSR | S_IWUSR },
- { .name = "synd", .fops = &synd_fops, .perm = S_IRUSR | S_IWUSR },
- { .name = "bank", .fops = &bank_fops, .perm = S_IRUSR | S_IWUSR },
- { .name = "flags", .fops = &flags_fops, .perm = S_IRUSR | S_IWUSR },
- { .name = "cpu", .fops = &extcpu_fops, .perm = S_IRUSR | S_IWUSR },
- { .name = "README", .fops = &readme_fops, .perm = S_IRUSR | S_IRGRP | S_IROTH },
-};
-
-static int __init init_mce_inject(void)
-{
- unsigned int i;
- u64 cap;
-
- rdmsrl(MSR_IA32_MCG_CAP, cap);
- n_banks = cap & MCG_BANKCNT_MASK;
-
- dfs_inj = debugfs_create_dir("mce-inject", NULL);
- if (!dfs_inj)
- return -EINVAL;
-
- for (i = 0; i < ARRAY_SIZE(dfs_fls); i++) {
- dfs_fls[i].d = debugfs_create_file(dfs_fls[i].name,
- dfs_fls[i].perm,
- dfs_inj,
- &i_mce,
- dfs_fls[i].fops);
-
- if (!dfs_fls[i].d)
- goto err_dfs_add;
- }
-
- return 0;
-
-err_dfs_add:
- while (i-- > 0)
- debugfs_remove(dfs_fls[i].d);
-
- debugfs_remove(dfs_inj);
- dfs_inj = NULL;
-
- return -ENODEV;
-}
-
-static void __exit exit_mce_inject(void)
-{
-
- debugfs_remove_recursive(dfs_inj);
- dfs_inj = NULL;
-
- memset(&dfs_fls, 0, sizeof(dfs_fls));
-}
-module_init(init_mce_inject);
-module_exit(exit_mce_inject);
-
-MODULE_LICENSE("GPL");
-MODULE_AUTHOR("Borislav Petkov <bp@alien8.de>");
-MODULE_AUTHOR("AMD Inc.");
-MODULE_DESCRIPTION("MCE injection facility for RAS testing");
trampoline_pgd = (u64 *) __va(real_mode_header->trampoline_pgd);
trampoline_pgd[0] = trampoline_pgd_entry.pgd;
- trampoline_pgd[511] = init_level4_pgt[511].pgd;
+ trampoline_pgd[511] = init_top_pgt[511].pgd;
#endif
}
+OBJECT_FILES_NON_STANDARD_xen-asm_$(BITS).o := y
+OBJECT_FILES_NON_STANDARD_xen-pvh.o := y
+
ifdef CONFIG_FUNCTION_TRACER
# Do not profile debug and lowlevel utilities
CFLAGS_REMOVE_spinlock.o = -pg
.get_apic_id = xen_get_apic_id,
.set_apic_id = xen_set_apic_id, /* Can be NULL on 32-bit. */
- .cpu_mask_to_apicid_and = flat_cpu_mask_to_apicid_and,
+ .cpu_mask_to_apicid = flat_cpu_mask_to_apicid,
#ifdef CONFIG_SMP
.send_IPI_mask = xen_send_IPI_mask,
.tables = EFI_INVALID_TABLE_ADDR /* Initialized later. */
};
-static const struct efi efi_xen __initconst = {
- .systab = NULL, /* Initialized later. */
- .runtime_version = 0, /* Initialized later. */
- .mps = EFI_INVALID_TABLE_ADDR,
- .acpi = EFI_INVALID_TABLE_ADDR,
- .acpi20 = EFI_INVALID_TABLE_ADDR,
- .smbios = EFI_INVALID_TABLE_ADDR,
- .smbios3 = EFI_INVALID_TABLE_ADDR,
- .sal_systab = EFI_INVALID_TABLE_ADDR,
- .boot_info = EFI_INVALID_TABLE_ADDR,
- .hcdp = EFI_INVALID_TABLE_ADDR,
- .uga = EFI_INVALID_TABLE_ADDR,
- .uv_systab = EFI_INVALID_TABLE_ADDR,
- .fw_vendor = EFI_INVALID_TABLE_ADDR,
- .runtime = EFI_INVALID_TABLE_ADDR,
- .config_table = EFI_INVALID_TABLE_ADDR,
- .get_time = xen_efi_get_time,
- .set_time = xen_efi_set_time,
- .get_wakeup_time = xen_efi_get_wakeup_time,
- .set_wakeup_time = xen_efi_set_wakeup_time,
- .get_variable = xen_efi_get_variable,
- .get_next_variable = xen_efi_get_next_variable,
- .set_variable = xen_efi_set_variable,
- .query_variable_info = xen_efi_query_variable_info,
- .update_capsule = xen_efi_update_capsule,
- .query_capsule_caps = xen_efi_query_capsule_caps,
- .get_next_high_mono_count = xen_efi_get_next_high_mono_count,
- .reset_system = xen_efi_reset_system,
- .set_virtual_address_map = NULL, /* Not used under Xen. */
- .flags = 0 /* Initialized later. */
-};
-
static efi_system_table_t __init *xen_efi_probe(void)
{
struct xen_platform_op op = {
/* Here we know that Xen runs on EFI platform. */
- efi = efi_xen;
+ efi.get_time = xen_efi_get_time;
+ efi.set_time = xen_efi_set_time;
+ efi.get_wakeup_time = xen_efi_get_wakeup_time;
+ efi.set_wakeup_time = xen_efi_set_wakeup_time;
+ efi.get_variable = xen_efi_get_variable;
+ efi.get_next_variable = xen_efi_get_next_variable;
+ efi.set_variable = xen_efi_set_variable;
+ efi.query_variable_info = xen_efi_query_variable_info;
+ efi.update_capsule = xen_efi_update_capsule;
+ efi.query_capsule_caps = xen_efi_query_capsule_caps;
+ efi.get_next_high_mono_count = xen_efi_get_next_high_mono_count;
+ efi.reset_system = xen_efi_reset_system;
efi_systab_xen.tables = info->cfg.addr;
efi_systab_xen.nr_tables = info->cfg.nent;
spin_unlock(&mm->page_table_lock);
}
-
-#ifdef CONFIG_SMP
-/* Another cpu may still have their %cr3 pointing at the pagetable, so
- we need to repoint it somewhere else before we can unpin it. */
-static void drop_other_mm_ref(void *info)
+static void drop_mm_ref_this_cpu(void *info)
{
struct mm_struct *mm = info;
- struct mm_struct *active_mm;
-
- active_mm = this_cpu_read(cpu_tlbstate.active_mm);
- if (active_mm == mm && this_cpu_read(cpu_tlbstate.state) != TLBSTATE_OK)
+ if (this_cpu_read(cpu_tlbstate.loaded_mm) == mm)
leave_mm(smp_processor_id());
- /* If this cpu still has a stale cr3 reference, then make sure
- it has been flushed. */
+ /*
+ * If this cpu still has a stale cr3 reference, then make sure
+ * it has been flushed.
+ */
if (this_cpu_read(xen_current_cr3) == __pa(mm->pgd))
- load_cr3(swapper_pg_dir);
+ xen_mc_flush();
}
+#ifdef CONFIG_SMP
+/*
+ * Another cpu may still have their %cr3 pointing at the pagetable, so
+ * we need to repoint it somewhere else before we can unpin it.
+ */
static void xen_drop_mm_ref(struct mm_struct *mm)
{
cpumask_var_t mask;
unsigned cpu;
- if (current->active_mm == mm) {
- if (current->mm == mm)
- load_cr3(swapper_pg_dir);
- else
- leave_mm(smp_processor_id());
- }
+ drop_mm_ref_this_cpu(mm);
/* Get the "official" set of cpus referring to our pagetable. */
if (!alloc_cpumask_var(&mask, GFP_ATOMIC)) {
if (!cpumask_test_cpu(cpu, mm_cpumask(mm))
&& per_cpu(xen_current_cr3, cpu) != __pa(mm->pgd))
continue;
- smp_call_function_single(cpu, drop_other_mm_ref, mm, 1);
+ smp_call_function_single(cpu, drop_mm_ref_this_cpu, mm, 1);
}
return;
}
cpumask_copy(mask, mm_cpumask(mm));
- /* It's possible that a vcpu may have a stale reference to our
- cr3, because its in lazy mode, and it hasn't yet flushed
- its set of pending hypercalls yet. In this case, we can
- look at its actual current cr3 value, and force it to flush
- if needed. */
+ /*
+ * It's possible that a vcpu may have a stale reference to our
+ * cr3, because its in lazy mode, and it hasn't yet flushed
+ * its set of pending hypercalls yet. In this case, we can
+ * look at its actual current cr3 value, and force it to flush
+ * if needed.
+ */
for_each_online_cpu(cpu) {
if (per_cpu(xen_current_cr3, cpu) == __pa(mm->pgd))
cpumask_set_cpu(cpu, mask);
}
- if (!cpumask_empty(mask))
- smp_call_function_many(mask, drop_other_mm_ref, mm, 1);
+ smp_call_function_many(mask, drop_mm_ref_this_cpu, mm, 1);
free_cpumask_var(mask);
}
#else
static void xen_drop_mm_ref(struct mm_struct *mm)
{
- if (current->active_mm == mm)
- load_cr3(swapper_pg_dir);
+ drop_mm_ref_this_cpu(mm);
}
#endif
}
static void xen_flush_tlb_others(const struct cpumask *cpus,
- struct mm_struct *mm, unsigned long start,
- unsigned long end)
+ const struct flush_tlb_info *info)
{
struct {
struct mmuext_op op;
} *args;
struct multicall_space mcs;
- trace_xen_mmu_flush_tlb_others(cpus, mm, start, end);
+ trace_xen_mmu_flush_tlb_others(cpus, info->mm, info->start, info->end);
if (cpumask_empty(cpus))
return; /* nothing to do */
cpumask_clear_cpu(smp_processor_id(), to_cpumask(args->mask));
args->op.cmd = MMUEXT_TLB_FLUSH_MULTI;
- if (end != TLB_FLUSH_ALL && (end - start) <= PAGE_SIZE) {
+ if (info->end != TLB_FLUSH_ALL &&
+ (info->end - info->start) <= PAGE_SIZE) {
args->op.cmd = MMUEXT_INVLPG_MULTI;
- args->op.arg1.linear_addr = start;
+ args->op.arg1.linear_addr = info->start;
}
MULTI_mmuext_op(mcs.mc, &args->op, 1, NULL, DOMID_SELF);
* At the start of the day - when Xen launches a guest, it has already
* built pagetables for the guest. We diligently look over them
* in xen_setup_kernel_pagetable and graft as appropriate them in the
- * init_level4_pgt and its friends. Then when we are happy we load
- * the new init_level4_pgt - and continue on.
+ * init_top_pgt and its friends. Then when we are happy we load
+ * the new init_top_pgt - and continue on.
*
* The generic code starts (start_kernel) and 'init_mem_mapping' sets
* up the rest of the pagetables. When it has completed it loads the cr3.
pt_end = pt_base + xen_start_info->nr_pt_frames;
/* Zap identity mapping */
- init_level4_pgt[0] = __pgd(0);
+ init_top_pgt[0] = __pgd(0);
/* Pre-constructed entries are in pfn, so convert to mfn */
/* L4[272] -> level3_ident_pgt */
/* L4[511] -> level3_kernel_pgt */
- convert_pfn_mfn(init_level4_pgt);
+ convert_pfn_mfn(init_top_pgt);
/* L3_i[0] -> level2_ident_pgt */
convert_pfn_mfn(level3_ident_pgt);
/* Copy the initial P->M table mappings if necessary. */
i = pgd_index(xen_start_info->mfn_list);
if (i && i < pgd_index(__START_KERNEL_map))
- init_level4_pgt[i] = ((pgd_t *)xen_start_info->pt_base)[i];
+ init_top_pgt[i] = ((pgd_t *)xen_start_info->pt_base)[i];
/* Make pagetable pieces RO */
- set_page_prot(init_level4_pgt, PAGE_KERNEL_RO);
+ set_page_prot(init_top_pgt, PAGE_KERNEL_RO);
set_page_prot(level3_ident_pgt, PAGE_KERNEL_RO);
set_page_prot(level3_kernel_pgt, PAGE_KERNEL_RO);
set_page_prot(level3_user_vsyscall, PAGE_KERNEL_RO);
/* Pin down new L4 */
pin_pagetable_pfn(MMUEXT_PIN_L4_TABLE,
- PFN_DOWN(__pa_symbol(init_level4_pgt)));
+ PFN_DOWN(__pa_symbol(init_top_pgt)));
/* Unpin Xen-provided one */
pin_pagetable_pfn(MMUEXT_UNPIN_TABLE, PFN_DOWN(__pa(pgd)));
* attach it to, so make sure we just set kernel pgd.
*/
xen_mc_batch();
- __xen_write_cr3(true, __pa(init_level4_pgt));
+ __xen_write_cr3(true, __pa(init_top_pgt));
xen_mc_issue(PARAVIRT_LAZY_CPU);
/* We can't that easily rip out L3 and L2, as the Xen pagetables are
pmd_t pmd;
pte_t pte;
- pa = read_cr3();
+ pa = read_cr3_pa();
pgd = native_make_pgd(xen_read_phys_ulong(pa + pgd_index(vaddr) *
sizeof(pgd)));
if (!pgd_present(pgd))
pt_phys = pmd_phys + PFN_PHYS(n_pmd);
p2m_pfn = PFN_DOWN(pt_phys) + n_pt;
- pgd = __va(read_cr3());
+ pgd = __va(read_cr3_pa());
new_p2m = (unsigned long *)(2 * PGDIR_SIZE);
idx_p4d = 0;
save_pud = n_pud;
{
unsigned long pfn = PFN_DOWN(__pa(swapper_pg_dir));
- BUG_ON(read_cr3() != __pa(initial_page_table));
+ BUG_ON(read_cr3_pa() != __pa(initial_page_table));
BUG_ON(cr3 != __pa(swapper_pg_dir));
/*
wrmsr
/* Enable pre-constructed page tables. */
- mov $_pa(init_level4_pgt), %eax
+ mov $_pa(init_top_pgt), %eax
mov %eax, %cr3
mov $(X86_CR0_PG | X86_CR0_PE), %eax
mov %eax, %cr0
generic-y += resource.h
generic-y += rwsem.h
generic-y += sections.h
-generic-y += siginfo.h
generic-y += statfs.h
generic-y += termios.h
generic-y += topology.h
#define release_segments(mm) do { } while(0)
#define forget_segments() do { } while (0)
-#define thread_saved_pc(tsk) (task_pt_regs(tsk)->pc)
-
extern unsigned long get_wchan(struct task_struct *p);
#define KSTK_EIP(tsk) (task_pt_regs(tsk)->pc)
# UAPI Header export list
include include/uapi/asm-generic/Kbuild.asm
+generic-y += siginfo.h
#define TIOCGPKT _IOR('T', 0x38, int) /* Get packet mode state */
#define TIOCGPTLCK _IOR('T', 0x39, int) /* Get Pty lock state */
#define TIOCGEXCL _IOR('T', 0x40, int) /* Get exclusive mode state */
+#define TIOCGPTPEER _IOR('T', 0x41, int) /* Safely open the slave */
#define TIOCSERCONFIG _IO('T', 83)
#define TIOCSERGWILD _IOR('T', 84, int)
local_timer_setup(0);
setup_irq(this_cpu_ptr(&ccount_timer)->evt.irq, &timer_irqaction);
sched_clock_register(ccount_sched_clock_read, 32, ccount_freq);
- clocksource_probe();
+ timer_probe();
}
/*
case 3:
if (newline != '\n')
return -EINVAL;
+ /* fall through */
case 2:
if (length <= 0)
return -EINVAL;
}
static void
-bfq_bfqq_resume_state(struct bfq_queue *bfqq, struct bfq_io_cq *bic)
+bfq_bfqq_resume_state(struct bfq_queue *bfqq, struct bfq_data *bfqd,
+ struct bfq_io_cq *bic, bool bfq_already_existing)
{
+ unsigned int old_wr_coeff = bfqq->wr_coeff;
+ bool busy = bfq_already_existing && bfq_bfqq_busy(bfqq);
+
if (bic->saved_idle_window)
bfq_mark_bfqq_idle_window(bfqq);
else
/* make sure weight will be updated, however we got here */
bfqq->entity.prio_changed = 1;
+
+ if (likely(!busy))
+ return;
+
+ if (old_wr_coeff == 1 && bfqq->wr_coeff > 1)
+ bfqd->wr_busy_queues++;
+ else if (old_wr_coeff > 1 && bfqq->wr_coeff == 1)
+ bfqd->wr_busy_queues--;
}
static int bfqq_process_refs(struct bfq_queue *bfqq)
bfq_put_queue(bfqq);
}
-static void bfq_put_rq_private(struct request_queue *q, struct request *rq)
+static void bfq_finish_request(struct request *rq)
{
- struct bfq_queue *bfqq = RQ_BFQQ(rq);
- struct bfq_data *bfqd = bfqq->bfqd;
+ struct bfq_queue *bfqq;
+ struct bfq_data *bfqd;
+
+ if (!rq->elv.icq)
+ return;
+
+ bfqq = RQ_BFQQ(rq);
+ bfqd = bfqq->bfqd;
if (rq->rq_flags & RQF_STARTED)
bfqg_stats_update_completion(bfqq_group(bfqq),
*/
if (!RB_EMPTY_NODE(&rq->rb_node))
- bfq_remove_request(q, rq);
+ bfq_remove_request(rq->q, rq);
bfq_put_rq_priv_body(bfqq);
}
/*
* Allocate bfq data structures associated with this request.
*/
-static int bfq_get_rq_private(struct request_queue *q, struct request *rq,
- struct bio *bio)
+static void bfq_prepare_request(struct request *rq, struct bio *bio)
{
+ struct request_queue *q = rq->q;
struct bfq_data *bfqd = q->elevator->elevator_data;
- struct bfq_io_cq *bic = icq_to_bic(rq->elv.icq);
+ struct bfq_io_cq *bic;
const int is_sync = rq_is_sync(rq);
struct bfq_queue *bfqq;
bool new_queue = false;
- bool split = false;
+ bool bfqq_already_existing = false, split = false;
- spin_lock_irq(&bfqd->lock);
+ if (!rq->elv.icq)
+ return;
+ bic = icq_to_bic(rq->elv.icq);
- if (!bic)
- goto queue_fail;
+ spin_lock_irq(&bfqd->lock);
bfq_check_ioprio_change(bic, bio);
bfqq = bfq_get_bfqq_handle_split(bfqd, bic, bio,
true, is_sync,
NULL);
+ else
+ bfqq_already_existing = true;
}
}
* queue: restore the idle window and the
* possible weight raising period.
*/
- bfq_bfqq_resume_state(bfqq, bic);
+ bfq_bfqq_resume_state(bfqq, bfqd, bic,
+ bfqq_already_existing);
}
}
bfq_handle_burst(bfqd, bfqq);
spin_unlock_irq(&bfqd->lock);
-
- return 0;
-
-queue_fail:
- spin_unlock_irq(&bfqd->lock);
-
- return 1;
}
static void bfq_idle_slice_timer_body(struct bfq_queue *bfqq)
static struct elevator_type iosched_bfq_mq = {
.ops.mq = {
- .get_rq_priv = bfq_get_rq_private,
- .put_rq_priv = bfq_put_rq_private,
+ .prepare_request = bfq_prepare_request,
+ .finish_request = bfq_finish_request,
.exit_icq = bfq_exit_icq,
.insert_requests = bfq_insert_requests,
.dispatch_request = bfq_dispatch_request,
* @bio: bio to generate/verify integrity metadata for
* @proc_fn: Pointer to the relevant processing function
*/
-static int bio_integrity_process(struct bio *bio,
+static blk_status_t bio_integrity_process(struct bio *bio,
integrity_processing_fn *proc_fn)
{
struct blk_integrity *bi = bdev_get_integrity(bio->bi_bdev);
struct bvec_iter bviter;
struct bio_vec bv;
struct bio_integrity_payload *bip = bio_integrity(bio);
- unsigned int ret = 0;
+ blk_status_t ret = BLK_STS_OK;
void *prot_buf = page_address(bip->bip_vec->bv_page) +
bip->bip_vec->bv_offset;
struct bio *bio = bip->bip_bio;
struct blk_integrity *bi = bdev_get_integrity(bio->bi_bdev);
- bio->bi_error = bio_integrity_process(bio, bi->profile->verify_fn);
+ bio->bi_status = bio_integrity_process(bio, bi->profile->verify_fn);
/* Restore original bio completion handler */
bio->bi_end_io = bip->bip_end_io;
* integrity metadata. Restore original bio end_io handler
* and run it.
*/
- if (bio->bi_error) {
+ if (bio->bi_status) {
bio->bi_end_io = bip->bip_end_io;
bio_endio(bio);
return bvl;
}
-static void __bio_free(struct bio *bio)
+void bio_uninit(struct bio *bio)
{
bio_disassociate_task(bio);
if (bio_integrity(bio))
bio_integrity_free(bio);
}
+EXPORT_SYMBOL(bio_uninit);
static void bio_free(struct bio *bio)
{
struct bio_set *bs = bio->bi_pool;
void *p;
- __bio_free(bio);
+ bio_uninit(bio);
if (bs) {
bvec_free(bs->bvec_pool, bio->bi_io_vec, BVEC_POOL_IDX(bio));
}
}
+/*
+ * Users of this function have their own bio allocation. Subsequently,
+ * they must remember to pair any call to bio_init() with bio_uninit()
+ * when IO has completed, or when the bio is released.
+ */
void bio_init(struct bio *bio, struct bio_vec *table,
unsigned short max_vecs)
{
{
unsigned long flags = bio->bi_flags & (~0UL << BIO_RESET_BITS);
- __bio_free(bio);
+ bio_uninit(bio);
memset(bio, 0, BIO_RESET_BYTES);
bio->bi_flags = flags;
{
struct bio *parent = bio->bi_private;
- if (!parent->bi_error)
- parent->bi_error = bio->bi_error;
+ if (!parent->bi_status)
+ parent->bi_status = bio->bi_status;
bio_put(bio);
return parent;
}
struct bio_list punt, nopunt;
struct bio *bio;
+ if (WARN_ON_ONCE(!bs->rescue_workqueue))
+ return;
/*
* In order to guarantee forward progress we must punt only bios that
* were allocated from this bio_set; otherwise, if there was a bio on
if (current->bio_list &&
(!bio_list_empty(¤t->bio_list[0]) ||
- !bio_list_empty(¤t->bio_list[1])))
+ !bio_list_empty(¤t->bio_list[1])) &&
+ bs->rescue_workqueue)
gfp_mask &= ~__GFP_DIRECT_RECLAIM;
p = mempool_alloc(bs->bio_pool, gfp_mask);
*
* Description:
* Put a reference to a &struct bio, either one you have gotten with
- * bio_alloc, bio_get or bio_clone. The last put of a bio will free it.
+ * bio_alloc, bio_get or bio_clone_*. The last put of a bio will free it.
**/
void bio_put(struct bio *bio)
{
bio->bi_bdev = bio_src->bi_bdev;
bio_set_flag(bio, BIO_CLONED);
bio->bi_opf = bio_src->bi_opf;
+ bio->bi_write_hint = bio_src->bi_write_hint;
bio->bi_iter = bio_src->bi_iter;
bio->bi_io_vec = bio_src->bi_io_vec;
return NULL;
bio->bi_bdev = bio_src->bi_bdev;
bio->bi_opf = bio_src->bi_opf;
+ bio->bi_write_hint = bio_src->bi_write_hint;
bio->bi_iter.bi_sector = bio_src->bi_iter.bi_sector;
bio->bi_iter.bi_size = bio_src->bi_iter.bi_size;
{
struct submit_bio_ret *ret = bio->bi_private;
- ret->error = bio->bi_error;
+ ret->error = blk_status_to_errno(bio->bi_status);
complete(&ret->event);
}
}
if (bio->bi_bdev && bio_flagged(bio, BIO_TRACE_COMPLETION)) {
- trace_block_bio_complete(bdev_get_queue(bio->bi_bdev),
- bio, bio->bi_error);
+ trace_block_bio_complete(bdev_get_queue(bio->bi_bdev), bio,
+ blk_status_to_errno(bio->bi_status));
bio_clear_flag(bio, BIO_TRACE_COMPLETION);
}
}
EXPORT_SYMBOL(bioset_free);
-static struct bio_set *__bioset_create(unsigned int pool_size,
- unsigned int front_pad,
- bool create_bvec_pool)
+/**
+ * bioset_create - Create a bio_set
+ * @pool_size: Number of bio and bio_vecs to cache in the mempool
+ * @front_pad: Number of bytes to allocate in front of the returned bio
+ * @flags: Flags to modify behavior, currently %BIOSET_NEED_BVECS
+ * and %BIOSET_NEED_RESCUER
+ *
+ * Description:
+ * Set up a bio_set to be used with @bio_alloc_bioset. Allows the caller
+ * to ask for a number of bytes to be allocated in front of the bio.
+ * Front pad allocation is useful for embedding the bio inside
+ * another structure, to avoid allocating extra data to go with the bio.
+ * Note that the bio must be embedded at the END of that structure always,
+ * or things will break badly.
+ * If %BIOSET_NEED_BVECS is set in @flags, a separate pool will be allocated
+ * for allocating iovecs. This pool is not needed e.g. for bio_clone_fast().
+ * If %BIOSET_NEED_RESCUER is set, a workqueue is created which can be used to
+ * dispatch queued requests when the mempool runs out of space.
+ *
+ */
+struct bio_set *bioset_create(unsigned int pool_size,
+ unsigned int front_pad,
+ int flags)
{
unsigned int back_pad = BIO_INLINE_VECS * sizeof(struct bio_vec);
struct bio_set *bs;
if (!bs->bio_pool)
goto bad;
- if (create_bvec_pool) {
+ if (flags & BIOSET_NEED_BVECS) {
bs->bvec_pool = biovec_create_pool(pool_size);
if (!bs->bvec_pool)
goto bad;
}
+ if (!(flags & BIOSET_NEED_RESCUER))
+ return bs;
+
bs->rescue_workqueue = alloc_workqueue("bioset", WQ_MEM_RECLAIM, 0);
if (!bs->rescue_workqueue)
goto bad;
bioset_free(bs);
return NULL;
}
-
-/**
- * bioset_create - Create a bio_set
- * @pool_size: Number of bio and bio_vecs to cache in the mempool
- * @front_pad: Number of bytes to allocate in front of the returned bio
- *
- * Description:
- * Set up a bio_set to be used with @bio_alloc_bioset. Allows the caller
- * to ask for a number of bytes to be allocated in front of the bio.
- * Front pad allocation is useful for embedding the bio inside
- * another structure, to avoid allocating extra data to go with the bio.
- * Note that the bio must be embedded at the END of that structure always,
- * or things will break badly.
- */
-struct bio_set *bioset_create(unsigned int pool_size, unsigned int front_pad)
-{
- return __bioset_create(pool_size, front_pad, true);
-}
EXPORT_SYMBOL(bioset_create);
-/**
- * bioset_create_nobvec - Create a bio_set without bio_vec mempool
- * @pool_size: Number of bio to cache in the mempool
- * @front_pad: Number of bytes to allocate in front of the returned bio
- *
- * Description:
- * Same functionality as bioset_create() except that mempool is not
- * created for bio_vecs. Saving some memory for bio_clone_fast() users.
- */
-struct bio_set *bioset_create_nobvec(unsigned int pool_size, unsigned int front_pad)
-{
- return __bioset_create(pool_size, front_pad, false);
-}
-EXPORT_SYMBOL(bioset_create_nobvec);
-
#ifdef CONFIG_BLK_CGROUP
/**
bio_integrity_init();
biovec_init_slabs();
- fs_bio_set = bioset_create(BIO_POOL_SIZE, 0);
+ fs_bio_set = bioset_create(BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS);
if (!fs_bio_set)
panic("bio: can't allocate bios\n");
}
EXPORT_SYMBOL(blk_rq_init);
+static const struct {
+ int errno;
+ const char *name;
+} blk_errors[] = {
+ [BLK_STS_OK] = { 0, "" },
+ [BLK_STS_NOTSUPP] = { -EOPNOTSUPP, "operation not supported" },
+ [BLK_STS_TIMEOUT] = { -ETIMEDOUT, "timeout" },
+ [BLK_STS_NOSPC] = { -ENOSPC, "critical space allocation" },
+ [BLK_STS_TRANSPORT] = { -ENOLINK, "recoverable transport" },
+ [BLK_STS_TARGET] = { -EREMOTEIO, "critical target" },
+ [BLK_STS_NEXUS] = { -EBADE, "critical nexus" },
+ [BLK_STS_MEDIUM] = { -ENODATA, "critical medium" },
+ [BLK_STS_PROTECTION] = { -EILSEQ, "protection" },
+ [BLK_STS_RESOURCE] = { -ENOMEM, "kernel resource" },
+ [BLK_STS_AGAIN] = { -EAGAIN, "nonblocking retry" },
+
+ /* device mapper special case, should not leak out: */
+ [BLK_STS_DM_REQUEUE] = { -EREMCHG, "dm internal retry" },
+
+ /* everything else not covered above: */
+ [BLK_STS_IOERR] = { -EIO, "I/O" },
+};
+
+blk_status_t errno_to_blk_status(int errno)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(blk_errors); i++) {
+ if (blk_errors[i].errno == errno)
+ return (__force blk_status_t)i;
+ }
+
+ return BLK_STS_IOERR;
+}
+EXPORT_SYMBOL_GPL(errno_to_blk_status);
+
+int blk_status_to_errno(blk_status_t status)
+{
+ int idx = (__force int)status;
+
+ if (WARN_ON_ONCE(idx >= ARRAY_SIZE(blk_errors)))
+ return -EIO;
+ return blk_errors[idx].errno;
+}
+EXPORT_SYMBOL_GPL(blk_status_to_errno);
+
+static void print_req_error(struct request *req, blk_status_t status)
+{
+ int idx = (__force int)status;
+
+ if (WARN_ON_ONCE(idx >= ARRAY_SIZE(blk_errors)))
+ return;
+
+ printk_ratelimited(KERN_ERR "%s: %s error, dev %s, sector %llu\n",
+ __func__, blk_errors[idx].name, req->rq_disk ?
+ req->rq_disk->disk_name : "?",
+ (unsigned long long)blk_rq_pos(req));
+}
+
static void req_bio_endio(struct request *rq, struct bio *bio,
- unsigned int nbytes, int error)
+ unsigned int nbytes, blk_status_t error)
{
if (error)
- bio->bi_error = error;
+ bio->bi_status = error;
if (unlikely(rq->rq_flags & RQF_QUIET))
bio_set_flag(bio, BIO_QUIET);
* Description:
* Sometimes queueing needs to be postponed for a little while, to allow
* resources to come back. This function will make sure that queueing is
- * restarted around the specified time. Queue lock must be held.
+ * restarted around the specified time.
*/
void blk_delay_queue(struct request_queue *q, unsigned long msecs)
{
+ lockdep_assert_held(q->queue_lock);
+ WARN_ON_ONCE(q->mq_ops);
+
if (likely(!blk_queue_dead(q)))
queue_delayed_work(kblockd_workqueue, &q->delay_work,
msecs_to_jiffies(msecs));
**/
void blk_start_queue_async(struct request_queue *q)
{
+ lockdep_assert_held(q->queue_lock);
+ WARN_ON_ONCE(q->mq_ops);
+
queue_flag_clear(QUEUE_FLAG_STOPPED, q);
blk_run_queue_async(q);
}
* Description:
* blk_start_queue() will clear the stop flag on the queue, and call
* the request_fn for the queue if it was in a stopped state when
- * entered. Also see blk_stop_queue(). Queue lock must be held.
+ * entered. Also see blk_stop_queue().
**/
void blk_start_queue(struct request_queue *q)
{
+ lockdep_assert_held(q->queue_lock);
WARN_ON(!irqs_disabled());
+ WARN_ON_ONCE(q->mq_ops);
queue_flag_clear(QUEUE_FLAG_STOPPED, q);
__blk_run_queue(q);
* or if it simply chooses not to queue more I/O at one point, it can
* call this function to prevent the request_fn from being called until
* the driver has signalled it's ready to go again. This happens by calling
- * blk_start_queue() to restart queue operations. Queue lock must be held.
+ * blk_start_queue() to restart queue operations.
**/
void blk_stop_queue(struct request_queue *q)
{
+ lockdep_assert_held(q->queue_lock);
+ WARN_ON_ONCE(q->mq_ops);
+
cancel_delayed_work(&q->delay_work);
queue_flag_set(QUEUE_FLAG_STOPPED, q);
}
*/
inline void __blk_run_queue_uncond(struct request_queue *q)
{
+ lockdep_assert_held(q->queue_lock);
+ WARN_ON_ONCE(q->mq_ops);
+
if (unlikely(blk_queue_dead(q)))
return;
* @q: The queue to run
*
* Description:
- * See @blk_run_queue. This variant must be called with the queue lock
- * held and interrupts disabled.
+ * See @blk_run_queue.
*/
void __blk_run_queue(struct request_queue *q)
{
+ lockdep_assert_held(q->queue_lock);
+ WARN_ON_ONCE(q->mq_ops);
+
if (unlikely(blk_queue_stopped(q)))
return;
*
* Description:
* Tells kblockd to perform the equivalent of @blk_run_queue on behalf
- * of us. The caller must hold the queue lock.
+ * of us.
+ *
+ * Note:
+ * Since it is not allowed to run q->delay_work after blk_cleanup_queue()
+ * has canceled q->delay_work, callers must hold the queue lock to avoid
+ * race conditions between blk_cleanup_queue() and blk_run_queue_async().
*/
void blk_run_queue_async(struct request_queue *q)
{
+ lockdep_assert_held(q->queue_lock);
+ WARN_ON_ONCE(q->mq_ops);
+
if (likely(!blk_queue_stopped(q) && !blk_queue_dead(q)))
mod_delayed_work(kblockd_workqueue, &q->delay_work, 0);
}
{
unsigned long flags;
+ WARN_ON_ONCE(q->mq_ops);
+
spin_lock_irqsave(q->queue_lock, flags);
__blk_run_queue(q);
spin_unlock_irqrestore(q->queue_lock, flags);
int i;
lockdep_assert_held(q->queue_lock);
+ WARN_ON_ONCE(q->mq_ops);
while (true) {
bool drain = false;
*/
void blk_queue_bypass_start(struct request_queue *q)
{
+ WARN_ON_ONCE(q->mq_ops);
+
spin_lock_irq(q->queue_lock);
q->bypass_depth++;
queue_flag_set(QUEUE_FLAG_BYPASS, q);
* @q: queue of interest
*
* Leave bypass mode and restore the normal queueing behavior.
+ *
+ * Note: although blk_queue_bypass_start() is only called for blk-sq queues,
+ * this function is called for both blk-sq and blk-mq queues.
*/
void blk_queue_bypass_end(struct request_queue *q)
{
if (q->id < 0)
goto fail_q;
- q->bio_split = bioset_create(BIO_POOL_SIZE, 0);
+ q->bio_split = bioset_create(BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS);
if (!q->bio_split)
goto fail_id;
int blk_init_allocated_queue(struct request_queue *q)
{
+ WARN_ON_ONCE(q->mq_ops);
+
q->fq = blk_alloc_flush_queue(q, NUMA_NO_NODE, q->cmd_size);
if (!q->fq)
return -ENOMEM;
struct request_list *rl;
int on_thresh, off_thresh;
+ WARN_ON_ONCE(q->mq_ops);
+
spin_lock_irq(q->queue_lock);
q->nr_requests = nr;
blk_queue_congestion_threshold(q);
int may_queue;
req_flags_t rq_flags = RQF_ALLOCED;
+ lockdep_assert_held(q->queue_lock);
+
if (unlikely(blk_queue_dying(q)))
return ERR_PTR(-ENODEV);
struct request_list *rl;
struct request *rq;
+ lockdep_assert_held(q->queue_lock);
+ WARN_ON_ONCE(q->mq_ops);
+
rl = blk_get_rl(q, bio); /* transferred to @rq on success */
retry:
rq = __get_request(rl, op, bio, gfp_mask);
if (!IS_ERR(rq))
return rq;
+ if (op & REQ_NOWAIT) {
+ blk_put_rl(rl);
+ return ERR_PTR(-EAGAIN);
+ }
+
if (!gfpflags_allow_blocking(gfp_mask) || unlikely(blk_queue_dying(q))) {
blk_put_rl(rl);
return rq;
goto retry;
}
-static struct request *blk_old_get_request(struct request_queue *q, int rw,
- gfp_t gfp_mask)
+static struct request *blk_old_get_request(struct request_queue *q,
+ unsigned int op, gfp_t gfp_mask)
{
struct request *rq;
+ WARN_ON_ONCE(q->mq_ops);
+
/* create ioc upfront */
create_io_context(gfp_mask, q->node);
spin_lock_irq(q->queue_lock);
- rq = get_request(q, rw, NULL, gfp_mask);
+ rq = get_request(q, op, NULL, gfp_mask);
if (IS_ERR(rq)) {
spin_unlock_irq(q->queue_lock);
return rq;
return rq;
}
-struct request *blk_get_request(struct request_queue *q, int rw, gfp_t gfp_mask)
+struct request *blk_get_request(struct request_queue *q, unsigned int op,
+ gfp_t gfp_mask)
{
- if (q->mq_ops)
- return blk_mq_alloc_request(q, rw,
+ struct request *req;
+
+ if (q->mq_ops) {
+ req = blk_mq_alloc_request(q, op,
(gfp_mask & __GFP_DIRECT_RECLAIM) ?
0 : BLK_MQ_REQ_NOWAIT);
- else
- return blk_old_get_request(q, rw, gfp_mask);
+ if (!IS_ERR(req) && q->mq_ops->initialize_rq_fn)
+ q->mq_ops->initialize_rq_fn(req);
+ } else {
+ req = blk_old_get_request(q, op, gfp_mask);
+ if (!IS_ERR(req) && q->initialize_rq_fn)
+ q->initialize_rq_fn(req);
+ }
+
+ return req;
}
EXPORT_SYMBOL(blk_get_request);
*/
void blk_requeue_request(struct request_queue *q, struct request *rq)
{
+ lockdep_assert_held(q->queue_lock);
+ WARN_ON_ONCE(q->mq_ops);
+
blk_delete_timer(rq);
blk_clear_rq_complete(rq);
trace_block_rq_requeue(q, rq);
static inline void blk_pm_put_request(struct request *rq) {}
#endif
-/*
- * queue lock must be held
- */
void __blk_put_request(struct request_queue *q, struct request *req)
{
req_flags_t rq_flags = req->rq_flags;
return;
}
+ lockdep_assert_held(q->queue_lock);
+
blk_pm_put_request(req);
elv_completed_request(q, req);
req->ioprio = ioc->ioprio;
else
req->ioprio = IOPRIO_PRIO_VALUE(IOPRIO_CLASS_NONE, 0);
+ req->write_hint = bio->bi_write_hint;
blk_rq_bio_prep(req->q, req, bio);
}
EXPORT_SYMBOL_GPL(blk_init_request_from_bio);
*/
blk_queue_bounce(q, &bio);
- blk_queue_split(q, &bio, q->bio_split);
+ blk_queue_split(q, &bio);
if (bio_integrity_enabled(bio) && bio_integrity_prep(bio)) {
- bio->bi_error = -EIO;
+ bio->bi_status = BLK_STS_IOERR;
bio_endio(bio);
return BLK_QC_T_NONE;
}
req = get_request(q, bio->bi_opf, bio, GFP_NOIO);
if (IS_ERR(req)) {
__wbt_done(q->rq_wb, wb_acct);
- bio->bi_error = PTR_ERR(req);
+ if (PTR_ERR(req) == -ENOMEM)
+ bio->bi_status = BLK_STS_RESOURCE;
+ else
+ bio->bi_status = BLK_STS_IOERR;
bio_endio(bio);
goto out_unlock;
}
{
struct request_queue *q;
int nr_sectors = bio_sectors(bio);
- int err = -EIO;
+ blk_status_t status = BLK_STS_IOERR;
char b[BDEVNAME_SIZE];
struct hd_struct *part;
goto end_io;
}
+ /*
+ * For a REQ_NOWAIT based request, return -EOPNOTSUPP
+ * if queue is not a request based queue.
+ */
+
+ if ((bio->bi_opf & REQ_NOWAIT) && !queue_is_rq_based(q))
+ goto not_supported;
+
part = bio->bi_bdev->bd_part;
if (should_fail_request(part, bio->bi_iter.bi_size) ||
should_fail_request(&part_to_disk(part)->part0,
!test_bit(QUEUE_FLAG_WC, &q->queue_flags)) {
bio->bi_opf &= ~(REQ_PREFLUSH | REQ_FUA);
if (!nr_sectors) {
- err = 0;
+ status = BLK_STS_OK;
goto end_io;
}
}
return true;
not_supported:
- err = -EOPNOTSUPP;
+ status = BLK_STS_NOTSUPP;
end_io:
- bio->bi_error = err;
+ bio->bi_status = status;
bio_endio(bio);
return false;
}
do {
struct request_queue *q = bdev_get_queue(bio->bi_bdev);
- if (likely(blk_queue_enter(q, false) == 0)) {
+ if (likely(blk_queue_enter(q, bio->bi_opf & REQ_NOWAIT) == 0)) {
struct bio_list lower, same;
/* Create a fresh bio_list for all subordinate requests */
bio_list_merge(&bio_list_on_stack[0], &same);
bio_list_merge(&bio_list_on_stack[0], &bio_list_on_stack[1]);
} else {
- bio_io_error(bio);
+ if (unlikely(!blk_queue_dying(q) &&
+ (bio->bi_opf & REQ_NOWAIT)))
+ bio_wouldblock_error(bio);
+ else
+ bio_io_error(bio);
}
bio = bio_list_pop(&bio_list_on_stack[0]);
} while (bio);
* @q: the queue to submit the request
* @rq: the request being queued
*/
-int blk_insert_cloned_request(struct request_queue *q, struct request *rq)
+blk_status_t blk_insert_cloned_request(struct request_queue *q, struct request *rq)
{
unsigned long flags;
int where = ELEVATOR_INSERT_BACK;
if (blk_cloned_rq_check_limits(q, rq))
- return -EIO;
+ return BLK_STS_IOERR;
if (rq->rq_disk &&
should_fail_request(&rq->rq_disk->part0, blk_rq_bytes(rq)))
- return -EIO;
+ return BLK_STS_IOERR;
if (q->mq_ops) {
if (blk_queue_io_stat(q))
blk_account_io_start(rq, true);
blk_mq_sched_insert_request(rq, false, true, false, false);
- return 0;
+ return BLK_STS_OK;
}
spin_lock_irqsave(q->queue_lock, flags);
if (unlikely(blk_queue_dying(q))) {
spin_unlock_irqrestore(q->queue_lock, flags);
- return -ENODEV;
+ return BLK_STS_IOERR;
}
/*
__blk_run_queue(q);
spin_unlock_irqrestore(q->queue_lock, flags);
- return 0;
+ return BLK_STS_OK;
}
EXPORT_SYMBOL_GPL(blk_insert_cloned_request);
*
* Return:
* The number of bytes to fail.
- *
- * Context:
- * queue_lock must be held.
*/
unsigned int blk_rq_err_bytes(const struct request *rq)
{
* Return:
* Pointer to the request at the top of @q if available. Null
* otherwise.
- *
- * Context:
- * queue_lock must be held.
*/
struct request *blk_peek_request(struct request_queue *q)
{
struct request *rq;
int ret;
+ lockdep_assert_held(q->queue_lock);
+ WARN_ON_ONCE(q->mq_ops);
+
while ((rq = __elv_next_request(q)) != NULL) {
rq = blk_pm_peek_request(q, rq);
rq = NULL;
break;
} else if (ret == BLKPREP_KILL || ret == BLKPREP_INVALID) {
- int err = (ret == BLKPREP_INVALID) ? -EREMOTEIO : -EIO;
-
rq->rq_flags |= RQF_QUIET;
/*
* Mark this request as started so we don't trigger
* any debug logic in the end I/O path.
*/
blk_start_request(rq);
- __blk_end_request_all(rq, err);
+ __blk_end_request_all(rq, ret == BLKPREP_INVALID ?
+ BLK_STS_TARGET : BLK_STS_IOERR);
} else {
printk(KERN_ERR "%s: bad return=%d\n", __func__, ret);
break;
*
* Block internal functions which don't want to start timer should
* call blk_dequeue_request().
- *
- * Context:
- * queue_lock must be held.
*/
void blk_start_request(struct request *req)
{
+ lockdep_assert_held(req->q->queue_lock);
+ WARN_ON_ONCE(req->q->mq_ops);
+
blk_dequeue_request(req);
if (test_bit(QUEUE_FLAG_STATS, &req->q->queue_flags)) {
* Return:
* Pointer to the request at the top of @q if available. Null
* otherwise.
- *
- * Context:
- * queue_lock must be held.
*/
struct request *blk_fetch_request(struct request_queue *q)
{
struct request *rq;
+ lockdep_assert_held(q->queue_lock);
+ WARN_ON_ONCE(q->mq_ops);
+
rq = blk_peek_request(q);
if (rq)
blk_start_request(rq);
/**
* blk_update_request - Special helper function for request stacking drivers
* @req: the request being processed
- * @error: %0 for success, < %0 for error
+ * @error: block status code
* @nr_bytes: number of bytes to complete @req
*
* Description:
* %false - this request doesn't have any more data
* %true - this request has more data
**/
-bool blk_update_request(struct request *req, int error, unsigned int nr_bytes)
+bool blk_update_request(struct request *req, blk_status_t error,
+ unsigned int nr_bytes)
{
int total_bytes;
- trace_block_rq_complete(req, error, nr_bytes);
+ trace_block_rq_complete(req, blk_status_to_errno(error), nr_bytes);
if (!req->bio)
return false;
- if (error && !blk_rq_is_passthrough(req) &&
- !(req->rq_flags & RQF_QUIET)) {
- char *error_type;
-
- switch (error) {
- case -ENOLINK:
- error_type = "recoverable transport";
- break;
- case -EREMOTEIO:
- error_type = "critical target";
- break;
- case -EBADE:
- error_type = "critical nexus";
- break;
- case -ETIMEDOUT:
- error_type = "timeout";
- break;
- case -ENOSPC:
- error_type = "critical space allocation";
- break;
- case -ENODATA:
- error_type = "critical medium";
- break;
- case -EIO:
- default:
- error_type = "I/O";
- break;
- }
- printk_ratelimited(KERN_ERR "%s: %s error, dev %s, sector %llu\n",
- __func__, error_type, req->rq_disk ?
- req->rq_disk->disk_name : "?",
- (unsigned long long)blk_rq_pos(req));
-
- }
+ if (unlikely(error && !blk_rq_is_passthrough(req) &&
+ !(req->rq_flags & RQF_QUIET)))
+ print_req_error(req, error);
blk_account_io_completion(req, nr_bytes);
}
EXPORT_SYMBOL_GPL(blk_update_request);
-static bool blk_update_bidi_request(struct request *rq, int error,
+static bool blk_update_bidi_request(struct request *rq, blk_status_t error,
unsigned int nr_bytes,
unsigned int bidi_bytes)
{
}
EXPORT_SYMBOL_GPL(blk_unprep_request);
-/*
- * queue lock must be held
- */
-void blk_finish_request(struct request *req, int error)
+void blk_finish_request(struct request *req, blk_status_t error)
{
struct request_queue *q = req->q;
+ lockdep_assert_held(req->q->queue_lock);
+ WARN_ON_ONCE(q->mq_ops);
+
if (req->rq_flags & RQF_STATS)
blk_stat_add(req);
/**
* blk_end_bidi_request - Complete a bidi request
* @rq: the request to complete
- * @error: %0 for success, < %0 for error
+ * @error: block status code
* @nr_bytes: number of bytes to complete @rq
* @bidi_bytes: number of bytes to complete @rq->next_rq
*
* %false - we are done with this request
* %true - still buffers pending for this request
**/
-static bool blk_end_bidi_request(struct request *rq, int error,
+static bool blk_end_bidi_request(struct request *rq, blk_status_t error,
unsigned int nr_bytes, unsigned int bidi_bytes)
{
struct request_queue *q = rq->q;
unsigned long flags;
+ WARN_ON_ONCE(q->mq_ops);
+
if (blk_update_bidi_request(rq, error, nr_bytes, bidi_bytes))
return true;
/**
* __blk_end_bidi_request - Complete a bidi request with queue lock held
* @rq: the request to complete
- * @error: %0 for success, < %0 for error
+ * @error: block status code
* @nr_bytes: number of bytes to complete @rq
* @bidi_bytes: number of bytes to complete @rq->next_rq
*
* %false - we are done with this request
* %true - still buffers pending for this request
**/
-static bool __blk_end_bidi_request(struct request *rq, int error,
+static bool __blk_end_bidi_request(struct request *rq, blk_status_t error,
unsigned int nr_bytes, unsigned int bidi_bytes)
{
+ lockdep_assert_held(rq->q->queue_lock);
+ WARN_ON_ONCE(rq->q->mq_ops);
+
if (blk_update_bidi_request(rq, error, nr_bytes, bidi_bytes))
return true;
/**
* blk_end_request - Helper function for drivers to complete the request.
* @rq: the request being processed
- * @error: %0 for success, < %0 for error
+ * @error: block status code
* @nr_bytes: number of bytes to complete
*
* Description:
* %false - we are done with this request
* %true - still buffers pending for this request
**/
-bool blk_end_request(struct request *rq, int error, unsigned int nr_bytes)
+bool blk_end_request(struct request *rq, blk_status_t error,
+ unsigned int nr_bytes)
{
+ WARN_ON_ONCE(rq->q->mq_ops);
return blk_end_bidi_request(rq, error, nr_bytes, 0);
}
EXPORT_SYMBOL(blk_end_request);
/**
* blk_end_request_all - Helper function for drives to finish the request.
* @rq: the request to finish
- * @error: %0 for success, < %0 for error
+ * @error: block status code
*
* Description:
* Completely finish @rq.
*/
-void blk_end_request_all(struct request *rq, int error)
+void blk_end_request_all(struct request *rq, blk_status_t error)
{
bool pending;
unsigned int bidi_bytes = 0;
/**
* __blk_end_request - Helper function for drivers to complete the request.
* @rq: the request being processed
- * @error: %0 for success, < %0 for error
+ * @error: block status code
* @nr_bytes: number of bytes to complete
*
* Description:
* %false - we are done with this request
* %true - still buffers pending for this request
**/
-bool __blk_end_request(struct request *rq, int error, unsigned int nr_bytes)
+bool __blk_end_request(struct request *rq, blk_status_t error,
+ unsigned int nr_bytes)
{
+ lockdep_assert_held(rq->q->queue_lock);
+ WARN_ON_ONCE(rq->q->mq_ops);
+
return __blk_end_bidi_request(rq, error, nr_bytes, 0);
}
EXPORT_SYMBOL(__blk_end_request);
/**
* __blk_end_request_all - Helper function for drives to finish the request.
* @rq: the request to finish
- * @error: %0 for success, < %0 for error
+ * @error: block status code
*
* Description:
* Completely finish @rq. Must be called with queue lock held.
*/
-void __blk_end_request_all(struct request *rq, int error)
+void __blk_end_request_all(struct request *rq, blk_status_t error)
{
bool pending;
unsigned int bidi_bytes = 0;
+ lockdep_assert_held(rq->q->queue_lock);
+ WARN_ON_ONCE(rq->q->mq_ops);
+
if (unlikely(blk_bidi_rq(rq)))
bidi_bytes = blk_rq_bytes(rq->next_rq);
/**
* __blk_end_request_cur - Helper function to finish the current request chunk.
* @rq: the request to finish the current chunk for
- * @error: %0 for success, < %0 for error
+ * @error: block status code
*
* Description:
* Complete the current consecutively mapped chunk from @rq. Must
* %false - we are done with this request
* %true - still buffers pending for this request
*/
-bool __blk_end_request_cur(struct request *rq, int error)
+bool __blk_end_request_cur(struct request *rq, blk_status_t error)
{
return __blk_end_request(rq, error, blk_rq_cur_bytes(rq));
}
bool from_schedule)
__releases(q->queue_lock)
{
+ lockdep_assert_held(q->queue_lock);
+
trace_block_unplug(q, depth, !from_schedule);
if (from_schedule)
* Short-circuit if @q is dead
*/
if (unlikely(blk_queue_dying(q))) {
- __blk_end_request_all(rq, -ENODEV);
+ __blk_end_request_all(rq, BLK_STS_IOERR);
continue;
}
* @rq: request to complete
* @error: end I/O status of the request
*/
-static void blk_end_sync_rq(struct request *rq, int error)
+static void blk_end_sync_rq(struct request *rq, blk_status_t error)
{
struct completion *waiting = rq->end_io_data;
if (unlikely(blk_queue_dying(q))) {
rq->rq_flags |= RQF_QUIET;
- __blk_end_request_all(rq, -ENXIO);
+ __blk_end_request_all(rq, BLK_STS_IOERR);
spin_unlock_irq(q->queue_lock);
return;
}
*/
static bool blk_flush_complete_seq(struct request *rq,
struct blk_flush_queue *fq,
- unsigned int seq, int error)
+ unsigned int seq, blk_status_t error)
{
struct request_queue *q = rq->q;
struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx];
return kicked | queued;
}
-static void flush_end_io(struct request *flush_rq, int error)
+static void flush_end_io(struct request *flush_rq, blk_status_t error)
{
struct request_queue *q = flush_rq->q;
struct list_head *running;
return blk_flush_queue_rq(flush_rq, false);
}
-static void flush_data_end_io(struct request *rq, int error)
+static void flush_data_end_io(struct request *rq, blk_status_t error)
{
struct request_queue *q = rq->q;
struct blk_flush_queue *fq = blk_get_flush_queue(q, NULL);
+ lockdep_assert_held(q->queue_lock);
+
/*
* Updating q->in_flight[] here for making this tag usable
* early. Because in blk_queue_start_tag(),
blk_run_queue_async(q);
}
-static void mq_flush_data_end_io(struct request *rq, int error)
+static void mq_flush_data_end_io(struct request *rq, blk_status_t error)
{
struct request_queue *q = rq->q;
struct blk_mq_hw_ctx *hctx;
* or __blk_mq_run_hw_queue() to dispatch request.
* @rq is being submitted. Analyze what needs to be done and put it on the
* right queue.
- *
- * CONTEXT:
- * spin_lock_irq(q->queue_lock) in !mq case
*/
void blk_insert_flush(struct request *rq)
{
unsigned int policy = blk_flush_policy(fflags, rq);
struct blk_flush_queue *fq = blk_get_flush_queue(q, rq->mq_ctx);
+ if (!q->mq_ops)
+ lockdep_assert_held(q->queue_lock);
+
/*
* @policy now records what operations need to be done. Adjust
* REQ_PREFLUSH and FUA for the driver.
.sysfs_ops = &integrity_ops,
};
-static int blk_integrity_nop_fn(struct blk_integrity_iter *iter)
+static blk_status_t blk_integrity_nop_fn(struct blk_integrity_iter *iter)
{
- return 0;
+ return BLK_STS_OK;
}
static const struct blk_integrity_profile nop_profile = {
*/
int blk_rq_append_bio(struct request *rq, struct bio *bio)
{
+ blk_queue_bounce(rq->q, &bio);
+
if (!rq->bio) {
blk_rq_bio_prep(rq->q, rq, bio);
} else {
map_data->offset += bio->bi_iter.bi_size;
orig_bio = bio;
- blk_queue_bounce(q, &bio);
/*
* We link the bounce buffer in and could have to traverse it
* later so we have to get a ref to prevent it from being freed
*/
- bio_get(bio);
-
ret = blk_rq_append_bio(rq, bio);
+ bio_get(bio);
if (ret) {
bio_endio(bio);
__blk_rq_unmap_user(orig_bio);
return ret;
}
- blk_queue_bounce(q, &rq->bio);
return 0;
}
EXPORT_SYMBOL(blk_rq_map_kern);
bool do_split = true;
struct bio *new = NULL;
const unsigned max_sectors = get_max_io_size(q, bio);
- unsigned bvecs = 0;
bio_for_each_segment(bv, bio, iter) {
- /*
- * With arbitrary bio size, the incoming bio may be very
- * big. We have to split the bio into small bios so that
- * each holds at most BIO_MAX_PAGES bvecs because
- * bio_clone() can fail to allocate big bvecs.
- *
- * It should have been better to apply the limit per
- * request queue in which bio_clone() is involved,
- * instead of globally. The biggest blocker is the
- * bio_clone() in bio bounce.
- *
- * If bio is splitted by this reason, we should have
- * allowed to continue bios merging, but don't do
- * that now for making the change simple.
- *
- * TODO: deal with bio bounce's bio_clone() gracefully
- * and convert the global limit into per-queue limit.
- */
- if (bvecs++ >= BIO_MAX_PAGES)
- goto split;
-
/*
* If the queue doesn't support SG gaps and adding this
* offset would create a gap, disallow it.
return do_split ? new : NULL;
}
-void blk_queue_split(struct request_queue *q, struct bio **bio,
- struct bio_set *bs)
+void blk_queue_split(struct request_queue *q, struct bio **bio)
{
struct bio *split, *res;
unsigned nsegs;
switch (bio_op(*bio)) {
case REQ_OP_DISCARD:
case REQ_OP_SECURE_ERASE:
- split = blk_bio_discard_split(q, *bio, bs, &nsegs);
+ split = blk_bio_discard_split(q, *bio, q->bio_split, &nsegs);
break;
case REQ_OP_WRITE_ZEROES:
- split = blk_bio_write_zeroes_split(q, *bio, bs, &nsegs);
+ split = blk_bio_write_zeroes_split(q, *bio, q->bio_split, &nsegs);
break;
case REQ_OP_WRITE_SAME:
- split = blk_bio_write_same_split(q, *bio, bs, &nsegs);
+ split = blk_bio_write_same_split(q, *bio, q->bio_split, &nsegs);
break;
default:
split = blk_bio_segment_split(q, *bio, q->bio_split, &nsegs);
static struct request *attempt_merge(struct request_queue *q,
struct request *req, struct request *next)
{
+ if (!q->mq_ops)
+ lockdep_assert_held(q->queue_lock);
+
if (!rq_mergeable(req) || !rq_mergeable(next))
return NULL;
!blk_write_same_mergeable(req->bio, next->bio))
return NULL;
+ /*
+ * Don't allow merge of different write hints, or for a hint with
+ * non-hint IO.
+ */
+ if (req->write_hint != next->write_hint)
+ return NULL;
+
/*
* If we are allowed to merge, then append bio list
* from next to rq and release next. merge_requests_fn
!blk_write_same_mergeable(rq->bio, bio))
return false;
+ /*
+ * Don't allow merge of different write hints, or for a hint with
+ * non-hint IO.
+ */
+ if (rq->write_hint != bio->bi_write_hint)
+ return false;
+
return true;
}
#include "blk.h"
#include "blk-mq.h"
-static int cpu_to_queue_index(unsigned int nr_cpus, unsigned int nr_queues,
- const int cpu)
+static int cpu_to_queue_index(unsigned int nr_queues, const int cpu)
{
- return cpu * nr_queues / nr_cpus;
+ /*
+ * Non online CPU will be mapped to queue index 0.
+ */
+ if (!cpu_online(cpu))
+ return 0;
+ return cpu % nr_queues;
}
static int get_first_sibling(unsigned int cpu)
{
unsigned int *map = set->mq_map;
unsigned int nr_queues = set->nr_hw_queues;
- const struct cpumask *online_mask = cpu_online_mask;
- unsigned int i, nr_cpus, nr_uniq_cpus, queue, first_sibling;
- cpumask_var_t cpus;
-
- if (!alloc_cpumask_var(&cpus, GFP_ATOMIC))
- return -ENOMEM;
-
- cpumask_clear(cpus);
- nr_cpus = nr_uniq_cpus = 0;
- for_each_cpu(i, online_mask) {
- nr_cpus++;
- first_sibling = get_first_sibling(i);
- if (!cpumask_test_cpu(first_sibling, cpus))
- nr_uniq_cpus++;
- cpumask_set_cpu(i, cpus);
- }
-
- queue = 0;
- for_each_possible_cpu(i) {
- if (!cpumask_test_cpu(i, online_mask)) {
- map[i] = 0;
- continue;
- }
+ unsigned int cpu, first_sibling;
+ for_each_possible_cpu(cpu) {
/*
- * Easy case - we have equal or more hardware queues. Or
- * there are no thread siblings to take into account. Do
- * 1:1 if enough, or sequential mapping if less.
+ * First do sequential mapping between CPUs and queues.
+ * In case we still have CPUs to map, and we have some number of
+ * threads per cores then map sibling threads to the same queue for
+ * performace optimizations.
*/
- if (nr_queues >= nr_cpus || nr_cpus == nr_uniq_cpus) {
- map[i] = cpu_to_queue_index(nr_cpus, nr_queues, queue);
- queue++;
- continue;
+ if (cpu < nr_queues) {
+ map[cpu] = cpu_to_queue_index(nr_queues, cpu);
+ } else {
+ first_sibling = get_first_sibling(cpu);
+ if (first_sibling == cpu)
+ map[cpu] = cpu_to_queue_index(nr_queues, cpu);
+ else
+ map[cpu] = map[first_sibling];
}
-
- /*
- * Less then nr_cpus queues, and we have some number of
- * threads per cores. Map sibling threads to the same
- * queue.
- */
- first_sibling = get_first_sibling(i);
- if (first_sibling == i) {
- map[i] = cpu_to_queue_index(nr_uniq_cpus, nr_queues,
- queue);
- queue++;
- } else
- map[i] = map[first_sibling];
}
- free_cpumask_var(cpus);
return 0;
}
EXPORT_SYMBOL_GPL(blk_mq_map_queues);
blk_mq_run_hw_queues(q, true);
} else if (strcmp(op, "start") == 0) {
blk_mq_start_stopped_hw_queues(q, true);
+ } else if (strcmp(op, "kick") == 0) {
+ blk_mq_kick_requeue_list(q);
} else {
pr_err("%s: unsupported operation '%s'\n", __func__, op);
inval:
- pr_err("%s: use either 'run' or 'start'\n", __func__);
+ pr_err("%s: use 'run', 'start' or 'kick'\n", __func__);
return -EINVAL;
}
return count;
}
}
+static int queue_write_hint_show(void *data, struct seq_file *m)
+{
+ struct request_queue *q = data;
+ int i;
+
+ for (i = 0; i < BLK_MAX_WRITE_HINTS; i++)
+ seq_printf(m, "hint%d: %llu\n", i, q->write_hints[i]);
+
+ return 0;
+}
+
+static ssize_t queue_write_hint_store(void *data, const char __user *buf,
+ size_t count, loff_t *ppos)
+{
+ struct request_queue *q = data;
+ int i;
+
+ for (i = 0; i < BLK_MAX_WRITE_HINTS; i++)
+ q->write_hints[i] = 0;
+
+ return count;
+}
+
static int queue_poll_stat_show(void *data, struct seq_file *m)
{
struct request_queue *q = data;
};
#undef RQF_NAME
+#define RQAF_NAME(name) [REQ_ATOM_##name] = #name
+static const char *const rqaf_name[] = {
+ RQAF_NAME(COMPLETE),
+ RQAF_NAME(STARTED),
+ RQAF_NAME(POLL_SLEPT),
+};
+#undef RQAF_NAME
+
int __blk_mq_debugfs_rq_show(struct seq_file *m, struct request *rq)
{
const struct blk_mq_ops *const mq_ops = rq->q->mq_ops;
seq_puts(m, ", .rq_flags=");
blk_flags_show(m, (__force unsigned int)rq->rq_flags, rqf_name,
ARRAY_SIZE(rqf_name));
+ seq_puts(m, ", .atomic_flags=");
+ blk_flags_show(m, rq->atomic_flags, rqaf_name, ARRAY_SIZE(rqaf_name));
seq_printf(m, ", .tag=%d, .internal_tag=%d", rq->tag,
rq->internal_tag);
if (mq_ops->show_rq)
}
EXPORT_SYMBOL_GPL(blk_mq_debugfs_rq_show);
+static void *queue_requeue_list_start(struct seq_file *m, loff_t *pos)
+ __acquires(&q->requeue_lock)
+{
+ struct request_queue *q = m->private;
+
+ spin_lock_irq(&q->requeue_lock);
+ return seq_list_start(&q->requeue_list, *pos);
+}
+
+static void *queue_requeue_list_next(struct seq_file *m, void *v, loff_t *pos)
+{
+ struct request_queue *q = m->private;
+
+ return seq_list_next(v, &q->requeue_list, pos);
+}
+
+static void queue_requeue_list_stop(struct seq_file *m, void *v)
+ __releases(&q->requeue_lock)
+{
+ struct request_queue *q = m->private;
+
+ spin_unlock_irq(&q->requeue_lock);
+}
+
+static const struct seq_operations queue_requeue_list_seq_ops = {
+ .start = queue_requeue_list_start,
+ .next = queue_requeue_list_next,
+ .stop = queue_requeue_list_stop,
+ .show = blk_mq_debugfs_rq_show,
+};
+
static void *hctx_dispatch_start(struct seq_file *m, loff_t *pos)
__acquires(&hctx->lock)
{
.show = blk_mq_debugfs_rq_show,
};
+struct show_busy_params {
+ struct seq_file *m;
+ struct blk_mq_hw_ctx *hctx;
+};
+
+/*
+ * Note: the state of a request may change while this function is in progress,
+ * e.g. due to a concurrent blk_mq_finish_request() call.
+ */
+static void hctx_show_busy_rq(struct request *rq, void *data, bool reserved)
+{
+ const struct show_busy_params *params = data;
+
+ if (blk_mq_map_queue(rq->q, rq->mq_ctx->cpu) == params->hctx &&
+ test_bit(REQ_ATOM_STARTED, &rq->atomic_flags))
+ __blk_mq_debugfs_rq_show(params->m,
+ list_entry_rq(&rq->queuelist));
+}
+
+static int hctx_busy_show(void *data, struct seq_file *m)
+{
+ struct blk_mq_hw_ctx *hctx = data;
+ struct show_busy_params params = { .m = m, .hctx = hctx };
+
+ blk_mq_tagset_busy_iter(hctx->queue->tag_set, hctx_show_busy_rq,
+ ¶ms);
+
+ return 0;
+}
+
static int hctx_ctx_map_show(void *data, struct seq_file *m)
{
struct blk_mq_hw_ctx *hctx = data;
static const struct blk_mq_debugfs_attr blk_mq_debugfs_queue_attrs[] = {
{"poll_stat", 0400, queue_poll_stat_show},
+ {"requeue_list", 0400, .seq_ops = &queue_requeue_list_seq_ops},
{"state", 0600, queue_state_show, queue_state_write},
+ {"write_hints", 0600, queue_write_hint_show, queue_write_hint_store},
{},
};
{"state", 0400, hctx_state_show},
{"flags", 0400, hctx_flags_show},
{"dispatch", 0400, .seq_ops = &hctx_dispatch_seq_ops},
+ {"busy", 0400, hctx_busy_show},
{"ctx_map", 0400, hctx_ctx_map_show},
{"tags", 0400, hctx_tags_show},
{"tags_bitmap", 0400, hctx_tags_bitmap_show},
}
EXPORT_SYMBOL_GPL(blk_mq_sched_free_hctx_data);
-static void __blk_mq_sched_assign_ioc(struct request_queue *q,
- struct request *rq,
- struct bio *bio,
- struct io_context *ioc)
+void blk_mq_sched_assign_ioc(struct request *rq, struct bio *bio)
{
+ struct request_queue *q = rq->q;
+ struct io_context *ioc = rq_ioc(bio);
struct io_cq *icq;
spin_lock_irq(q->queue_lock);
if (!icq)
return;
}
-
+ get_io_context(icq->ioc);
rq->elv.icq = icq;
- if (!blk_mq_sched_get_rq_priv(q, rq, bio)) {
- rq->rq_flags |= RQF_ELVPRIV;
- get_io_context(icq->ioc);
- return;
- }
-
- rq->elv.icq = NULL;
}
-static void blk_mq_sched_assign_ioc(struct request_queue *q,
- struct request *rq, struct bio *bio)
+/*
+ * Mark a hardware queue as needing a restart. For shared queues, maintain
+ * a count of how many hardware queues are marked for restart.
+ */
+static void blk_mq_sched_mark_restart_hctx(struct blk_mq_hw_ctx *hctx)
{
- struct io_context *ioc;
+ if (test_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state))
+ return;
+
+ if (hctx->flags & BLK_MQ_F_TAG_SHARED) {
+ struct request_queue *q = hctx->queue;
- ioc = rq_ioc(bio);
- if (ioc)
- __blk_mq_sched_assign_ioc(q, rq, bio, ioc);
+ if (!test_and_set_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state))
+ atomic_inc(&q->shared_hctx_restart);
+ } else
+ set_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state);
}
-struct request *blk_mq_sched_get_request(struct request_queue *q,
- struct bio *bio,
- unsigned int op,
- struct blk_mq_alloc_data *data)
+static bool blk_mq_sched_restart_hctx(struct blk_mq_hw_ctx *hctx)
{
- struct elevator_queue *e = q->elevator;
- struct request *rq;
-
- blk_queue_enter_live(q);
- data->q = q;
- if (likely(!data->ctx))
- data->ctx = blk_mq_get_ctx(q);
- if (likely(!data->hctx))
- data->hctx = blk_mq_map_queue(q, data->ctx->cpu);
-
- if (e) {
- data->flags |= BLK_MQ_REQ_INTERNAL;
-
- /*
- * Flush requests are special and go directly to the
- * dispatch list.
- */
- if (!op_is_flush(op) && e->type->ops.mq.get_request) {
- rq = e->type->ops.mq.get_request(q, op, data);
- if (rq)
- rq->rq_flags |= RQF_QUEUED;
- } else
- rq = __blk_mq_alloc_request(data, op);
- } else {
- rq = __blk_mq_alloc_request(data, op);
- }
-
- if (rq) {
- if (!op_is_flush(op)) {
- rq->elv.icq = NULL;
- if (e && e->type->icq_cache)
- blk_mq_sched_assign_ioc(q, rq, bio);
- }
- data->hctx->queued++;
- return rq;
- }
+ if (!test_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state))
+ return false;
- blk_queue_exit(q);
- return NULL;
-}
+ if (hctx->flags & BLK_MQ_F_TAG_SHARED) {
+ struct request_queue *q = hctx->queue;
-void blk_mq_sched_put_request(struct request *rq)
-{
- struct request_queue *q = rq->q;
- struct elevator_queue *e = q->elevator;
+ if (test_and_clear_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state))
+ atomic_dec(&q->shared_hctx_restart);
+ } else
+ clear_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state);
- if (rq->rq_flags & RQF_ELVPRIV) {
- blk_mq_sched_put_rq_priv(rq->q, rq);
- if (rq->elv.icq) {
- put_io_context(rq->elv.icq->ioc);
- rq->elv.icq = NULL;
- }
+ if (blk_mq_hctx_has_pending(hctx)) {
+ blk_mq_run_hw_queue(hctx, true);
+ return true;
}
- if ((rq->rq_flags & RQF_QUEUED) && e && e->type->ops.mq.put_request)
- e->type->ops.mq.put_request(rq);
- else
- blk_mq_finish_request(rq);
+ return false;
}
void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx)
bool did_work = false;
LIST_HEAD(rq_list);
- if (unlikely(blk_mq_hctx_stopped(hctx)))
+ /* RCU or SRCU read lock is needed before checking quiesced flag */
+ if (unlikely(blk_mq_hctx_stopped(hctx) || blk_queue_quiesced(q)))
return;
hctx->run++;
}
EXPORT_SYMBOL_GPL(blk_mq_sched_try_merge);
+/*
+ * Reverse check our software queue for entries that we could potentially
+ * merge with. Currently includes a hand-wavy stop count of 8, to not spend
+ * too much time checking for merges.
+ */
+static bool blk_mq_attempt_merge(struct request_queue *q,
+ struct blk_mq_ctx *ctx, struct bio *bio)
+{
+ struct request *rq;
+ int checked = 8;
+
+ lockdep_assert_held(&ctx->lock);
+
+ list_for_each_entry_reverse(rq, &ctx->rq_list, queuelist) {
+ bool merged = false;
+
+ if (!checked--)
+ break;
+
+ if (!blk_rq_merge_ok(rq, bio))
+ continue;
+
+ switch (blk_try_merge(rq, bio)) {
+ case ELEVATOR_BACK_MERGE:
+ if (blk_mq_sched_allow_merge(q, rq, bio))
+ merged = bio_attempt_back_merge(q, rq, bio);
+ break;
+ case ELEVATOR_FRONT_MERGE:
+ if (blk_mq_sched_allow_merge(q, rq, bio))
+ merged = bio_attempt_front_merge(q, rq, bio);
+ break;
+ case ELEVATOR_DISCARD_MERGE:
+ merged = bio_attempt_discard_merge(q, rq, bio);
+ break;
+ default:
+ continue;
+ }
+
+ if (merged)
+ ctx->rq_merged++;
+ return merged;
+ }
+
+ return false;
+}
+
bool __blk_mq_sched_bio_merge(struct request_queue *q, struct bio *bio)
{
struct elevator_queue *e = q->elevator;
+ struct blk_mq_ctx *ctx = blk_mq_get_ctx(q);
+ struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(q, ctx->cpu);
+ bool ret = false;
- if (e->type->ops.mq.bio_merge) {
- struct blk_mq_ctx *ctx = blk_mq_get_ctx(q);
- struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(q, ctx->cpu);
-
+ if (e && e->type->ops.mq.bio_merge) {
blk_mq_put_ctx(ctx);
return e->type->ops.mq.bio_merge(hctx, bio);
}
- return false;
+ if (hctx->flags & BLK_MQ_F_SHOULD_MERGE) {
+ /* default per sw-queue merge */
+ spin_lock(&ctx->lock);
+ ret = blk_mq_attempt_merge(q, ctx, bio);
+ spin_unlock(&ctx->lock);
+ }
+
+ blk_mq_put_ctx(ctx);
+ return ret;
}
bool blk_mq_sched_try_insert_merge(struct request_queue *q, struct request *rq)
return true;
}
-static bool blk_mq_sched_restart_hctx(struct blk_mq_hw_ctx *hctx)
-{
- if (test_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state)) {
- clear_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state);
- if (blk_mq_hctx_has_pending(hctx)) {
- blk_mq_run_hw_queue(hctx, true);
- return true;
- }
- }
- return false;
-}
-
/**
* list_for_each_entry_rcu_rr - iterate in a round-robin fashion over rcu list
* @pos: loop cursor.
unsigned int i, j;
if (set->flags & BLK_MQ_F_TAG_SHARED) {
+ /*
+ * If this is 0, then we know that no hardware queues
+ * have RESTART marked. We're done.
+ */
+ if (!atomic_read(&queue->shared_hctx_restart))
+ return;
+
rcu_read_lock();
list_for_each_entry_rcu_rr(q, queue, &set->tag_list,
tag_set_list) {
void blk_mq_sched_free_hctx_data(struct request_queue *q,
void (*exit)(struct blk_mq_hw_ctx *));
-struct request *blk_mq_sched_get_request(struct request_queue *q, struct bio *bio, unsigned int op, struct blk_mq_alloc_data *data);
-void blk_mq_sched_put_request(struct request *rq);
+void blk_mq_sched_assign_ioc(struct request *rq, struct bio *bio);
void blk_mq_sched_request_inserted(struct request *rq);
bool blk_mq_sched_try_merge(struct request_queue *q, struct bio *bio,
static inline bool
blk_mq_sched_bio_merge(struct request_queue *q, struct bio *bio)
{
- struct elevator_queue *e = q->elevator;
-
- if (!e || blk_queue_nomerges(q) || !bio_mergeable(bio))
+ if (blk_queue_nomerges(q) || !bio_mergeable(bio))
return false;
return __blk_mq_sched_bio_merge(q, bio);
}
-static inline int blk_mq_sched_get_rq_priv(struct request_queue *q,
- struct request *rq,
- struct bio *bio)
-{
- struct elevator_queue *e = q->elevator;
-
- if (e && e->type->ops.mq.get_rq_priv)
- return e->type->ops.mq.get_rq_priv(q, rq, bio);
-
- return 0;
-}
-
-static inline void blk_mq_sched_put_rq_priv(struct request_queue *q,
- struct request *rq)
-{
- struct elevator_queue *e = q->elevator;
-
- if (e && e->type->ops.mq.put_rq_priv)
- e->type->ops.mq.put_rq_priv(q, rq);
-}
-
static inline bool
blk_mq_sched_allow_merge(struct request_queue *q, struct request *rq,
struct bio *bio)
return false;
}
-/*
- * Mark a hardware queue as needing a restart.
- */
-static inline void blk_mq_sched_mark_restart_hctx(struct blk_mq_hw_ctx *hctx)
-{
- if (!test_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state))
- set_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state);
-}
-
static inline bool blk_mq_sched_needs_restart(struct blk_mq_hw_ctx *hctx)
{
return test_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state);
#include "blk-wbt.h"
#include "blk-mq-sched.h"
-static DEFINE_MUTEX(all_q_mutex);
-static LIST_HEAD(all_q_list);
-
static void blk_mq_poll_stats_start(struct request_queue *q);
static void blk_mq_poll_stats_fn(struct blk_stat_callback *cb);
-static void __blk_mq_stop_hw_queues(struct request_queue *q, bool sync);
static int blk_mq_poll_stats_bkt(const struct request *rq)
{
}
EXPORT_SYMBOL_GPL(blk_mq_unfreeze_queue);
+/*
+ * FIXME: replace the scsi_internal_device_*block_nowait() calls in the
+ * mpt3sas driver such that this function can be removed.
+ */
+void blk_mq_quiesce_queue_nowait(struct request_queue *q)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(q->queue_lock, flags);
+ queue_flag_set(QUEUE_FLAG_QUIESCED, q);
+ spin_unlock_irqrestore(q->queue_lock, flags);
+}
+EXPORT_SYMBOL_GPL(blk_mq_quiesce_queue_nowait);
+
/**
- * blk_mq_quiesce_queue() - wait until all ongoing queue_rq calls have finished
+ * blk_mq_quiesce_queue() - wait until all ongoing dispatches have finished
* @q: request queue.
*
* Note: this function does not prevent that the struct request end_io()
- * callback function is invoked. Additionally, it is not prevented that
- * new queue_rq() calls occur unless the queue has been stopped first.
+ * callback function is invoked. Once this function is returned, we make
+ * sure no dispatch can happen until the queue is unquiesced via
+ * blk_mq_unquiesce_queue().
*/
void blk_mq_quiesce_queue(struct request_queue *q)
{
unsigned int i;
bool rcu = false;
- __blk_mq_stop_hw_queues(q, true);
+ blk_mq_quiesce_queue_nowait(q);
queue_for_each_hw_ctx(q, hctx, i) {
if (hctx->flags & BLK_MQ_F_BLOCKING)
- synchronize_srcu(&hctx->queue_rq_srcu);
+ synchronize_srcu(hctx->queue_rq_srcu);
else
rcu = true;
}
}
EXPORT_SYMBOL_GPL(blk_mq_quiesce_queue);
+/*
+ * blk_mq_unquiesce_queue() - counterpart of blk_mq_quiesce_queue()
+ * @q: request queue.
+ *
+ * This function recovers queue into the state before quiescing
+ * which is done by blk_mq_quiesce_queue.
+ */
+void blk_mq_unquiesce_queue(struct request_queue *q)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(q->queue_lock, flags);
+ queue_flag_clear(QUEUE_FLAG_QUIESCED, q);
+ spin_unlock_irqrestore(q->queue_lock, flags);
+
+ /* dispatch requests which are inserted during quiescing */
+ blk_mq_run_hw_queues(q, true);
+}
+EXPORT_SYMBOL_GPL(blk_mq_unquiesce_queue);
+
void blk_mq_wake_waiters(struct request_queue *q)
{
struct blk_mq_hw_ctx *hctx;
}
EXPORT_SYMBOL(blk_mq_can_queue);
-void blk_mq_rq_ctx_init(struct request_queue *q, struct blk_mq_ctx *ctx,
- struct request *rq, unsigned int op)
+static struct request *blk_mq_rq_ctx_init(struct blk_mq_alloc_data *data,
+ unsigned int tag, unsigned int op)
{
+ struct blk_mq_tags *tags = blk_mq_tags_from_data(data);
+ struct request *rq = tags->static_rqs[tag];
+
+ rq->rq_flags = 0;
+
+ if (data->flags & BLK_MQ_REQ_INTERNAL) {
+ rq->tag = -1;
+ rq->internal_tag = tag;
+ } else {
+ if (blk_mq_tag_busy(data->hctx)) {
+ rq->rq_flags = RQF_MQ_INFLIGHT;
+ atomic_inc(&data->hctx->nr_active);
+ }
+ rq->tag = tag;
+ rq->internal_tag = -1;
+ data->hctx->tags->rqs[rq->tag] = rq;
+ }
+
INIT_LIST_HEAD(&rq->queuelist);
/* csd/requeue_work/fifo_time is initialized before use */
- rq->q = q;
- rq->mq_ctx = ctx;
+ rq->q = data->q;
+ rq->mq_ctx = data->ctx;
rq->cmd_flags = op;
- if (blk_queue_io_stat(q))
+ if (blk_queue_io_stat(data->q))
rq->rq_flags |= RQF_IO_STAT;
/* do not touch atomic flags, it needs atomic ops against the timer */
rq->cpu = -1;
rq->end_io_data = NULL;
rq->next_rq = NULL;
- ctx->rq_dispatched[op_is_sync(op)]++;
+ data->ctx->rq_dispatched[op_is_sync(op)]++;
+ return rq;
}
-EXPORT_SYMBOL_GPL(blk_mq_rq_ctx_init);
-struct request *__blk_mq_alloc_request(struct blk_mq_alloc_data *data,
- unsigned int op)
+static struct request *blk_mq_get_request(struct request_queue *q,
+ struct bio *bio, unsigned int op,
+ struct blk_mq_alloc_data *data)
{
+ struct elevator_queue *e = q->elevator;
struct request *rq;
unsigned int tag;
- tag = blk_mq_get_tag(data);
- if (tag != BLK_MQ_TAG_FAIL) {
- struct blk_mq_tags *tags = blk_mq_tags_from_data(data);
+ blk_queue_enter_live(q);
+ data->q = q;
+ if (likely(!data->ctx))
+ data->ctx = blk_mq_get_ctx(q);
+ if (likely(!data->hctx))
+ data->hctx = blk_mq_map_queue(q, data->ctx->cpu);
+ if (op & REQ_NOWAIT)
+ data->flags |= BLK_MQ_REQ_NOWAIT;
- rq = tags->static_rqs[tag];
+ if (e) {
+ data->flags |= BLK_MQ_REQ_INTERNAL;
- if (data->flags & BLK_MQ_REQ_INTERNAL) {
- rq->tag = -1;
- rq->internal_tag = tag;
- } else {
- if (blk_mq_tag_busy(data->hctx)) {
- rq->rq_flags = RQF_MQ_INFLIGHT;
- atomic_inc(&data->hctx->nr_active);
- }
- rq->tag = tag;
- rq->internal_tag = -1;
- data->hctx->tags->rqs[rq->tag] = rq;
- }
+ /*
+ * Flush requests are special and go directly to the
+ * dispatch list.
+ */
+ if (!op_is_flush(op) && e->type->ops.mq.limit_depth)
+ e->type->ops.mq.limit_depth(op, data);
+ }
- blk_mq_rq_ctx_init(data->q, data->ctx, rq, op);
- return rq;
+ tag = blk_mq_get_tag(data);
+ if (tag == BLK_MQ_TAG_FAIL) {
+ blk_queue_exit(q);
+ return NULL;
}
- return NULL;
+ rq = blk_mq_rq_ctx_init(data, tag, op);
+ if (!op_is_flush(op)) {
+ rq->elv.icq = NULL;
+ if (e && e->type->ops.mq.prepare_request) {
+ if (e->type->icq_cache && rq_ioc(bio))
+ blk_mq_sched_assign_ioc(rq, bio);
+
+ e->type->ops.mq.prepare_request(rq, bio);
+ rq->rq_flags |= RQF_ELVPRIV;
+ }
+ }
+ data->hctx->queued++;
+ return rq;
}
-EXPORT_SYMBOL_GPL(__blk_mq_alloc_request);
-struct request *blk_mq_alloc_request(struct request_queue *q, int rw,
+struct request *blk_mq_alloc_request(struct request_queue *q, unsigned int op,
unsigned int flags)
{
struct blk_mq_alloc_data alloc_data = { .flags = flags };
if (ret)
return ERR_PTR(ret);
- rq = blk_mq_sched_get_request(q, NULL, rw, &alloc_data);
+ rq = blk_mq_get_request(q, NULL, op, &alloc_data);
blk_mq_put_ctx(alloc_data.ctx);
blk_queue_exit(q);
}
EXPORT_SYMBOL(blk_mq_alloc_request);
-struct request *blk_mq_alloc_request_hctx(struct request_queue *q, int rw,
- unsigned int flags, unsigned int hctx_idx)
+struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
+ unsigned int op, unsigned int flags, unsigned int hctx_idx)
{
struct blk_mq_alloc_data alloc_data = { .flags = flags };
struct request *rq;
cpu = cpumask_first(alloc_data.hctx->cpumask);
alloc_data.ctx = __blk_mq_get_ctx(q, cpu);
- rq = blk_mq_sched_get_request(q, NULL, rw, &alloc_data);
+ rq = blk_mq_get_request(q, NULL, op, &alloc_data);
blk_queue_exit(q);
}
EXPORT_SYMBOL_GPL(blk_mq_alloc_request_hctx);
-void __blk_mq_finish_request(struct blk_mq_hw_ctx *hctx, struct blk_mq_ctx *ctx,
- struct request *rq)
+void blk_mq_free_request(struct request *rq)
{
- const int sched_tag = rq->internal_tag;
struct request_queue *q = rq->q;
+ struct elevator_queue *e = q->elevator;
+ struct blk_mq_ctx *ctx = rq->mq_ctx;
+ struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(q, ctx->cpu);
+ const int sched_tag = rq->internal_tag;
+
+ if (rq->rq_flags & RQF_ELVPRIV) {
+ if (e && e->type->ops.mq.finish_request)
+ e->type->ops.mq.finish_request(rq);
+ if (rq->elv.icq) {
+ put_io_context(rq->elv.icq->ioc);
+ rq->elv.icq = NULL;
+ }
+ }
+ ctx->rq_completed[rq_is_sync(rq)]++;
if (rq->rq_flags & RQF_MQ_INFLIGHT)
atomic_dec(&hctx->nr_active);
wbt_done(q->rq_wb, &rq->issue_stat);
- rq->rq_flags = 0;
clear_bit(REQ_ATOM_STARTED, &rq->atomic_flags);
clear_bit(REQ_ATOM_POLL_SLEPT, &rq->atomic_flags);
blk_mq_sched_restart(hctx);
blk_queue_exit(q);
}
-
-static void blk_mq_finish_hctx_request(struct blk_mq_hw_ctx *hctx,
- struct request *rq)
-{
- struct blk_mq_ctx *ctx = rq->mq_ctx;
-
- ctx->rq_completed[rq_is_sync(rq)]++;
- __blk_mq_finish_request(hctx, ctx, rq);
-}
-
-void blk_mq_finish_request(struct request *rq)
-{
- blk_mq_finish_hctx_request(blk_mq_map_queue(rq->q, rq->mq_ctx->cpu), rq);
-}
-EXPORT_SYMBOL_GPL(blk_mq_finish_request);
-
-void blk_mq_free_request(struct request *rq)
-{
- blk_mq_sched_put_request(rq);
-}
EXPORT_SYMBOL_GPL(blk_mq_free_request);
-inline void __blk_mq_end_request(struct request *rq, int error)
+inline void __blk_mq_end_request(struct request *rq, blk_status_t error)
{
blk_account_io_done(rq);
}
EXPORT_SYMBOL(__blk_mq_end_request);
-void blk_mq_end_request(struct request *rq, int error)
+void blk_mq_end_request(struct request *rq, blk_status_t error)
{
if (blk_update_request(rq, error, blk_rq_bytes(rq)))
BUG();
blk_queue_exit(q);
}
-/*
- * Reverse check our software queue for entries that we could potentially
- * merge with. Currently includes a hand-wavy stop count of 8, to not spend
- * too much time checking for merges.
- */
-static bool blk_mq_attempt_merge(struct request_queue *q,
- struct blk_mq_ctx *ctx, struct bio *bio)
-{
- struct request *rq;
- int checked = 8;
-
- list_for_each_entry_reverse(rq, &ctx->rq_list, queuelist) {
- bool merged = false;
-
- if (!checked--)
- break;
-
- if (!blk_rq_merge_ok(rq, bio))
- continue;
-
- switch (blk_try_merge(rq, bio)) {
- case ELEVATOR_BACK_MERGE:
- if (blk_mq_sched_allow_merge(q, rq, bio))
- merged = bio_attempt_back_merge(q, rq, bio);
- break;
- case ELEVATOR_FRONT_MERGE:
- if (blk_mq_sched_allow_merge(q, rq, bio))
- merged = bio_attempt_front_merge(q, rq, bio);
- break;
- case ELEVATOR_DISCARD_MERGE:
- merged = bio_attempt_discard_merge(q, rq, bio);
- break;
- default:
- continue;
- }
-
- if (merged)
- ctx->rq_merged++;
- return merged;
- }
-
- return false;
-}
-
struct flush_busy_ctx_data {
struct blk_mq_hw_ctx *hctx;
struct list_head *list;
return first != NULL;
}
-static int blk_mq_dispatch_wake(wait_queue_t *wait, unsigned mode, int flags,
+static int blk_mq_dispatch_wake(wait_queue_entry_t *wait, unsigned mode, int flags,
void *key)
{
struct blk_mq_hw_ctx *hctx;
hctx = container_of(wait, struct blk_mq_hw_ctx, dispatch_wait);
- list_del(&wait->task_list);
+ list_del(&wait->entry);
clear_bit_unlock(BLK_MQ_S_TAG_WAITING, &hctx->state);
blk_mq_run_hw_queue(hctx, true);
return 1;
{
struct blk_mq_hw_ctx *hctx;
struct request *rq;
- int errors, queued, ret = BLK_MQ_RQ_QUEUE_OK;
+ int errors, queued;
if (list_empty(list))
return false;
errors = queued = 0;
do {
struct blk_mq_queue_data bd;
+ blk_status_t ret;
rq = list_first_entry(list, struct request, queuelist);
if (!blk_mq_get_driver_tag(rq, &hctx, false)) {
}
ret = q->mq_ops->queue_rq(hctx, &bd);
- switch (ret) {
- case BLK_MQ_RQ_QUEUE_OK:
- queued++;
- break;
- case BLK_MQ_RQ_QUEUE_BUSY:
+ if (ret == BLK_STS_RESOURCE) {
blk_mq_put_driver_tag_hctx(hctx, rq);
list_add(&rq->queuelist, list);
__blk_mq_requeue_request(rq);
break;
- default:
- pr_err("blk-mq: bad return on queue: %d\n", ret);
- case BLK_MQ_RQ_QUEUE_ERROR:
+ }
+
+ if (unlikely(ret != BLK_STS_OK)) {
errors++;
- blk_mq_end_request(rq, -EIO);
- break;
+ blk_mq_end_request(rq, BLK_STS_IOERR);
+ continue;
}
- if (ret == BLK_MQ_RQ_QUEUE_BUSY)
- break;
+ queued++;
} while (!list_empty(list));
hctx->dispatched[queued_to_index(queued)]++;
* - blk_mq_run_hw_queue() checks whether or not a queue has
* been stopped before rerunning a queue.
* - Some but not all block drivers stop a queue before
- * returning BLK_MQ_RQ_QUEUE_BUSY. Two exceptions are scsi-mq
+ * returning BLK_STS_RESOURCE. Two exceptions are scsi-mq
* and dm-rq.
*/
if (!blk_mq_sched_needs_restart(hctx) &&
} else {
might_sleep();
- srcu_idx = srcu_read_lock(&hctx->queue_rq_srcu);
+ srcu_idx = srcu_read_lock(hctx->queue_rq_srcu);
blk_mq_sched_dispatch_requests(hctx);
- srcu_read_unlock(&hctx->queue_rq_srcu, srcu_idx);
+ srcu_read_unlock(hctx->queue_rq_srcu, srcu_idx);
}
}
static void __blk_mq_delay_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async,
unsigned long msecs)
{
- if (unlikely(blk_mq_hctx_stopped(hctx) ||
- !blk_mq_hw_queue_mapped(hctx)))
+ if (WARN_ON_ONCE(!blk_mq_hw_queue_mapped(hctx)))
+ return;
+
+ if (unlikely(blk_mq_hctx_stopped(hctx)))
return;
if (!async && !(hctx->flags & BLK_MQ_F_BLOCKING)) {
}
EXPORT_SYMBOL(blk_mq_queue_stopped);
-static void __blk_mq_stop_hw_queue(struct blk_mq_hw_ctx *hctx, bool sync)
+/*
+ * This function is often used for pausing .queue_rq() by driver when
+ * there isn't enough resource or some conditions aren't satisfied, and
+ * BLK_MQ_RQ_QUEUE_BUSY is usually returned.
+ *
+ * We do not guarantee that dispatch can be drained or blocked
+ * after blk_mq_stop_hw_queue() returns. Please use
+ * blk_mq_quiesce_queue() for that requirement.
+ */
+void blk_mq_stop_hw_queue(struct blk_mq_hw_ctx *hctx)
{
- if (sync)
- cancel_delayed_work_sync(&hctx->run_work);
- else
- cancel_delayed_work(&hctx->run_work);
+ cancel_delayed_work(&hctx->run_work);
set_bit(BLK_MQ_S_STOPPED, &hctx->state);
}
-
-void blk_mq_stop_hw_queue(struct blk_mq_hw_ctx *hctx)
-{
- __blk_mq_stop_hw_queue(hctx, false);
-}
EXPORT_SYMBOL(blk_mq_stop_hw_queue);
-static void __blk_mq_stop_hw_queues(struct request_queue *q, bool sync)
+/*
+ * This function is often used for pausing .queue_rq() by driver when
+ * there isn't enough resource or some conditions aren't satisfied, and
+ * BLK_MQ_RQ_QUEUE_BUSY is usually returned.
+ *
+ * We do not guarantee that dispatch can be drained or blocked
+ * after blk_mq_stop_hw_queues() returns. Please use
+ * blk_mq_quiesce_queue() for that requirement.
+ */
+void blk_mq_stop_hw_queues(struct request_queue *q)
{
struct blk_mq_hw_ctx *hctx;
int i;
queue_for_each_hw_ctx(q, hctx, i)
- __blk_mq_stop_hw_queue(hctx, sync);
-}
-
-void blk_mq_stop_hw_queues(struct request_queue *q)
-{
- __blk_mq_stop_hw_queues(q, false);
+ blk_mq_stop_hw_queue(hctx);
}
EXPORT_SYMBOL(blk_mq_stop_hw_queues);
void blk_mq_delay_queue(struct blk_mq_hw_ctx *hctx, unsigned long msecs)
{
- if (unlikely(!blk_mq_hw_queue_mapped(hctx)))
+ if (WARN_ON_ONCE(!blk_mq_hw_queue_mapped(hctx)))
return;
/*
{
struct blk_mq_ctx *ctx = rq->mq_ctx;
+ lockdep_assert_held(&ctx->lock);
+
trace_block_rq_insert(hctx->queue, rq);
if (at_head)
{
struct blk_mq_ctx *ctx = rq->mq_ctx;
+ lockdep_assert_held(&ctx->lock);
+
__blk_mq_insert_req_list(hctx, rq, at_head);
blk_mq_hctx_mark_pending(hctx, ctx);
}
!blk_queue_nomerges(hctx->queue);
}
-static inline bool blk_mq_merge_queue_io(struct blk_mq_hw_ctx *hctx,
- struct blk_mq_ctx *ctx,
- struct request *rq, struct bio *bio)
+static inline void blk_mq_queue_io(struct blk_mq_hw_ctx *hctx,
+ struct blk_mq_ctx *ctx,
+ struct request *rq)
{
- if (!hctx_allow_merges(hctx) || !bio_mergeable(bio)) {
- blk_mq_bio_to_request(rq, bio);
- spin_lock(&ctx->lock);
-insert_rq:
- __blk_mq_insert_request(hctx, rq, false);
- spin_unlock(&ctx->lock);
- return false;
- } else {
- struct request_queue *q = hctx->queue;
-
- spin_lock(&ctx->lock);
- if (!blk_mq_attempt_merge(q, ctx, bio)) {
- blk_mq_bio_to_request(rq, bio);
- goto insert_rq;
- }
-
- spin_unlock(&ctx->lock);
- __blk_mq_finish_request(hctx, ctx, rq);
- return true;
- }
+ spin_lock(&ctx->lock);
+ __blk_mq_insert_request(hctx, rq, false);
+ spin_unlock(&ctx->lock);
}
static blk_qc_t request_to_qc_t(struct blk_mq_hw_ctx *hctx, struct request *rq)
.last = true,
};
blk_qc_t new_cookie;
- int ret;
+ blk_status_t ret;
bool run_queue = true;
- if (blk_mq_hctx_stopped(hctx)) {
+ /* RCU or SRCU read lock is needed before checking quiesced flag */
+ if (blk_mq_hctx_stopped(hctx) || blk_queue_quiesced(q)) {
run_queue = false;
goto insert;
}
* would have done
*/
ret = q->mq_ops->queue_rq(hctx, &bd);
- if (ret == BLK_MQ_RQ_QUEUE_OK) {
+ switch (ret) {
+ case BLK_STS_OK:
*cookie = new_cookie;
return;
- }
-
- if (ret == BLK_MQ_RQ_QUEUE_ERROR) {
+ case BLK_STS_RESOURCE:
+ __blk_mq_requeue_request(rq);
+ goto insert;
+ default:
*cookie = BLK_QC_T_NONE;
- blk_mq_end_request(rq, -EIO);
+ blk_mq_end_request(rq, ret);
return;
}
- __blk_mq_requeue_request(rq);
insert:
blk_mq_sched_insert_request(rq, false, run_queue, false, may_sleep);
}
might_sleep();
- srcu_idx = srcu_read_lock(&hctx->queue_rq_srcu);
+ srcu_idx = srcu_read_lock(hctx->queue_rq_srcu);
__blk_mq_try_issue_directly(hctx, rq, cookie, true);
- srcu_read_unlock(&hctx->queue_rq_srcu, srcu_idx);
+ srcu_read_unlock(hctx->queue_rq_srcu, srcu_idx);
}
}
blk_queue_bounce(q, &bio);
- blk_queue_split(q, &bio, q->bio_split);
+ blk_queue_split(q, &bio);
if (bio_integrity_enabled(bio) && bio_integrity_prep(bio)) {
bio_io_error(bio);
trace_block_getrq(q, bio, bio->bi_opf);
- rq = blk_mq_sched_get_request(q, bio, bio->bi_opf, &data);
+ rq = blk_mq_get_request(q, bio, bio->bi_opf, &data);
if (unlikely(!rq)) {
__wbt_done(q->rq_wb, wb_acct);
+ if (bio->bi_opf & REQ_NOWAIT)
+ bio_wouldblock_error(bio);
return BLK_QC_T_NONE;
}
blk_mq_put_ctx(data.ctx);
blk_mq_bio_to_request(rq, bio);
blk_mq_sched_insert_request(rq, false, true, true, true);
- } else if (!blk_mq_merge_queue_io(data.hctx, data.ctx, rq, bio)) {
+ } else {
blk_mq_put_ctx(data.ctx);
+ blk_mq_bio_to_request(rq, bio);
+ blk_mq_queue_io(data.hctx, data.ctx, rq);
blk_mq_run_hw_queue(data.hctx, true);
- } else
- blk_mq_put_ctx(data.ctx);
+ }
return cookie;
}
set->ops->exit_hctx(hctx, hctx_idx);
if (hctx->flags & BLK_MQ_F_BLOCKING)
- cleanup_srcu_struct(&hctx->queue_rq_srcu);
+ cleanup_srcu_struct(hctx->queue_rq_srcu);
blk_mq_remove_cpuhp(hctx);
blk_free_flush_queue(hctx->fq);
spin_lock_init(&hctx->lock);
INIT_LIST_HEAD(&hctx->dispatch);
hctx->queue = q;
- hctx->queue_num = hctx_idx;
hctx->flags = set->flags & ~BLK_MQ_F_TAG_SHARED;
cpuhp_state_add_instance_nocalls(CPUHP_BLK_MQ_DEAD, &hctx->cpuhp_dead);
goto free_fq;
if (hctx->flags & BLK_MQ_F_BLOCKING)
- init_srcu_struct(&hctx->queue_rq_srcu);
+ init_srcu_struct(hctx->queue_rq_srcu);
blk_mq_debugfs_register_hctx(q, hctx);
INIT_LIST_HEAD(&__ctx->rq_list);
__ctx->queue = q;
- /* If the cpu isn't online, the cpu is mapped to first hctx */
- if (!cpu_online(i))
+ /* If the cpu isn't present, the cpu is mapped to first hctx */
+ if (!cpu_present(i))
continue;
hctx = blk_mq_map_queue(q, i);
}
}
-static void blk_mq_map_swqueue(struct request_queue *q,
- const struct cpumask *online_mask)
+static void blk_mq_map_swqueue(struct request_queue *q)
{
unsigned int i, hctx_idx;
struct blk_mq_hw_ctx *hctx;
}
/*
- * Map software to hardware queues
+ * Map software to hardware queues.
+ *
+ * If the cpu isn't present, the cpu is mapped to first hctx.
*/
- for_each_possible_cpu(i) {
- /* If the cpu isn't online, the cpu is mapped to first hctx */
- if (!cpumask_test_cpu(i, online_mask))
- continue;
-
+ for_each_present_cpu(i) {
hctx_idx = q->mq_map[i];
/* unmapped hw queue can be remapped after CPU topo changed */
if (!set->tags[hctx_idx] &&
}
}
+/*
+ * Caller needs to ensure that we're either frozen/quiesced, or that
+ * the queue isn't live yet.
+ */
static void queue_set_hctx_shared(struct request_queue *q, bool shared)
{
struct blk_mq_hw_ctx *hctx;
int i;
queue_for_each_hw_ctx(q, hctx, i) {
- if (shared)
+ if (shared) {
+ if (test_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state))
+ atomic_inc(&q->shared_hctx_restart);
hctx->flags |= BLK_MQ_F_TAG_SHARED;
- else
+ } else {
+ if (test_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state))
+ atomic_dec(&q->shared_hctx_restart);
hctx->flags &= ~BLK_MQ_F_TAG_SHARED;
+ }
}
}
-static void blk_mq_update_tag_set_depth(struct blk_mq_tag_set *set, bool shared)
+static void blk_mq_update_tag_set_depth(struct blk_mq_tag_set *set,
+ bool shared)
{
struct request_queue *q;
}
EXPORT_SYMBOL(blk_mq_init_queue);
+static int blk_mq_hw_ctx_size(struct blk_mq_tag_set *tag_set)
+{
+ int hw_ctx_size = sizeof(struct blk_mq_hw_ctx);
+
+ BUILD_BUG_ON(ALIGN(offsetof(struct blk_mq_hw_ctx, queue_rq_srcu),
+ __alignof__(struct blk_mq_hw_ctx)) !=
+ sizeof(struct blk_mq_hw_ctx));
+
+ if (tag_set->flags & BLK_MQ_F_BLOCKING)
+ hw_ctx_size += sizeof(struct srcu_struct);
+
+ return hw_ctx_size;
+}
+
static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
struct request_queue *q)
{
continue;
node = blk_mq_hw_queue_to_node(q->mq_map, i);
- hctxs[i] = kzalloc_node(sizeof(struct blk_mq_hw_ctx),
+ hctxs[i] = kzalloc_node(blk_mq_hw_ctx_size(set),
GFP_KERNEL, node);
if (!hctxs[i])
break;
blk_queue_softirq_done(q, set->ops->complete);
blk_mq_init_cpu_queues(q, set->nr_hw_queues);
-
- get_online_cpus();
- mutex_lock(&all_q_mutex);
-
- list_add_tail(&q->all_q_node, &all_q_list);
blk_mq_add_queue_tag_set(set, q);
- blk_mq_map_swqueue(q, cpu_online_mask);
-
- mutex_unlock(&all_q_mutex);
- put_online_cpus();
+ blk_mq_map_swqueue(q);
if (!(set->flags & BLK_MQ_F_NO_SCHED)) {
int ret;
{
struct blk_mq_tag_set *set = q->tag_set;
- mutex_lock(&all_q_mutex);
- list_del_init(&q->all_q_node);
- mutex_unlock(&all_q_mutex);
-
blk_mq_del_queue_tag_set(q);
-
blk_mq_exit_hw_queues(q, set, set->nr_hw_queues);
}
/* Basically redo blk_mq_init_queue with queue frozen */
-static void blk_mq_queue_reinit(struct request_queue *q,
- const struct cpumask *online_mask)
+static void blk_mq_queue_reinit(struct request_queue *q)
{
WARN_ON_ONCE(!atomic_read(&q->mq_freeze_depth));
* involves free and re-allocate memory, worthy doing?)
*/
- blk_mq_map_swqueue(q, online_mask);
+ blk_mq_map_swqueue(q);
blk_mq_sysfs_register(q);
blk_mq_debugfs_register_hctxs(q);
}
-/*
- * New online cpumask which is going to be set in this hotplug event.
- * Declare this cpumasks as global as cpu-hotplug operation is invoked
- * one-by-one and dynamically allocating this could result in a failure.
- */
-static struct cpumask cpuhp_online_new;
-
-static void blk_mq_queue_reinit_work(void)
-{
- struct request_queue *q;
-
- mutex_lock(&all_q_mutex);
- /*
- * We need to freeze and reinit all existing queues. Freezing
- * involves synchronous wait for an RCU grace period and doing it
- * one by one may take a long time. Start freezing all queues in
- * one swoop and then wait for the completions so that freezing can
- * take place in parallel.
- */
- list_for_each_entry(q, &all_q_list, all_q_node)
- blk_freeze_queue_start(q);
- list_for_each_entry(q, &all_q_list, all_q_node)
- blk_mq_freeze_queue_wait(q);
-
- list_for_each_entry(q, &all_q_list, all_q_node)
- blk_mq_queue_reinit(q, &cpuhp_online_new);
-
- list_for_each_entry(q, &all_q_list, all_q_node)
- blk_mq_unfreeze_queue(q);
-
- mutex_unlock(&all_q_mutex);
-}
-
-static int blk_mq_queue_reinit_dead(unsigned int cpu)
-{
- cpumask_copy(&cpuhp_online_new, cpu_online_mask);
- blk_mq_queue_reinit_work();
- return 0;
-}
-
-/*
- * Before hotadded cpu starts handling requests, new mappings must be
- * established. Otherwise, these requests in hw queue might never be
- * dispatched.
- *
- * For example, there is a single hw queue (hctx) and two CPU queues (ctx0
- * for CPU0, and ctx1 for CPU1).
- *
- * Now CPU1 is just onlined and a request is inserted into ctx1->rq_list
- * and set bit0 in pending bitmap as ctx1->index_hw is still zero.
- *
- * And then while running hw queue, blk_mq_flush_busy_ctxs() finds bit0 is set
- * in pending bitmap and tries to retrieve requests in hctx->ctxs[0]->rq_list.
- * But htx->ctxs[0] is a pointer to ctx0, so the request in ctx1->rq_list is
- * ignored.
- */
-static int blk_mq_queue_reinit_prepare(unsigned int cpu)
-{
- cpumask_copy(&cpuhp_online_new, cpu_online_mask);
- cpumask_set_cpu(cpu, &cpuhp_online_new);
- blk_mq_queue_reinit_work();
- return 0;
-}
-
static int __blk_mq_alloc_rq_maps(struct blk_mq_tag_set *set)
{
int i;
blk_mq_update_queue_map(set);
list_for_each_entry(q, &set->tag_list, tag_set_list) {
blk_mq_realloc_hw_ctxs(set, q);
- blk_mq_queue_reinit(q, cpu_online_mask);
+ blk_mq_queue_reinit(q);
}
list_for_each_entry(q, &set->tag_list, tag_set_list)
}
EXPORT_SYMBOL_GPL(blk_mq_poll);
-void blk_mq_disable_hotplug(void)
-{
- mutex_lock(&all_q_mutex);
-}
-
-void blk_mq_enable_hotplug(void)
-{
- mutex_unlock(&all_q_mutex);
-}
-
static int __init blk_mq_init(void)
{
cpuhp_setup_state_multi(CPUHP_BLK_MQ_DEAD, "block/mq:dead", NULL,
blk_mq_hctx_notify_dead);
-
- cpuhp_setup_state_nocalls(CPUHP_BLK_MQ_PREPARE, "block/mq:prepare",
- blk_mq_queue_reinit_prepare,
- blk_mq_queue_reinit_dead);
return 0;
}
subsys_initcall(blk_mq_init);
bool at_head);
void blk_mq_insert_requests(struct blk_mq_hw_ctx *hctx, struct blk_mq_ctx *ctx,
struct list_head *list);
-/*
- * CPU hotplug helpers
- */
-void blk_mq_enable_hotplug(void);
-void blk_mq_disable_hotplug(void);
/*
* CPU -> queue mappings
return data->hctx->tags;
}
-/*
- * Internal helpers for request allocation/init/free
- */
-void blk_mq_rq_ctx_init(struct request_queue *q, struct blk_mq_ctx *ctx,
- struct request *rq, unsigned int op);
-void __blk_mq_finish_request(struct blk_mq_hw_ctx *hctx, struct blk_mq_ctx *ctx,
- struct request *rq);
-void blk_mq_finish_request(struct request *rq);
-struct request *__blk_mq_alloc_request(struct blk_mq_alloc_data *data,
- unsigned int op);
-
static inline bool blk_mq_hctx_stopped(struct blk_mq_hw_ctx *hctx)
{
return test_bit(BLK_MQ_S_STOPPED, &hctx->state);
q->nr_batching = BLK_BATCH_REQ;
blk_set_default_limits(&q->limits);
-
- /*
- * by default assume old behaviour and bounce for any highmem page
- */
- blk_queue_bounce_limit(q, BLK_BOUNCE_HIGH);
}
EXPORT_SYMBOL(blk_queue_make_request);
* all transfers have been done for a request. It's important to call
* this function before end_that_request_last(), as that will put the
* request back on the free list thus corrupting the internal tag list.
- *
- * Notes:
- * queue lock must be held.
**/
void blk_queue_end_tag(struct request_queue *q, struct request *rq)
{
struct blk_queue_tag *bqt = q->queue_tags;
unsigned tag = rq->tag; /* negative tags invalid */
+ lockdep_assert_held(q->queue_lock);
+
BUG_ON(tag >= bqt->real_max_depth);
list_del_init(&rq->queuelist);
* calling this function. The request will also be removed from
* the request queue, so it's the drivers responsibility to readd
* it if it should need to be restarted for some reason.
- *
- * Notes:
- * queue lock must be held.
**/
int blk_queue_start_tag(struct request_queue *q, struct request *rq)
{
unsigned max_depth;
int tag;
+ lockdep_assert_held(q->queue_lock);
+
if (unlikely((rq->rq_flags & RQF_QUEUED))) {
printk(KERN_ERR
"%s: request %p for device [%s] already tagged %d",
* Hardware conditions may dictate a need to stop all pending requests.
* In this case, we will safely clear the block side of the tag queue and
* readd all requests to the request queue in the right order.
- *
- * Notes:
- * queue lock must be held.
**/
void blk_queue_invalidate_tags(struct request_queue *q)
{
struct list_head *tmp, *n;
+ lockdep_assert_held(q->queue_lock);
+
list_for_each_safe(tmp, n, &q->tag_busy_list)
blk_requeue_request(q, list_entry_rq(tmp));
}
* Notes:
* Each request has its own timer, and as it is added to the queue, we
* set up the timer. When the request completes, we cancel the timer.
- * Queue lock must be held for the non-mq case, mq case doesn't care.
*/
void blk_add_timer(struct request *req)
{
struct request_queue *q = req->q;
unsigned long expiry;
+ if (!q->mq_ops)
+ lockdep_assert_held(q->queue_lock);
+
/* blk-mq has its own handler, so we don't need ->rq_timed_out_fn */
if (!q->mq_ops && !q->rq_timed_out_fn)
return;
}
static inline bool may_queue(struct rq_wb *rwb, struct rq_wait *rqw,
- wait_queue_t *wait, unsigned long rw)
+ wait_queue_entry_t *wait, unsigned long rw)
{
/*
* inc it here even if disabled, since we'll dec it at completion.
* in line to be woken up, wait for our turn.
*/
if (waitqueue_active(&rqw->wait) &&
- rqw->wait.task_list.next != &wait->task_list)
+ rqw->wait.head.next != &wait->entry)
return false;
return atomic_inc_below(&rqw->inflight, get_limit(rwb, rw));
struct request *rq;
struct blk_flush_queue *fq = blk_get_flush_queue(q, NULL);
+ WARN_ON_ONCE(q->mq_ops);
+
while (1) {
if (!list_empty(&q->queue_head)) {
rq = list_entry_rq(q->queue_head.next);
static inline void blk_throtl_stat_add(struct request *rq, u64 time) { }
#endif
+#ifdef CONFIG_BOUNCE
+extern int init_emergency_isa_pool(void);
+extern void blk_queue_bounce(struct request_queue *q, struct bio **bio);
+#else
+static inline int init_emergency_isa_pool(void)
+{
+ return 0;
+}
+static inline void blk_queue_bounce(struct request_queue *q, struct bio **bio)
+{
+}
+#endif /* CONFIG_BOUNCE */
+
#endif /* BLK_INTERNAL_H */
#include <asm/tlbflush.h>
#include <trace/events/block.h>
+#include "blk.h"
#define POOL_SIZE 64
#define ISA_POOL_SIZE 16
+static struct bio_set *bounce_bio_set, *bounce_bio_split;
static mempool_t *page_pool, *isa_page_pool;
#if defined(CONFIG_HIGHMEM) || defined(CONFIG_NEED_BOUNCE_POOL)
BUG_ON(!page_pool);
pr_info("pool size: %d pages\n", POOL_SIZE);
+ bounce_bio_set = bioset_create(BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS);
+ BUG_ON(!bounce_bio_set);
+ if (bioset_integrity_create(bounce_bio_set, BIO_POOL_SIZE))
+ BUG_ON(1);
+
+ bounce_bio_split = bioset_create(BIO_POOL_SIZE, 0, 0);
+ BUG_ON(!bounce_bio_split);
+
return 0;
}
mempool_free(bvec->bv_page, pool);
}
- bio_orig->bi_error = bio->bi_error;
+ bio_orig->bi_status = bio->bi_status;
bio_endio(bio_orig);
bio_put(bio);
}
{
struct bio *bio_orig = bio->bi_private;
- if (!bio->bi_error)
+ if (!bio->bi_status)
copy_to_high_bio_irq(bio_orig, bio);
bounce_end_io(bio, pool);
int rw = bio_data_dir(*bio_orig);
struct bio_vec *to, from;
struct bvec_iter iter;
- unsigned i;
-
- bio_for_each_segment(from, *bio_orig, iter)
- if (page_to_pfn(from.bv_page) > queue_bounce_pfn(q))
- goto bounce;
+ unsigned i = 0;
+ bool bounce = false;
+ int sectors = 0;
+
+ bio_for_each_segment(from, *bio_orig, iter) {
+ if (i++ < BIO_MAX_PAGES)
+ sectors += from.bv_len >> 9;
+ if (page_to_pfn(from.bv_page) > q->limits.bounce_pfn)
+ bounce = true;
+ }
+ if (!bounce)
+ return;
- return;
-bounce:
- bio = bio_clone_bioset(*bio_orig, GFP_NOIO, fs_bio_set);
+ if (sectors < bio_sectors(*bio_orig)) {
+ bio = bio_split(*bio_orig, sectors, GFP_NOIO, bounce_bio_split);
+ bio_chain(bio, *bio_orig);
+ generic_make_request(*bio_orig);
+ *bio_orig = bio;
+ }
+ bio = bio_clone_bioset(*bio_orig, GFP_NOIO, bounce_bio_set);
bio_for_each_segment_all(to, bio, i) {
struct page *page = to->bv_page;
- if (page_to_pfn(page) <= queue_bounce_pfn(q))
+ if (page_to_pfn(page) <= q->limits.bounce_pfn)
continue;
to->bv_page = mempool_alloc(pool, q->bounce_gfp);
* don't waste time iterating over bio segments
*/
if (!(q->bounce_gfp & GFP_DMA)) {
- if (queue_bounce_pfn(q) >= blk_max_pfn)
+ if (q->limits.bounce_pfn >= blk_max_pfn)
return;
pool = page_pool;
} else {
*/
__blk_queue_bounce(q, bio_orig, pool);
}
-
-EXPORT_SYMBOL(blk_queue_bounce);
struct bsg_job *job = container_of(kref, struct bsg_job, kref);
struct request *rq = job->req;
- blk_end_request_all(rq, scsi_req(rq)->result);
+ blk_end_request_all(rq, BLK_STS_OK);
put_device(job->dev); /* release reference for the request */
ret = bsg_create_job(dev, req);
if (ret) {
scsi_req(req)->result = ret;
- blk_end_request_all(req, ret);
+ blk_end_request_all(req, BLK_STS_OK);
spin_lock_irq(q->queue_lock);
continue;
}
q->bsg_job_size = dd_job_size;
q->bsg_job_fn = job_fn;
queue_flag_set_unlocked(QUEUE_FLAG_BIDI, q);
+ queue_flag_set_unlocked(QUEUE_FLAG_SCSI_PASSTHROUGH, q);
blk_queue_softirq_done(q, bsg_softirq_done);
blk_queue_rq_timeout(q, BLK_DEFAULT_SG_TIMEOUT);
rq = blk_get_request(q, op, GFP_KERNEL);
if (IS_ERR(rq))
return rq;
- scsi_req_init(rq);
ret = blk_fill_sgv4_hdr_rq(q, rq, hdr, bd, has_write_perm);
if (ret)
* async completion call-back from the block layer, when scsi/ide/whatever
* calls end_that_request_last() on a request
*/
-static void bsg_rq_end_io(struct request *rq, int uptodate)
+static void bsg_rq_end_io(struct request *rq, blk_status_t status)
{
struct bsg_command *bc = rq->end_io_data;
struct bsg_device *bd = bc->bd;
unsigned long flags;
- dprintk("%s: finished rq %p bc %p, bio %p stat %d\n",
- bd->name, rq, bc, bc->bio, uptodate);
+ dprintk("%s: finished rq %p bc %p, bio %p\n",
+ bd->name, rq, bc, bc->bio);
bc->hdr.duration = jiffies_to_msecs(jiffies - bc->hdr.duration);
#ifdef BSG_DEBUG
unsigned char buf[32];
#endif
+
+ if (!blk_queue_scsi_passthrough(rq)) {
+ WARN_ONCE(true, "Attempt to register a non-SCSI queue\n");
+ return ERR_PTR(-EINVAL);
+ }
+
if (!blk_get_queue(rq))
return ERR_PTR(-ENXIO);
return min_vdisktime;
}
-static inline u64 min_vdisktime(u64 min_vdisktime, u64 vdisktime)
-{
- s64 delta = (s64)(vdisktime - min_vdisktime);
- if (delta < 0)
- min_vdisktime = vdisktime;
-
- return min_vdisktime;
-}
-
static void update_min_vdisktime(struct cfq_rb_root *st)
{
struct cfq_group *cfqg;
*/
if (elv_attempt_insert_merge(q, rq))
break;
+ /* fall through */
case ELEVATOR_INSERT_SORT:
BUG_ON(blk_rq_is_passthrough(rq));
rq->rq_flags |= RQF_SORTED;
static DEFINE_SPINLOCK(ext_devt_lock);
static DEFINE_IDR(ext_devt_idr);
-static struct device_type disk_type;
+static const struct device_type disk_type;
static void disk_check_events(struct disk_events *ev,
unsigned int *clearing_ptr);
return NULL;
}
-static struct device_type disk_type = {
+static const struct device_type disk_type = {
.name = "disk",
.groups = disk_attr_groups,
.release = disk_release,
case IOPRIO_CLASS_RT:
if (!capable(CAP_SYS_ADMIN))
return -EPERM;
- /* fall through, rt has prio field too */
+ /* fall through */
+ /* rt has prio field too */
case IOPRIO_CLASS_BE:
if (data >= IOPRIO_BE_NR || data < 0)
return -EINVAL;
struct list_head rqs[KYBER_NUM_DOMAINS];
unsigned int cur_domain;
unsigned int batching;
- wait_queue_t domain_wait[KYBER_NUM_DOMAINS];
+ wait_queue_entry_t domain_wait[KYBER_NUM_DOMAINS];
atomic_t wait_index[KYBER_NUM_DOMAINS];
};
for (i = 0; i < KYBER_NUM_DOMAINS; i++) {
INIT_LIST_HEAD(&khd->rqs[i]);
- INIT_LIST_HEAD(&khd->domain_wait[i].task_list);
+ INIT_LIST_HEAD(&khd->domain_wait[i].entry);
atomic_set(&khd->wait_index[i], 0);
}
}
}
-static struct request *kyber_get_request(struct request_queue *q,
- unsigned int op,
- struct blk_mq_alloc_data *data)
+static void kyber_limit_depth(unsigned int op, struct blk_mq_alloc_data *data)
{
- struct kyber_queue_data *kqd = q->elevator->elevator_data;
- struct request *rq;
-
/*
* We use the scheduler tags as per-hardware queue queueing tokens.
* Async requests can be limited at this stage.
*/
- if (!op_is_sync(op))
+ if (!op_is_sync(op)) {
+ struct kyber_queue_data *kqd = data->q->elevator->elevator_data;
+
data->shallow_depth = kqd->async_depth;
+ }
+}
- rq = __blk_mq_alloc_request(data, op);
- if (rq)
- rq_set_domain_token(rq, -1);
- return rq;
+static void kyber_prepare_request(struct request *rq, struct bio *bio)
+{
+ rq_set_domain_token(rq, -1);
}
-static void kyber_put_request(struct request *rq)
+static void kyber_finish_request(struct request *rq)
{
- struct request_queue *q = rq->q;
- struct kyber_queue_data *kqd = q->elevator->elevator_data;
+ struct kyber_queue_data *kqd = rq->q->elevator->elevator_data;
rq_clear_domain_token(kqd, rq);
- blk_mq_finish_request(rq);
}
static void kyber_completed_request(struct request *rq)
}
}
-static int kyber_domain_wake(wait_queue_t *wait, unsigned mode, int flags,
+static int kyber_domain_wake(wait_queue_entry_t *wait, unsigned mode, int flags,
void *key)
{
struct blk_mq_hw_ctx *hctx = READ_ONCE(wait->private);
- list_del_init(&wait->task_list);
+ list_del_init(&wait->entry);
blk_mq_run_hw_queue(hctx, true);
return 1;
}
{
unsigned int sched_domain = khd->cur_domain;
struct sbitmap_queue *domain_tokens = &kqd->domain_tokens[sched_domain];
- wait_queue_t *wait = &khd->domain_wait[sched_domain];
+ wait_queue_entry_t *wait = &khd->domain_wait[sched_domain];
struct sbq_wait_state *ws;
int nr;
* run when one becomes available. Note that this is serialized on
* khd->lock, but we still need to be careful about the waker.
*/
- if (list_empty_careful(&wait->task_list)) {
+ if (list_empty_careful(&wait->entry)) {
init_waitqueue_func_entry(wait, kyber_domain_wake);
wait->private = hctx;
ws = sbq_wait_ptr(domain_tokens,
{ \
struct blk_mq_hw_ctx *hctx = data; \
struct kyber_hctx_data *khd = hctx->sched_data; \
- wait_queue_t *wait = &khd->domain_wait[domain]; \
+ wait_queue_entry_t *wait = &khd->domain_wait[domain]; \
\
- seq_printf(m, "%d\n", !list_empty_careful(&wait->task_list)); \
+ seq_printf(m, "%d\n", !list_empty_careful(&wait->entry)); \
return 0; \
}
KYBER_DEBUGFS_DOMAIN_ATTRS(KYBER_READ, read)
.exit_sched = kyber_exit_sched,
.init_hctx = kyber_init_hctx,
.exit_hctx = kyber_exit_hctx,
- .get_request = kyber_get_request,
- .put_request = kyber_put_request,
+ .limit_depth = kyber_limit_depth,
+ .prepare_request = kyber_prepare_request,
+ .finish_request = kyber_finish_request,
.completed_request = kyber_completed_request,
.dispatch_request = kyber_dispatch_request,
.has_work = kyber_has_work,
ldm_error("PRIVHEAD disk size doesn't match real disk size");
return false;
}
- if (uuid_be_to_bin(data + 0x0030, (uuid_be *)ph->disk_id)) {
+ if (uuid_parse(data + 0x0030, &ph->disk_id)) {
ldm_error("PRIVHEAD contains an invalid GUID.");
return false;
}
(ph1->logical_disk_size == ph2->logical_disk_size) &&
(ph1->config_start == ph2->config_start) &&
(ph1->config_size == ph2->config_size) &&
- !memcmp (ph1->disk_id, ph2->disk_id, GUID_SIZE));
+ uuid_equal(&ph1->disk_id, &ph2->disk_id));
}
/**
list_for_each (item, &ldb->v_disk) {
struct vblk *v = list_entry (item, struct vblk, list);
- if (!memcmp (v->vblk.disk.disk_id, ldb->ph.disk_id, GUID_SIZE))
+ if (uuid_equal(&v->vblk.disk.disk_id, &ldb->ph.disk_id))
return v;
}
disk = &vb->vblk.disk;
ldm_get_vstr (buffer + 0x18 + r_diskid, disk->alt_name,
sizeof (disk->alt_name));
- if (uuid_be_to_bin(buffer + 0x19 + r_name, (uuid_be *)disk->disk_id))
+ if (uuid_parse(buffer + 0x19 + r_name, &disk->disk_id))
return false;
return true;
return false;
disk = &vb->vblk.disk;
- memcpy (disk->disk_id, buffer + 0x18 + r_name, GUID_SIZE);
+ uuid_copy(&disk->disk_id, (uuid_t *)(buffer + 0x18 + r_name));
return true;
}
/* In memory LDM database structures. */
-#define GUID_SIZE 16
-
struct privhead { /* Offsets and sizes are in sectors. */
u16 ver_major;
u16 ver_minor;
u64 logical_disk_size;
u64 config_start;
u64 config_size;
- u8 disk_id[GUID_SIZE];
+ uuid_t disk_id;
};
struct tocblock { /* We have exactly two bitmaps. */
};
struct vblk_disk { /* VBLK Disk */
- u8 disk_id[GUID_SIZE];
+ uuid_t disk_id;
u8 alt_name[128];
};
if (IS_ERR(rq))
return PTR_ERR(rq);
req = scsi_req(rq);
- scsi_req_init(rq);
if (hdr->cmd_len > BLK_MAX_CDB) {
req->cmd = kzalloc(hdr->cmd_len, GFP_KERNEL);
goto error_free_buffer;
}
req = scsi_req(rq);
- scsi_req_init(rq);
cmdlen = COMMAND_SIZE(opcode);
rq = blk_get_request(q, REQ_OP_SCSI_OUT, __GFP_RECLAIM);
if (IS_ERR(rq))
return PTR_ERR(rq);
- scsi_req_init(rq);
rq->timeout = BLK_DEFAULT_SG_TIMEOUT;
scsi_req(rq)->cmd[0] = cmd;
scsi_req(rq)->cmd[4] = data;
}
EXPORT_SYMBOL(scsi_cmd_blk_ioctl);
-void scsi_req_init(struct request *rq)
+/**
+ * scsi_req_init - initialize certain fields of a scsi_request structure
+ * @req: Pointer to a scsi_request structure.
+ * Initializes .__cmd[], .cmd, .cmd_len and .sense_len but no other members
+ * of struct scsi_request.
+ */
+void scsi_req_init(struct scsi_request *req)
{
- struct scsi_request *req = scsi_req(rq);
-
memset(req->__cmd, 0, sizeof(req->__cmd));
req->cmd = req->__cmd;
req->cmd_len = BLK_MAX_CDB;
* 16 bit app tag, 32 bit reference tag. Type 3 does not define the ref
* tag.
*/
-static int t10_pi_generate(struct blk_integrity_iter *iter, csum_fn *fn,
- unsigned int type)
+static blk_status_t t10_pi_generate(struct blk_integrity_iter *iter,
+ csum_fn *fn, unsigned int type)
{
unsigned int i;
iter->seed++;
}
- return 0;
+ return BLK_STS_OK;
}
-static int t10_pi_verify(struct blk_integrity_iter *iter, csum_fn *fn,
- unsigned int type)
+static blk_status_t t10_pi_verify(struct blk_integrity_iter *iter,
+ csum_fn *fn, unsigned int type)
{
unsigned int i;
"(rcvd %u)\n", iter->disk_name,
(unsigned long long)
iter->seed, be32_to_cpu(pi->ref_tag));
- return -EILSEQ;
+ return BLK_STS_PROTECTION;
}
break;
case 3:
"(rcvd %04x, want %04x)\n", iter->disk_name,
(unsigned long long)iter->seed,
be16_to_cpu(pi->guard_tag), be16_to_cpu(csum));
- return -EILSEQ;
+ return BLK_STS_PROTECTION;
}
next:
iter->seed++;
}
- return 0;
+ return BLK_STS_OK;
}
-static int t10_pi_type1_generate_crc(struct blk_integrity_iter *iter)
+static blk_status_t t10_pi_type1_generate_crc(struct blk_integrity_iter *iter)
{
return t10_pi_generate(iter, t10_pi_crc_fn, 1);
}
-static int t10_pi_type1_generate_ip(struct blk_integrity_iter *iter)
+static blk_status_t t10_pi_type1_generate_ip(struct blk_integrity_iter *iter)
{
return t10_pi_generate(iter, t10_pi_ip_fn, 1);
}
-static int t10_pi_type1_verify_crc(struct blk_integrity_iter *iter)
+static blk_status_t t10_pi_type1_verify_crc(struct blk_integrity_iter *iter)
{
return t10_pi_verify(iter, t10_pi_crc_fn, 1);
}
-static int t10_pi_type1_verify_ip(struct blk_integrity_iter *iter)
+static blk_status_t t10_pi_type1_verify_ip(struct blk_integrity_iter *iter)
{
return t10_pi_verify(iter, t10_pi_ip_fn, 1);
}
-static int t10_pi_type3_generate_crc(struct blk_integrity_iter *iter)
+static blk_status_t t10_pi_type3_generate_crc(struct blk_integrity_iter *iter)
{
return t10_pi_generate(iter, t10_pi_crc_fn, 3);
}
-static int t10_pi_type3_generate_ip(struct blk_integrity_iter *iter)
+static blk_status_t t10_pi_type3_generate_ip(struct blk_integrity_iter *iter)
{
return t10_pi_generate(iter, t10_pi_ip_fn, 3);
}
-static int t10_pi_type3_verify_crc(struct blk_integrity_iter *iter)
+static blk_status_t t10_pi_type3_verify_crc(struct blk_integrity_iter *iter)
{
return t10_pi_verify(iter, t10_pi_crc_fn, 3);
}
-static int t10_pi_type3_verify_ip(struct blk_integrity_iter *iter)
+static blk_status_t t10_pi_type3_verify_ip(struct blk_integrity_iter *iter)
{
return t10_pi_verify(iter, t10_pi_ip_fn, 3);
}
source "drivers/tee/Kconfig"
+source "drivers/mux/Kconfig"
+
endmenu
obj-$(CONFIG_FPGA) += fpga/
obj-$(CONFIG_FSI) += fsi/
obj-$(CONFIG_TEE) += tee/
+obj-$(CONFIG_MULTIPLEXER) += mux/
#include <linux/configfs.h>
#include <linux/acpi.h>
+#include "acpica/accommon.h"
+#include "acpica/actables.h"
+
static struct config_group *acpi_table_group;
struct acpi_table {
struct config_item cfg;
struct acpi_table_header *header;
+ u32 index;
};
static ssize_t acpi_table_aml_write(struct config_item *cfg,
if (!table->header)
return -ENOMEM;
- ret = acpi_load_table(table->header);
+ ACPI_INFO(("Host-directed Dynamic ACPI Table Load:"));
+ ret = acpi_tb_install_and_load_table(
+ ACPI_PTR_TO_PHYSADDR(table->header),
+ ACPI_TABLE_ORIGIN_EXTERNAL_VIRTUAL, FALSE,
+ &table->index);
if (ret) {
kfree(table->header);
table->header = NULL;
return &table->cfg;
}
+static void acpi_table_drop_item(struct config_group *group,
+ struct config_item *cfg)
+{
+ struct acpi_table *table = container_of(cfg, struct acpi_table, cfg);
+
+ ACPI_INFO(("Host-directed Dynamic ACPI Table Unload"));
+ acpi_tb_unload_table(table->index);
+}
+
struct configfs_group_operations acpi_table_group_ops = {
.make_item = acpi_table_make_item,
+ .drop_item = acpi_table_drop_item,
};
static struct config_item_type acpi_tables_type = {
int cpu = mce->extcpu;
struct acpi_hest_generic_status *estatus, *tmp;
struct acpi_hest_generic_data *gdata;
- const uuid_le *fru_id = &NULL_UUID_LE;
+ const guid_t *fru_id = &guid_null;
char *fru_text = "";
- uuid_le *sec_type;
+ guid_t *sec_type;
static u32 err_seq;
estatus = extlog_elog_entry_check(cpu, bank);
err_seq++;
gdata = (struct acpi_hest_generic_data *)(tmp + 1);
if (gdata->validation_bits & CPER_SEC_VALID_FRU_ID)
- fru_id = (uuid_le *)gdata->fru_id;
+ fru_id = (guid_t *)gdata->fru_id;
if (gdata->validation_bits & CPER_SEC_VALID_FRU_TEXT)
fru_text = gdata->fru_text;
- sec_type = (uuid_le *)gdata->section_type;
- if (!uuid_le_cmp(*sec_type, CPER_SEC_PLATFORM_MEM)) {
+ sec_type = (guid_t *)gdata->section_type;
+ if (guid_equal(sec_type, &CPER_SEC_PLATFORM_MEM)) {
struct cper_sec_mem_err *mem = (void *)(gdata + 1);
if (gdata->error_data_length >= sizeof(*mem))
trace_extlog_mem_event(mem, err_seq, fru_id, fru_text,
static bool __init extlog_get_l1addr(void)
{
- u8 uuid[16];
+ guid_t guid;
acpi_handle handle;
union acpi_object *obj;
- acpi_str_to_uuid(extlog_dsm_uuid, uuid);
-
+ if (guid_parse(extlog_dsm_uuid, &guid))
+ return false;
if (ACPI_FAILURE(acpi_get_handle(NULL, "\\_SB", &handle)))
return false;
- if (!acpi_check_dsm(handle, uuid, EXTLOG_DSM_REV, 1 << EXTLOG_FN_ADDR))
+ if (!acpi_check_dsm(handle, &guid, EXTLOG_DSM_REV, 1 << EXTLOG_FN_ADDR))
return false;
- obj = acpi_evaluate_dsm_typed(handle, uuid, EXTLOG_DSM_REV,
+ obj = acpi_evaluate_dsm_typed(handle, &guid, EXTLOG_DSM_REV,
EXTLOG_FN_ADDR, NULL, ACPI_TYPE_INTEGER);
if (!obj) {
return false;
utosi.o \
utownerid.o \
utpredef.o \
+ utresdecode.o \
utresrc.o \
utstate.o \
utstring.o \
acpi_owner_id owner_id);
void
-acpi_dm_convert_resource_indexes(union acpi_parse_object *parse_tree_root,
- struct acpi_namespace_node *namespace_root);
+acpi_dm_convert_parse_objects(union acpi_parse_object *parse_tree_root,
+ struct acpi_namespace_node *namespace_root);
/*
* adfile
ACPI_INIT_GLOBAL(u8, acpi_gbl_dm_opt_verbose, TRUE);
ACPI_INIT_GLOBAL(u8, acpi_gbl_dm_emit_external_opcodes, FALSE);
ACPI_INIT_GLOBAL(u8, acpi_gbl_do_disassembler_optimizations, TRUE);
+ACPI_INIT_GLOBAL(ACPI_PARSE_OBJECT_LIST, *acpi_gbl_temp_list_head, NULL);
ACPI_GLOBAL(u8, acpi_gbl_dm_opt_disasm);
ACPI_GLOBAL(u8, acpi_gbl_dm_opt_listing);
ACPI_GLOBAL(const char, *acpi_gbl_pld_horizontal_position_list[]);
ACPI_GLOBAL(const char, *acpi_gbl_pld_shape_list[]);
+ACPI_INIT_GLOBAL(u8, acpi_gbl_disasm_flag, FALSE);
+
#endif
/*
* and bytelists.
*/
struct acpi_parse_obj_named {
- ACPI_PARSE_COMMON u8 *path;
+ ACPI_PARSE_COMMON char *path;
u8 *data; /* AML body or bytelist data */
u32 length; /* AML length */
u32 name; /* 4-byte name or zero if no name */
#define ACPI_RESOURCE_NAME_ADDRESS64 0x8A
#define ACPI_RESOURCE_NAME_EXTENDED_ADDRESS64 0x8B
#define ACPI_RESOURCE_NAME_GPIO 0x8C
+#define ACPI_RESOURCE_NAME_PIN_FUNCTION 0x8D
#define ACPI_RESOURCE_NAME_SERIAL_BUS 0x8E
-#define ACPI_RESOURCE_NAME_LARGE_MAX 0x8E
+#define ACPI_RESOURCE_NAME_PIN_CONFIG 0x8F
+#define ACPI_RESOURCE_NAME_PIN_GROUP 0x90
+#define ACPI_RESOURCE_NAME_PIN_GROUP_FUNCTION 0x91
+#define ACPI_RESOURCE_NAME_PIN_GROUP_CONFIG 0x92
+#define ACPI_RESOURCE_NAME_LARGE_MAX 0x92
/*****************************************************************************
*
#define ACPI_EXT_INTERNAL_PATH_ALLOCATED 0x04 /* Deallocate internal path on completion */
#define ACPI_EXT_EXTERNAL_EMITTED 0x08 /* External() statement has been emitted */
#define ACPI_EXT_ORIGIN_FROM_OPCODE 0x10 /* External came from a External() opcode */
+#define ACPI_EXT_CONFLICTING_DECLARATION 0x20 /* External has a conflicting declaration within AML */
struct acpi_external_file {
char *path;
struct acpi_external_file *next;
};
+struct acpi_parse_object_list {
+ union acpi_parse_object *op;
+ struct acpi_parse_object_list *next;
+};
+
/*****************************************************************************
*
* Debugger
#define ARGP_DWORD_OP ARGP_LIST1 (ARGP_DWORDDATA)
#define ARGP_ELSE_OP ARGP_LIST2 (ARGP_PKGLENGTH, ARGP_TERMLIST)
#define ARGP_EVENT_OP ARGP_LIST1 (ARGP_NAME)
-#define ARGP_EXTERNAL_OP ARGP_LIST3 (ARGP_NAMESTRING, ARGP_BYTEDATA, ARGP_BYTEDATA)
+#define ARGP_EXTERNAL_OP ARGP_LIST3 (ARGP_NAME, ARGP_BYTEDATA, ARGP_BYTEDATA)
#define ARGP_FATAL_OP ARGP_LIST3 (ARGP_BYTEDATA, ARGP_DWORDDATA, ARGP_TERMARG)
#define ARGP_FIELD_OP ARGP_LIST4 (ARGP_PKGLENGTH, ARGP_NAMESTRING, ARGP_BYTEDATA, ARGP_FIELDLIST)
#define ARGP_FIND_SET_LEFT_BIT_OP ARGP_LIST2 (ARGP_TERMARG, ARGP_TARGET)
{{"_HID", METHOD_0ARGS,
METHOD_RETURNS(ACPI_RTYPE_INTEGER | ACPI_RTYPE_STRING)}},
+ {{"_HMA", METHOD_0ARGS,
+ METHOD_RETURNS(ACPI_RTYPE_BUFFER)}},
+
{{"_HOT", METHOD_0ARGS,
METHOD_RETURNS(ACPI_RTYPE_INTEGER)}},
ACPI_RTYPE_INTEGER | ACPI_RTYPE_BUFFER | ACPI_RTYPE_STRING,
10, 0),
+ {{"_LSI", METHOD_0ARGS,
+ METHOD_RETURNS(ACPI_RTYPE_PACKAGE)}},
+ PACKAGE_INFO(ACPI_PTYPE1_FIXED, ACPI_RTYPE_INTEGER, 3, 0, 0, 0),
+
+ {{"_LSR", METHOD_2ARGS(ACPI_TYPE_INTEGER, ACPI_TYPE_INTEGER),
+ METHOD_RETURNS(ACPI_RTYPE_PACKAGE)}},
+ PACKAGE_INFO(ACPI_PTYPE1_FIXED, ACPI_RTYPE_INTEGER, 1,
+ ACPI_RTYPE_BUFFER, 1, 0),
+
+ {{"_LSW",
+ METHOD_3ARGS(ACPI_TYPE_INTEGER, ACPI_TYPE_INTEGER, ACPI_TYPE_BUFFER),
+ METHOD_RETURNS(ACPI_RTYPE_INTEGER)}},
+
{{"_MAT", METHOD_0ARGS,
METHOD_RETURNS(ACPI_RTYPE_BUFFER)}},
ACPI_RSD_UINT16,
ACPI_RSD_UINT32,
ACPI_RSD_UINT64,
- ACPI_RSD_WORDLIST
+ ACPI_RSD_WORDLIST,
+ ACPI_RSD_LABEL,
+ ACPI_RSD_SOURCE_LABEL,
+
} ACPI_RSDUMP_OPCODES;
/* restore default alignment */
extern struct acpi_rsconvert_info acpi_rs_convert_i2c_serial_bus[];
extern struct acpi_rsconvert_info acpi_rs_convert_spi_serial_bus[];
extern struct acpi_rsconvert_info acpi_rs_convert_uart_serial_bus[];
+extern struct acpi_rsconvert_info acpi_rs_convert_pin_function[];
+extern struct acpi_rsconvert_info acpi_rs_convert_pin_config[];
+extern struct acpi_rsconvert_info acpi_rs_convert_pin_group[];
+extern struct acpi_rsconvert_info acpi_rs_convert_pin_group_function[];
+extern struct acpi_rsconvert_info acpi_rs_convert_pin_group_config[];
/* These resources require separate get/set tables */
extern struct acpi_rsdump_info acpi_rs_dump_ext_irq[];
extern struct acpi_rsdump_info acpi_rs_dump_generic_reg[];
extern struct acpi_rsdump_info acpi_rs_dump_gpio[];
+extern struct acpi_rsdump_info acpi_rs_dump_pin_function[];
extern struct acpi_rsdump_info acpi_rs_dump_fixed_dma[];
extern struct acpi_rsdump_info acpi_rs_dump_common_serial_bus[];
extern struct acpi_rsdump_info acpi_rs_dump_i2c_serial_bus[];
extern struct acpi_rsdump_info acpi_rs_dump_spi_serial_bus[];
extern struct acpi_rsdump_info acpi_rs_dump_uart_serial_bus[];
extern struct acpi_rsdump_info acpi_rs_dump_general_flags[];
+extern struct acpi_rsdump_info acpi_rs_dump_pin_config[];
+extern struct acpi_rsdump_info acpi_rs_dump_pin_group[];
+extern struct acpi_rsdump_info acpi_rs_dump_pin_group_function[];
+extern struct acpi_rsdump_info acpi_rs_dump_pin_group_config[];
#endif
#endif /* __ACRESRC_H__ */
extern const char *acpi_gbl_sb_decode[];
extern const char *acpi_gbl_fc_decode[];
extern const char *acpi_gbl_pt_decode[];
+extern const char *acpi_gbl_ptyp_decode[];
#endif
/*
* #A is the number of required arguments
* #T is the number of target operands
* #R indicates whether there is a return value
+ *
+ * These types are used for the top-level dispatch of the AML
+ * opcode. They group similar operators that can share common
+ * front-end code before dispatch to the final code that implements
+ * the operator.
*/
/*
* The opcode Type is used in a dispatch table, do not change
* or add anything new without updating the table.
*/
-#define AML_TYPE_EXEC_0A_0T_1R 0x00
-#define AML_TYPE_EXEC_1A_0T_0R 0x01 /* Monadic1 */
-#define AML_TYPE_EXEC_1A_0T_1R 0x02 /* Monadic2 */
-#define AML_TYPE_EXEC_1A_1T_0R 0x03
-#define AML_TYPE_EXEC_1A_1T_1R 0x04 /* monadic2_r */
-#define AML_TYPE_EXEC_2A_0T_0R 0x05 /* Dyadic1 */
-#define AML_TYPE_EXEC_2A_0T_1R 0x06 /* Dyadic2 */
-#define AML_TYPE_EXEC_2A_1T_1R 0x07 /* dyadic2_r */
-#define AML_TYPE_EXEC_2A_2T_1R 0x08
-#define AML_TYPE_EXEC_3A_0T_0R 0x09
-#define AML_TYPE_EXEC_3A_1T_1R 0x0A
-#define AML_TYPE_EXEC_6A_0T_1R 0x0B
+#define AML_TYPE_EXEC_0A_0T_1R 0x00 /* 0 Args, 0 Target, 1 ret_val */
+#define AML_TYPE_EXEC_1A_0T_0R 0x01 /* 1 Args, 0 Target, 0 ret_val */
+#define AML_TYPE_EXEC_1A_0T_1R 0x02 /* 1 Args, 0 Target, 1 ret_val */
+#define AML_TYPE_EXEC_1A_1T_0R 0x03 /* 1 Args, 1 Target, 0 ret_val */
+#define AML_TYPE_EXEC_1A_1T_1R 0x04 /* 1 Args, 1 Target, 1 ret_val */
+#define AML_TYPE_EXEC_2A_0T_0R 0x05 /* 2 Args, 0 Target, 0 ret_val */
+#define AML_TYPE_EXEC_2A_0T_1R 0x06 /* 2 Args, 0 Target, 1 ret_val */
+#define AML_TYPE_EXEC_2A_1T_1R 0x07 /* 2 Args, 1 Target, 1 ret_val */
+#define AML_TYPE_EXEC_2A_2T_1R 0x08 /* 2 Args, 2 Target, 1 ret_val */
+#define AML_TYPE_EXEC_3A_0T_0R 0x09 /* 3 Args, 0 Target, 0 ret_val */
+#define AML_TYPE_EXEC_3A_1T_1R 0x0A /* 3 Args, 1 Target, 1 ret_val */
+#define AML_TYPE_EXEC_6A_0T_1R 0x0B /* 6 Args, 0 Target, 1 ret_val */
/* End of types used in dispatch table */
-#define AML_TYPE_LITERAL 0x0B
-#define AML_TYPE_CONSTANT 0x0C
-#define AML_TYPE_METHOD_ARGUMENT 0x0D
-#define AML_TYPE_LOCAL_VARIABLE 0x0E
-#define AML_TYPE_DATA_TERM 0x0F
+#define AML_TYPE_LITERAL 0x0C
+#define AML_TYPE_CONSTANT 0x0D
+#define AML_TYPE_METHOD_ARGUMENT 0x0E
+#define AML_TYPE_LOCAL_VARIABLE 0x0F
+#define AML_TYPE_DATA_TERM 0x10
/* Generic for an op that returns a value */
-#define AML_TYPE_METHOD_CALL 0x10
+#define AML_TYPE_METHOD_CALL 0x11
/* Miscellaneous types */
-#define AML_TYPE_CREATE_FIELD 0x11
-#define AML_TYPE_CREATE_OBJECT 0x12
-#define AML_TYPE_CONTROL 0x13
-#define AML_TYPE_NAMED_NO_OBJ 0x14
-#define AML_TYPE_NAMED_FIELD 0x15
-#define AML_TYPE_NAMED_SIMPLE 0x16
-#define AML_TYPE_NAMED_COMPLEX 0x17
-#define AML_TYPE_RETURN 0x18
-#define AML_TYPE_UNDEFINED 0x19
-#define AML_TYPE_BOGUS 0x1A
+#define AML_TYPE_CREATE_FIELD 0x12
+#define AML_TYPE_CREATE_OBJECT 0x13
+#define AML_TYPE_CONTROL 0x14
+#define AML_TYPE_NAMED_NO_OBJ 0x15
+#define AML_TYPE_NAMED_FIELD 0x16
+#define AML_TYPE_NAMED_SIMPLE 0x17
+#define AML_TYPE_NAMED_COMPLEX 0x18
+#define AML_TYPE_RETURN 0x19
+#define AML_TYPE_UNDEFINED 0x1A
+#define AML_TYPE_BOGUS 0x1B
/* AML Package Length encodings */
#define ACPI_RESTAG_DRIVESTRENGTH "_DRS"
#define ACPI_RESTAG_ENDIANNESS "_END"
#define ACPI_RESTAG_FLOWCONTROL "_FLC"
+#define ACPI_RESTAG_FUNCTION "_FUN"
#define ACPI_RESTAG_GRANULARITY "_GRA"
#define ACPI_RESTAG_INTERRUPT "_INT"
#define ACPI_RESTAG_INTERRUPTLEVEL "_LL_" /* active_lo(1), active_hi(0) */
#define ACPI_RESTAG_PHASE "_PHA"
#define ACPI_RESTAG_PIN "_PIN"
#define ACPI_RESTAG_PINCONFIG "_PPI"
+#define ACPI_RESTAG_PINCONFIG_TYPE "_TYP"
+#define ACPI_RESTAG_PINCONFIG_VALUE "_VAL"
#define ACPI_RESTAG_POLARITY "_POL"
#define ACPI_RESTAG_REGISTERBITOFFSET "_RBO"
#define ACPI_RESTAG_REGISTERBITWIDTH "_RBW"
#define AML_RESOURCE_UART_TYPE_REVISION 1 /* ACPI 5.0 */
#define AML_RESOURCE_UART_MIN_DATA_LEN 10
+struct aml_resource_pin_function {
+ AML_RESOURCE_LARGE_HEADER_COMMON u8 revision_id;
+ u16 flags;
+ u8 pin_config;
+ u16 function_number;
+ u16 pin_table_offset;
+ u8 res_source_index;
+ u16 res_source_offset;
+ u16 vendor_offset;
+ u16 vendor_length;
+ /*
+ * Optional fields follow immediately:
+ * 1) PIN list (Words)
+ * 2) Resource Source String
+ * 3) Vendor Data bytes
+ */
+};
+
+#define AML_RESOURCE_PIN_FUNCTION_REVISION 1 /* ACPI 6.2 */
+
+struct aml_resource_pin_config {
+ AML_RESOURCE_LARGE_HEADER_COMMON u8 revision_id;
+ u16 flags;
+ u8 pin_config_type;
+ u32 pin_config_value;
+ u16 pin_table_offset;
+ u8 res_source_index;
+ u16 res_source_offset;
+ u16 vendor_offset;
+ u16 vendor_length;
+ /*
+ * Optional fields follow immediately:
+ * 1) PIN list (Words)
+ * 2) Resource Source String
+ * 3) Vendor Data bytes
+ */
+};
+
+#define AML_RESOURCE_PIN_CONFIG_REVISION 1 /* ACPI 6.2 */
+
+struct aml_resource_pin_group {
+ AML_RESOURCE_LARGE_HEADER_COMMON u8 revision_id;
+ u16 flags;
+ u16 pin_table_offset;
+ u16 label_offset;
+ u16 vendor_offset;
+ u16 vendor_length;
+ /*
+ * Optional fields follow immediately:
+ * 1) PIN list (Words)
+ * 2) Resource Label String
+ * 3) Vendor Data bytes
+ */
+};
+
+#define AML_RESOURCE_PIN_GROUP_REVISION 1 /* ACPI 6.2 */
+
+struct aml_resource_pin_group_function {
+ AML_RESOURCE_LARGE_HEADER_COMMON u8 revision_id;
+ u16 flags;
+ u16 function_number;
+ u8 res_source_index;
+ u16 res_source_offset;
+ u16 res_source_label_offset;
+ u16 vendor_offset;
+ u16 vendor_length;
+ /*
+ * Optional fields follow immediately:
+ * 1) Resource Source String
+ * 2) Resource Source Label String
+ * 3) Vendor Data bytes
+ */
+};
+
+#define AML_RESOURCE_PIN_GROUP_FUNCTION_REVISION 1 /* ACPI 6.2 */
+
+struct aml_resource_pin_group_config {
+ AML_RESOURCE_LARGE_HEADER_COMMON u8 revision_id;
+ u16 flags;
+ u8 pin_config_type;
+ u32 pin_config_value;
+ u8 res_source_index;
+ u16 res_source_offset;
+ u16 res_source_label_offset;
+ u16 vendor_offset;
+ u16 vendor_length;
+ /*
+ * Optional fields follow immediately:
+ * 1) Resource Source String
+ * 2) Resource Source Label String
+ * 3) Vendor Data bytes
+ */
+};
+
+#define AML_RESOURCE_PIN_GROUP_CONFIG_REVISION 1 /* ACPI 6.2 */
+
/* restore default alignment */
#pragma pack()
struct aml_resource_spi_serialbus spi_serial_bus;
struct aml_resource_uart_serialbus uart_serial_bus;
struct aml_resource_common_serialbus common_serial_bus;
+ struct aml_resource_pin_function pin_function;
+ struct aml_resource_pin_config pin_config;
+ struct aml_resource_pin_group pin_group;
+ struct aml_resource_pin_group_function pin_group_function;
+ struct aml_resource_pin_group_config pin_group_config;
/* Utility overlays */
acpi_gbl_method_executing = FALSE;
if (ACPI_FAILURE(status)) {
+ if ((status == AE_ABORT_METHOD) || acpi_gbl_abort_method) {
+
+ /* Clear the abort and fall back to the debugger prompt */
+
+ ACPI_EXCEPTION((AE_INFO, status,
+ "Aborting top-level method"));
+
+ acpi_gbl_abort_method = FALSE;
+ status = AE_OK;
+ goto cleanup;
+ }
+
ACPI_EXCEPTION((AE_INFO, status,
"while executing %s from debugger",
info->pathname));
if (display_locals) {
acpi_os_printf
- ("\nInitialized Local Variables for method [%4.4s]:\n",
+ ("\nInitialized Local Variables for Method [%4.4s]:\n",
acpi_ut_get_node_name(node));
for (i = 0; i < ACPI_METHOD_NUM_LOCALS; i++) {
}
} else {
acpi_os_printf
- ("No Local Variables are initialized for method [%4.4s]\n",
+ ("No Local Variables are initialized for Method [%4.4s]\n",
acpi_ut_get_node_name(node));
}
}
acpi_os_printf("Initialized Arguments for Method [%4.4s]: "
"(%X arguments defined for method invocation)\n",
acpi_ut_get_node_name(node),
- obj_desc->method.param_count);
+ node->object->method.param_count);
for (i = 0; i < ACPI_METHOD_NUM_ARGS; i++) {
obj_desc = walk_state->arguments[i].object;
if ((acpi_gbl_db_output_to_file) ||
(acpi_dbg_level & ACPI_LV_PARSE)) {
acpi_os_printf
- ("\n[AmlDebug] Next AML Opcode to execute:\n");
+ ("\nAML Debug: Next AML Opcode to execute:\n");
}
/*
union acpi_parse_object *op;
struct acpi_walk_state *walk_state;
- ACPI_FUNCTION_TRACE(ds_execute_arguments);
+ ACPI_FUNCTION_TRACE_PTR(ds_execute_arguments, aml_start);
/* Allocate a new parser op to be the root of the parsed tree */
return_ACPI_STATUS(AE_AML_INTERNAL);
}
- ACPI_DEBUG_PRINT((ACPI_DB_EXEC, "Package Arg Init\n"));
+ ACPI_DEBUG_PRINT((ACPI_DB_EXEC, "Package Argument Init, AML Ptr: %p\n",
+ obj_desc->package.aml_start));
/* Execute the AML code for the term_arg arguments */
op->common.next = NULL;
#ifdef ACPI_DISASSEMBLER
+ acpi_os_printf("Failed at ");
acpi_dm_disassemble(next_walk_state, op,
ACPI_UINT32_MAX);
#endif
acpi_ds_method_error(acpi_status status, struct acpi_walk_state *walk_state)
{
u32 aml_offset;
+ acpi_name name = 0;
ACPI_FUNCTION_ENTRY();
walk_state->parser_state.
aml_start);
- status = acpi_gbl_exception_handler(status,
- walk_state->method_node ?
- walk_state->method_node->
- name.integer : 0,
+ if (walk_state->method_node) {
+ name = walk_state->method_node->name.integer;
+ } else if (walk_state->deferred_node) {
+ name = walk_state->deferred_node->name.integer;
+ }
+
+ status = acpi_gbl_exception_handler(status, name,
walk_state->opcode,
aml_offset, NULL);
acpi_ex_enter_interpreter();
/* Entire field must fit within the current length of the buffer */
- if ((bit_offset + bit_count) > (8 * (u32) buffer_desc->buffer.length)) {
+ if ((bit_offset + bit_count) > (8 * (u32)buffer_desc->buffer.length)) {
ACPI_ERROR((AE_INFO,
- "Field [%4.4s] at %u exceeds Buffer [%4.4s] size %u (bits)",
- acpi_ut_get_node_name(result_desc),
- bit_offset + bit_count,
- acpi_ut_get_node_name(buffer_desc->buffer.node),
- 8 * (u32) buffer_desc->buffer.length));
+ "Field [%4.4s] at bit offset/length %u/%u "
+ "exceeds size of target Buffer (%u bits)",
+ acpi_ut_get_node_name(result_desc), bit_offset,
+ bit_count, 8 * (u32)buffer_desc->buffer.length));
status = AE_AML_BUFFER_LIMIT;
goto cleanup;
}
if ((op_info->flags & AML_HAS_RETVAL) ||
(arg->common.flags & ACPI_PARSEOP_IN_STACK)) {
- ACPI_DEBUG_PRINT((ACPI_DB_DISPATCH,
- "Argument previously created, already stacked\n"));
-
- acpi_db_display_argument_object(walk_state->
- operands[walk_state->
- num_operands -
- 1],
- walk_state);
-
/*
* Use value that was already previously returned
* by the evaluation of this argument
case AML_TYPE_CREATE_OBJECT:
ACPI_DEBUG_PRINT((ACPI_DB_EXEC,
- "Executing CreateObject (Buffer/Package) Op=%p\n",
- op));
+ "Executing CreateObject (Buffer/Package) Op=%p AMLPtr=%p\n",
+ op, op->named.data));
switch (op->common.parent->common.aml_opcode) {
case AML_NAME_OP:
/* Initialize the op */
#if (defined (ACPI_NO_METHOD_EXECUTION) || defined (ACPI_CONSTANT_EVAL_ONLY))
- op->named.path = ACPI_CAST_PTR(u8, path);
+ op->named.path = path;
#endif
if (node) {
acpi_object_type object_type;
acpi_status status = AE_OK;
+#ifdef ACPI_ASL_COMPILER
+ u8 param_count;
+#endif
+
ACPI_FUNCTION_TRACE(ds_load1_end_op);
op = walk_state->op;
}
}
}
+#ifdef ACPI_ASL_COMPILER
+ /*
+ * For external opcode, get the object type from the argument and
+ * get the parameter count from the argument's next.
+ */
+ if (acpi_gbl_disasm_flag &&
+ op->common.node && op->common.aml_opcode == AML_EXTERNAL_OP) {
+ /*
+ * Note, if this external is not a method
+ * Op->Common.Value.Arg->Common.Next->Common.Value.Integer == 0
+ * Therefore, param_count will be 0.
+ */
+ param_count =
+ (u8)op->common.value.arg->common.next->common.value.integer;
+ object_type = (u8)op->common.value.arg->common.value.integer;
+ op->common.node->flags |= ANOBJ_IS_EXTERNAL;
+ op->common.node->type = (u8)object_type;
+
+ acpi_dm_create_subobject_for_external((u8)object_type,
+ &op->common.node,
+ param_count);
+
+ /*
+ * Add the external to the external list because we may be
+ * emitting code based off of the items within the external list.
+ */
+ acpi_dm_add_op_to_external_list(op, op->named.path,
+ (u8)object_type, param_count,
+ ACPI_EXT_ORIGIN_FROM_OPCODE |
+ ACPI_EXT_RESOLVED_REFERENCE);
+ }
+#endif
/*
* If we are executing a method, do not create any namespace objects
/* Pop the scope stack (only if loading a table) */
- if (!walk_state->method_node && acpi_ns_opens_scope(object_type)) {
+ if (!walk_state->method_node &&
+ op->common.aml_opcode != AML_EXTERNAL_OP &&
+ acpi_ns_opens_scope(object_type)) {
ACPI_DEBUG_PRINT((ACPI_DB_DISPATCH,
"(%s): Popping scope for Op %p\n",
acpi_ut_get_type_name(object_type), op));
flags |= ACPI_NS_TEMPORARY;
}
}
+#ifdef ACPI_ASL_COMPILER
+
+ /*
+ * Do not open a scope for AML_EXTERNAL_OP
+ * acpi_ns_lookup can open a new scope based on the object type
+ * of this op. AML_EXTERNAL_OP is a declaration rather than a
+ * definition. In the case that this external is a method object,
+ * acpi_ns_lookup will open a new scope. However, an AML_EXTERNAL_OP
+ * associated with the ACPI_TYPE_METHOD is a declaration, rather than
+ * a definition. Flags is set to avoid opening a scope for any
+ * AML_EXTERNAL_OP.
+ */
+ if (walk_state->opcode == AML_EXTERNAL_OP) {
+ flags |= ACPI_NS_DONT_OPEN_SCOPE;
+ }
+#endif
/* Add new entry or lookup existing entry */
ACPI_FUNCTION_TRACE(acpi_enable_event);
+ /* If Hardware Reduced flag is set, there are no fixed events */
+
+ if (acpi_gbl_reduced_hardware) {
+ return_ACPI_STATUS(AE_OK);
+ }
+
/* Decode the Fixed Event */
if (event > ACPI_EVENT_MAX) {
ACPI_FUNCTION_TRACE(acpi_disable_event);
+ /* If Hardware Reduced flag is set, there are no fixed events */
+
+ if (acpi_gbl_reduced_hardware) {
+ return_ACPI_STATUS(AE_OK);
+ }
+
/* Decode the Fixed Event */
if (event > ACPI_EVENT_MAX) {
ACPI_FUNCTION_TRACE(acpi_clear_event);
+ /* If Hardware Reduced flag is set, there are no fixed events */
+
+ if (acpi_gbl_reduced_hardware) {
+ return_ACPI_STATUS(AE_OK);
+ }
+
/* Decode the Fixed Event */
if (event > ACPI_EVENT_MAX) {
timer = ((u32)acpi_os_get_timer() / 10);
timer &= 0x03FFFFFF;
- acpi_os_printf("[ACPI Debug T=0x%8.8X] %*s", timer,
+ acpi_os_printf("ACPI Debug: T=0x%8.8X %*s", timer,
level, " ");
} else {
- acpi_os_printf("[ACPI Debug] %*s", level, " ");
+ acpi_os_printf("ACPI Debug: %*s", level, " ");
}
}
/* obj_desc is a valid object */
if (depth > 0) {
- ACPI_DEBUG_PRINT((ACPI_DB_EXEC, "%*s[%u] %p ",
- depth, " ", depth, obj_desc));
+ ACPI_DEBUG_PRINT((ACPI_DB_EXEC, "%*s[%u] %p Refs=%u ",
+ depth, " ", depth, obj_desc,
+ obj_desc->common.reference_count));
} else {
- ACPI_DEBUG_PRINT((ACPI_DB_EXEC, "%p ", obj_desc));
+ ACPI_DEBUG_PRINT((ACPI_DB_EXEC, "%p Refs=%u ",
+ obj_desc, obj_desc->common.reference_count));
}
/* Decode object type */
case ACPI_REFCLASS_NAME:
- acpi_os_printf("- [%4.4s]\n",
- obj_desc->reference.node->name.ascii);
+ acpi_ut_repair_name(obj_desc->reference.node->name.
+ ascii);
+ acpi_os_printf("- [%4.4s] (Node %p)\n",
+ obj_desc->reference.node->name.ascii,
+ obj_desc->reference.node);
break;
case ACPI_REFCLASS_ARG:
status = acpi_ns_handle_to_pathname(obj_desc->reference.node,
&ret_buf, TRUE);
if (ACPI_FAILURE(status)) {
- acpi_os_printf(" Could not convert name to pathname\n");
+ acpi_os_printf
+ (" Could not convert name to pathname: %s\n",
+ acpi_format_exception(status));
} else {
- acpi_os_printf("%s\n", (char *)ret_buf.pointer);
+ acpi_os_printf("%s: %s\n",
+ acpi_ut_get_type_name(obj_desc->
+ reference.node->
+ type),
+ (char *)ret_buf.pointer);
ACPI_FREE(ret_buf.pointer);
}
} else if (obj_desc->reference.object) {
case ACPI_TYPE_LOCAL_REFERENCE:
- acpi_os_printf("[Object Reference] Type [%s] %2.2X",
- acpi_ut_get_reference_name(obj_desc),
- obj_desc->reference.class);
+ acpi_os_printf("[Object Reference] Class [%s]",
+ acpi_ut_get_reference_name(obj_desc));
acpi_ex_dump_reference_obj(obj_desc);
break;
* This is a deref_of (object_reference)
* Get the actual object from the Node (This is the dereference).
* This case may only happen when a local_x or arg_x is
- * dereferenced above.
+ * dereferenced above, or for references to device and
+ * thermal objects.
*/
- return_desc = acpi_ns_get_attached_object((struct
- acpi_namespace_node
- *)
- operand[0]);
- acpi_ut_add_reference(return_desc);
+ switch (((struct acpi_namespace_node *)operand[0])->
+ type) {
+ case ACPI_TYPE_DEVICE:
+ case ACPI_TYPE_THERMAL:
+
+ /* These types have no node subobject, return the NS node */
+
+ return_desc = operand[0];
+ break;
+
+ default:
+ /* For most types, get the object attached to the node */
+
+ return_desc = acpi_ns_get_attached_object((struct acpi_namespace_node *)operand[0]);
+ acpi_ut_add_reference(return_desc);
+ break;
+ }
} else {
/*
* This must be a reference object produced by either the
*)obj_desc);
}
- if (!obj_desc) {
- ACPI_ERROR((AE_INFO,
- "[%4.4s] Node is unresolved or uninitialized",
- acpi_ut_get_node_name(node)));
- return_ACPI_STATUS(AE_AML_UNINITIALIZED_NODE);
+ switch (type) {
+ case ACPI_TYPE_DEVICE:
+ case ACPI_TYPE_THERMAL:
+
+ /* These types have no attached subobject */
+ break;
+
+ default:
+
+ /* All other types require a subobject */
+
+ if (!obj_desc) {
+ ACPI_ERROR((AE_INFO,
+ "[%4.4s] Node is unresolved or uninitialized",
+ acpi_ut_get_node_name(node)));
+ return_ACPI_STATUS(AE_AML_UNINITIALIZED_NODE);
+ }
+ break;
}
break;
/* Legacy functions are optional, based upon ACPI_REDUCED_HARDWARE */
static struct acpi_sleep_functions acpi_sleep_dispatch[] = {
- {ACPI_HW_OPTIONAL_FUNCTION(acpi_hw_legacy_sleep),
- acpi_hw_extended_sleep},
- {ACPI_HW_OPTIONAL_FUNCTION(acpi_hw_legacy_wake_prep),
- acpi_hw_extended_wake_prep},
- {ACPI_HW_OPTIONAL_FUNCTION(acpi_hw_legacy_wake), acpi_hw_extended_wake}
+ {ACPI_STRUCT_INIT(legacy_function,
+ ACPI_HW_OPTIONAL_FUNCTION(acpi_hw_legacy_sleep)),
+ ACPI_STRUCT_INIT(extended_function, acpi_hw_extended_sleep) },
+ {ACPI_STRUCT_INIT(legacy_function,
+ ACPI_HW_OPTIONAL_FUNCTION(acpi_hw_legacy_wake_prep)),
+ ACPI_STRUCT_INIT(extended_function, acpi_hw_extended_wake_prep) },
+ {ACPI_STRUCT_INIT(legacy_function,
+ ACPI_HW_OPTIONAL_FUNCTION(acpi_hw_legacy_wake)),
+ ACPI_STRUCT_INIT(extended_function, acpi_hw_extended_wake) }
};
/*
#include "acnamesp.h"
#include "acdispat.h"
+#ifdef ACPI_ASL_COMPILER
+#include "acdisasm.h"
+#endif
+
#define _COMPONENT ACPI_NAMESPACE
ACPI_MODULE_NAME("nsaccess")
(char *)¤t_node->name,
current_node));
}
+#ifdef ACPI_ASL_COMPILER
+ /*
+ * If this ACPI name already exists within the namespace as an
+ * external declaration, then mark the external as a conflicting
+ * declaration and proceed to process the current node as if it did
+ * not exist in the namespace. If this node is not processed as
+ * normal, then it could cause improper namespace resolution
+ * by failing to open a new scope.
+ */
+ if (acpi_gbl_disasm_flag &&
+ (status == AE_ALREADY_EXISTS) &&
+ ((this_node->flags & ANOBJ_IS_EXTERNAL) ||
+ (walk_state
+ && walk_state->opcode == AML_EXTERNAL_OP))) {
+ this_node->flags &= ~ANOBJ_IS_EXTERNAL;
+ this_node->type = (u8)this_search_type;
+ if (walk_state->opcode != AML_EXTERNAL_OP) {
+ acpi_dm_mark_external_conflict
+ (this_node);
+ }
+ break;
+ }
+#endif
*return_node = this_node;
return_ACPI_STATUS(status);
(void)acpi_ns_build_normalized_path(node, buffer->pointer,
required_size, no_trailing);
- if (ACPI_FAILURE(status)) {
- return_ACPI_STATUS(status);
- }
ACPI_DEBUG_PRINT((ACPI_DB_EXEC, "%s [%X]\n",
(char *)buffer->pointer, (u32) required_size));
acpi_os_printf("%s ", message);
}
- acpi_os_printf("[%s] (Node %p)", (char *)buffer.pointer, node);
+ acpi_os_printf("%s", (char *)buffer.pointer);
ACPI_FREE(buffer.pointer);
}
}
{
acpi_status status;
u8 free_buffer_on_error = FALSE;
+ acpi_handle target_handle;
+ char *full_pathname;
ACPI_FUNCTION_TRACE(acpi_evaluate_object_typed);
free_buffer_on_error = TRUE;
}
+ status = acpi_get_handle(handle, pathname, &target_handle);
+ if (ACPI_FAILURE(status)) {
+ return_ACPI_STATUS(status);
+ }
+
+ full_pathname = acpi_ns_get_external_pathname(target_handle);
+ if (!full_pathname) {
+ return_ACPI_STATUS(AE_NO_MEMORY);
+ }
+
/* Evaluate the object */
- status = acpi_evaluate_object(handle, pathname,
- external_params, return_buffer);
+ status = acpi_evaluate_object(target_handle, NULL, external_params,
+ return_buffer);
if (ACPI_FAILURE(status)) {
- return_ACPI_STATUS(status);
+ goto exit;
}
- /* Type ANY means "don't care" */
+ /* Type ANY means "don't care about return value type" */
if (return_type == ACPI_TYPE_ANY) {
- return_ACPI_STATUS(AE_OK);
+ goto exit;
}
if (return_buffer->length == 0) {
/* Error because caller specifically asked for a return value */
- ACPI_ERROR((AE_INFO, "No return value"));
- return_ACPI_STATUS(AE_NULL_OBJECT);
+ ACPI_ERROR((AE_INFO, "%s did not return any object",
+ full_pathname));
+ status = AE_NULL_OBJECT;
+ goto exit;
}
/* Examine the object type returned from evaluate_object */
if (((union acpi_object *)return_buffer->pointer)->type == return_type) {
- return_ACPI_STATUS(AE_OK);
+ goto exit;
}
/* Return object type does not match requested type */
ACPI_ERROR((AE_INFO,
- "Incorrect return type [%s] requested [%s]",
+ "Incorrect return type from %s - received [%s], requested [%s]",
+ full_pathname,
acpi_ut_get_type_name(((union acpi_object *)return_buffer->
pointer)->type),
acpi_ut_get_type_name(return_type)));
}
return_buffer->length = 0;
- return_ACPI_STATUS(AE_TYPE);
+ status = AE_TYPE;
+
+exit:
+ ACPI_FREE(full_pathname);
+ return_ACPI_STATUS(status);
}
ACPI_EXPORT_SYMBOL(acpi_evaluate_object_typed)
(u32)(aml_offset +
sizeof(struct acpi_table_header)));
+ ACPI_ERROR((AE_INFO,
+ "Aborting disassembly, AML byte code is corrupt"));
+
/* Dump the context surrounding the invalid opcode */
acpi_ut_dump_buffer(((u8 *)walk_state->parser_state.
sizeof(struct acpi_table_header) -
16));
acpi_os_printf(" */\n");
+
+ /*
+ * Just abort the disassembly, cannot continue because the
+ * parser is essentially lost. The disassembler can then
+ * randomly fail because an ill-constructed parse tree
+ * can result.
+ */
+ return_ACPI_STATUS(AE_AML_BAD_OPCODE);
#endif
}
if (status == AE_CTRL_PARSE_CONTINUE) {
return_ACPI_STATUS(AE_CTRL_PARSE_CONTINUE);
}
+ if (ACPI_FAILURE(status)) {
+ return_ACPI_STATUS(status);
+ }
/* Create Op structure and append to parent's argument list */
/* ACPI 6.0 opcodes */
- /* 81 */ ACPI_OP("External", ARGP_EXTERNAL_OP, ARGI_EXTERNAL_OP,
- ACPI_TYPE_ANY, AML_CLASS_EXECUTE, /* ? */
- AML_TYPE_EXEC_3A_0T_0R, AML_FLAGS_EXEC_3A_0T_0R),
+/* 81 */ ACPI_OP("External", ARGP_EXTERNAL_OP, ARGI_EXTERNAL_OP,
+ ACPI_TYPE_ANY, AML_CLASS_NAMED_OBJECT,
+ AML_TYPE_NAMED_SIMPLE,
+ AML_HAS_ARGS | AML_NSOBJECT | AML_NSOPCODE |
+ AML_NSNODE | AML_NAMED),
/* 82 */ ACPI_OP("Comment", ARGP_COMMENT_OP, ARGI_COMMENT_OP,
ACPI_TYPE_STRING, AML_CLASS_ARGUMENT,
AML_TYPE_LITERAL, AML_CONSTANT)
#include "acdispat.h"
#include "amlcode.h"
#include "acinterp.h"
+#include "acnamesp.h"
#define _COMPONENT ACPI_PARSER
ACPI_MODULE_NAME("psparse")
/* Either the method parse or actual execution failed */
acpi_ex_exit_interpreter();
- ACPI_ERROR_METHOD("Method parse/execution failed",
- walk_state->method_node, NULL,
- status);
+ if (status == AE_ABORT_METHOD) {
+ acpi_ns_print_node_pathname(walk_state->
+ method_node,
+ "Method aborted:");
+ acpi_os_printf("\n");
+ } else {
+ ACPI_ERROR_METHOD
+ ("Method parse/execution failed",
+ walk_state->method_node, NULL, status);
+ }
acpi_ex_enter_interpreter();
/* Check for possible multi-thread reentrancy problem */
break;
+ case ACPI_RESOURCE_TYPE_PIN_FUNCTION:
+
+ total_size = (acpi_rs_length)(total_size +
+ (resource->data.
+ pin_function.
+ pin_table_length * 2) +
+ resource->data.
+ pin_function.
+ resource_source.
+ string_length +
+ resource->data.
+ pin_function.
+ vendor_length);
+
+ break;
+
case ACPI_RESOURCE_TYPE_SERIAL_BUS:
total_size =
break;
+ case ACPI_RESOURCE_TYPE_PIN_CONFIG:
+
+ total_size = (acpi_rs_length)(total_size +
+ (resource->data.
+ pin_config.
+ pin_table_length * 2) +
+ resource->data.pin_config.
+ resource_source.
+ string_length +
+ resource->data.pin_config.
+ vendor_length);
+
+ break;
+
+ case ACPI_RESOURCE_TYPE_PIN_GROUP:
+
+ total_size = (acpi_rs_length)(total_size +
+ (resource->data.pin_group.
+ pin_table_length * 2) +
+ resource->data.pin_group.
+ resource_label.
+ string_length +
+ resource->data.pin_group.
+ vendor_length);
+
+ break;
+
+ case ACPI_RESOURCE_TYPE_PIN_GROUP_FUNCTION:
+
+ total_size = (acpi_rs_length)(total_size +
+ resource->data.
+ pin_group_function.
+ resource_source.
+ string_length +
+ resource->data.
+ pin_group_function.
+ resource_source_label.
+ string_length +
+ resource->data.
+ pin_group_function.
+ vendor_length);
+
+ break;
+
+ case ACPI_RESOURCE_TYPE_PIN_GROUP_CONFIG:
+
+ total_size = (acpi_rs_length)(total_size +
+ resource->data.
+ pin_group_config.
+ resource_source.
+ string_length +
+ resource->data.
+ pin_group_config.
+ resource_source_label.
+ string_length +
+ resource->data.
+ pin_group_config.
+ vendor_length);
+
+ break;
+
default:
break;
}
break;
+ case ACPI_RESOURCE_NAME_PIN_FUNCTION:
+
+ /* Vendor data is optional */
+
+ if (aml_resource->pin_function.vendor_length) {
+ extra_struct_bytes +=
+ aml_resource->pin_function.vendor_offset -
+ aml_resource->pin_function.
+ pin_table_offset +
+ aml_resource->pin_function.vendor_length;
+ } else {
+ extra_struct_bytes +=
+ aml_resource->large_header.resource_length +
+ sizeof(struct aml_resource_large_header) -
+ aml_resource->pin_function.pin_table_offset;
+ }
+ break;
+
case ACPI_RESOURCE_NAME_SERIAL_BUS:
minimum_aml_resource_length =
minimum_aml_resource_length;
break;
+ case ACPI_RESOURCE_NAME_PIN_CONFIG:
+
+ /* Vendor data is optional */
+
+ if (aml_resource->pin_config.vendor_length) {
+ extra_struct_bytes +=
+ aml_resource->pin_config.vendor_offset -
+ aml_resource->pin_config.pin_table_offset +
+ aml_resource->pin_config.vendor_length;
+ } else {
+ extra_struct_bytes +=
+ aml_resource->large_header.resource_length +
+ sizeof(struct aml_resource_large_header) -
+ aml_resource->pin_config.pin_table_offset;
+ }
+ break;
+
+ case ACPI_RESOURCE_NAME_PIN_GROUP:
+
+ extra_struct_bytes +=
+ aml_resource->pin_group.vendor_offset -
+ aml_resource->pin_group.pin_table_offset +
+ aml_resource->pin_group.vendor_length;
+
+ break;
+
+ case ACPI_RESOURCE_NAME_PIN_GROUP_FUNCTION:
+
+ extra_struct_bytes +=
+ aml_resource->pin_group_function.vendor_offset -
+ aml_resource->pin_group_function.res_source_offset +
+ aml_resource->pin_group_function.vendor_length;
+
+ break;
+
+ case ACPI_RESOURCE_NAME_PIN_GROUP_CONFIG:
+
+ extra_struct_bytes +=
+ aml_resource->pin_group_config.vendor_offset -
+ aml_resource->pin_group_config.res_source_offset +
+ aml_resource->pin_group_config.vendor_length;
+
+ break;
+
default:
break;
static void
acpi_rs_dump_resource_source(struct acpi_resource_source *resource_source);
+static void
+acpi_rs_dump_resource_label(char *title,
+ struct acpi_resource_label *resource_label);
+
static void acpi_rs_dump_address_common(union acpi_resource_data *resource);
static void
target));
break;
+ case ACPI_RSD_LABEL:
+ /*
+ * resource_label
+ */
+ acpi_rs_dump_resource_label("Resource Label",
+ ACPI_CAST_PTR(struct
+ acpi_resource_label,
+ target));
+ break;
+
+ case ACPI_RSD_SOURCE_LABEL:
+ /*
+ * resource_source_label
+ */
+ acpi_rs_dump_resource_label("Resource Source Label",
+ ACPI_CAST_PTR(struct
+ acpi_resource_label,
+ target));
+ break;
+
default:
acpi_os_printf("**** Invalid table opcode [%X] ****\n",
resource_source->string_ptr : "[Not Specified]");
}
+/*******************************************************************************
+ *
+ * FUNCTION: acpi_rs_dump_resource_label
+ *
+ * PARAMETERS: title - Title of the dumped resource field
+ * resource_label - Pointer to a Resource Label struct
+ *
+ * RETURN: None
+ *
+ * DESCRIPTION: Common routine for dumping the resource_label
+ *
+ ******************************************************************************/
+
+static void
+acpi_rs_dump_resource_label(char *title,
+ struct acpi_resource_label *resource_label)
+{
+ ACPI_FUNCTION_ENTRY();
+
+ acpi_rs_out_string(title,
+ resource_label->string_ptr ?
+ resource_label->string_ptr : "[Not Specified]");
+}
+
/*******************************************************************************
*
* FUNCTION: acpi_rs_dump_address_common
NULL},
};
+struct acpi_rsdump_info acpi_rs_dump_pin_function[10] = {
+ {ACPI_RSD_TITLE, ACPI_RSD_TABLE_SIZE(acpi_rs_dump_pin_function),
+ "PinFunction", NULL},
+ {ACPI_RSD_UINT8, ACPI_RSD_OFFSET(pin_function.revision_id),
+ "RevisionId", NULL},
+ {ACPI_RSD_UINT8, ACPI_RSD_OFFSET(pin_function.pin_config), "PinConfig",
+ acpi_gbl_ppc_decode},
+ {ACPI_RSD_1BITFLAG, ACPI_RSD_OFFSET(pin_function.sharable), "Sharing",
+ acpi_gbl_shr_decode},
+ {ACPI_RSD_UINT16, ACPI_RSD_OFFSET(pin_function.function_number),
+ "FunctionNumber", NULL},
+ {ACPI_RSD_SOURCE, ACPI_RSD_OFFSET(pin_function.resource_source),
+ "ResourceSource", NULL},
+ {ACPI_RSD_UINT16, ACPI_RSD_OFFSET(pin_function.pin_table_length),
+ "PinTableLength", NULL},
+ {ACPI_RSD_WORDLIST, ACPI_RSD_OFFSET(pin_function.pin_table), "PinTable",
+ NULL},
+ {ACPI_RSD_UINT16, ACPI_RSD_OFFSET(pin_function.vendor_length),
+ "VendorLength", NULL},
+ {ACPI_RSD_SHORTLISTX, ACPI_RSD_OFFSET(pin_function.vendor_data),
+ "VendorData", NULL},
+};
+
+struct acpi_rsdump_info acpi_rs_dump_pin_config[11] = {
+ {ACPI_RSD_TITLE, ACPI_RSD_TABLE_SIZE(acpi_rs_dump_pin_config),
+ "PinConfig", NULL},
+ {ACPI_RSD_UINT8, ACPI_RSD_OFFSET(pin_config.revision_id), "RevisionId",
+ NULL},
+ {ACPI_RSD_1BITFLAG, ACPI_RSD_OFFSET(pin_config.producer_consumer),
+ "ProducerConsumer", acpi_gbl_consume_decode},
+ {ACPI_RSD_1BITFLAG, ACPI_RSD_OFFSET(pin_config.sharable), "Sharing",
+ acpi_gbl_shr_decode},
+ {ACPI_RSD_UINT8, ACPI_RSD_OFFSET(pin_config.pin_config_type),
+ "PinConfigType", NULL},
+ {ACPI_RSD_UINT32, ACPI_RSD_OFFSET(pin_config.pin_config_value),
+ "PinConfigValue", NULL},
+ {ACPI_RSD_SOURCE, ACPI_RSD_OFFSET(pin_config.resource_source),
+ "ResourceSource", NULL},
+ {ACPI_RSD_UINT16, ACPI_RSD_OFFSET(pin_config.pin_table_length),
+ "PinTableLength", NULL},
+ {ACPI_RSD_WORDLIST, ACPI_RSD_OFFSET(pin_config.pin_table), "PinTable",
+ NULL},
+ {ACPI_RSD_UINT16, ACPI_RSD_OFFSET(pin_config.vendor_length),
+ "VendorLength", NULL},
+ {ACPI_RSD_SHORTLISTX, ACPI_RSD_OFFSET(pin_config.vendor_data),
+ "VendorData", NULL},
+};
+
+struct acpi_rsdump_info acpi_rs_dump_pin_group[8] = {
+ {ACPI_RSD_TITLE, ACPI_RSD_TABLE_SIZE(acpi_rs_dump_pin_group),
+ "PinGroup", NULL},
+ {ACPI_RSD_UINT8, ACPI_RSD_OFFSET(pin_group.revision_id), "RevisionId",
+ NULL},
+ {ACPI_RSD_1BITFLAG, ACPI_RSD_OFFSET(pin_group.producer_consumer),
+ "ProducerConsumer", acpi_gbl_consume_decode},
+ {ACPI_RSD_UINT16, ACPI_RSD_OFFSET(pin_group.pin_table_length),
+ "PinTableLength", NULL},
+ {ACPI_RSD_WORDLIST, ACPI_RSD_OFFSET(pin_group.pin_table), "PinTable",
+ NULL},
+ {ACPI_RSD_LABEL, ACPI_RSD_OFFSET(pin_group.resource_label),
+ "ResourceLabel", NULL},
+ {ACPI_RSD_UINT16, ACPI_RSD_OFFSET(pin_group.vendor_length),
+ "VendorLength", NULL},
+ {ACPI_RSD_SHORTLISTX, ACPI_RSD_OFFSET(pin_group.vendor_data),
+ "VendorData", NULL},
+};
+
+struct acpi_rsdump_info acpi_rs_dump_pin_group_function[9] = {
+ {ACPI_RSD_TITLE, ACPI_RSD_TABLE_SIZE(acpi_rs_dump_pin_group_function),
+ "PinGroupFunction", NULL},
+ {ACPI_RSD_UINT8, ACPI_RSD_OFFSET(pin_group_function.revision_id),
+ "RevisionId", NULL},
+ {ACPI_RSD_1BITFLAG,
+ ACPI_RSD_OFFSET(pin_group_function.producer_consumer),
+ "ProducerConsumer", acpi_gbl_consume_decode},
+ {ACPI_RSD_1BITFLAG, ACPI_RSD_OFFSET(pin_group_function.sharable),
+ "Sharing", acpi_gbl_shr_decode},
+ {ACPI_RSD_UINT16, ACPI_RSD_OFFSET(pin_group_function.function_number),
+ "FunctionNumber", NULL},
+ {ACPI_RSD_SOURCE_LABEL,
+ ACPI_RSD_OFFSET(pin_group_function.resource_source_label),
+ "ResourceSourceLabel", NULL},
+ {ACPI_RSD_SOURCE, ACPI_RSD_OFFSET(pin_group_function.resource_source),
+ "ResourceSource", NULL},
+ {ACPI_RSD_UINT16, ACPI_RSD_OFFSET(pin_group_function.vendor_length),
+ "VendorLength", NULL},
+ {ACPI_RSD_SHORTLISTX, ACPI_RSD_OFFSET(pin_group_function.vendor_data),
+ "VendorData", NULL},
+};
+
+struct acpi_rsdump_info acpi_rs_dump_pin_group_config[10] = {
+ {ACPI_RSD_TITLE, ACPI_RSD_TABLE_SIZE(acpi_rs_dump_pin_group_config),
+ "PinGroupConfig", NULL},
+ {ACPI_RSD_UINT8, ACPI_RSD_OFFSET(pin_group_config.revision_id),
+ "RevisionId", NULL},
+ {ACPI_RSD_1BITFLAG, ACPI_RSD_OFFSET(pin_group_config.producer_consumer),
+ "ProducerConsumer", acpi_gbl_consume_decode},
+ {ACPI_RSD_1BITFLAG, ACPI_RSD_OFFSET(pin_group_config.sharable),
+ "Sharing", acpi_gbl_shr_decode},
+ {ACPI_RSD_UINT8, ACPI_RSD_OFFSET(pin_group_config.pin_config_type),
+ "PinConfigType", NULL},
+ {ACPI_RSD_UINT32, ACPI_RSD_OFFSET(pin_group_config.pin_config_value),
+ "PinConfigValue", NULL},
+ {ACPI_RSD_SOURCE_LABEL,
+ ACPI_RSD_OFFSET(pin_group_config.resource_source_label),
+ "ResourceSourceLabel", NULL},
+ {ACPI_RSD_SOURCE, ACPI_RSD_OFFSET(pin_group_config.resource_source),
+ "ResourceSource", NULL},
+ {ACPI_RSD_UINT16, ACPI_RSD_OFFSET(pin_group_config.vendor_length),
+ "VendorLength", NULL},
+ {ACPI_RSD_SHORTLISTX, ACPI_RSD_OFFSET(pin_group_config.vendor_data),
+ "VendorData", NULL},
+};
+
struct acpi_rsdump_info acpi_rs_dump_fixed_dma[4] = {
{ACPI_RSD_TITLE, ACPI_RSD_TABLE_SIZE(acpi_rs_dump_fixed_dma),
"FixedDma", NULL},
acpi_rs_convert_gpio, /* 0x11, ACPI_RESOURCE_TYPE_GPIO */
acpi_rs_convert_fixed_dma, /* 0x12, ACPI_RESOURCE_TYPE_FIXED_DMA */
NULL, /* 0x13, ACPI_RESOURCE_TYPE_SERIAL_BUS - Use subtype table below */
+ acpi_rs_convert_pin_function, /* 0x14, ACPI_RESOURCE_TYPE_PIN_FUNCTION */
+ acpi_rs_convert_pin_config, /* 0x15, ACPI_RESOURCE_TYPE_PIN_CONFIG */
+ acpi_rs_convert_pin_group, /* 0x16, ACPI_RESOURCE_TYPE_PIN_GROUP */
+ acpi_rs_convert_pin_group_function, /* 0x17, ACPI_RESOURCE_TYPE_PIN_GROUP_FUNCTION */
+ acpi_rs_convert_pin_group_config, /* 0x18, ACPI_RESOURCE_TYPE_PIN_GROUP_CONFIG */
};
/* Dispatch tables for AML-to-resource (Get Resource) conversion functions */
acpi_rs_convert_address64, /* 0x0A, ACPI_RESOURCE_NAME_ADDRESS64 */
acpi_rs_convert_ext_address64, /* 0x0B, ACPI_RESOURCE_NAME_EXTENDED_ADDRESS64 */
acpi_rs_convert_gpio, /* 0x0C, ACPI_RESOURCE_NAME_GPIO */
- NULL, /* 0x0D, Reserved */
+ acpi_rs_convert_pin_function, /* 0x0D, ACPI_RESOURCE_NAME_PIN_FUNCTION */
NULL, /* 0x0E, ACPI_RESOURCE_NAME_SERIAL_BUS - Use subtype table below */
+ acpi_rs_convert_pin_config, /* 0x0F, ACPI_RESOURCE_NAME_PIN_CONFIG */
+ acpi_rs_convert_pin_group, /* 0x10, ACPI_RESOURCE_NAME_PIN_GROUP */
+ acpi_rs_convert_pin_group_function, /* 0x11, ACPI_RESOURCE_NAME_PIN_GROUP_FUNCTION */
+ acpi_rs_convert_pin_group_config, /* 0x12, ACPI_RESOURCE_NAME_PIN_GROUP_CONFIG */
};
/* Subtype table for serial_bus -- I2C, SPI, and UART */
acpi_rs_dump_gpio, /* ACPI_RESOURCE_TYPE_GPIO */
acpi_rs_dump_fixed_dma, /* ACPI_RESOURCE_TYPE_FIXED_DMA */
NULL, /* ACPI_RESOURCE_TYPE_SERIAL_BUS */
+ acpi_rs_dump_pin_function, /* ACPI_RESOURCE_TYPE_PIN_FUNCTION */
+ acpi_rs_dump_pin_config, /* ACPI_RESOURCE_TYPE_PIN_CONFIG */
+ acpi_rs_dump_pin_group, /* ACPI_RESOURCE_TYPE_PIN_GROUP */
+ acpi_rs_dump_pin_group_function, /* ACPI_RESOURCE_TYPE_PIN_GROUP_FUNCTION */
+ acpi_rs_dump_pin_group_config, /* ACPI_RESOURCE_TYPE_PIN_GROUP_CONFIG */
};
struct acpi_rsdump_info *acpi_gbl_dump_serial_bus_dispatch[] = {
sizeof(struct aml_resource_gpio), /* ACPI_RESOURCE_TYPE_GPIO */
sizeof(struct aml_resource_fixed_dma), /* ACPI_RESOURCE_TYPE_FIXED_DMA */
sizeof(struct aml_resource_common_serialbus), /* ACPI_RESOURCE_TYPE_SERIAL_BUS */
+ sizeof(struct aml_resource_pin_function), /* ACPI_RESOURCE_TYPE_PIN_FUNCTION */
+ sizeof(struct aml_resource_pin_config), /* ACPI_RESOURCE_TYPE_PIN_CONFIG */
+ sizeof(struct aml_resource_pin_group), /* ACPI_RESOURCE_TYPE_PIN_GROUP */
+ sizeof(struct aml_resource_pin_group_function), /* ACPI_RESOURCE_TYPE_PIN_GROUP_FUNCTION */
+ sizeof(struct aml_resource_pin_group_config), /* ACPI_RESOURCE_TYPE_PIN_GROUP_CONFIG */
};
const u8 acpi_gbl_resource_struct_sizes[] = {
ACPI_RS_SIZE(struct acpi_resource_address64),
ACPI_RS_SIZE(struct acpi_resource_extended_address64),
ACPI_RS_SIZE(struct acpi_resource_gpio),
- ACPI_RS_SIZE(struct acpi_resource_common_serialbus)
+ ACPI_RS_SIZE(struct acpi_resource_pin_function),
+ ACPI_RS_SIZE(struct acpi_resource_common_serialbus),
+ ACPI_RS_SIZE(struct acpi_resource_pin_config),
+ ACPI_RS_SIZE(struct acpi_resource_pin_group),
+ ACPI_RS_SIZE(struct acpi_resource_pin_group_function),
+ ACPI_RS_SIZE(struct acpi_resource_pin_group_config),
};
const u8 acpi_gbl_aml_resource_serial_bus_sizes[] = {
/* Set vendor offset only if there is vendor data */
- if (resource->data.gpio.vendor_length) {
- ACPI_SET16(target, aml_length);
- }
+ ACPI_SET16(target, aml_length);
acpi_rs_set_resource_length(aml_length, aml);
break;
0},
};
+/*******************************************************************************
+ *
+ * acpi_rs_convert_pinfunction
+ *
+ ******************************************************************************/
+
+struct acpi_rsconvert_info acpi_rs_convert_pin_function[13] = {
+ {ACPI_RSC_INITGET, ACPI_RESOURCE_TYPE_PIN_FUNCTION,
+ ACPI_RS_SIZE(struct acpi_resource_pin_function),
+ ACPI_RSC_TABLE_SIZE(acpi_rs_convert_pin_function)},
+
+ {ACPI_RSC_INITSET, ACPI_RESOURCE_NAME_PIN_FUNCTION,
+ sizeof(struct aml_resource_pin_function),
+ 0},
+
+ {ACPI_RSC_MOVE8, ACPI_RS_OFFSET(data.pin_function.revision_id),
+ AML_OFFSET(pin_function.revision_id),
+ 1},
+
+ {ACPI_RSC_1BITFLAG, ACPI_RS_OFFSET(data.pin_function.sharable),
+ AML_OFFSET(pin_function.flags),
+ 0},
+
+ {ACPI_RSC_MOVE8, ACPI_RS_OFFSET(data.pin_function.pin_config),
+ AML_OFFSET(pin_function.pin_config),
+ 1},
+
+ {ACPI_RSC_MOVE16, ACPI_RS_OFFSET(data.pin_function.function_number),
+ AML_OFFSET(pin_function.function_number),
+ 2},
+
+ /* Pin Table */
+
+ /*
+ * It is OK to use GPIO operations here because none of them refer GPIO
+ * structures directly but instead use offsets given here.
+ */
+
+ {ACPI_RSC_COUNT_GPIO_PIN,
+ ACPI_RS_OFFSET(data.pin_function.pin_table_length),
+ AML_OFFSET(pin_function.pin_table_offset),
+ AML_OFFSET(pin_function.res_source_offset)},
+
+ {ACPI_RSC_MOVE_GPIO_PIN, ACPI_RS_OFFSET(data.pin_function.pin_table),
+ AML_OFFSET(pin_function.pin_table_offset),
+ 0},
+
+ /* Resource Source */
+
+ {ACPI_RSC_MOVE8,
+ ACPI_RS_OFFSET(data.pin_function.resource_source.index),
+ AML_OFFSET(pin_function.res_source_index),
+ 1},
+
+ {ACPI_RSC_COUNT_GPIO_RES,
+ ACPI_RS_OFFSET(data.pin_function.resource_source.string_length),
+ AML_OFFSET(pin_function.res_source_offset),
+ AML_OFFSET(pin_function.vendor_offset)},
+
+ {ACPI_RSC_MOVE_GPIO_RES,
+ ACPI_RS_OFFSET(data.pin_function.resource_source.string_ptr),
+ AML_OFFSET(pin_function.res_source_offset),
+ 0},
+
+ /* Vendor Data */
+
+ {ACPI_RSC_COUNT_GPIO_VEN,
+ ACPI_RS_OFFSET(data.pin_function.vendor_length),
+ AML_OFFSET(pin_function.vendor_length),
+ 1},
+
+ {ACPI_RSC_MOVE_GPIO_RES, ACPI_RS_OFFSET(data.pin_function.vendor_data),
+ AML_OFFSET(pin_function.vendor_offset),
+ 0},
+};
+
/*******************************************************************************
*
* acpi_rs_convert_i2c_serial_bus
AML_OFFSET(uart_serial_bus.default_baud_rate),
1},
};
+
+/*******************************************************************************
+ *
+ * acpi_rs_convert_pin_config
+ *
+ ******************************************************************************/
+
+struct acpi_rsconvert_info acpi_rs_convert_pin_config[14] = {
+ {ACPI_RSC_INITGET, ACPI_RESOURCE_TYPE_PIN_CONFIG,
+ ACPI_RS_SIZE(struct acpi_resource_pin_config),
+ ACPI_RSC_TABLE_SIZE(acpi_rs_convert_pin_config)},
+
+ {ACPI_RSC_INITSET, ACPI_RESOURCE_NAME_PIN_CONFIG,
+ sizeof(struct aml_resource_pin_config),
+ 0},
+
+ {ACPI_RSC_MOVE8, ACPI_RS_OFFSET(data.pin_config.revision_id),
+ AML_OFFSET(pin_config.revision_id),
+ 1},
+
+ {ACPI_RSC_1BITFLAG, ACPI_RS_OFFSET(data.pin_config.sharable),
+ AML_OFFSET(pin_config.flags),
+ 0},
+
+ {ACPI_RSC_1BITFLAG, ACPI_RS_OFFSET(data.pin_config.producer_consumer),
+ AML_OFFSET(pin_config.flags),
+ 1},
+
+ {ACPI_RSC_MOVE8, ACPI_RS_OFFSET(data.pin_config.pin_config_type),
+ AML_OFFSET(pin_config.pin_config_type),
+ 1},
+
+ {ACPI_RSC_MOVE32, ACPI_RS_OFFSET(data.pin_config.pin_config_value),
+ AML_OFFSET(pin_config.pin_config_value),
+ 1},
+
+ /* Pin Table */
+
+ /*
+ * It is OK to use GPIO operations here because none of them refer GPIO
+ * structures directly but instead use offsets given here.
+ */
+
+ {ACPI_RSC_COUNT_GPIO_PIN,
+ ACPI_RS_OFFSET(data.pin_config.pin_table_length),
+ AML_OFFSET(pin_config.pin_table_offset),
+ AML_OFFSET(pin_config.res_source_offset)},
+
+ {ACPI_RSC_MOVE_GPIO_PIN, ACPI_RS_OFFSET(data.pin_config.pin_table),
+ AML_OFFSET(pin_config.pin_table_offset),
+ 0},
+
+ /* Resource Source */
+
+ {ACPI_RSC_MOVE8, ACPI_RS_OFFSET(data.pin_config.resource_source.index),
+ AML_OFFSET(pin_config.res_source_index),
+ 1},
+
+ {ACPI_RSC_COUNT_GPIO_RES,
+ ACPI_RS_OFFSET(data.pin_config.resource_source.string_length),
+ AML_OFFSET(pin_config.res_source_offset),
+ AML_OFFSET(pin_config.vendor_offset)},
+
+ {ACPI_RSC_MOVE_GPIO_RES,
+ ACPI_RS_OFFSET(data.pin_config.resource_source.string_ptr),
+ AML_OFFSET(pin_config.res_source_offset),
+ 0},
+
+ /* Vendor Data */
+
+ {ACPI_RSC_COUNT_GPIO_VEN, ACPI_RS_OFFSET(data.pin_config.vendor_length),
+ AML_OFFSET(pin_config.vendor_length),
+ 1},
+
+ {ACPI_RSC_MOVE_GPIO_RES, ACPI_RS_OFFSET(data.pin_config.vendor_data),
+ AML_OFFSET(pin_config.vendor_offset),
+ 0},
+};
+
+/*******************************************************************************
+ *
+ * acpi_rs_convert_pin_group
+ *
+ ******************************************************************************/
+
+struct acpi_rsconvert_info acpi_rs_convert_pin_group[10] = {
+ {ACPI_RSC_INITGET, ACPI_RESOURCE_TYPE_PIN_GROUP,
+ ACPI_RS_SIZE(struct acpi_resource_pin_group),
+ ACPI_RSC_TABLE_SIZE(acpi_rs_convert_pin_group)},
+
+ {ACPI_RSC_INITSET, ACPI_RESOURCE_NAME_PIN_GROUP,
+ sizeof(struct aml_resource_pin_group),
+ 0},
+
+ {ACPI_RSC_MOVE8, ACPI_RS_OFFSET(data.pin_group.revision_id),
+ AML_OFFSET(pin_group.revision_id),
+ 1},
+
+ {ACPI_RSC_1BITFLAG, ACPI_RS_OFFSET(data.pin_group.producer_consumer),
+ AML_OFFSET(pin_group.flags),
+ 0},
+
+ /* Pin Table */
+
+ /*
+ * It is OK to use GPIO operations here because none of them refer GPIO
+ * structures directly but instead use offsets given here.
+ */
+
+ {ACPI_RSC_COUNT_GPIO_PIN,
+ ACPI_RS_OFFSET(data.pin_group.pin_table_length),
+ AML_OFFSET(pin_group.pin_table_offset),
+ AML_OFFSET(pin_group.label_offset)},
+
+ {ACPI_RSC_MOVE_GPIO_PIN, ACPI_RS_OFFSET(data.pin_group.pin_table),
+ AML_OFFSET(pin_group.pin_table_offset),
+ 0},
+
+ /* Resource Label */
+
+ {ACPI_RSC_COUNT_GPIO_RES,
+ ACPI_RS_OFFSET(data.pin_group.resource_label.string_length),
+ AML_OFFSET(pin_group.label_offset),
+ AML_OFFSET(pin_group.vendor_offset)},
+
+ {ACPI_RSC_MOVE_GPIO_RES,
+ ACPI_RS_OFFSET(data.pin_group.resource_label.string_ptr),
+ AML_OFFSET(pin_group.label_offset),
+ 0},
+
+ /* Vendor Data */
+
+ {ACPI_RSC_COUNT_GPIO_VEN, ACPI_RS_OFFSET(data.pin_group.vendor_length),
+ AML_OFFSET(pin_group.vendor_length),
+ 1},
+
+ {ACPI_RSC_MOVE_GPIO_RES, ACPI_RS_OFFSET(data.pin_group.vendor_data),
+ AML_OFFSET(pin_group.vendor_offset),
+ 0},
+};
+
+/*******************************************************************************
+ *
+ * acpi_rs_convert_pin_group_function
+ *
+ ******************************************************************************/
+
+struct acpi_rsconvert_info acpi_rs_convert_pin_group_function[13] = {
+ {ACPI_RSC_INITGET, ACPI_RESOURCE_TYPE_PIN_GROUP_FUNCTION,
+ ACPI_RS_SIZE(struct acpi_resource_pin_group_function),
+ ACPI_RSC_TABLE_SIZE(acpi_rs_convert_pin_group_function)},
+
+ {ACPI_RSC_INITSET, ACPI_RESOURCE_NAME_PIN_GROUP_FUNCTION,
+ sizeof(struct aml_resource_pin_group_function),
+ 0},
+
+ {ACPI_RSC_MOVE8, ACPI_RS_OFFSET(data.pin_group_function.revision_id),
+ AML_OFFSET(pin_group_function.revision_id),
+ 1},
+
+ {ACPI_RSC_1BITFLAG, ACPI_RS_OFFSET(data.pin_group_function.sharable),
+ AML_OFFSET(pin_group_function.flags),
+ 0},
+
+ {ACPI_RSC_1BITFLAG,
+ ACPI_RS_OFFSET(data.pin_group_function.producer_consumer),
+ AML_OFFSET(pin_group_function.flags),
+ 1},
+
+ {ACPI_RSC_MOVE16,
+ ACPI_RS_OFFSET(data.pin_group_function.function_number),
+ AML_OFFSET(pin_group_function.function_number),
+ 1},
+
+ /* Resource Source */
+
+ {ACPI_RSC_MOVE8,
+ ACPI_RS_OFFSET(data.pin_group_function.resource_source.index),
+ AML_OFFSET(pin_group_function.res_source_index),
+ 1},
+
+ {ACPI_RSC_COUNT_GPIO_RES,
+ ACPI_RS_OFFSET(data.pin_group_function.resource_source.string_length),
+ AML_OFFSET(pin_group_function.res_source_offset),
+ AML_OFFSET(pin_group_function.res_source_label_offset)},
+
+ {ACPI_RSC_MOVE_GPIO_RES,
+ ACPI_RS_OFFSET(data.pin_group_function.resource_source.string_ptr),
+ AML_OFFSET(pin_group_function.res_source_offset),
+ 0},
+
+ /* Resource Source Label */
+
+ {ACPI_RSC_COUNT_GPIO_RES,
+ ACPI_RS_OFFSET(data.pin_group_function.resource_source_label.
+ string_length),
+ AML_OFFSET(pin_group_function.res_source_label_offset),
+ AML_OFFSET(pin_group_function.vendor_offset)},
+
+ {ACPI_RSC_MOVE_GPIO_RES,
+ ACPI_RS_OFFSET(data.pin_group_function.resource_source_label.
+ string_ptr),
+ AML_OFFSET(pin_group_function.res_source_label_offset),
+ 0},
+
+ /* Vendor Data */
+
+ {ACPI_RSC_COUNT_GPIO_VEN,
+ ACPI_RS_OFFSET(data.pin_group_function.vendor_length),
+ AML_OFFSET(pin_group_function.vendor_length),
+ 1},
+
+ {ACPI_RSC_MOVE_GPIO_RES,
+ ACPI_RS_OFFSET(data.pin_group_function.vendor_data),
+ AML_OFFSET(pin_group_function.vendor_offset),
+ 0},
+};
+
+/*******************************************************************************
+ *
+ * acpi_rs_convert_pin_group_config
+ *
+ ******************************************************************************/
+
+struct acpi_rsconvert_info acpi_rs_convert_pin_group_config[14] = {
+ {ACPI_RSC_INITGET, ACPI_RESOURCE_TYPE_PIN_GROUP_CONFIG,
+ ACPI_RS_SIZE(struct acpi_resource_pin_group_config),
+ ACPI_RSC_TABLE_SIZE(acpi_rs_convert_pin_group_config)},
+
+ {ACPI_RSC_INITSET, ACPI_RESOURCE_NAME_PIN_GROUP_CONFIG,
+ sizeof(struct aml_resource_pin_group_config),
+ 0},
+
+ {ACPI_RSC_MOVE8, ACPI_RS_OFFSET(data.pin_group_config.revision_id),
+ AML_OFFSET(pin_group_config.revision_id),
+ 1},
+
+ {ACPI_RSC_1BITFLAG, ACPI_RS_OFFSET(data.pin_group_config.sharable),
+ AML_OFFSET(pin_group_config.flags),
+ 0},
+
+ {ACPI_RSC_1BITFLAG,
+ ACPI_RS_OFFSET(data.pin_group_config.producer_consumer),
+ AML_OFFSET(pin_group_config.flags),
+ 1},
+
+ {ACPI_RSC_MOVE8, ACPI_RS_OFFSET(data.pin_group_config.pin_config_type),
+ AML_OFFSET(pin_group_config.pin_config_type),
+ 1},
+
+ {ACPI_RSC_MOVE32,
+ ACPI_RS_OFFSET(data.pin_group_config.pin_config_value),
+ AML_OFFSET(pin_group_config.pin_config_value),
+ 1},
+
+ /* Resource Source */
+
+ {ACPI_RSC_MOVE8,
+ ACPI_RS_OFFSET(data.pin_group_config.resource_source.index),
+ AML_OFFSET(pin_group_config.res_source_index),
+ 1},
+
+ {ACPI_RSC_COUNT_GPIO_RES,
+ ACPI_RS_OFFSET(data.pin_group_config.resource_source.string_length),
+ AML_OFFSET(pin_group_config.res_source_offset),
+ AML_OFFSET(pin_group_config.res_source_label_offset)},
+
+ {ACPI_RSC_MOVE_GPIO_RES,
+ ACPI_RS_OFFSET(data.pin_group_config.resource_source.string_ptr),
+ AML_OFFSET(pin_group_config.res_source_offset),
+ 0},
+
+ /* Resource Source Label */
+
+ {ACPI_RSC_COUNT_GPIO_RES,
+ ACPI_RS_OFFSET(data.pin_group_config.resource_source_label.
+ string_length),
+ AML_OFFSET(pin_group_config.res_source_label_offset),
+ AML_OFFSET(pin_group_config.vendor_offset)},
+
+ {ACPI_RSC_MOVE_GPIO_RES,
+ ACPI_RS_OFFSET(data.pin_group_config.resource_source_label.string_ptr),
+ AML_OFFSET(pin_group_config.res_source_label_offset),
+ 0},
+
+ /* Vendor Data */
+
+ {ACPI_RSC_COUNT_GPIO_VEN,
+ ACPI_RS_OFFSET(data.pin_group_config.vendor_length),
+ AML_OFFSET(pin_group_config.vendor_length),
+ 1},
+
+ {ACPI_RSC_MOVE_GPIO_RES,
+ ACPI_RS_OFFSET(data.pin_group_config.vendor_data),
+ AML_OFFSET(pin_group_config.vendor_offset),
+ 0},
+};
return_ACPI_STATUS(status);
}
+ACPI_EXPORT_SYMBOL(acpi_tb_install_and_load_table)
+
/*******************************************************************************
*
* FUNCTION: acpi_tb_unload_table
acpi_tb_set_table_loaded_flag(table_index, FALSE);
return_ACPI_STATUS(status);
}
+
+ACPI_EXPORT_SYMBOL(acpi_tb_unload_table)
* The 64-bit X fields are optional extensions to the original 32-bit FADT
* V1.0 fields. Even if they are present in the FADT, they are optional and
* are unused if the BIOS sets them to zero. Therefore, we must copy/expand
- * 32-bit V1.0 fields to the 64-bit X fields if the the 64-bit X field is
- * originally zero.
+ * 32-bit V1.0 fields to the 64-bit X fields if the 64-bit X field is originally
+ * zero.
*
* For ACPI 1.0 FADTs (that contain no 64-bit addresses), all 32-bit address
* fields are expanded to the corresponding 64-bit X fields in the internal
*
* FUNCTION: acpi_tb_copy_dsdt
*
- * PARAMETERS: table_desc - Installed table to copy
+ * PARAMETERS: table_index - Index of installed table to copy
*
- * RETURN: None
+ * RETURN: The copied DSDT
*
* DESCRIPTION: Implements a subsystem option to copy the DSDT to local memory.
* Some very bad BIOSs are known to either corrupt the DSDT or
*
* FUNCTION: acpi_tb_parse_root_table
*
- * PARAMETERS: rsdp - Pointer to the RSDP
+ * PARAMETERS: rsdp_address - Pointer to the RSDP
*
* RETURN: Status
*
/* 09 */ "Device PLD Check",
/* 0A */ "Reserved",
/* 0B */ "System Locality Update",
- /* 0C */ "Shutdown Request",
- /* Reserved in ACPI 6.0 */
- /* 0D */ "System Resource Affinity Update"
+ /* 0C */ "Reserved (was previously Shutdown Request)",
+ /* Reserved in ACPI 6.0 */
+ /* 0D */ "System Resource Affinity Update",
+ /* 0E */ "Heterogeneous Memory Attributes Update"
+ /* ACPI 6.2 */
};
static const char *acpi_gbl_device_notify[5] = {
break;
}
- if (!(acpi_gbl_owner_id_mask[j] & (1 << k))) {
+ /*
+ * Note: the u32 cast ensures that 1 is stored as a unsigned
+ * integer. Omitting the cast may result in 1 being stored as an
+ * int. Some compilers or runtime error detection may flag this as
+ * an error.
+ */
+ if (!(acpi_gbl_owner_id_mask[j] & ((u32)1 << k))) {
/*
* Found a free ID. The actual ID is the bit index plus one,
* making zero an invalid Owner ID. Save this as the last ID
* allocated and update the global ID mask.
*/
- acpi_gbl_owner_id_mask[j] |= (1 << k);
+ acpi_gbl_owner_id_mask[j] |= ((u32)1 << k);
acpi_gbl_last_owner_id_index = (u8)j;
acpi_gbl_next_owner_id_offset = (u8)(k + 1);
/* Decode ID to index/offset pair */
index = ACPI_DIV_32(owner_id);
- bit = 1 << ACPI_MOD_32(owner_id);
+ bit = (u32)1 << ACPI_MOD_32(owner_id);
/* Free the owner ID only if it is valid */
--- /dev/null
+/*******************************************************************************
+ *
+ * Module Name: utresdecode - Resource descriptor keyword strings
+ *
+ ******************************************************************************/
+
+/*
+ * Copyright (C) 2000 - 2017, Intel Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions, and the following disclaimer,
+ * without modification.
+ * 2. Redistributions in binary form must reproduce at minimum a disclaimer
+ * substantially similar to the "NO WARRANTY" disclaimer below
+ * ("Disclaimer") and any redistribution must be conditioned upon
+ * including a substantially similar Disclaimer requirement for further
+ * binary redistribution.
+ * 3. Neither the names of the above-listed copyright holders nor the names
+ * of any contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * Alternatively, this software may be distributed under the terms of the
+ * GNU General Public License ("GPL") version 2 as published by the Free
+ * Software Foundation.
+ *
+ * NO WARRANTY
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTIBILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * HOLDERS OR CONTRIBUTORS BE LIABLE FOR SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
+ * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGES.
+ */
+
+#include <acpi/acpi.h>
+#include "accommon.h"
+#include "acresrc.h"
+
+#define _COMPONENT ACPI_UTILITIES
+ACPI_MODULE_NAME("utresdecode")
+
+#if defined (ACPI_DEBUG_OUTPUT) || \
+ defined (ACPI_DISASSEMBLER) || \
+ defined (ACPI_DEBUGGER)
+/*
+ * Strings used to decode resource descriptors.
+ * Used by both the disassembler and the debugger resource dump routines
+ */
+const char *acpi_gbl_bm_decode[] = {
+ "NotBusMaster",
+ "BusMaster"
+};
+
+const char *acpi_gbl_config_decode[] = {
+ "0 - Good Configuration",
+ "1 - Acceptable Configuration",
+ "2 - Suboptimal Configuration",
+ "3 - ***Invalid Configuration***",
+};
+
+const char *acpi_gbl_consume_decode[] = {
+ "ResourceProducer",
+ "ResourceConsumer"
+};
+
+const char *acpi_gbl_dec_decode[] = {
+ "PosDecode",
+ "SubDecode"
+};
+
+const char *acpi_gbl_he_decode[] = {
+ "Level",
+ "Edge"
+};
+
+const char *acpi_gbl_io_decode[] = {
+ "Decode10",
+ "Decode16"
+};
+
+const char *acpi_gbl_ll_decode[] = {
+ "ActiveHigh",
+ "ActiveLow",
+ "ActiveBoth",
+ "Reserved"
+};
+
+const char *acpi_gbl_max_decode[] = {
+ "MaxNotFixed",
+ "MaxFixed"
+};
+
+const char *acpi_gbl_mem_decode[] = {
+ "NonCacheable",
+ "Cacheable",
+ "WriteCombining",
+ "Prefetchable"
+};
+
+const char *acpi_gbl_min_decode[] = {
+ "MinNotFixed",
+ "MinFixed"
+};
+
+const char *acpi_gbl_mtp_decode[] = {
+ "AddressRangeMemory",
+ "AddressRangeReserved",
+ "AddressRangeACPI",
+ "AddressRangeNVS"
+};
+
+const char *acpi_gbl_rng_decode[] = {
+ "InvalidRanges",
+ "NonISAOnlyRanges",
+ "ISAOnlyRanges",
+ "EntireRange"
+};
+
+const char *acpi_gbl_rw_decode[] = {
+ "ReadOnly",
+ "ReadWrite"
+};
+
+const char *acpi_gbl_shr_decode[] = {
+ "Exclusive",
+ "Shared",
+ "ExclusiveAndWake", /* ACPI 5.0 */
+ "SharedAndWake" /* ACPI 5.0 */
+};
+
+const char *acpi_gbl_siz_decode[] = {
+ "Transfer8",
+ "Transfer8_16",
+ "Transfer16",
+ "InvalidSize"
+};
+
+const char *acpi_gbl_trs_decode[] = {
+ "DenseTranslation",
+ "SparseTranslation"
+};
+
+const char *acpi_gbl_ttp_decode[] = {
+ "TypeStatic",
+ "TypeTranslation"
+};
+
+const char *acpi_gbl_typ_decode[] = {
+ "Compatibility",
+ "TypeA",
+ "TypeB",
+ "TypeF"
+};
+
+const char *acpi_gbl_ppc_decode[] = {
+ "PullDefault",
+ "PullUp",
+ "PullDown",
+ "PullNone"
+};
+
+const char *acpi_gbl_ior_decode[] = {
+ "IoRestrictionNone",
+ "IoRestrictionInputOnly",
+ "IoRestrictionOutputOnly",
+ "IoRestrictionNoneAndPreserve"
+};
+
+const char *acpi_gbl_dts_decode[] = {
+ "Width8bit",
+ "Width16bit",
+ "Width32bit",
+ "Width64bit",
+ "Width128bit",
+ "Width256bit",
+};
+
+/* GPIO connection type */
+
+const char *acpi_gbl_ct_decode[] = {
+ "Interrupt",
+ "I/O"
+};
+
+/* Serial bus type */
+
+const char *acpi_gbl_sbt_decode[] = {
+ "/* UNKNOWN serial bus type */",
+ "I2C",
+ "SPI",
+ "UART"
+};
+
+/* I2C serial bus access mode */
+
+const char *acpi_gbl_am_decode[] = {
+ "AddressingMode7Bit",
+ "AddressingMode10Bit"
+};
+
+/* I2C serial bus slave mode */
+
+const char *acpi_gbl_sm_decode[] = {
+ "ControllerInitiated",
+ "DeviceInitiated"
+};
+
+/* SPI serial bus wire mode */
+
+const char *acpi_gbl_wm_decode[] = {
+ "FourWireMode",
+ "ThreeWireMode"
+};
+
+/* SPI serial clock phase */
+
+const char *acpi_gbl_cph_decode[] = {
+ "ClockPhaseFirst",
+ "ClockPhaseSecond"
+};
+
+/* SPI serial bus clock polarity */
+
+const char *acpi_gbl_cpo_decode[] = {
+ "ClockPolarityLow",
+ "ClockPolarityHigh"
+};
+
+/* SPI serial bus device polarity */
+
+const char *acpi_gbl_dp_decode[] = {
+ "PolarityLow",
+ "PolarityHigh"
+};
+
+/* UART serial bus endian */
+
+const char *acpi_gbl_ed_decode[] = {
+ "LittleEndian",
+ "BigEndian"
+};
+
+/* UART serial bus bits per byte */
+
+const char *acpi_gbl_bpb_decode[] = {
+ "DataBitsFive",
+ "DataBitsSix",
+ "DataBitsSeven",
+ "DataBitsEight",
+ "DataBitsNine",
+ "/* UNKNOWN Bits per byte */",
+ "/* UNKNOWN Bits per byte */",
+ "/* UNKNOWN Bits per byte */"
+};
+
+/* UART serial bus stop bits */
+
+const char *acpi_gbl_sb_decode[] = {
+ "StopBitsZero",
+ "StopBitsOne",
+ "StopBitsOnePlusHalf",
+ "StopBitsTwo"
+};
+
+/* UART serial bus flow control */
+
+const char *acpi_gbl_fc_decode[] = {
+ "FlowControlNone",
+ "FlowControlHardware",
+ "FlowControlXON",
+ "/* UNKNOWN flow control keyword */"
+};
+
+/* UART serial bus parity type */
+
+const char *acpi_gbl_pt_decode[] = {
+ "ParityTypeNone",
+ "ParityTypeEven",
+ "ParityTypeOdd",
+ "ParityTypeMark",
+ "ParityTypeSpace",
+ "/* UNKNOWN parity keyword */",
+ "/* UNKNOWN parity keyword */",
+ "/* UNKNOWN parity keyword */"
+};
+
+/* pin_config type */
+
+const char *acpi_gbl_ptyp_decode[] = {
+ "Default",
+ "Bias Pull-up",
+ "Bias Pull-down",
+ "Bias Default",
+ "Bias Disable",
+ "Bias High Impedance",
+ "Bias Bus Hold",
+ "Drive Open Drain",
+ "Drive Open Source",
+ "Drive Push Pull",
+ "Drive Strength",
+ "Slew Rate",
+ "Input Debounce",
+ "Input Schmitt Trigger",
+};
+
+#endif
#define _COMPONENT ACPI_UTILITIES
ACPI_MODULE_NAME("utresrc")
-#if defined(ACPI_DEBUG_OUTPUT) || defined (ACPI_DISASSEMBLER) || defined (ACPI_DEBUGGER)
-/*
- * Strings used to decode resource descriptors.
- * Used by both the disassembler and the debugger resource dump routines
- */
-const char *acpi_gbl_bm_decode[] = {
- "NotBusMaster",
- "BusMaster"
-};
-
-const char *acpi_gbl_config_decode[] = {
- "0 - Good Configuration",
- "1 - Acceptable Configuration",
- "2 - Suboptimal Configuration",
- "3 - ***Invalid Configuration***",
-};
-
-const char *acpi_gbl_consume_decode[] = {
- "ResourceProducer",
- "ResourceConsumer"
-};
-
-const char *acpi_gbl_dec_decode[] = {
- "PosDecode",
- "SubDecode"
-};
-
-const char *acpi_gbl_he_decode[] = {
- "Level",
- "Edge"
-};
-
-const char *acpi_gbl_io_decode[] = {
- "Decode10",
- "Decode16"
-};
-
-const char *acpi_gbl_ll_decode[] = {
- "ActiveHigh",
- "ActiveLow",
- "ActiveBoth",
- "Reserved"
-};
-
-const char *acpi_gbl_max_decode[] = {
- "MaxNotFixed",
- "MaxFixed"
-};
-
-const char *acpi_gbl_mem_decode[] = {
- "NonCacheable",
- "Cacheable",
- "WriteCombining",
- "Prefetchable"
-};
-
-const char *acpi_gbl_min_decode[] = {
- "MinNotFixed",
- "MinFixed"
-};
-
-const char *acpi_gbl_mtp_decode[] = {
- "AddressRangeMemory",
- "AddressRangeReserved",
- "AddressRangeACPI",
- "AddressRangeNVS"
-};
-
-const char *acpi_gbl_rng_decode[] = {
- "InvalidRanges",
- "NonISAOnlyRanges",
- "ISAOnlyRanges",
- "EntireRange"
-};
-
-const char *acpi_gbl_rw_decode[] = {
- "ReadOnly",
- "ReadWrite"
-};
-
-const char *acpi_gbl_shr_decode[] = {
- "Exclusive",
- "Shared",
- "ExclusiveAndWake", /* ACPI 5.0 */
- "SharedAndWake" /* ACPI 5.0 */
-};
-
-const char *acpi_gbl_siz_decode[] = {
- "Transfer8",
- "Transfer8_16",
- "Transfer16",
- "InvalidSize"
-};
-
-const char *acpi_gbl_trs_decode[] = {
- "DenseTranslation",
- "SparseTranslation"
-};
-
-const char *acpi_gbl_ttp_decode[] = {
- "TypeStatic",
- "TypeTranslation"
-};
-
-const char *acpi_gbl_typ_decode[] = {
- "Compatibility",
- "TypeA",
- "TypeB",
- "TypeF"
-};
-
-const char *acpi_gbl_ppc_decode[] = {
- "PullDefault",
- "PullUp",
- "PullDown",
- "PullNone"
-};
-
-const char *acpi_gbl_ior_decode[] = {
- "IoRestrictionNone",
- "IoRestrictionInputOnly",
- "IoRestrictionOutputOnly",
- "IoRestrictionNoneAndPreserve"
-};
-
-const char *acpi_gbl_dts_decode[] = {
- "Width8bit",
- "Width16bit",
- "Width32bit",
- "Width64bit",
- "Width128bit",
- "Width256bit",
-};
-
-/* GPIO connection type */
-
-const char *acpi_gbl_ct_decode[] = {
- "Interrupt",
- "I/O"
-};
-
-/* Serial bus type */
-
-const char *acpi_gbl_sbt_decode[] = {
- "/* UNKNOWN serial bus type */",
- "I2C",
- "SPI",
- "UART"
-};
-
-/* I2C serial bus access mode */
-
-const char *acpi_gbl_am_decode[] = {
- "AddressingMode7Bit",
- "AddressingMode10Bit"
-};
-
-/* I2C serial bus slave mode */
-
-const char *acpi_gbl_sm_decode[] = {
- "ControllerInitiated",
- "DeviceInitiated"
-};
-
-/* SPI serial bus wire mode */
-
-const char *acpi_gbl_wm_decode[] = {
- "FourWireMode",
- "ThreeWireMode"
-};
-
-/* SPI serial clock phase */
-
-const char *acpi_gbl_cph_decode[] = {
- "ClockPhaseFirst",
- "ClockPhaseSecond"
-};
-
-/* SPI serial bus clock polarity */
-
-const char *acpi_gbl_cpo_decode[] = {
- "ClockPolarityLow",
- "ClockPolarityHigh"
-};
-
-/* SPI serial bus device polarity */
-
-const char *acpi_gbl_dp_decode[] = {
- "PolarityLow",
- "PolarityHigh"
-};
-
-/* UART serial bus endian */
-
-const char *acpi_gbl_ed_decode[] = {
- "LittleEndian",
- "BigEndian"
-};
-
-/* UART serial bus bits per byte */
-
-const char *acpi_gbl_bpb_decode[] = {
- "DataBitsFive",
- "DataBitsSix",
- "DataBitsSeven",
- "DataBitsEight",
- "DataBitsNine",
- "/* UNKNOWN Bits per byte */",
- "/* UNKNOWN Bits per byte */",
- "/* UNKNOWN Bits per byte */"
-};
-
-/* UART serial bus stop bits */
-
-const char *acpi_gbl_sb_decode[] = {
- "StopBitsZero",
- "StopBitsOne",
- "StopBitsOnePlusHalf",
- "StopBitsTwo"
-};
-
-/* UART serial bus flow control */
-
-const char *acpi_gbl_fc_decode[] = {
- "FlowControlNone",
- "FlowControlHardware",
- "FlowControlXON",
- "/* UNKNOWN flow control keyword */"
-};
-
-/* UART serial bus parity type */
-
-const char *acpi_gbl_pt_decode[] = {
- "ParityTypeNone",
- "ParityTypeEven",
- "ParityTypeOdd",
- "ParityTypeMark",
- "ParityTypeSpace",
- "/* UNKNOWN parity keyword */",
- "/* UNKNOWN parity keyword */",
- "/* UNKNOWN parity keyword */"
-};
-
-#endif
-
/*
* Base sizes of the raw AML resource descriptors, indexed by resource type.
* Zero indicates a reserved (and therefore invalid) resource type.
ACPI_AML_SIZE_LARGE(struct aml_resource_address64),
ACPI_AML_SIZE_LARGE(struct aml_resource_extended_address64),
ACPI_AML_SIZE_LARGE(struct aml_resource_gpio),
- 0,
+ ACPI_AML_SIZE_LARGE(struct aml_resource_pin_function),
ACPI_AML_SIZE_LARGE(struct aml_resource_common_serialbus),
+ ACPI_AML_SIZE_LARGE(struct aml_resource_pin_config),
+ ACPI_AML_SIZE_LARGE(struct aml_resource_pin_group),
+ ACPI_AML_SIZE_LARGE(struct aml_resource_pin_group_function),
+ ACPI_AML_SIZE_LARGE(struct aml_resource_pin_group_config),
};
const u8 acpi_gbl_resource_aml_serial_bus_sizes[] = {
ACPI_VARIABLE_LENGTH, /* 0A Qword* address */
ACPI_FIXED_LENGTH, /* 0B Extended* address */
ACPI_VARIABLE_LENGTH, /* 0C Gpio* */
- 0,
- ACPI_VARIABLE_LENGTH /* 0E *serial_bus */
+ ACPI_VARIABLE_LENGTH, /* 0D pin_function */
+ ACPI_VARIABLE_LENGTH, /* 0E *serial_bus */
+ ACPI_VARIABLE_LENGTH, /* 0F pin_config */
+ ACPI_VARIABLE_LENGTH, /* 10 pin_group */
+ ACPI_VARIABLE_LENGTH, /* 11 pin_group_function */
+ ACPI_VARIABLE_LENGTH, /* 12 pin_group_config */
};
/*******************************************************************************
return (status);
}
+ACPI_EXPORT_SYMBOL(acpi_acquire_mutex)
+
/*******************************************************************************
*
* FUNCTION: acpi_release_mutex
* not both.
*
******************************************************************************/
-
acpi_status acpi_release_mutex(acpi_handle handle, acpi_string pathname)
{
acpi_status status;
acpi_os_release_mutex(mutex_obj->mutex.os_mutex);
return (AE_OK);
}
+
+ACPI_EXPORT_SYMBOL(acpi_release_mutex)
module_param_named(disable, ghes_disable, bool, 0);
/*
- * All error sources notified with SCI shares one notifier function,
- * so they need to be linked and checked one by one. This is applied
- * to NMI too.
+ * All error sources notified with HED (Hardware Error Device) share a
+ * single notifier callback, so they need to be linked and checked one
+ * by one. This holds true for NMI too.
*
* RCU is used for these lists, so ghes_list_mutex is only used for
* list changing, not for traversing.
*/
-static LIST_HEAD(ghes_sci);
+static LIST_HEAD(ghes_hed);
static DEFINE_MUTEX(ghes_list_mutex);
/*
{
int sev, sec_sev;
struct acpi_hest_generic_data *gdata;
+ guid_t *sec_type;
sev = ghes_severity(estatus->error_severity);
apei_estatus_for_each_section(estatus, gdata) {
+ sec_type = (guid_t *)gdata->section_type;
sec_sev = ghes_severity(gdata->error_severity);
- if (!uuid_le_cmp(*(uuid_le *)gdata->section_type,
- CPER_SEC_PLATFORM_MEM)) {
+ if (guid_equal(sec_type, &CPER_SEC_PLATFORM_MEM)) {
struct cper_sec_mem_err *mem_err;
mem_err = (struct cper_sec_mem_err *)(gdata+1);
ghes_edac_report_mem_error(ghes, sev, mem_err);
ghes_handle_memory_failure(gdata, sev);
}
#ifdef CONFIG_ACPI_APEI_PCIEAER
- else if (!uuid_le_cmp(*(uuid_le *)gdata->section_type,
- CPER_SEC_PCIE)) {
+ else if (guid_equal(sec_type, &CPER_SEC_PCIE)) {
struct cper_sec_pcie *pcie_err;
pcie_err = (struct cper_sec_pcie *)(gdata+1);
if (sev == GHES_SEV_RECOVERABLE &&
return IRQ_HANDLED;
}
-static int ghes_notify_sci(struct notifier_block *this,
- unsigned long event, void *data)
+static int ghes_notify_hed(struct notifier_block *this, unsigned long event,
+ void *data)
{
struct ghes *ghes;
int ret = NOTIFY_DONE;
rcu_read_lock();
- list_for_each_entry_rcu(ghes, &ghes_sci, list) {
+ list_for_each_entry_rcu(ghes, &ghes_hed, list) {
if (!ghes_proc(ghes))
ret = NOTIFY_OK;
}
return ret;
}
-static struct notifier_block ghes_notifier_sci = {
- .notifier_call = ghes_notify_sci,
+static struct notifier_block ghes_notifier_hed = {
+ .notifier_call = ghes_notify_hed,
};
#ifdef CONFIG_HAVE_ACPI_APEI_NMI
case ACPI_HEST_NOTIFY_POLLED:
case ACPI_HEST_NOTIFY_EXTERNAL:
case ACPI_HEST_NOTIFY_SCI:
+ case ACPI_HEST_NOTIFY_GSIV:
+ case ACPI_HEST_NOTIFY_GPIO:
break;
+
case ACPI_HEST_NOTIFY_NMI:
if (!IS_ENABLED(CONFIG_HAVE_ACPI_APEI_NMI)) {
pr_warn(GHES_PFX "Generic hardware error source: %d notified via NMI interrupt is not supported!\n",
goto err_edac_unreg;
}
break;
+
case ACPI_HEST_NOTIFY_SCI:
+ case ACPI_HEST_NOTIFY_GSIV:
+ case ACPI_HEST_NOTIFY_GPIO:
mutex_lock(&ghes_list_mutex);
- if (list_empty(&ghes_sci))
- register_acpi_hed_notifier(&ghes_notifier_sci);
- list_add_rcu(&ghes->list, &ghes_sci);
+ if (list_empty(&ghes_hed))
+ register_acpi_hed_notifier(&ghes_notifier_hed);
+ list_add_rcu(&ghes->list, &ghes_hed);
mutex_unlock(&ghes_list_mutex);
break;
+
case ACPI_HEST_NOTIFY_NMI:
ghes_nmi_add(ghes);
break;
case ACPI_HEST_NOTIFY_EXTERNAL:
free_irq(ghes->irq, ghes);
break;
+
case ACPI_HEST_NOTIFY_SCI:
+ case ACPI_HEST_NOTIFY_GSIV:
+ case ACPI_HEST_NOTIFY_GPIO:
mutex_lock(&ghes_list_mutex);
list_del_rcu(&ghes->list);
- if (list_empty(&ghes_sci))
- unregister_acpi_hed_notifier(&ghes_notifier_sci);
+ if (list_empty(&ghes_hed))
+ unregister_acpi_hed_notifier(&ghes_notifier_hed);
mutex_unlock(&ghes_list_mutex);
synchronize_rcu();
break;
+
case ACPI_HEST_NOTIFY_NMI:
ghes_nmi_remove(ghes);
break;
if ((battery->state & ACPI_BATTERY_STATE_CRITICAL) ||
(test_bit(ACPI_BATTERY_ALARM_PRESENT, &battery->flags) &&
(battery->capacity_now <= battery->alarm)))
- pm_wakeup_event(&battery->device->dev, 0);
+ acpi_pm_wakeup_event(&battery->device->dev);
return result;
}
pr_debug("\n");
}
-acpi_status acpi_str_to_uuid(char *str, u8 *uuid)
-{
- int i;
- static int opc_map_to_uuid[16] = {6, 4, 2, 0, 11, 9, 16, 14, 19, 21,
- 24, 26, 28, 30, 32, 34};
-
- if (strlen(str) != 36)
- return AE_BAD_PARAMETER;
- for (i = 0; i < 36; i++) {
- if (i == 8 || i == 13 || i == 18 || i == 23) {
- if (str[i] != '-')
- return AE_BAD_PARAMETER;
- } else if (!isxdigit(str[i]))
- return AE_BAD_PARAMETER;
- }
- for (i = 0; i < 16; i++) {
- uuid[i] = hex_to_bin(str[opc_map_to_uuid[i]]) << 4;
- uuid[i] |= hex_to_bin(str[opc_map_to_uuid[i] + 1]);
- }
- return AE_OK;
-}
-EXPORT_SYMBOL_GPL(acpi_str_to_uuid);
-
acpi_status acpi_run_osc(acpi_handle handle, struct acpi_osc_context *context)
{
acpi_status status;
struct acpi_object_list input;
union acpi_object in_params[4];
union acpi_object *out_obj;
- u8 uuid[16];
+ guid_t guid;
u32 errors;
struct acpi_buffer output = {ACPI_ALLOCATE_BUFFER, NULL};
if (!context)
return AE_ERROR;
- if (ACPI_FAILURE(acpi_str_to_uuid(context->uuid_str, uuid)))
+ if (guid_parse(context->uuid_str, &guid))
return AE_ERROR;
context->ret.length = ACPI_ALLOCATE_BUFFER;
context->ret.pointer = NULL;
input.pointer = in_params;
in_params[0].type = ACPI_TYPE_BUFFER;
in_params[0].buffer.length = 16;
- in_params[0].buffer.pointer = uuid;
+ in_params[0].buffer.pointer = (u8 *)&guid;
in_params[1].type = ACPI_TYPE_INTEGER;
in_params[1].integer.value = context->rev;
in_params[2].type = ACPI_TYPE_INTEGER;
(driver->flags & ACPI_DRIVER_ALL_NOTIFY_EVENTS))
driver->ops.notify(adev, type);
- if (hotplug_event && ACPI_SUCCESS(acpi_hotplug_schedule(adev, type)))
+ if (!hotplug_event) {
+ acpi_bus_put_acpi_device(adev);
+ return;
+ }
+
+ if (ACPI_SUCCESS(acpi_hotplug_schedule(adev, type)))
return;
acpi_bus_put_acpi_device(adev);
- return;
err:
acpi_evaluate_ost(handle, type, ost_code, NULL);
}
if (state)
- pm_wakeup_event(&device->dev, 0);
+ acpi_pm_wakeup_event(&device->dev);
ret = blocking_notifier_call_chain(&acpi_lid_notifier, state, device);
if (ret == NOTIFY_DONE)
} else {
int keycode;
- pm_wakeup_event(&device->dev, 0);
+ acpi_pm_wakeup_event(&device->dev);
if (button->suspended)
break;
lid_device = device;
}
+ device_init_wakeup(&device->dev, true);
printk(KERN_INFO PREFIX "%s [%s]\n", name, acpi_device_bid(device));
return 0;
#include <linux/pm_qos.h>
#include <linux/pm_domain.h>
#include <linux/pm_runtime.h>
+#include <linux/suspend.h>
#include "internal.h"
#ifdef CONFIG_PM
static DEFINE_MUTEX(acpi_pm_notifier_lock);
+void acpi_pm_wakeup_event(struct device *dev)
+{
+ pm_wakeup_dev_event(dev, 0, acpi_s2idle_wakeup());
+}
+EXPORT_SYMBOL_GPL(acpi_pm_wakeup_event);
+
static void acpi_pm_notify_handler(acpi_handle handle, u32 val, void *not_used)
{
struct acpi_device *adev;
mutex_lock(&acpi_pm_notifier_lock);
if (adev->wakeup.flags.notifier_present) {
- __pm_wakeup_event(adev->wakeup.ws, 0);
- if (adev->wakeup.context.work.func)
- queue_pm_work(&adev->wakeup.context.work);
+ pm_wakeup_ws_event(adev->wakeup.ws, 0, acpi_s2idle_wakeup());
+ if (adev->wakeup.context.func)
+ adev->wakeup.context.func(&adev->wakeup.context);
}
mutex_unlock(&acpi_pm_notifier_lock);
* acpi_add_pm_notifier - Register PM notify handler for given ACPI device.
* @adev: ACPI device to add the notify handler for.
* @dev: Device to generate a wakeup event for while handling the notification.
- * @work_func: Work function to execute when handling the notification.
+ * @func: Work function to execute when handling the notification.
*
* NOTE: @adev need not be a run-wake or wakeup device to be a valid source of
* PM wakeup events. For example, wakeup events may be generated for bridges
* bridge itself doesn't have a wakeup GPE associated with it.
*/
acpi_status acpi_add_pm_notifier(struct acpi_device *adev, struct device *dev,
- void (*work_func)(struct work_struct *work))
+ void (*func)(struct acpi_device_wakeup_context *context))
{
acpi_status status = AE_ALREADY_EXISTS;
- if (!dev && !work_func)
+ if (!dev && !func)
return AE_BAD_PARAMETER;
mutex_lock(&acpi_pm_notifier_lock);
adev->wakeup.ws = wakeup_source_register(dev_name(&adev->dev));
adev->wakeup.context.dev = dev;
- if (work_func)
- INIT_WORK(&adev->wakeup.context.work, work_func);
+ adev->wakeup.context.func = func;
status = acpi_install_notify_handler(adev->handle, ACPI_SYSTEM_NOTIFY,
acpi_pm_notify_handler, NULL);
if (ACPI_FAILURE(status))
goto out;
- if (adev->wakeup.context.work.func) {
- cancel_work_sync(&adev->wakeup.context.work);
- adev->wakeup.context.work.func = NULL;
- }
+ adev->wakeup.context.func = NULL;
adev->wakeup.context.dev = NULL;
wakeup_source_unregister(adev->wakeup.ws);
}
EXPORT_SYMBOL(acpi_bus_can_wakeup);
+bool acpi_pm_device_can_wakeup(struct device *dev)
+{
+ struct acpi_device *adev = ACPI_COMPANION(dev);
+
+ return adev ? acpi_device_can_wakeup(adev) : false;
+}
+
/**
* acpi_dev_pm_get_state - Get preferred power state of ACPI device.
* @dev: Device whose preferred target power state to return.
/**
* acpi_pm_notify_work_func - ACPI devices wakeup notification work function.
- * @work: Work item to handle.
+ * @context: Device wakeup context.
*/
-static void acpi_pm_notify_work_func(struct work_struct *work)
+static void acpi_pm_notify_work_func(struct acpi_device_wakeup_context *context)
{
- struct device *dev;
+ struct device *dev = context->dev;
- dev = container_of(work, struct acpi_device_wakeup_context, work)->dev;
if (dev) {
pm_wakeup_event(dev, 0);
- pm_runtime_resume(dev);
+ pm_request_resume(dev);
}
}
acpi_status res;
int error;
+ if (adev->wakeup.flags.enabled)
+ return 0;
+
error = acpi_enable_wakeup_device_power(adev, target_state);
if (error)
return error;
- if (adev->wakeup.flags.enabled)
- return 0;
-
res = acpi_enable_gpe(wakeup->gpe_device, wakeup->gpe_number);
- if (ACPI_SUCCESS(res)) {
- adev->wakeup.flags.enabled = 1;
- } else {
+ if (ACPI_FAILURE(res)) {
acpi_disable_wakeup_device_power(adev);
return -EIO;
}
- } else {
- if (adev->wakeup.flags.enabled) {
- acpi_disable_gpe(wakeup->gpe_device, wakeup->gpe_number);
- adev->wakeup.flags.enabled = 0;
- }
+ adev->wakeup.flags.enabled = 1;
+ } else if (adev->wakeup.flags.enabled) {
+ acpi_disable_gpe(wakeup->gpe_device, wakeup->gpe_number);
acpi_disable_wakeup_device_power(adev);
+ adev->wakeup.flags.enabled = 0;
}
return 0;
}
/**
- * acpi_pm_device_run_wake - Enable/disable remote wakeup for given device.
- * @dev: Device to enable/disable the platform to wake up.
+ * acpi_pm_set_device_wakeup - Enable/disable remote wakeup for given device.
+ * @dev: Device to enable/disable to generate wakeup events.
* @enable: Whether to enable or disable the wakeup functionality.
*/
-int acpi_pm_device_run_wake(struct device *phys_dev, bool enable)
-{
- struct acpi_device *adev;
-
- if (!device_run_wake(phys_dev))
- return -EINVAL;
-
- adev = ACPI_COMPANION(phys_dev);
- if (!adev) {
- dev_dbg(phys_dev, "ACPI companion missing in %s!\n", __func__);
- return -ENODEV;
- }
-
- return acpi_device_wakeup(adev, ACPI_STATE_S0, enable);
-}
-EXPORT_SYMBOL(acpi_pm_device_run_wake);
-
-#ifdef CONFIG_PM_SLEEP
-/**
- * acpi_pm_device_sleep_wake - Enable or disable device to wake up the system.
- * @dev: Device to enable/desible to wake up the system from sleep states.
- * @enable: Whether to enable or disable @dev to wake up the system.
- */
-int acpi_pm_device_sleep_wake(struct device *dev, bool enable)
+int acpi_pm_set_device_wakeup(struct device *dev, bool enable)
{
struct acpi_device *adev;
int error;
- if (!device_can_wakeup(dev))
- return -EINVAL;
-
adev = ACPI_COMPANION(dev);
if (!adev) {
dev_dbg(dev, "ACPI companion missing in %s!\n", __func__);
return -ENODEV;
}
+ if (!acpi_device_can_wakeup(adev))
+ return -EINVAL;
+
error = acpi_device_wakeup(adev, acpi_target_system_state(), enable);
if (!error)
- dev_info(dev, "System wakeup %s by ACPI\n",
- enable ? "enabled" : "disabled");
+ dev_dbg(dev, "Wakeup %s by ACPI\n", enable ? "enabled" : "disabled");
return error;
}
-#endif /* CONFIG_PM_SLEEP */
+EXPORT_SYMBOL(acpi_pm_set_device_wakeup);
/**
* acpi_dev_pm_low_power - Put ACPI device into a low-power state.
static int EC_FLAGS_QUERY_HANDSHAKE; /* Needs QR_EC issued when SCI_EVT set */
static int EC_FLAGS_CORRECT_ECDT; /* Needs ECDT port address correction */
+static int EC_FLAGS_IGNORE_DSDT_GPE; /* Needs ECDT GPE as correction setting */
/* --------------------------------------------------------------------------
* Logging/Debugging
ec->timestamp = jiffies;
}
-#ifdef DEBUG
+#if defined(DEBUG) || defined(CONFIG_DYNAMIC_DEBUG)
static const char *acpi_ec_cmd_string(u8 cmd)
{
switch (cmd) {
static void acpi_ec_submit_query(struct acpi_ec *ec)
{
- if (acpi_ec_event_enabled(ec) &&
- !test_and_set_bit(EC_FLAGS_QUERY_PENDING, &ec->flags)) {
+ acpi_ec_set_storm(ec, EC_FLAGS_COMMAND_STORM);
+ if (!acpi_ec_event_enabled(ec))
+ return;
+ if (!test_and_set_bit(EC_FLAGS_QUERY_PENDING, &ec->flags)) {
ec_dbg_evt("Command(%s) submitted/blocked",
acpi_ec_cmd_string(ACPI_EC_COMMAND_QUERY));
ec->nr_pending_queries++;
static void acpi_ec_complete_query(struct acpi_ec *ec)
{
- if (test_bit(EC_FLAGS_QUERY_PENDING, &ec->flags)) {
- clear_bit(EC_FLAGS_QUERY_PENDING, &ec->flags);
+ if (test_and_clear_bit(EC_FLAGS_QUERY_PENDING, &ec->flags))
ec_dbg_evt("Command(%s) unblocked",
acpi_ec_cmd_string(ACPI_EC_COMMAND_QUERY));
- }
+ acpi_ec_clear_storm(ec, EC_FLAGS_COMMAND_STORM);
}
static inline void __acpi_ec_enable_event(struct acpi_ec *ec)
ec_parse_io_ports, ec);
if (ACPI_FAILURE(status))
return status;
+ if (ec->data_addr == 0 || ec->command_addr == 0)
+ return AE_OK;
- /* Get GPE bit assignment (EC events). */
- /* TODO: Add support for _GPE returning a package */
- status = acpi_evaluate_integer(handle, "_GPE", NULL, &tmp);
- if (ACPI_FAILURE(status))
- return status;
- ec->gpe = tmp;
+ if (boot_ec && boot_ec_is_ecdt && EC_FLAGS_IGNORE_DSDT_GPE) {
+ /*
+ * Always inherit the GPE number setting from the ECDT
+ * EC.
+ */
+ ec->gpe = boot_ec->gpe;
+ } else {
+ /* Get GPE bit assignment (EC events). */
+ /* TODO: Add support for _GPE returning a package */
+ status = acpi_evaluate_integer(handle, "_GPE", NULL, &tmp);
+ if (ACPI_FAILURE(status))
+ return status;
+ ec->gpe = tmp;
+ }
/* Use the global lock for all EC transactions? */
tmp = 0;
acpi_evaluate_integer(handle, "_GLK", NULL, &tmp);
{"", 0},
};
+/*
+ * This function is not Windows-compatible as Windows never enumerates the
+ * namespace EC before the main ACPI device enumeration process. It is
+ * retained for historical reason and will be deprecated in the future.
+ */
int __init acpi_ec_dsdt_probe(void)
{
acpi_status status;
struct acpi_ec *ec;
int ret;
+ /*
+ * If a platform has ECDT, there is no need to proceed as the
+ * following probe is not a part of the ACPI device enumeration,
+ * executing _STA is not safe, and thus this probe may risk of
+ * picking up an invalid EC device.
+ */
+ if (boot_ec)
+ return -ENODEV;
+
ec = acpi_ec_alloc();
if (!ec)
return -ENOMEM;
return 0;
}
+/*
+ * Some DSDTs contain wrong GPE setting.
+ * Asus FX502VD/VE, GL702VMK, X550VXK, X580VD
+ * https://bugzilla.kernel.org/show_bug.cgi?id=195651
+ */
+static int ec_honor_ecdt_gpe(const struct dmi_system_id *id)
+{
+ pr_debug("Detected system needing ignore DSDT GPE setting.\n");
+ EC_FLAGS_IGNORE_DSDT_GPE = 1;
+ return 0;
+}
+
static struct dmi_system_id ec_dmi_table[] __initdata = {
{
ec_correct_ecdt, "MSI MS-171F", {
DMI_MATCH(DMI_SYS_VENDOR, "Micro-Star"),
DMI_MATCH(DMI_PRODUCT_NAME, "MS-171F"),}, NULL},
+ {
+ ec_honor_ecdt_gpe, "ASUS FX502VD", {
+ DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."),
+ DMI_MATCH(DMI_PRODUCT_NAME, "FX502VD"),}, NULL},
+ {
+ ec_honor_ecdt_gpe, "ASUS FX502VE", {
+ DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."),
+ DMI_MATCH(DMI_PRODUCT_NAME, "FX502VE"),}, NULL},
+ {
+ ec_honor_ecdt_gpe, "ASUS GL702VMK", {
+ DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."),
+ DMI_MATCH(DMI_PRODUCT_NAME, "GL702VMK"),}, NULL},
+ {
+ ec_honor_ecdt_gpe, "ASUS X550VXK", {
+ DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."),
+ DMI_MATCH(DMI_PRODUCT_NAME, "X550VXK"),}, NULL},
+ {
+ ec_honor_ecdt_gpe, "ASUS X580VD", {
+ DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."),
+ DMI_MATCH(DMI_PRODUCT_NAME, "X580VD"),}, NULL},
{},
};
struct acpi_ec *ec =
acpi_driver_data(to_acpi_device(dev));
- if (ec_freeze_events)
+ if (acpi_sleep_no_ec_events() && ec_freeze_events)
acpi_ec_disable_event(ec);
return 0;
}
Suspend/Resume
-------------------------------------------------------------------------- */
#ifdef CONFIG_ACPI_SYSTEM_POWER_STATES_SUPPORT
+extern bool acpi_s2idle_wakeup(void);
+extern bool acpi_sleep_no_ec_events(void);
extern int acpi_sleep_init(void);
#else
+static inline bool acpi_s2idle_wakeup(void) { return false; }
+static inline bool acpi_sleep_no_ec_events(void) { return true; }
static inline int acpi_sleep_init(void) { return -ENXIO; }
#endif
struct list_head flushes;
};
-static u8 nfit_uuid[NFIT_UUID_MAX][16];
+static guid_t nfit_uuid[NFIT_UUID_MAX];
-const u8 *to_nfit_uuid(enum nfit_uuids id)
+const guid_t *to_nfit_uuid(enum nfit_uuids id)
{
- return nfit_uuid[id];
+ return &nfit_uuid[id];
}
EXPORT_SYMBOL(to_nfit_uuid);
u32 offset, fw_status = 0;
acpi_handle handle;
unsigned int func;
- const u8 *uuid;
+ const guid_t *guid;
int rc, i;
func = cmd;
cmd_mask = nvdimm_cmd_mask(nvdimm);
dsm_mask = nfit_mem->dsm_mask;
desc = nd_cmd_dimm_desc(cmd);
- uuid = to_nfit_uuid(nfit_mem->family);
+ guid = to_nfit_uuid(nfit_mem->family);
handle = adev->handle;
} else {
struct acpi_device *adev = to_acpi_dev(acpi_desc);
cmd_mask = nd_desc->cmd_mask;
dsm_mask = cmd_mask;
desc = nd_cmd_bus_desc(cmd);
- uuid = to_nfit_uuid(NFIT_DEV_BUS);
+ guid = to_nfit_uuid(NFIT_DEV_BUS);
handle = adev->handle;
dimm_name = "bus";
}
in_buf.buffer.pointer,
min_t(u32, 256, in_buf.buffer.length), true);
- out_obj = acpi_evaluate_dsm(handle, uuid, 1, func, &in_obj);
+ out_obj = acpi_evaluate_dsm(handle, guid, 1, func, &in_obj);
if (!out_obj) {
dev_dbg(dev, "%s:%s _DSM failed cmd: %s\n", __func__, dimm_name,
cmd_name);
int i;
for (i = 0; i < NFIT_UUID_MAX; i++)
- if (memcmp(to_nfit_uuid(i), spa->range_guid, 16) == 0)
+ if (guid_equal(to_nfit_uuid(i), (guid_t *)&spa->range_guid))
return i;
return -1;
}
struct acpi_device *adev, *adev_dimm;
struct device *dev = acpi_desc->dev;
unsigned long dsm_mask;
- const u8 *uuid;
+ const guid_t *guid;
int i;
int family = -1;
/*
* Until standardization materializes we need to consider 4
* different command sets. Note, that checking for function0 (bit0)
- * tells us if any commands are reachable through this uuid.
+ * tells us if any commands are reachable through this GUID.
*/
for (i = NVDIMM_FAMILY_INTEL; i <= NVDIMM_FAMILY_MSFT; i++)
if (acpi_check_dsm(adev_dimm->handle, to_nfit_uuid(i), 1, 1))
return 0;
}
- uuid = to_nfit_uuid(nfit_mem->family);
+ guid = to_nfit_uuid(nfit_mem->family);
for_each_set_bit(i, &dsm_mask, BITS_PER_LONG)
- if (acpi_check_dsm(adev_dimm->handle, uuid, 1, 1ULL << i))
+ if (acpi_check_dsm(adev_dimm->handle, guid, 1, 1ULL << i))
set_bit(i, &nfit_mem->dsm_mask);
return 0;
static void acpi_nfit_init_dsms(struct acpi_nfit_desc *acpi_desc)
{
struct nvdimm_bus_descriptor *nd_desc = &acpi_desc->nd_desc;
- const u8 *uuid = to_nfit_uuid(NFIT_DEV_BUS);
+ const guid_t *guid = to_nfit_uuid(NFIT_DEV_BUS);
struct acpi_device *adev;
int i;
return;
for (i = ND_CMD_ARS_CAP; i <= ND_CMD_CLEAR_ERROR; i++)
- if (acpi_check_dsm(adev->handle, uuid, 1, 1ULL << i))
+ if (acpi_check_dsm(adev->handle, guid, 1, 1ULL << i))
set_bit(i, &nd_desc->cmd_mask);
}
BUILD_BUG_ON(sizeof(struct acpi_nfit_control_region) != 80);
BUILD_BUG_ON(sizeof(struct acpi_nfit_data_region) != 40);
- acpi_str_to_uuid(UUID_VOLATILE_MEMORY, nfit_uuid[NFIT_SPA_VOLATILE]);
- acpi_str_to_uuid(UUID_PERSISTENT_MEMORY, nfit_uuid[NFIT_SPA_PM]);
- acpi_str_to_uuid(UUID_CONTROL_REGION, nfit_uuid[NFIT_SPA_DCR]);
- acpi_str_to_uuid(UUID_DATA_REGION, nfit_uuid[NFIT_SPA_BDW]);
- acpi_str_to_uuid(UUID_VOLATILE_VIRTUAL_DISK, nfit_uuid[NFIT_SPA_VDISK]);
- acpi_str_to_uuid(UUID_VOLATILE_VIRTUAL_CD, nfit_uuid[NFIT_SPA_VCD]);
- acpi_str_to_uuid(UUID_PERSISTENT_VIRTUAL_DISK, nfit_uuid[NFIT_SPA_PDISK]);
- acpi_str_to_uuid(UUID_PERSISTENT_VIRTUAL_CD, nfit_uuid[NFIT_SPA_PCD]);
- acpi_str_to_uuid(UUID_NFIT_BUS, nfit_uuid[NFIT_DEV_BUS]);
- acpi_str_to_uuid(UUID_NFIT_DIMM, nfit_uuid[NFIT_DEV_DIMM]);
- acpi_str_to_uuid(UUID_NFIT_DIMM_N_HPE1, nfit_uuid[NFIT_DEV_DIMM_N_HPE1]);
- acpi_str_to_uuid(UUID_NFIT_DIMM_N_HPE2, nfit_uuid[NFIT_DEV_DIMM_N_HPE2]);
- acpi_str_to_uuid(UUID_NFIT_DIMM_N_MSFT, nfit_uuid[NFIT_DEV_DIMM_N_MSFT]);
+ guid_parse(UUID_VOLATILE_MEMORY, &nfit_uuid[NFIT_SPA_VOLATILE]);
+ guid_parse(UUID_PERSISTENT_MEMORY, &nfit_uuid[NFIT_SPA_PM]);
+ guid_parse(UUID_CONTROL_REGION, &nfit_uuid[NFIT_SPA_DCR]);
+ guid_parse(UUID_DATA_REGION, &nfit_uuid[NFIT_SPA_BDW]);
+ guid_parse(UUID_VOLATILE_VIRTUAL_DISK, &nfit_uuid[NFIT_SPA_VDISK]);
+ guid_parse(UUID_VOLATILE_VIRTUAL_CD, &nfit_uuid[NFIT_SPA_VCD]);
+ guid_parse(UUID_PERSISTENT_VIRTUAL_DISK, &nfit_uuid[NFIT_SPA_PDISK]);
+ guid_parse(UUID_PERSISTENT_VIRTUAL_CD, &nfit_uuid[NFIT_SPA_PCD]);
+ guid_parse(UUID_NFIT_BUS, &nfit_uuid[NFIT_DEV_BUS]);
+ guid_parse(UUID_NFIT_DIMM, &nfit_uuid[NFIT_DEV_DIMM]);
+ guid_parse(UUID_NFIT_DIMM_N_HPE1, &nfit_uuid[NFIT_DEV_DIMM_N_HPE1]);
+ guid_parse(UUID_NFIT_DIMM_N_HPE2, &nfit_uuid[NFIT_DEV_DIMM_N_HPE2]);
+ guid_parse(UUID_NFIT_DIMM_N_MSFT, &nfit_uuid[NFIT_DEV_DIMM_N_MSFT]);
nfit_wq = create_singlethread_workqueue("nfit");
if (!nfit_wq)
#include <linux/libnvdimm.h>
#include <linux/ndctl.h>
#include <linux/types.h>
-#include <linux/uuid.h>
#include <linux/acpi.h>
#include <acpi/acuuid.h>
return container_of(nd_desc, struct acpi_nfit_desc, nd_desc);
}
-const u8 *to_nfit_uuid(enum nfit_uuids id);
+const guid_t *to_nfit_uuid(enum nfit_uuids id);
int acpi_nfit_init(struct acpi_nfit_desc *acpi_desc, void *nfit, acpi_size sz);
void acpi_nfit_shutdown(void *data);
void __acpi_nfit_notify(struct device *dev, acpi_handle handle, u32 event);
__acpi_osi_setup_darwin(enable);
}
-void __init acpi_osi_dmi_linux(bool enable, const struct dmi_system_id *d)
+static void __init acpi_osi_dmi_linux(bool enable,
+ const struct dmi_system_id *d)
{
pr_notice("DMI detected to setup _OSI(\"Linux\"): %s\n", d->ident);
osi_config.linux_dmi = 1;
struct acpi_pci_root *root;
acpi_handle handle = device->handle;
int no_aspm = 0;
- bool hotadd = system_state != SYSTEM_BOOTING;
+ bool hotadd = system_state == SYSTEM_RUNNING;
root = kzalloc(sizeof(struct acpi_pci_root), GFP_KERNEL);
if (!root)
pcie_no_aspm();
pci_acpi_add_bus_pm_notifier(device);
- if (device->wakeup.flags.run_wake)
- device_set_run_wake(root->bus->bridge, true);
+ device_set_wakeup_capable(root->bus->bridge, device->wakeup.flags.valid);
if (hotadd) {
pcibios_resource_survey_bus(root->bus);
pci_stop_root_bus(root->bus);
pci_ioapic_remove(root);
- device_set_run_wake(root->bus->bridge, false);
+ device_set_wakeup_capable(root->bus->bridge, false);
pci_acpi_remove_bus_pm_notifier(device);
pci_remove_root_bus(root->bus);
#include "intel_pmic.h"
#define XPOWER_GPADC_LOW 0x5b
+#define XPOWER_GPI1_CTRL 0x92
+
+#define GPI1_LDO_MASK GENMASK(2, 0)
+#define GPI1_LDO_ON (3 << 0)
+#define GPI1_LDO_OFF (4 << 0)
static struct pmic_table power_table[] = {
{
.reg = 0x10,
.bit = 0x00
}, /* BUC6 */
+ {
+ .address = 0x4c,
+ .reg = 0x92,
+ }, /* GPI1 */
};
/* TMP0 - TMP5 are the same, all from GPADC */
if (regmap_read(regmap, reg, &data))
return -EIO;
- *value = (data & BIT(bit)) ? 1 : 0;
+ /* GPIO1 LDO regulator needs special handling */
+ if (reg == XPOWER_GPI1_CTRL)
+ *value = ((data & GPI1_LDO_MASK) == GPI1_LDO_ON);
+ else
+ *value = (data & BIT(bit)) ? 1 : 0;
+
return 0;
}
{
int data;
+ /* GPIO1 LDO regulator needs special handling */
+ if (reg == XPOWER_GPI1_CTRL)
+ return regmap_update_bits(regmap, reg, GPI1_LDO_MASK,
+ on ? GPI1_LDO_ON : GPI1_LDO_OFF);
+
if (regmap_read(regmap, reg, &data))
return -EIO;
if (!dev->physical_node_count) {
seq_printf(seq, "%c%-8s\n",
- dev->wakeup.flags.run_wake ? '*' : ' ',
+ dev->wakeup.flags.valid ? '*' : ' ',
device_may_wakeup(&dev->dev) ?
"enabled" : "disabled");
} else {
seq_printf(seq, "\t\t");
seq_printf(seq, "%c%-8s %s:%s\n",
- dev->wakeup.flags.run_wake ? '*' : ' ',
+ dev->wakeup.flags.valid ? '*' : ' ',
(device_may_wakeup(&dev->dev) ||
device_may_wakeup(ldev)) ?
"enabled" : "disabled",
return -ENODEV;
/* Protect against concurrent CPU hotplug operations */
- get_online_cpus();
+ cpu_hotplug_disable();
ret = __acpi_processor_start(device);
- put_online_cpus();
+ cpu_hotplug_enable();
return ret;
}
return pr->throttling.acpi_processor_get_throttling(pr);
}
+static int call_on_cpu(int cpu, long (*fn)(void *), void *arg, bool direct)
+{
+ if (direct || (is_percpu_thread() && cpu == smp_processor_id()))
+ return fn(arg);
+ return work_on_cpu(cpu, fn, arg);
+}
+
static int acpi_processor_get_throttling(struct acpi_processor *pr)
{
if (!pr)
if (!cpu_online(pr->id))
return -ENODEV;
- return work_on_cpu(pr->id, __acpi_processor_get_throttling, pr);
+ return call_on_cpu(pr->id, __acpi_processor_get_throttling, pr, false);
}
static int acpi_processor_get_fadt_info(struct acpi_processor *pr)
arg->target_state, arg->force);
}
-static int call_on_cpu(int cpu, long (*fn)(void *), void *arg, bool direct)
-{
- if (direct)
- return fn(arg);
- return work_on_cpu(cpu, fn, arg);
-}
-
static int __acpi_processor_set_throttling(struct acpi_processor *pr,
int state, bool force, bool direct)
{
error = dock_notify(adev, src);
} else if (adev->flags.hotplug_notify) {
error = acpi_generic_hotplug_event(adev, src);
- if (error == -EPERM) {
- ost_code = ACPI_OST_SC_EJECT_NOT_SUPPORTED;
- goto err_out;
- }
} else {
int (*notify)(struct acpi_device *, u32);
else
goto out;
}
- if (!error)
+ switch (error) {
+ case 0:
ost_code = ACPI_OST_SC_SUCCESS;
+ break;
+ case -EPERM:
+ ost_code = ACPI_OST_SC_EJECT_NOT_SUPPORTED;
+ break;
+ case -EBUSY:
+ ost_code = ACPI_OST_SC_DEVICE_BUSY;
+ break;
+ default:
+ ost_code = ACPI_OST_SC_NON_SPECIFIC_FAILURE;
+ break;
+ }
err_out:
acpi_evaluate_ost(adev->handle, src, ost_code, NULL);
return err;
}
-static void acpi_wakeup_gpe_init(struct acpi_device *device)
+static bool acpi_wakeup_gpe_init(struct acpi_device *device)
{
static const struct acpi_device_id button_device_ids[] = {
{"PNP0C0C", 0},
};
struct acpi_device_wakeup *wakeup = &device->wakeup;
acpi_status status;
- acpi_event_status event_status;
wakeup->flags.notifier_present = 0;
/* Power button, Lid switch always enable wakeup */
if (!acpi_match_device_ids(device, button_device_ids)) {
- wakeup->flags.run_wake = 1;
if (!acpi_match_device_ids(device, &button_device_ids[1])) {
/* Do not use Lid/sleep button for S5 wakeup */
if (wakeup->sleep_state == ACPI_STATE_S5)
}
acpi_mark_gpe_for_wake(wakeup->gpe_device, wakeup->gpe_number);
device_set_wakeup_capable(&device->dev, true);
- return;
+ return true;
}
- acpi_setup_gpe_for_wake(device->handle, wakeup->gpe_device,
- wakeup->gpe_number);
- status = acpi_get_gpe_status(wakeup->gpe_device, wakeup->gpe_number,
- &event_status);
- if (ACPI_FAILURE(status))
- return;
-
- wakeup->flags.run_wake = !!(event_status & ACPI_EVENT_FLAG_HAS_HANDLER);
+ status = acpi_setup_gpe_for_wake(device->handle, wakeup->gpe_device,
+ wakeup->gpe_number);
+ return ACPI_SUCCESS(status);
}
static void acpi_bus_get_wakeup_device_flags(struct acpi_device *device)
return;
}
- device->wakeup.flags.valid = 1;
+ device->wakeup.flags.valid = acpi_wakeup_gpe_init(device);
device->wakeup.prepare_count = 0;
- acpi_wakeup_gpe_init(device);
- /* Call _PSW/_DSW object to disable its ability to wake the sleeping
+ /*
+ * Call _PSW/_DSW object to disable its ability to wake the sleeping
* system for the ACPI device with the _PRW object.
* The _PSW object is depreciated in ACPI 3.0 and is replaced by _DSW.
* So it is necessary to call _DSW object first. Only when it is not
adev->flags.coherent_dma = cca;
}
+static int acpi_check_spi_i2c_slave(struct acpi_resource *ares, void *data)
+{
+ bool *is_spi_i2c_slave_p = data;
+
+ if (ares->type != ACPI_RESOURCE_TYPE_SERIAL_BUS)
+ return 1;
+
+ /*
+ * devices that are connected to UART still need to be enumerated to
+ * platform bus
+ */
+ if (ares->data.common_serial_bus.type != ACPI_RESOURCE_SERIAL_TYPE_UART)
+ *is_spi_i2c_slave_p = true;
+
+ /* no need to do more checking */
+ return -1;
+}
+
+static bool acpi_is_spi_i2c_slave(struct acpi_device *device)
+{
+ struct list_head resource_list;
+ bool is_spi_i2c_slave = false;
+
+ INIT_LIST_HEAD(&resource_list);
+ acpi_dev_get_resources(device, &resource_list, acpi_check_spi_i2c_slave,
+ &is_spi_i2c_slave);
+ acpi_dev_free_resource_list(&resource_list);
+
+ return is_spi_i2c_slave;
+}
+
void acpi_init_device_object(struct acpi_device *device, acpi_handle handle,
int type, unsigned long long sta)
{
acpi_bus_get_flags(device);
device->flags.match_driver = false;
device->flags.initialized = true;
+ device->flags.spi_i2c_slave = acpi_is_spi_i2c_slave(device);
acpi_device_clear_enumerated(device);
device_initialize(&device->dev);
dev_set_uevent_suppress(&device->dev, true);
return AE_OK;
}
-static int acpi_check_spi_i2c_slave(struct acpi_resource *ares, void *data)
-{
- bool *is_spi_i2c_slave_p = data;
-
- if (ares->type != ACPI_RESOURCE_TYPE_SERIAL_BUS)
- return 1;
-
- /*
- * devices that are connected to UART still need to be enumerated to
- * platform bus
- */
- if (ares->data.common_serial_bus.type != ACPI_RESOURCE_SERIAL_TYPE_UART)
- *is_spi_i2c_slave_p = true;
-
- /* no need to do more checking */
- return -1;
-}
-
static void acpi_default_enumeration(struct acpi_device *device)
{
- struct list_head resource_list;
- bool is_spi_i2c_slave = false;
-
/*
* Do not enumerate SPI/I2C slaves as they will be enumerated by their
* respective parents.
*/
- INIT_LIST_HEAD(&resource_list);
- acpi_dev_get_resources(device, &resource_list, acpi_check_spi_i2c_slave,
- &is_spi_i2c_slave);
- acpi_dev_free_resource_list(&resource_list);
- if (!is_spi_i2c_slave) {
+ if (!device->flags.spi_i2c_slave) {
acpi_create_platform_device(device, NULL);
acpi_device_set_enumerated(device);
} else {
return;
device->flags.match_driver = true;
- if (ret > 0) {
+ if (ret > 0 && !device->flags.spi_i2c_slave) {
acpi_device_set_enumerated(device);
goto ok;
}
if (ret < 0)
return;
- if (device->pnp.type.platform_id)
- acpi_default_enumeration(device);
- else
+ if (!device->pnp.type.platform_id && !device->flags.spi_i2c_slave)
acpi_device_set_enumerated(device);
+ else
+ acpi_default_enumeration(device);
ok:
list_for_each_entry(child, &device->children, node)
.recover = acpi_pm_finish,
};
+static bool s2idle_in_progress;
+static bool s2idle_wakeup;
+
+/*
+ * On platforms supporting the Low Power S0 Idle interface there is an ACPI
+ * device object with the PNP0D80 compatible device ID (System Power Management
+ * Controller) and a specific _DSM method under it. That method, if present,
+ * can be used to indicate to the platform that the OS is transitioning into a
+ * low-power state in which certain types of activity are not desirable or that
+ * it is leaving such a state, which allows the platform to adjust its operation
+ * mode accordingly.
+ */
+static const struct acpi_device_id lps0_device_ids[] = {
+ {"PNP0D80", },
+ {"", },
+};
+
+#define ACPI_LPS0_DSM_UUID "c4eb40a0-6cd2-11e2-bcfd-0800200c9a66"
+
+#define ACPI_LPS0_SCREEN_OFF 3
+#define ACPI_LPS0_SCREEN_ON 4
+#define ACPI_LPS0_ENTRY 5
+#define ACPI_LPS0_EXIT 6
+
+#define ACPI_S2IDLE_FUNC_MASK ((1 << ACPI_LPS0_ENTRY) | (1 << ACPI_LPS0_EXIT))
+
+static acpi_handle lps0_device_handle;
+static guid_t lps0_dsm_guid;
+static char lps0_dsm_func_mask;
+
+static void acpi_sleep_run_lps0_dsm(unsigned int func)
+{
+ union acpi_object *out_obj;
+
+ if (!(lps0_dsm_func_mask & (1 << func)))
+ return;
+
+ out_obj = acpi_evaluate_dsm(lps0_device_handle, &lps0_dsm_guid, 1, func, NULL);
+ ACPI_FREE(out_obj);
+
+ acpi_handle_debug(lps0_device_handle, "_DSM function %u evaluation %s\n",
+ func, out_obj ? "successful" : "failed");
+}
+
+static int lps0_device_attach(struct acpi_device *adev,
+ const struct acpi_device_id *not_used)
+{
+ union acpi_object *out_obj;
+
+ if (lps0_device_handle)
+ return 0;
+
+ if (!(acpi_gbl_FADT.flags & ACPI_FADT_LOW_POWER_S0))
+ return 0;
+
+ guid_parse(ACPI_LPS0_DSM_UUID, &lps0_dsm_guid);
+ /* Check if the _DSM is present and as expected. */
+ out_obj = acpi_evaluate_dsm(adev->handle, &lps0_dsm_guid, 1, 0, NULL);
+ if (out_obj && out_obj->type == ACPI_TYPE_BUFFER) {
+ char bitmask = *(char *)out_obj->buffer.pointer;
+
+ if ((bitmask & ACPI_S2IDLE_FUNC_MASK) == ACPI_S2IDLE_FUNC_MASK) {
+ lps0_dsm_func_mask = bitmask;
+ lps0_device_handle = adev->handle;
+ }
+
+ acpi_handle_debug(adev->handle, "_DSM function mask: 0x%x\n",
+ bitmask);
+ } else {
+ acpi_handle_debug(adev->handle,
+ "_DSM function 0 evaluation failed\n");
+ }
+ ACPI_FREE(out_obj);
+ return 0;
+}
+
+static struct acpi_scan_handler lps0_handler = {
+ .ids = lps0_device_ids,
+ .attach = lps0_device_attach,
+};
+
static int acpi_freeze_begin(void)
{
acpi_scan_lock_acquire();
+ s2idle_in_progress = true;
return 0;
}
static int acpi_freeze_prepare(void)
{
- acpi_enable_wakeup_devices(ACPI_STATE_S0);
- acpi_enable_all_wakeup_gpes();
- acpi_os_wait_events_complete();
+ if (lps0_device_handle) {
+ acpi_sleep_run_lps0_dsm(ACPI_LPS0_SCREEN_OFF);
+ acpi_sleep_run_lps0_dsm(ACPI_LPS0_ENTRY);
+ } else {
+ /*
+ * The configuration of GPEs is changed here to avoid spurious
+ * wakeups, but that should not be necessary if this is a
+ * "low-power S0" platform and the low-power S0 _DSM is present.
+ */
+ acpi_enable_all_wakeup_gpes();
+ acpi_os_wait_events_complete();
+ }
if (acpi_sci_irq_valid())
enable_irq_wake(acpi_sci_irq);
+
return 0;
}
+static void acpi_freeze_wake(void)
+{
+ /*
+ * If IRQD_WAKEUP_ARMED is not set for the SCI at this point, it means
+ * that the SCI has triggered while suspended, so cancel the wakeup in
+ * case it has not been a wakeup event (the GPEs will be checked later).
+ */
+ if (acpi_sci_irq_valid() &&
+ !irqd_is_wakeup_armed(irq_get_irq_data(acpi_sci_irq))) {
+ pm_system_cancel_wakeup();
+ s2idle_wakeup = true;
+ }
+}
+
+static void acpi_freeze_sync(void)
+{
+ /*
+ * Process all pending events in case there are any wakeup ones.
+ *
+ * The EC driver uses the system workqueue, so that one needs to be
+ * flushed too.
+ */
+ acpi_os_wait_events_complete();
+ flush_scheduled_work();
+ s2idle_wakeup = false;
+}
+
static void acpi_freeze_restore(void)
{
- acpi_disable_wakeup_devices(ACPI_STATE_S0);
if (acpi_sci_irq_valid())
disable_irq_wake(acpi_sci_irq);
- acpi_enable_all_runtime_gpes();
+
+ if (lps0_device_handle) {
+ acpi_sleep_run_lps0_dsm(ACPI_LPS0_EXIT);
+ acpi_sleep_run_lps0_dsm(ACPI_LPS0_SCREEN_ON);
+ } else {
+ acpi_enable_all_runtime_gpes();
+ }
}
static void acpi_freeze_end(void)
{
+ s2idle_in_progress = false;
acpi_scan_lock_release();
}
static const struct platform_freeze_ops acpi_freeze_ops = {
.begin = acpi_freeze_begin,
.prepare = acpi_freeze_prepare,
+ .wake = acpi_freeze_wake,
+ .sync = acpi_freeze_sync,
.restore = acpi_freeze_restore,
.end = acpi_freeze_end,
};
suspend_set_ops(old_suspend_ordering ?
&acpi_suspend_ops_old : &acpi_suspend_ops);
+
+ acpi_scan_add_handler(&lps0_handler);
freeze_set_ops(&acpi_freeze_ops);
}
#else /* !CONFIG_SUSPEND */
+#define s2idle_in_progress (false)
+#define s2idle_wakeup (false)
+#define lps0_device_handle (NULL)
static inline void acpi_sleep_suspend_setup(void) {}
#endif /* !CONFIG_SUSPEND */
+bool acpi_s2idle_wakeup(void)
+{
+ return s2idle_wakeup;
+}
+
+bool acpi_sleep_no_ec_events(void)
+{
+ return !s2idle_in_progress || !lps0_device_handle;
+}
+
#ifdef CONFIG_PM_SLEEP
static u32 saved_bm_rld;
return false;
}
+/*
+ * APM X-Gene v1 and v2 UART hardware is an 16550 like device but has its
+ * register aligned to 32-bit. In addition, the BIOS also encoded the
+ * access width to be 8 bits. This function detects this errata condition.
+ */
+static bool xgene_8250_erratum_present(struct acpi_table_spcr *tb)
+{
+ if (tb->interface_type != ACPI_DBG2_16550_COMPATIBLE)
+ return false;
+
+ if (memcmp(tb->header.oem_id, "APMC0D", ACPI_OEM_ID_SIZE))
+ return false;
+
+ if (!memcmp(tb->header.oem_table_id, "XGENESPC",
+ ACPI_OEM_TABLE_ID_SIZE) && tb->header.oem_revision == 0)
+ return true;
+
+ return false;
+}
+
/**
* parse_spcr() - parse ACPI SPCR table and add preferred console
*
goto done;
}
- iotype = table->serial_port.space_id == ACPI_ADR_SPACE_SYSTEM_MEMORY ?
- "mmio" : "io";
+ if (table->serial_port.space_id == ACPI_ADR_SPACE_SYSTEM_MEMORY) {
+ switch (table->serial_port.access_width) {
+ default:
+ pr_err("Unexpected SPCR Access Width. Defaulting to byte size\n");
+ case ACPI_ACCESS_SIZE_BYTE:
+ iotype = "mmio";
+ break;
+ case ACPI_ACCESS_SIZE_WORD:
+ iotype = "mmio16";
+ break;
+ case ACPI_ACCESS_SIZE_DWORD:
+ iotype = "mmio32";
+ break;
+ }
+ } else
+ iotype = "io";
switch (table->interface_type) {
case ACPI_DBG2_ARM_SBSA_32BIT:
if (qdf2400_erratum_44_present(&table->header))
uart = "qdf2400_e44";
+ if (xgene_8250_erratum_present(table))
+ iotype = "mmio32";
snprintf(opts, sizeof(opts), "%s,%s,0x%llx,%d", uart, iotype,
table->serial_port.address, baud_rate);
/**
* acpi_evaluate_dsm - evaluate device's _DSM method
* @handle: ACPI device handle
- * @uuid: UUID of requested functions, should be 16 bytes
+ * @guid: GUID of requested functions, should be 16 bytes
* @rev: revision number of requested function
* @func: requested function number
* @argv4: the function specific parameter
*
- * Evaluate device's _DSM method with specified UUID, revision id and
+ * Evaluate device's _DSM method with specified GUID, revision id and
* function number. Caller needs to free the returned object.
*
* Though ACPI defines the fourth parameter for _DSM should be a package,
* some old BIOSes do expect a buffer or an integer etc.
*/
union acpi_object *
-acpi_evaluate_dsm(acpi_handle handle, const u8 *uuid, u64 rev, u64 func,
+acpi_evaluate_dsm(acpi_handle handle, const guid_t *guid, u64 rev, u64 func,
union acpi_object *argv4)
{
acpi_status ret;
params[0].type = ACPI_TYPE_BUFFER;
params[0].buffer.length = 16;
- params[0].buffer.pointer = (char *)uuid;
+ params[0].buffer.pointer = (u8 *)guid;
params[1].type = ACPI_TYPE_INTEGER;
params[1].integer.value = rev;
params[2].type = ACPI_TYPE_INTEGER;
/**
* acpi_check_dsm - check if _DSM method supports requested functions.
* @handle: ACPI device handle
- * @uuid: UUID of requested functions, should be 16 bytes at least
+ * @guid: GUID of requested functions, should be 16 bytes at least
* @rev: revision number of requested functions
* @funcs: bitmap of requested functions
*
* functions. Currently only support 64 functions at maximum, should be
* enough for now.
*/
-bool acpi_check_dsm(acpi_handle handle, const u8 *uuid, u64 rev, u64 funcs)
+bool acpi_check_dsm(acpi_handle handle, const guid_t *guid, u64 rev, u64 funcs)
{
int i;
u64 mask = 0;
if (funcs == 0)
return false;
- obj = acpi_evaluate_dsm(handle, uuid, rev, 0, NULL);
+ obj = acpi_evaluate_dsm(handle, guid, rev, 0, NULL);
if (!obj)
return false;
/*
* Bit 0 indicates whether there's support for any functions other than
- * function 0 for the specified UUID and revision.
+ * function 0 for the specified GUID and revision.
*/
if ((mask & 0x1) && (mask & funcs) == funcs)
return true;
DMI_MATCH(DMI_PRODUCT_NAME, "Dell System XPS L702X"),
},
},
+ {
+ .callback = video_detect_force_native,
+ .ident = "Dell Precision 7510",
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc."),
+ DMI_MATCH(DMI_PRODUCT_NAME, "Precision 7510"),
+ },
+ },
{ },
};
return count;
}
+static DEVICE_ATTR_RW(driver_override);
#define amba_attr_func(name,fmt,arg...) \
static ssize_t name##_show(struct device *_dev, \
{ \
struct amba_device *dev = to_amba_device(_dev); \
return sprintf(buf, fmt, arg); \
-}
-
-#define amba_attr(name,fmt,arg...) \
-amba_attr_func(name,fmt,arg) \
-static DEVICE_ATTR(name, S_IRUGO, name##_show, NULL)
+} \
+static DEVICE_ATTR_RO(name)
amba_attr_func(id, "%08x\n", dev->periphid);
-amba_attr(irq0, "%u\n", dev->irq[0]);
-amba_attr(irq1, "%u\n", dev->irq[1]);
+amba_attr_func(irq0, "%u\n", dev->irq[0]);
+amba_attr_func(irq1, "%u\n", dev->irq[1]);
amba_attr_func(resource, "\t%016llx\t%016llx\t%016lx\n",
(unsigned long long)dev->res.start, (unsigned long long)dev->res.end,
dev->res.flags);
-static struct device_attribute amba_dev_attrs[] = {
- __ATTR_RO(id),
- __ATTR_RO(resource),
- __ATTR_RW(driver_override),
- __ATTR_NULL,
+static struct attribute *amba_dev_attrs[] = {
+ &dev_attr_id.attr,
+ &dev_attr_resource.attr,
+ &dev_attr_driver_override.attr,
+ NULL,
};
+ATTRIBUTE_GROUPS(amba_dev);
#ifdef CONFIG_PM
/*
*/
struct bus_type amba_bustype = {
.name = "amba",
- .dev_attrs = amba_dev_attrs,
+ .dev_groups = amba_dev_groups,
.match = amba_match,
.uevent = amba_uevent,
.pm = &amba_pm,
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* AHCI hardware documentation:
* http://www.intel.com/technology/serialata/pdf/rev1_0.pdf
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* AHCI hardware documentation:
* http://www.intel.com/technology/serialata/pdf/rev1_0.pdf
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* AHCI hardware documentation:
* http://www.intel.com/technology/serialata/pdf/rev1_0.pdf
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* Hardware documentation available at http://developer.intel.com/
*
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* AHCI hardware documentation:
* http://www.intel.com/technology/serialata/pdf/rev1_0.pdf
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* Hardware documentation available from http://www.t13.org/ and
* http://www.sata-io.org/
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* Hardware documentation available from http://www.t13.org/ and
* http://www.sata-io.org/
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* Hardware documentation available from
* - http://www.t10.org/
*
* Translate a SCSI WRITE SAME command to be either a DSM TRIM command or
* an SCT Write Same command.
- * Based on WRITE SAME has the UNMAP flag
- * When set translate to DSM TRIM
- * When clear translate to SCT Write Same
+ * Based on WRITE SAME has the UNMAP flag:
+ *
+ * - When set translate to DSM TRIM
+ * - When clear translate to SCT Write Same
*/
static unsigned int ata_scsi_write_same_xlat(struct ata_queued_cmd *qc)
{
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* Hardware documentation available from http://www.t13.org/ and
* http://www.sata-io.org/
sdev_disable_disk_events(dev->sdev);
zpodd->powered_off = true;
- device_set_run_wake(&dev->tdev, true);
- acpi_pm_device_run_wake(&dev->tdev, true);
+ acpi_pm_set_device_wakeup(&dev->tdev, true);
}
/* Disable runtime wake capability if it is enabled */
{
struct zpodd *zpodd = dev->zpodd;
- if (zpodd->powered_off) {
- acpi_pm_device_run_wake(&dev->tdev, false);
- device_set_run_wake(&dev->tdev, false);
- }
+ if (zpodd->powered_off)
+ acpi_pm_set_device_wakeup(&dev->tdev, false);
}
/*
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
*/
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* Hardware information only available under NDA.
*
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
*
* Supports ATA disks in single-packet ADMA mode.
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* No hardware documentation available outside of NVIDIA.
* This driver programs the NVIDIA SATA controller in a similar
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* Hardware information only available under NDA.
*
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
*/
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
*/
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* Documentation for SiI 3112:
* http://gkernel.sourceforge.net/specs/sii/3112A_SiI-DS-0095-B2.pdf.bz2
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* Hardware documentation available under NDA.
*
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* Hardware documentation available under NDA.
*
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* Hardware documentation available under NDA.
*
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* Hardware documentation available under NDA.
*
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* Hardware documentation available under NDA.
*
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* Vitesse hardware documentation presumably available under NDA.
* Intel 31244 (same hardware interface) documentation presumably
static void panel_process_inputs(void)
{
- struct list_head *item;
struct logical_input *input;
keypressed = 0;
inputs_stable = 1;
- list_for_each(item, &logical_inputs) {
- input = list_entry(item, struct logical_input, list);
-
+ list_for_each_entry(input, &logical_inputs, list) {
switch (input->state) {
case INPUT_ST_LOW:
if ((phys_curr & input->mask) != input->value)
endif
+config GENERIC_ARCH_TOPOLOGY
+ bool
+ help
+ Enable support for architectures common topology code: e.g., parsing
+ CPU capacity information from DT, usage of such information for
+ appropriate scaling, sysfs interface for changing capacity values at
+ runtime.
+
endmenu
obj-$(CONFIG_PINCTRL) += pinctrl.o
obj-$(CONFIG_DEV_COREDUMP) += devcoredump.o
obj-$(CONFIG_GENERIC_MSI_IRQ_DOMAIN) += platform-msi.o
+obj-$(CONFIG_GENERIC_ARCH_TOPOLOGY) += arch_topology.o
obj-y += test/
--- /dev/null
+/*
+ * Arch specific cpu topology information
+ *
+ * Copyright (C) 2016, ARM Ltd.
+ * Written by: Juri Lelli, ARM Ltd.
+ *
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License. See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Released under the GPLv2 only.
+ * SPDX-License-Identifier: GPL-2.0
+ */
+
+#include <linux/acpi.h>
+#include <linux/arch_topology.h>
+#include <linux/cpu.h>
+#include <linux/cpufreq.h>
+#include <linux/device.h>
+#include <linux/of.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <linux/sched/topology.h>
+
+static DEFINE_MUTEX(cpu_scale_mutex);
+static DEFINE_PER_CPU(unsigned long, cpu_scale) = SCHED_CAPACITY_SCALE;
+
+unsigned long topology_get_cpu_scale(struct sched_domain *sd, int cpu)
+{
+ return per_cpu(cpu_scale, cpu);
+}
+
+void topology_set_cpu_scale(unsigned int cpu, unsigned long capacity)
+{
+ per_cpu(cpu_scale, cpu) = capacity;
+}
+
+static ssize_t cpu_capacity_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct cpu *cpu = container_of(dev, struct cpu, dev);
+
+ return sprintf(buf, "%lu\n",
+ topology_get_cpu_scale(NULL, cpu->dev.id));
+}
+
+static ssize_t cpu_capacity_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ struct cpu *cpu = container_of(dev, struct cpu, dev);
+ int this_cpu = cpu->dev.id;
+ int i;
+ unsigned long new_capacity;
+ ssize_t ret;
+
+ if (!count)
+ return 0;
+
+ ret = kstrtoul(buf, 0, &new_capacity);
+ if (ret)
+ return ret;
+ if (new_capacity > SCHED_CAPACITY_SCALE)
+ return -EINVAL;
+
+ mutex_lock(&cpu_scale_mutex);
+ for_each_cpu(i, &cpu_topology[this_cpu].core_sibling)
+ topology_set_cpu_scale(i, new_capacity);
+ mutex_unlock(&cpu_scale_mutex);
+
+ return count;
+}
+
+static DEVICE_ATTR_RW(cpu_capacity);
+
+static int register_cpu_capacity_sysctl(void)
+{
+ int i;
+ struct device *cpu;
+
+ for_each_possible_cpu(i) {
+ cpu = get_cpu_device(i);
+ if (!cpu) {
+ pr_err("%s: too early to get CPU%d device!\n",
+ __func__, i);
+ continue;
+ }
+ device_create_file(cpu, &dev_attr_cpu_capacity);
+ }
+
+ return 0;
+}
+subsys_initcall(register_cpu_capacity_sysctl);
+
+static u32 capacity_scale;
+static u32 *raw_capacity;
+static bool cap_parsing_failed;
+
+void topology_normalize_cpu_scale(void)
+{
+ u64 capacity;
+ int cpu;
+
+ if (!raw_capacity || cap_parsing_failed)
+ return;
+
+ pr_debug("cpu_capacity: capacity_scale=%u\n", capacity_scale);
+ mutex_lock(&cpu_scale_mutex);
+ for_each_possible_cpu(cpu) {
+ pr_debug("cpu_capacity: cpu=%d raw_capacity=%u\n",
+ cpu, raw_capacity[cpu]);
+ capacity = (raw_capacity[cpu] << SCHED_CAPACITY_SHIFT)
+ / capacity_scale;
+ topology_set_cpu_scale(cpu, capacity);
+ pr_debug("cpu_capacity: CPU%d cpu_capacity=%lu\n",
+ cpu, topology_get_cpu_scale(NULL, cpu));
+ }
+ mutex_unlock(&cpu_scale_mutex);
+}
+
+int __init topology_parse_cpu_capacity(struct device_node *cpu_node, int cpu)
+{
+ int ret = 1;
+ u32 cpu_capacity;
+
+ if (cap_parsing_failed)
+ return !ret;
+
+ ret = of_property_read_u32(cpu_node,
+ "capacity-dmips-mhz",
+ &cpu_capacity);
+ if (!ret) {
+ if (!raw_capacity) {
+ raw_capacity = kcalloc(num_possible_cpus(),
+ sizeof(*raw_capacity),
+ GFP_KERNEL);
+ if (!raw_capacity) {
+ pr_err("cpu_capacity: failed to allocate memory for raw capacities\n");
+ cap_parsing_failed = true;
+ return 0;
+ }
+ }
+ capacity_scale = max(cpu_capacity, capacity_scale);
+ raw_capacity[cpu] = cpu_capacity;
+ pr_debug("cpu_capacity: %s cpu_capacity=%u (raw)\n",
+ cpu_node->full_name, raw_capacity[cpu]);
+ } else {
+ if (raw_capacity) {
+ pr_err("cpu_capacity: missing %s raw capacity\n",
+ cpu_node->full_name);
+ pr_err("cpu_capacity: partial information: fallback to 1024 for all CPUs\n");
+ }
+ cap_parsing_failed = true;
+ kfree(raw_capacity);
+ }
+
+ return !ret;
+}
+
+#ifdef CONFIG_CPU_FREQ
+static cpumask_var_t cpus_to_visit;
+static bool cap_parsing_done;
+static void parsing_done_workfn(struct work_struct *work);
+static DECLARE_WORK(parsing_done_work, parsing_done_workfn);
+
+static int
+init_cpu_capacity_callback(struct notifier_block *nb,
+ unsigned long val,
+ void *data)
+{
+ struct cpufreq_policy *policy = data;
+ int cpu;
+
+ if (cap_parsing_failed || cap_parsing_done)
+ return 0;
+
+ switch (val) {
+ case CPUFREQ_NOTIFY:
+ pr_debug("cpu_capacity: init cpu capacity for CPUs [%*pbl] (to_visit=%*pbl)\n",
+ cpumask_pr_args(policy->related_cpus),
+ cpumask_pr_args(cpus_to_visit));
+ cpumask_andnot(cpus_to_visit,
+ cpus_to_visit,
+ policy->related_cpus);
+ for_each_cpu(cpu, policy->related_cpus) {
+ raw_capacity[cpu] = topology_get_cpu_scale(NULL, cpu) *
+ policy->cpuinfo.max_freq / 1000UL;
+ capacity_scale = max(raw_capacity[cpu], capacity_scale);
+ }
+ if (cpumask_empty(cpus_to_visit)) {
+ topology_normalize_cpu_scale();
+ kfree(raw_capacity);
+ pr_debug("cpu_capacity: parsing done\n");
+ cap_parsing_done = true;
+ schedule_work(&parsing_done_work);
+ }
+ }
+ return 0;
+}
+
+static struct notifier_block init_cpu_capacity_notifier = {
+ .notifier_call = init_cpu_capacity_callback,
+};
+
+static int __init register_cpufreq_notifier(void)
+{
+ /*
+ * on ACPI-based systems we need to use the default cpu capacity
+ * until we have the necessary code to parse the cpu capacity, so
+ * skip registering cpufreq notifier.
+ */
+ if (!acpi_disabled || !raw_capacity)
+ return -EINVAL;
+
+ if (!alloc_cpumask_var(&cpus_to_visit, GFP_KERNEL)) {
+ pr_err("cpu_capacity: failed to allocate memory for cpus_to_visit\n");
+ return -ENOMEM;
+ }
+
+ cpumask_copy(cpus_to_visit, cpu_possible_mask);
+
+ return cpufreq_register_notifier(&init_cpu_capacity_notifier,
+ CPUFREQ_POLICY_NOTIFIER);
+}
+core_initcall(register_cpufreq_notifier);
+
+static void parsing_done_workfn(struct work_struct *work)
+{
+ cpufreq_unregister_notifier(&init_cpu_capacity_notifier,
+ CPUFREQ_POLICY_NOTIFIER);
+}
+
+#else
+static int __init free_raw_capacity(void)
+{
+ kfree(raw_capacity);
+
+ return 0;
+}
+core_initcall(free_raw_capacity);
+#endif
}
EXPORT_SYMBOL_GPL(bus_for_each_drv);
-static int device_add_attrs(struct bus_type *bus, struct device *dev)
-{
- int error = 0;
- int i;
-
- if (!bus->dev_attrs)
- return 0;
-
- for (i = 0; bus->dev_attrs[i].attr.name; i++) {
- error = device_create_file(dev, &bus->dev_attrs[i]);
- if (error) {
- while (--i >= 0)
- device_remove_file(dev, &bus->dev_attrs[i]);
- break;
- }
- }
- return error;
-}
-
-static void device_remove_attrs(struct bus_type *bus, struct device *dev)
-{
- int i;
-
- if (bus->dev_attrs) {
- for (i = 0; bus->dev_attrs[i].attr.name; i++)
- device_remove_file(dev, &bus->dev_attrs[i]);
- }
-}
-
/**
* bus_add_device - add device to bus
* @dev: device being added
if (bus) {
pr_debug("bus: '%s': add device %s\n", bus->name, dev_name(dev));
- error = device_add_attrs(bus, dev);
- if (error)
- goto out_put;
error = device_add_groups(dev, bus->dev_groups);
if (error)
- goto out_id;
+ goto out_put;
error = sysfs_create_link(&bus->p->devices_kset->kobj,
&dev->kobj, dev_name(dev));
if (error)
sysfs_remove_link(&bus->p->devices_kset->kobj, dev_name(dev));
out_groups:
device_remove_groups(dev, bus->dev_groups);
-out_id:
- device_remove_attrs(bus, dev);
out_put:
bus_put(dev->bus);
return error;
sysfs_remove_link(&dev->kobj, "subsystem");
sysfs_remove_link(&dev->bus->p->devices_kset->kobj,
dev_name(dev));
- device_remove_attrs(dev->bus, dev);
device_remove_groups(dev, dev->bus->dev_groups);
if (klist_node_attached(&dev->p->knode_bus))
klist_del(&dev->p->knode_bus);
static ssize_t uevent_store(struct device_driver *drv, const char *buf,
size_t count)
{
- enum kobject_action action;
-
- if (kobject_action_type(buf, count, &action) == 0)
- kobject_uevent(&drv->p->kobj, action);
+ kobject_synth_uevent(&drv->p->kobj, buf, count);
return count;
}
static DRIVER_ATTR_WO(uevent);
static ssize_t bus_uevent_store(struct bus_type *bus,
const char *buf, size_t count)
{
- enum kobject_action action;
-
- if (kobject_action_type(buf, count, &action) == 0)
- kobject_uevent(&bus->p->subsys.kobj, action);
+ kobject_synth_uevent(&bus->p->subsys.kobj, buf, count);
return count;
}
static BUS_ATTR(uevent, S_IWUSR, NULL, bus_uevent_store);
kset_put(&cls->p->subsys);
}
-static int add_class_attrs(struct class *cls)
-{
- int i;
- int error = 0;
-
- if (cls->class_attrs) {
- for (i = 0; cls->class_attrs[i].attr.name; i++) {
- error = class_create_file(cls, &cls->class_attrs[i]);
- if (error)
- goto error;
- }
- }
-done:
- return error;
-error:
- while (--i >= 0)
- class_remove_file(cls, &cls->class_attrs[i]);
- goto done;
-}
-
-static void remove_class_attrs(struct class *cls)
-{
- int i;
-
- if (cls->class_attrs) {
- for (i = 0; cls->class_attrs[i].attr.name; i++)
- class_remove_file(cls, &cls->class_attrs[i]);
- }
-}
-
static void klist_class_dev_get(struct klist_node *n)
{
struct device *dev = container_of(n, struct device, knode_class);
}
error = class_add_groups(class_get(cls), cls->class_groups);
class_put(cls);
- error = add_class_attrs(class_get(cls));
- class_put(cls);
return error;
}
EXPORT_SYMBOL_GPL(__class_register);
void class_unregister(struct class *cls)
{
pr_debug("device class '%s': unregistering\n", cls->name);
- remove_class_attrs(cls);
class_remove_groups(cls, cls->class_groups);
kset_unregister(&cls->p->subsys);
}
static ssize_t uevent_store(struct device *dev, struct device_attribute *attr,
const char *buf, size_t count)
{
- enum kobject_action action;
+ if (kobject_synth_uevent(&dev->kobj, buf, count))
+ dev_err(dev, "uevent: failed to send synthetic uevent\n");
- if (kobject_action_type(buf, count, &action) == 0)
- kobject_uevent(&dev->kobj, action);
- else
- dev_err(dev, "uevent: unknown action-string\n");
return count;
}
static DEVICE_ATTR_RW(uevent);
else
dev->fwnode = fwnode;
}
+
+/**
+ * device_set_of_node_from_dev - reuse device-tree node of another device
+ * @dev: device whose device-tree node is being set
+ * @dev2: device whose device-tree node is being reused
+ *
+ * Takes another reference to the new device-tree node after first dropping
+ * any reference held to the old node.
+ */
+void device_set_of_node_from_dev(struct device *dev, const struct device *dev2)
+{
+ of_node_put(dev->of_node);
+ dev->of_node = of_node_get(dev2->of_node);
+ dev->of_node_reused = true;
+}
+EXPORT_SYMBOL_GPL(device_set_of_node_from_dev);
EXPORT_SYMBOL(dma_common_mmap);
#ifdef CONFIG_MMU
+static struct vm_struct *__dma_common_pages_remap(struct page **pages,
+ size_t size, unsigned long vm_flags, pgprot_t prot,
+ const void *caller)
+{
+ struct vm_struct *area;
+
+ area = get_vm_area_caller(size, vm_flags, caller);
+ if (!area)
+ return NULL;
+
+ if (map_vm_area(area, prot, pages)) {
+ vunmap(area->addr);
+ return NULL;
+ }
+
+ return area;
+}
+
/*
* remaps an array of PAGE_SIZE pages into another vm_area
* Cannot be used in non-sleeping contexts
{
struct vm_struct *area;
- area = get_vm_area_caller(size, vm_flags, caller);
+ area = __dma_common_pages_remap(pages, size, vm_flags, prot, caller);
if (!area)
return NULL;
area->pages = pages;
- if (map_vm_area(area, prot, pages)) {
- vunmap(area->addr);
- return NULL;
- }
-
return area->addr;
}
{
int i;
struct page **pages;
- void *ptr;
+ struct vm_struct *area;
pages = kmalloc(sizeof(struct page *) << get_order(size), GFP_KERNEL);
if (!pages)
for (i = 0; i < (size >> PAGE_SHIFT); i++)
pages[i] = nth_page(page, i);
- ptr = dma_common_pages_remap(pages, size, vm_flags, prot, caller);
+ area = __dma_common_pages_remap(pages, size, vm_flags, prot, caller);
kfree(pages);
- return ptr;
+ if (!area)
+ return NULL;
+ return area->addr;
}
/*
* guarding for corner cases a global lock should be OK */
static DEFINE_MUTEX(fw_lock);
+static bool __enable_firmware = false;
+
+static void enable_firmware(void)
+{
+ mutex_lock(&fw_lock);
+ __enable_firmware = true;
+ mutex_unlock(&fw_lock);
+}
+
+static void disable_firmware(void)
+{
+ mutex_lock(&fw_lock);
+ __enable_firmware = false;
+ mutex_unlock(&fw_lock);
+}
+
+/*
+ * When disabled only the built-in firmware and the firmware cache will be
+ * used to look for firmware.
+ */
+static bool firmware_enabled(void)
+{
+ bool enabled = false;
+
+ mutex_lock(&fw_lock);
+ if (__enable_firmware)
+ enabled = true;
+ mutex_unlock(&fw_lock);
+
+ return enabled;
+}
+
static struct firmware_cache fw_cache;
static struct firmware_buf *__allocate_fw_buf(const char *fw_name,
}
#endif
+static int assign_firmware_buf(struct firmware *fw, struct device *device,
+ unsigned int opt_flags)
+{
+ struct firmware_buf *buf = fw->priv;
+
+ mutex_lock(&fw_lock);
+ if (!buf->size || fw_state_is_aborted(&buf->fw_st)) {
+ mutex_unlock(&fw_lock);
+ return -ENOENT;
+ }
+
+ /*
+ * add firmware name into devres list so that we can auto cache
+ * and uncache firmware for device.
+ *
+ * device may has been deleted already, but the problem
+ * should be fixed in devres or driver core.
+ */
+ /* don't cache firmware handled without uevent */
+ if (device && (opt_flags & FW_OPT_UEVENT) &&
+ !(opt_flags & FW_OPT_NOCACHE))
+ fw_add_devm_name(device, buf->fw_id);
+
+ /*
+ * After caching firmware image is started, let it piggyback
+ * on request firmware.
+ */
+ if (!(opt_flags & FW_OPT_NOCACHE) &&
+ buf->fwc->state == FW_LOADER_START_CACHE) {
+ if (fw_cache_piggyback_on_request(buf->fw_id))
+ kref_get(&buf->ref);
+ }
+
+ /* pass the pages buffer to driver at the last minute */
+ fw_set_page_data(buf, fw);
+ mutex_unlock(&fw_lock);
+ return 0;
+}
/*
* user-mode helper code
static LIST_HEAD(pending_fw_head);
-/* reboot notifier for avoid deadlock with usermode_lock */
-static int fw_shutdown_notify(struct notifier_block *unused1,
- unsigned long unused2, void *unused3)
+static void kill_pending_fw_fallback_reqs(bool only_kill_custom)
{
+ struct firmware_buf *buf;
+ struct firmware_buf *next;
+
mutex_lock(&fw_lock);
- while (!list_empty(&pending_fw_head))
- __fw_load_abort(list_first_entry(&pending_fw_head,
- struct firmware_buf,
- pending_list));
+ list_for_each_entry_safe(buf, next, &pending_fw_head, pending_list) {
+ if (!buf->need_uevent || !only_kill_custom)
+ __fw_load_abort(buf);
+ }
mutex_unlock(&fw_lock);
- return NOTIFY_DONE;
}
-static struct notifier_block fw_shutdown_nb = {
- .notifier_call = fw_shutdown_notify,
-};
-
static ssize_t timeout_show(struct class *class, struct class_attribute *attr,
char *buf)
{
static int fw_load_from_user_helper(struct firmware *firmware,
const char *name, struct device *device,
- unsigned int opt_flags, long timeout)
+ unsigned int opt_flags)
{
struct firmware_priv *fw_priv;
+ long timeout;
+ int ret;
+
+ timeout = firmware_loading_timeout();
+ if (opt_flags & FW_OPT_NOWAIT) {
+ timeout = usermodehelper_read_lock_wait(timeout);
+ if (!timeout) {
+ dev_dbg(device, "firmware: %s loading timed out\n",
+ name);
+ return -EBUSY;
+ }
+ } else {
+ ret = usermodehelper_read_trylock();
+ if (WARN_ON(ret)) {
+ dev_err(device, "firmware: %s will not be loaded\n",
+ name);
+ return ret;
+ }
+ }
fw_priv = fw_create_instance(firmware, name, device, opt_flags);
- if (IS_ERR(fw_priv))
- return PTR_ERR(fw_priv);
+ if (IS_ERR(fw_priv)) {
+ ret = PTR_ERR(fw_priv);
+ goto out_unlock;
+ }
fw_priv->buf = firmware->priv;
- return _request_firmware_load(fw_priv, opt_flags, timeout);
-}
+ ret = _request_firmware_load(fw_priv, opt_flags, timeout);
-#ifdef CONFIG_PM_SLEEP
-/* kill pending requests without uevent to avoid blocking suspend */
-static void kill_requests_without_uevent(void)
-{
- struct firmware_buf *buf;
- struct firmware_buf *next;
+ if (!ret)
+ ret = assign_firmware_buf(firmware, device, opt_flags);
- mutex_lock(&fw_lock);
- list_for_each_entry_safe(buf, next, &pending_fw_head, pending_list) {
- if (!buf->need_uevent)
- __fw_load_abort(buf);
- }
- mutex_unlock(&fw_lock);
+out_unlock:
+ usermodehelper_read_unlock();
+
+ return ret;
}
-#endif
#else /* CONFIG_FW_LOADER_USER_HELPER */
static inline int
fw_load_from_user_helper(struct firmware *firmware, const char *name,
- struct device *device, unsigned int opt_flags,
- long timeout)
+ struct device *device, unsigned int opt_flags)
{
return -ENOENT;
}
-#ifdef CONFIG_PM_SLEEP
-static inline void kill_requests_without_uevent(void) { }
-#endif
+static inline void kill_pending_fw_fallback_reqs(bool only_kill_custom) { }
#endif /* CONFIG_FW_LOADER_USER_HELPER */
return 1; /* need to load */
}
-static int assign_firmware_buf(struct firmware *fw, struct device *device,
- unsigned int opt_flags)
-{
- struct firmware_buf *buf = fw->priv;
-
- mutex_lock(&fw_lock);
- if (!buf->size || fw_state_is_aborted(&buf->fw_st)) {
- mutex_unlock(&fw_lock);
- return -ENOENT;
- }
-
- /*
- * add firmware name into devres list so that we can auto cache
- * and uncache firmware for device.
- *
- * device may has been deleted already, but the problem
- * should be fixed in devres or driver core.
- */
- /* don't cache firmware handled without uevent */
- if (device && (opt_flags & FW_OPT_UEVENT) &&
- !(opt_flags & FW_OPT_NOCACHE))
- fw_add_devm_name(device, buf->fw_id);
-
- /*
- * After caching firmware image is started, let it piggyback
- * on request firmware.
- */
- if (!(opt_flags & FW_OPT_NOCACHE) &&
- buf->fwc->state == FW_LOADER_START_CACHE) {
- if (fw_cache_piggyback_on_request(buf->fw_id))
- kref_get(&buf->ref);
- }
-
- /* pass the pages buffer to driver at the last minute */
- fw_set_page_data(buf, fw);
- mutex_unlock(&fw_lock);
- return 0;
-}
-
/* called from request_firmware() and request_firmware_work_func() */
static int
_request_firmware(const struct firmware **firmware_p, const char *name,
unsigned int opt_flags)
{
struct firmware *fw = NULL;
- long timeout;
int ret;
if (!firmware_p)
if (ret <= 0) /* error or already assigned */
goto out;
- ret = 0;
- timeout = firmware_loading_timeout();
- if (opt_flags & FW_OPT_NOWAIT) {
- timeout = usermodehelper_read_lock_wait(timeout);
- if (!timeout) {
- dev_dbg(device, "firmware: %s loading timed out\n",
- name);
- ret = -EBUSY;
- goto out;
- }
- } else {
- ret = usermodehelper_read_trylock();
- if (WARN_ON(ret)) {
- dev_err(device, "firmware: %s will not be loaded\n",
- name);
- goto out;
- }
+ if (!firmware_enabled()) {
+ WARN(1, "firmware request while host is not available\n");
+ ret = -EHOSTDOWN;
+ goto out;
}
ret = fw_get_filesystem_firmware(device, fw->priv);
if (opt_flags & FW_OPT_USERHELPER) {
dev_warn(device, "Falling back to user helper\n");
ret = fw_load_from_user_helper(fw, name, device,
- opt_flags, timeout);
+ opt_flags);
}
- }
-
- if (!ret)
+ } else
ret = assign_firmware_buf(fw, device, opt_flags);
- usermodehelper_read_unlock();
-
out:
if (ret < 0) {
release_firmware(fw);
msecs_to_jiffies(delay));
}
+/**
+ * fw_pm_notify - notifier for suspend/resume
+ * @notify_block: unused
+ * @mode: mode we are switching to
+ * @unused: unused
+ *
+ * Used to modify the firmware_class state as we move in between states.
+ * The firmware_class implements a firmware cache to enable device driver
+ * to fetch firmware upon resume before the root filesystem is ready. We
+ * disable API calls which do not use the built-in firmware or the firmware
+ * cache when we know these calls will not work.
+ *
+ * The inner logic behind all this is a bit complex so it is worth summarizing
+ * the kernel's own suspend/resume process with context and focus on how this
+ * can impact the firmware API.
+ *
+ * First a review on how we go to suspend::
+ *
+ * pm_suspend() --> enter_state() -->
+ * sys_sync()
+ * suspend_prepare() -->
+ * __pm_notifier_call_chain(PM_SUSPEND_PREPARE, ...);
+ * suspend_freeze_processes() -->
+ * freeze_processes() -->
+ * __usermodehelper_set_disable_depth(UMH_DISABLED);
+ * freeze all tasks ...
+ * freeze_kernel_threads()
+ * suspend_devices_and_enter() -->
+ * dpm_suspend_start() -->
+ * dpm_prepare()
+ * dpm_suspend()
+ * suspend_enter() -->
+ * platform_suspend_prepare()
+ * dpm_suspend_late()
+ * freeze_enter()
+ * syscore_suspend()
+ *
+ * When we resume we bail out of a loop from suspend_devices_and_enter() and
+ * unwind back out to the caller enter_state() where we were before as follows::
+ *
+ * enter_state() -->
+ * suspend_devices_and_enter() --> (bail from loop)
+ * dpm_resume_end() -->
+ * dpm_resume()
+ * dpm_complete()
+ * suspend_finish() -->
+ * suspend_thaw_processes() -->
+ * thaw_processes() -->
+ * __usermodehelper_set_disable_depth(UMH_FREEZING);
+ * thaw_workqueues();
+ * thaw all processes ...
+ * usermodehelper_enable();
+ * pm_notifier_call_chain(PM_POST_SUSPEND);
+ *
+ * fw_pm_notify() works through pm_notifier_call_chain().
+ */
static int fw_pm_notify(struct notifier_block *notify_block,
unsigned long mode, void *unused)
{
case PM_HIBERNATION_PREPARE:
case PM_SUSPEND_PREPARE:
case PM_RESTORE_PREPARE:
- kill_requests_without_uevent();
+ /*
+ * kill pending fallback requests with a custom fallback
+ * to avoid stalling suspend.
+ */
+ kill_pending_fw_fallback_reqs(true);
device_cache_fw_images();
+ disable_firmware();
break;
case PM_POST_SUSPEND:
mutex_lock(&fw_lock);
fw_cache.state = FW_LOADER_NO_CACHE;
mutex_unlock(&fw_lock);
+ enable_firmware();
device_uncache_fw_images_delay(10 * MSEC_PER_SEC);
break;
#endif
}
+static int fw_shutdown_notify(struct notifier_block *unused1,
+ unsigned long unused2, void *unused3)
+{
+ disable_firmware();
+ /*
+ * Kill all pending fallback requests to avoid both stalling shutdown,
+ * and avoid a deadlock with the usermode_lock.
+ */
+ kill_pending_fw_fallback_reqs(false);
+
+ return NOTIFY_DONE;
+}
+
+static struct notifier_block fw_shutdown_nb = {
+ .notifier_call = fw_shutdown_notify,
+};
+
static int __init firmware_class_init(void)
{
+ enable_firmware();
fw_cache_init();
-#ifdef CONFIG_FW_LOADER_USER_HELPER
register_reboot_notifier(&fw_shutdown_nb);
+#ifdef CONFIG_FW_LOADER_USER_HELPER
return class_register(&firmware_class);
#else
return 0;
static void __exit firmware_class_exit(void)
{
+ disable_firmware();
#ifdef CONFIG_PM_SLEEP
unregister_syscore_ops(&fw_syscore_ops);
unregister_pm_notifier(&fw_cache.pm_notify);
#endif
-#ifdef CONFIG_FW_LOADER_USER_HELPER
unregister_reboot_notifier(&fw_shutdown_nb);
+#ifdef CONFIG_FW_LOADER_USER_HELPER
class_unregister(&firmware_class);
#endif
}
if (!pfn_valid_within(pfn))
return -1;
#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
- if (system_state == SYSTEM_BOOTING)
+ if (system_state < SYSTEM_RUNNING)
return early_pfn_to_nid(pfn);
#endif
page = pfn_to_page(pfn);
{
int ret;
+ if (dev->of_node_reused)
+ return 0;
+
dev->pins = devm_kzalloc(dev, sizeof(*(dev->pins)), GFP_KERNEL);
if (!dev->pins)
return -ENOMEM;
domain = msi_create_irq_domain(fwnode, info, parent);
if (domain)
- domain->bus_token = DOMAIN_BUS_PLATFORM_MSI;
+ irq_domain_update_bus_token(domain, DOMAIN_BUS_PLATFORM_MSI);
return domain;
}
const char *buf, size_t count)
{
struct platform_device *pdev = to_platform_device(dev);
- char *driver_override, *old = pdev->driver_override, *cp;
+ char *driver_override, *old, *cp;
if (count > PATH_MAX)
return -EINVAL;
if (cp)
*cp = '\0';
+ device_lock(dev);
+ old = pdev->driver_override;
if (strlen(driver_override)) {
pdev->driver_override = driver_override;
} else {
kfree(driver_override);
pdev->driver_override = NULL;
}
+ device_unlock(dev);
kfree(old);
struct device_attribute *attr, char *buf)
{
struct platform_device *pdev = to_platform_device(dev);
+ ssize_t len;
- return sprintf(buf, "%s\n", pdev->driver_override);
+ device_lock(dev);
+ len = sprintf(buf, "%s\n", pdev->driver_override);
+ device_unlock(dev);
+ return len;
}
static DEVICE_ATTR_RW(driver_override);
#define genpd_is_always_on(genpd) (genpd->flags & GENPD_FLAG_ALWAYS_ON)
static inline bool irq_safe_dev_in_no_sleep_domain(struct device *dev,
- struct generic_pm_domain *genpd)
+ const struct generic_pm_domain *genpd)
{
bool ret;
return pd_to_genpd(dev->pm_domain);
}
-static int genpd_stop_dev(struct generic_pm_domain *genpd, struct device *dev)
+static int genpd_stop_dev(const struct generic_pm_domain *genpd,
+ struct device *dev)
{
return GENPD_DEV_CALLBACK(genpd, int, stop, dev);
}
-static int genpd_start_dev(struct generic_pm_domain *genpd, struct device *dev)
+static int genpd_start_dev(const struct generic_pm_domain *genpd,
+ struct device *dev)
{
return GENPD_DEV_CALLBACK(genpd, int, start, dev);
}
pdd = dev->power.subsys_data ?
dev->power.subsys_data->domain_data : NULL;
- if (pdd && pdd->dev) {
+ if (pdd) {
to_gpd_data(pdd)->td.constraint_changed = true;
genpd = dev_to_genpd(dev);
} else {
#ifdef CONFIG_PM_SLEEP
-static bool genpd_dev_active_wakeup(struct generic_pm_domain *genpd,
+static bool genpd_dev_active_wakeup(const struct generic_pm_domain *genpd,
struct device *dev)
{
return GENPD_DEV_CALLBACK(genpd, bool, active_wakeup, dev);
* signal remote wakeup from the system's working state as needed by runtime PM.
* Return 'true' in either of the above cases.
*/
-static bool resume_needed(struct device *dev, struct generic_pm_domain *genpd)
+static bool resume_needed(struct device *dev,
+ const struct generic_pm_domain *genpd)
{
bool active_wakeup;
}
/**
- * pm_genpd_suspend_noirq - Completion of suspend of device in an I/O PM domain.
+ * genpd_finish_suspend - Completion of suspend or hibernation of device in an
+ * I/O pm domain.
* @dev: Device to suspend.
+ * @poweroff: Specifies if this is a poweroff_noirq or suspend_noirq callback.
*
* Stop the device and remove power from the domain if all devices in it have
* been stopped.
*/
-static int pm_genpd_suspend_noirq(struct device *dev)
+static int genpd_finish_suspend(struct device *dev, bool poweroff)
{
struct generic_pm_domain *genpd;
int ret;
- dev_dbg(dev, "%s()\n", __func__);
-
genpd = dev_to_genpd(dev);
if (IS_ERR(genpd))
return -EINVAL;
if (dev->power.wakeup_path && genpd_dev_active_wakeup(genpd, dev))
return 0;
+ if (poweroff)
+ ret = pm_generic_poweroff_noirq(dev);
+ else
+ ret = pm_generic_suspend_noirq(dev);
+ if (ret)
+ return ret;
+
if (genpd->dev_ops.stop && genpd->dev_ops.start) {
ret = pm_runtime_force_suspend(dev);
if (ret)
return 0;
}
+/**
+ * pm_genpd_suspend_noirq - Completion of suspend of device in an I/O PM domain.
+ * @dev: Device to suspend.
+ *
+ * Stop the device and remove power from the domain if all devices in it have
+ * been stopped.
+ */
+static int pm_genpd_suspend_noirq(struct device *dev)
+{
+ dev_dbg(dev, "%s()\n", __func__);
+
+ return genpd_finish_suspend(dev, false);
+}
+
/**
* pm_genpd_resume_noirq - Start of resume of device in an I/O PM domain.
* @dev: Device to resume.
if (genpd->dev_ops.stop && genpd->dev_ops.start)
ret = pm_runtime_force_resume(dev);
+ ret = pm_generic_resume_noirq(dev);
+ if (ret)
+ return ret;
+
return ret;
}
*/
static int pm_genpd_freeze_noirq(struct device *dev)
{
- struct generic_pm_domain *genpd;
+ const struct generic_pm_domain *genpd;
int ret = 0;
dev_dbg(dev, "%s()\n", __func__);
if (IS_ERR(genpd))
return -EINVAL;
+ ret = pm_generic_freeze_noirq(dev);
+ if (ret)
+ return ret;
+
if (genpd->dev_ops.stop && genpd->dev_ops.start)
ret = pm_runtime_force_suspend(dev);
*/
static int pm_genpd_thaw_noirq(struct device *dev)
{
- struct generic_pm_domain *genpd;
+ const struct generic_pm_domain *genpd;
int ret = 0;
dev_dbg(dev, "%s()\n", __func__);
if (IS_ERR(genpd))
return -EINVAL;
- if (genpd->dev_ops.stop && genpd->dev_ops.start)
+ if (genpd->dev_ops.stop && genpd->dev_ops.start) {
ret = pm_runtime_force_resume(dev);
+ if (ret)
+ return ret;
+ }
- return ret;
+ return pm_generic_thaw_noirq(dev);
+}
+
+/**
+ * pm_genpd_poweroff_noirq - Completion of hibernation of device in an
+ * I/O PM domain.
+ * @dev: Device to poweroff.
+ *
+ * Stop the device and remove power from the domain if all devices in it have
+ * been stopped.
+ */
+static int pm_genpd_poweroff_noirq(struct device *dev)
+{
+ dev_dbg(dev, "%s()\n", __func__);
+
+ return genpd_finish_suspend(dev, true);
}
/**
genpd_sync_power_on(genpd, true, 0);
genpd_unlock(genpd);
- if (genpd->dev_ops.stop && genpd->dev_ops.start)
+ if (genpd->dev_ops.stop && genpd->dev_ops.start) {
ret = pm_runtime_force_resume(dev);
+ if (ret)
+ return ret;
+ }
- return ret;
+ return pm_generic_restore_noirq(dev);
}
/**
{
struct generic_pm_domain *genpd;
- genpd = dev_to_genpd(dev);
- if (!pm_genpd_present(genpd))
+ genpd = genpd_lookup_dev(dev);
+ if (!genpd)
return;
if (suspend) {
int pm_genpd_remove_subdomain(struct generic_pm_domain *genpd,
struct generic_pm_domain *subdomain)
{
- struct gpd_link *link;
+ struct gpd_link *l, *link;
int ret = -EINVAL;
if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(subdomain))
goto out;
}
- list_for_each_entry(link, &genpd->master_links, master_node) {
+ list_for_each_entry_safe(link, l, &genpd->master_links, master_node) {
if (link->slave != subdomain)
continue;
genpd->domain.ops.resume_noirq = pm_genpd_resume_noirq;
genpd->domain.ops.freeze_noirq = pm_genpd_freeze_noirq;
genpd->domain.ops.thaw_noirq = pm_genpd_thaw_noirq;
- genpd->domain.ops.poweroff_noirq = pm_genpd_suspend_noirq;
+ genpd->domain.ops.poweroff_noirq = pm_genpd_poweroff_noirq;
genpd->domain.ops.restore_noirq = pm_genpd_restore_noirq;
genpd->domain.ops.complete = pm_genpd_complete;
*/
void of_genpd_del_provider(struct device_node *np)
{
- struct of_genpd_provider *cp;
+ struct of_genpd_provider *cp, *tmp;
struct generic_pm_domain *gpd;
mutex_lock(&gpd_list_lock);
mutex_lock(&of_genpd_mutex);
- list_for_each_entry(cp, &of_genpd_providers, link) {
+ list_for_each_entry_safe(cp, tmp, &of_genpd_providers, link) {
if (cp->node == np) {
/*
* For each PM domain associated with the
*/
struct generic_pm_domain *of_genpd_remove_last(struct device_node *np)
{
- struct generic_pm_domain *gpd, *genpd = ERR_PTR(-ENOENT);
+ struct generic_pm_domain *gpd, *tmp, *genpd = ERR_PTR(-ENOENT);
int ret;
if (IS_ERR_OR_NULL(np))
return ERR_PTR(-EINVAL);
mutex_lock(&gpd_list_lock);
- list_for_each_entry(gpd, &gpd_list, gpd_list_node) {
+ list_for_each_entry_safe(gpd, tmp, &gpd_list, gpd_list_node) {
if (gpd->provider == &np->fwnode) {
ret = genpd_remove(gpd);
genpd = ret ? ERR_PTR(ret) : gpd;
return td->cached_suspend_ok;
}
-/**
- * default_power_down_ok - Default generic PM domain power off governor routine.
- * @pd: PM domain to check.
- *
- * This routine must be executed under the PM domain's lock.
- */
static bool __default_power_down_ok(struct dev_pm_domain *pd,
unsigned int state)
{
return true;
}
+/**
+ * default_power_down_ok - Default generic PM domain power off governor routine.
+ * @pd: PM domain to check.
+ *
+ * This routine must be executed under the PM domain's lock.
+ */
static bool default_power_down_ok(struct dev_pm_domain *pd)
{
struct generic_pm_domain *genpd = pd_to_genpd(pd);
static int async_error;
-static char *pm_verb(int event)
+static const char *pm_verb(int event)
{
switch (event) {
case PM_EVENT_SUSPEND:
}
static void initcall_debug_report(struct device *dev, ktime_t calltime,
- int error, pm_message_t state, char *info)
+ int error, pm_message_t state,
+ const char *info)
{
ktime_t rettime;
s64 nsecs;
return NULL;
}
-static void pm_dev_dbg(struct device *dev, pm_message_t state, char *info)
+static void pm_dev_dbg(struct device *dev, pm_message_t state, const char *info)
{
dev_dbg(dev, "%s%s%s\n", info, pm_verb(state.event),
((state.event & PM_EVENT_SLEEP) && device_may_wakeup(dev)) ?
", may wakeup" : "");
}
-static void pm_dev_err(struct device *dev, pm_message_t state, char *info,
+static void pm_dev_err(struct device *dev, pm_message_t state, const char *info,
int error)
{
printk(KERN_ERR "PM: Device %s failed to %s%s: error %d\n",
dev_name(dev), pm_verb(state.event), info, error);
}
-static void dpm_show_time(ktime_t starttime, pm_message_t state, char *info)
+#ifdef CONFIG_PM_DEBUG
+static void dpm_show_time(ktime_t starttime, pm_message_t state,
+ const char *info)
{
ktime_t calltime;
u64 usecs64;
info ?: "", info ? " " : "", pm_verb(state.event),
usecs / USEC_PER_MSEC, usecs % USEC_PER_MSEC);
}
+#else
+static inline void dpm_show_time(ktime_t starttime, pm_message_t state,
+ const char *info) {}
+#endif /* CONFIG_PM_DEBUG */
static int dpm_run_callback(pm_callback_t cb, struct device *dev,
- pm_message_t state, char *info)
+ pm_message_t state, const char *info)
{
ktime_t calltime;
int error;
static int device_resume_noirq(struct device *dev, pm_message_t state, bool async)
{
pm_callback_t callback = NULL;
- char *info = NULL;
+ const char *info = NULL;
int error = 0;
TRACE_DEVICE(dev);
static int device_resume_early(struct device *dev, pm_message_t state, bool async)
{
pm_callback_t callback = NULL;
- char *info = NULL;
+ const char *info = NULL;
int error = 0;
TRACE_DEVICE(dev);
static int device_resume(struct device *dev, pm_message_t state, bool async)
{
pm_callback_t callback = NULL;
- char *info = NULL;
+ const char *info = NULL;
int error = 0;
DECLARE_DPM_WATCHDOG_ON_STACK(wd);
static void device_complete(struct device *dev, pm_message_t state)
{
void (*callback)(struct device *) = NULL;
- char *info = NULL;
+ const char *info = NULL;
if (dev->power.syscore)
return;
static int __device_suspend_noirq(struct device *dev, pm_message_t state, bool async)
{
pm_callback_t callback = NULL;
- char *info = NULL;
+ const char *info = NULL;
int error = 0;
TRACE_DEVICE(dev);
if (async_error)
goto Complete;
- if (pm_wakeup_pending()) {
- async_error = -EBUSY;
- goto Complete;
- }
-
if (dev->power.syscore || dev->power.direct_complete)
goto Complete;
static int __device_suspend_late(struct device *dev, pm_message_t state, bool async)
{
pm_callback_t callback = NULL;
- char *info = NULL;
+ const char *info = NULL;
int error = 0;
TRACE_DEVICE(dev);
*/
static int legacy_suspend(struct device *dev, pm_message_t state,
int (*cb)(struct device *dev, pm_message_t state),
- char *info)
+ const char *info)
{
int error;
ktime_t calltime;
static int __device_suspend(struct device *dev, pm_message_t state, bool async)
{
pm_callback_t callback = NULL;
- char *info = NULL;
+ const char *info = NULL;
int error = 0;
DECLARE_DPM_WATCHDOG_ON_STACK(wd);
{
struct opp_table *opp_table;
struct dev_pm_opp *opp;
- struct regulator *reg, **regulators;
+ struct regulator *reg;
unsigned long latency_ns = 0;
int ret, i, count;
struct {
if (!count)
goto put_opp_table;
- regulators = kmalloc_array(count, sizeof(*regulators), GFP_KERNEL);
- if (!regulators)
- goto put_opp_table;
-
uV = kmalloc_array(count, sizeof(*uV), GFP_KERNEL);
if (!uV)
- goto free_regulators;
-
- memcpy(regulators, opp_table->regulators, count * sizeof(*regulators));
+ goto put_opp_table;
mutex_lock(&opp_table->lock);
* isn't freed, while we are executing this routine.
*/
for (i = 0; i < count; i++) {
- reg = regulators[i];
+ reg = opp_table->regulators[i];
ret = regulator_set_voltage_time(reg, uV[i].min, uV[i].max);
if (ret > 0)
latency_ns += ret * 1000;
}
kfree(uV);
-free_regulators:
- kfree(regulators);
put_opp_table:
dev_pm_opp_put_opp_table(opp_table);
return ret;
}
-static int _generic_set_opp(struct dev_pm_set_opp_data *data)
+static int _generic_set_opp_regulator(const struct opp_table *opp_table,
+ struct device *dev,
+ unsigned long old_freq,
+ unsigned long freq,
+ struct dev_pm_opp_supply *old_supply,
+ struct dev_pm_opp_supply *new_supply)
{
- struct dev_pm_opp_supply *old_supply = data->old_opp.supplies;
- struct dev_pm_opp_supply *new_supply = data->new_opp.supplies;
- unsigned long old_freq = data->old_opp.rate, freq = data->new_opp.rate;
- struct regulator *reg = data->regulators[0];
- struct device *dev= data->dev;
+ struct regulator *reg = opp_table->regulators[0];
int ret;
/* This function only supports single regulator per device */
- if (WARN_ON(data->regulator_count > 1)) {
+ if (WARN_ON(opp_table->regulator_count > 1)) {
dev_err(dev, "multiple regulators are not supported\n");
return -EINVAL;
}
}
/* Change frequency */
- ret = _generic_set_opp_clk_only(dev, data->clk, old_freq, freq);
+ ret = _generic_set_opp_clk_only(dev, opp_table->clk, old_freq, freq);
if (ret)
goto restore_voltage;
return 0;
restore_freq:
- if (_generic_set_opp_clk_only(dev, data->clk, freq, old_freq))
+ if (_generic_set_opp_clk_only(dev, opp_table->clk, freq, old_freq))
dev_err(dev, "%s: failed to restore old-freq (%lu Hz)\n",
__func__, old_freq);
restore_voltage:
/* This shouldn't harm even if the voltages weren't updated earlier */
- if (old_supply->u_volt)
+ if (old_supply)
_set_opp_voltage(dev, reg, old_supply);
return ret;
{
struct opp_table *opp_table;
unsigned long freq, old_freq;
- int (*set_opp)(struct dev_pm_set_opp_data *data);
struct dev_pm_opp *old_opp, *opp;
- struct regulator **regulators;
- struct dev_pm_set_opp_data *data;
struct clk *clk;
int ret, size;
dev_dbg(dev, "%s: switching OPP: %lu Hz --> %lu Hz\n", __func__,
old_freq, freq);
- regulators = opp_table->regulators;
-
/* Only frequency scaling */
- if (!regulators) {
+ if (!opp_table->regulators) {
ret = _generic_set_opp_clk_only(dev, clk, old_freq, freq);
- goto put_opps;
- }
+ } else if (!opp_table->set_opp) {
+ ret = _generic_set_opp_regulator(opp_table, dev, old_freq, freq,
+ IS_ERR(old_opp) ? NULL : old_opp->supplies,
+ opp->supplies);
+ } else {
+ struct dev_pm_set_opp_data *data;
- if (opp_table->set_opp)
- set_opp = opp_table->set_opp;
- else
- set_opp = _generic_set_opp;
-
- data = opp_table->set_opp_data;
- data->regulators = regulators;
- data->regulator_count = opp_table->regulator_count;
- data->clk = clk;
- data->dev = dev;
-
- data->old_opp.rate = old_freq;
- size = sizeof(*opp->supplies) * opp_table->regulator_count;
- if (IS_ERR(old_opp))
- memset(data->old_opp.supplies, 0, size);
- else
- memcpy(data->old_opp.supplies, old_opp->supplies, size);
+ data = opp_table->set_opp_data;
+ data->regulators = opp_table->regulators;
+ data->regulator_count = opp_table->regulator_count;
+ data->clk = clk;
+ data->dev = dev;
- data->new_opp.rate = freq;
- memcpy(data->new_opp.supplies, opp->supplies, size);
+ data->old_opp.rate = old_freq;
+ size = sizeof(*opp->supplies) * opp_table->regulator_count;
+ if (IS_ERR(old_opp))
+ memset(data->old_opp.supplies, 0, size);
+ else
+ memcpy(data->old_opp.supplies, old_opp->supplies, size);
- ret = set_opp(data);
+ data->new_opp.rate = freq;
+ memcpy(data->new_opp.supplies, opp->supplies, size);
+
+ ret = opp_table->set_opp(data);
+ }
-put_opps:
dev_pm_opp_put(opp);
put_old_opp:
if (!IS_ERR(old_opp))
}
EXPORT_SYMBOL_GPL(dev_pm_opp_put_regulators);
+/**
+ * dev_pm_opp_set_clkname() - Set clk name for the device
+ * @dev: Device for which clk name is being set.
+ * @name: Clk name.
+ *
+ * In order to support OPP switching, OPP layer needs to get pointer to the
+ * clock for the device. Simple cases work fine without using this routine (i.e.
+ * by passing connection-id as NULL), but for a device with multiple clocks
+ * available, the OPP core needs to know the exact name of the clk to use.
+ *
+ * This must be called before any OPPs are initialized for the device.
+ */
+struct opp_table *dev_pm_opp_set_clkname(struct device *dev, const char *name)
+{
+ struct opp_table *opp_table;
+ int ret;
+
+ opp_table = dev_pm_opp_get_opp_table(dev);
+ if (!opp_table)
+ return ERR_PTR(-ENOMEM);
+
+ /* This should be called before OPPs are initialized */
+ if (WARN_ON(!list_empty(&opp_table->opp_list))) {
+ ret = -EBUSY;
+ goto err;
+ }
+
+ /* Already have default clk set, free it */
+ if (!IS_ERR(opp_table->clk))
+ clk_put(opp_table->clk);
+
+ /* Find clk for the device */
+ opp_table->clk = clk_get(dev, name);
+ if (IS_ERR(opp_table->clk)) {
+ ret = PTR_ERR(opp_table->clk);
+ if (ret != -EPROBE_DEFER) {
+ dev_err(dev, "%s: Couldn't find clock: %d\n", __func__,
+ ret);
+ }
+ goto err;
+ }
+
+ return opp_table;
+
+err:
+ dev_pm_opp_put_opp_table(opp_table);
+
+ return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(dev_pm_opp_set_clkname);
+
+/**
+ * dev_pm_opp_put_clkname() - Releases resources blocked for clk.
+ * @opp_table: OPP table returned from dev_pm_opp_set_clkname().
+ */
+void dev_pm_opp_put_clkname(struct opp_table *opp_table)
+{
+ /* Make sure there are no concurrent readers while updating opp_table */
+ WARN_ON(!list_empty(&opp_table->opp_list));
+
+ clk_put(opp_table->clk);
+ opp_table->clk = ERR_PTR(-EINVAL);
+
+ dev_pm_opp_put_opp_table(opp_table);
+}
+EXPORT_SYMBOL_GPL(dev_pm_opp_put_clkname);
+
/**
* dev_pm_opp_register_set_opp_helper() - Register custom set OPP helper
* @dev: Device for which the helper is getting registered.
struct dentry *pdentry)
{
struct dentry *d;
- int i = 0;
+ int i;
char *name;
- /* Always create at least supply-0 directory */
- do {
+ for (i = 0; i < opp_table->regulator_count; i++) {
name = kasprintf(GFP_KERNEL, "supply-%d", i);
/* Create per-opp directory */
if (!debugfs_create_ulong("u_amp", S_IRUGO, d,
&opp->supplies[i].u_amp))
return false;
- } while (++i < opp_table->regulator_count);
+ }
return true;
}
prop = of_find_property(opp->np, name, NULL);
/* Missing property isn't a problem, but an invalid entry is */
- if (!prop)
- return 0;
+ if (!prop) {
+ if (!opp_table->regulator_count)
+ return 0;
+
+ dev_err(dev, "%s: opp-microvolt missing although OPP managing regulators\n",
+ __func__);
+ return -EINVAL;
+ }
}
vcount = of_property_count_u32_elems(opp->np, name);
#endif /* CONFIG_PM_ADVANCED_DEBUG */
NULL,
};
-static struct attribute_group pm_attr_group = {
+static const struct attribute_group pm_attr_group = {
.name = power_group_name,
.attrs = power_attrs,
};
#endif
NULL,
};
-static struct attribute_group pm_wakeup_attr_group = {
+static const struct attribute_group pm_wakeup_attr_group = {
.name = power_group_name,
.attrs = wakeup_attrs,
};
&dev_attr_autosuspend_delay_ms.attr,
NULL,
};
-static struct attribute_group pm_runtime_attr_group = {
+static const struct attribute_group pm_runtime_attr_group = {
.name = power_group_name,
.attrs = runtime_attrs,
};
&dev_attr_pm_qos_resume_latency_us.attr,
NULL,
};
-static struct attribute_group pm_qos_resume_latency_attr_group = {
+static const struct attribute_group pm_qos_resume_latency_attr_group = {
.name = power_group_name,
.attrs = pm_qos_resume_latency_attrs,
};
&dev_attr_pm_qos_latency_tolerance_us.attr,
NULL,
};
-static struct attribute_group pm_qos_latency_tolerance_attr_group = {
+static const struct attribute_group pm_qos_latency_tolerance_attr_group = {
.name = power_group_name,
.attrs = pm_qos_latency_tolerance_attrs,
};
&dev_attr_pm_qos_remote_wakeup.attr,
NULL,
};
-static struct attribute_group pm_qos_flags_attr_group = {
+static const struct attribute_group pm_qos_flags_attr_group = {
.name = power_group_name,
.attrs = pm_qos_flags_attrs,
};
/* First wakeup IRQ seen by the kernel in the last cycle. */
unsigned int pm_wakeup_irq __read_mostly;
-/* If set and the system is suspending, terminate the suspend. */
-static bool pm_abort_suspend __read_mostly;
+/* If greater than 0 and the system is suspending, terminate the suspend. */
+static atomic_t pm_abort_suspend __read_mostly;
/*
* Combined counters of registered wakeup events and wakeup events in progress.
static DECLARE_WAIT_QUEUE_HEAD(wakeup_count_wait_queue);
+DEFINE_STATIC_SRCU(wakeup_srcu);
+
static struct wakeup_source deleted_ws = {
.name = "deleted",
.lock = __SPIN_LOCK_UNLOCKED(deleted_ws.lock),
spin_lock_irqsave(&events_lock, flags);
list_del_rcu(&ws->entry);
spin_unlock_irqrestore(&events_lock, flags);
- synchronize_rcu();
+ synchronize_srcu(&wakeup_srcu);
}
EXPORT_SYMBOL_GPL(wakeup_source_remove);
void device_wakeup_arm_wake_irqs(void)
{
struct wakeup_source *ws;
+ int srcuidx;
- rcu_read_lock();
+ srcuidx = srcu_read_lock(&wakeup_srcu);
list_for_each_entry_rcu(ws, &wakeup_sources, entry)
dev_pm_arm_wake_irq(ws->wakeirq);
-
- rcu_read_unlock();
+ srcu_read_unlock(&wakeup_srcu, srcuidx);
}
/**
void device_wakeup_disarm_wake_irqs(void)
{
struct wakeup_source *ws;
+ int srcuidx;
- rcu_read_lock();
+ srcuidx = srcu_read_lock(&wakeup_srcu);
list_for_each_entry_rcu(ws, &wakeup_sources, entry)
dev_pm_disarm_wake_irq(ws->wakeirq);
-
- rcu_read_unlock();
+ srcu_read_unlock(&wakeup_srcu, srcuidx);
}
/**
void pm_print_active_wakeup_sources(void)
{
struct wakeup_source *ws;
- int active = 0;
+ int srcuidx, active = 0;
struct wakeup_source *last_activity_ws = NULL;
- rcu_read_lock();
+ srcuidx = srcu_read_lock(&wakeup_srcu);
list_for_each_entry_rcu(ws, &wakeup_sources, entry) {
if (ws->active) {
pr_debug("active wakeup source: %s\n", ws->name);
if (!active && last_activity_ws)
pr_debug("last active wakeup source: %s\n",
last_activity_ws->name);
- rcu_read_unlock();
+ srcu_read_unlock(&wakeup_srcu, srcuidx);
}
EXPORT_SYMBOL_GPL(pm_print_active_wakeup_sources);
pm_print_active_wakeup_sources();
}
- return ret || pm_abort_suspend;
+ return ret || atomic_read(&pm_abort_suspend) > 0;
}
void pm_system_wakeup(void)
{
- pm_abort_suspend = true;
+ atomic_inc(&pm_abort_suspend);
freeze_wake();
}
EXPORT_SYMBOL_GPL(pm_system_wakeup);
-void pm_wakeup_clear(void)
+void pm_system_cancel_wakeup(void)
+{
+ atomic_dec(&pm_abort_suspend);
+}
+
+void pm_wakeup_clear(bool reset)
{
- pm_abort_suspend = false;
pm_wakeup_irq = 0;
+ if (reset)
+ atomic_set(&pm_abort_suspend, 0);
}
void pm_system_irq_wakeup(unsigned int irq_number)
{
struct wakeup_source *ws;
ktime_t now = ktime_get();
+ int srcuidx;
- rcu_read_lock();
+ srcuidx = srcu_read_lock(&wakeup_srcu);
list_for_each_entry_rcu(ws, &wakeup_sources, entry) {
spin_lock_irq(&ws->lock);
if (ws->autosleep_enabled != set) {
}
spin_unlock_irq(&ws->lock);
}
- rcu_read_unlock();
+ srcu_read_unlock(&wakeup_srcu, srcuidx);
}
#endif /* CONFIG_PM_AUTOSLEEP */
static int wakeup_sources_stats_show(struct seq_file *m, void *unused)
{
struct wakeup_source *ws;
+ int srcuidx;
seq_puts(m, "name\t\tactive_count\tevent_count\twakeup_count\t"
"expire_count\tactive_since\ttotal_time\tmax_time\t"
"last_change\tprevent_suspend_time\n");
- rcu_read_lock();
+ srcuidx = srcu_read_lock(&wakeup_srcu);
list_for_each_entry_rcu(ws, &wakeup_sources, entry)
print_wakeup_source_stats(m, ws);
- rcu_read_unlock();
+ srcu_read_unlock(&wakeup_srcu, srcuidx);
print_wakeup_source_stats(m, &deleted_ws);
# subsystems should select the appropriate symbols.
config REGMAP
- default y if (REGMAP_I2C || REGMAP_SPI || REGMAP_SPMI || REGMAP_AC97 || REGMAP_MMIO || REGMAP_IRQ)
+ default y if (REGMAP_I2C || REGMAP_SPI || REGMAP_SPMI || REGMAP_W1 || REGMAP_AC97 || REGMAP_MMIO || REGMAP_IRQ)
+ select IRQ_DOMAIN if REGMAP_IRQ
+ bool
+
+config REGCACHE_COMPRESSED
select LZO_COMPRESS
select LZO_DECOMPRESS
- select IRQ_DOMAIN if REGMAP_IRQ
bool
config REGMAP_AC97
tristate
depends on SPMI
+config REGMAP_W1
+ tristate
+ depends on W1
+
config REGMAP_MMIO
tristate
CFLAGS_regmap.o := -I$(src)
obj-$(CONFIG_REGMAP) += regmap.o regcache.o
-obj-$(CONFIG_REGMAP) += regcache-rbtree.o regcache-lzo.o regcache-flat.o
+obj-$(CONFIG_REGMAP) += regcache-rbtree.o regcache-flat.o
+obj-$(CONFIG_REGCACHE_COMPRESSED) += regcache-lzo.o
obj-$(CONFIG_DEBUG_FS) += regmap-debugfs.o
obj-$(CONFIG_REGMAP_AC97) += regmap-ac97.o
obj-$(CONFIG_REGMAP_I2C) += regmap-i2c.o
obj-$(CONFIG_REGMAP_SPMI) += regmap-spmi.o
obj-$(CONFIG_REGMAP_MMIO) += regmap-mmio.o
obj-$(CONFIG_REGMAP_IRQ) += regmap-irq.o
+obj-$(CONFIG_REGMAP_W1) += regmap-w1.o
static const struct regcache_ops *cache_types[] = {
®cache_rbtree_ops,
+#if IS_ENABLED(CONFIG_REGCACHE_COMPRESSED)
®cache_lzo_ops,
+#endif
®cache_flat_ops,
};
mutex_lock(&d->lock);
}
+static int regmap_irq_update_bits(struct regmap_irq_chip_data *d,
+ unsigned int reg, unsigned int mask,
+ unsigned int val)
+{
+ if (d->chip->mask_writeonly)
+ return regmap_write_bits(d->map, reg, mask, val);
+ else
+ return regmap_update_bits(d->map, reg, mask, val);
+}
+
static void regmap_irq_sync_unlock(struct irq_data *data)
{
struct regmap_irq_chip_data *d = irq_data_get_irq_chip_data(data);
reg = d->chip->mask_base +
(i * map->reg_stride * d->irq_reg_stride);
if (d->chip->mask_invert) {
- ret = regmap_update_bits(d->map, reg,
+ ret = regmap_irq_update_bits(d, reg,
d->mask_buf_def[i], ~d->mask_buf[i]);
} else if (d->chip->unmask_base) {
/* set mask with mask_base register */
- ret = regmap_update_bits(d->map, reg,
+ ret = regmap_irq_update_bits(d, reg,
d->mask_buf_def[i], ~d->mask_buf[i]);
if (ret < 0)
dev_err(d->map->dev,
unmask_offset = d->chip->unmask_base -
d->chip->mask_base;
/* clear mask with unmask_base register */
- ret = regmap_update_bits(d->map,
+ ret = regmap_irq_update_bits(d,
reg + unmask_offset,
d->mask_buf_def[i],
d->mask_buf[i]);
} else {
- ret = regmap_update_bits(d->map, reg,
+ ret = regmap_irq_update_bits(d, reg,
d->mask_buf_def[i], d->mask_buf[i]);
}
if (ret != 0)
(i * map->reg_stride * d->irq_reg_stride);
if (d->wake_buf) {
if (d->chip->wake_invert)
- ret = regmap_update_bits(d->map, reg,
+ ret = regmap_irq_update_bits(d, reg,
d->mask_buf_def[i],
~d->wake_buf[i]);
else
- ret = regmap_update_bits(d->map, reg,
+ ret = regmap_irq_update_bits(d, reg,
d->mask_buf_def[i],
d->wake_buf[i]);
if (ret != 0)
reg = d->chip->type_base +
(i * map->reg_stride * d->type_reg_stride);
if (d->chip->type_invert)
- ret = regmap_update_bits(d->map, reg,
+ ret = regmap_irq_update_bits(d, reg,
d->type_buf_def[i], ~d->type_buf[i]);
else
- ret = regmap_update_bits(d->map, reg,
+ ret = regmap_irq_update_bits(d, reg,
d->type_buf_def[i], d->type_buf[i]);
if (ret != 0)
dev_err(d->map->dev, "Failed to sync type in %x\n",
static const struct irq_domain_ops regmap_domain_ops = {
.map = regmap_irq_map,
- .xlate = irq_domain_xlate_twocell,
+ .xlate = irq_domain_xlate_onetwocell,
};
/**
reg = chip->mask_base +
(i * map->reg_stride * d->irq_reg_stride);
if (chip->mask_invert)
- ret = regmap_update_bits(map, reg,
+ ret = regmap_irq_update_bits(d, reg,
d->mask_buf[i], ~d->mask_buf[i]);
else if (d->chip->unmask_base) {
unmask_offset = d->chip->unmask_base -
d->chip->mask_base;
- ret = regmap_update_bits(d->map,
+ ret = regmap_irq_update_bits(d,
reg + unmask_offset,
d->mask_buf[i],
d->mask_buf[i]);
} else
- ret = regmap_update_bits(map, reg,
+ ret = regmap_irq_update_bits(d, reg,
d->mask_buf[i], d->mask_buf[i]);
if (ret != 0) {
dev_err(map->dev, "Failed to set masks in 0x%x: %d\n",
(i * map->reg_stride * d->irq_reg_stride);
if (chip->wake_invert)
- ret = regmap_update_bits(map, reg,
+ ret = regmap_irq_update_bits(d, reg,
d->mask_buf_def[i],
0);
else
- ret = regmap_update_bits(map, reg,
+ ret = regmap_irq_update_bits(d, reg,
d->mask_buf_def[i],
d->wake_buf[i]);
if (ret != 0) {
reg = chip->type_base +
(i * map->reg_stride * d->type_reg_stride);
if (chip->type_invert)
- ret = regmap_update_bits(map, reg,
+ ret = regmap_irq_update_bits(d, reg,
d->type_buf_def[i], 0xFF);
else
- ret = regmap_update_bits(map, reg,
+ ret = regmap_irq_update_bits(d, reg,
d->type_buf_def[i], 0x0);
if (ret != 0) {
dev_err(map->dev,
--- /dev/null
+/*
+ * Register map access API - W1 (1-Wire) support
+ *
+ * Copyright (C) 2017 OAO Radioavionica
+ * Author: Alex A. Mihaylov <minimumlaw@rambler.ru>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation
+ */
+
+#include <linux/regmap.h>
+#include <linux/module.h>
+#include "../../w1/w1.h"
+
+#include "internal.h"
+
+#define W1_CMD_READ_DATA 0x69
+#define W1_CMD_WRITE_DATA 0x6C
+
+/*
+ * 1-Wire slaves registers with addess 8 bit and data 8 bit
+ */
+
+static int w1_reg_a8_v8_read(void *context, unsigned int reg, unsigned int *val)
+{
+ struct device *dev = context;
+ struct w1_slave *sl = container_of(dev, struct w1_slave, dev);
+ int ret = 0;
+
+ if (reg > 255)
+ return -EINVAL;
+
+ mutex_lock(&sl->master->bus_mutex);
+ if (!w1_reset_select_slave(sl)) {
+ w1_write_8(sl->master, W1_CMD_READ_DATA);
+ w1_write_8(sl->master, reg);
+ *val = w1_read_8(sl->master);
+ } else {
+ ret = -ENODEV;
+ }
+ mutex_unlock(&sl->master->bus_mutex);
+
+ return ret;
+}
+
+static int w1_reg_a8_v8_write(void *context, unsigned int reg, unsigned int val)
+{
+ struct device *dev = context;
+ struct w1_slave *sl = container_of(dev, struct w1_slave, dev);
+ int ret = 0;
+
+ if (reg > 255)
+ return -EINVAL;
+
+ mutex_lock(&sl->master->bus_mutex);
+ if (!w1_reset_select_slave(sl)) {
+ w1_write_8(sl->master, W1_CMD_WRITE_DATA);
+ w1_write_8(sl->master, reg);
+ w1_write_8(sl->master, val);
+ } else {
+ ret = -ENODEV;
+ }
+ mutex_unlock(&sl->master->bus_mutex);
+
+ return ret;
+}
+
+/*
+ * 1-Wire slaves registers with addess 8 bit and data 16 bit
+ */
+
+static int w1_reg_a8_v16_read(void *context, unsigned int reg,
+ unsigned int *val)
+{
+ struct device *dev = context;
+ struct w1_slave *sl = container_of(dev, struct w1_slave, dev);
+ int ret = 0;
+
+ if (reg > 255)
+ return -EINVAL;
+
+ mutex_lock(&sl->master->bus_mutex);
+ if (!w1_reset_select_slave(sl)) {
+ w1_write_8(sl->master, W1_CMD_READ_DATA);
+ w1_write_8(sl->master, reg);
+ *val = w1_read_8(sl->master);
+ *val |= w1_read_8(sl->master)<<8;
+ } else {
+ ret = -ENODEV;
+ }
+ mutex_unlock(&sl->master->bus_mutex);
+
+ return ret;
+}
+
+static int w1_reg_a8_v16_write(void *context, unsigned int reg,
+ unsigned int val)
+{
+ struct device *dev = context;
+ struct w1_slave *sl = container_of(dev, struct w1_slave, dev);
+ int ret = 0;
+
+ if (reg > 255)
+ return -EINVAL;
+
+ mutex_lock(&sl->master->bus_mutex);
+ if (!w1_reset_select_slave(sl)) {
+ w1_write_8(sl->master, W1_CMD_WRITE_DATA);
+ w1_write_8(sl->master, reg);
+ w1_write_8(sl->master, val & 0x00FF);
+ w1_write_8(sl->master, val>>8 & 0x00FF);
+ } else {
+ ret = -ENODEV;
+ }
+ mutex_unlock(&sl->master->bus_mutex);
+
+ return ret;
+}
+
+/*
+ * 1-Wire slaves registers with addess 16 bit and data 16 bit
+ */
+
+static int w1_reg_a16_v16_read(void *context, unsigned int reg,
+ unsigned int *val)
+{
+ struct device *dev = context;
+ struct w1_slave *sl = container_of(dev, struct w1_slave, dev);
+ int ret = 0;
+
+ if (reg > 65535)
+ return -EINVAL;
+
+ mutex_lock(&sl->master->bus_mutex);
+ if (!w1_reset_select_slave(sl)) {
+ w1_write_8(sl->master, W1_CMD_READ_DATA);
+ w1_write_8(sl->master, reg & 0x00FF);
+ w1_write_8(sl->master, reg>>8 & 0x00FF);
+ *val = w1_read_8(sl->master);
+ *val |= w1_read_8(sl->master)<<8;
+ } else {
+ ret = -ENODEV;
+ }
+ mutex_unlock(&sl->master->bus_mutex);
+
+ return ret;
+}
+
+static int w1_reg_a16_v16_write(void *context, unsigned int reg,
+ unsigned int val)
+{
+ struct device *dev = context;
+ struct w1_slave *sl = container_of(dev, struct w1_slave, dev);
+ int ret = 0;
+
+ if (reg > 65535)
+ return -EINVAL;
+
+ mutex_lock(&sl->master->bus_mutex);
+ if (!w1_reset_select_slave(sl)) {
+ w1_write_8(sl->master, W1_CMD_WRITE_DATA);
+ w1_write_8(sl->master, reg & 0x00FF);
+ w1_write_8(sl->master, reg>>8 & 0x00FF);
+ w1_write_8(sl->master, val & 0x00FF);
+ w1_write_8(sl->master, val>>8 & 0x00FF);
+ } else {
+ ret = -ENODEV;
+ }
+ mutex_unlock(&sl->master->bus_mutex);
+
+ return ret;
+}
+
+/*
+ * Various types of supported bus addressing
+ */
+
+static struct regmap_bus regmap_w1_bus_a8_v8 = {
+ .reg_read = w1_reg_a8_v8_read,
+ .reg_write = w1_reg_a8_v8_write,
+};
+
+static struct regmap_bus regmap_w1_bus_a8_v16 = {
+ .reg_read = w1_reg_a8_v16_read,
+ .reg_write = w1_reg_a8_v16_write,
+};
+
+static struct regmap_bus regmap_w1_bus_a16_v16 = {
+ .reg_read = w1_reg_a16_v16_read,
+ .reg_write = w1_reg_a16_v16_write,
+};
+
+static const struct regmap_bus *regmap_get_w1_bus(struct device *w1_dev,
+ const struct regmap_config *config)
+{
+ if (config->reg_bits == 8 && config->val_bits == 8)
+ return ®map_w1_bus_a8_v8;
+
+ if (config->reg_bits == 8 && config->val_bits == 16)
+ return ®map_w1_bus_a8_v16;
+
+ if (config->reg_bits == 16 && config->val_bits == 16)
+ return ®map_w1_bus_a16_v16;
+
+ return ERR_PTR(-ENOTSUPP);
+}
+
+struct regmap *__regmap_init_w1(struct device *w1_dev,
+ const struct regmap_config *config,
+ struct lock_class_key *lock_key,
+ const char *lock_name)
+{
+
+ const struct regmap_bus *bus = regmap_get_w1_bus(w1_dev, config);
+
+ if (IS_ERR(bus))
+ return ERR_CAST(bus);
+
+ return __regmap_init(w1_dev, bus, w1_dev, config,
+ lock_key, lock_name);
+
+ return NULL;
+}
+EXPORT_SYMBOL_GPL(__regmap_init_w1);
+
+struct regmap *__devm_regmap_init_w1(struct device *w1_dev,
+ const struct regmap_config *config,
+ struct lock_class_key *lock_key,
+ const char *lock_name)
+{
+
+ const struct regmap_bus *bus = regmap_get_w1_bus(w1_dev, config);
+
+ if (IS_ERR(bus))
+ return ERR_CAST(bus);
+
+ return __devm_regmap_init(w1_dev, bus, w1_dev, config,
+ lock_key, lock_name);
+
+ return NULL;
+}
+EXPORT_SYMBOL_GPL(__devm_regmap_init_w1);
+
+MODULE_LICENSE("GPL");
bool SuccessfulIO)
{
struct request *Request = Command->Request;
- int Error = SuccessfulIO ? 0 : -EIO;
+ blk_status_t Error = SuccessfulIO ? BLK_STS_OK : BLK_STS_IOERR;
pci_unmap_sg(Command->Controller->PCIDevice, Command->cmd_sglist,
Command->SegmentCount, Command->DmaDirection);
struct amiga_floppy_struct *floppy;
char *data;
unsigned long flags;
- int err;
+ blk_status_t err;
next_req:
rq = set_next_request();
next_segment:
/* Here someone could investigate to be more efficient */
- for (cnt = 0, err = 0; cnt < blk_rq_cur_sectors(rq); cnt++) {
+ for (cnt = 0, err = BLK_STS_OK; cnt < blk_rq_cur_sectors(rq); cnt++) {
#ifdef DEBUG
printk("fd: sector %ld + %d requested for %s\n",
blk_rq_pos(rq), cnt,
#endif
block = blk_rq_pos(rq) + cnt;
if ((int)block > floppy->blocks) {
- err = -EIO;
+ err = BLK_STS_IOERR;
break;
}
#endif
if (get_track(drive, track) == -1) {
- err = -EIO;
+ err = BLK_STS_IOERR;
break;
}
/* keep the drive spinning while writes are scheduled */
if (!fd_motor_on(drive)) {
- err = -EIO;
+ err = BLK_STS_IOERR;
break;
}
/*
d->aoemajor, d->aoeminor);
goto err_mempool;
}
+ blk_queue_bounce_limit(q, BLK_BOUNCE_HIGH);
spin_lock_irqsave(&d->lock, flags);
WARN_ON(!(d->flags & DEVFL_GD_NOW));
d->ip.rq = NULL;
do {
bio = rq->bio;
- bok = !fastfail && !bio->bi_error;
- } while (__blk_end_request(rq, bok ? 0 : -EIO, bio->bi_iter.bi_size));
+ bok = !fastfail && !bio->bi_status;
+ } while (__blk_end_request(rq, bok ? BLK_STS_OK : BLK_STS_IOERR, bio->bi_iter.bi_size));
/* cf. http://lkml.org/lkml/2006/10/31/28 */
if (!fastfail)
ahout->cmdstat, ahin->cmdstat,
d->aoemajor, d->aoeminor);
noskb: if (buf)
- buf->bio->bi_error = -EIO;
+ buf->bio->bi_status = BLK_STS_IOERR;
goto out;
}
"aoe: runt data size in read from",
(long) d->aoemajor, d->aoeminor,
skb->len, n);
- buf->bio->bi_error = -EIO;
+ buf->bio->bi_status = BLK_STS_IOERR;
break;
}
if (n > f->iter.bi_size) {
"aoe: too-large data size in read from",
(long) d->aoemajor, d->aoeminor,
n, f->iter.bi_size);
- buf->bio->bi_error = -EIO;
+ buf->bio->bi_status = BLK_STS_IOERR;
break;
}
bvcpy(skb, f->buf->bio, f->iter, n);
if (buf == NULL)
return;
buf->iter.bi_size = 0;
- buf->bio->bi_error = -EIO;
+ buf->bio->bi_status = BLK_STS_IOERR;
if (buf->nframesout == 0)
aoe_end_buf(d, buf);
}
if (rq == NULL)
return;
while ((bio = d->ip.nxbio)) {
- bio->bi_error = -EIO;
+ bio->bi_status = BLK_STS_IOERR;
d->ip.nxbio = bio->bi_next;
n = (unsigned long) rq->special;
rq->special = (void *) --n;
static DEFINE_TIMER(timeout_timer, fd_times_out, 0, 0);
static DEFINE_TIMER(fd_timer, check_change, 0, 0);
-static void fd_end_request_cur(int err)
+static void fd_end_request_cur(blk_status_t err)
{
if (!__blk_end_request_cur(fd_request, err))
fd_request = NULL;
fd_request->error_count++;
if (fd_request->error_count >= MAX_ERRORS) {
printk(KERN_ERR "fd%d: too many errors.\n", SelectedDrive );
- fd_end_request_cur(-EIO);
+ fd_end_request_cur(BLK_STS_IOERR);
}
else if (fd_request->error_count == RECALIBRATE_ERRORS) {
printk(KERN_WARNING "fd%d: recalibrating\n", SelectedDrive );
}
else {
/* all sectors finished */
- fd_end_request_cur(0);
+ fd_end_request_cur(BLK_STS_OK);
redo_fd_request();
return;
}
}
else {
/* all sectors finished */
- fd_end_request_cur(0);
+ fd_end_request_cur(BLK_STS_OK);
redo_fd_request();
}
return;
if (!UD.connected) {
/* drive not connected */
printk(KERN_ERR "Unknown Device: fd%d\n", drive );
- fd_end_request_cur(-EIO);
+ fd_end_request_cur(BLK_STS_IOERR);
goto repeat;
}
/* user supplied disk type */
if (--type >= NUM_DISK_MINORS) {
printk(KERN_WARNING "fd%d: invalid disk format", drive );
- fd_end_request_cur(-EIO);
+ fd_end_request_cur(BLK_STS_IOERR);
goto repeat;
}
if (minor2disktype[type].drive_types > DriveType) {
printk(KERN_WARNING "fd%d: unsupported disk format", drive );
- fd_end_request_cur(-EIO);
+ fd_end_request_cur(BLK_STS_IOERR);
goto repeat;
}
type = minor2disktype[type].index;
}
if (blk_rq_pos(fd_request) + 1 > UDT->blocks) {
- fd_end_request_cur(-EIO);
+ fd_end_request_cur(BLK_STS_IOERR);
goto repeat;
}
blk_queue_make_request(brd->brd_queue, brd_make_request);
blk_queue_max_hw_sectors(brd->brd_queue, 1024);
- blk_queue_bounce_limit(brd->brd_queue, BLK_BOUNCE_ANY);
/* This is so fdisk will align partitions on 4k, because of
* direct_access API needing 4k alignment, returning a PFN
/* set the residual count for pc requests */
if (blk_rq_is_passthrough(rq))
scsi_req(rq)->resid_len = c->err_info->ResidualCnt;
- blk_end_request_all(rq, scsi_req(rq)->result ? -EIO : 0);
+ blk_end_request_all(rq, scsi_req(rq)->result ?
+ BLK_STS_IOERR : BLK_STS_OK);
spin_lock_irqsave(&h->lock, flags);
cmd_free(h, c);
disk->queue->cmd_size = sizeof(struct scsi_request);
disk->queue->request_fn = do_cciss_request;
disk->queue->queue_lock = &h->lock;
+ queue_flag_set_unlocked(QUEUE_FLAG_SCSI_PASSTHROUGH, disk->queue);
if (blk_init_allocated_queue(disk->queue) < 0)
goto cleanup_queue;
else
submit_bio(bio);
wait_until_done_or_force_detached(device, bdev, &device->md_io.done);
- if (!bio->bi_error)
+ if (!bio->bi_status)
err = device->md_io.error;
out:
!bm_test_page_unchanged(b->bm_pages[idx]))
drbd_warn(device, "bitmap page idx %u changed during IO!\n", idx);
- if (bio->bi_error) {
+ if (bio->bi_status) {
/* ctx error will hold the completed-last non-zero error code,
* in case error codes differ. */
- ctx->error = bio->bi_error;
+ ctx->error = blk_status_to_errno(bio->bi_status);
bm_set_page_io_err(b->bm_pages[idx]);
/* Not identical to on disk version of it.
* Is BM_PAGE_IO_ERROR enough? */
if (__ratelimit(&drbd_ratelimit_state))
drbd_err(device, "IO ERROR %d on bitmap page idx %u\n",
- bio->bi_error, idx);
+ bio->bi_status, idx);
} else {
bm_clear_page_io_err(b->bm_pages[idx]);
dynamic_drbd_dbg(device, "bitmap page idx %u completed\n", idx);
/* to allocate from that set */
extern struct bio *bio_alloc_drbd(gfp_t gfp_mask);
+/* And a bio_set for cloning */
+extern struct bio_set *drbd_io_bio_set;
+
extern struct mutex resources_mutex;
extern int conn_lowest_minor(struct drbd_connection *connection);
__release(local);
if (!bio->bi_bdev) {
drbd_err(device, "drbd_generic_make_request: bio->bi_bdev == NULL\n");
- bio->bi_error = -ENODEV;
+ bio->bi_status = BLK_STS_IOERR;
bio_endio(bio);
return;
}
mempool_t *drbd_ee_mempool;
mempool_t *drbd_md_io_page_pool;
struct bio_set *drbd_md_io_bio_set;
+struct bio_set *drbd_io_bio_set;
/* I do not use a standard mempool, because:
1) I want to hand out the pre-allocated objects first.
/* D_ASSERT(device, atomic_read(&drbd_pp_vacant)==0); */
+ if (drbd_io_bio_set)
+ bioset_free(drbd_io_bio_set);
if (drbd_md_io_bio_set)
bioset_free(drbd_md_io_bio_set);
if (drbd_md_io_page_pool)
if (drbd_al_ext_cache)
kmem_cache_destroy(drbd_al_ext_cache);
+ drbd_io_bio_set = NULL;
drbd_md_io_bio_set = NULL;
drbd_md_io_page_pool = NULL;
drbd_ee_mempool = NULL;
drbd_pp_pool = NULL;
drbd_md_io_page_pool = NULL;
drbd_md_io_bio_set = NULL;
+ drbd_io_bio_set = NULL;
/* caches */
drbd_request_cache = kmem_cache_create(
goto Enomem;
/* mempools */
- drbd_md_io_bio_set = bioset_create(DRBD_MIN_POOL_PAGES, 0);
+ drbd_io_bio_set = bioset_create(BIO_POOL_SIZE, 0, BIOSET_NEED_RESCUER);
+ if (drbd_io_bio_set == NULL)
+ goto Enomem;
+
+ drbd_md_io_bio_set = bioset_create(DRBD_MIN_POOL_PAGES, 0,
+ BIOSET_NEED_BVECS |
+ BIOSET_NEED_RESCUER);
if (drbd_md_io_bio_set == NULL)
goto Enomem;
/* Setting the max_hw_sectors to an odd value of 8kibyte here
This triggers a max_bio_size message upon first attach or connect */
blk_queue_max_hw_sectors(q, DRBD_MAX_BIO_SIZE_SAFE >> 8);
- blk_queue_bounce_limit(q, BLK_BOUNCE_ANY);
q->queue_lock = &resource->req_lock;
device->md_io.page = alloc_page(GFP_KERNEL);
static enum drbd_ret_code
check_net_options(struct drbd_connection *connection, struct net_conf *new_net_conf)
{
- static enum drbd_ret_code rv;
+ enum drbd_ret_code rv;
struct drbd_peer_device *peer_device;
int i;
struct drbd_device *device = octx->device;
struct issue_flush_context *ctx = octx->ctx;
- if (bio->bi_error) {
- ctx->error = bio->bi_error;
- drbd_info(device, "local disk FLUSH FAILED with status %d\n", bio->bi_error);
+ if (bio->bi_status) {
+ ctx->error = blk_status_to_errno(bio->bi_status);
+ drbd_info(device, "local disk FLUSH FAILED with status %d\n", bio->bi_status);
}
kfree(octx);
bio_put(bio);
void complete_master_bio(struct drbd_device *device,
struct bio_and_error *m)
{
- m->bio->bi_error = m->error;
+ m->bio->bi_status = errno_to_blk_status(m->error);
bio_endio(m->bio);
dec_ap_bio(device);
}
if (blkdev_issue_zeroout(bdev, req->i.sector, req->i.size >> 9,
GFP_NOIO, 0))
- req->private_bio->bi_error = -EIO;
+ req->private_bio->bi_status = BLK_STS_IOERR;
bio_endio(req->private_bio);
}
/* only pass the error to the upper layers.
* if user cannot handle io errors, that's not our business. */
drbd_err(device, "could not kmalloc() req\n");
- bio->bi_error = -ENOMEM;
+ bio->bi_status = BLK_STS_RESOURCE;
bio_endio(bio);
return ERR_PTR(-ENOMEM);
}
struct drbd_device *device = (struct drbd_device *) q->queuedata;
unsigned long start_jif;
- blk_queue_split(q, &bio, q->bio_split);
+ blk_queue_split(q, &bio);
start_jif = jiffies;
static inline void drbd_req_make_private_bio(struct drbd_request *req, struct bio *bio_src)
{
struct bio *bio;
- bio = bio_clone(bio_src, GFP_NOIO); /* XXX cannot fail?? */
+ bio = bio_clone_fast(bio_src, GFP_NOIO, drbd_io_bio_set);
req->private_bio = bio;
struct drbd_device *device;
device = bio->bi_private;
- device->md_io.error = bio->bi_error;
+ device->md_io.error = blk_status_to_errno(bio->bi_status);
/* We grabbed an extra reference in _drbd_md_sync_page_io() to be able
* to timeout on the lower level device, and eventually detach from it.
bool is_discard = bio_op(bio) == REQ_OP_WRITE_ZEROES ||
bio_op(bio) == REQ_OP_DISCARD;
- if (bio->bi_error && __ratelimit(&drbd_ratelimit_state))
+ if (bio->bi_status && __ratelimit(&drbd_ratelimit_state))
drbd_warn(device, "%s: error=%d s=%llus\n",
is_write ? (is_discard ? "discard" : "write")
- : "read", bio->bi_error,
+ : "read", bio->bi_status,
(unsigned long long)peer_req->i.sector);
- if (bio->bi_error)
+ if (bio->bi_status)
set_bit(__EE_WAS_ERROR, &peer_req->flags);
bio_put(bio); /* no need for the bio anymore */
if (__ratelimit(&drbd_ratelimit_state))
drbd_emerg(device, "delayed completion of aborted local request; disk-timeout may be too aggressive\n");
- if (!bio->bi_error)
+ if (!bio->bi_status)
drbd_panic_after_delayed_completion_of_aborted_request(device);
}
/* to avoid recursion in __req_mod */
- if (unlikely(bio->bi_error)) {
+ if (unlikely(bio->bi_status)) {
switch (bio_op(bio)) {
case REQ_OP_WRITE_ZEROES:
case REQ_OP_DISCARD:
- if (bio->bi_error == -EOPNOTSUPP)
+ if (bio->bi_status == BLK_STS_NOTSUPP)
what = DISCARD_COMPLETED_NOTSUPP;
else
what = DISCARD_COMPLETED_WITH_ERROR;
}
bio_put(req->private_bio);
- req->private_bio = ERR_PTR(bio->bi_error);
+ req->private_bio = ERR_PTR(blk_status_to_errno(bio->bi_status));
/* not req_mod(), we need irqsave here! */
spin_lock_irqsave(&device->resource->req_lock, flags);
* =============================
*/
-static void floppy_end_request(struct request *req, int error)
+static void floppy_end_request(struct request *req, blk_status_t error)
{
unsigned int nr_sectors = current_count_sectors;
unsigned int drive = (unsigned long)req->rq_disk->private_data;
DRWE->last_error_generation = DRS->generation;
}
spin_lock_irqsave(q->queue_lock, flags);
- floppy_end_request(req, -EIO);
+ floppy_end_request(req, BLK_STS_IOERR);
spin_unlock_irqrestore(q->queue_lock, flags);
}
}
struct rb0_cbdata *cbdata = (struct rb0_cbdata *)bio->bi_private;
int drive = cbdata->drive;
- if (bio->bi_error) {
+ if (bio->bi_status) {
pr_info("floppy: error %d while reading block 0\n",
- bio->bi_error);
+ bio->bi_status);
set_bit(FD_OPEN_SHOULD_FAIL_BIT, &UDRS->flags);
}
complete(&cbdata->complete);
goto out_put_disk;
}
+ blk_queue_bounce_limit(disks[drive]->queue, BLK_BOUNCE_HIGH);
blk_queue_max_hw_sectors(disks[drive]->queue, 64);
disks[drive]->major = FLOPPY_MAJOR;
disks[drive]->first_minor = TOMINOR(drive);
}
static int
-figure_loop_size(struct loop_device *lo, loff_t offset, loff_t sizelimit)
+figure_loop_size(struct loop_device *lo, loff_t offset, loff_t sizelimit,
+ loff_t logical_blocksize)
{
loff_t size = get_size(offset, sizelimit, lo->lo_backing_file);
sector_t x = (sector_t)size;
lo->lo_offset = offset;
if (lo->lo_sizelimit != sizelimit)
lo->lo_sizelimit = sizelimit;
+ if (lo->lo_flags & LO_FLAGS_BLOCKSIZE) {
+ lo->lo_logical_blocksize = logical_blocksize;
+ blk_queue_physical_block_size(lo->lo_queue, lo->lo_blocksize);
+ blk_queue_logical_block_size(lo->lo_queue,
+ lo->lo_logical_blocksize);
+ }
set_capacity(lo->lo_disk, x);
bd_set_size(bdev, (loff_t)get_capacity(bdev->bd_disk) << 9);
/* let user-space know about the new size */
zero_fill_bio(bio);
}
- blk_mq_end_request(rq, cmd->ret < 0 ? -EIO : 0);
+ blk_mq_end_request(rq, cmd->ret < 0 ? BLK_STS_IOERR : BLK_STS_OK);
}
static void lo_rw_aio_complete(struct kiocb *iocb, long ret, long ret2)
struct file *file = lo->lo_backing_file;
struct inode *inode = file->f_mapping->host;
struct request_queue *q = lo->lo_queue;
+ int lo_bits = 9;
/*
* We use punch hole to reclaim the free space used by the
q->limits.discard_granularity = inode->i_sb->s_blocksize;
q->limits.discard_alignment = 0;
- blk_queue_max_discard_sectors(q, UINT_MAX >> 9);
- blk_queue_max_write_zeroes_sectors(q, UINT_MAX >> 9);
+ if (lo->lo_flags & LO_FLAGS_BLOCKSIZE)
+ lo_bits = blksize_bits(lo->lo_logical_blocksize);
+
+ blk_queue_max_discard_sectors(q, UINT_MAX >> lo_bits);
+ blk_queue_max_write_zeroes_sectors(q, UINT_MAX >> lo_bits);
queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, q);
}
kthread_stop(lo->worker_task);
}
+static int loop_kthread_worker_fn(void *worker_ptr)
+{
+ current->flags |= PF_LESS_THROTTLE;
+ return kthread_worker_fn(worker_ptr);
+}
+
static int loop_prepare_queue(struct loop_device *lo)
{
kthread_init_worker(&lo->worker);
- lo->worker_task = kthread_run(kthread_worker_fn,
+ lo->worker_task = kthread_run(loop_kthread_worker_fn,
&lo->worker, "loop%d", lo->lo_number);
if (IS_ERR(lo->worker_task))
return -ENOMEM;
lo->use_dio = false;
lo->lo_blocksize = lo_blocksize;
+ lo->lo_logical_blocksize = 512;
lo->lo_device = bdev;
lo->lo_flags = lo_flags;
lo->lo_backing_file = file;
int err;
struct loop_func_table *xfer;
kuid_t uid = current_uid();
+ int lo_flags = lo->lo_flags;
if (lo->lo_encrypt_key_size &&
!uid_eq(lo->lo_key_owner, uid) &&
if (err)
goto exit;
+ if (info->lo_flags & LO_FLAGS_BLOCKSIZE) {
+ if (!(lo->lo_flags & LO_FLAGS_BLOCKSIZE))
+ lo->lo_logical_blocksize = 512;
+ lo->lo_flags |= LO_FLAGS_BLOCKSIZE;
+ if (LO_INFO_BLOCKSIZE(info) != 512 &&
+ LO_INFO_BLOCKSIZE(info) != 1024 &&
+ LO_INFO_BLOCKSIZE(info) != 2048 &&
+ LO_INFO_BLOCKSIZE(info) != 4096)
+ return -EINVAL;
+ if (LO_INFO_BLOCKSIZE(info) > lo->lo_blocksize)
+ return -EINVAL;
+ }
+
if (lo->lo_offset != info->lo_offset ||
- lo->lo_sizelimit != info->lo_sizelimit)
- if (figure_loop_size(lo, info->lo_offset, info->lo_sizelimit)) {
+ lo->lo_sizelimit != info->lo_sizelimit ||
+ lo->lo_flags != lo_flags ||
+ ((lo->lo_flags & LO_FLAGS_BLOCKSIZE) &&
+ lo->lo_logical_blocksize != LO_INFO_BLOCKSIZE(info))) {
+ if (figure_loop_size(lo, info->lo_offset, info->lo_sizelimit,
+ LO_INFO_BLOCKSIZE(info))) {
err = -EFBIG;
goto exit;
}
+ }
loop_config_discard(lo);
return err;
}
-static int loop_set_capacity(struct loop_device *lo, struct block_device *bdev)
+static int loop_set_capacity(struct loop_device *lo)
{
if (unlikely(lo->lo_state != Lo_bound))
return -ENXIO;
- return figure_loop_size(lo, lo->lo_offset, lo->lo_sizelimit);
+ return figure_loop_size(lo, lo->lo_offset, lo->lo_sizelimit,
+ lo->lo_logical_blocksize);
}
static int loop_set_dio(struct loop_device *lo, unsigned long arg)
case LOOP_SET_CAPACITY:
err = -EPERM;
if ((mode & FMODE_WRITE) || capable(CAP_SYS_ADMIN))
- err = loop_set_capacity(lo, bdev);
+ err = loop_set_capacity(lo);
break;
case LOOP_SET_DIRECT_IO:
err = -EPERM;
EXPORT_SYMBOL(loop_register_transfer);
EXPORT_SYMBOL(loop_unregister_transfer);
-static int loop_queue_rq(struct blk_mq_hw_ctx *hctx,
+static blk_status_t loop_queue_rq(struct blk_mq_hw_ctx *hctx,
const struct blk_mq_queue_data *bd)
{
struct loop_cmd *cmd = blk_mq_rq_to_pdu(bd->rq);
blk_mq_start_request(bd->rq);
if (lo->lo_state != Lo_bound)
- return BLK_MQ_RQ_QUEUE_ERROR;
+ return BLK_STS_IOERR;
switch (req_op(cmd->rq)) {
case REQ_OP_FLUSH:
kthread_queue_work(&lo->worker, &cmd->work);
- return BLK_MQ_RQ_QUEUE_OK;
+ return BLK_STS_OK;
}
static void loop_handle_cmd(struct loop_cmd *cmd)
struct file * lo_backing_file;
struct block_device *lo_device;
unsigned lo_blocksize;
+ unsigned lo_logical_blocksize;
void *key_data;
gfp_t old_gfp_mask;
static int mtip_get_smart_attr(struct mtip_port *port, unsigned int id,
struct smart_attr *attrib);
-static void mtip_complete_command(struct mtip_cmd *cmd, int status)
+static void mtip_complete_command(struct mtip_cmd *cmd, blk_status_t status)
{
struct request *req = blk_mq_rq_from_pdu(cmd);
if (test_bit(MTIP_PF_IC_ACTIVE_BIT, &port->flags)) {
cmd = mtip_cmd_from_tag(dd, MTIP_TAG_INTERNAL);
dbg_printk(MTIP_DRV_NAME " TFE for the internal command\n");
- mtip_complete_command(cmd, -EIO);
+ mtip_complete_command(cmd, BLK_STS_IOERR);
return;
}
tag,
fail_reason != NULL ?
fail_reason : "unknown");
- mtip_complete_command(cmd, -ENODATA);
+ mtip_complete_command(cmd, BLK_STS_MEDIUM);
continue;
}
}
dev_warn(&port->dd->pdev->dev,
"retiring tag %d\n", tag);
- mtip_complete_command(cmd, -EIO);
+ mtip_complete_command(cmd, BLK_STS_IOERR);
}
}
print_tags(dd, "reissued (TFE)", tagaccum, cmd_cnt);
/* insert request and run queue */
blk_execute_rq(rq->q, NULL, rq, true);
- rv = int_cmd->status;
- if (rv < 0) {
- if (rv == -ERESTARTSYS) { /* interrupted */
- dev_err(&dd->pdev->dev,
- "Internal command [%02X] was interrupted after %u ms\n",
- fis->command,
- jiffies_to_msecs(jiffies - start));
- rv = -EINTR;
- goto exec_ic_exit;
- } else if (rv == 0) /* timeout */
- dev_err(&dd->pdev->dev,
- "Internal command did not complete [%02X] within timeout of %lu ms\n",
- fis->command, timeout);
- else
- dev_err(&dd->pdev->dev,
- "Internal command [%02X] wait returned code [%d] after %lu ms - unhandled\n",
- fis->command, rv, timeout);
+ if (int_cmd->status) {
+ dev_err(&dd->pdev->dev, "Internal command [%02X] failed %d\n",
+ fis->command, int_cmd->status);
+ rv = -EIO;
if (mtip_check_surprise_removal(dd->pdev) ||
test_bit(MTIP_DDF_REMOVE_PENDING_BIT,
dbg_printk(MTIP_DRV_NAME " Aborting request, tag = %d\n", req->tag);
clear_bit(req->tag, dd->port->cmds_to_issue);
- cmd->status = -EIO;
+ cmd->status = BLK_STS_IOERR;
mtip_softirq_done_fn(req);
}
int err;
err = mtip_send_trim(dd, blk_rq_pos(rq), blk_rq_sectors(rq));
- blk_mq_end_request(rq, err);
+ blk_mq_end_request(rq, err ? BLK_STS_IOERR : BLK_STS_OK);
return 0;
}
return false;
}
-static int mtip_issue_reserved_cmd(struct blk_mq_hw_ctx *hctx,
- struct request *rq)
+static blk_status_t mtip_issue_reserved_cmd(struct blk_mq_hw_ctx *hctx,
+ struct request *rq)
{
struct driver_data *dd = hctx->queue->queuedata;
struct mtip_int_cmd *icmd = rq->special;
struct mtip_cmd_sg *command_sg;
if (mtip_commands_active(dd->port))
- return BLK_MQ_RQ_QUEUE_BUSY;
+ return BLK_STS_RESOURCE;
/* Populate the SG list */
cmd->command_header->opts =
blk_mq_start_request(rq);
mtip_issue_non_ncq_command(dd->port, rq->tag);
- return BLK_MQ_RQ_QUEUE_OK;
+ return 0;
}
-static int mtip_queue_rq(struct blk_mq_hw_ctx *hctx,
+static blk_status_t mtip_queue_rq(struct blk_mq_hw_ctx *hctx,
const struct blk_mq_queue_data *bd)
{
struct request *rq = bd->rq;
return mtip_issue_reserved_cmd(hctx, rq);
if (unlikely(mtip_check_unal_depth(hctx, rq)))
- return BLK_MQ_RQ_QUEUE_BUSY;
+ return BLK_STS_RESOURCE;
blk_mq_start_request(rq);
ret = mtip_submit_request(hctx, rq);
if (likely(!ret))
- return BLK_MQ_RQ_QUEUE_OK;
-
- return BLK_MQ_RQ_QUEUE_ERROR;
+ return BLK_STS_OK;
+ return BLK_STS_IOERR;
}
static void mtip_free_cmd(struct blk_mq_tag_set *set, struct request *rq,
if (reserved) {
struct mtip_cmd *cmd = blk_mq_rq_to_pdu(req);
- cmd->status = -ETIME;
+ cmd->status = BLK_STS_TIMEOUT;
return BLK_EH_HANDLED;
}
{
struct mtip_cmd *cmd = blk_mq_rq_to_pdu(rq);
- cmd->status = -ENODEV;
+ cmd->status = BLK_STS_IOERR;
blk_mq_complete_request(rq);
}
int retries; /* The number of retries left for this command. */
int direction; /* Data transfer direction */
- int status;
+ blk_status_t status;
};
/* Structure used to describe a port. */
int index;
int cookie;
struct completion send_complete;
- int status;
+ blk_status_t status;
};
#if IS_ENABLED(CONFIG_DEBUG_FS)
struct nbd_config *config;
if (!refcount_inc_not_zero(&nbd->config_refs)) {
- cmd->status = -EIO;
+ cmd->status = BLK_STS_TIMEOUT;
return BLK_EH_HANDLED;
}
"Connection timed out\n");
}
set_bit(NBD_TIMEDOUT, &config->runtime_flags);
- cmd->status = -EIO;
+ cmd->status = BLK_STS_IOERR;
sock_shutdown(nbd);
nbd_config_put(nbd);
unsigned long size = blk_rq_bytes(req);
struct bio *bio;
u32 type;
+ u32 nbd_cmd_flags = 0;
u32 tag = blk_mq_unique_tag(req);
int sent = nsock->sent, skip = 0;
return -EIO;
}
+ if (req->cmd_flags & REQ_FUA)
+ nbd_cmd_flags |= NBD_CMD_FLAG_FUA;
+
/* We did a partial send previously, and we at least sent the whole
* request struct, so just go and send the rest of the pages in the
* request.
}
cmd->index = index;
cmd->cookie = nsock->cookie;
- request.type = htonl(type);
+ request.type = htonl(type | nbd_cmd_flags);
if (type != NBD_CMD_FLUSH) {
request.from = cpu_to_be64((u64)blk_rq_pos(req) << 9);
request.len = htonl(size);
nsock->pending = req;
nsock->sent = sent;
}
- return BLK_MQ_RQ_QUEUE_BUSY;
+ return BLK_STS_RESOURCE;
}
dev_err_ratelimited(disk_to_dev(nbd->disk),
"Send control failed (result %d)\n", result);
*/
nsock->pending = req;
nsock->sent = sent;
- return BLK_MQ_RQ_QUEUE_BUSY;
+ return BLK_STS_RESOURCE;
}
dev_err(disk_to_dev(nbd->disk),
"Send data failed (result %d)\n",
if (ntohl(reply.error)) {
dev_err(disk_to_dev(nbd->disk), "Other side returned error (%d)\n",
ntohl(reply.error));
- cmd->status = -EIO;
+ cmd->status = BLK_STS_IOERR;
return cmd;
}
*/
if (nbd_disconnected(config) ||
config->num_connections <= 1) {
- cmd->status = -EIO;
+ cmd->status = BLK_STS_IOERR;
return cmd;
}
return ERR_PTR(-EIO);
if (!blk_mq_request_started(req))
return;
cmd = blk_mq_rq_to_pdu(req);
- cmd->status = -EIO;
+ cmd->status = BLK_STS_IOERR;
blk_mq_complete_request(req);
}
nbd_config_put(nbd);
return -EINVAL;
}
- cmd->status = 0;
+ cmd->status = BLK_STS_OK;
again:
nsock = config->socks[index];
mutex_lock(&nsock->tx_lock);
return ret;
}
-static int nbd_queue_rq(struct blk_mq_hw_ctx *hctx,
+static blk_status_t nbd_queue_rq(struct blk_mq_hw_ctx *hctx,
const struct blk_mq_queue_data *bd)
{
struct nbd_cmd *cmd = blk_mq_rq_to_pdu(bd->rq);
* appropriate.
*/
ret = nbd_handle_cmd(cmd, hctx->queue_num);
- if (ret < 0)
- ret = BLK_MQ_RQ_QUEUE_ERROR;
- if (!ret)
- ret = BLK_MQ_RQ_QUEUE_OK;
complete(&cmd->send_complete);
- return ret;
+ return ret < 0 ? BLK_STS_IOERR : BLK_STS_OK;
}
static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
continue;
}
sk_set_memalloc(sock->sk);
+ sock->sk->sk_sndtimeo = nbd->tag_set.timeout;
atomic_inc(&config->recv_threads);
refcount_inc(&nbd->config_refs);
old = nsock->sock;
set_disk_ro(nbd->disk, false);
if (config->flags & NBD_FLAG_SEND_TRIM)
queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, nbd->disk->queue);
- if (config->flags & NBD_FLAG_SEND_FLUSH)
- blk_queue_write_cache(nbd->disk->queue, true, false);
+ if (config->flags & NBD_FLAG_SEND_FLUSH) {
+ if (config->flags & NBD_FLAG_SEND_FUA)
+ blk_queue_write_cache(nbd->disk->queue, true, true);
+ else
+ blk_queue_write_cache(nbd->disk->queue, true, false);
+ }
else
blk_queue_write_cache(nbd->disk->queue, false, false);
}
return -ENOMEM;
}
sk_set_memalloc(config->socks[i]->sock->sk);
+ config->socks[i]->sock->sk->sk_sndtimeo = nbd->tag_set.timeout;
atomic_inc(&config->recv_threads);
refcount_inc(&nbd->config_refs);
INIT_WORK(&args->work, recv_work);
seq_puts(s, "NBD_FLAG_READ_ONLY\n");
if (flags & NBD_FLAG_SEND_FLUSH)
seq_puts(s, "NBD_FLAG_SEND_FLUSH\n");
+ if (flags & NBD_FLAG_SEND_FUA)
+ seq_puts(s, "NBD_FLAG_SEND_FUA\n");
if (flags & NBD_FLAG_SEND_TRIM)
seq_puts(s, "NBD_FLAG_SEND_TRIM\n");
struct request_queue *q;
struct gendisk *disk;
struct nvm_dev *ndev;
- struct blk_mq_tag_set tag_set;
+ struct blk_mq_tag_set *tag_set;
+ struct blk_mq_tag_set __tag_set;
struct hrtimer timer;
unsigned int queue_depth;
spinlock_t lock;
static int null_major;
static int nullb_indexes;
static struct kmem_cache *ppa_cache;
+static struct blk_mq_tag_set tag_set;
enum {
NULL_IRQ_NONE = 0,
module_param(bs, int, S_IRUGO);
MODULE_PARM_DESC(bs, "Block size (in bytes)");
-static int nr_devices = 2;
+static int nr_devices = 1;
module_param(nr_devices, int, S_IRUGO);
MODULE_PARM_DESC(nr_devices, "Number of devices to register");
module_param(blocking, bool, S_IRUGO);
MODULE_PARM_DESC(blocking, "Register as a blocking blk-mq driver device");
+static bool shared_tags;
+module_param(shared_tags, bool, S_IRUGO);
+MODULE_PARM_DESC(shared_tags, "Share tag set between devices for blk-mq");
+
static int irqmode = NULL_IRQ_SOFTIRQ;
static int null_set_irqmode(const char *str, const struct kernel_param *kp)
switch (queue_mode) {
case NULL_Q_MQ:
- blk_mq_end_request(cmd->rq, 0);
+ blk_mq_end_request(cmd->rq, BLK_STS_OK);
return;
case NULL_Q_RQ:
INIT_LIST_HEAD(&cmd->rq->queuelist);
- blk_end_request_all(cmd->rq, 0);
+ blk_end_request_all(cmd->rq, BLK_STS_OK);
break;
case NULL_Q_BIO:
bio_endio(cmd->bio);
}
}
-static int null_queue_rq(struct blk_mq_hw_ctx *hctx,
+static blk_status_t null_queue_rq(struct blk_mq_hw_ctx *hctx,
const struct blk_mq_queue_data *bd)
{
struct nullb_cmd *cmd = blk_mq_rq_to_pdu(bd->rq);
blk_mq_start_request(bd->rq);
null_handle_cmd(cmd);
- return BLK_MQ_RQ_QUEUE_OK;
-}
-
-static void null_init_queue(struct nullb *nullb, struct nullb_queue *nq)
-{
- BUG_ON(!nullb);
- BUG_ON(!nq);
-
- init_waitqueue_head(&nq->wait);
- nq->queue_depth = nullb->queue_depth;
-}
-
-static int null_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
- unsigned int index)
-{
- struct nullb *nullb = data;
- struct nullb_queue *nq = &nullb->queues[index];
-
- hctx->driver_data = nq;
- null_init_queue(nullb, nq);
- nullb->nr_queues++;
-
- return 0;
+ return BLK_STS_OK;
}
static const struct blk_mq_ops null_mq_ops = {
.queue_rq = null_queue_rq,
- .init_hctx = null_init_hctx,
.complete = null_softirq_done_fn,
};
#ifdef CONFIG_NVM
-static void null_lnvm_end_io(struct request *rq, int error)
+static void null_lnvm_end_io(struct request *rq, blk_status_t status)
{
struct nvm_rq *rqd = rq->end_io_data;
- rqd->error = error;
+ /* XXX: lighnvm core seems to expect NVM_RSP_* values here.. */
+ rqd->error = status ? -EIO : 0;
nvm_end_io(rqd);
blk_put_request(rq);
else
del_gendisk(nullb->disk);
blk_cleanup_queue(nullb->q);
- if (queue_mode == NULL_Q_MQ)
- blk_mq_free_tag_set(&nullb->tag_set);
+ if (queue_mode == NULL_Q_MQ && nullb->tag_set == &nullb->__tag_set)
+ blk_mq_free_tag_set(nullb->tag_set);
if (!use_lightnvm)
put_disk(nullb->disk);
cleanup_queues(nullb);
.release = null_release,
};
+static void null_init_queue(struct nullb *nullb, struct nullb_queue *nq)
+{
+ BUG_ON(!nullb);
+ BUG_ON(!nq);
+
+ init_waitqueue_head(&nq->wait);
+ nq->queue_depth = nullb->queue_depth;
+}
+
+static void null_init_queues(struct nullb *nullb)
+{
+ struct request_queue *q = nullb->q;
+ struct blk_mq_hw_ctx *hctx;
+ struct nullb_queue *nq;
+ int i;
+
+ queue_for_each_hw_ctx(q, hctx, i) {
+ if (!hctx->nr_ctx || !hctx->tags)
+ continue;
+ nq = &nullb->queues[i];
+ hctx->driver_data = nq;
+ null_init_queue(nullb, nq);
+ nullb->nr_queues++;
+ }
+}
+
static int setup_commands(struct nullb_queue *nq)
{
struct nullb_cmd *cmd;
return 0;
}
+static int null_init_tag_set(struct blk_mq_tag_set *set)
+{
+ set->ops = &null_mq_ops;
+ set->nr_hw_queues = submit_queues;
+ set->queue_depth = hw_queue_depth;
+ set->numa_node = home_node;
+ set->cmd_size = sizeof(struct nullb_cmd);
+ set->flags = BLK_MQ_F_SHOULD_MERGE;
+ set->driver_data = NULL;
+
+ if (blocking)
+ set->flags |= BLK_MQ_F_BLOCKING;
+
+ return blk_mq_alloc_tag_set(set);
+}
+
static int null_add_dev(void)
{
struct nullb *nullb;
goto out_free_nullb;
if (queue_mode == NULL_Q_MQ) {
- nullb->tag_set.ops = &null_mq_ops;
- nullb->tag_set.nr_hw_queues = submit_queues;
- nullb->tag_set.queue_depth = hw_queue_depth;
- nullb->tag_set.numa_node = home_node;
- nullb->tag_set.cmd_size = sizeof(struct nullb_cmd);
- nullb->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
- nullb->tag_set.driver_data = nullb;
-
- if (blocking)
- nullb->tag_set.flags |= BLK_MQ_F_BLOCKING;
-
- rv = blk_mq_alloc_tag_set(&nullb->tag_set);
+ if (shared_tags) {
+ nullb->tag_set = &tag_set;
+ rv = 0;
+ } else {
+ nullb->tag_set = &nullb->__tag_set;
+ rv = null_init_tag_set(nullb->tag_set);
+ }
+
if (rv)
goto out_cleanup_queues;
- nullb->q = blk_mq_init_queue(&nullb->tag_set);
+ nullb->q = blk_mq_init_queue(nullb->tag_set);
if (IS_ERR(nullb->q)) {
rv = -ENOMEM;
goto out_cleanup_tags;
}
+ null_init_queues(nullb);
} else if (queue_mode == NULL_Q_BIO) {
nullb->q = blk_alloc_queue_node(GFP_KERNEL, home_node);
if (!nullb->q) {
out_cleanup_blk_queue:
blk_cleanup_queue(nullb->q);
out_cleanup_tags:
- if (queue_mode == NULL_Q_MQ)
- blk_mq_free_tag_set(&nullb->tag_set);
+ if (queue_mode == NULL_Q_MQ && nullb->tag_set == &nullb->__tag_set)
+ blk_mq_free_tag_set(nullb->tag_set);
out_cleanup_queues:
cleanup_queues(nullb);
out_free_nullb:
queue_mode = NULL_Q_MQ;
}
+ if (queue_mode == NULL_Q_MQ && shared_tags)
+ null_init_tag_set(&tag_set);
+
if (queue_mode == NULL_Q_MQ && use_per_node_hctx) {
if (submit_queues < nr_online_nodes) {
pr_warn("null_blk: submit_queues param is set to %u.",
}
mutex_unlock(&lock);
+ if (queue_mode == NULL_Q_MQ && shared_tags)
+ blk_mq_free_tag_set(&tag_set);
+
kmem_cache_destroy(ppa_cache);
}
put_disk(disk);
continue;
}
+ blk_queue_bounce_limit(disk->queue, BLK_BOUNCE_HIGH);
cd->disk = disk;
cd->pi = &cd->pia;
cd->present = 0;
ps_set_intr(do_pcd_read, NULL, 0, nice);
return;
} else {
- __blk_end_request_all(pcd_req, -EIO);
+ __blk_end_request_all(pcd_req, BLK_STS_IOERR);
pcd_req = NULL;
}
}
pcd_request();
}
-static inline void next_request(int err)
+static inline void next_request(blk_status_t err)
{
unsigned long saved_flags;
if (pcd_command(pcd_current, rd_cmd, 2048, "read block")) {
pcd_bufblk = -1;
- next_request(-EIO);
+ next_request(BLK_STS_IOERR);
return;
}
return;
}
pcd_bufblk = -1;
- next_request(-EIO);
+ next_request(BLK_STS_IOERR);
return;
}
phase = NULL;
spin_lock_irqsave(&pd_lock, saved_flags);
if (!__blk_end_request_cur(pd_req,
- res == Ok ? 0 : -EIO)) {
+ res == Ok ? 0 : BLK_STS_IOERR)) {
if (!set_next_request())
stop = 1;
}
return;
}
blk_queue_max_hw_sectors(p->queue, cluster);
+ blk_queue_bounce_limit(p->queue, BLK_BOUNCE_HIGH);
if (disk->drive == -1) {
for (disk->drive = 0; disk->drive <= 1; disk->drive++)
return;
}
blk_queue_max_segments(disk->queue, cluster);
+ blk_queue_bounce_limit(disk->queue, BLK_BOUNCE_HIGH);
pf->disk = disk;
pf->pi = &pf->pia;
pf->media_status = PF_NM;
return pf_req != NULL;
}
-static void pf_end_request(int err)
+static void pf_end_request(blk_status_t err)
{
if (pf_req && !__blk_end_request_cur(pf_req, err))
pf_req = NULL;
pf_count = blk_rq_cur_sectors(pf_req);
if (pf_block + pf_count > get_capacity(pf_req->rq_disk)) {
- pf_end_request(-EIO);
+ pf_end_request(BLK_STS_IOERR);
goto repeat;
}
pi_do_claimed(pf_current->pi, do_pf_write);
else {
pf_busy = 0;
- pf_end_request(-EIO);
+ pf_end_request(BLK_STS_IOERR);
goto repeat;
}
}
return 0;
}
-static inline void next_request(int err)
+static inline void next_request(blk_status_t err)
{
unsigned long saved_flags;
pi_do_claimed(pf_current->pi, do_pf_read_start);
return;
}
- next_request(-EIO);
+ next_request(BLK_STS_IOERR);
return;
}
pf_mask = STAT_DRQ;
pi_do_claimed(pf_current->pi, do_pf_read_start);
return;
}
- next_request(-EIO);
+ next_request(BLK_STS_IOERR);
return;
}
pi_read_block(pf_current->pi, pf_buf, 512);
pi_do_claimed(pf_current->pi, do_pf_write_start);
return;
}
- next_request(-EIO);
+ next_request(BLK_STS_IOERR);
return;
}
pi_do_claimed(pf_current->pi, do_pf_write_start);
return;
}
- next_request(-EIO);
+ next_request(BLK_STS_IOERR);
return;
}
pi_write_block(pf_current->pi, pf_buf, 512);
pi_do_claimed(pf_current->pi, do_pf_write_start);
return;
}
- next_request(-EIO);
+ next_request(BLK_STS_IOERR);
return;
}
pi_disconnect(pf_current->pi);
static int write_congestion_off = PKT_WRITE_CONGESTION_OFF;
static struct mutex ctl_mutex; /* Serialize open/close/setup/teardown */
static mempool_t *psd_pool;
+static struct bio_set *pkt_bio_set;
static struct class *class_pktcdvd = NULL; /* /sys/class/pktcdvd */
static struct dentry *pkt_debugfs_root = NULL; /* /sys/kernel/debug/pktcdvd */
{
kfree(cls);
}
-static ssize_t class_pktcdvd_show_map(struct class *c,
- struct class_attribute *attr,
- char *data)
+
+static ssize_t device_map_show(struct class *c, struct class_attribute *attr,
+ char *data)
{
int n = 0;
int idx;
mutex_unlock(&ctl_mutex);
return n;
}
+static CLASS_ATTR_RO(device_map);
-static ssize_t class_pktcdvd_store_add(struct class *c,
- struct class_attribute *attr,
- const char *buf,
- size_t count)
+static ssize_t add_store(struct class *c, struct class_attribute *attr,
+ const char *buf, size_t count)
{
unsigned int major, minor;
return -EINVAL;
}
+static CLASS_ATTR_WO(add);
-static ssize_t class_pktcdvd_store_remove(struct class *c,
- struct class_attribute *attr,
- const char *buf,
- size_t count)
+static ssize_t remove_store(struct class *c, struct class_attribute *attr,
+ const char *buf, size_t count)
{
unsigned int major, minor;
if (sscanf(buf, "%u:%u", &major, &minor) == 2) {
}
return -EINVAL;
}
+static CLASS_ATTR_WO(remove);
-static struct class_attribute class_pktcdvd_attrs[] = {
- __ATTR(add, 0200, NULL, class_pktcdvd_store_add),
- __ATTR(remove, 0200, NULL, class_pktcdvd_store_remove),
- __ATTR(device_map, 0444, class_pktcdvd_show_map, NULL),
- __ATTR_NULL
+static struct attribute *class_pktcdvd_attrs[] = {
+ &class_attr_add.attr,
+ &class_attr_remove.attr,
+ &class_attr_device_map.attr,
+ NULL,
};
-
+ATTRIBUTE_GROUPS(class_pktcdvd);
static int pkt_sysfs_init(void)
{
class_pktcdvd->name = DRIVER_NAME;
class_pktcdvd->owner = THIS_MODULE;
class_pktcdvd->class_release = class_pktcdvd_release;
- class_pktcdvd->class_attrs = class_pktcdvd_attrs;
+ class_pktcdvd->class_groups = class_pktcdvd_groups;
ret = class_register(class_pktcdvd);
if (ret) {
kfree(class_pktcdvd);
REQ_OP_SCSI_OUT : REQ_OP_SCSI_IN, __GFP_RECLAIM);
if (IS_ERR(rq))
return PTR_ERR(rq);
- scsi_req_init(rq);
if (cgc->buflen) {
ret = blk_rq_map_kern(q, rq, cgc->buffer, cgc->buflen,
pkt_dbg(2, pd, "bio=%p sec0=%llx sec=%llx err=%d\n",
bio, (unsigned long long)pkt->sector,
- (unsigned long long)bio->bi_iter.bi_sector, bio->bi_error);
+ (unsigned long long)bio->bi_iter.bi_sector, bio->bi_status);
- if (bio->bi_error)
+ if (bio->bi_status)
atomic_inc(&pkt->io_errors);
if (atomic_dec_and_test(&pkt->io_wait)) {
atomic_inc(&pkt->run_sm);
struct pktcdvd_device *pd = pkt->pd;
BUG_ON(!pd);
- pkt_dbg(2, pd, "id=%d, err=%d\n", pkt->id, bio->bi_error);
+ pkt_dbg(2, pd, "id=%d, err=%d\n", pkt->id, bio->bi_status);
pd->stats.pkt_ended++;
pkt_queue_bio(pd, pkt->w_bio);
}
-static void pkt_finish_packet(struct packet_data *pkt, int error)
+static void pkt_finish_packet(struct packet_data *pkt, blk_status_t status)
{
struct bio *bio;
- if (error)
+ if (status)
pkt->cache_valid = 0;
/* Finish all bios corresponding to this packet */
while ((bio = bio_list_pop(&pkt->orig_bios))) {
- bio->bi_error = error;
+ bio->bi_status = status;
bio_endio(bio);
}
}
if (atomic_read(&pkt->io_wait) > 0)
return;
- if (!pkt->w_bio->bi_error) {
+ if (!pkt->w_bio->bi_status) {
pkt_set_state(pkt, PACKET_FINISHED_STATE);
} else {
pkt_set_state(pkt, PACKET_RECOVERY_STATE);
break;
case PACKET_FINISHED_STATE:
- pkt_finish_packet(pkt, pkt->w_bio->bi_error);
+ pkt_finish_packet(pkt, pkt->w_bio->bi_status);
return;
default:
struct packet_stacked_data *psd = bio->bi_private;
struct pktcdvd_device *pd = psd->pd;
- psd->bio->bi_error = bio->bi_error;
+ psd->bio->bi_status = bio->bi_status;
bio_put(bio);
bio_endio(psd->bio);
mempool_free(psd, psd_pool);
static void pkt_make_request_read(struct pktcdvd_device *pd, struct bio *bio)
{
- struct bio *cloned_bio = bio_clone(bio, GFP_NOIO);
+ struct bio *cloned_bio = bio_clone_fast(bio, GFP_NOIO, pkt_bio_set);
struct packet_stacked_data *psd = mempool_alloc(psd_pool, GFP_NOIO);
psd->pd = pd;
char b[BDEVNAME_SIZE];
struct bio *split;
- blk_queue_bounce(q, &bio);
-
- blk_queue_split(q, &bio, q->bio_split);
+ blk_queue_split(q, &bio);
pd = q->queuedata;
if (!pd) {
split = bio_split(bio, last_zone -
bio->bi_iter.bi_sector,
- GFP_NOIO, fs_bio_set);
+ GFP_NOIO, pkt_bio_set);
bio_chain(split, bio);
} else {
split = bio;
bdev = bdget(dev);
if (!bdev)
return -ENOMEM;
+ if (!blk_queue_scsi_passthrough(bdev_get_queue(bdev))) {
+ WARN_ONCE(true, "Attempt to register a non-SCSI queue\n");
+ bdput(bdev);
+ return -EINVAL;
+ }
ret = blkdev_get(bdev, FMODE_READ | FMODE_NDELAY, NULL);
if (ret)
return ret;
sizeof(struct packet_stacked_data));
if (!psd_pool)
return -ENOMEM;
+ pkt_bio_set = bioset_create(BIO_POOL_SIZE, 0, 0);
+ if (!pkt_bio_set) {
+ mempool_destroy(psd_pool);
+ return -ENOMEM;
+ }
ret = register_blkdev(pktdev_major, DRIVER_NAME);
if (ret < 0) {
unregister_blkdev(pktdev_major, DRIVER_NAME);
out2:
mempool_destroy(psd_pool);
+ bioset_free(pkt_bio_set);
return ret;
}
unregister_blkdev(pktdev_major, DRIVER_NAME);
mempool_destroy(psd_pool);
+ bioset_free(pkt_bio_set);
}
MODULE_DESCRIPTION("Packet writing layer for CD/DVD drives");
if (res) {
dev_err(&dev->sbd.core, "%s:%u: %s failed %d\n", __func__,
__LINE__, op, res);
- __blk_end_request_all(req, -EIO);
+ __blk_end_request_all(req, BLK_STS_IOERR);
return 0;
}
if (res) {
dev_err(&dev->sbd.core, "%s:%u: sync cache failed 0x%llx\n",
__func__, __LINE__, res);
- __blk_end_request_all(req, -EIO);
+ __blk_end_request_all(req, BLK_STS_IOERR);
return 0;
}
break;
default:
blk_dump_rq_flags(req, DEVICE_NAME " bad request");
- __blk_end_request_all(req, -EIO);
+ __blk_end_request_all(req, BLK_STS_IOERR);
}
}
}
struct ps3_storage_device *dev = data;
struct ps3disk_private *priv;
struct request *req;
- int res, read, error;
+ int res, read;
+ blk_status_t error;
u64 tag, status;
const char *op;
if (status) {
dev_dbg(&dev->sbd.core, "%s:%u: %s failed 0x%llx\n", __func__,
__LINE__, op, status);
- error = -EIO;
+ error = BLK_STS_IOERR;
} else {
dev_dbg(&dev->sbd.core, "%s:%u: %s completed\n", __func__,
__LINE__, op);
kfree(priv->cache.tags);
}
-static int ps3vram_read(struct ps3_system_bus_device *dev, loff_t from,
+static blk_status_t ps3vram_read(struct ps3_system_bus_device *dev, loff_t from,
size_t len, size_t *retlen, u_char *buf)
{
struct ps3vram_priv *priv = ps3_system_bus_get_drvdata(dev);
(unsigned int)from, len);
if (from >= priv->size)
- return -EIO;
+ return BLK_STS_IOERR;
if (len > priv->size - from)
len = priv->size - from;
return 0;
}
-static int ps3vram_write(struct ps3_system_bus_device *dev, loff_t to,
+static blk_status_t ps3vram_write(struct ps3_system_bus_device *dev, loff_t to,
size_t len, size_t *retlen, const u_char *buf)
{
struct ps3vram_priv *priv = ps3_system_bus_get_drvdata(dev);
unsigned int cached, count;
if (to >= priv->size)
- return -EIO;
+ return BLK_STS_IOERR;
if (len > priv->size - to)
len = priv->size - to;
int write = bio_data_dir(bio) == WRITE;
const char *op = write ? "write" : "read";
loff_t offset = bio->bi_iter.bi_sector << 9;
- int error = 0;
+ blk_status_t error = 0;
struct bio_vec bvec;
struct bvec_iter iter;
struct bio *next;
if (retlen != len) {
dev_err(&dev->core, "Short %s\n", op);
- error = -EIO;
+ error = BLK_STS_IOERR;
goto out;
}
next = bio_list_peek(&priv->list);
spin_unlock_irq(&priv->lock);
- bio->bi_error = error;
+ bio->bi_status = error;
bio_endio(bio);
return next;
}
dev_dbg(&dev->core, "%s\n", __func__);
- blk_queue_split(q, &bio, q->bio_split);
+ blk_queue_split(q, &bio);
spin_lock_irq(&priv->lock);
busy = !bio_list_empty(&priv->list);
static struct kmem_cache *rbd_img_request_cache;
static struct kmem_cache *rbd_obj_request_cache;
+static struct bio_set *rbd_bio_clone;
+
static int rbd_major;
static DEFINE_IDA(rbd_dev_id_ida);
{
struct bio *bio;
- bio = bio_clone(bio_src, gfpmask);
+ bio = bio_clone_fast(bio_src, gfpmask, rbd_bio_clone);
if (!bio)
return NULL; /* ENOMEM */
rbd_assert(img_request->obj_request != NULL);
more = obj_request->which < img_request->obj_request_count - 1;
} else {
+ blk_status_t status = errno_to_blk_status(result);
+
rbd_assert(img_request->rq != NULL);
- more = blk_update_request(img_request->rq, result, xferred);
+ more = blk_update_request(img_request->rq, status, xferred);
if (!more)
- __blk_mq_end_request(img_request->rq, result);
+ __blk_mq_end_request(img_request->rq, status);
}
return more;
obj_op_name(op_type), length, offset, result);
ceph_put_snap_context(snapc);
err:
- blk_mq_end_request(rq, result);
+ blk_mq_end_request(rq, errno_to_blk_status(result));
}
-static int rbd_queue_rq(struct blk_mq_hw_ctx *hctx,
+static blk_status_t rbd_queue_rq(struct blk_mq_hw_ctx *hctx,
const struct blk_mq_queue_data *bd)
{
struct request *rq = bd->rq;
struct work_struct *work = blk_mq_rq_to_pdu(rq);
queue_work(rbd_wq, work);
- return BLK_MQ_RQ_QUEUE_OK;
+ return BLK_STS_OK;
}
static void rbd_free_disk(struct rbd_device *rbd_dev)
if (!rbd_obj_request_cache)
goto out_err;
+ rbd_assert(!rbd_bio_clone);
+ rbd_bio_clone = bioset_create(BIO_POOL_SIZE, 0, 0);
+ if (!rbd_bio_clone)
+ goto out_err_clone;
+
return 0;
+out_err_clone:
+ kmem_cache_destroy(rbd_obj_request_cache);
+ rbd_obj_request_cache = NULL;
out_err:
kmem_cache_destroy(rbd_img_request_cache);
rbd_img_request_cache = NULL;
rbd_assert(rbd_img_request_cache);
kmem_cache_destroy(rbd_img_request_cache);
rbd_img_request_cache = NULL;
+
+ rbd_assert(rbd_bio_clone);
+ bioset_free(rbd_bio_clone);
+ rbd_bio_clone = NULL;
}
static int __init rbd_init(void)
{
struct rsxx_cardinfo *card = q->queuedata;
struct rsxx_bio_meta *bio_meta;
- int st = -EINVAL;
+ blk_status_t st = BLK_STS_IOERR;
- blk_queue_split(q, &bio, q->bio_split);
+ blk_queue_split(q, &bio);
might_sleep();
if (bio_end_sector(bio) > get_capacity(card->gendisk))
goto req_err;
- if (unlikely(card->halt)) {
- st = -EFAULT;
+ if (unlikely(card->halt))
goto req_err;
- }
- if (unlikely(card->dma_fault)) {
- st = (-EFAULT);
+ if (unlikely(card->dma_fault))
goto req_err;
- }
if (bio->bi_iter.bi_size == 0) {
dev_err(CARD_TO_DEV(card), "size zero BIO!\n");
bio_meta = kmem_cache_alloc(bio_meta_pool, GFP_KERNEL);
if (!bio_meta) {
- st = -ENOMEM;
+ st = BLK_STS_RESOURCE;
goto req_err;
}
kmem_cache_free(bio_meta_pool, bio_meta);
req_err:
if (st)
- bio->bi_error = st;
+ bio->bi_status = st;
bio_endio(bio);
return BLK_QC_T_NONE;
}
}
blk_queue_make_request(card->queue, rsxx_make_request);
- blk_queue_bounce_limit(card->queue, BLK_BOUNCE_ANY);
blk_queue_max_hw_sectors(card->queue, blkdev_max_hw_sectors);
blk_queue_physical_block_size(card->queue, RSXX_HW_BLK_SIZE);
mutex_unlock(&ctrl->work_lock);
}
-static int rsxx_queue_discard(struct rsxx_cardinfo *card,
+static blk_status_t rsxx_queue_discard(struct rsxx_cardinfo *card,
struct list_head *q,
unsigned int laddr,
rsxx_dma_cb cb,
dma = kmem_cache_alloc(rsxx_dma_pool, GFP_KERNEL);
if (!dma)
- return -ENOMEM;
+ return BLK_STS_RESOURCE;
dma->cmd = HW_CMD_BLK_DISCARD;
dma->laddr = laddr;
return 0;
}
-static int rsxx_queue_dma(struct rsxx_cardinfo *card,
+static blk_status_t rsxx_queue_dma(struct rsxx_cardinfo *card,
struct list_head *q,
int dir,
unsigned int dma_off,
dma = kmem_cache_alloc(rsxx_dma_pool, GFP_KERNEL);
if (!dma)
- return -ENOMEM;
+ return BLK_STS_RESOURCE;
dma->cmd = dir ? HW_CMD_BLK_WRITE : HW_CMD_BLK_READ;
dma->laddr = laddr;
return 0;
}
-int rsxx_dma_queue_bio(struct rsxx_cardinfo *card,
+blk_status_t rsxx_dma_queue_bio(struct rsxx_cardinfo *card,
struct bio *bio,
atomic_t *n_dmas,
rsxx_dma_cb cb,
unsigned int dma_len;
int dma_cnt[RSXX_MAX_TARGETS];
int tgt;
- int st;
+ blk_status_t st;
int i;
addr8 = bio->bi_iter.bi_sector << 9; /* sectors are 512 bytes */
for (i = 0; i < card->n_targets; i++)
rsxx_cleanup_dma_queue(&card->ctrl[i], &dma_list[i],
FREE_DMA);
-
return st;
}
void rsxx_dma_cleanup(void);
void rsxx_dma_queue_reset(struct rsxx_cardinfo *card);
int rsxx_dma_configure(struct rsxx_cardinfo *card);
-int rsxx_dma_queue_bio(struct rsxx_cardinfo *card,
+blk_status_t rsxx_dma_queue_bio(struct rsxx_cardinfo *card,
struct bio *bio,
atomic_t *n_dmas,
rsxx_dma_cb cb,
struct skd_special_context *skspcl);
static void skd_request_fn(struct request_queue *rq);
static void skd_end_request(struct skd_device *skdev,
- struct skd_request_context *skreq, int error);
-static int skd_preop_sg_list(struct skd_device *skdev,
+ struct skd_request_context *skreq, blk_status_t status);
+static bool skd_preop_sg_list(struct skd_device *skdev,
struct skd_request_context *skreq);
static void skd_postop_sg_list(struct skd_device *skdev,
struct skd_request_context *skreq);
if (req == NULL)
break;
blk_start_request(req);
- __blk_end_request_all(req, -EIO);
+ __blk_end_request_all(req, BLK_STS_IOERR);
}
}
struct request *req = NULL;
struct skd_scsi_request *scsi_req;
unsigned long io_flags;
- int error;
u32 lba;
u32 count;
int data_dir;
if (!req->bio)
goto skip_sg;
- error = skd_preop_sg_list(skdev, skreq);
-
- if (error != 0) {
+ if (!skd_preop_sg_list(skdev, skreq)) {
/*
* Complete the native request with error.
* Note that the request context is still at the
*/
pr_debug("%s:%s:%d error Out\n",
skdev->name, __func__, __LINE__);
- skd_end_request(skdev, skreq, error);
+ skd_end_request(skdev, skreq, BLK_STS_RESOURCE);
continue;
}
}
static void skd_end_request(struct skd_device *skdev,
- struct skd_request_context *skreq, int error)
+ struct skd_request_context *skreq, blk_status_t error)
{
if (unlikely(error)) {
struct request *req = skreq->req;
__blk_end_request_all(skreq->req, error);
}
-static int skd_preop_sg_list(struct skd_device *skdev,
+static bool skd_preop_sg_list(struct skd_device *skdev,
struct skd_request_context *skreq)
{
struct request *req = skreq->req;
n_sg = blk_rq_map_sg(skdev->queue, req, sg);
if (n_sg <= 0)
- return -EINVAL;
+ return false;
/*
* Map scatterlist to PCI bus addresses.
*/
n_sg = pci_map_sg(skdev->pdev, sg, n_sg, pci_dir);
if (n_sg <= 0)
- return -EINVAL;
+ return false;
SKD_ASSERT(n_sg <= skdev->sgs_per_request);
}
}
- return 0;
+ return true;
}
static void skd_postop_sg_list(struct skd_device *skdev,
switch (skd_check_status(skdev, cmp_status, &skreq->err_info)) {
case SKD_CHECK_STATUS_REPORT_GOOD:
case SKD_CHECK_STATUS_REPORT_SMART_ALERT:
- skd_end_request(skdev, skreq, 0);
+ skd_end_request(skdev, skreq, BLK_STS_OK);
break;
case SKD_CHECK_STATUS_BUSY_IMMINENT:
case SKD_CHECK_STATUS_REPORT_ERROR:
default:
- skd_end_request(skdev, skreq, -EIO);
+ skd_end_request(skdev, skreq, BLK_STS_IOERR);
break;
}
}
* native request.
*/
if (likely(cmp_status == SAM_STAT_GOOD))
- skd_end_request(skdev, skreq, 0);
+ skd_end_request(skdev, skreq, BLK_STS_OK);
else
skd_resolve_req_exception(skdev, skreq);
}
SKD_MAX_RETRIES)
blk_requeue_request(skdev->queue, skreq->req);
else
- skd_end_request(skdev, skreq, -EIO);
+ skd_end_request(skdev, skreq, BLK_STS_IOERR);
skreq->req = NULL;
rc = -ENOMEM;
goto err_out;
}
+ blk_queue_bounce_limit(q, BLK_BOUNCE_HIGH);
skdev->queue = q;
disk->queue = q;
rqe->req = NULL;
- __blk_end_request(req, (desc->status ? -EIO : 0), desc->size);
+ __blk_end_request(req, (desc->status ? BLK_STS_IOERR : 0), desc->size);
vdc_blk_queue_start(port);
}
struct request *req;
while ((req = blk_fetch_request(port->disk->queue)) != NULL)
- __blk_end_request_all(req, -EIO);
+ __blk_end_request_all(req, BLK_STS_IOERR);
}
static void vdc_ldc_reset_timer(unsigned long _arg)
return ret;
}
-static int floppy_read_sectors(struct floppy_state *fs,
+static blk_status_t floppy_read_sectors(struct floppy_state *fs,
int req_sector, int sectors_nb,
unsigned char *buffer)
{
ret = swim_read_sector(fs, side, track, sector,
buffer);
if (try-- == 0)
- return -EIO;
+ return BLK_STS_IOERR;
} while (ret != 512);
buffer += ret;
req = swim_next_request(swd);
while (req) {
- int err = -EIO;
+ blk_status_t err = BLK_STS_IOERR;
fs = req->rq_disk->private_data;
if (blk_rq_pos(req) >= fs->total_secs)
put_disk(swd->unit[drive].disk);
goto exit_put_disks;
}
+ blk_queue_bounce_limit(swd->unit[drive].disk->queue,
+ BLK_BOUNCE_HIGH);
swd->unit[drive].disk->queue->queuedata = swd;
swd->unit[drive].swd = swd;
}
unsigned int clearing);
static int floppy_revalidate(struct gendisk *disk);
-static bool swim3_end_request(struct floppy_state *fs, int err, unsigned int nr_bytes)
+static bool swim3_end_request(struct floppy_state *fs, blk_status_t err, unsigned int nr_bytes)
{
struct request *req = fs->cur_req;
int rc;
if (fs->mdev->media_bay &&
check_media_bay(fs->mdev->media_bay) != MB_FD) {
swim3_dbg("%s", " media bay absent, dropping req\n");
- swim3_end_request(fs, -ENODEV, 0);
+ swim3_end_request(fs, BLK_STS_IOERR, 0);
continue;
}
if (blk_rq_pos(req) >= fs->total_secs) {
swim3_dbg(" pos out of bounds (%ld, max is %ld)\n",
(long)blk_rq_pos(req), (long)fs->total_secs);
- swim3_end_request(fs, -EIO, 0);
+ swim3_end_request(fs, BLK_STS_IOERR, 0);
continue;
}
if (fs->ejected) {
swim3_dbg("%s", " disk ejected\n");
- swim3_end_request(fs, -EIO, 0);
+ swim3_end_request(fs, BLK_STS_IOERR, 0);
continue;
}
fs->write_prot = swim3_readbit(fs, WRITE_PROT);
if (fs->write_prot) {
swim3_dbg("%s", " try to write, disk write protected\n");
- swim3_end_request(fs, -EIO, 0);
+ swim3_end_request(fs, BLK_STS_IOERR, 0);
continue;
}
}
if (fs->retries > 5) {
swim3_err("Wrong cylinder in transfer, want: %d got %d\n",
fs->req_cyl, fs->cur_cyl);
- swim3_end_request(fs, -EIO, 0);
+ swim3_end_request(fs, BLK_STS_IOERR, 0);
fs->state = idle;
return;
}
out_8(&sw->intr_enable, 0);
fs->cur_cyl = -1;
if (fs->retries > 5) {
- swim3_end_request(fs, -EIO, 0);
+ swim3_end_request(fs, BLK_STS_IOERR, 0);
fs->state = idle;
start_request(fs);
} else {
out_8(&sw->select, RELAX);
out_8(&sw->intr_enable, 0);
swim3_err("%s", "Seek timeout\n");
- swim3_end_request(fs, -EIO, 0);
+ swim3_end_request(fs, BLK_STS_IOERR, 0);
fs->state = idle;
start_request(fs);
spin_unlock_irqrestore(&swim3_lock, flags);
goto unlock;
}
swim3_err("%s", "Seek settle timeout\n");
- swim3_end_request(fs, -EIO, 0);
+ swim3_end_request(fs, BLK_STS_IOERR, 0);
fs->state = idle;
start_request(fs);
unlock:
swim3_err("Timeout %sing sector %ld\n",
(rq_data_dir(fs->cur_req)==WRITE? "writ": "read"),
(long)blk_rq_pos(fs->cur_req));
- swim3_end_request(fs, -EIO, 0);
+ swim3_end_request(fs, BLK_STS_IOERR, 0);
fs->state = idle;
start_request(fs);
spin_unlock_irqrestore(&swim3_lock, flags);
swim3_err("%s", "Seen sector but cyl=ff?\n");
fs->cur_cyl = -1;
if (fs->retries > 5) {
- swim3_end_request(fs, -EIO, 0);
+ swim3_end_request(fs, BLK_STS_IOERR, 0);
fs->state = idle;
start_request(fs);
} else {
swim3_err("Error %sing block %ld (err=%x)\n",
rq_data_dir(req) == WRITE? "writ": "read",
(long)blk_rq_pos(req), err);
- swim3_end_request(fs, -EIO, 0);
+ swim3_end_request(fs, BLK_STS_IOERR, 0);
fs->state = idle;
}
} else {
swim3_err("fd dma error: stat=%x resid=%d\n", stat, resid);
swim3_err(" state=%d, dir=%x, intr=%x, err=%x\n",
fs->state, rq_data_dir(req), intr, err);
- swim3_end_request(fs, -EIO, 0);
+ swim3_end_request(fs, BLK_STS_IOERR, 0);
fs->state = idle;
start_request(fs);
break;
put_disk(disk);
return -ENOMEM;
}
+ blk_queue_bounce_limit(disk->queue, BLK_BOUNCE_HIGH);
disk->queue->queuedata = &floppy_states[index];
if (index == 0) {
return 0;
}
-static struct of_device_id swim3_match[] =
+static const struct of_device_id swim3_match[] =
{
{
.name = "swim3",
static inline void carm_end_request_queued(struct carm_host *host,
struct carm_request *crq,
- int error)
+ blk_status_t error)
{
struct request *req = crq->rq;
int rc;
}
static inline void carm_end_rq(struct carm_host *host, struct carm_request *crq,
- int error)
+ blk_status_t error)
{
carm_end_request_queued(host, crq, error);
if (max_queue == 1)
sg = &crq->sg[0];
n_elem = blk_rq_map_sg(q, rq, sg);
if (n_elem <= 0) {
- carm_end_rq(host, crq, -EIO);
+ carm_end_rq(host, crq, BLK_STS_IOERR);
return; /* request with no s/g entries? */
}
/* map scatterlist to PCI bus addresses */
n_elem = pci_map_sg(host->pdev, sg, n_elem, pci_dir);
if (n_elem <= 0) {
- carm_end_rq(host, crq, -EIO);
+ carm_end_rq(host, crq, BLK_STS_IOERR);
return; /* request with no s/g entries? */
}
crq->n_elem = n_elem;
static void carm_handle_array_info(struct carm_host *host,
struct carm_request *crq, u8 *mem,
- int error)
+ blk_status_t error)
{
struct carm_port *port;
u8 *msg_data = mem + sizeof(struct carm_array_info);
static void carm_handle_scan_chan(struct carm_host *host,
struct carm_request *crq, u8 *mem,
- int error)
+ blk_status_t error)
{
u8 *msg_data = mem + IOC_SCAN_CHAN_OFFSET;
unsigned int i, dev_count = 0;
}
static void carm_handle_generic(struct carm_host *host,
- struct carm_request *crq, int error,
+ struct carm_request *crq, blk_status_t error,
int cur_state, int next_state)
{
DPRINTK("ENTER\n");
}
static inline void carm_handle_rw(struct carm_host *host,
- struct carm_request *crq, int error)
+ struct carm_request *crq, blk_status_t error)
{
int pci_dir;
u32 handle = le32_to_cpu(ret_handle_le);
unsigned int msg_idx;
struct carm_request *crq;
- int error = (status == RMSG_OK) ? 0 : -EIO;
+ blk_status_t error = (status == RMSG_OK) ? 0 : BLK_STS_IOERR;
u8 *mem;
VPRINTK("ENTER, handle == 0x%x\n", handle);
err_out:
printk(KERN_WARNING DRV_NAME "(%s): BUG: unhandled message type %d/%d\n",
pci_name(host->pdev), crq->msg_type, crq->msg_subtype);
- carm_end_rq(host, crq, -EIO);
+ carm_end_rq(host, crq, BLK_STS_IOERR);
}
static inline void carm_handle_responses(struct carm_host *host)
PCI_DMA_TODEVICE : PCI_DMA_FROMDEVICE);
if (control & DMASCR_HARD_ERROR) {
/* error */
- bio->bi_error = -EIO;
+ bio->bi_status = BLK_STS_IOERR;
dev_printk(KERN_WARNING, &card->dev->dev,
"I/O error on sector %d/%d\n",
le32_to_cpu(desc->local_addr)>>9,
(unsigned long long)bio->bi_iter.bi_sector,
bio->bi_iter.bi_size);
- blk_queue_split(q, &bio, q->bio_split);
+ blk_queue_split(q, &bio);
spin_lock_irq(&card->lock);
*card->biotail = bio;
struct scatterlist sg[];
};
-static inline int virtblk_result(struct virtblk_req *vbr)
+static inline blk_status_t virtblk_result(struct virtblk_req *vbr)
{
switch (vbr->status) {
case VIRTIO_BLK_S_OK:
- return 0;
+ return BLK_STS_OK;
case VIRTIO_BLK_S_UNSUPP:
- return -ENOTTY;
+ return BLK_STS_NOTSUPP;
default:
- return -EIO;
+ return BLK_STS_IOERR;
}
}
spin_unlock_irqrestore(&vblk->vqs[qid].lock, flags);
}
-static int virtio_queue_rq(struct blk_mq_hw_ctx *hctx,
+static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,
const struct blk_mq_queue_data *bd)
{
struct virtio_blk *vblk = hctx->queue->queuedata;
break;
default:
WARN_ON_ONCE(1);
- return BLK_MQ_RQ_QUEUE_ERROR;
+ return BLK_STS_IOERR;
}
vbr->out_hdr.type = cpu_to_virtio32(vblk->vdev, type);
/* Out of mem doesn't actually happen, since we fall back
* to direct descriptors */
if (err == -ENOMEM || err == -ENOSPC)
- return BLK_MQ_RQ_QUEUE_BUSY;
- return BLK_MQ_RQ_QUEUE_ERROR;
+ return BLK_STS_RESOURCE;
+ return BLK_STS_IOERR;
}
if (bd->last && virtqueue_kick_prepare(vblk->vqs[qid].vq))
if (notify)
virtqueue_notify(vblk->vqs[qid].vq);
- return BLK_MQ_RQ_QUEUE_OK;
+ return BLK_STS_OK;
}
/* return id (s/n) string for *disk to *id_str
goto out;
blk_execute_rq(vblk->disk->queue, vblk->disk, req, false);
- err = virtblk_result(blk_mq_rq_to_pdu(req));
+ err = blk_status_to_errno(virtblk_result(blk_mq_rq_to_pdu(req)));
out:
blk_put_request(req);
return err;
/* We can handle whatever the host told us to handle. */
blk_queue_max_segments(q, vblk->sg_elems-2);
- /* No need to bounce any requests */
- blk_queue_bounce_limit(q, BLK_BOUNCE_ANY);
-
/* No real sector limit. */
blk_queue_max_hw_sectors(q, -1U);
unsigned long timeout;
int ret;
- xen_blkif_get(blkif);
-
set_freezable();
while (!kthread_should_stop()) {
if (try_to_freeze())
print_stats(ring);
ring->xenblkd = NULL;
- xen_blkif_put(blkif);
return 0;
}
atomic_set(&blkif->drain, 0);
}
-/*
- * Completion callback on the bio's. Called as bh->b_end_io()
- */
-
-static void __end_block_io_op(struct pending_req *pending_req, int error)
+static void __end_block_io_op(struct pending_req *pending_req,
+ blk_status_t error)
{
/* An error fails the entire request. */
- if ((pending_req->operation == BLKIF_OP_FLUSH_DISKCACHE) &&
- (error == -EOPNOTSUPP)) {
+ if (pending_req->operation == BLKIF_OP_FLUSH_DISKCACHE &&
+ error == BLK_STS_NOTSUPP) {
pr_debug("flush diskcache op failed, not supported\n");
xen_blkbk_flush_diskcache(XBT_NIL, pending_req->ring->blkif->be, 0);
pending_req->status = BLKIF_RSP_EOPNOTSUPP;
- } else if ((pending_req->operation == BLKIF_OP_WRITE_BARRIER) &&
- (error == -EOPNOTSUPP)) {
+ } else if (pending_req->operation == BLKIF_OP_WRITE_BARRIER &&
+ error == BLK_STS_NOTSUPP) {
pr_debug("write barrier op failed, not supported\n");
xen_blkbk_barrier(XBT_NIL, pending_req->ring->blkif->be, 0);
pending_req->status = BLKIF_RSP_EOPNOTSUPP;
*/
static void end_block_io_op(struct bio *bio)
{
- __end_block_io_op(bio->bi_private, bio->bi_error);
+ __end_block_io_op(bio->bi_private, bio->bi_status);
bio_put(bio);
}
for (i = 0; i < nbio; i++)
bio_put(biolist[i]);
atomic_set(&pending_req->pendcnt, 1);
- __end_block_io_op(pending_req, -EINVAL);
+ __end_block_io_op(pending_req, BLK_STS_RESOURCE);
msleep(1); /* back off a bit */
return -EIO;
}
static void make_response(struct xen_blkif_ring *ring, u64 id,
unsigned short op, int st)
{
- struct blkif_response resp;
+ struct blkif_response *resp;
unsigned long flags;
union blkif_back_rings *blk_rings;
int notify;
- resp.id = id;
- resp.operation = op;
- resp.status = st;
-
spin_lock_irqsave(&ring->blk_ring_lock, flags);
blk_rings = &ring->blk_rings;
/* Place on the response ring for the relevant domain. */
switch (ring->blkif->blk_protocol) {
case BLKIF_PROTOCOL_NATIVE:
- memcpy(RING_GET_RESPONSE(&blk_rings->native, blk_rings->native.rsp_prod_pvt),
- &resp, sizeof(resp));
+ resp = RING_GET_RESPONSE(&blk_rings->native,
+ blk_rings->native.rsp_prod_pvt);
break;
case BLKIF_PROTOCOL_X86_32:
- memcpy(RING_GET_RESPONSE(&blk_rings->x86_32, blk_rings->x86_32.rsp_prod_pvt),
- &resp, sizeof(resp));
+ resp = RING_GET_RESPONSE(&blk_rings->x86_32,
+ blk_rings->x86_32.rsp_prod_pvt);
break;
case BLKIF_PROTOCOL_X86_64:
- memcpy(RING_GET_RESPONSE(&blk_rings->x86_64, blk_rings->x86_64.rsp_prod_pvt),
- &resp, sizeof(resp));
+ resp = RING_GET_RESPONSE(&blk_rings->x86_64,
+ blk_rings->x86_64.rsp_prod_pvt);
break;
default:
BUG();
}
+
+ resp->id = id;
+ resp->operation = op;
+ resp->status = st;
+
blk_rings->common.rsp_prod_pvt++;
RING_PUSH_RESPONSES_AND_CHECK_NOTIFY(&blk_rings->common, notify);
spin_unlock_irqrestore(&ring->blk_ring_lock, flags);
struct blkif_common_request {
char dummy;
};
-struct blkif_common_response {
- char dummy;
-};
+
+/* i386 protocol version */
struct blkif_x86_32_request_rw {
uint8_t nr_segments; /* number of segments */
} u;
} __attribute__((__packed__));
-/* i386 protocol version */
-#pragma pack(push, 4)
-struct blkif_x86_32_response {
- uint64_t id; /* copied from request */
- uint8_t operation; /* copied from request */
- int16_t status; /* BLKIF_RSP_??? */
-};
-#pragma pack(pop)
/* x86_64 protocol version */
struct blkif_x86_64_request_rw {
} u;
} __attribute__((__packed__));
-struct blkif_x86_64_response {
- uint64_t __attribute__((__aligned__(8))) id;
- uint8_t operation; /* copied from request */
- int16_t status; /* BLKIF_RSP_??? */
-};
-
DEFINE_RING_TYPES(blkif_common, struct blkif_common_request,
- struct blkif_common_response);
+ struct blkif_response);
DEFINE_RING_TYPES(blkif_x86_32, struct blkif_x86_32_request,
- struct blkif_x86_32_response);
+ struct blkif_response __packed);
DEFINE_RING_TYPES(blkif_x86_64, struct blkif_x86_64_request,
- struct blkif_x86_64_response);
+ struct blkif_response);
union blkif_back_rings {
struct blkif_back_ring native;
wait_queue_head_t wq;
atomic_t inflight;
+ bool active;
/* One thread per blkif ring. */
struct task_struct *xenblkd;
unsigned int waiting_reqs;
init_waitqueue_head(&ring->shutdown_wq);
ring->blkif = blkif;
ring->st_print = jiffies;
- xen_blkif_get(blkif);
+ ring->active = true;
}
return 0;
struct xen_blkif_ring *ring = &blkif->rings[r];
unsigned int i = 0;
+ if (!ring->active)
+ continue;
+
if (ring->xenblkd) {
kthread_stop(ring->xenblkd);
wake_up(&ring->shutdown_wq);
- ring->xenblkd = NULL;
}
/* The above kthread_stop() guarantees that at this point we
BUG_ON(ring->free_pages_num != 0);
BUG_ON(ring->persistent_gnt_c != 0);
WARN_ON(i != (XEN_BLKIF_REQS_PER_PAGE * blkif->nr_ring_pages));
- xen_blkif_put(blkif);
+ ring->active = false;
}
blkif->nr_ring_pages = 0;
/*
static void xen_blkif_free(struct xen_blkif *blkif)
{
-
- xen_blkif_disconnect(blkif);
+ WARN_ON(xen_blkif_disconnect(blkif));
xen_vbd_free(&blkif->vbd);
+ kfree(blkif->be->mode);
+ kfree(blkif->be);
/* Make sure everything is drained before shutting down */
kmem_cache_free(xen_blkif_cachep, blkif);
xen_blkif_put(be->blkif);
}
- kfree(be->mode);
- kfree(be);
return 0;
}
unsigned long associated_id;
};
-struct split_bio {
- struct bio *bio;
- atomic_t pending;
-};
-
struct blkif_req {
int error;
};
!info->feature_fua));
}
-static int blkif_queue_rq(struct blk_mq_hw_ctx *hctx,
+static blk_status_t blkif_queue_rq(struct blk_mq_hw_ctx *hctx,
const struct blk_mq_queue_data *qd)
{
unsigned long flags;
flush_requests(rinfo);
spin_unlock_irqrestore(&rinfo->ring_lock, flags);
- return BLK_MQ_RQ_QUEUE_OK;
+ return BLK_STS_OK;
out_err:
spin_unlock_irqrestore(&rinfo->ring_lock, flags);
- return BLK_MQ_RQ_QUEUE_ERROR;
+ return BLK_STS_IOERR;
out_busy:
spin_unlock_irqrestore(&rinfo->ring_lock, flags);
blk_mq_stop_hw_queue(hctx);
- return BLK_MQ_RQ_QUEUE_BUSY;
+ return BLK_STS_RESOURCE;
}
static void blkif_complete_rq(struct request *rq)
/* Make sure buffer addresses are sector-aligned. */
blk_queue_dma_alignment(rq, 511);
-
- /* Make sure we don't use bounce buffers. */
- blk_queue_bounce_limit(rq, BLK_BOUNCE_ANY);
}
static int xlvbd_init_blk_queue(struct gendisk *gd, u16 sector_size,
continue;
}
- blkif_req(req)->error = (bret->status == BLKIF_RSP_OKAY) ? 0 : -EIO;
+ if (bret->status == BLKIF_RSP_OKAY)
+ blkif_req(req)->error = BLK_STS_OK;
+ else
+ blkif_req(req)->error = BLK_STS_IOERR;
+
switch (bret->operation) {
case BLKIF_OP_DISCARD:
if (unlikely(bret->status == BLKIF_RSP_EOPNOTSUPP)) {
struct request_queue *rq = info->rq;
printk(KERN_WARNING "blkfront: %s: %s op failed\n",
info->gd->disk_name, op_name(bret->operation));
- blkif_req(req)->error = -EOPNOTSUPP;
+ blkif_req(req)->error = BLK_STS_NOTSUPP;
info->feature_discard = 0;
info->feature_secdiscard = 0;
queue_flag_clear(QUEUE_FLAG_DISCARD, rq);
rinfo->shadow[id].req.u.rw.nr_segments == 0)) {
printk(KERN_WARNING "blkfront: %s: empty %s op failed\n",
info->gd->disk_name, op_name(bret->operation));
- blkif_req(req)->error = -EOPNOTSUPP;
+ blkif_req(req)->error = BLK_STS_NOTSUPP;
}
if (unlikely(blkif_req(req)->error)) {
- if (blkif_req(req)->error == -EOPNOTSUPP)
- blkif_req(req)->error = 0;
+ if (blkif_req(req)->error == BLK_STS_NOTSUPP)
+ blkif_req(req)->error = BLK_STS_OK;
info->feature_fua = 0;
info->feature_flush = 0;
xlvbd_flush(info);
return 0;
}
-static void split_bio_end(struct bio *bio)
-{
- struct split_bio *split_bio = bio->bi_private;
-
- if (atomic_dec_and_test(&split_bio->pending)) {
- split_bio->bio->bi_phys_segments = 0;
- split_bio->bio->bi_error = bio->bi_error;
- bio_endio(split_bio->bio);
- kfree(split_bio);
- }
- bio_put(bio);
-}
-
static int blkif_recover(struct blkfront_info *info)
{
- unsigned int i, r_index;
+ unsigned int r_index;
struct request *req, *n;
int rc;
- struct bio *bio, *cloned_bio;
- unsigned int segs, offset;
- int pending, size;
- struct split_bio *split_bio;
+ struct bio *bio;
+ unsigned int segs;
blkfront_gather_backend_features(info);
/* Reset limits changed by blk_mq_update_nr_hw_queues(). */
while ((bio = bio_list_pop(&info->bio_list)) != NULL) {
/* Traverse the list of pending bios and re-queue them */
- if (bio_segments(bio) > segs) {
- /*
- * This bio has more segments than what we can
- * handle, we have to split it.
- */
- pending = (bio_segments(bio) + segs - 1) / segs;
- split_bio = kzalloc(sizeof(*split_bio), GFP_NOIO);
- BUG_ON(split_bio == NULL);
- atomic_set(&split_bio->pending, pending);
- split_bio->bio = bio;
- for (i = 0; i < pending; i++) {
- offset = (i * segs * XEN_PAGE_SIZE) >> 9;
- size = min((unsigned int)(segs * XEN_PAGE_SIZE) >> 9,
- (unsigned int)bio_sectors(bio) - offset);
- cloned_bio = bio_clone(bio, GFP_NOIO);
- BUG_ON(cloned_bio == NULL);
- bio_trim(cloned_bio, offset, size);
- cloned_bio->bi_private = split_bio;
- cloned_bio->bi_end_io = split_bio_end;
- submit_bio(cloned_bio);
- }
- /*
- * Now we have to wait for all those smaller bios to
- * end, so we can also end the "parent" bio.
- */
- continue;
- }
- /* We don't need to split this bio */
submit_bio(bio);
}
merge_bio.tail = shadow[j].request->biotail;
bio_list_merge(&info->bio_list, &merge_bio);
shadow[j].request->bio = NULL;
- blk_mq_end_request(shadow[j].request, 0);
+ blk_mq_end_request(shadow[j].request, BLK_STS_OK);
}
}
if (!blk_rq_is_passthrough(req))
break;
blk_start_request(req);
- __blk_end_request_all(req, -EIO);
+ __blk_end_request_all(req, BLK_STS_IOERR);
}
return req;
}
/* Drop all in-flight and pending requests */
if (ace->req) {
- __blk_end_request_all(ace->req, -EIO);
+ __blk_end_request_all(ace->req, BLK_STS_IOERR);
ace->req = NULL;
}
while ((req = blk_fetch_request(ace->queue)) != NULL)
- __blk_end_request_all(req, -EIO);
+ __blk_end_request_all(req, BLK_STS_IOERR);
/* Drop back to IDLE state and notify waiters */
ace->fsm_state = ACE_FSM_STATE_IDLE;
}
/* bio finished; is there another one? */
- if (__blk_end_request_cur(ace->req, 0)) {
+ if (__blk_end_request_cur(ace->req, BLK_STS_OK)) {
/* dev_dbg(ace->dev, "next block; h=%u c=%u\n",
* blk_rq_sectors(ace->req),
* blk_rq_cur_sectors(ace->req));
if (ace->queue == NULL)
goto err_blk_initq;
blk_queue_logical_block_size(ace->queue, 512);
+ blk_queue_bounce_limit(ace->queue, BLK_BOUNCE_HIGH);
/*
* Allocate and initialize GD structure
while (req) {
unsigned long start = blk_rq_pos(req) << 9;
unsigned long len = blk_rq_cur_bytes(req);
- int err = 0;
+ blk_status_t err = BLK_STS_OK;
if (start + len > z2ram_size) {
pr_err(DEVICE_NAME ": bad access: block=%llu, "
"count=%u\n",
(unsigned long long)blk_rq_pos(req),
blk_rq_cur_sectors(req));
- err = -EIO;
+ err = BLK_STS_IOERR;
goto done;
}
while (len) {
}
/* zram-control sysfs attributes */
+
+/*
+ * NOTE: hot_add attribute is not the usual read-only sysfs attribute. In a
+ * sense that reading from this file does alter the state of your system -- it
+ * creates a new un-initialized zram device and returns back this device's
+ * device_id (or an error code if it fails to create a new device).
+ */
static ssize_t hot_add_show(struct class *class,
struct class_attribute *attr,
char *buf)
return ret;
return scnprintf(buf, PAGE_SIZE, "%d\n", ret);
}
+static CLASS_ATTR_RO(hot_add);
static ssize_t hot_remove_store(struct class *class,
struct class_attribute *attr,
mutex_unlock(&zram_index_mutex);
return ret ? ret : count;
}
+static CLASS_ATTR_WO(hot_remove);
-/*
- * NOTE: hot_add attribute is not the usual read-only sysfs attribute. In a
- * sense that reading from this file does alter the state of your system -- it
- * creates a new un-initialized zram device and returns back this device's
- * device_id (or an error code if it fails to create a new device).
- */
-static struct class_attribute zram_control_class_attrs[] = {
- __ATTR(hot_add, 0400, hot_add_show, NULL),
- __ATTR_WO(hot_remove),
- __ATTR_NULL,
+static struct attribute *zram_control_class_attrs[] = {
+ &class_attr_hot_add.attr,
+ &class_attr_hot_remove.attr,
+ NULL,
};
+ATTRIBUTE_GROUPS(zram_control_class);
static struct class zram_control_class = {
.name = "zram-control",
.owner = THIS_MODULE,
- .class_attrs = zram_control_class_attrs,
+ .class_groups = zram_control_class_groups,
};
static int zram_remove_cb(int id, void *ptr, void *data)
struct btmrvl_thread *thread = data;
struct btmrvl_private *priv = thread->priv;
struct btmrvl_adapter *adapter = priv->adapter;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
struct sk_buff *skb;
ulong flags;
if (!q)
return -ENXIO;
+ if (!blk_queue_scsi_passthrough(q)) {
+ WARN_ONCE(true,
+ "Attempt read CDDA info through a non-SCSI queue\n");
+ return -EINVAL;
+ }
+
cdi->last_sense = 0;
while (nframes) {
break;
}
req = scsi_req(rq);
- scsi_req_init(rq);
ret = blk_rq_map_user(q, rq, NULL, ubuf, len, GFP_KERNEL);
if (ret) {
*/
static void gdrom_readdisk_dma(struct work_struct *work)
{
- int err, block, block_cnt;
+ int block, block_cnt;
+ blk_status_t err;
struct packet_command *read_command;
struct list_head *elem, *next;
struct request *req;
__raw_writeb(1, GDROM_DMA_STATUS_REG);
wait_event_interruptible_timeout(request_queue,
gd.transfer == 0, GDROM_DEFAULT_TIMEOUT);
- err = gd.transfer ? -EIO : 0;
+ err = gd.transfer ? BLK_STS_IOERR : BLK_STS_OK;
gd.transfer = 0;
gd.pending = 0;
/* now seek to take the request spinlock
break;
case REQ_OP_WRITE:
pr_notice("Read only device - write request ignored\n");
- __blk_end_request_all(req, -EIO);
+ __blk_end_request_all(req, BLK_STS_IOERR);
break;
default:
printk(KERN_DEBUG "gdrom: Non-fs request ignored\n");
- __blk_end_request_all(req, -EIO);
+ __blk_end_request_all(req, BLK_STS_IOERR);
break;
}
}
err = -ENOMEM;
goto probe_fail_requestq;
}
+ blk_queue_bounce_limit(gd.gdrom_rq, BLK_BOUNCE_HIGH);
err = probe_gdrom_setupqueue();
if (err)
out to lunch past a certain margin. It can reboot the system
or merely print a warning.
-config MMTIMER
- tristate "MMTIMER Memory mapped RTC for SGI Altix"
- depends on IA64_GENERIC || IA64_SGI_SN2
- depends on POSIX_TIMERS
- default y
- help
- The mmtimer device allows direct userspace access to the
- Altix system timer.
-
config UV_MMTIMER
tristate "UV_MMTIMER Memory mapped RTC for SGI UV"
depends on X86_UV
obj-$(CONFIG_RAW_DRIVER) += raw.o
obj-$(CONFIG_SGI_SNSC) += snsc.o snsc_event.o
obj-$(CONFIG_MSPEC) += mspec.o
-obj-$(CONFIG_MMTIMER) += mmtimer.o
obj-$(CONFIG_UV_MMTIMER) += uv_mmtimer.o
obj-$(CONFIG_IBM_BSR) += bsr.o
obj-$(CONFIG_SGI_MBCS) += mbcs.o
loff_t *ppos)
{
int rv = 0;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
if (count <= 0)
return 0;
+++ /dev/null
-/*
- * Timer device implementation for SGI SN platforms.
- *
- * This file is subject to the terms and conditions of the GNU General Public
- * License. See the file "COPYING" in the main directory of this archive
- * for more details.
- *
- * Copyright (c) 2001-2006 Silicon Graphics, Inc. All rights reserved.
- *
- * This driver exports an API that should be supportable by any HPET or IA-PC
- * multimedia timer. The code below is currently specific to the SGI Altix
- * SHub RTC, however.
- *
- * 11/01/01 - jbarnes - initial revision
- * 9/10/04 - Christoph Lameter - remove interrupt support for kernel inclusion
- * 10/1/04 - Christoph Lameter - provide posix clock CLOCK_SGI_CYCLE
- * 10/13/04 - Christoph Lameter, Dimitri Sivanich - provide timer interrupt
- * support via the posix timer interface
- */
-
-#include <linux/types.h>
-#include <linux/kernel.h>
-#include <linux/ioctl.h>
-#include <linux/module.h>
-#include <linux/init.h>
-#include <linux/errno.h>
-#include <linux/mm.h>
-#include <linux/fs.h>
-#include <linux/mmtimer.h>
-#include <linux/miscdevice.h>
-#include <linux/posix-timers.h>
-#include <linux/interrupt.h>
-#include <linux/time.h>
-#include <linux/math64.h>
-#include <linux/mutex.h>
-#include <linux/slab.h>
-
-#include <linux/uaccess.h>
-#include <asm/sn/addrs.h>
-#include <asm/sn/intr.h>
-#include <asm/sn/shub_mmr.h>
-#include <asm/sn/nodepda.h>
-#include <asm/sn/shubio.h>
-
-MODULE_AUTHOR("Jesse Barnes <jbarnes@sgi.com>");
-MODULE_DESCRIPTION("SGI Altix RTC Timer");
-MODULE_LICENSE("GPL");
-
-/* name of the device, usually in /dev */
-#define MMTIMER_NAME "mmtimer"
-#define MMTIMER_DESC "SGI Altix RTC Timer"
-#define MMTIMER_VERSION "2.1"
-
-#define RTC_BITS 55 /* 55 bits for this implementation */
-
-static struct k_clock sgi_clock;
-
-extern unsigned long sn_rtc_cycles_per_second;
-
-#define RTC_COUNTER_ADDR ((long *)LOCAL_MMR_ADDR(SH_RTC))
-
-#define rtc_time() (*RTC_COUNTER_ADDR)
-
-static DEFINE_MUTEX(mmtimer_mutex);
-static long mmtimer_ioctl(struct file *file, unsigned int cmd,
- unsigned long arg);
-static int mmtimer_mmap(struct file *file, struct vm_area_struct *vma);
-
-/*
- * Period in femtoseconds (10^-15 s)
- */
-static unsigned long mmtimer_femtoperiod = 0;
-
-static const struct file_operations mmtimer_fops = {
- .owner = THIS_MODULE,
- .mmap = mmtimer_mmap,
- .unlocked_ioctl = mmtimer_ioctl,
- .llseek = noop_llseek,
-};
-
-/*
- * We only have comparison registers RTC1-4 currently available per
- * node. RTC0 is used by SAL.
- */
-/* Check for an RTC interrupt pending */
-static int mmtimer_int_pending(int comparator)
-{
- if (HUB_L((unsigned long *)LOCAL_MMR_ADDR(SH_EVENT_OCCURRED)) &
- SH_EVENT_OCCURRED_RTC1_INT_MASK << comparator)
- return 1;
- else
- return 0;
-}
-
-/* Clear the RTC interrupt pending bit */
-static void mmtimer_clr_int_pending(int comparator)
-{
- HUB_S((u64 *)LOCAL_MMR_ADDR(SH_EVENT_OCCURRED_ALIAS),
- SH_EVENT_OCCURRED_RTC1_INT_MASK << comparator);
-}
-
-/* Setup timer on comparator RTC1 */
-static void mmtimer_setup_int_0(int cpu, u64 expires)
-{
- u64 val;
-
- /* Disable interrupt */
- HUB_S((u64 *)LOCAL_MMR_ADDR(SH_RTC1_INT_ENABLE), 0UL);
-
- /* Initialize comparator value */
- HUB_S((u64 *)LOCAL_MMR_ADDR(SH_INT_CMPB), -1L);
-
- /* Clear pending bit */
- mmtimer_clr_int_pending(0);
-
- val = ((u64)SGI_MMTIMER_VECTOR << SH_RTC1_INT_CONFIG_IDX_SHFT) |
- ((u64)cpu_physical_id(cpu) <<
- SH_RTC1_INT_CONFIG_PID_SHFT);
-
- /* Set configuration */
- HUB_S((u64 *)LOCAL_MMR_ADDR(SH_RTC1_INT_CONFIG), val);
-
- /* Enable RTC interrupts */
- HUB_S((u64 *)LOCAL_MMR_ADDR(SH_RTC1_INT_ENABLE), 1UL);
-
- /* Initialize comparator value */
- HUB_S((u64 *)LOCAL_MMR_ADDR(SH_INT_CMPB), expires);
-
-
-}
-
-/* Setup timer on comparator RTC2 */
-static void mmtimer_setup_int_1(int cpu, u64 expires)
-{
- u64 val;
-
- HUB_S((u64 *)LOCAL_MMR_ADDR(SH_RTC2_INT_ENABLE), 0UL);
-
- HUB_S((u64 *)LOCAL_MMR_ADDR(SH_INT_CMPC), -1L);
-
- mmtimer_clr_int_pending(1);
-
- val = ((u64)SGI_MMTIMER_VECTOR << SH_RTC2_INT_CONFIG_IDX_SHFT) |
- ((u64)cpu_physical_id(cpu) <<
- SH_RTC2_INT_CONFIG_PID_SHFT);
-
- HUB_S((u64 *)LOCAL_MMR_ADDR(SH_RTC2_INT_CONFIG), val);
-
- HUB_S((u64 *)LOCAL_MMR_ADDR(SH_RTC2_INT_ENABLE), 1UL);
-
- HUB_S((u64 *)LOCAL_MMR_ADDR(SH_INT_CMPC), expires);
-}
-
-/* Setup timer on comparator RTC3 */
-static void mmtimer_setup_int_2(int cpu, u64 expires)
-{
- u64 val;
-
- HUB_S((u64 *)LOCAL_MMR_ADDR(SH_RTC3_INT_ENABLE), 0UL);
-
- HUB_S((u64 *)LOCAL_MMR_ADDR(SH_INT_CMPD), -1L);
-
- mmtimer_clr_int_pending(2);
-
- val = ((u64)SGI_MMTIMER_VECTOR << SH_RTC3_INT_CONFIG_IDX_SHFT) |
- ((u64)cpu_physical_id(cpu) <<
- SH_RTC3_INT_CONFIG_PID_SHFT);
-
- HUB_S((u64 *)LOCAL_MMR_ADDR(SH_RTC3_INT_CONFIG), val);
-
- HUB_S((u64 *)LOCAL_MMR_ADDR(SH_RTC3_INT_ENABLE), 1UL);
-
- HUB_S((u64 *)LOCAL_MMR_ADDR(SH_INT_CMPD), expires);
-}
-
-/*
- * This function must be called with interrupts disabled and preemption off
- * in order to insure that the setup succeeds in a deterministic time frame.
- * It will check if the interrupt setup succeeded.
- */
-static int mmtimer_setup(int cpu, int comparator, unsigned long expires,
- u64 *set_completion_time)
-{
- switch (comparator) {
- case 0:
- mmtimer_setup_int_0(cpu, expires);
- break;
- case 1:
- mmtimer_setup_int_1(cpu, expires);
- break;
- case 2:
- mmtimer_setup_int_2(cpu, expires);
- break;
- }
- /* We might've missed our expiration time */
- *set_completion_time = rtc_time();
- if (*set_completion_time <= expires)
- return 1;
-
- /*
- * If an interrupt is already pending then its okay
- * if not then we failed
- */
- return mmtimer_int_pending(comparator);
-}
-
-static int mmtimer_disable_int(long nasid, int comparator)
-{
- switch (comparator) {
- case 0:
- nasid == -1 ? HUB_S((u64 *)LOCAL_MMR_ADDR(SH_RTC1_INT_ENABLE),
- 0UL) : REMOTE_HUB_S(nasid, SH_RTC1_INT_ENABLE, 0UL);
- break;
- case 1:
- nasid == -1 ? HUB_S((u64 *)LOCAL_MMR_ADDR(SH_RTC2_INT_ENABLE),
- 0UL) : REMOTE_HUB_S(nasid, SH_RTC2_INT_ENABLE, 0UL);
- break;
- case 2:
- nasid == -1 ? HUB_S((u64 *)LOCAL_MMR_ADDR(SH_RTC3_INT_ENABLE),
- 0UL) : REMOTE_HUB_S(nasid, SH_RTC3_INT_ENABLE, 0UL);
- break;
- default:
- return -EFAULT;
- }
- return 0;
-}
-
-#define COMPARATOR 1 /* The comparator to use */
-
-#define TIMER_OFF 0xbadcabLL /* Timer is not setup */
-#define TIMER_SET 0 /* Comparator is set for this timer */
-
-#define MMTIMER_INTERVAL_RETRY_INCREMENT_DEFAULT 40
-
-/* There is one of these for each timer */
-struct mmtimer {
- struct rb_node list;
- struct k_itimer *timer;
- int cpu;
-};
-
-struct mmtimer_node {
- spinlock_t lock ____cacheline_aligned;
- struct rb_root timer_head;
- struct rb_node *next;
- struct tasklet_struct tasklet;
-};
-static struct mmtimer_node *timers;
-
-static unsigned mmtimer_interval_retry_increment =
- MMTIMER_INTERVAL_RETRY_INCREMENT_DEFAULT;
-module_param(mmtimer_interval_retry_increment, uint, 0644);
-MODULE_PARM_DESC(mmtimer_interval_retry_increment,
- "RTC ticks to add to expiration on interval retry (default 40)");
-
-/*
- * Add a new mmtimer struct to the node's mmtimer list.
- * This function assumes the struct mmtimer_node is locked.
- */
-static void mmtimer_add_list(struct mmtimer *n)
-{
- int nodeid = n->timer->it.mmtimer.node;
- unsigned long expires = n->timer->it.mmtimer.expires;
- struct rb_node **link = &timers[nodeid].timer_head.rb_node;
- struct rb_node *parent = NULL;
- struct mmtimer *x;
-
- /*
- * Find the right place in the rbtree:
- */
- while (*link) {
- parent = *link;
- x = rb_entry(parent, struct mmtimer, list);
-
- if (expires < x->timer->it.mmtimer.expires)
- link = &(*link)->rb_left;
- else
- link = &(*link)->rb_right;
- }
-
- /*
- * Insert the timer to the rbtree and check whether it
- * replaces the first pending timer
- */
- rb_link_node(&n->list, parent, link);
- rb_insert_color(&n->list, &timers[nodeid].timer_head);
-
- if (!timers[nodeid].next || expires < rb_entry(timers[nodeid].next,
- struct mmtimer, list)->timer->it.mmtimer.expires)
- timers[nodeid].next = &n->list;
-}
-
-/*
- * Set the comparator for the next timer.
- * This function assumes the struct mmtimer_node is locked.
- */
-static void mmtimer_set_next_timer(int nodeid)
-{
- struct mmtimer_node *n = &timers[nodeid];
- struct mmtimer *x;
- struct k_itimer *t;
- u64 expires, exp, set_completion_time;
- int i;
-
-restart:
- if (n->next == NULL)
- return;
-
- x = rb_entry(n->next, struct mmtimer, list);
- t = x->timer;
- if (!t->it.mmtimer.incr) {
- /* Not an interval timer */
- if (!mmtimer_setup(x->cpu, COMPARATOR,
- t->it.mmtimer.expires,
- &set_completion_time)) {
- /* Late setup, fire now */
- tasklet_schedule(&n->tasklet);
- }
- return;
- }
-
- /* Interval timer */
- i = 0;
- expires = exp = t->it.mmtimer.expires;
- while (!mmtimer_setup(x->cpu, COMPARATOR, expires,
- &set_completion_time)) {
- int to;
-
- i++;
- expires = set_completion_time +
- mmtimer_interval_retry_increment + (1 << i);
- /* Calculate overruns as we go. */
- to = ((u64)(expires - exp) / t->it.mmtimer.incr);
- if (to) {
- t->it_overrun += to;
- t->it.mmtimer.expires += t->it.mmtimer.incr * to;
- exp = t->it.mmtimer.expires;
- }
- if (i > 20) {
- printk(KERN_ALERT "mmtimer: cannot reschedule timer\n");
- t->it.mmtimer.clock = TIMER_OFF;
- n->next = rb_next(&x->list);
- rb_erase(&x->list, &n->timer_head);
- kfree(x);
- goto restart;
- }
- }
-}
-
-/**
- * mmtimer_ioctl - ioctl interface for /dev/mmtimer
- * @file: file structure for the device
- * @cmd: command to execute
- * @arg: optional argument to command
- *
- * Executes the command specified by @cmd. Returns 0 for success, < 0 for
- * failure.
- *
- * Valid commands:
- *
- * %MMTIMER_GETOFFSET - Should return the offset (relative to the start
- * of the page where the registers are mapped) for the counter in question.
- *
- * %MMTIMER_GETRES - Returns the resolution of the clock in femto (10^-15)
- * seconds
- *
- * %MMTIMER_GETFREQ - Copies the frequency of the clock in Hz to the address
- * specified by @arg
- *
- * %MMTIMER_GETBITS - Returns the number of bits in the clock's counter
- *
- * %MMTIMER_MMAPAVAIL - Returns 1 if the registers can be mmap'd into userspace
- *
- * %MMTIMER_GETCOUNTER - Gets the current value in the counter and places it
- * in the address specified by @arg.
- */
-static long mmtimer_ioctl(struct file *file, unsigned int cmd,
- unsigned long arg)
-{
- int ret = 0;
-
- mutex_lock(&mmtimer_mutex);
-
- switch (cmd) {
- case MMTIMER_GETOFFSET: /* offset of the counter */
- /*
- * SN RTC registers are on their own 64k page
- */
- if(PAGE_SIZE <= (1 << 16))
- ret = (((long)RTC_COUNTER_ADDR) & (PAGE_SIZE-1)) / 8;
- else
- ret = -ENOSYS;
- break;
-
- case MMTIMER_GETRES: /* resolution of the clock in 10^-15 s */
- if(copy_to_user((unsigned long __user *)arg,
- &mmtimer_femtoperiod, sizeof(unsigned long)))
- ret = -EFAULT;
- break;
-
- case MMTIMER_GETFREQ: /* frequency in Hz */
- if(copy_to_user((unsigned long __user *)arg,
- &sn_rtc_cycles_per_second,
- sizeof(unsigned long)))
- ret = -EFAULT;
- break;
-
- case MMTIMER_GETBITS: /* number of bits in the clock */
- ret = RTC_BITS;
- break;
-
- case MMTIMER_MMAPAVAIL: /* can we mmap the clock into userspace? */
- ret = (PAGE_SIZE <= (1 << 16)) ? 1 : 0;
- break;
-
- case MMTIMER_GETCOUNTER:
- if(copy_to_user((unsigned long __user *)arg,
- RTC_COUNTER_ADDR, sizeof(unsigned long)))
- ret = -EFAULT;
- break;
- default:
- ret = -ENOTTY;
- break;
- }
- mutex_unlock(&mmtimer_mutex);
- return ret;
-}
-
-/**
- * mmtimer_mmap - maps the clock's registers into userspace
- * @file: file structure for the device
- * @vma: VMA to map the registers into
- *
- * Calls remap_pfn_range() to map the clock's registers into
- * the calling process' address space.
- */
-static int mmtimer_mmap(struct file *file, struct vm_area_struct *vma)
-{
- unsigned long mmtimer_addr;
-
- if (vma->vm_end - vma->vm_start != PAGE_SIZE)
- return -EINVAL;
-
- if (vma->vm_flags & VM_WRITE)
- return -EPERM;
-
- if (PAGE_SIZE > (1 << 16))
- return -ENOSYS;
-
- vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
-
- mmtimer_addr = __pa(RTC_COUNTER_ADDR);
- mmtimer_addr &= ~(PAGE_SIZE - 1);
- mmtimer_addr &= 0xfffffffffffffffUL;
-
- if (remap_pfn_range(vma, vma->vm_start, mmtimer_addr >> PAGE_SHIFT,
- PAGE_SIZE, vma->vm_page_prot)) {
- printk(KERN_ERR "remap_pfn_range failed in mmtimer.c\n");
- return -EAGAIN;
- }
-
- return 0;
-}
-
-static struct miscdevice mmtimer_miscdev = {
- .minor = SGI_MMTIMER,
- .name = MMTIMER_NAME,
- .fops = &mmtimer_fops
-};
-
-static struct timespec sgi_clock_offset;
-static int sgi_clock_period;
-
-/*
- * Posix Timer Interface
- */
-
-static struct timespec sgi_clock_offset;
-static int sgi_clock_period;
-
-static int sgi_clock_get(clockid_t clockid, struct timespec64 *tp)
-{
- u64 nsec;
-
- nsec = rtc_time() * sgi_clock_period
- + sgi_clock_offset.tv_nsec;
- *tp = ns_to_timespec64(nsec);
- tp->tv_sec += sgi_clock_offset.tv_sec;
- return 0;
-};
-
-static int sgi_clock_set(const clockid_t clockid, const struct timespec64 *tp)
-{
-
- u64 nsec;
- u32 rem;
-
- nsec = rtc_time() * sgi_clock_period;
-
- sgi_clock_offset.tv_sec = tp->tv_sec - div_u64_rem(nsec, NSEC_PER_SEC, &rem);
-
- if (rem <= tp->tv_nsec)
- sgi_clock_offset.tv_nsec = tp->tv_sec - rem;
- else {
- sgi_clock_offset.tv_nsec = tp->tv_sec + NSEC_PER_SEC - rem;
- sgi_clock_offset.tv_sec--;
- }
- return 0;
-}
-
-/**
- * mmtimer_interrupt - timer interrupt handler
- * @irq: irq received
- * @dev_id: device the irq came from
- *
- * Called when one of the comarators matches the counter, This
- * routine will send signals to processes that have requested
- * them.
- *
- * This interrupt is run in an interrupt context
- * by the SHUB. It is therefore safe to locally access SHub
- * registers.
- */
-static irqreturn_t
-mmtimer_interrupt(int irq, void *dev_id)
-{
- unsigned long expires = 0;
- int result = IRQ_NONE;
- unsigned indx = cpu_to_node(smp_processor_id());
- struct mmtimer *base;
-
- spin_lock(&timers[indx].lock);
- base = rb_entry(timers[indx].next, struct mmtimer, list);
- if (base == NULL) {
- spin_unlock(&timers[indx].lock);
- return result;
- }
-
- if (base->cpu == smp_processor_id()) {
- if (base->timer)
- expires = base->timer->it.mmtimer.expires;
- /* expires test won't work with shared irqs */
- if ((mmtimer_int_pending(COMPARATOR) > 0) ||
- (expires && (expires <= rtc_time()))) {
- mmtimer_clr_int_pending(COMPARATOR);
- tasklet_schedule(&timers[indx].tasklet);
- result = IRQ_HANDLED;
- }
- }
- spin_unlock(&timers[indx].lock);
- return result;
-}
-
-static void mmtimer_tasklet(unsigned long data)
-{
- int nodeid = data;
- struct mmtimer_node *mn = &timers[nodeid];
- struct mmtimer *x;
- struct k_itimer *t;
- unsigned long flags;
-
- /* Send signal and deal with periodic signals */
- spin_lock_irqsave(&mn->lock, flags);
- if (!mn->next)
- goto out;
-
- x = rb_entry(mn->next, struct mmtimer, list);
- t = x->timer;
-
- if (t->it.mmtimer.clock == TIMER_OFF)
- goto out;
-
- t->it_overrun = 0;
-
- mn->next = rb_next(&x->list);
- rb_erase(&x->list, &mn->timer_head);
-
- if (posix_timer_event(t, 0) != 0)
- t->it_overrun++;
-
- if(t->it.mmtimer.incr) {
- t->it.mmtimer.expires += t->it.mmtimer.incr;
- mmtimer_add_list(x);
- } else {
- /* Ensure we don't false trigger in mmtimer_interrupt */
- t->it.mmtimer.clock = TIMER_OFF;
- t->it.mmtimer.expires = 0;
- kfree(x);
- }
- /* Set comparator for next timer, if there is one */
- mmtimer_set_next_timer(nodeid);
-
- t->it_overrun_last = t->it_overrun;
-out:
- spin_unlock_irqrestore(&mn->lock, flags);
-}
-
-static int sgi_timer_create(struct k_itimer *timer)
-{
- /* Insure that a newly created timer is off */
- timer->it.mmtimer.clock = TIMER_OFF;
- return 0;
-}
-
-/* This does not really delete a timer. It just insures
- * that the timer is not active
- *
- * Assumption: it_lock is already held with irq's disabled
- */
-static int sgi_timer_del(struct k_itimer *timr)
-{
- cnodeid_t nodeid = timr->it.mmtimer.node;
- unsigned long irqflags;
-
- spin_lock_irqsave(&timers[nodeid].lock, irqflags);
- if (timr->it.mmtimer.clock != TIMER_OFF) {
- unsigned long expires = timr->it.mmtimer.expires;
- struct rb_node *n = timers[nodeid].timer_head.rb_node;
- struct mmtimer *uninitialized_var(t);
- int r = 0;
-
- timr->it.mmtimer.clock = TIMER_OFF;
- timr->it.mmtimer.expires = 0;
-
- while (n) {
- t = rb_entry(n, struct mmtimer, list);
- if (t->timer == timr)
- break;
-
- if (expires < t->timer->it.mmtimer.expires)
- n = n->rb_left;
- else
- n = n->rb_right;
- }
-
- if (!n) {
- spin_unlock_irqrestore(&timers[nodeid].lock, irqflags);
- return 0;
- }
-
- if (timers[nodeid].next == n) {
- timers[nodeid].next = rb_next(n);
- r = 1;
- }
-
- rb_erase(n, &timers[nodeid].timer_head);
- kfree(t);
-
- if (r) {
- mmtimer_disable_int(cnodeid_to_nasid(nodeid),
- COMPARATOR);
- mmtimer_set_next_timer(nodeid);
- }
- }
- spin_unlock_irqrestore(&timers[nodeid].lock, irqflags);
- return 0;
-}
-
-/* Assumption: it_lock is already held with irq's disabled */
-static void sgi_timer_get(struct k_itimer *timr, struct itimerspec64 *cur_setting)
-{
-
- if (timr->it.mmtimer.clock == TIMER_OFF) {
- cur_setting->it_interval.tv_nsec = 0;
- cur_setting->it_interval.tv_sec = 0;
- cur_setting->it_value.tv_nsec = 0;
- cur_setting->it_value.tv_sec =0;
- return;
- }
-
- cur_setting->it_interval = ns_to_timespec64(timr->it.mmtimer.incr * sgi_clock_period);
- cur_setting->it_value = ns_to_timespec64((timr->it.mmtimer.expires - rtc_time()) * sgi_clock_period);
-}
-
-
-static int sgi_timer_set(struct k_itimer *timr, int flags,
- struct itimerspec64 *new_setting,
- struct itimerspec64 *old_setting)
-{
- unsigned long when, period, irqflags;
- int err = 0;
- cnodeid_t nodeid;
- struct mmtimer *base;
- struct rb_node *n;
-
- if (old_setting)
- sgi_timer_get(timr, old_setting);
-
- sgi_timer_del(timr);
- when = timespec64_to_ns(&new_setting->it_value);
- period = timespec64_to_ns(&new_setting->it_interval);
-
- if (when == 0)
- /* Clear timer */
- return 0;
-
- base = kmalloc(sizeof(struct mmtimer), GFP_KERNEL);
- if (base == NULL)
- return -ENOMEM;
-
- if (flags & TIMER_ABSTIME) {
- struct timespec64 n;
- unsigned long now;
-
- getnstimeofday64(&n);
- now = timespec64_to_ns(&n);
- if (when > now)
- when -= now;
- else
- /* Fire the timer immediately */
- when = 0;
- }
-
- /*
- * Convert to sgi clock period. Need to keep rtc_time() as near as possible
- * to getnstimeofday() in order to be as faithful as possible to the time
- * specified.
- */
- when = (when + sgi_clock_period - 1) / sgi_clock_period + rtc_time();
- period = (period + sgi_clock_period - 1) / sgi_clock_period;
-
- /*
- * We are allocating a local SHub comparator. If we would be moved to another
- * cpu then another SHub may be local to us. Prohibit that by switching off
- * preemption.
- */
- preempt_disable();
-
- nodeid = cpu_to_node(smp_processor_id());
-
- /* Lock the node timer structure */
- spin_lock_irqsave(&timers[nodeid].lock, irqflags);
-
- base->timer = timr;
- base->cpu = smp_processor_id();
-
- timr->it.mmtimer.clock = TIMER_SET;
- timr->it.mmtimer.node = nodeid;
- timr->it.mmtimer.incr = period;
- timr->it.mmtimer.expires = when;
-
- n = timers[nodeid].next;
-
- /* Add the new struct mmtimer to node's timer list */
- mmtimer_add_list(base);
-
- if (timers[nodeid].next == n) {
- /* No need to reprogram comparator for now */
- spin_unlock_irqrestore(&timers[nodeid].lock, irqflags);
- preempt_enable();
- return err;
- }
-
- /* We need to reprogram the comparator */
- if (n)
- mmtimer_disable_int(cnodeid_to_nasid(nodeid), COMPARATOR);
-
- mmtimer_set_next_timer(nodeid);
-
- /* Unlock the node timer structure */
- spin_unlock_irqrestore(&timers[nodeid].lock, irqflags);
-
- preempt_enable();
-
- return err;
-}
-
-static int sgi_clock_getres(const clockid_t which_clock, struct timespec64 *tp)
-{
- tp->tv_sec = 0;
- tp->tv_nsec = sgi_clock_period;
- return 0;
-}
-
-static struct k_clock sgi_clock = {
- .clock_set = sgi_clock_set,
- .clock_get = sgi_clock_get,
- .clock_getres = sgi_clock_getres,
- .timer_create = sgi_timer_create,
- .timer_set = sgi_timer_set,
- .timer_del = sgi_timer_del,
- .timer_get = sgi_timer_get
-};
-
-/**
- * mmtimer_init - device initialization routine
- *
- * Does initial setup for the mmtimer device.
- */
-static int __init mmtimer_init(void)
-{
- cnodeid_t node, maxn = -1;
-
- if (!ia64_platform_is("sn2"))
- return 0;
-
- /*
- * Sanity check the cycles/sec variable
- */
- if (sn_rtc_cycles_per_second < 100000) {
- printk(KERN_ERR "%s: unable to determine clock frequency\n",
- MMTIMER_NAME);
- goto out1;
- }
-
- mmtimer_femtoperiod = ((unsigned long)1E15 + sn_rtc_cycles_per_second /
- 2) / sn_rtc_cycles_per_second;
-
- if (request_irq(SGI_MMTIMER_VECTOR, mmtimer_interrupt, IRQF_PERCPU, MMTIMER_NAME, NULL)) {
- printk(KERN_WARNING "%s: unable to allocate interrupt.",
- MMTIMER_NAME);
- goto out1;
- }
-
- if (misc_register(&mmtimer_miscdev)) {
- printk(KERN_ERR "%s: failed to register device\n",
- MMTIMER_NAME);
- goto out2;
- }
-
- /* Get max numbered node, calculate slots needed */
- for_each_online_node(node) {
- maxn = node;
- }
- maxn++;
-
- /* Allocate list of node ptrs to mmtimer_t's */
- timers = kzalloc(sizeof(struct mmtimer_node)*maxn, GFP_KERNEL);
- if (!timers) {
- printk(KERN_ERR "%s: failed to allocate memory for device\n",
- MMTIMER_NAME);
- goto out3;
- }
-
- /* Initialize struct mmtimer's for each online node */
- for_each_online_node(node) {
- spin_lock_init(&timers[node].lock);
- tasklet_init(&timers[node].tasklet, mmtimer_tasklet,
- (unsigned long) node);
- }
-
- sgi_clock_period = NSEC_PER_SEC / sn_rtc_cycles_per_second;
- posix_timers_register_clock(CLOCK_SGI_CYCLE, &sgi_clock);
-
- printk(KERN_INFO "%s: v%s, %ld MHz\n", MMTIMER_DESC, MMTIMER_VERSION,
- sn_rtc_cycles_per_second/(unsigned long)1E6);
-
- return 0;
-
-out3:
- misc_deregister(&mmtimer_miscdev);
-out2:
- free_irq(SGI_MMTIMER_VECTOR, NULL);
-out1:
- return -1;
-}
-
-module_init(mmtimer_init);
p[crng_init_cnt % CHACHA20_KEY_SIZE] ^= *cp;
cp++; crng_init_cnt++; len--;
}
+ spin_unlock_irqrestore(&primary_crng.lock, flags);
if (crng_init_cnt >= CRNG_INIT_CNT_THRESH) {
invalidate_batched_entropy();
crng_init = 1;
wake_up_interruptible(&crng_init_wait);
pr_notice("random: fast init done\n");
}
- spin_unlock_irqrestore(&primary_crng.lock, flags);
return 1;
}
}
memzero_explicit(&buf, sizeof(buf));
crng->init_time = jiffies;
+ spin_unlock_irqrestore(&primary_crng.lock, flags);
if (crng == &primary_crng && crng_init < 2) {
invalidate_batched_entropy();
crng_init = 2;
wake_up_interruptible(&crng_init_wait);
pr_notice("random: crng init done\n");
}
- spin_unlock_irqrestore(&primary_crng.lock, flags);
}
static inline void crng_wait_ready(void)
u64 get_random_u64(void)
{
u64 ret;
- bool use_lock = crng_init < 2;
- unsigned long flags;
+ bool use_lock = READ_ONCE(crng_init) < 2;
+ unsigned long flags = 0;
struct batched_entropy *batch;
#if BITS_PER_LONG == 64
u32 get_random_u32(void)
{
u32 ret;
- bool use_lock = crng_init < 2;
- unsigned long flags;
+ bool use_lock = READ_ONCE(crng_init) < 2;
+ unsigned long flags = 0;
struct batched_entropy *batch;
if (arch_get_random_int(&ret))
#define ACPI_SIG_TPM2 "TPM2"
-static const u8 CRB_ACPI_START_UUID[] = {
- /* 0000 */ 0xAB, 0x6C, 0xBF, 0x6B, 0x63, 0x54, 0x14, 0x47,
- /* 0008 */ 0xB7, 0xCD, 0xF0, 0x20, 0x3C, 0x03, 0x68, 0xD4
-};
+static const guid_t crb_acpi_start_guid =
+ GUID_INIT(0x6BBF6CAB, 0x5463, 0x4714,
+ 0xB7, 0xCD, 0xF0, 0x20, 0x3C, 0x03, 0x68, 0xD4);
enum crb_defaults {
CRB_ACPI_START_REVISION_ID = 1,
int rc;
obj = acpi_evaluate_dsm(chip->acpi_dev_handle,
- CRB_ACPI_START_UUID,
+ &crb_acpi_start_guid,
CRB_ACPI_START_REVISION_ID,
CRB_ACPI_START_INDEX,
NULL);
sm == ACPI_TPM2_COMMAND_BUFFER_WITH_START_METHOD)
priv->flags |= CRB_FL_ACPI_START;
- if (sm == ACPI_TPM2_COMMAND_BUFFER_WITH_SMC) {
+ if (sm == ACPI_TPM2_COMMAND_BUFFER_WITH_ARM_SMC) {
if (buf->header.length < (sizeof(*buf) + sizeof(*crb_smc))) {
dev_err(dev,
FW_BUG "TPM2 ACPI table has wrong size %u for start method type %d\n",
buf->header.length,
- ACPI_TPM2_COMMAND_BUFFER_WITH_SMC);
+ ACPI_TPM2_COMMAND_BUFFER_WITH_ARM_SMC);
return -EINVAL;
}
crb_smc = ACPI_ADD_PTR(struct tpm2_crb_smc, buf, sizeof(*buf));
#define PPI_VS_REQ_START 128
#define PPI_VS_REQ_END 255
-static const u8 tpm_ppi_uuid[] = {
- 0xA6, 0xFA, 0xDD, 0x3D,
- 0x1B, 0x36,
- 0xB4, 0x4E,
- 0xA4, 0x24,
- 0x8D, 0x10, 0x08, 0x9D, 0x16, 0x53
-};
+static const guid_t tpm_ppi_guid =
+ GUID_INIT(0x3DDDFAA6, 0x361B, 0x4EB4,
+ 0xA4, 0x24, 0x8D, 0x10, 0x08, 0x9D, 0x16, 0x53);
static inline union acpi_object *
tpm_eval_dsm(acpi_handle ppi_handle, int func, acpi_object_type type,
union acpi_object *argv4)
{
BUG_ON(!ppi_handle);
- return acpi_evaluate_dsm_typed(ppi_handle, tpm_ppi_uuid,
+ return acpi_evaluate_dsm_typed(ppi_handle, &tpm_ppi_guid,
TPM_PPI_REVISION_ID,
func, argv4, type);
}
* is updated with function index from SUBREQ to SUBREQ2 since PPI
* version 1.1
*/
- if (acpi_check_dsm(chip->acpi_dev_handle, tpm_ppi_uuid,
+ if (acpi_check_dsm(chip->acpi_dev_handle, &tpm_ppi_guid,
TPM_PPI_REVISION_ID, 1 << TPM_PPI_FN_SUBREQ2))
func = TPM_PPI_FN_SUBREQ2;
"User not required",
};
- if (!acpi_check_dsm(dev_handle, tpm_ppi_uuid, TPM_PPI_REVISION_ID,
+ if (!acpi_check_dsm(dev_handle, &tpm_ppi_guid, TPM_PPI_REVISION_ID,
1 << TPM_PPI_FN_GETOPR))
return -EPERM;
if (!chip->acpi_dev_handle)
return;
- if (!acpi_check_dsm(chip->acpi_dev_handle, tpm_ppi_uuid,
+ if (!acpi_check_dsm(chip->acpi_dev_handle, &tpm_ppi_guid,
TPM_PPI_REVISION_ID, 1 << TPM_PPI_FN_VERSION))
return;
/* Cache PPI version string. */
- obj = acpi_evaluate_dsm_typed(chip->acpi_dev_handle, tpm_ppi_uuid,
+ obj = acpi_evaluate_dsm_typed(chip->acpi_dev_handle, &tpm_ppi_guid,
TPM_PPI_REVISION_ID, TPM_PPI_FN_VERSION,
NULL, ACPI_TYPE_STRING);
if (obj) {
config COMMON_CLK_GXBB
bool
depends on COMMON_CLK_AMLOGIC
+ select RESET_CONTROLLER
help
Support for the clock controller on AmLogic S905 devices, aka gxbb.
Say Y if you want peripherals and CPU frequency scaling to work.
bool "Support for Allwinner SoCs' PRCM CCUs"
select SUNXI_CCU_DIV
select SUNXI_CCU_GATE
+ select SUNXI_CCU_MP
default MACH_SUN8I || (ARCH_SUNXI && ARM64)
endif
#define CLK_PLL_VIDEO0_2X 8
#define CLK_PLL_VE 9
#define CLK_PLL_DDR0 10
-#define CLK_PLL_PERIPH0 11
+
+/* PLL_PERIPH0 exported for PRCM */
+
#define CLK_PLL_PERIPH0_2X 12
#define CLK_PLL_PERIPH1 13
#define CLK_PLL_PERIPH1_2X 14
static SUNXI_CCU_GATE(ahb_dma_clk, "ahb-dma", "ahb",
0x060, BIT(6), 0);
static SUNXI_CCU_GATE(ahb_bist_clk, "ahb-bist", "ahb",
- 0x060, BIT(6), 0);
+ 0x060, BIT(7), 0);
static SUNXI_CCU_GATE(ahb_mmc0_clk, "ahb-mmc0", "ahb",
0x060, BIT(8), 0);
static SUNXI_CCU_GATE(ahb_mmc1_clk, "ahb-mmc1", "ahb",
0x12c, 0, 4, 24, 3, BIT(31),
CLK_SET_RATE_PARENT);
static SUNXI_CCU_M_WITH_MUX_GATE(lcd1_ch1_clk, "lcd1-ch1", lcd_ch1_parents,
- 0x12c, 0, 4, 24, 3, BIT(31),
+ 0x130, 0, 4, 24, 3, BIT(31),
CLK_SET_RATE_PARENT);
static const char * const csi_sclk_parents[] = { "pll-video0", "pll-video1",
#define CLK_PLL_VIDEO 6
#define CLK_PLL_VE 7
#define CLK_PLL_DDR 8
-#define CLK_PLL_PERIPH0 9
+
+/* PLL_PERIPH0 exported for PRCM */
+
#define CLK_PLL_PERIPH0_2X 10
#define CLK_PLL_GPU 11
#define CLK_PLL_PERIPH1 12
[RST_BUS_EMAC] = { 0x2c0, BIT(17) },
[RST_BUS_HSTIMER] = { 0x2c0, BIT(19) },
[RST_BUS_SPI0] = { 0x2c0, BIT(20) },
- [RST_BUS_OTG] = { 0x2c0, BIT(23) },
+ [RST_BUS_OTG] = { 0x2c0, BIT(24) },
[RST_BUS_EHCI0] = { 0x2c0, BIT(26) },
[RST_BUS_OHCI0] = { 0x2c0, BIT(29) },
menu "Clock Source drivers"
depends on !ARCH_USES_GETTIMEOFFSET
-config CLKSRC_OF
+config TIMER_OF
bool
- select CLKSRC_PROBE
-
-config CLKEVT_OF
- bool
- select CLKEVT_PROBE
-
-config CLKSRC_ACPI
- bool
- select CLKSRC_PROBE
+ depends on GENERIC_CLOCKEVENTS
+ select TIMER_PROBE
-config CLKSRC_PROBE
+config TIMER_ACPI
bool
+ select TIMER_PROBE
-config CLKEVT_PROBE
+config TIMER_PROBE
bool
config CLKSRC_I8253
config DW_APB_TIMER_OF
bool
select DW_APB_TIMER
- select CLKSRC_OF
+ select TIMER_OF
config FTTMR010_TIMER
bool "Faraday Technology timer driver" if COMPILE_TEST
depends on GENERIC_CLOCKEVENTS
depends on HAS_IOMEM
select CLKSRC_MMIO
- select CLKSRC_OF
+ select TIMER_OF
select MFD_SYSCON
help
Enables support for the Faraday Technology timer block
config ROCKCHIP_TIMER
bool "Rockchip timer driver" if COMPILE_TEST
depends on ARM || ARM64
- select CLKSRC_OF
+ select TIMER_OF
select CLKSRC_MMIO
help
Enables the support for the rockchip timer driver.
config ARMADA_370_XP_TIMER
bool "Armada 370 and XP timer driver" if COMPILE_TEST
depends on ARM
- select CLKSRC_OF
+ select TIMER_OF
select CLKSRC_MMIO
help
Enables the support for the Armada 370 and XP timer driver.
config ORION_TIMER
bool "Orion timer driver" if COMPILE_TEST
depends on ARM
- select CLKSRC_OF
+ select TIMER_OF
select CLKSRC_MMIO
help
Enables the support for the Orion timer driver
depends on GENERIC_CLOCKEVENTS
depends on HAS_IOMEM
select CLKSRC_MMIO
+ select TIMER_OF
help
Enables support for the Sun4i timer.
bool "ASM9260 timer driver" if COMPILE_TEST
depends on GENERIC_CLOCKEVENTS
select CLKSRC_MMIO
- select CLKSRC_OF
+ select TIMER_OF
help
Enables support for the ASM9260 timer.
help
Enables support for the Atlas7 timer.
-config MOXART_TIMER
- bool "Moxart timer driver" if COMPILE_TEST
- depends on GENERIC_CLOCKEVENTS
- select CLKSRC_MMIO
- help
- Enables support for the Moxart timer.
-
config MXS_TIMER
bool "Mxs timer driver" if COMPILE_TEST
depends on GENERIC_CLOCKEVENTS
depends on GENERIC_CLOCKEVENTS && HAS_IOMEM
depends on ARM
select CLKSRC_MMIO
- select CLKSRC_OF
+ select TIMER_OF
help
Support for the LPC32XX clocksource.
config CLKSRC_PISTACHIO
bool "Clocksource for Pistachio SoC" if COMPILE_TEST
depends on HAS_IOMEM
- select CLKSRC_OF
+ select TIMER_OF
help
Enables the clocksource for the Pistachio SoC.
config CLKSRC_TI_32K
bool "Texas Instruments 32.768 Hz Clocksource" if COMPILE_TEST
depends on GENERIC_SCHED_CLOCK
- select CLKSRC_OF if OF
+ select TIMER_OF if OF
help
This option enables support for Texas Instruments 32.768 Hz clocksource
available on many OMAP-like platforms.
bool "NPS400 clocksource driver" if COMPILE_TEST
depends on !PHYS_ADDR_T_64BIT
select CLKSRC_MMIO
- select CLKSRC_OF if OF
+ select TIMER_OF if OF
help
NPS400 clocksource support.
Got 64 bit counter with update rate up to 1000MHz.
bool "Clocksource for MPS2 SoCs" if COMPILE_TEST
depends on GENERIC_SCHED_CLOCK
select CLKSRC_MMIO
- select CLKSRC_OF
+ select TIMER_OF
config ARC_TIMERS
bool "Support for 32-bit TIMERn counters in ARC Cores" if COMPILE_TEST
depends on GENERIC_CLOCKEVENTS
- select CLKSRC_OF
+ select TIMER_OF
help
These are legacy 32-bit TIMER0 and TIMER1 counters found on all ARC cores
(ARC700 as well as ARC HS38).
bool "Support for 64-bit counters in ARC HS38 cores" if COMPILE_TEST
depends on GENERIC_CLOCKEVENTS
depends on ARC_TIMERS
- select CLKSRC_OF
+ select TIMER_OF
help
This enables 2 different 64-bit timers: RTC (for UP) and GFRC (for SMP)
RTC is implemented inside the core, while GFRC sits outside the core in
config ARM_ARCH_TIMER
bool
- select CLKSRC_OF if OF
- select CLKSRC_ACPI if ACPI
+ select TIMER_OF if OF
+ select TIMER_ACPI if ACPI
config ARM_ARCH_TIMER_EVTSTREAM
bool "Enable ARM architected timer event stream generation by default"
config ARM_GLOBAL_TIMER
bool "Support for the ARM global timer" if COMPILE_TEST
- select CLKSRC_OF if OF
+ select TIMER_OF if OF
depends on ARM
help
This options enables support for the ARM global timer unit
bool "Support for Dual Timer SP804 module"
depends on GENERIC_SCHED_CLOCK && CLKDEV_LOOKUP
select CLKSRC_MMIO
- select CLKSRC_OF if OF
+ select TIMER_OF if OF
config CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK
bool
config ARMV7M_SYSTICK
bool "Support for the ARMv7M system time" if COMPILE_TEST
- select CLKSRC_OF if OF
+ select TIMER_OF if OF
select CLKSRC_MMIO
help
This options enables support for the ARMv7M system timer unit
config ATMEL_PIT
- select CLKSRC_OF if OF
+ select TIMER_OF if OF
def_bool SOC_AT91SAM9 || SOC_SAMA5
config ATMEL_ST
bool "Atmel ST timer support" if COMPILE_TEST
depends on GENERIC_CLOCKEVENTS
- select CLKSRC_OF
+ select TIMER_OF
select MFD_SYSCON
help
Support for the Atmel ST timer.
config OXNAS_RPS_TIMER
bool "Oxford Semiconductor OXNAS RPS Timers driver" if COMPILE_TEST
depends on GENERIC_CLOCKEVENTS
- select CLKSRC_OF
+ select TIMER_OF
select CLKSRC_MMIO
help
This enables support for the Oxford Semiconductor OXNAS RPS timers.
config MTK_TIMER
bool "Mediatek timer driver" if COMPILE_TEST
depends on GENERIC_CLOCKEVENTS && HAS_IOMEM
- select CLKSRC_OF
+ select TIMER_OF
select CLKSRC_MMIO
help
Support for Mediatek timer driver.
config CLKSRC_QCOM
bool "Qualcomm MSM timer" if COMPILE_TEST
depends on ARM
- select CLKSRC_OF
+ select TIMER_OF
help
This enables the clocksource and the per CPU clockevent driver for the
Qualcomm SoCs.
config CLKSRC_VERSATILE
bool "ARM Versatile (Express) reference platforms clock source" if COMPILE_TEST
depends on GENERIC_SCHED_CLOCK && !ARCH_USES_GETTIMEOFFSET
- select CLKSRC_OF
+ select TIMER_OF
default y if MFD_VEXPRESS_SYSREG
help
This option enables clock source based on free running
config CLKSRC_MIPS_GIC
bool
depends on MIPS_GIC
- select CLKSRC_OF
+ select TIMER_OF
config CLKSRC_TANGO_XTAL
bool "Clocksource for Tango SoC" if COMPILE_TEST
depends on ARM
- select CLKSRC_OF
+ select TIMER_OF
select CLKSRC_MMIO
help
This enables the clocksource for Tango SoC
config CLKSRC_ST_LPC
bool "Low power clocksource found in the LPC" if COMPILE_TEST
- select CLKSRC_OF if OF
+ select TIMER_OF if OF
depends on HAS_IOMEM
select CLKSRC_MMIO
help
-obj-$(CONFIG_CLKSRC_PROBE) += clksrc-probe.o
-obj-$(CONFIG_CLKEVT_PROBE) += clkevt-probe.o
+obj-$(CONFIG_TIMER_OF) += timer-of.o
+obj-$(CONFIG_TIMER_PROBE) += timer-probe.o
obj-$(CONFIG_ATMEL_PIT) += timer-atmel-pit.o
obj-$(CONFIG_ATMEL_ST) += timer-atmel-st.o
obj-$(CONFIG_ATMEL_TCB_CLKSRC) += tcb_clksrc.o
obj-$(CONFIG_BCM2835_TIMER) += bcm2835_timer.o
obj-$(CONFIG_CLPS711X_TIMER) += clps711x-timer.o
obj-$(CONFIG_ATLAS7_TIMER) += timer-atlas7.o
-obj-$(CONFIG_MOXART_TIMER) += moxart_timer.o
obj-$(CONFIG_MXS_TIMER) += mxs_timer.o
obj-$(CONFIG_CLKSRC_PXA) += pxa_timer.o
obj-$(CONFIG_PRIMA2_TIMER) += timer-prima2.o
return clocksource_register_hz(&arc_counter_gfrc, arc_timer_freq);
}
-CLOCKSOURCE_OF_DECLARE(arc_gfrc, "snps,archs-timer-gfrc", arc_cs_setup_gfrc);
+TIMER_OF_DECLARE(arc_gfrc, "snps,archs-timer-gfrc", arc_cs_setup_gfrc);
#define AUX_RTC_CTRL 0x103
#define AUX_RTC_LOW 0x104
return clocksource_register_hz(&arc_counter_rtc, arc_timer_freq);
}
-CLOCKSOURCE_OF_DECLARE(arc_rtc, "snps,archs-timer-rtc", arc_cs_setup_rtc);
+TIMER_OF_DECLARE(arc_rtc, "snps,archs-timer-rtc", arc_cs_setup_rtc);
#endif
return ret;
}
-CLOCKSOURCE_OF_DECLARE(arc_clkevt, "snps,arc-timer", arc_of_timer_init);
+TIMER_OF_DECLARE(arc_clkevt, "snps,arc-timer", arc_of_timer_init);
return arch_timer_common_init();
}
-CLOCKSOURCE_OF_DECLARE(armv7_arch_timer, "arm,armv7-timer", arch_timer_of_init);
-CLOCKSOURCE_OF_DECLARE(armv8_arch_timer, "arm,armv8-timer", arch_timer_of_init);
+TIMER_OF_DECLARE(armv7_arch_timer, "arm,armv7-timer", arch_timer_of_init);
+TIMER_OF_DECLARE(armv8_arch_timer, "arm,armv8-timer", arch_timer_of_init);
static u32 __init
arch_timer_mem_frame_get_cntfrq(struct arch_timer_mem_frame *frame)
return 0;
}
- rate = readl_relaxed(frame + CNTFRQ);
+ rate = readl_relaxed(base + CNTFRQ);
- iounmap(frame);
+ iounmap(base);
return rate;
}
kfree(timer_mem);
return ret;
}
-CLOCKSOURCE_OF_DECLARE(armv7_arch_timer_mem, "arm,armv7-timer-mem",
+TIMER_OF_DECLARE(armv7_arch_timer_mem, "arm,armv7-timer-mem",
arch_timer_mem_of_init);
#ifdef CONFIG_ACPI_GTDT
return arch_timer_common_init();
}
-CLOCKSOURCE_ACPI_DECLARE(arch_timer, ACPI_SIG_GTDT, arch_timer_acpi_init);
+TIMER_ACPI_DECLARE(arch_timer, ACPI_SIG_GTDT, arch_timer_acpi_init);
#endif
}
/* Only tested on r2p2 and r3p0 */
-CLOCKSOURCE_OF_DECLARE(arm_gt, "arm,cortex-a9-global-timer",
+TIMER_OF_DECLARE(arm_gt, "arm,cortex-a9-global-timer",
global_timer_of_register);
return ret;
}
-CLOCKSOURCE_OF_DECLARE(arm_systick, "arm,armv7m-systick",
+TIMER_OF_DECLARE(arm_systick, "arm,armv7m-systick",
system_timer_of_register);
return 0;
}
-CLOCKSOURCE_OF_DECLARE(asm9260_timer, "alphascale,asm9260-timer",
+TIMER_OF_DECLARE(asm9260_timer, "alphascale,asm9260-timer",
asm9260_timer_init);
iounmap(base);
return ret;
}
-CLOCKSOURCE_OF_DECLARE(bcm2835, "brcm,bcm2835-system-timer",
+TIMER_OF_DECLARE(bcm2835, "brcm,bcm2835-system-timer",
bcm2835_timer_init);
return 0;
}
-CLOCKSOURCE_OF_DECLARE(brcm_kona, "brcm,kona-timer", kona_timer_init);
+TIMER_OF_DECLARE(brcm_kona, "brcm,kona-timer", kona_timer_init);
/*
* bcm,kona-timer is deprecated by brcm,kona-timer
* being kept here for driver compatibility
*/
-CLOCKSOURCE_OF_DECLARE(bcm_kona, "bcm,kona-timer", kona_timer_init);
+TIMER_OF_DECLARE(bcm_kona, "bcm,kona-timer", kona_timer_init);
#include <linux/clk.h>
#include <linux/interrupt.h>
#include <linux/clockchips.h>
+#include <linux/clocksource.h>
#include <linux/of_address.h>
#include <linux/of_irq.h>
#include <linux/slab.h>
return 0;
}
-CLOCKSOURCE_OF_DECLARE(ttc, "cdns,ttc", ttc_timer_init);
+TIMER_OF_DECLARE(ttc, "cdns,ttc", ttc_timer_init);
+++ /dev/null
-/*
- * Copyright (c) 2016, Linaro Ltd. All rights reserved.
- * Daniel Lezcano <daniel.lezcano@linaro.org>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms and conditions of the GNU General Public License,
- * version 2, as published by the Free Software Foundation.
- *
- * This program is distributed in the hope it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
- * more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program. If not, see <http://www.gnu.org/licenses/>.
- */
-
-#include <linux/init.h>
-#include <linux/of.h>
-#include <linux/clockchips.h>
-
-extern struct of_device_id __clkevt_of_table[];
-
-static const struct of_device_id __clkevt_of_table_sentinel
- __used __section(__clkevt_of_table_end);
-
-int __init clockevent_probe(void)
-{
- struct device_node *np;
- const struct of_device_id *match;
- of_init_fn_1_ret init_func;
- int ret, clockevents = 0;
-
- for_each_matching_node_and_match(np, __clkevt_of_table, &match) {
- if (!of_device_is_available(np))
- continue;
-
- init_func = match->data;
-
- ret = init_func(np);
- if (ret) {
- pr_warn("Failed to initialize '%s' (%d)\n",
- np->name, ret);
- continue;
- }
-
- clockevents++;
- }
-
- if (!clockevents) {
- pr_crit("%s: no matching clockevent found\n", __func__);
- return -ENODEV;
- }
-
- return 0;
-}
#endif
return clocksource_register_hz(&clocksource_dbx500_prcmu, RATE_32K);
}
-CLOCKSOURCE_OF_DECLARE(dbx500_prcmu, "stericsson,db8500-prcmu-timer-4",
+TIMER_OF_DECLARE(dbx500_prcmu, "stericsson,db8500-prcmu-timer-4",
clksrc_dbx500_prcmu_init);
return ret;
}
-CLOCKSOURCE_OF_DECLARE(ddata, "st,stih407-lpc", st_clksrc_of_register);
+TIMER_OF_DECLARE(ddata, "st,stih407-lpc", st_clksrc_of_register);
BUG_ON(_clps711x_clkevt_init(tc2, tc2_base, irq));
}
-#ifdef CONFIG_CLKSRC_OF
+#ifdef CONFIG_TIMER_OF
static int __init clps711x_timer_init(struct device_node *np)
{
unsigned int irq = irq_of_parse_and_map(np, 0);
return -EINVAL;
}
}
-CLOCKSOURCE_OF_DECLARE(clps711x, "cirrus,ep7209-timer", clps711x_timer_init);
+TIMER_OF_DECLARE(clps711x, "cirrus,ep7209-timer", clps711x_timer_init);
#endif
return 0;
}
-CLOCKSOURCE_OF_DECLARE(pc3x2_timer, "picochip,pc3x2-timer", dw_apb_timer_init);
-CLOCKSOURCE_OF_DECLARE(apb_timer_osc, "snps,dw-apb-timer-osc", dw_apb_timer_init);
-CLOCKSOURCE_OF_DECLARE(apb_timer_sp, "snps,dw-apb-timer-sp", dw_apb_timer_init);
-CLOCKSOURCE_OF_DECLARE(apb_timer, "snps,dw-apb-timer", dw_apb_timer_init);
+TIMER_OF_DECLARE(pc3x2_timer, "picochip,pc3x2-timer", dw_apb_timer_init);
+TIMER_OF_DECLARE(apb_timer_osc, "snps,dw-apb-timer-osc", dw_apb_timer_init);
+TIMER_OF_DECLARE(apb_timer_sp, "snps,dw-apb-timer-sp", dw_apb_timer_init);
+TIMER_OF_DECLARE(apb_timer, "snps,dw-apb-timer", dw_apb_timer_init);
{
return mct_init_dt(np, MCT_INT_PPI);
}
-CLOCKSOURCE_OF_DECLARE(exynos4210, "samsung,exynos4210-mct", mct_init_spi);
-CLOCKSOURCE_OF_DECLARE(exynos4412, "samsung,exynos4412-mct", mct_init_ppi);
+TIMER_OF_DECLARE(exynos4210, "samsung,exynos4210-mct", mct_init_spi);
+TIMER_OF_DECLARE(exynos4412, "samsung,exynos4412-mct", mct_init_ppi);
priv->clkevt_base = of_iomap(np, 0);
if (!priv->clkevt_base) {
pr_err("ftm: unable to map event timer registers\n");
- goto err;
+ goto err_clkevt;
}
priv->clksrc_base = of_iomap(np, 1);
if (!priv->clksrc_base) {
pr_err("ftm: unable to map source timer registers\n");
- goto err;
+ goto err_clksrc;
}
ret = -EINVAL;
return 0;
err:
+ iounmap(priv->clksrc_base);
+err_clksrc:
+ iounmap(priv->clkevt_base);
+err_clkevt:
kfree(priv);
return ret;
}
-CLOCKSOURCE_OF_DECLARE(flextimer, "fsl,ftm-timer", ftm_timer_init);
+TIMER_OF_DECLARE(flextimer, "fsl,ftm-timer", ftm_timer_init);
return ret;
}
-CLOCKSOURCE_OF_DECLARE(h8300_16bit, "renesas,16bit-timer",
+TIMER_OF_DECLARE(h8300_16bit, "renesas,16bit-timer",
h8300_16timer_init);
return ret;
}
-CLOCKSOURCE_OF_DECLARE(h8300_8bit, "renesas,8bit-timer", h8300_8timer_init);
+TIMER_OF_DECLARE(h8300_8bit, "renesas,8bit-timer", h8300_8timer_init);
return ret;
}
-CLOCKSOURCE_OF_DECLARE(h8300_tpu, "renesas,tpu", h8300_tpu_init);
+TIMER_OF_DECLARE(h8300_tpu, "renesas,tpu", h8300_tpu_init);
return 0;
}
-CLOCKSOURCE_OF_DECLARE(jcore_pit, "jcore,pit", jcore_pit_init);
+TIMER_OF_DECLARE(jcore_pit, "jcore,pit", jcore_pit_init);
1, 0xfffe);
return 0;
}
-CLOCKSOURCE_OF_DECLARE(meson6, "amlogic,meson6-timer",
+TIMER_OF_DECLARE(meson6, "amlogic,meson6-timer",
meson6_timer_init);
clk = of_clk_get(node, 0);
if (!IS_ERR(clk)) {
- if (clk_prepare_enable(clk) < 0) {
+ ret = clk_prepare_enable(clk);
+ if (ret < 0) {
pr_err("GIC failed to enable clock\n");
clk_put(clk);
- return PTR_ERR(clk);
+ return ret;
}
gic_frequency = clk_get_rate(clk);
return 0;
}
-CLOCKSOURCE_OF_DECLARE(mips_gic_timer, "mti,gic-timer",
+TIMER_OF_DECLARE(mips_gic_timer, "mti,gic-timer",
gic_clocksource_of_init);
+++ /dev/null
-/*
- * MOXA ART SoCs timer handling.
- *
- * Copyright (C) 2013 Jonas Jensen
- *
- * Jonas Jensen <jonas.jensen@gmail.com>
- *
- * This file is licensed under the terms of the GNU General Public
- * License version 2. This program is licensed "as is" without any
- * warranty of any kind, whether express or implied.
- */
-
-#include <linux/clk.h>
-#include <linux/clockchips.h>
-#include <linux/interrupt.h>
-#include <linux/irq.h>
-#include <linux/irqreturn.h>
-#include <linux/of.h>
-#include <linux/of_address.h>
-#include <linux/of_irq.h>
-#include <linux/io.h>
-#include <linux/clocksource.h>
-#include <linux/bitops.h>
-#include <linux/slab.h>
-
-#define TIMER1_BASE 0x00
-#define TIMER2_BASE 0x10
-#define TIMER3_BASE 0x20
-
-#define REG_COUNT 0x0 /* writable */
-#define REG_LOAD 0x4
-#define REG_MATCH1 0x8
-#define REG_MATCH2 0xC
-
-#define TIMER_CR 0x30
-#define TIMER_INTR_STATE 0x34
-#define TIMER_INTR_MASK 0x38
-
-/*
- * Moxart TIMER_CR flags:
- *
- * MOXART_CR_*_CLOCK 0: PCLK, 1: EXT1CLK
- * MOXART_CR_*_INT overflow interrupt enable bit
- */
-#define MOXART_CR_1_ENABLE BIT(0)
-#define MOXART_CR_1_CLOCK BIT(1)
-#define MOXART_CR_1_INT BIT(2)
-#define MOXART_CR_2_ENABLE BIT(3)
-#define MOXART_CR_2_CLOCK BIT(4)
-#define MOXART_CR_2_INT BIT(5)
-#define MOXART_CR_3_ENABLE BIT(6)
-#define MOXART_CR_3_CLOCK BIT(7)
-#define MOXART_CR_3_INT BIT(8)
-#define MOXART_CR_COUNT_UP BIT(9)
-
-#define MOXART_TIMER1_ENABLE (MOXART_CR_2_ENABLE | MOXART_CR_1_ENABLE)
-#define MOXART_TIMER1_DISABLE (MOXART_CR_2_ENABLE)
-
-/*
- * The ASpeed variant of the IP block has a different layout
- * for the control register
- */
-#define ASPEED_CR_1_ENABLE BIT(0)
-#define ASPEED_CR_1_CLOCK BIT(1)
-#define ASPEED_CR_1_INT BIT(2)
-#define ASPEED_CR_2_ENABLE BIT(4)
-#define ASPEED_CR_2_CLOCK BIT(5)
-#define ASPEED_CR_2_INT BIT(6)
-#define ASPEED_CR_3_ENABLE BIT(8)
-#define ASPEED_CR_3_CLOCK BIT(9)
-#define ASPEED_CR_3_INT BIT(10)
-
-#define ASPEED_TIMER1_ENABLE (ASPEED_CR_2_ENABLE | ASPEED_CR_1_ENABLE)
-#define ASPEED_TIMER1_DISABLE (ASPEED_CR_2_ENABLE)
-
-struct moxart_timer {
- void __iomem *base;
- unsigned int t1_disable_val;
- unsigned int t1_enable_val;
- unsigned int count_per_tick;
- struct clock_event_device clkevt;
-};
-
-static inline struct moxart_timer *to_moxart(struct clock_event_device *evt)
-{
- return container_of(evt, struct moxart_timer, clkevt);
-}
-
-static inline void moxart_disable(struct clock_event_device *evt)
-{
- struct moxart_timer *timer = to_moxart(evt);
-
- writel(timer->t1_disable_val, timer->base + TIMER_CR);
-}
-
-static inline void moxart_enable(struct clock_event_device *evt)
-{
- struct moxart_timer *timer = to_moxart(evt);
-
- writel(timer->t1_enable_val, timer->base + TIMER_CR);
-}
-
-static int moxart_shutdown(struct clock_event_device *evt)
-{
- moxart_disable(evt);
- return 0;
-}
-
-static int moxart_set_oneshot(struct clock_event_device *evt)
-{
- moxart_disable(evt);
- writel(~0, to_moxart(evt)->base + TIMER1_BASE + REG_LOAD);
- return 0;
-}
-
-static int moxart_set_periodic(struct clock_event_device *evt)
-{
- struct moxart_timer *timer = to_moxart(evt);
-
- moxart_disable(evt);
- writel(timer->count_per_tick, timer->base + TIMER1_BASE + REG_LOAD);
- writel(0, timer->base + TIMER1_BASE + REG_MATCH1);
- moxart_enable(evt);
- return 0;
-}
-
-static int moxart_clkevt_next_event(unsigned long cycles,
- struct clock_event_device *evt)
-{
- struct moxart_timer *timer = to_moxart(evt);
- u32 u;
-
- moxart_disable(evt);
-
- u = readl(timer->base + TIMER1_BASE + REG_COUNT) - cycles;
- writel(u, timer->base + TIMER1_BASE + REG_MATCH1);
-
- moxart_enable(evt);
-
- return 0;
-}
-
-static irqreturn_t moxart_timer_interrupt(int irq, void *dev_id)
-{
- struct clock_event_device *evt = dev_id;
- evt->event_handler(evt);
- return IRQ_HANDLED;
-}
-
-static int __init moxart_timer_init(struct device_node *node)
-{
- int ret, irq;
- unsigned long pclk;
- struct clk *clk;
- struct moxart_timer *timer;
-
- timer = kzalloc(sizeof(*timer), GFP_KERNEL);
- if (!timer)
- return -ENOMEM;
-
- timer->base = of_iomap(node, 0);
- if (!timer->base) {
- pr_err("%s: of_iomap failed\n", node->full_name);
- ret = -ENXIO;
- goto out_free;
- }
-
- irq = irq_of_parse_and_map(node, 0);
- if (irq <= 0) {
- pr_err("%s: irq_of_parse_and_map failed\n", node->full_name);
- ret = -EINVAL;
- goto out_unmap;
- }
-
- clk = of_clk_get(node, 0);
- if (IS_ERR(clk)) {
- pr_err("%s: of_clk_get failed\n", node->full_name);
- ret = PTR_ERR(clk);
- goto out_unmap;
- }
-
- pclk = clk_get_rate(clk);
-
- if (of_device_is_compatible(node, "moxa,moxart-timer")) {
- timer->t1_enable_val = MOXART_TIMER1_ENABLE;
- timer->t1_disable_val = MOXART_TIMER1_DISABLE;
- } else if (of_device_is_compatible(node, "aspeed,ast2400-timer")) {
- timer->t1_enable_val = ASPEED_TIMER1_ENABLE;
- timer->t1_disable_val = ASPEED_TIMER1_DISABLE;
- } else {
- pr_err("%s: unknown platform\n", node->full_name);
- ret = -EINVAL;
- goto out_unmap;
- }
-
- timer->count_per_tick = DIV_ROUND_CLOSEST(pclk, HZ);
-
- timer->clkevt.name = node->name;
- timer->clkevt.rating = 200;
- timer->clkevt.features = CLOCK_EVT_FEAT_PERIODIC |
- CLOCK_EVT_FEAT_ONESHOT;
- timer->clkevt.set_state_shutdown = moxart_shutdown;
- timer->clkevt.set_state_periodic = moxart_set_periodic;
- timer->clkevt.set_state_oneshot = moxart_set_oneshot;
- timer->clkevt.tick_resume = moxart_set_oneshot;
- timer->clkevt.set_next_event = moxart_clkevt_next_event;
- timer->clkevt.cpumask = cpumask_of(0);
- timer->clkevt.irq = irq;
-
- ret = clocksource_mmio_init(timer->base + TIMER2_BASE + REG_COUNT,
- "moxart_timer", pclk, 200, 32,
- clocksource_mmio_readl_down);
- if (ret) {
- pr_err("%s: clocksource_mmio_init failed\n", node->full_name);
- goto out_unmap;
- }
-
- ret = request_irq(irq, moxart_timer_interrupt, IRQF_TIMER,
- node->name, &timer->clkevt);
- if (ret) {
- pr_err("%s: setup_irq failed\n", node->full_name);
- goto out_unmap;
- }
-
- /* Clear match registers */
- writel(0, timer->base + TIMER1_BASE + REG_MATCH1);
- writel(0, timer->base + TIMER1_BASE + REG_MATCH2);
- writel(0, timer->base + TIMER2_BASE + REG_MATCH1);
- writel(0, timer->base + TIMER2_BASE + REG_MATCH2);
-
- /*
- * Start timer 2 rolling as our main wall clock source, keep timer 1
- * disabled
- */
- writel(0, timer->base + TIMER_CR);
- writel(~0, timer->base + TIMER2_BASE + REG_LOAD);
- writel(timer->t1_disable_val, timer->base + TIMER_CR);
-
- /*
- * documentation is not publicly available:
- * min_delta / max_delta obtained by trial-and-error,
- * max_delta 0xfffffffe should be ok because count
- * register size is u32
- */
- clockevents_config_and_register(&timer->clkevt, pclk, 0x4, 0xfffffffe);
-
- return 0;
-
-out_unmap:
- iounmap(timer->base);
-out_free:
- kfree(timer);
- return ret;
-}
-CLOCKSOURCE_OF_DECLARE(moxart, "moxa,moxart-timer", moxart_timer_init);
-CLOCKSOURCE_OF_DECLARE(aspeed, "aspeed,ast2400-timer", moxart_timer_init);
return 0;
}
-CLOCKSOURCE_OF_DECLARE(mps2_timer, "arm,mps2-timer", mps2_timer_init);
+TIMER_OF_DECLARE(mps2_timer, "arm,mps2-timer", mps2_timer_init);
return -EINVAL;
}
-CLOCKSOURCE_OF_DECLARE(mtk_mt6577, "mediatek,mt6577-timer", mtk_timer_init);
+TIMER_OF_DECLARE(mtk_mt6577, "mediatek,mt6577-timer", mtk_timer_init);
return setup_irq(irq, &mxs_timer_irq);
}
-CLOCKSOURCE_OF_DECLARE(mxs, "fsl,timrot", mxs_timer_init);
+TIMER_OF_DECLARE(mxs, "fsl,timrot", mxs_timer_init);
return nmdk_timer_init(base, irq, pclk, clk);
}
-CLOCKSOURCE_OF_DECLARE(nomadik_mtu, "st,nomadik-mtu",
+TIMER_OF_DECLARE(nomadik_mtu, "st,nomadik-mtu",
nmdk_timer_of_init);
return pxa_timer_common_init(irq, clk_get_rate(clk));
}
-CLOCKSOURCE_OF_DECLARE(pxa_timer, "marvell,pxa-timer", pxa_timer_dt_init);
+TIMER_OF_DECLARE(pxa_timer, "marvell,pxa-timer", pxa_timer_dt_init);
/*
* Legacy timer init for non device-tree boards.
return msm_timer_init(freq, 32, irq, !!percpu_offset);
}
-CLOCKSOURCE_OF_DECLARE(kpss_timer, "qcom,kpss-timer", msm_dt_timer_init);
-CLOCKSOURCE_OF_DECLARE(scss_timer, "qcom,scss-timer", msm_dt_timer_init);
+TIMER_OF_DECLARE(kpss_timer, "qcom,kpss-timer", msm_dt_timer_init);
+TIMER_OF_DECLARE(scss_timer, "qcom,scss-timer", msm_dt_timer_init);
return 0;
}
-CLOCKSOURCE_OF_DECLARE(ostm, "renesas,ostm", ostm_init);
+TIMER_OF_DECLARE(ostm, "renesas,ostm", ostm_init);
return -EINVAL;
}
-CLOCKSOURCE_OF_DECLARE(rk3288_timer, "rockchip,rk3288-timer", rk_timer_init);
-CLOCKSOURCE_OF_DECLARE(rk3399_timer, "rockchip,rk3399-timer", rk_timer_init);
+TIMER_OF_DECLARE(rk3288_timer, "rockchip,rk3288-timer", rk_timer_init);
+TIMER_OF_DECLARE(rk3399_timer, "rockchip,rk3399-timer", rk_timer_init);
_samsung_pwm_clocksource_init();
}
-#ifdef CONFIG_CLKSRC_OF
+#ifdef CONFIG_TIMER_OF
static int __init samsung_pwm_alloc(struct device_node *np,
const struct samsung_pwm_variant *variant)
{
{
return samsung_pwm_alloc(np, &s3c24xx_variant);
}
-CLOCKSOURCE_OF_DECLARE(s3c2410_pwm, "samsung,s3c2410-pwm", s3c2410_pwm_clocksource_init);
+TIMER_OF_DECLARE(s3c2410_pwm, "samsung,s3c2410-pwm", s3c2410_pwm_clocksource_init);
static const struct samsung_pwm_variant s3c64xx_variant = {
.bits = 32,
{
return samsung_pwm_alloc(np, &s3c64xx_variant);
}
-CLOCKSOURCE_OF_DECLARE(s3c6400_pwm, "samsung,s3c6400-pwm", s3c64xx_pwm_clocksource_init);
+TIMER_OF_DECLARE(s3c6400_pwm, "samsung,s3c6400-pwm", s3c64xx_pwm_clocksource_init);
static const struct samsung_pwm_variant s5p64x0_variant = {
.bits = 32,
{
return samsung_pwm_alloc(np, &s5p64x0_variant);
}
-CLOCKSOURCE_OF_DECLARE(s5p6440_pwm, "samsung,s5p6440-pwm", s5p64x0_pwm_clocksource_init);
+TIMER_OF_DECLARE(s5p6440_pwm, "samsung,s5p6440-pwm", s5p64x0_pwm_clocksource_init);
static const struct samsung_pwm_variant s5p_variant = {
.bits = 32,
{
return samsung_pwm_alloc(np, &s5p_variant);
}
-CLOCKSOURCE_OF_DECLARE(s5pc100_pwm, "samsung,s5pc100-pwm", s5p_pwm_clocksource_init);
+TIMER_OF_DECLARE(s5pc100_pwm, "samsung,s5pc100-pwm", s5p_pwm_clocksource_init);
#endif
#include <linux/of_address.h>
#include <linux/of_irq.h>
+#include "timer-of.h"
+
#define TIMER_IRQ_EN_REG 0x00
#define TIMER_IRQ_EN(val) BIT(val)
#define TIMER_IRQ_ST_REG 0x04
#define TIMER_SYNC_TICKS 3
-static void __iomem *timer_base;
-static u32 ticks_per_jiffy;
-
/*
* When we disable a timer, we need to wait at least for 2 cycles of
* the timer source clock. We will use for that the clocksource timer
* that is already setup and runs at the same frequency than the other
* timers, and we never will be disabled.
*/
-static void sun4i_clkevt_sync(void)
+static void sun4i_clkevt_sync(void __iomem *base)
{
- u32 old = readl(timer_base + TIMER_CNTVAL_REG(1));
+ u32 old = readl(base + TIMER_CNTVAL_REG(1));
- while ((old - readl(timer_base + TIMER_CNTVAL_REG(1))) < TIMER_SYNC_TICKS)
+ while ((old - readl(base + TIMER_CNTVAL_REG(1))) < TIMER_SYNC_TICKS)
cpu_relax();
}
-static void sun4i_clkevt_time_stop(u8 timer)
+static void sun4i_clkevt_time_stop(void __iomem *base, u8 timer)
{
- u32 val = readl(timer_base + TIMER_CTL_REG(timer));
- writel(val & ~TIMER_CTL_ENABLE, timer_base + TIMER_CTL_REG(timer));
- sun4i_clkevt_sync();
+ u32 val = readl(base + TIMER_CTL_REG(timer));
+ writel(val & ~TIMER_CTL_ENABLE, base + TIMER_CTL_REG(timer));
+ sun4i_clkevt_sync(base);
}
-static void sun4i_clkevt_time_setup(u8 timer, unsigned long delay)
+static void sun4i_clkevt_time_setup(void __iomem *base, u8 timer,
+ unsigned long delay)
{
- writel(delay, timer_base + TIMER_INTVAL_REG(timer));
+ writel(delay, base + TIMER_INTVAL_REG(timer));
}
-static void sun4i_clkevt_time_start(u8 timer, bool periodic)
+static void sun4i_clkevt_time_start(void __iomem *base, u8 timer,
+ bool periodic)
{
- u32 val = readl(timer_base + TIMER_CTL_REG(timer));
+ u32 val = readl(base + TIMER_CTL_REG(timer));
if (periodic)
val &= ~TIMER_CTL_ONESHOT;
val |= TIMER_CTL_ONESHOT;
writel(val | TIMER_CTL_ENABLE | TIMER_CTL_RELOAD,
- timer_base + TIMER_CTL_REG(timer));
+ base + TIMER_CTL_REG(timer));
}
static int sun4i_clkevt_shutdown(struct clock_event_device *evt)
{
- sun4i_clkevt_time_stop(0);
+ struct timer_of *to = to_timer_of(evt);
+
+ sun4i_clkevt_time_stop(timer_of_base(to), 0);
+
return 0;
}
static int sun4i_clkevt_set_oneshot(struct clock_event_device *evt)
{
- sun4i_clkevt_time_stop(0);
- sun4i_clkevt_time_start(0, false);
+ struct timer_of *to = to_timer_of(evt);
+
+ sun4i_clkevt_time_stop(timer_of_base(to), 0);
+ sun4i_clkevt_time_start(timer_of_base(to), 0, false);
+
return 0;
}
static int sun4i_clkevt_set_periodic(struct clock_event_device *evt)
{
- sun4i_clkevt_time_stop(0);
- sun4i_clkevt_time_setup(0, ticks_per_jiffy);
- sun4i_clkevt_time_start(0, true);
+ struct timer_of *to = to_timer_of(evt);
+
+ sun4i_clkevt_time_stop(timer_of_base(to), 0);
+ sun4i_clkevt_time_setup(timer_of_base(to), 0, timer_of_period(to));
+ sun4i_clkevt_time_start(timer_of_base(to), 0, true);
+
return 0;
}
static int sun4i_clkevt_next_event(unsigned long evt,
- struct clock_event_device *unused)
+ struct clock_event_device *clkevt)
{
- sun4i_clkevt_time_stop(0);
- sun4i_clkevt_time_setup(0, evt - TIMER_SYNC_TICKS);
- sun4i_clkevt_time_start(0, false);
+ struct timer_of *to = to_timer_of(clkevt);
+
+ sun4i_clkevt_time_stop(timer_of_base(to), 0);
+ sun4i_clkevt_time_setup(timer_of_base(to), 0, evt - TIMER_SYNC_TICKS);
+ sun4i_clkevt_time_start(timer_of_base(to), 0, false);
return 0;
}
-static struct clock_event_device sun4i_clockevent = {
- .name = "sun4i_tick",
- .rating = 350,
- .features = CLOCK_EVT_FEAT_PERIODIC | CLOCK_EVT_FEAT_ONESHOT,
- .set_state_shutdown = sun4i_clkevt_shutdown,
- .set_state_periodic = sun4i_clkevt_set_periodic,
- .set_state_oneshot = sun4i_clkevt_set_oneshot,
- .tick_resume = sun4i_clkevt_shutdown,
- .set_next_event = sun4i_clkevt_next_event,
-};
-
-static void sun4i_timer_clear_interrupt(void)
+static void sun4i_timer_clear_interrupt(void __iomem *base)
{
- writel(TIMER_IRQ_EN(0), timer_base + TIMER_IRQ_ST_REG);
+ writel(TIMER_IRQ_EN(0), base + TIMER_IRQ_ST_REG);
}
static irqreturn_t sun4i_timer_interrupt(int irq, void *dev_id)
{
struct clock_event_device *evt = (struct clock_event_device *)dev_id;
+ struct timer_of *to = to_timer_of(evt);
- sun4i_timer_clear_interrupt();
+ sun4i_timer_clear_interrupt(timer_of_base(to));
evt->event_handler(evt);
return IRQ_HANDLED;
}
-static struct irqaction sun4i_timer_irq = {
- .name = "sun4i_timer0",
- .flags = IRQF_TIMER | IRQF_IRQPOLL,
- .handler = sun4i_timer_interrupt,
- .dev_id = &sun4i_clockevent,
+static struct timer_of to = {
+ .flags = TIMER_OF_IRQ | TIMER_OF_CLOCK | TIMER_OF_BASE,
+
+ .clkevt = {
+ .name = "sun4i_tick",
+ .rating = 350,
+ .features = CLOCK_EVT_FEAT_PERIODIC | CLOCK_EVT_FEAT_ONESHOT,
+ .set_state_shutdown = sun4i_clkevt_shutdown,
+ .set_state_periodic = sun4i_clkevt_set_periodic,
+ .set_state_oneshot = sun4i_clkevt_set_oneshot,
+ .tick_resume = sun4i_clkevt_shutdown,
+ .set_next_event = sun4i_clkevt_next_event,
+ .cpumask = cpu_possible_mask,
+ },
+
+ .of_irq = {
+ .handler = sun4i_timer_interrupt,
+ .flags = IRQF_TIMER | IRQF_IRQPOLL,
+ },
};
static u64 notrace sun4i_timer_sched_read(void)
{
- return ~readl(timer_base + TIMER_CNTVAL_REG(1));
+ return ~readl(timer_of_base(&to) + TIMER_CNTVAL_REG(1));
}
static int __init sun4i_timer_init(struct device_node *node)
{
- unsigned long rate = 0;
- struct clk *clk;
- int ret, irq;
+ int ret;
u32 val;
- timer_base = of_iomap(node, 0);
- if (!timer_base) {
- pr_crit("Can't map registers\n");
- return -ENXIO;
- }
-
- irq = irq_of_parse_and_map(node, 0);
- if (irq <= 0) {
- pr_crit("Can't parse IRQ\n");
- return -EINVAL;
- }
-
- clk = of_clk_get(node, 0);
- if (IS_ERR(clk)) {
- pr_crit("Can't get timer clock\n");
- return PTR_ERR(clk);
- }
-
- ret = clk_prepare_enable(clk);
- if (ret) {
- pr_err("Failed to prepare clock\n");
+ ret = timer_of_init(node, &to);
+ if (ret)
return ret;
- }
-
- rate = clk_get_rate(clk);
- writel(~0, timer_base + TIMER_INTVAL_REG(1));
+ writel(~0, timer_of_base(&to) + TIMER_INTVAL_REG(1));
writel(TIMER_CTL_ENABLE | TIMER_CTL_RELOAD |
TIMER_CTL_CLK_SRC(TIMER_CTL_CLK_SRC_OSC24M),
- timer_base + TIMER_CTL_REG(1));
+ timer_of_base(&to) + TIMER_CTL_REG(1));
/*
* sched_clock_register does not have priorities, and on sun6i and
if (of_machine_is_compatible("allwinner,sun4i-a10") ||
of_machine_is_compatible("allwinner,sun5i-a13") ||
of_machine_is_compatible("allwinner,sun5i-a10s"))
- sched_clock_register(sun4i_timer_sched_read, 32, rate);
+ sched_clock_register(sun4i_timer_sched_read, 32,
+ timer_of_rate(&to));
- ret = clocksource_mmio_init(timer_base + TIMER_CNTVAL_REG(1), node->name,
- rate, 350, 32, clocksource_mmio_readl_down);
+ ret = clocksource_mmio_init(timer_of_base(&to) + TIMER_CNTVAL_REG(1),
+ node->name, timer_of_rate(&to), 350, 32,
+ clocksource_mmio_readl_down);
if (ret) {
pr_err("Failed to register clocksource\n");
return ret;
}
- ticks_per_jiffy = DIV_ROUND_UP(rate, HZ);
-
writel(TIMER_CTL_CLK_SRC(TIMER_CTL_CLK_SRC_OSC24M),
- timer_base + TIMER_CTL_REG(0));
+ timer_of_base(&to) + TIMER_CTL_REG(0));
/* Make sure timer is stopped before playing with interrupts */
- sun4i_clkevt_time_stop(0);
+ sun4i_clkevt_time_stop(timer_of_base(&to), 0);
/* clear timer0 interrupt */
- sun4i_timer_clear_interrupt();
-
- sun4i_clockevent.cpumask = cpu_possible_mask;
- sun4i_clockevent.irq = irq;
+ sun4i_timer_clear_interrupt(timer_of_base(&to));
- clockevents_config_and_register(&sun4i_clockevent, rate,
+ clockevents_config_and_register(&to.clkevt, timer_of_rate(&to),
TIMER_SYNC_TICKS, 0xffffffff);
- ret = setup_irq(irq, &sun4i_timer_irq);
- if (ret) {
- pr_err("failed to setup irq %d\n", irq);
- return ret;
- }
-
/* Enable timer0 interrupt */
- val = readl(timer_base + TIMER_IRQ_EN_REG);
- writel(val | TIMER_IRQ_EN(0), timer_base + TIMER_IRQ_EN_REG);
+ val = readl(timer_of_base(&to) + TIMER_IRQ_EN_REG);
+ writel(val | TIMER_IRQ_EN(0), timer_of_base(&to) + TIMER_IRQ_EN_REG);
return ret;
}
-CLOCKSOURCE_OF_DECLARE(sun4i, "allwinner,sun4i-a10-timer",
+TIMER_OF_DECLARE(sun4i, "allwinner,sun4i-a10-timer",
sun4i_timer_init);
return 0;
}
-CLOCKSOURCE_OF_DECLARE(tango, "sigma,tick-counter", tango_clocksource_init);
+TIMER_OF_DECLARE(tango, "sigma,tick-counter", tango_clocksource_init);
#include <linux/ioport.h>
#include <linux/io.h>
#include <linux/platform_device.h>
+#include <linux/syscore_ops.h>
#include <linux/atmel_tc.h>
*/
static void __iomem *tcaddr;
+static struct
+{
+ u32 cmr;
+ u32 imr;
+ u32 rc;
+ bool clken;
+} tcb_cache[3];
+static u32 bmr_cache;
static u64 tc_get_cycles(struct clocksource *cs)
{
raw_local_irq_save(flags);
do {
- upper = __raw_readl(tcaddr + ATMEL_TC_REG(1, CV));
- lower = __raw_readl(tcaddr + ATMEL_TC_REG(0, CV));
- } while (upper != __raw_readl(tcaddr + ATMEL_TC_REG(1, CV)));
+ upper = readl_relaxed(tcaddr + ATMEL_TC_REG(1, CV));
+ lower = readl_relaxed(tcaddr + ATMEL_TC_REG(0, CV));
+ } while (upper != readl_relaxed(tcaddr + ATMEL_TC_REG(1, CV)));
raw_local_irq_restore(flags);
return (upper << 16) | lower;
static u64 tc_get_cycles32(struct clocksource *cs)
{
- return __raw_readl(tcaddr + ATMEL_TC_REG(0, CV));
+ return readl_relaxed(tcaddr + ATMEL_TC_REG(0, CV));
+}
+
+void tc_clksrc_suspend(struct clocksource *cs)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(tcb_cache); i++) {
+ tcb_cache[i].cmr = readl(tcaddr + ATMEL_TC_REG(i, CMR));
+ tcb_cache[i].imr = readl(tcaddr + ATMEL_TC_REG(i, IMR));
+ tcb_cache[i].rc = readl(tcaddr + ATMEL_TC_REG(i, RC));
+ tcb_cache[i].clken = !!(readl(tcaddr + ATMEL_TC_REG(i, SR)) &
+ ATMEL_TC_CLKSTA);
+ }
+
+ bmr_cache = readl(tcaddr + ATMEL_TC_BMR);
+}
+
+void tc_clksrc_resume(struct clocksource *cs)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(tcb_cache); i++) {
+ /* Restore registers for the channel, RA and RB are not used */
+ writel(tcb_cache[i].cmr, tcaddr + ATMEL_TC_REG(i, CMR));
+ writel(tcb_cache[i].rc, tcaddr + ATMEL_TC_REG(i, RC));
+ writel(0, tcaddr + ATMEL_TC_REG(i, RA));
+ writel(0, tcaddr + ATMEL_TC_REG(i, RB));
+ /* Disable all the interrupts */
+ writel(0xff, tcaddr + ATMEL_TC_REG(i, IDR));
+ /* Reenable interrupts that were enabled before suspending */
+ writel(tcb_cache[i].imr, tcaddr + ATMEL_TC_REG(i, IER));
+ /* Start the clock if it was used */
+ if (tcb_cache[i].clken)
+ writel(ATMEL_TC_CLKEN, tcaddr + ATMEL_TC_REG(i, CCR));
+ }
+
+ /* Dual channel, chain channels */
+ writel(bmr_cache, tcaddr + ATMEL_TC_BMR);
+ /* Finally, trigger all the channels*/
+ writel(ATMEL_TC_SYNC, tcaddr + ATMEL_TC_BCR);
}
static struct clocksource clksrc = {
.read = tc_get_cycles,
.mask = CLOCKSOURCE_MASK(32),
.flags = CLOCK_SOURCE_IS_CONTINUOUS,
+ .suspend = tc_clksrc_suspend,
+ .resume = tc_clksrc_resume,
};
#ifdef CONFIG_GENERIC_CLOCKEVENTS
struct tc_clkevt_device *tcd = to_tc_clkevt(d);
void __iomem *regs = tcd->regs;
- __raw_writel(0xff, regs + ATMEL_TC_REG(2, IDR));
- __raw_writel(ATMEL_TC_CLKDIS, regs + ATMEL_TC_REG(2, CCR));
+ writel(0xff, regs + ATMEL_TC_REG(2, IDR));
+ writel(ATMEL_TC_CLKDIS, regs + ATMEL_TC_REG(2, CCR));
if (!clockevent_state_detached(d))
clk_disable(tcd->clk);
clk_enable(tcd->clk);
/* slow clock, count up to RC, then irq and stop */
- __raw_writel(timer_clock | ATMEL_TC_CPCSTOP | ATMEL_TC_WAVE |
+ writel(timer_clock | ATMEL_TC_CPCSTOP | ATMEL_TC_WAVE |
ATMEL_TC_WAVESEL_UP_AUTO, regs + ATMEL_TC_REG(2, CMR));
- __raw_writel(ATMEL_TC_CPCS, regs + ATMEL_TC_REG(2, IER));
+ writel(ATMEL_TC_CPCS, regs + ATMEL_TC_REG(2, IER));
/* set_next_event() configures and starts the timer */
return 0;
clk_enable(tcd->clk);
/* slow clock, count up to RC, then irq and restart */
- __raw_writel(timer_clock | ATMEL_TC_WAVE | ATMEL_TC_WAVESEL_UP_AUTO,
+ writel(timer_clock | ATMEL_TC_WAVE | ATMEL_TC_WAVESEL_UP_AUTO,
regs + ATMEL_TC_REG(2, CMR));
- __raw_writel((32768 + HZ / 2) / HZ, tcaddr + ATMEL_TC_REG(2, RC));
+ writel((32768 + HZ / 2) / HZ, tcaddr + ATMEL_TC_REG(2, RC));
/* Enable clock and interrupts on RC compare */
- __raw_writel(ATMEL_TC_CPCS, regs + ATMEL_TC_REG(2, IER));
+ writel(ATMEL_TC_CPCS, regs + ATMEL_TC_REG(2, IER));
/* go go gadget! */
- __raw_writel(ATMEL_TC_CLKEN | ATMEL_TC_SWTRG, regs +
+ writel(ATMEL_TC_CLKEN | ATMEL_TC_SWTRG, regs +
ATMEL_TC_REG(2, CCR));
return 0;
}
static int tc_next_event(unsigned long delta, struct clock_event_device *d)
{
- __raw_writel(delta, tcaddr + ATMEL_TC_REG(2, RC));
+ writel_relaxed(delta, tcaddr + ATMEL_TC_REG(2, RC));
/* go go gadget! */
- __raw_writel(ATMEL_TC_CLKEN | ATMEL_TC_SWTRG,
+ writel_relaxed(ATMEL_TC_CLKEN | ATMEL_TC_SWTRG,
tcaddr + ATMEL_TC_REG(2, CCR));
return 0;
}
struct tc_clkevt_device *dev = handle;
unsigned int sr;
- sr = __raw_readl(dev->regs + ATMEL_TC_REG(2, SR));
+ sr = readl_relaxed(dev->regs + ATMEL_TC_REG(2, SR));
if (sr & ATMEL_TC_CPCS) {
dev->clkevt.event_handler(&dev->clkevt);
return IRQ_HANDLED;
static void __init tcb_setup_dual_chan(struct atmel_tc *tc, int mck_divisor_idx)
{
/* channel 0: waveform mode, input mclk/8, clock TIOA0 on overflow */
- __raw_writel(mck_divisor_idx /* likely divide-by-8 */
+ writel(mck_divisor_idx /* likely divide-by-8 */
| ATMEL_TC_WAVE
| ATMEL_TC_WAVESEL_UP /* free-run */
| ATMEL_TC_ACPA_SET /* TIOA0 rises at 0 */
| ATMEL_TC_ACPC_CLEAR, /* (duty cycle 50%) */
tcaddr + ATMEL_TC_REG(0, CMR));
- __raw_writel(0x0000, tcaddr + ATMEL_TC_REG(0, RA));
- __raw_writel(0x8000, tcaddr + ATMEL_TC_REG(0, RC));
- __raw_writel(0xff, tcaddr + ATMEL_TC_REG(0, IDR)); /* no irqs */
- __raw_writel(ATMEL_TC_CLKEN, tcaddr + ATMEL_TC_REG(0, CCR));
+ writel(0x0000, tcaddr + ATMEL_TC_REG(0, RA));
+ writel(0x8000, tcaddr + ATMEL_TC_REG(0, RC));
+ writel(0xff, tcaddr + ATMEL_TC_REG(0, IDR)); /* no irqs */
+ writel(ATMEL_TC_CLKEN, tcaddr + ATMEL_TC_REG(0, CCR));
/* channel 1: waveform mode, input TIOA0 */
- __raw_writel(ATMEL_TC_XC1 /* input: TIOA0 */
+ writel(ATMEL_TC_XC1 /* input: TIOA0 */
| ATMEL_TC_WAVE
| ATMEL_TC_WAVESEL_UP, /* free-run */
tcaddr + ATMEL_TC_REG(1, CMR));
- __raw_writel(0xff, tcaddr + ATMEL_TC_REG(1, IDR)); /* no irqs */
- __raw_writel(ATMEL_TC_CLKEN, tcaddr + ATMEL_TC_REG(1, CCR));
+ writel(0xff, tcaddr + ATMEL_TC_REG(1, IDR)); /* no irqs */
+ writel(ATMEL_TC_CLKEN, tcaddr + ATMEL_TC_REG(1, CCR));
/* chain channel 0 to channel 1*/
- __raw_writel(ATMEL_TC_TC1XC1S_TIOA0, tcaddr + ATMEL_TC_BMR);
+ writel(ATMEL_TC_TC1XC1S_TIOA0, tcaddr + ATMEL_TC_BMR);
/* then reset all the timers */
- __raw_writel(ATMEL_TC_SYNC, tcaddr + ATMEL_TC_BCR);
+ writel(ATMEL_TC_SYNC, tcaddr + ATMEL_TC_BCR);
}
static void __init tcb_setup_single_chan(struct atmel_tc *tc, int mck_divisor_idx)
{
/* channel 0: waveform mode, input mclk/8 */
- __raw_writel(mck_divisor_idx /* likely divide-by-8 */
+ writel(mck_divisor_idx /* likely divide-by-8 */
| ATMEL_TC_WAVE
| ATMEL_TC_WAVESEL_UP, /* free-run */
tcaddr + ATMEL_TC_REG(0, CMR));
- __raw_writel(0xff, tcaddr + ATMEL_TC_REG(0, IDR)); /* no irqs */
- __raw_writel(ATMEL_TC_CLKEN, tcaddr + ATMEL_TC_REG(0, CCR));
+ writel(0xff, tcaddr + ATMEL_TC_REG(0, IDR)); /* no irqs */
+ writel(ATMEL_TC_CLKEN, tcaddr + ATMEL_TC_REG(0, CCR));
/* then reset all the timers */
- __raw_writel(ATMEL_TC_SYNC, tcaddr + ATMEL_TC_BCR);
+ writel(ATMEL_TC_SYNC, tcaddr + ATMEL_TC_BCR);
}
static int __init tcb_clksrc_init(void)
return 0;
}
-CLOCKSOURCE_OF_DECLARE(tegra20_timer, "nvidia,tegra20-timer", tegra20_init_timer);
+TIMER_OF_DECLARE(tegra20_timer, "nvidia,tegra20-timer", tegra20_init_timer);
static int __init tegra20_init_rtc(struct device_node *np)
{
return register_persistent_clock(NULL, tegra_read_persistent_clock64);
}
-CLOCKSOURCE_OF_DECLARE(tegra20_rtc, "nvidia,tegra20-rtc", tegra20_init_rtc);
+TIMER_OF_DECLARE(tegra20_rtc, "nvidia,tegra20-rtc", tegra20_init_rtc);
return armada_370_xp_timer_common_init(np);
}
-CLOCKSOURCE_OF_DECLARE(armada_xp, "marvell,armada-xp-timer",
+TIMER_OF_DECLARE(armada_xp, "marvell,armada-xp-timer",
armada_xp_timer_init);
static int __init armada_375_timer_init(struct device_node *np)
return armada_370_xp_timer_common_init(np);
}
-CLOCKSOURCE_OF_DECLARE(armada_375, "marvell,armada-375-timer",
+TIMER_OF_DECLARE(armada_375, "marvell,armada-375-timer",
armada_375_timer_init);
static int __init armada_370_timer_init(struct device_node *np)
return armada_370_xp_timer_common_init(np);
}
-CLOCKSOURCE_OF_DECLARE(armada_370, "marvell,armada-370-timer",
+TIMER_OF_DECLARE(armada_370, "marvell,armada-370-timer",
armada_370_timer_init);
return ret;
}
-CLOCKSOURCE_OF_DECLARE(efm32compat, "efm32,timer", efm32_timer_init);
-CLOCKSOURCE_OF_DECLARE(efm32, "energymicro,efm32-timer", efm32_timer_init);
+TIMER_OF_DECLARE(efm32compat, "efm32,timer", efm32_timer_init);
+TIMER_OF_DECLARE(efm32, "energymicro,efm32-timer", efm32_timer_init);
return ret;
}
-CLOCKSOURCE_OF_DECLARE(lpc32xx_timer, "nxp,lpc3220-timer", lpc32xx_timer_init);
+TIMER_OF_DECLARE(lpc32xx_timer, "nxp,lpc3220-timer", lpc32xx_timer_init);
return 0;
}
-CLOCKSOURCE_OF_DECLARE(orion_timer, "marvell,orion-timer", orion_timer_init);
+TIMER_OF_DECLARE(orion_timer, "marvell,orion-timer", orion_timer_init);
sched_clock_register(pistachio_read_sched_clock, 32, rate);
return clocksource_register_hz(&pcs_gpt.cs, rate);
}
-CLOCKSOURCE_OF_DECLARE(pistachio_gptimer, "img,pistachio-gptimer",
+TIMER_OF_DECLARE(pistachio_gptimer, "img,pistachio-gptimer",
pistachio_clksrc_of_init);
return sirfsoc_atlas7_timer_init(np);
}
-CLOCKSOURCE_OF_DECLARE(sirfsoc_atlas7_timer, "sirf,atlas7-tick", sirfsoc_of_timer_init);
+TIMER_OF_DECLARE(sirfsoc_atlas7_timer, "sirf,atlas7-tick", sirfsoc_of_timer_init);
return 0;
}
-CLOCKSOURCE_OF_DECLARE(at91sam926x_pit, "atmel,at91sam9260-pit",
+TIMER_OF_DECLARE(at91sam926x_pit, "atmel,at91sam9260-pit",
at91sam926x_pit_dt_init);
/* register clocksource */
return clocksource_register_hz(&clk32k, sclk_rate);
}
-CLOCKSOURCE_OF_DECLARE(atmel_st_timer, "atmel,at91rm9200-st",
+TIMER_OF_DECLARE(atmel_st_timer, "atmel,at91rm9200-st",
atmel_st_timer_init);
return 0;
}
-CLOCKSOURCE_OF_DECLARE(conexant_digicolor, "cnxt,cx92755-timer",
+TIMER_OF_DECLARE(conexant_digicolor, "cnxt,cx92755-timer",
digicolor_timer_init);
#include <linux/of.h>
#include <linux/of_address.h>
#include <linux/of_irq.h>
-#include <linux/mfd/syscon.h>
-#include <linux/regmap.h>
#include <linux/clockchips.h>
#include <linux/clocksource.h>
#include <linux/sched_clock.h>
#include <linux/clk.h>
+#include <linux/slab.h>
+#include <linux/bitops.h>
+#include <linux/delay.h>
/*
* Register definitions for the timers
#define TIMER_INTR_STATE (0x34)
#define TIMER_INTR_MASK (0x38)
-#define TIMER_1_CR_ENABLE (1 << 0)
-#define TIMER_1_CR_CLOCK (1 << 1)
-#define TIMER_1_CR_INT (1 << 2)
-#define TIMER_2_CR_ENABLE (1 << 3)
-#define TIMER_2_CR_CLOCK (1 << 4)
-#define TIMER_2_CR_INT (1 << 5)
-#define TIMER_3_CR_ENABLE (1 << 6)
-#define TIMER_3_CR_CLOCK (1 << 7)
-#define TIMER_3_CR_INT (1 << 8)
-#define TIMER_1_CR_UPDOWN (1 << 9)
-#define TIMER_2_CR_UPDOWN (1 << 10)
-#define TIMER_3_CR_UPDOWN (1 << 11)
-#define TIMER_DEFAULT_FLAGS (TIMER_1_CR_UPDOWN | \
- TIMER_3_CR_ENABLE | \
- TIMER_3_CR_UPDOWN)
-
-#define TIMER_1_INT_MATCH1 (1 << 0)
-#define TIMER_1_INT_MATCH2 (1 << 1)
-#define TIMER_1_INT_OVERFLOW (1 << 2)
-#define TIMER_2_INT_MATCH1 (1 << 3)
-#define TIMER_2_INT_MATCH2 (1 << 4)
-#define TIMER_2_INT_OVERFLOW (1 << 5)
-#define TIMER_3_INT_MATCH1 (1 << 6)
-#define TIMER_3_INT_MATCH2 (1 << 7)
-#define TIMER_3_INT_OVERFLOW (1 << 8)
+#define TIMER_1_CR_ENABLE BIT(0)
+#define TIMER_1_CR_CLOCK BIT(1)
+#define TIMER_1_CR_INT BIT(2)
+#define TIMER_2_CR_ENABLE BIT(3)
+#define TIMER_2_CR_CLOCK BIT(4)
+#define TIMER_2_CR_INT BIT(5)
+#define TIMER_3_CR_ENABLE BIT(6)
+#define TIMER_3_CR_CLOCK BIT(7)
+#define TIMER_3_CR_INT BIT(8)
+#define TIMER_1_CR_UPDOWN BIT(9)
+#define TIMER_2_CR_UPDOWN BIT(10)
+#define TIMER_3_CR_UPDOWN BIT(11)
+
+/*
+ * The Aspeed AST2400 moves bits around in the control register
+ * and lacks bits for setting the timer to count upwards.
+ */
+#define TIMER_1_CR_ASPEED_ENABLE BIT(0)
+#define TIMER_1_CR_ASPEED_CLOCK BIT(1)
+#define TIMER_1_CR_ASPEED_INT BIT(2)
+#define TIMER_2_CR_ASPEED_ENABLE BIT(4)
+#define TIMER_2_CR_ASPEED_CLOCK BIT(5)
+#define TIMER_2_CR_ASPEED_INT BIT(6)
+#define TIMER_3_CR_ASPEED_ENABLE BIT(8)
+#define TIMER_3_CR_ASPEED_CLOCK BIT(9)
+#define TIMER_3_CR_ASPEED_INT BIT(10)
+
+#define TIMER_1_INT_MATCH1 BIT(0)
+#define TIMER_1_INT_MATCH2 BIT(1)
+#define TIMER_1_INT_OVERFLOW BIT(2)
+#define TIMER_2_INT_MATCH1 BIT(3)
+#define TIMER_2_INT_MATCH2 BIT(4)
+#define TIMER_2_INT_OVERFLOW BIT(5)
+#define TIMER_3_INT_MATCH1 BIT(6)
+#define TIMER_3_INT_MATCH2 BIT(7)
+#define TIMER_3_INT_OVERFLOW BIT(8)
#define TIMER_INT_ALL_MASK 0x1ff
-static unsigned int tick_rate;
-static void __iomem *base;
+struct fttmr010 {
+ void __iomem *base;
+ unsigned int tick_rate;
+ bool count_down;
+ u32 t1_enable_val;
+ struct clock_event_device clkevt;
+#ifdef CONFIG_ARM
+ struct delay_timer delay_timer;
+#endif
+};
+
+/*
+ * A local singleton used by sched_clock and delay timer reads, which are
+ * fast and stateless
+ */
+static struct fttmr010 *local_fttmr;
+
+static inline struct fttmr010 *to_fttmr010(struct clock_event_device *evt)
+{
+ return container_of(evt, struct fttmr010, clkevt);
+}
+
+static unsigned long fttmr010_read_current_timer_up(void)
+{
+ return readl(local_fttmr->base + TIMER2_COUNT);
+}
+
+static unsigned long fttmr010_read_current_timer_down(void)
+{
+ return ~readl(local_fttmr->base + TIMER2_COUNT);
+}
+
+static u64 notrace fttmr010_read_sched_clock_up(void)
+{
+ return fttmr010_read_current_timer_up();
+}
-static u64 notrace fttmr010_read_sched_clock(void)
+static u64 notrace fttmr010_read_sched_clock_down(void)
{
- return readl(base + TIMER3_COUNT);
+ return fttmr010_read_current_timer_down();
}
static int fttmr010_timer_set_next_event(unsigned long cycles,
struct clock_event_device *evt)
{
+ struct fttmr010 *fttmr010 = to_fttmr010(evt);
u32 cr;
- /* Setup the match register */
- cr = readl(base + TIMER1_COUNT);
- writel(cr + cycles, base + TIMER1_MATCH1);
- if (readl(base + TIMER1_COUNT) - cr > cycles)
- return -ETIME;
+ /* Stop */
+ cr = readl(fttmr010->base + TIMER_CR);
+ cr &= ~fttmr010->t1_enable_val;
+ writel(cr, fttmr010->base + TIMER_CR);
+
+ /* Setup the match register forward/backward in time */
+ cr = readl(fttmr010->base + TIMER1_COUNT);
+ if (fttmr010->count_down)
+ cr -= cycles;
+ else
+ cr += cycles;
+ writel(cr, fttmr010->base + TIMER1_MATCH1);
+
+ /* Start */
+ cr = readl(fttmr010->base + TIMER_CR);
+ cr |= fttmr010->t1_enable_val;
+ writel(cr, fttmr010->base + TIMER_CR);
return 0;
}
static int fttmr010_timer_shutdown(struct clock_event_device *evt)
{
+ struct fttmr010 *fttmr010 = to_fttmr010(evt);
u32 cr;
- /*
- * Disable also for oneshot: the set_next() call will arm the timer
- * instead.
- */
- /* Stop timer and interrupt. */
- cr = readl(base + TIMER_CR);
- cr &= ~(TIMER_1_CR_ENABLE | TIMER_1_CR_INT);
- writel(cr, base + TIMER_CR);
+ /* Stop */
+ cr = readl(fttmr010->base + TIMER_CR);
+ cr &= ~fttmr010->t1_enable_val;
+ writel(cr, fttmr010->base + TIMER_CR);
+
+ return 0;
+}
+
+static int fttmr010_timer_set_oneshot(struct clock_event_device *evt)
+{
+ struct fttmr010 *fttmr010 = to_fttmr010(evt);
+ u32 cr;
+
+ /* Stop */
+ cr = readl(fttmr010->base + TIMER_CR);
+ cr &= ~fttmr010->t1_enable_val;
+ writel(cr, fttmr010->base + TIMER_CR);
- /* Setup counter start from 0 */
- writel(0, base + TIMER1_COUNT);
- writel(0, base + TIMER1_LOAD);
+ /* Setup counter start from 0 or ~0 */
+ writel(0, fttmr010->base + TIMER1_COUNT);
+ if (fttmr010->count_down)
+ writel(~0, fttmr010->base + TIMER1_LOAD);
+ else
+ writel(0, fttmr010->base + TIMER1_LOAD);
- /* enable interrupt */
- cr = readl(base + TIMER_INTR_MASK);
+ /* Enable interrupt */
+ cr = readl(fttmr010->base + TIMER_INTR_MASK);
cr &= ~(TIMER_1_INT_OVERFLOW | TIMER_1_INT_MATCH2);
cr |= TIMER_1_INT_MATCH1;
- writel(cr, base + TIMER_INTR_MASK);
-
- /* start the timer */
- cr = readl(base + TIMER_CR);
- cr |= TIMER_1_CR_ENABLE;
- writel(cr, base + TIMER_CR);
+ writel(cr, fttmr010->base + TIMER_INTR_MASK);
return 0;
}
static int fttmr010_timer_set_periodic(struct clock_event_device *evt)
{
- u32 period = DIV_ROUND_CLOSEST(tick_rate, HZ);
+ struct fttmr010 *fttmr010 = to_fttmr010(evt);
+ u32 period = DIV_ROUND_CLOSEST(fttmr010->tick_rate, HZ);
u32 cr;
- /* Stop timer and interrupt */
- cr = readl(base + TIMER_CR);
- cr &= ~(TIMER_1_CR_ENABLE | TIMER_1_CR_INT);
- writel(cr, base + TIMER_CR);
-
- /* Setup timer to fire at 1/HT intervals. */
- cr = 0xffffffff - (period - 1);
- writel(cr, base + TIMER1_COUNT);
- writel(cr, base + TIMER1_LOAD);
-
- /* enable interrupt on overflow */
- cr = readl(base + TIMER_INTR_MASK);
- cr &= ~(TIMER_1_INT_MATCH1 | TIMER_1_INT_MATCH2);
- cr |= TIMER_1_INT_OVERFLOW;
- writel(cr, base + TIMER_INTR_MASK);
+ /* Stop */
+ cr = readl(fttmr010->base + TIMER_CR);
+ cr &= ~fttmr010->t1_enable_val;
+ writel(cr, fttmr010->base + TIMER_CR);
+
+ /* Setup timer to fire at 1/HZ intervals. */
+ if (fttmr010->count_down) {
+ writel(period, fttmr010->base + TIMER1_LOAD);
+ writel(0, fttmr010->base + TIMER1_MATCH1);
+ } else {
+ cr = 0xffffffff - (period - 1);
+ writel(cr, fttmr010->base + TIMER1_COUNT);
+ writel(cr, fttmr010->base + TIMER1_LOAD);
+
+ /* Enable interrupt on overflow */
+ cr = readl(fttmr010->base + TIMER_INTR_MASK);
+ cr &= ~(TIMER_1_INT_MATCH1 | TIMER_1_INT_MATCH2);
+ cr |= TIMER_1_INT_OVERFLOW;
+ writel(cr, fttmr010->base + TIMER_INTR_MASK);
+ }
/* Start the timer */
- cr = readl(base + TIMER_CR);
- cr |= TIMER_1_CR_ENABLE;
- cr |= TIMER_1_CR_INT;
- writel(cr, base + TIMER_CR);
+ cr = readl(fttmr010->base + TIMER_CR);
+ cr |= fttmr010->t1_enable_val;
+ writel(cr, fttmr010->base + TIMER_CR);
return 0;
}
-/* Use TIMER1 as clock event */
-static struct clock_event_device fttmr010_clockevent = {
- .name = "TIMER1",
- /* Reasonably fast and accurate clock event */
- .rating = 300,
- .shift = 32,
- .features = CLOCK_EVT_FEAT_PERIODIC |
- CLOCK_EVT_FEAT_ONESHOT,
- .set_next_event = fttmr010_timer_set_next_event,
- .set_state_shutdown = fttmr010_timer_shutdown,
- .set_state_periodic = fttmr010_timer_set_periodic,
- .set_state_oneshot = fttmr010_timer_shutdown,
- .tick_resume = fttmr010_timer_shutdown,
-};
-
/*
* IRQ handler for the timer
*/
static irqreturn_t fttmr010_timer_interrupt(int irq, void *dev_id)
{
- struct clock_event_device *evt = &fttmr010_clockevent;
+ struct clock_event_device *evt = dev_id;
evt->event_handler(evt);
return IRQ_HANDLED;
}
-static struct irqaction fttmr010_timer_irq = {
- .name = "Faraday FTTMR010 Timer Tick",
- .flags = IRQF_TIMER,
- .handler = fttmr010_timer_interrupt,
-};
-
-static int __init fttmr010_timer_common_init(struct device_node *np)
+static int __init fttmr010_common_init(struct device_node *np, bool is_aspeed)
{
+ struct fttmr010 *fttmr010;
int irq;
+ struct clk *clk;
+ int ret;
+ u32 val;
+
+ /*
+ * These implementations require a clock reference.
+ * FIXME: we currently only support clocking using PCLK
+ * and using EXTCLK is not supported in the driver.
+ */
+ clk = of_clk_get_by_name(np, "PCLK");
+ if (IS_ERR(clk)) {
+ pr_err("could not get PCLK\n");
+ return PTR_ERR(clk);
+ }
+ ret = clk_prepare_enable(clk);
+ if (ret) {
+ pr_err("failed to enable PCLK\n");
+ return ret;
+ }
- base = of_iomap(np, 0);
- if (!base) {
+ fttmr010 = kzalloc(sizeof(*fttmr010), GFP_KERNEL);
+ if (!fttmr010) {
+ ret = -ENOMEM;
+ goto out_disable_clock;
+ }
+ fttmr010->tick_rate = clk_get_rate(clk);
+
+ fttmr010->base = of_iomap(np, 0);
+ if (!fttmr010->base) {
pr_err("Can't remap registers");
- return -ENXIO;
+ ret = -ENXIO;
+ goto out_free;
}
/* IRQ for timer 1 */
irq = irq_of_parse_and_map(np, 0);
if (irq <= 0) {
pr_err("Can't parse IRQ");
- return -EINVAL;
+ ret = -EINVAL;
+ goto out_unmap;
+ }
+
+ /*
+ * The Aspeed AST2400 moves bits around in the control register,
+ * otherwise it works the same.
+ */
+ if (is_aspeed) {
+ fttmr010->t1_enable_val = TIMER_1_CR_ASPEED_ENABLE |
+ TIMER_1_CR_ASPEED_INT;
+ /* Downward not available */
+ fttmr010->count_down = true;
+ } else {
+ fttmr010->t1_enable_val = TIMER_1_CR_ENABLE | TIMER_1_CR_INT;
}
/*
* Reset the interrupt mask and status
*/
- writel(TIMER_INT_ALL_MASK, base + TIMER_INTR_MASK);
- writel(0, base + TIMER_INTR_STATE);
- writel(TIMER_DEFAULT_FLAGS, base + TIMER_CR);
+ writel(TIMER_INT_ALL_MASK, fttmr010->base + TIMER_INTR_MASK);
+ writel(0, fttmr010->base + TIMER_INTR_STATE);
+
+ /*
+ * Enable timer 1 count up, timer 2 count up, except on Aspeed,
+ * where everything just counts down.
+ */
+ if (is_aspeed)
+ val = TIMER_2_CR_ASPEED_ENABLE;
+ else {
+ val = TIMER_2_CR_ENABLE;
+ if (!fttmr010->count_down)
+ val |= TIMER_1_CR_UPDOWN | TIMER_2_CR_UPDOWN;
+ }
+ writel(val, fttmr010->base + TIMER_CR);
/*
* Setup free-running clocksource timer (interrupts
* disabled.)
*/
- writel(0, base + TIMER3_COUNT);
- writel(0, base + TIMER3_LOAD);
- writel(0, base + TIMER3_MATCH1);
- writel(0, base + TIMER3_MATCH2);
- clocksource_mmio_init(base + TIMER3_COUNT,
- "fttmr010_clocksource", tick_rate,
- 300, 32, clocksource_mmio_readl_up);
- sched_clock_register(fttmr010_read_sched_clock, 32, tick_rate);
+ local_fttmr = fttmr010;
+ writel(0, fttmr010->base + TIMER2_COUNT);
+ writel(0, fttmr010->base + TIMER2_MATCH1);
+ writel(0, fttmr010->base + TIMER2_MATCH2);
+
+ if (fttmr010->count_down) {
+ writel(~0, fttmr010->base + TIMER2_LOAD);
+ clocksource_mmio_init(fttmr010->base + TIMER2_COUNT,
+ "FTTMR010-TIMER2",
+ fttmr010->tick_rate,
+ 300, 32, clocksource_mmio_readl_down);
+ sched_clock_register(fttmr010_read_sched_clock_down, 32,
+ fttmr010->tick_rate);
+ } else {
+ writel(0, fttmr010->base + TIMER2_LOAD);
+ clocksource_mmio_init(fttmr010->base + TIMER2_COUNT,
+ "FTTMR010-TIMER2",
+ fttmr010->tick_rate,
+ 300, 32, clocksource_mmio_readl_up);
+ sched_clock_register(fttmr010_read_sched_clock_up, 32,
+ fttmr010->tick_rate);
+ }
/*
- * Setup clockevent timer (interrupt-driven.)
+ * Setup clockevent timer (interrupt-driven) on timer 1.
*/
- writel(0, base + TIMER1_COUNT);
- writel(0, base + TIMER1_LOAD);
- writel(0, base + TIMER1_MATCH1);
- writel(0, base + TIMER1_MATCH2);
- setup_irq(irq, &fttmr010_timer_irq);
- fttmr010_clockevent.cpumask = cpumask_of(0);
- clockevents_config_and_register(&fttmr010_clockevent, tick_rate,
+ writel(0, fttmr010->base + TIMER1_COUNT);
+ writel(0, fttmr010->base + TIMER1_LOAD);
+ writel(0, fttmr010->base + TIMER1_MATCH1);
+ writel(0, fttmr010->base + TIMER1_MATCH2);
+ ret = request_irq(irq, fttmr010_timer_interrupt, IRQF_TIMER,
+ "FTTMR010-TIMER1", &fttmr010->clkevt);
+ if (ret) {
+ pr_err("FTTMR010-TIMER1 no IRQ\n");
+ goto out_unmap;
+ }
+
+ fttmr010->clkevt.name = "FTTMR010-TIMER1";
+ /* Reasonably fast and accurate clock event */
+ fttmr010->clkevt.rating = 300;
+ fttmr010->clkevt.features = CLOCK_EVT_FEAT_PERIODIC |
+ CLOCK_EVT_FEAT_ONESHOT;
+ fttmr010->clkevt.set_next_event = fttmr010_timer_set_next_event;
+ fttmr010->clkevt.set_state_shutdown = fttmr010_timer_shutdown;
+ fttmr010->clkevt.set_state_periodic = fttmr010_timer_set_periodic;
+ fttmr010->clkevt.set_state_oneshot = fttmr010_timer_set_oneshot;
+ fttmr010->clkevt.tick_resume = fttmr010_timer_shutdown;
+ fttmr010->clkevt.cpumask = cpumask_of(0);
+ fttmr010->clkevt.irq = irq;
+ clockevents_config_and_register(&fttmr010->clkevt,
+ fttmr010->tick_rate,
1, 0xffffffff);
- return 0;
-}
+#ifdef CONFIG_ARM
+ /* Also use this timer for delays */
+ if (fttmr010->count_down)
+ fttmr010->delay_timer.read_current_timer =
+ fttmr010_read_current_timer_down;
+ else
+ fttmr010->delay_timer.read_current_timer =
+ fttmr010_read_current_timer_up;
+ fttmr010->delay_timer.freq = fttmr010->tick_rate;
+ register_current_timer_delay(&fttmr010->delay_timer);
+#endif
-static int __init fttmr010_timer_of_init(struct device_node *np)
-{
- /*
- * These implementations require a clock reference.
- * FIXME: we currently only support clocking using PCLK
- * and using EXTCLK is not supported in the driver.
- */
- struct clk *clk;
+ return 0;
- clk = of_clk_get_by_name(np, "PCLK");
- if (IS_ERR(clk)) {
- pr_err("could not get PCLK");
- return PTR_ERR(clk);
- }
- tick_rate = clk_get_rate(clk);
+out_unmap:
+ iounmap(fttmr010->base);
+out_free:
+ kfree(fttmr010);
+out_disable_clock:
+ clk_disable_unprepare(clk);
- return fttmr010_timer_common_init(np);
+ return ret;
}
-CLOCKSOURCE_OF_DECLARE(fttmr010, "faraday,fttmr010", fttmr010_timer_of_init);
-/*
- * Gemini-specific: relevant registers in the global syscon
- */
-#define GLOBAL_STATUS 0x04
-#define CPU_AHB_RATIO_MASK (0x3 << 18)
-#define CPU_AHB_1_1 (0x0 << 18)
-#define CPU_AHB_3_2 (0x1 << 18)
-#define CPU_AHB_24_13 (0x2 << 18)
-#define CPU_AHB_2_1 (0x3 << 18)
-#define REG_TO_AHB_SPEED(reg) ((((reg) >> 15) & 0x7) * 10 + 130)
-
-static int __init gemini_timer_of_init(struct device_node *np)
+static __init int aspeed_timer_init(struct device_node *np)
{
- static struct regmap *map;
- int ret;
- u32 val;
-
- map = syscon_regmap_lookup_by_phandle(np, "syscon");
- if (IS_ERR(map)) {
- pr_err("Can't get regmap for syscon handle\n");
- return -ENODEV;
- }
- ret = regmap_read(map, GLOBAL_STATUS, &val);
- if (ret) {
- pr_err("Can't read syscon status register\n");
- return -ENXIO;
- }
-
- tick_rate = REG_TO_AHB_SPEED(val) * 1000000;
- pr_info("Bus: %dMHz ", tick_rate / 1000000);
-
- tick_rate /= 6; /* APB bus run AHB*(1/6) */
-
- switch (val & CPU_AHB_RATIO_MASK) {
- case CPU_AHB_1_1:
- pr_cont("(1/1)\n");
- break;
- case CPU_AHB_3_2:
- pr_cont("(3/2)\n");
- break;
- case CPU_AHB_24_13:
- pr_cont("(24/13)\n");
- break;
- case CPU_AHB_2_1:
- pr_cont("(2/1)\n");
- break;
- }
+ return fttmr010_common_init(np, true);
+}
- return fttmr010_timer_common_init(np);
+static __init int fttmr010_timer_init(struct device_node *np)
+{
+ return fttmr010_common_init(np, false);
}
-CLOCKSOURCE_OF_DECLARE(gemini, "cortina,gemini-timer", gemini_timer_of_init);
+
+TIMER_OF_DECLARE(fttmr010, "faraday,fttmr010", fttmr010_timer_init);
+TIMER_OF_DECLARE(gemini, "cortina,gemini-timer", fttmr010_timer_init);
+TIMER_OF_DECLARE(moxart, "moxa,moxart-timer", fttmr010_timer_init);
+TIMER_OF_DECLARE(ast2400, "aspeed,ast2400-timer", aspeed_timer_init);
+TIMER_OF_DECLARE(ast2500, "aspeed,ast2500-timer", aspeed_timer_init);
return mxc_timer_init_dt(np, GPT_TYPE_IMX6DL);
}
-CLOCKSOURCE_OF_DECLARE(imx1_timer, "fsl,imx1-gpt", imx1_timer_init_dt);
-CLOCKSOURCE_OF_DECLARE(imx21_timer, "fsl,imx21-gpt", imx21_timer_init_dt);
-CLOCKSOURCE_OF_DECLARE(imx27_timer, "fsl,imx27-gpt", imx21_timer_init_dt);
-CLOCKSOURCE_OF_DECLARE(imx31_timer, "fsl,imx31-gpt", imx31_timer_init_dt);
-CLOCKSOURCE_OF_DECLARE(imx25_timer, "fsl,imx25-gpt", imx31_timer_init_dt);
-CLOCKSOURCE_OF_DECLARE(imx50_timer, "fsl,imx50-gpt", imx31_timer_init_dt);
-CLOCKSOURCE_OF_DECLARE(imx51_timer, "fsl,imx51-gpt", imx31_timer_init_dt);
-CLOCKSOURCE_OF_DECLARE(imx53_timer, "fsl,imx53-gpt", imx31_timer_init_dt);
-CLOCKSOURCE_OF_DECLARE(imx6q_timer, "fsl,imx6q-gpt", imx31_timer_init_dt);
-CLOCKSOURCE_OF_DECLARE(imx6dl_timer, "fsl,imx6dl-gpt", imx6dl_timer_init_dt);
-CLOCKSOURCE_OF_DECLARE(imx6sl_timer, "fsl,imx6sl-gpt", imx6dl_timer_init_dt);
-CLOCKSOURCE_OF_DECLARE(imx6sx_timer, "fsl,imx6sx-gpt", imx6dl_timer_init_dt);
+TIMER_OF_DECLARE(imx1_timer, "fsl,imx1-gpt", imx1_timer_init_dt);
+TIMER_OF_DECLARE(imx21_timer, "fsl,imx21-gpt", imx21_timer_init_dt);
+TIMER_OF_DECLARE(imx27_timer, "fsl,imx27-gpt", imx21_timer_init_dt);
+TIMER_OF_DECLARE(imx31_timer, "fsl,imx31-gpt", imx31_timer_init_dt);
+TIMER_OF_DECLARE(imx25_timer, "fsl,imx25-gpt", imx31_timer_init_dt);
+TIMER_OF_DECLARE(imx50_timer, "fsl,imx50-gpt", imx31_timer_init_dt);
+TIMER_OF_DECLARE(imx51_timer, "fsl,imx51-gpt", imx31_timer_init_dt);
+TIMER_OF_DECLARE(imx53_timer, "fsl,imx53-gpt", imx31_timer_init_dt);
+TIMER_OF_DECLARE(imx6q_timer, "fsl,imx6q-gpt", imx31_timer_init_dt);
+TIMER_OF_DECLARE(imx6dl_timer, "fsl,imx6dl-gpt", imx6dl_timer_init_dt);
+TIMER_OF_DECLARE(imx6sl_timer, "fsl,imx6sl-gpt", imx6dl_timer_init_dt);
+TIMER_OF_DECLARE(imx6sx_timer, "fsl,imx6sx-gpt", imx6dl_timer_init_dt);
return 0;
}
-CLOCKSOURCE_OF_DECLARE(integrator_ap_timer, "arm,integrator-timer",
+TIMER_OF_DECLARE(integrator_ap_timer, "arm,integrator-timer",
integrator_ap_timer_init_of);
return error;
}
-CLOCKSOURCE_OF_DECLARE(keystone_timer, "ti,keystone-timer",
+TIMER_OF_DECLARE(keystone_timer, "ti,keystone-timer",
keystone_timer_init);
return ret;
}
-CLOCKSOURCE_OF_DECLARE(ezchip_nps400_clksrc, "ezchip,nps400-timer",
+TIMER_OF_DECLARE(ezchip_nps400_clksrc, "ezchip,nps400-timer",
nps_setup_clocksource);
-CLOCKSOURCE_OF_DECLARE(ezchip_nps400_clk_src, "ezchip,nps400-timer1",
+TIMER_OF_DECLARE(ezchip_nps400_clk_src, "ezchip,nps400-timer1",
nps_setup_clocksource);
#ifdef CONFIG_EZNPS_MTM_EXT
return 0;
}
-CLOCKSOURCE_OF_DECLARE(ezchip_nps400_clk_evt, "ezchip,nps400-timer0",
+TIMER_OF_DECLARE(ezchip_nps400_clk_evt, "ezchip,nps400-timer0",
nps_setup_clockevent);
#endif /* CONFIG_EZNPS_MTM_EXT */
--- /dev/null
+/*
+ * Copyright (c) 2017, Linaro Ltd. All rights reserved.
+ *
+ * Author: Daniel Lezcano <daniel.lezcano@linaro.org>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+#include <linux/clk.h>
+#include <linux/interrupt.h>
+#include <linux/of.h>
+#include <linux/of_address.h>
+#include <linux/of_irq.h>
+#include <linux/slab.h>
+
+#include "timer-of.h"
+
+static __init void timer_irq_exit(struct of_timer_irq *of_irq)
+{
+ struct timer_of *to = container_of(of_irq, struct timer_of, of_irq);
+
+ struct clock_event_device *clkevt = &to->clkevt;
+
+ of_irq->percpu ? free_percpu_irq(of_irq->irq, clkevt) :
+ free_irq(of_irq->irq, clkevt);
+}
+
+static __init int timer_irq_init(struct device_node *np,
+ struct of_timer_irq *of_irq)
+{
+ int ret;
+ struct timer_of *to = container_of(of_irq, struct timer_of, of_irq);
+ struct clock_event_device *clkevt = &to->clkevt;
+
+ of_irq->irq = of_irq->name ? of_irq_get_byname(np, of_irq->name):
+ irq_of_parse_and_map(np, of_irq->index);
+ if (!of_irq->irq) {
+ pr_err("Failed to map interrupt for %s\n", np->full_name);
+ return -EINVAL;
+ }
+
+ ret = of_irq->percpu ?
+ request_percpu_irq(of_irq->irq, of_irq->handler,
+ np->full_name, clkevt) :
+ request_irq(of_irq->irq, of_irq->handler,
+ of_irq->flags ? of_irq->flags : IRQF_TIMER,
+ np->full_name, clkevt);
+ if (ret) {
+ pr_err("Failed to request irq %d for %s\n", of_irq->irq,
+ np->full_name);
+ return ret;
+ }
+
+ clkevt->irq = of_irq->irq;
+
+ return 0;
+}
+
+static __init void timer_clk_exit(struct of_timer_clk *of_clk)
+{
+ of_clk->rate = 0;
+ clk_disable_unprepare(of_clk->clk);
+ clk_put(of_clk->clk);
+}
+
+static __init int timer_clk_init(struct device_node *np,
+ struct of_timer_clk *of_clk)
+{
+ int ret;
+
+ of_clk->clk = of_clk->name ? of_clk_get_by_name(np, of_clk->name) :
+ of_clk_get(np, of_clk->index);
+ if (IS_ERR(of_clk->clk)) {
+ pr_err("Failed to get clock for %s\n", np->full_name);
+ return PTR_ERR(of_clk->clk);
+ }
+
+ ret = clk_prepare_enable(of_clk->clk);
+ if (ret) {
+ pr_err("Failed for enable clock for %s\n", np->full_name);
+ goto out_clk_put;
+ }
+
+ of_clk->rate = clk_get_rate(of_clk->clk);
+ if (!of_clk->rate) {
+ ret = -EINVAL;
+ pr_err("Failed to get clock rate for %s\n", np->full_name);
+ goto out_clk_disable;
+ }
+
+ of_clk->period = DIV_ROUND_UP(of_clk->rate, HZ);
+out:
+ return ret;
+
+out_clk_disable:
+ clk_disable_unprepare(of_clk->clk);
+out_clk_put:
+ clk_put(of_clk->clk);
+
+ goto out;
+}
+
+static __init void timer_base_exit(struct of_timer_base *of_base)
+{
+ iounmap(of_base->base);
+}
+
+static __init int timer_base_init(struct device_node *np,
+ struct of_timer_base *of_base)
+{
+ const char *name = of_base->name ? of_base->name : np->full_name;
+
+ of_base->base = of_io_request_and_map(np, of_base->index, name);
+ if (!of_base->base) {
+ pr_err("Failed to iomap (%s)\n", name);
+ return -ENXIO;
+ }
+
+ return 0;
+}
+
+int __init timer_of_init(struct device_node *np, struct timer_of *to)
+{
+ int ret = -EINVAL;
+ int flags = 0;
+
+ if (to->flags & TIMER_OF_BASE) {
+ ret = timer_base_init(np, &to->of_base);
+ if (ret)
+ goto out_fail;
+ flags |= TIMER_OF_BASE;
+ }
+
+ if (to->flags & TIMER_OF_CLOCK) {
+ ret = timer_clk_init(np, &to->of_clk);
+ if (ret)
+ goto out_fail;
+ flags |= TIMER_OF_CLOCK;
+ }
+
+ if (to->flags & TIMER_OF_IRQ) {
+ ret = timer_irq_init(np, &to->of_irq);
+ if (ret)
+ goto out_fail;
+ flags |= TIMER_OF_IRQ;
+ }
+
+ if (!to->clkevt.name)
+ to->clkevt.name = np->name;
+ return ret;
+
+out_fail:
+ if (flags & TIMER_OF_IRQ)
+ timer_irq_exit(&to->of_irq);
+
+ if (flags & TIMER_OF_CLOCK)
+ timer_clk_exit(&to->of_clk);
+
+ if (flags & TIMER_OF_BASE)
+ timer_base_exit(&to->of_base);
+ return ret;
+}
--- /dev/null
+#ifndef __TIMER_OF_H__
+#define __TIMER_OF_H__
+
+#include <linux/clockchips.h>
+
+#define TIMER_OF_BASE 0x1
+#define TIMER_OF_CLOCK 0x2
+#define TIMER_OF_IRQ 0x4
+
+struct of_timer_irq {
+ int irq;
+ int index;
+ int percpu;
+ const char *name;
+ unsigned long flags;
+ irq_handler_t handler;
+};
+
+struct of_timer_base {
+ void __iomem *base;
+ const char *name;
+ int index;
+};
+
+struct of_timer_clk {
+ struct clk *clk;
+ const char *name;
+ int index;
+ unsigned long rate;
+ unsigned long period;
+};
+
+struct timer_of {
+ unsigned int flags;
+ struct clock_event_device clkevt;
+ struct of_timer_base of_base;
+ struct of_timer_irq of_irq;
+ struct of_timer_clk of_clk;
+ void *private_data;
+};
+
+static inline struct timer_of *to_timer_of(struct clock_event_device *clkevt)
+{
+ return container_of(clkevt, struct timer_of, clkevt);
+}
+
+static inline void __iomem *timer_of_base(struct timer_of *to)
+{
+ return to->of_base.base;
+}
+
+static inline int timer_of_irq(struct timer_of *to)
+{
+ return to->of_irq.irq;
+}
+
+static inline unsigned long timer_of_rate(struct timer_of *to)
+{
+ return to->of_clk.rate;
+}
+
+static inline unsigned long timer_of_period(struct timer_of *to)
+{
+ return to->of_clk.period;
+}
+
+extern int __init timer_of_init(struct device_node *np,
+ struct timer_of *to);
+#endif
return ret;
}
-CLOCKSOURCE_OF_DECLARE(ox810se_rps,
+TIMER_OF_DECLARE(ox810se_rps,
"oxsemi,ox810se-rps-timer", oxnas_rps_timer_init);
-CLOCKSOURCE_OF_DECLARE(ox820_rps,
+TIMER_OF_DECLARE(ox820_rps,
"oxsemi,ox820se-rps-timer", oxnas_rps_timer_init);
return 0;
}
-CLOCKSOURCE_OF_DECLARE(sirfsoc_prima2_timer,
+TIMER_OF_DECLARE(sirfsoc_prima2_timer,
"sirf,prima2-tick", sirfsoc_prima2_timer_init);
#include <linux/of.h>
#include <linux/clocksource.h>
-extern struct of_device_id __clksrc_of_table[];
+extern struct of_device_id __timer_of_table[];
-static const struct of_device_id __clksrc_of_table_sentinel
- __used __section(__clksrc_of_table_end);
+static const struct of_device_id __timer_of_table_sentinel
+ __used __section(__timer_of_table_end);
-void __init clocksource_probe(void)
+void __init timer_probe(void)
{
struct device_node *np;
const struct of_device_id *match;
of_init_fn_1_ret init_func_ret;
- unsigned clocksources = 0;
+ unsigned timers = 0;
int ret;
- for_each_matching_node_and_match(np, __clksrc_of_table, &match) {
+ for_each_matching_node_and_match(np, __timer_of_table, &match) {
if (!of_device_is_available(np))
continue;
continue;
}
- clocksources++;
+ timers++;
}
- clocksources += acpi_probe_device_table(clksrc);
+ timers += acpi_probe_device_table(timer);
- if (!clocksources)
- pr_crit("%s: no matching clocksources found\n", __func__);
+ if (!timers)
+ pr_crit("%s: no matching timers found\n", __func__);
}
iounmap(base);
return ret;
}
-CLOCKSOURCE_OF_DECLARE(sp804, "arm,sp804", sp804_of_init);
+TIMER_OF_DECLARE(sp804, "arm,sp804", sp804_of_init);
static int __init integrator_cp_of_init(struct device_node *np)
{
iounmap(base);
return ret;
}
-CLOCKSOURCE_OF_DECLARE(intcp, "arm,integrator-cp-timer", integrator_cp_of_init);
+TIMER_OF_DECLARE(intcp, "arm,integrator-cp-timer", integrator_cp_of_init);
return ret;
}
-CLOCKSOURCE_OF_DECLARE(stm32, "st,stm32-timer", stm32_clockevent_init);
+TIMER_OF_DECLARE(stm32, "st,stm32-timer", stm32_clockevent_init);
#include <linux/clk.h>
#include <linux/clockchips.h>
+#include <linux/clocksource.h>
#include <linux/delay.h>
#include <linux/interrupt.h>
#include <linux/irq.h>
return sun5i_setup_clockevent(node, timer_base, clk, irq);
}
-CLOCKSOURCE_OF_DECLARE(sun5i_a13, "allwinner,sun5i-a13-hstimer",
+TIMER_OF_DECLARE(sun5i_a13, "allwinner,sun5i-a13-hstimer",
sun5i_timer_init);
-CLOCKSOURCE_OF_DECLARE(sun7i_a20, "allwinner,sun7i-a20-hstimer",
+TIMER_OF_DECLARE(sun7i_a20, "allwinner,sun7i-a20-hstimer",
sun5i_timer_init);
return 0;
}
-CLOCKSOURCE_OF_DECLARE(ti_32k_timer, "ti,omap-counter32k",
+TIMER_OF_DECLARE(ti_32k_timer, "ti,omap-counter32k",
ti_32k_timer_init);
return 0;
}
-CLOCKSOURCE_OF_DECLARE(u300_timer, "stericsson,u300-apptimer",
+TIMER_OF_DECLARE(u300_timer, "stericsson,u300-apptimer",
u300_timer_init_of);
return 0;
}
-CLOCKSOURCE_OF_DECLARE(vexpress, "arm,vexpress-sysreg",
+TIMER_OF_DECLARE(vexpress, "arm,vexpress-sysreg",
versatile_sched_clock_init);
-CLOCKSOURCE_OF_DECLARE(versatile, "arm,versatile-sysreg",
+TIMER_OF_DECLARE(versatile, "arm,versatile-sysreg",
versatile_sched_clock_init);
return pit_clockevent_init(clk_rate, irq);
}
-CLOCKSOURCE_OF_DECLARE(vf610, "fsl,vf610-pit", pit_timer_init);
+TIMER_OF_DECLARE(vf610, "fsl,vf610-pit", pit_timer_init);
return 0;
}
-CLOCKSOURCE_OF_DECLARE(vt8500, "via,vt8500-timer", vt8500_timer_init);
+TIMER_OF_DECLARE(vt8500, "via,vt8500-timer", vt8500_timer_init);
return zevio_timer_add(node);
}
-CLOCKSOURCE_OF_DECLARE(zevio_timer, "lsi,zevio-timer", zevio_timer_init);
+TIMER_OF_DECLARE(zevio_timer, "lsi,zevio-timer", zevio_timer_init);
cppc_dmi_max_khz = cppc_get_dmi_max_khz();
- policy->min = cpu->perf_caps.lowest_perf * cppc_dmi_max_khz / cpu->perf_caps.highest_perf;
+ /*
+ * Set min to lowest nonlinear perf to avoid any efficiency penalty (see
+ * Section 8.4.7.1.1.5 of ACPI 6.1 spec)
+ */
+ policy->min = cpu->perf_caps.lowest_nonlinear_perf * cppc_dmi_max_khz /
+ cpu->perf_caps.highest_perf;
policy->max = cppc_dmi_max_khz;
- policy->cpuinfo.min_freq = policy->min;
- policy->cpuinfo.max_freq = policy->max;
+
+ /*
+ * Set cpuinfo.min_freq to Lowest to make the full range of performance
+ * available if userspace wants to use any perf between lowest & lowest
+ * nonlinear perf
+ */
+ policy->cpuinfo.min_freq = cpu->perf_caps.lowest_perf * cppc_dmi_max_khz /
+ cpu->perf_caps.highest_perf;
+ policy->cpuinfo.max_freq = cppc_dmi_max_khz;
+
policy->cpuinfo.transition_latency = cppc_get_transition_latency(cpu_num);
policy->shared_type = cpu->shared_type;
{ .compatible = "arm,integrator-ap", },
{ .compatible = "arm,integrator-cp", },
+ { .compatible = "hisilicon,hi3660", },
{ .compatible = "hisilicon,hi6220", },
{ .compatible = "fsl,imx27", },
show_one(scaling_min_freq, min);
show_one(scaling_max_freq, max);
+__weak unsigned int arch_freq_get_on_cpu(int cpu)
+{
+ return 0;
+}
+
static ssize_t show_scaling_cur_freq(struct cpufreq_policy *policy, char *buf)
{
ssize_t ret;
+ unsigned int freq;
- if (cpufreq_driver && cpufreq_driver->setpolicy && cpufreq_driver->get)
+ freq = arch_freq_get_on_cpu(policy->cpu);
+ if (freq)
+ ret = sprintf(buf, "%u\n", freq);
+ else if (cpufreq_driver && cpufreq_driver->setpolicy &&
+ cpufreq_driver->get)
ret = sprintf(buf, "%u\n", cpufreq_driver->get(policy->cpu));
else
ret = sprintf(buf, "%u\n", policy->cur);
struct freq_attr *fattr = to_attr(attr);
ssize_t ret = -EINVAL;
- get_online_cpus();
+ cpus_read_lock();
if (cpu_online(policy->cpu)) {
down_write(&policy->rwsem);
up_write(&policy->rwsem);
}
- put_online_cpus();
+ cpus_read_unlock();
return ret;
}
pr_debug("trying to register driver %s\n", driver_data->name);
/* Protect against concurrent CPU online/offline. */
- get_online_cpus();
+ cpus_read_lock();
write_lock_irqsave(&cpufreq_driver_lock, flags);
if (cpufreq_driver) {
goto err_if_unreg;
}
- ret = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "cpufreq:online",
- cpuhp_cpufreq_online,
- cpuhp_cpufreq_offline);
+ ret = cpuhp_setup_state_nocalls_cpuslocked(CPUHP_AP_ONLINE_DYN,
+ "cpufreq:online",
+ cpuhp_cpufreq_online,
+ cpuhp_cpufreq_offline);
if (ret < 0)
goto err_if_unreg;
hp_online = ret;
cpufreq_driver = NULL;
write_unlock_irqrestore(&cpufreq_driver_lock, flags);
out:
- put_online_cpus();
+ cpus_read_unlock();
return ret;
}
EXPORT_SYMBOL_GPL(cpufreq_register_driver);
pr_debug("unregistering driver %s\n", driver->name);
/* Protect against concurrent cpu hotplug */
- get_online_cpus();
+ cpus_read_lock();
subsys_interface_unregister(&cpufreq_interface);
remove_boost_sysfs_file();
- cpuhp_remove_state_nocalls(hp_online);
+ cpuhp_remove_state_nocalls_cpuslocked(hp_online);
write_lock_irqsave(&cpufreq_driver_lock, flags);
cpufreq_driver = NULL;
write_unlock_irqrestore(&cpufreq_driver_lock, flags);
- put_online_cpus();
+ cpus_read_unlock();
return 0;
}
/* Enable PSTATE Change Event */
tmp = __raw_readl(dvfs_info->base + XMU_PMUEVTEN);
tmp |= (1 << PSTATE_CHANGED_EVTEN_SHIFT);
- __raw_writel(tmp, dvfs_info->base + XMU_PMUEVTEN);
+ __raw_writel(tmp, dvfs_info->base + XMU_PMUEVTEN);
/* Enable PSTATE Change IRQ */
tmp = __raw_readl(dvfs_info->base + XMU_PMUIRQEN);
tmp |= (1 << PSTATE_CHANGED_IRQEN_SHIFT);
- __raw_writel(tmp, dvfs_info->base + XMU_PMUIRQEN);
+ __raw_writel(tmp, dvfs_info->base + XMU_PMUIRQEN);
/* Set initial performance index */
cpufreq_for_each_entry(pos, freq_table)
struct resource res;
unsigned int cur_frequency;
- np = pdev->dev.of_node;
+ np = pdev->dev.of_node;
if (!np)
return -ENODEV;
* - Reprogram pll1_sys_clk and reparent pll1_sw_clk back to it
* - Disable pll2_pfd2_396m_clk
*/
- if (of_machine_is_compatible("fsl,imx6ul")) {
+ if (of_machine_is_compatible("fsl,imx6ul") ||
+ of_machine_is_compatible("fsl,imx6ull")) {
/*
* When changing pll1_sw_clk's parent to pll1_sys_clk,
* CPU may run at higher than 528MHz, this will lead to
goto put_clk;
}
- if (of_machine_is_compatible("fsl,imx6ul")) {
+ if (of_machine_is_compatible("fsl,imx6ul") ||
+ of_machine_is_compatible("fsl,imx6ull")) {
pll2_bus_clk = clk_get(cpu_dev, "pll2_bus");
secondary_sel_clk = clk_get(cpu_dev, "secondary_sel");
if (IS_ERR(pll2_bus_clk) || IS_ERR(secondary_sel_clk)) {
* @prev_cummulative_iowait: IO Wait time difference from last and
* current sample
* @sample: Storage for storing last Sample data
- * @min_perf: Minimum capacity limit as a fraction of the maximum
- * turbo P-state capacity.
- * @max_perf: Maximum capacity limit as a fraction of the maximum
- * turbo P-state capacity.
+ * @min_perf_ratio: Minimum capacity in terms of PERF or HWP ratios
+ * @max_perf_ratio: Maximum capacity in terms of PERF or HWP ratios
* @acpi_perf_data: Stores ACPI perf information read from _PSS
* @valid_pss_table: Set to true for valid ACPI _PSS entries found
* @epp_powersave: Last saved HWP energy performance preference
u64 prev_tsc;
u64 prev_cummulative_iowait;
struct sample sample;
- int32_t min_perf;
- int32_t max_perf;
+ int32_t min_perf_ratio;
+ int32_t max_perf_ratio;
#ifdef CONFIG_ACPI
struct acpi_processor_performance acpi_perf_data;
bool valid_pss_table;
"power",
NULL
};
+static const unsigned int epp_values[] = {
+ HWP_EPP_PERFORMANCE,
+ HWP_EPP_BALANCE_PERFORMANCE,
+ HWP_EPP_BALANCE_POWERSAVE,
+ HWP_EPP_POWERSAVE
+};
static int intel_pstate_get_energy_pref_index(struct cpudata *cpu_data)
{
return epp;
if (static_cpu_has(X86_FEATURE_HWP_EPP)) {
- /*
- * Range:
- * 0x00-0x3F : Performance
- * 0x40-0x7F : Balance performance
- * 0x80-0xBF : Balance power
- * 0xC0-0xFF : Power
- * The EPP is a 8 bit value, but our ranges restrict the
- * value which can be set. Here only using top two bits
- * effectively.
- */
- index = (epp >> 6) + 1;
+ if (epp == HWP_EPP_PERFORMANCE)
+ return 1;
+ if (epp <= HWP_EPP_BALANCE_PERFORMANCE)
+ return 2;
+ if (epp <= HWP_EPP_BALANCE_POWERSAVE)
+ return 3;
+ else
+ return 4;
} else if (static_cpu_has(X86_FEATURE_EPB)) {
/*
* Range:
value &= ~GENMASK_ULL(31, 24);
- /*
- * If epp is not default, convert from index into
- * energy_perf_strings to epp value, by shifting 6
- * bits left to use only top two bits in epp.
- * The resultant epp need to shifted by 24 bits to
- * epp position in MSR_HWP_REQUEST.
- */
if (epp == -EINVAL)
- epp = (pref_index - 1) << 6;
+ epp = epp_values[pref_index - 1];
value |= (u64)epp << 24;
ret = wrmsrl_on_cpu(cpu_data->cpu, MSR_HWP_REQUEST, value);
NULL,
};
-static void intel_pstate_hwp_set(unsigned int cpu)
+static void intel_pstate_get_hwp_max(unsigned int cpu, int *phy_max,
+ int *current_max)
{
- struct cpudata *cpu_data = all_cpu_data[cpu];
- int min, hw_min, max, hw_max;
- u64 value, cap;
- s16 epp;
+ u64 cap;
rdmsrl_on_cpu(cpu, MSR_HWP_CAPABILITIES, &cap);
- hw_min = HWP_LOWEST_PERF(cap);
if (global.no_turbo)
- hw_max = HWP_GUARANTEED_PERF(cap);
+ *current_max = HWP_GUARANTEED_PERF(cap);
else
- hw_max = HWP_HIGHEST_PERF(cap);
+ *current_max = HWP_HIGHEST_PERF(cap);
+
+ *phy_max = HWP_HIGHEST_PERF(cap);
+}
+
+static void intel_pstate_hwp_set(unsigned int cpu)
+{
+ struct cpudata *cpu_data = all_cpu_data[cpu];
+ int max, min;
+ u64 value;
+ s16 epp;
+
+ max = cpu_data->max_perf_ratio;
+ min = cpu_data->min_perf_ratio;
- max = fp_ext_toint(hw_max * cpu_data->max_perf);
if (cpu_data->policy == CPUFREQ_POLICY_PERFORMANCE)
min = max;
- else
- min = fp_ext_toint(hw_max * cpu_data->min_perf);
rdmsrl_on_cpu(cpu, MSR_HWP_REQUEST, &value);
update_turbo_state();
pstate = intel_pstate_get_base_pstate(cpu);
- pstate = max(cpu->pstate.min_pstate,
- fp_ext_toint(pstate * cpu->max_perf));
+ pstate = max(cpu->pstate.min_pstate, cpu->max_perf_ratio);
intel_pstate_set_pstate(cpu, pstate);
}
int32_t busy_frac, boost;
int target, avg_pstate;
- if (cpu->policy == CPUFREQ_POLICY_PERFORMANCE)
- return cpu->pstate.turbo_pstate;
-
busy_frac = div_fp(sample->mperf, sample->tsc);
boost = cpu->iowait_boost;
int32_t perf_scaled, max_pstate, current_pstate, sample_ratio;
u64 duration_ns;
- if (cpu->policy == CPUFREQ_POLICY_PERFORMANCE)
- return cpu->pstate.turbo_pstate;
-
/*
* perf_scaled is the ratio of the average P-state during the last
* sampling period to the P-state requested last time (in percent).
int max_pstate = intel_pstate_get_base_pstate(cpu);
int min_pstate;
- min_pstate = max(cpu->pstate.min_pstate,
- fp_ext_toint(max_pstate * cpu->min_perf));
- max_pstate = max(min_pstate, fp_ext_toint(max_pstate * cpu->max_perf));
+ min_pstate = max(cpu->pstate.min_pstate, cpu->min_perf_ratio);
+ max_pstate = max(min_pstate, cpu->max_perf_ratio);
return clamp_t(int, pstate, min_pstate, max_pstate);
}
fp_toint(cpu->iowait_boost * 100));
}
-static void intel_pstate_update_util_hwp(struct update_util_data *data,
- u64 time, unsigned int flags)
-{
- struct cpudata *cpu = container_of(data, struct cpudata, update_util);
- u64 delta_ns = time - cpu->sample.time;
-
- if ((s64)delta_ns >= INTEL_PSTATE_HWP_SAMPLING_INTERVAL)
- intel_pstate_sample(cpu, time);
-}
-
static void intel_pstate_update_util_pid(struct update_util_data *data,
u64 time, unsigned int flags)
{
{
struct cpudata *cpu = all_cpu_data[cpu_num];
+ if (hwp_active)
+ return;
+
if (cpu->update_util_set)
return;
{
int max_freq = intel_pstate_get_max_freq(cpu);
int32_t max_policy_perf, min_policy_perf;
+ int max_state, turbo_max;
- max_policy_perf = div_ext_fp(policy->max, max_freq);
- max_policy_perf = clamp_t(int32_t, max_policy_perf, 0, int_ext_tofp(1));
+ /*
+ * HWP needs some special consideration, because on BDX the
+ * HWP_REQUEST uses abstract value to represent performance
+ * rather than pure ratios.
+ */
+ if (hwp_active) {
+ intel_pstate_get_hwp_max(cpu->cpu, &turbo_max, &max_state);
+ } else {
+ max_state = intel_pstate_get_base_pstate(cpu);
+ turbo_max = cpu->pstate.turbo_pstate;
+ }
+
+ max_policy_perf = max_state * policy->max / max_freq;
if (policy->max == policy->min) {
min_policy_perf = max_policy_perf;
} else {
- min_policy_perf = div_ext_fp(policy->min, max_freq);
+ min_policy_perf = max_state * policy->min / max_freq;
min_policy_perf = clamp_t(int32_t, min_policy_perf,
0, max_policy_perf);
}
+ pr_debug("cpu:%d max_state %d min_policy_perf:%d max_policy_perf:%d\n",
+ policy->cpu, max_state,
+ min_policy_perf, max_policy_perf);
+
/* Normalize user input to [min_perf, max_perf] */
if (per_cpu_limits) {
- cpu->min_perf = min_policy_perf;
- cpu->max_perf = max_policy_perf;
+ cpu->min_perf_ratio = min_policy_perf;
+ cpu->max_perf_ratio = max_policy_perf;
} else {
int32_t global_min, global_max;
/* Global limits are in percent of the maximum turbo P-state. */
- global_max = percent_ext_fp(global.max_perf_pct);
- global_min = percent_ext_fp(global.min_perf_pct);
- if (max_freq != cpu->pstate.turbo_freq) {
- int32_t turbo_factor;
-
- turbo_factor = div_ext_fp(cpu->pstate.turbo_pstate,
- cpu->pstate.max_pstate);
- global_min = mul_ext_fp(global_min, turbo_factor);
- global_max = mul_ext_fp(global_max, turbo_factor);
- }
+ global_max = DIV_ROUND_UP(turbo_max * global.max_perf_pct, 100);
+ global_min = DIV_ROUND_UP(turbo_max * global.min_perf_pct, 100);
global_min = clamp_t(int32_t, global_min, 0, global_max);
- cpu->min_perf = max(min_policy_perf, global_min);
- cpu->min_perf = min(cpu->min_perf, max_policy_perf);
- cpu->max_perf = min(max_policy_perf, global_max);
- cpu->max_perf = max(min_policy_perf, cpu->max_perf);
+ pr_debug("cpu:%d global_min:%d global_max:%d\n", policy->cpu,
+ global_min, global_max);
- /* Make sure min_perf <= max_perf */
- cpu->min_perf = min(cpu->min_perf, cpu->max_perf);
- }
+ cpu->min_perf_ratio = max(min_policy_perf, global_min);
+ cpu->min_perf_ratio = min(cpu->min_perf_ratio, max_policy_perf);
+ cpu->max_perf_ratio = min(max_policy_perf, global_max);
+ cpu->max_perf_ratio = max(min_policy_perf, cpu->max_perf_ratio);
- cpu->max_perf = round_up(cpu->max_perf, EXT_FRAC_BITS);
- cpu->min_perf = round_up(cpu->min_perf, EXT_FRAC_BITS);
+ /* Make sure min_perf <= max_perf */
+ cpu->min_perf_ratio = min(cpu->min_perf_ratio,
+ cpu->max_perf_ratio);
- pr_debug("cpu:%d max_perf_pct:%d min_perf_pct:%d\n", policy->cpu,
- fp_ext_toint(cpu->max_perf * 100),
- fp_ext_toint(cpu->min_perf * 100));
+ }
+ pr_debug("cpu:%d max_perf_ratio:%d min_perf_ratio:%d\n", policy->cpu,
+ cpu->max_perf_ratio,
+ cpu->min_perf_ratio);
}
static int intel_pstate_set_policy(struct cpufreq_policy *policy)
*/
intel_pstate_clear_update_util_hook(policy->cpu);
intel_pstate_max_within_limits(cpu);
+ } else {
+ intel_pstate_set_update_util_hook(policy->cpu);
}
- intel_pstate_set_update_util_hook(policy->cpu);
-
if (hwp_active)
intel_pstate_hwp_set(policy->cpu);
cpu = all_cpu_data[policy->cpu];
- cpu->max_perf = int_ext_tofp(1);
- cpu->min_perf = 0;
+ cpu->max_perf_ratio = 0xFF;
+ cpu->min_perf_ratio = 0;
policy->min = cpu->pstate.min_pstate * cpu->pstate.scaling;
policy->max = cpu->pstate.turbo_pstate * cpu->pstate.scaling;
} else {
hwp_active++;
intel_pstate.attr = hwp_cpufreq_attrs;
- pstate_funcs.update_util = intel_pstate_update_util_hwp;
goto hwp_cpu_matched;
}
} else {
* We don't support CPU hotplug. Don't unmap after the system
* has already made it to a running state.
*/
- if (system_state != SYSTEM_BOOTING)
+ if (system_state >= SYSTEM_RUNNING)
return 0;
if (sdcasr_mapbase)
#include <asm/msr.h>
-struct cpufreq_frequency_table *freq_table;
+static struct cpufreq_frequency_table *freq_table;
static struct sfi_freq_table_entry *sfi_cpufreq_array;
static int num_freq_table_entries;
config ARM_CPUIDLE
bool "Generic ARM/ARM64 CPU idle Driver"
select DT_IDLE_STATES
+ select CPU_IDLE_MULTIPLE_DRIVERS
help
Select this to enable generic cpuidle driver for ARM.
It provides a generic idle driver whose idle states are configured
#include <linux/module.h>
#include <linux/of.h>
#include <linux/slab.h>
+#include <linux/topology.h>
#include <asm/cpuidle.h>
return CPU_PM_CPU_IDLE_ENTER(arm_cpuidle_suspend, idx);
}
-static struct cpuidle_driver arm_idle_driver = {
+static struct cpuidle_driver arm_idle_driver __initdata = {
.name = "arm_idle",
.owner = THIS_MODULE,
/*
static int __init arm_idle_init(void)
{
int cpu, ret;
- struct cpuidle_driver *drv = &arm_idle_driver;
+ struct cpuidle_driver *drv;
struct cpuidle_device *dev;
- /*
- * Initialize idle states data, starting at index 1.
- * This driver is DT only, if no DT idle states are detected (ret == 0)
- * let the driver initialization fail accordingly since there is no
- * reason to initialize the idle driver if only wfi is supported.
- */
- ret = dt_init_idle_driver(drv, arm_idle_state_match, 1);
- if (ret <= 0)
- return ret ? : -ENODEV;
-
- ret = cpuidle_register_driver(drv);
- if (ret) {
- pr_err("Failed to register cpuidle driver\n");
- return ret;
- }
-
- /*
- * Call arch CPU operations in order to initialize
- * idle states suspend back-end specific data
- */
for_each_possible_cpu(cpu) {
+
+ drv = kmemdup(&arm_idle_driver, sizeof(*drv), GFP_KERNEL);
+ if (!drv) {
+ ret = -ENOMEM;
+ goto out_fail;
+ }
+
+ drv->cpumask = (struct cpumask *)cpumask_of(cpu);
+
+ /*
+ * Initialize idle states data, starting at index 1. This
+ * driver is DT only, if no DT idle states are detected (ret
+ * == 0) let the driver initialization fail accordingly since
+ * there is no reason to initialize the idle driver if only
+ * wfi is supported.
+ */
+ ret = dt_init_idle_driver(drv, arm_idle_state_match, 1);
+ if (ret <= 0) {
+ ret = ret ? : -ENODEV;
+ goto out_fail;
+ }
+
+ ret = cpuidle_register_driver(drv);
+ if (ret) {
+ pr_err("Failed to register cpuidle driver\n");
+ goto out_fail;
+ }
+
+ /*
+ * Call arch CPU operations in order to initialize
+ * idle states suspend back-end specific data
+ */
ret = arm_cpuidle_init(cpu);
/*
dev = per_cpu(cpuidle_devices, cpu);
cpuidle_unregister_device(dev);
kfree(dev);
+ drv = cpuidle_get_driver();
+ cpuidle_unregister_driver(drv);
+ kfree(drv);
}
- cpuidle_unregister_driver(drv);
-
return ret;
}
device_initcall(arm_idle_init);
entered_state = target_state->enter(dev, drv, index);
start_critical_timings();
+ sched_clock_idle_wakeup_event();
time_end = ns_to_ktime(local_clock());
trace_cpu_idle_rcuidle(PWR_EVENT_EXIT, dev->cpu);
struct device *device = get_cpu_device(dev->cpu);
int latency_req = pm_qos_request(PM_QOS_CPU_DMA_LATENCY);
int i;
+ int first_idx;
+ int idx;
unsigned int interactivity_req;
unsigned int expected_interval;
unsigned long nr_iowaiters, cpu_load;
if (data->next_timer_us > polling_threshold &&
latency_req > s->exit_latency && !s->disabled &&
!dev->states_usage[CPUIDLE_DRIVER_STATE_START].disable)
- data->last_state_idx = CPUIDLE_DRIVER_STATE_START;
+ first_idx = CPUIDLE_DRIVER_STATE_START;
else
- data->last_state_idx = CPUIDLE_DRIVER_STATE_START - 1;
+ first_idx = CPUIDLE_DRIVER_STATE_START - 1;
} else {
- data->last_state_idx = CPUIDLE_DRIVER_STATE_START;
+ first_idx = 0;
}
/*
* Find the idle state with the lowest power while satisfying
* our constraints.
*/
- for (i = data->last_state_idx + 1; i < drv->state_count; i++) {
+ idx = -1;
+ for (i = first_idx; i < drv->state_count; i++) {
struct cpuidle_state *s = &drv->states[i];
struct cpuidle_state_usage *su = &dev->states_usage[i];
if (s->disabled || su->disable)
continue;
+ if (idx == -1)
+ idx = i; /* first enabled state */
if (s->target_residency > data->predicted_us)
break;
if (s->exit_latency > latency_req)
break;
- data->last_state_idx = i;
+ idx = i;
}
+ if (idx == -1)
+ idx = 0; /* No states enabled. Must use 0. */
+
+ data->last_state_idx = idx;
+
return data->last_state_idx;
}
requires to have at least one CEX card in coprocessor mode
available at runtime.
+config CRYPTO_PAES_S390
+ tristate "PAES cipher algorithms"
+ depends on S390
+ depends on ZCRYPT
+ depends on PKEY
+ select CRYPTO_ALGAPI
+ select CRYPTO_BLKCIPHER
+ help
+ This is the s390 hardware accelerated implementation of the
+ AES cipher algorithms for use with protected key.
+
+ Select this option if you want to use the paes cipher
+ for example to use protected key encrypted devices.
+
config CRYPTO_SHA1_S390
tristate "SHA1 digest algorithm"
depends on S390
depends on S390
select CRYPTO_ALGAPI
select CRYPTO_BLKCIPHER
- select PKEY
help
This is the s390 hardware accelerated implementation of the
AES cipher algorithms (FIPS-197).
return NULL;
}
- /* When the port_window is used, one frame must cover the window */
- if (port_window) {
- burst = port_window;
- port_window_bytes = port_window * es_bytes[es];
- }
-
/* Now allocate and setup the descriptor. */
d = kzalloc(sizeof(*d) + sglen * sizeof(d->sg[0]), GFP_ATOMIC);
if (!d)
d->dev_addr = dev_addr;
d->es = es;
+ /* When the port_window is used, one frame must cover the window */
+ if (port_window) {
+ burst = port_window;
+ port_window_bytes = port_window * es_bytes[es];
+
+ d->ei = 1;
+ /*
+ * One frame covers the port_window and by configure
+ * the source frame index to be -1 * (port_window - 1)
+ * we instruct the sDMA that after a frame is processed
+ * it should move back to the start of the window.
+ */
+ d->fi = -(port_window_bytes - 1);
+ }
+
d->ccr = c->ccr | CCR_SYNC_FRAME;
if (dir == DMA_DEV_TO_MEM) {
d->csdp = CSDP_DST_BURST_64 | CSDP_DST_PACKED;
d->ccr |= CCR_SRC_AMODE_POSTINC;
if (port_window) {
d->ccr |= CCR_DST_AMODE_DBLIDX;
- d->ei = 1;
- /*
- * One frame covers the port_window and by configure
- * the source frame index to be -1 * (port_window - 1)
- * we instruct the sDMA that after a frame is processed
- * it should move back to the start of the window.
- */
- d->fi = -(port_window_bytes - 1);
if (port_window_bytes >= 64)
d->csdp |= CSDP_DST_BURST_64;
osg->addr = sg_dma_address(sgent);
osg->en = en;
osg->fn = sg_dma_len(sgent) / frame_bytes;
- if (port_window && dir == DMA_DEV_TO_MEM) {
- osg->ei = 1;
- /*
- * One frame covers the port_window and by configure
- * the source frame index to be -1 * (port_window - 1)
- * we instruct the sDMA that after a frame is processed
- * it should move back to the start of the window.
- */
- osg->fi = -(port_window_bytes - 1);
- }
if (d->using_ll) {
osg->t2_desc = dma_pool_alloc(od->desc_pool, GFP_ATOMIC,
static unsigned long get_total_mem(void)
{
struct device_node *np = NULL;
- const unsigned int *reg, *reg_end;
- int len, sw, aw;
- unsigned long start, size, total_mem = 0;
+ struct resource res;
+ int ret;
+ unsigned long total_mem = 0;
for_each_node_by_type(np, "memory") {
- aw = of_n_addr_cells(np);
- sw = of_n_size_cells(np);
- reg = (const unsigned int *)of_get_property(np, "reg", &len);
- reg_end = reg + (len / sizeof(u32));
-
- total_mem = 0;
- do {
- start = of_read_number(reg, aw);
- reg += aw;
- size = of_read_number(reg, sw);
- reg += sw;
- total_mem += size;
- } while (reg < reg_end);
+ ret = of_address_to_resource(np, 0, &res);
+ if (ret)
+ continue;
+
+ total_mem += resource_size(&res);
}
edac_dbg(0, "total_mem 0x%lx\n", total_mem);
return total_mem;
return 0;
}
-static struct irq_domain_ops a10_eccmgr_ic_ops = {
+static const struct irq_domain_ops a10_eccmgr_ic_ops = {
.map = a10_eccmgr_irqdomain_map,
.xlate = irq_domain_xlate_twocell,
};
#define NREC_RDWR(x) (((x)>>11) & 1)
#define NREC_RANK(x) (((x)>>8) & 0x7)
#define NRECMEMB 0xC0
-#define NREC_CAS(x) (((x)>>16) & 0xFFFFFF)
+#define NREC_CAS(x) (((x)>>16) & 0xFFF)
#define NREC_RAS(x) ((x) & 0x7FFF)
#define NRECFGLOG 0xC4
#define NREEECFBDA 0xC8
/* These registers are input ONLY if there was a
* Non-Recoverable Error */
u16 nrecmema; /* Non-Recoverable Mem log A */
- u16 nrecmemb; /* Non-Recoverable Mem log B */
+ u32 nrecmemb; /* Non-Recoverable Mem log B */
};
NERR_FAT_FBD, &info->nerr_fat_fbd);
pci_read_config_word(pvt->branchmap_werrors,
NRECMEMA, &info->nrecmema);
- pci_read_config_word(pvt->branchmap_werrors,
+ pci_read_config_dword(pvt->branchmap_werrors,
NRECMEMB, &info->nrecmemb);
/* Clear the error bits, by writing them back */
/* These registers are input ONLY if there was a Non-Rec Error */
u16 nrecmema; /* Non-Recoverable Mem log A */
- u16 nrecmemb; /* Non-Recoverable Mem log B */
+ u32 nrecmemb; /* Non-Recoverable Mem log B */
};
NERR_FAT_FBD, &info->nerr_fat_fbd);
pci_read_config_word(pvt->branchmap_werrors,
NRECMEMA, &info->nrecmema);
- pci_read_config_word(pvt->branchmap_werrors,
+ pci_read_config_dword(pvt->branchmap_werrors,
NRECMEMB, &info->nrecmemb);
/* Clear the error bits, by writing them back */
* 0c04: Xeon E3-1200 v3/4th Gen Core Processor DRAM Controller
* 0c08: Xeon E3-1200 v3 Processor DRAM Controller
* 1918: Xeon E3-1200 v5 Skylake Host Bridge/DRAM Registers
+ * 5918: Xeon E3-1200 Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers
*
* Based on Intel specification:
* http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e3-1200v3-vol-2-datasheet.pdf
* http://www.intel.com/content/www/us/en/processors/xeon/xeon-e3-1200-family-vol-2-datasheet.html
+ * http://www.intel.com/content/www/us/en/processors/core/7th-gen-core-family-mobile-h-processor-lines-datasheet-vol-2.html
*
* According to the above datasheet (p.16):
* "
#define PCI_DEVICE_ID_INTEL_IE31200_HB_6 0x0c04
#define PCI_DEVICE_ID_INTEL_IE31200_HB_7 0x0c08
#define PCI_DEVICE_ID_INTEL_IE31200_HB_8 0x1918
+#define PCI_DEVICE_ID_INTEL_IE31200_HB_9 0x5918
#define IE31200_DIMMS 4
#define IE31200_RANKS 8
void __iomem *window;
struct ie31200_priv *priv;
u32 addr_decode, mad_offset;
- bool skl = (pdev->device == PCI_DEVICE_ID_INTEL_IE31200_HB_8);
+
+ /*
+ * Kaby Lake seems to work like Skylake. Please re-visit this logic
+ * when adding new CPU support.
+ */
+ bool skl = (pdev->device >= PCI_DEVICE_ID_INTEL_IE31200_HB_8);
edac_dbg(0, "MC:\n");
{
PCI_VEND_DEV(INTEL, IE31200_HB_8), PCI_ANY_ID, PCI_ANY_ID, 0, 0,
IE31200},
+ {
+ PCI_VEND_DEV(INTEL, IE31200_HB_9), PCI_ANY_ID, PCI_ANY_ID, 0, 0,
+ IE31200},
{
0,
} /* 0 terminated list. */
"Sys Read data error thread 0",
"Sys read data error thread 1",
"DC tag error type 2",
- "DC data error type 1 (poison comsumption)",
+ "DC data error type 1 (poison consumption)",
"DC data error type 2",
"DC data error type 3",
"DC tag error type 4",
struct mv64x60_pci_pdata *pdata = pci->pvt_info;
u32 cause;
- cause = in_le32(pdata->pci_vbase + MV64X60_PCI_ERROR_CAUSE);
+ cause = readl(pdata->pci_vbase + MV64X60_PCI_ERROR_CAUSE);
if (!cause)
return;
printk(KERN_ERR "Error in PCI %d Interface\n", pdata->pci_hose);
printk(KERN_ERR "Cause register: 0x%08x\n", cause);
printk(KERN_ERR "Address Low: 0x%08x\n",
- in_le32(pdata->pci_vbase + MV64X60_PCI_ERROR_ADDR_LO));
+ readl(pdata->pci_vbase + MV64X60_PCI_ERROR_ADDR_LO));
printk(KERN_ERR "Address High: 0x%08x\n",
- in_le32(pdata->pci_vbase + MV64X60_PCI_ERROR_ADDR_HI));
+ readl(pdata->pci_vbase + MV64X60_PCI_ERROR_ADDR_HI));
printk(KERN_ERR "Attribute: 0x%08x\n",
- in_le32(pdata->pci_vbase + MV64X60_PCI_ERROR_ATTR));
+ readl(pdata->pci_vbase + MV64X60_PCI_ERROR_ATTR));
printk(KERN_ERR "Command: 0x%08x\n",
- in_le32(pdata->pci_vbase + MV64X60_PCI_ERROR_CMD));
- out_le32(pdata->pci_vbase + MV64X60_PCI_ERROR_CAUSE, ~cause);
+ readl(pdata->pci_vbase + MV64X60_PCI_ERROR_CMD));
+ writel(~cause, pdata->pci_vbase + MV64X60_PCI_ERROR_CAUSE);
if (cause & MV64X60_PCI_PE_MASK)
edac_pci_handle_pe(pci, pci->ctl_name);
struct mv64x60_pci_pdata *pdata = pci->pvt_info;
u32 val;
- val = in_le32(pdata->pci_vbase + MV64X60_PCI_ERROR_CAUSE);
+ val = readl(pdata->pci_vbase + MV64X60_PCI_ERROR_CAUSE);
if (!val)
return IRQ_NONE;
if (!pci_serr)
return -ENOMEM;
- out_le32(pci_serr, in_le32(pci_serr) & ~0x1);
+ writel(readl(pci_serr) & ~0x1, pci_serr);
iounmap(pci_serr);
return 0;
pdata = pci->pvt_info;
pdata->pci_hose = pdev->id;
- pdata->name = "mpc85xx_pci_err";
+ pdata->name = "mv64x60_pci_err";
platform_set_drvdata(pdev, pci);
pci->dev = &pdev->dev;
pci->dev_name = dev_name(&pdev->dev);
goto err;
}
- out_le32(pdata->pci_vbase + MV64X60_PCI_ERROR_CAUSE, 0);
- out_le32(pdata->pci_vbase + MV64X60_PCI_ERROR_MASK, 0);
- out_le32(pdata->pci_vbase + MV64X60_PCI_ERROR_MASK,
- MV64X60_PCIx_ERR_MASK_VAL);
+ writel(0, pdata->pci_vbase + MV64X60_PCI_ERROR_CAUSE);
+ writel(0, pdata->pci_vbase + MV64X60_PCI_ERROR_MASK);
+ writel(MV64X60_PCIx_ERR_MASK_VAL,
+ pdata->pci_vbase + MV64X60_PCI_ERROR_MASK);
if (edac_pci_add_device(pci, pdata->edac_idx) > 0) {
edac_dbg(3, "failed edac_pci_add_device()\n");
struct mv64x60_sram_pdata *pdata = edac_dev->pvt_info;
u32 cause;
- cause = in_le32(pdata->sram_vbase + MV64X60_SRAM_ERR_CAUSE);
+ cause = readl(pdata->sram_vbase + MV64X60_SRAM_ERR_CAUSE);
if (!cause)
return;
printk(KERN_ERR "Error in internal SRAM\n");
printk(KERN_ERR "Cause register: 0x%08x\n", cause);
printk(KERN_ERR "Address Low: 0x%08x\n",
- in_le32(pdata->sram_vbase + MV64X60_SRAM_ERR_ADDR_LO));
+ readl(pdata->sram_vbase + MV64X60_SRAM_ERR_ADDR_LO));
printk(KERN_ERR "Address High: 0x%08x\n",
- in_le32(pdata->sram_vbase + MV64X60_SRAM_ERR_ADDR_HI));
+ readl(pdata->sram_vbase + MV64X60_SRAM_ERR_ADDR_HI));
printk(KERN_ERR "Data Low: 0x%08x\n",
- in_le32(pdata->sram_vbase + MV64X60_SRAM_ERR_DATA_LO));
+ readl(pdata->sram_vbase + MV64X60_SRAM_ERR_DATA_LO));
printk(KERN_ERR "Data High: 0x%08x\n",
- in_le32(pdata->sram_vbase + MV64X60_SRAM_ERR_DATA_HI));
+ readl(pdata->sram_vbase + MV64X60_SRAM_ERR_DATA_HI));
printk(KERN_ERR "Parity: 0x%08x\n",
- in_le32(pdata->sram_vbase + MV64X60_SRAM_ERR_PARITY));
- out_le32(pdata->sram_vbase + MV64X60_SRAM_ERR_CAUSE, 0);
+ readl(pdata->sram_vbase + MV64X60_SRAM_ERR_PARITY));
+ writel(0, pdata->sram_vbase + MV64X60_SRAM_ERR_CAUSE);
edac_device_handle_ue(edac_dev, 0, 0, edac_dev->ctl_name);
}
struct mv64x60_sram_pdata *pdata = edac_dev->pvt_info;
u32 cause;
- cause = in_le32(pdata->sram_vbase + MV64X60_SRAM_ERR_CAUSE);
+ cause = readl(pdata->sram_vbase + MV64X60_SRAM_ERR_CAUSE);
if (!cause)
return IRQ_NONE;
}
/* setup SRAM err registers */
- out_le32(pdata->sram_vbase + MV64X60_SRAM_ERR_CAUSE, 0);
+ writel(0, pdata->sram_vbase + MV64X60_SRAM_ERR_CAUSE);
edac_dev->mod_name = EDAC_MOD_STR;
edac_dev->ctl_name = pdata->name;
struct mv64x60_cpu_pdata *pdata = edac_dev->pvt_info;
u32 cause;
- cause = in_le32(pdata->cpu_vbase[1] + MV64x60_CPU_ERR_CAUSE) &
+ cause = readl(pdata->cpu_vbase[1] + MV64x60_CPU_ERR_CAUSE) &
MV64x60_CPU_CAUSE_MASK;
if (!cause)
return;
printk(KERN_ERR "Error on CPU interface\n");
printk(KERN_ERR "Cause register: 0x%08x\n", cause);
printk(KERN_ERR "Address Low: 0x%08x\n",
- in_le32(pdata->cpu_vbase[0] + MV64x60_CPU_ERR_ADDR_LO));
+ readl(pdata->cpu_vbase[0] + MV64x60_CPU_ERR_ADDR_LO));
printk(KERN_ERR "Address High: 0x%08x\n",
- in_le32(pdata->cpu_vbase[0] + MV64x60_CPU_ERR_ADDR_HI));
+ readl(pdata->cpu_vbase[0] + MV64x60_CPU_ERR_ADDR_HI));
printk(KERN_ERR "Data Low: 0x%08x\n",
- in_le32(pdata->cpu_vbase[1] + MV64x60_CPU_ERR_DATA_LO));
+ readl(pdata->cpu_vbase[1] + MV64x60_CPU_ERR_DATA_LO));
printk(KERN_ERR "Data High: 0x%08x\n",
- in_le32(pdata->cpu_vbase[1] + MV64x60_CPU_ERR_DATA_HI));
+ readl(pdata->cpu_vbase[1] + MV64x60_CPU_ERR_DATA_HI));
printk(KERN_ERR "Parity: 0x%08x\n",
- in_le32(pdata->cpu_vbase[1] + MV64x60_CPU_ERR_PARITY));
- out_le32(pdata->cpu_vbase[1] + MV64x60_CPU_ERR_CAUSE, 0);
+ readl(pdata->cpu_vbase[1] + MV64x60_CPU_ERR_PARITY));
+ writel(0, pdata->cpu_vbase[1] + MV64x60_CPU_ERR_CAUSE);
edac_device_handle_ue(edac_dev, 0, 0, edac_dev->ctl_name);
}
struct mv64x60_cpu_pdata *pdata = edac_dev->pvt_info;
u32 cause;
- cause = in_le32(pdata->cpu_vbase[1] + MV64x60_CPU_ERR_CAUSE) &
+ cause = readl(pdata->cpu_vbase[1] + MV64x60_CPU_ERR_CAUSE) &
MV64x60_CPU_CAUSE_MASK;
if (!cause)
return IRQ_NONE;
}
/* setup CPU err registers */
- out_le32(pdata->cpu_vbase[1] + MV64x60_CPU_ERR_CAUSE, 0);
- out_le32(pdata->cpu_vbase[1] + MV64x60_CPU_ERR_MASK, 0);
- out_le32(pdata->cpu_vbase[1] + MV64x60_CPU_ERR_MASK, 0x000000ff);
+ writel(0, pdata->cpu_vbase[1] + MV64x60_CPU_ERR_CAUSE);
+ writel(0, pdata->cpu_vbase[1] + MV64x60_CPU_ERR_MASK);
+ writel(0x000000ff, pdata->cpu_vbase[1] + MV64x60_CPU_ERR_MASK);
edac_dev->mod_name = EDAC_MOD_STR;
edac_dev->ctl_name = pdata->name;
u32 comp_ecc;
u32 syndrome;
- reg = in_le32(pdata->mc_vbase + MV64X60_SDRAM_ERR_ADDR);
+ reg = readl(pdata->mc_vbase + MV64X60_SDRAM_ERR_ADDR);
if (!reg)
return;
err_addr = reg & ~0x3;
- sdram_ecc = in_le32(pdata->mc_vbase + MV64X60_SDRAM_ERR_ECC_RCVD);
- comp_ecc = in_le32(pdata->mc_vbase + MV64X60_SDRAM_ERR_ECC_CALC);
+ sdram_ecc = readl(pdata->mc_vbase + MV64X60_SDRAM_ERR_ECC_RCVD);
+ comp_ecc = readl(pdata->mc_vbase + MV64X60_SDRAM_ERR_ECC_CALC);
syndrome = sdram_ecc ^ comp_ecc;
/* first bit clear in ECC Err Reg, 1 bit error, correctable by HW */
mci->ctl_name, "");
/* clear the error */
- out_le32(pdata->mc_vbase + MV64X60_SDRAM_ERR_ADDR, 0);
+ writel(0, pdata->mc_vbase + MV64X60_SDRAM_ERR_ADDR);
}
static irqreturn_t mv64x60_mc_isr(int irq, void *dev_id)
struct mv64x60_mc_pdata *pdata = mci->pvt_info;
u32 reg;
- reg = in_le32(pdata->mc_vbase + MV64X60_SDRAM_ERR_ADDR);
+ reg = readl(pdata->mc_vbase + MV64X60_SDRAM_ERR_ADDR);
if (!reg)
return IRQ_NONE;
get_total_mem(pdata);
- ctl = in_le32(pdata->mc_vbase + MV64X60_SDRAM_CONFIG);
+ ctl = readl(pdata->mc_vbase + MV64X60_SDRAM_CONFIG);
csrow = mci->csrows[0];
dimm = csrow->channels[0]->dimm;
goto err;
}
- ctl = in_le32(pdata->mc_vbase + MV64X60_SDRAM_CONFIG);
+ ctl = readl(pdata->mc_vbase + MV64X60_SDRAM_CONFIG);
if (!(ctl & MV64X60_SDRAM_ECC)) {
/* Non-ECC RAM? */
printk(KERN_WARNING "%s: No ECC DIMMs discovered\n", __func__);
mv64x60_init_csrows(mci, pdata);
/* setup MC registers */
- out_le32(pdata->mc_vbase + MV64X60_SDRAM_ERR_ADDR, 0);
- ctl = in_le32(pdata->mc_vbase + MV64X60_SDRAM_ERR_ECC_CNTL);
+ writel(0, pdata->mc_vbase + MV64X60_SDRAM_ERR_ADDR);
+ ctl = readl(pdata->mc_vbase + MV64X60_SDRAM_ERR_ECC_CNTL);
ctl = (ctl & 0xff00ffff) | 0x10000;
- out_le32(pdata->mc_vbase + MV64X60_SDRAM_ERR_ECC_CNTL, ctl);
+ writel(ctl, pdata->mc_vbase + MV64X60_SDRAM_ERR_ECC_CNTL);
res = edac_mc_add_mc(mci);
if (res) {
static int __init mv64x60_edac_init(void)
{
- int ret = 0;
printk(KERN_INFO "Marvell MV64x60 EDAC driver " MV64x60_REVISION "\n");
printk(KERN_INFO "\t(C) 2006-2007 MontaVista Software\n");
+
/* make sure error reporting method is sane */
switch (edac_op_state) {
case EDAC_OPSTATE_POLL:
#ifdef CONFIG_X86_INTEL_SBI_APL
#include "linux/platform_data/sbi_apl.h"
-int sbi_send(int port, int off, int op, u32 *data)
+static int sbi_send(int port, int off, int op, u32 *data)
{
struct sbi_apl_message sbi_arg;
int ret, read = 0;
return ret;
}
#else
-int sbi_send(int port, int off, int op, u32 *data)
+static int sbi_send(int port, int off, int op, u32 *data)
{
return -EUNATCH;
}
static int apl_rd_reg(int port, int off, int op, void *data, size_t sz, char *name)
{
- int ret = 0;
+ int ret = 0;
edac_dbg(2, "Read %s port=%x off=%x op=%x\n", name, port, off, op);
switch (sz) {
case 8:
ret = sbi_send(port, off + 4, op, (u32 *)(data + 4));
+ /* fall through */
case 4:
- ret = sbi_send(port, off, op, (u32 *)data);
+ ret |= sbi_send(port, off, op, (u32 *)data);
pnd2_printk(KERN_DEBUG, "%s=%x%08x ret=%d\n", name,
sz == 8 ? *((u32 *)(data + 4)) : 0, *((u32 *)data), ret);
break;
static int apl_get_registers(void)
{
+ int ret = -ENODEV;
int i;
if (RD_REG(&asym_2way, b_cr_asym_2way_mem_region_mchbar))
return -ENODEV;
+ /*
+ * RD_REGP() will fail for unpopulated or non-existent
+ * DIMM slots. Return success if we find at least one DIMM.
+ */
for (i = 0; i < APL_NUM_CHANNELS; i++)
- if (RD_REGP(&drp0[i], d_cr_drp0, apl_dports[i]))
- return -ENODEV;
+ if (!RD_REGP(&drp0[i], d_cr_drp0, apl_dports[i]))
+ ret = 0;
- return 0;
+ return ret;
}
static int dnv_get_registers(void)
/*
* Alter this version for the module when modifications are made
*/
-#define SBRIDGE_REVISION " Ver: 1.1.1 "
+#define SBRIDGE_REVISION " Ver: 1.1.2 "
#define EDAC_MOD_STR "sbridge_edac"
/*
* sbridge structs
*/
-#define NUM_CHANNELS 8 /* 2MC per socket, four chan per MC */
+#define NUM_CHANNELS 4 /* Max channels per MC */
#define MAX_DIMMS 3 /* Max DIMMS per channel */
#define KNL_MAX_CHAS 38 /* KNL max num. of Cache Home Agents */
#define KNL_MAX_CHANNELS 6 /* KNL max num. of PCI channels */
KNIGHTS_LANDING,
};
+enum domain {
+ IMC0 = 0,
+ IMC1,
+ SOCK,
+};
+
struct sbridge_pvt;
struct sbridge_info {
enum type type;
struct pci_id_descr {
int dev_id;
int optional;
+ enum domain dom;
};
struct pci_id_table {
const struct pci_id_descr *descr;
- int n_devs;
+ int n_devs_per_imc;
+ int n_devs_per_sock;
+ int n_imcs_per_sock;
enum type type;
};
u8 bus, mc;
u8 node_id, source_id;
struct pci_dev **pdev;
+ enum domain dom;
int n_devs;
+ int i_devs;
struct mem_ctl_info *mci;
};
};
struct sbridge_pvt {
- struct pci_dev *pci_ta, *pci_ddrio, *pci_ras;
+ /* Devices per socket */
+ struct pci_dev *pci_ddrio;
struct pci_dev *pci_sad0, *pci_sad1;
- struct pci_dev *pci_ha0, *pci_ha1;
struct pci_dev *pci_br0, *pci_br1;
- struct pci_dev *pci_ha1_ta;
+ /* Devices per memory controller */
+ struct pci_dev *pci_ha, *pci_ta, *pci_ras;
struct pci_dev *pci_tad[NUM_CHANNELS];
struct sbridge_dev *sbridge_dev;
struct knl_pvt knl;
};
-#define PCI_DESCR(device_id, opt) \
+#define PCI_DESCR(device_id, opt, domain) \
.dev_id = (device_id), \
- .optional = opt
+ .optional = opt, \
+ .dom = domain
static const struct pci_id_descr pci_dev_descr_sbridge[] = {
/* Processor Home Agent */
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_HA0, 0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_HA0, 0, IMC0) },
/* Memory controller */
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TA, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_RAS, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD0, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD1, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD2, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD3, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_DDRIO, 1) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TA, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_RAS, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD0, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD1, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD2, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD3, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_DDRIO, 1, SOCK) },
/* System Address Decoder */
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_SAD0, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_SAD1, 0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_SAD0, 0, SOCK) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_SAD1, 0, SOCK) },
/* Broadcast Registers */
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_BR, 0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_SBRIDGE_BR, 0, SOCK) },
};
-#define PCI_ID_TABLE_ENTRY(A, T) { \
+#define PCI_ID_TABLE_ENTRY(A, N, M, T) { \
.descr = A, \
- .n_devs = ARRAY_SIZE(A), \
+ .n_devs_per_imc = N, \
+ .n_devs_per_sock = ARRAY_SIZE(A), \
+ .n_imcs_per_sock = M, \
.type = T \
}
static const struct pci_id_table pci_dev_descr_sbridge_table[] = {
- PCI_ID_TABLE_ENTRY(pci_dev_descr_sbridge, SANDY_BRIDGE),
+ PCI_ID_TABLE_ENTRY(pci_dev_descr_sbridge, ARRAY_SIZE(pci_dev_descr_sbridge), 1, SANDY_BRIDGE),
{0,} /* 0 terminated list. */
};
static const struct pci_id_descr pci_dev_descr_ibridge[] = {
/* Processor Home Agent */
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0, 0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0, 0, IMC0) },
/* Memory controller */
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_TA, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_RAS, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_TAD0, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_TAD1, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_TAD2, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_TAD3, 0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_TA, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_RAS, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_TAD0, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_TAD1, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_TAD2, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_TAD3, 0, IMC0) },
+
+ /* Optional, mode 2HA */
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1, 1, IMC1) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TA, 1, IMC1) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_RAS, 1, IMC1) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TAD0, 1, IMC1) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TAD1, 1, IMC1) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TAD2, 1, IMC1) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TAD3, 1, IMC1) },
+
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_1HA_DDRIO0, 1, SOCK) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_2HA_DDRIO0, 1, SOCK) },
/* System Address Decoder */
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_SAD, 0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_SAD, 0, SOCK) },
/* Broadcast Registers */
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_BR0, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_BR1, 0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_BR0, 1, SOCK) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_BR1, 0, SOCK) },
- /* Optional, mode 2HA */
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1, 1) },
-#if 0
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TA, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_RAS, 1) },
-#endif
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TAD0, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TAD1, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TAD2, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TAD3, 1) },
-
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_1HA_DDRIO0, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_2HA_DDRIO0, 1) },
};
static const struct pci_id_table pci_dev_descr_ibridge_table[] = {
- PCI_ID_TABLE_ENTRY(pci_dev_descr_ibridge, IVY_BRIDGE),
+ PCI_ID_TABLE_ENTRY(pci_dev_descr_ibridge, 12, 2, IVY_BRIDGE),
{0,} /* 0 terminated list. */
};
#define PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0 0x2fa0
#define PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1 0x2f60
#define PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TA 0x2fa8
-#define PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_THERMAL 0x2f71
+#define PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TM 0x2f71
#define PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TA 0x2f68
-#define PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_THERMAL 0x2f79
+#define PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TM 0x2f79
#define PCI_DEVICE_ID_INTEL_HASWELL_IMC_CBO_SAD0 0x2ffc
#define PCI_DEVICE_ID_INTEL_HASWELL_IMC_CBO_SAD1 0x2ffd
#define PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TAD0 0x2faa
#define PCI_DEVICE_ID_INTEL_HASWELL_IMC_DDRIO3 0x2fbb
static const struct pci_id_descr pci_dev_descr_haswell[] = {
/* first item must be the HA */
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0, 0) },
-
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_CBO_SAD0, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_CBO_SAD1, 0) },
-
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1, 1) },
-
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TA, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_THERMAL, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TAD0, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TAD1, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TAD2, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TAD3, 1) },
-
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_DDRIO0, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_DDRIO1, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_DDRIO2, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_DDRIO3, 1) },
-
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TA, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_THERMAL, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TAD0, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TAD1, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TAD2, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TAD3, 1) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1, 1, IMC1) },
+
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TA, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TM, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TAD0, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TAD1, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TAD2, 1, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TAD3, 1, IMC0) },
+
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TA, 1, IMC1) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TM, 1, IMC1) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TAD0, 1, IMC1) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TAD1, 1, IMC1) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TAD2, 1, IMC1) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TAD3, 1, IMC1) },
+
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_CBO_SAD0, 0, SOCK) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_CBO_SAD1, 0, SOCK) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_DDRIO0, 1, SOCK) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_DDRIO1, 1, SOCK) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_DDRIO2, 1, SOCK) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_HASWELL_IMC_DDRIO3, 1, SOCK) },
};
static const struct pci_id_table pci_dev_descr_haswell_table[] = {
- PCI_ID_TABLE_ENTRY(pci_dev_descr_haswell, HASWELL),
+ PCI_ID_TABLE_ENTRY(pci_dev_descr_haswell, 13, 2, HASWELL),
{0,} /* 0 terminated list. */
};
/* Memory controller, TAD tables, error injection - 2-8-0, 2-9-0 (2 of these) */
#define PCI_DEVICE_ID_INTEL_KNL_IMC_MC 0x7840
/* DRAM channel stuff; bank addrs, dimmmtr, etc.. 2-8-2 - 2-9-4 (6 of these) */
-#define PCI_DEVICE_ID_INTEL_KNL_IMC_CHANNEL 0x7843
+#define PCI_DEVICE_ID_INTEL_KNL_IMC_CHAN 0x7843
/* kdrwdbu TAD limits/offsets, MCMTR - 2-10-1, 2-11-1 (2 of these) */
#define PCI_DEVICE_ID_INTEL_KNL_IMC_TA 0x7844
/* CHA broadcast registers, dram rules - 1-29-0 (1 of these) */
*/
static const struct pci_id_descr pci_dev_descr_knl[] = {
- [0] = { PCI_DESCR(PCI_DEVICE_ID_INTEL_KNL_IMC_SAD0, 0) },
- [1] = { PCI_DESCR(PCI_DEVICE_ID_INTEL_KNL_IMC_SAD1, 0) },
- [2 ... 3] = { PCI_DESCR(PCI_DEVICE_ID_INTEL_KNL_IMC_MC, 0)},
- [4 ... 41] = { PCI_DESCR(PCI_DEVICE_ID_INTEL_KNL_IMC_CHA, 0) },
- [42 ... 47] = { PCI_DESCR(PCI_DEVICE_ID_INTEL_KNL_IMC_CHANNEL, 0) },
- [48] = { PCI_DESCR(PCI_DEVICE_ID_INTEL_KNL_IMC_TA, 0) },
- [49] = { PCI_DESCR(PCI_DEVICE_ID_INTEL_KNL_IMC_TOLHM, 0) },
+ [0 ... 1] = { PCI_DESCR(PCI_DEVICE_ID_INTEL_KNL_IMC_MC, 0, IMC0)},
+ [2 ... 7] = { PCI_DESCR(PCI_DEVICE_ID_INTEL_KNL_IMC_CHAN, 0, IMC0) },
+ [8] = { PCI_DESCR(PCI_DEVICE_ID_INTEL_KNL_IMC_TA, 0, IMC0) },
+ [9] = { PCI_DESCR(PCI_DEVICE_ID_INTEL_KNL_IMC_TOLHM, 0, IMC0) },
+ [10] = { PCI_DESCR(PCI_DEVICE_ID_INTEL_KNL_IMC_SAD0, 0, SOCK) },
+ [11] = { PCI_DESCR(PCI_DEVICE_ID_INTEL_KNL_IMC_SAD1, 0, SOCK) },
+ [12 ... 49] = { PCI_DESCR(PCI_DEVICE_ID_INTEL_KNL_IMC_CHA, 0, SOCK) },
};
static const struct pci_id_table pci_dev_descr_knl_table[] = {
- PCI_ID_TABLE_ENTRY(pci_dev_descr_knl, KNIGHTS_LANDING),
+ PCI_ID_TABLE_ENTRY(pci_dev_descr_knl, ARRAY_SIZE(pci_dev_descr_knl), 1, KNIGHTS_LANDING),
{0,}
};
#define PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0 0x6fa0
#define PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1 0x6f60
#define PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TA 0x6fa8
-#define PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_THERMAL 0x6f71
+#define PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TM 0x6f71
#define PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TA 0x6f68
-#define PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_THERMAL 0x6f79
+#define PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TM 0x6f79
#define PCI_DEVICE_ID_INTEL_BROADWELL_IMC_CBO_SAD0 0x6ffc
#define PCI_DEVICE_ID_INTEL_BROADWELL_IMC_CBO_SAD1 0x6ffd
#define PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TAD0 0x6faa
static const struct pci_id_descr pci_dev_descr_broadwell[] = {
/* first item must be the HA */
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0, 0) },
-
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_CBO_SAD0, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_CBO_SAD1, 0) },
-
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1, 1) },
-
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TA, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_THERMAL, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TAD0, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TAD1, 0) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TAD2, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TAD3, 1) },
-
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_DDRIO0, 1) },
-
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TA, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_THERMAL, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TAD0, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TAD1, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TAD2, 1) },
- { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TAD3, 1) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1, 1, IMC1) },
+
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TA, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TM, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TAD0, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TAD1, 0, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TAD2, 1, IMC0) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TAD3, 1, IMC0) },
+
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TA, 1, IMC1) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TM, 1, IMC1) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TAD0, 1, IMC1) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TAD1, 1, IMC1) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TAD2, 1, IMC1) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TAD3, 1, IMC1) },
+
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_CBO_SAD0, 0, SOCK) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_CBO_SAD1, 0, SOCK) },
+ { PCI_DESCR(PCI_DEVICE_ID_INTEL_BROADWELL_IMC_DDRIO0, 1, SOCK) },
};
static const struct pci_id_table pci_dev_descr_broadwell_table[] = {
- PCI_ID_TABLE_ENTRY(pci_dev_descr_broadwell, BROADWELL),
+ PCI_ID_TABLE_ENTRY(pci_dev_descr_broadwell, 10, 2, BROADWELL),
{0,} /* 0 terminated list. */
};
return 1 << cols;
}
-static struct sbridge_dev *get_sbridge_dev(u8 bus, int multi_bus)
+static struct sbridge_dev *get_sbridge_dev(u8 bus, enum domain dom, int multi_bus,
+ struct sbridge_dev *prev)
{
struct sbridge_dev *sbridge_dev;
struct sbridge_dev, list);
}
- list_for_each_entry(sbridge_dev, &sbridge_edac_list, list) {
- if (sbridge_dev->bus == bus)
+ sbridge_dev = list_entry(prev ? prev->list.next
+ : sbridge_edac_list.next, struct sbridge_dev, list);
+
+ list_for_each_entry_from(sbridge_dev, &sbridge_edac_list, list) {
+ if (sbridge_dev->bus == bus && (dom == SOCK || dom == sbridge_dev->dom))
return sbridge_dev;
}
return NULL;
}
-static struct sbridge_dev *alloc_sbridge_dev(u8 bus,
- const struct pci_id_table *table)
+static struct sbridge_dev *alloc_sbridge_dev(u8 bus, enum domain dom,
+ const struct pci_id_table *table)
{
struct sbridge_dev *sbridge_dev;
if (!sbridge_dev)
return NULL;
- sbridge_dev->pdev = kzalloc(sizeof(*sbridge_dev->pdev) * table->n_devs,
- GFP_KERNEL);
+ sbridge_dev->pdev = kcalloc(table->n_devs_per_imc,
+ sizeof(*sbridge_dev->pdev),
+ GFP_KERNEL);
if (!sbridge_dev->pdev) {
kfree(sbridge_dev);
return NULL;
}
sbridge_dev->bus = bus;
- sbridge_dev->n_devs = table->n_devs;
+ sbridge_dev->dom = dom;
+ sbridge_dev->n_devs = table->n_devs_per_imc;
list_add_tail(&sbridge_dev->list, &sbridge_edac_list);
return sbridge_dev;
return idx;
}
-/****************************************************************************
- Memory check routines
- ****************************************************************************/
-static struct pci_dev *get_pdev_same_bus(u8 bus, u32 id)
-{
- struct pci_dev *pdev = NULL;
-
- do {
- pdev = pci_get_device(PCI_VENDOR_ID_INTEL, id, pdev);
- if (pdev && pdev->bus->number == bus)
- break;
- } while (pdev);
-
- return pdev;
-}
-
-/**
- * check_if_ecc_is_active() - Checks if ECC is active
- * @bus: Device bus
- * @type: Memory controller type
- * returns: 0 in case ECC is active, -ENODEV if it can't be determined or
- * disabled
- */
-static int check_if_ecc_is_active(const u8 bus, enum type type)
-{
- struct pci_dev *pdev = NULL;
- u32 mcmtr, id;
-
- switch (type) {
- case IVY_BRIDGE:
- id = PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_TA;
- break;
- case HASWELL:
- id = PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TA;
- break;
- case SANDY_BRIDGE:
- id = PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TA;
- break;
- case BROADWELL:
- id = PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TA;
- break;
- case KNIGHTS_LANDING:
- /*
- * KNL doesn't group things by bus the same way
- * SB/IB/Haswell does.
- */
- id = PCI_DEVICE_ID_INTEL_KNL_IMC_TA;
- break;
- default:
- return -ENODEV;
- }
-
- if (type != KNIGHTS_LANDING)
- pdev = get_pdev_same_bus(bus, id);
- else
- pdev = pci_get_device(PCI_VENDOR_ID_INTEL, id, 0);
-
- if (!pdev) {
- sbridge_printk(KERN_ERR, "Couldn't find PCI device "
- "%04x:%04x! on bus %02d\n",
- PCI_VENDOR_ID_INTEL, id, bus);
- return -ENODEV;
- }
-
- pci_read_config_dword(pdev,
- type == KNIGHTS_LANDING ? KNL_MCMTR : MCMTR, &mcmtr);
- if (!IS_ECC_ENABLED(mcmtr)) {
- sbridge_printk(KERN_ERR, "ECC is disabled. Aborting\n");
- return -ENODEV;
- }
- return 0;
-}
-
/* Low bits of TAD limit, and some metadata. */
static const u32 knl_tad_dram_limit_lo[] = {
0x400, 0x500, 0x600, 0x700,
return 0;
}
-static int get_dimm_config(struct mem_ctl_info *mci)
+static void get_source_id(struct mem_ctl_info *mci)
{
struct sbridge_pvt *pvt = mci->pvt_info;
- struct dimm_info *dimm;
- unsigned i, j, banks, ranks, rows, cols, npages;
- u64 size;
u32 reg;
- enum edac_type mode;
- enum mem_type mtype;
- int channels = pvt->info.type == KNIGHTS_LANDING ?
- KNL_MAX_CHANNELS : NUM_CHANNELS;
- u64 knl_mc_sizes[KNL_MAX_CHANNELS];
- if (pvt->info.type == HASWELL || pvt->info.type == BROADWELL) {
- pci_read_config_dword(pvt->pci_ha0, HASWELL_HASYSDEFEATURE2, ®);
- pvt->is_chan_hash = GET_BITFIELD(reg, 21, 21);
- }
if (pvt->info.type == HASWELL || pvt->info.type == BROADWELL ||
- pvt->info.type == KNIGHTS_LANDING)
+ pvt->info.type == KNIGHTS_LANDING)
pci_read_config_dword(pvt->pci_sad1, SAD_TARGET, ®);
else
pci_read_config_dword(pvt->pci_br0, SAD_TARGET, ®);
pvt->sbridge_dev->source_id = SOURCE_ID_KNL(reg);
else
pvt->sbridge_dev->source_id = SOURCE_ID(reg);
+}
- pvt->sbridge_dev->node_id = pvt->info.get_node_id(pvt);
- edac_dbg(0, "mc#%d: Node ID: %d, source ID: %d\n",
- pvt->sbridge_dev->mc,
- pvt->sbridge_dev->node_id,
- pvt->sbridge_dev->source_id);
-
- /* KNL doesn't support mirroring or lockstep,
- * and is always closed page
- */
- if (pvt->info.type == KNIGHTS_LANDING) {
- mode = EDAC_S4ECD4ED;
- pvt->is_mirrored = false;
-
- if (knl_get_dimm_capacity(pvt, knl_mc_sizes) != 0)
- return -1;
- } else {
- pci_read_config_dword(pvt->pci_ras, RASENABLES, ®);
- if (IS_MIRROR_ENABLED(reg)) {
- edac_dbg(0, "Memory mirror is enabled\n");
- pvt->is_mirrored = true;
- } else {
- edac_dbg(0, "Memory mirror is disabled\n");
- pvt->is_mirrored = false;
- }
-
- pci_read_config_dword(pvt->pci_ta, MCMTR, &pvt->info.mcmtr);
- if (IS_LOCKSTEP_ENABLED(pvt->info.mcmtr)) {
- edac_dbg(0, "Lockstep is enabled\n");
- mode = EDAC_S8ECD8ED;
- pvt->is_lockstep = true;
- } else {
- edac_dbg(0, "Lockstep is disabled\n");
- mode = EDAC_S4ECD4ED;
- pvt->is_lockstep = false;
- }
- if (IS_CLOSE_PG(pvt->info.mcmtr)) {
- edac_dbg(0, "address map is on closed page mode\n");
- pvt->is_close_pg = true;
- } else {
- edac_dbg(0, "address map is on open page mode\n");
- pvt->is_close_pg = false;
- }
- }
+static int __populate_dimms(struct mem_ctl_info *mci,
+ u64 knl_mc_sizes[KNL_MAX_CHANNELS],
+ enum edac_type mode)
+{
+ struct sbridge_pvt *pvt = mci->pvt_info;
+ int channels = pvt->info.type == KNIGHTS_LANDING ? KNL_MAX_CHANNELS
+ : NUM_CHANNELS;
+ unsigned int i, j, banks, ranks, rows, cols, npages;
+ struct dimm_info *dimm;
+ enum mem_type mtype;
+ u64 size;
mtype = pvt->info.get_memory_type(pvt);
if (mtype == MEM_RDDR3 || mtype == MEM_RDDR4)
}
for (j = 0; j < max_dimms_per_channel; j++) {
- dimm = EDAC_DIMM_PTR(mci->layers, mci->dimms, mci->n_layers,
- i, j, 0);
+ dimm = EDAC_DIMM_PTR(mci->layers, mci->dimms, mci->n_layers, i, j, 0);
if (pvt->info.type == KNIGHTS_LANDING) {
pci_read_config_dword(pvt->knl.pci_channel[i],
knl_mtr_reg, &mtr);
}
edac_dbg(4, "Channel #%d MTR%d = %x\n", i, j, mtr);
if (IS_DIMM_PRESENT(mtr)) {
+ if (!IS_ECC_ENABLED(pvt->info.mcmtr)) {
+ sbridge_printk(KERN_ERR, "CPU SrcID #%d, Ha #%d, Channel #%d has DIMMs, but ECC is disabled\n",
+ pvt->sbridge_dev->source_id,
+ pvt->sbridge_dev->dom, i);
+ return -ENODEV;
+ }
pvt->channel[i].dimms++;
ranks = numrank(pvt->info.type, mtr);
npages = MiB_TO_PAGES(size);
edac_dbg(0, "mc#%d: ha %d channel %d, dimm %d, %lld Mb (%d pages) bank: %d, rank: %d, row: %#x, col: %#x\n",
- pvt->sbridge_dev->mc, i/4, i%4, j,
+ pvt->sbridge_dev->mc, pvt->sbridge_dev->dom, i, j,
size, npages,
banks, ranks, rows, cols);
dimm->mtype = mtype;
dimm->edac_mode = mode;
snprintf(dimm->label, sizeof(dimm->label),
- "CPU_SrcID#%u_Ha#%u_Chan#%u_DIMM#%u",
- pvt->sbridge_dev->source_id, i/4, i%4, j);
+ "CPU_SrcID#%u_Ha#%u_Chan#%u_DIMM#%u",
+ pvt->sbridge_dev->source_id, pvt->sbridge_dev->dom, i, j);
}
}
}
return 0;
}
+static int get_dimm_config(struct mem_ctl_info *mci)
+{
+ struct sbridge_pvt *pvt = mci->pvt_info;
+ u64 knl_mc_sizes[KNL_MAX_CHANNELS];
+ enum edac_type mode;
+ u32 reg;
+
+ if (pvt->info.type == HASWELL || pvt->info.type == BROADWELL) {
+ pci_read_config_dword(pvt->pci_ha, HASWELL_HASYSDEFEATURE2, ®);
+ pvt->is_chan_hash = GET_BITFIELD(reg, 21, 21);
+ }
+ pvt->sbridge_dev->node_id = pvt->info.get_node_id(pvt);
+ edac_dbg(0, "mc#%d: Node ID: %d, source ID: %d\n",
+ pvt->sbridge_dev->mc,
+ pvt->sbridge_dev->node_id,
+ pvt->sbridge_dev->source_id);
+
+ /* KNL doesn't support mirroring or lockstep,
+ * and is always closed page
+ */
+ if (pvt->info.type == KNIGHTS_LANDING) {
+ mode = EDAC_S4ECD4ED;
+ pvt->is_mirrored = false;
+
+ if (knl_get_dimm_capacity(pvt, knl_mc_sizes) != 0)
+ return -1;
+ pci_read_config_dword(pvt->pci_ta, KNL_MCMTR, &pvt->info.mcmtr);
+ } else {
+ pci_read_config_dword(pvt->pci_ras, RASENABLES, ®);
+ if (IS_MIRROR_ENABLED(reg)) {
+ edac_dbg(0, "Memory mirror is enabled\n");
+ pvt->is_mirrored = true;
+ } else {
+ edac_dbg(0, "Memory mirror is disabled\n");
+ pvt->is_mirrored = false;
+ }
+
+ pci_read_config_dword(pvt->pci_ta, MCMTR, &pvt->info.mcmtr);
+ if (IS_LOCKSTEP_ENABLED(pvt->info.mcmtr)) {
+ edac_dbg(0, "Lockstep is enabled\n");
+ mode = EDAC_S8ECD8ED;
+ pvt->is_lockstep = true;
+ } else {
+ edac_dbg(0, "Lockstep is disabled\n");
+ mode = EDAC_S4ECD4ED;
+ pvt->is_lockstep = false;
+ }
+ if (IS_CLOSE_PG(pvt->info.mcmtr)) {
+ edac_dbg(0, "address map is on closed page mode\n");
+ pvt->is_close_pg = true;
+ } else {
+ edac_dbg(0, "address map is on open page mode\n");
+ pvt->is_close_pg = false;
+ }
+ }
+
+ return __populate_dimms(mci, knl_mc_sizes, mode);
+}
+
static void get_memory_layout(const struct mem_ctl_info *mci)
{
struct sbridge_pvt *pvt = mci->pvt_info;
*/
prv = 0;
for (n_tads = 0; n_tads < MAX_TAD; n_tads++) {
- pci_read_config_dword(pvt->pci_ha0, tad_dram_rule[n_tads],
- ®);
+ pci_read_config_dword(pvt->pci_ha, tad_dram_rule[n_tads], ®);
limit = TAD_LIMIT(reg);
if (limit <= prv)
break;
}
}
-static struct mem_ctl_info *get_mci_for_node_id(u8 node_id)
+static struct mem_ctl_info *get_mci_for_node_id(u8 node_id, u8 ha)
{
struct sbridge_dev *sbridge_dev;
list_for_each_entry(sbridge_dev, &sbridge_edac_list, list) {
- if (sbridge_dev->node_id == node_id)
+ if (sbridge_dev->node_id == node_id && sbridge_dev->dom == ha)
return sbridge_dev->mci;
}
return NULL;
int interleave_mode, shiftup = 0;
unsigned sad_interleave[pvt->info.max_interleave];
u32 reg, dram_rule;
- u8 ch_way, sck_way, pkg, sad_ha = 0, ch_add = 0;
+ u8 ch_way, sck_way, pkg, sad_ha = 0;
u32 tad_offset;
u32 rir_way;
u32 mb, gb;
pkg = sad_pkg(pvt->info.interleave_pkg, reg, idx);
*socket = sad_pkg_socket(pkg);
sad_ha = sad_pkg_ha(pkg);
- if (sad_ha)
- ch_add = 4;
if (a7mode) {
/* MCChanShiftUpEnable */
- pci_read_config_dword(pvt->pci_ha0,
- HASWELL_HASYSDEFEATURE2, ®);
+ pci_read_config_dword(pvt->pci_ha, HASWELL_HASYSDEFEATURE2, ®);
shiftup = GET_BITFIELD(reg, 22, 22);
}
pkg = sad_pkg(pvt->info.interleave_pkg, reg, idx);
*socket = sad_pkg_socket(pkg);
sad_ha = sad_pkg_ha(pkg);
- if (sad_ha)
- ch_add = 4;
edac_dbg(0, "SAD interleave package: %d = CPU socket %d, HA %d\n",
idx, *socket, sad_ha);
}
* Move to the proper node structure, in order to access the
* right PCI registers
*/
- new_mci = get_mci_for_node_id(*socket);
+ new_mci = get_mci_for_node_id(*socket, sad_ha);
if (!new_mci) {
sprintf(msg, "Struct for socket #%u wasn't initialized",
*socket);
* Step 2) Get memory channel
*/
prv = 0;
- if (pvt->info.type == SANDY_BRIDGE)
- pci_ha = pvt->pci_ha0;
- else {
- if (sad_ha)
- pci_ha = pvt->pci_ha1;
- else
- pci_ha = pvt->pci_ha0;
- }
+ pci_ha = pvt->pci_ha;
for (n_tads = 0; n_tads < MAX_TAD; n_tads++) {
pci_read_config_dword(pci_ha, tad_dram_rule[n_tads], ®);
limit = TAD_LIMIT(reg);
}
*channel_mask = 1 << base_ch;
- pci_read_config_dword(pvt->pci_tad[ch_add + base_ch],
- tad_ch_nilv_offset[n_tads],
- &tad_offset);
+ pci_read_config_dword(pvt->pci_tad[base_ch], tad_ch_nilv_offset[n_tads], &tad_offset);
if (pvt->is_mirrored) {
*channel_mask |= 1 << ((base_ch + 2) % 4);
* Step 3) Decode rank
*/
for (n_rir = 0; n_rir < MAX_RIR_RANGES; n_rir++) {
- pci_read_config_dword(pvt->pci_tad[ch_add + base_ch],
- rir_way_limit[n_rir],
- ®);
+ pci_read_config_dword(pvt->pci_tad[base_ch], rir_way_limit[n_rir], ®);
if (!IS_RIR_VALID(reg))
continue;
idx = (ch_addr >> 13); /* FIXME: Datasheet says to shift by 15 */
idx %= 1 << rir_way;
- pci_read_config_dword(pvt->pci_tad[ch_add + base_ch],
- rir_offset[n_rir][idx],
- ®);
+ pci_read_config_dword(pvt->pci_tad[base_ch], rir_offset[n_rir][idx], ®);
*rank = RIR_RNK_TGT(pvt->info.type, reg);
edac_dbg(0, "RIR#%d: channel address 0x%08Lx < 0x%08Lx, RIR interleave %d, index %d\n",
const unsigned devno,
const int multi_bus)
{
- struct sbridge_dev *sbridge_dev;
+ struct sbridge_dev *sbridge_dev = NULL;
const struct pci_id_descr *dev_descr = &table->descr[devno];
struct pci_dev *pdev = NULL;
u8 bus = 0;
+ int i = 0;
sbridge_printk(KERN_DEBUG,
"Seeking for: PCI ID %04x:%04x\n",
}
bus = pdev->bus->number;
- sbridge_dev = get_sbridge_dev(bus, multi_bus);
+next_imc:
+ sbridge_dev = get_sbridge_dev(bus, dev_descr->dom, multi_bus, sbridge_dev);
if (!sbridge_dev) {
- sbridge_dev = alloc_sbridge_dev(bus, table);
+
+ if (dev_descr->dom == SOCK)
+ goto out_imc;
+
+ sbridge_dev = alloc_sbridge_dev(bus, dev_descr->dom, table);
if (!sbridge_dev) {
pci_dev_put(pdev);
return -ENOMEM;
(*num_mc)++;
}
- if (sbridge_dev->pdev[devno]) {
+ if (sbridge_dev->pdev[sbridge_dev->i_devs]) {
sbridge_printk(KERN_ERR,
"Duplicated device for %04x:%04x\n",
PCI_VENDOR_ID_INTEL, dev_descr->dev_id);
return -ENODEV;
}
- sbridge_dev->pdev[devno] = pdev;
+ sbridge_dev->pdev[sbridge_dev->i_devs++] = pdev;
+
+ /* pdev belongs to more than one IMC, do extra gets */
+ if (++i > 1)
+ pci_dev_get(pdev);
+ if (dev_descr->dom == SOCK && i < table->n_imcs_per_sock)
+ goto next_imc;
+
+out_imc:
/* Be sure that the device is enabled */
if (unlikely(pci_enable_device(pdev) < 0)) {
sbridge_printk(KERN_ERR,
if (table->type == KNIGHTS_LANDING)
allow_dups = multi_bus = 1;
while (table && table->descr) {
- for (i = 0; i < table->n_devs; i++) {
+ for (i = 0; i < table->n_devs_per_sock; i++) {
if (!allow_dups || i == 0 ||
table->descr[i].dev_id !=
table->descr[i-1].dev_id) {
table, i, multi_bus);
if (rc < 0) {
if (i == 0) {
- i = table->n_devs;
+ i = table->n_devs_per_sock;
break;
}
sbridge_put_all_devices();
return 0;
}
+/*
+ * Device IDs for {SBRIDGE,IBRIDGE,HASWELL,BROADWELL}_IMC_HA0_TAD0 are in
+ * the format: XXXa. So we can convert from a device to the corresponding
+ * channel like this
+ */
+#define TAD_DEV_TO_CHAN(dev) (((dev) & 0xf) - 0xa)
+
static int sbridge_mci_bind_devs(struct mem_ctl_info *mci,
struct sbridge_dev *sbridge_dev)
{
pvt->pci_br0 = pdev;
break;
case PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_HA0:
- pvt->pci_ha0 = pdev;
+ pvt->pci_ha = pdev;
break;
case PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TA:
pvt->pci_ta = pdev;
case PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD2:
case PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD3:
{
- int id = pdev->device - PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD0;
+ int id = TAD_DEV_TO_CHAN(pdev->device);
pvt->pci_tad[id] = pdev;
saw_chan_mask |= 1 << id;
}
}
/* Check if everything were registered */
- if (!pvt->pci_sad0 || !pvt->pci_sad1 || !pvt->pci_ha0 ||
+ if (!pvt->pci_sad0 || !pvt->pci_sad1 || !pvt->pci_ha ||
!pvt->pci_ras || !pvt->pci_ta)
goto enodev;
switch (pdev->device) {
case PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0:
- pvt->pci_ha0 = pdev;
+ case PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1:
+ pvt->pci_ha = pdev;
break;
case PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_TA:
+ case PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TA:
pvt->pci_ta = pdev;
case PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_RAS:
+ case PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_RAS:
pvt->pci_ras = pdev;
break;
case PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_TAD0:
case PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_TAD1:
case PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_TAD2:
case PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_TAD3:
+ case PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TAD0:
+ case PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TAD1:
+ case PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TAD2:
+ case PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TAD3:
{
- int id = pdev->device - PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA0_TAD0;
+ int id = TAD_DEV_TO_CHAN(pdev->device);
pvt->pci_tad[id] = pdev;
saw_chan_mask |= 1 << id;
}
case PCI_DEVICE_ID_INTEL_IBRIDGE_BR1:
pvt->pci_br1 = pdev;
break;
- case PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1:
- pvt->pci_ha1 = pdev;
- break;
- case PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TAD0:
- case PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TAD1:
- case PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TAD2:
- case PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TAD3:
- {
- int id = pdev->device - PCI_DEVICE_ID_INTEL_IBRIDGE_IMC_HA1_TAD0 + 4;
- pvt->pci_tad[id] = pdev;
- saw_chan_mask |= 1 << id;
- }
- break;
default:
goto error;
}
}
/* Check if everything were registered */
- if (!pvt->pci_sad0 || !pvt->pci_ha0 || !pvt->pci_br0 ||
+ if (!pvt->pci_sad0 || !pvt->pci_ha || !pvt->pci_br0 ||
!pvt->pci_br1 || !pvt->pci_ras || !pvt->pci_ta)
goto enodev;
- if (saw_chan_mask != 0x0f && /* -EN */
- saw_chan_mask != 0x33 && /* -EP */
- saw_chan_mask != 0xff) /* -EX */
+ if (saw_chan_mask != 0x0f && /* -EN/-EX */
+ saw_chan_mask != 0x03) /* -EP */
goto enodev;
return 0;
pvt->pci_sad1 = pdev;
break;
case PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0:
- pvt->pci_ha0 = pdev;
+ case PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1:
+ pvt->pci_ha = pdev;
break;
case PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TA:
+ case PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TA:
pvt->pci_ta = pdev;
break;
- case PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_THERMAL:
+ case PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TM:
+ case PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TM:
pvt->pci_ras = pdev;
break;
case PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TAD0:
case PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TAD1:
case PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TAD2:
case PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TAD3:
- {
- int id = pdev->device - PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA0_TAD0;
-
- pvt->pci_tad[id] = pdev;
- saw_chan_mask |= 1 << id;
- }
- break;
case PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TAD0:
case PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TAD1:
case PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TAD2:
case PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TAD3:
{
- int id = pdev->device - PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TAD0 + 4;
-
+ int id = TAD_DEV_TO_CHAN(pdev->device);
pvt->pci_tad[id] = pdev;
saw_chan_mask |= 1 << id;
}
if (!pvt->pci_ddrio)
pvt->pci_ddrio = pdev;
break;
- case PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1:
- pvt->pci_ha1 = pdev;
- break;
- case PCI_DEVICE_ID_INTEL_HASWELL_IMC_HA1_TA:
- pvt->pci_ha1_ta = pdev;
- break;
default:
break;
}
}
/* Check if everything were registered */
- if (!pvt->pci_sad0 || !pvt->pci_ha0 || !pvt->pci_sad1 ||
+ if (!pvt->pci_sad0 || !pvt->pci_ha || !pvt->pci_sad1 ||
!pvt->pci_ras || !pvt->pci_ta || !pvt->info.pci_vtd)
goto enodev;
- if (saw_chan_mask != 0x0f && /* -EN */
- saw_chan_mask != 0x33 && /* -EP */
- saw_chan_mask != 0xff) /* -EX */
+ if (saw_chan_mask != 0x0f && /* -EN/-EX */
+ saw_chan_mask != 0x03) /* -EP */
goto enodev;
return 0;
pvt->pci_sad1 = pdev;
break;
case PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0:
- pvt->pci_ha0 = pdev;
+ case PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1:
+ pvt->pci_ha = pdev;
break;
case PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TA:
+ case PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TA:
pvt->pci_ta = pdev;
break;
- case PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_THERMAL:
+ case PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TM:
+ case PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TM:
pvt->pci_ras = pdev;
break;
case PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TAD0:
case PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TAD1:
case PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TAD2:
case PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TAD3:
- {
- int id = pdev->device - PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA0_TAD0;
- pvt->pci_tad[id] = pdev;
- saw_chan_mask |= 1 << id;
- }
- break;
case PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TAD0:
case PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TAD1:
case PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TAD2:
case PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TAD3:
{
- int id = pdev->device - PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TAD0 + 4;
+ int id = TAD_DEV_TO_CHAN(pdev->device);
pvt->pci_tad[id] = pdev;
saw_chan_mask |= 1 << id;
}
case PCI_DEVICE_ID_INTEL_BROADWELL_IMC_DDRIO0:
pvt->pci_ddrio = pdev;
break;
- case PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1:
- pvt->pci_ha1 = pdev;
- break;
- case PCI_DEVICE_ID_INTEL_BROADWELL_IMC_HA1_TA:
- pvt->pci_ha1_ta = pdev;
- break;
default:
break;
}
}
/* Check if everything were registered */
- if (!pvt->pci_sad0 || !pvt->pci_ha0 || !pvt->pci_sad1 ||
+ if (!pvt->pci_sad0 || !pvt->pci_ha || !pvt->pci_sad1 ||
!pvt->pci_ras || !pvt->pci_ta || !pvt->info.pci_vtd)
goto enodev;
- if (saw_chan_mask != 0x0f && /* -EN */
- saw_chan_mask != 0x33 && /* -EP */
- saw_chan_mask != 0xff) /* -EX */
+ if (saw_chan_mask != 0x0f && /* -EN/-EX */
+ saw_chan_mask != 0x03) /* -EP */
goto enodev;
return 0;
pvt->knl.pci_cha[devidx] = pdev;
break;
- case PCI_DEVICE_ID_INTEL_KNL_IMC_CHANNEL:
+ case PCI_DEVICE_ID_INTEL_KNL_IMC_CHAN:
devidx = -1;
/*
if (rc < 0)
goto err_parsing;
- new_mci = get_mci_for_node_id(socket);
+ new_mci = get_mci_for_node_id(socket, ha);
if (!new_mci) {
strcpy(msg, "Error: socket got corrupted!");
goto err_parsing;
/* Call the helper to output message */
edac_mc_handle_error(tp_event, mci, core_err_cnt,
m->addr >> PAGE_SHIFT, m->addr & ~PAGE_MASK, 0,
- 4*ha+channel, dimm, -1,
+ channel, dimm, -1,
optype, msg);
return;
err_parsing:
if (edac_get_report_status() == EDAC_REPORTING_DISABLED)
return NOTIFY_DONE;
- mci = get_mci_for_node_id(mce->socketid);
+ mci = get_mci_for_node_id(mce->socketid, IMC0);
if (!mci)
return NOTIFY_DONE;
pvt = mci->pvt_info;
struct pci_dev *pdev = sbridge_dev->pdev[0];
int rc;
- /* Check the number of active and not disabled channels */
- rc = check_if_ecc_is_active(sbridge_dev->bus, type);
- if (unlikely(rc < 0))
- return rc;
-
/* allocate a new MC control structure */
layers[0].type = EDAC_MC_LAYER_CHANNEL;
layers[0].size = type == KNIGHTS_LANDING ?
MEM_FLAG_DDR4 : MEM_FLAG_DDR3;
mci->edac_ctl_cap = EDAC_FLAG_NONE;
mci->edac_cap = EDAC_FLAG_NONE;
- mci->mod_name = "sbridge_edac.c";
+ mci->mod_name = "sb_edac.c";
mci->mod_ver = SBRIDGE_REVISION;
mci->dev_name = pci_name(pdev);
mci->ctl_page_to_phys = NULL;
pvt->info.max_interleave = ARRAY_SIZE(ibridge_interleave_list);
pvt->info.interleave_pkg = ibridge_interleave_pkg;
pvt->info.get_width = ibridge_get_width;
- mci->ctl_name = kasprintf(GFP_KERNEL, "Ivy Bridge Socket#%d", mci->mc_idx);
/* Store pci devices at mci for faster access */
rc = ibridge_mci_bind_devs(mci, sbridge_dev);
if (unlikely(rc < 0))
goto fail0;
+ get_source_id(mci);
+ mci->ctl_name = kasprintf(GFP_KERNEL, "Ivy Bridge SrcID#%d_Ha#%d",
+ pvt->sbridge_dev->source_id, pvt->sbridge_dev->dom);
break;
case SANDY_BRIDGE:
pvt->info.rankcfgr = SB_RANK_CFG_A;
pvt->info.max_interleave = ARRAY_SIZE(sbridge_interleave_list);
pvt->info.interleave_pkg = sbridge_interleave_pkg;
pvt->info.get_width = sbridge_get_width;
- mci->ctl_name = kasprintf(GFP_KERNEL, "Sandy Bridge Socket#%d", mci->mc_idx);
/* Store pci devices at mci for faster access */
rc = sbridge_mci_bind_devs(mci, sbridge_dev);
if (unlikely(rc < 0))
goto fail0;
+ get_source_id(mci);
+ mci->ctl_name = kasprintf(GFP_KERNEL, "Sandy Bridge SrcID#%d_Ha#%d",
+ pvt->sbridge_dev->source_id, pvt->sbridge_dev->dom);
break;
case HASWELL:
/* rankcfgr isn't used */
pvt->info.max_interleave = ARRAY_SIZE(ibridge_interleave_list);
pvt->info.interleave_pkg = ibridge_interleave_pkg;
pvt->info.get_width = ibridge_get_width;
- mci->ctl_name = kasprintf(GFP_KERNEL, "Haswell Socket#%d", mci->mc_idx);
/* Store pci devices at mci for faster access */
rc = haswell_mci_bind_devs(mci, sbridge_dev);
if (unlikely(rc < 0))
goto fail0;
+ get_source_id(mci);
+ mci->ctl_name = kasprintf(GFP_KERNEL, "Haswell SrcID#%d_Ha#%d",
+ pvt->sbridge_dev->source_id, pvt->sbridge_dev->dom);
break;
case BROADWELL:
/* rankcfgr isn't used */
pvt->info.max_interleave = ARRAY_SIZE(ibridge_interleave_list);
pvt->info.interleave_pkg = ibridge_interleave_pkg;
pvt->info.get_width = broadwell_get_width;
- mci->ctl_name = kasprintf(GFP_KERNEL, "Broadwell Socket#%d", mci->mc_idx);
/* Store pci devices at mci for faster access */
rc = broadwell_mci_bind_devs(mci, sbridge_dev);
if (unlikely(rc < 0))
goto fail0;
+ get_source_id(mci);
+ mci->ctl_name = kasprintf(GFP_KERNEL, "Broadwell SrcID#%d_Ha#%d",
+ pvt->sbridge_dev->source_id, pvt->sbridge_dev->dom);
break;
case KNIGHTS_LANDING:
/* pvt->info.rankcfgr == ??? */
pvt->info.max_interleave = ARRAY_SIZE(knl_interleave_list);
pvt->info.interleave_pkg = ibridge_interleave_pkg;
pvt->info.get_width = knl_get_width;
- mci->ctl_name = kasprintf(GFP_KERNEL,
- "Knights Landing Socket#%d", mci->mc_idx);
rc = knl_mci_bind_devs(mci, sbridge_dev);
if (unlikely(rc < 0))
goto fail0;
+ get_source_id(mci);
+ mci->ctl_name = kasprintf(GFP_KERNEL, "Knights Landing SrcID#%d_Ha#%d",
+ pvt->sbridge_dev->source_id, pvt->sbridge_dev->dom);
break;
}
/* Get dimm basic config and the memory layout */
- get_dimm_config(mci);
+ rc = get_dimm_config(mci);
+ if (rc < 0) {
+ edac_dbg(0, "MC: failed to get_dimm_config()\n");
+ goto fail;
+ }
get_memory_layout(mci);
/* record ptr to the generic device */
if (unlikely(edac_mc_add_mc(mci))) {
edac_dbg(0, "MC: failed edac_mc_add_mc()\n");
rc = -EINVAL;
- goto fail0;
+ goto fail;
}
return 0;
-fail0:
+fail:
kfree(mci->ctl_name);
+fail0:
edac_mc_free(mci);
sbridge_dev->mci = NULL;
return rc;
if (IS_ENABLED(CONFIG_EDAC_DEBUG)) {
l2c->debugfs = edac_debugfs_create_dir(pdev->dev.kobj.name);
- thunderx_create_debugfs_nodes(l2c->debugfs, l2c_devattr,
+ ret = thunderx_create_debugfs_nodes(l2c->debugfs, l2c_devattr,
l2c, dfs_entries);
if (ret != dfs_entries) {
config EXTCON_QCOM_SPMI_MISC
tristate "Qualcomm USB extcon support"
+ depends on ARCH_QCOM || COMPILE_TEST
help
Say Y here to enable SPMI PMIC based USB cable detection
support on Qualcomm PMICs such as PM8941.
goto out;
nconfs /= entries_per_config;
-
- micd_configs = devm_kzalloc(dev,
- nconfs * sizeof(struct arizona_micd_range),
+ micd_configs = devm_kcalloc(dev, nconfs, sizeof(*micd_configs),
GFP_KERNEL);
if (!micd_configs) {
ret = -ENOMEM;
struct int3496_data *data;
int ret;
- ret = acpi_dev_add_driver_gpios(ACPI_COMPANION(dev),
- acpi_int3496_default_gpios);
+ ret = devm_acpi_dev_add_driver_gpios(dev, acpi_int3496_default_gpios);
if (ret) {
dev_err(dev, "can't add GPIO ACPI mapping\n");
return ret;
devm_free_irq(&pdev->dev, data->usb_id_irq, data);
cancel_delayed_work_sync(&data->work);
- acpi_dev_remove_driver_gpios(ACPI_COMPANION(&pdev->dev));
-
return 0;
}
/**
* extcon_register_notifier_all() - Register a notifier block for all connectors
- * @edev: the extcon device that has the external connecotr.
+ * @edev: the extcon device that has the external connector.
* @nb: a notifier block to be registered.
*
- * This fucntion registers a notifier block in order to receive the state
+ * This function registers a notifier block in order to receive the state
* change of all supported external connectors from extcon device.
- * And The second parameter given to the callback of nb (val) is
+ * And the second parameter given to the callback of nb (val) is
* the current state and third parameter is the edev pointer.
*
* Returns 0 if success or error number if fail
}
spin_lock_init(&edev->lock);
-
- edev->nh = devm_kzalloc(&edev->dev,
- sizeof(*edev->nh) * edev->max_supported, GFP_KERNEL);
+ edev->nh = devm_kcalloc(&edev->dev, edev->max_supported,
+ sizeof(*edev->nh), GFP_KERNEL);
if (!edev->nh) {
ret = -ENOMEM;
goto err_dev;
Most users should say N.
+config EFI_CAPSULE_QUIRK_QUARK_CSH
+ boolean "Add support for Quark capsules with non-standard headers"
+ depends on X86 && !64BIT
+ select EFI_CAPSULE_LOADER
+ default y
+ help
+ Add support for processing Quark X1000 EFI capsules, whose header
+ layout deviates from the layout mandated by the UEFI specification.
+
config EFI_TEST
tristate "EFI Runtime Service Tests Support"
depends on EFI
*
*/
+#include <linux/dmi.h>
#include <linux/efi.h>
#include <linux/io.h>
#include <linux/memblock.h>
efi_set_pgd(current->active_mm);
preempt_enable();
}
+
+
+static int __init arm_dmi_init(void)
+{
+ /*
+ * On arm64/ARM, DMI depends on UEFI, and dmi_scan_machine() needs to
+ * be called early because dmi_id_init(), which is an arch_initcall
+ * itself, depends on dmi_scan_machine() having been called already.
+ */
+ dmi_scan_machine();
+ if (dmi_available)
+ dmi_set_dump_stack_arch_desc();
+ return 0;
+}
+core_initcall(arm_dmi_init);
#define NO_FURTHER_WRITE_ACTION -1
-struct capsule_info {
- bool header_obtained;
- int reset_type;
- long index;
- size_t count;
- size_t total_size;
- struct page **pages;
- size_t page_bytes_remain;
-};
+#ifndef phys_to_page
+#define phys_to_page(x) pfn_to_page((x) >> PAGE_SHIFT)
+#endif
/**
* efi_free_all_buff_pages - free all previous allocated buffer pages
static void efi_free_all_buff_pages(struct capsule_info *cap_info)
{
while (cap_info->index > 0)
- __free_page(cap_info->pages[--cap_info->index]);
+ __free_page(phys_to_page(cap_info->pages[--cap_info->index]));
cap_info->index = NO_FURTHER_WRITE_ACTION;
}
-/**
- * efi_capsule_setup_info - obtain the efi capsule header in the binary and
- * setup capsule_info structure
- * @cap_info: pointer to current instance of capsule_info structure
- * @kbuff: a mapped first page buffer pointer
- * @hdr_bytes: the total received number of bytes for efi header
- **/
-static ssize_t efi_capsule_setup_info(struct capsule_info *cap_info,
- void *kbuff, size_t hdr_bytes)
+int __efi_capsule_setup_info(struct capsule_info *cap_info)
{
- efi_capsule_header_t *cap_hdr;
size_t pages_needed;
int ret;
void *temp_page;
- /* Only process data block that is larger than efi header size */
- if (hdr_bytes < sizeof(efi_capsule_header_t))
- return 0;
-
- /* Reset back to the correct offset of header */
- cap_hdr = kbuff - cap_info->count;
- pages_needed = ALIGN(cap_hdr->imagesize, PAGE_SIZE) >> PAGE_SHIFT;
+ pages_needed = ALIGN(cap_info->total_size, PAGE_SIZE) / PAGE_SIZE;
if (pages_needed == 0) {
- pr_err("%s: pages count invalid\n", __func__);
+ pr_err("invalid capsule size");
return -EINVAL;
}
/* Check if the capsule binary supported */
- ret = efi_capsule_supported(cap_hdr->guid, cap_hdr->flags,
- cap_hdr->imagesize,
+ ret = efi_capsule_supported(cap_info->header.guid,
+ cap_info->header.flags,
+ cap_info->header.imagesize,
&cap_info->reset_type);
if (ret) {
- pr_err("%s: efi_capsule_supported() failed\n",
- __func__);
+ pr_err("capsule not supported\n");
return ret;
}
- cap_info->total_size = cap_hdr->imagesize;
temp_page = krealloc(cap_info->pages,
pages_needed * sizeof(void *),
GFP_KERNEL | __GFP_ZERO);
- if (!temp_page) {
- pr_debug("%s: krealloc() failed\n", __func__);
+ if (!temp_page)
return -ENOMEM;
- }
cap_info->pages = temp_page;
- cap_info->header_obtained = true;
return 0;
}
+/**
+ * efi_capsule_setup_info - obtain the efi capsule header in the binary and
+ * setup capsule_info structure
+ * @cap_info: pointer to current instance of capsule_info structure
+ * @kbuff: a mapped first page buffer pointer
+ * @hdr_bytes: the total received number of bytes for efi header
+ *
+ * Platforms with non-standard capsule update mechanisms can override
+ * this __weak function so they can perform any required capsule
+ * image munging. See quark_quirk_function() for an example.
+ **/
+int __weak efi_capsule_setup_info(struct capsule_info *cap_info, void *kbuff,
+ size_t hdr_bytes)
+{
+ /* Only process data block that is larger than efi header size */
+ if (hdr_bytes < sizeof(efi_capsule_header_t))
+ return 0;
+
+ memcpy(&cap_info->header, kbuff, sizeof(cap_info->header));
+ cap_info->total_size = cap_info->header.imagesize;
+
+ return __efi_capsule_setup_info(cap_info);
+}
+
/**
* efi_capsule_submit_update - invoke the efi_capsule_update API once binary
* upload done
static ssize_t efi_capsule_submit_update(struct capsule_info *cap_info)
{
int ret;
- void *cap_hdr_temp;
-
- cap_hdr_temp = vmap(cap_info->pages, cap_info->index,
- VM_MAP, PAGE_KERNEL);
- if (!cap_hdr_temp) {
- pr_debug("%s: vmap() failed\n", __func__);
- return -EFAULT;
- }
- ret = efi_capsule_update(cap_hdr_temp, cap_info->pages);
- vunmap(cap_hdr_temp);
+ ret = efi_capsule_update(&cap_info->header, cap_info->pages);
if (ret) {
- pr_err("%s: efi_capsule_update() failed\n", __func__);
+ pr_err("capsule update failed\n");
return ret;
}
/* Indicate capsule binary uploading is done */
cap_info->index = NO_FURTHER_WRITE_ACTION;
- pr_info("%s: Successfully upload capsule file with reboot type '%s'\n",
- __func__, !cap_info->reset_type ? "RESET_COLD" :
+ pr_info("Successfully upload capsule file with reboot type '%s'\n",
+ !cap_info->reset_type ? "RESET_COLD" :
cap_info->reset_type == 1 ? "RESET_WARM" :
"RESET_SHUTDOWN");
return 0;
if (!cap_info->page_bytes_remain) {
page = alloc_page(GFP_KERNEL);
if (!page) {
- pr_debug("%s: alloc_page() failed\n", __func__);
ret = -ENOMEM;
goto failed;
}
- cap_info->pages[cap_info->index++] = page;
+ cap_info->pages[cap_info->index++] = page_to_phys(page);
cap_info->page_bytes_remain = PAGE_SIZE;
+ } else {
+ page = phys_to_page(cap_info->pages[cap_info->index - 1]);
}
- page = cap_info->pages[cap_info->index - 1];
-
kbuff = kmap(page);
- if (!kbuff) {
- pr_debug("%s: kmap() failed\n", __func__);
- ret = -EFAULT;
- goto failed;
- }
kbuff += PAGE_SIZE - cap_info->page_bytes_remain;
/* Copy capsule binary data from user space to kernel space buffer */
write_byte = min_t(size_t, count, cap_info->page_bytes_remain);
if (copy_from_user(kbuff, buff, write_byte)) {
- pr_debug("%s: copy_from_user() failed\n", __func__);
ret = -EFAULT;
goto fail_unmap;
}
cap_info->page_bytes_remain -= write_byte;
/* Setup capsule binary info structure */
- if (!cap_info->header_obtained) {
- ret = efi_capsule_setup_info(cap_info, kbuff,
+ if (cap_info->header.headersize == 0) {
+ ret = efi_capsule_setup_info(cap_info, kbuff - cap_info->count,
cap_info->count + write_byte);
if (ret)
goto fail_unmap;
kunmap(page);
/* Submit the full binary to efi_capsule_update() API */
- if (cap_info->header_obtained &&
+ if (cap_info->header.headersize > 0 &&
cap_info->count >= cap_info->total_size) {
if (cap_info->count > cap_info->total_size) {
- pr_err("%s: upload size exceeded header defined size\n",
- __func__);
+ pr_err("capsule upload size exceeded header defined size\n");
ret = -EINVAL;
goto failed;
}
struct capsule_info *cap_info = file->private_data;
if (cap_info->index > 0) {
- pr_err("%s: capsule upload not complete\n", __func__);
+ pr_err("capsule upload not complete\n");
efi_free_all_buff_pages(cap_info);
ret = -ECANCELED;
}
ret = misc_register(&efi_capsule_misc);
if (ret)
- pr_err("%s: Failed to register misc char file note\n",
- __func__);
+ pr_err("Unable to register capsule loader device\n");
return ret;
}
*
* Return 0 on success, a converted EFI status code on failure.
*/
-int efi_capsule_update(efi_capsule_header_t *capsule, struct page **pages)
+int efi_capsule_update(efi_capsule_header_t *capsule, phys_addr_t *pages)
{
u32 imagesize = capsule->imagesize;
efi_guid_t guid = capsule->guid;
efi_capsule_block_desc_t *sglist;
sglist = kmap(sg_pages[i]);
- if (!sglist) {
- rv = -ENOMEM;
- goto out;
- }
for (j = 0; j < SGLIST_PER_PAGE && count > 0; j++) {
- u64 sz = min_t(u64, imagesize, PAGE_SIZE);
+ u64 sz = min_t(u64, imagesize,
+ PAGE_SIZE - (u64)*pages % PAGE_SIZE);
sglist[j].length = sz;
- sglist[j].data = page_to_phys(*pages++);
+ sglist[j].data = *pages++;
imagesize -= sz;
count--;
}
}
- efi_memattr_init();
+ if (efi_enabled(EFI_MEMMAP))
+ efi_memattr_init();
/* Parse the EFI Properties table if it exists */
if (efi.properties_table != EFI_INVALID_TABLE_ADDR) {
if (!access_ok(VERIFY_READ, src, 1))
return -EFAULT;
- buf = kmalloc(len, GFP_KERNEL);
- if (!buf) {
+ buf = memdup_user(src, len);
+ if (IS_ERR(buf)) {
*dst = NULL;
- return -ENOMEM;
+ return PTR_ERR(buf);
}
*dst = buf;
- if (copy_from_user(*dst, src, len)) {
- kfree(buf);
- return -EFAULT;
- }
-
return 0;
}
/* CBMEM firmware console log descriptor. */
struct cbmem_cons {
- u32 buffer_size;
- u32 buffer_cursor;
- u8 buffer_body[0];
+ u32 size_dont_access_after_boot;
+ u32 cursor;
+ u8 body[0];
} __packed;
+#define CURSOR_MASK ((1 << 28) - 1)
+#define OVERFLOW (1 << 31)
+
static struct cbmem_cons __iomem *cbmem_console;
+static u32 cbmem_console_size;
+
+/*
+ * The cbmem_console structure is read again on every access because it may
+ * change at any time if runtime firmware logs new messages. This may rarely
+ * lead to race conditions where the firmware overwrites the beginning of the
+ * ring buffer with more lines after we have already read |cursor|. It should be
+ * rare and harmless enough that we don't spend extra effort working around it.
+ */
+static ssize_t memconsole_coreboot_read(char *buf, loff_t pos, size_t count)
+{
+ u32 cursor = cbmem_console->cursor & CURSOR_MASK;
+ u32 flags = cbmem_console->cursor & ~CURSOR_MASK;
+ u32 size = cbmem_console_size;
+ struct seg { /* describes ring buffer segments in logical order */
+ u32 phys; /* physical offset from start of mem buffer */
+ u32 len; /* length of segment */
+ } seg[2] = { {0}, {0} };
+ size_t done = 0;
+ int i;
+
+ if (flags & OVERFLOW) {
+ if (cursor > size) /* Shouldn't really happen, but... */
+ cursor = 0;
+ seg[0] = (struct seg){.phys = cursor, .len = size - cursor};
+ seg[1] = (struct seg){.phys = 0, .len = cursor};
+ } else {
+ seg[0] = (struct seg){.phys = 0, .len = min(cursor, size)};
+ }
+
+ for (i = 0; i < ARRAY_SIZE(seg) && count > done; i++) {
+ done += memory_read_from_buffer(buf + done, count - done, &pos,
+ cbmem_console->body + seg[i].phys, seg[i].len);
+ pos -= seg[i].len;
+ }
+ return done;
+}
static int memconsole_coreboot_init(phys_addr_t physaddr)
{
if (!tmp_cbmc)
return -ENOMEM;
+ /* Read size only once to prevent overrun attack through /dev/mem. */
+ cbmem_console_size = tmp_cbmc->size_dont_access_after_boot;
cbmem_console = memremap(physaddr,
- tmp_cbmc->buffer_size + sizeof(*cbmem_console),
+ cbmem_console_size + sizeof(*cbmem_console),
MEMREMAP_WB);
memunmap(tmp_cbmc);
if (!cbmem_console)
return -ENOMEM;
- memconsole_setup(cbmem_console->buffer_body,
- min(cbmem_console->buffer_cursor, cbmem_console->buffer_size));
-
+ memconsole_setup(memconsole_coreboot_read);
return 0;
}
};
} __packed;
+static char *memconsole_baseaddr;
+static size_t memconsole_length;
+
+static ssize_t memconsole_read(char *buf, loff_t pos, size_t count)
+{
+ return memory_read_from_buffer(buf, count, &pos, memconsole_baseaddr,
+ memconsole_length);
+}
+
static void found_v1_header(struct biosmemcon_ebda *hdr)
{
pr_info("memconsole: BIOS console v1 EBDA structure found at %p\n",
hdr->v1.buffer_addr, hdr->v1.start,
hdr->v1.end, hdr->v1.num_chars);
- memconsole_setup(phys_to_virt(hdr->v1.buffer_addr), hdr->v1.num_chars);
+ memconsole_baseaddr = phys_to_virt(hdr->v1.buffer_addr);
+ memconsole_length = hdr->v1.num_chars;
+ memconsole_setup(memconsole_read);
}
static void found_v2_header(struct biosmemcon_ebda *hdr)
hdr->v2.buffer_addr, hdr->v2.start,
hdr->v2.end, hdr->v2.num_bytes);
- memconsole_setup(phys_to_virt(hdr->v2.buffer_addr + hdr->v2.start),
- hdr->v2.end - hdr->v2.start);
+ memconsole_baseaddr = phys_to_virt(hdr->v2.buffer_addr + hdr->v2.start);
+ memconsole_length = hdr->v2.end - hdr->v2.start;
+ memconsole_setup(memconsole_read);
}
/*
#include "memconsole.h"
-static char *memconsole_baseaddr;
-static size_t memconsole_length;
+static ssize_t (*memconsole_read_func)(char *, loff_t, size_t);
static ssize_t memconsole_read(struct file *filp, struct kobject *kobp,
struct bin_attribute *bin_attr, char *buf,
loff_t pos, size_t count)
{
- return memory_read_from_buffer(buf, count, &pos, memconsole_baseaddr,
- memconsole_length);
+ if (WARN_ON_ONCE(!memconsole_read_func))
+ return -EIO;
+ return memconsole_read_func(buf, pos, count);
}
static struct bin_attribute memconsole_bin_attr = {
.read = memconsole_read,
};
-void memconsole_setup(void *baseaddr, size_t length)
+void memconsole_setup(ssize_t (*read_func)(char *, loff_t, size_t))
{
- memconsole_baseaddr = baseaddr;
- memconsole_length = length;
+ memconsole_read_func = read_func;
}
EXPORT_SYMBOL(memconsole_setup);
int memconsole_sysfs_init(void)
{
- memconsole_bin_attr.size = memconsole_length;
return sysfs_create_bin_file(firmware_kobj, &memconsole_bin_attr);
}
EXPORT_SYMBOL(memconsole_sysfs_init);
#ifndef __FIRMWARE_GOOGLE_MEMCONSOLE_H
#define __FIRMWARE_GOOGLE_MEMCONSOLE_H
+#include <linux/types.h>
+
/*
* memconsole_setup
*
- * Initialize the memory console from raw (virtual) base
- * address and length.
+ * Initialize the memory console, passing the function to handle read accesses.
*/
-void memconsole_setup(void *baseaddr, size_t length);
+void memconsole_setup(ssize_t (*read_func)(char *, loff_t, size_t));
/*
* memconsole_sysfs_init
info = kzalloc(sizeof(*info), GFP_KERNEL);
if (!info)
return -ENOMEM;
- info->key = kzalloc(key_len + 1, GFP_KERNEL);
+
+ info->key = kstrndup(key, key_len, GFP_KERNEL);
if (!info->key) {
ret = -ENOMEM;
goto free_info;
}
- memcpy(info->key, key, key_len);
-
sysfs_bin_attr_init(&info->bin_attr);
info->bin_attr.attr.name = info->key;
info->bin_attr.attr.mode = 0444;
static int vpd_section_init(const char *name, struct vpd_section *sec,
phys_addr_t physaddr, size_t size)
{
- int ret;
- int raw_len;
+ int err;
sec->baseaddr = memremap(physaddr, size, MEMREMAP_WB);
if (!sec->baseaddr)
sec->name = name;
/* We want to export the raw partion with name ${name}_raw */
- raw_len = strlen(name) + 5;
- sec->raw_name = kzalloc(raw_len, GFP_KERNEL);
- strncpy(sec->raw_name, name, raw_len);
- strncat(sec->raw_name, "_raw", raw_len);
+ sec->raw_name = kasprintf(GFP_KERNEL, "%s_raw", name);
+ if (!sec->raw_name) {
+ err = -ENOMEM;
+ goto err_iounmap;
+ }
sysfs_bin_attr_init(&sec->bin_attr);
sec->bin_attr.attr.name = sec->raw_name;
sec->bin_attr.read = vpd_section_read;
sec->bin_attr.private = sec;
- ret = sysfs_create_bin_file(vpd_kobj, &sec->bin_attr);
- if (ret)
- goto free_sec;
+ err = sysfs_create_bin_file(vpd_kobj, &sec->bin_attr);
+ if (err)
+ goto err_free_raw_name;
sec->kobj = kobject_create_and_add(name, vpd_kobj);
if (!sec->kobj) {
- ret = -EINVAL;
- goto sysfs_remove;
+ err = -EINVAL;
+ goto err_sysfs_remove;
}
INIT_LIST_HEAD(&sec->attribs);
return 0;
-sysfs_remove:
+err_sysfs_remove:
sysfs_remove_bin_file(vpd_kobj, &sec->bin_attr);
-
-free_sec:
+err_free_raw_name:
kfree(sec->raw_name);
+err_iounmap:
iounmap(sec->baseaddr);
-
- return ret;
+ return err;
}
static int vpd_section_destroy(struct vpd_section *sec)
if (!vpd_kobj)
return -ENOMEM;
- memset(&ro_vpd, 0, sizeof(ro_vpd));
- memset(&rw_vpd, 0, sizeof(rw_vpd));
-
platform_driver_register(&vpd_driver);
return 0;
config FSI
tristate "FSI support"
+ select CRC4
---help---
FSI - the FRU Support Interface - is a simple bus for low-level
access to POWER-based hardware.
+
+if FSI
+
+config FSI_MASTER_GPIO
+ tristate "GPIO-based FSI master"
+ depends on GPIOLIB
+ select CRC4
+ ---help---
+ This option enables a FSI master driver using GPIO lines.
+
+config FSI_MASTER_HUB
+ tristate "FSI hub master"
+ ---help---
+ This option enables a FSI hub master driver. Hub is a type of FSI
+ master that is connected to the upstream master via a slave. Hubs
+ allow chaining of FSI links to an arbitrary depth. This allows for
+ a high target device fanout.
+
+config FSI_SCOM
+ tristate "SCOM FSI client device driver"
+ ---help---
+ This option enables an FSI based SCOM device driver.
+
+endif
+
endmenu
obj-$(CONFIG_FSI) += fsi-core.o
+obj-$(CONFIG_FSI_MASTER_HUB) += fsi-master-hub.o
+obj-$(CONFIG_FSI_MASTER_GPIO) += fsi-master-gpio.o
+obj-$(CONFIG_FSI_SCOM) += fsi-scom.o
* GNU General Public License for more details.
*/
+#include <linux/crc4.h>
#include <linux/device.h>
#include <linux/fsi.h>
+#include <linux/idr.h>
#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/bitops.h>
+
+#include "fsi-master.h"
+
+#define CREATE_TRACE_POINTS
+#include <trace/events/fsi.h>
+
+#define FSI_SLAVE_CONF_NEXT_MASK GENMASK(31, 31)
+#define FSI_SLAVE_CONF_SLOTS_MASK GENMASK(23, 16)
+#define FSI_SLAVE_CONF_SLOTS_SHIFT 16
+#define FSI_SLAVE_CONF_VERSION_MASK GENMASK(15, 12)
+#define FSI_SLAVE_CONF_VERSION_SHIFT 12
+#define FSI_SLAVE_CONF_TYPE_MASK GENMASK(11, 4)
+#define FSI_SLAVE_CONF_TYPE_SHIFT 4
+#define FSI_SLAVE_CONF_CRC_SHIFT 4
+#define FSI_SLAVE_CONF_CRC_MASK GENMASK(3, 0)
+#define FSI_SLAVE_CONF_DATA_BITS 28
+
+#define FSI_PEEK_BASE 0x410
+
+static const int engine_page_size = 0x400;
+
+#define FSI_SLAVE_BASE 0x800
+
+/*
+ * FSI slave engine control register offsets
+ */
+#define FSI_SMODE 0x0 /* R/W: Mode register */
+#define FSI_SISC 0x8 /* R/W: Interrupt condition */
+#define FSI_SSTAT 0x14 /* R : Slave status */
+#define FSI_LLMODE 0x100 /* R/W: Link layer mode register */
+
+/*
+ * SMODE fields
+ */
+#define FSI_SMODE_WSC 0x80000000 /* Warm start done */
+#define FSI_SMODE_ECRC 0x20000000 /* Hw CRC check */
+#define FSI_SMODE_SID_SHIFT 24 /* ID shift */
+#define FSI_SMODE_SID_MASK 3 /* ID Mask */
+#define FSI_SMODE_ED_SHIFT 20 /* Echo delay shift */
+#define FSI_SMODE_ED_MASK 0xf /* Echo delay mask */
+#define FSI_SMODE_SD_SHIFT 16 /* Send delay shift */
+#define FSI_SMODE_SD_MASK 0xf /* Send delay mask */
+#define FSI_SMODE_LBCRR_SHIFT 8 /* Clk ratio shift */
+#define FSI_SMODE_LBCRR_MASK 0xf /* Clk ratio mask */
+
+/*
+ * LLMODE fields
+ */
+#define FSI_LLMODE_ASYNC 0x1
+
+#define FSI_SLAVE_SIZE_23b 0x800000
+
+static DEFINE_IDA(master_ida);
+
+struct fsi_slave {
+ struct device dev;
+ struct fsi_master *master;
+ int id;
+ int link;
+ uint32_t size; /* size of slave address space */
+};
+
+#define to_fsi_master(d) container_of(d, struct fsi_master, dev)
+#define to_fsi_slave(d) container_of(d, struct fsi_slave, dev)
+
+static const int slave_retries = 2;
+static int discard_errors;
+
+static int fsi_master_read(struct fsi_master *master, int link,
+ uint8_t slave_id, uint32_t addr, void *val, size_t size);
+static int fsi_master_write(struct fsi_master *master, int link,
+ uint8_t slave_id, uint32_t addr, const void *val, size_t size);
+static int fsi_master_break(struct fsi_master *master, int link);
+
+/*
+ * fsi_device_read() / fsi_device_write() / fsi_device_peek()
+ *
+ * FSI endpoint-device support
+ *
+ * Read / write / peek accessors for a client
+ *
+ * Parameters:
+ * dev: Structure passed to FSI client device drivers on probe().
+ * addr: FSI address of given device. Client should pass in its base address
+ * plus desired offset to access its register space.
+ * val: For read/peek this is the value read at the specified address. For
+ * write this is value to write to the specified address.
+ * The data in val must be FSI bus endian (big endian).
+ * size: Size in bytes of the operation. Sizes supported are 1, 2 and 4 bytes.
+ * Addresses must be aligned on size boundaries or an error will result.
+ */
+int fsi_device_read(struct fsi_device *dev, uint32_t addr, void *val,
+ size_t size)
+{
+ if (addr > dev->size || size > dev->size || addr > dev->size - size)
+ return -EINVAL;
+
+ return fsi_slave_read(dev->slave, dev->addr + addr, val, size);
+}
+EXPORT_SYMBOL_GPL(fsi_device_read);
+
+int fsi_device_write(struct fsi_device *dev, uint32_t addr, const void *val,
+ size_t size)
+{
+ if (addr > dev->size || size > dev->size || addr > dev->size - size)
+ return -EINVAL;
+
+ return fsi_slave_write(dev->slave, dev->addr + addr, val, size);
+}
+EXPORT_SYMBOL_GPL(fsi_device_write);
+
+int fsi_device_peek(struct fsi_device *dev, void *val)
+{
+ uint32_t addr = FSI_PEEK_BASE + ((dev->unit - 2) * sizeof(uint32_t));
+
+ return fsi_slave_read(dev->slave, addr, val, sizeof(uint32_t));
+}
+
+static void fsi_device_release(struct device *_device)
+{
+ struct fsi_device *device = to_fsi_dev(_device);
+
+ kfree(device);
+}
+
+static struct fsi_device *fsi_create_device(struct fsi_slave *slave)
+{
+ struct fsi_device *dev;
+
+ dev = kzalloc(sizeof(*dev), GFP_KERNEL);
+ if (!dev)
+ return NULL;
+
+ dev->dev.parent = &slave->dev;
+ dev->dev.bus = &fsi_bus_type;
+ dev->dev.release = fsi_device_release;
+
+ return dev;
+}
+
+/* FSI slave support */
+static int fsi_slave_calc_addr(struct fsi_slave *slave, uint32_t *addrp,
+ uint8_t *idp)
+{
+ uint32_t addr = *addrp;
+ uint8_t id = *idp;
+
+ if (addr > slave->size)
+ return -EINVAL;
+
+ /* For 23 bit addressing, we encode the extra two bits in the slave
+ * id (and the slave's actual ID needs to be 0).
+ */
+ if (addr > 0x1fffff) {
+ if (slave->id != 0)
+ return -EINVAL;
+ id = (addr >> 21) & 0x3;
+ addr &= 0x1fffff;
+ }
+
+ *addrp = addr;
+ *idp = id;
+ return 0;
+}
+
+int fsi_slave_report_and_clear_errors(struct fsi_slave *slave)
+{
+ struct fsi_master *master = slave->master;
+ uint32_t irq, stat;
+ int rc, link;
+ uint8_t id;
+
+ link = slave->link;
+ id = slave->id;
+
+ rc = fsi_master_read(master, link, id, FSI_SLAVE_BASE + FSI_SISC,
+ &irq, sizeof(irq));
+ if (rc)
+ return rc;
+
+ rc = fsi_master_read(master, link, id, FSI_SLAVE_BASE + FSI_SSTAT,
+ &stat, sizeof(stat));
+ if (rc)
+ return rc;
+
+ dev_info(&slave->dev, "status: 0x%08x, sisc: 0x%08x\n",
+ be32_to_cpu(stat), be32_to_cpu(irq));
+
+ /* clear interrupts */
+ return fsi_master_write(master, link, id, FSI_SLAVE_BASE + FSI_SISC,
+ &irq, sizeof(irq));
+}
+
+static int fsi_slave_set_smode(struct fsi_master *master, int link, int id);
+
+int fsi_slave_handle_error(struct fsi_slave *slave, bool write, uint32_t addr,
+ size_t size)
+{
+ struct fsi_master *master = slave->master;
+ int rc, link;
+ uint32_t reg;
+ uint8_t id;
+
+ if (discard_errors)
+ return -1;
+
+ link = slave->link;
+ id = slave->id;
+
+ dev_dbg(&slave->dev, "handling error on %s to 0x%08x[%zd]",
+ write ? "write" : "read", addr, size);
+
+ /* try a simple clear of error conditions, which may fail if we've lost
+ * communication with the slave
+ */
+ rc = fsi_slave_report_and_clear_errors(slave);
+ if (!rc)
+ return 0;
+
+ /* send a TERM and retry */
+ if (master->term) {
+ rc = master->term(master, link, id);
+ if (!rc) {
+ rc = fsi_master_read(master, link, id, 0,
+ ®, sizeof(reg));
+ if (!rc)
+ rc = fsi_slave_report_and_clear_errors(slave);
+ if (!rc)
+ return 0;
+ }
+ }
+
+ /* getting serious, reset the slave via BREAK */
+ rc = fsi_master_break(master, link);
+ if (rc)
+ return rc;
+
+ rc = fsi_slave_set_smode(master, link, id);
+ if (rc)
+ return rc;
+
+ return fsi_slave_report_and_clear_errors(slave);
+}
+
+int fsi_slave_read(struct fsi_slave *slave, uint32_t addr,
+ void *val, size_t size)
+{
+ uint8_t id = slave->id;
+ int rc, err_rc, i;
+
+ rc = fsi_slave_calc_addr(slave, &addr, &id);
+ if (rc)
+ return rc;
+
+ for (i = 0; i < slave_retries; i++) {
+ rc = fsi_master_read(slave->master, slave->link,
+ id, addr, val, size);
+ if (!rc)
+ break;
+
+ err_rc = fsi_slave_handle_error(slave, false, addr, size);
+ if (err_rc)
+ break;
+ }
+
+ return rc;
+}
+EXPORT_SYMBOL_GPL(fsi_slave_read);
+
+int fsi_slave_write(struct fsi_slave *slave, uint32_t addr,
+ const void *val, size_t size)
+{
+ uint8_t id = slave->id;
+ int rc, err_rc, i;
+
+ rc = fsi_slave_calc_addr(slave, &addr, &id);
+ if (rc)
+ return rc;
+
+ for (i = 0; i < slave_retries; i++) {
+ rc = fsi_master_write(slave->master, slave->link,
+ id, addr, val, size);
+ if (!rc)
+ break;
+
+ err_rc = fsi_slave_handle_error(slave, true, addr, size);
+ if (err_rc)
+ break;
+ }
+
+ return rc;
+}
+EXPORT_SYMBOL_GPL(fsi_slave_write);
+
+extern int fsi_slave_claim_range(struct fsi_slave *slave,
+ uint32_t addr, uint32_t size)
+{
+ if (addr + size < addr)
+ return -EINVAL;
+
+ if (addr + size > slave->size)
+ return -EINVAL;
+
+ /* todo: check for overlapping claims */
+ return 0;
+}
+EXPORT_SYMBOL_GPL(fsi_slave_claim_range);
+
+extern void fsi_slave_release_range(struct fsi_slave *slave,
+ uint32_t addr, uint32_t size)
+{
+}
+EXPORT_SYMBOL_GPL(fsi_slave_release_range);
+
+static int fsi_slave_scan(struct fsi_slave *slave)
+{
+ uint32_t engine_addr;
+ uint32_t conf;
+ int rc, i;
+
+ /*
+ * scan engines
+ *
+ * We keep the peek mode and slave engines for the core; so start
+ * at the third slot in the configuration table. We also need to
+ * skip the chip ID entry at the start of the address space.
+ */
+ engine_addr = engine_page_size * 3;
+ for (i = 2; i < engine_page_size / sizeof(uint32_t); i++) {
+ uint8_t slots, version, type, crc;
+ struct fsi_device *dev;
+
+ rc = fsi_slave_read(slave, (i + 1) * sizeof(conf),
+ &conf, sizeof(conf));
+ if (rc) {
+ dev_warn(&slave->dev,
+ "error reading slave registers\n");
+ return -1;
+ }
+ conf = be32_to_cpu(conf);
+
+ crc = crc4(0, conf, 32);
+ if (crc) {
+ dev_warn(&slave->dev,
+ "crc error in slave register at 0x%04x\n",
+ i);
+ return -1;
+ }
+
+ slots = (conf & FSI_SLAVE_CONF_SLOTS_MASK)
+ >> FSI_SLAVE_CONF_SLOTS_SHIFT;
+ version = (conf & FSI_SLAVE_CONF_VERSION_MASK)
+ >> FSI_SLAVE_CONF_VERSION_SHIFT;
+ type = (conf & FSI_SLAVE_CONF_TYPE_MASK)
+ >> FSI_SLAVE_CONF_TYPE_SHIFT;
+
+ /*
+ * Unused address areas are marked by a zero type value; this
+ * skips the defined address areas
+ */
+ if (type != 0 && slots != 0) {
+
+ /* create device */
+ dev = fsi_create_device(slave);
+ if (!dev)
+ return -ENOMEM;
+
+ dev->slave = slave;
+ dev->engine_type = type;
+ dev->version = version;
+ dev->unit = i;
+ dev->addr = engine_addr;
+ dev->size = slots * engine_page_size;
+
+ dev_dbg(&slave->dev,
+ "engine[%i]: type %x, version %x, addr %x size %x\n",
+ dev->unit, dev->engine_type, version,
+ dev->addr, dev->size);
+
+ dev_set_name(&dev->dev, "%02x:%02x:%02x:%02x",
+ slave->master->idx, slave->link,
+ slave->id, i - 2);
+
+ rc = device_register(&dev->dev);
+ if (rc) {
+ dev_warn(&slave->dev, "add failed: %d\n", rc);
+ put_device(&dev->dev);
+ }
+ }
+
+ engine_addr += slots * engine_page_size;
+
+ if (!(conf & FSI_SLAVE_CONF_NEXT_MASK))
+ break;
+ }
+
+ return 0;
+}
+
+static ssize_t fsi_slave_sysfs_raw_read(struct file *file,
+ struct kobject *kobj, struct bin_attribute *attr, char *buf,
+ loff_t off, size_t count)
+{
+ struct fsi_slave *slave = to_fsi_slave(kobj_to_dev(kobj));
+ size_t total_len, read_len;
+ int rc;
+
+ if (off < 0)
+ return -EINVAL;
+
+ if (off > 0xffffffff || count > 0xffffffff || off + count > 0xffffffff)
+ return -EINVAL;
+
+ for (total_len = 0; total_len < count; total_len += read_len) {
+ read_len = min_t(size_t, count, 4);
+ read_len -= off & 0x3;
+
+ rc = fsi_slave_read(slave, off, buf + total_len, read_len);
+ if (rc)
+ return rc;
+
+ off += read_len;
+ }
+
+ return count;
+}
+
+static ssize_t fsi_slave_sysfs_raw_write(struct file *file,
+ struct kobject *kobj, struct bin_attribute *attr,
+ char *buf, loff_t off, size_t count)
+{
+ struct fsi_slave *slave = to_fsi_slave(kobj_to_dev(kobj));
+ size_t total_len, write_len;
+ int rc;
+
+ if (off < 0)
+ return -EINVAL;
+
+ if (off > 0xffffffff || count > 0xffffffff || off + count > 0xffffffff)
+ return -EINVAL;
+
+ for (total_len = 0; total_len < count; total_len += write_len) {
+ write_len = min_t(size_t, count, 4);
+ write_len -= off & 0x3;
+
+ rc = fsi_slave_write(slave, off, buf + total_len, write_len);
+ if (rc)
+ return rc;
+
+ off += write_len;
+ }
+
+ return count;
+}
+
+static struct bin_attribute fsi_slave_raw_attr = {
+ .attr = {
+ .name = "raw",
+ .mode = 0600,
+ },
+ .size = 0,
+ .read = fsi_slave_sysfs_raw_read,
+ .write = fsi_slave_sysfs_raw_write,
+};
+
+static ssize_t fsi_slave_sysfs_term_write(struct file *file,
+ struct kobject *kobj, struct bin_attribute *attr,
+ char *buf, loff_t off, size_t count)
+{
+ struct fsi_slave *slave = to_fsi_slave(kobj_to_dev(kobj));
+ struct fsi_master *master = slave->master;
+
+ if (!master->term)
+ return -ENODEV;
+
+ master->term(master, slave->link, slave->id);
+ return count;
+}
+
+static struct bin_attribute fsi_slave_term_attr = {
+ .attr = {
+ .name = "term",
+ .mode = 0200,
+ },
+ .size = 0,
+ .write = fsi_slave_sysfs_term_write,
+};
+
+/* Encode slave local bus echo delay */
+static inline uint32_t fsi_smode_echodly(int x)
+{
+ return (x & FSI_SMODE_ED_MASK) << FSI_SMODE_ED_SHIFT;
+}
+
+/* Encode slave local bus send delay */
+static inline uint32_t fsi_smode_senddly(int x)
+{
+ return (x & FSI_SMODE_SD_MASK) << FSI_SMODE_SD_SHIFT;
+}
+
+/* Encode slave local bus clock rate ratio */
+static inline uint32_t fsi_smode_lbcrr(int x)
+{
+ return (x & FSI_SMODE_LBCRR_MASK) << FSI_SMODE_LBCRR_SHIFT;
+}
+
+/* Encode slave ID */
+static inline uint32_t fsi_smode_sid(int x)
+{
+ return (x & FSI_SMODE_SID_MASK) << FSI_SMODE_SID_SHIFT;
+}
+
+static const uint32_t fsi_slave_smode(int id)
+{
+ return FSI_SMODE_WSC | FSI_SMODE_ECRC
+ | fsi_smode_sid(id)
+ | fsi_smode_echodly(0xf) | fsi_smode_senddly(0xf)
+ | fsi_smode_lbcrr(0x8);
+}
+
+static int fsi_slave_set_smode(struct fsi_master *master, int link, int id)
+{
+ uint32_t smode;
+
+ /* set our smode register with the slave ID field to 0; this enables
+ * extended slave addressing
+ */
+ smode = fsi_slave_smode(id);
+ smode = cpu_to_be32(smode);
+
+ return fsi_master_write(master, link, id, FSI_SLAVE_BASE + FSI_SMODE,
+ &smode, sizeof(smode));
+}
+
+static void fsi_slave_release(struct device *dev)
+{
+ struct fsi_slave *slave = to_fsi_slave(dev);
+
+ kfree(slave);
+}
+
+static int fsi_slave_init(struct fsi_master *master, int link, uint8_t id)
+{
+ uint32_t chip_id, llmode;
+ struct fsi_slave *slave;
+ uint8_t crc;
+ int rc;
+
+ /* Currently, we only support single slaves on a link, and use the
+ * full 23-bit address range
+ */
+ if (id != 0)
+ return -EINVAL;
+
+ rc = fsi_master_read(master, link, id, 0, &chip_id, sizeof(chip_id));
+ if (rc) {
+ dev_dbg(&master->dev, "can't read slave %02x:%02x %d\n",
+ link, id, rc);
+ return -ENODEV;
+ }
+ chip_id = be32_to_cpu(chip_id);
+
+ crc = crc4(0, chip_id, 32);
+ if (crc) {
+ dev_warn(&master->dev, "slave %02x:%02x invalid chip id CRC!\n",
+ link, id);
+ return -EIO;
+ }
+
+ dev_info(&master->dev, "fsi: found chip %08x at %02x:%02x:%02x\n",
+ chip_id, master->idx, link, id);
+
+ rc = fsi_slave_set_smode(master, link, id);
+ if (rc) {
+ dev_warn(&master->dev,
+ "can't set smode on slave:%02x:%02x %d\n",
+ link, id, rc);
+ return -ENODEV;
+ }
+
+ /* If we're behind a master that doesn't provide a self-running bus
+ * clock, put the slave into async mode
+ */
+ if (master->flags & FSI_MASTER_FLAG_SWCLOCK) {
+ llmode = cpu_to_be32(FSI_LLMODE_ASYNC);
+ rc = fsi_master_write(master, link, id,
+ FSI_SLAVE_BASE + FSI_LLMODE,
+ &llmode, sizeof(llmode));
+ if (rc)
+ dev_warn(&master->dev,
+ "can't set llmode on slave:%02x:%02x %d\n",
+ link, id, rc);
+ }
+
+ /* We can communicate with a slave; create the slave device and
+ * register.
+ */
+ slave = kzalloc(sizeof(*slave), GFP_KERNEL);
+ if (!slave)
+ return -ENOMEM;
+
+ slave->master = master;
+ slave->dev.parent = &master->dev;
+ slave->dev.release = fsi_slave_release;
+ slave->link = link;
+ slave->id = id;
+ slave->size = FSI_SLAVE_SIZE_23b;
+
+ dev_set_name(&slave->dev, "slave@%02x:%02x", link, id);
+ rc = device_register(&slave->dev);
+ if (rc < 0) {
+ dev_warn(&master->dev, "failed to create slave device: %d\n",
+ rc);
+ put_device(&slave->dev);
+ return rc;
+ }
+
+ rc = device_create_bin_file(&slave->dev, &fsi_slave_raw_attr);
+ if (rc)
+ dev_warn(&slave->dev, "failed to create raw attr: %d\n", rc);
+
+ rc = device_create_bin_file(&slave->dev, &fsi_slave_term_attr);
+ if (rc)
+ dev_warn(&slave->dev, "failed to create term attr: %d\n", rc);
+
+ rc = fsi_slave_scan(slave);
+ if (rc)
+ dev_dbg(&master->dev, "failed during slave scan with: %d\n",
+ rc);
+
+ return rc;
+}
+
+/* FSI master support */
+static int fsi_check_access(uint32_t addr, size_t size)
+{
+ if (size != 1 && size != 2 && size != 4)
+ return -EINVAL;
+
+ if ((addr & 0x3) != (size & 0x3))
+ return -EINVAL;
+
+ return 0;
+}
+
+static int fsi_master_read(struct fsi_master *master, int link,
+ uint8_t slave_id, uint32_t addr, void *val, size_t size)
+{
+ int rc;
+
+ trace_fsi_master_read(master, link, slave_id, addr, size);
+
+ rc = fsi_check_access(addr, size);
+ if (!rc)
+ rc = master->read(master, link, slave_id, addr, val, size);
+
+ trace_fsi_master_rw_result(master, link, slave_id, addr, size,
+ false, val, rc);
+
+ return rc;
+}
+
+static int fsi_master_write(struct fsi_master *master, int link,
+ uint8_t slave_id, uint32_t addr, const void *val, size_t size)
+{
+ int rc;
+
+ trace_fsi_master_write(master, link, slave_id, addr, size, val);
+
+ rc = fsi_check_access(addr, size);
+ if (!rc)
+ rc = master->write(master, link, slave_id, addr, val, size);
+
+ trace_fsi_master_rw_result(master, link, slave_id, addr, size,
+ true, val, rc);
+
+ return rc;
+}
+
+static int fsi_master_link_enable(struct fsi_master *master, int link)
+{
+ if (master->link_enable)
+ return master->link_enable(master, link);
+
+ return 0;
+}
+
+/*
+ * Issue a break command on this link
+ */
+static int fsi_master_break(struct fsi_master *master, int link)
+{
+ trace_fsi_master_break(master, link);
+
+ if (master->send_break)
+ return master->send_break(master, link);
+
+ return 0;
+}
+
+static int fsi_master_scan(struct fsi_master *master)
+{
+ int link, rc;
+
+ for (link = 0; link < master->n_links; link++) {
+ rc = fsi_master_link_enable(master, link);
+ if (rc) {
+ dev_dbg(&master->dev,
+ "enable link %d failed: %d\n", link, rc);
+ continue;
+ }
+ rc = fsi_master_break(master, link);
+ if (rc) {
+ dev_dbg(&master->dev,
+ "break to link %d failed: %d\n", link, rc);
+ continue;
+ }
+
+ fsi_slave_init(master, link, 0);
+ }
+
+ return 0;
+}
+
+static int fsi_slave_remove_device(struct device *dev, void *arg)
+{
+ device_unregister(dev);
+ return 0;
+}
+
+static int fsi_master_remove_slave(struct device *dev, void *arg)
+{
+ device_for_each_child(dev, NULL, fsi_slave_remove_device);
+ device_unregister(dev);
+ return 0;
+}
+
+static void fsi_master_unscan(struct fsi_master *master)
+{
+ device_for_each_child(&master->dev, NULL, fsi_master_remove_slave);
+}
+
+static ssize_t master_rescan_store(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t count)
+{
+ struct fsi_master *master = to_fsi_master(dev);
+ int rc;
+
+ fsi_master_unscan(master);
+ rc = fsi_master_scan(master);
+ if (rc < 0)
+ return rc;
+
+ return count;
+}
+
+static DEVICE_ATTR(rescan, 0200, NULL, master_rescan_store);
+
+static ssize_t master_break_store(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t count)
+{
+ struct fsi_master *master = to_fsi_master(dev);
+
+ fsi_master_break(master, 0);
+
+ return count;
+}
+
+static DEVICE_ATTR(break, 0200, NULL, master_break_store);
+
+int fsi_master_register(struct fsi_master *master)
+{
+ int rc;
+
+ if (!master)
+ return -EINVAL;
+
+ master->idx = ida_simple_get(&master_ida, 0, INT_MAX, GFP_KERNEL);
+ dev_set_name(&master->dev, "fsi%d", master->idx);
+
+ rc = device_register(&master->dev);
+ if (rc) {
+ ida_simple_remove(&master_ida, master->idx);
+ return rc;
+ }
+
+ rc = device_create_file(&master->dev, &dev_attr_rescan);
+ if (rc) {
+ device_unregister(&master->dev);
+ ida_simple_remove(&master_ida, master->idx);
+ return rc;
+ }
+
+ rc = device_create_file(&master->dev, &dev_attr_break);
+ if (rc) {
+ device_unregister(&master->dev);
+ ida_simple_remove(&master_ida, master->idx);
+ return rc;
+ }
+
+ fsi_master_scan(master);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(fsi_master_register);
+
+void fsi_master_unregister(struct fsi_master *master)
+{
+ if (master->idx >= 0) {
+ ida_simple_remove(&master_ida, master->idx);
+ master->idx = -1;
+ }
+
+ fsi_master_unscan(master);
+ device_unregister(&master->dev);
+}
+EXPORT_SYMBOL_GPL(fsi_master_unregister);
/* FSI core & Linux bus type definitions */
return 0;
}
+int fsi_driver_register(struct fsi_driver *fsi_drv)
+{
+ if (!fsi_drv)
+ return -EINVAL;
+ if (!fsi_drv->id_table)
+ return -EINVAL;
+
+ return driver_register(&fsi_drv->drv);
+}
+EXPORT_SYMBOL_GPL(fsi_driver_register);
+
+void fsi_driver_unregister(struct fsi_driver *fsi_drv)
+{
+ driver_unregister(&fsi_drv->drv);
+}
+EXPORT_SYMBOL_GPL(fsi_driver_unregister);
+
struct bus_type fsi_bus_type = {
.name = "fsi",
.match = fsi_bus_match,
module_init(fsi_init);
module_exit(fsi_exit);
+module_param(discard_errors, int, 0664);
+MODULE_LICENSE("GPL");
+MODULE_PARM_DESC(discard_errors, "Don't invoke error handling on bus accesses");
--- /dev/null
+/*
+ * A FSI master controller, using a simple GPIO bit-banging interface
+ */
+
+#include <linux/crc4.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/fsi.h>
+#include <linux/gpio/consumer.h>
+#include <linux/io.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+
+#include "fsi-master.h"
+
+#define FSI_GPIO_STD_DLY 1 /* Standard pin delay in nS */
+#define FSI_ECHO_DELAY_CLOCKS 16 /* Number clocks for echo delay */
+#define FSI_PRE_BREAK_CLOCKS 50 /* Number clocks to prep for break */
+#define FSI_BREAK_CLOCKS 256 /* Number of clocks to issue break */
+#define FSI_POST_BREAK_CLOCKS 16000 /* Number clocks to set up cfam */
+#define FSI_INIT_CLOCKS 5000 /* Clock out any old data */
+#define FSI_GPIO_STD_DELAY 10 /* Standard GPIO delay in nS */
+ /* todo: adjust down as low as */
+ /* possible or eliminate */
+#define FSI_GPIO_CMD_DPOLL 0x2
+#define FSI_GPIO_CMD_TERM 0x3f
+#define FSI_GPIO_CMD_ABS_AR 0x4
+
+#define FSI_GPIO_DPOLL_CLOCKS 100 /* < 21 will cause slave to hang */
+
+/* Bus errors */
+#define FSI_GPIO_ERR_BUSY 1 /* Slave stuck in busy state */
+#define FSI_GPIO_RESP_ERRA 2 /* Any (misc) Error */
+#define FSI_GPIO_RESP_ERRC 3 /* Slave reports master CRC error */
+#define FSI_GPIO_MTOE 4 /* Master time out error */
+#define FSI_GPIO_CRC_INVAL 5 /* Master reports slave CRC error */
+
+/* Normal slave responses */
+#define FSI_GPIO_RESP_BUSY 1
+#define FSI_GPIO_RESP_ACK 0
+#define FSI_GPIO_RESP_ACKD 4
+
+#define FSI_GPIO_MAX_BUSY 100
+#define FSI_GPIO_MTOE_COUNT 1000
+#define FSI_GPIO_DRAIN_BITS 20
+#define FSI_GPIO_CRC_SIZE 4
+#define FSI_GPIO_MSG_ID_SIZE 2
+#define FSI_GPIO_MSG_RESPID_SIZE 2
+#define FSI_GPIO_PRIME_SLAVE_CLOCKS 100
+
+struct fsi_master_gpio {
+ struct fsi_master master;
+ struct device *dev;
+ spinlock_t cmd_lock; /* Lock for commands */
+ struct gpio_desc *gpio_clk;
+ struct gpio_desc *gpio_data;
+ struct gpio_desc *gpio_trans; /* Voltage translator */
+ struct gpio_desc *gpio_enable; /* FSI enable */
+ struct gpio_desc *gpio_mux; /* Mux control */
+};
+
+#define CREATE_TRACE_POINTS
+#include <trace/events/fsi_master_gpio.h>
+
+#define to_fsi_master_gpio(m) container_of(m, struct fsi_master_gpio, master)
+
+struct fsi_gpio_msg {
+ uint64_t msg;
+ uint8_t bits;
+};
+
+static void clock_toggle(struct fsi_master_gpio *master, int count)
+{
+ int i;
+
+ for (i = 0; i < count; i++) {
+ ndelay(FSI_GPIO_STD_DLY);
+ gpiod_set_value(master->gpio_clk, 0);
+ ndelay(FSI_GPIO_STD_DLY);
+ gpiod_set_value(master->gpio_clk, 1);
+ }
+}
+
+static int sda_in(struct fsi_master_gpio *master)
+{
+ int in;
+
+ ndelay(FSI_GPIO_STD_DLY);
+ in = gpiod_get_value(master->gpio_data);
+ return in ? 1 : 0;
+}
+
+static void sda_out(struct fsi_master_gpio *master, int value)
+{
+ gpiod_set_value(master->gpio_data, value);
+}
+
+static void set_sda_input(struct fsi_master_gpio *master)
+{
+ gpiod_direction_input(master->gpio_data);
+ gpiod_set_value(master->gpio_trans, 0);
+}
+
+static void set_sda_output(struct fsi_master_gpio *master, int value)
+{
+ gpiod_set_value(master->gpio_trans, 1);
+ gpiod_direction_output(master->gpio_data, value);
+}
+
+static void clock_zeros(struct fsi_master_gpio *master, int count)
+{
+ set_sda_output(master, 1);
+ clock_toggle(master, count);
+}
+
+static void serial_in(struct fsi_master_gpio *master, struct fsi_gpio_msg *msg,
+ uint8_t num_bits)
+{
+ uint8_t bit, in_bit;
+
+ set_sda_input(master);
+
+ for (bit = 0; bit < num_bits; bit++) {
+ clock_toggle(master, 1);
+ in_bit = sda_in(master);
+ msg->msg <<= 1;
+ msg->msg |= ~in_bit & 0x1; /* Data is active low */
+ }
+ msg->bits += num_bits;
+
+ trace_fsi_master_gpio_in(master, num_bits, msg->msg);
+}
+
+static void serial_out(struct fsi_master_gpio *master,
+ const struct fsi_gpio_msg *cmd)
+{
+ uint8_t bit;
+ uint64_t msg = ~cmd->msg; /* Data is active low */
+ uint64_t sda_mask = 0x1ULL << (cmd->bits - 1);
+ uint64_t last_bit = ~0;
+ int next_bit;
+
+ trace_fsi_master_gpio_out(master, cmd->bits, cmd->msg);
+
+ if (!cmd->bits) {
+ dev_warn(master->dev, "trying to output 0 bits\n");
+ return;
+ }
+ set_sda_output(master, 0);
+
+ /* Send the start bit */
+ sda_out(master, 0);
+ clock_toggle(master, 1);
+
+ /* Send the message */
+ for (bit = 0; bit < cmd->bits; bit++) {
+ next_bit = (msg & sda_mask) >> (cmd->bits - 1);
+ if (last_bit ^ next_bit) {
+ sda_out(master, next_bit);
+ last_bit = next_bit;
+ }
+ clock_toggle(master, 1);
+ msg <<= 1;
+ }
+}
+
+static void msg_push_bits(struct fsi_gpio_msg *msg, uint64_t data, int bits)
+{
+ msg->msg <<= bits;
+ msg->msg |= data & ((1ull << bits) - 1);
+ msg->bits += bits;
+}
+
+static void msg_push_crc(struct fsi_gpio_msg *msg)
+{
+ uint8_t crc;
+ int top;
+
+ top = msg->bits & 0x3;
+
+ /* start bit, and any non-aligned top bits */
+ crc = crc4(0, 1 << top | msg->msg >> (msg->bits - top), top + 1);
+
+ /* aligned bits */
+ crc = crc4(crc, msg->msg, msg->bits - top);
+
+ msg_push_bits(msg, crc, 4);
+}
+
+/*
+ * Encode an Absolute Address command
+ */
+static void build_abs_ar_command(struct fsi_gpio_msg *cmd,
+ uint8_t id, uint32_t addr, size_t size, const void *data)
+{
+ bool write = !!data;
+ uint8_t ds;
+ int i;
+
+ cmd->bits = 0;
+ cmd->msg = 0;
+
+ msg_push_bits(cmd, id, 2);
+ msg_push_bits(cmd, FSI_GPIO_CMD_ABS_AR, 3);
+ msg_push_bits(cmd, write ? 0 : 1, 1);
+
+ /*
+ * The read/write size is encoded in the lower bits of the address
+ * (as it must be naturally-aligned), and the following ds bit.
+ *
+ * size addr:1 addr:0 ds
+ * 1 x x 0
+ * 2 x 0 1
+ * 4 0 1 1
+ *
+ */
+ ds = size > 1 ? 1 : 0;
+ addr &= ~(size - 1);
+ if (size == 4)
+ addr |= 1;
+
+ msg_push_bits(cmd, addr & ((1 << 21) - 1), 21);
+ msg_push_bits(cmd, ds, 1);
+ for (i = 0; write && i < size; i++)
+ msg_push_bits(cmd, ((uint8_t *)data)[i], 8);
+
+ msg_push_crc(cmd);
+}
+
+static void build_dpoll_command(struct fsi_gpio_msg *cmd, uint8_t slave_id)
+{
+ cmd->bits = 0;
+ cmd->msg = 0;
+
+ msg_push_bits(cmd, slave_id, 2);
+ msg_push_bits(cmd, FSI_GPIO_CMD_DPOLL, 3);
+ msg_push_crc(cmd);
+}
+
+static void echo_delay(struct fsi_master_gpio *master)
+{
+ set_sda_output(master, 1);
+ clock_toggle(master, FSI_ECHO_DELAY_CLOCKS);
+}
+
+static void build_term_command(struct fsi_gpio_msg *cmd, uint8_t slave_id)
+{
+ cmd->bits = 0;
+ cmd->msg = 0;
+
+ msg_push_bits(cmd, slave_id, 2);
+ msg_push_bits(cmd, FSI_GPIO_CMD_TERM, 6);
+ msg_push_crc(cmd);
+}
+
+/*
+ * Store information on master errors so handler can detect and clean
+ * up the bus
+ */
+static void fsi_master_gpio_error(struct fsi_master_gpio *master, int error)
+{
+
+}
+
+static int read_one_response(struct fsi_master_gpio *master,
+ uint8_t data_size, struct fsi_gpio_msg *msgp, uint8_t *tagp)
+{
+ struct fsi_gpio_msg msg;
+ uint8_t id, tag;
+ uint32_t crc;
+ int i;
+
+ /* wait for the start bit */
+ for (i = 0; i < FSI_GPIO_MTOE_COUNT; i++) {
+ msg.bits = 0;
+ msg.msg = 0;
+ serial_in(master, &msg, 1);
+ if (msg.msg)
+ break;
+ }
+ if (i == FSI_GPIO_MTOE_COUNT) {
+ dev_dbg(master->dev,
+ "Master time out waiting for response\n");
+ fsi_master_gpio_error(master, FSI_GPIO_MTOE);
+ return -EIO;
+ }
+
+ msg.bits = 0;
+ msg.msg = 0;
+
+ /* Read slave ID & response tag */
+ serial_in(master, &msg, 4);
+
+ id = (msg.msg >> FSI_GPIO_MSG_RESPID_SIZE) & 0x3;
+ tag = msg.msg & 0x3;
+
+ /* If we have an ACK and we're expecting data, clock the data in too */
+ if (tag == FSI_GPIO_RESP_ACK && data_size)
+ serial_in(master, &msg, data_size * 8);
+
+ /* read CRC */
+ serial_in(master, &msg, FSI_GPIO_CRC_SIZE);
+
+ /* we have a whole message now; check CRC */
+ crc = crc4(0, 1, 1);
+ crc = crc4(crc, msg.msg, msg.bits);
+ if (crc) {
+ dev_dbg(master->dev, "ERR response CRC\n");
+ fsi_master_gpio_error(master, FSI_GPIO_CRC_INVAL);
+ return -EIO;
+ }
+
+ if (msgp)
+ *msgp = msg;
+ if (tagp)
+ *tagp = tag;
+
+ return 0;
+}
+
+static int issue_term(struct fsi_master_gpio *master, uint8_t slave)
+{
+ struct fsi_gpio_msg cmd;
+ uint8_t tag;
+ int rc;
+
+ build_term_command(&cmd, slave);
+ serial_out(master, &cmd);
+ echo_delay(master);
+
+ rc = read_one_response(master, 0, NULL, &tag);
+ if (rc < 0) {
+ dev_err(master->dev,
+ "TERM failed; lost communication with slave\n");
+ return -EIO;
+ } else if (tag != FSI_GPIO_RESP_ACK) {
+ dev_err(master->dev, "TERM failed; response %d\n", tag);
+ return -EIO;
+ }
+
+ return 0;
+}
+
+static int poll_for_response(struct fsi_master_gpio *master,
+ uint8_t slave, uint8_t size, void *data)
+{
+ struct fsi_gpio_msg response, cmd;
+ int busy_count = 0, rc, i;
+ uint8_t tag;
+ uint8_t *data_byte = data;
+
+retry:
+ rc = read_one_response(master, size, &response, &tag);
+ if (rc)
+ return rc;
+
+ switch (tag) {
+ case FSI_GPIO_RESP_ACK:
+ if (size && data) {
+ uint64_t val = response.msg;
+ /* clear crc & mask */
+ val >>= 4;
+ val &= (1ull << (size * 8)) - 1;
+
+ for (i = 0; i < size; i++) {
+ data_byte[size-i-1] = val;
+ val >>= 8;
+ }
+ }
+ break;
+ case FSI_GPIO_RESP_BUSY:
+ /*
+ * Its necessary to clock slave before issuing
+ * d-poll, not indicated in the hardware protocol
+ * spec. < 20 clocks causes slave to hang, 21 ok.
+ */
+ clock_zeros(master, FSI_GPIO_DPOLL_CLOCKS);
+ if (busy_count++ < FSI_GPIO_MAX_BUSY) {
+ build_dpoll_command(&cmd, slave);
+ serial_out(master, &cmd);
+ echo_delay(master);
+ goto retry;
+ }
+ dev_warn(master->dev,
+ "ERR slave is stuck in busy state, issuing TERM\n");
+ issue_term(master, slave);
+ rc = -EIO;
+ break;
+
+ case FSI_GPIO_RESP_ERRA:
+ case FSI_GPIO_RESP_ERRC:
+ dev_dbg(master->dev, "ERR%c received: 0x%x\n",
+ tag == FSI_GPIO_RESP_ERRA ? 'A' : 'C',
+ (int)response.msg);
+ fsi_master_gpio_error(master, response.msg);
+ rc = -EIO;
+ break;
+ }
+
+ /* Clock the slave enough to be ready for next operation */
+ clock_zeros(master, FSI_GPIO_PRIME_SLAVE_CLOCKS);
+ return rc;
+}
+
+static int fsi_master_gpio_xfer(struct fsi_master_gpio *master, uint8_t slave,
+ struct fsi_gpio_msg *cmd, size_t resp_len, void *resp)
+{
+ unsigned long flags;
+ int rc;
+
+ spin_lock_irqsave(&master->cmd_lock, flags);
+ serial_out(master, cmd);
+ echo_delay(master);
+ rc = poll_for_response(master, slave, resp_len, resp);
+ spin_unlock_irqrestore(&master->cmd_lock, flags);
+
+ return rc;
+}
+
+static int fsi_master_gpio_read(struct fsi_master *_master, int link,
+ uint8_t id, uint32_t addr, void *val, size_t size)
+{
+ struct fsi_master_gpio *master = to_fsi_master_gpio(_master);
+ struct fsi_gpio_msg cmd;
+
+ if (link != 0)
+ return -ENODEV;
+
+ build_abs_ar_command(&cmd, id, addr, size, NULL);
+ return fsi_master_gpio_xfer(master, id, &cmd, size, val);
+}
+
+static int fsi_master_gpio_write(struct fsi_master *_master, int link,
+ uint8_t id, uint32_t addr, const void *val, size_t size)
+{
+ struct fsi_master_gpio *master = to_fsi_master_gpio(_master);
+ struct fsi_gpio_msg cmd;
+
+ if (link != 0)
+ return -ENODEV;
+
+ build_abs_ar_command(&cmd, id, addr, size, val);
+ return fsi_master_gpio_xfer(master, id, &cmd, 0, NULL);
+}
+
+static int fsi_master_gpio_term(struct fsi_master *_master,
+ int link, uint8_t id)
+{
+ struct fsi_master_gpio *master = to_fsi_master_gpio(_master);
+ struct fsi_gpio_msg cmd;
+
+ if (link != 0)
+ return -ENODEV;
+
+ build_term_command(&cmd, id);
+ return fsi_master_gpio_xfer(master, id, &cmd, 0, NULL);
+}
+
+static int fsi_master_gpio_break(struct fsi_master *_master, int link)
+{
+ struct fsi_master_gpio *master = to_fsi_master_gpio(_master);
+
+ if (link != 0)
+ return -ENODEV;
+
+ trace_fsi_master_gpio_break(master);
+
+ set_sda_output(master, 1);
+ sda_out(master, 1);
+ clock_toggle(master, FSI_PRE_BREAK_CLOCKS);
+ sda_out(master, 0);
+ clock_toggle(master, FSI_BREAK_CLOCKS);
+ echo_delay(master);
+ sda_out(master, 1);
+ clock_toggle(master, FSI_POST_BREAK_CLOCKS);
+
+ /* Wait for logic reset to take effect */
+ udelay(200);
+
+ return 0;
+}
+
+static void fsi_master_gpio_init(struct fsi_master_gpio *master)
+{
+ gpiod_direction_output(master->gpio_mux, 1);
+ gpiod_direction_output(master->gpio_trans, 1);
+ gpiod_direction_output(master->gpio_enable, 1);
+ gpiod_direction_output(master->gpio_clk, 1);
+ gpiod_direction_output(master->gpio_data, 1);
+
+ /* todo: evaluate if clocks can be reduced */
+ clock_zeros(master, FSI_INIT_CLOCKS);
+}
+
+static int fsi_master_gpio_link_enable(struct fsi_master *_master, int link)
+{
+ struct fsi_master_gpio *master = to_fsi_master_gpio(_master);
+
+ if (link != 0)
+ return -ENODEV;
+ gpiod_set_value(master->gpio_enable, 1);
+
+ return 0;
+}
+
+static int fsi_master_gpio_probe(struct platform_device *pdev)
+{
+ struct fsi_master_gpio *master;
+ struct gpio_desc *gpio;
+
+ master = devm_kzalloc(&pdev->dev, sizeof(*master), GFP_KERNEL);
+ if (!master)
+ return -ENOMEM;
+
+ master->dev = &pdev->dev;
+ master->master.dev.parent = master->dev;
+
+ gpio = devm_gpiod_get(&pdev->dev, "clock", 0);
+ if (IS_ERR(gpio)) {
+ dev_err(&pdev->dev, "failed to get clock gpio\n");
+ return PTR_ERR(gpio);
+ }
+ master->gpio_clk = gpio;
+
+ gpio = devm_gpiod_get(&pdev->dev, "data", 0);
+ if (IS_ERR(gpio)) {
+ dev_err(&pdev->dev, "failed to get data gpio\n");
+ return PTR_ERR(gpio);
+ }
+ master->gpio_data = gpio;
+
+ /* Optional GPIOs */
+ gpio = devm_gpiod_get_optional(&pdev->dev, "trans", 0);
+ if (IS_ERR(gpio)) {
+ dev_err(&pdev->dev, "failed to get trans gpio\n");
+ return PTR_ERR(gpio);
+ }
+ master->gpio_trans = gpio;
+
+ gpio = devm_gpiod_get_optional(&pdev->dev, "enable", 0);
+ if (IS_ERR(gpio)) {
+ dev_err(&pdev->dev, "failed to get enable gpio\n");
+ return PTR_ERR(gpio);
+ }
+ master->gpio_enable = gpio;
+
+ gpio = devm_gpiod_get_optional(&pdev->dev, "mux", 0);
+ if (IS_ERR(gpio)) {
+ dev_err(&pdev->dev, "failed to get mux gpio\n");
+ return PTR_ERR(gpio);
+ }
+ master->gpio_mux = gpio;
+
+ master->master.n_links = 1;
+ master->master.flags = FSI_MASTER_FLAG_SWCLOCK;
+ master->master.read = fsi_master_gpio_read;
+ master->master.write = fsi_master_gpio_write;
+ master->master.term = fsi_master_gpio_term;
+ master->master.send_break = fsi_master_gpio_break;
+ master->master.link_enable = fsi_master_gpio_link_enable;
+ platform_set_drvdata(pdev, master);
+ spin_lock_init(&master->cmd_lock);
+
+ fsi_master_gpio_init(master);
+
+ return fsi_master_register(&master->master);
+}
+
+
+static int fsi_master_gpio_remove(struct platform_device *pdev)
+{
+ struct fsi_master_gpio *master = platform_get_drvdata(pdev);
+
+ devm_gpiod_put(&pdev->dev, master->gpio_clk);
+ devm_gpiod_put(&pdev->dev, master->gpio_data);
+ if (master->gpio_trans)
+ devm_gpiod_put(&pdev->dev, master->gpio_trans);
+ if (master->gpio_enable)
+ devm_gpiod_put(&pdev->dev, master->gpio_enable);
+ if (master->gpio_mux)
+ devm_gpiod_put(&pdev->dev, master->gpio_mux);
+ fsi_master_unregister(&master->master);
+
+ return 0;
+}
+
+static const struct of_device_id fsi_master_gpio_match[] = {
+ { .compatible = "fsi-master-gpio" },
+ { },
+};
+
+static struct platform_driver fsi_master_gpio_driver = {
+ .driver = {
+ .name = "fsi-master-gpio",
+ .of_match_table = fsi_master_gpio_match,
+ },
+ .probe = fsi_master_gpio_probe,
+ .remove = fsi_master_gpio_remove,
+};
+
+module_platform_driver(fsi_master_gpio_driver);
+MODULE_LICENSE("GPL");
--- /dev/null
+/*
+ * FSI hub master driver
+ *
+ * Copyright (C) IBM Corporation 2016
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/delay.h>
+#include <linux/fsi.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+
+#include "fsi-master.h"
+
+/* Control Registers */
+#define FSI_MMODE 0x0 /* R/W: mode */
+#define FSI_MDLYR 0x4 /* R/W: delay */
+#define FSI_MCRSP 0x8 /* R/W: clock rate */
+#define FSI_MENP0 0x10 /* R/W: enable */
+#define FSI_MLEVP0 0x18 /* R: plug detect */
+#define FSI_MSENP0 0x18 /* S: Set enable */
+#define FSI_MCENP0 0x20 /* C: Clear enable */
+#define FSI_MAEB 0x70 /* R: Error address */
+#define FSI_MVER 0x74 /* R: master version/type */
+#define FSI_MRESP0 0xd0 /* W: Port reset */
+#define FSI_MESRB0 0x1d0 /* R: Master error status */
+#define FSI_MRESB0 0x1d0 /* W: Reset bridge */
+#define FSI_MECTRL 0x2e0 /* W: Error control */
+
+/* MMODE: Mode control */
+#define FSI_MMODE_EIP 0x80000000 /* Enable interrupt polling */
+#define FSI_MMODE_ECRC 0x40000000 /* Enable error recovery */
+#define FSI_MMODE_EPC 0x10000000 /* Enable parity checking */
+#define FSI_MMODE_P8_TO_LSB 0x00000010 /* Timeout value LSB */
+ /* MSB=1, LSB=0 is 0.8 ms */
+ /* MSB=0, LSB=1 is 0.9 ms */
+#define FSI_MMODE_CRS0SHFT 18 /* Clk rate selection 0 shift */
+#define FSI_MMODE_CRS0MASK 0x3ff /* Clk rate selection 0 mask */
+#define FSI_MMODE_CRS1SHFT 8 /* Clk rate selection 1 shift */
+#define FSI_MMODE_CRS1MASK 0x3ff /* Clk rate selection 1 mask */
+
+/* MRESB: Reset brindge */
+#define FSI_MRESB_RST_GEN 0x80000000 /* General reset */
+#define FSI_MRESB_RST_ERR 0x40000000 /* Error Reset */
+
+/* MRESB: Reset port */
+#define FSI_MRESP_RST_ALL_MASTER 0x20000000 /* Reset all FSI masters */
+#define FSI_MRESP_RST_ALL_LINK 0x10000000 /* Reset all FSI port contr. */
+#define FSI_MRESP_RST_MCR 0x08000000 /* Reset FSI master reg. */
+#define FSI_MRESP_RST_PYE 0x04000000 /* Reset FSI parity error */
+#define FSI_MRESP_RST_ALL 0xfc000000 /* Reset any error */
+
+/* MECTRL: Error control */
+#define FSI_MECTRL_EOAE 0x8000 /* Enable machine check when */
+ /* master 0 in error */
+#define FSI_MECTRL_P8_AUTO_TERM 0x4000 /* Auto terminate */
+
+#define FSI_ENGID_HUB_MASTER 0x1c
+#define FSI_HUB_LINK_OFFSET 0x80000
+#define FSI_HUB_LINK_SIZE 0x80000
+#define FSI_HUB_MASTER_MAX_LINKS 8
+
+#define FSI_LINK_ENABLE_SETUP_TIME 10 /* in mS */
+
+/*
+ * FSI hub master support
+ *
+ * A hub master increases the number of potential target devices that the
+ * primary FSI master can access. For each link a primary master supports,
+ * each of those links can in turn be chained to a hub master with multiple
+ * links of its own.
+ *
+ * The hub is controlled by a set of control registers exposed as a regular fsi
+ * device (the hub->upstream device), and provides access to the downstream FSI
+ * bus as through an address range on the slave itself (->addr and ->size).
+ *
+ * [This differs from "cascaded" masters, which expose the entire downstream
+ * bus entirely through the fsi device address range, and so have a smaller
+ * accessible address space.]
+ */
+struct fsi_master_hub {
+ struct fsi_master master;
+ struct fsi_device *upstream;
+ uint32_t addr, size; /* slave-relative addr of */
+ /* master address space */
+};
+
+#define to_fsi_master_hub(m) container_of(m, struct fsi_master_hub, master)
+
+static int hub_master_read(struct fsi_master *master, int link,
+ uint8_t id, uint32_t addr, void *val, size_t size)
+{
+ struct fsi_master_hub *hub = to_fsi_master_hub(master);
+
+ if (id != 0)
+ return -EINVAL;
+
+ addr += hub->addr + (link * FSI_HUB_LINK_SIZE);
+ return fsi_slave_read(hub->upstream->slave, addr, val, size);
+}
+
+static int hub_master_write(struct fsi_master *master, int link,
+ uint8_t id, uint32_t addr, const void *val, size_t size)
+{
+ struct fsi_master_hub *hub = to_fsi_master_hub(master);
+
+ if (id != 0)
+ return -EINVAL;
+
+ addr += hub->addr + (link * FSI_HUB_LINK_SIZE);
+ return fsi_slave_write(hub->upstream->slave, addr, val, size);
+}
+
+static int hub_master_break(struct fsi_master *master, int link)
+{
+ uint32_t addr, cmd;
+
+ addr = 0x4;
+ cmd = cpu_to_be32(0xc0de0000);
+
+ return hub_master_write(master, link, 0, addr, &cmd, sizeof(cmd));
+}
+
+static int hub_master_link_enable(struct fsi_master *master, int link)
+{
+ struct fsi_master_hub *hub = to_fsi_master_hub(master);
+ int idx, bit;
+ __be32 reg;
+ int rc;
+
+ idx = link / 32;
+ bit = link % 32;
+
+ reg = cpu_to_be32(0x80000000 >> bit);
+
+ rc = fsi_device_write(hub->upstream, FSI_MSENP0 + (4 * idx), ®, 4);
+
+ mdelay(FSI_LINK_ENABLE_SETUP_TIME);
+
+ fsi_device_read(hub->upstream, FSI_MENP0 + (4 * idx), ®, 4);
+
+ return rc;
+}
+
+static void hub_master_release(struct device *dev)
+{
+ struct fsi_master_hub *hub = to_fsi_master_hub(dev_to_fsi_master(dev));
+
+ kfree(hub);
+}
+
+/* mmode encoders */
+static inline u32 fsi_mmode_crs0(u32 x)
+{
+ return (x & FSI_MMODE_CRS0MASK) << FSI_MMODE_CRS0SHFT;
+}
+
+static inline u32 fsi_mmode_crs1(u32 x)
+{
+ return (x & FSI_MMODE_CRS1MASK) << FSI_MMODE_CRS1SHFT;
+}
+
+static int hub_master_init(struct fsi_master_hub *hub)
+{
+ struct fsi_device *dev = hub->upstream;
+ __be32 reg;
+ int rc;
+
+ reg = cpu_to_be32(FSI_MRESP_RST_ALL_MASTER | FSI_MRESP_RST_ALL_LINK
+ | FSI_MRESP_RST_MCR | FSI_MRESP_RST_PYE);
+ rc = fsi_device_write(dev, FSI_MRESP0, ®, sizeof(reg));
+ if (rc)
+ return rc;
+
+ /* Initialize the MFSI (hub master) engine */
+ reg = cpu_to_be32(FSI_MRESP_RST_ALL_MASTER | FSI_MRESP_RST_ALL_LINK
+ | FSI_MRESP_RST_MCR | FSI_MRESP_RST_PYE);
+ rc = fsi_device_write(dev, FSI_MRESP0, ®, sizeof(reg));
+ if (rc)
+ return rc;
+
+ reg = cpu_to_be32(FSI_MECTRL_EOAE | FSI_MECTRL_P8_AUTO_TERM);
+ rc = fsi_device_write(dev, FSI_MECTRL, ®, sizeof(reg));
+ if (rc)
+ return rc;
+
+ reg = cpu_to_be32(FSI_MMODE_EIP | FSI_MMODE_ECRC | FSI_MMODE_EPC
+ | fsi_mmode_crs0(1) | fsi_mmode_crs1(1)
+ | FSI_MMODE_P8_TO_LSB);
+ rc = fsi_device_write(dev, FSI_MMODE, ®, sizeof(reg));
+ if (rc)
+ return rc;
+
+ reg = cpu_to_be32(0xffff0000);
+ rc = fsi_device_write(dev, FSI_MDLYR, ®, sizeof(reg));
+ if (rc)
+ return rc;
+
+ reg = ~0;
+ rc = fsi_device_write(dev, FSI_MSENP0, ®, sizeof(reg));
+ if (rc)
+ return rc;
+
+ /* Leave enabled long enough for master logic to set up */
+ mdelay(FSI_LINK_ENABLE_SETUP_TIME);
+
+ rc = fsi_device_write(dev, FSI_MCENP0, ®, sizeof(reg));
+ if (rc)
+ return rc;
+
+ rc = fsi_device_read(dev, FSI_MAEB, ®, sizeof(reg));
+ if (rc)
+ return rc;
+
+ reg = cpu_to_be32(FSI_MRESP_RST_ALL_MASTER | FSI_MRESP_RST_ALL_LINK);
+ rc = fsi_device_write(dev, FSI_MRESP0, ®, sizeof(reg));
+ if (rc)
+ return rc;
+
+ rc = fsi_device_read(dev, FSI_MLEVP0, ®, sizeof(reg));
+ if (rc)
+ return rc;
+
+ /* Reset the master bridge */
+ reg = cpu_to_be32(FSI_MRESB_RST_GEN);
+ rc = fsi_device_write(dev, FSI_MRESB0, ®, sizeof(reg));
+ if (rc)
+ return rc;
+
+ reg = cpu_to_be32(FSI_MRESB_RST_ERR);
+ return fsi_device_write(dev, FSI_MRESB0, ®, sizeof(reg));
+}
+
+static int hub_master_probe(struct device *dev)
+{
+ struct fsi_device *fsi_dev = to_fsi_dev(dev);
+ struct fsi_master_hub *hub;
+ uint32_t reg, links;
+ __be32 __reg;
+ int rc;
+
+ rc = fsi_device_read(fsi_dev, FSI_MVER, &__reg, sizeof(__reg));
+ if (rc)
+ return rc;
+
+ reg = be32_to_cpu(__reg);
+ links = (reg >> 8) & 0xff;
+ dev_info(dev, "hub version %08x (%d links)\n", reg, links);
+
+ rc = fsi_slave_claim_range(fsi_dev->slave, FSI_HUB_LINK_OFFSET,
+ FSI_HUB_LINK_SIZE * links);
+ if (rc) {
+ dev_err(dev, "can't claim slave address range for links");
+ return rc;
+ }
+
+ hub = kzalloc(sizeof(*hub), GFP_KERNEL);
+ if (!hub) {
+ rc = -ENOMEM;
+ goto err_release;
+ }
+
+ hub->addr = FSI_HUB_LINK_OFFSET;
+ hub->size = FSI_HUB_LINK_SIZE * links;
+ hub->upstream = fsi_dev;
+
+ hub->master.dev.parent = dev;
+ hub->master.dev.release = hub_master_release;
+
+ hub->master.n_links = links;
+ hub->master.read = hub_master_read;
+ hub->master.write = hub_master_write;
+ hub->master.send_break = hub_master_break;
+ hub->master.link_enable = hub_master_link_enable;
+
+ dev_set_drvdata(dev, hub);
+
+ hub_master_init(hub);
+
+ rc = fsi_master_register(&hub->master);
+ if (!rc)
+ return 0;
+
+ kfree(hub);
+err_release:
+ fsi_slave_release_range(fsi_dev->slave, FSI_HUB_LINK_OFFSET,
+ FSI_HUB_LINK_SIZE * links);
+ return rc;
+}
+
+static int hub_master_remove(struct device *dev)
+{
+ struct fsi_master_hub *hub = dev_get_drvdata(dev);
+
+ fsi_master_unregister(&hub->master);
+ fsi_slave_release_range(hub->upstream->slave, hub->addr, hub->size);
+ return 0;
+}
+
+static struct fsi_device_id hub_master_ids[] = {
+ {
+ .engine_type = FSI_ENGID_HUB_MASTER,
+ .version = FSI_VERSION_ANY,
+ },
+ { 0 }
+};
+
+static struct fsi_driver hub_master_driver = {
+ .id_table = hub_master_ids,
+ .drv = {
+ .name = "fsi-master-hub",
+ .bus = &fsi_bus_type,
+ .probe = hub_master_probe,
+ .remove = hub_master_remove,
+ }
+};
+
+module_fsi_driver(hub_master_driver);
+MODULE_LICENSE("GPL");
--- /dev/null
+/*
+ * FSI master definitions. These comprise the core <--> master interface,
+ * to allow the core to interact with the (hardware-specific) masters.
+ *
+ * Copyright (C) IBM Corporation 2016
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef DRIVERS_FSI_MASTER_H
+#define DRIVERS_FSI_MASTER_H
+
+#include <linux/device.h>
+
+#define FSI_MASTER_FLAG_SWCLOCK 0x1
+
+struct fsi_master {
+ struct device dev;
+ int idx;
+ int n_links;
+ int flags;
+ int (*read)(struct fsi_master *, int link, uint8_t id,
+ uint32_t addr, void *val, size_t size);
+ int (*write)(struct fsi_master *, int link, uint8_t id,
+ uint32_t addr, const void *val, size_t size);
+ int (*term)(struct fsi_master *, int link, uint8_t id);
+ int (*send_break)(struct fsi_master *, int link);
+ int (*link_enable)(struct fsi_master *, int link);
+};
+
+#define dev_to_fsi_master(d) container_of(d, struct fsi_master, dev)
+
+extern int fsi_master_register(struct fsi_master *master);
+extern void fsi_master_unregister(struct fsi_master *master);
+
+#endif /* DRIVERS_FSI_MASTER_H */
--- /dev/null
+/*
+ * SCOM FSI Client device driver
+ *
+ * Copyright (C) IBM Corporation 2016
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERGCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/fsi.h>
+#include <linux/module.h>
+#include <linux/cdev.h>
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/uaccess.h>
+#include <linux/slab.h>
+#include <linux/miscdevice.h>
+#include <linux/list.h>
+#include <linux/idr.h>
+
+#define FSI_ENGID_SCOM 0x5
+
+#define SCOM_FSI2PIB_DELAY 50
+
+/* SCOM engine register set */
+#define SCOM_DATA0_REG 0x00
+#define SCOM_DATA1_REG 0x04
+#define SCOM_CMD_REG 0x08
+#define SCOM_RESET_REG 0x1C
+
+#define SCOM_RESET_CMD 0x80000000
+#define SCOM_WRITE_CMD 0x80000000
+
+struct scom_device {
+ struct list_head link;
+ struct fsi_device *fsi_dev;
+ struct miscdevice mdev;
+ char name[32];
+ int idx;
+};
+
+#define to_scom_dev(x) container_of((x), struct scom_device, mdev)
+
+static struct list_head scom_devices;
+
+static DEFINE_IDA(scom_ida);
+
+static int put_scom(struct scom_device *scom_dev, uint64_t value,
+ uint32_t addr)
+{
+ int rc;
+ uint32_t data;
+
+ data = cpu_to_be32(SCOM_RESET_CMD);
+ rc = fsi_device_write(scom_dev->fsi_dev, SCOM_RESET_REG, &data,
+ sizeof(uint32_t));
+ if (rc)
+ return rc;
+
+ data = cpu_to_be32((value >> 32) & 0xffffffff);
+ rc = fsi_device_write(scom_dev->fsi_dev, SCOM_DATA0_REG, &data,
+ sizeof(uint32_t));
+ if (rc)
+ return rc;
+
+ data = cpu_to_be32(value & 0xffffffff);
+ rc = fsi_device_write(scom_dev->fsi_dev, SCOM_DATA1_REG, &data,
+ sizeof(uint32_t));
+ if (rc)
+ return rc;
+
+ data = cpu_to_be32(SCOM_WRITE_CMD | addr);
+ return fsi_device_write(scom_dev->fsi_dev, SCOM_CMD_REG, &data,
+ sizeof(uint32_t));
+}
+
+static int get_scom(struct scom_device *scom_dev, uint64_t *value,
+ uint32_t addr)
+{
+ uint32_t result, data;
+ int rc;
+
+ *value = 0ULL;
+ data = cpu_to_be32(addr);
+ rc = fsi_device_write(scom_dev->fsi_dev, SCOM_CMD_REG, &data,
+ sizeof(uint32_t));
+ if (rc)
+ return rc;
+
+ rc = fsi_device_read(scom_dev->fsi_dev, SCOM_DATA0_REG, &result,
+ sizeof(uint32_t));
+ if (rc)
+ return rc;
+
+ *value |= (uint64_t)cpu_to_be32(result) << 32;
+ rc = fsi_device_read(scom_dev->fsi_dev, SCOM_DATA1_REG, &result,
+ sizeof(uint32_t));
+ if (rc)
+ return rc;
+
+ *value |= cpu_to_be32(result);
+
+ return 0;
+}
+
+static ssize_t scom_read(struct file *filep, char __user *buf, size_t len,
+ loff_t *offset)
+{
+ int rc;
+ struct miscdevice *mdev =
+ (struct miscdevice *)filep->private_data;
+ struct scom_device *scom = to_scom_dev(mdev);
+ struct device *dev = &scom->fsi_dev->dev;
+ uint64_t val;
+
+ if (len != sizeof(uint64_t))
+ return -EINVAL;
+
+ rc = get_scom(scom, &val, *offset);
+ if (rc) {
+ dev_dbg(dev, "get_scom fail:%d\n", rc);
+ return rc;
+ }
+
+ rc = copy_to_user(buf, &val, len);
+ if (rc)
+ dev_dbg(dev, "copy to user failed:%d\n", rc);
+
+ return rc ? rc : len;
+}
+
+static ssize_t scom_write(struct file *filep, const char __user *buf,
+ size_t len, loff_t *offset)
+{
+ int rc;
+ struct miscdevice *mdev = filep->private_data;
+ struct scom_device *scom = to_scom_dev(mdev);
+ struct device *dev = &scom->fsi_dev->dev;
+ uint64_t val;
+
+ if (len != sizeof(uint64_t))
+ return -EINVAL;
+
+ rc = copy_from_user(&val, buf, len);
+ if (rc) {
+ dev_dbg(dev, "copy from user failed:%d\n", rc);
+ return -EINVAL;
+ }
+
+ rc = put_scom(scom, val, *offset);
+ if (rc) {
+ dev_dbg(dev, "put_scom failed with:%d\n", rc);
+ return rc;
+ }
+
+ return len;
+}
+
+static loff_t scom_llseek(struct file *file, loff_t offset, int whence)
+{
+ switch (whence) {
+ case SEEK_CUR:
+ break;
+ case SEEK_SET:
+ file->f_pos = offset;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return offset;
+}
+
+static const struct file_operations scom_fops = {
+ .owner = THIS_MODULE,
+ .llseek = scom_llseek,
+ .read = scom_read,
+ .write = scom_write,
+};
+
+static int scom_probe(struct device *dev)
+{
+ struct fsi_device *fsi_dev = to_fsi_dev(dev);
+ struct scom_device *scom;
+
+ scom = devm_kzalloc(dev, sizeof(*scom), GFP_KERNEL);
+ if (!scom)
+ return -ENOMEM;
+
+ scom->idx = ida_simple_get(&scom_ida, 1, INT_MAX, GFP_KERNEL);
+ snprintf(scom->name, sizeof(scom->name), "scom%d", scom->idx);
+ scom->fsi_dev = fsi_dev;
+ scom->mdev.minor = MISC_DYNAMIC_MINOR;
+ scom->mdev.fops = &scom_fops;
+ scom->mdev.name = scom->name;
+ scom->mdev.parent = dev;
+ list_add(&scom->link, &scom_devices);
+
+ return misc_register(&scom->mdev);
+}
+
+static int scom_remove(struct device *dev)
+{
+ struct scom_device *scom, *scom_tmp;
+ struct fsi_device *fsi_dev = to_fsi_dev(dev);
+
+ list_for_each_entry_safe(scom, scom_tmp, &scom_devices, link) {
+ if (scom->fsi_dev == fsi_dev) {
+ list_del(&scom->link);
+ ida_simple_remove(&scom_ida, scom->idx);
+ misc_deregister(&scom->mdev);
+ }
+ }
+
+ return 0;
+}
+
+static struct fsi_device_id scom_ids[] = {
+ {
+ .engine_type = FSI_ENGID_SCOM,
+ .version = FSI_VERSION_ANY,
+ },
+ { 0 }
+};
+
+static struct fsi_driver scom_drv = {
+ .id_table = scom_ids,
+ .drv = {
+ .name = "scom",
+ .bus = &fsi_bus_type,
+ .probe = scom_probe,
+ .remove = scom_remove,
+ }
+};
+
+static int scom_init(void)
+{
+ INIT_LIST_HEAD(&scom_devices);
+ return fsi_driver_register(&scom_drv);
+}
+
+static void scom_exit(void)
+{
+ struct list_head *pos;
+ struct scom_device *scom;
+
+ list_for_each(pos, &scom_devices) {
+ scom = list_entry(pos, struct scom_device, link);
+ misc_deregister(&scom->mdev);
+ devm_kfree(&scom->fsi_dev->dev, scom);
+ }
+ fsi_driver_unregister(&scom_drv);
+}
+
+module_init(scom_init);
+module_exit(scom_exit);
+MODULE_LICENSE("GPL");
u32 set;
if (!of_device_is_compatible(mvchip->chip.of_node,
- "marvell,armada-370-xp-gpio"))
+ "marvell,armada-370-gpio"))
return 0;
if (IS_ERR(mvchip->clk))
.data = (void *) MVEBU_GPIO_SOC_VARIANT_ARMADAXP,
},
{
- .compatible = "marvell,armada-370-xp-gpio",
+ .compatible = "marvell,armada-370-gpio",
.data = (void *) MVEBU_GPIO_SOC_VARIANT_ORION,
},
{
mvchip);
}
- /* Armada 370/XP has simple PWM support for GPIO lines */
+ /* Some MVEBU SoCs have simple PWM support for GPIO lines */
if (IS_ENABLED(CONFIG_PWM))
return mvebu_pwm_probe(pdev, mvchip, id);
handler = acpi_gpio_irq_handler_evt;
}
if (!handler)
- return AE_BAD_PARAMETER;
+ return AE_OK;
pin = acpi_gpiochip_pin_to_gpio_offset(chip->gpiodev, pin);
if (pin < 0)
pr_debug("%s: status %d\n", __func__, status);
return status ? : len;
}
+static CLASS_ATTR_WO(export);
static ssize_t unexport_store(struct class *class,
struct class_attribute *attr,
pr_debug("%s: status %d\n", __func__, status);
return status ? : len;
}
+static CLASS_ATTR_WO(unexport);
-static struct class_attribute gpio_class_attrs[] = {
- __ATTR(export, 0200, NULL, export_store),
- __ATTR(unexport, 0200, NULL, unexport_store),
- __ATTR_NULL,
+static struct attribute *gpio_class_attrs[] = {
+ &class_attr_export.attr,
+ &class_attr_unexport.attr,
+ NULL,
};
+ATTRIBUTE_GROUPS(gpio_class);
static struct class gpio_class = {
.name = "gpio",
.owner = THIS_MODULE,
- .class_attrs = gpio_class_attrs,
+ .class_groups = gpio_class_groups,
};
ge.timestamp = ktime_get_real_ns();
- if (le->eflags & GPIOEVENT_REQUEST_BOTH_EDGES) {
+ if (le->eflags & GPIOEVENT_REQUEST_RISING_EDGE
+ && le->eflags & GPIOEVENT_REQUEST_FALLING_EDGE) {
int level = gpiod_get_value_cansleep(le->desc);
if (level)
DRM_INFO("Changing default dispclk from %dMhz to 600Mhz\n",
adev->clock.default_dispclk / 100);
adev->clock.default_dispclk = 60000;
+ } else if (adev->clock.default_dispclk <= 60000) {
+ DRM_INFO("Changing default dispclk from %dMhz to 625Mhz\n",
+ adev->clock.default_dispclk / 100);
+ adev->clock.default_dispclk = 62500;
}
adev->clock.dp_extclk =
le16_to_cpu(firmware_info->info_21.usUniphyDPModeExtClkFreq);
{0x1002, 0x6986, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12},
{0x1002, 0x6987, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12},
{0x1002, 0x6995, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12},
+ {0x1002, 0x6997, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12},
{0x1002, 0x699F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12},
/* Vega 10 */
{0x1002, 0x6860, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10|AMD_EXP_HW_SUPPORT},
struct drm_device *dev = crtc->dev;
struct amdgpu_device *adev = dev->dev_private;
int index = GetIndexIntoMasterTable(COMMAND, EnableDispPowerGating);
- ENABLE_DISP_POWER_GATING_PARAMETERS_V2_1 args;
+ ENABLE_DISP_POWER_GATING_PS_ALLOCATION args;
memset(&args, 0, sizeof(args));
void amdgpu_atombios_crtc_powergate_init(struct amdgpu_device *adev)
{
int index = GetIndexIntoMasterTable(COMMAND, EnableDispPowerGating);
- ENABLE_DISP_POWER_GATING_PARAMETERS_V2_1 args;
+ ENABLE_DISP_POWER_GATING_PS_ALLOCATION args;
memset(&args, 0, sizeof(args));
if (!connector)
return -ENOENT;
- drm_modeset_lock(&dev->mode_config.connection_mutex, NULL);
- encoder = drm_connector_get_encoder(connector);
- if (encoder)
- out_resp->encoder_id = encoder->base.id;
- else
- out_resp->encoder_id = 0;
-
- ret = drm_mode_object_get_properties(&connector->base, file_priv->atomic,
- (uint32_t __user *)(unsigned long)(out_resp->props_ptr),
- (uint64_t __user *)(unsigned long)(out_resp->prop_values_ptr),
- &out_resp->count_props);
- drm_modeset_unlock(&dev->mode_config.connection_mutex);
- if (ret)
- goto out_unref;
-
for (i = 0; i < DRM_CONNECTOR_MAX_ENCODER; i++)
if (connector->encoder_ids[i] != 0)
encoders_count++;
if (put_user(connector->encoder_ids[i],
encoder_ptr + copied)) {
ret = -EFAULT;
- goto out_unref;
+ goto out;
}
copied++;
}
if (copy_to_user(mode_ptr + copied,
&u_mode, sizeof(u_mode))) {
ret = -EFAULT;
+ mutex_unlock(&dev->mode_config.mutex);
+
goto out;
}
copied++;
}
}
out_resp->count_modes = mode_count;
-out:
mutex_unlock(&dev->mode_config.mutex);
-out_unref:
+
+ drm_modeset_lock(&dev->mode_config.connection_mutex, NULL);
+ encoder = drm_connector_get_encoder(connector);
+ if (encoder)
+ out_resp->encoder_id = encoder->base.id;
+ else
+ out_resp->encoder_id = 0;
+
+ /* Only grab properties after probing, to make sure EDID and other
+ * properties reflect the latest status. */
+ ret = drm_mode_object_get_properties(&connector->base, file_priv->atomic,
+ (uint32_t __user *)(unsigned long)(out_resp->props_ptr),
+ (uint64_t __user *)(unsigned long)(out_resp->prop_values_ptr),
+ &out_resp->count_props);
+ drm_modeset_unlock(&dev->mode_config.connection_mutex);
+
+out:
drm_connector_put(connector);
return ret;
struct etnaviv_gpu *gpu;
struct ww_acquire_ctx ticket;
struct dma_fence *fence;
+ u32 flags;
unsigned int nr_bos;
struct etnaviv_gem_submit_bo bos[0];
- u32 flags;
+ /* No new members here, the previous one is variable-length! */
};
int etnaviv_gem_wait_bo(struct etnaviv_gpu *gpu, struct drm_gem_object *obj,
for (i = 0; i < submit->nr_bos; i++) {
struct etnaviv_gem_object *etnaviv_obj = submit->bos[i].obj;
bool write = submit->bos[i].flags & ETNA_SUBMIT_BO_WRITE;
- bool explicit = !(submit->flags & ETNA_SUBMIT_NO_IMPLICIT);
+ bool explicit = !!(submit->flags & ETNA_SUBMIT_NO_IMPLICIT);
ret = etnaviv_gpu_fence_sync_obj(etnaviv_obj, context, write,
explicit);
struct file_stats *stats = data;
struct i915_vma *vma;
+ lockdep_assert_held(&obj->base.dev->struct_mutex);
+
stats->count++;
stats->total += obj->base.size;
if (!obj->bind_count)
struct drm_i915_gem_request *request;
struct task_struct *task;
+ mutex_lock(&dev->struct_mutex);
+
memset(&stats, 0, sizeof(stats));
stats.file_priv = file->driver_priv;
spin_lock(&file->table_lock);
* still alive (e.g. get_pid(current) => fork() => exit()).
* Therefore, we need to protect this ->comm access using RCU.
*/
- mutex_lock(&dev->struct_mutex);
request = list_first_entry_or_null(&file_priv->mm.request_list,
struct drm_i915_gem_request,
client_link);
PIDTYPE_PID);
print_file_stats(m, task ? task->comm : "<unknown>", stats);
rcu_read_unlock();
+
mutex_unlock(&dev->struct_mutex);
}
mutex_unlock(&dev->filelist_mutex);
struct page *page;
unsigned long last_pfn = 0; /* suppress gcc warning */
unsigned int max_segment;
+ gfp_t noreclaim;
int ret;
- gfp_t gfp;
/* Assert that the object is not currently in any GPU domain. As it
* wasn't in the GTT, there shouldn't be any way it could have been in
* Fail silently without starting the shrinker
*/
mapping = obj->base.filp->f_mapping;
- gfp = mapping_gfp_constraint(mapping, ~(__GFP_IO | __GFP_RECLAIM));
- gfp |= __GFP_NORETRY | __GFP_NOWARN;
+ noreclaim = mapping_gfp_constraint(mapping,
+ ~(__GFP_IO | __GFP_RECLAIM));
+ noreclaim |= __GFP_NORETRY | __GFP_NOWARN;
+
sg = st->sgl;
st->nents = 0;
for (i = 0; i < page_count; i++) {
- page = shmem_read_mapping_page_gfp(mapping, i, gfp);
- if (unlikely(IS_ERR(page))) {
- i915_gem_shrink(dev_priv,
- page_count,
- I915_SHRINK_BOUND |
- I915_SHRINK_UNBOUND |
- I915_SHRINK_PURGEABLE);
+ const unsigned int shrink[] = {
+ I915_SHRINK_BOUND | I915_SHRINK_UNBOUND | I915_SHRINK_PURGEABLE,
+ 0,
+ }, *s = shrink;
+ gfp_t gfp = noreclaim;
+
+ do {
page = shmem_read_mapping_page_gfp(mapping, i, gfp);
- }
- if (unlikely(IS_ERR(page))) {
- gfp_t reclaim;
+ if (likely(!IS_ERR(page)))
+ break;
+
+ if (!*s) {
+ ret = PTR_ERR(page);
+ goto err_sg;
+ }
+
+ i915_gem_shrink(dev_priv, 2 * page_count, *s++);
+ cond_resched();
/* We've tried hard to allocate the memory by reaping
* our own buffer, now let the real VM do its job and
* defer the oom here by reporting the ENOMEM back
* to userspace.
*/
- reclaim = mapping_gfp_mask(mapping);
- reclaim |= __GFP_NORETRY; /* reclaim, but no oom */
-
- page = shmem_read_mapping_page_gfp(mapping, i, reclaim);
- if (IS_ERR(page)) {
- ret = PTR_ERR(page);
- goto err_sg;
+ if (!*s) {
+ /* reclaim and warn, but no oom */
+ gfp = mapping_gfp_mask(mapping);
+
+ /* Our bo are always dirty and so we require
+ * kswapd to reclaim our pages (direct reclaim
+ * does not effectively begin pageout of our
+ * buffers on its own). However, direct reclaim
+ * only waits for kswapd when under allocation
+ * congestion. So as a result __GFP_RECLAIM is
+ * unreliable and fails to actually reclaim our
+ * dirty pages -- unless you try over and over
+ * again with !__GFP_NORETRY. However, we still
+ * want to fail this allocation rather than
+ * trigger the out-of-memory killer and for
+ * this we want the future __GFP_MAYFAIL.
+ */
}
- }
+ } while (1);
+
if (!i ||
sg->length >= max_segment ||
page_to_pfn(page) != last_pfn + 1) {
mapping = obj->base.filp->f_mapping;
mapping_set_gfp_mask(mapping, mask);
+ GEM_BUG_ON(!(mapping_gfp_mask(mapping) & __GFP_RECLAIM));
i915_gem_object_init(obj, &i915_gem_object_ops);
}
static int
-i915_gem_execbuffer_relocate_entry(struct drm_i915_gem_object *obj,
+i915_gem_execbuffer_relocate_entry(struct i915_vma *vma,
struct eb_vmas *eb,
struct drm_i915_gem_relocation_entry *reloc,
struct reloc_cache *cache)
{
+ struct drm_i915_gem_object *obj = vma->obj;
struct drm_i915_private *dev_priv = to_i915(obj->base.dev);
struct drm_gem_object *target_obj;
struct drm_i915_gem_object *target_i915_obj;
return -EINVAL;
}
+ /*
+ * If we write into the object, we need to force the synchronisation
+ * barrier, either with an asynchronous clflush or if we executed the
+ * patching using the GPU (though that should be serialised by the
+ * timeline). To be completely sure, and since we are required to
+ * do relocations we are already stalling, disable the user's opt
+ * of our synchronisation.
+ */
+ vma->exec_entry->flags &= ~EXEC_OBJECT_ASYNC;
+
ret = relocate_entry(obj, reloc, cache, target_offset);
if (ret)
return ret;
do {
u64 offset = r->presumed_offset;
- ret = i915_gem_execbuffer_relocate_entry(vma->obj, eb, r, &cache);
+ ret = i915_gem_execbuffer_relocate_entry(vma, eb, r, &cache);
if (ret)
goto out;
reloc_cache_init(&cache, eb->i915);
for (i = 0; i < entry->relocation_count; i++) {
- ret = i915_gem_execbuffer_relocate_entry(vma->obj, eb, &relocs[i], &cache);
+ ret = i915_gem_execbuffer_relocate_entry(vma, eb, &relocs[i], &cache);
if (ret)
break;
}
* GPU processing the request, we never over-estimate the
* position of the head.
*/
- req->head = req->ring->tail;
+ req->head = req->ring->emit;
/* Check that we didn't interrupt ourselves with a new request */
GEM_BUG_ON(req->timeline->seqno != req->fence.seqno);
* It is used by the driver to then queue the request for execution.
*/
struct i915_sw_fence submit;
- wait_queue_t submitq;
+ wait_queue_entry_t submitq;
wait_queue_head_t execute;
/* A list of everyone we wait upon, and everyone who waits upon us.
GEM_BUG_ON(freespace < wqi_size);
/* The GuC firmware wants the tail index in QWords, not bytes */
- tail = rq->tail;
- assert_ring_tail_valid(rq->ring, rq->tail);
- tail >>= 3;
+ tail = intel_ring_set_tail(rq->ring, rq->tail) >> 3;
GEM_BUG_ON(tail > WQ_RING_TAIL_MAX);
/* For now workqueue item is 4 DWs; workqueue buffer is 2 pages. So we
struct list_head *continuation)
{
wait_queue_head_t *x = &fence->wait;
- wait_queue_t *pos, *next;
+ wait_queue_entry_t *pos, *next;
unsigned long flags;
debug_fence_deactivate(fence);
/*
* To prevent unbounded recursion as we traverse the graph of
- * i915_sw_fences, we move the task_list from this, the next ready
- * fence, to the tail of the original fence's task_list
+ * i915_sw_fences, we move the entry list from this, the next ready
+ * fence, to the tail of the original fence's entry list
* (and so added to the list to be woken).
*/
spin_lock_irqsave_nested(&x->lock, flags, 1 + !!continuation);
if (continuation) {
- list_for_each_entry_safe(pos, next, &x->task_list, task_list) {
+ list_for_each_entry_safe(pos, next, &x->head, entry) {
if (pos->func == autoremove_wake_function)
pos->func(pos, TASK_NORMAL, 0, continuation);
else
- list_move_tail(&pos->task_list, continuation);
+ list_move_tail(&pos->entry, continuation);
}
} else {
LIST_HEAD(extra);
do {
- list_for_each_entry_safe(pos, next,
- &x->task_list, task_list)
+ list_for_each_entry_safe(pos, next, &x->head, entry)
pos->func(pos, TASK_NORMAL, 0, &extra);
if (list_empty(&extra))
break;
- list_splice_tail_init(&extra, &x->task_list);
+ list_splice_tail_init(&extra, &x->head);
} while (1);
}
spin_unlock_irqrestore(&x->lock, flags);
__i915_sw_fence_commit(fence);
}
-static int i915_sw_fence_wake(wait_queue_t *wq, unsigned mode, int flags, void *key)
+static int i915_sw_fence_wake(wait_queue_entry_t *wq, unsigned mode, int flags, void *key)
{
- list_del(&wq->task_list);
+ list_del(&wq->entry);
__i915_sw_fence_complete(wq->private, key);
i915_sw_fence_put(wq->private);
if (wq->flags & I915_SW_FENCE_FLAG_ALLOC)
static bool __i915_sw_fence_check_if_after(struct i915_sw_fence *fence,
const struct i915_sw_fence * const signaler)
{
- wait_queue_t *wq;
+ wait_queue_entry_t *wq;
if (__test_and_set_bit(I915_SW_FENCE_CHECKED_BIT, &fence->flags))
return false;
if (fence == signaler)
return true;
- list_for_each_entry(wq, &fence->wait.task_list, task_list) {
+ list_for_each_entry(wq, &fence->wait.head, entry) {
if (wq->func != i915_sw_fence_wake)
continue;
static void __i915_sw_fence_clear_checked_bit(struct i915_sw_fence *fence)
{
- wait_queue_t *wq;
+ wait_queue_entry_t *wq;
if (!__test_and_clear_bit(I915_SW_FENCE_CHECKED_BIT, &fence->flags))
return;
- list_for_each_entry(wq, &fence->wait.task_list, task_list) {
+ list_for_each_entry(wq, &fence->wait.head, entry) {
if (wq->func != i915_sw_fence_wake)
continue;
static int __i915_sw_fence_await_sw_fence(struct i915_sw_fence *fence,
struct i915_sw_fence *signaler,
- wait_queue_t *wq, gfp_t gfp)
+ wait_queue_entry_t *wq, gfp_t gfp)
{
unsigned long flags;
int pending;
pending |= I915_SW_FENCE_FLAG_ALLOC;
}
- INIT_LIST_HEAD(&wq->task_list);
+ INIT_LIST_HEAD(&wq->entry);
wq->flags = pending;
wq->func = i915_sw_fence_wake;
wq->private = i915_sw_fence_get(fence);
spin_lock_irqsave(&signaler->wait.lock, flags);
if (likely(!i915_sw_fence_done(signaler))) {
- __add_wait_queue_tail(&signaler->wait, wq);
+ __add_wait_queue_entry_tail(&signaler->wait, wq);
pending = 1;
} else {
i915_sw_fence_wake(wq, 0, 0, NULL);
int i915_sw_fence_await_sw_fence(struct i915_sw_fence *fence,
struct i915_sw_fence *signaler,
- wait_queue_t *wq)
+ wait_queue_entry_t *wq)
{
return __i915_sw_fence_await_sw_fence(fence, signaler, wq, 0);
}
int i915_sw_fence_await_sw_fence(struct i915_sw_fence *fence,
struct i915_sw_fence *after,
- wait_queue_t *wq);
+ wait_queue_entry_t *wq);
int i915_sw_fence_await_sw_fence_gfp(struct i915_sw_fence *fence,
struct i915_sw_fence *after,
gfp_t gfp);
break;
}
+ if (!ret) {
+ ret = i915_gem_active_retire(&vma->last_fence,
+ &vma->vm->i915->drm.struct_mutex);
+ }
+
__i915_vma_unpin(vma);
if (ret)
return ret;
acpi_handle dhandle;
} intel_dsm_priv;
-static const u8 intel_dsm_guid[] = {
- 0xd3, 0x73, 0xd8, 0x7e,
- 0xd0, 0xc2,
- 0x4f, 0x4e,
- 0xa8, 0x54,
- 0x0f, 0x13, 0x17, 0xb0, 0x1c, 0x2c
-};
+static const guid_t intel_dsm_guid =
+ GUID_INIT(0x7ed873d3, 0xc2d0, 0x4e4f,
+ 0xa8, 0x54, 0x0f, 0x13, 0x17, 0xb0, 0x1c, 0x2c);
static char *intel_dsm_port_name(u8 id)
{
int i;
union acpi_object *pkg, *connector_count;
- pkg = acpi_evaluate_dsm_typed(intel_dsm_priv.dhandle, intel_dsm_guid,
+ pkg = acpi_evaluate_dsm_typed(intel_dsm_priv.dhandle, &intel_dsm_guid,
INTEL_DSM_REVISION_ID, INTEL_DSM_FN_PLATFORM_MUX_INFO,
NULL, ACPI_TYPE_PACKAGE);
if (!pkg) {
if (!dhandle)
return false;
- if (!acpi_check_dsm(dhandle, intel_dsm_guid, INTEL_DSM_REVISION_ID,
+ if (!acpi_check_dsm(dhandle, &intel_dsm_guid, INTEL_DSM_REVISION_ID,
1 << INTEL_DSM_FN_PLATFORM_MUX_INFO)) {
DRM_DEBUG_KMS("no _DSM method for intel device\n");
return false;
static void skylake_pfit_enable(struct intel_crtc *crtc);
static void ironlake_pfit_disable(struct intel_crtc *crtc, bool force);
static void ironlake_pfit_enable(struct intel_crtc *crtc);
-static void intel_modeset_setup_hw_state(struct drm_device *dev);
+static void intel_modeset_setup_hw_state(struct drm_device *dev,
+ struct drm_modeset_acquire_ctx *ctx);
static void intel_pre_disable_primary_noatomic(struct drm_crtc *crtc);
struct intel_limit {
struct drm_crtc *crtc;
int i, ret;
- intel_modeset_setup_hw_state(dev);
+ intel_modeset_setup_hw_state(dev, ctx);
i915_redisable_vga(to_i915(dev));
if (!state)
intel_update_watermarks(intel_crtc);
}
-static void intel_crtc_disable_noatomic(struct drm_crtc *crtc)
+static void intel_crtc_disable_noatomic(struct drm_crtc *crtc,
+ struct drm_modeset_acquire_ctx *ctx)
{
struct intel_encoder *encoder;
struct intel_crtc *intel_crtc = to_intel_crtc(crtc);
return;
}
- state->acquire_ctx = crtc->dev->mode_config.acquire_ctx;
+ state->acquire_ctx = ctx;
/* Everything's already locked, -EDEADLK can't happen. */
crtc_state = intel_atomic_get_crtc_state(state, intel_crtc);
intel_setup_outputs(dev_priv);
drm_modeset_lock_all(dev);
- intel_modeset_setup_hw_state(dev);
+ intel_modeset_setup_hw_state(dev, dev->mode_config.acquire_ctx);
drm_modeset_unlock_all(dev);
for_each_intel_crtc(dev, crtc) {
return 0;
}
-static void intel_enable_pipe_a(struct drm_device *dev)
+static void intel_enable_pipe_a(struct drm_device *dev,
+ struct drm_modeset_acquire_ctx *ctx)
{
struct intel_connector *connector;
struct drm_connector_list_iter conn_iter;
struct drm_connector *crt = NULL;
struct intel_load_detect_pipe load_detect_temp;
- struct drm_modeset_acquire_ctx *ctx = dev->mode_config.acquire_ctx;
int ret;
/* We can't just switch on the pipe A, we need to set things up with a
(HAS_PCH_LPT_H(dev_priv) && pch_transcoder == TRANSCODER_A);
}
-static void intel_sanitize_crtc(struct intel_crtc *crtc)
+static void intel_sanitize_crtc(struct intel_crtc *crtc,
+ struct drm_modeset_acquire_ctx *ctx)
{
struct drm_device *dev = crtc->base.dev;
struct drm_i915_private *dev_priv = to_i915(dev);
plane = crtc->plane;
crtc->base.primary->state->visible = true;
crtc->plane = !plane;
- intel_crtc_disable_noatomic(&crtc->base);
+ intel_crtc_disable_noatomic(&crtc->base, ctx);
crtc->plane = plane;
}
* resume. Force-enable the pipe to fix this, the update_dpms
* call below we restore the pipe to the right state, but leave
* the required bits on. */
- intel_enable_pipe_a(dev);
+ intel_enable_pipe_a(dev, ctx);
}
/* Adjust the state of the output pipe according to whether we
* have active connectors/encoders. */
if (crtc->active && !intel_crtc_has_encoders(crtc))
- intel_crtc_disable_noatomic(&crtc->base);
+ intel_crtc_disable_noatomic(&crtc->base, ctx);
if (crtc->active || HAS_GMCH_DISPLAY(dev_priv)) {
/*
* and sanitizes it to the current state
*/
static void
-intel_modeset_setup_hw_state(struct drm_device *dev)
+intel_modeset_setup_hw_state(struct drm_device *dev,
+ struct drm_modeset_acquire_ctx *ctx)
{
struct drm_i915_private *dev_priv = to_i915(dev);
enum pipe pipe;
for_each_pipe(dev_priv, pipe) {
crtc = intel_get_crtc_for_pipe(dev_priv, pipe);
- intel_sanitize_crtc(crtc);
+ intel_sanitize_crtc(crtc, ctx);
intel_dump_pipe_config(crtc, crtc->config,
"[setup_hw_state]");
}
struct intel_dp *intel_dp = enc_to_intel_dp(&connector->encoder->base);
struct intel_panel *panel = &connector->panel;
- intel_dp_aux_enable_backlight(connector);
-
if (intel_dp->edp_dpcd[2] & DP_EDP_BACKLIGHT_BRIGHTNESS_BYTE_COUNT)
panel->backlight.max = 0xFFFF;
else
rq->ctx->ppgtt ?: rq->i915->mm.aliasing_ppgtt;
u32 *reg_state = ce->lrc_reg_state;
- assert_ring_tail_valid(rq->ring, rq->tail);
- reg_state[CTX_RING_TAIL+1] = rq->tail;
+ reg_state[CTX_RING_TAIL+1] = intel_ring_set_tail(rq->ring, rq->tail);
/* True 32b PPGTT with dynamic page allocation: update PDP
* registers and point the unallocated PDPs to scratch page.
ce->state->obj->mm.dirty = true;
i915_gem_object_unpin_map(ce->state->obj);
- ce->ring->head = ce->ring->tail = 0;
- intel_ring_update_space(ce->ring);
+ intel_ring_reset(ce->ring, 0);
}
}
}
void intel_ring_update_space(struct intel_ring *ring)
{
- ring->space = __intel_ring_space(ring->head, ring->tail, ring->size);
+ ring->space = __intel_ring_space(ring->head, ring->emit, ring->size);
}
static int
i915_gem_request_submit(request);
- assert_ring_tail_valid(request->ring, request->tail);
- I915_WRITE_TAIL(request->engine, request->tail);
+ I915_WRITE_TAIL(request->engine,
+ intel_ring_set_tail(request->ring, request->tail));
}
static void i9xx_emit_breadcrumb(struct drm_i915_gem_request *req, u32 *cs)
return PTR_ERR(addr);
}
+void intel_ring_reset(struct intel_ring *ring, u32 tail)
+{
+ GEM_BUG_ON(!list_empty(&ring->request_list));
+ ring->tail = tail;
+ ring->head = tail;
+ ring->emit = tail;
+ intel_ring_update_space(ring);
+}
+
void intel_ring_unpin(struct intel_ring *ring)
{
GEM_BUG_ON(!ring->vma);
GEM_BUG_ON(!ring->vaddr);
+ /* Discard any unused bytes beyond that submitted to hw. */
+ intel_ring_reset(ring, ring->tail);
+
if (i915_vma_is_map_and_fenceable(ring->vma))
i915_vma_unpin_iomap(ring->vma);
else
struct intel_engine_cs *engine;
enum intel_engine_id id;
+ /* Restart from the beginning of the rings for convenience */
for_each_engine(engine, dev_priv, id)
- engine->buffer->head = engine->buffer->tail;
+ intel_ring_reset(engine->buffer, 0);
}
static int ring_request_alloc(struct drm_i915_gem_request *request)
unsigned space;
/* Would completion of this request free enough space? */
- space = __intel_ring_space(target->postfix, ring->tail,
+ space = __intel_ring_space(target->postfix, ring->emit,
ring->size);
if (space >= bytes)
break;
u32 *intel_ring_begin(struct drm_i915_gem_request *req, int num_dwords)
{
struct intel_ring *ring = req->ring;
- int remain_actual = ring->size - ring->tail;
- int remain_usable = ring->effective_size - ring->tail;
+ int remain_actual = ring->size - ring->emit;
+ int remain_usable = ring->effective_size - ring->emit;
int bytes = num_dwords * sizeof(u32);
int total_bytes, wait_bytes;
bool need_wrap = false;
if (unlikely(need_wrap)) {
GEM_BUG_ON(remain_actual > ring->space);
- GEM_BUG_ON(ring->tail + remain_actual > ring->size);
+ GEM_BUG_ON(ring->emit + remain_actual > ring->size);
/* Fill the tail with MI_NOOP */
- memset(ring->vaddr + ring->tail, 0, remain_actual);
- ring->tail = 0;
+ memset(ring->vaddr + ring->emit, 0, remain_actual);
+ ring->emit = 0;
ring->space -= remain_actual;
}
- GEM_BUG_ON(ring->tail > ring->size - bytes);
- cs = ring->vaddr + ring->tail;
- ring->tail += bytes;
+ GEM_BUG_ON(ring->emit > ring->size - bytes);
+ cs = ring->vaddr + ring->emit;
+ ring->emit += bytes;
ring->space -= bytes;
GEM_BUG_ON(ring->space < 0);
int intel_ring_cacheline_align(struct drm_i915_gem_request *req)
{
int num_dwords =
- (req->ring->tail & (CACHELINE_BYTES - 1)) / sizeof(uint32_t);
+ (req->ring->emit & (CACHELINE_BYTES - 1)) / sizeof(uint32_t);
u32 *cs;
if (num_dwords == 0)
u32 head;
u32 tail;
+ u32 emit;
int space;
int size;
struct intel_ring *
intel_engine_create_ring(struct intel_engine_cs *engine, int size);
int intel_ring_pin(struct intel_ring *ring, unsigned int offset_bias);
+void intel_ring_reset(struct intel_ring *ring, u32 tail);
+void intel_ring_update_space(struct intel_ring *ring);
void intel_ring_unpin(struct intel_ring *ring);
void intel_ring_free(struct intel_ring *ring);
* reserved for the command packet (i.e. the value passed to
* intel_ring_begin()).
*/
- GEM_BUG_ON((req->ring->vaddr + req->ring->tail) != cs);
+ GEM_BUG_ON((req->ring->vaddr + req->ring->emit) != cs);
}
static inline u32
GEM_BUG_ON(tail >= ring->size);
}
-void intel_ring_update_space(struct intel_ring *ring);
+static inline unsigned int
+intel_ring_set_tail(struct intel_ring *ring, unsigned int tail)
+{
+ /* Whilst writes to the tail are strictly order, there is no
+ * serialisation between readers and the writers. The tail may be
+ * read by i915_gem_request_retire() just as it is being updated
+ * by execlists, as although the breadcrumb is complete, the context
+ * switch hasn't been seen.
+ */
+ assert_ring_tail_valid(ring, tail);
+ ring->tail = tail;
+ return tail;
+}
void intel_engine_init_global_seqno(struct intel_engine_cs *engine, u32 seqno);
}
#ifdef CONFIG_VGA_SWITCHEROO
-static const char nouveau_dsm_muid[] = {
- 0xA0, 0xA0, 0x95, 0x9D, 0x60, 0x00, 0x48, 0x4D,
- 0xB3, 0x4D, 0x7E, 0x5F, 0xEA, 0x12, 0x9F, 0xD4,
-};
+static const guid_t nouveau_dsm_muid =
+ GUID_INIT(0x9D95A0A0, 0x0060, 0x4D48,
+ 0xB3, 0x4D, 0x7E, 0x5F, 0xEA, 0x12, 0x9F, 0xD4);
-static const char nouveau_op_dsm_muid[] = {
- 0xF8, 0xD8, 0x86, 0xA4, 0xDA, 0x0B, 0x1B, 0x47,
- 0xA7, 0x2B, 0x60, 0x42, 0xA6, 0xB5, 0xBE, 0xE0,
-};
+static const guid_t nouveau_op_dsm_muid =
+ GUID_INIT(0xA486D8F8, 0x0BDA, 0x471B,
+ 0xA7, 0x2B, 0x60, 0x42, 0xA6, 0xB5, 0xBE, 0xE0);
static int nouveau_optimus_dsm(acpi_handle handle, int func, int arg, uint32_t *result)
{
args_buff[i] = (arg >> i * 8) & 0xFF;
*result = 0;
- obj = acpi_evaluate_dsm_typed(handle, nouveau_op_dsm_muid, 0x00000100,
+ obj = acpi_evaluate_dsm_typed(handle, &nouveau_op_dsm_muid, 0x00000100,
func, &argv4, ACPI_TYPE_BUFFER);
if (!obj) {
acpi_handle_info(handle, "failed to evaluate _DSM\n");
.integer.value = arg,
};
- obj = acpi_evaluate_dsm_typed(handle, nouveau_dsm_muid, 0x00000102,
+ obj = acpi_evaluate_dsm_typed(handle, &nouveau_dsm_muid, 0x00000102,
func, &argv4, ACPI_TYPE_INTEGER);
if (!obj) {
acpi_handle_info(handle, "failed to evaluate _DSM\n");
if (!acpi_has_method(dhandle, "_DSM"))
return;
- supports_mux = acpi_check_dsm(dhandle, nouveau_dsm_muid, 0x00000102,
+ supports_mux = acpi_check_dsm(dhandle, &nouveau_dsm_muid, 0x00000102,
1 << NOUVEAU_DSM_POWER);
optimus_funcs = nouveau_dsm_get_optimus_functions(dhandle);
{
struct nvkm_subdev *subdev = &mxm->subdev;
struct nvkm_device *device = subdev->device;
- static char muid[] = {
- 0x00, 0xA4, 0x04, 0x40, 0x7D, 0x91, 0xF2, 0x4C,
- 0xB8, 0x9C, 0x79, 0xB6, 0x2F, 0xD5, 0x56, 0x65
- };
+ static guid_t muid =
+ GUID_INIT(0x4004A400, 0x917D, 0x4CF2,
+ 0xB8, 0x9C, 0x79, 0xB6, 0x2F, 0xD5, 0x56, 0x65);
u32 mxms_args[] = { 0x00000000 };
union acpi_object argv4 = {
.buffer.type = ACPI_TYPE_BUFFER,
* unless you pass in exactly the version it supports..
*/
rev = (version & 0xf0) << 4 | (version & 0x0f);
- obj = acpi_evaluate_dsm(handle, muid, rev, 0x00000010, &argv4);
+ obj = acpi_evaluate_dsm(handle, &muid, rev, 0x00000010, &argv4);
if (!obj) {
nvkm_debug(subdev, "DSM MXMS failed\n");
return false;
unsigned ring;
bool is_vm_update;
- wait_queue_t fence_wake;
+ wait_queue_entry_t fence_wake;
};
int radeon_fence_driver_start_ring(struct radeon_device *rdev, int ring);
rdev->pdev->subsystem_vendor == 0x103c &&
rdev->pdev->subsystem_device == 0x280a)
return;
+ /* quirk for rs4xx Toshiba Sattellite L20-183 latop to make it resume
+ * - it hangs on resume inside the dynclk 1 table.
+ */
+ if (rdev->family == CHIP_RS400 &&
+ rdev->pdev->subsystem_vendor == 0x1179 &&
+ rdev->pdev->subsystem_device == 0xff31)
+ return;
/* DYN CLK 1 */
table = combios_get_table_offset(dev, COMBIOS_DYN_CLK_1_TABLE);
* https://bugzilla.kernel.org/show_bug.cgi?id=51381
*/
{ PCI_VENDOR_ID_ATI, 0x6840, 0x1043, 0x2122, RADEON_PX_QUIRK_DISABLE_PX },
+ /* Asus K53TK laptop with AMD A6-3420M APU and Radeon 7670m GPU
+ * https://bugs.freedesktop.org/show_bug.cgi?id=101491
+ */
+ { PCI_VENDOR_ID_ATI, 0x6741, 0x1043, 0x2122, RADEON_PX_QUIRK_DISABLE_PX },
/* macbook pro 8.2 */
{ PCI_VENDOR_ID_ATI, 0x6741, PCI_VENDOR_ID_APPLE, 0x00e2, RADEON_PX_QUIRK_LONG_WAKEUP },
{ 0, 0, 0, 0, 0 },
* for the fence locking itself, so unlocked variants are used for
* fence_signal, and remove_wait_queue.
*/
-static int radeon_fence_check_signaled(wait_queue_t *wait, unsigned mode, int flags, void *key)
+static int radeon_fence_check_signaled(wait_queue_entry_t *wait, unsigned mode, int flags, void *key)
{
struct radeon_fence *fence;
u64 seq;
list_for_each_entry_safe(entry, next, &man->list, head)
vmw_cmdbuf_res_free(man, entry);
+ drm_ht_remove(&man->resources);
kfree(man);
}
{
struct vga_device *vgadev, *conflict;
unsigned long flags;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
int rc = 0;
vga_check_first_use();
* Adds a new dynamic hid device ID to this driver,
* and causes the driver to probe for all devices again.
*/
-static ssize_t store_new_id(struct device_driver *drv, const char *buf,
+static ssize_t new_id_store(struct device_driver *drv, const char *buf,
size_t count)
{
struct hid_driver *hdrv = to_hid_driver(drv);
return ret ? : count;
}
-static DRIVER_ATTR(new_id, S_IWUSR, NULL, store_new_id);
+static DRIVER_ATTR_WO(new_id);
+
+static struct attribute *hid_drv_attrs[] = {
+ &driver_attr_new_id.attr,
+ NULL,
+};
+ATTRIBUTE_GROUPS(hid_drv);
static void hid_free_dynids(struct hid_driver *hdrv)
{
static struct bus_type hid_bus_type = {
.name = "hid",
.dev_groups = hid_dev_groups,
+ .drv_groups = hid_drv_groups,
.match = hid_bus_match,
.probe = hid_device_probe,
.remove = hid_device_remove,
int __hid_register_driver(struct hid_driver *hdrv, struct module *owner,
const char *mod_name)
{
- int ret;
-
hdrv->driver.name = hdrv->name;
hdrv->driver.bus = &hid_bus_type;
hdrv->driver.owner = owner;
INIT_LIST_HEAD(&hdrv->dyn_list);
spin_lock_init(&hdrv->dyn_lock);
- ret = driver_register(&hdrv->driver);
- if (ret)
- return ret;
-
- ret = driver_create_file(&hdrv->driver, &driver_attr_new_id);
- if (ret)
- driver_unregister(&hdrv->driver);
-
- return ret;
+ return driver_register(&hdrv->driver);
}
EXPORT_SYMBOL_GPL(__hid_register_driver);
void hid_unregister_driver(struct hid_driver *hdrv)
{
- driver_remove_file(&hdrv->driver, &driver_attr_new_id);
driver_unregister(&hdrv->driver);
hid_free_dynids(hdrv);
}
#define USB_VENDOR_ID_DELCOM 0x0fc5
#define USB_DEVICE_ID_DELCOM_VISUAL_IND 0xb080
+#define USB_VENDOR_ID_DELL 0x413c
+#define USB_DEVICE_ID_DELL_PIXART_USB_OPTICAL_MOUSE 0x301a
+
#define USB_VENDOR_ID_DELORME 0x1163
#define USB_DEVICE_ID_DELORME_EARTHMATE 0x0100
#define USB_DEVICE_ID_DELORME_EM_LT20 0x0200
if (input->id.product == USB_DEVICE_ID_APPLE_MAGICMOUSE) {
magicmouse_emit_buttons(msc, clicks & 3);
- input_mt_report_pointer_emulation(input, true);
input_report_rel(input, REL_X, x);
input_report_rel(input, REL_Y, y);
} else { /* USB_DEVICE_ID_APPLE_MAGICTRACKPAD */
__clear_bit(BTN_RIGHT, input->keybit);
__clear_bit(BTN_MIDDLE, input->keybit);
__set_bit(BTN_MOUSE, input->keybit);
+ __set_bit(BTN_TOOL_FINGER, input->keybit);
+ __set_bit(BTN_TOOL_DOUBLETAP, input->keybit);
+ __set_bit(BTN_TOOL_TRIPLETAP, input->keybit);
+ __set_bit(BTN_TOOL_QUADTAP, input->keybit);
+ __set_bit(BTN_TOOL_QUINTTAP, input->keybit);
+ __set_bit(BTN_TOUCH, input->keybit);
+ __set_bit(INPUT_PROP_POINTER, input->propbit);
__set_bit(INPUT_PROP_BUTTONPAD, input->propbit);
}
- __set_bit(BTN_TOOL_FINGER, input->keybit);
- __set_bit(BTN_TOOL_DOUBLETAP, input->keybit);
- __set_bit(BTN_TOOL_TRIPLETAP, input->keybit);
- __set_bit(BTN_TOOL_QUADTAP, input->keybit);
- __set_bit(BTN_TOOL_QUINTTAP, input->keybit);
- __set_bit(BTN_TOUCH, input->keybit);
- __set_bit(INPUT_PROP_POINTER, input->propbit);
__set_bit(EV_ABS, input->evbit);
static int i2c_hid_acpi_pdata(struct i2c_client *client,
struct i2c_hid_platform_data *pdata)
{
- static u8 i2c_hid_guid[] = {
- 0xF7, 0xF6, 0xDF, 0x3C, 0x67, 0x42, 0x55, 0x45,
- 0xAD, 0x05, 0xB3, 0x0A, 0x3D, 0x89, 0x38, 0xDE,
- };
+ static guid_t i2c_hid_guid =
+ GUID_INIT(0x3CDFF6F7, 0x4267, 0x4555,
+ 0xAD, 0x05, 0xB3, 0x0A, 0x3D, 0x89, 0x38, 0xDE);
union acpi_object *obj;
struct acpi_device *adev;
acpi_handle handle;
if (!handle || acpi_bus_get_device(handle, &adev))
return -ENODEV;
- obj = acpi_evaluate_dsm_typed(handle, i2c_hid_guid, 1, 1, NULL,
+ obj = acpi_evaluate_dsm_typed(handle, &i2c_hid_guid, 1, 1, NULL,
ACPI_TYPE_INTEGER);
if (!obj) {
dev_err(&client->dev, "device _DSM execution failed\n");
len = snprintf(buf, PAGE_SIZE, "ishtp:%s\n", dev_name(dev));
return (len >= PAGE_SIZE) ? (PAGE_SIZE - 1) : len;
}
+static DEVICE_ATTR_RO(modalias);
-static struct device_attribute ishtp_cl_dev_attrs[] = {
- __ATTR_RO(modalias),
- __ATTR_NULL,
+static struct attribute *ishtp_cl_dev_attrs[] = {
+ &dev_attr_modalias.attr,
+ NULL,
};
+ATTRIBUTE_GROUPS(ishtp_cl_dev);
static int ishtp_cl_uevent(struct device *dev, struct kobj_uevent_env *env)
{
static struct bus_type ishtp_cl_bus_type = {
.name = "ishtp",
- .dev_attrs = ishtp_cl_dev_attrs,
+ .dev_groups = ishtp_cl_dev_groups,
.probe = ishtp_cl_device_probe,
.remove = ishtp_cl_device_remove,
.pm = &ishtp_cl_bus_dev_pm_ops,
{ USB_VENDOR_ID_CORSAIR, USB_DEVICE_ID_CORSAIR_K65RGB_RAPIDFIRE, HID_QUIRK_NO_INIT_REPORTS | HID_QUIRK_ALWAYS_POLL },
{ USB_VENDOR_ID_CORSAIR, USB_DEVICE_ID_CORSAIR_SCIMITAR_PRO_RGB, HID_QUIRK_NO_INIT_REPORTS | HID_QUIRK_ALWAYS_POLL },
{ USB_VENDOR_ID_CREATIVELABS, USB_DEVICE_ID_CREATIVE_SB_OMNI_SURROUND_51, HID_QUIRK_NOGET },
+ { USB_VENDOR_ID_DELL, USB_DEVICE_ID_DELL_PIXART_USB_OPTICAL_MOUSE, HID_QUIRK_ALWAYS_POLL },
{ USB_VENDOR_ID_DMI, USB_DEVICE_ID_DMI_ENC, HID_QUIRK_NOGET },
{ USB_VENDOR_ID_DRAGONRISE, USB_DEVICE_ID_DRAGONRISE_WIIU, HID_QUIRK_MULTI_INPUT },
{ USB_VENDOR_ID_DRAGONRISE, USB_DEVICE_ID_DRAGONRISE_PS3, HID_QUIRK_MULTI_INPUT },
*/
list_for_each_safe(cur, tmp, &channel->sc_list) {
cur_channel = list_entry(cur, struct vmbus_channel, sc_list);
- if (cur_channel->state != CHANNEL_OPENED_STATE)
- continue;
vmbus_close_internal(cur_channel);
+ if (cur_channel->rescind) {
+ mutex_lock(&vmbus_connection.channel_mutex);
+ hv_process_channel_removal(cur_channel,
+ cur_channel->offermsg.child_relid);
+ mutex_unlock(&vmbus_connection.channel_mutex);
+ }
}
/*
* Now close the primary.
{
struct vmbus_channel *channel, *tmp;
- mutex_lock(&vmbus_connection.channel_mutex);
list_for_each_entry_safe(channel, tmp, &vmbus_connection.chn_list,
listentry) {
/* hv_process_channel_removal() needs this */
vmbus_device_unregister(channel->device_obj);
}
- mutex_unlock(&vmbus_connection.channel_mutex);
}
/*
list_add_tail(&newchannel->sc_list, &channel->sc_list);
channel->num_sc++;
spin_unlock_irqrestore(&channel->lock, flags);
- } else
+ } else {
+ atomic_dec(&vmbus_connection.offer_in_progress);
goto err_free_chan;
+ }
}
dev_type = hv_get_dev_type(newchannel);
if (!fnew) {
if (channel->sc_creation_callback != NULL)
channel->sc_creation_callback(newchannel);
+ atomic_dec(&vmbus_connection.offer_in_progress);
return;
}
* binding which eventually invokes the device driver's AddDevice()
* method.
*/
- mutex_lock(&vmbus_connection.channel_mutex);
ret = vmbus_device_register(newchannel->device_obj);
- mutex_unlock(&vmbus_connection.channel_mutex);
if (ret != 0) {
pr_err("unable to add child device object (relid %d)\n",
kfree(newchannel->device_obj);
goto err_deq_chan;
}
+
+ atomic_dec(&vmbus_connection.offer_in_progress);
return;
err_deq_chan:
newchannel = alloc_channel();
if (!newchannel) {
vmbus_release_relid(offer->child_relid);
+ atomic_dec(&vmbus_connection.offer_in_progress);
pr_err("Unable to allocate channel object\n");
return;
}
rescind = (struct vmbus_channel_rescind_offer *)hdr;
+ /*
+ * The offer msg and the corresponding rescind msg
+ * from the host are guranteed to be ordered -
+ * offer comes in first and then the rescind.
+ * Since we process these events in work elements,
+ * and with preemption, we may end up processing
+ * the events out of order. Given that we handle these
+ * work elements on the same CPU, this is possible only
+ * in the case of preemption. In any case wait here
+ * until the offer processing has moved beyond the
+ * point where the channel is discoverable.
+ */
+
+ while (atomic_read(&vmbus_connection.offer_in_progress) != 0) {
+ /*
+ * We wait here until any channel offer is currently
+ * being processed.
+ */
+ msleep(1);
+ }
+
mutex_lock(&vmbus_connection.channel_mutex);
channel = relid2channel(rescind->child_relid);
+ mutex_unlock(&vmbus_connection.channel_mutex);
if (channel == NULL) {
/*
- * This is very impossible, because in
- * vmbus_process_offer(), we have already invoked
- * vmbus_release_relid() on error.
+ * We failed in processing the offer message;
+ * we would have cleaned up the relid in that
+ * failure path.
*/
- goto out;
+ return;
}
spin_lock_irqsave(&channel->lock, flags);
if (channel->device_obj) {
if (channel->chn_rescind_callback) {
channel->chn_rescind_callback(channel);
- goto out;
+ return;
}
/*
* We will have to unregister this device from the
vmbus_device_unregister(channel->device_obj);
put_device(dev);
}
- } else {
- hv_process_channel_removal(channel,
- channel->offermsg.child_relid);
}
-
-out:
- mutex_unlock(&vmbus_connection.channel_mutex);
+ if (channel->primary_channel != NULL) {
+ /*
+ * Sub-channel is being rescinded. Following is the channel
+ * close sequence when initiated from the driveri (refer to
+ * vmbus_close() for details):
+ * 1. Close all sub-channels first
+ * 2. Then close the primary channel.
+ */
+ if (channel->state == CHANNEL_OPEN_STATE) {
+ /*
+ * The channel is currently not open;
+ * it is safe for us to cleanup the channel.
+ */
+ mutex_lock(&vmbus_connection.channel_mutex);
+ hv_process_channel_removal(channel,
+ channel->offermsg.child_relid);
+ mutex_unlock(&vmbus_connection.channel_mutex);
+ }
+ }
}
void vmbus_hvsock_device_unregister(struct vmbus_channel *channel)
* all the CPUs. This is needed for kexec to work correctly where
* the CPU attempting to connect may not be CPU 0.
*/
- if (version >= VERSION_WIN8_1)
+ if (version >= VERSION_WIN8_1) {
msg->target_vcpu = hv_context.vp_index[smp_processor_id()];
- else
+ vmbus_connection.connect_cpu = smp_processor_id();
+ } else {
msg->target_vcpu = 0;
+ vmbus_connection.connect_cpu = 0;
+ }
/*
* Add to list before we send the request since we may
break;
case HV_STATUS_INSUFFICIENT_MEMORY:
case HV_STATUS_INSUFFICIENT_BUFFERS:
- ret = -ENOMEM;
+ ret = -ENOBUFS;
break;
case HV_STATUS_SUCCESS:
return ret;
else
mdelay(usec / 1000);
- if (usec < 256000)
+ if (retries < 22)
usec *= 2;
}
return ret;
aligned_msg->message_type = message_type;
aligned_msg->payload_size = payload_size;
memcpy((void *)aligned_msg->payload, payload, payload_size);
- put_cpu_ptr(hv_cpu);
status = hv_do_hypercall(HVCALL_POST_MESSAGE, aligned_msg, NULL);
+ /* Preemption must remain disabled until after the hypercall
+ * so some other thread can't get scheduled onto this cpu and
+ * corrupt the per-cpu post_msg_page
+ */
+ put_cpu_ptr(hv_cpu);
+
return status & 0xFFFF;
}
WARN_ON(!clockevent_state_oneshot(evt));
- hv_get_current_tick(current_tick);
+ current_tick = hyperv_cs->read(NULL);
current_tick += delta;
hv_init_timer(HV_X64_MSR_STIMER0_COUNT, current_tick);
return 0;
{
/* Transaction is finished, reset the state here to avoid races. */
kvp_transaction.state = HVUTIL_READY;
- hv_kvp_onchannelcallback(channel);
+ tasklet_schedule(&((struct vmbus_channel *)channel)->callback_event);
}
static void kvp_register_done(void)
static void kvp_host_handshake_func(struct work_struct *dummy)
{
- hv_poll_channel(kvp_transaction.recv_channel, hv_kvp_onchannelcallback);
+ tasklet_schedule(&kvp_transaction.recv_channel->callback_event);
}
static int kvp_handle_handshake(struct hv_kvp_msg *msg)
NEGO_IN_PROGRESS,
NEGO_FINISHED} host_negotiatied = NEGO_NOT_STARTED;
- if (host_negotiatied == NEGO_NOT_STARTED &&
- kvp_transaction.state < HVUTIL_READY) {
+ if (kvp_transaction.state < HVUTIL_READY) {
/*
* If userspace daemon is not connected and host is asking
* us to negotiate we need to delay to not lose messages.
* This is important for Failover IP setting.
*/
- host_negotiatied = NEGO_IN_PROGRESS;
- schedule_delayed_work(&kvp_host_handshake_work,
+ if (host_negotiatied == NEGO_NOT_STARTED) {
+ host_negotiatied = NEGO_IN_PROGRESS;
+ schedule_delayed_work(&kvp_host_handshake_work,
HV_UTIL_NEGO_TIMEOUT * HZ);
+ }
return;
}
if (kvp_transaction.state > HVUTIL_READY)
VM_PKT_DATA_INBAND, 0);
host_negotiatied = NEGO_FINISHED;
+ hv_poll_channel(kvp_transaction.recv_channel, kvp_poll_wrapper);
}
}
/*
* Set the host time in a process context.
*/
+static struct work_struct adj_time_work;
-struct adj_time_work {
- struct work_struct work;
- u64 host_time;
- u64 ref_time;
- u8 flags;
-};
+/*
+ * The last time sample, received from the host. PTP device responds to
+ * requests by using this data and the current partition-wide time reference
+ * count.
+ */
+static struct {
+ u64 host_time;
+ u64 ref_time;
+ spinlock_t lock;
+} host_ts;
-static void hv_set_host_time(struct work_struct *work)
+static struct timespec64 hv_get_adj_host_time(void)
{
- struct adj_time_work *wrk;
- struct timespec64 host_ts;
- u64 reftime, newtime;
-
- wrk = container_of(work, struct adj_time_work, work);
+ struct timespec64 ts;
+ u64 newtime, reftime;
+ unsigned long flags;
+ spin_lock_irqsave(&host_ts.lock, flags);
reftime = hyperv_cs->read(hyperv_cs);
- newtime = wrk->host_time + (reftime - wrk->ref_time);
- host_ts = ns_to_timespec64((newtime - WLTIMEDELTA) * 100);
+ newtime = host_ts.host_time + (reftime - host_ts.ref_time);
+ ts = ns_to_timespec64((newtime - WLTIMEDELTA) * 100);
+ spin_unlock_irqrestore(&host_ts.lock, flags);
- do_settimeofday64(&host_ts);
+ return ts;
+}
+
+static void hv_set_host_time(struct work_struct *work)
+{
+ struct timespec64 ts = hv_get_adj_host_time();
+
+ do_settimeofday64(&ts);
}
/*
* typically used as a hint to the guest. The guest is under no obligation
* to discipline the clock.
*/
-static struct adj_time_work wrk;
-
-/*
- * The last time sample, received from the host. PTP device responds to
- * requests by using this data and the current partition-wide time reference
- * count.
- */
-static struct {
- u64 host_time;
- u64 ref_time;
- struct system_time_snapshot snap;
- spinlock_t lock;
-} host_ts;
-
static inline void adj_guesttime(u64 hosttime, u64 reftime, u8 adj_flags)
{
unsigned long flags;
u64 cur_reftime;
/*
- * This check is safe since we are executing in the
- * interrupt context and time synch messages are always
- * delivered on the same CPU.
+ * Save the adjusted time sample from the host and the snapshot
+ * of the current system time.
*/
- if (adj_flags & ICTIMESYNCFLAG_SYNC) {
- /* Queue a job to do do_settimeofday64() */
- if (work_pending(&wrk.work))
- return;
-
- wrk.host_time = hosttime;
- wrk.ref_time = reftime;
- wrk.flags = adj_flags;
- schedule_work(&wrk.work);
- } else {
- /*
- * Save the adjusted time sample from the host and the snapshot
- * of the current system time for PTP device.
- */
- spin_lock_irqsave(&host_ts.lock, flags);
-
- cur_reftime = hyperv_cs->read(hyperv_cs);
- host_ts.host_time = hosttime;
- host_ts.ref_time = cur_reftime;
- ktime_get_snapshot(&host_ts.snap);
-
- /*
- * TimeSync v4 messages contain reference time (guest's Hyper-V
- * clocksource read when the time sample was generated), we can
- * improve the precision by adding the delta between now and the
- * time of generation.
- */
- if (ts_srv_version > TS_VERSION_3)
- host_ts.host_time += (cur_reftime - reftime);
-
- spin_unlock_irqrestore(&host_ts.lock, flags);
- }
+ spin_lock_irqsave(&host_ts.lock, flags);
+
+ cur_reftime = hyperv_cs->read(hyperv_cs);
+ host_ts.host_time = hosttime;
+ host_ts.ref_time = cur_reftime;
+
+ /*
+ * TimeSync v4 messages contain reference time (guest's Hyper-V
+ * clocksource read when the time sample was generated), we can
+ * improve the precision by adding the delta between now and the
+ * time of generation. For older protocols we set
+ * reftime == cur_reftime on call.
+ */
+ host_ts.host_time += (cur_reftime - reftime);
+
+ spin_unlock_irqrestore(&host_ts.lock, flags);
+
+ /* Schedule work to do do_settimeofday64() */
+ if (adj_flags & ICTIMESYNCFLAG_SYNC)
+ schedule_work(&adj_time_work);
}
/*
sizeof(struct vmbuspipe_hdr) +
sizeof(struct icmsg_hdr)];
adj_guesttime(timedatap->parenttime,
- 0,
- timedatap->flags);
+ hyperv_cs->read(hyperv_cs),
+ timedatap->flags);
}
}
static int hv_ptp_gettime(struct ptp_clock_info *info, struct timespec64 *ts)
{
- unsigned long flags;
- u64 newtime, reftime;
-
- spin_lock_irqsave(&host_ts.lock, flags);
- reftime = hyperv_cs->read(hyperv_cs);
- newtime = host_ts.host_time + (reftime - host_ts.ref_time);
- *ts = ns_to_timespec64((newtime - WLTIMEDELTA) * 100);
- spin_unlock_irqrestore(&host_ts.lock, flags);
+ *ts = hv_get_adj_host_time();
return 0;
}
-static int hv_ptp_get_syncdevicetime(ktime_t *device,
- struct system_counterval_t *system,
- void *ctx)
-{
- system->cs = hyperv_cs;
- system->cycles = host_ts.ref_time;
- *device = ns_to_ktime((host_ts.host_time - WLTIMEDELTA) * 100);
-
- return 0;
-}
-
-static int hv_ptp_getcrosststamp(struct ptp_clock_info *ptp,
- struct system_device_crosststamp *xtstamp)
-{
- unsigned long flags;
- int ret;
-
- spin_lock_irqsave(&host_ts.lock, flags);
-
- /*
- * host_ts contains the last time sample from the host and the snapshot
- * of system time. We don't need to calculate the time delta between
- * the reception and now as get_device_system_crosststamp() does the
- * required interpolation.
- */
- ret = get_device_system_crosststamp(hv_ptp_get_syncdevicetime,
- NULL, &host_ts.snap, xtstamp);
-
- spin_unlock_irqrestore(&host_ts.lock, flags);
-
- return ret;
-}
-
static struct ptp_clock_info ptp_hyperv_info = {
.name = "hyperv",
.enable = hv_ptp_enable,
.adjtime = hv_ptp_adjtime,
.adjfreq = hv_ptp_adjfreq,
.gettime64 = hv_ptp_gettime,
- .getcrosststamp = hv_ptp_getcrosststamp,
.settime64 = hv_ptp_settime,
.owner = THIS_MODULE,
};
spin_lock_init(&host_ts.lock);
- INIT_WORK(&wrk.work, hv_set_host_time);
+ INIT_WORK(&adj_time_work, hv_set_host_time);
/*
* ptp_clock_register() returns NULL when CONFIG_PTP_1588_CLOCK is
{
if (hv_ptp_clock)
ptp_clock_unregister(hv_ptp_clock);
- cancel_work_sync(&wrk.work);
+ cancel_work_sync(&adj_time_work);
}
static int __init init_hyperv_utils(void)
#define MAX_SIZE_CHANNEL_MESSAGE HV_MESSAGE_PAYLOAD_BYTE_COUNT
struct vmbus_connection {
+ /*
+ * CPU on which the initial host contact was made.
+ */
+ int connect_cpu;
+
+ atomic_t offer_in_progress;
+
enum vmbus_connect_state conn_state;
atomic_t next_gpadl_handle;
if (!channel)
return;
+ if (in_interrupt() && (channel->target_cpu == smp_processor_id())) {
+ cb(channel);
+ return;
+ }
smp_call_function_single(channel->target_cpu, cb, channel, true);
}
spin_unlock(&drv->dynids.lock);
}
-/* Parse string of form: 1b4e28ba-2fa1-11d2-883f-b9a761bde3f */
-static int get_uuid_le(const char *str, uuid_le *uu)
-{
- unsigned int b[16];
- int i;
-
- if (strlen(str) < 37)
- return -1;
-
- for (i = 0; i < 36; i++) {
- switch (i) {
- case 8: case 13: case 18: case 23:
- if (str[i] != '-')
- return -1;
- break;
- default:
- if (!isxdigit(str[i]))
- return -1;
- }
- }
-
- /* unparse little endian output byte order */
- if (sscanf(str,
- "%2x%2x%2x%2x-%2x%2x-%2x%2x-%2x%2x-%2x%2x%2x%2x%2x%2x",
- &b[3], &b[2], &b[1], &b[0],
- &b[5], &b[4], &b[7], &b[6], &b[8], &b[9],
- &b[10], &b[11], &b[12], &b[13], &b[14], &b[15]) != 16)
- return -1;
-
- for (i = 0; i < 16; i++)
- uu->b[i] = b[i];
- return 0;
-}
-
/*
* store_new_id - sysfs frontend to vmbus_add_dynid()
*
size_t count)
{
struct hv_driver *drv = drv_to_hv_drv(driver);
- uuid_le guid = NULL_UUID_LE;
+ uuid_le guid;
ssize_t retval;
- if (get_uuid_le(buf, &guid) != 0)
- return -EINVAL;
+ retval = uuid_le_to_bin(buf, &guid);
+ if (retval)
+ return retval;
if (hv_vmbus_get_id(drv, &guid))
return -EEXIST;
{
struct hv_driver *drv = drv_to_hv_drv(driver);
struct vmbus_dynid *dynid, *n;
- uuid_le guid = NULL_UUID_LE;
- size_t retval = -ENODEV;
+ uuid_le guid;
+ ssize_t retval;
- if (get_uuid_le(buf, &guid))
- return -EINVAL;
+ retval = uuid_le_to_bin(buf, &guid);
+ if (retval)
+ return retval;
+ retval = -ENODEV;
spin_lock(&drv->dynids.lock);
list_for_each_entry_safe(dynid, n, &drv->dynids.list, node) {
struct hv_vmbus_device_id *id = &dynid->id;
struct hv_device *hv_dev = device_to_hv_device(device);
struct vmbus_channel *channel = hv_dev->channel;
+ mutex_lock(&vmbus_connection.channel_mutex);
hv_process_channel_removal(channel,
channel->offermsg.child_relid);
+ mutex_unlock(&vmbus_connection.channel_mutex);
kfree(hv_dev);
}
INIT_WORK(&ctx->work, vmbus_onmessage_work);
memcpy(&ctx->msg, msg, sizeof(*msg));
- queue_work(vmbus_connection.work_queue, &ctx->work);
+ /*
+ * The host can generate a rescind message while we
+ * may still be handling the original offer. We deal with
+ * this condition by ensuring the processing is done on the
+ * same CPU.
+ */
+ switch (hdr->msgtype) {
+ case CHANNELMSG_RESCIND_CHANNELOFFER:
+ /*
+ * If we are handling the rescind message;
+ * schedule the work on the global work queue.
+ */
+ schedule_work_on(vmbus_connection.connect_cpu,
+ &ctx->work);
+ break;
+
+ case CHANNELMSG_OFFERCHANNEL:
+ atomic_inc(&vmbus_connection.offer_in_progress);
+ queue_work_on(vmbus_connection.connect_cpu,
+ vmbus_connection.work_queue,
+ &ctx->work);
+ break;
+
+ default:
+ queue_work(vmbus_connection.work_queue, &ctx->work);
+ }
} else
entry->message_handler(hdr);
#include <linux/of_device.h>
#include <linux/of.h>
-#include <linux/i2c/ads1015.h>
+#include <linux/platform_data/ads1015.h>
/* ADS1015 registers */
enum {
#include <linux/hwmon-vid.h>
#include <linux/err.h>
#include <linux/jiffies.h>
+#include <linux/util_macros.h>
/* Indexes for the sysfs hooks */
#define REG_TEMP_TRANGE_BASE 0x5F
+#define REG_ENHANCE_ACOUSTICS1 0x62
+#define REG_ENHANCE_ACOUSTICS2 0x63
+
#define REG_PWM_MIN_BASE 0x64
#define REG_TEMP_TMIN_BASE 0x67
u8 range[3];
u8 pwmctl[3];
u8 pwmchan[3];
+ u8 enh_acoustics[2];
u8 vid;
u8 vrm;
i2c_smbus_write_byte_data(client, reg, val & 0xFF);
}
-/*
- * Find the nearest value in a table - used for pwm frequency and
- * auto temp range
- */
-static int find_nearest(long val, const int *array, int size)
-{
- int i;
-
- if (val < array[0])
- return 0;
-
- if (val > array[size - 1])
- return size - 1;
-
- for (i = 0; i < size - 1; i++) {
- int a, b;
-
- if (val > array[i + 1])
- continue;
-
- a = val - array[i];
- b = array[i + 1] - val;
-
- return (a <= b) ? i : i + 1;
- }
-
- return 0;
-}
-
static ssize_t show_voltage(struct device *dev, struct device_attribute *attr,
char *buf)
{
return count;
}
+/* Assuming CONFIG6[SLOW] is 0 */
+static const int ad7475_st_map[] = {
+ 37500, 18800, 12500, 7500, 4700, 3100, 1600, 800,
+};
+
+static ssize_t show_temp_st(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct sensor_device_attribute_2 *sattr = to_sensor_dev_attr_2(attr);
+ struct i2c_client *client = to_i2c_client(dev);
+ struct adt7475_data *data = i2c_get_clientdata(client);
+ long val;
+
+ switch (sattr->index) {
+ case 0:
+ val = data->enh_acoustics[0] & 0xf;
+ break;
+ case 1:
+ val = (data->enh_acoustics[1] >> 4) & 0xf;
+ break;
+ case 2:
+ default:
+ val = data->enh_acoustics[1] & 0xf;
+ break;
+ }
+
+ if (val & 0x8)
+ return sprintf(buf, "%d\n", ad7475_st_map[val & 0x7]);
+ else
+ return sprintf(buf, "0\n");
+}
+
+static ssize_t set_temp_st(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct sensor_device_attribute_2 *sattr = to_sensor_dev_attr_2(attr);
+ struct i2c_client *client = to_i2c_client(dev);
+ struct adt7475_data *data = i2c_get_clientdata(client);
+ unsigned char reg;
+ int shift, idx;
+ ulong val;
+
+ if (kstrtoul(buf, 10, &val))
+ return -EINVAL;
+
+ switch (sattr->index) {
+ case 0:
+ reg = REG_ENHANCE_ACOUSTICS1;
+ shift = 0;
+ idx = 0;
+ break;
+ case 1:
+ reg = REG_ENHANCE_ACOUSTICS2;
+ shift = 0;
+ idx = 1;
+ break;
+ case 2:
+ default:
+ reg = REG_ENHANCE_ACOUSTICS2;
+ shift = 4;
+ idx = 1;
+ break;
+ }
+
+ if (val > 0) {
+ val = find_closest_descending(val, ad7475_st_map,
+ ARRAY_SIZE(ad7475_st_map));
+ val |= 0x8;
+ }
+
+ mutex_lock(&data->lock);
+
+ data->enh_acoustics[idx] &= ~(0xf << shift);
+ data->enh_acoustics[idx] |= (val << shift);
+
+ i2c_smbus_write_byte_data(client, reg, data->enh_acoustics[idx]);
+
+ mutex_unlock(&data->lock);
+
+ return count;
+}
+
/*
* Table of autorange values - the user will write the value in millidegrees,
* and we'll convert it
val -= temp;
/* Find the nearest table entry to what the user wrote */
- val = find_nearest(val, autorange_table, ARRAY_SIZE(autorange_table));
+ val = find_closest(val, autorange_table, ARRAY_SIZE(autorange_table));
data->range[sattr->index] &= ~0xF0;
data->range[sattr->index] |= val << 4;
data->pwm[sattr->nr][sattr->index] = clamp_val(val, 0, 0xFF);
i2c_smbus_write_byte_data(client, reg,
data->pwm[sattr->nr][sattr->index]);
+ mutex_unlock(&data->lock);
+
+ return count;
+}
+
+static ssize_t show_stall_disable(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct sensor_device_attribute_2 *sattr = to_sensor_dev_attr_2(attr);
+ struct i2c_client *client = to_i2c_client(dev);
+ struct adt7475_data *data = i2c_get_clientdata(client);
+ u8 mask = BIT(5 + sattr->index);
+
+ return sprintf(buf, "%d\n", !!(data->enh_acoustics[0] & mask));
+}
+
+static ssize_t set_stall_disable(struct device *dev,
+ struct device_attribute *attr, const char *buf,
+ size_t count)
+{
+ struct sensor_device_attribute_2 *sattr = to_sensor_dev_attr_2(attr);
+ struct i2c_client *client = to_i2c_client(dev);
+ struct adt7475_data *data = i2c_get_clientdata(client);
+ long val;
+ u8 mask = BIT(5 + sattr->index);
+
+ if (kstrtol(buf, 10, &val))
+ return -EINVAL;
+
+ mutex_lock(&data->lock);
+
+ data->enh_acoustics[0] &= ~mask;
+ if (val)
+ data->enh_acoustics[0] |= mask;
+
+ i2c_smbus_write_byte_data(client, REG_ENHANCE_ACOUSTICS1,
+ data->enh_acoustics[0]);
mutex_unlock(&data->lock);
/* List of frequencies for the PWM */
static const int pwmfreq_table[] = {
- 11, 14, 22, 29, 35, 44, 58, 88
+ 11, 14, 22, 29, 35, 44, 58, 88, 22500
};
static ssize_t show_pwmfreq(struct device *dev, struct device_attribute *attr,
{
struct adt7475_data *data = adt7475_update_device(dev);
struct sensor_device_attribute_2 *sattr = to_sensor_dev_attr_2(attr);
+ int i = clamp_val(data->range[sattr->index] & 0xf, 0,
+ ARRAY_SIZE(pwmfreq_table) - 1);
- return sprintf(buf, "%d\n",
- pwmfreq_table[data->range[sattr->index] & 7]);
+ return sprintf(buf, "%d\n", pwmfreq_table[i]);
}
static ssize_t set_pwmfreq(struct device *dev, struct device_attribute *attr,
if (kstrtol(buf, 10, &val))
return -EINVAL;
- out = find_nearest(val, pwmfreq_table, ARRAY_SIZE(pwmfreq_table));
+ out = find_closest(val, pwmfreq_table, ARRAY_SIZE(pwmfreq_table));
mutex_lock(&data->lock);
data->range[sattr->index] =
adt7475_read(TEMP_TRANGE_REG(sattr->index));
- data->range[sattr->index] &= ~7;
+ data->range[sattr->index] &= ~0xf;
data->range[sattr->index] |= out;
i2c_smbus_write_byte_data(client, TEMP_TRANGE_REG(sattr->index),
THERM, 0);
static SENSOR_DEVICE_ATTR_2(temp1_crit_hyst, S_IRUGO | S_IWUSR, show_temp,
set_temp, HYSTERSIS, 0);
+static SENSOR_DEVICE_ATTR_2(temp1_smoothing, S_IRUGO | S_IWUSR, show_temp_st,
+ set_temp_st, 0, 0);
static SENSOR_DEVICE_ATTR_2(temp2_input, S_IRUGO, show_temp, NULL, INPUT, 1);
static SENSOR_DEVICE_ATTR_2(temp2_alarm, S_IRUGO, show_temp, NULL, ALARM, 1);
static SENSOR_DEVICE_ATTR_2(temp2_max, S_IRUGO | S_IWUSR, show_temp, set_temp,
THERM, 1);
static SENSOR_DEVICE_ATTR_2(temp2_crit_hyst, S_IRUGO | S_IWUSR, show_temp,
set_temp, HYSTERSIS, 1);
+static SENSOR_DEVICE_ATTR_2(temp2_smoothing, S_IRUGO | S_IWUSR, show_temp_st,
+ set_temp_st, 0, 1);
static SENSOR_DEVICE_ATTR_2(temp3_input, S_IRUGO, show_temp, NULL, INPUT, 2);
static SENSOR_DEVICE_ATTR_2(temp3_alarm, S_IRUGO, show_temp, NULL, ALARM, 2);
static SENSOR_DEVICE_ATTR_2(temp3_fault, S_IRUGO, show_temp, NULL, FAULT, 2);
THERM, 2);
static SENSOR_DEVICE_ATTR_2(temp3_crit_hyst, S_IRUGO | S_IWUSR, show_temp,
set_temp, HYSTERSIS, 2);
+static SENSOR_DEVICE_ATTR_2(temp3_smoothing, S_IRUGO | S_IWUSR, show_temp_st,
+ set_temp_st, 0, 2);
static SENSOR_DEVICE_ATTR_2(fan1_input, S_IRUGO, show_tach, NULL, INPUT, 0);
static SENSOR_DEVICE_ATTR_2(fan1_min, S_IRUGO | S_IWUSR, show_tach, set_tach,
MIN, 0);
set_pwm, MIN, 0);
static SENSOR_DEVICE_ATTR_2(pwm1_auto_point2_pwm, S_IRUGO | S_IWUSR, show_pwm,
set_pwm, MAX, 0);
+static SENSOR_DEVICE_ATTR_2(pwm1_stall_disable, S_IRUGO | S_IWUSR,
+ show_stall_disable, set_stall_disable, 0, 0);
static SENSOR_DEVICE_ATTR_2(pwm2, S_IRUGO | S_IWUSR, show_pwm, set_pwm, INPUT,
1);
static SENSOR_DEVICE_ATTR_2(pwm2_freq, S_IRUGO | S_IWUSR, show_pwmfreq,
set_pwm, MIN, 1);
static SENSOR_DEVICE_ATTR_2(pwm2_auto_point2_pwm, S_IRUGO | S_IWUSR, show_pwm,
set_pwm, MAX, 1);
+static SENSOR_DEVICE_ATTR_2(pwm2_stall_disable, S_IRUGO | S_IWUSR,
+ show_stall_disable, set_stall_disable, 0, 1);
static SENSOR_DEVICE_ATTR_2(pwm3, S_IRUGO | S_IWUSR, show_pwm, set_pwm, INPUT,
2);
static SENSOR_DEVICE_ATTR_2(pwm3_freq, S_IRUGO | S_IWUSR, show_pwmfreq,
set_pwm, MIN, 2);
static SENSOR_DEVICE_ATTR_2(pwm3_auto_point2_pwm, S_IRUGO | S_IWUSR, show_pwm,
set_pwm, MAX, 2);
+static SENSOR_DEVICE_ATTR_2(pwm3_stall_disable, S_IRUGO | S_IWUSR,
+ show_stall_disable, set_stall_disable, 0, 2);
/* Non-standard name, might need revisiting */
static DEVICE_ATTR_RW(pwm_use_point2_pwm_at_crit);
&sensor_dev_attr_temp1_auto_point2_temp.dev_attr.attr,
&sensor_dev_attr_temp1_crit.dev_attr.attr,
&sensor_dev_attr_temp1_crit_hyst.dev_attr.attr,
+ &sensor_dev_attr_temp1_smoothing.dev_attr.attr,
&sensor_dev_attr_temp2_input.dev_attr.attr,
&sensor_dev_attr_temp2_alarm.dev_attr.attr,
&sensor_dev_attr_temp2_max.dev_attr.attr,
&sensor_dev_attr_temp2_auto_point2_temp.dev_attr.attr,
&sensor_dev_attr_temp2_crit.dev_attr.attr,
&sensor_dev_attr_temp2_crit_hyst.dev_attr.attr,
+ &sensor_dev_attr_temp2_smoothing.dev_attr.attr,
&sensor_dev_attr_temp3_input.dev_attr.attr,
&sensor_dev_attr_temp3_fault.dev_attr.attr,
&sensor_dev_attr_temp3_alarm.dev_attr.attr,
&sensor_dev_attr_temp3_auto_point2_temp.dev_attr.attr,
&sensor_dev_attr_temp3_crit.dev_attr.attr,
&sensor_dev_attr_temp3_crit_hyst.dev_attr.attr,
+ &sensor_dev_attr_temp3_smoothing.dev_attr.attr,
&sensor_dev_attr_fan1_input.dev_attr.attr,
&sensor_dev_attr_fan1_min.dev_attr.attr,
&sensor_dev_attr_fan1_alarm.dev_attr.attr,
&sensor_dev_attr_pwm1_auto_channels_temp.dev_attr.attr,
&sensor_dev_attr_pwm1_auto_point1_pwm.dev_attr.attr,
&sensor_dev_attr_pwm1_auto_point2_pwm.dev_attr.attr,
+ &sensor_dev_attr_pwm1_stall_disable.dev_attr.attr,
&sensor_dev_attr_pwm3.dev_attr.attr,
&sensor_dev_attr_pwm3_freq.dev_attr.attr,
&sensor_dev_attr_pwm3_enable.dev_attr.attr,
&sensor_dev_attr_pwm3_auto_channels_temp.dev_attr.attr,
&sensor_dev_attr_pwm3_auto_point1_pwm.dev_attr.attr,
&sensor_dev_attr_pwm3_auto_point2_pwm.dev_attr.attr,
+ &sensor_dev_attr_pwm3_stall_disable.dev_attr.attr,
&dev_attr_pwm_use_point2_pwm_at_crit.attr,
NULL,
};
&sensor_dev_attr_pwm2_auto_channels_temp.dev_attr.attr,
&sensor_dev_attr_pwm2_auto_point1_pwm.dev_attr.attr,
&sensor_dev_attr_pwm2_auto_point2_pwm.dev_attr.attr,
+ &sensor_dev_attr_pwm2_stall_disable.dev_attr.attr,
NULL
};
#define PWM_MAX 255
+#define BOTH_EDGES 0x02 /* 10b */
+
#define M_PWM_DIV_H 0x00
#define M_PWM_DIV_L 0x05
#define M_PWM_PERIOD 0x5F
#define M_TACH_CLK_DIV 0x00
-#define M_TACH_MODE 0x00
-#define M_TACH_UNIT 0x1000
+/*
+ * 5:4 Type N fan tach mode selection bit:
+ * 00: falling
+ * 01: rising
+ * 10: both
+ * 11: reserved.
+ */
+#define M_TACH_MODE 0x02 /* 10b */
+#define M_TACH_UNIT 0x00c0
#define INIT_FAN_CTRL 0xFF
+/* How long we sleep in us while waiting for an RPM result. */
+#define ASPEED_RPM_STATUS_SLEEP_USEC 500
+
struct aspeed_pwm_tacho_data {
struct regmap *regmap;
unsigned long clk_freq;
u8 type_pwm_clock_division_h[3];
u8 type_pwm_clock_division_l[3];
u8 type_fan_tach_clock_division[3];
+ u8 type_fan_tach_mode[3];
u16 type_fan_tach_unit[3];
u8 pwm_port_type[8];
u8 pwm_port_fan_ctrl[8];
static int aspeed_get_fan_tach_ch_rpm(struct aspeed_pwm_tacho_data *priv,
u8 fan_tach_ch)
{
- u32 raw_data, tach_div, clk_source, sec, val;
- u8 fan_tach_ch_source, type;
+ u32 raw_data, tach_div, clk_source, msec, usec, val;
+ u8 fan_tach_ch_source, type, mode, both;
+ int ret;
regmap_write(priv->regmap, ASPEED_PTCR_TRIGGER, 0);
regmap_write(priv->regmap, ASPEED_PTCR_TRIGGER, 0x1 << fan_tach_ch);
fan_tach_ch_source = priv->fan_tach_ch_source[fan_tach_ch];
type = priv->pwm_port_type[fan_tach_ch_source];
- sec = (1000 / aspeed_get_fan_tach_ch_measure_period(priv, type));
- msleep(sec);
+ msec = (1000 / aspeed_get_fan_tach_ch_measure_period(priv, type));
+ usec = msec * 1000;
+
+ ret = regmap_read_poll_timeout(
+ priv->regmap,
+ ASPEED_PTCR_RESULT,
+ val,
+ (val & RESULT_STATUS_MASK),
+ ASPEED_RPM_STATUS_SLEEP_USEC,
+ usec);
- regmap_read(priv->regmap, ASPEED_PTCR_RESULT, &val);
- if (!(val & RESULT_STATUS_MASK))
- return -ETIMEDOUT;
+ /* return -ETIMEDOUT if we didn't get an answer. */
+ if (ret)
+ return ret;
raw_data = val & RESULT_VALUE_MASK;
tach_div = priv->type_fan_tach_clock_division[type];
- tach_div = 0x4 << (tach_div * 2);
+ /*
+ * We need the mode to determine if the raw_data is double (from
+ * counting both edges).
+ */
+ mode = priv->type_fan_tach_mode[type];
+ both = (mode & BOTH_EDGES) ? 1 : 0;
+
+ tach_div = (0x4 << both) << (tach_div * 2);
clk_source = priv->clk_freq;
if (raw_data == 0)
aspeed_set_tacho_type_enable(priv->regmap, TYPEM, true);
priv->type_fan_tach_clock_division[TYPEM] = M_TACH_CLK_DIV;
priv->type_fan_tach_unit[TYPEM] = M_TACH_UNIT;
+ priv->type_fan_tach_mode[TYPEM] = M_TACH_MODE;
aspeed_set_tacho_type_values(priv->regmap, TYPEM, M_TACH_MODE,
M_TACH_UNIT, M_TACH_CLK_DIV);
}
#include <linux/err.h>
#include <linux/mutex.h>
#include <linux/sysfs.h>
-#include <linux/i2c/ds620.h>
+#include <linux/platform_data/ds620.h>
/*
* Many DS620 constants specified below
TEMP,
POWER_SUPPLY,
POWER_INPUT,
+ CURRENT,
MAX_SENSOR_TYPE,
};
#define INVALID_INDEX (-1U)
+/*
+ * 'compatible' string properties for sensor types as defined in old
+ * PowerNV firmware (skiboot). These are ordered as 'enum sensors'.
+ */
+static const char * const legacy_compatibles[] = {
+ "ibm,opal-sensor-cooling-fan",
+ "ibm,opal-sensor-amb-temp",
+ "ibm,opal-sensor-power-supply",
+ "ibm,opal-sensor-power"
+};
+
static struct sensor_group {
- const char *name;
- const char *compatible;
+ const char *name; /* matches property 'sensor-type' */
struct attribute_group group;
u32 attr_count;
u32 hwmon_index;
} sensor_groups[] = {
- {"fan", "ibm,opal-sensor-cooling-fan"},
- {"temp", "ibm,opal-sensor-amb-temp"},
- {"in", "ibm,opal-sensor-power-supply"},
- {"power", "ibm,opal-sensor-power"}
+ { "fan" },
+ { "temp" },
+ { "in" },
+ { "power" },
+ { "curr" },
};
struct sensor_data {
enum sensors type;
const char *str;
- for (type = 0; type < MAX_SENSOR_TYPE; type++) {
- if (of_device_is_compatible(np, sensor_groups[type].compatible))
+ for (type = 0; type < ARRAY_SIZE(legacy_compatibles); type++) {
+ if (of_device_is_compatible(np, legacy_compatibles[type]))
return type;
}
sensor_groups[type].attr_count++;
/*
- * add a new attribute for labels
+ * add attributes for labels, min and max
*/
if (!of_property_read_string(np, "label", &label))
sensor_groups[type].attr_count++;
+ if (of_find_property(np, "sensor-data-min", NULL))
+ sensor_groups[type].attr_count++;
+ if (of_find_property(np, "sensor-data-max", NULL))
+ sensor_groups[type].attr_count++;
}
of_node_put(opal);
sdata->dev_attr.show = show;
}
+static void populate_sensor(struct sensor_data *sdata, int od, int hd, int sid,
+ const char *attr_name, enum sensors type,
+ const struct attribute_group *pgroup,
+ ssize_t (*show)(struct device *dev,
+ struct device_attribute *attr,
+ char *buf))
+{
+ sdata->id = sid;
+ sdata->type = type;
+ sdata->opal_index = od;
+ sdata->hwmon_index = hd;
+ create_hwmon_attr(sdata, attr_name, show);
+ pgroup->attrs[sensor_groups[type].attr_count++] = &sdata->dev_attr.attr;
+}
+
+static char *get_max_attr(enum sensors type)
+{
+ switch (type) {
+ case POWER_INPUT:
+ return "input_highest";
+ default:
+ return "highest";
+ }
+}
+
+static char *get_min_attr(enum sensors type)
+{
+ switch (type) {
+ case POWER_INPUT:
+ return "input_lowest";
+ default:
+ return "lowest";
+ }
+}
+
/*
* Iterate through the device tree for each child of 'sensors' node, create
* a sysfs attribute file, the file is named by translating the DT node name
* attribute. They are related to the same
* sensor.
*/
- sdata[count].type = type;
- sdata[count].opal_index = sdata[count - 1].opal_index;
- sdata[count].hwmon_index = sdata[count - 1].hwmon_index;
make_sensor_label(np, &sdata[count], label);
+ populate_sensor(&sdata[count], opal_index,
+ sdata[count - 1].hwmon_index,
+ sensor_id, "label", type, pgroups[type],
+ show_label);
+ count++;
+ }
- create_hwmon_attr(&sdata[count], "label", show_label);
+ if (!of_property_read_u32(np, "sensor-data-max", &sensor_id)) {
+ attr_name = get_max_attr(type);
+ populate_sensor(&sdata[count], opal_index,
+ sdata[count - 1].hwmon_index,
+ sensor_id, attr_name, type,
+ pgroups[type], show_sensor);
+ count++;
+ }
- pgroups[type]->attrs[sensor_groups[type].attr_count++] =
- &sdata[count++].dev_attr.attr;
+ if (!of_property_read_u32(np, "sensor-data-min", &sensor_id)) {
+ attr_name = get_min_attr(type);
+ populate_sensor(&sdata[count], opal_index,
+ sdata[count - 1].hwmon_index,
+ sensor_id, attr_name, type,
+ pgroups[type], show_sensor);
+ count++;
}
}
#include <linux/hwmon.h>
#include <linux/hwmon-sysfs.h>
#include <linux/jiffies.h>
-#include <linux/i2c/ltc4245.h>
+#include <linux/platform_data/ltc4245.h>
/* Here are names of the chip's registers (a.k.a. commands) */
enum ltc4245_cmd {
#include <linux/hwmon-sysfs.h>
#include <linux/err.h>
#include <linux/mutex.h>
-#include <linux/i2c/max6639.h>
+#include <linux/platform_data/max6639.h>
/* Addresses to scan */
static const unsigned short normal_i2c[] = { 0x2c, 0x2e, 0x2f, I2C_CLIENT_END };
* nct6791d 15 6 6 2+6 0xc800 0xc1 0x5ca3
* nct6792d 15 6 6 2+6 0xc910 0xc1 0x5ca3
* nct6793d 15 6 6 2+6 0xd120 0xc1 0x5ca3
+ * nct6795d 14 6 6 2+6 0xd350 0xc1 0x5ca3
+ *
*
* #temp lists the number of monitored temperature sources (first value) plus
* the number of directly connectable temperature sensors (second value).
#include <linux/err.h>
#include <linux/mutex.h>
#include <linux/acpi.h>
+#include <linux/bitops.h>
#include <linux/dmi.h>
#include <linux/io.h>
#include "lm75.h"
#define USE_ALTERNATE
-enum kinds { nct6106, nct6775, nct6776, nct6779, nct6791, nct6792, nct6793 };
+enum kinds { nct6106, nct6775, nct6776, nct6779, nct6791, nct6792, nct6793,
+ nct6795 };
/* used to set data->name = nct6775_device_names[data->sio_kind] */
static const char * const nct6775_device_names[] = {
"nct6791",
"nct6792",
"nct6793",
+ "nct6795",
};
static const char * const nct6775_sio_names[] __initconst = {
"NCT6791D",
"NCT6792D",
"NCT6793D",
+ "NCT6795D",
};
static unsigned short force_id;
#define NCT6775_LD_ACPI 0x0a
#define NCT6775_LD_HWM 0x0b
#define NCT6775_LD_VID 0x0d
+#define NCT6775_LD_12 0x12
#define SIO_REG_LDSEL 0x07 /* Logical device select */
#define SIO_REG_DEVID 0x20 /* Device ID (2 bytes) */
#define SIO_NCT6791_ID 0xc800
#define SIO_NCT6792_ID 0xc910
#define SIO_NCT6793_ID 0xd120
+#define SIO_NCT6795_ID 0xd350
#define SIO_ID_MASK 0xFFF0
enum pwm_enable { off, manual, thermal_cruise, speed_cruise, sf3, sf4 };
"PCH_DIM3_TEMP"
};
-static const u16 NCT6775_REG_TEMP_ALTERNATE[ARRAY_SIZE(nct6775_temp_label) - 1]
- = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x661, 0x662, 0x664 };
+#define NCT6775_TEMP_MASK 0x001ffffe
+
+static const u16 NCT6775_REG_TEMP_ALTERNATE[32] = {
+ [13] = 0x661,
+ [14] = 0x662,
+ [15] = 0x664,
+};
-static const u16 NCT6775_REG_TEMP_CRIT[ARRAY_SIZE(nct6775_temp_label) - 1]
- = { 0, 0, 0, 0, 0xa00, 0xa01, 0xa02, 0xa03, 0xa04, 0xa05, 0xa06,
- 0xa07 };
+static const u16 NCT6775_REG_TEMP_CRIT[32] = {
+ [4] = 0xa00,
+ [5] = 0xa01,
+ [6] = 0xa02,
+ [7] = 0xa03,
+ [8] = 0xa04,
+ [9] = 0xa05,
+ [10] = 0xa06,
+ [11] = 0xa07
+};
/* NCT6776 specific data */
"BYTE_TEMP"
};
-static const u16 NCT6776_REG_TEMP_ALTERNATE[ARRAY_SIZE(nct6776_temp_label) - 1]
- = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x401, 0x402, 0x404 };
+#define NCT6776_TEMP_MASK 0x007ffffe
-static const u16 NCT6776_REG_TEMP_CRIT[ARRAY_SIZE(nct6776_temp_label) - 1]
- = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x709, 0x70a };
+static const u16 NCT6776_REG_TEMP_ALTERNATE[32] = {
+ [14] = 0x401,
+ [15] = 0x402,
+ [16] = 0x404,
+};
+
+static const u16 NCT6776_REG_TEMP_CRIT[32] = {
+ [11] = 0x709,
+ [12] = 0x70a,
+};
/* NCT6779 specific data */
"Virtual_TEMP"
};
-#define NCT6779_NUM_LABELS (ARRAY_SIZE(nct6779_temp_label) - 5)
-#define NCT6791_NUM_LABELS ARRAY_SIZE(nct6779_temp_label)
+#define NCT6779_TEMP_MASK 0x07ffff7e
+#define NCT6791_TEMP_MASK 0x87ffff7e
-static const u16 NCT6779_REG_TEMP_ALTERNATE[NCT6791_NUM_LABELS - 1]
+static const u16 NCT6779_REG_TEMP_ALTERNATE[32]
= { 0x490, 0x491, 0x492, 0x493, 0x494, 0x495, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0,
0, 0x400, 0x401, 0x402, 0x404, 0x405, 0x406, 0x407,
0x408, 0 };
-static const u16 NCT6779_REG_TEMP_CRIT[NCT6791_NUM_LABELS - 1]
- = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x709, 0x70a };
+static const u16 NCT6779_REG_TEMP_CRIT[32] = {
+ [15] = 0x709,
+ [16] = 0x70a,
+};
/* NCT6791 specific data */
"Virtual_TEMP"
};
+#define NCT6792_TEMP_MASK 0x9fffff7e
+
static const char *const nct6793_temp_label[] = {
"",
"SYSTIN",
"Virtual_TEMP"
};
+#define NCT6793_TEMP_MASK 0xbfff037e
+
+static const char *const nct6795_temp_label[] = {
+ "",
+ "SYSTIN",
+ "CPUTIN",
+ "AUXTIN0",
+ "AUXTIN1",
+ "AUXTIN2",
+ "AUXTIN3",
+ "",
+ "SMBUSMASTER 0",
+ "SMBUSMASTER 1",
+ "SMBUSMASTER 2",
+ "SMBUSMASTER 3",
+ "SMBUSMASTER 4",
+ "SMBUSMASTER 5",
+ "SMBUSMASTER 6",
+ "SMBUSMASTER 7",
+ "PECI Agent 0",
+ "PECI Agent 1",
+ "PCH_CHIP_CPU_MAX_TEMP",
+ "PCH_CHIP_TEMP",
+ "PCH_CPU_TEMP",
+ "PCH_MCH_TEMP",
+ "PCH_DIM0_TEMP",
+ "PCH_DIM1_TEMP",
+ "PCH_DIM2_TEMP",
+ "PCH_DIM3_TEMP",
+ "BYTE_TEMP0",
+ "BYTE_TEMP1",
+ "PECI Agent 0 Calibration",
+ "PECI Agent 1 Calibration",
+ "",
+ "Virtual_TEMP"
+};
+
+#define NCT6795_TEMP_MASK 0xbfffff7e
+
/* NCT6102D/NCT6106D specific data */
#define NCT6106_REG_VBAT 0x318
34, -1 /* intrusion0, intrusion1 */
};
-static const u16 NCT6106_REG_TEMP_ALTERNATE[ARRAY_SIZE(nct6776_temp_label) - 1]
- = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x51, 0x52, 0x54 };
+static const u16 NCT6106_REG_TEMP_ALTERNATE[32] = {
+ [14] = 0x51,
+ [15] = 0x52,
+ [16] = 0x54,
+};
-static const u16 NCT6106_REG_TEMP_CRIT[ARRAY_SIZE(nct6776_temp_label) - 1]
- = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x204, 0x205 };
+static const u16 NCT6106_REG_TEMP_CRIT[32] = {
+ [11] = 0x204,
+ [12] = 0x205,
+};
static enum pwm_enable reg_to_pwm_enable(int pwm, int mode)
{
static inline unsigned int
div_from_reg(u8 reg)
{
- return 1 << reg;
+ return BIT(reg);
}
/*
u8 temp_src[NUM_TEMP];
u16 reg_temp_config[NUM_TEMP];
const char * const *temp_label;
- int temp_label_num;
+ u32 temp_mask;
u16 REG_CONFIG;
u16 REG_VBAT;
case nct6791:
case nct6792:
case nct6793:
+ case nct6795:
return reg == 0x150 || reg == 0x153 || reg == 0x155 ||
((reg & 0xfff0) == 0x4b0 && (reg & 0x000f) < 0x0b) ||
reg == 0x402 ||
data->fan_div[1] = (i & 0x70) >> 4;
i = nct6775_read_value(data, NCT6775_REG_FANDIV2);
data->fan_div[2] = i & 0x7;
- if (data->has_fan & (1 << 3))
+ if (data->has_fan & BIT(3))
data->fan_div[3] = (i & 0x70) >> 4;
}
* We'll compute a better divider later on.
*/
for (i = 0; i < ARRAY_SIZE(data->fan_div); i++) {
- if (!(data->has_fan & (1 << i)))
+ if (!(data->has_fan & BIT(i)))
continue;
if (data->fan_div[i] == 0) {
data->fan_div[i] = 7;
* prevents the unnecessary warning when fanX_min is reported as 0.
*/
for (i = 0; i < ARRAY_SIZE(data->fan_min); i++) {
- if (data->has_fan_min & (1 << i)) {
+ if (data->has_fan_min & BIT(i)) {
reg = nct6775_read_value(data, data->REG_FAN_MIN[i]);
if (!reg)
nct6775_write_value(data, data->REG_FAN_MIN[i],
div_from_reg(fan_div));
/* Preserve min limit if possible */
- if (data->has_fan_min & (1 << nr)) {
+ if (data->has_fan_min & BIT(nr)) {
fan_min = data->fan_min[nr];
if (fan_div > data->fan_div[nr]) {
if (fan_min != 255 && fan_min > 1)
bool duty_is_dc;
for (i = 0; i < data->pwm_num; i++) {
- if (!(data->has_pwm & (1 << i)))
+ if (!(data->has_pwm & BIT(i)))
continue;
duty_is_dc = data->REG_PWM_MODE[i] &&
u16 reg_t;
for (i = 0; i < data->pwm_num; i++) {
- if (!(data->has_pwm & (1 << i)))
+ if (!(data->has_pwm & BIT(i)))
continue;
for (j = 0; j < ARRAY_SIZE(data->fan_time); j++) {
case nct6791:
case nct6792:
case nct6793:
+ case nct6795:
reg = nct6775_read_value(data,
data->REG_CRITICAL_PWM_ENABLE[i]);
if (reg & data->CRITICAL_PWM_ENABLE_MASK)
/* Measured voltages and limits */
for (i = 0; i < data->in_num; i++) {
- if (!(data->have_in & (1 << i)))
+ if (!(data->have_in & BIT(i)))
continue;
data->in[i][0] = nct6775_read_value(data,
for (i = 0; i < ARRAY_SIZE(data->rpm); i++) {
u16 reg;
- if (!(data->has_fan & (1 << i)))
+ if (!(data->has_fan & BIT(i)))
continue;
reg = nct6775_read_value(data, data->REG_FAN[i]);
data->rpm[i] = data->fan_from_reg(reg,
data->fan_div[i]);
- if (data->has_fan_min & (1 << i))
+ if (data->has_fan_min & BIT(i))
data->fan_min[i] = nct6775_read_value(data,
data->REG_FAN_MIN[i]);
data->fan_pulses[i] =
/* Measured temperatures and limits */
for (i = 0; i < NUM_TEMP; i++) {
- if (!(data->have_temp & (1 << i)))
+ if (!(data->have_temp & BIT(i)))
continue;
for (j = 0; j < ARRAY_SIZE(data->reg_temp); j++) {
if (data->reg_temp[j][i])
data->reg_temp[j][i]);
}
if (i >= NUM_TEMP_FIXED ||
- !(data->have_temp_fixed & (1 << i)))
+ !(data->have_temp_fixed & BIT(i)))
continue;
data->temp_offset[i]
= nct6775_read_value(data, data->REG_TEMP_OFFSET[i]);
struct nct6775_data *data = dev_get_drvdata(dev);
int in = index / 5; /* voltage index */
- if (!(data->have_in & (1 << in)))
+ if (!(data->have_in & BIT(in)))
return 0;
return attr->mode;
* even with the highest divider (128)
*/
data->fan_min[nr] = 254;
- new_div = 7; /* 128 == (1 << 7) */
+ new_div = 7; /* 128 == BIT(7) */
dev_warn(dev,
"fan%u low limit %lu below minimum %u, set to minimum\n",
nr + 1, val, data->fan_from_reg_min(254, 7));
* even with the lowest divider (1)
*/
data->fan_min[nr] = 1;
- new_div = 0; /* 1 == (1 << 0) */
+ new_div = 0; /* 1 == BIT(0) */
dev_warn(dev,
"fan%u low limit %lu above maximum %u, set to maximum\n",
nr + 1, val, data->fan_from_reg_min(1, 0));
int fan = index / 6; /* fan index */
int nr = index % 6; /* attribute index */
- if (!(data->has_fan & (1 << fan)))
+ if (!(data->has_fan & BIT(fan)))
return 0;
if (nr == 1 && data->ALARM_BITS[FAN_ALARM_BASE + fan] == -1)
return 0;
if (nr == 2 && data->BEEP_BITS[FAN_ALARM_BASE + fan] == -1)
return 0;
- if (nr == 4 && !(data->has_fan_min & (1 << fan)))
+ if (nr == 4 && !(data->has_fan_min & BIT(fan)))
return 0;
if (nr == 5 && data->kind != nct6775)
return 0;
int temp = index / 10; /* temp index */
int nr = index % 10; /* attribute index */
- if (!(data->have_temp & (1 << temp)))
+ if (!(data->have_temp & BIT(temp)))
+ return 0;
+
+ if (nr == 1 && !data->temp_label)
return 0;
if (nr == 2 && find_temp_source(data, temp, data->num_temp_alarms) < 0)
return 0;
/* offset and type only apply to fixed sensors */
- if (nr > 7 && !(data->have_temp_fixed & (1 << temp)))
+ if (nr > 7 && !(data->have_temp_fixed & BIT(temp)))
return 0;
return attr->mode;
int i, sel = 0;
for (i = 0; i < NUM_TEMP; i++) {
- if (!(data->have_temp & (1 << i)))
+ if (!(data->have_temp & BIT(i)))
continue;
if (src == data->temp_src[i]) {
sel = i + 1;
return err;
if (val == 0 || val > NUM_TEMP)
return -EINVAL;
- if (!(data->have_temp & (1 << (val - 1))) || !data->temp_src[val - 1])
+ if (!(data->have_temp & BIT(val - 1)) || !data->temp_src[val - 1])
return -EINVAL;
mutex_lock(&data->update_lock);
return err;
if (val > NUM_TEMP)
return -EINVAL;
- if (val && (!(data->have_temp & (1 << (val - 1))) ||
+ if (val && (!(data->have_temp & BIT(val - 1)) ||
!data->temp_src[val - 1]))
return -EINVAL;
case nct6791:
case nct6792:
case nct6793:
+ case nct6795:
nct6775_write_value(data, data->REG_CRITICAL_PWM[nr],
val);
reg = nct6775_read_value(data,
int pwm = index / 36; /* pwm index */
int nr = index % 36; /* attribute index */
- if (!(data->has_pwm & (1 << pwm)))
+ if (!(data->has_pwm & BIT(pwm)))
return 0;
if ((nr >= 14 && nr <= 18) || nr == 21) /* weight */
/* Enable temperature sensors if needed */
for (i = 0; i < NUM_TEMP; i++) {
- if (!(data->have_temp & (1 << i)))
+ if (!(data->have_temp & BIT(i)))
continue;
if (!data->reg_temp_config[i])
continue;
diode = nct6775_read_value(data, data->REG_DIODE);
for (i = 0; i < data->temp_fixed_num; i++) {
- if (!(data->have_temp_fixed & (1 << i)))
+ if (!(data->have_temp_fixed & BIT(i)))
continue;
if ((tmp & (data->DIODE_MASK << i))) /* diode */
data->temp_type[i]
if (data->kind == nct6775) {
regval = superio_inb(sioreg, 0x2c);
- fan3pin = regval & (1 << 6);
- pwm3pin = regval & (1 << 7);
+ fan3pin = regval & BIT(6);
+ pwm3pin = regval & BIT(7);
/* On NCT6775, fan4 shares pins with the fdc interface */
fan4pin = !(superio_inb(sioreg, 0x2A) & 0x80);
pwm4pin = false;
pwm5pin = false;
pwm6pin = false;
- } else { /* NCT6779D, NCT6791D, NCT6792D, or NCT6793D */
- regval = superio_inb(sioreg, 0x1c);
+ } else { /* NCT6779D, NCT6791D, NCT6792D, NCT6793D, or NCT6795D */
+ int regval_1b, regval_2a, regval_eb;
- fan3pin = !(regval & (1 << 5));
- fan4pin = !(regval & (1 << 6));
- fan5pin = !(regval & (1 << 7));
+ regval = superio_inb(sioreg, 0x1c);
- pwm3pin = !(regval & (1 << 0));
- pwm4pin = !(regval & (1 << 1));
- pwm5pin = !(regval & (1 << 2));
+ fan3pin = !(regval & BIT(5));
+ fan4pin = !(regval & BIT(6));
+ fan5pin = !(regval & BIT(7));
- fan4min = fan4pin;
+ pwm3pin = !(regval & BIT(0));
+ pwm4pin = !(regval & BIT(1));
+ pwm5pin = !(regval & BIT(2));
- if (data->kind == nct6791 || data->kind == nct6792 ||
- data->kind == nct6793) {
- regval = superio_inb(sioreg, 0x2d);
- fan6pin = (regval & (1 << 1));
- pwm6pin = (regval & (1 << 0));
- } else { /* NCT6779D */
+ regval = superio_inb(sioreg, 0x2d);
+ switch (data->kind) {
+ case nct6791:
+ case nct6792:
+ fan6pin = regval & BIT(1);
+ pwm6pin = regval & BIT(0);
+ break;
+ case nct6793:
+ case nct6795:
+ regval_1b = superio_inb(sioreg, 0x1b);
+ regval_2a = superio_inb(sioreg, 0x2a);
+
+ if (!pwm5pin)
+ pwm5pin = regval & BIT(7);
+ fan6pin = regval & BIT(1);
+ pwm6pin = regval & BIT(0);
+ if (!fan5pin)
+ fan5pin = regval_1b & BIT(5);
+
+ superio_select(sioreg, NCT6775_LD_12);
+ regval_eb = superio_inb(sioreg, 0xeb);
+ if (!fan5pin)
+ fan5pin = regval_eb & BIT(5);
+ if (!pwm5pin)
+ pwm5pin = (regval_eb & BIT(4)) &&
+ !(regval_2a & BIT(0));
+ if (!fan6pin)
+ fan6pin = regval_eb & BIT(3);
+ if (!pwm6pin)
+ pwm6pin = regval_eb & BIT(2);
+ break;
+ default: /* NCT6779D */
fan6pin = false;
pwm6pin = false;
+ break;
}
+
+ fan4min = fan4pin;
}
/* fan 1 and 2 (0x03) are always present */
continue;
src = nct6775_read_value(data, regp[i]);
src &= 0x1f;
- if (!src || (*mask & (1 << src)))
+ if (!src || (*mask & BIT(src)))
continue;
- if (src >= data->temp_label_num ||
- !strlen(data->temp_label[src]))
+ if (!(data->temp_mask & BIT(src)))
continue;
index = __ffs(*available);
nct6775_write_value(data, data->REG_TEMP_SOURCE[index], src);
- *available &= ~(1 << index);
- *mask |= 1 << src;
+ *available &= ~BIT(index);
+ *mask |= BIT(src);
}
}
data->fan_from_reg_min = fan_from_reg13;
data->temp_label = nct6776_temp_label;
- data->temp_label_num = ARRAY_SIZE(nct6776_temp_label);
+ data->temp_mask = NCT6776_TEMP_MASK;
data->REG_VBAT = NCT6106_REG_VBAT;
data->REG_DIODE = NCT6106_REG_DIODE;
data->speed_tolerance_limit = 15;
data->temp_label = nct6775_temp_label;
- data->temp_label_num = ARRAY_SIZE(nct6775_temp_label);
+ data->temp_mask = NCT6775_TEMP_MASK;
data->REG_CONFIG = NCT6775_REG_CONFIG;
data->REG_VBAT = NCT6775_REG_VBAT;
data->speed_tolerance_limit = 63;
data->temp_label = nct6776_temp_label;
- data->temp_label_num = ARRAY_SIZE(nct6776_temp_label);
+ data->temp_mask = NCT6776_TEMP_MASK;
data->REG_CONFIG = NCT6775_REG_CONFIG;
data->REG_VBAT = NCT6775_REG_VBAT;
data->speed_tolerance_limit = 63;
data->temp_label = nct6779_temp_label;
- data->temp_label_num = NCT6779_NUM_LABELS;
+ data->temp_mask = NCT6779_TEMP_MASK;
data->REG_CONFIG = NCT6775_REG_CONFIG;
data->REG_VBAT = NCT6775_REG_VBAT;
case nct6791:
case nct6792:
case nct6793:
+ case nct6795:
data->in_num = 15;
data->pwm_num = 6;
data->auto_pwm_num = 4;
default:
case nct6791:
data->temp_label = nct6779_temp_label;
+ data->temp_mask = NCT6791_TEMP_MASK;
break;
case nct6792:
data->temp_label = nct6792_temp_label;
+ data->temp_mask = NCT6792_TEMP_MASK;
break;
case nct6793:
data->temp_label = nct6793_temp_label;
+ data->temp_mask = NCT6793_TEMP_MASK;
+ break;
+ case nct6795:
+ data->temp_label = nct6795_temp_label;
+ data->temp_mask = NCT6795_TEMP_MASK;
break;
}
- data->temp_label_num = NCT6791_NUM_LABELS;
data->REG_CONFIG = NCT6775_REG_CONFIG;
data->REG_VBAT = NCT6775_REG_VBAT;
default:
return -ENODEV;
}
- data->have_in = (1 << data->in_num) - 1;
+ data->have_in = BIT(data->in_num) - 1;
data->have_temp = 0;
/*
continue;
src = nct6775_read_value(data, data->REG_TEMP_SOURCE[i]) & 0x1f;
- if (!src || (mask & (1 << src)))
- available |= 1 << i;
+ if (!src || (mask & BIT(src)))
+ available |= BIT(i);
- mask |= 1 << src;
+ mask |= BIT(src);
}
/*
continue;
src = nct6775_read_value(data, data->REG_TEMP_SOURCE[i]) & 0x1f;
- if (!src || (mask & (1 << src)))
+ if (!src || (mask & BIT(src)))
continue;
- if (src >= data->temp_label_num ||
- !strlen(data->temp_label[src])) {
+ if (!(data->temp_mask & BIT(src))) {
dev_info(dev,
"Invalid temperature source %d at index %d, source register 0x%x, temp register 0x%x\n",
src, i, data->REG_TEMP_SOURCE[i], reg_temp[i]);
continue;
}
- mask |= 1 << src;
+ mask |= BIT(src);
/* Use fixed index for SYSTIN(1), CPUTIN(2), AUXTIN(3) */
if (src <= data->temp_fixed_num) {
- data->have_temp |= 1 << (src - 1);
- data->have_temp_fixed |= 1 << (src - 1);
+ data->have_temp |= BIT(src - 1);
+ data->have_temp_fixed |= BIT(src - 1);
data->reg_temp[0][src - 1] = reg_temp[i];
data->reg_temp[1][src - 1] = reg_temp_over[i];
data->reg_temp[2][src - 1] = reg_temp_hyst[i];
continue;
/* Use dynamic index for other sources */
- data->have_temp |= 1 << s;
+ data->have_temp |= BIT(s);
data->reg_temp[0][s] = reg_temp[i];
data->reg_temp[1][s] = reg_temp_over[i];
data->reg_temp[2][s] = reg_temp_hyst[i];
if (!src)
continue;
- if (src >= data->temp_label_num ||
- !strlen(data->temp_label[src])) {
+ if (!(data->temp_mask & BIT(src))) {
dev_info(dev,
"Invalid temperature source %d at index %d, source register 0x%x, temp register 0x%x\n",
src, i, data->REG_TEMP_SEL[i],
* are no duplicates.
*/
if (src != TEMP_SOURCE_VIRTUAL) {
- if (mask & (1 << src))
+ if (mask & BIT(src))
continue;
- mask |= 1 << src;
+ mask |= BIT(src);
}
/* Use fixed index for SYSTIN(1), CPUTIN(2), AUXTIN(3) */
if (src <= data->temp_fixed_num) {
- if (data->have_temp & (1 << (src - 1)))
+ if (data->have_temp & BIT(src - 1))
continue;
- data->have_temp |= 1 << (src - 1);
- data->have_temp_fixed |= 1 << (src - 1);
+ data->have_temp |= BIT(src - 1);
+ data->have_temp_fixed |= BIT(src - 1);
data->reg_temp[0][src - 1] = reg_temp_mon[i];
data->temp_src[src - 1] = src;
continue;
continue;
/* Use dynamic index for other sources */
- data->have_temp |= 1 << s;
+ data->have_temp |= BIT(s);
data->reg_temp[0][s] = reg_temp_mon[i];
data->temp_src[s] = src;
s++;
* The temperature is already monitored if the respective bit in <mask>
* is set.
*/
- for (i = 0; i < data->temp_label_num - 1; i++) {
+ for (i = 0; i < 32; i++) {
+ if (!(data->temp_mask & BIT(i + 1)))
+ continue;
if (!reg_temp_alternate[i])
continue;
- if (mask & (1 << (i + 1)))
+ if (mask & BIT(i + 1))
continue;
if (i < data->temp_fixed_num) {
- if (data->have_temp & (1 << i))
+ if (data->have_temp & BIT(i))
continue;
- data->have_temp |= 1 << i;
- data->have_temp_fixed |= 1 << i;
+ data->have_temp |= BIT(i);
+ data->have_temp_fixed |= BIT(i);
data->reg_temp[0][i] = reg_temp_alternate[i];
if (i < num_reg_temp) {
data->reg_temp[1][i] = reg_temp_over[i];
if (s >= NUM_TEMP) /* Abort if no more space */
break;
- data->have_temp |= 1 << s;
+ data->have_temp |= BIT(s);
data->reg_temp[0][s] = reg_temp_alternate[i];
data->temp_src[s] = i + 1;
s++;
case nct6791:
case nct6792:
case nct6793:
+ case nct6795:
break;
}
case nct6791:
case nct6792:
case nct6793:
+ case nct6795:
tmp |= 0x7e;
break;
}
superio_outb(sioreg, SIO_REG_ENABLE, data->sio_reg_enable);
if (data->kind == nct6791 || data->kind == nct6792 ||
- data->kind == nct6793)
+ data->kind == nct6793 || data->kind == nct6795)
nct6791_enable_io_mapping(sioreg);
superio_exit(sioreg);
/* Restore limits */
for (i = 0; i < data->in_num; i++) {
- if (!(data->have_in & (1 << i)))
+ if (!(data->have_in & BIT(i)))
continue;
nct6775_write_value(data, data->REG_IN_MINMAX[0][i],
}
for (i = 0; i < ARRAY_SIZE(data->fan_min); i++) {
- if (!(data->has_fan_min & (1 << i)))
+ if (!(data->has_fan_min & BIT(i)))
continue;
nct6775_write_value(data, data->REG_FAN_MIN[i],
}
for (i = 0; i < NUM_TEMP; i++) {
- if (!(data->have_temp & (1 << i)))
+ if (!(data->have_temp & BIT(i)))
continue;
for (j = 1; j < ARRAY_SIZE(data->reg_temp); j++)
case SIO_NCT6793_ID:
sio_data->kind = nct6793;
break;
+ case SIO_NCT6795_ID:
+ sio_data->kind = nct6795;
+ break;
default:
if (val != 0xffff)
pr_debug("unsupported chip ID: 0x%04x\n", val);
}
if (sio_data->kind == nct6791 || sio_data->kind == nct6792 ||
- sio_data->kind == nct6793)
+ sio_data->kind == nct6793 || sio_data->kind == nct6795)
nct6791_enable_io_mapping(sioaddr);
superio_exit(sioaddr);
This driver can also be built as a module. If so, the module will
be called adm1275.
+config SENSORS_IR35221
+ tristate "Infineon IR35221"
+ default n
+ help
+ If you say yes here you get hardware monitoring support for the
+ Infineon IR35221 controller.
+
+ This driver can also be built as a module. If so, the module will
+ be called ir35521.
+
config SENSORS_LM25066
tristate "National Semiconductor LM25066 and compatibles"
default n
obj-$(CONFIG_PMBUS) += pmbus_core.o
obj-$(CONFIG_SENSORS_PMBUS) += pmbus.o
obj-$(CONFIG_SENSORS_ADM1275) += adm1275.o
+obj-$(CONFIG_SENSORS_IR35221) += ir35221.o
obj-$(CONFIG_SENSORS_LM25066) += lm25066.o
obj-$(CONFIG_SENSORS_LTC2978) += ltc2978.o
obj-$(CONFIG_SENSORS_LTC3815) += ltc3815.o
--- /dev/null
+/*
+ * Hardware monitoring driver for IR35221
+ *
+ * Copyright (C) IBM Corporation 2017.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#include <linux/err.h>
+#include <linux/i2c.h>
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include "pmbus.h"
+
+#define IR35221_MFR_VIN_PEAK 0xc5
+#define IR35221_MFR_VOUT_PEAK 0xc6
+#define IR35221_MFR_IOUT_PEAK 0xc7
+#define IR35221_MFR_TEMP_PEAK 0xc8
+#define IR35221_MFR_VIN_VALLEY 0xc9
+#define IR35221_MFR_VOUT_VALLEY 0xca
+#define IR35221_MFR_IOUT_VALLEY 0xcb
+#define IR35221_MFR_TEMP_VALLEY 0xcc
+
+static long ir35221_reg2data(int data, enum pmbus_sensor_classes class)
+{
+ s16 exponent;
+ s32 mantissa;
+ long val;
+
+ /* We only modify LINEAR11 formats */
+ exponent = ((s16)data) >> 11;
+ mantissa = ((s16)((data & 0x7ff) << 5)) >> 5;
+
+ val = mantissa * 1000L;
+
+ /* scale result to micro-units for power sensors */
+ if (class == PSC_POWER)
+ val = val * 1000L;
+
+ if (exponent >= 0)
+ val <<= exponent;
+ else
+ val >>= -exponent;
+
+ return val;
+}
+
+#define MAX_MANTISSA (1023 * 1000)
+#define MIN_MANTISSA (511 * 1000)
+
+static u16 ir35221_data2reg(long val, enum pmbus_sensor_classes class)
+{
+ s16 exponent = 0, mantissa;
+ bool negative = false;
+
+ if (val == 0)
+ return 0;
+
+ if (val < 0) {
+ negative = true;
+ val = -val;
+ }
+
+ /* Power is in uW. Convert to mW before converting. */
+ if (class == PSC_POWER)
+ val = DIV_ROUND_CLOSEST(val, 1000L);
+
+ /* Reduce large mantissa until it fits into 10 bit */
+ while (val >= MAX_MANTISSA && exponent < 15) {
+ exponent++;
+ val >>= 1;
+ }
+ /* Increase small mantissa to improve precision */
+ while (val < MIN_MANTISSA && exponent > -15) {
+ exponent--;
+ val <<= 1;
+ }
+
+ /* Convert mantissa from milli-units to units */
+ mantissa = DIV_ROUND_CLOSEST(val, 1000);
+
+ /* Ensure that resulting number is within range */
+ if (mantissa > 0x3ff)
+ mantissa = 0x3ff;
+
+ /* restore sign */
+ if (negative)
+ mantissa = -mantissa;
+
+ /* Convert to 5 bit exponent, 11 bit mantissa */
+ return (mantissa & 0x7ff) | ((exponent << 11) & 0xf800);
+}
+
+static u16 ir35221_scale_result(s16 data, int shift,
+ enum pmbus_sensor_classes class)
+{
+ long val;
+
+ val = ir35221_reg2data(data, class);
+
+ if (shift < 0)
+ val >>= -shift;
+ else
+ val <<= shift;
+
+ return ir35221_data2reg(val, class);
+}
+
+static int ir35221_read_word_data(struct i2c_client *client, int page, int reg)
+{
+ int ret;
+
+ switch (reg) {
+ case PMBUS_IOUT_OC_FAULT_LIMIT:
+ case PMBUS_IOUT_OC_WARN_LIMIT:
+ ret = pmbus_read_word_data(client, page, reg);
+ if (ret < 0)
+ break;
+ ret = ir35221_scale_result(ret, 1, PSC_CURRENT_OUT);
+ break;
+ case PMBUS_VIN_OV_FAULT_LIMIT:
+ case PMBUS_VIN_OV_WARN_LIMIT:
+ case PMBUS_VIN_UV_WARN_LIMIT:
+ ret = pmbus_read_word_data(client, page, reg);
+ ret = ir35221_scale_result(ret, -4, PSC_VOLTAGE_IN);
+ break;
+ case PMBUS_IIN_OC_WARN_LIMIT:
+ ret = pmbus_read_word_data(client, page, reg);
+ if (ret < 0)
+ break;
+ ret = ir35221_scale_result(ret, -1, PSC_CURRENT_IN);
+ break;
+ case PMBUS_READ_VIN:
+ ret = pmbus_read_word_data(client, page, PMBUS_READ_VIN);
+ if (ret < 0)
+ break;
+ ret = ir35221_scale_result(ret, -5, PSC_VOLTAGE_IN);
+ break;
+ case PMBUS_READ_IIN:
+ ret = pmbus_read_word_data(client, page, PMBUS_READ_IIN);
+ if (ret < 0)
+ break;
+ if (page == 0)
+ ret = ir35221_scale_result(ret, -4, PSC_CURRENT_IN);
+ else
+ ret = ir35221_scale_result(ret, -5, PSC_CURRENT_IN);
+ break;
+ case PMBUS_READ_POUT:
+ ret = pmbus_read_word_data(client, page, PMBUS_READ_POUT);
+ if (ret < 0)
+ break;
+ ret = ir35221_scale_result(ret, -1, PSC_POWER);
+ break;
+ case PMBUS_READ_PIN:
+ ret = pmbus_read_word_data(client, page, PMBUS_READ_PIN);
+ if (ret < 0)
+ break;
+ ret = ir35221_scale_result(ret, -1, PSC_POWER);
+ break;
+ case PMBUS_READ_IOUT:
+ ret = pmbus_read_word_data(client, page, PMBUS_READ_IOUT);
+ if (ret < 0)
+ break;
+ if (page == 0)
+ ret = ir35221_scale_result(ret, -1, PSC_CURRENT_OUT);
+ else
+ ret = ir35221_scale_result(ret, -2, PSC_CURRENT_OUT);
+ break;
+ case PMBUS_VIRT_READ_VIN_MAX:
+ ret = pmbus_read_word_data(client, page, IR35221_MFR_VIN_PEAK);
+ if (ret < 0)
+ break;
+ ret = ir35221_scale_result(ret, -5, PSC_VOLTAGE_IN);
+ break;
+ case PMBUS_VIRT_READ_VOUT_MAX:
+ ret = pmbus_read_word_data(client, page, IR35221_MFR_VOUT_PEAK);
+ break;
+ case PMBUS_VIRT_READ_IOUT_MAX:
+ ret = pmbus_read_word_data(client, page, IR35221_MFR_IOUT_PEAK);
+ if (ret < 0)
+ break;
+ if (page == 0)
+ ret = ir35221_scale_result(ret, -1, PSC_CURRENT_IN);
+ else
+ ret = ir35221_scale_result(ret, -2, PSC_CURRENT_IN);
+ break;
+ case PMBUS_VIRT_READ_TEMP_MAX:
+ ret = pmbus_read_word_data(client, page, IR35221_MFR_TEMP_PEAK);
+ break;
+ case PMBUS_VIRT_READ_VIN_MIN:
+ ret = pmbus_read_word_data(client, page,
+ IR35221_MFR_VIN_VALLEY);
+ if (ret < 0)
+ break;
+ ret = ir35221_scale_result(ret, -5, PSC_VOLTAGE_IN);
+ break;
+ case PMBUS_VIRT_READ_VOUT_MIN:
+ ret = pmbus_read_word_data(client, page,
+ IR35221_MFR_VOUT_VALLEY);
+ break;
+ case PMBUS_VIRT_READ_IOUT_MIN:
+ ret = pmbus_read_word_data(client, page,
+ IR35221_MFR_IOUT_VALLEY);
+ if (ret < 0)
+ break;
+ if (page == 0)
+ ret = ir35221_scale_result(ret, -1, PSC_CURRENT_IN);
+ else
+ ret = ir35221_scale_result(ret, -2, PSC_CURRENT_IN);
+ break;
+ case PMBUS_VIRT_READ_TEMP_MIN:
+ ret = pmbus_read_word_data(client, page,
+ IR35221_MFR_TEMP_VALLEY);
+ break;
+ default:
+ ret = -ENODATA;
+ break;
+ }
+
+ return ret;
+}
+
+static int ir35221_write_word_data(struct i2c_client *client, int page, int reg,
+ u16 word)
+{
+ int ret;
+ u16 val;
+
+ switch (reg) {
+ case PMBUS_IOUT_OC_FAULT_LIMIT:
+ case PMBUS_IOUT_OC_WARN_LIMIT:
+ val = ir35221_scale_result(word, -1, PSC_CURRENT_OUT);
+ ret = pmbus_write_word_data(client, page, reg, val);
+ break;
+ case PMBUS_VIN_OV_FAULT_LIMIT:
+ case PMBUS_VIN_OV_WARN_LIMIT:
+ case PMBUS_VIN_UV_WARN_LIMIT:
+ val = ir35221_scale_result(word, 4, PSC_VOLTAGE_IN);
+ ret = pmbus_write_word_data(client, page, reg, val);
+ break;
+ case PMBUS_IIN_OC_WARN_LIMIT:
+ val = ir35221_scale_result(word, 1, PSC_CURRENT_IN);
+ ret = pmbus_write_word_data(client, page, reg, val);
+ break;
+ default:
+ ret = -ENODATA;
+ break;
+ }
+
+ return ret;
+}
+
+static int ir35221_probe(struct i2c_client *client,
+ const struct i2c_device_id *id)
+{
+ struct pmbus_driver_info *info;
+ u8 buf[I2C_SMBUS_BLOCK_MAX];
+ int ret;
+
+ if (!i2c_check_functionality(client->adapter,
+ I2C_FUNC_SMBUS_READ_BYTE_DATA
+ | I2C_FUNC_SMBUS_READ_WORD_DATA
+ | I2C_FUNC_SMBUS_READ_BLOCK_DATA))
+ return -ENODEV;
+
+ ret = i2c_smbus_read_block_data(client, PMBUS_MFR_ID, buf);
+ if (ret < 0) {
+ dev_err(&client->dev, "Failed to read PMBUS_MFR_ID\n");
+ return ret;
+ }
+ if (ret != 2 || strncmp(buf, "RI", strlen("RI"))) {
+ dev_err(&client->dev, "MFR_ID unrecognised\n");
+ return -ENODEV;
+ }
+
+ ret = i2c_smbus_read_block_data(client, PMBUS_MFR_MODEL, buf);
+ if (ret < 0) {
+ dev_err(&client->dev, "Failed to read PMBUS_MFR_MODEL\n");
+ return ret;
+ }
+ if (ret != 2 || !(buf[0] == 0x6c && buf[1] == 0x00)) {
+ dev_err(&client->dev, "MFR_MODEL unrecognised\n");
+ return -ENODEV;
+ }
+
+ info = devm_kzalloc(&client->dev, sizeof(struct pmbus_driver_info),
+ GFP_KERNEL);
+ if (!info)
+ return -ENOMEM;
+
+ info->write_word_data = ir35221_write_word_data;
+ info->read_word_data = ir35221_read_word_data;
+
+ info->pages = 2;
+ info->format[PSC_VOLTAGE_IN] = linear;
+ info->format[PSC_VOLTAGE_OUT] = linear;
+ info->format[PSC_CURRENT_IN] = linear;
+ info->format[PSC_CURRENT_OUT] = linear;
+ info->format[PSC_POWER] = linear;
+ info->format[PSC_TEMPERATURE] = linear;
+
+ info->func[0] = PMBUS_HAVE_VIN
+ | PMBUS_HAVE_VOUT | PMBUS_HAVE_IIN
+ | PMBUS_HAVE_IOUT | PMBUS_HAVE_PIN
+ | PMBUS_HAVE_POUT | PMBUS_HAVE_TEMP
+ | PMBUS_HAVE_STATUS_VOUT | PMBUS_HAVE_STATUS_IOUT
+ | PMBUS_HAVE_STATUS_INPUT | PMBUS_HAVE_STATUS_TEMP;
+ info->func[1] = info->func[0];
+
+ return pmbus_do_probe(client, id, info);
+}
+
+static const struct i2c_device_id ir35221_id[] = {
+ {"ir35221", 0},
+ {}
+};
+
+MODULE_DEVICE_TABLE(i2c, ir35221_id);
+
+static struct i2c_driver ir35221_driver = {
+ .driver = {
+ .name = "ir35221",
+ },
+ .probe = ir35221_probe,
+ .remove = pmbus_do_remove,
+ .id_table = ir35221_id,
+};
+
+module_i2c_driver(ir35221_driver);
+
+MODULE_AUTHOR("Samuel Mendoza-Jonas <sam@mendozajonas.com");
+MODULE_DESCRIPTION("PMBus driver for IR35221");
+MODULE_LICENSE("GPL");
#include <linux/slab.h>
#include <linux/mutex.h>
#include <linux/i2c.h>
-#include <linux/i2c/pmbus.h>
+#include <linux/pmbus.h>
#include "pmbus.h"
/*
#include <linux/hwmon.h>
#include <linux/hwmon-sysfs.h>
#include <linux/jiffies.h>
-#include <linux/i2c/pmbus.h>
+#include <linux/pmbus.h>
#include <linux/regulator/driver.h>
#include <linux/regulator/machine.h>
#include "pmbus.h"
#include <linux/err.h>
#include <linux/slab.h>
#include <linux/i2c.h>
-#include <linux/i2c/pmbus.h>
+#include <linux/pmbus.h>
#include "pmbus.h"
enum chips { ucd9000, ucd90120, ucd90124, ucd90160, ucd9090, ucd90910 };
#include <linux/err.h>
#include <linux/slab.h>
#include <linux/i2c.h>
-#include <linux/i2c/pmbus.h>
+#include <linux/pmbus.h>
#include "pmbus.h"
#define UCD9200_PHASE_INFO 0xd2
static int __set_pwm(struct pwm_fan_ctx *ctx, unsigned long pwm)
{
- struct pwm_args pargs;
- unsigned long duty;
+ unsigned long period;
int ret = 0;
-
- pwm_get_args(ctx->pwm, &pargs);
+ struct pwm_state state = { };
mutex_lock(&ctx->lock);
if (ctx->pwm_value == pwm)
goto exit_set_pwm_err;
- duty = DIV_ROUND_UP(pwm * (pargs.period - 1), MAX_PWM);
- ret = pwm_config(ctx->pwm, duty, pargs.period);
- if (ret)
- goto exit_set_pwm_err;
-
- if (pwm == 0)
- pwm_disable(ctx->pwm);
-
- if (ctx->pwm_value == 0) {
- ret = pwm_enable(ctx->pwm);
- if (ret)
- goto exit_set_pwm_err;
- }
+ pwm_init_state(ctx->pwm, &state);
+ period = ctx->pwm->args.period;
+ state.duty_cycle = DIV_ROUND_UP(pwm * (period - 1), MAX_PWM);
+ state.enabled = pwm ? true : false;
- ctx->pwm_value = pwm;
+ ret = pwm_apply_state(ctx->pwm, &state);
+ if (!ret)
+ ctx->pwm_value = pwm;
exit_set_pwm_err:
mutex_unlock(&ctx->lock);
return ret;
{
struct thermal_cooling_device *cdev;
struct pwm_fan_ctx *ctx;
- struct pwm_args pargs;
struct device *hwmon;
- int duty_cycle;
int ret;
+ struct pwm_state state = { };
ctx = devm_kzalloc(&pdev->dev, sizeof(*ctx), GFP_KERNEL);
if (!ctx)
platform_set_drvdata(pdev, ctx);
- /*
- * FIXME: pwm_apply_args() should be removed when switching to the
- * atomic PWM API.
- */
- pwm_apply_args(ctx->pwm);
-
- /* Set duty cycle to maximum allowed */
- pwm_get_args(ctx->pwm, &pargs);
-
- duty_cycle = pargs.period - 1;
ctx->pwm_value = MAX_PWM;
- ret = pwm_config(ctx->pwm, duty_cycle, pargs.period);
- if (ret) {
- dev_err(&pdev->dev, "Failed to configure PWM\n");
- return ret;
- }
+ /* Set duty cycle to maximum allowed and enable PWM output */
+ pwm_init_state(ctx->pwm, &state);
+ state.duty_cycle = ctx->pwm->args.period - 1;
+ state.enabled = true;
- /* Enbale PWM output */
- ret = pwm_enable(ctx->pwm);
+ ret = pwm_apply_state(ctx->pwm, &state);
if (ret) {
- dev_err(&pdev->dev, "Failed to enable PWM\n");
+ dev_err(&pdev->dev, "Failed to configure PWM\n");
return ret;
}
ctx, pwm_fan_groups);
if (IS_ERR(hwmon)) {
dev_err(&pdev->dev, "Failed to register hwmon device\n");
- pwm_disable(ctx->pwm);
- return PTR_ERR(hwmon);
+ ret = PTR_ERR(hwmon);
+ goto err_pwm_disable;
}
ret = pwm_fan_of_get_cooling_data(&pdev->dev, ctx);
if (IS_ERR(cdev)) {
dev_err(&pdev->dev,
"Failed to register pwm-fan as cooling device");
- pwm_disable(ctx->pwm);
- return PTR_ERR(cdev);
+ ret = PTR_ERR(cdev);
+ goto err_pwm_disable;
}
ctx->cdev = cdev;
thermal_cdev_update(cdev);
}
return 0;
+
+err_pwm_disable:
+ state.enabled = false;
+ pwm_apply_state(ctx->pwm, &state);
+
+ return ret;
}
static int pwm_fan_remove(struct platform_device *pdev)
#include <linux/hwmon.h>
#include <linux/module.h>
+#include <linux/of_device.h>
#include <linux/platform_device.h>
#include <linux/scpi_protocol.h>
#include <linux/slab.h>
#include <linux/thermal.h>
struct sensor_data {
+ unsigned int scale;
struct scpi_sensor_info info;
struct device_attribute dev_attr_input;
struct device_attribute dev_attr_label;
const struct attribute_group *groups[2];
};
+static const u32 gxbb_scpi_scale[] = {
+ [TEMPERATURE] = 1, /* (celsius) */
+ [VOLTAGE] = 1000, /* (millivolts) */
+ [CURRENT] = 1000, /* (milliamperes) */
+ [POWER] = 1000000, /* (microwatts) */
+ [ENERGY] = 1000000, /* (microjoules) */
+};
+
+static const u32 scpi_scale[] = {
+ [TEMPERATURE] = 1000, /* (millicelsius) */
+ [VOLTAGE] = 1000, /* (millivolts) */
+ [CURRENT] = 1000, /* (milliamperes) */
+ [POWER] = 1000000, /* (microwatts) */
+ [ENERGY] = 1000000, /* (microjoules) */
+};
+
+static void scpi_scale_reading(u64 *value, struct sensor_data *sensor)
+{
+ if (scpi_scale[sensor->info.class] != sensor->scale) {
+ *value *= scpi_scale[sensor->info.class];
+ do_div(*value, sensor->scale);
+ }
+}
+
static int scpi_read_temp(void *dev, int *temp)
{
struct scpi_thermal_zone *zone = dev;
if (ret)
return ret;
+ scpi_scale_reading(&value, sensor);
+
*temp = value;
return 0;
}
if (ret)
return ret;
+ scpi_scale_reading(&value, sensor);
+
return sprintf(buf, "%llu\n", value);
}
.get_temp = scpi_read_temp,
};
+static const struct of_device_id scpi_of_match[] = {
+ {.compatible = "arm,scpi-sensors", .data = &scpi_scale},
+ {.compatible = "amlogic,meson-gxbb-scpi-sensors", .data = &gxbb_scpi_scale},
+ {},
+};
+MODULE_DEVICE_TABLE(of, scpi_of_match);
+
static int scpi_hwmon_probe(struct platform_device *pdev)
{
u16 nr_sensors, i;
+ const u32 *scale;
int num_temp = 0, num_volt = 0, num_current = 0, num_power = 0;
int num_energy = 0;
struct scpi_ops *scpi_ops;
struct device *hwdev, *dev = &pdev->dev;
struct scpi_sensors *scpi_sensors;
+ const struct of_device_id *of_id;
int idx, ret;
scpi_ops = get_scpi_ops();
scpi_sensors->scpi_ops = scpi_ops;
+ of_id = of_match_device(scpi_of_match, &pdev->dev);
+ if (!of_id) {
+ dev_err(&pdev->dev, "Unable to initialize scpi-hwmon data\n");
+ return -ENODEV;
+ }
+ scale = of_id->data;
+
for (i = 0, idx = 0; i < nr_sensors; i++) {
struct sensor_data *sensor = &scpi_sensors->data[idx];
continue;
}
+ sensor->scale = scale[sensor->info.class];
+
sensor->dev_attr_input.attr.mode = S_IRUGO;
sensor->dev_attr_input.show = scpi_show_sensor;
sensor->dev_attr_input.attr.name = sensor->input;
return 0;
}
-static const struct of_device_id scpi_of_match[] = {
- {.compatible = "arm,scpi-sensors"},
- {},
-};
-MODULE_DEVICE_TABLE(of, scpi_of_match);
-
static struct platform_driver scpi_hwmon_platdrv = {
.driver = {
.name = "scpi-hwmon",
logging useful software events or data coming from various entities
in the system, possibly running different OSs
+config CORESIGHT_CPU_DEBUG
+ tristate "CoreSight CPU Debug driver"
+ depends on ARM || ARM64
+ depends on DEBUG_FS
+ help
+ This driver provides support for coresight debugging module. This
+ is primarily used to dump sample-based profiling registers when
+ system triggers panic, the driver will parse context registers so
+ can quickly get to know program counter (PC), secure state,
+ exception level, etc. Before use debugging functionality, platform
+ needs to ensure the clock domain and power domain are enabled
+ properly, please refer Documentation/trace/coresight-cpu-debug.txt
+ for detailed description and the example for usage.
+
endif
coresight-etm4x-sysfs.o
obj-$(CONFIG_CORESIGHT_QCOM_REPLICATOR) += coresight-replicator-qcom.o
obj-$(CONFIG_CORESIGHT_STM) += coresight-stm.o
+obj-$(CONFIG_CORESIGHT_CPU_DEBUG) += coresight-cpu-debug.o
--- /dev/null
+/*
+ * Copyright (c) 2017 Linaro Limited. All rights reserved.
+ *
+ * Author: Leo Yan <leo.yan@linaro.org>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published by
+ * the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program. If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+#include <linux/amba/bus.h>
+#include <linux/coresight.h>
+#include <linux/cpu.h>
+#include <linux/debugfs.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/init.h>
+#include <linux/io.h>
+#include <linux/iopoll.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/moduleparam.h>
+#include <linux/pm_qos.h>
+#include <linux/slab.h>
+#include <linux/smp.h>
+#include <linux/types.h>
+#include <linux/uaccess.h>
+
+#include "coresight-priv.h"
+
+#define EDPCSR 0x0A0
+#define EDCIDSR 0x0A4
+#define EDVIDSR 0x0A8
+#define EDPCSR_HI 0x0AC
+#define EDOSLAR 0x300
+#define EDPRCR 0x310
+#define EDPRSR 0x314
+#define EDDEVID1 0xFC4
+#define EDDEVID 0xFC8
+
+#define EDPCSR_PROHIBITED 0xFFFFFFFF
+
+/* bits definition for EDPCSR */
+#define EDPCSR_THUMB BIT(0)
+#define EDPCSR_ARM_INST_MASK GENMASK(31, 2)
+#define EDPCSR_THUMB_INST_MASK GENMASK(31, 1)
+
+/* bits definition for EDPRCR */
+#define EDPRCR_COREPURQ BIT(3)
+#define EDPRCR_CORENPDRQ BIT(0)
+
+/* bits definition for EDPRSR */
+#define EDPRSR_DLK BIT(6)
+#define EDPRSR_PU BIT(0)
+
+/* bits definition for EDVIDSR */
+#define EDVIDSR_NS BIT(31)
+#define EDVIDSR_E2 BIT(30)
+#define EDVIDSR_E3 BIT(29)
+#define EDVIDSR_HV BIT(28)
+#define EDVIDSR_VMID GENMASK(7, 0)
+
+/*
+ * bits definition for EDDEVID1:PSCROffset
+ *
+ * NOTE: armv8 and armv7 have different definition for the register,
+ * so consolidate the bits definition as below:
+ *
+ * 0b0000 - Sample offset applies based on the instruction state, we
+ * rely on EDDEVID to check if EDPCSR is implemented or not
+ * 0b0001 - No offset applies.
+ * 0b0010 - No offset applies, but do not use in AArch32 mode
+ *
+ */
+#define EDDEVID1_PCSR_OFFSET_MASK GENMASK(3, 0)
+#define EDDEVID1_PCSR_OFFSET_INS_SET (0x0)
+#define EDDEVID1_PCSR_NO_OFFSET_DIS_AARCH32 (0x2)
+
+/* bits definition for EDDEVID */
+#define EDDEVID_PCSAMPLE_MODE GENMASK(3, 0)
+#define EDDEVID_IMPL_EDPCSR (0x1)
+#define EDDEVID_IMPL_EDPCSR_EDCIDSR (0x2)
+#define EDDEVID_IMPL_FULL (0x3)
+
+#define DEBUG_WAIT_SLEEP 1000
+#define DEBUG_WAIT_TIMEOUT 32000
+
+struct debug_drvdata {
+ void __iomem *base;
+ struct device *dev;
+ int cpu;
+
+ bool edpcsr_present;
+ bool edcidsr_present;
+ bool edvidsr_present;
+ bool pc_has_offset;
+
+ u32 edpcsr;
+ u32 edpcsr_hi;
+ u32 edprsr;
+ u32 edvidsr;
+ u32 edcidsr;
+};
+
+static DEFINE_MUTEX(debug_lock);
+static DEFINE_PER_CPU(struct debug_drvdata *, debug_drvdata);
+static int debug_count;
+static struct dentry *debug_debugfs_dir;
+
+static bool debug_enable;
+module_param_named(enable, debug_enable, bool, 0600);
+MODULE_PARM_DESC(enable, "Control to enable coresight CPU debug functionality");
+
+static void debug_os_unlock(struct debug_drvdata *drvdata)
+{
+ /* Unlocks the debug registers */
+ writel_relaxed(0x0, drvdata->base + EDOSLAR);
+
+ /* Make sure the registers are unlocked before accessing */
+ wmb();
+}
+
+/*
+ * According to ARM DDI 0487A.k, before access external debug
+ * registers should firstly check the access permission; if any
+ * below condition has been met then cannot access debug
+ * registers to avoid lockup issue:
+ *
+ * - CPU power domain is powered off;
+ * - The OS Double Lock is locked;
+ *
+ * By checking EDPRSR can get to know if meet these conditions.
+ */
+static bool debug_access_permitted(struct debug_drvdata *drvdata)
+{
+ /* CPU is powered off */
+ if (!(drvdata->edprsr & EDPRSR_PU))
+ return false;
+
+ /* The OS Double Lock is locked */
+ if (drvdata->edprsr & EDPRSR_DLK)
+ return false;
+
+ return true;
+}
+
+static void debug_force_cpu_powered_up(struct debug_drvdata *drvdata)
+{
+ u32 edprcr;
+
+try_again:
+
+ /*
+ * Send request to power management controller and assert
+ * DBGPWRUPREQ signal; if power management controller has
+ * sane implementation, it should enable CPU power domain
+ * in case CPU is in low power state.
+ */
+ edprcr = readl_relaxed(drvdata->base + EDPRCR);
+ edprcr |= EDPRCR_COREPURQ;
+ writel_relaxed(edprcr, drvdata->base + EDPRCR);
+
+ /* Wait for CPU to be powered up (timeout~=32ms) */
+ if (readx_poll_timeout_atomic(readl_relaxed, drvdata->base + EDPRSR,
+ drvdata->edprsr, (drvdata->edprsr & EDPRSR_PU),
+ DEBUG_WAIT_SLEEP, DEBUG_WAIT_TIMEOUT)) {
+ /*
+ * Unfortunately the CPU cannot be powered up, so return
+ * back and later has no permission to access other
+ * registers. For this case, should disable CPU low power
+ * states to ensure CPU power domain is enabled!
+ */
+ dev_err(drvdata->dev, "%s: power up request for CPU%d failed\n",
+ __func__, drvdata->cpu);
+ return;
+ }
+
+ /*
+ * At this point the CPU is powered up, so set the no powerdown
+ * request bit so we don't lose power and emulate power down.
+ */
+ edprcr = readl_relaxed(drvdata->base + EDPRCR);
+ edprcr |= EDPRCR_COREPURQ | EDPRCR_CORENPDRQ;
+ writel_relaxed(edprcr, drvdata->base + EDPRCR);
+
+ drvdata->edprsr = readl_relaxed(drvdata->base + EDPRSR);
+
+ /* The core power domain got switched off on use, try again */
+ if (unlikely(!(drvdata->edprsr & EDPRSR_PU)))
+ goto try_again;
+}
+
+static void debug_read_regs(struct debug_drvdata *drvdata)
+{
+ u32 save_edprcr;
+
+ CS_UNLOCK(drvdata->base);
+
+ /* Unlock os lock */
+ debug_os_unlock(drvdata);
+
+ /* Save EDPRCR register */
+ save_edprcr = readl_relaxed(drvdata->base + EDPRCR);
+
+ /*
+ * Ensure CPU power domain is enabled to let registers
+ * are accessiable.
+ */
+ debug_force_cpu_powered_up(drvdata);
+
+ if (!debug_access_permitted(drvdata))
+ goto out;
+
+ drvdata->edpcsr = readl_relaxed(drvdata->base + EDPCSR);
+
+ /*
+ * As described in ARM DDI 0487A.k, if the processing
+ * element (PE) is in debug state, or sample-based
+ * profiling is prohibited, EDPCSR reads as 0xFFFFFFFF;
+ * EDCIDSR, EDVIDSR and EDPCSR_HI registers also become
+ * UNKNOWN state. So directly bail out for this case.
+ */
+ if (drvdata->edpcsr == EDPCSR_PROHIBITED)
+ goto out;
+
+ /*
+ * A read of the EDPCSR normally has the side-effect of
+ * indirectly writing to EDCIDSR, EDVIDSR and EDPCSR_HI;
+ * at this point it's safe to read value from them.
+ */
+ if (IS_ENABLED(CONFIG_64BIT))
+ drvdata->edpcsr_hi = readl_relaxed(drvdata->base + EDPCSR_HI);
+
+ if (drvdata->edcidsr_present)
+ drvdata->edcidsr = readl_relaxed(drvdata->base + EDCIDSR);
+
+ if (drvdata->edvidsr_present)
+ drvdata->edvidsr = readl_relaxed(drvdata->base + EDVIDSR);
+
+out:
+ /* Restore EDPRCR register */
+ writel_relaxed(save_edprcr, drvdata->base + EDPRCR);
+
+ CS_LOCK(drvdata->base);
+}
+
+#ifdef CONFIG_64BIT
+static unsigned long debug_adjust_pc(struct debug_drvdata *drvdata)
+{
+ return (unsigned long)drvdata->edpcsr_hi << 32 |
+ (unsigned long)drvdata->edpcsr;
+}
+#else
+static unsigned long debug_adjust_pc(struct debug_drvdata *drvdata)
+{
+ unsigned long arm_inst_offset = 0, thumb_inst_offset = 0;
+ unsigned long pc;
+
+ pc = (unsigned long)drvdata->edpcsr;
+
+ if (drvdata->pc_has_offset) {
+ arm_inst_offset = 8;
+ thumb_inst_offset = 4;
+ }
+
+ /* Handle thumb instruction */
+ if (pc & EDPCSR_THUMB) {
+ pc = (pc & EDPCSR_THUMB_INST_MASK) - thumb_inst_offset;
+ return pc;
+ }
+
+ /*
+ * Handle arm instruction offset, if the arm instruction
+ * is not 4 byte alignment then it's possible the case
+ * for implementation defined; keep original value for this
+ * case and print info for notice.
+ */
+ if (pc & BIT(1))
+ dev_emerg(drvdata->dev,
+ "Instruction offset is implementation defined\n");
+ else
+ pc = (pc & EDPCSR_ARM_INST_MASK) - arm_inst_offset;
+
+ return pc;
+}
+#endif
+
+static void debug_dump_regs(struct debug_drvdata *drvdata)
+{
+ struct device *dev = drvdata->dev;
+ unsigned long pc;
+
+ dev_emerg(dev, " EDPRSR: %08x (Power:%s DLK:%s)\n",
+ drvdata->edprsr,
+ drvdata->edprsr & EDPRSR_PU ? "On" : "Off",
+ drvdata->edprsr & EDPRSR_DLK ? "Lock" : "Unlock");
+
+ if (!debug_access_permitted(drvdata)) {
+ dev_emerg(dev, "No permission to access debug registers!\n");
+ return;
+ }
+
+ if (drvdata->edpcsr == EDPCSR_PROHIBITED) {
+ dev_emerg(dev, "CPU is in Debug state or profiling is prohibited!\n");
+ return;
+ }
+
+ pc = debug_adjust_pc(drvdata);
+ dev_emerg(dev, " EDPCSR: [<%p>] %pS\n", (void *)pc, (void *)pc);
+
+ if (drvdata->edcidsr_present)
+ dev_emerg(dev, " EDCIDSR: %08x\n", drvdata->edcidsr);
+
+ if (drvdata->edvidsr_present)
+ dev_emerg(dev, " EDVIDSR: %08x (State:%s Mode:%s Width:%dbits VMID:%x)\n",
+ drvdata->edvidsr,
+ drvdata->edvidsr & EDVIDSR_NS ?
+ "Non-secure" : "Secure",
+ drvdata->edvidsr & EDVIDSR_E3 ? "EL3" :
+ (drvdata->edvidsr & EDVIDSR_E2 ?
+ "EL2" : "EL1/0"),
+ drvdata->edvidsr & EDVIDSR_HV ? 64 : 32,
+ drvdata->edvidsr & (u32)EDVIDSR_VMID);
+}
+
+static void debug_init_arch_data(void *info)
+{
+ struct debug_drvdata *drvdata = info;
+ u32 mode, pcsr_offset;
+ u32 eddevid, eddevid1;
+
+ CS_UNLOCK(drvdata->base);
+
+ /* Read device info */
+ eddevid = readl_relaxed(drvdata->base + EDDEVID);
+ eddevid1 = readl_relaxed(drvdata->base + EDDEVID1);
+
+ CS_LOCK(drvdata->base);
+
+ /* Parse implementation feature */
+ mode = eddevid & EDDEVID_PCSAMPLE_MODE;
+ pcsr_offset = eddevid1 & EDDEVID1_PCSR_OFFSET_MASK;
+
+ drvdata->edpcsr_present = false;
+ drvdata->edcidsr_present = false;
+ drvdata->edvidsr_present = false;
+ drvdata->pc_has_offset = false;
+
+ switch (mode) {
+ case EDDEVID_IMPL_FULL:
+ drvdata->edvidsr_present = true;
+ /* Fall through */
+ case EDDEVID_IMPL_EDPCSR_EDCIDSR:
+ drvdata->edcidsr_present = true;
+ /* Fall through */
+ case EDDEVID_IMPL_EDPCSR:
+ /*
+ * In ARM DDI 0487A.k, the EDDEVID1.PCSROffset is used to
+ * define if has the offset for PC sampling value; if read
+ * back EDDEVID1.PCSROffset == 0x2, then this means the debug
+ * module does not sample the instruction set state when
+ * armv8 CPU in AArch32 state.
+ */
+ drvdata->edpcsr_present =
+ ((IS_ENABLED(CONFIG_64BIT) && pcsr_offset != 0) ||
+ (pcsr_offset != EDDEVID1_PCSR_NO_OFFSET_DIS_AARCH32));
+
+ drvdata->pc_has_offset =
+ (pcsr_offset == EDDEVID1_PCSR_OFFSET_INS_SET);
+ break;
+ default:
+ break;
+ }
+}
+
+/*
+ * Dump out information on panic.
+ */
+static int debug_notifier_call(struct notifier_block *self,
+ unsigned long v, void *p)
+{
+ int cpu;
+ struct debug_drvdata *drvdata;
+
+ mutex_lock(&debug_lock);
+
+ /* Bail out if the functionality is disabled */
+ if (!debug_enable)
+ goto skip_dump;
+
+ pr_emerg("ARM external debug module:\n");
+
+ for_each_possible_cpu(cpu) {
+ drvdata = per_cpu(debug_drvdata, cpu);
+ if (!drvdata)
+ continue;
+
+ dev_emerg(drvdata->dev, "CPU[%d]:\n", drvdata->cpu);
+
+ debug_read_regs(drvdata);
+ debug_dump_regs(drvdata);
+ }
+
+skip_dump:
+ mutex_unlock(&debug_lock);
+ return 0;
+}
+
+static struct notifier_block debug_notifier = {
+ .notifier_call = debug_notifier_call,
+};
+
+static int debug_enable_func(void)
+{
+ struct debug_drvdata *drvdata;
+ int cpu, ret = 0;
+ cpumask_t mask;
+
+ /*
+ * Use cpumask to track which debug power domains have
+ * been powered on and use it to handle failure case.
+ */
+ cpumask_clear(&mask);
+
+ for_each_possible_cpu(cpu) {
+ drvdata = per_cpu(debug_drvdata, cpu);
+ if (!drvdata)
+ continue;
+
+ ret = pm_runtime_get_sync(drvdata->dev);
+ if (ret < 0)
+ goto err;
+ else
+ cpumask_set_cpu(cpu, &mask);
+ }
+
+ return 0;
+
+err:
+ /*
+ * If pm_runtime_get_sync() has failed, need rollback on
+ * all the other CPUs that have been enabled before that.
+ */
+ for_each_cpu(cpu, &mask) {
+ drvdata = per_cpu(debug_drvdata, cpu);
+ pm_runtime_put_noidle(drvdata->dev);
+ }
+
+ return ret;
+}
+
+static int debug_disable_func(void)
+{
+ struct debug_drvdata *drvdata;
+ int cpu, ret, err = 0;
+
+ /*
+ * Disable debug power domains, records the error and keep
+ * circling through all other CPUs when an error has been
+ * encountered.
+ */
+ for_each_possible_cpu(cpu) {
+ drvdata = per_cpu(debug_drvdata, cpu);
+ if (!drvdata)
+ continue;
+
+ ret = pm_runtime_put(drvdata->dev);
+ if (ret < 0)
+ err = ret;
+ }
+
+ return err;
+}
+
+static ssize_t debug_func_knob_write(struct file *f,
+ const char __user *buf, size_t count, loff_t *ppos)
+{
+ u8 val;
+ int ret;
+
+ ret = kstrtou8_from_user(buf, count, 2, &val);
+ if (ret)
+ return ret;
+
+ mutex_lock(&debug_lock);
+
+ if (val == debug_enable)
+ goto out;
+
+ if (val)
+ ret = debug_enable_func();
+ else
+ ret = debug_disable_func();
+
+ if (ret) {
+ pr_err("%s: unable to %s debug function: %d\n",
+ __func__, val ? "enable" : "disable", ret);
+ goto err;
+ }
+
+ debug_enable = val;
+out:
+ ret = count;
+err:
+ mutex_unlock(&debug_lock);
+ return ret;
+}
+
+static ssize_t debug_func_knob_read(struct file *f,
+ char __user *ubuf, size_t count, loff_t *ppos)
+{
+ ssize_t ret;
+ char buf[3];
+
+ mutex_lock(&debug_lock);
+ snprintf(buf, sizeof(buf), "%d\n", debug_enable);
+ mutex_unlock(&debug_lock);
+
+ ret = simple_read_from_buffer(ubuf, count, ppos, buf, sizeof(buf));
+ return ret;
+}
+
+static const struct file_operations debug_func_knob_fops = {
+ .open = simple_open,
+ .read = debug_func_knob_read,
+ .write = debug_func_knob_write,
+};
+
+static int debug_func_init(void)
+{
+ struct dentry *file;
+ int ret;
+
+ /* Create debugfs node */
+ debug_debugfs_dir = debugfs_create_dir("coresight_cpu_debug", NULL);
+ if (!debug_debugfs_dir) {
+ pr_err("%s: unable to create debugfs directory\n", __func__);
+ return -ENOMEM;
+ }
+
+ file = debugfs_create_file("enable", 0644, debug_debugfs_dir, NULL,
+ &debug_func_knob_fops);
+ if (!file) {
+ pr_err("%s: unable to create enable knob file\n", __func__);
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ /* Register function to be called for panic */
+ ret = atomic_notifier_chain_register(&panic_notifier_list,
+ &debug_notifier);
+ if (ret) {
+ pr_err("%s: unable to register notifier: %d\n",
+ __func__, ret);
+ goto err;
+ }
+
+ return 0;
+
+err:
+ debugfs_remove_recursive(debug_debugfs_dir);
+ return ret;
+}
+
+static void debug_func_exit(void)
+{
+ atomic_notifier_chain_unregister(&panic_notifier_list,
+ &debug_notifier);
+ debugfs_remove_recursive(debug_debugfs_dir);
+}
+
+static int debug_probe(struct amba_device *adev, const struct amba_id *id)
+{
+ void __iomem *base;
+ struct device *dev = &adev->dev;
+ struct debug_drvdata *drvdata;
+ struct resource *res = &adev->res;
+ struct device_node *np = adev->dev.of_node;
+ int ret;
+
+ drvdata = devm_kzalloc(dev, sizeof(*drvdata), GFP_KERNEL);
+ if (!drvdata)
+ return -ENOMEM;
+
+ drvdata->cpu = np ? of_coresight_get_cpu(np) : 0;
+ if (per_cpu(debug_drvdata, drvdata->cpu)) {
+ dev_err(dev, "CPU%d drvdata has already been initialized\n",
+ drvdata->cpu);
+ return -EBUSY;
+ }
+
+ drvdata->dev = &adev->dev;
+ amba_set_drvdata(adev, drvdata);
+
+ /* Validity for the resource is already checked by the AMBA core */
+ base = devm_ioremap_resource(dev, res);
+ if (IS_ERR(base))
+ return PTR_ERR(base);
+
+ drvdata->base = base;
+
+ get_online_cpus();
+ per_cpu(debug_drvdata, drvdata->cpu) = drvdata;
+ ret = smp_call_function_single(drvdata->cpu, debug_init_arch_data,
+ drvdata, 1);
+ put_online_cpus();
+
+ if (ret) {
+ dev_err(dev, "CPU%d debug arch init failed\n", drvdata->cpu);
+ goto err;
+ }
+
+ if (!drvdata->edpcsr_present) {
+ dev_err(dev, "CPU%d sample-based profiling isn't implemented\n",
+ drvdata->cpu);
+ ret = -ENXIO;
+ goto err;
+ }
+
+ if (!debug_count++) {
+ ret = debug_func_init();
+ if (ret)
+ goto err_func_init;
+ }
+
+ mutex_lock(&debug_lock);
+ /* Turn off debug power domain if debugging is disabled */
+ if (!debug_enable)
+ pm_runtime_put(dev);
+ mutex_unlock(&debug_lock);
+
+ dev_info(dev, "Coresight debug-CPU%d initialized\n", drvdata->cpu);
+ return 0;
+
+err_func_init:
+ debug_count--;
+err:
+ per_cpu(debug_drvdata, drvdata->cpu) = NULL;
+ return ret;
+}
+
+static int debug_remove(struct amba_device *adev)
+{
+ struct device *dev = &adev->dev;
+ struct debug_drvdata *drvdata = amba_get_drvdata(adev);
+
+ per_cpu(debug_drvdata, drvdata->cpu) = NULL;
+
+ mutex_lock(&debug_lock);
+ /* Turn off debug power domain before rmmod the module */
+ if (debug_enable)
+ pm_runtime_put(dev);
+ mutex_unlock(&debug_lock);
+
+ if (!--debug_count)
+ debug_func_exit();
+
+ return 0;
+}
+
+static struct amba_id debug_ids[] = {
+ { /* Debug for Cortex-A53 */
+ .id = 0x000bbd03,
+ .mask = 0x000fffff,
+ },
+ { /* Debug for Cortex-A57 */
+ .id = 0x000bbd07,
+ .mask = 0x000fffff,
+ },
+ { /* Debug for Cortex-A72 */
+ .id = 0x000bbd08,
+ .mask = 0x000fffff,
+ },
+ { 0, 0 },
+};
+
+static struct amba_driver debug_driver = {
+ .drv = {
+ .name = "coresight-cpu-debug",
+ .suppress_bind_attrs = true,
+ },
+ .probe = debug_probe,
+ .remove = debug_remove,
+ .id_table = debug_ids,
+};
+
+module_amba_driver(debug_driver);
+
+MODULE_AUTHOR("Leo Yan <leo.yan@linaro.org>");
+MODULE_DESCRIPTION("ARM Coresight CPU Debug Driver");
+MODULE_LICENSE("GPL");
/*
* Entries should be aligned to the frame size. If they are not
- * go back to the last alignement point to give decoding tools a
+ * go back to the last alignment point to give decoding tools a
* chance to fix things.
*/
if (write_ptr % ETB_FRAME_SIZE_WORDS) {
drvdata->buf = devm_kzalloc(dev,
drvdata->buffer_depth * 4, GFP_KERNEL);
- if (!drvdata->buf) {
- dev_err(dev, "Failed to allocate %u bytes for buffer data\n",
- drvdata->buffer_depth * 4);
+ if (!drvdata->buf)
return -ENOMEM;
- }
desc.type = CORESIGHT_DEV_TYPE_SINK;
desc.subtype.sink_subtype = CORESIGHT_DEV_SUBTYPE_SINK_BUFFER;
event_data = alloc_event_data(event_cpu);
if (!event_data)
return NULL;
+ INIT_WORK(&event_data->work, free_event_data);
/*
* In theory nothing prevent tracers in a trace session from being
if (!sink)
goto err;
- INIT_WORK(&event_data->work, free_event_data);
-
mask = &event_data->mask;
/* Setup the path for each CPU in a trace session */
* after cpu online mask indicates the cpu is offline but before the
* DYING hotplug callback is serviced by the ETM driver.
*/
- get_online_cpus();
+ cpus_read_lock();
spin_lock(&drvdata->spinlock);
/*
smp_call_function_single(drvdata->cpu, etm_disable_hw, drvdata, 1);
spin_unlock(&drvdata->spinlock);
- put_online_cpus();
+ cpus_read_unlock();
dev_info(drvdata->dev, "ETM tracing disabled\n");
}
drvdata->cpu = pdata ? pdata->cpu : 0;
- get_online_cpus();
+ cpus_read_lock();
etmdrvdata[drvdata->cpu] = drvdata;
if (smp_call_function_single(drvdata->cpu,
dev_err(dev, "ETM arch init failed\n");
if (!etm_count++) {
- cpuhp_setup_state_nocalls(CPUHP_AP_ARM_CORESIGHT_STARTING,
- "arm/coresight:starting",
- etm_starting_cpu, etm_dying_cpu);
- ret = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
- "arm/coresight:online",
- etm_online_cpu, NULL);
+ cpuhp_setup_state_nocalls_cpuslocked(CPUHP_AP_ARM_CORESIGHT_STARTING,
+ "arm/coresight:starting",
+ etm_starting_cpu, etm_dying_cpu);
+ ret = cpuhp_setup_state_nocalls_cpuslocked(CPUHP_AP_ONLINE_DYN,
+ "arm/coresight:online",
+ etm_online_cpu, NULL);
if (ret < 0)
goto err_arch_supported;
hp_online = ret;
}
- put_online_cpus();
+ cpus_read_unlock();
if (etm_arch_supported(drvdata->arch) == false) {
ret = -EINVAL;
* after cpu online mask indicates the cpu is offline but before the
* DYING hotplug callback is serviced by the ETM driver.
*/
- get_online_cpus();
+ cpus_read_lock();
spin_lock(&drvdata->spinlock);
/*
smp_call_function_single(drvdata->cpu, etm4_disable_hw, drvdata, 1);
spin_unlock(&drvdata->spinlock);
- put_online_cpus();
+ cpus_read_unlock();
dev_info(drvdata->dev, "ETM tracing disabled\n");
}
drvdata->cpu = pdata ? pdata->cpu : 0;
- get_online_cpus();
+ cpus_read_lock();
etmdrvdata[drvdata->cpu] = drvdata;
if (smp_call_function_single(drvdata->cpu,
dev_err(dev, "ETM arch init failed\n");
if (!etm4_count++) {
- cpuhp_setup_state_nocalls(CPUHP_AP_ARM_CORESIGHT_STARTING,
- "arm/coresight4:starting",
- etm4_starting_cpu, etm4_dying_cpu);
- ret = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
- "arm/coresight4:online",
- etm4_online_cpu, NULL);
+ cpuhp_setup_state_nocalls_cpuslocked(CPUHP_AP_ARM_CORESIGHT_STARTING,
+ "arm/coresight4:starting",
+ etm4_starting_cpu, etm4_dying_cpu);
+ ret = cpuhp_setup_state_nocalls_cpuslocked(CPUHP_AP_ONLINE_DYN,
+ "arm/coresight4:online",
+ etm4_online_cpu, NULL);
if (ret < 0)
goto err_arch_supported;
hp_online = ret;
}
- put_online_cpus();
+ cpus_read_unlock();
if (etm4_arch_supported(drvdata->arch) == false) {
ret = -EINVAL;
if (!used)
kfree(buf);
- if (!ret)
- dev_info(drvdata->dev, "TMC-ETB/ETF enabled\n");
-
return ret;
}
static int tmc_enable_etf_sink(struct coresight_device *csdev, u32 mode)
{
+ int ret;
+ struct tmc_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
+
switch (mode) {
case CS_MODE_SYSFS:
- return tmc_enable_etf_sink_sysfs(csdev);
+ ret = tmc_enable_etf_sink_sysfs(csdev);
+ break;
case CS_MODE_PERF:
- return tmc_enable_etf_sink_perf(csdev);
+ ret = tmc_enable_etf_sink_perf(csdev);
+ break;
+ /* We shouldn't be here */
+ default:
+ ret = -EINVAL;
+ break;
}
- /* We shouldn't be here */
- return -EINVAL;
+ if (ret)
+ return ret;
+
+ dev_info(drvdata->dev, "TMC-ETB/ETF enabled\n");
+ return 0;
}
static void tmc_disable_etf_sink(struct coresight_device *csdev)
drvdata->mode = CS_MODE_DISABLED;
spin_unlock_irqrestore(&drvdata->spinlock, flags);
- dev_info(drvdata->dev, "TMC disabled\n");
+ dev_info(drvdata->dev, "TMC-ETF disabled\n");
}
static void *tmc_alloc_etf_buffer(struct coresight_device *csdev, int cpu,
desc.type = CORESIGHT_DEV_TYPE_SINK;
desc.subtype.sink_subtype = CORESIGHT_DEV_SUBTYPE_SINK_BUFFER;
desc.ops = &tmc_etr_cs_ops;
+ /*
+ * ETR configuration uses a 40-bit AXI master in place of
+ * the embedded SRAM of ETB/ETF.
+ */
+ ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(40));
+ if (ret)
+ goto out;
} else {
desc.type = CORESIGHT_DEV_TYPE_LINKSINK;
desc.subtype.link_subtype = CORESIGHT_DEV_SUBTYPE_LINK_FIFO;
return 0;
}
-static void coresight_disable_source(struct coresight_device *csdev)
+/**
+ * coresight_disable_source - Drop the reference count by 1 and disable
+ * the device if there are no users left.
+ *
+ * @csdev - The coresight device to disable
+ *
+ * Returns true if the device has been disabled.
+ */
+static bool coresight_disable_source(struct coresight_device *csdev)
{
if (atomic_dec_return(csdev->refcnt) == 0) {
- if (source_ops(csdev)->disable) {
+ if (source_ops(csdev)->disable)
source_ops(csdev)->disable(csdev, NULL);
- csdev->enable = false;
- }
+ csdev->enable = false;
}
+ return !csdev->enable;
}
void coresight_disable_path(struct list_head *path)
int cpu, ret = 0;
struct coresight_device *sink;
struct list_head *path;
+ enum coresight_dev_subtype_source subtype;
+
+ subtype = csdev->subtype.source_subtype;
mutex_lock(&coresight_mutex);
if (ret)
goto out;
- if (csdev->enable)
+ if (csdev->enable) {
+ /*
+ * There could be multiple applications driving the software
+ * source. So keep the refcount for each such user when the
+ * source is already enabled.
+ */
+ if (subtype == CORESIGHT_DEV_SUBTYPE_SOURCE_SOFTWARE)
+ atomic_inc(csdev->refcnt);
goto out;
+ }
/*
* Search for a valid sink for this session but don't reset the
if (ret)
goto err_source;
- switch (csdev->subtype.source_subtype) {
+ switch (subtype) {
case CORESIGHT_DEV_SUBTYPE_SOURCE_PROC:
/*
* When working from sysFS it is important to keep track
if (ret)
goto out;
- if (!csdev->enable)
+ if (!csdev->enable || !coresight_disable_source(csdev))
goto out;
switch (csdev->subtype.source_subtype) {
break;
}
- coresight_disable_source(csdev);
coresight_disable_path(path);
coresight_release_path(path);
endpoint, of_dev_node_match);
}
-static void of_coresight_get_ports(struct device_node *node,
+static void of_coresight_get_ports(const struct device_node *node,
int *nr_inport, int *nr_outport)
{
struct device_node *ep = NULL;
return 0;
}
-struct coresight_platform_data *of_get_coresight_platform_data(
- struct device *dev, struct device_node *node)
+int of_coresight_get_cpu(const struct device_node *node)
{
- int i = 0, ret = 0, cpu;
+ int cpu;
+ bool found;
+ struct device_node *dn, *np;
+
+ dn = of_parse_phandle(node, "cpu", 0);
+
+ /* Affinity defaults to CPU0 */
+ if (!dn)
+ return 0;
+
+ for_each_possible_cpu(cpu) {
+ np = of_cpu_device_node_get(cpu);
+ found = (dn == np);
+ of_node_put(np);
+ if (found)
+ break;
+ }
+ of_node_put(dn);
+
+ /* Affinity to CPU0 if no cpu nodes are found */
+ return found ? cpu : 0;
+}
+EXPORT_SYMBOL_GPL(of_coresight_get_cpu);
+
+struct coresight_platform_data *
+of_get_coresight_platform_data(struct device *dev,
+ const struct device_node *node)
+{
+ int i = 0, ret = 0;
struct coresight_platform_data *pdata;
struct of_endpoint endpoint, rendpoint;
struct device *rdev;
- struct device_node *dn;
struct device_node *ep = NULL;
struct device_node *rparent = NULL;
struct device_node *rport = NULL;
} while (ep);
}
- /* Affinity defaults to CPU0 */
- pdata->cpu = 0;
- dn = of_parse_phandle(node, "cpu", 0);
- for (cpu = 0; dn && cpu < nr_cpu_ids; cpu++) {
- if (dn == of_get_cpu_node(cpu, NULL)) {
- pdata->cpu = cpu;
- break;
- }
- }
- of_node_put(dn);
+ pdata->cpu = of_coresight_get_cpu(node);
return pdata;
}
static struct bus_type intel_th_bus = {
.name = "intel_th",
- .dev_attrs = NULL,
.match = intel_th_match,
.probe = intel_th_probe,
.remove = intel_th_remove,
* the first read operation, otherwise the first read cost
* one extra clock cycle.
*/
- temp = readb(i2c_imx->base + IMX_I2C_I2CR);
+ temp = imx_i2c_read_reg(i2c_imx, IMX_I2C_I2CR);
temp |= I2CR_MTX;
- writeb(temp, i2c_imx->base + IMX_I2C_I2CR);
+ imx_i2c_write_reg(temp, i2c_imx, IMX_I2C_I2CR);
}
msgs->buf[msgs->len-1] = imx_i2c_read_reg(i2c_imx, IMX_I2C_I2DR);
* the first read operation, otherwise the first read cost
* one extra clock cycle.
*/
- temp = readb(i2c_imx->base + IMX_I2C_I2CR);
+ temp = imx_i2c_read_reg(i2c_imx, IMX_I2C_I2CR);
temp |= I2CR_MTX;
- writeb(temp, i2c_imx->base + IMX_I2C_I2CR);
+ imx_i2c_write_reg(temp, i2c_imx, IMX_I2C_I2CR);
}
} else if (i == (msgs->len - 2)) {
dev_dbg(&i2c_imx->adapter.dev,
This driver can also be built as a module. If so, the module
will be called i2c-mux-gpio.
+config I2C_MUX_GPMUX
+ tristate "General Purpose I2C multiplexer"
+ select MULTIPLEXER
+ depends on OF || COMPILE_TEST
+ help
+ If you say yes to this option, support will be included for a
+ general purpose I2C multiplexer. This driver provides access to
+ I2C busses connected through a MUX, which in turn is controlled
+ by a MUX-controller from the MUX subsystem.
+
+ This driver can also be built as a module. If so, the module
+ will be called i2c-mux-gpmux.
+
config I2C_MUX_LTC4306
tristate "LTC LTC4306/5 I2C multiplexer"
select GPIOLIB
obj-$(CONFIG_I2C_DEMUX_PINCTRL) += i2c-demux-pinctrl.o
obj-$(CONFIG_I2C_MUX_GPIO) += i2c-mux-gpio.o
+obj-$(CONFIG_I2C_MUX_GPMUX) += i2c-mux-gpmux.o
obj-$(CONFIG_I2C_MUX_LTC4306) += i2c-mux-ltc4306.o
obj-$(CONFIG_I2C_MUX_MLXCPLD) += i2c-mux-mlxcpld.o
obj-$(CONFIG_I2C_MUX_PCA9541) += i2c-mux-pca9541.o
--- /dev/null
+/*
+ * General Purpose I2C multiplexer
+ *
+ * Copyright (C) 2017 Axentia Technologies AB
+ *
+ * Author: Peter Rosin <peda@axentia.se>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/i2c.h>
+#include <linux/i2c-mux.h>
+#include <linux/module.h>
+#include <linux/mux/consumer.h>
+#include <linux/of_device.h>
+#include <linux/platform_device.h>
+
+struct mux {
+ struct mux_control *control;
+
+ bool do_not_deselect;
+};
+
+static int i2c_mux_select(struct i2c_mux_core *muxc, u32 chan)
+{
+ struct mux *mux = i2c_mux_priv(muxc);
+ int ret;
+
+ ret = mux_control_select(mux->control, chan);
+ mux->do_not_deselect = ret < 0;
+
+ return ret;
+}
+
+static int i2c_mux_deselect(struct i2c_mux_core *muxc, u32 chan)
+{
+ struct mux *mux = i2c_mux_priv(muxc);
+
+ if (mux->do_not_deselect)
+ return 0;
+
+ return mux_control_deselect(mux->control);
+}
+
+static struct i2c_adapter *mux_parent_adapter(struct device *dev)
+{
+ struct device_node *np = dev->of_node;
+ struct device_node *parent_np;
+ struct i2c_adapter *parent;
+
+ parent_np = of_parse_phandle(np, "i2c-parent", 0);
+ if (!parent_np) {
+ dev_err(dev, "Cannot parse i2c-parent\n");
+ return ERR_PTR(-ENODEV);
+ }
+ parent = of_find_i2c_adapter_by_node(parent_np);
+ of_node_put(parent_np);
+ if (!parent)
+ return ERR_PTR(-EPROBE_DEFER);
+
+ return parent;
+}
+
+static const struct of_device_id i2c_mux_of_match[] = {
+ { .compatible = "i2c-mux", },
+ {},
+};
+MODULE_DEVICE_TABLE(of, i2c_mux_of_match);
+
+static int i2c_mux_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct device_node *np = dev->of_node;
+ struct device_node *child;
+ struct i2c_mux_core *muxc;
+ struct mux *mux;
+ struct i2c_adapter *parent;
+ int children;
+ int ret;
+
+ if (!np)
+ return -ENODEV;
+
+ mux = devm_kzalloc(dev, sizeof(*mux), GFP_KERNEL);
+ if (!mux)
+ return -ENOMEM;
+
+ mux->control = devm_mux_control_get(dev, NULL);
+ if (IS_ERR(mux->control)) {
+ if (PTR_ERR(mux->control) != -EPROBE_DEFER)
+ dev_err(dev, "failed to get control-mux\n");
+ return PTR_ERR(mux->control);
+ }
+
+ parent = mux_parent_adapter(dev);
+ if (IS_ERR(parent)) {
+ if (PTR_ERR(parent) != -EPROBE_DEFER)
+ dev_err(dev, "failed to get i2c-parent adapter\n");
+ return PTR_ERR(parent);
+ }
+
+ children = of_get_child_count(np);
+
+ muxc = i2c_mux_alloc(parent, dev, children, 0, 0,
+ i2c_mux_select, i2c_mux_deselect);
+ if (!muxc) {
+ ret = -ENOMEM;
+ goto err_parent;
+ }
+ muxc->priv = mux;
+
+ platform_set_drvdata(pdev, muxc);
+
+ muxc->mux_locked = of_property_read_bool(np, "mux-locked");
+
+ for_each_child_of_node(np, child) {
+ u32 chan;
+
+ ret = of_property_read_u32(child, "reg", &chan);
+ if (ret < 0) {
+ dev_err(dev, "no reg property for node '%s'\n",
+ child->name);
+ goto err_children;
+ }
+
+ if (chan >= mux_control_states(mux->control)) {
+ dev_err(dev, "invalid reg %u\n", chan);
+ ret = -EINVAL;
+ goto err_children;
+ }
+
+ ret = i2c_mux_add_adapter(muxc, 0, chan, 0);
+ if (ret)
+ goto err_children;
+ }
+
+ dev_info(dev, "%d-port mux on %s adapter\n", children, parent->name);
+
+ return 0;
+
+err_children:
+ i2c_mux_del_adapters(muxc);
+err_parent:
+ i2c_put_adapter(parent);
+
+ return ret;
+}
+
+static int i2c_mux_remove(struct platform_device *pdev)
+{
+ struct i2c_mux_core *muxc = platform_get_drvdata(pdev);
+
+ i2c_mux_del_adapters(muxc);
+ i2c_put_adapter(muxc->parent);
+
+ return 0;
+}
+
+static struct platform_driver i2c_mux_driver = {
+ .probe = i2c_mux_probe,
+ .remove = i2c_mux_remove,
+ .driver = {
+ .name = "i2c-mux-gpmux",
+ .of_match_table = i2c_mux_of_match,
+ },
+};
+module_platform_driver(i2c_mux_driver);
+
+MODULE_DESCRIPTION("General Purpose I2C multiplexer driver");
+MODULE_AUTHOR("Peter Rosin <peda@axentia.se>");
+MODULE_LICENSE("GPL v2");
int error;
rq = blk_get_request(drive->queue, REQ_OP_DRV_IN, __GFP_RECLAIM);
- scsi_req_init(rq);
ide_req(rq)->type = ATA_PRIV_MISC;
rq->special = (char *)pc;
memset(sense, 0, sizeof(*sense));
blk_rq_init(rq->q, sense_rq);
- scsi_req_init(sense_rq);
+ scsi_req_init(req);
err = blk_rq_map_kern(drive->queue, sense_rq, sense, sense_len,
GFP_NOIO);
ide_requeue_and_plug(drive, failed_rq);
if (ide_queue_sense_rq(drive, pc)) {
blk_start_request(failed_rq);
- ide_complete_rq(drive, -EIO, blk_rq_bytes(failed_rq));
+ ide_complete_rq(drive, BLK_STS_IOERR, blk_rq_bytes(failed_rq));
}
}
EXPORT_SYMBOL_GPL(ide_retry_pc);
/* No more interrupts */
if ((stat & ATA_DRQ) == 0) {
- int uptodate, error;
+ int uptodate;
+ blk_status_t error;
debug_log("Packet command completed, %d bytes transferred\n",
blk_rq_bytes(rq));
if (ata_misc_request(rq)) {
scsi_req(rq)->result = 0;
- error = 0;
+ error = BLK_STS_OK;
} else {
if (blk_rq_is_passthrough(rq) && uptodate <= 0) {
scsi_req(rq)->result = -EIO;
}
- error = uptodate ? 0 : -EIO;
+ error = uptodate ? BLK_STS_OK : BLK_STS_IOERR;
}
ide_complete_rq(drive, error, blk_rq_bytes(rq));
scsi_req(failed)->sense_len = scsi_req(rq)->sense_len;
cdrom_analyze_sense_data(drive, failed);
- if (ide_end_rq(drive, failed, -EIO, blk_rq_bytes(failed)))
+ if (ide_end_rq(drive, failed, BLK_STS_IOERR, blk_rq_bytes(failed)))
BUG();
} else
cdrom_analyze_sense_data(drive, NULL);
rq = blk_get_request(drive->queue,
write ? REQ_OP_DRV_OUT : REQ_OP_DRV_IN, __GFP_RECLAIM);
- scsi_req_init(rq);
memcpy(scsi_req(rq)->cmd, cmd, BLK_MAX_CDB);
ide_req(rq)->type = ATA_PRIV_PC;
rq->rq_flags |= rq_flags;
nr_bytes -= cmd->last_xfer_len;
if (nr_bytes > 0) {
- ide_complete_rq(drive, 0, nr_bytes);
+ ide_complete_rq(drive, BLK_STS_OK, nr_bytes);
return true;
}
out_end:
if (blk_rq_is_scsi(rq) && rc == 0) {
scsi_req(rq)->resid_len = 0;
- blk_end_request_all(rq, 0);
+ blk_end_request_all(rq, BLK_STS_OK);
hwif->rq = NULL;
} else {
if (sense && uptodate)
scsi_req(rq)->resid_len += cmd->last_xfer_len;
}
- ide_complete_rq(drive, uptodate ? 0 : -EIO, blk_rq_bytes(rq));
+ ide_complete_rq(drive, uptodate ? BLK_STS_OK : BLK_STS_IOERR, blk_rq_bytes(rq));
if (sense && rc == 2)
ide_error(drive, "request sense failure", stat);
if (nsectors == 0)
nsectors = 1;
- ide_complete_rq(drive, uptodate ? 0 : -EIO, nsectors << 9);
+ ide_complete_rq(drive, uptodate ? BLK_STS_OK : BLK_STS_IOERR, nsectors << 9);
return ide_stopped;
}
int ret;
rq = blk_get_request(drive->queue, REQ_OP_DRV_IN, __GFP_RECLAIM);
- scsi_req_init(rq);
ide_req(rq)->type = ATA_PRIV_MISC;
rq->rq_flags = RQF_QUIET;
blk_execute_rq(drive->queue, cd->disk, rq, 0);
return setting->set(drive, arg);
rq = blk_get_request(q, REQ_OP_DRV_IN, __GFP_RECLAIM);
- scsi_req_init(rq);
ide_req(rq)->type = ATA_PRIV_MISC;
scsi_req(rq)->cmd_len = 5;
scsi_req(rq)->cmd[0] = REQ_DEVSET_EXEC;
return -EBUSY;
rq = blk_get_request(drive->queue, REQ_OP_DRV_IN, __GFP_RECLAIM);
- scsi_req_init(rq);
ide_req(rq)->type = ATA_PRIV_TASKFILE;
drive->mult_req = arg;
if ((cmd->tf_flags & IDE_TFLAG_FS) == 0)
ide_finish_cmd(drive, cmd, stat);
else
- ide_complete_rq(drive, 0,
+ ide_complete_rq(drive, BLK_STS_OK,
blk_rq_sectors(cmd->rq) << 9);
return ide_stopped;
}
return ide_stopped;
}
scsi_req(rq)->result = err;
- ide_complete_rq(drive, err ? -EIO : 0, blk_rq_bytes(rq));
+ ide_complete_rq(drive, err ? BLK_STS_IOERR : BLK_STS_OK, blk_rq_bytes(rq));
return ide_stopped;
}
}
EXPORT_SYMBOL_GPL(ide_error);
-static inline void ide_complete_drive_reset(ide_drive_t *drive, int err)
+static inline void ide_complete_drive_reset(ide_drive_t *drive, blk_status_t err)
{
struct request *rq = drive->hwif->rq;
scsi_req(rq)->cmd[0] == REQ_DRIVE_RESET) {
if (err <= 0 && scsi_req(rq)->result == 0)
scsi_req(rq)->result = -EIO;
- ide_complete_rq(drive, err ? err : 0, blk_rq_bytes(rq));
+ ide_complete_rq(drive, err, blk_rq_bytes(rq));
}
}
}
/* done polling */
hwif->polling = 0;
- ide_complete_drive_reset(drive, 0);
+ ide_complete_drive_reset(drive, BLK_STS_OK);
return ide_stopped;
}
ide_hwif_t *hwif = drive->hwif;
const struct ide_port_ops *port_ops = hwif->port_ops;
u8 tmp;
- int err = 0;
+ blk_status_t err = BLK_STS_OK;
if (port_ops && port_ops->reset_poll) {
err = port_ops->reset_poll(drive);
printk(KERN_ERR "%s: reset timed-out, status=0x%02x\n",
hwif->name, tmp);
drive->failures++;
- err = -EIO;
+ err = BLK_STS_IOERR;
} else {
tmp = ide_read_error(drive);
} else {
ide_reset_report_error(hwif, tmp);
drive->failures++;
- err = -EIO;
+ err = BLK_STS_IOERR;
}
}
out:
if (io_ports->ctl_addr == 0) {
spin_unlock_irqrestore(&hwif->lock, flags);
- ide_complete_drive_reset(drive, -ENXIO);
+ ide_complete_drive_reset(drive, BLK_STS_IOERR);
return ide_stopped;
}
drive->failed_pc = NULL;
drive->pc_callback(drive, 0);
- ide_complete_rq(drive, -EIO, done);
+ ide_complete_rq(drive, BLK_STS_IOERR, done);
return ide_stopped;
}
if (ata_misc_request(rq)) {
scsi_req(rq)->result = 0;
- ide_complete_rq(drive, 0, blk_rq_bytes(rq));
+ ide_complete_rq(drive, BLK_STS_OK, blk_rq_bytes(rq));
return ide_stopped;
} else
goto out_end;
drive->failed_pc = NULL;
if (blk_rq_is_passthrough(rq) && scsi_req(rq)->result == 0)
scsi_req(rq)->result = -EIO;
- ide_complete_rq(drive, -EIO, blk_rq_bytes(rq));
+ ide_complete_rq(drive, BLK_STS_IOERR, blk_rq_bytes(rq));
return ide_stopped;
}
#include <linux/uaccess.h>
#include <asm/io.h>
-int ide_end_rq(ide_drive_t *drive, struct request *rq, int error,
+int ide_end_rq(ide_drive_t *drive, struct request *rq, blk_status_t error,
unsigned int nr_bytes)
{
/*
}
}
-int ide_complete_rq(ide_drive_t *drive, int error, unsigned int nr_bytes)
+int ide_complete_rq(ide_drive_t *drive, blk_status_t error, unsigned int nr_bytes)
{
ide_hwif_t *hwif = drive->hwif;
struct request *rq = hwif->rq;
* if failfast is set on a request, override number of sectors
* and complete the whole request right now
*/
- if (blk_noretry_request(rq) && error <= 0)
+ if (blk_noretry_request(rq) && error)
nr_bytes = blk_rq_sectors(rq) << 9;
rc = ide_end_rq(drive, rq, error, nr_bytes);
scsi_req(rq)->result = -EIO;
}
- ide_complete_rq(drive, -EIO, blk_rq_bytes(rq));
+ ide_complete_rq(drive, BLK_STS_IOERR, blk_rq_bytes(rq));
}
static void ide_tf_set_specify_cmd(ide_drive_t *drive, struct ide_taskfile *tf)
printk("%s: DRIVE_CMD (null)\n", drive->name);
#endif
scsi_req(rq)->result = 0;
- ide_complete_rq(drive, 0, blk_rq_bytes(rq));
+ ide_complete_rq(drive, BLK_STS_OK, blk_rq_bytes(rq));
return ide_stopped;
}
struct request *rq;
rq = blk_get_request(drive->queue, REQ_OP_DRV_IN, __GFP_RECLAIM);
- scsi_req_init(rq);
ide_req(rq)->type = ATA_PRIV_TASKFILE;
blk_execute_rq(drive->queue, NULL, rq, 0);
err = scsi_req(rq)->result ? -EIO : 0;
int ret = 0;
rq = blk_get_request(drive->queue, REQ_OP_DRV_IN, __GFP_RECLAIM);
- scsi_req_init(rq);
ide_req(rq)->type = ATA_PRIV_MISC;
scsi_req(rq)->cmd_len = 1;
scsi_req(rq)->cmd[0] = REQ_DRIVE_RESET;
spin_unlock_irq(&hwif->lock);
rq = blk_get_request(q, REQ_OP_DRV_IN, __GFP_RECLAIM);
- scsi_req_init(rq);
scsi_req(rq)->cmd[0] = REQ_PARK_HEADS;
scsi_req(rq)->cmd_len = 1;
ide_req(rq)->type = ATA_PRIV_MISC;
* timeout has expired, so power management will be reenabled.
*/
rq = blk_get_request(q, REQ_OP_DRV_IN, GFP_NOWAIT);
- scsi_req_init(rq);
if (IS_ERR(rq))
goto out;
memset(&rqpm, 0, sizeof(rqpm));
rq = blk_get_request(drive->queue, REQ_OP_DRV_IN, __GFP_RECLAIM);
- scsi_req_init(rq);
ide_req(rq)->type = ATA_PRIV_PM_SUSPEND;
rq->special = &rqpm;
rqpm.pm_step = IDE_PM_START_SUSPEND;
return ret;
}
-static void ide_end_sync_rq(struct request *rq, int error)
+static void ide_end_sync_rq(struct request *rq, blk_status_t error)
{
complete(rq->end_io_data);
}
if (unlikely(blk_queue_dying(q))) {
rq->rq_flags |= RQF_QUIET;
scsi_req(rq)->result = -ENXIO;
- __blk_end_request_all(rq, 0);
+ __blk_end_request_all(rq, BLK_STS_OK);
spin_unlock_irq(q->queue_lock);
return -ENXIO;
}
memset(&rqpm, 0, sizeof(rqpm));
rq = blk_get_request(drive->queue, REQ_OP_DRV_IN, __GFP_RECLAIM);
- scsi_req_init(rq);
ide_req(rq)->type = ATA_PRIV_PM_RESUME;
rq->rq_flags |= RQF_PREEMPT;
rq->special = &rqpm;
drive->hwif->rq = NULL;
- if (blk_end_request(rq, 0, 0))
+ if (blk_end_request(rq, BLK_STS_OK, 0))
BUG();
}
}
}
-static int ide_init_rq(struct request_queue *q, struct request *rq, gfp_t gfp)
+static void ide_initialize_rq(struct request *rq)
{
struct ide_request *req = blk_mq_rq_to_pdu(rq);
+ scsi_req_init(&req->sreq);
req->sreq.sense = req->sense;
- return 0;
}
/*
return 1;
q->request_fn = do_ide_request;
- q->init_rq_fn = ide_init_rq;
+ q->initialize_rq_fn = ide_initialize_rq;
q->cmd_size = sizeof(struct ide_request);
+ queue_flag_set_unlocked(QUEUE_FLAG_SCSI_PASSTHROUGH, q);
if (blk_init_allocated_queue(q) < 0) {
blk_cleanup_queue(q);
return 1;
drive->failed_pc = NULL;
drive->pc_callback(drive, 0);
- ide_complete_rq(drive, -EIO, blk_rq_bytes(rq));
+ ide_complete_rq(drive, BLK_STS_IOERR, blk_rq_bytes(rq));
return ide_stopped;
}
ide_debug_log(IDE_DBG_SENSE, "retry #%d, cmd: 0x%02x", pc->retries,
BUG_ON(size < 0 || size % tape->blk_size);
rq = blk_get_request(drive->queue, REQ_OP_DRV_IN, __GFP_RECLAIM);
- scsi_req_init(rq);
ide_req(rq)->type = ATA_PRIV_MISC;
scsi_req(rq)->cmd[13] = cmd;
rq->rq_disk = tape->disk;
}
if (nr_bytes > 0)
- ide_complete_rq(drive, 0, nr_bytes);
+ ide_complete_rq(drive, BLK_STS_OK, nr_bytes);
}
}
ide_driveid_update(drive);
}
- ide_complete_rq(drive, err ? -EIO : 0, blk_rq_bytes(rq));
+ ide_complete_rq(drive, err ? BLK_STS_IOERR : BLK_STS_OK, blk_rq_bytes(rq));
}
/*
if ((cmd->tf_flags & IDE_TFLAG_FS) == 0)
ide_finish_cmd(drive, cmd, stat);
else
- ide_complete_rq(drive, 0, blk_rq_sectors(cmd->rq) << 9);
+ ide_complete_rq(drive, BLK_STS_OK, blk_rq_sectors(cmd->rq) << 9);
return ide_stopped;
out_err:
ide_error_cmd(drive, cmd);
rq = blk_get_request(drive->queue,
(cmd->tf_flags & IDE_TFLAG_WRITE) ?
REQ_OP_DRV_OUT : REQ_OP_DRV_IN, __GFP_RECLAIM);
- scsi_req_init(rq);
ide_req(rq)->type = ATA_PRIV_TASKFILE;
/*
* yet.
*/
-static int sil_sata_reset_poll(ide_drive_t *drive)
+static blk_status_t sil_sata_reset_poll(ide_drive_t *drive)
{
ide_hwif_t *hwif = drive->hwif;
void __iomem *sata_status_addr
if ((sata_stat & 0x03) != 0x03) {
printk(KERN_WARNING "%s: reset phy dead, status=0x%08x\n",
hwif->name, sata_stat);
- return -ENXIO;
+ return BLK_STS_IOERR;
}
}
- return 0;
+ return BLK_STS_OK;
}
/**
/* un-comment DEBUG to enable pr_debug() statements */
#define DEBUG
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
#include <linux/kernel.h>
#include <linux/cpuidle.h>
#include <linux/tick.h>
#include <asm/msr.h>
#define INTEL_IDLE_VERSION "0.4.1"
-#define PREFIX "intel_idle: "
static struct cpuidle_driver intel_idle_driver = {
.name = "intel_idle",
const struct x86_cpu_id *id;
if (max_cstate == 0) {
- pr_debug(PREFIX "disabled\n");
+ pr_debug("disabled\n");
return -EPERM;
}
if (!id) {
if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL &&
boot_cpu_data.x86 == 6)
- pr_debug(PREFIX "does not run on family %d model %d\n",
- boot_cpu_data.x86, boot_cpu_data.x86_model);
+ pr_debug("does not run on family %d model %d\n",
+ boot_cpu_data.x86, boot_cpu_data.x86_model);
return -ENODEV;
}
!mwait_substates)
return -ENODEV;
- pr_debug(PREFIX "MWAIT substates: 0x%x\n", mwait_substates);
+ pr_debug("MWAIT substates: 0x%x\n", mwait_substates);
icpu = (const struct idle_cpu *)id->driver_data;
cpuidle_state_table = icpu->state_table;
- pr_debug(PREFIX "v" INTEL_IDLE_VERSION
- " model 0x%X\n", boot_cpu_data.x86_model);
+ pr_debug("v" INTEL_IDLE_VERSION " model 0x%X\n",
+ boot_cpu_data.x86_model);
return 0;
}
break;
if (cstate + 1 > max_cstate) {
- printk(PREFIX "max_cstate %d reached\n",
- max_cstate);
+ pr_info("max_cstate %d reached\n", max_cstate);
break;
}
/* if state marked as disabled, skip it */
if (cpuidle_state_table[cstate].disabled != 0) {
- pr_debug(PREFIX "state %s is disabled",
- cpuidle_state_table[cstate].name);
+ pr_debug("state %s is disabled\n",
+ cpuidle_state_table[cstate].name);
continue;
}
dev->cpu = cpu;
if (cpuidle_register_device(dev)) {
- pr_debug(PREFIX "cpuidle_register_device %d failed!\n", cpu);
+ pr_debug("cpuidle_register_device %d failed!\n", cpu);
return -EIO;
}
retval = cpuidle_register_driver(&intel_idle_driver);
if (retval) {
struct cpuidle_driver *drv = cpuidle_get_driver();
- printk(KERN_DEBUG PREFIX "intel_idle yielding to %s",
- drv ? drv->name : "none");
+ printk(KERN_DEBUG pr_fmt("intel_idle yielding to %s\n"),
+ drv ? drv->name : "none");
goto init_driver_fail;
}
if (retval < 0)
goto hp_setup_fail;
- pr_debug(PREFIX "lapic_timer_reliable_states 0x%x\n",
- lapic_timer_reliable_states);
+ pr_debug("lapic_timer_reliable_states 0x%x\n",
+ lapic_timer_reliable_states);
return 0;
source "drivers/iio/imu/Kconfig"
source "drivers/iio/light/Kconfig"
source "drivers/iio/magnetometer/Kconfig"
+source "drivers/iio/multiplexer/Kconfig"
source "drivers/iio/orientation/Kconfig"
if IIO_TRIGGER
source "drivers/iio/trigger/Kconfig"
obj-y += imu/
obj-y += light/
obj-y += magnetometer/
+obj-y += multiplexer/
obj-y += orientation/
obj-y += potentiometer/
obj-y += potentiostat/
static int hid_accel_3d_probe(struct platform_device *pdev)
{
int ret = 0;
- static const char *name;
+ const char *name;
struct iio_dev *indio_dev;
struct accel_3d_state *accel_state;
const struct iio_chan_spec *channel_spec;
#define MMA9551_DRV_NAME "mma9551"
#define MMA9551_IRQ_NAME "mma9551_event"
-#define MMA9551_GPIO_NAME "mma9551_int"
#define MMA9551_GPIO_COUNT 4
/* Tilt application (inclination in IIO terms). */
struct device *dev = &data->client->dev;
for (i = 0; i < MMA9551_GPIO_COUNT; i++) {
- gpio = devm_gpiod_get_index(dev, MMA9551_GPIO_NAME, i,
- GPIOD_IN);
+ gpio = devm_gpiod_get_index(dev, NULL, i, GPIOD_IN);
if (IS_ERR(gpio)) {
dev_err(dev, "acpi gpio get index failed\n");
return PTR_ERR(gpio);
int st_accel_common_probe(struct iio_dev *indio_dev)
{
struct st_sensor_data *adata = iio_priv(indio_dev);
+ struct st_sensors_platform_data *pdata =
+ (struct st_sensors_platform_data *)adata->dev->platform_data;
int irq = adata->get_irq_data_ready(indio_dev);
int err;
&adata->sensor_settings->fs.fs_avl[0];
adata->odr = adata->sensor_settings->odr.odr_avl[0].hz;
- if (!adata->dev->platform_data)
- adata->dev->platform_data =
- (struct st_sensors_platform_data *)&default_accel_pdata;
+ if (!pdata)
+ pdata = (struct st_sensors_platform_data *)&default_accel_pdata;
err = st_sensors_init_sensor(indio_dev, adata->dev->platform_data);
if (err < 0)
}
static const struct spi_device_id st_accel_id_table[] = {
- { LSM303DLH_ACCEL_DEV_NAME },
- { LSM303DLHC_ACCEL_DEV_NAME },
{ LIS3DH_ACCEL_DEV_NAME },
{ LSM330D_ACCEL_DEV_NAME },
{ LSM330DL_ACCEL_DEV_NAME },
{ LSM330DLC_ACCEL_DEV_NAME },
{ LIS331DLH_ACCEL_DEV_NAME },
- { LSM303DL_ACCEL_DEV_NAME },
- { LSM303DLM_ACCEL_DEV_NAME },
{ LSM330_ACCEL_DEV_NAME },
{ LSM303AGR_ACCEL_DEV_NAME },
{ LIS2DH12_ACCEL_DEV_NAME },
This driver can also be built as a module. If so, the module will be
called ti-adc0832.
+config TI_ADC084S021
+ tristate "Texas Instruments ADC084S021"
+ depends on SPI
+ select IIO_BUFFER
+ select IIO_TRIGGERED_BUFFER
+ help
+ If you say yes here you get support for Texas Instruments ADC084S021
+ chips.
+
+ This driver can also be built as a module. If so, the module will be
+ called ti-adc084s021.
+
config TI_ADC12138
tristate "Texas Instruments ADC12130/ADC12132/ADC12138"
depends on SPI
This driver can also be built as a module. If so, the module will be
called ti-adc12138.
+config TI_ADC108S102
+ tristate "Texas Instruments ADC108S102 and ADC128S102 driver"
+ depends on SPI
+ select IIO_BUFFER
+ select IIO_TRIGGERED_BUFFER
+ help
+ Say yes here to build support for Texas Instruments ADC108S102 and
+ ADC128S102 ADC.
+
+ To compile this driver as a module, choose M here: the module will
+ be called ti-adc108s102.
+
config TI_ADC128S052
tristate "Texas Instruments ADC128S052/ADC122S021/ADC124S021"
depends on SPI
obj-$(CONFIG_STM32_ADC) += stm32-adc.o
obj-$(CONFIG_TI_ADC081C) += ti-adc081c.o
obj-$(CONFIG_TI_ADC0832) += ti-adc0832.o
+obj-$(CONFIG_TI_ADC084S021) += ti-adc084s021.o
obj-$(CONFIG_TI_ADC12138) += ti-adc12138.o
+obj-$(CONFIG_TI_ADC108S102) += ti-adc108s102.o
obj-$(CONFIG_TI_ADC128S052) += ti-adc128s052.o
obj-$(CONFIG_TI_ADC161S626) += ti-adc161s626.o
obj-$(CONFIG_TI_ADS1015) += ti-ads1015.o
struct ad7791_state *st = iio_priv(indio_dev);
int i, ret;
- for (i = 0; i < ARRAY_SIZE(ad7791_sample_freq_avail); i++)
- if (sysfs_streq(ad7791_sample_freq_avail[i], buf))
- break;
- if (i == ARRAY_SIZE(ad7791_sample_freq_avail))
- return -EINVAL;
+ i = sysfs_match_string(ad7791_sample_freq_avail, buf);
+ if (i < 0)
+ return i;
ret = iio_device_claim_direct_mode(indio_dev);
if (ret)
}
/* Start all channels in normal mode. */
- clk_prepare_enable(data->clk_scaler->clk);
+ ret = clk_prepare_enable(data->clk_scaler->clk);
+ if (ret)
+ goto clk_enable_error;
+
adc_engine_control_reg_val = GENMASK(31, 16) |
ASPEED_OPERATION_MODE_NORMAL | ASPEED_ENGINE_ENABLE;
writel(adc_engine_control_reg_val,
writel(ASPEED_OPERATION_MODE_POWER_DOWN,
data->base + ASPEED_REG_ENGINE_CONTROL);
clk_disable_unprepare(data->clk_scaler->clk);
+clk_enable_error:
clk_hw_unregister_divider(data->clk_scaler);
scaler_error:
#define CPCAP_BIT_RAND0 BIT(1) /* Set with CAL_MODE */
#define CPCAP_BIT_ADEN BIT(0) /* Currently unused */
+#define CPCAP_REG_ADCC1_DEFAULTS (CPCAP_BIT_ADEN_AUTO_CLR | \
+ CPCAP_BIT_ADC_CLK_SEL0 | \
+ CPCAP_BIT_RAND1)
+
/* Register CPCAP_REG_ADCC2 bits */
#define CPCAP_BIT_CAL_FACTOR_ENABLE BIT(15) /* Currently unused */
#define CPCAP_BIT_BATDETB_EN BIT(14) /* Currently unused */
#define CPCAP_BIT_ADC_PS_FACTOR0 BIT(9)
#define CPCAP_BIT_AD4_SELECT BIT(8) /* Currently unused */
#define CPCAP_BIT_ADC_BUSY BIT(7) /* Currently unused */
-#define CPCAP_BIT_THERMBIAS_EN BIT(6) /* Currently unused */
+#define CPCAP_BIT_THERMBIAS_EN BIT(6) /* Bias for AD0_BATTDETB */
#define CPCAP_BIT_ADTRIG_DIS BIT(5) /* Disable interrupt */
#define CPCAP_BIT_LIADC BIT(4) /* Currently unused */
#define CPCAP_BIT_TS_REFEN BIT(3) /* Currently unused */
#define CPCAP_BIT_TS_M1 BIT(1) /* Currently unused */
#define CPCAP_BIT_TS_M0 BIT(0) /* Currently unused */
+#define CPCAP_REG_ADCC2_DEFAULTS (CPCAP_BIT_AD4_SELECT | \
+ CPCAP_BIT_ADTRIG_DIS | \
+ CPCAP_BIT_LIADC | \
+ CPCAP_BIT_TS_M2 | \
+ CPCAP_BIT_TS_M1)
+
#define CPCAP_MAX_TEMP_LVL 27
#define CPCAP_FOUR_POINT_TWO_ADC 801
#define ST_ADC_CAL_CHRGI_HIGH_THRESHOLD 530
#define ST_ADC_CAL_BATTI_LOW_THRESHOLD 494
#define ST_ADC_CALIBRATE_DIFF_THRESHOLD 3
-#define CPCAP_ADC_MAX_RETRIES 5 /* Calibration and quirk */
+#define CPCAP_ADC_MAX_RETRIES 5 /* Calibration */
/**
* struct cpcap_adc_ato - timing settings for cpcap adc
*/
enum cpcap_adc_channel {
/* Bank0 channels */
- CPCAP_ADC_AD0_BATTDETB, /* Battery detection */
+ CPCAP_ADC_AD0, /* Battery temperature */
CPCAP_ADC_BATTP, /* Battery voltage */
CPCAP_ADC_VBUS, /* USB VBUS voltage */
- CPCAP_ADC_AD3, /* Battery temperature when charging */
+ CPCAP_ADC_AD3, /* Die temperature when charging */
CPCAP_ADC_BPLUS_AD4, /* Another battery or system voltage */
CPCAP_ADC_CHG_ISENSE, /* Calibrated charge current */
CPCAP_ADC_BATTI, /* Calibrated system current */
/* Phasing table for channels. Note that channels 16 & 17 use BATTP and BATTI */
static const struct cpcap_adc_phasing_tbl bank_phasing[] = {
/* Bank0 */
- [CPCAP_ADC_AD0_BATTDETB] = {0, 0x80, 0x80, 0, 1023},
+ [CPCAP_ADC_AD0] = {0, 0x80, 0x80, 0, 1023},
[CPCAP_ADC_BATTP] = {0, 0x80, 0x80, 0, 1023},
[CPCAP_ADC_VBUS] = {0, 0x80, 0x80, 0, 1023},
[CPCAP_ADC_AD3] = {0, 0x80, 0x80, 0, 1023},
*/
static struct cpcap_adc_conversion_tbl bank_conversion[] = {
/* Bank0 */
- [CPCAP_ADC_AD0_BATTDETB] = {
+ [CPCAP_ADC_AD0] = {
IIO_CHAN_INFO_PROCESSED, 0, 0, 0, 1, 1,
},
[CPCAP_ADC_BATTP] = {
return;
switch (req->channel) {
+ case CPCAP_ADC_AD0:
+ value2 |= CPCAP_BIT_THERMBIAS_EN;
+ error = regmap_update_bits(ddata->reg, CPCAP_REG_ADCC2,
+ CPCAP_BIT_THERMBIAS_EN,
+ value2);
+ if (error)
+ return;
+ usleep_range(800, 1000);
+ break;
case CPCAP_ADC_AD8 ... CPCAP_ADC_TSY2_AD15:
value1 |= CPCAP_BIT_AD_SEL1;
break;
error = regmap_update_bits(ddata->reg, CPCAP_REG_ADCC2,
CPCAP_BIT_ATOX_PS_FACTOR |
CPCAP_BIT_ADC_PS_FACTOR1 |
- CPCAP_BIT_ADC_PS_FACTOR0,
+ CPCAP_BIT_ADC_PS_FACTOR0 |
+ CPCAP_BIT_THERMBIAS_EN,
value2);
if (error)
return;
}
}
-/*
- * Occasionally the ADC does not seem to start and there will be no
- * interrupt. Let's re-init interrupt to prevent the ADC from hanging
- * for the next request. It is unclear why this happens, but the next
- * request will usually work after doing this.
- */
-static void cpcap_adc_quirk_reset_lost_irq(struct cpcap_adc *ddata)
-{
- int error;
-
- dev_info(ddata->dev, "lost ADC irq, attempting to reinit\n");
- disable_irq(ddata->irq);
- error = regmap_update_bits(ddata->reg, CPCAP_REG_ADCC2,
- CPCAP_BIT_ADTRIG_DIS,
- CPCAP_BIT_ADTRIG_DIS);
- if (error)
- dev_warn(ddata->dev, "%s reset failed: %i\n",
- __func__, error);
- enable_irq(ddata->irq);
-}
-
static int cpcap_adc_start_bank(struct cpcap_adc *ddata,
struct cpcap_adc_request *req)
{
return 0;
if (error == 0) {
- cpcap_adc_quirk_reset_lost_irq(ddata);
error = -ETIMEDOUT;
continue;
}
return error;
}
+static int cpcap_adc_stop_bank(struct cpcap_adc *ddata)
+{
+ int error;
+
+ error = regmap_update_bits(ddata->reg, CPCAP_REG_ADCC1,
+ 0xffff,
+ CPCAP_REG_ADCC1_DEFAULTS);
+ if (error)
+ return error;
+
+ return regmap_update_bits(ddata->reg, CPCAP_REG_ADCC2,
+ 0xffff,
+ CPCAP_REG_ADCC2_DEFAULTS);
+}
+
static void cpcap_adc_phase(struct cpcap_adc_request *req)
{
const struct cpcap_adc_conversion_tbl *conv_tbl = req->conv_tbl;
return;
/* Temperatures use a lookup table instead of conversion table */
- if ((req->channel == CPCAP_ADC_AD0_BATTDETB) ||
+ if ((req->channel == CPCAP_ADC_AD0) ||
(req->channel == CPCAP_ADC_AD3)) {
req->result =
cpcap_adc_table_to_millicelcius(req->result);
req->conv_tbl = bank_conversion;
switch (channel) {
- case CPCAP_ADC_AD0_BATTDETB ... CPCAP_ADC_USB_ID:
+ case CPCAP_ADC_AD0 ... CPCAP_ADC_USB_ID:
req->bank_index = channel;
break;
case CPCAP_ADC_AD8 ... CPCAP_ADC_TSY2_AD15:
return 0;
}
+static int cpcap_adc_read_st_die_temp(struct cpcap_adc *ddata,
+ int addr, int *val)
+{
+ int error;
+
+ error = regmap_read(ddata->reg, addr, val);
+ if (error)
+ return error;
+
+ *val -= 282;
+ *val *= 114;
+ *val += 25000;
+
+ return 0;
+}
+
static int cpcap_adc_read(struct iio_dev *indio_dev,
struct iio_chan_spec const *chan,
int *val, int *val2, long mask)
if (error)
goto err_unlock;
error = regmap_read(ddata->reg, chan->address, val);
+ if (error)
+ goto err_unlock;
+ error = cpcap_adc_stop_bank(ddata);
if (error)
goto err_unlock;
mutex_unlock(&ddata->lock);
error = cpcap_adc_start_bank(ddata, &req);
if (error)
goto err_unlock;
- error = cpcap_adc_read_bank_scaled(ddata, &req);
+ if ((ddata->vendor == CPCAP_VENDOR_ST) &&
+ (chan->channel == CPCAP_ADC_AD3)) {
+ error = cpcap_adc_read_st_die_temp(ddata,
+ chan->address,
+ &req.result);
+ if (error)
+ goto err_unlock;
+ } else {
+ error = cpcap_adc_read_bank_scaled(ddata, &req);
+ if (error)
+ goto err_unlock;
+ }
+ error = cpcap_adc_stop_bank(ddata);
if (error)
goto err_unlock;
mutex_unlock(&ddata->lock);
return spi_write(priv->spi, priv->reg_buffer, 3);
}
+static int hi8435_read_raw(struct iio_dev *idev,
+ const struct iio_chan_spec *chan,
+ int *val, int *val2, long mask)
+{
+ struct hi8435_priv *priv = iio_priv(idev);
+ u32 tmp;
+ int ret;
+
+ switch (mask) {
+ case IIO_CHAN_INFO_RAW:
+ ret = hi8435_readl(priv, HI8435_SO31_0_REG, &tmp);
+ if (ret < 0)
+ return ret;
+ *val = !!(tmp & BIT(chan->channel));
+ return IIO_VAL_INT;
+ default:
+ return -EINVAL;
+ }
+}
+
static int hi8435_read_event_config(struct iio_dev *idev,
const struct iio_chan_spec *chan,
enum iio_event_type type,
enum iio_event_direction dir, int state)
{
struct hi8435_priv *priv = iio_priv(idev);
+ int ret;
+ u32 tmp;
+
+ if (state) {
+ ret = hi8435_readl(priv, HI8435_SO31_0_REG, &tmp);
+ if (ret < 0)
+ return ret;
+ if (tmp & BIT(chan->channel))
+ priv->event_prev_val |= BIT(chan->channel);
+ else
+ priv->event_prev_val &= ~BIT(chan->channel);
- priv->event_scan_mask &= ~BIT(chan->channel);
- if (state)
priv->event_scan_mask |= BIT(chan->channel);
+ } else
+ priv->event_scan_mask &= ~BIT(chan->channel);
return 0;
}
static const struct iio_chan_spec_ext_info hi8435_ext_info[] = {
IIO_ENUM("sensing_mode", IIO_SEPARATE, &hi8435_sensing_mode),
+ IIO_ENUM_AVAILABLE("sensing_mode", &hi8435_sensing_mode),
{},
};
.type = IIO_VOLTAGE, \
.indexed = 1, \
.channel = num, \
+ .info_mask_separate = BIT(IIO_CHAN_INFO_RAW), \
.event_spec = hi8435_events, \
.num_event_specs = ARRAY_SIZE(hi8435_events), \
.ext_info = hi8435_ext_info, \
static const struct iio_info hi8435_info = {
.driver_module = THIS_MODULE,
- .read_event_config = &hi8435_read_event_config,
+ .read_raw = hi8435_read_raw,
+ .read_event_config = hi8435_read_event_config,
.write_event_config = hi8435_write_event_config,
- .read_event_value = &hi8435_read_event_value,
- .write_event_value = &hi8435_write_event_value,
- .debugfs_reg_access = &hi8435_debugfs_reg_access,
+ .read_event_value = hi8435_read_event_value,
+ .write_event_value = hi8435_write_event_value,
+ .debugfs_reg_access = hi8435_debugfs_reg_access,
};
static void hi8435_iio_push_event(struct iio_dev *idev, unsigned int val)
#define INA2XX_CURRENT 0x04 /* readonly */
#define INA2XX_CALIBRATION 0x05
-#define INA226_ALERT_MASK GENMASK(2, 1)
-#define INA266_CVRF BIT(3)
+#define INA226_MASK_ENABLE 0x06
+#define INA226_CVRF BIT(3)
#define INA2XX_MAX_REGISTERS 8
/* settings - depend on use case */
#define INA219_CONFIG_DEFAULT 0x399F /* PGA=8 */
+#define INA219_DEFAULT_IT 532
#define INA226_CONFIG_DEFAULT 0x4327
#define INA226_DEFAULT_AVG 4
#define INA226_DEFAULT_IT 1110
#define INA2XX_RSHUNT_DEFAULT 10000
/*
- * bit mask for reading the averaging setting in the configuration register
+ * bit masks for reading the settings in the configuration register
* FIXME: use regmap_fields.
*/
#define INA2XX_MODE_MASK GENMASK(3, 0)
+/* Averaging for VBus/VShunt/Power */
#define INA226_AVG_MASK GENMASK(11, 9)
#define INA226_SHIFT_AVG(val) ((val) << 9)
/* Integration time for VBus */
+#define INA219_ITB_MASK GENMASK(10, 7)
+#define INA219_SHIFT_ITB(val) ((val) << 7)
#define INA226_ITB_MASK GENMASK(8, 6)
#define INA226_SHIFT_ITB(val) ((val) << 6)
/* Integration time for VShunt */
+#define INA219_ITS_MASK GENMASK(6, 3)
+#define INA219_SHIFT_ITS(val) ((val) << 3)
#define INA226_ITS_MASK GENMASK(5, 3)
#define INA226_SHIFT_ITS(val) ((val) << 3)
int bus_voltage_shift;
int bus_voltage_lsb; /* uV */
int power_lsb; /* uW */
+ enum ina2xx_ids chip_id;
};
struct ina2xx_chip_info {
.bus_voltage_shift = 3,
.bus_voltage_lsb = 4000,
.power_lsb = 20000,
+ .chip_id = ina219,
},
[ina226] = {
.config_default = INA226_CONFIG_DEFAULT,
.bus_voltage_shift = 0,
.bus_voltage_lsb = 1250,
.power_lsb = 25000,
+ .chip_id = ina226,
},
};
return 0;
}
+/* Conversion times in uS. */
+static const int ina219_conv_time_tab_subsample[] = { 84, 148, 276, 532 };
+static const int ina219_conv_time_tab_average[] = { 532, 1060, 2130, 4260,
+ 8510, 17020, 34050, 68100};
+
+static int ina219_lookup_int_time(unsigned int *val_us, int *bits)
+{
+ if (*val_us > 68100 || *val_us < 84)
+ return -EINVAL;
+
+ if (*val_us <= 532) {
+ *bits = find_closest(*val_us, ina219_conv_time_tab_subsample,
+ ARRAY_SIZE(ina219_conv_time_tab_subsample));
+ *val_us = ina219_conv_time_tab_subsample[*bits];
+ } else {
+ *bits = find_closest(*val_us, ina219_conv_time_tab_average,
+ ARRAY_SIZE(ina219_conv_time_tab_average));
+ *val_us = ina219_conv_time_tab_average[*bits];
+ *bits |= 0x8;
+ }
+
+ return 0;
+}
+
+static int ina219_set_int_time_vbus(struct ina2xx_chip_info *chip,
+ unsigned int val_us, unsigned int *config)
+{
+ int bits, ret;
+ unsigned int val_us_best = val_us;
+
+ ret = ina219_lookup_int_time(&val_us_best, &bits);
+ if (ret)
+ return ret;
+
+ chip->int_time_vbus = val_us_best;
+
+ *config &= ~INA219_ITB_MASK;
+ *config |= INA219_SHIFT_ITB(bits) & INA219_ITB_MASK;
+
+ return 0;
+}
+
+static int ina219_set_int_time_vshunt(struct ina2xx_chip_info *chip,
+ unsigned int val_us, unsigned int *config)
+{
+ int bits, ret;
+ unsigned int val_us_best = val_us;
+
+ ret = ina219_lookup_int_time(&val_us_best, &bits);
+ if (ret)
+ return ret;
+
+ chip->int_time_vshunt = val_us_best;
+
+ *config &= ~INA219_ITS_MASK;
+ *config |= INA219_SHIFT_ITS(bits) & INA219_ITS_MASK;
+
+ return 0;
+}
+
static int ina2xx_write_raw(struct iio_dev *indio_dev,
struct iio_chan_spec const *chan,
int val, int val2, long mask)
break;
case IIO_CHAN_INFO_INT_TIME:
- if (chan->address == INA2XX_SHUNT_VOLTAGE)
- ret = ina226_set_int_time_vshunt(chip, val2, &tmp);
- else
- ret = ina226_set_int_time_vbus(chip, val2, &tmp);
+ if (chip->config->chip_id == ina226) {
+ if (chan->address == INA2XX_SHUNT_VOLTAGE)
+ ret = ina226_set_int_time_vshunt(chip, val2,
+ &tmp);
+ else
+ ret = ina226_set_int_time_vbus(chip, val2,
+ &tmp);
+ } else {
+ if (chan->address == INA2XX_SHUNT_VOLTAGE)
+ ret = ina219_set_int_time_vshunt(chip, val2,
+ &tmp);
+ else
+ ret = ina219_set_int_time_vbus(chip, val2,
+ &tmp);
+ }
break;
default:
return len;
}
-#define INA2XX_CHAN(_type, _index, _address) { \
+#define INA219_CHAN(_type, _index, _address) { \
+ .type = (_type), \
+ .address = (_address), \
+ .indexed = 1, \
+ .channel = (_index), \
+ .info_mask_separate = BIT(IIO_CHAN_INFO_RAW) | \
+ BIT(IIO_CHAN_INFO_SCALE), \
+ .info_mask_shared_by_dir = BIT(IIO_CHAN_INFO_SAMP_FREQ), \
+ .scan_index = (_index), \
+ .scan_type = { \
+ .sign = 'u', \
+ .realbits = 16, \
+ .storagebits = 16, \
+ .endianness = IIO_CPU, \
+ } \
+}
+
+#define INA226_CHAN(_type, _index, _address) { \
.type = (_type), \
.address = (_address), \
.indexed = 1, \
.channel = (_index), \
- .info_mask_separate = BIT(IIO_CHAN_INFO_RAW) \
- | BIT(IIO_CHAN_INFO_SCALE), \
+ .info_mask_separate = BIT(IIO_CHAN_INFO_RAW) | \
+ BIT(IIO_CHAN_INFO_SCALE), \
.info_mask_shared_by_dir = BIT(IIO_CHAN_INFO_SAMP_FREQ) | \
BIT(IIO_CHAN_INFO_OVERSAMPLING_RATIO), \
.scan_index = (_index), \
* Sampling Freq is a consequence of the integration times of
* the Voltage channels.
*/
-#define INA2XX_CHAN_VOLTAGE(_index, _address) { \
+#define INA219_CHAN_VOLTAGE(_index, _address) { \
+ .type = IIO_VOLTAGE, \
+ .address = (_address), \
+ .indexed = 1, \
+ .channel = (_index), \
+ .info_mask_separate = BIT(IIO_CHAN_INFO_RAW) | \
+ BIT(IIO_CHAN_INFO_SCALE) | \
+ BIT(IIO_CHAN_INFO_INT_TIME), \
+ .info_mask_shared_by_dir = BIT(IIO_CHAN_INFO_SAMP_FREQ), \
+ .scan_index = (_index), \
+ .scan_type = { \
+ .sign = 'u', \
+ .realbits = 16, \
+ .storagebits = 16, \
+ .endianness = IIO_LE, \
+ } \
+}
+
+#define INA226_CHAN_VOLTAGE(_index, _address) { \
.type = IIO_VOLTAGE, \
.address = (_address), \
.indexed = 1, \
.info_mask_separate = BIT(IIO_CHAN_INFO_RAW) | \
BIT(IIO_CHAN_INFO_SCALE) | \
BIT(IIO_CHAN_INFO_INT_TIME), \
+ .info_mask_shared_by_dir = BIT(IIO_CHAN_INFO_SAMP_FREQ) | \
+ BIT(IIO_CHAN_INFO_OVERSAMPLING_RATIO), \
.scan_index = (_index), \
.scan_type = { \
.sign = 'u', \
} \
}
-static const struct iio_chan_spec ina2xx_channels[] = {
- INA2XX_CHAN_VOLTAGE(0, INA2XX_SHUNT_VOLTAGE),
- INA2XX_CHAN_VOLTAGE(1, INA2XX_BUS_VOLTAGE),
- INA2XX_CHAN(IIO_POWER, 2, INA2XX_POWER),
- INA2XX_CHAN(IIO_CURRENT, 3, INA2XX_CURRENT),
+
+static const struct iio_chan_spec ina226_channels[] = {
+ INA226_CHAN_VOLTAGE(0, INA2XX_SHUNT_VOLTAGE),
+ INA226_CHAN_VOLTAGE(1, INA2XX_BUS_VOLTAGE),
+ INA226_CHAN(IIO_POWER, 2, INA2XX_POWER),
+ INA226_CHAN(IIO_CURRENT, 3, INA2XX_CURRENT),
+ IIO_CHAN_SOFT_TIMESTAMP(4),
+};
+
+static const struct iio_chan_spec ina219_channels[] = {
+ INA219_CHAN_VOLTAGE(0, INA2XX_SHUNT_VOLTAGE),
+ INA219_CHAN_VOLTAGE(1, INA2XX_BUS_VOLTAGE),
+ INA219_CHAN(IIO_POWER, 2, INA2XX_POWER),
+ INA219_CHAN(IIO_CURRENT, 3, INA2XX_CURRENT),
IIO_CHAN_SOFT_TIMESTAMP(4),
};
*/
if (!chip->allow_async_readout)
do {
- ret = regmap_read(chip->regmap, INA226_ALERT_MASK,
+ ret = regmap_read(chip->regmap, INA226_MASK_ENABLE,
&alert);
if (ret < 0)
return ret;
- alert &= INA266_CVRF;
+ alert &= INA226_CVRF;
} while (!alert);
/*
}
/* Possible integration times for vshunt and vbus */
-static IIO_CONST_ATTR_INT_TIME_AVAIL("0.000140 0.000204 0.000332 0.000588 0.001100 0.002116 0.004156 0.008244");
+static IIO_CONST_ATTR_NAMED(ina219_integration_time_available,
+ integration_time_available,
+ "0.000084 0.000148 0.000276 0.000532 0.001060 0.002130 0.004260 0.008510 0.017020 0.034050 0.068100");
+
+static IIO_CONST_ATTR_NAMED(ina226_integration_time_available,
+ integration_time_available,
+ "0.000140 0.000204 0.000332 0.000588 0.001100 0.002116 0.004156 0.008244");
+
static IIO_DEVICE_ATTR(in_allow_async_readout, S_IRUGO | S_IWUSR,
ina2xx_allow_async_readout_show,
ina2xx_shunt_resistor_show,
ina2xx_shunt_resistor_store, 0);
-static struct attribute *ina2xx_attributes[] = {
+static struct attribute *ina219_attributes[] = {
+ &iio_dev_attr_in_allow_async_readout.dev_attr.attr,
+ &iio_const_attr_ina219_integration_time_available.dev_attr.attr,
+ &iio_dev_attr_in_shunt_resistor.dev_attr.attr,
+ NULL,
+};
+
+static struct attribute *ina226_attributes[] = {
&iio_dev_attr_in_allow_async_readout.dev_attr.attr,
- &iio_const_attr_integration_time_available.dev_attr.attr,
+ &iio_const_attr_ina226_integration_time_available.dev_attr.attr,
&iio_dev_attr_in_shunt_resistor.dev_attr.attr,
NULL,
};
-static const struct attribute_group ina2xx_attribute_group = {
- .attrs = ina2xx_attributes,
+static const struct attribute_group ina219_attribute_group = {
+ .attrs = ina219_attributes,
+};
+
+static const struct attribute_group ina226_attribute_group = {
+ .attrs = ina226_attributes,
};
-static const struct iio_info ina2xx_info = {
+static const struct iio_info ina219_info = {
.driver_module = THIS_MODULE,
- .attrs = &ina2xx_attribute_group,
+ .attrs = &ina219_attribute_group,
+ .read_raw = ina2xx_read_raw,
+ .write_raw = ina2xx_write_raw,
+ .debugfs_reg_access = ina2xx_debug_reg,
+};
+
+static const struct iio_info ina226_info = {
+ .driver_module = THIS_MODULE,
+ .attrs = &ina226_attribute_group,
.read_raw = ina2xx_read_raw,
.write_raw = ina2xx_write_raw,
.debugfs_reg_access = ina2xx_debug_reg,
ina226_set_average(chip, INA226_DEFAULT_AVG, &val);
ina226_set_int_time_vbus(chip, INA226_DEFAULT_IT, &val);
ina226_set_int_time_vshunt(chip, INA226_DEFAULT_IT, &val);
+ } else {
+ chip->avg = 1;
+ ina219_set_int_time_vbus(chip, INA219_DEFAULT_IT, &val);
+ ina219_set_int_time_vshunt(chip, INA219_DEFAULT_IT, &val);
}
ret = ina2xx_init(chip, val);
indio_dev->modes = INDIO_DIRECT_MODE | INDIO_BUFFER_SOFTWARE;
indio_dev->dev.parent = &client->dev;
indio_dev->dev.of_node = client->dev.of_node;
- indio_dev->channels = ina2xx_channels;
- indio_dev->num_channels = ARRAY_SIZE(ina2xx_channels);
+ if (id->driver_data == ina226) {
+ indio_dev->channels = ina226_channels;
+ indio_dev->num_channels = ARRAY_SIZE(ina226_channels);
+ indio_dev->info = &ina226_info;
+ } else {
+ indio_dev->channels = ina219_channels;
+ indio_dev->num_channels = ARRAY_SIZE(ina219_channels);
+ indio_dev->info = &ina219_info;
+ }
indio_dev->name = id->name;
- indio_dev->info = &ina2xx_info;
indio_dev->setup_ops = &ina2xx_setup_ops;
buffer = devm_iio_kfifo_allocate(&indio_dev->dev);
long mask)
{
struct lpc32xx_adc_state *st = iio_priv(indio_dev);
-
+ int ret;
if (mask == IIO_CHAN_INFO_RAW) {
mutex_lock(&indio_dev->mlock);
- clk_prepare_enable(st->clk);
+ ret = clk_prepare_enable(st->clk);
+ if (ret) {
+ mutex_unlock(&indio_dev->mlock);
+ return ret;
+ }
/* Measurement setup */
__raw_writel(LPC32XXAD_INTERNAL | (chan->address) |
LPC32XXAD_REFp | LPC32XXAD_REFm,
};
struct meson_sar_adc_data {
+ bool has_bl30_integration;
unsigned int resolution;
const char *name;
};
mutex_lock(&indio_dev->mlock);
- /* prevent BL30 from using the SAR ADC while we are using it */
- regmap_update_bits(priv->regmap, MESON_SAR_ADC_DELAY,
- MESON_SAR_ADC_DELAY_KERNEL_BUSY,
- MESON_SAR_ADC_DELAY_KERNEL_BUSY);
-
- /* wait until BL30 releases it's lock (so we can use the SAR ADC) */
- do {
- udelay(1);
- regmap_read(priv->regmap, MESON_SAR_ADC_DELAY, &val);
- } while (val & MESON_SAR_ADC_DELAY_BL30_BUSY && timeout--);
-
- if (timeout < 0)
- return -ETIMEDOUT;
+ if (priv->data->has_bl30_integration) {
+ /* prevent BL30 from using the SAR ADC while we are using it */
+ regmap_update_bits(priv->regmap, MESON_SAR_ADC_DELAY,
+ MESON_SAR_ADC_DELAY_KERNEL_BUSY,
+ MESON_SAR_ADC_DELAY_KERNEL_BUSY);
+
+ /*
+ * wait until BL30 releases it's lock (so we can use the SAR
+ * ADC)
+ */
+ do {
+ udelay(1);
+ regmap_read(priv->regmap, MESON_SAR_ADC_DELAY, &val);
+ } while (val & MESON_SAR_ADC_DELAY_BL30_BUSY && timeout--);
+
+ if (timeout < 0)
+ return -ETIMEDOUT;
+ }
return 0;
}
{
struct meson_sar_adc_priv *priv = iio_priv(indio_dev);
- /* allow BL30 to use the SAR ADC again */
- regmap_update_bits(priv->regmap, MESON_SAR_ADC_DELAY,
- MESON_SAR_ADC_DELAY_KERNEL_BUSY, 0);
+ if (priv->data->has_bl30_integration)
+ /* allow BL30 to use the SAR ADC again */
+ regmap_update_bits(priv->regmap, MESON_SAR_ADC_DELAY,
+ MESON_SAR_ADC_DELAY_KERNEL_BUSY, 0);
mutex_unlock(&indio_dev->mlock);
}
*/
meson_sar_adc_set_chan7_mux(indio_dev, CHAN7_MUX_CH7_INPUT);
- /*
- * leave sampling delay and the input clocks as configured by BL30 to
- * make sure BL30 gets the values it expects when reading the
- * temperature sensor.
- */
- regmap_read(priv->regmap, MESON_SAR_ADC_REG3, ®val);
- if (regval & MESON_SAR_ADC_REG3_BL30_INITIALIZED)
- return 0;
+ if (priv->data->has_bl30_integration) {
+ /*
+ * leave sampling delay and the input clocks as configured by
+ * BL30 to make sure BL30 gets the values it expects when
+ * reading the temperature sensor.
+ */
+ regmap_read(priv->regmap, MESON_SAR_ADC_REG3, ®val);
+ if (regval & MESON_SAR_ADC_REG3_BL30_INITIALIZED)
+ return 0;
+ }
meson_sar_adc_stop_sample_engine(indio_dev);
.driver_module = THIS_MODULE,
};
-struct meson_sar_adc_data meson_sar_adc_gxbb_data = {
+static const struct meson_sar_adc_data meson_sar_adc_meson8_data = {
+ .has_bl30_integration = false,
+ .resolution = 10,
+ .name = "meson-meson8-saradc",
+};
+
+static const struct meson_sar_adc_data meson_sar_adc_meson8b_data = {
+ .has_bl30_integration = false,
+ .resolution = 10,
+ .name = "meson-meson8b-saradc",
+};
+
+static const struct meson_sar_adc_data meson_sar_adc_gxbb_data = {
+ .has_bl30_integration = true,
.resolution = 10,
.name = "meson-gxbb-saradc",
};
-struct meson_sar_adc_data meson_sar_adc_gxl_data = {
+static const struct meson_sar_adc_data meson_sar_adc_gxl_data = {
+ .has_bl30_integration = true,
.resolution = 12,
.name = "meson-gxl-saradc",
};
-struct meson_sar_adc_data meson_sar_adc_gxm_data = {
+static const struct meson_sar_adc_data meson_sar_adc_gxm_data = {
+ .has_bl30_integration = true,
.resolution = 12,
.name = "meson-gxm-saradc",
};
static const struct of_device_id meson_sar_adc_of_match[] = {
+ {
+ .compatible = "amlogic,meson8-saradc",
+ .data = &meson_sar_adc_meson8_data,
+ },
+ {
+ .compatible = "amlogic,meson8b-saradc",
+ .data = &meson_sar_adc_meson8b_data,
+ },
{
.compatible = "amlogic,meson-gxbb-saradc",
.data = &meson_sar_adc_gxbb_data,
#define VREF_MV_BASE 1850
-const char *mx23_lradc_adc_irq_names[] = {
+static const char *mx23_lradc_adc_irq_names[] = {
"mxs-lradc-channel0",
"mxs-lradc-channel1",
"mxs-lradc-channel2",
"mxs-lradc-channel5",
};
-const char *mx28_lradc_adc_irq_names[] = {
+static const char *mx28_lradc_adc_irq_names[] = {
"mxs-lradc-thresh0",
"mxs-lradc-thresh1",
"mxs-lradc-channel0",
IIO_DEVICE_ATTR(in_voltage##ch##_scale_available, 0444,\
mxs_lradc_adc_show_scale_avail, NULL, ch)
-SHOW_SCALE_AVAILABLE_ATTR(0);
-SHOW_SCALE_AVAILABLE_ATTR(1);
-SHOW_SCALE_AVAILABLE_ATTR(2);
-SHOW_SCALE_AVAILABLE_ATTR(3);
-SHOW_SCALE_AVAILABLE_ATTR(4);
-SHOW_SCALE_AVAILABLE_ATTR(5);
-SHOW_SCALE_AVAILABLE_ATTR(6);
-SHOW_SCALE_AVAILABLE_ATTR(7);
-SHOW_SCALE_AVAILABLE_ATTR(10);
-SHOW_SCALE_AVAILABLE_ATTR(11);
-SHOW_SCALE_AVAILABLE_ATTR(12);
-SHOW_SCALE_AVAILABLE_ATTR(13);
-SHOW_SCALE_AVAILABLE_ATTR(14);
-SHOW_SCALE_AVAILABLE_ATTR(15);
+static SHOW_SCALE_AVAILABLE_ATTR(0);
+static SHOW_SCALE_AVAILABLE_ATTR(1);
+static SHOW_SCALE_AVAILABLE_ATTR(2);
+static SHOW_SCALE_AVAILABLE_ATTR(3);
+static SHOW_SCALE_AVAILABLE_ATTR(4);
+static SHOW_SCALE_AVAILABLE_ATTR(5);
+static SHOW_SCALE_AVAILABLE_ATTR(6);
+static SHOW_SCALE_AVAILABLE_ATTR(7);
+static SHOW_SCALE_AVAILABLE_ATTR(10);
+static SHOW_SCALE_AVAILABLE_ATTR(11);
+static SHOW_SCALE_AVAILABLE_ATTR(12);
+static SHOW_SCALE_AVAILABLE_ATTR(13);
+static SHOW_SCALE_AVAILABLE_ATTR(14);
+static SHOW_SCALE_AVAILABLE_ATTR(15);
static struct attribute *mxs_lradc_adc_attributes[] = {
&iio_dev_attr_in_voltage0_scale_available.dev_attr.attr,
struct rcar_gyroadc {
struct device *dev;
void __iomem *regs;
- struct clk *iclk;
+ struct clk *clk;
struct regulator *vref[8];
unsigned int num_channels;
enum rcar_gyroadc_model model;
static void rcar_gyroadc_hw_init(struct rcar_gyroadc *priv)
{
- const unsigned long clk_mhz = clk_get_rate(priv->iclk) / 1000000;
+ const unsigned long clk_mhz = clk_get_rate(priv->clk) / 1000000;
const unsigned long clk_mul =
(priv->mode == RCAR_GYROADC_MODE_SELECT_1_MB88101A) ? 10 : 5;
unsigned long clk_len = clk_mhz * clk_mul;
if (IS_ERR(priv->regs))
return PTR_ERR(priv->regs);
- priv->iclk = devm_clk_get(dev, "if");
- if (IS_ERR(priv->iclk)) {
- ret = PTR_ERR(priv->iclk);
+ priv->clk = devm_clk_get(dev, "fck");
+ if (IS_ERR(priv->clk)) {
+ ret = PTR_ERR(priv->clk);
if (ret != -EPROBE_DEFER)
dev_err(dev, "Failed to get IF clock (ret=%i)\n", ret);
return ret;
indio_dev->info = &rcar_gyroadc_iio_info;
indio_dev->modes = INDIO_DIRECT_MODE;
- ret = clk_prepare_enable(priv->iclk);
+ ret = clk_prepare_enable(priv->clk);
if (ret) {
dev_err(dev, "Could not prepare or enable the IF clock.\n");
goto err_clk_if_enable;
pm_runtime_put_sync(dev);
pm_runtime_disable(dev);
pm_runtime_set_suspended(dev);
- clk_disable_unprepare(priv->iclk);
+ clk_disable_unprepare(priv->clk);
err_clk_if_enable:
rcar_gyroadc_deinit_supplies(indio_dev);
pm_runtime_put_sync(dev);
pm_runtime_disable(dev);
pm_runtime_set_suspended(dev);
- clk_disable_unprepare(priv->iclk);
+ clk_disable_unprepare(priv->clk);
rcar_gyroadc_deinit_supplies(indio_dev);
return 0;
/* STM32 F4 maximum analog clock rate (from datasheet) */
#define STM32F4_ADC_MAX_CLK_RATE 36000000
+/* STM32H7 - common registers for all ADC instances */
+#define STM32H7_ADC_CSR (STM32_ADCX_COMN_OFFSET + 0x00)
+#define STM32H7_ADC_CCR (STM32_ADCX_COMN_OFFSET + 0x08)
+
+/* STM32H7_ADC_CSR - bit fields */
+#define STM32H7_EOC_SLV BIT(18)
+#define STM32H7_EOC_MST BIT(2)
+
+/* STM32H7_ADC_CCR - bit fields */
+#define STM32H7_PRESC_SHIFT 18
+#define STM32H7_PRESC_MASK GENMASK(21, 18)
+#define STM32H7_CKMODE_SHIFT 16
+#define STM32H7_CKMODE_MASK GENMASK(17, 16)
+
+/* STM32 H7 maximum analog clock rate (from datasheet) */
+#define STM32H7_ADC_MAX_CLK_RATE 72000000
+
+/**
+ * stm32_adc_common_regs - stm32 common registers, compatible dependent data
+ * @csr: common status register offset
+ * @eoc1: adc1 end of conversion flag in @csr
+ * @eoc2: adc2 end of conversion flag in @csr
+ * @eoc3: adc3 end of conversion flag in @csr
+ */
+struct stm32_adc_common_regs {
+ u32 csr;
+ u32 eoc1_msk;
+ u32 eoc2_msk;
+ u32 eoc3_msk;
+};
+
+struct stm32_adc_priv;
+
+/**
+ * stm32_adc_priv_cfg - stm32 core compatible configuration data
+ * @regs: common registers for all instances
+ * @clk_sel: clock selection routine
+ */
+struct stm32_adc_priv_cfg {
+ const struct stm32_adc_common_regs *regs;
+ int (*clk_sel)(struct platform_device *, struct stm32_adc_priv *);
+};
+
/**
* struct stm32_adc_priv - stm32 ADC core private data
* @irq: irq for ADC block
* @domain: irq domain reference
* @aclk: clock reference for the analog circuitry
+ * @bclk: bus clock common for all ADCs, depends on part used
* @vref: regulator reference
+ * @cfg: compatible configuration data
* @common: common data for all ADC instances
*/
struct stm32_adc_priv {
int irq;
struct irq_domain *domain;
struct clk *aclk;
+ struct clk *bclk;
struct regulator *vref;
+ const struct stm32_adc_priv_cfg *cfg;
struct stm32_adc_common common;
};
u32 val;
int i;
+ /* stm32f4 has one clk input for analog (mandatory), enforce it here */
+ if (!priv->aclk) {
+ dev_err(&pdev->dev, "No 'adc' clock found\n");
+ return -ENOENT;
+ }
+
rate = clk_get_rate(priv->aclk);
for (i = 0; i < ARRAY_SIZE(stm32f4_pclk_div); i++) {
if ((rate / stm32f4_pclk_div[i]) <= STM32F4_ADC_MAX_CLK_RATE)
break;
}
- if (i >= ARRAY_SIZE(stm32f4_pclk_div))
+ if (i >= ARRAY_SIZE(stm32f4_pclk_div)) {
+ dev_err(&pdev->dev, "adc clk selection failed\n");
return -EINVAL;
+ }
+ priv->common.rate = rate;
val = readl_relaxed(priv->common.base + STM32F4_ADC_CCR);
val &= ~STM32F4_ADC_ADCPRE_MASK;
val |= i << STM32F4_ADC_ADCPRE_SHIFT;
return 0;
}
+/**
+ * struct stm32h7_adc_ck_spec - specification for stm32h7 adc clock
+ * @ckmode: ADC clock mode, Async or sync with prescaler.
+ * @presc: prescaler bitfield for async clock mode
+ * @div: prescaler division ratio
+ */
+struct stm32h7_adc_ck_spec {
+ u32 ckmode;
+ u32 presc;
+ int div;
+};
+
+const struct stm32h7_adc_ck_spec stm32h7_adc_ckmodes_spec[] = {
+ /* 00: CK_ADC[1..3]: Asynchronous clock modes */
+ { 0, 0, 1 },
+ { 0, 1, 2 },
+ { 0, 2, 4 },
+ { 0, 3, 6 },
+ { 0, 4, 8 },
+ { 0, 5, 10 },
+ { 0, 6, 12 },
+ { 0, 7, 16 },
+ { 0, 8, 32 },
+ { 0, 9, 64 },
+ { 0, 10, 128 },
+ { 0, 11, 256 },
+ /* HCLK used: Synchronous clock modes (1, 2 or 4 prescaler) */
+ { 1, 0, 1 },
+ { 2, 0, 2 },
+ { 3, 0, 4 },
+};
+
+static int stm32h7_adc_clk_sel(struct platform_device *pdev,
+ struct stm32_adc_priv *priv)
+{
+ u32 ckmode, presc, val;
+ unsigned long rate;
+ int i, div;
+
+ /* stm32h7 bus clock is common for all ADC instances (mandatory) */
+ if (!priv->bclk) {
+ dev_err(&pdev->dev, "No 'bus' clock found\n");
+ return -ENOENT;
+ }
+
+ /*
+ * stm32h7 can use either 'bus' or 'adc' clock for analog circuitry.
+ * So, choice is to have bus clock mandatory and adc clock optional.
+ * If optional 'adc' clock has been found, then try to use it first.
+ */
+ if (priv->aclk) {
+ /*
+ * Asynchronous clock modes (e.g. ckmode == 0)
+ * From spec: PLL output musn't exceed max rate
+ */
+ rate = clk_get_rate(priv->aclk);
+
+ for (i = 0; i < ARRAY_SIZE(stm32h7_adc_ckmodes_spec); i++) {
+ ckmode = stm32h7_adc_ckmodes_spec[i].ckmode;
+ presc = stm32h7_adc_ckmodes_spec[i].presc;
+ div = stm32h7_adc_ckmodes_spec[i].div;
+
+ if (ckmode)
+ continue;
+
+ if ((rate / div) <= STM32H7_ADC_MAX_CLK_RATE)
+ goto out;
+ }
+ }
+
+ /* Synchronous clock modes (e.g. ckmode is 1, 2 or 3) */
+ rate = clk_get_rate(priv->bclk);
+
+ for (i = 0; i < ARRAY_SIZE(stm32h7_adc_ckmodes_spec); i++) {
+ ckmode = stm32h7_adc_ckmodes_spec[i].ckmode;
+ presc = stm32h7_adc_ckmodes_spec[i].presc;
+ div = stm32h7_adc_ckmodes_spec[i].div;
+
+ if (!ckmode)
+ continue;
+
+ if ((rate / div) <= STM32H7_ADC_MAX_CLK_RATE)
+ goto out;
+ }
+
+ dev_err(&pdev->dev, "adc clk selection failed\n");
+ return -EINVAL;
+
+out:
+ /* rate used later by each ADC instance to control BOOST mode */
+ priv->common.rate = rate;
+
+ /* Set common clock mode and prescaler */
+ val = readl_relaxed(priv->common.base + STM32H7_ADC_CCR);
+ val &= ~(STM32H7_CKMODE_MASK | STM32H7_PRESC_MASK);
+ val |= ckmode << STM32H7_CKMODE_SHIFT;
+ val |= presc << STM32H7_PRESC_SHIFT;
+ writel_relaxed(val, priv->common.base + STM32H7_ADC_CCR);
+
+ dev_dbg(&pdev->dev, "Using %s clock/%d source at %ld kHz\n",
+ ckmode ? "bus" : "adc", div, rate / (div * 1000));
+
+ return 0;
+}
+
+/* STM32F4 common registers definitions */
+static const struct stm32_adc_common_regs stm32f4_adc_common_regs = {
+ .csr = STM32F4_ADC_CSR,
+ .eoc1_msk = STM32F4_EOC1,
+ .eoc2_msk = STM32F4_EOC2,
+ .eoc3_msk = STM32F4_EOC3,
+};
+
+/* STM32H7 common registers definitions */
+static const struct stm32_adc_common_regs stm32h7_adc_common_regs = {
+ .csr = STM32H7_ADC_CSR,
+ .eoc1_msk = STM32H7_EOC_MST,
+ .eoc2_msk = STM32H7_EOC_SLV,
+};
+
/* ADC common interrupt for all instances */
static void stm32_adc_irq_handler(struct irq_desc *desc)
{
u32 status;
chained_irq_enter(chip, desc);
- status = readl_relaxed(priv->common.base + STM32F4_ADC_CSR);
+ status = readl_relaxed(priv->common.base + priv->cfg->regs->csr);
- if (status & STM32F4_EOC1)
+ if (status & priv->cfg->regs->eoc1_msk)
generic_handle_irq(irq_find_mapping(priv->domain, 0));
- if (status & STM32F4_EOC2)
+ if (status & priv->cfg->regs->eoc2_msk)
generic_handle_irq(irq_find_mapping(priv->domain, 1));
- if (status & STM32F4_EOC3)
+ if (status & priv->cfg->regs->eoc3_msk)
generic_handle_irq(irq_find_mapping(priv->domain, 2));
chained_irq_exit(chip, desc);
static int stm32_adc_probe(struct platform_device *pdev)
{
struct stm32_adc_priv *priv;
+ struct device *dev = &pdev->dev;
struct device_node *np = pdev->dev.of_node;
struct resource *res;
int ret;
if (!priv)
return -ENOMEM;
+ priv->cfg = (const struct stm32_adc_priv_cfg *)
+ of_match_device(dev->driver->of_match_table, dev)->data;
+
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
priv->common.base = devm_ioremap_resource(&pdev->dev, res);
if (IS_ERR(priv->common.base))
priv->aclk = devm_clk_get(&pdev->dev, "adc");
if (IS_ERR(priv->aclk)) {
ret = PTR_ERR(priv->aclk);
- dev_err(&pdev->dev, "Can't get 'adc' clock\n");
- goto err_regulator_disable;
+ if (ret == -ENOENT) {
+ priv->aclk = NULL;
+ } else {
+ dev_err(&pdev->dev, "Can't get 'adc' clock\n");
+ goto err_regulator_disable;
+ }
}
- ret = clk_prepare_enable(priv->aclk);
- if (ret < 0) {
- dev_err(&pdev->dev, "adc clk enable failed\n");
- goto err_regulator_disable;
+ if (priv->aclk) {
+ ret = clk_prepare_enable(priv->aclk);
+ if (ret < 0) {
+ dev_err(&pdev->dev, "adc clk enable failed\n");
+ goto err_regulator_disable;
+ }
}
- ret = stm32f4_adc_clk_sel(pdev, priv);
- if (ret < 0) {
- dev_err(&pdev->dev, "adc clk selection failed\n");
- goto err_clk_disable;
+ priv->bclk = devm_clk_get(&pdev->dev, "bus");
+ if (IS_ERR(priv->bclk)) {
+ ret = PTR_ERR(priv->bclk);
+ if (ret == -ENOENT) {
+ priv->bclk = NULL;
+ } else {
+ dev_err(&pdev->dev, "Can't get 'bus' clock\n");
+ goto err_aclk_disable;
+ }
+ }
+
+ if (priv->bclk) {
+ ret = clk_prepare_enable(priv->bclk);
+ if (ret < 0) {
+ dev_err(&pdev->dev, "adc clk enable failed\n");
+ goto err_aclk_disable;
+ }
}
+ ret = priv->cfg->clk_sel(pdev, priv);
+ if (ret < 0)
+ goto err_bclk_disable;
+
ret = stm32_adc_irq_probe(pdev, priv);
if (ret < 0)
- goto err_clk_disable;
+ goto err_bclk_disable;
platform_set_drvdata(pdev, &priv->common);
err_irq_remove:
stm32_adc_irq_remove(pdev, priv);
-err_clk_disable:
- clk_disable_unprepare(priv->aclk);
+err_bclk_disable:
+ if (priv->bclk)
+ clk_disable_unprepare(priv->bclk);
+
+err_aclk_disable:
+ if (priv->aclk)
+ clk_disable_unprepare(priv->aclk);
err_regulator_disable:
regulator_disable(priv->vref);
of_platform_depopulate(&pdev->dev);
stm32_adc_irq_remove(pdev, priv);
- clk_disable_unprepare(priv->aclk);
+ if (priv->bclk)
+ clk_disable_unprepare(priv->bclk);
+ if (priv->aclk)
+ clk_disable_unprepare(priv->aclk);
regulator_disable(priv->vref);
return 0;
}
+static const struct stm32_adc_priv_cfg stm32f4_adc_priv_cfg = {
+ .regs = &stm32f4_adc_common_regs,
+ .clk_sel = stm32f4_adc_clk_sel,
+};
+
+static const struct stm32_adc_priv_cfg stm32h7_adc_priv_cfg = {
+ .regs = &stm32h7_adc_common_regs,
+ .clk_sel = stm32h7_adc_clk_sel,
+};
+
static const struct of_device_id stm32_adc_of_match[] = {
- { .compatible = "st,stm32f4-adc-core" },
- {},
+ {
+ .compatible = "st,stm32f4-adc-core",
+ .data = (void *)&stm32f4_adc_priv_cfg
+ }, {
+ .compatible = "st,stm32h7-adc-core",
+ .data = (void *)&stm32h7_adc_priv_cfg
+ }, {
+ },
};
MODULE_DEVICE_TABLE(of, stm32_adc_of_match);
* struct stm32_adc_common - stm32 ADC driver common data (for all instances)
* @base: control registers base cpu addr
* @phys_base: control registers base physical addr
+ * @rate: clock rate used for analog circuitry
* @vref_mv: vref voltage (mv)
*/
struct stm32_adc_common {
void __iomem *base;
phys_addr_t phys_base;
+ unsigned long rate;
int vref_mv;
};
#include <linux/iio/triggered_buffer.h>
#include <linux/interrupt.h>
#include <linux/io.h>
+#include <linux/iopoll.h>
#include <linux/module.h>
#include <linux/platform_device.h>
#include <linux/of.h>
+#include <linux/of_device.h>
#include "stm32-adc-core.h"
#define STM32F4_DMA BIT(8)
#define STM32F4_ADON BIT(0)
+/* STM32H7 - Registers for each ADC instance */
+#define STM32H7_ADC_ISR 0x00
+#define STM32H7_ADC_IER 0x04
+#define STM32H7_ADC_CR 0x08
+#define STM32H7_ADC_CFGR 0x0C
+#define STM32H7_ADC_PCSEL 0x1C
+#define STM32H7_ADC_SQR1 0x30
+#define STM32H7_ADC_SQR2 0x34
+#define STM32H7_ADC_SQR3 0x38
+#define STM32H7_ADC_SQR4 0x3C
+#define STM32H7_ADC_DR 0x40
+#define STM32H7_ADC_CALFACT 0xC4
+#define STM32H7_ADC_CALFACT2 0xC8
+
+/* STM32H7_ADC_ISR - bit fields */
+#define STM32H7_EOC BIT(2)
+#define STM32H7_ADRDY BIT(0)
+
+/* STM32H7_ADC_IER - bit fields */
+#define STM32H7_EOCIE STM32H7_EOC
+
+/* STM32H7_ADC_CR - bit fields */
+#define STM32H7_ADCAL BIT(31)
+#define STM32H7_ADCALDIF BIT(30)
+#define STM32H7_DEEPPWD BIT(29)
+#define STM32H7_ADVREGEN BIT(28)
+#define STM32H7_LINCALRDYW6 BIT(27)
+#define STM32H7_LINCALRDYW5 BIT(26)
+#define STM32H7_LINCALRDYW4 BIT(25)
+#define STM32H7_LINCALRDYW3 BIT(24)
+#define STM32H7_LINCALRDYW2 BIT(23)
+#define STM32H7_LINCALRDYW1 BIT(22)
+#define STM32H7_ADCALLIN BIT(16)
+#define STM32H7_BOOST BIT(8)
+#define STM32H7_ADSTP BIT(4)
+#define STM32H7_ADSTART BIT(2)
+#define STM32H7_ADDIS BIT(1)
+#define STM32H7_ADEN BIT(0)
+
+/* STM32H7_ADC_CFGR bit fields */
+#define STM32H7_EXTEN_SHIFT 10
+#define STM32H7_EXTEN_MASK GENMASK(11, 10)
+#define STM32H7_EXTSEL_SHIFT 5
+#define STM32H7_EXTSEL_MASK GENMASK(9, 5)
+#define STM32H7_RES_SHIFT 2
+#define STM32H7_RES_MASK GENMASK(4, 2)
+#define STM32H7_DMNGT_SHIFT 0
+#define STM32H7_DMNGT_MASK GENMASK(1, 0)
+
+enum stm32h7_adc_dmngt {
+ STM32H7_DMNGT_DR_ONLY, /* Regular data in DR only */
+ STM32H7_DMNGT_DMA_ONESHOT, /* DMA one shot mode */
+ STM32H7_DMNGT_DFSDM, /* DFSDM mode */
+ STM32H7_DMNGT_DMA_CIRC, /* DMA circular mode */
+};
+
+/* STM32H7_ADC_CALFACT - bit fields */
+#define STM32H7_CALFACT_D_SHIFT 16
+#define STM32H7_CALFACT_D_MASK GENMASK(26, 16)
+#define STM32H7_CALFACT_S_SHIFT 0
+#define STM32H7_CALFACT_S_MASK GENMASK(10, 0)
+
+/* STM32H7_ADC_CALFACT2 - bit fields */
+#define STM32H7_LINCALFACT_SHIFT 0
+#define STM32H7_LINCALFACT_MASK GENMASK(29, 0)
+
+/* Number of linear calibration shadow registers / LINCALRDYW control bits */
+#define STM32H7_LINCALFACT_NUM 6
+
+/* BOOST bit must be set on STM32H7 when ADC clock is above 20MHz */
+#define STM32H7_BOOST_CLKRATE 20000000UL
+
#define STM32_ADC_MAX_SQ 16 /* SQ1..SQ16 */
#define STM32_ADC_TIMEOUT_US 100000
#define STM32_ADC_TIMEOUT (msecs_to_jiffies(STM32_ADC_TIMEOUT_US / 1000))
enum stm32_adc_extsel extsel;
};
+/**
+ * struct stm32_adc_calib - optional adc calibration data
+ * @calfact_s: Calibration offset for single ended channels
+ * @calfact_d: Calibration offset in differential
+ * @lincalfact: Linearity calibration factor
+ */
+struct stm32_adc_calib {
+ u32 calfact_s;
+ u32 calfact_d;
+ u32 lincalfact[STM32H7_LINCALFACT_NUM];
+};
+
/**
* stm32_adc_regs - stm32 ADC misc registers & bitfield desc
* @reg: register offset
int shift;
};
+/**
+ * stm32_adc_regspec - stm32 registers definition, compatible dependent data
+ * @dr: data register offset
+ * @ier_eoc: interrupt enable register & eocie bitfield
+ * @isr_eoc: interrupt status register & eoc bitfield
+ * @sqr: reference to sequence registers array
+ * @exten: trigger control register & bitfield
+ * @extsel: trigger selection register & bitfield
+ * @res: resolution selection register & bitfield
+ */
+struct stm32_adc_regspec {
+ const u32 dr;
+ const struct stm32_adc_regs ier_eoc;
+ const struct stm32_adc_regs isr_eoc;
+ const struct stm32_adc_regs *sqr;
+ const struct stm32_adc_regs exten;
+ const struct stm32_adc_regs extsel;
+ const struct stm32_adc_regs res;
+};
+
+struct stm32_adc;
+
+/**
+ * stm32_adc_cfg - stm32 compatible configuration data
+ * @regs: registers descriptions
+ * @adc_info: per instance input channels definitions
+ * @trigs: external trigger sources
+ * @clk_required: clock is required
+ * @selfcalib: optional routine for self-calibration
+ * @prepare: optional prepare routine (power-up, enable)
+ * @start_conv: routine to start conversions
+ * @stop_conv: routine to stop conversions
+ * @unprepare: optional unprepare routine (disable, power-down)
+ */
+struct stm32_adc_cfg {
+ const struct stm32_adc_regspec *regs;
+ const struct stm32_adc_info *adc_info;
+ struct stm32_adc_trig_info *trigs;
+ bool clk_required;
+ int (*selfcalib)(struct stm32_adc *);
+ int (*prepare)(struct stm32_adc *);
+ void (*start_conv)(struct stm32_adc *, bool dma);
+ void (*stop_conv)(struct stm32_adc *);
+ void (*unprepare)(struct stm32_adc *);
+};
+
/**
* struct stm32_adc - private data of each ADC IIO instance
* @common: reference to ADC block common data
* @offset: ADC instance register offset in ADC block
+ * @cfg: compatible configuration data
* @completion: end of single conversion completion
* @buffer: data buffer
* @clk: clock for this adc instance
* @rx_buf: dma rx buffer cpu address
* @rx_dma_buf: dma rx buffer bus address
* @rx_buf_sz: dma rx buffer size
+ * @pcsel bitmask to preselect channels on some devices
+ * @cal: optional calibration data on some devices
*/
struct stm32_adc {
struct stm32_adc_common *common;
u32 offset;
+ const struct stm32_adc_cfg *cfg;
struct completion completion;
u16 buffer[STM32_ADC_MAX_SQ];
struct clk *clk;
u8 *rx_buf;
dma_addr_t rx_dma_buf;
unsigned int rx_buf_sz;
+ u32 pcsel;
+ struct stm32_adc_calib cal;
};
/**
const char *name;
};
-/* Input definitions common for all STM32F4 instances */
-static const struct stm32_adc_chan_spec stm32f4_adc123_channels[] = {
+/**
+ * struct stm32_adc_info - stm32 ADC, per instance config data
+ * @channels: Reference to stm32 channels spec
+ * @max_channels: Number of channels
+ * @resolutions: available resolutions
+ * @num_res: number of available resolutions
+ */
+struct stm32_adc_info {
+ const struct stm32_adc_chan_spec *channels;
+ int max_channels;
+ const unsigned int *resolutions;
+ const unsigned int num_res;
+};
+
+/*
+ * Input definitions common for all instances:
+ * stm32f4 can have up to 16 channels
+ * stm32h7 can have up to 20 channels
+ */
+static const struct stm32_adc_chan_spec stm32_adc_channels[] = {
{ IIO_VOLTAGE, 0, "in0" },
{ IIO_VOLTAGE, 1, "in1" },
{ IIO_VOLTAGE, 2, "in2" },
{ IIO_VOLTAGE, 13, "in13" },
{ IIO_VOLTAGE, 14, "in14" },
{ IIO_VOLTAGE, 15, "in15" },
+ { IIO_VOLTAGE, 16, "in16" },
+ { IIO_VOLTAGE, 17, "in17" },
+ { IIO_VOLTAGE, 18, "in18" },
+ { IIO_VOLTAGE, 19, "in19" },
};
static const unsigned int stm32f4_adc_resolutions[] = {
12, 10, 8, 6,
};
+static const struct stm32_adc_info stm32f4_adc_info = {
+ .channels = stm32_adc_channels,
+ .max_channels = 16,
+ .resolutions = stm32f4_adc_resolutions,
+ .num_res = ARRAY_SIZE(stm32f4_adc_resolutions),
+};
+
+static const unsigned int stm32h7_adc_resolutions[] = {
+ /* sorted values so the index matches RES[2:0] in STM32H7_ADC_CFGR */
+ 16, 14, 12, 10, 8,
+};
+
+static const struct stm32_adc_info stm32h7_adc_info = {
+ .channels = stm32_adc_channels,
+ .max_channels = 20,
+ .resolutions = stm32h7_adc_resolutions,
+ .num_res = ARRAY_SIZE(stm32h7_adc_resolutions),
+};
+
/**
* stm32f4_sq - describe regular sequence registers
* - L: sequence len (register & bit field)
{}, /* sentinel */
};
+static const struct stm32_adc_regspec stm32f4_adc_regspec = {
+ .dr = STM32F4_ADC_DR,
+ .ier_eoc = { STM32F4_ADC_CR1, STM32F4_EOCIE },
+ .isr_eoc = { STM32F4_ADC_SR, STM32F4_EOC },
+ .sqr = stm32f4_sq,
+ .exten = { STM32F4_ADC_CR2, STM32F4_EXTEN_MASK, STM32F4_EXTEN_SHIFT },
+ .extsel = { STM32F4_ADC_CR2, STM32F4_EXTSEL_MASK,
+ STM32F4_EXTSEL_SHIFT },
+ .res = { STM32F4_ADC_CR1, STM32F4_RES_MASK, STM32F4_RES_SHIFT },
+};
+
+static const struct stm32_adc_regs stm32h7_sq[STM32_ADC_MAX_SQ + 1] = {
+ /* L: len bit field description to be kept as first element */
+ { STM32H7_ADC_SQR1, GENMASK(3, 0), 0 },
+ /* SQ1..SQ16 registers & bit fields (reg, mask, shift) */
+ { STM32H7_ADC_SQR1, GENMASK(10, 6), 6 },
+ { STM32H7_ADC_SQR1, GENMASK(16, 12), 12 },
+ { STM32H7_ADC_SQR1, GENMASK(22, 18), 18 },
+ { STM32H7_ADC_SQR1, GENMASK(28, 24), 24 },
+ { STM32H7_ADC_SQR2, GENMASK(4, 0), 0 },
+ { STM32H7_ADC_SQR2, GENMASK(10, 6), 6 },
+ { STM32H7_ADC_SQR2, GENMASK(16, 12), 12 },
+ { STM32H7_ADC_SQR2, GENMASK(22, 18), 18 },
+ { STM32H7_ADC_SQR2, GENMASK(28, 24), 24 },
+ { STM32H7_ADC_SQR3, GENMASK(4, 0), 0 },
+ { STM32H7_ADC_SQR3, GENMASK(10, 6), 6 },
+ { STM32H7_ADC_SQR3, GENMASK(16, 12), 12 },
+ { STM32H7_ADC_SQR3, GENMASK(22, 18), 18 },
+ { STM32H7_ADC_SQR3, GENMASK(28, 24), 24 },
+ { STM32H7_ADC_SQR4, GENMASK(4, 0), 0 },
+ { STM32H7_ADC_SQR4, GENMASK(10, 6), 6 },
+};
+
+/* STM32H7 external trigger sources for all instances */
+static struct stm32_adc_trig_info stm32h7_adc_trigs[] = {
+ { TIM1_CH1, STM32_EXT0 },
+ { TIM1_CH2, STM32_EXT1 },
+ { TIM1_CH3, STM32_EXT2 },
+ { TIM2_CH2, STM32_EXT3 },
+ { TIM3_TRGO, STM32_EXT4 },
+ { TIM4_CH4, STM32_EXT5 },
+ { TIM8_TRGO, STM32_EXT7 },
+ { TIM8_TRGO2, STM32_EXT8 },
+ { TIM1_TRGO, STM32_EXT9 },
+ { TIM1_TRGO2, STM32_EXT10 },
+ { TIM2_TRGO, STM32_EXT11 },
+ { TIM4_TRGO, STM32_EXT12 },
+ { TIM6_TRGO, STM32_EXT13 },
+ { TIM3_CH4, STM32_EXT15 },
+ {},
+};
+
+static const struct stm32_adc_regspec stm32h7_adc_regspec = {
+ .dr = STM32H7_ADC_DR,
+ .ier_eoc = { STM32H7_ADC_IER, STM32H7_EOCIE },
+ .isr_eoc = { STM32H7_ADC_ISR, STM32H7_EOC },
+ .sqr = stm32h7_sq,
+ .exten = { STM32H7_ADC_CFGR, STM32H7_EXTEN_MASK, STM32H7_EXTEN_SHIFT },
+ .extsel = { STM32H7_ADC_CFGR, STM32H7_EXTSEL_MASK,
+ STM32H7_EXTSEL_SHIFT },
+ .res = { STM32H7_ADC_CFGR, STM32H7_RES_MASK, STM32H7_RES_SHIFT },
+};
+
/**
* STM32 ADC registers access routines
* @adc: stm32 adc instance
return readl_relaxed(adc->common->base + adc->offset + reg);
}
+#define stm32_adc_readl_addr(addr) stm32_adc_readl(adc, addr)
+
+#define stm32_adc_readl_poll_timeout(reg, val, cond, sleep_us, timeout_us) \
+ readx_poll_timeout(stm32_adc_readl_addr, reg, val, \
+ cond, sleep_us, timeout_us)
+
static u16 stm32_adc_readw(struct stm32_adc *adc, u32 reg)
{
return readw_relaxed(adc->common->base + adc->offset + reg);
*/
static void stm32_adc_conv_irq_enable(struct stm32_adc *adc)
{
- stm32_adc_set_bits(adc, STM32F4_ADC_CR1, STM32F4_EOCIE);
+ stm32_adc_set_bits(adc, adc->cfg->regs->ier_eoc.reg,
+ adc->cfg->regs->ier_eoc.mask);
};
/**
*/
static void stm32_adc_conv_irq_disable(struct stm32_adc *adc)
{
- stm32_adc_clr_bits(adc, STM32F4_ADC_CR1, STM32F4_EOCIE);
+ stm32_adc_clr_bits(adc, adc->cfg->regs->ier_eoc.reg,
+ adc->cfg->regs->ier_eoc.mask);
}
static void stm32_adc_set_res(struct stm32_adc *adc)
{
- u32 val = stm32_adc_readl(adc, STM32F4_ADC_CR1);
+ const struct stm32_adc_regs *res = &adc->cfg->regs->res;
+ u32 val;
- val = (val & ~STM32F4_RES_MASK) | (adc->res << STM32F4_RES_SHIFT);
- stm32_adc_writel(adc, STM32F4_ADC_CR1, val);
+ val = stm32_adc_readl(adc, res->reg);
+ val = (val & ~res->mask) | (adc->res << res->shift);
+ stm32_adc_writel(adc, res->reg, val);
}
/**
- * stm32_adc_start_conv() - Start conversions for regular channels.
+ * stm32f4_adc_start_conv() - Start conversions for regular channels.
* @adc: stm32 adc instance
* @dma: use dma to transfer conversion result
*
* conversions, in IIO buffer modes. Otherwise, use ADC interrupt with direct
* DR read instead (e.g. read_raw, or triggered buffer mode without DMA).
*/
-static void stm32_adc_start_conv(struct stm32_adc *adc, bool dma)
+static void stm32f4_adc_start_conv(struct stm32_adc *adc, bool dma)
{
stm32_adc_set_bits(adc, STM32F4_ADC_CR1, STM32F4_SCAN);
stm32_adc_set_bits(adc, STM32F4_ADC_CR2, STM32F4_SWSTART);
}
-static void stm32_adc_stop_conv(struct stm32_adc *adc)
+static void stm32f4_adc_stop_conv(struct stm32_adc *adc)
{
stm32_adc_clr_bits(adc, STM32F4_ADC_CR2, STM32F4_EXTEN_MASK);
stm32_adc_clr_bits(adc, STM32F4_ADC_SR, STM32F4_STRT);
STM32F4_ADON | STM32F4_DMA | STM32F4_DDS);
}
+static void stm32h7_adc_start_conv(struct stm32_adc *adc, bool dma)
+{
+ enum stm32h7_adc_dmngt dmngt;
+ unsigned long flags;
+ u32 val;
+
+ if (dma)
+ dmngt = STM32H7_DMNGT_DMA_CIRC;
+ else
+ dmngt = STM32H7_DMNGT_DR_ONLY;
+
+ spin_lock_irqsave(&adc->lock, flags);
+ val = stm32_adc_readl(adc, STM32H7_ADC_CFGR);
+ val = (val & ~STM32H7_DMNGT_MASK) | (dmngt << STM32H7_DMNGT_SHIFT);
+ stm32_adc_writel(adc, STM32H7_ADC_CFGR, val);
+ spin_unlock_irqrestore(&adc->lock, flags);
+
+ stm32_adc_set_bits(adc, STM32H7_ADC_CR, STM32H7_ADSTART);
+}
+
+static void stm32h7_adc_stop_conv(struct stm32_adc *adc)
+{
+ struct iio_dev *indio_dev = iio_priv_to_dev(adc);
+ int ret;
+ u32 val;
+
+ stm32_adc_set_bits(adc, STM32H7_ADC_CR, STM32H7_ADSTP);
+
+ ret = stm32_adc_readl_poll_timeout(STM32H7_ADC_CR, val,
+ !(val & (STM32H7_ADSTART)),
+ 100, STM32_ADC_TIMEOUT_US);
+ if (ret)
+ dev_warn(&indio_dev->dev, "stop failed\n");
+
+ stm32_adc_clr_bits(adc, STM32H7_ADC_CFGR, STM32H7_DMNGT_MASK);
+}
+
+static void stm32h7_adc_exit_pwr_down(struct stm32_adc *adc)
+{
+ /* Exit deep power down, then enable ADC voltage regulator */
+ stm32_adc_clr_bits(adc, STM32H7_ADC_CR, STM32H7_DEEPPWD);
+ stm32_adc_set_bits(adc, STM32H7_ADC_CR, STM32H7_ADVREGEN);
+
+ if (adc->common->rate > STM32H7_BOOST_CLKRATE)
+ stm32_adc_set_bits(adc, STM32H7_ADC_CR, STM32H7_BOOST);
+
+ /* Wait for startup time */
+ usleep_range(10, 20);
+}
+
+static void stm32h7_adc_enter_pwr_down(struct stm32_adc *adc)
+{
+ stm32_adc_clr_bits(adc, STM32H7_ADC_CR, STM32H7_BOOST);
+
+ /* Setting DEEPPWD disables ADC vreg and clears ADVREGEN */
+ stm32_adc_set_bits(adc, STM32H7_ADC_CR, STM32H7_DEEPPWD);
+}
+
+static int stm32h7_adc_enable(struct stm32_adc *adc)
+{
+ struct iio_dev *indio_dev = iio_priv_to_dev(adc);
+ int ret;
+ u32 val;
+
+ /* Clear ADRDY by writing one, then enable ADC */
+ stm32_adc_set_bits(adc, STM32H7_ADC_ISR, STM32H7_ADRDY);
+ stm32_adc_set_bits(adc, STM32H7_ADC_CR, STM32H7_ADEN);
+
+ /* Poll for ADRDY to be set (after adc startup time) */
+ ret = stm32_adc_readl_poll_timeout(STM32H7_ADC_ISR, val,
+ val & STM32H7_ADRDY,
+ 100, STM32_ADC_TIMEOUT_US);
+ if (ret) {
+ stm32_adc_clr_bits(adc, STM32H7_ADC_CR, STM32H7_ADEN);
+ dev_err(&indio_dev->dev, "Failed to enable ADC\n");
+ }
+
+ return ret;
+}
+
+static void stm32h7_adc_disable(struct stm32_adc *adc)
+{
+ struct iio_dev *indio_dev = iio_priv_to_dev(adc);
+ int ret;
+ u32 val;
+
+ /* Disable ADC and wait until it's effectively disabled */
+ stm32_adc_set_bits(adc, STM32H7_ADC_CR, STM32H7_ADDIS);
+ ret = stm32_adc_readl_poll_timeout(STM32H7_ADC_CR, val,
+ !(val & STM32H7_ADEN), 100,
+ STM32_ADC_TIMEOUT_US);
+ if (ret)
+ dev_warn(&indio_dev->dev, "Failed to disable\n");
+}
+
+/**
+ * stm32h7_adc_read_selfcalib() - read calibration shadow regs, save result
+ * @adc: stm32 adc instance
+ */
+static int stm32h7_adc_read_selfcalib(struct stm32_adc *adc)
+{
+ struct iio_dev *indio_dev = iio_priv_to_dev(adc);
+ int i, ret;
+ u32 lincalrdyw_mask, val;
+
+ /* Enable adc so LINCALRDYW1..6 bits are writable */
+ ret = stm32h7_adc_enable(adc);
+ if (ret)
+ return ret;
+
+ /* Read linearity calibration */
+ lincalrdyw_mask = STM32H7_LINCALRDYW6;
+ for (i = STM32H7_LINCALFACT_NUM - 1; i >= 0; i--) {
+ /* Clear STM32H7_LINCALRDYW[6..1]: transfer calib to CALFACT2 */
+ stm32_adc_clr_bits(adc, STM32H7_ADC_CR, lincalrdyw_mask);
+
+ /* Poll: wait calib data to be ready in CALFACT2 register */
+ ret = stm32_adc_readl_poll_timeout(STM32H7_ADC_CR, val,
+ !(val & lincalrdyw_mask),
+ 100, STM32_ADC_TIMEOUT_US);
+ if (ret) {
+ dev_err(&indio_dev->dev, "Failed to read calfact\n");
+ goto disable;
+ }
+
+ val = stm32_adc_readl(adc, STM32H7_ADC_CALFACT2);
+ adc->cal.lincalfact[i] = (val & STM32H7_LINCALFACT_MASK);
+ adc->cal.lincalfact[i] >>= STM32H7_LINCALFACT_SHIFT;
+
+ lincalrdyw_mask >>= 1;
+ }
+
+ /* Read offset calibration */
+ val = stm32_adc_readl(adc, STM32H7_ADC_CALFACT);
+ adc->cal.calfact_s = (val & STM32H7_CALFACT_S_MASK);
+ adc->cal.calfact_s >>= STM32H7_CALFACT_S_SHIFT;
+ adc->cal.calfact_d = (val & STM32H7_CALFACT_D_MASK);
+ adc->cal.calfact_d >>= STM32H7_CALFACT_D_SHIFT;
+
+disable:
+ stm32h7_adc_disable(adc);
+
+ return ret;
+}
+
+/**
+ * stm32h7_adc_restore_selfcalib() - Restore saved self-calibration result
+ * @adc: stm32 adc instance
+ * Note: ADC must be enabled, with no on-going conversions.
+ */
+static int stm32h7_adc_restore_selfcalib(struct stm32_adc *adc)
+{
+ struct iio_dev *indio_dev = iio_priv_to_dev(adc);
+ int i, ret;
+ u32 lincalrdyw_mask, val;
+
+ val = (adc->cal.calfact_s << STM32H7_CALFACT_S_SHIFT) |
+ (adc->cal.calfact_d << STM32H7_CALFACT_D_SHIFT);
+ stm32_adc_writel(adc, STM32H7_ADC_CALFACT, val);
+
+ lincalrdyw_mask = STM32H7_LINCALRDYW6;
+ for (i = STM32H7_LINCALFACT_NUM - 1; i >= 0; i--) {
+ /*
+ * Write saved calibration data to shadow registers:
+ * Write CALFACT2, and set LINCALRDYW[6..1] bit to trigger
+ * data write. Then poll to wait for complete transfer.
+ */
+ val = adc->cal.lincalfact[i] << STM32H7_LINCALFACT_SHIFT;
+ stm32_adc_writel(adc, STM32H7_ADC_CALFACT2, val);
+ stm32_adc_set_bits(adc, STM32H7_ADC_CR, lincalrdyw_mask);
+ ret = stm32_adc_readl_poll_timeout(STM32H7_ADC_CR, val,
+ val & lincalrdyw_mask,
+ 100, STM32_ADC_TIMEOUT_US);
+ if (ret) {
+ dev_err(&indio_dev->dev, "Failed to write calfact\n");
+ return ret;
+ }
+
+ /*
+ * Read back calibration data, has two effects:
+ * - It ensures bits LINCALRDYW[6..1] are kept cleared
+ * for next time calibration needs to be restored.
+ * - BTW, bit clear triggers a read, then check data has been
+ * correctly written.
+ */
+ stm32_adc_clr_bits(adc, STM32H7_ADC_CR, lincalrdyw_mask);
+ ret = stm32_adc_readl_poll_timeout(STM32H7_ADC_CR, val,
+ !(val & lincalrdyw_mask),
+ 100, STM32_ADC_TIMEOUT_US);
+ if (ret) {
+ dev_err(&indio_dev->dev, "Failed to read calfact\n");
+ return ret;
+ }
+ val = stm32_adc_readl(adc, STM32H7_ADC_CALFACT2);
+ if (val != adc->cal.lincalfact[i] << STM32H7_LINCALFACT_SHIFT) {
+ dev_err(&indio_dev->dev, "calfact not consistent\n");
+ return -EIO;
+ }
+
+ lincalrdyw_mask >>= 1;
+ }
+
+ return 0;
+}
+
+/**
+ * Fixed timeout value for ADC calibration.
+ * worst cases:
+ * - low clock frequency
+ * - maximum prescalers
+ * Calibration requires:
+ * - 131,072 ADC clock cycle for the linear calibration
+ * - 20 ADC clock cycle for the offset calibration
+ *
+ * Set to 100ms for now
+ */
+#define STM32H7_ADC_CALIB_TIMEOUT_US 100000
+
+/**
+ * stm32h7_adc_selfcalib() - Procedure to calibrate ADC (from power down)
+ * @adc: stm32 adc instance
+ * Exit from power down, calibrate ADC, then return to power down.
+ */
+static int stm32h7_adc_selfcalib(struct stm32_adc *adc)
+{
+ struct iio_dev *indio_dev = iio_priv_to_dev(adc);
+ int ret;
+ u32 val;
+
+ stm32h7_adc_exit_pwr_down(adc);
+
+ /*
+ * Select calibration mode:
+ * - Offset calibration for single ended inputs
+ * - No linearity calibration (do it later, before reading it)
+ */
+ stm32_adc_clr_bits(adc, STM32H7_ADC_CR, STM32H7_ADCALDIF);
+ stm32_adc_clr_bits(adc, STM32H7_ADC_CR, STM32H7_ADCALLIN);
+
+ /* Start calibration, then wait for completion */
+ stm32_adc_set_bits(adc, STM32H7_ADC_CR, STM32H7_ADCAL);
+ ret = stm32_adc_readl_poll_timeout(STM32H7_ADC_CR, val,
+ !(val & STM32H7_ADCAL), 100,
+ STM32H7_ADC_CALIB_TIMEOUT_US);
+ if (ret) {
+ dev_err(&indio_dev->dev, "calibration failed\n");
+ goto pwr_dwn;
+ }
+
+ /*
+ * Select calibration mode, then start calibration:
+ * - Offset calibration for differential input
+ * - Linearity calibration (needs to be done only once for single/diff)
+ * will run simultaneously with offset calibration.
+ */
+ stm32_adc_set_bits(adc, STM32H7_ADC_CR,
+ STM32H7_ADCALDIF | STM32H7_ADCALLIN);
+ stm32_adc_set_bits(adc, STM32H7_ADC_CR, STM32H7_ADCAL);
+ ret = stm32_adc_readl_poll_timeout(STM32H7_ADC_CR, val,
+ !(val & STM32H7_ADCAL), 100,
+ STM32H7_ADC_CALIB_TIMEOUT_US);
+ if (ret) {
+ dev_err(&indio_dev->dev, "calibration failed\n");
+ goto pwr_dwn;
+ }
+
+ stm32_adc_clr_bits(adc, STM32H7_ADC_CR,
+ STM32H7_ADCALDIF | STM32H7_ADCALLIN);
+
+ /* Read calibration result for future reference */
+ ret = stm32h7_adc_read_selfcalib(adc);
+
+pwr_dwn:
+ stm32h7_adc_enter_pwr_down(adc);
+
+ return ret;
+}
+
+/**
+ * stm32h7_adc_prepare() - Leave power down mode to enable ADC.
+ * @adc: stm32 adc instance
+ * Leave power down mode.
+ * Enable ADC.
+ * Restore calibration data.
+ * Pre-select channels that may be used in PCSEL (required by input MUX / IO).
+ */
+static int stm32h7_adc_prepare(struct stm32_adc *adc)
+{
+ int ret;
+
+ stm32h7_adc_exit_pwr_down(adc);
+
+ ret = stm32h7_adc_enable(adc);
+ if (ret)
+ goto pwr_dwn;
+
+ ret = stm32h7_adc_restore_selfcalib(adc);
+ if (ret)
+ goto disable;
+
+ stm32_adc_writel(adc, STM32H7_ADC_PCSEL, adc->pcsel);
+
+ return 0;
+
+disable:
+ stm32h7_adc_disable(adc);
+pwr_dwn:
+ stm32h7_adc_enter_pwr_down(adc);
+
+ return ret;
+}
+
+static void stm32h7_adc_unprepare(struct stm32_adc *adc)
+{
+ stm32h7_adc_disable(adc);
+ stm32h7_adc_enter_pwr_down(adc);
+}
+
/**
* stm32_adc_conf_scan_seq() - Build regular channels scan sequence
* @indio_dev: IIO device
const unsigned long *scan_mask)
{
struct stm32_adc *adc = iio_priv(indio_dev);
+ const struct stm32_adc_regs *sqr = adc->cfg->regs->sqr;
const struct iio_chan_spec *chan;
u32 val, bit;
int i = 0;
dev_dbg(&indio_dev->dev, "%s chan %d to SQ%d\n",
__func__, chan->channel, i);
- val = stm32_adc_readl(adc, stm32f4_sq[i].reg);
- val &= ~stm32f4_sq[i].mask;
- val |= chan->channel << stm32f4_sq[i].shift;
- stm32_adc_writel(adc, stm32f4_sq[i].reg, val);
+ val = stm32_adc_readl(adc, sqr[i].reg);
+ val &= ~sqr[i].mask;
+ val |= chan->channel << sqr[i].shift;
+ stm32_adc_writel(adc, sqr[i].reg, val);
}
if (!i)
return -EINVAL;
/* Sequence len */
- val = stm32_adc_readl(adc, stm32f4_sq[0].reg);
- val &= ~stm32f4_sq[0].mask;
- val |= ((i - 1) << stm32f4_sq[0].shift);
- stm32_adc_writel(adc, stm32f4_sq[0].reg, val);
+ val = stm32_adc_readl(adc, sqr[0].reg);
+ val &= ~sqr[0].mask;
+ val |= ((i - 1) << sqr[0].shift);
+ stm32_adc_writel(adc, sqr[0].reg, val);
return 0;
}
*
* Returns trigger extsel value, if trig matches, -EINVAL otherwise.
*/
-static int stm32_adc_get_trig_extsel(struct iio_trigger *trig)
+static int stm32_adc_get_trig_extsel(struct iio_dev *indio_dev,
+ struct iio_trigger *trig)
{
+ struct stm32_adc *adc = iio_priv(indio_dev);
int i;
/* lookup triggers registered by stm32 timer trigger driver */
- for (i = 0; stm32f4_adc_trigs[i].name; i++) {
+ for (i = 0; adc->cfg->trigs[i].name; i++) {
/**
* Checking both stm32 timer trigger type and trig name
* should be safe against arbitrary trigger names.
*/
if (is_stm32_timer_trigger(trig) &&
- !strcmp(stm32f4_adc_trigs[i].name, trig->name)) {
- return stm32f4_adc_trigs[i].extsel;
+ !strcmp(adc->cfg->trigs[i].name, trig->name)) {
+ return adc->cfg->trigs[i].extsel;
}
}
int ret;
if (trig) {
- ret = stm32_adc_get_trig_extsel(trig);
+ ret = stm32_adc_get_trig_extsel(indio_dev, trig);
if (ret < 0)
return ret;
}
spin_lock_irqsave(&adc->lock, flags);
- val = stm32_adc_readl(adc, STM32F4_ADC_CR2);
- val &= ~(STM32F4_EXTEN_MASK | STM32F4_EXTSEL_MASK);
- val |= exten << STM32F4_EXTEN_SHIFT;
- val |= extsel << STM32F4_EXTSEL_SHIFT;
- stm32_adc_writel(adc, STM32F4_ADC_CR2, val);
+ val = stm32_adc_readl(adc, adc->cfg->regs->exten.reg);
+ val &= ~(adc->cfg->regs->exten.mask | adc->cfg->regs->extsel.mask);
+ val |= exten << adc->cfg->regs->exten.shift;
+ val |= extsel << adc->cfg->regs->extsel.shift;
+ stm32_adc_writel(adc, adc->cfg->regs->exten.reg, val);
spin_unlock_irqrestore(&adc->lock, flags);
return 0;
int *res)
{
struct stm32_adc *adc = iio_priv(indio_dev);
+ const struct stm32_adc_regspec *regs = adc->cfg->regs;
long timeout;
u32 val;
int ret;
adc->bufi = 0;
+ if (adc->cfg->prepare) {
+ ret = adc->cfg->prepare(adc);
+ if (ret)
+ return ret;
+ }
+
/* Program chan number in regular sequence (SQ1) */
- val = stm32_adc_readl(adc, stm32f4_sq[1].reg);
- val &= ~stm32f4_sq[1].mask;
- val |= chan->channel << stm32f4_sq[1].shift;
- stm32_adc_writel(adc, stm32f4_sq[1].reg, val);
+ val = stm32_adc_readl(adc, regs->sqr[1].reg);
+ val &= ~regs->sqr[1].mask;
+ val |= chan->channel << regs->sqr[1].shift;
+ stm32_adc_writel(adc, regs->sqr[1].reg, val);
/* Set regular sequence len (0 for 1 conversion) */
- stm32_adc_clr_bits(adc, stm32f4_sq[0].reg, stm32f4_sq[0].mask);
+ stm32_adc_clr_bits(adc, regs->sqr[0].reg, regs->sqr[0].mask);
/* Trigger detection disabled (conversion can be launched in SW) */
- stm32_adc_clr_bits(adc, STM32F4_ADC_CR2, STM32F4_EXTEN_MASK);
+ stm32_adc_clr_bits(adc, regs->exten.reg, regs->exten.mask);
stm32_adc_conv_irq_enable(adc);
- stm32_adc_start_conv(adc, false);
+ adc->cfg->start_conv(adc, false);
timeout = wait_for_completion_interruptible_timeout(
&adc->completion, STM32_ADC_TIMEOUT);
ret = IIO_VAL_INT;
}
- stm32_adc_stop_conv(adc);
+ adc->cfg->stop_conv(adc);
stm32_adc_conv_irq_disable(adc);
+ if (adc->cfg->unprepare)
+ adc->cfg->unprepare(adc);
+
return ret;
}
{
struct stm32_adc *adc = data;
struct iio_dev *indio_dev = iio_priv_to_dev(adc);
- u32 status = stm32_adc_readl(adc, STM32F4_ADC_SR);
+ const struct stm32_adc_regspec *regs = adc->cfg->regs;
+ u32 status = stm32_adc_readl(adc, regs->isr_eoc.reg);
- if (status & STM32F4_EOC) {
+ if (status & regs->isr_eoc.mask) {
/* Reading DR also clears EOC status flag */
- adc->buffer[adc->bufi] = stm32_adc_readw(adc, STM32F4_ADC_DR);
+ adc->buffer[adc->bufi] = stm32_adc_readw(adc, regs->dr);
if (iio_buffer_enabled(indio_dev)) {
adc->bufi++;
if (adc->bufi >= adc->num_conv) {
static int stm32_adc_validate_trigger(struct iio_dev *indio_dev,
struct iio_trigger *trig)
{
- return stm32_adc_get_trig_extsel(trig) < 0 ? -EINVAL : 0;
+ return stm32_adc_get_trig_extsel(indio_dev, trig) < 0 ? -EINVAL : 0;
}
static int stm32_adc_set_watermark(struct iio_dev *indio_dev, unsigned int val)
struct stm32_adc *adc = iio_priv(indio_dev);
int ret;
+ if (adc->cfg->prepare) {
+ ret = adc->cfg->prepare(adc);
+ if (ret)
+ return ret;
+ }
+
ret = stm32_adc_set_trig(indio_dev, indio_dev->trig);
if (ret) {
dev_err(&indio_dev->dev, "Can't set trigger\n");
- return ret;
+ goto err_unprepare;
}
ret = stm32_adc_dma_start(indio_dev);
if (!adc->dma_chan)
stm32_adc_conv_irq_enable(adc);
- stm32_adc_start_conv(adc, !!adc->dma_chan);
+ adc->cfg->start_conv(adc, !!adc->dma_chan);
return 0;
dmaengine_terminate_all(adc->dma_chan);
err_clr_trig:
stm32_adc_set_trig(indio_dev, NULL);
+err_unprepare:
+ if (adc->cfg->unprepare)
+ adc->cfg->unprepare(adc);
return ret;
}
struct stm32_adc *adc = iio_priv(indio_dev);
int ret;
- stm32_adc_stop_conv(adc);
+ adc->cfg->stop_conv(adc);
if (!adc->dma_chan)
stm32_adc_conv_irq_disable(adc);
if (stm32_adc_set_trig(indio_dev, NULL))
dev_err(&indio_dev->dev, "Can't clear trigger\n");
+ if (adc->cfg->unprepare)
+ adc->cfg->unprepare(adc);
+
return ret;
}
u32 res;
if (of_property_read_u32(node, "assigned-resolution-bits", &res))
- res = stm32f4_adc_resolutions[0];
+ res = adc->cfg->adc_info->resolutions[0];
- for (i = 0; i < ARRAY_SIZE(stm32f4_adc_resolutions); i++)
- if (res == stm32f4_adc_resolutions[i])
+ for (i = 0; i < adc->cfg->adc_info->num_res; i++)
+ if (res == adc->cfg->adc_info->resolutions[i])
break;
- if (i >= ARRAY_SIZE(stm32f4_adc_resolutions)) {
+ if (i >= adc->cfg->adc_info->num_res) {
dev_err(&indio_dev->dev, "Bad resolution: %u bits\n", res);
return -EINVAL;
}
chan->info_mask_separate = BIT(IIO_CHAN_INFO_RAW);
chan->info_mask_shared_by_type = BIT(IIO_CHAN_INFO_SCALE);
chan->scan_type.sign = 'u';
- chan->scan_type.realbits = stm32f4_adc_resolutions[adc->res];
+ chan->scan_type.realbits = adc->cfg->adc_info->resolutions[adc->res];
chan->scan_type.storagebits = 16;
chan->ext_info = stm32_adc_ext_info;
+
+ /* pre-build selected channels mask */
+ adc->pcsel |= BIT(chan->channel);
}
static int stm32_adc_chan_of_init(struct iio_dev *indio_dev)
{
struct device_node *node = indio_dev->dev.of_node;
+ struct stm32_adc *adc = iio_priv(indio_dev);
+ const struct stm32_adc_info *adc_info = adc->cfg->adc_info;
struct property *prop;
const __be32 *cur;
struct iio_chan_spec *channels;
num_channels = of_property_count_u32_elems(node, "st,adc-channels");
if (num_channels < 0 ||
- num_channels >= ARRAY_SIZE(stm32f4_adc123_channels)) {
+ num_channels >= adc_info->max_channels) {
dev_err(&indio_dev->dev, "Bad st,adc-channels?\n");
return num_channels < 0 ? num_channels : -EINVAL;
}
return -ENOMEM;
of_property_for_each_u32(node, "st,adc-channels", prop, cur, val) {
- if (val >= ARRAY_SIZE(stm32f4_adc123_channels)) {
+ if (val >= adc_info->max_channels) {
dev_err(&indio_dev->dev, "Invalid channel %d\n", val);
return -EINVAL;
}
stm32_adc_chan_init_one(indio_dev, &channels[scan_index],
- &stm32f4_adc123_channels[val],
+ &adc_info->channels[val],
scan_index);
scan_index++;
}
/* Configure DMA channel to read data register */
memset(&config, 0, sizeof(config));
config.src_addr = (dma_addr_t)adc->common->phys_base;
- config.src_addr += adc->offset + STM32F4_ADC_DR;
+ config.src_addr += adc->offset + adc->cfg->regs->dr;
config.src_addr_width = DMA_SLAVE_BUSWIDTH_2_BYTES;
ret = dmaengine_slave_config(adc->dma_chan, &config);
static int stm32_adc_probe(struct platform_device *pdev)
{
struct iio_dev *indio_dev;
+ struct device *dev = &pdev->dev;
struct stm32_adc *adc;
int ret;
adc->common = dev_get_drvdata(pdev->dev.parent);
spin_lock_init(&adc->lock);
init_completion(&adc->completion);
+ adc->cfg = (const struct stm32_adc_cfg *)
+ of_match_device(dev->driver->of_match_table, dev)->data;
indio_dev->name = dev_name(&pdev->dev);
indio_dev->dev.parent = &pdev->dev;
adc->clk = devm_clk_get(&pdev->dev, NULL);
if (IS_ERR(adc->clk)) {
- dev_err(&pdev->dev, "Can't get clock\n");
- return PTR_ERR(adc->clk);
+ ret = PTR_ERR(adc->clk);
+ if (ret == -ENOENT && !adc->cfg->clk_required) {
+ adc->clk = NULL;
+ } else {
+ dev_err(&pdev->dev, "Can't get clock\n");
+ return ret;
+ }
}
- ret = clk_prepare_enable(adc->clk);
- if (ret < 0) {
- dev_err(&pdev->dev, "clk enable failed\n");
- return ret;
+ if (adc->clk) {
+ ret = clk_prepare_enable(adc->clk);
+ if (ret < 0) {
+ dev_err(&pdev->dev, "clk enable failed\n");
+ return ret;
+ }
}
ret = stm32_adc_of_get_resolution(indio_dev);
goto err_clk_disable;
stm32_adc_set_res(adc);
+ if (adc->cfg->selfcalib) {
+ ret = adc->cfg->selfcalib(adc);
+ if (ret)
+ goto err_clk_disable;
+ }
+
ret = stm32_adc_chan_of_init(indio_dev);
if (ret < 0)
goto err_clk_disable;
dma_release_channel(adc->dma_chan);
}
err_clk_disable:
- clk_disable_unprepare(adc->clk);
+ if (adc->clk)
+ clk_disable_unprepare(adc->clk);
return ret;
}
adc->rx_buf, adc->rx_dma_buf);
dma_release_channel(adc->dma_chan);
}
- clk_disable_unprepare(adc->clk);
+ if (adc->clk)
+ clk_disable_unprepare(adc->clk);
return 0;
}
+static const struct stm32_adc_cfg stm32f4_adc_cfg = {
+ .regs = &stm32f4_adc_regspec,
+ .adc_info = &stm32f4_adc_info,
+ .trigs = stm32f4_adc_trigs,
+ .clk_required = true,
+ .start_conv = stm32f4_adc_start_conv,
+ .stop_conv = stm32f4_adc_stop_conv,
+};
+
+static const struct stm32_adc_cfg stm32h7_adc_cfg = {
+ .regs = &stm32h7_adc_regspec,
+ .adc_info = &stm32h7_adc_info,
+ .trigs = stm32h7_adc_trigs,
+ .selfcalib = stm32h7_adc_selfcalib,
+ .start_conv = stm32h7_adc_start_conv,
+ .stop_conv = stm32h7_adc_stop_conv,
+ .prepare = stm32h7_adc_prepare,
+ .unprepare = stm32h7_adc_unprepare,
+};
+
static const struct of_device_id stm32_adc_of_match[] = {
- { .compatible = "st,stm32f4-adc" },
+ { .compatible = "st,stm32f4-adc", .data = (void *)&stm32f4_adc_cfg },
+ { .compatible = "st,stm32h7-adc", .data = (void *)&stm32h7_adc_cfg },
{},
};
MODULE_DEVICE_TABLE(of, stm32_adc_of_match);
--- /dev/null
+/**
+ * Copyright (C) 2017 Axis Communications AB
+ *
+ * Driver for Texas Instruments' ADC084S021 ADC chip.
+ * Datasheets can be found here:
+ * http://www.ti.com/lit/ds/symlink/adc084s021.pdf
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/err.h>
+#include <linux/spi/spi.h>
+#include <linux/module.h>
+#include <linux/interrupt.h>
+#include <linux/iio/iio.h>
+#include <linux/iio/buffer.h>
+#include <linux/iio/triggered_buffer.h>
+#include <linux/iio/trigger_consumer.h>
+#include <linux/regulator/consumer.h>
+
+#define ADC084S021_DRIVER_NAME "adc084s021"
+
+struct adc084s021 {
+ struct spi_device *spi;
+ struct spi_message message;
+ struct spi_transfer spi_trans;
+ struct regulator *reg;
+ struct mutex lock;
+ /*
+ * DMA (thus cache coherency maintenance) requires the
+ * transfer buffers to live in their own cache line.
+ */
+ u16 tx_buf[4] ____cacheline_aligned;
+ __be16 rx_buf[5]; /* First 16-bits are trash */
+};
+
+#define ADC084S021_VOLTAGE_CHANNEL(num) \
+ { \
+ .type = IIO_VOLTAGE, \
+ .channel = (num), \
+ .indexed = 1, \
+ .scan_index = (num), \
+ .scan_type = { \
+ .sign = 'u', \
+ .realbits = 8, \
+ .storagebits = 16, \
+ .shift = 4, \
+ .endianness = IIO_BE, \
+ }, \
+ .info_mask_separate = BIT(IIO_CHAN_INFO_RAW), \
+ .info_mask_shared_by_type = BIT(IIO_CHAN_INFO_SCALE),\
+ }
+
+static const struct iio_chan_spec adc084s021_channels[] = {
+ ADC084S021_VOLTAGE_CHANNEL(0),
+ ADC084S021_VOLTAGE_CHANNEL(1),
+ ADC084S021_VOLTAGE_CHANNEL(2),
+ ADC084S021_VOLTAGE_CHANNEL(3),
+ IIO_CHAN_SOFT_TIMESTAMP(4),
+};
+
+/**
+ * Read an ADC channel and return its value.
+ *
+ * @adc: The ADC SPI data.
+ * @data: Buffer for converted data.
+ */
+static int adc084s021_adc_conversion(struct adc084s021 *adc, void *data)
+{
+ int n_words = (adc->spi_trans.len >> 1) - 1; /* Discard first word */
+ int ret, i = 0;
+ u16 *p = data;
+
+ /* Do the transfer */
+ ret = spi_sync(adc->spi, &adc->message);
+ if (ret < 0)
+ return ret;
+
+ for (; i < n_words; i++)
+ *(p + i) = adc->rx_buf[i + 1];
+
+ return ret;
+}
+
+static int adc084s021_read_raw(struct iio_dev *indio_dev,
+ struct iio_chan_spec const *channel, int *val,
+ int *val2, long mask)
+{
+ struct adc084s021 *adc = iio_priv(indio_dev);
+ int ret;
+
+ switch (mask) {
+ case IIO_CHAN_INFO_RAW:
+ ret = iio_device_claim_direct_mode(indio_dev);
+ if (ret < 0)
+ return ret;
+
+ ret = regulator_enable(adc->reg);
+ if (ret) {
+ iio_device_release_direct_mode(indio_dev);
+ return ret;
+ }
+
+ adc->tx_buf[0] = channel->channel << 3;
+ ret = adc084s021_adc_conversion(adc, val);
+ iio_device_release_direct_mode(indio_dev);
+ regulator_disable(adc->reg);
+ if (ret < 0)
+ return ret;
+
+ *val = be16_to_cpu(*val);
+ *val = (*val >> channel->scan_type.shift) & 0xff;
+
+ return IIO_VAL_INT;
+ case IIO_CHAN_INFO_SCALE:
+ ret = regulator_enable(adc->reg);
+ if (ret)
+ return ret;
+
+ ret = regulator_get_voltage(adc->reg);
+ regulator_disable(adc->reg);
+ if (ret < 0)
+ return ret;
+
+ *val = ret / 1000;
+
+ return IIO_VAL_INT;
+ default:
+ return -EINVAL;
+ }
+}
+
+/**
+ * Read enabled ADC channels and push data to the buffer.
+ *
+ * @irq: The interrupt number (not used).
+ * @pollfunc: Pointer to the poll func.
+ */
+static irqreturn_t adc084s021_buffer_trigger_handler(int irq, void *pollfunc)
+{
+ struct iio_poll_func *pf = pollfunc;
+ struct iio_dev *indio_dev = pf->indio_dev;
+ struct adc084s021 *adc = iio_priv(indio_dev);
+ __be16 data[8] = {0}; /* 4 * 16-bit words of data + 8 bytes timestamp */
+
+ mutex_lock(&adc->lock);
+
+ if (adc084s021_adc_conversion(adc, &data) < 0)
+ dev_err(&adc->spi->dev, "Failed to read data\n");
+
+ iio_push_to_buffers_with_timestamp(indio_dev, data,
+ iio_get_time_ns(indio_dev));
+ mutex_unlock(&adc->lock);
+ iio_trigger_notify_done(indio_dev->trig);
+
+ return IRQ_HANDLED;
+}
+
+static int adc084s021_buffer_preenable(struct iio_dev *indio_dev)
+{
+ struct adc084s021 *adc = iio_priv(indio_dev);
+ int scan_index;
+ int i = 0;
+
+ for_each_set_bit(scan_index, indio_dev->active_scan_mask,
+ indio_dev->masklength) {
+ const struct iio_chan_spec *channel =
+ &indio_dev->channels[scan_index];
+ adc->tx_buf[i++] = channel->channel << 3;
+ }
+ adc->spi_trans.len = 2 + (i * sizeof(__be16)); /* Trash + channels */
+
+ return regulator_enable(adc->reg);
+}
+
+static int adc084s021_buffer_postdisable(struct iio_dev *indio_dev)
+{
+ struct adc084s021 *adc = iio_priv(indio_dev);
+
+ adc->spi_trans.len = 4; /* Trash + single channel */
+
+ return regulator_disable(adc->reg);
+}
+
+static const struct iio_info adc084s021_info = {
+ .read_raw = adc084s021_read_raw,
+ .driver_module = THIS_MODULE,
+};
+
+static const struct iio_buffer_setup_ops adc084s021_buffer_setup_ops = {
+ .preenable = adc084s021_buffer_preenable,
+ .postenable = iio_triggered_buffer_postenable,
+ .predisable = iio_triggered_buffer_predisable,
+ .postdisable = adc084s021_buffer_postdisable,
+};
+
+static int adc084s021_probe(struct spi_device *spi)
+{
+ struct iio_dev *indio_dev;
+ struct adc084s021 *adc;
+ int ret;
+
+ indio_dev = devm_iio_device_alloc(&spi->dev, sizeof(*adc));
+ if (!indio_dev) {
+ dev_err(&spi->dev, "Failed to allocate IIO device\n");
+ return -ENOMEM;
+ }
+
+ adc = iio_priv(indio_dev);
+ adc->spi = spi;
+
+ /* Connect the SPI device and the iio dev */
+ spi_set_drvdata(spi, indio_dev);
+
+ /* Initiate the Industrial I/O device */
+ indio_dev->dev.parent = &spi->dev;
+ indio_dev->dev.of_node = spi->dev.of_node;
+ indio_dev->name = spi_get_device_id(spi)->name;
+ indio_dev->modes = INDIO_DIRECT_MODE;
+ indio_dev->info = &adc084s021_info;
+ indio_dev->channels = adc084s021_channels;
+ indio_dev->num_channels = ARRAY_SIZE(adc084s021_channels);
+
+ /* Create SPI transfer for channel reads */
+ adc->spi_trans.tx_buf = adc->tx_buf;
+ adc->spi_trans.rx_buf = adc->rx_buf;
+ adc->spi_trans.len = 4; /* Trash + single channel */
+ spi_message_init_with_transfers(&adc->message, &adc->spi_trans, 1);
+
+ adc->reg = devm_regulator_get(&spi->dev, "vref");
+ if (IS_ERR(adc->reg))
+ return PTR_ERR(adc->reg);
+
+ mutex_init(&adc->lock);
+
+ /* Setup triggered buffer with pollfunction */
+ ret = devm_iio_triggered_buffer_setup(&spi->dev, indio_dev, NULL,
+ adc084s021_buffer_trigger_handler,
+ &adc084s021_buffer_setup_ops);
+ if (ret) {
+ dev_err(&spi->dev, "Failed to setup triggered buffer\n");
+ return ret;
+ }
+
+ return devm_iio_device_register(&spi->dev, indio_dev);
+}
+
+static const struct of_device_id adc084s021_of_match[] = {
+ { .compatible = "ti,adc084s021", },
+ {},
+};
+MODULE_DEVICE_TABLE(of, adc084s021_of_match);
+
+static const struct spi_device_id adc084s021_id[] = {
+ { ADC084S021_DRIVER_NAME, 0},
+ {}
+};
+MODULE_DEVICE_TABLE(spi, adc084s021_id);
+
+static struct spi_driver adc084s021_driver = {
+ .driver = {
+ .name = ADC084S021_DRIVER_NAME,
+ .of_match_table = of_match_ptr(adc084s021_of_match),
+ },
+ .probe = adc084s021_probe,
+ .id_table = adc084s021_id,
+};
+module_spi_driver(adc084s021_driver);
+
+MODULE_AUTHOR("Mårten Lindahl <martenli@axis.com>");
+MODULE_DESCRIPTION("Texas Instruments ADC084S021");
+MODULE_LICENSE("GPL v2");
+MODULE_VERSION("1.0");
--- /dev/null
+/*
+ * TI ADC108S102 SPI ADC driver
+ *
+ * Copyright (c) 2013-2015 Intel Corporation.
+ * Copyright (c) 2017 Siemens AG
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ * This IIO device driver is designed to work with the following
+ * analog to digital converters from Texas Instruments:
+ * ADC108S102
+ * ADC128S102
+ * The communication with ADC chip is via the SPI bus (mode 3).
+ */
+
+#include <linux/acpi.h>
+#include <linux/iio/iio.h>
+#include <linux/iio/buffer.h>
+#include <linux/iio/types.h>
+#include <linux/iio/triggered_buffer.h>
+#include <linux/iio/trigger_consumer.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/property.h>
+#include <linux/regulator/consumer.h>
+#include <linux/spi/spi.h>
+
+/*
+ * In case of ACPI, we use the hard-wired 5000 mV of the Galileo and IOT2000
+ * boards as default for the reference pin VA. Device tree users encode that
+ * via the vref-supply regulator.
+ */
+#define ADC108S102_VA_MV_ACPI_DEFAULT 5000
+
+/*
+ * Defining the ADC resolution being 12 bits, we can use the same driver for
+ * both ADC108S102 (10 bits resolution) and ADC128S102 (12 bits resolution)
+ * chips. The ADC108S102 effectively returns a 12-bit result with the 2
+ * least-significant bits unset.
+ */
+#define ADC108S102_BITS 12
+#define ADC108S102_MAX_CHANNELS 8
+
+/*
+ * 16-bit SPI command format:
+ * [15:14] Ignored
+ * [13:11] 3-bit channel address
+ * [10:0] Ignored
+ */
+#define ADC108S102_CMD(ch) ((u16)(ch) << 11)
+
+/*
+ * 16-bit SPI response format:
+ * [15:12] Zeros
+ * [11:0] 12-bit ADC sample (for ADC108S102, [1:0] will always be 0).
+ */
+#define ADC108S102_RES_DATA(res) ((u16)res & GENMASK(11, 0))
+
+struct adc108s102_state {
+ struct spi_device *spi;
+ struct regulator *reg;
+ u32 va_millivolt;
+ /* SPI transfer used by triggered buffer handler*/
+ struct spi_transfer ring_xfer;
+ /* SPI transfer used by direct scan */
+ struct spi_transfer scan_single_xfer;
+ /* SPI message used by ring_xfer SPI transfer */
+ struct spi_message ring_msg;
+ /* SPI message used by scan_single_xfer SPI transfer */
+ struct spi_message scan_single_msg;
+
+ /*
+ * SPI message buffers:
+ * tx_buf: |C0|C1|C2|C3|C4|C5|C6|C7|XX|
+ * rx_buf: |XX|R0|R1|R2|R3|R4|R5|R6|R7|tt|tt|tt|tt|
+ *
+ * tx_buf: 8 channel read commands, plus 1 dummy command
+ * rx_buf: 1 dummy response, 8 channel responses, plus 64-bit timestamp
+ */
+ __be16 rx_buf[13] ____cacheline_aligned;
+ __be16 tx_buf[9] ____cacheline_aligned;
+};
+
+#define ADC108S102_V_CHAN(index) \
+ { \
+ .type = IIO_VOLTAGE, \
+ .indexed = 1, \
+ .channel = index, \
+ .info_mask_separate = BIT(IIO_CHAN_INFO_RAW) | \
+ BIT(IIO_CHAN_INFO_SCALE), \
+ .address = index, \
+ .scan_index = index, \
+ .scan_type = { \
+ .sign = 'u', \
+ .realbits = ADC108S102_BITS, \
+ .storagebits = 16, \
+ .endianness = IIO_BE, \
+ }, \
+ }
+
+static const struct iio_chan_spec adc108s102_channels[] = {
+ ADC108S102_V_CHAN(0),
+ ADC108S102_V_CHAN(1),
+ ADC108S102_V_CHAN(2),
+ ADC108S102_V_CHAN(3),
+ ADC108S102_V_CHAN(4),
+ ADC108S102_V_CHAN(5),
+ ADC108S102_V_CHAN(6),
+ ADC108S102_V_CHAN(7),
+ IIO_CHAN_SOFT_TIMESTAMP(8),
+};
+
+static int adc108s102_update_scan_mode(struct iio_dev *indio_dev,
+ unsigned long const *active_scan_mask)
+{
+ struct adc108s102_state *st = iio_priv(indio_dev);
+ unsigned int bit, cmds;
+
+ /*
+ * Fill in the first x shorts of tx_buf with the number of channels
+ * enabled for sampling by the triggered buffer.
+ */
+ cmds = 0;
+ for_each_set_bit(bit, active_scan_mask, ADC108S102_MAX_CHANNELS)
+ st->tx_buf[cmds++] = cpu_to_be16(ADC108S102_CMD(bit));
+
+ /* One dummy command added, to clock in the last response */
+ st->tx_buf[cmds++] = 0x00;
+
+ /* build SPI ring message */
+ st->ring_xfer.tx_buf = &st->tx_buf[0];
+ st->ring_xfer.rx_buf = &st->rx_buf[0];
+ st->ring_xfer.len = cmds * sizeof(st->tx_buf[0]);
+
+ spi_message_init_with_transfers(&st->ring_msg, &st->ring_xfer, 1);
+
+ return 0;
+}
+
+static irqreturn_t adc108s102_trigger_handler(int irq, void *p)
+{
+ struct iio_poll_func *pf = p;
+ struct iio_dev *indio_dev = pf->indio_dev;
+ struct adc108s102_state *st = iio_priv(indio_dev);
+ int ret;
+
+ ret = spi_sync(st->spi, &st->ring_msg);
+ if (ret < 0)
+ goto out_notify;
+
+ /* Skip the dummy response in the first slot */
+ iio_push_to_buffers_with_timestamp(indio_dev,
+ (u8 *)&st->rx_buf[1],
+ iio_get_time_ns(indio_dev));
+
+out_notify:
+ iio_trigger_notify_done(indio_dev->trig);
+
+ return IRQ_HANDLED;
+}
+
+static int adc108s102_scan_direct(struct adc108s102_state *st, unsigned int ch)
+{
+ int ret;
+
+ st->tx_buf[0] = cpu_to_be16(ADC108S102_CMD(ch));
+ ret = spi_sync(st->spi, &st->scan_single_msg);
+ if (ret)
+ return ret;
+
+ /* Skip the dummy response in the first slot */
+ return be16_to_cpu(st->rx_buf[1]);
+}
+
+static int adc108s102_read_raw(struct iio_dev *indio_dev,
+ struct iio_chan_spec const *chan,
+ int *val, int *val2, long m)
+{
+ struct adc108s102_state *st = iio_priv(indio_dev);
+ int ret;
+
+ switch (m) {
+ case IIO_CHAN_INFO_RAW:
+ ret = iio_device_claim_direct_mode(indio_dev);
+ if (ret)
+ return ret;
+
+ ret = adc108s102_scan_direct(st, chan->address);
+
+ iio_device_release_direct_mode(indio_dev);
+
+ if (ret < 0)
+ return ret;
+
+ *val = ADC108S102_RES_DATA(ret);
+
+ return IIO_VAL_INT;
+ case IIO_CHAN_INFO_SCALE:
+ if (chan->type != IIO_VOLTAGE)
+ break;
+
+ *val = st->va_millivolt;
+ *val2 = chan->scan_type.realbits;
+
+ return IIO_VAL_FRACTIONAL_LOG2;
+ default:
+ break;
+ }
+
+ return -EINVAL;
+}
+
+static const struct iio_info adc108s102_info = {
+ .read_raw = &adc108s102_read_raw,
+ .update_scan_mode = &adc108s102_update_scan_mode,
+ .driver_module = THIS_MODULE,
+};
+
+static int adc108s102_probe(struct spi_device *spi)
+{
+ struct adc108s102_state *st;
+ struct iio_dev *indio_dev;
+ int ret;
+
+ indio_dev = devm_iio_device_alloc(&spi->dev, sizeof(*st));
+ if (!indio_dev)
+ return -ENOMEM;
+
+ st = iio_priv(indio_dev);
+
+ if (ACPI_COMPANION(&spi->dev)) {
+ st->va_millivolt = ADC108S102_VA_MV_ACPI_DEFAULT;
+ } else {
+ st->reg = devm_regulator_get(&spi->dev, "vref");
+ if (IS_ERR(st->reg))
+ return PTR_ERR(st->reg);
+
+ ret = regulator_enable(st->reg);
+ if (ret < 0) {
+ dev_err(&spi->dev, "Cannot enable vref regulator\n");
+ return ret;
+ }
+
+ ret = regulator_get_voltage(st->reg);
+ if (ret < 0) {
+ dev_err(&spi->dev, "vref get voltage failed\n");
+ return ret;
+ }
+
+ st->va_millivolt = ret / 1000;
+ }
+
+ spi_set_drvdata(spi, indio_dev);
+ st->spi = spi;
+
+ indio_dev->name = spi->modalias;
+ indio_dev->dev.parent = &spi->dev;
+ indio_dev->modes = INDIO_DIRECT_MODE;
+ indio_dev->channels = adc108s102_channels;
+ indio_dev->num_channels = ARRAY_SIZE(adc108s102_channels);
+ indio_dev->info = &adc108s102_info;
+
+ /* Setup default message */
+ st->scan_single_xfer.tx_buf = st->tx_buf;
+ st->scan_single_xfer.rx_buf = st->rx_buf;
+ st->scan_single_xfer.len = 2 * sizeof(st->tx_buf[0]);
+
+ spi_message_init_with_transfers(&st->scan_single_msg,
+ &st->scan_single_xfer, 1);
+
+ ret = iio_triggered_buffer_setup(indio_dev, NULL,
+ &adc108s102_trigger_handler, NULL);
+ if (ret)
+ goto error_disable_reg;
+
+ ret = iio_device_register(indio_dev);
+ if (ret) {
+ dev_err(&spi->dev, "Failed to register IIO device\n");
+ goto error_cleanup_triggered_buffer;
+ }
+ return 0;
+
+error_cleanup_triggered_buffer:
+ iio_triggered_buffer_cleanup(indio_dev);
+
+error_disable_reg:
+ regulator_disable(st->reg);
+
+ return ret;
+}
+
+static int adc108s102_remove(struct spi_device *spi)
+{
+ struct iio_dev *indio_dev = spi_get_drvdata(spi);
+ struct adc108s102_state *st = iio_priv(indio_dev);
+
+ iio_device_unregister(indio_dev);
+ iio_triggered_buffer_cleanup(indio_dev);
+
+ regulator_disable(st->reg);
+
+ return 0;
+}
+
+#ifdef CONFIG_OF
+static const struct of_device_id adc108s102_of_match[] = {
+ { .compatible = "ti,adc108s102" },
+ { }
+};
+MODULE_DEVICE_TABLE(of, adc108s102_of_match);
+#endif
+
+#ifdef CONFIG_ACPI
+static const struct acpi_device_id adc108s102_acpi_ids[] = {
+ { "INT3495", 0 },
+ { }
+};
+MODULE_DEVICE_TABLE(acpi, adc108s102_acpi_ids);
+#endif
+
+static const struct spi_device_id adc108s102_id[] = {
+ { "adc108s102", 0 },
+ { }
+};
+MODULE_DEVICE_TABLE(spi, adc108s102_id);
+
+static struct spi_driver adc108s102_driver = {
+ .driver = {
+ .name = "adc108s102",
+ .of_match_table = of_match_ptr(adc108s102_of_match),
+ .acpi_match_table = ACPI_PTR(adc108s102_acpi_ids),
+ },
+ .probe = adc108s102_probe,
+ .remove = adc108s102_remove,
+ .id_table = adc108s102_id,
+};
+module_spi_driver(adc108s102_driver);
+
+MODULE_AUTHOR("Bogdan Pricop <bogdan.pricop@emutex.com>");
+MODULE_DESCRIPTION("Texas Instruments ADC108S102 and ADC128S102 driver");
+MODULE_LICENSE("GPL v2");
#include <linux/mutex.h>
#include <linux/delay.h>
-#include <linux/i2c/ads1015.h>
+#include <linux/platform_data/ads1015.h>
#include <linux/iio/iio.h>
#include <linux/iio/types.h>
#include <linux/platform_device.h>
#include <linux/slab.h>
#include <linux/i2c/twl.h>
-#include <linux/i2c/twl4030-madc.h>
#include <linux/module.h>
#include <linux/stddef.h>
#include <linux/mutex.h>
#include <linux/iio/iio.h>
+#define TWL4030_MADC_MAX_CHANNELS 16
+
+#define TWL4030_MADC_CTRL1 0x00
+#define TWL4030_MADC_CTRL2 0x01
+
+#define TWL4030_MADC_RTSELECT_LSB 0x02
+#define TWL4030_MADC_SW1SELECT_LSB 0x06
+#define TWL4030_MADC_SW2SELECT_LSB 0x0A
+
+#define TWL4030_MADC_RTAVERAGE_LSB 0x04
+#define TWL4030_MADC_SW1AVERAGE_LSB 0x08
+#define TWL4030_MADC_SW2AVERAGE_LSB 0x0C
+
+#define TWL4030_MADC_CTRL_SW1 0x12
+#define TWL4030_MADC_CTRL_SW2 0x13
+
+#define TWL4030_MADC_RTCH0_LSB 0x17
+#define TWL4030_MADC_GPCH0_LSB 0x37
+
+#define TWL4030_MADC_MADCON (1 << 0) /* MADC power on */
+#define TWL4030_MADC_BUSY (1 << 0) /* MADC busy */
+/* MADC conversion completion */
+#define TWL4030_MADC_EOC_SW (1 << 1)
+/* MADC SWx start conversion */
+#define TWL4030_MADC_SW_START (1 << 5)
+#define TWL4030_MADC_ADCIN0 (1 << 0)
+#define TWL4030_MADC_ADCIN1 (1 << 1)
+#define TWL4030_MADC_ADCIN2 (1 << 2)
+#define TWL4030_MADC_ADCIN3 (1 << 3)
+#define TWL4030_MADC_ADCIN4 (1 << 4)
+#define TWL4030_MADC_ADCIN5 (1 << 5)
+#define TWL4030_MADC_ADCIN6 (1 << 6)
+#define TWL4030_MADC_ADCIN7 (1 << 7)
+#define TWL4030_MADC_ADCIN8 (1 << 8)
+#define TWL4030_MADC_ADCIN9 (1 << 9)
+#define TWL4030_MADC_ADCIN10 (1 << 10)
+#define TWL4030_MADC_ADCIN11 (1 << 11)
+#define TWL4030_MADC_ADCIN12 (1 << 12)
+#define TWL4030_MADC_ADCIN13 (1 << 13)
+#define TWL4030_MADC_ADCIN14 (1 << 14)
+#define TWL4030_MADC_ADCIN15 (1 << 15)
+
+/* Fixed channels */
+#define TWL4030_MADC_BTEMP TWL4030_MADC_ADCIN1
+#define TWL4030_MADC_VBUS TWL4030_MADC_ADCIN8
+#define TWL4030_MADC_VBKB TWL4030_MADC_ADCIN9
+#define TWL4030_MADC_ICHG TWL4030_MADC_ADCIN10
+#define TWL4030_MADC_VCHG TWL4030_MADC_ADCIN11
+#define TWL4030_MADC_VBAT TWL4030_MADC_ADCIN12
+
+/* Step size and prescaler ratio */
+#define TEMP_STEP_SIZE 147
+#define TEMP_PSR_R 100
+#define CURR_STEP_SIZE 147
+#define CURR_PSR_R1 44
+#define CURR_PSR_R2 88
+
+#define TWL4030_BCI_BCICTL1 0x23
+#define TWL4030_BCI_CGAIN 0x020
+#define TWL4030_BCI_MESBAT (1 << 1)
+#define TWL4030_BCI_TYPEN (1 << 4)
+#define TWL4030_BCI_ITHEN (1 << 3)
+
+#define REG_BCICTL2 0x024
+#define TWL4030_BCI_ITHSENS 0x007
+
+/* Register and bits for GPBR1 register */
+#define TWL4030_REG_GPBR1 0x0c
+#define TWL4030_GPBR1_MADC_HFCLK_EN (1 << 7)
+
#define TWL4030_USB_SEL_MADC_MCPC (1<<3)
#define TWL4030_USB_CARKIT_ANA_CTRL 0xBB
+struct twl4030_madc_conversion_method {
+ u8 sel;
+ u8 avg;
+ u8 rbase;
+ u8 ctrl;
+};
+
+/**
+ * struct twl4030_madc_request - madc request packet for channel conversion
+ * @channels: 16 bit bitmap for individual channels
+ * @do_avg: sample the input channel for 4 consecutive cycles
+ * @method: RT, SW1, SW2
+ * @type: Polling or interrupt based method
+ * @active: Flag if request is active
+ * @result_pending: Flag from irq handler, that result is ready
+ * @raw: Return raw value, do not convert it
+ * @rbuf: Result buffer
+ */
+struct twl4030_madc_request {
+ unsigned long channels;
+ bool do_avg;
+ u16 method;
+ u16 type;
+ bool active;
+ bool result_pending;
+ bool raw;
+ int rbuf[TWL4030_MADC_MAX_CHANNELS];
+};
+
+enum conversion_methods {
+ TWL4030_MADC_RT,
+ TWL4030_MADC_SW1,
+ TWL4030_MADC_SW2,
+ TWL4030_MADC_NUM_METHODS
+};
+
+enum sample_type {
+ TWL4030_MADC_WAIT,
+ TWL4030_MADC_IRQ_ONESHOT,
+ TWL4030_MADC_IRQ_REARM
+};
+
/**
* struct twl4030_madc_data - a container for madc info
* @dev: Pointer to device structure for madc
u8 isr;
};
+static int twl4030_madc_conversion(struct twl4030_madc_request *req);
+
static int twl4030_madc_read(struct iio_dev *iio_dev,
const struct iio_chan_spec *chan,
int *val, int *val2, long mask)
req.channels = BIT(chan->channel);
req.active = false;
- req.func_cb = NULL;
req.type = TWL4030_MADC_WAIT;
req.raw = !(mask == IIO_CHAN_INFO_PROCESSED);
req.do_avg = (mask == IIO_CHAN_INFO_AVERAGE_RAW);
return count;
}
-/*
- * Enables irq.
- * @madc - pointer to twl4030_madc_data struct
- * @id - irq number to be enabled
- * can take one of TWL4030_MADC_RT, TWL4030_MADC_SW1, TWL4030_MADC_SW2
- * corresponding to RT, SW1, SW2 conversion requests.
- * If the i2c read fails it returns an error else returns 0.
- */
-static int twl4030_madc_enable_irq(struct twl4030_madc_data *madc, u8 id)
-{
- u8 val;
- int ret;
-
- ret = twl_i2c_read_u8(TWL4030_MODULE_MADC, &val, madc->imr);
- if (ret) {
- dev_err(madc->dev, "unable to read imr register 0x%X\n",
- madc->imr);
- return ret;
- }
-
- val &= ~(1 << id);
- ret = twl_i2c_write_u8(TWL4030_MODULE_MADC, val, madc->imr);
- if (ret) {
- dev_err(madc->dev,
- "unable to write imr register 0x%X\n", madc->imr);
- return ret;
- }
-
- return 0;
-}
-
/*
* Disables irq.
* @madc - pointer to twl4030_madc_data struct
/* Read results */
len = twl4030_madc_read_channels(madc, method->rbase,
r->channels, r->rbuf, r->raw);
- /* Return results to caller */
- if (r->func_cb != NULL) {
- r->func_cb(len, r->channels, r->rbuf);
- r->func_cb = NULL;
- }
/* Free request */
r->result_pending = 0;
r->active = 0;
/* Read results */
len = twl4030_madc_read_channels(madc, method->rbase,
r->channels, r->rbuf, r->raw);
- /* Return results to caller */
- if (r->func_cb != NULL) {
- r->func_cb(len, r->channels, r->rbuf);
- r->func_cb = NULL;
- }
/* Free request */
r->result_pending = 0;
r->active = 0;
return IRQ_HANDLED;
}
-static int twl4030_madc_set_irq(struct twl4030_madc_data *madc,
- struct twl4030_madc_request *req)
-{
- struct twl4030_madc_request *p;
- int ret;
-
- p = &madc->requests[req->method];
- memcpy(p, req, sizeof(*req));
- ret = twl4030_madc_enable_irq(madc, req->method);
- if (ret < 0) {
- dev_err(madc->dev, "enable irq failed!!\n");
- return ret;
- }
-
- return 0;
-}
-
/*
* Function which enables the madc conversion
* by writing to the control register.
* be a negative error value in the corresponding array element.
* returns 0 if succeeds else error value
*/
-int twl4030_madc_conversion(struct twl4030_madc_request *req)
+static int twl4030_madc_conversion(struct twl4030_madc_request *req)
{
const struct twl4030_madc_conversion_method *method;
int ret;
goto out;
}
}
- if (req->type == TWL4030_MADC_IRQ_ONESHOT && req->func_cb != NULL) {
- ret = twl4030_madc_set_irq(twl4030_madc, req);
- if (ret < 0)
- goto out;
- ret = twl4030_madc_start_conversion(twl4030_madc, req->method);
- if (ret < 0)
- goto out;
- twl4030_madc->requests[req->method].active = 1;
- ret = 0;
- goto out;
- }
/* With RT method we should not be here anymore */
if (req->method == TWL4030_MADC_RT) {
ret = -EINVAL;
return ret;
}
-EXPORT_SYMBOL_GPL(twl4030_madc_conversion);
-
-int twl4030_get_madc_conversion(int channel_no)
-{
- struct twl4030_madc_request req;
- int temp = 0;
- int ret;
-
- req.channels = (1 << channel_no);
- req.method = TWL4030_MADC_SW2;
- req.active = 0;
- req.raw = 0;
- req.func_cb = NULL;
- ret = twl4030_madc_conversion(&req);
- if (ret < 0)
- return ret;
- if (req.rbuf[channel_no] > 0)
- temp = req.rbuf[channel_no];
-
- return temp;
-}
-EXPORT_SYMBOL_GPL(twl4030_get_madc_conversion);
/**
* twl4030_madc_set_current_generator() - setup bias current
ret = PTR_ERR(xadc->clk);
goto err_free_samplerate_trigger;
}
- clk_prepare_enable(xadc->clk);
+
+ ret = clk_prepare_enable(xadc->clk);
+ if (ret)
+ goto err_free_samplerate_trigger;
ret = xadc->ops->setup(pdev, indio_dev, irq);
if (ret)
config HID_SENSOR_IIO_TRIGGER
tristate "Common module (trigger) for all HID Sensor IIO drivers"
- depends on HID_SENSOR_HUB && HID_SENSOR_IIO_COMMON
+ depends on HID_SENSOR_HUB && HID_SENSOR_IIO_COMMON && IIO_BUFFER
select IIO_TRIGGER
help
Say yes here to build trigger support for HID sensors.
{HID_USAGE_SENSOR_TIME_TIMESTAMP, HID_USAGE_SENSOR_UNITS_MILLISECOND,
1000000, 0},
+ {HID_USAGE_SENSOR_DEVICE_ORIENTATION, 0, 1, 0},
+
+ {HID_USAGE_SENSOR_RELATIVE_ORIENTATION, 0, 1, 0},
+
+ {HID_USAGE_SENSOR_GEOMAGNETIC_ORIENTATION, 0, 1, 0},
+
{HID_USAGE_SENSOR_TEMPERATURE, 0, 1000, 0},
{HID_USAGE_SENSOR_TEMPERATURE, HID_USAGE_SENSOR_UNITS_DEGREES, 1000, 0},
ret = sensor_hub_set_feature(st->hsdev, st->poll.report_id,
st->poll.index, sizeof(value), &value);
if (ret < 0 || value < 0)
- ret = -EINVAL;
+ return -EINVAL;
ret = sensor_hub_get_feature(st->hsdev,
st->poll.report_id,
st->sensitivity.index, sizeof(value),
&value);
if (ret < 0 || value < 0)
- ret = -EINVAL;
+ return -EINVAL;
ret = sensor_hub_get_feature(st->hsdev,
st->sensitivity.report_id,
}
+static void hid_sensor_get_report_latency_info(struct hid_sensor_hub_device *hsdev,
+ u32 usage_id,
+ struct hid_sensor_common *st)
+{
+ sensor_hub_input_get_attribute_info(hsdev, HID_FEATURE_REPORT,
+ usage_id,
+ HID_USAGE_SENSOR_PROP_REPORT_LATENCY,
+ &st->report_latency);
+
+ hid_dbg(hsdev->hdev, "Report latency attributes: %x:%x\n",
+ st->report_latency.index, st->report_latency.report_id);
+}
+
+int hid_sensor_get_report_latency(struct hid_sensor_common *st)
+{
+ int ret;
+ int value;
+
+ ret = sensor_hub_get_feature(st->hsdev, st->report_latency.report_id,
+ st->report_latency.index, sizeof(value),
+ &value);
+ if (ret < 0)
+ return ret;
+
+ return value;
+}
+EXPORT_SYMBOL(hid_sensor_get_report_latency);
+
+int hid_sensor_set_report_latency(struct hid_sensor_common *st, int latency_ms)
+{
+ return sensor_hub_set_feature(st->hsdev, st->report_latency.report_id,
+ st->report_latency.index,
+ sizeof(latency_ms), &latency_ms);
+}
+EXPORT_SYMBOL(hid_sensor_set_report_latency);
+
+bool hid_sensor_batch_mode_supported(struct hid_sensor_common *st)
+{
+ return st->report_latency.index > 0 && st->report_latency.report_id > 0;
+}
+EXPORT_SYMBOL(hid_sensor_batch_mode_supported);
+
int hid_sensor_parse_common_attributes(struct hid_sensor_hub_device *hsdev,
u32 usage_id,
struct hid_sensor_common *st)
} else
st->timestamp_ns_scale = 1000000000;
+ hid_sensor_get_report_latency_info(hsdev, usage_id, st);
+
hid_dbg(hsdev->hdev, "common attributes: %x:%x, %x:%x, %x:%x %x:%x %x:%x\n",
st->poll.index, st->poll.report_id,
st->report_state.index, st->report_state.report_id,
#include <linux/hid-sensor-hub.h>
#include <linux/iio/iio.h>
#include <linux/iio/trigger.h>
+#include <linux/iio/buffer.h>
#include <linux/iio/sysfs.h>
#include "hid-sensor-trigger.h"
+static ssize_t _hid_sensor_set_report_latency(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t len)
+{
+ struct iio_dev *indio_dev = dev_to_iio_dev(dev);
+ struct hid_sensor_common *attrb = iio_device_get_drvdata(indio_dev);
+ int integer, fract, ret;
+ int latency;
+
+ ret = iio_str_to_fixpoint(buf, 100000, &integer, &fract);
+ if (ret)
+ return ret;
+
+ latency = integer * 1000 + fract / 1000;
+ ret = hid_sensor_set_report_latency(attrb, latency);
+ if (ret < 0)
+ return len;
+
+ attrb->latency_ms = hid_sensor_get_report_latency(attrb);
+
+ return len;
+}
+
+static ssize_t _hid_sensor_get_report_latency(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct iio_dev *indio_dev = dev_to_iio_dev(dev);
+ struct hid_sensor_common *attrb = iio_device_get_drvdata(indio_dev);
+ int latency;
+
+ latency = hid_sensor_get_report_latency(attrb);
+ if (latency < 0)
+ return latency;
+
+ return sprintf(buf, "%d.%06u\n", latency / 1000, (latency % 1000) * 1000);
+}
+
+static ssize_t _hid_sensor_get_fifo_state(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct iio_dev *indio_dev = dev_to_iio_dev(dev);
+ struct hid_sensor_common *attrb = iio_device_get_drvdata(indio_dev);
+ int latency;
+
+ latency = hid_sensor_get_report_latency(attrb);
+ if (latency < 0)
+ return latency;
+
+ return sprintf(buf, "%d\n", !!latency);
+}
+
+static IIO_DEVICE_ATTR(hwfifo_timeout, 0644,
+ _hid_sensor_get_report_latency,
+ _hid_sensor_set_report_latency, 0);
+static IIO_DEVICE_ATTR(hwfifo_enabled, 0444,
+ _hid_sensor_get_fifo_state, NULL, 0);
+
+static const struct attribute *hid_sensor_fifo_attributes[] = {
+ &iio_dev_attr_hwfifo_timeout.dev_attr.attr,
+ &iio_dev_attr_hwfifo_enabled.dev_attr.attr,
+ NULL,
+};
+
+static void hid_sensor_setup_batch_mode(struct iio_dev *indio_dev,
+ struct hid_sensor_common *st)
+{
+ if (!hid_sensor_batch_mode_supported(st))
+ return;
+
+ iio_buffer_set_attrs(indio_dev->buffer, hid_sensor_fifo_attributes);
+}
+
static int _hid_sensor_power_state(struct hid_sensor_common *st, bool state)
{
int state_val;
sizeof(attrb->raw_hystersis),
&attrb->raw_hystersis);
+ if (attrb->latency_ms > 0)
+ hid_sensor_set_report_latency(attrb, attrb->latency_ms);
+
_hid_sensor_power_state(attrb, true);
}
attrb->trigger = trig;
indio_dev->trig = iio_trigger_get(trig);
+ hid_sensor_setup_batch_mode(indio_dev, attrb);
+
ret = pm_runtime_set_active(&indio_dev->dev);
if (ret)
goto error_unreg_trigger;
AD5045, AD5064, AD5064-1, AD5065, AD5625, AD5625R, AD5627, AD5627R,
AD5628, AD5629R, AD5645R, AD5647R, AD5648, AD5665, AD5665R, AD5666,
AD5667, AD5667R, AD5668, AD5669R, LTC2606, LTC2607, LTC2609, LTC2616,
- LTC2617, LTC2619, LTC2626, LTC2627, LTC2629 Digital to Analog Converter.
+ LTC2617, LTC2619, LTC2626, LTC2627, LTC2629, LTC2631, LTC2633, LTC2635
+ Digital to Analog Converter.
To compile this driver as a module, choose M here: the
module will be called ad5064.
* AD5024, AD5025, AD5044, AD5045, AD5064, AD5064-1, AD5065, AD5625, AD5625R,
* AD5627, AD5627R, AD5628, AD5629R, AD5645R, AD5647R, AD5648, AD5665, AD5665R,
* AD5666, AD5667, AD5667R, AD5668, AD5669R, LTC2606, LTC2607, LTC2609, LTC2616,
- * LTC2617, LTC2619, LTC2626, LTC2627, LTC2629 Digital to analog converters
- * driver
+ * LTC2617, LTC2619, LTC2626, LTC2627, LTC2629, LTC2631, LTC2633, LTC2635
+ * Digital to analog converters driver
*
* Copyright 2011 Analog Devices Inc.
*
ID_LTC2626,
ID_LTC2627,
ID_LTC2629,
+ ID_LTC2631_L12,
+ ID_LTC2631_H12,
+ ID_LTC2631_L10,
+ ID_LTC2631_H10,
+ ID_LTC2631_L8,
+ ID_LTC2631_H8,
+ ID_LTC2633_L12,
+ ID_LTC2633_H12,
+ ID_LTC2633_L10,
+ ID_LTC2633_H10,
+ ID_LTC2633_L8,
+ ID_LTC2633_H8,
+ ID_LTC2635_L12,
+ ID_LTC2635_H12,
+ ID_LTC2635_L10,
+ ID_LTC2635_H10,
+ ID_LTC2635_L8,
+ ID_LTC2635_H8,
};
static int ad5064_write(struct ad5064_state *st, unsigned int cmd,
static DECLARE_AD5064_CHANNELS(ltc2607_channels, 16, 0, ltc2617_ext_info);
static DECLARE_AD5064_CHANNELS(ltc2617_channels, 14, 2, ltc2617_ext_info);
static DECLARE_AD5064_CHANNELS(ltc2627_channels, 12, 4, ltc2617_ext_info);
+#define ltc2631_12_channels ltc2627_channels
+static DECLARE_AD5064_CHANNELS(ltc2631_10_channels, 10, 6, ltc2617_ext_info);
+static DECLARE_AD5064_CHANNELS(ltc2631_8_channels, 8, 8, ltc2617_ext_info);
+
+#define LTC2631_INFO(vref, pchannels, nchannels) \
+ { \
+ .shared_vref = true, \
+ .internal_vref = vref, \
+ .channels = pchannels, \
+ .num_channels = nchannels, \
+ .regmap_type = AD5064_REGMAP_LTC, \
+ }
+
static const struct ad5064_chip_info ad5064_chip_info_tbl[] = {
[ID_AD5024] = {
.num_channels = 4,
.regmap_type = AD5064_REGMAP_LTC,
},
+ [ID_LTC2631_L12] = LTC2631_INFO(2500000, ltc2631_12_channels, 1),
+ [ID_LTC2631_H12] = LTC2631_INFO(4096000, ltc2631_12_channels, 1),
+ [ID_LTC2631_L10] = LTC2631_INFO(2500000, ltc2631_10_channels, 1),
+ [ID_LTC2631_H10] = LTC2631_INFO(4096000, ltc2631_10_channels, 1),
+ [ID_LTC2631_L8] = LTC2631_INFO(2500000, ltc2631_8_channels, 1),
+ [ID_LTC2631_H8] = LTC2631_INFO(4096000, ltc2631_8_channels, 1),
+ [ID_LTC2633_L12] = LTC2631_INFO(2500000, ltc2631_12_channels, 2),
+ [ID_LTC2633_H12] = LTC2631_INFO(4096000, ltc2631_12_channels, 2),
+ [ID_LTC2633_L10] = LTC2631_INFO(2500000, ltc2631_10_channels, 2),
+ [ID_LTC2633_H10] = LTC2631_INFO(4096000, ltc2631_10_channels, 2),
+ [ID_LTC2633_L8] = LTC2631_INFO(2500000, ltc2631_8_channels, 2),
+ [ID_LTC2633_H8] = LTC2631_INFO(4096000, ltc2631_8_channels, 2),
+ [ID_LTC2635_L12] = LTC2631_INFO(2500000, ltc2631_12_channels, 4),
+ [ID_LTC2635_H12] = LTC2631_INFO(4096000, ltc2631_12_channels, 4),
+ [ID_LTC2635_L10] = LTC2631_INFO(2500000, ltc2631_10_channels, 4),
+ [ID_LTC2635_H10] = LTC2631_INFO(4096000, ltc2631_10_channels, 4),
+ [ID_LTC2635_L8] = LTC2631_INFO(2500000, ltc2631_8_channels, 4),
+ [ID_LTC2635_H8] = LTC2631_INFO(4096000, ltc2631_8_channels, 4),
};
static inline unsigned int ad5064_num_vref(struct ad5064_state *st)
{"ltc2626", ID_LTC2626},
{"ltc2627", ID_LTC2627},
{"ltc2629", ID_LTC2629},
+ {"ltc2631-l12", ID_LTC2631_L12},
+ {"ltc2631-h12", ID_LTC2631_H12},
+ {"ltc2631-l10", ID_LTC2631_L10},
+ {"ltc2631-h10", ID_LTC2631_H10},
+ {"ltc2631-l8", ID_LTC2631_L8},
+ {"ltc2631-h8", ID_LTC2631_H8},
+ {"ltc2633-l12", ID_LTC2633_L12},
+ {"ltc2633-h12", ID_LTC2633_H12},
+ {"ltc2633-l10", ID_LTC2633_L10},
+ {"ltc2633-h10", ID_LTC2633_H10},
+ {"ltc2633-l8", ID_LTC2633_L8},
+ {"ltc2633-h8", ID_LTC2633_H8},
+ {"ltc2635-l12", ID_LTC2635_L12},
+ {"ltc2635-h12", ID_LTC2635_H12},
+ {"ltc2635-l10", ID_LTC2635_L10},
+ {"ltc2635-h10", ID_LTC2635_H10},
+ {"ltc2635-l8", ID_LTC2635_L8},
+ {"ltc2635-h8", ID_LTC2635_H8},
{}
};
MODULE_DEVICE_TABLE(i2c, ad5064_i2c_ids);
struct hts221_sensor sensors[HTS221_SENSOR_MAX];
+ bool enabled;
u8 odr;
const struct hts221_transfer_function *tf;
struct hts221_transfer_buffer tb;
};
+extern const struct dev_pm_ops hts221_pm_ops;
+
int hts221_config_drdy(struct hts221_hw *hw, bool enable);
int hts221_probe(struct iio_dev *iio_dev);
int hts221_power_on(struct hts221_hw *hw);
#include <linux/device.h>
#include <linux/iio/sysfs.h>
#include <linux/delay.h>
+#include <linux/pm.h>
#include <asm/unaligned.h>
#include "hts221.h"
int hts221_power_on(struct hts221_hw *hw)
{
- return hts221_update_odr(hw, hw->odr);
+ int err;
+
+ err = hts221_update_odr(hw, hw->odr);
+ if (err < 0)
+ return err;
+
+ hw->enabled = true;
+
+ return 0;
}
int hts221_power_off(struct hts221_hw *hw)
{
- u8 data[] = {0x00, 0x00};
+ __le16 data = 0;
+ int err;
- return hw->tf->write(hw->dev, HTS221_REG_CNTRL1_ADDR, sizeof(data),
- data);
+ err = hw->tf->write(hw->dev, HTS221_REG_CNTRL1_ADDR, sizeof(data),
+ (u8 *)&data);
+ if (err < 0)
+ return err;
+
+ hw->enabled = false;
+
+ return 0;
}
static int hts221_parse_temp_caldata(struct hts221_hw *hw)
}
EXPORT_SYMBOL(hts221_probe);
+static int __maybe_unused hts221_suspend(struct device *dev)
+{
+ struct iio_dev *iio_dev = dev_get_drvdata(dev);
+ struct hts221_hw *hw = iio_priv(iio_dev);
+ __le16 data = 0;
+ int err;
+
+ err = hw->tf->write(hw->dev, HTS221_REG_CNTRL1_ADDR, sizeof(data),
+ (u8 *)&data);
+
+ return err < 0 ? err : 0;
+}
+
+static int __maybe_unused hts221_resume(struct device *dev)
+{
+ struct iio_dev *iio_dev = dev_get_drvdata(dev);
+ struct hts221_hw *hw = iio_priv(iio_dev);
+ int err = 0;
+
+ if (hw->enabled)
+ err = hts221_update_odr(hw, hw->odr);
+
+ return err;
+}
+
+const struct dev_pm_ops hts221_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(hts221_suspend, hts221_resume)
+};
+EXPORT_SYMBOL(hts221_pm_ops);
+
MODULE_AUTHOR("Lorenzo Bianconi <lorenzo.bianconi@st.com>");
MODULE_DESCRIPTION("STMicroelectronics hts221 sensor driver");
MODULE_LICENSE("GPL v2");
static struct i2c_driver hts221_driver = {
.driver = {
.name = "hts221_i2c",
+ .pm = &hts221_pm_ops,
.of_match_table = of_match_ptr(hts221_i2c_of_match),
.acpi_match_table = ACPI_PTR(hts221_acpi_match),
},
static struct spi_driver hts221_driver = {
.driver = {
.name = "hts221_spi",
+ .pm = &hts221_pm_ops,
.of_match_table = of_match_ptr(hts221_spi_of_match),
},
.probe = hts221_spi_probe,
int result = 0;
if (power_on) {
- /* Already under indio-dev->mlock mutex */
if (!st->powerup_count)
result = regmap_write(st->map, st->reg->pwr_mgmt_1, 0);
if (!result)
int result;
ret = IIO_VAL_INT;
- result = 0;
- mutex_lock(&indio_dev->mlock);
- if (!st->chip_config.enable) {
- result = inv_mpu6050_set_power_itg(st, true);
- if (result)
- goto error_read_raw;
- }
- /* when enable is on, power is already on */
+ mutex_lock(&st->lock);
+ result = iio_device_claim_direct_mode(indio_dev);
+ if (result)
+ goto error_read_raw_unlock;
+ result = inv_mpu6050_set_power_itg(st, true);
+ if (result)
+ goto error_read_raw_release;
switch (chan->type) {
case IIO_ANGL_VEL:
- if (!st->chip_config.gyro_fifo_enable ||
- !st->chip_config.enable) {
- result = inv_mpu6050_switch_engine(st, true,
- INV_MPU6050_BIT_PWR_GYRO_STBY);
- if (result)
- goto error_read_raw;
- }
+ result = inv_mpu6050_switch_engine(st, true,
+ INV_MPU6050_BIT_PWR_GYRO_STBY);
+ if (result)
+ goto error_read_raw_power_off;
ret = inv_mpu6050_sensor_show(st, st->reg->raw_gyro,
chan->channel2, val);
- if (!st->chip_config.gyro_fifo_enable ||
- !st->chip_config.enable) {
- result = inv_mpu6050_switch_engine(st, false,
- INV_MPU6050_BIT_PWR_GYRO_STBY);
- if (result)
- goto error_read_raw;
- }
+ result = inv_mpu6050_switch_engine(st, false,
+ INV_MPU6050_BIT_PWR_GYRO_STBY);
+ if (result)
+ goto error_read_raw_power_off;
break;
case IIO_ACCEL:
- if (!st->chip_config.accl_fifo_enable ||
- !st->chip_config.enable) {
- result = inv_mpu6050_switch_engine(st, true,
- INV_MPU6050_BIT_PWR_ACCL_STBY);
- if (result)
- goto error_read_raw;
- }
+ result = inv_mpu6050_switch_engine(st, true,
+ INV_MPU6050_BIT_PWR_ACCL_STBY);
+ if (result)
+ goto error_read_raw_power_off;
ret = inv_mpu6050_sensor_show(st, st->reg->raw_accl,
chan->channel2, val);
- if (!st->chip_config.accl_fifo_enable ||
- !st->chip_config.enable) {
- result = inv_mpu6050_switch_engine(st, false,
- INV_MPU6050_BIT_PWR_ACCL_STBY);
- if (result)
- goto error_read_raw;
- }
+ result = inv_mpu6050_switch_engine(st, false,
+ INV_MPU6050_BIT_PWR_ACCL_STBY);
+ if (result)
+ goto error_read_raw_power_off;
break;
case IIO_TEMP:
/* wait for stablization */
ret = -EINVAL;
break;
}
-error_read_raw:
- if (!st->chip_config.enable)
- result |= inv_mpu6050_set_power_itg(st, false);
- mutex_unlock(&indio_dev->mlock);
+error_read_raw_power_off:
+ result |= inv_mpu6050_set_power_itg(st, false);
+error_read_raw_release:
+ iio_device_release_direct_mode(indio_dev);
+error_read_raw_unlock:
+ mutex_unlock(&st->lock);
if (result)
return result;
case IIO_CHAN_INFO_SCALE:
switch (chan->type) {
case IIO_ANGL_VEL:
+ mutex_lock(&st->lock);
*val = 0;
*val2 = gyro_scale_6050[st->chip_config.fsr];
+ mutex_unlock(&st->lock);
return IIO_VAL_INT_PLUS_NANO;
case IIO_ACCEL:
+ mutex_lock(&st->lock);
*val = 0;
*val2 = accel_scale[st->chip_config.accl_fs];
+ mutex_unlock(&st->lock);
return IIO_VAL_INT_PLUS_MICRO;
case IIO_TEMP:
case IIO_CHAN_INFO_CALIBBIAS:
switch (chan->type) {
case IIO_ANGL_VEL:
+ mutex_lock(&st->lock);
ret = inv_mpu6050_sensor_show(st, st->reg->gyro_offset,
chan->channel2, val);
+ mutex_unlock(&st->lock);
return IIO_VAL_INT;
case IIO_ACCEL:
+ mutex_lock(&st->lock);
ret = inv_mpu6050_sensor_show(st, st->reg->accl_offset,
chan->channel2, val);
+ mutex_unlock(&st->lock);
return IIO_VAL_INT;
default:
struct inv_mpu6050_state *st = iio_priv(indio_dev);
int result;
- mutex_lock(&indio_dev->mlock);
+ mutex_lock(&st->lock);
/*
* we should only update scale when the chip is disabled, i.e.
* not running
*/
- if (st->chip_config.enable) {
- result = -EBUSY;
- goto error_write_raw;
- }
+ result = iio_device_claim_direct_mode(indio_dev);
+ if (result)
+ goto error_write_raw_unlock;
result = inv_mpu6050_set_power_itg(st, true);
if (result)
- goto error_write_raw;
+ goto error_write_raw_release;
switch (mask) {
case IIO_CHAN_INFO_SCALE:
break;
}
-error_write_raw:
result |= inv_mpu6050_set_power_itg(st, false);
- mutex_unlock(&indio_dev->mlock);
+error_write_raw_release:
+ iio_device_release_direct_mode(indio_dev);
+error_write_raw_unlock:
+ mutex_unlock(&st->lock);
return result;
}
if (fifo_rate < INV_MPU6050_MIN_FIFO_RATE ||
fifo_rate > INV_MPU6050_MAX_FIFO_RATE)
return -EINVAL;
- if (fifo_rate == st->chip_config.fifo_rate)
- return count;
- mutex_lock(&indio_dev->mlock);
- if (st->chip_config.enable) {
- result = -EBUSY;
- goto fifo_rate_fail;
+ mutex_lock(&st->lock);
+ if (fifo_rate == st->chip_config.fifo_rate) {
+ result = 0;
+ goto fifo_rate_fail_unlock;
}
+ result = iio_device_claim_direct_mode(indio_dev);
+ if (result)
+ goto fifo_rate_fail_unlock;
result = inv_mpu6050_set_power_itg(st, true);
if (result)
- goto fifo_rate_fail;
+ goto fifo_rate_fail_release;
d = INV_MPU6050_ONE_K_HZ / fifo_rate - 1;
result = regmap_write(st->map, st->reg->sample_rate_div, d);
if (result)
- goto fifo_rate_fail;
+ goto fifo_rate_fail_power_off;
st->chip_config.fifo_rate = fifo_rate;
result = inv_mpu6050_set_lpf(st, fifo_rate);
if (result)
- goto fifo_rate_fail;
+ goto fifo_rate_fail_power_off;
-fifo_rate_fail:
+fifo_rate_fail_power_off:
result |= inv_mpu6050_set_power_itg(st, false);
- mutex_unlock(&indio_dev->mlock);
+fifo_rate_fail_release:
+ iio_device_release_direct_mode(indio_dev);
+fifo_rate_fail_unlock:
+ mutex_unlock(&st->lock);
if (result)
return result;
char *buf)
{
struct inv_mpu6050_state *st = iio_priv(dev_to_iio_dev(dev));
+ unsigned fifo_rate;
+
+ mutex_lock(&st->lock);
+ fifo_rate = st->chip_config.fifo_rate;
+ mutex_unlock(&st->lock);
- return sprintf(buf, "%d\n", st->chip_config.fifo_rate);
+ return scnprintf(buf, PAGE_SIZE, "%u\n", fifo_rate);
}
/**
case ATTR_ACCL_MATRIX:
m = st->plat_data.orientation;
- return sprintf(buf, "%d, %d, %d; %d, %d, %d; %d, %d, %d\n",
+ return scnprintf(buf, PAGE_SIZE,
+ "%d, %d, %d; %d, %d, %d; %d, %d, %d\n",
m[0], m[1], m[2], m[3], m[4], m[5], m[6], m[7], m[8]);
default:
return -EINVAL;
{
int result;
unsigned int regval;
+ int i;
st->hw = &hw_info[st->chip_type];
st->reg = hw_info[st->chip_type].reg;
- /* reset to make sure previous state are not there */
- result = regmap_write(st->map, st->reg->pwr_mgmt_1,
- INV_MPU6050_BIT_H_RESET);
- if (result)
- return result;
- msleep(INV_MPU6050_POWER_UP_TIME);
-
/* check chip self-identification */
result = regmap_read(st->map, INV_MPU6050_REG_WHOAMI, ®val);
if (result)
return result;
if (regval != st->hw->whoami) {
- dev_warn(regmap_get_device(st->map),
- "whoami mismatch got %#02x expected %#02hhx for %s\n",
+ /* check whoami against all possible values */
+ for (i = 0; i < INV_NUM_PARTS; ++i) {
+ if (regval == hw_info[i].whoami) {
+ dev_warn(regmap_get_device(st->map),
+ "whoami mismatch got %#02x (%s)"
+ "expected %#02hhx (%s)\n",
+ regval, hw_info[i].name,
+ st->hw->whoami, st->hw->name);
+ break;
+ }
+ }
+ if (i >= INV_NUM_PARTS) {
+ dev_err(regmap_get_device(st->map),
+ "invalid whoami %#02x expected %#02hhx (%s)\n",
regval, st->hw->whoami, st->hw->name);
+ return -ENODEV;
+ }
}
+ /* reset to make sure previous state are not there */
+ result = regmap_write(st->map, st->reg->pwr_mgmt_1,
+ INV_MPU6050_BIT_H_RESET);
+ if (result)
+ return result;
+ msleep(INV_MPU6050_POWER_UP_TIME);
+
/*
* toggle power state. After reset, the sleep bit could be on
* or off depending on the OTP settings. Toggling power would
return -ENODEV;
}
st = iio_priv(indio_dev);
+ mutex_init(&st->lock);
st->chip_type = chip_type;
st->powerup_count = 0;
st->irq = irq;
static int inv_mpu_resume(struct device *dev)
{
- return inv_mpu6050_set_power_itg(iio_priv(dev_get_drvdata(dev)), true);
+ struct inv_mpu6050_state *st = iio_priv(dev_get_drvdata(dev));
+ int result;
+
+ mutex_lock(&st->lock);
+ result = inv_mpu6050_set_power_itg(st, true);
+ mutex_unlock(&st->lock);
+
+ return result;
}
static int inv_mpu_suspend(struct device *dev)
{
- return inv_mpu6050_set_power_itg(iio_priv(dev_get_drvdata(dev)), false);
+ struct inv_mpu6050_state *st = iio_priv(dev_get_drvdata(dev));
+ int result;
+
+ mutex_lock(&st->lock);
+ result = inv_mpu6050_set_power_itg(st, false);
+ mutex_unlock(&st->lock);
+
+ return result;
}
#endif /* CONFIG_PM_SLEEP */
int ret = 0;
/* Use the same mutex which was used everywhere to protect power-op */
- mutex_lock(&indio_dev->mlock);
+ mutex_lock(&st->lock);
if (!st->powerup_count) {
ret = regmap_write(st->map, st->reg->pwr_mgmt_1, 0);
if (ret)
INV_MPU6050_BIT_BYPASS_EN);
}
write_error:
- mutex_unlock(&indio_dev->mlock);
+ mutex_unlock(&st->lock);
return ret;
}
struct iio_dev *indio_dev = i2c_mux_priv(muxc);
struct inv_mpu6050_state *st = iio_priv(indio_dev);
- mutex_lock(&indio_dev->mlock);
+ mutex_lock(&st->lock);
/* It doesn't really mattter, if any of the calls fails */
regmap_write(st->map, st->reg->int_pin_cfg, INV_MPU6050_INT_PIN_CFG);
st->powerup_count--;
if (!st->powerup_count)
regmap_write(st->map, st->reg->pwr_mgmt_1,
INV_MPU6050_BIT_SLEEP);
- mutex_unlock(&indio_dev->mlock);
+ mutex_unlock(&st->lock);
return 0;
}
#include <linux/i2c-mux.h>
#include <linux/kfifo.h>
#include <linux/spinlock.h>
+#include <linux/mutex.h>
#include <linux/iio/iio.h>
#include <linux/iio/buffer.h>
#include <linux/regmap.h>
* @fsr: Full scale range.
* @lpf: Digital low pass filter frequency.
* @accl_fs: accel full scale range.
- * @enable: master enable state.
* @accl_fifo_enable: enable accel data output
* @gyro_fifo_enable: enable gyro data output
* @fifo_rate: FIFO update rate.
unsigned int fsr:2;
unsigned int lpf:3;
unsigned int accl_fs:2;
- unsigned int enable:1;
unsigned int accl_fifo_enable:1;
unsigned int gyro_fifo_enable:1;
u16 fifo_rate;
/*
* struct inv_mpu6050_state - Driver state variables.
* @TIMESTAMP_FIFO_SIZE: fifo size for timestamp.
+ * @lock: Chip access lock.
* @trig: IIO trigger.
* @chip_config: Cached attribute information.
* @reg: Map of important registers.
*/
struct inv_mpu6050_state {
#define TIMESTAMP_FIFO_SIZE 16
+ struct mutex lock;
struct iio_trigger *trig;
struct inv_mpu6050_chip_config chip_config;
const struct inv_mpu6050_reg_map *reg;
u16 fifo_count;
s64 timestamp;
- mutex_lock(&indio_dev->mlock);
+ mutex_lock(&st->lock);
if (!(st->chip_config.accl_fifo_enable |
st->chip_config.gyro_fifo_enable))
goto end_session;
}
end_session:
- mutex_unlock(&indio_dev->mlock);
+ mutex_unlock(&st->lock);
iio_trigger_notify_done(indio_dev->trig);
return IRQ_HANDLED;
flush_fifo:
/* Flush HW and SW FIFOs. */
inv_reset_fifo(indio_dev);
- mutex_unlock(&indio_dev->mlock);
+ mutex_unlock(&st->lock);
iio_trigger_notify_done(indio_dev->trig);
return IRQ_HANDLED;
if (result)
return result;
}
- st->chip_config.enable = enable;
return 0;
}
static int inv_mpu_data_rdy_trigger_set_state(struct iio_trigger *trig,
bool state)
{
- return inv_mpu6050_set_enable(iio_trigger_get_drvdata(trig), state);
+ struct iio_dev *indio_dev = iio_trigger_get_drvdata(trig);
+ struct inv_mpu6050_state *st = iio_priv(indio_dev);
+ int result;
+
+ mutex_lock(&st->lock);
+ result = inv_mpu6050_set_enable(indio_dev, state);
+ mutex_unlock(&st->lock);
+
+ return result;
}
static const struct iio_trigger_ops inv_mpu_trigger_ops = {
#endif /* CONFIG_SPI_MASTER */
};
+extern const struct dev_pm_ops st_lsm6dsx_pm_ops;
+
int st_lsm6dsx_probe(struct device *dev, int irq, int hw_id, const char *name,
const struct st_lsm6dsx_transfer_function *tf_ops);
int st_lsm6dsx_sensor_enable(struct st_lsm6dsx_sensor *sensor);
u8 val);
int st_lsm6dsx_update_watermark(struct st_lsm6dsx_sensor *sensor,
u16 watermark);
+int st_lsm6dsx_flush_fifo(struct st_lsm6dsx_hw *hw);
+int st_lsm6dsx_set_fifo_mode(struct st_lsm6dsx_hw *hw,
+ enum st_lsm6dsx_fifo_mode fifo_mode);
#endif /* ST_LSM6DSX_H */
#define ST_LSM6DSX_REG_FIFO_THH_ADDR 0x07
#define ST_LSM6DSX_FIFO_TH_MASK GENMASK(11, 0)
#define ST_LSM6DSX_REG_FIFO_DEC_GXL_ADDR 0x08
+#define ST_LSM6DSX_REG_HLACTIVE_ADDR 0x12
+#define ST_LSM6DSX_REG_HLACTIVE_MASK BIT(5)
#define ST_LSM6DSX_REG_FIFO_MODE_ADDR 0x0a
#define ST_LSM6DSX_FIFO_MODE_MASK GENMASK(2, 0)
#define ST_LSM6DSX_FIFO_ODR_MASK GENMASK(6, 3)
return 0;
}
-static int st_lsm6dsx_set_fifo_mode(struct st_lsm6dsx_hw *hw,
- enum st_lsm6dsx_fifo_mode fifo_mode)
+int st_lsm6dsx_set_fifo_mode(struct st_lsm6dsx_hw *hw,
+ enum st_lsm6dsx_fifo_mode fifo_mode)
{
u8 data;
int err;
return read_len;
}
-static int st_lsm6dsx_flush_fifo(struct st_lsm6dsx_hw *hw)
+int st_lsm6dsx_flush_fifo(struct st_lsm6dsx_hw *hw)
{
int err;
{
struct iio_buffer *buffer;
unsigned long irq_type;
+ bool irq_active_low;
int i, err;
irq_type = irqd_get_trigger_type(irq_get_irq_data(hw->irq));
switch (irq_type) {
case IRQF_TRIGGER_HIGH:
case IRQF_TRIGGER_RISING:
+ irq_active_low = false;
+ break;
+ case IRQF_TRIGGER_LOW:
+ case IRQF_TRIGGER_FALLING:
+ irq_active_low = true;
break;
default:
dev_info(hw->dev, "mode %lx unsupported\n", irq_type);
return -EINVAL;
}
+ err = st_lsm6dsx_write_with_mask(hw, ST_LSM6DSX_REG_HLACTIVE_ADDR,
+ ST_LSM6DSX_REG_HLACTIVE_MASK,
+ irq_active_low);
+ if (err < 0)
+ return err;
+
err = devm_request_threaded_irq(hw->dev, hw->irq,
st_lsm6dsx_handler_irq,
st_lsm6dsx_handler_thread,
#include <linux/delay.h>
#include <linux/iio/iio.h>
#include <linux/iio/sysfs.h>
+#include <linux/pm.h>
#include <linux/platform_data/st_sensors_pdata.h>
}
EXPORT_SYMBOL(st_lsm6dsx_probe);
+static int __maybe_unused st_lsm6dsx_suspend(struct device *dev)
+{
+ struct st_lsm6dsx_hw *hw = dev_get_drvdata(dev);
+ struct st_lsm6dsx_sensor *sensor;
+ int i, err = 0;
+
+ for (i = 0; i < ST_LSM6DSX_ID_MAX; i++) {
+ sensor = iio_priv(hw->iio_devs[i]);
+ if (!(hw->enable_mask & BIT(sensor->id)))
+ continue;
+
+ err = st_lsm6dsx_write_with_mask(hw,
+ st_lsm6dsx_odr_table[sensor->id].reg.addr,
+ st_lsm6dsx_odr_table[sensor->id].reg.mask, 0);
+ if (err < 0)
+ return err;
+ }
+
+ if (hw->fifo_mode != ST_LSM6DSX_FIFO_BYPASS)
+ err = st_lsm6dsx_flush_fifo(hw);
+
+ return err;
+}
+
+static int __maybe_unused st_lsm6dsx_resume(struct device *dev)
+{
+ struct st_lsm6dsx_hw *hw = dev_get_drvdata(dev);
+ struct st_lsm6dsx_sensor *sensor;
+ int i, err = 0;
+
+ for (i = 0; i < ST_LSM6DSX_ID_MAX; i++) {
+ sensor = iio_priv(hw->iio_devs[i]);
+ if (!(hw->enable_mask & BIT(sensor->id)))
+ continue;
+
+ err = st_lsm6dsx_set_odr(sensor, sensor->odr);
+ if (err < 0)
+ return err;
+ }
+
+ if (hw->enable_mask)
+ err = st_lsm6dsx_set_fifo_mode(hw, ST_LSM6DSX_FIFO_CONT);
+
+ return err;
+}
+
+const struct dev_pm_ops st_lsm6dsx_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(st_lsm6dsx_suspend, st_lsm6dsx_resume)
+};
+EXPORT_SYMBOL(st_lsm6dsx_pm_ops);
+
MODULE_AUTHOR("Lorenzo Bianconi <lorenzo.bianconi@st.com>");
MODULE_AUTHOR("Denis Ciocca <denis.ciocca@st.com>");
MODULE_DESCRIPTION("STMicroelectronics st_lsm6dsx driver");
static struct i2c_driver st_lsm6dsx_driver = {
.driver = {
.name = "st_lsm6dsx_i2c",
+ .pm = &st_lsm6dsx_pm_ops,
.of_match_table = of_match_ptr(st_lsm6dsx_i2c_of_match),
},
.probe = st_lsm6dsx_i2c_probe,
static struct spi_driver st_lsm6dsx_driver = {
.driver = {
.name = "st_lsm6dsx_spi",
+ .pm = &st_lsm6dsx_pm_ops,
.of_match_table = of_match_ptr(st_lsm6dsx_spi_of_match),
},
.probe = st_lsm6dsx_spi_probe,
size_t len)
{
const struct iio_enum *e = (const struct iio_enum *)priv;
- unsigned int i;
int ret;
if (!e->set)
return -EINVAL;
- for (i = 0; i < e->num_items; i++) {
- if (sysfs_streq(buf, e->items[i]))
- break;
- }
-
- if (i == e->num_items)
- return -EINVAL;
+ ret = __sysfs_match_string(e->items, e->num_items, buf);
+ if (ret < 0)
+ return ret;
- ret = e->set(indio_dev, chan, i);
+ ret = e->set(indio_dev, chan, ret);
return ret ? ret : len;
}
EXPORT_SYMBOL_GPL(iio_enum_write);
{
int i, ret, attrcount = 0;
- for_each_set_bit(i, infomask, sizeof(infomask)*8) {
+ for_each_set_bit(i, infomask, sizeof(*infomask)*8) {
if (i >= ARRAY_SIZE(iio_chan_info_postfix))
return -EINVAL;
ret = __iio_add_chan_devattr(iio_chan_info_postfix[i],
int i, ret, attrcount = 0;
char *avail_postfix;
- for_each_set_bit(i, infomask, sizeof(infomask) * 8) {
+ for_each_set_bit(i, infomask, sizeof(*infomask) * 8) {
avail_postfix = kasprintf(GFP_KERNEL,
"%s_available",
iio_chan_info_postfix[i]);
static void iio_dev_release(struct device *device)
{
struct iio_dev *indio_dev = dev_to_iio_dev(device);
- if (indio_dev->modes & (INDIO_BUFFER_TRIGGERED | INDIO_EVENT_TRIGGERED))
+ if (indio_dev->modes & INDIO_ALL_TRIGGERED_MODES)
iio_device_unregister_trigger_consumer(indio_dev);
iio_device_unregister_eventset(indio_dev);
iio_device_unregister_sysfs(indio_dev);
"Failed to register event set\n");
goto error_free_sysfs;
}
- if (indio_dev->modes & (INDIO_BUFFER_TRIGGERED | INDIO_EVENT_TRIGGERED))
+ if (indio_dev->modes & INDIO_ALL_TRIGGERED_MODES)
iio_device_register_trigger_consumer(indio_dev);
if ((indio_dev->modes & INDIO_ALL_BUFFER_MODES) &&
err_unlock:
mutex_unlock(&chan->indio_dev->info_exist_lock);
- if (ret >= 0 && type != IIO_VAL_INT) {
+ if (ret >= 0 && type != IIO_VAL_INT)
/* raw values are assumed to be IIO_VAL_INT */
ret = -EINVAL;
- goto err_unlock;
- }
return ret;
}
return ret;
}
EXPORT_SYMBOL_GPL(iio_write_channel_raw);
+
+unsigned int iio_get_channel_ext_info_count(struct iio_channel *chan)
+{
+ const struct iio_chan_spec_ext_info *ext_info;
+ unsigned int i = 0;
+
+ if (!chan->channel->ext_info)
+ return i;
+
+ for (ext_info = chan->channel->ext_info; ext_info->name; ext_info++)
+ ++i;
+
+ return i;
+}
+EXPORT_SYMBOL_GPL(iio_get_channel_ext_info_count);
+
+static const struct iio_chan_spec_ext_info *iio_lookup_ext_info(
+ const struct iio_channel *chan,
+ const char *attr)
+{
+ const struct iio_chan_spec_ext_info *ext_info;
+
+ if (!chan->channel->ext_info)
+ return NULL;
+
+ for (ext_info = chan->channel->ext_info; ext_info->name; ++ext_info) {
+ if (!strcmp(attr, ext_info->name))
+ return ext_info;
+ }
+
+ return NULL;
+}
+
+ssize_t iio_read_channel_ext_info(struct iio_channel *chan,
+ const char *attr, char *buf)
+{
+ const struct iio_chan_spec_ext_info *ext_info;
+
+ ext_info = iio_lookup_ext_info(chan, attr);
+ if (!ext_info)
+ return -EINVAL;
+
+ return ext_info->read(chan->indio_dev, ext_info->private,
+ chan->channel, buf);
+}
+EXPORT_SYMBOL_GPL(iio_read_channel_ext_info);
+
+ssize_t iio_write_channel_ext_info(struct iio_channel *chan, const char *attr,
+ const char *buf, size_t len)
+{
+ const struct iio_chan_spec_ext_info *ext_info;
+
+ ext_info = iio_lookup_ext_info(chan, attr);
+ if (!ext_info)
+ return -EINVAL;
+
+ return ext_info->write(chan->indio_dev, ext_info->private,
+ chan->channel, buf, len);
+}
+EXPORT_SYMBOL_GPL(iio_write_channel_ext_info);
in lux, proximity infrared sensing and normal infrared sensing.
Data from sensor is accessible via sysfs.
+config SENSORS_ISL29028
+ tristate "Intersil ISL29028 Concurrent Light and Proximity Sensor"
+ depends on I2C
+ select REGMAP_I2C
+ help
+ Provides driver for the Intersil's ISL29028 device.
+ This driver supports the sysfs interface to get the ALS, IR intensity,
+ Proximity value via iio. The ISL29028 provides the concurrent sensing
+ of ambient light and proximity.
+
config ISL29125
tristate "Intersil ISL29125 digital color light sensor"
depends on I2C
obj-$(CONFIG_HID_SENSOR_ALS) += hid-sensor-als.o
obj-$(CONFIG_HID_SENSOR_PROX) += hid-sensor-prox.o
obj-$(CONFIG_SENSORS_ISL29018) += isl29018.o
+obj-$(CONFIG_SENSORS_ISL29028) += isl29028.o
obj-$(CONFIG_ISL29125) += isl29125.o
obj-$(CONFIG_JSA1212) += jsa1212.o
obj-$(CONFIG_SENSORS_LM3533) += lm3533-als.o
#define ISL29018_PM_OPS NULL
#endif
+#ifdef CONFIG_ACPI
static const struct acpi_device_id isl29018_acpi_match[] = {
{"ISL29018", isl29018},
{"ISL29023", isl29023},
{},
};
MODULE_DEVICE_TABLE(acpi, isl29018_acpi_match);
+#endif
static const struct i2c_device_id isl29018_id[] = {
{"isl29018", isl29018},
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Datasheets:
+ * - http://www.intersil.com/content/dam/Intersil/documents/isl2/isl29028.pdf
+ * - http://www.intersil.com/content/dam/Intersil/documents/isl2/isl29030.pdf
*/
#include <linux/module.h>
#define ISL29028_POWER_OFF_DELAY_MS 2000
-static const unsigned int isl29028_prox_sleep_time[] = {800, 400, 200, 100, 75,
- 50, 12, 0};
+struct isl29028_prox_data {
+ int sampling_int;
+ int sampling_fract;
+ int sleep_time;
+};
+
+static const struct isl29028_prox_data isl29028_prox_data[] = {
+ { 1, 250000, 800 },
+ { 2, 500000, 400 },
+ { 5, 0, 200 },
+ { 10, 0, 100 },
+ { 13, 300000, 75 },
+ { 20, 0, 50 },
+ { 80, 0, 13 }, /*
+ * Note: Data sheet lists 12.5 ms sleep time.
+ * Round up a half millisecond for msleep().
+ */
+ { 100, 0, 0 }
+};
enum isl29028_als_ir_mode {
ISL29028_MODE_NONE = 0,
struct isl29028_chip {
struct mutex lock;
struct regmap *regmap;
- unsigned int prox_sampling;
+ int prox_sampling_int;
+ int prox_sampling_frac;
bool enable_prox;
int lux_scale;
enum isl29028_als_ir_mode als_ir_mode;
};
-static int isl29028_find_prox_sleep_time_index(int sampling)
+static int isl29028_find_prox_sleep_index(int sampling_int, int sampling_fract)
{
- unsigned int period = DIV_ROUND_UP(1000, sampling);
int i;
- for (i = 0; i < ARRAY_SIZE(isl29028_prox_sleep_time); ++i) {
- if (period >= isl29028_prox_sleep_time[i])
- break;
+ for (i = 0; i < ARRAY_SIZE(isl29028_prox_data); ++i) {
+ if (isl29028_prox_data[i].sampling_int == sampling_int &&
+ isl29028_prox_data[i].sampling_fract == sampling_fract)
+ return i;
}
- return i;
+ return -EINVAL;
}
static int isl29028_set_proxim_sampling(struct isl29028_chip *chip,
- unsigned int sampling)
+ int sampling_int, int sampling_fract)
{
struct device *dev = regmap_get_device(chip->regmap);
int sleep_index, ret;
- sleep_index = isl29028_find_prox_sleep_time_index(sampling);
+ sleep_index = isl29028_find_prox_sleep_index(sampling_int,
+ sampling_fract);
+ if (sleep_index < 0)
+ return sleep_index;
+
ret = regmap_update_bits(chip->regmap, ISL29028_REG_CONFIGURE,
ISL29028_CONF_PROX_SLP_MASK,
sleep_index << ISL29028_CONF_PROX_SLP_SH);
return ret;
}
- chip->prox_sampling = sampling;
+ chip->prox_sampling_int = sampling_int;
+ chip->prox_sampling_frac = sampling_fract;
return ret;
}
static int isl29028_enable_proximity(struct isl29028_chip *chip)
{
- int sleep_index, ret;
+ int prox_index, ret;
- ret = isl29028_set_proxim_sampling(chip, chip->prox_sampling);
+ ret = isl29028_set_proxim_sampling(chip, chip->prox_sampling_int,
+ chip->prox_sampling_frac);
if (ret < 0)
return ret;
return ret;
/* Wait for conversion to be complete for first sample */
- sleep_index = isl29028_find_prox_sleep_time_index(chip->prox_sampling);
- msleep(isl29028_prox_sleep_time[sleep_index]);
+ prox_index = isl29028_find_prox_sleep_index(chip->prox_sampling_int,
+ chip->prox_sampling_frac);
+ if (prox_index < 0)
+ return prox_index;
+
+ msleep(isl29028_prox_data[prox_index].sleep_time);
return 0;
}
break;
}
- ret = isl29028_set_proxim_sampling(chip, val);
+ ret = isl29028_set_proxim_sampling(chip, val, val2);
break;
case IIO_LIGHT:
if (mask != IIO_CHAN_INFO_SCALE) {
if (chan->type != IIO_PROXIMITY)
break;
- *val = chip->prox_sampling;
+ *val = chip->prox_sampling_int;
+ *val2 = chip->prox_sampling_frac;
ret = IIO_VAL_INT;
break;
case IIO_CHAN_INFO_SCALE:
}
static IIO_CONST_ATTR(in_proximity_sampling_frequency_available,
- "1 3 5 10 13 20 83 100");
+ "1.25 2.5 5 10 13.3 20 80 100");
static IIO_CONST_ATTR(in_illuminance_scale_available, "125 2000");
#define ISL29028_CONST_ATTR(name) (&iio_const_attr_##name.dev_attr.attr)
}
chip->enable_prox = false;
- chip->prox_sampling = 20;
+ chip->prox_sampling_int = 20;
+ chip->prox_sampling_frac = 0;
chip->lux_scale = 2000;
ret = regmap_write(chip->regmap, ISL29028_REG_TEST1_MODE, 0x0);
static const struct i2c_device_id isl29028_id[] = {
{"isl29028", 0},
+ {"isl29030", 0},
{}
};
MODULE_DEVICE_TABLE(i2c, isl29028_id);
static const struct of_device_id isl29028_of_match[] = {
{ .compatible = "isl,isl29028", }, /* for backward compat., don't use */
{ .compatible = "isil,isl29028", },
+ { .compatible = "isil,isl29030", },
{ },
};
MODULE_DEVICE_TABLE(of, isl29028_of_match);
*
* IIO driver for RPR-0521RS (7-bit I2C slave address 0x38).
*
- * TODO: illuminance channel, PM support, buffer
+ * TODO: illuminance channel, buffer
*/
#include <linux/module.h>
#define RPR0521_REG_PXS_DATA 0x44 /* 16-bit, little endian */
#define RPR0521_REG_ALS_DATA0 0x46 /* 16-bit, little endian */
#define RPR0521_REG_ALS_DATA1 0x48 /* 16-bit, little endian */
+#define RPR0521_REG_PS_OFFSET_LSB 0x53
#define RPR0521_REG_ID 0x92
#define RPR0521_MODE_ALS_MASK BIT(7)
};
enum rpr0521_channel {
+ RPR0521_CHAN_PXS,
RPR0521_CHAN_ALS_DATA0,
RPR0521_CHAN_ALS_DATA1,
- RPR0521_CHAN_PXS,
};
struct rpr0521_reg_desc {
};
static const struct rpr0521_reg_desc rpr0521_data_reg[] = {
+ [RPR0521_CHAN_PXS] = {
+ .address = RPR0521_REG_PXS_DATA,
+ .device_mask = RPR0521_MODE_PXS_MASK,
+ },
[RPR0521_CHAN_ALS_DATA0] = {
.address = RPR0521_REG_ALS_DATA0,
.device_mask = RPR0521_MODE_ALS_MASK,
.address = RPR0521_REG_ALS_DATA1,
.device_mask = RPR0521_MODE_ALS_MASK,
},
- [RPR0521_CHAN_PXS] = {
- .address = RPR0521_REG_PXS_DATA,
- .device_mask = RPR0521_MODE_PXS_MASK,
- },
};
static const struct rpr0521_gain_info {
const struct rpr0521_gain *gain;
int size;
} rpr0521_gain[] = {
+ [RPR0521_CHAN_PXS] = {
+ .reg = RPR0521_REG_PXS_CTRL,
+ .mask = RPR0521_PXS_GAIN_MASK,
+ .shift = RPR0521_PXS_GAIN_SHIFT,
+ .gain = rpr0521_pxs_gain,
+ .size = ARRAY_SIZE(rpr0521_pxs_gain),
+ },
[RPR0521_CHAN_ALS_DATA0] = {
.reg = RPR0521_REG_ALS_CTRL,
.mask = RPR0521_ALS_DATA0_GAIN_MASK,
.gain = rpr0521_als_gain,
.size = ARRAY_SIZE(rpr0521_als_gain),
},
- [RPR0521_CHAN_PXS] = {
- .reg = RPR0521_REG_PXS_CTRL,
- .mask = RPR0521_PXS_GAIN_MASK,
- .shift = RPR0521_PXS_GAIN_SHIFT,
- .gain = rpr0521_pxs_gain,
- .size = ARRAY_SIZE(rpr0521_pxs_gain),
- },
+};
+
+struct rpr0521_samp_freq {
+ int als_hz;
+ int als_uhz;
+ int pxs_hz;
+ int pxs_uhz;
+};
+
+static const struct rpr0521_samp_freq rpr0521_samp_freq_i[13] = {
+/* {ALS, PXS}, W==currently writable option */
+ {0, 0, 0, 0}, /* W0000, 0=standby */
+ {0, 0, 100, 0}, /* 0001 */
+ {0, 0, 25, 0}, /* 0010 */
+ {0, 0, 10, 0}, /* 0011 */
+ {0, 0, 2, 500000}, /* 0100 */
+ {10, 0, 20, 0}, /* 0101 */
+ {10, 0, 10, 0}, /* W0110 */
+ {10, 0, 2, 500000}, /* 0111 */
+ {2, 500000, 20, 0}, /* 1000, measurement 100ms, sleep 300ms */
+ {2, 500000, 10, 0}, /* 1001, measurement 100ms, sleep 300ms */
+ {2, 500000, 0, 0}, /* 1010, high sensitivity mode */
+ {2, 500000, 2, 500000}, /* W1011, high sensitivity mode */
+ {20, 0, 20, 0} /* 1100, ALS_data x 0.5, see specification P.18 */
};
struct rpr0521_data {
bool als_dev_en;
bool pxs_dev_en;
- /* optimize runtime pm ops - enable device only if needed */
+ /* optimize runtime pm ops - enable/disable device only if needed */
bool als_ps_need_en;
bool pxs_ps_need_en;
+ bool als_need_dis;
+ bool pxs_need_dis;
struct regmap *regmap;
};
static IIO_CONST_ATTR(in_intensity_scale_available, RPR0521_ALS_SCALE_AVAIL);
static IIO_CONST_ATTR(in_proximity_scale_available, RPR0521_PXS_SCALE_AVAIL);
+/*
+ * Start with easy freq first, whole table of freq combinations is more
+ * complicated.
+ */
+static IIO_CONST_ATTR_SAMP_FREQ_AVAIL("2.5 10");
+
static struct attribute *rpr0521_attributes[] = {
&iio_const_attr_in_intensity_scale_available.dev_attr.attr,
&iio_const_attr_in_proximity_scale_available.dev_attr.attr,
+ &iio_const_attr_sampling_frequency_available.dev_attr.attr,
NULL,
};
};
static const struct iio_chan_spec rpr0521_channels[] = {
+ {
+ .type = IIO_PROXIMITY,
+ .address = RPR0521_CHAN_PXS,
+ .info_mask_separate = BIT(IIO_CHAN_INFO_RAW) |
+ BIT(IIO_CHAN_INFO_OFFSET) |
+ BIT(IIO_CHAN_INFO_SCALE),
+ .info_mask_shared_by_all = BIT(IIO_CHAN_INFO_SAMP_FREQ),
+ },
{
.type = IIO_INTENSITY,
.modified = 1,
.channel2 = IIO_MOD_LIGHT_BOTH,
.info_mask_separate = BIT(IIO_CHAN_INFO_RAW) |
BIT(IIO_CHAN_INFO_SCALE),
+ .info_mask_shared_by_all = BIT(IIO_CHAN_INFO_SAMP_FREQ),
},
{
.type = IIO_INTENSITY,
.channel2 = IIO_MOD_LIGHT_IR,
.info_mask_separate = BIT(IIO_CHAN_INFO_RAW) |
BIT(IIO_CHAN_INFO_SCALE),
+ .info_mask_shared_by_all = BIT(IIO_CHAN_INFO_SAMP_FREQ),
},
- {
- .type = IIO_PROXIMITY,
- .address = RPR0521_CHAN_PXS,
- .info_mask_separate = BIT(IIO_CHAN_INFO_RAW) |
- BIT(IIO_CHAN_INFO_SCALE),
- }
};
static int rpr0521_als_enable(struct rpr0521_data *data, u8 status)
if (ret < 0)
return ret;
- data->als_dev_en = true;
+ if (status & RPR0521_MODE_ALS_MASK)
+ data->als_dev_en = true;
+ else
+ data->als_dev_en = false;
return 0;
}
if (ret < 0)
return ret;
- data->pxs_dev_en = true;
+ if (status & RPR0521_MODE_PXS_MASK)
+ data->pxs_dev_en = true;
+ else
+ data->pxs_dev_en = false;
return 0;
}
* @on: state to be set for devices in @device_mask
* @device_mask: bitmask specifying for which device we need to update @on state
*
- * We rely on rpr0521_runtime_resume to enable our @device_mask devices, but
- * if (for example) PXS was enabled (pxs_dev_en = true) by a previous call to
- * rpr0521_runtime_resume and we want to enable ALS we MUST set ALS enable
- * bit of RPR0521_REG_MODE_CTRL here because rpr0521_runtime_resume will not
- * be called twice.
+ * Calls for this function must be balanced so that each ON should have matching
+ * OFF. Otherwise pm usage_count gets out of sync.
*/
static int rpr0521_set_power_state(struct rpr0521_data *data, bool on,
u8 device_mask)
{
#ifdef CONFIG_PM
int ret;
- u8 update_mask = 0;
if (device_mask & RPR0521_MODE_ALS_MASK) {
- if (on && !data->als_ps_need_en && data->pxs_dev_en)
- update_mask |= RPR0521_MODE_ALS_MASK;
- else
- data->als_ps_need_en = on;
+ data->als_ps_need_en = on;
+ data->als_need_dis = !on;
}
if (device_mask & RPR0521_MODE_PXS_MASK) {
- if (on && !data->pxs_ps_need_en && data->als_dev_en)
- update_mask |= RPR0521_MODE_PXS_MASK;
- else
- data->pxs_ps_need_en = on;
- }
-
- if (update_mask) {
- ret = regmap_update_bits(data->regmap, RPR0521_REG_MODE_CTRL,
- update_mask, update_mask);
- if (ret < 0)
- return ret;
+ data->pxs_ps_need_en = on;
+ data->pxs_need_dis = !on;
}
+ /*
+ * On: _resume() is called only when we are suspended
+ * Off: _suspend() is called after delay if _resume() is not
+ * called before that.
+ * Note: If either measurement is re-enabled before _suspend(),
+ * both stay enabled until _suspend().
+ */
if (on) {
ret = pm_runtime_get_sync(&data->client->dev);
} else {
return ret;
}
+
+ if (on) {
+ /* If _resume() was not called, enable measurement now. */
+ if (data->als_ps_need_en) {
+ ret = rpr0521_als_enable(data, RPR0521_MODE_ALS_ENABLE);
+ if (ret)
+ return ret;
+ data->als_ps_need_en = false;
+ }
+
+ if (data->pxs_ps_need_en) {
+ ret = rpr0521_pxs_enable(data, RPR0521_MODE_PXS_ENABLE);
+ if (ret)
+ return ret;
+ data->pxs_ps_need_en = false;
+ }
+ }
#endif
return 0;
}
idx << rpr0521_gain[chan].shift);
}
+static int rpr0521_read_samp_freq(struct rpr0521_data *data,
+ enum iio_chan_type chan_type,
+ int *val, int *val2)
+{
+ int reg, ret;
+
+ ret = regmap_read(data->regmap, RPR0521_REG_MODE_CTRL, ®);
+ if (ret < 0)
+ return ret;
+
+ reg &= RPR0521_MODE_MEAS_TIME_MASK;
+ if (reg >= ARRAY_SIZE(rpr0521_samp_freq_i))
+ return -EINVAL;
+
+ switch (chan_type) {
+ case IIO_INTENSITY:
+ *val = rpr0521_samp_freq_i[reg].als_hz;
+ *val2 = rpr0521_samp_freq_i[reg].als_uhz;
+ return 0;
+
+ case IIO_PROXIMITY:
+ *val = rpr0521_samp_freq_i[reg].pxs_hz;
+ *val2 = rpr0521_samp_freq_i[reg].pxs_uhz;
+ return 0;
+
+ default:
+ return -EINVAL;
+ }
+}
+
+static int rpr0521_write_samp_freq_common(struct rpr0521_data *data,
+ enum iio_chan_type chan_type,
+ int val, int val2)
+{
+ int i;
+
+ /*
+ * Ignore channel
+ * both pxs and als are setup only to same freq because of simplicity
+ */
+ switch (val) {
+ case 0:
+ i = 0;
+ break;
+
+ case 2:
+ if (val2 != 500000)
+ return -EINVAL;
+
+ i = 11;
+ break;
+
+ case 10:
+ i = 6;
+ break;
+
+ default:
+ return -EINVAL;
+ }
+
+ return regmap_update_bits(data->regmap,
+ RPR0521_REG_MODE_CTRL,
+ RPR0521_MODE_MEAS_TIME_MASK,
+ i);
+}
+
+static int rpr0521_read_ps_offset(struct rpr0521_data *data, int *offset)
+{
+ int ret;
+ __le16 buffer;
+
+ ret = regmap_bulk_read(data->regmap,
+ RPR0521_REG_PS_OFFSET_LSB, &buffer, sizeof(buffer));
+
+ if (ret < 0) {
+ dev_err(&data->client->dev, "Failed to read PS OFFSET register\n");
+ return ret;
+ }
+ *offset = le16_to_cpu(buffer);
+
+ return ret;
+}
+
+static int rpr0521_write_ps_offset(struct rpr0521_data *data, int offset)
+{
+ int ret;
+ __le16 buffer;
+
+ buffer = cpu_to_le16(offset & 0x3ff);
+ ret = regmap_raw_write(data->regmap,
+ RPR0521_REG_PS_OFFSET_LSB, &buffer, sizeof(buffer));
+
+ if (ret < 0) {
+ dev_err(&data->client->dev, "Failed to write PS OFFSET register\n");
+ return ret;
+ }
+
+ return ret;
+}
+
static int rpr0521_read_raw(struct iio_dev *indio_dev,
struct iio_chan_spec const *chan, int *val,
int *val2, long mask)
ret = regmap_bulk_read(data->regmap,
rpr0521_data_reg[chan->address].address,
- &raw_data, 2);
+ &raw_data, sizeof(raw_data));
if (ret < 0) {
rpr0521_set_power_state(data, false, device_mask);
mutex_unlock(&data->lock);
*val = le16_to_cpu(raw_data);
return IIO_VAL_INT;
+
case IIO_CHAN_INFO_SCALE:
mutex_lock(&data->lock);
ret = rpr0521_get_gain(data, chan->address, val, val2);
return ret;
return IIO_VAL_INT_PLUS_MICRO;
+
+ case IIO_CHAN_INFO_SAMP_FREQ:
+ mutex_lock(&data->lock);
+ ret = rpr0521_read_samp_freq(data, chan->type, val, val2);
+ mutex_unlock(&data->lock);
+ if (ret < 0)
+ return ret;
+
+ return IIO_VAL_INT_PLUS_MICRO;
+
+ case IIO_CHAN_INFO_OFFSET:
+ mutex_lock(&data->lock);
+ ret = rpr0521_read_ps_offset(data, val);
+ mutex_unlock(&data->lock);
+ if (ret < 0)
+ return ret;
+
+ return IIO_VAL_INT;
+
default:
return -EINVAL;
}
mutex_unlock(&data->lock);
return ret;
+
+ case IIO_CHAN_INFO_SAMP_FREQ:
+ mutex_lock(&data->lock);
+ ret = rpr0521_write_samp_freq_common(data, chan->type,
+ val, val2);
+ mutex_unlock(&data->lock);
+
+ return ret;
+
+ case IIO_CHAN_INFO_OFFSET:
+ mutex_lock(&data->lock);
+ ret = rpr0521_write_ps_offset(data, val);
+ mutex_unlock(&data->lock);
+
+ return ret;
+
default:
return -EINVAL;
}
return ret;
}
+#ifndef CONFIG_PM
ret = rpr0521_als_enable(data, RPR0521_MODE_ALS_ENABLE);
if (ret < 0)
return ret;
ret = rpr0521_pxs_enable(data, RPR0521_MODE_PXS_ENABLE);
if (ret < 0)
return ret;
+#endif
return 0;
}
ret = pm_runtime_set_active(&client->dev);
if (ret < 0)
- return ret;
+ goto err_poweroff;
pm_runtime_enable(&client->dev);
pm_runtime_set_autosuspend_delay(&client->dev, RPR0521_SLEEP_DELAY_MS);
pm_runtime_use_autosuspend(&client->dev);
- return iio_device_register(indio_dev);
+ ret = iio_device_register(indio_dev);
+ if (ret)
+ goto err_pm_disable;
+
+ return 0;
+
+err_pm_disable:
+ pm_runtime_disable(&client->dev);
+ pm_runtime_set_suspended(&client->dev);
+ pm_runtime_put_noidle(&client->dev);
+err_poweroff:
+ rpr0521_poweroff(data);
+
+ return ret;
}
static int rpr0521_remove(struct i2c_client *client)
struct rpr0521_data *data = iio_priv(indio_dev);
int ret;
- /* disable channels and sets {als,pxs}_dev_en to false */
mutex_lock(&data->lock);
+ /* If measurements are enabled, enable them on resume */
+ if (!data->als_need_dis)
+ data->als_ps_need_en = data->als_dev_en;
+ if (!data->pxs_need_dis)
+ data->pxs_ps_need_en = data->pxs_dev_en;
+
+ /* disable channels and sets {als,pxs}_dev_en to false */
ret = rpr0521_poweroff(data);
+ regcache_mark_dirty(data->regmap);
mutex_unlock(&data->lock);
return ret;
struct rpr0521_data *data = iio_priv(indio_dev);
int ret;
+ regcache_sync(data->regmap);
if (data->als_ps_need_en) {
ret = rpr0521_als_enable(data, RPR0521_MODE_ALS_ENABLE);
if (ret < 0)
return ret;
data->pxs_ps_need_en = false;
}
+ msleep(100); //wait for first measurement result
return 0;
}
* within the TAOS tsl258x family of devices (tsl2580, tsl2581, tsl2583).
*
* Copyright (c) 2011, TAOS Corporation.
- * Copyright (c) 2016 Brian Masney <masneyb@onstation.org>
+ * Copyright (c) 2016-2017 Brian Masney <masneyb@onstation.org>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
#include <linux/module.h>
#include <linux/iio/iio.h>
#include <linux/iio/sysfs.h>
+#include <linux/pm_runtime.h>
/* Device Registers and Masks */
#define TSL2583_CNTRL 0x00
#define TSL2583_CHIP_ID 0x90
#define TSL2583_CHIP_ID_MASK 0xf0
+#define TSL2583_POWER_OFF_DELAY_MS 2000
+
/* Per-device data */
struct tsl2583_als_info {
u16 als_ch0;
struct tsl2583_settings als_settings;
int als_time_scale;
int als_saturation;
- bool suspended;
};
struct gainadj {
if (ret < 0)
return ret;
- chip->suspended = false;
-
return ret;
}
mutex_lock(&chip->als_mutex);
- if (chip->suspended) {
- ret = -EBUSY;
- goto done;
- }
-
ret = tsl2583_als_calibrate(indio_dev);
if (ret < 0)
goto done;
},
};
+static int tsl2583_set_pm_runtime_busy(struct tsl2583_chip *chip, bool on)
+{
+ int ret;
+
+ if (on) {
+ ret = pm_runtime_get_sync(&chip->client->dev);
+ if (ret < 0)
+ pm_runtime_put_noidle(&chip->client->dev);
+ } else {
+ pm_runtime_mark_last_busy(&chip->client->dev);
+ ret = pm_runtime_put_autosuspend(&chip->client->dev);
+ }
+
+ return ret;
+}
+
static int tsl2583_read_raw(struct iio_dev *indio_dev,
struct iio_chan_spec const *chan,
int *val, int *val2, long mask)
{
struct tsl2583_chip *chip = iio_priv(indio_dev);
- int ret = -EINVAL;
+ int ret, pm_ret;
- mutex_lock(&chip->als_mutex);
+ ret = tsl2583_set_pm_runtime_busy(chip, true);
+ if (ret < 0)
+ return ret;
- if (chip->suspended) {
- ret = -EBUSY;
- goto read_done;
- }
+ mutex_lock(&chip->als_mutex);
+ ret = -EINVAL;
switch (mask) {
case IIO_CHAN_INFO_RAW:
if (chan->type == IIO_LIGHT) {
read_done:
mutex_unlock(&chip->als_mutex);
+ if (ret < 0)
+ return ret;
+
+ /*
+ * Preserve the ret variable if the call to
+ * tsl2583_set_pm_runtime_busy() is successful so the reading
+ * (if applicable) is returned to user space.
+ */
+ pm_ret = tsl2583_set_pm_runtime_busy(chip, false);
+ if (pm_ret < 0)
+ return pm_ret;
+
return ret;
}
int val, int val2, long mask)
{
struct tsl2583_chip *chip = iio_priv(indio_dev);
- int ret = -EINVAL;
+ int ret;
- mutex_lock(&chip->als_mutex);
+ ret = tsl2583_set_pm_runtime_busy(chip, true);
+ if (ret < 0)
+ return ret;
- if (chip->suspended) {
- ret = -EBUSY;
- goto write_done;
- }
+ mutex_lock(&chip->als_mutex);
+ ret = -EINVAL;
switch (mask) {
case IIO_CHAN_INFO_CALIBBIAS:
if (chan->type == IIO_LIGHT) {
break;
}
-write_done:
mutex_unlock(&chip->als_mutex);
+ if (ret < 0)
+ return ret;
+
+ ret = tsl2583_set_pm_runtime_busy(chip, false);
+ if (ret < 0)
+ return ret;
+
return ret;
}
i2c_set_clientdata(clientp, indio_dev);
mutex_init(&chip->als_mutex);
- chip->suspended = true;
ret = i2c_smbus_read_byte_data(clientp,
TSL2583_CMD_REG | TSL2583_CHIPID);
indio_dev->modes = INDIO_DIRECT_MODE;
indio_dev->name = chip->client->name;
+ pm_runtime_enable(&clientp->dev);
+ pm_runtime_set_autosuspend_delay(&clientp->dev,
+ TSL2583_POWER_OFF_DELAY_MS);
+ pm_runtime_use_autosuspend(&clientp->dev);
+
ret = devm_iio_device_register(indio_dev->dev.parent, indio_dev);
if (ret) {
dev_err(&clientp->dev, "%s: iio registration failed\n",
/* Load up the V2 defaults (these are hard coded defaults for now) */
tsl2583_defaults(chip);
- /* Make sure the chip is on */
- ret = tsl2583_chip_init_and_power_on(indio_dev);
- if (ret < 0)
- return ret;
-
dev_info(&clientp->dev, "Light sensor found.\n");
return 0;
}
+static int tsl2583_remove(struct i2c_client *client)
+{
+ struct iio_dev *indio_dev = i2c_get_clientdata(client);
+ struct tsl2583_chip *chip = iio_priv(indio_dev);
+
+ iio_device_unregister(indio_dev);
+
+ pm_runtime_disable(&client->dev);
+ pm_runtime_set_suspended(&client->dev);
+ pm_runtime_put_noidle(&client->dev);
+
+ return tsl2583_set_power_state(chip, TSL2583_CNTL_PWR_OFF);
+}
+
static int __maybe_unused tsl2583_suspend(struct device *dev)
{
struct iio_dev *indio_dev = i2c_get_clientdata(to_i2c_client(dev));
mutex_lock(&chip->als_mutex);
ret = tsl2583_set_power_state(chip, TSL2583_CNTL_PWR_OFF);
- chip->suspended = true;
mutex_unlock(&chip->als_mutex);
return ret;
}
-static SIMPLE_DEV_PM_OPS(tsl2583_pm_ops, tsl2583_suspend, tsl2583_resume);
+static const struct dev_pm_ops tsl2583_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
+ pm_runtime_force_resume)
+ SET_RUNTIME_PM_OPS(tsl2583_suspend, tsl2583_resume, NULL)
+};
static struct i2c_device_id tsl2583_idtable[] = {
{ "tsl2580", 0 },
},
.id_table = tsl2583_idtable,
.probe = tsl2583_probe,
+ .remove = tsl2583_remove,
};
module_i2c_driver(tsl2583_driver);
}
static const struct spi_device_id st_magn_id_table[] = {
- { LSM303DLHC_MAGN_DEV_NAME },
- { LSM303DLM_MAGN_DEV_NAME },
{ LIS3MDL_MAGN_DEV_NAME },
{ LSM303AGR_MAGN_DEV_NAME },
{},
--- /dev/null
+#
+# Multiplexer drivers
+#
+# When adding new entries keep the list in alphabetical order
+
+menu "Multiplexers"
+
+config IIO_MUX
+ tristate "IIO multiplexer driver"
+ select MULTIPLEXER
+ depends on OF || COMPILE_TEST
+ help
+ Say yes here to build support for the IIO multiplexer.
+
+ To compile this driver as a module, choose M here: the
+ module will be called iio-mux.
+
+endmenu
--- /dev/null
+#
+# Makefile for industrial I/O multiplexer drivers
+#
+
+# When adding new entries keep the list in alphabetical order
+obj-$(CONFIG_IIO_MUX) += iio-mux.o
--- /dev/null
+/*
+ * IIO multiplexer driver
+ *
+ * Copyright (C) 2017 Axentia Technologies AB
+ *
+ * Author: Peter Rosin <peda@axentia.se>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/err.h>
+#include <linux/iio/consumer.h>
+#include <linux/iio/iio.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/mux/consumer.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+
+struct mux_ext_info_cache {
+ char *data;
+ ssize_t size;
+};
+
+struct mux_child {
+ struct mux_ext_info_cache *ext_info_cache;
+};
+
+struct mux {
+ int cached_state;
+ struct mux_control *control;
+ struct iio_channel *parent;
+ struct iio_dev *indio_dev;
+ struct iio_chan_spec *chan;
+ struct iio_chan_spec_ext_info *ext_info;
+ struct mux_child *child;
+};
+
+static int iio_mux_select(struct mux *mux, int idx)
+{
+ struct mux_child *child = &mux->child[idx];
+ struct iio_chan_spec const *chan = &mux->chan[idx];
+ int ret;
+ int i;
+
+ ret = mux_control_select(mux->control, chan->channel);
+ if (ret < 0) {
+ mux->cached_state = -1;
+ return ret;
+ }
+
+ if (mux->cached_state == chan->channel)
+ return 0;
+
+ if (chan->ext_info) {
+ for (i = 0; chan->ext_info[i].name; ++i) {
+ const char *attr = chan->ext_info[i].name;
+ struct mux_ext_info_cache *cache;
+
+ cache = &child->ext_info_cache[i];
+
+ if (cache->size < 0)
+ continue;
+
+ ret = iio_write_channel_ext_info(mux->parent, attr,
+ cache->data,
+ cache->size);
+
+ if (ret < 0) {
+ mux_control_deselect(mux->control);
+ mux->cached_state = -1;
+ return ret;
+ }
+ }
+ }
+ mux->cached_state = chan->channel;
+
+ return 0;
+}
+
+static void iio_mux_deselect(struct mux *mux)
+{
+ mux_control_deselect(mux->control);
+}
+
+static int mux_read_raw(struct iio_dev *indio_dev,
+ struct iio_chan_spec const *chan,
+ int *val, int *val2, long mask)
+{
+ struct mux *mux = iio_priv(indio_dev);
+ int idx = chan - mux->chan;
+ int ret;
+
+ ret = iio_mux_select(mux, idx);
+ if (ret < 0)
+ return ret;
+
+ switch (mask) {
+ case IIO_CHAN_INFO_RAW:
+ ret = iio_read_channel_raw(mux->parent, val);
+ break;
+
+ case IIO_CHAN_INFO_SCALE:
+ ret = iio_read_channel_scale(mux->parent, val, val2);
+ break;
+
+ default:
+ ret = -EINVAL;
+ }
+
+ iio_mux_deselect(mux);
+
+ return ret;
+}
+
+static int mux_read_avail(struct iio_dev *indio_dev,
+ struct iio_chan_spec const *chan,
+ const int **vals, int *type, int *length,
+ long mask)
+{
+ struct mux *mux = iio_priv(indio_dev);
+ int idx = chan - mux->chan;
+ int ret;
+
+ ret = iio_mux_select(mux, idx);
+ if (ret < 0)
+ return ret;
+
+ switch (mask) {
+ case IIO_CHAN_INFO_RAW:
+ *type = IIO_VAL_INT;
+ ret = iio_read_avail_channel_raw(mux->parent, vals, length);
+ break;
+
+ default:
+ ret = -EINVAL;
+ }
+
+ iio_mux_deselect(mux);
+
+ return ret;
+}
+
+static int mux_write_raw(struct iio_dev *indio_dev,
+ struct iio_chan_spec const *chan,
+ int val, int val2, long mask)
+{
+ struct mux *mux = iio_priv(indio_dev);
+ int idx = chan - mux->chan;
+ int ret;
+
+ ret = iio_mux_select(mux, idx);
+ if (ret < 0)
+ return ret;
+
+ switch (mask) {
+ case IIO_CHAN_INFO_RAW:
+ ret = iio_write_channel_raw(mux->parent, val);
+ break;
+
+ default:
+ ret = -EINVAL;
+ }
+
+ iio_mux_deselect(mux);
+
+ return ret;
+}
+
+static const struct iio_info mux_info = {
+ .read_raw = mux_read_raw,
+ .read_avail = mux_read_avail,
+ .write_raw = mux_write_raw,
+ .driver_module = THIS_MODULE,
+};
+
+static ssize_t mux_read_ext_info(struct iio_dev *indio_dev, uintptr_t private,
+ struct iio_chan_spec const *chan, char *buf)
+{
+ struct mux *mux = iio_priv(indio_dev);
+ int idx = chan - mux->chan;
+ ssize_t ret;
+
+ ret = iio_mux_select(mux, idx);
+ if (ret < 0)
+ return ret;
+
+ ret = iio_read_channel_ext_info(mux->parent,
+ mux->ext_info[private].name,
+ buf);
+
+ iio_mux_deselect(mux);
+
+ return ret;
+}
+
+static ssize_t mux_write_ext_info(struct iio_dev *indio_dev, uintptr_t private,
+ struct iio_chan_spec const *chan,
+ const char *buf, size_t len)
+{
+ struct device *dev = indio_dev->dev.parent;
+ struct mux *mux = iio_priv(indio_dev);
+ int idx = chan - mux->chan;
+ char *new;
+ ssize_t ret;
+
+ if (len >= PAGE_SIZE)
+ return -EINVAL;
+
+ ret = iio_mux_select(mux, idx);
+ if (ret < 0)
+ return ret;
+
+ new = devm_kmemdup(dev, buf, len + 1, GFP_KERNEL);
+ if (!new) {
+ iio_mux_deselect(mux);
+ return -ENOMEM;
+ }
+
+ new[len] = 0;
+
+ ret = iio_write_channel_ext_info(mux->parent,
+ mux->ext_info[private].name,
+ buf, len);
+ if (ret < 0) {
+ iio_mux_deselect(mux);
+ devm_kfree(dev, new);
+ return ret;
+ }
+
+ devm_kfree(dev, mux->child[idx].ext_info_cache[private].data);
+ mux->child[idx].ext_info_cache[private].data = new;
+ mux->child[idx].ext_info_cache[private].size = len;
+
+ iio_mux_deselect(mux);
+
+ return ret;
+}
+
+static int mux_configure_channel(struct device *dev, struct mux *mux,
+ u32 state, const char *label, int idx)
+{
+ struct mux_child *child = &mux->child[idx];
+ struct iio_chan_spec *chan = &mux->chan[idx];
+ struct iio_chan_spec const *pchan = mux->parent->channel;
+ char *page = NULL;
+ int num_ext_info;
+ int i;
+ int ret;
+
+ chan->indexed = 1;
+ chan->output = pchan->output;
+ chan->datasheet_name = label;
+ chan->ext_info = mux->ext_info;
+
+ ret = iio_get_channel_type(mux->parent, &chan->type);
+ if (ret < 0) {
+ dev_err(dev, "failed to get parent channel type\n");
+ return ret;
+ }
+
+ if (iio_channel_has_info(pchan, IIO_CHAN_INFO_RAW))
+ chan->info_mask_separate |= BIT(IIO_CHAN_INFO_RAW);
+ if (iio_channel_has_info(pchan, IIO_CHAN_INFO_SCALE))
+ chan->info_mask_separate |= BIT(IIO_CHAN_INFO_SCALE);
+
+ if (iio_channel_has_available(pchan, IIO_CHAN_INFO_RAW))
+ chan->info_mask_separate_available |= BIT(IIO_CHAN_INFO_RAW);
+
+ if (state >= mux_control_states(mux->control)) {
+ dev_err(dev, "too many channels\n");
+ return -EINVAL;
+ }
+
+ chan->channel = state;
+
+ num_ext_info = iio_get_channel_ext_info_count(mux->parent);
+ if (num_ext_info) {
+ page = devm_kzalloc(dev, PAGE_SIZE, GFP_KERNEL);
+ if (!page)
+ return -ENOMEM;
+ }
+ child->ext_info_cache = devm_kzalloc(dev,
+ sizeof(*child->ext_info_cache) *
+ num_ext_info, GFP_KERNEL);
+ for (i = 0; i < num_ext_info; ++i) {
+ child->ext_info_cache[i].size = -1;
+
+ if (!pchan->ext_info[i].write)
+ continue;
+ if (!pchan->ext_info[i].read)
+ continue;
+
+ ret = iio_read_channel_ext_info(mux->parent,
+ mux->ext_info[i].name,
+ page);
+ if (ret < 0) {
+ dev_err(dev, "failed to get ext_info '%s'\n",
+ pchan->ext_info[i].name);
+ return ret;
+ }
+ if (ret >= PAGE_SIZE) {
+ dev_err(dev, "too large ext_info '%s'\n",
+ pchan->ext_info[i].name);
+ return -EINVAL;
+ }
+
+ child->ext_info_cache[i].data = devm_kmemdup(dev, page, ret + 1,
+ GFP_KERNEL);
+ child->ext_info_cache[i].data[ret] = 0;
+ child->ext_info_cache[i].size = ret;
+ }
+
+ if (page)
+ devm_kfree(dev, page);
+
+ return 0;
+}
+
+/*
+ * Same as of_property_for_each_string(), but also keeps track of the
+ * index of each string.
+ */
+#define of_property_for_each_string_index(np, propname, prop, s, i) \
+ for (prop = of_find_property(np, propname, NULL), \
+ s = of_prop_next_string(prop, NULL), \
+ i = 0; \
+ s; \
+ s = of_prop_next_string(prop, s), \
+ i++)
+
+static int mux_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct device_node *np = pdev->dev.of_node;
+ struct iio_dev *indio_dev;
+ struct iio_channel *parent;
+ struct mux *mux;
+ struct property *prop;
+ const char *label;
+ u32 state;
+ int sizeof_ext_info;
+ int children;
+ int sizeof_priv;
+ int i;
+ int ret;
+
+ if (!np)
+ return -ENODEV;
+
+ parent = devm_iio_channel_get(dev, "parent");
+ if (IS_ERR(parent)) {
+ if (PTR_ERR(parent) != -EPROBE_DEFER)
+ dev_err(dev, "failed to get parent channel\n");
+ return PTR_ERR(parent);
+ }
+
+ sizeof_ext_info = iio_get_channel_ext_info_count(parent);
+ if (sizeof_ext_info) {
+ sizeof_ext_info += 1; /* one extra entry for the sentinel */
+ sizeof_ext_info *= sizeof(*mux->ext_info);
+ }
+
+ children = 0;
+ of_property_for_each_string(np, "channels", prop, label) {
+ if (*label)
+ children++;
+ }
+ if (children <= 0) {
+ dev_err(dev, "not even a single child\n");
+ return -EINVAL;
+ }
+
+ sizeof_priv = sizeof(*mux);
+ sizeof_priv += sizeof(*mux->child) * children;
+ sizeof_priv += sizeof(*mux->chan) * children;
+ sizeof_priv += sizeof_ext_info;
+
+ indio_dev = devm_iio_device_alloc(dev, sizeof_priv);
+ if (!indio_dev)
+ return -ENOMEM;
+
+ mux = iio_priv(indio_dev);
+ mux->child = (struct mux_child *)(mux + 1);
+ mux->chan = (struct iio_chan_spec *)(mux->child + children);
+
+ platform_set_drvdata(pdev, indio_dev);
+
+ mux->parent = parent;
+ mux->cached_state = -1;
+
+ indio_dev->name = dev_name(dev);
+ indio_dev->dev.parent = dev;
+ indio_dev->info = &mux_info;
+ indio_dev->modes = INDIO_DIRECT_MODE;
+ indio_dev->channels = mux->chan;
+ indio_dev->num_channels = children;
+ if (sizeof_ext_info) {
+ mux->ext_info = devm_kmemdup(dev,
+ parent->channel->ext_info,
+ sizeof_ext_info, GFP_KERNEL);
+ if (!mux->ext_info)
+ return -ENOMEM;
+
+ for (i = 0; mux->ext_info[i].name; ++i) {
+ if (parent->channel->ext_info[i].read)
+ mux->ext_info[i].read = mux_read_ext_info;
+ if (parent->channel->ext_info[i].write)
+ mux->ext_info[i].write = mux_write_ext_info;
+ mux->ext_info[i].private = i;
+ }
+ }
+
+ mux->control = devm_mux_control_get(dev, NULL);
+ if (IS_ERR(mux->control)) {
+ if (PTR_ERR(mux->control) != -EPROBE_DEFER)
+ dev_err(dev, "failed to get control-mux\n");
+ return PTR_ERR(mux->control);
+ }
+
+ i = 0;
+ of_property_for_each_string_index(np, "channels", prop, label, state) {
+ if (!*label)
+ continue;
+
+ ret = mux_configure_channel(dev, mux, state, label, i++);
+ if (ret < 0)
+ return ret;
+ }
+
+ ret = devm_iio_device_register(dev, indio_dev);
+ if (ret) {
+ dev_err(dev, "failed to register iio device\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static const struct of_device_id mux_match[] = {
+ { .compatible = "io-channel-mux" },
+ { /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, mux_match);
+
+static struct platform_driver mux_driver = {
+ .probe = mux_probe,
+ .driver = {
+ .name = "iio-mux",
+ .of_match_table = mux_match,
+ },
+};
+module_platform_driver(mux_driver);
+
+MODULE_DESCRIPTION("IIO multiplexer driver");
+MODULE_AUTHOR("Peter Rosin <peda@axentia.se>");
+MODULE_LICENSE("GPL v2");
struct hid_sensor_common common_attributes;
struct hid_sensor_hub_attribute_info quaternion;
u32 sampled_vals[4];
+ int scale_pre_decml;
+ int scale_post_decml;
+ int scale_precision;
+ int value_offset;
};
/* Channel definitions */
.channel2 = IIO_MOD_QUATERNION,
.info_mask_separate = BIT(IIO_CHAN_INFO_RAW),
.info_mask_shared_by_type = BIT(IIO_CHAN_INFO_SAMP_FREQ) |
+ BIT(IIO_CHAN_INFO_OFFSET) |
+ BIT(IIO_CHAN_INFO_SCALE) |
BIT(IIO_CHAN_INFO_HYSTERESIS)
}
};
} else
ret_type = -EINVAL;
break;
+ case IIO_CHAN_INFO_SCALE:
+ vals[0] = rot_state->scale_pre_decml;
+ vals[1] = rot_state->scale_post_decml;
+ return rot_state->scale_precision;
+
+ case IIO_CHAN_INFO_OFFSET:
+ *vals = rot_state->value_offset;
+ return IIO_VAL_INT;
+
case IIO_CHAN_INFO_SAMP_FREQ:
ret_type = hid_sensor_read_samp_freq_value(
&rot_state->common_attributes, &vals[0], &vals[1]);
dev_dbg(&pdev->dev, "dev_rot: attrib size %d\n",
st->quaternion.size);
+ st->scale_precision = hid_sensor_format_scale(
+ hsdev->usage,
+ &st->quaternion,
+ &st->scale_pre_decml, &st->scale_post_decml);
+
/* Set Sensitivity field ids, when there is no individual modifier */
if (st->common_attributes.sensitivity.index < 0) {
sensor_hub_input_get_attribute_info(hsdev,
static int hid_dev_rot_probe(struct platform_device *pdev)
{
int ret;
- static char *name = "dev_rotation";
+ static char *name;
struct iio_dev *indio_dev;
struct dev_rot_state *rot_state;
struct hid_sensor_hub_device *hsdev = pdev->dev.platform_data;
rot_state->common_attributes.hsdev = hsdev;
rot_state->common_attributes.pdev = pdev;
- ret = hid_sensor_parse_common_attributes(hsdev,
- HID_USAGE_SENSOR_DEVICE_ORIENTATION,
+ switch (hsdev->usage) {
+ case HID_USAGE_SENSOR_DEVICE_ORIENTATION:
+ name = "dev_rotation";
+ break;
+ case HID_USAGE_SENSOR_RELATIVE_ORIENTATION:
+ name = "relative_orientation";
+ break;
+ case HID_USAGE_SENSOR_GEOMAGNETIC_ORIENTATION:
+ name = "geomagnetic_orientation";
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ ret = hid_sensor_parse_common_attributes(hsdev, hsdev->usage,
&rot_state->common_attributes);
if (ret) {
dev_err(&pdev->dev, "failed to setup common attributes\n");
ret = dev_rot_parse_report(pdev, hsdev,
(struct iio_chan_spec *)indio_dev->channels,
- HID_USAGE_SENSOR_DEVICE_ORIENTATION,
- rot_state);
+ hsdev->usage, rot_state);
if (ret) {
dev_err(&pdev->dev, "failed to setup attributes\n");
return ret;
rot_state->callbacks.send_event = dev_rot_proc_event;
rot_state->callbacks.capture_sample = dev_rot_capture_sample;
rot_state->callbacks.pdev = pdev;
- ret = sensor_hub_register_callback(hsdev,
- HID_USAGE_SENSOR_DEVICE_ORIENTATION,
+ ret = sensor_hub_register_callback(hsdev, hsdev->usage,
&rot_state->callbacks);
if (ret) {
dev_err(&pdev->dev, "callback reg failed\n");
struct iio_dev *indio_dev = platform_get_drvdata(pdev);
struct dev_rot_state *rot_state = iio_priv(indio_dev);
- sensor_hub_remove_callback(hsdev, HID_USAGE_SENSOR_DEVICE_ORIENTATION);
+ sensor_hub_remove_callback(hsdev, hsdev->usage);
iio_device_unregister(indio_dev);
hid_sensor_remove_trigger(&rot_state->common_attributes);
iio_triggered_buffer_cleanup(indio_dev);
/* Format: HID-SENSOR-usage_id_in_hex_lowercase */
.name = "HID-SENSOR-20008a",
},
+ {
+ /* Relative orientation(AG) sensor */
+ .name = "HID-SENSOR-20008e",
+ },
+ {
+ /* Geomagnetic orientation(AM) sensor */
+ .name = "HID-SENSOR-2000c1",
+ },
{ /* sentinel */ }
};
MODULE_DEVICE_TABLE(platform, hid_dev_rot_ids);
select BMP280_SPI if (SPI_MASTER)
help
Say yes here to build support for Bosch Sensortec BMP180 and BMP280
- pressure and temperature sensors. Also supports the BE280 with
+ pressure and temperature sensors. Also supports the BME280 with
an additional humidity sensor channel.
To compile this driver as a module, choose M here: the core module
int st_press_common_probe(struct iio_dev *indio_dev)
{
struct st_sensor_data *press_data = iio_priv(indio_dev);
+ struct st_sensors_platform_data *pdata =
+ (struct st_sensors_platform_data *)press_data->dev->platform_data;
int irq = press_data->get_irq_data_ready(indio_dev);
int err;
press_data->odr = press_data->sensor_settings->odr.odr_avl[0].hz;
/* Some devices don't support a data ready pin. */
- if (!press_data->dev->platform_data &&
- press_data->sensor_settings->drdy_irq.addr)
- press_data->dev->platform_data =
- (struct st_sensors_platform_data *)&default_press_pdata;
+ if (!pdata && press_data->sensor_settings->drdy_irq.addr)
+ pdata = (struct st_sensors_platform_data *)&default_press_pdata;
err = st_sensors_init_sensor(indio_dev, press_data->dev->platform_data);
if (err < 0)
{
int ret;
unsigned int val;
+ long timeout;
zpa2326_dbg(indio_dev, "waiting for one shot completion interrupt");
- ret = wait_for_completion_interruptible_timeout(
+ timeout = wait_for_completion_interruptible_timeout(
&private->data_ready, ZPA2326_CONVERSION_JIFFIES);
- if (ret > 0)
+ if (timeout > 0)
/*
* Interrupt handler completed before timeout: return operation
* status.
/* Clear all interrupts just to be sure. */
regmap_read(private->regmap, ZPA2326_INT_SOURCE_REG, &val);
- if (!ret)
+ if (!timeout) {
/* Timed out. */
+ zpa2326_warn(indio_dev, "no one shot interrupt occurred (%ld)",
+ timeout);
ret = -ETIME;
-
- if (ret != -ERESTARTSYS)
- zpa2326_warn(indio_dev, "no one shot interrupt occurred (%d)",
- ret);
+ } else if (timeout < 0) {
+ zpa2326_warn(indio_dev,
+ "wait for one shot interrupt cancelled");
+ ret = -ERESTARTSYS;
+ }
return ret;
}
if (ret)
return ret;
- if (m == IIO_CHAN_INFO_RAW)
- return IIO_VAL_INT;
-
/* storm out of range */
if (*val == AS3935_DATA_MASK)
return -EINVAL;
+ if (m == IIO_CHAN_INFO_RAW)
+ return IIO_VAL_INT;
+
if (m == IIO_CHAN_INFO_PROCESSED)
*val *= 1000;
break;
dev = &client->dev;
- data->gpiod_rst = devm_gpiod_get_index(dev, SX9500_GPIO_RESET,
- 0, GPIOD_OUT_HIGH);
+ data->gpiod_rst = devm_gpiod_get(dev, SX9500_GPIO_RESET, GPIOD_OUT_HIGH);
if (IS_ERR(data->gpiod_rst)) {
dev_warn(dev, "gpio get reset pin failed\n");
data->gpiod_rst = NULL;
static const struct spi_device_id maxim_thermocouple_id[] = {
{"max6675", MAX6675},
{"max31855", MAX31855},
+ {"max31856", MAX31855},
{},
};
MODULE_DEVICE_TABLE(spi, maxim_thermocouple_id);
#include <linux/module.h>
#include <linux/platform_device.h>
-#define MAX_TRIGGERS 6
+#define MAX_TRIGGERS 7
#define MAX_VALIDS 5
/* List the triggers created by each timer */
static const void *triggers_table[][MAX_TRIGGERS] = {
- { TIM1_TRGO, TIM1_CH1, TIM1_CH2, TIM1_CH3, TIM1_CH4,},
+ { TIM1_TRGO, TIM1_TRGO2, TIM1_CH1, TIM1_CH2, TIM1_CH3, TIM1_CH4,},
{ TIM2_TRGO, TIM2_CH1, TIM2_CH2, TIM2_CH3, TIM2_CH4,},
{ TIM3_TRGO, TIM3_CH1, TIM3_CH2, TIM3_CH3, TIM3_CH4,},
{ TIM4_TRGO, TIM4_CH1, TIM4_CH2, TIM4_CH3, TIM4_CH4,},
{ TIM5_TRGO, TIM5_CH1, TIM5_CH2, TIM5_CH3, TIM5_CH4,},
{ TIM6_TRGO,},
{ TIM7_TRGO,},
- { TIM8_TRGO, TIM8_CH1, TIM8_CH2, TIM8_CH3, TIM8_CH4,},
+ { TIM8_TRGO, TIM8_TRGO2, TIM8_CH1, TIM8_CH2, TIM8_CH3, TIM8_CH4,},
{ TIM9_TRGO, TIM9_CH1, TIM9_CH2,},
{ }, /* timer 10 */
{ }, /* timer 11 */
u32 max_arr;
const void *triggers;
const void *valids;
+ bool has_trgo2;
};
+static bool stm32_timer_is_trgo2_name(const char *name)
+{
+ return !!strstr(name, "trgo2");
+}
+
static int stm32_timer_start(struct stm32_timer_trigger *priv,
+ struct iio_trigger *trig,
unsigned int frequency)
{
unsigned long long prd, div;
regmap_update_bits(priv->regmap, TIM_CR1, TIM_CR1_ARPE, TIM_CR1_ARPE);
/* Force master mode to update mode */
- regmap_update_bits(priv->regmap, TIM_CR2, TIM_CR2_MMS, 0x20);
+ if (stm32_timer_is_trgo2_name(trig->name))
+ regmap_update_bits(priv->regmap, TIM_CR2, TIM_CR2_MMS2,
+ 0x2 << TIM_CR2_MMS2_SHIFT);
+ else
+ regmap_update_bits(priv->regmap, TIM_CR2, TIM_CR2_MMS,
+ 0x2 << TIM_CR2_MMS_SHIFT);
/* Make sure that registers are updated */
regmap_update_bits(priv->regmap, TIM_EGR, TIM_EGR_UG, TIM_EGR_UG);
if (freq == 0) {
stm32_timer_stop(priv);
} else {
- ret = stm32_timer_start(priv, freq);
+ ret = stm32_timer_start(priv, trig, freq);
if (ret)
return ret;
}
stm32_tt_read_frequency,
stm32_tt_store_frequency);
+#define MASTER_MODE_MAX 7
+#define MASTER_MODE2_MAX 15
+
static char *master_mode_table[] = {
"reset",
"enable",
"OC1REF",
"OC2REF",
"OC3REF",
- "OC4REF"
+ "OC4REF",
+ /* Master mode selection 2 only */
+ "OC5REF",
+ "OC6REF",
+ "compare_pulse_OC4REF",
+ "compare_pulse_OC6REF",
+ "compare_pulse_OC4REF_r_or_OC6REF_r",
+ "compare_pulse_OC4REF_r_or_OC6REF_f",
+ "compare_pulse_OC5REF_r_or_OC6REF_r",
+ "compare_pulse_OC5REF_r_or_OC6REF_f",
};
static ssize_t stm32_tt_show_master_mode(struct device *dev,
char *buf)
{
struct stm32_timer_trigger *priv = dev_get_drvdata(dev);
+ struct iio_trigger *trig = to_iio_trigger(dev);
u32 cr2;
regmap_read(priv->regmap, TIM_CR2, &cr2);
- cr2 = (cr2 & TIM_CR2_MMS) >> TIM_CR2_MMS_SHIFT;
+
+ if (stm32_timer_is_trgo2_name(trig->name))
+ cr2 = (cr2 & TIM_CR2_MMS2) >> TIM_CR2_MMS2_SHIFT;
+ else
+ cr2 = (cr2 & TIM_CR2_MMS) >> TIM_CR2_MMS_SHIFT;
return snprintf(buf, PAGE_SIZE, "%s\n", master_mode_table[cr2]);
}
const char *buf, size_t len)
{
struct stm32_timer_trigger *priv = dev_get_drvdata(dev);
+ struct iio_trigger *trig = to_iio_trigger(dev);
+ u32 mask, shift, master_mode_max;
int i;
- for (i = 0; i < ARRAY_SIZE(master_mode_table); i++) {
+ if (stm32_timer_is_trgo2_name(trig->name)) {
+ mask = TIM_CR2_MMS2;
+ shift = TIM_CR2_MMS2_SHIFT;
+ master_mode_max = MASTER_MODE2_MAX;
+ } else {
+ mask = TIM_CR2_MMS;
+ shift = TIM_CR2_MMS_SHIFT;
+ master_mode_max = MASTER_MODE_MAX;
+ }
+
+ for (i = 0; i <= master_mode_max; i++) {
if (!strncmp(master_mode_table[i], buf,
strlen(master_mode_table[i]))) {
- regmap_update_bits(priv->regmap, TIM_CR2,
- TIM_CR2_MMS, i << TIM_CR2_MMS_SHIFT);
+ regmap_update_bits(priv->regmap, TIM_CR2, mask,
+ i << shift);
/* Make sure that registers are updated */
regmap_update_bits(priv->regmap, TIM_EGR,
TIM_EGR_UG, TIM_EGR_UG);
return -EINVAL;
}
-static IIO_CONST_ATTR(master_mode_available,
- "reset enable update compare_pulse OC1REF OC2REF OC3REF OC4REF");
+static ssize_t stm32_tt_show_master_mode_avail(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct iio_trigger *trig = to_iio_trigger(dev);
+ unsigned int i, master_mode_max;
+ size_t len = 0;
+
+ if (stm32_timer_is_trgo2_name(trig->name))
+ master_mode_max = MASTER_MODE2_MAX;
+ else
+ master_mode_max = MASTER_MODE_MAX;
+
+ for (i = 0; i <= master_mode_max; i++)
+ len += scnprintf(buf + len, PAGE_SIZE - len,
+ "%s ", master_mode_table[i]);
+
+ /* replace trailing space by newline */
+ buf[len - 1] = '\n';
+
+ return len;
+}
+
+static IIO_DEVICE_ATTR(master_mode_available, 0444,
+ stm32_tt_show_master_mode_avail, NULL, 0);
static IIO_DEVICE_ATTR(master_mode, 0660,
stm32_tt_show_master_mode,
static struct attribute *stm32_trigger_attrs[] = {
&iio_dev_attr_sampling_frequency.dev_attr.attr,
&iio_dev_attr_master_mode.dev_attr.attr,
- &iio_const_attr_master_mode_available.dev_attr.attr,
+ &iio_dev_attr_master_mode_available.dev_attr.attr,
NULL,
};
while (cur && *cur) {
struct iio_trigger *trig;
+ bool cur_is_trgo2 = stm32_timer_is_trgo2_name(*cur);
+
+ if (cur_is_trgo2 && !priv->has_trgo2) {
+ cur++;
+ continue;
+ }
trig = devm_iio_trigger_alloc(priv->dev, "%s", *cur);
if (!trig)
* should only be available on trgo trigger which
* is always the first in the list.
*/
- if (cur == priv->triggers)
+ if (cur == priv->triggers || cur_is_trgo2)
trig->dev.groups = stm32_trigger_attr_groups;
iio_trigger_set_drvdata(trig, priv);
return -EINVAL;
}
+static int stm32_counter_validate_trigger(struct iio_dev *indio_dev,
+ struct iio_trigger *trig)
+{
+ struct stm32_timer_trigger *priv = iio_priv(indio_dev);
+ const char * const *cur = priv->valids;
+ unsigned int i = 0;
+
+ if (!is_stm32_timer_trigger(trig))
+ return -EINVAL;
+
+ while (cur && *cur) {
+ if (!strncmp(trig->name, *cur, strlen(trig->name))) {
+ regmap_update_bits(priv->regmap,
+ TIM_SMCR, TIM_SMCR_TS,
+ i << TIM_SMCR_TS_SHIFT);
+ return 0;
+ }
+ cur++;
+ i++;
+ }
+
+ return -EINVAL;
+}
+
static const struct iio_info stm32_trigger_info = {
.driver_module = THIS_MODULE,
+ .validate_trigger = stm32_counter_validate_trigger,
.read_raw = stm32_counter_read_raw,
.write_raw = stm32_counter_write_raw
};
+static const char *const stm32_trigger_modes[] = {
+ "trigger",
+};
+
+static int stm32_set_trigger_mode(struct iio_dev *indio_dev,
+ const struct iio_chan_spec *chan,
+ unsigned int mode)
+{
+ struct stm32_timer_trigger *priv = iio_priv(indio_dev);
+
+ regmap_update_bits(priv->regmap, TIM_SMCR, TIM_SMCR_SMS, TIM_SMCR_SMS);
+
+ return 0;
+}
+
+static int stm32_get_trigger_mode(struct iio_dev *indio_dev,
+ const struct iio_chan_spec *chan)
+{
+ struct stm32_timer_trigger *priv = iio_priv(indio_dev);
+ u32 smcr;
+
+ regmap_read(priv->regmap, TIM_SMCR, &smcr);
+
+ return smcr == TIM_SMCR_SMS ? 0 : -EINVAL;
+}
+
+static const struct iio_enum stm32_trigger_mode_enum = {
+ .items = stm32_trigger_modes,
+ .num_items = ARRAY_SIZE(stm32_trigger_modes),
+ .set = stm32_set_trigger_mode,
+ .get = stm32_get_trigger_mode
+};
+
static const char *const stm32_enable_modes[] = {
"always",
"gated",
IIO_ENUM_AVAILABLE("quadrature_mode", &stm32_quadrature_mode_enum),
IIO_ENUM("enable_mode", IIO_SEPARATE, &stm32_enable_mode_enum),
IIO_ENUM_AVAILABLE("enable_mode", &stm32_enable_mode_enum),
+ IIO_ENUM("trigger_mode", IIO_SEPARATE, &stm32_trigger_mode_enum),
+ IIO_ENUM_AVAILABLE("trigger_mode", &stm32_trigger_mode_enum),
{}
};
indio_dev->name = dev_name(dev);
indio_dev->dev.parent = dev;
indio_dev->info = &stm32_trigger_info;
+ indio_dev->modes = INDIO_HARDWARE_TRIGGERED;
indio_dev->num_channels = 1;
indio_dev->channels = &stm32_trigger_channel;
indio_dev->dev.of_node = dev->of_node;
}
EXPORT_SYMBOL(is_stm32_timer_trigger);
+static void stm32_timer_detect_trgo2(struct stm32_timer_trigger *priv)
+{
+ u32 val;
+
+ /*
+ * Master mode selection 2 bits can only be written and read back when
+ * timer supports it.
+ */
+ regmap_update_bits(priv->regmap, TIM_CR2, TIM_CR2_MMS2, TIM_CR2_MMS2);
+ regmap_read(priv->regmap, TIM_CR2, &val);
+ regmap_update_bits(priv->regmap, TIM_CR2, TIM_CR2_MMS2, 0);
+ priv->has_trgo2 = !!val;
+}
+
static int stm32_timer_trigger_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
priv->max_arr = ddata->max_arr;
priv->triggers = triggers_table[index];
priv->valids = valids_table[index];
+ stm32_timer_detect_trgo2(priv);
ret = stm32_setup_iio_triggers(priv);
if (ret)
bool i40iw_vf_clear_to_send(struct i40iw_sc_dev *dev)
{
struct i40iw_device *iwdev;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
iwdev = dev->back_dev;
.remove = nes_remove,
};
-static ssize_t nes_show_adapter(struct device_driver *ddp, char *buf)
+static ssize_t adapter_show(struct device_driver *ddp, char *buf)
{
unsigned int devfn = 0xffffffff;
unsigned char bus_number = 0xff;
return snprintf(buf, PAGE_SIZE, "%x:%x\n", bus_number, devfn);
}
-static ssize_t nes_store_adapter(struct device_driver *ddp,
+static ssize_t adapter_store(struct device_driver *ddp,
const char *buf, size_t count)
{
char *p = (char *)buf;
return strnlen(buf, count);
}
-static ssize_t nes_show_ee_cmd(struct device_driver *ddp, char *buf)
+static ssize_t eeprom_cmd_show(struct device_driver *ddp, char *buf)
{
u32 eeprom_cmd = 0xdead;
u32 i = 0;
return snprintf(buf, PAGE_SIZE, "0x%x\n", eeprom_cmd);
}
-static ssize_t nes_store_ee_cmd(struct device_driver *ddp,
+static ssize_t eeprom_cmd_store(struct device_driver *ddp,
const char *buf, size_t count)
{
char *p = (char *)buf;
return strnlen(buf, count);
}
-static ssize_t nes_show_ee_data(struct device_driver *ddp, char *buf)
+static ssize_t eeprom_data_show(struct device_driver *ddp, char *buf)
{
u32 eeprom_data = 0xdead;
u32 i = 0;
return snprintf(buf, PAGE_SIZE, "0x%x\n", eeprom_data);
}
-static ssize_t nes_store_ee_data(struct device_driver *ddp,
+static ssize_t eeprom_data_store(struct device_driver *ddp,
const char *buf, size_t count)
{
char *p = (char *)buf;
return strnlen(buf, count);
}
-static ssize_t nes_show_flash_cmd(struct device_driver *ddp, char *buf)
+static ssize_t flash_cmd_show(struct device_driver *ddp, char *buf)
{
u32 flash_cmd = 0xdead;
u32 i = 0;
return snprintf(buf, PAGE_SIZE, "0x%x\n", flash_cmd);
}
-static ssize_t nes_store_flash_cmd(struct device_driver *ddp,
+static ssize_t flash_cmd_store(struct device_driver *ddp,
const char *buf, size_t count)
{
char *p = (char *)buf;
return strnlen(buf, count);
}
-static ssize_t nes_show_flash_data(struct device_driver *ddp, char *buf)
+static ssize_t flash_data_show(struct device_driver *ddp, char *buf)
{
u32 flash_data = 0xdead;
u32 i = 0;
return snprintf(buf, PAGE_SIZE, "0x%x\n", flash_data);
}
-static ssize_t nes_store_flash_data(struct device_driver *ddp,
+static ssize_t flash_data_store(struct device_driver *ddp,
const char *buf, size_t count)
{
char *p = (char *)buf;
return strnlen(buf, count);
}
-static ssize_t nes_show_nonidx_addr(struct device_driver *ddp, char *buf)
+static ssize_t nonidx_addr_show(struct device_driver *ddp, char *buf)
{
return snprintf(buf, PAGE_SIZE, "0x%x\n", sysfs_nonidx_addr);
}
-static ssize_t nes_store_nonidx_addr(struct device_driver *ddp,
+static ssize_t nonidx_addr_store(struct device_driver *ddp,
const char *buf, size_t count)
{
char *p = (char *)buf;
return strnlen(buf, count);
}
-static ssize_t nes_show_nonidx_data(struct device_driver *ddp, char *buf)
+static ssize_t nonidx_data_show(struct device_driver *ddp, char *buf)
{
u32 nonidx_data = 0xdead;
u32 i = 0;
return snprintf(buf, PAGE_SIZE, "0x%x\n", nonidx_data);
}
-static ssize_t nes_store_nonidx_data(struct device_driver *ddp,
+static ssize_t nonidx_data_store(struct device_driver *ddp,
const char *buf, size_t count)
{
char *p = (char *)buf;
return strnlen(buf, count);
}
-static ssize_t nes_show_idx_addr(struct device_driver *ddp, char *buf)
+static ssize_t idx_addr_show(struct device_driver *ddp, char *buf)
{
return snprintf(buf, PAGE_SIZE, "0x%x\n", sysfs_idx_addr);
}
-static ssize_t nes_store_idx_addr(struct device_driver *ddp,
+static ssize_t idx_addr_store(struct device_driver *ddp,
const char *buf, size_t count)
{
char *p = (char *)buf;
return strnlen(buf, count);
}
-static ssize_t nes_show_idx_data(struct device_driver *ddp, char *buf)
+static ssize_t idx_data_show(struct device_driver *ddp, char *buf)
{
u32 idx_data = 0xdead;
u32 i = 0;
return snprintf(buf, PAGE_SIZE, "0x%x\n", idx_data);
}
-static ssize_t nes_store_idx_data(struct device_driver *ddp,
+static ssize_t idx_data_store(struct device_driver *ddp,
const char *buf, size_t count)
{
char *p = (char *)buf;
return strnlen(buf, count);
}
-
-/**
- * nes_show_wqm_quanta
- */
-static ssize_t nes_show_wqm_quanta(struct device_driver *ddp, char *buf)
+static ssize_t wqm_quanta_show(struct device_driver *ddp, char *buf)
{
u32 wqm_quanta_value = 0xdead;
u32 i = 0;
return snprintf(buf, PAGE_SIZE, "0x%X\n", wqm_quanta_value);
}
-
-/**
- * nes_store_wqm_quanta
- */
-static ssize_t nes_store_wqm_quanta(struct device_driver *ddp,
- const char *buf, size_t count)
+static ssize_t wqm_quanta_store(struct device_driver *ddp, const char *buf,
+ size_t count)
{
unsigned long wqm_quanta_value;
u32 wqm_config1;
return strnlen(buf, count);
}
-static DRIVER_ATTR(adapter, S_IRUSR | S_IWUSR,
- nes_show_adapter, nes_store_adapter);
-static DRIVER_ATTR(eeprom_cmd, S_IRUSR | S_IWUSR,
- nes_show_ee_cmd, nes_store_ee_cmd);
-static DRIVER_ATTR(eeprom_data, S_IRUSR | S_IWUSR,
- nes_show_ee_data, nes_store_ee_data);
-static DRIVER_ATTR(flash_cmd, S_IRUSR | S_IWUSR,
- nes_show_flash_cmd, nes_store_flash_cmd);
-static DRIVER_ATTR(flash_data, S_IRUSR | S_IWUSR,
- nes_show_flash_data, nes_store_flash_data);
-static DRIVER_ATTR(nonidx_addr, S_IRUSR | S_IWUSR,
- nes_show_nonidx_addr, nes_store_nonidx_addr);
-static DRIVER_ATTR(nonidx_data, S_IRUSR | S_IWUSR,
- nes_show_nonidx_data, nes_store_nonidx_data);
-static DRIVER_ATTR(idx_addr, S_IRUSR | S_IWUSR,
- nes_show_idx_addr, nes_store_idx_addr);
-static DRIVER_ATTR(idx_data, S_IRUSR | S_IWUSR,
- nes_show_idx_data, nes_store_idx_data);
-static DRIVER_ATTR(wqm_quanta, S_IRUSR | S_IWUSR,
- nes_show_wqm_quanta, nes_store_wqm_quanta);
+static DRIVER_ATTR_RW(adapter);
+static DRIVER_ATTR_RW(eeprom_cmd);
+static DRIVER_ATTR_RW(eeprom_data);
+static DRIVER_ATTR_RW(flash_cmd);
+static DRIVER_ATTR_RW(flash_data);
+static DRIVER_ATTR_RW(nonidx_addr);
+static DRIVER_ATTR_RW(nonidx_data);
+static DRIVER_ATTR_RW(idx_addr);
+static DRIVER_ATTR_RW(idx_data);
+static DRIVER_ATTR_RW(wqm_quanta);
static int nes_create_driver_sysfs(struct pci_driver *drv)
{
if (!btns_desc) {
dev_err(dev, "ACPI Button Descriptors not found\n");
- return ERR_PTR(-ENODEV);
+ button_info = ERR_PTR(-ENODEV);
+ goto out;
}
/* The first package describes the collection */
}
if (collection_uid == -1) {
dev_err(dev, "Invalid Button Collection Descriptor\n");
- return ERR_PTR(-ENODEV);
+ button_info = ERR_PTR(-ENODEV);
+ goto out;
}
/* There are package.count - 1 buttons + 1 terminating empty entry */
button_info = devm_kcalloc(dev, btns_desc->package.count,
sizeof(*button_info), GFP_KERNEL);
- if (!button_info)
- return ERR_PTR(-ENOMEM);
+ if (!button_info) {
+ button_info = ERR_PTR(-ENOMEM);
+ goto out;
+ }
/* Parse the button descriptors */
for (i = 1, btn = 0; i < btns_desc->package.count; i++, btn++) {
if (soc_button_parse_btn_desc(dev,
&btns_desc->package.elements[i],
collection_uid,
- &button_info[btn]))
- return ERR_PTR(-ENODEV);
+ &button_info[btn])) {
+ button_info = ERR_PTR(-ENODEV);
+ goto out;
+ }
}
+out:
+ kfree(buf.pointer);
return button_info;
}
#define F54_GET_REPORT 1
#define F54_FORCE_CAL 2
-/* Fixed sizes of reports */
-#define F54_QUERY_LEN 27
-
/* F54 capabilities */
#define F54_CAP_BASELINE (1 << 2)
#define F54_CAP_IMAGE8 (1 << 3)
struct f54_data {
struct rmi_function *fn;
- u8 qry[F54_QUERY_LEN];
u8 num_rx_electrodes;
u8 num_tx_electrodes;
u8 capabilities;
{
int error;
struct f54_data *f54;
+ u8 buf[6];
f54 = dev_get_drvdata(&fn->dev);
error = rmi_read_block(fn->rmi_dev, fn->fd.query_base_addr,
- &f54->qry, sizeof(f54->qry));
+ buf, sizeof(buf));
if (error) {
dev_err(&fn->dev, "%s: Failed to query F54 properties\n",
__func__);
return error;
}
- f54->num_rx_electrodes = f54->qry[0];
- f54->num_tx_electrodes = f54->qry[1];
- f54->capabilities = f54->qry[2];
- f54->clock_rate = f54->qry[3] | (f54->qry[4] << 8);
- f54->family = f54->qry[5];
+ f54->num_rx_electrodes = buf[0];
+ f54->num_tx_electrodes = buf[1];
+ f54->capabilities = buf[2];
+ f54->clock_rate = buf[3] | (buf[4] << 8);
+ f54->family = buf[5];
rmi_dbg(RMI_DEBUG_FN, &fn->dev, "F54 num_rx_electrodes: %d\n",
f54->num_rx_electrodes);
DMI_MATCH(DMI_PRODUCT_NAME, "LIFEBOOK U574"),
},
},
+ {
+ /* Fujitsu UH554 laptop */
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "FUJITSU"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "LIFEBOOK UH544"),
+ },
+ },
{ }
};
u8 vector, u32 dest_apicid, int devid)
{
struct irte_ga *irte = (struct irte_ga *) entry;
- struct iommu_dev_data *dev_data = search_dev_data(devid);
irte->lo.val = 0;
irte->hi.val = 0;
- irte->lo.fields_remap.guest_mode = dev_data ? dev_data->use_vapic : 0;
irte->lo.fields_remap.int_type = delivery_mode;
irte->lo.fields_remap.dm = dest_mode;
irte->hi.fields.vector = vector;
struct irte_ga *irte = (struct irte_ga *) entry;
struct iommu_dev_data *dev_data = search_dev_data(devid);
- if (!dev_data || !dev_data->use_vapic) {
+ if (!dev_data || !dev_data->use_vapic ||
+ !irte->lo.fields_remap.guest_mode) {
irte->hi.fields.vector = vector;
irte->lo.fields_remap.destination = dest_apicid;
- irte->lo.fields_remap.guest_mode = 0;
modify_irte_ga(devid, index, irte, NULL);
}
}
}
static struct irq_chip amd_ir_chip = {
- .irq_ack = ir_ack_apic_edge,
- .irq_set_affinity = amd_ir_set_affinity,
- .irq_set_vcpu_affinity = amd_ir_set_vcpu_affinity,
- .irq_compose_msi_msg = ir_compose_msi_msg,
+ .name = "AMD-IR",
+ .irq_ack = ir_ack_apic_edge,
+ .irq_set_affinity = amd_ir_set_affinity,
+ .irq_set_vcpu_affinity = amd_ir_set_vcpu_affinity,
+ .irq_compose_msi_msg = ir_compose_msi_msg,
};
int amd_iommu_create_irq_domain(struct amd_iommu *iommu)
{
- iommu->ir_domain = irq_domain_add_tree(NULL, &amd_ir_domain_ops, iommu);
+ struct fwnode_handle *fn;
+
+ fn = irq_domain_alloc_named_id_fwnode("AMD-IR", iommu->index);
+ if (!fn)
+ return -ENOMEM;
+ iommu->ir_domain = irq_domain_create_tree(fn, &amd_ir_domain_ops, iommu);
+ irq_domain_free_fwnode(fn);
if (!iommu->ir_domain)
return -ENOMEM;
iommu->ir_domain->parent = arch_get_ir_parent_domain();
- iommu->msi_domain = arch_create_msi_irq_domain(iommu->ir_domain);
-
+ iommu->msi_domain = arch_create_remap_msi_irq_domain(iommu->ir_domain,
+ "AMD-IR-MSI",
+ iommu->index);
return 0;
}
* for Directed-IO Architecture Specifiction, Rev 2.2, Section 8.8
* "Remapping Hardware Unit Hot Plug".
*/
-static u8 dmar_hp_uuid[] = {
- /* 0000 */ 0xA6, 0xA3, 0xC1, 0xD8, 0x9B, 0xBE, 0x9B, 0x4C,
- /* 0008 */ 0x91, 0xBF, 0xC3, 0xCB, 0x81, 0xFC, 0x5D, 0xAF
-};
+static guid_t dmar_hp_guid =
+ GUID_INIT(0xD8C1A3A6, 0xBE9B, 0x4C9B,
+ 0x91, 0xBF, 0xC3, 0xCB, 0x81, 0xFC, 0x5D, 0xAF);
/*
* Currently there's only one revision and BIOS will not check the revision id,
static inline bool dmar_detect_dsm(acpi_handle handle, int func)
{
- return acpi_check_dsm(handle, dmar_hp_uuid, DMAR_DSM_REV_ID, 1 << func);
+ return acpi_check_dsm(handle, &dmar_hp_guid, DMAR_DSM_REV_ID, 1 << func);
}
static int dmar_walk_dsm_resource(acpi_handle handle, int func,
if (!dmar_detect_dsm(handle, func))
return 0;
- obj = acpi_evaluate_dsm_typed(handle, dmar_hp_uuid, DMAR_DSM_REV_ID,
+ obj = acpi_evaluate_dsm_typed(handle, &dmar_hp_guid, DMAR_DSM_REV_ID,
func, NULL, ACPI_TYPE_BUFFER);
if (!obj)
return -ENODEV;
struct acpi_dmar_atsr *atsr;
struct dmar_atsr_unit *atsru;
- if (system_state != SYSTEM_BOOTING && !intel_iommu_enabled)
+ if (system_state >= SYSTEM_RUNNING && !intel_iommu_enabled)
return 0;
atsr = container_of(hdr, struct acpi_dmar_atsr, header);
struct acpi_dmar_atsr *atsr;
struct acpi_dmar_reserved_memory *rmrr;
- if (!intel_iommu_enabled && system_state != SYSTEM_BOOTING)
+ if (!intel_iommu_enabled && system_state >= SYSTEM_RUNNING)
return 0;
list_for_each_entry(rmrru, &dmar_rmrr_units, list) {
static int intel_setup_irq_remapping(struct intel_iommu *iommu)
{
struct ir_table *ir_table;
- struct page *pages;
+ struct fwnode_handle *fn;
unsigned long *bitmap;
+ struct page *pages;
if (iommu->ir_table)
return 0;
goto out_free_pages;
}
- iommu->ir_domain = irq_domain_add_hierarchy(arch_get_ir_parent_domain(),
- 0, INTR_REMAP_TABLE_ENTRIES,
- NULL, &intel_ir_domain_ops,
- iommu);
+ fn = irq_domain_alloc_named_id_fwnode("INTEL-IR", iommu->seq_id);
+ if (!fn)
+ goto out_free_bitmap;
+
+ iommu->ir_domain =
+ irq_domain_create_hierarchy(arch_get_ir_parent_domain(),
+ 0, INTR_REMAP_TABLE_ENTRIES,
+ fn, &intel_ir_domain_ops,
+ iommu);
+ irq_domain_free_fwnode(fn);
if (!iommu->ir_domain) {
pr_err("IR%d: failed to allocate irqdomain\n", iommu->seq_id);
goto out_free_bitmap;
}
- iommu->ir_msi_domain = arch_create_msi_irq_domain(iommu->ir_domain);
+ iommu->ir_msi_domain =
+ arch_create_remap_msi_irq_domain(iommu->ir_domain,
+ "INTEL-IR-MSI",
+ iommu->seq_id);
ir_table->base = page_address(pages);
ir_table->bitmap = bitmap;
}
static struct irq_chip intel_ir_chip = {
- .irq_ack = ir_ack_apic_edge,
- .irq_set_affinity = intel_ir_set_affinity,
- .irq_compose_msi_msg = intel_ir_compose_msi_msg,
- .irq_set_vcpu_affinity = intel_ir_set_vcpu_affinity,
+ .name = "INTEL-IR",
+ .irq_ack = ir_ack_apic_edge,
+ .irq_set_affinity = intel_ir_set_affinity,
+ .irq_compose_msi_msg = intel_ir_compose_msi_msg,
+ .irq_set_vcpu_affinity = intel_ir_set_vcpu_affinity,
};
static void intel_irq_remapping_prepare_irte(struct intel_ir_data *data,
* it never will be. We don't want to defer indefinitely, nor attempt
* to dereference __iommu_of_table after it's been freed.
*/
- if (system_state > SYSTEM_BOOTING)
+ if (system_state >= SYSTEM_RUNNING)
return false;
return of_match_node(&__iommu_of_table, np);
int bus_nr;
struct ipack_bus_device *bus;
- bus = kzalloc(sizeof(struct ipack_bus_device), GFP_KERNEL);
+ bus = kzalloc(sizeof(*bus), GFP_KERNEL);
if (!bus)
return NULL;
* ID ROM contents */
dev->id = kmalloc(dev->id_avail, GFP_KERNEL);
if (!dev->id) {
- dev_err(&dev->dev, "dev->id alloc failed.\n");
ret = -ENOMEM;
goto out;
}
select IRQ_DOMAIN
select STMP_DEVICE
+config MVEBU_GICP
+ bool
+
+config MVEBU_ICU
+ bool
+
config MVEBU_ODMI
bool
select GENERIC_MSI_IRQ_DOMAIN
obj-$(CONFIG_INGENIC_IRQ) += irq-ingenic.o
obj-$(CONFIG_IMX_GPCV2) += irq-imx-gpcv2.o
obj-$(CONFIG_PIC32_EVIC) += irq-pic32-evic.o
+obj-$(CONFIG_MVEBU_GICP) += irq-mvebu-gicp.o
+obj-$(CONFIG_MVEBU_ICU) += irq-mvebu-icu.o
obj-$(CONFIG_MVEBU_ODMI) += irq-mvebu-odmi.o
obj-$(CONFIG_MVEBU_PIC) += irq-mvebu-pic.o
obj-$(CONFIG_LS_SCFG_MSI) += irq-ls-scfg-msi.o
obj-$(CONFIG_EZNPS_GIC) += irq-eznps.o
-obj-$(CONFIG_ARCH_ASPEED) += irq-aspeed-vic.o
+obj-$(CONFIG_ARCH_ASPEED) += irq-aspeed-vic.o irq-aspeed-i2c-ic.o
obj-$(CONFIG_STM32_EXTI) += irq-stm32-exti.o
obj-$(CONFIG_QCOM_IRQ_COMBINER) += qcom-irq-combiner.o
#include <asm/smp_plat.h>
#include <asm/mach/irq.h>
-/* Interrupt Controller Registers Map */
-#define ARMADA_370_XP_INT_SET_MASK_OFFS (0x48)
-#define ARMADA_370_XP_INT_CLEAR_MASK_OFFS (0x4C)
-#define ARMADA_370_XP_INT_FABRIC_MASK_OFFS (0x54)
-#define ARMADA_370_XP_INT_CAUSE_PERF(cpu) (1 << cpu)
+/*
+ * Overall diagram of the Armada XP interrupt controller:
+ *
+ * To CPU 0 To CPU 1
+ *
+ * /\ /\
+ * || ||
+ * +---------------+ +---------------+
+ * | | | |
+ * | per-CPU | | per-CPU |
+ * | mask/unmask | | mask/unmask |
+ * | CPU0 | | CPU1 |
+ * | | | |
+ * +---------------+ +---------------+
+ * /\ /\
+ * || ||
+ * \\_______________________//
+ * ||
+ * +-------------------+
+ * | |
+ * | Global interrupt |
+ * | mask/unmask |
+ * | |
+ * +-------------------+
+ * /\
+ * ||
+ * interrupt from
+ * device
+ *
+ * The "global interrupt mask/unmask" is modified using the
+ * ARMADA_370_XP_INT_SET_ENABLE_OFFS and
+ * ARMADA_370_XP_INT_CLEAR_ENABLE_OFFS registers, which are relative
+ * to "main_int_base".
+ *
+ * The "per-CPU mask/unmask" is modified using the
+ * ARMADA_370_XP_INT_SET_MASK_OFFS and
+ * ARMADA_370_XP_INT_CLEAR_MASK_OFFS registers, which are relative to
+ * "per_cpu_int_base". This base address points to a special address,
+ * which automatically accesses the registers of the current CPU.
+ *
+ * The per-CPU mask/unmask can also be adjusted using the global
+ * per-interrupt ARMADA_370_XP_INT_SOURCE_CTL register, which we use
+ * to configure interrupt affinity.
+ *
+ * Due to this model, all interrupts need to be mask/unmasked at two
+ * different levels: at the global level and at the per-CPU level.
+ *
+ * This driver takes the following approach to deal with this:
+ *
+ * - For global interrupts:
+ *
+ * At ->map() time, a global interrupt is unmasked at the per-CPU
+ * mask/unmask level. It is therefore unmasked at this level for
+ * the current CPU, running the ->map() code. This allows to have
+ * the interrupt unmasked at this level in non-SMP
+ * configurations. In SMP configurations, the ->set_affinity()
+ * callback is called, which using the
+ * ARMADA_370_XP_INT_SOURCE_CTL() readjusts the per-CPU mask/unmask
+ * for the interrupt.
+ *
+ * The ->mask() and ->unmask() operations only mask/unmask the
+ * interrupt at the "global" level.
+ *
+ * So, a global interrupt is enabled at the per-CPU level as soon
+ * as it is mapped. At run time, the masking/unmasking takes place
+ * at the global level.
+ *
+ * - For per-CPU interrupts
+ *
+ * At ->map() time, a per-CPU interrupt is unmasked at the global
+ * mask/unmask level.
+ *
+ * The ->mask() and ->unmask() operations mask/unmask the interrupt
+ * at the per-CPU level.
+ *
+ * So, a per-CPU interrupt is enabled at the global level as soon
+ * as it is mapped. At run time, the masking/unmasking takes place
+ * at the per-CPU level.
+ */
+/* Registers relative to main_int_base */
#define ARMADA_370_XP_INT_CONTROL (0x00)
+#define ARMADA_370_XP_SW_TRIG_INT_OFFS (0x04)
#define ARMADA_370_XP_INT_SET_ENABLE_OFFS (0x30)
#define ARMADA_370_XP_INT_CLEAR_ENABLE_OFFS (0x34)
#define ARMADA_370_XP_INT_SOURCE_CTL(irq) (0x100 + irq*4)
#define ARMADA_370_XP_INT_SOURCE_CPU_MASK 0xF
#define ARMADA_370_XP_INT_IRQ_FIQ_MASK(cpuid) ((BIT(0) | BIT(8)) << cpuid)
-#define ARMADA_370_XP_CPU_INTACK_OFFS (0x44)
+/* Registers relative to per_cpu_int_base */
+#define ARMADA_370_XP_IN_DRBEL_CAUSE_OFFS (0x08)
+#define ARMADA_370_XP_IN_DRBEL_MSK_OFFS (0x0c)
#define ARMADA_375_PPI_CAUSE (0x10)
-
-#define ARMADA_370_XP_SW_TRIG_INT_OFFS (0x4)
-#define ARMADA_370_XP_IN_DRBEL_MSK_OFFS (0xc)
-#define ARMADA_370_XP_IN_DRBEL_CAUSE_OFFS (0x8)
+#define ARMADA_370_XP_CPU_INTACK_OFFS (0x44)
+#define ARMADA_370_XP_INT_SET_MASK_OFFS (0x48)
+#define ARMADA_370_XP_INT_CLEAR_MASK_OFFS (0x4C)
+#define ARMADA_370_XP_INT_FABRIC_MASK_OFFS (0x54)
+#define ARMADA_370_XP_INT_CAUSE_PERF(cpu) (1 << cpu)
#define ARMADA_370_XP_MAX_PER_CPU_IRQS (28)
irq_set_percpu_devid(virq);
irq_set_chip_and_handler(virq, &armada_370_xp_irq_chip,
handle_percpu_devid_irq);
-
} else {
irq_set_chip_and_handler(virq, &armada_370_xp_irq_chip,
handle_level_irq);
}
irq_set_probe(virq);
- irq_clear_status_flags(virq, IRQ_NOAUTOEN);
return 0;
}
ARMADA_370_XP_SW_TRIG_INT_OFFS);
}
+static void armada_xp_mpic_reenable_percpu(void)
+{
+ unsigned int irq;
+
+ /* Re-enable per-CPU interrupts that were enabled before suspend */
+ for (irq = 0; irq < ARMADA_370_XP_MAX_PER_CPU_IRQS; irq++) {
+ struct irq_data *data;
+ int virq;
+
+ virq = irq_linear_revmap(armada_370_xp_mpic_domain, irq);
+ if (virq == 0)
+ continue;
+
+ data = irq_get_irq_data(virq);
+
+ if (!irq_percpu_is_enabled(virq))
+ continue;
+
+ armada_370_xp_irq_unmask(data);
+ }
+}
+
static int armada_xp_mpic_starting_cpu(unsigned int cpu)
{
armada_xp_mpic_perf_init();
armada_xp_mpic_smp_cpu_init();
+ armada_xp_mpic_reenable_percpu();
return 0;
}
static int mpic_cascaded_starting_cpu(unsigned int cpu)
{
armada_xp_mpic_perf_init();
+ armada_xp_mpic_reenable_percpu();
enable_percpu_irq(parent_irq, IRQ_TYPE_NONE);
return 0;
}
if (virq == 0)
continue;
- if (!is_percpu_irq(irq))
+ data = irq_get_irq_data(virq);
+
+ if (!is_percpu_irq(irq)) {
+ /* Non per-CPU interrupts */
writel(irq, per_cpu_int_base +
ARMADA_370_XP_INT_CLEAR_MASK_OFFS);
- else
+ if (!irqd_irq_disabled(data))
+ armada_370_xp_irq_unmask(data);
+ } else {
+ /* Per-CPU interrupts */
writel(irq, main_int_base +
ARMADA_370_XP_INT_SET_ENABLE_OFFS);
- data = irq_get_irq_data(virq);
- if (!irqd_irq_disabled(data))
- armada_370_xp_irq_unmask(data);
+ /*
+ * Re-enable on the current CPU,
+ * armada_xp_mpic_reenable_percpu() will take
+ * care of secondary CPUs when they come up.
+ */
+ if (irq_percpu_is_enabled(virq))
+ armada_370_xp_irq_unmask(data);
+ }
}
/* Reconfigure doorbells for IPIs and MSIs */
irq_domain_add_linear(node, nr_irqs,
&armada_370_xp_mpic_irq_ops, NULL);
BUG_ON(!armada_370_xp_mpic_domain);
- armada_370_xp_mpic_domain->bus_token = DOMAIN_BUS_WIRED;
+ irq_domain_update_bus_token(armada_370_xp_mpic_domain, DOMAIN_BUS_WIRED);
/* Setup for the boot CPU */
armada_xp_mpic_perf_init();
--- /dev/null
+/*
+ * Aspeed 24XX/25XX I2C Interrupt Controller.
+ *
+ * Copyright (C) 2012-2017 ASPEED Technology Inc.
+ * Copyright 2017 IBM Corporation
+ * Copyright 2017 Google, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/irq.h>
+#include <linux/irqchip.h>
+#include <linux/irqchip/chained_irq.h>
+#include <linux/irqdomain.h>
+#include <linux/of_address.h>
+#include <linux/of_irq.h>
+#include <linux/io.h>
+
+
+#define ASPEED_I2C_IC_NUM_BUS 14
+
+struct aspeed_i2c_ic {
+ void __iomem *base;
+ int parent_irq;
+ struct irq_domain *irq_domain;
+};
+
+/*
+ * The aspeed chip provides a single hardware interrupt for all of the I2C
+ * busses, so we use a dummy interrupt chip to translate this single interrupt
+ * into multiple interrupts, each associated with a single I2C bus.
+ */
+static void aspeed_i2c_ic_irq_handler(struct irq_desc *desc)
+{
+ struct aspeed_i2c_ic *i2c_ic = irq_desc_get_handler_data(desc);
+ struct irq_chip *chip = irq_desc_get_chip(desc);
+ unsigned long bit, status;
+ unsigned int bus_irq;
+
+ chained_irq_enter(chip, desc);
+ status = readl(i2c_ic->base);
+ for_each_set_bit(bit, &status, ASPEED_I2C_IC_NUM_BUS) {
+ bus_irq = irq_find_mapping(i2c_ic->irq_domain, bit);
+ generic_handle_irq(bus_irq);
+ }
+ chained_irq_exit(chip, desc);
+}
+
+/*
+ * Set simple handler and mark IRQ as valid. Nothing interesting to do here
+ * since we are using a dummy interrupt chip.
+ */
+static int aspeed_i2c_ic_map_irq_domain(struct irq_domain *domain,
+ unsigned int irq, irq_hw_number_t hwirq)
+{
+ irq_set_chip_and_handler(irq, &dummy_irq_chip, handle_simple_irq);
+ irq_set_chip_data(irq, domain->host_data);
+
+ return 0;
+}
+
+static const struct irq_domain_ops aspeed_i2c_ic_irq_domain_ops = {
+ .map = aspeed_i2c_ic_map_irq_domain,
+};
+
+static int __init aspeed_i2c_ic_of_init(struct device_node *node,
+ struct device_node *parent)
+{
+ struct aspeed_i2c_ic *i2c_ic;
+ int ret = 0;
+
+ i2c_ic = kzalloc(sizeof(*i2c_ic), GFP_KERNEL);
+ if (!i2c_ic)
+ return -ENOMEM;
+
+ i2c_ic->base = of_iomap(node, 0);
+ if (IS_ERR(i2c_ic->base)) {
+ ret = PTR_ERR(i2c_ic->base);
+ goto err_free_ic;
+ }
+
+ i2c_ic->parent_irq = irq_of_parse_and_map(node, 0);
+ if (i2c_ic->parent_irq < 0) {
+ ret = i2c_ic->parent_irq;
+ goto err_iounmap;
+ }
+
+ i2c_ic->irq_domain = irq_domain_add_linear(node, ASPEED_I2C_IC_NUM_BUS,
+ &aspeed_i2c_ic_irq_domain_ops,
+ NULL);
+ if (!i2c_ic->irq_domain) {
+ ret = -ENOMEM;
+ goto err_iounmap;
+ }
+
+ i2c_ic->irq_domain->name = "aspeed-i2c-domain";
+
+ irq_set_chained_handler_and_data(i2c_ic->parent_irq,
+ aspeed_i2c_ic_irq_handler, i2c_ic);
+
+ pr_info("i2c controller registered, irq %d\n", i2c_ic->parent_irq);
+
+ return 0;
+
+err_iounmap:
+ iounmap(i2c_ic->base);
+err_free_ic:
+ kfree(i2c_ic);
+ return ret;
+}
+
+IRQCHIP_DECLARE(ast2400_i2c_ic, "aspeed,ast2400-i2c-ic", aspeed_i2c_ic_of_init);
+IRQCHIP_DECLARE(ast2500_i2c_ic, "aspeed,ast2500-i2c-ic", aspeed_i2c_ic_of_init);
return 0;
}
-static struct irq_domain_ops avic_dom_ops = {
+static const struct irq_domain_ops avic_dom_ops = {
.map = avic_map,
.xlate = irq_domain_xlate_onetwocell,
};
return 0;
}
-IRQCHIP_DECLARE(aspeed_new_vic, "aspeed,ast2400-vic", avic_of_init);
+IRQCHIP_DECLARE(ast2400_vic, "aspeed,ast2400-vic", avic_of_init);
+IRQCHIP_DECLARE(ast2500_vic, "aspeed,ast2500-vic", avic_of_init);
return -ENOMEM;
}
- inner_domain->bus_token = DOMAIN_BUS_NEXUS;
+ irq_domain_update_bus_token(inner_domain, DOMAIN_BUS_NEXUS);
inner_domain->parent = parent;
pci_domain = pci_msi_create_irq_domain(v2m->fwnode,
&gicv2m_msi_domain_info,
.irq_write_msi_msg = pci_msi_domain_write_msg,
};
-struct its_pci_alias {
- struct pci_dev *pdev;
- u32 count;
-};
-
-static int its_pci_msi_vec_count(struct pci_dev *pdev)
+static int its_pci_msi_vec_count(struct pci_dev *pdev, void *data)
{
- int msi, msix;
+ int msi, msix, *count = data;
msi = max(pci_msi_vec_count(pdev), 0);
msix = max(pci_msix_vec_count(pdev), 0);
+ *count += max(msi, msix);
- return max(msi, msix);
+ return 0;
}
static int its_get_pci_alias(struct pci_dev *pdev, u16 alias, void *data)
{
- struct its_pci_alias *dev_alias = data;
+ struct pci_dev **alias_dev = data;
- if (pdev != dev_alias->pdev)
- dev_alias->count += its_pci_msi_vec_count(pdev);
+ *alias_dev = pdev;
return 0;
}
static int its_pci_msi_prepare(struct irq_domain *domain, struct device *dev,
int nvec, msi_alloc_info_t *info)
{
- struct pci_dev *pdev;
- struct its_pci_alias dev_alias;
+ struct pci_dev *pdev, *alias_dev;
struct msi_domain_info *msi_info;
+ int alias_count = 0;
if (!dev_is_pci(dev))
return -EINVAL;
msi_info = msi_get_domain_info(domain->parent);
pdev = to_pci_dev(dev);
- dev_alias.pdev = pdev;
- dev_alias.count = nvec;
-
- pci_for_each_dma_alias(pdev, its_get_pci_alias, &dev_alias);
+ /*
+ * If pdev is downstream of any aliasing bridges, take an upper
+ * bound of how many other vectors could map to the same DevID.
+ */
+ pci_for_each_dma_alias(pdev, its_get_pci_alias, &alias_dev);
+ if (alias_dev != pdev && alias_dev->subordinate)
+ pci_walk_bus(alias_dev->subordinate, its_pci_msi_vec_count,
+ &alias_count);
/* ITS specific DeviceID, as the core ITS ignores dev. */
info->scratchpad[0].ul = pci_msi_domain_get_msi_rid(domain, pdev);
return msi_info->ops->msi_prepare(domain->parent,
- dev, dev_alias.count, info);
+ dev, max(nvec, alias_count), info);
}
static struct msi_domain_ops its_pci_msi_ops = {
.chip = &its_pmsi_irq_chip,
};
-static struct of_device_id its_device_id[] = {
+static const struct of_device_id its_device_id[] = {
{ .compatible = "arm,gic-v3-its", },
{},
};
if (cpu >= nr_cpu_ids)
return -EINVAL;
- target_col = &its_dev->its->collections[cpu];
- its_send_movi(its_dev, target_col, id);
- its_dev->event_map.col_map[id] = cpu;
+ /* don't set the affinity when the target cpu is same as current one */
+ if (cpu != its_dev->event_map.col_map[id]) {
+ target_col = &its_dev->its->collections[cpu];
+ its_send_movi(its_dev, target_col, id);
+ its_dev->event_map.col_map[id] = cpu;
+ }
return IRQ_SET_MASK_OK_DONE;
}
*/
#define IRQS_PER_CHUNK_SHIFT 5
#define IRQS_PER_CHUNK (1 << IRQS_PER_CHUNK_SHIFT)
+#define ITS_MAX_LPI_NRBITS 16 /* 64K LPIs */
static unsigned long *lpi_bitmap;
static u32 lpi_chunks;
+static u32 lpi_id_bits;
static DEFINE_SPINLOCK(lpi_lock);
static int its_lpi_to_chunk(int lpi)
}
/*
- * We allocate 64kB for PROPBASE. That gives us at most 64K LPIs to
+ * We allocate memory for PROPBASE to cover 2 ^ lpi_id_bits LPIs to
* deal with (one configuration byte per interrupt). PENDBASE has to
* be 64kB aligned (one bit per LPI, plus 8192 bits for SPI/PPI/SGI).
*/
-#define LPI_PROPBASE_SZ SZ_64K
-#define LPI_PENDBASE_SZ (LPI_PROPBASE_SZ / 8 + SZ_1K)
-
-/*
- * This is how many bits of ID we need, including the useless ones.
- */
-#define LPI_NRBITS ilog2(LPI_PROPBASE_SZ + SZ_8K)
+#define LPI_NRBITS lpi_id_bits
+#define LPI_PROPBASE_SZ ALIGN(BIT(LPI_NRBITS), SZ_64K)
+#define LPI_PENDBASE_SZ ALIGN(BIT(LPI_NRBITS) / 8, SZ_64K)
#define LPI_PROP_DEFAULT_PRIO 0xa0
{
phys_addr_t paddr;
+ lpi_id_bits = min_t(u32, gic_rdists->id_bits, ITS_MAX_LPI_NRBITS);
gic_rdists->prop_page = alloc_pages(GFP_NOWAIT,
get_order(LPI_PROPBASE_SZ));
if (!gic_rdists->prop_page) {
/* Make sure the GIC will observe the written configuration */
gic_flush_dcache_to_poc(page_address(gic_rdists->prop_page), LPI_PROPBASE_SZ);
- return 0;
+ return its_lpi_init(lpi_id_bits);
}
static const char *its_base_type_string[] = {
* hence the 'max(LPI_PENDBASE_SZ, SZ_64K)' below.
*/
pend_page = alloc_pages(GFP_NOWAIT | __GFP_ZERO,
- get_order(max(LPI_PENDBASE_SZ, SZ_64K)));
+ get_order(max_t(u32, LPI_PENDBASE_SZ, SZ_64K)));
if (!pend_page) {
pr_err("Failed to allocate PENDBASE for CPU%d\n",
smp_processor_id());
}
inner_domain->parent = its_parent;
- inner_domain->bus_token = DOMAIN_BUS_NEXUS;
+ irq_domain_update_bus_token(inner_domain, DOMAIN_BUS_NEXUS);
inner_domain->flags |= IRQ_DOMAIN_FLAG_MSI_REMAP;
info->ops = &its_msi_domain_ops;
info->data = its;
return 0;
}
-static struct of_device_id its_device_id[] = {
+static const struct of_device_id its_device_id[] = {
{ .compatible = "arm,gic-v3-its", },
{},
};
#define ACPI_GICV3_ITS_MEM_SIZE (SZ_128K)
+#if defined(CONFIG_ACPI_NUMA) && (ACPI_CA_VERSION >= 0x20170531)
+struct its_srat_map {
+ /* numa node id */
+ u32 numa_node;
+ /* GIC ITS ID */
+ u32 its_id;
+};
+
+static struct its_srat_map its_srat_maps[MAX_NUMNODES] __initdata;
+static int its_in_srat __initdata;
+
+static int __init acpi_get_its_numa_node(u32 its_id)
+{
+ int i;
+
+ for (i = 0; i < its_in_srat; i++) {
+ if (its_id == its_srat_maps[i].its_id)
+ return its_srat_maps[i].numa_node;
+ }
+ return NUMA_NO_NODE;
+}
+
+static int __init gic_acpi_parse_srat_its(struct acpi_subtable_header *header,
+ const unsigned long end)
+{
+ int node;
+ struct acpi_srat_gic_its_affinity *its_affinity;
+
+ its_affinity = (struct acpi_srat_gic_its_affinity *)header;
+ if (!its_affinity)
+ return -EINVAL;
+
+ if (its_affinity->header.length < sizeof(*its_affinity)) {
+ pr_err("SRAT: Invalid header length %d in ITS affinity\n",
+ its_affinity->header.length);
+ return -EINVAL;
+ }
+
+ if (its_in_srat >= MAX_NUMNODES) {
+ pr_err("SRAT: ITS affinity exceeding max count[%d]\n",
+ MAX_NUMNODES);
+ return -EINVAL;
+ }
+
+ node = acpi_map_pxm_to_node(its_affinity->proximity_domain);
+
+ if (node == NUMA_NO_NODE || node >= MAX_NUMNODES) {
+ pr_err("SRAT: Invalid NUMA node %d in ITS affinity\n", node);
+ return 0;
+ }
+
+ its_srat_maps[its_in_srat].numa_node = node;
+ its_srat_maps[its_in_srat].its_id = its_affinity->its_id;
+ its_in_srat++;
+ pr_info("SRAT: PXM %d -> ITS %d -> Node %d\n",
+ its_affinity->proximity_domain, its_affinity->its_id, node);
+
+ return 0;
+}
+
+static void __init acpi_table_parse_srat_its(void)
+{
+ acpi_table_parse_entries(ACPI_SIG_SRAT,
+ sizeof(struct acpi_table_srat),
+ ACPI_SRAT_TYPE_GIC_ITS_AFFINITY,
+ gic_acpi_parse_srat_its, 0);
+}
+#else
+static void __init acpi_table_parse_srat_its(void) { }
+static int __init acpi_get_its_numa_node(u32 its_id) { return NUMA_NO_NODE; }
+#endif
+
static int __init gic_acpi_parse_madt_its(struct acpi_subtable_header *header,
const unsigned long end)
{
goto dom_err;
}
- err = its_probe_one(&res, dom_handle, NUMA_NO_NODE);
+ err = its_probe_one(&res, dom_handle,
+ acpi_get_its_numa_node(its_entry->translation_id));
if (!err)
return 0;
static void __init its_acpi_probe(void)
{
+ acpi_table_parse_srat_its();
acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_TRANSLATOR,
gic_acpi_parse_madt_its, 0);
}
}
gic_rdists = rdists;
- its_alloc_lpi_tables();
- its_lpi_init(rdists->id_bits);
-
- return 0;
+ return its_alloc_lpi_tables();
}
int enabled;
u64 val;
+ if (cpu >= nr_cpu_ids)
+ return -EINVAL;
+
if (gic_irq_in_rdist(d))
return -EINVAL;
return 0;
}
-static struct irq_domain_ops i8259A_ops = {
+static const struct irq_domain_ops i8259A_ops = {
.map = i8259A_irq_domain_map,
.xlate = irq_domain_xlate_onecell,
};
&parent_fwspec);
}
-static struct irq_domain_ops gpcv2_irqchip_data_domain_ops = {
+static const struct irq_domain_ops gpcv2_irqchip_data_domain_ops = {
.translate = imx_gpcv2_domain_translate,
.alloc = imx_gpcv2_domain_alloc,
.free = irq_domain_free_irqs_common,
return 0;
}
-static struct irq_domain_ops mbigen_domain_ops = {
+static const struct irq_domain_ops mbigen_domain_ops = {
.translate = mbigen_domain_translate,
.alloc = mbigen_irq_domain_alloc,
.free = irq_domain_free_irqs_common,
ipi_domain_state);
if (!ipi_domain)
panic("Failed to add MIPS CPU IPI domain");
- ipi_domain->bus_token = DOMAIN_BUS_IPI;
+ irq_domain_update_bus_token(ipi_domain, DOMAIN_BUS_IPI);
}
#else /* !CONFIG_GENERIC_IRQ_IPI */
}
#ifdef CONFIG_CLKSRC_MIPS_GIC
-u64 gic_read_count(void)
+u64 notrace gic_read_count(void)
{
unsigned int hi, hi2, lo;
return bits;
}
-void gic_write_compare(u64 cnt)
+void notrace gic_write_compare(u64 cnt)
{
if (mips_cm_is64) {
gic_write(GIC_REG(VPE_LOCAL, GIC_VPE_COMPARE), cnt);
}
}
-void gic_write_cpu_compare(u64 cnt, int cpu)
+void notrace gic_write_cpu_compare(u64 cnt, int cpu)
{
unsigned long flags;
}
}
-static struct irq_domain_ops gic_ipi_domain_ops = {
+static const struct irq_domain_ops gic_ipi_domain_ops = {
.xlate = gic_ipi_domain_xlate,
.alloc = gic_ipi_domain_alloc,
.free = gic_ipi_domain_free,
panic("Failed to add GIC IPI domain");
gic_ipi_domain->name = "mips-gic-ipi";
- gic_ipi_domain->bus_token = DOMAIN_BUS_IPI;
+ irq_domain_update_bus_token(gic_ipi_domain, DOMAIN_BUS_IPI);
if (node &&
!of_property_read_u32_array(node, "mti,reserved-ipi-vectors", v, 2)) {
--- /dev/null
+/*
+ * Copyright (C) 2017 Marvell
+ *
+ * Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
+ *
+ * This file is licensed under the terms of the GNU General Public
+ * License version 2. This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#include <linux/io.h>
+#include <linux/irq.h>
+#include <linux/irqdomain.h>
+#include <linux/msi.h>
+#include <linux/of.h>
+#include <linux/of_irq.h>
+#include <linux/of_platform.h>
+#include <linux/platform_device.h>
+
+#include <dt-bindings/interrupt-controller/arm-gic.h>
+
+#include "irq-mvebu-gicp.h"
+
+#define GICP_SETSPI_NSR_OFFSET 0x0
+#define GICP_CLRSPI_NSR_OFFSET 0x8
+
+struct mvebu_gicp_spi_range {
+ unsigned int start;
+ unsigned int count;
+};
+
+struct mvebu_gicp {
+ struct mvebu_gicp_spi_range *spi_ranges;
+ unsigned int spi_ranges_cnt;
+ unsigned int spi_cnt;
+ unsigned long *spi_bitmap;
+ spinlock_t spi_lock;
+ struct resource *res;
+ struct device *dev;
+};
+
+static int gicp_idx_to_spi(struct mvebu_gicp *gicp, int idx)
+{
+ int i;
+
+ for (i = 0; i < gicp->spi_ranges_cnt; i++) {
+ struct mvebu_gicp_spi_range *r = &gicp->spi_ranges[i];
+
+ if (idx < r->count)
+ return r->start + idx;
+
+ idx -= r->count;
+ }
+
+ return -EINVAL;
+}
+
+int mvebu_gicp_get_doorbells(struct device_node *dn, phys_addr_t *setspi,
+ phys_addr_t *clrspi)
+{
+ struct platform_device *pdev;
+ struct mvebu_gicp *gicp;
+
+ pdev = of_find_device_by_node(dn);
+ if (!pdev)
+ return -ENODEV;
+
+ gicp = platform_get_drvdata(pdev);
+ if (!gicp)
+ return -ENODEV;
+
+ *setspi = gicp->res->start + GICP_SETSPI_NSR_OFFSET;
+ *clrspi = gicp->res->start + GICP_CLRSPI_NSR_OFFSET;
+
+ return 0;
+}
+
+static void gicp_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
+{
+ struct mvebu_gicp *gicp = data->chip_data;
+ phys_addr_t setspi = gicp->res->start + GICP_SETSPI_NSR_OFFSET;
+
+ msg->data = data->hwirq;
+ msg->address_lo = lower_32_bits(setspi);
+ msg->address_hi = upper_32_bits(setspi);
+}
+
+static struct irq_chip gicp_irq_chip = {
+ .name = "GICP",
+ .irq_mask = irq_chip_mask_parent,
+ .irq_unmask = irq_chip_unmask_parent,
+ .irq_eoi = irq_chip_eoi_parent,
+ .irq_set_affinity = irq_chip_set_affinity_parent,
+ .irq_set_type = irq_chip_set_type_parent,
+ .irq_compose_msi_msg = gicp_compose_msi_msg,
+};
+
+static int gicp_irq_domain_alloc(struct irq_domain *domain, unsigned int virq,
+ unsigned int nr_irqs, void *args)
+{
+ struct mvebu_gicp *gicp = domain->host_data;
+ struct irq_fwspec fwspec;
+ unsigned int hwirq;
+ int ret;
+
+ spin_lock(&gicp->spi_lock);
+ hwirq = find_first_zero_bit(gicp->spi_bitmap, gicp->spi_cnt);
+ if (hwirq == gicp->spi_cnt) {
+ spin_unlock(&gicp->spi_lock);
+ return -ENOSPC;
+ }
+ __set_bit(hwirq, gicp->spi_bitmap);
+ spin_unlock(&gicp->spi_lock);
+
+ fwspec.fwnode = domain->parent->fwnode;
+ fwspec.param_count = 3;
+ fwspec.param[0] = GIC_SPI;
+ fwspec.param[1] = gicp_idx_to_spi(gicp, hwirq) - 32;
+ /*
+ * Assume edge rising for now, it will be properly set when
+ * ->set_type() is called
+ */
+ fwspec.param[2] = IRQ_TYPE_EDGE_RISING;
+
+ ret = irq_domain_alloc_irqs_parent(domain, virq, 1, &fwspec);
+ if (ret) {
+ dev_err(gicp->dev, "Cannot allocate parent IRQ\n");
+ goto free_hwirq;
+ }
+
+ ret = irq_domain_set_hwirq_and_chip(domain, virq, hwirq,
+ &gicp_irq_chip, gicp);
+ if (ret)
+ goto free_irqs_parent;
+
+ return 0;
+
+free_irqs_parent:
+ irq_domain_free_irqs_parent(domain, virq, nr_irqs);
+free_hwirq:
+ spin_lock(&gicp->spi_lock);
+ __clear_bit(hwirq, gicp->spi_bitmap);
+ spin_unlock(&gicp->spi_lock);
+ return ret;
+}
+
+static void gicp_irq_domain_free(struct irq_domain *domain,
+ unsigned int virq, unsigned int nr_irqs)
+{
+ struct mvebu_gicp *gicp = domain->host_data;
+ struct irq_data *d = irq_domain_get_irq_data(domain, virq);
+
+ if (d->hwirq >= gicp->spi_cnt) {
+ dev_err(gicp->dev, "Invalid hwirq %lu\n", d->hwirq);
+ return;
+ }
+
+ irq_domain_free_irqs_parent(domain, virq, nr_irqs);
+
+ spin_lock(&gicp->spi_lock);
+ __clear_bit(d->hwirq, gicp->spi_bitmap);
+ spin_unlock(&gicp->spi_lock);
+}
+
+static const struct irq_domain_ops gicp_domain_ops = {
+ .alloc = gicp_irq_domain_alloc,
+ .free = gicp_irq_domain_free,
+};
+
+static struct irq_chip gicp_msi_irq_chip = {
+ .name = "GICP",
+ .irq_set_type = irq_chip_set_type_parent,
+};
+
+static struct msi_domain_ops gicp_msi_ops = {
+};
+
+static struct msi_domain_info gicp_msi_domain_info = {
+ .flags = (MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS),
+ .ops = &gicp_msi_ops,
+ .chip = &gicp_msi_irq_chip,
+};
+
+static int mvebu_gicp_probe(struct platform_device *pdev)
+{
+ struct mvebu_gicp *gicp;
+ struct irq_domain *inner_domain, *plat_domain, *parent_domain;
+ struct device_node *node = pdev->dev.of_node;
+ struct device_node *irq_parent_dn;
+ int ret, i;
+
+ gicp = devm_kzalloc(&pdev->dev, sizeof(*gicp), GFP_KERNEL);
+ if (!gicp)
+ return -ENOMEM;
+
+ gicp->dev = &pdev->dev;
+
+ gicp->res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!gicp->res)
+ return -ENODEV;
+
+ ret = of_property_count_u32_elems(node, "marvell,spi-ranges");
+ if (ret < 0)
+ return ret;
+
+ gicp->spi_ranges_cnt = ret / 2;
+
+ gicp->spi_ranges =
+ devm_kzalloc(&pdev->dev,
+ gicp->spi_ranges_cnt *
+ sizeof(struct mvebu_gicp_spi_range),
+ GFP_KERNEL);
+ if (!gicp->spi_ranges)
+ return -ENOMEM;
+
+ for (i = 0; i < gicp->spi_ranges_cnt; i++) {
+ of_property_read_u32_index(node, "marvell,spi-ranges",
+ i * 2,
+ &gicp->spi_ranges[i].start);
+
+ of_property_read_u32_index(node, "marvell,spi-ranges",
+ i * 2 + 1,
+ &gicp->spi_ranges[i].count);
+
+ gicp->spi_cnt += gicp->spi_ranges[i].count;
+ }
+
+ gicp->spi_bitmap = devm_kzalloc(&pdev->dev,
+ BITS_TO_LONGS(gicp->spi_cnt) * sizeof(long),
+ GFP_KERNEL);
+ if (!gicp->spi_bitmap)
+ return -ENOMEM;
+
+ irq_parent_dn = of_irq_find_parent(node);
+ if (!irq_parent_dn) {
+ dev_err(&pdev->dev, "failed to find parent IRQ node\n");
+ return -ENODEV;
+ }
+
+ parent_domain = irq_find_host(irq_parent_dn);
+ if (!parent_domain) {
+ dev_err(&pdev->dev, "failed to find parent IRQ domain\n");
+ return -ENODEV;
+ }
+
+ inner_domain = irq_domain_create_hierarchy(parent_domain, 0,
+ gicp->spi_cnt,
+ of_node_to_fwnode(node),
+ &gicp_domain_ops, gicp);
+ if (!inner_domain)
+ return -ENOMEM;
+
+
+ plat_domain = platform_msi_create_irq_domain(of_node_to_fwnode(node),
+ &gicp_msi_domain_info,
+ inner_domain);
+ if (!plat_domain) {
+ irq_domain_remove(inner_domain);
+ return -ENOMEM;
+ }
+
+ platform_set_drvdata(pdev, gicp);
+
+ return 0;
+}
+
+static const struct of_device_id mvebu_gicp_of_match[] = {
+ { .compatible = "marvell,ap806-gicp", },
+ {},
+};
+
+static struct platform_driver mvebu_gicp_driver = {
+ .probe = mvebu_gicp_probe,
+ .driver = {
+ .name = "mvebu-gicp",
+ .of_match_table = mvebu_gicp_of_match,
+ },
+};
+builtin_platform_driver(mvebu_gicp_driver);
--- /dev/null
+#ifndef __MVEBU_GICP_H__
+#define __MVEBU_GICP_H__
+
+#include <linux/types.h>
+
+struct device_node;
+
+int mvebu_gicp_get_doorbells(struct device_node *dn, phys_addr_t *setspi,
+ phys_addr_t *clrspi);
+
+#endif /* __MVEBU_GICP_H__ */
--- /dev/null
+/*
+ * Copyright (C) 2017 Marvell
+ *
+ * Hanna Hawa <hannah@marvell.com>
+ * Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
+ *
+ * This file is licensed under the terms of the GNU General Public
+ * License version 2. This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#include <linux/interrupt.h>
+#include <linux/irq.h>
+#include <linux/irqchip.h>
+#include <linux/irqdomain.h>
+#include <linux/kernel.h>
+#include <linux/msi.h>
+#include <linux/of_irq.h>
+#include <linux/of_platform.h>
+#include <linux/platform_device.h>
+
+#include <dt-bindings/interrupt-controller/mvebu-icu.h>
+
+#include "irq-mvebu-gicp.h"
+
+/* ICU registers */
+#define ICU_SETSPI_NSR_AL 0x10
+#define ICU_SETSPI_NSR_AH 0x14
+#define ICU_CLRSPI_NSR_AL 0x18
+#define ICU_CLRSPI_NSR_AH 0x1c
+#define ICU_INT_CFG(x) (0x100 + 4 * (x))
+#define ICU_INT_ENABLE BIT(24)
+#define ICU_IS_EDGE BIT(28)
+#define ICU_GROUP_SHIFT 29
+
+/* ICU definitions */
+#define ICU_MAX_IRQS 207
+#define ICU_SATA0_ICU_ID 109
+#define ICU_SATA1_ICU_ID 107
+
+struct mvebu_icu {
+ struct irq_chip irq_chip;
+ void __iomem *base;
+ struct irq_domain *domain;
+ struct device *dev;
+};
+
+struct mvebu_icu_irq_data {
+ struct mvebu_icu *icu;
+ unsigned int icu_group;
+ unsigned int type;
+};
+
+static void mvebu_icu_write_msg(struct msi_desc *desc, struct msi_msg *msg)
+{
+ struct irq_data *d = irq_get_irq_data(desc->irq);
+ struct mvebu_icu_irq_data *icu_irqd = d->chip_data;
+ struct mvebu_icu *icu = icu_irqd->icu;
+ unsigned int icu_int;
+
+ if (msg->address_lo || msg->address_hi) {
+ /* Configure the ICU with irq number & type */
+ icu_int = msg->data | ICU_INT_ENABLE;
+ if (icu_irqd->type & IRQ_TYPE_EDGE_RISING)
+ icu_int |= ICU_IS_EDGE;
+ icu_int |= icu_irqd->icu_group << ICU_GROUP_SHIFT;
+ } else {
+ /* De-configure the ICU */
+ icu_int = 0;
+ }
+
+ writel_relaxed(icu_int, icu->base + ICU_INT_CFG(d->hwirq));
+
+ /*
+ * The SATA unit has 2 ports, and a dedicated ICU entry per
+ * port. The ahci sata driver supports only one irq interrupt
+ * per SATA unit. To solve this conflict, we configure the 2
+ * SATA wired interrupts in the south bridge into 1 GIC
+ * interrupt in the north bridge. Even if only a single port
+ * is enabled, if sata node is enabled, both interrupts are
+ * configured (regardless of which port is actually in use).
+ */
+ if (d->hwirq == ICU_SATA0_ICU_ID || d->hwirq == ICU_SATA1_ICU_ID) {
+ writel_relaxed(icu_int,
+ icu->base + ICU_INT_CFG(ICU_SATA0_ICU_ID));
+ writel_relaxed(icu_int,
+ icu->base + ICU_INT_CFG(ICU_SATA1_ICU_ID));
+ }
+}
+
+static int
+mvebu_icu_irq_domain_translate(struct irq_domain *d, struct irq_fwspec *fwspec,
+ unsigned long *hwirq, unsigned int *type)
+{
+ struct mvebu_icu *icu = d->host_data;
+ unsigned int icu_group;
+
+ /* Check the count of the parameters in dt */
+ if (WARN_ON(fwspec->param_count < 3)) {
+ dev_err(icu->dev, "wrong ICU parameter count %d\n",
+ fwspec->param_count);
+ return -EINVAL;
+ }
+
+ /* Only ICU group type is handled */
+ icu_group = fwspec->param[0];
+ if (icu_group != ICU_GRP_NSR && icu_group != ICU_GRP_SR &&
+ icu_group != ICU_GRP_SEI && icu_group != ICU_GRP_REI) {
+ dev_err(icu->dev, "wrong ICU group type %x\n", icu_group);
+ return -EINVAL;
+ }
+
+ *hwirq = fwspec->param[1];
+ if (*hwirq >= ICU_MAX_IRQS) {
+ dev_err(icu->dev, "invalid interrupt number %ld\n", *hwirq);
+ return -EINVAL;
+ }
+
+ /* Mask the type to prevent wrong DT configuration */
+ *type = fwspec->param[2] & IRQ_TYPE_SENSE_MASK;
+
+ return 0;
+}
+
+static int
+mvebu_icu_irq_domain_alloc(struct irq_domain *domain, unsigned int virq,
+ unsigned int nr_irqs, void *args)
+{
+ int err;
+ unsigned long hwirq;
+ struct irq_fwspec *fwspec = args;
+ struct mvebu_icu *icu = platform_msi_get_host_data(domain);
+ struct mvebu_icu_irq_data *icu_irqd;
+
+ icu_irqd = kmalloc(sizeof(*icu_irqd), GFP_KERNEL);
+ if (!icu_irqd)
+ return -ENOMEM;
+
+ err = mvebu_icu_irq_domain_translate(domain, fwspec, &hwirq,
+ &icu_irqd->type);
+ if (err) {
+ dev_err(icu->dev, "failed to translate ICU parameters\n");
+ goto free_irqd;
+ }
+
+ icu_irqd->icu_group = fwspec->param[0];
+ icu_irqd->icu = icu;
+
+ err = platform_msi_domain_alloc(domain, virq, nr_irqs);
+ if (err) {
+ dev_err(icu->dev, "failed to allocate ICU interrupt in parent domain\n");
+ goto free_irqd;
+ }
+
+ /* Make sure there is no interrupt left pending by the firmware */
+ err = irq_set_irqchip_state(virq, IRQCHIP_STATE_PENDING, false);
+ if (err)
+ goto free_msi;
+
+ err = irq_domain_set_hwirq_and_chip(domain, virq, hwirq,
+ &icu->irq_chip, icu_irqd);
+ if (err) {
+ dev_err(icu->dev, "failed to set the data to IRQ domain\n");
+ goto free_msi;
+ }
+
+ return 0;
+
+free_msi:
+ platform_msi_domain_free(domain, virq, nr_irqs);
+free_irqd:
+ kfree(icu_irqd);
+ return err;
+}
+
+static void
+mvebu_icu_irq_domain_free(struct irq_domain *domain, unsigned int virq,
+ unsigned int nr_irqs)
+{
+ struct irq_data *d = irq_get_irq_data(virq);
+ struct mvebu_icu_irq_data *icu_irqd = d->chip_data;
+
+ kfree(icu_irqd);
+
+ platform_msi_domain_free(domain, virq, nr_irqs);
+}
+
+static const struct irq_domain_ops mvebu_icu_domain_ops = {
+ .translate = mvebu_icu_irq_domain_translate,
+ .alloc = mvebu_icu_irq_domain_alloc,
+ .free = mvebu_icu_irq_domain_free,
+};
+
+static int mvebu_icu_probe(struct platform_device *pdev)
+{
+ struct mvebu_icu *icu;
+ struct device_node *node = pdev->dev.of_node;
+ struct device_node *gicp_dn;
+ struct resource *res;
+ phys_addr_t setspi, clrspi;
+ u32 i, icu_int;
+ int ret;
+
+ icu = devm_kzalloc(&pdev->dev, sizeof(struct mvebu_icu),
+ GFP_KERNEL);
+ if (!icu)
+ return -ENOMEM;
+
+ icu->dev = &pdev->dev;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ icu->base = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(icu->base)) {
+ dev_err(&pdev->dev, "Failed to map icu base address.\n");
+ return PTR_ERR(icu->base);
+ }
+
+ icu->irq_chip.name = devm_kasprintf(&pdev->dev, GFP_KERNEL,
+ "ICU.%x",
+ (unsigned int)res->start);
+ if (!icu->irq_chip.name)
+ return -ENOMEM;
+
+ icu->irq_chip.irq_mask = irq_chip_mask_parent;
+ icu->irq_chip.irq_unmask = irq_chip_unmask_parent;
+ icu->irq_chip.irq_eoi = irq_chip_eoi_parent;
+ icu->irq_chip.irq_set_type = irq_chip_set_type_parent;
+#ifdef CONFIG_SMP
+ icu->irq_chip.irq_set_affinity = irq_chip_set_affinity_parent;
+#endif
+
+ /*
+ * We're probed after MSI domains have been resolved, so force
+ * resolution here.
+ */
+ pdev->dev.msi_domain = of_msi_get_domain(&pdev->dev, node,
+ DOMAIN_BUS_PLATFORM_MSI);
+ if (!pdev->dev.msi_domain)
+ return -EPROBE_DEFER;
+
+ gicp_dn = irq_domain_get_of_node(pdev->dev.msi_domain);
+ if (!gicp_dn)
+ return -ENODEV;
+
+ ret = mvebu_gicp_get_doorbells(gicp_dn, &setspi, &clrspi);
+ if (ret)
+ return ret;
+
+ /* Set Clear/Set ICU SPI message address in AP */
+ writel_relaxed(upper_32_bits(setspi), icu->base + ICU_SETSPI_NSR_AH);
+ writel_relaxed(lower_32_bits(setspi), icu->base + ICU_SETSPI_NSR_AL);
+ writel_relaxed(upper_32_bits(clrspi), icu->base + ICU_CLRSPI_NSR_AH);
+ writel_relaxed(lower_32_bits(clrspi), icu->base + ICU_CLRSPI_NSR_AL);
+
+ /*
+ * Clean all ICU interrupts with type SPI_NSR, required to
+ * avoid unpredictable SPI assignments done by firmware.
+ */
+ for (i = 0 ; i < ICU_MAX_IRQS ; i++) {
+ icu_int = readl(icu->base + ICU_INT_CFG(i));
+ if ((icu_int >> ICU_GROUP_SHIFT) == ICU_GRP_NSR)
+ writel_relaxed(0x0, icu->base + ICU_INT_CFG(i));
+ }
+
+ icu->domain =
+ platform_msi_create_device_domain(&pdev->dev, ICU_MAX_IRQS,
+ mvebu_icu_write_msg,
+ &mvebu_icu_domain_ops, icu);
+ if (!icu->domain) {
+ dev_err(&pdev->dev, "Failed to create ICU domain\n");
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+static const struct of_device_id mvebu_icu_of_match[] = {
+ { .compatible = "marvell,cp110-icu", },
+ {},
+};
+
+static struct platform_driver mvebu_icu_driver = {
+ .probe = mvebu_icu_probe,
+ .driver = {
+ .name = "mvebu-icu",
+ .of_match_table = mvebu_icu_of_match,
+ },
+};
+builtin_platform_driver(mvebu_icu_driver);
.name = "or1k-PIC-level",
.irq_unmask = or1k_pic_unmask,
.irq_mask = or1k_pic_mask,
- .irq_mask_ack = or1k_pic_mask,
+ .irq_mask_ack = or1k_pic_mask_ack,
},
.handle = handle_level_irq,
.flags = IRQ_LEVEL | IRQ_NOPROBE,
return 0;
}
-static struct irq_domain_ops irq_ops = {
+static const struct irq_domain_ops irq_ops = {
.map = irq_map,
.xlate = irq_domain_xlate_onecell,
};
return 0;
}
-static struct irq_domain_ops irq_ops = {
+static const struct irq_domain_ops irq_ops = {
.map = irq_map,
.xlate = irq_domain_xlate_onecell,
};
#define SUNXI_NMI_SRC_TYPE_MASK 0x00000003
+#define SUNXI_NMI_IRQ_BIT BIT(0)
+
+#define SUN6I_R_INTC_CTRL 0x0c
+#define SUN6I_R_INTC_PENDING 0x10
+#define SUN6I_R_INTC_ENABLE 0x40
+
+/*
+ * For deprecated sun6i-a31-sc-nmi compatible.
+ * Registers are offset by 0x0c.
+ */
+#define SUN6I_R_INTC_NMI_OFFSET 0x0c
+#define SUN6I_NMI_CTRL (SUN6I_R_INTC_CTRL - SUN6I_R_INTC_NMI_OFFSET)
+#define SUN6I_NMI_PENDING (SUN6I_R_INTC_PENDING - SUN6I_R_INTC_NMI_OFFSET)
+#define SUN6I_NMI_ENABLE (SUN6I_R_INTC_ENABLE - SUN6I_R_INTC_NMI_OFFSET)
+
+#define SUN7I_NMI_CTRL 0x00
+#define SUN7I_NMI_PENDING 0x04
+#define SUN7I_NMI_ENABLE 0x08
+
+#define SUN9I_NMI_CTRL 0x00
+#define SUN9I_NMI_ENABLE 0x04
+#define SUN9I_NMI_PENDING 0x08
+
enum {
SUNXI_SRC_TYPE_LEVEL_LOW = 0,
SUNXI_SRC_TYPE_EDGE_FALLING,
u32 enable;
};
-static struct sunxi_sc_nmi_reg_offs sun7i_reg_offs = {
- .ctrl = 0x00,
- .pend = 0x04,
- .enable = 0x08,
+static const struct sunxi_sc_nmi_reg_offs sun6i_r_intc_reg_offs __initconst = {
+ .ctrl = SUN6I_R_INTC_CTRL,
+ .pend = SUN6I_R_INTC_PENDING,
+ .enable = SUN6I_R_INTC_ENABLE,
};
-static struct sunxi_sc_nmi_reg_offs sun6i_reg_offs = {
- .ctrl = 0x00,
- .pend = 0x04,
- .enable = 0x34,
+static const struct sunxi_sc_nmi_reg_offs sun6i_reg_offs __initconst = {
+ .ctrl = SUN6I_NMI_CTRL,
+ .pend = SUN6I_NMI_PENDING,
+ .enable = SUN6I_NMI_ENABLE,
};
-static struct sunxi_sc_nmi_reg_offs sun9i_reg_offs = {
- .ctrl = 0x00,
- .pend = 0x08,
- .enable = 0x04,
+static const struct sunxi_sc_nmi_reg_offs sun7i_reg_offs __initconst = {
+ .ctrl = SUN7I_NMI_CTRL,
+ .pend = SUN7I_NMI_PENDING,
+ .enable = SUN7I_NMI_ENABLE,
+};
+
+static const struct sunxi_sc_nmi_reg_offs sun9i_reg_offs __initconst = {
+ .ctrl = SUN9I_NMI_CTRL,
+ .pend = SUN9I_NMI_PENDING,
+ .enable = SUN9I_NMI_ENABLE,
};
static inline void sunxi_sc_nmi_write(struct irq_chip_generic *gc, u32 off,
}
static int __init sunxi_sc_nmi_irq_init(struct device_node *node,
- struct sunxi_sc_nmi_reg_offs *reg_offs)
+ const struct sunxi_sc_nmi_reg_offs *reg_offs)
{
struct irq_domain *domain;
struct irq_chip_generic *gc;
gc->chip_types[1].regs.type = reg_offs->ctrl;
gc->chip_types[1].handler = handle_edge_irq;
+ /* Disable any active interrupts */
sunxi_sc_nmi_write(gc, reg_offs->enable, 0);
- sunxi_sc_nmi_write(gc, reg_offs->pend, 0x1);
+
+ /* Clear any pending NMI interrupts */
+ sunxi_sc_nmi_write(gc, reg_offs->pend, SUNXI_NMI_IRQ_BIT);
irq_set_chained_handler_and_data(irq, sunxi_sc_nmi_handle_irq, domain);
return ret;
}
+static int __init sun6i_r_intc_irq_init(struct device_node *node,
+ struct device_node *parent)
+{
+ return sunxi_sc_nmi_irq_init(node, &sun6i_r_intc_reg_offs);
+}
+IRQCHIP_DECLARE(sun6i_r_intc, "allwinner,sun6i-a31-r-intc",
+ sun6i_r_intc_irq_init);
+
static int __init sun6i_sc_nmi_irq_init(struct device_node *node,
struct device_node *parent)
{
},
.probe = combiner_probe,
};
-
-static int __init register_qcom_irq_combiner(void)
-{
- return platform_driver_register(&qcom_irq_combiner_probe);
-}
-device_initcall(register_qcom_irq_combiner);
+builtin_platform_driver(qcom_irq_combiner_probe);
}
mutex_unlock(&dev->mlock);
- if (nvm_reserve_luns(dev, s->lun_begin, s->lun_end))
- return -ENOMEM;
+ ret = nvm_reserve_luns(dev, s->lun_begin, s->lun_end);
+ if (ret)
+ return ret;
t = kmalloc(sizeof(struct nvm_target), GFP_KERNEL);
if (!t) {
int nvm_submit_io(struct nvm_tgt_dev *tgt_dev, struct nvm_rq *rqd)
{
struct nvm_dev *dev = tgt_dev->parent;
+ int ret;
if (!dev->ops->submit_io)
return -ENODEV;
nvm_rq_tgt_to_dev(tgt_dev, rqd);
rqd->dev = tgt_dev;
- return dev->ops->submit_io(dev, rqd);
+
+ /* In case of error, fail with right address format */
+ ret = dev->ops->submit_io(dev, rqd);
+ if (ret)
+ nvm_rq_dev_to_tgt(tgt_dev, rqd);
+ return ret;
}
EXPORT_SYMBOL(nvm_submit_io);
*/
retry:
ret = pblk_rb_may_write_user(&pblk->rwb, bio, nr_entries, &bpos);
- if (ret == NVM_IO_REQUEUE) {
+ switch (ret) {
+ case NVM_IO_REQUEUE:
io_schedule();
goto retry;
+ case NVM_IO_ERR:
+ pblk_pipeline_stop(pblk);
+ goto out;
}
if (unlikely(!bio_has_data(bio)))
atomic_long_add(nr_entries, &pblk->req_writes);
#endif
+ pblk_rl_inserted(&pblk->rl, nr_entries);
+
out:
pblk_write_should_kick(pblk);
return ret;
*/
#include "pblk.h"
-#include <linux/time.h>
static void pblk_mark_bb(struct pblk *pblk, struct pblk_line *line,
struct ppa_addr *ppa)
pr_err("pblk: attempted to erase bb: line:%d, pos:%d\n",
line->id, pos);
- pblk_line_run_ws(pblk, NULL, ppa, pblk_line_mark_bb);
+ pblk_line_run_ws(pblk, NULL, ppa, pblk_line_mark_bb, pblk->bb_wq);
}
static void __pblk_end_io_erase(struct pblk *pblk, struct nvm_rq *rqd)
*ppa = rqd->ppa_addr;
pblk_mark_bb(pblk, line, ppa);
}
+
+ atomic_dec(&pblk->inflight_io);
}
/* Erase completion assumes that only one block is erased at the time */
{
struct pblk *pblk = rqd->private;
- up(&pblk->erase_sem);
__pblk_end_io_erase(pblk, rqd);
- mempool_free(rqd, pblk->r_rq_pool);
+ mempool_free(rqd, pblk->g_rq_pool);
}
-static void __pblk_map_invalidate(struct pblk *pblk, struct pblk_line *line,
- u64 paddr)
+void __pblk_map_invalidate(struct pblk *pblk, struct pblk_line *line,
+ u64 paddr)
{
struct pblk_line_mgmt *l_mg = &pblk->l_mg;
struct list_head *move_list = NULL;
spin_unlock(&line->lock);
return;
}
- line->vsc--;
+ le32_add_cpu(line->vsc, -1);
if (line->state == PBLK_LINESTATE_CLOSED)
move_list = pblk_line_gc_list(pblk, line);
__pblk_map_invalidate(pblk, line, paddr);
}
-void pblk_map_pad_invalidate(struct pblk *pblk, struct pblk_line *line,
- u64 paddr)
-{
- __pblk_map_invalidate(pblk, line, paddr);
-
- pblk_rb_sync_init(&pblk->rwb, NULL);
- line->left_ssecs--;
- if (!line->left_ssecs)
- pblk_line_run_ws(pblk, line, NULL, pblk_line_close_ws);
- pblk_rb_sync_end(&pblk->rwb, NULL);
-}
-
static void pblk_invalidate_range(struct pblk *pblk, sector_t slba,
unsigned int nr_secs)
{
pool = pblk->w_rq_pool;
rq_size = pblk_w_rq_size;
} else {
- pool = pblk->r_rq_pool;
- rq_size = pblk_r_rq_size;
+ pool = pblk->g_rq_pool;
+ rq_size = pblk_g_rq_size;
}
rqd = mempool_alloc(pool, GFP_KERNEL);
if (rw == WRITE)
pool = pblk->w_rq_pool;
else
- pool = pblk->r_rq_pool;
+ pool = pblk->g_rq_pool;
mempool_free(rqd, pool);
}
complete(waiting);
}
-void pblk_flush_writer(struct pblk *pblk)
+void pblk_wait_for_meta(struct pblk *pblk)
{
- struct bio *bio;
- int ret;
- DECLARE_COMPLETION_ONSTACK(wait);
-
- bio = bio_alloc(GFP_KERNEL, 1);
- if (!bio)
- return;
-
- bio->bi_iter.bi_sector = 0; /* internal bio */
- bio_set_op_attrs(bio, REQ_OP_WRITE, REQ_OP_FLUSH);
- bio->bi_private = &wait;
- bio->bi_end_io = pblk_end_bio_sync;
+ do {
+ if (!atomic_read(&pblk->inflight_io))
+ break;
- ret = pblk_write_to_cache(pblk, bio, 0);
- if (ret == NVM_IO_OK) {
- if (!wait_for_completion_io_timeout(&wait,
- msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
- pr_err("pblk: flush cache timed out\n");
- }
- } else if (ret != NVM_IO_DONE) {
- pr_err("pblk: tear down bio failed\n");
- }
+ schedule();
+ } while (1);
+}
- if (bio->bi_error)
- pr_err("pblk: flush sync write failed (%u)\n", bio->bi_error);
+static void pblk_flush_writer(struct pblk *pblk)
+{
+ pblk_rb_flush(&pblk->rwb);
+ do {
+ if (!pblk_rb_sync_count(&pblk->rwb))
+ break;
- bio_put(bio);
+ pblk_write_kick(pblk);
+ schedule();
+ } while (1);
}
struct list_head *pblk_line_gc_list(struct pblk *pblk, struct pblk_line *line)
struct pblk_line_meta *lm = &pblk->lm;
struct pblk_line_mgmt *l_mg = &pblk->l_mg;
struct list_head *move_list = NULL;
+ int vsc = le32_to_cpu(*line->vsc);
- if (!line->vsc) {
+ lockdep_assert_held(&line->lock);
+
+ if (!vsc) {
if (line->gc_group != PBLK_LINEGC_FULL) {
line->gc_group = PBLK_LINEGC_FULL;
move_list = &l_mg->gc_full_list;
}
- } else if (line->vsc < lm->mid_thrs) {
+ } else if (vsc < lm->high_thrs) {
if (line->gc_group != PBLK_LINEGC_HIGH) {
line->gc_group = PBLK_LINEGC_HIGH;
move_list = &l_mg->gc_high_list;
}
- } else if (line->vsc < lm->high_thrs) {
+ } else if (vsc < lm->mid_thrs) {
if (line->gc_group != PBLK_LINEGC_MID) {
line->gc_group = PBLK_LINEGC_MID;
move_list = &l_mg->gc_mid_list;
}
- } else if (line->vsc < line->sec_in_line) {
+ } else if (vsc < line->sec_in_line) {
if (line->gc_group != PBLK_LINEGC_LOW) {
line->gc_group = PBLK_LINEGC_LOW;
move_list = &l_mg->gc_low_list;
}
- } else if (line->vsc == line->sec_in_line) {
+ } else if (vsc == line->sec_in_line) {
if (line->gc_group != PBLK_LINEGC_EMPTY) {
line->gc_group = PBLK_LINEGC_EMPTY;
move_list = &l_mg->gc_empty_list;
line->gc_group = PBLK_LINEGC_NONE;
move_list = &l_mg->corrupt_list;
pr_err("pblk: corrupted vsc for line %d, vsc:%d (%d/%d/%d)\n",
- line->id, line->vsc,
+ line->id, vsc,
line->sec_in_line,
lm->high_thrs, lm->mid_thrs);
}
#endif
}
+void pblk_set_sec_per_write(struct pblk *pblk, int sec_per_write)
+{
+ pblk->sec_per_write = sec_per_write;
+}
+
int pblk_submit_io(struct pblk *pblk, struct nvm_rq *rqd)
{
struct nvm_tgt_dev *dev = pblk->dev;
}
}
#endif
+
+ atomic_inc(&pblk->inflight_io);
+
return nvm_submit_io(dev, rqd);
}
struct bio *pblk_bio_map_addr(struct pblk *pblk, void *data,
unsigned int nr_secs, unsigned int len,
- gfp_t gfp_mask)
+ int alloc_type, gfp_t gfp_mask)
{
struct nvm_tgt_dev *dev = pblk->dev;
- struct pblk_line_mgmt *l_mg = &pblk->l_mg;
void *kaddr = data;
struct page *page;
struct bio *bio;
int i, ret;
- if (l_mg->emeta_alloc_type == PBLK_KMALLOC_META)
+ if (alloc_type == PBLK_KMALLOC_META)
return bio_map_kern(dev->q, kaddr, len, gfp_mask);
bio = bio_kmalloc(gfp_mask, nr_secs);
int pblk_calc_secs(struct pblk *pblk, unsigned long secs_avail,
unsigned long secs_to_flush)
{
- int max = pblk->max_write_pgs;
+ int max = pblk->sec_per_write;
int min = pblk->min_write_pgs;
int secs_to_sync = 0;
return secs_to_sync;
}
-static u64 __pblk_alloc_page(struct pblk *pblk, struct pblk_line *line,
- int nr_secs)
+void pblk_dealloc_page(struct pblk *pblk, struct pblk_line *line, int nr_secs)
+{
+ u64 addr;
+ int i;
+
+ addr = find_next_zero_bit(line->map_bitmap,
+ pblk->lm.sec_per_line, line->cur_sec);
+ line->cur_sec = addr - nr_secs;
+
+ for (i = 0; i < nr_secs; i++, line->cur_sec--)
+ WARN_ON(!test_and_clear_bit(line->cur_sec, line->map_bitmap));
+}
+
+u64 __pblk_alloc_page(struct pblk *pblk, struct pblk_line *line, int nr_secs)
{
u64 addr;
int i;
+ lockdep_assert_held(&line->lock);
+
/* logic error: ppa out-of-bounds. Prevent generating bad address */
if (line->cur_sec + nr_secs > pblk->lm.sec_per_line) {
WARN(1, "pblk: page allocation out of bounds\n");
return addr;
}
+u64 pblk_lookup_page(struct pblk *pblk, struct pblk_line *line)
+{
+ u64 paddr;
+
+ spin_lock(&line->lock);
+ paddr = find_next_zero_bit(line->map_bitmap,
+ pblk->lm.sec_per_line, line->cur_sec);
+ spin_unlock(&line->lock);
+
+ return paddr;
+}
+
/*
* Submit emeta to one LUN in the raid line at the time to avoid a deadlock when
* taking the per LUN semaphore.
*/
static int pblk_line_submit_emeta_io(struct pblk *pblk, struct pblk_line *line,
- u64 paddr, int dir)
+ void *emeta_buf, u64 paddr, int dir)
{
struct nvm_tgt_dev *dev = pblk->dev;
struct nvm_geo *geo = &dev->geo;
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
struct pblk_line_meta *lm = &pblk->lm;
+ void *ppa_list, *meta_list;
struct bio *bio;
struct nvm_rq rqd;
- struct ppa_addr *ppa_list;
- dma_addr_t dma_ppa_list;
- void *emeta = line->emeta;
+ dma_addr_t dma_ppa_list, dma_meta_list;
int min = pblk->min_write_pgs;
- int left_ppas = lm->emeta_sec;
+ int left_ppas = lm->emeta_sec[0];
int id = line->id;
int rq_ppas, rq_len;
int cmd_op, bio_op;
- int flags;
int i, j;
int ret;
DECLARE_COMPLETION_ONSTACK(wait);
if (dir == WRITE) {
bio_op = REQ_OP_WRITE;
cmd_op = NVM_OP_PWRITE;
- flags = pblk_set_progr_mode(pblk, WRITE);
} else if (dir == READ) {
bio_op = REQ_OP_READ;
cmd_op = NVM_OP_PREAD;
- flags = pblk_set_read_mode(pblk);
} else
return -EINVAL;
- ppa_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL, &dma_ppa_list);
- if (!ppa_list)
+ meta_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL,
+ &dma_meta_list);
+ if (!meta_list)
return -ENOMEM;
+ ppa_list = meta_list + pblk_dma_meta_size;
+ dma_ppa_list = dma_meta_list + pblk_dma_meta_size;
+
next_rq:
memset(&rqd, 0, sizeof(struct nvm_rq));
rq_ppas = pblk_calc_secs(pblk, left_ppas, 0);
rq_len = rq_ppas * geo->sec_size;
- bio = pblk_bio_map_addr(pblk, emeta, rq_ppas, rq_len, GFP_KERNEL);
+ bio = pblk_bio_map_addr(pblk, emeta_buf, rq_ppas, rq_len,
+ l_mg->emeta_alloc_type, GFP_KERNEL);
if (IS_ERR(bio)) {
ret = PTR_ERR(bio);
goto free_rqd_dma;
bio_set_op_attrs(bio, bio_op, 0);
rqd.bio = bio;
- rqd.opcode = cmd_op;
- rqd.flags = flags;
- rqd.nr_ppas = rq_ppas;
+ rqd.meta_list = meta_list;
rqd.ppa_list = ppa_list;
+ rqd.dma_meta_list = dma_meta_list;
rqd.dma_ppa_list = dma_ppa_list;
+ rqd.opcode = cmd_op;
+ rqd.nr_ppas = rq_ppas;
rqd.end_io = pblk_end_io_sync;
rqd.private = &wait;
if (dir == WRITE) {
+ struct pblk_sec_meta *meta_list = rqd.meta_list;
+
+ rqd.flags = pblk_set_progr_mode(pblk, WRITE);
for (i = 0; i < rqd.nr_ppas; ) {
spin_lock(&line->lock);
paddr = __pblk_alloc_page(pblk, line, min);
spin_unlock(&line->lock);
- for (j = 0; j < min; j++, i++, paddr++)
+ for (j = 0; j < min; j++, i++, paddr++) {
+ meta_list[i].lba = cpu_to_le64(ADDR_EMPTY);
rqd.ppa_list[i] =
addr_to_gen_ppa(pblk, paddr, id);
+ }
}
} else {
for (i = 0; i < rqd.nr_ppas; ) {
struct ppa_addr ppa = addr_to_gen_ppa(pblk, paddr, id);
int pos = pblk_dev_ppa_to_pos(geo, ppa);
+ int read_type = PBLK_READ_RANDOM;
+
+ if (pblk_io_aligned(pblk, rq_ppas))
+ read_type = PBLK_READ_SEQUENTIAL;
+ rqd.flags = pblk_set_read_mode(pblk, read_type);
while (test_bit(pos, line->blk_bitmap)) {
paddr += min;
msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
pr_err("pblk: emeta I/O timed out\n");
}
+ atomic_dec(&pblk->inflight_io);
reinit_completion(&wait);
- bio_put(bio);
+ if (likely(pblk->l_mg.emeta_alloc_type == PBLK_VMALLOC_META))
+ bio_put(bio);
if (rqd.error) {
if (dir == WRITE)
pblk_log_read_err(pblk, &rqd);
}
- emeta += rq_len;
+ emeta_buf += rq_len;
left_ppas -= rq_ppas;
if (left_ppas)
goto next_rq;
free_rqd_dma:
- nvm_dev_dma_free(dev->parent, ppa_list, dma_ppa_list);
+ nvm_dev_dma_free(dev->parent, rqd.meta_list, rqd.dma_meta_list);
return ret;
}
bio_op = REQ_OP_WRITE;
cmd_op = NVM_OP_PWRITE;
flags = pblk_set_progr_mode(pblk, WRITE);
- lba_list = pblk_line_emeta_to_lbas(line->emeta);
+ lba_list = emeta_to_lbas(pblk, line->emeta->buf);
} else if (dir == READ) {
bio_op = REQ_OP_READ;
cmd_op = NVM_OP_PREAD;
- flags = pblk_set_read_mode(pblk);
+ flags = pblk_set_read_mode(pblk, PBLK_READ_SEQUENTIAL);
} else
return -EINVAL;
memset(&rqd, 0, sizeof(struct nvm_rq));
- rqd.ppa_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL,
- &rqd.dma_ppa_list);
- if (!rqd.ppa_list)
+ rqd.meta_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL,
+ &rqd.dma_meta_list);
+ if (!rqd.meta_list)
return -ENOMEM;
+ rqd.ppa_list = rqd.meta_list + pblk_dma_meta_size;
+ rqd.dma_ppa_list = rqd.dma_meta_list + pblk_dma_meta_size;
+
bio = bio_map_kern(dev->q, line->smeta, lm->smeta_len, GFP_KERNEL);
if (IS_ERR(bio)) {
ret = PTR_ERR(bio);
rqd.private = &wait;
for (i = 0; i < lm->smeta_sec; i++, paddr++) {
+ struct pblk_sec_meta *meta_list = rqd.meta_list;
+
rqd.ppa_list[i] = addr_to_gen_ppa(pblk, paddr, line->id);
- if (dir == WRITE)
- lba_list[paddr] = cpu_to_le64(ADDR_EMPTY);
+
+ if (dir == WRITE) {
+ __le64 addr_empty = cpu_to_le64(ADDR_EMPTY);
+
+ meta_list[i].lba = lba_list[paddr] = addr_empty;
+ }
}
/*
msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
pr_err("pblk: smeta I/O timed out\n");
}
+ atomic_dec(&pblk->inflight_io);
if (rqd.error) {
if (dir == WRITE)
}
free_ppa_list:
- nvm_dev_dma_free(dev->parent, rqd.ppa_list, rqd.dma_ppa_list);
+ nvm_dev_dma_free(dev->parent, rqd.meta_list, rqd.dma_meta_list);
return ret;
}
return pblk_line_submit_smeta_io(pblk, line, bpaddr, READ);
}
-int pblk_line_read_emeta(struct pblk *pblk, struct pblk_line *line)
+int pblk_line_read_emeta(struct pblk *pblk, struct pblk_line *line,
+ void *emeta_buf)
{
- return pblk_line_submit_emeta_io(pblk, line, line->emeta_ssec, READ);
+ return pblk_line_submit_emeta_io(pblk, line, emeta_buf,
+ line->emeta_ssec, READ);
}
static void pblk_setup_e_rq(struct pblk *pblk, struct nvm_rq *rqd,
static int pblk_blk_erase_sync(struct pblk *pblk, struct ppa_addr ppa)
{
struct nvm_rq rqd;
- int ret;
+ int ret = 0;
DECLARE_COMPLETION_ONSTACK(wait);
memset(&rqd, 0, sizeof(struct nvm_rq));
rqd.private = pblk;
__pblk_end_io_erase(pblk, &rqd);
- return 0;
+ return ret;
}
int pblk_line_erase(struct pblk *pblk, struct pblk_line *line)
{
struct pblk_line_meta *lm = &pblk->lm;
struct ppa_addr ppa;
- int bit = -1;
+ int ret, bit = -1;
/* Erase only good blocks, one at a time */
do {
WARN_ON(test_and_set_bit(bit, line->erase_bitmap));
spin_unlock(&line->lock);
- if (pblk_blk_erase_sync(pblk, ppa)) {
+ ret = pblk_blk_erase_sync(pblk, ppa);
+ if (ret) {
pr_err("pblk: failed to erase line %d\n", line->id);
- return -ENOMEM;
+ return ret;
}
} while (1);
return 0;
}
+static void pblk_line_setup_metadata(struct pblk_line *line,
+ struct pblk_line_mgmt *l_mg,
+ struct pblk_line_meta *lm)
+{
+ int meta_line;
+
+ lockdep_assert_held(&l_mg->free_lock);
+
+retry_meta:
+ meta_line = find_first_zero_bit(&l_mg->meta_bitmap, PBLK_DATA_LINES);
+ if (meta_line == PBLK_DATA_LINES) {
+ spin_unlock(&l_mg->free_lock);
+ io_schedule();
+ spin_lock(&l_mg->free_lock);
+ goto retry_meta;
+ }
+
+ set_bit(meta_line, &l_mg->meta_bitmap);
+ line->meta_line = meta_line;
+
+ line->smeta = l_mg->sline_meta[meta_line];
+ line->emeta = l_mg->eline_meta[meta_line];
+
+ memset(line->smeta, 0, lm->smeta_len);
+ memset(line->emeta->buf, 0, lm->emeta_len[0]);
+
+ line->emeta->mem = 0;
+ atomic_set(&line->emeta->sync, 0);
+}
+
/* For now lines are always assumed full lines. Thus, smeta former and current
* lun bitmaps are omitted.
*/
-static int pblk_line_set_metadata(struct pblk *pblk, struct pblk_line *line,
+static int pblk_line_init_metadata(struct pblk *pblk, struct pblk_line *line,
struct pblk_line *cur)
{
struct nvm_tgt_dev *dev = pblk->dev;
struct nvm_geo *geo = &dev->geo;
struct pblk_line_meta *lm = &pblk->lm;
struct pblk_line_mgmt *l_mg = &pblk->l_mg;
- struct line_smeta *smeta = line->smeta;
- struct line_emeta *emeta = line->emeta;
+ struct pblk_emeta *emeta = line->emeta;
+ struct line_emeta *emeta_buf = emeta->buf;
+ struct line_smeta *smeta_buf = (struct line_smeta *)line->smeta;
int nr_blk_line;
/* After erasing the line, new bad blocks might appear and we risk
}
/* Run-time metadata */
- line->lun_bitmap = ((void *)(smeta)) + sizeof(struct line_smeta);
+ line->lun_bitmap = ((void *)(smeta_buf)) + sizeof(struct line_smeta);
/* Mark LUNs allocated in this line (all for now) */
bitmap_set(line->lun_bitmap, 0, lm->lun_bitmap_len);
- smeta->header.identifier = cpu_to_le32(PBLK_MAGIC);
- memcpy(smeta->header.uuid, pblk->instance_uuid, 16);
- smeta->header.id = cpu_to_le32(line->id);
- smeta->header.type = cpu_to_le16(line->type);
- smeta->header.version = cpu_to_le16(1);
+ smeta_buf->header.identifier = cpu_to_le32(PBLK_MAGIC);
+ memcpy(smeta_buf->header.uuid, pblk->instance_uuid, 16);
+ smeta_buf->header.id = cpu_to_le32(line->id);
+ smeta_buf->header.type = cpu_to_le16(line->type);
+ smeta_buf->header.version = cpu_to_le16(1);
/* Start metadata */
- smeta->seq_nr = cpu_to_le64(line->seq_nr);
- smeta->window_wr_lun = cpu_to_le32(geo->nr_luns);
+ smeta_buf->seq_nr = cpu_to_le64(line->seq_nr);
+ smeta_buf->window_wr_lun = cpu_to_le32(geo->nr_luns);
/* Fill metadata among lines */
if (cur) {
memcpy(line->lun_bitmap, cur->lun_bitmap, lm->lun_bitmap_len);
- smeta->prev_id = cpu_to_le32(cur->id);
- cur->emeta->next_id = cpu_to_le32(line->id);
+ smeta_buf->prev_id = cpu_to_le32(cur->id);
+ cur->emeta->buf->next_id = cpu_to_le32(line->id);
} else {
- smeta->prev_id = cpu_to_le32(PBLK_LINE_EMPTY);
+ smeta_buf->prev_id = cpu_to_le32(PBLK_LINE_EMPTY);
}
/* All smeta must be set at this point */
- smeta->header.crc = cpu_to_le32(pblk_calc_meta_header_crc(pblk, smeta));
- smeta->crc = cpu_to_le32(pblk_calc_smeta_crc(pblk, smeta));
+ smeta_buf->header.crc = cpu_to_le32(
+ pblk_calc_meta_header_crc(pblk, &smeta_buf->header));
+ smeta_buf->crc = cpu_to_le32(pblk_calc_smeta_crc(pblk, smeta_buf));
/* End metadata */
- memcpy(&emeta->header, &smeta->header, sizeof(struct line_header));
- emeta->seq_nr = cpu_to_le64(line->seq_nr);
- emeta->nr_lbas = cpu_to_le64(line->sec_in_line);
- emeta->nr_valid_lbas = cpu_to_le64(0);
- emeta->next_id = cpu_to_le32(PBLK_LINE_EMPTY);
- emeta->crc = cpu_to_le32(0);
- emeta->prev_id = smeta->prev_id;
+ memcpy(&emeta_buf->header, &smeta_buf->header,
+ sizeof(struct line_header));
+ emeta_buf->seq_nr = cpu_to_le64(line->seq_nr);
+ emeta_buf->nr_lbas = cpu_to_le64(line->sec_in_line);
+ emeta_buf->nr_valid_lbas = cpu_to_le64(0);
+ emeta_buf->next_id = cpu_to_le32(PBLK_LINE_EMPTY);
+ emeta_buf->crc = cpu_to_le32(0);
+ emeta_buf->prev_id = smeta_buf->prev_id;
return 1;
}
/* Mark smeta metadata sectors as bad sectors */
bit = find_first_zero_bit(line->blk_bitmap, lm->blk_per_line);
off = bit * geo->sec_per_pl;
-retry_smeta:
bitmap_set(line->map_bitmap, off, lm->smeta_sec);
line->sec_in_line -= lm->smeta_sec;
line->smeta_ssec = off;
if (init && pblk_line_submit_smeta_io(pblk, line, off, WRITE)) {
pr_debug("pblk: line smeta I/O failed. Retry\n");
- off += geo->sec_per_pl;
- goto retry_smeta;
+ return 1;
}
bitmap_copy(line->invalid_bitmap, line->map_bitmap, lm->sec_per_line);
* blocks to make sure that there are enough sectors to store emeta
*/
bit = lm->sec_per_line;
- off = lm->sec_per_line - lm->emeta_sec;
- bitmap_set(line->invalid_bitmap, off, lm->emeta_sec);
+ off = lm->sec_per_line - lm->emeta_sec[0];
+ bitmap_set(line->invalid_bitmap, off, lm->emeta_sec[0]);
while (nr_bb) {
off -= geo->sec_per_pl;
if (!test_bit(off, line->invalid_bitmap)) {
}
}
- line->sec_in_line -= lm->emeta_sec;
+ line->sec_in_line -= lm->emeta_sec[0];
line->emeta_ssec = off;
- line->vsc = line->left_ssecs = line->left_msecs = line->sec_in_line;
+ line->nr_valid_lbas = 0;
+ line->left_msecs = line->sec_in_line;
+ *line->vsc = cpu_to_le32(line->sec_in_line);
if (lm->sec_per_line - line->sec_in_line !=
bitmap_weight(line->invalid_bitmap, lm->sec_per_line)) {
spin_lock(&line->lock);
if (line->state != PBLK_LINESTATE_FREE) {
+ mempool_free(line->invalid_bitmap, pblk->line_meta_pool);
+ mempool_free(line->map_bitmap, pblk->line_meta_pool);
spin_unlock(&line->lock);
- WARN(1, "pblk: corrupted line state\n");
- return -EINTR;
+ WARN(1, "pblk: corrupted line %d, state %d\n",
+ line->id, line->state);
+ return -EAGAIN;
}
+
line->state = PBLK_LINESTATE_OPEN;
atomic_set(&line->left_eblks, blk_in_line);
atomic_set(&line->left_seblks, blk_in_line);
+
+ line->meta_distance = lm->meta_distance;
spin_unlock(&line->lock);
/* Bad blocks do not need to be erased */
{
struct pblk_line_mgmt *l_mg = &pblk->l_mg;
struct pblk_line_meta *lm = &pblk->lm;
- struct pblk_line *line = NULL;
- int bit;
+ struct pblk_line *line;
+ int ret, bit;
lockdep_assert_held(&l_mg->free_lock);
-retry_get:
+retry:
if (list_empty(&l_mg->free_list)) {
pr_err("pblk: no free lines\n");
- goto out;
+ return NULL;
}
line = list_first_entry(&l_mg->free_list, struct pblk_line, list);
list_add_tail(&line->list, &l_mg->bad_list);
pr_debug("pblk: line %d is bad\n", line->id);
- goto retry_get;
+ goto retry;
}
- if (pblk_line_prepare(pblk, line)) {
- pr_err("pblk: failed to prepare line %d\n", line->id);
- list_add(&line->list, &l_mg->free_list);
- return NULL;
+ ret = pblk_line_prepare(pblk, line);
+ if (ret) {
+ if (ret == -EAGAIN) {
+ list_add(&line->list, &l_mg->corrupt_list);
+ goto retry;
+ } else {
+ pr_err("pblk: failed to prepare line %d\n", line->id);
+ list_add(&line->list, &l_mg->free_list);
+ l_mg->nr_free_lines++;
+ return NULL;
+ }
}
-out:
return line;
}
struct pblk_line_mgmt *l_mg = &pblk->l_mg;
struct pblk_line *retry_line;
+retry:
spin_lock(&l_mg->free_lock);
retry_line = pblk_line_get(pblk);
if (!retry_line) {
l_mg->data_line = retry_line;
spin_unlock(&l_mg->free_lock);
- if (pblk_line_erase(pblk, retry_line)) {
- spin_lock(&l_mg->free_lock);
- l_mg->data_line = NULL;
- spin_unlock(&l_mg->free_lock);
- return NULL;
- }
-
pblk_rl_free_lines_dec(&pblk->rl, retry_line);
+ if (pblk_line_erase(pblk, retry_line))
+ goto retry;
+
return retry_line;
}
+static void pblk_set_space_limit(struct pblk *pblk)
+{
+ struct pblk_rl *rl = &pblk->rl;
+
+ atomic_set(&rl->rb_space, 0);
+}
+
struct pblk_line *pblk_line_get_first_data(struct pblk *pblk)
{
struct pblk_line_mgmt *l_mg = &pblk->l_mg;
struct pblk_line *line;
- int meta_line;
int is_next = 0;
spin_lock(&l_mg->free_lock);
line->type = PBLK_LINETYPE_DATA;
l_mg->data_line = line;
- meta_line = find_first_zero_bit(&l_mg->meta_bitmap, PBLK_DATA_LINES);
- set_bit(meta_line, &l_mg->meta_bitmap);
- line->smeta = l_mg->sline_meta[meta_line].meta;
- line->emeta = l_mg->eline_meta[meta_line].meta;
- line->meta_line = meta_line;
+ pblk_line_setup_metadata(line, l_mg, &pblk->lm);
/* Allocate next line for preparation */
l_mg->data_next = pblk_line_get(pblk);
- if (l_mg->data_next) {
+ if (!l_mg->data_next) {
+ /* If we cannot get a new line, we need to stop the pipeline.
+ * Only allow as many writes in as we can store safely and then
+ * fail gracefully
+ */
+ pblk_set_space_limit(pblk);
+
+ l_mg->data_next = NULL;
+ } else {
l_mg->data_next->seq_nr = l_mg->d_seq_nr++;
l_mg->data_next->type = PBLK_LINETYPE_DATA;
is_next = 1;
}
spin_unlock(&l_mg->free_lock);
+ if (pblk_line_erase(pblk, line)) {
+ line = pblk_line_retry(pblk, line);
+ if (!line)
+ return NULL;
+ }
+
pblk_rl_free_lines_dec(&pblk->rl, line);
if (is_next)
pblk_rl_free_lines_dec(&pblk->rl, l_mg->data_next);
- if (pblk_line_erase(pblk, line))
- return NULL;
-
retry_setup:
- if (!pblk_line_set_metadata(pblk, line, NULL)) {
+ if (!pblk_line_init_metadata(pblk, line, NULL)) {
line = pblk_line_retry(pblk, line);
if (!line)
return NULL;
return line;
}
-struct pblk_line *pblk_line_replace_data(struct pblk *pblk)
+static void pblk_stop_writes(struct pblk *pblk, struct pblk_line *line)
+{
+ lockdep_assert_held(&pblk->l_mg.free_lock);
+
+ pblk_set_space_limit(pblk);
+ pblk->state = PBLK_STATE_STOPPING;
+}
+
+void pblk_pipeline_stop(struct pblk *pblk)
+{
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ int ret;
+
+ spin_lock(&l_mg->free_lock);
+ if (pblk->state == PBLK_STATE_RECOVERING ||
+ pblk->state == PBLK_STATE_STOPPED) {
+ spin_unlock(&l_mg->free_lock);
+ return;
+ }
+ pblk->state = PBLK_STATE_RECOVERING;
+ spin_unlock(&l_mg->free_lock);
+
+ pblk_flush_writer(pblk);
+ pblk_wait_for_meta(pblk);
+
+ ret = pblk_recov_pad(pblk);
+ if (ret) {
+ pr_err("pblk: could not close data on teardown(%d)\n", ret);
+ return;
+ }
+
+ flush_workqueue(pblk->bb_wq);
+ pblk_line_close_meta_sync(pblk);
+
+ spin_lock(&l_mg->free_lock);
+ pblk->state = PBLK_STATE_STOPPED;
+ l_mg->data_line = NULL;
+ l_mg->data_next = NULL;
+ spin_unlock(&l_mg->free_lock);
+}
+
+void pblk_line_replace_data(struct pblk *pblk)
{
- struct pblk_line_meta *lm = &pblk->lm;
struct pblk_line_mgmt *l_mg = &pblk->l_mg;
struct pblk_line *cur, *new;
unsigned int left_seblks;
- int meta_line;
int is_next = 0;
cur = l_mg->data_line;
new = l_mg->data_next;
if (!new)
- return NULL;
+ return;
l_mg->data_line = new;
-retry_line:
+ spin_lock(&l_mg->free_lock);
+ if (pblk->state != PBLK_STATE_RUNNING) {
+ l_mg->data_line = NULL;
+ l_mg->data_next = NULL;
+ spin_unlock(&l_mg->free_lock);
+ return;
+ }
+
+ pblk_line_setup_metadata(new, l_mg, &pblk->lm);
+ spin_unlock(&l_mg->free_lock);
+
+retry_erase:
left_seblks = atomic_read(&new->left_seblks);
if (left_seblks) {
/* If line is not fully erased, erase it */
if (atomic_read(&new->left_eblks)) {
if (pblk_line_erase(pblk, new))
- return NULL;
+ return;
} else {
io_schedule();
}
- goto retry_line;
+ goto retry_erase;
}
- spin_lock(&l_mg->free_lock);
- /* Allocate next line for preparation */
- l_mg->data_next = pblk_line_get(pblk);
- if (l_mg->data_next) {
- l_mg->data_next->seq_nr = l_mg->d_seq_nr++;
- l_mg->data_next->type = PBLK_LINETYPE_DATA;
- is_next = 1;
- }
-
-retry_meta:
- meta_line = find_first_zero_bit(&l_mg->meta_bitmap, PBLK_DATA_LINES);
- if (meta_line == PBLK_DATA_LINES) {
- spin_unlock(&l_mg->free_lock);
- io_schedule();
- spin_lock(&l_mg->free_lock);
- goto retry_meta;
- }
-
- set_bit(meta_line, &l_mg->meta_bitmap);
- new->smeta = l_mg->sline_meta[meta_line].meta;
- new->emeta = l_mg->eline_meta[meta_line].meta;
- new->meta_line = meta_line;
-
- memset(new->smeta, 0, lm->smeta_len);
- memset(new->emeta, 0, lm->emeta_len);
- spin_unlock(&l_mg->free_lock);
-
- if (is_next)
- pblk_rl_free_lines_dec(&pblk->rl, l_mg->data_next);
-
retry_setup:
- if (!pblk_line_set_metadata(pblk, new, cur)) {
+ if (!pblk_line_init_metadata(pblk, new, cur)) {
new = pblk_line_retry(pblk, new);
if (!new)
- return NULL;
+ return;
goto retry_setup;
}
if (!pblk_line_init_bb(pblk, new, 1)) {
new = pblk_line_retry(pblk, new);
if (!new)
- return NULL;
+ return;
goto retry_setup;
}
- return new;
+ /* Allocate next line for preparation */
+ spin_lock(&l_mg->free_lock);
+ l_mg->data_next = pblk_line_get(pblk);
+ if (!l_mg->data_next) {
+ /* If we cannot get a new line, we need to stop the pipeline.
+ * Only allow as many writes in as we can store safely and then
+ * fail gracefully
+ */
+ pblk_stop_writes(pblk, new);
+ l_mg->data_next = NULL;
+ } else {
+ l_mg->data_next->seq_nr = l_mg->d_seq_nr++;
+ l_mg->data_next->type = PBLK_LINETYPE_DATA;
+ is_next = 1;
+ }
+ spin_unlock(&l_mg->free_lock);
+
+ if (is_next)
+ pblk_rl_free_lines_dec(&pblk->rl, l_mg->data_next);
}
void pblk_line_free(struct pblk *pblk, struct pblk_line *line)
if (line->invalid_bitmap)
mempool_free(line->invalid_bitmap, pblk->line_meta_pool);
+ *line->vsc = cpu_to_le32(EMPTY_ENTRY);
+
line->map_bitmap = NULL;
line->invalid_bitmap = NULL;
line->smeta = NULL;
struct nvm_rq *rqd;
int err;
- rqd = mempool_alloc(pblk->r_rq_pool, GFP_KERNEL);
- memset(rqd, 0, pblk_r_rq_size);
+ rqd = mempool_alloc(pblk->g_rq_pool, GFP_KERNEL);
+ memset(rqd, 0, pblk_g_rq_size);
pblk_setup_e_rq(pblk, rqd, ppa);
return pblk->l_mg.data_line;
}
-struct pblk_line *pblk_line_get_data_next(struct pblk *pblk)
+/* For now, always erase next line */
+struct pblk_line *pblk_line_get_erase(struct pblk *pblk)
{
return pblk->l_mg.data_next;
}
return (line->left_msecs == 0);
}
+void pblk_line_close_meta_sync(struct pblk *pblk)
+{
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct pblk_line *line, *tline;
+ LIST_HEAD(list);
+
+ spin_lock(&l_mg->close_lock);
+ if (list_empty(&l_mg->emeta_list)) {
+ spin_unlock(&l_mg->close_lock);
+ return;
+ }
+
+ list_cut_position(&list, &l_mg->emeta_list, l_mg->emeta_list.prev);
+ spin_unlock(&l_mg->close_lock);
+
+ list_for_each_entry_safe(line, tline, &list, list) {
+ struct pblk_emeta *emeta = line->emeta;
+
+ while (emeta->mem < lm->emeta_len[0]) {
+ int ret;
+
+ ret = pblk_submit_meta_io(pblk, line);
+ if (ret) {
+ pr_err("pblk: sync meta line %d failed (%d)\n",
+ line->id, ret);
+ return;
+ }
+ }
+ }
+
+ pblk_wait_for_meta(pblk);
+ flush_workqueue(pblk->close_wq);
+}
+
+static void pblk_line_should_sync_meta(struct pblk *pblk)
+{
+ if (pblk_rl_is_limit(&pblk->rl))
+ pblk_line_close_meta_sync(pblk);
+}
+
void pblk_line_close(struct pblk *pblk, struct pblk_line *line)
{
struct pblk_line_mgmt *l_mg = &pblk->l_mg;
struct list_head *move_list;
- line->emeta->crc = cpu_to_le32(pblk_calc_emeta_crc(pblk, line->emeta));
-
- if (pblk_line_submit_emeta_io(pblk, line, line->cur_sec, WRITE))
- pr_err("pblk: line %d close I/O failed\n", line->id);
+#ifdef CONFIG_NVM_DEBUG
+ struct pblk_line_meta *lm = &pblk->lm;
- WARN(!bitmap_full(line->map_bitmap, line->sec_in_line),
+ WARN(!bitmap_full(line->map_bitmap, lm->sec_per_line),
"pblk: corrupt closed line %d\n", line->id);
+#endif
spin_lock(&l_mg->free_lock);
WARN_ON(!test_and_clear_bit(line->meta_line, &l_mg->meta_bitmap));
spin_unlock(&line->lock);
spin_unlock(&l_mg->gc_lock);
+
+ pblk_gc_should_kick(pblk);
+}
+
+void pblk_line_close_meta(struct pblk *pblk, struct pblk_line *line)
+{
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct pblk_emeta *emeta = line->emeta;
+ struct line_emeta *emeta_buf = emeta->buf;
+
+ /* No need for exact vsc value; avoid a big line lock and take aprox. */
+ memcpy(emeta_to_vsc(pblk, emeta_buf), l_mg->vsc_list, lm->vsc_list_len);
+ memcpy(emeta_to_bb(emeta_buf), line->blk_bitmap, lm->blk_bitmap_len);
+
+ emeta_buf->nr_valid_lbas = cpu_to_le64(line->nr_valid_lbas);
+ emeta_buf->crc = cpu_to_le32(pblk_calc_emeta_crc(pblk, emeta_buf));
+
+ spin_lock(&l_mg->close_lock);
+ spin_lock(&line->lock);
+ list_add_tail(&line->list, &l_mg->emeta_list);
+ spin_unlock(&line->lock);
+ spin_unlock(&l_mg->close_lock);
+
+ pblk_line_should_sync_meta(pblk);
}
void pblk_line_close_ws(struct work_struct *work)
}
void pblk_line_run_ws(struct pblk *pblk, struct pblk_line *line, void *priv,
- void (*work)(struct work_struct *))
+ void (*work)(struct work_struct *),
+ struct workqueue_struct *wq)
{
struct pblk_line_ws *line_ws;
line_ws->priv = priv;
INIT_WORK(&line_ws->ws, work);
- queue_work(pblk->kw_wq, &line_ws->ws);
+ queue_work(wq, &line_ws->ws);
}
void pblk_down_rq(struct pblk *pblk, struct ppa_addr *ppa_list, int nr_ppas,
struct nvm_tgt_dev *dev = pblk->dev;
struct nvm_geo *geo = &dev->geo;
struct pblk_lun *rlun;
- int lun_id = ppa_list[0].g.ch * geo->luns_per_chnl + ppa_list[0].g.lun;
+ int pos = pblk_ppa_to_pos(geo, ppa_list[0]);
int ret;
/*
/* If the LUN has been locked for this same request, do no attempt to
* lock it again
*/
- if (test_and_set_bit(lun_id, lun_bitmap))
+ if (test_and_set_bit(pos, lun_bitmap))
return;
- rlun = &pblk->luns[lun_id];
+ rlun = &pblk->luns[pos];
ret = down_timeout(&rlun->wr_sem, msecs_to_jiffies(5000));
if (ret) {
switch (ret) {
static void pblk_gc_free_gc_rq(struct pblk_gc_rq *gc_rq)
{
- kfree(gc_rq->data);
- kfree(gc_rq->lba_list);
+ vfree(gc_rq->data);
kfree(gc_rq);
}
return 1;
}
- list_for_each_entry_safe(gc_rq, tgc_rq, &gc->w_list, list) {
- list_move_tail(&gc_rq->list, &w_list);
- gc->w_entries--;
- }
+ list_cut_position(&w_list, &gc->w_list, gc->w_list.prev);
+ gc->w_entries = 0;
spin_unlock(&gc->w_lock);
list_for_each_entry_safe(gc_rq, tgc_rq, &w_list, list) {
gc_rq->nr_secs, gc_rq->secs_to_gc,
gc_rq->line, PBLK_IOTYPE_GC);
- kref_put(&gc_rq->line->ref, pblk_line_put);
-
list_del(&gc_rq->list);
+ kref_put(&gc_rq->line->ref, pblk_line_put);
pblk_gc_free_gc_rq(gc_rq);
}
* Responsible for managing all memory related to a gc request. Also in case of
* failure
*/
-static int pblk_gc_move_valid_secs(struct pblk *pblk, struct pblk_line *line,
- u64 *lba_list, unsigned int nr_secs)
+static int pblk_gc_move_valid_secs(struct pblk *pblk, struct pblk_gc_rq *gc_rq)
{
struct nvm_tgt_dev *dev = pblk->dev;
struct nvm_geo *geo = &dev->geo;
struct pblk_gc *gc = &pblk->gc;
- struct pblk_gc_rq *gc_rq;
+ struct pblk_line *line = gc_rq->line;
void *data;
unsigned int secs_to_gc;
- int ret = NVM_IO_OK;
+ int ret = 0;
- data = kmalloc(nr_secs * geo->sec_size, GFP_KERNEL);
+ data = vmalloc(gc_rq->nr_secs * geo->sec_size);
if (!data) {
- ret = NVM_IO_ERR;
- goto free_lba_list;
+ ret = -ENOMEM;
+ goto out;
}
/* Read from GC victim block */
- if (pblk_submit_read_gc(pblk, lba_list, data, nr_secs,
+ if (pblk_submit_read_gc(pblk, gc_rq->lba_list, data, gc_rq->nr_secs,
&secs_to_gc, line)) {
- ret = NVM_IO_ERR;
+ ret = -EFAULT;
goto free_data;
}
if (!secs_to_gc)
- goto free_data;
-
- gc_rq = kmalloc(sizeof(struct pblk_gc_rq), GFP_KERNEL);
- if (!gc_rq) {
- ret = NVM_IO_ERR;
- goto free_data;
- }
+ goto free_rq;
- gc_rq->line = line;
gc_rq->data = data;
- gc_rq->lba_list = lba_list;
- gc_rq->nr_secs = nr_secs;
gc_rq->secs_to_gc = secs_to_gc;
- kref_get(&line->ref);
-
retry:
spin_lock(&gc->w_lock);
- if (gc->w_entries > 256) {
+ if (gc->w_entries >= PBLK_GC_W_QD) {
spin_unlock(&gc->w_lock);
- usleep_range(256, 1024);
+ pblk_gc_writer_kick(&pblk->gc);
+ usleep_range(128, 256);
goto retry;
}
gc->w_entries++;
pblk_gc_writer_kick(&pblk->gc);
- return NVM_IO_OK;
+ return 0;
+free_rq:
+ kfree(gc_rq);
free_data:
- kfree(data);
-free_lba_list:
- kfree(lba_list);
-
+ vfree(data);
+out:
+ kref_put(&line->ref, pblk_line_put);
return ret;
}
}
static void pblk_gc_line_ws(struct work_struct *work)
+{
+ struct pblk_line_ws *line_rq_ws = container_of(work,
+ struct pblk_line_ws, ws);
+ struct pblk *pblk = line_rq_ws->pblk;
+ struct pblk_gc *gc = &pblk->gc;
+ struct pblk_line *line = line_rq_ws->line;
+ struct pblk_gc_rq *gc_rq = line_rq_ws->priv;
+
+ up(&gc->gc_sem);
+
+ if (pblk_gc_move_valid_secs(pblk, gc_rq)) {
+ pr_err("pblk: could not GC all sectors: line:%d (%d/%d)\n",
+ line->id, *line->vsc,
+ gc_rq->nr_secs);
+ }
+
+ mempool_free(line_rq_ws, pblk->line_ws_pool);
+}
+
+static void pblk_gc_line_prepare_ws(struct work_struct *work)
{
struct pblk_line_ws *line_ws = container_of(work, struct pblk_line_ws,
ws);
struct pblk *pblk = line_ws->pblk;
- struct pblk_line_mgmt *l_mg = &pblk->l_mg;
struct pblk_line *line = line_ws->line;
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
struct pblk_line_meta *lm = &pblk->lm;
- __le64 *lba_list = line_ws->priv;
- u64 *gc_list;
- int sec_left;
- int nr_ppas, bit;
- int put_line = 1;
+ struct pblk_gc *gc = &pblk->gc;
+ struct line_emeta *emeta_buf;
+ struct pblk_line_ws *line_rq_ws;
+ struct pblk_gc_rq *gc_rq;
+ __le64 *lba_list;
+ int sec_left, nr_secs, bit;
+ int ret;
- pr_debug("pblk: line '%d' being reclaimed for GC\n", line->id);
+ emeta_buf = pblk_malloc(lm->emeta_len[0], l_mg->emeta_alloc_type,
+ GFP_KERNEL);
+ if (!emeta_buf) {
+ pr_err("pblk: cannot use GC emeta\n");
+ return;
+ }
- spin_lock(&line->lock);
- sec_left = line->vsc;
- if (!sec_left) {
- /* Lines are erased before being used (l_mg->data_/log_next) */
- spin_unlock(&line->lock);
- goto out;
+ ret = pblk_line_read_emeta(pblk, line, emeta_buf);
+ if (ret) {
+ pr_err("pblk: line %d read emeta failed (%d)\n", line->id, ret);
+ goto fail_free_emeta;
+ }
+
+ /* If this read fails, it means that emeta is corrupted. For now, leave
+ * the line untouched. TODO: Implement a recovery routine that scans and
+ * moves all sectors on the line.
+ */
+ lba_list = pblk_recov_get_lba_list(pblk, emeta_buf);
+ if (!lba_list) {
+ pr_err("pblk: could not interpret emeta (line %d)\n", line->id);
+ goto fail_free_emeta;
}
- spin_unlock(&line->lock);
+ sec_left = pblk_line_vsc(line);
if (sec_left < 0) {
pr_err("pblk: corrupted GC line (%d)\n", line->id);
- put_line = 0;
- pblk_put_line_back(pblk, line);
- goto out;
+ goto fail_free_emeta;
}
bit = -1;
next_rq:
- gc_list = kmalloc_array(pblk->max_write_pgs, sizeof(u64), GFP_KERNEL);
- if (!gc_list) {
- put_line = 0;
- pblk_put_line_back(pblk, line);
- goto out;
- }
+ gc_rq = kmalloc(sizeof(struct pblk_gc_rq), GFP_KERNEL);
+ if (!gc_rq)
+ goto fail_free_emeta;
- nr_ppas = 0;
+ nr_secs = 0;
do {
bit = find_next_zero_bit(line->invalid_bitmap, lm->sec_per_line,
bit + 1);
if (bit > line->emeta_ssec)
break;
- gc_list[nr_ppas++] = le64_to_cpu(lba_list[bit]);
- } while (nr_ppas < pblk->max_write_pgs);
+ gc_rq->lba_list[nr_secs++] = le64_to_cpu(lba_list[bit]);
+ } while (nr_secs < pblk->max_write_pgs);
- if (unlikely(!nr_ppas)) {
- kfree(gc_list);
+ if (unlikely(!nr_secs)) {
+ kfree(gc_rq);
goto out;
}
- if (pblk_gc_move_valid_secs(pblk, line, gc_list, nr_ppas)) {
- pr_err("pblk: could not GC all sectors: line:%d (%d/%d/%d)\n",
- line->id, line->vsc,
- nr_ppas, nr_ppas);
- put_line = 0;
- pblk_put_line_back(pblk, line);
- goto out;
- }
+ gc_rq->nr_secs = nr_secs;
+ gc_rq->line = line;
+
+ line_rq_ws = mempool_alloc(pblk->line_ws_pool, GFP_KERNEL);
+ if (!line_rq_ws)
+ goto fail_free_gc_rq;
- sec_left -= nr_ppas;
+ line_rq_ws->pblk = pblk;
+ line_rq_ws->line = line;
+ line_rq_ws->priv = gc_rq;
+
+ down(&gc->gc_sem);
+ kref_get(&line->ref);
+
+ INIT_WORK(&line_rq_ws->ws, pblk_gc_line_ws);
+ queue_work(gc->gc_line_reader_wq, &line_rq_ws->ws);
+
+ sec_left -= nr_secs;
if (sec_left > 0)
goto next_rq;
out:
- pblk_mfree(line->emeta, l_mg->emeta_alloc_type);
+ pblk_mfree(emeta_buf, l_mg->emeta_alloc_type);
mempool_free(line_ws, pblk->line_ws_pool);
- atomic_dec(&pblk->gc.inflight_gc);
- if (put_line)
- kref_put(&line->ref, pblk_line_put);
+
+ kref_put(&line->ref, pblk_line_put);
+ atomic_dec(&gc->inflight_gc);
+
+ return;
+
+fail_free_gc_rq:
+ kfree(gc_rq);
+fail_free_emeta:
+ pblk_mfree(emeta_buf, l_mg->emeta_alloc_type);
+ pblk_put_line_back(pblk, line);
+ kref_put(&line->ref, pblk_line_put);
+ mempool_free(line_ws, pblk->line_ws_pool);
+ atomic_dec(&gc->inflight_gc);
+
+ pr_err("pblk: Failed to GC line %d\n", line->id);
}
static int pblk_gc_line(struct pblk *pblk, struct pblk_line *line)
{
- struct pblk_line_mgmt *l_mg = &pblk->l_mg;
- struct pblk_line_meta *lm = &pblk->lm;
+ struct pblk_gc *gc = &pblk->gc;
struct pblk_line_ws *line_ws;
- __le64 *lba_list;
- int ret;
- line_ws = mempool_alloc(pblk->line_ws_pool, GFP_KERNEL);
- line->emeta = pblk_malloc(lm->emeta_len, l_mg->emeta_alloc_type,
- GFP_KERNEL);
- if (!line->emeta) {
- pr_err("pblk: cannot use GC emeta\n");
- goto fail_free_ws;
- }
-
- ret = pblk_line_read_emeta(pblk, line);
- if (ret) {
- pr_err("pblk: line %d read emeta failed (%d)\n", line->id, ret);
- goto fail_free_emeta;
- }
+ pr_debug("pblk: line '%d' being reclaimed for GC\n", line->id);
- /* If this read fails, it means that emeta is corrupted. For now, leave
- * the line untouched. TODO: Implement a recovery routine that scans and
- * moves all sectors on the line.
- */
- lba_list = pblk_recov_get_lba_list(pblk, line->emeta);
- if (!lba_list) {
- pr_err("pblk: could not interpret emeta (line %d)\n", line->id);
- goto fail_free_emeta;
- }
+ line_ws = mempool_alloc(pblk->line_ws_pool, GFP_KERNEL);
+ if (!line_ws)
+ return -ENOMEM;
line_ws->pblk = pblk;
line_ws->line = line;
- line_ws->priv = lba_list;
- INIT_WORK(&line_ws->ws, pblk_gc_line_ws);
- queue_work(pblk->gc.gc_reader_wq, &line_ws->ws);
+ INIT_WORK(&line_ws->ws, pblk_gc_line_prepare_ws);
+ queue_work(gc->gc_reader_wq, &line_ws->ws);
return 0;
+}
-fail_free_emeta:
- pblk_mfree(line->emeta, l_mg->emeta_alloc_type);
-fail_free_ws:
- mempool_free(line_ws, pblk->line_ws_pool);
- pblk_put_line_back(pblk, line);
+static int pblk_gc_read(struct pblk *pblk)
+{
+ struct pblk_gc *gc = &pblk->gc;
+ struct pblk_line *line;
+
+ spin_lock(&gc->r_lock);
+ if (list_empty(&gc->r_list)) {
+ spin_unlock(&gc->r_lock);
+ return 1;
+ }
+
+ line = list_first_entry(&gc->r_list, struct pblk_line, list);
+ list_del(&line->list);
+ spin_unlock(&gc->r_lock);
+
+ pblk_gc_kick(pblk);
- return 1;
+ if (pblk_gc_line(pblk, line))
+ pr_err("pblk: failed to GC line %d\n", line->id);
+
+ return 0;
}
-static void pblk_gc_lines(struct pblk *pblk, struct list_head *gc_list)
+static void pblk_gc_reader_kick(struct pblk_gc *gc)
{
- struct pblk_line *line, *tline;
+ wake_up_process(gc->gc_reader_ts);
+}
- list_for_each_entry_safe(line, tline, gc_list, list) {
- if (pblk_gc_line(pblk, line))
- pr_err("pblk: failed to GC line %d\n", line->id);
- list_del(&line->list);
+static struct pblk_line *pblk_gc_get_victim_line(struct pblk *pblk,
+ struct list_head *group_list)
+{
+ struct pblk_line *line, *victim;
+ int line_vsc, victim_vsc;
+
+ victim = list_first_entry(group_list, struct pblk_line, list);
+ list_for_each_entry(line, group_list, list) {
+ line_vsc = le32_to_cpu(*line->vsc);
+ victim_vsc = le32_to_cpu(*victim->vsc);
+ if (line_vsc < victim_vsc)
+ victim = line;
}
+
+ return victim;
+}
+
+static bool pblk_gc_should_run(struct pblk_gc *gc, struct pblk_rl *rl)
+{
+ unsigned int nr_blocks_free, nr_blocks_need;
+
+ nr_blocks_need = pblk_rl_high_thrs(rl);
+ nr_blocks_free = pblk_rl_nr_free_blks(rl);
+
+ /* This is not critical, no need to take lock here */
+ return ((gc->gc_active) && (nr_blocks_need > nr_blocks_free));
}
/*
{
struct pblk_line_mgmt *l_mg = &pblk->l_mg;
struct pblk_gc *gc = &pblk->gc;
- struct pblk_line *line, *tline;
- unsigned int nr_blocks_free, nr_blocks_need;
+ struct pblk_line *line;
struct list_head *group_list;
- int run_gc, gc_group = 0;
- int prev_gc = 0;
- int inflight_gc = atomic_read(&gc->inflight_gc);
- LIST_HEAD(gc_list);
+ bool run_gc;
+ int inflight_gc, gc_group = 0, prev_group = 0;
+
+ do {
+ spin_lock(&l_mg->gc_lock);
+ if (list_empty(&l_mg->gc_full_list)) {
+ spin_unlock(&l_mg->gc_lock);
+ break;
+ }
+
+ line = list_first_entry(&l_mg->gc_full_list,
+ struct pblk_line, list);
- spin_lock(&l_mg->gc_lock);
- list_for_each_entry_safe(line, tline, &l_mg->gc_full_list, list) {
spin_lock(&line->lock);
WARN_ON(line->state != PBLK_LINESTATE_CLOSED);
line->state = PBLK_LINESTATE_GC;
spin_unlock(&line->lock);
list_del(&line->list);
+ spin_unlock(&l_mg->gc_lock);
+
kref_put(&line->ref, pblk_line_put);
- }
- spin_unlock(&l_mg->gc_lock);
+ } while (1);
- nr_blocks_need = pblk_rl_gc_thrs(&pblk->rl);
- nr_blocks_free = pblk_rl_nr_free_blks(&pblk->rl);
- run_gc = (nr_blocks_need > nr_blocks_free || gc->gc_forced);
+ run_gc = pblk_gc_should_run(&pblk->gc, &pblk->rl);
+ if (!run_gc || (atomic_read(&gc->inflight_gc) >= PBLK_GC_L_QD))
+ return;
next_gc_group:
group_list = l_mg->gc_lists[gc_group++];
- spin_lock(&l_mg->gc_lock);
- while (run_gc && !list_empty(group_list)) {
- /* No need to queue up more GC lines than we can handle */
- if (!run_gc || inflight_gc > gc->gc_jobs_active) {
+
+ do {
+ spin_lock(&l_mg->gc_lock);
+ if (list_empty(group_list)) {
spin_unlock(&l_mg->gc_lock);
- pblk_gc_lines(pblk, &gc_list);
- return;
+ break;
}
- line = list_first_entry(group_list, struct pblk_line, list);
- nr_blocks_free += atomic_read(&line->blk_in_line);
+ line = pblk_gc_get_victim_line(pblk, group_list);
spin_lock(&line->lock);
WARN_ON(line->state != PBLK_LINESTATE_CLOSED);
line->state = PBLK_LINESTATE_GC;
- list_move_tail(&line->list, &gc_list);
- atomic_inc(&gc->inflight_gc);
- inflight_gc++;
spin_unlock(&line->lock);
- prev_gc = 1;
- run_gc = (nr_blocks_need > nr_blocks_free || gc->gc_forced);
- }
- spin_unlock(&l_mg->gc_lock);
+ list_del(&line->list);
+ spin_unlock(&l_mg->gc_lock);
+
+ spin_lock(&gc->r_lock);
+ list_add_tail(&line->list, &gc->r_list);
+ spin_unlock(&gc->r_lock);
- pblk_gc_lines(pblk, &gc_list);
+ inflight_gc = atomic_inc_return(&gc->inflight_gc);
+ pblk_gc_reader_kick(gc);
- if (!prev_gc && pblk->rl.rb_state > gc_group &&
- gc_group < PBLK_NR_GC_LISTS)
+ prev_group = 1;
+
+ /* No need to queue up more GC lines than we can handle */
+ run_gc = pblk_gc_should_run(&pblk->gc, &pblk->rl);
+ if (!run_gc || inflight_gc >= PBLK_GC_L_QD)
+ break;
+ } while (1);
+
+ if (!prev_group && pblk->rl.rb_state > gc_group &&
+ gc_group < PBLK_GC_NR_LISTS)
goto next_gc_group;
}
-
-static void pblk_gc_kick(struct pblk *pblk)
+void pblk_gc_kick(struct pblk *pblk)
{
struct pblk_gc *gc = &pblk->gc;
wake_up_process(gc->gc_ts);
pblk_gc_writer_kick(gc);
+ pblk_gc_reader_kick(gc);
mod_timer(&gc->gc_timer, jiffies + msecs_to_jiffies(GC_TIME_MSECS));
}
return 0;
}
-static void pblk_gc_start(struct pblk *pblk)
+static int pblk_gc_reader_ts(void *data)
{
- pblk->gc.gc_active = 1;
+ struct pblk *pblk = data;
- pr_debug("pblk: gc start\n");
+ while (!kthread_should_stop()) {
+ if (!pblk_gc_read(pblk))
+ continue;
+ set_current_state(TASK_INTERRUPTIBLE);
+ io_schedule();
+ }
+
+ return 0;
}
-int pblk_gc_status(struct pblk *pblk)
+static void pblk_gc_start(struct pblk *pblk)
{
- struct pblk_gc *gc = &pblk->gc;
- int ret;
-
- spin_lock(&gc->lock);
- ret = gc->gc_active;
- spin_unlock(&gc->lock);
-
- return ret;
+ pblk->gc.gc_active = 1;
+ pr_debug("pblk: gc start\n");
}
-static void __pblk_gc_should_start(struct pblk *pblk)
+void pblk_gc_should_start(struct pblk *pblk)
{
struct pblk_gc *gc = &pblk->gc;
- lockdep_assert_held(&gc->lock);
-
if (gc->gc_enabled && !gc->gc_active)
pblk_gc_start(pblk);
-}
-void pblk_gc_should_start(struct pblk *pblk)
-{
- struct pblk_gc *gc = &pblk->gc;
-
- spin_lock(&gc->lock);
- __pblk_gc_should_start(pblk);
- spin_unlock(&gc->lock);
+ pblk_gc_kick(pblk);
}
/*
*/
static void pblk_gc_stop(struct pblk *pblk, int flush_wq)
{
- spin_lock(&pblk->gc.lock);
pblk->gc.gc_active = 0;
- spin_unlock(&pblk->gc.lock);
-
pr_debug("pblk: gc stop\n");
}
spin_unlock(&gc->lock);
}
-void pblk_gc_sysfs_force(struct pblk *pblk, int force)
+int pblk_gc_sysfs_force(struct pblk *pblk, int force)
{
struct pblk_gc *gc = &pblk->gc;
- int rsv = 0;
+
+ if (force < 0 || force > 1)
+ return -EINVAL;
spin_lock(&gc->lock);
- if (force) {
- gc->gc_enabled = 1;
- rsv = 64;
- }
- pblk_rl_set_gc_rsc(&pblk->rl, rsv);
gc->gc_forced = force;
- __pblk_gc_should_start(pblk);
+
+ if (force)
+ gc->gc_enabled = 1;
+ else
+ gc->gc_enabled = 0;
spin_unlock(&gc->lock);
+
+ pblk_gc_should_start(pblk);
+
+ return 0;
}
int pblk_gc_init(struct pblk *pblk)
goto fail_free_main_kthread;
}
+ gc->gc_reader_ts = kthread_create(pblk_gc_reader_ts, pblk,
+ "pblk-gc-reader-ts");
+ if (IS_ERR(gc->gc_reader_ts)) {
+ pr_err("pblk: could not allocate GC reader kthread\n");
+ ret = PTR_ERR(gc->gc_reader_ts);
+ goto fail_free_writer_kthread;
+ }
+
setup_timer(&gc->gc_timer, pblk_gc_timer, (unsigned long)pblk);
mod_timer(&gc->gc_timer, jiffies + msecs_to_jiffies(GC_TIME_MSECS));
gc->gc_active = 0;
gc->gc_forced = 0;
gc->gc_enabled = 1;
- gc->gc_jobs_active = 8;
gc->w_entries = 0;
atomic_set(&gc->inflight_gc, 0);
- gc->gc_reader_wq = alloc_workqueue("pblk-gc-reader-wq",
- WQ_MEM_RECLAIM | WQ_UNBOUND, gc->gc_jobs_active);
+ /* Workqueue that reads valid sectors from a line and submit them to the
+ * GC writer to be recycled.
+ */
+ gc->gc_line_reader_wq = alloc_workqueue("pblk-gc-line-reader-wq",
+ WQ_MEM_RECLAIM | WQ_UNBOUND, PBLK_GC_MAX_READERS);
+ if (!gc->gc_line_reader_wq) {
+ pr_err("pblk: could not allocate GC line reader workqueue\n");
+ ret = -ENOMEM;
+ goto fail_free_reader_kthread;
+ }
+
+ /* Workqueue that prepare lines for GC */
+ gc->gc_reader_wq = alloc_workqueue("pblk-gc-line_wq",
+ WQ_MEM_RECLAIM | WQ_UNBOUND, 1);
if (!gc->gc_reader_wq) {
pr_err("pblk: could not allocate GC reader workqueue\n");
ret = -ENOMEM;
- goto fail_free_writer_kthread;
+ goto fail_free_reader_line_wq;
}
spin_lock_init(&gc->lock);
spin_lock_init(&gc->w_lock);
+ spin_lock_init(&gc->r_lock);
+
+ sema_init(&gc->gc_sem, 128);
+
INIT_LIST_HEAD(&gc->w_list);
+ INIT_LIST_HEAD(&gc->r_list);
return 0;
+fail_free_reader_line_wq:
+ destroy_workqueue(gc->gc_line_reader_wq);
+fail_free_reader_kthread:
+ kthread_stop(gc->gc_reader_ts);
fail_free_writer_kthread:
kthread_stop(gc->gc_writer_ts);
fail_free_main_kthread:
struct pblk_gc *gc = &pblk->gc;
flush_workqueue(gc->gc_reader_wq);
+ flush_workqueue(gc->gc_line_reader_wq);
del_timer(&gc->gc_timer);
pblk_gc_stop(pblk, 1);
if (gc->gc_ts)
kthread_stop(gc->gc_ts);
- if (pblk->gc.gc_reader_wq)
- destroy_workqueue(pblk->gc.gc_reader_wq);
+ if (gc->gc_reader_wq)
+ destroy_workqueue(gc->gc_reader_wq);
+
+ if (gc->gc_line_reader_wq)
+ destroy_workqueue(gc->gc_line_reader_wq);
if (gc->gc_writer_ts)
kthread_stop(gc->gc_writer_ts);
+
+ if (gc->gc_reader_ts)
+ kthread_stop(gc->gc_reader_ts);
}
#include "pblk.h"
-static struct kmem_cache *pblk_blk_ws_cache, *pblk_rec_cache, *pblk_r_rq_cache,
- *pblk_w_rq_cache, *pblk_line_meta_cache;
+static struct kmem_cache *pblk_blk_ws_cache, *pblk_rec_cache, *pblk_g_rq_cache,
+ *pblk_w_rq_cache, *pblk_line_meta_cache;
static DECLARE_RWSEM(pblk_lock);
+struct bio_set *pblk_bio_set;
static int pblk_rw_io(struct request_queue *q, struct pblk *pblk,
struct bio *bio)
* constraint. Writes can be of arbitrary size.
*/
if (bio_data_dir(bio) == READ) {
- blk_queue_split(q, &bio, q->bio_split);
+ blk_queue_split(q, &bio);
ret = pblk_submit_read(pblk, bio);
if (ret == NVM_IO_DONE && bio_flagged(bio, BIO_CLONED))
bio_put(bio);
* available for user I/O.
*/
if (unlikely(pblk_get_secs(bio) >= pblk_rl_sysfs_rate_show(&pblk->rl)))
- blk_queue_split(q, &bio, q->bio_split);
+ blk_queue_split(q, &bio);
return pblk_write_to_cache(pblk, bio, PBLK_IOTYPE_USER);
}
return -ENOMEM;
}
- pblk_r_rq_cache = kmem_cache_create("pblk_r_rq", pblk_r_rq_size,
+ pblk_g_rq_cache = kmem_cache_create("pblk_g_rq", pblk_g_rq_size,
0, 0, NULL);
- if (!pblk_r_rq_cache) {
+ if (!pblk_g_rq_cache) {
kmem_cache_destroy(pblk_blk_ws_cache);
kmem_cache_destroy(pblk_rec_cache);
up_write(&pblk_lock);
if (!pblk_w_rq_cache) {
kmem_cache_destroy(pblk_blk_ws_cache);
kmem_cache_destroy(pblk_rec_cache);
- kmem_cache_destroy(pblk_r_rq_cache);
+ kmem_cache_destroy(pblk_g_rq_cache);
up_write(&pblk_lock);
return -ENOMEM;
}
if (!pblk_line_meta_cache) {
kmem_cache_destroy(pblk_blk_ws_cache);
kmem_cache_destroy(pblk_rec_cache);
- kmem_cache_destroy(pblk_r_rq_cache);
+ kmem_cache_destroy(pblk_g_rq_cache);
kmem_cache_destroy(pblk_w_rq_cache);
up_write(&pblk_lock);
return -ENOMEM;
{
struct nvm_tgt_dev *dev = pblk->dev;
struct nvm_geo *geo = &dev->geo;
- int max_write_ppas;
- int mod;
- pblk->min_write_pgs = geo->sec_per_pl * (geo->sec_size / PAGE_SIZE);
- max_write_ppas = pblk->min_write_pgs * geo->nr_luns;
- pblk->max_write_pgs = (max_write_ppas < nvm_max_phys_sects(dev)) ?
- max_write_ppas : nvm_max_phys_sects(dev);
pblk->pgs_in_buffer = NVM_MEM_PAGE_WRITE * geo->sec_per_pg *
geo->nr_planes * geo->nr_luns;
- if (pblk->max_write_pgs > PBLK_MAX_REQ_ADDRS) {
- pr_err("pblk: cannot support device max_phys_sect\n");
- return -EINVAL;
- }
-
- div_u64_rem(geo->sec_per_blk, pblk->min_write_pgs, &mod);
- if (mod) {
- pr_err("pblk: bad configuration of sectors/pages\n");
- return -EINVAL;
- }
-
if (pblk_init_global_caches(pblk))
return -ENOMEM;
if (!pblk->page_pool)
return -ENOMEM;
- pblk->line_ws_pool = mempool_create_slab_pool(geo->nr_luns,
+ pblk->line_ws_pool = mempool_create_slab_pool(PBLK_WS_POOL_SIZE,
pblk_blk_ws_cache);
if (!pblk->line_ws_pool)
goto free_page_pool;
if (!pblk->rec_pool)
goto free_blk_ws_pool;
- pblk->r_rq_pool = mempool_create_slab_pool(64, pblk_r_rq_cache);
- if (!pblk->r_rq_pool)
+ pblk->g_rq_pool = mempool_create_slab_pool(PBLK_READ_REQ_POOL_SIZE,
+ pblk_g_rq_cache);
+ if (!pblk->g_rq_pool)
goto free_rec_pool;
- pblk->w_rq_pool = mempool_create_slab_pool(64, pblk_w_rq_cache);
+ pblk->w_rq_pool = mempool_create_slab_pool(geo->nr_luns * 2,
+ pblk_w_rq_cache);
if (!pblk->w_rq_pool)
- goto free_r_rq_pool;
+ goto free_g_rq_pool;
pblk->line_meta_pool =
- mempool_create_slab_pool(16, pblk_line_meta_cache);
+ mempool_create_slab_pool(PBLK_META_POOL_SIZE,
+ pblk_line_meta_cache);
if (!pblk->line_meta_pool)
goto free_w_rq_pool;
- pblk->kw_wq = alloc_workqueue("pblk-aux-wq",
- WQ_MEM_RECLAIM | WQ_UNBOUND, 1);
- if (!pblk->kw_wq)
+ pblk->close_wq = alloc_workqueue("pblk-close-wq",
+ WQ_MEM_RECLAIM | WQ_UNBOUND, PBLK_NR_CLOSE_JOBS);
+ if (!pblk->close_wq)
goto free_line_meta_pool;
+ pblk->bb_wq = alloc_workqueue("pblk-bb-wq",
+ WQ_MEM_RECLAIM | WQ_UNBOUND, 0);
+ if (!pblk->bb_wq)
+ goto free_close_wq;
+
if (pblk_set_ppaf(pblk))
- goto free_kw_wq;
+ goto free_bb_wq;
if (pblk_rwb_init(pblk))
- goto free_kw_wq;
+ goto free_bb_wq;
INIT_LIST_HEAD(&pblk->compl_list);
return 0;
-free_kw_wq:
- destroy_workqueue(pblk->kw_wq);
+free_bb_wq:
+ destroy_workqueue(pblk->bb_wq);
+free_close_wq:
+ destroy_workqueue(pblk->close_wq);
free_line_meta_pool:
mempool_destroy(pblk->line_meta_pool);
free_w_rq_pool:
mempool_destroy(pblk->w_rq_pool);
-free_r_rq_pool:
- mempool_destroy(pblk->r_rq_pool);
+free_g_rq_pool:
+ mempool_destroy(pblk->g_rq_pool);
free_rec_pool:
mempool_destroy(pblk->rec_pool);
free_blk_ws_pool:
static void pblk_core_free(struct pblk *pblk)
{
- if (pblk->kw_wq)
- destroy_workqueue(pblk->kw_wq);
+ if (pblk->close_wq)
+ destroy_workqueue(pblk->close_wq);
+
+ if (pblk->bb_wq)
+ destroy_workqueue(pblk->bb_wq);
mempool_destroy(pblk->page_pool);
mempool_destroy(pblk->line_ws_pool);
mempool_destroy(pblk->rec_pool);
- mempool_destroy(pblk->r_rq_pool);
+ mempool_destroy(pblk->g_rq_pool);
mempool_destroy(pblk->w_rq_pool);
mempool_destroy(pblk->line_meta_pool);
kmem_cache_destroy(pblk_blk_ws_cache);
kmem_cache_destroy(pblk_rec_cache);
- kmem_cache_destroy(pblk_r_rq_cache);
+ kmem_cache_destroy(pblk_g_rq_cache);
kmem_cache_destroy(pblk_w_rq_cache);
kmem_cache_destroy(pblk_line_meta_cache);
}
kfree(pblk->luns);
}
+static void pblk_free_line_bitmaps(struct pblk_line *line)
+{
+ kfree(line->blk_bitmap);
+ kfree(line->erase_bitmap);
+}
+
static void pblk_lines_free(struct pblk *pblk)
{
struct pblk_line_mgmt *l_mg = &pblk->l_mg;
line = &pblk->lines[i];
pblk_line_free(pblk, line);
- kfree(line->blk_bitmap);
- kfree(line->erase_bitmap);
+ pblk_free_line_bitmaps(line);
}
spin_unlock(&l_mg->free_lock);
}
kfree(l_mg->bb_template);
kfree(l_mg->bb_aux);
+ kfree(l_mg->vsc_list);
+ spin_lock(&l_mg->free_lock);
for (i = 0; i < PBLK_DATA_LINES; i++) {
- pblk_mfree(l_mg->sline_meta[i].meta, l_mg->smeta_alloc_type);
- pblk_mfree(l_mg->eline_meta[i].meta, l_mg->emeta_alloc_type);
+ kfree(l_mg->sline_meta[i]);
+ pblk_mfree(l_mg->eline_meta[i]->buf, l_mg->emeta_alloc_type);
+ kfree(l_mg->eline_meta[i]);
}
+ spin_unlock(&l_mg->free_lock);
kfree(pblk->lines);
}
return ret;
}
-static int pblk_bb_line(struct pblk *pblk, struct pblk_line *line)
+static int pblk_bb_line(struct pblk *pblk, struct pblk_line *line,
+ int blk_per_line)
{
- struct pblk_line_meta *lm = &pblk->lm;
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
struct pblk_lun *rlun;
int bb_cnt = 0;
int i;
+ for (i = 0; i < blk_per_line; i++) {
+ rlun = &pblk->luns[i];
+ if (rlun->bb_list[line->id] == NVM_BLK_T_FREE)
+ continue;
+
+ set_bit(pblk_ppa_to_pos(geo, rlun->bppa), line->blk_bitmap);
+ bb_cnt++;
+ }
+
+ return bb_cnt;
+}
+
+static int pblk_alloc_line_bitmaps(struct pblk *pblk, struct pblk_line *line)
+{
+ struct pblk_line_meta *lm = &pblk->lm;
+
line->blk_bitmap = kzalloc(lm->blk_bitmap_len, GFP_KERNEL);
if (!line->blk_bitmap)
return -ENOMEM;
return -ENOMEM;
}
- for (i = 0; i < lm->blk_per_line; i++) {
- rlun = &pblk->luns[i];
- if (rlun->bb_list[line->id] == NVM_BLK_T_FREE)
- continue;
-
- set_bit(i, line->blk_bitmap);
- bb_cnt++;
- }
-
- return bb_cnt;
+ return 0;
}
static int pblk_luns_init(struct pblk *pblk, struct ppa_addr *luns)
}
/* See comment over struct line_emeta definition */
-static unsigned int calc_emeta_len(struct pblk *pblk, struct pblk_line_meta *lm)
+static unsigned int calc_emeta_len(struct pblk *pblk)
{
- return (sizeof(struct line_emeta) +
- ((lm->sec_per_line - lm->emeta_sec) * sizeof(u64)) +
- (pblk->l_mg.nr_lines * sizeof(u32)) +
- lm->blk_bitmap_len);
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+
+ /* Round to sector size so that lba_list starts on its own sector */
+ lm->emeta_sec[1] = DIV_ROUND_UP(
+ sizeof(struct line_emeta) + lm->blk_bitmap_len,
+ geo->sec_size);
+ lm->emeta_len[1] = lm->emeta_sec[1] * geo->sec_size;
+
+ /* Round to sector size so that vsc_list starts on its own sector */
+ lm->dsec_per_line = lm->sec_per_line - lm->emeta_sec[0];
+ lm->emeta_sec[2] = DIV_ROUND_UP(lm->dsec_per_line * sizeof(u64),
+ geo->sec_size);
+ lm->emeta_len[2] = lm->emeta_sec[2] * geo->sec_size;
+
+ lm->emeta_sec[3] = DIV_ROUND_UP(l_mg->nr_lines * sizeof(u32),
+ geo->sec_size);
+ lm->emeta_len[3] = lm->emeta_sec[3] * geo->sec_size;
+
+ lm->vsc_list_len = l_mg->nr_lines * sizeof(u32);
+
+ return (lm->emeta_len[1] + lm->emeta_len[2] + lm->emeta_len[3]);
}
static void pblk_set_provision(struct pblk *pblk, long nr_free_blks)
atomic_set(&pblk->rl.free_blocks, nr_free_blks);
}
+static int pblk_lines_alloc_metadata(struct pblk *pblk)
+{
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct pblk_line_meta *lm = &pblk->lm;
+ int i;
+
+ /* smeta is always small enough to fit on a kmalloc memory allocation,
+ * emeta depends on the number of LUNs allocated to the pblk instance
+ */
+ for (i = 0; i < PBLK_DATA_LINES; i++) {
+ l_mg->sline_meta[i] = kmalloc(lm->smeta_len, GFP_KERNEL);
+ if (!l_mg->sline_meta[i])
+ goto fail_free_smeta;
+ }
+
+ /* emeta allocates three different buffers for managing metadata with
+ * in-memory and in-media layouts
+ */
+ for (i = 0; i < PBLK_DATA_LINES; i++) {
+ struct pblk_emeta *emeta;
+
+ emeta = kmalloc(sizeof(struct pblk_emeta), GFP_KERNEL);
+ if (!emeta)
+ goto fail_free_emeta;
+
+ if (lm->emeta_len[0] > KMALLOC_MAX_CACHE_SIZE) {
+ l_mg->emeta_alloc_type = PBLK_VMALLOC_META;
+
+ emeta->buf = vmalloc(lm->emeta_len[0]);
+ if (!emeta->buf) {
+ kfree(emeta);
+ goto fail_free_emeta;
+ }
+
+ emeta->nr_entries = lm->emeta_sec[0];
+ l_mg->eline_meta[i] = emeta;
+ } else {
+ l_mg->emeta_alloc_type = PBLK_KMALLOC_META;
+
+ emeta->buf = kmalloc(lm->emeta_len[0], GFP_KERNEL);
+ if (!emeta->buf) {
+ kfree(emeta);
+ goto fail_free_emeta;
+ }
+
+ emeta->nr_entries = lm->emeta_sec[0];
+ l_mg->eline_meta[i] = emeta;
+ }
+ }
+
+ l_mg->vsc_list = kcalloc(l_mg->nr_lines, sizeof(__le32), GFP_KERNEL);
+ if (!l_mg->vsc_list)
+ goto fail_free_emeta;
+
+ for (i = 0; i < l_mg->nr_lines; i++)
+ l_mg->vsc_list[i] = cpu_to_le32(EMPTY_ENTRY);
+
+ return 0;
+
+fail_free_emeta:
+ while (--i >= 0) {
+ vfree(l_mg->eline_meta[i]->buf);
+ kfree(l_mg->eline_meta[i]);
+ }
+
+fail_free_smeta:
+ for (i = 0; i < PBLK_DATA_LINES; i++)
+ kfree(l_mg->sline_meta[i]);
+
+ return -ENOMEM;
+}
+
static int pblk_lines_init(struct pblk *pblk)
{
struct nvm_tgt_dev *dev = pblk->dev;
struct pblk_line_meta *lm = &pblk->lm;
struct pblk_line *line;
unsigned int smeta_len, emeta_len;
- long nr_bad_blks, nr_meta_blks, nr_free_blks;
- int bb_distance;
- int i;
- int ret;
+ long nr_bad_blks, nr_free_blks;
+ int bb_distance, max_write_ppas, mod;
+ int i, ret;
+
+ pblk->min_write_pgs = geo->sec_per_pl * (geo->sec_size / PAGE_SIZE);
+ max_write_ppas = pblk->min_write_pgs * geo->nr_luns;
+ pblk->max_write_pgs = (max_write_ppas < nvm_max_phys_sects(dev)) ?
+ max_write_ppas : nvm_max_phys_sects(dev);
+ pblk_set_sec_per_write(pblk, pblk->min_write_pgs);
+
+ if (pblk->max_write_pgs > PBLK_MAX_REQ_ADDRS) {
+ pr_err("pblk: cannot support device max_phys_sect\n");
+ return -EINVAL;
+ }
+
+ div_u64_rem(geo->sec_per_blk, pblk->min_write_pgs, &mod);
+ if (mod) {
+ pr_err("pblk: bad configuration of sectors/pages\n");
+ return -EINVAL;
+ }
+
+ l_mg->nr_lines = geo->blks_per_lun;
+ l_mg->log_line = l_mg->data_line = NULL;
+ l_mg->l_seq_nr = l_mg->d_seq_nr = 0;
+ l_mg->nr_free_lines = 0;
+ bitmap_zero(&l_mg->meta_bitmap, PBLK_DATA_LINES);
lm->sec_per_line = geo->sec_per_blk * geo->nr_luns;
lm->blk_per_line = geo->nr_luns;
lm->lun_bitmap_len = BITS_TO_LONGS(geo->nr_luns) * sizeof(long);
lm->high_thrs = lm->sec_per_line / 2;
lm->mid_thrs = lm->sec_per_line / 4;
+ lm->meta_distance = (geo->nr_luns / 2) * pblk->min_write_pgs;
/* Calculate necessary pages for smeta. See comment over struct
* line_smeta definition
*/
- lm->smeta_len = sizeof(struct line_smeta) +
- PBLK_LINE_NR_LUN_BITMAP * lm->lun_bitmap_len;
-
i = 1;
add_smeta_page:
lm->smeta_sec = i * geo->sec_per_pl;
lm->smeta_len = lm->smeta_sec * geo->sec_size;
- smeta_len = sizeof(struct line_smeta) +
- PBLK_LINE_NR_LUN_BITMAP * lm->lun_bitmap_len;
+ smeta_len = sizeof(struct line_smeta) + lm->lun_bitmap_len;
if (smeta_len > lm->smeta_len) {
i++;
goto add_smeta_page;
*/
i = 1;
add_emeta_page:
- lm->emeta_sec = i * geo->sec_per_pl;
- lm->emeta_len = lm->emeta_sec * geo->sec_size;
+ lm->emeta_sec[0] = i * geo->sec_per_pl;
+ lm->emeta_len[0] = lm->emeta_sec[0] * geo->sec_size;
- emeta_len = calc_emeta_len(pblk, lm);
- if (emeta_len > lm->emeta_len) {
+ emeta_len = calc_emeta_len(pblk);
+ if (emeta_len > lm->emeta_len[0]) {
i++;
goto add_emeta_page;
}
- lm->emeta_bb = geo->nr_luns - i;
-
- nr_meta_blks = (lm->smeta_sec + lm->emeta_sec +
- (geo->sec_per_blk / 2)) / geo->sec_per_blk;
- lm->min_blk_line = nr_meta_blks + 1;
-
- l_mg->nr_lines = geo->blks_per_lun;
- l_mg->log_line = l_mg->data_line = NULL;
- l_mg->l_seq_nr = l_mg->d_seq_nr = 0;
- l_mg->nr_free_lines = 0;
- bitmap_zero(&l_mg->meta_bitmap, PBLK_DATA_LINES);
- /* smeta is always small enough to fit on a kmalloc memory allocation,
- * emeta depends on the number of LUNs allocated to the pblk instance
- */
- l_mg->smeta_alloc_type = PBLK_KMALLOC_META;
- for (i = 0; i < PBLK_DATA_LINES; i++) {
- l_mg->sline_meta[i].meta = kmalloc(lm->smeta_len, GFP_KERNEL);
- if (!l_mg->sline_meta[i].meta)
- while (--i >= 0) {
- kfree(l_mg->sline_meta[i].meta);
- ret = -ENOMEM;
- goto fail;
- }
+ lm->emeta_bb = geo->nr_luns - i;
+ lm->min_blk_line = 1 + DIV_ROUND_UP(lm->smeta_sec + lm->emeta_sec[0],
+ geo->sec_per_blk);
+ if (lm->min_blk_line > lm->blk_per_line) {
+ pr_err("pblk: config. not supported. Min. LUN in line:%d\n",
+ lm->blk_per_line);
+ ret = -EINVAL;
+ goto fail;
}
- if (lm->emeta_len > KMALLOC_MAX_CACHE_SIZE) {
- l_mg->emeta_alloc_type = PBLK_VMALLOC_META;
-
- for (i = 0; i < PBLK_DATA_LINES; i++) {
- l_mg->eline_meta[i].meta = vmalloc(lm->emeta_len);
- if (!l_mg->eline_meta[i].meta)
- while (--i >= 0) {
- vfree(l_mg->eline_meta[i].meta);
- ret = -ENOMEM;
- goto fail;
- }
- }
- } else {
- l_mg->emeta_alloc_type = PBLK_KMALLOC_META;
-
- for (i = 0; i < PBLK_DATA_LINES; i++) {
- l_mg->eline_meta[i].meta =
- kmalloc(lm->emeta_len, GFP_KERNEL);
- if (!l_mg->eline_meta[i].meta)
- while (--i >= 0) {
- kfree(l_mg->eline_meta[i].meta);
- ret = -ENOMEM;
- goto fail;
- }
- }
- }
+ ret = pblk_lines_alloc_metadata(pblk);
+ if (ret)
+ goto fail;
l_mg->bb_template = kzalloc(lm->sec_bitmap_len, GFP_KERNEL);
if (!l_mg->bb_template) {
INIT_LIST_HEAD(&l_mg->gc_low_list);
INIT_LIST_HEAD(&l_mg->gc_empty_list);
+ INIT_LIST_HEAD(&l_mg->emeta_list);
+
l_mg->gc_lists[0] = &l_mg->gc_high_list;
l_mg->gc_lists[1] = &l_mg->gc_mid_list;
l_mg->gc_lists[2] = &l_mg->gc_low_list;
spin_lock_init(&l_mg->free_lock);
+ spin_lock_init(&l_mg->close_lock);
spin_lock_init(&l_mg->gc_lock);
pblk->lines = kcalloc(l_mg->nr_lines, sizeof(struct pblk_line),
line->type = PBLK_LINETYPE_FREE;
line->state = PBLK_LINESTATE_FREE;
line->gc_group = PBLK_LINEGC_NONE;
+ line->vsc = &l_mg->vsc_list[i];
spin_lock_init(&line->lock);
- nr_bad_blks = pblk_bb_line(pblk, line);
+ ret = pblk_alloc_line_bitmaps(pblk, line);
+ if (ret)
+ goto fail_free_lines;
+
+ nr_bad_blks = pblk_bb_line(pblk, line, lm->blk_per_line);
if (nr_bad_blks < 0 || nr_bad_blks > lm->blk_per_line) {
+ pblk_free_line_bitmaps(line);
ret = -EINVAL;
goto fail_free_lines;
}
pblk_set_provision(pblk, nr_free_blks);
- sema_init(&pblk->erase_sem, 1);
-
/* Cleanup per-LUN bad block lists - managed within lines on run-time */
for (i = 0; i < geo->nr_luns; i++)
kfree(pblk->luns[i].bb_list);
return 0;
fail_free_lines:
- kfree(pblk->lines);
+ while (--i >= 0)
+ pblk_free_line_bitmaps(&pblk->lines[i]);
fail_free_bb_aux:
kfree(l_mg->bb_aux);
fail_free_bb_template:
kfree(l_mg->bb_template);
fail_free_meta:
- for (i = 0; i < PBLK_DATA_LINES; i++) {
- pblk_mfree(l_mg->sline_meta[i].meta, l_mg->smeta_alloc_type);
- pblk_mfree(l_mg->eline_meta[i].meta, l_mg->emeta_alloc_type);
- }
+ pblk_line_meta_free(pblk);
fail:
for (i = 0; i < geo->nr_luns; i++)
kfree(pblk->luns[i].bb_list);
static void pblk_writer_stop(struct pblk *pblk)
{
+ /* The pipeline must be stopped and the write buffer emptied before the
+ * write thread is stopped
+ */
+ WARN(pblk_rb_read_count(&pblk->rwb),
+ "Stopping not fully persisted write buffer\n");
+
+ WARN(pblk_rb_sync_count(&pblk->rwb),
+ "Stopping not fully synced write buffer\n");
+
if (pblk->writer_ts)
kthread_stop(pblk->writer_ts);
del_timer(&pblk->wtimer);
static void pblk_tear_down(struct pblk *pblk)
{
- pblk_flush_writer(pblk);
+ pblk_pipeline_stop(pblk);
pblk_writer_stop(pblk);
pblk_rb_sync_l2p(&pblk->rwb);
- pblk_recov_pad(pblk);
pblk_rwb_free(pblk);
pblk_rl_free(&pblk->rl);
pblk->dev = dev;
pblk->disk = tdisk;
+ pblk->state = PBLK_STATE_RUNNING;
spin_lock_init(&pblk->trans_lock);
spin_lock_init(&pblk->lock);
atomic_long_set(&pblk->req_writes, 0);
atomic_long_set(&pblk->sub_writes, 0);
atomic_long_set(&pblk->sync_writes, 0);
- atomic_long_set(&pblk->compl_writes, 0);
atomic_long_set(&pblk->inflight_reads, 0);
+ atomic_long_set(&pblk->cache_reads, 0);
atomic_long_set(&pblk->sync_reads, 0);
atomic_long_set(&pblk->recov_writes, 0);
atomic_long_set(&pblk->recov_writes, 0);
static int __init pblk_module_init(void)
{
- return nvm_register_tgt_type(&tt_pblk);
+ int ret;
+
+ pblk_bio_set = bioset_create(BIO_POOL_SIZE, 0, 0);
+ if (!pblk_bio_set)
+ return -ENOMEM;
+ ret = nvm_register_tgt_type(&tt_pblk);
+ if (ret)
+ bioset_free(pblk_bio_set);
+ return ret;
}
static void pblk_module_exit(void)
{
+ bioset_free(pblk_bio_set);
nvm_unregister_tgt_type(&tt_pblk);
}
unsigned int valid_secs)
{
struct pblk_line *line = pblk_line_get_data(pblk);
- struct line_emeta *emeta = line->emeta;
+ struct pblk_emeta *emeta = line->emeta;
struct pblk_w_ctx *w_ctx;
- __le64 *lba_list = pblk_line_emeta_to_lbas(emeta);
+ __le64 *lba_list = emeta_to_lbas(pblk, emeta->buf);
u64 paddr;
int nr_secs = pblk->min_write_pgs;
int i;
w_ctx->ppa = ppa_list[i];
meta_list[i].lba = cpu_to_le64(w_ctx->lba);
lba_list[paddr] = cpu_to_le64(w_ctx->lba);
- le64_add_cpu(&line->emeta->nr_valid_lbas, 1);
+ line->nr_valid_lbas++;
} else {
- meta_list[i].lba = cpu_to_le64(ADDR_EMPTY);
- lba_list[paddr] = cpu_to_le64(ADDR_EMPTY);
- pblk_map_pad_invalidate(pblk, line, paddr);
+ __le64 addr_empty = cpu_to_le64(ADDR_EMPTY);
+
+ lba_list[paddr] = meta_list[i].lba = addr_empty;
+ __pblk_map_invalidate(pblk, line, paddr);
}
}
if (pblk_line_is_full(line)) {
- line = pblk_line_replace_data(pblk);
- if (!line)
- return;
+ struct pblk_line *prev_line = line;
+
+ pblk_line_replace_data(pblk);
+ pblk_line_close_meta(pblk, prev_line);
}
pblk_down_rq(pblk, ppa_list, nr_secs, lun_bitmap);
{
struct nvm_tgt_dev *dev = pblk->dev;
struct nvm_geo *geo = &dev->geo;
- struct pblk_line *e_line = pblk_line_get_data_next(pblk);
+ struct pblk_line_meta *lm = &pblk->lm;
struct pblk_sec_meta *meta_list = rqd->meta_list;
+ struct pblk_line *e_line, *d_line;
unsigned int map_secs;
int min = pblk->min_write_pgs;
int i, erase_lun;
pblk_map_page_data(pblk, sentry + i, &rqd->ppa_list[i],
lun_bitmap, &meta_list[i], map_secs);
- erase_lun = rqd->ppa_list[i].g.lun * geo->nr_chnls +
- rqd->ppa_list[i].g.ch;
+ erase_lun = pblk_ppa_to_pos(geo, rqd->ppa_list[i]);
- if (!test_bit(erase_lun, e_line->erase_bitmap)) {
- if (down_trylock(&pblk->erase_sem))
- continue;
+ /* line can change after page map. We might also be writing the
+ * last line.
+ */
+ e_line = pblk_line_get_erase(pblk);
+ if (!e_line)
+ return pblk_map_rq(pblk, rqd, sentry, lun_bitmap,
+ valid_secs, i + min);
+ spin_lock(&e_line->lock);
+ if (!test_bit(erase_lun, e_line->erase_bitmap)) {
set_bit(erase_lun, e_line->erase_bitmap);
atomic_dec(&e_line->left_eblks);
+
*erase_ppa = rqd->ppa_list[i];
erase_ppa->g.blk = e_line->id;
+ spin_unlock(&e_line->lock);
+
/* Avoid evaluating e_line->left_eblks */
return pblk_map_rq(pblk, rqd, sentry, lun_bitmap,
valid_secs, i + min);
}
+ spin_unlock(&e_line->lock);
}
- /* Erase blocks that are bad in this line but might not be in next */
- if (unlikely(ppa_empty(*erase_ppa))) {
- struct pblk_line_meta *lm = &pblk->lm;
+ d_line = pblk_line_get_data(pblk);
+
+ /* line can change after page map. We might also be writing the
+ * last line.
+ */
+ e_line = pblk_line_get_erase(pblk);
+ if (!e_line)
+ return;
- i = find_first_zero_bit(e_line->erase_bitmap, lm->blk_per_line);
- if (i == lm->blk_per_line)
+ /* Erase blocks that are bad in this line but might not be in next */
+ if (unlikely(ppa_empty(*erase_ppa)) &&
+ bitmap_weight(d_line->blk_bitmap, lm->blk_per_line)) {
+ int bit = -1;
+
+retry:
+ bit = find_next_bit(d_line->blk_bitmap,
+ lm->blk_per_line, bit + 1);
+ if (bit >= lm->blk_per_line)
return;
- set_bit(i, e_line->erase_bitmap);
+ spin_lock(&e_line->lock);
+ if (test_bit(bit, e_line->erase_bitmap)) {
+ spin_unlock(&e_line->lock);
+ goto retry;
+ }
+ spin_unlock(&e_line->lock);
+
+ set_bit(bit, e_line->erase_bitmap);
atomic_dec(&e_line->left_eblks);
- *erase_ppa = pblk->luns[i].bppa; /* set ch and lun */
+ *erase_ppa = pblk->luns[bit].bppa; /* set ch and lun */
erase_ppa->g.blk = e_line->id;
}
}
/* Release flags on context. Protect from writes and reads */
smp_store_release(&w_ctx->flags, PBLK_WRITABLE_ENTRY);
pblk_ppa_set_empty(&w_ctx->ppa);
+ w_ctx->lba = ADDR_EMPTY;
}
#define pblk_rb_ring_count(head, tail, size) CIRC_CNT(head, tail, size)
return pblk_rb_ring_count(mem, subm, rb->nr_entries);
}
+unsigned int pblk_rb_sync_count(struct pblk_rb *rb)
+{
+ unsigned int mem = READ_ONCE(rb->mem);
+ unsigned int sync = READ_ONCE(rb->sync);
+
+ return pblk_rb_ring_count(mem, sync, rb->nr_entries);
+}
+
unsigned int pblk_rb_read_commit(struct pblk_rb *rb, unsigned int nr_entries)
{
unsigned int subm;
struct pblk_line *line;
struct pblk_rb_entry *entry;
struct pblk_w_ctx *w_ctx;
+ unsigned int user_io = 0, gc_io = 0;
unsigned int i;
+ int flags;
for (i = 0; i < to_update; i++) {
entry = &rb->entries[*l2p_upd];
w_ctx = &entry->w_ctx;
+ flags = READ_ONCE(entry->w_ctx.flags);
+ if (flags & PBLK_IOTYPE_USER)
+ user_io++;
+ else if (flags & PBLK_IOTYPE_GC)
+ gc_io++;
+ else
+ WARN(1, "pblk: unknown IO type\n");
+
pblk_update_map_dev(pblk, w_ctx->lba, w_ctx->ppa,
entry->cacheline);
*l2p_upd = (*l2p_upd + 1) & (rb->nr_entries - 1);
}
+ pblk_rl_out(&pblk->rl, user_io, gc_io);
+
return 0;
}
/* Protect syncs */
smp_store_release(&rb->sync_point, sync_point);
+ if (!bio)
+ return 0;
+
spin_lock_irq(&rb->s_lock);
bio_list_add(&entry->w_ctx.bios, bio);
spin_unlock_irq(&rb->s_lock);
return 1;
}
+void pblk_rb_flush(struct pblk_rb *rb)
+{
+ struct pblk *pblk = container_of(rb, struct pblk, rwb);
+ unsigned int mem = READ_ONCE(rb->mem);
+
+ if (pblk_rb_sync_point_set(rb, NULL, mem))
+ return;
+
+ pblk_write_should_kick(pblk);
+}
+
static int pblk_rb_may_write_flush(struct pblk_rb *rb, unsigned int nr_entries,
unsigned int *pos, struct bio *bio,
int *io_ret)
unsigned int nr_entries, unsigned int *pos)
{
struct pblk *pblk = container_of(rb, struct pblk, rwb);
- int flush_done;
+ int io_ret;
spin_lock(&rb->w_lock);
- if (!pblk_rl_user_may_insert(&pblk->rl, nr_entries)) {
+ io_ret = pblk_rl_user_may_insert(&pblk->rl, nr_entries);
+ if (io_ret) {
spin_unlock(&rb->w_lock);
- return NVM_IO_REQUEUE;
+ return io_ret;
}
- if (!pblk_rb_may_write_flush(rb, nr_entries, pos, bio, &flush_done)) {
+ if (!pblk_rb_may_write_flush(rb, nr_entries, pos, bio, &io_ret)) {
spin_unlock(&rb->w_lock);
return NVM_IO_REQUEUE;
}
pblk_rl_user_in(&pblk->rl, nr_entries);
spin_unlock(&rb->w_lock);
- return flush_done;
+ return io_ret;
}
/*
* This function is used by the write thread to form the write bio that will
* persist data on the write buffer to the media.
*/
-unsigned int pblk_rb_read_to_bio(struct pblk_rb *rb, struct bio *bio,
- struct pblk_c_ctx *c_ctx,
- unsigned int pos,
- unsigned int nr_entries,
- unsigned int count)
+unsigned int pblk_rb_read_to_bio(struct pblk_rb *rb, struct nvm_rq *rqd,
+ struct bio *bio, unsigned int pos,
+ unsigned int nr_entries, unsigned int count)
{
struct pblk *pblk = container_of(rb, struct pblk, rwb);
+ struct request_queue *q = pblk->dev->q;
+ struct pblk_c_ctx *c_ctx = nvm_rq_to_pdu(rqd);
struct pblk_rb_entry *entry;
struct page *page;
- unsigned int pad = 0, read = 0, to_read = nr_entries;
- unsigned int user_io = 0, gc_io = 0;
+ unsigned int pad = 0, to_read = nr_entries;
unsigned int i;
int flags;
- int ret;
if (count < nr_entries) {
pad = nr_entries - count;
*/
try:
flags = READ_ONCE(entry->w_ctx.flags);
- if (!(flags & PBLK_WRITTEN_DATA))
+ if (!(flags & PBLK_WRITTEN_DATA)) {
+ io_schedule();
goto try;
-
- if (flags & PBLK_IOTYPE_USER)
- user_io++;
- else if (flags & PBLK_IOTYPE_GC)
- gc_io++;
- else
- WARN(1, "pblk: unknown IO type\n");
+ }
page = virt_to_page(entry->data);
if (!page) {
flags |= PBLK_SUBMITTED_ENTRY;
/* Release flags on context. Protect from writes */
smp_store_release(&entry->w_ctx.flags, flags);
- goto out;
+ return NVM_IO_ERR;
}
- ret = bio_add_page(bio, page, rb->seg_size, 0);
- if (ret != rb->seg_size) {
+ if (bio_add_pc_page(q, bio, page, rb->seg_size, 0) !=
+ rb->seg_size) {
pr_err("pblk: could not add page to write bio\n");
flags &= ~PBLK_WRITTEN_DATA;
flags |= PBLK_SUBMITTED_ENTRY;
/* Release flags on context. Protect from writes */
smp_store_release(&entry->w_ctx.flags, flags);
- goto out;
+ return NVM_IO_ERR;
}
if (flags & PBLK_FLUSH_ENTRY) {
pos = (pos + 1) & (rb->nr_entries - 1);
}
- read = to_read;
- pblk_rl_out(&pblk->rl, user_io, gc_io);
+ if (pad) {
+ if (pblk_bio_add_pages(pblk, bio, GFP_KERNEL, pad)) {
+ pr_err("pblk: could not pad page in write bio\n");
+ return NVM_IO_ERR;
+ }
+ }
+
#ifdef CONFIG_NVM_DEBUG
atomic_long_add(pad, &((struct pblk *)
(container_of(rb, struct pblk, rwb)))->padded_writes);
#endif
-out:
- return read;
+
+ return NVM_IO_OK;
}
/*
* be directed to disk.
*/
int pblk_rb_copy_to_bio(struct pblk_rb *rb, struct bio *bio, sector_t lba,
- u64 pos, int bio_iter)
+ struct ppa_addr ppa, int bio_iter)
{
+ struct pblk *pblk = container_of(rb, struct pblk, rwb);
struct pblk_rb_entry *entry;
struct pblk_w_ctx *w_ctx;
+ struct ppa_addr l2p_ppa;
+ u64 pos = pblk_addr_to_cacheline(ppa);
void *data;
int flags;
int ret = 1;
- spin_lock(&rb->w_lock);
#ifdef CONFIG_NVM_DEBUG
/* Caller must ensure that the access will not cause an overflow */
w_ctx = &entry->w_ctx;
flags = READ_ONCE(w_ctx->flags);
+ spin_lock(&rb->w_lock);
+ spin_lock(&pblk->trans_lock);
+ l2p_ppa = pblk_trans_map_get(pblk, lba);
+ spin_unlock(&pblk->trans_lock);
+
/* Check if the entry has been overwritten or is scheduled to be */
- if (w_ctx->lba != lba || flags & PBLK_WRITABLE_ENTRY) {
+ if (!pblk_ppa_comp(l2p_ppa, ppa) || w_ctx->lba != lba ||
+ flags & PBLK_WRITABLE_ENTRY) {
ret = 0;
goto out;
}
BUG_ON(!pblk_addr_in_cache(ppa));
#endif
- return pblk_rb_copy_to_bio(&pblk->rwb, bio, lba,
- pblk_addr_to_cacheline(ppa), bio_iter);
+ return pblk_rb_copy_to_bio(&pblk->rwb, bio, lba, ppa, bio_iter);
}
static void pblk_read_ppalist_rq(struct pblk *pblk, struct nvm_rq *rqd,
}
WARN_ON(test_and_set_bit(i, read_bitmap));
advanced_bio = 1;
+#ifdef CONFIG_NVM_DEBUG
+ atomic_long_inc(&pblk->cache_reads);
+#endif
} else {
/* Read from media non-cached sectors */
rqd->ppa_list[j++] = p;
bio_advance(bio, PBLK_EXPOSED_PAGE_SIZE);
}
+ if (pblk_io_aligned(pblk, nr_secs))
+ rqd->flags = pblk_set_read_mode(pblk, PBLK_READ_SEQUENTIAL);
+ else
+ rqd->flags = pblk_set_read_mode(pblk, PBLK_READ_RANDOM);
+
#ifdef CONFIG_NVM_DEBUG
atomic_long_add(nr_secs, &pblk->inflight_reads);
#endif
{
int err;
- rqd->flags = pblk_set_read_mode(pblk);
-
err = pblk_submit_io(pblk, rqd);
if (err)
return NVM_IO_ERR;
{
struct pblk *pblk = rqd->private;
struct nvm_tgt_dev *dev = pblk->dev;
- struct pblk_r_ctx *r_ctx = nvm_rq_to_pdu(rqd);
+ struct pblk_g_ctx *r_ctx = nvm_rq_to_pdu(rqd);
struct bio *bio = rqd->bio;
if (rqd->error)
pblk_log_read_err(pblk, rqd);
#ifdef CONFIG_NVM_DEBUG
else
- WARN_ONCE(bio->bi_error, "pblk: corrupted read error\n");
+ WARN_ONCE(bio->bi_status, "pblk: corrupted read error\n");
#endif
- if (rqd->nr_ppas > 1)
- nvm_dev_dma_free(dev->parent, rqd->ppa_list, rqd->dma_ppa_list);
+ nvm_dev_dma_free(dev->parent, rqd->meta_list, rqd->dma_meta_list);
bio_put(bio);
- if (r_ctx->orig_bio) {
+ if (r_ctx->private) {
+ struct bio *orig_bio = r_ctx->private;
+
#ifdef CONFIG_NVM_DEBUG
- WARN_ONCE(r_ctx->orig_bio->bi_error,
- "pblk: corrupted read bio\n");
+ WARN_ONCE(orig_bio->bi_status, "pblk: corrupted read bio\n");
#endif
- bio_endio(r_ctx->orig_bio);
- bio_put(r_ctx->orig_bio);
+ bio_endio(orig_bio);
+ bio_put(orig_bio);
}
#ifdef CONFIG_NVM_DEBUG
#endif
pblk_free_rqd(pblk, rqd, READ);
+ atomic_dec(&pblk->inflight_io);
}
static int pblk_fill_partial_read_bio(struct pblk *pblk, struct nvm_rq *rqd,
rqd->bio = new_bio;
rqd->nr_ppas = nr_holes;
+ rqd->flags = pblk_set_read_mode(pblk, PBLK_READ_RANDOM);
rqd->end_io = NULL;
if (unlikely(nr_secs > 1 && nr_holes == 1)) {
goto retry;
}
WARN_ON(test_and_set_bit(0, read_bitmap));
+#ifdef CONFIG_NVM_DEBUG
+ atomic_long_inc(&pblk->cache_reads);
+#endif
} else {
rqd->ppa_addr = ppa;
}
+
+ rqd->flags = pblk_set_read_mode(pblk, PBLK_READ_RANDOM);
}
int pblk_submit_read(struct pblk *pblk, struct bio *bio)
*/
bio_init_idx = pblk_get_bi_idx(bio);
+ rqd->meta_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL,
+ &rqd->dma_meta_list);
+ if (!rqd->meta_list) {
+ pr_err("pblk: not able to allocate ppa list\n");
+ goto fail_rqd_free;
+ }
+
if (nr_secs > 1) {
- rqd->ppa_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL,
- &rqd->dma_ppa_list);
- if (!rqd->ppa_list) {
- pr_err("pblk: not able to allocate ppa list\n");
- goto fail_rqd_free;
- }
+ rqd->ppa_list = rqd->meta_list + pblk_dma_meta_size;
+ rqd->dma_ppa_list = rqd->dma_meta_list + pblk_dma_meta_size;
pblk_read_ppalist_rq(pblk, rqd, &read_bitmap);
} else {
bio_get(bio);
if (bitmap_full(&read_bitmap, nr_secs)) {
bio_endio(bio);
+ atomic_inc(&pblk->inflight_io);
pblk_end_io_read(rqd);
return NVM_IO_OK;
}
/* All sectors are to be read from the device */
if (bitmap_empty(&read_bitmap, rqd->nr_ppas)) {
struct bio *int_bio = NULL;
- struct pblk_r_ctx *r_ctx = nvm_rq_to_pdu(rqd);
+ struct pblk_g_ctx *r_ctx = nvm_rq_to_pdu(rqd);
/* Clone read bio to deal with read errors internally */
- int_bio = bio_clone_bioset(bio, GFP_KERNEL, fs_bio_set);
+ int_bio = bio_clone_fast(bio, GFP_KERNEL, pblk_bio_set);
if (!int_bio) {
pr_err("pblk: could not clone read bio\n");
return NVM_IO_ERR;
}
rqd->bio = int_bio;
- r_ctx->orig_bio = bio;
+ r_ctx->private = bio;
ret = pblk_submit_read_io(pblk, rqd);
if (ret) {
{
struct nvm_tgt_dev *dev = pblk->dev;
struct nvm_geo *geo = &dev->geo;
- struct request_queue *q = dev->q;
struct bio *bio;
struct nvm_rq rqd;
int ret, data_len;
memset(&rqd, 0, sizeof(struct nvm_rq));
+ rqd.meta_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL,
+ &rqd.dma_meta_list);
+ if (!rqd.meta_list)
+ return NVM_IO_ERR;
+
if (nr_secs > 1) {
- rqd.ppa_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL,
- &rqd.dma_ppa_list);
- if (!rqd.ppa_list)
- return NVM_IO_ERR;
+ rqd.ppa_list = rqd.meta_list + pblk_dma_meta_size;
+ rqd.dma_ppa_list = rqd.dma_meta_list + pblk_dma_meta_size;
*secs_to_gc = read_ppalist_rq_gc(pblk, &rqd, line, lba_list,
nr_secs);
- if (*secs_to_gc == 1) {
- struct ppa_addr ppa;
-
- ppa = rqd.ppa_list[0];
- nvm_dev_dma_free(dev->parent, rqd.ppa_list,
- rqd.dma_ppa_list);
- rqd.ppa_addr = ppa;
- }
+ if (*secs_to_gc == 1)
+ rqd.ppa_addr = rqd.ppa_list[0];
} else {
*secs_to_gc = read_rq_gc(pblk, &rqd, line, lba_list[0]);
}
goto out;
data_len = (*secs_to_gc) * geo->sec_size;
- bio = bio_map_kern(q, data, data_len, GFP_KERNEL);
+ bio = pblk_bio_map_addr(pblk, data, *secs_to_gc, data_len,
+ PBLK_KMALLOC_META, GFP_KERNEL);
if (IS_ERR(bio)) {
pr_err("pblk: could not allocate GC bio (%lu)\n", PTR_ERR(bio));
goto err_free_dma;
rqd.end_io = pblk_end_io_sync;
rqd.private = &wait;
rqd.nr_ppas = *secs_to_gc;
+ rqd.flags = pblk_set_read_mode(pblk, PBLK_READ_RANDOM);
rqd.bio = bio;
ret = pblk_submit_read_io(pblk, &rqd);
msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
pr_err("pblk: GC read I/O timed out\n");
}
+ atomic_dec(&pblk->inflight_io);
if (rqd.error) {
atomic_long_inc(&pblk->read_failed_gc);
#endif
out:
- if (rqd.nr_ppas > 1)
- nvm_dev_dma_free(dev->parent, rqd.ppa_list, rqd.dma_ppa_list);
+ nvm_dev_dma_free(dev->parent, rqd.meta_list, rqd.dma_meta_list);
return NVM_IO_OK;
err_free_dma:
- if (rqd.nr_ppas > 1)
- nvm_dev_dma_free(dev->parent, rqd.ppa_list, rqd.dma_ppa_list);
+ nvm_dev_dma_free(dev->parent, rqd.meta_list, rqd.dma_meta_list);
return NVM_IO_ERR;
}
return 0;
}
-__le64 *pblk_recov_get_lba_list(struct pblk *pblk, struct line_emeta *emeta)
+__le64 *pblk_recov_get_lba_list(struct pblk *pblk, struct line_emeta *emeta_buf)
{
u32 crc;
- crc = pblk_calc_emeta_crc(pblk, emeta);
- if (le32_to_cpu(emeta->crc) != crc)
+ crc = pblk_calc_emeta_crc(pblk, emeta_buf);
+ if (le32_to_cpu(emeta_buf->crc) != crc)
return NULL;
- if (le32_to_cpu(emeta->header.identifier) != PBLK_MAGIC)
+ if (le32_to_cpu(emeta_buf->header.identifier) != PBLK_MAGIC)
return NULL;
- return pblk_line_emeta_to_lbas(emeta);
+ return emeta_to_lbas(pblk, emeta_buf);
}
static int pblk_recov_l2p_from_emeta(struct pblk *pblk, struct pblk_line *line)
struct nvm_tgt_dev *dev = pblk->dev;
struct nvm_geo *geo = &dev->geo;
struct pblk_line_meta *lm = &pblk->lm;
- struct line_emeta *emeta = line->emeta;
+ struct pblk_emeta *emeta = line->emeta;
+ struct line_emeta *emeta_buf = emeta->buf;
__le64 *lba_list;
int data_start;
int nr_data_lbas, nr_valid_lbas, nr_lbas = 0;
int i;
- lba_list = pblk_recov_get_lba_list(pblk, emeta);
+ lba_list = pblk_recov_get_lba_list(pblk, emeta_buf);
if (!lba_list)
return 1;
data_start = pblk_line_smeta_start(pblk, line) + lm->smeta_sec;
- nr_data_lbas = lm->sec_per_line - lm->emeta_sec;
- nr_valid_lbas = le64_to_cpu(emeta->nr_valid_lbas);
+ nr_data_lbas = lm->sec_per_line - lm->emeta_sec[0];
+ nr_valid_lbas = le64_to_cpu(emeta_buf->nr_valid_lbas);
for (i = data_start; i < nr_data_lbas && nr_lbas < nr_valid_lbas; i++) {
struct ppa_addr ppa;
if (test_and_set_bit(i, line->invalid_bitmap))
WARN_ONCE(1, "pblk: rec. double invalidate:\n");
else
- line->vsc--;
+ le32_add_cpu(line->vsc, -1);
spin_unlock(&line->lock);
continue;
if (nr_valid_lbas != nr_lbas)
pr_err("pblk: line %d - inconsistent lba list(%llu/%d)\n",
- line->id, line->emeta->nr_valid_lbas, nr_lbas);
+ line->id, emeta_buf->nr_valid_lbas, nr_lbas);
line->left_msecs = 0;
struct pblk_line_meta *lm = &pblk->lm;
int nr_bb = bitmap_weight(line->blk_bitmap, lm->blk_per_line);
- return lm->sec_per_line - lm->smeta_sec - lm->emeta_sec -
+ return lm->sec_per_line - lm->smeta_sec - lm->emeta_sec[0] -
nr_bb * geo->sec_per_blk;
}
r_ptr_int = r_ptr;
next_read_rq:
- memset(rqd, 0, pblk_r_rq_size);
+ memset(rqd, 0, pblk_g_rq_size);
rq_ppas = pblk_calc_secs(pblk, left_ppas, 0);
if (!rq_ppas)
rqd->bio = bio;
rqd->opcode = NVM_OP_PREAD;
- rqd->flags = pblk_set_read_mode(pblk);
rqd->meta_list = meta_list;
rqd->nr_ppas = rq_ppas;
rqd->ppa_list = ppa_list;
rqd->end_io = pblk_end_io_sync;
rqd->private = &wait;
+ if (pblk_io_aligned(pblk, rq_ppas))
+ rqd->flags = pblk_set_read_mode(pblk, PBLK_READ_SEQUENTIAL);
+ else
+ rqd->flags = pblk_set_read_mode(pblk, PBLK_READ_RANDOM);
+
for (i = 0; i < rqd->nr_ppas; ) {
struct ppa_addr ppa;
int pos;
pr_err("pblk: L2P recovery read timed out\n");
return -EINTR;
}
-
+ atomic_dec(&pblk->inflight_io);
reinit_completion(&wait);
/* At this point, the read should not fail. If it does, it is a problem
return 0;
}
+static void pblk_recov_complete(struct kref *ref)
+{
+ struct pblk_pad_rq *pad_rq = container_of(ref, struct pblk_pad_rq, ref);
+
+ complete(&pad_rq->wait);
+}
+
+static void pblk_end_io_recov(struct nvm_rq *rqd)
+{
+ struct pblk_pad_rq *pad_rq = rqd->private;
+ struct pblk *pblk = pad_rq->pblk;
+ struct nvm_tgt_dev *dev = pblk->dev;
+
+ kref_put(&pad_rq->ref, pblk_recov_complete);
+ nvm_dev_dma_free(dev->parent, rqd->meta_list, rqd->dma_meta_list);
+ pblk_free_rqd(pblk, rqd, WRITE);
+}
+
static int pblk_recov_pad_oob(struct pblk *pblk, struct pblk_line *line,
- struct pblk_recov_alloc p, int left_ppas)
+ int left_ppas)
{
struct nvm_tgt_dev *dev = pblk->dev;
struct nvm_geo *geo = &dev->geo;
struct ppa_addr *ppa_list;
struct pblk_sec_meta *meta_list;
+ struct pblk_pad_rq *pad_rq;
struct nvm_rq *rqd;
struct bio *bio;
void *data;
dma_addr_t dma_ppa_list, dma_meta_list;
- __le64 *lba_list = pblk_line_emeta_to_lbas(line->emeta);
+ __le64 *lba_list = emeta_to_lbas(pblk, line->emeta->buf);
u64 w_ptr = line->cur_sec;
- int left_line_ppas = line->left_msecs;
- int rq_ppas, rq_len;
+ int left_line_ppas, rq_ppas, rq_len;
int i, j;
int ret = 0;
- DECLARE_COMPLETION_ONSTACK(wait);
- ppa_list = p.ppa_list;
- meta_list = p.meta_list;
- rqd = p.rqd;
- data = p.data;
- dma_ppa_list = p.dma_ppa_list;
- dma_meta_list = p.dma_meta_list;
+ spin_lock(&line->lock);
+ left_line_ppas = line->left_msecs;
+ spin_unlock(&line->lock);
+
+ pad_rq = kmalloc(sizeof(struct pblk_pad_rq), GFP_KERNEL);
+ if (!pad_rq)
+ return -ENOMEM;
+
+ data = vzalloc(pblk->max_write_pgs * geo->sec_size);
+ if (!data) {
+ ret = -ENOMEM;
+ goto free_rq;
+ }
+
+ pad_rq->pblk = pblk;
+ init_completion(&pad_rq->wait);
+ kref_init(&pad_rq->ref);
next_pad_rq:
rq_ppas = pblk_calc_secs(pblk, left_ppas, 0);
- if (!rq_ppas)
- rq_ppas = pblk->min_write_pgs;
+ if (rq_ppas < pblk->min_write_pgs) {
+ pr_err("pblk: corrupted pad line %d\n", line->id);
+ goto free_rq;
+ }
+
rq_len = rq_ppas * geo->sec_size;
+ meta_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL, &dma_meta_list);
+ if (!meta_list) {
+ ret = -ENOMEM;
+ goto free_data;
+ }
+
+ ppa_list = (void *)(meta_list) + pblk_dma_meta_size;
+ dma_ppa_list = dma_meta_list + pblk_dma_meta_size;
+
+ rqd = pblk_alloc_rqd(pblk, WRITE);
+ if (IS_ERR(rqd)) {
+ ret = PTR_ERR(rqd);
+ goto fail_free_meta;
+ }
+ memset(rqd, 0, pblk_w_rq_size);
+
bio = bio_map_kern(dev->q, data, rq_len, GFP_KERNEL);
- if (IS_ERR(bio))
- return PTR_ERR(bio);
+ if (IS_ERR(bio)) {
+ ret = PTR_ERR(bio);
+ goto fail_free_rqd;
+ }
bio->bi_iter.bi_sector = 0; /* internal bio */
bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
- memset(rqd, 0, pblk_r_rq_size);
-
rqd->bio = bio;
rqd->opcode = NVM_OP_PWRITE;
rqd->flags = pblk_set_progr_mode(pblk, WRITE);
rqd->ppa_list = ppa_list;
rqd->dma_ppa_list = dma_ppa_list;
rqd->dma_meta_list = dma_meta_list;
- rqd->end_io = pblk_end_io_sync;
- rqd->private = &wait;
+ rqd->end_io = pblk_end_io_recov;
+ rqd->private = pad_rq;
for (i = 0; i < rqd->nr_ppas; ) {
struct ppa_addr ppa;
for (j = 0; j < pblk->min_write_pgs; j++, i++, w_ptr++) {
struct ppa_addr dev_ppa;
+ __le64 addr_empty = cpu_to_le64(ADDR_EMPTY);
dev_ppa = addr_to_gen_ppa(pblk, w_ptr, line->id);
pblk_map_invalidate(pblk, dev_ppa);
- meta_list[i].lba = cpu_to_le64(ADDR_EMPTY);
- lba_list[w_ptr] = cpu_to_le64(ADDR_EMPTY);
+ lba_list[w_ptr] = meta_list[i].lba = addr_empty;
rqd->ppa_list[i] = dev_ppa;
}
}
+ kref_get(&pad_rq->ref);
+
ret = pblk_submit_io(pblk, rqd);
if (ret) {
pr_err("pblk: I/O submission failed: %d\n", ret);
- return ret;
+ goto free_data;
}
- if (!wait_for_completion_io_timeout(&wait,
- msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
- pr_err("pblk: L2P recovery write timed out\n");
- }
- reinit_completion(&wait);
+ atomic_dec(&pblk->inflight_io);
left_line_ppas -= rq_ppas;
left_ppas -= rq_ppas;
- if (left_ppas > 0 && left_line_ppas)
+ if (left_ppas && left_line_ppas)
goto next_pad_rq;
- return 0;
+ kref_put(&pad_rq->ref, pblk_recov_complete);
+
+ if (!wait_for_completion_io_timeout(&pad_rq->wait,
+ msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
+ pr_err("pblk: pad write timed out\n");
+ ret = -ETIME;
+ }
+
+free_rq:
+ kfree(pad_rq);
+free_data:
+ vfree(data);
+ return ret;
+
+fail_free_rqd:
+ pblk_free_rqd(pblk, rqd, WRITE);
+fail_free_meta:
+ nvm_dev_dma_free(dev->parent, meta_list, dma_meta_list);
+ kfree(pad_rq);
+ return ret;
}
/* When this function is called, it means that not all upper pages have been
rec_round = 0;
next_rq:
- memset(rqd, 0, pblk_r_rq_size);
+ memset(rqd, 0, pblk_g_rq_size);
rq_ppas = pblk_calc_secs(pblk, left_ppas, 0);
if (!rq_ppas)
rqd->bio = bio;
rqd->opcode = NVM_OP_PREAD;
- rqd->flags = pblk_set_read_mode(pblk);
rqd->meta_list = meta_list;
rqd->nr_ppas = rq_ppas;
rqd->ppa_list = ppa_list;
rqd->end_io = pblk_end_io_sync;
rqd->private = &wait;
+ if (pblk_io_aligned(pblk, rq_ppas))
+ rqd->flags = pblk_set_read_mode(pblk, PBLK_READ_SEQUENTIAL);
+ else
+ rqd->flags = pblk_set_read_mode(pblk, PBLK_READ_RANDOM);
+
for (i = 0; i < rqd->nr_ppas; ) {
struct ppa_addr ppa;
int pos;
msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
pr_err("pblk: L2P recovery read timed out\n");
}
+ atomic_dec(&pblk->inflight_io);
reinit_completion(&wait);
/* This should not happen since the read failed during normal recovery,
if (pad_secs > line->left_msecs)
pad_secs = line->left_msecs;
- ret = pblk_recov_pad_oob(pblk, line, p, pad_secs);
+ ret = pblk_recov_pad_oob(pblk, line, pad_secs);
if (ret)
pr_err("pblk: OOB padding failed (err:%d)\n", ret);
if (ret)
pr_err("pblk: OOB read failed (err:%d)\n", ret);
- line->left_ssecs = line->left_msecs;
left_ppas = 0;
}
*done = 1;
next_rq:
- memset(rqd, 0, pblk_r_rq_size);
+ memset(rqd, 0, pblk_g_rq_size);
rq_ppas = pblk_calc_secs(pblk, left_ppas, 0);
if (!rq_ppas)
rqd->bio = bio;
rqd->opcode = NVM_OP_PREAD;
- rqd->flags = pblk_set_read_mode(pblk);
rqd->meta_list = meta_list;
rqd->nr_ppas = rq_ppas;
rqd->ppa_list = ppa_list;
rqd->end_io = pblk_end_io_sync;
rqd->private = &wait;
+ if (pblk_io_aligned(pblk, rq_ppas))
+ rqd->flags = pblk_set_read_mode(pblk, PBLK_READ_SEQUENTIAL);
+ else
+ rqd->flags = pblk_set_read_mode(pblk, PBLK_READ_RANDOM);
+
for (i = 0; i < rqd->nr_ppas; ) {
struct ppa_addr ppa;
int pos;
msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
pr_err("pblk: L2P recovery read timed out\n");
}
+ atomic_dec(&pblk->inflight_io);
reinit_completion(&wait);
/* Reached the end of the written line */
/* Roll back failed sectors */
line->cur_sec -= nr_error_bits;
line->left_msecs += nr_error_bits;
- line->left_ssecs = line->left_msecs;
bitmap_clear(line->map_bitmap, line->cur_sec, nr_error_bits);
left_ppas = 0;
struct pblk_line_meta *lm = &pblk->lm;
struct pblk_line_mgmt *l_mg = &pblk->l_mg;
struct pblk_line *line, *tline, *data_line = NULL;
- struct line_smeta *smeta;
- struct line_emeta *emeta;
+ struct pblk_smeta *smeta;
+ struct pblk_emeta *emeta;
+ struct line_smeta *smeta_buf;
int found_lines = 0, recovered_lines = 0, open_lines = 0;
int is_next = 0;
int meta_line;
spin_lock(&l_mg->free_lock);
meta_line = find_first_zero_bit(&l_mg->meta_bitmap, PBLK_DATA_LINES);
set_bit(meta_line, &l_mg->meta_bitmap);
- smeta = l_mg->sline_meta[meta_line].meta;
- emeta = l_mg->eline_meta[meta_line].meta;
+ smeta = l_mg->sline_meta[meta_line];
+ emeta = l_mg->eline_meta[meta_line];
+ smeta_buf = (struct line_smeta *)smeta;
spin_unlock(&l_mg->free_lock);
/* Order data lines using their sequence number */
memset(smeta, 0, lm->smeta_len);
line->smeta = smeta;
- line->lun_bitmap = ((void *)(smeta)) +
+ line->lun_bitmap = ((void *)(smeta_buf)) +
sizeof(struct line_smeta);
/* Lines that cannot be read are assumed as not written here */
if (pblk_line_read_smeta(pblk, line))
continue;
- crc = pblk_calc_smeta_crc(pblk, smeta);
- if (le32_to_cpu(smeta->crc) != crc)
+ crc = pblk_calc_smeta_crc(pblk, smeta_buf);
+ if (le32_to_cpu(smeta_buf->crc) != crc)
continue;
- if (le32_to_cpu(smeta->header.identifier) != PBLK_MAGIC)
+ if (le32_to_cpu(smeta_buf->header.identifier) != PBLK_MAGIC)
continue;
- if (le16_to_cpu(smeta->header.version) != 1) {
+ if (le16_to_cpu(smeta_buf->header.version) != 1) {
pr_err("pblk: found incompatible line version %u\n",
- smeta->header.version);
+ smeta_buf->header.version);
return ERR_PTR(-EINVAL);
}
/* The first valid instance uuid is used for initialization */
if (!valid_uuid) {
- memcpy(pblk->instance_uuid, smeta->header.uuid, 16);
+ memcpy(pblk->instance_uuid, smeta_buf->header.uuid, 16);
valid_uuid = 1;
}
- if (memcmp(pblk->instance_uuid, smeta->header.uuid, 16)) {
+ if (memcmp(pblk->instance_uuid, smeta_buf->header.uuid, 16)) {
pr_debug("pblk: ignore line %u due to uuid mismatch\n",
i);
continue;
/* Update line metadata */
spin_lock(&line->lock);
- line->id = le32_to_cpu(line->smeta->header.id);
- line->type = le16_to_cpu(line->smeta->header.type);
- line->seq_nr = le64_to_cpu(line->smeta->seq_nr);
+ line->id = le32_to_cpu(smeta_buf->header.id);
+ line->type = le16_to_cpu(smeta_buf->header.type);
+ line->seq_nr = le64_to_cpu(smeta_buf->seq_nr);
spin_unlock(&line->lock);
/* Update general metadata */
pblk_recov_line_add_ordered(&recov_list, line);
found_lines++;
pr_debug("pblk: recovering data line %d, seq:%llu\n",
- line->id, smeta->seq_nr);
+ line->id, smeta_buf->seq_nr);
}
if (!found_lines) {
recovered_lines++;
/* Calculate where emeta starts based on the line bb */
- off = lm->sec_per_line - lm->emeta_sec;
+ off = lm->sec_per_line - lm->emeta_sec[0];
nr_bb = bitmap_weight(line->blk_bitmap, lm->blk_per_line);
off -= nr_bb * geo->sec_per_pl;
- memset(emeta, 0, lm->emeta_len);
- line->emeta = emeta;
line->emeta_ssec = off;
+ line->emeta = emeta;
+ memset(line->emeta->buf, 0, lm->emeta_len[0]);
- if (pblk_line_read_emeta(pblk, line)) {
+ if (pblk_line_read_emeta(pblk, line, line->emeta->buf)) {
pblk_recov_l2p_from_oob(pblk, line);
goto next;
}
}
/*
- * Pad until smeta can be read on current data line
+ * Pad current line
*/
-void pblk_recov_pad(struct pblk *pblk)
+int pblk_recov_pad(struct pblk *pblk)
{
- struct nvm_tgt_dev *dev = pblk->dev;
- struct nvm_geo *geo = &dev->geo;
struct pblk_line *line;
struct pblk_line_mgmt *l_mg = &pblk->l_mg;
- struct nvm_rq *rqd;
- struct pblk_recov_alloc p;
- struct ppa_addr *ppa_list;
- struct pblk_sec_meta *meta_list;
- void *data;
- dma_addr_t dma_ppa_list, dma_meta_list;
+ int left_msecs;
+ int ret = 0;
spin_lock(&l_mg->free_lock);
line = l_mg->data_line;
+ left_msecs = line->left_msecs;
spin_unlock(&l_mg->free_lock);
- rqd = pblk_alloc_rqd(pblk, READ);
- if (IS_ERR(rqd))
- return;
-
- meta_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL, &dma_meta_list);
- if (!meta_list)
- goto free_rqd;
-
- ppa_list = (void *)(meta_list) + pblk_dma_meta_size;
- dma_ppa_list = dma_meta_list + pblk_dma_meta_size;
-
- data = kcalloc(pblk->max_write_pgs, geo->sec_size, GFP_KERNEL);
- if (!data)
- goto free_meta_list;
-
- p.ppa_list = ppa_list;
- p.meta_list = meta_list;
- p.rqd = rqd;
- p.data = data;
- p.dma_ppa_list = dma_ppa_list;
- p.dma_meta_list = dma_meta_list;
-
- if (pblk_recov_pad_oob(pblk, line, p, line->left_msecs)) {
- pr_err("pblk: Tear down padding failed\n");
- goto free_data;
+ ret = pblk_recov_pad_oob(pblk, line, left_msecs);
+ if (ret) {
+ pr_err("pblk: Tear down padding failed (%d)\n", ret);
+ return ret;
}
- pblk_line_close(pblk, line);
-
-free_data:
- kfree(data);
-free_meta_list:
- nvm_dev_dma_free(dev->parent, meta_list, dma_meta_list);
-free_rqd:
- pblk_free_rqd(pblk, rqd, READ);
+ pblk_line_close_meta(pblk, line);
+ return ret;
}
mod_timer(&rl->u_timer, jiffies + msecs_to_jiffies(5000));
}
+int pblk_rl_is_limit(struct pblk_rl *rl)
+{
+ int rb_space;
+
+ rb_space = atomic_read(&rl->rb_space);
+
+ return (rb_space == 0);
+}
+
int pblk_rl_user_may_insert(struct pblk_rl *rl, int nr_entries)
{
int rb_user_cnt = atomic_read(&rl->rb_user_cnt);
+ int rb_space = atomic_read(&rl->rb_space);
- return (!(rb_user_cnt + nr_entries > rl->rb_user_max));
+ if (unlikely(rb_space >= 0) && (rb_space - nr_entries < 0))
+ return NVM_IO_ERR;
+
+ if (rb_user_cnt >= rl->rb_user_max)
+ return NVM_IO_REQUEUE;
+
+ return NVM_IO_OK;
+}
+
+void pblk_rl_inserted(struct pblk_rl *rl, int nr_entries)
+{
+ int rb_space = atomic_read(&rl->rb_space);
+
+ if (unlikely(rb_space >= 0))
+ atomic_sub(nr_entries, &rl->rb_space);
}
int pblk_rl_gc_may_insert(struct pblk_rl *rl, int nr_entries)
/* If there is no user I/O let GC take over space on the write buffer */
rb_user_active = READ_ONCE(rl->rb_user_active);
- return (!(rb_gc_cnt + nr_entries > rl->rb_gc_max && rb_user_active));
+ return (!(rb_gc_cnt >= rl->rb_gc_max && rb_user_active));
}
void pblk_rl_user_in(struct pblk_rl *rl, int nr_entries)
unsigned long free_blocks = pblk_rl_nr_free_blks(rl);
if (free_blocks >= rl->high) {
- rl->rb_user_max = max - rl->rb_gc_rsv;
- rl->rb_gc_max = rl->rb_gc_rsv;
+ rl->rb_user_max = max;
+ rl->rb_gc_max = 0;
rl->rb_state = PBLK_RL_HIGH;
} else if (free_blocks < rl->high) {
int shift = rl->high_pw - rl->rb_windows_pw;
int user_windows = free_blocks >> shift;
int user_max = user_windows << PBLK_MAX_REQ_ADDRS_PW;
- int gc_max;
rl->rb_user_max = user_max;
- gc_max = max - rl->rb_user_max;
- rl->rb_gc_max = max(gc_max, rl->rb_gc_rsv);
-
- if (free_blocks > rl->low)
- rl->rb_state = PBLK_RL_MID;
- else
- rl->rb_state = PBLK_RL_LOW;
+ rl->rb_gc_max = max - user_max;
+
+ if (free_blocks <= rl->rsv_blocks) {
+ rl->rb_user_max = 0;
+ rl->rb_gc_max = max;
+ }
+
+ /* In the worst case, we will need to GC lines in the low list
+ * (high valid sector count). If there are lines to GC on high
+ * or mid lists, these will be prioritized
+ */
+ rl->rb_state = PBLK_RL_LOW;
}
return rl->rb_state;
}
-void pblk_rl_set_gc_rsc(struct pblk_rl *rl, int rsv)
-{
- rl->rb_gc_rsv = rl->rb_gc_max = rsv;
-}
-
void pblk_rl_free_lines_inc(struct pblk_rl *rl, struct pblk_line *line)
{
struct pblk *pblk = container_of(rl, struct pblk, rl);
void pblk_rl_free_lines_dec(struct pblk_rl *rl, struct pblk_line *line)
{
- struct pblk *pblk = container_of(rl, struct pblk, rl);
int blk_in_line = atomic_read(&line->blk_in_line);
- int ret;
atomic_sub(blk_in_line, &rl->free_blocks);
+}
+
+void pblk_gc_should_kick(struct pblk *pblk)
+{
+ struct pblk_rl *rl = &pblk->rl;
+ int ret;
/* Rates will not change that often - no need to lock update */
ret = pblk_rl_update_rates(rl, rl->rb_budget);
pblk_gc_should_stop(pblk);
}
-int pblk_rl_gc_thrs(struct pblk_rl *rl)
+int pblk_rl_high_thrs(struct pblk_rl *rl)
{
return rl->high;
}
+int pblk_rl_low_thrs(struct pblk_rl *rl)
+{
+ return rl->low;
+}
+
int pblk_rl_sysfs_rate_show(struct pblk_rl *rl)
{
return rl->rb_user_max;
void pblk_rl_init(struct pblk_rl *rl, int budget)
{
+ struct pblk *pblk = container_of(rl, struct pblk, rl);
+ struct pblk_line_meta *lm = &pblk->lm;
+ int min_blocks = lm->blk_per_line * PBLK_GC_RSV_LINE;
unsigned int rb_windows;
rl->high = rl->total_blocks / PBLK_USER_HIGH_THRS;
- rl->low = rl->total_blocks / PBLK_USER_LOW_THRS;
rl->high_pw = get_count_order(rl->high);
+ rl->low = rl->total_blocks / PBLK_USER_LOW_THRS;
+ if (rl->low < min_blocks)
+ rl->low = min_blocks;
+
+ rl->rsv_blocks = min_blocks;
+
/* This will always be a power-of-2 */
rb_windows = budget / PBLK_MAX_REQ_ADDRS;
- rl->rb_windows_pw = get_count_order(rb_windows) + 1;
+ rl->rb_windows_pw = get_count_order(rb_windows);
/* To start with, all buffer is available to user I/O writers */
rl->rb_budget = budget;
rl->rb_user_max = budget;
- atomic_set(&rl->rb_user_cnt, 0);
rl->rb_gc_max = 0;
rl->rb_state = PBLK_RL_HIGH;
+
+ atomic_set(&rl->rb_user_cnt, 0);
atomic_set(&rl->rb_gc_cnt, 0);
+ atomic_set(&rl->rb_space, -1);
setup_timer(&rl->u_timer, pblk_rl_u_timer, (unsigned long)rl);
+
rl->rb_user_active = 0;
+ rl->rb_gc_active = 0;
}
static ssize_t pblk_sysfs_rate_limiter(struct pblk *pblk, char *page)
{
- struct nvm_tgt_dev *dev = pblk->dev;
- struct nvm_geo *geo = &dev->geo;
int free_blocks, total_blocks;
int rb_user_max, rb_user_cnt;
- int rb_gc_max, rb_gc_rsv, rb_gc_cnt, rb_budget, rb_state;
+ int rb_gc_max, rb_gc_cnt, rb_budget, rb_state;
free_blocks = atomic_read(&pblk->rl.free_blocks);
rb_user_max = pblk->rl.rb_user_max;
rb_user_cnt = atomic_read(&pblk->rl.rb_user_cnt);
rb_gc_max = pblk->rl.rb_gc_max;
- rb_gc_rsv = pblk->rl.rb_gc_rsv;
rb_gc_cnt = atomic_read(&pblk->rl.rb_gc_cnt);
rb_budget = pblk->rl.rb_budget;
rb_state = pblk->rl.rb_state;
- total_blocks = geo->blks_per_lun * geo->nr_luns;
+ total_blocks = pblk->rl.total_blocks;
return snprintf(page, PAGE_SIZE,
- "u:%u/%u,gc:%u/%u/%u(%u/%u)(stop:<%u,full:>%u,free:%d/%d)-%d\n",
+ "u:%u/%u,gc:%u/%u(%u/%u)(stop:<%u,full:>%u,free:%d/%d)-%d\n",
rb_user_cnt,
rb_user_max,
rb_gc_cnt,
rb_gc_max,
- rb_gc_rsv,
rb_state,
rb_budget,
pblk->rl.low,
ssize_t sz = 0;
int nr_free_lines;
int cur_data, cur_log;
- int free_line_cnt = 0, closed_line_cnt = 0;
+ int free_line_cnt = 0, closed_line_cnt = 0, emeta_line_cnt = 0;
int d_line_cnt = 0, l_line_cnt = 0;
int gc_full = 0, gc_high = 0, gc_mid = 0, gc_low = 0, gc_empty = 0;
- int free = 0, bad = 0, cor = 0;
- int msecs = 0, ssecs = 0, cur_sec = 0, vsc = 0, sec_in_line = 0;
+ int bad = 0, cor = 0;
+ int msecs = 0, cur_sec = 0, vsc = 0, sec_in_line = 0;
int map_weight = 0, meta_weight = 0;
spin_lock(&l_mg->free_lock);
free_line_cnt++;
spin_unlock(&l_mg->free_lock);
+ spin_lock(&l_mg->close_lock);
+ list_for_each_entry(line, &l_mg->emeta_list, list)
+ emeta_line_cnt++;
+ spin_unlock(&l_mg->close_lock);
+
spin_lock(&l_mg->gc_lock);
list_for_each_entry(line, &l_mg->gc_full_list, list) {
if (line->type == PBLK_LINETYPE_DATA)
gc_empty++;
}
- list_for_each_entry(line, &l_mg->free_list, list)
- free++;
list_for_each_entry(line, &l_mg->bad_list, list)
bad++;
list_for_each_entry(line, &l_mg->corrupt_list, list)
if (l_mg->data_line) {
cur_sec = l_mg->data_line->cur_sec;
msecs = l_mg->data_line->left_msecs;
- ssecs = l_mg->data_line->left_ssecs;
- vsc = l_mg->data_line->vsc;
+ vsc = le32_to_cpu(*l_mg->data_line->vsc);
sec_in_line = l_mg->data_line->sec_in_line;
meta_weight = bitmap_weight(&l_mg->meta_bitmap,
PBLK_DATA_LINES);
spin_unlock(&l_mg->free_lock);
if (nr_free_lines != free_line_cnt)
- pr_err("pblk: corrupted free line list\n");
+ pr_err("pblk: corrupted free line list:%d/%d\n",
+ nr_free_lines, free_line_cnt);
sz = snprintf(page, PAGE_SIZE - sz,
"line: nluns:%d, nblks:%d, nsecs:%d\n",
geo->nr_luns, lm->blk_per_line, lm->sec_per_line);
sz += snprintf(page + sz, PAGE_SIZE - sz,
- "lines:d:%d,l:%d-f:%d(%d),b:%d,co:%d,c:%d(d:%d,l:%d)t:%d\n",
+ "lines:d:%d,l:%d-f:%d,m:%d/%d,c:%d,b:%d,co:%d(d:%d,l:%d)t:%d\n",
cur_data, cur_log,
- free, nr_free_lines, bad, cor,
+ nr_free_lines,
+ emeta_line_cnt, meta_weight,
closed_line_cnt,
+ bad, cor,
d_line_cnt, l_line_cnt,
l_mg->nr_lines);
atomic_read(&pblk->gc.inflight_gc));
sz += snprintf(page + sz, PAGE_SIZE - sz,
- "data (%d) cur:%d, left:%d/%d, vsc:%d, s:%d, map:%d/%d (%d)\n",
- cur_data, cur_sec, msecs, ssecs, vsc, sec_in_line,
- map_weight, lm->sec_per_line, meta_weight);
+ "data (%d) cur:%d, left:%d, vsc:%d, s:%d, map:%d/%d (%d)\n",
+ cur_data, cur_sec, msecs, vsc, sec_in_line,
+ map_weight, lm->sec_per_line,
+ atomic_read(&pblk->inflight_io));
return sz;
}
lm->smeta_len, lm->smeta_sec);
sz += snprintf(page + sz, PAGE_SIZE - sz,
"emeta - len:%d, sec:%d, bb_start:%d\n",
- lm->emeta_len, lm->emeta_sec,
+ lm->emeta_len[0], lm->emeta_sec[0],
lm->emeta_bb);
sz += snprintf(page + sz, PAGE_SIZE - sz,
"bitmap lengths: sec:%d, blk:%d, lun:%d\n",
return sz;
}
+static ssize_t pblk_sysfs_get_sec_per_write(struct pblk *pblk, char *page)
+{
+ return snprintf(page, PAGE_SIZE, "%d\n", pblk->sec_per_write);
+}
+
#ifdef CONFIG_NVM_DEBUG
static ssize_t pblk_sysfs_stats_debug(struct pblk *pblk, char *page)
{
atomic_long_read(&pblk->padded_wb),
atomic_long_read(&pblk->sub_writes),
atomic_long_read(&pblk->sync_writes),
- atomic_long_read(&pblk->compl_writes),
atomic_long_read(&pblk->recov_writes),
atomic_long_read(&pblk->recov_gc_writes),
atomic_long_read(&pblk->recov_gc_reads),
+ atomic_long_read(&pblk->cache_reads),
atomic_long_read(&pblk->sync_reads));
}
#endif
-static ssize_t pblk_sysfs_rate_store(struct pblk *pblk, const char *page,
- size_t len)
+static ssize_t pblk_sysfs_gc_force(struct pblk *pblk, const char *page,
+ size_t len)
{
- struct pblk_gc *gc = &pblk->gc;
size_t c_len;
- int value;
+ int force;
c_len = strcspn(page, "\n");
if (c_len >= len)
return -EINVAL;
- if (kstrtouint(page, 0, &value))
+ if (kstrtouint(page, 0, &force))
return -EINVAL;
- spin_lock(&gc->lock);
- pblk_rl_set_gc_rsc(&pblk->rl, value);
- spin_unlock(&gc->lock);
+ pblk_gc_sysfs_force(pblk, force);
return len;
}
-static ssize_t pblk_sysfs_gc_force(struct pblk *pblk, const char *page,
- size_t len)
+static ssize_t pblk_sysfs_set_sec_per_write(struct pblk *pblk,
+ const char *page, size_t len)
{
size_t c_len;
- int force;
+ int sec_per_write;
c_len = strcspn(page, "\n");
if (c_len >= len)
return -EINVAL;
- if (kstrtouint(page, 0, &force))
+ if (kstrtouint(page, 0, &sec_per_write))
return -EINVAL;
- if (force < 0 || force > 1)
+ if (sec_per_write < pblk->min_write_pgs
+ || sec_per_write > pblk->max_write_pgs
+ || sec_per_write % pblk->min_write_pgs != 0)
return -EINVAL;
- pblk_gc_sysfs_force(pblk, force);
+ pblk_set_sec_per_write(pblk, sec_per_write);
return len;
}
.mode = 0200,
};
-static struct attribute sys_gc_rl_max = {
- .name = "gc_rl_max",
- .mode = 0200,
+static struct attribute sys_max_sec_per_write = {
+ .name = "max_sec_per_write",
+ .mode = 0644,
};
#ifdef CONFIG_NVM_DEBUG
&sys_errors_attr,
&sys_gc_state,
&sys_gc_force,
- &sys_gc_rl_max,
+ &sys_max_sec_per_write,
&sys_rb_attr,
&sys_stats_ppaf_attr,
&sys_lines_attr,
return pblk_sysfs_lines(pblk, buf);
else if (strcmp(attr->name, "lines_info") == 0)
return pblk_sysfs_lines_info(pblk, buf);
+ else if (strcmp(attr->name, "max_sec_per_write") == 0)
+ return pblk_sysfs_get_sec_per_write(pblk, buf);
#ifdef CONFIG_NVM_DEBUG
else if (strcmp(attr->name, "stats") == 0)
return pblk_sysfs_stats_debug(pblk, buf);
{
struct pblk *pblk = container_of(kobj, struct pblk, kobj);
- if (strcmp(attr->name, "gc_rl_max") == 0)
- return pblk_sysfs_rate_store(pblk, buf, len);
- else if (strcmp(attr->name, "gc_force") == 0)
+ if (strcmp(attr->name, "gc_force") == 0)
return pblk_sysfs_gc_force(pblk, buf, len);
+ else if (strcmp(attr->name, "max_sec_per_write") == 0)
+ return pblk_sysfs_set_sec_per_write(pblk, buf, len);
return 0;
}
#include "pblk.h"
-static void pblk_sync_line(struct pblk *pblk, struct pblk_line *line)
-{
-#ifdef CONFIG_NVM_DEBUG
- atomic_long_inc(&pblk->sync_writes);
-#endif
-
- /* Counter protected by rb sync lock */
- line->left_ssecs--;
- if (!line->left_ssecs)
- pblk_line_run_ws(pblk, line, NULL, pblk_line_close_ws);
-}
-
static unsigned long pblk_end_w_bio(struct pblk *pblk, struct nvm_rq *rqd,
struct pblk_c_ctx *c_ctx)
{
for (i = 0; i < c_ctx->nr_valid; i++) {
struct pblk_w_ctx *w_ctx;
- struct ppa_addr p;
- struct pblk_line *line;
w_ctx = pblk_rb_w_ctx(&pblk->rwb, c_ctx->sentry + i);
-
- p = rqd->ppa_list[i];
- line = &pblk->lines[pblk_dev_ppa_to_line(p)];
- pblk_sync_line(pblk, line);
-
while ((original_bio = bio_list_pop(&w_ctx->bios)))
bio_endio(original_bio);
}
#ifdef CONFIG_NVM_DEBUG
- atomic_long_add(c_ctx->nr_valid, &pblk->compl_writes);
+ atomic_long_add(c_ctx->nr_valid, &pblk->sync_writes);
#endif
ret = pblk_rb_sync_advance(&pblk->rwb, c_ctx->nr_valid);
}
INIT_WORK(&recovery->ws_rec, pblk_submit_rec);
- queue_work(pblk->kw_wq, &recovery->ws_rec);
+ queue_work(pblk->close_wq, &recovery->ws_rec);
out:
pblk_complete_write(pblk, rqd, c_ctx);
}
#ifdef CONFIG_NVM_DEBUG
else
- WARN_ONCE(rqd->bio->bi_error, "pblk: corrupted write error\n");
+ WARN_ONCE(rqd->bio->bi_status, "pblk: corrupted write error\n");
#endif
pblk_complete_write(pblk, rqd, c_ctx);
+ atomic_dec(&pblk->inflight_io);
+}
+
+static void pblk_end_io_write_meta(struct nvm_rq *rqd)
+{
+ struct pblk *pblk = rqd->private;
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_g_ctx *m_ctx = nvm_rq_to_pdu(rqd);
+ struct pblk_line *line = m_ctx->private;
+ struct pblk_emeta *emeta = line->emeta;
+ int pos = pblk_ppa_to_pos(geo, rqd->ppa_list[0]);
+ struct pblk_lun *rlun = &pblk->luns[pos];
+ int sync;
+
+ up(&rlun->wr_sem);
+
+ if (rqd->error) {
+ pblk_log_write_err(pblk, rqd);
+ pr_err("pblk: metadata I/O failed. Line %d\n", line->id);
+ }
+#ifdef CONFIG_NVM_DEBUG
+ else
+ WARN_ONCE(rqd->bio->bi_status, "pblk: corrupted write error\n");
+#endif
+
+ sync = atomic_add_return(rqd->nr_ppas, &emeta->sync);
+ if (sync == emeta->nr_entries)
+ pblk_line_run_ws(pblk, line, NULL, pblk_line_close_ws,
+ pblk->close_wq);
+
+ bio_put(rqd->bio);
+ pblk_free_rqd(pblk, rqd, READ);
+
+ atomic_dec(&pblk->inflight_io);
}
static int pblk_alloc_w_rq(struct pblk *pblk, struct nvm_rq *rqd,
- unsigned int nr_secs)
+ unsigned int nr_secs,
+ nvm_end_io_fn(*end_io))
{
struct nvm_tgt_dev *dev = pblk->dev;
rqd->nr_ppas = nr_secs;
rqd->flags = pblk_set_progr_mode(pblk, WRITE);
rqd->private = pblk;
- rqd->end_io = pblk_end_io_write;
+ rqd->end_io = end_io;
rqd->meta_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL,
&rqd->dma_meta_list);
}
static int pblk_setup_w_rq(struct pblk *pblk, struct nvm_rq *rqd,
- struct pblk_c_ctx *c_ctx)
+ struct pblk_c_ctx *c_ctx, struct ppa_addr *erase_ppa)
{
struct pblk_line_meta *lm = &pblk->lm;
- struct pblk_line *e_line = pblk_line_get_data_next(pblk);
- struct ppa_addr erase_ppa;
+ struct pblk_line *e_line = pblk_line_get_erase(pblk);
unsigned int valid = c_ctx->nr_valid;
unsigned int padded = c_ctx->nr_padded;
unsigned int nr_secs = valid + padded;
int ret = 0;
lun_bitmap = kzalloc(lm->lun_bitmap_len, GFP_KERNEL);
- if (!lun_bitmap) {
- ret = -ENOMEM;
- goto out;
- }
+ if (!lun_bitmap)
+ return -ENOMEM;
c_ctx->lun_bitmap = lun_bitmap;
- ret = pblk_alloc_w_rq(pblk, rqd, nr_secs);
+ ret = pblk_alloc_w_rq(pblk, rqd, nr_secs, pblk_end_io_write);
if (ret) {
kfree(lun_bitmap);
- goto out;
+ return ret;
}
- ppa_set_empty(&erase_ppa);
if (likely(!e_line || !atomic_read(&e_line->left_eblks)))
pblk_map_rq(pblk, rqd, c_ctx->sentry, lun_bitmap, valid, 0);
else
pblk_map_erase_rq(pblk, rqd, c_ctx->sentry, lun_bitmap,
- valid, &erase_ppa);
-
-out:
- if (unlikely(e_line && !ppa_empty(erase_ppa))) {
- if (pblk_blk_erase_async(pblk, erase_ppa)) {
- struct nvm_tgt_dev *dev = pblk->dev;
- struct nvm_geo *geo = &dev->geo;
- int bit;
-
- atomic_inc(&e_line->left_eblks);
- bit = erase_ppa.g.lun * geo->nr_chnls + erase_ppa.g.ch;
- WARN_ON(!test_and_clear_bit(bit, e_line->erase_bitmap));
- up(&pblk->erase_sem);
- }
- }
+ valid, erase_ppa);
- return ret;
+ return 0;
}
int pblk_setup_w_rec_rq(struct pblk *pblk, struct nvm_rq *rqd,
c_ctx->lun_bitmap = lun_bitmap;
- ret = pblk_alloc_w_rq(pblk, rqd, rqd->nr_ppas);
+ ret = pblk_alloc_w_rq(pblk, rqd, rqd->nr_ppas, pblk_end_io_write);
if (ret)
return ret;
return secs_to_sync;
}
+static inline int pblk_valid_meta_ppa(struct pblk *pblk,
+ struct pblk_line *meta_line,
+ struct ppa_addr *ppa_list, int nr_ppas)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_line *data_line;
+ struct ppa_addr ppa, ppa_opt;
+ u64 paddr;
+ int i;
+
+ data_line = &pblk->lines[pblk_dev_ppa_to_line(ppa_list[0])];
+ paddr = pblk_lookup_page(pblk, meta_line);
+ ppa = addr_to_gen_ppa(pblk, paddr, 0);
+
+ if (test_bit(pblk_ppa_to_pos(geo, ppa), data_line->blk_bitmap))
+ return 1;
+
+ /* Schedule a metadata I/O that is half the distance from the data I/O
+ * with regards to the number of LUNs forming the pblk instance. This
+ * balances LUN conflicts across every I/O.
+ *
+ * When the LUN configuration changes (e.g., due to GC), this distance
+ * can align, which would result on a LUN deadlock. In this case, modify
+ * the distance to not be optimal, but allow metadata I/Os to succeed.
+ */
+ ppa_opt = addr_to_gen_ppa(pblk, paddr + data_line->meta_distance, 0);
+ if (unlikely(ppa_opt.ppa == ppa.ppa)) {
+ data_line->meta_distance--;
+ return 0;
+ }
+
+ for (i = 0; i < nr_ppas; i += pblk->min_write_pgs)
+ if (ppa_list[i].g.ch == ppa_opt.g.ch &&
+ ppa_list[i].g.lun == ppa_opt.g.lun)
+ return 1;
+
+ if (test_bit(pblk_ppa_to_pos(geo, ppa_opt), data_line->blk_bitmap)) {
+ for (i = 0; i < nr_ppas; i += pblk->min_write_pgs)
+ if (ppa_list[i].g.ch == ppa.g.ch &&
+ ppa_list[i].g.lun == ppa.g.lun)
+ return 0;
+
+ return 1;
+ }
+
+ return 0;
+}
+
+int pblk_submit_meta_io(struct pblk *pblk, struct pblk_line *meta_line)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct pblk_emeta *emeta = meta_line->emeta;
+ struct pblk_g_ctx *m_ctx;
+ struct pblk_lun *rlun;
+ struct bio *bio;
+ struct nvm_rq *rqd;
+ void *data;
+ u64 paddr;
+ int rq_ppas = pblk->min_write_pgs;
+ int id = meta_line->id;
+ int rq_len;
+ int i, j;
+ int ret;
+
+ rqd = pblk_alloc_rqd(pblk, READ);
+ if (IS_ERR(rqd)) {
+ pr_err("pblk: cannot allocate write req.\n");
+ return PTR_ERR(rqd);
+ }
+ m_ctx = nvm_rq_to_pdu(rqd);
+ m_ctx->private = meta_line;
+
+ rq_len = rq_ppas * geo->sec_size;
+ data = ((void *)emeta->buf) + emeta->mem;
+
+ bio = pblk_bio_map_addr(pblk, data, rq_ppas, rq_len,
+ l_mg->emeta_alloc_type, GFP_KERNEL);
+ if (IS_ERR(bio)) {
+ ret = PTR_ERR(bio);
+ goto fail_free_rqd;
+ }
+ bio->bi_iter.bi_sector = 0; /* internal bio */
+ bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
+ rqd->bio = bio;
+
+ ret = pblk_alloc_w_rq(pblk, rqd, rq_ppas, pblk_end_io_write_meta);
+ if (ret)
+ goto fail_free_bio;
+
+ for (i = 0; i < rqd->nr_ppas; ) {
+ spin_lock(&meta_line->lock);
+ paddr = __pblk_alloc_page(pblk, meta_line, rq_ppas);
+ spin_unlock(&meta_line->lock);
+ for (j = 0; j < rq_ppas; j++, i++, paddr++)
+ rqd->ppa_list[i] = addr_to_gen_ppa(pblk, paddr, id);
+ }
+
+ rlun = &pblk->luns[pblk_ppa_to_pos(geo, rqd->ppa_list[0])];
+ ret = down_timeout(&rlun->wr_sem, msecs_to_jiffies(5000));
+ if (ret) {
+ pr_err("pblk: lun semaphore timed out (%d)\n", ret);
+ goto fail_free_bio;
+ }
+
+ emeta->mem += rq_len;
+ if (emeta->mem >= lm->emeta_len[0]) {
+ spin_lock(&l_mg->close_lock);
+ list_del(&meta_line->list);
+ WARN(!bitmap_full(meta_line->map_bitmap, lm->sec_per_line),
+ "pblk: corrupt meta line %d\n", meta_line->id);
+ spin_unlock(&l_mg->close_lock);
+ }
+
+ ret = pblk_submit_io(pblk, rqd);
+ if (ret) {
+ pr_err("pblk: emeta I/O submission failed: %d\n", ret);
+ goto fail_rollback;
+ }
+
+ return NVM_IO_OK;
+
+fail_rollback:
+ spin_lock(&l_mg->close_lock);
+ pblk_dealloc_page(pblk, meta_line, rq_ppas);
+ list_add(&meta_line->list, &meta_line->list);
+ spin_unlock(&l_mg->close_lock);
+fail_free_bio:
+ if (likely(l_mg->emeta_alloc_type == PBLK_VMALLOC_META))
+ bio_put(bio);
+fail_free_rqd:
+ pblk_free_rqd(pblk, rqd, READ);
+ return ret;
+}
+
+static int pblk_sched_meta_io(struct pblk *pblk, struct ppa_addr *prev_list,
+ int prev_n)
+{
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct pblk_line *meta_line;
+
+ spin_lock(&l_mg->close_lock);
+retry:
+ if (list_empty(&l_mg->emeta_list)) {
+ spin_unlock(&l_mg->close_lock);
+ return 0;
+ }
+ meta_line = list_first_entry(&l_mg->emeta_list, struct pblk_line, list);
+ if (bitmap_full(meta_line->map_bitmap, lm->sec_per_line))
+ goto retry;
+ spin_unlock(&l_mg->close_lock);
+
+ if (!pblk_valid_meta_ppa(pblk, meta_line, prev_list, prev_n))
+ return 0;
+
+ return pblk_submit_meta_io(pblk, meta_line);
+}
+
+static int pblk_submit_io_set(struct pblk *pblk, struct nvm_rq *rqd)
+{
+ struct pblk_c_ctx *c_ctx = nvm_rq_to_pdu(rqd);
+ struct ppa_addr erase_ppa;
+ int err;
+
+ ppa_set_empty(&erase_ppa);
+
+ /* Assign lbas to ppas and populate request structure */
+ err = pblk_setup_w_rq(pblk, rqd, c_ctx, &erase_ppa);
+ if (err) {
+ pr_err("pblk: could not setup write request: %d\n", err);
+ return NVM_IO_ERR;
+ }
+
+ if (likely(ppa_empty(erase_ppa))) {
+ /* Submit metadata write for previous data line */
+ err = pblk_sched_meta_io(pblk, rqd->ppa_list, rqd->nr_ppas);
+ if (err) {
+ pr_err("pblk: metadata I/O submission failed: %d", err);
+ return NVM_IO_ERR;
+ }
+
+ /* Submit data write for current data line */
+ err = pblk_submit_io(pblk, rqd);
+ if (err) {
+ pr_err("pblk: data I/O submission failed: %d\n", err);
+ return NVM_IO_ERR;
+ }
+ } else {
+ /* Submit data write for current data line */
+ err = pblk_submit_io(pblk, rqd);
+ if (err) {
+ pr_err("pblk: data I/O submission failed: %d\n", err);
+ return NVM_IO_ERR;
+ }
+
+ /* Submit available erase for next data line */
+ if (pblk_blk_erase_async(pblk, erase_ppa)) {
+ struct pblk_line *e_line = pblk_line_get_erase(pblk);
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ int bit;
+
+ atomic_inc(&e_line->left_eblks);
+ bit = pblk_ppa_to_pos(geo, erase_ppa);
+ WARN_ON(!test_and_clear_bit(bit, e_line->erase_bitmap));
+ }
+ }
+
+ return NVM_IO_OK;
+}
+
+static void pblk_free_write_rqd(struct pblk *pblk, struct nvm_rq *rqd)
+{
+ struct pblk_c_ctx *c_ctx = nvm_rq_to_pdu(rqd);
+ struct bio *bio = rqd->bio;
+
+ if (c_ctx->nr_padded)
+ pblk_bio_free_pages(pblk, bio, rqd->nr_ppas, c_ctx->nr_padded);
+}
+
static int pblk_submit_write(struct pblk *pblk)
{
struct bio *bio;
struct nvm_rq *rqd;
- struct pblk_c_ctx *c_ctx;
- unsigned int pgs_read;
unsigned int secs_avail, secs_to_sync, secs_to_com;
unsigned int secs_to_flush;
unsigned long pos;
- int err;
/* If there are no sectors in the cache, flushes (bios without data)
* will be cleared on the cache threads
pr_err("pblk: cannot allocate write req.\n");
return 1;
}
- c_ctx = nvm_rq_to_pdu(rqd);
bio = bio_alloc(GFP_KERNEL, pblk->max_write_pgs);
if (!bio) {
secs_to_com = (secs_to_sync > secs_avail) ? secs_avail : secs_to_sync;
pos = pblk_rb_read_commit(&pblk->rwb, secs_to_com);
- pgs_read = pblk_rb_read_to_bio(&pblk->rwb, bio, c_ctx, pos,
- secs_to_sync, secs_avail);
- if (!pgs_read) {
+ if (pblk_rb_read_to_bio(&pblk->rwb, rqd, bio, pos, secs_to_sync,
+ secs_avail)) {
pr_err("pblk: corrupted write bio\n");
goto fail_put_bio;
}
- if (c_ctx->nr_padded)
- if (pblk_bio_add_pages(pblk, bio, GFP_KERNEL, c_ctx->nr_padded))
- goto fail_put_bio;
-
- /* Assign lbas to ppas and populate request structure */
- err = pblk_setup_w_rq(pblk, rqd, c_ctx);
- if (err) {
- pr_err("pblk: could not setup write request\n");
- goto fail_free_bio;
- }
-
- err = pblk_submit_io(pblk, rqd);
- if (err) {
- pr_err("pblk: I/O submission failed: %d\n", err);
+ if (pblk_submit_io_set(pblk, rqd))
goto fail_free_bio;
- }
#ifdef CONFIG_NVM_DEBUG
atomic_long_add(secs_to_sync, &pblk->sub_writes);
return 0;
fail_free_bio:
- if (c_ctx->nr_padded)
- pblk_bio_free_pages(pblk, bio, secs_to_sync, c_ctx->nr_padded);
+ pblk_free_write_rqd(pblk, rqd);
fail_put_bio:
bio_put(bio);
fail_free_rqd:
#define PBLK_MAX_REQ_ADDRS (64)
#define PBLK_MAX_REQ_ADDRS_PW (6)
+#define PBLK_WS_POOL_SIZE (128)
+#define PBLK_META_POOL_SIZE (128)
+#define PBLK_READ_REQ_POOL_SIZE (1024)
+
+#define PBLK_NR_CLOSE_JOBS (4)
+
#define PBLK_CACHE_NAME_LEN (DISK_NAME_LEN + 16)
#define PBLK_COMMAND_TIMEOUT_MS 30000
PBLK_BLK_ST_CLOSED = 0x2,
};
+struct pblk_sec_meta {
+ u64 reserved;
+ __le64 lba;
+};
+
/* The number of GC lists and the rate-limiter states go together. This way the
* rate-limiter can dictate how much GC is needed based on resource utilization.
*/
-#define PBLK_NR_GC_LISTS 3
-#define PBLK_MAX_GC_JOBS 32
+#define PBLK_GC_NR_LISTS 3
enum {
PBLK_RL_HIGH = 1,
PBLK_RL_LOW = 3,
};
-struct pblk_sec_meta {
- u64 reserved;
- __le64 lba;
-};
-
#define pblk_dma_meta_size (sizeof(struct pblk_sec_meta) * PBLK_MAX_REQ_ADDRS)
-/* write completion context */
+/* write buffer completion context */
struct pblk_c_ctx {
struct list_head list; /* Head for out-of-order completion */
unsigned int nr_padded;
};
-/* Read context */
-struct pblk_r_ctx {
- struct bio *orig_bio;
+/* generic context */
+struct pblk_g_ctx {
+ void *private;
+};
+
+/* Pad context */
+struct pblk_pad_rq {
+ struct pblk *pblk;
+ struct completion wait;
+ struct kref ref;
};
/* Recovery context */
struct pblk_gc_rq {
struct pblk_line *line;
void *data;
- u64 *lba_list;
+ u64 lba_list[PBLK_MAX_REQ_ADDRS];
int nr_secs;
int secs_to_gc;
struct list_head list;
};
struct pblk_gc {
+ /* These states are not protected by a lock since (i) they are in the
+ * fast path, and (ii) they are not critical.
+ */
int gc_active;
int gc_enabled;
int gc_forced;
- int gc_jobs_active;
- atomic_t inflight_gc;
struct task_struct *gc_ts;
struct task_struct *gc_writer_ts;
+ struct task_struct *gc_reader_ts;
+
+ struct workqueue_struct *gc_line_reader_wq;
struct workqueue_struct *gc_reader_wq;
+
struct timer_list gc_timer;
+ struct semaphore gc_sem;
+ atomic_t inflight_gc;
int w_entries;
+
struct list_head w_list;
+ struct list_head r_list;
spinlock_t lock;
spinlock_t w_lock;
+ spinlock_t r_lock;
};
struct pblk_rl {
*/
unsigned int high_pw; /* High rounded up as a power of 2 */
-#define PBLK_USER_HIGH_THRS 2 /* Begin write limit at 50 percent
- * available blks
- */
-#define PBLK_USER_LOW_THRS 20 /* Aggressive GC at 5% available blocks */
+#define PBLK_USER_HIGH_THRS 8 /* Begin write limit at 12% available blks */
+#define PBLK_USER_LOW_THRS 10 /* Aggressive GC at 10% available blocks */
int rb_windows_pw; /* Number of rate windows in the write buffer
* given as a power-of-2. This guarantees that
*/
int rb_budget; /* Total number of entries available for I/O */
int rb_user_max; /* Max buffer entries available for user I/O */
- atomic_t rb_user_cnt; /* User I/O buffer counter */
int rb_gc_max; /* Max buffer entries available for GC I/O */
int rb_gc_rsv; /* Reserved buffer entries for GC I/O */
int rb_state; /* Rate-limiter current state */
+
+ atomic_t rb_user_cnt; /* User I/O buffer counter */
atomic_t rb_gc_cnt; /* GC I/O buffer counter */
+ atomic_t rb_space; /* Space limit in case of reaching capacity */
+
+ int rsv_blocks; /* Reserved blocks for GC */
int rb_user_active;
+ int rb_gc_active;
+
struct timer_list u_timer;
unsigned long long nr_secs;
atomic_t free_blocks;
};
-#define PBLK_LINE_NR_LUN_BITMAP 2
-#define PBLK_LINE_NR_SEC_BITMAP 2
#define PBLK_LINE_EMPTY (~0U)
enum {
__le32 window_wr_lun; /* Number of parallel LUNs to write */
__le32 rsvd[2];
+
+ __le64 lun_bitmap[];
};
/*
- * Metadata Layout:
- * 1. struct pblk_emeta
- * 2. nr_lbas u64 forming lba list
- * 3. nr_lines (all) u32 valid sector count (vsc) (~0U: non-alloc line)
- * 4. nr_luns bits (u64 format) forming line bad block bitmap
- *
- * 3. and 4. will be part of FTL log
+ * Metadata layout in media:
+ * First sector:
+ * 1. struct line_emeta
+ * 2. bad block bitmap (u64 * window_wr_lun)
+ * Mid sectors (start at lbas_sector):
+ * 3. nr_lbas (u64) forming lba list
+ * Last sectors (start at vsc_sector):
+ * 4. u32 valid sector count (vsc) for all lines (~0U: free line)
*/
struct line_emeta {
struct line_header header;
__le32 next_id; /* Line id for next line */
__le64 nr_lbas; /* Number of lbas mapped in line */
__le64 nr_valid_lbas; /* Number of valid lbas mapped in line */
+ __le64 bb_bitmap[]; /* Updated bad block bitmap for line */
+};
+
+struct pblk_emeta {
+ struct line_emeta *buf; /* emeta buffer in media format */
+ int mem; /* Write offset - points to next
+ * writable entry in memory
+ */
+ atomic_t sync; /* Synced - backpointer that signals the
+ * last entry that has been successfully
+ * persisted to media
+ */
+ unsigned int nr_entries; /* Number of emeta entries */
+};
+
+struct pblk_smeta {
+ struct line_smeta *buf; /* smeta buffer in persistent format */
};
struct pblk_line {
unsigned long *lun_bitmap; /* Bitmap for LUNs mapped in line */
- struct line_smeta *smeta; /* Start metadata */
- struct line_emeta *emeta; /* End metadata */
+ struct pblk_smeta *smeta; /* Start metadata */
+ struct pblk_emeta *emeta; /* End medatada */
+
int meta_line; /* Metadata line id */
+ int meta_distance; /* Distance between data and metadata */
+
u64 smeta_ssec; /* Sector where smeta starts */
u64 emeta_ssec; /* Sector where emeta starts */
atomic_t left_seblks; /* Blocks left for sync erasing */
int left_msecs; /* Sectors left for mapping */
- int left_ssecs; /* Sectors left to sync */
unsigned int cur_sec; /* Sector map pointer */
- unsigned int vsc; /* Valid sector count in line */
+ unsigned int nr_valid_lbas; /* Number of valid lbas in line */
+
+ __le32 *vsc; /* Valid sector count in line */
struct kref ref; /* Write buffer L2P references */
#define PBLK_DATA_LINES 4
-enum{
+enum {
PBLK_KMALLOC_META = 1,
PBLK_VMALLOC_META = 2,
};
-struct pblk_line_metadata {
- void *meta;
+enum {
+ PBLK_EMETA_TYPE_HEADER = 1, /* struct line_emeta first sector */
+ PBLK_EMETA_TYPE_LLBA = 2, /* lba list - type: __le64 */
+ PBLK_EMETA_TYPE_VSC = 3, /* vsc list - type: __le32 */
};
struct pblk_line_mgmt {
struct list_head bad_list; /* Full lines bad */
/* GC lists - use gc_lock */
- struct list_head *gc_lists[PBLK_NR_GC_LISTS];
+ struct list_head *gc_lists[PBLK_GC_NR_LISTS];
struct list_head gc_high_list; /* Full lines ready to GC, high isc */
struct list_head gc_mid_list; /* Full lines ready to GC, mid isc */
struct list_head gc_low_list; /* Full lines ready to GC, low isc */
struct pblk_line *log_next; /* Next FTL log line */
struct pblk_line *data_next; /* Next data line */
+ struct list_head emeta_list; /* Lines queued to schedule emeta */
+
+ __le32 *vsc_list; /* Valid sector counts for all lines */
+
/* Metadata allocation type: VMALLOC | KMALLOC */
- int smeta_alloc_type;
int emeta_alloc_type;
/* Pre-allocated metadata for data lines */
- struct pblk_line_metadata sline_meta[PBLK_DATA_LINES];
- struct pblk_line_metadata eline_meta[PBLK_DATA_LINES];
+ struct pblk_smeta *sline_meta[PBLK_DATA_LINES];
+ struct pblk_emeta *eline_meta[PBLK_DATA_LINES];
unsigned long meta_bitmap;
/* Helpers for fast bitmap calculations */
unsigned long l_seq_nr; /* Log line unique sequence number */
spinlock_t free_lock;
+ spinlock_t close_lock;
spinlock_t gc_lock;
};
struct pblk_line_meta {
unsigned int smeta_len; /* Total length for smeta */
- unsigned int smeta_sec; /* Sectors needed for smeta*/
- unsigned int emeta_len; /* Total length for emeta */
- unsigned int emeta_sec; /* Sectors needed for emeta*/
+ unsigned int smeta_sec; /* Sectors needed for smeta */
+
+ unsigned int emeta_len[4]; /* Lengths for emeta:
+ * [0]: Total length
+ * [1]: struct line_emeta length
+ * [2]: L2P portion length
+ * [3]: vsc list length
+ */
+ unsigned int emeta_sec[4]; /* Sectors needed for emeta. Same layout
+ * as emeta_len
+ */
+
unsigned int emeta_bb; /* Boundary for bb that affects emeta */
+
+ unsigned int vsc_list_len; /* Length for vsc list */
unsigned int sec_bitmap_len; /* Length for sector bitmap in line */
unsigned int blk_bitmap_len; /* Length for block bitmap in line */
unsigned int lun_bitmap_len; /* Length for lun bitmap in line */
unsigned int blk_per_line; /* Number of blocks in a full line */
unsigned int sec_per_line; /* Number of sectors in a line */
+ unsigned int dsec_per_line; /* Number of data sectors in a line */
unsigned int min_blk_line; /* Min. number of good blocks in line */
unsigned int mid_thrs; /* Threshold for GC mid list */
unsigned int high_thrs; /* Threshold for GC high list */
+
+ unsigned int meta_distance; /* Distance between data and metadata */
};
struct pblk_addr_format {
u8 sec_offset;
};
+enum {
+ PBLK_STATE_RUNNING = 0,
+ PBLK_STATE_STOPPING = 1,
+ PBLK_STATE_RECOVERING = 2,
+ PBLK_STATE_STOPPED = 3,
+};
+
struct pblk {
struct nvm_tgt_dev *dev;
struct gendisk *disk;
struct pblk_rb rwb;
+ int state; /* pblk line state */
+
int min_write_pgs; /* Minimum amount of pages required by controller */
int max_write_pgs; /* Maximum amount of pages supported by controller */
int pgs_in_buffer; /* Number of pages that need to be held in buffer to
/* pblk provisioning values. Used by rate limiter */
struct pblk_rl rl;
- struct semaphore erase_sem;
+ int sec_per_write;
unsigned char instance_uuid[16];
#ifdef CONFIG_NVM_DEBUG
atomic_long_t req_writes; /* Sectors stored on write buffer */
atomic_long_t sub_writes; /* Sectors submitted from buffer */
atomic_long_t sync_writes; /* Sectors synced to media */
- atomic_long_t compl_writes; /* Sectors completed in write bio */
atomic_long_t inflight_reads; /* Inflight sector read requests */
+ atomic_long_t cache_reads; /* Read requests that hit the cache */
atomic_long_t sync_reads; /* Completed sector read requests */
atomic_long_t recov_writes; /* Sectors submitted from recovery */
atomic_long_t recov_gc_writes; /* Sectors submitted from write GC */
atomic_long_t write_failed;
atomic_long_t erase_failed;
+ atomic_t inflight_io; /* General inflight I/O counter */
+
struct task_struct *writer_ts;
/* Simple translation map of logical addresses to physical addresses.
mempool_t *page_pool;
mempool_t *line_ws_pool;
mempool_t *rec_pool;
- mempool_t *r_rq_pool;
+ mempool_t *g_rq_pool;
mempool_t *w_rq_pool;
mempool_t *line_meta_pool;
- struct workqueue_struct *kw_wq;
+ struct workqueue_struct *close_wq;
+ struct workqueue_struct *bb_wq;
+
struct timer_list wtimer;
struct pblk_gc gc;
struct work_struct ws;
};
-#define pblk_r_rq_size (sizeof(struct nvm_rq) + sizeof(struct pblk_r_ctx))
+#define pblk_g_rq_size (sizeof(struct nvm_rq) + sizeof(struct pblk_g_ctx))
#define pblk_w_rq_size (sizeof(struct nvm_rq) + sizeof(struct pblk_c_ctx))
/*
struct pblk_w_ctx w_ctx, struct pblk_line *gc_line,
unsigned int pos);
struct pblk_w_ctx *pblk_rb_w_ctx(struct pblk_rb *rb, unsigned int pos);
+void pblk_rb_flush(struct pblk_rb *rb);
void pblk_rb_sync_l2p(struct pblk_rb *rb);
-unsigned int pblk_rb_read_to_bio(struct pblk_rb *rb, struct bio *bio,
- struct pblk_c_ctx *c_ctx,
- unsigned int pos,
- unsigned int nr_entries,
- unsigned int count);
+unsigned int pblk_rb_read_to_bio(struct pblk_rb *rb, struct nvm_rq *rqd,
+ struct bio *bio, unsigned int pos,
+ unsigned int nr_entries, unsigned int count);
unsigned int pblk_rb_read_to_bio_list(struct pblk_rb *rb, struct bio *bio,
struct list_head *list,
unsigned int max);
int pblk_rb_copy_to_bio(struct pblk_rb *rb, struct bio *bio, sector_t lba,
- u64 pos, int bio_iter);
+ struct ppa_addr ppa, int bio_iter);
unsigned int pblk_rb_read_commit(struct pblk_rb *rb, unsigned int entries);
unsigned int pblk_rb_sync_init(struct pblk_rb *rb, unsigned long *flags);
unsigned int pblk_rb_sync_point_count(struct pblk_rb *rb);
unsigned int pblk_rb_read_count(struct pblk_rb *rb);
+unsigned int pblk_rb_sync_count(struct pblk_rb *rb);
unsigned int pblk_rb_wrap_pos(struct pblk_rb *rb, unsigned int pos);
int pblk_rb_tear_down_check(struct pblk_rb *rb);
* pblk core
*/
struct nvm_rq *pblk_alloc_rqd(struct pblk *pblk, int rw);
+void pblk_set_sec_per_write(struct pblk *pblk, int sec_per_write);
int pblk_setup_w_rec_rq(struct pblk *pblk, struct nvm_rq *rqd,
struct pblk_c_ctx *c_ctx);
void pblk_free_rqd(struct pblk *pblk, struct nvm_rq *rqd, int rw);
-void pblk_flush_writer(struct pblk *pblk);
+void pblk_wait_for_meta(struct pblk *pblk);
struct ppa_addr pblk_get_lba_map(struct pblk *pblk, sector_t lba);
void pblk_discard(struct pblk *pblk, struct bio *bio);
void pblk_log_write_err(struct pblk *pblk, struct nvm_rq *rqd);
void pblk_log_read_err(struct pblk *pblk, struct nvm_rq *rqd);
int pblk_submit_io(struct pblk *pblk, struct nvm_rq *rqd);
+int pblk_submit_meta_io(struct pblk *pblk, struct pblk_line *meta_line);
struct bio *pblk_bio_map_addr(struct pblk *pblk, void *data,
unsigned int nr_secs, unsigned int len,
- gfp_t gfp_mask);
+ int alloc_type, gfp_t gfp_mask);
struct pblk_line *pblk_line_get(struct pblk *pblk);
struct pblk_line *pblk_line_get_first_data(struct pblk *pblk);
-struct pblk_line *pblk_line_replace_data(struct pblk *pblk);
+void pblk_line_replace_data(struct pblk *pblk);
int pblk_line_recov_alloc(struct pblk *pblk, struct pblk_line *line);
void pblk_line_recov_close(struct pblk *pblk, struct pblk_line *line);
struct pblk_line *pblk_line_get_data(struct pblk *pblk);
-struct pblk_line *pblk_line_get_data_next(struct pblk *pblk);
+struct pblk_line *pblk_line_get_erase(struct pblk *pblk);
int pblk_line_erase(struct pblk *pblk, struct pblk_line *line);
int pblk_line_is_full(struct pblk_line *line);
void pblk_line_free(struct pblk *pblk, struct pblk_line *line);
-void pblk_line_close_ws(struct work_struct *work);
+void pblk_line_close_meta(struct pblk *pblk, struct pblk_line *line);
void pblk_line_close(struct pblk *pblk, struct pblk_line *line);
+void pblk_line_close_meta_sync(struct pblk *pblk);
+void pblk_line_close_ws(struct work_struct *work);
+void pblk_pipeline_stop(struct pblk *pblk);
void pblk_line_mark_bb(struct work_struct *work);
void pblk_line_run_ws(struct pblk *pblk, struct pblk_line *line, void *priv,
- void (*work)(struct work_struct *));
+ void (*work)(struct work_struct *),
+ struct workqueue_struct *wq);
u64 pblk_line_smeta_start(struct pblk *pblk, struct pblk_line *line);
int pblk_line_read_smeta(struct pblk *pblk, struct pblk_line *line);
-int pblk_line_read_emeta(struct pblk *pblk, struct pblk_line *line);
+int pblk_line_read_emeta(struct pblk *pblk, struct pblk_line *line,
+ void *emeta_buf);
int pblk_blk_erase_async(struct pblk *pblk, struct ppa_addr erase_ppa);
void pblk_line_put(struct kref *ref);
struct list_head *pblk_line_gc_list(struct pblk *pblk, struct pblk_line *line);
+u64 pblk_lookup_page(struct pblk *pblk, struct pblk_line *line);
+void pblk_dealloc_page(struct pblk *pblk, struct pblk_line *line, int nr_secs);
u64 pblk_alloc_page(struct pblk *pblk, struct pblk_line *line, int nr_secs);
+u64 __pblk_alloc_page(struct pblk *pblk, struct pblk_line *line, int nr_secs);
int pblk_calc_secs(struct pblk *pblk, unsigned long secs_avail,
unsigned long secs_to_flush);
void pblk_down_rq(struct pblk *pblk, struct ppa_addr *ppa_list, int nr_ppas,
void pblk_end_io_sync(struct nvm_rq *rqd);
int pblk_bio_add_pages(struct pblk *pblk, struct bio *bio, gfp_t flags,
int nr_pages);
-void pblk_map_pad_invalidate(struct pblk *pblk, struct pblk_line *line,
- u64 paddr);
void pblk_bio_free_pages(struct pblk *pblk, struct bio *bio, int off,
int nr_pages);
void pblk_map_invalidate(struct pblk *pblk, struct ppa_addr ppa);
+void __pblk_map_invalidate(struct pblk *pblk, struct pblk_line *line,
+ u64 paddr);
void pblk_update_map(struct pblk *pblk, sector_t lba, struct ppa_addr ppa);
void pblk_update_map_cache(struct pblk *pblk, sector_t lba,
struct ppa_addr ppa);
/*
* pblk read path
*/
+extern struct bio_set *pblk_bio_set;
int pblk_submit_read(struct pblk *pblk, struct bio *bio);
int pblk_submit_read_gc(struct pblk *pblk, u64 *lba_list, void *data,
unsigned int nr_secs, unsigned int *secs_to_gc,
*/
void pblk_submit_rec(struct work_struct *work);
struct pblk_line *pblk_recov_l2p(struct pblk *pblk);
-void pblk_recov_pad(struct pblk *pblk);
+int pblk_recov_pad(struct pblk *pblk);
__le64 *pblk_recov_get_lba_list(struct pblk *pblk, struct line_emeta *emeta);
int pblk_recov_setup_rq(struct pblk *pblk, struct pblk_c_ctx *c_ctx,
struct pblk_rec_ctx *recovery, u64 *comp_bits,
/*
* pblk gc
*/
-#define PBLK_GC_TRIES 3
+#define PBLK_GC_MAX_READERS 8 /* Max number of outstanding GC reader jobs */
+#define PBLK_GC_W_QD 128 /* Queue depth for inflight GC write I/Os */
+#define PBLK_GC_L_QD 4 /* Queue depth for inflight GC lines */
+#define PBLK_GC_RSV_LINE 1 /* Reserved lines for GC */
int pblk_gc_init(struct pblk *pblk);
void pblk_gc_exit(struct pblk *pblk);
void pblk_gc_should_start(struct pblk *pblk);
void pblk_gc_should_stop(struct pblk *pblk);
-int pblk_gc_status(struct pblk *pblk);
+void pblk_gc_should_kick(struct pblk *pblk);
+void pblk_gc_kick(struct pblk *pblk);
void pblk_gc_sysfs_state_show(struct pblk *pblk, int *gc_enabled,
int *gc_active);
-void pblk_gc_sysfs_force(struct pblk *pblk, int force);
+int pblk_gc_sysfs_force(struct pblk *pblk, int force);
/*
* pblk rate limiter
*/
void pblk_rl_init(struct pblk_rl *rl, int budget);
void pblk_rl_free(struct pblk_rl *rl);
-int pblk_rl_gc_thrs(struct pblk_rl *rl);
+int pblk_rl_high_thrs(struct pblk_rl *rl);
+int pblk_rl_low_thrs(struct pblk_rl *rl);
unsigned long pblk_rl_nr_free_blks(struct pblk_rl *rl);
int pblk_rl_user_may_insert(struct pblk_rl *rl, int nr_entries);
+void pblk_rl_inserted(struct pblk_rl *rl, int nr_entries);
void pblk_rl_user_in(struct pblk_rl *rl, int nr_entries);
int pblk_rl_gc_may_insert(struct pblk_rl *rl, int nr_entries);
void pblk_rl_gc_in(struct pblk_rl *rl, int nr_entries);
void pblk_rl_out(struct pblk_rl *rl, int nr_user, int nr_gc);
-void pblk_rl_set_gc_rsc(struct pblk_rl *rl, int rsv);
int pblk_rl_sysfs_rate_show(struct pblk_rl *rl);
void pblk_rl_free_lines_inc(struct pblk_rl *rl, struct pblk_line *line);
void pblk_rl_free_lines_dec(struct pblk_rl *rl, struct pblk_line *line);
+void pblk_rl_set_space_limit(struct pblk_rl *rl, int entries_left);
+int pblk_rl_is_limit(struct pblk_rl *rl);
/*
* pblk sysfs
return c_ctx - sizeof(struct nvm_rq);
}
-static inline void *pblk_line_emeta_to_lbas(struct line_emeta *emeta)
+static inline void *emeta_to_bb(struct line_emeta *emeta)
+{
+ return emeta->bb_bitmap;
+}
+
+static inline void *emeta_to_lbas(struct pblk *pblk, struct line_emeta *emeta)
+{
+ return ((void *)emeta + pblk->lm.emeta_len[1]);
+}
+
+static inline void *emeta_to_vsc(struct pblk *pblk, struct line_emeta *emeta)
{
- return (emeta) + 1;
+ return (emeta_to_lbas(pblk, emeta) + pblk->lm.emeta_len[2]);
+}
+
+static inline int pblk_line_vsc(struct pblk_line *line)
+{
+ int vsc;
+
+ spin_lock(&line->lock);
+ vsc = le32_to_cpu(*line->vsc);
+ spin_unlock(&line->lock);
+
+ return vsc;
}
#define NVM_MEM_PAGE_WRITE (8)
ppa_addr->ppa = ADDR_EMPTY;
}
+static inline bool pblk_ppa_comp(struct ppa_addr lppa, struct ppa_addr rppa)
+{
+ if (lppa.ppa == rppa.ppa)
+ return true;
+
+ return false;
+}
+
static inline int pblk_addr_in_cache(struct ppa_addr ppa)
{
return (ppa.ppa != ADDR_EMPTY && ppa.c.is_cached);
}
static inline u32 pblk_calc_meta_header_crc(struct pblk *pblk,
- struct line_smeta *smeta)
+ struct line_header *header)
{
u32 crc = ~(u32)0;
- crc = crc32_le(crc, (unsigned char *)smeta + sizeof(crc),
+ crc = crc32_le(crc, (unsigned char *)header + sizeof(crc),
sizeof(struct line_header) - sizeof(crc));
return crc;
crc = crc32_le(crc, (unsigned char *)emeta +
sizeof(struct line_header) + sizeof(crc),
- lm->emeta_len -
+ lm->emeta_len[0] -
sizeof(struct line_header) - sizeof(crc));
return crc;
return flags;
}
-static inline int pblk_set_read_mode(struct pblk *pblk)
+enum {
+ PBLK_READ_RANDOM = 0,
+ PBLK_READ_SEQUENTIAL = 1,
+};
+
+static inline int pblk_set_read_mode(struct pblk *pblk, int type)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ int flags;
+
+ flags = NVM_IO_SUSPEND | NVM_IO_SCRAMBLE_ENABLE;
+ if (type == PBLK_READ_SEQUENTIAL)
+ flags |= geo->plane_mode >> 1;
+
+ return flags;
+}
+
+static inline int pblk_io_aligned(struct pblk *pblk, int nr_secs)
{
- return NVM_IO_SNGL_ACCESS | NVM_IO_SUSPEND | NVM_IO_SCRAMBLE_ENABLE;
+ return !(nr_secs % pblk->min_write_pgs);
}
#ifdef CONFIG_NVM_DEBUG
{
struct completion *waiting = bio->bi_private;
- if (bio->bi_error)
- pr_err("nvm: gc request failed (%u).\n", bio->bi_error);
+ if (bio->bi_status)
+ pr_err("nvm: gc request failed (%u).\n", bio->bi_status);
complete(waiting);
}
goto finished;
}
wait_for_completion_io(&wait);
- if (bio->bi_error) {
+ if (bio->bi_status) {
rrpc_inflight_laddr_release(rrpc, rqd);
goto finished;
}
wait_for_completion_io(&wait);
rrpc_inflight_laddr_release(rrpc, rqd);
- if (bio->bi_error)
+ if (bio->bi_status)
goto finished;
bio_reset(bio);
struct nvm_rq *rqd;
int err;
- blk_queue_split(q, &bio, q->bio_split);
+ blk_queue_split(q, &bio);
if (bio_op(bio) == REQ_OP_DISCARD) {
rrpc_discard(rrpc, bio);
return 0;
}
-extern struct device_attribute macio_dev_attrs[];
+extern const struct attribute_group *macio_dev_groups[];
struct bus_type macio_bus_type = {
.name = "macio",
.shutdown = macio_device_shutdown,
.suspend = macio_device_suspend,
.resume = macio_device_resume,
- .dev_attrs = macio_dev_attrs,
+ .dev_groups = macio_dev_groups,
};
static int __init macio_bus_driver_init(void)
{ \
struct macio_dev *mdev = to_macio_device (dev); \
return sprintf (buf, format_string, mdev->ofdev.dev.of_node->field); \
-}
+} \
+static DEVICE_ATTR_RO(field);
static ssize_t
compatible_show (struct device *dev, struct device_attribute *attr, char *buf)
return length;
}
+static DEVICE_ATTR_RO(compatible);
static ssize_t modalias_show (struct device *dev, struct device_attribute *attr,
char *buf)
ofdev = to_platform_device(dev);
return sprintf(buf, "%s\n", ofdev->dev.of_node->full_name);
}
+static DEVICE_ATTR_RO(modalias);
+static DEVICE_ATTR_RO(devspec);
macio_config_of_attr (name, "%s\n");
macio_config_of_attr (type, "%s\n");
-struct device_attribute macio_dev_attrs[] = {
- __ATTR_RO(name),
- __ATTR_RO(type),
- __ATTR_RO(compatible),
- __ATTR_RO(modalias),
- __ATTR_RO(devspec),
- __ATTR_NULL
+static struct attribute *macio_dev_attrs[] = {
+ &dev_attr_name.attr,
+ &dev_attr_type.attr,
+ &dev_attr_compatible.attr,
+ &dev_attr_modalias.attr,
+ &dev_attr_devspec.attr,
+ NULL,
+};
+
+static const struct attribute_group macio_dev_group = {
+ .attrs = macio_dev_attrs,
+};
+
+const struct attribute_group *macio_dev_groups[] = {
+ &macio_dev_group,
+ NULL,
};
struct acpi_pcct_hw_reduced_type2 *pcct2_ss = chan->con_priv;
u32 id = chan - pcc_mbox_channels;
- doorbell_ack = &pcct2_ss->doorbell_ack_register;
+ doorbell_ack = &pcct2_ss->platform_ack_register;
doorbell_ack_preserve = pcct2_ss->ack_preserve_mask;
doorbell_ack_write = pcct2_ss->ack_write_mask;
static int pcc_parse_subspace_irq(int id,
struct acpi_pcct_hw_reduced *pcct_ss)
{
- pcc_doorbell_irq[id] = pcc_map_interrupt(pcct_ss->doorbell_interrupt,
+ pcc_doorbell_irq[id] = pcc_map_interrupt(pcct_ss->platform_interrupt,
(u32)pcct_ss->flags);
if (pcc_doorbell_irq[id] <= 0) {
pr_err("PCC GSI %d not registered\n",
- pcct_ss->doorbell_interrupt);
+ pcct_ss->platform_interrupt);
return -EINVAL;
}
struct acpi_pcct_hw_reduced_type2 *pcct2_ss = (void *)pcct_ss;
pcc_doorbell_ack_vaddr[id] = acpi_os_ioremap(
- pcct2_ss->doorbell_ack_register.address,
- pcct2_ss->doorbell_ack_register.bit_width / 8);
+ pcct2_ss->platform_ack_register.address,
+ pcct2_ss->platform_ack_register.bit_width / 8);
if (!pcc_doorbell_ack_vaddr[id]) {
pr_err("Failed to ioremap PCC ACK register\n");
return -ENOMEM;
/* Forward declarations */
-void bch_count_io_errors(struct cache *, int, const char *);
+void bch_count_io_errors(struct cache *, blk_status_t, const char *);
void bch_bbio_count_io_errors(struct cache_set *, struct bio *,
- int, const char *);
-void bch_bbio_endio(struct cache_set *, struct bio *, int, const char *);
+ blk_status_t, const char *);
+void bch_bbio_endio(struct cache_set *, struct bio *, blk_status_t,
+ const char *);
void bch_bbio_free(struct bio *, struct cache_set *);
struct bio *bch_bbio_alloc(struct cache_set *);
bch_submit_bbio(bio, b->c, &b->key, 0);
closure_sync(&cl);
- if (bio->bi_error)
+ if (bio->bi_status)
set_btree_node_io_error(b);
bch_bbio_free(bio, b->c);
struct closure *cl = bio->bi_private;
struct btree *b = container_of(cl, struct btree, io);
- if (bio->bi_error)
+ if (bio->bi_status)
set_btree_node_io_error(b);
- bch_bbio_count_io_errors(b->c, bio, bio->bi_error, "writing btree");
+ bch_bbio_count_io_errors(b->c, bio, bio->bi_status, "writing btree");
closure_put(cl);
}
struct btree_op {
/* for waiting on btree reserve in btree_split() */
- wait_queue_t wait;
+ wait_queue_entry_t wait;
/* Btree level at which we start taking write locks */
short lock;
struct bio_vec bv, cbv;
struct bvec_iter iter, citer = { 0 };
- check = bio_clone(bio, GFP_NOIO);
+ check = bio_clone_kmalloc(bio, GFP_NOIO);
if (!check)
return;
check->bi_opf = REQ_OP_READ;
/* IO errors */
-void bch_count_io_errors(struct cache *ca, int error, const char *m)
+void bch_count_io_errors(struct cache *ca, blk_status_t error, const char *m)
{
/*
* The halflife of an error is:
}
void bch_bbio_count_io_errors(struct cache_set *c, struct bio *bio,
- int error, const char *m)
+ blk_status_t error, const char *m)
{
struct bbio *b = container_of(bio, struct bbio, bio);
struct cache *ca = PTR_CACHE(c, &b->key, 0);
}
void bch_bbio_endio(struct cache_set *c, struct bio *bio,
- int error, const char *m)
+ blk_status_t error, const char *m)
{
struct closure *cl = bio->bi_private;
{
struct journal_write *w = bio->bi_private;
- cache_set_err_on(bio->bi_error, w->c, "journal io error");
+ cache_set_err_on(bio->bi_status, w->c, "journal io error");
closure_put(&w->c->journal.io);
}
struct moving_io *io = container_of(bio->bi_private,
struct moving_io, cl);
- if (bio->bi_error)
- io->op.error = bio->bi_error;
+ if (bio->bi_status)
+ io->op.status = bio->bi_status;
else if (!KEY_DIRTY(&b->key) &&
ptr_stale(io->op.c, &b->key, 0)) {
- io->op.error = -EINTR;
+ io->op.status = BLK_STS_IOERR;
}
- bch_bbio_endio(io->op.c, bio, bio->bi_error, "reading data to move");
+ bch_bbio_endio(io->op.c, bio, bio->bi_status, "reading data to move");
}
static void moving_init(struct moving_io *io)
struct moving_io *io = container_of(cl, struct moving_io, cl);
struct data_insert_op *op = &io->op;
- if (!op->error) {
+ if (!op->status) {
moving_init(io);
io->bio.bio.bi_iter.bi_sector = KEY_START(&io->w->key);
if (ret == -ESRCH) {
op->replace_collision = true;
} else if (ret) {
- op->error = -ENOMEM;
+ op->status = BLK_STS_RESOURCE;
op->insert_data_done = true;
}
struct closure *cl = bio->bi_private;
struct data_insert_op *op = container_of(cl, struct data_insert_op, cl);
- if (bio->bi_error) {
+ if (bio->bi_status) {
/* TODO: We could try to recover from this. */
if (op->writeback)
- op->error = bio->bi_error;
+ op->status = bio->bi_status;
else if (!op->replace)
set_closure_fn(cl, bch_data_insert_error, op->wq);
else
set_closure_fn(cl, NULL, NULL);
}
- bch_bbio_endio(op->c, bio, bio->bi_error, "writing data to cache");
+ bch_bbio_endio(op->c, bio, bio->bi_status, "writing data to cache");
}
static void bch_data_insert_start(struct closure *cl)
* from the backing device.
*/
- if (bio->bi_error)
- s->iop.error = bio->bi_error;
+ if (bio->bi_status)
+ s->iop.status = bio->bi_status;
else if (!KEY_DIRTY(&b->key) &&
ptr_stale(s->iop.c, &b->key, 0)) {
atomic_long_inc(&s->iop.c->cache_read_races);
- s->iop.error = -EINTR;
+ s->iop.status = BLK_STS_IOERR;
}
- bch_bbio_endio(s->iop.c, bio, bio->bi_error, "reading from cache");
+ bch_bbio_endio(s->iop.c, bio, bio->bi_status, "reading from cache");
}
/*
{
struct closure *cl = bio->bi_private;
- if (bio->bi_error) {
+ if (bio->bi_status) {
struct search *s = container_of(cl, struct search, cl);
- s->iop.error = bio->bi_error;
+ s->iop.status = bio->bi_status;
/* Only cache read errors are recoverable */
s->recoverable = false;
}
&s->d->disk->part0, s->start_time);
trace_bcache_request_end(s->d, s->orig_bio);
- s->orig_bio->bi_error = s->iop.error;
+ s->orig_bio->bi_status = s->iop.status;
bio_endio(s->orig_bio);
s->orig_bio = NULL;
}
s->iop.inode = d->id;
s->iop.write_point = hash_long((unsigned long) current, 16);
s->iop.write_prio = 0;
- s->iop.error = 0;
+ s->iop.status = 0;
s->iop.flags = 0;
s->iop.flush_journal = op_is_flush(bio->bi_opf);
s->iop.wq = bcache_wq;
/* Retry from the backing device: */
trace_bcache_read_retry(s->orig_bio);
- s->iop.error = 0;
+ s->iop.status = 0;
do_bio_hook(s, s->orig_bio);
/* XXX: invalidate cache */
!s->cache_miss, s->iop.bypass);
trace_bcache_read(s->orig_bio, !s->cache_miss, s->iop.bypass);
- if (s->iop.error)
+ if (s->iop.status)
continue_at_nobarrier(cl, cached_dev_read_error, bcache_wq);
else if (s->iop.bio || verify(dc, &s->bio.bio))
continue_at_nobarrier(cl, cached_dev_read_done, bcache_wq);
unsigned inode;
uint16_t write_point;
uint16_t write_prio;
- short error;
+ blk_status_t status;
union {
uint16_t flags;
{
struct cache *ca = bio->bi_private;
- bch_count_io_errors(ca, bio->bi_error, "writing superblock");
+ bch_count_io_errors(ca, bio->bi_status, "writing superblock");
closure_put(&ca->set->sb_write);
}
struct closure *cl = bio->bi_private;
struct cache_set *c = container_of(cl, struct cache_set, uuid_write);
- cache_set_err_on(bio->bi_error, c, "accessing uuids");
+ cache_set_err_on(bio->bi_status, c, "accessing uuids");
bch_bbio_free(bio, c);
closure_put(cl);
}
{
struct cache *ca = bio->bi_private;
- cache_set_err_on(bio->bi_error, ca->set, "accessing priorities");
+ cache_set_err_on(bio->bi_status, ca->set, "accessing priorities");
bch_bbio_free(bio, ca->set);
closure_put(&ca->prio);
}
minor *= BCACHE_MINORS;
- if (!(d->bio_split = bioset_create(4, offsetof(struct bbio, bio))) ||
+ if (!(d->bio_split = bioset_create(4, offsetof(struct bbio, bio),
+ BIOSET_NEED_BVECS |
+ BIOSET_NEED_RESCUER)) ||
!(d->disk = alloc_disk(BCACHE_MINORS))) {
ida_simple_remove(&bcache_minor, minor);
return -ENOMEM;
sizeof(struct bbio) + sizeof(struct bio_vec) *
bucket_pages(c))) ||
!(c->fill_iter = mempool_create_kmalloc_pool(1, iter_size)) ||
- !(c->bio_split = bioset_create(4, offsetof(struct bbio, bio))) ||
+ !(c->bio_split = bioset_create(4, offsetof(struct bbio, bio),
+ BIOSET_NEED_BVECS |
+ BIOSET_NEED_RESCUER)) ||
!(c->uuids = alloc_bucket_pages(GFP_KERNEL, c)) ||
!(c->moving_gc_wq = alloc_workqueue("bcache_gc",
WQ_MEM_RECLAIM, 0)) ||
struct keybuf_key *w = bio->bi_private;
struct dirty_io *io = w->private;
- if (bio->bi_error)
+ if (bio->bi_status)
SET_KEY_DIRTY(&w->key, false);
closure_put(&io->cl);
struct dirty_io *io = w->private;
bch_count_io_errors(PTR_CACHE(io->dc->disk.c, &w->key, 0),
- bio->bi_error, "reading dirty data from cache");
+ bio->bi_status, "reading dirty data from cache");
dirty_endio(bio);
}
EXPORT_SYMBOL_GPL(dm_cell_release_no_holder);
void dm_cell_error(struct dm_bio_prison *prison,
- struct dm_bio_prison_cell *cell, int error)
+ struct dm_bio_prison_cell *cell, blk_status_t error)
{
struct bio_list bios;
struct bio *bio;
dm_cell_release(prison, cell, &bios);
while ((bio = bio_list_pop(&bios))) {
- bio->bi_error = error;
+ bio->bi_status = error;
bio_endio(bio);
}
}
struct dm_bio_prison_cell *cell,
struct bio_list *inmates);
void dm_cell_error(struct dm_bio_prison *prison,
- struct dm_bio_prison_cell *cell, int error);
+ struct dm_bio_prison_cell *cell, blk_status_t error);
/*
* Visits the cell and then releases. Guarantees no new inmates are
enum data_mode data_mode;
unsigned char list_mode; /* LIST_* */
unsigned hold_count;
- int read_error;
- int write_error;
+ blk_status_t read_error;
+ blk_status_t write_error;
unsigned long state;
unsigned long last_accessed;
struct dm_bufio_client *c;
{
struct dm_buffer *b = context;
- b->bio.bi_error = error ? -EIO : 0;
+ b->bio.bi_status = error ? BLK_STS_IOERR : 0;
b->bio.bi_end_io(&b->bio);
}
r = dm_io(&io_req, 1, ®ion, NULL);
if (r) {
- b->bio.bi_error = r;
+ b->bio.bi_status = errno_to_blk_status(r);
end_io(&b->bio);
}
}
static void inline_endio(struct bio *bio)
{
bio_end_io_t *end_fn = bio->bi_private;
- int error = bio->bi_error;
+ blk_status_t status = bio->bi_status;
/*
* Reset the bio to free any attached resources
*/
bio_reset(bio);
- bio->bi_error = error;
+ bio->bi_status = status;
end_fn(bio);
}
{
struct dm_buffer *b = container_of(bio, struct dm_buffer, bio);
- b->write_error = bio->bi_error;
- if (unlikely(bio->bi_error)) {
+ b->write_error = bio->bi_status;
+ if (unlikely(bio->bi_status)) {
struct dm_bufio_client *c = b->c;
- int error = bio->bi_error;
- (void)cmpxchg(&c->async_write_error, 0, error);
+
+ (void)cmpxchg(&c->async_write_error, 0,
+ blk_status_to_errno(bio->bi_status));
}
BUG_ON(!test_bit(B_WRITING, &b->state));
{
struct dm_buffer *b = container_of(bio, struct dm_buffer, bio);
- b->read_error = bio->bi_error;
+ b->read_error = bio->bi_status;
BUG_ON(!test_bit(B_READING, &b->state));
wait_on_bit_io(&b->state, B_READING, TASK_UNINTERRUPTIBLE);
if (b->read_error) {
- int error = b->read_error;
+ int error = blk_status_to_errno(b->read_error);
dm_bufio_release(b);
*/
int dm_bufio_write_dirty_buffers(struct dm_bufio_client *c)
{
- int a, f;
+ blk_status_t a;
+ int f;
unsigned long buffers_processed = 0;
struct dm_buffer *b, *tmp;
*/
struct continuation {
struct work_struct ws;
- int input;
+ blk_status_t input;
};
static inline void init_continuation(struct continuation *k,
/*
* The operation that everyone is waiting for.
*/
- int (*commit_op)(void *context);
+ blk_status_t (*commit_op)(void *context);
void *commit_context;
/*
static void __commit(struct work_struct *_ws)
{
struct batcher *b = container_of(_ws, struct batcher, commit_work);
-
- int r;
+ blk_status_t r;
unsigned long flags;
struct list_head work_items;
struct work_struct *ws, *tmp;
while ((bio = bio_list_pop(&bios))) {
if (r) {
- bio->bi_error = r;
+ bio->bi_status = r;
bio_endio(bio);
} else
b->issue_op(bio, b->issue_context);
}
static void batcher_init(struct batcher *b,
- int (*commit_op)(void *),
+ blk_status_t (*commit_op)(void *),
void *commit_context,
void (*issue_op)(struct bio *bio, void *),
void *issue_context,
dm_unhook_bio(&pb->hook_info, bio);
- if (bio->bi_error) {
+ if (bio->bi_status) {
bio_endio(bio);
return;
}
struct dm_cache_migration *mg = container_of(context, struct dm_cache_migration, k);
if (read_err || write_err)
- mg->k.input = -EIO;
+ mg->k.input = BLK_STS_IOERR;
queue_continuation(mg->cache->wq, &mg->k);
}
dm_unhook_bio(&pb->hook_info, bio);
- if (bio->bi_error)
- mg->k.input = bio->bi_error;
+ if (bio->bi_status)
+ mg->k.input = bio->bi_status;
queue_continuation(mg->cache->wq, &mg->k);
}
if (mg->overwrite_bio) {
if (success)
force_set_dirty(cache, cblock);
+ else if (mg->k.input)
+ mg->overwrite_bio->bi_status = mg->k.input;
else
- mg->overwrite_bio->bi_error = (mg->k.input ? : -EIO);
+ mg->overwrite_bio->bi_status = BLK_STS_IOERR;
bio_endio(mg->overwrite_bio);
} else {
if (success)
r = copy(mg, is_policy_promote);
if (r) {
DMERR_LIMIT("%s: migration copy failed", cache_device_name(cache));
- mg->k.input = -EIO;
+ mg->k.input = BLK_STS_IOERR;
mg_complete(mg, false);
}
}
/*
* Used by the batcher.
*/
-static int commit_op(void *context)
+static blk_status_t commit_op(void *context)
{
struct cache *cache = context;
if (dm_cache_changed_this_transaction(cache->cmd))
- return commit(cache, false);
+ return errno_to_blk_status(commit(cache, false));
return 0;
}
bio_list_init(&cache->deferred_bios);
while ((bio = bio_list_pop(&bios))) {
- bio->bi_error = DM_ENDIO_REQUEUE;
+ bio->bi_status = BLK_STS_DM_REQUEUE;
bio_endio(bio);
}
}
return r;
}
-static int cache_end_io(struct dm_target *ti, struct bio *bio, int error)
+static int cache_end_io(struct dm_target *ti, struct bio *bio,
+ blk_status_t *error)
{
struct cache *cache = ti->private;
unsigned long flags;
bio_drop_shared_lock(cache, bio);
accounted_complete(cache, bio);
- return 0;
+ return DM_ENDIO_DONE;
}
static int write_dirty_bitset(struct cache *cache)
struct convert_context ctx;
atomic_t io_pending;
- int error;
+ blk_status_t error;
sector_t sector;
struct rb_node rb_node;
/*
* Encrypt / decrypt data from one bio to another one (can be the same one)
*/
-static int crypt_convert(struct crypt_config *cc,
+static blk_status_t crypt_convert(struct crypt_config *cc,
struct convert_context *ctx)
{
unsigned int tag_offset = 0;
*/
case -EBADMSG:
atomic_dec(&ctx->cc_pending);
- return -EILSEQ;
+ return BLK_STS_PROTECTION;
/*
* There was an error while processing the request.
*/
default:
atomic_dec(&ctx->cc_pending);
- return -EIO;
+ return BLK_STS_IOERR;
}
}
{
struct crypt_config *cc = io->cc;
struct bio *base_bio = io->base_bio;
- int error = io->error;
+ blk_status_t error = io->error;
if (!atomic_dec_and_test(&io->io_pending))
return;
else
kfree(io->integrity_metadata);
- base_bio->bi_error = error;
+ base_bio->bi_status = error;
bio_endio(base_bio);
}
struct dm_crypt_io *io = clone->bi_private;
struct crypt_config *cc = io->cc;
unsigned rw = bio_data_dir(clone);
- int error;
+ blk_status_t error;
/*
* free the processed pages
if (rw == WRITE)
crypt_free_buffer_pages(cc, clone);
- error = clone->bi_error;
+ error = clone->bi_status;
bio_put(clone);
if (rw == READ && !error) {
crypt_inc_pending(io);
if (kcryptd_io_read(io, GFP_NOIO))
- io->error = -ENOMEM;
+ io->error = BLK_STS_RESOURCE;
crypt_dec_pending(io);
}
sector_t sector;
struct rb_node **rbp, *parent;
- if (unlikely(io->error < 0)) {
+ if (unlikely(io->error)) {
crypt_free_buffer_pages(cc, clone);
bio_put(clone);
crypt_dec_pending(io);
struct bio *clone;
int crypt_finished;
sector_t sector = io->sector;
- int r;
+ blk_status_t r;
/*
* Prevent io from disappearing until this function completes.
clone = crypt_alloc_buffer(io, io->base_bio->bi_iter.bi_size);
if (unlikely(!clone)) {
- io->error = -EIO;
+ io->error = BLK_STS_IOERR;
goto dec;
}
crypt_inc_pending(io);
r = crypt_convert(cc, &io->ctx);
- if (r < 0)
+ if (r)
io->error = r;
crypt_finished = atomic_dec_and_test(&io->ctx.cc_pending);
static void kcryptd_crypt_read_convert(struct dm_crypt_io *io)
{
struct crypt_config *cc = io->cc;
- int r = 0;
+ blk_status_t r;
crypt_inc_pending(io);
io->sector);
r = crypt_convert(cc, &io->ctx);
- if (r < 0)
+ if (r)
io->error = r;
if (atomic_dec_and_test(&io->ctx.cc_pending))
if (error == -EBADMSG) {
DMERR_LIMIT("INTEGRITY AEAD ERROR, sector %llu",
(unsigned long long)le64_to_cpu(*org_sector_of_dmreq(cc, dmreq)));
- io->error = -EILSEQ;
+ io->error = BLK_STS_PROTECTION;
} else if (error < 0)
- io->error = -EIO;
+ io->error = BLK_STS_IOERR;
crypt_free_req(cc, req_of_dmreq(cc, dmreq), io->base_bio);
goto bad;
}
- cc->bs = bioset_create(MIN_IOS, 0);
+ cc->bs = bioset_create(MIN_IOS, 0, (BIOSET_NEED_BVECS |
+ BIOSET_NEED_RESCUER));
if (!cc->bs) {
ti->error = "Cannot allocate crypt bioset";
goto bad;
* and is aligned to this size as defined in IO hints.
*/
if (unlikely((bio->bi_iter.bi_sector & ((cc->sector_size >> SECTOR_SHIFT) - 1)) != 0))
- return -EIO;
+ return DM_MAPIO_KILL;
if (unlikely(bio->bi_iter.bi_size & (cc->sector_size - 1)))
- return -EIO;
+ return DM_MAPIO_KILL;
io = dm_per_bio_data(bio, cc->per_bio_data_size);
crypt_io_init(io, cc, bio, dm_target_offset(ti, bio->bi_iter.bi_sector));
if (bio_data_dir(bio) == READ) {
if (!fc->corrupt_bio_byte && !test_bit(DROP_WRITES, &fc->flags) &&
!test_bit(ERROR_WRITES, &fc->flags))
- return -EIO;
+ return DM_MAPIO_KILL;
goto map_bio;
}
/*
* By default, error all I/O.
*/
- return -EIO;
+ return DM_MAPIO_KILL;
}
map_bio:
return DM_MAPIO_REMAPPED;
}
-static int flakey_end_io(struct dm_target *ti, struct bio *bio, int error)
+static int flakey_end_io(struct dm_target *ti, struct bio *bio,
+ blk_status_t *error)
{
struct flakey_c *fc = ti->private;
struct per_bio_data *pb = dm_per_bio_data(bio, sizeof(struct per_bio_data));
- if (!error && pb->bio_submitted && (bio_data_dir(bio) == READ)) {
+ if (!*error && pb->bio_submitted && (bio_data_dir(bio) == READ)) {
if (fc->corrupt_bio_byte && (fc->corrupt_bio_rw == READ) &&
all_corrupt_bio_flags_match(bio, fc)) {
/*
* Error read during the down_interval if drop_writes
* and error_writes were not configured.
*/
- return -EIO;
+ *error = BLK_STS_IOERR;
}
}
- return error;
+ return DM_ENDIO_DONE;
}
static void flakey_status(struct dm_target *ti, status_type_t type,
unsigned metadata_offset;
atomic_t in_flight;
- int bi_error;
+ blk_status_t bi_status;
struct completion *completion;
static void submit_flush_bio(struct dm_integrity_c *ic, struct dm_integrity_io *dio)
{
struct bio *bio;
- spin_lock_irq(&ic->endio_wait.lock);
+ unsigned long flags;
+
+ spin_lock_irqsave(&ic->endio_wait.lock, flags);
bio = dm_bio_from_per_bio_data(dio, sizeof(struct dm_integrity_io));
bio_list_add(&ic->flush_bio_list, bio);
- spin_unlock_irq(&ic->endio_wait.lock);
+ spin_unlock_irqrestore(&ic->endio_wait.lock, flags);
+
queue_work(ic->commit_wq, &ic->commit_work);
}
static void do_endio(struct dm_integrity_c *ic, struct bio *bio)
{
int r = dm_integrity_failed(ic);
- if (unlikely(r) && !bio->bi_error)
- bio->bi_error = r;
+ if (unlikely(r) && !bio->bi_status)
+ bio->bi_status = errno_to_blk_status(r);
bio_endio(bio);
}
{
struct bio *bio = dm_bio_from_per_bio_data(dio, sizeof(struct dm_integrity_io));
- if (unlikely(dio->fua) && likely(!bio->bi_error) && likely(!dm_integrity_failed(ic)))
+ if (unlikely(dio->fua) && likely(!bio->bi_status) && likely(!dm_integrity_failed(ic)))
submit_flush_bio(ic, dio);
else
do_endio(ic, bio);
bio = dm_bio_from_per_bio_data(dio, sizeof(struct dm_integrity_io));
- if (unlikely(dio->bi_error) && !bio->bi_error)
- bio->bi_error = dio->bi_error;
- if (likely(!bio->bi_error) && unlikely(bio_sectors(bio) != dio->range.n_sectors)) {
+ if (unlikely(dio->bi_status) && !bio->bi_status)
+ bio->bi_status = dio->bi_status;
+ if (likely(!bio->bi_status) && unlikely(bio_sectors(bio) != dio->range.n_sectors)) {
dio->range.logical_sector += dio->range.n_sectors;
bio_advance(bio, dio->range.n_sectors << SECTOR_SHIFT);
INIT_WORK(&dio->work, integrity_bio_wait);
dec_in_flight(dio);
return;
error:
- dio->bi_error = r;
+ dio->bi_status = errno_to_blk_status(r);
dec_in_flight(dio);
}
sector_t area, offset;
dio->ic = ic;
- dio->bi_error = 0;
+ dio->bi_status = 0;
if (unlikely(bio->bi_opf & REQ_PREFLUSH)) {
submit_flush_bio(ic, dio);
DMERR("Too big sector number: 0x%llx + 0x%x > 0x%llx",
(unsigned long long)dio->range.logical_sector, bio_sectors(bio),
(unsigned long long)ic->provided_data_sectors);
- return -EIO;
+ return DM_MAPIO_KILL;
}
if (unlikely((dio->range.logical_sector | bio_sectors(bio)) & (unsigned)(ic->sectors_per_block - 1))) {
DMERR("Bio not aligned on %u sectors: 0x%llx, 0x%x",
ic->sectors_per_block,
(unsigned long long)dio->range.logical_sector, bio_sectors(bio));
- return -EIO;
+ return DM_MAPIO_KILL;
}
if (ic->sectors_per_block > 1) {
if (unlikely((bv.bv_offset | bv.bv_len) & ((ic->sectors_per_block << SECTOR_SHIFT) - 1))) {
DMERR("Bio vector (%u,%u) is not aligned on %u-sector boundary",
bv.bv_offset, bv.bv_len, ic->sectors_per_block);
- return -EIO;
+ return DM_MAPIO_KILL;
}
}
}
wanted_tag_size *= ic->tag_size;
if (unlikely(wanted_tag_size != bip->bip_iter.bi_size)) {
DMERR("Invalid integrity data size %u, expected %u", bip->bip_iter.bi_size, wanted_tag_size);
- return -EIO;
+ return DM_MAPIO_KILL;
}
}
} else {
if (unlikely(bip != NULL)) {
DMERR("Unexpected integrity data when using internal hash");
- return -EIO;
+ return DM_MAPIO_KILL;
}
}
if (unlikely(ic->mode == 'R') && unlikely(dio->write))
- return -EIO;
+ return DM_MAPIO_KILL;
get_area_and_offset(ic, dio->range.logical_sector, &area, &offset);
dio->metadata_block = get_metadata_sector_and_offset(ic, area, offset, &dio->metadata_offset);
ti->error = "The device is too small";
goto bad;
}
+ if (ti->len > ic->provided_data_sectors) {
+ r = -EINVAL;
+ ti->error = "Not enough provided sectors for requested mapping size";
+ goto bad;
+ }
if (!buffer_sectors)
buffer_sectors = 1;
if (!client->pool)
goto bad;
- client->bios = bioset_create(min_ios, 0);
+ client->bios = bioset_create(min_ios, 0, (BIOSET_NEED_BVECS |
+ BIOSET_NEED_RESCUER));
if (!client->bios)
goto bad;
fn(error_bits, context);
}
-static void dec_count(struct io *io, unsigned int region, int error)
+static void dec_count(struct io *io, unsigned int region, blk_status_t error)
{
if (error)
set_bit(region, &io->error_bits);
{
struct io *io;
unsigned region;
- int error;
+ blk_status_t error;
- if (bio->bi_error && bio_data_dir(bio) == READ)
+ if (bio->bi_status && bio_data_dir(bio) == READ)
zero_fill_bio(bio);
/*
*/
retrieve_io_and_region_from_bio(bio, &io, ®ion);
- error = bio->bi_error;
+ error = bio->bi_status;
bio_put(bio);
dec_count(io, region, error);
else if (op == REQ_OP_WRITE_SAME)
special_cmd_max_sectors = q->limits.max_write_same_sectors;
if ((op == REQ_OP_DISCARD || op == REQ_OP_WRITE_ZEROES ||
- op == REQ_OP_WRITE_SAME) &&
- special_cmd_max_sectors == 0) {
- dec_count(io, region, -EOPNOTSUPP);
+ op == REQ_OP_WRITE_SAME) && special_cmd_max_sectors == 0) {
+ atomic_inc(&io->count);
+ dec_count(io, region, BLK_STS_NOTSUPP);
return;
}
{
struct log_writes_c *lc = bio->bi_private;
- if (bio->bi_error) {
+ if (bio->bi_status) {
unsigned long flags;
- DMERR("Error writing log block, error=%d", bio->bi_error);
+ DMERR("Error writing log block, error=%d", bio->bi_status);
spin_lock_irqsave(&lc->blocks_lock, flags);
lc->logging_enabled = false;
spin_unlock_irqrestore(&lc->blocks_lock, flags);
spin_lock_irq(&lc->blocks_lock);
lc->logging_enabled = false;
spin_unlock_irq(&lc->blocks_lock);
- return -ENOMEM;
+ return DM_MAPIO_KILL;
}
INIT_LIST_HEAD(&block->list);
pb->block = block;
spin_lock_irq(&lc->blocks_lock);
lc->logging_enabled = false;
spin_unlock_irq(&lc->blocks_lock);
- return -ENOMEM;
+ return DM_MAPIO_KILL;
}
src = kmap_atomic(bv.bv_page);
return DM_MAPIO_REMAPPED;
}
-static int normal_end_io(struct dm_target *ti, struct bio *bio, int error)
+static int normal_end_io(struct dm_target *ti, struct bio *bio,
+ blk_status_t *error)
{
struct log_writes_c *lc = ti->private;
struct per_bio_data *pb = dm_per_bio_data(bio, sizeof(struct per_bio_data));
spin_unlock_irqrestore(&lc->blocks_lock, flags);
}
- return error;
+ return DM_ENDIO_DONE;
}
/*
if (test_bit(MPATHF_QUEUE_IF_NO_PATH, &m->flags))
return DM_MAPIO_REQUEUE;
dm_report_EIO(m);
- return -EIO;
+ return DM_MAPIO_KILL;
}
mpio->pgpath = pgpath;
mpio->nr_bytes = nr_bytes;
- bio->bi_error = 0;
+ bio->bi_status = 0;
bio->bi_bdev = pgpath->path.dev->bdev;
bio->bi_opf |= REQ_FAILFAST_TRANSPORT;
blk_start_plug(&plug);
while ((bio = bio_list_pop(&bios))) {
r = __multipath_map_bio(m, bio, get_mpio_from_bio(bio));
- if (r < 0 || r == DM_MAPIO_REQUEUE) {
- bio->bi_error = r;
+ switch (r) {
+ case DM_MAPIO_KILL:
+ bio->bi_status = BLK_STS_IOERR;
+ bio_endio(bio);
+ break;
+ case DM_MAPIO_REQUEUE:
+ bio->bi_status = BLK_STS_DM_REQUEUE;
bio_endio(bio);
- } else if (r == DM_MAPIO_REMAPPED)
+ break;
+ case DM_MAPIO_REMAPPED:
generic_make_request(bio);
+ break;
+ }
}
blk_finish_plug(&plug);
}
activate_or_offline_path(pgpath);
}
-static int noretry_error(int error)
+static int noretry_error(blk_status_t error)
{
switch (error) {
- case -EBADE:
- /*
- * EBADE signals an reservation conflict.
- * We shouldn't fail the path here as we can communicate with
- * the target. We should failover to the next path, but in
- * doing so we might be causing a ping-pong between paths.
- * So just return the reservation conflict error.
- */
- case -EOPNOTSUPP:
- case -EREMOTEIO:
- case -EILSEQ:
- case -ENODATA:
- case -ENOSPC:
+ case BLK_STS_NOTSUPP:
+ case BLK_STS_NOSPC:
+ case BLK_STS_TARGET:
+ case BLK_STS_NEXUS:
+ case BLK_STS_MEDIUM:
+ case BLK_STS_RESOURCE:
return 1;
}
}
static int multipath_end_io(struct dm_target *ti, struct request *clone,
- int error, union map_info *map_context)
+ blk_status_t error, union map_info *map_context)
{
struct dm_mpath_io *mpio = get_mpio(map_context);
struct pgpath *pgpath = mpio->pgpath;
if (atomic_read(&m->nr_valid_paths) == 0 &&
!test_bit(MPATHF_QUEUE_IF_NO_PATH, &m->flags)) {
- if (error == -EIO)
+ if (error == BLK_STS_IOERR)
dm_report_EIO(m);
/* complete with the original error */
r = DM_ENDIO_DONE;
return r;
}
-static int do_end_io_bio(struct multipath *m, struct bio *clone,
- int error, struct dm_mpath_io *mpio)
+static int multipath_end_io_bio(struct dm_target *ti, struct bio *clone,
+ blk_status_t *error)
{
+ struct multipath *m = ti->private;
+ struct dm_mpath_io *mpio = get_mpio_from_bio(clone);
+ struct pgpath *pgpath = mpio->pgpath;
unsigned long flags;
+ int r = DM_ENDIO_DONE;
- if (!error)
- return 0; /* I/O complete */
-
- if (noretry_error(error))
- return error;
+ if (!*error || noretry_error(*error))
+ goto done;
- if (mpio->pgpath)
- fail_path(mpio->pgpath);
+ if (pgpath)
+ fail_path(pgpath);
if (atomic_read(&m->nr_valid_paths) == 0 &&
!test_bit(MPATHF_QUEUE_IF_NO_PATH, &m->flags)) {
dm_report_EIO(m);
- return -EIO;
+ *error = BLK_STS_IOERR;
+ goto done;
}
/* Queue for the daemon to resubmit */
if (!test_bit(MPATHF_QUEUE_IO, &m->flags))
queue_work(kmultipathd, &m->process_queued_bios);
- return DM_ENDIO_INCOMPLETE;
-}
-
-static int multipath_end_io_bio(struct dm_target *ti, struct bio *clone, int error)
-{
- struct multipath *m = ti->private;
- struct dm_mpath_io *mpio = get_mpio_from_bio(clone);
- struct pgpath *pgpath;
- struct path_selector *ps;
- int r;
-
- BUG_ON(!mpio);
-
- r = do_end_io_bio(m, clone, error, mpio);
- pgpath = mpio->pgpath;
+ r = DM_ENDIO_INCOMPLETE;
+done:
if (pgpath) {
- ps = &pgpath->pg->ps;
+ struct path_selector *ps = &pgpath->pg->ps;
+
if (ps->type->end_io)
ps->type->end_io(ps, &pgpath->path, mpio->nr_bytes);
}
/********************************************************************
* BELOW FOLLOW V1.9.0 EXTENSIONS TO THE PRISTINE SUPERBLOCK FORMAT!!!
*
- * FEATURE_FLAG_SUPPORTS_V190 in the features member indicates that those exist
+ * FEATURE_FLAG_SUPPORTS_V190 in the compat_features member indicates that those exist
*/
__le32 flags; /* Flags defining array states for reshaping */
sb->layout = cpu_to_le32(mddev->layout);
sb->stripe_sectors = cpu_to_le32(mddev->chunk_sectors);
+ /********************************************************************
+ * BELOW FOLLOW V1.9.0 EXTENSIONS TO THE PRISTINE SUPERBLOCK FORMAT!!!
+ *
+ * FEATURE_FLAG_SUPPORTS_V190 in the compat_features member indicates that those exist
+ */
sb->new_level = cpu_to_le32(mddev->new_level);
sb->new_layout = cpu_to_le32(mddev->new_layout);
sb->new_stripe_sectors = cpu_to_le32(mddev->new_chunk_sectors);
mddev->bitmap_info.default_offset = mddev->bitmap_info.offset;
if (!test_and_clear_bit(FirstUse, &rdev->flags)) {
- /* Retrieve device size stored in superblock to be prepared for shrink */
- rdev->sectors = le64_to_cpu(sb->sectors);
+ /*
+ * Retrieve rdev size stored in superblock to be prepared for shrink.
+ * Check extended superblock members are present otherwise the size
+ * will not be set!
+ */
+ if (le32_to_cpu(sb->compat_features) & FEATURE_FLAG_SUPPORTS_V190)
+ rdev->sectors = le64_to_cpu(sb->sectors);
+
rdev->recovery_offset = le64_to_cpu(sb->disk_recovery_offset);
if (rdev->recovery_offset == MaxSector)
set_bit(In_sync, &rdev->flags);
struct dm_raid1_bio_record {
struct mirror *m;
+ /* if details->bi_bdev == NULL, details were not saved */
struct dm_bio_details details;
region_t write_region;
};
* If device is suspended, complete the bio.
*/
if (dm_noflush_suspending(ms->ti))
- bio->bi_error = DM_ENDIO_REQUEUE;
+ bio->bi_status = BLK_STS_DM_REQUEUE;
else
- bio->bi_error = -EIO;
+ bio->bi_status = BLK_STS_IOERR;
bio_endio(bio);
return;
* degrade the array.
*/
if (bio_op(bio) == REQ_OP_DISCARD) {
- bio->bi_error = -EOPNOTSUPP;
+ bio->bi_status = BLK_STS_NOTSUPP;
bio_endio(bio);
return;
}
struct dm_raid1_bio_record *bio_record =
dm_per_bio_data(bio, sizeof(struct dm_raid1_bio_record));
+ bio_record->details.bi_bdev = NULL;
+
if (rw == WRITE) {
/* Save region for mirror_end_io() handler */
bio_record->write_region = dm_rh_bio_to_region(ms->rh, bio);
r = log->type->in_sync(log, dm_rh_bio_to_region(ms->rh, bio), 0);
if (r < 0 && r != -EWOULDBLOCK)
- return r;
+ return DM_MAPIO_KILL;
/*
* If region is not in-sync queue the bio.
*/
if (!r || (r == -EWOULDBLOCK)) {
if (bio->bi_opf & REQ_RAHEAD)
- return -EWOULDBLOCK;
+ return DM_MAPIO_KILL;
queue_bio(ms, bio, rw);
return DM_MAPIO_SUBMITTED;
*/
m = choose_mirror(ms, bio->bi_iter.bi_sector);
if (unlikely(!m))
- return -EIO;
+ return DM_MAPIO_KILL;
dm_bio_record(&bio_record->details, bio);
bio_record->m = m;
return DM_MAPIO_REMAPPED;
}
-static int mirror_end_io(struct dm_target *ti, struct bio *bio, int error)
+static int mirror_end_io(struct dm_target *ti, struct bio *bio,
+ blk_status_t *error)
{
int rw = bio_data_dir(bio);
struct mirror_set *ms = (struct mirror_set *) ti->private;
if (!(bio->bi_opf & REQ_PREFLUSH) &&
bio_op(bio) != REQ_OP_DISCARD)
dm_rh_dec(ms->rh, bio_record->write_region);
- return error;
+ return DM_ENDIO_DONE;
}
- if (error == -EOPNOTSUPP)
- return error;
+ if (*error == BLK_STS_NOTSUPP)
+ goto out;
+
+ if (bio->bi_opf & REQ_RAHEAD)
+ goto out;
- if ((error == -EWOULDBLOCK) && (bio->bi_opf & REQ_RAHEAD))
- return error;
+ if (unlikely(*error)) {
+ if (!bio_record->details.bi_bdev) {
+ /*
+ * There wasn't enough memory to record necessary
+ * information for a retry or there was no other
+ * mirror in-sync.
+ */
+ DMERR_LIMIT("Mirror read failed.");
+ return DM_ENDIO_DONE;
+ }
- if (unlikely(error)) {
m = bio_record->m;
DMERR("Mirror read failed from %s. Trying alternative device.",
bd = &bio_record->details;
dm_bio_restore(bd, bio);
- bio->bi_error = 0;
+ bio_record->details.bi_bdev = NULL;
+ bio->bi_status = 0;
queue_bio(ms, bio, rw);
return DM_ENDIO_INCOMPLETE;
DMERR("All replicated volumes dead, failing I/O");
}
- return error;
+out:
+ bio_record->details.bi_bdev = NULL;
+
+ return DM_ENDIO_DONE;
}
static void mirror_presuspend(struct dm_target *ti)
static void dm_mq_start_queue(struct request_queue *q)
{
- blk_mq_start_stopped_hw_queues(q, true);
+ blk_mq_unquiesce_queue(q);
blk_mq_kick_requeue_list(q);
}
struct dm_rq_target_io *tio = info->tio;
struct bio *bio = info->orig;
unsigned int nr_bytes = info->orig->bi_iter.bi_size;
- int error = clone->bi_error;
+ blk_status_t error = clone->bi_status;
bio_put(clone);
* Do not use blk_end_request() here, because it may complete
* the original request before the clone, and break the ordering.
*/
- blk_update_request(tio->orig, 0, nr_bytes);
+ blk_update_request(tio->orig, BLK_STS_OK, nr_bytes);
}
static struct dm_rq_target_io *tio_from_request(struct request *rq)
* Must be called without clone's queue lock held,
* see end_clone_request() for more details.
*/
-static void dm_end_request(struct request *clone, int error)
+static void dm_end_request(struct request *clone, blk_status_t error)
{
int rw = rq_data_dir(clone);
struct dm_rq_target_io *tio = clone->end_io_data;
rq_completed(md, rw, false);
}
-static void dm_done(struct request *clone, int error, bool mapped)
+static void dm_done(struct request *clone, blk_status_t error, bool mapped)
{
int r = DM_ENDIO_DONE;
struct dm_rq_target_io *tio = clone->end_io_data;
r = rq_end_io(tio->ti, clone, error, &tio->info);
}
- if (unlikely(error == -EREMOTEIO)) {
+ if (unlikely(error == BLK_STS_TARGET)) {
if (req_op(clone) == REQ_OP_WRITE_SAME &&
!clone->q->limits.max_write_same_sectors)
disable_write_same(tio->md);
* Complete the clone and the original request with the error status
* through softirq context.
*/
-static void dm_complete_request(struct request *rq, int error)
+static void dm_complete_request(struct request *rq, blk_status_t error)
{
struct dm_rq_target_io *tio = tio_from_request(rq);
* Target's rq_end_io() function isn't called.
* This may be used when the target's map_rq() or clone_and_map_rq() functions fail.
*/
-static void dm_kill_unmapped_request(struct request *rq, int error)
+static void dm_kill_unmapped_request(struct request *rq, blk_status_t error)
{
rq->rq_flags |= RQF_FAILED;
dm_complete_request(rq, error);
/*
* Called with the clone's queue lock held (in the case of .request_fn)
*/
-static void end_clone_request(struct request *clone, int error)
+static void end_clone_request(struct request *clone, blk_status_t error)
{
struct dm_rq_target_io *tio = clone->end_io_data;
static void dm_dispatch_clone_request(struct request *clone, struct request *rq)
{
- int r;
+ blk_status_t r;
if (blk_queue_io_stat(clone->q))
clone->rq_flags |= RQF_IO_STAT;
break;
case DM_MAPIO_KILL:
/* The target wants to complete the I/O */
- dm_kill_unmapped_request(rq, -EIO);
+ dm_kill_unmapped_request(rq, BLK_STS_IOERR);
break;
default:
DMWARN("unimplemented target map return value: %d", r);
return __dm_rq_init_rq(set->driver_data, rq);
}
-static int dm_mq_queue_rq(struct blk_mq_hw_ctx *hctx,
+static blk_status_t dm_mq_queue_rq(struct blk_mq_hw_ctx *hctx,
const struct blk_mq_queue_data *bd)
{
struct request *rq = bd->rq;
}
if (ti->type->busy && ti->type->busy(ti))
- return BLK_MQ_RQ_QUEUE_BUSY;
+ return BLK_STS_RESOURCE;
dm_start_request(md, rq);
rq_end_stats(md, rq);
rq_completed(md, rq_data_dir(rq), false);
blk_mq_delay_run_hw_queue(hctx, 100/*ms*/);
- return BLK_MQ_RQ_QUEUE_BUSY;
+ return BLK_STS_RESOURCE;
}
- return BLK_MQ_RQ_QUEUE_OK;
+ return BLK_STS_OK;
}
static const struct blk_mq_ops dm_mq_ops = {
struct dm_target *ti;
struct request *orig, *clone;
struct kthread_work work;
- int error;
+ blk_status_t error;
union map_info info;
struct dm_stats_aux stats_aux;
unsigned long duration_jiffies;
{
void *callback_data = bio->bi_private;
- dm_kcopyd_do_callback(callback_data, 0, bio->bi_error ? 1 : 0);
+ dm_kcopyd_do_callback(callback_data, 0, bio->bi_status ? 1 : 0);
}
static void start_full_bio(struct dm_snap_pending_exception *pe,
/* Full snapshots are not usable */
/* To get here the table must be live so s->active is always set. */
if (!s->valid)
- return -EIO;
+ return DM_MAPIO_KILL;
/* FIXME: should only take write lock if we need
* to copy an exception */
if (!s->valid || (unlikely(s->snapshot_overflowed) &&
bio_data_dir(bio) == WRITE)) {
- r = -EIO;
+ r = DM_MAPIO_KILL;
goto out_unlock;
}
if (!s->valid || s->snapshot_overflowed) {
free_pending_exception(pe);
- r = -EIO;
+ r = DM_MAPIO_KILL;
goto out_unlock;
}
DMERR("Snapshot overflowed: Unable to allocate exception.");
} else
__invalidate_snapshot(s, -ENOMEM);
- r = -EIO;
+ r = DM_MAPIO_KILL;
goto out_unlock;
}
}
return r;
}
-static int snapshot_end_io(struct dm_target *ti, struct bio *bio, int error)
+static int snapshot_end_io(struct dm_target *ti, struct bio *bio,
+ blk_status_t *error)
{
struct dm_snapshot *s = ti->private;
if (is_bio_tracked(bio))
stop_tracking_chunk(s, bio);
- return 0;
+ return DM_ENDIO_DONE;
}
static void snapshot_merge_presuspend(struct dm_target *ti)
}
}
-static int stripe_end_io(struct dm_target *ti, struct bio *bio, int error)
+static int stripe_end_io(struct dm_target *ti, struct bio *bio,
+ blk_status_t *error)
{
unsigned i;
char major_minor[16];
struct stripe_c *sc = ti->private;
- if (!error)
- return 0; /* I/O complete */
+ if (!*error)
+ return DM_ENDIO_DONE; /* I/O complete */
- if ((error == -EWOULDBLOCK) && (bio->bi_opf & REQ_RAHEAD))
- return error;
+ if (bio->bi_opf & REQ_RAHEAD)
+ return DM_ENDIO_DONE;
- if (error == -EOPNOTSUPP)
- return error;
+ if (*error == BLK_STS_NOTSUPP)
+ return DM_ENDIO_DONE;
memset(major_minor, 0, sizeof(major_minor));
sprintf(major_minor, "%d:%d",
schedule_work(&sc->trigger_event);
}
- return error;
+ return DM_ENDIO_DONE;
}
static int stripe_iterate_devices(struct dm_target *ti,
static int io_err_map(struct dm_target *tt, struct bio *bio)
{
- return -EIO;
+ return DM_MAPIO_KILL;
}
static int io_err_clone_and_map_rq(struct dm_target *ti, struct request *rq,
* Even if r is set, there could be sub discards in flight that we
* need to wait for.
*/
- if (r && !op->parent_bio->bi_error)
- op->parent_bio->bi_error = r;
+ if (r && !op->parent_bio->bi_status)
+ op->parent_bio->bi_status = errno_to_blk_status(r);
bio_endio(op->parent_bio);
}
}
static void cell_error_with_code(struct pool *pool,
- struct dm_bio_prison_cell *cell, int error_code)
+ struct dm_bio_prison_cell *cell, blk_status_t error_code)
{
dm_cell_error(pool->prison, cell, error_code);
dm_bio_prison_free_cell(pool->prison, cell);
}
-static int get_pool_io_error_code(struct pool *pool)
+static blk_status_t get_pool_io_error_code(struct pool *pool)
{
- return pool->out_of_data_space ? -ENOSPC : -EIO;
+ return pool->out_of_data_space ? BLK_STS_NOSPC : BLK_STS_IOERR;
}
static void cell_error(struct pool *pool, struct dm_bio_prison_cell *cell)
{
- int error = get_pool_io_error_code(pool);
-
- cell_error_with_code(pool, cell, error);
+ cell_error_with_code(pool, cell, get_pool_io_error_code(pool));
}
static void cell_success(struct pool *pool, struct dm_bio_prison_cell *cell)
static void cell_requeue(struct pool *pool, struct dm_bio_prison_cell *cell)
{
- cell_error_with_code(pool, cell, DM_ENDIO_REQUEUE);
+ cell_error_with_code(pool, cell, BLK_STS_DM_REQUEUE);
}
/*----------------------------------------------------------------*/
bio_list_init(master);
}
-static void error_bio_list(struct bio_list *bios, int error)
+static void error_bio_list(struct bio_list *bios, blk_status_t error)
{
struct bio *bio;
while ((bio = bio_list_pop(bios))) {
- bio->bi_error = error;
+ bio->bi_status = error;
bio_endio(bio);
}
}
-static void error_thin_bio_list(struct thin_c *tc, struct bio_list *master, int error)
+static void error_thin_bio_list(struct thin_c *tc, struct bio_list *master,
+ blk_status_t error)
{
struct bio_list bios;
unsigned long flags;
__merge_bio_list(&bios, &tc->retry_on_resume_list);
spin_unlock_irqrestore(&tc->lock, flags);
- error_bio_list(&bios, DM_ENDIO_REQUEUE);
+ error_bio_list(&bios, BLK_STS_DM_REQUEUE);
requeue_deferred_cells(tc);
}
-static void error_retry_list_with_code(struct pool *pool, int error)
+static void error_retry_list_with_code(struct pool *pool, blk_status_t error)
{
struct thin_c *tc;
static void error_retry_list(struct pool *pool)
{
- int error = get_pool_io_error_code(pool);
-
- error_retry_list_with_code(pool, error);
+ error_retry_list_with_code(pool, get_pool_io_error_code(pool));
}
/*
*/
atomic_t prepare_actions;
- int err;
+ blk_status_t status;
struct thin_c *tc;
dm_block_t virt_begin, virt_end;
dm_block_t data_block;
{
struct dm_thin_new_mapping *m = context;
- m->err = read_err || write_err ? -EIO : 0;
+ m->status = read_err || write_err ? BLK_STS_IOERR : 0;
complete_mapping_preparation(m);
}
bio->bi_end_io = m->saved_bi_end_io;
- m->err = bio->bi_error;
+ m->status = bio->bi_status;
complete_mapping_preparation(m);
}
struct bio *bio = m->bio;
int r;
- if (m->err) {
+ if (m->status) {
cell_error(pool, m->cell);
goto out;
}
return;
}
+ /*
+ * Increment the unmapped blocks. This prevents a race between the
+ * passdown io and reallocation of freed blocks.
+ */
+ r = dm_pool_inc_data_range(pool->pmd, m->data_block, data_end);
+ if (r) {
+ metadata_operation_failed(pool, "dm_pool_inc_data_range", r);
+ bio_io_error(m->bio);
+ cell_defer_no_holder(tc, m->cell);
+ mempool_free(m, pool->mapping_pool);
+ return;
+ }
+
discard_parent = bio_alloc(GFP_NOIO, 1);
if (!discard_parent) {
DMWARN("%s: unable to allocate top level discard bio for passdown. Skipping passdown.",
end_discard(&op, r);
}
}
-
- /*
- * Increment the unmapped blocks. This prevents a race between the
- * passdown io and reallocation of freed blocks.
- */
- r = dm_pool_inc_data_range(pool->pmd, m->data_block, data_end);
- if (r) {
- metadata_operation_failed(pool, "dm_pool_inc_data_range", r);
- bio_io_error(m->bio);
- cell_defer_no_holder(tc, m->cell);
- mempool_free(m, pool->mapping_pool);
- return;
- }
}
static void process_prepared_discard_passdown_pt2(struct dm_thin_new_mapping *m)
spin_unlock_irqrestore(&tc->lock, flags);
}
-static int should_error_unserviceable_bio(struct pool *pool)
+static blk_status_t should_error_unserviceable_bio(struct pool *pool)
{
enum pool_mode m = get_pool_mode(pool);
case PM_WRITE:
/* Shouldn't get here */
DMERR_LIMIT("bio unserviceable, yet pool is in PM_WRITE mode");
- return -EIO;
+ return BLK_STS_IOERR;
case PM_OUT_OF_DATA_SPACE:
- return pool->pf.error_if_no_space ? -ENOSPC : 0;
+ return pool->pf.error_if_no_space ? BLK_STS_NOSPC : 0;
case PM_READ_ONLY:
case PM_FAIL:
- return -EIO;
+ return BLK_STS_IOERR;
default:
/* Shouldn't get here */
DMERR_LIMIT("bio unserviceable, yet pool has an unknown mode");
- return -EIO;
+ return BLK_STS_IOERR;
}
}
static void handle_unserviceable_bio(struct pool *pool, struct bio *bio)
{
- int error = should_error_unserviceable_bio(pool);
+ blk_status_t error = should_error_unserviceable_bio(pool);
if (error) {
- bio->bi_error = error;
+ bio->bi_status = error;
bio_endio(bio);
} else
retry_on_resume(bio);
{
struct bio *bio;
struct bio_list bios;
- int error;
+ blk_status_t error;
error = should_error_unserviceable_bio(pool);
if (error) {
unsigned count = 0;
if (tc->requeue_mode) {
- error_thin_bio_list(tc, &tc->deferred_bio_list, DM_ENDIO_REQUEUE);
+ error_thin_bio_list(tc, &tc->deferred_bio_list,
+ BLK_STS_DM_REQUEUE);
return;
}
if (get_pool_mode(pool) == PM_OUT_OF_DATA_SPACE && !pool->pf.error_if_no_space) {
pool->pf.error_if_no_space = true;
notify_of_pool_mode_change_to_oods(pool);
- error_retry_list_with_code(pool, -ENOSPC);
+ error_retry_list_with_code(pool, BLK_STS_NOSPC);
}
}
thin_hook_bio(tc, bio);
if (tc->requeue_mode) {
- bio->bi_error = DM_ENDIO_REQUEUE;
+ bio->bi_status = BLK_STS_DM_REQUEUE;
bio_endio(bio);
return DM_MAPIO_SUBMITTED;
}
return thin_bio_map(ti, bio);
}
-static int thin_endio(struct dm_target *ti, struct bio *bio, int err)
+static int thin_endio(struct dm_target *ti, struct bio *bio,
+ blk_status_t *err)
{
unsigned long flags;
struct dm_thin_endio_hook *h = dm_per_bio_data(bio, sizeof(struct dm_thin_endio_hook));
if (h->cell)
cell_defer_no_holder(h->tc, h->cell);
- return 0;
+ return DM_ENDIO_DONE;
}
static void thin_presuspend(struct dm_target *ti)
/*
* End one "io" structure with a given error.
*/
-static void verity_finish_io(struct dm_verity_io *io, int error)
+static void verity_finish_io(struct dm_verity_io *io, blk_status_t status)
{
struct dm_verity *v = io->v;
struct bio *bio = dm_bio_from_per_bio_data(io, v->ti->per_io_data_size);
bio->bi_end_io = io->orig_bi_end_io;
- bio->bi_error = error;
+ bio->bi_status = status;
verity_fec_finish_io(io);
{
struct dm_verity_io *io = container_of(w, struct dm_verity_io, work);
- verity_finish_io(io, verity_verify_io(io));
+ verity_finish_io(io, errno_to_blk_status(verity_verify_io(io)));
}
static void verity_end_io(struct bio *bio)
{
struct dm_verity_io *io = bio->bi_private;
- if (bio->bi_error && !verity_fec_is_enabled(io->v)) {
- verity_finish_io(io, bio->bi_error);
+ if (bio->bi_status && !verity_fec_is_enabled(io->v)) {
+ verity_finish_io(io, bio->bi_status);
return;
}
if (((unsigned)bio->bi_iter.bi_sector | bio_sectors(bio)) &
((1 << (v->data_dev_block_bits - SECTOR_SHIFT)) - 1)) {
DMERR_LIMIT("unaligned io");
- return -EIO;
+ return DM_MAPIO_KILL;
}
if (bio_end_sector(bio) >>
(v->data_dev_block_bits - SECTOR_SHIFT) > v->data_blocks) {
DMERR_LIMIT("io out of range");
- return -EIO;
+ return DM_MAPIO_KILL;
}
if (bio_data_dir(bio) == WRITE)
- return -EIO;
+ return DM_MAPIO_KILL;
io = dm_per_bio_data(bio, ti->per_io_data_size);
io->v = v;
case REQ_OP_READ:
if (bio->bi_opf & REQ_RAHEAD) {
/* readahead of null bytes only wastes buffer cache */
- return -EIO;
+ return DM_MAPIO_KILL;
}
zero_fill_bio(bio);
break;
/* writes get silently dropped */
break;
default:
- return -EIO;
+ return DM_MAPIO_KILL;
}
bio_endio(bio);
*/
struct dm_io {
struct mapped_device *md;
- int error;
+ blk_status_t status;
atomic_t io_count;
struct bio *bio;
unsigned long start_time;
* Decrements the number of outstanding ios that a bio has been
* cloned into, completing the original io if necc.
*/
-static void dec_pending(struct dm_io *io, int error)
+static void dec_pending(struct dm_io *io, blk_status_t error)
{
unsigned long flags;
- int io_error;
+ blk_status_t io_error;
struct bio *bio;
struct mapped_device *md = io->md;
/* Push-back supersedes any I/O errors */
if (unlikely(error)) {
spin_lock_irqsave(&io->endio_lock, flags);
- if (!(io->error > 0 && __noflush_suspending(md)))
- io->error = error;
+ if (!(io->status == BLK_STS_DM_REQUEUE &&
+ __noflush_suspending(md)))
+ io->status = error;
spin_unlock_irqrestore(&io->endio_lock, flags);
}
if (atomic_dec_and_test(&io->io_count)) {
- if (io->error == DM_ENDIO_REQUEUE) {
+ if (io->status == BLK_STS_DM_REQUEUE) {
/*
* Target requested pushing back the I/O.
*/
bio_list_add_head(&md->deferred, io->bio);
else
/* noflush suspend was interrupted. */
- io->error = -EIO;
+ io->status = BLK_STS_IOERR;
spin_unlock_irqrestore(&md->deferred_lock, flags);
}
- io_error = io->error;
+ io_error = io->status;
bio = io->bio;
end_io_acct(io);
free_io(md, io);
- if (io_error == DM_ENDIO_REQUEUE)
+ if (io_error == BLK_STS_DM_REQUEUE)
return;
if ((bio->bi_opf & REQ_PREFLUSH) && bio->bi_iter.bi_size) {
queue_io(md, bio);
} else {
/* done with normal IO or empty flush */
- bio->bi_error = io_error;
+ bio->bi_status = io_error;
bio_endio(bio);
}
}
static void clone_endio(struct bio *bio)
{
- int error = bio->bi_error;
- int r = error;
+ blk_status_t error = bio->bi_status;
struct dm_target_io *tio = container_of(bio, struct dm_target_io, clone);
struct dm_io *io = tio->io;
struct mapped_device *md = tio->io->md;
dm_endio_fn endio = tio->ti->type->end_io;
- if (endio) {
- r = endio(tio->ti, bio, error);
- if (r < 0 || r == DM_ENDIO_REQUEUE)
- /*
- * error and requeue request are handled
- * in dec_pending().
- */
- error = r;
- else if (r == DM_ENDIO_INCOMPLETE)
- /* The target will handle the io */
- return;
- else if (r) {
- DMWARN("unimplemented target endio return value: %d", r);
- BUG();
- }
- }
-
- if (unlikely(r == -EREMOTEIO)) {
+ if (unlikely(error == BLK_STS_TARGET)) {
if (bio_op(bio) == REQ_OP_WRITE_SAME &&
!bdev_get_queue(bio->bi_bdev)->limits.max_write_same_sectors)
disable_write_same(md);
disable_write_zeroes(md);
}
+ if (endio) {
+ int r = endio(tio->ti, bio, &error);
+ switch (r) {
+ case DM_ENDIO_REQUEUE:
+ error = BLK_STS_DM_REQUEUE;
+ /*FALLTHRU*/
+ case DM_ENDIO_DONE:
+ break;
+ case DM_ENDIO_INCOMPLETE:
+ /* The target will handle the io */
+ return;
+ default:
+ DMWARN("unimplemented target endio return value: %d", r);
+ BUG();
+ }
+ }
+
free_tio(tio);
dec_pending(io, error);
}
while ((bio = bio_list_pop(&list))) {
struct bio_set *bs = bio->bi_pool;
- if (unlikely(!bs) || bs == fs_bio_set) {
+ if (unlikely(!bs) || bs == fs_bio_set ||
+ !bs->rescue_workqueue) {
bio_list_add(¤t->bio_list[i], bio);
continue;
}
r = ti->type->map(ti, clone);
dm_offload_end(&o);
- if (r == DM_MAPIO_REMAPPED) {
+ switch (r) {
+ case DM_MAPIO_SUBMITTED:
+ break;
+ case DM_MAPIO_REMAPPED:
/* the bio has been remapped so dispatch it */
-
trace_block_bio_remap(bdev_get_queue(clone->bi_bdev), clone,
tio->io->bio->bi_bdev->bd_dev, sector);
-
generic_make_request(clone);
- } else if (r < 0 || r == DM_MAPIO_REQUEUE) {
- /* error the io and bail out, or requeue it if needed */
- dec_pending(tio->io, r);
+ break;
+ case DM_MAPIO_KILL:
+ dec_pending(tio->io, BLK_STS_IOERR);
+ free_tio(tio);
+ break;
+ case DM_MAPIO_REQUEUE:
+ dec_pending(tio->io, BLK_STS_DM_REQUEUE);
free_tio(tio);
- } else if (r != DM_MAPIO_SUBMITTED) {
+ break;
+ default:
DMWARN("unimplemented target map return value: %d", r);
BUG();
}
ci.map = map;
ci.md = md;
ci.io = alloc_io(md);
- ci.io->error = 0;
+ ci.io->status = 0;
atomic_set(&ci.io->io_count, 1);
ci.io->bio = bio;
ci.io->md = md;
* Initialize aspects of queue that aren't relevant for blk-mq
*/
md->queue->backing_dev_info->congested_fn = dm_any_congested;
- blk_queue_bounce_limit(md->queue, BLK_BOUNCE_ANY);
}
static void cleanup_mapped_device(struct mapped_device *md)
BUG();
}
- pools->bs = bioset_create_nobvec(pool_size, front_pad);
+ pools->bs = bioset_create(pool_size, front_pad, BIOSET_NEED_RESCUER);
if (!pools->bs)
goto out;
static bool create_on_open = true;
/* bio_clone_mddev
- * like bio_clone, but with a local bio set
+ * like bio_clone_bioset, but with a local bio set
*/
struct bio *bio_alloc_mddev(gfp_t gfp_mask, int nr_iovecs,
unsigned int sectors;
int cpu;
- blk_queue_split(q, &bio, q->bio_split);
+ blk_queue_split(q, &bio);
if (mddev == NULL || mddev->pers == NULL) {
bio_io_error(bio);
}
if (mddev->ro == 1 && unlikely(rw == WRITE)) {
if (bio_sectors(bio) != 0)
- bio->bi_error = -EROFS;
+ bio->bi_status = BLK_STS_IOERR;
bio_endio(bio);
return BLK_QC_T_NONE;
}
struct md_rdev *rdev = bio->bi_private;
struct mddev *mddev = rdev->mddev;
- if (bio->bi_error) {
- pr_err("md: super_written gets error=%d\n", bio->bi_error);
+ if (bio->bi_status) {
+ pr_err("md: super_written gets error=%d\n", bio->bi_status);
md_error(mddev, rdev);
if (!test_bit(Faulty, &rdev->flags)
&& (bio->bi_opf & MD_FAILFAST)) {
submit_bio_wait(bio);
- ret = !bio->bi_error;
+ ret = !bio->bi_status;
bio_put(bio);
return ret;
}
return -EINVAL;
}
-static int uuid_equal(mdp_super_t *sb1, mdp_super_t *sb2)
+static int md_uuid_equal(mdp_super_t *sb1, mdp_super_t *sb2)
{
return sb1->set_uuid0 == sb2->set_uuid0 &&
sb1->set_uuid1 == sb2->set_uuid1 &&
sb1->set_uuid3 == sb2->set_uuid3;
}
-static int sb_equal(mdp_super_t *sb1, mdp_super_t *sb2)
+static int md_sb_equal(mdp_super_t *sb1, mdp_super_t *sb2)
{
int ret;
mdp_super_t *tmp1, *tmp2;
} else {
__u64 ev1, ev2;
mdp_super_t *refsb = page_address(refdev->sb_page);
- if (!uuid_equal(refsb, sb)) {
+ if (!md_uuid_equal(refsb, sb)) {
pr_warn("md: %s has different UUID to %s\n",
b, bdevname(refdev->bdev,b2));
goto abort;
}
- if (!sb_equal(refsb, sb)) {
+ if (!md_sb_equal(refsb, sb)) {
pr_warn("md: %s has same UUID but different superblock to %s\n",
b, bdevname(refdev->bdev, b2));
goto abort;
}
if (mddev->bio_set == NULL) {
- mddev->bio_set = bioset_create(BIO_POOL_SIZE, 0);
+ mddev->bio_set = bioset_create(BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS);
if (!mddev->bio_set)
return -ENOMEM;
}
* operation and are ready to return a success/failure code to the buffer
* cache layer.
*/
-static void multipath_end_bh_io (struct multipath_bh *mp_bh, int err)
+static void multipath_end_bh_io(struct multipath_bh *mp_bh, blk_status_t status)
{
struct bio *bio = mp_bh->master_bio;
struct mpconf *conf = mp_bh->mddev->private;
- bio->bi_error = err;
+ bio->bi_status = status;
bio_endio(bio);
mempool_free(mp_bh, conf->pool);
}
struct mpconf *conf = mp_bh->mddev->private;
struct md_rdev *rdev = conf->multipaths[mp_bh->path].rdev;
- if (!bio->bi_error)
+ if (!bio->bi_status)
multipath_end_bh_io(mp_bh, 0);
else if (!(bio->bi_opf & REQ_RAHEAD)) {
/*
(unsigned long long)bio->bi_iter.bi_sector);
multipath_reschedule_retry(mp_bh);
} else
- multipath_end_bh_io(mp_bh, bio->bi_error);
+ multipath_end_bh_io(mp_bh, bio->bi_status);
rdev_dec_pending(rdev, conf->mddev);
}
pr_err("multipath: %s: unrecoverable IO read error for block %llu\n",
bdevname(bio->bi_bdev,b),
(unsigned long long)bio->bi_iter.bi_sector);
- multipath_end_bh_io(mp_bh, -EIO);
+ multipath_end_bh_io(mp_bh, BLK_STS_IOERR);
} else {
pr_err("multipath: %s: redirecting sector %llu to another IO path\n",
bdevname(bio->bi_bdev,b),
struct r1conf *conf = r1_bio->mddev->private;
if (!test_bit(R1BIO_Uptodate, &r1_bio->state))
- bio->bi_error = -EIO;
+ bio->bi_status = BLK_STS_IOERR;
bio_endio(bio);
/*
static void raid1_end_read_request(struct bio *bio)
{
- int uptodate = !bio->bi_error;
+ int uptodate = !bio->bi_status;
struct r1bio *r1_bio = bio->bi_private;
struct r1conf *conf = r1_bio->mddev->private;
struct md_rdev *rdev = conf->mirrors[r1_bio->read_disk].rdev;
struct md_rdev *rdev = conf->mirrors[mirror].rdev;
bool discard_error;
- discard_error = bio->bi_error && bio_op(bio) == REQ_OP_DISCARD;
+ discard_error = bio->bi_status && bio_op(bio) == REQ_OP_DISCARD;
/*
* 'one mirror IO has finished' event handler:
*/
- if (bio->bi_error && !discard_error) {
+ if (bio->bi_status && !discard_error) {
set_bit(WriteErrorSeen, &rdev->flags);
if (!test_and_set_bit(WantReplacement, &rdev->flags))
set_bit(MD_RECOVERY_NEEDED, &
bio->bi_next = NULL;
bio->bi_bdev = rdev->bdev;
if (test_bit(Faulty, &rdev->flags)) {
- bio->bi_error = -EIO;
+ bio->bi_status = BLK_STS_IOERR;
bio_endio(bio);
} else if (unlikely((bio_op(bio) == REQ_OP_DISCARD) &&
!blk_queue_discard(bdev_get_queue(bio->bi_bdev))))
* or re-read if the read failed.
* We don't do much here, just schedule handling by raid1d
*/
- if (!bio->bi_error)
+ if (!bio->bi_status)
set_bit(R1BIO_Uptodate, &r1_bio->state);
if (atomic_dec_and_test(&r1_bio->remaining))
static void end_sync_write(struct bio *bio)
{
- int uptodate = !bio->bi_error;
+ int uptodate = !bio->bi_status;
struct r1bio *r1_bio = get_resync_r1bio(bio);
struct mddev *mddev = r1_bio->mddev;
struct r1conf *conf = mddev->private;
idx ++;
}
set_bit(R1BIO_Uptodate, &r1_bio->state);
- bio->bi_error = 0;
+ bio->bi_status = 0;
return 1;
}
for (i = 0; i < conf->raid_disks * 2; i++) {
int j;
int size;
- int error;
+ blk_status_t status;
struct bio_vec *bi;
struct bio *b = r1_bio->bios[i];
struct resync_pages *rp = get_resync_pages(b);
if (b->bi_end_io != end_sync_read)
continue;
/* fixup the bio for reuse, but preserve errno */
- error = b->bi_error;
+ status = b->bi_status;
bio_reset(b);
- b->bi_error = error;
+ b->bi_status = status;
b->bi_vcnt = vcnt;
b->bi_iter.bi_size = r1_bio->sectors << 9;
b->bi_iter.bi_sector = r1_bio->sector +
}
for (primary = 0; primary < conf->raid_disks * 2; primary++)
if (r1_bio->bios[primary]->bi_end_io == end_sync_read &&
- !r1_bio->bios[primary]->bi_error) {
+ !r1_bio->bios[primary]->bi_status) {
r1_bio->bios[primary]->bi_end_io = NULL;
rdev_dec_pending(conf->mirrors[primary].rdev, mddev);
break;
int j;
struct bio *pbio = r1_bio->bios[primary];
struct bio *sbio = r1_bio->bios[i];
- int error = sbio->bi_error;
+ blk_status_t status = sbio->bi_status;
struct page **ppages = get_resync_pages(pbio)->pages;
struct page **spages = get_resync_pages(sbio)->pages;
struct bio_vec *bi;
if (sbio->bi_end_io != end_sync_read)
continue;
/* Now we can 'fixup' the error value */
- sbio->bi_error = 0;
+ sbio->bi_status = 0;
bio_for_each_segment_all(bi, sbio, j)
page_len[j] = bi->bv_len;
- if (!error) {
+ if (!status) {
for (j = vcnt; j-- ; ) {
if (memcmp(page_address(ppages[j]),
page_address(spages[j]),
if (j >= 0)
atomic64_add(r1_bio->sectors, &mddev->resync_mismatches);
if (j < 0 || (test_bit(MD_RECOVERY_CHECK, &mddev->recovery)
- && !error)) {
+ && !status)) {
/* No need to write to this device. */
sbio->bi_end_io = NULL;
rdev_dec_pending(conf->mirrors[i].rdev, mddev);
struct bio *bio = r1_bio->bios[m];
if (bio->bi_end_io == NULL)
continue;
- if (!bio->bi_error &&
+ if (!bio->bi_status &&
test_bit(R1BIO_MadeGood, &r1_bio->state)) {
rdev_clear_badblocks(rdev, r1_bio->sector, s, 0);
}
- if (bio->bi_error &&
+ if (bio->bi_status &&
test_bit(R1BIO_WriteError, &r1_bio->state)) {
if (!rdev_set_badblocks(rdev, r1_bio->sector, s, 0))
md_error(conf->mddev, rdev);
if (!conf->r1bio_pool)
goto abort;
- conf->bio_split = bioset_create(BIO_POOL_SIZE, 0);
+ conf->bio_split = bioset_create(BIO_POOL_SIZE, 0, 0);
if (!conf->bio_split)
goto abort;
struct r10conf *conf = r10_bio->mddev->private;
if (!test_bit(R10BIO_Uptodate, &r10_bio->state))
- bio->bi_error = -EIO;
+ bio->bi_status = BLK_STS_IOERR;
bio_endio(bio);
/*
static void raid10_end_read_request(struct bio *bio)
{
- int uptodate = !bio->bi_error;
+ int uptodate = !bio->bi_status;
struct r10bio *r10_bio = bio->bi_private;
int slot, dev;
struct md_rdev *rdev;
struct bio *to_put = NULL;
bool discard_error;
- discard_error = bio->bi_error && bio_op(bio) == REQ_OP_DISCARD;
+ discard_error = bio->bi_status && bio_op(bio) == REQ_OP_DISCARD;
dev = find_bio_disk(conf, r10_bio, bio, &slot, &repl);
/*
* this branch is our 'one mirror IO has finished' event handler:
*/
- if (bio->bi_error && !discard_error) {
+ if (bio->bi_status && !discard_error) {
if (repl)
/* Never record new bad blocks to replacement,
* just fail it.
bio->bi_next = NULL;
bio->bi_bdev = rdev->bdev;
if (test_bit(Faulty, &rdev->flags)) {
- bio->bi_error = -EIO;
+ bio->bi_status = BLK_STS_IOERR;
bio_endio(bio);
} else if (unlikely((bio_op(bio) == REQ_OP_DISCARD) &&
!blk_queue_discard(bdev_get_queue(bio->bi_bdev))))
bio->bi_next = NULL;
bio->bi_bdev = rdev->bdev;
if (test_bit(Faulty, &rdev->flags)) {
- bio->bi_error = -EIO;
+ bio->bi_status = BLK_STS_IOERR;
bio_endio(bio);
} else if (unlikely((bio_op(bio) == REQ_OP_DISCARD) &&
!blk_queue_discard(bdev_get_queue(bio->bi_bdev))))
{
struct r10conf *conf = r10_bio->mddev->private;
- if (!bio->bi_error)
+ if (!bio->bi_status)
set_bit(R10BIO_Uptodate, &r10_bio->state);
else
/* The write handler will notice the lack of
else
rdev = conf->mirrors[d].rdev;
- if (bio->bi_error) {
+ if (bio->bi_status) {
if (repl)
md_error(mddev, rdev);
else {
/* find the first device with a block */
for (i=0; i<conf->copies; i++)
- if (!r10_bio->devs[i].bio->bi_error)
+ if (!r10_bio->devs[i].bio->bi_status)
break;
if (i == conf->copies)
tpages = get_resync_pages(tbio)->pages;
d = r10_bio->devs[i].devnum;
rdev = conf->mirrors[d].rdev;
- if (!r10_bio->devs[i].bio->bi_error) {
+ if (!r10_bio->devs[i].bio->bi_status) {
/* We know that the bi_io_vec layout is the same for
* both 'first' and 'i', so we just compare them.
* All vec entries are PAGE_SIZE;
rdev = conf->mirrors[dev].rdev;
if (r10_bio->devs[m].bio == NULL)
continue;
- if (!r10_bio->devs[m].bio->bi_error) {
+ if (!r10_bio->devs[m].bio->bi_status) {
rdev_clear_badblocks(
rdev,
r10_bio->devs[m].addr,
if (r10_bio->devs[m].repl_bio == NULL)
continue;
- if (!r10_bio->devs[m].repl_bio->bi_error) {
+ if (!r10_bio->devs[m].repl_bio->bi_status) {
rdev_clear_badblocks(
rdev,
r10_bio->devs[m].addr,
r10_bio->devs[m].addr,
r10_bio->sectors, 0);
rdev_dec_pending(rdev, conf->mddev);
- } else if (bio != NULL && bio->bi_error) {
+ } else if (bio != NULL && bio->bi_status) {
fail = true;
if (!narrow_write_error(r10_bio, m)) {
md_error(conf->mddev, rdev);
r10_bio->devs[i].repl_bio->bi_end_io = NULL;
bio = r10_bio->devs[i].bio;
- bio->bi_error = -EIO;
+ bio->bi_status = BLK_STS_IOERR;
rcu_read_lock();
rdev = rcu_dereference(conf->mirrors[d].rdev);
if (rdev == NULL || test_bit(Faulty, &rdev->flags)) {
/* Need to set up for writing to the replacement */
bio = r10_bio->devs[i].repl_bio;
- bio->bi_error = -EIO;
+ bio->bi_status = BLK_STS_IOERR;
sector = r10_bio->devs[i].addr;
bio->bi_next = biolist;
if (bio->bi_end_io == end_sync_read) {
md_sync_acct(bio->bi_bdev, nr_sectors);
- bio->bi_error = 0;
+ bio->bi_status = 0;
generic_make_request(bio);
}
}
if (!conf->r10bio_pool)
goto out;
- conf->bio_split = bioset_create(BIO_POOL_SIZE, 0);
+ conf->bio_split = bioset_create(BIO_POOL_SIZE, 0, 0);
if (!conf->bio_split)
goto out;
read_bio->bi_end_io = end_reshape_read;
bio_set_op_attrs(read_bio, REQ_OP_READ, 0);
read_bio->bi_flags &= (~0UL << BIO_RESET_BITS);
- read_bio->bi_error = 0;
+ read_bio->bi_status = 0;
read_bio->bi_vcnt = 0;
read_bio->bi_iter.bi_size = 0;
r10_bio->master_bio = read_bio;
rdev = conf->mirrors[d].rdev;
}
- if (bio->bi_error) {
+ if (bio->bi_status) {
/* FIXME should record badblock */
md_error(mddev, rdev);
}
struct r5l_log *log = io->log;
unsigned long flags;
- if (bio->bi_error)
+ if (bio->bi_status)
md_error(log->rdev->mddev, log->rdev);
bio_put(bio);
unsigned long flags;
struct r5l_io_unit *io;
- if (bio->bi_error)
+ if (bio->bi_status)
md_error(log->rdev->mddev, log->rdev);
spin_lock_irqsave(&log->io_list_lock, flags);
if (!log->io_pool)
goto io_pool;
- log->bs = bioset_create(R5L_POOL_SIZE, 0);
+ log->bs = bioset_create(R5L_POOL_SIZE, 0, BIOSET_NEED_BVECS);
if (!log->bs)
goto io_bs;
pr_debug("%s: seq: %llu\n", __func__, io->seq);
- if (bio->bi_error)
+ if (bio->bi_status)
md_error(ppl_conf->mddev, log->rdev);
list_for_each_entry_safe(sh, next, &io->stripe_list, log_list) {
goto err;
}
- ppl_conf->bs = bioset_create(conf->raid_disks, 0);
+ ppl_conf->bs = bioset_create(conf->raid_disks, 0, 0);
if (!ppl_conf->bs) {
ret = -ENOMEM;
goto err;
pr_debug("end_read_request %llu/%d, count: %d, error %d.\n",
(unsigned long long)sh->sector, i, atomic_read(&sh->count),
- bi->bi_error);
+ bi->bi_status);
if (i == disks) {
bio_reset(bi);
BUG();
s = sh->sector + rdev->new_data_offset;
else
s = sh->sector + rdev->data_offset;
- if (!bi->bi_error) {
+ if (!bi->bi_status) {
set_bit(R5_UPTODATE, &sh->dev[i].flags);
if (test_bit(R5_ReadError, &sh->dev[i].flags)) {
/* Note that this cannot happen on a
}
pr_debug("end_write_request %llu/%d, count %d, error: %d.\n",
(unsigned long long)sh->sector, i, atomic_read(&sh->count),
- bi->bi_error);
+ bi->bi_status);
if (i == disks) {
bio_reset(bi);
BUG();
}
if (replacement) {
- if (bi->bi_error)
+ if (bi->bi_status)
md_error(conf->mddev, rdev);
else if (is_badblock(rdev, sh->sector,
STRIPE_SECTORS,
&first_bad, &bad_sectors))
set_bit(R5_MadeGoodRepl, &sh->dev[i].flags);
} else {
- if (bi->bi_error) {
+ if (bi->bi_status) {
set_bit(STRIPE_DEGRADED, &sh->state);
set_bit(WriteErrorSeen, &rdev->flags);
set_bit(R5_WriteError, &sh->dev[i].flags);
}
rdev_dec_pending(rdev, conf->mddev);
- if (sh->batch_head && bi->bi_error && !replacement)
+ if (sh->batch_head && bi->bi_status && !replacement)
set_bit(STRIPE_BATCH_ERR, &sh->batch_head->state);
bio_reset(bi);
sh->dev[i].sector + STRIPE_SECTORS) {
struct bio *nextbi = r5_next_bio(bi, sh->dev[i].sector);
- bi->bi_error = -EIO;
+ bi->bi_status = BLK_STS_IOERR;
md_write_end(conf->mddev);
bio_endio(bi);
bi = nextbi;
sh->dev[i].sector + STRIPE_SECTORS) {
struct bio *bi2 = r5_next_bio(bi, sh->dev[i].sector);
- bi->bi_error = -EIO;
+ bi->bi_status = BLK_STS_IOERR;
md_write_end(conf->mddev);
bio_endio(bi);
bi = bi2;
struct bio *nextbi =
r5_next_bio(bi, sh->dev[i].sector);
- bi->bi_error = -EIO;
+ bi->bi_status = BLK_STS_IOERR;
bio_endio(bi);
bi = nextbi;
}
struct mddev *mddev;
struct r5conf *conf;
struct md_rdev *rdev;
- int error = bi->bi_error;
+ blk_status_t error = bi->bi_status;
bio_put(bi);
release_stripe_plug(mddev, sh);
} else {
/* cannot get stripe for read-ahead, just give-up */
- bi->bi_error = -EIO;
+ bi->bi_status = BLK_STS_IOERR;
break;
}
}
goto abort;
}
- conf->bio_split = bioset_create(BIO_POOL_SIZE, 0);
+ conf->bio_split = bioset_create(BIO_POOL_SIZE, 0, 0);
if (!conf->bio_split)
goto abort;
conf->mddev = mddev;
return PTR_ERR(aemif->clk);
}
- clk_prepare_enable(aemif->clk);
+ ret = clk_prepare_enable(aemif->clk);
+ if (ret)
+ return ret;
+
aemif->clk_rate = clk_get_rate(aemif->clk) / MSEC_PER_SEC;
if (of_device_is_compatible(np, "ti,da850-aemif"))
spin_lock_irqsave(&msb->q_lock, flags);
if (len)
- if (!__blk_end_request(msb->req, 0, len))
+ if (!__blk_end_request(msb->req, BLK_STS_OK, len))
msb->req = NULL;
if (error && msb->req) {
+ blk_status_t ret = errno_to_blk_status(error);
dbg_verbose("IO: ending one sector of the request with error");
- if (!__blk_end_request(msb->req, error, msb->page_size))
+ if (!__blk_end_request(msb->req, ret, msb->page_size))
msb->req = NULL;
}
WARN_ON(!msb->io_queue_stopped);
while ((req = blk_fetch_request(q)) != NULL)
- __blk_end_request_all(req, -ENODEV);
+ __blk_end_request_all(req, BLK_STS_IOERR);
return;
}
msb->req_sg);
if (!msb->seg_count) {
- chunk = __blk_end_request_cur(msb->block_req, -ENOMEM);
+ chunk = __blk_end_request_cur(msb->block_req,
+ BLK_STS_RESOURCE);
continue;
}
if (error && !t_len)
t_len = blk_rq_cur_bytes(msb->block_req);
- chunk = __blk_end_request(msb->block_req, error, t_len);
+ chunk = __blk_end_request(msb->block_req,
+ errno_to_blk_status(error), t_len);
error = mspro_block_issue_req(card, chunk);
if (msb->eject) {
while ((req = blk_fetch_request(q)) != NULL)
- __blk_end_request_all(req, -ENODEV);
+ __blk_end_request_all(req, BLK_STS_IOERR);
return;
}
int ret;
ret = regmap_read_poll_timeout(arizona->regmap,
- ARIZONA_INTERRUPT_RAW_STATUS_5, val,
- ((val & mask) == target),
+ reg, val, ((val & mask) == target),
ARIZONA_REG_POLL_DELAY_US,
timeout_ms * 1000);
if (ret)
goto err_sleep_init;
}
- /* Return if there is no sleep keepon data. */
- if (!pmic_pdata->slp_keepon)
- return 0;
-
- if (pmic_pdata->slp_keepon->therm_keepon) {
+ if (pmic_pdata->slp_keepon.therm_keepon) {
ret = tps65910_reg_set_bits(tps65910,
TPS65910_SLEEP_KEEP_RES_ON,
SLEEP_KEEP_RES_ON_THERM_KEEPON_MASK);
}
}
- if (pmic_pdata->slp_keepon->clkout32k_keepon) {
+ if (pmic_pdata->slp_keepon.clkout32k_keepon) {
ret = tps65910_reg_set_bits(tps65910,
TPS65910_SLEEP_KEEP_RES_ON,
SLEEP_KEEP_RES_ON_CLKOUT32K_KEEPON_MASK);
}
}
- if (pmic_pdata->slp_keepon->i2chs_keepon) {
+ if (pmic_pdata->slp_keepon.i2chs_keepon) {
ret = tps65910_reg_set_bits(tps65910,
TPS65910_SLEEP_KEEP_RES_ON,
SLEEP_KEEP_RES_ON_I2CHS_KEEPON_MASK);
prop = of_property_read_bool(np, "ti,en-ck32k-xtal");
board_info->en_ck32k_xtal = prop;
+ prop = of_property_read_bool(np, "ti,sleep-enable");
+ board_info->en_dev_slp = prop;
+
+ prop = of_property_read_bool(np, "ti,sleep-keep-therm");
+ board_info->slp_keepon.therm_keepon = prop;
+
+ prop = of_property_read_bool(np, "ti,sleep-keep-ck32k");
+ board_info->slp_keepon.clkout32k_keepon = prop;
+
+ prop = of_property_read_bool(np, "ti,sleep-keep-hsclk");
+ board_info->slp_keepon.i2chs_keepon = prop;
+
board_info->irq = client->irq;
board_info->irq_base = -1;
board_info->pm_off = of_property_read_bool(np,
ioctl()s, the driver also provides a read/write interface to a BMC ram
region where the host LPC read/write region can be buffered.
+config ASPEED_LPC_SNOOP
+ tristate "Aspeed ast2500 HOST LPC snoop support"
+ depends on (ARCH_ASPEED || COMPILE_TEST) && REGMAP && MFD_SYSCON
+ help
+ Provides a driver to control the LPC snoop interface which
+ allows the BMC to listen on and save the data written by
+ the host to an arbitrary LPC I/O port.
+
config PCI_ENDPOINT_TEST
depends on PCI
select CRC32
obj-$(CONFIG_VEXPRESS_SYSCFG) += vexpress-syscfg.o
obj-$(CONFIG_CXL_BASE) += cxl/
obj-$(CONFIG_ASPEED_LPC_CTRL) += aspeed-lpc-ctrl.o
+obj-$(CONFIG_ASPEED_LPC_SNOOP) += aspeed-lpc-snoop.o
obj-$(CONFIG_PCI_ENDPOINT_TEST) += pci_endpoint_test.o
lkdtm-$(CONFIG_LKDTM) += lkdtm_core.o
#include <linux/delay.h>
#include <linux/wait.h>
#include <linux/slab.h>
-#include <linux/i2c/apds990x.h>
+#include <linux/platform_data/apds990x.h>
/* Register map */
#define APDS990X_ENABLE 0x00 /* Enable of states and interrupts */
static DEVICE_ATTR(prox0_raw_en, S_IRUGO | S_IWUSR, apds990x_prox_enable_show,
apds990x_prox_enable_store);
-static const char reporting_modes[][9] = {"trigger", "periodic"};
+static const char *reporting_modes[] = {"trigger", "periodic"};
static ssize_t apds990x_prox_reporting_mode_show(struct device *dev,
struct device_attribute *attr, char *buf)
const char *buf, size_t len)
{
struct apds990x_chip *chip = dev_get_drvdata(dev);
+ int ret;
- if (sysfs_streq(buf, reporting_modes[0]))
- chip->prox_continuous_mode = 0;
- else if (sysfs_streq(buf, reporting_modes[1]))
- chip->prox_continuous_mode = 1;
- else
- return -EINVAL;
+ ret = sysfs_match_string(reporting_modes, buf);
+ if (ret < 0)
+ return ret;
+
+ chip->prox_continuous_mode = ret;
return len;
}
--- /dev/null
+/*
+ * Copyright 2017 Google Inc
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ *
+ * Provides a simple driver to control the ASPEED LPC snoop interface which
+ * allows the BMC to listen on and save the data written by
+ * the host to an arbitrary LPC I/O port.
+ *
+ * Typically used by the BMC to "watch" host boot progress via port
+ * 0x80 writes made by the BIOS during the boot process.
+ */
+
+#include <linux/bitops.h>
+#include <linux/interrupt.h>
+#include <linux/kfifo.h>
+#include <linux/mfd/syscon.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+
+#define DEVICE_NAME "aspeed-lpc-snoop"
+
+#define NUM_SNOOP_CHANNELS 2
+#define SNOOP_FIFO_SIZE 2048
+
+#define HICR5 0x0
+#define HICR5_EN_SNP0W BIT(0)
+#define HICR5_ENINT_SNP0W BIT(1)
+#define HICR5_EN_SNP1W BIT(2)
+#define HICR5_ENINT_SNP1W BIT(3)
+
+#define HICR6 0x4
+#define HICR6_STR_SNP0W BIT(0)
+#define HICR6_STR_SNP1W BIT(1)
+#define SNPWADR 0x10
+#define SNPWADR_CH0_MASK GENMASK(15, 0)
+#define SNPWADR_CH0_SHIFT 0
+#define SNPWADR_CH1_MASK GENMASK(31, 16)
+#define SNPWADR_CH1_SHIFT 16
+#define SNPWDR 0x14
+#define SNPWDR_CH0_MASK GENMASK(7, 0)
+#define SNPWDR_CH0_SHIFT 0
+#define SNPWDR_CH1_MASK GENMASK(15, 8)
+#define SNPWDR_CH1_SHIFT 8
+#define HICRB 0x80
+#define HICRB_ENSNP0D BIT(14)
+#define HICRB_ENSNP1D BIT(15)
+
+struct aspeed_lpc_snoop {
+ struct regmap *regmap;
+ int irq;
+ struct kfifo snoop_fifo[NUM_SNOOP_CHANNELS];
+};
+
+/* Save a byte to a FIFO and discard the oldest byte if FIFO is full */
+static void put_fifo_with_discard(struct kfifo *fifo, u8 val)
+{
+ if (!kfifo_initialized(fifo))
+ return;
+ if (kfifo_is_full(fifo))
+ kfifo_skip(fifo);
+ kfifo_put(fifo, val);
+}
+
+static irqreturn_t aspeed_lpc_snoop_irq(int irq, void *arg)
+{
+ struct aspeed_lpc_snoop *lpc_snoop = arg;
+ u32 reg, data;
+
+ if (regmap_read(lpc_snoop->regmap, HICR6, ®))
+ return IRQ_NONE;
+
+ /* Check if one of the snoop channels is interrupting */
+ reg &= (HICR6_STR_SNP0W | HICR6_STR_SNP1W);
+ if (!reg)
+ return IRQ_NONE;
+
+ /* Ack pending IRQs */
+ regmap_write(lpc_snoop->regmap, HICR6, reg);
+
+ /* Read and save most recent snoop'ed data byte to FIFO */
+ regmap_read(lpc_snoop->regmap, SNPWDR, &data);
+
+ if (reg & HICR6_STR_SNP0W) {
+ u8 val = (data & SNPWDR_CH0_MASK) >> SNPWDR_CH0_SHIFT;
+
+ put_fifo_with_discard(&lpc_snoop->snoop_fifo[0], val);
+ }
+ if (reg & HICR6_STR_SNP1W) {
+ u8 val = (data & SNPWDR_CH1_MASK) >> SNPWDR_CH1_SHIFT;
+
+ put_fifo_with_discard(&lpc_snoop->snoop_fifo[1], val);
+ }
+
+ return IRQ_HANDLED;
+}
+
+static int aspeed_lpc_snoop_config_irq(struct aspeed_lpc_snoop *lpc_snoop,
+ struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ int rc;
+
+ lpc_snoop->irq = platform_get_irq(pdev, 0);
+ if (!lpc_snoop->irq)
+ return -ENODEV;
+
+ rc = devm_request_irq(dev, lpc_snoop->irq,
+ aspeed_lpc_snoop_irq, IRQF_SHARED,
+ DEVICE_NAME, lpc_snoop);
+ if (rc < 0) {
+ dev_warn(dev, "Unable to request IRQ %d\n", lpc_snoop->irq);
+ lpc_snoop->irq = 0;
+ return rc;
+ }
+
+ return 0;
+}
+
+static int aspeed_lpc_enable_snoop(struct aspeed_lpc_snoop *lpc_snoop,
+ int channel, u16 lpc_port)
+{
+ int rc = 0;
+ u32 hicr5_en, snpwadr_mask, snpwadr_shift, hicrb_en;
+
+ /* Create FIFO datastructure */
+ rc = kfifo_alloc(&lpc_snoop->snoop_fifo[channel],
+ SNOOP_FIFO_SIZE, GFP_KERNEL);
+ if (rc)
+ return rc;
+
+ /* Enable LPC snoop channel at requested port */
+ switch (channel) {
+ case 0:
+ hicr5_en = HICR5_EN_SNP0W | HICR5_ENINT_SNP0W;
+ snpwadr_mask = SNPWADR_CH0_MASK;
+ snpwadr_shift = SNPWADR_CH0_SHIFT;
+ hicrb_en = HICRB_ENSNP0D;
+ break;
+ case 1:
+ hicr5_en = HICR5_EN_SNP1W | HICR5_ENINT_SNP1W;
+ snpwadr_mask = SNPWADR_CH1_MASK;
+ snpwadr_shift = SNPWADR_CH1_SHIFT;
+ hicrb_en = HICRB_ENSNP1D;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ regmap_update_bits(lpc_snoop->regmap, HICR5, hicr5_en, hicr5_en);
+ regmap_update_bits(lpc_snoop->regmap, SNPWADR, snpwadr_mask,
+ lpc_port << snpwadr_shift);
+ regmap_update_bits(lpc_snoop->regmap, HICRB, hicrb_en, hicrb_en);
+
+ return rc;
+}
+
+static void aspeed_lpc_disable_snoop(struct aspeed_lpc_snoop *lpc_snoop,
+ int channel)
+{
+ switch (channel) {
+ case 0:
+ regmap_update_bits(lpc_snoop->regmap, HICR5,
+ HICR5_EN_SNP0W | HICR5_ENINT_SNP0W,
+ 0);
+ break;
+ case 1:
+ regmap_update_bits(lpc_snoop->regmap, HICR5,
+ HICR5_EN_SNP1W | HICR5_ENINT_SNP1W,
+ 0);
+ break;
+ default:
+ return;
+ }
+
+ kfifo_free(&lpc_snoop->snoop_fifo[channel]);
+}
+
+static int aspeed_lpc_snoop_probe(struct platform_device *pdev)
+{
+ struct aspeed_lpc_snoop *lpc_snoop;
+ struct device *dev;
+ u32 port;
+ int rc;
+
+ dev = &pdev->dev;
+
+ lpc_snoop = devm_kzalloc(dev, sizeof(*lpc_snoop), GFP_KERNEL);
+ if (!lpc_snoop)
+ return -ENOMEM;
+
+ lpc_snoop->regmap = syscon_node_to_regmap(
+ pdev->dev.parent->of_node);
+ if (IS_ERR(lpc_snoop->regmap)) {
+ dev_err(dev, "Couldn't get regmap\n");
+ return -ENODEV;
+ }
+
+ dev_set_drvdata(&pdev->dev, lpc_snoop);
+
+ rc = of_property_read_u32_index(dev->of_node, "snoop-ports", 0, &port);
+ if (rc) {
+ dev_err(dev, "no snoop ports configured\n");
+ return -ENODEV;
+ }
+
+ rc = aspeed_lpc_snoop_config_irq(lpc_snoop, pdev);
+ if (rc)
+ return rc;
+
+ rc = aspeed_lpc_enable_snoop(lpc_snoop, 0, port);
+ if (rc)
+ return rc;
+
+ /* Configuration of 2nd snoop channel port is optional */
+ if (of_property_read_u32_index(dev->of_node, "snoop-ports",
+ 1, &port) == 0) {
+ rc = aspeed_lpc_enable_snoop(lpc_snoop, 1, port);
+ if (rc)
+ aspeed_lpc_disable_snoop(lpc_snoop, 0);
+ }
+
+ return rc;
+}
+
+static int aspeed_lpc_snoop_remove(struct platform_device *pdev)
+{
+ struct aspeed_lpc_snoop *lpc_snoop = dev_get_drvdata(&pdev->dev);
+
+ /* Disable both snoop channels */
+ aspeed_lpc_disable_snoop(lpc_snoop, 0);
+ aspeed_lpc_disable_snoop(lpc_snoop, 1);
+
+ return 0;
+}
+
+static const struct of_device_id aspeed_lpc_snoop_match[] = {
+ { .compatible = "aspeed,ast2500-lpc-snoop" },
+ { },
+};
+
+static struct platform_driver aspeed_lpc_snoop_driver = {
+ .driver = {
+ .name = DEVICE_NAME,
+ .of_match_table = aspeed_lpc_snoop_match,
+ },
+ .probe = aspeed_lpc_snoop_probe,
+ .remove = aspeed_lpc_snoop_remove,
+};
+
+module_platform_driver(aspeed_lpc_snoop_driver);
+
+MODULE_DEVICE_TABLE(of, aspeed_lpc_snoop_match);
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Robert Lippert <rlippert@google.com>");
+MODULE_DESCRIPTION("Linux driver to control Aspeed LPC snoop functionality");
#include <linux/i2c.h>
#include <linux/interrupt.h>
#include <linux/mutex.h>
-#include <linux/i2c/bh1770glc.h>
+#include <linux/platform_data/bh1770glc.h>
#include <linux/regulator/consumer.h>
#include <linux/pm_runtime.h>
#include <linux/workqueue.h>
mutex_init(&ctx->mapping_lock);
ctx->mapping = NULL;
- if (cxl_is_psl8(afu)) {
+ if (cxl_is_power8()) {
spin_lock_init(&ctx->sste_lock);
/*
if (start + len > ctx->afu->adapter->ps_size)
return -EINVAL;
- if (cxl_is_psl9(ctx->afu)) {
+ if (cxl_is_power9()) {
/*
* Make sure there is a valid problem state
* area space for this AFU.
{
struct cxl_context *ctx = container_of(rcu, struct cxl_context, rcu);
- if (cxl_is_psl8(ctx->afu))
+ if (cxl_is_power8())
free_page((u64)ctx->sstp);
if (ctx->ff_page)
__free_page(ctx->ff_page);
#define CXL_PSL9_DSISR_An_PF_RGP 0x0000000000000090ULL /* PTE not found (Radix Guest (parent)) 0b10010000 */
#define CXL_PSL9_DSISR_An_PF_HRH 0x0000000000000094ULL /* PTE not found (HPT/Radix Host) 0b10010100 */
#define CXL_PSL9_DSISR_An_PF_STEG 0x000000000000009CULL /* PTE not found (STEG VA) 0b10011100 */
+#define CXL_PSL9_DSISR_An_URTCH 0x00000000000000B4ULL /* Unsupported Radix Tree Configuration 0b10110100 */
/****** CXL_PSL_TFC_An ******************************************************/
#define CXL_PSL_TFC_An_A (1ull << (63-28)) /* Acknowledge non-translation fault */
static inline bool cxl_is_power9(void)
{
- /* intermediate solution */
- if (!cxl_is_power8() &&
- (cpu_has_feature(CPU_FTRS_POWER9) ||
- cpu_has_feature(CPU_FTR_POWER9_DD1)))
+ if (pvr_version_is(PVR_POWER9))
return true;
return false;
}
-static inline bool cxl_is_psl8(struct cxl_afu *afu)
+static inline bool cxl_is_power9_dd1(void)
{
- if (afu->adapter->caia_major == 1)
- return true;
- return false;
-}
-
-static inline bool cxl_is_psl9(struct cxl_afu *afu)
-{
- if (afu->adapter->caia_major == 2)
+ if ((pvr_version_is(PVR_POWER9)) &&
+ cpu_has_feature(CPU_FTR_POWER9_DD1))
return true;
return false;
}
static bool cxl_is_segment_miss(struct cxl_context *ctx, u64 dsisr)
{
- if ((cxl_is_psl8(ctx->afu)) && (dsisr & CXL_PSL_DSISR_An_DS))
+ if ((cxl_is_power8() && (dsisr & CXL_PSL_DSISR_An_DS)))
return true;
return false;
static bool cxl_is_page_fault(struct cxl_context *ctx, u64 dsisr)
{
- if ((cxl_is_psl8(ctx->afu)) && (dsisr & CXL_PSL_DSISR_An_DM))
- return true;
+ u64 crs; /* Translation Checkout Response Status */
- if ((cxl_is_psl9(ctx->afu)) &&
- ((dsisr & CXL_PSL9_DSISR_An_CO_MASK) &
- (CXL_PSL9_DSISR_An_PF_SLR | CXL_PSL9_DSISR_An_PF_RGC |
- CXL_PSL9_DSISR_An_PF_RGP | CXL_PSL9_DSISR_An_PF_HRH |
- CXL_PSL9_DSISR_An_PF_STEG)))
+ if ((cxl_is_power8()) && (dsisr & CXL_PSL_DSISR_An_DM))
return true;
+ if (cxl_is_power9()) {
+ crs = (dsisr & CXL_PSL9_DSISR_An_CO_MASK);
+ if ((crs == CXL_PSL9_DSISR_An_PF_SLR) ||
+ (crs == CXL_PSL9_DSISR_An_PF_RGC) ||
+ (crs == CXL_PSL9_DSISR_An_PF_RGP) ||
+ (crs == CXL_PSL9_DSISR_An_PF_HRH) ||
+ (crs == CXL_PSL9_DSISR_An_PF_STEG) ||
+ (crs == CXL_PSL9_DSISR_An_URTCH)) {
+ return true;
+ }
+ }
+
return false;
}
cxl_debugfs_init();
- if ((rc = register_cxl_calls(&cxl_calls)))
- goto err;
+ /*
+ * we don't register the callback on P9. slb callack is only
+ * used for the PSL8 MMU and CX4.
+ */
+ if (cxl_is_power8()) {
+ rc = register_cxl_calls(&cxl_calls);
+ if (rc)
+ goto err;
+ }
if (cpu_has_feature(CPU_FTR_HVMODE)) {
cxl_ops = &cxl_native_ops;
return 0;
err1:
- unregister_cxl_calls(&cxl_calls);
+ if (cxl_is_power8())
+ unregister_cxl_calls(&cxl_calls);
err:
cxl_debugfs_exit();
cxl_file_exit();
cxl_debugfs_exit();
cxl_file_exit();
- unregister_cxl_calls(&cxl_calls);
+ if (cxl_is_power8())
+ unregister_cxl_calls(&cxl_calls);
idr_destroy(&cxl_adapter_idr);
}
CXL_AFU_Cntl_An_RS_MASK | CXL_AFU_Cntl_An_ES_MASK,
false);
- /* Re-enable any masked interrupts */
- serr = cxl_p1n_read(afu, CXL_PSL_SERR_An);
- serr &= ~CXL_PSL_SERR_An_IRQ_MASKS;
- cxl_p1n_write(afu, CXL_PSL_SERR_An, serr);
-
+ /*
+ * Re-enable any masked interrupts when the AFU is not
+ * activated to avoid side effects after attaching a process
+ * in dedicated mode.
+ */
+ if (afu->current_mode == 0) {
+ serr = cxl_p1n_read(afu, CXL_PSL_SERR_An);
+ serr &= ~CXL_PSL_SERR_An_IRQ_MASKS;
+ cxl_p1n_write(afu, CXL_PSL_SERR_An, serr);
+ }
return rc;
}
pr_devel("PSL purge request\n");
- if (cxl_is_psl8(afu))
+ if (cxl_is_power8())
trans_fault = CXL_PSL_DSISR_TRANS;
- if (cxl_is_psl9(afu))
+ if (cxl_is_power9())
trans_fault = CXL_PSL9_DSISR_An_TF;
if (!cxl_ops->link_ok(afu->adapter, afu)) {
if (!test_tsk_thread_flag(current, TIF_32BIT))
sr |= CXL_PSL_SR_An_SF;
}
- if (cxl_is_psl9(ctx->afu)) {
+ if (cxl_is_power9()) {
if (radix_enabled())
sr |= CXL_PSL_SR_An_XLAT_ror;
else
static bool cxl_is_translation_fault(struct cxl_afu *afu, u64 dsisr)
{
- if ((cxl_is_psl8(afu)) && (dsisr & CXL_PSL_DSISR_TRANS))
+ if ((cxl_is_power8()) && (dsisr & CXL_PSL_DSISR_TRANS))
return true;
- if ((cxl_is_psl9(afu)) && (dsisr & CXL_PSL9_DSISR_An_TF))
+ if ((cxl_is_power9()) && (dsisr & CXL_PSL9_DSISR_An_TF))
return true;
return false;
if (ph != ctx->pe)
return;
dsisr = cxl_p2n_read(ctx->afu, CXL_PSL_DSISR_An);
- if (cxl_is_psl8(ctx->afu) &&
+ if (cxl_is_power8() &&
((dsisr & CXL_PSL_DSISR_PENDING) == 0))
return;
- if (cxl_is_psl9(ctx->afu) &&
+ if (cxl_is_power9() &&
((dsisr & CXL_PSL9_DSISR_PENDING) == 0))
return;
/*
/* nMMU_ID Defaults to: b’000001001’*/
xsl_dsnctl |= ((u64)0x09 << (63-28));
- if (cxl_is_power9() && !cpu_has_feature(CPU_FTR_POWER9_DD1)) {
+ if (!(cxl_is_power9_dd1())) {
/*
* Used to identify CAPI packets which should be sorted into
* the Non-Blocking queues by the PHB. This field should match
cxl_p1_write(adapter, CXL_PSL9_APCDEDTYPE, 0x40000003FFFF0000ULL);
/* Disable vc dd1 fix */
- if ((cxl_is_power9() && cpu_has_feature(CPU_FTR_POWER9_DD1)))
+ if (cxl_is_power9_dd1())
cxl_p1_write(adapter, CXL_PSL9_GP_CT, 0x0400000000000001ULL);
return 0;
* The adapter is about to be reset, so ignore errors.
* Not supported on P9 DD1
*/
- if ((cxl_is_power8()) ||
- ((cxl_is_power9() && !cpu_has_feature(CPU_FTR_POWER9_DD1))))
+ if ((cxl_is_power8()) || (!(cxl_is_power9_dd1())))
cxl_data_cache_flush(adapter);
/* pcie_warm_reset requests a fundamental pci reset which includes a
.debugfs_add_adapter_regs = cxl_debugfs_add_adapter_regs_psl9,
.debugfs_add_afu_regs = cxl_debugfs_add_afu_regs_psl9,
.psl_irq_dump_registers = cxl_native_irq_dump_regs_psl9,
- .err_irq_dump_registers = cxl_native_err_irq_dump_regs,
.debugfs_stop_trace = cxl_stop_trace_psl9,
.write_timebase_ctrl = write_timebase_ctrl_psl9,
.timebase_read = timebase_read_psl9,
* Flush adapter datacache as its about to be removed.
* Not supported on P9 DD1.
*/
- if ((cxl_is_power8()) ||
- ((cxl_is_power9() && !cpu_has_feature(CPU_FTR_POWER9_DD1))))
+ if ((cxl_is_power8()) || (!(cxl_is_power9_dd1())))
cxl_data_cache_flush(adapter);
cxl_deconfigure_adapter(adapter);
*
* @bus: mei device
*/
-void mei_cl_bus_rescan(struct mei_device *bus)
+static void mei_cl_bus_rescan(struct mei_device *bus)
{
struct mei_cl_device *cldev, *n;
struct mei_me_client *me_cl;
#define HBM_MAJOR_VERSION_DOT 2
/*
- * MEI version with notifcation support
+ * MEI version with notification support
*/
#define HBM_MINOR_VERSION_EV 0
#define HBM_MAJOR_VERSION_EV 2
}
} while (ret);
- /* we cannot start the device w/o hbm start message completed */
- if (dev->dev_state == MEI_DEV_DISABLED) {
- dev_err(dev->dev, "reset failed");
- goto err;
- }
-
if (mei_hbm_start_wait(dev)) {
dev_err(dev->dev, "HBM haven't started");
goto err;
return mei_hdr->host_addr == 0 && mei_hdr->me_addr != 0;
}
+static inline int hdr_is_valid(u32 msg_hdr)
+{
+ struct mei_msg_hdr *mei_hdr;
+
+ mei_hdr = (struct mei_msg_hdr *)&msg_hdr;
+ if (!msg_hdr || mei_hdr->reserved)
+ return -EBADMSG;
+
+ return 0;
+}
+
/**
* mei_irq_read_handler - bottom half read routine after ISR to
* handle the read processing.
dev->rd_msg_hdr = mei_read_hdr(dev);
(*slots)--;
dev_dbg(dev->dev, "slots =%08x.\n", *slots);
- }
- mei_hdr = (struct mei_msg_hdr *) &dev->rd_msg_hdr;
- dev_dbg(dev->dev, MEI_HDR_FMT, MEI_HDR_PRM(mei_hdr));
- if (mei_hdr->reserved || !dev->rd_msg_hdr) {
- dev_err(dev->dev, "corrupted message header 0x%08X\n",
+ ret = hdr_is_valid(dev->rd_msg_hdr);
+ if (ret) {
+ dev_err(dev->dev, "corrupted message header 0x%08X\n",
dev->rd_msg_hdr);
- ret = -EBADMSG;
- goto end;
+ goto end;
+ }
}
+ mei_hdr = (struct mei_msg_hdr *)&dev->rd_msg_hdr;
+ dev_dbg(dev->dev, MEI_HDR_FMT, MEI_HDR_PRM(mei_hdr));
+
if (mei_slots2data(*slots) < mei_hdr->length) {
dev_err(dev->dev, "less data available than length=%08x.\n",
*slots);
};
/* MEI bus API*/
-void mei_cl_bus_rescan(struct mei_device *bus);
void mei_cl_bus_rescan_work(struct work_struct *work);
void mei_cl_bus_dev_fixup(struct mei_cl_device *dev);
ssize_t __mei_cl_send(struct mei_cl *cl, u8 *buf, size_t length,
#include <linux/mm.h>
#include <linux/sram.h>
+#include <asm/fncpy.h>
#include <asm/set_memory.h>
#include "sram.h"
* @src: Source address for the data to copy
* @size: Size of copy to perform, which starting from dst, must reside in pool
*
+ * Return: Address for copied data that can safely be called through function
+ * pointer, or NULL if problem.
+ *
* This helper function allows sram driver to act as central control location
* of 'protect-exec' pools which are normal sram pools but are always set
* read-only and executable except when copying data to them, at which point
* they are set to read-write non-executable, to make sure no memory is
* writeable and executable at the same time. This region must be page-aligned
* and is checked during probe, otherwise page attribute manipulation would
- * not be possible.
+ * not be possible. Care must be taken to only call the returned address as
+ * dst address is not guaranteed to be safely callable.
+ *
+ * NOTE: This function uses the fncpy macro to move code to the executable
+ * region. Some architectures have strict requirements for relocating
+ * executable code, so fncpy is a macro that must be defined by any arch
+ * making use of this functionality that guarantees a safe copy of exec
+ * data and returns a safe address that can be called as a C function
+ * pointer.
*/
-int sram_exec_copy(struct gen_pool *pool, void *dst, void *src,
- size_t size)
+void *sram_exec_copy(struct gen_pool *pool, void *dst, void *src,
+ size_t size)
{
struct sram_partition *part = NULL, *p;
unsigned long base;
int pages;
+ void *dst_cpy;
mutex_lock(&exec_pool_list_mutex);
list_for_each_entry(p, &exec_pool_list, list) {
mutex_unlock(&exec_pool_list_mutex);
if (!part)
- return -EINVAL;
+ return NULL;
if (!addr_in_gen_pool(pool, (unsigned long)dst, size))
- return -EINVAL;
+ return NULL;
base = (unsigned long)part->base;
pages = PAGE_ALIGN(size) / PAGE_SIZE;
set_memory_nx((unsigned long)base, pages);
set_memory_rw((unsigned long)base, pages);
- memcpy(dst, src, size);
+ dst_cpy = fncpy(dst, src, size);
set_memory_ro((unsigned long)base, pages);
set_memory_x((unsigned long)base, pages);
mutex_unlock(&part->lock);
- return 0;
+ return dst_cpy;
}
EXPORT_SYMBOL_GPL(sram_exec_copy);
If unsure, say 8 here.
-config MMC_BLOCK_BOUNCE
- bool "Use bounce buffer for simple hosts"
- depends on MMC_BLOCK
- default y
- help
- SD/MMC is a high latency protocol where it is crucial to
- send large requests in order to get high performance. Many
- controllers, however, are restricted to continuous memory
- (i.e. they can't do scatter-gather), something the kernel
- rarely can provide.
-
- Say Y here to help these restricted hosts by bouncing
- requests back and forth from a large buffer. You will get
- a big performance gain at the cost of up to 64 KiB of
- physical memory.
-
- If unsure, say Y here.
-
config SDIO_UART
tristate "SDIO UART/GPS class support"
depends on TTY
static inline int mmc_blk_part_switch(struct mmc_card *card,
struct mmc_blk_data *md);
-static int get_card_status(struct mmc_card *card, u32 *status, int retries);
-
-static void mmc_blk_requeue(struct request_queue *q, struct request *req)
-{
- spin_lock_irq(q->queue_lock);
- blk_requeue_request(q, req);
- spin_unlock_irq(q->queue_lock);
-}
static struct mmc_blk_data *mmc_blk_get(struct gendisk *disk)
{
int ret;
struct mmc_blk_data *md, *part_md;
struct mmc_card *card;
+ struct mmc_queue *mq;
+ struct request *req;
unsigned long set;
if (kstrtoul(buf, 0, &set))
return count;
md = mmc_blk_get(dev_to_disk(dev));
+ mq = &md->queue;
card = md->queue.card;
- mmc_get_card(card);
-
- ret = mmc_switch(card, EXT_CSD_CMD_SET_NORMAL, EXT_CSD_BOOT_WP,
- card->ext_csd.boot_ro_lock |
- EXT_CSD_BOOT_WP_B_PWR_WP_EN,
- card->ext_csd.part_time);
- if (ret)
- pr_err("%s: Locking boot partition ro until next power on failed: %d\n", md->disk->disk_name, ret);
- else
- card->ext_csd.boot_ro_lock |= EXT_CSD_BOOT_WP_B_PWR_WP_EN;
-
- mmc_put_card(card);
+ /* Dispatch locking to the block layer */
+ req = blk_get_request(mq->queue, REQ_OP_DRV_OUT, __GFP_RECLAIM);
+ req_to_mmc_queue_req(req)->drv_op = MMC_DRV_OP_BOOT_WP;
+ blk_execute_rq(mq->queue, NULL, req, 0);
+ ret = req_to_mmc_queue_req(req)->drv_op_result;
if (!ret) {
pr_info("%s: Locking boot partition ro until next power on\n",
return -EINVAL;
do {
- err = get_card_status(card, status, 5);
+ err = __mmc_send_status(card, status, 5);
if (err)
break;
struct mmc_request mrq = {};
struct scatterlist sg;
int err;
- int is_rpmb = false;
+ bool is_rpmb = false;
u32 status = 0;
if (!card || !md || !idata)
struct mmc_ioc_cmd __user *ic_ptr)
{
struct mmc_blk_ioc_data *idata;
+ struct mmc_blk_ioc_data *idatas[1];
struct mmc_blk_data *md;
+ struct mmc_queue *mq;
struct mmc_card *card;
int err = 0, ioc_err = 0;
+ struct request *req;
/*
* The caller must have CAP_SYS_RAWIO, and must be calling this on the
goto cmd_done;
}
- mmc_get_card(card);
-
- ioc_err = __mmc_blk_ioctl_cmd(card, md, idata);
-
- /* Always switch back to main area after RPMB access */
- if (md->area_type & MMC_BLK_DATA_AREA_RPMB)
- mmc_blk_part_switch(card, dev_get_drvdata(&card->dev));
-
- mmc_put_card(card);
-
+ /*
+ * Dispatch the ioctl() into the block request queue.
+ */
+ mq = &md->queue;
+ req = blk_get_request(mq->queue,
+ idata->ic.write_flag ? REQ_OP_DRV_OUT : REQ_OP_DRV_IN,
+ __GFP_RECLAIM);
+ idatas[0] = idata;
+ req_to_mmc_queue_req(req)->drv_op = MMC_DRV_OP_IOCTL;
+ req_to_mmc_queue_req(req)->idata = idatas;
+ req_to_mmc_queue_req(req)->ioc_count = 1;
+ blk_execute_rq(mq->queue, NULL, req, 0);
+ ioc_err = req_to_mmc_queue_req(req)->drv_op_result;
err = mmc_blk_ioctl_copy_to_user(ic_ptr, idata);
+ blk_put_request(req);
cmd_done:
mmc_blk_put(md);
struct mmc_ioc_cmd __user *cmds = user->cmds;
struct mmc_card *card;
struct mmc_blk_data *md;
+ struct mmc_queue *mq;
int i, err = 0, ioc_err = 0;
__u64 num_of_cmds;
+ struct request *req;
/*
* The caller must have CAP_SYS_RAWIO, and must be calling this on the
goto cmd_done;
}
- mmc_get_card(card);
-
- for (i = 0; i < num_of_cmds && !ioc_err; i++)
- ioc_err = __mmc_blk_ioctl_cmd(card, md, idata[i]);
-
- /* Always switch back to main area after RPMB access */
- if (md->area_type & MMC_BLK_DATA_AREA_RPMB)
- mmc_blk_part_switch(card, dev_get_drvdata(&card->dev));
- mmc_put_card(card);
+ /*
+ * Dispatch the ioctl()s into the block request queue.
+ */
+ mq = &md->queue;
+ req = blk_get_request(mq->queue,
+ idata[0]->ic.write_flag ? REQ_OP_DRV_OUT : REQ_OP_DRV_IN,
+ __GFP_RECLAIM);
+ req_to_mmc_queue_req(req)->drv_op = MMC_DRV_OP_IOCTL;
+ req_to_mmc_queue_req(req)->idata = idata;
+ req_to_mmc_queue_req(req)->ioc_count = num_of_cmds;
+ blk_execute_rq(mq->queue, NULL, req, 0);
+ ioc_err = req_to_mmc_queue_req(req)->drv_op_result;
/* copy to user if data and response */
for (i = 0; i < num_of_cmds && !err; i++)
err = mmc_blk_ioctl_copy_to_user(&cmds[i], idata[i]);
+ blk_put_request(req);
+
cmd_done:
mmc_blk_put(md);
cmd_err:
return 0;
}
-static int get_card_status(struct mmc_card *card, u32 *status, int retries)
-{
- struct mmc_command cmd = {};
- int err;
-
- cmd.opcode = MMC_SEND_STATUS;
- if (!mmc_host_is_spi(card->host))
- cmd.arg = card->rca << 16;
- cmd.flags = MMC_RSP_SPI_R2 | MMC_RSP_R1 | MMC_CMD_AC;
- err = mmc_wait_for_cmd(card->host, &cmd, retries);
- if (err == 0)
- *status = cmd.resp[0];
- return err;
-}
-
static int card_busy_detect(struct mmc_card *card, unsigned int timeout_ms,
bool hw_busy_detect, struct request *req, bool *gen_err)
{
u32 status;
do {
- err = get_card_status(card, &status, 5);
+ err = __mmc_send_status(card, &status, 5);
if (err) {
pr_err("%s: error %d requesting status\n",
req->rq_disk->disk_name, err);
* we can't be sure the returned status is for the r/w command.
*/
for (retry = 2; retry >= 0; retry--) {
- err = get_card_status(card, &status, 0);
+ err = __mmc_send_status(card, &status, 0);
if (!err)
break;
return false;
}
+/*
+ * The non-block commands come back from the block layer after it queued it and
+ * processed it with all other requests and then they get issued in this
+ * function.
+ */
+static void mmc_blk_issue_drv_op(struct mmc_queue *mq, struct request *req)
+{
+ struct mmc_queue_req *mq_rq;
+ struct mmc_card *card = mq->card;
+ struct mmc_blk_data *md = mq->blkdata;
+ int ret;
+ int i;
+
+ mq_rq = req_to_mmc_queue_req(req);
+
+ switch (mq_rq->drv_op) {
+ case MMC_DRV_OP_IOCTL:
+ for (i = 0; i < mq_rq->ioc_count; i++) {
+ ret = __mmc_blk_ioctl_cmd(card, md, mq_rq->idata[i]);
+ if (ret)
+ break;
+ }
+ /* Always switch back to main area after RPMB access */
+ if (md->area_type & MMC_BLK_DATA_AREA_RPMB)
+ mmc_blk_part_switch(card, dev_get_drvdata(&card->dev));
+ break;
+ case MMC_DRV_OP_BOOT_WP:
+ ret = mmc_switch(card, EXT_CSD_CMD_SET_NORMAL, EXT_CSD_BOOT_WP,
+ card->ext_csd.boot_ro_lock |
+ EXT_CSD_BOOT_WP_B_PWR_WP_EN,
+ card->ext_csd.part_time);
+ if (ret)
+ pr_err("%s: Locking boot partition ro until next power on failed: %d\n",
+ md->disk->disk_name, ret);
+ else
+ card->ext_csd.boot_ro_lock |=
+ EXT_CSD_BOOT_WP_B_PWR_WP_EN;
+ break;
+ default:
+ pr_err("%s: unknown driver specific operation\n",
+ md->disk->disk_name);
+ ret = -EINVAL;
+ break;
+ }
+ mq_rq->drv_op_result = ret;
+ blk_end_request_all(req, ret);
+}
+
static void mmc_blk_issue_discard_rq(struct mmc_queue *mq, struct request *req)
{
struct mmc_blk_data *md = mq->blkdata;
struct mmc_card *card = md->queue.card;
unsigned int from, nr, arg;
int err = 0, type = MMC_BLK_DISCARD;
+ blk_status_t status = BLK_STS_OK;
if (!mmc_can_erase(card)) {
- err = -EOPNOTSUPP;
+ status = BLK_STS_NOTSUPP;
goto fail;
}
if (!err)
err = mmc_erase(card, from, nr, arg);
} while (err == -EIO && !mmc_blk_reset(md, card->host, type));
- if (!err)
+ if (err)
+ status = BLK_STS_IOERR;
+ else
mmc_blk_reset_success(md, type);
fail:
- blk_end_request(req, err, blk_rq_bytes(req));
+ blk_end_request(req, status, blk_rq_bytes(req));
}
static void mmc_blk_issue_secdiscard_rq(struct mmc_queue *mq,
struct mmc_card *card = md->queue.card;
unsigned int from, nr, arg;
int err = 0, type = MMC_BLK_SECDISCARD;
+ blk_status_t status = BLK_STS_OK;
if (!(mmc_can_secure_erase_trim(card))) {
- err = -EOPNOTSUPP;
+ status = BLK_STS_NOTSUPP;
goto out;
}
err = mmc_erase(card, from, nr, arg);
if (err == -EIO)
goto out_retry;
- if (err)
+ if (err) {
+ status = BLK_STS_IOERR;
goto out;
+ }
if (arg == MMC_SECURE_TRIM1_ARG) {
if (card->quirks & MMC_QUIRK_INAND_CMD38) {
err = mmc_erase(card, from, nr, MMC_SECURE_TRIM2_ARG);
if (err == -EIO)
goto out_retry;
- if (err)
+ if (err) {
+ status = BLK_STS_IOERR;
goto out;
+ }
}
out_retry:
if (!err)
mmc_blk_reset_success(md, type);
out:
- blk_end_request(req, err, blk_rq_bytes(req));
+ blk_end_request(req, status, blk_rq_bytes(req));
}
static void mmc_blk_issue_flush(struct mmc_queue *mq, struct request *req)
int ret = 0;
ret = mmc_flush_cache(card);
- if (ret)
- ret = -EIO;
-
- blk_end_request_all(req, ret);
+ blk_end_request_all(req, ret ? BLK_STS_IOERR : BLK_STS_OK);
}
/*
R1_ADDRESS_ERROR | /* Misaligned address */ \
R1_BLOCK_LEN_ERROR | /* Transferred block length incorrect */\
R1_WP_VIOLATION | /* Tried to write to protected block */ \
+ R1_CARD_ECC_FAILED | /* Card ECC failed */ \
R1_CC_ERROR | /* Card controller error */ \
R1_ERROR) /* General/unknown error */
+static bool mmc_blk_has_cmd_err(struct mmc_command *cmd)
+{
+ if (!cmd->error && cmd->resp[0] & CMD_ERRORS)
+ cmd->error = -EIO;
+
+ return cmd->error;
+}
+
static enum mmc_blk_status mmc_blk_err_check(struct mmc_card *card,
struct mmc_async_req *areq)
{
struct mmc_queue_req *mq_mrq = container_of(areq, struct mmc_queue_req,
areq);
struct mmc_blk_request *brq = &mq_mrq->brq;
- struct request *req = mq_mrq->req;
+ struct request *req = mmc_queue_req_to_req(mq_mrq);
int need_retune = card->host->need_retune;
bool ecc_err = false;
bool gen_err = false;
* stop.error indicates a problem with the stop command. Data
* may have been transferred, or may still be transferring.
*/
- if (brq->sbc.error || brq->cmd.error || brq->stop.error ||
+ if (brq->sbc.error || brq->cmd.error || mmc_blk_has_cmd_err(&brq->stop) ||
brq->data.error) {
switch (mmc_blk_cmd_recovery(card, req, brq, &ecc_err, &gen_err)) {
case ERR_RETRY:
return MMC_BLK_RETRY;
}
- if (brq->data.error) {
+ /* Some errors (ECC) are flagged on the next commmand, so check stop, too */
+ if (brq->data.error || brq->stop.error) {
if (need_retune && !brq->retune_retry_done) {
pr_debug("%s: retrying because a re-tune was needed\n",
req->rq_disk->disk_name);
return MMC_BLK_RETRY;
}
pr_err("%s: error %d transferring data, sector %u, nr %u, cmd response %#x, card status %#x\n",
- req->rq_disk->disk_name, brq->data.error,
+ req->rq_disk->disk_name, brq->data.error ?: brq->stop.error,
(unsigned)blk_rq_pos(req),
(unsigned)blk_rq_sectors(req),
brq->cmd.resp[0], brq->stop.resp[0]);
struct mmc_blk_data *md = mq->blkdata;
struct mmc_card *card = md->queue.card;
struct mmc_blk_request *brq = &mqrq->brq;
- struct request *req = mqrq->req;
+ struct request *req = mmc_queue_req_to_req(mqrq);
/*
* Reliable writes are used to implement Forced Unit Access and
{
u32 readcmd, writecmd;
struct mmc_blk_request *brq = &mqrq->brq;
- struct request *req = mqrq->req;
+ struct request *req = mmc_queue_req_to_req(mqrq);
struct mmc_blk_data *md = mq->blkdata;
bool do_rel_wr, do_data_tag;
{
if (mmc_card_removed(card))
req->rq_flags |= RQF_QUIET;
- while (blk_end_request(req, -EIO, blk_rq_cur_bytes(req)));
- mmc_queue_req_free(mq, mqrq);
+ while (blk_end_request(req, BLK_STS_IOERR, blk_rq_cur_bytes(req)));
+ mq->qcnt--;
}
/**
*/
if (mmc_card_removed(mq->card)) {
req->rq_flags |= RQF_QUIET;
- blk_end_request_all(req, -EIO);
- mmc_queue_req_free(mq, mqrq);
+ blk_end_request_all(req, BLK_STS_IOERR);
+ mq->qcnt--; /* FIXME: just set to 0? */
return;
}
/* Else proceed and try to restart the current async request */
bool req_pending = true;
if (new_req) {
- mqrq_cur = mmc_queue_req_find(mq, new_req);
- if (!mqrq_cur) {
- WARN_ON(1);
- mmc_blk_requeue(mq->queue, new_req);
- new_req = NULL;
- }
+ mqrq_cur = req_to_mmc_queue_req(new_req);
+ mq->qcnt++;
}
if (!mq->qcnt)
*/
mq_rq = container_of(old_areq, struct mmc_queue_req, areq);
brq = &mq_rq->brq;
- old_req = mq_rq->req;
+ old_req = mmc_queue_req_to_req(mq_rq);
type = rq_data_dir(old_req) == READ ? MMC_BLK_READ : MMC_BLK_WRITE;
mmc_queue_bounce_post(mq_rq);
*/
mmc_blk_reset_success(md, type);
- req_pending = blk_end_request(old_req, 0,
+ req_pending = blk_end_request(old_req, BLK_STS_OK,
brq->data.bytes_xfered);
/*
* If the blk_end_request function returns non-zero even
if (req_pending)
mmc_blk_rw_cmd_abort(mq, card, old_req, mq_rq);
else
- mmc_queue_req_free(mq, mq_rq);
+ mq->qcnt--;
mmc_blk_rw_try_restart(mq, new_req, mqrq_cur);
return;
}
if (!req_pending) {
- mmc_queue_req_free(mq, mq_rq);
+ mq->qcnt--;
mmc_blk_rw_try_restart(mq, new_req, mqrq_cur);
return;
}
* time, so we only reach here after trying to
* read a single sector.
*/
- req_pending = blk_end_request(old_req, -EIO,
+ req_pending = blk_end_request(old_req, BLK_STS_IOERR,
brq->data.blksz);
if (!req_pending) {
- mmc_queue_req_free(mq, mq_rq);
+ mq->qcnt--;
mmc_blk_rw_try_restart(mq, new_req, mqrq_cur);
return;
}
}
} while (req_pending);
- mmc_queue_req_free(mq, mq_rq);
+ mq->qcnt--;
}
void mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req)
ret = mmc_blk_part_switch(card, md);
if (ret) {
if (req) {
- blk_end_request_all(req, -EIO);
+ blk_end_request_all(req, BLK_STS_IOERR);
}
goto out;
}
- if (req && req_op(req) == REQ_OP_DISCARD) {
- /* complete ongoing async transfer before issuing discard */
- if (mq->qcnt)
- mmc_blk_issue_rw_rq(mq, NULL);
- mmc_blk_issue_discard_rq(mq, req);
- } else if (req && req_op(req) == REQ_OP_SECURE_ERASE) {
- /* complete ongoing async transfer before issuing secure erase*/
- if (mq->qcnt)
- mmc_blk_issue_rw_rq(mq, NULL);
- mmc_blk_issue_secdiscard_rq(mq, req);
- } else if (req && req_op(req) == REQ_OP_FLUSH) {
- /* complete ongoing async transfer before issuing flush */
- if (mq->qcnt)
- mmc_blk_issue_rw_rq(mq, NULL);
- mmc_blk_issue_flush(mq, req);
+ if (req) {
+ switch (req_op(req)) {
+ case REQ_OP_DRV_IN:
+ case REQ_OP_DRV_OUT:
+ /*
+ * Complete ongoing async transfer before issuing
+ * ioctl()s
+ */
+ if (mq->qcnt)
+ mmc_blk_issue_rw_rq(mq, NULL);
+ mmc_blk_issue_drv_op(mq, req);
+ break;
+ case REQ_OP_DISCARD:
+ /*
+ * Complete ongoing async transfer before issuing
+ * discard.
+ */
+ if (mq->qcnt)
+ mmc_blk_issue_rw_rq(mq, NULL);
+ mmc_blk_issue_discard_rq(mq, req);
+ break;
+ case REQ_OP_SECURE_ERASE:
+ /*
+ * Complete ongoing async transfer before issuing
+ * secure erase.
+ */
+ if (mq->qcnt)
+ mmc_blk_issue_rw_rq(mq, NULL);
+ mmc_blk_issue_secdiscard_rq(mq, req);
+ break;
+ case REQ_OP_FLUSH:
+ /*
+ * Complete ongoing async transfer before issuing
+ * flush.
+ */
+ if (mq->qcnt)
+ mmc_blk_issue_rw_rq(mq, NULL);
+ mmc_blk_issue_flush(mq, req);
+ break;
+ default:
+ /* Normal request, just issue it */
+ mmc_blk_issue_rw_rq(mq, req);
+ card->host->context_info.is_waiting_last_req = false;
+ break;
+ }
} else {
- mmc_blk_issue_rw_rq(mq, req);
+ /* No request, flushing the pipeline with NULL */
+ mmc_blk_issue_rw_rq(mq, NULL);
card->host->context_info.is_waiting_last_req = false;
}
{
struct mmc_blk_data *md, *part_md;
char cap_str[10];
- int ret;
/*
* Check that the card supports the command class(es) we need.
mmc_fixup_device(card, mmc_blk_fixups);
- ret = mmc_queue_alloc_shared_queue(card);
- if (ret)
- return ret;
-
md = mmc_blk_alloc(card);
- if (IS_ERR(md)) {
- mmc_queue_free_shared_queue(card);
+ if (IS_ERR(md))
return PTR_ERR(md);
- }
string_get_size((u64)get_capacity(md->disk), 512, STRING_UNITS_2,
cap_str, sizeof(cap_str));
out:
mmc_blk_remove_parts(card, md);
mmc_blk_remove_req(md);
- mmc_queue_free_shared_queue(card);
return 0;
}
pm_runtime_put_noidle(&card->dev);
mmc_blk_remove_req(md);
dev_set_drvdata(&card->dev, NULL);
- mmc_queue_free_shared_queue(card);
}
static int _mmc_blk_suspend(struct mmc_card *card)
/* If the device is not responding */
#define MMC_CORE_TIMEOUT_MS (10 * 60 * 1000) /* 10 minute timeout */
-/*
- * Background operations can take a long time, depending on the housekeeping
- * operations the card has to perform.
- */
-#define MMC_BKOPS_MAX_TIMEOUT (4 * 60 * 1000) /* max time to wait in ms */
-
/* The max erase timeout, used when host->max_busy_timeout isn't specified */
#define MMC_ERASE_TIMEOUT_MS (60 * 1000) /* 60 s */
return 0;
}
-/**
- * mmc_start_bkops - start BKOPS for supported cards
- * @card: MMC card to start BKOPS
- * @form_exception: A flag to indicate if this function was
- * called due to an exception raised by the card
- *
- * Start background operations whenever requested.
- * When the urgent BKOPS bit is set in a R1 command response
- * then background operations should be started immediately.
-*/
-void mmc_start_bkops(struct mmc_card *card, bool from_exception)
-{
- int err;
- int timeout;
- bool use_busy_signal;
-
- if (!card->ext_csd.man_bkops_en || mmc_card_doing_bkops(card))
- return;
-
- err = mmc_read_bkops_status(card);
- if (err) {
- pr_err("%s: Failed to read bkops status: %d\n",
- mmc_hostname(card->host), err);
- return;
- }
-
- if (!card->ext_csd.raw_bkops_status)
- return;
-
- if (card->ext_csd.raw_bkops_status < EXT_CSD_BKOPS_LEVEL_2 &&
- from_exception)
- return;
-
- mmc_claim_host(card->host);
- if (card->ext_csd.raw_bkops_status >= EXT_CSD_BKOPS_LEVEL_2) {
- timeout = MMC_BKOPS_MAX_TIMEOUT;
- use_busy_signal = true;
- } else {
- timeout = 0;
- use_busy_signal = false;
- }
-
- mmc_retune_hold(card->host);
-
- err = __mmc_switch(card, EXT_CSD_CMD_SET_NORMAL,
- EXT_CSD_BKOPS_START, 1, timeout, 0,
- use_busy_signal, true, false);
- if (err) {
- pr_warn("%s: Error %d starting bkops\n",
- mmc_hostname(card->host), err);
- mmc_retune_release(card->host);
- goto out;
- }
-
- /*
- * For urgent bkops status (LEVEL_2 and more)
- * bkops executed synchronously, otherwise
- * the operation is in progress
- */
- if (!use_busy_signal)
- mmc_card_set_doing_bkops(card);
- else
- mmc_retune_release(card->host);
-out:
- mmc_release_host(card->host);
-}
-EXPORT_SYMBOL(mmc_start_bkops);
-
/*
* mmc_wait_data_done() - done callback for data request
* @mrq: done data request
}
EXPORT_SYMBOL(mmc_wait_for_req);
-/**
- * mmc_interrupt_hpi - Issue for High priority Interrupt
- * @card: the MMC card associated with the HPI transfer
- *
- * Issued High Priority Interrupt, and check for card status
- * until out-of prg-state.
- */
-int mmc_interrupt_hpi(struct mmc_card *card)
-{
- int err;
- u32 status;
- unsigned long prg_wait;
-
- if (!card->ext_csd.hpi_en) {
- pr_info("%s: HPI enable bit unset\n", mmc_hostname(card->host));
- return 1;
- }
-
- mmc_claim_host(card->host);
- err = mmc_send_status(card, &status);
- if (err) {
- pr_err("%s: Get card status fail\n", mmc_hostname(card->host));
- goto out;
- }
-
- switch (R1_CURRENT_STATE(status)) {
- case R1_STATE_IDLE:
- case R1_STATE_READY:
- case R1_STATE_STBY:
- case R1_STATE_TRAN:
- /*
- * In idle and transfer states, HPI is not needed and the caller
- * can issue the next intended command immediately
- */
- goto out;
- case R1_STATE_PRG:
- break;
- default:
- /* In all other states, it's illegal to issue HPI */
- pr_debug("%s: HPI cannot be sent. Card state=%d\n",
- mmc_hostname(card->host), R1_CURRENT_STATE(status));
- err = -EINVAL;
- goto out;
- }
-
- err = mmc_send_hpi_cmd(card, &status);
- if (err)
- goto out;
-
- prg_wait = jiffies + msecs_to_jiffies(card->ext_csd.out_of_int_time);
- do {
- err = mmc_send_status(card, &status);
-
- if (!err && R1_CURRENT_STATE(status) == R1_STATE_TRAN)
- break;
- if (time_after(jiffies, prg_wait))
- err = -ETIMEDOUT;
- } while (!err);
-
-out:
- mmc_release_host(card->host);
- return err;
-}
-EXPORT_SYMBOL(mmc_interrupt_hpi);
-
/**
* mmc_wait_for_cmd - start a command and wait for completion
* @host: MMC host to start command
EXPORT_SYMBOL(mmc_wait_for_cmd);
-/**
- * mmc_stop_bkops - stop ongoing BKOPS
- * @card: MMC card to check BKOPS
- *
- * Send HPI command to stop ongoing background operations to
- * allow rapid servicing of foreground operations, e.g. read/
- * writes. Wait until the card comes out of the programming state
- * to avoid errors in servicing read/write requests.
- */
-int mmc_stop_bkops(struct mmc_card *card)
-{
- int err = 0;
-
- err = mmc_interrupt_hpi(card);
-
- /*
- * If err is EINVAL, we can't issue an HPI.
- * It should complete the BKOPS.
- */
- if (!err || (err == -EINVAL)) {
- mmc_card_clr_doing_bkops(card);
- mmc_retune_release(card->host);
- err = 0;
- }
-
- return err;
-}
-EXPORT_SYMBOL(mmc_stop_bkops);
-
-int mmc_read_bkops_status(struct mmc_card *card)
-{
- int err;
- u8 *ext_csd;
-
- mmc_claim_host(card->host);
- err = mmc_get_ext_csd(card, &ext_csd);
- mmc_release_host(card->host);
- if (err)
- return err;
-
- card->ext_csd.raw_bkops_status = ext_csd[EXT_CSD_BKOPS_STATUS];
- card->ext_csd.raw_exception_status = ext_csd[EXT_CSD_EXP_EVENTS_STATUS];
- kfree(ext_csd);
- return 0;
-}
-EXPORT_SYMBOL(mmc_read_bkops_status);
-
/**
* mmc_set_data_timeout - set the timeout for a data command
* @data: data phase for command
static void mmc_hw_reset_for_init(struct mmc_host *host)
{
+ mmc_pwrseq_reset(host);
+
if (!(host->caps & MMC_CAP_HW_RESET) || !host->ops->hw_reset)
return;
host->ops->hw_reset(host);
host->removed = 1;
spin_unlock_irqrestore(&host->lock, flags);
#endif
- if (host->slot.cd_irq >= 0)
+ if (host->slot.cd_irq >= 0) {
+ if (host->slot.cd_wake_enabled)
+ disable_irq_wake(host->slot.cd_irq);
disable_irq(host->slot.cd_irq);
+ }
host->rescan_disable = 1;
cancel_delayed_work_sync(&host->detect);
}
EXPORT_SYMBOL(mmc_power_restore_host);
-/*
- * Flush the cache to the non-volatile storage.
- */
-int mmc_flush_cache(struct mmc_card *card)
-{
- int err = 0;
-
- if (mmc_card_mmc(card) &&
- (card->ext_csd.cache_size > 0) &&
- (card->ext_csd.cache_ctrl & 1)) {
- err = mmc_switch(card, EXT_CSD_CMD_SET_NORMAL,
- EXT_CSD_FLUSH_CACHE, 1, 0);
- if (err)
- pr_err("%s: cache flush error %d\n",
- mmc_hostname(card->host), err);
- }
-
- return err;
-}
-EXPORT_SYMBOL(mmc_flush_cache);
-
#ifdef CONFIG_PM_SLEEP
/* Do the card removal on suspend if card is assumed removeable
* Do that in pm notifier while userspace isn't yet frozen, so we will be able
#include "host.h"
#include "slot-gpio.h"
#include "pwrseq.h"
+#include "sdio_ops.h"
#define cls_dev_to_mmc_host(d) container_of(d, struct mmc_host, class_dev)
*/
int mmc_of_parse(struct mmc_host *host)
{
- struct device_node *np;
+ struct device *dev = host->parent;
u32 bus_width;
int ret;
bool cd_cap_invert, cd_gpio_invert = false;
bool ro_cap_invert, ro_gpio_invert = false;
- if (!host->parent || !host->parent->of_node)
+ if (!dev || !dev_fwnode(dev))
return 0;
- np = host->parent->of_node;
-
/* "bus-width" is translated to MMC_CAP_*_BIT_DATA flags */
- if (of_property_read_u32(np, "bus-width", &bus_width) < 0) {
+ if (device_property_read_u32(dev, "bus-width", &bus_width) < 0) {
dev_dbg(host->parent,
"\"bus-width\" property is missing, assuming 1 bit.\n");
bus_width = 1;
}
/* f_max is obtained from the optional "max-frequency" property */
- of_property_read_u32(np, "max-frequency", &host->f_max);
+ device_property_read_u32(dev, "max-frequency", &host->f_max);
/*
* Configure CD and WP pins. They are both by default active low to
*/
/* Parse Card Detection */
- if (of_property_read_bool(np, "non-removable")) {
+ if (device_property_read_bool(dev, "non-removable")) {
host->caps |= MMC_CAP_NONREMOVABLE;
} else {
- cd_cap_invert = of_property_read_bool(np, "cd-inverted");
+ cd_cap_invert = device_property_read_bool(dev, "cd-inverted");
- if (of_property_read_bool(np, "broken-cd"))
+ if (device_property_read_bool(dev, "broken-cd"))
host->caps |= MMC_CAP_NEEDS_POLL;
ret = mmc_gpiod_request_cd(host, "cd", 0, true,
}
/* Parse Write Protection */
- ro_cap_invert = of_property_read_bool(np, "wp-inverted");
+ ro_cap_invert = device_property_read_bool(dev, "wp-inverted");
ret = mmc_gpiod_request_ro(host, "wp", 0, false, 0, &ro_gpio_invert);
if (!ret)
else if (ret != -ENOENT && ret != -ENOSYS)
return ret;
- if (of_property_read_bool(np, "disable-wp"))
+ if (device_property_read_bool(dev, "disable-wp"))
host->caps2 |= MMC_CAP2_NO_WRITE_PROTECT;
/* See the comment on CD inversion above */
if (ro_cap_invert ^ ro_gpio_invert)
host->caps2 |= MMC_CAP2_RO_ACTIVE_HIGH;
- if (of_property_read_bool(np, "cap-sd-highspeed"))
+ if (device_property_read_bool(dev, "cap-sd-highspeed"))
host->caps |= MMC_CAP_SD_HIGHSPEED;
- if (of_property_read_bool(np, "cap-mmc-highspeed"))
+ if (device_property_read_bool(dev, "cap-mmc-highspeed"))
host->caps |= MMC_CAP_MMC_HIGHSPEED;
- if (of_property_read_bool(np, "sd-uhs-sdr12"))
+ if (device_property_read_bool(dev, "sd-uhs-sdr12"))
host->caps |= MMC_CAP_UHS_SDR12;
- if (of_property_read_bool(np, "sd-uhs-sdr25"))
+ if (device_property_read_bool(dev, "sd-uhs-sdr25"))
host->caps |= MMC_CAP_UHS_SDR25;
- if (of_property_read_bool(np, "sd-uhs-sdr50"))
+ if (device_property_read_bool(dev, "sd-uhs-sdr50"))
host->caps |= MMC_CAP_UHS_SDR50;
- if (of_property_read_bool(np, "sd-uhs-sdr104"))
+ if (device_property_read_bool(dev, "sd-uhs-sdr104"))
host->caps |= MMC_CAP_UHS_SDR104;
- if (of_property_read_bool(np, "sd-uhs-ddr50"))
+ if (device_property_read_bool(dev, "sd-uhs-ddr50"))
host->caps |= MMC_CAP_UHS_DDR50;
- if (of_property_read_bool(np, "cap-power-off-card"))
+ if (device_property_read_bool(dev, "cap-power-off-card"))
host->caps |= MMC_CAP_POWER_OFF_CARD;
- if (of_property_read_bool(np, "cap-mmc-hw-reset"))
+ if (device_property_read_bool(dev, "cap-mmc-hw-reset"))
host->caps |= MMC_CAP_HW_RESET;
- if (of_property_read_bool(np, "cap-sdio-irq"))
+ if (device_property_read_bool(dev, "cap-sdio-irq"))
host->caps |= MMC_CAP_SDIO_IRQ;
- if (of_property_read_bool(np, "full-pwr-cycle"))
+ if (device_property_read_bool(dev, "full-pwr-cycle"))
host->caps2 |= MMC_CAP2_FULL_PWR_CYCLE;
- if (of_property_read_bool(np, "keep-power-in-suspend"))
+ if (device_property_read_bool(dev, "keep-power-in-suspend"))
host->pm_caps |= MMC_PM_KEEP_POWER;
- if (of_property_read_bool(np, "wakeup-source") ||
- of_property_read_bool(np, "enable-sdio-wakeup")) /* legacy */
+ if (device_property_read_bool(dev, "wakeup-source") ||
+ device_property_read_bool(dev, "enable-sdio-wakeup")) /* legacy */
host->pm_caps |= MMC_PM_WAKE_SDIO_IRQ;
- if (of_property_read_bool(np, "mmc-ddr-3_3v"))
+ if (device_property_read_bool(dev, "mmc-ddr-3_3v"))
host->caps |= MMC_CAP_3_3V_DDR;
- if (of_property_read_bool(np, "mmc-ddr-1_8v"))
+ if (device_property_read_bool(dev, "mmc-ddr-1_8v"))
host->caps |= MMC_CAP_1_8V_DDR;
- if (of_property_read_bool(np, "mmc-ddr-1_2v"))
+ if (device_property_read_bool(dev, "mmc-ddr-1_2v"))
host->caps |= MMC_CAP_1_2V_DDR;
- if (of_property_read_bool(np, "mmc-hs200-1_8v"))
+ if (device_property_read_bool(dev, "mmc-hs200-1_8v"))
host->caps2 |= MMC_CAP2_HS200_1_8V_SDR;
- if (of_property_read_bool(np, "mmc-hs200-1_2v"))
+ if (device_property_read_bool(dev, "mmc-hs200-1_2v"))
host->caps2 |= MMC_CAP2_HS200_1_2V_SDR;
- if (of_property_read_bool(np, "mmc-hs400-1_8v"))
+ if (device_property_read_bool(dev, "mmc-hs400-1_8v"))
host->caps2 |= MMC_CAP2_HS400_1_8V | MMC_CAP2_HS200_1_8V_SDR;
- if (of_property_read_bool(np, "mmc-hs400-1_2v"))
+ if (device_property_read_bool(dev, "mmc-hs400-1_2v"))
host->caps2 |= MMC_CAP2_HS400_1_2V | MMC_CAP2_HS200_1_2V_SDR;
- if (of_property_read_bool(np, "mmc-hs400-enhanced-strobe"))
+ if (device_property_read_bool(dev, "mmc-hs400-enhanced-strobe"))
host->caps2 |= MMC_CAP2_HS400_ES;
- if (of_property_read_bool(np, "no-sdio"))
+ if (device_property_read_bool(dev, "no-sdio"))
host->caps2 |= MMC_CAP2_NO_SDIO;
- if (of_property_read_bool(np, "no-sd"))
+ if (device_property_read_bool(dev, "no-sd"))
host->caps2 |= MMC_CAP2_NO_SD;
- if (of_property_read_bool(np, "no-mmc"))
+ if (device_property_read_bool(dev, "no-mmc"))
host->caps2 |= MMC_CAP2_NO_MMC;
- host->dsr_req = !of_property_read_u32(np, "dsr", &host->dsr);
+ host->dsr_req = !device_property_read_u32(dev, "dsr", &host->dsr);
if (host->dsr_req && (host->dsr & ~0xffff)) {
dev_err(host->parent,
"device tree specified broken value for DSR: 0x%x, ignoring\n",
spin_lock_init(&host->lock);
init_waitqueue_head(&host->wq);
INIT_DELAYED_WORK(&host->detect, mmc_rescan);
+ INIT_DELAYED_WORK(&host->sdio_irq_work, sdio_irq_work);
setup_timer(&host->retune_timer, mmc_retune_timer, (unsigned long)host);
/*
#include "mmc_ops.h"
#include "quirks.h"
#include "sd_ops.h"
+#include "pwrseq.h"
#define DEFAULT_CMD6_TIMEOUT_MS 500
/*
* Fetch CID from card.
*/
- if (mmc_host_is_spi(host))
- err = mmc_send_cid(host, cid);
- else
- err = mmc_all_send_cid(host, cid);
+ err = mmc_send_cid(host, cid);
if (err)
goto err;
mmc_set_erase_size(card);
}
- /*
- * If enhanced_area_en is TRUE, host needs to enable ERASE_GRP_DEF
- * bit. This bit will be lost every time after a reset or power off.
- */
- if (card->ext_csd.partition_setting_completed ||
- (card->ext_csd.rev >= 3 && (host->caps2 & MMC_CAP2_HC_ERASE_SZ))) {
+ /* Enable ERASE_GRP_DEF. This bit is lost after a reset or power off. */
+ if (card->ext_csd.rev >= 3) {
err = mmc_switch(card, EXT_CSD_CMD_SET_NORMAL,
EXT_CSD_ERASE_GROUP_DEF, 1,
card->ext_csd.generic_cmd6_time);
return 0;
}
-int mmc_can_reset(struct mmc_card *card)
+static int mmc_can_reset(struct mmc_card *card)
{
u8 rst_n_function;
return 0;
return 1;
}
-EXPORT_SYMBOL(mmc_can_reset);
static int mmc_reset(struct mmc_host *host)
{
} else {
/* Do a brute force power cycle */
mmc_power_cycle(host, card->ocr);
+ mmc_pwrseq_reset(host);
}
return mmc_init_card(host, card->ocr, card);
}
#include <linux/mmc/mmc.h>
#include "core.h"
+#include "card.h"
#include "host.h"
#include "mmc_ops.h"
0xff, 0x77, 0x77, 0xff, 0x77, 0xbb, 0xdd, 0xee,
};
-int mmc_send_status(struct mmc_card *card, u32 *status)
+int __mmc_send_status(struct mmc_card *card, u32 *status, unsigned int retries)
{
int err;
struct mmc_command cmd = {};
cmd.arg = card->rca << 16;
cmd.flags = MMC_RSP_SPI_R2 | MMC_RSP_R1 | MMC_CMD_AC;
- err = mmc_wait_for_cmd(card->host, &cmd, MMC_CMD_RETRIES);
+ err = mmc_wait_for_cmd(card->host, &cmd, retries);
if (err)
return err;
return 0;
}
+EXPORT_SYMBOL_GPL(__mmc_send_status);
+
+int mmc_send_status(struct mmc_card *card, u32 *status)
+{
+ return __mmc_send_status(card, status, MMC_CMD_RETRIES);
+}
static int _mmc_select_card(struct mmc_host *host, struct mmc_card *card)
{
return err;
}
-int mmc_all_send_cid(struct mmc_host *host, u32 *cid)
-{
- int err;
- struct mmc_command cmd = {};
-
- cmd.opcode = MMC_ALL_SEND_CID;
- cmd.arg = 0;
- cmd.flags = MMC_RSP_R2 | MMC_CMD_BCR;
-
- err = mmc_wait_for_cmd(host, &cmd, MMC_CMD_RETRIES);
- if (err)
- return err;
-
- memcpy(cid, cmd.resp, sizeof(u32) * 4);
-
- return 0;
-}
-
int mmc_set_relative_addr(struct mmc_card *card)
{
struct mmc_command cmd = {};
return 0;
}
-int mmc_send_csd(struct mmc_card *card, u32 *csd)
+static int mmc_spi_send_csd(struct mmc_card *card, u32 *csd)
{
int ret, i;
__be32 *csd_tmp;
- if (!mmc_host_is_spi(card->host))
- return mmc_send_cxd_native(card->host, card->rca << 16,
- csd, MMC_SEND_CSD);
-
csd_tmp = kzalloc(16, GFP_KERNEL);
if (!csd_tmp)
return -ENOMEM;
return ret;
}
-int mmc_send_cid(struct mmc_host *host, u32 *cid)
+int mmc_send_csd(struct mmc_card *card, u32 *csd)
+{
+ if (mmc_host_is_spi(card->host))
+ return mmc_spi_send_csd(card, csd);
+
+ return mmc_send_cxd_native(card->host, card->rca << 16, csd,
+ MMC_SEND_CSD);
+}
+
+static int mmc_spi_send_cid(struct mmc_host *host, u32 *cid)
{
int ret, i;
__be32 *cid_tmp;
- if (!mmc_host_is_spi(host)) {
- if (!host->card)
- return -EINVAL;
- return mmc_send_cxd_native(host, host->card->rca << 16,
- cid, MMC_SEND_CID);
- }
-
cid_tmp = kzalloc(16, GFP_KERNEL);
if (!cid_tmp)
return -ENOMEM;
return ret;
}
+int mmc_send_cid(struct mmc_host *host, u32 *cid)
+{
+ if (mmc_host_is_spi(host))
+ return mmc_spi_send_cid(host, cid);
+
+ return mmc_send_cxd_native(host, 0, cid, MMC_ALL_SEND_CID);
+}
+
int mmc_get_ext_csd(struct mmc_card *card, u8 **new_ext_csd)
{
int err;
return mmc_send_bus_test(card, card->host, MMC_BUS_TEST_R, width);
}
-int mmc_send_hpi_cmd(struct mmc_card *card, u32 *status)
+static int mmc_send_hpi_cmd(struct mmc_card *card, u32 *status)
{
struct mmc_command cmd = {};
unsigned int opcode;
return 0;
}
+/**
+ * mmc_interrupt_hpi - Issue for High priority Interrupt
+ * @card: the MMC card associated with the HPI transfer
+ *
+ * Issued High Priority Interrupt, and check for card status
+ * until out-of prg-state.
+ */
+int mmc_interrupt_hpi(struct mmc_card *card)
+{
+ int err;
+ u32 status;
+ unsigned long prg_wait;
+
+ if (!card->ext_csd.hpi_en) {
+ pr_info("%s: HPI enable bit unset\n", mmc_hostname(card->host));
+ return 1;
+ }
+
+ mmc_claim_host(card->host);
+ err = mmc_send_status(card, &status);
+ if (err) {
+ pr_err("%s: Get card status fail\n", mmc_hostname(card->host));
+ goto out;
+ }
+
+ switch (R1_CURRENT_STATE(status)) {
+ case R1_STATE_IDLE:
+ case R1_STATE_READY:
+ case R1_STATE_STBY:
+ case R1_STATE_TRAN:
+ /*
+ * In idle and transfer states, HPI is not needed and the caller
+ * can issue the next intended command immediately
+ */
+ goto out;
+ case R1_STATE_PRG:
+ break;
+ default:
+ /* In all other states, it's illegal to issue HPI */
+ pr_debug("%s: HPI cannot be sent. Card state=%d\n",
+ mmc_hostname(card->host), R1_CURRENT_STATE(status));
+ err = -EINVAL;
+ goto out;
+ }
+
+ err = mmc_send_hpi_cmd(card, &status);
+ if (err)
+ goto out;
+
+ prg_wait = jiffies + msecs_to_jiffies(card->ext_csd.out_of_int_time);
+ do {
+ err = mmc_send_status(card, &status);
+
+ if (!err && R1_CURRENT_STATE(status) == R1_STATE_TRAN)
+ break;
+ if (time_after(jiffies, prg_wait))
+ err = -ETIMEDOUT;
+ } while (!err);
+
+out:
+ mmc_release_host(card->host);
+ return err;
+}
+
int mmc_can_ext_csd(struct mmc_card *card)
{
return (card && card->csd.mmca_vsn > CSD_SPEC_VER_3);
}
+/**
+ * mmc_stop_bkops - stop ongoing BKOPS
+ * @card: MMC card to check BKOPS
+ *
+ * Send HPI command to stop ongoing background operations to
+ * allow rapid servicing of foreground operations, e.g. read/
+ * writes. Wait until the card comes out of the programming state
+ * to avoid errors in servicing read/write requests.
+ */
+int mmc_stop_bkops(struct mmc_card *card)
+{
+ int err = 0;
+
+ err = mmc_interrupt_hpi(card);
+
+ /*
+ * If err is EINVAL, we can't issue an HPI.
+ * It should complete the BKOPS.
+ */
+ if (!err || (err == -EINVAL)) {
+ mmc_card_clr_doing_bkops(card);
+ mmc_retune_release(card->host);
+ err = 0;
+ }
+
+ return err;
+}
+
+static int mmc_read_bkops_status(struct mmc_card *card)
+{
+ int err;
+ u8 *ext_csd;
+
+ mmc_claim_host(card->host);
+ err = mmc_get_ext_csd(card, &ext_csd);
+ mmc_release_host(card->host);
+ if (err)
+ return err;
+
+ card->ext_csd.raw_bkops_status = ext_csd[EXT_CSD_BKOPS_STATUS];
+ card->ext_csd.raw_exception_status = ext_csd[EXT_CSD_EXP_EVENTS_STATUS];
+ kfree(ext_csd);
+ return 0;
+}
+
+/**
+ * mmc_start_bkops - start BKOPS for supported cards
+ * @card: MMC card to start BKOPS
+ * @form_exception: A flag to indicate if this function was
+ * called due to an exception raised by the card
+ *
+ * Start background operations whenever requested.
+ * When the urgent BKOPS bit is set in a R1 command response
+ * then background operations should be started immediately.
+*/
+void mmc_start_bkops(struct mmc_card *card, bool from_exception)
+{
+ int err;
+ int timeout;
+ bool use_busy_signal;
+
+ if (!card->ext_csd.man_bkops_en || mmc_card_doing_bkops(card))
+ return;
+
+ err = mmc_read_bkops_status(card);
+ if (err) {
+ pr_err("%s: Failed to read bkops status: %d\n",
+ mmc_hostname(card->host), err);
+ return;
+ }
+
+ if (!card->ext_csd.raw_bkops_status)
+ return;
+
+ if (card->ext_csd.raw_bkops_status < EXT_CSD_BKOPS_LEVEL_2 &&
+ from_exception)
+ return;
+
+ mmc_claim_host(card->host);
+ if (card->ext_csd.raw_bkops_status >= EXT_CSD_BKOPS_LEVEL_2) {
+ timeout = MMC_OPS_TIMEOUT_MS;
+ use_busy_signal = true;
+ } else {
+ timeout = 0;
+ use_busy_signal = false;
+ }
+
+ mmc_retune_hold(card->host);
+
+ err = __mmc_switch(card, EXT_CSD_CMD_SET_NORMAL,
+ EXT_CSD_BKOPS_START, 1, timeout, 0,
+ use_busy_signal, true, false);
+ if (err) {
+ pr_warn("%s: Error %d starting bkops\n",
+ mmc_hostname(card->host), err);
+ mmc_retune_release(card->host);
+ goto out;
+ }
+
+ /*
+ * For urgent bkops status (LEVEL_2 and more)
+ * bkops executed synchronously, otherwise
+ * the operation is in progress
+ */
+ if (!use_busy_signal)
+ mmc_card_set_doing_bkops(card);
+ else
+ mmc_retune_release(card->host);
+out:
+ mmc_release_host(card->host);
+}
+
+/*
+ * Flush the cache to the non-volatile storage.
+ */
+int mmc_flush_cache(struct mmc_card *card)
+{
+ int err = 0;
+
+ if (mmc_card_mmc(card) &&
+ (card->ext_csd.cache_size > 0) &&
+ (card->ext_csd.cache_ctrl & 1)) {
+ err = mmc_switch(card, EXT_CSD_CMD_SET_NORMAL,
+ EXT_CSD_FLUSH_CACHE, 1, 0);
+ if (err)
+ pr_err("%s: cache flush error %d\n",
+ mmc_hostname(card->host), err);
+ }
+
+ return err;
+}
+EXPORT_SYMBOL(mmc_flush_cache);
+
static int mmc_cmdq_switch(struct mmc_card *card, bool enable)
{
u8 val = enable ? EXT_CSD_CMDQ_MODE_ENABLED : 0;
int mmc_set_dsr(struct mmc_host *host);
int mmc_go_idle(struct mmc_host *host);
int mmc_send_op_cond(struct mmc_host *host, u32 ocr, u32 *rocr);
-int mmc_all_send_cid(struct mmc_host *host, u32 *cid);
int mmc_set_relative_addr(struct mmc_card *card);
int mmc_send_csd(struct mmc_card *card, u32 *csd);
+int __mmc_send_status(struct mmc_card *card, u32 *status, unsigned int retries);
int mmc_send_status(struct mmc_card *card, u32 *status);
int mmc_send_cid(struct mmc_host *host, u32 *cid);
int mmc_spi_read_ocr(struct mmc_host *host, int highcap, u32 *ocrp);
int mmc_spi_set_crc(struct mmc_host *host, int use_crc);
int mmc_bus_test(struct mmc_card *card, u8 bus_width);
-int mmc_send_hpi_cmd(struct mmc_card *card, u32 *status);
int mmc_interrupt_hpi(struct mmc_card *card);
int mmc_can_ext_csd(struct mmc_card *card);
int mmc_get_ext_csd(struct mmc_card *card, u8 **new_ext_csd);
int mmc_switch(struct mmc_card *card, u8 set, u8 index, u8 value,
unsigned int timeout_ms);
int mmc_stop_bkops(struct mmc_card *card);
-int mmc_read_bkops_status(struct mmc_card *card);
void mmc_start_bkops(struct mmc_card *card, bool from_exception);
-int mmc_can_reset(struct mmc_card *card);
int mmc_flush_cache(struct mmc_card *card);
int mmc_cmdq_enable(struct mmc_card *card);
int mmc_cmdq_disable(struct mmc_card *card);
df = kmalloc(sizeof(*df), GFP_KERNEL);
if (!df) {
debugfs_remove(file);
- dev_err(&card->dev,
- "Can't allocate memory for internal usage.\n");
return -ENOMEM;
}
pwrseq->ops->power_off(host);
}
+void mmc_pwrseq_reset(struct mmc_host *host)
+{
+ struct mmc_pwrseq *pwrseq = host->pwrseq;
+
+ if (pwrseq && pwrseq->ops->reset)
+ pwrseq->ops->reset(host);
+}
+
void mmc_pwrseq_free(struct mmc_host *host)
{
struct mmc_pwrseq *pwrseq = host->pwrseq;
void (*pre_power_on)(struct mmc_host *host);
void (*post_power_on)(struct mmc_host *host);
void (*power_off)(struct mmc_host *host);
+ void (*reset)(struct mmc_host *host);
};
struct mmc_pwrseq {
void mmc_pwrseq_pre_power_on(struct mmc_host *host);
void mmc_pwrseq_post_power_on(struct mmc_host *host);
void mmc_pwrseq_power_off(struct mmc_host *host);
+void mmc_pwrseq_reset(struct mmc_host *host);
void mmc_pwrseq_free(struct mmc_host *host);
#else
static inline void mmc_pwrseq_pre_power_on(struct mmc_host *host) {}
static inline void mmc_pwrseq_post_power_on(struct mmc_host *host) {}
static inline void mmc_pwrseq_power_off(struct mmc_host *host) {}
+static inline void mmc_pwrseq_reset(struct mmc_host *host) {}
static inline void mmc_pwrseq_free(struct mmc_host *host) {}
#endif
}
static const struct mmc_pwrseq_ops mmc_pwrseq_emmc_ops = {
- .post_power_on = mmc_pwrseq_emmc_reset,
+ .reset = mmc_pwrseq_emmc_reset,
};
static int mmc_pwrseq_emmc_probe(struct platform_device *pdev)
return BLKPREP_OK;
}
-struct mmc_queue_req *mmc_queue_req_find(struct mmc_queue *mq,
- struct request *req)
-{
- struct mmc_queue_req *mqrq;
- int i = ffz(mq->qslots);
-
- if (i >= mq->qdepth)
- return NULL;
-
- mqrq = &mq->mqrq[i];
- WARN_ON(mqrq->req || mq->qcnt >= mq->qdepth ||
- test_bit(mqrq->task_id, &mq->qslots));
- mqrq->req = req;
- mq->qcnt += 1;
- __set_bit(mqrq->task_id, &mq->qslots);
-
- return mqrq;
-}
-
-void mmc_queue_req_free(struct mmc_queue *mq,
- struct mmc_queue_req *mqrq)
-{
- WARN_ON(!mqrq->req || mq->qcnt < 1 ||
- !test_bit(mqrq->task_id, &mq->qslots));
- mqrq->req = NULL;
- mq->qcnt -= 1;
- __clear_bit(mqrq->task_id, &mq->qslots);
-}
-
static int mmc_queue_thread(void *d)
{
struct mmc_queue *mq = d;
if (!mq) {
while ((req = blk_fetch_request(q)) != NULL) {
req->rq_flags |= RQF_QUIET;
- __blk_end_request_all(req, -EIO);
+ __blk_end_request_all(req, BLK_STS_IOERR);
}
return;
}
wake_up_process(mq->thread);
}
-static struct scatterlist *mmc_alloc_sg(int sg_len)
+static struct scatterlist *mmc_alloc_sg(int sg_len, gfp_t gfp)
{
struct scatterlist *sg;
- sg = kmalloc_array(sg_len, sizeof(*sg), GFP_KERNEL);
+ sg = kmalloc_array(sg_len, sizeof(*sg), gfp);
if (sg)
sg_init_table(sg, sg_len);
queue_flag_set_unlocked(QUEUE_FLAG_SECERASE, q);
}
-static void mmc_queue_req_free_bufs(struct mmc_queue_req *mqrq)
-{
- kfree(mqrq->bounce_sg);
- mqrq->bounce_sg = NULL;
-
- kfree(mqrq->sg);
- mqrq->sg = NULL;
-
- kfree(mqrq->bounce_buf);
- mqrq->bounce_buf = NULL;
-}
-
-static void mmc_queue_reqs_free_bufs(struct mmc_queue_req *mqrq, int qdepth)
-{
- int i;
-
- for (i = 0; i < qdepth; i++)
- mmc_queue_req_free_bufs(&mqrq[i]);
-}
-
-static void mmc_queue_free_mqrqs(struct mmc_queue_req *mqrq, int qdepth)
-{
- mmc_queue_reqs_free_bufs(mqrq, qdepth);
- kfree(mqrq);
-}
-
-static struct mmc_queue_req *mmc_queue_alloc_mqrqs(int qdepth)
-{
- struct mmc_queue_req *mqrq;
- int i;
-
- mqrq = kcalloc(qdepth, sizeof(*mqrq), GFP_KERNEL);
- if (mqrq) {
- for (i = 0; i < qdepth; i++)
- mqrq[i].task_id = i;
- }
-
- return mqrq;
-}
-
-#ifdef CONFIG_MMC_BLOCK_BOUNCE
-static int mmc_queue_alloc_bounce_bufs(struct mmc_queue_req *mqrq, int qdepth,
- unsigned int bouncesz)
-{
- int i;
-
- for (i = 0; i < qdepth; i++) {
- mqrq[i].bounce_buf = kmalloc(bouncesz, GFP_KERNEL);
- if (!mqrq[i].bounce_buf)
- return -ENOMEM;
-
- mqrq[i].sg = mmc_alloc_sg(1);
- if (!mqrq[i].sg)
- return -ENOMEM;
-
- mqrq[i].bounce_sg = mmc_alloc_sg(bouncesz / 512);
- if (!mqrq[i].bounce_sg)
- return -ENOMEM;
- }
-
- return 0;
-}
-
-static bool mmc_queue_alloc_bounce(struct mmc_queue_req *mqrq, int qdepth,
- unsigned int bouncesz)
-{
- int ret;
-
- ret = mmc_queue_alloc_bounce_bufs(mqrq, qdepth, bouncesz);
- if (ret)
- mmc_queue_reqs_free_bufs(mqrq, qdepth);
-
- return !ret;
-}
-
static unsigned int mmc_queue_calc_bouncesz(struct mmc_host *host)
{
unsigned int bouncesz = MMC_QUEUE_BOUNCESZ;
- if (host->max_segs != 1)
+ if (host->max_segs != 1 || (host->caps & MMC_CAP_NO_BOUNCE_BUFF))
return 0;
if (bouncesz > host->max_req_size)
return bouncesz;
}
-#else
-static inline bool mmc_queue_alloc_bounce(struct mmc_queue_req *mqrq,
- int qdepth, unsigned int bouncesz)
-{
- return false;
-}
-static unsigned int mmc_queue_calc_bouncesz(struct mmc_host *host)
-{
- return 0;
-}
-#endif
-
-static int mmc_queue_alloc_sgs(struct mmc_queue_req *mqrq, int qdepth,
- int max_segs)
+/**
+ * mmc_init_request() - initialize the MMC-specific per-request data
+ * @q: the request queue
+ * @req: the request
+ * @gfp: memory allocation policy
+ */
+static int mmc_init_request(struct request_queue *q, struct request *req,
+ gfp_t gfp)
{
- int i;
+ struct mmc_queue_req *mq_rq = req_to_mmc_queue_req(req);
+ struct mmc_queue *mq = q->queuedata;
+ struct mmc_card *card = mq->card;
+ struct mmc_host *host = card->host;
- for (i = 0; i < qdepth; i++) {
- mqrq[i].sg = mmc_alloc_sg(max_segs);
- if (!mqrq[i].sg)
+ if (card->bouncesz) {
+ mq_rq->bounce_buf = kmalloc(card->bouncesz, gfp);
+ if (!mq_rq->bounce_buf)
+ return -ENOMEM;
+ if (card->bouncesz > 512) {
+ mq_rq->sg = mmc_alloc_sg(1, gfp);
+ if (!mq_rq->sg)
+ return -ENOMEM;
+ mq_rq->bounce_sg = mmc_alloc_sg(card->bouncesz / 512,
+ gfp);
+ if (!mq_rq->bounce_sg)
+ return -ENOMEM;
+ }
+ } else {
+ mq_rq->bounce_buf = NULL;
+ mq_rq->bounce_sg = NULL;
+ mq_rq->sg = mmc_alloc_sg(host->max_segs, gfp);
+ if (!mq_rq->sg)
return -ENOMEM;
}
return 0;
}
-void mmc_queue_free_shared_queue(struct mmc_card *card)
+static void mmc_exit_request(struct request_queue *q, struct request *req)
{
- if (card->mqrq) {
- mmc_queue_free_mqrqs(card->mqrq, card->qdepth);
- card->mqrq = NULL;
- }
-}
-
-static int __mmc_queue_alloc_shared_queue(struct mmc_card *card, int qdepth)
-{
- struct mmc_host *host = card->host;
- struct mmc_queue_req *mqrq;
- unsigned int bouncesz;
- int ret = 0;
-
- if (card->mqrq)
- return -EINVAL;
+ struct mmc_queue_req *mq_rq = req_to_mmc_queue_req(req);
- mqrq = mmc_queue_alloc_mqrqs(qdepth);
- if (!mqrq)
- return -ENOMEM;
-
- card->mqrq = mqrq;
- card->qdepth = qdepth;
-
- bouncesz = mmc_queue_calc_bouncesz(host);
-
- if (bouncesz && !mmc_queue_alloc_bounce(mqrq, qdepth, bouncesz)) {
- bouncesz = 0;
- pr_warn("%s: unable to allocate bounce buffers\n",
- mmc_card_name(card));
- }
+ /* It is OK to kfree(NULL) so this will be smooth */
+ kfree(mq_rq->bounce_sg);
+ mq_rq->bounce_sg = NULL;
- card->bouncesz = bouncesz;
+ kfree(mq_rq->bounce_buf);
+ mq_rq->bounce_buf = NULL;
- if (!bouncesz) {
- ret = mmc_queue_alloc_sgs(mqrq, qdepth, host->max_segs);
- if (ret)
- goto out_err;
- }
-
- return ret;
-
-out_err:
- mmc_queue_free_shared_queue(card);
- return ret;
-}
-
-int mmc_queue_alloc_shared_queue(struct mmc_card *card)
-{
- return __mmc_queue_alloc_shared_queue(card, 2);
+ kfree(mq_rq->sg);
+ mq_rq->sg = NULL;
}
/**
limit = (u64)dma_max_pfn(mmc_dev(host)) << PAGE_SHIFT;
mq->card = card;
- mq->queue = blk_init_queue(mmc_request_fn, lock);
+ mq->queue = blk_alloc_queue(GFP_KERNEL);
if (!mq->queue)
return -ENOMEM;
-
- mq->mqrq = card->mqrq;
- mq->qdepth = card->qdepth;
+ mq->queue->queue_lock = lock;
+ mq->queue->request_fn = mmc_request_fn;
+ mq->queue->init_rq_fn = mmc_init_request;
+ mq->queue->exit_rq_fn = mmc_exit_request;
+ mq->queue->cmd_size = sizeof(struct mmc_queue_req);
mq->queue->queuedata = mq;
+ mq->qcnt = 0;
+ ret = blk_init_allocated_queue(mq->queue);
+ if (ret) {
+ blk_cleanup_queue(mq->queue);
+ return ret;
+ }
blk_queue_prep_rq(mq->queue, mmc_prep_request);
queue_flag_set_unlocked(QUEUE_FLAG_NONROT, mq->queue);
if (mmc_can_erase(card))
mmc_queue_setup_discard(mq->queue, card);
+ card->bouncesz = mmc_queue_calc_bouncesz(host);
if (card->bouncesz) {
- blk_queue_bounce_limit(mq->queue, BLK_BOUNCE_ANY);
blk_queue_max_hw_sectors(mq->queue, card->bouncesz / 512);
blk_queue_max_segments(mq->queue, card->bouncesz / 512);
blk_queue_max_segment_size(mq->queue, card->bouncesz);
return 0;
cleanup_queue:
- mq->mqrq = NULL;
blk_cleanup_queue(mq->queue);
return ret;
}
blk_start_queue(q);
spin_unlock_irqrestore(q->queue_lock, flags);
- mq->mqrq = NULL;
mq->card = NULL;
}
EXPORT_SYMBOL(mmc_cleanup_queue);
unsigned int sg_len;
size_t buflen;
struct scatterlist *sg;
+ struct request *req = mmc_queue_req_to_req(mqrq);
int i;
if (!mqrq->bounce_buf)
- return blk_rq_map_sg(mq->queue, mqrq->req, mqrq->sg);
+ return blk_rq_map_sg(mq->queue, req, mqrq->sg);
- sg_len = blk_rq_map_sg(mq->queue, mqrq->req, mqrq->bounce_sg);
+ sg_len = blk_rq_map_sg(mq->queue, req, mqrq->bounce_sg);
mqrq->bounce_sg_len = sg_len;
if (!mqrq->bounce_buf)
return;
- if (rq_data_dir(mqrq->req) != WRITE)
+ if (rq_data_dir(mmc_queue_req_to_req(mqrq)) != WRITE)
return;
sg_copy_to_buffer(mqrq->bounce_sg, mqrq->bounce_sg_len,
if (!mqrq->bounce_buf)
return;
- if (rq_data_dir(mqrq->req) != READ)
+ if (rq_data_dir(mmc_queue_req_to_req(mqrq)) != READ)
return;
sg_copy_from_buffer(mqrq->bounce_sg, mqrq->bounce_sg_len,
#include <linux/types.h>
#include <linux/blkdev.h>
+#include <linux/blk-mq.h>
#include <linux/mmc/core.h>
#include <linux/mmc/host.h>
-static inline bool mmc_req_is_special(struct request *req)
+static inline struct mmc_queue_req *req_to_mmc_queue_req(struct request *rq)
{
- return req &&
- (req_op(req) == REQ_OP_FLUSH ||
- req_op(req) == REQ_OP_DISCARD ||
- req_op(req) == REQ_OP_SECURE_ERASE);
+ return blk_mq_rq_to_pdu(rq);
+}
+
+struct mmc_queue_req;
+
+static inline struct request *mmc_queue_req_to_req(struct mmc_queue_req *mqr)
+{
+ return blk_mq_rq_from_pdu(mqr);
}
struct task_struct;
struct mmc_blk_data;
+struct mmc_blk_ioc_data;
struct mmc_blk_request {
struct mmc_request mrq;
int retune_retry_done;
};
+/**
+ * enum mmc_drv_op - enumerates the operations in the mmc_queue_req
+ * @MMC_DRV_OP_IOCTL: ioctl operation
+ * @MMC_DRV_OP_BOOT_WP: write protect boot partitions
+ */
+enum mmc_drv_op {
+ MMC_DRV_OP_IOCTL,
+ MMC_DRV_OP_BOOT_WP,
+};
+
struct mmc_queue_req {
- struct request *req;
struct mmc_blk_request brq;
struct scatterlist *sg;
char *bounce_buf;
struct scatterlist *bounce_sg;
unsigned int bounce_sg_len;
struct mmc_async_req areq;
- int task_id;
+ enum mmc_drv_op drv_op;
+ int drv_op_result;
+ struct mmc_blk_ioc_data **idata;
+ unsigned int ioc_count;
};
struct mmc_queue {
bool asleep;
struct mmc_blk_data *blkdata;
struct request_queue *queue;
- struct mmc_queue_req *mqrq;
- int qdepth;
+ /*
+ * FIXME: this counter is not a very reliable way of keeping
+ * track of how many requests that are ongoing. Switch to just
+ * letting the block core keep track of requests and per-request
+ * associated mmc_queue_req data.
+ */
int qcnt;
- unsigned long qslots;
};
-extern int mmc_queue_alloc_shared_queue(struct mmc_card *card);
-extern void mmc_queue_free_shared_queue(struct mmc_card *card);
extern int mmc_init_queue(struct mmc_queue *, struct mmc_card *, spinlock_t *,
const char *);
extern void mmc_cleanup_queue(struct mmc_queue *);
extern int mmc_access_rpmb(struct mmc_queue *);
-extern struct mmc_queue_req *mmc_queue_req_find(struct mmc_queue *,
- struct request *);
-extern void mmc_queue_req_free(struct mmc_queue *, struct mmc_queue_req *);
-
#endif
err = -EIO;
status = kmalloc(64, GFP_KERNEL);
- if (!status) {
- pr_err("%s: could not allocate a buffer for "
- "switch capabilities.\n",
- mmc_hostname(card->host));
+ if (!status)
return -ENOMEM;
- }
/*
* Find out the card's support bits with a mode 0 operation.
return 0;
status = kmalloc(64, GFP_KERNEL);
- if (!status) {
- pr_err("%s: could not allocate a buffer for "
- "switch capabilities.\n", mmc_hostname(card->host));
+ if (!status)
return -ENOMEM;
- }
err = mmc_sd_switch(card, 1, 0, 1, status);
if (err)
return 0;
status = kmalloc(64, GFP_KERNEL);
- if (!status) {
- pr_err("%s: could not allocate a buffer for "
- "switch capabilities.\n", mmc_hostname(card->host));
+ if (!status)
return -ENOMEM;
- }
/* Set 4-bit bus width */
if ((card->host->caps & MMC_CAP_4_BIT_DATA) &&
}
}
- if (mmc_host_is_spi(host))
- err = mmc_send_cid(host, cid);
- else
- err = mmc_all_send_cid(host, cid);
-
+ err = mmc_send_cid(host, cid);
return err;
}
* Enable runtime PM only if supported by host+card+board
*/
if (host->caps & MMC_CAP_POWER_OFF_CARD) {
+ /*
+ * Do not allow runtime suspend until after SDIO function
+ * devices are added.
+ */
+ pm_runtime_get_noresume(&card->dev);
+
/*
* Let runtime PM core know our card is active
*/
goto remove_added;
}
+ if (host->caps & MMC_CAP_POWER_OFF_CARD)
+ pm_runtime_put(&card->dev);
+
mmc_claim_host(host);
return 0;
-remove_added:
- /* Remove without lock if the device has been added. */
- mmc_sdio_remove(host);
- mmc_claim_host(host);
remove:
- /* And with lock if it hasn't been added. */
mmc_release_host(host);
- if (host->card)
- mmc_sdio_remove(host);
+remove_added:
+ /*
+ * The devices are being deleted so it is not necessary to disable
+ * runtime PM. Similarly we also don't pm_runtime_put() the SDIO card
+ * because it needs to be active to remove any function devices that
+ * were probed, and after that it gets deleted.
+ */
+ mmc_sdio_remove(host);
mmc_claim_host(host);
err:
mmc_detach_bus(host);
void sdio_run_irqs(struct mmc_host *host)
{
mmc_claim_host(host);
- host->sdio_irq_pending = true;
- process_sdio_pending_irqs(host);
+ if (host->sdio_irqs) {
+ host->sdio_irq_pending = true;
+ process_sdio_pending_irqs(host);
+ if (host->ops->ack_sdio_irq)
+ host->ops->ack_sdio_irq(host);
+ }
mmc_release_host(host);
}
EXPORT_SYMBOL_GPL(sdio_run_irqs);
+void sdio_irq_work(struct work_struct *work)
+{
+ struct mmc_host *host =
+ container_of(work, struct mmc_host, sdio_irq_work.work);
+
+ sdio_run_irqs(host);
+}
+
+void sdio_signal_irq(struct mmc_host *host)
+{
+ queue_delayed_work(system_wq, &host->sdio_irq_work, 0);
+}
+EXPORT_SYMBOL_GPL(sdio_signal_irq);
+
static int sdio_irq_thread(void *_host)
{
struct mmc_host *host = _host;
struct mmc_host;
struct mmc_card;
+struct work_struct;
int mmc_send_io_op_cond(struct mmc_host *host, u32 ocr, u32 *rocr);
int mmc_io_rw_direct(struct mmc_card *card, int write, unsigned fn,
unsigned addr, int incr_addr, u8 *buf, unsigned blocks, unsigned blksz);
int sdio_reset(struct mmc_host *host);
unsigned int mmc_align_data_size(struct mmc_card *card, unsigned int sz);
+void sdio_irq_work(struct work_struct *work);
static inline bool sdio_is_io_busy(u32 opcode, u32 arg)
{
if (irq < 0)
host->caps |= MMC_CAP_NEEDS_POLL;
+ else if ((host->caps & MMC_CAP_CD_WAKE) && !enable_irq_wake(irq))
+ host->slot.cd_wake_enabled = true;
}
EXPORT_SYMBOL(mmc_gpiod_request_cd_irq);
config MMC_ATMELMCI
tristate "Atmel SD/MMC Driver (Multimedia Card Interface)"
- depends on AVR32 || ARCH_AT91
+ depends on ARCH_AT91
help
- This selects the Atmel Multimedia Card Interface driver. If
- you have an AT32 (AVR32) or AT91 platform with a Multimedia
- Card slot, say Y or M here.
+ This selects the Atmel Multimedia Card Interface driver.
+ If you have an AT91 platform with a Multimedia Card slot,
+ say Y or M here.
If unsure, say N.
T7L66XB and also HTC ASIC3
config MMC_SDHI
- tristate "SH-Mobile SDHI SD/SDIO controller support"
+ tristate "Renesas SDHI SD/SDIO controller support"
depends on SUPERH || ARM || ARM64
depends on SUPERH || ARCH_RENESAS || COMPILE_TEST
select MMC_TMIO_CORE
help
This provides support for the SDHI SD/SDIO controller found in
- SuperH and ARM SH-Mobile SoCs
+ Renesas SuperH, ARM and ARM64 based SoCs
config MMC_CB710
tristate "ENE CB710 MMC/SD Interface support"
obj-$(CONFIG_MMC_SDRICOH_CS) += sdricoh_cs.o
obj-$(CONFIG_MMC_TMIO) += tmio_mmc.o
obj-$(CONFIG_MMC_TMIO_CORE) += tmio_mmc_core.o
-tmio_mmc_core-y := tmio_mmc_pio.o
-tmio_mmc_core-$(subst m,y,$(CONFIG_MMC_SDHI)) += tmio_mmc_dma.o
-obj-$(CONFIG_MMC_SDHI) += sh_mobile_sdhi.o
+obj-$(CONFIG_MMC_SDHI) += renesas_sdhi_core.o renesas_sdhi_sys_dmac.o
obj-$(CONFIG_MMC_CB710) += cb710-mmc.o
obj-$(CONFIG_MMC_VIA_SDMMC) += via-sdmmc.o
obj-$(CONFIG_SDH_BFIN) += bfin_sdh.o
#include <asm/unaligned.h>
/*
- * Superset of MCI IP registers integrated in Atmel AVR32 and AT91 Processors
+ * Superset of MCI IP registers integrated in Atmel AT91 Processor
* Registers and bitfields marked with [2] are only available in MCI2
*/
#define atmci_writel(port, reg, value) \
__raw_writel((value), (port)->regs + reg)
-/* On AVR chips the Peripheral DMA Controller is not connected to MCI. */
-#ifdef CONFIG_AVR32
-# define ATMCI_PDC_CONNECTED 0
-#else
-# define ATMCI_PDC_CONNECTED 1
-#endif
-
#define AUTOSUSPEND_DELAY 50
#define ATMCI_DATA_ERROR_FLAGS (ATMCI_DCRCE | ATMCI_DTOE | ATMCI_OVRE | ATMCI_UNRE)
}
pdata = devm_kzalloc(&pdev->dev, sizeof(*pdata), GFP_KERNEL);
- if (!pdata) {
- dev_err(&pdev->dev, "could not allocate memory for pdata\n");
+ if (!pdata)
return ERR_PTR(-ENOMEM);
- }
for_each_child_of_node(np, cnp) {
if (of_property_read_u32(cnp, "reg", &slot_id)) {
mmc_regulator_set_ocr(mmc, mmc->supply.vmmc, ios->vdd);
break;
default:
- /*
- * TODO: None of the currently available AVR32-based
- * boards allow MMC power to be turned off. Implement
- * power control when this can be tested properly.
- *
- * We also need to hook this into the clock management
- * somehow so that newly inserted cards aren't
- * subjected to a fast clock before we have a chance
- * to figure out what the maximum rate is. Currently,
- * there's no way to avoid this, and there never will
- * be for boards that don't support power control.
- */
break;
}
-
}
static int atmci_get_ro(struct mmc_host *mmc)
"version: 0x%x\n", version);
host->caps.has_dma_conf_reg = 0;
- host->caps.has_pdc = ATMCI_PDC_CONNECTED;
+ host->caps.has_pdc = 1;
host->caps.has_cfg_reg = 0;
host->caps.has_cstor_reg = 0;
host->caps.has_highspeed = 0;
if (mrq->data && !is_power_of_2(mrq->data->blksz)) {
dev_err(dev, "unsupported block size (%d bytes)\n",
mrq->data->blksz);
- mrq->cmd->error = -EINVAL;
+
+ if (mrq->cmd)
+ mrq->cmd->error = -EINVAL;
+
mmc_request_done(mmc, mrq);
return;
}
readl(host->ioaddr + SDCMD) & SDCMD_CMD_MASK,
edm);
bcm2835_dumpregs(host);
- mrq->cmd->error = -EILSEQ;
+
+ if (mrq->cmd)
+ mrq->cmd->error = -EILSEQ;
+
bcm2835_finish_request(host);
mutex_unlock(&host->mutex);
return;
if (!host->use_busy)
bcm2835_finish_command(host);
}
- } else if (bcm2835_send_command(host, mrq->cmd)) {
+ } else if (mrq->cmd && bcm2835_send_command(host, mrq->cmd)) {
if (host->data && host->dma_desc) {
/* DMA transfer starts now, PIO starts after irq */
bcm2835_start_dma(host);
* We only have a 3.3v supply, we cannot support any
* of the UHS modes. We do support the high speed DDR
* modes up to 52MHz.
+ *
+ * Disable bounce buffers for max_segs = 1
*/
mmc->caps |= MMC_CAP_MMC_HIGHSPEED | MMC_CAP_SD_HIGHSPEED |
MMC_CAP_ERASE | MMC_CAP_CMD23 | MMC_CAP_POWER_OFF_CARD |
- MMC_CAP_3_3V_DDR;
+ MMC_CAP_3_3V_DDR | MMC_CAP_NO_BOUNCE_BUFF;
if (host->use_sg)
mmc->max_segs = 16;
* HOLD register should be bypassed in case there is no phase shift
* applied on CMD/DATA that is sent to the card.
*/
- if (!SDMMC_CLKSEL_GET_DRV_WD3(clksel) && host->cur_slot)
- set_bit(DW_MMC_CARD_NO_USE_HOLD, &host->cur_slot->flags);
+ if (!SDMMC_CLKSEL_GET_DRV_WD3(clksel) && host->slot)
+ set_bit(DW_MMC_CARD_NO_USE_HOLD, &host->slot->flags);
}
#ifdef CONFIG_PM
struct clk *drv_clk;
struct clk *sample_clk;
int default_sample_phase;
+ int num_phases;
};
static void dw_mci_rk3288_set_ios(struct dw_mci *host, struct mmc_ios *ios)
}
}
-#define NUM_PHASES 360
-#define TUNING_ITERATION_TO_PHASE(i) (DIV_ROUND_UP((i) * 360, NUM_PHASES))
+#define TUNING_ITERATION_TO_PHASE(i, num_phases) \
+ (DIV_ROUND_UP((i) * 360, num_phases))
static int dw_mci_rk3288_execute_tuning(struct dw_mci_slot *slot, u32 opcode)
{
return -EIO;
}
- ranges = kmalloc_array(NUM_PHASES / 2 + 1, sizeof(*ranges), GFP_KERNEL);
+ ranges = kmalloc_array(priv->num_phases / 2 + 1,
+ sizeof(*ranges), GFP_KERNEL);
if (!ranges)
return -ENOMEM;
/* Try each phase and extract good ranges */
- for (i = 0; i < NUM_PHASES; ) {
- clk_set_phase(priv->sample_clk, TUNING_ITERATION_TO_PHASE(i));
+ for (i = 0; i < priv->num_phases; ) {
+ clk_set_phase(priv->sample_clk,
+ TUNING_ITERATION_TO_PHASE(i, priv->num_phases));
v = !mmc_send_tuning(mmc, opcode, NULL);
if (v) {
ranges[range_count-1].end = i;
i++;
- } else if (i == NUM_PHASES - 1) {
+ } else if (i == priv->num_phases - 1) {
/* No extra skipping rules if we're at the end */
i++;
} else {
* one since testing bad phases is slow. Skip
* 20 degrees.
*/
- i += DIV_ROUND_UP(20 * NUM_PHASES, 360);
+ i += DIV_ROUND_UP(20 * priv->num_phases, 360);
/* Always test the last one */
- if (i >= NUM_PHASES)
- i = NUM_PHASES - 1;
+ if (i >= priv->num_phases)
+ i = priv->num_phases - 1;
}
prev_v = v;
range_count--;
}
- if (ranges[0].start == 0 && ranges[0].end == NUM_PHASES - 1) {
+ if (ranges[0].start == 0 && ranges[0].end == priv->num_phases - 1) {
clk_set_phase(priv->sample_clk, priv->default_sample_phase);
dev_info(host->dev, "All phases work, using default phase %d.",
priv->default_sample_phase);
int len = (ranges[i].end - ranges[i].start + 1);
if (len < 0)
- len += NUM_PHASES;
+ len += priv->num_phases;
if (longest_range_len < len) {
longest_range_len = len;
}
dev_dbg(host->dev, "Good phase range %d-%d (%d len)\n",
- TUNING_ITERATION_TO_PHASE(ranges[i].start),
- TUNING_ITERATION_TO_PHASE(ranges[i].end),
+ TUNING_ITERATION_TO_PHASE(ranges[i].start,
+ priv->num_phases),
+ TUNING_ITERATION_TO_PHASE(ranges[i].end,
+ priv->num_phases),
len
);
}
dev_dbg(host->dev, "Best phase range %d-%d (%d len)\n",
- TUNING_ITERATION_TO_PHASE(ranges[longest_range].start),
- TUNING_ITERATION_TO_PHASE(ranges[longest_range].end),
+ TUNING_ITERATION_TO_PHASE(ranges[longest_range].start,
+ priv->num_phases),
+ TUNING_ITERATION_TO_PHASE(ranges[longest_range].end,
+ priv->num_phases),
longest_range_len
);
middle_phase = ranges[longest_range].start + longest_range_len / 2;
- middle_phase %= NUM_PHASES;
+ middle_phase %= priv->num_phases;
dev_info(host->dev, "Successfully tuned phase to %d\n",
- TUNING_ITERATION_TO_PHASE(middle_phase));
+ TUNING_ITERATION_TO_PHASE(middle_phase, priv->num_phases));
clk_set_phase(priv->sample_clk,
- TUNING_ITERATION_TO_PHASE(middle_phase));
+ TUNING_ITERATION_TO_PHASE(middle_phase,
+ priv->num_phases));
free:
kfree(ranges);
if (!priv)
return -ENOMEM;
+ if (of_property_read_u32(np, "rockchip,desired-num-phases",
+ &priv->num_phases))
+ priv->num_phases = 360;
+
if (of_property_read_u32(np, "rockchip,default-sample-phase",
&priv->default_sample_phase))
priv->default_sample_phase = 0;
cmdr = stop->opcode | SDMMC_CMD_STOP |
SDMMC_CMD_RESP_CRC | SDMMC_CMD_RESP_EXP;
- if (!test_bit(DW_MMC_CARD_NO_USE_HOLD, &host->cur_slot->flags))
+ if (!test_bit(DW_MMC_CARD_NO_USE_HOLD, &host->slot->flags))
cmdr |= SDMMC_CMD_USE_HOLD_REG;
return cmdr;
if ((host->use_dma == TRANS_MODE_EDMAC) &&
data && (data->flags & MMC_DATA_READ))
/* Invalidate cache after read */
- dma_sync_sg_for_cpu(mmc_dev(host->cur_slot->mmc),
+ dma_sync_sg_for_cpu(mmc_dev(host->slot->mmc),
data->sg,
data->sg_len,
DMA_FROM_DEVICE);
/* Flush cache before write */
if (host->data->flags & MMC_DATA_WRITE)
- dma_sync_sg_for_device(mmc_dev(host->cur_slot->mmc), sgl,
+ dma_sync_sg_for_device(mmc_dev(host->slot->mmc), sgl,
sg_elems, DMA_TO_DEVICE);
dma_async_issue_pending(host->dms->ch);
mrq = slot->mrq;
- host->cur_slot = slot;
host->mrq = mrq;
host->pending_events = 0;
if (card->type == MMC_TYPE_SDIO ||
card->type == MMC_TYPE_SD_COMBO) {
- if (!test_bit(DW_MMC_CARD_NO_LOW_PWR, &slot->flags)) {
- pm_runtime_get_noresume(mmc->parent);
- set_bit(DW_MMC_CARD_NO_LOW_PWR, &slot->flags);
- }
+ set_bit(DW_MMC_CARD_NO_LOW_PWR, &slot->flags);
clk_en_a = clk_en_a_old & ~clken_low_pwr;
} else {
- if (test_bit(DW_MMC_CARD_NO_LOW_PWR, &slot->flags)) {
- pm_runtime_put_noidle(mmc->parent);
- clear_bit(DW_MMC_CARD_NO_LOW_PWR, &slot->flags);
- }
+ clear_bit(DW_MMC_CARD_NO_LOW_PWR, &slot->flags);
clk_en_a = clk_en_a_old | clken_low_pwr;
}
}
}
-static void dw_mci_enable_sdio_irq(struct mmc_host *mmc, int enb)
+static void __dw_mci_enable_sdio_irq(struct dw_mci_slot *slot, int enb)
{
- struct dw_mci_slot *slot = mmc_priv(mmc);
struct dw_mci *host = slot->host;
unsigned long irqflags;
u32 int_mask;
spin_unlock_irqrestore(&host->irq_lock, irqflags);
}
+static void dw_mci_enable_sdio_irq(struct mmc_host *mmc, int enb)
+{
+ struct dw_mci_slot *slot = mmc_priv(mmc);
+ struct dw_mci *host = slot->host;
+
+ __dw_mci_enable_sdio_irq(slot, enb);
+
+ /* Avoid runtime suspending the device when SDIO IRQ is enabled */
+ if (enb)
+ pm_runtime_get_noresume(host->dev);
+ else
+ pm_runtime_put_noidle(host->dev);
+}
+
+static void dw_mci_ack_sdio_irq(struct mmc_host *mmc)
+{
+ struct dw_mci_slot *slot = mmc_priv(mmc);
+
+ __dw_mci_enable_sdio_irq(slot, 1);
+}
+
static int dw_mci_execute_tuning(struct mmc_host *mmc, u32 opcode)
{
struct dw_mci_slot *slot = mmc_priv(mmc);
ciu_out:
/* After a CTRL reset we need to have CIU set clock registers */
- mci_send_cmd(host->cur_slot, SDMMC_CMD_UPD_CLK, 0);
+ mci_send_cmd(host->slot, SDMMC_CMD_UPD_CLK, 0);
return ret;
}
.get_cd = dw_mci_get_cd,
.hw_reset = dw_mci_hw_reset,
.enable_sdio_irq = dw_mci_enable_sdio_irq,
+ .ack_sdio_irq = dw_mci_ack_sdio_irq,
.execute_tuning = dw_mci_execute_tuning,
.card_busy = dw_mci_card_busy,
.start_signal_voltage_switch = dw_mci_switch_voltage,
__acquires(&host->lock)
{
struct dw_mci_slot *slot;
- struct mmc_host *prev_mmc = host->cur_slot->mmc;
+ struct mmc_host *prev_mmc = host->slot->mmc;
WARN_ON(host->cmd || host->data);
- host->cur_slot->mrq = NULL;
+ host->slot->mrq = NULL;
host->mrq = NULL;
if (!list_empty(&host->queue)) {
slot = list_entry(host->queue.next,
err = dw_mci_command_complete(host, cmd);
if (cmd == mrq->sbc && !err) {
prev_state = state = STATE_SENDING_CMD;
- __dw_mci_start_request(host, host->cur_slot,
+ __dw_mci_start_request(host, host->slot,
mrq->cmd);
goto unlock;
}
static void dw_mci_handle_cd(struct dw_mci *host)
{
- int i;
+ struct dw_mci_slot *slot = host->slot;
- for (i = 0; i < host->num_slots; i++) {
- struct dw_mci_slot *slot = host->slot[i];
-
- if (!slot)
- continue;
-
- if (slot->mmc->ops->card_event)
- slot->mmc->ops->card_event(slot->mmc);
- mmc_detect_change(slot->mmc,
- msecs_to_jiffies(host->pdata->detect_delay_ms));
- }
+ if (slot->mmc->ops->card_event)
+ slot->mmc->ops->card_event(slot->mmc);
+ mmc_detect_change(slot->mmc,
+ msecs_to_jiffies(host->pdata->detect_delay_ms));
}
static irqreturn_t dw_mci_interrupt(int irq, void *dev_id)
{
struct dw_mci *host = dev_id;
u32 pending;
- int i;
+ struct dw_mci_slot *slot = host->slot;
pending = mci_readl(host, MINTSTS); /* read-only mask reg */
dw_mci_handle_cd(host);
}
- /* Handle SDIO Interrupts */
- for (i = 0; i < host->num_slots; i++) {
- struct dw_mci_slot *slot = host->slot[i];
-
- if (!slot)
- continue;
-
- if (pending & SDMMC_INT_SDIO(slot->sdio_id)) {
- mci_writel(host, RINTSTS,
- SDMMC_INT_SDIO(slot->sdio_id));
- mmc_signal_sdio_irq(slot->mmc);
- }
+ if (pending & SDMMC_INT_SDIO(slot->sdio_id)) {
+ mci_writel(host, RINTSTS,
+ SDMMC_INT_SDIO(slot->sdio_id));
+ __dw_mci_enable_sdio_irq(slot, 0);
+ sdio_signal_irq(slot->mmc);
}
}
return IRQ_HANDLED;
}
-static int dw_mci_init_slot(struct dw_mci *host, unsigned int id)
+static int dw_mci_init_slot(struct dw_mci *host)
{
struct mmc_host *mmc;
struct dw_mci_slot *slot;
return -ENOMEM;
slot = mmc_priv(mmc);
- slot->id = id;
- slot->sdio_id = host->sdio_id0 + id;
+ slot->id = 0;
+ slot->sdio_id = host->sdio_id0 + slot->id;
slot->mmc = mmc;
slot->host = host;
- host->slot[id] = slot;
+ host->slot = slot;
mmc->ops = &dw_mci_ops;
- if (of_property_read_u32_array(host->dev->of_node,
- "clock-freq-min-max", freq, 2)) {
+ if (device_property_read_u32_array(host->dev, "clock-freq-min-max",
+ freq, 2)) {
mmc->f_min = DW_MCI_FREQ_MIN;
mmc->f_max = DW_MCI_FREQ_MAX;
} else {
if (ret)
goto err_host_allocated;
+ /* Process SDIO IRQs through the sdio_irq_work. */
+ if (mmc->caps & MMC_CAP_SDIO_IRQ)
+ mmc->caps2 |= MMC_CAP2_SDIO_IRQ_NOTHREAD;
+
/* Useful defaults if platform data is unset. */
if (host->use_dma == TRANS_MODE_IDMAC) {
mmc->max_segs = host->ring_size;
return ret;
}
-static void dw_mci_cleanup_slot(struct dw_mci_slot *slot, unsigned int id)
+static void dw_mci_cleanup_slot(struct dw_mci_slot *slot)
{
/* Debugfs stuff is cleaned up by mmc core */
mmc_remove_host(slot->mmc);
- slot->host->slot[id] = NULL;
+ slot->host->slot = NULL;
mmc_free_host(slot->mmc);
}
{
int addr_config;
struct device *dev = host->dev;
- struct device_node *np = dev->of_node;
/*
* Check tansfer mode from HCON[17:16]
dev_info(host->dev, "Using internal DMA controller.\n");
} else {
/* TRANS_MODE_EDMAC: check dma bindings again */
- if ((of_property_count_strings(np, "dma-names") < 0) ||
- (!of_find_property(np, "dmas", NULL))) {
+ if ((device_property_read_string_array(dev, "dma-names",
+ NULL, 0) < 0) ||
+ !device_property_present(dev, "dmas")) {
goto no_dma;
}
host->dma_ops = &dw_mci_edmac_ops;
{
struct dw_mci_board *pdata;
struct device *dev = host->dev;
- struct device_node *np = dev->of_node;
const struct dw_mci_drv_data *drv_data = host->drv_data;
int ret;
u32 clock_frequency;
}
/* find out number of slots supported */
- of_property_read_u32(np, "num-slots", &pdata->num_slots);
+ if (device_property_read_u32(dev, "num-slots", &pdata->num_slots))
+ dev_info(dev, "'num-slots' was deprecated.\n");
- if (of_property_read_u32(np, "fifo-depth", &pdata->fifo_depth))
+ if (device_property_read_u32(dev, "fifo-depth", &pdata->fifo_depth))
dev_info(dev,
"fifo-depth property not found, using value of FIFOTH register as default\n");
- of_property_read_u32(np, "card-detect-delay", &pdata->detect_delay_ms);
+ device_property_read_u32(dev, "card-detect-delay",
+ &pdata->detect_delay_ms);
- of_property_read_u32(np, "data-addr", &host->data_addr_override);
+ device_property_read_u32(dev, "data-addr", &host->data_addr_override);
- if (of_get_property(np, "fifo-watermark-aligned", NULL))
+ if (device_property_present(dev, "fifo-watermark-aligned"))
host->wm_aligned = true;
- if (!of_property_read_u32(np, "clock-frequency", &clock_frequency))
+ if (!device_property_read_u32(dev, "clock-frequency", &clock_frequency))
pdata->bus_hz = clock_frequency;
if (drv_data && drv_data->parse_dt) {
{
unsigned long irqflags;
u32 temp;
- int i;
- struct dw_mci_slot *slot;
/*
* No need for CD if all slots have a non-error GPIO
* as well as broken card detection is found.
*/
- for (i = 0; i < host->num_slots; i++) {
- slot = host->slot[i];
- if (slot->mmc->caps & MMC_CAP_NEEDS_POLL)
- return;
-
- if (mmc_gpio_get_cd(slot->mmc) < 0)
- break;
- }
- if (i == host->num_slots)
+ if (host->slot->mmc->caps & MMC_CAP_NEEDS_POLL)
return;
- spin_lock_irqsave(&host->irq_lock, irqflags);
- temp = mci_readl(host, INTMASK);
- temp |= SDMMC_INT_CD;
- mci_writel(host, INTMASK, temp);
- spin_unlock_irqrestore(&host->irq_lock, irqflags);
+ if (mmc_gpio_get_cd(host->slot->mmc) < 0) {
+ spin_lock_irqsave(&host->irq_lock, irqflags);
+ temp = mci_readl(host, INTMASK);
+ temp |= SDMMC_INT_CD;
+ mci_writel(host, INTMASK, temp);
+ spin_unlock_irqrestore(&host->irq_lock, irqflags);
+ }
}
int dw_mci_probe(struct dw_mci *host)
const struct dw_mci_drv_data *drv_data = host->drv_data;
int width, i, ret = 0;
u32 fifo_size;
- int init_slots = 0;
if (!host->pdata) {
host->pdata = dw_mci_parse_dt(host);
if (ret)
goto err_dmaunmap;
- if (host->pdata->num_slots)
- host->num_slots = host->pdata->num_slots;
- else
- host->num_slots = 1;
-
- if (host->num_slots < 1 ||
- host->num_slots > SDMMC_GET_SLOT_NUM(mci_readl(host, HCON))) {
- dev_err(host->dev,
- "Platform data must supply correct num_slots.\n");
- ret = -ENODEV;
- goto err_clk_ciu;
- }
-
/*
* Enable interrupts for command done, data over, data empty,
* receive ready and error such as transmit, receive timeout, crc error
host->irq, width, fifo_size);
/* We need at least one slot to succeed */
- for (i = 0; i < host->num_slots; i++) {
- ret = dw_mci_init_slot(host, i);
- if (ret)
- dev_dbg(host->dev, "slot %d init failed\n", i);
- else
- init_slots++;
- }
-
- if (init_slots) {
- dev_info(host->dev, "%d slots initialized\n", init_slots);
- } else {
- dev_dbg(host->dev,
- "attempted to initialize %d slots, but failed on all\n",
- host->num_slots);
+ ret = dw_mci_init_slot(host);
+ if (ret) {
+ dev_dbg(host->dev, "slot %d init failed\n", i);
goto err_dmaunmap;
}
void dw_mci_remove(struct dw_mci *host)
{
- int i;
-
- for (i = 0; i < host->num_slots; i++) {
- dev_dbg(host->dev, "remove slot %d\n", i);
- if (host->slot[i])
- dw_mci_cleanup_slot(host->slot[i], i);
- }
+ dev_dbg(host->dev, "remove slot\n");
+ if (host->slot)
+ dw_mci_cleanup_slot(host->slot);
mci_writel(host, RINTSTS, 0xFFFFFFFF);
mci_writel(host, INTMASK, 0); /* disable all mmc interrupt first */
clk_disable_unprepare(host->ciu_clk);
- if (host->cur_slot &&
- (mmc_can_gpio_cd(host->cur_slot->mmc) ||
- !mmc_card_is_removable(host->cur_slot->mmc)))
+ if (host->slot &&
+ (mmc_can_gpio_cd(host->slot->mmc) ||
+ !mmc_card_is_removable(host->slot->mmc)))
clk_disable_unprepare(host->biu_clk);
return 0;
int dw_mci_runtime_resume(struct device *dev)
{
- int i, ret = 0;
+ int ret = 0;
struct dw_mci *host = dev_get_drvdata(dev);
- if (host->cur_slot &&
- (mmc_can_gpio_cd(host->cur_slot->mmc) ||
- !mmc_card_is_removable(host->cur_slot->mmc))) {
+ if (host->slot &&
+ (mmc_can_gpio_cd(host->slot->mmc) ||
+ !mmc_card_is_removable(host->slot->mmc))) {
ret = clk_prepare_enable(host->biu_clk);
if (ret)
return ret;
DW_MCI_ERROR_FLAGS);
mci_writel(host, CTRL, SDMMC_CTRL_INT_ENABLE);
- for (i = 0; i < host->num_slots; i++) {
- struct dw_mci_slot *slot = host->slot[i];
- if (!slot)
- continue;
- if (slot->mmc->pm_flags & MMC_PM_KEEP_POWER)
- dw_mci_set_ios(slot->mmc, &slot->mmc->ios);
+ if (host->slot->mmc->pm_flags & MMC_PM_KEEP_POWER)
+ dw_mci_set_ios(host->slot->mmc, &host->slot->mmc->ios);
- /* Force setup bus to guarantee available clock output */
- dw_mci_setup_bus(slot, true);
- }
+ /* Force setup bus to guarantee available clock output */
+ dw_mci_setup_bus(host->slot, true);
/* Now that slots are all setup, we can enable card detect */
dw_mci_enable_cd(host);
return 0;
err:
- if (host->cur_slot &&
- (mmc_can_gpio_cd(host->cur_slot->mmc) ||
- !mmc_card_is_removable(host->cur_slot->mmc)))
+ if (host->slot &&
+ (mmc_can_gpio_cd(host->slot->mmc) ||
+ !mmc_card_is_removable(host->slot->mmc)))
clk_disable_unprepare(host->biu_clk);
return ret;
#include <linux/reset.h>
#include <linux/interrupt.h>
-#define MAX_MCI_SLOTS 2
-
enum dw_mci_state {
STATE_IDLE = 0,
STATE_SENDING_CMD,
* =======
*
* @lock is a softirq-safe spinlock protecting @queue as well as
- * @cur_slot, @mrq and @state. These must always be updated
* at the same time while holding @lock.
*
* @irq_lock is an irq-safe spinlock protecting the INTMASK register
struct scatterlist *sg;
struct sg_mapping_iter sg_miter;
- struct dw_mci_slot *cur_slot;
struct mmc_request *mrq;
struct mmc_command *cmd;
struct mmc_data *data;
u32 bus_hz;
u32 current_speed;
- u32 num_slots;
u32 fifoth_val;
u16 verid;
struct device *dev;
void *priv;
struct clk *biu_clk;
struct clk *ciu_clk;
- struct dw_mci_slot *slot[MAX_MCI_SLOTS];
+ struct dw_mci_slot *slot;
/* FIFO push and pull */
int fifo_depth;
pm_runtime_disable(host->dev);
pm_runtime_put_noidle(host->dev);
dma_free_coherent(&pdev->dev,
- sizeof(struct mt_gpdma_desc),
+ 2 * sizeof(struct mt_gpdma_desc),
host->dma.gpd, host->dma.gpd_addr);
dma_free_coherent(&pdev->dev, MAX_BD_NUM * sizeof(struct mt_bdma_desc),
host->dma.bd, host->dma.bd_addr);
struct omap_hsmmc_host *host = mmc_priv(mmc);
struct mmc_ios *ios = &mmc->ios;
- if (mmc->supply.vmmc) {
+ if (!IS_ERR(mmc->supply.vmmc)) {
ret = mmc_regulator_set_ocr(mmc, mmc->supply.vmmc, ios->vdd);
if (ret)
return ret;
}
/* Enable interface voltage rail, if needed */
- if (mmc->supply.vqmmc && !host->vqmmc_enabled) {
+ if (!IS_ERR(mmc->supply.vqmmc) && !host->vqmmc_enabled) {
ret = regulator_enable(mmc->supply.vqmmc);
if (ret) {
dev_err(mmc_dev(mmc), "vmmc_aux reg enable failed\n");
return 0;
err_vqmmc:
- if (mmc->supply.vmmc)
+ if (!IS_ERR(mmc->supply.vmmc))
mmc_regulator_set_ocr(mmc, mmc->supply.vmmc, 0);
return ret;
int status;
struct omap_hsmmc_host *host = mmc_priv(mmc);
- if (mmc->supply.vqmmc && host->vqmmc_enabled) {
+ if (!IS_ERR(mmc->supply.vqmmc) && host->vqmmc_enabled) {
ret = regulator_disable(mmc->supply.vqmmc);
if (ret) {
dev_err(mmc_dev(mmc), "vmmc_aux reg disable failed\n");
host->vqmmc_enabled = 0;
}
- if (mmc->supply.vmmc) {
+ if (!IS_ERR(mmc->supply.vmmc)) {
ret = mmc_regulator_set_ocr(mmc, mmc->supply.vmmc, 0);
if (ret)
goto err_set_ocr;
return 0;
err_set_ocr:
- if (mmc->supply.vqmmc) {
+ if (!IS_ERR(mmc->supply.vqmmc)) {
status = regulator_enable(mmc->supply.vqmmc);
if (status)
dev_err(mmc_dev(mmc), "vmmc_aux re-enable failed\n");
{
int ret;
- if (!host->pbias)
+ if (IS_ERR(host->pbias))
return 0;
if (power_on) {
* If we don't see a Vcc regulator, assume it's a fixed
* voltage always-on regulator.
*/
- if (!mmc->supply.vmmc)
+ if (IS_ERR(mmc->supply.vmmc))
return 0;
if (mmc_pdata(host)->before_set_reg)
{
int ret;
- if (!reg)
+ if (IS_ERR(reg))
return 0;
if (regulator_is_enabled(reg)) {
static int omap_hsmmc_reg_get(struct omap_hsmmc_host *host)
{
- int ocr_value = 0;
int ret;
struct mmc_host *mmc = host->mmc;
if (mmc_pdata(host)->set_power)
return 0;
- mmc->supply.vmmc = devm_regulator_get_optional(host->dev, "vmmc");
- if (IS_ERR(mmc->supply.vmmc)) {
- ret = PTR_ERR(mmc->supply.vmmc);
- if ((ret != -ENODEV) && host->dev->of_node)
- return ret;
- dev_dbg(host->dev, "unable to get vmmc regulator %ld\n",
- PTR_ERR(mmc->supply.vmmc));
- mmc->supply.vmmc = NULL;
- } else {
- ocr_value = mmc_regulator_get_ocrmask(mmc->supply.vmmc);
- if (ocr_value > 0)
- mmc_pdata(host)->ocr_mask = ocr_value;
- }
+ ret = mmc_regulator_get_supply(mmc);
+ if (ret == -EPROBE_DEFER)
+ return ret;
/* Allow an aux regulator */
- mmc->supply.vqmmc = devm_regulator_get_optional(host->dev, "vmmc_aux");
if (IS_ERR(mmc->supply.vqmmc)) {
- ret = PTR_ERR(mmc->supply.vqmmc);
- if ((ret != -ENODEV) && host->dev->of_node)
- return ret;
- dev_dbg(host->dev, "unable to get vmmc_aux regulator %ld\n",
- PTR_ERR(mmc->supply.vqmmc));
- mmc->supply.vqmmc = NULL;
+ mmc->supply.vqmmc = devm_regulator_get_optional(host->dev,
+ "vmmc_aux");
+ if (IS_ERR(mmc->supply.vqmmc)) {
+ ret = PTR_ERR(mmc->supply.vqmmc);
+ if ((ret != -ENODEV) && host->dev->of_node)
+ return ret;
+ dev_dbg(host->dev, "unable to get vmmc_aux regulator %ld\n",
+ PTR_ERR(mmc->supply.vqmmc));
+ }
}
host->pbias = devm_regulator_get_optional(host->dev, "pbias");
}
dev_dbg(host->dev, "unable to get pbias regulator %ld\n",
PTR_ERR(host->pbias));
- host->pbias = NULL;
}
/* For eMMC do not power off when not in sleep state */
if (ret)
goto err_irq;
- mmc->ocr_avail = mmc_pdata(host)->ocr_mask;
+ if (!mmc->ocr_avail)
+ mmc->ocr_avail = mmc_pdata(host)->ocr_mask;
omap_hsmmc_disable_irq(host);
pxamci_init_ocr(host);
- mmc->caps = 0;
+ /*
+ * This architecture used to disable bounce buffers through its
+ * defconfig, now it is done at runtime as a host property.
+ */
+ mmc->caps = MMC_CAP_NO_BOUNCE_BUFF;
host->cmdat = 0;
if (!cpu_is_pxa25x()) {
mmc->caps |= MMC_CAP_4_BIT_DATA | MMC_CAP_SDIO_IRQ;
--- /dev/null
+/*
+ * Renesas Mobile SDHI
+ *
+ * Copyright (C) 2017 Horms Solutions Ltd., Simon Horman
+ * Copyright (C) 2017 Renesas Electronics Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef RENESAS_SDHI_H
+#define RENESAS_SDHI_H
+
+#include <linux/platform_device.h>
+#include "tmio_mmc.h"
+
+struct renesas_sdhi_scc {
+ unsigned long clk_rate; /* clock rate for SDR104 */
+ u32 tap; /* sampling clock position for SDR104 */
+};
+
+struct renesas_sdhi_of_data {
+ unsigned long tmio_flags;
+ u32 tmio_ocr_mask;
+ unsigned long capabilities;
+ unsigned long capabilities2;
+ enum dma_slave_buswidth dma_buswidth;
+ dma_addr_t dma_rx_offset;
+ unsigned int bus_shift;
+ int scc_offset;
+ struct renesas_sdhi_scc *taps;
+ int taps_num;
+};
+
+int renesas_sdhi_probe(struct platform_device *pdev,
+ const struct tmio_mmc_dma_ops *dma_ops);
+int renesas_sdhi_remove(struct platform_device *pdev);
+#endif
/*
- * SuperH Mobile SDHI
+ * Renesas SDHI
*
- * Copyright (C) 2016 Sang Engineering, Wolfram Sang
- * Copyright (C) 2015-16 Renesas Electronics Corporation
+ * Copyright (C) 2015-17 Renesas Electronics Corporation
+ * Copyright (C) 2016-17 Sang Engineering, Wolfram Sang
+ * Copyright (C) 2016-17 Horms Solutions, Simon Horman
* Copyright (C) 2009 Magnus Damm
*
* This program is free software; you can redistribute it and/or modify
#include <linux/kernel.h>
#include <linux/clk.h>
#include <linux/slab.h>
-#include <linux/mod_devicetable.h>
-#include <linux/module.h>
#include <linux/of_device.h>
#include <linux/platform_device.h>
#include <linux/mmc/host.h>
#include <linux/pinctrl/pinctrl-state.h>
#include <linux/regulator/consumer.h>
+#include "renesas_sdhi.h"
#include "tmio_mmc.h"
#define EXT_ACC 0xe4
#define SDHI_VER_GEN3_SD 0xcc10
#define SDHI_VER_GEN3_SDMMC 0xcd10
-#define host_to_priv(host) container_of((host)->pdata, struct sh_mobile_sdhi, mmc_data)
+#define host_to_priv(host) \
+ container_of((host)->pdata, struct renesas_sdhi, mmc_data)
-struct sh_mobile_sdhi_scc {
- unsigned long clk_rate; /* clock rate for SDR104 */
- u32 tap; /* sampling clock position for SDR104 */
-};
-
-struct sh_mobile_sdhi_of_data {
- unsigned long tmio_flags;
- u32 tmio_ocr_mask;
- unsigned long capabilities;
- unsigned long capabilities2;
- enum dma_slave_buswidth dma_buswidth;
- dma_addr_t dma_rx_offset;
- unsigned bus_shift;
- int scc_offset;
- struct sh_mobile_sdhi_scc *taps;
- int taps_num;
-};
-
-static const struct sh_mobile_sdhi_of_data of_default_cfg = {
- .tmio_flags = TMIO_MMC_HAS_IDLE_WAIT,
-};
-
-static const struct sh_mobile_sdhi_of_data of_rz_compatible = {
- .tmio_flags = TMIO_MMC_HAS_IDLE_WAIT | TMIO_MMC_32BIT_DATA_PORT,
- .tmio_ocr_mask = MMC_VDD_32_33,
- .capabilities = MMC_CAP_SD_HIGHSPEED | MMC_CAP_SDIO_IRQ,
-};
-
-static const struct sh_mobile_sdhi_of_data of_rcar_gen1_compatible = {
- .tmio_flags = TMIO_MMC_HAS_IDLE_WAIT | TMIO_MMC_WRPROTECT_DISABLE |
- TMIO_MMC_CLK_ACTUAL,
- .capabilities = MMC_CAP_SD_HIGHSPEED | MMC_CAP_SDIO_IRQ,
-};
-
-/* Definitions for sampling clocks */
-static struct sh_mobile_sdhi_scc rcar_gen2_scc_taps[] = {
- {
- .clk_rate = 156000000,
- .tap = 0x00000703,
- },
- {
- .clk_rate = 0,
- .tap = 0x00000300,
- },
-};
-
-static const struct sh_mobile_sdhi_of_data of_rcar_gen2_compatible = {
- .tmio_flags = TMIO_MMC_HAS_IDLE_WAIT | TMIO_MMC_WRPROTECT_DISABLE |
- TMIO_MMC_CLK_ACTUAL | TMIO_MMC_MIN_RCAR2,
- .capabilities = MMC_CAP_SD_HIGHSPEED | MMC_CAP_SDIO_IRQ,
- .dma_buswidth = DMA_SLAVE_BUSWIDTH_4_BYTES,
- .dma_rx_offset = 0x2000,
- .scc_offset = 0x0300,
- .taps = rcar_gen2_scc_taps,
- .taps_num = ARRAY_SIZE(rcar_gen2_scc_taps),
-};
-
-/* Definitions for sampling clocks */
-static struct sh_mobile_sdhi_scc rcar_gen3_scc_taps[] = {
- {
- .clk_rate = 0,
- .tap = 0x00000300,
- },
-};
-
-static const struct sh_mobile_sdhi_of_data of_rcar_gen3_compatible = {
- .tmio_flags = TMIO_MMC_HAS_IDLE_WAIT | TMIO_MMC_WRPROTECT_DISABLE |
- TMIO_MMC_CLK_ACTUAL | TMIO_MMC_MIN_RCAR2,
- .capabilities = MMC_CAP_SD_HIGHSPEED | MMC_CAP_SDIO_IRQ,
- .bus_shift = 2,
- .scc_offset = 0x1000,
- .taps = rcar_gen3_scc_taps,
- .taps_num = ARRAY_SIZE(rcar_gen3_scc_taps),
-};
-
-static const struct of_device_id sh_mobile_sdhi_of_match[] = {
- { .compatible = "renesas,sdhi-shmobile" },
- { .compatible = "renesas,sdhi-sh73a0", .data = &of_default_cfg, },
- { .compatible = "renesas,sdhi-r8a73a4", .data = &of_default_cfg, },
- { .compatible = "renesas,sdhi-r8a7740", .data = &of_default_cfg, },
- { .compatible = "renesas,sdhi-r7s72100", .data = &of_rz_compatible, },
- { .compatible = "renesas,sdhi-r8a7778", .data = &of_rcar_gen1_compatible, },
- { .compatible = "renesas,sdhi-r8a7779", .data = &of_rcar_gen1_compatible, },
- { .compatible = "renesas,sdhi-r8a7790", .data = &of_rcar_gen2_compatible, },
- { .compatible = "renesas,sdhi-r8a7791", .data = &of_rcar_gen2_compatible, },
- { .compatible = "renesas,sdhi-r8a7792", .data = &of_rcar_gen2_compatible, },
- { .compatible = "renesas,sdhi-r8a7793", .data = &of_rcar_gen2_compatible, },
- { .compatible = "renesas,sdhi-r8a7794", .data = &of_rcar_gen2_compatible, },
- { .compatible = "renesas,sdhi-r8a7795", .data = &of_rcar_gen3_compatible, },
- { .compatible = "renesas,sdhi-r8a7796", .data = &of_rcar_gen3_compatible, },
- {},
-};
-MODULE_DEVICE_TABLE(of, sh_mobile_sdhi_of_match);
-
-struct sh_mobile_sdhi {
+struct renesas_sdhi {
struct clk *clk;
struct clk *clk_cd;
struct tmio_mmc_data mmc_data;
void __iomem *scc_ctl;
};
-static void sh_mobile_sdhi_sdbuf_width(struct tmio_mmc_host *host, int width)
+static void renesas_sdhi_sdbuf_width(struct tmio_mmc_host *host, int width)
{
u32 val;
/*
* see also
- * sh_mobile_sdhi_of_data :: dma_buswidth
+ * renesas_sdhi_of_data :: dma_buswidth
*/
switch (sd_ctrl_read16(host, CTL_VERSION)) {
case SDHI_VER_GEN2_SDR50:
sd_ctrl_write16(host, EXT_ACC, val);
}
-static int sh_mobile_sdhi_clk_enable(struct tmio_mmc_host *host)
+static int renesas_sdhi_clk_enable(struct tmio_mmc_host *host)
{
struct mmc_host *mmc = host->mmc;
- struct sh_mobile_sdhi *priv = host_to_priv(host);
+ struct renesas_sdhi *priv = host_to_priv(host);
int ret = clk_prepare_enable(priv->clk);
+
if (ret < 0)
return ret;
mmc->f_min = max(clk_round_rate(priv->clk, 1) / 512, 1L);
/* enable 16bit data access on SDBUF as default */
- sh_mobile_sdhi_sdbuf_width(host, 16);
+ renesas_sdhi_sdbuf_width(host, 16);
return 0;
}
-static unsigned int sh_mobile_sdhi_clk_update(struct tmio_mmc_host *host,
- unsigned int new_clock)
+static unsigned int renesas_sdhi_clk_update(struct tmio_mmc_host *host,
+ unsigned int new_clock)
{
- struct sh_mobile_sdhi *priv = host_to_priv(host);
+ struct renesas_sdhi *priv = host_to_priv(host);
unsigned int freq, diff, best_freq = 0, diff_min = ~0;
int i, ret;
- /* tested only on RCar Gen2+ currently; may work for others */
+ /* tested only on R-Car Gen2+ currently; may work for others */
if (!(host->pdata->flags & TMIO_MMC_MIN_RCAR2))
return clk_get_rate(priv->clk);
return ret == 0 ? best_freq : clk_get_rate(priv->clk);
}
-static void sh_mobile_sdhi_clk_disable(struct tmio_mmc_host *host)
+static void renesas_sdhi_clk_disable(struct tmio_mmc_host *host)
{
- struct sh_mobile_sdhi *priv = host_to_priv(host);
+ struct renesas_sdhi *priv = host_to_priv(host);
clk_disable_unprepare(priv->clk);
clk_disable_unprepare(priv->clk_cd);
}
-static int sh_mobile_sdhi_card_busy(struct mmc_host *mmc)
+static int renesas_sdhi_card_busy(struct mmc_host *mmc)
{
struct tmio_mmc_host *host = mmc_priv(mmc);
- return !(sd_ctrl_read16_and_16_as_32(host, CTL_STATUS) & TMIO_STAT_DAT0);
+ return !(sd_ctrl_read16_and_16_as_32(host, CTL_STATUS) &
+ TMIO_STAT_DAT0);
}
-static int sh_mobile_sdhi_start_signal_voltage_switch(struct mmc_host *mmc,
- struct mmc_ios *ios)
+static int renesas_sdhi_start_signal_voltage_switch(struct mmc_host *mmc,
+ struct mmc_ios *ios)
{
struct tmio_mmc_host *host = mmc_priv(mmc);
- struct sh_mobile_sdhi *priv = host_to_priv(host);
+ struct renesas_sdhi *priv = host_to_priv(host);
struct pinctrl_state *pin_state;
int ret;
#define SH_MOBILE_SDHI_SCC_RVSREQ_RVSERR BIT(2)
static inline u32 sd_scc_read32(struct tmio_mmc_host *host,
- struct sh_mobile_sdhi *priv, int addr)
+ struct renesas_sdhi *priv, int addr)
{
return readl(priv->scc_ctl + (addr << host->bus_shift));
}
static inline void sd_scc_write32(struct tmio_mmc_host *host,
- struct sh_mobile_sdhi *priv,
+ struct renesas_sdhi *priv,
int addr, u32 val)
{
writel(val, priv->scc_ctl + (addr << host->bus_shift));
}
-static unsigned int sh_mobile_sdhi_init_tuning(struct tmio_mmc_host *host)
+static unsigned int renesas_sdhi_init_tuning(struct tmio_mmc_host *host)
{
- struct sh_mobile_sdhi *priv;
+ struct renesas_sdhi *priv;
priv = host_to_priv(host);
SH_MOBILE_SDHI_SCC_DTCNTL_TAPNUM_MASK;
}
-static void sh_mobile_sdhi_prepare_tuning(struct tmio_mmc_host *host,
- unsigned long tap)
+static void renesas_sdhi_prepare_tuning(struct tmio_mmc_host *host,
+ unsigned long tap)
{
- struct sh_mobile_sdhi *priv = host_to_priv(host);
+ struct renesas_sdhi *priv = host_to_priv(host);
/* Set sampling clock position */
sd_scc_write32(host, priv, SH_MOBILE_SDHI_SCC_TAPSET, tap);
#define SH_MOBILE_SDHI_MAX_TAP 3
-static int sh_mobile_sdhi_select_tuning(struct tmio_mmc_host *host)
+static int renesas_sdhi_select_tuning(struct tmio_mmc_host *host)
{
- struct sh_mobile_sdhi *priv = host_to_priv(host);
+ struct renesas_sdhi *priv = host_to_priv(host);
unsigned long tap_cnt; /* counter of tuning success */
unsigned long tap_set; /* tap position */
unsigned long tap_start;/* start position of tuning success */
tap_start = 0;
tap_end = 0;
for (i = 0; i < host->tap_num * 2; i++) {
- if (test_bit(i, host->taps))
+ if (test_bit(i, host->taps)) {
ntap++;
- else {
+ } else {
if (ntap > tap_cnt) {
tap_start = i - ntap;
tap_end = i - 1;
return 0;
}
-
-static bool sh_mobile_sdhi_check_scc_error(struct tmio_mmc_host *host)
+static bool renesas_sdhi_check_scc_error(struct tmio_mmc_host *host)
{
- struct sh_mobile_sdhi *priv = host_to_priv(host);
+ struct renesas_sdhi *priv = host_to_priv(host);
/* Check SCC error */
if (sd_scc_read32(host, priv, SH_MOBILE_SDHI_SCC_RVSCNTL) &
return false;
}
-static void sh_mobile_sdhi_hw_reset(struct tmio_mmc_host *host)
+static void renesas_sdhi_hw_reset(struct tmio_mmc_host *host)
{
- struct sh_mobile_sdhi *priv;
+ struct renesas_sdhi *priv;
priv = host_to_priv(host);
sd_scc_read32(host, priv, SH_MOBILE_SDHI_SCC_RVSCNTL));
}
-static int sh_mobile_sdhi_wait_idle(struct tmio_mmc_host *host)
+static int renesas_sdhi_wait_idle(struct tmio_mmc_host *host)
{
int timeout = 1000;
return 0;
}
-static int sh_mobile_sdhi_write16_hook(struct tmio_mmc_host *host, int addr)
+static int renesas_sdhi_write16_hook(struct tmio_mmc_host *host, int addr)
{
- switch (addr)
- {
+ switch (addr) {
case CTL_SD_CMD:
case CTL_STOP_INTERNAL_ACTION:
case CTL_XFER_BLK_COUNT:
case CTL_TRANSACTION_CTL:
case CTL_DMA_ENABLE:
case EXT_ACC:
- return sh_mobile_sdhi_wait_idle(host);
+ return renesas_sdhi_wait_idle(host);
}
return 0;
}
-static int sh_mobile_sdhi_multi_io_quirk(struct mmc_card *card,
- unsigned int direction, int blk_size)
+static int renesas_sdhi_multi_io_quirk(struct mmc_card *card,
+ unsigned int direction, int blk_size)
{
/*
* In Renesas controllers, when performing a
return blk_size;
}
-static void sh_mobile_sdhi_enable_dma(struct tmio_mmc_host *host, bool enable)
+static void renesas_sdhi_enable_dma(struct tmio_mmc_host *host, bool enable)
{
sd_ctrl_write16(host, CTL_DMA_ENABLE, enable ? 2 : 0);
/* enable 32bit access if DMA mode if possibile */
- sh_mobile_sdhi_sdbuf_width(host, enable ? 32 : 16);
+ renesas_sdhi_sdbuf_width(host, enable ? 32 : 16);
}
-static int sh_mobile_sdhi_probe(struct platform_device *pdev)
+int renesas_sdhi_probe(struct platform_device *pdev,
+ const struct tmio_mmc_dma_ops *dma_ops)
{
- const struct sh_mobile_sdhi_of_data *of_data = of_device_get_match_data(&pdev->dev);
- struct sh_mobile_sdhi *priv;
- struct tmio_mmc_data *mmc_data;
struct tmio_mmc_data *mmd = pdev->dev.platform_data;
+ const struct renesas_sdhi_of_data *of_data;
+ struct tmio_mmc_data *mmc_data;
+ struct tmio_mmc_dma *dma_priv;
struct tmio_mmc_host *host;
+ struct renesas_sdhi *priv;
struct resource *res;
int irq, ret, i;
- struct tmio_mmc_dma *dma_priv;
+
+ of_data = of_device_get_match_data(&pdev->dev);
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
if (!res)
return -EINVAL;
- priv = devm_kzalloc(&pdev->dev, sizeof(struct sh_mobile_sdhi), GFP_KERNEL);
+ priv = devm_kzalloc(&pdev->dev, sizeof(struct renesas_sdhi),
+ GFP_KERNEL);
if (!priv)
return -ENOMEM;
goto eprobe;
}
-
if (of_data) {
mmc_data->flags |= of_data->tmio_flags;
mmc_data->ocr_mask = of_data->tmio_ocr_mask;
}
host->dma = dma_priv;
- host->write16_hook = sh_mobile_sdhi_write16_hook;
- host->clk_enable = sh_mobile_sdhi_clk_enable;
- host->clk_update = sh_mobile_sdhi_clk_update;
- host->clk_disable = sh_mobile_sdhi_clk_disable;
- host->multi_io_quirk = sh_mobile_sdhi_multi_io_quirk;
+ host->write16_hook = renesas_sdhi_write16_hook;
+ host->clk_enable = renesas_sdhi_clk_enable;
+ host->clk_update = renesas_sdhi_clk_update;
+ host->clk_disable = renesas_sdhi_clk_disable;
+ host->multi_io_quirk = renesas_sdhi_multi_io_quirk;
/* SDR speeds are only available on Gen2+ */
if (mmc_data->flags & TMIO_MMC_MIN_RCAR2) {
/* card_busy caused issues on r8a73a4 (pre-Gen2) CD-less SDHI */
- host->card_busy = sh_mobile_sdhi_card_busy;
+ host->card_busy = renesas_sdhi_card_busy;
host->start_signal_voltage_switch =
- sh_mobile_sdhi_start_signal_voltage_switch;
+ renesas_sdhi_start_signal_voltage_switch;
}
/* Orginally registers were 16 bit apart, could be 32 or 64 nowadays */
*mmc_data = *mmd;
dma_priv->filter = shdma_chan_filter;
- dma_priv->enable = sh_mobile_sdhi_enable_dma;
+ dma_priv->enable = renesas_sdhi_enable_dma;
mmc_data->alignment_shift = 1; /* 2-byte alignment */
mmc_data->capabilities |= MMC_CAP_MMC_HIGHSPEED;
*/
mmc_data->flags |= TMIO_MMC_SDIO_IRQ;
- /*
- * All SDHI have CMD12 controll bit
- */
+ /* All SDHI have CMD12 control bit */
mmc_data->flags |= TMIO_MMC_HAVE_CMD12_CTRL;
/* All SDHI have SDIO status bits which must be 1 */
mmc_data->flags |= TMIO_MMC_SDIO_STATUS_SETBITS;
- ret = tmio_mmc_host_probe(host, mmc_data);
+ ret = tmio_mmc_host_probe(host, mmc_data, dma_ops);
if (ret < 0)
goto efree;
if (of_data && of_data->scc_offset &&
(host->mmc->caps & MMC_CAP_UHS_SDR104 ||
host->mmc->caps2 & MMC_CAP2_HS200_1_8V_SDR)) {
- const struct sh_mobile_sdhi_scc *taps = of_data->taps;
+ const struct renesas_sdhi_scc *taps = of_data->taps;
bool hit = false;
host->mmc->caps |= MMC_CAP_HW_RESET;
dev_warn(&host->pdev->dev, "Unknown clock rate for SDR104\n");
priv->scc_ctl = host->ctl + of_data->scc_offset;
- host->init_tuning = sh_mobile_sdhi_init_tuning;
- host->prepare_tuning = sh_mobile_sdhi_prepare_tuning;
- host->select_tuning = sh_mobile_sdhi_select_tuning;
- host->check_scc_error = sh_mobile_sdhi_check_scc_error;
- host->hw_reset = sh_mobile_sdhi_hw_reset;
+ host->init_tuning = renesas_sdhi_init_tuning;
+ host->prepare_tuning = renesas_sdhi_prepare_tuning;
+ host->select_tuning = renesas_sdhi_select_tuning;
+ host->check_scc_error = renesas_sdhi_check_scc_error;
+ host->hw_reset = renesas_sdhi_hw_reset;
}
i = 0;
break;
i++;
ret = devm_request_irq(&pdev->dev, irq, tmio_mmc_irq, 0,
- dev_name(&pdev->dev), host);
+ dev_name(&pdev->dev), host);
if (ret)
goto eirq;
}
eprobe:
return ret;
}
+EXPORT_SYMBOL_GPL(renesas_sdhi_probe);
-static int sh_mobile_sdhi_remove(struct platform_device *pdev)
+int renesas_sdhi_remove(struct platform_device *pdev)
{
struct mmc_host *mmc = platform_get_drvdata(pdev);
struct tmio_mmc_host *host = mmc_priv(mmc);
return 0;
}
-
-static const struct dev_pm_ops tmio_mmc_dev_pm_ops = {
- SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
- pm_runtime_force_resume)
- SET_RUNTIME_PM_OPS(tmio_mmc_host_runtime_suspend,
- tmio_mmc_host_runtime_resume,
- NULL)
-};
-
-static struct platform_driver sh_mobile_sdhi_driver = {
- .driver = {
- .name = "sh_mobile_sdhi",
- .pm = &tmio_mmc_dev_pm_ops,
- .of_match_table = sh_mobile_sdhi_of_match,
- },
- .probe = sh_mobile_sdhi_probe,
- .remove = sh_mobile_sdhi_remove,
-};
-
-module_platform_driver(sh_mobile_sdhi_driver);
-
-MODULE_DESCRIPTION("SuperH Mobile SDHI driver");
-MODULE_AUTHOR("Magnus Damm");
-MODULE_LICENSE("GPL v2");
-MODULE_ALIAS("platform:sh_mobile_sdhi");
+EXPORT_SYMBOL_GPL(renesas_sdhi_remove);
/*
- * linux/drivers/mmc/tmio_mmc_dma.c
+ * DMA function for TMIO MMC implementations
*
+ * Copyright (C) 2016-17 Renesas Electronics Corporation
+ * Copyright (C) 2016-17 Sang Engineering, Wolfram Sang
+ * Copyright (C) 2017 Horms Solutions, Simon Horman
* Copyright (C) 2010-2011 Guennadi Liakhovetski
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
- * DMA function for TMIO MMC implementations
*/
#include <linux/device.h>
#include <linux/dmaengine.h>
#include <linux/mfd/tmio.h>
#include <linux/mmc/host.h>
+#include <linux/mod_devicetable.h>
+#include <linux/module.h>
#include <linux/pagemap.h>
#include <linux/scatterlist.h>
+#include "renesas_sdhi.h"
#include "tmio_mmc.h"
#define TMIO_MMC_MIN_DMA_LEN 8
-void tmio_mmc_enable_dma(struct tmio_mmc_host *host, bool enable)
+static const struct renesas_sdhi_of_data of_default_cfg = {
+ .tmio_flags = TMIO_MMC_HAS_IDLE_WAIT,
+};
+
+static const struct renesas_sdhi_of_data of_rz_compatible = {
+ .tmio_flags = TMIO_MMC_HAS_IDLE_WAIT | TMIO_MMC_32BIT_DATA_PORT,
+ .tmio_ocr_mask = MMC_VDD_32_33,
+ .capabilities = MMC_CAP_SD_HIGHSPEED | MMC_CAP_SDIO_IRQ,
+};
+
+static const struct renesas_sdhi_of_data of_rcar_gen1_compatible = {
+ .tmio_flags = TMIO_MMC_HAS_IDLE_WAIT | TMIO_MMC_WRPROTECT_DISABLE |
+ TMIO_MMC_CLK_ACTUAL,
+ .capabilities = MMC_CAP_SD_HIGHSPEED | MMC_CAP_SDIO_IRQ,
+};
+
+/* Definitions for sampling clocks */
+static struct renesas_sdhi_scc rcar_gen2_scc_taps[] = {
+ {
+ .clk_rate = 156000000,
+ .tap = 0x00000703,
+ },
+ {
+ .clk_rate = 0,
+ .tap = 0x00000300,
+ },
+};
+
+static const struct renesas_sdhi_of_data of_rcar_gen2_compatible = {
+ .tmio_flags = TMIO_MMC_HAS_IDLE_WAIT | TMIO_MMC_WRPROTECT_DISABLE |
+ TMIO_MMC_CLK_ACTUAL | TMIO_MMC_MIN_RCAR2,
+ .capabilities = MMC_CAP_SD_HIGHSPEED | MMC_CAP_SDIO_IRQ |
+ MMC_CAP_CMD23,
+ .dma_buswidth = DMA_SLAVE_BUSWIDTH_4_BYTES,
+ .dma_rx_offset = 0x2000,
+ .scc_offset = 0x0300,
+ .taps = rcar_gen2_scc_taps,
+ .taps_num = ARRAY_SIZE(rcar_gen2_scc_taps),
+};
+
+/* Definitions for sampling clocks */
+static struct renesas_sdhi_scc rcar_gen3_scc_taps[] = {
+ {
+ .clk_rate = 0,
+ .tap = 0x00000300,
+ },
+};
+
+static const struct renesas_sdhi_of_data of_rcar_gen3_compatible = {
+ .tmio_flags = TMIO_MMC_HAS_IDLE_WAIT | TMIO_MMC_WRPROTECT_DISABLE |
+ TMIO_MMC_CLK_ACTUAL | TMIO_MMC_MIN_RCAR2,
+ .capabilities = MMC_CAP_SD_HIGHSPEED | MMC_CAP_SDIO_IRQ |
+ MMC_CAP_CMD23,
+ .bus_shift = 2,
+ .scc_offset = 0x1000,
+ .taps = rcar_gen3_scc_taps,
+ .taps_num = ARRAY_SIZE(rcar_gen3_scc_taps),
+};
+
+static const struct of_device_id renesas_sdhi_sys_dmac_of_match[] = {
+ { .compatible = "renesas,sdhi-shmobile" },
+ { .compatible = "renesas,sdhi-sh73a0", .data = &of_default_cfg, },
+ { .compatible = "renesas,sdhi-r8a73a4", .data = &of_default_cfg, },
+ { .compatible = "renesas,sdhi-r8a7740", .data = &of_default_cfg, },
+ { .compatible = "renesas,sdhi-r7s72100", .data = &of_rz_compatible, },
+ { .compatible = "renesas,sdhi-r8a7778", .data = &of_rcar_gen1_compatible, },
+ { .compatible = "renesas,sdhi-r8a7779", .data = &of_rcar_gen1_compatible, },
+ { .compatible = "renesas,sdhi-r8a7790", .data = &of_rcar_gen2_compatible, },
+ { .compatible = "renesas,sdhi-r8a7791", .data = &of_rcar_gen2_compatible, },
+ { .compatible = "renesas,sdhi-r8a7792", .data = &of_rcar_gen2_compatible, },
+ { .compatible = "renesas,sdhi-r8a7793", .data = &of_rcar_gen2_compatible, },
+ { .compatible = "renesas,sdhi-r8a7794", .data = &of_rcar_gen2_compatible, },
+ { .compatible = "renesas,sdhi-r8a7795", .data = &of_rcar_gen3_compatible, },
+ { .compatible = "renesas,sdhi-r8a7796", .data = &of_rcar_gen3_compatible, },
+ {},
+};
+MODULE_DEVICE_TABLE(of, renesas_sdhi_sys_dmac_of_match);
+
+static void renesas_sdhi_sys_dmac_enable_dma(struct tmio_mmc_host *host,
+ bool enable)
{
if (!host->chan_tx || !host->chan_rx)
return;
host->dma->enable(host, enable);
}
-void tmio_mmc_abort_dma(struct tmio_mmc_host *host)
+static void renesas_sdhi_sys_dmac_abort_dma(struct tmio_mmc_host *host)
{
- tmio_mmc_enable_dma(host, false);
+ renesas_sdhi_sys_dmac_enable_dma(host, false);
if (host->chan_rx)
dmaengine_terminate_all(host->chan_rx);
if (host->chan_tx)
dmaengine_terminate_all(host->chan_tx);
- tmio_mmc_enable_dma(host, true);
+ renesas_sdhi_sys_dmac_enable_dma(host, true);
}
-static void tmio_mmc_dma_callback(void *arg)
+static void renesas_sdhi_sys_dmac_dma_callback(void *arg)
{
struct tmio_mmc_host *host = arg;
spin_unlock_irq(&host->lock);
}
-static void tmio_mmc_start_dma_rx(struct tmio_mmc_host *host)
+static void renesas_sdhi_sys_dmac_start_dma_rx(struct tmio_mmc_host *host)
{
struct scatterlist *sg = host->sg_ptr, *sg_tmp;
struct dma_async_tx_descriptor *desc = NULL;
ret = dma_map_sg(chan->device->dev, sg, host->sg_len, DMA_FROM_DEVICE);
if (ret > 0)
- desc = dmaengine_prep_slave_sg(chan, sg, ret,
- DMA_DEV_TO_MEM, DMA_CTRL_ACK);
+ desc = dmaengine_prep_slave_sg(chan, sg, ret, DMA_DEV_TO_MEM,
+ DMA_CTRL_ACK);
if (desc) {
reinit_completion(&host->dma_dataend);
- desc->callback = tmio_mmc_dma_callback;
+ desc->callback = renesas_sdhi_sys_dmac_dma_callback;
desc->callback_param = host;
cookie = dmaengine_submit(desc);
pio:
if (!desc) {
/* DMA failed, fall back to PIO */
- tmio_mmc_enable_dma(host, false);
+ renesas_sdhi_sys_dmac_enable_dma(host, false);
if (ret >= 0)
ret = -EIO;
host->chan_rx = NULL;
}
}
-static void tmio_mmc_start_dma_tx(struct tmio_mmc_host *host)
+static void renesas_sdhi_sys_dmac_start_dma_tx(struct tmio_mmc_host *host)
{
struct scatterlist *sg = host->sg_ptr, *sg_tmp;
struct dma_async_tx_descriptor *desc = NULL;
if (!aligned) {
unsigned long flags;
void *sg_vaddr = tmio_mmc_kmap_atomic(sg, &flags);
+
sg_init_one(&host->bounce_sg, host->bounce_buf, sg->length);
memcpy(host->bounce_buf, sg_vaddr, host->bounce_sg.length);
tmio_mmc_kunmap_atomic(sg, &flags, sg_vaddr);
ret = dma_map_sg(chan->device->dev, sg, host->sg_len, DMA_TO_DEVICE);
if (ret > 0)
- desc = dmaengine_prep_slave_sg(chan, sg, ret,
- DMA_MEM_TO_DEV, DMA_CTRL_ACK);
+ desc = dmaengine_prep_slave_sg(chan, sg, ret, DMA_MEM_TO_DEV,
+ DMA_CTRL_ACK);
if (desc) {
reinit_completion(&host->dma_dataend);
- desc->callback = tmio_mmc_dma_callback;
+ desc->callback = renesas_sdhi_sys_dmac_dma_callback;
desc->callback_param = host;
cookie = dmaengine_submit(desc);
pio:
if (!desc) {
/* DMA failed, fall back to PIO */
- tmio_mmc_enable_dma(host, false);
+ renesas_sdhi_sys_dmac_enable_dma(host, false);
if (ret >= 0)
ret = -EIO;
host->chan_tx = NULL;
}
}
-void tmio_mmc_start_dma(struct tmio_mmc_host *host,
- struct mmc_data *data)
+static void renesas_sdhi_sys_dmac_start_dma(struct tmio_mmc_host *host,
+ struct mmc_data *data)
{
if (data->flags & MMC_DATA_READ) {
if (host->chan_rx)
- tmio_mmc_start_dma_rx(host);
+ renesas_sdhi_sys_dmac_start_dma_rx(host);
} else {
if (host->chan_tx)
- tmio_mmc_start_dma_tx(host);
+ renesas_sdhi_sys_dmac_start_dma_tx(host);
}
}
-static void tmio_mmc_issue_tasklet_fn(unsigned long priv)
+static void renesas_sdhi_sys_dmac_issue_tasklet_fn(unsigned long priv)
{
struct tmio_mmc_host *host = (struct tmio_mmc_host *)priv;
struct dma_chan *chan = NULL;
dma_async_issue_pending(chan);
}
-void tmio_mmc_request_dma(struct tmio_mmc_host *host, struct tmio_mmc_data *pdata)
+static void renesas_sdhi_sys_dmac_request_dma(struct tmio_mmc_host *host,
+ struct tmio_mmc_data *pdata)
{
/* We can only either use DMA for both Tx and Rx or not use it at all */
if (!host->dma || (!host->pdev->dev.of_node &&
- (!pdata->chan_priv_tx || !pdata->chan_priv_rx)))
+ (!pdata->chan_priv_tx || !pdata->chan_priv_rx)))
return;
if (!host->chan_tx && !host->chan_rx) {
return;
cfg.direction = DMA_MEM_TO_DEV;
- cfg.dst_addr = res->start + (CTL_SD_DATA_PORT << host->bus_shift);
+ cfg.dst_addr = res->start +
+ (CTL_SD_DATA_PORT << host->bus_shift);
cfg.dst_addr_width = host->dma->dma_buswidth;
if (!cfg.dst_addr_width)
cfg.dst_addr_width = DMA_SLAVE_BUSWIDTH_2_BYTES;
goto ebouncebuf;
init_completion(&host->dma_dataend);
- tasklet_init(&host->dma_issue, tmio_mmc_issue_tasklet_fn, (unsigned long)host);
+ tasklet_init(&host->dma_issue,
+ renesas_sdhi_sys_dmac_issue_tasklet_fn,
+ (unsigned long)host);
}
- tmio_mmc_enable_dma(host, true);
+ renesas_sdhi_sys_dmac_enable_dma(host, true);
return;
host->chan_tx = NULL;
}
-void tmio_mmc_release_dma(struct tmio_mmc_host *host)
+static void renesas_sdhi_sys_dmac_release_dma(struct tmio_mmc_host *host)
{
if (host->chan_tx) {
struct dma_chan *chan = host->chan_tx;
+
host->chan_tx = NULL;
dma_release_channel(chan);
}
if (host->chan_rx) {
struct dma_chan *chan = host->chan_rx;
+
host->chan_rx = NULL;
dma_release_channel(chan);
}
host->bounce_buf = NULL;
}
}
+
+static const struct tmio_mmc_dma_ops renesas_sdhi_sys_dmac_dma_ops = {
+ .start = renesas_sdhi_sys_dmac_start_dma,
+ .enable = renesas_sdhi_sys_dmac_enable_dma,
+ .request = renesas_sdhi_sys_dmac_request_dma,
+ .release = renesas_sdhi_sys_dmac_release_dma,
+ .abort = renesas_sdhi_sys_dmac_abort_dma,
+};
+
+static int renesas_sdhi_sys_dmac_probe(struct platform_device *pdev)
+{
+ return renesas_sdhi_probe(pdev, &renesas_sdhi_sys_dmac_dma_ops);
+}
+
+static const struct dev_pm_ops renesas_sdhi_sys_dmac_dev_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
+ pm_runtime_force_resume)
+ SET_RUNTIME_PM_OPS(tmio_mmc_host_runtime_suspend,
+ tmio_mmc_host_runtime_resume,
+ NULL)
+};
+
+static struct platform_driver renesas_sys_dmac_sdhi_driver = {
+ .driver = {
+ .name = "sh_mobile_sdhi",
+ .pm = &renesas_sdhi_sys_dmac_dev_pm_ops,
+ .of_match_table = renesas_sdhi_sys_dmac_of_match,
+ },
+ .probe = renesas_sdhi_sys_dmac_probe,
+ .remove = renesas_sdhi_remove,
+};
+
+module_platform_driver(renesas_sys_dmac_sdhi_driver);
+
+MODULE_DESCRIPTION("Renesas SDHI driver");
+MODULE_AUTHOR("Magnus Damm");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:sh_mobile_sdhi");
.caps = MMC_CAP_8_BIT_DATA | MMC_CAP_NONREMOVABLE |
MMC_CAP_HW_RESET | MMC_CAP_1_8V_DDR |
MMC_CAP_CMD_DURING_TFR | MMC_CAP_WAIT_WHILE_BUSY,
- .caps2 = MMC_CAP2_HC_ERASE_SZ,
.flags = SDHCI_ACPI_RUNTIME_PM,
.quirks = SDHCI_QUIRK_NO_ENDATTR_IN_NOPDESC,
.quirks2 = SDHCI_QUIRK2_PRESET_VALUE_BROKEN |
if (child->status.present && child->status.enabled)
acpi_device_fix_up_power(child);
- if (acpi_bus_get_status(device) || !device->status.present)
- return -ENODEV;
-
if (sdhci_acpi_byt_defer(dev))
return -EPROBE_DEFER;
goto err_clk;
}
- /* Enable MMC_CAP2_HC_ERASE_SZ for better max discard calculations */
- host->mmc->caps2 |= MMC_CAP2_HC_ERASE_SZ;
-
sdhci_get_of_property(pdev);
mmc_of_parse(host->mmc);
#define ESDHC_CTRL_BUSWIDTH_MASK (0x3 << 1)
/*
- * There is an INT DMA ERR mis-match between eSDHC and STD SDHC SPEC:
+ * There is an INT DMA ERR mismatch between eSDHC and STD SDHC SPEC:
* Bit25 is used in STD SPEC, and is reserved in fsl eSDHC design,
* but bit28 is used as the INT DMA ERR in fsl eSDHC design.
* Define this macro DMA error INT for fsl eSDHC
* In exact block transfer, the controller doesn't complete the
* operations automatically as required at the end of the
* transfer and remains on hold if the abort command is not sent.
- * As a result, the TC flag is not asserted and SW received timeout
- * exeception. Bit1 of Vendor Spec registor is used to fix it.
+ * As a result, the TC flag is not asserted and SW received timeout
+ * exception. Bit1 of Vendor Spec register is used to fix it.
*/
#define ESDHC_FLAG_MULTIBLK_NO_INT BIT(1)
-/*
- * The flag enables the workaround for ESDHC errata ENGcm07207 which
- * affects i.MX25 and i.MX35.
- */
-#define ESDHC_FLAG_ENGCM07207 BIT(2)
/*
* The flag tells that the ESDHC controller is an USDHC block that is
* integrated on the i.MX6 series.
/* The IP has SDHCI_CAPABILITIES_1 register */
#define ESDHC_FLAG_HAVE_CAP1 BIT(6)
/*
- * The IP has errata ERR004536
+ * The IP has erratum ERR004536
* uSDHC: ADMA Length Mismatch Error occurs if the AHB read access is slow,
* when reading data from the card
+ * This flag is also set for i.MX25 and i.MX35 in order to get
+ * SDHCI_QUIRK_BROKEN_ADMA, but for different reasons (ADMA capability bits).
*/
#define ESDHC_FLAG_ERR004536 BIT(7)
/* The IP supports HS200 mode */
/* The IP supports HS400 mode */
#define ESDHC_FLAG_HS400 BIT(9)
-/* A higher clock ferquency than this rate requires strobell dll control */
+/* A clock frequency higher than this rate requires strobe dll control */
#define ESDHC_STROBE_DLL_CLK_FREQ 100000000
struct esdhc_soc_data {
};
static struct esdhc_soc_data esdhc_imx25_data = {
- .flags = ESDHC_FLAG_ENGCM07207,
+ .flags = ESDHC_FLAG_ERR004536,
};
static struct esdhc_soc_data esdhc_imx35_data = {
- .flags = ESDHC_FLAG_ENGCM07207,
+ .flags = ESDHC_FLAG_ERR004536,
};
static struct esdhc_soc_data esdhc_imx51_data = {
struct clk *clk_ahb;
struct clk *clk_per;
enum {
- NO_CMD_PENDING, /* no multiblock command pending*/
+ NO_CMD_PENDING, /* no multiblock command pending */
MULTIBLK_IN_PROCESS, /* exact multiblock cmd in process */
WAIT_FOR_INT, /* sent CMD12, waiting for response INT */
} multiblock_status;
* ADMA2 capability of esdhc, but this bit is messed up on
* some SOCs (e.g. on MX25, MX35 this bit is set, but they
* don't actually support ADMA2). So set the BROKEN_ADMA
- * uirk on MX25/35 platforms.
+ * quirk on MX25/35 platforms.
*/
if (val & SDHCI_CAN_DO_ADMA1) {
if ((val & SDHCI_INT_CARD_INT) && !esdhc_is_usdhc(imx_data)) {
/*
* Clear and then set D3CD bit to avoid missing the
- * card interrupt. This is a eSDHC controller problem
+ * card interrupt. This is an eSDHC controller problem
* so we need to apply the following workaround: clear
* and set D3CD bit will make eSDHC re-sample the card
* interrupt. In case a card interrupt was lost,
{
struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
struct pltfm_imx_data *imx_data = sdhci_pltfm_priv(pltfm_host);
- u32 new_val;
+ u32 new_val = 0;
u32 mask;
switch (reg) {
* Do not touch buswidth bits here. This is done in
* esdhc_pltfm_bus_width.
* Do not touch the D3CD bit either which is used for the
- * SDIO interrupt errata workaround.
+ * SDIO interrupt erratum workaround.
*/
mask = 0xffff & ~(ESDHC_CTRL_BUSWIDTH_MASK | ESDHC_CTRL_D3CD);
esdhc_clrset_le(host, mask, new_val, reg);
return;
+ case SDHCI_SOFTWARE_RESET:
+ if (val & SDHCI_RESET_DATA)
+ new_val = readl(host->ioaddr + SDHCI_HOST_CONTROL);
+ break;
}
esdhc_clrset_le(host, 0xff, val, reg);
- /*
- * The esdhc has a design violation to SDHC spec which tells
- * that software reset should not affect card detection circuit.
- * But esdhc clears its SYSCTL register bits [0..2] during the
- * software reset. This will stop those clocks that card detection
- * circuit relies on. To work around it, we turn the clocks on back
- * to keep card detection circuit functional.
- */
- if ((reg == SDHCI_SOFTWARE_RESET) && (val & 1)) {
- esdhc_clrset_le(host, 0x7, 0x7, ESDHC_SYSTEM_CONTROL);
- /*
- * The reset on usdhc fails to clear MIX_CTRL register.
- * Do it manually here.
- */
- if (esdhc_is_usdhc(imx_data)) {
- /* the tuning bits should be kept during reset */
- new_val = readl(host->ioaddr + ESDHC_MIX_CTRL);
- writel(new_val & ESDHC_MIX_CTRL_TUNING_MASK,
- host->ioaddr + ESDHC_MIX_CTRL);
- imx_data->is_ddr = 0;
+ if (reg == SDHCI_SOFTWARE_RESET) {
+ if (val & SDHCI_RESET_ALL) {
+ /*
+ * The esdhc has a design violation to SDHC spec which
+ * tells that software reset should not affect card
+ * detection circuit. But esdhc clears its SYSCTL
+ * register bits [0..2] during the software reset. This
+ * will stop those clocks that card detection circuit
+ * relies on. To work around it, we turn the clocks on
+ * back to keep card detection circuit functional.
+ */
+ esdhc_clrset_le(host, 0x7, 0x7, ESDHC_SYSTEM_CONTROL);
+ /*
+ * The reset on usdhc fails to clear MIX_CTRL register.
+ * Do it manually here.
+ */
+ if (esdhc_is_usdhc(imx_data)) {
+ /*
+ * the tuning bits should be kept during reset
+ */
+ new_val = readl(host->ioaddr + ESDHC_MIX_CTRL);
+ writel(new_val & ESDHC_MIX_CTRL_TUNING_MASK,
+ host->ioaddr + ESDHC_MIX_CTRL);
+ imx_data->is_ddr = 0;
+ }
+ } else if (val & SDHCI_RESET_DATA) {
+ /*
+ * The eSDHC DAT line software reset clears at least the
+ * data transfer width on i.MX25, so make sure that the
+ * Host Control register is unaffected.
+ */
+ esdhc_clrset_le(host, 0xff, new_val,
+ SDHCI_HOST_CONTROL);
}
}
}
struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
struct pltfm_imx_data *imx_data = sdhci_pltfm_priv(pltfm_host);
unsigned int host_clock = pltfm_host->clock;
- int pre_div = 2;
+ int ddr_pre_div = imx_data->is_ddr ? 2 : 1;
+ int pre_div = 1;
int div = 1;
u32 temp, val;
return;
}
- if (esdhc_is_usdhc(imx_data) && !imx_data->is_ddr)
- pre_div = 1;
-
temp = sdhci_readl(host, ESDHC_SYSTEM_CONTROL);
temp &= ~(ESDHC_CLOCK_IPGEN | ESDHC_CLOCK_HCKEN | ESDHC_CLOCK_PEREN
| ESDHC_CLOCK_MASK);
sdhci_writel(host, temp, ESDHC_SYSTEM_CONTROL);
- while (host_clock / pre_div / 16 > clock && pre_div < 256)
+ while (host_clock / (16 * pre_div * ddr_pre_div) > clock &&
+ pre_div < 256)
pre_div *= 2;
- while (host_clock / pre_div / div > clock && div < 16)
+ while (host_clock / (div * pre_div * ddr_pre_div) > clock && div < 16)
div++;
- host->mmc->actual_clock = host_clock / pre_div / div;
+ host->mmc->actual_clock = host_clock / (div * pre_div * ddr_pre_div);
dev_dbg(mmc_dev(host->mmc), "desired SD clock: %d, actual: %d\n",
clock, host->mmc->actual_clock);
- if (imx_data->is_ddr)
- pre_div >>= 2;
- else
- pre_div >>= 1;
+ pre_div >>= 1;
div--;
temp = sdhci_readl(host, ESDHC_SYSTEM_CONTROL);
writel(reg, host->ioaddr + ESDHC_MIX_CTRL);
writel(val << 8, host->ioaddr + ESDHC_TUNE_CTRL_STATUS);
dev_dbg(mmc_dev(host->mmc),
- "tunning with delay 0x%x ESDHC_TUNE_CTRL_STATUS 0x%x\n",
+ "tuning with delay 0x%x ESDHC_TUNE_CTRL_STATUS 0x%x\n",
val, readl(host->ioaddr + ESDHC_TUNE_CTRL_STATUS));
}
ret = mmc_send_tuning(host->mmc, opcode, NULL);
esdhc_post_tuning(host);
- dev_dbg(mmc_dev(host->mmc), "tunning %s at 0x%x ret %d\n",
+ dev_dbg(mmc_dev(host->mmc), "tuning %s at 0x%x ret %d\n",
ret ? "failed" : "passed", avg, ret);
return ret;
}
/*
- * For HS400 eMMC, there is a data_strobe line, this signal is generated
+ * For HS400 eMMC, there is a data_strobe line. This signal is generated
* by the device and used for data output and CRC status response output
* in HS400 mode. The frequency of this signal follows the frequency of
- * CLK generated by host. Host receive the data which is aligned to the
+ * CLK generated by host. The host receives the data which is aligned to the
* edge of data_strobe line. Due to the time delay between CLK line and
* data_strobe line, if the delay time is larger than one clock cycle,
- * then CLK and data_strobe line will misaligned, read error shows up.
+ * then CLK and data_strobe line will be misaligned, read error shows up.
* So when the CLK is higher than 100MHz, each clock cycle is short enough,
- * host should config the delay target.
+ * host should configure the delay target.
*/
static void esdhc_set_strobe_dll(struct sdhci_host *host)
{
struct pltfm_imx_data *imx_data = sdhci_pltfm_priv(pltfm_host);
u32 ctrl;
- /* Rest the tuning circurt */
+ /* Reset the tuning circuit */
if (esdhc_is_usdhc(imx_data)) {
if (imx_data->socdata->flags & ESDHC_FLAG_MAN_TUNING) {
ctrl = readl(host->ioaddr + ESDHC_MIX_CTRL);
struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
struct pltfm_imx_data *imx_data = sdhci_pltfm_priv(pltfm_host);
- /* Doc Errata: the uSDHC actual maximum timeout count is 1 << 29 */
+ /* Doc Erratum: the uSDHC actual maximum timeout count is 1 << 29 */
return esdhc_is_usdhc(imx_data) ? 1 << 29 : 1 << 27;
}
/*
* ROM code will change the bit burst_length_enable setting
- * to zero if this usdhc is choosed to boot system. Change
+ * to zero if this usdhc is chosen to boot system. Change
* it back here, otherwise it will impact the performance a
* lot. This bit is used to enable/disable the burst length
- * for the external AHB2AXI bridge, it's usefully especially
+ * for the external AHB2AXI bridge. It's useful especially
* for INCR transfer because without burst length indicator,
* the AHB2AXI bridge does not know the burst length in
* advance. And without burst length indicator, AHB INCR
| ESDHC_BURST_LEN_EN_INCR,
host->ioaddr + SDHCI_HOST_CONTROL);
/*
- * errata ESDHC_FLAG_ERR004536 fix for MX6Q TO1.2 and MX6DL
+ * erratum ESDHC_FLAG_ERR004536 fix for MX6Q TO1.2 and MX6DL
* TO1.1, it's harmless for MX6SL
*/
writel(readl(host->ioaddr + 0x6c) | BIT(7),
mmc_of_parse_voltage(np, &host->ocr_mask);
- /* sdr50 and sdr104 needs work on 1.8v signal voltage */
+ /* sdr50 and sdr104 need work on 1.8v signal voltage */
if ((boarddata->support_vsel) && esdhc_is_usdhc(imx_data) &&
!IS_ERR(imx_data->pins_default)) {
imx_data->pins_100mhz = pinctrl_lookup_state(imx_data->pinctrl,
dev_warn(mmc_dev(host->mmc),
"could not get ultra high speed state, work on normal mode\n");
/*
- * fall back to not support uhs by specify no 1.8v quirk
+ * fall back to not supporting uhs by specifying no
+ * 1.8v quirk
*/
host->quirks2 |= SDHCI_QUIRK2_NO_1_8_V;
}
pltfm_host->clk = imx_data->clk_per;
pltfm_host->clock = clk_get_rate(pltfm_host->clk);
- clk_prepare_enable(imx_data->clk_per);
- clk_prepare_enable(imx_data->clk_ipg);
- clk_prepare_enable(imx_data->clk_ahb);
+ err = clk_prepare_enable(imx_data->clk_per);
+ if (err)
+ goto free_sdhci;
+ err = clk_prepare_enable(imx_data->clk_ipg);
+ if (err)
+ goto disable_per_clk;
+ err = clk_prepare_enable(imx_data->clk_ahb);
+ if (err)
+ goto disable_ipg_clk;
imx_data->pinctrl = devm_pinctrl_get(&pdev->dev);
if (IS_ERR(imx_data->pinctrl)) {
err = PTR_ERR(imx_data->pinctrl);
- goto disable_clk;
+ goto disable_ahb_clk;
}
imx_data->pins_default = pinctrl_lookup_state(imx_data->pinctrl,
if (IS_ERR(imx_data->pins_default))
dev_warn(mmc_dev(host->mmc), "could not get default state\n");
- if (imx_data->socdata->flags & ESDHC_FLAG_ENGCM07207)
- /* Fix errata ENGcm07207 present on i.MX25 and i.MX35 */
- host->quirks |= SDHCI_QUIRK_NO_MULTIBLOCK
- | SDHCI_QUIRK_BROKEN_ADMA;
-
if (esdhc_is_usdhc(imx_data)) {
host->quirks2 |= SDHCI_QUIRK2_PRESET_VALUE_BROKEN;
host->mmc->caps |= MMC_CAP_1_8V_DDR;
else
err = sdhci_esdhc_imx_probe_nondt(pdev, host, imx_data);
if (err)
- goto disable_clk;
+ goto disable_ahb_clk;
sdhci_esdhc_imx_hwinit(host);
err = sdhci_add_host(host);
if (err)
- goto disable_clk;
+ goto disable_ahb_clk;
pm_runtime_set_active(&pdev->dev);
pm_runtime_set_autosuspend_delay(&pdev->dev, 50);
return 0;
-disable_clk:
- clk_disable_unprepare(imx_data->clk_per);
- clk_disable_unprepare(imx_data->clk_ipg);
+disable_ahb_clk:
clk_disable_unprepare(imx_data->clk_ahb);
+disable_ipg_clk:
+ clk_disable_unprepare(imx_data->clk_ipg);
+disable_per_clk:
+ clk_disable_unprepare(imx_data->clk_per);
free_sdhci:
sdhci_pltfm_free(pdev);
return err;
struct sdhci_host *host = dev_get_drvdata(dev);
struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
struct pltfm_imx_data *imx_data = sdhci_pltfm_priv(pltfm_host);
+ int err;
if (!sdhci_sdio_irq_enabled(host)) {
- clk_prepare_enable(imx_data->clk_per);
- clk_prepare_enable(imx_data->clk_ipg);
+ err = clk_prepare_enable(imx_data->clk_per);
+ if (err)
+ return err;
+ err = clk_prepare_enable(imx_data->clk_ipg);
+ if (err)
+ goto disable_per_clk;
}
- clk_prepare_enable(imx_data->clk_ahb);
+ err = clk_prepare_enable(imx_data->clk_ahb);
+ if (err)
+ goto disable_ipg_clk;
+ err = sdhci_runtime_resume_host(host);
+ if (err)
+ goto disable_ahb_clk;
+
+ return 0;
- return sdhci_runtime_resume_host(host);
+disable_ahb_clk:
+ clk_disable_unprepare(imx_data->clk_ahb);
+disable_ipg_clk:
+ if (!sdhci_sdio_irq_enabled(host))
+ clk_disable_unprepare(imx_data->clk_ipg);
+disable_per_clk:
+ if (!sdhci_sdio_irq_enabled(host))
+ clk_disable_unprepare(imx_data->clk_per);
+ return err;
}
#endif
*/
#define ESDHC_DEFAULT_QUIRKS (SDHCI_QUIRK_FORCE_BLK_SZ_2048 | \
+ SDHCI_QUIRK_32BIT_DMA_ADDR | \
SDHCI_QUIRK_NO_BUSY_IRQ | \
SDHCI_QUIRK_DATA_TIMEOUT_USES_SDCLK | \
SDHCI_QUIRK_PIO_NEEDS_DELAY | \
ret = mmc_of_parse(host->mmc);
if (ret) {
- dev_err(&pdev->dev, "parsing dt failed (%u)\n", ret);
+ dev_err(&pdev->dev, "parsing dt failed (%d)\n", ret);
goto unreg_clk;
}
static int mfd_emmc_probe_slot(struct sdhci_pci_slot *slot)
{
slot->host->mmc->caps |= MMC_CAP_8_BIT_DATA | MMC_CAP_NONREMOVABLE;
- slot->host->mmc->caps2 |= MMC_CAP2_BOOTPART_NOACC |
- MMC_CAP2_HC_ERASE_SZ;
+ slot->host->mmc->caps2 |= MMC_CAP2_BOOTPART_NOACC;
return 0;
}
bool d3_retune;
};
-const u8 intel_dsm_uuid[] = {
- 0xA5, 0x3E, 0xC1, 0xF6, 0xCD, 0x65, 0x1F, 0x46,
- 0xAB, 0x7A, 0x29, 0xF7, 0xE8, 0xD5, 0xBD, 0x61,
-};
+static const guid_t intel_dsm_guid =
+ GUID_INIT(0xF6C13EA5, 0x65CD, 0x461F,
+ 0xAB, 0x7A, 0x29, 0xF7, 0xE8, 0xD5, 0xBD, 0x61);
static int __intel_dsm(struct intel_host *intel_host, struct device *dev,
unsigned int fn, u32 *result)
int err = 0;
size_t len;
- obj = acpi_evaluate_dsm(ACPI_HANDLE(dev), intel_dsm_uuid, 0, fn, NULL);
+ obj = acpi_evaluate_dsm(ACPI_HANDLE(dev), &intel_dsm_guid, 0, fn, NULL);
if (!obj)
return -EOPNOTSUPP;
}
}
+#define INTEL_HS400_ES_REG 0x78
+#define INTEL_HS400_ES_BIT BIT(0)
+
+static void intel_hs400_enhanced_strobe(struct mmc_host *mmc,
+ struct mmc_ios *ios)
+{
+ struct sdhci_host *host = mmc_priv(mmc);
+ u32 val;
+
+ val = sdhci_readl(host, INTEL_HS400_ES_REG);
+ if (ios->enhanced_strobe)
+ val |= INTEL_HS400_ES_BIT;
+ else
+ val &= ~INTEL_HS400_ES_BIT;
+ sdhci_writel(host, val, INTEL_HS400_ES_REG);
+}
+
static const struct sdhci_ops sdhci_intel_byt_ops = {
.set_clock = sdhci_set_clock,
.set_power = sdhci_intel_set_power,
MMC_CAP_HW_RESET | MMC_CAP_1_8V_DDR |
MMC_CAP_CMD_DURING_TFR |
MMC_CAP_WAIT_WHILE_BUSY;
- slot->host->mmc->caps2 |= MMC_CAP2_HC_ERASE_SZ;
slot->hw_reset = sdhci_pci_int_hw_reset;
if (slot->chip->pdev->device == PCI_DEVICE_ID_INTEL_BSW_EMMC)
slot->host->timeout_clk = 1000; /* 1000 kHz i.e. 1 MHz */
return 0;
}
+static int glk_emmc_probe_slot(struct sdhci_pci_slot *slot)
+{
+ int ret = byt_emmc_probe_slot(slot);
+
+ if (slot->chip->pdev->device != PCI_DEVICE_ID_INTEL_GLK_EMMC) {
+ slot->host->mmc->caps2 |= MMC_CAP2_HS400_ES,
+ slot->host->mmc_host_ops.hs400_enhanced_strobe =
+ intel_hs400_enhanced_strobe;
+ }
+
+ return ret;
+}
+
#ifdef CONFIG_ACPI
static int ni_set_max_freq(struct sdhci_pci_slot *slot)
{
{
byt_read_dsm(slot);
slot->host->mmc->caps |= MMC_CAP_WAIT_WHILE_BUSY |
- MMC_CAP_AGGRESSIVE_PM;
+ MMC_CAP_AGGRESSIVE_PM | MMC_CAP_CD_WAKE;
slot->cd_idx = 0;
slot->cd_override_level = true;
if (slot->chip->pdev->device == PCI_DEVICE_ID_INTEL_BXT_SD ||
.priv_size = sizeof(struct intel_host),
};
+static const struct sdhci_pci_fixes sdhci_intel_glk_emmc = {
+ .allow_runtime_pm = true,
+ .probe_slot = glk_emmc_probe_slot,
+ .quirks = SDHCI_QUIRK_NO_ENDATTR_IN_NOPDESC,
+ .quirks2 = SDHCI_QUIRK2_PRESET_VALUE_BROKEN |
+ SDHCI_QUIRK2_CAPS_BIT63_FOR_HS400 |
+ SDHCI_QUIRK2_STOP_WITH_TC,
+ .ops = &sdhci_intel_byt_ops,
+ .priv_size = sizeof(struct intel_host),
+};
+
static const struct sdhci_pci_fixes sdhci_ni_byt_sdio = {
.quirks = SDHCI_QUIRK_NO_ENDATTR_IN_NOPDESC,
.quirks2 = SDHCI_QUIRK2_HOST_OFF_CARD_ON |
};
static const struct pci_device_id pci_ids[] = {
- {
- .vendor = PCI_VENDOR_ID_RICOH,
- .device = PCI_DEVICE_ID_RICOH_R5C822,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_ricoh,
- },
-
- {
- .vendor = PCI_VENDOR_ID_RICOH,
- .device = 0x843,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_ricoh_mmc,
- },
-
- {
- .vendor = PCI_VENDOR_ID_RICOH,
- .device = 0xe822,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_ricoh_mmc,
- },
-
- {
- .vendor = PCI_VENDOR_ID_RICOH,
- .device = 0xe823,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_ricoh_mmc,
- },
-
- {
- .vendor = PCI_VENDOR_ID_ENE,
- .device = PCI_DEVICE_ID_ENE_CB712_SD,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_ene_712,
- },
-
- {
- .vendor = PCI_VENDOR_ID_ENE,
- .device = PCI_DEVICE_ID_ENE_CB712_SD_2,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_ene_712,
- },
-
- {
- .vendor = PCI_VENDOR_ID_ENE,
- .device = PCI_DEVICE_ID_ENE_CB714_SD,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_ene_714,
- },
-
- {
- .vendor = PCI_VENDOR_ID_ENE,
- .device = PCI_DEVICE_ID_ENE_CB714_SD_2,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_ene_714,
- },
-
- {
- .vendor = PCI_VENDOR_ID_MARVELL,
- .device = PCI_DEVICE_ID_MARVELL_88ALP01_SD,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_cafe,
- },
-
- {
- .vendor = PCI_VENDOR_ID_JMICRON,
- .device = PCI_DEVICE_ID_JMICRON_JMB38X_SD,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_jmicron,
- },
-
- {
- .vendor = PCI_VENDOR_ID_JMICRON,
- .device = PCI_DEVICE_ID_JMICRON_JMB38X_MMC,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_jmicron,
- },
-
- {
- .vendor = PCI_VENDOR_ID_JMICRON,
- .device = PCI_DEVICE_ID_JMICRON_JMB388_SD,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_jmicron,
- },
-
- {
- .vendor = PCI_VENDOR_ID_JMICRON,
- .device = PCI_DEVICE_ID_JMICRON_JMB388_ESD,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_jmicron,
- },
-
- {
- .vendor = PCI_VENDOR_ID_SYSKONNECT,
- .device = 0x8000,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_syskt,
- },
-
- {
- .vendor = PCI_VENDOR_ID_VIA,
- .device = 0x95d0,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_via,
- },
-
- {
- .vendor = PCI_VENDOR_ID_REALTEK,
- .device = 0x5250,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_rtsx,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_QRK_SD,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_qrk,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_MRST_SD0,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_mrst_hc0,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_MRST_SD1,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_mrst_hc1_hc2,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_MRST_SD2,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_mrst_hc1_hc2,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_MFD_SD,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_mfd_sd,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_MFD_SDIO1,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_mfd_sdio,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_MFD_SDIO2,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_mfd_sdio,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_MFD_EMMC0,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_mfd_emmc,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_MFD_EMMC1,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_mfd_emmc,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_PCH_SDIO0,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_pch_sdio,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_PCH_SDIO1,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_pch_sdio,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_BYT_EMMC,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_emmc,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_BYT_SDIO,
- .subvendor = PCI_VENDOR_ID_NI,
- .subdevice = 0x7884,
- .driver_data = (kernel_ulong_t)&sdhci_ni_byt_sdio,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_BYT_SDIO,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_sdio,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_BYT_SD,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_sd,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_BYT_EMMC2,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_emmc,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_BSW_EMMC,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_emmc,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_BSW_SDIO,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_sdio,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_BSW_SD,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_sd,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_CLV_SDIO0,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_mfd_sd,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_CLV_SDIO1,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_mfd_sdio,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_CLV_SDIO2,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_mfd_sdio,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_CLV_EMMC0,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_mfd_emmc,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_CLV_EMMC1,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_mfd_emmc,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_MRFLD_MMC,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_mrfld_mmc,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_SPT_EMMC,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_emmc,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_SPT_SDIO,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_sdio,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_SPT_SD,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_sd,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_DNV_EMMC,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_emmc,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_BXT_EMMC,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_emmc,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_BXT_SDIO,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_sdio,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_BXT_SD,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_sd,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_BXTM_EMMC,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_emmc,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_BXTM_SDIO,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_sdio,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_BXTM_SD,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_sd,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_APL_EMMC,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_emmc,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_APL_SDIO,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_sdio,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_APL_SD,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_sd,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_GLK_EMMC,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_emmc,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_GLK_SDIO,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_sdio,
- },
-
- {
- .vendor = PCI_VENDOR_ID_INTEL,
- .device = PCI_DEVICE_ID_INTEL_GLK_SD,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_intel_byt_sd,
- },
-
- {
- .vendor = PCI_VENDOR_ID_O2,
- .device = PCI_DEVICE_ID_O2_8120,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_o2,
- },
-
- {
- .vendor = PCI_VENDOR_ID_O2,
- .device = PCI_DEVICE_ID_O2_8220,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_o2,
- },
-
- {
- .vendor = PCI_VENDOR_ID_O2,
- .device = PCI_DEVICE_ID_O2_8221,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_o2,
- },
-
- {
- .vendor = PCI_VENDOR_ID_O2,
- .device = PCI_DEVICE_ID_O2_8320,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_o2,
- },
-
- {
- .vendor = PCI_VENDOR_ID_O2,
- .device = PCI_DEVICE_ID_O2_8321,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_o2,
- },
-
- {
- .vendor = PCI_VENDOR_ID_O2,
- .device = PCI_DEVICE_ID_O2_FUJIN2,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_o2,
- },
-
- {
- .vendor = PCI_VENDOR_ID_O2,
- .device = PCI_DEVICE_ID_O2_SDS0,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_o2,
- },
-
- {
- .vendor = PCI_VENDOR_ID_O2,
- .device = PCI_DEVICE_ID_O2_SDS1,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_o2,
- },
-
- {
- .vendor = PCI_VENDOR_ID_O2,
- .device = PCI_DEVICE_ID_O2_SEABIRD0,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_o2,
- },
-
- {
- .vendor = PCI_VENDOR_ID_O2,
- .device = PCI_DEVICE_ID_O2_SEABIRD1,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_o2,
- },
- {
- .vendor = PCI_VENDOR_ID_AMD,
- .device = PCI_ANY_ID,
- .class = PCI_CLASS_SYSTEM_SDHCI << 8,
- .class_mask = 0xFFFF00,
- .subvendor = PCI_ANY_ID,
- .subdevice = PCI_ANY_ID,
- .driver_data = (kernel_ulong_t)&sdhci_amd,
- },
- { /* Generic SD host controller */
- PCI_DEVICE_CLASS((PCI_CLASS_SYSTEM_SDHCI << 8), 0xFFFF00)
- },
-
+ SDHCI_PCI_DEVICE(RICOH, R5C822, ricoh),
+ SDHCI_PCI_DEVICE(RICOH, R5C843, ricoh_mmc),
+ SDHCI_PCI_DEVICE(RICOH, R5CE822, ricoh_mmc),
+ SDHCI_PCI_DEVICE(RICOH, R5CE823, ricoh_mmc),
+ SDHCI_PCI_DEVICE(ENE, CB712_SD, ene_712),
+ SDHCI_PCI_DEVICE(ENE, CB712_SD_2, ene_712),
+ SDHCI_PCI_DEVICE(ENE, CB714_SD, ene_714),
+ SDHCI_PCI_DEVICE(ENE, CB714_SD_2, ene_714),
+ SDHCI_PCI_DEVICE(MARVELL, 88ALP01_SD, cafe),
+ SDHCI_PCI_DEVICE(JMICRON, JMB38X_SD, jmicron),
+ SDHCI_PCI_DEVICE(JMICRON, JMB38X_MMC, jmicron),
+ SDHCI_PCI_DEVICE(JMICRON, JMB388_SD, jmicron),
+ SDHCI_PCI_DEVICE(JMICRON, JMB388_ESD, jmicron),
+ SDHCI_PCI_DEVICE(SYSKONNECT, 8000, syskt),
+ SDHCI_PCI_DEVICE(VIA, 95D0, via),
+ SDHCI_PCI_DEVICE(REALTEK, 5250, rtsx),
+ SDHCI_PCI_DEVICE(INTEL, QRK_SD, intel_qrk),
+ SDHCI_PCI_DEVICE(INTEL, MRST_SD0, intel_mrst_hc0),
+ SDHCI_PCI_DEVICE(INTEL, MRST_SD1, intel_mrst_hc1_hc2),
+ SDHCI_PCI_DEVICE(INTEL, MRST_SD2, intel_mrst_hc1_hc2),
+ SDHCI_PCI_DEVICE(INTEL, MFD_SD, intel_mfd_sd),
+ SDHCI_PCI_DEVICE(INTEL, MFD_SDIO1, intel_mfd_sdio),
+ SDHCI_PCI_DEVICE(INTEL, MFD_SDIO2, intel_mfd_sdio),
+ SDHCI_PCI_DEVICE(INTEL, MFD_EMMC0, intel_mfd_emmc),
+ SDHCI_PCI_DEVICE(INTEL, MFD_EMMC1, intel_mfd_emmc),
+ SDHCI_PCI_DEVICE(INTEL, PCH_SDIO0, intel_pch_sdio),
+ SDHCI_PCI_DEVICE(INTEL, PCH_SDIO1, intel_pch_sdio),
+ SDHCI_PCI_DEVICE(INTEL, BYT_EMMC, intel_byt_emmc),
+ SDHCI_PCI_SUBDEVICE(INTEL, BYT_SDIO, NI, 7884, ni_byt_sdio),
+ SDHCI_PCI_DEVICE(INTEL, BYT_SDIO, intel_byt_sdio),
+ SDHCI_PCI_DEVICE(INTEL, BYT_SD, intel_byt_sd),
+ SDHCI_PCI_DEVICE(INTEL, BYT_EMMC2, intel_byt_emmc),
+ SDHCI_PCI_DEVICE(INTEL, BSW_EMMC, intel_byt_emmc),
+ SDHCI_PCI_DEVICE(INTEL, BSW_SDIO, intel_byt_sdio),
+ SDHCI_PCI_DEVICE(INTEL, BSW_SD, intel_byt_sd),
+ SDHCI_PCI_DEVICE(INTEL, CLV_SDIO0, intel_mfd_sd),
+ SDHCI_PCI_DEVICE(INTEL, CLV_SDIO1, intel_mfd_sdio),
+ SDHCI_PCI_DEVICE(INTEL, CLV_SDIO2, intel_mfd_sdio),
+ SDHCI_PCI_DEVICE(INTEL, CLV_EMMC0, intel_mfd_emmc),
+ SDHCI_PCI_DEVICE(INTEL, CLV_EMMC1, intel_mfd_emmc),
+ SDHCI_PCI_DEVICE(INTEL, MRFLD_MMC, intel_mrfld_mmc),
+ SDHCI_PCI_DEVICE(INTEL, SPT_EMMC, intel_byt_emmc),
+ SDHCI_PCI_DEVICE(INTEL, SPT_SDIO, intel_byt_sdio),
+ SDHCI_PCI_DEVICE(INTEL, SPT_SD, intel_byt_sd),
+ SDHCI_PCI_DEVICE(INTEL, DNV_EMMC, intel_byt_emmc),
+ SDHCI_PCI_DEVICE(INTEL, BXT_EMMC, intel_byt_emmc),
+ SDHCI_PCI_DEVICE(INTEL, BXT_SDIO, intel_byt_sdio),
+ SDHCI_PCI_DEVICE(INTEL, BXT_SD, intel_byt_sd),
+ SDHCI_PCI_DEVICE(INTEL, BXTM_EMMC, intel_byt_emmc),
+ SDHCI_PCI_DEVICE(INTEL, BXTM_SDIO, intel_byt_sdio),
+ SDHCI_PCI_DEVICE(INTEL, BXTM_SD, intel_byt_sd),
+ SDHCI_PCI_DEVICE(INTEL, APL_EMMC, intel_byt_emmc),
+ SDHCI_PCI_DEVICE(INTEL, APL_SDIO, intel_byt_sdio),
+ SDHCI_PCI_DEVICE(INTEL, APL_SD, intel_byt_sd),
+ SDHCI_PCI_DEVICE(INTEL, GLK_EMMC, intel_glk_emmc),
+ SDHCI_PCI_DEVICE(INTEL, GLK_SDIO, intel_byt_sdio),
+ SDHCI_PCI_DEVICE(INTEL, GLK_SD, intel_byt_sd),
+ SDHCI_PCI_DEVICE(INTEL, CNP_EMMC, intel_glk_emmc),
+ SDHCI_PCI_DEVICE(INTEL, CNP_SD, intel_byt_sd),
+ SDHCI_PCI_DEVICE(INTEL, CNPH_SD, intel_byt_sd),
+ SDHCI_PCI_DEVICE(O2, 8120, o2),
+ SDHCI_PCI_DEVICE(O2, 8220, o2),
+ SDHCI_PCI_DEVICE(O2, 8221, o2),
+ SDHCI_PCI_DEVICE(O2, 8320, o2),
+ SDHCI_PCI_DEVICE(O2, 8321, o2),
+ SDHCI_PCI_DEVICE(O2, FUJIN2, o2),
+ SDHCI_PCI_DEVICE(O2, SDS0, o2),
+ SDHCI_PCI_DEVICE(O2, SDS1, o2),
+ SDHCI_PCI_DEVICE(O2, SEABIRD0, o2),
+ SDHCI_PCI_DEVICE(O2, SEABIRD1, o2),
+ SDHCI_PCI_DEVICE_CLASS(AMD, SYSTEM_SDHCI, PCI_CLASS_MASK, amd),
+ /* Generic SD host controller */
+ {PCI_DEVICE_CLASS(SYSTEM_SDHCI, PCI_CLASS_MASK)},
{ /* end: all zeroes */ },
};
#define __SDHCI_PCI_H
/*
- * PCI device IDs
+ * PCI device IDs, sub IDs
*/
#define PCI_DEVICE_ID_INTEL_PCH_SDIO0 0x8809
#define PCI_DEVICE_ID_INTEL_GLK_SD 0x31ca
#define PCI_DEVICE_ID_INTEL_GLK_EMMC 0x31cc
#define PCI_DEVICE_ID_INTEL_GLK_SDIO 0x31d0
+#define PCI_DEVICE_ID_INTEL_CNP_EMMC 0x9dc4
+#define PCI_DEVICE_ID_INTEL_CNP_SD 0x9df5
+#define PCI_DEVICE_ID_INTEL_CNPH_SD 0xa375
+
+#define PCI_DEVICE_ID_SYSKONNECT_8000 0x8000
+#define PCI_DEVICE_ID_VIA_95D0 0x95d0
+#define PCI_DEVICE_ID_REALTEK_5250 0x5250
+
+#define PCI_SUBDEVICE_ID_NI_7884 0x7884
+
+/*
+ * PCI device class and mask
+ */
+
+#define SYSTEM_SDHCI (PCI_CLASS_SYSTEM_SDHCI << 8)
+#define PCI_CLASS_MASK 0xFFFF00
+
+/*
+ * Macros for PCI device-description
+ */
+
+#define _PCI_VEND(vend) PCI_VENDOR_ID_##vend
+#define _PCI_DEV(vend, dev) PCI_DEVICE_ID_##vend##_##dev
+#define _PCI_SUBDEV(subvend, subdev) PCI_SUBDEVICE_ID_##subvend##_##subdev
+
+#define SDHCI_PCI_DEVICE(vend, dev, cfg) { \
+ .vendor = _PCI_VEND(vend), .device = _PCI_DEV(vend, dev), \
+ .subvendor = PCI_ANY_ID, .subdevice = PCI_ANY_ID, \
+ .driver_data = (kernel_ulong_t)&(sdhci_##cfg) \
+}
+
+#define SDHCI_PCI_SUBDEVICE(vend, dev, subvend, subdev, cfg) { \
+ .vendor = _PCI_VEND(vend), .device = _PCI_DEV(vend, dev), \
+ .subvendor = _PCI_VEND(subvend), \
+ .subdevice = _PCI_SUBDEV(subvend, subdev), \
+ .driver_data = (kernel_ulong_t)&(sdhci_##cfg) \
+}
+
+#define SDHCI_PCI_DEVICE_CLASS(vend, cl, cl_msk, cfg) { \
+ .vendor = _PCI_VEND(vend), .device = PCI_ANY_ID, \
+ .subvendor = PCI_ANY_ID, .subdevice = PCI_ANY_ID, \
+ .class = (cl), .class_mask = (cl_msk), \
+ .driver_data = (kernel_ulong_t)&(sdhci_##cfg) \
+}
/*
* PCI registers
}
}
- if (len)
- return -EIO;
-
return 0;
}
/*
- * linux/drivers/mmc/host/tmio_mmc.c
+ * Driver for the MMC / SD / SDIO cell found in:
+ *
+ * TC6393XB TC6391XB TC6387XB T7L66XB ASIC3
*
+ * Copyright (C) 2017 Renesas Electronics Corporation
+ * Copyright (C) 2017 Horms Solutions, Simon Horman
* Copyright (C) 2007 Ian Molton
* Copyright (C) 2004 Ian Molton
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
- * Driver for the MMC / SD / SDIO cell found in:
- *
- * TC6393XB TC6391XB TC6387XB T7L66XB ASIC3
*/
#include <linux/device.h>
/* SD control register space size is 0x200, 0x400 for bus_shift=1 */
host->bus_shift = resource_size(res) >> 10;
- ret = tmio_mmc_host_probe(host, pdata);
+ ret = tmio_mmc_host_probe(host, pdata, NULL);
if (ret)
goto host_free;
ret = devm_request_irq(&pdev->dev, irq, tmio_mmc_irq,
- IRQF_TRIGGER_FALLING,
- dev_name(&pdev->dev), host);
+ IRQF_TRIGGER_FALLING,
+ dev_name(&pdev->dev), host);
if (ret)
goto host_remove;
if (mmc) {
struct tmio_mmc_host *host = mmc_priv(mmc);
+
tmio_mmc_host_remove(host);
if (cell->disable)
cell->disable(pdev);
static const struct dev_pm_ops tmio_mmc_dev_pm_ops = {
SET_SYSTEM_SLEEP_PM_OPS(tmio_mmc_suspend, tmio_mmc_resume)
SET_RUNTIME_PM_OPS(tmio_mmc_host_runtime_suspend,
- tmio_mmc_host_runtime_resume,
- NULL)
+ tmio_mmc_host_runtime_resume, NULL)
};
static struct platform_driver tmio_mmc_driver = {
/*
- * linux/drivers/mmc/host/tmio_mmc.h
+ * Driver for the MMC / SD / SDIO cell found in:
+ *
+ * TC6393XB TC6391XB TC6387XB T7L66XB ASIC3
*
- * Copyright (C) 2016 Sang Engineering, Wolfram Sang
- * Copyright (C) 2015-16 Renesas Electronics Corporation
+ * Copyright (C) 2015-17 Renesas Electronics Corporation
+ * Copyright (C) 2016-17 Sang Engineering, Wolfram Sang
+ * Copyright (C) 2016-17 Horms Solutions, Simon Horman
* Copyright (C) 2007 Ian Molton
* Copyright (C) 2004 Ian Molton
*
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*
- * Driver for the MMC / SD / SDIO cell found in:
- *
- * TC6393XB TC6391XB TC6387XB T7L66XB ASIC3
*/
#ifndef TMIO_MMC_H
void (*enable)(struct tmio_mmc_host *host, bool enable);
};
+struct tmio_mmc_dma_ops {
+ void (*start)(struct tmio_mmc_host *host, struct mmc_data *data);
+ void (*enable)(struct tmio_mmc_host *host, bool enable);
+ void (*request)(struct tmio_mmc_host *host,
+ struct tmio_mmc_data *pdata);
+ void (*release)(struct tmio_mmc_host *host);
+ void (*abort)(struct tmio_mmc_host *host);
+};
+
struct tmio_mmc_host {
void __iomem *ctl;
struct mmc_command *cmd;
/* Tuning values: 1 for success, 0 for failure */
DECLARE_BITMAP(taps, BITS_PER_BYTE * sizeof(long));
unsigned int tap_num;
+
+ const struct tmio_mmc_dma_ops *dma_ops;
};
struct tmio_mmc_host *tmio_mmc_host_alloc(struct platform_device *pdev);
void tmio_mmc_host_free(struct tmio_mmc_host *host);
int tmio_mmc_host_probe(struct tmio_mmc_host *host,
- struct tmio_mmc_data *pdata);
+ struct tmio_mmc_data *pdata,
+ const struct tmio_mmc_dma_ops *dma_ops);
void tmio_mmc_host_remove(struct tmio_mmc_host *host);
void tmio_mmc_do_data_irq(struct tmio_mmc_host *host);
local_irq_restore(*flags);
}
-#if defined(CONFIG_MMC_SDHI) || defined(CONFIG_MMC_SDHI_MODULE)
-void tmio_mmc_start_dma(struct tmio_mmc_host *host, struct mmc_data *data);
-void tmio_mmc_enable_dma(struct tmio_mmc_host *host, bool enable);
-void tmio_mmc_request_dma(struct tmio_mmc_host *host, struct tmio_mmc_data *pdata);
-void tmio_mmc_release_dma(struct tmio_mmc_host *host);
-void tmio_mmc_abort_dma(struct tmio_mmc_host *host);
-#else
-static inline void tmio_mmc_start_dma(struct tmio_mmc_host *host,
- struct mmc_data *data)
-{
-}
-
-static inline void tmio_mmc_enable_dma(struct tmio_mmc_host *host, bool enable)
-{
-}
-
-static inline void tmio_mmc_request_dma(struct tmio_mmc_host *host,
- struct tmio_mmc_data *pdata)
-{
- host->chan_tx = NULL;
- host->chan_rx = NULL;
-}
-
-static inline void tmio_mmc_release_dma(struct tmio_mmc_host *host)
-{
-}
-
-static inline void tmio_mmc_abort_dma(struct tmio_mmc_host *host)
-{
-}
-#endif
-
#ifdef CONFIG_PM
int tmio_mmc_host_runtime_suspend(struct device *dev);
int tmio_mmc_host_runtime_resume(struct device *dev);
}
static inline void sd_ctrl_read16_rep(struct tmio_mmc_host *host, int addr,
- u16 *buf, int count)
+ u16 *buf, int count)
{
readsw(host->ctl + (addr << host->bus_shift), buf, count);
}
-static inline u32 sd_ctrl_read16_and_16_as_32(struct tmio_mmc_host *host, int addr)
+static inline u32 sd_ctrl_read16_and_16_as_32(struct tmio_mmc_host *host,
+ int addr)
{
return readw(host->ctl + (addr << host->bus_shift)) |
readw(host->ctl + ((addr + 2) << host->bus_shift)) << 16;
}
static inline void sd_ctrl_read32_rep(struct tmio_mmc_host *host, int addr,
- u32 *buf, int count)
+ u32 *buf, int count)
{
readsl(host->ctl + (addr << host->bus_shift), buf, count);
}
-static inline void sd_ctrl_write16(struct tmio_mmc_host *host, int addr, u16 val)
+static inline void sd_ctrl_write16(struct tmio_mmc_host *host, int addr,
+ u16 val)
{
/* If there is a hook and it returns non-zero then there
* is an error and the write should be skipped
}
static inline void sd_ctrl_write16_rep(struct tmio_mmc_host *host, int addr,
- u16 *buf, int count)
+ u16 *buf, int count)
{
writesw(host->ctl + (addr << host->bus_shift), buf, count);
}
-static inline void sd_ctrl_write32_as_16_and_16(struct tmio_mmc_host *host, int addr, u32 val)
+static inline void sd_ctrl_write32_as_16_and_16(struct tmio_mmc_host *host,
+ int addr, u32 val)
{
writew(val & 0xffff, host->ctl + (addr << host->bus_shift));
writew(val >> 16, host->ctl + ((addr + 2) << host->bus_shift));
}
static inline void sd_ctrl_write32_rep(struct tmio_mmc_host *host, int addr,
- const u32 *buf, int count)
+ const u32 *buf, int count)
{
writesl(host->ctl + (addr << host->bus_shift), buf, count);
}
/*
- * linux/drivers/mmc/host/tmio_mmc_pio.c
+ * Driver for the MMC / SD / SDIO IP found in:
+ *
+ * TC6393XB, TC6391XB, TC6387XB, T7L66XB, ASIC3, SH-Mobile SoCs
*
- * Copyright (C) 2016 Sang Engineering, Wolfram Sang
- * Copyright (C) 2015-16 Renesas Electronics Corporation
+ * Copyright (C) 2015-17 Renesas Electronics Corporation
+ * Copyright (C) 2016-17 Sang Engineering, Wolfram Sang
+ * Copyright (C) 2017 Horms Solutions, Simon Horman
* Copyright (C) 2011 Guennadi Liakhovetski
* Copyright (C) 2007 Ian Molton
* Copyright (C) 2004 Ian Molton
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*
- * Driver for the MMC / SD / SDIO IP found in:
- *
- * TC6393XB, TC6391XB, TC6387XB, T7L66XB, ASIC3, SH-Mobile SoCs
- *
* This driver draws mainly on scattered spec sheets, Reverse engineering
* of the toshiba e800 SD driver and some parts of the 2.4 ASIC3 driver (4 bit
* support). (Further 4 bit support from a later datasheet).
#include "tmio_mmc.h"
+static inline void tmio_mmc_start_dma(struct tmio_mmc_host *host,
+ struct mmc_data *data)
+{
+ if (host->dma_ops)
+ host->dma_ops->start(host, data);
+}
+
+static inline void tmio_mmc_enable_dma(struct tmio_mmc_host *host, bool enable)
+{
+ if (host->dma_ops)
+ host->dma_ops->enable(host, enable);
+}
+
+static inline void tmio_mmc_request_dma(struct tmio_mmc_host *host,
+ struct tmio_mmc_data *pdata)
+{
+ if (host->dma_ops) {
+ host->dma_ops->request(host, pdata);
+ } else {
+ host->chan_tx = NULL;
+ host->chan_rx = NULL;
+ }
+}
+
+static inline void tmio_mmc_release_dma(struct tmio_mmc_host *host)
+{
+ if (host->dma_ops)
+ host->dma_ops->release(host);
+}
+
+static inline void tmio_mmc_abort_dma(struct tmio_mmc_host *host)
+{
+ if (host->dma_ops)
+ host->dma_ops->abort(host);
+}
+
void tmio_mmc_enable_mmc_irqs(struct tmio_mmc_host *host, u32 i)
{
host->sdcard_irq_mask &= ~(i & TMIO_MASK_IRQ);
sd_ctrl_write32_as_16_and_16(host, CTL_IRQ_MASK, host->sdcard_irq_mask);
}
+EXPORT_SYMBOL_GPL(tmio_mmc_enable_mmc_irqs);
void tmio_mmc_disable_mmc_irqs(struct tmio_mmc_host *host, u32 i)
{
host->sdcard_irq_mask |= (i & TMIO_MASK_IRQ);
sd_ctrl_write32_as_16_and_16(host, CTL_IRQ_MASK, host->sdcard_irq_mask);
}
+EXPORT_SYMBOL_GPL(tmio_mmc_disable_mmc_irqs);
static void tmio_mmc_ack_mmc_irqs(struct tmio_mmc_host *host, u32 i)
{
#define STATUS_TO_TEXT(a, status, i) \
do { \
- if (status & TMIO_STAT_##a) { \
- if (i++) \
- printk(" | "); \
- printk(#a); \
+ if ((status) & TMIO_STAT_##a) { \
+ if ((i)++) \
+ printk(KERN_DEBUG " | "); \
+ printk(KERN_DEBUG #a); \
} \
} while (0)
static void pr_debug_status(u32 status)
{
int i = 0;
+
pr_debug("status: %08x = ", status);
STATUS_TO_TEXT(CARD_REMOVE, status, i);
STATUS_TO_TEXT(CARD_INSERT, status, i);
pm_runtime_get_sync(mmc_dev(mmc));
host->sdio_irq_enabled = true;
- host->sdio_irq_mask = TMIO_SDIO_MASK_ALL &
- ~TMIO_SDIO_STAT_IOIRQ;
+ host->sdio_irq_mask = TMIO_SDIO_MASK_ALL & ~TMIO_SDIO_STAT_IOIRQ;
/* Clear obsolete interrupts before enabling */
sdio_status = sd_ctrl_read16(host, CTL_SDIO_STATUS) & ~TMIO_SDIO_MASK_ALL;
}
static void tmio_mmc_set_clock(struct tmio_mmc_host *host,
- unsigned int new_clock)
+ unsigned int new_clock)
{
u32 clk = 0, clock;
if (host->pdata->flags & TMIO_MMC_HAVE_HIGH_REG)
sd_ctrl_write16(host, CTL_RESET_SDIO, 0x0001);
msleep(10);
+
+ if (host->pdata->flags & TMIO_MMC_SDIO_IRQ) {
+ sd_ctrl_write16(host, CTL_SDIO_IRQ_MASK, host->sdio_irq_mask);
+ sd_ctrl_write16(host, CTL_TRANSACTION_CTL, 0x0001);
+ }
+
}
static void tmio_mmc_reset_work(struct work_struct *work)
* cancel_delayed_work(), it can happen, that a .set_ios() call preempts
* us, so, have to check for IS_ERR(host->mrq)
*/
- if (IS_ERR_OR_NULL(mrq)
- || time_is_after_jiffies(host->last_req_ts +
- msecs_to_jiffies(CMDREQ_TIMEOUT))) {
+ if (IS_ERR_OR_NULL(mrq) ||
+ time_is_after_jiffies(host->last_req_ts +
+ msecs_to_jiffies(CMDREQ_TIMEOUT))) {
spin_unlock_irqrestore(&host->lock, flags);
return;
}
dev_warn(&host->pdev->dev,
- "timeout waiting for hardware interrupt (CMD%u)\n",
- mrq->cmd->opcode);
+ "timeout waiting for hardware interrupt (CMD%u)\n",
+ mrq->cmd->opcode);
if (host->data)
host->data->error = -ETIMEDOUT;
mmc_request_done(host->mmc, mrq);
}
-/* called with host->lock held, interrupts disabled */
-static void tmio_mmc_finish_request(struct tmio_mmc_host *host)
-{
- struct mmc_request *mrq;
- unsigned long flags;
-
- spin_lock_irqsave(&host->lock, flags);
-
- mrq = host->mrq;
- if (IS_ERR_OR_NULL(mrq)) {
- spin_unlock_irqrestore(&host->lock, flags);
- return;
- }
-
- host->cmd = NULL;
- host->data = NULL;
- host->force_pio = false;
-
- cancel_delayed_work(&host->delayed_reset_work);
-
- host->mrq = NULL;
- spin_unlock_irqrestore(&host->lock, flags);
-
- if (mrq->cmd->error || (mrq->data && mrq->data->error))
- tmio_mmc_abort_dma(host);
-
- if (host->check_scc_error)
- host->check_scc_error(host);
-
- mmc_request_done(host->mmc, mrq);
-}
-
-static void tmio_mmc_done_work(struct work_struct *work)
-{
- struct tmio_mmc_host *host = container_of(work, struct tmio_mmc_host,
- done);
- tmio_mmc_finish_request(host);
-}
-
/* These are the bitmasks the tmio chip requires to implement the MMC response
* types. Note that R1 and R6 are the same in this scheme. */
#define APP_CMD 0x0040
#define SECURITY_CMD 0x4000
#define NO_CMD12_ISSUE 0x4000 /* TMIO_MMC_HAVE_CMD12_CTRL */
-static int tmio_mmc_start_command(struct tmio_mmc_host *host, struct mmc_command *cmd)
+static int tmio_mmc_start_command(struct tmio_mmc_host *host,
+ struct mmc_command *cmd)
{
struct mmc_data *data = host->data;
int c = cmd->opcode;
c |= TRANSFER_MULTI;
/*
- * Disable auto CMD12 at IO_RW_EXTENDED when
- * multiple block transfer
+ * Disable auto CMD12 at IO_RW_EXTENDED and
+ * SET_BLOCK_COUNT when doing multiple block transfer
*/
if ((host->pdata->flags & TMIO_MMC_HAVE_CMD12_CTRL) &&
- (cmd->opcode == SD_IO_RW_EXTENDED))
+ (cmd->opcode == SD_IO_RW_EXTENDED || host->mrq->sbc))
c |= NO_CMD12_ISSUE;
}
if (data->flags & MMC_DATA_READ)
if (host->sg_off == host->sg_ptr->length)
tmio_mmc_next_sg(host);
-
- return;
}
static void tmio_mmc_check_bounce_buffer(struct tmio_mmc_host *host)
if (host->sg_ptr == &host->bounce_sg) {
unsigned long flags;
void *sg_vaddr = tmio_mmc_kmap_atomic(host->sg_orig, &flags);
+
memcpy(sg_vaddr, host->bounce_buf, host->bounce_sg.length);
tmio_mmc_kunmap_atomic(host->sg_orig, &flags, sg_vaddr);
}
host->mrq);
}
- if (stop) {
+ if (stop && !host->mrq->sbc) {
if (stop->opcode != MMC_STOP_TRANSMISSION || stop->arg)
dev_err(&host->pdev->dev, "unsupported stop: CMD%u,0x%x. We did CMD12,0\n",
stop->opcode, stop->arg);
schedule_work(&host->done);
}
+EXPORT_SYMBOL_GPL(tmio_mmc_do_data_irq);
static void tmio_mmc_data_irq(struct tmio_mmc_host *host, unsigned int stat)
{
struct mmc_data *data;
+
spin_lock(&host->lock);
data = host->data;
spin_unlock(&host->lock);
}
-static void tmio_mmc_cmd_irq(struct tmio_mmc_host *host,
- unsigned int stat)
+static void tmio_mmc_cmd_irq(struct tmio_mmc_host *host, unsigned int stat)
{
struct mmc_command *cmd = host->cmd;
int i, addr;
}
static bool __tmio_mmc_card_detect_irq(struct tmio_mmc_host *host,
- int ireg, int status)
+ int ireg, int status)
{
struct mmc_host *mmc = host->mmc;
return false;
}
-static bool __tmio_mmc_sdcard_irq(struct tmio_mmc_host *host,
- int ireg, int status)
+static bool __tmio_mmc_sdcard_irq(struct tmio_mmc_host *host, int ireg,
+ int status)
{
/* Command completion */
if (ireg & (TMIO_STAT_CMDRESPEND | TMIO_STAT_CMDTIMEOUT)) {
- tmio_mmc_ack_mmc_irqs(host,
- TMIO_STAT_CMDRESPEND |
- TMIO_STAT_CMDTIMEOUT);
+ tmio_mmc_ack_mmc_irqs(host, TMIO_STAT_CMDRESPEND |
+ TMIO_STAT_CMDTIMEOUT);
tmio_mmc_cmd_irq(host, status);
return true;
}
return IRQ_HANDLED;
}
-EXPORT_SYMBOL(tmio_mmc_irq);
+EXPORT_SYMBOL_GPL(tmio_mmc_irq);
static int tmio_mmc_start_data(struct tmio_mmc_host *host,
- struct mmc_data *data)
+ struct mmc_data *data)
{
struct tmio_mmc_data *pdata = host->pdata;
if (host->tap_num * 2 >= sizeof(host->taps) * BITS_PER_BYTE) {
dev_warn_once(&host->pdev->dev,
- "Too many taps, skipping tuning. Please consider updating size of taps field of tmio_mmc_host\n");
+ "Too many taps, skipping tuning. Please consider updating size of taps field of tmio_mmc_host\n");
goto out;
}
return ret;
}
+static void tmio_process_mrq(struct tmio_mmc_host *host,
+ struct mmc_request *mrq)
+{
+ struct mmc_command *cmd;
+ int ret;
+
+ if (mrq->sbc && host->cmd != mrq->sbc) {
+ cmd = mrq->sbc;
+ } else {
+ cmd = mrq->cmd;
+ if (mrq->data) {
+ ret = tmio_mmc_start_data(host, mrq->data);
+ if (ret)
+ goto fail;
+ }
+ }
+
+ ret = tmio_mmc_start_command(host, cmd);
+ if (ret)
+ goto fail;
+
+ schedule_delayed_work(&host->delayed_reset_work,
+ msecs_to_jiffies(CMDREQ_TIMEOUT));
+ return;
+
+fail:
+ host->force_pio = false;
+ host->mrq = NULL;
+ mrq->cmd->error = ret;
+ mmc_request_done(host->mmc, mrq);
+}
+
/* Process requests from the MMC layer */
static void tmio_mmc_request(struct mmc_host *mmc, struct mmc_request *mrq)
{
struct tmio_mmc_host *host = mmc_priv(mmc);
unsigned long flags;
- int ret;
spin_lock_irqsave(&host->lock, flags);
spin_unlock_irqrestore(&host->lock, flags);
- if (mrq->data) {
- ret = tmio_mmc_start_data(host, mrq->data);
- if (ret)
- goto fail;
+ tmio_process_mrq(host, mrq);
+}
+
+static void tmio_mmc_finish_request(struct tmio_mmc_host *host)
+{
+ struct mmc_request *mrq;
+ unsigned long flags;
+
+ spin_lock_irqsave(&host->lock, flags);
+
+ mrq = host->mrq;
+ if (IS_ERR_OR_NULL(mrq)) {
+ spin_unlock_irqrestore(&host->lock, flags);
+ return;
+ }
+
+ /* If not SET_BLOCK_COUNT, clear old data */
+ if (host->cmd != mrq->sbc) {
+ host->cmd = NULL;
+ host->data = NULL;
+ host->force_pio = false;
+ host->mrq = NULL;
}
- ret = tmio_mmc_start_command(host, mrq->cmd);
- if (!ret) {
- schedule_delayed_work(&host->delayed_reset_work,
- msecs_to_jiffies(CMDREQ_TIMEOUT));
+ cancel_delayed_work(&host->delayed_reset_work);
+
+ spin_unlock_irqrestore(&host->lock, flags);
+
+ if (mrq->cmd->error || (mrq->data && mrq->data->error))
+ tmio_mmc_abort_dma(host);
+
+ if (host->check_scc_error)
+ host->check_scc_error(host);
+
+ /* If SET_BLOCK_COUNT, continue with main command */
+ if (host->mrq) {
+ tmio_process_mrq(host, mrq);
return;
}
-fail:
- host->force_pio = false;
- host->mrq = NULL;
- mrq->cmd->error = ret;
- mmc_request_done(mmc, mrq);
+ mmc_request_done(host->mmc, mrq);
+}
+
+static void tmio_mmc_done_work(struct work_struct *work)
+{
+ struct tmio_mmc_host *host = container_of(work, struct tmio_mmc_host,
+ done);
+ tmio_mmc_finish_request(host);
}
static int tmio_mmc_clk_enable(struct tmio_mmc_host *host)
}
static void tmio_mmc_set_bus_width(struct tmio_mmc_host *host,
- unsigned char bus_width)
+ unsigned char bus_width)
{
u16 reg = sd_ctrl_read16(host, CTL_SD_MEM_CARD_OPT)
& ~(CARD_OPT_WIDTH | CARD_OPT_WIDTH8);
dev_dbg(dev,
"%s.%d: CMD%u active since %lu, now %lu!\n",
current->comm, task_pid_nr(current),
- host->mrq->cmd->opcode, host->last_req_ts, jiffies);
+ host->mrq->cmd->opcode, host->last_req_ts,
+ jiffies);
}
spin_unlock_irqrestore(&host->lock, flags);
struct tmio_mmc_host *host = mmc_priv(mmc);
struct tmio_mmc_data *pdata = host->pdata;
int ret = mmc_gpio_get_ro(mmc);
+
if (ret >= 0)
return ret;
struct tmio_mmc_data *pdata)
{
const struct device_node *np = pdev->dev.of_node;
+
if (!np)
return;
return host;
}
-EXPORT_SYMBOL(tmio_mmc_host_alloc);
+EXPORT_SYMBOL_GPL(tmio_mmc_host_alloc);
void tmio_mmc_host_free(struct tmio_mmc_host *host)
{
mmc_free_host(host->mmc);
}
-EXPORT_SYMBOL(tmio_mmc_host_free);
+EXPORT_SYMBOL_GPL(tmio_mmc_host_free);
int tmio_mmc_host_probe(struct tmio_mmc_host *_host,
- struct tmio_mmc_data *pdata)
+ struct tmio_mmc_data *pdata,
+ const struct tmio_mmc_dma_ops *dma_ops)
{
struct platform_device *pdev = _host->pdev;
struct mmc_host *mmc = _host->mmc;
return -ENOMEM;
tmio_mmc_ops.card_busy = _host->card_busy;
- tmio_mmc_ops.start_signal_voltage_switch = _host->start_signal_voltage_switch;
+ tmio_mmc_ops.start_signal_voltage_switch =
+ _host->start_signal_voltage_switch;
mmc->ops = &tmio_mmc_ops;
mmc->caps |= MMC_CAP_4_BIT_DATA | pdata->capabilities;
if (_host->native_hotplug)
pm_runtime_get_noresume(&pdev->dev);
+ _host->sdio_irq_enabled = false;
+ if (pdata->flags & TMIO_MMC_SDIO_IRQ)
+ _host->sdio_irq_mask = TMIO_SDIO_MASK_ALL;
+
tmio_mmc_clk_stop(_host);
tmio_mmc_reset(_host);
_host->sdcard_irq_mask &= ~irq_mask;
- _host->sdio_irq_enabled = false;
- if (pdata->flags & TMIO_MMC_SDIO_IRQ) {
- _host->sdio_irq_mask = TMIO_SDIO_MASK_ALL;
- sd_ctrl_write16(_host, CTL_SDIO_IRQ_MASK, _host->sdio_irq_mask);
- sd_ctrl_write16(_host, CTL_TRANSACTION_CTL, 0x0001);
- }
-
spin_lock_init(&_host->lock);
mutex_init(&_host->ios_lock);
INIT_WORK(&_host->done, tmio_mmc_done_work);
/* See if we also get DMA */
+ _host->dma_ops = dma_ops;
tmio_mmc_request_dma(_host, pdata);
pm_runtime_set_active(&pdev->dev);
return 0;
}
-EXPORT_SYMBOL(tmio_mmc_host_probe);
+EXPORT_SYMBOL_GPL(tmio_mmc_host_probe);
void tmio_mmc_host_remove(struct tmio_mmc_host *host)
{
tmio_mmc_clk_disable(host);
}
-EXPORT_SYMBOL(tmio_mmc_host_remove);
+EXPORT_SYMBOL_GPL(tmio_mmc_host_remove);
#ifdef CONFIG_PM
int tmio_mmc_host_runtime_suspend(struct device *dev)
return 0;
}
-EXPORT_SYMBOL(tmio_mmc_host_runtime_suspend);
+EXPORT_SYMBOL_GPL(tmio_mmc_host_runtime_suspend);
static bool tmio_mmc_can_retune(struct tmio_mmc_host *host)
{
return 0;
}
-EXPORT_SYMBOL(tmio_mmc_host_runtime_resume);
+EXPORT_SYMBOL_GPL(tmio_mmc_host_runtime_resume);
#endif
MODULE_LICENSE("GPL v2");
usb_string(udev, udev->descriptor.iSerialNumber, serial_number,
sizeof(serial_number));
dev_info(&udev->dev, "probing VID:PID(%04X:%04X) %s %s %s\n",
- udev->descriptor.idVendor, udev->descriptor.idProduct,
+ le16_to_cpu(udev->descriptor.idVendor),
+ le16_to_cpu(udev->descriptor.idProduct),
manufacturer, product, serial_number);
command_out_urb = usb_alloc_urb(0, GFP_KERNEL);
if (!command_out_urb) {
}
-static int do_blktrans_request(struct mtd_blktrans_ops *tr,
+static blk_status_t do_blktrans_request(struct mtd_blktrans_ops *tr,
struct mtd_blktrans_dev *dev,
struct request *req)
{
nsect = blk_rq_cur_bytes(req) >> tr->blkshift;
buf = bio_data(req->bio);
- if (req_op(req) == REQ_OP_FLUSH)
- return tr->flush(dev);
+ if (req_op(req) == REQ_OP_FLUSH) {
+ if (tr->flush(dev))
+ return BLK_STS_IOERR;
+ return BLK_STS_OK;
+ }
if (blk_rq_pos(req) + blk_rq_cur_sectors(req) >
get_capacity(req->rq_disk))
- return -EIO;
+ return BLK_STS_IOERR;
switch (req_op(req)) {
case REQ_OP_DISCARD:
- return tr->discard(dev, block, nsect);
+ if (tr->discard(dev, block, nsect))
+ return BLK_STS_IOERR;
+ return BLK_STS_OK;
case REQ_OP_READ:
for (; nsect > 0; nsect--, block++, buf += tr->blksize)
if (tr->readsect(dev, block, buf))
- return -EIO;
+ return BLK_STS_IOERR;
rq_flush_dcache_pages(req);
- return 0;
+ return BLK_STS_OK;
case REQ_OP_WRITE:
if (!tr->writesect)
- return -EIO;
+ return BLK_STS_IOERR;
rq_flush_dcache_pages(req);
for (; nsect > 0; nsect--, block++, buf += tr->blksize)
if (tr->writesect(dev, block, buf))
- return -EIO;
- return 0;
+ return BLK_STS_IOERR;
default:
- return -EIO;
+ return BLK_STS_IOERR;
}
}
spin_lock_irq(rq->queue_lock);
while (1) {
- int res;
+ blk_status_t res;
dev->bg_stop = false;
if (!req && !(req = blk_fetch_request(rq))) {
if (!dev)
while ((req = blk_fetch_request(rq)) != NULL)
- __blk_end_request_all(req, -ENODEV);
+ __blk_end_request_all(req, BLK_STS_IOERR);
else
queue_work(dev->wq, &dev->work);
}
new->rq->queuedata = new;
blk_queue_logical_block_size(new->rq, tr->blksize);
+ blk_queue_bounce_limit(new->rq, BLK_BOUNCE_HIGH);
queue_flag_set_unlocked(QUEUE_FLAG_NONROT, new->rq);
queue_flag_clear_unlocked(QUEUE_FLAG_ADD_RANDOM, new->rq);
* specify how to write bad block markers to OOB (chip->block_markbad).
*
* We try operations in the following order:
+ *
* (1) erase the affected block, to allow OOB marker to be written cleanly
* (2) write bad block marker to OOB area of affected block (unless flag
* NAND_BBT_NO_OOB_BBM is present)
* (3) update the BBT
+ *
* Note that we retain the first error encountered in (2) or (3), finish the
* procedures, and dump the error in the end.
*/
* @mtd: mtd info
* @ofs: offset to start unlock from
* @len: length to unlock
- * @invert: when = 0, unlock the range of blocks within the lower and
+ * @invert:
+ * - when = 0, unlock the range of blocks within the lower and
* upper boundary address
- * when = 1, unlock the range of blocks outside the boundaries
+ * - when = 1, unlock the range of blocks outside the boundaries
* of the lower and upper boundary address
*
* Returs unlock status.
ret = ubiblock_read(pdu);
rq_flush_dcache_pages(req);
- blk_mq_end_request(req, ret);
+ blk_mq_end_request(req, errno_to_blk_status(ret));
}
-static int ubiblock_queue_rq(struct blk_mq_hw_ctx *hctx,
+static blk_status_t ubiblock_queue_rq(struct blk_mq_hw_ctx *hctx,
const struct blk_mq_queue_data *bd)
{
struct request *req = bd->rq;
case REQ_OP_READ:
ubi_sgl_init(&pdu->usgl);
queue_work(dev->wq, &pdu->work);
- return BLK_MQ_RQ_QUEUE_OK;
+ return BLK_STS_OK;
default:
- return BLK_MQ_RQ_QUEUE_ERROR;
+ return BLK_STS_IOERR;
}
}
static DEFINE_SPINLOCK(ubi_devices_lock);
/* "Show" method for files in '/<sysfs>/class/ubi/' */
-static ssize_t ubi_version_show(struct class *class,
- struct class_attribute *attr, char *buf)
+/* UBI version attribute ('/<sysfs>/class/ubi/version') */
+static ssize_t version_show(struct class *class, struct class_attribute *attr,
+ char *buf)
{
return sprintf(buf, "%d\n", UBI_VERSION);
}
+static CLASS_ATTR_RO(version);
-/* UBI version attribute ('/<sysfs>/class/ubi/version') */
-static struct class_attribute ubi_class_attrs[] = {
- __ATTR(version, S_IRUGO, ubi_version_show, NULL),
- __ATTR_NULL
+static struct attribute *ubi_class_attrs[] = {
+ &class_attr_version.attr,
+ NULL,
};
+ATTRIBUTE_GROUPS(ubi_class);
/* Root UBI "class" object (corresponds to '/<sysfs>/class/ubi/') */
struct class ubi_class = {
.name = UBI_NAME_STR,
.owner = THIS_MODULE,
- .class_attrs = ubi_class_attrs,
+ .class_groups = ubi_class_groups,
};
static ssize_t dev_attribute_show(struct device *dev,
--- /dev/null
+#
+# Multiplexer devices
+#
+
+menuconfig MULTIPLEXER
+ tristate "Multiplexer subsystem"
+ help
+ Multiplexer controller subsystem. Multiplexers are used in a
+ variety of settings, and this subsystem abstracts their use
+ so that the rest of the kernel sees a common interface. When
+ multiple parallel multiplexers are controlled by one single
+ multiplexer controller, this subsystem also coordinates the
+ multiplexer accesses.
+
+ To compile the subsystem as a module, choose M here: the module will
+ be called mux-core.
+
+if MULTIPLEXER
+
+config MUX_ADG792A
+ tristate "Analog Devices ADG792A/ADG792G Multiplexers"
+ depends on I2C
+ help
+ ADG792A and ADG792G Wide Bandwidth Triple 4:1 Multiplexers
+
+ The driver supports both operating the three multiplexers in
+ parallel and operating them independently.
+
+ To compile the driver as a module, choose M here: the module will
+ be called mux-adg792a.
+
+config MUX_GPIO
+ tristate "GPIO-controlled Multiplexer"
+ depends on GPIOLIB || COMPILE_TEST
+ help
+ GPIO-controlled Multiplexer controller.
+
+ The driver builds a single multiplexer controller using a number
+ of gpio pins. For N pins, there will be 2^N possible multiplexer
+ states. The GPIO pins can be connected (by the hardware) to several
+ multiplexers, which in that case will be operated in parallel.
+
+ To compile the driver as a module, choose M here: the module will
+ be called mux-gpio.
+
+config MUX_MMIO
+ tristate "MMIO register bitfield-controlled Multiplexer"
+ depends on (OF && MFD_SYSCON) || COMPILE_TEST
+ help
+ MMIO register bitfield-controlled Multiplexer controller.
+
+ The driver builds multiplexer controllers for bitfields in a syscon
+ register. For N bit wide bitfields, there will be 2^N possible
+ multiplexer states.
+
+ To compile the driver as a module, choose M here: the module will
+ be called mux-mmio.
+
+endif
--- /dev/null
+#
+# Makefile for multiplexer devices.
+#
+
+obj-$(CONFIG_MULTIPLEXER) += mux-core.o
+obj-$(CONFIG_MUX_ADG792A) += mux-adg792a.o
+obj-$(CONFIG_MUX_GPIO) += mux-gpio.o
+obj-$(CONFIG_MUX_MMIO) += mux-mmio.o
--- /dev/null
+/*
+ * Multiplexer driver for Analog Devices ADG792A/G Triple 4:1 mux
+ *
+ * Copyright (C) 2017 Axentia Technologies AB
+ *
+ * Author: Peter Rosin <peda@axentia.se>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/err.h>
+#include <linux/i2c.h>
+#include <linux/module.h>
+#include <linux/mux/driver.h>
+#include <linux/property.h>
+
+#define ADG792A_LDSW BIT(0)
+#define ADG792A_RESETB BIT(1)
+#define ADG792A_DISABLE(mux) (0x50 | (mux))
+#define ADG792A_DISABLE_ALL (0x5f)
+#define ADG792A_MUX(mux, state) (0xc0 | (((mux) + 1) << 2) | (state))
+#define ADG792A_MUX_ALL(state) (0xc0 | (state))
+
+static int adg792a_write_cmd(struct i2c_client *i2c, u8 cmd, int reset)
+{
+ u8 data = ADG792A_RESETB | ADG792A_LDSW;
+
+ /* ADG792A_RESETB is active low, the chip resets when it is zero. */
+ if (reset)
+ data &= ~ADG792A_RESETB;
+
+ return i2c_smbus_write_byte_data(i2c, cmd, data);
+}
+
+static int adg792a_set(struct mux_control *mux, int state)
+{
+ struct i2c_client *i2c = to_i2c_client(mux->chip->dev.parent);
+ u8 cmd;
+
+ if (mux->chip->controllers == 1) {
+ /* parallel mux controller operation */
+ if (state == MUX_IDLE_DISCONNECT)
+ cmd = ADG792A_DISABLE_ALL;
+ else
+ cmd = ADG792A_MUX_ALL(state);
+ } else {
+ unsigned int controller = mux_control_get_index(mux);
+
+ if (state == MUX_IDLE_DISCONNECT)
+ cmd = ADG792A_DISABLE(controller);
+ else
+ cmd = ADG792A_MUX(controller, state);
+ }
+
+ return adg792a_write_cmd(i2c, cmd, 0);
+}
+
+static const struct mux_control_ops adg792a_ops = {
+ .set = adg792a_set,
+};
+
+static int adg792a_probe(struct i2c_client *i2c,
+ const struct i2c_device_id *id)
+{
+ struct device *dev = &i2c->dev;
+ struct mux_chip *mux_chip;
+ s32 idle_state[3];
+ u32 cells;
+ int ret;
+ int i;
+
+ if (!i2c_check_functionality(i2c->adapter, I2C_FUNC_SMBUS_BYTE_DATA))
+ return -ENODEV;
+
+ ret = device_property_read_u32(dev, "#mux-control-cells", &cells);
+ if (ret < 0)
+ return ret;
+ if (cells >= 2)
+ return -EINVAL;
+
+ mux_chip = devm_mux_chip_alloc(dev, cells ? 3 : 1, 0);
+ if (IS_ERR(mux_chip))
+ return PTR_ERR(mux_chip);
+
+ mux_chip->ops = &adg792a_ops;
+
+ ret = adg792a_write_cmd(i2c, ADG792A_DISABLE_ALL, 1);
+ if (ret < 0)
+ return ret;
+
+ ret = device_property_read_u32_array(dev, "idle-state",
+ (u32 *)idle_state,
+ mux_chip->controllers);
+ if (ret < 0) {
+ idle_state[0] = MUX_IDLE_AS_IS;
+ idle_state[1] = MUX_IDLE_AS_IS;
+ idle_state[2] = MUX_IDLE_AS_IS;
+ }
+
+ for (i = 0; i < mux_chip->controllers; ++i) {
+ struct mux_control *mux = &mux_chip->mux[i];
+
+ mux->states = 4;
+
+ switch (idle_state[i]) {
+ case MUX_IDLE_DISCONNECT:
+ case MUX_IDLE_AS_IS:
+ case 0 ... 4:
+ mux->idle_state = idle_state[i];
+ break;
+ default:
+ dev_err(dev, "invalid idle-state %d\n", idle_state[i]);
+ return -EINVAL;
+ }
+ }
+
+ ret = devm_mux_chip_register(dev, mux_chip);
+ if (ret < 0)
+ return ret;
+
+ if (cells)
+ dev_info(dev, "3x single pole quadruple throw muxes registered\n");
+ else
+ dev_info(dev, "triple pole quadruple throw mux registered\n");
+
+ return 0;
+}
+
+static const struct i2c_device_id adg792a_id[] = {
+ { .name = "adg792a", },
+ { .name = "adg792g", },
+ { }
+};
+MODULE_DEVICE_TABLE(i2c, adg792a_id);
+
+static const struct of_device_id adg792a_of_match[] = {
+ { .compatible = "adi,adg792a", },
+ { .compatible = "adi,adg792g", },
+ { }
+};
+MODULE_DEVICE_TABLE(of, adg792a_of_match);
+
+static struct i2c_driver adg792a_driver = {
+ .driver = {
+ .name = "adg792a",
+ .of_match_table = of_match_ptr(adg792a_of_match),
+ },
+ .probe = adg792a_probe,
+ .id_table = adg792a_id,
+};
+module_i2c_driver(adg792a_driver);
+
+MODULE_DESCRIPTION("Analog Devices ADG792A/G Triple 4:1 mux driver");
+MODULE_AUTHOR("Peter Rosin <peda@axentia.se>");
+MODULE_LICENSE("GPL v2");
--- /dev/null
+/*
+ * Multiplexer subsystem
+ *
+ * Copyright (C) 2017 Axentia Technologies AB
+ *
+ * Author: Peter Rosin <peda@axentia.se>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#define pr_fmt(fmt) "mux-core: " fmt
+
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/export.h>
+#include <linux/idr.h>
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/mux/consumer.h>
+#include <linux/mux/driver.h>
+#include <linux/of.h>
+#include <linux/of_platform.h>
+#include <linux/slab.h>
+
+/*
+ * The idle-as-is "state" is not an actual state that may be selected, it
+ * only implies that the state should not be changed. So, use that state
+ * as indication that the cached state of the multiplexer is unknown.
+ */
+#define MUX_CACHE_UNKNOWN MUX_IDLE_AS_IS
+
+static struct class mux_class = {
+ .name = "mux",
+ .owner = THIS_MODULE,
+};
+
+static DEFINE_IDA(mux_ida);
+
+static int __init mux_init(void)
+{
+ ida_init(&mux_ida);
+ return class_register(&mux_class);
+}
+
+static void __exit mux_exit(void)
+{
+ class_register(&mux_class);
+ ida_destroy(&mux_ida);
+}
+
+static void mux_chip_release(struct device *dev)
+{
+ struct mux_chip *mux_chip = to_mux_chip(dev);
+
+ ida_simple_remove(&mux_ida, mux_chip->id);
+ kfree(mux_chip);
+}
+
+static struct device_type mux_type = {
+ .name = "mux-chip",
+ .release = mux_chip_release,
+};
+
+/**
+ * mux_chip_alloc() - Allocate a mux-chip.
+ * @dev: The parent device implementing the mux interface.
+ * @controllers: The number of mux controllers to allocate for this chip.
+ * @sizeof_priv: Size of extra memory area for private use by the caller.
+ *
+ * After allocating the mux-chip with the desired number of mux controllers
+ * but before registering the chip, the mux driver is required to configure
+ * the number of valid mux states in the mux_chip->mux[N].states members and
+ * the desired idle state in the returned mux_chip->mux[N].idle_state members.
+ * The default idle state is MUX_IDLE_AS_IS. The mux driver also needs to
+ * provide a pointer to the operations struct in the mux_chip->ops member
+ * before registering the mux-chip with mux_chip_register.
+ *
+ * Return: A pointer to the new mux-chip, or an ERR_PTR with a negative errno.
+ */
+struct mux_chip *mux_chip_alloc(struct device *dev,
+ unsigned int controllers, size_t sizeof_priv)
+{
+ struct mux_chip *mux_chip;
+ int i;
+
+ if (WARN_ON(!dev || !controllers))
+ return ERR_PTR(-EINVAL);
+
+ mux_chip = kzalloc(sizeof(*mux_chip) +
+ controllers * sizeof(*mux_chip->mux) +
+ sizeof_priv, GFP_KERNEL);
+ if (!mux_chip)
+ return ERR_PTR(-ENOMEM);
+
+ mux_chip->mux = (struct mux_control *)(mux_chip + 1);
+ mux_chip->dev.class = &mux_class;
+ mux_chip->dev.type = &mux_type;
+ mux_chip->dev.parent = dev;
+ mux_chip->dev.of_node = dev->of_node;
+ dev_set_drvdata(&mux_chip->dev, mux_chip);
+
+ mux_chip->id = ida_simple_get(&mux_ida, 0, 0, GFP_KERNEL);
+ if (mux_chip->id < 0) {
+ int err = mux_chip->id;
+
+ pr_err("muxchipX failed to get a device id\n");
+ kfree(mux_chip);
+ return ERR_PTR(err);
+ }
+ dev_set_name(&mux_chip->dev, "muxchip%d", mux_chip->id);
+
+ mux_chip->controllers = controllers;
+ for (i = 0; i < controllers; ++i) {
+ struct mux_control *mux = &mux_chip->mux[i];
+
+ mux->chip = mux_chip;
+ sema_init(&mux->lock, 1);
+ mux->cached_state = MUX_CACHE_UNKNOWN;
+ mux->idle_state = MUX_IDLE_AS_IS;
+ }
+
+ device_initialize(&mux_chip->dev);
+
+ return mux_chip;
+}
+EXPORT_SYMBOL_GPL(mux_chip_alloc);
+
+static int mux_control_set(struct mux_control *mux, int state)
+{
+ int ret = mux->chip->ops->set(mux, state);
+
+ mux->cached_state = ret < 0 ? MUX_CACHE_UNKNOWN : state;
+
+ return ret;
+}
+
+/**
+ * mux_chip_register() - Register a mux-chip, thus readying the controllers
+ * for use.
+ * @mux_chip: The mux-chip to register.
+ *
+ * Do not retry registration of the same mux-chip on failure. You should
+ * instead put it away with mux_chip_free() and allocate a new one, if you
+ * for some reason would like to retry registration.
+ *
+ * Return: Zero on success or a negative errno on error.
+ */
+int mux_chip_register(struct mux_chip *mux_chip)
+{
+ int i;
+ int ret;
+
+ for (i = 0; i < mux_chip->controllers; ++i) {
+ struct mux_control *mux = &mux_chip->mux[i];
+
+ if (mux->idle_state == mux->cached_state)
+ continue;
+
+ ret = mux_control_set(mux, mux->idle_state);
+ if (ret < 0) {
+ dev_err(&mux_chip->dev, "unable to set idle state\n");
+ return ret;
+ }
+ }
+
+ ret = device_add(&mux_chip->dev);
+ if (ret < 0)
+ dev_err(&mux_chip->dev,
+ "device_add failed in %s: %d\n", __func__, ret);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(mux_chip_register);
+
+/**
+ * mux_chip_unregister() - Take the mux-chip off-line.
+ * @mux_chip: The mux-chip to unregister.
+ *
+ * mux_chip_unregister() reverses the effects of mux_chip_register().
+ * But not completely, you should not try to call mux_chip_register()
+ * on a mux-chip that has been registered before.
+ */
+void mux_chip_unregister(struct mux_chip *mux_chip)
+{
+ device_del(&mux_chip->dev);
+}
+EXPORT_SYMBOL_GPL(mux_chip_unregister);
+
+/**
+ * mux_chip_free() - Free the mux-chip for good.
+ * @mux_chip: The mux-chip to free.
+ *
+ * mux_chip_free() reverses the effects of mux_chip_alloc().
+ */
+void mux_chip_free(struct mux_chip *mux_chip)
+{
+ if (!mux_chip)
+ return;
+
+ put_device(&mux_chip->dev);
+}
+EXPORT_SYMBOL_GPL(mux_chip_free);
+
+static void devm_mux_chip_release(struct device *dev, void *res)
+{
+ struct mux_chip *mux_chip = *(struct mux_chip **)res;
+
+ mux_chip_free(mux_chip);
+}
+
+/**
+ * devm_mux_chip_alloc() - Resource-managed version of mux_chip_alloc().
+ * @dev: The parent device implementing the mux interface.
+ * @controllers: The number of mux controllers to allocate for this chip.
+ * @sizeof_priv: Size of extra memory area for private use by the caller.
+ *
+ * See mux_chip_alloc() for more details.
+ *
+ * Return: A pointer to the new mux-chip, or an ERR_PTR with a negative errno.
+ */
+struct mux_chip *devm_mux_chip_alloc(struct device *dev,
+ unsigned int controllers,
+ size_t sizeof_priv)
+{
+ struct mux_chip **ptr, *mux_chip;
+
+ ptr = devres_alloc(devm_mux_chip_release, sizeof(*ptr), GFP_KERNEL);
+ if (!ptr)
+ return ERR_PTR(-ENOMEM);
+
+ mux_chip = mux_chip_alloc(dev, controllers, sizeof_priv);
+ if (IS_ERR(mux_chip)) {
+ devres_free(ptr);
+ return mux_chip;
+ }
+
+ *ptr = mux_chip;
+ devres_add(dev, ptr);
+
+ return mux_chip;
+}
+EXPORT_SYMBOL_GPL(devm_mux_chip_alloc);
+
+static void devm_mux_chip_reg_release(struct device *dev, void *res)
+{
+ struct mux_chip *mux_chip = *(struct mux_chip **)res;
+
+ mux_chip_unregister(mux_chip);
+}
+
+/**
+ * devm_mux_chip_register() - Resource-managed version mux_chip_register().
+ * @dev: The parent device implementing the mux interface.
+ * @mux_chip: The mux-chip to register.
+ *
+ * See mux_chip_register() for more details.
+ *
+ * Return: Zero on success or a negative errno on error.
+ */
+int devm_mux_chip_register(struct device *dev,
+ struct mux_chip *mux_chip)
+{
+ struct mux_chip **ptr;
+ int res;
+
+ ptr = devres_alloc(devm_mux_chip_reg_release, sizeof(*ptr), GFP_KERNEL);
+ if (!ptr)
+ return -ENOMEM;
+
+ res = mux_chip_register(mux_chip);
+ if (res) {
+ devres_free(ptr);
+ return res;
+ }
+
+ *ptr = mux_chip;
+ devres_add(dev, ptr);
+
+ return res;
+}
+EXPORT_SYMBOL_GPL(devm_mux_chip_register);
+
+/**
+ * mux_control_states() - Query the number of multiplexer states.
+ * @mux: The mux-control to query.
+ *
+ * Return: The number of multiplexer states.
+ */
+unsigned int mux_control_states(struct mux_control *mux)
+{
+ return mux->states;
+}
+EXPORT_SYMBOL_GPL(mux_control_states);
+
+/*
+ * The mux->lock must be down when calling this function.
+ */
+static int __mux_control_select(struct mux_control *mux, int state)
+{
+ int ret;
+
+ if (WARN_ON(state < 0 || state >= mux->states))
+ return -EINVAL;
+
+ if (mux->cached_state == state)
+ return 0;
+
+ ret = mux_control_set(mux, state);
+ if (ret >= 0)
+ return 0;
+
+ /* The mux update failed, try to revert if appropriate... */
+ if (mux->idle_state != MUX_IDLE_AS_IS)
+ mux_control_set(mux, mux->idle_state);
+
+ return ret;
+}
+
+/**
+ * mux_control_select() - Select the given multiplexer state.
+ * @mux: The mux-control to request a change of state from.
+ * @state: The new requested state.
+ *
+ * On successfully selecting the mux-control state, it will be locked until
+ * there is a call to mux_control_deselect(). If the mux-control is already
+ * selected when mux_control_select() is called, the caller will be blocked
+ * until mux_control_deselect() is called (by someone else).
+ *
+ * Therefore, make sure to call mux_control_deselect() when the operation is
+ * complete and the mux-control is free for others to use, but do not call
+ * mux_control_deselect() if mux_control_select() fails.
+ *
+ * Return: 0 when the mux-control state has the requested state or a negative
+ * errno on error.
+ */
+int mux_control_select(struct mux_control *mux, unsigned int state)
+{
+ int ret;
+
+ ret = down_killable(&mux->lock);
+ if (ret < 0)
+ return ret;
+
+ ret = __mux_control_select(mux, state);
+
+ if (ret < 0)
+ up(&mux->lock);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(mux_control_select);
+
+/**
+ * mux_control_try_select() - Try to select the given multiplexer state.
+ * @mux: The mux-control to request a change of state from.
+ * @state: The new requested state.
+ *
+ * On successfully selecting the mux-control state, it will be locked until
+ * mux_control_deselect() called.
+ *
+ * Therefore, make sure to call mux_control_deselect() when the operation is
+ * complete and the mux-control is free for others to use, but do not call
+ * mux_control_deselect() if mux_control_try_select() fails.
+ *
+ * Return: 0 when the mux-control state has the requested state or a negative
+ * errno on error. Specifically -EBUSY if the mux-control is contended.
+ */
+int mux_control_try_select(struct mux_control *mux, unsigned int state)
+{
+ int ret;
+
+ if (down_trylock(&mux->lock))
+ return -EBUSY;
+
+ ret = __mux_control_select(mux, state);
+
+ if (ret < 0)
+ up(&mux->lock);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(mux_control_try_select);
+
+/**
+ * mux_control_deselect() - Deselect the previously selected multiplexer state.
+ * @mux: The mux-control to deselect.
+ *
+ * It is required that a single call is made to mux_control_deselect() for
+ * each and every successful call made to either of mux_control_select() or
+ * mux_control_try_select().
+ *
+ * Return: 0 on success and a negative errno on error. An error can only
+ * occur if the mux has an idle state. Note that even if an error occurs, the
+ * mux-control is unlocked and is thus free for the next access.
+ */
+int mux_control_deselect(struct mux_control *mux)
+{
+ int ret = 0;
+
+ if (mux->idle_state != MUX_IDLE_AS_IS &&
+ mux->idle_state != mux->cached_state)
+ ret = mux_control_set(mux, mux->idle_state);
+
+ up(&mux->lock);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(mux_control_deselect);
+
+static int of_dev_node_match(struct device *dev, const void *data)
+{
+ return dev->of_node == data;
+}
+
+static struct mux_chip *of_find_mux_chip_by_node(struct device_node *np)
+{
+ struct device *dev;
+
+ dev = class_find_device(&mux_class, NULL, np, of_dev_node_match);
+
+ return dev ? to_mux_chip(dev) : NULL;
+}
+
+/**
+ * mux_control_get() - Get the mux-control for a device.
+ * @dev: The device that needs a mux-control.
+ * @mux_name: The name identifying the mux-control.
+ *
+ * Return: A pointer to the mux-control, or an ERR_PTR with a negative errno.
+ */
+struct mux_control *mux_control_get(struct device *dev, const char *mux_name)
+{
+ struct device_node *np = dev->of_node;
+ struct of_phandle_args args;
+ struct mux_chip *mux_chip;
+ unsigned int controller;
+ int index = 0;
+ int ret;
+
+ if (mux_name) {
+ index = of_property_match_string(np, "mux-control-names",
+ mux_name);
+ if (index < 0) {
+ dev_err(dev, "mux controller '%s' not found\n",
+ mux_name);
+ return ERR_PTR(index);
+ }
+ }
+
+ ret = of_parse_phandle_with_args(np,
+ "mux-controls", "#mux-control-cells",
+ index, &args);
+ if (ret) {
+ dev_err(dev, "%s: failed to get mux-control %s(%i)\n",
+ np->full_name, mux_name ?: "", index);
+ return ERR_PTR(ret);
+ }
+
+ mux_chip = of_find_mux_chip_by_node(args.np);
+ of_node_put(args.np);
+ if (!mux_chip)
+ return ERR_PTR(-EPROBE_DEFER);
+
+ if (args.args_count > 1 ||
+ (!args.args_count && (mux_chip->controllers > 1))) {
+ dev_err(dev, "%s: wrong #mux-control-cells for %s\n",
+ np->full_name, args.np->full_name);
+ return ERR_PTR(-EINVAL);
+ }
+
+ controller = 0;
+ if (args.args_count)
+ controller = args.args[0];
+
+ if (controller >= mux_chip->controllers) {
+ dev_err(dev, "%s: bad mux controller %u specified in %s\n",
+ np->full_name, controller, args.np->full_name);
+ return ERR_PTR(-EINVAL);
+ }
+
+ get_device(&mux_chip->dev);
+ return &mux_chip->mux[controller];
+}
+EXPORT_SYMBOL_GPL(mux_control_get);
+
+/**
+ * mux_control_put() - Put away the mux-control for good.
+ * @mux: The mux-control to put away.
+ *
+ * mux_control_put() reverses the effects of mux_control_get().
+ */
+void mux_control_put(struct mux_control *mux)
+{
+ put_device(&mux->chip->dev);
+}
+EXPORT_SYMBOL_GPL(mux_control_put);
+
+static void devm_mux_control_release(struct device *dev, void *res)
+{
+ struct mux_control *mux = *(struct mux_control **)res;
+
+ mux_control_put(mux);
+}
+
+/**
+ * devm_mux_control_get() - Get the mux-control for a device, with resource
+ * management.
+ * @dev: The device that needs a mux-control.
+ * @mux_name: The name identifying the mux-control.
+ *
+ * Return: Pointer to the mux-control, or an ERR_PTR with a negative errno.
+ */
+struct mux_control *devm_mux_control_get(struct device *dev,
+ const char *mux_name)
+{
+ struct mux_control **ptr, *mux;
+
+ ptr = devres_alloc(devm_mux_control_release, sizeof(*ptr), GFP_KERNEL);
+ if (!ptr)
+ return ERR_PTR(-ENOMEM);
+
+ mux = mux_control_get(dev, mux_name);
+ if (IS_ERR(mux)) {
+ devres_free(ptr);
+ return mux;
+ }
+
+ *ptr = mux;
+ devres_add(dev, ptr);
+
+ return mux;
+}
+EXPORT_SYMBOL_GPL(devm_mux_control_get);
+
+/*
+ * Using subsys_initcall instead of module_init here to try to ensure - for
+ * the non-modular case - that the subsystem is initialized when mux consumers
+ * and mux controllers start to use it.
+ * For the modular case, the ordering is ensured with module dependencies.
+ */
+subsys_initcall(mux_init);
+module_exit(mux_exit);
+
+MODULE_DESCRIPTION("Multiplexer subsystem");
+MODULE_AUTHOR("Peter Rosin <peda@axentia.se>");
+MODULE_LICENSE("GPL v2");
--- /dev/null
+/*
+ * GPIO-controlled multiplexer driver
+ *
+ * Copyright (C) 2017 Axentia Technologies AB
+ *
+ * Author: Peter Rosin <peda@axentia.se>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/err.h>
+#include <linux/gpio/consumer.h>
+#include <linux/module.h>
+#include <linux/mux/driver.h>
+#include <linux/of_platform.h>
+#include <linux/platform_device.h>
+#include <linux/property.h>
+
+struct mux_gpio {
+ struct gpio_descs *gpios;
+ int *val;
+};
+
+static int mux_gpio_set(struct mux_control *mux, int state)
+{
+ struct mux_gpio *mux_gpio = mux_chip_priv(mux->chip);
+ int i;
+
+ for (i = 0; i < mux_gpio->gpios->ndescs; i++)
+ mux_gpio->val[i] = (state >> i) & 1;
+
+ gpiod_set_array_value_cansleep(mux_gpio->gpios->ndescs,
+ mux_gpio->gpios->desc,
+ mux_gpio->val);
+
+ return 0;
+}
+
+static const struct mux_control_ops mux_gpio_ops = {
+ .set = mux_gpio_set,
+};
+
+static const struct of_device_id mux_gpio_dt_ids[] = {
+ { .compatible = "gpio-mux", },
+ { /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, mux_gpio_dt_ids);
+
+static int mux_gpio_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct mux_chip *mux_chip;
+ struct mux_gpio *mux_gpio;
+ int pins;
+ s32 idle_state;
+ int ret;
+
+ pins = gpiod_count(dev, "mux");
+ if (pins < 0)
+ return pins;
+
+ mux_chip = devm_mux_chip_alloc(dev, 1, sizeof(*mux_gpio) +
+ pins * sizeof(*mux_gpio->val));
+ if (IS_ERR(mux_chip))
+ return PTR_ERR(mux_chip);
+
+ mux_gpio = mux_chip_priv(mux_chip);
+ mux_gpio->val = (int *)(mux_gpio + 1);
+ mux_chip->ops = &mux_gpio_ops;
+
+ mux_gpio->gpios = devm_gpiod_get_array(dev, "mux", GPIOD_OUT_LOW);
+ if (IS_ERR(mux_gpio->gpios)) {
+ ret = PTR_ERR(mux_gpio->gpios);
+ if (ret != -EPROBE_DEFER)
+ dev_err(dev, "failed to get gpios\n");
+ return ret;
+ }
+ WARN_ON(pins != mux_gpio->gpios->ndescs);
+ mux_chip->mux->states = 1 << pins;
+
+ ret = device_property_read_u32(dev, "idle-state", (u32 *)&idle_state);
+ if (ret >= 0 && idle_state != MUX_IDLE_AS_IS) {
+ if (idle_state < 0 || idle_state >= mux_chip->mux->states) {
+ dev_err(dev, "invalid idle-state %u\n", idle_state);
+ return -EINVAL;
+ }
+
+ mux_chip->mux->idle_state = idle_state;
+ }
+
+ ret = devm_mux_chip_register(dev, mux_chip);
+ if (ret < 0)
+ return ret;
+
+ dev_info(dev, "%u-way mux-controller registered\n",
+ mux_chip->mux->states);
+
+ return 0;
+}
+
+static struct platform_driver mux_gpio_driver = {
+ .driver = {
+ .name = "gpio-mux",
+ .of_match_table = of_match_ptr(mux_gpio_dt_ids),
+ },
+ .probe = mux_gpio_probe,
+};
+module_platform_driver(mux_gpio_driver);
+
+MODULE_DESCRIPTION("GPIO-controlled multiplexer driver");
+MODULE_AUTHOR("Peter Rosin <peda@axentia.se>");
+MODULE_LICENSE("GPL v2");
--- /dev/null
+/*
+ * MMIO register bitfield-controlled multiplexer driver
+ *
+ * Copyright (C) 2017 Pengutronix, Philipp Zabel <kernel@pengutronix.de>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/bitops.h>
+#include <linux/err.h>
+#include <linux/mfd/syscon.h>
+#include <linux/module.h>
+#include <linux/mux/driver.h>
+#include <linux/of_platform.h>
+#include <linux/platform_device.h>
+#include <linux/property.h>
+#include <linux/regmap.h>
+
+static int mux_mmio_set(struct mux_control *mux, int state)
+{
+ struct regmap_field **fields = mux_chip_priv(mux->chip);
+
+ return regmap_field_write(fields[mux_control_get_index(mux)], state);
+}
+
+static const struct mux_control_ops mux_mmio_ops = {
+ .set = mux_mmio_set,
+};
+
+static const struct of_device_id mux_mmio_dt_ids[] = {
+ { .compatible = "mmio-mux", },
+ { /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, mux_mmio_dt_ids);
+
+static int mux_mmio_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct device_node *np = dev->of_node;
+ struct regmap_field **fields;
+ struct mux_chip *mux_chip;
+ struct regmap *regmap;
+ int num_fields;
+ int ret;
+ int i;
+
+ regmap = syscon_node_to_regmap(np->parent);
+ if (IS_ERR(regmap)) {
+ ret = PTR_ERR(regmap);
+ dev_err(dev, "failed to get regmap: %d\n", ret);
+ return ret;
+ }
+
+ ret = of_property_count_u32_elems(np, "mux-reg-masks");
+ if (ret == 0 || ret % 2)
+ ret = -EINVAL;
+ if (ret < 0) {
+ dev_err(dev, "mux-reg-masks property missing or invalid: %d\n",
+ ret);
+ return ret;
+ }
+ num_fields = ret / 2;
+
+ mux_chip = devm_mux_chip_alloc(dev, num_fields, num_fields *
+ sizeof(*fields));
+ if (IS_ERR(mux_chip))
+ return PTR_ERR(mux_chip);
+
+ fields = mux_chip_priv(mux_chip);
+
+ for (i = 0; i < num_fields; i++) {
+ struct mux_control *mux = &mux_chip->mux[i];
+ struct reg_field field;
+ s32 idle_state = MUX_IDLE_AS_IS;
+ u32 reg, mask;
+ int bits;
+
+ ret = of_property_read_u32_index(np, "mux-reg-masks",
+ 2 * i, ®);
+ if (!ret)
+ ret = of_property_read_u32_index(np, "mux-reg-masks",
+ 2 * i + 1, &mask);
+ if (ret < 0) {
+ dev_err(dev, "bitfield %d: failed to read mux-reg-masks property: %d\n",
+ i, ret);
+ return ret;
+ }
+
+ field.reg = reg;
+ field.msb = fls(mask) - 1;
+ field.lsb = ffs(mask) - 1;
+
+ if (mask != GENMASK(field.msb, field.lsb)) {
+ dev_err(dev, "bitfield %d: invalid mask 0x%x\n",
+ i, mask);
+ return -EINVAL;
+ }
+
+ fields[i] = devm_regmap_field_alloc(dev, regmap, field);
+ if (IS_ERR(fields[i])) {
+ ret = PTR_ERR(fields[i]);
+ dev_err(dev, "bitfield %d: failed allocate: %d\n",
+ i, ret);
+ return ret;
+ }
+
+ bits = 1 + field.msb - field.lsb;
+ mux->states = 1 << bits;
+
+ of_property_read_u32_index(np, "idle-states", i,
+ (u32 *)&idle_state);
+ if (idle_state != MUX_IDLE_AS_IS) {
+ if (idle_state < 0 || idle_state >= mux->states) {
+ dev_err(dev, "bitfield: %d: out of range idle state %d\n",
+ i, idle_state);
+ return -EINVAL;
+ }
+
+ mux->idle_state = idle_state;
+ }
+ }
+
+ mux_chip->ops = &mux_mmio_ops;
+
+ return devm_mux_chip_register(dev, mux_chip);
+}
+
+static struct platform_driver mux_mmio_driver = {
+ .driver = {
+ .name = "mmio-mux",
+ .of_match_table = of_match_ptr(mux_mmio_dt_ids),
+ },
+ .probe = mux_mmio_probe,
+};
+module_platform_driver(mux_mmio_driver);
+
+MODULE_DESCRIPTION("MMIO register bitfield-controlled multiplexer driver");
+MODULE_AUTHOR("Philipp Zabel <p.zabel@pengutronix.de>");
+MODULE_LICENSE("GPL v2");
struct net_device *dev = dev_id;
struct arcnet_local *lp;
int recbuf, status, diagstatus, didsomething, boguscount;
+ unsigned long flags;
int retval = IRQ_NONE;
arc_printk(D_DURING, dev, "\n");
lp = netdev_priv(dev);
BUG_ON(!lp);
- spin_lock(&lp->lock);
+ spin_lock_irqsave(&lp->lock, flags);
/* RESET flag was enabled - if device is not running, we must
* clear it right away (but nothing else).
if (lp->hw.status(dev) & RESETflag)
lp->hw.command(dev, CFLAGScmd | RESETclear);
lp->hw.intmask(dev, 0);
- spin_unlock(&lp->lock);
+ spin_unlock_irqrestore(&lp->lock, flags);
return retval;
}
udelay(1);
lp->hw.intmask(dev, lp->intmask);
- spin_unlock(&lp->lock);
+ spin_unlock_irqrestore(&lp->lock, flags);
return retval;
}
EXPORT_SYMBOL(arcnet_interrupt);
ackpkt->soft.cap.proto = 0; /* using protocol 0 for acknowledge */
ackpkt->soft.cap.mes.ack = acked;
- arc_printk(D_PROTO, dev, "Ackknowledge for cap packet %x.\n",
+ arc_printk(D_PROTO, dev, "Acknowledge for cap packet %x.\n",
*((int *)&ackpkt->soft.cap.cookie[0]));
ackskb->protocol = cpu_to_be16(ETH_P_ARCNET);
for (i = 0; i < ci->devcount; i++) {
struct com20020_pci_channel_map *cm = &ci->chan_map_tbl[i];
struct com20020_dev *card;
+ int dev_id_mask = 0xf;
dev = alloc_arcdev(device);
if (!dev) {
arcnet_outb(0x00, ioaddr, COM20020_REG_W_COMMAND);
arcnet_inb(ioaddr, COM20020_REG_R_DIAGSTAT);
+ SET_NETDEV_DEV(dev, &pdev->dev);
dev->base_addr = ioaddr;
dev->dev_addr[0] = node;
dev->irq = pdev->irq;
/* Get the dev_id from the PLX rotary coder */
if (!strncmp(ci->name, "EAE PLX-PCI MA1", 15))
- dev->dev_id = 0xc;
- dev->dev_id ^= inb(priv->misc + ci->rotary) >> 4;
+ dev_id_mask = 0x3;
+ dev->dev_id = (inb(priv->misc + ci->rotary) >> 4) & dev_id_mask;
snprintf(dev->name, sizeof(dev->name), "arc%d-%d", dev->dev_id, i);
return -ENODEV;
}
- dev->base_addr = ioaddr;
-
arc_printk(D_NORMAL, dev, "%s: station %02Xh found at %03lXh, IRQ %d.\n",
lp->card_name, dev->dev_addr[0], dev->base_addr, dev->irq);
static spinlock_t cfspi_list_lock;
/* SPI uplink head alignment. */
-static ssize_t show_up_head_align(struct device_driver *driver, char *buf)
+static ssize_t up_head_align_show(struct device_driver *driver, char *buf)
{
return sprintf(buf, "%d\n", spi_up_head_align);
}
-static DRIVER_ATTR(up_head_align, S_IRUSR, show_up_head_align, NULL);
+static DRIVER_ATTR_RO(up_head_align);
/* SPI uplink tail alignment. */
-static ssize_t show_up_tail_align(struct device_driver *driver, char *buf)
+static ssize_t up_tail_align_show(struct device_driver *driver, char *buf)
{
return sprintf(buf, "%d\n", spi_up_tail_align);
}
-static DRIVER_ATTR(up_tail_align, S_IRUSR, show_up_tail_align, NULL);
+static DRIVER_ATTR_RO(up_tail_align);
/* SPI downlink head alignment. */
-static ssize_t show_down_head_align(struct device_driver *driver, char *buf)
+static ssize_t down_head_align_show(struct device_driver *driver, char *buf)
{
return sprintf(buf, "%d\n", spi_down_head_align);
}
-static DRIVER_ATTR(down_head_align, S_IRUSR, show_down_head_align, NULL);
+static DRIVER_ATTR_RO(down_head_align);
/* SPI downlink tail alignment. */
-static ssize_t show_down_tail_align(struct device_driver *driver, char *buf)
+static ssize_t down_tail_align_show(struct device_driver *driver, char *buf)
{
return sprintf(buf, "%d\n", spi_down_tail_align);
}
-static DRIVER_ATTR(down_tail_align, S_IRUSR, show_down_tail_align, NULL);
+static DRIVER_ATTR_RO(down_tail_align);
/* SPI frame alignment. */
-static ssize_t show_frame_align(struct device_driver *driver, char *buf)
+static ssize_t frame_align_show(struct device_driver *driver, char *buf)
{
return sprintf(buf, "%d\n", spi_frm_align);
}
-static DRIVER_ATTR(frame_align, S_IRUSR, show_frame_align, NULL);
+static DRIVER_ATTR_RO(frame_align);
int cfspi_xmitfrm(struct cfspi *cfspi, u8 *buf, size_t len)
{
} else {
/* If no mc addresses are required, flush the configuration */
rc = bnx2x_config_mcast(bp, &rparam, BNX2X_MCAST_CMD_DEL);
- if (rc)
+ if (rc < 0)
BNX2X_ERR("Failed to clear multicast configuration %d\n",
rc);
}
cp_cons = NEXT_CMP(cp_cons);
}
- if (unlikely(agg_bufs > MAX_SKB_FRAGS)) {
+ if (unlikely(agg_bufs > MAX_SKB_FRAGS || TPA_END_ERRORS(tpa_end1))) {
bnxt_abort_tpa(bp, bnapi, cp_cons, agg_bufs);
- netdev_warn(bp->dev, "TPA frags %d exceeded MAX_SKB_FRAGS %d\n",
- agg_bufs, (int)MAX_SKB_FRAGS);
+ if (agg_bufs > MAX_SKB_FRAGS)
+ netdev_warn(bp->dev, "TPA frags %d exceeded MAX_SKB_FRAGS %d\n",
+ agg_bufs, (int)MAX_SKB_FRAGS);
return NULL;
}
return rc;
}
+/* In netpoll mode, if we are using a combined completion ring, we need to
+ * discard the rx packets and recycle the buffers.
+ */
+static int bnxt_force_rx_discard(struct bnxt *bp, struct bnxt_napi *bnapi,
+ u32 *raw_cons, u8 *event)
+{
+ struct bnxt_cp_ring_info *cpr = &bnapi->cp_ring;
+ u32 tmp_raw_cons = *raw_cons;
+ struct rx_cmp_ext *rxcmp1;
+ struct rx_cmp *rxcmp;
+ u16 cp_cons;
+ u8 cmp_type;
+
+ cp_cons = RING_CMP(tmp_raw_cons);
+ rxcmp = (struct rx_cmp *)
+ &cpr->cp_desc_ring[CP_RING(cp_cons)][CP_IDX(cp_cons)];
+
+ tmp_raw_cons = NEXT_RAW_CMP(tmp_raw_cons);
+ cp_cons = RING_CMP(tmp_raw_cons);
+ rxcmp1 = (struct rx_cmp_ext *)
+ &cpr->cp_desc_ring[CP_RING(cp_cons)][CP_IDX(cp_cons)];
+
+ if (!RX_CMP_VALID(rxcmp1, tmp_raw_cons))
+ return -EBUSY;
+
+ cmp_type = RX_CMP_TYPE(rxcmp);
+ if (cmp_type == CMP_TYPE_RX_L2_CMP) {
+ rxcmp1->rx_cmp_cfa_code_errors_v2 |=
+ cpu_to_le32(RX_CMPL_ERRORS_CRC_ERROR);
+ } else if (cmp_type == CMP_TYPE_RX_L2_TPA_END_CMP) {
+ struct rx_tpa_end_cmp_ext *tpa_end1;
+
+ tpa_end1 = (struct rx_tpa_end_cmp_ext *)rxcmp1;
+ tpa_end1->rx_tpa_end_cmp_errors_v2 |=
+ cpu_to_le32(RX_TPA_END_CMP_ERRORS);
+ }
+ return bnxt_rx_pkt(bp, bnapi, raw_cons, event);
+}
+
#define BNXT_GET_EVENT_PORT(data) \
((data) & \
ASYNC_EVENT_CMPL_PORT_CONN_NOT_ALLOWED_EVENT_DATA1_PORT_ID_MASK)
if (unlikely(tx_pkts > bp->tx_wake_thresh))
rx_pkts = budget;
} else if ((TX_CMP_TYPE(txcmp) & 0x30) == 0x10) {
- rc = bnxt_rx_pkt(bp, bnapi, &raw_cons, &event);
+ if (likely(budget))
+ rc = bnxt_rx_pkt(bp, bnapi, &raw_cons, &event);
+ else
+ rc = bnxt_force_rx_discard(bp, bnapi, &raw_cons,
+ &event);
if (likely(rc >= 0))
rx_pkts += rc;
else if (rc == -EBUSY) /* partial completion */
struct bnxt *bp = netdev_priv(dev);
int i;
- for (i = 0; i < bp->cp_nr_rings; i++) {
- struct bnxt_irq *irq = &bp->irq_tbl[i];
+ /* Only process tx rings/combined rings in netpoll mode. */
+ for (i = 0; i < bp->tx_nr_rings; i++) {
+ struct bnxt_tx_ring_info *txr = &bp->tx_ring[i];
- disable_irq(irq->vector);
- irq->handler(irq->vector, bp->bnapi[i]);
- enable_irq(irq->vector);
+ napi_schedule(&txr->bnapi->napi);
}
}
#endif
__le32 rx_tpa_end_cmp_errors_v2;
#define RX_TPA_END_CMP_V2 (0x1 << 0)
- #define RX_TPA_END_CMP_ERRORS (0x7fff << 1)
+ #define RX_TPA_END_CMP_ERRORS (0x3 << 1)
#define RX_TPA_END_CMPL_ERRORS_SHIFT 1
u32 rx_tpa_end_cmp_start_opaque;
};
+#define TPA_END_ERRORS(rx_tpa_end_ext) \
+ ((rx_tpa_end_ext)->rx_tpa_end_cmp_errors_v2 & \
+ cpu_to_le32(RX_TPA_END_CMP_ERRORS))
+
#define DB_IDX_MASK 0xffffff
#define DB_IDX_VALID (0x1 << 26)
#define DB_IRQ_DIS (0x1 << 27)
sleep_cond(wait_queue_head_t *wait_queue, int *condition)
{
int errno = 0;
- wait_queue_t we;
+ wait_queue_entry_t we;
init_waitqueue_entry(&we, current);
add_wait_queue(wait_queue, &we);
int *condition,
int timeout)
{
- wait_queue_t we;
+ wait_queue_entry_t we;
init_waitqueue_entry(&we, current);
add_wait_queue(wait_queue, &we);
{
int err;
+ mutex_lock(&uld_mutex);
err = setup_sge_queues(adap);
if (err)
- goto out;
+ goto rel_lock;
err = setup_rss(adap);
if (err)
goto freeq;
goto irq_err;
}
- mutex_lock(&uld_mutex);
enable_rx(adap);
t4_sge_start(adap);
t4_intr_enable(adap);
#endif
/* Initialize hash mac addr list*/
INIT_LIST_HEAD(&adap->mac_hlist);
- out:
return err;
+
irq_err:
dev_err(adap->pdev_dev, "request_irq failed, err %d\n", err);
freeq:
t4_free_sge_resources(adap);
- goto out;
+ rel_lock:
+ mutex_unlock(&uld_mutex);
+ return err;
}
static void cxgb_down(struct adapter *adapter)
priv->buf_layout[TX].priv_data_size = DPAA_TX_PRIV_DATA_SIZE; /* Tx */
/* device used for DMA mapping */
- arch_setup_dma_ops(dev, 0, 0, NULL, false);
+ set_dma_ops(dev, get_dma_ops(&pdev->dev));
err = dma_coerce_mask_and_coherent(dev, DMA_BIT_MASK(40));
if (err) {
dev_err(dev, "dma_coerce_mask_and_coherent() failed\n");
tristate "FMan support"
depends on FSL_SOC || ARCH_LAYERSCAPE || COMPILE_TEST
select GENERIC_ALLOCATOR
+ depends on HAS_DMA
select PHYLIB
default n
help
goto no_mem;
}
+ set_dma_ops(&pdev->dev, get_dma_ops(priv->dev));
+
ret = platform_device_add_data(pdev, &data, sizeof(data));
if (ret)
goto err;
HNS_ROCE_RESET_FUNC = 0x7,
};
-const u8 hns_dsaf_acpi_dsm_uuid[] = {
- 0x1A, 0xAA, 0x85, 0x1A, 0x93, 0xE2, 0x5E, 0x41,
- 0x8E, 0x28, 0x8D, 0x69, 0x0A, 0x0F, 0x82, 0x0A
-};
+static const guid_t hns_dsaf_acpi_dsm_guid =
+ GUID_INIT(0x1A85AA1A, 0xE293, 0x415E,
+ 0x8E, 0x28, 0x8D, 0x69, 0x0A, 0x0F, 0x82, 0x0A);
static void dsaf_write_sub(struct dsaf_device *dsaf_dev, u32 reg, u32 val)
{
argv4.package.elements = obj_args;
obj = acpi_evaluate_dsm(ACPI_HANDLE(dsaf_dev->dev),
- hns_dsaf_acpi_dsm_uuid, 0, op_type, &argv4);
+ &hns_dsaf_acpi_dsm_guid, 0, op_type, &argv4);
if (!obj) {
dev_warn(dsaf_dev->dev, "reset port_type%d port%d fail!",
port_type, port);
argv4.package.elements = &obj_args,
obj = acpi_evaluate_dsm(ACPI_HANDLE(mac_cb->dev),
- hns_dsaf_acpi_dsm_uuid, 0,
+ &hns_dsaf_acpi_dsm_guid, 0,
HNS_OP_GET_PORT_TYPE_FUNC, &argv4);
if (!obj || obj->type != ACPI_TYPE_INTEGER)
argv4.package.elements = &obj_args,
obj = acpi_evaluate_dsm(ACPI_HANDLE(mac_cb->dev),
- hns_dsaf_acpi_dsm_uuid, 0,
+ &hns_dsaf_acpi_dsm_guid, 0,
HNS_OP_GET_SFP_STAT_FUNC, &argv4);
if (!obj || obj->type != ACPI_TYPE_INTEGER)
argv4.package.elements = obj_args;
obj = acpi_evaluate_dsm(ACPI_HANDLE(mac_cb->dsaf_dev->dev),
- hns_dsaf_acpi_dsm_uuid, 0,
+ &hns_dsaf_acpi_dsm_guid, 0,
HNS_OP_SERDES_LP_FUNC, &argv4);
if (!obj) {
dev_warn(mac_cb->dsaf_dev->dev, "set port%d serdes lp fail!",
/* Force 1000M Link, Default is 0x0200 */
phy_write(phy_dev, 7, 0x20C);
- phy_write(phy_dev, HNS_PHY_PAGE_REG, 0);
- /* Enable PHY loop-back */
+ /* Powerup Fiber */
+ phy_write(phy_dev, HNS_PHY_PAGE_REG, 1);
+ val = phy_read(phy_dev, COPPER_CONTROL_REG);
+ val &= ~PHY_POWER_DOWN;
+ phy_write(phy_dev, COPPER_CONTROL_REG, val);
+
+ /* Enable Phy Loopback */
+ phy_write(phy_dev, HNS_PHY_PAGE_REG, 0);
val = phy_read(phy_dev, COPPER_CONTROL_REG);
val |= PHY_LOOP_BACK;
val &= ~PHY_POWER_DOWN;
phy_write(phy_dev, HNS_PHY_PAGE_REG, 0xFA);
phy_write(phy_dev, 1, 0x400);
phy_write(phy_dev, 7, 0x200);
+
+ phy_write(phy_dev, HNS_PHY_PAGE_REG, 1);
+ val = phy_read(phy_dev, COPPER_CONTROL_REG);
+ val |= PHY_POWER_DOWN;
+ phy_write(phy_dev, COPPER_CONTROL_REG, val);
+
phy_write(phy_dev, HNS_PHY_PAGE_REG, 0);
phy_write(phy_dev, 9, 0xF00);
return ret;
}
-static ssize_t ehea_show_capabilities(struct device_driver *drv,
- char *buf)
+static ssize_t capabilities_show(struct device_driver *drv, char *buf)
{
return sprintf(buf, "%d", EHEA_CAPABILITIES);
}
-static DRIVER_ATTR(capabilities, S_IRUSR | S_IRGRP | S_IROTH,
- ehea_show_capabilities, NULL);
+static DRIVER_ATTR_RO(capabilities);
static int __init ehea_module_init(void)
{
SOF_TIMESTAMPING_RX_HARDWARE |
SOF_TIMESTAMPING_RAW_HARDWARE;
- info->tx_types = (BIT(1) << HWTSTAMP_TX_OFF) |
- (BIT(1) << HWTSTAMP_TX_ON);
+ info->tx_types = BIT(HWTSTAMP_TX_OFF) |
+ BIT(HWTSTAMP_TX_ON);
- info->rx_filters = (BIT(1) << HWTSTAMP_FILTER_NONE) |
- (BIT(1) << HWTSTAMP_FILTER_ALL);
+ info->rx_filters = BIT(HWTSTAMP_FILTER_NONE) |
+ BIT(HWTSTAMP_FILTER_ALL);
return 0;
}
return netdev;
err_cleanup_nic:
- profile->cleanup(priv);
+ if (profile->cleanup)
+ profile->cleanup(priv);
free_netdev(netdev);
return NULL;
params->tx_max_inline = mlx5e_get_max_inline_cap(mdev);
params->num_tc = 1;
params->lro_wqe_sz = MLX5E_PARAMS_DEFAULT_LRO_WQE_SZ;
+
+ mlx5_query_min_inline(mdev, ¶ms->tx_min_inline_mode);
}
static void mlx5e_build_rep_netdev(struct net_device *netdev)
{MLX5_ACTION_IN_FIELD_OUT_SMAC_15_0, 2, offsetof(struct pedit_headers, eth.h_source[4])},
{MLX5_ACTION_IN_FIELD_OUT_ETHERTYPE, 2, offsetof(struct pedit_headers, eth.h_proto)},
- {MLX5_ACTION_IN_FIELD_OUT_IP_DSCP, 1, offsetof(struct pedit_headers, ip4.tos)},
{MLX5_ACTION_IN_FIELD_OUT_IP_TTL, 1, offsetof(struct pedit_headers, ip4.ttl)},
{MLX5_ACTION_IN_FIELD_OUT_SIPV4, 4, offsetof(struct pedit_headers, ip4.saddr)},
{MLX5_ACTION_IN_FIELD_OUT_DIPV4, 4, offsetof(struct pedit_headers, ip4.daddr)},
return 0;
}
-int mlx5_devlink_eswitch_mode_set(struct devlink *devlink, u16 mode)
+static int mlx5_devlink_eswitch_check(struct devlink *devlink)
{
- struct mlx5_core_dev *dev;
- u16 cur_mlx5_mode, mlx5_mode = 0;
+ struct mlx5_core_dev *dev = devlink_priv(devlink);
- dev = devlink_priv(devlink);
+ if (MLX5_CAP_GEN(dev, port_type) != MLX5_CAP_PORT_TYPE_ETH)
+ return -EOPNOTSUPP;
if (!MLX5_CAP_GEN(dev, vport_group_manager))
return -EOPNOTSUPP;
- cur_mlx5_mode = dev->priv.eswitch->mode;
-
- if (cur_mlx5_mode == SRIOV_NONE)
+ if (dev->priv.eswitch->mode == SRIOV_NONE)
return -EOPNOTSUPP;
+ return 0;
+}
+
+int mlx5_devlink_eswitch_mode_set(struct devlink *devlink, u16 mode)
+{
+ struct mlx5_core_dev *dev = devlink_priv(devlink);
+ u16 cur_mlx5_mode, mlx5_mode = 0;
+ int err;
+
+ err = mlx5_devlink_eswitch_check(devlink);
+ if (err)
+ return err;
+
+ cur_mlx5_mode = dev->priv.eswitch->mode;
+
if (esw_mode_from_devlink(mode, &mlx5_mode))
return -EINVAL;
int mlx5_devlink_eswitch_mode_get(struct devlink *devlink, u16 *mode)
{
- struct mlx5_core_dev *dev;
-
- dev = devlink_priv(devlink);
-
- if (!MLX5_CAP_GEN(dev, vport_group_manager))
- return -EOPNOTSUPP;
+ struct mlx5_core_dev *dev = devlink_priv(devlink);
+ int err;
- if (dev->priv.eswitch->mode == SRIOV_NONE)
- return -EOPNOTSUPP;
+ err = mlx5_devlink_eswitch_check(devlink);
+ if (err)
+ return err;
return esw_mode_to_devlink(dev->priv.eswitch->mode, mode);
}
{
struct mlx5_core_dev *dev = devlink_priv(devlink);
struct mlx5_eswitch *esw = dev->priv.eswitch;
- int num_vports = esw->enabled_vports;
int err, vport;
u8 mlx5_mode;
- if (!MLX5_CAP_GEN(dev, vport_group_manager))
- return -EOPNOTSUPP;
-
- if (esw->mode == SRIOV_NONE)
- return -EOPNOTSUPP;
+ err = mlx5_devlink_eswitch_check(devlink);
+ if (err)
+ return err;
switch (MLX5_CAP_ETH(dev, wqe_inline_mode)) {
case MLX5_CAP_INLINE_MODE_NOT_REQUIRED:
if (err)
goto out;
- for (vport = 1; vport < num_vports; vport++) {
+ for (vport = 1; vport < esw->enabled_vports; vport++) {
err = mlx5_modify_nic_vport_min_inline(dev, vport, mlx5_mode);
if (err) {
esw_warn(dev, "Failed to set min inline on vport %d\n",
{
struct mlx5_core_dev *dev = devlink_priv(devlink);
struct mlx5_eswitch *esw = dev->priv.eswitch;
+ int err;
- if (!MLX5_CAP_GEN(dev, vport_group_manager))
- return -EOPNOTSUPP;
-
- if (esw->mode == SRIOV_NONE)
- return -EOPNOTSUPP;
+ err = mlx5_devlink_eswitch_check(devlink);
+ if (err)
+ return err;
return esw_inline_mode_to_devlink(esw->offloads.inline_mode, mode);
}
struct mlx5_eswitch *esw = dev->priv.eswitch;
int err;
- if (!MLX5_CAP_GEN(dev, vport_group_manager))
- return -EOPNOTSUPP;
-
- if (esw->mode == SRIOV_NONE)
- return -EOPNOTSUPP;
+ err = mlx5_devlink_eswitch_check(devlink);
+ if (err)
+ return err;
if (encap != DEVLINK_ESWITCH_ENCAP_MODE_NONE &&
(!MLX5_CAP_ESW_FLOWTABLE_FDB(dev, encap) ||
{
struct mlx5_core_dev *dev = devlink_priv(devlink);
struct mlx5_eswitch *esw = dev->priv.eswitch;
+ int err;
- if (!MLX5_CAP_GEN(dev, vport_group_manager))
- return -EOPNOTSUPP;
-
- if (esw->mode == SRIOV_NONE)
- return -EOPNOTSUPP;
+ err = mlx5_devlink_eswitch_check(devlink);
+ if (err)
+ return err;
*encap = esw->offloads.encap;
return 0;
},
};
-#define FW_INIT_TIMEOUT_MILI 2000
-#define FW_INIT_WAIT_MS 2
+#define FW_INIT_TIMEOUT_MILI 2000
+#define FW_INIT_WAIT_MS 2
+#define FW_PRE_INIT_TIMEOUT_MILI 10000
static int wait_fw_init(struct mlx5_core_dev *dev, u32 max_wait_mili)
{
*/
dev->state = MLX5_DEVICE_STATE_UP;
+ /* wait for firmware to accept initialization segments configurations
+ */
+ err = wait_fw_init(dev, FW_PRE_INIT_TIMEOUT_MILI);
+ if (err) {
+ dev_err(&dev->pdev->dev, "Firmware over %d MS in pre-initializing state, aborting\n",
+ FW_PRE_INIT_TIMEOUT_MILI);
+ goto out;
+ }
+
err = mlx5_cmd_init(dev);
if (err) {
dev_err(&pdev->dev, "Failed initializing command interface, aborting\n");
struct mlxsw_sp *mlxsw_sp = mlxsw_sp_lower_get(vlan_dev);
u16 vid = vlan_dev_vlan_id(vlan_dev);
+ if (netif_is_bridge_port(vlan_dev))
+ return 0;
+
if (mlxsw_sp_port_dev_check(real_dev))
return mlxsw_sp_inetaddr_vport_event(vlan_dev, real_dev, event,
vid);
*index = entry->index;
resolved = false;
} else if (removing) {
- ofdpa_neigh_del(trans, found);
*index = found->index;
+ ofdpa_neigh_del(trans, found);
} else if (updating) {
ofdpa_neigh_update(found, trans, NULL, false);
resolved = !is_zero_ether_addr(found->eth_dst);
* recipients
*/
if (is_mc_recip) {
- MCDI_DECLARE_BUF(inbuf, MC_CMD_FILTER_OP_IN_LEN);
+ MCDI_DECLARE_BUF(inbuf, MC_CMD_FILTER_OP_EXT_IN_LEN);
unsigned int depth, i;
memset(inbuf, 0, sizeof(inbuf));
efx_ef10_filter_set_entry(table, filter_idx, NULL, 0);
} else {
efx_mcdi_display_error(efx, MC_CMD_FILTER_OP,
- MC_CMD_FILTER_OP_IN_LEN,
+ MC_CMD_FILTER_OP_EXT_IN_LEN,
NULL, 0, rc);
}
}
struct efx_filter_spec *spec)
{
struct efx_ef10_filter_table *table = efx->filter_state;
- MCDI_DECLARE_BUF(inbuf, MC_CMD_FILTER_OP_IN_LEN);
+ MCDI_DECLARE_BUF(inbuf, MC_CMD_FILTER_OP_EXT_IN_LEN);
struct efx_filter_spec *saved_spec;
unsigned int hash, i, depth = 1;
bool replacing = false;
static void efx_ef10_filter_table_remove(struct efx_nic *efx)
{
struct efx_ef10_filter_table *table = efx->filter_state;
- MCDI_DECLARE_BUF(inbuf, MC_CMD_FILTER_OP_IN_LEN);
+ MCDI_DECLARE_BUF(inbuf, MC_CMD_FILTER_OP_EXT_IN_LEN);
struct efx_filter_spec *spec;
unsigned int filter_idx;
int rc;
/* Insert/renew filters */
for (i = 0; i < addr_count; i++) {
+ EFX_WARN_ON_PARANOID(ids[i] != EFX_EF10_FILTER_ID_INVALID);
efx_filter_init_rx(&spec, EFX_FILTER_PRI_AUTO, filter_flags, 0);
efx_filter_set_eth_local(&spec, vlan->vid, addr_list[i].addr);
rc = efx_ef10_filter_insert(efx, &spec, true);
}
return rc;
} else {
- /* mark as not inserted, and carry on */
- rc = EFX_EF10_FILTER_ID_INVALID;
+ /* keep invalid ID, and carry on */
}
+ } else {
+ ids[i] = efx_ef10_filter_get_unsafe_id(rc);
}
- ids[i] = efx_ef10_filter_get_unsafe_id(rc);
}
if (multicast && rollback) {
up_write(&vf->efx->filter_sem);
mutex_unlock(&vf->efx->mac_lock);
- up_write(&vf->efx->filter_sem);
-
rc2 = efx_net_open(vf->efx->net_dev);
if (rc2)
goto reset_nic;
tx_q->tx_skbuff_dma[first_entry].buf = des;
tx_q->tx_skbuff_dma[first_entry].len = skb_headlen(skb);
- tx_q->tx_skbuff[first_entry] = skb;
first->des0 = cpu_to_le32(des);
tx_q->tx_skbuff_dma[tx_q->cur_tx].last_segment = true;
+ /* Only the last descriptor gets to point to the skb. */
+ tx_q->tx_skbuff[tx_q->cur_tx] = skb;
+
+ /* We've used all descriptors we need for this skb, however,
+ * advance cur_tx so that it references a fresh descriptor.
+ * ndo_start_xmit will fill this descriptor the next time it's
+ * called and stmmac_tx_clean may clean up to this descriptor.
+ */
tx_q->cur_tx = STMMAC_GET_ENTRY(tx_q->cur_tx, DMA_TX_SIZE);
if (unlikely(stmmac_tx_avail(priv, queue) <= (MAX_SKB_FRAGS + 1))) {
first = desc;
- tx_q->tx_skbuff[first_entry] = skb;
-
enh_desc = priv->plat->enh_desc;
/* To program the descriptors according to the size of the frame */
if (enh_desc)
skb->len);
}
- entry = STMMAC_GET_ENTRY(entry, DMA_TX_SIZE);
+ /* Only the last descriptor gets to point to the skb. */
+ tx_q->tx_skbuff[entry] = skb;
+ /* We've used all descriptors we need for this skb, however,
+ * advance cur_tx so that it references a fresh descriptor.
+ * ndo_start_xmit will fill this descriptor the next time it's
+ * called and stmmac_tx_clean may clean up to this descriptor.
+ */
+ entry = STMMAC_GET_ENTRY(entry, DMA_TX_SIZE);
tx_q->cur_tx = entry;
if (netif_msg_pktdata(priv)) {
if (of_device_is_compatible(dev->of_node, "ti,dm816-emac"))
return cpsw_am33xx_cm_get_macid(dev, 0x30, slave, mac_addr);
- if (of_machine_is_compatible("ti,am4372"))
+ if (of_machine_is_compatible("ti,am43"))
return cpsw_am33xx_cm_get_macid(dev, 0x630, slave, mac_addr);
if (of_machine_is_compatible("ti,dra7"))
channels->rx_count || channels->tx_count || channels->other_count)
return -EINVAL;
- if (count > net->num_tx_queues || count > net->num_rx_queues)
+ if (count > net->num_tx_queues || count > VRSS_CHANNEL_MAX)
return -EINVAL;
if (!nvdev || nvdev->destroy)
rndis_dev = ndev->extension;
if (indir) {
for (i = 0; i < ITAB_NUM; i++)
- if (indir[i] >= dev->num_rx_queues)
+ if (indir[i] >= VRSS_CHANNEL_MAX)
return -EINVAL;
for (i = 0; i < ITAB_NUM; i++)
#define MACVLAN_HASH_SIZE (1<<MACVLAN_HASH_BITS)
#define MACVLAN_BC_QUEUE_LEN 1000
+#define MACVLAN_F_PASSTHRU 1
+#define MACVLAN_F_ADDRCHANGE 2
+
struct macvlan_port {
struct net_device *dev;
struct hlist_head vlan_hash[MACVLAN_HASH_SIZE];
struct list_head vlans;
struct sk_buff_head bc_queue;
struct work_struct bc_work;
- bool passthru;
+ u32 flags;
int count;
struct hlist_head vlan_source_hash[MACVLAN_HASH_SIZE];
DECLARE_BITMAP(mc_filter, MACVLAN_MC_FILTER_SZ);
+ unsigned char perm_addr[ETH_ALEN];
};
struct macvlan_source_entry {
static void macvlan_port_destroy(struct net_device *dev);
+static inline bool macvlan_passthru(const struct macvlan_port *port)
+{
+ return port->flags & MACVLAN_F_PASSTHRU;
+}
+
+static inline void macvlan_set_passthru(struct macvlan_port *port)
+{
+ port->flags |= MACVLAN_F_PASSTHRU;
+}
+
+static inline bool macvlan_addr_change(const struct macvlan_port *port)
+{
+ return port->flags & MACVLAN_F_ADDRCHANGE;
+}
+
+static inline void macvlan_set_addr_change(struct macvlan_port *port)
+{
+ port->flags |= MACVLAN_F_ADDRCHANGE;
+}
+
+static inline void macvlan_clear_addr_change(struct macvlan_port *port)
+{
+ port->flags &= ~MACVLAN_F_ADDRCHANGE;
+}
+
/* Hash Ethernet address */
static u32 macvlan_eth_hash(const unsigned char *addr)
{
static bool macvlan_addr_busy(const struct macvlan_port *port,
const unsigned char *addr)
{
- /* Test to see if the specified multicast address is
+ /* Test to see if the specified address is
* currently in use by the underlying device or
* another macvlan.
*/
- if (ether_addr_equal_64bits(port->dev->dev_addr, addr))
+ if (!macvlan_passthru(port) && !macvlan_addr_change(port) &&
+ ether_addr_equal_64bits(port->dev->dev_addr, addr))
return true;
if (macvlan_hash_lookup(port, addr))
}
macvlan_forward_source(skb, port, eth->h_source);
- if (port->passthru)
+ if (macvlan_passthru(port))
vlan = list_first_or_null_rcu(&port->vlans,
struct macvlan_dev, list);
else
struct net_device *lowerdev = vlan->lowerdev;
int err;
- if (vlan->port->passthru) {
+ if (macvlan_passthru(vlan->port)) {
if (!(vlan->flags & MACVLAN_FLAG_NOPROMISC)) {
err = dev_set_promiscuity(lowerdev, 1);
if (err < 0)
dev_uc_unsync(lowerdev, dev);
dev_mc_unsync(lowerdev, dev);
- if (vlan->port->passthru) {
+ if (macvlan_passthru(vlan->port)) {
if (!(vlan->flags & MACVLAN_FLAG_NOPROMISC))
dev_set_promiscuity(lowerdev, -1);
goto hash_del;
{
struct macvlan_dev *vlan = netdev_priv(dev);
struct net_device *lowerdev = vlan->lowerdev;
+ struct macvlan_port *port = vlan->port;
int err;
if (!(dev->flags & IFF_UP)) {
if (macvlan_addr_busy(vlan->port, addr))
return -EBUSY;
- if (!vlan->port->passthru) {
+ if (!macvlan_passthru(port)) {
err = dev_uc_add(lowerdev, addr);
if (err)
return err;
macvlan_hash_change_addr(vlan, addr);
}
+ if (macvlan_passthru(port) && !macvlan_addr_change(port)) {
+ /* Since addr_change isn't set, we are here due to lower
+ * device change. Save the lower-dev address so we can
+ * restore it later.
+ */
+ ether_addr_copy(vlan->port->perm_addr,
+ lowerdev->dev_addr);
+ }
+ macvlan_clear_addr_change(port);
return 0;
}
if (!is_valid_ether_addr(addr->sa_data))
return -EADDRNOTAVAIL;
+ /* If the addresses are the same, this is a no-op */
+ if (ether_addr_equal(dev->dev_addr, addr->sa_data))
+ return 0;
+
if (vlan->mode == MACVLAN_MODE_PASSTHRU) {
+ macvlan_set_addr_change(vlan->port);
dev_set_mac_address(vlan->lowerdev, addr);
return 0;
}
/* Support unicast filter only on passthru devices.
* Multicast filter should be allowed on all devices.
*/
- if (!vlan->port->passthru && is_unicast_ether_addr(addr))
+ if (!macvlan_passthru(vlan->port) && is_unicast_ether_addr(addr))
return -EOPNOTSUPP;
if (flags & NLM_F_REPLACE)
/* Support unicast filter only on passthru devices.
* Multicast filter should be allowed on all devices.
*/
- if (!vlan->port->passthru && is_unicast_ether_addr(addr))
+ if (!macvlan_passthru(vlan->port) && is_unicast_ether_addr(addr))
return -EOPNOTSUPP;
if (is_unicast_ether_addr(addr))
if (port == NULL)
return -ENOMEM;
- port->passthru = false;
port->dev = dev;
+ ether_addr_copy(port->perm_addr, dev->dev_addr);
INIT_LIST_HEAD(&port->vlans);
for (i = 0; i < MACVLAN_HASH_SIZE; i++)
INIT_HLIST_HEAD(&port->vlan_hash[i]);
kfree_skb(skb);
}
+ /* If the lower device address has been changed by passthru
+ * macvlan, put it back.
+ */
+ if (macvlan_passthru(port) &&
+ !ether_addr_equal(port->dev->dev_addr, port->perm_addr)) {
+ struct sockaddr sa;
+
+ sa.sa_family = port->dev->type;
+ memcpy(&sa.sa_data, port->perm_addr, port->dev->addr_len);
+ dev_set_mac_address(port->dev, &sa);
+ }
+
kfree(port);
}
port = macvlan_port_get_rtnl(lowerdev);
/* Only 1 macvlan device can be created in passthru mode */
- if (port->passthru) {
+ if (macvlan_passthru(port)) {
/* The macvlan port must be not created this time,
* still goto destroy_macvlan_port for readability.
*/
err = -EINVAL;
goto destroy_macvlan_port;
}
- port->passthru = true;
+ macvlan_set_passthru(port);
eth_hw_addr_inherit(dev, lowerdev);
}
if (data && data[IFLA_MACVLAN_FLAGS]) {
__u16 flags = nla_get_u16(data[IFLA_MACVLAN_FLAGS]);
bool promisc = (flags ^ vlan->flags) & MACVLAN_FLAG_NOPROMISC;
- if (vlan->port->passthru && promisc) {
+ if (macvlan_passthru(vlan->port) && promisc) {
int err;
if (flags & MACVLAN_FLAG_NOPROMISC)
}
break;
case NETDEV_CHANGEADDR:
- if (!port->passthru)
+ if (!macvlan_passthru(port))
return NOTIFY_DONE;
vlan = list_first_entry_or_null(&port->vlans,
if (overflow) {
pr_debug("tx timestamp queue overflow, count %d\n", overflow);
while (skb) {
- skb_complete_tx_timestamp(skb, NULL);
+ kfree_skb(skb);
skb = skb_dequeue(&dp83640->tx_queue);
}
return;
if ((regval & 0xFF) == 0xFF) {
phy_init_hw(phydev);
phydev->link = 0;
+ if (phydev->drv->config_intr && phy_interrupt_is_valid(phydev))
+ phydev->drv->config_intr(phydev);
}
return 0;
* @cmd: ethtool_cmd
*
* A few notes about parameter checking:
+ *
* - We don't set port or transceiver, so we don't care what they
* were set to.
* - phy_start_aneg() will make sure forced settings are sane, and
.tx_fixup = ax88179_tx_fixup,
};
+static const struct driver_info belkin_info = {
+ .description = "Belkin USB Ethernet Adapter",
+ .bind = ax88179_bind,
+ .unbind = ax88179_unbind,
+ .status = ax88179_status,
+ .link_reset = ax88179_link_reset,
+ .reset = ax88179_reset,
+ .flags = FLAG_ETHER | FLAG_FRAMING_AX,
+ .rx_fixup = ax88179_rx_fixup,
+ .tx_fixup = ax88179_tx_fixup,
+};
+
static const struct usb_device_id products[] = {
{
/* ASIX AX88179 10/100/1000 */
/* Lenovo OneLinkDock Gigabit LAN */
USB_DEVICE(0x17ef, 0x304b),
.driver_info = (unsigned long)&lenovo_info,
+}, {
+ /* Belkin B2B128 USB 3.0 Hub + Gigabit Ethernet Adapter */
+ USB_DEVICE(0x050d, 0x0128),
+ .driver_info = (unsigned long)&belkin_info,
},
{ },
};
tbp = tb;
}
- if (tbp[IFLA_IFNAME]) {
+ if (ifmp && tbp[IFLA_IFNAME]) {
nla_strlcpy(ifname, tbp[IFLA_IFNAME], IFNAMSIZ);
name_assign_type = NET_NAME_USER;
} else {
return PTR_ERR(peer);
}
- if (tbp[IFLA_ADDRESS] == NULL)
+ if (!ifmp || !tbp[IFLA_ADDRESS])
eth_hw_addr_random(peer);
if (ifmp && (dev->ifindex != 0))
flush_work(&vi->config_work);
netif_device_detach(vi->dev);
+ netif_tx_disable(vi->dev);
cancel_delayed_work_sync(&vi->refill);
if (netif_running(vi->dev)) {
const char *nvram_name;
u16 domain_nr;
u16 bus_nr;
- void (*done)(struct device *dev, const struct firmware *fw,
+ void (*done)(struct device *dev, int err, const struct firmware *fw,
void *nvram_image, u32 nvram_len);
};
if (!nvram && !(fwctx->flags & BRCMF_FW_REQ_NV_OPTIONAL))
goto fail;
- fwctx->done(fwctx->dev, fwctx->code, nvram, nvram_length);
+ fwctx->done(fwctx->dev, 0, fwctx->code, nvram, nvram_length);
kfree(fwctx);
return;
fail:
brcmf_dbg(TRACE, "failed: dev=%s\n", dev_name(fwctx->dev));
release_firmware(fwctx->code);
- device_release_driver(fwctx->dev);
+ fwctx->done(fwctx->dev, -ENOENT, NULL, NULL, 0);
kfree(fwctx);
}
static void brcmf_fw_request_code_done(const struct firmware *fw, void *ctx)
{
struct brcmf_fw *fwctx = ctx;
- int ret;
+ int ret = 0;
brcmf_dbg(TRACE, "enter: dev=%s\n", dev_name(fwctx->dev));
- if (!fw)
+ if (!fw) {
+ ret = -ENOENT;
goto fail;
-
- /* only requested code so done here */
- if (!(fwctx->flags & BRCMF_FW_REQUEST_NVRAM)) {
- fwctx->done(fwctx->dev, fw, NULL, 0);
- kfree(fwctx);
- return;
}
+ /* only requested code so done here */
+ if (!(fwctx->flags & BRCMF_FW_REQUEST_NVRAM))
+ goto done;
+
fwctx->code = fw;
ret = request_firmware_nowait(THIS_MODULE, true, fwctx->nvram_name,
fwctx->dev, GFP_KERNEL, fwctx,
brcmf_fw_request_nvram_done);
- if (!ret)
- return;
-
- brcmf_fw_request_nvram_done(NULL, fwctx);
+ /* pass NULL to nvram callback for bcm47xx fallback */
+ if (ret)
+ brcmf_fw_request_nvram_done(NULL, fwctx);
return;
fail:
brcmf_dbg(TRACE, "failed: dev=%s\n", dev_name(fwctx->dev));
- device_release_driver(fwctx->dev);
+done:
+ fwctx->done(fwctx->dev, ret, fw, NULL, 0);
kfree(fwctx);
}
int brcmf_fw_get_firmwares_pcie(struct device *dev, u16 flags,
const char *code, const char *nvram,
- void (*fw_cb)(struct device *dev,
+ void (*fw_cb)(struct device *dev, int err,
const struct firmware *fw,
void *nvram_image, u32 nvram_len),
u16 domain_nr, u16 bus_nr)
int brcmf_fw_get_firmwares(struct device *dev, u16 flags,
const char *code, const char *nvram,
- void (*fw_cb)(struct device *dev,
+ void (*fw_cb)(struct device *dev, int err,
const struct firmware *fw,
void *nvram_image, u32 nvram_len))
{
*/
int brcmf_fw_get_firmwares_pcie(struct device *dev, u16 flags,
const char *code, const char *nvram,
- void (*fw_cb)(struct device *dev,
+ void (*fw_cb)(struct device *dev, int err,
const struct firmware *fw,
void *nvram_image, u32 nvram_len),
u16 domain_nr, u16 bus_nr);
int brcmf_fw_get_firmwares(struct device *dev, u16 flags,
const char *code, const char *nvram,
- void (*fw_cb)(struct device *dev,
+ void (*fw_cb)(struct device *dev, int err,
const struct firmware *fw,
void *nvram_image, u32 nvram_len));
struct brcmf_fws_info *fws = drvr_to_fws(ifp->drvr);
struct brcmf_fws_mac_descriptor *entry;
- if (!ifp->ndev || fws->fcmode == BRCMF_FWS_FCMODE_NONE)
+ if (!ifp->ndev || !brcmf_fws_queue_skbs(fws))
return;
entry = &fws->desc.iface[ifp->ifidx];
.write32 = brcmf_pcie_buscore_write32,
};
-static void brcmf_pcie_setup(struct device *dev, const struct firmware *fw,
+static void brcmf_pcie_setup(struct device *dev, int ret,
+ const struct firmware *fw,
void *nvram, u32 nvram_len)
{
- struct brcmf_bus *bus = dev_get_drvdata(dev);
- struct brcmf_pciedev *pcie_bus_dev = bus->bus_priv.pcie;
- struct brcmf_pciedev_info *devinfo = pcie_bus_dev->devinfo;
+ struct brcmf_bus *bus;
+ struct brcmf_pciedev *pcie_bus_dev;
+ struct brcmf_pciedev_info *devinfo;
struct brcmf_commonring **flowrings;
- int ret;
u32 i;
+ /* check firmware loading result */
+ if (ret)
+ goto fail;
+
+ bus = dev_get_drvdata(dev);
+ pcie_bus_dev = bus->bus_priv.pcie;
+ devinfo = pcie_bus_dev->devinfo;
brcmf_pcie_attach(devinfo);
/* Some of the firmwares have the size of the memory of the device
.get_memdump = brcmf_sdio_bus_get_memdump,
};
-static void brcmf_sdio_firmware_callback(struct device *dev,
+static void brcmf_sdio_firmware_callback(struct device *dev, int err,
const struct firmware *code,
void *nvram, u32 nvram_len)
{
- struct brcmf_bus *bus_if = dev_get_drvdata(dev);
- struct brcmf_sdio_dev *sdiodev = bus_if->bus_priv.sdio;
- struct brcmf_sdio *bus = sdiodev->bus;
- int err = 0;
+ struct brcmf_bus *bus_if;
+ struct brcmf_sdio_dev *sdiodev;
+ struct brcmf_sdio *bus;
u8 saveclk;
- brcmf_dbg(TRACE, "Enter: dev=%s\n", dev_name(dev));
+ brcmf_dbg(TRACE, "Enter: dev=%s, err=%d\n", dev_name(dev), err);
+ bus_if = dev_get_drvdata(dev);
+ sdiodev = bus_if->bus_priv.sdio;
+ if (err)
+ goto fail;
if (!bus_if->drvr)
return;
+ bus = sdiodev->bus;
+
/* try to download image and nvram to the dongle */
bus->alp_only = true;
err = brcmf_sdio_download_firmware(bus, code, nvram, nvram_len);
fail:
brcmf_dbg(TRACE, "failed: dev=%s, err=%d\n", dev_name(dev), err);
device_release_driver(dev);
+ device_release_driver(&sdiodev->func[2]->dev);
}
struct brcmf_sdio *brcmf_sdio_probe(struct brcmf_sdio_dev *sdiodev)
return ret;
}
-static void brcmf_usb_probe_phase2(struct device *dev,
+static void brcmf_usb_probe_phase2(struct device *dev, int ret,
const struct firmware *fw,
void *nvram, u32 nvlen)
{
struct brcmf_bus *bus = dev_get_drvdata(dev);
- struct brcmf_usbdev_info *devinfo;
- int ret;
+ struct brcmf_usbdev_info *devinfo = bus->bus_priv.usb->devinfo;
+
+ if (ret)
+ goto error;
brcmf_dbg(USB, "Start fw downloading\n");
- devinfo = bus->bus_priv.usb->devinfo;
ret = check_file(fw->data);
if (ret < 0) {
brcmf_err("invalid firmware\n");
if (ai->jobs) {
locked = down_interruptible(&ai->sem);
} else {
- wait_queue_t wait;
+ wait_queue_entry_t wait;
init_waitqueue_entry(&wait, current);
add_wait_queue(&ai->thr_wait, &wait);
static DEVICE_ATTR(bssinfo, S_IRUGO, show_bssinfo, NULL);
#ifdef CONFIG_IPW2100_DEBUG
-static ssize_t show_debug_level(struct device_driver *d, char *buf)
+static ssize_t debug_level_show(struct device_driver *d, char *buf)
{
return sprintf(buf, "0x%08X\n", ipw2100_debug_level);
}
-static ssize_t store_debug_level(struct device_driver *d,
+static ssize_t debug_level_store(struct device_driver *d,
const char *buf, size_t count)
{
u32 val;
return strnlen(buf, count);
}
-
-static DRIVER_ATTR(debug_level, S_IWUSR | S_IRUGO, show_debug_level,
- store_debug_level);
+static DRIVER_ATTR_RW(debug_level);
#endif /* CONFIG_IPW2100_DEBUG */
static ssize_t show_fatal_error(struct device *d,
*
* See the level definitions in ipw for details.
*/
-static ssize_t show_debug_level(struct device_driver *d, char *buf)
+static ssize_t debug_level_show(struct device_driver *d, char *buf)
{
return sprintf(buf, "0x%08X\n", ipw_debug_level);
}
-static ssize_t store_debug_level(struct device_driver *d, const char *buf,
+static ssize_t debug_level_store(struct device_driver *d, const char *buf,
size_t count)
{
char *p = (char *)buf;
return strnlen(buf, count);
}
-
-static DRIVER_ATTR(debug_level, S_IWUSR | S_IRUGO,
- show_debug_level, store_debug_level);
+static DRIVER_ATTR_RW(debug_level);
static inline u32 ipw_get_event_log_len(struct ipw_priv *priv)
{
ret = -EINVAL;
}
if (local->iw_mode == IW_MODE_MASTER) {
- wait_queue_t __wait;
+ wait_queue_entry_t __wait;
init_waitqueue_entry(&__wait, current);
add_wait_queue(&local->hostscan_wq, &__wait);
set_current_state(TASK_INTERRUPTIBLE);
{
struct net_device *dev = data;
struct lbs_private *priv = dev->ml_priv;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
lbs_deb_enter(LBS_DEB_THREAD);
unsigned long remaining_credit;
struct timer_list credit_timeout;
u64 credit_window_start;
+ bool rate_limited;
/* Statistics */
struct xenvif_stats stats;
if (work_done < budget) {
napi_complete_done(napi, work_done);
- xenvif_napi_schedule_or_enable_events(queue);
+ /* If the queue is rate-limited, it shall be
+ * rescheduled in the timer callback.
+ */
+ if (likely(!queue->rate_limited))
+ xenvif_napi_schedule_or_enable_events(queue);
}
return work_done;
max_credit = ULONG_MAX; /* wrapped: clamp to ULONG_MAX */
queue->remaining_credit = min(max_credit, max_burst);
+ queue->rate_limited = false;
}
void xenvif_tx_credit_callback(unsigned long data)
msecs_to_jiffies(queue->credit_usec / 1000);
/* Timer could already be pending in rare cases. */
- if (timer_pending(&queue->credit_timeout))
+ if (timer_pending(&queue->credit_timeout)) {
+ queue->rate_limited = true;
return true;
+ }
/* Passed the point where we can replenish credit? */
if (time_after_eq64(now, next_credit)) {
mod_timer(&queue->credit_timeout,
next_credit);
queue->credit_window_start = next_credit;
+ queue->rate_limited = true;
return true;
}
.link_is_up = xeon_link_is_up,
.db_ioread = skx_db_ioread,
.db_iowrite = skx_db_iowrite,
- .db_size = sizeof(u64),
+ .db_size = sizeof(u32),
.ntb_ctl = SKX_NTBCNTL_OFFSET,
.mw_bar = {2, 4},
};
u64 rx_err_ver;
u64 rx_memcpy;
u64 rx_async;
- u64 dma_rx_prep_err;
u64 tx_bytes;
u64 tx_pkts;
u64 tx_ring_full;
u64 tx_err_no_buf;
u64 tx_memcpy;
u64 tx_async;
- u64 dma_tx_prep_err;
};
struct ntb_transport_mw {
#define QP_TO_MW(nt, qp) ((qp) % nt->mw_count)
#define NTB_QP_DEF_NUM_ENTRIES 100
#define NTB_LINK_DOWN_TIMEOUT 10
-#define DMA_RETRIES 20
-#define DMA_OUT_RESOURCE_TO msecs_to_jiffies(50)
static void ntb_transport_rxc_db(unsigned long data);
static const struct ntb_ctx_ops ntb_transport_ops;
out_offset += snprintf(buf + out_offset, out_count - out_offset,
"free tx - \t%u\n",
ntb_transport_tx_free_entry(qp));
- out_offset += snprintf(buf + out_offset, out_count - out_offset,
- "DMA tx prep err - \t%llu\n",
- qp->dma_tx_prep_err);
- out_offset += snprintf(buf + out_offset, out_count - out_offset,
- "DMA rx prep err - \t%llu\n",
- qp->dma_rx_prep_err);
out_offset += snprintf(buf + out_offset, out_count - out_offset,
"\n");
if (!mw->virt_addr)
return -ENOMEM;
- if (qp_count % mw_count && mw_num + 1 < qp_count / mw_count)
+ if (mw_num < qp_count % mw_count)
num_qps_mw = qp_count / mw_count + 1;
else
num_qps_mw = qp_count / mw_count;
qp->tx_err_no_buf = 0;
qp->tx_memcpy = 0;
qp->tx_async = 0;
- qp->dma_tx_prep_err = 0;
- qp->dma_rx_prep_err = 0;
}
static void ntb_qp_link_cleanup(struct ntb_transport_qp *qp)
qp->event_handler = NULL;
ntb_qp_link_down_reset(qp);
- if (qp_count % mw_count && mw_num + 1 < qp_count / mw_count)
+ if (mw_num < qp_count % mw_count)
num_qps_mw = qp_count / mw_count + 1;
else
num_qps_mw = qp_count / mw_count;
qp_count = ilog2(qp_bitmap);
if (max_num_clients && max_num_clients < qp_count)
qp_count = max_num_clients;
- else if (mw_count < qp_count)
- qp_count = mw_count;
+ else if (nt->mw_count < qp_count)
+ qp_count = nt->mw_count;
qp_bitmap &= BIT_ULL(qp_count) - 1;
struct dmaengine_unmap_data *unmap;
dma_cookie_t cookie;
void *buf = entry->buf;
- int retries = 0;
len = entry->len;
device = chan->device;
unmap->from_cnt = 1;
- for (retries = 0; retries < DMA_RETRIES; retries++) {
- txd = device->device_prep_dma_memcpy(chan,
- unmap->addr[1],
- unmap->addr[0], len,
- DMA_PREP_INTERRUPT);
- if (txd)
- break;
-
- set_current_state(TASK_INTERRUPTIBLE);
- schedule_timeout(DMA_OUT_RESOURCE_TO);
- }
-
- if (!txd) {
- qp->dma_rx_prep_err++;
+ txd = device->device_prep_dma_memcpy(chan, unmap->addr[1],
+ unmap->addr[0], len,
+ DMA_PREP_INTERRUPT);
+ if (!txd)
goto err_get_unmap;
- }
txd->callback_result = ntb_rx_copy_callback;
txd->callback_param = entry;
struct dmaengine_unmap_data *unmap;
dma_addr_t dest;
dma_cookie_t cookie;
- int retries = 0;
device = chan->device;
dest = qp->tx_mw_phys + qp->tx_max_frame * entry->tx_index;
unmap->to_cnt = 1;
- for (retries = 0; retries < DMA_RETRIES; retries++) {
- txd = device->device_prep_dma_memcpy(chan, dest,
- unmap->addr[0], len,
- DMA_PREP_INTERRUPT);
- if (txd)
- break;
-
- set_current_state(TASK_INTERRUPTIBLE);
- schedule_timeout(DMA_OUT_RESOURCE_TO);
- }
-
- if (!txd) {
- qp->dma_tx_prep_err++;
+ txd = device->device_prep_dma_memcpy(chan, dest, unmap->addr[0], len,
+ DMA_PREP_INTERRUPT);
+ if (!txd)
goto err_get_unmap;
- }
txd->callback_result = ntb_tx_copy_callback;
txd->callback_param = entry;
static unsigned int seg_order = 19; /* 512K */
module_param(seg_order, uint, 0644);
-MODULE_PARM_DESC(seg_order, "size order [n^2] of buffer segment for testing");
+MODULE_PARM_DESC(seg_order, "size order [2^n] of buffer segment for testing");
static unsigned int run_order = 32; /* 4G */
module_param(run_order, uint, 0644);
-MODULE_PARM_DESC(run_order, "size order [n^2] of total data to transfer");
+MODULE_PARM_DESC(run_order, "size order [2^n] of total data to transfer");
static bool use_dma; /* default to 0 */
module_param(use_dma, bool, 0644);
#include <linux/nubus.h>
#include <linux/errno.h>
#include <linux/init.h>
-#include <linux/delay.h>
#include <linux/module.h>
#include <linux/slab.h>
#include <asm/setup.h>
#define NUBUS_TEST_PATTERN 0x5A932BC7
-/* Define this if you like to live dangerously - it is known not to
- work on pretty much every machine except the Quadra 630 and the LC
- III. */
-#undef I_WANT_TO_PROBE_SLOT_ZERO
-
-/* This sometimes helps combat failure to boot */
-#undef TRY_TO_DODGE_WSOD
-
/* Globals */
struct nubus_dev *nubus_devices;
{
unsigned char *p = *ptr;
- /* Sanity check */
- if (len > 65536)
- pr_err("rewind of 0x%08x!\n", len);
while (len) {
do {
p--;
{
unsigned char *p = *ptr;
- if (len > 65536)
- pr_err("advance of 0x%08x!\n", len);
while (len) {
while (not_useful(p, map))
p++;
static void nubus_move(unsigned char **ptr, int len, int map)
{
+ unsigned long slot_space = (unsigned long)*ptr & 0xFF000000;
+
if (len > 0)
nubus_advance(ptr, len, map);
else if (len < 0)
nubus_rewind(ptr, -len, map);
+
+ if (((unsigned long)*ptr & 0xFF000000) != slot_space)
+ pr_err("%s: moved out of slot address space!\n", __func__);
}
/* Now, functions to read the sResource tree */
pr_info(" Function 0x%02x:\n", parent->type);
nubus_get_subdir(parent, &dir);
- /* Apple seems to have botched the ROM on the IIx */
- if (slot == 0 && (unsigned long)dir.base % 2)
- dir.base += 1;
-
pr_debug("%s: parent is 0x%p, dir is 0x%p\n",
__func__, parent->base, dir.base);
return 0;
}
-/* Attempt to bypass the somewhat non-obvious arrangement of
- sResources in the motherboard ROM */
-static void __init nubus_find_rom_dir(struct nubus_board* board)
-{
- unsigned char *rp;
- unsigned char *romdir;
- struct nubus_dir dir;
- struct nubus_dirent ent;
-
- /* Check for the extra directory just under the format block */
- rp = board->fblock;
- nubus_rewind(&rp, 4, board->lanes);
- if (nubus_get_rom(&rp, 4, board->lanes) != NUBUS_TEST_PATTERN) {
- /* OK, the ROM was telling the truth */
- board->directory = board->fblock;
- nubus_move(&board->directory,
- nubus_expand32(board->doffset),
- board->lanes);
- return;
- }
-
- /* On "slot zero", you have to walk down a few more
- directories to get to the equivalent of a real card's root
- directory. We don't know what they were smoking when they
- came up with this. */
- romdir = nubus_rom_addr(board->slot);
- nubus_rewind(&romdir, ROM_DIR_OFFSET, board->lanes);
- dir.base = dir.ptr = romdir;
- dir.done = 0;
- dir.mask = board->lanes;
-
- /* This one points to an "Unknown Macintosh" directory */
- if (nubus_readdir(&dir, &ent) == -1)
- goto badrom;
-
- if (console_loglevel >= CONSOLE_LOGLEVEL_DEBUG)
- printk(KERN_INFO "nubus_get_rom_dir: entry %02x %06x\n", ent.type, ent.data);
- /* This one takes us to where we want to go. */
- if (nubus_readdir(&dir, &ent) == -1)
- goto badrom;
- if (console_loglevel >= CONSOLE_LOGLEVEL_DEBUG)
- printk(KERN_DEBUG "nubus_get_rom_dir: entry %02x %06x\n", ent.type, ent.data);
- nubus_get_subdir(&ent, &dir);
-
- /* Resource ID 01, also an "Unknown Macintosh" */
- if (nubus_readdir(&dir, &ent) == -1)
- goto badrom;
- if (console_loglevel >= CONSOLE_LOGLEVEL_DEBUG)
- printk(KERN_DEBUG "nubus_get_rom_dir: entry %02x %06x\n", ent.type, ent.data);
-
- /* FIXME: the first one is *not* always the right one. We
- suspect this has something to do with the ROM revision.
- "The HORROR ROM" (LC-series) uses 0x7e, while "The HORROR
- Continues" (Q630) uses 0x7b. The DAFB Macs evidently use
- something else. Please run "Slots" on your Mac (see
- include/linux/nubus.h for where to get this program) and
- tell us where the 'SiDirPtr' for Slot 0 is. If you feel
- brave, you should also use MacsBug to walk down the ROM
- directories like this function does and try to find the
- path to that address... */
- if (nubus_readdir(&dir, &ent) == -1)
- goto badrom;
- if (console_loglevel >= CONSOLE_LOGLEVEL_DEBUG)
- printk(KERN_DEBUG "nubus_get_rom_dir: entry %02x %06x\n", ent.type, ent.data);
-
- /* Bwahahahaha... */
- nubus_get_subdir(&ent, &dir);
- board->directory = dir.base;
- return;
-
- /* Even more evil laughter... */
- badrom:
- board->directory = board->fblock;
- nubus_move(&board->directory, nubus_expand32(board->doffset), board->lanes);
- printk(KERN_ERR "nubus_get_rom_dir: ROM weirdness! Notify the developers...\n");
-}
-
/* Add a board (might be many devices) to the list */
static struct nubus_board * __init nubus_add_board(int slot, int bytelanes)
{
* since the initial Macintosh ROM releases skipped the check.
*/
- /* Attempt to work around slot zero weirdness */
- nubus_find_rom_dir(board);
+ /* Set up the directory pointer */
+ board->directory = board->fblock;
+ nubus_move(&board->directory, nubus_expand32(board->doffset),
+ board->lanes);
+
nubus_get_root_dir(board, &dir);
/* We're ready to rock */
nubus_get_board_resource(board, slot, &ent);
}
- /* Aaaarrrrgghh! The LC III motherboard has *two* board
- resources. I have no idea WTF to do about this. */
-
while (nubus_readdir(&dir, &ent) != -1) {
struct nubus_dev *dev;
struct nubus_dev **devp;
continue;
dp = *rp;
- if(dp == 0)
- continue;
/* The last byte of the format block consists of two
nybbles which are "mirror images" of each other.
continue;
/* Check that this value is actually *on* one of the
bytelanes it claims are valid! */
- if ((dp & 0x0F) >= (1 << i))
+ if (not_useful(rp, dp))
continue;
/* Looks promising. Let's put it on the list. */
{
int slot;
- /* This might not work on your machine */
-#ifdef I_WANT_TO_PROBE_SLOT_ZERO
- nubus_probe_slot(0);
-#endif
for (slot = 9; slot < 15; slot++) {
nubus_probe_slot(slot);
}
via_nubus_init();
}
-#ifdef TRY_TO_DODGE_WSOD
- /* Rogue Ethernet interrupts can kill the machine if we don't
- do this. Obviously this is bogus. Hopefully the local VIA
- gurus can fix the real cause of the problem. */
- mdelay(1000);
-#endif
-
/* And probe */
pr_info("NuBus: Scanning NuBus slots.\n");
nubus_devices = NULL;
* another kernel subsystem, and we just pass it through.
*/
if (bio_integrity_enabled(bio) && bio_integrity_prep(bio)) {
- bio->bi_error = -EIO;
+ bio->bi_status = BLK_STS_IOERR;
goto out;
}
"io error in %s sector %lld, len %d,\n",
(rw == READ) ? "READ" : "WRITE",
(unsigned long long) iter.bi_sector, len);
- bio->bi_error = err;
+ bio->bi_status = errno_to_blk_status(err);
break;
}
}
blk_queue_make_request(q, nd_blk_make_request);
blk_queue_max_hw_sectors(q, UINT_MAX);
- blk_queue_bounce_limit(q, BLK_BOUNCE_ANY);
blk_queue_logical_block_size(q, nsblk_sector_size(nsblk));
queue_flag_set_unlocked(QUEUE_FLAG_NONROT, q);
q->queuedata = nsblk;
* another kernel subsystem, and we just pass it through.
*/
if (bio_integrity_enabled(bio) && bio_integrity_prep(bio)) {
- bio->bi_error = -EIO;
+ bio->bi_status = BLK_STS_IOERR;
goto out;
}
(op_is_write(bio_op(bio))) ? "WRITE" :
"READ",
(unsigned long long) iter.bi_sector, len);
- bio->bi_error = err;
+ bio->bi_status = errno_to_blk_status(err);
break;
}
}
blk_queue_make_request(btt->btt_queue, btt_make_request);
blk_queue_logical_block_size(btt->btt_queue, btt->sector_size);
blk_queue_max_hw_sectors(btt->btt_queue, UINT_MAX);
- blk_queue_bounce_limit(btt->btt_queue, BLK_BOUNCE_ANY);
queue_flag_set_unlocked(QUEUE_FLAG_NONROT, btt->btt_queue);
btt->btt_queue->queuedata = btt;
return dev;
}
-static bool uuid_is_null(u8 *uuid)
-{
- static const u8 null_uuid[16];
-
- return (memcmp(uuid, null_uuid, 16) == 0);
-}
-
/**
* nd_btt_arena_is_valid - check if the metadata layout is valid
* @nd_btt: device with BTT geometry and backing device info
if (memcmp(super->signature, BTT_SIG, BTT_SIG_LEN) != 0)
return false;
- if (!uuid_is_null(super->parent_uuid))
+ if (!guid_is_null((guid_t *)&super->parent_uuid))
if (memcmp(super->parent_uuid, parent_uuid, 16) != 0)
return false;
return to_nd_region(to_dev(pmem)->parent);
}
-static int pmem_clear_poison(struct pmem_device *pmem, phys_addr_t offset,
- unsigned int len)
+static blk_status_t pmem_clear_poison(struct pmem_device *pmem,
+ phys_addr_t offset, unsigned int len)
{
struct device *dev = to_dev(pmem);
sector_t sector;
long cleared;
- int rc = 0;
+ blk_status_t rc = BLK_STS_OK;
sector = (offset - pmem->data_offset) / 512;
cleared = nvdimm_clear_poison(dev, pmem->phys_addr + offset, len);
if (cleared < len)
- rc = -EIO;
+ rc = BLK_STS_IOERR;
if (cleared > 0 && cleared / 512) {
cleared /= 512;
dev_dbg(dev, "%s: %#llx clear %ld sector%s\n", __func__,
kunmap_atomic(mem);
}
-static int read_pmem(struct page *page, unsigned int off,
+static blk_status_t read_pmem(struct page *page, unsigned int off,
void *pmem_addr, unsigned int len)
{
int rc;
rc = memcpy_mcsafe(mem + off, pmem_addr, len);
kunmap_atomic(mem);
if (rc)
- return -EIO;
- return 0;
+ return BLK_STS_IOERR;
+ return BLK_STS_OK;
}
-static int pmem_do_bvec(struct pmem_device *pmem, struct page *page,
+static blk_status_t pmem_do_bvec(struct pmem_device *pmem, struct page *page,
unsigned int len, unsigned int off, bool is_write,
sector_t sector)
{
- int rc = 0;
+ blk_status_t rc = BLK_STS_OK;
bool bad_pmem = false;
phys_addr_t pmem_off = sector * 512 + pmem->data_offset;
void *pmem_addr = pmem->virt_addr + pmem_off;
if (!is_write) {
if (unlikely(bad_pmem))
- rc = -EIO;
+ rc = BLK_STS_IOERR;
else {
rc = read_pmem(page, off, pmem_addr, len);
flush_dcache_page(page);
static blk_qc_t pmem_make_request(struct request_queue *q, struct bio *bio)
{
- int rc = 0;
+ blk_status_t rc = 0;
bool do_acct;
unsigned long start;
struct bio_vec bvec;
bvec.bv_offset, op_is_write(bio_op(bio)),
iter.bi_sector);
if (rc) {
- bio->bi_error = rc;
+ bio->bi_status = rc;
break;
}
}
struct page *page, bool is_write)
{
struct pmem_device *pmem = bdev->bd_queue->queuedata;
- int rc;
+ blk_status_t rc;
rc = pmem_do_bvec(pmem, page, PAGE_SIZE, 0, is_write, sector);
if (rc == 0)
page_endio(page, is_write, 0);
- return rc;
+ return blk_status_to_errno(rc);
}
/* see "strong" declaration in tools/testing/nvdimm/pmem-dax.c */
blk_queue_make_request(q, pmem_make_request);
blk_queue_physical_block_size(q, PAGE_SIZE);
blk_queue_max_hw_sectors(q, UINT_MAX);
- blk_queue_bounce_limit(q, BLK_BOUNCE_ANY);
queue_flag_set_unlocked(QUEUE_FLAG_NONROT, q);
queue_flag_set_unlocked(QUEUE_FLAG_DAX, q);
q->queuedata = pmem;
To compile this driver as a module, choose M here: the
module will be called nvme.
-config BLK_DEV_NVME_SCSI
- bool "SCSI emulation for NVMe device nodes"
- depends on NVME_CORE
- ---help---
- This adds support for the SG_IO ioctl on the NVMe character
- and block devices nodes, as well as a translation for a small
- number of selected SCSI commands to NVMe commands to the NVMe
- driver. If you don't know what this means you probably want
- to say N here, unless you run a distro that abuses the SCSI
- emulation to provide stable device names for mount by id, like
- some OpenSuSE and SLES versions.
-
config NVME_FABRICS
tristate
obj-$(CONFIG_NVME_FC) += nvme-fc.o
nvme-core-y := core.o
-nvme-core-$(CONFIG_BLK_DEV_NVME_SCSI) += scsi.o
nvme-core-$(CONFIG_NVM) += lightnvm.o
nvme-y += pci.o
#include <linux/nvme_ioctl.h>
#include <linux/t10-pi.h>
#include <linux/pm_qos.h>
-#include <scsi/sg.h>
#include <asm/unaligned.h>
#include "nvme.h"
MODULE_PARM_DESC(io_timeout, "timeout in seconds for I/O");
EXPORT_SYMBOL_GPL(nvme_io_timeout);
-unsigned char shutdown_timeout = 5;
+static unsigned char shutdown_timeout = 5;
module_param(shutdown_timeout, byte, 0644);
MODULE_PARM_DESC(shutdown_timeout, "timeout in seconds for controller shutdown");
module_param(force_apst, bool, 0644);
MODULE_PARM_DESC(force_apst, "allow APST for newly enumerated devices even if quirked off");
+static bool streams;
+module_param(streams, bool, 0644);
+MODULE_PARM_DESC(streams, "turn on support for Streams write directives");
+
+struct workqueue_struct *nvme_wq;
+EXPORT_SYMBOL_GPL(nvme_wq);
+
static LIST_HEAD(nvme_ctrl_list);
static DEFINE_SPINLOCK(dev_list_lock);
static struct class *nvme_class;
-static int nvme_error_status(struct request *req)
+int nvme_reset_ctrl(struct nvme_ctrl *ctrl)
+{
+ if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING))
+ return -EBUSY;
+ if (!queue_work(nvme_wq, &ctrl->reset_work))
+ return -EBUSY;
+ return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_reset_ctrl);
+
+static int nvme_reset_ctrl_sync(struct nvme_ctrl *ctrl)
+{
+ int ret;
+
+ ret = nvme_reset_ctrl(ctrl);
+ if (!ret)
+ flush_work(&ctrl->reset_work);
+ return ret;
+}
+
+static blk_status_t nvme_error_status(struct request *req)
{
switch (nvme_req(req)->status & 0x7ff) {
case NVME_SC_SUCCESS:
- return 0;
+ return BLK_STS_OK;
case NVME_SC_CAP_EXCEEDED:
- return -ENOSPC;
- default:
- return -EIO;
-
- /*
- * XXX: these errors are a nasty side-band protocol to
- * drivers/md/dm-mpath.c:noretry_error() that aren't documented
- * anywhere..
- */
- case NVME_SC_CMD_SEQ_ERROR:
- return -EILSEQ;
+ return BLK_STS_NOSPC;
case NVME_SC_ONCS_NOT_SUPPORTED:
- return -EOPNOTSUPP;
+ return BLK_STS_NOTSUPP;
case NVME_SC_WRITE_FAULT:
case NVME_SC_READ_ERROR:
case NVME_SC_UNWRITTEN_BLOCK:
- return -ENODATA;
+ return BLK_STS_MEDIUM;
+ default:
+ return BLK_STS_IOERR;
}
}
switch (old_state) {
case NVME_CTRL_NEW:
case NVME_CTRL_LIVE:
- case NVME_CTRL_RECONNECTING:
changed = true;
/* FALLTHRU */
default:
}
EXPORT_SYMBOL_GPL(nvme_alloc_request);
+static int nvme_toggle_streams(struct nvme_ctrl *ctrl, bool enable)
+{
+ struct nvme_command c;
+
+ memset(&c, 0, sizeof(c));
+
+ c.directive.opcode = nvme_admin_directive_send;
+ c.directive.nsid = cpu_to_le32(0xffffffff);
+ c.directive.doper = NVME_DIR_SND_ID_OP_ENABLE;
+ c.directive.dtype = NVME_DIR_IDENTIFY;
+ c.directive.tdtype = NVME_DIR_STREAMS;
+ c.directive.endir = enable ? NVME_DIR_ENDIR : 0;
+
+ return nvme_submit_sync_cmd(ctrl->admin_q, &c, NULL, 0);
+}
+
+static int nvme_disable_streams(struct nvme_ctrl *ctrl)
+{
+ return nvme_toggle_streams(ctrl, false);
+}
+
+static int nvme_enable_streams(struct nvme_ctrl *ctrl)
+{
+ return nvme_toggle_streams(ctrl, true);
+}
+
+static int nvme_get_stream_params(struct nvme_ctrl *ctrl,
+ struct streams_directive_params *s, u32 nsid)
+{
+ struct nvme_command c;
+
+ memset(&c, 0, sizeof(c));
+ memset(s, 0, sizeof(*s));
+
+ c.directive.opcode = nvme_admin_directive_recv;
+ c.directive.nsid = cpu_to_le32(nsid);
+ c.directive.numd = sizeof(*s);
+ c.directive.doper = NVME_DIR_RCV_ST_OP_PARAM;
+ c.directive.dtype = NVME_DIR_STREAMS;
+
+ return nvme_submit_sync_cmd(ctrl->admin_q, &c, s, sizeof(*s));
+}
+
+static int nvme_configure_directives(struct nvme_ctrl *ctrl)
+{
+ struct streams_directive_params s;
+ int ret;
+
+ if (!(ctrl->oacs & NVME_CTRL_OACS_DIRECTIVES))
+ return 0;
+ if (!streams)
+ return 0;
+
+ ret = nvme_enable_streams(ctrl);
+ if (ret)
+ return ret;
+
+ ret = nvme_get_stream_params(ctrl, &s, 0xffffffff);
+ if (ret)
+ return ret;
+
+ ctrl->nssa = le16_to_cpu(s.nssa);
+ if (ctrl->nssa < BLK_MAX_WRITE_HINTS - 1) {
+ dev_info(ctrl->device, "too few streams (%u) available\n",
+ ctrl->nssa);
+ nvme_disable_streams(ctrl);
+ return 0;
+ }
+
+ ctrl->nr_streams = min_t(unsigned, ctrl->nssa, BLK_MAX_WRITE_HINTS - 1);
+ dev_info(ctrl->device, "Using %u streams\n", ctrl->nr_streams);
+ return 0;
+}
+
+/*
+ * Check if 'req' has a write hint associated with it. If it does, assign
+ * a valid namespace stream to the write.
+ */
+static void nvme_assign_write_stream(struct nvme_ctrl *ctrl,
+ struct request *req, u16 *control,
+ u32 *dsmgmt)
+{
+ enum rw_hint streamid = req->write_hint;
+
+ if (streamid == WRITE_LIFE_NOT_SET || streamid == WRITE_LIFE_NONE)
+ streamid = 0;
+ else {
+ streamid--;
+ if (WARN_ON_ONCE(streamid > ctrl->nr_streams))
+ return;
+
+ *control |= NVME_RW_DTYPE_STREAMS;
+ *dsmgmt |= streamid << 16;
+ }
+
+ if (streamid < ARRAY_SIZE(req->q->write_hints))
+ req->q->write_hints[streamid] += blk_rq_bytes(req) >> 9;
+}
+
static inline void nvme_setup_flush(struct nvme_ns *ns,
struct nvme_command *cmnd)
{
cmnd->common.nsid = cpu_to_le32(ns->ns_id);
}
-static inline int nvme_setup_discard(struct nvme_ns *ns, struct request *req,
+static blk_status_t nvme_setup_discard(struct nvme_ns *ns, struct request *req,
struct nvme_command *cmnd)
{
unsigned short segments = blk_rq_nr_discard_segments(req), n = 0;
range = kmalloc_array(segments, sizeof(*range), GFP_ATOMIC);
if (!range)
- return BLK_MQ_RQ_QUEUE_BUSY;
+ return BLK_STS_RESOURCE;
__rq_for_each_bio(bio, req) {
u64 slba = nvme_block_nr(ns, bio->bi_iter.bi_sector);
if (WARN_ON_ONCE(n != segments)) {
kfree(range);
- return BLK_MQ_RQ_QUEUE_ERROR;
+ return BLK_STS_IOERR;
}
memset(cmnd, 0, sizeof(*cmnd));
req->special_vec.bv_len = sizeof(*range) * segments;
req->rq_flags |= RQF_SPECIAL_PAYLOAD;
- return BLK_MQ_RQ_QUEUE_OK;
+ return BLK_STS_OK;
}
-static inline void nvme_setup_rw(struct nvme_ns *ns, struct request *req,
- struct nvme_command *cmnd)
+static inline blk_status_t nvme_setup_rw(struct nvme_ns *ns,
+ struct request *req, struct nvme_command *cmnd)
{
+ struct nvme_ctrl *ctrl = ns->ctrl;
u16 control = 0;
u32 dsmgmt = 0;
+ /*
+ * If formated with metadata, require the block layer provide a buffer
+ * unless this namespace is formated such that the metadata can be
+ * stripped/generated by the controller with PRACT=1.
+ */
+ if (ns && ns->ms &&
+ (!ns->pi_type || ns->ms != sizeof(struct t10_pi_tuple)) &&
+ !blk_integrity_rq(req) && !blk_rq_is_passthrough(req))
+ return BLK_STS_NOTSUPP;
+
if (req->cmd_flags & REQ_FUA)
control |= NVME_RW_FUA;
if (req->cmd_flags & (REQ_FAILFAST_DEV | REQ_RAHEAD))
cmnd->rw.slba = cpu_to_le64(nvme_block_nr(ns, blk_rq_pos(req)));
cmnd->rw.length = cpu_to_le16((blk_rq_bytes(req) >> ns->lba_shift) - 1);
+ if (req_op(req) == REQ_OP_WRITE && ctrl->nr_streams)
+ nvme_assign_write_stream(ctrl, req, &control, &dsmgmt);
+
if (ns->ms) {
switch (ns->pi_type) {
case NVME_NS_DPS_PI_TYPE3:
cmnd->rw.control = cpu_to_le16(control);
cmnd->rw.dsmgmt = cpu_to_le32(dsmgmt);
+ return 0;
}
-int nvme_setup_cmd(struct nvme_ns *ns, struct request *req,
+blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req,
struct nvme_command *cmd)
{
- int ret = BLK_MQ_RQ_QUEUE_OK;
+ blk_status_t ret = BLK_STS_OK;
if (!(req->rq_flags & RQF_DONTPREP)) {
nvme_req(req)->retries = 0;
break;
case REQ_OP_READ:
case REQ_OP_WRITE:
- nvme_setup_rw(ns, req, cmd);
+ ret = nvme_setup_rw(ns, req, cmd);
break;
default:
WARN_ON_ONCE(1);
- return BLK_MQ_RQ_QUEUE_ERROR;
+ return BLK_STS_IOERR;
}
cmd->common.command_id = req->tag;
result, timeout);
}
-static void nvme_keep_alive_end_io(struct request *rq, int error)
+static void nvme_keep_alive_end_io(struct request *rq, blk_status_t status)
{
struct nvme_ctrl *ctrl = rq->end_io_data;
blk_mq_free_request(rq);
- if (error) {
+ if (status) {
dev_err(ctrl->device,
- "failed nvme_keep_alive_end_io error=%d\n", error);
+ "failed nvme_keep_alive_end_io error=%d\n",
+ status);
return;
}
if (nvme_keep_alive(ctrl)) {
/* allocation failure, reset the controller */
dev_err(ctrl->device, "keep-alive failed\n");
- ctrl->ops->reset_ctrl(ctrl);
+ nvme_reset_ctrl(ctrl);
return;
}
}
}
EXPORT_SYMBOL_GPL(nvme_stop_keep_alive);
-int nvme_identify_ctrl(struct nvme_ctrl *dev, struct nvme_id_ctrl **id)
+static int nvme_identify_ctrl(struct nvme_ctrl *dev, struct nvme_id_ctrl **id)
{
struct nvme_command c = { };
int error;
return error;
}
+static int nvme_identify_ns_descs(struct nvme_ns *ns, unsigned nsid)
+{
+ struct nvme_command c = { };
+ int status;
+ void *data;
+ int pos;
+ int len;
+
+ c.identify.opcode = nvme_admin_identify;
+ c.identify.nsid = cpu_to_le32(nsid);
+ c.identify.cns = NVME_ID_CNS_NS_DESC_LIST;
+
+ data = kzalloc(NVME_IDENTIFY_DATA_SIZE, GFP_KERNEL);
+ if (!data)
+ return -ENOMEM;
+
+ status = nvme_submit_sync_cmd(ns->ctrl->admin_q, &c, data,
+ NVME_IDENTIFY_DATA_SIZE);
+ if (status)
+ goto free_data;
+
+ for (pos = 0; pos < NVME_IDENTIFY_DATA_SIZE; pos += len) {
+ struct nvme_ns_id_desc *cur = data + pos;
+
+ if (cur->nidl == 0)
+ break;
+
+ switch (cur->nidt) {
+ case NVME_NIDT_EUI64:
+ if (cur->nidl != NVME_NIDT_EUI64_LEN) {
+ dev_warn(ns->ctrl->device,
+ "ctrl returned bogus length: %d for NVME_NIDT_EUI64\n",
+ cur->nidl);
+ goto free_data;
+ }
+ len = NVME_NIDT_EUI64_LEN;
+ memcpy(ns->eui, data + pos + sizeof(*cur), len);
+ break;
+ case NVME_NIDT_NGUID:
+ if (cur->nidl != NVME_NIDT_NGUID_LEN) {
+ dev_warn(ns->ctrl->device,
+ "ctrl returned bogus length: %d for NVME_NIDT_NGUID\n",
+ cur->nidl);
+ goto free_data;
+ }
+ len = NVME_NIDT_NGUID_LEN;
+ memcpy(ns->nguid, data + pos + sizeof(*cur), len);
+ break;
+ case NVME_NIDT_UUID:
+ if (cur->nidl != NVME_NIDT_UUID_LEN) {
+ dev_warn(ns->ctrl->device,
+ "ctrl returned bogus length: %d for NVME_NIDT_UUID\n",
+ cur->nidl);
+ goto free_data;
+ }
+ len = NVME_NIDT_UUID_LEN;
+ uuid_copy(&ns->uuid, data + pos + sizeof(*cur));
+ break;
+ default:
+ /* Skip unnkown types */
+ len = cur->nidl;
+ break;
+ }
+
+ len += sizeof(*cur);
+ }
+free_data:
+ kfree(data);
+ return status;
+}
+
static int nvme_identify_ns_list(struct nvme_ctrl *dev, unsigned nsid, __le32 *ns_list)
{
struct nvme_command c = { };
return nvme_submit_sync_cmd(dev->admin_q, &c, ns_list, 0x1000);
}
-int nvme_identify_ns(struct nvme_ctrl *dev, unsigned nsid,
+static int nvme_identify_ns(struct nvme_ctrl *dev, unsigned nsid,
struct nvme_id_ns **id)
{
struct nvme_command c = { };
return error;
}
-int nvme_get_features(struct nvme_ctrl *dev, unsigned fid, unsigned nsid,
- void *buffer, size_t buflen, u32 *result)
-{
- struct nvme_command c;
- union nvme_result res;
- int ret;
-
- memset(&c, 0, sizeof(c));
- c.features.opcode = nvme_admin_get_features;
- c.features.nsid = cpu_to_le32(nsid);
- c.features.fid = cpu_to_le32(fid);
-
- ret = __nvme_submit_sync_cmd(dev->admin_q, &c, &res, buffer, buflen, 0,
- NVME_QID_ANY, 0, 0);
- if (ret >= 0 && result)
- *result = le32_to_cpu(res.u32);
- return ret;
-}
-
-int nvme_set_features(struct nvme_ctrl *dev, unsigned fid, unsigned dword11,
+static int nvme_set_features(struct nvme_ctrl *dev, unsigned fid, unsigned dword11,
void *buffer, size_t buflen, u32 *result)
{
struct nvme_command c;
return ret;
}
-int nvme_get_log_page(struct nvme_ctrl *dev, struct nvme_smart_log **log)
-{
- struct nvme_command c = { };
- int error;
-
- c.common.opcode = nvme_admin_get_log_page,
- c.common.nsid = cpu_to_le32(0xFFFFFFFF),
- c.common.cdw10[0] = cpu_to_le32(
- (((sizeof(struct nvme_smart_log) / 4) - 1) << 16) |
- NVME_LOG_SMART),
-
- *log = kmalloc(sizeof(struct nvme_smart_log), GFP_KERNEL);
- if (!*log)
- return -ENOMEM;
-
- error = nvme_submit_sync_cmd(dev->admin_q, &c, *log,
- sizeof(struct nvme_smart_log));
- if (error)
- kfree(*log);
- return error;
-}
-
int nvme_set_queue_count(struct nvme_ctrl *ctrl, int *count)
{
u32 q_count = (*count - 1) | ((*count - 1) << 16);
* access to the admin queue, as that might be only way to fix them up.
*/
if (status > 0) {
- dev_err(ctrl->dev, "Could not set queue count (%d)\n", status);
+ dev_err(ctrl->device, "Could not set queue count (%d)\n", status);
*count = 0;
} else {
nr_io_queues = min(result & 0xffff, result >> 16) + 1;
return nvme_user_cmd(ns->ctrl, ns, (void __user *)arg);
case NVME_IOCTL_SUBMIT_IO:
return nvme_submit_io(ns, (void __user *)arg);
-#ifdef CONFIG_BLK_DEV_NVME_SCSI
- case SG_GET_VERSION_NUM:
- return nvme_sg_get_version_num((void __user *)arg);
- case SG_IO:
- return nvme_sg_io(ns, (void __user *)arg);
-#endif
default:
#ifdef CONFIG_NVM
if (ns->ndev)
static int nvme_compat_ioctl(struct block_device *bdev, fmode_t mode,
unsigned int cmd, unsigned long arg)
{
- switch (cmd) {
- case SG_IO:
- return -ENOIOCTLCMD;
- }
return nvme_ioctl(bdev, mode, cmd, arg);
}
#else
}
#endif /* CONFIG_BLK_DEV_INTEGRITY */
+static void nvme_set_chunk_size(struct nvme_ns *ns)
+{
+ u32 chunk_size = (((u32)ns->noiob) << (ns->lba_shift - 9));
+ blk_queue_chunk_sectors(ns->queue, rounddown_pow_of_two(chunk_size));
+}
+
static void nvme_config_discard(struct nvme_ns *ns)
{
struct nvme_ctrl *ctrl = ns->ctrl;
BUILD_BUG_ON(PAGE_SIZE / sizeof(struct nvme_dsm_range) <
NVME_DSM_MAX_RANGES);
- ns->queue->limits.discard_alignment = logical_block_size;
- ns->queue->limits.discard_granularity = logical_block_size;
+ if (ctrl->nr_streams && ns->sws && ns->sgs) {
+ unsigned int sz = logical_block_size * ns->sws * ns->sgs;
+
+ ns->queue->limits.discard_alignment = sz;
+ ns->queue->limits.discard_granularity = sz;
+ } else {
+ ns->queue->limits.discard_alignment = logical_block_size;
+ ns->queue->limits.discard_granularity = logical_block_size;
+ }
blk_queue_max_discard_sectors(ns->queue, UINT_MAX);
blk_queue_max_discard_segments(ns->queue, NVME_DSM_MAX_RANGES);
queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, ns->queue);
if (ns->ctrl->vs >= NVME_VS(1, 1, 0))
memcpy(ns->eui, (*id)->eui64, sizeof(ns->eui));
if (ns->ctrl->vs >= NVME_VS(1, 2, 0))
- memcpy(ns->uuid, (*id)->nguid, sizeof(ns->uuid));
+ memcpy(ns->nguid, (*id)->nguid, sizeof(ns->nguid));
+ if (ns->ctrl->vs >= NVME_VS(1, 3, 0)) {
+ /* Don't treat error as fatal we potentially
+ * already have a NGUID or EUI-64
+ */
+ if (nvme_identify_ns_descs(ns, ns->ns_id))
+ dev_warn(ns->ctrl->device,
+ "%s: Identify Descriptors failed\n", __func__);
+ }
return 0;
}
static void __nvme_revalidate_disk(struct gendisk *disk, struct nvme_id_ns *id)
{
struct nvme_ns *ns = disk->private_data;
+ struct nvme_ctrl *ctrl = ns->ctrl;
u16 bs;
/*
if (ns->lba_shift == 0)
ns->lba_shift = 9;
bs = 1 << ns->lba_shift;
+ ns->noiob = le16_to_cpu(id->noiob);
blk_mq_freeze_queue(disk->queue);
- if (ns->ctrl->ops->flags & NVME_F_METADATA_SUPPORTED)
+ if (ctrl->ops->flags & NVME_F_METADATA_SUPPORTED)
nvme_prep_integrity(disk, id, bs);
blk_queue_logical_block_size(ns->queue, bs);
+ if (ns->noiob)
+ nvme_set_chunk_size(ns);
if (ns->ms && !blk_get_integrity(disk) && !ns->ext)
nvme_init_integrity(ns);
if (ns->ms && !(ns->ms == 8 && ns->pi_type) && !blk_get_integrity(disk))
else
set_capacity(disk, le64_to_cpup(&id->nsze) << (ns->lba_shift - 9));
- if (ns->ctrl->oncs & NVME_CTRL_ONCS_DSM)
+ if (ctrl->oncs & NVME_CTRL_ONCS_DSM)
nvme_config_discard(ns);
blk_mq_unfreeze_queue(disk->queue);
}
int nvme_shutdown_ctrl(struct nvme_ctrl *ctrl)
{
- unsigned long timeout = SHUTDOWN_TIMEOUT + jiffies;
+ unsigned long timeout = jiffies + (shutdown_timeout * HZ);
u32 csts;
int ret;
if (!table)
return;
- if (ctrl->ps_max_latency_us == 0) {
+ if (!ctrl->apst_enabled || ctrl->ps_max_latency_us == 0) {
/* Turn off APST. */
apste = 0;
dev_dbg(ctrl->device, "APST disabled\n");
string_matches(id->fr, q->fr, sizeof(id->fr));
}
+static void nvme_init_subnqn(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
+{
+ size_t nqnlen;
+ int off;
+
+ nqnlen = strnlen(id->subnqn, NVMF_NQN_SIZE);
+ if (nqnlen > 0 && nqnlen < NVMF_NQN_SIZE) {
+ strcpy(ctrl->subnqn, id->subnqn);
+ return;
+ }
+
+ if (ctrl->vs >= NVME_VS(1, 2, 1))
+ dev_warn(ctrl->device, "missing or invalid SUBNQN field.\n");
+
+ /* Generate a "fake" NQN per Figure 254 in NVMe 1.3 + ECN 001 */
+ off = snprintf(ctrl->subnqn, NVMF_NQN_SIZE,
+ "nqn.2014.08.org.nvmexpress:%4x%4x",
+ le16_to_cpu(id->vid), le16_to_cpu(id->ssvid));
+ memcpy(ctrl->subnqn + off, id->sn, sizeof(id->sn));
+ off += sizeof(id->sn);
+ memcpy(ctrl->subnqn + off, id->mn, sizeof(id->mn));
+ off += sizeof(id->mn);
+ memset(ctrl->subnqn + off, 0, sizeof(ctrl->subnqn) - off);
+}
+
/*
* Initialize the cached copies of the Identify data and various controller
* register in our nvme_ctrl structure. This should be called as soon as
u64 cap;
int ret, page_shift;
u32 max_hw_sectors;
- u8 prev_apsta;
+ bool prev_apst_enabled;
ret = ctrl->ops->reg_read32(ctrl, NVME_REG_VS, &ctrl->vs);
if (ret) {
return -EIO;
}
+ nvme_init_subnqn(ctrl, id);
+
if (!ctrl->identified) {
/*
* Check for quirks. Quirk can depend on firmware version,
}
if (force_apst && (ctrl->quirks & NVME_QUIRK_NO_DEEPEST_PS)) {
- dev_warn(ctrl->dev, "forcibly allowing all power states due to nvme_core.force_apst -- use at your own risk\n");
+ dev_warn(ctrl->device, "forcibly allowing all power states due to nvme_core.force_apst -- use at your own risk\n");
ctrl->quirks &= ~NVME_QUIRK_NO_DEEPEST_PS;
}
ctrl->kas = le16_to_cpu(id->kas);
ctrl->npss = id->npss;
- prev_apsta = ctrl->apsta;
+ ctrl->apsta = id->apsta;
+ prev_apst_enabled = ctrl->apst_enabled;
if (ctrl->quirks & NVME_QUIRK_NO_APST) {
if (force_apst && id->apsta) {
- dev_warn(ctrl->dev, "forcibly allowing APST due to nvme_core.force_apst -- use at your own risk\n");
- ctrl->apsta = 1;
+ dev_warn(ctrl->device, "forcibly allowing APST due to nvme_core.force_apst -- use at your own risk\n");
+ ctrl->apst_enabled = true;
} else {
- ctrl->apsta = 0;
+ ctrl->apst_enabled = false;
}
} else {
- ctrl->apsta = id->apsta;
+ ctrl->apst_enabled = id->apsta;
}
memcpy(ctrl->psd, id->psd, sizeof(ctrl->psd));
ret = -EINVAL;
if (!ctrl->opts->discovery_nqn && !ctrl->kas) {
- dev_err(ctrl->dev,
+ dev_err(ctrl->device,
"keep-alive support is mandatory for fabrics\n");
ret = -EINVAL;
}
} else {
ctrl->cntlid = le16_to_cpu(id->cntlid);
+ ctrl->hmpre = le32_to_cpu(id->hmpre);
+ ctrl->hmmin = le32_to_cpu(id->hmmin);
}
kfree(id);
- if (ctrl->apsta && !prev_apsta)
+ if (ctrl->apst_enabled && !prev_apst_enabled)
dev_pm_qos_expose_latency_tolerance(ctrl->device);
- else if (!ctrl->apsta && prev_apsta)
+ else if (!ctrl->apst_enabled && prev_apst_enabled)
dev_pm_qos_hide_latency_tolerance(ctrl->device);
nvme_configure_apst(ctrl);
+ nvme_configure_directives(ctrl);
ctrl->identified = true;
return nvme_dev_user_cmd(ctrl, argp);
case NVME_IOCTL_RESET:
dev_warn(ctrl->device, "resetting controller\n");
- return ctrl->ops->reset_ctrl(ctrl);
+ return nvme_reset_ctrl_sync(ctrl);
case NVME_IOCTL_SUBSYS_RESET:
return nvme_reset_subsystem(ctrl);
case NVME_IOCTL_RESCAN:
struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
int ret;
- ret = ctrl->ops->reset_ctrl(ctrl);
+ ret = nvme_reset_ctrl_sync(ctrl);
if (ret < 0)
return ret;
return count;
int serial_len = sizeof(ctrl->serial);
int model_len = sizeof(ctrl->model);
- if (memchr_inv(ns->uuid, 0, sizeof(ns->uuid)))
- return sprintf(buf, "eui.%16phN\n", ns->uuid);
+ if (memchr_inv(ns->nguid, 0, sizeof(ns->nguid)))
+ return sprintf(buf, "eui.%16phN\n", ns->nguid);
if (memchr_inv(ns->eui, 0, sizeof(ns->eui)))
return sprintf(buf, "eui.%8phN\n", ns->eui);
}
static DEVICE_ATTR(wwid, S_IRUGO, wwid_show, NULL);
+static ssize_t nguid_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
+ return sprintf(buf, "%pU\n", ns->nguid);
+}
+static DEVICE_ATTR(nguid, S_IRUGO, nguid_show, NULL);
+
static ssize_t uuid_show(struct device *dev, struct device_attribute *attr,
char *buf)
{
struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
- return sprintf(buf, "%pU\n", ns->uuid);
+
+ /* For backward compatibility expose the NGUID to userspace if
+ * we have no UUID set
+ */
+ if (uuid_is_null(&ns->uuid)) {
+ printk_ratelimited(KERN_WARNING
+ "No UUID available providing old NGUID\n");
+ return sprintf(buf, "%pU\n", ns->nguid);
+ }
+ return sprintf(buf, "%pU\n", &ns->uuid);
}
static DEVICE_ATTR(uuid, S_IRUGO, uuid_show, NULL);
static struct attribute *nvme_ns_attrs[] = {
&dev_attr_wwid.attr,
&dev_attr_uuid.attr,
+ &dev_attr_nguid.attr,
&dev_attr_eui.attr,
&dev_attr_nsid.attr,
NULL,
struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
if (a == &dev_attr_uuid.attr) {
- if (!memchr_inv(ns->uuid, 0, sizeof(ns->uuid)))
+ if (uuid_is_null(&ns->uuid) ||
+ !memchr_inv(ns->nguid, 0, sizeof(ns->nguid)))
+ return 0;
+ }
+ if (a == &dev_attr_nguid.attr) {
+ if (!memchr_inv(ns->nguid, 0, sizeof(ns->nguid)))
return 0;
}
if (a == &dev_attr_eui.attr) {
{
struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
- return snprintf(buf, PAGE_SIZE, "%s\n",
- ctrl->ops->get_subsysnqn(ctrl));
+ return snprintf(buf, PAGE_SIZE, "%s\n", ctrl->subnqn);
}
static DEVICE_ATTR(subsysnqn, S_IRUGO, nvme_sysfs_show_subsysnqn, NULL);
NULL
};
-#define CHECK_ATTR(ctrl, a, name) \
- if ((a) == &dev_attr_##name.attr && \
- !(ctrl)->ops->get_##name) \
- return 0
-
static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
struct attribute *a, int n)
{
struct device *dev = container_of(kobj, struct device, kobj);
struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
- if (a == &dev_attr_delete_controller.attr) {
- if (!ctrl->ops->delete_ctrl)
- return 0;
- }
-
- CHECK_ATTR(ctrl, a, subsysnqn);
- CHECK_ATTR(ctrl, a, address);
+ if (a == &dev_attr_delete_controller.attr && !ctrl->ops->delete_ctrl)
+ return 0;
+ if (a == &dev_attr_address.attr && !ctrl->ops->get_address)
+ return 0;
return a->mode;
}
return ret;
}
+static int nvme_setup_streams_ns(struct nvme_ctrl *ctrl, struct nvme_ns *ns)
+{
+ struct streams_directive_params s;
+ int ret;
+
+ if (!ctrl->nr_streams)
+ return 0;
+
+ ret = nvme_get_stream_params(ctrl, &s, ns->ns_id);
+ if (ret)
+ return ret;
+
+ ns->sws = le32_to_cpu(s.sws);
+ ns->sgs = le16_to_cpu(s.sgs);
+
+ if (ns->sws) {
+ unsigned int bs = 1 << ns->lba_shift;
+
+ blk_queue_io_min(ns->queue, bs * ns->sws);
+ if (ns->sgs)
+ blk_queue_io_opt(ns->queue, bs * ns->sws * ns->sgs);
+ }
+
+ return 0;
+}
+
static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
{
struct nvme_ns *ns;
blk_queue_logical_block_size(ns->queue, 1 << ns->lba_shift);
nvme_set_queue_limits(ctrl, ns->queue);
+ nvme_setup_streams_ns(ctrl, ns);
sprintf(disk_name, "nvme%dn%d", ctrl->instance, ns->instance);
if (nvme_nvm_ns_supported(ns, id) &&
nvme_nvm_register(ns, disk_name, node)) {
- dev_warn(ctrl->dev, "%s: LightNVM init failure\n", __func__);
+ dev_warn(ctrl->device, "%s: LightNVM init failure\n", __func__);
goto out_free_id;
}
* removal.
*/
if (ctrl->state == NVME_CTRL_LIVE)
- schedule_work(&ctrl->scan_work);
+ queue_work(nvme_wq, &ctrl->scan_work);
}
EXPORT_SYMBOL_GPL(nvme_queue_scan);
/*FALLTHRU*/
case NVME_SC_ABORT_REQ:
++ctrl->event_limit;
- schedule_work(&ctrl->async_event_work);
+ queue_work(nvme_wq, &ctrl->async_event_work);
break;
default:
break;
void nvme_queue_async_events(struct nvme_ctrl *ctrl)
{
ctrl->event_limit = NVME_NR_AERS;
- schedule_work(&ctrl->async_event_work);
+ queue_work(nvme_wq, &ctrl->async_event_work);
}
EXPORT_SYMBOL_GPL(nvme_queue_async_events);
mutex_lock(&ctrl->namespaces_mutex);
+ /* Forcibly unquiesce queues to avoid blocking dispatch */
+ blk_mq_unquiesce_queue(ctrl->admin_q);
+
/* Forcibly start all queues to avoid having stuck requests */
blk_mq_start_hw_queues(ctrl->admin_q);
revalidate_disk(ns->disk);
blk_set_queue_dying(ns->queue);
+ /* Forcibly unquiesce queues to avoid blocking dispatch */
+ blk_mq_unquiesce_queue(ns->queue);
+
/*
* Forcibly start all queues to avoid having stuck requests.
* Note that we must ensure the queues are not stopped
mutex_lock(&ctrl->namespaces_mutex);
list_for_each_entry(ns, &ctrl->namespaces, list) {
- blk_mq_start_stopped_hw_queues(ns->queue, true);
+ blk_mq_unquiesce_queue(ns->queue);
blk_mq_kick_requeue_list(ns->queue);
}
mutex_unlock(&ctrl->namespaces_mutex);
{
int result;
+ nvme_wq = alloc_workqueue("nvme-wq",
+ WQ_UNBOUND | WQ_MEM_RECLAIM | WQ_SYSFS, 0);
+ if (!nvme_wq)
+ return -ENOMEM;
+
result = __register_chrdev(nvme_char_major, 0, NVME_MINORS, "nvme",
&nvme_dev_fops);
if (result < 0)
- return result;
+ goto destroy_wq;
else if (result > 0)
nvme_char_major = result;
return 0;
- unregister_chrdev:
+unregister_chrdev:
__unregister_chrdev(nvme_char_major, 0, NVME_MINORS, "nvme");
+destroy_wq:
+ destroy_workqueue(nvme_wq);
return result;
}
{
class_destroy(nvme_class);
__unregister_chrdev(nvme_char_major, 0, NVME_MINORS, "nvme");
+ destroy_workqueue(nvme_wq);
}
MODULE_LICENSE("GPL");
kref_init(&host->ref);
memcpy(host->nqn, hostnqn, NVMF_NQN_SIZE);
- uuid_be_gen(&host->id);
list_add_tail(&host->list, &nvmf_hosts);
out_unlock:
return NULL;
kref_init(&host->ref);
- uuid_be_gen(&host->id);
snprintf(host->nqn, NVMF_NQN_SIZE,
"nqn.2014-08.org.nvmexpress:NVMf:uuid:%pUb", &host->id);
}
EXPORT_SYMBOL_GPL(nvmf_get_address);
-/**
- * nvmf_get_subsysnqn() - Get subsystem NQN
- * @ctrl: Host NVMe controller instance which we got the NQN
- */
-const char *nvmf_get_subsysnqn(struct nvme_ctrl *ctrl)
-{
- return ctrl->opts->subsysnqn;
-}
-EXPORT_SYMBOL_GPL(nvmf_get_subsysnqn);
-
/**
* nvmf_reg_read32() - NVMe Fabrics "Property Get" API function.
* @ctrl: Host NVMe controller instance maintaining the admin
}
}
break;
+
+ case NVME_SC_CONNECT_INVALID_HOST:
+ dev_err(ctrl->device,
+ "Connect for subsystem %s is not allowed, hostnqn: %s\n",
+ data->subsysnqn, data->hostnqn);
+ break;
+
+ case NVME_SC_CONNECT_CTRL_BUSY:
+ dev_err(ctrl->device,
+ "Connect command failed: controller is busy or not available\n");
+ break;
+
+ case NVME_SC_CONNECT_FORMAT:
+ dev_err(ctrl->device,
+ "Connect incompatible format: %d",
+ cmd->connect.recfmt);
+ break;
+
default:
dev_err(ctrl->device,
"Connect command failed, error wo/DNR bit: %d\n",
cmd.connect.opcode = nvme_fabrics_command;
cmd.connect.fctype = nvme_fabrics_type_connect;
cmd.connect.qid = 0;
-
- /*
- * fabrics spec sets a minimum of depth 32 for admin queue,
- * so set the queue with this depth always until
- * justification otherwise.
- */
- cmd.connect.sqsize = cpu_to_le16(NVMF_AQ_DEPTH - 1);
+ cmd.connect.sqsize = cpu_to_le16(NVME_AQ_DEPTH - 1);
/*
* Set keep-alive timeout in seconds granularity (ms * 1000)
if (!data)
return -ENOMEM;
- memcpy(&data->hostid, &ctrl->opts->host->id, sizeof(uuid_be));
+ uuid_copy(&data->hostid, &ctrl->opts->host->id);
data->cntlid = cpu_to_le16(0xffff);
strncpy(data->subsysnqn, ctrl->opts->subsysnqn, NVMF_NQN_SIZE);
strncpy(data->hostnqn, ctrl->opts->host->nqn, NVMF_NQN_SIZE);
if (!data)
return -ENOMEM;
- memcpy(&data->hostid, &ctrl->opts->host->id, sizeof(uuid_be));
+ uuid_copy(&data->hostid, &ctrl->opts->host->id);
data->cntlid = cpu_to_le16(ctrl->cntlid);
strncpy(data->subsysnqn, ctrl->opts->subsysnqn, NVMF_NQN_SIZE);
strncpy(data->hostnqn, ctrl->opts->host->nqn, NVMF_NQN_SIZE);
bool nvmf_should_reconnect(struct nvme_ctrl *ctrl)
{
if (ctrl->opts->max_reconnects != -1 &&
- ctrl->opts->nr_reconnects < ctrl->opts->max_reconnects)
+ ctrl->nr_reconnects < ctrl->opts->max_reconnects)
return true;
return false;
{ NVMF_OPT_KATO, "keep_alive_tmo=%d" },
{ NVMF_OPT_HOSTNQN, "hostnqn=%s" },
{ NVMF_OPT_HOST_TRADDR, "host_traddr=%s" },
+ { NVMF_OPT_HOST_ID, "hostid=%s" },
{ NVMF_OPT_ERR, NULL }
};
int token, ret = 0;
size_t nqnlen = 0;
int ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO;
+ uuid_t hostid;
/* Set defaults */
opts->queue_size = NVMF_DEF_QUEUE_SIZE;
if (!options)
return -ENOMEM;
+ uuid_gen(&hostid);
+
while ((p = strsep(&o, ",\n")) != NULL) {
if (!*p)
continue;
}
opts->host_traddr = p;
break;
+ case NVMF_OPT_HOST_ID:
+ p = match_strdup(args);
+ if (!p) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ if (uuid_parse(p, &hostid)) {
+ ret = -EINVAL;
+ goto out;
+ }
+ break;
default:
pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
p);
opts->host = nvmf_default_host;
}
+ uuid_copy(&opts->host->id, &hostid);
+
out:
if (!opts->discovery_nqn && !opts->kato)
opts->kato = NVME_DEFAULT_KATO;
#define NVMF_REQUIRED_OPTS (NVMF_OPT_TRANSPORT | NVMF_OPT_NQN)
#define NVMF_ALLOWED_OPTS (NVMF_OPT_QUEUE_SIZE | NVMF_OPT_NR_IO_QUEUES | \
- NVMF_OPT_KATO | NVMF_OPT_HOSTNQN)
+ NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
+ NVMF_OPT_HOST_ID)
static struct nvme_ctrl *
nvmf_create_ctrl(struct device *dev, const char *buf, size_t count)
goto out_unlock;
}
+ if (strcmp(ctrl->subnqn, opts->subsysnqn)) {
+ dev_warn(ctrl->device,
+ "controller returned incorrect NQN: \"%s\".\n",
+ ctrl->subnqn);
+ mutex_unlock(&nvmf_transports_mutex);
+ ctrl->ops->delete_ctrl(ctrl);
+ return ERR_PTR(-EINVAL);
+ }
+
mutex_unlock(&nvmf_transports_mutex);
return ctrl;
struct kref ref;
struct list_head list;
char nqn[NVMF_NQN_SIZE];
- uuid_be id;
+ uuid_t id;
};
/**
NVMF_OPT_RECONNECT_DELAY = 1 << 9,
NVMF_OPT_HOST_TRADDR = 1 << 10,
NVMF_OPT_CTRL_LOSS_TMO = 1 << 11,
+ NVMF_OPT_HOST_ID = 1 << 12,
};
/**
* @discovery_nqn: indicates if the subsysnqn is the well-known discovery NQN.
* @kato: Keep-alive timeout.
* @host: Virtual NVMe host, contains the NQN and Host ID.
- * @nr_reconnects: number of reconnect attempted since the last ctrl failure
* @max_reconnects: maximum number of allowed reconnect attempts before removing
* the controller, (-1) means reconnect forever, zero means remove
* immediately;
bool discovery_nqn;
unsigned int kato;
struct nvmf_host *host;
- int nr_reconnects;
int max_reconnects;
};
int nvmf_register_transport(struct nvmf_transport_ops *ops);
void nvmf_unregister_transport(struct nvmf_transport_ops *ops);
void nvmf_free_options(struct nvmf_ctrl_options *opts);
-const char *nvmf_get_subsysnqn(struct nvme_ctrl *ctrl);
int nvmf_get_address(struct nvme_ctrl *ctrl, char *buf, int size);
bool nvmf_should_reconnect(struct nvme_ctrl *ctrl);
*/
#define NVME_FC_NR_AEN_COMMANDS 1
#define NVME_FC_AQ_BLKMQ_DEPTH \
- (NVMF_AQ_DEPTH - NVME_FC_NR_AEN_COMMANDS)
+ (NVME_AQ_DEPTH - NVME_FC_NR_AEN_COMMANDS)
#define AEN_CMDID_BASE (NVME_FC_AQ_BLKMQ_DEPTH + 1)
enum nvme_fc_queue_flags {
struct blk_mq_tag_set tag_set;
struct work_struct delete_work;
- struct work_struct reset_work;
struct delayed_work connect_work;
struct kref ref;
u32 flags;
u32 iocnt;
+ wait_queue_head_t ioabort_wait;
struct nvme_fc_fcp_op aen_ops[NVME_FC_NR_AEN_COMMANDS];
static DEFINE_IDA(nvme_fc_local_port_cnt);
static DEFINE_IDA(nvme_fc_ctrl_cnt);
-static struct workqueue_struct *nvme_fc_wq;
assoc_rqst->assoc_cmd.sqsize = cpu_to_be16(qsize);
/* Linux supports only Dynamic controllers */
assoc_rqst->assoc_cmd.cntlid = cpu_to_be16(0xffff);
- memcpy(&assoc_rqst->assoc_cmd.hostid, &ctrl->ctrl.opts->host->id,
- min_t(size_t, FCNVME_ASSOC_HOSTID_LEN, sizeof(uuid_be)));
+ uuid_copy(&assoc_rqst->assoc_cmd.hostid, &ctrl->ctrl.opts->host->id);
strncpy(assoc_rqst->assoc_cmd.hostnqn, ctrl->ctrl.opts->host->nqn,
min(FCNVME_ASSOC_HOSTNQN_LEN, NVMF_NQN_SIZE));
strncpy(assoc_rqst->assoc_cmd.subnqn, ctrl->ctrl.opts->subsysnqn,
spin_lock_irqsave(&ctrl->lock, flags);
if (unlikely(op->flags & FCOP_FLAGS_TERMIO)) {
- if (ctrl->flags & FCCTRL_TERMIO)
- ctrl->iocnt--;
+ if (ctrl->flags & FCCTRL_TERMIO) {
+ if (!--ctrl->iocnt)
+ wake_up(&ctrl->ioabort_wait);
+ }
}
if (op->flags & FCOP_FLAGS_RELEASED)
complete_rq = true;
{
struct nvme_fc_ctrl *ctrl = set->driver_data;
struct nvme_fc_fcp_op *op = blk_mq_rq_to_pdu(rq);
- struct nvme_fc_queue *queue = &ctrl->queues[hctx_idx+1];
-
- return __nvme_fc_init_request(ctrl, queue, op, rq, queue->rqcnt++);
-}
-
-static int
-nvme_fc_init_admin_request(struct blk_mq_tag_set *set, struct request *rq,
- unsigned int hctx_idx, unsigned int numa_node)
-{
- struct nvme_fc_ctrl *ctrl = set->driver_data;
- struct nvme_fc_fcp_op *op = blk_mq_rq_to_pdu(rq);
- struct nvme_fc_queue *queue = &ctrl->queues[0];
+ int queue_idx = (set == &ctrl->tag_set) ? hctx_idx + 1 : 0;
+ struct nvme_fc_queue *queue = &ctrl->queues[queue_idx];
return __nvme_fc_init_request(ctrl, queue, op, rq, queue->rqcnt++);
}
static void
nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl, char *errmsg)
{
+ /* only proceed if in LIVE state - e.g. on first error */
+ if (ctrl->ctrl.state != NVME_CTRL_LIVE)
+ return;
+
dev_warn(ctrl->ctrl.device,
"NVME-FC{%d}: transport association error detected: %s\n",
ctrl->cnum, errmsg);
dev_warn(ctrl->ctrl.device,
"NVME-FC{%d}: resetting controller\n", ctrl->cnum);
- /* stop the queues on error, cleanup is in reset thread */
- if (ctrl->queue_count > 1)
- nvme_stop_queues(&ctrl->ctrl);
-
if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RECONNECTING)) {
dev_err(ctrl->ctrl.device,
"NVME-FC{%d}: error_recovery: Couldn't change state "
return;
}
- if (!queue_work(nvme_fc_wq, &ctrl->reset_work))
- dev_err(ctrl->ctrl.device,
- "NVME-FC{%d}: error_recovery: Failed to schedule "
- "reset work\n", ctrl->cnum);
+ nvme_reset_ctrl(&ctrl->ctrl);
}
static enum blk_eh_timer_return
* level FC exchange resource that is also outstanding. This must be
* considered in all cleanup operations.
*/
-static int
+static blk_status_t
nvme_fc_start_fcp_op(struct nvme_fc_ctrl *ctrl, struct nvme_fc_queue *queue,
struct nvme_fc_fcp_op *op, u32 data_len,
enum nvmefc_fcp_datadir io_dir)
* the target device is present
*/
if (ctrl->rport->remoteport.port_state != FC_OBJSTATE_ONLINE)
- return BLK_MQ_RQ_QUEUE_ERROR;
+ return BLK_STS_IOERR;
if (!nvme_fc_ctrl_get(ctrl))
- return BLK_MQ_RQ_QUEUE_ERROR;
+ return BLK_STS_IOERR;
/* format the FC-NVME CMD IU and fcp_req */
cmdiu->connection_id = cpu_to_be64(queue->connection_id);
if (ret < 0) {
nvme_cleanup_cmd(op->rq);
nvme_fc_ctrl_put(ctrl);
- return (ret == -ENOMEM || ret == -EAGAIN) ?
- BLK_MQ_RQ_QUEUE_BUSY : BLK_MQ_RQ_QUEUE_ERROR;
+ if (ret == -ENOMEM || ret == -EAGAIN)
+ return BLK_STS_RESOURCE;
+ return BLK_STS_IOERR;
}
}
queue->lldd_handle, &op->fcp_req);
if (ret) {
- if (op->rq) { /* normal request */
+ if (op->rq) /* normal request */
nvme_fc_unmap_data(ctrl, op->rq, op);
- nvme_cleanup_cmd(op->rq);
- }
/* else - aen. no cleanup needed */
nvme_fc_ctrl_put(ctrl);
if (ret != -EBUSY)
- return BLK_MQ_RQ_QUEUE_ERROR;
+ return BLK_STS_IOERR;
if (op->rq) {
blk_mq_stop_hw_queues(op->rq->q);
blk_mq_delay_queue(queue->hctx, NVMEFC_QUEUE_DELAY);
}
- return BLK_MQ_RQ_QUEUE_BUSY;
+ return BLK_STS_RESOURCE;
}
- return BLK_MQ_RQ_QUEUE_OK;
+ return BLK_STS_OK;
}
-static int
+static blk_status_t
nvme_fc_queue_rq(struct blk_mq_hw_ctx *hctx,
const struct blk_mq_queue_data *bd)
{
struct nvme_command *sqe = &cmdiu->sqe;
enum nvmefc_fcp_datadir io_dir;
u32 data_len;
- int ret;
+ blk_status_t ret;
ret = nvme_setup_cmd(ns, rq, sqe);
if (ret)
struct nvme_fc_fcp_op *aen_op;
unsigned long flags;
bool terminating = false;
- int ret;
+ blk_status_t ret;
if (aer_idx > NVME_FC_NR_AEN_COMMANDS)
return;
op->flags &= ~(FCOP_FLAGS_TERMIO | FCOP_FLAGS_RELEASED |
FCOP_FLAGS_COMPLETE);
- nvme_cleanup_cmd(rq);
nvme_fc_unmap_data(ctrl, rq, op);
nvme_complete_rq(rq);
nvme_fc_ctrl_put(ctrl);
int ret;
bool changed;
- ++ctrl->ctrl.opts->nr_reconnects;
+ ++ctrl->ctrl.nr_reconnects;
/*
* Create the admin queue
changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
WARN_ON_ONCE(!changed);
- ctrl->ctrl.opts->nr_reconnects = 0;
+ ctrl->ctrl.nr_reconnects = 0;
if (ctrl->queue_count > 1) {
nvme_start_queues(&ctrl->ctrl);
/* wait for all io that had to be aborted */
spin_lock_irqsave(&ctrl->lock, flags);
- while (ctrl->iocnt) {
- spin_unlock_irqrestore(&ctrl->lock, flags);
- msleep(1000);
- spin_lock_irqsave(&ctrl->lock, flags);
- }
+ wait_event_lock_irq(ctrl->ioabort_wait, ctrl->iocnt == 0, ctrl->lock);
ctrl->flags &= ~FCCTRL_TERMIO;
spin_unlock_irqrestore(&ctrl->lock, flags);
struct nvme_fc_ctrl *ctrl =
container_of(work, struct nvme_fc_ctrl, delete_work);
- cancel_work_sync(&ctrl->reset_work);
+ cancel_work_sync(&ctrl->ctrl.reset_work);
cancel_delayed_work_sync(&ctrl->connect_work);
/*
if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_DELETING))
return true;
- if (!queue_work(nvme_fc_wq, &ctrl->delete_work))
+ if (!queue_work(nvme_wq, &ctrl->delete_work))
return true;
return false;
ret = __nvme_fc_del_ctrl(ctrl);
if (!ret)
- flush_workqueue(nvme_fc_wq);
+ flush_workqueue(nvme_wq);
nvme_put_ctrl(&ctrl->ctrl);
dev_info(ctrl->ctrl.device,
"NVME-FC{%d}: Reconnect attempt in %d seconds.\n",
ctrl->cnum, ctrl->ctrl.opts->reconnect_delay);
- queue_delayed_work(nvme_fc_wq, &ctrl->connect_work,
+ queue_delayed_work(nvme_wq, &ctrl->connect_work,
ctrl->ctrl.opts->reconnect_delay * HZ);
} else {
dev_warn(ctrl->ctrl.device,
"NVME-FC{%d}: Max reconnect attempts (%d) "
"reached. Removing controller\n",
- ctrl->cnum, ctrl->ctrl.opts->nr_reconnects);
+ ctrl->cnum, ctrl->ctrl.nr_reconnects);
WARN_ON(__nvme_fc_schedule_delete_work(ctrl));
}
}
nvme_fc_reset_ctrl_work(struct work_struct *work)
{
struct nvme_fc_ctrl *ctrl =
- container_of(work, struct nvme_fc_ctrl, reset_work);
+ container_of(work, struct nvme_fc_ctrl, ctrl.reset_work);
int ret;
/* will block will waiting for io to terminate */
"NVME-FC{%d}: controller reset complete\n", ctrl->cnum);
}
-/*
- * called by the nvme core layer, for sysfs interface that requests
- * a reset of the nvme controller
- */
-static int
-nvme_fc_reset_nvme_ctrl(struct nvme_ctrl *nctrl)
-{
- struct nvme_fc_ctrl *ctrl = to_fc_ctrl(nctrl);
-
- dev_info(ctrl->ctrl.device,
- "NVME-FC{%d}: admin requested controller reset\n", ctrl->cnum);
-
- if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RESETTING))
- return -EBUSY;
-
- if (!queue_work(nvme_fc_wq, &ctrl->reset_work))
- return -EBUSY;
-
- flush_work(&ctrl->reset_work);
-
- return 0;
-}
-
static const struct nvme_ctrl_ops nvme_fc_ctrl_ops = {
.name = "fc",
.module = THIS_MODULE,
.reg_read32 = nvmf_reg_read32,
.reg_read64 = nvmf_reg_read64,
.reg_write32 = nvmf_reg_write32,
- .reset_ctrl = nvme_fc_reset_nvme_ctrl,
.free_ctrl = nvme_fc_nvme_ctrl_freed,
.submit_async_event = nvme_fc_submit_async_event,
.delete_ctrl = nvme_fc_del_nvme_ctrl,
- .get_subsysnqn = nvmf_get_subsysnqn,
.get_address = nvmf_get_address,
};
static const struct blk_mq_ops nvme_fc_admin_mq_ops = {
.queue_rq = nvme_fc_queue_rq,
.complete = nvme_fc_complete_rq,
- .init_request = nvme_fc_init_admin_request,
+ .init_request = nvme_fc_init_request,
.exit_request = nvme_fc_exit_request,
.reinit_request = nvme_fc_reinit_request,
.init_hctx = nvme_fc_init_admin_hctx,
kref_init(&ctrl->ref);
INIT_WORK(&ctrl->delete_work, nvme_fc_delete_ctrl_work);
- INIT_WORK(&ctrl->reset_work, nvme_fc_reset_ctrl_work);
+ INIT_WORK(&ctrl->ctrl.reset_work, nvme_fc_reset_ctrl_work);
INIT_DELAYED_WORK(&ctrl->connect_work, nvme_fc_connect_ctrl_work);
spin_lock_init(&ctrl->lock);
nvme_uninit_ctrl(&ctrl->ctrl);
nvme_put_ctrl(&ctrl->ctrl);
+ /* Remove core ctrl ref. */
+ nvme_put_ctrl(&ctrl->ctrl);
+
/* as we're past the point where we transition to the ref
* counting teardown path, if we return a bad pointer here,
* the calling routine, thinking it's prior to the
static int __init nvme_fc_init_module(void)
{
- int ret;
-
- nvme_fc_wq = create_workqueue("nvme_fc_wq");
- if (!nvme_fc_wq)
- return -ENOMEM;
-
- ret = nvmf_register_transport(&nvme_fc_transport);
- if (ret)
- goto err;
-
- return 0;
-err:
- destroy_workqueue(nvme_fc_wq);
- return ret;
+ return nvmf_register_transport(&nvme_fc_transport);
}
static void __exit nvme_fc_exit_module(void)
nvmf_unregister_transport(&nvme_fc_transport);
- destroy_workqueue(nvme_fc_wq);
-
ida_destroy(&nvme_fc_local_port_cnt);
ida_destroy(&nvme_fc_ctrl_cnt);
}
BUILD_BUG_ON(sizeof(struct nvme_nvm_erase_blk) != 64);
BUILD_BUG_ON(sizeof(struct nvme_nvm_id_group) != 960);
BUILD_BUG_ON(sizeof(struct nvme_nvm_addr_format) != 16);
- BUILD_BUG_ON(sizeof(struct nvme_nvm_id) != 4096);
+ BUILD_BUG_ON(sizeof(struct nvme_nvm_id) != NVME_IDENTIFY_DATA_SIZE);
BUILD_BUG_ON(sizeof(struct nvme_nvm_bb_tbl) != 64);
}
rqd->bio->bi_iter.bi_sector));
}
-static void nvme_nvm_end_io(struct request *rq, int error)
+static void nvme_nvm_end_io(struct request *rq, blk_status_t status)
{
struct nvm_rq *rqd = rq->end_io_data;
rq = nvme_alloc_request(q, (struct nvme_command *)cmd, 0, NVME_QID_ANY);
if (IS_ERR(rq)) {
kfree(cmd);
- return -ENOMEM;
+ return PTR_ERR(rq);
}
rq->cmd_flags &= ~REQ_FAILFAST_DRIVER;
.max_phys_sect = 64,
};
-static void nvme_nvm_end_user_vio(struct request *rq, int error)
-{
- struct completion *waiting = rq->end_io_data;
-
- complete(waiting);
-}
-
static int nvme_nvm_submit_user_cmd(struct request_queue *q,
struct nvme_ns *ns,
struct nvme_nvm_command *vcmd,
rq->timeout = timeout ? timeout : ADMIN_TIMEOUT;
rq->cmd_flags &= ~REQ_FAILFAST_DRIVER;
- rq->end_io_data = &wait;
if (ppa_buf && ppa_len) {
ppa_list = dma_pool_alloc(dev->dma_pool, GFP_KERNEL, &ppa_dma);
}
submit:
- blk_execute_rq_nowait(q, NULL, rq, 0, nvme_nvm_end_user_vio);
-
- wait_for_completion_io(&wait);
+ blk_execute_rq(q, NULL, rq, 0);
if (nvme_req(rq)->flags & NVME_REQ_CANCELLED)
ret = -EINTR;
extern unsigned char admin_timeout;
#define ADMIN_TIMEOUT (admin_timeout * HZ)
-extern unsigned char shutdown_timeout;
-#define SHUTDOWN_TIMEOUT (shutdown_timeout * HZ)
-
#define NVME_DEFAULT_KATO 5
#define NVME_KATO_GRACE 10
+extern struct workqueue_struct *nvme_wq;
+
enum {
NVME_NS_LBA = 0,
NVME_NS_LIGHTNVM = 1,
struct device *device; /* char device */
struct list_head node;
struct ida ns_ida;
+ struct work_struct reset_work;
struct opal_dev *opal_dev;
char serial[20];
char model[40];
char firmware_rev[8];
+ char subnqn[NVMF_NQN_SIZE];
u16 cntlid;
u32 ctrl_config;
u16 oncs;
u16 vid;
u16 oacs;
+ u16 nssa;
+ u16 nr_streams;
atomic_t abort_limit;
u8 event_limit;
u8 vwc;
/* Power saving configuration */
u64 ps_max_latency_us;
+ bool apst_enabled;
+
+ u32 hmpre;
+ u32 hmmin;
/* Fabrics only */
u16 sqsize;
u32 iorcsz;
u16 icdoff;
u16 maxcmd;
+ int nr_reconnects;
struct nvmf_ctrl_options *opts;
};
-/*
- * An NVM Express namespace is equivalent to a SCSI LUN
- */
struct nvme_ns {
struct list_head list;
int instance;
u8 eui[8];
- u8 uuid[16];
+ u8 nguid[16];
+ uuid_t uuid;
unsigned ns_id;
int lba_shift;
u16 ms;
+ u16 sgs;
+ u32 sws;
bool ext;
u8 pi_type;
unsigned long flags;
+ u16 noiob;
#define NVME_NS_REMOVING 0
#define NVME_NS_DEAD 1
int (*reg_read32)(struct nvme_ctrl *ctrl, u32 off, u32 *val);
int (*reg_write32)(struct nvme_ctrl *ctrl, u32 off, u32 val);
int (*reg_read64)(struct nvme_ctrl *ctrl, u32 off, u64 *val);
- int (*reset_ctrl)(struct nvme_ctrl *ctrl);
void (*free_ctrl)(struct nvme_ctrl *ctrl);
void (*submit_async_event)(struct nvme_ctrl *ctrl, int aer_idx);
int (*delete_ctrl)(struct nvme_ctrl *ctrl);
- const char *(*get_subsysnqn)(struct nvme_ctrl *ctrl);
int (*get_address)(struct nvme_ctrl *ctrl, char *buf, int size);
};
#define NVME_QID_ANY -1
struct request *nvme_alloc_request(struct request_queue *q,
struct nvme_command *cmd, unsigned int flags, int qid);
-int nvme_setup_cmd(struct nvme_ns *ns, struct request *req,
+blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req,
struct nvme_command *cmd);
int nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd,
void *buf, unsigned bufflen);
void __user *ubuffer, unsigned bufflen,
void __user *meta_buffer, unsigned meta_len, u32 meta_seed,
u32 *result, unsigned timeout);
-int nvme_identify_ctrl(struct nvme_ctrl *dev, struct nvme_id_ctrl **id);
-int nvme_identify_ns(struct nvme_ctrl *dev, unsigned nsid,
- struct nvme_id_ns **id);
-int nvme_get_log_page(struct nvme_ctrl *dev, struct nvme_smart_log **log);
-int nvme_get_features(struct nvme_ctrl *dev, unsigned fid, unsigned nsid,
- void *buffer, size_t buflen, u32 *result);
-int nvme_set_features(struct nvme_ctrl *dev, unsigned fid, unsigned dword11,
- void *buffer, size_t buflen, u32 *result);
int nvme_set_queue_count(struct nvme_ctrl *ctrl, int *count);
void nvme_start_keep_alive(struct nvme_ctrl *ctrl);
void nvme_stop_keep_alive(struct nvme_ctrl *ctrl);
-
-struct sg_io_hdr;
-
-int nvme_sg_io(struct nvme_ns *ns, struct sg_io_hdr __user *u_hdr);
-int nvme_sg_io32(struct nvme_ns *ns, unsigned long arg);
-int nvme_sg_get_version_num(int __user *ip);
+int nvme_reset_ctrl(struct nvme_ctrl *ctrl);
#ifdef CONFIG_NVM
int nvme_nvm_ns_supported(struct nvme_ns *ns, struct nvme_id_ns *id);
#include <linux/blkdev.h>
#include <linux/blk-mq.h>
#include <linux/blk-mq-pci.h>
-#include <linux/cpu.h>
-#include <linux/delay.h>
#include <linux/dmi.h>
-#include <linux/errno.h>
-#include <linux/fs.h>
-#include <linux/genhd.h>
-#include <linux/hdreg.h>
-#include <linux/idr.h>
#include <linux/init.h>
#include <linux/interrupt.h>
#include <linux/io.h>
-#include <linux/kdev_t.h>
-#include <linux/kernel.h>
#include <linux/mm.h>
#include <linux/module.h>
-#include <linux/moduleparam.h>
#include <linux/mutex.h>
#include <linux/pci.h>
#include <linux/poison.h>
-#include <linux/ptrace.h>
-#include <linux/sched.h>
-#include <linux/slab.h>
#include <linux/t10-pi.h>
#include <linux/timer.h>
#include <linux/types.h>
#include "nvme.h"
#define NVME_Q_DEPTH 1024
-#define NVME_AQ_DEPTH 256
#define SQ_SIZE(depth) (depth * sizeof(struct nvme_command))
#define CQ_SIZE(depth) (depth * sizeof(struct nvme_completion))
module_param(use_cmb_sqes, bool, 0644);
MODULE_PARM_DESC(use_cmb_sqes, "use controller's memory buffer for I/O SQes");
-static struct workqueue_struct *nvme_workq;
+static unsigned int max_host_mem_size_mb = 128;
+module_param(max_host_mem_size_mb, uint, 0444);
+MODULE_PARM_DESC(max_host_mem_size_mb,
+ "Maximum Host Memory Buffer (HMB) size per controller (in MiB)");
struct nvme_dev;
struct nvme_queue;
-static int nvme_reset(struct nvme_dev *dev);
static void nvme_process_cq(struct nvme_queue *nvmeq);
static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown);
int q_depth;
u32 db_stride;
void __iomem *bar;
- struct work_struct reset_work;
+ unsigned long bar_mapped_size;
struct work_struct remove_work;
- struct timer_list watchdog_timer;
struct mutex shutdown_lock;
bool subsystem;
void __iomem *cmb;
u32 cmbloc;
struct nvme_ctrl ctrl;
struct completion ioq_wait;
+
+ /* shadow doorbell buffer support: */
u32 *dbbuf_dbs;
dma_addr_t dbbuf_dbs_dma_addr;
u32 *dbbuf_eis;
dma_addr_t dbbuf_eis_dma_addr;
+
+ /* host memory buffer support: */
+ u64 host_mem_size;
+ u32 nr_host_mem_descs;
+ struct nvme_host_mem_buf_desc *host_mem_descs;
+ void **host_mem_desc_bufs;
};
static inline unsigned int sq_idx(unsigned int qid, u32 stride)
BUILD_BUG_ON(sizeof(struct nvme_format_cmd) != 64);
BUILD_BUG_ON(sizeof(struct nvme_abort_cmd) != 64);
BUILD_BUG_ON(sizeof(struct nvme_command) != 64);
- BUILD_BUG_ON(sizeof(struct nvme_id_ctrl) != 4096);
- BUILD_BUG_ON(sizeof(struct nvme_id_ns) != 4096);
+ BUILD_BUG_ON(sizeof(struct nvme_id_ctrl) != NVME_IDENTIFY_DATA_SIZE);
+ BUILD_BUG_ON(sizeof(struct nvme_id_ns) != NVME_IDENTIFY_DATA_SIZE);
BUILD_BUG_ON(sizeof(struct nvme_lba_range_type) != 64);
BUILD_BUG_ON(sizeof(struct nvme_smart_log) != 512);
BUILD_BUG_ON(sizeof(struct nvme_dbbuf) != 64);
nvmeq->tags = NULL;
}
-static int nvme_admin_init_request(struct blk_mq_tag_set *set,
- struct request *req, unsigned int hctx_idx,
- unsigned int numa_node)
-{
- struct nvme_dev *dev = set->driver_data;
- struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
- struct nvme_queue *nvmeq = dev->queues[0];
-
- BUG_ON(!nvmeq);
- iod->nvmeq = nvmeq;
- return 0;
-}
-
static int nvme_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
unsigned int hctx_idx)
{
{
struct nvme_dev *dev = set->driver_data;
struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
- struct nvme_queue *nvmeq = dev->queues[hctx_idx + 1];
+ int queue_idx = (set == &dev->tagset) ? hctx_idx + 1 : 0;
+ struct nvme_queue *nvmeq = dev->queues[queue_idx];
BUG_ON(!nvmeq);
iod->nvmeq = nvmeq;
return (__le64 **)(iod->sg + blk_rq_nr_phys_segments(req));
}
-static int nvme_init_iod(struct request *rq, struct nvme_dev *dev)
+static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
{
struct nvme_iod *iod = blk_mq_rq_to_pdu(rq);
int nseg = blk_rq_nr_phys_segments(rq);
if (nseg > NVME_INT_PAGES || size > NVME_INT_BYTES(dev)) {
iod->sg = kmalloc(nvme_iod_alloc_size(dev, size, nseg), GFP_ATOMIC);
if (!iod->sg)
- return BLK_MQ_RQ_QUEUE_BUSY;
+ return BLK_STS_RESOURCE;
} else {
iod->sg = iod->inline_sg;
}
iod->nents = 0;
iod->length = size;
- return BLK_MQ_RQ_QUEUE_OK;
+ return BLK_STS_OK;
}
static void nvme_free_iod(struct nvme_dev *dev, struct request *req)
return true;
}
-static int nvme_map_data(struct nvme_dev *dev, struct request *req,
+static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
struct nvme_command *cmnd)
{
struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
struct request_queue *q = req->q;
enum dma_data_direction dma_dir = rq_data_dir(req) ?
DMA_TO_DEVICE : DMA_FROM_DEVICE;
- int ret = BLK_MQ_RQ_QUEUE_ERROR;
+ blk_status_t ret = BLK_STS_IOERR;
sg_init_table(iod->sg, blk_rq_nr_phys_segments(req));
iod->nents = blk_rq_map_sg(q, req, iod->sg);
if (!iod->nents)
goto out;
- ret = BLK_MQ_RQ_QUEUE_BUSY;
+ ret = BLK_STS_RESOURCE;
if (!dma_map_sg_attrs(dev->dev, iod->sg, iod->nents, dma_dir,
DMA_ATTR_NO_WARN))
goto out;
if (!nvme_setup_prps(dev, req))
goto out_unmap;
- ret = BLK_MQ_RQ_QUEUE_ERROR;
+ ret = BLK_STS_IOERR;
if (blk_integrity_rq(req)) {
if (blk_rq_count_integrity_sg(q, req->bio) != 1)
goto out_unmap;
cmnd->rw.dptr.prp2 = cpu_to_le64(iod->first_dma);
if (blk_integrity_rq(req))
cmnd->rw.metadata = cpu_to_le64(sg_dma_address(&iod->meta_sg));
- return BLK_MQ_RQ_QUEUE_OK;
+ return BLK_STS_OK;
out_unmap:
dma_unmap_sg(dev->dev, iod->sg, iod->nents, dma_dir);
/*
* NOTE: ns is NULL when called on the admin queue.
*/
-static int nvme_queue_rq(struct blk_mq_hw_ctx *hctx,
+static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx,
const struct blk_mq_queue_data *bd)
{
struct nvme_ns *ns = hctx->queue->queuedata;
struct nvme_dev *dev = nvmeq->dev;
struct request *req = bd->rq;
struct nvme_command cmnd;
- int ret = BLK_MQ_RQ_QUEUE_OK;
-
- /*
- * If formated with metadata, require the block layer provide a buffer
- * unless this namespace is formated such that the metadata can be
- * stripped/generated by the controller with PRACT=1.
- */
- if (ns && ns->ms && !blk_integrity_rq(req)) {
- if (!(ns->pi_type && ns->ms == 8) &&
- !blk_rq_is_passthrough(req)) {
- blk_mq_end_request(req, -EFAULT);
- return BLK_MQ_RQ_QUEUE_OK;
- }
- }
+ blk_status_t ret;
ret = nvme_setup_cmd(ns, req, &cmnd);
- if (ret != BLK_MQ_RQ_QUEUE_OK)
+ if (ret)
return ret;
ret = nvme_init_iod(req, dev);
- if (ret != BLK_MQ_RQ_QUEUE_OK)
+ if (ret)
goto out_free_cmd;
- if (blk_rq_nr_phys_segments(req))
+ if (blk_rq_nr_phys_segments(req)) {
ret = nvme_map_data(dev, req, &cmnd);
-
- if (ret != BLK_MQ_RQ_QUEUE_OK)
- goto out_cleanup_iod;
+ if (ret)
+ goto out_cleanup_iod;
+ }
blk_mq_start_request(req);
spin_lock_irq(&nvmeq->q_lock);
if (unlikely(nvmeq->cq_vector < 0)) {
- ret = BLK_MQ_RQ_QUEUE_ERROR;
+ ret = BLK_STS_IOERR;
spin_unlock_irq(&nvmeq->q_lock);
goto out_cleanup_iod;
}
__nvme_submit_cmd(nvmeq, &cmnd);
nvme_process_cq(nvmeq);
spin_unlock_irq(&nvmeq->q_lock);
- return BLK_MQ_RQ_QUEUE_OK;
+ return BLK_STS_OK;
out_cleanup_iod:
nvme_free_iod(dev, req);
out_free_cmd:
return (le16_to_cpu(nvmeq->cqes[head].status) & 1) == phase;
}
-static void __nvme_process_cq(struct nvme_queue *nvmeq, unsigned int *tag)
+static inline void nvme_ring_cq_doorbell(struct nvme_queue *nvmeq)
{
- u16 head, phase;
-
- head = nvmeq->cq_head;
- phase = nvmeq->cq_phase;
-
- while (nvme_cqe_valid(nvmeq, head, phase)) {
- struct nvme_completion cqe = nvmeq->cqes[head];
- struct request *req;
-
- if (++head == nvmeq->q_depth) {
- head = 0;
- phase = !phase;
- }
-
- if (tag && *tag == cqe.command_id)
- *tag = -1;
+ u16 head = nvmeq->cq_head;
- if (unlikely(cqe.command_id >= nvmeq->q_depth)) {
- dev_warn(nvmeq->dev->ctrl.device,
- "invalid id %d completed on queue %d\n",
- cqe.command_id, le16_to_cpu(cqe.sq_id));
- continue;
- }
+ if (likely(nvmeq->cq_vector >= 0)) {
+ if (nvme_dbbuf_update_and_check_event(head, nvmeq->dbbuf_cq_db,
+ nvmeq->dbbuf_cq_ei))
+ writel(head, nvmeq->q_db + nvmeq->dev->db_stride);
+ }
+}
- /*
- * AEN requests are special as they don't time out and can
- * survive any kind of queue freeze and often don't respond to
- * aborts. We don't even bother to allocate a struct request
- * for them but rather special case them here.
- */
- if (unlikely(nvmeq->qid == 0 &&
- cqe.command_id >= NVME_AQ_BLKMQ_DEPTH)) {
- nvme_complete_async_event(&nvmeq->dev->ctrl,
- cqe.status, &cqe.result);
- continue;
- }
+static inline void nvme_handle_cqe(struct nvme_queue *nvmeq,
+ struct nvme_completion *cqe)
+{
+ struct request *req;
- req = blk_mq_tag_to_rq(*nvmeq->tags, cqe.command_id);
- nvme_end_request(req, cqe.status, cqe.result);
+ if (unlikely(cqe->command_id >= nvmeq->q_depth)) {
+ dev_warn(nvmeq->dev->ctrl.device,
+ "invalid id %d completed on queue %d\n",
+ cqe->command_id, le16_to_cpu(cqe->sq_id));
+ return;
}
- if (head == nvmeq->cq_head && phase == nvmeq->cq_phase)
+ /*
+ * AEN requests are special as they don't time out and can
+ * survive any kind of queue freeze and often don't respond to
+ * aborts. We don't even bother to allocate a struct request
+ * for them but rather special case them here.
+ */
+ if (unlikely(nvmeq->qid == 0 &&
+ cqe->command_id >= NVME_AQ_BLKMQ_DEPTH)) {
+ nvme_complete_async_event(&nvmeq->dev->ctrl,
+ cqe->status, &cqe->result);
return;
+ }
- if (likely(nvmeq->cq_vector >= 0))
- if (nvme_dbbuf_update_and_check_event(head, nvmeq->dbbuf_cq_db,
- nvmeq->dbbuf_cq_ei))
- writel(head, nvmeq->q_db + nvmeq->dev->db_stride);
- nvmeq->cq_head = head;
- nvmeq->cq_phase = phase;
+ req = blk_mq_tag_to_rq(*nvmeq->tags, cqe->command_id);
+ nvme_end_request(req, cqe->status, cqe->result);
+}
- nvmeq->cqe_seen = 1;
+static inline bool nvme_read_cqe(struct nvme_queue *nvmeq,
+ struct nvme_completion *cqe)
+{
+ if (nvme_cqe_valid(nvmeq, nvmeq->cq_head, nvmeq->cq_phase)) {
+ *cqe = nvmeq->cqes[nvmeq->cq_head];
+
+ if (++nvmeq->cq_head == nvmeq->q_depth) {
+ nvmeq->cq_head = 0;
+ nvmeq->cq_phase = !nvmeq->cq_phase;
+ }
+ return true;
+ }
+ return false;
}
static void nvme_process_cq(struct nvme_queue *nvmeq)
{
- __nvme_process_cq(nvmeq, NULL);
+ struct nvme_completion cqe;
+ int consumed = 0;
+
+ while (nvme_read_cqe(nvmeq, &cqe)) {
+ nvme_handle_cqe(nvmeq, &cqe);
+ consumed++;
+ }
+
+ if (consumed) {
+ nvme_ring_cq_doorbell(nvmeq);
+ nvmeq->cqe_seen = 1;
+ }
}
static irqreturn_t nvme_irq(int irq, void *data)
static int __nvme_poll(struct nvme_queue *nvmeq, unsigned int tag)
{
- if (nvme_cqe_valid(nvmeq, nvmeq->cq_head, nvmeq->cq_phase)) {
- spin_lock_irq(&nvmeq->q_lock);
- __nvme_process_cq(nvmeq, &tag);
- spin_unlock_irq(&nvmeq->q_lock);
+ struct nvme_completion cqe;
+ int found = 0, consumed = 0;
- if (tag == -1)
- return 1;
- }
+ if (!nvme_cqe_valid(nvmeq, nvmeq->cq_head, nvmeq->cq_phase))
+ return 0;
- return 0;
+ spin_lock_irq(&nvmeq->q_lock);
+ while (nvme_read_cqe(nvmeq, &cqe)) {
+ nvme_handle_cqe(nvmeq, &cqe);
+ consumed++;
+
+ if (tag == cqe.command_id) {
+ found = 1;
+ break;
+ }
+ }
+
+ if (consumed)
+ nvme_ring_cq_doorbell(nvmeq);
+ spin_unlock_irq(&nvmeq->q_lock);
+
+ return found;
}
static int nvme_poll(struct blk_mq_hw_ctx *hctx, unsigned int tag)
return adapter_delete_queue(dev, nvme_admin_delete_sq, sqid);
}
-static void abort_endio(struct request *req, int error)
+static void abort_endio(struct request *req, blk_status_t error)
{
struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
struct nvme_queue *nvmeq = iod->nvmeq;
blk_mq_free_request(req);
}
+static bool nvme_should_reset(struct nvme_dev *dev, u32 csts)
+{
+
+ /* If true, indicates loss of adapter communication, possibly by a
+ * NVMe Subsystem reset.
+ */
+ bool nssro = dev->subsystem && (csts & NVME_CSTS_NSSRO);
+
+ /* If there is a reset ongoing, we shouldn't reset again. */
+ if (dev->ctrl.state == NVME_CTRL_RESETTING)
+ return false;
+
+ /* We shouldn't reset unless the controller is on fatal error state
+ * _or_ if we lost the communication with it.
+ */
+ if (!(csts & NVME_CSTS_CFS) && !nssro)
+ return false;
+
+ /* If PCI error recovery process is happening, we cannot reset or
+ * the recovery mechanism will surely fail.
+ */
+ if (pci_channel_offline(to_pci_dev(dev->dev)))
+ return false;
+
+ return true;
+}
+
+static void nvme_warn_reset(struct nvme_dev *dev, u32 csts)
+{
+ /* Read a config register to help see what died. */
+ u16 pci_status;
+ int result;
+
+ result = pci_read_config_word(to_pci_dev(dev->dev), PCI_STATUS,
+ &pci_status);
+ if (result == PCIBIOS_SUCCESSFUL)
+ dev_warn(dev->ctrl.device,
+ "controller is down; will reset: CSTS=0x%x, PCI_STATUS=0x%hx\n",
+ csts, pci_status);
+ else
+ dev_warn(dev->ctrl.device,
+ "controller is down; will reset: CSTS=0x%x, PCI_STATUS read failed (%d)\n",
+ csts, result);
+}
+
static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)
{
struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
struct nvme_dev *dev = nvmeq->dev;
struct request *abort_req;
struct nvme_command cmd;
+ u32 csts = readl(dev->bar + NVME_REG_CSTS);
+
+ /*
+ * Reset immediately if the controller is failed
+ */
+ if (nvme_should_reset(dev, csts)) {
+ nvme_warn_reset(dev, csts);
+ nvme_dev_disable(dev, false);
+ nvme_reset_ctrl(&dev->ctrl);
+ return BLK_EH_HANDLED;
+ }
/*
* Did we miss an interrupt?
"I/O %d QID %d timeout, reset controller\n",
req->tag, nvmeq->qid);
nvme_dev_disable(dev, false);
- nvme_reset(dev);
+ nvme_reset_ctrl(&dev->ctrl);
/*
* Mark the request as handled, since the inline shutdown
.complete = nvme_pci_complete_rq,
.init_hctx = nvme_admin_init_hctx,
.exit_hctx = nvme_admin_exit_hctx,
- .init_request = nvme_admin_init_request,
+ .init_request = nvme_init_request,
.timeout = nvme_timeout,
};
return 0;
}
+static unsigned long db_bar_size(struct nvme_dev *dev, unsigned nr_io_queues)
+{
+ return NVME_REG_DBS + ((nr_io_queues + 1) * 8 * dev->db_stride);
+}
+
+static int nvme_remap_bar(struct nvme_dev *dev, unsigned long size)
+{
+ struct pci_dev *pdev = to_pci_dev(dev->dev);
+
+ if (size <= dev->bar_mapped_size)
+ return 0;
+ if (size > pci_resource_len(pdev, 0))
+ return -ENOMEM;
+ if (dev->bar)
+ iounmap(dev->bar);
+ dev->bar = ioremap(pci_resource_start(pdev, 0), size);
+ if (!dev->bar) {
+ dev->bar_mapped_size = 0;
+ return -ENOMEM;
+ }
+ dev->bar_mapped_size = size;
+ dev->dbs = dev->bar + NVME_REG_DBS;
+
+ return 0;
+}
+
static int nvme_configure_admin_queue(struct nvme_dev *dev)
{
int result;
u64 cap = lo_hi_readq(dev->bar + NVME_REG_CAP);
struct nvme_queue *nvmeq;
+ result = nvme_remap_bar(dev, db_bar_size(dev, 0));
+ if (result < 0)
+ return result;
+
dev->subsystem = readl(dev->bar + NVME_REG_VS) >= NVME_VS(1, 1, 0) ?
NVME_CAP_NSSRC(cap) : 0;
return result;
}
-static bool nvme_should_reset(struct nvme_dev *dev, u32 csts)
-{
-
- /* If true, indicates loss of adapter communication, possibly by a
- * NVMe Subsystem reset.
- */
- bool nssro = dev->subsystem && (csts & NVME_CSTS_NSSRO);
-
- /* If there is a reset ongoing, we shouldn't reset again. */
- if (dev->ctrl.state == NVME_CTRL_RESETTING)
- return false;
-
- /* We shouldn't reset unless the controller is on fatal error state
- * _or_ if we lost the communication with it.
- */
- if (!(csts & NVME_CSTS_CFS) && !nssro)
- return false;
-
- /* If PCI error recovery process is happening, we cannot reset or
- * the recovery mechanism will surely fail.
- */
- if (pci_channel_offline(to_pci_dev(dev->dev)))
- return false;
-
- return true;
-}
-
-static void nvme_warn_reset(struct nvme_dev *dev, u32 csts)
-{
- /* Read a config register to help see what died. */
- u16 pci_status;
- int result;
-
- result = pci_read_config_word(to_pci_dev(dev->dev), PCI_STATUS,
- &pci_status);
- if (result == PCIBIOS_SUCCESSFUL)
- dev_warn(dev->ctrl.device,
- "controller is down; will reset: CSTS=0x%x, PCI_STATUS=0x%hx\n",
- csts, pci_status);
- else
- dev_warn(dev->ctrl.device,
- "controller is down; will reset: CSTS=0x%x, PCI_STATUS read failed (%d)\n",
- csts, result);
-}
-
-static void nvme_watchdog_timer(unsigned long data)
-{
- struct nvme_dev *dev = (struct nvme_dev *)data;
- u32 csts = readl(dev->bar + NVME_REG_CSTS);
-
- /* Skip controllers under certain specific conditions. */
- if (nvme_should_reset(dev, csts)) {
- if (!nvme_reset(dev))
- nvme_warn_reset(dev, csts);
- return;
- }
-
- mod_timer(&dev->watchdog_timer, round_jiffies(jiffies + HZ));
-}
-
static int nvme_create_io_queues(struct nvme_dev *dev)
{
unsigned i, max;
}
}
-static size_t db_bar_size(struct nvme_dev *dev, unsigned nr_io_queues)
+static int nvme_set_host_mem(struct nvme_dev *dev, u32 bits)
+{
+ size_t len = dev->nr_host_mem_descs * sizeof(*dev->host_mem_descs);
+ struct nvme_command c;
+ u64 dma_addr;
+ int ret;
+
+ dma_addr = dma_map_single(dev->dev, dev->host_mem_descs, len,
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(dev->dev, dma_addr))
+ return -ENOMEM;
+
+ memset(&c, 0, sizeof(c));
+ c.features.opcode = nvme_admin_set_features;
+ c.features.fid = cpu_to_le32(NVME_FEAT_HOST_MEM_BUF);
+ c.features.dword11 = cpu_to_le32(bits);
+ c.features.dword12 = cpu_to_le32(dev->host_mem_size >>
+ ilog2(dev->ctrl.page_size));
+ c.features.dword13 = cpu_to_le32(lower_32_bits(dma_addr));
+ c.features.dword14 = cpu_to_le32(upper_32_bits(dma_addr));
+ c.features.dword15 = cpu_to_le32(dev->nr_host_mem_descs);
+
+ ret = nvme_submit_sync_cmd(dev->ctrl.admin_q, &c, NULL, 0);
+ if (ret) {
+ dev_warn(dev->ctrl.device,
+ "failed to set host mem (err %d, flags %#x).\n",
+ ret, bits);
+ }
+ dma_unmap_single(dev->dev, dma_addr, len, DMA_TO_DEVICE);
+ return ret;
+}
+
+static void nvme_free_host_mem(struct nvme_dev *dev)
+{
+ int i;
+
+ for (i = 0; i < dev->nr_host_mem_descs; i++) {
+ struct nvme_host_mem_buf_desc *desc = &dev->host_mem_descs[i];
+ size_t size = le32_to_cpu(desc->size) * dev->ctrl.page_size;
+
+ dma_free_coherent(dev->dev, size, dev->host_mem_desc_bufs[i],
+ le64_to_cpu(desc->addr));
+ }
+
+ kfree(dev->host_mem_desc_bufs);
+ dev->host_mem_desc_bufs = NULL;
+ kfree(dev->host_mem_descs);
+ dev->host_mem_descs = NULL;
+}
+
+static int nvme_alloc_host_mem(struct nvme_dev *dev, u64 min, u64 preferred)
{
- return 4096 + ((nr_io_queues + 1) * 8 * dev->db_stride);
+ struct nvme_host_mem_buf_desc *descs;
+ u32 chunk_size, max_entries, i = 0;
+ void **bufs;
+ u64 size, tmp;
+
+ /* start big and work our way down */
+ chunk_size = min(preferred, (u64)PAGE_SIZE << MAX_ORDER);
+retry:
+ tmp = (preferred + chunk_size - 1);
+ do_div(tmp, chunk_size);
+ max_entries = tmp;
+ descs = kcalloc(max_entries, sizeof(*descs), GFP_KERNEL);
+ if (!descs)
+ goto out;
+
+ bufs = kcalloc(max_entries, sizeof(*bufs), GFP_KERNEL);
+ if (!bufs)
+ goto out_free_descs;
+
+ for (size = 0; size < preferred; size += chunk_size) {
+ u32 len = min_t(u64, chunk_size, preferred - size);
+ dma_addr_t dma_addr;
+
+ bufs[i] = dma_alloc_attrs(dev->dev, len, &dma_addr, GFP_KERNEL,
+ DMA_ATTR_NO_KERNEL_MAPPING | DMA_ATTR_NO_WARN);
+ if (!bufs[i])
+ break;
+
+ descs[i].addr = cpu_to_le64(dma_addr);
+ descs[i].size = cpu_to_le32(len / dev->ctrl.page_size);
+ i++;
+ }
+
+ if (!size || (min && size < min)) {
+ dev_warn(dev->ctrl.device,
+ "failed to allocate host memory buffer.\n");
+ goto out_free_bufs;
+ }
+
+ dev_info(dev->ctrl.device,
+ "allocated %lld MiB host memory buffer.\n",
+ size >> ilog2(SZ_1M));
+ dev->nr_host_mem_descs = i;
+ dev->host_mem_size = size;
+ dev->host_mem_descs = descs;
+ dev->host_mem_desc_bufs = bufs;
+ return 0;
+
+out_free_bufs:
+ while (--i >= 0) {
+ size_t size = le32_to_cpu(descs[i].size) * dev->ctrl.page_size;
+
+ dma_free_coherent(dev->dev, size, bufs[i],
+ le64_to_cpu(descs[i].addr));
+ }
+
+ kfree(bufs);
+out_free_descs:
+ kfree(descs);
+out:
+ /* try a smaller chunk size if we failed early */
+ if (chunk_size >= PAGE_SIZE * 2 && (i == 0 || size < min)) {
+ chunk_size /= 2;
+ goto retry;
+ }
+ dev->host_mem_descs = NULL;
+ return -ENOMEM;
+}
+
+static void nvme_setup_host_mem(struct nvme_dev *dev)
+{
+ u64 max = (u64)max_host_mem_size_mb * SZ_1M;
+ u64 preferred = (u64)dev->ctrl.hmpre * 4096;
+ u64 min = (u64)dev->ctrl.hmmin * 4096;
+ u32 enable_bits = NVME_HOST_MEM_ENABLE;
+
+ preferred = min(preferred, max);
+ if (min > max) {
+ dev_warn(dev->ctrl.device,
+ "min host memory (%lld MiB) above limit (%d MiB).\n",
+ min >> ilog2(SZ_1M), max_host_mem_size_mb);
+ nvme_free_host_mem(dev);
+ return;
+ }
+
+ /*
+ * If we already have a buffer allocated check if we can reuse it.
+ */
+ if (dev->host_mem_descs) {
+ if (dev->host_mem_size >= min)
+ enable_bits |= NVME_HOST_MEM_RETURN;
+ else
+ nvme_free_host_mem(dev);
+ }
+
+ if (!dev->host_mem_descs) {
+ if (nvme_alloc_host_mem(dev, min, preferred))
+ return;
+ }
+
+ if (nvme_set_host_mem(dev, enable_bits))
+ nvme_free_host_mem(dev);
}
static int nvme_setup_io_queues(struct nvme_dev *dev)
{
struct nvme_queue *adminq = dev->queues[0];
struct pci_dev *pdev = to_pci_dev(dev->dev);
- int result, nr_io_queues, size;
+ int result, nr_io_queues;
+ unsigned long size;
- nr_io_queues = num_online_cpus();
+ nr_io_queues = num_present_cpus();
result = nvme_set_queue_count(&dev->ctrl, &nr_io_queues);
if (result < 0)
return result;
nvme_release_cmb(dev);
}
- size = db_bar_size(dev, nr_io_queues);
- if (size > 8192) {
- iounmap(dev->bar);
- do {
- dev->bar = ioremap(pci_resource_start(pdev, 0), size);
- if (dev->bar)
- break;
- if (!--nr_io_queues)
- return -ENOMEM;
- size = db_bar_size(dev, nr_io_queues);
- } while (1);
- dev->dbs = dev->bar + 4096;
- adminq->q_db = dev->dbs;
- }
+ do {
+ size = db_bar_size(dev, nr_io_queues);
+ result = nvme_remap_bar(dev, size);
+ if (!result)
+ break;
+ if (!--nr_io_queues)
+ return -ENOMEM;
+ } while (1);
+ adminq->q_db = dev->dbs;
/* Deregister the admin queue's interrupt */
pci_free_irq(pdev, 0, adminq);
return nvme_create_io_queues(dev);
}
-static void nvme_del_queue_end(struct request *req, int error)
+static void nvme_del_queue_end(struct request *req, blk_status_t error)
{
struct nvme_queue *nvmeq = req->end_io_data;
complete(&nvmeq->dev->ioq_wait);
}
-static void nvme_del_cq_end(struct request *req, int error)
+static void nvme_del_cq_end(struct request *req, blk_status_t error)
{
struct nvme_queue *nvmeq = req->end_io_data;
bool dead = true;
struct pci_dev *pdev = to_pci_dev(dev->dev);
- del_timer_sync(&dev->watchdog_timer);
-
mutex_lock(&dev->shutdown_lock);
if (pci_is_enabled(pdev)) {
u32 csts = readl(dev->bar + NVME_REG_CSTS);
- if (dev->ctrl.state == NVME_CTRL_LIVE)
+ if (dev->ctrl.state == NVME_CTRL_LIVE ||
+ dev->ctrl.state == NVME_CTRL_RESETTING)
nvme_start_freeze(&dev->ctrl);
dead = !!((csts & NVME_CSTS_CFS) || !(csts & NVME_CSTS_RDY) ||
pdev->error_state != pci_channel_io_normal);
* Give the controller a chance to complete all entered requests if
* doing a safe shutdown.
*/
- if (!dead && shutdown)
- nvme_wait_freeze_timeout(&dev->ctrl, NVME_IO_TIMEOUT);
+ if (!dead) {
+ if (shutdown)
+ nvme_wait_freeze_timeout(&dev->ctrl, NVME_IO_TIMEOUT);
+
+ /*
+ * If the controller is still alive tell it to stop using the
+ * host memory buffer. In theory the shutdown / reset should
+ * make sure that it doesn't access the host memoery anymore,
+ * but I'd rather be safe than sorry..
+ */
+ if (dev->host_mem_descs)
+ nvme_set_host_mem(dev, 0);
+
+ }
nvme_stop_queues(&dev->ctrl);
queues = dev->online_queues - 1;
static void nvme_reset_work(struct work_struct *work)
{
- struct nvme_dev *dev = container_of(work, struct nvme_dev, reset_work);
+ struct nvme_dev *dev =
+ container_of(work, struct nvme_dev, ctrl.reset_work);
bool was_suspend = !!(dev->ctrl.ctrl_config & NVME_CC_SHN_NORMAL);
int result = -ENODEV;
"unable to allocate dma for dbbuf\n");
}
+ if (dev->ctrl.hmpre)
+ nvme_setup_host_mem(dev);
+
result = nvme_setup_io_queues(dev);
if (result)
goto out;
if (dev->online_queues > 1)
nvme_queue_async_events(&dev->ctrl);
- mod_timer(&dev->watchdog_timer, round_jiffies(jiffies + HZ));
-
/*
* Keep the controller around but remove all namespaces if we don't have
* any working I/O queue.
nvme_put_ctrl(&dev->ctrl);
}
-static int nvme_reset(struct nvme_dev *dev)
-{
- if (!dev->ctrl.admin_q || blk_queue_dying(dev->ctrl.admin_q))
- return -ENODEV;
- if (!nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_RESETTING))
- return -EBUSY;
- if (!queue_work(nvme_workq, &dev->reset_work))
- return -EBUSY;
- return 0;
-}
-
static int nvme_pci_reg_read32(struct nvme_ctrl *ctrl, u32 off, u32 *val)
{
*val = readl(to_nvme_dev(ctrl)->bar + off);
return 0;
}
-static int nvme_pci_reset_ctrl(struct nvme_ctrl *ctrl)
-{
- struct nvme_dev *dev = to_nvme_dev(ctrl);
- int ret = nvme_reset(dev);
-
- if (!ret)
- flush_work(&dev->reset_work);
- return ret;
-}
-
static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = {
.name = "pcie",
.module = THIS_MODULE,
.reg_read32 = nvme_pci_reg_read32,
.reg_write32 = nvme_pci_reg_write32,
.reg_read64 = nvme_pci_reg_read64,
- .reset_ctrl = nvme_pci_reset_ctrl,
.free_ctrl = nvme_pci_free_ctrl,
.submit_async_event = nvme_pci_submit_async_event,
};
if (pci_request_mem_regions(pdev, "nvme"))
return -ENODEV;
- dev->bar = ioremap(pci_resource_start(pdev, 0), 8192);
- if (!dev->bar)
+ if (nvme_remap_bar(dev, NVME_REG_DBS + 4096))
goto release;
return 0;
if (result)
goto free;
- INIT_WORK(&dev->reset_work, nvme_reset_work);
+ INIT_WORK(&dev->ctrl.reset_work, nvme_reset_work);
INIT_WORK(&dev->remove_work, nvme_remove_dead_ctrl_work);
- setup_timer(&dev->watchdog_timer, nvme_watchdog_timer,
- (unsigned long)dev);
mutex_init(&dev->shutdown_lock);
init_completion(&dev->ioq_wait);
nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_RESETTING);
dev_info(dev->ctrl.device, "pci function %s\n", dev_name(&pdev->dev));
- queue_work(nvme_workq, &dev->reset_work);
+ queue_work(nvme_wq, &dev->ctrl.reset_work);
return 0;
release_pools:
if (prepare)
nvme_dev_disable(dev, false);
else
- nvme_reset(dev);
+ nvme_reset_ctrl(&dev->ctrl);
}
static void nvme_shutdown(struct pci_dev *pdev)
nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_DELETING);
- cancel_work_sync(&dev->reset_work);
+ cancel_work_sync(&dev->ctrl.reset_work);
pci_set_drvdata(pdev, NULL);
if (!pci_device_is_present(pdev)) {
nvme_dev_disable(dev, false);
}
- flush_work(&dev->reset_work);
+ flush_work(&dev->ctrl.reset_work);
nvme_uninit_ctrl(&dev->ctrl);
nvme_dev_disable(dev, true);
+ nvme_free_host_mem(dev);
nvme_dev_remove_admin(dev);
nvme_free_queues(dev, 0);
nvme_release_prp_pools(dev);
struct pci_dev *pdev = to_pci_dev(dev);
struct nvme_dev *ndev = pci_get_drvdata(pdev);
- nvme_reset(ndev);
+ nvme_reset_ctrl(&ndev->ctrl);
return 0;
}
#endif
dev_info(dev->ctrl.device, "restart after slot reset\n");
pci_restore_state(pdev);
- nvme_reset(dev);
+ nvme_reset_ctrl(&dev->ctrl);
return PCI_ERS_RESULT_RECOVERED;
}
static int __init nvme_init(void)
{
- int result;
-
- nvme_workq = alloc_workqueue("nvme", WQ_UNBOUND | WQ_MEM_RECLAIM, 0);
- if (!nvme_workq)
- return -ENOMEM;
-
- result = pci_register_driver(&nvme_driver);
- if (result)
- destroy_workqueue(nvme_workq);
- return result;
+ return pci_register_driver(&nvme_driver);
}
static void __exit nvme_exit(void)
{
pci_unregister_driver(&nvme_driver);
- destroy_workqueue(nvme_workq);
_nvme_check_size();
}
*/
#define NVME_RDMA_NR_AEN_COMMANDS 1
#define NVME_RDMA_AQ_BLKMQ_DEPTH \
- (NVMF_AQ_DEPTH - NVME_RDMA_NR_AEN_COMMANDS)
+ (NVME_AQ_DEPTH - NVME_RDMA_NR_AEN_COMMANDS)
struct nvme_rdma_device {
struct ib_device *dev;
};
enum nvme_rdma_queue_flags {
- NVME_RDMA_Q_CONNECTED = (1 << 0),
- NVME_RDMA_IB_QUEUE_ALLOCATED = (1 << 1),
- NVME_RDMA_Q_DELETING = (1 << 2),
- NVME_RDMA_Q_LIVE = (1 << 3),
+ NVME_RDMA_Q_LIVE = 0,
+ NVME_RDMA_Q_DELETING = 1,
};
struct nvme_rdma_queue {
};
struct nvme_rdma_ctrl {
- /* read and written in the hot path */
- spinlock_t lock;
-
/* read only in the hot path */
struct nvme_rdma_queue *queues;
u32 queue_count;
/* other member variables */
struct blk_mq_tag_set tag_set;
struct work_struct delete_work;
- struct work_struct reset_work;
struct work_struct err_work;
struct nvme_rdma_qe async_event_sqe;
static LIST_HEAD(nvme_rdma_ctrl_list);
static DEFINE_MUTEX(nvme_rdma_ctrl_mutex);
-static struct workqueue_struct *nvme_rdma_wq;
-
/*
* Disabling this option makes small I/O goes faster, but is fundamentally
* unsafe. With it turned off we will have to register a global rkey that
return ret;
}
-static void __nvme_rdma_exit_request(struct nvme_rdma_ctrl *ctrl,
- struct request *rq, unsigned int queue_idx)
+static void nvme_rdma_exit_request(struct blk_mq_tag_set *set,
+ struct request *rq, unsigned int hctx_idx)
{
+ struct nvme_rdma_ctrl *ctrl = set->driver_data;
struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq);
+ int queue_idx = (set == &ctrl->tag_set) ? hctx_idx + 1 : 0;
struct nvme_rdma_queue *queue = &ctrl->queues[queue_idx];
struct nvme_rdma_device *dev = queue->device;
DMA_TO_DEVICE);
}
-static void nvme_rdma_exit_request(struct blk_mq_tag_set *set,
- struct request *rq, unsigned int hctx_idx)
-{
- return __nvme_rdma_exit_request(set->driver_data, rq, hctx_idx + 1);
-}
-
-static void nvme_rdma_exit_admin_request(struct blk_mq_tag_set *set,
- struct request *rq, unsigned int hctx_idx)
-{
- return __nvme_rdma_exit_request(set->driver_data, rq, 0);
-}
-
-static int __nvme_rdma_init_request(struct nvme_rdma_ctrl *ctrl,
- struct request *rq, unsigned int queue_idx)
+static int nvme_rdma_init_request(struct blk_mq_tag_set *set,
+ struct request *rq, unsigned int hctx_idx,
+ unsigned int numa_node)
{
+ struct nvme_rdma_ctrl *ctrl = set->driver_data;
struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq);
+ int queue_idx = (set == &ctrl->tag_set) ? hctx_idx + 1 : 0;
struct nvme_rdma_queue *queue = &ctrl->queues[queue_idx];
struct nvme_rdma_device *dev = queue->device;
struct ib_device *ibdev = dev->dev;
return -ENOMEM;
}
-static int nvme_rdma_init_request(struct blk_mq_tag_set *set,
- struct request *rq, unsigned int hctx_idx,
- unsigned int numa_node)
-{
- return __nvme_rdma_init_request(set->driver_data, rq, hctx_idx + 1);
-}
-
-static int nvme_rdma_init_admin_request(struct blk_mq_tag_set *set,
- struct request *rq, unsigned int hctx_idx,
- unsigned int numa_node)
-{
- return __nvme_rdma_init_request(set->driver_data, rq, 0);
-}
-
static int nvme_rdma_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
unsigned int hctx_idx)
{
struct nvme_rdma_device *dev;
struct ib_device *ibdev;
- if (!test_and_clear_bit(NVME_RDMA_IB_QUEUE_ALLOCATED, &queue->flags))
- return;
-
dev = queue->device;
ibdev = dev->dev;
rdma_destroy_qp(queue->cm_id);
nvme_rdma_dev_put(dev);
}
-static int nvme_rdma_create_queue_ib(struct nvme_rdma_queue *queue,
- struct nvme_rdma_device *dev)
+static int nvme_rdma_create_queue_ib(struct nvme_rdma_queue *queue)
{
- struct ib_device *ibdev = dev->dev;
+ struct ib_device *ibdev;
const int send_wr_factor = 3; /* MR, SEND, INV */
const int cq_factor = send_wr_factor + 1; /* + RECV */
int comp_vector, idx = nvme_rdma_queue_idx(queue);
-
int ret;
- queue->device = dev;
+ queue->device = nvme_rdma_find_get_device(queue->cm_id);
+ if (!queue->device) {
+ dev_err(queue->cm_id->device->dev.parent,
+ "no client data found!\n");
+ return -ECONNREFUSED;
+ }
+ ibdev = queue->device->dev;
/*
* The admin queue is barely used once the controller is live, so don't
/* +1 for ib_stop_cq */
- queue->ib_cq = ib_alloc_cq(dev->dev, queue,
- cq_factor * queue->queue_size + 1, comp_vector,
- IB_POLL_SOFTIRQ);
+ queue->ib_cq = ib_alloc_cq(ibdev, queue,
+ cq_factor * queue->queue_size + 1,
+ comp_vector, IB_POLL_SOFTIRQ);
if (IS_ERR(queue->ib_cq)) {
ret = PTR_ERR(queue->ib_cq);
- goto out;
+ goto out_put_dev;
}
ret = nvme_rdma_create_qp(queue, send_wr_factor);
ret = -ENOMEM;
goto out_destroy_qp;
}
- set_bit(NVME_RDMA_IB_QUEUE_ALLOCATED, &queue->flags);
return 0;
ib_destroy_qp(queue->qp);
out_destroy_ib_cq:
ib_free_cq(queue->ib_cq);
-out:
+out_put_dev:
+ nvme_rdma_dev_put(queue->device);
return ret;
}
}
clear_bit(NVME_RDMA_Q_DELETING, &queue->flags);
- set_bit(NVME_RDMA_Q_CONNECTED, &queue->flags);
return 0;
out_destroy_cm_id:
- nvme_rdma_destroy_queue_ib(queue);
rdma_destroy_id(queue->cm_id);
return ret;
}
if (nvmf_should_reconnect(&ctrl->ctrl)) {
dev_info(ctrl->ctrl.device, "Reconnecting in %d seconds...\n",
ctrl->ctrl.opts->reconnect_delay);
- queue_delayed_work(nvme_rdma_wq, &ctrl->reconnect_work,
+ queue_delayed_work(nvme_wq, &ctrl->reconnect_work,
ctrl->ctrl.opts->reconnect_delay * HZ);
} else {
dev_info(ctrl->ctrl.device, "Removing controller...\n");
- queue_work(nvme_rdma_wq, &ctrl->delete_work);
+ queue_work(nvme_wq, &ctrl->delete_work);
}
}
bool changed;
int ret;
- ++ctrl->ctrl.opts->nr_reconnects;
+ ++ctrl->ctrl.nr_reconnects;
if (ctrl->queue_count > 1) {
nvme_rdma_free_io_queues(ctrl);
if (ret)
goto requeue;
- ret = nvme_rdma_init_queue(ctrl, 0, NVMF_AQ_DEPTH);
+ ret = nvme_rdma_init_queue(ctrl, 0, NVME_AQ_DEPTH);
if (ret)
goto requeue;
changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
WARN_ON_ONCE(!changed);
- ctrl->ctrl.opts->nr_reconnects = 0;
+ ctrl->ctrl.nr_reconnects = 0;
if (ctrl->queue_count > 1) {
nvme_queue_scan(&ctrl->ctrl);
requeue:
dev_info(ctrl->ctrl.device, "Failed reconnect attempt %d\n",
- ctrl->ctrl.opts->nr_reconnects);
+ ctrl->ctrl.nr_reconnects);
nvme_rdma_reconnect_or_remove(ctrl);
}
nvme_stop_keep_alive(&ctrl->ctrl);
- for (i = 0; i < ctrl->queue_count; i++) {
- clear_bit(NVME_RDMA_Q_CONNECTED, &ctrl->queues[i].flags);
+ for (i = 0; i < ctrl->queue_count; i++)
clear_bit(NVME_RDMA_Q_LIVE, &ctrl->queues[i].flags);
- }
if (ctrl->queue_count > 1)
nvme_stop_queues(&ctrl->ctrl);
if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RECONNECTING))
return;
- queue_work(nvme_rdma_wq, &ctrl->err_work);
+ queue_work(nvme_wq, &ctrl->err_work);
}
static void nvme_rdma_wr_error(struct ib_cq *cq, struct ib_wc *wc,
static int nvme_rdma_addr_resolved(struct nvme_rdma_queue *queue)
{
- struct nvme_rdma_device *dev;
int ret;
- dev = nvme_rdma_find_get_device(queue->cm_id);
- if (!dev) {
- dev_err(queue->cm_id->device->dev.parent,
- "no client data found!\n");
- return -ECONNREFUSED;
- }
-
- ret = nvme_rdma_create_queue_ib(queue, dev);
- if (ret) {
- nvme_rdma_dev_put(dev);
- goto out;
- }
+ ret = nvme_rdma_create_queue_ib(queue);
+ if (ret)
+ return ret;
ret = rdma_resolve_route(queue->cm_id, NVME_RDMA_CONNECT_TIMEOUT_MS);
if (ret) {
out_destroy_queue:
nvme_rdma_destroy_queue_ib(queue);
-out:
return ret;
}
* specified by the Fabrics standard.
*/
if (priv.qid == 0) {
- priv.hrqsize = cpu_to_le16(NVMF_AQ_DEPTH);
- priv.hsqsize = cpu_to_le16(NVMF_AQ_DEPTH - 1);
+ priv.hrqsize = cpu_to_le16(NVME_AQ_DEPTH);
+ priv.hsqsize = cpu_to_le16(NVME_AQ_DEPTH - 1);
} else {
/*
* current interpretation of the fabrics spec
complete(&queue->cm_done);
return 0;
case RDMA_CM_EVENT_REJECTED:
+ nvme_rdma_destroy_queue_ib(queue);
cm_error = nvme_rdma_conn_rejected(queue, ev);
break;
- case RDMA_CM_EVENT_ADDR_ERROR:
case RDMA_CM_EVENT_ROUTE_ERROR:
case RDMA_CM_EVENT_CONNECT_ERROR:
case RDMA_CM_EVENT_UNREACHABLE:
+ nvme_rdma_destroy_queue_ib(queue);
+ case RDMA_CM_EVENT_ADDR_ERROR:
dev_dbg(queue->ctrl->ctrl.device,
"CM error event %d\n", ev->event);
cm_error = -ECONNRESET;
/*
* We cannot accept any other command until the Connect command has completed.
*/
-static inline int nvme_rdma_queue_is_ready(struct nvme_rdma_queue *queue,
- struct request *rq)
+static inline blk_status_t
+nvme_rdma_queue_is_ready(struct nvme_rdma_queue *queue, struct request *rq)
{
if (unlikely(!test_bit(NVME_RDMA_Q_LIVE, &queue->flags))) {
struct nvme_command *cmd = nvme_req(rq)->cmd;
* failover.
*/
if (queue->ctrl->ctrl.state == NVME_CTRL_RECONNECTING)
- return -EIO;
- else
- return -EAGAIN;
+ return BLK_STS_IOERR;
+ return BLK_STS_RESOURCE; /* try again later */
}
}
return 0;
}
-static int nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx,
+static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx,
const struct blk_mq_queue_data *bd)
{
struct nvme_ns *ns = hctx->queue->queuedata;
struct nvme_command *c = sqe->data;
bool flush = false;
struct ib_device *dev;
- int ret;
+ blk_status_t ret;
+ int err;
WARN_ON_ONCE(rq->tag < 0);
ret = nvme_rdma_queue_is_ready(queue, rq);
if (unlikely(ret))
- goto err;
+ return ret;
dev = queue->device->dev;
ib_dma_sync_single_for_cpu(dev, sqe->dma,
sizeof(struct nvme_command), DMA_TO_DEVICE);
ret = nvme_setup_cmd(ns, rq, c);
- if (ret != BLK_MQ_RQ_QUEUE_OK)
+ if (ret)
return ret;
blk_mq_start_request(rq);
- ret = nvme_rdma_map_data(queue, rq, c);
- if (ret < 0) {
+ err = nvme_rdma_map_data(queue, rq, c);
+ if (err < 0) {
dev_err(queue->ctrl->ctrl.device,
- "Failed to map data (%d)\n", ret);
+ "Failed to map data (%d)\n", err);
nvme_cleanup_cmd(rq);
goto err;
}
if (req_op(rq) == REQ_OP_FLUSH)
flush = true;
- ret = nvme_rdma_post_send(queue, sqe, req->sge, req->num_sge,
+ err = nvme_rdma_post_send(queue, sqe, req->sge, req->num_sge,
req->mr->need_inval ? &req->reg_wr.wr : NULL, flush);
- if (ret) {
+ if (err) {
nvme_rdma_unmap_data(queue, rq);
goto err;
}
- return BLK_MQ_RQ_QUEUE_OK;
+ return BLK_STS_OK;
err:
- return (ret == -ENOMEM || ret == -EAGAIN) ?
- BLK_MQ_RQ_QUEUE_BUSY : BLK_MQ_RQ_QUEUE_ERROR;
+ if (err == -ENOMEM || err == -EAGAIN)
+ return BLK_STS_RESOURCE;
+ return BLK_STS_IOERR;
}
static int nvme_rdma_poll(struct blk_mq_hw_ctx *hctx, unsigned int tag)
struct ib_wc wc;
int found = 0;
- ib_req_notify_cq(cq, IB_CQ_NEXT_COMP);
while (ib_poll_cq(cq, 1, &wc) > 0) {
struct ib_cqe *cqe = wc.wr_cqe;
static const struct blk_mq_ops nvme_rdma_admin_mq_ops = {
.queue_rq = nvme_rdma_queue_rq,
.complete = nvme_rdma_complete_rq,
- .init_request = nvme_rdma_init_admin_request,
- .exit_request = nvme_rdma_exit_admin_request,
+ .init_request = nvme_rdma_init_request,
+ .exit_request = nvme_rdma_exit_request,
.reinit_request = nvme_rdma_reinit_request,
.init_hctx = nvme_rdma_init_admin_hctx,
.timeout = nvme_rdma_timeout,
{
int error;
- error = nvme_rdma_init_queue(ctrl, 0, NVMF_AQ_DEPTH);
+ error = nvme_rdma_init_queue(ctrl, 0, NVME_AQ_DEPTH);
if (error)
return error;
nvme_rdma_free_io_queues(ctrl);
}
- if (test_bit(NVME_RDMA_Q_CONNECTED, &ctrl->queues[0].flags))
+ if (test_bit(NVME_RDMA_Q_LIVE, &ctrl->queues[0].flags))
nvme_shutdown_ctrl(&ctrl->ctrl);
blk_mq_stop_hw_queues(ctrl->ctrl.admin_q);
if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_DELETING))
return -EBUSY;
- if (!queue_work(nvme_rdma_wq, &ctrl->delete_work))
+ if (!queue_work(nvme_wq, &ctrl->delete_work))
return -EBUSY;
return 0;
static void nvme_rdma_reset_ctrl_work(struct work_struct *work)
{
- struct nvme_rdma_ctrl *ctrl = container_of(work,
- struct nvme_rdma_ctrl, reset_work);
+ struct nvme_rdma_ctrl *ctrl =
+ container_of(work, struct nvme_rdma_ctrl, ctrl.reset_work);
int ret;
bool changed;
del_dead_ctrl:
/* Deleting this dead controller... */
dev_warn(ctrl->ctrl.device, "Removing after reset failure\n");
- WARN_ON(!queue_work(nvme_rdma_wq, &ctrl->delete_work));
-}
-
-static int nvme_rdma_reset_ctrl(struct nvme_ctrl *nctrl)
-{
- struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl);
-
- if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RESETTING))
- return -EBUSY;
-
- if (!queue_work(nvme_rdma_wq, &ctrl->reset_work))
- return -EBUSY;
-
- flush_work(&ctrl->reset_work);
-
- return 0;
+ WARN_ON(!queue_work(nvme_wq, &ctrl->delete_work));
}
static const struct nvme_ctrl_ops nvme_rdma_ctrl_ops = {
.reg_read32 = nvmf_reg_read32,
.reg_read64 = nvmf_reg_read64,
.reg_write32 = nvmf_reg_write32,
- .reset_ctrl = nvme_rdma_reset_ctrl,
.free_ctrl = nvme_rdma_free_ctrl,
.submit_async_event = nvme_rdma_submit_async_event,
.delete_ctrl = nvme_rdma_del_ctrl,
- .get_subsysnqn = nvmf_get_subsysnqn,
.get_address = nvmf_get_address,
};
nvme_rdma_reconnect_ctrl_work);
INIT_WORK(&ctrl->err_work, nvme_rdma_error_recovery_work);
INIT_WORK(&ctrl->delete_work, nvme_rdma_del_ctrl_work);
- INIT_WORK(&ctrl->reset_work, nvme_rdma_reset_ctrl_work);
- spin_lock_init(&ctrl->lock);
+ INIT_WORK(&ctrl->ctrl.reset_work, nvme_rdma_reset_ctrl_work);
ctrl->queue_count = opts->nr_io_queues + 1; /* +1 for admin queue */
ctrl->ctrl.sqsize = opts->queue_size - 1;
/* sanity check icdoff */
if (ctrl->ctrl.icdoff) {
dev_err(ctrl->ctrl.device, "icdoff is not supported!\n");
+ ret = -EINVAL;
goto out_remove_admin_queue;
}
/* sanity check keyed sgls */
if (!(ctrl->ctrl.sgls & (1 << 20))) {
dev_err(ctrl->ctrl.device, "Mandatory keyed sgls are not support\n");
+ ret = -EINVAL;
goto out_remove_admin_queue;
}
}
mutex_unlock(&nvme_rdma_ctrl_mutex);
- flush_workqueue(nvme_rdma_wq);
+ flush_workqueue(nvme_wq);
}
static struct ib_client nvme_rdma_ib_client = {
{
int ret;
- nvme_rdma_wq = create_workqueue("nvme_rdma_wq");
- if (!nvme_rdma_wq)
- return -ENOMEM;
-
ret = ib_register_client(&nvme_rdma_ib_client);
if (ret)
- goto err_destroy_wq;
+ return ret;
ret = nvmf_register_transport(&nvme_rdma_transport);
if (ret)
err_unreg_client:
ib_unregister_client(&nvme_rdma_ib_client);
-err_destroy_wq:
- destroy_workqueue(nvme_rdma_wq);
return ret;
}
{
nvmf_unregister_transport(&nvme_rdma_transport);
ib_unregister_client(&nvme_rdma_ib_client);
- destroy_workqueue(nvme_rdma_wq);
}
module_init(nvme_rdma_init_module);
+++ /dev/null
-/*
- * NVM Express device driver
- * Copyright (c) 2011-2014, Intel Corporation.
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms and conditions of the GNU General Public License,
- * version 2, as published by the Free Software Foundation.
- *
- * This program is distributed in the hope it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
- * more details.
- */
-
-/*
- * Refer to the SCSI-NVMe Translation spec for details on how
- * each command is translated.
- */
-
-#include <linux/bio.h>
-#include <linux/bitops.h>
-#include <linux/blkdev.h>
-#include <linux/compat.h>
-#include <linux/delay.h>
-#include <linux/errno.h>
-#include <linux/fs.h>
-#include <linux/genhd.h>
-#include <linux/idr.h>
-#include <linux/init.h>
-#include <linux/interrupt.h>
-#include <linux/io.h>
-#include <linux/kdev_t.h>
-#include <linux/kthread.h>
-#include <linux/kernel.h>
-#include <linux/mm.h>
-#include <linux/module.h>
-#include <linux/moduleparam.h>
-#include <linux/pci.h>
-#include <linux/poison.h>
-#include <linux/sched.h>
-#include <linux/slab.h>
-#include <linux/types.h>
-#include <asm/unaligned.h>
-#include <scsi/sg.h>
-#include <scsi/scsi.h>
-#include <scsi/scsi_request.h>
-
-#include "nvme.h"
-
-static int sg_version_num = 30534; /* 2 digits for each component */
-
-/* VPD Page Codes */
-#define VPD_SUPPORTED_PAGES 0x00
-#define VPD_SERIAL_NUMBER 0x80
-#define VPD_DEVICE_IDENTIFIERS 0x83
-#define VPD_EXTENDED_INQUIRY 0x86
-#define VPD_BLOCK_LIMITS 0xB0
-#define VPD_BLOCK_DEV_CHARACTERISTICS 0xB1
-
-/* format unit paramter list offsets */
-#define FORMAT_UNIT_SHORT_PARM_LIST_LEN 4
-#define FORMAT_UNIT_LONG_PARM_LIST_LEN 8
-#define FORMAT_UNIT_PROT_INT_OFFSET 3
-#define FORMAT_UNIT_PROT_FIELD_USAGE_OFFSET 0
-#define FORMAT_UNIT_PROT_FIELD_USAGE_MASK 0x07
-
-/* Misc. defines */
-#define FIXED_SENSE_DATA 0x70
-#define DESC_FORMAT_SENSE_DATA 0x72
-#define FIXED_SENSE_DATA_ADD_LENGTH 10
-#define LUN_ENTRY_SIZE 8
-#define LUN_DATA_HEADER_SIZE 8
-#define ALL_LUNS_RETURNED 0x02
-#define ALL_WELL_KNOWN_LUNS_RETURNED 0x01
-#define RESTRICTED_LUNS_RETURNED 0x00
-#define DOWNLOAD_SAVE_ACTIVATE 0x05
-#define DOWNLOAD_SAVE_DEFER_ACTIVATE 0x0E
-#define ACTIVATE_DEFERRED_MICROCODE 0x0F
-#define FORMAT_UNIT_IMMED_MASK 0x2
-#define FORMAT_UNIT_IMMED_OFFSET 1
-#define KELVIN_TEMP_FACTOR 273
-#define FIXED_FMT_SENSE_DATA_SIZE 18
-#define DESC_FMT_SENSE_DATA_SIZE 8
-
-/* SCSI/NVMe defines and bit masks */
-#define INQ_STANDARD_INQUIRY_PAGE 0x00
-#define INQ_SUPPORTED_VPD_PAGES_PAGE 0x00
-#define INQ_UNIT_SERIAL_NUMBER_PAGE 0x80
-#define INQ_DEVICE_IDENTIFICATION_PAGE 0x83
-#define INQ_EXTENDED_INQUIRY_DATA_PAGE 0x86
-#define INQ_BDEV_LIMITS_PAGE 0xB0
-#define INQ_BDEV_CHARACTERISTICS_PAGE 0xB1
-#define INQ_SERIAL_NUMBER_LENGTH 0x14
-#define INQ_NUM_SUPPORTED_VPD_PAGES 6
-#define VERSION_SPC_4 0x06
-#define ACA_UNSUPPORTED 0
-#define STANDARD_INQUIRY_LENGTH 36
-#define ADDITIONAL_STD_INQ_LENGTH 31
-#define EXTENDED_INQUIRY_DATA_PAGE_LENGTH 0x3C
-#define RESERVED_FIELD 0
-
-/* Mode Sense/Select defines */
-#define MODE_PAGE_INFO_EXCEP 0x1C
-#define MODE_PAGE_CACHING 0x08
-#define MODE_PAGE_CONTROL 0x0A
-#define MODE_PAGE_POWER_CONDITION 0x1A
-#define MODE_PAGE_RETURN_ALL 0x3F
-#define MODE_PAGE_BLK_DES_LEN 0x08
-#define MODE_PAGE_LLBAA_BLK_DES_LEN 0x10
-#define MODE_PAGE_CACHING_LEN 0x14
-#define MODE_PAGE_CONTROL_LEN 0x0C
-#define MODE_PAGE_POW_CND_LEN 0x28
-#define MODE_PAGE_INF_EXC_LEN 0x0C
-#define MODE_PAGE_ALL_LEN 0x54
-#define MODE_SENSE6_MPH_SIZE 4
-#define MODE_SENSE_PAGE_CONTROL_MASK 0xC0
-#define MODE_SENSE_PAGE_CODE_OFFSET 2
-#define MODE_SENSE_PAGE_CODE_MASK 0x3F
-#define MODE_SENSE_LLBAA_MASK 0x10
-#define MODE_SENSE_LLBAA_SHIFT 4
-#define MODE_SENSE_DBD_MASK 8
-#define MODE_SENSE_DBD_SHIFT 3
-#define MODE_SENSE10_MPH_SIZE 8
-#define MODE_SELECT_CDB_PAGE_FORMAT_MASK 0x10
-#define MODE_SELECT_CDB_SAVE_PAGES_MASK 0x1
-#define MODE_SELECT_6_BD_OFFSET 3
-#define MODE_SELECT_10_BD_OFFSET 6
-#define MODE_SELECT_10_LLBAA_OFFSET 4
-#define MODE_SELECT_10_LLBAA_MASK 1
-#define MODE_SELECT_6_MPH_SIZE 4
-#define MODE_SELECT_10_MPH_SIZE 8
-#define CACHING_MODE_PAGE_WCE_MASK 0x04
-#define MODE_SENSE_BLK_DESC_ENABLED 0
-#define MODE_SENSE_BLK_DESC_COUNT 1
-#define MODE_SELECT_PAGE_CODE_MASK 0x3F
-#define SHORT_DESC_BLOCK 8
-#define LONG_DESC_BLOCK 16
-#define MODE_PAGE_POW_CND_LEN_FIELD 0x26
-#define MODE_PAGE_INF_EXC_LEN_FIELD 0x0A
-#define MODE_PAGE_CACHING_LEN_FIELD 0x12
-#define MODE_PAGE_CONTROL_LEN_FIELD 0x0A
-#define MODE_SENSE_PC_CURRENT_VALUES 0
-
-/* Log Sense defines */
-#define LOG_PAGE_SUPPORTED_LOG_PAGES_PAGE 0x00
-#define LOG_PAGE_SUPPORTED_LOG_PAGES_LENGTH 0x07
-#define LOG_PAGE_INFORMATIONAL_EXCEPTIONS_PAGE 0x2F
-#define LOG_PAGE_TEMPERATURE_PAGE 0x0D
-#define LOG_SENSE_CDB_SP_NOT_ENABLED 0
-#define LOG_SENSE_CDB_PC_MASK 0xC0
-#define LOG_SENSE_CDB_PC_SHIFT 6
-#define LOG_SENSE_CDB_PC_CUMULATIVE_VALUES 1
-#define LOG_SENSE_CDB_PAGE_CODE_MASK 0x3F
-#define REMAINING_INFO_EXCP_PAGE_LENGTH 0x8
-#define LOG_INFO_EXCP_PAGE_LENGTH 0xC
-#define REMAINING_TEMP_PAGE_LENGTH 0xC
-#define LOG_TEMP_PAGE_LENGTH 0x10
-#define LOG_TEMP_UNKNOWN 0xFF
-#define SUPPORTED_LOG_PAGES_PAGE_LENGTH 0x3
-
-/* Read Capacity defines */
-#define READ_CAP_10_RESP_SIZE 8
-#define READ_CAP_16_RESP_SIZE 32
-
-/* NVMe Namespace and Command Defines */
-#define BYTES_TO_DWORDS 4
-#define NVME_MAX_FIRMWARE_SLOT 7
-
-/* Report LUNs defines */
-#define REPORT_LUNS_FIRST_LUN_OFFSET 8
-
-/* SCSI ADDITIONAL SENSE Codes */
-
-#define SCSI_ASC_NO_SENSE 0x00
-#define SCSI_ASC_PERIPHERAL_DEV_WRITE_FAULT 0x03
-#define SCSI_ASC_LUN_NOT_READY 0x04
-#define SCSI_ASC_WARNING 0x0B
-#define SCSI_ASC_LOG_BLOCK_GUARD_CHECK_FAILED 0x10
-#define SCSI_ASC_LOG_BLOCK_APPTAG_CHECK_FAILED 0x10
-#define SCSI_ASC_LOG_BLOCK_REFTAG_CHECK_FAILED 0x10
-#define SCSI_ASC_UNRECOVERED_READ_ERROR 0x11
-#define SCSI_ASC_MISCOMPARE_DURING_VERIFY 0x1D
-#define SCSI_ASC_ACCESS_DENIED_INVALID_LUN_ID 0x20
-#define SCSI_ASC_ILLEGAL_COMMAND 0x20
-#define SCSI_ASC_ILLEGAL_BLOCK 0x21
-#define SCSI_ASC_INVALID_CDB 0x24
-#define SCSI_ASC_INVALID_LUN 0x25
-#define SCSI_ASC_INVALID_PARAMETER 0x26
-#define SCSI_ASC_FORMAT_COMMAND_FAILED 0x31
-#define SCSI_ASC_INTERNAL_TARGET_FAILURE 0x44
-
-/* SCSI ADDITIONAL SENSE Code Qualifiers */
-
-#define SCSI_ASCQ_CAUSE_NOT_REPORTABLE 0x00
-#define SCSI_ASCQ_FORMAT_COMMAND_FAILED 0x01
-#define SCSI_ASCQ_LOG_BLOCK_GUARD_CHECK_FAILED 0x01
-#define SCSI_ASCQ_LOG_BLOCK_APPTAG_CHECK_FAILED 0x02
-#define SCSI_ASCQ_LOG_BLOCK_REFTAG_CHECK_FAILED 0x03
-#define SCSI_ASCQ_FORMAT_IN_PROGRESS 0x04
-#define SCSI_ASCQ_POWER_LOSS_EXPECTED 0x08
-#define SCSI_ASCQ_INVALID_LUN_ID 0x09
-
-/* copied from drivers/usb/gadget/function/storage_common.h */
-static inline u32 get_unaligned_be24(u8 *buf)
-{
- return 0xffffff & (u32) get_unaligned_be32(buf - 1);
-}
-
-/* Struct to gather data that needs to be extracted from a SCSI CDB.
- Not conforming to any particular CDB variant, but compatible with all. */
-
-struct nvme_trans_io_cdb {
- u8 fua;
- u8 prot_info;
- u64 lba;
- u32 xfer_len;
-};
-
-
-/* Internal Helper Functions */
-
-
-/* Copy data to userspace memory */
-
-static int nvme_trans_copy_to_user(struct sg_io_hdr *hdr, void *from,
- unsigned long n)
-{
- int i;
- void *index = from;
- size_t remaining = n;
- size_t xfer_len;
-
- if (hdr->iovec_count > 0) {
- struct sg_iovec sgl;
-
- for (i = 0; i < hdr->iovec_count; i++) {
- if (copy_from_user(&sgl, hdr->dxferp +
- i * sizeof(struct sg_iovec),
- sizeof(struct sg_iovec)))
- return -EFAULT;
- xfer_len = min(remaining, sgl.iov_len);
- if (copy_to_user(sgl.iov_base, index, xfer_len))
- return -EFAULT;
-
- index += xfer_len;
- remaining -= xfer_len;
- if (remaining == 0)
- break;
- }
- return 0;
- }
-
- if (copy_to_user(hdr->dxferp, from, n))
- return -EFAULT;
- return 0;
-}
-
-/* Copy data from userspace memory */
-
-static int nvme_trans_copy_from_user(struct sg_io_hdr *hdr, void *to,
- unsigned long n)
-{
- int i;
- void *index = to;
- size_t remaining = n;
- size_t xfer_len;
-
- if (hdr->iovec_count > 0) {
- struct sg_iovec sgl;
-
- for (i = 0; i < hdr->iovec_count; i++) {
- if (copy_from_user(&sgl, hdr->dxferp +
- i * sizeof(struct sg_iovec),
- sizeof(struct sg_iovec)))
- return -EFAULT;
- xfer_len = min(remaining, sgl.iov_len);
- if (copy_from_user(index, sgl.iov_base, xfer_len))
- return -EFAULT;
- index += xfer_len;
- remaining -= xfer_len;
- if (remaining == 0)
- break;
- }
- return 0;
- }
-
- if (copy_from_user(to, hdr->dxferp, n))
- return -EFAULT;
- return 0;
-}
-
-/* Status/Sense Buffer Writeback */
-
-static int nvme_trans_completion(struct sg_io_hdr *hdr, u8 status, u8 sense_key,
- u8 asc, u8 ascq)
-{
- u8 xfer_len;
- u8 resp[DESC_FMT_SENSE_DATA_SIZE];
-
- if (scsi_status_is_good(status)) {
- hdr->status = SAM_STAT_GOOD;
- hdr->masked_status = GOOD;
- hdr->host_status = DID_OK;
- hdr->driver_status = DRIVER_OK;
- hdr->sb_len_wr = 0;
- } else {
- hdr->status = status;
- hdr->masked_status = status >> 1;
- hdr->host_status = DID_OK;
- hdr->driver_status = DRIVER_OK;
-
- memset(resp, 0, DESC_FMT_SENSE_DATA_SIZE);
- resp[0] = DESC_FORMAT_SENSE_DATA;
- resp[1] = sense_key;
- resp[2] = asc;
- resp[3] = ascq;
-
- xfer_len = min_t(u8, hdr->mx_sb_len, DESC_FMT_SENSE_DATA_SIZE);
- hdr->sb_len_wr = xfer_len;
- if (copy_to_user(hdr->sbp, resp, xfer_len) > 0)
- return -EFAULT;
- }
-
- return 0;
-}
-
-/*
- * Take a status code from a lowlevel routine, and if it was a positive NVMe
- * error code update the sense data based on it. In either case the passed
- * in value is returned again, unless an -EFAULT from copy_to_user overrides
- * it.
- */
-static int nvme_trans_status_code(struct sg_io_hdr *hdr, int nvme_sc)
-{
- u8 status, sense_key, asc, ascq;
- int res;
-
- /* For non-nvme (Linux) errors, simply return the error code */
- if (nvme_sc < 0)
- return nvme_sc;
-
- /* Mask DNR, More, and reserved fields */
- switch (nvme_sc & 0x7FF) {
- /* Generic Command Status */
- case NVME_SC_SUCCESS:
- status = SAM_STAT_GOOD;
- sense_key = NO_SENSE;
- asc = SCSI_ASC_NO_SENSE;
- ascq = SCSI_ASCQ_CAUSE_NOT_REPORTABLE;
- break;
- case NVME_SC_INVALID_OPCODE:
- status = SAM_STAT_CHECK_CONDITION;
- sense_key = ILLEGAL_REQUEST;
- asc = SCSI_ASC_ILLEGAL_COMMAND;
- ascq = SCSI_ASCQ_CAUSE_NOT_REPORTABLE;
- break;
- case NVME_SC_INVALID_FIELD:
- status = SAM_STAT_CHECK_CONDITION;
- sense_key = ILLEGAL_REQUEST;
- asc = SCSI_ASC_INVALID_CDB;
- ascq = SCSI_ASCQ_CAUSE_NOT_REPORTABLE;
- break;
- case NVME_SC_DATA_XFER_ERROR:
- status = SAM_STAT_CHECK_CONDITION;
- sense_key = MEDIUM_ERROR;
- asc = SCSI_ASC_NO_SENSE;
- ascq = SCSI_ASCQ_CAUSE_NOT_REPORTABLE;
- break;
- case NVME_SC_POWER_LOSS:
- status = SAM_STAT_TASK_ABORTED;
- sense_key = ABORTED_COMMAND;
- asc = SCSI_ASC_WARNING;
- ascq = SCSI_ASCQ_POWER_LOSS_EXPECTED;
- break;
- case NVME_SC_INTERNAL:
- status = SAM_STAT_CHECK_CONDITION;
- sense_key = HARDWARE_ERROR;
- asc = SCSI_ASC_INTERNAL_TARGET_FAILURE;
- ascq = SCSI_ASCQ_CAUSE_NOT_REPORTABLE;
- break;
- case NVME_SC_ABORT_REQ:
- status = SAM_STAT_TASK_ABORTED;
- sense_key = ABORTED_COMMAND;
- asc = SCSI_ASC_NO_SENSE;
- ascq = SCSI_ASCQ_CAUSE_NOT_REPORTABLE;
- break;
- case NVME_SC_ABORT_QUEUE:
- status = SAM_STAT_TASK_ABORTED;
- sense_key = ABORTED_COMMAND;
- asc = SCSI_ASC_NO_SENSE;
- ascq = SCSI_ASCQ_CAUSE_NOT_REPORTABLE;
- break;
- case NVME_SC_FUSED_FAIL:
- status = SAM_STAT_TASK_ABORTED;
- sense_key = ABORTED_COMMAND;
- asc = SCSI_ASC_NO_SENSE;
- ascq = SCSI_ASCQ_CAUSE_NOT_REPORTABLE;
- break;
- case NVME_SC_FUSED_MISSING:
- status = SAM_STAT_TASK_ABORTED;
- sense_key = ABORTED_COMMAND;
- asc = SCSI_ASC_NO_SENSE;
- ascq = SCSI_ASCQ_CAUSE_NOT_REPORTABLE;
- break;
- case NVME_SC_INVALID_NS:
- status = SAM_STAT_CHECK_CONDITION;
- sense_key = ILLEGAL_REQUEST;
- asc = SCSI_ASC_ACCESS_DENIED_INVALID_LUN_ID;
- ascq = SCSI_ASCQ_INVALID_LUN_ID;
- break;
- case NVME_SC_LBA_RANGE:
- status = SAM_STAT_CHECK_CONDITION;
- sense_key = ILLEGAL_REQUEST;
- asc = SCSI_ASC_ILLEGAL_BLOCK;
- ascq = SCSI_ASCQ_CAUSE_NOT_REPORTABLE;
- break;
- case NVME_SC_CAP_EXCEEDED:
- status = SAM_STAT_CHECK_CONDITION;
- sense_key = MEDIUM_ERROR;
- asc = SCSI_ASC_NO_SENSE;
- ascq = SCSI_ASCQ_CAUSE_NOT_REPORTABLE;
- break;
- case NVME_SC_NS_NOT_READY:
- status = SAM_STAT_CHECK_CONDITION;
- sense_key = NOT_READY;
- asc = SCSI_ASC_LUN_NOT_READY;
- ascq = SCSI_ASCQ_CAUSE_NOT_REPORTABLE;
- break;
-
- /* Command Specific Status */
- case NVME_SC_INVALID_FORMAT:
- status = SAM_STAT_CHECK_CONDITION;
- sense_key = ILLEGAL_REQUEST;
- asc = SCSI_ASC_FORMAT_COMMAND_FAILED;
- ascq = SCSI_ASCQ_FORMAT_COMMAND_FAILED;
- break;
- case NVME_SC_BAD_ATTRIBUTES:
- status = SAM_STAT_CHECK_CONDITION;
- sense_key = ILLEGAL_REQUEST;
- asc = SCSI_ASC_INVALID_CDB;
- ascq = SCSI_ASCQ_CAUSE_NOT_REPORTABLE;
- break;
-
- /* Media Errors */
- case NVME_SC_WRITE_FAULT:
- status = SAM_STAT_CHECK_CONDITION;
- sense_key = MEDIUM_ERROR;
- asc = SCSI_ASC_PERIPHERAL_DEV_WRITE_FAULT;
- ascq = SCSI_ASCQ_CAUSE_NOT_REPORTABLE;
- break;
- case NVME_SC_READ_ERROR:
- status = SAM_STAT_CHECK_CONDITION;
- sense_key = MEDIUM_ERROR;
- asc = SCSI_ASC_UNRECOVERED_READ_ERROR;
- ascq = SCSI_ASCQ_CAUSE_NOT_REPORTABLE;
- break;
- case NVME_SC_GUARD_CHECK:
- status = SAM_STAT_CHECK_CONDITION;
- sense_key = MEDIUM_ERROR;
- asc = SCSI_ASC_LOG_BLOCK_GUARD_CHECK_FAILED;
- ascq = SCSI_ASCQ_LOG_BLOCK_GUARD_CHECK_FAILED;
- break;
- case NVME_SC_APPTAG_CHECK:
- status = SAM_STAT_CHECK_CONDITION;
- sense_key = MEDIUM_ERROR;
- asc = SCSI_ASC_LOG_BLOCK_APPTAG_CHECK_FAILED;
- ascq = SCSI_ASCQ_LOG_BLOCK_APPTAG_CHECK_FAILED;
- break;
- case NVME_SC_REFTAG_CHECK:
- status = SAM_STAT_CHECK_CONDITION;
- sense_key = MEDIUM_ERROR;
- asc = SCSI_ASC_LOG_BLOCK_REFTAG_CHECK_FAILED;
- ascq = SCSI_ASCQ_LOG_BLOCK_REFTAG_CHECK_FAILED;
- break;
- case NVME_SC_COMPARE_FAILED:
- status = SAM_STAT_CHECK_CONDITION;
- sense_key = MISCOMPARE;
- asc = SCSI_ASC_MISCOMPARE_DURING_VERIFY;
- ascq = SCSI_ASCQ_CAUSE_NOT_REPORTABLE;
- break;
- case NVME_SC_ACCESS_DENIED:
- status = SAM_STAT_CHECK_CONDITION;
- sense_key = ILLEGAL_REQUEST;
- asc = SCSI_ASC_ACCESS_DENIED_INVALID_LUN_ID;
- ascq = SCSI_ASCQ_INVALID_LUN_ID;
- break;
-
- /* Unspecified/Default */
- case NVME_SC_CMDID_CONFLICT:
- case NVME_SC_CMD_SEQ_ERROR:
- case NVME_SC_CQ_INVALID:
- case NVME_SC_QID_INVALID:
- case NVME_SC_QUEUE_SIZE:
- case NVME_SC_ABORT_LIMIT:
- case NVME_SC_ABORT_MISSING:
- case NVME_SC_ASYNC_LIMIT:
- case NVME_SC_FIRMWARE_SLOT:
- case NVME_SC_FIRMWARE_IMAGE:
- case NVME_SC_INVALID_VECTOR:
- case NVME_SC_INVALID_LOG_PAGE:
- default:
- status = SAM_STAT_CHECK_CONDITION;
- sense_key = ILLEGAL_REQUEST;
- asc = SCSI_ASC_NO_SENSE;
- ascq = SCSI_ASCQ_CAUSE_NOT_REPORTABLE;
- break;
- }
-
- res = nvme_trans_completion(hdr, status, sense_key, asc, ascq);
- return res ? res : nvme_sc;
-}
-
-/* INQUIRY Helper Functions */
-
-static int nvme_trans_standard_inquiry_page(struct nvme_ns *ns,
- struct sg_io_hdr *hdr, u8 *inq_response,
- int alloc_len)
-{
- struct nvme_ctrl *ctrl = ns->ctrl;
- struct nvme_id_ns *id_ns;
- int res;
- int nvme_sc;
- int xfer_len;
- u8 resp_data_format = 0x02;
- u8 protect;
- u8 cmdque = 0x01 << 1;
- u8 fw_offset = sizeof(ctrl->firmware_rev);
-
- /* nvme ns identify - use DPS value for PROTECT field */
- nvme_sc = nvme_identify_ns(ctrl, ns->ns_id, &id_ns);
- res = nvme_trans_status_code(hdr, nvme_sc);
- if (res)
- return res;
-
- if (id_ns->dps)
- protect = 0x01;
- else
- protect = 0;
- kfree(id_ns);
-
- memset(inq_response, 0, STANDARD_INQUIRY_LENGTH);
- inq_response[2] = VERSION_SPC_4;
- inq_response[3] = resp_data_format; /*normaca=0 | hisup=0 */
- inq_response[4] = ADDITIONAL_STD_INQ_LENGTH;
- inq_response[5] = protect; /* sccs=0 | acc=0 | tpgs=0 | pc3=0 */
- inq_response[7] = cmdque; /* wbus16=0 | sync=0 | vs=0 */
- strncpy(&inq_response[8], "NVMe ", 8);
- strncpy(&inq_response[16], ctrl->model, 16);
-
- while (ctrl->firmware_rev[fw_offset - 1] == ' ' && fw_offset > 4)
- fw_offset--;
- fw_offset -= 4;
- strncpy(&inq_response[32], ctrl->firmware_rev + fw_offset, 4);
-
- xfer_len = min(alloc_len, STANDARD_INQUIRY_LENGTH);
- return nvme_trans_copy_to_user(hdr, inq_response, xfer_len);
-}
-
-static int nvme_trans_supported_vpd_pages(struct nvme_ns *ns,
- struct sg_io_hdr *hdr, u8 *inq_response,
- int alloc_len)
-{
- int xfer_len;
-
- memset(inq_response, 0, STANDARD_INQUIRY_LENGTH);
- inq_response[1] = INQ_SUPPORTED_VPD_PAGES_PAGE; /* Page Code */
- inq_response[3] = INQ_NUM_SUPPORTED_VPD_PAGES; /* Page Length */
- inq_response[4] = INQ_SUPPORTED_VPD_PAGES_PAGE;
- inq_response[5] = INQ_UNIT_SERIAL_NUMBER_PAGE;
- inq_response[6] = INQ_DEVICE_IDENTIFICATION_PAGE;
- inq_response[7] = INQ_EXTENDED_INQUIRY_DATA_PAGE;
- inq_response[8] = INQ_BDEV_CHARACTERISTICS_PAGE;
- inq_response[9] = INQ_BDEV_LIMITS_PAGE;
-
- xfer_len = min(alloc_len, STANDARD_INQUIRY_LENGTH);
- return nvme_trans_copy_to_user(hdr, inq_response, xfer_len);
-}
-
-static int nvme_trans_unit_serial_page(struct nvme_ns *ns,
- struct sg_io_hdr *hdr, u8 *inq_response,
- int alloc_len)
-{
- int xfer_len;
-
- memset(inq_response, 0, STANDARD_INQUIRY_LENGTH);
- inq_response[1] = INQ_UNIT_SERIAL_NUMBER_PAGE; /* Page Code */
- inq_response[3] = INQ_SERIAL_NUMBER_LENGTH; /* Page Length */
- strncpy(&inq_response[4], ns->ctrl->serial, INQ_SERIAL_NUMBER_LENGTH);
-
- xfer_len = min(alloc_len, STANDARD_INQUIRY_LENGTH);
- return nvme_trans_copy_to_user(hdr, inq_response, xfer_len);
-}
-
-static int nvme_fill_device_id_eui64(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 *inq_response, int alloc_len)
-{
- struct nvme_id_ns *id_ns;
- int nvme_sc, res;
- size_t len;
- void *eui;
-
- nvme_sc = nvme_identify_ns(ns->ctrl, ns->ns_id, &id_ns);
- res = nvme_trans_status_code(hdr, nvme_sc);
- if (res)
- return res;
-
- eui = id_ns->eui64;
- len = sizeof(id_ns->eui64);
-
- if (ns->ctrl->vs >= NVME_VS(1, 2, 0)) {
- if (bitmap_empty(eui, len * 8)) {
- eui = id_ns->nguid;
- len = sizeof(id_ns->nguid);
- }
- }
-
- if (bitmap_empty(eui, len * 8)) {
- res = -EOPNOTSUPP;
- goto out_free_id;
- }
-
- memset(inq_response, 0, alloc_len);
- inq_response[1] = INQ_DEVICE_IDENTIFICATION_PAGE;
- inq_response[3] = 4 + len; /* Page Length */
-
- /* Designation Descriptor start */
- inq_response[4] = 0x01; /* Proto ID=0h | Code set=1h */
- inq_response[5] = 0x02; /* PIV=0b | Asso=00b | Designator Type=2h */
- inq_response[6] = 0x00; /* Rsvd */
- inq_response[7] = len; /* Designator Length */
- memcpy(&inq_response[8], eui, len);
-
- res = nvme_trans_copy_to_user(hdr, inq_response, alloc_len);
-out_free_id:
- kfree(id_ns);
- return res;
-}
-
-static int nvme_fill_device_id_scsi_string(struct nvme_ns *ns,
- struct sg_io_hdr *hdr, u8 *inq_response, int alloc_len)
-{
- struct nvme_ctrl *ctrl = ns->ctrl;
- struct nvme_id_ctrl *id_ctrl;
- int nvme_sc, res;
-
- if (alloc_len < 72) {
- return nvme_trans_completion(hdr,
- SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- }
-
- nvme_sc = nvme_identify_ctrl(ctrl, &id_ctrl);
- res = nvme_trans_status_code(hdr, nvme_sc);
- if (res)
- return res;
-
- memset(inq_response, 0, alloc_len);
- inq_response[1] = INQ_DEVICE_IDENTIFICATION_PAGE;
- inq_response[3] = 0x48; /* Page Length */
-
- /* Designation Descriptor start */
- inq_response[4] = 0x03; /* Proto ID=0h | Code set=3h */
- inq_response[5] = 0x08; /* PIV=0b | Asso=00b | Designator Type=8h */
- inq_response[6] = 0x00; /* Rsvd */
- inq_response[7] = 0x44; /* Designator Length */
-
- sprintf(&inq_response[8], "%04x", le16_to_cpu(id_ctrl->vid));
- memcpy(&inq_response[12], ctrl->model, sizeof(ctrl->model));
- sprintf(&inq_response[52], "%04x", cpu_to_be32(ns->ns_id));
- memcpy(&inq_response[56], ctrl->serial, sizeof(ctrl->serial));
-
- res = nvme_trans_copy_to_user(hdr, inq_response, alloc_len);
- kfree(id_ctrl);
- return res;
-}
-
-static int nvme_trans_device_id_page(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 *resp, int alloc_len)
-{
- int res;
-
- if (ns->ctrl->vs >= NVME_VS(1, 1, 0)) {
- res = nvme_fill_device_id_eui64(ns, hdr, resp, alloc_len);
- if (res != -EOPNOTSUPP)
- return res;
- }
-
- return nvme_fill_device_id_scsi_string(ns, hdr, resp, alloc_len);
-}
-
-static int nvme_trans_ext_inq_page(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- int alloc_len)
-{
- u8 *inq_response;
- int res;
- int nvme_sc;
- struct nvme_ctrl *ctrl = ns->ctrl;
- struct nvme_id_ctrl *id_ctrl;
- struct nvme_id_ns *id_ns;
- int xfer_len;
- u8 microcode = 0x80;
- u8 spt;
- u8 spt_lut[8] = {0, 0, 2, 1, 4, 6, 5, 7};
- u8 grd_chk, app_chk, ref_chk, protect;
- u8 uask_sup = 0x20;
- u8 v_sup;
- u8 luiclr = 0x01;
-
- inq_response = kmalloc(EXTENDED_INQUIRY_DATA_PAGE_LENGTH, GFP_KERNEL);
- if (inq_response == NULL)
- return -ENOMEM;
-
- nvme_sc = nvme_identify_ns(ctrl, ns->ns_id, &id_ns);
- res = nvme_trans_status_code(hdr, nvme_sc);
- if (res)
- goto out_free_inq;
-
- spt = spt_lut[id_ns->dpc & 0x07] << 3;
- if (id_ns->dps)
- protect = 0x01;
- else
- protect = 0;
- kfree(id_ns);
-
- grd_chk = protect << 2;
- app_chk = protect << 1;
- ref_chk = protect;
-
- nvme_sc = nvme_identify_ctrl(ctrl, &id_ctrl);
- res = nvme_trans_status_code(hdr, nvme_sc);
- if (res)
- goto out_free_inq;
-
- v_sup = id_ctrl->vwc;
- kfree(id_ctrl);
-
- memset(inq_response, 0, EXTENDED_INQUIRY_DATA_PAGE_LENGTH);
- inq_response[1] = INQ_EXTENDED_INQUIRY_DATA_PAGE; /* Page Code */
- inq_response[2] = 0x00; /* Page Length MSB */
- inq_response[3] = 0x3C; /* Page Length LSB */
- inq_response[4] = microcode | spt | grd_chk | app_chk | ref_chk;
- inq_response[5] = uask_sup;
- inq_response[6] = v_sup;
- inq_response[7] = luiclr;
- inq_response[8] = 0;
- inq_response[9] = 0;
-
- xfer_len = min(alloc_len, EXTENDED_INQUIRY_DATA_PAGE_LENGTH);
- res = nvme_trans_copy_to_user(hdr, inq_response, xfer_len);
-
- out_free_inq:
- kfree(inq_response);
- return res;
-}
-
-static int nvme_trans_bdev_limits_page(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 *inq_response, int alloc_len)
-{
- __be32 max_sectors = cpu_to_be32(
- nvme_block_nr(ns, queue_max_hw_sectors(ns->queue)));
- __be32 max_discard = cpu_to_be32(ns->queue->limits.max_discard_sectors);
- __be32 discard_desc_count = cpu_to_be32(0x100);
-
- memset(inq_response, 0, STANDARD_INQUIRY_LENGTH);
- inq_response[1] = VPD_BLOCK_LIMITS;
- inq_response[3] = 0x3c; /* Page Length */
- memcpy(&inq_response[8], &max_sectors, sizeof(u32));
- memcpy(&inq_response[20], &max_discard, sizeof(u32));
-
- if (max_discard)
- memcpy(&inq_response[24], &discard_desc_count, sizeof(u32));
-
- return nvme_trans_copy_to_user(hdr, inq_response, 0x3c);
-}
-
-static int nvme_trans_bdev_char_page(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- int alloc_len)
-{
- u8 *inq_response;
- int res;
- int xfer_len;
-
- inq_response = kzalloc(EXTENDED_INQUIRY_DATA_PAGE_LENGTH, GFP_KERNEL);
- if (inq_response == NULL) {
- res = -ENOMEM;
- goto out_mem;
- }
-
- inq_response[1] = INQ_BDEV_CHARACTERISTICS_PAGE; /* Page Code */
- inq_response[2] = 0x00; /* Page Length MSB */
- inq_response[3] = 0x3C; /* Page Length LSB */
- inq_response[4] = 0x00; /* Medium Rotation Rate MSB */
- inq_response[5] = 0x01; /* Medium Rotation Rate LSB */
- inq_response[6] = 0x00; /* Form Factor */
-
- xfer_len = min(alloc_len, EXTENDED_INQUIRY_DATA_PAGE_LENGTH);
- res = nvme_trans_copy_to_user(hdr, inq_response, xfer_len);
-
- kfree(inq_response);
- out_mem:
- return res;
-}
-
-/* LOG SENSE Helper Functions */
-
-static int nvme_trans_log_supp_pages(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- int alloc_len)
-{
- int res;
- int xfer_len;
- u8 *log_response;
-
- log_response = kzalloc(LOG_PAGE_SUPPORTED_LOG_PAGES_LENGTH, GFP_KERNEL);
- if (log_response == NULL) {
- res = -ENOMEM;
- goto out_mem;
- }
-
- log_response[0] = LOG_PAGE_SUPPORTED_LOG_PAGES_PAGE;
- /* Subpage=0x00, Page Length MSB=0 */
- log_response[3] = SUPPORTED_LOG_PAGES_PAGE_LENGTH;
- log_response[4] = LOG_PAGE_SUPPORTED_LOG_PAGES_PAGE;
- log_response[5] = LOG_PAGE_INFORMATIONAL_EXCEPTIONS_PAGE;
- log_response[6] = LOG_PAGE_TEMPERATURE_PAGE;
-
- xfer_len = min(alloc_len, LOG_PAGE_SUPPORTED_LOG_PAGES_LENGTH);
- res = nvme_trans_copy_to_user(hdr, log_response, xfer_len);
-
- kfree(log_response);
- out_mem:
- return res;
-}
-
-static int nvme_trans_log_info_exceptions(struct nvme_ns *ns,
- struct sg_io_hdr *hdr, int alloc_len)
-{
- int res;
- int xfer_len;
- u8 *log_response;
- struct nvme_smart_log *smart_log;
- u8 temp_c;
- u16 temp_k;
-
- log_response = kzalloc(LOG_INFO_EXCP_PAGE_LENGTH, GFP_KERNEL);
- if (log_response == NULL)
- return -ENOMEM;
-
- res = nvme_get_log_page(ns->ctrl, &smart_log);
- if (res < 0)
- goto out_free_response;
-
- if (res != NVME_SC_SUCCESS) {
- temp_c = LOG_TEMP_UNKNOWN;
- } else {
- temp_k = (smart_log->temperature[1] << 8) +
- (smart_log->temperature[0]);
- temp_c = temp_k - KELVIN_TEMP_FACTOR;
- }
- kfree(smart_log);
-
- log_response[0] = LOG_PAGE_INFORMATIONAL_EXCEPTIONS_PAGE;
- /* Subpage=0x00, Page Length MSB=0 */
- log_response[3] = REMAINING_INFO_EXCP_PAGE_LENGTH;
- /* Informational Exceptions Log Parameter 1 Start */
- /* Parameter Code=0x0000 bytes 4,5 */
- log_response[6] = 0x23; /* DU=0, TSD=1, ETC=0, TMC=0, FMT_AND_LNK=11b */
- log_response[7] = 0x04; /* PARAMETER LENGTH */
- /* Add sense Code and qualifier = 0x00 each */
- /* Use Temperature from NVMe Get Log Page, convert to C from K */
- log_response[10] = temp_c;
-
- xfer_len = min(alloc_len, LOG_INFO_EXCP_PAGE_LENGTH);
- res = nvme_trans_copy_to_user(hdr, log_response, xfer_len);
-
- out_free_response:
- kfree(log_response);
- return res;
-}
-
-static int nvme_trans_log_temperature(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- int alloc_len)
-{
- int res;
- int xfer_len;
- u8 *log_response;
- struct nvme_smart_log *smart_log;
- u32 feature_resp;
- u8 temp_c_cur, temp_c_thresh;
- u16 temp_k;
-
- log_response = kzalloc(LOG_TEMP_PAGE_LENGTH, GFP_KERNEL);
- if (log_response == NULL)
- return -ENOMEM;
-
- res = nvme_get_log_page(ns->ctrl, &smart_log);
- if (res < 0)
- goto out_free_response;
-
- if (res != NVME_SC_SUCCESS) {
- temp_c_cur = LOG_TEMP_UNKNOWN;
- } else {
- temp_k = (smart_log->temperature[1] << 8) +
- (smart_log->temperature[0]);
- temp_c_cur = temp_k - KELVIN_TEMP_FACTOR;
- }
- kfree(smart_log);
-
- /* Get Features for Temp Threshold */
- res = nvme_get_features(ns->ctrl, NVME_FEAT_TEMP_THRESH, 0, NULL, 0,
- &feature_resp);
- if (res != NVME_SC_SUCCESS)
- temp_c_thresh = LOG_TEMP_UNKNOWN;
- else
- temp_c_thresh = (feature_resp & 0xFFFF) - KELVIN_TEMP_FACTOR;
-
- log_response[0] = LOG_PAGE_TEMPERATURE_PAGE;
- /* Subpage=0x00, Page Length MSB=0 */
- log_response[3] = REMAINING_TEMP_PAGE_LENGTH;
- /* Temperature Log Parameter 1 (Temperature) Start */
- /* Parameter Code = 0x0000 */
- log_response[6] = 0x01; /* Format and Linking = 01b */
- log_response[7] = 0x02; /* Parameter Length */
- /* Use Temperature from NVMe Get Log Page, convert to C from K */
- log_response[9] = temp_c_cur;
- /* Temperature Log Parameter 2 (Reference Temperature) Start */
- log_response[11] = 0x01; /* Parameter Code = 0x0001 */
- log_response[12] = 0x01; /* Format and Linking = 01b */
- log_response[13] = 0x02; /* Parameter Length */
- /* Use Temperature Thresh from NVMe Get Log Page, convert to C from K */
- log_response[15] = temp_c_thresh;
-
- xfer_len = min(alloc_len, LOG_TEMP_PAGE_LENGTH);
- res = nvme_trans_copy_to_user(hdr, log_response, xfer_len);
-
- out_free_response:
- kfree(log_response);
- return res;
-}
-
-/* MODE SENSE Helper Functions */
-
-static int nvme_trans_fill_mode_parm_hdr(u8 *resp, int len, u8 cdb10, u8 llbaa,
- u16 mode_data_length, u16 blk_desc_len)
-{
- /* Quick check to make sure I don't stomp on my own memory... */
- if ((cdb10 && len < 8) || (!cdb10 && len < 4))
- return -EINVAL;
-
- if (cdb10) {
- resp[0] = (mode_data_length & 0xFF00) >> 8;
- resp[1] = (mode_data_length & 0x00FF);
- resp[3] = 0x10 /* DPOFUA */;
- resp[4] = llbaa;
- resp[5] = RESERVED_FIELD;
- resp[6] = (blk_desc_len & 0xFF00) >> 8;
- resp[7] = (blk_desc_len & 0x00FF);
- } else {
- resp[0] = (mode_data_length & 0x00FF);
- resp[2] = 0x10 /* DPOFUA */;
- resp[3] = (blk_desc_len & 0x00FF);
- }
-
- return 0;
-}
-
-static int nvme_trans_fill_blk_desc(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 *resp, int len, u8 llbaa)
-{
- int res;
- int nvme_sc;
- struct nvme_id_ns *id_ns;
- u8 flbas;
- u32 lba_length;
-
- if (llbaa == 0 && len < MODE_PAGE_BLK_DES_LEN)
- return -EINVAL;
- else if (llbaa > 0 && len < MODE_PAGE_LLBAA_BLK_DES_LEN)
- return -EINVAL;
-
- nvme_sc = nvme_identify_ns(ns->ctrl, ns->ns_id, &id_ns);
- res = nvme_trans_status_code(hdr, nvme_sc);
- if (res)
- return res;
-
- flbas = (id_ns->flbas) & 0x0F;
- lba_length = (1 << (id_ns->lbaf[flbas].ds));
-
- if (llbaa == 0) {
- __be32 tmp_cap = cpu_to_be32(le64_to_cpu(id_ns->ncap));
- /* Byte 4 is reserved */
- __be32 tmp_len = cpu_to_be32(lba_length & 0x00FFFFFF);
-
- memcpy(resp, &tmp_cap, sizeof(u32));
- memcpy(&resp[4], &tmp_len, sizeof(u32));
- } else {
- __be64 tmp_cap = cpu_to_be64(le64_to_cpu(id_ns->ncap));
- __be32 tmp_len = cpu_to_be32(lba_length);
-
- memcpy(resp, &tmp_cap, sizeof(u64));
- /* Bytes 8, 9, 10, 11 are reserved */
- memcpy(&resp[12], &tmp_len, sizeof(u32));
- }
-
- kfree(id_ns);
- return res;
-}
-
-static int nvme_trans_fill_control_page(struct nvme_ns *ns,
- struct sg_io_hdr *hdr, u8 *resp,
- int len)
-{
- if (len < MODE_PAGE_CONTROL_LEN)
- return -EINVAL;
-
- resp[0] = MODE_PAGE_CONTROL;
- resp[1] = MODE_PAGE_CONTROL_LEN_FIELD;
- resp[2] = 0x0E; /* TST=000b, TMF_ONLY=0, DPICZ=1,
- * D_SENSE=1, GLTSD=1, RLEC=0 */
- resp[3] = 0x12; /* Q_ALGO_MODIFIER=1h, NUAR=0, QERR=01b */
- /* Byte 4: VS=0, RAC=0, UA_INT=0, SWP=0 */
- resp[5] = 0x40; /* ATO=0, TAS=1, ATMPE=0, RWWP=0, AUTOLOAD=0 */
- /* resp[6] and [7] are obsolete, thus zero */
- resp[8] = 0xFF; /* Busy timeout period = 0xffff */
- resp[9] = 0xFF;
- /* Bytes 10,11: Extended selftest completion time = 0x0000 */
-
- return 0;
-}
-
-static int nvme_trans_fill_caching_page(struct nvme_ns *ns,
- struct sg_io_hdr *hdr,
- u8 *resp, int len)
-{
- int res = 0;
- int nvme_sc;
- u32 feature_resp;
- u8 vwc;
-
- if (len < MODE_PAGE_CACHING_LEN)
- return -EINVAL;
-
- nvme_sc = nvme_get_features(ns->ctrl, NVME_FEAT_VOLATILE_WC, 0, NULL, 0,
- &feature_resp);
- res = nvme_trans_status_code(hdr, nvme_sc);
- if (res)
- return res;
-
- vwc = feature_resp & 0x00000001;
-
- resp[0] = MODE_PAGE_CACHING;
- resp[1] = MODE_PAGE_CACHING_LEN_FIELD;
- resp[2] = vwc << 2;
- return 0;
-}
-
-static int nvme_trans_fill_pow_cnd_page(struct nvme_ns *ns,
- struct sg_io_hdr *hdr, u8 *resp,
- int len)
-{
- if (len < MODE_PAGE_POW_CND_LEN)
- return -EINVAL;
-
- resp[0] = MODE_PAGE_POWER_CONDITION;
- resp[1] = MODE_PAGE_POW_CND_LEN_FIELD;
- /* All other bytes are zero */
-
- return 0;
-}
-
-static int nvme_trans_fill_inf_exc_page(struct nvme_ns *ns,
- struct sg_io_hdr *hdr, u8 *resp,
- int len)
-{
- if (len < MODE_PAGE_INF_EXC_LEN)
- return -EINVAL;
-
- resp[0] = MODE_PAGE_INFO_EXCEP;
- resp[1] = MODE_PAGE_INF_EXC_LEN_FIELD;
- resp[2] = 0x88;
- /* All other bytes are zero */
-
- return 0;
-}
-
-static int nvme_trans_fill_all_pages(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 *resp, int len)
-{
- int res;
- u16 mode_pages_offset_1 = 0;
- u16 mode_pages_offset_2, mode_pages_offset_3, mode_pages_offset_4;
-
- mode_pages_offset_2 = mode_pages_offset_1 + MODE_PAGE_CACHING_LEN;
- mode_pages_offset_3 = mode_pages_offset_2 + MODE_PAGE_CONTROL_LEN;
- mode_pages_offset_4 = mode_pages_offset_3 + MODE_PAGE_POW_CND_LEN;
-
- res = nvme_trans_fill_caching_page(ns, hdr, &resp[mode_pages_offset_1],
- MODE_PAGE_CACHING_LEN);
- if (res)
- return res;
- res = nvme_trans_fill_control_page(ns, hdr, &resp[mode_pages_offset_2],
- MODE_PAGE_CONTROL_LEN);
- if (res)
- return res;
- res = nvme_trans_fill_pow_cnd_page(ns, hdr, &resp[mode_pages_offset_3],
- MODE_PAGE_POW_CND_LEN);
- if (res)
- return res;
- return nvme_trans_fill_inf_exc_page(ns, hdr, &resp[mode_pages_offset_4],
- MODE_PAGE_INF_EXC_LEN);
-}
-
-static inline int nvme_trans_get_blk_desc_len(u8 dbd, u8 llbaa)
-{
- if (dbd == MODE_SENSE_BLK_DESC_ENABLED) {
- /* SPC-4: len = 8 x Num_of_descriptors if llbaa = 0, 16x if 1 */
- return 8 * (llbaa + 1) * MODE_SENSE_BLK_DESC_COUNT;
- } else {
- return 0;
- }
-}
-
-static int nvme_trans_mode_page_create(struct nvme_ns *ns,
- struct sg_io_hdr *hdr, u8 *cmd,
- u16 alloc_len, u8 cdb10,
- int (*mode_page_fill_func)
- (struct nvme_ns *,
- struct sg_io_hdr *hdr, u8 *, int),
- u16 mode_pages_tot_len)
-{
- int res;
- int xfer_len;
- u8 *response;
- u8 dbd, llbaa;
- u16 resp_size;
- int mph_size;
- u16 mode_pages_offset_1;
- u16 blk_desc_len, blk_desc_offset, mode_data_length;
-
- dbd = (cmd[1] & MODE_SENSE_DBD_MASK) >> MODE_SENSE_DBD_SHIFT;
- llbaa = (cmd[1] & MODE_SENSE_LLBAA_MASK) >> MODE_SENSE_LLBAA_SHIFT;
- mph_size = cdb10 ? MODE_SENSE10_MPH_SIZE : MODE_SENSE6_MPH_SIZE;
-
- blk_desc_len = nvme_trans_get_blk_desc_len(dbd, llbaa);
-
- resp_size = mph_size + blk_desc_len + mode_pages_tot_len;
- /* Refer spc4r34 Table 440 for calculation of Mode data Length field */
- mode_data_length = 3 + (3 * cdb10) + blk_desc_len + mode_pages_tot_len;
-
- blk_desc_offset = mph_size;
- mode_pages_offset_1 = blk_desc_offset + blk_desc_len;
-
- response = kzalloc(resp_size, GFP_KERNEL);
- if (response == NULL) {
- res = -ENOMEM;
- goto out_mem;
- }
-
- res = nvme_trans_fill_mode_parm_hdr(&response[0], mph_size, cdb10,
- llbaa, mode_data_length, blk_desc_len);
- if (res)
- goto out_free;
- if (blk_desc_len > 0) {
- res = nvme_trans_fill_blk_desc(ns, hdr,
- &response[blk_desc_offset],
- blk_desc_len, llbaa);
- if (res)
- goto out_free;
- }
- res = mode_page_fill_func(ns, hdr, &response[mode_pages_offset_1],
- mode_pages_tot_len);
- if (res)
- goto out_free;
-
- xfer_len = min(alloc_len, resp_size);
- res = nvme_trans_copy_to_user(hdr, response, xfer_len);
-
- out_free:
- kfree(response);
- out_mem:
- return res;
-}
-
-/* Read Capacity Helper Functions */
-
-static void nvme_trans_fill_read_cap(u8 *response, struct nvme_id_ns *id_ns,
- u8 cdb16)
-{
- u8 flbas;
- u32 lba_length;
- u64 rlba;
- u8 prot_en;
- u8 p_type_lut[4] = {0, 0, 1, 2};
- __be64 tmp_rlba;
- __be32 tmp_rlba_32;
- __be32 tmp_len;
-
- flbas = (id_ns->flbas) & 0x0F;
- lba_length = (1 << (id_ns->lbaf[flbas].ds));
- rlba = le64_to_cpup(&id_ns->nsze) - 1;
- (id_ns->dps) ? (prot_en = 0x01) : (prot_en = 0);
-
- if (!cdb16) {
- if (rlba > 0xFFFFFFFF)
- rlba = 0xFFFFFFFF;
- tmp_rlba_32 = cpu_to_be32(rlba);
- tmp_len = cpu_to_be32(lba_length);
- memcpy(response, &tmp_rlba_32, sizeof(u32));
- memcpy(&response[4], &tmp_len, sizeof(u32));
- } else {
- tmp_rlba = cpu_to_be64(rlba);
- tmp_len = cpu_to_be32(lba_length);
- memcpy(response, &tmp_rlba, sizeof(u64));
- memcpy(&response[8], &tmp_len, sizeof(u32));
- response[12] = (p_type_lut[id_ns->dps & 0x3] << 1) | prot_en;
- /* P_I_Exponent = 0x0 | LBPPBE = 0x0 */
- /* LBPME = 0 | LBPRZ = 0 | LALBA = 0x00 */
- /* Bytes 16-31 - Reserved */
- }
-}
-
-/* Start Stop Unit Helper Functions */
-
-static int nvme_trans_send_activate_fw_cmd(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 buffer_id)
-{
- struct nvme_command c;
- int nvme_sc;
-
- memset(&c, 0, sizeof(c));
- c.common.opcode = nvme_admin_activate_fw;
- c.common.cdw10[0] = cpu_to_le32(buffer_id | NVME_FWACT_REPL_ACTV);
-
- nvme_sc = nvme_submit_sync_cmd(ns->queue, &c, NULL, 0);
- return nvme_trans_status_code(hdr, nvme_sc);
-}
-
-static int nvme_trans_send_download_fw_cmd(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 opcode, u32 tot_len, u32 offset,
- u8 buffer_id)
-{
- int nvme_sc;
- struct nvme_command c;
-
- if (hdr->iovec_count > 0) {
- /* Assuming SGL is not allowed for this command */
- return nvme_trans_completion(hdr,
- SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST,
- SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- }
-
- memset(&c, 0, sizeof(c));
- c.common.opcode = nvme_admin_download_fw;
- c.dlfw.numd = cpu_to_le32((tot_len/BYTES_TO_DWORDS) - 1);
- c.dlfw.offset = cpu_to_le32(offset/BYTES_TO_DWORDS);
-
- nvme_sc = nvme_submit_user_cmd(ns->ctrl->admin_q, &c,
- hdr->dxferp, tot_len, NULL, 0);
- return nvme_trans_status_code(hdr, nvme_sc);
-}
-
-/* Mode Select Helper Functions */
-
-static inline void nvme_trans_modesel_get_bd_len(u8 *parm_list, u8 cdb10,
- u16 *bd_len, u8 *llbaa)
-{
- if (cdb10) {
- /* 10 Byte CDB */
- *bd_len = (parm_list[MODE_SELECT_10_BD_OFFSET] << 8) +
- parm_list[MODE_SELECT_10_BD_OFFSET + 1];
- *llbaa = parm_list[MODE_SELECT_10_LLBAA_OFFSET] &
- MODE_SELECT_10_LLBAA_MASK;
- } else {
- /* 6 Byte CDB */
- *bd_len = parm_list[MODE_SELECT_6_BD_OFFSET];
- }
-}
-
-static void nvme_trans_modesel_save_bd(struct nvme_ns *ns, u8 *parm_list,
- u16 idx, u16 bd_len, u8 llbaa)
-{
- /* Store block descriptor info if a FORMAT UNIT comes later */
- /* TODO Saving 1st BD info; what to do if multiple BD received? */
- if (llbaa == 0) {
- /* Standard Block Descriptor - spc4r34 7.5.5.1 */
- ns->mode_select_num_blocks =
- (parm_list[idx + 1] << 16) +
- (parm_list[idx + 2] << 8) +
- (parm_list[idx + 3]);
-
- ns->mode_select_block_len =
- (parm_list[idx + 5] << 16) +
- (parm_list[idx + 6] << 8) +
- (parm_list[idx + 7]);
- } else {
- /* Long LBA Block Descriptor - sbc3r27 6.4.2.3 */
- ns->mode_select_num_blocks =
- (((u64)parm_list[idx + 0]) << 56) +
- (((u64)parm_list[idx + 1]) << 48) +
- (((u64)parm_list[idx + 2]) << 40) +
- (((u64)parm_list[idx + 3]) << 32) +
- (((u64)parm_list[idx + 4]) << 24) +
- (((u64)parm_list[idx + 5]) << 16) +
- (((u64)parm_list[idx + 6]) << 8) +
- ((u64)parm_list[idx + 7]);
-
- ns->mode_select_block_len =
- (parm_list[idx + 12] << 24) +
- (parm_list[idx + 13] << 16) +
- (parm_list[idx + 14] << 8) +
- (parm_list[idx + 15]);
- }
-}
-
-static int nvme_trans_modesel_get_mp(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 *mode_page, u8 page_code)
-{
- int res = 0;
- int nvme_sc;
- unsigned dword11;
-
- switch (page_code) {
- case MODE_PAGE_CACHING:
- dword11 = ((mode_page[2] & CACHING_MODE_PAGE_WCE_MASK) ? 1 : 0);
- nvme_sc = nvme_set_features(ns->ctrl, NVME_FEAT_VOLATILE_WC,
- dword11, NULL, 0, NULL);
- res = nvme_trans_status_code(hdr, nvme_sc);
- break;
- case MODE_PAGE_CONTROL:
- break;
- case MODE_PAGE_POWER_CONDITION:
- /* Verify the OS is not trying to set timers */
- if ((mode_page[2] & 0x01) != 0 || (mode_page[3] & 0x0F) != 0) {
- res = nvme_trans_completion(hdr,
- SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST,
- SCSI_ASC_INVALID_PARAMETER,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- break;
- }
- break;
- default:
- res = nvme_trans_completion(hdr, SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- break;
- }
-
- return res;
-}
-
-static int nvme_trans_modesel_data(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 *cmd, u16 parm_list_len, u8 pf,
- u8 sp, u8 cdb10)
-{
- int res;
- u8 *parm_list;
- u16 bd_len;
- u8 llbaa = 0;
- u16 index, saved_index;
- u8 page_code;
- u16 mp_size;
-
- /* Get parm list from data-in/out buffer */
- parm_list = kmalloc(parm_list_len, GFP_KERNEL);
- if (parm_list == NULL) {
- res = -ENOMEM;
- goto out;
- }
-
- res = nvme_trans_copy_from_user(hdr, parm_list, parm_list_len);
- if (res)
- goto out_mem;
-
- nvme_trans_modesel_get_bd_len(parm_list, cdb10, &bd_len, &llbaa);
- index = (cdb10) ? (MODE_SELECT_10_MPH_SIZE) : (MODE_SELECT_6_MPH_SIZE);
-
- if (bd_len != 0) {
- /* Block Descriptors present, parse */
- nvme_trans_modesel_save_bd(ns, parm_list, index, bd_len, llbaa);
- index += bd_len;
- }
- saved_index = index;
-
- /* Multiple mode pages may be present; iterate through all */
- /* In 1st Iteration, don't do NVME Command, only check for CDB errors */
- do {
- page_code = parm_list[index] & MODE_SELECT_PAGE_CODE_MASK;
- mp_size = parm_list[index + 1] + 2;
- if ((page_code != MODE_PAGE_CACHING) &&
- (page_code != MODE_PAGE_CONTROL) &&
- (page_code != MODE_PAGE_POWER_CONDITION)) {
- res = nvme_trans_completion(hdr,
- SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST,
- SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- goto out_mem;
- }
- index += mp_size;
- } while (index < parm_list_len);
-
- /* In 2nd Iteration, do the NVME Commands */
- index = saved_index;
- do {
- page_code = parm_list[index] & MODE_SELECT_PAGE_CODE_MASK;
- mp_size = parm_list[index + 1] + 2;
- res = nvme_trans_modesel_get_mp(ns, hdr, &parm_list[index],
- page_code);
- if (res)
- break;
- index += mp_size;
- } while (index < parm_list_len);
-
- out_mem:
- kfree(parm_list);
- out:
- return res;
-}
-
-/* Format Unit Helper Functions */
-
-static int nvme_trans_fmt_set_blk_size_count(struct nvme_ns *ns,
- struct sg_io_hdr *hdr)
-{
- int res = 0;
- int nvme_sc;
- u8 flbas;
-
- /*
- * SCSI Expects a MODE SELECT would have been issued prior to
- * a FORMAT UNIT, and the block size and number would be used
- * from the block descriptor in it. If a MODE SELECT had not
- * been issued, FORMAT shall use the current values for both.
- */
-
- if (ns->mode_select_num_blocks == 0 || ns->mode_select_block_len == 0) {
- struct nvme_id_ns *id_ns;
-
- nvme_sc = nvme_identify_ns(ns->ctrl, ns->ns_id, &id_ns);
- res = nvme_trans_status_code(hdr, nvme_sc);
- if (res)
- return res;
-
- if (ns->mode_select_num_blocks == 0)
- ns->mode_select_num_blocks = le64_to_cpu(id_ns->ncap);
- if (ns->mode_select_block_len == 0) {
- flbas = (id_ns->flbas) & 0x0F;
- ns->mode_select_block_len =
- (1 << (id_ns->lbaf[flbas].ds));
- }
-
- kfree(id_ns);
- }
-
- return 0;
-}
-
-static int nvme_trans_fmt_get_parm_header(struct sg_io_hdr *hdr, u8 len,
- u8 format_prot_info, u8 *nvme_pf_code)
-{
- int res;
- u8 *parm_list;
- u8 pf_usage, pf_code;
-
- parm_list = kmalloc(len, GFP_KERNEL);
- if (parm_list == NULL) {
- res = -ENOMEM;
- goto out;
- }
- res = nvme_trans_copy_from_user(hdr, parm_list, len);
- if (res)
- goto out_mem;
-
- if ((parm_list[FORMAT_UNIT_IMMED_OFFSET] &
- FORMAT_UNIT_IMMED_MASK) != 0) {
- res = nvme_trans_completion(hdr, SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- goto out_mem;
- }
-
- if (len == FORMAT_UNIT_LONG_PARM_LIST_LEN &&
- (parm_list[FORMAT_UNIT_PROT_INT_OFFSET] & 0x0F) != 0) {
- res = nvme_trans_completion(hdr, SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- goto out_mem;
- }
- pf_usage = parm_list[FORMAT_UNIT_PROT_FIELD_USAGE_OFFSET] &
- FORMAT_UNIT_PROT_FIELD_USAGE_MASK;
- pf_code = (pf_usage << 2) | format_prot_info;
- switch (pf_code) {
- case 0:
- *nvme_pf_code = 0;
- break;
- case 2:
- *nvme_pf_code = 1;
- break;
- case 3:
- *nvme_pf_code = 2;
- break;
- case 7:
- *nvme_pf_code = 3;
- break;
- default:
- res = nvme_trans_completion(hdr, SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- break;
- }
-
- out_mem:
- kfree(parm_list);
- out:
- return res;
-}
-
-static int nvme_trans_fmt_send_cmd(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 prot_info)
-{
- int res;
- int nvme_sc;
- struct nvme_id_ns *id_ns;
- u8 i;
- u8 nlbaf;
- u8 selected_lbaf = 0xFF;
- u32 cdw10 = 0;
- struct nvme_command c;
-
- /* Loop thru LBAF's in id_ns to match reqd lbaf, put in cdw10 */
- nvme_sc = nvme_identify_ns(ns->ctrl, ns->ns_id, &id_ns);
- res = nvme_trans_status_code(hdr, nvme_sc);
- if (res)
- return res;
-
- nlbaf = id_ns->nlbaf;
-
- for (i = 0; i < nlbaf; i++) {
- if (ns->mode_select_block_len == (1 << (id_ns->lbaf[i].ds))) {
- selected_lbaf = i;
- break;
- }
- }
- if (selected_lbaf > 0x0F) {
- res = nvme_trans_completion(hdr, SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_INVALID_PARAMETER,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- }
- if (ns->mode_select_num_blocks != le64_to_cpu(id_ns->ncap)) {
- res = nvme_trans_completion(hdr, SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_INVALID_PARAMETER,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- }
-
- cdw10 |= prot_info << 5;
- cdw10 |= selected_lbaf & 0x0F;
- memset(&c, 0, sizeof(c));
- c.format.opcode = nvme_admin_format_nvm;
- c.format.nsid = cpu_to_le32(ns->ns_id);
- c.format.cdw10 = cpu_to_le32(cdw10);
-
- nvme_sc = nvme_submit_sync_cmd(ns->ctrl->admin_q, &c, NULL, 0);
- res = nvme_trans_status_code(hdr, nvme_sc);
-
- kfree(id_ns);
- return res;
-}
-
-static inline u32 nvme_trans_io_get_num_cmds(struct sg_io_hdr *hdr,
- struct nvme_trans_io_cdb *cdb_info,
- u32 max_blocks)
-{
- /* If using iovecs, send one nvme command per vector */
- if (hdr->iovec_count > 0)
- return hdr->iovec_count;
- else if (cdb_info->xfer_len > max_blocks)
- return ((cdb_info->xfer_len - 1) / max_blocks) + 1;
- else
- return 1;
-}
-
-static u16 nvme_trans_io_get_control(struct nvme_ns *ns,
- struct nvme_trans_io_cdb *cdb_info)
-{
- u16 control = 0;
-
- /* When Protection information support is added, implement here */
-
- if (cdb_info->fua > 0)
- control |= NVME_RW_FUA;
-
- return control;
-}
-
-static int nvme_trans_do_nvme_io(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- struct nvme_trans_io_cdb *cdb_info, u8 is_write)
-{
- int nvme_sc = NVME_SC_SUCCESS;
- u32 num_cmds;
- u64 unit_len;
- u64 unit_num_blocks; /* Number of blocks to xfer in each nvme cmd */
- u32 retcode;
- u32 i = 0;
- u64 nvme_offset = 0;
- void __user *next_mapping_addr;
- struct nvme_command c;
- u8 opcode = (is_write ? nvme_cmd_write : nvme_cmd_read);
- u16 control;
- u32 max_blocks = queue_max_hw_sectors(ns->queue) >> (ns->lba_shift - 9);
-
- num_cmds = nvme_trans_io_get_num_cmds(hdr, cdb_info, max_blocks);
-
- /*
- * This loop handles two cases.
- * First, when an SGL is used in the form of an iovec list:
- * - Use iov_base as the next mapping address for the nvme command_id
- * - Use iov_len as the data transfer length for the command.
- * Second, when we have a single buffer
- * - If larger than max_blocks, split into chunks, offset
- * each nvme command accordingly.
- */
- for (i = 0; i < num_cmds; i++) {
- memset(&c, 0, sizeof(c));
- if (hdr->iovec_count > 0) {
- struct sg_iovec sgl;
-
- retcode = copy_from_user(&sgl, hdr->dxferp +
- i * sizeof(struct sg_iovec),
- sizeof(struct sg_iovec));
- if (retcode)
- return -EFAULT;
- unit_len = sgl.iov_len;
- unit_num_blocks = unit_len >> ns->lba_shift;
- next_mapping_addr = sgl.iov_base;
- } else {
- unit_num_blocks = min((u64)max_blocks,
- (cdb_info->xfer_len - nvme_offset));
- unit_len = unit_num_blocks << ns->lba_shift;
- next_mapping_addr = hdr->dxferp +
- ((1 << ns->lba_shift) * nvme_offset);
- }
-
- c.rw.opcode = opcode;
- c.rw.nsid = cpu_to_le32(ns->ns_id);
- c.rw.slba = cpu_to_le64(cdb_info->lba + nvme_offset);
- c.rw.length = cpu_to_le16(unit_num_blocks - 1);
- control = nvme_trans_io_get_control(ns, cdb_info);
- c.rw.control = cpu_to_le16(control);
-
- if (get_capacity(ns->disk) - unit_num_blocks <
- cdb_info->lba + nvme_offset) {
- nvme_sc = NVME_SC_LBA_RANGE;
- break;
- }
- nvme_sc = nvme_submit_user_cmd(ns->queue, &c,
- next_mapping_addr, unit_len, NULL, 0);
- if (nvme_sc)
- break;
-
- nvme_offset += unit_num_blocks;
- }
-
- return nvme_trans_status_code(hdr, nvme_sc);
-}
-
-
-/* SCSI Command Translation Functions */
-
-static int nvme_trans_io(struct nvme_ns *ns, struct sg_io_hdr *hdr, u8 is_write,
- u8 *cmd)
-{
- int res = 0;
- struct nvme_trans_io_cdb cdb_info = { 0, };
- u8 opcode = cmd[0];
- u64 xfer_bytes;
- u64 sum_iov_len = 0;
- struct sg_iovec sgl;
- int i;
- size_t not_copied;
-
- /*
- * The FUA and WPROTECT fields are not supported in 6-byte CDBs,
- * but always in the same place for all others.
- */
- switch (opcode) {
- case WRITE_6:
- case READ_6:
- break;
- default:
- cdb_info.fua = cmd[1] & 0x8;
- cdb_info.prot_info = (cmd[1] & 0xe0) >> 5;
- if (cdb_info.prot_info && !ns->pi_type) {
- return nvme_trans_completion(hdr,
- SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST,
- SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- }
- }
-
- switch (opcode) {
- case WRITE_6:
- case READ_6:
- cdb_info.lba = get_unaligned_be24(&cmd[1]);
- cdb_info.xfer_len = cmd[4];
- if (cdb_info.xfer_len == 0)
- cdb_info.xfer_len = 256;
- break;
- case WRITE_10:
- case READ_10:
- cdb_info.lba = get_unaligned_be32(&cmd[2]);
- cdb_info.xfer_len = get_unaligned_be16(&cmd[7]);
- break;
- case WRITE_12:
- case READ_12:
- cdb_info.lba = get_unaligned_be32(&cmd[2]);
- cdb_info.xfer_len = get_unaligned_be32(&cmd[6]);
- break;
- case WRITE_16:
- case READ_16:
- cdb_info.lba = get_unaligned_be64(&cmd[2]);
- cdb_info.xfer_len = get_unaligned_be32(&cmd[10]);
- break;
- default:
- /* Will never really reach here */
- res = -EIO;
- goto out;
- }
-
- /* Calculate total length of transfer (in bytes) */
- if (hdr->iovec_count > 0) {
- for (i = 0; i < hdr->iovec_count; i++) {
- not_copied = copy_from_user(&sgl, hdr->dxferp +
- i * sizeof(struct sg_iovec),
- sizeof(struct sg_iovec));
- if (not_copied)
- return -EFAULT;
- sum_iov_len += sgl.iov_len;
- /* IO vector sizes should be multiples of block size */
- if (sgl.iov_len % (1 << ns->lba_shift) != 0) {
- res = nvme_trans_completion(hdr,
- SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST,
- SCSI_ASC_INVALID_PARAMETER,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- goto out;
- }
- }
- } else {
- sum_iov_len = hdr->dxfer_len;
- }
-
- /* As Per sg ioctl howto, if the lengths differ, use the lower one */
- xfer_bytes = min(((u64)hdr->dxfer_len), sum_iov_len);
-
- /* If block count and actual data buffer size dont match, error out */
- if (xfer_bytes != (cdb_info.xfer_len << ns->lba_shift)) {
- res = -EINVAL;
- goto out;
- }
-
- /* Check for 0 length transfer - it is not illegal */
- if (cdb_info.xfer_len == 0)
- goto out;
-
- /* Send NVMe IO Command(s) */
- res = nvme_trans_do_nvme_io(ns, hdr, &cdb_info, is_write);
- if (res)
- goto out;
-
- out:
- return res;
-}
-
-static int nvme_trans_inquiry(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 *cmd)
-{
- int res = 0;
- u8 evpd;
- u8 page_code;
- int alloc_len;
- u8 *inq_response;
-
- evpd = cmd[1] & 0x01;
- page_code = cmd[2];
- alloc_len = get_unaligned_be16(&cmd[3]);
-
- inq_response = kmalloc(max(alloc_len, STANDARD_INQUIRY_LENGTH),
- GFP_KERNEL);
- if (inq_response == NULL) {
- res = -ENOMEM;
- goto out_mem;
- }
-
- if (evpd == 0) {
- if (page_code == INQ_STANDARD_INQUIRY_PAGE) {
- res = nvme_trans_standard_inquiry_page(ns, hdr,
- inq_response, alloc_len);
- } else {
- res = nvme_trans_completion(hdr,
- SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST,
- SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- }
- } else {
- switch (page_code) {
- case VPD_SUPPORTED_PAGES:
- res = nvme_trans_supported_vpd_pages(ns, hdr,
- inq_response, alloc_len);
- break;
- case VPD_SERIAL_NUMBER:
- res = nvme_trans_unit_serial_page(ns, hdr, inq_response,
- alloc_len);
- break;
- case VPD_DEVICE_IDENTIFIERS:
- res = nvme_trans_device_id_page(ns, hdr, inq_response,
- alloc_len);
- break;
- case VPD_EXTENDED_INQUIRY:
- res = nvme_trans_ext_inq_page(ns, hdr, alloc_len);
- break;
- case VPD_BLOCK_LIMITS:
- res = nvme_trans_bdev_limits_page(ns, hdr, inq_response,
- alloc_len);
- break;
- case VPD_BLOCK_DEV_CHARACTERISTICS:
- res = nvme_trans_bdev_char_page(ns, hdr, alloc_len);
- break;
- default:
- res = nvme_trans_completion(hdr,
- SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST,
- SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- break;
- }
- }
- kfree(inq_response);
- out_mem:
- return res;
-}
-
-static int nvme_trans_log_sense(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 *cmd)
-{
- int res;
- u16 alloc_len;
- u8 pc;
- u8 page_code;
-
- if (cmd[1] != LOG_SENSE_CDB_SP_NOT_ENABLED) {
- res = nvme_trans_completion(hdr, SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- goto out;
- }
-
- page_code = cmd[2] & LOG_SENSE_CDB_PAGE_CODE_MASK;
- pc = (cmd[2] & LOG_SENSE_CDB_PC_MASK) >> LOG_SENSE_CDB_PC_SHIFT;
- if (pc != LOG_SENSE_CDB_PC_CUMULATIVE_VALUES) {
- res = nvme_trans_completion(hdr, SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- goto out;
- }
- alloc_len = get_unaligned_be16(&cmd[7]);
- switch (page_code) {
- case LOG_PAGE_SUPPORTED_LOG_PAGES_PAGE:
- res = nvme_trans_log_supp_pages(ns, hdr, alloc_len);
- break;
- case LOG_PAGE_INFORMATIONAL_EXCEPTIONS_PAGE:
- res = nvme_trans_log_info_exceptions(ns, hdr, alloc_len);
- break;
- case LOG_PAGE_TEMPERATURE_PAGE:
- res = nvme_trans_log_temperature(ns, hdr, alloc_len);
- break;
- default:
- res = nvme_trans_completion(hdr, SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- break;
- }
-
- out:
- return res;
-}
-
-static int nvme_trans_mode_select(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 *cmd)
-{
- u8 cdb10 = 0;
- u16 parm_list_len;
- u8 page_format;
- u8 save_pages;
-
- page_format = cmd[1] & MODE_SELECT_CDB_PAGE_FORMAT_MASK;
- save_pages = cmd[1] & MODE_SELECT_CDB_SAVE_PAGES_MASK;
-
- if (cmd[0] == MODE_SELECT) {
- parm_list_len = cmd[4];
- } else {
- parm_list_len = cmd[7];
- cdb10 = 1;
- }
-
- if (parm_list_len != 0) {
- /*
- * According to SPC-4 r24, a paramter list length field of 0
- * shall not be considered an error
- */
- return nvme_trans_modesel_data(ns, hdr, cmd, parm_list_len,
- page_format, save_pages, cdb10);
- }
-
- return 0;
-}
-
-static int nvme_trans_mode_sense(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 *cmd)
-{
- int res = 0;
- u16 alloc_len;
- u8 cdb10 = 0;
-
- if (cmd[0] == MODE_SENSE) {
- alloc_len = cmd[4];
- } else {
- alloc_len = get_unaligned_be16(&cmd[7]);
- cdb10 = 1;
- }
-
- if ((cmd[2] & MODE_SENSE_PAGE_CONTROL_MASK) !=
- MODE_SENSE_PC_CURRENT_VALUES) {
- res = nvme_trans_completion(hdr, SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- goto out;
- }
-
- switch (cmd[2] & MODE_SENSE_PAGE_CODE_MASK) {
- case MODE_PAGE_CACHING:
- res = nvme_trans_mode_page_create(ns, hdr, cmd, alloc_len,
- cdb10,
- &nvme_trans_fill_caching_page,
- MODE_PAGE_CACHING_LEN);
- break;
- case MODE_PAGE_CONTROL:
- res = nvme_trans_mode_page_create(ns, hdr, cmd, alloc_len,
- cdb10,
- &nvme_trans_fill_control_page,
- MODE_PAGE_CONTROL_LEN);
- break;
- case MODE_PAGE_POWER_CONDITION:
- res = nvme_trans_mode_page_create(ns, hdr, cmd, alloc_len,
- cdb10,
- &nvme_trans_fill_pow_cnd_page,
- MODE_PAGE_POW_CND_LEN);
- break;
- case MODE_PAGE_INFO_EXCEP:
- res = nvme_trans_mode_page_create(ns, hdr, cmd, alloc_len,
- cdb10,
- &nvme_trans_fill_inf_exc_page,
- MODE_PAGE_INF_EXC_LEN);
- break;
- case MODE_PAGE_RETURN_ALL:
- res = nvme_trans_mode_page_create(ns, hdr, cmd, alloc_len,
- cdb10,
- &nvme_trans_fill_all_pages,
- MODE_PAGE_ALL_LEN);
- break;
- default:
- res = nvme_trans_completion(hdr, SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- break;
- }
-
- out:
- return res;
-}
-
-static int nvme_trans_read_capacity(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 *cmd, u8 cdb16)
-{
- int res;
- int nvme_sc;
- u32 alloc_len;
- u32 resp_size;
- u32 xfer_len;
- struct nvme_id_ns *id_ns;
- u8 *response;
-
- if (cdb16) {
- alloc_len = get_unaligned_be32(&cmd[10]);
- resp_size = READ_CAP_16_RESP_SIZE;
- } else {
- alloc_len = READ_CAP_10_RESP_SIZE;
- resp_size = READ_CAP_10_RESP_SIZE;
- }
-
- nvme_sc = nvme_identify_ns(ns->ctrl, ns->ns_id, &id_ns);
- res = nvme_trans_status_code(hdr, nvme_sc);
- if (res)
- return res;
-
- response = kzalloc(resp_size, GFP_KERNEL);
- if (response == NULL) {
- res = -ENOMEM;
- goto out_free_id;
- }
- nvme_trans_fill_read_cap(response, id_ns, cdb16);
-
- xfer_len = min(alloc_len, resp_size);
- res = nvme_trans_copy_to_user(hdr, response, xfer_len);
-
- kfree(response);
- out_free_id:
- kfree(id_ns);
- return res;
-}
-
-static int nvme_trans_report_luns(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 *cmd)
-{
- int res;
- int nvme_sc;
- u32 alloc_len, xfer_len, resp_size;
- u8 *response;
- struct nvme_id_ctrl *id_ctrl;
- u32 ll_length, lun_id;
- u8 lun_id_offset = REPORT_LUNS_FIRST_LUN_OFFSET;
- __be32 tmp_len;
-
- switch (cmd[2]) {
- default:
- return nvme_trans_completion(hdr, SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- case ALL_LUNS_RETURNED:
- case ALL_WELL_KNOWN_LUNS_RETURNED:
- case RESTRICTED_LUNS_RETURNED:
- nvme_sc = nvme_identify_ctrl(ns->ctrl, &id_ctrl);
- res = nvme_trans_status_code(hdr, nvme_sc);
- if (res)
- return res;
-
- ll_length = le32_to_cpu(id_ctrl->nn) * LUN_ENTRY_SIZE;
- resp_size = ll_length + LUN_DATA_HEADER_SIZE;
-
- alloc_len = get_unaligned_be32(&cmd[6]);
- if (alloc_len < resp_size) {
- res = nvme_trans_completion(hdr,
- SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- goto out_free_id;
- }
-
- response = kzalloc(resp_size, GFP_KERNEL);
- if (response == NULL) {
- res = -ENOMEM;
- goto out_free_id;
- }
-
- /* The first LUN ID will always be 0 per the SAM spec */
- for (lun_id = 0; lun_id < le32_to_cpu(id_ctrl->nn); lun_id++) {
- /*
- * Set the LUN Id and then increment to the next LUN
- * location in the parameter data.
- */
- __be64 tmp_id = cpu_to_be64(lun_id);
- memcpy(&response[lun_id_offset], &tmp_id, sizeof(u64));
- lun_id_offset += LUN_ENTRY_SIZE;
- }
- tmp_len = cpu_to_be32(ll_length);
- memcpy(response, &tmp_len, sizeof(u32));
- }
-
- xfer_len = min(alloc_len, resp_size);
- res = nvme_trans_copy_to_user(hdr, response, xfer_len);
-
- kfree(response);
- out_free_id:
- kfree(id_ctrl);
- return res;
-}
-
-static int nvme_trans_request_sense(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 *cmd)
-{
- int res;
- u8 alloc_len, xfer_len, resp_size;
- u8 desc_format;
- u8 *response;
-
- desc_format = cmd[1] & 0x01;
- alloc_len = cmd[4];
-
- resp_size = ((desc_format) ? (DESC_FMT_SENSE_DATA_SIZE) :
- (FIXED_FMT_SENSE_DATA_SIZE));
- response = kzalloc(resp_size, GFP_KERNEL);
- if (response == NULL) {
- res = -ENOMEM;
- goto out;
- }
-
- if (desc_format) {
- /* Descriptor Format Sense Data */
- response[0] = DESC_FORMAT_SENSE_DATA;
- response[1] = NO_SENSE;
- /* TODO How is LOW POWER CONDITION ON handled? (byte 2) */
- response[2] = SCSI_ASC_NO_SENSE;
- response[3] = SCSI_ASCQ_CAUSE_NOT_REPORTABLE;
- /* SDAT_OVFL = 0 | Additional Sense Length = 0 */
- } else {
- /* Fixed Format Sense Data */
- response[0] = FIXED_SENSE_DATA;
- /* Byte 1 = Obsolete */
- response[2] = NO_SENSE; /* FM, EOM, ILI, SDAT_OVFL = 0 */
- /* Bytes 3-6 - Information - set to zero */
- response[7] = FIXED_SENSE_DATA_ADD_LENGTH;
- /* Bytes 8-11 - Cmd Specific Information - set to zero */
- response[12] = SCSI_ASC_NO_SENSE;
- response[13] = SCSI_ASCQ_CAUSE_NOT_REPORTABLE;
- /* Byte 14 = Field Replaceable Unit Code = 0 */
- /* Bytes 15-17 - SKSV=0; Sense Key Specific = 0 */
- }
-
- xfer_len = min(alloc_len, resp_size);
- res = nvme_trans_copy_to_user(hdr, response, xfer_len);
-
- kfree(response);
- out:
- return res;
-}
-
-static int nvme_trans_synchronize_cache(struct nvme_ns *ns,
- struct sg_io_hdr *hdr)
-{
- int nvme_sc;
- struct nvme_command c;
-
- memset(&c, 0, sizeof(c));
- c.common.opcode = nvme_cmd_flush;
- c.common.nsid = cpu_to_le32(ns->ns_id);
-
- nvme_sc = nvme_submit_sync_cmd(ns->queue, &c, NULL, 0);
- return nvme_trans_status_code(hdr, nvme_sc);
-}
-
-static int nvme_trans_format_unit(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 *cmd)
-{
- int res;
- u8 parm_hdr_len = 0;
- u8 nvme_pf_code = 0;
- u8 format_prot_info, long_list, format_data;
-
- format_prot_info = (cmd[1] & 0xc0) >> 6;
- long_list = cmd[1] & 0x20;
- format_data = cmd[1] & 0x10;
-
- if (format_data != 0) {
- if (format_prot_info != 0) {
- if (long_list == 0)
- parm_hdr_len = FORMAT_UNIT_SHORT_PARM_LIST_LEN;
- else
- parm_hdr_len = FORMAT_UNIT_LONG_PARM_LIST_LEN;
- }
- } else if (format_data == 0 && format_prot_info != 0) {
- res = nvme_trans_completion(hdr, SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- goto out;
- }
-
- /* Get parm header from data-in/out buffer */
- /*
- * According to the translation spec, the only fields in the parameter
- * list we are concerned with are in the header. So allocate only that.
- */
- if (parm_hdr_len > 0) {
- res = nvme_trans_fmt_get_parm_header(hdr, parm_hdr_len,
- format_prot_info, &nvme_pf_code);
- if (res)
- goto out;
- }
-
- /* Attempt to activate any previously downloaded firmware image */
- res = nvme_trans_send_activate_fw_cmd(ns, hdr, 0);
-
- /* Determine Block size and count and send format command */
- res = nvme_trans_fmt_set_blk_size_count(ns, hdr);
- if (res)
- goto out;
-
- res = nvme_trans_fmt_send_cmd(ns, hdr, nvme_pf_code);
-
- out:
- return res;
-}
-
-static int nvme_trans_test_unit_ready(struct nvme_ns *ns,
- struct sg_io_hdr *hdr,
- u8 *cmd)
-{
- if (nvme_ctrl_ready(ns->ctrl))
- return nvme_trans_completion(hdr, SAM_STAT_CHECK_CONDITION,
- NOT_READY, SCSI_ASC_LUN_NOT_READY,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- else
- return nvme_trans_completion(hdr, SAM_STAT_GOOD, NO_SENSE, 0, 0);
-}
-
-static int nvme_trans_write_buffer(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 *cmd)
-{
- int res = 0;
- u32 buffer_offset, parm_list_length;
- u8 buffer_id, mode;
-
- parm_list_length = get_unaligned_be24(&cmd[6]);
- if (parm_list_length % BYTES_TO_DWORDS != 0) {
- /* NVMe expects Firmware file to be a whole number of DWORDS */
- res = nvme_trans_completion(hdr, SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- goto out;
- }
- buffer_id = cmd[2];
- if (buffer_id > NVME_MAX_FIRMWARE_SLOT) {
- res = nvme_trans_completion(hdr, SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- goto out;
- }
- mode = cmd[1] & 0x1f;
- buffer_offset = get_unaligned_be24(&cmd[3]);
-
- switch (mode) {
- case DOWNLOAD_SAVE_ACTIVATE:
- res = nvme_trans_send_download_fw_cmd(ns, hdr, nvme_admin_download_fw,
- parm_list_length, buffer_offset,
- buffer_id);
- if (res)
- goto out;
- res = nvme_trans_send_activate_fw_cmd(ns, hdr, buffer_id);
- break;
- case DOWNLOAD_SAVE_DEFER_ACTIVATE:
- res = nvme_trans_send_download_fw_cmd(ns, hdr, nvme_admin_download_fw,
- parm_list_length, buffer_offset,
- buffer_id);
- break;
- case ACTIVATE_DEFERRED_MICROCODE:
- res = nvme_trans_send_activate_fw_cmd(ns, hdr, buffer_id);
- break;
- default:
- res = nvme_trans_completion(hdr, SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_INVALID_CDB,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- break;
- }
-
- out:
- return res;
-}
-
-struct scsi_unmap_blk_desc {
- __be64 slba;
- __be32 nlb;
- u32 resv;
-};
-
-struct scsi_unmap_parm_list {
- __be16 unmap_data_len;
- __be16 unmap_blk_desc_data_len;
- u32 resv;
- struct scsi_unmap_blk_desc desc[0];
-};
-
-static int nvme_trans_unmap(struct nvme_ns *ns, struct sg_io_hdr *hdr,
- u8 *cmd)
-{
- struct scsi_unmap_parm_list *plist;
- struct nvme_dsm_range *range;
- struct nvme_command c;
- int i, nvme_sc, res;
- u16 ndesc, list_len;
-
- list_len = get_unaligned_be16(&cmd[7]);
- if (!list_len)
- return -EINVAL;
-
- plist = kmalloc(list_len, GFP_KERNEL);
- if (!plist)
- return -ENOMEM;
-
- res = nvme_trans_copy_from_user(hdr, plist, list_len);
- if (res)
- goto out;
-
- ndesc = be16_to_cpu(plist->unmap_blk_desc_data_len) >> 4;
- if (!ndesc || ndesc > 256) {
- res = -EINVAL;
- goto out;
- }
-
- range = kcalloc(ndesc, sizeof(*range), GFP_KERNEL);
- if (!range) {
- res = -ENOMEM;
- goto out;
- }
-
- for (i = 0; i < ndesc; i++) {
- range[i].nlb = cpu_to_le32(be32_to_cpu(plist->desc[i].nlb));
- range[i].slba = cpu_to_le64(be64_to_cpu(plist->desc[i].slba));
- range[i].cattr = 0;
- }
-
- memset(&c, 0, sizeof(c));
- c.dsm.opcode = nvme_cmd_dsm;
- c.dsm.nsid = cpu_to_le32(ns->ns_id);
- c.dsm.nr = cpu_to_le32(ndesc - 1);
- c.dsm.attributes = cpu_to_le32(NVME_DSMGMT_AD);
-
- nvme_sc = nvme_submit_sync_cmd(ns->queue, &c, range,
- ndesc * sizeof(*range));
- res = nvme_trans_status_code(hdr, nvme_sc);
-
- kfree(range);
- out:
- kfree(plist);
- return res;
-}
-
-static int nvme_scsi_translate(struct nvme_ns *ns, struct sg_io_hdr *hdr)
-{
- u8 cmd[16];
- int retcode;
- unsigned int opcode;
-
- if (hdr->cmdp == NULL)
- return -EMSGSIZE;
- if (hdr->cmd_len > sizeof(cmd))
- return -EINVAL;
- if (copy_from_user(cmd, hdr->cmdp, hdr->cmd_len))
- return -EFAULT;
-
- /*
- * Prime the hdr with good status for scsi commands that don't require
- * an nvme command for translation.
- */
- retcode = nvme_trans_status_code(hdr, NVME_SC_SUCCESS);
- if (retcode)
- return retcode;
-
- opcode = cmd[0];
-
- switch (opcode) {
- case READ_6:
- case READ_10:
- case READ_12:
- case READ_16:
- retcode = nvme_trans_io(ns, hdr, 0, cmd);
- break;
- case WRITE_6:
- case WRITE_10:
- case WRITE_12:
- case WRITE_16:
- retcode = nvme_trans_io(ns, hdr, 1, cmd);
- break;
- case INQUIRY:
- retcode = nvme_trans_inquiry(ns, hdr, cmd);
- break;
- case LOG_SENSE:
- retcode = nvme_trans_log_sense(ns, hdr, cmd);
- break;
- case MODE_SELECT:
- case MODE_SELECT_10:
- retcode = nvme_trans_mode_select(ns, hdr, cmd);
- break;
- case MODE_SENSE:
- case MODE_SENSE_10:
- retcode = nvme_trans_mode_sense(ns, hdr, cmd);
- break;
- case READ_CAPACITY:
- retcode = nvme_trans_read_capacity(ns, hdr, cmd, 0);
- break;
- case SERVICE_ACTION_IN_16:
- switch (cmd[1]) {
- case SAI_READ_CAPACITY_16:
- retcode = nvme_trans_read_capacity(ns, hdr, cmd, 1);
- break;
- default:
- goto out;
- }
- break;
- case REPORT_LUNS:
- retcode = nvme_trans_report_luns(ns, hdr, cmd);
- break;
- case REQUEST_SENSE:
- retcode = nvme_trans_request_sense(ns, hdr, cmd);
- break;
- case SYNCHRONIZE_CACHE:
- retcode = nvme_trans_synchronize_cache(ns, hdr);
- break;
- case FORMAT_UNIT:
- retcode = nvme_trans_format_unit(ns, hdr, cmd);
- break;
- case TEST_UNIT_READY:
- retcode = nvme_trans_test_unit_ready(ns, hdr, cmd);
- break;
- case WRITE_BUFFER:
- retcode = nvme_trans_write_buffer(ns, hdr, cmd);
- break;
- case UNMAP:
- retcode = nvme_trans_unmap(ns, hdr, cmd);
- break;
- default:
- out:
- retcode = nvme_trans_completion(hdr, SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_ILLEGAL_COMMAND,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
- break;
- }
- return retcode;
-}
-
-int nvme_sg_io(struct nvme_ns *ns, struct sg_io_hdr __user *u_hdr)
-{
- struct sg_io_hdr hdr;
- int retcode;
-
- if (!capable(CAP_SYS_ADMIN))
- return -EACCES;
- if (copy_from_user(&hdr, u_hdr, sizeof(hdr)))
- return -EFAULT;
- if (hdr.interface_id != 'S')
- return -EINVAL;
-
- /*
- * A positive return code means a NVMe status, which has been
- * translated to sense data.
- */
- retcode = nvme_scsi_translate(ns, &hdr);
- if (retcode < 0)
- return retcode;
- if (copy_to_user(u_hdr, &hdr, sizeof(sg_io_hdr_t)) > 0)
- return -EFAULT;
- return 0;
-}
-
-int nvme_sg_get_version_num(int __user *ip)
-{
- return put_user(sg_version_num, ip);
-}
static void nvmet_execute_identify_nslist(struct nvmet_req *req)
{
- static const int buf_size = 4096;
+ static const int buf_size = NVME_IDENTIFY_DATA_SIZE;
struct nvmet_ctrl *ctrl = req->sq->ctrl;
struct nvmet_ns *ns;
u32 min_nsid = le32_to_cpu(req->cmd->identify.nsid);
nvmet_req_complete(req, status);
}
+static u16 nvmet_copy_ns_identifier(struct nvmet_req *req, u8 type, u8 len,
+ void *id, off_t *off)
+{
+ struct nvme_ns_id_desc desc = {
+ .nidt = type,
+ .nidl = len,
+ };
+ u16 status;
+
+ status = nvmet_copy_to_sgl(req, *off, &desc, sizeof(desc));
+ if (status)
+ return status;
+ *off += sizeof(desc);
+
+ status = nvmet_copy_to_sgl(req, *off, id, len);
+ if (status)
+ return status;
+ *off += len;
+
+ return 0;
+}
+
+static void nvmet_execute_identify_desclist(struct nvmet_req *req)
+{
+ struct nvmet_ns *ns;
+ u16 status = 0;
+ off_t off = 0;
+
+ ns = nvmet_find_namespace(req->sq->ctrl, req->cmd->identify.nsid);
+ if (!ns) {
+ status = NVME_SC_INVALID_NS | NVME_SC_DNR;
+ goto out;
+ }
+
+ if (memchr_inv(&ns->uuid, 0, sizeof(ns->uuid))) {
+ status = nvmet_copy_ns_identifier(req, NVME_NIDT_UUID,
+ NVME_NIDT_UUID_LEN,
+ &ns->uuid, &off);
+ if (status)
+ goto out_put_ns;
+ }
+ if (memchr_inv(ns->nguid, 0, sizeof(ns->nguid))) {
+ status = nvmet_copy_ns_identifier(req, NVME_NIDT_NGUID,
+ NVME_NIDT_NGUID_LEN,
+ &ns->nguid, &off);
+ if (status)
+ goto out_put_ns;
+ }
+
+ if (sg_zero_buffer(req->sg, req->sg_cnt, NVME_IDENTIFY_DATA_SIZE - off,
+ off) != NVME_IDENTIFY_DATA_SIZE - off)
+ status = NVME_SC_INTERNAL | NVME_SC_DNR;
+out_put_ns:
+ nvmet_put_namespace(ns);
+out:
+ nvmet_req_complete(req, status);
+}
+
/*
* A "mimimum viable" abort implementation: the command is mandatory in the
* spec, but we are not required to do any useful work. We couldn't really
}
break;
case nvme_admin_identify:
- req->data_len = 4096;
+ req->data_len = NVME_IDENTIFY_DATA_SIZE;
switch (cmd->identify.cns) {
case NVME_ID_CNS_NS:
req->execute = nvmet_execute_identify_ns;
case NVME_ID_CNS_NS_ACTIVE_LIST:
req->execute = nvmet_execute_identify_nslist;
return 0;
+ case NVME_ID_CNS_NS_DESC_LIST:
+ req->execute = nvmet_execute_identify_desclist;
+ return 0;
}
break;
case nvme_admin_abort_cmd:
CONFIGFS_ATTR(nvmet_ns_, device_path);
+static ssize_t nvmet_ns_device_uuid_show(struct config_item *item, char *page)
+{
+ return sprintf(page, "%pUb\n", &to_nvmet_ns(item)->uuid);
+}
+
+static ssize_t nvmet_ns_device_uuid_store(struct config_item *item,
+ const char *page, size_t count)
+{
+ struct nvmet_ns *ns = to_nvmet_ns(item);
+ struct nvmet_subsys *subsys = ns->subsys;
+ int ret = 0;
+
+
+ mutex_lock(&subsys->lock);
+ if (ns->enabled) {
+ ret = -EBUSY;
+ goto out_unlock;
+ }
+
+
+ if (uuid_parse(page, &ns->uuid))
+ ret = -EINVAL;
+
+out_unlock:
+ mutex_unlock(&subsys->lock);
+ return ret ? ret : count;
+}
+
static ssize_t nvmet_ns_device_nguid_show(struct config_item *item, char *page)
{
return sprintf(page, "%pUb\n", &to_nvmet_ns(item)->nguid);
}
+CONFIGFS_ATTR(nvmet_ns_, device_uuid);
+
static ssize_t nvmet_ns_device_nguid_store(struct config_item *item,
const char *page, size_t count)
{
static struct configfs_attribute *nvmet_ns_attrs[] = {
&nvmet_ns_attr_device_path,
&nvmet_ns_attr_device_nguid,
+ &nvmet_ns_attr_device_uuid,
&nvmet_ns_attr_enable,
NULL,
};
CONFIGFS_ATTR(nvmet_subsys_, attr_allow_any_host);
+static ssize_t nvmet_subsys_version_show(struct config_item *item,
+ char *page)
+{
+ struct nvmet_subsys *subsys = to_subsys(item);
+
+ if (NVME_TERTIARY(subsys->ver))
+ return snprintf(page, PAGE_SIZE, "%d.%d.%d\n",
+ (int)NVME_MAJOR(subsys->ver),
+ (int)NVME_MINOR(subsys->ver),
+ (int)NVME_TERTIARY(subsys->ver));
+ else
+ return snprintf(page, PAGE_SIZE, "%d.%d\n",
+ (int)NVME_MAJOR(subsys->ver),
+ (int)NVME_MINOR(subsys->ver));
+}
+
+static ssize_t nvmet_subsys_version_store(struct config_item *item,
+ const char *page, size_t count)
+{
+ struct nvmet_subsys *subsys = to_subsys(item);
+ int major, minor, tertiary = 0;
+ int ret;
+
+
+ ret = sscanf(page, "%d.%d.%d\n", &major, &minor, &tertiary);
+ if (ret != 2 && ret != 3)
+ return -EINVAL;
+
+ down_write(&nvmet_config_sem);
+ subsys->ver = NVME_VS(major, minor, tertiary);
+ up_write(&nvmet_config_sem);
+
+ return count;
+}
+CONFIGFS_ATTR(nvmet_subsys_, version);
+
static struct configfs_attribute *nvmet_subsys_attrs[] = {
&nvmet_subsys_attr_attr_allow_any_host,
+ &nvmet_subsys_attr_version,
NULL,
};
ns->nsid = nsid;
ns->subsys = subsys;
+ uuid_gen(&ns->uuid);
return ns;
}
if (!subsys)
return NULL;
- subsys->ver = NVME_VS(1, 2, 1); /* NVMe 1.2.1 */
+ subsys->ver = NVME_VS(1, 3, 0); /* NVMe 1.3.0 */
switch (type) {
case NVME_NQN_NVME:
e->portid = port->disc_addr.portid;
/* we support only dynamic controllers */
e->cntlid = cpu_to_le16(NVME_CNTLID_DYNAMIC);
- e->asqsz = cpu_to_le16(NVMF_AQ_DEPTH);
+ e->asqsz = cpu_to_le16(NVME_AQ_DEPTH);
e->subtype = type;
memcpy(e->trsvcid, port->disc_addr.trsvcid, NVMF_TRSVCID_SIZE);
memcpy(e->traddr, port->disc_addr.traddr, NVMF_TRADDR_SIZE);
return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
}
case nvme_admin_identify:
- req->data_len = 4096;
+ req->data_len = NVME_IDENTIFY_DATA_SIZE;
switch (cmd->identify.cns) {
case NVME_ID_CNS_CTRL:
req->execute =
/* clear any response payload */
memset(&fod->rspiubuf, 0, sizeof(fod->rspiubuf));
+ fod->data_sg = NULL;
+ fod->data_sg_cnt = 0;
+
ret = nvmet_req_init(&fod->req,
&fod->queue->nvme_cq,
&fod->queue->nvme_sq,
&nvmet_fc_tgt_fcp_ops);
- if (!ret) { /* bad SQE content or invalid ctrl state */
- nvmet_fc_abort_op(tgtport, fod);
+ if (!ret) {
+ /* bad SQE content or invalid ctrl state */
+ /* nvmet layer has already called op done to send rsp. */
return;
}
/* keep a running counter of tail position */
atomic_inc(&fod->queue->sqtail);
- fod->data_sg = NULL;
- fod->data_sg_cnt = 0;
if (fod->total_length) {
ret = nvmet_fc_alloc_tgt_pgs(fod);
if (ret) {
struct nvmefc_tgt_fcp_req *tgt_fcpreq)
{
struct fcloop_fcpreq *tfcp_req = tgt_fcp_req_to_fcpreq(tgt_fcpreq);
- int active;
/*
* mark aborted only in case there were 2 threads in transport
* after the abort request
*/
spin_lock(&tfcp_req->reqlock);
- active = tfcp_req->active;
tfcp_req->aborted = true;
spin_unlock(&tfcp_req->reqlock);
struct nvmet_req *req = bio->bi_private;
nvmet_req_complete(req,
- bio->bi_error ? NVME_SC_INTERNAL | NVME_SC_DNR : 0);
+ bio->bi_status ? NVME_SC_INTERNAL | NVME_SC_DNR : 0);
if (bio != &req->inline_bio)
bio_put(bio);
bio->bi_private = req;
bio->bi_end_io = nvmet_bio_done;
if (status) {
- bio->bi_error = -EIO;
+ bio->bi_status = BLK_STS_IOERR;
bio_endio(bio);
} else {
submit_bio(bio);
#include "../host/nvme.h"
#include "../host/fabrics.h"
-#define NVME_LOOP_AQ_DEPTH 256
-
#define NVME_LOOP_MAX_SEGMENTS 256
/*
*/
#define NVME_LOOP_NR_AEN_COMMANDS 1
#define NVME_LOOP_AQ_BLKMQ_DEPTH \
- (NVME_LOOP_AQ_DEPTH - NVME_LOOP_NR_AEN_COMMANDS)
+ (NVME_AQ_DEPTH - NVME_LOOP_NR_AEN_COMMANDS)
struct nvme_loop_iod {
struct nvme_request nvme_req;
};
struct nvme_loop_ctrl {
- spinlock_t lock;
struct nvme_loop_queue *queues;
u32 queue_count;
struct nvmet_ctrl *target_ctrl;
struct work_struct delete_work;
- struct work_struct reset_work;
};
static inline struct nvme_loop_ctrl *to_loop_ctrl(struct nvme_ctrl *ctrl)
struct nvme_loop_iod *iod = blk_mq_rq_to_pdu(rq);
/* queue error recovery */
- schedule_work(&iod->queue->ctrl->reset_work);
+ nvme_reset_ctrl(&iod->queue->ctrl->ctrl);
/* fail with DNR on admin cmd timeout */
nvme_req(rq)->status = NVME_SC_ABORT_REQ | NVME_SC_DNR;
return BLK_EH_HANDLED;
}
-static int nvme_loop_queue_rq(struct blk_mq_hw_ctx *hctx,
+static blk_status_t nvme_loop_queue_rq(struct blk_mq_hw_ctx *hctx,
const struct blk_mq_queue_data *bd)
{
struct nvme_ns *ns = hctx->queue->queuedata;
struct nvme_loop_queue *queue = hctx->driver_data;
struct request *req = bd->rq;
struct nvme_loop_iod *iod = blk_mq_rq_to_pdu(req);
- int ret;
+ blk_status_t ret;
ret = nvme_setup_cmd(ns, req, &iod->cmd);
- if (ret != BLK_MQ_RQ_QUEUE_OK)
+ if (ret)
return ret;
iod->cmd.common.flags |= NVME_CMD_SGL_METABUF;
nvme_cleanup_cmd(req);
blk_mq_start_request(req);
nvme_loop_queue_response(&iod->req);
- return BLK_MQ_RQ_QUEUE_OK;
+ return BLK_STS_OK;
}
if (blk_rq_bytes(req)) {
iod->sg_table.sgl = iod->first_sgl;
- ret = sg_alloc_table_chained(&iod->sg_table,
+ if (sg_alloc_table_chained(&iod->sg_table,
blk_rq_nr_phys_segments(req),
- iod->sg_table.sgl);
- if (ret)
- return BLK_MQ_RQ_QUEUE_BUSY;
+ iod->sg_table.sgl))
+ return BLK_STS_RESOURCE;
iod->req.sg = iod->sg_table.sgl;
iod->req.sg_cnt = blk_rq_map_sg(req->q, req, iod->sg_table.sgl);
blk_mq_start_request(req);
schedule_work(&iod->work);
- return BLK_MQ_RQ_QUEUE_OK;
+ return BLK_STS_OK;
}
static void nvme_loop_submit_async_event(struct nvme_ctrl *arg, int aer_idx)
struct request *req, unsigned int hctx_idx,
unsigned int numa_node)
{
- return nvme_loop_init_iod(set->driver_data, blk_mq_rq_to_pdu(req),
- hctx_idx + 1);
-}
+ struct nvme_loop_ctrl *ctrl = set->driver_data;
-static int nvme_loop_init_admin_request(struct blk_mq_tag_set *set,
- struct request *req, unsigned int hctx_idx,
- unsigned int numa_node)
-{
- return nvme_loop_init_iod(set->driver_data, blk_mq_rq_to_pdu(req), 0);
+ return nvme_loop_init_iod(ctrl, blk_mq_rq_to_pdu(req),
+ (set == &ctrl->tag_set) ? hctx_idx + 1 : 0);
}
static int nvme_loop_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
static const struct blk_mq_ops nvme_loop_admin_mq_ops = {
.queue_rq = nvme_loop_queue_rq,
.complete = nvme_loop_complete_rq,
- .init_request = nvme_loop_init_admin_request,
+ .init_request = nvme_loop_init_request,
.init_hctx = nvme_loop_init_admin_hctx,
.timeout = nvme_loop_timeout,
};
if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_DELETING))
return -EBUSY;
- if (!schedule_work(&ctrl->delete_work))
+ if (!queue_work(nvme_wq, &ctrl->delete_work))
return -EBUSY;
return 0;
static void nvme_loop_reset_ctrl_work(struct work_struct *work)
{
- struct nvme_loop_ctrl *ctrl = container_of(work,
- struct nvme_loop_ctrl, reset_work);
+ struct nvme_loop_ctrl *ctrl =
+ container_of(work, struct nvme_loop_ctrl, ctrl.reset_work);
bool changed;
int ret;
nvme_put_ctrl(&ctrl->ctrl);
}
-static int nvme_loop_reset_ctrl(struct nvme_ctrl *nctrl)
-{
- struct nvme_loop_ctrl *ctrl = to_loop_ctrl(nctrl);
-
- if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RESETTING))
- return -EBUSY;
-
- if (!schedule_work(&ctrl->reset_work))
- return -EBUSY;
-
- flush_work(&ctrl->reset_work);
-
- return 0;
-}
-
static const struct nvme_ctrl_ops nvme_loop_ctrl_ops = {
.name = "loop",
.module = THIS_MODULE,
.reg_read32 = nvmf_reg_read32,
.reg_read64 = nvmf_reg_read64,
.reg_write32 = nvmf_reg_write32,
- .reset_ctrl = nvme_loop_reset_ctrl,
.free_ctrl = nvme_loop_free_ctrl,
.submit_async_event = nvme_loop_submit_async_event,
.delete_ctrl = nvme_loop_del_ctrl,
- .get_subsysnqn = nvmf_get_subsysnqn,
};
static int nvme_loop_create_io_queues(struct nvme_loop_ctrl *ctrl)
INIT_LIST_HEAD(&ctrl->list);
INIT_WORK(&ctrl->delete_work, nvme_loop_del_ctrl_work);
- INIT_WORK(&ctrl->reset_work, nvme_loop_reset_ctrl_work);
+ INIT_WORK(&ctrl->ctrl.reset_work, nvme_loop_reset_ctrl_work);
ret = nvme_init_ctrl(&ctrl->ctrl, dev, &nvme_loop_ctrl_ops,
0 /* no quirks, we're perfect! */);
if (ret)
goto out_put_ctrl;
- spin_lock_init(&ctrl->lock);
-
ret = -ENOMEM;
ctrl->ctrl.sqsize = opts->queue_size - 1;
__nvme_loop_del_ctrl(ctrl);
mutex_unlock(&nvme_loop_ctrl_mutex);
- flush_scheduled_work();
+ flush_workqueue(nvme_wq);
}
module_init(nvme_loop_init_module);
#include <linux/percpu-refcount.h>
#include <linux/list.h>
#include <linux/mutex.h>
+#include <linux/uuid.h>
#include <linux/nvme.h>
#include <linux/configfs.h>
#include <linux/rcupdate.h>
u32 blksize_shift;
loff_t size;
u8 nguid[16];
+ uuid_t uuid;
bool enabled;
struct nvmet_subsys *subsys;
queue->recv_queue_size = le16_to_cpu(req->hsqsize) + 1;
queue->send_queue_size = le16_to_cpu(req->hrqsize);
- if (!queue->host_qid && queue->recv_queue_size > NVMF_AQ_DEPTH)
+ if (!queue->host_qid && queue->recv_queue_size > NVME_AQ_DEPTH)
return NVME_RDMA_CM_INVALID_HSQSIZE;
/* XXX: Should we enforce some kind of max for IO queues? */
/**
* nvme_rdma_device_removal() - Handle RDMA device removal
+ * @cm_id: rdma_cm id, used for nvmet port
* @queue: nvmet rdma queue (cm id qp_context)
- * @addr: nvmet address (cm_id context)
*
* DEVICE_REMOVAL event notifies us that the RDMA device is about
- * to unplug so we should take care of destroying our RDMA resources.
- * This event will be generated for each allocated cm_id.
+ * to unplug. Note that this event can be generated on a normal
+ * queue cm_id and/or a device bound listener cm_id (where in this
+ * case queue will be null).
*
- * Note that this event can be generated on a normal queue cm_id
- * and/or a device bound listener cm_id (where in this case
- * queue will be null).
- *
- * we claim ownership on destroying the cm_id. For queues we move
- * the queue state to NVMET_RDMA_IN_DEVICE_REMOVAL and for port
+ * We registered an ib_client to handle device removal for queues,
+ * so we only need to handle the listening port cm_ids. In this case
* we nullify the priv to prevent double cm_id destruction and destroying
* the cm_id implicitely by returning a non-zero rc to the callout.
*/
static int nvmet_rdma_device_removal(struct rdma_cm_id *cm_id,
struct nvmet_rdma_queue *queue)
{
- unsigned long flags;
-
- if (!queue) {
- struct nvmet_port *port = cm_id->context;
+ struct nvmet_port *port;
+ if (queue) {
/*
- * This is a listener cm_id. Make sure that
- * future remove_port won't invoke a double
- * cm_id destroy. use atomic xchg to make sure
- * we don't compete with remove_port.
- */
- if (xchg(&port->priv, NULL) != cm_id)
- return 0;
- } else {
- /*
- * This is a queue cm_id. Make sure that
- * release queue will not destroy the cm_id
- * and schedule all ctrl queues removal (only
- * if the queue is not disconnecting already).
+ * This is a queue cm_id. we have registered
+ * an ib_client to handle queues removal
+ * so don't interfear and just return.
*/
- spin_lock_irqsave(&queue->state_lock, flags);
- if (queue->state != NVMET_RDMA_Q_DISCONNECTING)
- queue->state = NVMET_RDMA_IN_DEVICE_REMOVAL;
- spin_unlock_irqrestore(&queue->state_lock, flags);
- nvmet_rdma_queue_disconnect(queue);
- flush_scheduled_work();
+ return 0;
}
+ port = cm_id->context;
+
+ /*
+ * This is a listener cm_id. Make sure that
+ * future remove_port won't invoke a double
+ * cm_id destroy. use atomic xchg to make sure
+ * we don't compete with remove_port.
+ */
+ if (xchg(&port->priv, NULL) != cm_id)
+ return 0;
+
/*
* We need to return 1 so that the core will destroy
* it's own ID. What a great API design..
.delete_ctrl = nvmet_rdma_delete_ctrl,
};
+static void nvmet_rdma_add_one(struct ib_device *ib_device)
+{
+}
+
+static void nvmet_rdma_remove_one(struct ib_device *ib_device, void *client_data)
+{
+ struct nvmet_rdma_queue *queue;
+
+ /* Device is being removed, delete all queues using this device */
+ mutex_lock(&nvmet_rdma_queue_mutex);
+ list_for_each_entry(queue, &nvmet_rdma_queue_list, queue_list) {
+ if (queue->dev->device != ib_device)
+ continue;
+
+ pr_info("Removing queue %d\n", queue->idx);
+ __nvmet_rdma_queue_disconnect(queue);
+ }
+ mutex_unlock(&nvmet_rdma_queue_mutex);
+
+ flush_scheduled_work();
+}
+
+static struct ib_client nvmet_rdma_ib_client = {
+ .name = "nvmet_rdma",
+ .add = nvmet_rdma_add_one,
+ .remove = nvmet_rdma_remove_one
+};
+
static int __init nvmet_rdma_init(void)
{
- return nvmet_register_transport(&nvmet_rdma_ops);
+ int ret;
+
+ ret = ib_register_client(&nvmet_rdma_ib_client);
+ if (ret)
+ return ret;
+
+ ret = nvmet_register_transport(&nvmet_rdma_ops);
+ if (ret)
+ goto err_ib_client;
+
+ return 0;
+
+err_ib_client:
+ ib_unregister_client(&nvmet_rdma_ib_client);
+ return ret;
}
static void __exit nvmet_rdma_exit(void)
mutex_unlock(&nvmet_rdma_queue_mutex);
flush_scheduled_work();
+ ib_unregister_client(&nvmet_rdma_ib_client);
ida_destroy(&nvmet_rdma_queue_ida);
}
#define OTPC_CMD_READ 0x0
#define OTPC_CMD_OTP_PROG_ENABLE 0x2
#define OTPC_CMD_OTP_PROG_DISABLE 0x3
-#define OTPC_CMD_PROGRAM 0xA
+#define OTPC_CMD_PROGRAM 0x8
/* OTPC Status Bits */
#define OTPC_STAT_CMD_DONE BIT(1)
set_command(priv->base, OTPC_CMD_PROGRAM);
set_cpu_address(priv->base, address++);
for (i = 0; i < priv->map->otpc_row_size; i++) {
- writel(*buf, priv->base + priv->map->data_r_offset[i]);
+ writel(*buf, priv->base + priv->map->data_w_offset[i]);
buf++;
bytes_written += sizeof(*buf);
}
{
struct nvmem_cell *p;
+ mutex_lock(&nvmem_cells_mutex);
+
list_for_each_entry(p, &nvmem_cells, node)
- if (p && !strcmp(p->name, cell_id))
+ if (p && !strcmp(p->name, cell_id)) {
+ mutex_unlock(&nvmem_cells_mutex);
return p;
+ }
+
+ mutex_unlock(&nvmem_cells_mutex);
return NULL;
}
rval = device_add(&nvmem->dev);
if (rval)
- goto out;
+ goto err_put_device;
if (config->compat) {
rval = nvmem_setup_compat(nvmem, config);
if (rval)
- goto out;
+ goto err_device_del;
}
if (config->cells)
nvmem_add_cells(nvmem, config);
return nvmem;
-out:
- ida_simple_remove(&nvmem_ida, nvmem->id);
- kfree(nvmem);
+
+err_device_del:
+ device_del(&nvmem->dev);
+err_put_device:
+ put_device(&nvmem->dev);
+
return ERR_PTR(rval);
}
EXPORT_SYMBOL_GPL(nvmem_register);
nvmem_device_remove_all_cells(nvmem);
device_del(&nvmem->dev);
+ put_device(&nvmem->dev);
return 0;
}
.compatible = "rockchip,rk3188-efuse",
.data = (void *)&rockchip_rk3288_efuse_read,
},
+ {
+ .compatible = "rockchip,rk322x-efuse",
+ .data = (void *)&rockchip_rk3288_efuse_read,
+ },
{
.compatible = "rockchip,rk3288-efuse",
.data = (void *)&rockchip_rk3288_efuse_read,
BUG_ON(!dev);
ioc = GET_IOC(dev);
+ if (!ioc)
+ return DMA_ERROR_CODE;
BUG_ON(size <= 0);
BUG_ON(!dev);
ioc = GET_IOC(dev);
+ if (!ioc) {
+ WARN_ON(!ioc);
+ return;
+ }
DBG_RUN("%s() iovp 0x%lx/%x\n",
__func__, (long)iova, size);
BUG_ON(!dev);
ioc = GET_IOC(dev);
+ if (!ioc)
+ return 0;
DBG_RUN_SG("%s() START %d entries\n", __func__, nents);
BUG_ON(!dev);
ioc = GET_IOC(dev);
+ if (!ioc) {
+ WARN_ON(!ioc);
+ return;
+ }
DBG_RUN_SG("%s() START %d entries, %p,%x\n",
__func__, nents, sg_virt(sglist), sglist->length);
};
/* Looks nice and keeps the compiler happy */
-#define DINO_DEV(d) ((struct dino_device *) d)
+#define DINO_DEV(d) ({ \
+ void *__pdata = d; \
+ BUG_ON(!__pdata); \
+ (struct dino_device *)__pdata; })
/*
/* Looks nice and keeps the compiler happy */
-#define LBA_DEV(d) ((struct lba_device *) (d))
-
+#define LBA_DEV(d) ({ \
+ void *__pdata = d; \
+ BUG_ON(!__pdata); \
+ (struct lba_device *)__pdata; })
/*
** Only allow 8 subsidiary busses per LBA
return 0;
ioc = GET_IOC(dev);
+ if (!ioc)
+ return 0;
/*
* check if mask is >= than the current max IO Virt Address
int pide;
ioc = GET_IOC(dev);
+ if (!ioc)
+ return DMA_ERROR_CODE;
/* save offset bits */
offset = ((dma_addr_t) (long) addr) & ~IOVP_MASK;
DBG_RUN("%s() iovp 0x%lx/%x\n", __func__, (long) iova, size);
ioc = GET_IOC(dev);
+ if (!ioc) {
+ WARN_ON(!ioc);
+ return;
+ }
offset = iova & ~IOVP_MASK;
iova ^= offset; /* clear offset bits */
size += offset;
DBG_RUN_SG("%s() START %d entries\n", __func__, nents);
ioc = GET_IOC(dev);
+ if (!ioc)
+ return 0;
/* Fast path single entry scatterlists. */
if (nents == 1) {
__func__, nents, sg_virt(sglist), sglist->length);
ioc = GET_IOC(dev);
+ if (!ioc) {
+ WARN_ON(!ioc);
+ return;
+ }
#ifdef SBA_COLLECT_STATS
ioc->usg_calls++;
config PCI_ECAM
bool
+config PCI_LOCKLESS_CONFIG
+ bool
+
config PCI_IOV
bool "PCI IOV support"
depends on PCI
#define PCI_word_BAD (pos & 1)
#define PCI_dword_BAD (pos & 3)
+#ifdef CONFIG_PCI_LOCKLESS_CONFIG
+# define pci_lock_config(f) do { (void)(f); } while (0)
+# define pci_unlock_config(f) do { (void)(f); } while (0)
+#else
+# define pci_lock_config(f) raw_spin_lock_irqsave(&pci_lock, f)
+# define pci_unlock_config(f) raw_spin_unlock_irqrestore(&pci_lock, f)
+#endif
+
#define PCI_OP_READ(size, type, len) \
int pci_bus_read_config_##size \
(struct pci_bus *bus, unsigned int devfn, int pos, type *value) \
unsigned long flags; \
u32 data = 0; \
if (PCI_##size##_BAD) return PCIBIOS_BAD_REGISTER_NUMBER; \
- raw_spin_lock_irqsave(&pci_lock, flags); \
+ pci_lock_config(flags); \
res = bus->ops->read(bus, devfn, pos, len, &data); \
*value = (type)data; \
- raw_spin_unlock_irqrestore(&pci_lock, flags); \
+ pci_unlock_config(flags); \
return res; \
}
int res; \
unsigned long flags; \
if (PCI_##size##_BAD) return PCIBIOS_BAD_REGISTER_NUMBER; \
- raw_spin_lock_irqsave(&pci_lock, flags); \
+ pci_lock_config(flags); \
res = bus->ops->write(bus, devfn, pos, len, value); \
- raw_spin_unlock_irqrestore(&pci_lock, flags); \
+ pci_unlock_config(flags); \
return res; \
}
static int vmd_enable_domain(struct vmd_dev *vmd)
{
struct pci_sysdata *sd = &vmd->sysdata;
+ struct fwnode_handle *fn;
struct resource *res;
u32 upper_bits;
unsigned long flags;
sd->node = pcibus_to_node(vmd->dev->bus);
- vmd->irq_domain = pci_msi_create_irq_domain(NULL, &vmd_msi_domain_info,
+ fn = irq_domain_alloc_named_id_fwnode("VMD-MSI", vmd->sysdata.domain);
+ if (!fn)
+ return -ENODEV;
+
+ vmd->irq_domain = pci_msi_create_irq_domain(fn, &vmd_msi_domain_info,
x86_vector_domain);
+ irq_domain_free_fwnode(fn);
if (!vmd->irq_domain)
return -ENODEV;
if (!domain)
return NULL;
- domain->bus_token = DOMAIN_BUS_PCI_MSI;
+ irq_domain_update_bus_token(domain, DOMAIN_BUS_PCI_MSI);
return domain;
}
EXPORT_SYMBOL_GPL(pci_msi_create_irq_domain);
#include "pci.h"
/*
- * The UUID is defined in the PCI Firmware Specification available here:
+ * The GUID is defined in the PCI Firmware Specification available here:
* https://www.pcisig.com/members/downloads/pcifw_r3_1_13Dec10.pdf
*/
-const u8 pci_acpi_dsm_uuid[] = {
- 0xd0, 0x37, 0xc9, 0xe5, 0x53, 0x35, 0x7a, 0x4d,
- 0x91, 0x17, 0xea, 0x4d, 0x19, 0xc3, 0x43, 0x4d
-};
+const guid_t pci_acpi_dsm_guid =
+ GUID_INIT(0xe5c937d0, 0x3553, 0x4d7a,
+ 0x91, 0x17, 0xea, 0x4d, 0x19, 0xc3, 0x43, 0x4d);
#if defined(CONFIG_PCI_QUIRKS) && defined(CONFIG_ARM64)
static int acpi_get_rc_addr(struct acpi_device *adev, struct resource *res)
/**
* pci_acpi_wake_bus - Root bus wakeup notification fork function.
- * @work: Work item to handle.
+ * @context: Device wakeup context.
*/
-static void pci_acpi_wake_bus(struct work_struct *work)
+static void pci_acpi_wake_bus(struct acpi_device_wakeup_context *context)
{
struct acpi_device *adev;
struct acpi_pci_root *root;
- adev = container_of(work, struct acpi_device, wakeup.context.work);
+ adev = container_of(context, struct acpi_device, wakeup.context);
root = acpi_driver_data(adev);
pci_pme_wakeup_bus(root->bus);
}
/**
* pci_acpi_wake_dev - PCI device wakeup notification work function.
- * @handle: ACPI handle of a device the notification is for.
- * @work: Work item to handle.
+ * @context: Device wakeup context.
*/
-static void pci_acpi_wake_dev(struct work_struct *work)
+static void pci_acpi_wake_dev(struct acpi_device_wakeup_context *context)
{
- struct acpi_device_wakeup_context *context;
struct pci_dev *pci_dev;
- context = container_of(work, struct acpi_device_wakeup_context, work);
pci_dev = to_pci_dev(context->dev);
if (pci_dev->pme_poll)
if (pci_dev->current_state == PCI_D3cold) {
pci_wakeup_event(pci_dev);
- pm_runtime_resume(&pci_dev->dev);
+ pm_request_resume(&pci_dev->dev);
return;
}
pci_check_pme_status(pci_dev);
pci_wakeup_event(pci_dev);
- pm_runtime_resume(&pci_dev->dev);
+ pm_request_resume(&pci_dev->dev);
pci_pme_wakeup_bus(pci_dev->subordinate);
}
return state_conv[state];
}
-static bool acpi_pci_can_wakeup(struct pci_dev *dev)
-{
- struct acpi_device *adev = ACPI_COMPANION(&dev->dev);
- return adev ? acpi_device_can_wakeup(adev) : false;
-}
-
-static void acpi_pci_propagate_wakeup_enable(struct pci_bus *bus, bool enable)
+static int acpi_pci_propagate_wakeup(struct pci_bus *bus, bool enable)
{
while (bus->parent) {
- if (!acpi_pm_device_sleep_wake(&bus->self->dev, enable))
- return;
- bus = bus->parent;
- }
+ if (acpi_pm_device_can_wakeup(&bus->self->dev))
+ return acpi_pm_set_device_wakeup(&bus->self->dev, enable);
- /* We have reached the root bus. */
- if (bus->bridge)
- acpi_pm_device_sleep_wake(bus->bridge, enable);
-}
-
-static int acpi_pci_sleep_wake(struct pci_dev *dev, bool enable)
-{
- if (acpi_pci_can_wakeup(dev))
- return acpi_pm_device_sleep_wake(&dev->dev, enable);
-
- acpi_pci_propagate_wakeup_enable(dev->bus, enable);
- return 0;
-}
-
-static void acpi_pci_propagate_run_wake(struct pci_bus *bus, bool enable)
-{
- while (bus->parent) {
- struct pci_dev *bridge = bus->self;
-
- if (bridge->pme_interrupt)
- return;
- if (!acpi_pm_device_run_wake(&bridge->dev, enable))
- return;
bus = bus->parent;
}
/* We have reached the root bus. */
- if (bus->bridge)
- acpi_pm_device_run_wake(bus->bridge, enable);
+ if (bus->bridge) {
+ if (acpi_pm_device_can_wakeup(bus->bridge))
+ return acpi_pm_set_device_wakeup(bus->bridge, enable);
+ }
+ return 0;
}
-static int acpi_pci_run_wake(struct pci_dev *dev, bool enable)
+static int acpi_pci_wakeup(struct pci_dev *dev, bool enable)
{
- /*
- * Per PCI Express Base Specification Revision 2.0 section
- * 5.3.3.2 Link Wakeup, platform support is needed for D3cold
- * waking up to power on the main link even if there is PME
- * support for D3cold
- */
- if (dev->pme_interrupt && !dev->runtime_d3cold)
- return 0;
-
- if (!acpi_pm_device_run_wake(&dev->dev, enable))
- return 0;
+ if (acpi_pm_device_can_wakeup(&dev->dev))
+ return acpi_pm_set_device_wakeup(&dev->dev, enable);
- acpi_pci_propagate_run_wake(dev->bus, enable);
- return 0;
+ return acpi_pci_propagate_wakeup(dev->bus, enable);
}
static bool acpi_pci_need_resume(struct pci_dev *dev)
.set_state = acpi_pci_set_power_state,
.get_state = acpi_pci_get_power_state,
.choose_state = acpi_pci_choose_state,
- .sleep_wake = acpi_pci_sleep_wake,
- .run_wake = acpi_pci_run_wake,
+ .set_wakeup = acpi_pci_wakeup,
.need_resume = acpi_pci_need_resume,
};
if (!pci_is_root_bus(bus))
return;
- obj = acpi_evaluate_dsm(ACPI_HANDLE(bus->bridge), pci_acpi_dsm_uuid, 3,
+ obj = acpi_evaluate_dsm(ACPI_HANDLE(bus->bridge), &pci_acpi_dsm_guid, 3,
RESET_DELAY_DSM, NULL);
if (!obj)
return;
if (bridge->ignore_reset_delay)
pdev->d3cold_delay = 0;
- obj = acpi_evaluate_dsm(handle, pci_acpi_dsm_uuid, 3,
+ obj = acpi_evaluate_dsm(handle, &pci_acpi_dsm_guid, 3,
FUNCTION_DELAY_DSM, NULL);
if (!obj)
return;
return;
device_set_wakeup_capable(dev, true);
- acpi_pci_sleep_wake(pci_dev, false);
- if (adev->wakeup.flags.run_wake)
- device_set_run_wake(dev, true);
+ acpi_pci_wakeup(pci_dev, false);
}
static void pci_acpi_cleanup(struct device *dev)
return;
pci_acpi_remove_pm_notifier(adev);
- if (adev->wakeup.flags.valid) {
+ if (adev->wakeup.flags.valid)
device_set_wakeup_capable(dev, false);
- device_set_run_wake(dev, false);
- }
}
static bool pci_acpi_bus_match(struct device *dev)
*
* Allow PCI IDs to be added to an existing driver via sysfs.
*/
-static ssize_t store_new_id(struct device_driver *driver, const char *buf,
+static ssize_t new_id_store(struct device_driver *driver, const char *buf,
size_t count)
{
struct pci_driver *pdrv = to_pci_driver(driver);
return retval;
return count;
}
-static DRIVER_ATTR(new_id, S_IWUSR, NULL, store_new_id);
+static DRIVER_ATTR_WO(new_id);
/**
* store_remove_id - remove a PCI device ID from this driver
*
* Removes a dynamic pci device ID to this driver.
*/
-static ssize_t store_remove_id(struct device_driver *driver, const char *buf,
+static ssize_t remove_id_store(struct device_driver *driver, const char *buf,
size_t count)
{
struct pci_dynid *dynid, *n;
return retval;
}
-static DRIVER_ATTR(remove_id, S_IWUSR, NULL, store_remove_id);
+static DRIVER_ATTR_WO(remove_id);
static struct attribute *pci_drv_attrs[] = {
&driver_attr_new_id.attr,
return 0;
}
+static bool pci_physfn_is_probed(struct pci_dev *dev)
+{
+#ifdef CONFIG_PCI_IOV
+ return dev->is_virtfn && dev->physfn->is_probed;
+#else
+ return false;
+#endif
+}
+
static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
const struct pci_device_id *id)
{
- int error, node;
+ int error, node, cpu;
struct drv_dev_and_id ddi = { drv, dev, id };
/*
* on the right node.
*/
node = dev_to_node(&dev->dev);
+ dev->is_probed = 1;
+
+ cpu_hotplug_disable();
/*
- * On NUMA systems, we are likely to call a PF probe function using
- * work_on_cpu(). If that probe calls pci_enable_sriov() (which
- * adds the VF devices via pci_bus_add_device()), we may re-enter
- * this function to call the VF probe function. Calling
- * work_on_cpu() again will cause a lockdep warning. Since VFs are
- * always on the same node as the PF, we can work around this by
- * avoiding work_on_cpu() when we're already on the correct node.
- *
- * Preemption is enabled, so it's theoretically unsafe to use
- * numa_node_id(), but even if we run the probe function on the
- * wrong node, it should be functionally correct.
+ * Prevent nesting work_on_cpu() for the case where a Virtual Function
+ * device is probed from work_on_cpu() of the Physical device.
*/
- if (node >= 0 && node != numa_node_id()) {
- int cpu;
-
- get_online_cpus();
+ if (node < 0 || node >= MAX_NUMNODES || !node_online(node) ||
+ pci_physfn_is_probed(dev))
+ cpu = nr_cpu_ids;
+ else
cpu = cpumask_any_and(cpumask_of_node(node), cpu_online_mask);
- if (cpu < nr_cpu_ids)
- error = work_on_cpu(cpu, local_pci_probe, &ddi);
- else
- error = local_pci_probe(&ddi);
- put_online_cpus();
- } else
+
+ if (cpu < nr_cpu_ids)
+ error = work_on_cpu(cpu, local_pci_probe, &ddi);
+ else
error = local_pci_probe(&ddi);
+ dev->is_probed = 0;
+ cpu_hotplug_enable();
return error;
}
pci_restore_standard_config(pci_dev);
pci_fixup_device(pci_fixup_resume_early, pci_dev);
- __pci_enable_wake(pci_dev, PCI_D0, true, false);
+ pci_enable_wake(pci_dev, PCI_D0, false);
pci_fixup_device(pci_fixup_resume, pci_dev);
rc = pm->runtime_resume(dev);
if (!handle)
return -1;
- obj = acpi_evaluate_dsm(handle, pci_acpi_dsm_uuid, 0x2,
+ obj = acpi_evaluate_dsm(handle, &pci_acpi_dsm_guid, 0x2,
DEVICE_LABEL_DSM, NULL);
if (!obj)
return -1;
if (!handle)
return false;
- return !!acpi_check_dsm(handle, pci_acpi_dsm_uuid, 0x2,
+ return !!acpi_check_dsm(handle, &pci_acpi_dsm_guid, 0x2,
1 << DEVICE_LABEL_DSM);
}
return PCI_D3hot;
}
-static int mid_pci_sleep_wake(struct pci_dev *dev, bool enable)
-{
- return 0;
-}
-
-static int mid_pci_run_wake(struct pci_dev *dev, bool enable)
+static int mid_pci_wakeup(struct pci_dev *dev, bool enable)
{
return 0;
}
.set_state = mid_pci_set_power_state,
.get_state = mid_pci_get_power_state,
.choose_state = mid_pci_choose_state,
- .sleep_wake = mid_pci_sleep_wake,
- .run_wake = mid_pci_run_wake,
+ .set_wakeup = mid_pci_wakeup,
.need_resume = mid_pci_need_resume,
};
int pci_set_platform_pm(const struct pci_platform_pm_ops *ops)
{
if (!ops->is_manageable || !ops->set_state || !ops->get_state ||
- !ops->choose_state || !ops->sleep_wake || !ops->run_wake ||
- !ops->need_resume)
+ !ops->choose_state || !ops->set_wakeup || !ops->need_resume)
return -EINVAL;
pci_platform_pm = ops;
return 0;
pci_platform_pm->choose_state(dev) : PCI_POWER_ERROR;
}
-static inline int platform_pci_sleep_wake(struct pci_dev *dev, bool enable)
+static inline int platform_pci_set_wakeup(struct pci_dev *dev, bool enable)
{
return pci_platform_pm ?
- pci_platform_pm->sleep_wake(dev, enable) : -ENODEV;
-}
-
-static inline int platform_pci_run_wake(struct pci_dev *dev, bool enable)
-{
- return pci_platform_pm ?
- pci_platform_pm->run_wake(dev, enable) : -ENODEV;
+ pci_platform_pm->set_wakeup(dev, enable) : -ENODEV;
}
static inline bool platform_pci_need_resume(struct pci_dev *dev)
pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, pmcsr);
}
+static void pci_pme_restore(struct pci_dev *dev)
+{
+ u16 pmcsr;
+
+ if (!dev->pme_support)
+ return;
+
+ pci_read_config_word(dev, dev->pm_cap + PCI_PM_CTRL, &pmcsr);
+ if (dev->wakeup_prepared) {
+ pmcsr |= PCI_PM_CTRL_PME_ENABLE;
+ } else {
+ pmcsr &= ~PCI_PM_CTRL_PME_ENABLE;
+ pmcsr |= PCI_PM_CTRL_PME_STATUS;
+ }
+ pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, pmcsr);
+}
+
/**
* pci_pme_active - enable or disable PCI device's PME# function
* @dev: PCI device to handle.
EXPORT_SYMBOL(pci_pme_active);
/**
- * __pci_enable_wake - enable PCI device as wakeup event source
+ * pci_enable_wake - enable PCI device as wakeup event source
* @dev: PCI device affected
* @state: PCI state from which device will issue wakeup events
- * @runtime: True if the events are to be generated at run time
* @enable: True to enable event generation; false to disable
*
* This enables the device as a wakeup event source, or disables it.
* Error code depending on the platform is returned if both the platform and
* the native mechanism fail to enable the generation of wake-up events
*/
-int __pci_enable_wake(struct pci_dev *dev, pci_power_t state,
- bool runtime, bool enable)
+int pci_enable_wake(struct pci_dev *dev, pci_power_t state, bool enable)
{
int ret = 0;
- if (enable && !runtime && !device_may_wakeup(&dev->dev))
- return -EINVAL;
-
- /* Don't do the same thing twice in a row for one device. */
- if (!!enable == !!dev->wakeup_prepared)
+ /*
+ * Don't do the same thing twice in a row for one device, but restore
+ * PME Enable in case it has been updated by config space restoration.
+ */
+ if (!!enable == !!dev->wakeup_prepared) {
+ pci_pme_restore(dev);
return 0;
+ }
/*
* According to "PCI System Architecture" 4th ed. by Tom Shanley & Don
pci_pme_active(dev, true);
else
ret = 1;
- error = runtime ? platform_pci_run_wake(dev, true) :
- platform_pci_sleep_wake(dev, true);
+ error = platform_pci_set_wakeup(dev, true);
if (ret)
ret = error;
if (!ret)
dev->wakeup_prepared = true;
} else {
- if (runtime)
- platform_pci_run_wake(dev, false);
- else
- platform_pci_sleep_wake(dev, false);
+ platform_pci_set_wakeup(dev, false);
pci_pme_active(dev, false);
dev->wakeup_prepared = false;
}
return ret;
}
-EXPORT_SYMBOL(__pci_enable_wake);
+EXPORT_SYMBOL(pci_enable_wake);
/**
* pci_wake_from_d3 - enable/disable device to wake up from D3_hot or D3_cold
dev->runtime_d3cold = target_state == PCI_D3cold;
- __pci_enable_wake(dev, target_state, true, pci_dev_run_wake(dev));
+ pci_enable_wake(dev, target_state, pci_dev_run_wake(dev));
error = pci_set_power_state(dev, target_state);
if (error) {
- __pci_enable_wake(dev, target_state, true, false);
+ pci_enable_wake(dev, target_state, false);
dev->runtime_d3cold = false;
}
{
struct pci_bus *bus = dev->bus;
- if (device_run_wake(&dev->dev))
+ if (device_can_wakeup(&dev->dev))
return true;
if (!dev->pme_support)
while (bus->parent) {
struct pci_dev *bridge = bus->self;
- if (device_run_wake(&bridge->dev))
+ if (device_can_wakeup(&bridge->dev))
return true;
bus = bus->parent;
/* We have reached the root bus. */
if (bus->bridge)
- return device_run_wake(bus->bridge);
+ return device_can_wakeup(bus->bridge);
return false;
}
* platform; to be used during system-wide transitions from a
* sleeping state to the working state and vice versa
*
- * @sleep_wake: enables/disables the system wake up capability of given device
- *
- * @run_wake: enables/disables the platform to generate run-time wake-up events
- * for given device (the device's wake-up capability has to be
- * enabled by @sleep_wake for this feature to work)
+ * @set_wakeup: enables/disables wakeup capability for the device
*
* @need_resume: returns 'true' if the given device (which is currently
* suspended) needs to be resumed to be configured for system
int (*set_state)(struct pci_dev *dev, pci_power_t state);
pci_power_t (*get_state)(struct pci_dev *dev);
pci_power_t (*choose_state)(struct pci_dev *dev);
- int (*sleep_wake)(struct pci_dev *dev, bool enable);
- int (*run_wake)(struct pci_dev *dev, bool enable);
+ int (*set_wakeup)(struct pci_dev *dev, bool enable);
bool (*need_resume)(struct pci_dev *dev);
};
}
/**
- * pcie_pme_set_native - Set the PME interrupt flag for given device.
+ * pcie_pme_can_wakeup - Set the wakeup capability flag.
* @dev: PCI device to handle.
* @ign: Ignored.
*/
-static int pcie_pme_set_native(struct pci_dev *dev, void *ign)
+static int pcie_pme_can_wakeup(struct pci_dev *dev, void *ign)
{
- device_set_run_wake(&dev->dev, true);
- dev->pme_interrupt = true;
+ device_set_wakeup_capable(&dev->dev, true);
return 0;
}
/**
- * pcie_pme_mark_devices - Set the PME interrupt flag for devices below a port.
+ * pcie_pme_mark_devices - Set the wakeup flag for devices below a port.
* @port: PCIe root port or event collector to handle.
*
* For each device below given root port, including the port itself (or for each
* root complex integrated endpoint if @port is a root complex event collector)
- * set the flag indicating that it can signal run-time wake-up events via PCIe
- * PME interrupts.
+ * set the flag indicating that it can signal run-time wake-up events.
*/
static void pcie_pme_mark_devices(struct pci_dev *port)
{
- pcie_pme_set_native(port, NULL);
+ pcie_pme_can_wakeup(port, NULL);
if (port->subordinate)
- pci_walk_bus(port->subordinate, pcie_pme_set_native, NULL);
+ pci_walk_bus(port->subordinate, pcie_pme_can_wakeup, NULL);
}
/**
* and causes the driver to probe for all devices again.
*/
static ssize_t
-pcmcia_store_new_id(struct device_driver *driver, const char *buf, size_t count)
+new_id_store(struct device_driver *driver, const char *buf, size_t count)
{
struct pcmcia_dynid *dynid;
struct pcmcia_driver *pdrv = to_pcmcia_drv(driver);
return retval;
return count;
}
-static DRIVER_ATTR(new_id, S_IWUSR, NULL, pcmcia_store_new_id);
+static DRIVER_ATTR_WO(new_id);
static void
pcmcia_free_dynids(struct pcmcia_driver *drv)
phy users can obtain reference to the PHY. All the users of this
framework should select this config.
-config PHY_BCM_NS_USB2
- tristate "Broadcom Northstar USB 2.0 PHY Driver"
- depends on ARCH_BCM_IPROC || COMPILE_TEST
- depends on HAS_IOMEM && OF
- select GENERIC_PHY
- help
- Enable this to support Broadcom USB 2.0 PHY connected to the USB
- controller on Northstar family.
-
-config PHY_BCM_NS_USB3
- tristate "Broadcom Northstar USB 3.0 PHY Driver"
- depends on ARCH_BCM_IPROC || COMPILE_TEST
- depends on HAS_IOMEM && OF
- select GENERIC_PHY
- help
- Enable this to support Broadcom USB 3.0 PHY connected to the USB
- controller on Northstar family.
-
-config PHY_BERLIN_USB
- tristate "Marvell Berlin USB PHY Driver"
- depends on ARCH_BERLIN && RESET_CONTROLLER && HAS_IOMEM && OF
- select GENERIC_PHY
- help
- Enable this to support the USB PHY on Marvell Berlin SoCs.
-
-config PHY_BERLIN_SATA
- tristate "Marvell Berlin SATA PHY driver"
- depends on ARCH_BERLIN && HAS_IOMEM && OF
- select GENERIC_PHY
- help
- Enable this to support the SATA PHY on Marvell Berlin SoCs.
-
-config ARMADA375_USBCLUSTER_PHY
- def_bool y
- depends on MACH_ARMADA_375 || COMPILE_TEST
- depends on OF && HAS_IOMEM
- select GENERIC_PHY
-
-config PHY_DA8XX_USB
- tristate "TI DA8xx USB PHY Driver"
- depends on ARCH_DAVINCI_DA8XX
- select GENERIC_PHY
- select MFD_SYSCON
- help
- Enable this to support the USB PHY on DA8xx SoCs.
-
- This driver controls both the USB 1.1 PHY and the USB 2.0 PHY.
-
-config PHY_DM816X_USB
- tristate "TI dm816x USB PHY driver"
- depends on ARCH_OMAP2PLUS
- depends on USB_SUPPORT
- select GENERIC_PHY
- select USB_PHY
- help
- Enable this for dm816x USB to work.
-
-config PHY_EXYNOS_MIPI_VIDEO
- tristate "S5P/EXYNOS SoC series MIPI CSI-2/DSI PHY driver"
- depends on HAS_IOMEM
- depends on ARCH_S5PV210 || ARCH_EXYNOS || COMPILE_TEST
- select GENERIC_PHY
- default y if ARCH_S5PV210 || ARCH_EXYNOS
- help
- Support for MIPI CSI-2 and MIPI DSI DPHY found on Samsung S5P
- and EXYNOS SoCs.
-
config PHY_LPC18XX_USB_OTG
tristate "NXP LPC18xx/43xx SoC USB OTG PHY driver"
depends on OF && (ARCH_LPC18XX || COMPILE_TEST)
This driver is need for USB0 support on LPC18xx/43xx and takes
care of enabling and clock setup.
-config PHY_PXA_28NM_HSIC
- tristate "Marvell USB HSIC 28nm PHY Driver"
- depends on HAS_IOMEM
- select GENERIC_PHY
- help
- Enable this to support Marvell USB HSIC PHY driver for Marvell
- SoC. This driver will do the PHY initialization and shutdown.
- The PHY driver will be used by Marvell ehci driver.
-
- To compile this driver as a module, choose M here.
-
-config PHY_PXA_28NM_USB2
- tristate "Marvell USB 2.0 28nm PHY Driver"
- depends on HAS_IOMEM
- select GENERIC_PHY
- help
- Enable this to support Marvell USB 2.0 PHY driver for Marvell
- SoC. This driver will do the PHY initialization and shutdown.
- The PHY driver will be used by Marvell udc/ehci/otg driver.
-
- To compile this driver as a module, choose M here.
-
-config PHY_MVEBU_SATA
- def_bool y
- depends on ARCH_DOVE || MACH_DOVE || MACH_KIRKWOOD
- depends on OF
- select GENERIC_PHY
-
-config PHY_MIPHY28LP
- tristate "STMicroelectronics MIPHY28LP PHY driver for STiH407"
- depends on ARCH_STI
- select GENERIC_PHY
- help
- Enable this to support the miphy transceiver (for SATA/PCIE/USB3)
- that is part of STMicroelectronics STiH407 SoC.
-
-config PHY_RCAR_GEN2
- tristate "Renesas R-Car generation 2 USB PHY driver"
- depends on ARCH_RENESAS
- depends on GENERIC_PHY
- help
- Support for USB PHY found on Renesas R-Car generation 2 SoCs.
-
-config PHY_RCAR_GEN3_USB2
- tristate "Renesas R-Car generation 3 USB 2.0 PHY driver"
- depends on ARCH_RENESAS
- depends on EXTCON
- select GENERIC_PHY
- help
- Support for USB 2.0 PHY found on Renesas R-Car generation 3 SoCs.
-
-config OMAP_CONTROL_PHY
- tristate "OMAP CONTROL PHY Driver"
- depends on ARCH_OMAP2PLUS || COMPILE_TEST
- help
- Enable this to add support for the PHY part present in the control
- module. This driver has API to power on the USB2 PHY and to write to
- the mailbox. The mailbox is present only in omap4 and the register to
- power on the USB2 PHY is present in OMAP4 and OMAP5. OMAP5 has an
- additional register to power on USB3 PHY/SATA PHY/PCIE PHY
- (PIPE3 PHY).
-
-config OMAP_USB2
- tristate "OMAP USB2 PHY Driver"
- depends on ARCH_OMAP2PLUS
- depends on USB_SUPPORT
- select GENERIC_PHY
- select USB_PHY
- select OMAP_CONTROL_PHY
- depends on OMAP_OCP2SCP
- help
- Enable this to support the transceiver that is part of SOC. This
- driver takes care of all the PHY functionality apart from comparator.
- The USB OTG controller communicates with the comparator using this
- driver.
-
-config TI_PIPE3
- tristate "TI PIPE3 PHY Driver"
- depends on ARCH_OMAP2PLUS || COMPILE_TEST
- select GENERIC_PHY
- select OMAP_CONTROL_PHY
- depends on OMAP_OCP2SCP
- help
- Enable this to support the PIPE3 PHY that is part of TI SOCs. This
- driver takes care of all the PHY functionality apart from comparator.
- This driver interacts with the "OMAP Control PHY Driver" to power
- on/off the PHY.
-
-config TWL4030_USB
- tristate "TWL4030 USB Transceiver Driver"
- depends on TWL4030_CORE && REGULATOR_TWL4030 && USB_MUSB_OMAP2PLUS
- depends on USB_SUPPORT
- depends on USB_GADGET || !USB_GADGET # if USB_GADGET=m, this can't 'y'
- select GENERIC_PHY
- select USB_PHY
- help
- Enable this to support the USB OTG transceiver on TWL4030
- family chips (including the TWL5030 and TPS659x0 devices).
- This transceiver supports high and full speed devices plus,
- in host mode, low speed.
-
-config PHY_EXYNOS_DP_VIDEO
- tristate "EXYNOS SoC series Display Port PHY driver"
- depends on OF
- depends on ARCH_EXYNOS || COMPILE_TEST
- default ARCH_EXYNOS
- select GENERIC_PHY
- help
- Support for Display Port PHY found on Samsung EXYNOS SoCs.
-
-config BCM_KONA_USB2_PHY
- tristate "Broadcom Kona USB2 PHY Driver"
- depends on HAS_IOMEM
- select GENERIC_PHY
- help
- Enable this to support the Broadcom Kona USB 2.0 PHY.
-
-config PHY_EXYNOS5250_SATA
- tristate "Exynos5250 Sata SerDes/PHY driver"
- depends on SOC_EXYNOS5250
- depends on HAS_IOMEM
- depends on OF
- select GENERIC_PHY
- select I2C
- select I2C_S3C2410
- select MFD_SYSCON
- help
- Enable this to support SATA SerDes/Phy found on Samsung's
- Exynos5250 based SoCs.This SerDes/Phy supports SATA 1.5 Gb/s,
- SATA 3.0 Gb/s, SATA 6.0 Gb/s speeds. It supports one SATA host
- port to accept one SATA device.
-
-config PHY_HIX5HD2_SATA
- tristate "HIX5HD2 SATA PHY Driver"
- depends on ARCH_HIX5HD2 && OF && HAS_IOMEM
- select GENERIC_PHY
- select MFD_SYSCON
- help
- Support for SATA PHY on Hisilicon hix5hd2 Soc.
-
config PHY_MT65XX_USB3
tristate "Mediatek USB3.0 PHY Driver"
depends on ARCH_MEDIATEK && OF
Say 'Y' here to add support for Mediatek USB3.0 PHY driver,
it supports multiple usb2.0 and usb3.0 ports.
-config PHY_HI6220_USB
- tristate "hi6220 USB PHY support"
- depends on (ARCH_HISI && ARM64) || COMPILE_TEST
- select GENERIC_PHY
- select MFD_SYSCON
- help
- Enable this to support the HISILICON HI6220 USB PHY.
-
- To compile this driver as a module, choose M here.
-
-config PHY_SUN4I_USB
- tristate "Allwinner sunxi SoC USB PHY driver"
- depends on ARCH_SUNXI && HAS_IOMEM && OF
- depends on RESET_CONTROLLER
- depends on EXTCON
- depends on POWER_SUPPLY
- depends on USB_SUPPORT
- select GENERIC_PHY
- select USB_COMMON
- help
- Enable this to support the transceiver that is part of Allwinner
- sunxi SoCs.
-
- This driver controls the entire USB PHY block, both the USB OTG
- parts, as well as the 2 regular USB 2 host PHYs.
-
-config PHY_SUN9I_USB
- tristate "Allwinner sun9i SoC USB PHY driver"
- depends on ARCH_SUNXI && HAS_IOMEM && OF
- depends on RESET_CONTROLLER
- depends on USB_SUPPORT
- select USB_COMMON
- select GENERIC_PHY
- help
- Enable this to support the transceiver that is part of Allwinner
- sun9i SoCs.
-
- This driver controls each individual USB 2 host PHY.
-
-config PHY_SAMSUNG_USB2
- tristate "Samsung USB 2.0 PHY driver"
- depends on HAS_IOMEM
- depends on USB_EHCI_EXYNOS || USB_OHCI_EXYNOS || USB_DWC2
- select GENERIC_PHY
- select MFD_SYSCON
- default ARCH_EXYNOS
- help
- Enable this to support the Samsung USB 2.0 PHY driver for Samsung
- SoCs. This driver provides the interface for USB 2.0 PHY. Support
- for particular PHYs will be enabled based on the SoC type in addition
- to this driver.
-
-config PHY_S5PV210_USB2
- bool "Support for S5PV210"
- depends on PHY_SAMSUNG_USB2
- depends on ARCH_S5PV210
- help
- Enable USB PHY support for S5PV210. This option requires that Samsung
- USB 2.0 PHY driver is enabled and means that support for this
- particular SoC is compiled in the driver. In case of S5PV210 two phys
- are available - device and host.
-
-config PHY_EXYNOS4210_USB2
- bool
- depends on PHY_SAMSUNG_USB2
- default CPU_EXYNOS4210
-
-config PHY_EXYNOS4X12_USB2
- bool
- depends on PHY_SAMSUNG_USB2
- default SOC_EXYNOS3250 || SOC_EXYNOS4212 || SOC_EXYNOS4412
-
-config PHY_EXYNOS5250_USB2
- bool
- depends on PHY_SAMSUNG_USB2
- default SOC_EXYNOS5250 || SOC_EXYNOS5420
-
-config PHY_EXYNOS5_USBDRD
- tristate "Exynos5 SoC series USB DRD PHY driver"
- depends on ARCH_EXYNOS && OF
- depends on HAS_IOMEM
- depends on USB_DWC3_EXYNOS
- select GENERIC_PHY
- select MFD_SYSCON
- default y
- help
- Enable USB DRD PHY support for Exynos 5 SoC series.
- This driver provides PHY interface for USB 3.0 DRD controller
- present on Exynos5 SoC series.
-
-config PHY_EXYNOS_PCIE
- bool "Exynos PCIe PHY driver"
- depends on OF && (ARCH_EXYNOS || COMPILE_TEST)
- select GENERIC_PHY
- help
- Enable PCIe PHY support for Exynos SoC series.
- This driver provides PHY interface for Exynos PCIe controller.
-
config PHY_PISTACHIO_USB
tristate "IMG Pistachio USB2.0 PHY driver"
depends on MACH_PISTACHIO
help
Enable this to support the USB2.0 PHY on the IMG Pistachio SoC.
-config PHY_QCOM_APQ8064_SATA
- tristate "Qualcomm APQ8064 SATA SerDes/PHY driver"
- depends on ARCH_QCOM
- depends on HAS_IOMEM
- depends on OF
- select GENERIC_PHY
-
-config PHY_QCOM_IPQ806X_SATA
- tristate "Qualcomm IPQ806x SATA SerDes/PHY driver"
- depends on ARCH_QCOM
- depends on HAS_IOMEM
- depends on OF
- select GENERIC_PHY
-
-config PHY_ROCKCHIP_USB
- tristate "Rockchip USB2 PHY Driver"
- depends on ARCH_ROCKCHIP && OF
- select GENERIC_PHY
- help
- Enable this to support the Rockchip USB 2.0 PHY.
-
-config PHY_ROCKCHIP_INNO_USB2
- tristate "Rockchip INNO USB2PHY Driver"
- depends on (ARCH_ROCKCHIP || COMPILE_TEST) && OF
- depends on COMMON_CLK
- depends on EXTCON
- depends on USB_SUPPORT
- select GENERIC_PHY
- select USB_COMMON
- help
- Support for Rockchip USB2.0 PHY with Innosilicon IP block.
-
-config PHY_ROCKCHIP_EMMC
- tristate "Rockchip EMMC PHY Driver"
- depends on ARCH_ROCKCHIP && OF
- select GENERIC_PHY
- help
- Enable this to support the Rockchip EMMC PHY.
-
-config PHY_ROCKCHIP_DP
- tristate "Rockchip Display Port PHY Driver"
- depends on ARCH_ROCKCHIP && OF
- select GENERIC_PHY
- help
- Enable this to support the Rockchip Display Port PHY.
-
-config PHY_ROCKCHIP_PCIE
- tristate "Rockchip PCIe PHY Driver"
- depends on (ARCH_ROCKCHIP && OF) || COMPILE_TEST
- select GENERIC_PHY
- select MFD_SYSCON
- help
- Enable this to support the Rockchip PCIe PHY.
-
-config PHY_ROCKCHIP_TYPEC
- tristate "Rockchip TYPEC PHY Driver"
- depends on OF && (ARCH_ROCKCHIP || COMPILE_TEST)
- select EXTCON
- select GENERIC_PHY
- select RESET_CONTROLLER
- help
- Enable this to support the Rockchip USB TYPEC PHY.
-
-config PHY_ST_SPEAR1310_MIPHY
- tristate "ST SPEAR1310-MIPHY driver"
- select GENERIC_PHY
- depends on MACH_SPEAR1310 || COMPILE_TEST
- help
- Support for ST SPEAr1310 MIPHY which can be used for PCIe and SATA.
-
-config PHY_ST_SPEAR1340_MIPHY
- tristate "ST SPEAR1340-MIPHY driver"
- select GENERIC_PHY
- depends on MACH_SPEAR1340 || COMPILE_TEST
- help
- Support for ST SPEAr1340 MIPHY which can be used for PCIe and SATA.
-
config PHY_XGENE
tristate "APM X-Gene 15Gbps PHY support"
depends on HAS_IOMEM && OF && (ARM64 || COMPILE_TEST)
help
This option enables support for APM X-Gene SoC multi-purpose PHY.
-config PHY_STIH407_USB
- tristate "STMicroelectronics USB2 picoPHY driver for STiH407 family"
- depends on RESET_CONTROLLER
- depends on ARCH_STI || COMPILE_TEST
- select GENERIC_PHY
- help
- Enable this support to enable the picoPHY device used by USB2
- and USB3 controllers on STMicroelectronics STiH407 SoC families.
-
-config PHY_QCOM_QMP
- tristate "Qualcomm QMP PHY Driver"
- depends on OF && COMMON_CLK && (ARCH_QCOM || COMPILE_TEST)
- select GENERIC_PHY
- help
- Enable this to support the QMP PHY transceiver that is used
- with controllers such as PCIe, UFS, and USB on Qualcomm chips.
-
-config PHY_QCOM_QUSB2
- tristate "Qualcomm QUSB2 PHY Driver"
- depends on OF && (ARCH_QCOM || COMPILE_TEST)
- depends on NVMEM || !NVMEM
- select GENERIC_PHY
- help
- Enable this to support the HighSpeed QUSB2 PHY transceiver for USB
- controllers on Qualcomm chips. This driver supports the high-speed
- PHY which is usually paired with either the ChipIdea or Synopsys DWC3
- USB IPs on MSM SOCs.
-
-config PHY_QCOM_UFS
- tristate "Qualcomm UFS PHY driver"
- depends on OF && ARCH_QCOM
- select GENERIC_PHY
- help
- Support for UFS PHY on QCOM chipsets.
-
-config PHY_QCOM_USB_HS
- tristate "Qualcomm USB HS PHY module"
- depends on USB_ULPI_BUS
- depends on EXTCON || !EXTCON # if EXTCON=m, this cannot be built-in
- select GENERIC_PHY
- help
- Support for the USB high-speed ULPI compliant phy on Qualcomm
- chipsets.
-
-config PHY_QCOM_USB_HSIC
- tristate "Qualcomm USB HSIC ULPI PHY module"
- depends on USB_ULPI_BUS
- select GENERIC_PHY
- help
- Support for the USB HSIC ULPI compliant PHY on QCOM chipsets.
-
-config PHY_TUSB1210
- tristate "TI TUSB1210 ULPI PHY module"
- depends on USB_ULPI_BUS
- select GENERIC_PHY
- help
- Support for TI TUSB1210 USB ULPI PHY.
-
-config PHY_BRCM_SATA
- tristate "Broadcom SATA PHY driver"
- depends on ARCH_BRCMSTB || ARCH_BCM_IPROC || BMIPS_GENERIC || COMPILE_TEST
- depends on OF
- select GENERIC_PHY
- default ARCH_BCM_IPROC
- help
- Enable this to support the Broadcom SATA PHY.
- If unsure, say N.
-
-config PHY_CYGNUS_PCIE
- tristate "Broadcom Cygnus PCIe PHY driver"
- depends on OF && (ARCH_BCM_CYGNUS || COMPILE_TEST)
- select GENERIC_PHY
- default ARCH_BCM_CYGNUS
- help
- Enable this to support the Broadcom Cygnus PCIe PHY.
- If unsure, say N.
-
+source "drivers/phy/allwinner/Kconfig"
+source "drivers/phy/amlogic/Kconfig"
+source "drivers/phy/broadcom/Kconfig"
+source "drivers/phy/hisilicon/Kconfig"
+source "drivers/phy/marvell/Kconfig"
+source "drivers/phy/motorola/Kconfig"
+source "drivers/phy/qualcomm/Kconfig"
+source "drivers/phy/renesas/Kconfig"
+source "drivers/phy/rockchip/Kconfig"
+source "drivers/phy/samsung/Kconfig"
+source "drivers/phy/st/Kconfig"
source "drivers/phy/tegra/Kconfig"
-
-config PHY_NS2_PCIE
- tristate "Broadcom Northstar2 PCIe PHY driver"
- depends on OF && MDIO_BUS_MUX_BCM_IPROC
- select GENERIC_PHY
- default ARCH_BCM_IPROC
- help
- Enable this to support the Broadcom Northstar2 PCIe PHY.
- If unsure, say N.
-
-config PHY_MESON8B_USB2
- tristate "Meson8b and GXBB USB2 PHY driver"
- default ARCH_MESON
- depends on OF && (ARCH_MESON || COMPILE_TEST)
- depends on USB_SUPPORT
- select USB_COMMON
- select GENERIC_PHY
- help
- Enable this to support the Meson USB2 PHYs found in Meson8b
- and GXBB SoCs.
- If unsure, say N.
+source "drivers/phy/ti/Kconfig"
endmenu
#
obj-$(CONFIG_GENERIC_PHY) += phy-core.o
-obj-$(CONFIG_PHY_BCM_NS_USB2) += phy-bcm-ns-usb2.o
-obj-$(CONFIG_PHY_BCM_NS_USB3) += phy-bcm-ns-usb3.o
-obj-$(CONFIG_PHY_BERLIN_USB) += phy-berlin-usb.o
-obj-$(CONFIG_PHY_BERLIN_SATA) += phy-berlin-sata.o
-obj-$(CONFIG_PHY_DA8XX_USB) += phy-da8xx-usb.o
-obj-$(CONFIG_PHY_DM816X_USB) += phy-dm816x-usb.o
-obj-$(CONFIG_ARMADA375_USBCLUSTER_PHY) += phy-armada375-usb2.o
-obj-$(CONFIG_BCM_KONA_USB2_PHY) += phy-bcm-kona-usb2.o
-obj-$(CONFIG_PHY_EXYNOS_DP_VIDEO) += phy-exynos-dp-video.o
-obj-$(CONFIG_PHY_EXYNOS_MIPI_VIDEO) += phy-exynos-mipi-video.o
obj-$(CONFIG_PHY_LPC18XX_USB_OTG) += phy-lpc18xx-usb-otg.o
-obj-$(CONFIG_PHY_PXA_28NM_USB2) += phy-pxa-28nm-usb2.o
-obj-$(CONFIG_PHY_PXA_28NM_HSIC) += phy-pxa-28nm-hsic.o
-obj-$(CONFIG_PHY_MVEBU_SATA) += phy-mvebu-sata.o
-obj-$(CONFIG_PHY_MIPHY28LP) += phy-miphy28lp.o
-obj-$(CONFIG_PHY_RCAR_GEN2) += phy-rcar-gen2.o
-obj-$(CONFIG_PHY_RCAR_GEN3_USB2) += phy-rcar-gen3-usb2.o
-obj-$(CONFIG_OMAP_CONTROL_PHY) += phy-omap-control.o
-obj-$(CONFIG_OMAP_USB2) += phy-omap-usb2.o
-obj-$(CONFIG_TI_PIPE3) += phy-ti-pipe3.o
-obj-$(CONFIG_TWL4030_USB) += phy-twl4030-usb.o
-obj-$(CONFIG_PHY_EXYNOS5250_SATA) += phy-exynos5250-sata.o
-obj-$(CONFIG_PHY_HIX5HD2_SATA) += phy-hix5hd2-sata.o
-obj-$(CONFIG_PHY_HI6220_USB) += phy-hi6220-usb.o
obj-$(CONFIG_PHY_MT65XX_USB3) += phy-mt65xx-usb3.o
-obj-$(CONFIG_PHY_SUN4I_USB) += phy-sun4i-usb.o
-obj-$(CONFIG_PHY_SUN9I_USB) += phy-sun9i-usb.o
-obj-$(CONFIG_PHY_SAMSUNG_USB2) += phy-exynos-usb2.o
-phy-exynos-usb2-y += phy-samsung-usb2.o
-phy-exynos-usb2-$(CONFIG_PHY_EXYNOS4210_USB2) += phy-exynos4210-usb2.o
-phy-exynos-usb2-$(CONFIG_PHY_EXYNOS4X12_USB2) += phy-exynos4x12-usb2.o
-phy-exynos-usb2-$(CONFIG_PHY_EXYNOS5250_USB2) += phy-exynos5250-usb2.o
-phy-exynos-usb2-$(CONFIG_PHY_S5PV210_USB2) += phy-s5pv210-usb2.o
-obj-$(CONFIG_PHY_EXYNOS5_USBDRD) += phy-exynos5-usbdrd.o
-obj-$(CONFIG_PHY_EXYNOS_PCIE) += phy-exynos-pcie.o
-obj-$(CONFIG_PHY_QCOM_APQ8064_SATA) += phy-qcom-apq8064-sata.o
-obj-$(CONFIG_PHY_ROCKCHIP_USB) += phy-rockchip-usb.o
-obj-$(CONFIG_PHY_ROCKCHIP_INNO_USB2) += phy-rockchip-inno-usb2.o
-obj-$(CONFIG_PHY_ROCKCHIP_EMMC) += phy-rockchip-emmc.o
-obj-$(CONFIG_PHY_ROCKCHIP_PCIE) += phy-rockchip-pcie.o
-obj-$(CONFIG_PHY_ROCKCHIP_DP) += phy-rockchip-dp.o
-obj-$(CONFIG_PHY_ROCKCHIP_TYPEC) += phy-rockchip-typec.o
-obj-$(CONFIG_PHY_QCOM_IPQ806X_SATA) += phy-qcom-ipq806x-sata.o
-obj-$(CONFIG_PHY_ST_SPEAR1310_MIPHY) += phy-spear1310-miphy.o
-obj-$(CONFIG_PHY_ST_SPEAR1340_MIPHY) += phy-spear1340-miphy.o
obj-$(CONFIG_PHY_XGENE) += phy-xgene.o
-obj-$(CONFIG_PHY_STIH407_USB) += phy-stih407-usb.o
-obj-$(CONFIG_PHY_QCOM_QMP) += phy-qcom-qmp.o
-obj-$(CONFIG_PHY_QCOM_QUSB2) += phy-qcom-qusb2.o
-obj-$(CONFIG_PHY_QCOM_UFS) += phy-qcom-ufs.o
-obj-$(CONFIG_PHY_QCOM_UFS) += phy-qcom-ufs-qmp-20nm.o
-obj-$(CONFIG_PHY_QCOM_UFS) += phy-qcom-ufs-qmp-14nm.o
-obj-$(CONFIG_PHY_QCOM_USB_HS) += phy-qcom-usb-hs.o
-obj-$(CONFIG_PHY_QCOM_USB_HSIC) += phy-qcom-usb-hsic.o
-obj-$(CONFIG_PHY_TUSB1210) += phy-tusb1210.o
-obj-$(CONFIG_PHY_BRCM_SATA) += phy-brcm-sata.o
obj-$(CONFIG_PHY_PISTACHIO_USB) += phy-pistachio-usb.o
-obj-$(CONFIG_PHY_CYGNUS_PCIE) += phy-bcm-cygnus-pcie.o
-obj-$(CONFIG_ARCH_TEGRA) += tegra/
-obj-$(CONFIG_PHY_NS2_PCIE) += phy-bcm-ns2-pcie.o
-obj-$(CONFIG_PHY_MESON8B_USB2) += phy-meson8b-usb2.o
+
+obj-$(CONFIG_ARCH_SUNXI) += allwinner/
+obj-$(CONFIG_ARCH_MESON) += amlogic/
+obj-$(CONFIG_ARCH_RENESAS) += renesas/
+obj-$(CONFIG_ARCH_ROCKCHIP) += rockchip/
+obj-$(CONFIG_ARCH_TEGRA) += tegra/
+obj-y += broadcom/ \
+ hisilicon/ \
+ marvell/ \
+ motorola/ \
+ qualcomm/ \
+ samsung/ \
+ st/ \
+ ti/
--- /dev/null
+#
+# Phy drivers for Allwinner platforms
+#
+config PHY_SUN4I_USB
+ tristate "Allwinner sunxi SoC USB PHY driver"
+ depends on ARCH_SUNXI && HAS_IOMEM && OF
+ depends on RESET_CONTROLLER
+ depends on EXTCON
+ depends on POWER_SUPPLY
+ depends on USB_SUPPORT
+ select GENERIC_PHY
+ select USB_COMMON
+ help
+ Enable this to support the transceiver that is part of Allwinner
+ sunxi SoCs.
+
+ This driver controls the entire USB PHY block, both the USB OTG
+ parts, as well as the 2 regular USB 2 host PHYs.
+
+config PHY_SUN9I_USB
+ tristate "Allwinner sun9i SoC USB PHY driver"
+ depends on ARCH_SUNXI && HAS_IOMEM && OF
+ depends on RESET_CONTROLLER
+ depends on USB_SUPPORT
+ select USB_COMMON
+ select GENERIC_PHY
+ help
+ Enable this to support the transceiver that is part of Allwinner
+ sun9i SoCs.
+
+ This driver controls each individual USB 2 host PHY.
--- /dev/null
+obj-$(CONFIG_PHY_SUN4I_USB) += phy-sun4i-usb.o
+obj-$(CONFIG_PHY_SUN9I_USB) += phy-sun9i-usb.o
--- /dev/null
+#
+# Phy drivers for Amlogic platforms
+#
+config PHY_MESON8B_USB2
+ tristate "Meson8, Meson8b and GXBB USB2 PHY driver"
+ default ARCH_MESON
+ depends on OF && (ARCH_MESON || COMPILE_TEST)
+ depends on USB_SUPPORT
+ select USB_COMMON
+ select GENERIC_PHY
+ help
+ Enable this to support the Meson USB2 PHYs found in Meson8,
+ Meson8b and GXBB SoCs.
+ If unsure, say N.
+
+config PHY_MESON_GXL_USB2
+ tristate "Meson GXL and GXM USB2 PHY drivers"
+ default ARCH_MESON
+ depends on OF && (ARCH_MESON || COMPILE_TEST)
+ depends on USB_SUPPORT
+ select USB_COMMON
+ select GENERIC_PHY
+ select REGMAP_MMIO
+ help
+ Enable this to support the Meson USB2 PHYs found in Meson
+ GXL and GXM SoCs.
+ If unsure, say N.
--- /dev/null
+obj-$(CONFIG_PHY_MESON8B_USB2) += phy-meson8b-usb2.o
+obj-$(CONFIG_PHY_MESON_GXL_USB2) += phy-meson-gxl-usb2.o
--- /dev/null
+/*
+ * Meson GXL and GXM USB2 PHY driver
+ *
+ * Copyright (C) 2017 Martin Blumenstingl <martin.blumenstingl@googlemail.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/delay.h>
+#include <linux/io.h>
+#include <linux/module.h>
+#include <linux/of_device.h>
+#include <linux/regmap.h>
+#include <linux/phy/phy.h>
+#include <linux/platform_device.h>
+#include <linux/usb/of.h>
+
+/* bits [31:27] are read-only */
+#define U2P_R0 0x0
+ #define U2P_R0_BYPASS_SEL BIT(0)
+ #define U2P_R0_BYPASS_DM_EN BIT(1)
+ #define U2P_R0_BYPASS_DP_EN BIT(2)
+ #define U2P_R0_TXBITSTUFF_ENH BIT(3)
+ #define U2P_R0_TXBITSTUFF_EN BIT(4)
+ #define U2P_R0_DM_PULLDOWN BIT(5)
+ #define U2P_R0_DP_PULLDOWN BIT(6)
+ #define U2P_R0_DP_VBUS_VLD_EXT_SEL BIT(7)
+ #define U2P_R0_DP_VBUS_VLD_EXT BIT(8)
+ #define U2P_R0_ADP_PRB_EN BIT(9)
+ #define U2P_R0_ADP_DISCHARGE BIT(10)
+ #define U2P_R0_ADP_CHARGE BIT(11)
+ #define U2P_R0_DRV_VBUS BIT(12)
+ #define U2P_R0_ID_PULLUP BIT(13)
+ #define U2P_R0_LOOPBACK_EN_B BIT(14)
+ #define U2P_R0_OTG_DISABLE BIT(15)
+ #define U2P_R0_COMMON_ONN BIT(16)
+ #define U2P_R0_FSEL_MASK GENMASK(19, 17)
+ #define U2P_R0_REF_CLK_SEL_MASK GENMASK(21, 20)
+ #define U2P_R0_POWER_ON_RESET BIT(22)
+ #define U2P_R0_V_ATE_TEST_EN_B_MASK GENMASK(24, 23)
+ #define U2P_R0_ID_SET_ID_DQ BIT(25)
+ #define U2P_R0_ATE_RESET BIT(26)
+ #define U2P_R0_FSV_MINUS BIT(27)
+ #define U2P_R0_FSV_PLUS BIT(28)
+ #define U2P_R0_BYPASS_DM_DATA BIT(29)
+ #define U2P_R0_BYPASS_DP_DATA BIT(30)
+
+#define U2P_R1 0x4
+ #define U2P_R1_BURN_IN_TEST BIT(0)
+ #define U2P_R1_ACA_ENABLE BIT(1)
+ #define U2P_R1_DCD_ENABLE BIT(2)
+ #define U2P_R1_VDAT_SRC_EN_B BIT(3)
+ #define U2P_R1_VDAT_DET_EN_B BIT(4)
+ #define U2P_R1_CHARGES_SEL BIT(5)
+ #define U2P_R1_TX_PREEMP_PULSE_TUNE BIT(6)
+ #define U2P_R1_TX_PREEMP_AMP_TUNE_MASK GENMASK(8, 7)
+ #define U2P_R1_TX_RES_TUNE_MASK GENMASK(10, 9)
+ #define U2P_R1_TX_RISE_TUNE_MASK GENMASK(12, 11)
+ #define U2P_R1_TX_VREF_TUNE_MASK GENMASK(16, 13)
+ #define U2P_R1_TX_FSLS_TUNE_MASK GENMASK(20, 17)
+ #define U2P_R1_TX_HSXV_TUNE_MASK GENMASK(22, 21)
+ #define U2P_R1_OTG_TUNE_MASK GENMASK(25, 23)
+ #define U2P_R1_SQRX_TUNE_MASK GENMASK(28, 26)
+ #define U2P_R1_COMP_DIS_TUNE_MASK GENMASK(31, 29)
+
+/* bits [31:14] are read-only */
+#define U2P_R2 0x8
+ #define U2P_R2_DATA_IN_MASK GENMASK(3, 0)
+ #define U2P_R2_DATA_IN_EN_MASK GENMASK(7, 4)
+ #define U2P_R2_ADDR_MASK GENMASK(11, 8)
+ #define U2P_R2_DATA_OUT_SEL BIT(12)
+ #define U2P_R2_CLK BIT(13)
+ #define U2P_R2_DATA_OUT_MASK GENMASK(17, 14)
+ #define U2P_R2_ACA_PIN_RANGE_C BIT(18)
+ #define U2P_R2_ACA_PIN_RANGE_B BIT(19)
+ #define U2P_R2_ACA_PIN_RANGE_A BIT(20)
+ #define U2P_R2_ACA_PIN_GND BIT(21)
+ #define U2P_R2_ACA_PIN_FLOAT BIT(22)
+ #define U2P_R2_CHARGE_DETECT BIT(23)
+ #define U2P_R2_DEVICE_SESSION_VALID BIT(24)
+ #define U2P_R2_ADP_PROBE BIT(25)
+ #define U2P_R2_ADP_SENSE BIT(26)
+ #define U2P_R2_SESSION_END BIT(27)
+ #define U2P_R2_VBUS_VALID BIT(28)
+ #define U2P_R2_B_VALID BIT(29)
+ #define U2P_R2_A_VALID BIT(30)
+ #define U2P_R2_ID_DIG BIT(31)
+
+#define U2P_R3 0xc
+
+#define RESET_COMPLETE_TIME 500
+
+struct phy_meson_gxl_usb2_priv {
+ struct regmap *regmap;
+ enum phy_mode mode;
+ int is_enabled;
+};
+
+static const struct regmap_config phy_meson_gxl_usb2_regmap_conf = {
+ .reg_bits = 8,
+ .val_bits = 32,
+ .reg_stride = 4,
+ .max_register = U2P_R3,
+};
+
+static int phy_meson_gxl_usb2_reset(struct phy *phy)
+{
+ struct phy_meson_gxl_usb2_priv *priv = phy_get_drvdata(phy);
+
+ if (priv->is_enabled) {
+ /* reset the PHY and wait until settings are stabilized */
+ regmap_update_bits(priv->regmap, U2P_R0, U2P_R0_POWER_ON_RESET,
+ U2P_R0_POWER_ON_RESET);
+ udelay(RESET_COMPLETE_TIME);
+ regmap_update_bits(priv->regmap, U2P_R0, U2P_R0_POWER_ON_RESET,
+ 0);
+ udelay(RESET_COMPLETE_TIME);
+ }
+
+ return 0;
+}
+
+static int phy_meson_gxl_usb2_set_mode(struct phy *phy, enum phy_mode mode)
+{
+ struct phy_meson_gxl_usb2_priv *priv = phy_get_drvdata(phy);
+
+ switch (mode) {
+ case PHY_MODE_USB_HOST:
+ case PHY_MODE_USB_OTG:
+ regmap_update_bits(priv->regmap, U2P_R0, U2P_R0_DM_PULLDOWN,
+ U2P_R0_DM_PULLDOWN);
+ regmap_update_bits(priv->regmap, U2P_R0, U2P_R0_DP_PULLDOWN,
+ U2P_R0_DP_PULLDOWN);
+ regmap_update_bits(priv->regmap, U2P_R0, U2P_R0_ID_PULLUP, 0);
+ break;
+
+ case PHY_MODE_USB_DEVICE:
+ regmap_update_bits(priv->regmap, U2P_R0, U2P_R0_DM_PULLDOWN,
+ 0);
+ regmap_update_bits(priv->regmap, U2P_R0, U2P_R0_DP_PULLDOWN,
+ 0);
+ regmap_update_bits(priv->regmap, U2P_R0, U2P_R0_ID_PULLUP,
+ U2P_R0_ID_PULLUP);
+ break;
+
+ default:
+ return -EINVAL;
+ }
+
+ phy_meson_gxl_usb2_reset(phy);
+
+ priv->mode = mode;
+
+ return 0;
+}
+
+static int phy_meson_gxl_usb2_power_off(struct phy *phy)
+{
+ struct phy_meson_gxl_usb2_priv *priv = phy_get_drvdata(phy);
+
+ priv->is_enabled = 0;
+
+ /* power off the PHY by putting it into reset mode */
+ regmap_update_bits(priv->regmap, U2P_R0, U2P_R0_POWER_ON_RESET,
+ U2P_R0_POWER_ON_RESET);
+
+ return 0;
+}
+
+static int phy_meson_gxl_usb2_power_on(struct phy *phy)
+{
+ struct phy_meson_gxl_usb2_priv *priv = phy_get_drvdata(phy);
+ int ret;
+
+ priv->is_enabled = 1;
+
+ /* power on the PHY by taking it out of reset mode */
+ regmap_update_bits(priv->regmap, U2P_R0, U2P_R0_POWER_ON_RESET, 0);
+
+ ret = phy_meson_gxl_usb2_set_mode(phy, priv->mode);
+ if (ret) {
+ phy_meson_gxl_usb2_power_off(phy);
+
+ dev_err(&phy->dev, "Failed to initialize PHY with mode %d\n",
+ priv->mode);
+ return ret;
+ }
+
+ return 0;
+}
+
+static const struct phy_ops phy_meson_gxl_usb2_ops = {
+ .power_on = phy_meson_gxl_usb2_power_on,
+ .power_off = phy_meson_gxl_usb2_power_off,
+ .set_mode = phy_meson_gxl_usb2_set_mode,
+ .reset = phy_meson_gxl_usb2_reset,
+ .owner = THIS_MODULE,
+};
+
+static int phy_meson_gxl_usb2_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct phy_provider *phy_provider;
+ struct resource *res;
+ struct phy_meson_gxl_usb2_priv *priv;
+ struct phy *phy;
+ void __iomem *base;
+
+ priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, priv);
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ base = devm_ioremap_resource(dev, res);
+ if (IS_ERR(base))
+ return PTR_ERR(base);
+
+ switch (of_usb_get_dr_mode_by_phy(dev->of_node, -1)) {
+ case USB_DR_MODE_PERIPHERAL:
+ priv->mode = PHY_MODE_USB_DEVICE;
+ break;
+ case USB_DR_MODE_OTG:
+ priv->mode = PHY_MODE_USB_OTG;
+ break;
+ case USB_DR_MODE_HOST:
+ default:
+ priv->mode = PHY_MODE_USB_HOST;
+ break;
+ }
+
+ priv->regmap = devm_regmap_init_mmio(dev, base,
+ &phy_meson_gxl_usb2_regmap_conf);
+ if (IS_ERR(priv->regmap))
+ return PTR_ERR(priv->regmap);
+
+ phy = devm_phy_create(dev, NULL, &phy_meson_gxl_usb2_ops);
+ if (IS_ERR(phy)) {
+ dev_err(dev, "failed to create PHY\n");
+ return PTR_ERR(phy);
+ }
+
+ phy_set_drvdata(phy, priv);
+
+ phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
+
+ return PTR_ERR_OR_ZERO(phy_provider);
+}
+
+static const struct of_device_id phy_meson_gxl_usb2_of_match[] = {
+ { .compatible = "amlogic,meson-gxl-usb2-phy", },
+ { },
+};
+MODULE_DEVICE_TABLE(of, phy_meson_gxl_usb2_of_match);
+
+static struct platform_driver phy_meson_gxl_usb2_driver = {
+ .probe = phy_meson_gxl_usb2_probe,
+ .driver = {
+ .name = "phy-meson-gxl-usb2",
+ .of_match_table = phy_meson_gxl_usb2_of_match,
+ },
+};
+module_platform_driver(phy_meson_gxl_usb2_driver);
+
+MODULE_AUTHOR("Martin Blumenstingl <martin.blumenstingl@googlemail.com>");
+MODULE_DESCRIPTION("Meson GXL and GXM USB2 PHY driver");
+MODULE_LICENSE("GPL v2");
/*
- * Meson8b and GXBB USB2 PHY driver
+ * Meson8, Meson8b and GXBB USB2 PHY driver
*
* Copyright (C) 2016 Martin Blumenstingl <martin.blumenstingl@googlemail.com>
*
}
static const struct of_device_id phy_meson8b_usb2_of_match[] = {
+ { .compatible = "amlogic,meson8-usb2-phy", },
{ .compatible = "amlogic,meson8b-usb2-phy", },
{ .compatible = "amlogic,meson-gxbb-usb2-phy", },
{ },
module_platform_driver(phy_meson8b_usb2_driver);
MODULE_AUTHOR("Martin Blumenstingl <martin.blumenstingl@googlemail.com>");
-MODULE_DESCRIPTION("Meson8b and GXBB USB2 PHY driver");
+MODULE_DESCRIPTION("Meson8, Meson8b and GXBB USB2 PHY driver");
MODULE_LICENSE("GPL");
--- /dev/null
+#
+# Phy drivers for Broadcom platforms
+#
+config PHY_CYGNUS_PCIE
+ tristate "Broadcom Cygnus PCIe PHY driver"
+ depends on OF && (ARCH_BCM_CYGNUS || COMPILE_TEST)
+ select GENERIC_PHY
+ default ARCH_BCM_CYGNUS
+ help
+ Enable this to support the Broadcom Cygnus PCIe PHY.
+ If unsure, say N.
+
+config BCM_KONA_USB2_PHY
+ tristate "Broadcom Kona USB2 PHY Driver"
+ depends on HAS_IOMEM
+ select GENERIC_PHY
+ help
+ Enable this to support the Broadcom Kona USB 2.0 PHY.
+
+config PHY_BCM_NS_USB2
+ tristate "Broadcom Northstar USB 2.0 PHY Driver"
+ depends on ARCH_BCM_IPROC || COMPILE_TEST
+ depends on HAS_IOMEM && OF
+ select GENERIC_PHY
+ help
+ Enable this to support Broadcom USB 2.0 PHY connected to the USB
+ controller on Northstar family.
+
+config PHY_BCM_NS_USB3
+ tristate "Broadcom Northstar USB 3.0 PHY Driver"
+ depends on ARCH_BCM_IPROC || COMPILE_TEST
+ depends on HAS_IOMEM && OF
+ select GENERIC_PHY
+ select MDIO_DEVICE
+ help
+ Enable this to support Broadcom USB 3.0 PHY connected to the USB
+ controller on Northstar family.
+
+config PHY_NS2_PCIE
+ tristate "Broadcom Northstar2 PCIe PHY driver"
+ depends on OF && MDIO_BUS_MUX_BCM_IPROC
+ select GENERIC_PHY
+ default ARCH_BCM_IPROC
+ help
+ Enable this to support the Broadcom Northstar2 PCIe PHY.
+ If unsure, say N.
+
+config PHY_NS2_USB_DRD
+ tristate "Broadcom Northstar2 USB DRD PHY support"
+ depends on OF && (ARCH_BCM_IPROC || COMPILE_TEST)
+ select GENERIC_PHY
+ select EXTCON
+ default ARCH_BCM_IPROC
+ help
+ Enable this to support the Broadcom Northstar2 USB DRD PHY.
+ This driver initializes the PHY in either HOST or DEVICE mode.
+ The host or device configuration is read from device tree.
+
+ If unsure, say N.
+
+config PHY_BRCM_SATA
+ tristate "Broadcom SATA PHY driver"
+ depends on ARCH_BRCMSTB || ARCH_BCM_IPROC || BMIPS_GENERIC || COMPILE_TEST
+ depends on OF
+ select GENERIC_PHY
+ default ARCH_BCM_IPROC
+ help
+ Enable this to support the Broadcom SATA PHY.
+ If unsure, say N.
--- /dev/null
+obj-$(CONFIG_PHY_CYGNUS_PCIE) += phy-bcm-cygnus-pcie.o
+obj-$(CONFIG_BCM_KONA_USB2_PHY) += phy-bcm-kona-usb2.o
+obj-$(CONFIG_PHY_BCM_NS_USB2) += phy-bcm-ns-usb2.o
+obj-$(CONFIG_PHY_BCM_NS_USB3) += phy-bcm-ns-usb3.o
+obj-$(CONFIG_PHY_NS2_PCIE) += phy-bcm-ns2-pcie.o
+obj-$(CONFIG_PHY_NS2_USB_DRD) += phy-bcm-ns2-usbdrd.o
+obj-$(CONFIG_PHY_BRCM_SATA) += phy-brcm-sata.o
#include <linux/bcma/bcma.h>
#include <linux/delay.h>
#include <linux/err.h>
+#include <linux/mdio.h>
#include <linux/module.h>
+#include <linux/of_address.h>
#include <linux/of_platform.h>
#include <linux/platform_device.h>
#include <linux/phy/phy.h>
enum bcm_ns_family family;
void __iomem *dmp;
void __iomem *ccb_mii;
+ struct mdio_device *mdiodev;
struct phy *phy;
+
+ int (*phy_write)(struct bcm_ns_usb3 *usb3, u16 reg, u16 value);
};
static const struct of_device_id bcm_ns_usb3_id_table[] = {
};
MODULE_DEVICE_TABLE(of, bcm_ns_usb3_id_table);
-static int bcm_ns_usb3_wait_reg(struct bcm_ns_usb3 *usb3, void __iomem *addr,
- u32 mask, u32 value, unsigned long timeout)
-{
- unsigned long deadline = jiffies + timeout;
- u32 val;
-
- do {
- val = readl(addr);
- if ((val & mask) == value)
- return 0;
- cpu_relax();
- udelay(10);
- } while (!time_after_eq(jiffies, deadline));
-
- dev_err(usb3->dev, "Timeout waiting for register %p\n", addr);
-
- return -EBUSY;
-}
-
-static inline int bcm_ns_usb3_mii_mng_wait_idle(struct bcm_ns_usb3 *usb3)
-{
- return bcm_ns_usb3_wait_reg(usb3, usb3->ccb_mii + BCMA_CCB_MII_MNG_CTL,
- 0x0100, 0x0000,
- usecs_to_jiffies(BCM_NS_USB3_MII_MNG_TIMEOUT_US));
-}
-
static int bcm_ns_usb3_mdio_phy_write(struct bcm_ns_usb3 *usb3, u16 reg,
u16 value)
{
- u32 tmp = 0;
- int err;
-
- err = bcm_ns_usb3_mii_mng_wait_idle(usb3);
- if (err < 0) {
- dev_err(usb3->dev, "Couldn't write 0x%08x value\n", value);
- return err;
- }
-
- /* TODO: Use a proper MDIO bus layer */
- tmp |= 0x58020000; /* Magic value for MDIO PHY write */
- tmp |= reg << 18;
- tmp |= value;
- writel(tmp, usb3->ccb_mii + BCMA_CCB_MII_MNG_CMD_DATA);
-
- return 0;
+ return usb3->phy_write(usb3, reg, value);
}
static int bcm_ns_usb3_phy_init_ns_bx(struct bcm_ns_usb3 *usb3)
{
int err;
- /* Enable MDIO. Setting MDCDIV as 26 */
- writel(0x0000009a, usb3->ccb_mii + BCMA_CCB_MII_MNG_CTL);
-
- /* Wait for MDIO? */
- udelay(2);
-
/* USB3 PLL Block */
err = bcm_ns_usb3_mdio_phy_write(usb3, BCM_NS_USB3_PHY_BASE_ADDR_REG,
BCM_NS_USB3_PHY_PLL30_BLOCK);
/* Deaaserting PLL Reset */
bcm_ns_usb3_mdio_phy_write(usb3, BCM_NS_USB3_PLLA_CONTROL1, 0x8000);
- /* Waiting MII Mgt interface idle */
- bcm_ns_usb3_mii_mng_wait_idle(usb3);
-
/* Deasserting USB3 system reset */
writel(0, usb3->dmp + BCMA_RESET_CTL);
/* Enabling SSC */
bcm_ns_usb3_mdio_phy_write(usb3, BCM_NS_USB3_TX_PMD_CONTROL1, 0x1003);
- /* Waiting MII Mgt interface idle */
- bcm_ns_usb3_mii_mng_wait_idle(usb3);
-
return 0;
}
{
int err;
- /* Enable MDIO. Setting MDCDIV as 26 */
- writel(0x0000009a, usb3->ccb_mii + BCMA_CCB_MII_MNG_CTL);
-
- /* Wait for MDIO? */
- udelay(2);
-
/* PLL30 block */
err = bcm_ns_usb3_mdio_phy_write(usb3, BCM_NS_USB3_PHY_BASE_ADDR_REG,
BCM_NS_USB3_PHY_PLL30_BLOCK);
bcm_ns_usb3_mdio_phy_write(usb3, BCM_NS_USB3_TX_PMD_CONTROL1, 0x1003);
- /* Waiting MII Mgt interface idle */
- bcm_ns_usb3_mii_mng_wait_idle(usb3);
-
/* Deasserting USB3 system reset */
writel(0, usb3->dmp + BCMA_RESET_CTL);
.owner = THIS_MODULE,
};
+/**************************************************
+ * MDIO driver code
+ **************************************************/
+
+static int bcm_ns_usb3_mdiodev_phy_write(struct bcm_ns_usb3 *usb3, u16 reg,
+ u16 value)
+{
+ struct mdio_device *mdiodev = usb3->mdiodev;
+
+ return mdiobus_write(mdiodev->bus, mdiodev->addr, reg, value);
+}
+
+static int bcm_ns_usb3_mdio_probe(struct mdio_device *mdiodev)
+{
+ struct device *dev = &mdiodev->dev;
+ const struct of_device_id *of_id;
+ struct phy_provider *phy_provider;
+ struct device_node *syscon_np;
+ struct bcm_ns_usb3 *usb3;
+ struct resource res;
+ int err;
+
+ usb3 = devm_kzalloc(dev, sizeof(*usb3), GFP_KERNEL);
+ if (!usb3)
+ return -ENOMEM;
+
+ usb3->dev = dev;
+ usb3->mdiodev = mdiodev;
+
+ of_id = of_match_device(bcm_ns_usb3_id_table, dev);
+ if (!of_id)
+ return -EINVAL;
+ usb3->family = (enum bcm_ns_family)of_id->data;
+
+ syscon_np = of_parse_phandle(dev->of_node, "usb3-dmp-syscon", 0);
+ err = of_address_to_resource(syscon_np, 0, &res);
+ of_node_put(syscon_np);
+ if (err)
+ return err;
+
+ usb3->dmp = devm_ioremap_resource(dev, &res);
+ if (IS_ERR(usb3->dmp)) {
+ dev_err(dev, "Failed to map DMP regs\n");
+ return PTR_ERR(usb3->dmp);
+ }
+
+ usb3->phy_write = bcm_ns_usb3_mdiodev_phy_write;
+
+ usb3->phy = devm_phy_create(dev, NULL, &ops);
+ if (IS_ERR(usb3->phy)) {
+ dev_err(dev, "Failed to create PHY\n");
+ return PTR_ERR(usb3->phy);
+ }
+
+ phy_set_drvdata(usb3->phy, usb3);
+
+ phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
+
+ return PTR_ERR_OR_ZERO(phy_provider);
+}
+
+static struct mdio_driver bcm_ns_usb3_mdio_driver = {
+ .mdiodrv = {
+ .driver = {
+ .name = "bcm_ns_mdio_usb3",
+ .of_match_table = bcm_ns_usb3_id_table,
+ },
+ },
+ .probe = bcm_ns_usb3_mdio_probe,
+};
+
+/**************************************************
+ * Platform driver code
+ **************************************************/
+
+static int bcm_ns_usb3_wait_reg(struct bcm_ns_usb3 *usb3, void __iomem *addr,
+ u32 mask, u32 value, unsigned long timeout)
+{
+ unsigned long deadline = jiffies + timeout;
+ u32 val;
+
+ do {
+ val = readl(addr);
+ if ((val & mask) == value)
+ return 0;
+ cpu_relax();
+ udelay(10);
+ } while (!time_after_eq(jiffies, deadline));
+
+ dev_err(usb3->dev, "Timeout waiting for register %p\n", addr);
+
+ return -EBUSY;
+}
+
+static inline int bcm_ns_usb3_mii_mng_wait_idle(struct bcm_ns_usb3 *usb3)
+{
+ return bcm_ns_usb3_wait_reg(usb3, usb3->ccb_mii + BCMA_CCB_MII_MNG_CTL,
+ 0x0100, 0x0000,
+ usecs_to_jiffies(BCM_NS_USB3_MII_MNG_TIMEOUT_US));
+}
+
+static int bcm_ns_usb3_platform_phy_write(struct bcm_ns_usb3 *usb3, u16 reg,
+ u16 value)
+{
+ u32 tmp = 0;
+ int err;
+
+ err = bcm_ns_usb3_mii_mng_wait_idle(usb3);
+ if (err < 0) {
+ dev_err(usb3->dev, "Couldn't write 0x%08x value\n", value);
+ return err;
+ }
+
+ /* TODO: Use a proper MDIO bus layer */
+ tmp |= 0x58020000; /* Magic value for MDIO PHY write */
+ tmp |= reg << 18;
+ tmp |= value;
+ writel(tmp, usb3->ccb_mii + BCMA_CCB_MII_MNG_CMD_DATA);
+
+ return bcm_ns_usb3_mii_mng_wait_idle(usb3);
+}
+
static int bcm_ns_usb3_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
return PTR_ERR(usb3->ccb_mii);
}
+ /* Enable MDIO. Setting MDCDIV as 26 */
+ writel(0x0000009a, usb3->ccb_mii + BCMA_CCB_MII_MNG_CTL);
+
+ /* Wait for MDIO? */
+ udelay(2);
+
+ usb3->phy_write = bcm_ns_usb3_platform_phy_write;
+
usb3->phy = devm_phy_create(dev, NULL, &ops);
if (IS_ERR(usb3->phy)) {
dev_err(dev, "Failed to create PHY\n");
.of_match_table = bcm_ns_usb3_id_table,
},
};
-module_platform_driver(bcm_ns_usb3_driver);
+
+static int __init bcm_ns_usb3_module_init(void)
+{
+ int err;
+
+ /*
+ * For backward compatibility we register as MDIO and platform driver.
+ * After getting MDIO binding commonly used (e.g. switching all DT files
+ * to use it) we should deprecate the old binding and eventually drop
+ * support for it.
+ */
+
+ err = mdio_driver_register(&bcm_ns_usb3_mdio_driver);
+ if (err)
+ return err;
+
+ err = platform_driver_register(&bcm_ns_usb3_driver);
+ if (err)
+ mdio_driver_unregister(&bcm_ns_usb3_mdio_driver);
+
+ return err;
+}
+module_init(bcm_ns_usb3_module_init);
+
+static void __exit bcm_ns_usb3_module_exit(void)
+{
+ platform_driver_unregister(&bcm_ns_usb3_driver);
+ mdio_driver_unregister(&bcm_ns_usb3_mdio_driver);
+}
+module_exit(bcm_ns_usb3_module_exit)
MODULE_LICENSE("GPL v2");
--- /dev/null
+/*
+ * Copyright (C) 2017 Broadcom
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation version 2.
+ *
+ * This program is distributed "as is" WITHOUT ANY WARRANTY of any
+ * kind, whether express or implied; without even the implied warranty
+ * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/delay.h>
+#include <linux/extcon.h>
+#include <linux/gpio.h>
+#include <linux/gpio/consumer.h>
+#include <linux/init.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/irq.h>
+#include <linux/mfd/syscon.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_address.h>
+#include <linux/phy/phy.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+#include <linux/slab.h>
+#include <linux/workqueue.h>
+
+#define ICFG_DRD_AFE 0x0
+#define ICFG_MISC_STAT 0x18
+#define ICFG_DRD_P0CTL 0x1C
+#define ICFG_STRAP_CTRL 0x20
+#define ICFG_FSM_CTRL 0x24
+
+#define ICFG_DEV_BIT BIT(2)
+#define IDM_RST_BIT BIT(0)
+#define AFE_CORERDY_VDDC BIT(18)
+#define PHY_PLL_RESETB BIT(15)
+#define PHY_RESETB BIT(14)
+#define PHY_PLL_LOCK BIT(0)
+
+#define DRD_DEV_MODE BIT(20)
+#define OHCI_OVRCUR_POL BIT(11)
+#define ICFG_OFF_MODE BIT(6)
+#define PLL_LOCK_RETRY 1000
+
+#define EVT_DEVICE 0
+#define EVT_HOST 1
+
+#define DRD_HOST_MODE (BIT(2) | BIT(3))
+#define DRD_DEVICE_MODE (BIT(4) | BIT(5))
+#define DRD_HOST_VAL 0x803
+#define DRD_DEV_VAL 0x807
+#define GPIO_DELAY 20
+
+struct ns2_phy_data;
+struct ns2_phy_driver {
+ void __iomem *icfgdrd_regs;
+ void __iomem *idmdrd_rst_ctrl;
+ void __iomem *crmu_usb2_ctrl;
+ void __iomem *usb2h_strap_reg;
+ struct ns2_phy_data *data;
+ struct extcon_dev *edev;
+ struct gpio_desc *vbus_gpiod;
+ struct gpio_desc *id_gpiod;
+ int id_irq;
+ int vbus_irq;
+ unsigned long debounce_jiffies;
+ struct delayed_work wq_extcon;
+};
+
+struct ns2_phy_data {
+ struct ns2_phy_driver *driver;
+ struct phy *phy;
+ int new_state;
+};
+
+static const unsigned int usb_extcon_cable[] = {
+ EXTCON_USB,
+ EXTCON_USB_HOST,
+ EXTCON_NONE,
+};
+
+static inline int pll_lock_stat(u32 usb_reg, int reg_mask,
+ struct ns2_phy_driver *driver)
+{
+ int retry = PLL_LOCK_RETRY;
+ u32 val;
+
+ do {
+ udelay(1);
+ val = readl(driver->icfgdrd_regs + usb_reg);
+ if (val & reg_mask)
+ return 0;
+ } while (--retry > 0);
+
+ return -EBUSY;
+}
+
+static int ns2_drd_phy_init(struct phy *phy)
+{
+ struct ns2_phy_data *data = phy_get_drvdata(phy);
+ struct ns2_phy_driver *driver = data->driver;
+ u32 val;
+
+ val = readl(driver->icfgdrd_regs + ICFG_FSM_CTRL);
+
+ if (data->new_state == EVT_HOST) {
+ val &= ~DRD_DEVICE_MODE;
+ val |= DRD_HOST_MODE;
+ } else {
+ val &= ~DRD_HOST_MODE;
+ val |= DRD_DEVICE_MODE;
+ }
+ writel(val, driver->icfgdrd_regs + ICFG_FSM_CTRL);
+
+ return 0;
+}
+
+static int ns2_drd_phy_poweroff(struct phy *phy)
+{
+ struct ns2_phy_data *data = phy_get_drvdata(phy);
+ struct ns2_phy_driver *driver = data->driver;
+ u32 val;
+
+ val = readl(driver->crmu_usb2_ctrl);
+ val &= ~AFE_CORERDY_VDDC;
+ writel(val, driver->crmu_usb2_ctrl);
+
+ val = readl(driver->crmu_usb2_ctrl);
+ val &= ~DRD_DEV_MODE;
+ writel(val, driver->crmu_usb2_ctrl);
+
+ /* Disable Host and Device Mode */
+ val = readl(driver->icfgdrd_regs + ICFG_FSM_CTRL);
+ val &= ~(DRD_HOST_MODE | DRD_DEVICE_MODE | ICFG_OFF_MODE);
+ writel(val, driver->icfgdrd_regs + ICFG_FSM_CTRL);
+
+ return 0;
+}
+
+static int ns2_drd_phy_poweron(struct phy *phy)
+{
+ struct ns2_phy_data *data = phy_get_drvdata(phy);
+ struct ns2_phy_driver *driver = data->driver;
+ u32 extcon_event = data->new_state;
+ int ret;
+ u32 val;
+
+ if (extcon_event == EVT_DEVICE) {
+ writel(DRD_DEV_VAL, driver->icfgdrd_regs + ICFG_DRD_P0CTL);
+
+ val = readl(driver->idmdrd_rst_ctrl);
+ val &= ~IDM_RST_BIT;
+ writel(val, driver->idmdrd_rst_ctrl);
+
+ val = readl(driver->crmu_usb2_ctrl);
+ val |= (AFE_CORERDY_VDDC | DRD_DEV_MODE);
+ writel(val, driver->crmu_usb2_ctrl);
+
+ /* Bring PHY and PHY_PLL out of Reset */
+ val = readl(driver->crmu_usb2_ctrl);
+ val |= (PHY_PLL_RESETB | PHY_RESETB);
+ writel(val, driver->crmu_usb2_ctrl);
+
+ ret = pll_lock_stat(ICFG_MISC_STAT, PHY_PLL_LOCK, driver);
+ if (ret < 0) {
+ dev_err(&phy->dev, "Phy PLL lock failed\n");
+ return ret;
+ }
+ } else {
+ writel(DRD_HOST_VAL, driver->icfgdrd_regs + ICFG_DRD_P0CTL);
+
+ val = readl(driver->crmu_usb2_ctrl);
+ val |= AFE_CORERDY_VDDC;
+ writel(val, driver->crmu_usb2_ctrl);
+
+ ret = pll_lock_stat(ICFG_MISC_STAT, PHY_PLL_LOCK, driver);
+ if (ret < 0) {
+ dev_err(&phy->dev, "Phy PLL lock failed\n");
+ return ret;
+ }
+
+ val = readl(driver->idmdrd_rst_ctrl);
+ val &= ~IDM_RST_BIT;
+ writel(val, driver->idmdrd_rst_ctrl);
+
+ /* port over current Polarity */
+ val = readl(driver->usb2h_strap_reg);
+ val |= OHCI_OVRCUR_POL;
+ writel(val, driver->usb2h_strap_reg);
+ }
+
+ return 0;
+}
+
+static void connect_change(struct ns2_phy_driver *driver)
+{
+ u32 extcon_event;
+ u32 val;
+
+ extcon_event = driver->data->new_state;
+ val = readl(driver->icfgdrd_regs + ICFG_FSM_CTRL);
+
+ switch (extcon_event) {
+ case EVT_DEVICE:
+ val &= ~(DRD_HOST_MODE | DRD_DEVICE_MODE);
+ writel(val, driver->icfgdrd_regs + ICFG_FSM_CTRL);
+
+ val = (val & ~DRD_HOST_MODE) | DRD_DEVICE_MODE;
+ writel(val, driver->icfgdrd_regs + ICFG_FSM_CTRL);
+
+ val = readl(driver->icfgdrd_regs + ICFG_DRD_P0CTL);
+ val |= ICFG_DEV_BIT;
+ writel(val, driver->icfgdrd_regs + ICFG_DRD_P0CTL);
+ break;
+
+ case EVT_HOST:
+ val &= ~(DRD_HOST_MODE | DRD_DEVICE_MODE);
+ writel(val, driver->icfgdrd_regs + ICFG_FSM_CTRL);
+
+ val = (val & ~DRD_DEVICE_MODE) | DRD_HOST_MODE;
+ writel(val, driver->icfgdrd_regs + ICFG_FSM_CTRL);
+
+ val = readl(driver->usb2h_strap_reg);
+ val |= OHCI_OVRCUR_POL;
+ writel(val, driver->usb2h_strap_reg);
+
+ val = readl(driver->icfgdrd_regs + ICFG_DRD_P0CTL);
+ val &= ~ICFG_DEV_BIT;
+ writel(val, driver->icfgdrd_regs + ICFG_DRD_P0CTL);
+ break;
+
+ default:
+ pr_err("Invalid extcon event\n");
+ break;
+ }
+}
+
+static void extcon_work(struct work_struct *work)
+{
+ struct ns2_phy_driver *driver;
+ int vbus;
+ int id;
+
+ driver = container_of(to_delayed_work(work),
+ struct ns2_phy_driver, wq_extcon);
+
+ id = gpiod_get_value_cansleep(driver->id_gpiod);
+ vbus = gpiod_get_value_cansleep(driver->vbus_gpiod);
+
+ if (!id && vbus) { /* Host connected */
+ extcon_set_cable_state_(driver->edev, EXTCON_USB_HOST, true);
+ pr_debug("Host cable connected\n");
+ driver->data->new_state = EVT_HOST;
+ connect_change(driver);
+ } else if (id && !vbus) { /* Disconnected */
+ extcon_set_cable_state_(driver->edev, EXTCON_USB_HOST, false);
+ extcon_set_cable_state_(driver->edev, EXTCON_USB, false);
+ pr_debug("Cable disconnected\n");
+ } else if (id && vbus) { /* Device connected */
+ extcon_set_cable_state_(driver->edev, EXTCON_USB, true);
+ pr_debug("Device cable connected\n");
+ driver->data->new_state = EVT_DEVICE;
+ connect_change(driver);
+ }
+}
+
+static irqreturn_t gpio_irq_handler(int irq, void *dev_id)
+{
+ struct ns2_phy_driver *driver = dev_id;
+
+ queue_delayed_work(system_power_efficient_wq, &driver->wq_extcon,
+ driver->debounce_jiffies);
+
+ return IRQ_HANDLED;
+}
+
+static struct phy_ops ops = {
+ .init = ns2_drd_phy_init,
+ .power_on = ns2_drd_phy_poweron,
+ .power_off = ns2_drd_phy_poweroff,
+ .owner = THIS_MODULE,
+};
+
+static const struct of_device_id ns2_drd_phy_dt_ids[] = {
+ { .compatible = "brcm,ns2-drd-phy", },
+ { }
+};
+MODULE_DEVICE_TABLE(of, ns2_drd_phy_dt_ids);
+
+static int ns2_drd_phy_probe(struct platform_device *pdev)
+{
+ struct phy_provider *phy_provider;
+ struct device *dev = &pdev->dev;
+ struct ns2_phy_driver *driver;
+ struct ns2_phy_data *data;
+ struct resource *res;
+ int ret;
+ u32 val;
+
+ driver = devm_kzalloc(dev, sizeof(struct ns2_phy_driver),
+ GFP_KERNEL);
+ if (!driver)
+ return -ENOMEM;
+
+ driver->data = devm_kzalloc(dev, sizeof(struct ns2_phy_data),
+ GFP_KERNEL);
+ if (!driver->data)
+ return -ENOMEM;
+
+ res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "icfg");
+ driver->icfgdrd_regs = devm_ioremap_resource(dev, res);
+ if (IS_ERR(driver->icfgdrd_regs))
+ return PTR_ERR(driver->icfgdrd_regs);
+
+ res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "rst-ctrl");
+ driver->idmdrd_rst_ctrl = devm_ioremap_resource(dev, res);
+ if (IS_ERR(driver->idmdrd_rst_ctrl))
+ return PTR_ERR(driver->idmdrd_rst_ctrl);
+
+ res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "crmu-ctrl");
+ driver->crmu_usb2_ctrl = devm_ioremap_resource(dev, res);
+ if (IS_ERR(driver->crmu_usb2_ctrl))
+ return PTR_ERR(driver->crmu_usb2_ctrl);
+
+ res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "usb2-strap");
+ driver->usb2h_strap_reg = devm_ioremap_resource(dev, res);
+ if (IS_ERR(driver->usb2h_strap_reg))
+ return PTR_ERR(driver->usb2h_strap_reg);
+
+ /* create extcon */
+ driver->id_gpiod = devm_gpiod_get(&pdev->dev, "id", GPIOD_IN);
+ if (IS_ERR(driver->id_gpiod)) {
+ dev_err(dev, "failed to get ID GPIO\n");
+ return PTR_ERR(driver->id_gpiod);
+ }
+ driver->vbus_gpiod = devm_gpiod_get(&pdev->dev, "vbus", GPIOD_IN);
+ if (IS_ERR(driver->vbus_gpiod)) {
+ dev_err(dev, "failed to get VBUS GPIO\n");
+ return PTR_ERR(driver->vbus_gpiod);
+ }
+
+ driver->edev = devm_extcon_dev_allocate(dev, usb_extcon_cable);
+ if (IS_ERR(driver->edev)) {
+ dev_err(dev, "failed to allocate extcon device\n");
+ return -ENOMEM;
+ }
+
+ ret = devm_extcon_dev_register(dev, driver->edev);
+ if (ret < 0) {
+ dev_err(dev, "failed to register extcon device\n");
+ return ret;
+ }
+
+ ret = gpiod_set_debounce(driver->id_gpiod, GPIO_DELAY * 1000);
+ if (ret < 0)
+ driver->debounce_jiffies = msecs_to_jiffies(GPIO_DELAY);
+
+ INIT_DELAYED_WORK(&driver->wq_extcon, extcon_work);
+
+ driver->id_irq = gpiod_to_irq(driver->id_gpiod);
+ if (driver->id_irq < 0) {
+ dev_err(dev, "failed to get ID IRQ\n");
+ return driver->id_irq;
+ }
+
+ driver->vbus_irq = gpiod_to_irq(driver->vbus_gpiod);
+ if (driver->vbus_irq < 0) {
+ dev_err(dev, "failed to get ID IRQ\n");
+ return driver->vbus_irq;
+ }
+
+ ret = devm_request_irq(dev, driver->id_irq, gpio_irq_handler,
+ IRQF_TRIGGER_RISING | IRQF_TRIGGER_FALLING,
+ "usb_id", driver);
+ if (ret < 0) {
+ dev_err(dev, "failed to request handler for ID IRQ\n");
+ return ret;
+ }
+
+ ret = devm_request_irq(dev, driver->vbus_irq, gpio_irq_handler,
+ IRQF_TRIGGER_RISING | IRQF_TRIGGER_FALLING,
+ "usb_vbus", driver);
+ if (ret < 0) {
+ dev_err(dev, "failed to request handler for VBUS IRQ\n");
+ return ret;
+ }
+
+ dev_set_drvdata(dev, driver);
+
+ /* Shutdown all ports. They can be powered up as required */
+ val = readl(driver->crmu_usb2_ctrl);
+ val &= ~(AFE_CORERDY_VDDC | PHY_RESETB);
+ writel(val, driver->crmu_usb2_ctrl);
+
+ data = driver->data;
+ data->phy = devm_phy_create(dev, dev->of_node, &ops);
+ if (IS_ERR(data->phy)) {
+ dev_err(dev, "Failed to create usb drd phy\n");
+ return PTR_ERR(data->phy);
+ }
+
+ data->driver = driver;
+ phy_set_drvdata(data->phy, data);
+
+ phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
+ if (IS_ERR(phy_provider)) {
+ dev_err(dev, "Failed to register as phy provider\n");
+ return PTR_ERR(phy_provider);
+ }
+
+ platform_set_drvdata(pdev, driver);
+
+ dev_info(dev, "Registered NS2 DRD Phy device\n");
+ queue_delayed_work(system_power_efficient_wq, &driver->wq_extcon,
+ driver->debounce_jiffies);
+
+ return 0;
+}
+
+static struct platform_driver ns2_drd_phy_driver = {
+ .probe = ns2_drd_phy_probe,
+ .driver = {
+ .name = "bcm-ns2-usbphy",
+ .of_match_table = of_match_ptr(ns2_drd_phy_dt_ids),
+ },
+};
+module_platform_driver(ns2_drd_phy_driver);
+
+MODULE_ALIAS("platform:bcm-ns2-drd-phy");
+MODULE_AUTHOR("Broadcom");
+MODULE_DESCRIPTION("Broadcom NS2 USB2 PHY driver");
+MODULE_LICENSE("GPL v2");
BRCM_SATA_PHY_STB_40NM,
BRCM_SATA_PHY_IPROC_NS2,
BRCM_SATA_PHY_IPROC_NSP,
+ BRCM_SATA_PHY_IPROC_SR,
};
struct brcm_sata_port {
PLL_ACTRL2 = 0x8b,
PLL_ACTRL2_SELDIV_MASK = 0x1f,
PLL_ACTRL2_SELDIV_SHIFT = 9,
+ PLL_ACTRL6 = 0x86,
PLL1_REG_BANK = 0x060,
PLL1_ACTRL2 = 0x82,
PLL1_ACTRL3 = 0x83,
PLL1_ACTRL4 = 0x84,
+ TX_REG_BANK = 0x070,
+ TX_ACTRL0 = 0x80,
+ TX_ACTRL0_TXPOL_FLIP = BIT(6),
+
OOB_REG_BANK = 0x150,
OOB1_REG_BANK = 0x160,
OOB_CTRL1 = 0x80,
return 0;
}
+/* SR PHY PLL0 registers */
+#define SR_PLL0_ACTRL6_MAGIC 0xa
+
+/* SR PHY PLL1 registers */
+#define SR_PLL1_ACTRL2_MAGIC 0x32
+#define SR_PLL1_ACTRL3_MAGIC 0x2
+#define SR_PLL1_ACTRL4_MAGIC 0x3e8
+
+static int brcm_sr_sata_init(struct brcm_sata_port *port)
+{
+ struct brcm_sata_phy *priv = port->phy_priv;
+ struct device *dev = port->phy_priv->dev;
+ void __iomem *base = priv->phy_base;
+ unsigned int val, try;
+
+ /* Configure PHY PLL register bank 1 */
+ val = SR_PLL1_ACTRL2_MAGIC;
+ brcm_sata_phy_wr(base, PLL1_REG_BANK, PLL1_ACTRL2, 0x0, val);
+ val = SR_PLL1_ACTRL3_MAGIC;
+ brcm_sata_phy_wr(base, PLL1_REG_BANK, PLL1_ACTRL3, 0x0, val);
+ val = SR_PLL1_ACTRL4_MAGIC;
+ brcm_sata_phy_wr(base, PLL1_REG_BANK, PLL1_ACTRL4, 0x0, val);
+
+ /* Configure PHY PLL register bank 0 */
+ val = SR_PLL0_ACTRL6_MAGIC;
+ brcm_sata_phy_wr(base, PLL_REG_BANK_0, PLL_ACTRL6, 0x0, val);
+
+ /* Wait for PHY PLL lock by polling pll_lock bit */
+ try = 50;
+ do {
+ val = brcm_sata_phy_rd(base, BLOCK0_REG_BANK,
+ BLOCK0_XGXSSTATUS);
+ if (val & BLOCK0_XGXSSTATUS_PLL_LOCK)
+ break;
+ msleep(20);
+ try--;
+ } while (try);
+
+ if ((val & BLOCK0_XGXSSTATUS_PLL_LOCK) == 0) {
+ /* PLL did not lock; give up */
+ dev_err(dev, "port%d PLL did not lock\n", port->portnum);
+ return -ETIMEDOUT;
+ }
+
+ /* Invert Tx polarity */
+ brcm_sata_phy_wr(base, TX_REG_BANK, TX_ACTRL0,
+ ~TX_ACTRL0_TXPOL_FLIP, TX_ACTRL0_TXPOL_FLIP);
+
+ /* Configure OOB control to handle 100MHz reference clock */
+ val = ((0xc << OOB_CTRL1_BURST_MAX_SHIFT) |
+ (0x4 << OOB_CTRL1_BURST_MIN_SHIFT) |
+ (0x8 << OOB_CTRL1_WAKE_IDLE_MAX_SHIFT) |
+ (0x3 << OOB_CTRL1_WAKE_IDLE_MIN_SHIFT));
+ brcm_sata_phy_wr(base, OOB_REG_BANK, OOB_CTRL1, 0x0, val);
+ val = ((0x1b << OOB_CTRL2_RESET_IDLE_MAX_SHIFT) |
+ (0x2 << OOB_CTRL2_BURST_CNT_SHIFT) |
+ (0x9 << OOB_CTRL2_RESET_IDLE_MIN_SHIFT));
+ brcm_sata_phy_wr(base, OOB_REG_BANK, OOB_CTRL2, 0x0, val);
+
+ return 0;
+}
+
static int brcm_sata_phy_init(struct phy *phy)
{
int rc;
case BRCM_SATA_PHY_IPROC_NSP:
rc = brcm_nsp_sata_init(port);
break;
+ case BRCM_SATA_PHY_IPROC_SR:
+ rc = brcm_sr_sata_init(port);
+ break;
default:
rc = -ENODEV;
}
.data = (void *)BRCM_SATA_PHY_IPROC_NS2 },
{ .compatible = "brcm,iproc-nsp-sata-phy",
.data = (void *)BRCM_SATA_PHY_IPROC_NSP },
+ { .compatible = "brcm,iproc-sr-sata-phy",
+ .data = (void *)BRCM_SATA_PHY_IPROC_SR },
{},
};
MODULE_DEVICE_TABLE(of, brcm_sata_phy_of_match);
--- /dev/null
+#
+# Phy drivers for Hisilicon platforms
+#
+config PHY_HI6220_USB
+ tristate "hi6220 USB PHY support"
+ depends on (ARCH_HISI && ARM64) || COMPILE_TEST
+ select GENERIC_PHY
+ select MFD_SYSCON
+ help
+ Enable this to support the HISILICON HI6220 USB PHY.
+
+ To compile this driver as a module, choose M here.
+
+config PHY_HIX5HD2_SATA
+ tristate "HIX5HD2 SATA PHY Driver"
+ depends on ARCH_HIX5HD2 && OF && HAS_IOMEM
+ select GENERIC_PHY
+ select MFD_SYSCON
+ help
+ Support for SATA PHY on Hisilicon hix5hd2 Soc.
--- /dev/null
+obj-$(CONFIG_PHY_HI6220_USB) += phy-hi6220-usb.o
+obj-$(CONFIG_PHY_HIX5HD2_SATA) += phy-hix5hd2-sata.o
--- /dev/null
+#
+# Phy drivers for Marvell platforms
+#
+config ARMADA375_USBCLUSTER_PHY
+ def_bool y
+ depends on MACH_ARMADA_375 || COMPILE_TEST
+ depends on OF && HAS_IOMEM
+ select GENERIC_PHY
+
+config PHY_BERLIN_SATA
+ tristate "Marvell Berlin SATA PHY driver"
+ depends on ARCH_BERLIN && HAS_IOMEM && OF
+ select GENERIC_PHY
+ help
+ Enable this to support the SATA PHY on Marvell Berlin SoCs.
+
+config PHY_BERLIN_USB
+ tristate "Marvell Berlin USB PHY Driver"
+ depends on ARCH_BERLIN && RESET_CONTROLLER && HAS_IOMEM && OF
+ select GENERIC_PHY
+ help
+ Enable this to support the USB PHY on Marvell Berlin SoCs.
+
+config PHY_MVEBU_SATA
+ def_bool y
+ depends on ARCH_DOVE || MACH_DOVE || MACH_KIRKWOOD
+ depends on OF
+ select GENERIC_PHY
+
+config PHY_PXA_28NM_HSIC
+ tristate "Marvell USB HSIC 28nm PHY Driver"
+ depends on HAS_IOMEM
+ select GENERIC_PHY
+ help
+ Enable this to support Marvell USB HSIC PHY driver for Marvell
+ SoC. This driver will do the PHY initialization and shutdown.
+ The PHY driver will be used by Marvell ehci driver.
+
+ To compile this driver as a module, choose M here.
+
+config PHY_PXA_28NM_USB2
+ tristate "Marvell USB 2.0 28nm PHY Driver"
+ depends on HAS_IOMEM
+ select GENERIC_PHY
+ help
+ Enable this to support Marvell USB 2.0 PHY driver for Marvell
+ SoC. This driver will do the PHY initialization and shutdown.
+ The PHY driver will be used by Marvell udc/ehci/otg driver.
+
+ To compile this driver as a module, choose M here.
--- /dev/null
+obj-$(CONFIG_ARMADA375_USBCLUSTER_PHY) += phy-armada375-usb2.o
+obj-$(CONFIG_PHY_BERLIN_SATA) += phy-berlin-sata.o
+obj-$(CONFIG_PHY_BERLIN_USB) += phy-berlin-usb.o
+obj-$(CONFIG_PHY_MVEBU_SATA) += phy-mvebu-sata.o
+obj-$(CONFIG_PHY_PXA_28NM_HSIC) += phy-pxa-28nm-hsic.o
+obj-$(CONFIG_PHY_PXA_28NM_USB2) += phy-pxa-28nm-usb2.o
--- /dev/null
+#
+# Phy drivers for Motorola devices
+#
+config PHY_CPCAP_USB
+ tristate "CPCAP PMIC USB PHY driver"
+ depends on USB_SUPPORT && IIO
+ depends on USB_MUSB_HDRC || USB_MUSB_HDRC=n
+ select GENERIC_PHY
+ select USB_PHY
+ help
+ Enable this for USB to work on Motorola phones and tablets
+ such as Droid 4.
--- /dev/null
+#
+# Makefile for the phy drivers.
+#
+
+obj-$(CONFIG_PHY_CPCAP_USB) += phy-cpcap-usb.o
--- /dev/null
+/*
+ * Motorola CPCAP PMIC USB PHY driver
+ * Copyright (C) 2017 Tony Lindgren <tony@atomide.com>
+ *
+ * Some parts based on earlier Motorola Linux kernel tree code in
+ * board-mapphone-usb.c and cpcap-usb-det.c:
+ * Copyright (C) 2007 - 2011 Motorola, Inc.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation version 2.
+ *
+ * This program is distributed "as is" WITHOUT ANY WARRANTY of any
+ * kind, whether express or implied; without even the implied warranty
+ * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/atomic.h>
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/err.h>
+#include <linux/io.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_platform.h>
+#include <linux/iio/consumer.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+#include <linux/slab.h>
+
+#include <linux/gpio/consumer.h>
+#include <linux/mfd/motorola-cpcap.h>
+#include <linux/phy/omap_usb.h>
+#include <linux/phy/phy.h>
+#include <linux/regulator/consumer.h>
+#include <linux/usb/musb.h>
+
+/* CPCAP_REG_USBC1 register bits */
+#define CPCAP_BIT_IDPULSE BIT(15)
+#define CPCAP_BIT_ID100KPU BIT(14)
+#define CPCAP_BIT_IDPUCNTRL BIT(13)
+#define CPCAP_BIT_IDPU BIT(12)
+#define CPCAP_BIT_IDPD BIT(11)
+#define CPCAP_BIT_VBUSCHRGTMR3 BIT(10)
+#define CPCAP_BIT_VBUSCHRGTMR2 BIT(9)
+#define CPCAP_BIT_VBUSCHRGTMR1 BIT(8)
+#define CPCAP_BIT_VBUSCHRGTMR0 BIT(7)
+#define CPCAP_BIT_VBUSPU BIT(6)
+#define CPCAP_BIT_VBUSPD BIT(5)
+#define CPCAP_BIT_DMPD BIT(4)
+#define CPCAP_BIT_DPPD BIT(3)
+#define CPCAP_BIT_DM1K5PU BIT(2)
+#define CPCAP_BIT_DP1K5PU BIT(1)
+#define CPCAP_BIT_DP150KPU BIT(0)
+
+/* CPCAP_REG_USBC2 register bits */
+#define CPCAP_BIT_ZHSDRV1 BIT(15)
+#define CPCAP_BIT_ZHSDRV0 BIT(14)
+#define CPCAP_BIT_DPLLCLKREQ BIT(13)
+#define CPCAP_BIT_SE0CONN BIT(12)
+#define CPCAP_BIT_UARTTXTRI BIT(11)
+#define CPCAP_BIT_UARTSWAP BIT(10)
+#define CPCAP_BIT_UARTMUX1 BIT(9)
+#define CPCAP_BIT_UARTMUX0 BIT(8)
+#define CPCAP_BIT_ULPISTPLOW BIT(7)
+#define CPCAP_BIT_TXENPOL BIT(6)
+#define CPCAP_BIT_USBXCVREN BIT(5)
+#define CPCAP_BIT_USBCNTRL BIT(4)
+#define CPCAP_BIT_USBSUSPEND BIT(3)
+#define CPCAP_BIT_EMUMODE2 BIT(2)
+#define CPCAP_BIT_EMUMODE1 BIT(1)
+#define CPCAP_BIT_EMUMODE0 BIT(0)
+
+/* CPCAP_REG_USBC3 register bits */
+#define CPCAP_BIT_SPARE_898_15 BIT(15)
+#define CPCAP_BIT_IHSTX03 BIT(14)
+#define CPCAP_BIT_IHSTX02 BIT(13)
+#define CPCAP_BIT_IHSTX01 BIT(12)
+#define CPCAP_BIT_IHSTX0 BIT(11)
+#define CPCAP_BIT_IDPU_SPI BIT(10)
+#define CPCAP_BIT_UNUSED_898_9 BIT(9)
+#define CPCAP_BIT_VBUSSTBY_EN BIT(8)
+#define CPCAP_BIT_VBUSEN_SPI BIT(7)
+#define CPCAP_BIT_VBUSPU_SPI BIT(6)
+#define CPCAP_BIT_VBUSPD_SPI BIT(5)
+#define CPCAP_BIT_DMPD_SPI BIT(4)
+#define CPCAP_BIT_DPPD_SPI BIT(3)
+#define CPCAP_BIT_SUSPEND_SPI BIT(2)
+#define CPCAP_BIT_PU_SPI BIT(1)
+#define CPCAP_BIT_ULPI_SPI_SEL BIT(0)
+
+struct cpcap_usb_ints_state {
+ bool id_ground;
+ bool id_float;
+ bool chrg_det;
+ bool rvrs_chrg;
+ bool vbusov;
+
+ bool chrg_se1b;
+ bool se0conn;
+ bool rvrs_mode;
+ bool chrgcurr1;
+ bool vbusvld;
+ bool sessvld;
+ bool sessend;
+ bool se1;
+
+ bool battdetb;
+ bool dm;
+ bool dp;
+};
+
+enum cpcap_gpio_mode {
+ CPCAP_DM_DP,
+ CPCAP_MDM_RX_TX,
+ CPCAP_UNKNOWN,
+ CPCAP_OTG_DM_DP,
+};
+
+struct cpcap_phy_ddata {
+ struct regmap *reg;
+ struct device *dev;
+ struct clk *refclk;
+ struct usb_phy phy;
+ struct delayed_work detect_work;
+ struct pinctrl *pins;
+ struct pinctrl_state *pins_ulpi;
+ struct pinctrl_state *pins_utmi;
+ struct pinctrl_state *pins_uart;
+ struct gpio_desc *gpio[2];
+ struct iio_channel *vbus;
+ struct iio_channel *id;
+ struct regulator *vusb;
+ atomic_t active;
+};
+
+static bool cpcap_usb_vbus_valid(struct cpcap_phy_ddata *ddata)
+{
+ int error, value = 0;
+
+ error = iio_read_channel_processed(ddata->vbus, &value);
+ if (error >= 0)
+ return value > 3900 ? true : false;
+
+ dev_err(ddata->dev, "error reading VBUS: %i\n", error);
+
+ return false;
+}
+
+static int cpcap_usb_phy_set_host(struct usb_otg *otg, struct usb_bus *host)
+{
+ otg->host = host;
+ if (!host)
+ otg->state = OTG_STATE_UNDEFINED;
+
+ return 0;
+}
+
+static int cpcap_usb_phy_set_peripheral(struct usb_otg *otg,
+ struct usb_gadget *gadget)
+{
+ otg->gadget = gadget;
+ if (!gadget)
+ otg->state = OTG_STATE_UNDEFINED;
+
+ return 0;
+}
+
+static const struct phy_ops ops = {
+ .owner = THIS_MODULE,
+};
+
+static int cpcap_phy_get_ints_state(struct cpcap_phy_ddata *ddata,
+ struct cpcap_usb_ints_state *s)
+{
+ int val, error;
+
+ error = regmap_read(ddata->reg, CPCAP_REG_INTS1, &val);
+ if (error)
+ return error;
+
+ s->id_ground = val & BIT(15);
+ s->id_float = val & BIT(14);
+ s->vbusov = val & BIT(11);
+
+ error = regmap_read(ddata->reg, CPCAP_REG_INTS2, &val);
+ if (error)
+ return error;
+
+ s->vbusvld = val & BIT(3);
+ s->sessvld = val & BIT(2);
+ s->sessend = val & BIT(1);
+ s->se1 = val & BIT(0);
+
+ error = regmap_read(ddata->reg, CPCAP_REG_INTS4, &val);
+ if (error)
+ return error;
+
+ s->dm = val & BIT(1);
+ s->dp = val & BIT(0);
+
+ return 0;
+}
+
+static int cpcap_usb_set_uart_mode(struct cpcap_phy_ddata *ddata);
+static int cpcap_usb_set_usb_mode(struct cpcap_phy_ddata *ddata);
+
+static void cpcap_usb_detect(struct work_struct *work)
+{
+ struct cpcap_phy_ddata *ddata;
+ struct cpcap_usb_ints_state s;
+ bool vbus = false;
+ int error;
+
+ ddata = container_of(work, struct cpcap_phy_ddata, detect_work.work);
+
+ error = cpcap_phy_get_ints_state(ddata, &s);
+ if (error)
+ return;
+
+ if (s.id_ground) {
+ dev_dbg(ddata->dev, "id ground, USB host mode\n");
+ error = cpcap_usb_set_usb_mode(ddata);
+ if (error)
+ goto out_err;
+
+ error = musb_mailbox(MUSB_ID_GROUND);
+ if (error)
+ goto out_err;
+
+ error = regmap_update_bits(ddata->reg, CPCAP_REG_USBC3,
+ CPCAP_BIT_VBUSSTBY_EN,
+ CPCAP_BIT_VBUSSTBY_EN);
+ if (error)
+ goto out_err;
+
+ return;
+ }
+
+ error = regmap_update_bits(ddata->reg, CPCAP_REG_USBC3,
+ CPCAP_BIT_VBUSSTBY_EN, 0);
+ if (error)
+ goto out_err;
+
+ vbus = cpcap_usb_vbus_valid(ddata);
+
+ if (vbus) {
+ /* Are we connected to a docking station with vbus? */
+ if (s.id_ground) {
+ dev_dbg(ddata->dev, "connected to a dock\n");
+
+ /* No VBUS needed with docks */
+ error = cpcap_usb_set_usb_mode(ddata);
+ if (error)
+ goto out_err;
+ error = musb_mailbox(MUSB_ID_GROUND);
+ if (error)
+ goto out_err;
+
+ return;
+ }
+
+ /* Otherwise assume we're connected to a USB host */
+ dev_dbg(ddata->dev, "connected to USB host\n");
+ error = cpcap_usb_set_usb_mode(ddata);
+ if (error)
+ goto out_err;
+ error = musb_mailbox(MUSB_VBUS_VALID);
+ if (error)
+ goto out_err;
+
+ return;
+ }
+
+ /* Default to debug UART mode */
+ error = cpcap_usb_set_uart_mode(ddata);
+ if (error)
+ goto out_err;
+
+ error = musb_mailbox(MUSB_VBUS_OFF);
+ if (error)
+ goto out_err;
+
+ dev_dbg(ddata->dev, "set UART mode\n");
+
+ return;
+
+out_err:
+ dev_err(ddata->dev, "error setting cable state: %i\n", error);
+}
+
+static irqreturn_t cpcap_phy_irq_thread(int irq, void *data)
+{
+ struct cpcap_phy_ddata *ddata = data;
+
+ if (!atomic_read(&ddata->active))
+ return IRQ_NONE;
+
+ schedule_delayed_work(&ddata->detect_work, msecs_to_jiffies(1));
+
+ return IRQ_HANDLED;
+}
+
+static int cpcap_usb_init_irq(struct platform_device *pdev,
+ struct cpcap_phy_ddata *ddata,
+ const char *name)
+{
+ int irq, error;
+
+ irq = platform_get_irq_byname(pdev, name);
+ if (!irq)
+ return -ENODEV;
+
+ error = devm_request_threaded_irq(ddata->dev, irq, NULL,
+ cpcap_phy_irq_thread,
+ IRQF_SHARED,
+ name, ddata);
+ if (error) {
+ dev_err(ddata->dev, "could not get irq %s: %i\n",
+ name, error);
+
+ return error;
+ }
+
+ return 0;
+}
+
+static const char * const cpcap_phy_irqs[] = {
+ /* REG_INT_0 */
+ "id_ground", "id_float",
+
+ /* REG_INT1 */
+ "se0conn", "vbusvld", "sessvld", "sessend", "se1",
+
+ /* REG_INT_3 */
+ "dm", "dp",
+};
+
+static int cpcap_usb_init_interrupts(struct platform_device *pdev,
+ struct cpcap_phy_ddata *ddata)
+{
+ int i, error;
+
+ for (i = 0; i < ARRAY_SIZE(cpcap_phy_irqs); i++) {
+ error = cpcap_usb_init_irq(pdev, ddata, cpcap_phy_irqs[i]);
+ if (error)
+ return error;
+ }
+
+ return 0;
+}
+
+/*
+ * Optional pins and modes. At least Motorola mapphone devices
+ * are using two GPIOs and dynamic pinctrl to multiplex PHY pins
+ * to UART, ULPI or UTMI mode.
+ */
+
+static int cpcap_usb_gpio_set_mode(struct cpcap_phy_ddata *ddata,
+ enum cpcap_gpio_mode mode)
+{
+ if (!ddata->gpio[0] || !ddata->gpio[1])
+ return 0;
+
+ gpiod_set_value(ddata->gpio[0], mode & 1);
+ gpiod_set_value(ddata->gpio[1], mode >> 1);
+
+ return 0;
+}
+
+static int cpcap_usb_set_uart_mode(struct cpcap_phy_ddata *ddata)
+{
+ int error;
+
+ error = cpcap_usb_gpio_set_mode(ddata, CPCAP_DM_DP);
+ if (error)
+ goto out_err;
+
+ if (ddata->pins_uart) {
+ error = pinctrl_select_state(ddata->pins, ddata->pins_uart);
+ if (error)
+ goto out_err;
+ }
+
+ error = regmap_update_bits(ddata->reg, CPCAP_REG_USBC1,
+ CPCAP_BIT_VBUSPD,
+ CPCAP_BIT_VBUSPD);
+ if (error)
+ goto out_err;
+
+ error = regmap_update_bits(ddata->reg, CPCAP_REG_USBC2,
+ 0xffff, CPCAP_BIT_UARTMUX0 |
+ CPCAP_BIT_EMUMODE0);
+ if (error)
+ goto out_err;
+
+ error = regmap_update_bits(ddata->reg, CPCAP_REG_USBC3, 0x7fff,
+ CPCAP_BIT_IDPU_SPI);
+ if (error)
+ goto out_err;
+
+ return 0;
+
+out_err:
+ dev_err(ddata->dev, "%s failed with %i\n", __func__, error);
+
+ return error;
+}
+
+static int cpcap_usb_set_usb_mode(struct cpcap_phy_ddata *ddata)
+{
+ int error;
+
+ error = cpcap_usb_gpio_set_mode(ddata, CPCAP_OTG_DM_DP);
+ if (error)
+ return error;
+
+ if (ddata->pins_utmi) {
+ error = pinctrl_select_state(ddata->pins, ddata->pins_utmi);
+ if (error) {
+ dev_err(ddata->dev, "could not set usb mode: %i\n",
+ error);
+
+ return error;
+ }
+ }
+
+ error = regmap_update_bits(ddata->reg, CPCAP_REG_USBC1,
+ CPCAP_BIT_VBUSPD, 0);
+ if (error)
+ goto out_err;
+
+ error = regmap_update_bits(ddata->reg, CPCAP_REG_USBC2,
+ CPCAP_BIT_USBXCVREN,
+ CPCAP_BIT_USBXCVREN);
+ if (error)
+ goto out_err;
+
+ error = regmap_update_bits(ddata->reg, CPCAP_REG_USBC3,
+ CPCAP_BIT_PU_SPI |
+ CPCAP_BIT_DMPD_SPI |
+ CPCAP_BIT_DPPD_SPI |
+ CPCAP_BIT_SUSPEND_SPI |
+ CPCAP_BIT_ULPI_SPI_SEL, 0);
+ if (error)
+ goto out_err;
+
+ error = regmap_update_bits(ddata->reg, CPCAP_REG_USBC2,
+ CPCAP_BIT_USBXCVREN,
+ CPCAP_BIT_USBXCVREN);
+ if (error)
+ goto out_err;
+
+ return 0;
+
+out_err:
+ dev_err(ddata->dev, "%s failed with %i\n", __func__, error);
+
+ return error;
+}
+
+static int cpcap_usb_init_optional_pins(struct cpcap_phy_ddata *ddata)
+{
+ ddata->pins = devm_pinctrl_get(ddata->dev);
+ if (IS_ERR(ddata->pins)) {
+ dev_info(ddata->dev, "default pins not configured: %ld\n",
+ PTR_ERR(ddata->pins));
+ ddata->pins = NULL;
+
+ return 0;
+ }
+
+ ddata->pins_ulpi = pinctrl_lookup_state(ddata->pins, "ulpi");
+ if (IS_ERR(ddata->pins_ulpi)) {
+ dev_info(ddata->dev, "ulpi pins not configured\n");
+ ddata->pins_ulpi = NULL;
+ }
+
+ ddata->pins_utmi = pinctrl_lookup_state(ddata->pins, "utmi");
+ if (IS_ERR(ddata->pins_utmi)) {
+ dev_info(ddata->dev, "utmi pins not configured\n");
+ ddata->pins_utmi = NULL;
+ }
+
+ ddata->pins_uart = pinctrl_lookup_state(ddata->pins, "uart");
+ if (IS_ERR(ddata->pins_uart)) {
+ dev_info(ddata->dev, "uart pins not configured\n");
+ ddata->pins_uart = NULL;
+ }
+
+ if (ddata->pins_uart)
+ return pinctrl_select_state(ddata->pins, ddata->pins_uart);
+
+ return 0;
+}
+
+static void cpcap_usb_init_optional_gpios(struct cpcap_phy_ddata *ddata)
+{
+ int i;
+
+ for (i = 0; i < 2; i++) {
+ ddata->gpio[i] = devm_gpiod_get_index(ddata->dev, "mode",
+ i, GPIOD_OUT_HIGH);
+ if (IS_ERR(ddata->gpio[i])) {
+ dev_info(ddata->dev, "no mode change GPIO%i: %li\n",
+ i, PTR_ERR(ddata->gpio[i]));
+ ddata->gpio[i] = NULL;
+ }
+ }
+}
+
+static int cpcap_usb_init_iio(struct cpcap_phy_ddata *ddata)
+{
+ enum iio_chan_type type;
+ int error;
+
+ ddata->vbus = devm_iio_channel_get(ddata->dev, "vbus");
+ if (IS_ERR(ddata->vbus)) {
+ error = PTR_ERR(ddata->vbus);
+ goto out_err;
+ }
+
+ if (!ddata->vbus->indio_dev) {
+ error = -ENXIO;
+ goto out_err;
+ }
+
+ error = iio_get_channel_type(ddata->vbus, &type);
+ if (error < 0)
+ goto out_err;
+
+ if (type != IIO_VOLTAGE) {
+ error = -EINVAL;
+ goto out_err;
+ }
+
+ return 0;
+
+out_err:
+ dev_err(ddata->dev, "could not initialize VBUS or ID IIO: %i\n",
+ error);
+
+ return error;
+}
+
+#ifdef CONFIG_OF
+static const struct of_device_id cpcap_usb_phy_id_table[] = {
+ {
+ .compatible = "motorola,cpcap-usb-phy",
+ },
+ {
+ .compatible = "motorola,mapphone-cpcap-usb-phy",
+ },
+ {},
+};
+MODULE_DEVICE_TABLE(of, cpcap_usb_phy_id_table);
+#endif
+
+static int cpcap_usb_phy_probe(struct platform_device *pdev)
+{
+ struct cpcap_phy_ddata *ddata;
+ struct phy *generic_phy;
+ struct phy_provider *phy_provider;
+ struct usb_otg *otg;
+ const struct of_device_id *of_id;
+ int error;
+
+ of_id = of_match_device(of_match_ptr(cpcap_usb_phy_id_table),
+ &pdev->dev);
+ if (!of_id)
+ return -EINVAL;
+
+ ddata = devm_kzalloc(&pdev->dev, sizeof(*ddata), GFP_KERNEL);
+ if (!ddata)
+ return -ENOMEM;
+
+ ddata->reg = dev_get_regmap(pdev->dev.parent, NULL);
+ if (!ddata->reg)
+ return -ENODEV;
+
+ otg = devm_kzalloc(&pdev->dev, sizeof(*otg), GFP_KERNEL);
+ if (!otg)
+ return -ENOMEM;
+
+ ddata->dev = &pdev->dev;
+ ddata->phy.dev = ddata->dev;
+ ddata->phy.label = "cpcap_usb_phy";
+ ddata->phy.otg = otg;
+ ddata->phy.type = USB_PHY_TYPE_USB2;
+ otg->set_host = cpcap_usb_phy_set_host;
+ otg->set_peripheral = cpcap_usb_phy_set_peripheral;
+ otg->usb_phy = &ddata->phy;
+ INIT_DELAYED_WORK(&ddata->detect_work, cpcap_usb_detect);
+ platform_set_drvdata(pdev, ddata);
+
+ ddata->vusb = devm_regulator_get(&pdev->dev, "vusb");
+ if (IS_ERR(ddata->vusb))
+ return PTR_ERR(ddata->vusb);
+
+ error = regulator_enable(ddata->vusb);
+ if (error)
+ return error;
+
+ generic_phy = devm_phy_create(ddata->dev, NULL, &ops);
+ if (IS_ERR(generic_phy)) {
+ error = PTR_ERR(generic_phy);
+ return PTR_ERR(generic_phy);
+ }
+
+ phy_set_drvdata(generic_phy, ddata);
+
+ phy_provider = devm_of_phy_provider_register(ddata->dev,
+ of_phy_simple_xlate);
+ if (IS_ERR(phy_provider))
+ return PTR_ERR(phy_provider);
+
+ error = cpcap_usb_init_optional_pins(ddata);
+ if (error)
+ return error;
+
+ cpcap_usb_init_optional_gpios(ddata);
+
+ error = cpcap_usb_init_iio(ddata);
+ if (error)
+ return error;
+
+ error = cpcap_usb_init_interrupts(pdev, ddata);
+ if (error)
+ return error;
+
+ usb_add_phy_dev(&ddata->phy);
+ atomic_set(&ddata->active, 1);
+ schedule_delayed_work(&ddata->detect_work, msecs_to_jiffies(1));
+
+ return 0;
+}
+
+static int cpcap_usb_phy_remove(struct platform_device *pdev)
+{
+ struct cpcap_phy_ddata *ddata = platform_get_drvdata(pdev);
+ int error;
+
+ atomic_set(&ddata->active, 0);
+ error = cpcap_usb_set_uart_mode(ddata);
+ if (error)
+ dev_err(ddata->dev, "could not set UART mode\n");
+
+ error = musb_mailbox(MUSB_VBUS_OFF);
+ if (error)
+ dev_err(ddata->dev, "could not set mailbox\n");
+
+ usb_remove_phy(&ddata->phy);
+ cancel_delayed_work_sync(&ddata->detect_work);
+ clk_unprepare(ddata->refclk);
+ regulator_disable(ddata->vusb);
+
+ return 0;
+}
+
+static struct platform_driver cpcap_usb_phy_driver = {
+ .probe = cpcap_usb_phy_probe,
+ .remove = cpcap_usb_phy_remove,
+ .driver = {
+ .name = "cpcap-usb-phy",
+ .of_match_table = of_match_ptr(cpcap_usb_phy_id_table),
+ },
+};
+
+module_platform_driver(cpcap_usb_phy_driver);
+
+MODULE_ALIAS("platform:cpcap_usb");
+MODULE_AUTHOR("Tony Lindgren <tony@atomide.com>");
+MODULE_DESCRIPTION("CPCAP usb phy driver");
+MODULE_LICENSE("GPL v2");
--- /dev/null
+#
+# Phy drivers for Qualcomm platforms
+#
+config PHY_QCOM_APQ8064_SATA
+ tristate "Qualcomm APQ8064 SATA SerDes/PHY driver"
+ depends on ARCH_QCOM
+ depends on HAS_IOMEM
+ depends on OF
+ select GENERIC_PHY
+
+config PHY_QCOM_IPQ806X_SATA
+ tristate "Qualcomm IPQ806x SATA SerDes/PHY driver"
+ depends on ARCH_QCOM
+ depends on HAS_IOMEM
+ depends on OF
+ select GENERIC_PHY
+
+config PHY_QCOM_QMP
+ tristate "Qualcomm QMP PHY Driver"
+ depends on OF && COMMON_CLK && (ARCH_QCOM || COMPILE_TEST)
+ select GENERIC_PHY
+ help
+ Enable this to support the QMP PHY transceiver that is used
+ with controllers such as PCIe, UFS, and USB on Qualcomm chips.
+
+config PHY_QCOM_QUSB2
+ tristate "Qualcomm QUSB2 PHY Driver"
+ depends on OF && (ARCH_QCOM || COMPILE_TEST)
+ depends on NVMEM || !NVMEM
+ select GENERIC_PHY
+ help
+ Enable this to support the HighSpeed QUSB2 PHY transceiver for USB
+ controllers on Qualcomm chips. This driver supports the high-speed
+ PHY which is usually paired with either the ChipIdea or Synopsys DWC3
+ USB IPs on MSM SOCs.
+
+config PHY_QCOM_UFS
+ tristate "Qualcomm UFS PHY driver"
+ depends on OF && ARCH_QCOM
+ select GENERIC_PHY
+ help
+ Support for UFS PHY on QCOM chipsets.
+
+config PHY_QCOM_USB_HS
+ tristate "Qualcomm USB HS PHY module"
+ depends on USB_ULPI_BUS
+ depends on EXTCON || !EXTCON # if EXTCON=m, this cannot be built-in
+ select GENERIC_PHY
+ help
+ Support for the USB high-speed ULPI compliant phy on Qualcomm
+ chipsets.
+
+config PHY_QCOM_USB_HSIC
+ tristate "Qualcomm USB HSIC ULPI PHY module"
+ depends on USB_ULPI_BUS
+ select GENERIC_PHY
+ help
+ Support for the USB HSIC ULPI compliant PHY on QCOM chipsets.
--- /dev/null
+obj-$(CONFIG_PHY_QCOM_APQ8064_SATA) += phy-qcom-apq8064-sata.o
+obj-$(CONFIG_PHY_QCOM_IPQ806X_SATA) += phy-qcom-ipq806x-sata.o
+obj-$(CONFIG_PHY_QCOM_QMP) += phy-qcom-qmp.o
+obj-$(CONFIG_PHY_QCOM_QUSB2) += phy-qcom-qusb2.o
+obj-$(CONFIG_PHY_QCOM_UFS) += phy-qcom-ufs.o
+obj-$(CONFIG_PHY_QCOM_UFS) += phy-qcom-ufs-qmp-14nm.o
+obj-$(CONFIG_PHY_QCOM_UFS) += phy-qcom-ufs-qmp-20nm.o
+obj-$(CONFIG_PHY_QCOM_USB_HS) += phy-qcom-usb-hs.o
+obj-$(CONFIG_PHY_QCOM_USB_HSIC) += phy-qcom-usb-hsic.o
#include <linux/clk.h>
#include <linux/regulator/consumer.h>
#include <linux/of_device.h>
+#include <linux/phy/phy.h>
#include <linux/reset.h>
#include <linux/extcon.h>
#include <linux/notifier.h>
-#include "ulpi_phy.h"
-
#define ULPI_PWR_CLK_MNG_REG 0x88
# define ULPI_PWR_OTG_COMP_DISABLE BIT(0)
#include <linux/module.h>
#include <linux/ulpi/driver.h>
#include <linux/ulpi/regs.h>
+#include <linux/phy/phy.h>
#include <linux/pinctrl/consumer.h>
#include <linux/pinctrl/pinctrl-state.h>
#include <linux/delay.h>
#include <linux/clk.h>
-#include "ulpi_phy.h"
-
#define ULPI_HSIC_CFG 0x30
#define ULPI_HSIC_IO_CAL 0x33
--- /dev/null
+#
+# Phy drivers for Renesas platforms
+#
+config PHY_RCAR_GEN2
+ tristate "Renesas R-Car generation 2 USB PHY driver"
+ depends on ARCH_RENESAS
+ depends on GENERIC_PHY
+ help
+ Support for USB PHY found on Renesas R-Car generation 2 SoCs.
+
+config PHY_RCAR_GEN3_USB2
+ tristate "Renesas R-Car generation 3 USB 2.0 PHY driver"
+ depends on ARCH_RENESAS
+ depends on EXTCON
+ select GENERIC_PHY
+ help
+ Support for USB 2.0 PHY found on Renesas R-Car generation 3 SoCs.
+
+config PHY_RCAR_GEN3_USB3
+ tristate "Renesas R-Car generation 3 USB 3.0 PHY driver"
+ depends on ARCH_RENESAS || COMPILE_TEST
+ select GENERIC_PHY
+ help
+ Support for USB 3.0 PHY found on Renesas R-Car generation 3 SoCs.
--- /dev/null
+obj-$(CONFIG_PHY_RCAR_GEN2) += phy-rcar-gen2.o
+obj-$(CONFIG_PHY_RCAR_GEN3_USB2) += phy-rcar-gen3-usb2.o
+obj-$(CONFIG_PHY_RCAR_GEN3_USB3) += phy-rcar-gen3-usb3.o
--- /dev/null
+/*
+ * Renesas R-Car Gen3 for USB3.0 PHY driver
+ *
+ * Copyright (C) 2017 Renesas Electronics Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/io.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/phy/phy.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+
+#define USB30_CLKSET0 0x034
+#define USB30_CLKSET1 0x036
+#define USB30_SSC_SET 0x038
+#define USB30_PHY_ENABLE 0x060
+#define USB30_VBUS_EN 0x064
+
+/* USB30_CLKSET0 */
+#define CLKSET0_PRIVATE 0x05c0
+#define CLKSET0_USB30_FSEL_USB_EXTAL 0x0002
+
+/* USB30_CLKSET1 */
+#define CLKSET1_USB30_PLL_MULTI_SHIFT 6
+#define CLKSET1_USB30_PLL_MULTI_USB_EXTAL (0x64 << \
+ CLKSET1_USB30_PLL_MULTI_SHIFT)
+#define CLKSET1_PHYRESET BIT(4) /* 1: reset */
+#define CLKSET1_REF_CLKDIV BIT(3) /* 1: USB_EXTAL */
+#define CLKSET1_PRIVATE_2_1 BIT(1) /* Write B'01 */
+#define CLKSET1_REF_CLK_SEL BIT(0) /* 1: USB3S0_CLK_P */
+
+/* USB30_SSC_SET */
+#define SSC_SET_SSC_EN BIT(12)
+#define SSC_SET_RANGE_SHIFT 9
+#define SSC_SET_RANGE_4980 (0x0 << SSC_SET_RANGE_SHIFT)
+#define SSC_SET_RANGE_4492 (0x1 << SSC_SET_RANGE_SHIFT)
+#define SSC_SET_RANGE_4003 (0x2 << SSC_SET_RANGE_SHIFT)
+
+/* USB30_PHY_ENABLE */
+#define PHY_ENABLE_RESET_EN BIT(4)
+
+/* USB30_VBUS_EN */
+#define VBUS_EN_VBUS_EN BIT(1)
+
+struct rcar_gen3_usb3 {
+ void __iomem *base;
+ struct phy *phy;
+ u32 ssc_range;
+ bool usb3s_clk;
+ bool usb_extal;
+};
+
+static void write_clkset1_for_usb_extal(struct rcar_gen3_usb3 *r, bool reset)
+{
+ u16 val = CLKSET1_USB30_PLL_MULTI_USB_EXTAL |
+ CLKSET1_REF_CLKDIV | CLKSET1_PRIVATE_2_1;
+
+ if (reset)
+ val |= CLKSET1_PHYRESET;
+
+ writew(val, r->base + USB30_CLKSET1);
+}
+
+static void rcar_gen3_phy_usb3_enable_ssc(struct rcar_gen3_usb3 *r)
+{
+ u16 val = SSC_SET_SSC_EN;
+
+ switch (r->ssc_range) {
+ case 4980:
+ val |= SSC_SET_RANGE_4980;
+ break;
+ case 4492:
+ val |= SSC_SET_RANGE_4492;
+ break;
+ case 4003:
+ val |= SSC_SET_RANGE_4003;
+ break;
+ default:
+ dev_err(&r->phy->dev, "%s: unsupported range (%x)\n", __func__,
+ r->ssc_range);
+ return;
+ }
+
+ writew(val, r->base + USB30_SSC_SET);
+}
+
+static void rcar_gen3_phy_usb3_select_usb_extal(struct rcar_gen3_usb3 *r)
+{
+ write_clkset1_for_usb_extal(r, false);
+ if (r->ssc_range)
+ rcar_gen3_phy_usb3_enable_ssc(r);
+ writew(CLKSET0_PRIVATE | CLKSET0_USB30_FSEL_USB_EXTAL,
+ r->base + USB30_CLKSET0);
+ writew(PHY_ENABLE_RESET_EN, r->base + USB30_PHY_ENABLE);
+ write_clkset1_for_usb_extal(r, true);
+ usleep_range(10, 20);
+ write_clkset1_for_usb_extal(r, false);
+}
+
+static int rcar_gen3_phy_usb3_init(struct phy *p)
+{
+ struct rcar_gen3_usb3 *r = phy_get_drvdata(p);
+
+ dev_vdbg(&r->phy->dev, "%s: enter (%d, %d, %d)\n", __func__,
+ r->usb3s_clk, r->usb_extal, r->ssc_range);
+
+ if (!r->usb3s_clk && r->usb_extal)
+ rcar_gen3_phy_usb3_select_usb_extal(r);
+
+ /* Enables VBUS detection anyway */
+ writew(VBUS_EN_VBUS_EN, r->base + USB30_VBUS_EN);
+
+ return 0;
+}
+
+static const struct phy_ops rcar_gen3_phy_usb3_ops = {
+ .init = rcar_gen3_phy_usb3_init,
+ .owner = THIS_MODULE,
+};
+
+static const struct of_device_id rcar_gen3_phy_usb3_match_table[] = {
+ { .compatible = "renesas,rcar-gen3-usb3-phy" },
+ { }
+};
+MODULE_DEVICE_TABLE(of, rcar_gen3_phy_usb3_match_table);
+
+static int rcar_gen3_phy_usb3_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct rcar_gen3_usb3 *r;
+ struct phy_provider *provider;
+ struct resource *res;
+ int ret = 0;
+ struct clk *clk;
+
+ if (!dev->of_node) {
+ dev_err(dev, "This driver needs device tree\n");
+ return -EINVAL;
+ }
+
+ r = devm_kzalloc(dev, sizeof(*r), GFP_KERNEL);
+ if (!r)
+ return -ENOMEM;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ r->base = devm_ioremap_resource(dev, res);
+ if (IS_ERR(r->base))
+ return PTR_ERR(r->base);
+
+ clk = devm_clk_get(dev, "usb3s_clk");
+ if (!IS_ERR(clk) && !clk_prepare_enable(clk)) {
+ r->usb3s_clk = !!clk_get_rate(clk);
+ clk_disable_unprepare(clk);
+ }
+ clk = devm_clk_get(dev, "usb_extal");
+ if (!IS_ERR(clk) && !clk_prepare_enable(clk)) {
+ r->usb_extal = !!clk_get_rate(clk);
+ clk_disable_unprepare(clk);
+ }
+
+ if (!r->usb3s_clk && !r->usb_extal) {
+ dev_err(dev, "This driver needs usb3s_clk and/or usb_extal\n");
+ ret = -EINVAL;
+ goto error;
+ }
+
+ /*
+ * devm_phy_create() will call pm_runtime_enable(&phy->dev);
+ * And then, phy-core will manage runtime pm for this device.
+ */
+ pm_runtime_enable(dev);
+
+ r->phy = devm_phy_create(dev, NULL, &rcar_gen3_phy_usb3_ops);
+ if (IS_ERR(r->phy)) {
+ dev_err(dev, "Failed to create USB3 PHY\n");
+ ret = PTR_ERR(r->phy);
+ goto error;
+ }
+
+ of_property_read_u32(dev->of_node, "renesas,ssc-range", &r->ssc_range);
+
+ platform_set_drvdata(pdev, r);
+ phy_set_drvdata(r->phy, r);
+
+ provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
+ if (IS_ERR(provider)) {
+ dev_err(dev, "Failed to register PHY provider\n");
+ ret = PTR_ERR(provider);
+ goto error;
+ }
+
+ return 0;
+
+error:
+ pm_runtime_disable(dev);
+
+ return ret;
+}
+
+static int rcar_gen3_phy_usb3_remove(struct platform_device *pdev)
+{
+ pm_runtime_disable(&pdev->dev);
+
+ return 0;
+};
+
+static struct platform_driver rcar_gen3_phy_usb3_driver = {
+ .driver = {
+ .name = "phy_rcar_gen3_usb3",
+ .of_match_table = rcar_gen3_phy_usb3_match_table,
+ },
+ .probe = rcar_gen3_phy_usb3_probe,
+ .remove = rcar_gen3_phy_usb3_remove,
+};
+module_platform_driver(rcar_gen3_phy_usb3_driver);
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("Renesas R-Car Gen3 USB 3.0 PHY");
+MODULE_AUTHOR("Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>");
--- /dev/null
+#
+# Phy drivers for Rockchip platforms
+#
+config PHY_ROCKCHIP_DP
+ tristate "Rockchip Display Port PHY Driver"
+ depends on ARCH_ROCKCHIP && OF
+ select GENERIC_PHY
+ help
+ Enable this to support the Rockchip Display Port PHY.
+
+config PHY_ROCKCHIP_EMMC
+ tristate "Rockchip EMMC PHY Driver"
+ depends on ARCH_ROCKCHIP && OF
+ select GENERIC_PHY
+ help
+ Enable this to support the Rockchip EMMC PHY.
+
+config PHY_ROCKCHIP_INNO_USB2
+ tristate "Rockchip INNO USB2PHY Driver"
+ depends on (ARCH_ROCKCHIP || COMPILE_TEST) && OF
+ depends on COMMON_CLK
+ depends on EXTCON
+ depends on USB_SUPPORT
+ select GENERIC_PHY
+ select USB_COMMON
+ help
+ Support for Rockchip USB2.0 PHY with Innosilicon IP block.
+
+config PHY_ROCKCHIP_PCIE
+ tristate "Rockchip PCIe PHY Driver"
+ depends on (ARCH_ROCKCHIP && OF) || COMPILE_TEST
+ select GENERIC_PHY
+ select MFD_SYSCON
+ help
+ Enable this to support the Rockchip PCIe PHY.
+
+config PHY_ROCKCHIP_TYPEC
+ tristate "Rockchip TYPEC PHY Driver"
+ depends on OF && (ARCH_ROCKCHIP || COMPILE_TEST)
+ select EXTCON
+ select GENERIC_PHY
+ select RESET_CONTROLLER
+ help
+ Enable this to support the Rockchip USB TYPEC PHY.
+
+config PHY_ROCKCHIP_USB
+ tristate "Rockchip USB2 PHY Driver"
+ depends on ARCH_ROCKCHIP && OF
+ select GENERIC_PHY
+ help
+ Enable this to support the Rockchip USB 2.0 PHY.
--- /dev/null
+obj-$(CONFIG_PHY_ROCKCHIP_DP) += phy-rockchip-dp.o
+obj-$(CONFIG_PHY_ROCKCHIP_EMMC) += phy-rockchip-emmc.o
+obj-$(CONFIG_PHY_ROCKCHIP_INNO_USB2) += phy-rockchip-inno-usb2.o
+obj-$(CONFIG_PHY_ROCKCHIP_PCIE) += phy-rockchip-pcie.o
+obj-$(CONFIG_PHY_ROCKCHIP_TYPEC) += phy-rockchip-typec.o
+obj-$(CONFIG_PHY_ROCKCHIP_USB) += phy-rockchip-usb.o
mutex_lock(&rport->mutex);
if (rport->port_id == USB2PHY_PORT_OTG) {
- if (rport->mode != USB_DR_MODE_HOST) {
+ if (rport->mode != USB_DR_MODE_HOST &&
+ rport->mode != USB_DR_MODE_UNKNOWN) {
/* clear bvalid status and enable bvalid detect irq */
ret = property_enable(rphy,
&rport->port_cfg->bvalid_det_clr,
goto out;
schedule_delayed_work(&rport->otg_sm_work,
- OTG_SCHEDULE_DELAY);
+ OTG_SCHEDULE_DELAY * 3);
} else {
/* If OTG works in host only mode, do nothing. */
dev_dbg(&rport->phy->dev, "mode %d\n", rport->mode);
if (ret)
return ret;
+ /* waiting for the utmi_clk to become stable */
+ usleep_range(1500, 2000);
+
rport->suspended = false;
return 0;
}
struct rockchip_usb2phy_port *rport = phy_get_drvdata(phy);
if (rport->port_id == USB2PHY_PORT_OTG &&
- rport->mode != USB_DR_MODE_HOST) {
+ rport->mode != USB_DR_MODE_HOST &&
+ rport->mode != USB_DR_MODE_UNKNOWN) {
cancel_delayed_work_sync(&rport->otg_sm_work);
cancel_delayed_work_sync(&rport->chg_work);
} else if (rport->port_id == USB2PHY_PORT_HOST)
mutex_init(&rport->mutex);
rport->mode = of_usb_get_dr_mode_by_phy(child_np, -1);
- if (rport->mode == USB_DR_MODE_HOST) {
+ if (rport->mode == USB_DR_MODE_HOST ||
+ rport->mode == USB_DR_MODE_UNKNOWN) {
ret = 0;
goto out;
}
return ret;
}
+static const struct rockchip_usb2phy_cfg rk3228_phy_cfgs[] = {
+ {
+ .reg = 0x760,
+ .num_ports = 2,
+ .clkout_ctl = { 0x0768, 4, 4, 1, 0 },
+ .port_cfgs = {
+ [USB2PHY_PORT_OTG] = {
+ .phy_sus = { 0x0760, 15, 0, 0, 0x1d1 },
+ .bvalid_det_en = { 0x0680, 3, 3, 0, 1 },
+ .bvalid_det_st = { 0x0690, 3, 3, 0, 1 },
+ .bvalid_det_clr = { 0x06a0, 3, 3, 0, 1 },
+ .ls_det_en = { 0x0680, 2, 2, 0, 1 },
+ .ls_det_st = { 0x0690, 2, 2, 0, 1 },
+ .ls_det_clr = { 0x06a0, 2, 2, 0, 1 },
+ .utmi_bvalid = { 0x0480, 4, 4, 0, 1 },
+ .utmi_ls = { 0x0480, 3, 2, 0, 1 },
+ },
+ [USB2PHY_PORT_HOST] = {
+ .phy_sus = { 0x0764, 15, 0, 0, 0x1d1 },
+ .ls_det_en = { 0x0680, 4, 4, 0, 1 },
+ .ls_det_st = { 0x0690, 4, 4, 0, 1 },
+ .ls_det_clr = { 0x06a0, 4, 4, 0, 1 }
+ }
+ },
+ .chg_det = {
+ .opmode = { 0x0760, 3, 0, 5, 1 },
+ .cp_det = { 0x0884, 4, 4, 0, 1 },
+ .dcp_det = { 0x0884, 3, 3, 0, 1 },
+ .dp_det = { 0x0884, 5, 5, 0, 1 },
+ .idm_sink_en = { 0x0768, 8, 8, 0, 1 },
+ .idp_sink_en = { 0x0768, 7, 7, 0, 1 },
+ .idp_src_en = { 0x0768, 9, 9, 0, 1 },
+ .rdm_pdwn_en = { 0x0768, 10, 10, 0, 1 },
+ .vdm_src_en = { 0x0768, 12, 12, 0, 1 },
+ .vdp_src_en = { 0x0768, 11, 11, 0, 1 },
+ },
+ },
+ {
+ .reg = 0x800,
+ .num_ports = 2,
+ .clkout_ctl = { 0x0808, 4, 4, 1, 0 },
+ .port_cfgs = {
+ [USB2PHY_PORT_OTG] = {
+ .phy_sus = { 0x800, 15, 0, 0, 0x1d1 },
+ .ls_det_en = { 0x0684, 0, 0, 0, 1 },
+ .ls_det_st = { 0x0694, 0, 0, 0, 1 },
+ .ls_det_clr = { 0x06a4, 0, 0, 0, 1 }
+ },
+ [USB2PHY_PORT_HOST] = {
+ .phy_sus = { 0x804, 15, 0, 0, 0x1d1 },
+ .ls_det_en = { 0x0684, 1, 1, 0, 1 },
+ .ls_det_st = { 0x0694, 1, 1, 0, 1 },
+ .ls_det_clr = { 0x06a4, 1, 1, 0, 1 }
+ }
+ },
+ },
+ { /* sentinel */ }
+};
+
static const struct rockchip_usb2phy_cfg rk3328_phy_cfgs[] = {
{
.reg = 0x100,
};
static const struct of_device_id rockchip_usb2phy_dt_match[] = {
+ { .compatible = "rockchip,rk3228-usb2phy", .data = &rk3228_phy_cfgs },
{ .compatible = "rockchip,rk3328-usb2phy", .data = &rk3328_phy_cfgs },
{ .compatible = "rockchip,rk3366-usb2phy", .data = &rk3366_phy_cfgs },
{ .compatible = "rockchip,rk3399-usb2phy", .data = &rk3399_phy_cfgs },
--- /dev/null
+#
+# Phy drivers for Samsung platforms
+#
+config PHY_EXYNOS_DP_VIDEO
+ tristate "EXYNOS SoC series Display Port PHY driver"
+ depends on OF
+ depends on ARCH_EXYNOS || COMPILE_TEST
+ default ARCH_EXYNOS
+ select GENERIC_PHY
+ help
+ Support for Display Port PHY found on Samsung EXYNOS SoCs.
+
+config PHY_EXYNOS_MIPI_VIDEO
+ tristate "S5P/EXYNOS SoC series MIPI CSI-2/DSI PHY driver"
+ depends on HAS_IOMEM
+ depends on ARCH_S5PV210 || ARCH_EXYNOS || COMPILE_TEST
+ select GENERIC_PHY
+ default y if ARCH_S5PV210 || ARCH_EXYNOS
+ help
+ Support for MIPI CSI-2 and MIPI DSI DPHY found on Samsung S5P
+ and EXYNOS SoCs.
+
+config PHY_EXYNOS_PCIE
+ bool "Exynos PCIe PHY driver"
+ depends on OF && (ARCH_EXYNOS || COMPILE_TEST)
+ select GENERIC_PHY
+ help
+ Enable PCIe PHY support for Exynos SoC series.
+ This driver provides PHY interface for Exynos PCIe controller.
+
+config PHY_SAMSUNG_USB2
+ tristate "Samsung USB 2.0 PHY driver"
+ depends on HAS_IOMEM
+ depends on USB_EHCI_EXYNOS || USB_OHCI_EXYNOS || USB_DWC2
+ select GENERIC_PHY
+ select MFD_SYSCON
+ default ARCH_EXYNOS
+ help
+ Enable this to support the Samsung USB 2.0 PHY driver for Samsung
+ SoCs. This driver provides the interface for USB 2.0 PHY. Support
+ for particular PHYs will be enabled based on the SoC type in addition
+ to this driver.
+
+config PHY_EXYNOS4210_USB2
+ bool
+ depends on PHY_SAMSUNG_USB2
+ default CPU_EXYNOS4210
+
+config PHY_EXYNOS4X12_USB2
+ bool
+ depends on PHY_SAMSUNG_USB2
+ default SOC_EXYNOS3250 || SOC_EXYNOS4212 || SOC_EXYNOS4412
+
+config PHY_EXYNOS5250_USB2
+ bool
+ depends on PHY_SAMSUNG_USB2
+ default SOC_EXYNOS5250 || SOC_EXYNOS5420
+
+config PHY_S5PV210_USB2
+ bool "Support for S5PV210"
+ depends on PHY_SAMSUNG_USB2
+ depends on ARCH_S5PV210
+ help
+ Enable USB PHY support for S5PV210. This option requires that Samsung
+ USB 2.0 PHY driver is enabled and means that support for this
+ particular SoC is compiled in the driver. In case of S5PV210 two phys
+ are available - device and host.
+
+config PHY_EXYNOS5_USBDRD
+ tristate "Exynos5 SoC series USB DRD PHY driver"
+ depends on ARCH_EXYNOS && OF
+ depends on HAS_IOMEM
+ depends on USB_DWC3_EXYNOS
+ select GENERIC_PHY
+ select MFD_SYSCON
+ default y
+ help
+ Enable USB DRD PHY support for Exynos 5 SoC series.
+ This driver provides PHY interface for USB 3.0 DRD controller
+ present on Exynos5 SoC series.
+
+config PHY_EXYNOS5250_SATA
+ tristate "Exynos5250 Sata SerDes/PHY driver"
+ depends on SOC_EXYNOS5250
+ depends on HAS_IOMEM
+ depends on OF
+ select GENERIC_PHY
+ select I2C
+ select I2C_S3C2410
+ select MFD_SYSCON
+ help
+ Enable this to support SATA SerDes/Phy found on Samsung's
+ Exynos5250 based SoCs.This SerDes/Phy supports SATA 1.5 Gb/s,
+ SATA 3.0 Gb/s, SATA 6.0 Gb/s speeds. It supports one SATA host
+ port to accept one SATA device.
--- /dev/null
+obj-$(CONFIG_PHY_EXYNOS_DP_VIDEO) += phy-exynos-dp-video.o
+obj-$(CONFIG_PHY_EXYNOS_MIPI_VIDEO) += phy-exynos-mipi-video.o
+obj-$(CONFIG_PHY_EXYNOS_PCIE) += phy-exynos-pcie.o
+obj-$(CONFIG_PHY_SAMSUNG_USB2) += phy-exynos-usb2.o
+phy-exynos-usb2-y += phy-samsung-usb2.o
+phy-exynos-usb2-$(CONFIG_PHY_EXYNOS4210_USB2) += phy-exynos4210-usb2.o
+phy-exynos-usb2-$(CONFIG_PHY_EXYNOS4X12_USB2) += phy-exynos4x12-usb2.o
+phy-exynos-usb2-$(CONFIG_PHY_EXYNOS5250_USB2) += phy-exynos5250-usb2.o
+phy-exynos-usb2-$(CONFIG_PHY_S5PV210_USB2) += phy-s5pv210-usb2.o
+obj-$(CONFIG_PHY_EXYNOS5_USBDRD) += phy-exynos5-usbdrd.o
+obj-$(CONFIG_PHY_EXYNOS5250_SATA) += phy-exynos5250-sata.o
--- /dev/null
+#
+# Phy drivers for STMicro platforms
+#
+config PHY_MIPHY28LP
+ tristate "STMicroelectronics MIPHY28LP PHY driver for STiH407"
+ depends on ARCH_STI
+ select GENERIC_PHY
+ help
+ Enable this to support the miphy transceiver (for SATA/PCIE/USB3)
+ that is part of STMicroelectronics STiH407 SoC.
+
+config PHY_ST_SPEAR1310_MIPHY
+ tristate "ST SPEAR1310-MIPHY driver"
+ select GENERIC_PHY
+ depends on MACH_SPEAR1310 || COMPILE_TEST
+ help
+ Support for ST SPEAr1310 MIPHY which can be used for PCIe and SATA.
+
+config PHY_ST_SPEAR1340_MIPHY
+ tristate "ST SPEAR1340-MIPHY driver"
+ select GENERIC_PHY
+ depends on MACH_SPEAR1340 || COMPILE_TEST
+ help
+ Support for ST SPEAr1340 MIPHY which can be used for PCIe and SATA.
+
+config PHY_STIH407_USB
+ tristate "STMicroelectronics USB2 picoPHY driver for STiH407 family"
+ depends on RESET_CONTROLLER
+ depends on ARCH_STI || COMPILE_TEST
+ select GENERIC_PHY
+ help
+ Enable this support to enable the picoPHY device used by USB2
+ and USB3 controllers on STMicroelectronics STiH407 SoC families.
--- /dev/null
+obj-$(CONFIG_PHY_MIPHY28LP) += phy-miphy28lp.o
+obj-$(CONFIG_PHY_ST_SPEAR1310_MIPHY) += phy-spear1310-miphy.o
+obj-$(CONFIG_PHY_ST_SPEAR1340_MIPHY) += phy-spear1340-miphy.o
+obj-$(CONFIG_PHY_STIH407_USB) += phy-stih407-usb.o
--- /dev/null
+#
+# Phy drivers for TI platforms
+#
+config PHY_DA8XX_USB
+ tristate "TI DA8xx USB PHY Driver"
+ depends on ARCH_DAVINCI_DA8XX
+ select GENERIC_PHY
+ select MFD_SYSCON
+ help
+ Enable this to support the USB PHY on DA8xx SoCs.
+
+ This driver controls both the USB 1.1 PHY and the USB 2.0 PHY.
+
+config PHY_DM816X_USB
+ tristate "TI dm816x USB PHY driver"
+ depends on ARCH_OMAP2PLUS
+ depends on USB_SUPPORT
+ select GENERIC_PHY
+ select USB_PHY
+ help
+ Enable this for dm816x USB to work.
+
+config OMAP_CONTROL_PHY
+ tristate "OMAP CONTROL PHY Driver"
+ depends on ARCH_OMAP2PLUS || COMPILE_TEST
+ help
+ Enable this to add support for the PHY part present in the control
+ module. This driver has API to power on the USB2 PHY and to write to
+ the mailbox. The mailbox is present only in omap4 and the register to
+ power on the USB2 PHY is present in OMAP4 and OMAP5. OMAP5 has an
+ additional register to power on USB3 PHY/SATA PHY/PCIE PHY
+ (PIPE3 PHY).
+
+config OMAP_USB2
+ tristate "OMAP USB2 PHY Driver"
+ depends on ARCH_OMAP2PLUS
+ depends on USB_SUPPORT
+ select GENERIC_PHY
+ select USB_PHY
+ select OMAP_CONTROL_PHY
+ depends on OMAP_OCP2SCP
+ help
+ Enable this to support the transceiver that is part of SOC. This
+ driver takes care of all the PHY functionality apart from comparator.
+ The USB OTG controller communicates with the comparator using this
+ driver.
+
+config TI_PIPE3
+ tristate "TI PIPE3 PHY Driver"
+ depends on ARCH_OMAP2PLUS || COMPILE_TEST
+ select GENERIC_PHY
+ select OMAP_CONTROL_PHY
+ depends on OMAP_OCP2SCP
+ help
+ Enable this to support the PIPE3 PHY that is part of TI SOCs. This
+ driver takes care of all the PHY functionality apart from comparator.
+ This driver interacts with the "OMAP Control PHY Driver" to power
+ on/off the PHY.
+
+config PHY_TUSB1210
+ tristate "TI TUSB1210 ULPI PHY module"
+ depends on USB_ULPI_BUS
+ select GENERIC_PHY
+ help
+ Support for TI TUSB1210 USB ULPI PHY.
+
+config TWL4030_USB
+ tristate "TWL4030 USB Transceiver Driver"
+ depends on TWL4030_CORE && REGULATOR_TWL4030 && USB_MUSB_OMAP2PLUS
+ depends on USB_SUPPORT
+ depends on USB_GADGET || !USB_GADGET # if USB_GADGET=m, this can't 'y'
+ select GENERIC_PHY
+ select USB_PHY
+ help
+ Enable this to support the USB OTG transceiver on TWL4030
+ family chips (including the TWL5030 and TPS659x0 devices).
+ This transceiver supports high and full speed devices plus,
+ in host mode, low speed.
--- /dev/null
+obj-$(CONFIG_PHY_DA8XX_USB) += phy-da8xx-usb.o
+obj-$(CONFIG_PHY_DM816X_USB) += phy-dm816x-usb.o
+obj-$(CONFIG_OMAP_CONTROL_PHY) += phy-omap-control.o
+obj-$(CONFIG_OMAP_USB2) += phy-omap-usb2.o
+obj-$(CONFIG_TI_PIPE3) += phy-ti-pipe3.o
+obj-$(CONFIG_PHY_TUSB1210) += phy-tusb1210.o
+obj-$(CONFIG_TWL4030_USB) += phy-twl4030-usb.o
*/
#include <linux/module.h>
#include <linux/ulpi/driver.h>
+#include <linux/ulpi/regs.h>
#include <linux/gpio/consumer.h>
-
-#include "ulpi_phy.h"
+#include <linux/phy/ulpi_phy.h>
#define TUSB1210_VENDOR_SPECIFIC2 0x80
#define TUSB1210_VENDOR_SPECIFIC2_IHSTX_SHIFT 0
return 0;
}
+static int tusb1210_set_mode(struct phy *phy, enum phy_mode mode)
+{
+ struct tusb1210 *tusb = phy_get_drvdata(phy);
+ int ret;
+
+ ret = ulpi_read(tusb->ulpi, ULPI_OTG_CTRL);
+ if (ret < 0)
+ return ret;
+
+ switch (mode) {
+ case PHY_MODE_USB_HOST:
+ ret |= (ULPI_OTG_CTRL_DRVVBUS_EXT
+ | ULPI_OTG_CTRL_ID_PULLUP
+ | ULPI_OTG_CTRL_DP_PULLDOWN
+ | ULPI_OTG_CTRL_DM_PULLDOWN);
+ ulpi_write(tusb->ulpi, ULPI_OTG_CTRL, ret);
+ ret |= ULPI_OTG_CTRL_DRVVBUS;
+ break;
+ case PHY_MODE_USB_DEVICE:
+ ret &= ~(ULPI_OTG_CTRL_DRVVBUS
+ | ULPI_OTG_CTRL_DP_PULLDOWN
+ | ULPI_OTG_CTRL_DM_PULLDOWN);
+ ulpi_write(tusb->ulpi, ULPI_OTG_CTRL, ret);
+ ret &= ~ULPI_OTG_CTRL_DRVVBUS_EXT;
+ break;
+ default:
+ /* nothing */
+ return 0;
+ }
+
+ return ulpi_write(tusb->ulpi, ULPI_OTG_CTRL, ret);
+}
+
static const struct phy_ops phy_ops = {
.power_on = tusb1210_power_on,
.power_off = tusb1210_power_off,
+ .set_mode = tusb1210_set_mode,
.owner = THIS_MODULE,
};
#define TI_VENDOR_ID 0x0451
static const struct ulpi_device_id tusb1210_ulpi_id[] = {
- { TI_VENDOR_ID, 0x1507, },
+ { TI_VENDOR_ID, 0x1507, }, /* TUSB1210 */
+ { TI_VENDOR_ID, 0x1508, }, /* TUSB1211 */
{ },
};
MODULE_DEVICE_TABLE(ulpi, tusb1210_ulpi_id);
.flags = IRQCHIP_SKIP_SET_WAKE,
};
-static void amd_gpio_irq_handler(struct irq_desc *desc)
+#define PIN_IRQ_PENDING (BIT(INTERRUPT_STS_OFF) | BIT(WAKE_STS_OFF))
+
+static irqreturn_t amd_gpio_irq_handler(int irq, void *dev_id)
{
- u32 i;
- u32 off;
- u32 reg;
- u32 pin_reg;
- u64 reg64;
- int handled = 0;
- unsigned int irq;
+ struct amd_gpio *gpio_dev = dev_id;
+ struct gpio_chip *gc = &gpio_dev->gc;
+ irqreturn_t ret = IRQ_NONE;
+ unsigned int i, irqnr;
unsigned long flags;
- struct irq_chip *chip = irq_desc_get_chip(desc);
- struct gpio_chip *gc = irq_desc_get_handler_data(desc);
- struct amd_gpio *gpio_dev = gpiochip_get_data(gc);
+ u32 *regs, regval;
+ u64 status, mask;
- chained_irq_enter(chip, desc);
- /*enable GPIO interrupt again*/
+ /* Read the wake status */
raw_spin_lock_irqsave(&gpio_dev->lock, flags);
- reg = readl(gpio_dev->base + WAKE_INT_STATUS_REG1);
- reg64 = reg;
- reg64 = reg64 << 32;
-
- reg = readl(gpio_dev->base + WAKE_INT_STATUS_REG0);
- reg64 |= reg;
+ status = readl(gpio_dev->base + WAKE_INT_STATUS_REG1);
+ status <<= 32;
+ status |= readl(gpio_dev->base + WAKE_INT_STATUS_REG0);
raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
- /*
- * first 46 bits indicates interrupt status.
- * one bit represents four interrupt sources.
- */
- for (off = 0; off < 46 ; off++) {
- if (reg64 & BIT(off)) {
- for (i = 0; i < 4; i++) {
- pin_reg = readl(gpio_dev->base +
- (off * 4 + i) * 4);
- if ((pin_reg & BIT(INTERRUPT_STS_OFF)) ||
- (pin_reg & BIT(WAKE_STS_OFF))) {
- irq = irq_find_mapping(gc->irqdomain,
- off * 4 + i);
- generic_handle_irq(irq);
- writel(pin_reg,
- gpio_dev->base
- + (off * 4 + i) * 4);
- handled++;
- }
- }
+ /* Bit 0-45 contain the relevant status bits */
+ status &= (1ULL << 46) - 1;
+ regs = gpio_dev->base;
+ for (mask = 1, irqnr = 0; status; mask <<= 1, regs += 4, irqnr += 4) {
+ if (!(status & mask))
+ continue;
+ status &= ~mask;
+
+ /* Each status bit covers four pins */
+ for (i = 0; i < 4; i++) {
+ regval = readl(regs + i);
+ if (!(regval & PIN_IRQ_PENDING))
+ continue;
+ irq = irq_find_mapping(gc->irqdomain, irqnr + i);
+ generic_handle_irq(irq);
+ /* Clear interrupt */
+ writel(regval, regs + i);
+ ret = IRQ_HANDLED;
}
}
- if (handled == 0)
- handle_bad_irq(desc);
-
+ /* Signal EOI to the GPIO unit */
raw_spin_lock_irqsave(&gpio_dev->lock, flags);
- reg = readl(gpio_dev->base + WAKE_INT_MASTER_REG);
- reg |= EOI_MASK;
- writel(reg, gpio_dev->base + WAKE_INT_MASTER_REG);
+ regval = readl(gpio_dev->base + WAKE_INT_MASTER_REG);
+ regval |= EOI_MASK;
+ writel(regval, gpio_dev->base + WAKE_INT_MASTER_REG);
raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
- chained_irq_exit(chip, desc);
+ return ret;
}
static int amd_get_groups_count(struct pinctrl_dev *pctldev)
goto out2;
}
- gpiochip_set_chained_irqchip(&gpio_dev->gc,
- &amd_gpio_irqchip,
- irq_base,
- amd_gpio_irq_handler);
+ ret = devm_request_irq(&pdev->dev, irq_base, amd_gpio_irq_handler, 0,
+ KBUILD_MODNAME, gpio_dev);
+ if (ret)
+ goto out2;
+
platform_set_drvdata(pdev, gpio_dev);
dev_dbg(&pdev->dev, "amd gpio driver loaded\n");
* @gpio_chip: gpiolib chip
* @grange: gpio range
* @slock: spinlock for the gpio bank
- * @irq_lock: bus lock for irq chip
- * @new_irqs: newly configured irqs which must be muxed as GPIOs in
- * irq_bus_sync_unlock()
*/
struct rockchip_pin_bank {
void __iomem *reg_base;
struct pinctrl_gpio_range grange;
raw_spinlock_t slock;
u32 toggle_edge_mode;
- struct mutex irq_lock;
- u32 new_irqs;
};
#define PIN_BANK(id, pins, label) \
int ret;
/* make sure the pin is configured as gpio input */
- ret = rockchip_verify_mux(bank, d->hwirq, RK_FUNC_GPIO);
+ ret = rockchip_set_mux(bank, d->hwirq, RK_FUNC_GPIO);
if (ret < 0)
return ret;
- bank->new_irqs |= mask;
-
+ clk_enable(bank->clk);
raw_spin_lock_irqsave(&bank->slock, flags);
data = readl_relaxed(bank->reg_base + GPIO_SWPORT_DDR);
default:
irq_gc_unlock(gc);
raw_spin_unlock_irqrestore(&bank->slock, flags);
+ clk_disable(bank->clk);
return -EINVAL;
}
irq_gc_unlock(gc);
raw_spin_unlock_irqrestore(&bank->slock, flags);
+ clk_disable(bank->clk);
return 0;
}
clk_disable(bank->clk);
}
-static void rockchip_irq_bus_lock(struct irq_data *d)
-{
- struct irq_chip_generic *gc = irq_data_get_irq_chip_data(d);
- struct rockchip_pin_bank *bank = gc->private;
-
- clk_enable(bank->clk);
- mutex_lock(&bank->irq_lock);
-}
-
-static void rockchip_irq_bus_sync_unlock(struct irq_data *d)
-{
- struct irq_chip_generic *gc = irq_data_get_irq_chip_data(d);
- struct rockchip_pin_bank *bank = gc->private;
-
- while (bank->new_irqs) {
- unsigned int irq = __ffs(bank->new_irqs);
- int ret;
-
- ret = rockchip_set_mux(bank, irq, RK_FUNC_GPIO);
- WARN_ON(ret < 0);
-
- bank->new_irqs &= ~BIT(irq);
- }
-
- mutex_unlock(&bank->irq_lock);
- clk_disable(bank->clk);
-}
-
static int rockchip_interrupts_register(struct platform_device *pdev,
struct rockchip_pinctrl *info)
{
gc->chip_types[0].chip.irq_suspend = rockchip_irq_suspend;
gc->chip_types[0].chip.irq_resume = rockchip_irq_resume;
gc->chip_types[0].chip.irq_set_type = rockchip_irq_set_type;
- gc->chip_types[0].chip.irq_bus_lock = rockchip_irq_bus_lock;
- gc->chip_types[0].chip.irq_bus_sync_unlock =
- rockchip_irq_bus_sync_unlock;
gc->wake_enabled = IRQ_MSK(bank->nr_pins);
irq_set_chained_handler_and_data(bank->irq,
int bank_pins = 0;
raw_spin_lock_init(&bank->slock);
- mutex_init(&bank->irq_lock);
bank->drvdata = d;
bank->pin_base = ctrl->nr_pins;
ctrl->nr_pins += bank->nr_pins;
break;
case PIN_CONFIG_OUTPUT:
__stm32_gpio_set(bank, offset, arg);
- ret = stm32_pmx_gpio_set_direction(pctldev, NULL, pin, false);
+ ret = stm32_pmx_gpio_set_direction(pctldev, range, pin, false);
break;
default:
ret = -EINVAL;
unsigned char __iomem *base;
};
-struct goldfish_pipe_dev pipe_dev[1] = {};
+static struct goldfish_pipe_dev pipe_dev[1] = {};
static int goldfish_cmd_locked(struct goldfish_pipe *pipe, enum PipeCmdCode cmd)
{
This driver instantiates i2c-clients for these, so that standard
i2c drivers for these chips can bind to the them.
+config INTEL_INT0002_VGPIO
+ tristate "Intel ACPI INT0002 Virtual GPIO driver"
+ depends on GPIOLIB && ACPI
+ select GPIOLIB_IRQCHIP
+ ---help---
+ Some peripherals on Bay Trail and Cherry Trail platforms signal a
+ Power Management Event (PME) to the Power Management Controller (PMC)
+ to wakeup the system. When this happens software needs to explicitly
+ clear the PME bus 0 status bit in the GPE0a_STS register to avoid an
+ IRQ storm on IRQ 9.
+
+ This is modelled in ACPI through the INT0002 ACPI device, which is
+ called a "Virtual GPIO controller" in ACPI because it defines the
+ event handler to call when the PME triggers through _AEI and _L02
+ methods as would be done for a real GPIO interrupt in ACPI.
+
+ To compile this driver as a module, choose M here: the module will
+ be called intel_int0002_vgpio.
+
config INTEL_HID_EVENT
tristate "INTEL HID Event"
depends on ACPI
obj-$(CONFIG_TOSHIBA_HAPS) += toshiba_haps.o
obj-$(CONFIG_TOSHIBA_WMI) += toshiba-wmi.o
obj-$(CONFIG_INTEL_CHT_INT33FE) += intel_cht_int33fe.o
+obj-$(CONFIG_INTEL_INT0002_VGPIO) += intel_int0002_vgpio.o
obj-$(CONFIG_INTEL_HID_EVENT) += intel-hid.o
obj-$(CONFIG_INTEL_VBTN) += intel-vbtn.o
obj-$(CONFIG_INTEL_SCU_IPC) += intel_scu_ipc.o
#include <linux/platform_device.h>
#include <linux/input/sparse-keymap.h>
#include <linux/acpi.h>
+#include <linux/suspend.h>
#include <acpi/acpi_bus.h>
MODULE_LICENSE("GPL");
struct intel_hid_priv {
struct input_dev *input_dev;
struct input_dev *array;
+ bool wakeup_mode;
};
static int intel_hid_set_enable(struct device *device, bool enable)
dev_warn(device, "failed to set button capability\n");
}
-static int intel_hid_pl_suspend_handler(struct device *device)
+static int intel_hid_pm_prepare(struct device *device)
{
- intel_hid_set_enable(device, false);
- intel_button_array_enable(device, false);
+ struct intel_hid_priv *priv = dev_get_drvdata(device);
+
+ priv->wakeup_mode = true;
+ return 0;
+}
+static int intel_hid_pl_suspend_handler(struct device *device)
+{
+ if (pm_suspend_via_firmware()) {
+ intel_hid_set_enable(device, false);
+ intel_button_array_enable(device, false);
+ }
return 0;
}
static int intel_hid_pl_resume_handler(struct device *device)
{
- intel_hid_set_enable(device, true);
- intel_button_array_enable(device, true);
+ struct intel_hid_priv *priv = dev_get_drvdata(device);
+ priv->wakeup_mode = false;
+ if (pm_resume_via_firmware()) {
+ intel_hid_set_enable(device, true);
+ intel_button_array_enable(device, true);
+ }
return 0;
}
static const struct dev_pm_ops intel_hid_pl_pm_ops = {
+ .prepare = intel_hid_pm_prepare,
.freeze = intel_hid_pl_suspend_handler,
.thaw = intel_hid_pl_resume_handler,
.restore = intel_hid_pl_resume_handler,
unsigned long long ev_index;
acpi_status status;
+ if (priv->wakeup_mode) {
+ /* Wake up on 5-button array events only. */
+ if (event == 0xc0 || !priv->array)
+ return;
+
+ if (sparse_keymap_entry_from_scancode(priv->array, event))
+ pm_wakeup_hard_event(&device->dev);
+ else
+ dev_info(&device->dev, "unknown event 0x%x\n", event);
+
+ return;
+ }
+
/* 0xC0 is for HID events, other values are for 5 button array */
if (event != 0xc0) {
if (!priv->array ||
"failed to enable HID power button\n");
}
+ device_init_wakeup(&device->dev, true);
return 0;
err_remove_notify:
#include <linux/platform_device.h>
#include <linux/input/sparse-keymap.h>
#include <linux/acpi.h>
+#include <linux/suspend.h>
#include <acpi/acpi_bus.h>
MODULE_LICENSE("GPL");
struct intel_vbtn_priv {
struct input_dev *input_dev;
+ bool wakeup_mode;
};
static int intel_vbtn_input_setup(struct platform_device *device)
struct platform_device *device = context;
struct intel_vbtn_priv *priv = dev_get_drvdata(&device->dev);
- if (!sparse_keymap_report_event(priv->input_dev, event, 1, true))
- dev_info(&device->dev, "unknown event index 0x%x\n",
- event);
+ if (priv->wakeup_mode) {
+ if (sparse_keymap_entry_from_scancode(priv->input_dev, event)) {
+ pm_wakeup_hard_event(&device->dev);
+ return;
+ }
+ } else if (sparse_keymap_report_event(priv->input_dev, event, 1, true)) {
+ return;
+ }
+ dev_info(&device->dev, "unknown event index 0x%x\n", event);
}
static int intel_vbtn_probe(struct platform_device *device)
if (ACPI_FAILURE(status))
return -EBUSY;
+ device_init_wakeup(&device->dev, true);
return 0;
}
return 0;
}
+static int intel_vbtn_pm_prepare(struct device *dev)
+{
+ struct intel_vbtn_priv *priv = dev_get_drvdata(dev);
+
+ priv->wakeup_mode = true;
+ return 0;
+}
+
+static int intel_vbtn_pm_resume(struct device *dev)
+{
+ struct intel_vbtn_priv *priv = dev_get_drvdata(dev);
+
+ priv->wakeup_mode = false;
+ return 0;
+}
+
+static const struct dev_pm_ops intel_vbtn_pm_ops = {
+ .prepare = intel_vbtn_pm_prepare,
+ .resume = intel_vbtn_pm_resume,
+ .restore = intel_vbtn_pm_resume,
+ .thaw = intel_vbtn_pm_resume,
+};
+
static struct platform_driver intel_vbtn_pl_driver = {
.driver = {
.name = "intel-vbtn",
.acpi_match_table = intel_vbtn_ids,
+ .pm = &intel_vbtn_pm_ops,
},
.probe = intel_vbtn_probe,
.remove = intel_vbtn_remove,
--- /dev/null
+/*
+ * Intel INT0002 "Virtual GPIO" driver
+ *
+ * Copyright (C) 2017 Hans de Goede <hdegoede@redhat.com>
+ *
+ * Loosely based on android x86 kernel code which is:
+ *
+ * Copyright (c) 2014, Intel Corporation.
+ *
+ * Author: Dyut Kumar Sil <dyut.k.sil@intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * Some peripherals on Bay Trail and Cherry Trail platforms signal a Power
+ * Management Event (PME) to the Power Management Controller (PMC) to wakeup
+ * the system. When this happens software needs to clear the PME bus 0 status
+ * bit in the GPE0a_STS register to avoid an IRQ storm on IRQ 9.
+ *
+ * This is modelled in ACPI through the INT0002 ACPI device, which is
+ * called a "Virtual GPIO controller" in ACPI because it defines the event
+ * handler to call when the PME triggers through _AEI and _L02 / _E02
+ * methods as would be done for a real GPIO interrupt in ACPI. Note this
+ * is a hack to define an AML event handler for the PME while using existing
+ * ACPI mechanisms, this is not a real GPIO at all.
+ *
+ * This driver will bind to the INT0002 device, and register as a GPIO
+ * controller, letting gpiolib-acpi.c call the _L02 handler as it would
+ * for a real GPIO controller.
+ */
+
+#include <linux/acpi.h>
+#include <linux/bitmap.h>
+#include <linux/gpio/driver.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+#include <linux/suspend.h>
+
+#include <asm/cpu_device_id.h>
+#include <asm/intel-family.h>
+
+#define DRV_NAME "INT0002 Virtual GPIO"
+
+/* For some reason the virtual GPIO pin tied to the GPE is numbered pin 2 */
+#define GPE0A_PME_B0_VIRT_GPIO_PIN 2
+
+#define GPE0A_PME_B0_STS_BIT BIT(13)
+#define GPE0A_PME_B0_EN_BIT BIT(13)
+#define GPE0A_STS_PORT 0x420
+#define GPE0A_EN_PORT 0x428
+
+#define ICPU(model) { X86_VENDOR_INTEL, 6, model, X86_FEATURE_ANY, }
+
+static const struct x86_cpu_id int0002_cpu_ids[] = {
+/*
+ * Limit ourselves to Cherry Trail for now, until testing shows we
+ * need to handle the INT0002 device on Baytrail too.
+ * ICPU(INTEL_FAM6_ATOM_SILVERMONT1), * Valleyview, Bay Trail *
+ */
+ ICPU(INTEL_FAM6_ATOM_AIRMONT), /* Braswell, Cherry Trail */
+ {}
+};
+
+/*
+ * As this is not a real GPIO at all, but just a hack to model an event in
+ * ACPI the get / set functions are dummy functions.
+ */
+
+static int int0002_gpio_get(struct gpio_chip *chip, unsigned int offset)
+{
+ return 0;
+}
+
+static void int0002_gpio_set(struct gpio_chip *chip, unsigned int offset,
+ int value)
+{
+}
+
+static int int0002_gpio_direction_output(struct gpio_chip *chip,
+ unsigned int offset, int value)
+{
+ return 0;
+}
+
+static void int0002_irq_ack(struct irq_data *data)
+{
+ outl(GPE0A_PME_B0_STS_BIT, GPE0A_STS_PORT);
+}
+
+static void int0002_irq_unmask(struct irq_data *data)
+{
+ u32 gpe_en_reg;
+
+ gpe_en_reg = inl(GPE0A_EN_PORT);
+ gpe_en_reg |= GPE0A_PME_B0_EN_BIT;
+ outl(gpe_en_reg, GPE0A_EN_PORT);
+}
+
+static void int0002_irq_mask(struct irq_data *data)
+{
+ u32 gpe_en_reg;
+
+ gpe_en_reg = inl(GPE0A_EN_PORT);
+ gpe_en_reg &= ~GPE0A_PME_B0_EN_BIT;
+ outl(gpe_en_reg, GPE0A_EN_PORT);
+}
+
+static irqreturn_t int0002_irq(int irq, void *data)
+{
+ struct gpio_chip *chip = data;
+ u32 gpe_sts_reg;
+
+ gpe_sts_reg = inl(GPE0A_STS_PORT);
+ if (!(gpe_sts_reg & GPE0A_PME_B0_STS_BIT))
+ return IRQ_NONE;
+
+ generic_handle_irq(irq_find_mapping(chip->irqdomain,
+ GPE0A_PME_B0_VIRT_GPIO_PIN));
+
+ pm_system_wakeup();
+
+ return IRQ_HANDLED;
+}
+
+static struct irq_chip int0002_irqchip = {
+ .name = DRV_NAME,
+ .irq_ack = int0002_irq_ack,
+ .irq_mask = int0002_irq_mask,
+ .irq_unmask = int0002_irq_unmask,
+};
+
+static int int0002_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ const struct x86_cpu_id *cpu_id;
+ struct gpio_chip *chip;
+ int irq, ret;
+
+ /* Menlow has a different INT0002 device? <sigh> */
+ cpu_id = x86_match_cpu(int0002_cpu_ids);
+ if (!cpu_id)
+ return -ENODEV;
+
+ irq = platform_get_irq(pdev, 0);
+ if (irq < 0) {
+ dev_err(dev, "Error getting IRQ: %d\n", irq);
+ return irq;
+ }
+
+ chip = devm_kzalloc(dev, sizeof(*chip), GFP_KERNEL);
+ if (!chip)
+ return -ENOMEM;
+
+ chip->label = DRV_NAME;
+ chip->parent = dev;
+ chip->owner = THIS_MODULE;
+ chip->get = int0002_gpio_get;
+ chip->set = int0002_gpio_set;
+ chip->direction_input = int0002_gpio_get;
+ chip->direction_output = int0002_gpio_direction_output;
+ chip->base = -1;
+ chip->ngpio = GPE0A_PME_B0_VIRT_GPIO_PIN + 1;
+ chip->irq_need_valid_mask = true;
+
+ ret = devm_gpiochip_add_data(&pdev->dev, chip, NULL);
+ if (ret) {
+ dev_err(dev, "Error adding gpio chip: %d\n", ret);
+ return ret;
+ }
+
+ bitmap_clear(chip->irq_valid_mask, 0, GPE0A_PME_B0_VIRT_GPIO_PIN);
+
+ /*
+ * We manually request the irq here instead of passing a flow-handler
+ * to gpiochip_set_chained_irqchip, because the irq is shared.
+ */
+ ret = devm_request_irq(dev, irq, int0002_irq,
+ IRQF_SHARED | IRQF_NO_THREAD, "INT0002", chip);
+ if (ret) {
+ dev_err(dev, "Error requesting IRQ %d: %d\n", irq, ret);
+ return ret;
+ }
+
+ ret = gpiochip_irqchip_add(chip, &int0002_irqchip, 0, handle_edge_irq,
+ IRQ_TYPE_NONE);
+ if (ret) {
+ dev_err(dev, "Error adding irqchip: %d\n", ret);
+ return ret;
+ }
+
+ gpiochip_set_chained_irqchip(chip, &int0002_irqchip, irq, NULL);
+
+ return 0;
+}
+
+static const struct acpi_device_id int0002_acpi_ids[] = {
+ { "INT0002", 0 },
+ { },
+};
+MODULE_DEVICE_TABLE(acpi, int0002_acpi_ids);
+
+static struct platform_driver int0002_driver = {
+ .driver = {
+ .name = DRV_NAME,
+ .acpi_match_table = int0002_acpi_ids,
+ },
+ .probe = int0002_probe,
+};
+
+module_platform_driver(int0002_driver);
+
+MODULE_AUTHOR("Hans de Goede <hdegoede@redhat.com>");
+MODULE_DESCRIPTION("Intel INT0002 Virtual GPIO driver");
+MODULE_LICENSE("GPL");
*/
/* interface_version --------------------------------------------------- */
-static ssize_t tpacpi_driver_interface_version_show(
- struct device_driver *drv,
- char *buf)
+static ssize_t interface_version_show(struct device_driver *drv, char *buf)
{
return snprintf(buf, PAGE_SIZE, "0x%08x\n", TPACPI_SYSFS_VERSION);
}
-
-static DRIVER_ATTR(interface_version, S_IRUGO,
- tpacpi_driver_interface_version_show, NULL);
+static DRIVER_ATTR_RO(interface_version);
/* debug_level --------------------------------------------------------- */
-static ssize_t tpacpi_driver_debug_show(struct device_driver *drv,
- char *buf)
+static ssize_t debug_level_show(struct device_driver *drv, char *buf)
{
return snprintf(buf, PAGE_SIZE, "0x%04x\n", dbg_level);
}
-static ssize_t tpacpi_driver_debug_store(struct device_driver *drv,
- const char *buf, size_t count)
+static ssize_t debug_level_store(struct device_driver *drv, const char *buf,
+ size_t count)
{
unsigned long t;
return count;
}
-
-static DRIVER_ATTR(debug_level, S_IWUSR | S_IRUGO,
- tpacpi_driver_debug_show, tpacpi_driver_debug_store);
+static DRIVER_ATTR_RW(debug_level);
/* version ------------------------------------------------------------- */
-static ssize_t tpacpi_driver_version_show(struct device_driver *drv,
- char *buf)
+static ssize_t version_show(struct device_driver *drv, char *buf)
{
return snprintf(buf, PAGE_SIZE, "%s v%s\n",
TPACPI_DESC, TPACPI_VERSION);
}
-
-static DRIVER_ATTR(version, S_IRUGO,
- tpacpi_driver_version_show, NULL);
+static DRIVER_ATTR_RO(version);
/* --------------------------------------------------------------------- */
#ifdef CONFIG_THINKPAD_ACPI_DEBUGFACILITIES
/* wlsw_emulstate ------------------------------------------------------ */
-static ssize_t tpacpi_driver_wlsw_emulstate_show(struct device_driver *drv,
- char *buf)
+static ssize_t wlsw_emulstate_show(struct device_driver *drv, char *buf)
{
return snprintf(buf, PAGE_SIZE, "%d\n", !!tpacpi_wlsw_emulstate);
}
-static ssize_t tpacpi_driver_wlsw_emulstate_store(struct device_driver *drv,
- const char *buf, size_t count)
+static ssize_t wlsw_emulstate_store(struct device_driver *drv, const char *buf,
+ size_t count)
{
unsigned long t;
return count;
}
-
-static DRIVER_ATTR(wlsw_emulstate, S_IWUSR | S_IRUGO,
- tpacpi_driver_wlsw_emulstate_show,
- tpacpi_driver_wlsw_emulstate_store);
+static DRIVER_ATTR_RW(wlsw_emulstate);
/* bluetooth_emulstate ------------------------------------------------- */
-static ssize_t tpacpi_driver_bluetooth_emulstate_show(
- struct device_driver *drv,
- char *buf)
+static ssize_t bluetooth_emulstate_show(struct device_driver *drv, char *buf)
{
return snprintf(buf, PAGE_SIZE, "%d\n", !!tpacpi_bluetooth_emulstate);
}
-static ssize_t tpacpi_driver_bluetooth_emulstate_store(
- struct device_driver *drv,
- const char *buf, size_t count)
+static ssize_t bluetooth_emulstate_store(struct device_driver *drv,
+ const char *buf, size_t count)
{
unsigned long t;
return count;
}
-
-static DRIVER_ATTR(bluetooth_emulstate, S_IWUSR | S_IRUGO,
- tpacpi_driver_bluetooth_emulstate_show,
- tpacpi_driver_bluetooth_emulstate_store);
+static DRIVER_ATTR_RW(bluetooth_emulstate);
/* wwan_emulstate ------------------------------------------------- */
-static ssize_t tpacpi_driver_wwan_emulstate_show(
- struct device_driver *drv,
- char *buf)
+static ssize_t wwan_emulstate_show(struct device_driver *drv, char *buf)
{
return snprintf(buf, PAGE_SIZE, "%d\n", !!tpacpi_wwan_emulstate);
}
-static ssize_t tpacpi_driver_wwan_emulstate_store(
- struct device_driver *drv,
- const char *buf, size_t count)
+static ssize_t wwan_emulstate_store(struct device_driver *drv, const char *buf,
+ size_t count)
{
unsigned long t;
return count;
}
-
-static DRIVER_ATTR(wwan_emulstate, S_IWUSR | S_IRUGO,
- tpacpi_driver_wwan_emulstate_show,
- tpacpi_driver_wwan_emulstate_store);
+static DRIVER_ATTR_RW(wwan_emulstate);
/* uwb_emulstate ------------------------------------------------- */
-static ssize_t tpacpi_driver_uwb_emulstate_show(
- struct device_driver *drv,
- char *buf)
+static ssize_t uwb_emulstate_show(struct device_driver *drv, char *buf)
{
return snprintf(buf, PAGE_SIZE, "%d\n", !!tpacpi_uwb_emulstate);
}
-static ssize_t tpacpi_driver_uwb_emulstate_store(
- struct device_driver *drv,
- const char *buf, size_t count)
+static ssize_t uwb_emulstate_store(struct device_driver *drv, const char *buf,
+ size_t count)
{
unsigned long t;
return count;
}
-
-static DRIVER_ATTR(uwb_emulstate, S_IWUSR | S_IRUGO,
- tpacpi_driver_uwb_emulstate_show,
- tpacpi_driver_uwb_emulstate_store);
+static DRIVER_ATTR_RW(uwb_emulstate);
#endif
/* --------------------------------------------------------------------- */
static DEVICE_ATTR(fan2_input, S_IRUGO, fan_fan2_input_show, NULL);
/* sysfs fan fan_watchdog (hwmon driver) ------------------------------- */
-static ssize_t fan_fan_watchdog_show(struct device_driver *drv,
- char *buf)
+static ssize_t fan_watchdog_show(struct device_driver *drv, char *buf)
{
return snprintf(buf, PAGE_SIZE, "%u\n", fan_watchdog_maxinterval);
}
-static ssize_t fan_fan_watchdog_store(struct device_driver *drv,
- const char *buf, size_t count)
+static ssize_t fan_watchdog_store(struct device_driver *drv, const char *buf,
+ size_t count)
{
unsigned long t;
return count;
}
-
-static DRIVER_ATTR(fan_watchdog, S_IWUSR | S_IRUGO,
- fan_fan_watchdog_show, fan_fan_watchdog_store);
+static DRIVER_ATTR_RW(fan_watchdog);
/* --------------------------------------------------------------------- */
static struct attribute *fan_attributes[] = {
}
if (device_can_wakeup(&dev->dev)) {
- error = acpi_pm_device_sleep_wake(&dev->dev,
- device_may_wakeup(&dev->dev));
+ error = acpi_pm_set_device_wakeup(&dev->dev,
+ device_may_wakeup(&dev->dev));
if (error)
return error;
}
}
if (device_may_wakeup(&dev->dev))
- acpi_pm_device_sleep_wake(&dev->dev, false);
+ acpi_pm_set_device_wakeup(&dev->dev, false);
if (acpi_device_power_manageable(acpi_dev))
error = acpi_device_set_power(acpi_dev, ACPI_STATE_D0);
},
};
+static const struct rockchip_iodomain_soc_data soc_data_rk3228 = {
+ .grf_offset = 0x418,
+ .supply_names = {
+ "vccio1",
+ "vccio2",
+ "vccio3",
+ "vccio4",
+ },
+};
+
static const struct rockchip_iodomain_soc_data soc_data_rk3288 = {
.grf_offset = 0x380,
.supply_names = {
.compatible = "rockchip,rk3188-io-voltage-domain",
.data = (void *)&soc_data_rk3188
},
+ {
+ .compatible = "rockchip,rk3228-io-voltage-domain",
+ .data = (void *)&soc_data_rk3228
+ },
{
.compatible = "rockchip,rk3288-io-voltage-domain",
.data = (void *)&soc_data_rk3288
#include <linux/platform_device.h>
#include <linux/power_supply.h>
-#include "../../w1/w1.h"
+#include <linux/w1.h>
#include "../../w1/slaves/w1_ds2760.h"
struct ds2760_device_info {
#include <linux/power_supply.h>
#include <linux/idr.h>
-#include "../../w1/w1.h"
+#include <linux/w1.h>
#include "../../w1/slaves/w1_ds2780.h"
/* Current unit measurement in uA for a 1 milli-ohm sense resistor */
#include <linux/power_supply.h>
#include <linux/idr.h>
-#include "../../w1/w1.h"
+#include <linux/w1.h>
#include "../../w1/slaves/w1_ds2781.h"
/* Current unit measurement in uA for a 1 milli-ohm sense resistor */
cpu = rd->rp->lead_cpu;
bits = rapl_unit_xlate(rd, rp->unit, value, 1);
- bits |= bits << rp->shift;
+ bits <<= rp->shift;
+ bits &= rp->mask;
+
memset(&ma, 0, sizeof(ma));
ma.msr_no = rd->msrs[rp->id];
# PPS support configuration
#
-menu "PPS support"
-
-config PPS
+menuconfig PPS
tristate "PPS support"
---help---
PPS (Pulse Per Second) is a special pulse provided by some GPS
To compile this driver as a module, choose M here: the module
will be called pps_core.ko.
-if PPS
config PPS_DEBUG
bool "PPS debugging messages"
+ depends on PPS
help
Say Y here if you want the PPS support to produce a bunch of debug
messages to the system log. Select this if you are having a
config NTP_PPS
bool "PPS kernel consumer support"
- depends on !NO_HZ_COMMON
+ depends on PPS && !NO_HZ_COMMON
help
This option adds support for direct in-kernel time
synchronization using an external PPS signal.
It doesn't work on tickless systems at the moment.
-endif
-
source drivers/pps/clients/Kconfig
source drivers/pps/generators/Kconfig
-
-endmenu
# PPS clients configuration
#
-if PPS
-
comment "PPS clients support"
+ depends on PPS
config PPS_CLIENT_KTIMER
tristate "Kernel timer client (Testing client, use for debug)"
+ depends on PPS
help
If you say yes here you get support for a PPS debugging client
which uses a kernel timer to generate the PPS signal.
GPIO. To be useful you must also register a platform device
specifying the GPIO pin and other options, usually in your board
setup.
-
-endif
#
comment "PPS generators support"
+ depends on PPS
config PPS_GENERATOR_PARPORT
tristate "Parallel port PPS signal generator"
- depends on PARPORT && BROKEN
+ depends on PPS && PARPORT && BROKEN
help
If you say yes here you get support for a PPS signal generator which
utilizes STROBE pin of a parallel port to send PPS signals. It uses
count = debugfs_create_file("count_threshold", S_IRUSR | S_IWUSR, d,
&count_threshold, &count_threshold_ops);
- if (!decay) {
+ if (!count) {
pr_warn("Error creating count_threshold debugfs node!\n");
goto err;
}
EXPORT_TRACEPOINT_SYMBOL_GPL(mc_event);
-int __init parse_ras_param(char *str)
+static int __init parse_ras_param(char *str)
{
#ifdef CONFIG_RAS_CEC
parse_cec_param(str);
will be called da9055-regulator.
config REGULATOR_DA9062
- tristate "Dialog Semiconductor DA9062 regulators"
+ tristate "Dialog Semiconductor DA9061/62 regulators"
depends on MFD_DA9062
help
Say y here to support the BUCKs and LDOs regulators found on
- DA9062 PMICs.
+ DA9061 and DA9062 PMICs.
This driver can also be built as a module. If so, the module
will be called da9062-regulator.
21 general purpose LDOs, 3 dedicated LDOs, and 5 BUCKs. All
of them come with support to either ECO (idle) or sleep mode.
+config REGULATOR_HI6421V530
+ tristate "HiSilicon Hi6421v530 PMIC voltage regulator support"
+ depends on MFD_HI6421_PMIC && OF
+ help
+ This driver provides support for the voltage regulators on
+ HiSilicon Hi6421v530 PMU / Codec IC.
+ Hi6421v530 is a multi-function device which, on regulator part,
+ provides 5 general purpose LDOs, and all of them come with support
+ to either ECO (idle) or sleep mode.
+
config REGULATOR_HI655X
tristate "Hisilicon HI655X PMIC regulators support"
depends on ARCH_HISI || COMPILE_TEST
chip contains six step-down DC/DC converters which can support
9 mode multiphase configuration.
+config REGULATOR_LP87565
+ tristate "TI LP87565 Power regulators"
+ depends on MFD_TI_LP87565 && OF
+ help
+ This driver supports LP87565 voltage regulator chips. LP87565
+ provides four step-down converters. It supports software based
+ voltage control for different voltage domains
+
config REGULATOR_LP8788
tristate "TI LP8788 Power Regulators"
depends on MFD_LP8788
obj-$(CONFIG_REGULATOR_FAN53555) += fan53555.o
obj-$(CONFIG_REGULATOR_GPIO) += gpio-regulator.o
obj-$(CONFIG_REGULATOR_HI6421) += hi6421-regulator.o
+obj-$(CONFIG_REGULATOR_HI6421V530) += hi6421v530-regulator.o
obj-$(CONFIG_REGULATOR_HI655X) += hi655x-regulator.o
obj-$(CONFIG_REGULATOR_ISL6271A) += isl6271a-regulator.o
obj-$(CONFIG_REGULATOR_ISL9305) += isl9305.o
obj-$(CONFIG_REGULATOR_LP3972) += lp3972.o
obj-$(CONFIG_REGULATOR_LP872X) += lp872x.o
obj-$(CONFIG_REGULATOR_LP873X) += lp873x-regulator.o
+obj-$(CONFIG_REGULATOR_LP87565) += lp87565-regulator.o
obj-$(CONFIG_REGULATOR_LP8788) += lp8788-buck.o
obj-$(CONFIG_REGULATOR_LP8788) += lp8788-ldo.o
obj-$(CONFIG_REGULATOR_LP8755) += lp8755.o
.ops = &axp20x_ops_sw,
};
+static const struct regulator_linear_range axp803_dcdc234_ranges[] = {
+ REGULATOR_LINEAR_RANGE(500000, 0x0, 0x46, 10000),
+ REGULATOR_LINEAR_RANGE(1220000, 0x47, 0x4b, 20000),
+};
+
+static const struct regulator_linear_range axp803_dcdc5_ranges[] = {
+ REGULATOR_LINEAR_RANGE(800000, 0x0, 0x20, 10000),
+ REGULATOR_LINEAR_RANGE(1140000, 0x21, 0x44, 20000),
+};
+
+static const struct regulator_linear_range axp803_dcdc6_ranges[] = {
+ REGULATOR_LINEAR_RANGE(600000, 0x0, 0x32, 10000),
+ REGULATOR_LINEAR_RANGE(1120000, 0x33, 0x47, 20000),
+};
+
+/* AXP806's CLDO2 and AXP809's DLDO1 shares the same range */
+static const struct regulator_linear_range axp803_dldo2_ranges[] = {
+ REGULATOR_LINEAR_RANGE(700000, 0x0, 0x1a, 100000),
+ REGULATOR_LINEAR_RANGE(3400000, 0x1b, 0x1f, 200000),
+};
+
+static const struct regulator_desc axp803_regulators[] = {
+ AXP_DESC(AXP803, DCDC1, "dcdc1", "vin1", 1600, 3400, 100,
+ AXP803_DCDC1_V_OUT, 0x1f, AXP22X_PWR_OUT_CTRL1, BIT(0)),
+ AXP_DESC_RANGES(AXP803, DCDC2, "dcdc2", "vin2", axp803_dcdc234_ranges,
+ 76, AXP803_DCDC2_V_OUT, 0x7f, AXP22X_PWR_OUT_CTRL1,
+ BIT(1)),
+ AXP_DESC_RANGES(AXP803, DCDC3, "dcdc3", "vin3", axp803_dcdc234_ranges,
+ 76, AXP803_DCDC3_V_OUT, 0x7f, AXP22X_PWR_OUT_CTRL1,
+ BIT(2)),
+ AXP_DESC_RANGES(AXP803, DCDC4, "dcdc4", "vin4", axp803_dcdc234_ranges,
+ 76, AXP803_DCDC4_V_OUT, 0x7f, AXP22X_PWR_OUT_CTRL1,
+ BIT(3)),
+ AXP_DESC_RANGES(AXP803, DCDC5, "dcdc5", "vin5", axp803_dcdc5_ranges,
+ 68, AXP803_DCDC5_V_OUT, 0x7f, AXP22X_PWR_OUT_CTRL1,
+ BIT(4)),
+ AXP_DESC_RANGES(AXP803, DCDC6, "dcdc6", "vin6", axp803_dcdc6_ranges,
+ 72, AXP803_DCDC6_V_OUT, 0x7f, AXP22X_PWR_OUT_CTRL1,
+ BIT(5)),
+ /* secondary switchable output of DCDC1 */
+ AXP_DESC_SW(AXP803, DC1SW, "dc1sw", NULL, AXP22X_PWR_OUT_CTRL2,
+ BIT(7)),
+ AXP_DESC(AXP803, ALDO1, "aldo1", "aldoin", 700, 3300, 100,
+ AXP22X_ALDO1_V_OUT, 0x1f, AXP22X_PWR_OUT_CTRL3, BIT(5)),
+ AXP_DESC(AXP803, ALDO2, "aldo2", "aldoin", 700, 3300, 100,
+ AXP22X_ALDO2_V_OUT, 0x1f, AXP22X_PWR_OUT_CTRL3, BIT(6)),
+ AXP_DESC(AXP803, ALDO3, "aldo3", "aldoin", 700, 3300, 100,
+ AXP22X_ALDO3_V_OUT, 0x1f, AXP22X_PWR_OUT_CTRL3, BIT(7)),
+ AXP_DESC(AXP803, DLDO1, "dldo1", "dldoin", 700, 3300, 100,
+ AXP22X_DLDO1_V_OUT, 0x1f, AXP22X_PWR_OUT_CTRL2, BIT(3)),
+ AXP_DESC_RANGES(AXP803, DLDO2, "dldo2", "dldoin", axp803_dldo2_ranges,
+ 32, AXP22X_DLDO2_V_OUT, 0x1f, AXP22X_PWR_OUT_CTRL2,
+ BIT(4)),
+ AXP_DESC(AXP803, DLDO3, "dldo3", "dldoin", 700, 3300, 100,
+ AXP22X_DLDO3_V_OUT, 0x1f, AXP22X_PWR_OUT_CTRL2, BIT(5)),
+ AXP_DESC(AXP803, DLDO4, "dldo4", "dldoin", 700, 3300, 100,
+ AXP22X_DLDO4_V_OUT, 0x1f, AXP22X_PWR_OUT_CTRL2, BIT(6)),
+ AXP_DESC(AXP803, ELDO1, "eldo1", "eldoin", 700, 1900, 50,
+ AXP22X_ELDO1_V_OUT, 0x1f, AXP22X_PWR_OUT_CTRL2, BIT(0)),
+ AXP_DESC(AXP803, ELDO2, "eldo2", "eldoin", 700, 1900, 50,
+ AXP22X_ELDO2_V_OUT, 0x1f, AXP22X_PWR_OUT_CTRL2, BIT(1)),
+ AXP_DESC(AXP803, ELDO3, "eldo3", "eldoin", 700, 1900, 50,
+ AXP22X_ELDO3_V_OUT, 0x1f, AXP22X_PWR_OUT_CTRL2, BIT(2)),
+ AXP_DESC(AXP803, FLDO1, "fldo1", "fldoin", 700, 1450, 50,
+ AXP803_FLDO1_V_OUT, 0x0f, AXP22X_PWR_OUT_CTRL3, BIT(2)),
+ AXP_DESC(AXP803, FLDO2, "fldo2", "fldoin", 700, 1450, 50,
+ AXP803_FLDO2_V_OUT, 0x0f, AXP22X_PWR_OUT_CTRL3, BIT(3)),
+ AXP_DESC_IO(AXP803, LDO_IO0, "ldo-io0", "ips", 700, 3300, 100,
+ AXP22X_LDO_IO0_V_OUT, 0x1f, AXP20X_GPIO0_CTRL, 0x07,
+ AXP22X_IO_ENABLED, AXP22X_IO_DISABLED),
+ AXP_DESC_IO(AXP803, LDO_IO1, "ldo-io1", "ips", 700, 3300, 100,
+ AXP22X_LDO_IO1_V_OUT, 0x1f, AXP20X_GPIO1_CTRL, 0x07,
+ AXP22X_IO_ENABLED, AXP22X_IO_DISABLED),
+ AXP_DESC_FIXED(AXP803, RTC_LDO, "rtc-ldo", "ips", 3000),
+};
+
static const struct regulator_linear_range axp806_dcdca_ranges[] = {
REGULATOR_LINEAR_RANGE(600000, 0x0, 0x32, 10000),
REGULATOR_LINEAR_RANGE(1120000, 0x33, 0x47, 20000),
REGULATOR_LINEAR_RANGE(1600000, 0x2e, 0x3f, 100000),
};
-static const struct regulator_linear_range axp806_cldo2_ranges[] = {
- REGULATOR_LINEAR_RANGE(700000, 0x0, 0x1a, 100000),
- REGULATOR_LINEAR_RANGE(3400000, 0x1b, 0x1f, 200000),
-};
-
static const struct regulator_desc axp806_regulators[] = {
AXP_DESC_RANGES(AXP806, DCDCA, "dcdca", "vina", axp806_dcdca_ranges,
72, AXP806_DCDCA_V_CTRL, 0x7f, AXP806_PWR_OUT_CTRL1,
AXP806_BLDO4_V_CTRL, 0x0f, AXP806_PWR_OUT_CTRL2, BIT(3)),
AXP_DESC(AXP806, CLDO1, "cldo1", "cldoin", 700, 3300, 100,
AXP806_CLDO1_V_CTRL, 0x1f, AXP806_PWR_OUT_CTRL2, BIT(4)),
- AXP_DESC_RANGES(AXP806, CLDO2, "cldo2", "cldoin", axp806_cldo2_ranges,
+ AXP_DESC_RANGES(AXP806, CLDO2, "cldo2", "cldoin", axp803_dldo2_ranges,
32, AXP806_CLDO2_V_CTRL, 0x1f, AXP806_PWR_OUT_CTRL2,
BIT(5)),
AXP_DESC(AXP806, CLDO3, "cldo3", "cldoin", 700, 3300, 100,
AXP22X_ALDO2_V_OUT, 0x1f, AXP22X_PWR_OUT_CTRL1, BIT(7)),
AXP_DESC(AXP809, ALDO3, "aldo3", "aldoin", 700, 3300, 100,
AXP22X_ALDO3_V_OUT, 0x1f, AXP22X_PWR_OUT_CTRL2, BIT(5)),
- AXP_DESC_RANGES(AXP809, DLDO1, "dldo1", "dldoin", axp806_cldo2_ranges,
+ AXP_DESC_RANGES(AXP809, DLDO1, "dldo1", "dldoin", axp803_dldo2_ranges,
32, AXP22X_DLDO1_V_OUT, 0x1f, AXP22X_PWR_OUT_CTRL2,
BIT(3)),
AXP_DESC(AXP809, DLDO2, "dldo2", "dldoin", 700, 3300, 100,
def = 1500;
step = 75;
break;
- case AXP806_ID:
+ case AXP803_ID:
/*
- * AXP806 DCDC work frequency setting has the same range and
+ * AXP803 DCDC work frequency setting has the same range and
* step as AXP22X, but at a different register.
* Fall through to the check below.
* (See include/linux/mfd/axp20x.h)
*/
- reg = AXP806_DCDC_FREQ_CTRL;
+ reg = AXP803_DCDC_FREQ_CTRL;
+ case AXP806_ID:
+ /*
+ * AXP806 also have DCDC work frequency setting register at a
+ * different position.
+ */
+ if (axp20x->variant == AXP806_ID)
+ reg = AXP806_DCDC_FREQ_CTRL;
case AXP221_ID:
case AXP223_ID:
case AXP809_ID:
workmode <<= id - AXP22X_DCDC1;
break;
+ case AXP803_ID:
+ if (id < AXP803_DCDC1 || id > AXP803_DCDC6)
+ return -EINVAL;
+
+ mask = AXP22X_WORKMODE_DCDCX_MASK(id - AXP803_DCDC1);
+ workmode <<= id - AXP803_DCDC1;
+ break;
+
default:
/* should not happen */
WARN_ON(1);
{
u32 reg = 0;
- /* Only AXP806 has poly-phase outputs */
- if (axp20x->variant != AXP806_ID)
- return false;
+ /*
+ * Currently in our supported AXP variants, only AXP803 and AXP806
+ * have polyphase regulators.
+ */
+ switch (axp20x->variant) {
+ case AXP803_ID:
+ regmap_read(axp20x->regmap, AXP803_POLYPHASE_CTRL, ®);
+
+ switch (id) {
+ case AXP803_DCDC3:
+ return !!(reg & BIT(6));
+ case AXP803_DCDC6:
+ return !!(reg & BIT(7));
+ }
+ break;
- regmap_read(axp20x->regmap, AXP806_DCDC_MODE_CTRL2, ®);
+ case AXP806_ID:
+ regmap_read(axp20x->regmap, AXP806_DCDC_MODE_CTRL2, ®);
+
+ switch (id) {
+ case AXP806_DCDCB:
+ return (((reg & GENMASK(7, 6)) == BIT(6)) ||
+ ((reg & GENMASK(7, 6)) == BIT(7)));
+ case AXP806_DCDCC:
+ return ((reg & GENMASK(7, 6)) == BIT(7));
+ case AXP806_DCDCE:
+ return !!(reg & BIT(5));
+ }
+ break;
- switch (id) {
- case AXP806_DCDCB:
- return (((reg & GENMASK(7, 6)) == BIT(6)) ||
- ((reg & GENMASK(7, 6)) == BIT(7)));
- case AXP806_DCDCC:
- return ((reg & GENMASK(7, 6)) == BIT(7));
- case AXP806_DCDCE:
- return !!(reg & BIT(5));
+ default:
+ return false;
}
return false;
drivevbus = of_property_read_bool(pdev->dev.parent->of_node,
"x-powers,drive-vbus-en");
break;
+ case AXP803_ID:
+ regulators = axp803_regulators;
+ nregulators = AXP803_REG_ID_MAX;
+ break;
case AXP806_ID:
regulators = axp806_regulators;
nregulators = AXP806_REG_ID_MAX;
* name.
*/
if ((regulators == axp22x_regulators && i == AXP22X_DC1SW) ||
+ (regulators == axp803_regulators && i == AXP803_DC1SW) ||
(regulators == axp809_regulators && i == AXP809_DC1SW)) {
new_desc = devm_kzalloc(&pdev->dev, sizeof(*desc),
GFP_KERNEL);
.linear_min_sel = _lmin, \
}
-int bd9571mwv_avs_get_moni_state(struct regulator_dev *rdev)
+static int bd9571mwv_avs_get_moni_state(struct regulator_dev *rdev)
{
unsigned int val;
int ret;
return val & BD9571MWV_AVS_SET_MONI_MASK;
}
-int bd9571mwv_avs_set_voltage_sel_regmap(struct regulator_dev *rdev,
- unsigned int sel)
+static int bd9571mwv_avs_set_voltage_sel_regmap(struct regulator_dev *rdev,
+ unsigned int sel)
{
int ret;
rdev->desc->vsel_mask, sel);
}
-int bd9571mwv_avs_get_voltage_sel_regmap(struct regulator_dev *rdev)
+static int bd9571mwv_avs_get_voltage_sel_regmap(struct regulator_dev *rdev)
{
unsigned int val;
int ret;
return val;
}
-int bd9571mwv_reg_set_voltage_sel_regmap(struct regulator_dev *rdev,
- unsigned int sel)
+static int bd9571mwv_reg_set_voltage_sel_regmap(struct regulator_dev *rdev,
+ unsigned int sel)
{
return regmap_write_bits(rdev->regmap, BD9571MWV_DVFS_SETVID,
rdev->desc->vsel_mask, sel);
static struct regulator_dev *regulator_dev_lookup(struct device *dev,
const char *supply)
{
- struct regulator_dev *r;
+ struct regulator_dev *r = NULL;
struct device_node *node;
struct regulator_map *map;
const char *devname = NULL;
if (dev)
devname = dev_name(dev);
- r = regulator_lookup_by_name(supply);
- if (r)
- return r;
-
mutex_lock(®ulator_list_mutex);
list_for_each_entry(map, ®ulator_map_list, list) {
/* If the mapping has a device set up it must match */
}
mutex_unlock(®ulator_list_mutex);
+ if (r)
+ return r;
+
+ r = regulator_lookup_by_name(supply);
if (r)
return r;
ramp_delay = rdev->desc->ramp_delay;
else if (rdev->constraints->settling_time)
return rdev->constraints->settling_time;
+ else if (rdev->constraints->settling_time_up &&
+ (new_uV > old_uV))
+ return rdev->constraints->settling_time_up;
+ else if (rdev->constraints->settling_time_down &&
+ (new_uV < old_uV))
+ return rdev->constraints->settling_time_down;
if (ramp_delay == 0) {
rdev_dbg(rdev, "ramp_delay not set\n");
if (rdev->supply &&
regulator_ops_is_valid(rdev->supply->rdev,
REGULATOR_CHANGE_VOLTAGE) &&
- (rdev->desc->min_dropout_uV || !rdev->desc->ops->get_voltage)) {
+ (rdev->desc->min_dropout_uV || !(rdev->desc->ops->get_voltage ||
+ rdev->desc->ops->get_voltage_sel))) {
int current_supply_uV;
int selector;
EXPORT_SYMBOL_GPL(regulator_get_init_drvdata);
#ifdef CONFIG_DEBUG_FS
-static ssize_t supply_map_read_file(struct file *file, char __user *user_buf,
- size_t count, loff_t *ppos)
+static int supply_map_show(struct seq_file *sf, void *data)
{
- char *buf = kmalloc(PAGE_SIZE, GFP_KERNEL);
- ssize_t len, ret = 0;
struct regulator_map *map;
- if (!buf)
- return -ENOMEM;
-
list_for_each_entry(map, ®ulator_map_list, list) {
- len = snprintf(buf + ret, PAGE_SIZE - ret,
- "%s -> %s.%s\n",
- rdev_get_name(map->regulator), map->dev_name,
- map->supply);
- if (len >= 0)
- ret += len;
- if (ret > PAGE_SIZE) {
- ret = PAGE_SIZE;
- break;
- }
+ seq_printf(sf, "%s -> %s.%s\n",
+ rdev_get_name(map->regulator), map->dev_name,
+ map->supply);
}
- ret = simple_read_from_buffer(user_buf, count, ppos, buf, ret);
-
- kfree(buf);
+ return 0;
+}
- return ret;
+static int supply_map_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, supply_map_show, inode->i_private);
}
#endif
static const struct file_operations supply_map_fops = {
#ifdef CONFIG_DEBUG_FS
- .read = supply_map_read_file,
- .llseek = default_llseek,
+ .open = supply_map_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
#endif
};
/*
- * da9062-regulator.c - REGULATOR device driver for DA9062
- * Copyright (C) 2015 Dialog Semiconductor Ltd.
+ * Regulator device driver for DA9061 and DA9062.
+ * Copyright (C) 2015-2017 Dialog Semiconductor
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
#include <linux/mfd/da9062/registers.h>
/* Regulator IDs */
+enum {
+ DA9061_ID_BUCK1,
+ DA9061_ID_BUCK2,
+ DA9061_ID_BUCK3,
+ DA9061_ID_LDO1,
+ DA9061_ID_LDO2,
+ DA9061_ID_LDO3,
+ DA9061_ID_LDO4,
+ DA9061_MAX_REGULATORS,
+};
+
enum {
DA9062_ID_BUCK1,
DA9062_ID_BUCK2,
/* Regulator operations */
-/* Current limits array (in uA) BUCK1 and BUCK3.
- Entry indexes corresponds to register values. */
+/* Current limits array (in uA)
+ * - DA9061_ID_[BUCK1|BUCK3]
+ * - DA9062_ID_[BUCK1|BUCK2|BUCK4]
+ * Entry indexes corresponds to register values.
+ */
static const int da9062_buck_a_limits[] = {
500000, 600000, 700000, 800000, 900000, 1000000, 1100000, 1200000,
1300000, 1400000, 1500000, 1600000, 1700000, 1800000, 1900000, 2000000
};
-/* Current limits array (in uA) for BUCK2.
- Entry indexes corresponds to register values. */
+/* Current limits array (in uA)
+ * - DA9061_ID_BUCK2
+ * - DA9062_ID_BUCK3
+ * Entry indexes corresponds to register values.
+ */
static const int da9062_buck_b_limits[] = {
1500000, 1600000, 1700000, 1800000, 1900000, 2000000, 2100000, 2200000,
2300000, 2400000, 2500000, 2600000, 2700000, 2800000, 2900000, 3000000
.set_suspend_mode = da9062_ldo_set_suspend_mode,
};
-/* Regulator information */
-static const struct da9062_regulator_info local_regulator_info[] = {
+/* DA9061 Regulator information */
+static const struct da9062_regulator_info local_da9061_regulator_info[] = {
+ {
+ .desc.id = DA9061_ID_BUCK1,
+ .desc.name = "DA9061 BUCK1",
+ .desc.of_match = of_match_ptr("buck1"),
+ .desc.regulators_node = of_match_ptr("regulators"),
+ .desc.ops = &da9062_buck_ops,
+ .desc.min_uV = (300) * 1000,
+ .desc.uV_step = (10) * 1000,
+ .desc.n_voltages = ((1570) - (300))/(10) + 1,
+ .current_limits = da9062_buck_a_limits,
+ .n_current_limits = ARRAY_SIZE(da9062_buck_a_limits),
+ .desc.enable_reg = DA9062AA_BUCK1_CONT,
+ .desc.enable_mask = DA9062AA_BUCK1_EN_MASK,
+ .desc.vsel_reg = DA9062AA_VBUCK1_A,
+ .desc.vsel_mask = DA9062AA_VBUCK1_A_MASK,
+ .desc.linear_min_sel = 0,
+ .sleep = REG_FIELD(DA9062AA_VBUCK1_A,
+ __builtin_ffs((int)DA9062AA_BUCK1_SL_A_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_BUCK1_SL_A_MASK)) - 1),
+ .suspend_sleep = REG_FIELD(DA9062AA_VBUCK1_B,
+ __builtin_ffs((int)DA9062AA_BUCK1_SL_B_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_BUCK1_SL_B_MASK)) - 1),
+ .suspend_vsel_reg = DA9062AA_VBUCK1_B,
+ .mode = REG_FIELD(DA9062AA_BUCK1_CFG,
+ __builtin_ffs((int)DA9062AA_BUCK1_MODE_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_BUCK1_MODE_MASK)) - 1),
+ .suspend = REG_FIELD(DA9062AA_DVC_1,
+ __builtin_ffs((int)DA9062AA_VBUCK1_SEL_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_VBUCK1_SEL_MASK)) - 1),
+ .ilimit = REG_FIELD(DA9062AA_BUCK_ILIM_C,
+ __builtin_ffs((int)DA9062AA_BUCK1_ILIM_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_BUCK1_ILIM_MASK)) - 1),
+ },
+ {
+ .desc.id = DA9061_ID_BUCK2,
+ .desc.name = "DA9061 BUCK2",
+ .desc.of_match = of_match_ptr("buck2"),
+ .desc.regulators_node = of_match_ptr("regulators"),
+ .desc.ops = &da9062_buck_ops,
+ .desc.min_uV = (800) * 1000,
+ .desc.uV_step = (20) * 1000,
+ .desc.n_voltages = ((3340) - (800))/(20) + 1,
+ .current_limits = da9062_buck_b_limits,
+ .n_current_limits = ARRAY_SIZE(da9062_buck_b_limits),
+ .desc.enable_reg = DA9062AA_BUCK3_CONT,
+ .desc.enable_mask = DA9062AA_BUCK3_EN_MASK,
+ .desc.vsel_reg = DA9062AA_VBUCK3_A,
+ .desc.vsel_mask = DA9062AA_VBUCK3_A_MASK,
+ .desc.linear_min_sel = 0,
+ .sleep = REG_FIELD(DA9062AA_VBUCK3_A,
+ __builtin_ffs((int)DA9062AA_BUCK3_SL_A_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_BUCK3_SL_A_MASK)) - 1),
+ .suspend_sleep = REG_FIELD(DA9062AA_VBUCK3_B,
+ __builtin_ffs((int)DA9062AA_BUCK3_SL_B_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_BUCK3_SL_B_MASK)) - 1),
+ .suspend_vsel_reg = DA9062AA_VBUCK3_B,
+ .mode = REG_FIELD(DA9062AA_BUCK3_CFG,
+ __builtin_ffs((int)DA9062AA_BUCK3_MODE_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_BUCK3_MODE_MASK)) - 1),
+ .suspend = REG_FIELD(DA9062AA_DVC_1,
+ __builtin_ffs((int)DA9062AA_VBUCK3_SEL_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_VBUCK3_SEL_MASK)) - 1),
+ .ilimit = REG_FIELD(DA9062AA_BUCK_ILIM_A,
+ __builtin_ffs((int)DA9062AA_BUCK3_ILIM_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_BUCK3_ILIM_MASK)) - 1),
+ },
+ {
+ .desc.id = DA9061_ID_BUCK3,
+ .desc.name = "DA9061 BUCK3",
+ .desc.of_match = of_match_ptr("buck3"),
+ .desc.regulators_node = of_match_ptr("regulators"),
+ .desc.ops = &da9062_buck_ops,
+ .desc.min_uV = (530) * 1000,
+ .desc.uV_step = (10) * 1000,
+ .desc.n_voltages = ((1800) - (530))/(10) + 1,
+ .current_limits = da9062_buck_a_limits,
+ .n_current_limits = ARRAY_SIZE(da9062_buck_a_limits),
+ .desc.enable_reg = DA9062AA_BUCK4_CONT,
+ .desc.enable_mask = DA9062AA_BUCK4_EN_MASK,
+ .desc.vsel_reg = DA9062AA_VBUCK4_A,
+ .desc.vsel_mask = DA9062AA_VBUCK4_A_MASK,
+ .desc.linear_min_sel = 0,
+ .sleep = REG_FIELD(DA9062AA_VBUCK4_A,
+ __builtin_ffs((int)DA9062AA_BUCK4_SL_A_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_BUCK4_SL_A_MASK)) - 1),
+ .suspend_sleep = REG_FIELD(DA9062AA_VBUCK4_B,
+ __builtin_ffs((int)DA9062AA_BUCK4_SL_B_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_BUCK4_SL_B_MASK)) - 1),
+ .suspend_vsel_reg = DA9062AA_VBUCK4_B,
+ .mode = REG_FIELD(DA9062AA_BUCK4_CFG,
+ __builtin_ffs((int)DA9062AA_BUCK4_MODE_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_BUCK4_MODE_MASK)) - 1),
+ .suspend = REG_FIELD(DA9062AA_DVC_1,
+ __builtin_ffs((int)DA9062AA_VBUCK4_SEL_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_VBUCK4_SEL_MASK)) - 1),
+ .ilimit = REG_FIELD(DA9062AA_BUCK_ILIM_B,
+ __builtin_ffs((int)DA9062AA_BUCK4_ILIM_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_BUCK4_ILIM_MASK)) - 1),
+ },
+ {
+ .desc.id = DA9061_ID_LDO1,
+ .desc.name = "DA9061 LDO1",
+ .desc.of_match = of_match_ptr("ldo1"),
+ .desc.regulators_node = of_match_ptr("regulators"),
+ .desc.ops = &da9062_ldo_ops,
+ .desc.min_uV = (900) * 1000,
+ .desc.uV_step = (50) * 1000,
+ .desc.n_voltages = ((3600) - (900))/(50) + 1,
+ .desc.enable_reg = DA9062AA_LDO1_CONT,
+ .desc.enable_mask = DA9062AA_LDO1_EN_MASK,
+ .desc.vsel_reg = DA9062AA_VLDO1_A,
+ .desc.vsel_mask = DA9062AA_VLDO1_A_MASK,
+ .desc.linear_min_sel = 0,
+ .sleep = REG_FIELD(DA9062AA_VLDO1_A,
+ __builtin_ffs((int)DA9062AA_LDO1_SL_A_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_LDO1_SL_A_MASK)) - 1),
+ .suspend_sleep = REG_FIELD(DA9062AA_VLDO1_B,
+ __builtin_ffs((int)DA9062AA_LDO1_SL_B_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_LDO1_SL_B_MASK)) - 1),
+ .suspend_vsel_reg = DA9062AA_VLDO1_B,
+ .suspend = REG_FIELD(DA9062AA_DVC_1,
+ __builtin_ffs((int)DA9062AA_VLDO1_SEL_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_VLDO1_SEL_MASK)) - 1),
+ .oc_event = REG_FIELD(DA9062AA_STATUS_D,
+ __builtin_ffs((int)DA9062AA_LDO1_ILIM_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_LDO1_ILIM_MASK)) - 1),
+ },
+ {
+ .desc.id = DA9061_ID_LDO2,
+ .desc.name = "DA9061 LDO2",
+ .desc.of_match = of_match_ptr("ldo2"),
+ .desc.regulators_node = of_match_ptr("regulators"),
+ .desc.ops = &da9062_ldo_ops,
+ .desc.min_uV = (900) * 1000,
+ .desc.uV_step = (50) * 1000,
+ .desc.n_voltages = ((3600) - (600))/(50) + 1,
+ .desc.enable_reg = DA9062AA_LDO2_CONT,
+ .desc.enable_mask = DA9062AA_LDO2_EN_MASK,
+ .desc.vsel_reg = DA9062AA_VLDO2_A,
+ .desc.vsel_mask = DA9062AA_VLDO2_A_MASK,
+ .desc.linear_min_sel = 0,
+ .sleep = REG_FIELD(DA9062AA_VLDO2_A,
+ __builtin_ffs((int)DA9062AA_LDO2_SL_A_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_LDO2_SL_A_MASK)) - 1),
+ .suspend_sleep = REG_FIELD(DA9062AA_VLDO2_B,
+ __builtin_ffs((int)DA9062AA_LDO2_SL_B_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_LDO2_SL_B_MASK)) - 1),
+ .suspend_vsel_reg = DA9062AA_VLDO2_B,
+ .suspend = REG_FIELD(DA9062AA_DVC_1,
+ __builtin_ffs((int)DA9062AA_VLDO2_SEL_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_VLDO2_SEL_MASK)) - 1),
+ .oc_event = REG_FIELD(DA9062AA_STATUS_D,
+ __builtin_ffs((int)DA9062AA_LDO2_ILIM_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_LDO2_ILIM_MASK)) - 1),
+ },
+ {
+ .desc.id = DA9061_ID_LDO3,
+ .desc.name = "DA9061 LDO3",
+ .desc.of_match = of_match_ptr("ldo3"),
+ .desc.regulators_node = of_match_ptr("regulators"),
+ .desc.ops = &da9062_ldo_ops,
+ .desc.min_uV = (900) * 1000,
+ .desc.uV_step = (50) * 1000,
+ .desc.n_voltages = ((3600) - (900))/(50) + 1,
+ .desc.enable_reg = DA9062AA_LDO3_CONT,
+ .desc.enable_mask = DA9062AA_LDO3_EN_MASK,
+ .desc.vsel_reg = DA9062AA_VLDO3_A,
+ .desc.vsel_mask = DA9062AA_VLDO3_A_MASK,
+ .desc.linear_min_sel = 0,
+ .sleep = REG_FIELD(DA9062AA_VLDO3_A,
+ __builtin_ffs((int)DA9062AA_LDO3_SL_A_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_LDO3_SL_A_MASK)) - 1),
+ .suspend_sleep = REG_FIELD(DA9062AA_VLDO3_B,
+ __builtin_ffs((int)DA9062AA_LDO3_SL_B_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_LDO3_SL_B_MASK)) - 1),
+ .suspend_vsel_reg = DA9062AA_VLDO3_B,
+ .suspend = REG_FIELD(DA9062AA_DVC_1,
+ __builtin_ffs((int)DA9062AA_VLDO3_SEL_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_VLDO3_SEL_MASK)) - 1),
+ .oc_event = REG_FIELD(DA9062AA_STATUS_D,
+ __builtin_ffs((int)DA9062AA_LDO3_ILIM_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_LDO3_ILIM_MASK)) - 1),
+ },
+ {
+ .desc.id = DA9061_ID_LDO4,
+ .desc.name = "DA9061 LDO4",
+ .desc.of_match = of_match_ptr("ldo4"),
+ .desc.regulators_node = of_match_ptr("regulators"),
+ .desc.ops = &da9062_ldo_ops,
+ .desc.min_uV = (900) * 1000,
+ .desc.uV_step = (50) * 1000,
+ .desc.n_voltages = ((3600) - (900))/(50) + 1,
+ .desc.enable_reg = DA9062AA_LDO4_CONT,
+ .desc.enable_mask = DA9062AA_LDO4_EN_MASK,
+ .desc.vsel_reg = DA9062AA_VLDO4_A,
+ .desc.vsel_mask = DA9062AA_VLDO4_A_MASK,
+ .desc.linear_min_sel = 0,
+ .sleep = REG_FIELD(DA9062AA_VLDO4_A,
+ __builtin_ffs((int)DA9062AA_LDO4_SL_A_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_LDO4_SL_A_MASK)) - 1),
+ .suspend_sleep = REG_FIELD(DA9062AA_VLDO4_B,
+ __builtin_ffs((int)DA9062AA_LDO4_SL_B_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_LDO4_SL_B_MASK)) - 1),
+ .suspend_vsel_reg = DA9062AA_VLDO4_B,
+ .suspend = REG_FIELD(DA9062AA_DVC_1,
+ __builtin_ffs((int)DA9062AA_VLDO4_SEL_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_VLDO4_SEL_MASK)) - 1),
+ .oc_event = REG_FIELD(DA9062AA_STATUS_D,
+ __builtin_ffs((int)DA9062AA_LDO4_ILIM_MASK) - 1,
+ sizeof(unsigned int) * 8 -
+ __builtin_clz((DA9062AA_LDO4_ILIM_MASK)) - 1),
+ },
+};
+
+/* DA9062 Regulator information */
+static const struct da9062_regulator_info local_da9062_regulator_info[] = {
{
.desc.id = DA9062_ID_BUCK1,
.desc.name = "DA9062 BUCK1",
struct da9062_regulators *regulators;
struct da9062_regulator *regl;
struct regulator_config config = { };
+ const struct da9062_regulator_info *rinfo;
int irq, n, ret;
size_t size;
+ int max_regulators;
+
+ switch (chip->chip_type) {
+ case COMPAT_TYPE_DA9061:
+ max_regulators = DA9061_MAX_REGULATORS;
+ rinfo = local_da9061_regulator_info;
+ break;
+ case COMPAT_TYPE_DA9062:
+ max_regulators = DA9062_MAX_REGULATORS;
+ rinfo = local_da9062_regulator_info;
+ break;
+ default:
+ dev_err(chip->dev, "Unrecognised chip type\n");
+ return -ENODEV;
+ }
/* Allocate memory required by usable regulators */
size = sizeof(struct da9062_regulators) +
- DA9062_MAX_REGULATORS * sizeof(struct da9062_regulator);
+ max_regulators * sizeof(struct da9062_regulator);
regulators = devm_kzalloc(&pdev->dev, size, GFP_KERNEL);
if (!regulators)
return -ENOMEM;
- regulators->n_regulators = DA9062_MAX_REGULATORS;
+ regulators->n_regulators = max_regulators;
platform_set_drvdata(pdev, regulators);
n = 0;
/* Initialise regulator structure */
regl = ®ulators->regulator[n];
regl->hw = chip;
- regl->info = &local_regulator_info[n];
+ regl->info = &rinfo[n];
regl->desc = regl->info->desc;
regl->desc.type = REGULATOR_VOLTAGE;
regl->desc.owner = THIS_MODULE;
/* Module information */
MODULE_AUTHOR("S Twiss <stwiss.opensource@diasemi.com>");
-MODULE_DESCRIPTION("REGULATOR device driver for Dialog DA9062");
+MODULE_DESCRIPTION("REGULATOR device driver for Dialog DA9062 and DA9061");
MODULE_LICENSE("GPL");
MODULE_ALIAS("platform:da9062-regulators");
return 0;
}
+static const struct platform_device_id hi6421_regulator_table[] = {
+ { .name = "hi6421-regulator" },
+ {},
+};
+MODULE_DEVICE_TABLE(platform, hi6421_regulator_table);
+
static struct platform_driver hi6421_regulator_driver = {
+ .id_table = hi6421_regulator_table,
.driver = {
.name = "hi6421-regulator",
},
--- /dev/null
+/*
+ * Device driver for regulators in Hi6421V530 IC
+ *
+ * Copyright (c) <2017> HiSilicon Technologies Co., Ltd.
+ * http://www.hisilicon.com
+ * Copyright (c) <2017> Linaro Ltd.
+ * http://www.linaro.org
+ *
+ * Author: Wang Xiaoyin <hw.wangxiaoyin@hisilicon.com>
+ * Guodong Xu <guodong.xu@linaro.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/mfd/hi6421-pmic.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+#include <linux/regulator/driver.h>
+
+/*
+ * struct hi6421v530_regulator_info - hi6421v530 regulator information
+ * @desc: regulator description
+ * @mode_mask: ECO mode bitmask of LDOs; for BUCKs, this masks sleep
+ * @eco_microamp: eco mode load upper limit (in uA), valid for LDOs only
+ */
+struct hi6421v530_regulator_info {
+ struct regulator_desc rdesc;
+ u8 mode_mask;
+ u32 eco_microamp;
+};
+
+/* HI6421v530 regulators */
+enum hi6421v530_regulator_id {
+ HI6421V530_LDO3,
+ HI6421V530_LDO9,
+ HI6421V530_LDO11,
+ HI6421V530_LDO15,
+ HI6421V530_LDO16,
+};
+
+static const unsigned int ldo_3_voltages[] = {
+ 1800000, 1825000, 1850000, 1875000,
+ 1900000, 1925000, 1950000, 1975000,
+ 2000000, 2025000, 2050000, 2075000,
+ 2100000, 2125000, 2150000, 2200000,
+};
+
+static const unsigned int ldo_9_11_voltages[] = {
+ 1750000, 1800000, 1825000, 2800000,
+ 2850000, 2950000, 3000000, 3300000,
+};
+
+static const unsigned int ldo_15_16_voltages[] = {
+ 1750000, 1800000, 2400000, 2600000,
+ 2700000, 2850000, 2950000, 3000000,
+};
+
+static const struct regulator_ops hi6421v530_ldo_ops;
+
+#define HI6421V530_LDO_ENABLE_TIME (350)
+
+/*
+ * _id - LDO id name string
+ * v_table - voltage table
+ * vreg - voltage select register
+ * vmask - voltage select mask
+ * ereg - enable register
+ * emask - enable mask
+ * odelay - off/on delay time in uS
+ * ecomask - eco mode mask
+ * ecoamp - eco mode load uppler limit in uA
+ */
+#define HI6421V530_LDO(_ID, v_table, vreg, vmask, ereg, emask, \
+ odelay, ecomask, ecoamp) { \
+ .rdesc = { \
+ .name = #_ID, \
+ .of_match = of_match_ptr(#_ID), \
+ .regulators_node = of_match_ptr("regulators"), \
+ .ops = &hi6421v530_ldo_ops, \
+ .type = REGULATOR_VOLTAGE, \
+ .id = HI6421V530_##_ID, \
+ .owner = THIS_MODULE, \
+ .n_voltages = ARRAY_SIZE(v_table), \
+ .volt_table = v_table, \
+ .vsel_reg = HI6421_REG_TO_BUS_ADDR(vreg), \
+ .vsel_mask = vmask, \
+ .enable_reg = HI6421_REG_TO_BUS_ADDR(ereg), \
+ .enable_mask = emask, \
+ .enable_time = HI6421V530_LDO_ENABLE_TIME, \
+ .off_on_delay = odelay, \
+ }, \
+ .mode_mask = ecomask, \
+ .eco_microamp = ecoamp, \
+}
+
+/* HI6421V530 regulator information */
+
+static struct hi6421v530_regulator_info hi6421v530_regulator_info[] = {
+ HI6421V530_LDO(LDO3, ldo_3_voltages, 0x061, 0xf, 0x060, 0x2,
+ 20000, 0x6, 8000),
+ HI6421V530_LDO(LDO9, ldo_9_11_voltages, 0x06b, 0x7, 0x06a, 0x2,
+ 40000, 0x6, 8000),
+ HI6421V530_LDO(LDO11, ldo_9_11_voltages, 0x06f, 0x7, 0x06e, 0x2,
+ 40000, 0x6, 8000),
+ HI6421V530_LDO(LDO15, ldo_15_16_voltages, 0x077, 0x7, 0x076, 0x2,
+ 40000, 0x6, 8000),
+ HI6421V530_LDO(LDO16, ldo_15_16_voltages, 0x079, 0x7, 0x078, 0x2,
+ 40000, 0x6, 8000),
+};
+
+static unsigned int hi6421v530_regulator_ldo_get_mode(
+ struct regulator_dev *rdev)
+{
+ struct hi6421v530_regulator_info *info;
+ unsigned int reg_val;
+
+ info = rdev_get_drvdata(rdev);
+ regmap_read(rdev->regmap, rdev->desc->enable_reg, ®_val);
+
+ if (reg_val & (info->mode_mask))
+ return REGULATOR_MODE_IDLE;
+
+ return REGULATOR_MODE_NORMAL;
+}
+
+static int hi6421v530_regulator_ldo_set_mode(struct regulator_dev *rdev,
+ unsigned int mode)
+{
+ struct hi6421v530_regulator_info *info;
+ unsigned int new_mode;
+
+ info = rdev_get_drvdata(rdev);
+ switch (mode) {
+ case REGULATOR_MODE_NORMAL:
+ new_mode = 0;
+ break;
+ case REGULATOR_MODE_IDLE:
+ new_mode = info->mode_mask;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ regmap_update_bits(rdev->regmap, rdev->desc->enable_reg,
+ info->mode_mask, new_mode);
+
+ return 0;
+}
+
+
+static const struct regulator_ops hi6421v530_ldo_ops = {
+ .is_enabled = regulator_is_enabled_regmap,
+ .enable = regulator_enable_regmap,
+ .disable = regulator_disable_regmap,
+ .list_voltage = regulator_list_voltage_table,
+ .map_voltage = regulator_map_voltage_ascend,
+ .get_voltage_sel = regulator_get_voltage_sel_regmap,
+ .set_voltage_sel = regulator_set_voltage_sel_regmap,
+ .get_mode = hi6421v530_regulator_ldo_get_mode,
+ .set_mode = hi6421v530_regulator_ldo_set_mode,
+};
+
+static int hi6421v530_regulator_probe(struct platform_device *pdev)
+{
+ struct hi6421_pmic *pmic;
+ struct regulator_dev *rdev;
+ struct regulator_config config = { };
+ unsigned int i;
+
+ pmic = dev_get_drvdata(pdev->dev.parent);
+ if (!pmic) {
+ dev_err(&pdev->dev, "no pmic in the regulator parent node\n");
+ return -ENODEV;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(hi6421v530_regulator_info); i++) {
+ config.dev = pdev->dev.parent;
+ config.regmap = pmic->regmap;
+ config.driver_data = &hi6421v530_regulator_info[i];
+
+ rdev = devm_regulator_register(&pdev->dev,
+ &hi6421v530_regulator_info[i].rdesc,
+ &config);
+ if (IS_ERR(rdev)) {
+ dev_err(&pdev->dev, "failed to register regulator %s\n",
+ hi6421v530_regulator_info[i].rdesc.name);
+ return PTR_ERR(rdev);
+ }
+ }
+ return 0;
+}
+
+static const struct platform_device_id hi6421v530_regulator_table[] = {
+ { .name = "hi6421v530-regulator" },
+ {},
+};
+MODULE_DEVICE_TABLE(platform, hi6421v530_regulator_table);
+
+static struct platform_driver hi6421v530_regulator_driver = {
+ .id_table = hi6421v530_regulator_table,
+ .driver = {
+ .name = "hi6421v530-regulator",
+ },
+ .probe = hi6421v530_regulator_probe,
+};
+module_platform_driver(hi6421v530_regulator_driver);
+
+MODULE_AUTHOR("Wang Xiaoyin <hw.wangxiaoyin@hisilicon.com>");
+MODULE_DESCRIPTION("Hi6421v530 regulator driver");
+MODULE_LICENSE("GPL v2");
ret = lp8755_read(pchip, 0x12 + id, ®val);
if (ret < 0) {
- dev_err(pchip->dev, "i2c acceess error %s\n", __func__);
+ dev_err(pchip->dev, "i2c access error %s\n", __func__);
return ret;
}
return (regval & 0xff) * 100;
goto err_i2c;
return ret;
err_i2c:
- dev_err(pchip->dev, "i2c acceess error %s\n", __func__);
+ dev_err(pchip->dev, "i2c access error %s\n", __func__);
return ret;
}
return REGULATOR_MODE_NORMAL;
err_i2c:
- dev_err(pchip->dev, "i2c acceess error %s\n", __func__);
+ dev_err(pchip->dev, "i2c access error %s\n", __func__);
return 0;
}
goto err_i2c;
return ret;
err_i2c:
- dev_err(pchip->dev, "i2c acceess error %s\n", __func__);
+ dev_err(pchip->dev, "i2c access error %s\n", __func__);
return ret;
}
return ret;
out_i2c_error:
- dev_err(pchip->dev, "i2c acceess error %s\n", __func__);
+ dev_err(pchip->dev, "i2c access error %s\n", __func__);
return ret;
}
return IRQ_HANDLED;
err_i2c:
- dev_err(pchip->dev, "i2c acceess error %s\n", __func__);
+ dev_err(pchip->dev, "i2c access error %s\n", __func__);
return IRQ_NONE;
}
ret = lp8755_read(pchip, 0x0F, ®val);
if (ret < 0) {
- dev_err(pchip->dev, "i2c acceess error %s\n", __func__);
+ dev_err(pchip->dev, "i2c access error %s\n", __func__);
return ret;
}
--- /dev/null
+/*
+ * Regulator driver for LP87565 PMIC
+ *
+ * Copyright (C) 2017 Texas Instruments Incorporated - http://www.ti.com/
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2.
+ */
+
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+
+#include <linux/mfd/lp87565.h>
+
+#define LP87565_REGULATOR(_name, _id, _of, _ops, _n, _vr, _vm, _er, _em, \
+ _delay, _lr, _cr) \
+ [_id] = { \
+ .desc = { \
+ .name = _name, \
+ .supply_name = _of "-in", \
+ .id = _id, \
+ .of_match = of_match_ptr(_of), \
+ .regulators_node = of_match_ptr("regulators"),\
+ .ops = &_ops, \
+ .n_voltages = _n, \
+ .type = REGULATOR_VOLTAGE, \
+ .owner = THIS_MODULE, \
+ .vsel_reg = _vr, \
+ .vsel_mask = _vm, \
+ .enable_reg = _er, \
+ .enable_mask = _em, \
+ .ramp_delay = _delay, \
+ .linear_ranges = _lr, \
+ .n_linear_ranges = ARRAY_SIZE(_lr), \
+ }, \
+ .ctrl2_reg = _cr, \
+ }
+
+struct lp87565_regulator {
+ struct regulator_desc desc;
+ unsigned int ctrl2_reg;
+};
+
+static const struct lp87565_regulator regulators[];
+
+static const struct regulator_linear_range buck0_1_2_3_ranges[] = {
+ REGULATOR_LINEAR_RANGE(600000, 0xA, 0x17, 10000),
+ REGULATOR_LINEAR_RANGE(735000, 0x18, 0x9d, 5000),
+ REGULATOR_LINEAR_RANGE(1420000, 0x9e, 0xff, 20000),
+};
+
+static unsigned int lp87565_buck_ramp_delay[] = {
+ 30000, 15000, 10000, 7500, 3800, 1900, 940, 470
+};
+
+/* LP87565 BUCK current limit */
+static const unsigned int lp87565_buck_uA[] = {
+ 1500000, 2000000, 2500000, 3000000, 3500000, 4000000, 4500000, 5000000,
+};
+
+static int lp87565_buck_set_ramp_delay(struct regulator_dev *rdev,
+ int ramp_delay)
+{
+ int id = rdev_get_id(rdev);
+ struct lp87565 *lp87565 = rdev_get_drvdata(rdev);
+ unsigned int reg;
+ int ret;
+
+ if (ramp_delay <= 470)
+ reg = 7;
+ else if (ramp_delay <= 940)
+ reg = 6;
+ else if (ramp_delay <= 1900)
+ reg = 5;
+ else if (ramp_delay <= 3800)
+ reg = 4;
+ else if (ramp_delay <= 7500)
+ reg = 3;
+ else if (ramp_delay <= 10000)
+ reg = 2;
+ else if (ramp_delay <= 15000)
+ reg = 1;
+ else
+ reg = 0;
+
+ ret = regmap_update_bits(lp87565->regmap, regulators[id].ctrl2_reg,
+ LP87565_BUCK_CTRL_2_SLEW_RATE,
+ reg << __ffs(LP87565_BUCK_CTRL_2_SLEW_RATE));
+ if (ret) {
+ dev_err(lp87565->dev, "SLEW RATE write failed: %d\n", ret);
+ return ret;
+ }
+
+ rdev->constraints->ramp_delay = lp87565_buck_ramp_delay[reg];
+
+ return 0;
+}
+
+static int lp87565_buck_set_current_limit(struct regulator_dev *rdev,
+ int min_uA, int max_uA)
+{
+ int id = rdev_get_id(rdev);
+ struct lp87565 *lp87565 = rdev_get_drvdata(rdev);
+ int i;
+
+ for (i = ARRAY_SIZE(lp87565_buck_uA) - 1; i >= 0; i--) {
+ if (lp87565_buck_uA[i] >= min_uA &&
+ lp87565_buck_uA[i] <= max_uA)
+ return regmap_update_bits(lp87565->regmap,
+ regulators[id].ctrl2_reg,
+ LP87565_BUCK_CTRL_2_ILIM,
+ i << __ffs(LP87565_BUCK_CTRL_2_ILIM));
+ }
+
+ return -EINVAL;
+}
+
+static int lp87565_buck_get_current_limit(struct regulator_dev *rdev)
+{
+ int id = rdev_get_id(rdev);
+ struct lp87565 *lp87565 = rdev_get_drvdata(rdev);
+ int ret;
+ unsigned int val;
+
+ ret = regmap_read(lp87565->regmap, regulators[id].ctrl2_reg, &val);
+ if (ret)
+ return ret;
+
+ val = (val & LP87565_BUCK_CTRL_2_ILIM) >>
+ __ffs(LP87565_BUCK_CTRL_2_ILIM);
+
+ return (val < ARRAY_SIZE(lp87565_buck_uA)) ?
+ lp87565_buck_uA[val] : -EINVAL;
+}
+
+/* Operations permitted on BUCK0, BUCK1 */
+static struct regulator_ops lp87565_buck_ops = {
+ .is_enabled = regulator_is_enabled_regmap,
+ .enable = regulator_enable_regmap,
+ .disable = regulator_disable_regmap,
+ .get_voltage_sel = regulator_get_voltage_sel_regmap,
+ .set_voltage_sel = regulator_set_voltage_sel_regmap,
+ .list_voltage = regulator_list_voltage_linear_range,
+ .map_voltage = regulator_map_voltage_linear_range,
+ .set_voltage_time_sel = regulator_set_voltage_time_sel,
+ .set_ramp_delay = lp87565_buck_set_ramp_delay,
+ .set_current_limit = lp87565_buck_set_current_limit,
+ .get_current_limit = lp87565_buck_get_current_limit,
+};
+
+static const struct lp87565_regulator regulators[] = {
+ LP87565_REGULATOR("BUCK0", LP87565_BUCK_0, "buck0", lp87565_buck_ops,
+ 256, LP87565_REG_BUCK0_VOUT, LP87565_BUCK_VSET,
+ LP87565_REG_BUCK0_CTRL_1,
+ LP87565_BUCK_CTRL_1_EN, 3800,
+ buck0_1_2_3_ranges, LP87565_REG_BUCK0_CTRL_2),
+ LP87565_REGULATOR("BUCK1", LP87565_BUCK_1, "buck1", lp87565_buck_ops,
+ 256, LP87565_REG_BUCK1_VOUT, LP87565_BUCK_VSET,
+ LP87565_REG_BUCK1_CTRL_1,
+ LP87565_BUCK_CTRL_1_EN, 3800,
+ buck0_1_2_3_ranges, LP87565_REG_BUCK1_CTRL_2),
+ LP87565_REGULATOR("BUCK2", LP87565_BUCK_2, "buck2", lp87565_buck_ops,
+ 256, LP87565_REG_BUCK2_VOUT, LP87565_BUCK_VSET,
+ LP87565_REG_BUCK2_CTRL_1,
+ LP87565_BUCK_CTRL_1_EN, 3800,
+ buck0_1_2_3_ranges, LP87565_REG_BUCK2_CTRL_2),
+ LP87565_REGULATOR("BUCK3", LP87565_BUCK_3, "buck3", lp87565_buck_ops,
+ 256, LP87565_REG_BUCK3_VOUT, LP87565_BUCK_VSET,
+ LP87565_REG_BUCK3_CTRL_1,
+ LP87565_BUCK_CTRL_1_EN, 3800,
+ buck0_1_2_3_ranges, LP87565_REG_BUCK3_CTRL_2),
+ LP87565_REGULATOR("BUCK10", LP87565_BUCK_10, "buck10", lp87565_buck_ops,
+ 256, LP87565_REG_BUCK0_VOUT, LP87565_BUCK_VSET,
+ LP87565_REG_BUCK0_CTRL_1,
+ LP87565_BUCK_CTRL_1_EN, 3800,
+ buck0_1_2_3_ranges, LP87565_REG_BUCK0_CTRL_2),
+ LP87565_REGULATOR("BUCK23", LP87565_BUCK_23, "buck23", lp87565_buck_ops,
+ 256, LP87565_REG_BUCK2_VOUT, LP87565_BUCK_VSET,
+ LP87565_REG_BUCK2_CTRL_1,
+ LP87565_BUCK_CTRL_1_EN, 3800,
+ buck0_1_2_3_ranges, LP87565_REG_BUCK2_CTRL_2),
+};
+
+static int lp87565_regulator_probe(struct platform_device *pdev)
+{
+ struct lp87565 *lp87565 = dev_get_drvdata(pdev->dev.parent);
+ struct regulator_config config = { };
+ struct regulator_dev *rdev;
+ int i, min_idx = LP87565_BUCK_1, max_idx = LP87565_BUCK_3;
+
+ platform_set_drvdata(pdev, lp87565);
+
+ config.dev = &pdev->dev;
+ config.dev->of_node = lp87565->dev->of_node;
+ config.driver_data = lp87565;
+ config.regmap = lp87565->regmap;
+
+ if (lp87565->dev_type == LP87565_DEVICE_TYPE_LP87565_Q1) {
+ min_idx = LP87565_BUCK_10;
+ max_idx = LP87565_BUCK_23;
+ }
+
+ for (i = min_idx; i <= max_idx; i++) {
+ rdev = devm_regulator_register(&pdev->dev, ®ulators[i].desc,
+ &config);
+ if (IS_ERR(rdev)) {
+ dev_err(lp87565->dev, "failed to register %s regulator\n",
+ pdev->name);
+ return PTR_ERR(rdev);
+ }
+ }
+
+ return 0;
+}
+
+static const struct platform_device_id lp87565_regulator_id_table[] = {
+ { "lp87565-regulator", },
+ { "lp87565-q1-regulator", },
+ { /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(platform, lp87565_regulator_id_table);
+
+static struct platform_driver lp87565_regulator_driver = {
+ .driver = {
+ .name = "lp87565-pmic",
+ },
+ .probe = lp87565_regulator_probe,
+ .id_table = lp87565_regulator_id_table,
+};
+module_platform_driver(lp87565_regulator_driver);
+
+MODULE_AUTHOR("J Keerthy <j-keerthy@ti.com>");
+MODULE_DESCRIPTION("LP87565 voltage regulator driver");
+MODULE_LICENSE("GPL v2");
if (max_uV < 4000000 || min_uV > 4350000)
return -EINVAL;
- if (min_uV <= 4000000) {
- if (max_uV >= 4000000)
- return -EINVAL;
- else
- val = 0x1;
- } else if (min_uV <= 4200000 && max_uV >= 4200000)
+ if (min_uV <= 4000000)
+ val = 0x1;
+ else if (min_uV <= 4200000 && max_uV >= 4200000)
val = 0x0;
else {
lb = (min_uV - 4000001) / 20000 + 2;
if (!ret)
constraints->settling_time = pval;
+ ret = of_property_read_u32(np, "regulator-settling-time-up-us", &pval);
+ if (!ret)
+ constraints->settling_time_up = pval;
+ if (constraints->settling_time_up && constraints->settling_time) {
+ pr_warn("%s: ambiguous configuration for settling time, ignoring 'regulator-settling-time-up-us'\n",
+ np->name);
+ constraints->settling_time_up = 0;
+ }
+
+ ret = of_property_read_u32(np, "regulator-settling-time-down-us",
+ &pval);
+ if (!ret)
+ constraints->settling_time_down = pval;
+ if (constraints->settling_time_down && constraints->settling_time) {
+ pr_warn("%s: ambiguous configuration for settling time, ignoring 'regulator-settling-time-down-us'\n",
+ np->name);
+ constraints->settling_time_down = 0;
+ }
+
ret = of_property_read_u32(np, "regulator-enable-ramp-delay", &pval);
if (!ret)
constraints->enable_time = pval;
.ctrl_addr = TPS65917_SMPS5_CTRL,
.sleep_id = TPS65917_EXTERNAL_REQSTR_ID_SMPS5,
},
+ {
+ .name = "SMPS12",
+ .sname = "smps1-in",
+ .vsel_addr = TPS65917_SMPS1_VOLTAGE,
+ .ctrl_addr = TPS65917_SMPS1_CTRL,
+ .sleep_id = TPS65917_EXTERNAL_REQSTR_ID_SMPS12,
+ },
{
.name = "LDO1",
.sname = "ldo1-in",
EXTERNAL_REQUESTOR_TPS65917(SMPS3, 1, 2),
EXTERNAL_REQUESTOR_TPS65917(SMPS4, 1, 3),
EXTERNAL_REQUESTOR_TPS65917(SMPS5, 1, 4),
+ EXTERNAL_REQUESTOR_TPS65917(SMPS12, 1, 5),
EXTERNAL_REQUESTOR_TPS65917(LDO1, 2, 0),
EXTERNAL_REQUESTOR_TPS65917(LDO2, 2, 1),
EXTERNAL_REQUESTOR_TPS65917(LDO3, 2, 2),
*/
desc = &pmic->desc[id];
desc->n_linear_ranges = 3;
- if ((id == TPS65917_REG_SMPS2) && pmic->smps12)
+ if ((id == TPS65917_REG_SMPS2 || id == TPS65917_REG_SMPS1) &&
+ pmic->smps12)
continue;
/* Initialise sleep/init values from platform data */
{ .name = "smps3", },
{ .name = "smps4", },
{ .name = "smps5", },
+ { .name = "smps12",},
{ .name = "ldo1", },
{ .name = "ldo2", },
{ .name = "ldo3", },
static struct palmas_pmic_driver_data tps65917_ddata = {
.smps_start = TPS65917_REG_SMPS1,
- .smps_end = TPS65917_REG_SMPS5,
+ .smps_end = TPS65917_REG_SMPS12,
.ldo_begin = TPS65917_REG_LDO1,
.ldo_end = TPS65917_REG_LDO5,
.max_reg = TPS65917_NUM_REGS,
}
for (idx = 0; idx < ddata->max_reg; idx++) {
- static struct of_regulator_match *match;
+ struct of_regulator_match *match;
struct palmas_reg_init *rinit;
struct device_node *np;
if (ret)
return ret;
- if (reg & PALMAS_SMPS_CTRL_SMPS12_SMPS123_EN)
+ if (reg & PALMAS_SMPS_CTRL_SMPS12_SMPS123_EN) {
pmic->smps123 = 1;
+ pmic->smps12 = 1;
+ }
if (reg & PALMAS_SMPS_CTRL_SMPS45_SMPS457_EN)
pmic->smps457 = 1;
switch (tps65910_chip_id(tps65910)) {
case TPS65910:
+ BUILD_BUG_ON(TPS65910_NUM_REGS < ARRAY_SIZE(tps65910_regs));
pmic->get_ctrl_reg = &tps65910_get_ctrl_register;
pmic->num_regulators = ARRAY_SIZE(tps65910_regs);
pmic->ext_sleep_control = tps65910_ext_sleep_control;
DCDCCTRL_DCDCCKSYNC_MASK);
break;
case TPS65911:
+ BUILD_BUG_ON(TPS65910_NUM_REGS < ARRAY_SIZE(tps65911_regs));
pmic->get_ctrl_reg = &tps65911_get_ctrl_register;
pmic->num_regulators = ARRAY_SIZE(tps65911_regs);
pmic->ext_sleep_control = tps65911_ext_sleep_control;
if (!pmic->rdev)
return -ENOMEM;
- for (i = 0; i < pmic->num_regulators && i < TPS65910_NUM_REGS;
- i++, info++) {
+ for (i = 0; i < pmic->num_regulators; i++, info++) {
/* Register the regulators */
pmic->info[i] = info;
struct rpmsg_device *rpdev = to_rpmsg_device(dev); \
\
return sprintf(buf, format_string, rpdev->path); \
-}
+} \
+static DEVICE_ATTR_RO(field);
/* for more info, see Documentation/ABI/testing/sysfs-bus-rpmsg */
rpmsg_show_attr(name, id.name, "%s\n");
return sprintf(buf, RPMSG_DEVICE_MODALIAS_FMT "\n", rpdev->id.name);
}
-
-static struct device_attribute rpmsg_dev_attrs[] = {
- __ATTR_RO(name),
- __ATTR_RO(modalias),
- __ATTR_RO(dst),
- __ATTR_RO(src),
- __ATTR_RO(announce),
- __ATTR_NULL
+static DEVICE_ATTR_RO(modalias);
+
+static struct attribute *rpmsg_dev_attrs[] = {
+ &dev_attr_name.attr,
+ &dev_attr_modalias.attr,
+ &dev_attr_dst.attr,
+ &dev_attr_src.attr,
+ &dev_attr_announce.attr,
+ NULL,
};
+ATTRIBUTE_GROUPS(rpmsg_dev);
/* rpmsg devices and drivers are matched using the service name */
static inline int rpmsg_id_match(const struct rpmsg_device *rpdev,
static struct bus_type rpmsg_bus = {
.name = "rpmsg",
.match = rpmsg_dev_match,
- .dev_attrs = rpmsg_dev_attrs,
+ .dev_groups = rpmsg_dev_groups,
.uevent = rpmsg_uevent,
.probe = rpmsg_dev_probe,
.remove = rpmsg_dev_remove,
/*If the write wait queue is empty then there is no pending
operations. It means the interrupt is for DryIce -Security.
IRQ must be returned as none.*/
- if (list_empty_careful(&imxdi->write_wait.task_list))
+ if (list_empty_careful(&imxdi->write_wait.head))
return rc;
/* DSR_WCF clears itself on DSR read */
To compile this driver as a module, choose M here: the
module will be called scm_block.
-
-config SCM_BLOCK_CLUSTER_WRITE
- def_bool y
- prompt "SCM force cluster writes"
- depends on SCM_BLOCK
- help
- Force writes to Storage Class Memory (SCM) to be in done in clusters.
obj-$(CONFIG_DCSSBLK) += dcssblk.o
scm_block-objs := scm_drv.o scm_blk.o
-ifdef CONFIG_SCM_BLOCK_CLUSTER_WRITE
-scm_block-objs += scm_blk_cluster.o
-endif
obj-$(CONFIG_SCM_BLOCK) += scm_block.o
{
int mask = ~(DASD_STOPPED_DC_WAIT | DASD_UNRESUMED_PM);
- if (test_bit(DASD_FLAG_OFFLINE, &device->flags)) {
- /* dasd is being set offline. */
+ if (test_bit(DASD_FLAG_OFFLINE, &device->flags) &&
+ !test_bit(DASD_FLAG_SAFE_OFFLINE_RUNNING, &device->flags)) {
+ /*
+ * dasd is being set offline
+ * but it is no safe offline where we have to allow I/O
+ */
return 1;
}
if (device->stopped) {
*/
if (basedev->state < DASD_STATE_READY) {
while ((req = blk_fetch_request(block->request_queue)))
- __blk_end_request_all(req, -EIO);
+ __blk_end_request_all(req, BLK_STS_IOERR);
return;
}
"Rejecting write request %p",
req);
blk_start_request(req);
- __blk_end_request_all(req, -EIO);
+ __blk_end_request_all(req, BLK_STS_IOERR);
continue;
}
if (test_bit(DASD_FLAG_ABORTALL, &basedev->flags) &&
"Rejecting failfast request %p",
req);
blk_start_request(req);
- __blk_end_request_all(req, -ETIMEDOUT);
+ __blk_end_request_all(req, BLK_STS_TIMEOUT);
continue;
}
cqr = basedev->discipline->build_cp(basedev, block, req);
"on request %p",
PTR_ERR(cqr), req);
blk_start_request(req);
- __blk_end_request_all(req, -EIO);
+ __blk_end_request_all(req, BLK_STS_IOERR);
continue;
}
/*
{
struct request *req;
int status;
- int error = 0;
+ blk_status_t error = BLK_STS_OK;
req = (struct request *) cqr->callback_data;
dasd_profile_end(cqr->block, cqr, req);
+
status = cqr->block->base->discipline->free_cp(cqr, req);
if (status < 0)
- error = status;
+ error = errno_to_blk_status(status);
else if (status == 0) {
- if (cqr->intrc == -EPERM)
- error = -EBADE;
- else if (cqr->intrc == -ENOLINK ||
- cqr->intrc == -ETIMEDOUT)
- error = cqr->intrc;
- else
- error = -EIO;
+ switch (cqr->intrc) {
+ case -EPERM:
+ error = BLK_STS_NEXUS;
+ break;
+ case -ENOLINK:
+ error = BLK_STS_TRANSPORT;
+ break;
+ case -ETIMEDOUT:
+ error = BLK_STS_TIMEOUT;
+ break;
+ default:
+ error = BLK_STS_IOERR;
+ break;
+ }
}
__blk_end_request_all(req, error);
}
spin_lock_irq(&block->request_queue_lock);
while ((req = blk_fetch_request(block->request_queue)))
- __blk_end_request_all(req, -EIO);
+ __blk_end_request_all(req, BLK_STS_IOERR);
spin_unlock_irq(&block->request_queue_lock);
}
else
pr_warn("%s: The DASD cannot be set offline while it is in use\n",
dev_name(&cdev->dev));
- clear_bit(DASD_FLAG_OFFLINE, &device->flags);
- goto out_busy;
+ rc = -EBUSY;
+ goto out_err;
}
}
- if (test_bit(DASD_FLAG_SAFE_OFFLINE_RUNNING, &device->flags)) {
- /*
- * safe offline already running
- * could only be called by normal offline so safe_offline flag
- * needs to be removed to run normal offline and kill all I/O
- */
- if (test_and_set_bit(DASD_FLAG_OFFLINE, &device->flags))
- /* Already doing normal offline processing */
- goto out_busy;
- else
- clear_bit(DASD_FLAG_SAFE_OFFLINE, &device->flags);
- } else {
- if (test_bit(DASD_FLAG_OFFLINE, &device->flags))
- /* Already doing offline processing */
- goto out_busy;
+ /*
+ * Test if the offline processing is already running and exit if so.
+ * If a safe offline is being processed this could only be a normal
+ * offline that should be able to overtake the safe offline and
+ * cancel any I/O we do not want to wait for any longer
+ */
+ if (test_bit(DASD_FLAG_OFFLINE, &device->flags)) {
+ if (test_bit(DASD_FLAG_SAFE_OFFLINE_RUNNING, &device->flags)) {
+ clear_bit(DASD_FLAG_SAFE_OFFLINE_RUNNING,
+ &device->flags);
+ } else {
+ rc = -EBUSY;
+ goto out_err;
+ }
}
-
set_bit(DASD_FLAG_OFFLINE, &device->flags);
- spin_unlock_irqrestore(get_ccwdev_lock(cdev), flags);
/*
- * if safe_offline called set safe_offline_running flag and
+ * if safe_offline is called set safe_offline_running flag and
* clear safe_offline so that a call to normal offline
* can overrun safe_offline processing
*/
if (test_and_clear_bit(DASD_FLAG_SAFE_OFFLINE, &device->flags) &&
!test_and_set_bit(DASD_FLAG_SAFE_OFFLINE_RUNNING, &device->flags)) {
+ /* need to unlock here to wait for outstanding I/O */
+ spin_unlock_irqrestore(get_ccwdev_lock(cdev), flags);
/*
* If we want to set the device safe offline all IO operations
* should be finished before continuing the offline process
* so sync bdev first and then wait for our queues to become
* empty
*/
- /* sync blockdev and partitions */
if (device->block) {
rc = fsync_bdev(device->block->bdev);
if (rc != 0)
goto interrupted;
}
- /* schedule device tasklet and wait for completion */
dasd_schedule_device_bh(device);
rc = wait_event_interruptible(shutdown_waitq,
_wait_for_empty_queues(device));
if (rc != 0)
goto interrupted;
+
+ /*
+ * check if a normal offline process overtook the offline
+ * processing in this case simply do nothing beside returning
+ * that we got interrupted
+ * otherwise mark safe offline as not running any longer and
+ * continue with normal offline
+ */
+ spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
+ if (!test_bit(DASD_FLAG_SAFE_OFFLINE_RUNNING, &device->flags)) {
+ rc = -ERESTARTSYS;
+ goto out_err;
+ }
+ clear_bit(DASD_FLAG_SAFE_OFFLINE_RUNNING, &device->flags);
}
+ spin_unlock_irqrestore(get_ccwdev_lock(cdev), flags);
dasd_set_target_state(device, DASD_STATE_NEW);
/* dasd_delete_device destroys the device reference. */
*/
if (block)
dasd_free_block(block);
+
return 0;
interrupted:
/* interrupted by signal */
- clear_bit(DASD_FLAG_SAFE_OFFLINE, &device->flags);
+ spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
clear_bit(DASD_FLAG_SAFE_OFFLINE_RUNNING, &device->flags);
clear_bit(DASD_FLAG_OFFLINE, &device->flags);
- dasd_put_device(device);
-
- return rc;
-
-out_busy:
+out_err:
dasd_put_device(device);
spin_unlock_irqrestore(get_ccwdev_lock(cdev), flags);
-
- return -EBUSY;
+ return rc;
}
EXPORT_SYMBOL_GPL(dasd_generic_set_offline);
char *features_str = NULL;
char *from_str = NULL;
char *to_str = NULL;
- size_t len = strlen(range) + 1;
- char tmp[len];
+ int rc = 0;
+ char *tmp;
- strlcpy(tmp, range, len);
+ tmp = kstrdup(range, GFP_KERNEL);
+ if (!tmp)
+ return -ENOMEM;
- if (dasd_evaluate_range_param(tmp, &from_str, &to_str, &features_str))
- goto out_err;
+ if (dasd_evaluate_range_param(tmp, &from_str, &to_str, &features_str)) {
+ rc = -EINVAL;
+ goto out;
+ }
- if (dasd_busid(from_str, &from_id0, &from_id1, &from))
- goto out_err;
+ if (dasd_busid(from_str, &from_id0, &from_id1, &from)) {
+ rc = -EINVAL;
+ goto out;
+ }
to = from;
to_id0 = from_id0;
to_id1 = from_id1;
if (to_str) {
- if (dasd_busid(to_str, &to_id0, &to_id1, &to))
- goto out_err;
+ if (dasd_busid(to_str, &to_id0, &to_id1, &to)) {
+ rc = -EINVAL;
+ goto out;
+ }
if (from_id0 != to_id0 || from_id1 != to_id1 || from > to) {
pr_err("%s is not a valid device range\n", range);
- goto out_err;
+ rc = -EINVAL;
+ goto out;
}
}
features = dasd_feature_list(features_str);
- if (features < 0)
- goto out_err;
+ if (features < 0) {
+ rc = -EINVAL;
+ goto out;
+ }
/* each device in dasd= parameter should be set initially online */
features |= DASD_FEATURE_INITIAL_ONLINE;
while (from <= to) {
sprintf(bus_id, "%01x.%01x.%04x", from_id0, from_id1, from++);
devmap = dasd_add_busid(bus_id, features);
- if (IS_ERR(devmap))
- return PTR_ERR(devmap);
+ if (IS_ERR(devmap)) {
+ rc = PTR_ERR(devmap);
+ goto out;
+ }
}
- return 0;
+out:
+ kfree(tmp);
-out_err:
- return -EINVAL;
+ return rc;
}
/*
dasd_ro_show(struct device *dev, struct device_attribute *attr, char *buf)
{
struct dasd_devmap *devmap;
- int ro_flag;
+ struct dasd_device *device;
+ int ro_flag = 0;
devmap = dasd_find_busid(dev_name(dev));
- if (!IS_ERR(devmap))
- ro_flag = (devmap->features & DASD_FEATURE_READONLY) != 0;
- else
- ro_flag = (DASD_FEATURE_DEFAULT & DASD_FEATURE_READONLY) != 0;
+ if (IS_ERR(devmap))
+ goto out;
+
+ ro_flag = !!(devmap->features & DASD_FEATURE_READONLY);
+
+ spin_lock(&dasd_devmap_lock);
+ device = devmap->device;
+ if (device)
+ ro_flag |= test_bit(DASD_FLAG_DEVICE_RO, &device->flags);
+ spin_unlock(&dasd_devmap_lock);
+
+out:
return snprintf(buf, PAGE_SIZE, ro_flag ? "1\n" : "0\n");
}
device = dasd_device_from_cdev(cdev);
if (IS_ERR(device))
- return PTR_ERR(device);
+ return count;
spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
val = val || test_bit(DASD_FLAG_DEVICE_RO, &device->flags);
{
struct ccw_device *cdev = to_ccwdev(dev);
struct dasd_device *device;
+ unsigned long flags;
int rc;
- device = dasd_device_from_cdev(cdev);
+ spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
+ device = dasd_device_from_cdev_locked(cdev);
if (IS_ERR(device)) {
rc = PTR_ERR(device);
+ spin_unlock_irqrestore(get_ccwdev_lock(cdev), flags);
goto out;
}
test_bit(DASD_FLAG_SAFE_OFFLINE_RUNNING, &device->flags)) {
/* Already doing offline processing */
dasd_put_device(device);
+ spin_unlock_irqrestore(get_ccwdev_lock(cdev), flags);
rc = -EBUSY;
goto out;
}
set_bit(DASD_FLAG_SAFE_OFFLINE, &device->flags);
dasd_put_device(device);
+ spin_unlock_irqrestore(get_ccwdev_lock(cdev), flags);
rc = ccw_device_set_offline(cdev);
unsigned long source_addr;
unsigned long bytes_done;
- blk_queue_split(q, &bio, q->bio_split);
+ blk_queue_split(q, &bio);
bytes_done = 0;
dev_info = bio->bi_bdev->bd_disk->private_data;
#include <linux/mempool.h>
#include <linux/module.h>
#include <linux/blkdev.h>
+#include <linux/blk-mq.h>
#include <linux/genhd.h>
#include <linux/slab.h>
#include <linux/list.h>
struct aob_rq_header *aobrq = to_aobrq(scmrq);
free_page((unsigned long) scmrq->aob);
- __scm_free_rq_cluster(scmrq);
kfree(scmrq->request);
kfree(aobrq);
}
if (!scmrq->request)
goto free;
- if (__scm_alloc_rq_cluster(scmrq))
- goto free;
-
INIT_LIST_HEAD(&scmrq->list);
spin_lock_irq(&list_lock);
list_add(&scmrq->list, &inactive_requests);
{
struct scm_request *scmrq = NULL;
- spin_lock(&list_lock);
+ spin_lock_irq(&list_lock);
if (list_empty(&inactive_requests))
goto out;
scmrq = list_first_entry(&inactive_requests, struct scm_request, list);
list_del(&scmrq->list);
out:
- spin_unlock(&list_lock);
+ spin_unlock_irq(&list_lock);
return scmrq;
}
aob->request.data = (u64) aobrq;
scmrq->bdev = bdev;
scmrq->retries = 4;
- scmrq->error = 0;
+ scmrq->error = BLK_STS_OK;
/* We don't use all msbs - place aidaws at the end of the aob page. */
scmrq->next_aidaw = (void *) &aob->msb[nr_requests_per_io];
- scm_request_cluster_init(scmrq);
}
-static void scm_ensure_queue_restart(struct scm_blk_dev *bdev)
-{
- if (atomic_read(&bdev->queued_reqs)) {
- /* Queue restart is triggered by the next interrupt. */
- return;
- }
- blk_delay_queue(bdev->rq, SCM_QUEUE_DELAY);
-}
-
-void scm_request_requeue(struct scm_request *scmrq)
+static void scm_request_requeue(struct scm_request *scmrq)
{
struct scm_blk_dev *bdev = scmrq->bdev;
int i;
- scm_release_cluster(scmrq);
for (i = 0; i < nr_requests_per_io && scmrq->request[i]; i++)
- blk_requeue_request(bdev->rq, scmrq->request[i]);
+ blk_mq_requeue_request(scmrq->request[i], false);
atomic_dec(&bdev->queued_reqs);
scm_request_done(scmrq);
- scm_ensure_queue_restart(bdev);
+ blk_mq_kick_requeue_list(bdev->rq);
}
-void scm_request_finish(struct scm_request *scmrq)
+static void scm_request_finish(struct scm_request *scmrq)
{
struct scm_blk_dev *bdev = scmrq->bdev;
int i;
- scm_release_cluster(scmrq);
- for (i = 0; i < nr_requests_per_io && scmrq->request[i]; i++)
- blk_end_request_all(scmrq->request[i], scmrq->error);
+ for (i = 0; i < nr_requests_per_io && scmrq->request[i]; i++) {
+ if (scmrq->error)
+ blk_mq_end_request(scmrq->request[i], scmrq->error);
+ else
+ blk_mq_complete_request(scmrq->request[i]);
+ }
atomic_dec(&bdev->queued_reqs);
scm_request_done(scmrq);
}
-static int scm_request_start(struct scm_request *scmrq)
+static void scm_request_start(struct scm_request *scmrq)
{
struct scm_blk_dev *bdev = scmrq->bdev;
- int ret;
atomic_inc(&bdev->queued_reqs);
- if (!scmrq->aob->request.msb_count) {
- scm_request_requeue(scmrq);
- return -EINVAL;
- }
-
- ret = eadm_start_aob(scmrq->aob);
- if (ret) {
+ if (eadm_start_aob(scmrq->aob)) {
SCM_LOG(5, "no subchannel");
scm_request_requeue(scmrq);
}
- return ret;
}
-static void scm_blk_request(struct request_queue *rq)
+struct scm_queue {
+ struct scm_request *scmrq;
+ spinlock_t lock;
+};
+
+static int scm_blk_request(struct blk_mq_hw_ctx *hctx,
+ const struct blk_mq_queue_data *qd)
{
- struct scm_device *scmdev = rq->queuedata;
+ struct scm_device *scmdev = hctx->queue->queuedata;
struct scm_blk_dev *bdev = dev_get_drvdata(&scmdev->dev);
- struct scm_request *scmrq = NULL;
- struct request *req;
+ struct scm_queue *sq = hctx->driver_data;
+ struct request *req = qd->rq;
+ struct scm_request *scmrq;
- while ((req = blk_peek_request(rq))) {
- if (!scm_permit_request(bdev, req))
- goto out;
+ spin_lock(&sq->lock);
+ if (!scm_permit_request(bdev, req)) {
+ spin_unlock(&sq->lock);
+ return BLK_MQ_RQ_QUEUE_BUSY;
+ }
+ scmrq = sq->scmrq;
+ if (!scmrq) {
+ scmrq = scm_request_fetch();
if (!scmrq) {
- scmrq = scm_request_fetch();
- if (!scmrq) {
- SCM_LOG(5, "no request");
- goto out;
- }
- scm_request_init(bdev, scmrq);
+ SCM_LOG(5, "no request");
+ spin_unlock(&sq->lock);
+ return BLK_MQ_RQ_QUEUE_BUSY;
}
- scm_request_set(scmrq, req);
+ scm_request_init(bdev, scmrq);
+ sq->scmrq = scmrq;
+ }
+ scm_request_set(scmrq, req);
- if (!scm_reserve_cluster(scmrq)) {
- SCM_LOG(5, "cluster busy");
- scm_request_set(scmrq, NULL);
- if (scmrq->aob->request.msb_count)
- goto out;
+ if (scm_request_prepare(scmrq)) {
+ SCM_LOG(5, "aidaw alloc failed");
+ scm_request_set(scmrq, NULL);
- scm_request_done(scmrq);
- return;
- }
+ if (scmrq->aob->request.msb_count)
+ scm_request_start(scmrq);
- if (scm_need_cluster_request(scmrq)) {
- if (scmrq->aob->request.msb_count) {
- /* Start cluster requests separately. */
- scm_request_set(scmrq, NULL);
- if (scm_request_start(scmrq))
- return;
- } else {
- atomic_inc(&bdev->queued_reqs);
- blk_start_request(req);
- scm_initiate_cluster_request(scmrq);
- }
- scmrq = NULL;
- continue;
- }
+ sq->scmrq = NULL;
+ spin_unlock(&sq->lock);
+ return BLK_MQ_RQ_QUEUE_BUSY;
+ }
+ blk_mq_start_request(req);
- if (scm_request_prepare(scmrq)) {
- SCM_LOG(5, "aidaw alloc failed");
- scm_request_set(scmrq, NULL);
- goto out;
- }
- blk_start_request(req);
+ if (qd->last || scmrq->aob->request.msb_count == nr_requests_per_io) {
+ scm_request_start(scmrq);
+ sq->scmrq = NULL;
+ }
+ spin_unlock(&sq->lock);
+ return BLK_MQ_RQ_QUEUE_OK;
+}
- if (scmrq->aob->request.msb_count < nr_requests_per_io)
- continue;
+static int scm_blk_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
+ unsigned int idx)
+{
+ struct scm_queue *qd = kzalloc(sizeof(*qd), GFP_KERNEL);
- if (scm_request_start(scmrq))
- return;
+ if (!qd)
+ return -ENOMEM;
- scmrq = NULL;
- }
-out:
- if (scmrq)
- scm_request_start(scmrq);
- else
- scm_ensure_queue_restart(bdev);
+ spin_lock_init(&qd->lock);
+ hctx->driver_data = qd;
+
+ return 0;
+}
+
+static void scm_blk_exit_hctx(struct blk_mq_hw_ctx *hctx, unsigned int idx)
+{
+ struct scm_queue *qd = hctx->driver_data;
+
+ WARN_ON(qd->scmrq);
+ kfree(hctx->driver_data);
+ hctx->driver_data = NULL;
}
static void __scmrq_log_error(struct scm_request *scmrq)
{
struct aob *aob = scmrq->aob;
- if (scmrq->error == -ETIMEDOUT)
+ if (scmrq->error == BLK_STS_TIMEOUT)
SCM_LOG(1, "Request timeout");
else {
SCM_LOG(1, "Request error");
scmrq->error);
}
-void scm_blk_irq(struct scm_device *scmdev, void *data, int error)
-{
- struct scm_request *scmrq = data;
- struct scm_blk_dev *bdev = scmrq->bdev;
-
- scmrq->error = error;
- if (error)
- __scmrq_log_error(scmrq);
-
- spin_lock(&bdev->lock);
- list_add_tail(&scmrq->list, &bdev->finished_requests);
- spin_unlock(&bdev->lock);
- tasklet_hi_schedule(&bdev->tasklet);
-}
-
static void scm_blk_handle_error(struct scm_request *scmrq)
{
struct scm_blk_dev *bdev = scmrq->bdev;
unsigned long flags;
- if (scmrq->error != -EIO)
+ if (scmrq->error != BLK_STS_IOERR)
goto restart;
/* For -EIO the response block is valid. */
return;
requeue:
- spin_lock_irqsave(&bdev->rq_lock, flags);
scm_request_requeue(scmrq);
- spin_unlock_irqrestore(&bdev->rq_lock, flags);
}
-static void scm_blk_tasklet(struct scm_blk_dev *bdev)
+void scm_blk_irq(struct scm_device *scmdev, void *data, blk_status_t error)
{
- struct scm_request *scmrq;
- unsigned long flags;
-
- spin_lock_irqsave(&bdev->lock, flags);
- while (!list_empty(&bdev->finished_requests)) {
- scmrq = list_first_entry(&bdev->finished_requests,
- struct scm_request, list);
- list_del(&scmrq->list);
- spin_unlock_irqrestore(&bdev->lock, flags);
+ struct scm_request *scmrq = data;
- if (scmrq->error && scmrq->retries-- > 0) {
+ scmrq->error = error;
+ if (error) {
+ __scmrq_log_error(scmrq);
+ if (scmrq->retries-- > 0) {
scm_blk_handle_error(scmrq);
-
- /* Request restarted or requeued, handle next. */
- spin_lock_irqsave(&bdev->lock, flags);
- continue;
+ return;
}
+ }
- if (scm_test_cluster_request(scmrq)) {
- scm_cluster_request_irq(scmrq);
- spin_lock_irqsave(&bdev->lock, flags);
- continue;
- }
+ scm_request_finish(scmrq);
+}
- scm_request_finish(scmrq);
- spin_lock_irqsave(&bdev->lock, flags);
- }
- spin_unlock_irqrestore(&bdev->lock, flags);
- /* Look out for more requests. */
- blk_run_queue(bdev->rq);
+static void scm_blk_request_done(struct request *req)
+{
+ blk_mq_end_request(req, 0);
}
static const struct block_device_operations scm_blk_devops = {
.owner = THIS_MODULE,
};
+static const struct blk_mq_ops scm_mq_ops = {
+ .queue_rq = scm_blk_request,
+ .complete = scm_blk_request_done,
+ .init_hctx = scm_blk_init_hctx,
+ .exit_hctx = scm_blk_exit_hctx,
+};
+
int scm_blk_dev_setup(struct scm_blk_dev *bdev, struct scm_device *scmdev)
{
- struct request_queue *rq;
- int len, ret = -ENOMEM;
unsigned int devindex, nr_max_blk;
+ struct request_queue *rq;
+ int len, ret;
devindex = atomic_inc_return(&nr_devices) - 1;
/* scma..scmz + scmaa..scmzz */
bdev->scmdev = scmdev;
bdev->state = SCM_OPER;
- spin_lock_init(&bdev->rq_lock);
spin_lock_init(&bdev->lock);
- INIT_LIST_HEAD(&bdev->finished_requests);
atomic_set(&bdev->queued_reqs, 0);
- tasklet_init(&bdev->tasklet,
- (void (*)(unsigned long)) scm_blk_tasklet,
- (unsigned long) bdev);
- rq = blk_init_queue(scm_blk_request, &bdev->rq_lock);
- if (!rq)
+ bdev->tag_set.ops = &scm_mq_ops;
+ bdev->tag_set.nr_hw_queues = nr_requests;
+ bdev->tag_set.queue_depth = nr_requests_per_io * nr_requests;
+ bdev->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
+
+ ret = blk_mq_alloc_tag_set(&bdev->tag_set);
+ if (ret)
goto out;
+ rq = blk_mq_init_queue(&bdev->tag_set);
+ if (IS_ERR(rq)) {
+ ret = PTR_ERR(rq);
+ goto out_tag;
+ }
bdev->rq = rq;
nr_max_blk = min(scmdev->nr_max_block,
(unsigned int) (PAGE_SIZE / sizeof(struct aidaw)));
blk_queue_max_segments(rq, nr_max_blk);
queue_flag_set_unlocked(QUEUE_FLAG_NONROT, rq);
queue_flag_clear_unlocked(QUEUE_FLAG_ADD_RANDOM, rq);
- scm_blk_dev_cluster_setup(bdev);
bdev->gendisk = alloc_disk(SCM_NR_PARTS);
- if (!bdev->gendisk)
+ if (!bdev->gendisk) {
+ ret = -ENOMEM;
goto out_queue;
-
+ }
rq->queuedata = scmdev;
bdev->gendisk->private_data = scmdev;
bdev->gendisk->fops = &scm_blk_devops;
out_queue:
blk_cleanup_queue(rq);
+out_tag:
+ blk_mq_free_tag_set(&bdev->tag_set);
out:
atomic_dec(&nr_devices);
return ret;
void scm_blk_dev_cleanup(struct scm_blk_dev *bdev)
{
- tasklet_kill(&bdev->tasklet);
del_gendisk(bdev->gendisk);
blk_cleanup_queue(bdev->gendisk->queue);
+ blk_mq_free_tag_set(&bdev->tag_set);
put_disk(bdev->gendisk);
}
if (!nr_requests_per_io || nr_requests_per_io > 64)
return false;
- return scm_cluster_size_valid();
+ return true;
}
static int __init scm_blk_init(void)
#include <linux/interrupt.h>
#include <linux/spinlock.h>
#include <linux/blkdev.h>
+#include <linux/blk-mq.h>
#include <linux/genhd.h>
#include <linux/list.h>
#define SCM_QUEUE_DELAY 5
struct scm_blk_dev {
- struct tasklet_struct tasklet;
struct request_queue *rq;
struct gendisk *gendisk;
+ struct blk_mq_tag_set tag_set;
struct scm_device *scmdev;
- spinlock_t rq_lock; /* guard the request queue */
- spinlock_t lock; /* guard the rest of the blockdev */
+ spinlock_t lock;
atomic_t queued_reqs;
enum {SCM_OPER, SCM_WR_PROHIBIT} state;
struct list_head finished_requests;
-#ifdef CONFIG_SCM_BLOCK_CLUSTER_WRITE
- struct list_head cluster_list;
-#endif
};
struct scm_request {
struct aob *aob;
struct list_head list;
u8 retries;
- int error;
-#ifdef CONFIG_SCM_BLOCK_CLUSTER_WRITE
- struct {
- enum {CLUSTER_NONE, CLUSTER_READ, CLUSTER_WRITE} state;
- struct list_head list;
- void **buf;
- } cluster;
-#endif
+ blk_status_t error;
};
#define to_aobrq(rq) container_of((void *) rq, struct aob_rq_header, data)
int scm_blk_dev_setup(struct scm_blk_dev *, struct scm_device *);
void scm_blk_dev_cleanup(struct scm_blk_dev *);
void scm_blk_set_available(struct scm_blk_dev *);
-void scm_blk_irq(struct scm_device *, void *, int);
-
-void scm_request_finish(struct scm_request *);
-void scm_request_requeue(struct scm_request *);
+void scm_blk_irq(struct scm_device *, void *, blk_status_t);
struct aidaw *scm_aidaw_fetch(struct scm_request *scmrq, unsigned int bytes);
int scm_drv_init(void);
void scm_drv_cleanup(void);
-#ifdef CONFIG_SCM_BLOCK_CLUSTER_WRITE
-void __scm_free_rq_cluster(struct scm_request *);
-int __scm_alloc_rq_cluster(struct scm_request *);
-void scm_request_cluster_init(struct scm_request *);
-bool scm_reserve_cluster(struct scm_request *);
-void scm_release_cluster(struct scm_request *);
-void scm_blk_dev_cluster_setup(struct scm_blk_dev *);
-bool scm_need_cluster_request(struct scm_request *);
-void scm_initiate_cluster_request(struct scm_request *);
-void scm_cluster_request_irq(struct scm_request *);
-bool scm_test_cluster_request(struct scm_request *);
-bool scm_cluster_size_valid(void);
-#else /* CONFIG_SCM_BLOCK_CLUSTER_WRITE */
-static inline void __scm_free_rq_cluster(struct scm_request *scmrq) {}
-static inline int __scm_alloc_rq_cluster(struct scm_request *scmrq)
-{
- return 0;
-}
-static inline void scm_request_cluster_init(struct scm_request *scmrq) {}
-static inline bool scm_reserve_cluster(struct scm_request *scmrq)
-{
- return true;
-}
-static inline void scm_release_cluster(struct scm_request *scmrq) {}
-static inline void scm_blk_dev_cluster_setup(struct scm_blk_dev *bdev) {}
-static inline bool scm_need_cluster_request(struct scm_request *scmrq)
-{
- return false;
-}
-static inline void scm_initiate_cluster_request(struct scm_request *scmrq) {}
-static inline void scm_cluster_request_irq(struct scm_request *scmrq) {}
-static inline bool scm_test_cluster_request(struct scm_request *scmrq)
-{
- return false;
-}
-static inline bool scm_cluster_size_valid(void)
-{
- return true;
-}
-#endif /* CONFIG_SCM_BLOCK_CLUSTER_WRITE */
-
extern debug_info_t *scm_debug;
#define SCM_LOG(imp, txt) do { \
+++ /dev/null
-/*
- * Block driver for s390 storage class memory.
- *
- * Copyright IBM Corp. 2012
- * Author(s): Sebastian Ott <sebott@linux.vnet.ibm.com>
- */
-
-#include <linux/spinlock.h>
-#include <linux/module.h>
-#include <linux/blkdev.h>
-#include <linux/genhd.h>
-#include <linux/slab.h>
-#include <linux/list.h>
-#include <asm/eadm.h>
-#include "scm_blk.h"
-
-static unsigned int write_cluster_size = 64;
-module_param(write_cluster_size, uint, S_IRUGO);
-MODULE_PARM_DESC(write_cluster_size,
- "Number of pages used for contiguous writes.");
-
-#define CLUSTER_SIZE (write_cluster_size * PAGE_SIZE)
-
-void __scm_free_rq_cluster(struct scm_request *scmrq)
-{
- int i;
-
- if (!scmrq->cluster.buf)
- return;
-
- for (i = 0; i < 2 * write_cluster_size; i++)
- free_page((unsigned long) scmrq->cluster.buf[i]);
-
- kfree(scmrq->cluster.buf);
-}
-
-int __scm_alloc_rq_cluster(struct scm_request *scmrq)
-{
- int i;
-
- scmrq->cluster.buf = kzalloc(sizeof(void *) * 2 * write_cluster_size,
- GFP_KERNEL);
- if (!scmrq->cluster.buf)
- return -ENOMEM;
-
- for (i = 0; i < 2 * write_cluster_size; i++) {
- scmrq->cluster.buf[i] = (void *) get_zeroed_page(GFP_DMA);
- if (!scmrq->cluster.buf[i])
- return -ENOMEM;
- }
- INIT_LIST_HEAD(&scmrq->cluster.list);
- return 0;
-}
-
-void scm_request_cluster_init(struct scm_request *scmrq)
-{
- scmrq->cluster.state = CLUSTER_NONE;
-}
-
-static bool clusters_intersect(struct request *A, struct request *B)
-{
- unsigned long firstA, lastA, firstB, lastB;
-
- firstA = ((u64) blk_rq_pos(A) << 9) / CLUSTER_SIZE;
- lastA = (((u64) blk_rq_pos(A) << 9) +
- blk_rq_bytes(A) - 1) / CLUSTER_SIZE;
-
- firstB = ((u64) blk_rq_pos(B) << 9) / CLUSTER_SIZE;
- lastB = (((u64) blk_rq_pos(B) << 9) +
- blk_rq_bytes(B) - 1) / CLUSTER_SIZE;
-
- return (firstB <= lastA && firstA <= lastB);
-}
-
-bool scm_reserve_cluster(struct scm_request *scmrq)
-{
- struct request *req = scmrq->request[scmrq->aob->request.msb_count];
- struct scm_blk_dev *bdev = scmrq->bdev;
- struct scm_request *iter;
- int pos, add = 1;
-
- if (write_cluster_size == 0)
- return true;
-
- spin_lock(&bdev->lock);
- list_for_each_entry(iter, &bdev->cluster_list, cluster.list) {
- if (iter == scmrq) {
- /*
- * We don't have to use clusters_intersect here, since
- * cluster requests are always started separately.
- */
- add = 0;
- continue;
- }
- for (pos = 0; pos < iter->aob->request.msb_count; pos++) {
- if (clusters_intersect(req, iter->request[pos]) &&
- (rq_data_dir(req) == WRITE ||
- rq_data_dir(iter->request[pos]) == WRITE)) {
- spin_unlock(&bdev->lock);
- return false;
- }
- }
- }
- if (add)
- list_add(&scmrq->cluster.list, &bdev->cluster_list);
- spin_unlock(&bdev->lock);
-
- return true;
-}
-
-void scm_release_cluster(struct scm_request *scmrq)
-{
- struct scm_blk_dev *bdev = scmrq->bdev;
- unsigned long flags;
-
- if (write_cluster_size == 0)
- return;
-
- spin_lock_irqsave(&bdev->lock, flags);
- list_del(&scmrq->cluster.list);
- spin_unlock_irqrestore(&bdev->lock, flags);
-}
-
-void scm_blk_dev_cluster_setup(struct scm_blk_dev *bdev)
-{
- INIT_LIST_HEAD(&bdev->cluster_list);
- blk_queue_io_opt(bdev->rq, CLUSTER_SIZE);
-}
-
-static int scm_prepare_cluster_request(struct scm_request *scmrq)
-{
- struct scm_blk_dev *bdev = scmrq->bdev;
- struct scm_device *scmdev = bdev->gendisk->private_data;
- struct request *req = scmrq->request[0];
- struct msb *msb = &scmrq->aob->msb[0];
- struct req_iterator iter;
- struct aidaw *aidaw;
- struct bio_vec bv;
- int i = 0;
- u64 addr;
-
- switch (scmrq->cluster.state) {
- case CLUSTER_NONE:
- scmrq->cluster.state = CLUSTER_READ;
- /* fall through */
- case CLUSTER_READ:
- msb->bs = MSB_BS_4K;
- msb->oc = MSB_OC_READ;
- msb->flags = MSB_FLAG_IDA;
- msb->blk_count = write_cluster_size;
-
- addr = scmdev->address + ((u64) blk_rq_pos(req) << 9);
- msb->scm_addr = round_down(addr, CLUSTER_SIZE);
-
- if (msb->scm_addr !=
- round_down(addr + (u64) blk_rq_bytes(req) - 1,
- CLUSTER_SIZE))
- msb->blk_count = 2 * write_cluster_size;
-
- aidaw = scm_aidaw_fetch(scmrq, msb->blk_count * PAGE_SIZE);
- if (!aidaw)
- return -ENOMEM;
-
- scmrq->aob->request.msb_count = 1;
- msb->data_addr = (u64) aidaw;
- for (i = 0; i < msb->blk_count; i++) {
- aidaw->data_addr = (u64) scmrq->cluster.buf[i];
- aidaw++;
- }
-
- break;
- case CLUSTER_WRITE:
- aidaw = (void *) msb->data_addr;
- msb->oc = MSB_OC_WRITE;
-
- for (addr = msb->scm_addr;
- addr < scmdev->address + ((u64) blk_rq_pos(req) << 9);
- addr += PAGE_SIZE) {
- aidaw->data_addr = (u64) scmrq->cluster.buf[i];
- aidaw++;
- i++;
- }
- rq_for_each_segment(bv, req, iter) {
- aidaw->data_addr = (u64) page_address(bv.bv_page);
- aidaw++;
- i++;
- }
- for (; i < msb->blk_count; i++) {
- aidaw->data_addr = (u64) scmrq->cluster.buf[i];
- aidaw++;
- }
- break;
- }
- return 0;
-}
-
-bool scm_need_cluster_request(struct scm_request *scmrq)
-{
- int pos = scmrq->aob->request.msb_count;
-
- if (rq_data_dir(scmrq->request[pos]) == READ)
- return false;
-
- return blk_rq_bytes(scmrq->request[pos]) < CLUSTER_SIZE;
-}
-
-/* Called with queue lock held. */
-void scm_initiate_cluster_request(struct scm_request *scmrq)
-{
- if (scm_prepare_cluster_request(scmrq))
- goto requeue;
- if (eadm_start_aob(scmrq->aob))
- goto requeue;
- return;
-requeue:
- scm_request_requeue(scmrq);
-}
-
-bool scm_test_cluster_request(struct scm_request *scmrq)
-{
- return scmrq->cluster.state != CLUSTER_NONE;
-}
-
-void scm_cluster_request_irq(struct scm_request *scmrq)
-{
- struct scm_blk_dev *bdev = scmrq->bdev;
- unsigned long flags;
-
- switch (scmrq->cluster.state) {
- case CLUSTER_NONE:
- BUG();
- break;
- case CLUSTER_READ:
- if (scmrq->error) {
- scm_request_finish(scmrq);
- break;
- }
- scmrq->cluster.state = CLUSTER_WRITE;
- spin_lock_irqsave(&bdev->rq_lock, flags);
- scm_initiate_cluster_request(scmrq);
- spin_unlock_irqrestore(&bdev->rq_lock, flags);
- break;
- case CLUSTER_WRITE:
- scm_request_finish(scmrq);
- break;
- }
-}
-
-bool scm_cluster_size_valid(void)
-{
- if (write_cluster_size == 1 || write_cluster_size > 128)
- return false;
-
- return !(write_cluster_size & (write_cluster_size - 1));
-}
unsigned long page_addr;
unsigned long bytes;
- blk_queue_split(q, &bio, q->bio_split);
+ blk_queue_split(q, &bio);
if ((bio->bi_iter.bi_sector & 7) != 0 ||
(bio->bi_iter.bi_size & 4095) != 0)
.restore = sclp_restore,
};
-static ssize_t sclp_show_console_pages(struct device_driver *dev, char *buf)
+static ssize_t con_pages_show(struct device_driver *dev, char *buf)
{
return sprintf(buf, "%i\n", sclp_console_pages);
}
-static DRIVER_ATTR(con_pages, S_IRUSR, sclp_show_console_pages, NULL);
+static DRIVER_ATTR_RO(con_pages);
-static ssize_t sclp_show_con_drop(struct device_driver *dev, char *buf)
+static ssize_t con_drop_show(struct device_driver *dev, char *buf)
{
return sprintf(buf, "%i\n", sclp_console_drop);
}
-static DRIVER_ATTR(con_drop, S_IRUSR, sclp_show_con_drop, NULL);
+static DRIVER_ATTR_RO(con_drop);
-static ssize_t sclp_show_console_full(struct device_driver *dev, char *buf)
+static ssize_t con_full_show(struct device_driver *dev, char *buf)
{
return sprintf(buf, "%lu\n", sclp_console_full);
}
-static DRIVER_ATTR(con_full, S_IRUSR, sclp_show_console_full, NULL);
+static DRIVER_ATTR_RO(con_full);
static struct attribute *sclp_drv_attrs[] = {
&driver_attr_con_pages.attr,
static DEVICE_ATTR(recording, 0200, NULL, vmlogrdr_recording_store);
-static ssize_t vmlogrdr_recording_status_show(struct device_driver *driver,
- char *buf)
+static ssize_t recording_status_show(struct device_driver *driver, char *buf)
{
-
static const char cp_command[] = "QUERY RECORDING ";
int len;
len = strlen(buf);
return len;
}
-static DRIVER_ATTR(recording_status, 0444, vmlogrdr_recording_status_show,
- NULL);
+static DRIVER_ATTR_RO(recording_status);
static struct attribute *vmlogrdr_drv_attrs[] = {
&driver_attr_recording_status.attr,
NULL,
NULL,
};
+static ssize_t chpids_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct subchannel *sch = to_subchannel(dev);
+ struct chsc_ssd_info *ssd = &sch->ssd_info;
+ ssize_t ret = 0;
+ int mask;
+ int chp;
+
+ for (chp = 0; chp < 8; chp++) {
+ mask = 0x80 >> chp;
+ if (ssd->path_mask & mask)
+ ret += sprintf(buf + ret, "%02x ", ssd->chpid[chp].id);
+ else
+ ret += sprintf(buf + ret, "00 ");
+ }
+ ret += sprintf(buf + ret, "\n");
+ return ret;
+}
+static DEVICE_ATTR(chpids, 0444, chpids_show, NULL);
+
+static ssize_t pimpampom_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct subchannel *sch = to_subchannel(dev);
+ struct pmcw *pmcw = &sch->schib.pmcw;
+
+ return sprintf(buf, "%02x %02x %02x\n",
+ pmcw->pim, pmcw->pam, pmcw->pom);
+}
+static DEVICE_ATTR(pimpampom, 0444, pimpampom_show, NULL);
+
+static struct attribute *io_subchannel_type_attrs[] = {
+ &dev_attr_chpids.attr,
+ &dev_attr_pimpampom.attr,
+ NULL,
+};
+ATTRIBUTE_GROUPS(io_subchannel_type);
+
+static const struct device_type io_subchannel_type = {
+ .groups = io_subchannel_type_groups,
+};
+
int css_register_subchannel(struct subchannel *sch)
{
int ret;
sch->dev.parent = &channel_subsystems[0]->device;
sch->dev.bus = &css_bus_type;
sch->dev.groups = default_subch_attr_groups;
+
+ if (sch->st == SUBCHANNEL_TYPE_IO)
+ sch->dev.type = &io_subchannel_type;
+
/*
* We don't want to generate uevents for I/O subchannels that don't
* have a working ccw device behind them since they will be
/************************ device handling **************************/
-/*
- * A ccw_device has some interfaces in sysfs in addition to the
- * standard ones.
- * The following entries are designed to export the information which
- * resided in 2.4 in /proc/subchannels. Subchannel and device number
- * are obvious, so they don't have an entry :)
- * TODO: Split chpids and pimpampom up? Where is "in use" in the tree?
- */
-static ssize_t
-chpids_show (struct device * dev, struct device_attribute *attr, char * buf)
-{
- struct subchannel *sch = to_subchannel(dev);
- struct chsc_ssd_info *ssd = &sch->ssd_info;
- ssize_t ret = 0;
- int chp;
- int mask;
-
- for (chp = 0; chp < 8; chp++) {
- mask = 0x80 >> chp;
- if (ssd->path_mask & mask)
- ret += sprintf(buf + ret, "%02x ", ssd->chpid[chp].id);
- else
- ret += sprintf(buf + ret, "00 ");
- }
- ret += sprintf (buf+ret, "\n");
- return min((ssize_t)PAGE_SIZE, ret);
-}
-
-static ssize_t
-pimpampom_show (struct device * dev, struct device_attribute *attr, char * buf)
-{
- struct subchannel *sch = to_subchannel(dev);
- struct pmcw *pmcw = &sch->schib.pmcw;
-
- return sprintf (buf, "%02x %02x %02x\n",
- pmcw->pim, pmcw->pam, pmcw->pom);
-}
-
static ssize_t
devtype_show (struct device *dev, struct device_attribute *attr, char *buf)
{
return sprintf(buf, "%02x\n", sch->vpm);
}
-static DEVICE_ATTR(chpids, 0444, chpids_show, NULL);
-static DEVICE_ATTR(pimpampom, 0444, pimpampom_show, NULL);
static DEVICE_ATTR(devtype, 0444, devtype_show, NULL);
static DEVICE_ATTR(cutype, 0444, cutype_show, NULL);
static DEVICE_ATTR(modalias, 0444, modalias_show, NULL);
static DEVICE_ATTR(vpm, 0444, vpm_show, NULL);
static struct attribute *io_subchannel_attrs[] = {
- &dev_attr_chpids.attr,
- &dev_attr_pimpampom.attr,
&dev_attr_logging.attr,
&dev_attr_vpm.attr,
NULL,
struct eadm_private *private = get_eadm_private(sch);
struct eadm_scsw *scsw = &sch->schib.scsw.eadm;
struct irb *irb = this_cpu_ptr(&cio_irb);
- int error = 0;
+ blk_status_t error = BLK_STS_OK;
EADM_LOG(6, "irq");
EADM_LOG_HEX(6, irb, sizeof(*irb));
if ((scsw->stctl & (SCSW_STCTL_ALERT_STATUS | SCSW_STCTL_STATUS_PEND))
&& scsw->eswf == 1 && irb->esw.eadm.erw.r)
- error = -EIO;
+ error = BLK_STS_IOERR;
if (scsw->fctl & SCSW_FCTL_CLEAR_FUNC)
- error = -ETIMEDOUT;
+ error = BLK_STS_TIMEOUT;
eadm_subchannel_set_timeout(sch, 0);
}
EXPORT_SYMBOL_GPL(scm_driver_unregister);
-void scm_irq_handler(struct aob *aob, int error)
+void scm_irq_handler(struct aob *aob, blk_status_t error)
{
struct aob_rq_header *aobrq = (void *) aob->request.data;
struct scm_device *scmdev = aobrq->scmdev;
private->state = VFIO_CCW_STATE_IDLE;
}
-/*
- * Sysfs interfaces
- */
-static ssize_t chpids_show(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct subchannel *sch = to_subchannel(dev);
- struct chsc_ssd_info *ssd = &sch->ssd_info;
- ssize_t ret = 0;
- int chp;
- int mask;
-
- for (chp = 0; chp < 8; chp++) {
- mask = 0x80 >> chp;
- if (ssd->path_mask & mask)
- ret += sprintf(buf + ret, "%02x ", ssd->chpid[chp].id);
- else
- ret += sprintf(buf + ret, "00 ");
- }
- ret += sprintf(buf+ret, "\n");
- return ret;
-}
-
-static ssize_t pimpampom_show(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct subchannel *sch = to_subchannel(dev);
- struct pmcw *pmcw = &sch->schib.pmcw;
-
- return sprintf(buf, "%02x %02x %02x\n",
- pmcw->pim, pmcw->pam, pmcw->pom);
-}
-
-static DEVICE_ATTR(chpids, 0444, chpids_show, NULL);
-static DEVICE_ATTR(pimpampom, 0444, pimpampom_show, NULL);
-
-static struct attribute *vfio_subchannel_attrs[] = {
- &dev_attr_chpids.attr,
- &dev_attr_pimpampom.attr,
- NULL,
-};
-
-static struct attribute_group vfio_subchannel_attr_group = {
- .attrs = vfio_subchannel_attrs,
-};
-
/*
* Css driver callbacks
*/
if (ret)
goto out_free;
- ret = sysfs_create_group(&sch->dev.kobj, &vfio_subchannel_attr_group);
- if (ret)
- goto out_disable;
-
ret = vfio_ccw_mdev_reg(sch);
if (ret)
- goto out_rm_group;
+ goto out_disable;
INIT_WORK(&private->io_work, vfio_ccw_sch_io_todo);
atomic_set(&private->avail, 1);
return 0;
-out_rm_group:
- sysfs_remove_group(&sch->dev.kobj, &vfio_subchannel_attr_group);
out_disable:
cio_disable_subchannel(sch);
out_free:
vfio_ccw_mdev_unreg(sch);
- sysfs_remove_group(&sch->dev.kobj, &vfio_subchannel_attr_group);
-
dev_set_drvdata(&sch->dev, NULL);
kfree(private);
ap_domain_index = domain;
spin_unlock_bh(&ap_domain_lock);
- AP_DBF(DBF_DEBUG, "store new default domain=%d\n", domain);
+ AP_DBF(DBF_DEBUG, "stored new default domain=%d\n", domain);
return count;
}
}
if (best_domain >= 0){
ap_domain_index = best_domain;
+ AP_DBF(DBF_DEBUG, "new ap_domain_index=%d\n", ap_domain_index);
spin_unlock_bh(&ap_domain_lock);
return 0;
}
ap_qid_t qid;
int depth = 0, type = 0;
unsigned int functions = 0;
- int rc, id, dom, borked, domains;
+ int rc, id, dom, borked, domains, defdomdevs = 0;
AP_DBF(DBF_DEBUG, "ap_scan_bus running\n");
put_device(dev);
if (!borked) {
domains++;
+ if (dom == ap_domain_index)
+ defdomdevs++;
continue;
}
}
continue;
}
domains++;
+ if (dom == ap_domain_index)
+ defdomdevs++;
} /* end domain loop */
if (ac) {
/* remove card dev if there are no queue devices */
put_device(&ac->ap_dev.device);
}
} /* end device loop */
+
+ if (defdomdevs < 1)
+ AP_DBF(DBF_INFO, "no queue device with default domain %d available\n",
+ ap_domain_index);
+
out:
mod_timer(&ap_config_timer, jiffies + ap_config_time * HZ);
}
ap_init_configuration();
if (ap_configuration)
- max_domain_id = ap_max_domain_id ? : (AP_DOMAINS - 1);
+ max_domain_id =
+ ap_max_domain_id ? ap_max_domain_id : AP_DOMAINS - 1;
else
max_domain_id = 15;
if (ap_domain_index < -1 || ap_domain_index > max_domain_id) {
pr_warn("%d is not a valid cryptographic domain\n",
ap_domain_index);
- rc = -EINVAL;
- goto out_free;
+ ap_domain_index = -1;
}
/* In resume callback we need to know if the user had set the domain.
* If so, we can not just reset it.
unregister_reset_call(&ap_reset_call);
if (ap_using_interrupts())
unregister_adapter_interrupt(&ap_airq);
-out_free:
kfree(ap_configuration);
return rc;
}
pxcrb->user_defined = (cardnr == 0xFFFF ? AUTOSELECT : cardnr);
pxcrb->request_control_blk_length =
preqcblk->cprb_len + preqcblk->req_parml;
- pxcrb->request_control_blk_addr = (void *) preqcblk;
+ pxcrb->request_control_blk_addr = (void __user *) preqcblk;
pxcrb->reply_control_blk_length = preqcblk->rpl_msgbl;
- pxcrb->reply_control_blk_addr = (void *) prepcblk;
+ pxcrb->reply_control_blk_addr = (void __user *) prepcblk;
}
/*
/*
* Module init
*/
-int __init pkey_init(void)
+static int __init pkey_init(void)
{
cpacf_mask_t pckmo_functions;
do {
rc = zcrypt_rsa_modexpo(&mex);
} while (rc == -EAGAIN);
- if (rc)
+ if (rc) {
+ ZCRYPT_DBF(DBF_DEBUG, "ioctl ICARSAMODEXPO rc=%d", rc);
return rc;
+ }
return put_user(mex.outputdatalength, &umex->outputdatalength);
}
case ICARSACRT: {
do {
rc = zcrypt_rsa_crt(&crt);
} while (rc == -EAGAIN);
- if (rc)
+ if (rc) {
+ ZCRYPT_DBF(DBF_DEBUG, "ioctl ICARSACRT rc=%d", rc);
return rc;
+ }
return put_user(crt.outputdatalength, &ucrt->outputdatalength);
}
case ZSECSENDCPRB: {
do {
rc = zcrypt_send_cprb(&xcRB);
} while (rc == -EAGAIN);
+ if (rc)
+ ZCRYPT_DBF(DBF_DEBUG, "ioctl ZSENDCPRB rc=%d", rc);
if (copy_to_user(uxcRB, &xcRB, sizeof(xcRB)))
return -EFAULT;
return rc;
do {
rc = zcrypt_send_ep11_cprb(&xcrb);
} while (rc == -EAGAIN);
+ if (rc)
+ ZCRYPT_DBF(DBF_DEBUG, "ioctl ZSENDEP11CPRB rc=%d", rc);
if (copy_to_user(uxcrb, &xcrb, sizeof(xcrb)))
return -EFAULT;
return rc;
#define CCA_TKN_HDR_ID_EXT 0x1E
-/**
- * mapping for the cca private ME section
- */
-struct cca_private_ext_ME_sec {
- unsigned char section_identifier;
- unsigned char version;
- unsigned short section_length;
- unsigned char private_key_hash[20];
- unsigned char reserved1[4];
- unsigned char key_format;
- unsigned char reserved2;
- unsigned char key_name_hash[20];
- unsigned char key_use_flags[4];
- unsigned char reserved3[6];
- unsigned char reserved4[24];
- unsigned char confounder[24];
- unsigned char exponent[128];
- unsigned char modulus[128];
-} __attribute__((packed));
-
#define CCA_PVT_USAGE_ALL 0x80
/**
#define CCA_PVT_EXT_CRT_SEC_ID_PVT 0x08
#define CCA_PVT_EXT_CRT_SEC_FMT_CL 0x40
-/**
- * Set up private key fields of a type6 MEX message.
- * Note that all numerics in the key token are big-endian,
- * while the entries in the key block header are little-endian.
- *
- * @mex: pointer to user input data
- * @p: pointer to memory area for the key
- *
- * Returns the size of the key area or -EFAULT
- */
-static inline int zcrypt_type6_mex_key_de(struct ica_rsa_modexpo *mex,
- void *p, int big_endian)
-{
- static struct cca_token_hdr static_pvt_me_hdr = {
- .token_identifier = 0x1E,
- .token_length = 0x0183,
- };
- static struct cca_private_ext_ME_sec static_pvt_me_sec = {
- .section_identifier = 0x02,
- .section_length = 0x016C,
- .key_use_flags = {0x80,0x00,0x00,0x00},
- };
- static struct cca_public_sec static_pub_me_sec = {
- .section_identifier = 0x04,
- .section_length = 0x000F,
- .exponent_len = 0x0003,
- };
- static char pk_exponent[3] = { 0x01, 0x00, 0x01 };
- struct {
- struct T6_keyBlock_hdr t6_hdr;
- struct cca_token_hdr pvtMeHdr;
- struct cca_private_ext_ME_sec pvtMeSec;
- struct cca_public_sec pubMeSec;
- char exponent[3];
- } __attribute__((packed)) *key = p;
- unsigned char *temp;
-
- memset(key, 0, sizeof(*key));
-
- if (big_endian) {
- key->t6_hdr.blen = cpu_to_be16(0x189);
- key->t6_hdr.ulen = cpu_to_be16(0x189 - 2);
- } else {
- key->t6_hdr.blen = cpu_to_le16(0x189);
- key->t6_hdr.ulen = cpu_to_le16(0x189 - 2);
- }
- key->pvtMeHdr = static_pvt_me_hdr;
- key->pvtMeSec = static_pvt_me_sec;
- key->pubMeSec = static_pub_me_sec;
- /*
- * In a private key, the modulus doesn't appear in the public
- * section. So, an arbitrary public exponent of 0x010001 will be
- * used.
- */
- memcpy(key->exponent, pk_exponent, 3);
-
- /* key parameter block */
- temp = key->pvtMeSec.exponent +
- sizeof(key->pvtMeSec.exponent) - mex->inputdatalength;
- if (copy_from_user(temp, mex->b_key, mex->inputdatalength))
- return -EFAULT;
-
- /* modulus */
- temp = key->pvtMeSec.modulus +
- sizeof(key->pvtMeSec.modulus) - mex->inputdatalength;
- if (copy_from_user(temp, mex->n_modulus, mex->inputdatalength))
- return -EFAULT;
- key->pubMeSec.modulus_bit_len = 8 * mex->inputdatalength;
- return sizeof(*key);
-}
-
/**
* Set up private key fields of a type6 MEX message. The _pad variant
* strips leading zeroes from the b_key.
*
* Returns the size of the key area or -EFAULT
*/
-static inline int zcrypt_type6_mex_key_en(struct ica_rsa_modexpo *mex,
- void *p, int big_endian)
+static inline int zcrypt_type6_mex_key_en(struct ica_rsa_modexpo *mex, void *p)
{
static struct cca_token_hdr static_pub_hdr = {
.token_identifier = 0x1E,
2*mex->inputdatalength - i;
key->pubHdr.token_length =
key->pubSec.section_length + sizeof(key->pubHdr);
- if (big_endian) {
- key->t6_hdr.ulen = cpu_to_be16(key->pubHdr.token_length + 4);
- key->t6_hdr.blen = cpu_to_be16(key->pubHdr.token_length + 6);
- } else {
- key->t6_hdr.ulen = cpu_to_le16(key->pubHdr.token_length + 4);
- key->t6_hdr.blen = cpu_to_le16(key->pubHdr.token_length + 6);
- }
+ key->t6_hdr.ulen = key->pubHdr.token_length + 4;
+ key->t6_hdr.blen = key->pubHdr.token_length + 6;
return sizeof(*key) + 2*mex->inputdatalength - i;
}
*
* Returns the size of the key area or -EFAULT
*/
-static inline int zcrypt_type6_crt_key(struct ica_rsa_modexpo_crt *crt,
- void *p, int big_endian)
+static inline int zcrypt_type6_crt_key(struct ica_rsa_modexpo_crt *crt, void *p)
{
static struct cca_public_sec static_cca_pub_sec = {
.section_identifier = 4,
size = sizeof(*key) + key_len + sizeof(*pub) + 3;
/* parameter block.key block */
- if (big_endian) {
- key->t6_hdr.blen = cpu_to_be16(size);
- key->t6_hdr.ulen = cpu_to_be16(size - 2);
- } else {
- key->t6_hdr.blen = cpu_to_le16(size);
- key->t6_hdr.ulen = cpu_to_le16(size - 2);
- }
+ key->t6_hdr.blen = size;
+ key->t6_hdr.ulen = size - 2;
/* key token header */
key->token.token_identifier = CCA_TKN_HDR_ID_EXT;
return -EFAULT;
/* Set up key which is located after the variable length text. */
- size = zcrypt_type6_mex_key_en(mex, msg->text+mex->inputdatalength, 1);
+ size = zcrypt_type6_mex_key_en(mex, msg->text+mex->inputdatalength);
if (size < 0)
return size;
size += sizeof(*msg) + mex->inputdatalength;
return -EFAULT;
/* Set up key which is located after the variable length text. */
- size = zcrypt_type6_crt_key(crt, msg->text + crt->inputdatalength, 1);
+ size = zcrypt_type6_crt_key(crt, msg->text + crt->inputdatalength);
if (size < 0)
return size;
size += sizeof(*msg) + crt->inputdatalength; /* total size of msg */
.restore = ctcm_pm_resume,
};
-static ssize_t ctcm_driver_group_store(struct device_driver *ddrv,
- const char *buf, size_t count)
+static ssize_t group_store(struct device_driver *ddrv, const char *buf,
+ size_t count)
{
int err;
err = ccwgroup_create_dev(ctcm_root_dev, &ctcm_group_driver, 2, buf);
return err ? err : count;
}
-static DRIVER_ATTR(group, 0200, NULL, ctcm_driver_group_store);
+static DRIVER_ATTR_WO(group);
static struct attribute *ctcm_drv_attrs[] = {
&driver_attr_group.attr,
.restore = lcs_restore,
};
-static ssize_t lcs_driver_group_store(struct device_driver *ddrv,
- const char *buf, size_t count)
+static ssize_t group_store(struct device_driver *ddrv, const char *buf,
+ size_t count)
{
int err;
err = ccwgroup_create_dev(lcs_root_dev, &lcs_group_driver, 2, buf);
return err ? err : count;
}
-static DRIVER_ATTR(group, 0200, NULL, lcs_driver_group_store);
+static DRIVER_ATTR_WO(group);
static struct attribute *lcs_drv_attrs[] = {
&driver_attr_group.attr,
return NULL;
}
-static ssize_t conn_write(struct device_driver *drv,
- const char *buf, size_t count)
+static ssize_t connection_store(struct device_driver *drv, const char *buf,
+ size_t count)
{
char username[9];
char userdata[17];
netiucv_free_netdevice(dev);
return rc;
}
+static DRIVER_ATTR_WO(connection);
-static DRIVER_ATTR(connection, 0200, NULL, conn_write);
-
-static ssize_t remove_write (struct device_driver *drv,
- const char *buf, size_t count)
+static ssize_t remove_store(struct device_driver *drv, const char *buf,
+ size_t count)
{
struct iucv_connection *cp;
struct net_device *ndev;
IUCV_DBF_TEXT(data, 2, "remove_write: unknown device\n");
return -EINVAL;
}
-
-static DRIVER_ATTR(remove, 0200, NULL, remove_write);
+static DRIVER_ATTR_WO(remove);
static struct attribute * netiucv_drv_attrs[] = {
&driver_attr_connection.attr,
.restore = qeth_core_restore,
};
-static ssize_t qeth_core_driver_group_store(struct device_driver *ddrv,
- const char *buf, size_t count)
+static ssize_t group_store(struct device_driver *ddrv, const char *buf,
+ size_t count)
{
int err;
return err ? err : count;
}
-static DRIVER_ATTR(group, 0200, NULL, qeth_core_driver_group_store);
+static DRIVER_ATTR_WO(group);
static struct attribute *qeth_drv_attrs[] = {
&driver_attr_group.attr,
struct jsfd_part *jdp = req->rq_disk->private_data;
unsigned long offset = blk_rq_pos(req) << 9;
size_t len = blk_rq_cur_bytes(req);
- int err = -EIO;
+ blk_status_t err = BLK_STS_IOERR;
if ((offset + len) > jdp->dsize)
goto end;
}
jsfd_read(bio_data(req->bio), jdp->dbase + offset, len);
- err = 0;
+ err = BLK_STS_OK;
end:
if (!__blk_end_request_cur(req, err))
req = jsfd_next_request();
put_disk(disk);
goto out;
}
+ blk_queue_bounce_limit(disk->queue, BLK_BOUNCE_HIGH);
jsfd_disk[i] = disk;
}
#include <linux/wait.h>
typedef wait_queue_head_t adpt_wait_queue_head_t;
#define ADPT_DECLARE_WAIT_QUEUE_HEAD(wait) DECLARE_WAIT_QUEUE_HEAD_ONSTACK(wait)
-typedef wait_queue_t adpt_wait_queue_t;
+typedef wait_queue_entry_t adpt_wait_queue_entry_t;
/*
* message structures
static void ibmvscsis_dev_release(struct device *dev) {};
-static struct class_attribute ibmvscsis_class_attrs[] = {
- __ATTR_NULL,
-};
-
static struct device_attribute dev_attr_system_id =
__ATTR(system_id, S_IRUGO, system_id_show, NULL);
static struct class ibmvscsis_class = {
.name = "ibmvscsis",
.dev_release = ibmvscsis_dev_release,
- .class_attrs = ibmvscsis_class_attrs,
.dev_groups = ibmvscsis_dev_groups,
};
static uint32_t ips_statupd_morpheus(ips_ha_t *);
static ips_scb_t *ips_getscb(ips_ha_t *);
static void ips_putq_scb_head(ips_scb_queue_t *, ips_scb_t *);
-static void ips_putq_wait_tail(ips_wait_queue_t *, struct scsi_cmnd *);
+static void ips_putq_wait_tail(ips_wait_queue_entry_t *, struct scsi_cmnd *);
static void ips_putq_copp_tail(ips_copp_queue_t *,
ips_copp_wait_item_t *);
static ips_scb_t *ips_removeq_scb_head(ips_scb_queue_t *);
static ips_scb_t *ips_removeq_scb(ips_scb_queue_t *, ips_scb_t *);
-static struct scsi_cmnd *ips_removeq_wait_head(ips_wait_queue_t *);
-static struct scsi_cmnd *ips_removeq_wait(ips_wait_queue_t *,
+static struct scsi_cmnd *ips_removeq_wait_head(ips_wait_queue_entry_t *);
+static struct scsi_cmnd *ips_removeq_wait(ips_wait_queue_entry_t *,
struct scsi_cmnd *);
static ips_copp_wait_item_t *ips_removeq_copp(ips_copp_queue_t *,
ips_copp_wait_item_t *);
/* ASSUMED to be called from within the HA lock */
/* */
/****************************************************************************/
-static void ips_putq_wait_tail(ips_wait_queue_t *queue, struct scsi_cmnd *item)
+static void ips_putq_wait_tail(ips_wait_queue_entry_t *queue, struct scsi_cmnd *item)
{
METHOD_TRACE("ips_putq_wait_tail", 1);
/* ASSUMED to be called from within the HA lock */
/* */
/****************************************************************************/
-static struct scsi_cmnd *ips_removeq_wait_head(ips_wait_queue_t *queue)
+static struct scsi_cmnd *ips_removeq_wait_head(ips_wait_queue_entry_t *queue)
{
struct scsi_cmnd *item;
/* ASSUMED to be called from within the HA lock */
/* */
/****************************************************************************/
-static struct scsi_cmnd *ips_removeq_wait(ips_wait_queue_t *queue,
+static struct scsi_cmnd *ips_removeq_wait(ips_wait_queue_entry_t *queue,
struct scsi_cmnd *item)
{
struct scsi_cmnd *p;
struct scsi_cmnd *head;
struct scsi_cmnd *tail;
int count;
-} ips_wait_queue_t;
+} ips_wait_queue_entry_t;
typedef struct ips_copp_wait_item {
struct scsi_cmnd *scsi_cmd;
ips_stat_t sp; /* Status packer pointer */
struct ips_scb *scbs; /* Array of all CCBS */
struct ips_scb *scb_freelist; /* SCB free list */
- ips_wait_queue_t scb_waitlist; /* Pending SCB list */
+ ips_wait_queue_entry_t scb_waitlist; /* Pending SCB list */
ips_copp_queue_t copp_waitlist; /* Pending PT list */
ips_scb_queue_t scb_activelist; /* Active SCB list */
IPS_IO_CMD *dummy; /* dummy command */
* code paths.
*/
if (unlikely(rq->bio))
- blk_end_request(rq, -ENOMEM, blk_rq_bytes(rq));
+ blk_end_request(rq, BLK_STS_IOERR, blk_rq_bytes(rq));
else
blk_put_request(rq);
}
EXPORT_SYMBOL(osd_end_request);
static void _set_error_resid(struct osd_request *or, struct request *req,
- int error)
+ blk_status_t error)
{
or->async_error = error;
- or->req_errors = scsi_req(req)->result ? : error;
+ or->req_errors = scsi_req(req)->result;
or->sense_len = scsi_req(req)->sense_len;
if (or->sense_len)
memcpy(or->sense, scsi_req(req)->sense, or->sense_len);
int osd_execute_request(struct osd_request *or)
{
- int error;
-
blk_execute_rq(or->request->q, NULL, or->request, 0);
- error = scsi_req(or->request)->result ? -EIO : 0;
- _set_error_resid(or, or->request, error);
- return error;
+ if (scsi_req(or->request)->result) {
+ _set_error_resid(or, or->request, BLK_STS_IOERR);
+ return -EIO;
+ }
+
+ _set_error_resid(or, or->request, BLK_STS_OK);
+ return 0;
}
EXPORT_SYMBOL(osd_execute_request);
-static void osd_request_async_done(struct request *req, int error)
+static void osd_request_async_done(struct request *req, blk_status_t error)
{
struct osd_request *or = req->end_io_data;
flags);
if (IS_ERR(req))
return req;
- scsi_req_init(req);
for_each_bio(bio) {
- struct bio *bounce_bio = bio;
-
- blk_queue_bounce(req->q, &bounce_bio);
- ret = blk_rq_append_bio(req, bounce_bio);
+ ret = blk_rq_append_bio(req, bio);
if (ret)
return ERR_PTR(ret);
}
ret = PTR_ERR(req);
goto out;
}
- scsi_req_init(req);
or->in.req = or->request->next_rq = req;
}
} else if (has_in)
/* scsi sense is Empty, the request was never issued to target
* linux return code might tell us what happened.
*/
- if (or->async_error == -ENOMEM)
+ if (or->async_error == BLK_STS_RESOURCE)
osi->osd_err_pri = OSD_ERR_PRI_RESOURCE;
else
osi->osd_err_pri = OSD_ERR_PRI_UNREACHABLE;
/* Wakeup from interrupt */
-static void osst_end_async(struct request *req, int update)
+static void osst_end_async(struct request *req, blk_status_t status)
{
struct scsi_request *rq = scsi_req(req);
struct osst_request *SRpnt = req->end_io_data;
return DRIVER_ERROR << 24;
rq = scsi_req(req);
- scsi_req_init(req);
req->rq_flags |= RQF_QUIET;
SRpnt->bio = NULL;
QEDI_ERR(&qedi->dbg_ctx,
"Delayed or untracked cleanup response, itt=0x%x, tid=0x%x, cid=0x%x, task=%p\n",
protoitt, cqe->itid, qedi_conn->iscsi_conn_id, task);
- WARN_ON(1);
}
}
void qedi_clear_task_idx(struct qedi_ctx *qedi, int idx)
{
- if (!test_and_clear_bit(idx, qedi->task_idx_map)) {
+ if (!test_and_clear_bit(idx, qedi->task_idx_map))
QEDI_ERR(&qedi->dbg_ctx,
"FW task context, already cleared, tid=0x%x\n", idx);
- WARN_ON(1);
- }
}
void qedi_update_itt_map(struct qedi_ctx *qedi, u32 tid, u32 proto_itt,
* On PCI bus, order reverses and write of 6 posts, then index 5,
* causing chip to issue full queue of stale commands
* The mmiowb() prevents future writes from crossing the barrier.
- * See Documentation/DocBook/deviceiobook.tmpl for more information.
+ * See Documentation/driver-api/device-io.rst for more information.
*/
WRT_REG_WORD(®->mailbox4, ha->req_ring_index);
mmiowb();
unsigned int channel;
unsigned int target;
u64 lun;
- uuid_be lu_name;
+ uuid_t lu_name;
struct sdebug_host_info *sdbg_host;
unsigned long uas_bm[1];
atomic_t num_in_q;
static int inquiry_vpd_83(unsigned char *arr, int port_group_id,
int target_dev_id, int dev_id_num,
const char *dev_id_str, int dev_id_str_len,
- const uuid_be *lu_name)
+ const uuid_t *lu_name)
{
int num, port_a;
char b[32];
}
static bool got_shared_uuid;
-static uuid_be shared_uuid;
+static uuid_t shared_uuid;
static struct sdebug_dev_info *sdebug_device_create(
struct sdebug_host_info *sdbg_host, gfp_t flags)
devip = kzalloc(sizeof(*devip), flags);
if (devip) {
if (sdebug_uuid_ctl == 1)
- uuid_be_gen(&devip->lu_name);
+ uuid_gen(&devip->lu_name);
else if (sdebug_uuid_ctl == 2) {
if (got_shared_uuid)
devip->lu_name = shared_uuid;
else {
- uuid_be_gen(&shared_uuid);
+ uuid_gen(&shared_uuid);
got_shared_uuid = true;
devip->lu_name = shared_uuid;
}
}
}
-static void eh_lock_door_done(struct request *req, int uptodate)
+static void eh_lock_door_done(struct request *req, blk_status_t status)
{
__blk_put_request(req->q, req);
}
if (IS_ERR(req))
return;
rq = scsi_req(req);
- scsi_req_init(req);
rq->cmd[0] = ALLOW_MEDIUM_REMOVAL;
rq->cmd[1] = 0;
if (IS_ERR(req))
return ret;
rq = scsi_req(req);
- scsi_req_init(req);
if (bufflen && blk_rq_map_kern(sdev->request_queue, req,
buffer, bufflen, __GFP_RECLAIM))
cmd->request->next_rq->special = NULL;
}
-static bool scsi_end_request(struct request *req, int error,
+static bool scsi_end_request(struct request *req, blk_status_t error,
unsigned int bytes, unsigned int bidi_bytes)
{
struct scsi_cmnd *cmd = req->special;
* @cmd: SCSI command (unused)
* @result: scsi error code
*
- * Translate SCSI error code into standard UNIX errno.
- * Return values:
- * -ENOLINK temporary transport failure
- * -EREMOTEIO permanent target failure, do not retry
- * -EBADE permanent nexus failure, retry on other path
- * -ENOSPC No write space available
- * -ENODATA Medium error
- * -EIO unspecified I/O error
+ * Translate SCSI error code into block errors.
*/
-static int __scsi_error_from_host_byte(struct scsi_cmnd *cmd, int result)
+static blk_status_t __scsi_error_from_host_byte(struct scsi_cmnd *cmd,
+ int result)
{
- int error = 0;
-
- switch(host_byte(result)) {
+ switch (host_byte(result)) {
case DID_TRANSPORT_FAILFAST:
- error = -ENOLINK;
- break;
+ return BLK_STS_TRANSPORT;
case DID_TARGET_FAILURE:
set_host_byte(cmd, DID_OK);
- error = -EREMOTEIO;
- break;
+ return BLK_STS_TARGET;
case DID_NEXUS_FAILURE:
- set_host_byte(cmd, DID_OK);
- error = -EBADE;
- break;
+ return BLK_STS_NEXUS;
case DID_ALLOC_FAILURE:
set_host_byte(cmd, DID_OK);
- error = -ENOSPC;
- break;
+ return BLK_STS_NOSPC;
case DID_MEDIUM_ERROR:
set_host_byte(cmd, DID_OK);
- error = -ENODATA;
- break;
+ return BLK_STS_MEDIUM;
default:
- error = -EIO;
- break;
+ return BLK_STS_IOERR;
}
-
- return error;
}
/*
int result = cmd->result;
struct request_queue *q = cmd->device->request_queue;
struct request *req = cmd->request;
- int error = 0;
+ blk_status_t error = BLK_STS_OK;
struct scsi_sense_hdr sshdr;
bool sense_valid = false;
int sense_deferred = 0, level = 0;
* both sides at once.
*/
scsi_req(req->next_rq)->resid_len = scsi_in(cmd)->resid;
- if (scsi_end_request(req, 0, blk_rq_bytes(req),
+ if (scsi_end_request(req, BLK_STS_OK, blk_rq_bytes(req),
blk_rq_bytes(req->next_rq)))
BUG();
return;
scsi_print_sense(cmd);
result = 0;
/* for passthrough error may be set */
- error = 0;
+ error = BLK_STS_OK;
}
/*
action = ACTION_REPREP;
} else if (sshdr.asc == 0x10) /* DIX */ {
action = ACTION_FAIL;
- error = -EILSEQ;
+ error = BLK_STS_PROTECTION;
/* INVALID COMMAND OPCODE or INVALID FIELD IN CDB */
} else if (sshdr.asc == 0x20 || sshdr.asc == 0x24) {
action = ACTION_FAIL;
- error = -EREMOTEIO;
+ error = BLK_STS_TARGET;
} else
action = ACTION_FAIL;
break;
case ABORTED_COMMAND:
action = ACTION_FAIL;
if (sshdr.asc == 0x10) /* DIF */
- error = -EILSEQ;
+ error = BLK_STS_PROTECTION;
break;
case NOT_READY:
/* If the device is in the process of becoming
}
EXPORT_SYMBOL(scsi_init_io);
+/**
+ * scsi_initialize_rq - initialize struct scsi_cmnd.req
+ *
+ * Called from inside blk_get_request().
+ */
+void scsi_initialize_rq(struct request *rq)
+{
+ struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(rq);
+
+ scsi_req_init(&cmd->req);
+}
+EXPORT_SYMBOL(scsi_initialize_rq);
+
+/* Called after a request has been started. */
void scsi_init_command(struct scsi_device *dev, struct scsi_cmnd *cmd)
{
void *buf = cmd->sense_buffer;
blk_delay_queue(q, SCSI_QUEUE_DELAY);
}
-static inline int prep_to_mq(int ret)
+static inline blk_status_t prep_to_mq(int ret)
{
switch (ret) {
case BLKPREP_OK:
- return BLK_MQ_RQ_QUEUE_OK;
+ return BLK_STS_OK;
case BLKPREP_DEFER:
- return BLK_MQ_RQ_QUEUE_BUSY;
+ return BLK_STS_RESOURCE;
default:
- return BLK_MQ_RQ_QUEUE_ERROR;
+ return BLK_STS_IOERR;
}
}
blk_mq_complete_request(cmd->request);
}
-static int scsi_queue_rq(struct blk_mq_hw_ctx *hctx,
+static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,
const struct blk_mq_queue_data *bd)
{
struct request *req = bd->rq;
struct scsi_device *sdev = q->queuedata;
struct Scsi_Host *shost = sdev->host;
struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(req);
- int ret;
+ blk_status_t ret;
int reason;
ret = prep_to_mq(scsi_prep_state_check(sdev, req));
- if (ret != BLK_MQ_RQ_QUEUE_OK)
+ if (ret != BLK_STS_OK)
goto out;
- ret = BLK_MQ_RQ_QUEUE_BUSY;
+ ret = BLK_STS_RESOURCE;
if (!get_device(&sdev->sdev_gendev))
goto out;
if (!(req->rq_flags & RQF_DONTPREP)) {
ret = prep_to_mq(scsi_mq_prep_fn(req));
- if (ret != BLK_MQ_RQ_QUEUE_OK)
+ if (ret != BLK_STS_OK)
goto out_dec_host_busy;
req->rq_flags |= RQF_DONTPREP;
} else {
reason = scsi_dispatch_cmd(cmd);
if (reason) {
scsi_set_blocked(cmd, reason);
- ret = BLK_MQ_RQ_QUEUE_BUSY;
+ ret = BLK_STS_RESOURCE;
goto out_dec_host_busy;
}
- return BLK_MQ_RQ_QUEUE_OK;
+ return BLK_STS_OK;
out_dec_host_busy:
atomic_dec(&shost->host_busy);
put_device(&sdev->sdev_gendev);
out:
switch (ret) {
- case BLK_MQ_RQ_QUEUE_BUSY:
+ case BLK_STS_OK:
+ break;
+ case BLK_STS_RESOURCE:
if (atomic_read(&sdev->device_busy) == 0 &&
!scsi_device_blocked(sdev))
blk_mq_delay_run_hw_queue(hctx, SCSI_QUEUE_DELAY);
break;
- case BLK_MQ_RQ_QUEUE_ERROR:
+ default:
/*
* Make sure to release all allocated ressources when
* we hit an error, as we will never see this command
if (req->rq_flags & RQF_DONTPREP)
scsi_mq_uninit_cmd(cmd);
break;
- default:
- break;
}
return ret;
}
{
struct device *dev = shost->dma_dev;
+ queue_flag_set_unlocked(QUEUE_FLAG_SCSI_PASSTHROUGH, q);
+
/*
* this limit is imposed by hardware restrictions
*/
q->request_fn = scsi_request_fn;
q->init_rq_fn = scsi_init_rq;
q->exit_rq_fn = scsi_exit_rq;
+ q->initialize_rq_fn = scsi_initialize_rq;
if (blk_init_allocated_queue(q) < 0) {
blk_cleanup_queue(q);
#endif
.init_request = scsi_init_request,
.exit_request = scsi_exit_request,
+ .initialize_rq_fn = scsi_initialize_rq,
.map_queues = scsi_map_queues,
};
if (wait)
blk_mq_quiesce_queue(q);
else
- blk_mq_stop_hw_queues(q);
+ blk_mq_quiesce_queue_nowait(q);
} else {
spin_lock_irqsave(q->queue_lock, flags);
blk_stop_queue(q);
return -EINVAL;
if (q->mq_ops) {
- blk_mq_start_stopped_hw_queues(q, false);
+ blk_mq_unquiesce_queue(q);
} else {
spin_lock_irqsave(q->queue_lock, flags);
blk_start_queue(q);
* allocate and set it up by calling scsi_add_lun.
*
* Return:
- * SCSI_SCAN_NO_RESPONSE: could not allocate or setup a scsi_device
- * SCSI_SCAN_TARGET_PRESENT: target responded, but no device is
+ *
+ * - SCSI_SCAN_NO_RESPONSE: could not allocate or setup a scsi_device
+ * - SCSI_SCAN_TARGET_PRESENT: target responded, but no device is
* attached at the LUN
- * SCSI_SCAN_LUN_PRESENT: a new scsi_device was allocated and initialized
+ * - SCSI_SCAN_LUN_PRESENT: a new scsi_device was allocated and initialized
**/
static int scsi_probe_and_add_lun(struct scsi_target *starget,
u64 lun, int *bflagsp,
* port is no longer part of the topology. Note: Although a port
* may no longer be part of the topology, it may persist in the remote
* ports displayed by the fc_host. We do this under 2 conditions:
+ *
* 1) If the port was a scsi target, we delay its deletion by "blocking" it.
- * This allows the port to temporarily disappear, then reappear without
- * disrupting the SCSI device tree attached to it. During the "blocked"
- * period the port will still exist.
+ * This allows the port to temporarily disappear, then reappear without
+ * disrupting the SCSI device tree attached to it. During the "blocked"
+ * period the port will still exist.
+ *
* 2) If the port was a scsi target and disappears for longer than we
- * expect, we'll delete the port and the tear down the SCSI device tree
- * attached to it. However, we want to semi-persist the target id assigned
- * to that port if it eventually does exist. The port structure will
- * remain (although with minimal information) so that the target id
- * bindings remails.
+ * expect, we'll delete the port and the tear down the SCSI device tree
+ * attached to it. However, we want to semi-persist the target id assigned
+ * to that port if it eventually does exist. The port structure will
+ * remain (although with minimal information) so that the target id
+ * bindings remails.
*
* If the remote port is not an FCP Target, it will be fully torn down
* and deallocated, including the fc_remote_port class device.
#include <linux/bsg.h>
#include <scsi/scsi.h>
+#include <scsi/scsi_cmnd.h>
#include <scsi/scsi_request.h>
#include <scsi/scsi_device.h>
#include <scsi/scsi_host.h>
struct sas_rphy *rphy)
{
struct request *req;
- int ret;
+ blk_status_t ret;
int (*handler)(struct Scsi_Host *, struct sas_rphy *, struct request *);
while ((req = blk_fetch_request(q)) != NULL) {
q = blk_alloc_queue(GFP_KERNEL);
if (!q)
return -ENOMEM;
+ q->initialize_rq_fn = scsi_initialize_rq;
q->cmd_size = sizeof(struct scsi_request);
if (rphy) {
if (error)
goto out_cleanup_queue;
+ /*
+ * by default assume old behaviour and bounce for any highmem page
+ */
+ blk_queue_bounce_limit(q, BLK_BOUNCE_HIGH);
+
error = bsg_register_queue(q, dev, name, release);
if (error)
goto out_cleanup_queue;
q->queuedata = shost;
queue_flag_set_unlocked(QUEUE_FLAG_BIDI, q);
+ queue_flag_set_unlocked(QUEUE_FLAG_SCSI_PASSTHROUGH, q);
return 0;
out_cleanup_queue:
* @hds: put heads here
* @secs: put sectors here
*
- * Description: determine the BIOS mapping/geometry used to create the partition
- * table, storing the results in *cyls, *hds, and *secs
+ * Determine the BIOS mapping/geometry used to create the partition
+ * table, storing the results in @cyls, @hds, and @secs
*
* Returns: -1 on failure, 0 on success.
*/
} Sg_device;
/* tasklet or soft irq callback */
-static void sg_rq_end_io(struct request *rq, int uptodate);
+static void sg_rq_end_io(struct request *rq, blk_status_t status);
static int sg_start_req(Sg_request *srp, unsigned char *cmd);
static int sg_finish_rem_req(Sg_request * srp);
static int sg_build_indirect(Sg_scatter_hold * schp, Sg_fd * sfp, int buff_size);
if (atomic_read(&sdp->detaching)) {
if (srp->bio) {
scsi_req_free_cmd(scsi_req(srp->rq));
- blk_end_request_all(srp->rq, -EIO);
+ blk_end_request_all(srp->rq, BLK_STS_IOERR);
srp->rq = NULL;
}
* level when a command is completed (or has failed).
*/
static void
-sg_rq_end_io(struct request *rq, int uptodate)
+sg_rq_end_io(struct request *rq, blk_status_t status)
{
struct sg_request *srp = rq->end_io_data;
struct scsi_request *req = scsi_req(rq);
}
req = scsi_req(rq);
- scsi_req_init(rq);
-
if (hp->cmd_len > BLK_MAX_CDB)
req->cmd = long_cmdp;
memcpy(req->cmd, cmd, hp->cmd_len);
atomic64_dec(&STp->stats->in_flight);
}
-static void st_scsi_execute_end(struct request *req, int uptodate)
+static void st_scsi_execute_end(struct request *req, blk_status_t status)
{
struct st_request *SRpnt = req->end_io_data;
struct scsi_request *rq = scsi_req(req);
if (IS_ERR(req))
return DRIVER_ERROR << 24;
rq = scsi_req(req);
- scsi_req_init(req);
req->rq_flags |= RQF_QUIET;
mdata->null_mapped = 1;
{ \
struct superhyway_device *s = to_superhyway_device(dev); \
return sprintf(buf, fmt, s->field); \
-}
+} \
+static DEVICE_ATTR_RO(name);
/* VCR flags */
superhyway_ro_attr(perr_flags, "0x%02x\n", vcr.perr_flags);
/* Misc */
superhyway_ro_attr(resource, "0x%08lx\n", resource[0].start);
-struct device_attribute superhyway_dev_attrs[] = {
- __ATTR_RO(perr_flags),
- __ATTR_RO(merr_flags),
- __ATTR_RO(mod_vers),
- __ATTR_RO(mod_id),
- __ATTR_RO(bot_mb),
- __ATTR_RO(top_mb),
- __ATTR_RO(resource),
- __ATTR_NULL,
+static struct attribute *superhyway_dev_attrs[] = {
+ &dev_attr_perr_flags.attr,
+ &dev_attr_merr_flags.attr,
+ &dev_attr_mod_vers.attr,
+ &dev_attr_mod_id.attr,
+ &dev_attr_bot_mb.attr,
+ &dev_attr_top_mb.attr,
+ &dev_attr_resource.attr,
+ NULL,
};
+static const struct attribute_group superhyway_dev_group = {
+ .attrs = superhyway_dev_attrs,
+};
+
+const struct attribute_group *superhyway_dev_groups[] = {
+ &superhyway_dev_group,
+ NULL,
+};
.name = "superhyway",
.match = superhyway_bus_match,
#ifdef CONFIG_SYSFS
- .dev_attrs = superhyway_dev_attrs,
+ .dev_groups = superhyway_dev_groups,
#endif
.probe = superhyway_device_probe,
.remove = superhyway_device_remove,
From MPC8536, 85xx platform uses the controller, and all P10xx,
P20xx, P30xx,P40xx, P50xx uses this controller.
+config SPI_MESON_SPICC
+ tristate "Amlogic Meson SPICC controller"
+ depends on ARCH_MESON || COMPILE_TEST
+ help
+ This enables master mode support for the SPICC (SPI communication
+ controller) available in Amlogic Meson SoCs.
+
config SPI_MESON_SPIFC
tristate "Amlogic Meson SPIFC controller"
depends on ARCH_MESON || COMPILE_TEST
config SPI_TI_QSPI
tristate "DRA7xxx QSPI controller support"
+ depends on HAS_DMA
depends on ARCH_OMAP2PLUS || COMPILE_TEST
help
QSPI master controller for DRA7xxx used for flash devices.
help
SPI driver for CSR SiRFprimaII SoCs
+config SPI_STM32
+ tristate "STMicroelectronics STM32 SPI controller"
+ depends on ARCH_STM32 || COMPILE_TEST
+ help
+ SPI driver for STMicroelectonics STM32 SoCs.
+
+ STM32 SPI controller supports DMA and PIO modes. When DMA
+ is not available, the driver automatically falls back to
+ PIO mode.
+
config SPI_ST_SSC4
tristate "STMicroelectronics SPI SSC-based driver"
depends on ARCH_STI || COMPILE_TEST
endif # SPI_MASTER
-# (slave support would go here)
+#
+# SLAVE side ... listening to other SPI masters
+#
+
+config SPI_SLAVE
+ bool "SPI slave protocol handlers"
+ help
+ If your system has a slave-capable SPI controller, you can enable
+ slave protocol handlers.
+
+if SPI_SLAVE
+
+config SPI_SLAVE_TIME
+ tristate "SPI slave handler reporting boot up time"
+ help
+ SPI slave handler responding with the time of reception of the last
+ SPI message.
+
+config SPI_SLAVE_SYSTEM_CONTROL
+ tristate "SPI slave handler controlling system state"
+ help
+ SPI slave handler to allow remote control of system reboot, power
+ off, halt, and suspend.
+
+endif # SPI_SLAVE
endif # SPI
obj-$(CONFIG_SPI_JCORE) += spi-jcore.o
obj-$(CONFIG_SPI_LM70_LLP) += spi-lm70llp.o
obj-$(CONFIG_SPI_LP8841_RTC) += spi-lp8841-rtc.o
+obj-$(CONFIG_SPI_MESON_SPICC) += spi-meson-spicc.o
obj-$(CONFIG_SPI_MESON_SPIFC) += spi-meson-spifc.o
obj-$(CONFIG_SPI_MPC512x_PSC) += spi-mpc512x-psc.o
obj-$(CONFIG_SPI_MPC52xx_PSC) += spi-mpc52xx-psc.o
obj-$(CONFIG_SPI_SH_MSIOF) += spi-sh-msiof.o
obj-$(CONFIG_SPI_SH_SCI) += spi-sh-sci.o
obj-$(CONFIG_SPI_SIRF) += spi-sirf.o
+obj-$(CONFIG_SPI_STM32) += spi-stm32.o
obj-$(CONFIG_SPI_ST_SSC4) += spi-st-ssc4.o
obj-$(CONFIG_SPI_SUN4I) += spi-sun4i.o
obj-$(CONFIG_SPI_SUN6I) += spi-sun6i.o
obj-$(CONFIG_SPI_XLP) += spi-xlp.o
obj-$(CONFIG_SPI_XTENSA_XTFPGA) += spi-xtensa-xtfpga.o
obj-$(CONFIG_SPI_ZYNQMP_GQSPI) += spi-zynqmp-gqspi.o
+
+# SPI slave protocol handlers
+obj-$(CONFIG_SPI_SLAVE_TIME) += spi-slave-time.o
+obj-$(CONFIG_SPI_SLAVE_SYSTEM_CONTROL) += spi-slave-system-control.o
bool is_spi2;
bool has_wdrbt;
bool has_dma_support;
+ bool has_pdc_support;
};
/*
unsigned int version;
version = atmel_get_version(as);
- dev_info(&as->pdev->dev, "version: 0x%x\n", version);
as->caps.is_spi2 = version > 0x121;
as->caps.has_wdrbt = version >= 0x210;
+#ifdef CONFIG_SOC_SAM_V4_V5
+ /*
+ * Atmel SoCs based on ARM9 (SAM9x) cores should not use spi_map_buf()
+ * since this later function tries to map buffers with dma_map_sg()
+ * even if they have not been allocated inside DMA-safe areas.
+ * On SoCs based on Cortex A5 (SAMA5Dx), it works anyway because for
+ * those ARM cores, the data cache follows the PIPT model.
+ * Also the L2 cache controller of SAMA5D2 uses the PIPT model too.
+ * In case of PIPT caches, there cannot be cache aliases.
+ * However on ARM9 cores, the data cache follows the VIVT model, hence
+ * the cache aliases issue can occur when buffers are allocated from
+ * DMA-unsafe areas, by vmalloc() for instance, where cache coherency is
+ * not taken into account or at least not handled completely (cache
+ * lines of aliases are not invalidated).
+ * This is not a theorical issue: it was reproduced when trying to mount
+ * a UBI file-system on a at91sam9g35ek board.
+ */
+ as->caps.has_dma_support = false;
+#else
as->caps.has_dma_support = version >= 0x212;
+#endif
+ as->caps.has_pdc_support = version < 0x212;
}
/*-------------------------------------------------------------------------*/
} else if (ret == -EPROBE_DEFER) {
return ret;
}
- } else {
+ } else if (as->caps.has_pdc_support) {
as->use_pdc = true;
}
goto out_free_dma;
/* go! */
- dev_info(&pdev->dev, "Atmel SPI Controller at 0x%08lx (irq %d)\n",
- (unsigned long)regs->start, irq);
+ dev_info(&pdev->dev, "Atmel SPI Controller version 0x%x at 0x%08lx (irq %d)\n",
+ atmel_get_version(as), (unsigned long)regs->start,
+ irq);
return 0;
{ .compatible = "brcm,bcm6328-hsspi", },
{ },
};
+MODULE_DEVICE_TABLE(of, bcm63xx_hsspi_of_match);
static struct platform_driver bcm63xx_hsspi_driver = {
.driver = {
/* Platform data */
const unsigned long *reg_offsets;
- unsigned fifo_size;
+ unsigned int fifo_size;
unsigned int msg_type_shift;
unsigned int msg_ctl_width;
#endif
}
-static const unsigned bcm63xx_spi_freq_table[SPI_CLK_MASK][2] = {
+static const unsigned int bcm63xx_spi_freq_table[SPI_CLK_MASK][2] = {
{ 20000000, SPI_CLK_20MHZ },
{ 12500000, SPI_CLK_12_50MHZ },
{ 6250000, SPI_CLK_6_250MHZ },
return 0;
}
#else
-static struct davinci_spi_platform_data
- *spi_davinci_get_pdata(struct platform_device *pdev,
- struct davinci_spi *dspi)
+static int spi_davinci_get_pdata(struct platform_device *pdev,
+ struct davinci_spi *dspi)
{
return -ENODEV;
}
ret = -ENODEV;
goto free_master;
}
- clk_prepare_enable(dspi->clk);
+ ret = clk_prepare_enable(dspi->clk);
+ if (ret)
+ goto free_master;
master->dev.of_node = pdev->dev.of_node;
master->bus_num = pdev->id;
goto out_master_put;
if (dspi->devtype_data->trans_mode == DSPI_DMA_MODE) {
- if (dspi_request_dma(dspi, res->start)) {
+ ret = dspi_request_dma(dspi, res->start);
+ if (ret < 0) {
dev_err(&pdev->dev, "can't get dma channels\n");
goto out_clk_put;
}
/* The maximum bytes that a sdma BD can transfer.*/
#define MAX_SDMA_BD_BYTES (1 << 15)
-struct spi_imx_config {
- unsigned int speed_hz;
- unsigned int bpw;
-};
enum spi_imx_devtype {
IMX1_CSPI,
struct spi_imx_devtype_data {
void (*intctrl)(struct spi_imx_data *, int);
- int (*config)(struct spi_device *, struct spi_imx_config *);
+ int (*config)(struct spi_device *);
void (*trigger)(struct spi_imx_data *);
int (*rx_available)(struct spi_imx_data *);
void (*reset)(struct spi_imx_data *);
unsigned long spi_clk;
unsigned int spi_bus_clk;
- unsigned int bytes_per_word;
+ unsigned int speed_hz;
+ unsigned int bits_per_word;
unsigned int spi_drctl;
unsigned int count;
return i;
}
-static int spi_imx_bytes_per_word(const int bpw)
+static int spi_imx_bytes_per_word(const int bits_per_word)
{
- return DIV_ROUND_UP(bpw, BITS_PER_BYTE);
+ return DIV_ROUND_UP(bits_per_word, BITS_PER_BYTE);
}
static bool spi_imx_can_dma(struct spi_master *master, struct spi_device *spi,
struct spi_transfer *transfer)
{
struct spi_imx_data *spi_imx = spi_master_get_devdata(master);
- unsigned int bpw, i;
+ unsigned int bytes_per_word, i;
if (!master->dma_rx)
return false;
- if (!transfer)
- return false;
-
- bpw = transfer->bits_per_word;
- if (!bpw)
- bpw = spi->bits_per_word;
-
- bpw = spi_imx_bytes_per_word(bpw);
+ bytes_per_word = spi_imx_bytes_per_word(transfer->bits_per_word);
- if (bpw != 1 && bpw != 2 && bpw != 4)
+ if (bytes_per_word != 1 && bytes_per_word != 2 && bytes_per_word != 4)
return false;
for (i = spi_imx_get_fifosize(spi_imx) / 2; i > 0; i--) {
- if (!(transfer->len % (i * bpw)))
+ if (!(transfer->len % (i * bytes_per_word)))
break;
}
writel(reg, spi_imx->base + MX51_ECSPI_CTRL);
}
-static int mx51_ecspi_config(struct spi_device *spi,
- struct spi_imx_config *config)
+static int mx51_ecspi_config(struct spi_device *spi)
{
struct spi_imx_data *spi_imx = spi_master_get_devdata(spi->master);
u32 ctrl = MX51_ECSPI_CTRL_ENABLE;
- u32 clk = config->speed_hz, delay, reg;
+ u32 clk = spi_imx->speed_hz, delay, reg;
u32 cfg = readl(spi_imx->base + MX51_ECSPI_CONFIG);
/*
ctrl |= MX51_ECSPI_CTRL_DRCTL(spi_imx->spi_drctl);
/* set clock speed */
- ctrl |= mx51_ecspi_clkdiv(spi_imx, config->speed_hz, &clk);
+ ctrl |= mx51_ecspi_clkdiv(spi_imx, spi_imx->speed_hz, &clk);
spi_imx->spi_bus_clk = clk;
/* set chip select to use */
ctrl |= MX51_ECSPI_CTRL_CS(spi->chip_select);
- ctrl |= (config->bpw - 1) << MX51_ECSPI_CTRL_BL_OFFSET;
+ ctrl |= (spi_imx->bits_per_word - 1) << MX51_ECSPI_CTRL_BL_OFFSET;
cfg |= MX51_ECSPI_CONFIG_SBBCTRL(spi->chip_select);
writel(reg, spi_imx->base + MXC_CSPICTRL);
}
-static int mx31_config(struct spi_device *spi, struct spi_imx_config *config)
+static int mx31_config(struct spi_device *spi)
{
struct spi_imx_data *spi_imx = spi_master_get_devdata(spi->master);
unsigned int reg = MX31_CSPICTRL_ENABLE | MX31_CSPICTRL_MASTER;
unsigned int clk;
- reg |= spi_imx_clkdiv_2(spi_imx->spi_clk, config->speed_hz, &clk) <<
+ reg |= spi_imx_clkdiv_2(spi_imx->spi_clk, spi_imx->speed_hz, &clk) <<
MX31_CSPICTRL_DR_SHIFT;
spi_imx->spi_bus_clk = clk;
if (is_imx35_cspi(spi_imx)) {
- reg |= (config->bpw - 1) << MX35_CSPICTRL_BL_SHIFT;
+ reg |= (spi_imx->bits_per_word - 1) << MX35_CSPICTRL_BL_SHIFT;
reg |= MX31_CSPICTRL_SSCTL;
} else {
- reg |= (config->bpw - 1) << MX31_CSPICTRL_BC_SHIFT;
+ reg |= (spi_imx->bits_per_word - 1) << MX31_CSPICTRL_BC_SHIFT;
}
if (spi->mode & SPI_CPHA)
writel(reg, spi_imx->base + MXC_CSPICTRL);
}
-static int mx21_config(struct spi_device *spi, struct spi_imx_config *config)
+static int mx21_config(struct spi_device *spi)
{
struct spi_imx_data *spi_imx = spi_master_get_devdata(spi->master);
unsigned int reg = MX21_CSPICTRL_ENABLE | MX21_CSPICTRL_MASTER;
unsigned int max = is_imx27_cspi(spi_imx) ? 16 : 18;
unsigned int clk;
- reg |= spi_imx_clkdiv_1(spi_imx->spi_clk, config->speed_hz, max, &clk)
+ reg |= spi_imx_clkdiv_1(spi_imx->spi_clk, spi_imx->speed_hz, max, &clk)
<< MX21_CSPICTRL_DR_SHIFT;
spi_imx->spi_bus_clk = clk;
- reg |= config->bpw - 1;
+ reg |= spi_imx->bits_per_word - 1;
if (spi->mode & SPI_CPHA)
reg |= MX21_CSPICTRL_PHA;
writel(reg, spi_imx->base + MXC_CSPICTRL);
}
-static int mx1_config(struct spi_device *spi, struct spi_imx_config *config)
+static int mx1_config(struct spi_device *spi)
{
struct spi_imx_data *spi_imx = spi_master_get_devdata(spi->master);
unsigned int reg = MX1_CSPICTRL_ENABLE | MX1_CSPICTRL_MASTER;
unsigned int clk;
- reg |= spi_imx_clkdiv_2(spi_imx->spi_clk, config->speed_hz, &clk) <<
+ reg |= spi_imx_clkdiv_2(spi_imx->spi_clk, spi_imx->speed_hz, &clk) <<
MX1_CSPICTRL_DR_SHIFT;
spi_imx->spi_bus_clk = clk;
- reg |= config->bpw - 1;
+ reg |= spi_imx->bits_per_word - 1;
if (spi->mode & SPI_CPHA)
reg |= MX1_CSPICTRL_PHA;
return IRQ_HANDLED;
}
-static int spi_imx_dma_configure(struct spi_master *master,
- int bytes_per_word)
+static int spi_imx_dma_configure(struct spi_master *master)
{
int ret;
enum dma_slave_buswidth buswidth;
struct dma_slave_config rx = {}, tx = {};
struct spi_imx_data *spi_imx = spi_master_get_devdata(master);
- switch (bytes_per_word) {
+ switch (spi_imx_bytes_per_word(spi_imx->bits_per_word)) {
case 4:
buswidth = DMA_SLAVE_BUSWIDTH_4_BYTES;
break;
return ret;
}
- spi_imx->bytes_per_word = bytes_per_word;
-
return 0;
}
struct spi_transfer *t)
{
struct spi_imx_data *spi_imx = spi_master_get_devdata(spi->master);
- struct spi_imx_config config;
int ret;
- config.bpw = t ? t->bits_per_word : spi->bits_per_word;
- config.speed_hz = t ? t->speed_hz : spi->max_speed_hz;
+ if (!t)
+ return 0;
- if (!config.speed_hz)
- config.speed_hz = spi->max_speed_hz;
- if (!config.bpw)
- config.bpw = spi->bits_per_word;
+ spi_imx->bits_per_word = t->bits_per_word;
+ spi_imx->speed_hz = t->speed_hz;
/* Initialize the functions for transfer */
- if (config.bpw <= 8) {
+ if (spi_imx->bits_per_word <= 8) {
spi_imx->rx = spi_imx_buf_rx_u8;
spi_imx->tx = spi_imx_buf_tx_u8;
- } else if (config.bpw <= 16) {
+ } else if (spi_imx->bits_per_word <= 16) {
spi_imx->rx = spi_imx_buf_rx_u16;
spi_imx->tx = spi_imx_buf_tx_u16;
} else {
spi_imx->usedma = 0;
if (spi_imx->usedma) {
- ret = spi_imx_dma_configure(spi->master,
- spi_imx_bytes_per_word(config.bpw));
+ ret = spi_imx_dma_configure(spi->master);
if (ret)
return ret;
}
- spi_imx->devtype_data->config(spi, &config);
+ spi_imx->devtype_data->config(spi);
return 0;
}
goto err;
}
- spi_imx_dma_configure(master, 1);
-
init_completion(&spi_imx->dma_rx_completion);
init_completion(&spi_imx->dma_tx_completion);
master->can_dma = spi_imx_can_dma;
}
master = spi_alloc_master(&pdev->dev, sizeof(struct spi_imx_data));
+ if (!master)
+ return -ENOMEM;
+
ret = of_property_read_u32(np, "fsl,spi-rdy-drctl", &spi_drctl);
if ((ret < 0) || (spi_drctl >= 0x3)) {
/* '11' is reserved */
spi_drctl = 0;
}
- if (!master)
- return -ENOMEM;
-
platform_set_drvdata(pdev, master);
master->bits_per_word_mask = SPI_BPW_RANGE_MASK(1, 32);
test->elapsed_time = ktime_to_ns(ktime_sub(ktime_get(), start));
if (ret == -ETIMEDOUT) {
dev_info(&spi->dev,
- "spi-message timed out - reruning...\n");
+ "spi-message timed out - rerunning...\n");
/* rerun after a few explicit schedules */
for (i = 0; i < 16; i++)
schedule();
rx = vmalloc(SPI_TEST_MAX_SIZE_PLUS);
else
rx = kzalloc(SPI_TEST_MAX_SIZE_PLUS, GFP_KERNEL);
- if (!rx) {
- ret = -ENOMEM;
- goto out;
- }
+ if (!rx)
+ return -ENOMEM;
+
if (use_vmalloc)
tx = vmalloc(SPI_TEST_MAX_SIZE_PLUS);
tx = kzalloc(SPI_TEST_MAX_SIZE_PLUS, GFP_KERNEL);
if (!tx) {
ret = -ENOMEM;
- goto out;
+ goto err_tx;
}
/* now run the individual tests in the table */
}
out:
- kvfree(rx);
kvfree(tx);
+err_tx:
+ kvfree(rx);
return ret;
}
EXPORT_SYMBOL_GPL(spi_test_run_tests);
--- /dev/null
+/*
+ * Driver for Amlogic Meson SPI communication controller (SPICC)
+ *
+ * Copyright (C) BayLibre, SAS
+ * Author: Neil Armstrong <narmstrong@baylibre.com>
+ *
+ * SPDX-License-Identifier: GPL-2.0+
+ */
+
+#include <linux/bitfield.h>
+#include <linux/clk.h>
+#include <linux/device.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+#include <linux/spi/spi.h>
+#include <linux/types.h>
+#include <linux/interrupt.h>
+#include <linux/reset.h>
+#include <linux/gpio.h>
+
+/*
+ * The Meson SPICC controller could support DMA based transfers, but is not
+ * implemented by the vendor code, and while having the registers documentation
+ * it has never worked on the GXL Hardware.
+ * The PIO mode is the only mode implemented, and due to badly designed HW :
+ * - all transfers are cutted in 16 words burst because the FIFO hangs on
+ * TX underflow, and there is no TX "Half-Empty" interrupt, so we go by
+ * FIFO max size chunk only
+ * - CS management is dumb, and goes UP between every burst, so is really a
+ * "Data Valid" signal than a Chip Select, GPIO link should be used instead
+ * to have a CS go down over the full transfer
+ */
+
+#define SPICC_MAX_FREQ 30000000
+#define SPICC_MAX_BURST 128
+
+/* Register Map */
+#define SPICC_RXDATA 0x00
+
+#define SPICC_TXDATA 0x04
+
+#define SPICC_CONREG 0x08
+#define SPICC_ENABLE BIT(0)
+#define SPICC_MODE_MASTER BIT(1)
+#define SPICC_XCH BIT(2)
+#define SPICC_SMC BIT(3)
+#define SPICC_POL BIT(4)
+#define SPICC_PHA BIT(5)
+#define SPICC_SSCTL BIT(6)
+#define SPICC_SSPOL BIT(7)
+#define SPICC_DRCTL_MASK GENMASK(9, 8)
+#define SPICC_DRCTL_IGNORE 0
+#define SPICC_DRCTL_FALLING 1
+#define SPICC_DRCTL_LOWLEVEL 2
+#define SPICC_CS_MASK GENMASK(13, 12)
+#define SPICC_DATARATE_MASK GENMASK(18, 16)
+#define SPICC_DATARATE_DIV4 0
+#define SPICC_DATARATE_DIV8 1
+#define SPICC_DATARATE_DIV16 2
+#define SPICC_DATARATE_DIV32 3
+#define SPICC_BITLENGTH_MASK GENMASK(24, 19)
+#define SPICC_BURSTLENGTH_MASK GENMASK(31, 25)
+
+#define SPICC_INTREG 0x0c
+#define SPICC_TE_EN BIT(0) /* TX FIFO Empty Interrupt */
+#define SPICC_TH_EN BIT(1) /* TX FIFO Half-Full Interrupt */
+#define SPICC_TF_EN BIT(2) /* TX FIFO Full Interrupt */
+#define SPICC_RR_EN BIT(3) /* RX FIFO Ready Interrupt */
+#define SPICC_RH_EN BIT(4) /* RX FIFO Half-Full Interrupt */
+#define SPICC_RF_EN BIT(5) /* RX FIFO Full Interrupt */
+#define SPICC_RO_EN BIT(6) /* RX FIFO Overflow Interrupt */
+#define SPICC_TC_EN BIT(7) /* Transfert Complete Interrupt */
+
+#define SPICC_DMAREG 0x10
+#define SPICC_DMA_ENABLE BIT(0)
+#define SPICC_TXFIFO_THRESHOLD_MASK GENMASK(5, 1)
+#define SPICC_RXFIFO_THRESHOLD_MASK GENMASK(10, 6)
+#define SPICC_READ_BURST_MASK GENMASK(14, 11)
+#define SPICC_WRITE_BURST_MASK GENMASK(18, 15)
+#define SPICC_DMA_URGENT BIT(19)
+#define SPICC_DMA_THREADID_MASK GENMASK(25, 20)
+#define SPICC_DMA_BURSTNUM_MASK GENMASK(31, 26)
+
+#define SPICC_STATREG 0x14
+#define SPICC_TE BIT(0) /* TX FIFO Empty Interrupt */
+#define SPICC_TH BIT(1) /* TX FIFO Half-Full Interrupt */
+#define SPICC_TF BIT(2) /* TX FIFO Full Interrupt */
+#define SPICC_RR BIT(3) /* RX FIFO Ready Interrupt */
+#define SPICC_RH BIT(4) /* RX FIFO Half-Full Interrupt */
+#define SPICC_RF BIT(5) /* RX FIFO Full Interrupt */
+#define SPICC_RO BIT(6) /* RX FIFO Overflow Interrupt */
+#define SPICC_TC BIT(7) /* Transfert Complete Interrupt */
+
+#define SPICC_PERIODREG 0x18
+#define SPICC_PERIOD GENMASK(14, 0) /* Wait cycles */
+
+#define SPICC_TESTREG 0x1c
+#define SPICC_TXCNT_MASK GENMASK(4, 0) /* TX FIFO Counter */
+#define SPICC_RXCNT_MASK GENMASK(9, 5) /* RX FIFO Counter */
+#define SPICC_SMSTATUS_MASK GENMASK(12, 10) /* State Machine Status */
+#define SPICC_LBC_RO BIT(13) /* Loop Back Control Read-Only */
+#define SPICC_LBC_W1 BIT(14) /* Loop Back Control Write-Only */
+#define SPICC_SWAP_RO BIT(14) /* RX FIFO Data Swap Read-Only */
+#define SPICC_SWAP_W1 BIT(15) /* RX FIFO Data Swap Write-Only */
+#define SPICC_DLYCTL_RO_MASK GENMASK(20, 15) /* Delay Control Read-Only */
+#define SPICC_DLYCTL_W1_MASK GENMASK(21, 16) /* Delay Control Write-Only */
+#define SPICC_FIFORST_RO_MASK GENMASK(22, 21) /* FIFO Softreset Read-Only */
+#define SPICC_FIFORST_W1_MASK GENMASK(23, 22) /* FIFO Softreset Write-Only */
+
+#define SPICC_DRADDR 0x20 /* Read Address of DMA */
+
+#define SPICC_DWADDR 0x24 /* Write Address of DMA */
+
+#define writel_bits_relaxed(mask, val, addr) \
+ writel_relaxed((readl_relaxed(addr) & ~(mask)) | (val), addr)
+
+#define SPICC_BURST_MAX 16
+#define SPICC_FIFO_HALF 10
+
+struct meson_spicc_device {
+ struct spi_master *master;
+ struct platform_device *pdev;
+ void __iomem *base;
+ struct clk *core;
+ struct spi_message *message;
+ struct spi_transfer *xfer;
+ u8 *tx_buf;
+ u8 *rx_buf;
+ unsigned int bytes_per_word;
+ unsigned long tx_remain;
+ unsigned long txb_remain;
+ unsigned long rx_remain;
+ unsigned long rxb_remain;
+ unsigned long xfer_remain;
+ bool is_burst_end;
+ bool is_last_burst;
+};
+
+static inline bool meson_spicc_txfull(struct meson_spicc_device *spicc)
+{
+ return !!FIELD_GET(SPICC_TF,
+ readl_relaxed(spicc->base + SPICC_STATREG));
+}
+
+static inline bool meson_spicc_rxready(struct meson_spicc_device *spicc)
+{
+ return FIELD_GET(SPICC_RH | SPICC_RR | SPICC_RF_EN,
+ readl_relaxed(spicc->base + SPICC_STATREG));
+}
+
+static inline u32 meson_spicc_pull_data(struct meson_spicc_device *spicc)
+{
+ unsigned int bytes = spicc->bytes_per_word;
+ unsigned int byte_shift = 0;
+ u32 data = 0;
+ u8 byte;
+
+ while (bytes--) {
+ byte = *spicc->tx_buf++;
+ data |= (byte & 0xff) << byte_shift;
+ byte_shift += 8;
+ }
+
+ spicc->tx_remain--;
+ return data;
+}
+
+static inline void meson_spicc_push_data(struct meson_spicc_device *spicc,
+ u32 data)
+{
+ unsigned int bytes = spicc->bytes_per_word;
+ unsigned int byte_shift = 0;
+ u8 byte;
+
+ while (bytes--) {
+ byte = (data >> byte_shift) & 0xff;
+ *spicc->rx_buf++ = byte;
+ byte_shift += 8;
+ }
+
+ spicc->rx_remain--;
+}
+
+static inline void meson_spicc_rx(struct meson_spicc_device *spicc)
+{
+ /* Empty RX FIFO */
+ while (spicc->rx_remain &&
+ meson_spicc_rxready(spicc))
+ meson_spicc_push_data(spicc,
+ readl_relaxed(spicc->base + SPICC_RXDATA));
+}
+
+static inline void meson_spicc_tx(struct meson_spicc_device *spicc)
+{
+ /* Fill Up TX FIFO */
+ while (spicc->tx_remain &&
+ !meson_spicc_txfull(spicc))
+ writel_relaxed(meson_spicc_pull_data(spicc),
+ spicc->base + SPICC_TXDATA);
+}
+
+static inline u32 meson_spicc_setup_rx_irq(struct meson_spicc_device *spicc,
+ u32 irq_ctrl)
+{
+ if (spicc->rx_remain > SPICC_FIFO_HALF)
+ irq_ctrl |= SPICC_RH_EN;
+ else
+ irq_ctrl |= SPICC_RR_EN;
+
+ return irq_ctrl;
+}
+
+static inline void meson_spicc_setup_burst(struct meson_spicc_device *spicc,
+ unsigned int burst_len)
+{
+ /* Setup Xfer variables */
+ spicc->tx_remain = burst_len;
+ spicc->rx_remain = burst_len;
+ spicc->xfer_remain -= burst_len * spicc->bytes_per_word;
+ spicc->is_burst_end = false;
+ if (burst_len < SPICC_BURST_MAX || !spicc->xfer_remain)
+ spicc->is_last_burst = true;
+ else
+ spicc->is_last_burst = false;
+
+ /* Setup burst length */
+ writel_bits_relaxed(SPICC_BURSTLENGTH_MASK,
+ FIELD_PREP(SPICC_BURSTLENGTH_MASK,
+ burst_len),
+ spicc->base + SPICC_CONREG);
+
+ /* Fill TX FIFO */
+ meson_spicc_tx(spicc);
+}
+
+static irqreturn_t meson_spicc_irq(int irq, void *data)
+{
+ struct meson_spicc_device *spicc = (void *) data;
+ u32 ctrl = readl_relaxed(spicc->base + SPICC_INTREG);
+ u32 stat = readl_relaxed(spicc->base + SPICC_STATREG) & ctrl;
+
+ ctrl &= ~(SPICC_RH_EN | SPICC_RR_EN);
+
+ /* Empty RX FIFO */
+ meson_spicc_rx(spicc);
+
+ /* Enable TC interrupt since we transferred everything */
+ if (!spicc->tx_remain && !spicc->rx_remain) {
+ spicc->is_burst_end = true;
+
+ /* Enable TC interrupt */
+ ctrl |= SPICC_TC_EN;
+
+ /* Reload IRQ status */
+ stat = readl_relaxed(spicc->base + SPICC_STATREG) & ctrl;
+ }
+
+ /* Check transfer complete */
+ if ((stat & SPICC_TC) && spicc->is_burst_end) {
+ unsigned int burst_len;
+
+ /* Clear TC bit */
+ writel_relaxed(SPICC_TC, spicc->base + SPICC_STATREG);
+
+ /* Disable TC interrupt */
+ ctrl &= ~SPICC_TC_EN;
+
+ if (spicc->is_last_burst) {
+ /* Disable all IRQs */
+ writel(0, spicc->base + SPICC_INTREG);
+
+ spi_finalize_current_transfer(spicc->master);
+
+ return IRQ_HANDLED;
+ }
+
+ burst_len = min_t(unsigned int,
+ spicc->xfer_remain / spicc->bytes_per_word,
+ SPICC_BURST_MAX);
+
+ /* Setup burst */
+ meson_spicc_setup_burst(spicc, burst_len);
+
+ /* Restart burst */
+ writel_bits_relaxed(SPICC_XCH, SPICC_XCH,
+ spicc->base + SPICC_CONREG);
+ }
+
+ /* Setup RX interrupt trigger */
+ ctrl = meson_spicc_setup_rx_irq(spicc, ctrl);
+
+ /* Reconfigure interrupts */
+ writel(ctrl, spicc->base + SPICC_INTREG);
+
+ return IRQ_HANDLED;
+}
+
+static u32 meson_spicc_setup_speed(struct meson_spicc_device *spicc, u32 conf,
+ u32 speed)
+{
+ unsigned long parent, value;
+ unsigned int i, div;
+
+ parent = clk_get_rate(spicc->core);
+
+ /* Find closest inferior/equal possible speed */
+ for (i = 0 ; i < 7 ; ++i) {
+ /* 2^(data_rate+2) */
+ value = parent >> (i + 2);
+
+ if (value <= speed)
+ break;
+ }
+
+ /* If provided speed it lower than max divider, use max divider */
+ if (i > 7) {
+ div = 7;
+ dev_warn_once(&spicc->pdev->dev, "unable to get close to speed %u\n",
+ speed);
+ } else
+ div = i;
+
+ dev_dbg(&spicc->pdev->dev, "parent %lu, speed %u -> %lu (%u)\n",
+ parent, speed, value, div);
+
+ conf &= ~SPICC_DATARATE_MASK;
+ conf |= FIELD_PREP(SPICC_DATARATE_MASK, div);
+
+ return conf;
+}
+
+static void meson_spicc_setup_xfer(struct meson_spicc_device *spicc,
+ struct spi_transfer *xfer)
+{
+ u32 conf, conf_orig;
+
+ /* Read original configuration */
+ conf = conf_orig = readl_relaxed(spicc->base + SPICC_CONREG);
+
+ /* Select closest divider */
+ conf = meson_spicc_setup_speed(spicc, conf, xfer->speed_hz);
+
+ /* Setup word width */
+ conf &= ~SPICC_BITLENGTH_MASK;
+ conf |= FIELD_PREP(SPICC_BITLENGTH_MASK,
+ (spicc->bytes_per_word << 3) - 1);
+
+ /* Ignore if unchanged */
+ if (conf != conf_orig)
+ writel_relaxed(conf, spicc->base + SPICC_CONREG);
+}
+
+static int meson_spicc_transfer_one(struct spi_master *master,
+ struct spi_device *spi,
+ struct spi_transfer *xfer)
+{
+ struct meson_spicc_device *spicc = spi_master_get_devdata(master);
+ unsigned int burst_len;
+ u32 irq = 0;
+
+ /* Store current transfer */
+ spicc->xfer = xfer;
+
+ /* Setup transfer parameters */
+ spicc->tx_buf = (u8 *)xfer->tx_buf;
+ spicc->rx_buf = (u8 *)xfer->rx_buf;
+ spicc->xfer_remain = xfer->len;
+
+ /* Pre-calculate word size */
+ spicc->bytes_per_word =
+ DIV_ROUND_UP(spicc->xfer->bits_per_word, 8);
+
+ /* Setup transfer parameters */
+ meson_spicc_setup_xfer(spicc, xfer);
+
+ burst_len = min_t(unsigned int,
+ spicc->xfer_remain / spicc->bytes_per_word,
+ SPICC_BURST_MAX);
+
+ meson_spicc_setup_burst(spicc, burst_len);
+
+ irq = meson_spicc_setup_rx_irq(spicc, irq);
+
+ /* Start burst */
+ writel_bits_relaxed(SPICC_XCH, SPICC_XCH, spicc->base + SPICC_CONREG);
+
+ /* Enable interrupts */
+ writel_relaxed(irq, spicc->base + SPICC_INTREG);
+
+ return 1;
+}
+
+static int meson_spicc_prepare_message(struct spi_master *master,
+ struct spi_message *message)
+{
+ struct meson_spicc_device *spicc = spi_master_get_devdata(master);
+ struct spi_device *spi = message->spi;
+ u32 conf = 0;
+
+ /* Store current message */
+ spicc->message = message;
+
+ /* Enable Master */
+ conf |= SPICC_ENABLE;
+ conf |= SPICC_MODE_MASTER;
+
+ /* SMC = 0 */
+
+ /* Setup transfer mode */
+ if (spi->mode & SPI_CPOL)
+ conf |= SPICC_POL;
+ else
+ conf &= ~SPICC_POL;
+
+ if (spi->mode & SPI_CPHA)
+ conf |= SPICC_PHA;
+ else
+ conf &= ~SPICC_PHA;
+
+ /* SSCTL = 0 */
+
+ if (spi->mode & SPI_CS_HIGH)
+ conf |= SPICC_SSPOL;
+ else
+ conf &= ~SPICC_SSPOL;
+
+ if (spi->mode & SPI_READY)
+ conf |= FIELD_PREP(SPICC_DRCTL_MASK, SPICC_DRCTL_LOWLEVEL);
+ else
+ conf |= FIELD_PREP(SPICC_DRCTL_MASK, SPICC_DRCTL_IGNORE);
+
+ /* Select CS */
+ conf |= FIELD_PREP(SPICC_CS_MASK, spi->chip_select);
+
+ /* Default Clock rate core/4 */
+
+ /* Default 8bit word */
+ conf |= FIELD_PREP(SPICC_BITLENGTH_MASK, 8 - 1);
+
+ writel_relaxed(conf, spicc->base + SPICC_CONREG);
+
+ /* Setup no wait cycles by default */
+ writel_relaxed(0, spicc->base + SPICC_PERIODREG);
+
+ writel_bits_relaxed(BIT(24), BIT(24), spicc->base + SPICC_TESTREG);
+
+ return 0;
+}
+
+static int meson_spicc_unprepare_transfer(struct spi_master *master)
+{
+ struct meson_spicc_device *spicc = spi_master_get_devdata(master);
+
+ /* Disable all IRQs */
+ writel(0, spicc->base + SPICC_INTREG);
+
+ /* Disable controller */
+ writel_bits_relaxed(SPICC_ENABLE, 0, spicc->base + SPICC_CONREG);
+
+ device_reset_optional(&spicc->pdev->dev);
+
+ return 0;
+}
+
+static int meson_spicc_setup(struct spi_device *spi)
+{
+ int ret = 0;
+
+ if (!spi->controller_state)
+ spi->controller_state = spi_master_get_devdata(spi->master);
+ else if (gpio_is_valid(spi->cs_gpio))
+ goto out_gpio;
+ else if (spi->cs_gpio == -ENOENT)
+ return 0;
+
+ if (gpio_is_valid(spi->cs_gpio)) {
+ ret = gpio_request(spi->cs_gpio, dev_name(&spi->dev));
+ if (ret) {
+ dev_err(&spi->dev, "failed to request cs gpio\n");
+ return ret;
+ }
+ }
+
+out_gpio:
+ ret = gpio_direction_output(spi->cs_gpio,
+ !(spi->mode & SPI_CS_HIGH));
+
+ return ret;
+}
+
+static void meson_spicc_cleanup(struct spi_device *spi)
+{
+ if (gpio_is_valid(spi->cs_gpio))
+ gpio_free(spi->cs_gpio);
+
+ spi->controller_state = NULL;
+}
+
+static int meson_spicc_probe(struct platform_device *pdev)
+{
+ struct spi_master *master;
+ struct meson_spicc_device *spicc;
+ struct resource *res;
+ int ret, irq, rate;
+
+ master = spi_alloc_master(&pdev->dev, sizeof(*spicc));
+ if (!master) {
+ dev_err(&pdev->dev, "master allocation failed\n");
+ return -ENOMEM;
+ }
+ spicc = spi_master_get_devdata(master);
+ spicc->master = master;
+
+ spicc->pdev = pdev;
+ platform_set_drvdata(pdev, spicc);
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ spicc->base = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(spicc->base)) {
+ dev_err(&pdev->dev, "io resource mapping failed\n");
+ ret = PTR_ERR(spicc->base);
+ goto out_master;
+ }
+
+ /* Disable all IRQs */
+ writel_relaxed(0, spicc->base + SPICC_INTREG);
+
+ irq = platform_get_irq(pdev, 0);
+ ret = devm_request_irq(&pdev->dev, irq, meson_spicc_irq,
+ 0, NULL, spicc);
+ if (ret) {
+ dev_err(&pdev->dev, "irq request failed\n");
+ goto out_master;
+ }
+
+ spicc->core = devm_clk_get(&pdev->dev, "core");
+ if (IS_ERR(spicc->core)) {
+ dev_err(&pdev->dev, "core clock request failed\n");
+ ret = PTR_ERR(spicc->core);
+ goto out_master;
+ }
+
+ ret = clk_prepare_enable(spicc->core);
+ if (ret) {
+ dev_err(&pdev->dev, "core clock enable failed\n");
+ goto out_master;
+ }
+ rate = clk_get_rate(spicc->core);
+
+ device_reset_optional(&pdev->dev);
+
+ master->num_chipselect = 4;
+ master->dev.of_node = pdev->dev.of_node;
+ master->mode_bits = SPI_CPHA | SPI_CPOL | SPI_CS_HIGH;
+ master->bits_per_word_mask = SPI_BPW_MASK(32) |
+ SPI_BPW_MASK(24) |
+ SPI_BPW_MASK(16) |
+ SPI_BPW_MASK(8);
+ master->flags = (SPI_MASTER_MUST_RX | SPI_MASTER_MUST_TX);
+ master->min_speed_hz = rate >> 9;
+ master->setup = meson_spicc_setup;
+ master->cleanup = meson_spicc_cleanup;
+ master->prepare_message = meson_spicc_prepare_message;
+ master->unprepare_transfer_hardware = meson_spicc_unprepare_transfer;
+ master->transfer_one = meson_spicc_transfer_one;
+
+ /* Setup max rate according to the Meson GX datasheet */
+ if ((rate >> 2) > SPICC_MAX_FREQ)
+ master->max_speed_hz = SPICC_MAX_FREQ;
+ else
+ master->max_speed_hz = rate >> 2;
+
+ ret = devm_spi_register_master(&pdev->dev, master);
+ if (!ret)
+ return 0;
+
+ dev_err(&pdev->dev, "spi master registration failed\n");
+
+out_master:
+ spi_master_put(master);
+
+ return ret;
+}
+
+static int meson_spicc_remove(struct platform_device *pdev)
+{
+ struct meson_spicc_device *spicc = platform_get_drvdata(pdev);
+
+ /* Disable SPI */
+ writel(0, spicc->base + SPICC_CONREG);
+
+ clk_disable_unprepare(spicc->core);
+
+ return 0;
+}
+
+static const struct of_device_id meson_spicc_of_match[] = {
+ { .compatible = "amlogic,meson-gx-spicc", },
+ { /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, meson_spicc_of_match);
+
+static struct platform_driver meson_spicc_driver = {
+ .probe = meson_spicc_probe,
+ .remove = meson_spicc_remove,
+ .driver = {
+ .name = "meson-spicc",
+ .of_match_table = of_match_ptr(meson_spicc_of_match),
+ },
+};
+
+module_platform_driver(meson_spicc_driver);
+
+MODULE_DESCRIPTION("Meson SPI Communication Controller driver");
+MODULE_AUTHOR("Neil Armstrong <narmstrong@baylibre.com>");
+MODULE_LICENSE("GPL");
#define SPI_CMD_REG 0x0018
#define SPI_STATUS0_REG 0x001c
#define SPI_PAD_SEL_REG 0x0024
+#define SPI_CFG2_REG 0x0028
#define SPI_CFG0_SCK_HIGH_OFFSET 0
#define SPI_CFG0_SCK_LOW_OFFSET 8
#define SPI_CFG0_CS_HOLD_OFFSET 16
#define SPI_CFG0_CS_SETUP_OFFSET 24
+#define SPI_ADJUST_CFG0_SCK_LOW_OFFSET 16
+#define SPI_ADJUST_CFG0_CS_HOLD_OFFSET 0
+#define SPI_ADJUST_CFG0_CS_SETUP_OFFSET 16
#define SPI_CFG1_CS_IDLE_OFFSET 0
#define SPI_CFG1_PACKET_LOOP_OFFSET 8
#define SPI_CMD_RST BIT(2)
#define SPI_CMD_PAUSE_EN BIT(4)
#define SPI_CMD_DEASSERT BIT(5)
+#define SPI_CMD_SAMPLE_SEL BIT(6)
+#define SPI_CMD_CS_POL BIT(7)
#define SPI_CMD_CPHA BIT(8)
#define SPI_CMD_CPOL BIT(9)
#define SPI_CMD_RX_DMA BIT(10)
bool need_pad_sel;
/* Must explicitly send dummy Tx bytes to do Rx only transfer */
bool must_tx;
+ /* some IC design adjust cfg register to enhance time accuracy */
+ bool enhance_timing;
};
struct mtk_spi {
};
static const struct mtk_spi_compatible mtk_common_compat;
+
+static const struct mtk_spi_compatible mt2712_compat = {
+ .must_tx = true,
+};
+
+static const struct mtk_spi_compatible mt7622_compat = {
+ .must_tx = true,
+ .enhance_timing = true,
+};
+
static const struct mtk_spi_compatible mt8173_compat = {
.need_pad_sel = true,
.must_tx = true,
static const struct mtk_chip_config mtk_default_chip_info = {
.rx_mlsb = 1,
.tx_mlsb = 1,
+ .cs_pol = 0,
+ .sample_sel = 0,
};
static const struct of_device_id mtk_spi_of_match[] = {
{ .compatible = "mediatek,mt2701-spi",
.data = (void *)&mtk_common_compat,
},
+ { .compatible = "mediatek,mt2712-spi",
+ .data = (void *)&mt2712_compat,
+ },
{ .compatible = "mediatek,mt6589-spi",
.data = (void *)&mtk_common_compat,
},
+ { .compatible = "mediatek,mt7622-spi",
+ .data = (void *)&mt7622_compat,
+ },
{ .compatible = "mediatek,mt8135-spi",
.data = (void *)&mtk_common_compat,
},
reg_val |= SPI_CMD_RX_ENDIAN;
#endif
+ if (mdata->dev_comp->enhance_timing) {
+ if (chip_config->cs_pol)
+ reg_val |= SPI_CMD_CS_POL;
+ else
+ reg_val &= ~SPI_CMD_CS_POL;
+ if (chip_config->sample_sel)
+ reg_val |= SPI_CMD_SAMPLE_SEL;
+ else
+ reg_val &= ~SPI_CMD_SAMPLE_SEL;
+ }
+
/* set finish and pause interrupt always enable */
reg_val |= SPI_CMD_FINISH_IE | SPI_CMD_PAUSE_IE;
sck_time = (div + 1) / 2;
cs_time = sck_time * 2;
- reg_val |= (((sck_time - 1) & 0xff) << SPI_CFG0_SCK_HIGH_OFFSET);
- reg_val |= (((sck_time - 1) & 0xff) << SPI_CFG0_SCK_LOW_OFFSET);
- reg_val |= (((cs_time - 1) & 0xff) << SPI_CFG0_CS_HOLD_OFFSET);
- reg_val |= (((cs_time - 1) & 0xff) << SPI_CFG0_CS_SETUP_OFFSET);
- writel(reg_val, mdata->base + SPI_CFG0_REG);
+ if (mdata->dev_comp->enhance_timing) {
+ reg_val |= (((sck_time - 1) & 0xffff)
+ << SPI_CFG0_SCK_HIGH_OFFSET);
+ reg_val |= (((sck_time - 1) & 0xffff)
+ << SPI_ADJUST_CFG0_SCK_LOW_OFFSET);
+ writel(reg_val, mdata->base + SPI_CFG2_REG);
+ reg_val |= (((cs_time - 1) & 0xffff)
+ << SPI_ADJUST_CFG0_CS_HOLD_OFFSET);
+ reg_val |= (((cs_time - 1) & 0xffff)
+ << SPI_ADJUST_CFG0_CS_SETUP_OFFSET);
+ writel(reg_val, mdata->base + SPI_CFG0_REG);
+ } else {
+ reg_val |= (((sck_time - 1) & 0xff)
+ << SPI_CFG0_SCK_HIGH_OFFSET);
+ reg_val |= (((sck_time - 1) & 0xff) << SPI_CFG0_SCK_LOW_OFFSET);
+ reg_val |= (((cs_time - 1) & 0xff) << SPI_CFG0_CS_HOLD_OFFSET);
+ reg_val |= (((cs_time - 1) & 0xff) << SPI_CFG0_CS_SETUP_OFFSET);
+ writel(reg_val, mdata->base + SPI_CFG0_REG);
+ }
reg_val = readl(mdata->base + SPI_CFG1_REG);
reg_val &= ~SPI_CFG1_CS_IDLE_MASK;
sprintf(mcspi->dma_channels[i].dma_tx_ch_name, "tx%d", i);
}
- if (status < 0)
- goto free_master;
-
pm_runtime_use_autosuspend(&pdev->dev);
pm_runtime_set_autosuspend_delay(&pdev->dev, SPI_AUTOSUSPEND_TIMEOUT);
pm_runtime_enable(&pdev->dev);
#include <linux/of_device.h>
#include <linux/clk.h>
#include <linux/sizes.h>
+#include <linux/gpio.h>
#include <asm/unaligned.h>
#define DRIVER_NAME "orion_spi"
static void orion_spi_set_cs(struct spi_device *spi, bool enable)
{
struct orion_spi *orion_spi;
+ int cs;
+
+ if (gpio_is_valid(spi->cs_gpio))
+ cs = 0;
+ else
+ cs = spi->chip_select;
orion_spi = spi_master_get_devdata(spi->master);
orion_spi_clrbits(orion_spi, ORION_SPI_IF_CTRL_REG, ORION_SPI_CS_MASK);
orion_spi_setbits(orion_spi, ORION_SPI_IF_CTRL_REG,
- ORION_SPI_CS(spi->chip_select));
+ ORION_SPI_CS(cs));
/* Chip select logic is inverted from spi_set_cs */
if (!enable)
master->setup = orion_spi_setup;
master->bits_per_word_mask = SPI_BPW_MASK(8) | SPI_BPW_MASK(16);
master->auto_runtime_pm = true;
+ master->flags = SPI_MASTER_GPIO_SS;
platform_set_drvdata(pdev, master);
.cs_sel_shift = 8,
.cs_sel_mask = 3 << 8,
},
+ { /* LPSS_CNL_SSP */
+ .offset = 0x200,
+ .reg_general = -1,
+ .reg_ssp = 0x20,
+ .reg_cs_ctrl = 0x24,
+ .reg_capabilities = 0xfc,
+ .rx_threshold = 1,
+ .tx_threshold_lo = 32,
+ .tx_threshold_hi = 56,
+ .cs_sel_shift = 8,
+ .cs_sel_mask = 3 << 8,
+ },
};
static inline const struct lpss_config
case LPSS_BSW_SSP:
case LPSS_SPT_SSP:
case LPSS_BXT_SSP:
+ case LPSS_CNL_SSP:
return true;
default:
return false;
case LPSS_BSW_SSP:
case LPSS_SPT_SSP:
case LPSS_BXT_SSP:
+ case LPSS_CNL_SSP:
config = lpss_get_config(drv_data);
tx_thres = config->tx_threshold_lo;
tx_hi_thres = config->tx_threshold_hi;
{ PCI_VDEVICE(INTEL, 0x5ac2), LPSS_BXT_SSP },
{ PCI_VDEVICE(INTEL, 0x5ac4), LPSS_BXT_SSP },
{ PCI_VDEVICE(INTEL, 0x5ac6), LPSS_BXT_SSP },
+ /* CNL-LP */
+ { PCI_VDEVICE(INTEL, 0x9daa), LPSS_CNL_SSP },
+ { PCI_VDEVICE(INTEL, 0x9dab), LPSS_CNL_SSP },
+ { PCI_VDEVICE(INTEL, 0x9dfb), LPSS_CNL_SSP },
+ /* CNL-H */
+ { PCI_VDEVICE(INTEL, 0xa32a), LPSS_CNL_SSP },
+ { PCI_VDEVICE(INTEL, 0xa32b), LPSS_CNL_SSP },
+ { PCI_VDEVICE(INTEL, 0xa37b), LPSS_CNL_SSP },
{ },
};
#define DRIVER_NAME "rockchip-spi"
+#define ROCKCHIP_SPI_CLR_BITS(reg, bits) \
+ writel_relaxed(readl_relaxed(reg) & ~(bits), reg)
+#define ROCKCHIP_SPI_SET_BITS(reg, bits) \
+ writel_relaxed(readl_relaxed(reg) | (bits), reg)
+
/* SPI register offsets */
#define ROCKCHIP_SPI_CTRLR0 0x0000
#define ROCKCHIP_SPI_CTRLR1 0x0004
*/
#define ROCKCHIP_SPI_MAX_TRANLEN 0xffff
+#define ROCKCHIP_SPI_MAX_CS_NUM 2
+
enum rockchip_ssi_type {
SSI_MOTO_SPI = 0,
SSI_TI_SSP,
/* protect state */
spinlock_t lock;
+ bool cs_asserted[ROCKCHIP_SPI_MAX_CS_NUM];
+
u32 use_dma;
struct sg_table tx_sg;
struct sg_table rx_sg;
static void rockchip_spi_set_cs(struct spi_device *spi, bool enable)
{
- u32 ser;
struct spi_master *master = spi->master;
struct rockchip_spi *rs = spi_master_get_devdata(master);
+ bool cs_asserted = !enable;
- pm_runtime_get_sync(rs->dev);
+ /* Return immediately for no-op */
+ if (cs_asserted == rs->cs_asserted[spi->chip_select])
+ return;
- ser = readl_relaxed(rs->regs + ROCKCHIP_SPI_SER) & SER_MASK;
+ if (cs_asserted) {
+ /* Keep things powered as long as CS is asserted */
+ pm_runtime_get_sync(rs->dev);
- /*
- * drivers/spi/spi.c:
- * static void spi_set_cs(struct spi_device *spi, bool enable)
- * {
- * if (spi->mode & SPI_CS_HIGH)
- * enable = !enable;
- *
- * if (spi->cs_gpio >= 0)
- * gpio_set_value(spi->cs_gpio, !enable);
- * else if (spi->master->set_cs)
- * spi->master->set_cs(spi, !enable);
- * }
- *
- * Note: enable(rockchip_spi_set_cs) = !enable(spi_set_cs)
- */
- if (!enable)
- ser |= 1 << spi->chip_select;
- else
- ser &= ~(1 << spi->chip_select);
+ ROCKCHIP_SPI_SET_BITS(rs->regs + ROCKCHIP_SPI_SER,
+ BIT(spi->chip_select));
+ } else {
+ ROCKCHIP_SPI_CLR_BITS(rs->regs + ROCKCHIP_SPI_SER,
+ BIT(spi->chip_select));
- writel_relaxed(ser, rs->regs + ROCKCHIP_SPI_SER);
+ /* Drop reference from when we first asserted CS */
+ pm_runtime_put(rs->dev);
+ }
- pm_runtime_put_sync(rs->dev);
+ rs->cs_asserted[spi->chip_select] = cs_asserted;
}
static int rockchip_spi_prepare_message(struct spi_master *master,
rs->regs = devm_ioremap_resource(&pdev->dev, mem);
if (IS_ERR(rs->regs)) {
ret = PTR_ERR(rs->regs);
- goto err_ioremap_resource;
+ goto err_put_master;
}
rs->apb_pclk = devm_clk_get(&pdev->dev, "apb_pclk");
if (IS_ERR(rs->apb_pclk)) {
dev_err(&pdev->dev, "Failed to get apb_pclk\n");
ret = PTR_ERR(rs->apb_pclk);
- goto err_ioremap_resource;
+ goto err_put_master;
}
rs->spiclk = devm_clk_get(&pdev->dev, "spiclk");
if (IS_ERR(rs->spiclk)) {
dev_err(&pdev->dev, "Failed to get spi_pclk\n");
ret = PTR_ERR(rs->spiclk);
- goto err_ioremap_resource;
+ goto err_put_master;
}
ret = clk_prepare_enable(rs->apb_pclk);
if (ret) {
dev_err(&pdev->dev, "Failed to enable apb_pclk\n");
- goto err_ioremap_resource;
+ goto err_put_master;
}
ret = clk_prepare_enable(rs->spiclk);
if (ret) {
dev_err(&pdev->dev, "Failed to enable spi_clk\n");
- goto err_spiclk_enable;
+ goto err_disable_apbclk;
}
spi_enable_chip(rs, 0);
if (!rs->fifo_len) {
dev_err(&pdev->dev, "Failed to get fifo length\n");
ret = -EINVAL;
- goto err_get_fifo_len;
+ goto err_disable_spiclk;
}
spin_lock_init(&rs->lock);
master->auto_runtime_pm = true;
master->bus_num = pdev->id;
master->mode_bits = SPI_CPOL | SPI_CPHA | SPI_LOOP;
- master->num_chipselect = 2;
+ master->num_chipselect = ROCKCHIP_SPI_MAX_CS_NUM;
master->dev.of_node = pdev->dev.of_node;
master->bits_per_word_mask = SPI_BPW_MASK(16) | SPI_BPW_MASK(8);
master->transfer_one = rockchip_spi_transfer_one;
master->max_transfer_size = rockchip_spi_max_transfer_size;
master->handle_err = rockchip_spi_handle_err;
+ master->flags = SPI_MASTER_GPIO_SS;
rs->dma_tx.ch = dma_request_chan(rs->dev, "tx");
if (IS_ERR(rs->dma_tx.ch)) {
/* Check tx to see if we need defer probing driver */
if (PTR_ERR(rs->dma_tx.ch) == -EPROBE_DEFER) {
ret = -EPROBE_DEFER;
- goto err_get_fifo_len;
+ goto err_disable_pm_runtime;
}
dev_warn(rs->dev, "Failed to request TX DMA channel\n");
rs->dma_tx.ch = NULL;
ret = devm_spi_register_master(&pdev->dev, master);
if (ret) {
dev_err(&pdev->dev, "Failed to register master\n");
- goto err_register_master;
+ goto err_free_dma_rx;
}
return 0;
-err_register_master:
- pm_runtime_disable(&pdev->dev);
+err_free_dma_rx:
if (rs->dma_rx.ch)
dma_release_channel(rs->dma_rx.ch);
err_free_dma_tx:
if (rs->dma_tx.ch)
dma_release_channel(rs->dma_tx.ch);
-err_get_fifo_len:
+err_disable_pm_runtime:
+ pm_runtime_disable(&pdev->dev);
+err_disable_spiclk:
clk_disable_unprepare(rs->spiclk);
-err_spiclk_enable:
+err_disable_apbclk:
clk_disable_unprepare(rs->apb_pclk);
-err_ioremap_resource:
+err_put_master:
spi_master_put(master);
return ret;
* SuperH MSIOF SPI Master Interface
*
* Copyright (c) 2009 Magnus Damm
- * Copyright (C) 2014 Glider bvba
+ * Copyright (C) 2014 Renesas Electronics Corporation
+ * Copyright (C) 2014-2017 Glider bvba
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
#include <asm/unaligned.h>
-
struct sh_msiof_chipdata {
u16 tx_fifo_size;
u16 rx_fifo_size;
void *rx_dma_page;
dma_addr_t tx_dma_addr;
dma_addr_t rx_dma_addr;
+ bool slave_aborted;
};
#define TMDR1 0x00 /* Transmit Mode Register 1 */
tmp |= !cs_high << MDR1_SYNCAC_SHIFT;
tmp |= lsb_first << MDR1_BITLSB_SHIFT;
tmp |= sh_msiof_spi_get_dtdl_and_syncdl(p);
- sh_msiof_write(p, TMDR1, tmp | MDR1_TRMD | TMDR1_PCON);
+ if (spi_controller_is_slave(p->master))
+ sh_msiof_write(p, TMDR1, tmp | TMDR1_PCON);
+ else
+ sh_msiof_write(p, TMDR1, tmp | MDR1_TRMD | TMDR1_PCON);
if (p->master->flags & SPI_MASTER_MUST_TX) {
/* These bits are reserved if RX needs TX */
tmp &= ~0x0000ffff;
static int sh_msiof_spi_start(struct sh_msiof_spi_priv *p, void *rx_buf)
{
- int ret;
+ bool slave = spi_controller_is_slave(p->master);
+ int ret = 0;
/* setup clock and rx/tx signals */
- ret = sh_msiof_modify_ctr_wait(p, 0, CTR_TSCKE);
+ if (!slave)
+ ret = sh_msiof_modify_ctr_wait(p, 0, CTR_TSCKE);
if (rx_buf && !ret)
ret = sh_msiof_modify_ctr_wait(p, 0, CTR_RXE);
if (!ret)
ret = sh_msiof_modify_ctr_wait(p, 0, CTR_TXE);
/* start by setting frame bit */
- if (!ret)
+ if (!ret && !slave)
ret = sh_msiof_modify_ctr_wait(p, 0, CTR_TFSE);
return ret;
static int sh_msiof_spi_stop(struct sh_msiof_spi_priv *p, void *rx_buf)
{
- int ret;
+ bool slave = spi_controller_is_slave(p->master);
+ int ret = 0;
/* shut down frame, rx/tx and clock signals */
- ret = sh_msiof_modify_ctr_wait(p, CTR_TFSE, 0);
+ if (!slave)
+ ret = sh_msiof_modify_ctr_wait(p, CTR_TFSE, 0);
if (!ret)
ret = sh_msiof_modify_ctr_wait(p, CTR_TXE, 0);
if (rx_buf && !ret)
ret = sh_msiof_modify_ctr_wait(p, CTR_RXE, 0);
- if (!ret)
+ if (!ret && !slave)
ret = sh_msiof_modify_ctr_wait(p, CTR_TSCKE, 0);
return ret;
}
+static int sh_msiof_slave_abort(struct spi_master *master)
+{
+ struct sh_msiof_spi_priv *p = spi_master_get_devdata(master);
+
+ p->slave_aborted = true;
+ complete(&p->done);
+ return 0;
+}
+
+static int sh_msiof_wait_for_completion(struct sh_msiof_spi_priv *p)
+{
+ if (spi_controller_is_slave(p->master)) {
+ if (wait_for_completion_interruptible(&p->done) ||
+ p->slave_aborted) {
+ dev_dbg(&p->pdev->dev, "interrupted\n");
+ return -EINTR;
+ }
+ } else {
+ if (!wait_for_completion_timeout(&p->done, HZ)) {
+ dev_err(&p->pdev->dev, "timeout\n");
+ return -ETIMEDOUT;
+ }
+ }
+
+ return 0;
+}
+
static int sh_msiof_spi_txrx_once(struct sh_msiof_spi_priv *p,
void (*tx_fifo)(struct sh_msiof_spi_priv *,
const void *, int, int),
tx_fifo(p, tx_buf, words, fifo_shift);
reinit_completion(&p->done);
+ p->slave_aborted = false;
ret = sh_msiof_spi_start(p, rx_buf);
if (ret) {
}
/* wait for tx fifo to be emptied / rx fifo to be filled */
- if (!wait_for_completion_timeout(&p->done, HZ)) {
- dev_err(&p->pdev->dev, "PIO timeout\n");
- ret = -ETIMEDOUT;
+ ret = sh_msiof_wait_for_completion(p);
+ if (ret)
goto stop_reset;
- }
/* read rx fifo */
if (rx_buf)
sh_msiof_write(p, IER, ier_bits);
reinit_completion(&p->done);
+ p->slave_aborted = false;
/* Now start DMA */
if (rx)
}
/* wait for tx fifo to be emptied / rx fifo to be filled */
- if (!wait_for_completion_timeout(&p->done, HZ)) {
- dev_err(&p->pdev->dev, "DMA timeout\n");
- ret = -ETIMEDOUT;
+ ret = sh_msiof_wait_for_completion(p);
+ if (ret)
goto stop_reset;
- }
/* clear status bits */
sh_msiof_reset_str(p);
int ret;
/* setup clocks (clock already enabled in chipselect()) */
- sh_msiof_spi_set_clk_regs(p, clk_get_rate(p->clk), t->speed_hz);
+ if (!spi_controller_is_slave(p->master))
+ sh_msiof_spi_set_clk_regs(p, clk_get_rate(p->clk), t->speed_hz);
while (master->dma_tx && len > 15) {
/*
if (!info)
return NULL;
+ info->mode = of_property_read_bool(np, "spi-slave") ? MSIOF_SPI_SLAVE
+ : MSIOF_SPI_MASTER;
+
/* Parse the MSIOF properties */
- of_property_read_u32(np, "num-cs", &num_cs);
+ if (info->mode == MSIOF_SPI_MASTER)
+ of_property_read_u32(np, "num-cs", &num_cs);
of_property_read_u32(np, "renesas,tx-fifo-size",
&info->tx_fifo_override);
of_property_read_u32(np, "renesas,rx-fifo-size",
struct spi_master *master;
const struct sh_msiof_chipdata *chipdata;
const struct of_device_id *of_id;
+ struct sh_msiof_spi_info *info;
struct sh_msiof_spi_priv *p;
int i;
int ret;
- master = spi_alloc_master(&pdev->dev, sizeof(struct sh_msiof_spi_priv));
- if (master == NULL)
- return -ENOMEM;
-
- p = spi_master_get_devdata(master);
-
- platform_set_drvdata(pdev, p);
- p->master = master;
-
of_id = of_match_device(sh_msiof_match, &pdev->dev);
if (of_id) {
chipdata = of_id->data;
- p->info = sh_msiof_spi_parse_dt(&pdev->dev);
+ info = sh_msiof_spi_parse_dt(&pdev->dev);
} else {
chipdata = (const void *)pdev->id_entry->driver_data;
- p->info = dev_get_platdata(&pdev->dev);
+ info = dev_get_platdata(&pdev->dev);
}
- if (!p->info) {
+ if (!info) {
dev_err(&pdev->dev, "failed to obtain device info\n");
- ret = -ENXIO;
- goto err1;
+ return -ENXIO;
}
+ if (info->mode == MSIOF_SPI_SLAVE)
+ master = spi_alloc_slave(&pdev->dev,
+ sizeof(struct sh_msiof_spi_priv));
+ else
+ master = spi_alloc_master(&pdev->dev,
+ sizeof(struct sh_msiof_spi_priv));
+ if (master == NULL)
+ return -ENOMEM;
+
+ p = spi_master_get_devdata(master);
+
+ platform_set_drvdata(pdev, p);
+ p->master = master;
+ p->info = info;
+
init_completion(&p->done);
p->clk = devm_clk_get(&pdev->dev, NULL);
master->num_chipselect = p->info->num_chipselect;
master->setup = sh_msiof_spi_setup;
master->prepare_message = sh_msiof_prepare_message;
+ master->slave_abort = sh_msiof_slave_abort;
master->bits_per_word_mask = SPI_BPW_RANGE_MASK(8, 32);
master->auto_runtime_pm = true;
master->transfer_one = sh_msiof_transfer_one;
ret = spi_bitbang_start(&sspi->bitbang);
if (ret)
goto free_clk;
- dev_info(&pdev->dev, "registerred, bus number = %d\n", master->bus_num);
+ dev_info(&pdev->dev, "registered, bus number = %d\n", master->bus_num);
return 0;
free_clk:
--- /dev/null
+/*
+ * SPI slave handler controlling system state
+ *
+ * This SPI slave handler allows remote control of system reboot, power off,
+ * halt, and suspend.
+ *
+ * Copyright (C) 2016-2017 Glider bvba
+ *
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License. See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Usage (assuming /dev/spidev2.0 corresponds to the SPI master on the remote
+ * system):
+ *
+ * # reboot='\x7c\x50'
+ * # poweroff='\x71\x3f'
+ * # halt='\x38\x76'
+ * # suspend='\x1b\x1b'
+ * # spidev_test -D /dev/spidev2.0 -p $suspend # or $reboot, $poweroff, $halt
+ */
+
+#include <linux/completion.h>
+#include <linux/module.h>
+#include <linux/reboot.h>
+#include <linux/suspend.h>
+#include <linux/spi/spi.h>
+
+/*
+ * The numbers are chosen to display something human-readable on two 7-segment
+ * displays connected to two 74HC595 shift registers
+ */
+#define CMD_REBOOT 0x7c50 /* rb */
+#define CMD_POWEROFF 0x713f /* OF */
+#define CMD_HALT 0x3876 /* HL */
+#define CMD_SUSPEND 0x1b1b /* ZZ */
+
+struct spi_slave_system_control_priv {
+ struct spi_device *spi;
+ struct completion finished;
+ struct spi_transfer xfer;
+ struct spi_message msg;
+ __be16 cmd;
+};
+
+static
+int spi_slave_system_control_submit(struct spi_slave_system_control_priv *priv);
+
+static void spi_slave_system_control_complete(void *arg)
+{
+ struct spi_slave_system_control_priv *priv = arg;
+ u16 cmd;
+ int ret;
+
+ if (priv->msg.status)
+ goto terminate;
+
+ cmd = be16_to_cpu(priv->cmd);
+ switch (cmd) {
+ case CMD_REBOOT:
+ dev_info(&priv->spi->dev, "Rebooting system...\n");
+ kernel_restart(NULL);
+
+ case CMD_POWEROFF:
+ dev_info(&priv->spi->dev, "Powering off system...\n");
+ kernel_power_off();
+ break;
+
+ case CMD_HALT:
+ dev_info(&priv->spi->dev, "Halting system...\n");
+ kernel_halt();
+ break;
+
+ case CMD_SUSPEND:
+ dev_info(&priv->spi->dev, "Suspending system...\n");
+ pm_suspend(PM_SUSPEND_MEM);
+ break;
+
+ default:
+ dev_warn(&priv->spi->dev, "Unknown command 0x%x\n", cmd);
+ break;
+ }
+
+ ret = spi_slave_system_control_submit(priv);
+ if (ret)
+ goto terminate;
+
+ return;
+
+terminate:
+ dev_info(&priv->spi->dev, "Terminating\n");
+ complete(&priv->finished);
+}
+
+static
+int spi_slave_system_control_submit(struct spi_slave_system_control_priv *priv)
+{
+ int ret;
+
+ spi_message_init_with_transfers(&priv->msg, &priv->xfer, 1);
+
+ priv->msg.complete = spi_slave_system_control_complete;
+ priv->msg.context = priv;
+
+ ret = spi_async(priv->spi, &priv->msg);
+ if (ret)
+ dev_err(&priv->spi->dev, "spi_async() failed %d\n", ret);
+
+ return ret;
+}
+
+static int spi_slave_system_control_probe(struct spi_device *spi)
+{
+ struct spi_slave_system_control_priv *priv;
+ int ret;
+
+ priv = devm_kzalloc(&spi->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ priv->spi = spi;
+ init_completion(&priv->finished);
+ priv->xfer.rx_buf = &priv->cmd;
+ priv->xfer.len = sizeof(priv->cmd);
+
+ ret = spi_slave_system_control_submit(priv);
+ if (ret)
+ return ret;
+
+ spi_set_drvdata(spi, priv);
+ return 0;
+}
+
+static int spi_slave_system_control_remove(struct spi_device *spi)
+{
+ struct spi_slave_system_control_priv *priv = spi_get_drvdata(spi);
+
+ spi_slave_abort(spi);
+ wait_for_completion(&priv->finished);
+ return 0;
+}
+
+static struct spi_driver spi_slave_system_control_driver = {
+ .driver = {
+ .name = "spi-slave-system-control",
+ },
+ .probe = spi_slave_system_control_probe,
+ .remove = spi_slave_system_control_remove,
+};
+module_spi_driver(spi_slave_system_control_driver);
+
+MODULE_AUTHOR("Geert Uytterhoeven <geert+renesas@glider.be>");
+MODULE_DESCRIPTION("SPI slave handler controlling system state");
+MODULE_LICENSE("GPL v2");
--- /dev/null
+/*
+ * SPI slave handler reporting uptime at reception of previous SPI message
+ *
+ * This SPI slave handler sends the time of reception of the last SPI message
+ * as two 32-bit unsigned integers in binary format and in network byte order,
+ * representing the number of seconds and fractional seconds (in microseconds)
+ * since boot up.
+ *
+ * Copyright (C) 2016-2017 Glider bvba
+ *
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License. See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Usage (assuming /dev/spidev2.0 corresponds to the SPI master on the remote
+ * system):
+ *
+ * # spidev_test -D /dev/spidev2.0 -p dummy-8B
+ * spi mode: 0x0
+ * bits per word: 8
+ * max speed: 500000 Hz (500 KHz)
+ * RX | 00 00 04 6D 00 09 5B BB ...
+ * ^^^^^ ^^^^^^^^
+ * seconds microseconds
+ */
+
+#include <linux/completion.h>
+#include <linux/module.h>
+#include <linux/sched/clock.h>
+#include <linux/spi/spi.h>
+
+
+struct spi_slave_time_priv {
+ struct spi_device *spi;
+ struct completion finished;
+ struct spi_transfer xfer;
+ struct spi_message msg;
+ __be32 buf[2];
+};
+
+static int spi_slave_time_submit(struct spi_slave_time_priv *priv);
+
+static void spi_slave_time_complete(void *arg)
+{
+ struct spi_slave_time_priv *priv = arg;
+ int ret;
+
+ ret = priv->msg.status;
+ if (ret)
+ goto terminate;
+
+ ret = spi_slave_time_submit(priv);
+ if (ret)
+ goto terminate;
+
+ return;
+
+terminate:
+ dev_info(&priv->spi->dev, "Terminating\n");
+ complete(&priv->finished);
+}
+
+static int spi_slave_time_submit(struct spi_slave_time_priv *priv)
+{
+ u32 rem_us;
+ int ret;
+ u64 ts;
+
+ ts = local_clock();
+ rem_us = do_div(ts, 1000000000) / 1000;
+
+ priv->buf[0] = cpu_to_be32(ts);
+ priv->buf[1] = cpu_to_be32(rem_us);
+
+ spi_message_init_with_transfers(&priv->msg, &priv->xfer, 1);
+
+ priv->msg.complete = spi_slave_time_complete;
+ priv->msg.context = priv;
+
+ ret = spi_async(priv->spi, &priv->msg);
+ if (ret)
+ dev_err(&priv->spi->dev, "spi_async() failed %d\n", ret);
+
+ return ret;
+}
+
+static int spi_slave_time_probe(struct spi_device *spi)
+{
+ struct spi_slave_time_priv *priv;
+ int ret;
+
+ priv = devm_kzalloc(&spi->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ priv->spi = spi;
+ init_completion(&priv->finished);
+ priv->xfer.tx_buf = priv->buf;
+ priv->xfer.len = sizeof(priv->buf);
+
+ ret = spi_slave_time_submit(priv);
+ if (ret)
+ return ret;
+
+ spi_set_drvdata(spi, priv);
+ return 0;
+}
+
+static int spi_slave_time_remove(struct spi_device *spi)
+{
+ struct spi_slave_time_priv *priv = spi_get_drvdata(spi);
+
+ spi_slave_abort(spi);
+ wait_for_completion(&priv->finished);
+ return 0;
+}
+
+static struct spi_driver spi_slave_time_driver = {
+ .driver = {
+ .name = "spi-slave-time",
+ },
+ .probe = spi_slave_time_probe,
+ .remove = spi_slave_time_remove,
+};
+module_spi_driver(spi_slave_time_driver);
+
+MODULE_AUTHOR("Geert Uytterhoeven <geert+renesas@glider.be>");
+MODULE_DESCRIPTION("SPI slave reporting uptime at previous SPI message");
+MODULE_LICENSE("GPL v2");
"setting baudrate:target= %u hz, actual= %u hz, sscbrg= %u\n",
hz, spi_st->baud, sscbrg);
- /* Set SSC_CTL and enable SSC */
- var = readl_relaxed(spi_st->base + SSC_CTL);
- var |= SSC_CTL_MS;
+ /* Set SSC_CTL and enable SSC */
+ var = readl_relaxed(spi_st->base + SSC_CTL);
+ var |= SSC_CTL_MS;
- if (spi->mode & SPI_CPOL)
+ if (spi->mode & SPI_CPOL)
var |= SSC_CTL_PO;
- else
+ else
var &= ~SSC_CTL_PO;
- if (spi->mode & SPI_CPHA)
+ if (spi->mode & SPI_CPHA)
var |= SSC_CTL_PH;
- else
+ else
var &= ~SSC_CTL_PH;
- if ((spi->mode & SPI_LSB_FIRST) == 0)
+ if ((spi->mode & SPI_LSB_FIRST) == 0)
var |= SSC_CTL_HB;
- else
+ else
var &= ~SSC_CTL_HB;
- if (spi->mode & SPI_LOOP)
+ if (spi->mode & SPI_LOOP)
var |= SSC_CTL_LPB;
- else
+ else
var &= ~SSC_CTL_LPB;
- var &= ~SSC_CTL_DATA_WIDTH_MSK;
- var |= (spi->bits_per_word - 1);
+ var &= ~SSC_CTL_DATA_WIDTH_MSK;
+ var |= (spi->bits_per_word - 1);
- var |= SSC_CTL_EN_TX_FIFO | SSC_CTL_EN_RX_FIFO;
- var |= SSC_CTL_EN;
+ var |= SSC_CTL_EN_TX_FIFO | SSC_CTL_EN_RX_FIFO;
+ var |= SSC_CTL_EN;
- writel_relaxed(var, spi_st->base + SSC_CTL);
+ writel_relaxed(var, spi_st->base + SSC_CTL);
- /* Clear the status register */
- readl_relaxed(spi_st->base + SSC_RBUF);
+ /* Clear the status register */
+ readl_relaxed(spi_st->base + SSC_RBUF);
- return 0;
+ return 0;
out_free_gpio:
gpio_free(cs);
--- /dev/null
+/*
+ * STMicroelectronics STM32 SPI Controller driver (master mode only)
+ *
+ * Copyright (C) 2017, STMicroelectronics - All Rights Reserved
+ * Author(s): Amelie Delaunay <amelie.delaunay@st.com> for STMicroelectronics.
+ *
+ * License terms: GPL V2.0.
+ *
+ * spi_stm32 driver is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published by
+ * the Free Software Foundation.
+ *
+ * spi_stm32 driver is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
+ * details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * spi_stm32 driver. If not, see <http://www.gnu.org/licenses/>.
+ */
+#include <linux/debugfs.h>
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/dmaengine.h>
+#include <linux/gpio.h>
+#include <linux/interrupt.h>
+#include <linux/iopoll.h>
+#include <linux/module.h>
+#include <linux/of_platform.h>
+#include <linux/pm_runtime.h>
+#include <linux/reset.h>
+#include <linux/spi/spi.h>
+
+#define DRIVER_NAME "spi_stm32"
+
+/* STM32 SPI registers */
+#define STM32_SPI_CR1 0x00
+#define STM32_SPI_CR2 0x04
+#define STM32_SPI_CFG1 0x08
+#define STM32_SPI_CFG2 0x0C
+#define STM32_SPI_IER 0x10
+#define STM32_SPI_SR 0x14
+#define STM32_SPI_IFCR 0x18
+#define STM32_SPI_TXDR 0x20
+#define STM32_SPI_RXDR 0x30
+#define STM32_SPI_I2SCFGR 0x50
+
+/* STM32_SPI_CR1 bit fields */
+#define SPI_CR1_SPE BIT(0)
+#define SPI_CR1_MASRX BIT(8)
+#define SPI_CR1_CSTART BIT(9)
+#define SPI_CR1_CSUSP BIT(10)
+#define SPI_CR1_HDDIR BIT(11)
+#define SPI_CR1_SSI BIT(12)
+
+/* STM32_SPI_CR2 bit fields */
+#define SPI_CR2_TSIZE_SHIFT 0
+#define SPI_CR2_TSIZE GENMASK(15, 0)
+
+/* STM32_SPI_CFG1 bit fields */
+#define SPI_CFG1_DSIZE_SHIFT 0
+#define SPI_CFG1_DSIZE GENMASK(4, 0)
+#define SPI_CFG1_FTHLV_SHIFT 5
+#define SPI_CFG1_FTHLV GENMASK(8, 5)
+#define SPI_CFG1_RXDMAEN BIT(14)
+#define SPI_CFG1_TXDMAEN BIT(15)
+#define SPI_CFG1_MBR_SHIFT 28
+#define SPI_CFG1_MBR GENMASK(30, 28)
+#define SPI_CFG1_MBR_MIN 0
+#define SPI_CFG1_MBR_MAX (GENMASK(30, 28) >> 28)
+
+/* STM32_SPI_CFG2 bit fields */
+#define SPI_CFG2_MIDI_SHIFT 4
+#define SPI_CFG2_MIDI GENMASK(7, 4)
+#define SPI_CFG2_COMM_SHIFT 17
+#define SPI_CFG2_COMM GENMASK(18, 17)
+#define SPI_CFG2_SP_SHIFT 19
+#define SPI_CFG2_SP GENMASK(21, 19)
+#define SPI_CFG2_MASTER BIT(22)
+#define SPI_CFG2_LSBFRST BIT(23)
+#define SPI_CFG2_CPHA BIT(24)
+#define SPI_CFG2_CPOL BIT(25)
+#define SPI_CFG2_SSM BIT(26)
+#define SPI_CFG2_AFCNTR BIT(31)
+
+/* STM32_SPI_IER bit fields */
+#define SPI_IER_RXPIE BIT(0)
+#define SPI_IER_TXPIE BIT(1)
+#define SPI_IER_DXPIE BIT(2)
+#define SPI_IER_EOTIE BIT(3)
+#define SPI_IER_TXTFIE BIT(4)
+#define SPI_IER_OVRIE BIT(6)
+#define SPI_IER_MODFIE BIT(9)
+#define SPI_IER_ALL GENMASK(10, 0)
+
+/* STM32_SPI_SR bit fields */
+#define SPI_SR_RXP BIT(0)
+#define SPI_SR_TXP BIT(1)
+#define SPI_SR_EOT BIT(3)
+#define SPI_SR_OVR BIT(6)
+#define SPI_SR_MODF BIT(9)
+#define SPI_SR_SUSP BIT(11)
+#define SPI_SR_RXPLVL_SHIFT 13
+#define SPI_SR_RXPLVL GENMASK(14, 13)
+#define SPI_SR_RXWNE BIT(15)
+
+/* STM32_SPI_IFCR bit fields */
+#define SPI_IFCR_ALL GENMASK(11, 3)
+
+/* STM32_SPI_I2SCFGR bit fields */
+#define SPI_I2SCFGR_I2SMOD BIT(0)
+
+/* SPI Master Baud Rate min/max divisor */
+#define SPI_MBR_DIV_MIN (2 << SPI_CFG1_MBR_MIN)
+#define SPI_MBR_DIV_MAX (2 << SPI_CFG1_MBR_MAX)
+
+/* SPI Communication mode */
+#define SPI_FULL_DUPLEX 0
+#define SPI_SIMPLEX_TX 1
+#define SPI_SIMPLEX_RX 2
+#define SPI_HALF_DUPLEX 3
+
+#define SPI_1HZ_NS 1000000000
+
+/**
+ * struct stm32_spi - private data of the SPI controller
+ * @dev: driver model representation of the controller
+ * @master: controller master interface
+ * @base: virtual memory area
+ * @clk: hw kernel clock feeding the SPI clock generator
+ * @clk_rate: rate of the hw kernel clock feeding the SPI clock generator
+ * @rst: SPI controller reset line
+ * @lock: prevent I/O concurrent access
+ * @irq: SPI controller interrupt line
+ * @fifo_size: size of the embedded fifo in bytes
+ * @cur_midi: master inter-data idleness in ns
+ * @cur_speed: speed configured in Hz
+ * @cur_bpw: number of bits in a single SPI data frame
+ * @cur_fthlv: fifo threshold level (data frames in a single data packet)
+ * @cur_comm: SPI communication mode
+ * @cur_xferlen: current transfer length in bytes
+ * @cur_usedma: boolean to know if dma is used in current transfer
+ * @tx_buf: data to be written, or NULL
+ * @rx_buf: data to be read, or NULL
+ * @tx_len: number of data to be written in bytes
+ * @rx_len: number of data to be read in bytes
+ * @dma_tx: dma channel for TX transfer
+ * @dma_rx: dma channel for RX transfer
+ * @phys_addr: SPI registers physical base address
+ */
+struct stm32_spi {
+ struct device *dev;
+ struct spi_master *master;
+ void __iomem *base;
+ struct clk *clk;
+ u32 clk_rate;
+ struct reset_control *rst;
+ spinlock_t lock; /* prevent I/O concurrent access */
+ int irq;
+ unsigned int fifo_size;
+
+ unsigned int cur_midi;
+ unsigned int cur_speed;
+ unsigned int cur_bpw;
+ unsigned int cur_fthlv;
+ unsigned int cur_comm;
+ unsigned int cur_xferlen;
+ bool cur_usedma;
+
+ const void *tx_buf;
+ void *rx_buf;
+ int tx_len;
+ int rx_len;
+ struct dma_chan *dma_tx;
+ struct dma_chan *dma_rx;
+ dma_addr_t phys_addr;
+};
+
+static inline void stm32_spi_set_bits(struct stm32_spi *spi,
+ u32 offset, u32 bits)
+{
+ writel_relaxed(readl_relaxed(spi->base + offset) | bits,
+ spi->base + offset);
+}
+
+static inline void stm32_spi_clr_bits(struct stm32_spi *spi,
+ u32 offset, u32 bits)
+{
+ writel_relaxed(readl_relaxed(spi->base + offset) & ~bits,
+ spi->base + offset);
+}
+
+/**
+ * stm32_spi_get_fifo_size - Return fifo size
+ * @spi: pointer to the spi controller data structure
+ */
+static int stm32_spi_get_fifo_size(struct stm32_spi *spi)
+{
+ unsigned long flags;
+ u32 count = 0;
+
+ spin_lock_irqsave(&spi->lock, flags);
+
+ stm32_spi_set_bits(spi, STM32_SPI_CR1, SPI_CR1_SPE);
+
+ while (readl_relaxed(spi->base + STM32_SPI_SR) & SPI_SR_TXP)
+ writeb_relaxed(++count, spi->base + STM32_SPI_TXDR);
+
+ stm32_spi_clr_bits(spi, STM32_SPI_CR1, SPI_CR1_SPE);
+
+ spin_unlock_irqrestore(&spi->lock, flags);
+
+ dev_dbg(spi->dev, "%d x 8-bit fifo size\n", count);
+
+ return count;
+}
+
+/**
+ * stm32_spi_get_bpw_mask - Return bits per word mask
+ * @spi: pointer to the spi controller data structure
+ */
+static int stm32_spi_get_bpw_mask(struct stm32_spi *spi)
+{
+ unsigned long flags;
+ u32 cfg1, max_bpw;
+
+ spin_lock_irqsave(&spi->lock, flags);
+
+ /*
+ * The most significant bit at DSIZE bit field is reserved when the
+ * maximum data size of periperal instances is limited to 16-bit
+ */
+ stm32_spi_set_bits(spi, STM32_SPI_CFG1, SPI_CFG1_DSIZE);
+
+ cfg1 = readl_relaxed(spi->base + STM32_SPI_CFG1);
+ max_bpw = (cfg1 & SPI_CFG1_DSIZE) >> SPI_CFG1_DSIZE_SHIFT;
+ max_bpw += 1;
+
+ spin_unlock_irqrestore(&spi->lock, flags);
+
+ dev_dbg(spi->dev, "%d-bit maximum data frame\n", max_bpw);
+
+ return SPI_BPW_RANGE_MASK(4, max_bpw);
+}
+
+/**
+ * stm32_spi_prepare_mbr - Determine SPI_CFG1.MBR value
+ * @spi: pointer to the spi controller data structure
+ * @speed_hz: requested speed
+ *
+ * Return SPI_CFG1.MBR value in case of success or -EINVAL
+ */
+static int stm32_spi_prepare_mbr(struct stm32_spi *spi, u32 speed_hz)
+{
+ u32 div, mbrdiv;
+
+ div = DIV_ROUND_UP(spi->clk_rate, speed_hz);
+
+ /*
+ * SPI framework set xfer->speed_hz to master->max_speed_hz if
+ * xfer->speed_hz is greater than master->max_speed_hz, and it returns
+ * an error when xfer->speed_hz is lower than master->min_speed_hz, so
+ * no need to check it there.
+ * However, we need to ensure the following calculations.
+ */
+ if ((div < SPI_MBR_DIV_MIN) &&
+ (div > SPI_MBR_DIV_MAX))
+ return -EINVAL;
+
+ /* Determine the first power of 2 greater than or equal to div */
+ if (div & (div - 1))
+ mbrdiv = fls(div);
+ else
+ mbrdiv = fls(div) - 1;
+
+ spi->cur_speed = spi->clk_rate / (1 << mbrdiv);
+
+ return mbrdiv - 1;
+}
+
+/**
+ * stm32_spi_prepare_fthlv - Determine FIFO threshold level
+ * @spi: pointer to the spi controller data structure
+ */
+static u32 stm32_spi_prepare_fthlv(struct stm32_spi *spi)
+{
+ u32 fthlv, half_fifo;
+
+ /* data packet should not exceed 1/2 of fifo space */
+ half_fifo = (spi->fifo_size / 2);
+
+ if (spi->cur_bpw <= 8)
+ fthlv = half_fifo;
+ else if (spi->cur_bpw <= 16)
+ fthlv = half_fifo / 2;
+ else
+ fthlv = half_fifo / 4;
+
+ /* align packet size with data registers access */
+ if (spi->cur_bpw > 8)
+ fthlv -= (fthlv % 2); /* multiple of 2 */
+ else
+ fthlv -= (fthlv % 4); /* multiple of 4 */
+
+ return fthlv;
+}
+
+/**
+ * stm32_spi_write_txfifo - Write bytes in Transmit Data Register
+ * @spi: pointer to the spi controller data structure
+ *
+ * Read from tx_buf depends on remaining bytes to avoid to read beyond
+ * tx_buf end.
+ */
+static void stm32_spi_write_txfifo(struct stm32_spi *spi)
+{
+ while ((spi->tx_len > 0) &&
+ (readl_relaxed(spi->base + STM32_SPI_SR) & SPI_SR_TXP)) {
+ u32 offs = spi->cur_xferlen - spi->tx_len;
+
+ if (spi->tx_len >= sizeof(u32)) {
+ const u32 *tx_buf32 = (const u32 *)(spi->tx_buf + offs);
+
+ writel_relaxed(*tx_buf32, spi->base + STM32_SPI_TXDR);
+ spi->tx_len -= sizeof(u32);
+ } else if (spi->tx_len >= sizeof(u16)) {
+ const u16 *tx_buf16 = (const u16 *)(spi->tx_buf + offs);
+
+ writew_relaxed(*tx_buf16, spi->base + STM32_SPI_TXDR);
+ spi->tx_len -= sizeof(u16);
+ } else {
+ const u8 *tx_buf8 = (const u8 *)(spi->tx_buf + offs);
+
+ writeb_relaxed(*tx_buf8, spi->base + STM32_SPI_TXDR);
+ spi->tx_len -= sizeof(u8);
+ }
+ }
+
+ dev_dbg(spi->dev, "%s: %d bytes left\n", __func__, spi->tx_len);
+}
+
+/**
+ * stm32_spi_read_rxfifo - Read bytes in Receive Data Register
+ * @spi: pointer to the spi controller data structure
+ *
+ * Write in rx_buf depends on remaining bytes to avoid to write beyond
+ * rx_buf end.
+ */
+static void stm32_spi_read_rxfifo(struct stm32_spi *spi, bool flush)
+{
+ u32 sr = readl_relaxed(spi->base + STM32_SPI_SR);
+ u32 rxplvl = (sr & SPI_SR_RXPLVL) >> SPI_SR_RXPLVL_SHIFT;
+
+ while ((spi->rx_len > 0) &&
+ ((sr & SPI_SR_RXP) ||
+ (flush && ((sr & SPI_SR_RXWNE) || (rxplvl > 0))))) {
+ u32 offs = spi->cur_xferlen - spi->rx_len;
+
+ if ((spi->rx_len >= sizeof(u32)) ||
+ (flush && (sr & SPI_SR_RXWNE))) {
+ u32 *rx_buf32 = (u32 *)(spi->rx_buf + offs);
+
+ *rx_buf32 = readl_relaxed(spi->base + STM32_SPI_RXDR);
+ spi->rx_len -= sizeof(u32);
+ } else if ((spi->rx_len >= sizeof(u16)) ||
+ (flush && (rxplvl >= 2 || spi->cur_bpw > 8))) {
+ u16 *rx_buf16 = (u16 *)(spi->rx_buf + offs);
+
+ *rx_buf16 = readw_relaxed(spi->base + STM32_SPI_RXDR);
+ spi->rx_len -= sizeof(u16);
+ } else {
+ u8 *rx_buf8 = (u8 *)(spi->rx_buf + offs);
+
+ *rx_buf8 = readb_relaxed(spi->base + STM32_SPI_RXDR);
+ spi->rx_len -= sizeof(u8);
+ }
+
+ sr = readl_relaxed(spi->base + STM32_SPI_SR);
+ rxplvl = (sr & SPI_SR_RXPLVL) >> SPI_SR_RXPLVL_SHIFT;
+ }
+
+ dev_dbg(spi->dev, "%s%s: %d bytes left\n", __func__,
+ flush ? "(flush)" : "", spi->rx_len);
+}
+
+/**
+ * stm32_spi_enable - Enable SPI controller
+ * @spi: pointer to the spi controller data structure
+ *
+ * SPI data transfer is enabled but spi_ker_ck is idle.
+ * SPI_CFG1 and SPI_CFG2 are now write protected.
+ */
+static void stm32_spi_enable(struct stm32_spi *spi)
+{
+ dev_dbg(spi->dev, "enable controller\n");
+
+ stm32_spi_set_bits(spi, STM32_SPI_CR1, SPI_CR1_SPE);
+}
+
+/**
+ * stm32_spi_disable - Disable SPI controller
+ * @spi: pointer to the spi controller data structure
+ *
+ * RX-Fifo is flushed when SPI controller is disabled. To prevent any data
+ * loss, use stm32_spi_read_rxfifo(flush) to read the remaining bytes in
+ * RX-Fifo.
+ */
+static void stm32_spi_disable(struct stm32_spi *spi)
+{
+ unsigned long flags;
+ u32 cr1, sr;
+
+ dev_dbg(spi->dev, "disable controller\n");
+
+ spin_lock_irqsave(&spi->lock, flags);
+
+ cr1 = readl_relaxed(spi->base + STM32_SPI_CR1);
+
+ if (!(cr1 & SPI_CR1_SPE)) {
+ spin_unlock_irqrestore(&spi->lock, flags);
+ return;
+ }
+
+ /* Wait on EOT or suspend the flow */
+ if (readl_relaxed_poll_timeout_atomic(spi->base + STM32_SPI_SR,
+ sr, !(sr & SPI_SR_EOT),
+ 10, 100000) < 0) {
+ if (cr1 & SPI_CR1_CSTART) {
+ writel_relaxed(cr1 | SPI_CR1_CSUSP,
+ spi->base + STM32_SPI_CR1);
+ if (readl_relaxed_poll_timeout_atomic(
+ spi->base + STM32_SPI_SR,
+ sr, !(sr & SPI_SR_SUSP),
+ 10, 100000) < 0)
+ dev_warn(spi->dev,
+ "Suspend request timeout\n");
+ }
+ }
+
+ if (!spi->cur_usedma && spi->rx_buf && (spi->rx_len > 0))
+ stm32_spi_read_rxfifo(spi, true);
+
+ if (spi->cur_usedma && spi->tx_buf)
+ dmaengine_terminate_all(spi->dma_tx);
+ if (spi->cur_usedma && spi->rx_buf)
+ dmaengine_terminate_all(spi->dma_rx);
+
+ stm32_spi_clr_bits(spi, STM32_SPI_CR1, SPI_CR1_SPE);
+
+ stm32_spi_clr_bits(spi, STM32_SPI_CFG1, SPI_CFG1_TXDMAEN |
+ SPI_CFG1_RXDMAEN);
+
+ /* Disable interrupts and clear status flags */
+ writel_relaxed(0, spi->base + STM32_SPI_IER);
+ writel_relaxed(SPI_IFCR_ALL, spi->base + STM32_SPI_IFCR);
+
+ spin_unlock_irqrestore(&spi->lock, flags);
+}
+
+/**
+ * stm32_spi_can_dma - Determine if the transfer is eligible for DMA use
+ *
+ * If the current transfer size is greater than fifo size, use DMA.
+ */
+static bool stm32_spi_can_dma(struct spi_master *master,
+ struct spi_device *spi_dev,
+ struct spi_transfer *transfer)
+{
+ struct stm32_spi *spi = spi_master_get_devdata(master);
+
+ dev_dbg(spi->dev, "%s: %s\n", __func__,
+ (transfer->len > spi->fifo_size) ? "true" : "false");
+
+ return (transfer->len > spi->fifo_size);
+}
+
+/**
+ * stm32_spi_irq - Interrupt handler for SPI controller events
+ * @irq: interrupt line
+ * @dev_id: SPI controller master interface
+ */
+static irqreturn_t stm32_spi_irq(int irq, void *dev_id)
+{
+ struct spi_master *master = dev_id;
+ struct stm32_spi *spi = spi_master_get_devdata(master);
+ u32 sr, ier, mask;
+ unsigned long flags;
+ bool end = false;
+
+ spin_lock_irqsave(&spi->lock, flags);
+
+ sr = readl_relaxed(spi->base + STM32_SPI_SR);
+ ier = readl_relaxed(spi->base + STM32_SPI_IER);
+
+ mask = ier;
+ /* EOTIE is triggered on EOT, SUSP and TXC events. */
+ mask |= SPI_SR_SUSP;
+ /*
+ * When TXTF is set, DXPIE and TXPIE are cleared. So in case of
+ * Full-Duplex, need to poll RXP event to know if there are remaining
+ * data, before disabling SPI.
+ */
+ if (spi->rx_buf && !spi->cur_usedma)
+ mask |= SPI_SR_RXP;
+
+ if (!(sr & mask)) {
+ dev_dbg(spi->dev, "spurious IT (sr=0x%08x, ier=0x%08x)\n",
+ sr, ier);
+ spin_unlock_irqrestore(&spi->lock, flags);
+ return IRQ_NONE;
+ }
+
+ if (sr & SPI_SR_SUSP) {
+ dev_warn(spi->dev, "Communication suspended\n");
+ if (!spi->cur_usedma && (spi->rx_buf && (spi->rx_len > 0)))
+ stm32_spi_read_rxfifo(spi, false);
+ /*
+ * If communication is suspended while using DMA, it means
+ * that something went wrong, so stop the current transfer
+ */
+ if (spi->cur_usedma)
+ end = true;
+ }
+
+ if (sr & SPI_SR_MODF) {
+ dev_warn(spi->dev, "Mode fault: transfer aborted\n");
+ end = true;
+ }
+
+ if (sr & SPI_SR_OVR) {
+ dev_warn(spi->dev, "Overrun: received value discarded\n");
+ if (!spi->cur_usedma && (spi->rx_buf && (spi->rx_len > 0)))
+ stm32_spi_read_rxfifo(spi, false);
+ /*
+ * If overrun is detected while using DMA, it means that
+ * something went wrong, so stop the current transfer
+ */
+ if (spi->cur_usedma)
+ end = true;
+ }
+
+ if (sr & SPI_SR_EOT) {
+ if (!spi->cur_usedma && (spi->rx_buf && (spi->rx_len > 0)))
+ stm32_spi_read_rxfifo(spi, true);
+ end = true;
+ }
+
+ if (sr & SPI_SR_TXP)
+ if (!spi->cur_usedma && (spi->tx_buf && (spi->tx_len > 0)))
+ stm32_spi_write_txfifo(spi);
+
+ if (sr & SPI_SR_RXP)
+ if (!spi->cur_usedma && (spi->rx_buf && (spi->rx_len > 0)))
+ stm32_spi_read_rxfifo(spi, false);
+
+ writel_relaxed(mask, spi->base + STM32_SPI_IFCR);
+
+ spin_unlock_irqrestore(&spi->lock, flags);
+
+ if (end) {
+ spi_finalize_current_transfer(master);
+ stm32_spi_disable(spi);
+ }
+
+ return IRQ_HANDLED;
+}
+
+/**
+ * stm32_spi_setup - setup device chip select
+ */
+static int stm32_spi_setup(struct spi_device *spi_dev)
+{
+ int ret = 0;
+
+ if (!gpio_is_valid(spi_dev->cs_gpio)) {
+ dev_err(&spi_dev->dev, "%d is not a valid gpio\n",
+ spi_dev->cs_gpio);
+ return -EINVAL;
+ }
+
+ dev_dbg(&spi_dev->dev, "%s: set gpio%d output %s\n", __func__,
+ spi_dev->cs_gpio,
+ (spi_dev->mode & SPI_CS_HIGH) ? "low" : "high");
+
+ ret = gpio_direction_output(spi_dev->cs_gpio,
+ !(spi_dev->mode & SPI_CS_HIGH));
+
+ return ret;
+}
+
+/**
+ * stm32_spi_prepare_msg - set up the controller to transfer a single message
+ */
+static int stm32_spi_prepare_msg(struct spi_master *master,
+ struct spi_message *msg)
+{
+ struct stm32_spi *spi = spi_master_get_devdata(master);
+ struct spi_device *spi_dev = msg->spi;
+ struct device_node *np = spi_dev->dev.of_node;
+ unsigned long flags;
+ u32 cfg2_clrb = 0, cfg2_setb = 0;
+
+ /* SPI slave device may need time between data frames */
+ spi->cur_midi = 0;
+ if (np && !of_property_read_u32(np, "st,spi-midi-ns", &spi->cur_midi))
+ dev_dbg(spi->dev, "%dns inter-data idleness\n", spi->cur_midi);
+
+ if (spi_dev->mode & SPI_CPOL)
+ cfg2_setb |= SPI_CFG2_CPOL;
+ else
+ cfg2_clrb |= SPI_CFG2_CPOL;
+
+ if (spi_dev->mode & SPI_CPHA)
+ cfg2_setb |= SPI_CFG2_CPHA;
+ else
+ cfg2_clrb |= SPI_CFG2_CPHA;
+
+ if (spi_dev->mode & SPI_LSB_FIRST)
+ cfg2_setb |= SPI_CFG2_LSBFRST;
+ else
+ cfg2_clrb |= SPI_CFG2_LSBFRST;
+
+ dev_dbg(spi->dev, "cpol=%d cpha=%d lsb_first=%d cs_high=%d\n",
+ spi_dev->mode & SPI_CPOL,
+ spi_dev->mode & SPI_CPHA,
+ spi_dev->mode & SPI_LSB_FIRST,
+ spi_dev->mode & SPI_CS_HIGH);
+
+ spin_lock_irqsave(&spi->lock, flags);
+
+ if (cfg2_clrb || cfg2_setb)
+ writel_relaxed(
+ (readl_relaxed(spi->base + STM32_SPI_CFG2) &
+ ~cfg2_clrb) | cfg2_setb,
+ spi->base + STM32_SPI_CFG2);
+
+ spin_unlock_irqrestore(&spi->lock, flags);
+
+ return 0;
+}
+
+/**
+ * stm32_spi_dma_cb - dma callback
+ *
+ * DMA callback is called when the transfer is complete or when an error
+ * occurs. If the transfer is complete, EOT flag is raised.
+ */
+static void stm32_spi_dma_cb(void *data)
+{
+ struct stm32_spi *spi = data;
+ unsigned long flags;
+ u32 sr;
+
+ spin_lock_irqsave(&spi->lock, flags);
+
+ sr = readl_relaxed(spi->base + STM32_SPI_SR);
+
+ spin_unlock_irqrestore(&spi->lock, flags);
+
+ if (!(sr & SPI_SR_EOT))
+ dev_warn(spi->dev, "DMA error (sr=0x%08x)\n", sr);
+
+ /* Now wait for EOT, or SUSP or OVR in case of error */
+}
+
+/**
+ * stm32_spi_dma_config - configure dma slave channel depending on current
+ * transfer bits_per_word.
+ */
+static void stm32_spi_dma_config(struct stm32_spi *spi,
+ struct dma_slave_config *dma_conf,
+ enum dma_transfer_direction dir)
+{
+ enum dma_slave_buswidth buswidth;
+ u32 maxburst;
+
+ if (spi->cur_bpw <= 8)
+ buswidth = DMA_SLAVE_BUSWIDTH_1_BYTE;
+ else if (spi->cur_bpw <= 16)
+ buswidth = DMA_SLAVE_BUSWIDTH_2_BYTES;
+ else
+ buswidth = DMA_SLAVE_BUSWIDTH_4_BYTES;
+
+ /* Valid for DMA Half or Full Fifo threshold */
+ if (spi->cur_fthlv == 2)
+ maxburst = 1;
+ else
+ maxburst = spi->cur_fthlv;
+
+ memset(dma_conf, 0, sizeof(struct dma_slave_config));
+ dma_conf->direction = dir;
+ if (dma_conf->direction == DMA_DEV_TO_MEM) { /* RX */
+ dma_conf->src_addr = spi->phys_addr + STM32_SPI_RXDR;
+ dma_conf->src_addr_width = buswidth;
+ dma_conf->src_maxburst = maxburst;
+
+ dev_dbg(spi->dev, "Rx DMA config buswidth=%d, maxburst=%d\n",
+ buswidth, maxburst);
+ } else if (dma_conf->direction == DMA_MEM_TO_DEV) { /* TX */
+ dma_conf->dst_addr = spi->phys_addr + STM32_SPI_TXDR;
+ dma_conf->dst_addr_width = buswidth;
+ dma_conf->dst_maxburst = maxburst;
+
+ dev_dbg(spi->dev, "Tx DMA config buswidth=%d, maxburst=%d\n",
+ buswidth, maxburst);
+ }
+}
+
+/**
+ * stm32_spi_transfer_one_irq - transfer a single spi_transfer using
+ * interrupts
+ *
+ * It must returns 0 if the transfer is finished or 1 if the transfer is still
+ * in progress.
+ */
+static int stm32_spi_transfer_one_irq(struct stm32_spi *spi)
+{
+ unsigned long flags;
+ u32 ier = 0;
+
+ /* Enable the interrupts relative to the current communication mode */
+ if (spi->tx_buf && spi->rx_buf) /* Full Duplex */
+ ier |= SPI_IER_DXPIE;
+ else if (spi->tx_buf) /* Half-Duplex TX dir or Simplex TX */
+ ier |= SPI_IER_TXPIE;
+ else if (spi->rx_buf) /* Half-Duplex RX dir or Simplex RX */
+ ier |= SPI_IER_RXPIE;
+
+ /* Enable the interrupts relative to the end of transfer */
+ ier |= SPI_IER_EOTIE | SPI_IER_TXTFIE | SPI_IER_OVRIE | SPI_IER_MODFIE;
+
+ spin_lock_irqsave(&spi->lock, flags);
+
+ stm32_spi_enable(spi);
+
+ /* Be sure to have data in fifo before starting data transfer */
+ if (spi->tx_buf)
+ stm32_spi_write_txfifo(spi);
+
+ stm32_spi_set_bits(spi, STM32_SPI_CR1, SPI_CR1_CSTART);
+
+ writel_relaxed(ier, spi->base + STM32_SPI_IER);
+
+ spin_unlock_irqrestore(&spi->lock, flags);
+
+ return 1;
+}
+
+/**
+ * stm32_spi_transfer_one_dma - transfer a single spi_transfer using DMA
+ *
+ * It must returns 0 if the transfer is finished or 1 if the transfer is still
+ * in progress.
+ */
+static int stm32_spi_transfer_one_dma(struct stm32_spi *spi,
+ struct spi_transfer *xfer)
+{
+ struct dma_slave_config tx_dma_conf, rx_dma_conf;
+ struct dma_async_tx_descriptor *tx_dma_desc, *rx_dma_desc;
+ unsigned long flags;
+ u32 ier = 0;
+
+ spin_lock_irqsave(&spi->lock, flags);
+
+ rx_dma_desc = NULL;
+ if (spi->rx_buf) {
+ stm32_spi_dma_config(spi, &rx_dma_conf, DMA_DEV_TO_MEM);
+ dmaengine_slave_config(spi->dma_rx, &rx_dma_conf);
+
+ /* Enable Rx DMA request */
+ stm32_spi_set_bits(spi, STM32_SPI_CFG1, SPI_CFG1_RXDMAEN);
+
+ rx_dma_desc = dmaengine_prep_slave_sg(
+ spi->dma_rx, xfer->rx_sg.sgl,
+ xfer->rx_sg.nents,
+ rx_dma_conf.direction,
+ DMA_PREP_INTERRUPT);
+ }
+
+ tx_dma_desc = NULL;
+ if (spi->tx_buf) {
+ stm32_spi_dma_config(spi, &tx_dma_conf, DMA_MEM_TO_DEV);
+ dmaengine_slave_config(spi->dma_tx, &tx_dma_conf);
+
+ tx_dma_desc = dmaengine_prep_slave_sg(
+ spi->dma_tx, xfer->tx_sg.sgl,
+ xfer->tx_sg.nents,
+ tx_dma_conf.direction,
+ DMA_PREP_INTERRUPT);
+ }
+
+ if ((spi->tx_buf && !tx_dma_desc) ||
+ (spi->rx_buf && !rx_dma_desc))
+ goto dma_desc_error;
+
+ if (rx_dma_desc) {
+ rx_dma_desc->callback = stm32_spi_dma_cb;
+ rx_dma_desc->callback_param = spi;
+
+ if (dma_submit_error(dmaengine_submit(rx_dma_desc))) {
+ dev_err(spi->dev, "Rx DMA submit failed\n");
+ goto dma_desc_error;
+ }
+ /* Enable Rx DMA channel */
+ dma_async_issue_pending(spi->dma_rx);
+ }
+
+ if (tx_dma_desc) {
+ if (spi->cur_comm == SPI_SIMPLEX_TX) {
+ tx_dma_desc->callback = stm32_spi_dma_cb;
+ tx_dma_desc->callback_param = spi;
+ }
+
+ if (dma_submit_error(dmaengine_submit(tx_dma_desc))) {
+ dev_err(spi->dev, "Tx DMA submit failed\n");
+ goto dma_submit_error;
+ }
+ /* Enable Tx DMA channel */
+ dma_async_issue_pending(spi->dma_tx);
+
+ /* Enable Tx DMA request */
+ stm32_spi_set_bits(spi, STM32_SPI_CFG1, SPI_CFG1_TXDMAEN);
+ }
+
+ /* Enable the interrupts relative to the end of transfer */
+ ier |= SPI_IER_EOTIE | SPI_IER_TXTFIE | SPI_IER_OVRIE | SPI_IER_MODFIE;
+ writel_relaxed(ier, spi->base + STM32_SPI_IER);
+
+ stm32_spi_enable(spi);
+
+ stm32_spi_set_bits(spi, STM32_SPI_CR1, SPI_CR1_CSTART);
+
+ spin_unlock_irqrestore(&spi->lock, flags);
+
+ return 1;
+
+dma_submit_error:
+ if (spi->rx_buf)
+ dmaengine_terminate_all(spi->dma_rx);
+
+dma_desc_error:
+ stm32_spi_clr_bits(spi, STM32_SPI_CFG1, SPI_CFG1_RXDMAEN);
+
+ spin_unlock_irqrestore(&spi->lock, flags);
+
+ dev_info(spi->dev, "DMA issue: fall back to irq transfer\n");
+
+ return stm32_spi_transfer_one_irq(spi);
+}
+
+/**
+ * stm32_spi_transfer_one_setup - common setup to transfer a single
+ * spi_transfer either using DMA or
+ * interrupts.
+ */
+static int stm32_spi_transfer_one_setup(struct stm32_spi *spi,
+ struct spi_device *spi_dev,
+ struct spi_transfer *transfer)
+{
+ unsigned long flags;
+ u32 cfg1_clrb = 0, cfg1_setb = 0, cfg2_clrb = 0, cfg2_setb = 0;
+ u32 mode, nb_words;
+ int ret = 0;
+
+ spin_lock_irqsave(&spi->lock, flags);
+
+ if (spi->cur_bpw != transfer->bits_per_word) {
+ u32 bpw, fthlv;
+
+ spi->cur_bpw = transfer->bits_per_word;
+ bpw = spi->cur_bpw - 1;
+
+ cfg1_clrb |= SPI_CFG1_DSIZE;
+ cfg1_setb |= (bpw << SPI_CFG1_DSIZE_SHIFT) & SPI_CFG1_DSIZE;
+
+ spi->cur_fthlv = stm32_spi_prepare_fthlv(spi);
+ fthlv = spi->cur_fthlv - 1;
+
+ cfg1_clrb |= SPI_CFG1_FTHLV;
+ cfg1_setb |= (fthlv << SPI_CFG1_FTHLV_SHIFT) & SPI_CFG1_FTHLV;
+ }
+
+ if (spi->cur_speed != transfer->speed_hz) {
+ int mbr;
+
+ /* Update spi->cur_speed with real clock speed */
+ mbr = stm32_spi_prepare_mbr(spi, transfer->speed_hz);
+ if (mbr < 0) {
+ ret = mbr;
+ goto out;
+ }
+
+ transfer->speed_hz = spi->cur_speed;
+
+ cfg1_clrb |= SPI_CFG1_MBR;
+ cfg1_setb |= ((u32)mbr << SPI_CFG1_MBR_SHIFT) & SPI_CFG1_MBR;
+ }
+
+ if (cfg1_clrb || cfg1_setb)
+ writel_relaxed((readl_relaxed(spi->base + STM32_SPI_CFG1) &
+ ~cfg1_clrb) | cfg1_setb,
+ spi->base + STM32_SPI_CFG1);
+
+ mode = SPI_FULL_DUPLEX;
+ if (spi_dev->mode & SPI_3WIRE) { /* MISO/MOSI signals shared */
+ /*
+ * SPI_3WIRE and xfer->tx_buf != NULL and xfer->rx_buf != NULL
+ * is forbidden und unvalidated by SPI subsystem so depending
+ * on the valid buffer, we can determine the direction of the
+ * transfer.
+ */
+ mode = SPI_HALF_DUPLEX;
+ if (!transfer->tx_buf)
+ stm32_spi_clr_bits(spi, STM32_SPI_CR1, SPI_CR1_HDDIR);
+ else if (!transfer->rx_buf)
+ stm32_spi_set_bits(spi, STM32_SPI_CR1, SPI_CR1_HDDIR);
+ } else {
+ if (!transfer->tx_buf)
+ mode = SPI_SIMPLEX_RX;
+ else if (!transfer->rx_buf)
+ mode = SPI_SIMPLEX_TX;
+ }
+ if (spi->cur_comm != mode) {
+ spi->cur_comm = mode;
+
+ cfg2_clrb |= SPI_CFG2_COMM;
+ cfg2_setb |= (mode << SPI_CFG2_COMM_SHIFT) & SPI_CFG2_COMM;
+ }
+
+ cfg2_clrb |= SPI_CFG2_MIDI;
+ if ((transfer->len > 1) && (spi->cur_midi > 0)) {
+ u32 sck_period_ns = DIV_ROUND_UP(SPI_1HZ_NS, spi->cur_speed);
+ u32 midi = min((u32)DIV_ROUND_UP(spi->cur_midi, sck_period_ns),
+ (u32)SPI_CFG2_MIDI >> SPI_CFG2_MIDI_SHIFT);
+
+ dev_dbg(spi->dev, "period=%dns, midi=%d(=%dns)\n",
+ sck_period_ns, midi, midi * sck_period_ns);
+
+ cfg2_setb |= (midi << SPI_CFG2_MIDI_SHIFT) & SPI_CFG2_MIDI;
+ }
+
+ if (cfg2_clrb || cfg2_setb)
+ writel_relaxed((readl_relaxed(spi->base + STM32_SPI_CFG2) &
+ ~cfg2_clrb) | cfg2_setb,
+ spi->base + STM32_SPI_CFG2);
+
+ if (spi->cur_bpw <= 8)
+ nb_words = transfer->len;
+ else if (spi->cur_bpw <= 16)
+ nb_words = DIV_ROUND_UP(transfer->len * 8, 16);
+ else
+ nb_words = DIV_ROUND_UP(transfer->len * 8, 32);
+ nb_words <<= SPI_CR2_TSIZE_SHIFT;
+
+ if (nb_words <= SPI_CR2_TSIZE) {
+ writel_relaxed(nb_words, spi->base + STM32_SPI_CR2);
+ } else {
+ ret = -EMSGSIZE;
+ goto out;
+ }
+
+ spi->cur_xferlen = transfer->len;
+
+ dev_dbg(spi->dev, "transfer communication mode set to %d\n",
+ spi->cur_comm);
+ dev_dbg(spi->dev,
+ "data frame of %d-bit, data packet of %d data frames\n",
+ spi->cur_bpw, spi->cur_fthlv);
+ dev_dbg(spi->dev, "speed set to %dHz\n", spi->cur_speed);
+ dev_dbg(spi->dev, "transfer of %d bytes (%d data frames)\n",
+ spi->cur_xferlen, nb_words);
+ dev_dbg(spi->dev, "dma %s\n",
+ (spi->cur_usedma) ? "enabled" : "disabled");
+
+out:
+ spin_unlock_irqrestore(&spi->lock, flags);
+
+ return ret;
+}
+
+/**
+ * stm32_spi_transfer_one - transfer a single spi_transfer
+ *
+ * It must return 0 if the transfer is finished or 1 if the transfer is still
+ * in progress.
+ */
+static int stm32_spi_transfer_one(struct spi_master *master,
+ struct spi_device *spi_dev,
+ struct spi_transfer *transfer)
+{
+ struct stm32_spi *spi = spi_master_get_devdata(master);
+ int ret;
+
+ spi->tx_buf = transfer->tx_buf;
+ spi->rx_buf = transfer->rx_buf;
+ spi->tx_len = spi->tx_buf ? transfer->len : 0;
+ spi->rx_len = spi->rx_buf ? transfer->len : 0;
+
+ spi->cur_usedma = (master->can_dma &&
+ stm32_spi_can_dma(master, spi_dev, transfer));
+
+ ret = stm32_spi_transfer_one_setup(spi, spi_dev, transfer);
+ if (ret) {
+ dev_err(spi->dev, "SPI transfer setup failed\n");
+ return ret;
+ }
+
+ if (spi->cur_usedma)
+ return stm32_spi_transfer_one_dma(spi, transfer);
+ else
+ return stm32_spi_transfer_one_irq(spi);
+}
+
+/**
+ * stm32_spi_unprepare_msg - relax the hardware
+ *
+ * Normally, if TSIZE has been configured, we should relax the hardware at the
+ * reception of the EOT interrupt. But in case of error, EOT will not be
+ * raised. So the subsystem unprepare_message call allows us to properly
+ * complete the transfer from an hardware point of view.
+ */
+static int stm32_spi_unprepare_msg(struct spi_master *master,
+ struct spi_message *msg)
+{
+ struct stm32_spi *spi = spi_master_get_devdata(master);
+
+ stm32_spi_disable(spi);
+
+ return 0;
+}
+
+/**
+ * stm32_spi_config - Configure SPI controller as SPI master
+ */
+static int stm32_spi_config(struct stm32_spi *spi)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&spi->lock, flags);
+
+ /* Ensure I2SMOD bit is kept cleared */
+ stm32_spi_clr_bits(spi, STM32_SPI_I2SCFGR, SPI_I2SCFGR_I2SMOD);
+
+ /*
+ * - SS input value high
+ * - transmitter half duplex direction
+ * - automatic communication suspend when RX-Fifo is full
+ */
+ stm32_spi_set_bits(spi, STM32_SPI_CR1, SPI_CR1_SSI |
+ SPI_CR1_HDDIR |
+ SPI_CR1_MASRX);
+
+ /*
+ * - Set the master mode (default Motorola mode)
+ * - Consider 1 master/n slaves configuration and
+ * SS input value is determined by the SSI bit
+ * - keep control of all associated GPIOs
+ */
+ stm32_spi_set_bits(spi, STM32_SPI_CFG2, SPI_CFG2_MASTER |
+ SPI_CFG2_SSM |
+ SPI_CFG2_AFCNTR);
+
+ spin_unlock_irqrestore(&spi->lock, flags);
+
+ return 0;
+}
+
+static const struct of_device_id stm32_spi_of_match[] = {
+ { .compatible = "st,stm32h7-spi", },
+ {},
+};
+MODULE_DEVICE_TABLE(of, stm32_spi_of_match);
+
+static int stm32_spi_probe(struct platform_device *pdev)
+{
+ struct spi_master *master;
+ struct stm32_spi *spi;
+ struct resource *res;
+ int i, ret;
+
+ master = spi_alloc_master(&pdev->dev, sizeof(struct stm32_spi));
+ if (!master) {
+ dev_err(&pdev->dev, "spi master allocation failed\n");
+ return -ENOMEM;
+ }
+ platform_set_drvdata(pdev, master);
+
+ spi = spi_master_get_devdata(master);
+ spi->dev = &pdev->dev;
+ spi->master = master;
+ spin_lock_init(&spi->lock);
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ spi->base = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(spi->base)) {
+ ret = PTR_ERR(spi->base);
+ goto err_master_put;
+ }
+ spi->phys_addr = (dma_addr_t)res->start;
+
+ spi->irq = platform_get_irq(pdev, 0);
+ if (spi->irq <= 0) {
+ dev_err(&pdev->dev, "no irq: %d\n", spi->irq);
+ ret = -ENOENT;
+ goto err_master_put;
+ }
+ ret = devm_request_threaded_irq(&pdev->dev, spi->irq, NULL,
+ stm32_spi_irq, IRQF_ONESHOT,
+ pdev->name, master);
+ if (ret) {
+ dev_err(&pdev->dev, "irq%d request failed: %d\n", spi->irq,
+ ret);
+ goto err_master_put;
+ }
+
+ spi->clk = devm_clk_get(&pdev->dev, 0);
+ if (IS_ERR(spi->clk)) {
+ ret = PTR_ERR(spi->clk);
+ dev_err(&pdev->dev, "clk get failed: %d\n", ret);
+ goto err_master_put;
+ }
+
+ ret = clk_prepare_enable(spi->clk);
+ if (ret) {
+ dev_err(&pdev->dev, "clk enable failed: %d\n", ret);
+ goto err_master_put;
+ }
+ spi->clk_rate = clk_get_rate(spi->clk);
+ if (!spi->clk_rate) {
+ dev_err(&pdev->dev, "clk rate = 0\n");
+ ret = -EINVAL;
+ goto err_master_put;
+ }
+
+ spi->rst = devm_reset_control_get(&pdev->dev, NULL);
+ if (!IS_ERR(spi->rst)) {
+ reset_control_assert(spi->rst);
+ udelay(2);
+ reset_control_deassert(spi->rst);
+ }
+
+ spi->fifo_size = stm32_spi_get_fifo_size(spi);
+
+ ret = stm32_spi_config(spi);
+ if (ret) {
+ dev_err(&pdev->dev, "controller configuration failed: %d\n",
+ ret);
+ goto err_clk_disable;
+ }
+
+ master->dev.of_node = pdev->dev.of_node;
+ master->auto_runtime_pm = true;
+ master->bus_num = pdev->id;
+ master->mode_bits = SPI_MODE_3 | SPI_CS_HIGH | SPI_LSB_FIRST |
+ SPI_3WIRE | SPI_LOOP;
+ master->bits_per_word_mask = stm32_spi_get_bpw_mask(spi);
+ master->max_speed_hz = spi->clk_rate / SPI_MBR_DIV_MIN;
+ master->min_speed_hz = spi->clk_rate / SPI_MBR_DIV_MAX;
+ master->setup = stm32_spi_setup;
+ master->prepare_message = stm32_spi_prepare_msg;
+ master->transfer_one = stm32_spi_transfer_one;
+ master->unprepare_message = stm32_spi_unprepare_msg;
+
+ spi->dma_tx = dma_request_slave_channel(spi->dev, "tx");
+ if (!spi->dma_tx)
+ dev_warn(&pdev->dev, "failed to request tx dma channel\n");
+ else
+ master->dma_tx = spi->dma_tx;
+
+ spi->dma_rx = dma_request_slave_channel(spi->dev, "rx");
+ if (!spi->dma_rx)
+ dev_warn(&pdev->dev, "failed to request rx dma channel\n");
+ else
+ master->dma_rx = spi->dma_rx;
+
+ if (spi->dma_tx || spi->dma_rx)
+ master->can_dma = stm32_spi_can_dma;
+
+ pm_runtime_set_active(&pdev->dev);
+ pm_runtime_enable(&pdev->dev);
+
+ ret = devm_spi_register_master(&pdev->dev, master);
+ if (ret) {
+ dev_err(&pdev->dev, "spi master registration failed: %d\n",
+ ret);
+ goto err_dma_release;
+ }
+
+ if (!master->cs_gpios) {
+ dev_err(&pdev->dev, "no CS gpios available\n");
+ ret = -EINVAL;
+ goto err_dma_release;
+ }
+
+ for (i = 0; i < master->num_chipselect; i++) {
+ if (!gpio_is_valid(master->cs_gpios[i])) {
+ dev_err(&pdev->dev, "%i is not a valid gpio\n",
+ master->cs_gpios[i]);
+ ret = -EINVAL;
+ goto err_dma_release;
+ }
+
+ ret = devm_gpio_request(&pdev->dev, master->cs_gpios[i],
+ DRIVER_NAME);
+ if (ret) {
+ dev_err(&pdev->dev, "can't get CS gpio %i\n",
+ master->cs_gpios[i]);
+ goto err_dma_release;
+ }
+ }
+
+ dev_info(&pdev->dev, "driver initialized\n");
+
+ return 0;
+
+err_dma_release:
+ if (spi->dma_tx)
+ dma_release_channel(spi->dma_tx);
+ if (spi->dma_rx)
+ dma_release_channel(spi->dma_rx);
+
+ pm_runtime_disable(&pdev->dev);
+err_clk_disable:
+ clk_disable_unprepare(spi->clk);
+err_master_put:
+ spi_master_put(master);
+
+ return ret;
+}
+
+static int stm32_spi_remove(struct platform_device *pdev)
+{
+ struct spi_master *master = platform_get_drvdata(pdev);
+ struct stm32_spi *spi = spi_master_get_devdata(master);
+
+ stm32_spi_disable(spi);
+
+ if (master->dma_tx)
+ dma_release_channel(master->dma_tx);
+ if (master->dma_rx)
+ dma_release_channel(master->dma_rx);
+
+ clk_disable_unprepare(spi->clk);
+
+ pm_runtime_disable(&pdev->dev);
+
+ return 0;
+}
+
+#ifdef CONFIG_PM
+static int stm32_spi_runtime_suspend(struct device *dev)
+{
+ struct spi_master *master = dev_get_drvdata(dev);
+ struct stm32_spi *spi = spi_master_get_devdata(master);
+
+ clk_disable_unprepare(spi->clk);
+
+ return 0;
+}
+
+static int stm32_spi_runtime_resume(struct device *dev)
+{
+ struct spi_master *master = dev_get_drvdata(dev);
+ struct stm32_spi *spi = spi_master_get_devdata(master);
+
+ return clk_prepare_enable(spi->clk);
+}
+#endif
+
+#ifdef CONFIG_PM_SLEEP
+static int stm32_spi_suspend(struct device *dev)
+{
+ struct spi_master *master = dev_get_drvdata(dev);
+ int ret;
+
+ ret = spi_master_suspend(master);
+ if (ret)
+ return ret;
+
+ return pm_runtime_force_suspend(dev);
+}
+
+static int stm32_spi_resume(struct device *dev)
+{
+ struct spi_master *master = dev_get_drvdata(dev);
+ struct stm32_spi *spi = spi_master_get_devdata(master);
+ int ret;
+
+ ret = pm_runtime_force_resume(dev);
+ if (ret)
+ return ret;
+
+ ret = spi_master_resume(master);
+ if (ret)
+ clk_disable_unprepare(spi->clk);
+
+ return ret;
+}
+#endif
+
+static const struct dev_pm_ops stm32_spi_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(stm32_spi_suspend, stm32_spi_resume)
+ SET_RUNTIME_PM_OPS(stm32_spi_runtime_suspend,
+ stm32_spi_runtime_resume, NULL)
+};
+
+static struct platform_driver stm32_spi_driver = {
+ .probe = stm32_spi_probe,
+ .remove = stm32_spi_remove,
+ .driver = {
+ .name = DRIVER_NAME,
+ .pm = &stm32_spi_pm_ops,
+ .of_match_table = stm32_spi_of_match,
+ },
+};
+
+module_platform_driver(stm32_spi_driver);
+
+MODULE_ALIAS("platform:" DRIVER_NAME);
+MODULE_DESCRIPTION("STMicroelectronics STM32 SPI Controller driver");
+MODULE_AUTHOR("Amelie Delaunay <amelie.delaunay@st.com>");
+MODULE_LICENSE("GPL v2");
{
struct spi_device *spi = to_spi_device(dev);
- /* spi masters may cleanup for released devices */
- if (spi->master->cleanup)
- spi->master->cleanup(spi);
+ /* spi controllers may cleanup for released devices */
+ if (spi->controller->cleanup)
+ spi->controller->cleanup(spi);
- spi_master_put(spi->master);
+ spi_controller_put(spi->controller);
kfree(spi);
}
static DEVICE_ATTR_RO(modalias);
#define SPI_STATISTICS_ATTRS(field, file) \
-static ssize_t spi_master_##field##_show(struct device *dev, \
- struct device_attribute *attr, \
- char *buf) \
+static ssize_t spi_controller_##field##_show(struct device *dev, \
+ struct device_attribute *attr, \
+ char *buf) \
{ \
- struct spi_master *master = container_of(dev, \
- struct spi_master, dev); \
- return spi_statistics_##field##_show(&master->statistics, buf); \
+ struct spi_controller *ctlr = container_of(dev, \
+ struct spi_controller, dev); \
+ return spi_statistics_##field##_show(&ctlr->statistics, buf); \
} \
-static struct device_attribute dev_attr_spi_master_##field = { \
- .attr = { .name = file, .mode = S_IRUGO }, \
- .show = spi_master_##field##_show, \
+static struct device_attribute dev_attr_spi_controller_##field = { \
+ .attr = { .name = file, .mode = 0444 }, \
+ .show = spi_controller_##field##_show, \
}; \
static ssize_t spi_device_##field##_show(struct device *dev, \
struct device_attribute *attr, \
return spi_statistics_##field##_show(&spi->statistics, buf); \
} \
static struct device_attribute dev_attr_spi_device_##field = { \
- .attr = { .name = file, .mode = S_IRUGO }, \
+ .attr = { .name = file, .mode = 0444 }, \
.show = spi_device_##field##_show, \
}
NULL,
};
-static struct attribute *spi_master_statistics_attrs[] = {
- &dev_attr_spi_master_messages.attr,
- &dev_attr_spi_master_transfers.attr,
- &dev_attr_spi_master_errors.attr,
- &dev_attr_spi_master_timedout.attr,
- &dev_attr_spi_master_spi_sync.attr,
- &dev_attr_spi_master_spi_sync_immediate.attr,
- &dev_attr_spi_master_spi_async.attr,
- &dev_attr_spi_master_bytes.attr,
- &dev_attr_spi_master_bytes_rx.attr,
- &dev_attr_spi_master_bytes_tx.attr,
- &dev_attr_spi_master_transfer_bytes_histo0.attr,
- &dev_attr_spi_master_transfer_bytes_histo1.attr,
- &dev_attr_spi_master_transfer_bytes_histo2.attr,
- &dev_attr_spi_master_transfer_bytes_histo3.attr,
- &dev_attr_spi_master_transfer_bytes_histo4.attr,
- &dev_attr_spi_master_transfer_bytes_histo5.attr,
- &dev_attr_spi_master_transfer_bytes_histo6.attr,
- &dev_attr_spi_master_transfer_bytes_histo7.attr,
- &dev_attr_spi_master_transfer_bytes_histo8.attr,
- &dev_attr_spi_master_transfer_bytes_histo9.attr,
- &dev_attr_spi_master_transfer_bytes_histo10.attr,
- &dev_attr_spi_master_transfer_bytes_histo11.attr,
- &dev_attr_spi_master_transfer_bytes_histo12.attr,
- &dev_attr_spi_master_transfer_bytes_histo13.attr,
- &dev_attr_spi_master_transfer_bytes_histo14.attr,
- &dev_attr_spi_master_transfer_bytes_histo15.attr,
- &dev_attr_spi_master_transfer_bytes_histo16.attr,
- &dev_attr_spi_master_transfers_split_maxsize.attr,
+static struct attribute *spi_controller_statistics_attrs[] = {
+ &dev_attr_spi_controller_messages.attr,
+ &dev_attr_spi_controller_transfers.attr,
+ &dev_attr_spi_controller_errors.attr,
+ &dev_attr_spi_controller_timedout.attr,
+ &dev_attr_spi_controller_spi_sync.attr,
+ &dev_attr_spi_controller_spi_sync_immediate.attr,
+ &dev_attr_spi_controller_spi_async.attr,
+ &dev_attr_spi_controller_bytes.attr,
+ &dev_attr_spi_controller_bytes_rx.attr,
+ &dev_attr_spi_controller_bytes_tx.attr,
+ &dev_attr_spi_controller_transfer_bytes_histo0.attr,
+ &dev_attr_spi_controller_transfer_bytes_histo1.attr,
+ &dev_attr_spi_controller_transfer_bytes_histo2.attr,
+ &dev_attr_spi_controller_transfer_bytes_histo3.attr,
+ &dev_attr_spi_controller_transfer_bytes_histo4.attr,
+ &dev_attr_spi_controller_transfer_bytes_histo5.attr,
+ &dev_attr_spi_controller_transfer_bytes_histo6.attr,
+ &dev_attr_spi_controller_transfer_bytes_histo7.attr,
+ &dev_attr_spi_controller_transfer_bytes_histo8.attr,
+ &dev_attr_spi_controller_transfer_bytes_histo9.attr,
+ &dev_attr_spi_controller_transfer_bytes_histo10.attr,
+ &dev_attr_spi_controller_transfer_bytes_histo11.attr,
+ &dev_attr_spi_controller_transfer_bytes_histo12.attr,
+ &dev_attr_spi_controller_transfer_bytes_histo13.attr,
+ &dev_attr_spi_controller_transfer_bytes_histo14.attr,
+ &dev_attr_spi_controller_transfer_bytes_histo15.attr,
+ &dev_attr_spi_controller_transfer_bytes_histo16.attr,
+ &dev_attr_spi_controller_transfers_split_maxsize.attr,
NULL,
};
-static const struct attribute_group spi_master_statistics_group = {
+static const struct attribute_group spi_controller_statistics_group = {
.name = "statistics",
- .attrs = spi_master_statistics_attrs,
+ .attrs = spi_controller_statistics_attrs,
};
static const struct attribute_group *spi_master_groups[] = {
- &spi_master_statistics_group,
+ &spi_controller_statistics_group,
NULL,
};
void spi_statistics_add_transfer_stats(struct spi_statistics *stats,
struct spi_transfer *xfer,
- struct spi_master *master)
+ struct spi_controller *ctlr)
{
unsigned long flags;
int l2len = min(fls(xfer->len), SPI_STATISTICS_HISTO_SIZE) - 1;
stats->bytes += xfer->len;
if ((xfer->tx_buf) &&
- (xfer->tx_buf != master->dummy_tx))
+ (xfer->tx_buf != ctlr->dummy_tx))
stats->bytes_tx += xfer->len;
if ((xfer->rx_buf) &&
- (xfer->rx_buf != master->dummy_rx))
+ (xfer->rx_buf != ctlr->dummy_rx))
stats->bytes_rx += xfer->len;
spin_unlock_irqrestore(&stats->lock, flags);
/*-------------------------------------------------------------------------*/
/* SPI devices should normally not be created by SPI device drivers; that
- * would make them board-specific. Similarly with SPI master drivers.
+ * would make them board-specific. Similarly with SPI controller drivers.
* Device registration normally goes into like arch/.../mach.../board-YYY.c
* with other readonly (flashable) information about mainboard devices.
*/
};
static LIST_HEAD(board_list);
-static LIST_HEAD(spi_master_list);
+static LIST_HEAD(spi_controller_list);
/*
* Used to protect add/del opertion for board_info list and
- * spi_master list, and their matching process
+ * spi_controller list, and their matching process
*/
static DEFINE_MUTEX(board_lock);
/**
* spi_alloc_device - Allocate a new SPI device
- * @master: Controller to which device is connected
+ * @ctlr: Controller to which device is connected
* Context: can sleep
*
* Allows a driver to allocate and initialize a spi_device without
* spi_add_device() on it.
*
* Caller is responsible to call spi_add_device() on the returned
- * spi_device structure to add it to the SPI master. If the caller
+ * spi_device structure to add it to the SPI controller. If the caller
* needs to discard the spi_device without adding it, then it should
* call spi_dev_put() on it.
*
* Return: a pointer to the new device, or NULL.
*/
-struct spi_device *spi_alloc_device(struct spi_master *master)
+struct spi_device *spi_alloc_device(struct spi_controller *ctlr)
{
struct spi_device *spi;
- if (!spi_master_get(master))
+ if (!spi_controller_get(ctlr))
return NULL;
spi = kzalloc(sizeof(*spi), GFP_KERNEL);
if (!spi) {
- spi_master_put(master);
+ spi_controller_put(ctlr);
return NULL;
}
- spi->master = master;
- spi->dev.parent = &master->dev;
+ spi->master = spi->controller = ctlr;
+ spi->dev.parent = &ctlr->dev;
spi->dev.bus = &spi_bus_type;
spi->dev.release = spidev_release;
spi->cs_gpio = -ENOENT;
return;
}
- dev_set_name(&spi->dev, "%s.%u", dev_name(&spi->master->dev),
+ dev_set_name(&spi->dev, "%s.%u", dev_name(&spi->controller->dev),
spi->chip_select);
}
struct spi_device *spi = to_spi_device(dev);
struct spi_device *new_spi = data;
- if (spi->master == new_spi->master &&
+ if (spi->controller == new_spi->controller &&
spi->chip_select == new_spi->chip_select)
return -EBUSY;
return 0;
int spi_add_device(struct spi_device *spi)
{
static DEFINE_MUTEX(spi_add_lock);
- struct spi_master *master = spi->master;
- struct device *dev = master->dev.parent;
+ struct spi_controller *ctlr = spi->controller;
+ struct device *dev = ctlr->dev.parent;
int status;
/* Chipselects are numbered 0..max; validate. */
- if (spi->chip_select >= master->num_chipselect) {
- dev_err(dev, "cs%d >= max %d\n",
- spi->chip_select,
- master->num_chipselect);
+ if (spi->chip_select >= ctlr->num_chipselect) {
+ dev_err(dev, "cs%d >= max %d\n", spi->chip_select,
+ ctlr->num_chipselect);
return -EINVAL;
}
goto done;
}
- if (master->cs_gpios)
- spi->cs_gpio = master->cs_gpios[spi->chip_select];
+ if (ctlr->cs_gpios)
+ spi->cs_gpio = ctlr->cs_gpios[spi->chip_select];
/* Drivers may modify this initial i/o setup, but will
* normally rely on the device being setup. Devices
/**
* spi_new_device - instantiate one new SPI device
- * @master: Controller to which device is connected
+ * @ctlr: Controller to which device is connected
* @chip: Describes the SPI device
* Context: can sleep
*
*
* Return: the new device, or NULL.
*/
-struct spi_device *spi_new_device(struct spi_master *master,
+struct spi_device *spi_new_device(struct spi_controller *ctlr,
struct spi_board_info *chip)
{
struct spi_device *proxy;
* suggests syslogged diagnostics are best here (ugh).
*/
- proxy = spi_alloc_device(master);
+ proxy = spi_alloc_device(ctlr);
if (!proxy)
return NULL;
if (chip->properties) {
status = device_add_properties(&proxy->dev, chip->properties);
if (status) {
- dev_err(&master->dev,
+ dev_err(&ctlr->dev,
"failed to add properties to '%s': %d\n",
chip->modalias, status);
goto err_dev_put;
* @spi: spi_device to unregister
*
* Start making the passed SPI device vanish. Normally this would be handled
- * by spi_unregister_master().
+ * by spi_unregister_controller().
*/
void spi_unregister_device(struct spi_device *spi)
{
}
EXPORT_SYMBOL_GPL(spi_unregister_device);
-static void spi_match_master_to_boardinfo(struct spi_master *master,
- struct spi_board_info *bi)
+static void spi_match_controller_to_boardinfo(struct spi_controller *ctlr,
+ struct spi_board_info *bi)
{
struct spi_device *dev;
- if (master->bus_num != bi->bus_num)
+ if (ctlr->bus_num != bi->bus_num)
return;
- dev = spi_new_device(master, bi);
+ dev = spi_new_device(ctlr, bi);
if (!dev)
- dev_err(master->dev.parent, "can't create new device for %s\n",
+ dev_err(ctlr->dev.parent, "can't create new device for %s\n",
bi->modalias);
}
return -ENOMEM;
for (i = 0; i < n; i++, bi++, info++) {
- struct spi_master *master;
+ struct spi_controller *ctlr;
memcpy(&bi->board_info, info, sizeof(*info));
if (info->properties) {
mutex_lock(&board_lock);
list_add_tail(&bi->list, &board_list);
- list_for_each_entry(master, &spi_master_list, list)
- spi_match_master_to_boardinfo(master, &bi->board_info);
+ list_for_each_entry(ctlr, &spi_controller_list, list)
+ spi_match_controller_to_boardinfo(ctlr,
+ &bi->board_info);
mutex_unlock(&board_lock);
}
if (gpio_is_valid(spi->cs_gpio)) {
gpio_set_value(spi->cs_gpio, !enable);
/* Some SPI masters need both GPIO CS & slave_select */
- if ((spi->master->flags & SPI_MASTER_GPIO_SS) &&
- spi->master->set_cs)
- spi->master->set_cs(spi, !enable);
- } else if (spi->master->set_cs) {
- spi->master->set_cs(spi, !enable);
+ if ((spi->controller->flags & SPI_MASTER_GPIO_SS) &&
+ spi->controller->set_cs)
+ spi->controller->set_cs(spi, !enable);
+ } else if (spi->controller->set_cs) {
+ spi->controller->set_cs(spi, !enable);
}
}
#ifdef CONFIG_HAS_DMA
-static int spi_map_buf(struct spi_master *master, struct device *dev,
+static int spi_map_buf(struct spi_controller *ctlr, struct device *dev,
struct sg_table *sgt, void *buf, size_t len,
enum dma_data_direction dir)
{
desc_len = min_t(int, max_seg_size, PAGE_SIZE);
sgs = DIV_ROUND_UP(len + offset_in_page(buf), desc_len);
} else if (virt_addr_valid(buf)) {
- desc_len = min_t(int, max_seg_size, master->max_dma_len);
+ desc_len = min_t(int, max_seg_size, ctlr->max_dma_len);
sgs = DIV_ROUND_UP(len, desc_len);
} else {
return -EINVAL;
return 0;
}
-static void spi_unmap_buf(struct spi_master *master, struct device *dev,
+static void spi_unmap_buf(struct spi_controller *ctlr, struct device *dev,
struct sg_table *sgt, enum dma_data_direction dir)
{
if (sgt->orig_nents) {
}
}
-static int __spi_map_msg(struct spi_master *master, struct spi_message *msg)
+static int __spi_map_msg(struct spi_controller *ctlr, struct spi_message *msg)
{
struct device *tx_dev, *rx_dev;
struct spi_transfer *xfer;
int ret;
- if (!master->can_dma)
+ if (!ctlr->can_dma)
return 0;
- if (master->dma_tx)
- tx_dev = master->dma_tx->device->dev;
+ if (ctlr->dma_tx)
+ tx_dev = ctlr->dma_tx->device->dev;
else
- tx_dev = master->dev.parent;
+ tx_dev = ctlr->dev.parent;
- if (master->dma_rx)
- rx_dev = master->dma_rx->device->dev;
+ if (ctlr->dma_rx)
+ rx_dev = ctlr->dma_rx->device->dev;
else
- rx_dev = master->dev.parent;
+ rx_dev = ctlr->dev.parent;
list_for_each_entry(xfer, &msg->transfers, transfer_list) {
- if (!master->can_dma(master, msg->spi, xfer))
+ if (!ctlr->can_dma(ctlr, msg->spi, xfer))
continue;
if (xfer->tx_buf != NULL) {
- ret = spi_map_buf(master, tx_dev, &xfer->tx_sg,
+ ret = spi_map_buf(ctlr, tx_dev, &xfer->tx_sg,
(void *)xfer->tx_buf, xfer->len,
DMA_TO_DEVICE);
if (ret != 0)
}
if (xfer->rx_buf != NULL) {
- ret = spi_map_buf(master, rx_dev, &xfer->rx_sg,
+ ret = spi_map_buf(ctlr, rx_dev, &xfer->rx_sg,
xfer->rx_buf, xfer->len,
DMA_FROM_DEVICE);
if (ret != 0) {
- spi_unmap_buf(master, tx_dev, &xfer->tx_sg,
+ spi_unmap_buf(ctlr, tx_dev, &xfer->tx_sg,
DMA_TO_DEVICE);
return ret;
}
}
}
- master->cur_msg_mapped = true;
+ ctlr->cur_msg_mapped = true;
return 0;
}
-static int __spi_unmap_msg(struct spi_master *master, struct spi_message *msg)
+static int __spi_unmap_msg(struct spi_controller *ctlr, struct spi_message *msg)
{
struct spi_transfer *xfer;
struct device *tx_dev, *rx_dev;
- if (!master->cur_msg_mapped || !master->can_dma)
+ if (!ctlr->cur_msg_mapped || !ctlr->can_dma)
return 0;
- if (master->dma_tx)
- tx_dev = master->dma_tx->device->dev;
+ if (ctlr->dma_tx)
+ tx_dev = ctlr->dma_tx->device->dev;
else
- tx_dev = master->dev.parent;
+ tx_dev = ctlr->dev.parent;
- if (master->dma_rx)
- rx_dev = master->dma_rx->device->dev;
+ if (ctlr->dma_rx)
+ rx_dev = ctlr->dma_rx->device->dev;
else
- rx_dev = master->dev.parent;
+ rx_dev = ctlr->dev.parent;
list_for_each_entry(xfer, &msg->transfers, transfer_list) {
- if (!master->can_dma(master, msg->spi, xfer))
+ if (!ctlr->can_dma(ctlr, msg->spi, xfer))
continue;
- spi_unmap_buf(master, rx_dev, &xfer->rx_sg, DMA_FROM_DEVICE);
- spi_unmap_buf(master, tx_dev, &xfer->tx_sg, DMA_TO_DEVICE);
+ spi_unmap_buf(ctlr, rx_dev, &xfer->rx_sg, DMA_FROM_DEVICE);
+ spi_unmap_buf(ctlr, tx_dev, &xfer->tx_sg, DMA_TO_DEVICE);
}
return 0;
}
#else /* !CONFIG_HAS_DMA */
-static inline int spi_map_buf(struct spi_master *master,
- struct device *dev, struct sg_table *sgt,
- void *buf, size_t len,
+static inline int spi_map_buf(struct spi_controller *ctlr, struct device *dev,
+ struct sg_table *sgt, void *buf, size_t len,
enum dma_data_direction dir)
{
return -EINVAL;
}
-static inline void spi_unmap_buf(struct spi_master *master,
+static inline void spi_unmap_buf(struct spi_controller *ctlr,
struct device *dev, struct sg_table *sgt,
enum dma_data_direction dir)
{
}
-static inline int __spi_map_msg(struct spi_master *master,
+static inline int __spi_map_msg(struct spi_controller *ctlr,
struct spi_message *msg)
{
return 0;
}
-static inline int __spi_unmap_msg(struct spi_master *master,
+static inline int __spi_unmap_msg(struct spi_controller *ctlr,
struct spi_message *msg)
{
return 0;
}
#endif /* !CONFIG_HAS_DMA */
-static inline int spi_unmap_msg(struct spi_master *master,
+static inline int spi_unmap_msg(struct spi_controller *ctlr,
struct spi_message *msg)
{
struct spi_transfer *xfer;
* Restore the original value of tx_buf or rx_buf if they are
* NULL.
*/
- if (xfer->tx_buf == master->dummy_tx)
+ if (xfer->tx_buf == ctlr->dummy_tx)
xfer->tx_buf = NULL;
- if (xfer->rx_buf == master->dummy_rx)
+ if (xfer->rx_buf == ctlr->dummy_rx)
xfer->rx_buf = NULL;
}
- return __spi_unmap_msg(master, msg);
+ return __spi_unmap_msg(ctlr, msg);
}
-static int spi_map_msg(struct spi_master *master, struct spi_message *msg)
+static int spi_map_msg(struct spi_controller *ctlr, struct spi_message *msg)
{
struct spi_transfer *xfer;
void *tmp;
unsigned int max_tx, max_rx;
- if (master->flags & (SPI_MASTER_MUST_RX | SPI_MASTER_MUST_TX)) {
+ if (ctlr->flags & (SPI_CONTROLLER_MUST_RX | SPI_CONTROLLER_MUST_TX)) {
max_tx = 0;
max_rx = 0;
list_for_each_entry(xfer, &msg->transfers, transfer_list) {
- if ((master->flags & SPI_MASTER_MUST_TX) &&
+ if ((ctlr->flags & SPI_CONTROLLER_MUST_TX) &&
!xfer->tx_buf)
max_tx = max(xfer->len, max_tx);
- if ((master->flags & SPI_MASTER_MUST_RX) &&
+ if ((ctlr->flags & SPI_CONTROLLER_MUST_RX) &&
!xfer->rx_buf)
max_rx = max(xfer->len, max_rx);
}
if (max_tx) {
- tmp = krealloc(master->dummy_tx, max_tx,
+ tmp = krealloc(ctlr->dummy_tx, max_tx,
GFP_KERNEL | GFP_DMA);
if (!tmp)
return -ENOMEM;
- master->dummy_tx = tmp;
+ ctlr->dummy_tx = tmp;
memset(tmp, 0, max_tx);
}
if (max_rx) {
- tmp = krealloc(master->dummy_rx, max_rx,
+ tmp = krealloc(ctlr->dummy_rx, max_rx,
GFP_KERNEL | GFP_DMA);
if (!tmp)
return -ENOMEM;
- master->dummy_rx = tmp;
+ ctlr->dummy_rx = tmp;
}
if (max_tx || max_rx) {
list_for_each_entry(xfer, &msg->transfers,
transfer_list) {
if (!xfer->tx_buf)
- xfer->tx_buf = master->dummy_tx;
+ xfer->tx_buf = ctlr->dummy_tx;
if (!xfer->rx_buf)
- xfer->rx_buf = master->dummy_rx;
+ xfer->rx_buf = ctlr->dummy_rx;
}
}
}
- return __spi_map_msg(master, msg);
+ return __spi_map_msg(ctlr, msg);
}
/*
* drivers which implement a transfer_one() operation. It provides
* standard handling of delays and chip select management.
*/
-static int spi_transfer_one_message(struct spi_master *master,
+static int spi_transfer_one_message(struct spi_controller *ctlr,
struct spi_message *msg)
{
struct spi_transfer *xfer;
bool keep_cs = false;
int ret = 0;
unsigned long long ms = 1;
- struct spi_statistics *statm = &master->statistics;
+ struct spi_statistics *statm = &ctlr->statistics;
struct spi_statistics *stats = &msg->spi->statistics;
spi_set_cs(msg->spi, true);
list_for_each_entry(xfer, &msg->transfers, transfer_list) {
trace_spi_transfer_start(msg, xfer);
- spi_statistics_add_transfer_stats(statm, xfer, master);
- spi_statistics_add_transfer_stats(stats, xfer, master);
+ spi_statistics_add_transfer_stats(statm, xfer, ctlr);
+ spi_statistics_add_transfer_stats(stats, xfer, ctlr);
if (xfer->tx_buf || xfer->rx_buf) {
- reinit_completion(&master->xfer_completion);
+ reinit_completion(&ctlr->xfer_completion);
- ret = master->transfer_one(master, msg->spi, xfer);
+ ret = ctlr->transfer_one(ctlr, msg->spi, xfer);
if (ret < 0) {
SPI_STATISTICS_INCREMENT_FIELD(statm,
errors);
if (ms > UINT_MAX)
ms = UINT_MAX;
- ms = wait_for_completion_timeout(&master->xfer_completion,
+ ms = wait_for_completion_timeout(&ctlr->xfer_completion,
msecs_to_jiffies(ms));
}
if (msg->status == -EINPROGRESS)
msg->status = ret;
- if (msg->status && master->handle_err)
- master->handle_err(master, msg);
+ if (msg->status && ctlr->handle_err)
+ ctlr->handle_err(ctlr, msg);
- spi_res_release(master, msg);
+ spi_res_release(ctlr, msg);
- spi_finalize_current_message(master);
+ spi_finalize_current_message(ctlr);
return ret;
}
/**
* spi_finalize_current_transfer - report completion of a transfer
- * @master: the master reporting completion
+ * @ctlr: the controller reporting completion
*
* Called by SPI drivers using the core transfer_one_message()
* implementation to notify it that the current interrupt driven
* transfer has finished and the next one may be scheduled.
*/
-void spi_finalize_current_transfer(struct spi_master *master)
+void spi_finalize_current_transfer(struct spi_controller *ctlr)
{
- complete(&master->xfer_completion);
+ complete(&ctlr->xfer_completion);
}
EXPORT_SYMBOL_GPL(spi_finalize_current_transfer);
/**
* __spi_pump_messages - function which processes spi message queue
- * @master: master to process queue for
+ * @ctlr: controller to process queue for
* @in_kthread: true if we are in the context of the message pump thread
*
* This function checks if there is any spi message in the queue that
* inside spi_sync(); the queue extraction handling at the top of the
* function should deal with this safely.
*/
-static void __spi_pump_messages(struct spi_master *master, bool in_kthread)
+static void __spi_pump_messages(struct spi_controller *ctlr, bool in_kthread)
{
unsigned long flags;
bool was_busy = false;
int ret;
/* Lock queue */
- spin_lock_irqsave(&master->queue_lock, flags);
+ spin_lock_irqsave(&ctlr->queue_lock, flags);
/* Make sure we are not already running a message */
- if (master->cur_msg) {
- spin_unlock_irqrestore(&master->queue_lock, flags);
+ if (ctlr->cur_msg) {
+ spin_unlock_irqrestore(&ctlr->queue_lock, flags);
return;
}
/* If another context is idling the device then defer */
- if (master->idling) {
- kthread_queue_work(&master->kworker, &master->pump_messages);
- spin_unlock_irqrestore(&master->queue_lock, flags);
+ if (ctlr->idling) {
+ kthread_queue_work(&ctlr->kworker, &ctlr->pump_messages);
+ spin_unlock_irqrestore(&ctlr->queue_lock, flags);
return;
}
/* Check if the queue is idle */
- if (list_empty(&master->queue) || !master->running) {
- if (!master->busy) {
- spin_unlock_irqrestore(&master->queue_lock, flags);
+ if (list_empty(&ctlr->queue) || !ctlr->running) {
+ if (!ctlr->busy) {
+ spin_unlock_irqrestore(&ctlr->queue_lock, flags);
return;
}
/* Only do teardown in the thread */
if (!in_kthread) {
- kthread_queue_work(&master->kworker,
- &master->pump_messages);
- spin_unlock_irqrestore(&master->queue_lock, flags);
+ kthread_queue_work(&ctlr->kworker,
+ &ctlr->pump_messages);
+ spin_unlock_irqrestore(&ctlr->queue_lock, flags);
return;
}
- master->busy = false;
- master->idling = true;
- spin_unlock_irqrestore(&master->queue_lock, flags);
-
- kfree(master->dummy_rx);
- master->dummy_rx = NULL;
- kfree(master->dummy_tx);
- master->dummy_tx = NULL;
- if (master->unprepare_transfer_hardware &&
- master->unprepare_transfer_hardware(master))
- dev_err(&master->dev,
+ ctlr->busy = false;
+ ctlr->idling = true;
+ spin_unlock_irqrestore(&ctlr->queue_lock, flags);
+
+ kfree(ctlr->dummy_rx);
+ ctlr->dummy_rx = NULL;
+ kfree(ctlr->dummy_tx);
+ ctlr->dummy_tx = NULL;
+ if (ctlr->unprepare_transfer_hardware &&
+ ctlr->unprepare_transfer_hardware(ctlr))
+ dev_err(&ctlr->dev,
"failed to unprepare transfer hardware\n");
- if (master->auto_runtime_pm) {
- pm_runtime_mark_last_busy(master->dev.parent);
- pm_runtime_put_autosuspend(master->dev.parent);
+ if (ctlr->auto_runtime_pm) {
+ pm_runtime_mark_last_busy(ctlr->dev.parent);
+ pm_runtime_put_autosuspend(ctlr->dev.parent);
}
- trace_spi_master_idle(master);
+ trace_spi_controller_idle(ctlr);
- spin_lock_irqsave(&master->queue_lock, flags);
- master->idling = false;
- spin_unlock_irqrestore(&master->queue_lock, flags);
+ spin_lock_irqsave(&ctlr->queue_lock, flags);
+ ctlr->idling = false;
+ spin_unlock_irqrestore(&ctlr->queue_lock, flags);
return;
}
/* Extract head of queue */
- master->cur_msg =
- list_first_entry(&master->queue, struct spi_message, queue);
+ ctlr->cur_msg =
+ list_first_entry(&ctlr->queue, struct spi_message, queue);
- list_del_init(&master->cur_msg->queue);
- if (master->busy)
+ list_del_init(&ctlr->cur_msg->queue);
+ if (ctlr->busy)
was_busy = true;
else
- master->busy = true;
- spin_unlock_irqrestore(&master->queue_lock, flags);
+ ctlr->busy = true;
+ spin_unlock_irqrestore(&ctlr->queue_lock, flags);
- mutex_lock(&master->io_mutex);
+ mutex_lock(&ctlr->io_mutex);
- if (!was_busy && master->auto_runtime_pm) {
- ret = pm_runtime_get_sync(master->dev.parent);
+ if (!was_busy && ctlr->auto_runtime_pm) {
+ ret = pm_runtime_get_sync(ctlr->dev.parent);
if (ret < 0) {
- dev_err(&master->dev, "Failed to power device: %d\n",
+ dev_err(&ctlr->dev, "Failed to power device: %d\n",
ret);
- mutex_unlock(&master->io_mutex);
+ mutex_unlock(&ctlr->io_mutex);
return;
}
}
if (!was_busy)
- trace_spi_master_busy(master);
+ trace_spi_controller_busy(ctlr);
- if (!was_busy && master->prepare_transfer_hardware) {
- ret = master->prepare_transfer_hardware(master);
+ if (!was_busy && ctlr->prepare_transfer_hardware) {
+ ret = ctlr->prepare_transfer_hardware(ctlr);
if (ret) {
- dev_err(&master->dev,
+ dev_err(&ctlr->dev,
"failed to prepare transfer hardware\n");
- if (master->auto_runtime_pm)
- pm_runtime_put(master->dev.parent);
- mutex_unlock(&master->io_mutex);
+ if (ctlr->auto_runtime_pm)
+ pm_runtime_put(ctlr->dev.parent);
+ mutex_unlock(&ctlr->io_mutex);
return;
}
}
- trace_spi_message_start(master->cur_msg);
+ trace_spi_message_start(ctlr->cur_msg);
- if (master->prepare_message) {
- ret = master->prepare_message(master, master->cur_msg);
+ if (ctlr->prepare_message) {
+ ret = ctlr->prepare_message(ctlr, ctlr->cur_msg);
if (ret) {
- dev_err(&master->dev,
- "failed to prepare message: %d\n", ret);
- master->cur_msg->status = ret;
- spi_finalize_current_message(master);
+ dev_err(&ctlr->dev, "failed to prepare message: %d\n",
+ ret);
+ ctlr->cur_msg->status = ret;
+ spi_finalize_current_message(ctlr);
goto out;
}
- master->cur_msg_prepared = true;
+ ctlr->cur_msg_prepared = true;
}
- ret = spi_map_msg(master, master->cur_msg);
+ ret = spi_map_msg(ctlr, ctlr->cur_msg);
if (ret) {
- master->cur_msg->status = ret;
- spi_finalize_current_message(master);
+ ctlr->cur_msg->status = ret;
+ spi_finalize_current_message(ctlr);
goto out;
}
- ret = master->transfer_one_message(master, master->cur_msg);
+ ret = ctlr->transfer_one_message(ctlr, ctlr->cur_msg);
if (ret) {
- dev_err(&master->dev,
+ dev_err(&ctlr->dev,
"failed to transfer one message from queue\n");
goto out;
}
out:
- mutex_unlock(&master->io_mutex);
+ mutex_unlock(&ctlr->io_mutex);
/* Prod the scheduler in case transfer_one() was busy waiting */
if (!ret)
/**
* spi_pump_messages - kthread work function which processes spi message queue
- * @work: pointer to kthread work struct contained in the master struct
+ * @work: pointer to kthread work struct contained in the controller struct
*/
static void spi_pump_messages(struct kthread_work *work)
{
- struct spi_master *master =
- container_of(work, struct spi_master, pump_messages);
+ struct spi_controller *ctlr =
+ container_of(work, struct spi_controller, pump_messages);
- __spi_pump_messages(master, true);
+ __spi_pump_messages(ctlr, true);
}
-static int spi_init_queue(struct spi_master *master)
+static int spi_init_queue(struct spi_controller *ctlr)
{
struct sched_param param = { .sched_priority = MAX_RT_PRIO - 1 };
- master->running = false;
- master->busy = false;
+ ctlr->running = false;
+ ctlr->busy = false;
- kthread_init_worker(&master->kworker);
- master->kworker_task = kthread_run(kthread_worker_fn,
- &master->kworker, "%s",
- dev_name(&master->dev));
- if (IS_ERR(master->kworker_task)) {
- dev_err(&master->dev, "failed to create message pump task\n");
- return PTR_ERR(master->kworker_task);
+ kthread_init_worker(&ctlr->kworker);
+ ctlr->kworker_task = kthread_run(kthread_worker_fn, &ctlr->kworker,
+ "%s", dev_name(&ctlr->dev));
+ if (IS_ERR(ctlr->kworker_task)) {
+ dev_err(&ctlr->dev, "failed to create message pump task\n");
+ return PTR_ERR(ctlr->kworker_task);
}
- kthread_init_work(&master->pump_messages, spi_pump_messages);
+ kthread_init_work(&ctlr->pump_messages, spi_pump_messages);
/*
- * Master config will indicate if this controller should run the
+ * Controller config will indicate if this controller should run the
* message pump with high (realtime) priority to reduce the transfer
* latency on the bus by minimising the delay between a transfer
* request and the scheduling of the message pump thread. Without this
* setting the message pump thread will remain at default priority.
*/
- if (master->rt) {
- dev_info(&master->dev,
+ if (ctlr->rt) {
+ dev_info(&ctlr->dev,
"will run message pump with realtime priority\n");
- sched_setscheduler(master->kworker_task, SCHED_FIFO, ¶m);
+ sched_setscheduler(ctlr->kworker_task, SCHED_FIFO, ¶m);
}
return 0;
/**
* spi_get_next_queued_message() - called by driver to check for queued
* messages
- * @master: the master to check for queued messages
+ * @ctlr: the controller to check for queued messages
*
* If there are more messages in the queue, the next message is returned from
* this call.
*
* Return: the next message in the queue, else NULL if the queue is empty.
*/
-struct spi_message *spi_get_next_queued_message(struct spi_master *master)
+struct spi_message *spi_get_next_queued_message(struct spi_controller *ctlr)
{
struct spi_message *next;
unsigned long flags;
/* get a pointer to the next message, if any */
- spin_lock_irqsave(&master->queue_lock, flags);
- next = list_first_entry_or_null(&master->queue, struct spi_message,
+ spin_lock_irqsave(&ctlr->queue_lock, flags);
+ next = list_first_entry_or_null(&ctlr->queue, struct spi_message,
queue);
- spin_unlock_irqrestore(&master->queue_lock, flags);
+ spin_unlock_irqrestore(&ctlr->queue_lock, flags);
return next;
}
/**
* spi_finalize_current_message() - the current message is complete
- * @master: the master to return the message to
+ * @ctlr: the controller to return the message to
*
* Called by the driver to notify the core that the message in the front of the
* queue is complete and can be removed from the queue.
*/
-void spi_finalize_current_message(struct spi_master *master)
+void spi_finalize_current_message(struct spi_controller *ctlr)
{
struct spi_message *mesg;
unsigned long flags;
int ret;
- spin_lock_irqsave(&master->queue_lock, flags);
- mesg = master->cur_msg;
- spin_unlock_irqrestore(&master->queue_lock, flags);
+ spin_lock_irqsave(&ctlr->queue_lock, flags);
+ mesg = ctlr->cur_msg;
+ spin_unlock_irqrestore(&ctlr->queue_lock, flags);
- spi_unmap_msg(master, mesg);
+ spi_unmap_msg(ctlr, mesg);
- if (master->cur_msg_prepared && master->unprepare_message) {
- ret = master->unprepare_message(master, mesg);
+ if (ctlr->cur_msg_prepared && ctlr->unprepare_message) {
+ ret = ctlr->unprepare_message(ctlr, mesg);
if (ret) {
- dev_err(&master->dev,
- "failed to unprepare message: %d\n", ret);
+ dev_err(&ctlr->dev, "failed to unprepare message: %d\n",
+ ret);
}
}
- spin_lock_irqsave(&master->queue_lock, flags);
- master->cur_msg = NULL;
- master->cur_msg_prepared = false;
- kthread_queue_work(&master->kworker, &master->pump_messages);
- spin_unlock_irqrestore(&master->queue_lock, flags);
+ spin_lock_irqsave(&ctlr->queue_lock, flags);
+ ctlr->cur_msg = NULL;
+ ctlr->cur_msg_prepared = false;
+ kthread_queue_work(&ctlr->kworker, &ctlr->pump_messages);
+ spin_unlock_irqrestore(&ctlr->queue_lock, flags);
trace_spi_message_done(mesg);
}
EXPORT_SYMBOL_GPL(spi_finalize_current_message);
-static int spi_start_queue(struct spi_master *master)
+static int spi_start_queue(struct spi_controller *ctlr)
{
unsigned long flags;
- spin_lock_irqsave(&master->queue_lock, flags);
+ spin_lock_irqsave(&ctlr->queue_lock, flags);
- if (master->running || master->busy) {
- spin_unlock_irqrestore(&master->queue_lock, flags);
+ if (ctlr->running || ctlr->busy) {
+ spin_unlock_irqrestore(&ctlr->queue_lock, flags);
return -EBUSY;
}
- master->running = true;
- master->cur_msg = NULL;
- spin_unlock_irqrestore(&master->queue_lock, flags);
+ ctlr->running = true;
+ ctlr->cur_msg = NULL;
+ spin_unlock_irqrestore(&ctlr->queue_lock, flags);
- kthread_queue_work(&master->kworker, &master->pump_messages);
+ kthread_queue_work(&ctlr->kworker, &ctlr->pump_messages);
return 0;
}
-static int spi_stop_queue(struct spi_master *master)
+static int spi_stop_queue(struct spi_controller *ctlr)
{
unsigned long flags;
unsigned limit = 500;
int ret = 0;
- spin_lock_irqsave(&master->queue_lock, flags);
+ spin_lock_irqsave(&ctlr->queue_lock, flags);
/*
* This is a bit lame, but is optimized for the common execution path.
- * A wait_queue on the master->busy could be used, but then the common
+ * A wait_queue on the ctlr->busy could be used, but then the common
* execution path (pump_messages) would be required to call wake_up or
* friends on every SPI message. Do this instead.
*/
- while ((!list_empty(&master->queue) || master->busy) && limit--) {
- spin_unlock_irqrestore(&master->queue_lock, flags);
+ while ((!list_empty(&ctlr->queue) || ctlr->busy) && limit--) {
+ spin_unlock_irqrestore(&ctlr->queue_lock, flags);
usleep_range(10000, 11000);
- spin_lock_irqsave(&master->queue_lock, flags);
+ spin_lock_irqsave(&ctlr->queue_lock, flags);
}
- if (!list_empty(&master->queue) || master->busy)
+ if (!list_empty(&ctlr->queue) || ctlr->busy)
ret = -EBUSY;
else
- master->running = false;
+ ctlr->running = false;
- spin_unlock_irqrestore(&master->queue_lock, flags);
+ spin_unlock_irqrestore(&ctlr->queue_lock, flags);
if (ret) {
- dev_warn(&master->dev,
- "could not stop message queue\n");
+ dev_warn(&ctlr->dev, "could not stop message queue\n");
return ret;
}
return ret;
}
-static int spi_destroy_queue(struct spi_master *master)
+static int spi_destroy_queue(struct spi_controller *ctlr)
{
int ret;
- ret = spi_stop_queue(master);
+ ret = spi_stop_queue(ctlr);
/*
* kthread_flush_worker will block until all work is done.
* return anyway.
*/
if (ret) {
- dev_err(&master->dev, "problem destroying queue\n");
+ dev_err(&ctlr->dev, "problem destroying queue\n");
return ret;
}
- kthread_flush_worker(&master->kworker);
- kthread_stop(master->kworker_task);
+ kthread_flush_worker(&ctlr->kworker);
+ kthread_stop(ctlr->kworker_task);
return 0;
}
struct spi_message *msg,
bool need_pump)
{
- struct spi_master *master = spi->master;
+ struct spi_controller *ctlr = spi->controller;
unsigned long flags;
- spin_lock_irqsave(&master->queue_lock, flags);
+ spin_lock_irqsave(&ctlr->queue_lock, flags);
- if (!master->running) {
- spin_unlock_irqrestore(&master->queue_lock, flags);
+ if (!ctlr->running) {
+ spin_unlock_irqrestore(&ctlr->queue_lock, flags);
return -ESHUTDOWN;
}
msg->actual_length = 0;
msg->status = -EINPROGRESS;
- list_add_tail(&msg->queue, &master->queue);
- if (!master->busy && need_pump)
- kthread_queue_work(&master->kworker, &master->pump_messages);
+ list_add_tail(&msg->queue, &ctlr->queue);
+ if (!ctlr->busy && need_pump)
+ kthread_queue_work(&ctlr->kworker, &ctlr->pump_messages);
- spin_unlock_irqrestore(&master->queue_lock, flags);
+ spin_unlock_irqrestore(&ctlr->queue_lock, flags);
return 0;
}
return __spi_queued_transfer(spi, msg, true);
}
-static int spi_master_initialize_queue(struct spi_master *master)
+static int spi_controller_initialize_queue(struct spi_controller *ctlr)
{
int ret;
- master->transfer = spi_queued_transfer;
- if (!master->transfer_one_message)
- master->transfer_one_message = spi_transfer_one_message;
+ ctlr->transfer = spi_queued_transfer;
+ if (!ctlr->transfer_one_message)
+ ctlr->transfer_one_message = spi_transfer_one_message;
/* Initialize and start queue */
- ret = spi_init_queue(master);
+ ret = spi_init_queue(ctlr);
if (ret) {
- dev_err(&master->dev, "problem initializing queue\n");
+ dev_err(&ctlr->dev, "problem initializing queue\n");
goto err_init_queue;
}
- master->queued = true;
- ret = spi_start_queue(master);
+ ctlr->queued = true;
+ ret = spi_start_queue(ctlr);
if (ret) {
- dev_err(&master->dev, "problem starting queue\n");
+ dev_err(&ctlr->dev, "problem starting queue\n");
goto err_start_queue;
}
return 0;
err_start_queue:
- spi_destroy_queue(master);
+ spi_destroy_queue(ctlr);
err_init_queue:
return ret;
}
/*-------------------------------------------------------------------------*/
#if defined(CONFIG_OF)
-static int of_spi_parse_dt(struct spi_master *master, struct spi_device *spi,
+static int of_spi_parse_dt(struct spi_controller *ctlr, struct spi_device *spi,
struct device_node *nc)
{
u32 value;
int rc;
- /* Device address */
- rc = of_property_read_u32(nc, "reg", &value);
- if (rc) {
- dev_err(&master->dev, "%s has no valid 'reg' property (%d)\n",
- nc->full_name, rc);
- return rc;
- }
- spi->chip_select = value;
-
/* Mode (clock phase/polarity/etc.) */
if (of_find_property(nc, "spi-cpha", NULL))
spi->mode |= SPI_CPHA;
spi->mode |= SPI_TX_QUAD;
break;
default:
- dev_warn(&master->dev,
+ dev_warn(&ctlr->dev,
"spi-tx-bus-width %d not supported\n",
value);
break;
spi->mode |= SPI_RX_QUAD;
break;
default:
- dev_warn(&master->dev,
+ dev_warn(&ctlr->dev,
"spi-rx-bus-width %d not supported\n",
value);
break;
}
}
+ if (spi_controller_is_slave(ctlr)) {
+ if (strcmp(nc->name, "slave")) {
+ dev_err(&ctlr->dev, "%s is not called 'slave'\n",
+ nc->full_name);
+ return -EINVAL;
+ }
+ return 0;
+ }
+
+ /* Device address */
+ rc = of_property_read_u32(nc, "reg", &value);
+ if (rc) {
+ dev_err(&ctlr->dev, "%s has no valid 'reg' property (%d)\n",
+ nc->full_name, rc);
+ return rc;
+ }
+ spi->chip_select = value;
+
/* Device speed */
rc = of_property_read_u32(nc, "spi-max-frequency", &value);
if (rc) {
- dev_err(&master->dev, "%s has no valid 'spi-max-frequency' property (%d)\n",
+ dev_err(&ctlr->dev,
+ "%s has no valid 'spi-max-frequency' property (%d)\n",
nc->full_name, rc);
return rc;
}
}
static struct spi_device *
-of_register_spi_device(struct spi_master *master, struct device_node *nc)
+of_register_spi_device(struct spi_controller *ctlr, struct device_node *nc)
{
struct spi_device *spi;
int rc;
/* Alloc an spi_device */
- spi = spi_alloc_device(master);
+ spi = spi_alloc_device(ctlr);
if (!spi) {
- dev_err(&master->dev, "spi_device alloc error for %s\n",
+ dev_err(&ctlr->dev, "spi_device alloc error for %s\n",
nc->full_name);
rc = -ENOMEM;
goto err_out;
rc = of_modalias_node(nc, spi->modalias,
sizeof(spi->modalias));
if (rc < 0) {
- dev_err(&master->dev, "cannot find modalias for %s\n",
+ dev_err(&ctlr->dev, "cannot find modalias for %s\n",
nc->full_name);
goto err_out;
}
- rc = of_spi_parse_dt(master, spi, nc);
+ rc = of_spi_parse_dt(ctlr, spi, nc);
if (rc)
goto err_out;
/* Register the new device */
rc = spi_add_device(spi);
if (rc) {
- dev_err(&master->dev, "spi_device register error %s\n",
+ dev_err(&ctlr->dev, "spi_device register error %s\n",
nc->full_name);
goto err_of_node_put;
}
/**
* of_register_spi_devices() - Register child devices onto the SPI bus
- * @master: Pointer to spi_master device
+ * @ctlr: Pointer to spi_controller device
*
- * Registers an spi_device for each child node of master node which has a 'reg'
- * property.
+ * Registers an spi_device for each child node of controller node which
+ * represents a valid SPI slave.
*/
-static void of_register_spi_devices(struct spi_master *master)
+static void of_register_spi_devices(struct spi_controller *ctlr)
{
struct spi_device *spi;
struct device_node *nc;
- if (!master->dev.of_node)
+ if (!ctlr->dev.of_node)
return;
- for_each_available_child_of_node(master->dev.of_node, nc) {
+ for_each_available_child_of_node(ctlr->dev.of_node, nc) {
if (of_node_test_and_set_flag(nc, OF_POPULATED))
continue;
- spi = of_register_spi_device(master, nc);
+ spi = of_register_spi_device(ctlr, nc);
if (IS_ERR(spi)) {
- dev_warn(&master->dev, "Failed to create SPI device for %s\n",
- nc->full_name);
+ dev_warn(&ctlr->dev,
+ "Failed to create SPI device for %s\n",
+ nc->full_name);
of_node_clear_flag(nc, OF_POPULATED);
}
}
}
#else
-static void of_register_spi_devices(struct spi_master *master) { }
+static void of_register_spi_devices(struct spi_controller *ctlr) { }
#endif
#ifdef CONFIG_ACPI
static int acpi_spi_add_resource(struct acpi_resource *ares, void *data)
{
struct spi_device *spi = data;
- struct spi_master *master = spi->master;
+ struct spi_controller *ctlr = spi->controller;
if (ares->type == ACPI_RESOURCE_TYPE_SERIAL_BUS) {
struct acpi_resource_spi_serialbus *sb;
* 0 .. max - 1 so we need to ask the driver to
* translate between the two schemes.
*/
- if (master->fw_translate_cs) {
- int cs = master->fw_translate_cs(master,
+ if (ctlr->fw_translate_cs) {
+ int cs = ctlr->fw_translate_cs(ctlr,
sb->device_selection);
if (cs < 0)
return cs;
return 1;
}
-static acpi_status acpi_register_spi_device(struct spi_master *master,
+static acpi_status acpi_register_spi_device(struct spi_controller *ctlr,
struct acpi_device *adev)
{
struct list_head resource_list;
acpi_device_enumerated(adev))
return AE_OK;
- spi = spi_alloc_device(master);
+ spi = spi_alloc_device(ctlr);
if (!spi) {
- dev_err(&master->dev, "failed to allocate SPI device for %s\n",
+ dev_err(&ctlr->dev, "failed to allocate SPI device for %s\n",
dev_name(&adev->dev));
return AE_NO_MEMORY;
}
adev->power.flags.ignore_parent = true;
if (spi_add_device(spi)) {
adev->power.flags.ignore_parent = false;
- dev_err(&master->dev, "failed to add SPI device %s from ACPI\n",
+ dev_err(&ctlr->dev, "failed to add SPI device %s from ACPI\n",
dev_name(&adev->dev));
spi_dev_put(spi);
}
static acpi_status acpi_spi_add_device(acpi_handle handle, u32 level,
void *data, void **return_value)
{
- struct spi_master *master = data;
+ struct spi_controller *ctlr = data;
struct acpi_device *adev;
if (acpi_bus_get_device(handle, &adev))
return AE_OK;
- return acpi_register_spi_device(master, adev);
+ return acpi_register_spi_device(ctlr, adev);
}
-static void acpi_register_spi_devices(struct spi_master *master)
+static void acpi_register_spi_devices(struct spi_controller *ctlr)
{
acpi_status status;
acpi_handle handle;
- handle = ACPI_HANDLE(master->dev.parent);
+ handle = ACPI_HANDLE(ctlr->dev.parent);
if (!handle)
return;
status = acpi_walk_namespace(ACPI_TYPE_DEVICE, handle, 1,
- acpi_spi_add_device, NULL,
- master, NULL);
+ acpi_spi_add_device, NULL, ctlr, NULL);
if (ACPI_FAILURE(status))
- dev_warn(&master->dev, "failed to enumerate SPI slaves\n");
+ dev_warn(&ctlr->dev, "failed to enumerate SPI slaves\n");
}
#else
-static inline void acpi_register_spi_devices(struct spi_master *master) {}
+static inline void acpi_register_spi_devices(struct spi_controller *ctlr) {}
#endif /* CONFIG_ACPI */
-static void spi_master_release(struct device *dev)
+static void spi_controller_release(struct device *dev)
{
- struct spi_master *master;
+ struct spi_controller *ctlr;
- master = container_of(dev, struct spi_master, dev);
- kfree(master);
+ ctlr = container_of(dev, struct spi_controller, dev);
+ kfree(ctlr);
}
static struct class spi_master_class = {
.name = "spi_master",
.owner = THIS_MODULE,
- .dev_release = spi_master_release,
+ .dev_release = spi_controller_release,
.dev_groups = spi_master_groups,
};
+#ifdef CONFIG_SPI_SLAVE
+/**
+ * spi_slave_abort - abort the ongoing transfer request on an SPI slave
+ * controller
+ * @spi: device used for the current transfer
+ */
+int spi_slave_abort(struct spi_device *spi)
+{
+ struct spi_controller *ctlr = spi->controller;
+
+ if (spi_controller_is_slave(ctlr) && ctlr->slave_abort)
+ return ctlr->slave_abort(ctlr);
+
+ return -ENOTSUPP;
+}
+EXPORT_SYMBOL_GPL(spi_slave_abort);
+
+static int match_true(struct device *dev, void *data)
+{
+ return 1;
+}
+
+static ssize_t spi_slave_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct spi_controller *ctlr = container_of(dev, struct spi_controller,
+ dev);
+ struct device *child;
+
+ child = device_find_child(&ctlr->dev, NULL, match_true);
+ return sprintf(buf, "%s\n",
+ child ? to_spi_device(child)->modalias : NULL);
+}
+
+static ssize_t spi_slave_store(struct device *dev,
+ struct device_attribute *attr, const char *buf,
+ size_t count)
+{
+ struct spi_controller *ctlr = container_of(dev, struct spi_controller,
+ dev);
+ struct spi_device *spi;
+ struct device *child;
+ char name[32];
+ int rc;
+
+ rc = sscanf(buf, "%31s", name);
+ if (rc != 1 || !name[0])
+ return -EINVAL;
+
+ child = device_find_child(&ctlr->dev, NULL, match_true);
+ if (child) {
+ /* Remove registered slave */
+ device_unregister(child);
+ put_device(child);
+ }
+
+ if (strcmp(name, "(null)")) {
+ /* Register new slave */
+ spi = spi_alloc_device(ctlr);
+ if (!spi)
+ return -ENOMEM;
+
+ strlcpy(spi->modalias, name, sizeof(spi->modalias));
+
+ rc = spi_add_device(spi);
+ if (rc) {
+ spi_dev_put(spi);
+ return rc;
+ }
+ }
+
+ return count;
+}
+
+static DEVICE_ATTR(slave, 0644, spi_slave_show, spi_slave_store);
+
+static struct attribute *spi_slave_attrs[] = {
+ &dev_attr_slave.attr,
+ NULL,
+};
+
+static const struct attribute_group spi_slave_group = {
+ .attrs = spi_slave_attrs,
+};
+
+static const struct attribute_group *spi_slave_groups[] = {
+ &spi_controller_statistics_group,
+ &spi_slave_group,
+ NULL,
+};
+
+static struct class spi_slave_class = {
+ .name = "spi_slave",
+ .owner = THIS_MODULE,
+ .dev_release = spi_controller_release,
+ .dev_groups = spi_slave_groups,
+};
+#else
+extern struct class spi_slave_class; /* dummy */
+#endif
/**
- * spi_alloc_master - allocate SPI master controller
+ * __spi_alloc_controller - allocate an SPI master or slave controller
* @dev: the controller, possibly using the platform_bus
* @size: how much zeroed driver-private data to allocate; the pointer to this
* memory is in the driver_data field of the returned device,
- * accessible with spi_master_get_devdata().
+ * accessible with spi_controller_get_devdata().
+ * @slave: flag indicating whether to allocate an SPI master (false) or SPI
+ * slave (true) controller
* Context: can sleep
*
- * This call is used only by SPI master controller drivers, which are the
+ * This call is used only by SPI controller drivers, which are the
* only ones directly touching chip registers. It's how they allocate
- * an spi_master structure, prior to calling spi_register_master().
+ * an spi_controller structure, prior to calling spi_register_controller().
*
* This must be called from context that can sleep.
*
- * The caller is responsible for assigning the bus number and initializing
- * the master's methods before calling spi_register_master(); and (after errors
- * adding the device) calling spi_master_put() to prevent a memory leak.
+ * The caller is responsible for assigning the bus number and initializing the
+ * controller's methods before calling spi_register_controller(); and (after
+ * errors adding the device) calling spi_controller_put() to prevent a memory
+ * leak.
*
- * Return: the SPI master structure on success, else NULL.
+ * Return: the SPI controller structure on success, else NULL.
*/
-struct spi_master *spi_alloc_master(struct device *dev, unsigned size)
+struct spi_controller *__spi_alloc_controller(struct device *dev,
+ unsigned int size, bool slave)
{
- struct spi_master *master;
+ struct spi_controller *ctlr;
if (!dev)
return NULL;
- master = kzalloc(size + sizeof(*master), GFP_KERNEL);
- if (!master)
+ ctlr = kzalloc(size + sizeof(*ctlr), GFP_KERNEL);
+ if (!ctlr)
return NULL;
- device_initialize(&master->dev);
- master->bus_num = -1;
- master->num_chipselect = 1;
- master->dev.class = &spi_master_class;
- master->dev.parent = dev;
- pm_suspend_ignore_children(&master->dev, true);
- spi_master_set_devdata(master, &master[1]);
+ device_initialize(&ctlr->dev);
+ ctlr->bus_num = -1;
+ ctlr->num_chipselect = 1;
+ ctlr->slave = slave;
+ if (IS_ENABLED(CONFIG_SPI_SLAVE) && slave)
+ ctlr->dev.class = &spi_slave_class;
+ else
+ ctlr->dev.class = &spi_master_class;
+ ctlr->dev.parent = dev;
+ pm_suspend_ignore_children(&ctlr->dev, true);
+ spi_controller_set_devdata(ctlr, &ctlr[1]);
- return master;
+ return ctlr;
}
-EXPORT_SYMBOL_GPL(spi_alloc_master);
+EXPORT_SYMBOL_GPL(__spi_alloc_controller);
#ifdef CONFIG_OF
-static int of_spi_register_master(struct spi_master *master)
+static int of_spi_register_master(struct spi_controller *ctlr)
{
int nb, i, *cs;
- struct device_node *np = master->dev.of_node;
+ struct device_node *np = ctlr->dev.of_node;
if (!np)
return 0;
nb = of_gpio_named_count(np, "cs-gpios");
- master->num_chipselect = max_t(int, nb, master->num_chipselect);
+ ctlr->num_chipselect = max_t(int, nb, ctlr->num_chipselect);
/* Return error only for an incorrectly formed cs-gpios property */
if (nb == 0 || nb == -ENOENT)
else if (nb < 0)
return nb;
- cs = devm_kzalloc(&master->dev,
- sizeof(int) * master->num_chipselect,
+ cs = devm_kzalloc(&ctlr->dev, sizeof(int) * ctlr->num_chipselect,
GFP_KERNEL);
- master->cs_gpios = cs;
+ ctlr->cs_gpios = cs;
- if (!master->cs_gpios)
+ if (!ctlr->cs_gpios)
return -ENOMEM;
- for (i = 0; i < master->num_chipselect; i++)
+ for (i = 0; i < ctlr->num_chipselect; i++)
cs[i] = -ENOENT;
for (i = 0; i < nb; i++)
return 0;
}
#else
-static int of_spi_register_master(struct spi_master *master)
+static int of_spi_register_master(struct spi_controller *ctlr)
{
return 0;
}
#endif
/**
- * spi_register_master - register SPI master controller
- * @master: initialized master, originally from spi_alloc_master()
+ * spi_register_controller - register SPI master or slave controller
+ * @ctlr: initialized master, originally from spi_alloc_master() or
+ * spi_alloc_slave()
* Context: can sleep
*
- * SPI master controllers connect to their drivers using some non-SPI bus,
+ * SPI controllers connect to their drivers using some non-SPI bus,
* such as the platform bus. The final stage of probe() in that code
- * includes calling spi_register_master() to hook up to this SPI bus glue.
+ * includes calling spi_register_controller() to hook up to this SPI bus glue.
*
* SPI controllers use board specific (often SOC specific) bus numbers,
* and board-specific addressing for SPI devices combines those numbers
* chip is at which address.
*
* This must be called from context that can sleep. It returns zero on
- * success, else a negative error code (dropping the master's refcount).
+ * success, else a negative error code (dropping the controller's refcount).
* After a successful return, the caller is responsible for calling
- * spi_unregister_master().
+ * spi_unregister_controller().
*
* Return: zero on success, else a negative error code.
*/
-int spi_register_master(struct spi_master *master)
+int spi_register_controller(struct spi_controller *ctlr)
{
static atomic_t dyn_bus_id = ATOMIC_INIT((1<<15) - 1);
- struct device *dev = master->dev.parent;
+ struct device *dev = ctlr->dev.parent;
struct boardinfo *bi;
int status = -ENODEV;
int dynamic = 0;
if (!dev)
return -ENODEV;
- status = of_spi_register_master(master);
- if (status)
- return status;
+ if (!spi_controller_is_slave(ctlr)) {
+ status = of_spi_register_master(ctlr);
+ if (status)
+ return status;
+ }
/* even if it's just one always-selected device, there must
* be at least one chipselect
*/
- if (master->num_chipselect == 0)
+ if (ctlr->num_chipselect == 0)
return -EINVAL;
- if ((master->bus_num < 0) && master->dev.of_node)
- master->bus_num = of_alias_get_id(master->dev.of_node, "spi");
+ if ((ctlr->bus_num < 0) && ctlr->dev.of_node)
+ ctlr->bus_num = of_alias_get_id(ctlr->dev.of_node, "spi");
/* convention: dynamically assigned bus IDs count down from the max */
- if (master->bus_num < 0) {
+ if (ctlr->bus_num < 0) {
/* FIXME switch to an IDR based scheme, something like
* I2C now uses, so we can't run out of "dynamic" IDs
*/
- master->bus_num = atomic_dec_return(&dyn_bus_id);
+ ctlr->bus_num = atomic_dec_return(&dyn_bus_id);
dynamic = 1;
}
- INIT_LIST_HEAD(&master->queue);
- spin_lock_init(&master->queue_lock);
- spin_lock_init(&master->bus_lock_spinlock);
- mutex_init(&master->bus_lock_mutex);
- mutex_init(&master->io_mutex);
- master->bus_lock_flag = 0;
- init_completion(&master->xfer_completion);
- if (!master->max_dma_len)
- master->max_dma_len = INT_MAX;
+ INIT_LIST_HEAD(&ctlr->queue);
+ spin_lock_init(&ctlr->queue_lock);
+ spin_lock_init(&ctlr->bus_lock_spinlock);
+ mutex_init(&ctlr->bus_lock_mutex);
+ mutex_init(&ctlr->io_mutex);
+ ctlr->bus_lock_flag = 0;
+ init_completion(&ctlr->xfer_completion);
+ if (!ctlr->max_dma_len)
+ ctlr->max_dma_len = INT_MAX;
/* register the device, then userspace will see it.
* registration fails if the bus ID is in use.
*/
- dev_set_name(&master->dev, "spi%u", master->bus_num);
- status = device_add(&master->dev);
+ dev_set_name(&ctlr->dev, "spi%u", ctlr->bus_num);
+ status = device_add(&ctlr->dev);
if (status < 0)
goto done;
- dev_dbg(dev, "registered master %s%s\n", dev_name(&master->dev),
- dynamic ? " (dynamic)" : "");
+ dev_dbg(dev, "registered %s %s%s\n",
+ spi_controller_is_slave(ctlr) ? "slave" : "master",
+ dev_name(&ctlr->dev), dynamic ? " (dynamic)" : "");
/* If we're using a queued driver, start the queue */
- if (master->transfer)
- dev_info(dev, "master is unqueued, this is deprecated\n");
+ if (ctlr->transfer)
+ dev_info(dev, "controller is unqueued, this is deprecated\n");
else {
- status = spi_master_initialize_queue(master);
+ status = spi_controller_initialize_queue(ctlr);
if (status) {
- device_del(&master->dev);
+ device_del(&ctlr->dev);
goto done;
}
}
/* add statistics */
- spin_lock_init(&master->statistics.lock);
+ spin_lock_init(&ctlr->statistics.lock);
mutex_lock(&board_lock);
- list_add_tail(&master->list, &spi_master_list);
+ list_add_tail(&ctlr->list, &spi_controller_list);
list_for_each_entry(bi, &board_list, list)
- spi_match_master_to_boardinfo(master, &bi->board_info);
+ spi_match_controller_to_boardinfo(ctlr, &bi->board_info);
mutex_unlock(&board_lock);
/* Register devices from the device tree and ACPI */
- of_register_spi_devices(master);
- acpi_register_spi_devices(master);
+ of_register_spi_devices(ctlr);
+ acpi_register_spi_devices(ctlr);
done:
return status;
}
-EXPORT_SYMBOL_GPL(spi_register_master);
+EXPORT_SYMBOL_GPL(spi_register_controller);
static void devm_spi_unregister(struct device *dev, void *res)
{
- spi_unregister_master(*(struct spi_master **)res);
+ spi_unregister_controller(*(struct spi_controller **)res);
}
/**
- * dev_spi_register_master - register managed SPI master controller
- * @dev: device managing SPI master
- * @master: initialized master, originally from spi_alloc_master()
+ * devm_spi_register_controller - register managed SPI master or slave
+ * controller
+ * @dev: device managing SPI controller
+ * @ctlr: initialized controller, originally from spi_alloc_master() or
+ * spi_alloc_slave()
* Context: can sleep
*
- * Register a SPI device as with spi_register_master() which will
+ * Register a SPI device as with spi_register_controller() which will
* automatically be unregister
*
* Return: zero on success, else a negative error code.
*/
-int devm_spi_register_master(struct device *dev, struct spi_master *master)
+int devm_spi_register_controller(struct device *dev,
+ struct spi_controller *ctlr)
{
- struct spi_master **ptr;
+ struct spi_controller **ptr;
int ret;
ptr = devres_alloc(devm_spi_unregister, sizeof(*ptr), GFP_KERNEL);
if (!ptr)
return -ENOMEM;
- ret = spi_register_master(master);
+ ret = spi_register_controller(ctlr);
if (!ret) {
- *ptr = master;
+ *ptr = ctlr;
devres_add(dev, ptr);
} else {
devres_free(ptr);
return ret;
}
-EXPORT_SYMBOL_GPL(devm_spi_register_master);
+EXPORT_SYMBOL_GPL(devm_spi_register_controller);
static int __unregister(struct device *dev, void *null)
{
}
/**
- * spi_unregister_master - unregister SPI master controller
- * @master: the master being unregistered
+ * spi_unregister_controller - unregister SPI master or slave controller
+ * @ctlr: the controller being unregistered
* Context: can sleep
*
- * This call is used only by SPI master controller drivers, which are the
+ * This call is used only by SPI controller drivers, which are the
* only ones directly touching chip registers.
*
* This must be called from context that can sleep.
*/
-void spi_unregister_master(struct spi_master *master)
+void spi_unregister_controller(struct spi_controller *ctlr)
{
int dummy;
- if (master->queued) {
- if (spi_destroy_queue(master))
- dev_err(&master->dev, "queue remove failed\n");
+ if (ctlr->queued) {
+ if (spi_destroy_queue(ctlr))
+ dev_err(&ctlr->dev, "queue remove failed\n");
}
mutex_lock(&board_lock);
- list_del(&master->list);
+ list_del(&ctlr->list);
mutex_unlock(&board_lock);
- dummy = device_for_each_child(&master->dev, NULL, __unregister);
- device_unregister(&master->dev);
+ dummy = device_for_each_child(&ctlr->dev, NULL, __unregister);
+ device_unregister(&ctlr->dev);
}
-EXPORT_SYMBOL_GPL(spi_unregister_master);
+EXPORT_SYMBOL_GPL(spi_unregister_controller);
-int spi_master_suspend(struct spi_master *master)
+int spi_controller_suspend(struct spi_controller *ctlr)
{
int ret;
- /* Basically no-ops for non-queued masters */
- if (!master->queued)
+ /* Basically no-ops for non-queued controllers */
+ if (!ctlr->queued)
return 0;
- ret = spi_stop_queue(master);
+ ret = spi_stop_queue(ctlr);
if (ret)
- dev_err(&master->dev, "queue stop failed\n");
+ dev_err(&ctlr->dev, "queue stop failed\n");
return ret;
}
-EXPORT_SYMBOL_GPL(spi_master_suspend);
+EXPORT_SYMBOL_GPL(spi_controller_suspend);
-int spi_master_resume(struct spi_master *master)
+int spi_controller_resume(struct spi_controller *ctlr)
{
int ret;
- if (!master->queued)
+ if (!ctlr->queued)
return 0;
- ret = spi_start_queue(master);
+ ret = spi_start_queue(ctlr);
if (ret)
- dev_err(&master->dev, "queue restart failed\n");
+ dev_err(&ctlr->dev, "queue restart failed\n");
return ret;
}
-EXPORT_SYMBOL_GPL(spi_master_resume);
+EXPORT_SYMBOL_GPL(spi_controller_resume);
-static int __spi_master_match(struct device *dev, const void *data)
+static int __spi_controller_match(struct device *dev, const void *data)
{
- struct spi_master *m;
+ struct spi_controller *ctlr;
const u16 *bus_num = data;
- m = container_of(dev, struct spi_master, dev);
- return m->bus_num == *bus_num;
+ ctlr = container_of(dev, struct spi_controller, dev);
+ return ctlr->bus_num == *bus_num;
}
/**
*
* This call may be used with devices that are registered after
* arch init time. It returns a refcounted pointer to the relevant
- * spi_master (which the caller must release), or NULL if there is
+ * spi_controller (which the caller must release), or NULL if there is
* no such master registered.
*
* Return: the SPI master structure on success, else NULL.
*/
-struct spi_master *spi_busnum_to_master(u16 bus_num)
+struct spi_controller *spi_busnum_to_master(u16 bus_num)
{
struct device *dev;
- struct spi_master *master = NULL;
+ struct spi_controller *ctlr = NULL;
dev = class_find_device(&spi_master_class, NULL, &bus_num,
- __spi_master_match);
+ __spi_controller_match);
if (dev)
- master = container_of(dev, struct spi_master, dev);
+ ctlr = container_of(dev, struct spi_controller, dev);
/* reference got in class_find_device */
- return master;
+ return ctlr;
}
EXPORT_SYMBOL_GPL(spi_busnum_to_master);
* Return: the pointer to the allocated data
*
* This may get enhanced in the future to allocate from a memory pool
- * of the @spi_device or @spi_master to avoid repeated allocations.
+ * of the @spi_device or @spi_controller to avoid repeated allocations.
*/
void *spi_res_alloc(struct spi_device *spi,
spi_res_release_t release,
/**
* spi_res_release - release all spi resources for this message
- * @master: the @spi_master
+ * @ctlr: the @spi_controller
* @message: the @spi_message
*/
-void spi_res_release(struct spi_master *master,
- struct spi_message *message)
+void spi_res_release(struct spi_controller *ctlr, struct spi_message *message)
{
struct spi_res *res;
struct spi_res, entry);
if (res->release)
- res->release(master, message, res->data);
+ res->release(ctlr, message, res->data);
list_del(&res->entry);
/* Core methods for spi_message alterations */
-static void __spi_replace_transfers_release(struct spi_master *master,
+static void __spi_replace_transfers_release(struct spi_controller *ctlr,
struct spi_message *msg,
void *res)
{
/* call extra callback if requested */
if (rxfer->release)
- rxfer->release(master, msg, res);
+ rxfer->release(ctlr, msg, res);
/* insert replaced transfers back into the message */
list_splice(&rxfer->replaced_transfers, rxfer->replaced_after);
}
EXPORT_SYMBOL_GPL(spi_replace_transfers);
-static int __spi_split_transfer_maxsize(struct spi_master *master,
+static int __spi_split_transfer_maxsize(struct spi_controller *ctlr,
struct spi_message *msg,
struct spi_transfer **xferp,
size_t maxsize,
*xferp = &xfers[count - 1];
/* increment statistics counters */
- SPI_STATISTICS_INCREMENT_FIELD(&master->statistics,
+ SPI_STATISTICS_INCREMENT_FIELD(&ctlr->statistics,
transfers_split_maxsize);
SPI_STATISTICS_INCREMENT_FIELD(&msg->spi->statistics,
transfers_split_maxsize);
* spi_split_tranfers_maxsize - split spi transfers into multiple transfers
* when an individual transfer exceeds a
* certain size
- * @master: the @spi_master for this transfer
+ * @ctlr: the @spi_controller for this transfer
* @msg: the @spi_message to transform
* @maxsize: the maximum when to apply this
* @gfp: GFP allocation flags
*
* Return: status of transformation
*/
-int spi_split_transfers_maxsize(struct spi_master *master,
+int spi_split_transfers_maxsize(struct spi_controller *ctlr,
struct spi_message *msg,
size_t maxsize,
gfp_t gfp)
*/
list_for_each_entry(xfer, &msg->transfers, transfer_list) {
if (xfer->len > maxsize) {
- ret = __spi_split_transfer_maxsize(
- master, msg, &xfer, maxsize, gfp);
+ ret = __spi_split_transfer_maxsize(ctlr, msg, &xfer,
+ maxsize, gfp);
if (ret)
return ret;
}
/*-------------------------------------------------------------------------*/
-/* Core methods for SPI master protocol drivers. Some of the
+/* Core methods for SPI controller protocol drivers. Some of the
* other core methods are currently defined as inline functions.
*/
-static int __spi_validate_bits_per_word(struct spi_master *master, u8 bits_per_word)
+static int __spi_validate_bits_per_word(struct spi_controller *ctlr,
+ u8 bits_per_word)
{
- if (master->bits_per_word_mask) {
+ if (ctlr->bits_per_word_mask) {
/* Only 32 bits fit in the mask */
if (bits_per_word > 32)
return -EINVAL;
- if (!(master->bits_per_word_mask &
- SPI_BPW_MASK(bits_per_word)))
+ if (!(ctlr->bits_per_word_mask & SPI_BPW_MASK(bits_per_word)))
return -EINVAL;
}
(SPI_TX_DUAL | SPI_TX_QUAD | SPI_RX_DUAL | SPI_RX_QUAD)))
return -EINVAL;
/* help drivers fail *cleanly* when they need options
- * that aren't supported with their current master
+ * that aren't supported with their current controller
*/
- bad_bits = spi->mode & ~spi->master->mode_bits;
+ bad_bits = spi->mode & ~spi->controller->mode_bits;
ugly_bits = bad_bits &
(SPI_TX_DUAL | SPI_TX_QUAD | SPI_RX_DUAL | SPI_RX_QUAD);
if (ugly_bits) {
if (!spi->bits_per_word)
spi->bits_per_word = 8;
- status = __spi_validate_bits_per_word(spi->master, spi->bits_per_word);
+ status = __spi_validate_bits_per_word(spi->controller,
+ spi->bits_per_word);
if (status)
return status;
if (!spi->max_speed_hz)
- spi->max_speed_hz = spi->master->max_speed_hz;
+ spi->max_speed_hz = spi->controller->max_speed_hz;
- if (spi->master->setup)
- status = spi->master->setup(spi);
+ if (spi->controller->setup)
+ status = spi->controller->setup(spi);
spi_set_cs(spi, false);
static int __spi_validate(struct spi_device *spi, struct spi_message *message)
{
- struct spi_master *master = spi->master;
+ struct spi_controller *ctlr = spi->controller;
struct spi_transfer *xfer;
int w_size;
* either MOSI or MISO is missing. They can also be caused by
* software limitations.
*/
- if ((master->flags & SPI_MASTER_HALF_DUPLEX)
- || (spi->mode & SPI_3WIRE)) {
- unsigned flags = master->flags;
+ if ((ctlr->flags & SPI_CONTROLLER_HALF_DUPLEX) ||
+ (spi->mode & SPI_3WIRE)) {
+ unsigned flags = ctlr->flags;
list_for_each_entry(xfer, &message->transfers, transfer_list) {
if (xfer->rx_buf && xfer->tx_buf)
return -EINVAL;
- if ((flags & SPI_MASTER_NO_TX) && xfer->tx_buf)
+ if ((flags & SPI_CONTROLLER_NO_TX) && xfer->tx_buf)
return -EINVAL;
- if ((flags & SPI_MASTER_NO_RX) && xfer->rx_buf)
+ if ((flags & SPI_CONTROLLER_NO_RX) && xfer->rx_buf)
return -EINVAL;
}
}
if (!xfer->speed_hz)
xfer->speed_hz = spi->max_speed_hz;
if (!xfer->speed_hz)
- xfer->speed_hz = master->max_speed_hz;
+ xfer->speed_hz = ctlr->max_speed_hz;
- if (master->max_speed_hz &&
- xfer->speed_hz > master->max_speed_hz)
- xfer->speed_hz = master->max_speed_hz;
+ if (ctlr->max_speed_hz && xfer->speed_hz > ctlr->max_speed_hz)
+ xfer->speed_hz = ctlr->max_speed_hz;
- if (__spi_validate_bits_per_word(master, xfer->bits_per_word))
+ if (__spi_validate_bits_per_word(ctlr, xfer->bits_per_word))
return -EINVAL;
/*
if (xfer->len % w_size)
return -EINVAL;
- if (xfer->speed_hz && master->min_speed_hz &&
- xfer->speed_hz < master->min_speed_hz)
+ if (xfer->speed_hz && ctlr->min_speed_hz &&
+ xfer->speed_hz < ctlr->min_speed_hz)
return -EINVAL;
if (xfer->tx_buf && !xfer->tx_nbits)
static int __spi_async(struct spi_device *spi, struct spi_message *message)
{
- struct spi_master *master = spi->master;
+ struct spi_controller *ctlr = spi->controller;
message->spi = spi;
- SPI_STATISTICS_INCREMENT_FIELD(&master->statistics, spi_async);
+ SPI_STATISTICS_INCREMENT_FIELD(&ctlr->statistics, spi_async);
SPI_STATISTICS_INCREMENT_FIELD(&spi->statistics, spi_async);
trace_spi_message_submit(message);
- return master->transfer(spi, message);
+ return ctlr->transfer(spi, message);
}
/**
*/
int spi_async(struct spi_device *spi, struct spi_message *message)
{
- struct spi_master *master = spi->master;
+ struct spi_controller *ctlr = spi->controller;
int ret;
unsigned long flags;
if (ret != 0)
return ret;
- spin_lock_irqsave(&master->bus_lock_spinlock, flags);
+ spin_lock_irqsave(&ctlr->bus_lock_spinlock, flags);
- if (master->bus_lock_flag)
+ if (ctlr->bus_lock_flag)
ret = -EBUSY;
else
ret = __spi_async(spi, message);
- spin_unlock_irqrestore(&master->bus_lock_spinlock, flags);
+ spin_unlock_irqrestore(&ctlr->bus_lock_spinlock, flags);
return ret;
}
*/
int spi_async_locked(struct spi_device *spi, struct spi_message *message)
{
- struct spi_master *master = spi->master;
+ struct spi_controller *ctlr = spi->controller;
int ret;
unsigned long flags;
if (ret != 0)
return ret;
- spin_lock_irqsave(&master->bus_lock_spinlock, flags);
+ spin_lock_irqsave(&ctlr->bus_lock_spinlock, flags);
ret = __spi_async(spi, message);
- spin_unlock_irqrestore(&master->bus_lock_spinlock, flags);
+ spin_unlock_irqrestore(&ctlr->bus_lock_spinlock, flags);
return ret;
struct spi_flash_read_message *msg)
{
- struct spi_master *master = spi->master;
+ struct spi_controller *master = spi->controller;
struct device *rx_dev = NULL;
int ret;
/*-------------------------------------------------------------------------*/
-/* Utility methods for SPI master protocol drivers, layered on
+/* Utility methods for SPI protocol drivers, layered on
* top of the core. Some other utility methods are defined as
* inline functions.
*/
{
DECLARE_COMPLETION_ONSTACK(done);
int status;
- struct spi_master *master = spi->master;
+ struct spi_controller *ctlr = spi->controller;
unsigned long flags;
status = __spi_validate(spi, message);
message->context = &done;
message->spi = spi;
- SPI_STATISTICS_INCREMENT_FIELD(&master->statistics, spi_sync);
+ SPI_STATISTICS_INCREMENT_FIELD(&ctlr->statistics, spi_sync);
SPI_STATISTICS_INCREMENT_FIELD(&spi->statistics, spi_sync);
/* If we're not using the legacy transfer method then we will
* This code would be less tricky if we could remove the
* support for driver implemented message queues.
*/
- if (master->transfer == spi_queued_transfer) {
- spin_lock_irqsave(&master->bus_lock_spinlock, flags);
+ if (ctlr->transfer == spi_queued_transfer) {
+ spin_lock_irqsave(&ctlr->bus_lock_spinlock, flags);
trace_spi_message_submit(message);
status = __spi_queued_transfer(spi, message, false);
- spin_unlock_irqrestore(&master->bus_lock_spinlock, flags);
+ spin_unlock_irqrestore(&ctlr->bus_lock_spinlock, flags);
} else {
status = spi_async_locked(spi, message);
}
/* Push out the messages in the calling context if we
* can.
*/
- if (master->transfer == spi_queued_transfer) {
- SPI_STATISTICS_INCREMENT_FIELD(&master->statistics,
+ if (ctlr->transfer == spi_queued_transfer) {
+ SPI_STATISTICS_INCREMENT_FIELD(&ctlr->statistics,
spi_sync_immediate);
SPI_STATISTICS_INCREMENT_FIELD(&spi->statistics,
spi_sync_immediate);
- __spi_pump_messages(master, false);
+ __spi_pump_messages(ctlr, false);
}
wait_for_completion(&done);
{
int ret;
- mutex_lock(&spi->master->bus_lock_mutex);
+ mutex_lock(&spi->controller->bus_lock_mutex);
ret = __spi_sync(spi, message);
- mutex_unlock(&spi->master->bus_lock_mutex);
+ mutex_unlock(&spi->controller->bus_lock_mutex);
return ret;
}
/**
* spi_bus_lock - obtain a lock for exclusive SPI bus usage
- * @master: SPI bus master that should be locked for exclusive bus access
+ * @ctlr: SPI bus master that should be locked for exclusive bus access
* Context: can sleep
*
* This call may only be used from a context that may sleep. The sleep
*
* Return: always zero.
*/
-int spi_bus_lock(struct spi_master *master)
+int spi_bus_lock(struct spi_controller *ctlr)
{
unsigned long flags;
- mutex_lock(&master->bus_lock_mutex);
+ mutex_lock(&ctlr->bus_lock_mutex);
- spin_lock_irqsave(&master->bus_lock_spinlock, flags);
- master->bus_lock_flag = 1;
- spin_unlock_irqrestore(&master->bus_lock_spinlock, flags);
+ spin_lock_irqsave(&ctlr->bus_lock_spinlock, flags);
+ ctlr->bus_lock_flag = 1;
+ spin_unlock_irqrestore(&ctlr->bus_lock_spinlock, flags);
/* mutex remains locked until spi_bus_unlock is called */
/**
* spi_bus_unlock - release the lock for exclusive SPI bus usage
- * @master: SPI bus master that was locked for exclusive bus access
+ * @ctlr: SPI bus master that was locked for exclusive bus access
* Context: can sleep
*
* This call may only be used from a context that may sleep. The sleep
*
* Return: always zero.
*/
-int spi_bus_unlock(struct spi_master *master)
+int spi_bus_unlock(struct spi_controller *ctlr)
{
- master->bus_lock_flag = 0;
+ ctlr->bus_lock_flag = 0;
- mutex_unlock(&master->bus_lock_mutex);
+ mutex_unlock(&ctlr->bus_lock_mutex);
return 0;
}
return dev ? to_spi_device(dev) : NULL;
}
-static int __spi_of_master_match(struct device *dev, const void *data)
+static int __spi_of_controller_match(struct device *dev, const void *data)
{
return dev->of_node == data;
}
-/* the spi masters are not using spi_bus, so we find it with another way */
-static struct spi_master *of_find_spi_master_by_node(struct device_node *node)
+/* the spi controllers are not using spi_bus, so we find it with another way */
+static struct spi_controller *of_find_spi_controller_by_node(struct device_node *node)
{
struct device *dev;
dev = class_find_device(&spi_master_class, NULL, node,
- __spi_of_master_match);
+ __spi_of_controller_match);
+ if (!dev && IS_ENABLED(CONFIG_SPI_SLAVE))
+ dev = class_find_device(&spi_slave_class, NULL, node,
+ __spi_of_controller_match);
if (!dev)
return NULL;
/* reference got in class_find_device */
- return container_of(dev, struct spi_master, dev);
+ return container_of(dev, struct spi_controller, dev);
}
static int of_spi_notify(struct notifier_block *nb, unsigned long action,
void *arg)
{
struct of_reconfig_data *rd = arg;
- struct spi_master *master;
+ struct spi_controller *ctlr;
struct spi_device *spi;
switch (of_reconfig_get_state_change(action, arg)) {
case OF_RECONFIG_CHANGE_ADD:
- master = of_find_spi_master_by_node(rd->dn->parent);
- if (master == NULL)
+ ctlr = of_find_spi_controller_by_node(rd->dn->parent);
+ if (ctlr == NULL)
return NOTIFY_OK; /* not for us */
if (of_node_test_and_set_flag(rd->dn, OF_POPULATED)) {
- put_device(&master->dev);
+ put_device(&ctlr->dev);
return NOTIFY_OK;
}
- spi = of_register_spi_device(master, rd->dn);
- put_device(&master->dev);
+ spi = of_register_spi_device(ctlr, rd->dn);
+ put_device(&ctlr->dev);
if (IS_ERR(spi)) {
pr_err("%s: failed to create for '%s'\n",
#endif /* IS_ENABLED(CONFIG_OF_DYNAMIC) */
#if IS_ENABLED(CONFIG_ACPI)
-static int spi_acpi_master_match(struct device *dev, const void *data)
+static int spi_acpi_controller_match(struct device *dev, const void *data)
{
return ACPI_COMPANION(dev->parent) == data;
}
return ACPI_COMPANION(dev) == data;
}
-static struct spi_master *acpi_spi_find_master_by_adev(struct acpi_device *adev)
+static struct spi_controller *acpi_spi_find_controller_by_adev(struct acpi_device *adev)
{
struct device *dev;
dev = class_find_device(&spi_master_class, NULL, adev,
- spi_acpi_master_match);
+ spi_acpi_controller_match);
+ if (!dev && IS_ENABLED(CONFIG_SPI_SLAVE))
+ dev = class_find_device(&spi_slave_class, NULL, adev,
+ spi_acpi_controller_match);
if (!dev)
return NULL;
- return container_of(dev, struct spi_master, dev);
+ return container_of(dev, struct spi_controller, dev);
}
static struct spi_device *acpi_spi_find_device_by_adev(struct acpi_device *adev)
void *arg)
{
struct acpi_device *adev = arg;
- struct spi_master *master;
+ struct spi_controller *ctlr;
struct spi_device *spi;
switch (value) {
case ACPI_RECONFIG_DEVICE_ADD:
- master = acpi_spi_find_master_by_adev(adev->parent);
- if (!master)
+ ctlr = acpi_spi_find_controller_by_adev(adev->parent);
+ if (!ctlr)
break;
- acpi_register_spi_device(master, adev);
- put_device(&master->dev);
+ acpi_register_spi_device(ctlr, adev);
+ put_device(&ctlr->dev);
break;
case ACPI_RECONFIG_DEVICE_REMOVE:
if (!acpi_device_enumerated(adev))
if (status < 0)
goto err2;
+ if (IS_ENABLED(CONFIG_SPI_SLAVE)) {
+ status = class_register(&spi_slave_class);
+ if (status < 0)
+ goto err3;
+ }
+
if (IS_ENABLED(CONFIG_OF_DYNAMIC))
WARN_ON(of_reconfig_notifier_register(&spi_of_notifier));
if (IS_ENABLED(CONFIG_ACPI))
return 0;
+err3:
+ class_unregister(&spi_master_class);
err2:
bus_unregister(&spi_bus_type);
err1:
static ssize_t
spidev_sync(struct spidev_data *spidev, struct spi_message *message)
{
- DECLARE_COMPLETION_ONSTACK(done);
int status;
struct spi_device *spi;
spidev_get_ioc_message(unsigned int cmd, struct spi_ioc_transfer __user *u_ioc,
unsigned *n_ioc)
{
- struct spi_ioc_transfer *ioc;
u32 tmp;
/* Check type, command number and direction */
return NULL;
/* copy into scratch area */
- ioc = kmalloc(tmp, GFP_KERNEL);
- if (!ioc)
- return ERR_PTR(-ENOMEM);
- if (__copy_from_user(ioc, u_ioc, tmp)) {
- kfree(ioc);
- return ERR_PTR(-EFAULT);
- }
- return ioc;
+ return memdup_user(u_ioc, tmp);
}
static long
/* PMIC Arbiter configuration registers */
#define PMIC_ARB_VERSION 0x0000
#define PMIC_ARB_VERSION_V2_MIN 0x20010000
+#define PMIC_ARB_VERSION_V3_MIN 0x30000000
#define PMIC_ARB_INT_EN 0x0004
/* PMIC Arbiter channel registers offsets */
/* Channel Status fields */
enum pmic_arb_chnl_status {
- PMIC_ARB_STATUS_DONE = (1 << 0),
- PMIC_ARB_STATUS_FAILURE = (1 << 1),
- PMIC_ARB_STATUS_DENIED = (1 << 2),
- PMIC_ARB_STATUS_DROPPED = (1 << 3),
+ PMIC_ARB_STATUS_DONE = BIT(0),
+ PMIC_ARB_STATUS_FAILURE = BIT(1),
+ PMIC_ARB_STATUS_DENIED = BIT(2),
+ PMIC_ARB_STATUS_DROPPED = BIT(3),
};
/* Command register fields */
/* interrupt enable bit */
#define SPMI_PIC_ACC_ENABLE_BIT BIT(0)
+#define HWIRQ(slave_id, periph_id, irq_id, apid) \
+ ((((slave_id) & 0xF) << 28) | \
+ (((periph_id) & 0xFF) << 20) | \
+ (((irq_id) & 0x7) << 16) | \
+ (((apid) & 0x1FF) << 0))
+
+#define HWIRQ_SID(hwirq) (((hwirq) >> 28) & 0xF)
+#define HWIRQ_PER(hwirq) (((hwirq) >> 20) & 0xFF)
+#define HWIRQ_IRQ(hwirq) (((hwirq) >> 16) & 0x7)
+#define HWIRQ_APID(hwirq) (((hwirq) >> 0) & 0x1FF)
+
struct pmic_arb_ver_ops;
+struct apid_data {
+ u16 ppid;
+ u8 owner;
+};
+
/**
- * spmi_pmic_arb_dev - SPMI PMIC Arbiter object
+ * spmi_pmic_arb - SPMI PMIC Arbiter object
*
* @rd_base: on v1 "core", on v2 "observer" register base off DT.
* @wr_base: on v1 "core", on v2 "chnls" register base off DT.
* @ee: the current Execution Environment
* @min_apid: minimum APID (used for bounding IRQ search)
* @max_apid: maximum APID
+ * @max_periph: maximum number of PMIC peripherals supported by HW.
* @mapping_table: in-memory copy of PPID -> APID mapping table.
* @domain: irq domain object for PMIC IRQ domain
* @spmic: SPMI controller object
- * @apid_to_ppid: in-memory copy of APID -> PPID mapping table.
* @ver_ops: version dependent operations.
- * @ppid_to_chan in-memory copy of PPID -> channel (APID) mapping table.
+ * @ppid_to_apid in-memory copy of PPID -> channel (APID) mapping table.
* v2 only.
*/
-struct spmi_pmic_arb_dev {
+struct spmi_pmic_arb {
void __iomem *rd_base;
void __iomem *wr_base;
void __iomem *intr;
u8 ee;
u16 min_apid;
u16 max_apid;
+ u16 max_periph;
u32 *mapping_table;
DECLARE_BITMAP(mapping_table_valid, PMIC_ARB_MAX_PERIPHS);
struct irq_domain *domain;
struct spmi_controller *spmic;
- u16 *apid_to_ppid;
const struct pmic_arb_ver_ops *ver_ops;
- u16 *ppid_to_chan;
- u16 last_channel;
+ u16 *ppid_to_apid;
+ u16 last_apid;
+ struct apid_data apid_data[PMIC_ARB_MAX_PERIPHS];
};
/**
* pmic_arb_ver: version dependent functionality.
*
+ * @ver_str: version string.
+ * @ppid_to_apid: finds the apid for a given ppid.
+ * @mode: access rights to specified pmic peripheral.
* @non_data_cmd: on v1 issues an spmi non-data command.
* on v2 no HW support, returns -EOPNOTSUPP.
* @offset: on v1 offset of per-ee channel.
* on v2 offset of SPMI_PIC_IRQ_CLEARn.
*/
struct pmic_arb_ver_ops {
+ const char *ver_str;
+ int (*ppid_to_apid)(struct spmi_pmic_arb *pa, u8 sid, u16 addr,
+ u16 *apid);
+ int (*mode)(struct spmi_pmic_arb *dev, u8 sid, u16 addr,
+ mode_t *mode);
/* spmi commands (read_cmd, write_cmd, cmd) functionality */
- int (*offset)(struct spmi_pmic_arb_dev *dev, u8 sid, u16 addr,
+ int (*offset)(struct spmi_pmic_arb *dev, u8 sid, u16 addr,
u32 *offset);
u32 (*fmt_cmd)(u8 opc, u8 sid, u16 addr, u8 bc);
int (*non_data_cmd)(struct spmi_controller *ctrl, u8 opc, u8 sid);
/* Interrupts controller functionality (offset of PIC registers) */
- u32 (*owner_acc_status)(u8 m, u8 n);
- u32 (*acc_enable)(u8 n);
- u32 (*irq_status)(u8 n);
- u32 (*irq_clear)(u8 n);
+ u32 (*owner_acc_status)(u8 m, u16 n);
+ u32 (*acc_enable)(u16 n);
+ u32 (*irq_status)(u16 n);
+ u32 (*irq_clear)(u16 n);
};
-static inline void pmic_arb_base_write(struct spmi_pmic_arb_dev *dev,
+static inline void pmic_arb_base_write(struct spmi_pmic_arb *pa,
u32 offset, u32 val)
{
- writel_relaxed(val, dev->wr_base + offset);
+ writel_relaxed(val, pa->wr_base + offset);
}
-static inline void pmic_arb_set_rd_cmd(struct spmi_pmic_arb_dev *dev,
+static inline void pmic_arb_set_rd_cmd(struct spmi_pmic_arb *pa,
u32 offset, u32 val)
{
- writel_relaxed(val, dev->rd_base + offset);
+ writel_relaxed(val, pa->rd_base + offset);
}
/**
* @reg: register's address
* @buf: output parameter, length must be bc + 1
*/
-static void pa_read_data(struct spmi_pmic_arb_dev *dev, u8 *buf, u32 reg, u8 bc)
+static void pa_read_data(struct spmi_pmic_arb *pa, u8 *buf, u32 reg, u8 bc)
{
- u32 data = __raw_readl(dev->rd_base + reg);
+ u32 data = __raw_readl(pa->rd_base + reg);
+
memcpy(buf, &data, (bc & 3) + 1);
}
* @buf: buffer to write. length must be bc + 1.
*/
static void
-pa_write_data(struct spmi_pmic_arb_dev *dev, const u8 *buf, u32 reg, u8 bc)
+pa_write_data(struct spmi_pmic_arb *pa, const u8 *buf, u32 reg, u8 bc)
{
u32 data = 0;
+
memcpy(&data, buf, (bc & 3) + 1);
- __raw_writel(data, dev->wr_base + reg);
+ pmic_arb_base_write(pa, reg, data);
}
static int pmic_arb_wait_for_done(struct spmi_controller *ctrl,
void __iomem *base, u8 sid, u16 addr)
{
- struct spmi_pmic_arb_dev *dev = spmi_controller_get_drvdata(ctrl);
+ struct spmi_pmic_arb *pa = spmi_controller_get_drvdata(ctrl);
u32 status = 0;
u32 timeout = PMIC_ARB_TIMEOUT_US;
u32 offset;
int rc;
- rc = dev->ver_ops->offset(dev, sid, addr, &offset);
+ rc = pa->ver_ops->offset(pa, sid, addr, &offset);
if (rc)
return rc;
static int
pmic_arb_non_data_cmd_v1(struct spmi_controller *ctrl, u8 opc, u8 sid)
{
- struct spmi_pmic_arb_dev *pmic_arb = spmi_controller_get_drvdata(ctrl);
+ struct spmi_pmic_arb *pa = spmi_controller_get_drvdata(ctrl);
unsigned long flags;
u32 cmd;
int rc;
u32 offset;
- rc = pmic_arb->ver_ops->offset(pmic_arb, sid, 0, &offset);
+ rc = pa->ver_ops->offset(pa, sid, 0, &offset);
if (rc)
return rc;
cmd = ((opc | 0x40) << 27) | ((sid & 0xf) << 20);
- raw_spin_lock_irqsave(&pmic_arb->lock, flags);
- pmic_arb_base_write(pmic_arb, offset + PMIC_ARB_CMD, cmd);
- rc = pmic_arb_wait_for_done(ctrl, pmic_arb->wr_base, sid, 0);
- raw_spin_unlock_irqrestore(&pmic_arb->lock, flags);
+ raw_spin_lock_irqsave(&pa->lock, flags);
+ pmic_arb_base_write(pa, offset + PMIC_ARB_CMD, cmd);
+ rc = pmic_arb_wait_for_done(ctrl, pa->wr_base, sid, 0);
+ raw_spin_unlock_irqrestore(&pa->lock, flags);
return rc;
}
/* Non-data command */
static int pmic_arb_cmd(struct spmi_controller *ctrl, u8 opc, u8 sid)
{
- struct spmi_pmic_arb_dev *pmic_arb = spmi_controller_get_drvdata(ctrl);
+ struct spmi_pmic_arb *pa = spmi_controller_get_drvdata(ctrl);
dev_dbg(&ctrl->dev, "cmd op:0x%x sid:%d\n", opc, sid);
if (opc < SPMI_CMD_RESET || opc > SPMI_CMD_WAKEUP)
return -EINVAL;
- return pmic_arb->ver_ops->non_data_cmd(ctrl, opc, sid);
+ return pa->ver_ops->non_data_cmd(ctrl, opc, sid);
}
static int pmic_arb_read_cmd(struct spmi_controller *ctrl, u8 opc, u8 sid,
u16 addr, u8 *buf, size_t len)
{
- struct spmi_pmic_arb_dev *pmic_arb = spmi_controller_get_drvdata(ctrl);
+ struct spmi_pmic_arb *pa = spmi_controller_get_drvdata(ctrl);
unsigned long flags;
u8 bc = len - 1;
u32 cmd;
int rc;
u32 offset;
+ mode_t mode;
+
+ rc = pa->ver_ops->offset(pa, sid, addr, &offset);
+ if (rc)
+ return rc;
- rc = pmic_arb->ver_ops->offset(pmic_arb, sid, addr, &offset);
+ rc = pa->ver_ops->mode(pa, sid, addr, &mode);
if (rc)
return rc;
+ if (!(mode & S_IRUSR)) {
+ dev_err(&pa->spmic->dev,
+ "error: impermissible read from peripheral sid:%d addr:0x%x\n",
+ sid, addr);
+ return -EPERM;
+ }
+
if (bc >= PMIC_ARB_MAX_TRANS_BYTES) {
dev_err(&ctrl->dev,
"pmic-arb supports 1..%d bytes per trans, but:%zu requested",
else
return -EINVAL;
- cmd = pmic_arb->ver_ops->fmt_cmd(opc, sid, addr, bc);
+ cmd = pa->ver_ops->fmt_cmd(opc, sid, addr, bc);
- raw_spin_lock_irqsave(&pmic_arb->lock, flags);
- pmic_arb_set_rd_cmd(pmic_arb, offset + PMIC_ARB_CMD, cmd);
- rc = pmic_arb_wait_for_done(ctrl, pmic_arb->rd_base, sid, addr);
+ raw_spin_lock_irqsave(&pa->lock, flags);
+ pmic_arb_set_rd_cmd(pa, offset + PMIC_ARB_CMD, cmd);
+ rc = pmic_arb_wait_for_done(ctrl, pa->rd_base, sid, addr);
if (rc)
goto done;
- pa_read_data(pmic_arb, buf, offset + PMIC_ARB_RDATA0,
+ pa_read_data(pa, buf, offset + PMIC_ARB_RDATA0,
min_t(u8, bc, 3));
if (bc > 3)
- pa_read_data(pmic_arb, buf + 4,
- offset + PMIC_ARB_RDATA1, bc - 4);
+ pa_read_data(pa, buf + 4, offset + PMIC_ARB_RDATA1, bc - 4);
done:
- raw_spin_unlock_irqrestore(&pmic_arb->lock, flags);
+ raw_spin_unlock_irqrestore(&pa->lock, flags);
return rc;
}
static int pmic_arb_write_cmd(struct spmi_controller *ctrl, u8 opc, u8 sid,
u16 addr, const u8 *buf, size_t len)
{
- struct spmi_pmic_arb_dev *pmic_arb = spmi_controller_get_drvdata(ctrl);
+ struct spmi_pmic_arb *pa = spmi_controller_get_drvdata(ctrl);
unsigned long flags;
u8 bc = len - 1;
u32 cmd;
int rc;
u32 offset;
+ mode_t mode;
+
+ rc = pa->ver_ops->offset(pa, sid, addr, &offset);
+ if (rc)
+ return rc;
- rc = pmic_arb->ver_ops->offset(pmic_arb, sid, addr, &offset);
+ rc = pa->ver_ops->mode(pa, sid, addr, &mode);
if (rc)
return rc;
+ if (!(mode & S_IWUSR)) {
+ dev_err(&pa->spmic->dev,
+ "error: impermissible write to peripheral sid:%d addr:0x%x\n",
+ sid, addr);
+ return -EPERM;
+ }
+
if (bc >= PMIC_ARB_MAX_TRANS_BYTES) {
dev_err(&ctrl->dev,
"pmic-arb supports 1..%d bytes per trans, but:%zu requested",
else
return -EINVAL;
- cmd = pmic_arb->ver_ops->fmt_cmd(opc, sid, addr, bc);
+ cmd = pa->ver_ops->fmt_cmd(opc, sid, addr, bc);
/* Write data to FIFOs */
- raw_spin_lock_irqsave(&pmic_arb->lock, flags);
- pa_write_data(pmic_arb, buf, offset + PMIC_ARB_WDATA0,
- min_t(u8, bc, 3));
+ raw_spin_lock_irqsave(&pa->lock, flags);
+ pa_write_data(pa, buf, offset + PMIC_ARB_WDATA0, min_t(u8, bc, 3));
if (bc > 3)
- pa_write_data(pmic_arb, buf + 4,
- offset + PMIC_ARB_WDATA1, bc - 4);
+ pa_write_data(pa, buf + 4, offset + PMIC_ARB_WDATA1, bc - 4);
/* Start the transaction */
- pmic_arb_base_write(pmic_arb, offset + PMIC_ARB_CMD, cmd);
- rc = pmic_arb_wait_for_done(ctrl, pmic_arb->wr_base, sid, addr);
- raw_spin_unlock_irqrestore(&pmic_arb->lock, flags);
+ pmic_arb_base_write(pa, offset + PMIC_ARB_CMD, cmd);
+ rc = pmic_arb_wait_for_done(ctrl, pa->wr_base, sid, addr);
+ raw_spin_unlock_irqrestore(&pa->lock, flags);
return rc;
}
static void qpnpint_spmi_write(struct irq_data *d, u8 reg, void *buf,
size_t len)
{
- struct spmi_pmic_arb_dev *pa = irq_data_get_irq_chip_data(d);
- u8 sid = d->hwirq >> 24;
- u8 per = d->hwirq >> 16;
+ struct spmi_pmic_arb *pa = irq_data_get_irq_chip_data(d);
+ u8 sid = HWIRQ_SID(d->hwirq);
+ u8 per = HWIRQ_PER(d->hwirq);
if (pmic_arb_write_cmd(pa->spmic, SPMI_CMD_EXT_WRITEL, sid,
(per << 8) + reg, buf, len))
static void qpnpint_spmi_read(struct irq_data *d, u8 reg, void *buf, size_t len)
{
- struct spmi_pmic_arb_dev *pa = irq_data_get_irq_chip_data(d);
- u8 sid = d->hwirq >> 24;
- u8 per = d->hwirq >> 16;
+ struct spmi_pmic_arb *pa = irq_data_get_irq_chip_data(d);
+ u8 sid = HWIRQ_SID(d->hwirq);
+ u8 per = HWIRQ_PER(d->hwirq);
if (pmic_arb_read_cmd(pa->spmic, SPMI_CMD_EXT_READL, sid,
(per << 8) + reg, buf, len))
d->irq);
}
-static void periph_interrupt(struct spmi_pmic_arb_dev *pa, u8 apid)
+static void cleanup_irq(struct spmi_pmic_arb *pa, u16 apid, int id)
+{
+ u16 ppid = pa->apid_data[apid].ppid;
+ u8 sid = ppid >> 8;
+ u8 per = ppid & 0xFF;
+ u8 irq_mask = BIT(id);
+
+ writel_relaxed(irq_mask, pa->intr + pa->ver_ops->irq_clear(apid));
+
+ if (pmic_arb_write_cmd(pa->spmic, SPMI_CMD_EXT_WRITEL, sid,
+ (per << 8) + QPNPINT_REG_LATCHED_CLR, &irq_mask, 1))
+ dev_err_ratelimited(&pa->spmic->dev,
+ "failed to ack irq_mask = 0x%x for ppid = %x\n",
+ irq_mask, ppid);
+
+ if (pmic_arb_write_cmd(pa->spmic, SPMI_CMD_EXT_WRITEL, sid,
+ (per << 8) + QPNPINT_REG_EN_CLR, &irq_mask, 1))
+ dev_err_ratelimited(&pa->spmic->dev,
+ "failed to ack irq_mask = 0x%x for ppid = %x\n",
+ irq_mask, ppid);
+}
+
+static void periph_interrupt(struct spmi_pmic_arb *pa, u16 apid)
{
unsigned int irq;
u32 status;
int id;
+ u8 sid = (pa->apid_data[apid].ppid >> 8) & 0xF;
+ u8 per = pa->apid_data[apid].ppid & 0xFF;
status = readl_relaxed(pa->intr + pa->ver_ops->irq_status(apid));
while (status) {
id = ffs(status) - 1;
- status &= ~(1 << id);
- irq = irq_find_mapping(pa->domain,
- pa->apid_to_ppid[apid] << 16
- | id << 8
- | apid);
+ status &= ~BIT(id);
+ irq = irq_find_mapping(pa->domain, HWIRQ(sid, per, id, apid));
+ if (irq == 0) {
+ cleanup_irq(pa, apid, id);
+ continue;
+ }
generic_handle_irq(irq);
}
}
static void pmic_arb_chained_irq(struct irq_desc *desc)
{
- struct spmi_pmic_arb_dev *pa = irq_desc_get_handler_data(desc);
+ struct spmi_pmic_arb *pa = irq_desc_get_handler_data(desc);
struct irq_chip *chip = irq_desc_get_chip(desc);
void __iomem *intr = pa->intr;
int first = pa->min_apid >> 5;
int last = pa->max_apid >> 5;
- u32 status;
- int i, id;
+ u32 status, enable;
+ int i, id, apid;
chained_irq_enter(chip, desc);
pa->ver_ops->owner_acc_status(pa->ee, i));
while (status) {
id = ffs(status) - 1;
- status &= ~(1 << id);
- periph_interrupt(pa, id + i * 32);
+ status &= ~BIT(id);
+ apid = id + i * 32;
+ enable = readl_relaxed(intr +
+ pa->ver_ops->acc_enable(apid));
+ if (enable & SPMI_PIC_ACC_ENABLE_BIT)
+ periph_interrupt(pa, apid);
}
}
static void qpnpint_irq_ack(struct irq_data *d)
{
- struct spmi_pmic_arb_dev *pa = irq_data_get_irq_chip_data(d);
- u8 irq = d->hwirq >> 8;
- u8 apid = d->hwirq;
- unsigned long flags;
+ struct spmi_pmic_arb *pa = irq_data_get_irq_chip_data(d);
+ u8 irq = HWIRQ_IRQ(d->hwirq);
+ u16 apid = HWIRQ_APID(d->hwirq);
u8 data;
- raw_spin_lock_irqsave(&pa->lock, flags);
- writel_relaxed(1 << irq, pa->intr + pa->ver_ops->irq_clear(apid));
- raw_spin_unlock_irqrestore(&pa->lock, flags);
+ writel_relaxed(BIT(irq), pa->intr + pa->ver_ops->irq_clear(apid));
- data = 1 << irq;
+ data = BIT(irq);
qpnpint_spmi_write(d, QPNPINT_REG_LATCHED_CLR, &data, 1);
}
static void qpnpint_irq_mask(struct irq_data *d)
{
- struct spmi_pmic_arb_dev *pa = irq_data_get_irq_chip_data(d);
- u8 irq = d->hwirq >> 8;
- u8 apid = d->hwirq;
- unsigned long flags;
- u32 status;
- u8 data;
+ u8 irq = HWIRQ_IRQ(d->hwirq);
+ u8 data = BIT(irq);
- raw_spin_lock_irqsave(&pa->lock, flags);
- status = readl_relaxed(pa->intr + pa->ver_ops->acc_enable(apid));
- if (status & SPMI_PIC_ACC_ENABLE_BIT) {
- status = status & ~SPMI_PIC_ACC_ENABLE_BIT;
- writel_relaxed(status, pa->intr +
- pa->ver_ops->acc_enable(apid));
- }
- raw_spin_unlock_irqrestore(&pa->lock, flags);
-
- data = 1 << irq;
qpnpint_spmi_write(d, QPNPINT_REG_EN_CLR, &data, 1);
}
static void qpnpint_irq_unmask(struct irq_data *d)
{
- struct spmi_pmic_arb_dev *pa = irq_data_get_irq_chip_data(d);
- u8 irq = d->hwirq >> 8;
- u8 apid = d->hwirq;
- unsigned long flags;
- u32 status;
- u8 data;
-
- raw_spin_lock_irqsave(&pa->lock, flags);
- status = readl_relaxed(pa->intr + pa->ver_ops->acc_enable(apid));
- if (!(status & SPMI_PIC_ACC_ENABLE_BIT)) {
- writel_relaxed(status | SPMI_PIC_ACC_ENABLE_BIT,
- pa->intr + pa->ver_ops->acc_enable(apid));
+ struct spmi_pmic_arb *pa = irq_data_get_irq_chip_data(d);
+ u8 irq = HWIRQ_IRQ(d->hwirq);
+ u16 apid = HWIRQ_APID(d->hwirq);
+ u8 buf[2];
+
+ writel_relaxed(SPMI_PIC_ACC_ENABLE_BIT,
+ pa->intr + pa->ver_ops->acc_enable(apid));
+
+ qpnpint_spmi_read(d, QPNPINT_REG_EN_SET, &buf[0], 1);
+ if (!(buf[0] & BIT(irq))) {
+ /*
+ * Since the interrupt is currently disabled, write to both the
+ * LATCHED_CLR and EN_SET registers so that a spurious interrupt
+ * cannot be triggered when the interrupt is enabled
+ */
+ buf[0] = BIT(irq);
+ buf[1] = BIT(irq);
+ qpnpint_spmi_write(d, QPNPINT_REG_LATCHED_CLR, &buf, 2);
}
- raw_spin_unlock_irqrestore(&pa->lock, flags);
-
- data = 1 << irq;
- qpnpint_spmi_write(d, QPNPINT_REG_EN_SET, &data, 1);
-}
-
-static void qpnpint_irq_enable(struct irq_data *d)
-{
- u8 irq = d->hwirq >> 8;
- u8 data;
-
- qpnpint_irq_unmask(d);
-
- data = 1 << irq;
- qpnpint_spmi_write(d, QPNPINT_REG_LATCHED_CLR, &data, 1);
}
static int qpnpint_irq_set_type(struct irq_data *d, unsigned int flow_type)
{
struct spmi_pmic_arb_qpnpint_type type;
- u8 irq = d->hwirq >> 8;
+ u8 irq = HWIRQ_IRQ(d->hwirq);
+ u8 bit_mask_irq = BIT(irq);
qpnpint_spmi_read(d, QPNPINT_REG_SET_TYPE, &type, sizeof(type));
if (flow_type & (IRQF_TRIGGER_RISING | IRQF_TRIGGER_FALLING)) {
- type.type |= 1 << irq;
+ type.type |= bit_mask_irq;
if (flow_type & IRQF_TRIGGER_RISING)
- type.polarity_high |= 1 << irq;
+ type.polarity_high |= bit_mask_irq;
if (flow_type & IRQF_TRIGGER_FALLING)
- type.polarity_low |= 1 << irq;
+ type.polarity_low |= bit_mask_irq;
} else {
if ((flow_type & (IRQF_TRIGGER_HIGH)) &&
(flow_type & (IRQF_TRIGGER_LOW)))
return -EINVAL;
- type.type &= ~(1 << irq); /* level trig */
+ type.type &= ~bit_mask_irq; /* level trig */
if (flow_type & IRQF_TRIGGER_HIGH)
- type.polarity_high |= 1 << irq;
+ type.polarity_high |= bit_mask_irq;
else
- type.polarity_low |= 1 << irq;
+ type.polarity_low |= bit_mask_irq;
}
qpnpint_spmi_write(d, QPNPINT_REG_SET_TYPE, &type, sizeof(type));
+
+ if (flow_type & IRQ_TYPE_EDGE_BOTH)
+ irq_set_handler_locked(d, handle_edge_irq);
+ else
+ irq_set_handler_locked(d, handle_level_irq);
+
return 0;
}
enum irqchip_irq_state which,
bool *state)
{
- u8 irq = d->hwirq >> 8;
+ u8 irq = HWIRQ_IRQ(d->hwirq);
u8 status = 0;
if (which != IRQCHIP_STATE_LINE_LEVEL)
static struct irq_chip pmic_arb_irqchip = {
.name = "pmic_arb",
- .irq_enable = qpnpint_irq_enable,
.irq_ack = qpnpint_irq_ack,
.irq_mask = qpnpint_irq_mask,
.irq_unmask = qpnpint_irq_unmask,
| IRQCHIP_SKIP_SET_WAKE,
};
-struct spmi_pmic_arb_irq_spec {
- unsigned slave:4;
- unsigned per:8;
- unsigned irq:3;
-};
-
-static int search_mapping_table(struct spmi_pmic_arb_dev *pa,
- struct spmi_pmic_arb_irq_spec *spec,
- u8 *apid)
-{
- u16 ppid = spec->slave << 8 | spec->per;
- u32 *mapping_table = pa->mapping_table;
- int index = 0, i;
- u32 data;
-
- for (i = 0; i < SPMI_MAPPING_TABLE_TREE_DEPTH; ++i) {
- if (!test_and_set_bit(index, pa->mapping_table_valid))
- mapping_table[index] = readl_relaxed(pa->cnfg +
- SPMI_MAPPING_TABLE_REG(index));
-
- data = mapping_table[index];
-
- if (ppid & (1 << SPMI_MAPPING_BIT_INDEX(data))) {
- if (SPMI_MAPPING_BIT_IS_1_FLAG(data)) {
- index = SPMI_MAPPING_BIT_IS_1_RESULT(data);
- } else {
- *apid = SPMI_MAPPING_BIT_IS_1_RESULT(data);
- return 0;
- }
- } else {
- if (SPMI_MAPPING_BIT_IS_0_FLAG(data)) {
- index = SPMI_MAPPING_BIT_IS_0_RESULT(data);
- } else {
- *apid = SPMI_MAPPING_BIT_IS_0_RESULT(data);
- return 0;
- }
- }
- }
-
- return -ENODEV;
-}
-
static int qpnpint_irq_domain_dt_translate(struct irq_domain *d,
struct device_node *controller,
const u32 *intspec,
unsigned long *out_hwirq,
unsigned int *out_type)
{
- struct spmi_pmic_arb_dev *pa = d->host_data;
- struct spmi_pmic_arb_irq_spec spec;
- int err;
- u8 apid;
+ struct spmi_pmic_arb *pa = d->host_data;
+ int rc;
+ u16 apid;
dev_dbg(&pa->spmic->dev,
"intspec[0] 0x%1x intspec[1] 0x%02x intspec[2] 0x%02x\n",
if (intspec[0] > 0xF || intspec[1] > 0xFF || intspec[2] > 0x7)
return -EINVAL;
- spec.slave = intspec[0];
- spec.per = intspec[1];
- spec.irq = intspec[2];
-
- err = search_mapping_table(pa, &spec, &apid);
- if (err)
- return err;
-
- pa->apid_to_ppid[apid] = spec.slave << 8 | spec.per;
+ rc = pa->ver_ops->ppid_to_apid(pa, intspec[0],
+ (intspec[1] << 8), &apid);
+ if (rc < 0) {
+ dev_err(&pa->spmic->dev,
+ "failed to xlate sid = 0x%x, periph = 0x%x, irq = %x rc = %d\n",
+ intspec[0], intspec[1], intspec[2], rc);
+ return rc;
+ }
/* Keep track of {max,min}_apid for bounding search during interrupt */
if (apid > pa->max_apid)
if (apid < pa->min_apid)
pa->min_apid = apid;
- *out_hwirq = spec.slave << 24
- | spec.per << 16
- | spec.irq << 8
- | apid;
+ *out_hwirq = HWIRQ(intspec[0], intspec[1], intspec[2], apid);
*out_type = intspec[3] & IRQ_TYPE_SENSE_MASK;
dev_dbg(&pa->spmic->dev, "out_hwirq = %lu\n", *out_hwirq);
unsigned int virq,
irq_hw_number_t hwirq)
{
- struct spmi_pmic_arb_dev *pa = d->host_data;
+ struct spmi_pmic_arb *pa = d->host_data;
dev_dbg(&pa->spmic->dev, "virq = %u, hwirq = %lu\n", virq, hwirq);
return 0;
}
+static int
+pmic_arb_ppid_to_apid_v1(struct spmi_pmic_arb *pa, u8 sid, u16 addr, u16 *apid)
+{
+ u16 ppid = sid << 8 | ((addr >> 8) & 0xFF);
+ u32 *mapping_table = pa->mapping_table;
+ int index = 0, i;
+ u16 apid_valid;
+ u32 data;
+
+ apid_valid = pa->ppid_to_apid[ppid];
+ if (apid_valid & PMIC_ARB_CHAN_VALID) {
+ *apid = (apid_valid & ~PMIC_ARB_CHAN_VALID);
+ return 0;
+ }
+
+ for (i = 0; i < SPMI_MAPPING_TABLE_TREE_DEPTH; ++i) {
+ if (!test_and_set_bit(index, pa->mapping_table_valid))
+ mapping_table[index] = readl_relaxed(pa->cnfg +
+ SPMI_MAPPING_TABLE_REG(index));
+
+ data = mapping_table[index];
+
+ if (ppid & BIT(SPMI_MAPPING_BIT_INDEX(data))) {
+ if (SPMI_MAPPING_BIT_IS_1_FLAG(data)) {
+ index = SPMI_MAPPING_BIT_IS_1_RESULT(data);
+ } else {
+ *apid = SPMI_MAPPING_BIT_IS_1_RESULT(data);
+ pa->ppid_to_apid[ppid]
+ = *apid | PMIC_ARB_CHAN_VALID;
+ pa->apid_data[*apid].ppid = ppid;
+ return 0;
+ }
+ } else {
+ if (SPMI_MAPPING_BIT_IS_0_FLAG(data)) {
+ index = SPMI_MAPPING_BIT_IS_0_RESULT(data);
+ } else {
+ *apid = SPMI_MAPPING_BIT_IS_0_RESULT(data);
+ pa->ppid_to_apid[ppid]
+ = *apid | PMIC_ARB_CHAN_VALID;
+ pa->apid_data[*apid].ppid = ppid;
+ return 0;
+ }
+ }
+ }
+
+ return -ENODEV;
+}
+
+static int
+pmic_arb_mode_v1_v3(struct spmi_pmic_arb *pa, u8 sid, u16 addr, mode_t *mode)
+{
+ *mode = S_IRUSR | S_IWUSR;
+ return 0;
+}
+
/* v1 offset per ee */
static int
-pmic_arb_offset_v1(struct spmi_pmic_arb_dev *pa, u8 sid, u16 addr, u32 *offset)
+pmic_arb_offset_v1(struct spmi_pmic_arb *pa, u8 sid, u16 addr, u32 *offset)
{
*offset = 0x800 + 0x80 * pa->channel;
return 0;
}
-static u16 pmic_arb_find_chan(struct spmi_pmic_arb_dev *pa, u16 ppid)
+static u16 pmic_arb_find_apid(struct spmi_pmic_arb *pa, u16 ppid)
{
u32 regval, offset;
- u16 chan;
+ u16 apid;
u16 id;
/*
* PMIC_ARB_REG_CHNL is a table in HW mapping channel to ppid.
- * ppid_to_chan is an in-memory invert of that table.
+ * ppid_to_apid is an in-memory invert of that table.
*/
- for (chan = pa->last_channel; ; chan++) {
- offset = PMIC_ARB_REG_CHNL(chan);
+ for (apid = pa->last_apid; apid < pa->max_periph; apid++) {
+ regval = readl_relaxed(pa->cnfg +
+ SPMI_OWNERSHIP_TABLE_REG(apid));
+ pa->apid_data[apid].owner = SPMI_OWNERSHIP_PERIPH2OWNER(regval);
+
+ offset = PMIC_ARB_REG_CHNL(apid);
if (offset >= pa->core_size)
break;
continue;
id = (regval >> 8) & PMIC_ARB_PPID_MASK;
- pa->ppid_to_chan[id] = chan | PMIC_ARB_CHAN_VALID;
+ pa->ppid_to_apid[id] = apid | PMIC_ARB_CHAN_VALID;
+ pa->apid_data[apid].ppid = id;
if (id == ppid) {
- chan |= PMIC_ARB_CHAN_VALID;
+ apid |= PMIC_ARB_CHAN_VALID;
break;
}
}
- pa->last_channel = chan & ~PMIC_ARB_CHAN_VALID;
+ pa->last_apid = apid & ~PMIC_ARB_CHAN_VALID;
- return chan;
+ return apid;
}
-/* v2 offset per ppid (chan) and per ee */
static int
-pmic_arb_offset_v2(struct spmi_pmic_arb_dev *pa, u8 sid, u16 addr, u32 *offset)
+pmic_arb_ppid_to_apid_v2(struct spmi_pmic_arb *pa, u8 sid, u16 addr, u16 *apid)
{
u16 ppid = (sid << 8) | (addr >> 8);
- u16 chan;
+ u16 apid_valid;
- chan = pa->ppid_to_chan[ppid];
- if (!(chan & PMIC_ARB_CHAN_VALID))
- chan = pmic_arb_find_chan(pa, ppid);
- if (!(chan & PMIC_ARB_CHAN_VALID))
+ apid_valid = pa->ppid_to_apid[ppid];
+ if (!(apid_valid & PMIC_ARB_CHAN_VALID))
+ apid_valid = pmic_arb_find_apid(pa, ppid);
+ if (!(apid_valid & PMIC_ARB_CHAN_VALID))
return -ENODEV;
- chan &= ~PMIC_ARB_CHAN_VALID;
- *offset = 0x1000 * pa->ee + 0x8000 * chan;
+ *apid = (apid_valid & ~PMIC_ARB_CHAN_VALID);
+ return 0;
+}
+
+static int
+pmic_arb_mode_v2(struct spmi_pmic_arb *pa, u8 sid, u16 addr, mode_t *mode)
+{
+ u16 apid;
+ int rc;
+
+ rc = pmic_arb_ppid_to_apid_v2(pa, sid, addr, &apid);
+ if (rc < 0)
+ return rc;
+
+ *mode = 0;
+ *mode |= S_IRUSR;
+
+ if (pa->ee == pa->apid_data[apid].owner)
+ *mode |= S_IWUSR;
+ return 0;
+}
+
+/* v2 offset per ppid and per ee */
+static int
+pmic_arb_offset_v2(struct spmi_pmic_arb *pa, u8 sid, u16 addr, u32 *offset)
+{
+ u16 apid;
+ int rc;
+
+ rc = pmic_arb_ppid_to_apid_v2(pa, sid, addr, &apid);
+ if (rc < 0)
+ return rc;
+
+ *offset = 0x1000 * pa->ee + 0x8000 * apid;
return 0;
}
return (opc << 27) | ((addr & 0xff) << 4) | (bc & 0x7);
}
-static u32 pmic_arb_owner_acc_status_v1(u8 m, u8 n)
+static u32 pmic_arb_owner_acc_status_v1(u8 m, u16 n)
{
return 0x20 * m + 0x4 * n;
}
-static u32 pmic_arb_owner_acc_status_v2(u8 m, u8 n)
+static u32 pmic_arb_owner_acc_status_v2(u8 m, u16 n)
{
return 0x100000 + 0x1000 * m + 0x4 * n;
}
-static u32 pmic_arb_acc_enable_v1(u8 n)
+static u32 pmic_arb_owner_acc_status_v3(u8 m, u16 n)
+{
+ return 0x200000 + 0x1000 * m + 0x4 * n;
+}
+
+static u32 pmic_arb_acc_enable_v1(u16 n)
{
return 0x200 + 0x4 * n;
}
-static u32 pmic_arb_acc_enable_v2(u8 n)
+static u32 pmic_arb_acc_enable_v2(u16 n)
{
return 0x1000 * n;
}
-static u32 pmic_arb_irq_status_v1(u8 n)
+static u32 pmic_arb_irq_status_v1(u16 n)
{
return 0x600 + 0x4 * n;
}
-static u32 pmic_arb_irq_status_v2(u8 n)
+static u32 pmic_arb_irq_status_v2(u16 n)
{
return 0x4 + 0x1000 * n;
}
-static u32 pmic_arb_irq_clear_v1(u8 n)
+static u32 pmic_arb_irq_clear_v1(u16 n)
{
return 0xA00 + 0x4 * n;
}
-static u32 pmic_arb_irq_clear_v2(u8 n)
+static u32 pmic_arb_irq_clear_v2(u16 n)
{
return 0x8 + 0x1000 * n;
}
static const struct pmic_arb_ver_ops pmic_arb_v1 = {
+ .ver_str = "v1",
+ .ppid_to_apid = pmic_arb_ppid_to_apid_v1,
+ .mode = pmic_arb_mode_v1_v3,
.non_data_cmd = pmic_arb_non_data_cmd_v1,
.offset = pmic_arb_offset_v1,
.fmt_cmd = pmic_arb_fmt_cmd_v1,
};
static const struct pmic_arb_ver_ops pmic_arb_v2 = {
+ .ver_str = "v2",
+ .ppid_to_apid = pmic_arb_ppid_to_apid_v2,
+ .mode = pmic_arb_mode_v2,
.non_data_cmd = pmic_arb_non_data_cmd_v2,
.offset = pmic_arb_offset_v2,
.fmt_cmd = pmic_arb_fmt_cmd_v2,
.irq_clear = pmic_arb_irq_clear_v2,
};
+static const struct pmic_arb_ver_ops pmic_arb_v3 = {
+ .ver_str = "v3",
+ .ppid_to_apid = pmic_arb_ppid_to_apid_v2,
+ .mode = pmic_arb_mode_v1_v3,
+ .non_data_cmd = pmic_arb_non_data_cmd_v2,
+ .offset = pmic_arb_offset_v2,
+ .fmt_cmd = pmic_arb_fmt_cmd_v2,
+ .owner_acc_status = pmic_arb_owner_acc_status_v3,
+ .acc_enable = pmic_arb_acc_enable_v2,
+ .irq_status = pmic_arb_irq_status_v2,
+ .irq_clear = pmic_arb_irq_clear_v2,
+};
+
static const struct irq_domain_ops pmic_arb_irq_domain_ops = {
.map = qpnpint_irq_domain_map,
.xlate = qpnpint_irq_domain_dt_translate,
static int spmi_pmic_arb_probe(struct platform_device *pdev)
{
- struct spmi_pmic_arb_dev *pa;
+ struct spmi_pmic_arb *pa;
struct spmi_controller *ctrl;
struct resource *res;
void __iomem *core;
u32 channel, ee, hw_ver;
int err;
- bool is_v1;
ctrl = spmi_controller_alloc(&pdev->dev, sizeof(*pa));
if (!ctrl)
res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "core");
pa->core_size = resource_size(res);
+ if (pa->core_size <= 0x800) {
+ dev_err(&pdev->dev, "core_size is smaller than 0x800. Failing Probe\n");
+ err = -EINVAL;
+ goto err_put_ctrl;
+ }
+
core = devm_ioremap_resource(&ctrl->dev, res);
if (IS_ERR(core)) {
err = PTR_ERR(core);
}
hw_ver = readl_relaxed(core + PMIC_ARB_VERSION);
- is_v1 = (hw_ver < PMIC_ARB_VERSION_V2_MIN);
-
- dev_info(&ctrl->dev, "PMIC Arb Version-%d (0x%x)\n", (is_v1 ? 1 : 2),
- hw_ver);
- if (is_v1) {
+ if (hw_ver < PMIC_ARB_VERSION_V2_MIN) {
pa->ver_ops = &pmic_arb_v1;
pa->wr_base = core;
pa->rd_base = core;
} else {
pa->core = core;
- pa->ver_ops = &pmic_arb_v2;
+
+ if (hw_ver < PMIC_ARB_VERSION_V3_MIN)
+ pa->ver_ops = &pmic_arb_v2;
+ else
+ pa->ver_ops = &pmic_arb_v3;
+
+ /* the apid to ppid table starts at PMIC_ARB_REG_CHNL(0) */
+ pa->max_periph = (pa->core_size - PMIC_ARB_REG_CHNL(0)) / 4;
res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
"obsrvr");
goto err_put_ctrl;
}
- pa->ppid_to_chan = devm_kcalloc(&ctrl->dev,
+ pa->ppid_to_apid = devm_kcalloc(&ctrl->dev,
PMIC_ARB_MAX_PPID,
- sizeof(*pa->ppid_to_chan),
+ sizeof(*pa->ppid_to_apid),
GFP_KERNEL);
- if (!pa->ppid_to_chan) {
+ if (!pa->ppid_to_apid) {
err = -ENOMEM;
goto err_put_ctrl;
}
}
+ dev_info(&ctrl->dev, "PMIC arbiter version %s (0x%x)\n",
+ pa->ver_ops->ver_str, hw_ver);
+
res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "intr");
pa->intr = devm_ioremap_resource(&ctrl->dev, res);
if (IS_ERR(pa->intr)) {
pa->ee = ee;
- pa->apid_to_ppid = devm_kcalloc(&ctrl->dev, PMIC_ARB_MAX_PERIPHS,
- sizeof(*pa->apid_to_ppid),
- GFP_KERNEL);
- if (!pa->apid_to_ppid) {
- err = -ENOMEM;
- goto err_put_ctrl;
- }
-
pa->mapping_table = devm_kcalloc(&ctrl->dev, PMIC_ARB_MAX_PERIPHS - 1,
sizeof(*pa->mapping_table), GFP_KERNEL);
if (!pa->mapping_table) {
}
irq_set_chained_handler_and_data(pa->irq, pmic_arb_chained_irq, pa);
+ enable_irq_wake(pa->irq);
err = spmi_controller_add(ctrl);
if (err)
static int spmi_pmic_arb_remove(struct platform_device *pdev)
{
struct spmi_controller *ctrl = platform_get_drvdata(pdev);
- struct spmi_pmic_arb_dev *pa = spmi_controller_get_drvdata(ctrl);
+ struct spmi_pmic_arb *pa = spmi_controller_get_drvdata(ctrl);
spmi_controller_remove(ctrl);
irq_set_chained_handler_and_data(pa->irq, NULL, NULL);
irq_domain_remove(pa->domain);
int fd;
fd = ion_alloc(data.allocation.len,
- data.allocation.heap_id_mask,
- data.allocation.flags);
+ data.allocation.heap_id_mask,
+ data.allocation.flags);
if (fd < 0)
return fd;
goto err2;
}
- if (buffer->sg_table == NULL) {
+ if (!buffer->sg_table) {
WARN_ONCE(1, "This heap needs to set the sgtable");
ret = -EINVAL;
goto err1;
buffer->dev = dev;
buffer->size = len;
- INIT_LIST_HEAD(&buffer->vmas);
INIT_LIST_HEAD(&buffer->attachments);
mutex_init(&buffer->lock);
mutex_lock(&dev->buffer_lock);
if (WARN_ON(buffer->kmap_cnt > 0))
buffer->heap->ops->unmap_kernel(buffer->heap, buffer);
buffer->heap->ops->free(buffer);
- vfree(buffer->pages);
kfree(buffer);
}
return buffer->vaddr;
}
vaddr = buffer->heap->ops->map_kernel(buffer->heap, buffer);
- if (WARN_ONCE(vaddr == NULL,
+ if (WARN_ONCE(!vaddr,
"heap->ops->map_kernel should return ERR_PTR on error"))
return ERR_PTR(-EINVAL);
if (IS_ERR(vaddr))
};
static int ion_dma_buf_attach(struct dma_buf *dmabuf, struct device *dev,
- struct dma_buf_attachment *attachment)
+ struct dma_buf_attachment *attachment)
{
struct ion_dma_buf_attachment *a;
struct sg_table *table;
kfree(a);
}
-
static struct sg_table *ion_map_dma_buf(struct dma_buf_attachment *attachment,
enum dma_data_direction direction)
{
struct ion_dma_buf_attachment *a = attachment->priv;
struct sg_table *table;
- int ret;
table = a->table;
if (!dma_map_sg(attachment->dev, table->sgl, table->nents,
- direction)){
- ret = -ENOMEM;
- goto err;
- }
- return table;
+ direction))
+ return ERR_PTR(-ENOMEM);
-err:
- free_duped_table(table);
- return ERR_PTR(ret);
+ return table;
}
static void ion_unmap_dma_buf(struct dma_buf_attachment *attachment,
mutex_unlock(&buffer->lock);
}
-
mutex_lock(&buffer->lock);
list_for_each_entry(a, &buffer->attachments, list) {
dma_sync_sg_for_cpu(a->dev, a->table->sgl, a->table->nents,
- DMA_BIDIRECTIONAL);
+ DMA_BIDIRECTIONAL);
}
mutex_unlock(&buffer->lock);
mutex_lock(&buffer->lock);
list_for_each_entry(a, &buffer->attachments, list) {
dma_sync_sg_for_device(a->dev, a->table->sgl, a->table->nents,
- DMA_BIDIRECTIONAL);
+ DMA_BIDIRECTIONAL);
}
mutex_unlock(&buffer->lock);
}
up_read(&dev->lock);
- if (buffer == NULL)
+ if (!buffer)
return -ENODEV;
if (IS_ERR(buffer))
}
EXPORT_SYMBOL(ion_device_add_heap);
-int ion_device_create(void)
+static int ion_device_create(void)
{
struct ion_device *idev;
int ret;
* @kmap_cnt: number of times the buffer is mapped to the kernel
* @vaddr: the kernel mapping if kmap_cnt is not zero
* @sg_table: the sg table for the buffer if dmap_cnt is not zero
- * @pages: flat array of pages in the buffer -- used by fault
- * handler and only valid for buffers that are faulted in
- * @vmas: list of vma's mapping this buffer
- * @handle_count: count of handles referencing this buffer
- * @task_comm: taskcomm of last client to reference this buffer in a
- * handle, used for debugging
- * @pid: pid of last client to reference this buffer in a
- * handle, used for debugging
*/
struct ion_buffer {
union {
int kmap_cnt;
void *vaddr;
struct sg_table *sg_table;
- struct page **pages;
- struct list_head vmas;
struct list_head attachments;
- /* used to track orphaned buffers */
- int handle_count;
- char task_comm[TASK_COMM_LEN];
- pid_t pid;
};
void ion_buffer_destroy(struct ion_buffer *buffer);
};
static phys_addr_t ion_carveout_allocate(struct ion_heap *heap,
- unsigned long size)
+ unsigned long size)
{
struct ion_carveout_heap *carveout_heap =
container_of(heap, struct ion_carveout_heap, heap);
max_order = compound_order(page);
i++;
}
- table = kmalloc(sizeof(struct sg_table), GFP_KERNEL);
+ table = kmalloc(sizeof(*table), GFP_KERNEL);
if (!table)
goto free_pages;
for (i = len >> PAGE_SHIFT; i < (1 << order); i++)
__free_page(page + i);
- table = kmalloc(sizeof(struct sg_table), GFP_KERNEL);
+ table = kmalloc(sizeof(*table), GFP_KERNEL);
if (!table) {
ret = -ENOMEM;
goto free_pages;
{
struct ion_heap *heap;
- heap = kzalloc(sizeof(struct ion_heap), GFP_KERNEL);
+ heap = kzalloc(sizeof(*heap), GFP_KERNEL);
if (!heap)
return ERR_PTR(-ENOMEM);
heap->ops = &kmalloc_ops;
*/
#define ION_FLAG_CACHED 1
-/*
- * mappings of this buffer will created at mmap time, if this is set
- * caches must be managed manually
- */
-#define ION_FLAG_CACHED_NEEDS_SYNC 2
-
/**
* DOC: Ion Userspace API
*
#define ION_IOC_ALLOC _IOWR(ION_IOC_MAGIC, 0, \
struct ion_allocation_data)
-/**
- * DOC: ION_IOC_FREE - free memory
- *
- * Takes an ion_handle_data struct and frees the handle.
- */
-#define ION_IOC_FREE _IOWR(ION_IOC_MAGIC, 1, struct ion_handle_data)
-
-/**
- * DOC: ION_IOC_SHARE - creates a file descriptor to use to share an allocation
- *
- * Takes an ion_fd_data struct with the handle field populated with a valid
- * opaque handle. Returns the struct with the fd field set to a file
- * descriptor open in the current address space. This file descriptor
- * can then be passed to another process. The corresponding opaque handle can
- * be retrieved via ION_IOC_IMPORT.
- */
-#define ION_IOC_SHARE _IOWR(ION_IOC_MAGIC, 4, struct ion_fd_data)
-
/**
* DOC: ION_IOC_HEAP_QUERY - information about available heaps
*
select CRYPTO_CTR
select CRYPTO_XTS
help
- Say 'Y' to enable a driver for the Arm TrustZone CryptoCell
+ Say 'Y' to enable a driver for the Arm TrustZone CryptoCell
C7xx. Currently only the CryptoCell 712 REE is supported.
Choose this if you wish to use hardware acceleration of
cryptographic operations on the system REE.
Say 'Y' to enable support for FIPS compliant mode by the
CCREE driver.
If unsure say N.
-
-config CCREE_DISABLE_COHERENT_DMA_OPS
- bool "Disable Coherent DMA operations for the CCREE driver"
- depends on CRYPTO_DEV_CCREE
- default n
- help
- Say 'Y' to disable the use of coherent DMA operations by the
- CCREE driver for debugging purposes.
- If unsure say N.
obj-$(CONFIG_CRYPTO_DEV_CCREE) := ccree.o
-ccree-y := ssi_driver.o ssi_sysfs.o ssi_buffer_mgr.o ssi_request_mgr.o ssi_cipher.o ssi_hash.o ssi_aead.o ssi_ivgen.o ssi_sram_mgr.o ssi_pm.o ssi_pm_ext.o
+ccree-y := ssi_driver.o ssi_sysfs.o ssi_buffer_mgr.o ssi_request_mgr.o ssi_cipher.o ssi_hash.o ssi_aead.o ssi_ivgen.o ssi_sram_mgr.o ssi_pm.o
ccree-$(CCREE_FIPS_SUPPORT) += ssi_fips.o ssi_fips_ll.o ssi_fips_ext.o ssi_fips_local.o
+++ /dev/null
-/*
- * Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, see <http://www.gnu.org/licenses/>.
- */
-
-/*!
- * \file cc_bitops.h
- * Bit fields operations macros.
- */
-#ifndef _CC_BITOPS_H_
-#define _CC_BITOPS_H_
-
-#define BITMASK(mask_size) (((mask_size) < 32) ? \
- ((1UL << (mask_size)) - 1) : 0xFFFFFFFFUL)
-#define BITMASK_AT(mask_size, mask_offset) (BITMASK(mask_size) << (mask_offset))
-
-#define BITFIELD_GET(word, bit_offset, bit_size) \
- (((word) >> (bit_offset)) & BITMASK(bit_size))
-#define BITFIELD_SET(word, bit_offset, bit_size, new_val) do { \
- word = ((word) & ~BITMASK_AT(bit_size, bit_offset)) | \
- (((new_val) & BITMASK(bit_size)) << (bit_offset)); \
-} while (0)
-
-/* Is val aligned to "align" ("align" must be power of 2) */
-#ifndef IS_ALIGNED
-#define IS_ALIGNED(val, align) \
- (((uintptr_t)(val) & ((align) - 1)) == 0)
-#endif
-
-#define SWAP_ENDIAN(word) \
- (((word) >> 24) | (((word) & 0x00FF0000) >> 8) | \
- (((word) & 0x0000FF00) << 8) | (((word) & 0x000000FF) << 24))
-
-#ifdef BIG__ENDIAN
-#define SWAP_TO_LE(word) SWAP_ENDIAN(word)
-#define SWAP_TO_BE(word) word
-#else
-#define SWAP_TO_LE(word) word
-#define SWAP_TO_BE(word) SWAP_ENDIAN(word)
-#endif
-
-
-
-/* Is val a multiple of "mult" ("mult" must be power of 2) */
-#define IS_MULT(val, mult) \
- (((val) & ((mult) - 1)) == 0)
-
-#define IS_NULL_ADDR(adr) \
- (!(adr))
-
-#endif /*_CC_BITOPS_H_*/
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
-
#ifndef _CC_CRYPTO_CTX_H_
#define _CC_CRYPTO_CTX_H_
-#ifdef __KERNEL__
#include <linux/types.h>
-#define INT32_MAX 0x7FFFFFFFL
-#else
-#include <stdint.h>
-#endif
-
-
-#ifndef max
-#define max(a, b) ((a) > (b) ? (a) : (b))
-#define min(a, b) ((a) < (b) ? (a) : (b))
-#endif
/* context size */
#ifndef CC_CTX_SIZE_LOG2
#define CC_CTX_SIZE_LOG2 7
#endif
#endif
-#define CC_CTX_SIZE (1<<CC_CTX_SIZE_LOG2)
+#define CC_CTX_SIZE BIT(CC_CTX_SIZE_LOG2)
#define CC_DRV_CTX_SIZE_WORDS (CC_CTX_SIZE >> 2)
#define CC_DRV_DES_IV_SIZE 8
#define CC_AES_KEY_SIZE_MAX CC_AES_256_BIT_KEY_SIZE
#define CC_AES_KEY_SIZE_WORDS_MAX (CC_AES_KEY_SIZE_MAX >> 2)
-#define CC_MD5_DIGEST_SIZE 16
-#define CC_SHA1_DIGEST_SIZE 20
-#define CC_SHA224_DIGEST_SIZE 28
-#define CC_SHA256_DIGEST_SIZE 32
+#define CC_MD5_DIGEST_SIZE 16
+#define CC_SHA1_DIGEST_SIZE 20
+#define CC_SHA224_DIGEST_SIZE 28
+#define CC_SHA256_DIGEST_SIZE 32
#define CC_SHA256_DIGEST_SIZE_IN_WORDS 8
-#define CC_SHA384_DIGEST_SIZE 48
-#define CC_SHA512_DIGEST_SIZE 64
+#define CC_SHA384_DIGEST_SIZE 48
+#define CC_SHA512_DIGEST_SIZE 64
#define CC_SHA1_BLOCK_SIZE 64
#define CC_SHA1_BLOCK_SIZE_IN_WORDS 16
#define CC_HMAC_BLOCK_SIZE_MAX CC_HASH_BLOCK_SIZE_MAX
-#define CC_MULTI2_SYSTEM_KEY_SIZE 32
-#define CC_MULTI2_DATA_KEY_SIZE 8
-#define CC_MULTI2_SYSTEM_N_DATA_KEY_SIZE (CC_MULTI2_SYSTEM_KEY_SIZE + CC_MULTI2_DATA_KEY_SIZE)
+#define CC_MULTI2_SYSTEM_KEY_SIZE 32
+#define CC_MULTI2_DATA_KEY_SIZE 8
+#define CC_MULTI2_SYSTEM_N_DATA_KEY_SIZE \
+ (CC_MULTI2_SYSTEM_KEY_SIZE + CC_MULTI2_DATA_KEY_SIZE)
#define CC_MULTI2_BLOCK_SIZE 8
#define CC_MULTI2_IV_SIZE 8
#define CC_MULTI2_MIN_NUM_ROUNDS 8
#define CC_MULTI2_MAX_NUM_ROUNDS 128
-
#define CC_DRV_ALG_MAX_BLOCK_SIZE CC_HASH_BLOCK_SIZE_MAX
-
enum drv_engine_type {
DRV_ENGINE_NULL = 0,
DRV_ENGINE_AES = 1,
DRV_ENGINE_HASH = 3,
DRV_ENGINE_RC4 = 4,
DRV_ENGINE_DOUT = 5,
- DRV_ENGINE_RESERVE32B = INT32_MAX,
+ DRV_ENGINE_RESERVE32B = S32_MAX,
};
enum drv_crypto_alg {
DRV_CRYPTO_ALG_AEAD = 5,
DRV_CRYPTO_ALG_BYPASS = 6,
DRV_CRYPTO_ALG_NUM = 7,
- DRV_CRYPTO_ALG_RESERVE32B = INT32_MAX
+ DRV_CRYPTO_ALG_RESERVE32B = S32_MAX
};
enum drv_crypto_direction {
DRV_CRYPTO_DIRECTION_ENCRYPT = 0,
DRV_CRYPTO_DIRECTION_DECRYPT = 1,
DRV_CRYPTO_DIRECTION_DECRYPT_ENCRYPT = 3,
- DRV_CRYPTO_DIRECTION_RESERVE32B = INT32_MAX
+ DRV_CRYPTO_DIRECTION_RESERVE32B = S32_MAX
};
enum drv_cipher_mode {
DRV_CIPHER_GCTR = 12,
DRV_CIPHER_ESSIV = 13,
DRV_CIPHER_BITLOCKER = 14,
- DRV_CIPHER_RESERVE32B = INT32_MAX
+ DRV_CIPHER_RESERVE32B = S32_MAX
};
enum drv_hash_mode {
DRV_HASH_SHA512 = 3,
DRV_HASH_SHA384 = 4,
DRV_HASH_MD5 = 5,
- DRV_HASH_CBC_MAC = 6,
+ DRV_HASH_CBC_MAC = 6,
DRV_HASH_XCBC_MAC = 7,
DRV_HASH_CMAC = 8,
DRV_HASH_MODE_NUM = 9,
- DRV_HASH_RESERVE32B = INT32_MAX
+ DRV_HASH_RESERVE32B = S32_MAX
};
enum drv_hash_hw_mode {
DRV_HASH_HW_SHA512 = 4,
DRV_HASH_HW_SHA384 = 12,
DRV_HASH_HW_GHASH = 6,
- DRV_HASH_HW_RESERVE32B = INT32_MAX
+ DRV_HASH_HW_RESERVE32B = S32_MAX
};
enum drv_multi2_mode {
DRV_MULTI2_ECB = 0,
DRV_MULTI2_CBC = 1,
DRV_MULTI2_OFB = 2,
- DRV_MULTI2_RESERVE32B = INT32_MAX
+ DRV_MULTI2_RESERVE32B = S32_MAX
};
-
/* drv_crypto_key_type[1:0] is mapped to cipher_do[1:0] */
/* drv_crypto_key_type[2] is mapped to cipher_config2 */
enum drv_crypto_key_type {
DRV_APPLET_KEY = 4, /* NA */
DRV_PLATFORM_KEY = 5, /* 0x101 */
DRV_CUSTOMER_KEY = 6, /* 0x110 */
- DRV_END_OF_KEYS = INT32_MAX,
+ DRV_END_OF_KEYS = S32_MAX,
};
enum drv_crypto_padding_type {
DRV_PADDING_NONE = 0,
DRV_PADDING_PKCS7 = 1,
- DRV_PADDING_RESERVE32B = INT32_MAX
+ DRV_PADDING_RESERVE32B = S32_MAX
};
-/*******************************************************************/
-/***************** DESCRIPTOR BASED CONTEXTS ***********************/
-/*******************************************************************/
-
- /* Generic context ("super-class") */
-struct drv_ctx_generic {
- enum drv_crypto_alg alg;
-} __attribute__((__may_alias__));
-
-
-struct drv_ctx_hash {
- enum drv_crypto_alg alg; /* DRV_CRYPTO_ALG_HASH */
- enum drv_hash_mode mode;
- uint8_t digest[CC_DIGEST_SIZE_MAX];
- /* reserve to end of allocated context size */
- uint8_t reserved[CC_CTX_SIZE - 2 * sizeof(uint32_t) -
- CC_DIGEST_SIZE_MAX];
-};
-
-/* !!!! drv_ctx_hmac should have the same structure as drv_ctx_hash except
- k0, k0_size fields */
-struct drv_ctx_hmac {
- enum drv_crypto_alg alg; /* DRV_CRYPTO_ALG_HMAC */
- enum drv_hash_mode mode;
- uint8_t digest[CC_DIGEST_SIZE_MAX];
- uint32_t k0[CC_HMAC_BLOCK_SIZE_MAX/sizeof(uint32_t)];
- uint32_t k0_size;
- /* reserve to end of allocated context size */
- uint8_t reserved[CC_CTX_SIZE - 3 * sizeof(uint32_t) -
- CC_DIGEST_SIZE_MAX - CC_HMAC_BLOCK_SIZE_MAX];
-};
-
-struct drv_ctx_cipher {
- enum drv_crypto_alg alg; /* DRV_CRYPTO_ALG_AES */
- enum drv_cipher_mode mode;
- enum drv_crypto_direction direction;
- enum drv_crypto_key_type crypto_key_type;
- enum drv_crypto_padding_type padding_type;
- uint32_t key_size; /* numeric value in bytes */
- uint32_t data_unit_size; /* required for XTS */
- /* block_state is the AES engine block state.
- * It is used by the host to pass IV or counter at initialization.
- * It is used by SeP for intermediate block chaining state and for
- * returning MAC algorithms results. */
- uint8_t block_state[CC_AES_BLOCK_SIZE];
- uint8_t key[CC_AES_KEY_SIZE_MAX];
- uint8_t xex_key[CC_AES_KEY_SIZE_MAX];
- /* reserve to end of allocated context size */
- uint32_t reserved[CC_DRV_CTX_SIZE_WORDS - 7 -
- CC_AES_BLOCK_SIZE/sizeof(uint32_t) - 2 *
- (CC_AES_KEY_SIZE_MAX/sizeof(uint32_t))];
-};
-
-/* authentication and encryption with associated data class */
-struct drv_ctx_aead {
- enum drv_crypto_alg alg; /* DRV_CRYPTO_ALG_AES */
- enum drv_cipher_mode mode;
- enum drv_crypto_direction direction;
- uint32_t key_size; /* numeric value in bytes */
- uint32_t nonce_size; /* nonce size (octets) */
- uint32_t header_size; /* finit additional data size (octets) */
- uint32_t text_size; /* finit text data size (octets) */
- uint32_t tag_size; /* mac size, element of {4, 6, 8, 10, 12, 14, 16} */
- /* block_state1/2 is the AES engine block state */
- uint8_t block_state[CC_AES_BLOCK_SIZE];
- uint8_t mac_state[CC_AES_BLOCK_SIZE]; /* MAC result */
- uint8_t nonce[CC_AES_BLOCK_SIZE]; /* nonce buffer */
- uint8_t key[CC_AES_KEY_SIZE_MAX];
- /* reserve to end of allocated context size */
- uint32_t reserved[CC_DRV_CTX_SIZE_WORDS - 8 -
- 3 * (CC_AES_BLOCK_SIZE/sizeof(uint32_t)) -
- CC_AES_KEY_SIZE_MAX/sizeof(uint32_t)];
-};
-
-/*******************************************************************/
-/***************** MESSAGE BASED CONTEXTS **************************/
-/*******************************************************************/
-
-
-/* Get the address of a @member within a given @ctx address
- @ctx: The context address
- @type: Type of context structure
- @member: Associated context field */
-#define GET_CTX_FIELD_ADDR(ctx, type, member) (ctx + offsetof(type, member))
-
#endif /* _CC_CRYPTO_CTX_H_ */
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
-/* pseudo cc_hal.h for cc7x_perf_test_driver (to be able to include code from CC drivers) */
+/* pseudo cc_hal.h for cc7x_perf_test_driver (to be able to include code from
+ * CC drivers).
+ */
#ifndef __CC_HAL_H__
#define __CC_HAL_H__
#define READ_REGISTER(_addr) ioread32((_addr))
#define WRITE_REGISTER(_addr, _data) iowrite32((_data), (_addr))
-#define CC_HAL_WRITE_REGISTER(offset, val) WRITE_REGISTER(cc_base + offset, val)
-#define CC_HAL_READ_REGISTER(offset) READ_REGISTER(cc_base + offset)
+#define CC_HAL_WRITE_REGISTER(offset, val) \
+ WRITE_REGISTER(cc_base + (offset), val)
+#define CC_HAL_READ_REGISTER(offset) READ_REGISTER(cc_base + (offset))
#endif
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
#ifndef __CC_HW_QUEUE_DEFS_H__
#define __CC_HW_QUEUE_DEFS_H__
-#include "cc_pal_log.h"
-#include "cc_regs.h"
-#include "dx_crys_kernel.h"
-
-#ifdef __KERNEL__
#include <linux/types.h>
-#define UINT32_MAX 0xFFFFFFFFL
-#define INT32_MAX 0x7FFFFFFFL
-#define UINT16_MAX 0xFFFFL
-#else
-#include <stdint.h>
-#endif
-
-/******************************************************************************
-* DEFINITIONS
-******************************************************************************/
+#include "dx_crys_kernel.h"
+#include <linux/bitfield.h>
-/* Dma AXI Secure bit */
-#define AXI_SECURE 0
-#define AXI_NOT_SECURE 1
+/******************************************************************************
+ * DEFINITIONS
+ ******************************************************************************/
#define HW_DESC_SIZE_WORDS 6
#define HW_QUEUE_SLOTS_MAX 15 /* Max. available slots in HW queue */
-#define _HW_DESC_MONITOR_KICK 0x7FFFC00
+#define CC_REG_NAME(word, name) DX_DSCRPTR_QUEUE_WORD ## word ## _ ## name
+
+#define CC_REG_LOW(word, name) \
+ (DX_DSCRPTR_QUEUE_WORD ## word ## _ ## name ## _BIT_SHIFT)
+
+#define CC_REG_HIGH(word, name) \
+ (CC_REG_LOW(word, name) + \
+ DX_DSCRPTR_QUEUE_WORD ## word ## _ ## name ## _BIT_SIZE - 1)
+
+#define CC_GENMASK(word, name) \
+ GENMASK(CC_REG_HIGH(word, name), CC_REG_LOW(word, name))
+
+#define WORD0_VALUE CC_GENMASK(0, VALUE)
+#define WORD1_DIN_CONST_VALUE CC_GENMASK(1, DIN_CONST_VALUE)
+#define WORD1_DIN_DMA_MODE CC_GENMASK(1, DIN_DMA_MODE)
+#define WORD1_DIN_SIZE CC_GENMASK(1, DIN_SIZE)
+#define WORD1_NOT_LAST CC_GENMASK(1, NOT_LAST)
+#define WORD1_NS_BIT CC_GENMASK(1, NS_BIT)
+#define WORD2_VALUE CC_GENMASK(2, VALUE)
+#define WORD3_DOUT_DMA_MODE CC_GENMASK(3, DOUT_DMA_MODE)
+#define WORD3_DOUT_LAST_IND CC_GENMASK(3, DOUT_LAST_IND)
+#define WORD3_DOUT_SIZE CC_GENMASK(3, DOUT_SIZE)
+#define WORD3_HASH_XOR_BIT CC_GENMASK(3, HASH_XOR_BIT)
+#define WORD3_NS_BIT CC_GENMASK(3, NS_BIT)
+#define WORD3_QUEUE_LAST_IND CC_GENMASK(3, QUEUE_LAST_IND)
+#define WORD4_ACK_NEEDED CC_GENMASK(4, ACK_NEEDED)
+#define WORD4_AES_SEL_N_HASH CC_GENMASK(4, AES_SEL_N_HASH)
+#define WORD4_BYTES_SWAP CC_GENMASK(4, BYTES_SWAP)
+#define WORD4_CIPHER_CONF0 CC_GENMASK(4, CIPHER_CONF0)
+#define WORD4_CIPHER_CONF1 CC_GENMASK(4, CIPHER_CONF1)
+#define WORD4_CIPHER_CONF2 CC_GENMASK(4, CIPHER_CONF2)
+#define WORD4_CIPHER_DO CC_GENMASK(4, CIPHER_DO)
+#define WORD4_CIPHER_MODE CC_GENMASK(4, CIPHER_MODE)
+#define WORD4_CMAC_SIZE0 CC_GENMASK(4, CMAC_SIZE0)
+#define WORD4_DATA_FLOW_MODE CC_GENMASK(4, DATA_FLOW_MODE)
+#define WORD4_KEY_SIZE CC_GENMASK(4, KEY_SIZE)
+#define WORD4_SETUP_OPERATION CC_GENMASK(4, SETUP_OPERATION)
+#define WORD5_DIN_ADDR_HIGH CC_GENMASK(5, DIN_ADDR_HIGH)
+#define WORD5_DOUT_ADDR_HIGH CC_GENMASK(5, DOUT_ADDR_HIGH)
/******************************************************************************
-* TYPE DEFINITIONS
-******************************************************************************/
+ * TYPE DEFINITIONS
+ ******************************************************************************/
+
+struct cc_hw_desc {
+ union {
+ u32 word[HW_DESC_SIZE_WORDS];
+ u16 hword[HW_DESC_SIZE_WORDS * 2];
+ };
+};
-typedef struct HwDesc {
- uint32_t word[HW_DESC_SIZE_WORDS];
-} HwDesc_s;
+enum cc_axi_sec {
+ AXI_SECURE = 0,
+ AXI_NOT_SECURE = 1
+};
-typedef enum DescDirection {
+enum cc_desc_direction {
DESC_DIRECTION_ILLEGAL = -1,
DESC_DIRECTION_ENCRYPT_ENCRYPT = 0,
DESC_DIRECTION_DECRYPT_DECRYPT = 1,
DESC_DIRECTION_DECRYPT_ENCRYPT = 3,
- DESC_DIRECTION_END = INT32_MAX,
-}DescDirection_t;
+ DESC_DIRECTION_END = S32_MAX,
+};
-typedef enum DmaMode {
+enum cc_dma_mode {
DMA_MODE_NULL = -1,
- NO_DMA = 0,
+ NO_DMA = 0,
DMA_SRAM = 1,
DMA_DLLI = 2,
DMA_MLLI = 3,
- DmaMode_OPTIONTS,
- DmaMode_END = INT32_MAX,
-}DmaMode_t;
+ DMA_MODE_END = S32_MAX,
+};
-typedef enum FlowMode {
+enum cc_flow_mode {
FLOW_MODE_NULL = -1,
/* data flows */
- BYPASS = 0,
+ BYPASS = 0,
DIN_AES_DOUT = 1,
AES_to_HASH = 2,
AES_and_HASH = 3,
DIN_AES_AESMAC = 17,
HASH_to_DOUT = 18,
/* setup flows */
- S_DIN_to_AES = 32,
+ S_DIN_to_AES = 32,
S_DIN_to_AES2 = 33,
S_DIN_to_DES = 34,
S_DIN_to_RC4 = 35,
- S_DIN_to_MULTI2 = 36,
+ S_DIN_to_MULTI2 = 36,
S_DIN_to_HASH = 37,
S_AES_to_DOUT = 38,
S_AES2_to_DOUT = 39,
S_DES_to_DOUT = 42,
S_HASH_to_DOUT = 43,
SET_FLOW_ID = 44,
- FlowMode_OPTIONTS,
- FlowMode_END = INT32_MAX,
-}FlowMode_t;
+ FLOW_MODE_END = S32_MAX,
+};
-typedef enum TunnelOp {
+enum cc_tunnel_op {
TUNNEL_OP_INVALID = -1,
TUNNEL_OFF = 0,
TUNNEL_ON = 1,
- TunnelOp_OPTIONS,
- TunnelOp_END = INT32_MAX,
-} TunnelOp_t;
+ TUNNEL_OP_END = S32_MAX,
+};
-typedef enum SetupOp {
+enum cc_setup_op {
SETUP_LOAD_NOP = 0,
SETUP_LOAD_STATE0 = 1,
SETUP_LOAD_STATE1 = 2,
SETUP_LOAD_STATE2 = 3,
SETUP_LOAD_KEY0 = 4,
SETUP_LOAD_XEX_KEY = 5,
- SETUP_WRITE_STATE0 = 8,
+ SETUP_WRITE_STATE0 = 8,
SETUP_WRITE_STATE1 = 9,
SETUP_WRITE_STATE2 = 10,
SETUP_WRITE_STATE3 = 11,
- setupOp_OPTIONTS,
- setupOp_END = INT32_MAX,
-}SetupOp_t;
+ SETUP_OP_END = S32_MAX,
+};
-enum AesMacSelector {
+enum cc_aes_mac_selector {
AES_SK = 1,
AES_CMAC_INIT = 2,
AES_CMAC_SIZE0 = 3,
- AesMacEnd = INT32_MAX,
+ AES_MAC_END = S32_MAX,
};
-#define HW_KEY_MASK_CIPHER_DO 0x3
+#define HW_KEY_MASK_CIPHER_DO 0x3
#define HW_KEY_SHIFT_CIPHER_CFG2 2
-
/* HwCryptoKey[1:0] is mapped to cipher_do[1:0] */
/* HwCryptoKey[2:3] is mapped to cipher_config2[1:0] */
-typedef enum HwCryptoKey {
+enum cc_hw_crypto_key {
USER_KEY = 0, /* 0x0000 */
ROOT_KEY = 1, /* 0x0001 */
PROVISIONING_KEY = 2, /* 0x0010 */ /* ==KCP */
KFDE1_KEY = 9, /* 0x1001 */
KFDE2_KEY = 10, /* 0x1010 */
KFDE3_KEY = 11, /* 0x1011 */
- END_OF_KEYS = INT32_MAX,
-}HwCryptoKey_t;
+ END_OF_KEYS = S32_MAX,
+};
-typedef enum HwAesKeySize {
+enum cc_hw_aes_key_size {
AES_128_KEY = 0,
AES_192_KEY = 1,
AES_256_KEY = 2,
- END_OF_AES_KEYS = INT32_MAX,
-}HwAesKeySize_t;
+ END_OF_AES_KEYS = S32_MAX,
+};
-typedef enum HwDesKeySize {
+enum cc_hw_des_key_size {
DES_ONE_KEY = 0,
DES_TWO_KEYS = 1,
DES_THREE_KEYS = 2,
- END_OF_DES_KEYS = INT32_MAX,
-}HwDesKeySize_t;
+ END_OF_DES_KEYS = S32_MAX,
+};
/*****************************/
/* Descriptor packing macros */
/*****************************/
-#define GET_HW_Q_DESC_WORD_IDX(descWordIdx) (CC_REG_OFFSET(CRY_KERNEL, DSCRPTR_QUEUE_WORD ## descWordIdx) )
-
-#define HW_DESC_INIT(pDesc) do { \
- (pDesc)->word[0] = 0; \
- (pDesc)->word[1] = 0; \
- (pDesc)->word[2] = 0; \
- (pDesc)->word[3] = 0; \
- (pDesc)->word[4] = 0; \
- (pDesc)->word[5] = 0; \
-} while (0)
-
-/* HW descriptor debug functions */
-int createDetailedDump(HwDesc_s *pDesc);
-void descriptor_log(HwDesc_s *desc);
-
-#if defined(HW_DESCRIPTOR_LOG) || defined(HW_DESC_DUMP_HOST_BUF)
-#define LOG_HW_DESC(pDesc) descriptor_log(pDesc)
-#else
-#define LOG_HW_DESC(pDesc)
-#endif
-
-#if (CC_PAL_MAX_LOG_LEVEL >= CC_PAL_LOG_LEVEL_TRACE) || defined(OEMFW_LOG)
-
-#ifdef UART_PRINTF
-#define CREATE_DETAILED_DUMP(pDesc) createDetailedDump(pDesc)
-#else
-#define CREATE_DETAILED_DUMP(pDesc)
-#endif
-
-#define HW_DESC_DUMP(pDesc) do { \
- CC_PAL_LOG_TRACE("\n---------------------------------------------------\n"); \
- CREATE_DETAILED_DUMP(pDesc); \
- CC_PAL_LOG_TRACE("0x%08X, ", (unsigned int)(pDesc)->word[0]); \
- CC_PAL_LOG_TRACE("0x%08X, ", (unsigned int)(pDesc)->word[1]); \
- CC_PAL_LOG_TRACE("0x%08X, ", (unsigned int)(pDesc)->word[2]); \
- CC_PAL_LOG_TRACE("0x%08X, ", (unsigned int)(pDesc)->word[3]); \
- CC_PAL_LOG_TRACE("0x%08X, ", (unsigned int)(pDesc)->word[4]); \
- CC_PAL_LOG_TRACE("0x%08X\n", (unsigned int)(pDesc)->word[5]); \
- CC_PAL_LOG_TRACE("---------------------------------------------------\n\n"); \
-} while (0)
-
-#else
-#define HW_DESC_DUMP(pDesc) do {} while (0)
-#endif
-
-
-/*!
- * This macro indicates the end of current HW descriptors flow and release the HW engines.
- *
- * \param pDesc pointer HW descriptor struct
- */
-#define HW_DESC_SET_QUEUE_LAST_IND(pDesc) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD3, QUEUE_LAST_IND, (pDesc)->word[3], 1); \
- } while (0)
-
-/*!
- * This macro signs the end of HW descriptors flow by asking for completion ack, and release the HW engines
- *
- * \param pDesc pointer HW descriptor struct
- */
-#define HW_DESC_SET_ACK_LAST(pDesc) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD3, QUEUE_LAST_IND, (pDesc)->word[3], 1); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD4, ACK_NEEDED, (pDesc)->word[4], 1); \
- } while (0)
-
-
-#define MSB64(_addr) (sizeof(_addr) == 4 ? 0 : ((_addr) >> 32)&UINT16_MAX)
-
-/*!
- * This macro sets the DIN field of a HW descriptors
- *
- * \param pDesc pointer HW descriptor struct
- * \param dmaMode The DMA mode: NO_DMA, SRAM, DLLI, MLLI, CONSTANT
- * \param dinAdr DIN address
- * \param dinSize Data size in bytes
- * \param axiNs AXI secure bit
+/*
+ * Init a HW descriptor struct
+ * @pdesc: pointer HW descriptor struct
*/
-#define HW_DESC_SET_DIN_TYPE(pDesc, dmaMode, dinAdr, dinSize, axiNs) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD0, VALUE, (pDesc)->word[0], (dinAdr)&UINT32_MAX ); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD5, DIN_ADDR_HIGH, (pDesc)->word[5], MSB64(dinAdr) ); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD1, DIN_DMA_MODE, (pDesc)->word[1], (dmaMode)); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD1, DIN_SIZE, (pDesc)->word[1], (dinSize)); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD1, NS_BIT, (pDesc)->word[1], (axiNs)); \
- } while (0)
-
-
-/*!
- * This macro sets the DIN field of a HW descriptors to NO DMA mode. Used for NOP descriptor, register patches and
- * other special modes
- *
- * \param pDesc pointer HW descriptor struct
- * \param dinAdr DIN address
- * \param dinSize Data size in bytes
+static inline void hw_desc_init(struct cc_hw_desc *pdesc)
+{
+ memset(pdesc, 0, sizeof(struct cc_hw_desc));
+}
+
+/*
+ * Indicates the end of current HW descriptors flow and release the HW engines.
+ *
+ * @pdesc: pointer HW descriptor struct
*/
-#define HW_DESC_SET_DIN_NO_DMA(pDesc, dinAdr, dinSize) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD0, VALUE, (pDesc)->word[0], (uint32_t)(dinAdr)); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD1, DIN_SIZE, (pDesc)->word[1], (dinSize)); \
- } while (0)
-
-/*!
- * This macro sets the DIN field of a HW descriptors to SRAM mode.
- * Note: No need to check SRAM alignment since host requests do not use SRAM and
- * adaptor will enforce alignment check.
- *
- * \param pDesc pointer HW descriptor struct
- * \param dinAdr DIN address
- * \param dinSize Data size in bytes
+static inline void set_queue_last_ind(struct cc_hw_desc *pdesc)
+{
+ pdesc->word[3] |= FIELD_PREP(WORD3_QUEUE_LAST_IND, 1);
+}
+
+/*
+ * Set the DIN field of a HW descriptors
+ *
+ * @pdesc: pointer HW descriptor struct
+ * @dma_mode: dmaMode The DMA mode: NO_DMA, SRAM, DLLI, MLLI, CONSTANT
+ * @addr: dinAdr DIN address
+ * @size: Data size in bytes
+ * @axi_sec: AXI secure bit
*/
-#define HW_DESC_SET_DIN_SRAM(pDesc, dinAdr, dinSize) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD0, VALUE, (pDesc)->word[0], (uint32_t)(dinAdr)); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD1, DIN_DMA_MODE, (pDesc)->word[1], DMA_SRAM); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD1, DIN_SIZE, (pDesc)->word[1], (dinSize)); \
- } while (0)
-
-/*! This macro sets the DIN field of a HW descriptors to CONST mode
- *
- * \param pDesc pointer HW descriptor struct
- * \param val DIN const value
- * \param dinSize Data size in bytes
+static inline void set_din_type(struct cc_hw_desc *pdesc,
+ enum cc_dma_mode dma_mode, dma_addr_t addr,
+ u32 size, enum cc_axi_sec axi_sec)
+{
+ pdesc->word[0] = (u32)addr;
+#ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT
+ pdesc->word[5] |= FIELD_PREP(WORD5_DIN_ADDR_HIGH, ((u16)(addr >> 32)));
+#endif
+ pdesc->word[1] |= FIELD_PREP(WORD1_DIN_DMA_MODE, dma_mode) |
+ FIELD_PREP(WORD1_DIN_SIZE, size) |
+ FIELD_PREP(WORD1_NS_BIT, axi_sec);
+}
+
+/*
+ * Set the DIN field of a HW descriptors to NO DMA mode.
+ * Used for NOP descriptor, register patches and other special modes.
+ *
+ * @pdesc: pointer HW descriptor struct
+ * @addr: DIN address
+ * @size: Data size in bytes
*/
-#define HW_DESC_SET_DIN_CONST(pDesc, val, dinSize) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD0, VALUE, (pDesc)->word[0], (uint32_t)(val)); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD1, DIN_CONST_VALUE, (pDesc)->word[1], 1); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD1, DIN_DMA_MODE, (pDesc)->word[1], DMA_SRAM); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD1, DIN_SIZE, (pDesc)->word[1], (dinSize)); \
- } while (0)
-
-/*!
- * This macro sets the DIN not last input data indicator
- *
- * \param pDesc pointer HW descriptor struct
+static inline void set_din_no_dma(struct cc_hw_desc *pdesc, u32 addr, u32 size)
+{
+ pdesc->word[0] = addr;
+ pdesc->word[1] |= FIELD_PREP(WORD1_DIN_SIZE, size);
+}
+
+/*
+ * Set the DIN field of a HW descriptors to SRAM mode.
+ * Note: No need to check SRAM alignment since host requests do not use SRAM and
+ * adaptor will enforce alignment check.
+ *
+ * @pdesc: pointer HW descriptor struct
+ * @addr: DIN address
+ * @size Data size in bytes
*/
-#define HW_DESC_SET_DIN_NOT_LAST_INDICATION(pDesc) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD1, NOT_LAST, (pDesc)->word[1], 1); \
- } while (0)
-
-/*!
- * This macro sets the DOUT field of a HW descriptors
- *
- * \param pDesc pointer HW descriptor struct
- * \param dmaMode The DMA mode: NO_DMA, SRAM, DLLI, MLLI, CONSTANT
- * \param doutAdr DOUT address
- * \param doutSize Data size in bytes
- * \param axiNs AXI secure bit
+static inline void set_din_sram(struct cc_hw_desc *pdesc, dma_addr_t addr,
+ u32 size)
+{
+ pdesc->word[0] = (u32)addr;
+ pdesc->word[1] |= FIELD_PREP(WORD1_DIN_SIZE, size) |
+ FIELD_PREP(WORD1_DIN_DMA_MODE, DMA_SRAM);
+}
+
+/*
+ * Set the DIN field of a HW descriptors to CONST mode
+ *
+ * @pdesc: pointer HW descriptor struct
+ * @val: DIN const value
+ * @size: Data size in bytes
*/
-#define HW_DESC_SET_DOUT_TYPE(pDesc, dmaMode, doutAdr, doutSize, axiNs) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD2, VALUE, (pDesc)->word[2], (doutAdr)&UINT32_MAX ); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD5, DOUT_ADDR_HIGH, (pDesc)->word[5], MSB64(doutAdr) ); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD3, DOUT_DMA_MODE, (pDesc)->word[3], (dmaMode)); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD3, DOUT_SIZE, (pDesc)->word[3], (doutSize)); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD3, NS_BIT, (pDesc)->word[3], (axiNs)); \
- } while (0)
-
-/*!
- * This macro sets the DOUT field of a HW descriptors to DLLI type
- * The LAST INDICATION is provided by the user
- *
- * \param pDesc pointer HW descriptor struct
- * \param doutAdr DOUT address
- * \param doutSize Data size in bytes
- * \param lastInd The last indication bit
- * \param axiNs AXI secure bit
+static inline void set_din_const(struct cc_hw_desc *pdesc, u32 val, u32 size)
+{
+ pdesc->word[0] = val;
+ pdesc->word[1] |= FIELD_PREP(WORD1_DIN_CONST_VALUE, 1) |
+ FIELD_PREP(WORD1_DIN_DMA_MODE, DMA_SRAM) |
+ FIELD_PREP(WORD1_DIN_SIZE, size);
+}
+
+/*
+ * Set the DIN not last input data indicator
+ *
+ * @pdesc: pointer HW descriptor struct
*/
-#define HW_DESC_SET_DOUT_DLLI(pDesc, doutAdr, doutSize, axiNs ,lastInd) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD2, VALUE, (pDesc)->word[2], (doutAdr)&UINT32_MAX ); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD5, DOUT_ADDR_HIGH, (pDesc)->word[5], MSB64(doutAdr) ); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD3, DOUT_DMA_MODE, (pDesc)->word[3], DMA_DLLI); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD3, DOUT_SIZE, (pDesc)->word[3], (doutSize)); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD3, DOUT_LAST_IND, (pDesc)->word[3], lastInd); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD3, NS_BIT, (pDesc)->word[3], (axiNs)); \
- } while (0)
-
-/*!
- * This macro sets the DOUT field of a HW descriptors to DLLI type
- * The LAST INDICATION is provided by the user
- *
- * \param pDesc pointer HW descriptor struct
- * \param doutAdr DOUT address
- * \param doutSize Data size in bytes
- * \param lastInd The last indication bit
- * \param axiNs AXI secure bit
+static inline void set_din_not_last_indication(struct cc_hw_desc *pdesc)
+{
+ pdesc->word[1] |= FIELD_PREP(WORD1_NOT_LAST, 1);
+}
+
+/*
+ * Set the DOUT field of a HW descriptors
+ *
+ * @pdesc: pointer HW descriptor struct
+ * @dma_mode: The DMA mode: NO_DMA, SRAM, DLLI, MLLI, CONSTANT
+ * @addr: DOUT address
+ * @size: Data size in bytes
+ * @axi_sec: AXI secure bit
*/
-#define HW_DESC_SET_DOUT_MLLI(pDesc, doutAdr, doutSize, axiNs ,lastInd) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD2, VALUE, (pDesc)->word[2], (doutAdr)&UINT32_MAX ); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD5, DOUT_ADDR_HIGH, (pDesc)->word[5], MSB64(doutAdr) ); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD3, DOUT_DMA_MODE, (pDesc)->word[3], DMA_MLLI); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD3, DOUT_SIZE, (pDesc)->word[3], (doutSize)); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD3, DOUT_LAST_IND, (pDesc)->word[3], lastInd); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD3, NS_BIT, (pDesc)->word[3], (axiNs)); \
- } while (0)
-
-/*!
- * This macro sets the DOUT field of a HW descriptors to NO DMA mode. Used for NOP descriptor, register patches and
- * other special modes
- *
- * \param pDesc pointer HW descriptor struct
- * \param doutAdr DOUT address
- * \param doutSize Data size in bytes
- * \param registerWriteEnable Enables a write operation to a register
+static inline void set_dout_type(struct cc_hw_desc *pdesc,
+ enum cc_dma_mode dma_mode, dma_addr_t addr,
+ u32 size, enum cc_axi_sec axi_sec)
+{
+ pdesc->word[2] = (u32)addr;
+#ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT
+ pdesc->word[5] |= FIELD_PREP(WORD5_DOUT_ADDR_HIGH, ((u16)(addr >> 32)));
+#endif
+ pdesc->word[3] |= FIELD_PREP(WORD3_DOUT_DMA_MODE, dma_mode) |
+ FIELD_PREP(WORD3_DOUT_SIZE, size) |
+ FIELD_PREP(WORD3_NS_BIT, axi_sec);
+}
+
+/*
+ * Set the DOUT field of a HW descriptors to DLLI type
+ * The LAST INDICATION is provided by the user
+ *
+ * @pdesc pointer HW descriptor struct
+ * @addr: DOUT address
+ * @size: Data size in bytes
+ * @last_ind: The last indication bit
+ * @axi_sec: AXI secure bit
*/
-#define HW_DESC_SET_DOUT_NO_DMA(pDesc, doutAdr, doutSize, registerWriteEnable) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD2, VALUE, (pDesc)->word[2], (uint32_t)(doutAdr)); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD3, DOUT_SIZE, (pDesc)->word[3], (doutSize)); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD3, DOUT_LAST_IND, (pDesc)->word[3], (registerWriteEnable)); \
- } while (0)
-
-/*!
- * This macro sets the word for the XOR operation.
- *
- * \param pDesc pointer HW descriptor struct
- * \param xorVal xor data value
+static inline void set_dout_dlli(struct cc_hw_desc *pdesc, dma_addr_t addr,
+ u32 size, enum cc_axi_sec axi_sec,
+ u32 last_ind)
+{
+ set_dout_type(pdesc, DMA_DLLI, addr, size, axi_sec);
+ pdesc->word[3] |= FIELD_PREP(WORD3_DOUT_LAST_IND, last_ind);
+}
+
+/*
+ * Set the DOUT field of a HW descriptors to DLLI type
+ * The LAST INDICATION is provided by the user
+ *
+ * @pdesc: pointer HW descriptor struct
+ * @addr: DOUT address
+ * @size: Data size in bytes
+ * @last_ind: The last indication bit
+ * @axi_sec: AXI secure bit
*/
-#define HW_DESC_SET_XOR_VAL(pDesc, xorVal) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD2, VALUE, (pDesc)->word[2], (uint32_t)(xorVal)); \
- } while (0)
-
-/*!
- * This macro sets the XOR indicator bit in the descriptor
- *
- * \param pDesc pointer HW descriptor struct
+static inline void set_dout_mlli(struct cc_hw_desc *pdesc, dma_addr_t addr,
+ u32 size, enum cc_axi_sec axi_sec,
+ bool last_ind)
+{
+ set_dout_type(pdesc, DMA_MLLI, addr, size, axi_sec);
+ pdesc->word[3] |= FIELD_PREP(WORD3_DOUT_LAST_IND, last_ind);
+}
+
+/*
+ * Set the DOUT field of a HW descriptors to NO DMA mode.
+ * Used for NOP descriptor, register patches and other special modes.
+ *
+ * @pdesc: pointer HW descriptor struct
+ * @addr: DOUT address
+ * @size: Data size in bytes
+ * @write_enable: Enables a write operation to a register
*/
-#define HW_DESC_SET_XOR_ACTIVE(pDesc) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD3, HASH_XOR_BIT, (pDesc)->word[3], 1); \
- } while (0)
-
-/*!
- * This macro selects the AES engine instead of HASH engine when setting up combined mode with AES XCBC MAC
- *
- * \param pDesc pointer HW descriptor struct
+static inline void set_dout_no_dma(struct cc_hw_desc *pdesc, u32 addr,
+ u32 size, bool write_enable)
+{
+ pdesc->word[2] = addr;
+ pdesc->word[3] |= FIELD_PREP(WORD3_DOUT_SIZE, size) |
+ FIELD_PREP(WORD3_DOUT_LAST_IND, write_enable);
+}
+
+/*
+ * Set the word for the XOR operation.
+ *
+ * @pdesc: pointer HW descriptor struct
+ * @val: xor data value
*/
-#define HW_DESC_SET_AES_NOT_HASH_MODE(pDesc) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD4, AES_SEL_N_HASH, (pDesc)->word[4], 1); \
- } while (0)
-
-/*!
- * This macro sets the DOUT field of a HW descriptors to SRAM mode
- * Note: No need to check SRAM alignment since host requests do not use SRAM and
- * adaptor will enforce alignment check.
- *
- * \param pDesc pointer HW descriptor struct
- * \param doutAdr DOUT address
- * \param doutSize Data size in bytes
+static inline void set_xor_val(struct cc_hw_desc *pdesc, u32 val)
+{
+ pdesc->word[2] = val;
+}
+
+/*
+ * Sets the XOR indicator bit in the descriptor
+ *
+ * @pdesc: pointer HW descriptor struct
*/
-#define HW_DESC_SET_DOUT_SRAM(pDesc, doutAdr, doutSize) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD2, VALUE, (pDesc)->word[2], (uint32_t)(doutAdr)); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD3, DOUT_DMA_MODE, (pDesc)->word[3], DMA_SRAM); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD3, DOUT_SIZE, (pDesc)->word[3], (doutSize)); \
- } while (0)
-
-
-/*!
- * This macro sets the data unit size for XEX mode in data_out_addr[15:0]
- *
- * \param pDesc pointer HW descriptor struct
- * \param dataUnitSize data unit size for XEX mode
+static inline void set_xor_active(struct cc_hw_desc *pdesc)
+{
+ pdesc->word[3] |= FIELD_PREP(WORD3_HASH_XOR_BIT, 1);
+}
+
+/*
+ * Select the AES engine instead of HASH engine when setting up combined mode
+ * with AES XCBC MAC
+ *
+ * @pdesc: pointer HW descriptor struct
*/
-#define HW_DESC_SET_XEX_DATA_UNIT_SIZE(pDesc, dataUnitSize) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD2, VALUE, (pDesc)->word[2], (uint32_t)(dataUnitSize)); \
- } while (0)
+static inline void set_aes_not_hash_mode(struct cc_hw_desc *pdesc)
+{
+ pdesc->word[4] |= FIELD_PREP(WORD4_AES_SEL_N_HASH, 1);
+}
-/*!
- * This macro sets the number of rounds for Multi2 in data_out_addr[15:0]
+/*
+ * Set the DOUT field of a HW descriptors to SRAM mode
+ * Note: No need to check SRAM alignment since host requests do not use SRAM and
+ * adaptor will enforce alignment check.
*
- * \param pDesc pointer HW descriptor struct
- * \param numRounds number of rounds for Multi2
-*/
-#define HW_DESC_SET_MULTI2_NUM_ROUNDS(pDesc, numRounds) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD2, VALUE, (pDesc)->word[2], (uint32_t)(numRounds)); \
- } while (0)
-
-/*!
- * This macro sets the flow mode.
+ * @pdesc: pointer HW descriptor struct
+ * @addr: DOUT address
+ * @size: Data size in bytes
+ */
+static inline void set_dout_sram(struct cc_hw_desc *pdesc, u32 addr, u32 size)
+{
+ pdesc->word[2] = addr;
+ pdesc->word[3] |= FIELD_PREP(WORD3_DOUT_DMA_MODE, DMA_SRAM) |
+ FIELD_PREP(WORD3_DOUT_SIZE, size);
+}
+
+/*
+ * Sets the data unit size for XEX mode in data_out_addr[15:0]
*
- * \param pDesc pointer HW descriptor struct
- * \param flowMode Any one of the modes defined in [CC7x-DESC]
-*/
+ * @pdesc: pDesc pointer HW descriptor struct
+ * @size: data unit size for XEX mode
+ */
+static inline void set_xex_data_unit_size(struct cc_hw_desc *pdesc, u32 size)
+{
+ pdesc->word[2] = size;
+}
-#define HW_DESC_SET_FLOW_MODE(pDesc, flowMode) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD4, DATA_FLOW_MODE, (pDesc)->word[4], (flowMode)); \
- } while (0)
+/*
+ * Set the number of rounds for Multi2 in data_out_addr[15:0]
+ *
+ * @pdesc: pointer HW descriptor struct
+ * @num: number of rounds for Multi2
+ */
+static inline void set_multi2_num_rounds(struct cc_hw_desc *pdesc, u32 num)
+{
+ pdesc->word[2] = num;
+}
-/*!
- * This macro sets the cipher mode.
+/*
+ * Set the flow mode.
*
- * \param pDesc pointer HW descriptor struct
- * \param cipherMode Any one of the modes defined in [CC7x-DESC]
-*/
-#define HW_DESC_SET_CIPHER_MODE(pDesc, cipherMode) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD4, CIPHER_MODE, (pDesc)->word[4], (cipherMode)); \
- } while (0)
-
-/*!
- * This macro sets the cipher configuration fields.
+ * @pdesc: pointer HW descriptor struct
+ * @mode: Any one of the modes defined in [CC7x-DESC]
+ */
+static inline void set_flow_mode(struct cc_hw_desc *pdesc,
+ enum cc_flow_mode mode)
+{
+ pdesc->word[4] |= FIELD_PREP(WORD4_DATA_FLOW_MODE, mode);
+}
+
+/*
+ * Set the cipher mode.
*
- * \param pDesc pointer HW descriptor struct
- * \param cipherConfig Any one of the modes defined in [CC7x-DESC]
-*/
-#define HW_DESC_SET_CIPHER_CONFIG0(pDesc, cipherConfig) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD4, CIPHER_CONF0, (pDesc)->word[4], (cipherConfig)); \
- } while (0)
-
-/*!
- * This macro sets the cipher configuration fields.
+ * @pdesc: pointer HW descriptor struct
+ * @mode: Any one of the modes defined in [CC7x-DESC]
+ */
+static inline void set_cipher_mode(struct cc_hw_desc *pdesc,
+ enum drv_cipher_mode mode)
+{
+ pdesc->word[4] |= FIELD_PREP(WORD4_CIPHER_MODE, mode);
+}
+
+/*
+ * Set the cipher configuration fields.
*
- * \param pDesc pointer HW descriptor struct
- * \param cipherConfig Any one of the modes defined in [CC7x-DESC]
-*/
-#define HW_DESC_SET_CIPHER_CONFIG1(pDesc, cipherConfig) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD4, CIPHER_CONF1, (pDesc)->word[4], (cipherConfig)); \
- } while (0)
-
-/*!
- * This macro sets HW key configuration fields.
+ * @pdesc: pointer HW descriptor struct
+ * @mode: Any one of the modes defined in [CC7x-DESC]
+ */
+static inline void set_cipher_config0(struct cc_hw_desc *pdesc,
+ enum drv_crypto_direction mode)
+{
+ pdesc->word[4] |= FIELD_PREP(WORD4_CIPHER_CONF0, mode);
+}
+
+/*
+ * Set the cipher configuration fields.
*
- * \param pDesc pointer HW descriptor struct
- * \param hwKey The hw key number as in enun HwCryptoKey
-*/
-#define HW_DESC_SET_HW_CRYPTO_KEY(pDesc, hwKey) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD4, CIPHER_DO, (pDesc)->word[4], (hwKey)&HW_KEY_MASK_CIPHER_DO); \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD4, CIPHER_CONF2, (pDesc)->word[4], (hwKey>>HW_KEY_SHIFT_CIPHER_CFG2)); \
- } while (0)
-
-/*!
- * This macro changes the bytes order of all setup-finalize descriptosets.
+ * @pdesc: pointer HW descriptor struct
+ * @config: Any one of the modes defined in [CC7x-DESC]
+ */
+static inline void set_cipher_config1(struct cc_hw_desc *pdesc,
+ enum cc_hash_conf_pad config)
+{
+ pdesc->word[4] |= FIELD_PREP(WORD4_CIPHER_CONF1, config);
+}
+
+/*
+ * Set HW key configuration fields.
*
- * \param pDesc pointer HW descriptor struct
- * \param swapConfig Any one of the modes defined in [CC7x-DESC]
-*/
-#define HW_DESC_SET_BYTES_SWAP(pDesc, swapConfig) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD4, BYTES_SWAP, (pDesc)->word[4], (swapConfig)); \
- } while (0)
-
-/*!
- * This macro sets the CMAC_SIZE0 mode.
+ * @pdesc: pointer HW descriptor struct
+ * @hw_key: The HW key slot asdefined in enum cc_hw_crypto_key
+ */
+static inline void set_hw_crypto_key(struct cc_hw_desc *pdesc,
+ enum cc_hw_crypto_key hw_key)
+{
+ pdesc->word[4] |= FIELD_PREP(WORD4_CIPHER_DO,
+ (hw_key & HW_KEY_MASK_CIPHER_DO)) |
+ FIELD_PREP(WORD4_CIPHER_CONF2,
+ (hw_key >> HW_KEY_SHIFT_CIPHER_CFG2));
+}
+
+/*
+ * Set byte order of all setup-finalize descriptors.
*
- * \param pDesc pointer HW descriptor struct
-*/
-#define HW_DESC_SET_CMAC_SIZE0_MODE(pDesc) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD4, CMAC_SIZE0, (pDesc)->word[4], 0x1); \
- } while (0)
-
-/*!
- * This macro sets the key size for AES engine.
+ * @pdesc: pointer HW descriptor struct
+ * @config: Any one of the modes defined in [CC7x-DESC]
+ */
+static inline void set_bytes_swap(struct cc_hw_desc *pdesc, bool config)
+{
+ pdesc->word[4] |= FIELD_PREP(WORD4_BYTES_SWAP, config);
+}
+
+/*
+ * Set CMAC_SIZE0 mode.
*
- * \param pDesc pointer HW descriptor struct
- * \param keySize key size in bytes (NOT size code)
-*/
-#define HW_DESC_SET_KEY_SIZE_AES(pDesc, keySize) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD4, KEY_SIZE, (pDesc)->word[4], ((keySize) >> 3) - 2); \
- } while (0)
-
-/*!
- * This macro sets the key size for DES engine.
+ * @pdesc: pointer HW descriptor struct
+ */
+static inline void set_cmac_size0_mode(struct cc_hw_desc *pdesc)
+{
+ pdesc->word[4] |= FIELD_PREP(WORD4_CMAC_SIZE0, 1);
+}
+
+/*
+ * Set key size descriptor field.
*
- * \param pDesc pointer HW descriptor struct
- * \param keySize key size in bytes (NOT size code)
-*/
-#define HW_DESC_SET_KEY_SIZE_DES(pDesc, keySize) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD4, KEY_SIZE, (pDesc)->word[4], ((keySize) >> 3) - 1); \
- } while (0)
-
-/*!
- * This macro sets the descriptor's setup mode
+ * @pdesc: pointer HW descriptor struct
+ * @size: key size in bytes (NOT size code)
+ */
+static inline void set_key_size(struct cc_hw_desc *pdesc, u32 size)
+{
+ pdesc->word[4] |= FIELD_PREP(WORD4_KEY_SIZE, size);
+}
+
+/*
+ * Set AES key size.
*
- * \param pDesc pointer HW descriptor struct
- * \param setupMode Any one of the setup modes defined in [CC7x-DESC]
-*/
-#define HW_DESC_SET_SETUP_MODE(pDesc, setupMode) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD4, SETUP_OPERATION, (pDesc)->word[4], (setupMode)); \
- } while (0)
-
-/*!
- * This macro sets the descriptor's cipher do
+ * @pdesc: pointer HW descriptor struct
+ * @size: key size in bytes (NOT size code)
+ */
+static inline void set_key_size_aes(struct cc_hw_desc *pdesc, u32 size)
+{
+ set_key_size(pdesc, ((size >> 3) - 2));
+}
+
+/*
+ * Set DES key size.
*
- * \param pDesc pointer HW descriptor struct
- * \param cipherDo Any one of the cipher do defined in [CC7x-DESC]
-*/
-#define HW_DESC_SET_CIPHER_DO(pDesc, cipherDo) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_QUEUE_WORD4, CIPHER_DO, (pDesc)->word[4], (cipherDo)&HW_KEY_MASK_CIPHER_DO); \
- } while (0)
-
-/*!
- * This macro sets the DIN field of a HW descriptors to star/stop monitor descriptor.
- * Used for performance measurements and debug purposes.
- *
- * \param pDesc pointer HW descriptor struct
+ * @pdesc: pointer HW descriptor struct
+ * @size: key size in bytes (NOT size code)
*/
-#define HW_DESC_SET_DIN_MONITOR_CNTR(pDesc) \
- do { \
- CC_REG_FLD_SET(CRY_KERNEL, DSCRPTR_MEASURE_CNTR, VALUE, (pDesc)->word[1], _HW_DESC_MONITOR_KICK); \
- } while (0)
+static inline void set_key_size_des(struct cc_hw_desc *pdesc, u32 size)
+{
+ set_key_size(pdesc, ((size >> 3) - 1));
+}
+/*
+ * Set the descriptor setup mode
+ *
+ * @pdesc: pointer HW descriptor struct
+ * @mode: Any one of the setup modes defined in [CC7x-DESC]
+ */
+static inline void set_setup_mode(struct cc_hw_desc *pdesc,
+ enum cc_setup_op mode)
+{
+ pdesc->word[4] |= FIELD_PREP(WORD4_SETUP_OPERATION, mode);
+}
+/*
+ * Set the descriptor cipher DO
+ *
+ * @pdesc: pointer HW descriptor struct
+ * @config: Any one of the cipher do defined in [CC7x-DESC]
+ */
+static inline void set_cipher_do(struct cc_hw_desc *pdesc,
+ enum cc_hash_cipher_pad config)
+{
+ pdesc->word[4] |= FIELD_PREP(WORD4_CIPHER_DO,
+ (config & HW_KEY_MASK_CIPHER_DO));
+}
#endif /*__CC_HW_QUEUE_DEFS_H__*/
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
-
#ifndef _CC_LLI_DEFS_H_
#define _CC_LLI_DEFS_H_
-#ifdef __KERNEL__
-#include <linux/types.h>
-#else
-#include <stdint.h>
-#endif
-#include "cc_bitops.h"
-/* Max DLLI size */
-#define DLLI_SIZE_BIT_SIZE 0x18 // DX_DSCRPTR_QUEUE_WORD1_DIN_SIZE_BIT_SIZE
-
-#define CC_MAX_MLLI_ENTRY_SIZE 0x10000
+#include <linux/types.h>
-#define MSB64(_addr) (sizeof(_addr) == 4 ? 0 : ((_addr) >> 32)&UINT16_MAX)
+/* Max DLLI size
+ * AKA DX_DSCRPTR_QUEUE_WORD1_DIN_SIZE_BIT_SIZE
+ */
+#define DLLI_SIZE_BIT_SIZE 0x18
-#define LLI_SET_ADDR(lli_p, addr) \
- BITFIELD_SET(((uint32_t *)(lli_p))[LLI_WORD0_OFFSET], LLI_LADDR_BIT_OFFSET, LLI_LADDR_BIT_SIZE, (addr & UINT32_MAX)); \
- BITFIELD_SET(((uint32_t *)(lli_p))[LLI_WORD1_OFFSET], LLI_HADDR_BIT_OFFSET, LLI_HADDR_BIT_SIZE, MSB64(addr));
+#define CC_MAX_MLLI_ENTRY_SIZE 0xFFFF
-#define LLI_SET_SIZE(lli_p, size) \
- BITFIELD_SET(((uint32_t *)(lli_p))[LLI_WORD1_OFFSET], LLI_SIZE_BIT_OFFSET, LLI_SIZE_BIT_SIZE, size)
+#define LLI_MAX_NUM_OF_DATA_ENTRIES 128
+#define LLI_MAX_NUM_OF_ASSOC_DATA_ENTRIES 4
+#define MLLI_TABLE_MIN_ALIGNMENT 4 /* 32 bit alignment */
+#define MAX_NUM_OF_BUFFERS_IN_MLLI 4
+#define MAX_NUM_OF_TOTAL_MLLI_ENTRIES \
+ (2 * LLI_MAX_NUM_OF_DATA_ENTRIES + \
+ LLI_MAX_NUM_OF_ASSOC_DATA_ENTRIES)
/* Size of entry */
#define LLI_ENTRY_WORD_SIZE 2
-#define LLI_ENTRY_BYTE_SIZE (LLI_ENTRY_WORD_SIZE * sizeof(uint32_t))
+#define LLI_ENTRY_BYTE_SIZE (LLI_ENTRY_WORD_SIZE * sizeof(u32))
/* Word0[31:0] = ADDR[31:0] */
#define LLI_WORD0_OFFSET 0
#define LLI_HADDR_BIT_OFFSET 16
#define LLI_HADDR_BIT_SIZE 16
+#define LLI_SIZE_MASK GENMASK((LLI_SIZE_BIT_SIZE - 1), LLI_SIZE_BIT_OFFSET)
+#define LLI_HADDR_MASK GENMASK( \
+ (LLI_HADDR_BIT_OFFSET + LLI_HADDR_BIT_SIZE - 1),\
+ LLI_HADDR_BIT_OFFSET)
+
+static inline void cc_lli_set_addr(u32 *lli_p, dma_addr_t addr)
+{
+ lli_p[LLI_WORD0_OFFSET] = (addr & U32_MAX);
+#ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT
+ lli_p[LLI_WORD1_OFFSET] &= ~LLI_HADDR_MASK;
+ lli_p[LLI_WORD1_OFFSET] |= FIELD_PREP(LLI_HADDR_MASK, (addr >> 16));
+#endif /* CONFIG_ARCH_DMA_ADDR_T_64BIT */
+}
+
+static inline void cc_lli_set_size(u32 *lli_p, u16 size)
+{
+ lli_p[LLI_WORD1_OFFSET] &= ~LLI_SIZE_MASK;
+ lli_p[LLI_WORD1_OFFSET] |= FIELD_PREP(LLI_SIZE_MASK, size);
+}
#endif /*_CC_LLI_DEFS_H_*/
+++ /dev/null
-/*
- * Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, see <http://www.gnu.org/licenses/>.
- */
-
-#ifndef _CC_PAL_LOG_H_
-#define _CC_PAL_LOG_H_
-
-#include "cc_pal_types.h"
-#include "cc_pal_log_plat.h"
-
-/*!
-@file
-@brief This file contains the PAL layer log definitions, by default the log is disabled.
-@defgroup cc_pal_log CryptoCell PAL logging APIs and definitions
-@{
-@ingroup cc_pal
-*/
-
-/* PAL log levels (to be used in CC_PAL_logLevel) */
-/*! PAL log level - disabled. */
-#define CC_PAL_LOG_LEVEL_NULL (-1) /*!< \internal Disable logging */
-/*! PAL log level - error. */
-#define CC_PAL_LOG_LEVEL_ERR 0
-/*! PAL log level - warning. */
-#define CC_PAL_LOG_LEVEL_WARN 1
-/*! PAL log level - info. */
-#define CC_PAL_LOG_LEVEL_INFO 2
-/*! PAL log level - debug. */
-#define CC_PAL_LOG_LEVEL_DEBUG 3
-/*! PAL log level - trace. */
-#define CC_PAL_LOG_LEVEL_TRACE 4
-/*! PAL log level - data. */
-#define CC_PAL_LOG_LEVEL_DATA 5
-
-#ifndef CC_PAL_LOG_CUR_COMPONENT
-/* Setting default component mask in case caller did not define */
-/* (a mask that is always on for every log mask value but full masking) */
-/*! Default log debugged component.*/
-#define CC_PAL_LOG_CUR_COMPONENT 0xFFFFFFFF
-#endif
-#ifndef CC_PAL_LOG_CUR_COMPONENT_NAME
-/*! Default log debugged component.*/
-#define CC_PAL_LOG_CUR_COMPONENT_NAME "CC"
-#endif
-
-/* Select compile time log level (default if not explicitly specified by caller) */
-#ifndef CC_PAL_MAX_LOG_LEVEL /* Can be overriden by external definition of this constant */
-#ifdef DEBUG
-/*! Default debug log level (when debug is set to on).*/
-#define CC_PAL_MAX_LOG_LEVEL CC_PAL_LOG_LEVEL_ERR /*CC_PAL_LOG_LEVEL_DEBUG*/
-#else /* Disable logging */
-/*! Default debug log level (when debug is set to on).*/
-#define CC_PAL_MAX_LOG_LEVEL CC_PAL_LOG_LEVEL_NULL
-#endif
-#endif /*CC_PAL_MAX_LOG_LEVEL*/
-/*! Evaluate CC_PAL_MAX_LOG_LEVEL in case provided by caller */
-#define __CC_PAL_LOG_LEVEL_EVAL(level) level
-/*! Maximal log level defintion.*/
-#define _CC_PAL_MAX_LOG_LEVEL __CC_PAL_LOG_LEVEL_EVAL(CC_PAL_MAX_LOG_LEVEL)
-
-
-#ifdef ARM_DSM
-/*! Log init function. */
-#define CC_PalLogInit() do {} while (0)
-/*! Log set level function - sets the level of logging in case of debug. */
-#define CC_PalLogLevelSet(setLevel) do {} while (0)
-/*! Log set mask function - sets the component masking in case of debug. */
-#define CC_PalLogMaskSet(setMask) do {} while (0)
-#else
-#if _CC_PAL_MAX_LOG_LEVEL > CC_PAL_LOG_LEVEL_NULL
-/*! Log init function. */
-void CC_PalLogInit(void);
-/*! Log set level function - sets the level of logging in case of debug. */
-void CC_PalLogLevelSet(int setLevel);
-/*! Log set mask function - sets the component masking in case of debug. */
-void CC_PalLogMaskSet(uint32_t setMask);
-/*! Global variable for log level */
-extern int CC_PAL_logLevel;
-/*! Global variable for log mask */
-extern uint32_t CC_PAL_logMask;
-#else /* No log */
-/*! Log init function. */
-static inline void CC_PalLogInit(void) {}
-/*! Log set level function - sets the level of logging in case of debug. */
-static inline void CC_PalLogLevelSet(int setLevel) {CC_UNUSED_PARAM(setLevel);}
-/*! Log set mask function - sets the component masking in case of debug. */
-static inline void CC_PalLogMaskSet(uint32_t setMask) {CC_UNUSED_PARAM(setMask);}
-#endif
-#endif
-
-/*! Filter logging based on logMask and dispatch to platform specific logging mechanism. */
-#define _CC_PAL_LOG(level, format, ...) \
- if (CC_PAL_logMask & CC_PAL_LOG_CUR_COMPONENT) \
- __CC_PAL_LOG_PLAT(CC_PAL_LOG_LEVEL_ ## level, "%s:%s: " format, CC_PAL_LOG_CUR_COMPONENT_NAME, __func__, ##__VA_ARGS__)
-
-#if (_CC_PAL_MAX_LOG_LEVEL >= CC_PAL_LOG_LEVEL_ERR)
-/*! Log messages according to log level.*/
-#define CC_PAL_LOG_ERR(format, ... ) \
- _CC_PAL_LOG(ERR, format, ##__VA_ARGS__)
-#else
-/*! Log messages according to log level.*/
-#define CC_PAL_LOG_ERR( ... ) do {} while (0)
-#endif
-
-#if (_CC_PAL_MAX_LOG_LEVEL >= CC_PAL_LOG_LEVEL_WARN)
-/*! Log messages according to log level.*/
-#define CC_PAL_LOG_WARN(format, ... ) \
- if (CC_PAL_logLevel >= CC_PAL_LOG_LEVEL_WARN) \
- _CC_PAL_LOG(WARN, format, ##__VA_ARGS__)
-#else
-/*! Log messages according to log level.*/
-#define CC_PAL_LOG_WARN( ... ) do {} while (0)
-#endif
-
-#if (_CC_PAL_MAX_LOG_LEVEL >= CC_PAL_LOG_LEVEL_INFO)
-/*! Log messages according to log level.*/
-#define CC_PAL_LOG_INFO(format, ... ) \
- if (CC_PAL_logLevel >= CC_PAL_LOG_LEVEL_INFO) \
- _CC_PAL_LOG(INFO, format, ##__VA_ARGS__)
-#else
-/*! Log messages according to log level.*/
-#define CC_PAL_LOG_INFO( ... ) do {} while (0)
-#endif
-
-#if (_CC_PAL_MAX_LOG_LEVEL >= CC_PAL_LOG_LEVEL_DEBUG)
-/*! Log messages according to log level.*/
-#define CC_PAL_LOG_DEBUG(format, ... ) \
- if (CC_PAL_logLevel >= CC_PAL_LOG_LEVEL_DEBUG) \
- _CC_PAL_LOG(DEBUG, format, ##__VA_ARGS__)
-
-/*! Log message buffer.*/
-#define CC_PAL_LOG_DUMP_BUF(msg, buf, size) \
- do { \
- int i; \
- uint8_t *pData = (uint8_t*)buf; \
- \
- PRINTF("%s (%d):\n", msg, size); \
- for (i = 0; i < size; i++) { \
- PRINTF("0x%02X ", pData[i]); \
- if ((i & 0xF) == 0xF) { \
- PRINTF("\n"); \
- } \
- } \
- PRINTF("\n"); \
- } while (0)
-#else
-/*! Log debug messages.*/
-#define CC_PAL_LOG_DEBUG( ... ) do {} while (0)
-/*! Log debug buffer.*/
-#define CC_PAL_LOG_DUMP_BUF(msg, buf, size) do {} while (0)
-#endif
-
-#if (_CC_PAL_MAX_LOG_LEVEL >= CC_PAL_LOG_LEVEL_TRACE)
-/*! Log debug trace.*/
-#define CC_PAL_LOG_TRACE(format, ... ) \
- if (CC_PAL_logLevel >= CC_PAL_LOG_LEVEL_TRACE) \
- _CC_PAL_LOG(TRACE, format, ##__VA_ARGS__)
-#else
-/*! Log debug trace.*/
-#define CC_PAL_LOG_TRACE(...) do {} while (0)
-#endif
-
-#if (_CC_PAL_MAX_LOG_LEVEL >= CC_PAL_LOG_LEVEL_TRACE)
-/*! Log debug data.*/
-#define CC_PAL_LOG_DATA(format, ...) \
- if (CC_PAL_logLevel >= CC_PAL_LOG_LEVEL_TRACE) \
- _CC_PAL_LOG(DATA, format, ##__VA_ARGS__)
-#else
-/*! Log debug data.*/
-#define CC_PAL_LOG_DATA( ...) do {} while (0)
-#endif
-/**
-@}
- */
-
-#endif /*_CC_PAL_LOG_H_*/
+++ /dev/null
-/*
- * Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, see <http://www.gnu.org/licenses/>.
- */
-
-/* Dummy pal_log_plat for test driver in kernel */
-
-#ifndef _SSI_PAL_LOG_PLAT_H_
-#define _SSI_PAL_LOG_PLAT_H_
-
-#if defined(DEBUG)
-
-#define __CC_PAL_LOG_PLAT(level, format, ...) printk(level "cc7x_test::" format , ##__VA_ARGS__)
-
-#else /* Disable all prints */
-
-#define __CC_PAL_LOG_PLAT(...) do {} while (0)
-
-#endif
-
-#endif /*_SASI_PAL_LOG_PLAT_H_*/
-
+++ /dev/null
-/*
- * Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, see <http://www.gnu.org/licenses/>.
- */
-
-#ifndef CC_PAL_TYPES_H
-#define CC_PAL_TYPES_H
-
-/*!
-@file
-@brief This file contains platform-dependent definitions and types.
-@defgroup cc_pal_types CryptoCell PAL platform dependant types
-@{
-@ingroup cc_pal
-
-*/
-
-#include "cc_pal_types_plat.h"
-
-/*! Boolean definition.*/
-typedef enum {
- /*! Boolean false definition.*/
- CC_FALSE = 0,
- /*! Boolean true definition.*/
- CC_TRUE = 1
-} CCBool;
-
-/*! Success definition. */
-#define CC_SUCCESS 0UL
-/*! Failure definition. */
-#define CC_FAIL 1UL
-
-/*! Defintion of 1KB in bytes. */
-#define CC_1K_SIZE_IN_BYTES 1024
-/*! Defintion of number of bits in a byte. */
-#define CC_BITS_IN_BYTE 8
-/*! Defintion of number of bits in a 32bits word. */
-#define CC_BITS_IN_32BIT_WORD 32
-/*! Defintion of number of bytes in a 32bits word. */
-#define CC_32BIT_WORD_SIZE (sizeof(uint32_t))
-
-/*! Success (OK) defintion. */
-#define CC_OK 0
-
-/*! Macro that handles unused parameters in the code (to avoid compilation warnings). */
-#define CC_UNUSED_PARAM(prm) ((void)prm)
-
-/*! Maximal uint32 value.*/
-#define CC_MAX_UINT32_VAL (0xFFFFFFFF)
-
-
-/* Minimum and Maximum macros */
-#ifdef min
-/*! Definition for minimum. */
-#define CC_MIN(a,b) min( a , b )
-#else
-/*! Definition for minimum. */
-#define CC_MIN( a , b ) ( ( (a) < (b) ) ? (a) : (b) )
-#endif
-
-#ifdef max
-/*! Definition for maximum. */
-#define CC_MAX(a,b) max( a , b )
-#else
-/*! Definition for maximum. */
-#define CC_MAX( a , b ) ( ( (a) > (b) ) ? (a) : (b) )
-#endif
-
-/*! Macro that calculates number of full bytes from bits (i.e. 7 bits are 1 byte). */
-#define CALC_FULL_BYTES(numBits) ((numBits)/CC_BITS_IN_BYTE + (((numBits) & (CC_BITS_IN_BYTE-1)) > 0))
-/*! Macro that calculates number of full 32bits words from bits (i.e. 31 bits are 1 word). */
-#define CALC_FULL_32BIT_WORDS(numBits) ((numBits)/CC_BITS_IN_32BIT_WORD + (((numBits) & (CC_BITS_IN_32BIT_WORD-1)) > 0))
-/*! Macro that calculates number of full 32bits words from bytes (i.e. 3 bytes are 1 word). */
-#define CALC_32BIT_WORDS_FROM_BYTES(sizeBytes) ((sizeBytes)/CC_32BIT_WORD_SIZE + (((sizeBytes) & (CC_32BIT_WORD_SIZE-1)) > 0))
-/*! Macro that round up bits to 32bits words. */
-#define ROUNDUP_BITS_TO_32BIT_WORD(numBits) (CALC_FULL_32BIT_WORDS(numBits) * CC_BITS_IN_32BIT_WORD)
-/*! Macro that round up bits to bytes. */
-#define ROUNDUP_BITS_TO_BYTES(numBits) (CALC_FULL_BYTES(numBits) * CC_BITS_IN_BYTE)
-/*! Macro that round up bytes to 32bits words. */
-#define ROUNDUP_BYTES_TO_32BIT_WORD(sizeBytes) (CALC_32BIT_WORDS_FROM_BYTES(sizeBytes) * CC_32BIT_WORD_SIZE)
-
-
-/**
-@}
- */
-#endif
+++ /dev/null
-/*
- * Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, see <http://www.gnu.org/licenses/>.
- */
-
-
-#ifndef SSI_PAL_TYPES_PLAT_H
-#define SSI_PAL_TYPES_PLAT_H
-/* Linux kernel types */
-
-#include <linux/types.h>
-
-#ifndef NULL /* Missing in Linux kernel */
-#define NULL (0x0L)
-#endif
-
-
-#endif /*SSI_PAL_TYPES_PLAT_H*/
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
-
/*!
- * @file
- * @brief This file contains macro definitions for accessing ARM TrustZone CryptoCell register space.
+ * @file
+ * @brief This file contains macro definitions for accessing ARM TrustZone
+ * CryptoCell register space.
*/
#ifndef _CC_REGS_H_
#define _CC_REGS_H_
-#include "cc_bitops.h"
+#include <linux/bitfield.h>
+
+#define AXIM_MON_BASE_OFFSET CC_REG_OFFSET(CRY_KERNEL, AXIM_MON_COMP)
+#define AXIM_MON_COMP_VALUE GENMASK(DX_AXIM_MON_COMP_VALUE_BIT_SIZE + \
+ DX_AXIM_MON_COMP_VALUE_BIT_SHIFT, \
+ DX_AXIM_MON_COMP_VALUE_BIT_SHIFT)
+
+#define AXIM_MON_BASE_OFFSET CC_REG_OFFSET(CRY_KERNEL, AXIM_MON_COMP)
+#define AXIM_MON_COMP_VALUE GENMASK(DX_AXIM_MON_COMP_VALUE_BIT_SIZE + \
+ DX_AXIM_MON_COMP_VALUE_BIT_SHIFT, \
+ DX_AXIM_MON_COMP_VALUE_BIT_SHIFT)
/* Register Offset macro */
#define CC_REG_OFFSET(unit_name, reg_name) \
(DX_BASE_ ## unit_name + DX_ ## reg_name ## _REG_OFFSET)
-#define CC_REG_BIT_SHIFT(reg_name, field_name) \
- (DX_ ## reg_name ## _ ## field_name ## _BIT_SHIFT)
-
-/* Register Offset macros (from registers base address in host) */
-#include "dx_reg_base_host.h"
-
-/* Read-Modify-Write a field of a register */
-#define MODIFY_REGISTER_FLD(unitName, regName, fldName, fldVal) \
-do { \
- uint32_t regVal; \
- regVal = READ_REGISTER(CC_REG_ADDR(unitName, regName)); \
- CC_REG_FLD_SET(unitName, regName, fldName, regVal, fldVal); \
- WRITE_REGISTER(CC_REG_ADDR(unitName, regName), regVal); \
-} while (0)
-
-/* Registers address macros for ENV registers (development FPGA only) */
-#ifdef DX_BASE_ENV_REGS
-
-/* This offset should be added to mapping address of DX_BASE_ENV_REGS */
-#define CC_ENV_REG_OFFSET(reg_name) (DX_ENV_ ## reg_name ## _REG_OFFSET)
-
-#endif /*DX_BASE_ENV_REGS*/
-
-/*! Bit fields get */
-#define CC_REG_FLD_GET(unit_name, reg_name, fld_name, reg_val) \
- (DX_ ## reg_name ## _ ## fld_name ## _BIT_SIZE == 0x20 ? \
- reg_val /*!< \internal Optimization for 32b fields */ : \
- BITFIELD_GET(reg_val, DX_ ## reg_name ## _ ## fld_name ## _BIT_SHIFT, \
- DX_ ## reg_name ## _ ## fld_name ## _BIT_SIZE))
-
-/*! Bit fields access */
-#define CC_REG_FLD_GET2(unit_name, reg_name, fld_name, reg_val) \
- (CC_ ## reg_name ## _ ## fld_name ## _BIT_SIZE == 0x20 ? \
- reg_val /*!< \internal Optimization for 32b fields */ : \
- BITFIELD_GET(reg_val, CC_ ## reg_name ## _ ## fld_name ## _BIT_SHIFT, \
- CC_ ## reg_name ## _ ## fld_name ## _BIT_SIZE))
-
-/* yael TBD !!! - *
-* all HW includes should start with CC_ and not DX_ !! */
-
-
-/*! Bit fields set */
-#define CC_REG_FLD_SET( \
- unit_name, reg_name, fld_name, reg_shadow_var, new_fld_val) \
-do { \
- if (DX_ ## reg_name ## _ ## fld_name ## _BIT_SIZE == 0x20) \
- reg_shadow_var = new_fld_val; /*!< \internal Optimization for 32b fields */\
- else \
- BITFIELD_SET(reg_shadow_var, \
- DX_ ## reg_name ## _ ## fld_name ## _BIT_SHIFT, \
- DX_ ## reg_name ## _ ## fld_name ## _BIT_SIZE, \
- new_fld_val); \
-} while (0)
-
-/*! Bit fields set */
-#define CC_REG_FLD_SET2( \
- unit_name, reg_name, fld_name, reg_shadow_var, new_fld_val) \
-do { \
- if (CC_ ## reg_name ## _ ## fld_name ## _BIT_SIZE == 0x20) \
- reg_shadow_var = new_fld_val; /*!< \internal Optimization for 32b fields */\
- else \
- BITFIELD_SET(reg_shadow_var, \
- CC_ ## reg_name ## _ ## fld_name ## _BIT_SHIFT, \
- CC_ ## reg_name ## _ ## fld_name ## _BIT_SIZE, \
- new_fld_val); \
-} while (0)
-
-/* Usage example:
- uint32_t reg_shadow = READ_REGISTER(CC_REG_ADDR(CRY_KERNEL,AES_CONTROL));
- CC_REG_FLD_SET(CRY_KERNEL,AES_CONTROL,NK_KEY0,reg_shadow, 3);
- CC_REG_FLD_SET(CRY_KERNEL,AES_CONTROL,NK_KEY1,reg_shadow, 1);
- WRITE_REGISTER(CC_REG_ADDR(CRY_KERNEL,AES_CONTROL), reg_shadow);
- */
-
#endif /*_CC_REGS_H_*/
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
// --------------------------------------
// BLOCK: DSCRPTR
// --------------------------------------
-#define DX_DSCRPTR_COMPLETION_COUNTER_REG_OFFSET 0xE00UL
-#define DX_DSCRPTR_COMPLETION_COUNTER_COMPLETION_COUNTER_BIT_SHIFT 0x0UL
-#define DX_DSCRPTR_COMPLETION_COUNTER_COMPLETION_COUNTER_BIT_SIZE 0x6UL
-#define DX_DSCRPTR_COMPLETION_COUNTER_OVERFLOW_COUNTER_BIT_SHIFT 0x6UL
-#define DX_DSCRPTR_COMPLETION_COUNTER_OVERFLOW_COUNTER_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_SW_RESET_REG_OFFSET 0xE40UL
-#define DX_DSCRPTR_SW_RESET_VALUE_BIT_SHIFT 0x0UL
-#define DX_DSCRPTR_SW_RESET_VALUE_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_QUEUE_SRAM_SIZE_REG_OFFSET 0xE60UL
-#define DX_DSCRPTR_QUEUE_SRAM_SIZE_NUM_OF_DSCRPTR_BIT_SHIFT 0x0UL
-#define DX_DSCRPTR_QUEUE_SRAM_SIZE_NUM_OF_DSCRPTR_BIT_SIZE 0xAUL
-#define DX_DSCRPTR_QUEUE_SRAM_SIZE_DSCRPTR_SRAM_SIZE_BIT_SHIFT 0xAUL
-#define DX_DSCRPTR_QUEUE_SRAM_SIZE_DSCRPTR_SRAM_SIZE_BIT_SIZE 0xCUL
-#define DX_DSCRPTR_QUEUE_SRAM_SIZE_SRAM_SIZE_BIT_SHIFT 0x16UL
-#define DX_DSCRPTR_QUEUE_SRAM_SIZE_SRAM_SIZE_BIT_SIZE 0x3UL
-#define DX_DSCRPTR_SINGLE_ADDR_EN_REG_OFFSET 0xE64UL
-#define DX_DSCRPTR_SINGLE_ADDR_EN_VALUE_BIT_SHIFT 0x0UL
-#define DX_DSCRPTR_SINGLE_ADDR_EN_VALUE_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_MEASURE_CNTR_REG_OFFSET 0xE68UL
-#define DX_DSCRPTR_MEASURE_CNTR_VALUE_BIT_SHIFT 0x0UL
-#define DX_DSCRPTR_MEASURE_CNTR_VALUE_BIT_SIZE 0x20UL
-#define DX_DSCRPTR_QUEUE_WORD0_REG_OFFSET 0xE80UL
-#define DX_DSCRPTR_QUEUE_WORD0_VALUE_BIT_SHIFT 0x0UL
-#define DX_DSCRPTR_QUEUE_WORD0_VALUE_BIT_SIZE 0x20UL
-#define DX_DSCRPTR_QUEUE_WORD1_REG_OFFSET 0xE84UL
-#define DX_DSCRPTR_QUEUE_WORD1_DIN_DMA_MODE_BIT_SHIFT 0x0UL
-#define DX_DSCRPTR_QUEUE_WORD1_DIN_DMA_MODE_BIT_SIZE 0x2UL
-#define DX_DSCRPTR_QUEUE_WORD1_DIN_SIZE_BIT_SHIFT 0x2UL
-#define DX_DSCRPTR_QUEUE_WORD1_DIN_SIZE_BIT_SIZE 0x18UL
-#define DX_DSCRPTR_QUEUE_WORD1_NS_BIT_BIT_SHIFT 0x1AUL
-#define DX_DSCRPTR_QUEUE_WORD1_NS_BIT_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_QUEUE_WORD1_DIN_CONST_VALUE_BIT_SHIFT 0x1BUL
-#define DX_DSCRPTR_QUEUE_WORD1_DIN_CONST_VALUE_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_QUEUE_WORD1_NOT_LAST_BIT_SHIFT 0x1CUL
-#define DX_DSCRPTR_QUEUE_WORD1_NOT_LAST_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_QUEUE_WORD1_LOCK_QUEUE_BIT_SHIFT 0x1DUL
-#define DX_DSCRPTR_QUEUE_WORD1_LOCK_QUEUE_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_QUEUE_WORD1_NOT_USED_BIT_SHIFT 0x1EUL
-#define DX_DSCRPTR_QUEUE_WORD1_NOT_USED_BIT_SIZE 0x2UL
-#define DX_DSCRPTR_QUEUE_WORD2_REG_OFFSET 0xE88UL
-#define DX_DSCRPTR_QUEUE_WORD2_VALUE_BIT_SHIFT 0x0UL
-#define DX_DSCRPTR_QUEUE_WORD2_VALUE_BIT_SIZE 0x20UL
-#define DX_DSCRPTR_QUEUE_WORD3_REG_OFFSET 0xE8CUL
-#define DX_DSCRPTR_QUEUE_WORD3_DOUT_DMA_MODE_BIT_SHIFT 0x0UL
-#define DX_DSCRPTR_QUEUE_WORD3_DOUT_DMA_MODE_BIT_SIZE 0x2UL
-#define DX_DSCRPTR_QUEUE_WORD3_DOUT_SIZE_BIT_SHIFT 0x2UL
-#define DX_DSCRPTR_QUEUE_WORD3_DOUT_SIZE_BIT_SIZE 0x18UL
-#define DX_DSCRPTR_QUEUE_WORD3_NS_BIT_BIT_SHIFT 0x1AUL
-#define DX_DSCRPTR_QUEUE_WORD3_NS_BIT_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_QUEUE_WORD3_DOUT_LAST_IND_BIT_SHIFT 0x1BUL
-#define DX_DSCRPTR_QUEUE_WORD3_DOUT_LAST_IND_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_QUEUE_WORD3_HASH_XOR_BIT_BIT_SHIFT 0x1DUL
-#define DX_DSCRPTR_QUEUE_WORD3_HASH_XOR_BIT_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_QUEUE_WORD3_NOT_USED_BIT_SHIFT 0x1EUL
-#define DX_DSCRPTR_QUEUE_WORD3_NOT_USED_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_QUEUE_WORD3_QUEUE_LAST_IND_BIT_SHIFT 0x1FUL
-#define DX_DSCRPTR_QUEUE_WORD3_QUEUE_LAST_IND_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_QUEUE_WORD4_REG_OFFSET 0xE90UL
-#define DX_DSCRPTR_QUEUE_WORD4_DATA_FLOW_MODE_BIT_SHIFT 0x0UL
-#define DX_DSCRPTR_QUEUE_WORD4_DATA_FLOW_MODE_BIT_SIZE 0x6UL
-#define DX_DSCRPTR_QUEUE_WORD4_AES_SEL_N_HASH_BIT_SHIFT 0x6UL
-#define DX_DSCRPTR_QUEUE_WORD4_AES_SEL_N_HASH_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_QUEUE_WORD4_AES_XOR_CRYPTO_KEY_BIT_SHIFT 0x7UL
-#define DX_DSCRPTR_QUEUE_WORD4_AES_XOR_CRYPTO_KEY_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_QUEUE_WORD4_ACK_NEEDED_BIT_SHIFT 0x8UL
-#define DX_DSCRPTR_QUEUE_WORD4_ACK_NEEDED_BIT_SIZE 0x2UL
-#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_MODE_BIT_SHIFT 0xAUL
-#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_MODE_BIT_SIZE 0x4UL
-#define DX_DSCRPTR_QUEUE_WORD4_CMAC_SIZE0_BIT_SHIFT 0xEUL
-#define DX_DSCRPTR_QUEUE_WORD4_CMAC_SIZE0_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_DO_BIT_SHIFT 0xFUL
-#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_DO_BIT_SIZE 0x2UL
-#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_CONF0_BIT_SHIFT 0x11UL
-#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_CONF0_BIT_SIZE 0x2UL
-#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_CONF1_BIT_SHIFT 0x13UL
-#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_CONF1_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_CONF2_BIT_SHIFT 0x14UL
-#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_CONF2_BIT_SIZE 0x2UL
-#define DX_DSCRPTR_QUEUE_WORD4_KEY_SIZE_BIT_SHIFT 0x16UL
-#define DX_DSCRPTR_QUEUE_WORD4_KEY_SIZE_BIT_SIZE 0x2UL
-#define DX_DSCRPTR_QUEUE_WORD4_SETUP_OPERATION_BIT_SHIFT 0x18UL
-#define DX_DSCRPTR_QUEUE_WORD4_SETUP_OPERATION_BIT_SIZE 0x4UL
-#define DX_DSCRPTR_QUEUE_WORD4_DIN_SRAM_ENDIANNESS_BIT_SHIFT 0x1CUL
-#define DX_DSCRPTR_QUEUE_WORD4_DIN_SRAM_ENDIANNESS_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_QUEUE_WORD4_DOUT_SRAM_ENDIANNESS_BIT_SHIFT 0x1DUL
-#define DX_DSCRPTR_QUEUE_WORD4_DOUT_SRAM_ENDIANNESS_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_QUEUE_WORD4_WORD_SWAP_BIT_SHIFT 0x1EUL
-#define DX_DSCRPTR_QUEUE_WORD4_WORD_SWAP_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_QUEUE_WORD4_BYTES_SWAP_BIT_SHIFT 0x1FUL
-#define DX_DSCRPTR_QUEUE_WORD4_BYTES_SWAP_BIT_SIZE 0x1UL
-#define DX_DSCRPTR_QUEUE_WORD5_REG_OFFSET 0xE94UL
-#define DX_DSCRPTR_QUEUE_WORD5_DIN_ADDR_HIGH_BIT_SHIFT 0x0UL
-#define DX_DSCRPTR_QUEUE_WORD5_DIN_ADDR_HIGH_BIT_SIZE 0x10UL
-#define DX_DSCRPTR_QUEUE_WORD5_DOUT_ADDR_HIGH_BIT_SHIFT 0x10UL
-#define DX_DSCRPTR_QUEUE_WORD5_DOUT_ADDR_HIGH_BIT_SIZE 0x10UL
-#define DX_DSCRPTR_QUEUE_WATERMARK_REG_OFFSET 0xE98UL
-#define DX_DSCRPTR_QUEUE_WATERMARK_VALUE_BIT_SHIFT 0x0UL
-#define DX_DSCRPTR_QUEUE_WATERMARK_VALUE_BIT_SIZE 0xAUL
-#define DX_DSCRPTR_QUEUE_CONTENT_REG_OFFSET 0xE9CUL
-#define DX_DSCRPTR_QUEUE_CONTENT_VALUE_BIT_SHIFT 0x0UL
-#define DX_DSCRPTR_QUEUE_CONTENT_VALUE_BIT_SIZE 0xAUL
+#define DX_DSCRPTR_COMPLETION_COUNTER_REG_OFFSET 0xE00UL
+#define DX_DSCRPTR_COMPLETION_COUNTER_COMPLETION_COUNTER_BIT_SHIFT 0x0UL
+#define DX_DSCRPTR_COMPLETION_COUNTER_COMPLETION_COUNTER_BIT_SIZE 0x6UL
+#define DX_DSCRPTR_COMPLETION_COUNTER_OVERFLOW_COUNTER_BIT_SHIFT 0x6UL
+#define DX_DSCRPTR_COMPLETION_COUNTER_OVERFLOW_COUNTER_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_SW_RESET_REG_OFFSET 0xE40UL
+#define DX_DSCRPTR_SW_RESET_VALUE_BIT_SHIFT 0x0UL
+#define DX_DSCRPTR_SW_RESET_VALUE_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_QUEUE_SRAM_SIZE_REG_OFFSET 0xE60UL
+#define DX_DSCRPTR_QUEUE_SRAM_SIZE_NUM_OF_DSCRPTR_BIT_SHIFT 0x0UL
+#define DX_DSCRPTR_QUEUE_SRAM_SIZE_NUM_OF_DSCRPTR_BIT_SIZE 0xAUL
+#define DX_DSCRPTR_QUEUE_SRAM_SIZE_DSCRPTR_SRAM_SIZE_BIT_SHIFT 0xAUL
+#define DX_DSCRPTR_QUEUE_SRAM_SIZE_DSCRPTR_SRAM_SIZE_BIT_SIZE 0xCUL
+#define DX_DSCRPTR_QUEUE_SRAM_SIZE_SRAM_SIZE_BIT_SHIFT 0x16UL
+#define DX_DSCRPTR_QUEUE_SRAM_SIZE_SRAM_SIZE_BIT_SIZE 0x3UL
+#define DX_DSCRPTR_SINGLE_ADDR_EN_REG_OFFSET 0xE64UL
+#define DX_DSCRPTR_SINGLE_ADDR_EN_VALUE_BIT_SHIFT 0x0UL
+#define DX_DSCRPTR_SINGLE_ADDR_EN_VALUE_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_MEASURE_CNTR_REG_OFFSET 0xE68UL
+#define DX_DSCRPTR_MEASURE_CNTR_VALUE_BIT_SHIFT 0x0UL
+#define DX_DSCRPTR_MEASURE_CNTR_VALUE_BIT_SIZE 0x20UL
+#define DX_DSCRPTR_QUEUE_WORD0_REG_OFFSET 0xE80UL
+#define DX_DSCRPTR_QUEUE_WORD0_VALUE_BIT_SHIFT 0x0UL
+#define DX_DSCRPTR_QUEUE_WORD0_VALUE_BIT_SIZE 0x20UL
+#define DX_DSCRPTR_QUEUE_WORD1_REG_OFFSET 0xE84UL
+#define DX_DSCRPTR_QUEUE_WORD1_DIN_DMA_MODE_BIT_SHIFT 0x0UL
+#define DX_DSCRPTR_QUEUE_WORD1_DIN_DMA_MODE_BIT_SIZE 0x2UL
+#define DX_DSCRPTR_QUEUE_WORD1_DIN_SIZE_BIT_SHIFT 0x2UL
+#define DX_DSCRPTR_QUEUE_WORD1_DIN_SIZE_BIT_SIZE 0x18UL
+#define DX_DSCRPTR_QUEUE_WORD1_NS_BIT_BIT_SHIFT 0x1AUL
+#define DX_DSCRPTR_QUEUE_WORD1_NS_BIT_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_QUEUE_WORD1_DIN_CONST_VALUE_BIT_SHIFT 0x1BUL
+#define DX_DSCRPTR_QUEUE_WORD1_DIN_CONST_VALUE_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_QUEUE_WORD1_NOT_LAST_BIT_SHIFT 0x1CUL
+#define DX_DSCRPTR_QUEUE_WORD1_NOT_LAST_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_QUEUE_WORD1_LOCK_QUEUE_BIT_SHIFT 0x1DUL
+#define DX_DSCRPTR_QUEUE_WORD1_LOCK_QUEUE_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_QUEUE_WORD1_NOT_USED_BIT_SHIFT 0x1EUL
+#define DX_DSCRPTR_QUEUE_WORD1_NOT_USED_BIT_SIZE 0x2UL
+#define DX_DSCRPTR_QUEUE_WORD2_REG_OFFSET 0xE88UL
+#define DX_DSCRPTR_QUEUE_WORD2_VALUE_BIT_SHIFT 0x0UL
+#define DX_DSCRPTR_QUEUE_WORD2_VALUE_BIT_SIZE 0x20UL
+#define DX_DSCRPTR_QUEUE_WORD3_REG_OFFSET 0xE8CUL
+#define DX_DSCRPTR_QUEUE_WORD3_DOUT_DMA_MODE_BIT_SHIFT 0x0UL
+#define DX_DSCRPTR_QUEUE_WORD3_DOUT_DMA_MODE_BIT_SIZE 0x2UL
+#define DX_DSCRPTR_QUEUE_WORD3_DOUT_SIZE_BIT_SHIFT 0x2UL
+#define DX_DSCRPTR_QUEUE_WORD3_DOUT_SIZE_BIT_SIZE 0x18UL
+#define DX_DSCRPTR_QUEUE_WORD3_NS_BIT_BIT_SHIFT 0x1AUL
+#define DX_DSCRPTR_QUEUE_WORD3_NS_BIT_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_QUEUE_WORD3_DOUT_LAST_IND_BIT_SHIFT 0x1BUL
+#define DX_DSCRPTR_QUEUE_WORD3_DOUT_LAST_IND_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_QUEUE_WORD3_HASH_XOR_BIT_BIT_SHIFT 0x1DUL
+#define DX_DSCRPTR_QUEUE_WORD3_HASH_XOR_BIT_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_QUEUE_WORD3_NOT_USED_BIT_SHIFT 0x1EUL
+#define DX_DSCRPTR_QUEUE_WORD3_NOT_USED_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_QUEUE_WORD3_QUEUE_LAST_IND_BIT_SHIFT 0x1FUL
+#define DX_DSCRPTR_QUEUE_WORD3_QUEUE_LAST_IND_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_QUEUE_WORD4_REG_OFFSET 0xE90UL
+#define DX_DSCRPTR_QUEUE_WORD4_DATA_FLOW_MODE_BIT_SHIFT 0x0UL
+#define DX_DSCRPTR_QUEUE_WORD4_DATA_FLOW_MODE_BIT_SIZE 0x6UL
+#define DX_DSCRPTR_QUEUE_WORD4_AES_SEL_N_HASH_BIT_SHIFT 0x6UL
+#define DX_DSCRPTR_QUEUE_WORD4_AES_SEL_N_HASH_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_QUEUE_WORD4_AES_XOR_CRYPTO_KEY_BIT_SHIFT 0x7UL
+#define DX_DSCRPTR_QUEUE_WORD4_AES_XOR_CRYPTO_KEY_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_QUEUE_WORD4_ACK_NEEDED_BIT_SHIFT 0x8UL
+#define DX_DSCRPTR_QUEUE_WORD4_ACK_NEEDED_BIT_SIZE 0x2UL
+#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_MODE_BIT_SHIFT 0xAUL
+#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_MODE_BIT_SIZE 0x4UL
+#define DX_DSCRPTR_QUEUE_WORD4_CMAC_SIZE0_BIT_SHIFT 0xEUL
+#define DX_DSCRPTR_QUEUE_WORD4_CMAC_SIZE0_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_DO_BIT_SHIFT 0xFUL
+#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_DO_BIT_SIZE 0x2UL
+#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_CONF0_BIT_SHIFT 0x11UL
+#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_CONF0_BIT_SIZE 0x2UL
+#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_CONF1_BIT_SHIFT 0x13UL
+#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_CONF1_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_CONF2_BIT_SHIFT 0x14UL
+#define DX_DSCRPTR_QUEUE_WORD4_CIPHER_CONF2_BIT_SIZE 0x2UL
+#define DX_DSCRPTR_QUEUE_WORD4_KEY_SIZE_BIT_SHIFT 0x16UL
+#define DX_DSCRPTR_QUEUE_WORD4_KEY_SIZE_BIT_SIZE 0x2UL
+#define DX_DSCRPTR_QUEUE_WORD4_SETUP_OPERATION_BIT_SHIFT 0x18UL
+#define DX_DSCRPTR_QUEUE_WORD4_SETUP_OPERATION_BIT_SIZE 0x4UL
+#define DX_DSCRPTR_QUEUE_WORD4_DIN_SRAM_ENDIANNESS_BIT_SHIFT 0x1CUL
+#define DX_DSCRPTR_QUEUE_WORD4_DIN_SRAM_ENDIANNESS_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_QUEUE_WORD4_DOUT_SRAM_ENDIANNESS_BIT_SHIFT 0x1DUL
+#define DX_DSCRPTR_QUEUE_WORD4_DOUT_SRAM_ENDIANNESS_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_QUEUE_WORD4_WORD_SWAP_BIT_SHIFT 0x1EUL
+#define DX_DSCRPTR_QUEUE_WORD4_WORD_SWAP_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_QUEUE_WORD4_BYTES_SWAP_BIT_SHIFT 0x1FUL
+#define DX_DSCRPTR_QUEUE_WORD4_BYTES_SWAP_BIT_SIZE 0x1UL
+#define DX_DSCRPTR_QUEUE_WORD5_REG_OFFSET 0xE94UL
+#define DX_DSCRPTR_QUEUE_WORD5_DIN_ADDR_HIGH_BIT_SHIFT 0x0UL
+#define DX_DSCRPTR_QUEUE_WORD5_DIN_ADDR_HIGH_BIT_SIZE 0x10UL
+#define DX_DSCRPTR_QUEUE_WORD5_DOUT_ADDR_HIGH_BIT_SHIFT 0x10UL
+#define DX_DSCRPTR_QUEUE_WORD5_DOUT_ADDR_HIGH_BIT_SIZE 0x10UL
+#define DX_DSCRPTR_QUEUE_WATERMARK_REG_OFFSET 0xE98UL
+#define DX_DSCRPTR_QUEUE_WATERMARK_VALUE_BIT_SHIFT 0x0UL
+#define DX_DSCRPTR_QUEUE_WATERMARK_VALUE_BIT_SIZE 0xAUL
+#define DX_DSCRPTR_QUEUE_CONTENT_REG_OFFSET 0xE9CUL
+#define DX_DSCRPTR_QUEUE_CONTENT_VALUE_BIT_SHIFT 0x0UL
+#define DX_DSCRPTR_QUEUE_CONTENT_VALUE_BIT_SIZE 0xAUL
// --------------------------------------
// BLOCK: AXI_P
// --------------------------------------
-#define DX_AXIM_MON_INFLIGHT_REG_OFFSET 0xB00UL
-#define DX_AXIM_MON_INFLIGHT_VALUE_BIT_SHIFT 0x0UL
-#define DX_AXIM_MON_INFLIGHT_VALUE_BIT_SIZE 0x8UL
-#define DX_AXIM_MON_INFLIGHTLAST_REG_OFFSET 0xB40UL
-#define DX_AXIM_MON_INFLIGHTLAST_VALUE_BIT_SHIFT 0x0UL
-#define DX_AXIM_MON_INFLIGHTLAST_VALUE_BIT_SIZE 0x8UL
-#define DX_AXIM_MON_COMP_REG_OFFSET 0xB80UL
-#define DX_AXIM_MON_COMP_VALUE_BIT_SHIFT 0x0UL
-#define DX_AXIM_MON_COMP_VALUE_BIT_SIZE 0x10UL
-#define DX_AXIM_MON_ERR_REG_OFFSET 0xBC4UL
-#define DX_AXIM_MON_ERR_BRESP_BIT_SHIFT 0x0UL
-#define DX_AXIM_MON_ERR_BRESP_BIT_SIZE 0x2UL
-#define DX_AXIM_MON_ERR_BID_BIT_SHIFT 0x2UL
-#define DX_AXIM_MON_ERR_BID_BIT_SIZE 0x4UL
-#define DX_AXIM_MON_ERR_RRESP_BIT_SHIFT 0x10UL
-#define DX_AXIM_MON_ERR_RRESP_BIT_SIZE 0x2UL
-#define DX_AXIM_MON_ERR_RID_BIT_SHIFT 0x12UL
-#define DX_AXIM_MON_ERR_RID_BIT_SIZE 0x4UL
-#define DX_AXIM_CFG_REG_OFFSET 0xBE8UL
-#define DX_AXIM_CFG_BRESPMASK_BIT_SHIFT 0x4UL
-#define DX_AXIM_CFG_BRESPMASK_BIT_SIZE 0x1UL
-#define DX_AXIM_CFG_RRESPMASK_BIT_SHIFT 0x5UL
-#define DX_AXIM_CFG_RRESPMASK_BIT_SIZE 0x1UL
-#define DX_AXIM_CFG_INFLTMASK_BIT_SHIFT 0x6UL
-#define DX_AXIM_CFG_INFLTMASK_BIT_SIZE 0x1UL
-#define DX_AXIM_CFG_COMPMASK_BIT_SHIFT 0x7UL
-#define DX_AXIM_CFG_COMPMASK_BIT_SIZE 0x1UL
-#define DX_AXIM_ACE_CONST_REG_OFFSET 0xBECUL
-#define DX_AXIM_ACE_CONST_ARDOMAIN_BIT_SHIFT 0x0UL
-#define DX_AXIM_ACE_CONST_ARDOMAIN_BIT_SIZE 0x2UL
-#define DX_AXIM_ACE_CONST_AWDOMAIN_BIT_SHIFT 0x2UL
-#define DX_AXIM_ACE_CONST_AWDOMAIN_BIT_SIZE 0x2UL
-#define DX_AXIM_ACE_CONST_ARBAR_BIT_SHIFT 0x4UL
-#define DX_AXIM_ACE_CONST_ARBAR_BIT_SIZE 0x2UL
-#define DX_AXIM_ACE_CONST_AWBAR_BIT_SHIFT 0x6UL
-#define DX_AXIM_ACE_CONST_AWBAR_BIT_SIZE 0x2UL
-#define DX_AXIM_ACE_CONST_ARSNOOP_BIT_SHIFT 0x8UL
-#define DX_AXIM_ACE_CONST_ARSNOOP_BIT_SIZE 0x4UL
-#define DX_AXIM_ACE_CONST_AWSNOOP_NOT_ALIGNED_BIT_SHIFT 0xCUL
-#define DX_AXIM_ACE_CONST_AWSNOOP_NOT_ALIGNED_BIT_SIZE 0x3UL
-#define DX_AXIM_ACE_CONST_AWSNOOP_ALIGNED_BIT_SHIFT 0xFUL
-#define DX_AXIM_ACE_CONST_AWSNOOP_ALIGNED_BIT_SIZE 0x3UL
-#define DX_AXIM_ACE_CONST_AWADDR_NOT_MASKED_BIT_SHIFT 0x12UL
-#define DX_AXIM_ACE_CONST_AWADDR_NOT_MASKED_BIT_SIZE 0x7UL
-#define DX_AXIM_ACE_CONST_AWLEN_VAL_BIT_SHIFT 0x19UL
-#define DX_AXIM_ACE_CONST_AWLEN_VAL_BIT_SIZE 0x4UL
-#define DX_AXIM_CACHE_PARAMS_REG_OFFSET 0xBF0UL
-#define DX_AXIM_CACHE_PARAMS_AWCACHE_LAST_BIT_SHIFT 0x0UL
-#define DX_AXIM_CACHE_PARAMS_AWCACHE_LAST_BIT_SIZE 0x4UL
-#define DX_AXIM_CACHE_PARAMS_AWCACHE_BIT_SHIFT 0x4UL
-#define DX_AXIM_CACHE_PARAMS_AWCACHE_BIT_SIZE 0x4UL
-#define DX_AXIM_CACHE_PARAMS_ARCACHE_BIT_SHIFT 0x8UL
-#define DX_AXIM_CACHE_PARAMS_ARCACHE_BIT_SIZE 0x4UL
+#define DX_AXIM_MON_INFLIGHT_REG_OFFSET 0xB00UL
+#define DX_AXIM_MON_INFLIGHT_VALUE_BIT_SHIFT 0x0UL
+#define DX_AXIM_MON_INFLIGHT_VALUE_BIT_SIZE 0x8UL
+#define DX_AXIM_MON_INFLIGHTLAST_REG_OFFSET 0xB40UL
+#define DX_AXIM_MON_INFLIGHTLAST_VALUE_BIT_SHIFT 0x0UL
+#define DX_AXIM_MON_INFLIGHTLAST_VALUE_BIT_SIZE 0x8UL
+#define DX_AXIM_MON_COMP_REG_OFFSET 0xB80UL
+#define DX_AXIM_MON_COMP_VALUE_BIT_SHIFT 0x0UL
+#define DX_AXIM_MON_COMP_VALUE_BIT_SIZE 0x10UL
+#define DX_AXIM_MON_ERR_REG_OFFSET 0xBC4UL
+#define DX_AXIM_MON_ERR_BRESP_BIT_SHIFT 0x0UL
+#define DX_AXIM_MON_ERR_BRESP_BIT_SIZE 0x2UL
+#define DX_AXIM_MON_ERR_BID_BIT_SHIFT 0x2UL
+#define DX_AXIM_MON_ERR_BID_BIT_SIZE 0x4UL
+#define DX_AXIM_MON_ERR_RRESP_BIT_SHIFT 0x10UL
+#define DX_AXIM_MON_ERR_RRESP_BIT_SIZE 0x2UL
+#define DX_AXIM_MON_ERR_RID_BIT_SHIFT 0x12UL
+#define DX_AXIM_MON_ERR_RID_BIT_SIZE 0x4UL
+#define DX_AXIM_CFG_REG_OFFSET 0xBE8UL
+#define DX_AXIM_CFG_BRESPMASK_BIT_SHIFT 0x4UL
+#define DX_AXIM_CFG_BRESPMASK_BIT_SIZE 0x1UL
+#define DX_AXIM_CFG_RRESPMASK_BIT_SHIFT 0x5UL
+#define DX_AXIM_CFG_RRESPMASK_BIT_SIZE 0x1UL
+#define DX_AXIM_CFG_INFLTMASK_BIT_SHIFT 0x6UL
+#define DX_AXIM_CFG_INFLTMASK_BIT_SIZE 0x1UL
+#define DX_AXIM_CFG_COMPMASK_BIT_SHIFT 0x7UL
+#define DX_AXIM_CFG_COMPMASK_BIT_SIZE 0x1UL
+#define DX_AXIM_ACE_CONST_REG_OFFSET 0xBECUL
+#define DX_AXIM_ACE_CONST_ARDOMAIN_BIT_SHIFT 0x0UL
+#define DX_AXIM_ACE_CONST_ARDOMAIN_BIT_SIZE 0x2UL
+#define DX_AXIM_ACE_CONST_AWDOMAIN_BIT_SHIFT 0x2UL
+#define DX_AXIM_ACE_CONST_AWDOMAIN_BIT_SIZE 0x2UL
+#define DX_AXIM_ACE_CONST_ARBAR_BIT_SHIFT 0x4UL
+#define DX_AXIM_ACE_CONST_ARBAR_BIT_SIZE 0x2UL
+#define DX_AXIM_ACE_CONST_AWBAR_BIT_SHIFT 0x6UL
+#define DX_AXIM_ACE_CONST_AWBAR_BIT_SIZE 0x2UL
+#define DX_AXIM_ACE_CONST_ARSNOOP_BIT_SHIFT 0x8UL
+#define DX_AXIM_ACE_CONST_ARSNOOP_BIT_SIZE 0x4UL
+#define DX_AXIM_ACE_CONST_AWSNOOP_NOT_ALIGNED_BIT_SHIFT 0xCUL
+#define DX_AXIM_ACE_CONST_AWSNOOP_NOT_ALIGNED_BIT_SIZE 0x3UL
+#define DX_AXIM_ACE_CONST_AWSNOOP_ALIGNED_BIT_SHIFT 0xFUL
+#define DX_AXIM_ACE_CONST_AWSNOOP_ALIGNED_BIT_SIZE 0x3UL
+#define DX_AXIM_ACE_CONST_AWADDR_NOT_MASKED_BIT_SHIFT 0x12UL
+#define DX_AXIM_ACE_CONST_AWADDR_NOT_MASKED_BIT_SIZE 0x7UL
+#define DX_AXIM_ACE_CONST_AWLEN_VAL_BIT_SHIFT 0x19UL
+#define DX_AXIM_ACE_CONST_AWLEN_VAL_BIT_SIZE 0x4UL
+#define DX_AXIM_CACHE_PARAMS_REG_OFFSET 0xBF0UL
+#define DX_AXIM_CACHE_PARAMS_AWCACHE_LAST_BIT_SHIFT 0x0UL
+#define DX_AXIM_CACHE_PARAMS_AWCACHE_LAST_BIT_SIZE 0x4UL
+#define DX_AXIM_CACHE_PARAMS_AWCACHE_BIT_SHIFT 0x4UL
+#define DX_AXIM_CACHE_PARAMS_AWCACHE_BIT_SIZE 0x4UL
+#define DX_AXIM_CACHE_PARAMS_ARCACHE_BIT_SHIFT 0x8UL
+#define DX_AXIM_CACHE_PARAMS_ARCACHE_BIT_SIZE 0x4UL
#endif // __DX_CRYS_KERNEL_H__
+++ /dev/null
-/*
- * Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, see <http://www.gnu.org/licenses/>.
- */
-
-#ifndef __DX_ENV_H__
-#define __DX_ENV_H__
-
-// --------------------------------------
-// BLOCK: FPGA_ENV_REGS
-// --------------------------------------
-#define DX_ENV_PKA_DEBUG_MODE_REG_OFFSET 0x024UL
-#define DX_ENV_PKA_DEBUG_MODE_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_PKA_DEBUG_MODE_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_SCAN_MODE_REG_OFFSET 0x030UL
-#define DX_ENV_SCAN_MODE_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_SCAN_MODE_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_CC_ALLOW_SCAN_REG_OFFSET 0x034UL
-#define DX_ENV_CC_ALLOW_SCAN_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_CC_ALLOW_SCAN_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_CC_HOST_INT_REG_OFFSET 0x0A0UL
-#define DX_ENV_CC_HOST_INT_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_CC_HOST_INT_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_CC_PUB_HOST_INT_REG_OFFSET 0x0A4UL
-#define DX_ENV_CC_PUB_HOST_INT_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_CC_PUB_HOST_INT_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_CC_RST_N_REG_OFFSET 0x0A8UL
-#define DX_ENV_CC_RST_N_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_CC_RST_N_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_RST_OVERRIDE_REG_OFFSET 0x0ACUL
-#define DX_ENV_RST_OVERRIDE_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_RST_OVERRIDE_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_CC_POR_N_ADDR_REG_OFFSET 0x0E0UL
-#define DX_ENV_CC_POR_N_ADDR_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_CC_POR_N_ADDR_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_CC_COLD_RST_REG_OFFSET 0x0FCUL
-#define DX_ENV_CC_COLD_RST_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_CC_COLD_RST_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_DUMMY_ADDR_REG_OFFSET 0x108UL
-#define DX_ENV_DUMMY_ADDR_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_DUMMY_ADDR_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_COUNTER_CLR_REG_OFFSET 0x118UL
-#define DX_ENV_COUNTER_CLR_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_COUNTER_CLR_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_COUNTER_RD_REG_OFFSET 0x11CUL
-#define DX_ENV_COUNTER_RD_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_COUNTER_RD_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_RNG_DEBUG_ENABLE_REG_OFFSET 0x430UL
-#define DX_ENV_RNG_DEBUG_ENABLE_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_RNG_DEBUG_ENABLE_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_CC_LCS_REG_OFFSET 0x43CUL
-#define DX_ENV_CC_LCS_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_CC_LCS_VALUE_BIT_SIZE 0x8UL
-#define DX_ENV_CC_IS_CM_DM_SECURE_RMA_REG_OFFSET 0x440UL
-#define DX_ENV_CC_IS_CM_DM_SECURE_RMA_IS_CM_BIT_SHIFT 0x0UL
-#define DX_ENV_CC_IS_CM_DM_SECURE_RMA_IS_CM_BIT_SIZE 0x1UL
-#define DX_ENV_CC_IS_CM_DM_SECURE_RMA_IS_DM_BIT_SHIFT 0x1UL
-#define DX_ENV_CC_IS_CM_DM_SECURE_RMA_IS_DM_BIT_SIZE 0x1UL
-#define DX_ENV_CC_IS_CM_DM_SECURE_RMA_IS_SECURE_BIT_SHIFT 0x2UL
-#define DX_ENV_CC_IS_CM_DM_SECURE_RMA_IS_SECURE_BIT_SIZE 0x1UL
-#define DX_ENV_CC_IS_CM_DM_SECURE_RMA_IS_RMA_BIT_SHIFT 0x3UL
-#define DX_ENV_CC_IS_CM_DM_SECURE_RMA_IS_RMA_BIT_SIZE 0x1UL
-#define DX_ENV_DCU_EN_REG_OFFSET 0x444UL
-#define DX_ENV_DCU_EN_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_DCU_EN_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_CC_LCS_IS_VALID_REG_OFFSET 0x448UL
-#define DX_ENV_CC_LCS_IS_VALID_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_CC_LCS_IS_VALID_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_POWER_DOWN_REG_OFFSET 0x478UL
-#define DX_ENV_POWER_DOWN_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_POWER_DOWN_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_DCU_H_EN_REG_OFFSET 0x484UL
-#define DX_ENV_DCU_H_EN_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_DCU_H_EN_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_VERSION_REG_OFFSET 0x488UL
-#define DX_ENV_VERSION_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_VERSION_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_ROSC_WRITE_REG_OFFSET 0x48CUL
-#define DX_ENV_ROSC_WRITE_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_ROSC_WRITE_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_ROSC_ADDR_REG_OFFSET 0x490UL
-#define DX_ENV_ROSC_ADDR_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_ROSC_ADDR_VALUE_BIT_SIZE 0x8UL
-#define DX_ENV_RESET_SESSION_KEY_REG_OFFSET 0x494UL
-#define DX_ENV_RESET_SESSION_KEY_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_RESET_SESSION_KEY_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_SESSION_KEY_0_REG_OFFSET 0x4A0UL
-#define DX_ENV_SESSION_KEY_0_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_SESSION_KEY_0_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_SESSION_KEY_1_REG_OFFSET 0x4A4UL
-#define DX_ENV_SESSION_KEY_1_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_SESSION_KEY_1_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_SESSION_KEY_2_REG_OFFSET 0x4A8UL
-#define DX_ENV_SESSION_KEY_2_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_SESSION_KEY_2_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_SESSION_KEY_3_REG_OFFSET 0x4ACUL
-#define DX_ENV_SESSION_KEY_3_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_SESSION_KEY_3_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_SESSION_KEY_VALID_REG_OFFSET 0x4B0UL
-#define DX_ENV_SESSION_KEY_VALID_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_SESSION_KEY_VALID_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_SPIDEN_REG_OFFSET 0x4D0UL
-#define DX_ENV_SPIDEN_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_SPIDEN_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_AXIM_USER_PARAMS_REG_OFFSET 0x600UL
-#define DX_ENV_AXIM_USER_PARAMS_ARUSER_BIT_SHIFT 0x0UL
-#define DX_ENV_AXIM_USER_PARAMS_ARUSER_BIT_SIZE 0x5UL
-#define DX_ENV_AXIM_USER_PARAMS_AWUSER_BIT_SHIFT 0x5UL
-#define DX_ENV_AXIM_USER_PARAMS_AWUSER_BIT_SIZE 0x5UL
-#define DX_ENV_SECURITY_MODE_OVERRIDE_REG_OFFSET 0x604UL
-#define DX_ENV_SECURITY_MODE_OVERRIDE_AWPROT_NS_BIT_BIT_SHIFT 0x0UL
-#define DX_ENV_SECURITY_MODE_OVERRIDE_AWPROT_NS_BIT_BIT_SIZE 0x1UL
-#define DX_ENV_SECURITY_MODE_OVERRIDE_AWPROT_NS_OVERRIDE_BIT_SHIFT 0x1UL
-#define DX_ENV_SECURITY_MODE_OVERRIDE_AWPROT_NS_OVERRIDE_BIT_SIZE 0x1UL
-#define DX_ENV_SECURITY_MODE_OVERRIDE_ARPROT_NS_BIT_BIT_SHIFT 0x2UL
-#define DX_ENV_SECURITY_MODE_OVERRIDE_ARPROT_NS_BIT_BIT_SIZE 0x1UL
-#define DX_ENV_SECURITY_MODE_OVERRIDE_ARPROT_NS_OVERRIDE_BIT_SHIFT 0x3UL
-#define DX_ENV_SECURITY_MODE_OVERRIDE_ARPROT_NS_OVERRIDE_BIT_SIZE 0x1UL
-#define DX_ENV_AO_CC_KPLT_0_REG_OFFSET 0x620UL
-#define DX_ENV_AO_CC_KPLT_0_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_AO_CC_KPLT_0_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_AO_CC_KPLT_1_REG_OFFSET 0x624UL
-#define DX_ENV_AO_CC_KPLT_1_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_AO_CC_KPLT_1_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_AO_CC_KPLT_2_REG_OFFSET 0x628UL
-#define DX_ENV_AO_CC_KPLT_2_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_AO_CC_KPLT_2_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_AO_CC_KPLT_3_REG_OFFSET 0x62CUL
-#define DX_ENV_AO_CC_KPLT_3_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_AO_CC_KPLT_3_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_AO_CC_KCST_0_REG_OFFSET 0x630UL
-#define DX_ENV_AO_CC_KCST_0_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_AO_CC_KCST_0_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_AO_CC_KCST_1_REG_OFFSET 0x634UL
-#define DX_ENV_AO_CC_KCST_1_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_AO_CC_KCST_1_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_AO_CC_KCST_2_REG_OFFSET 0x638UL
-#define DX_ENV_AO_CC_KCST_2_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_AO_CC_KCST_2_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_AO_CC_KCST_3_REG_OFFSET 0x63CUL
-#define DX_ENV_AO_CC_KCST_3_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_AO_CC_KCST_3_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_APB_FIPS_ADDR_REG_OFFSET 0x650UL
-#define DX_ENV_APB_FIPS_ADDR_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_APB_FIPS_ADDR_VALUE_BIT_SIZE 0xCUL
-#define DX_ENV_APB_FIPS_VAL_REG_OFFSET 0x654UL
-#define DX_ENV_APB_FIPS_VAL_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_APB_FIPS_VAL_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_APB_FIPS_MASK_REG_OFFSET 0x658UL
-#define DX_ENV_APB_FIPS_MASK_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_APB_FIPS_MASK_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_APB_FIPS_CNT_REG_OFFSET 0x65CUL
-#define DX_ENV_APB_FIPS_CNT_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_APB_FIPS_CNT_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_APB_FIPS_NEW_ADDR_REG_OFFSET 0x660UL
-#define DX_ENV_APB_FIPS_NEW_ADDR_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_APB_FIPS_NEW_ADDR_VALUE_BIT_SIZE 0xCUL
-#define DX_ENV_APB_FIPS_NEW_VAL_REG_OFFSET 0x664UL
-#define DX_ENV_APB_FIPS_NEW_VAL_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_APB_FIPS_NEW_VAL_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_APBP_FIPS_ADDR_REG_OFFSET 0x670UL
-#define DX_ENV_APBP_FIPS_ADDR_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_APBP_FIPS_ADDR_VALUE_BIT_SIZE 0xCUL
-#define DX_ENV_APBP_FIPS_VAL_REG_OFFSET 0x674UL
-#define DX_ENV_APBP_FIPS_VAL_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_APBP_FIPS_VAL_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_APBP_FIPS_MASK_REG_OFFSET 0x678UL
-#define DX_ENV_APBP_FIPS_MASK_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_APBP_FIPS_MASK_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_APBP_FIPS_CNT_REG_OFFSET 0x67CUL
-#define DX_ENV_APBP_FIPS_CNT_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_APBP_FIPS_CNT_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_APBP_FIPS_NEW_ADDR_REG_OFFSET 0x680UL
-#define DX_ENV_APBP_FIPS_NEW_ADDR_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_APBP_FIPS_NEW_ADDR_VALUE_BIT_SIZE 0xCUL
-#define DX_ENV_APBP_FIPS_NEW_VAL_REG_OFFSET 0x684UL
-#define DX_ENV_APBP_FIPS_NEW_VAL_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_APBP_FIPS_NEW_VAL_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_CC_POWERDOWN_EN_REG_OFFSET 0x690UL
-#define DX_ENV_CC_POWERDOWN_EN_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_CC_POWERDOWN_EN_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_CC_POWERDOWN_RST_EN_REG_OFFSET 0x694UL
-#define DX_ENV_CC_POWERDOWN_RST_EN_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_CC_POWERDOWN_RST_EN_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_POWERDOWN_RST_CNTR_REG_OFFSET 0x698UL
-#define DX_ENV_POWERDOWN_RST_CNTR_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_POWERDOWN_RST_CNTR_VALUE_BIT_SIZE 0x20UL
-#define DX_ENV_POWERDOWN_EN_DEBUG_REG_OFFSET 0x69CUL
-#define DX_ENV_POWERDOWN_EN_DEBUG_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_POWERDOWN_EN_DEBUG_VALUE_BIT_SIZE 0x1UL
-// --------------------------------------
-// BLOCK: ENV_CC_MEMORIES
-// --------------------------------------
-#define DX_ENV_FUSE_READY_REG_OFFSET 0x000UL
-#define DX_ENV_FUSE_READY_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_FUSE_READY_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_PERF_RAM_MASTER_REG_OFFSET 0x0ECUL
-#define DX_ENV_PERF_RAM_MASTER_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_PERF_RAM_MASTER_VALUE_BIT_SIZE 0x1UL
-#define DX_ENV_PERF_RAM_ADDR_HIGH4_REG_OFFSET 0x0F0UL
-#define DX_ENV_PERF_RAM_ADDR_HIGH4_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_PERF_RAM_ADDR_HIGH4_VALUE_BIT_SIZE 0x2UL
-#define DX_ENV_FUSES_RAM_REG_OFFSET 0x3ECUL
-#define DX_ENV_FUSES_RAM_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_FUSES_RAM_VALUE_BIT_SIZE 0x20UL
-// --------------------------------------
-// BLOCK: ENV_PERF_RAM_BASE
-// --------------------------------------
-#define DX_ENV_PERF_RAM_BASE_REG_OFFSET 0x000UL
-#define DX_ENV_PERF_RAM_BASE_VALUE_BIT_SHIFT 0x0UL
-#define DX_ENV_PERF_RAM_BASE_VALUE_BIT_SIZE 0x20UL
-
-#endif /*__DX_ENV_H__*/
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
// --------------------------------------
// BLOCK: HOST_P
// --------------------------------------
-#define DX_HOST_IRR_REG_OFFSET 0xA00UL
-#define DX_HOST_IRR_DSCRPTR_COMPLETION_LOW_INT_BIT_SHIFT 0x2UL
-#define DX_HOST_IRR_DSCRPTR_COMPLETION_LOW_INT_BIT_SIZE 0x1UL
-#define DX_HOST_IRR_AXI_ERR_INT_BIT_SHIFT 0x8UL
-#define DX_HOST_IRR_AXI_ERR_INT_BIT_SIZE 0x1UL
-#define DX_HOST_IRR_GPR0_BIT_SHIFT 0xBUL
-#define DX_HOST_IRR_GPR0_BIT_SIZE 0x1UL
-#define DX_HOST_IRR_DSCRPTR_WATERMARK_INT_BIT_SHIFT 0x13UL
-#define DX_HOST_IRR_DSCRPTR_WATERMARK_INT_BIT_SIZE 0x1UL
-#define DX_HOST_IRR_AXIM_COMP_INT_BIT_SHIFT 0x17UL
-#define DX_HOST_IRR_AXIM_COMP_INT_BIT_SIZE 0x1UL
-#define DX_HOST_IMR_REG_OFFSET 0xA04UL
-#define DX_HOST_IMR_NOT_USED_MASK_BIT_SHIFT 0x1UL
-#define DX_HOST_IMR_NOT_USED_MASK_BIT_SIZE 0x1UL
-#define DX_HOST_IMR_DSCRPTR_COMPLETION_MASK_BIT_SHIFT 0x2UL
-#define DX_HOST_IMR_DSCRPTR_COMPLETION_MASK_BIT_SIZE 0x1UL
-#define DX_HOST_IMR_AXI_ERR_MASK_BIT_SHIFT 0x8UL
-#define DX_HOST_IMR_AXI_ERR_MASK_BIT_SIZE 0x1UL
-#define DX_HOST_IMR_GPR0_BIT_SHIFT 0xBUL
-#define DX_HOST_IMR_GPR0_BIT_SIZE 0x1UL
-#define DX_HOST_IMR_DSCRPTR_WATERMARK_MASK0_BIT_SHIFT 0x13UL
-#define DX_HOST_IMR_DSCRPTR_WATERMARK_MASK0_BIT_SIZE 0x1UL
-#define DX_HOST_IMR_AXIM_COMP_INT_MASK_BIT_SHIFT 0x17UL
-#define DX_HOST_IMR_AXIM_COMP_INT_MASK_BIT_SIZE 0x1UL
-#define DX_HOST_ICR_REG_OFFSET 0xA08UL
-#define DX_HOST_ICR_DSCRPTR_COMPLETION_BIT_SHIFT 0x2UL
-#define DX_HOST_ICR_DSCRPTR_COMPLETION_BIT_SIZE 0x1UL
-#define DX_HOST_ICR_AXI_ERR_CLEAR_BIT_SHIFT 0x8UL
-#define DX_HOST_ICR_AXI_ERR_CLEAR_BIT_SIZE 0x1UL
-#define DX_HOST_ICR_GPR_INT_CLEAR_BIT_SHIFT 0xBUL
-#define DX_HOST_ICR_GPR_INT_CLEAR_BIT_SIZE 0x1UL
-#define DX_HOST_ICR_DSCRPTR_WATERMARK_QUEUE0_CLEAR_BIT_SHIFT 0x13UL
-#define DX_HOST_ICR_DSCRPTR_WATERMARK_QUEUE0_CLEAR_BIT_SIZE 0x1UL
-#define DX_HOST_ICR_AXIM_COMP_INT_CLEAR_BIT_SHIFT 0x17UL
-#define DX_HOST_ICR_AXIM_COMP_INT_CLEAR_BIT_SIZE 0x1UL
-#define DX_HOST_SIGNATURE_REG_OFFSET 0xA24UL
-#define DX_HOST_SIGNATURE_VALUE_BIT_SHIFT 0x0UL
-#define DX_HOST_SIGNATURE_VALUE_BIT_SIZE 0x20UL
-#define DX_HOST_BOOT_REG_OFFSET 0xA28UL
-#define DX_HOST_BOOT_SYNTHESIS_CONFIG_BIT_SHIFT 0x0UL
-#define DX_HOST_BOOT_SYNTHESIS_CONFIG_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_LARGE_RKEK_LOCAL_BIT_SHIFT 0x1UL
-#define DX_HOST_BOOT_LARGE_RKEK_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_HASH_IN_FUSES_LOCAL_BIT_SHIFT 0x2UL
-#define DX_HOST_BOOT_HASH_IN_FUSES_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_EXT_MEM_SECURED_LOCAL_BIT_SHIFT 0x3UL
-#define DX_HOST_BOOT_EXT_MEM_SECURED_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_RKEK_ECC_EXISTS_LOCAL_N_BIT_SHIFT 0x5UL
-#define DX_HOST_BOOT_RKEK_ECC_EXISTS_LOCAL_N_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_SRAM_SIZE_LOCAL_BIT_SHIFT 0x6UL
-#define DX_HOST_BOOT_SRAM_SIZE_LOCAL_BIT_SIZE 0x3UL
-#define DX_HOST_BOOT_DSCRPTR_EXISTS_LOCAL_BIT_SHIFT 0x9UL
-#define DX_HOST_BOOT_DSCRPTR_EXISTS_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_PAU_EXISTS_LOCAL_BIT_SHIFT 0xAUL
-#define DX_HOST_BOOT_PAU_EXISTS_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_RNG_EXISTS_LOCAL_BIT_SHIFT 0xBUL
-#define DX_HOST_BOOT_RNG_EXISTS_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_PKA_EXISTS_LOCAL_BIT_SHIFT 0xCUL
-#define DX_HOST_BOOT_PKA_EXISTS_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_RC4_EXISTS_LOCAL_BIT_SHIFT 0xDUL
-#define DX_HOST_BOOT_RC4_EXISTS_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_SHA_512_PRSNT_LOCAL_BIT_SHIFT 0xEUL
-#define DX_HOST_BOOT_SHA_512_PRSNT_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_SHA_256_PRSNT_LOCAL_BIT_SHIFT 0xFUL
-#define DX_HOST_BOOT_SHA_256_PRSNT_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_MD5_PRSNT_LOCAL_BIT_SHIFT 0x10UL
-#define DX_HOST_BOOT_MD5_PRSNT_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_HASH_EXISTS_LOCAL_BIT_SHIFT 0x11UL
-#define DX_HOST_BOOT_HASH_EXISTS_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_C2_EXISTS_LOCAL_BIT_SHIFT 0x12UL
-#define DX_HOST_BOOT_C2_EXISTS_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_DES_EXISTS_LOCAL_BIT_SHIFT 0x13UL
-#define DX_HOST_BOOT_DES_EXISTS_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_AES_XCBC_MAC_EXISTS_LOCAL_BIT_SHIFT 0x14UL
-#define DX_HOST_BOOT_AES_XCBC_MAC_EXISTS_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_AES_CMAC_EXISTS_LOCAL_BIT_SHIFT 0x15UL
-#define DX_HOST_BOOT_AES_CMAC_EXISTS_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_AES_CCM_EXISTS_LOCAL_BIT_SHIFT 0x16UL
-#define DX_HOST_BOOT_AES_CCM_EXISTS_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_AES_XEX_HW_T_CALC_LOCAL_BIT_SHIFT 0x17UL
-#define DX_HOST_BOOT_AES_XEX_HW_T_CALC_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_AES_XEX_EXISTS_LOCAL_BIT_SHIFT 0x18UL
-#define DX_HOST_BOOT_AES_XEX_EXISTS_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_CTR_EXISTS_LOCAL_BIT_SHIFT 0x19UL
-#define DX_HOST_BOOT_CTR_EXISTS_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_AES_DIN_BYTE_RESOLUTION_LOCAL_BIT_SHIFT 0x1AUL
-#define DX_HOST_BOOT_AES_DIN_BYTE_RESOLUTION_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_TUNNELING_ENB_LOCAL_BIT_SHIFT 0x1BUL
-#define DX_HOST_BOOT_TUNNELING_ENB_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_SUPPORT_256_192_KEY_LOCAL_BIT_SHIFT 0x1CUL
-#define DX_HOST_BOOT_SUPPORT_256_192_KEY_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_ONLY_ENCRYPT_LOCAL_BIT_SHIFT 0x1DUL
-#define DX_HOST_BOOT_ONLY_ENCRYPT_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_BOOT_AES_EXISTS_LOCAL_BIT_SHIFT 0x1EUL
-#define DX_HOST_BOOT_AES_EXISTS_LOCAL_BIT_SIZE 0x1UL
-#define DX_HOST_VERSION_REG_OFFSET 0xA40UL
-#define DX_HOST_VERSION_VALUE_BIT_SHIFT 0x0UL
-#define DX_HOST_VERSION_VALUE_BIT_SIZE 0x20UL
-#define DX_HOST_KFDE0_VALID_REG_OFFSET 0xA60UL
-#define DX_HOST_KFDE0_VALID_VALUE_BIT_SHIFT 0x0UL
-#define DX_HOST_KFDE0_VALID_VALUE_BIT_SIZE 0x1UL
-#define DX_HOST_KFDE1_VALID_REG_OFFSET 0xA64UL
-#define DX_HOST_KFDE1_VALID_VALUE_BIT_SHIFT 0x0UL
-#define DX_HOST_KFDE1_VALID_VALUE_BIT_SIZE 0x1UL
-#define DX_HOST_KFDE2_VALID_REG_OFFSET 0xA68UL
-#define DX_HOST_KFDE2_VALID_VALUE_BIT_SHIFT 0x0UL
-#define DX_HOST_KFDE2_VALID_VALUE_BIT_SIZE 0x1UL
-#define DX_HOST_KFDE3_VALID_REG_OFFSET 0xA6CUL
-#define DX_HOST_KFDE3_VALID_VALUE_BIT_SHIFT 0x0UL
-#define DX_HOST_KFDE3_VALID_VALUE_BIT_SIZE 0x1UL
-#define DX_HOST_GPR0_REG_OFFSET 0xA70UL
-#define DX_HOST_GPR0_VALUE_BIT_SHIFT 0x0UL
-#define DX_HOST_GPR0_VALUE_BIT_SIZE 0x20UL
-#define DX_GPR_HOST_REG_OFFSET 0xA74UL
-#define DX_GPR_HOST_VALUE_BIT_SHIFT 0x0UL
-#define DX_GPR_HOST_VALUE_BIT_SIZE 0x20UL
-#define DX_HOST_POWER_DOWN_EN_REG_OFFSET 0xA78UL
-#define DX_HOST_POWER_DOWN_EN_VALUE_BIT_SHIFT 0x0UL
-#define DX_HOST_POWER_DOWN_EN_VALUE_BIT_SIZE 0x1UL
+#define DX_HOST_IRR_REG_OFFSET 0xA00UL
+#define DX_HOST_IRR_DSCRPTR_COMPLETION_LOW_INT_BIT_SHIFT 0x2UL
+#define DX_HOST_IRR_DSCRPTR_COMPLETION_LOW_INT_BIT_SIZE 0x1UL
+#define DX_HOST_IRR_AXI_ERR_INT_BIT_SHIFT 0x8UL
+#define DX_HOST_IRR_AXI_ERR_INT_BIT_SIZE 0x1UL
+#define DX_HOST_IRR_GPR0_BIT_SHIFT 0xBUL
+#define DX_HOST_IRR_GPR0_BIT_SIZE 0x1UL
+#define DX_HOST_IRR_DSCRPTR_WATERMARK_INT_BIT_SHIFT 0x13UL
+#define DX_HOST_IRR_DSCRPTR_WATERMARK_INT_BIT_SIZE 0x1UL
+#define DX_HOST_IRR_AXIM_COMP_INT_BIT_SHIFT 0x17UL
+#define DX_HOST_IRR_AXIM_COMP_INT_BIT_SIZE 0x1UL
+#define DX_HOST_IMR_REG_OFFSET 0xA04UL
+#define DX_HOST_IMR_NOT_USED_MASK_BIT_SHIFT 0x1UL
+#define DX_HOST_IMR_NOT_USED_MASK_BIT_SIZE 0x1UL
+#define DX_HOST_IMR_DSCRPTR_COMPLETION_MASK_BIT_SHIFT 0x2UL
+#define DX_HOST_IMR_DSCRPTR_COMPLETION_MASK_BIT_SIZE 0x1UL
+#define DX_HOST_IMR_AXI_ERR_MASK_BIT_SHIFT 0x8UL
+#define DX_HOST_IMR_AXI_ERR_MASK_BIT_SIZE 0x1UL
+#define DX_HOST_IMR_GPR0_BIT_SHIFT 0xBUL
+#define DX_HOST_IMR_GPR0_BIT_SIZE 0x1UL
+#define DX_HOST_IMR_DSCRPTR_WATERMARK_MASK0_BIT_SHIFT 0x13UL
+#define DX_HOST_IMR_DSCRPTR_WATERMARK_MASK0_BIT_SIZE 0x1UL
+#define DX_HOST_IMR_AXIM_COMP_INT_MASK_BIT_SHIFT 0x17UL
+#define DX_HOST_IMR_AXIM_COMP_INT_MASK_BIT_SIZE 0x1UL
+#define DX_HOST_ICR_REG_OFFSET 0xA08UL
+#define DX_HOST_ICR_DSCRPTR_COMPLETION_BIT_SHIFT 0x2UL
+#define DX_HOST_ICR_DSCRPTR_COMPLETION_BIT_SIZE 0x1UL
+#define DX_HOST_ICR_AXI_ERR_CLEAR_BIT_SHIFT 0x8UL
+#define DX_HOST_ICR_AXI_ERR_CLEAR_BIT_SIZE 0x1UL
+#define DX_HOST_ICR_GPR_INT_CLEAR_BIT_SHIFT 0xBUL
+#define DX_HOST_ICR_GPR_INT_CLEAR_BIT_SIZE 0x1UL
+#define DX_HOST_ICR_DSCRPTR_WATERMARK_QUEUE0_CLEAR_BIT_SHIFT 0x13UL
+#define DX_HOST_ICR_DSCRPTR_WATERMARK_QUEUE0_CLEAR_BIT_SIZE 0x1UL
+#define DX_HOST_ICR_AXIM_COMP_INT_CLEAR_BIT_SHIFT 0x17UL
+#define DX_HOST_ICR_AXIM_COMP_INT_CLEAR_BIT_SIZE 0x1UL
+#define DX_HOST_SIGNATURE_REG_OFFSET 0xA24UL
+#define DX_HOST_SIGNATURE_VALUE_BIT_SHIFT 0x0UL
+#define DX_HOST_SIGNATURE_VALUE_BIT_SIZE 0x20UL
+#define DX_HOST_BOOT_REG_OFFSET 0xA28UL
+#define DX_HOST_BOOT_SYNTHESIS_CONFIG_BIT_SHIFT 0x0UL
+#define DX_HOST_BOOT_SYNTHESIS_CONFIG_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_LARGE_RKEK_LOCAL_BIT_SHIFT 0x1UL
+#define DX_HOST_BOOT_LARGE_RKEK_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_HASH_IN_FUSES_LOCAL_BIT_SHIFT 0x2UL
+#define DX_HOST_BOOT_HASH_IN_FUSES_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_EXT_MEM_SECURED_LOCAL_BIT_SHIFT 0x3UL
+#define DX_HOST_BOOT_EXT_MEM_SECURED_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_RKEK_ECC_EXISTS_LOCAL_N_BIT_SHIFT 0x5UL
+#define DX_HOST_BOOT_RKEK_ECC_EXISTS_LOCAL_N_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_SRAM_SIZE_LOCAL_BIT_SHIFT 0x6UL
+#define DX_HOST_BOOT_SRAM_SIZE_LOCAL_BIT_SIZE 0x3UL
+#define DX_HOST_BOOT_DSCRPTR_EXISTS_LOCAL_BIT_SHIFT 0x9UL
+#define DX_HOST_BOOT_DSCRPTR_EXISTS_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_PAU_EXISTS_LOCAL_BIT_SHIFT 0xAUL
+#define DX_HOST_BOOT_PAU_EXISTS_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_RNG_EXISTS_LOCAL_BIT_SHIFT 0xBUL
+#define DX_HOST_BOOT_RNG_EXISTS_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_PKA_EXISTS_LOCAL_BIT_SHIFT 0xCUL
+#define DX_HOST_BOOT_PKA_EXISTS_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_RC4_EXISTS_LOCAL_BIT_SHIFT 0xDUL
+#define DX_HOST_BOOT_RC4_EXISTS_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_SHA_512_PRSNT_LOCAL_BIT_SHIFT 0xEUL
+#define DX_HOST_BOOT_SHA_512_PRSNT_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_SHA_256_PRSNT_LOCAL_BIT_SHIFT 0xFUL
+#define DX_HOST_BOOT_SHA_256_PRSNT_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_MD5_PRSNT_LOCAL_BIT_SHIFT 0x10UL
+#define DX_HOST_BOOT_MD5_PRSNT_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_HASH_EXISTS_LOCAL_BIT_SHIFT 0x11UL
+#define DX_HOST_BOOT_HASH_EXISTS_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_C2_EXISTS_LOCAL_BIT_SHIFT 0x12UL
+#define DX_HOST_BOOT_C2_EXISTS_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_DES_EXISTS_LOCAL_BIT_SHIFT 0x13UL
+#define DX_HOST_BOOT_DES_EXISTS_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_AES_XCBC_MAC_EXISTS_LOCAL_BIT_SHIFT 0x14UL
+#define DX_HOST_BOOT_AES_XCBC_MAC_EXISTS_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_AES_CMAC_EXISTS_LOCAL_BIT_SHIFT 0x15UL
+#define DX_HOST_BOOT_AES_CMAC_EXISTS_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_AES_CCM_EXISTS_LOCAL_BIT_SHIFT 0x16UL
+#define DX_HOST_BOOT_AES_CCM_EXISTS_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_AES_XEX_HW_T_CALC_LOCAL_BIT_SHIFT 0x17UL
+#define DX_HOST_BOOT_AES_XEX_HW_T_CALC_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_AES_XEX_EXISTS_LOCAL_BIT_SHIFT 0x18UL
+#define DX_HOST_BOOT_AES_XEX_EXISTS_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_CTR_EXISTS_LOCAL_BIT_SHIFT 0x19UL
+#define DX_HOST_BOOT_CTR_EXISTS_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_AES_DIN_BYTE_RESOLUTION_LOCAL_BIT_SHIFT 0x1AUL
+#define DX_HOST_BOOT_AES_DIN_BYTE_RESOLUTION_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_TUNNELING_ENB_LOCAL_BIT_SHIFT 0x1BUL
+#define DX_HOST_BOOT_TUNNELING_ENB_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_SUPPORT_256_192_KEY_LOCAL_BIT_SHIFT 0x1CUL
+#define DX_HOST_BOOT_SUPPORT_256_192_KEY_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_ONLY_ENCRYPT_LOCAL_BIT_SHIFT 0x1DUL
+#define DX_HOST_BOOT_ONLY_ENCRYPT_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_BOOT_AES_EXISTS_LOCAL_BIT_SHIFT 0x1EUL
+#define DX_HOST_BOOT_AES_EXISTS_LOCAL_BIT_SIZE 0x1UL
+#define DX_HOST_VERSION_REG_OFFSET 0xA40UL
+#define DX_HOST_VERSION_VALUE_BIT_SHIFT 0x0UL
+#define DX_HOST_VERSION_VALUE_BIT_SIZE 0x20UL
+#define DX_HOST_KFDE0_VALID_REG_OFFSET 0xA60UL
+#define DX_HOST_KFDE0_VALID_VALUE_BIT_SHIFT 0x0UL
+#define DX_HOST_KFDE0_VALID_VALUE_BIT_SIZE 0x1UL
+#define DX_HOST_KFDE1_VALID_REG_OFFSET 0xA64UL
+#define DX_HOST_KFDE1_VALID_VALUE_BIT_SHIFT 0x0UL
+#define DX_HOST_KFDE1_VALID_VALUE_BIT_SIZE 0x1UL
+#define DX_HOST_KFDE2_VALID_REG_OFFSET 0xA68UL
+#define DX_HOST_KFDE2_VALID_VALUE_BIT_SHIFT 0x0UL
+#define DX_HOST_KFDE2_VALID_VALUE_BIT_SIZE 0x1UL
+#define DX_HOST_KFDE3_VALID_REG_OFFSET 0xA6CUL
+#define DX_HOST_KFDE3_VALID_VALUE_BIT_SHIFT 0x0UL
+#define DX_HOST_KFDE3_VALID_VALUE_BIT_SIZE 0x1UL
+#define DX_HOST_GPR0_REG_OFFSET 0xA70UL
+#define DX_HOST_GPR0_VALUE_BIT_SHIFT 0x0UL
+#define DX_HOST_GPR0_VALUE_BIT_SIZE 0x20UL
+#define DX_GPR_HOST_REG_OFFSET 0xA74UL
+#define DX_GPR_HOST_VALUE_BIT_SHIFT 0x0UL
+#define DX_GPR_HOST_VALUE_BIT_SIZE 0x20UL
+#define DX_HOST_POWER_DOWN_EN_REG_OFFSET 0xA78UL
+#define DX_HOST_POWER_DOWN_EN_VALUE_BIT_SHIFT 0x0UL
+#define DX_HOST_POWER_DOWN_EN_VALUE_BIT_SIZE 0x1UL
// --------------------------------------
// BLOCK: HOST_SRAM
// --------------------------------------
-#define DX_SRAM_DATA_REG_OFFSET 0xF00UL
-#define DX_SRAM_DATA_VALUE_BIT_SHIFT 0x0UL
-#define DX_SRAM_DATA_VALUE_BIT_SIZE 0x20UL
-#define DX_SRAM_ADDR_REG_OFFSET 0xF04UL
-#define DX_SRAM_ADDR_VALUE_BIT_SHIFT 0x0UL
-#define DX_SRAM_ADDR_VALUE_BIT_SIZE 0xFUL
-#define DX_SRAM_DATA_READY_REG_OFFSET 0xF08UL
-#define DX_SRAM_DATA_READY_VALUE_BIT_SHIFT 0x0UL
-#define DX_SRAM_DATA_READY_VALUE_BIT_SIZE 0x1UL
+#define DX_SRAM_DATA_REG_OFFSET 0xF00UL
+#define DX_SRAM_DATA_VALUE_BIT_SHIFT 0x0UL
+#define DX_SRAM_DATA_VALUE_BIT_SIZE 0x20UL
+#define DX_SRAM_ADDR_REG_OFFSET 0xF04UL
+#define DX_SRAM_ADDR_VALUE_BIT_SHIFT 0x0UL
+#define DX_SRAM_ADDR_VALUE_BIT_SIZE 0xFUL
+#define DX_SRAM_DATA_READY_REG_OFFSET 0xF08UL
+#define DX_SRAM_DATA_READY_VALUE_BIT_SHIFT 0x0UL
+#define DX_SRAM_DATA_READY_VALUE_BIT_SIZE 0x1UL
#endif //__DX_HOST_H__
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
#ifndef __DX_REG_BASE_HOST_H__
#define __DX_REG_BASE_HOST_H__
-/* Identify platform: Xilinx Zynq7000 ZC706 */
-#define DX_PLAT_ZYNQ7000 1
-#define DX_PLAT_ZYNQ7000_ZC706 1
-
#define DX_BASE_CC 0x80000000
-
-#define DX_BASE_ENV_REGS 0x40008000
-#define DX_BASE_ENV_CC_MEMORIES 0x40008000
-#define DX_BASE_ENV_PERF_RAM 0x40009000
-
#define DX_BASE_HOST_RGF 0x0UL
#define DX_BASE_CRY_KERNEL 0x0UL
#define DX_BASE_ROM 0x40000000
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
#define DX_DEV_SIGNATURE 0xDCC71200UL
-#define CC_HW_VERSION 0xef840015UL
+#define CC_HW_VERSION 0xef840015UL
#define DX_DEV_SHA_MAX 512
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
-#ifndef _HASH_DEFS_H__
-#define _HASH_DEFS_H__
+#ifndef _HASH_DEFS_H_
+#define _HASH_DEFS_H_
#include "cc_crypto_ctx.h"
-/* this files provides definitions required for hash engine drivers */
-#ifndef CC_CONFIG_HASH_SHA_512_SUPPORTED
-#define SEP_HASH_LENGTH_WORDS 2
-#else
-#define SEP_HASH_LENGTH_WORDS 4
-#endif
-
-#ifdef BIG__ENDIAN
-#define OPAD_CURRENT_LENGTH 0x40000000, 0x00000000 , 0x00000000, 0x00000000
-#define HASH_LARVAL_MD5 0x76543210, 0xFEDCBA98, 0x89ABCDEF, 0x01234567
-#define HASH_LARVAL_SHA1 0xF0E1D2C3, 0x76543210, 0xFEDCBA98, 0x89ABCDEF, 0x01234567
-#define HASH_LARVAL_SHA224 0XA44FFABE, 0XA78FF964, 0X11155868, 0X310BC0FF, 0X39590EF7, 0X17DD7030, 0X07D57C36, 0XD89E05C1
-#define HASH_LARVAL_SHA256 0X19CDE05B, 0XABD9831F, 0X8C68059B, 0X7F520E51, 0X3AF54FA5, 0X72F36E3C, 0X85AE67BB, 0X67E6096A
-#define HASH_LARVAL_SHA384 0X1D48B547, 0XA44FFABE, 0X0D2E0CDB, 0XA78FF964, 0X874AB48E, 0X11155868, 0X67263367, 0X310BC0FF, 0XD8EC2F15, 0X39590EF7, 0X5A015991, 0X17DD7030, 0X2A299A62, 0X07D57C36, 0X5D9DBBCB, 0XD89E05C1
-#define HASH_LARVAL_SHA512 0X19CDE05B, 0X79217E13, 0XABD9831F, 0X6BBD41FB, 0X8C68059B, 0X1F6C3E2B, 0X7F520E51, 0XD182E6AD, 0X3AF54FA5, 0XF1361D5F, 0X72F36E3C, 0X2BF894FE, 0X85AE67BB, 0X3BA7CA84, 0X67E6096A, 0X08C9BCF3
-#else
-#define OPAD_CURRENT_LENGTH 0x00000040, 0x00000000, 0x00000000, 0x00000000
-#define HASH_LARVAL_MD5 0x10325476, 0x98BADCFE, 0xEFCDAB89, 0x67452301
-#define HASH_LARVAL_SHA1 0xC3D2E1F0, 0x10325476, 0x98BADCFE, 0xEFCDAB89, 0x67452301
-#define HASH_LARVAL_SHA224 0xbefa4fa4, 0x64f98fa7, 0x68581511, 0xffc00b31, 0xf70e5939, 0x3070dd17, 0x367cd507, 0xc1059ed8
-#define HASH_LARVAL_SHA256 0x5be0cd19, 0x1f83d9ab, 0x9b05688c, 0x510e527f, 0xa54ff53a, 0x3c6ef372, 0xbb67ae85, 0x6a09e667
-#define HASH_LARVAL_SHA384 0X47B5481D, 0XBEFA4FA4, 0XDB0C2E0D, 0X64F98FA7, 0X8EB44A87, 0X68581511, 0X67332667, 0XFFC00B31, 0X152FECD8, 0XF70E5939, 0X9159015A, 0X3070DD17, 0X629A292A, 0X367CD507, 0XCBBB9D5D, 0XC1059ED8
-#define HASH_LARVAL_SHA512 0x5be0cd19, 0x137e2179, 0x1f83d9ab, 0xfb41bd6b, 0x9b05688c, 0x2b3e6c1f, 0x510e527f, 0xade682d1, 0xa54ff53a, 0x5f1d36f1, 0x3c6ef372, 0xfe94f82b, 0xbb67ae85, 0x84caa73b, 0x6a09e667, 0xf3bcc908
-#endif
-
-enum HashConfig1Padding {
+enum cc_hash_conf_pad {
HASH_PADDING_DISABLED = 0,
HASH_PADDING_ENABLED = 1,
HASH_DIGEST_RESULT_LITTLE_ENDIAN = 2,
- HASH_CONFIG1_PADDING_RESERVE32 = INT32_MAX,
+ HASH_CONFIG1_PADDING_RESERVE32 = S32_MAX,
};
-enum HashCipherDoPadding {
+enum cc_hash_cipher_pad {
DO_NOT_PAD = 0,
DO_PAD = 1,
- HASH_CIPHER_DO_PADDING_RESERVE32 = INT32_MAX,
+ HASH_CIPHER_DO_PADDING_RESERVE32 = S32_MAX,
};
-typedef struct SepHashPrivateContext {
- /* The current length is placed at the end of the context buffer because the hash
- context is used for all HMAC operations as well. HMAC context includes a 64 bytes
- K0 field. The size of struct drv_ctx_hash reserved field is 88/184 bytes depend if t
- he SHA512 is supported ( in this case teh context size is 256 bytes).
- The size of struct drv_ctx_hash reseved field is 20 or 52 depend if the SHA512 is supported.
- This means that this structure size (without the reserved field can be up to 20 bytes ,
- in case sha512 is not suppported it is 20 bytes (SEP_HASH_LENGTH_WORDS define to 2 ) and in the other
- case it is 28 (SEP_HASH_LENGTH_WORDS define to 4) */
- uint32_t reserved[(sizeof(struct drv_ctx_hash)/sizeof(uint32_t)) - SEP_HASH_LENGTH_WORDS - 3];
- uint32_t CurrentDigestedLength[SEP_HASH_LENGTH_WORDS];
- uint32_t KeyType;
- uint32_t dataCompleted;
- uint32_t hmacFinalization;
- /* no space left */
-} SepHashPrivateContext_s;
-
-#endif /*_HASH_DEFS_H__*/
+#endif /*_HASH_DEFS_H_*/
+++ /dev/null
-/*
- * Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, see <http://www.gnu.org/licenses/>.
- */
-
-#ifndef __HW_QUEUE_DEFS_PLAT_H__
-#define __HW_QUEUE_DEFS_PLAT_H__
-
-
-/*****************************/
-/* Descriptor packing macros */
-/*****************************/
-
-#define HW_QUEUE_FREE_SLOTS_GET() (CC_HAL_READ_REGISTER(CC_REG_OFFSET(CRY_KERNEL, DSCRPTR_QUEUE_CONTENT)) & HW_QUEUE_SLOTS_MAX)
-
-#define HW_QUEUE_POLL_QUEUE_UNTIL_FREE_SLOTS(seqLen) \
- do { \
- } while (HW_QUEUE_FREE_SLOTS_GET() < (seqLen))
-
-#define HW_DESC_PUSH_TO_QUEUE(pDesc) do { \
- LOG_HW_DESC(pDesc); \
- HW_DESC_DUMP(pDesc); \
- CC_HAL_WRITE_REGISTER(GET_HW_Q_DESC_WORD_IDX(0), (pDesc)->word[0]); \
- CC_HAL_WRITE_REGISTER(GET_HW_Q_DESC_WORD_IDX(1), (pDesc)->word[1]); \
- CC_HAL_WRITE_REGISTER(GET_HW_Q_DESC_WORD_IDX(2), (pDesc)->word[2]); \
- CC_HAL_WRITE_REGISTER(GET_HW_Q_DESC_WORD_IDX(3), (pDesc)->word[3]); \
- CC_HAL_WRITE_REGISTER(GET_HW_Q_DESC_WORD_IDX(4), (pDesc)->word[4]); \
- wmb(); \
- CC_HAL_WRITE_REGISTER(GET_HW_Q_DESC_WORD_IDX(5), (pDesc)->word[5]); \
-} while (0)
-
-#endif /*__HW_QUEUE_DEFS_PLAT_H__*/
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
#define AES_CCM_RFC4309_NONCE_SIZE 3
#define MAX_NONCE_SIZE CTR_RFC3686_NONCE_SIZE
-
/* Value of each ICV_CMP byte (of 8) in case of success */
-#define ICV_VERIF_OK 0x01
+#define ICV_VERIF_OK 0x01
struct ssi_aead_handle {
ssi_sram_addr_t sram_workspace_addr;
struct ssi_aead_ctx {
struct ssi_drvdata *drvdata;
- uint8_t ctr_nonce[MAX_NONCE_SIZE]; /* used for ctr3686 iv and aes ccm */
- uint8_t *enckey;
+ u8 ctr_nonce[MAX_NONCE_SIZE]; /* used for ctr3686 iv and aes ccm */
+ u8 *enckey;
dma_addr_t enckey_dma_addr;
union {
struct {
- uint8_t *padded_authkey;
- uint8_t *ipad_opad; /* IPAD, OPAD*/
+ u8 *padded_authkey;
+ u8 *ipad_opad; /* IPAD, OPAD*/
dma_addr_t padded_authkey_dma_addr;
dma_addr_t ipad_opad_dma_addr;
} hmac;
struct {
- uint8_t *xcbc_keys; /* K1,K2,K3 */
+ u8 *xcbc_keys; /* K1,K2,K3 */
dma_addr_t xcbc_keys_dma_addr;
} xcbc;
} auth_state;
unsigned int auth_keylen;
unsigned int authsize; /* Actual (reduced?) size of the MAC/ICv */
enum drv_cipher_mode cipher_mode;
- enum FlowMode flow_mode;
+ enum cc_flow_mode flow_mode;
enum drv_hash_mode auth_mode;
};
SSI_LOG_DEBUG("Clearing context @%p for %s\n",
crypto_aead_ctx(tfm), crypto_tfm_alg_name(&(tfm->base)));
- dev = &ctx->drvdata->plat_dev->dev;
+ dev = &ctx->drvdata->plat_dev->dev;
/* Unmap enckey buffer */
- if (ctx->enckey != NULL) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(ctx->enckey_dma_addr);
+ if (ctx->enckey) {
dma_free_coherent(dev, AES_MAX_KEY_SIZE, ctx->enckey, ctx->enckey_dma_addr);
SSI_LOG_DEBUG("Freed enckey DMA buffer enckey_dma_addr=0x%llX\n",
(unsigned long long)ctx->enckey_dma_addr);
ctx->enckey_dma_addr = 0;
ctx->enckey = NULL;
}
-
+
if (ctx->auth_mode == DRV_HASH_XCBC_MAC) { /* XCBC authetication */
- if (ctx->auth_state.xcbc.xcbc_keys != NULL) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(
- ctx->auth_state.xcbc.xcbc_keys_dma_addr);
+ if (ctx->auth_state.xcbc.xcbc_keys) {
dma_free_coherent(dev, CC_AES_128_BIT_KEY_SIZE * 3,
- ctx->auth_state.xcbc.xcbc_keys,
+ ctx->auth_state.xcbc.xcbc_keys,
ctx->auth_state.xcbc.xcbc_keys_dma_addr);
}
SSI_LOG_DEBUG("Freed xcbc_keys DMA buffer xcbc_keys_dma_addr=0x%llX\n",
ctx->auth_state.xcbc.xcbc_keys_dma_addr = 0;
ctx->auth_state.xcbc.xcbc_keys = NULL;
} else if (ctx->auth_mode != DRV_HASH_NULL) { /* HMAC auth. */
- if (ctx->auth_state.hmac.ipad_opad != NULL) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(
- ctx->auth_state.hmac.ipad_opad_dma_addr);
+ if (ctx->auth_state.hmac.ipad_opad) {
dma_free_coherent(dev, 2 * MAX_HMAC_DIGEST_SIZE,
ctx->auth_state.hmac.ipad_opad,
ctx->auth_state.hmac.ipad_opad_dma_addr);
ctx->auth_state.hmac.ipad_opad_dma_addr = 0;
ctx->auth_state.hmac.ipad_opad = NULL;
}
- if (ctx->auth_state.hmac.padded_authkey != NULL) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(
- ctx->auth_state.hmac.padded_authkey_dma_addr);
+ if (ctx->auth_state.hmac.padded_authkey) {
dma_free_coherent(dev, MAX_HMAC_BLOCK_SIZE,
ctx->auth_state.hmac.padded_authkey,
ctx->auth_state.hmac.padded_authkey_dma_addr);
ctx->auth_mode = ssi_alg->auth_mode;
ctx->drvdata = ssi_alg->drvdata;
dev = &ctx->drvdata->plat_dev->dev;
- crypto_aead_set_reqsize(tfm,sizeof(struct aead_req_ctx));
+ crypto_aead_set_reqsize(tfm, sizeof(struct aead_req_ctx));
/* Allocate key buffer, cache line aligned */
ctx->enckey = dma_alloc_coherent(dev, AES_MAX_KEY_SIZE,
&ctx->enckey_dma_addr, GFP_KERNEL);
- if (ctx->enckey == NULL) {
+ if (!ctx->enckey) {
SSI_LOG_ERR("Failed allocating key buffer\n");
goto init_failed;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(ctx->enckey_dma_addr, AES_MAX_KEY_SIZE);
SSI_LOG_DEBUG("Allocated enckey buffer in context ctx->enckey=@%p\n", ctx->enckey);
/* Set default authlen value */
ctx->auth_state.xcbc.xcbc_keys = dma_alloc_coherent(dev,
CC_AES_128_BIT_KEY_SIZE * 3,
&ctx->auth_state.xcbc.xcbc_keys_dma_addr, GFP_KERNEL);
- if (ctx->auth_state.xcbc.xcbc_keys == NULL) {
+ if (!ctx->auth_state.xcbc.xcbc_keys) {
SSI_LOG_ERR("Failed allocating buffer for XCBC keys\n");
goto init_failed;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(
- ctx->auth_state.xcbc.xcbc_keys_dma_addr,
- CC_AES_128_BIT_KEY_SIZE * 3);
} else if (ctx->auth_mode != DRV_HASH_NULL) { /* HMAC authentication */
/* Allocate dma-coherent buffer for IPAD + OPAD */
ctx->auth_state.hmac.ipad_opad = dma_alloc_coherent(dev,
2 * MAX_HMAC_DIGEST_SIZE,
&ctx->auth_state.hmac.ipad_opad_dma_addr, GFP_KERNEL);
- if (ctx->auth_state.hmac.ipad_opad == NULL) {
+ if (!ctx->auth_state.hmac.ipad_opad) {
SSI_LOG_ERR("Failed allocating IPAD/OPAD buffer\n");
goto init_failed;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(
- ctx->auth_state.hmac.ipad_opad_dma_addr,
- 2 * MAX_HMAC_DIGEST_SIZE);
SSI_LOG_DEBUG("Allocated authkey buffer in context ctx->authkey=@%p\n",
ctx->auth_state.hmac.ipad_opad);
-
+
ctx->auth_state.hmac.padded_authkey = dma_alloc_coherent(dev,
MAX_HMAC_BLOCK_SIZE,
&ctx->auth_state.hmac.padded_authkey_dma_addr, GFP_KERNEL);
- if (ctx->auth_state.hmac.padded_authkey == NULL) {
+ if (!ctx->auth_state.hmac.padded_authkey) {
SSI_LOG_ERR("failed to allocate padded_authkey\n");
goto init_failed;
- }
- SSI_UPDATE_DMA_ADDR_TO_48BIT(
- ctx->auth_state.hmac.padded_authkey_dma_addr,
- MAX_HMAC_BLOCK_SIZE);
+ }
} else {
ctx->auth_state.hmac.ipad_opad = NULL;
ctx->auth_state.hmac.padded_authkey = NULL;
ssi_aead_exit(tfm);
return -ENOMEM;
}
-
static void ssi_aead_complete(struct device *dev, void *ssi_req, void __iomem *cc_base)
{
struct crypto_aead *tfm = crypto_aead_reqtfm(ssi_req);
struct ssi_aead_ctx *ctx = crypto_aead_ctx(tfm);
int err = 0;
- DECL_CYCLE_COUNT_RESOURCES;
-
- START_CYCLE_COUNT();
ssi_buffer_mgr_unmap_aead_request(dev, areq);
"(auth-size=%d, cipher=%d).\n",
ctx->authsize, ctx->cipher_mode);
/* In case of payload authentication failure, MUST NOT
- revealed the decrypted message --> zero its memory. */
+ * revealed the decrypted message --> zero its memory.
+ */
ssi_buffer_mgr_zero_sgl(areq->dst, areq_ctx->cryptlen);
err = -EBADMSG;
}
} else { /*ENCRYPT*/
- if (unlikely(areq_ctx->is_icv_fragmented == true))
+ if (unlikely(areq_ctx->is_icv_fragmented))
ssi_buffer_mgr_copy_scatterlist_portion(
- areq_ctx->mac_buf, areq_ctx->dstSgl, areq->cryptlen+areq_ctx->dstOffset,
- areq->cryptlen+areq_ctx->dstOffset + ctx->authsize, SSI_SG_FROM_BUF);
+ areq_ctx->mac_buf, areq_ctx->dstSgl, areq->cryptlen + areq_ctx->dstOffset,
+ areq->cryptlen + areq_ctx->dstOffset + ctx->authsize, SSI_SG_FROM_BUF);
/* If an IV was generated, copy it back to the user provided buffer. */
- if (areq_ctx->backup_giv != NULL) {
- if (ctx->cipher_mode == DRV_CIPHER_CTR) {
+ if (areq_ctx->backup_giv) {
+ if (ctx->cipher_mode == DRV_CIPHER_CTR)
memcpy(areq_ctx->backup_giv, areq_ctx->ctr_iv + CTR_RFC3686_NONCE_SIZE, CTR_RFC3686_IV_SIZE);
- } else if (ctx->cipher_mode == DRV_CIPHER_CCM) {
+ else if (ctx->cipher_mode == DRV_CIPHER_CCM)
memcpy(areq_ctx->backup_giv, areq_ctx->ctr_iv + CCM_BLOCK_IV_OFFSET, CCM_BLOCK_IV_SIZE);
- }
}
}
- END_CYCLE_COUNT(STAT_OP_TYPE_GENERIC, STAT_PHASE_4);
aead_request_complete(areq, err);
}
-static int xcbc_setkey(HwDesc_s *desc, struct ssi_aead_ctx *ctx)
+static int xcbc_setkey(struct cc_hw_desc *desc, struct ssi_aead_ctx *ctx)
{
/* Load the AES key */
- HW_DESC_INIT(&desc[0]);
+ hw_desc_init(&desc[0]);
/* We are using for the source/user key the same buffer as for the output keys,
- because after this key loading it is not needed anymore */
- HW_DESC_SET_DIN_TYPE(&desc[0], DMA_DLLI, ctx->auth_state.xcbc.xcbc_keys_dma_addr, ctx->auth_keylen, NS_BIT);
- HW_DESC_SET_CIPHER_MODE(&desc[0], DRV_CIPHER_ECB);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[0], DRV_CRYPTO_DIRECTION_ENCRYPT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[0], ctx->auth_keylen);
- HW_DESC_SET_FLOW_MODE(&desc[0], S_DIN_to_AES);
- HW_DESC_SET_SETUP_MODE(&desc[0], SETUP_LOAD_KEY0);
-
- HW_DESC_INIT(&desc[1]);
- HW_DESC_SET_DIN_CONST(&desc[1], 0x01010101, CC_AES_128_BIT_KEY_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[1], DIN_AES_DOUT);
- HW_DESC_SET_DOUT_DLLI(&desc[1], ctx->auth_state.xcbc.xcbc_keys_dma_addr, AES_KEYSIZE_128, NS_BIT, 0);
-
- HW_DESC_INIT(&desc[2]);
- HW_DESC_SET_DIN_CONST(&desc[2], 0x02020202, CC_AES_128_BIT_KEY_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[2], DIN_AES_DOUT);
- HW_DESC_SET_DOUT_DLLI(&desc[2], (ctx->auth_state.xcbc.xcbc_keys_dma_addr
+ * because after this key loading it is not needed anymore
+ */
+ set_din_type(&desc[0], DMA_DLLI,
+ ctx->auth_state.xcbc.xcbc_keys_dma_addr, ctx->auth_keylen,
+ NS_BIT);
+ set_cipher_mode(&desc[0], DRV_CIPHER_ECB);
+ set_cipher_config0(&desc[0], DRV_CRYPTO_DIRECTION_ENCRYPT);
+ set_key_size_aes(&desc[0], ctx->auth_keylen);
+ set_flow_mode(&desc[0], S_DIN_to_AES);
+ set_setup_mode(&desc[0], SETUP_LOAD_KEY0);
+
+ hw_desc_init(&desc[1]);
+ set_din_const(&desc[1], 0x01010101, CC_AES_128_BIT_KEY_SIZE);
+ set_flow_mode(&desc[1], DIN_AES_DOUT);
+ set_dout_dlli(&desc[1], ctx->auth_state.xcbc.xcbc_keys_dma_addr,
+ AES_KEYSIZE_128, NS_BIT, 0);
+
+ hw_desc_init(&desc[2]);
+ set_din_const(&desc[2], 0x02020202, CC_AES_128_BIT_KEY_SIZE);
+ set_flow_mode(&desc[2], DIN_AES_DOUT);
+ set_dout_dlli(&desc[2], (ctx->auth_state.xcbc.xcbc_keys_dma_addr
+ AES_KEYSIZE_128),
AES_KEYSIZE_128, NS_BIT, 0);
- HW_DESC_INIT(&desc[3]);
- HW_DESC_SET_DIN_CONST(&desc[3], 0x03030303, CC_AES_128_BIT_KEY_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[3], DIN_AES_DOUT);
- HW_DESC_SET_DOUT_DLLI(&desc[3], (ctx->auth_state.xcbc.xcbc_keys_dma_addr
+ hw_desc_init(&desc[3]);
+ set_din_const(&desc[3], 0x03030303, CC_AES_128_BIT_KEY_SIZE);
+ set_flow_mode(&desc[3], DIN_AES_DOUT);
+ set_dout_dlli(&desc[3], (ctx->auth_state.xcbc.xcbc_keys_dma_addr
+ 2 * AES_KEYSIZE_128),
AES_KEYSIZE_128, NS_BIT, 0);
return 4;
}
-static int hmac_setkey(HwDesc_s *desc, struct ssi_aead_ctx *ctx)
+static int hmac_setkey(struct cc_hw_desc *desc, struct ssi_aead_ctx *ctx)
{
unsigned int hmacPadConst[2] = { HMAC_IPAD_CONST, HMAC_OPAD_CONST };
unsigned int digest_ofs = 0;
- unsigned int hash_mode = (ctx->auth_mode == DRV_HASH_SHA1) ?
+ unsigned int hash_mode = (ctx->auth_mode == DRV_HASH_SHA1) ?
DRV_HASH_HW_SHA1 : DRV_HASH_HW_SHA256;
- unsigned int digest_size = (ctx->auth_mode == DRV_HASH_SHA1) ?
+ unsigned int digest_size = (ctx->auth_mode == DRV_HASH_SHA1) ?
CC_SHA1_DIGEST_SIZE : CC_SHA256_DIGEST_SIZE;
int idx = 0;
/* calc derived HMAC key */
for (i = 0; i < 2; i++) {
/* Load hash initial state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hash_mode);
- HW_DESC_SET_DIN_SRAM(&desc[idx],
- ssi_ahash_get_larval_digest_sram_addr(
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hash_mode);
+ set_din_sram(&desc[idx],
+ ssi_ahash_get_larval_digest_sram_addr(
ctx->drvdata, ctx->auth_mode),
- digest_size);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
+ digest_size);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
idx++;
/* Load the hash current length*/
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hash_mode);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0, HASH_LEN_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hash_mode);
+ set_din_const(&desc[idx], 0, HASH_LEN_SIZE);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
/* Prepare ipad key */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_XOR_VAL(&desc[idx], hmacPadConst[i]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hash_mode);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE1);
+ hw_desc_init(&desc[idx]);
+ set_xor_val(&desc[idx], hmacPadConst[i]);
+ set_cipher_mode(&desc[idx], hash_mode);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE1);
idx++;
/* Perform HASH update */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- ctx->auth_state.hmac.padded_authkey_dma_addr,
- SHA256_BLOCK_SIZE, NS_BIT);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hash_mode);
- HW_DESC_SET_XOR_ACTIVE(&desc[idx]);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_HASH);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI,
+ ctx->auth_state.hmac.padded_authkey_dma_addr,
+ SHA256_BLOCK_SIZE, NS_BIT);
+ set_cipher_mode(&desc[idx], hash_mode);
+ set_xor_active(&desc[idx]);
+ set_flow_mode(&desc[idx], DIN_HASH);
idx++;
/* Get the digset */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hash_mode);
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- (ctx->auth_state.hmac.ipad_opad_dma_addr +
- digest_ofs),
- digest_size, NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_DISABLED);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hash_mode);
+ set_dout_dlli(&desc[idx],
+ (ctx->auth_state.hmac.ipad_opad_dma_addr +
+ digest_ofs), digest_size, NS_BIT, 0);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
+ set_cipher_config1(&desc[idx], HASH_PADDING_DISABLED);
idx++;
digest_ofs += digest_size;
return 0; /* All tests of keys sizes passed */
}
-/*This function prepers the user key so it can pass to the hmac processing
- (copy to intenral buffer or hash in case of key longer than block */
+
+/* This function prepers the user key so it can pass to the hmac processing
+ * (copy to intenral buffer or hash in case of key longer than block
+ */
static int
ssi_get_plain_hmac_key(struct crypto_aead *tfm, const u8 *key, unsigned int keylen)
{
dma_addr_t key_dma_addr = 0;
struct ssi_aead_ctx *ctx = crypto_aead_ctx(tfm);
struct device *dev = &ctx->drvdata->plat_dev->dev;
- uint32_t larval_addr = ssi_ahash_get_larval_digest_sram_addr(
+ u32 larval_addr = ssi_ahash_get_larval_digest_sram_addr(
ctx->drvdata, ctx->auth_mode);
struct ssi_crypto_req ssi_req = {};
unsigned int blocksize;
unsigned int hashmode;
unsigned int idx = 0;
int rc = 0;
- HwDesc_s desc[MAX_AEAD_SETKEY_SEQ];
- dma_addr_t padded_authkey_dma_addr =
+ struct cc_hw_desc desc[MAX_AEAD_SETKEY_SEQ];
+ dma_addr_t padded_authkey_dma_addr =
ctx->auth_state.hmac.padded_authkey_dma_addr;
switch (ctx->auth_mode) { /* auth_key required and >0 */
" DMA failed\n", key, keylen);
return -ENOMEM;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(key_dma_addr, keylen);
if (keylen > blocksize) {
/* Load hash initial state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hashmode);
- HW_DESC_SET_DIN_SRAM(&desc[idx], larval_addr, digestsize);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hashmode);
+ set_din_sram(&desc[idx], larval_addr, digestsize);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
idx++;
-
+
/* Load the hash current length*/
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hashmode);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0, HASH_LEN_SIZE);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_ENABLED);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hashmode);
+ set_din_const(&desc[idx], 0, HASH_LEN_SIZE);
+ set_cipher_config1(&desc[idx], HASH_PADDING_ENABLED);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
-
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- key_dma_addr,
- keylen, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_HASH);
+
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI,
+ key_dma_addr, keylen, NS_BIT);
+ set_flow_mode(&desc[idx], DIN_HASH);
idx++;
-
+
/* Get hashed key */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hashmode);
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- padded_authkey_dma_addr,
- digestsize,
- NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx],
- HASH_PADDING_DISABLED);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx],
- HASH_DIGEST_RESULT_LITTLE_ENDIAN);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hashmode);
+ set_dout_dlli(&desc[idx], padded_authkey_dma_addr,
+ digestsize, NS_BIT, 0);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
+ set_cipher_config1(&desc[idx], HASH_PADDING_DISABLED);
+ set_cipher_config0(&desc[idx],
+ HASH_DIGEST_RESULT_LITTLE_ENDIAN);
idx++;
-
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0, (blocksize - digestsize));
- HW_DESC_SET_FLOW_MODE(&desc[idx], BYPASS);
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- (padded_authkey_dma_addr + digestsize),
- (blocksize - digestsize),
- NS_BIT, 0);
+
+ hw_desc_init(&desc[idx]);
+ set_din_const(&desc[idx], 0, (blocksize - digestsize));
+ set_flow_mode(&desc[idx], BYPASS);
+ set_dout_dlli(&desc[idx], (padded_authkey_dma_addr +
+ digestsize), (blocksize - digestsize),
+ NS_BIT, 0);
idx++;
} else {
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- key_dma_addr,
- keylen, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], BYPASS);
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- (padded_authkey_dma_addr),
- keylen, NS_BIT, 0);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, key_dma_addr,
+ keylen, NS_BIT);
+ set_flow_mode(&desc[idx], BYPASS);
+ set_dout_dlli(&desc[idx], padded_authkey_dma_addr,
+ keylen, NS_BIT, 0);
idx++;
-
+
if ((blocksize - keylen) != 0) {
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0,
- (blocksize - keylen));
- HW_DESC_SET_FLOW_MODE(&desc[idx], BYPASS);
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- (padded_authkey_dma_addr + keylen),
- (blocksize - keylen),
- NS_BIT, 0);
+ hw_desc_init(&desc[idx]);
+ set_din_const(&desc[idx], 0,
+ (blocksize - keylen));
+ set_flow_mode(&desc[idx], BYPASS);
+ set_dout_dlli(&desc[idx],
+ (padded_authkey_dma_addr +
+ keylen),
+ (blocksize - keylen), NS_BIT, 0);
idx++;
}
}
} else {
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0,
- (blocksize - keylen));
- HW_DESC_SET_FLOW_MODE(&desc[idx], BYPASS);
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- padded_authkey_dma_addr,
- blocksize,
- NS_BIT, 0);
+ hw_desc_init(&desc[idx]);
+ set_din_const(&desc[idx], 0, (blocksize - keylen));
+ set_flow_mode(&desc[idx], BYPASS);
+ set_dout_dlli(&desc[idx], padded_authkey_dma_addr,
+ blocksize, NS_BIT, 0);
idx++;
}
-#ifdef ENABLE_CYCLE_COUNT
- ssi_req.op_type = STAT_OP_TYPE_SETKEY;
-#endif
-
rc = send_request(ctx->drvdata, &ssi_req, desc, idx, 0);
if (unlikely(rc != 0))
SSI_LOG_ERR("send_request() failed (rc=%d)\n", rc);
- if (likely(key_dma_addr != 0)) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(key_dma_addr);
+ if (likely(key_dma_addr != 0))
dma_unmap_single(dev, key_dma_addr, keylen, DMA_TO_DEVICE);
- }
return rc;
}
-
static int
ssi_aead_setkey(struct crypto_aead *tfm, const u8 *key, unsigned int keylen)
{
struct rtattr *rta = (struct rtattr *)key;
struct ssi_crypto_req ssi_req = {};
struct crypto_authenc_key_param *param;
- HwDesc_s desc[MAX_AEAD_SETKEY_SEQ];
+ struct cc_hw_desc desc[MAX_AEAD_SETKEY_SEQ];
int seq_len = 0, rc = -EINVAL;
- DECL_CYCLE_COUNT_RESOURCES;
SSI_LOG_DEBUG("Setting key in context @%p for %s. key=%p keylen=%u\n",
ctx, crypto_tfm_alg_name(crypto_aead_tfm(tfm)), key, keylen);
CHECK_AND_RETURN_UPON_FIPS_ERROR();
/* STAT_PHASE_0: Init and sanity checks */
- START_CYCLE_COUNT();
if (ctx->auth_mode != DRV_HASH_NULL) { /* authenc() alg. */
if (!RTA_OK(rta, keylen))
(AES_MIN_KEY_SIZE + CTR_RFC3686_NONCE_SIZE))
goto badkey;
/* Copy nonce from last 4 bytes in CTR key to
- * first 4 bytes in CTR IV */
+ * first 4 bytes in CTR IV
+ */
memcpy(ctx->ctr_nonce, key + ctx->auth_keylen + ctx->enc_keylen -
CTR_RFC3686_NONCE_SIZE, CTR_RFC3686_NONCE_SIZE);
/* Set CTR key size */
if (unlikely(rc != 0))
goto badkey;
- END_CYCLE_COUNT(STAT_OP_TYPE_SETKEY, STAT_PHASE_0);
/* STAT_PHASE_1: Copy key to ctx */
- START_CYCLE_COUNT();
/* Get key material */
memcpy(ctx->enckey, key + ctx->auth_keylen, ctx->enc_keylen);
goto badkey;
}
- END_CYCLE_COUNT(STAT_OP_TYPE_SETKEY, STAT_PHASE_1);
-
/* STAT_PHASE_2: Create sequence */
- START_CYCLE_COUNT();
switch (ctx->auth_mode) {
case DRV_HASH_SHA1:
goto badkey;
}
- END_CYCLE_COUNT(STAT_OP_TYPE_SETKEY, STAT_PHASE_2);
-
/* STAT_PHASE_3: Submit sequence to HW */
- START_CYCLE_COUNT();
-
+
if (seq_len > 0) { /* For CCM there is no sequence to setup the key */
-#ifdef ENABLE_CYCLE_COUNT
- ssi_req.op_type = STAT_OP_TYPE_SETKEY;
-#endif
rc = send_request(ctx->drvdata, &ssi_req, desc, seq_len, 0);
if (unlikely(rc != 0)) {
SSI_LOG_ERR("send_request() failed (rc=%d)\n", rc);
}
/* Update STAT_PHASE_3 */
- END_CYCLE_COUNT(STAT_OP_TYPE_SETKEY, STAT_PHASE_3);
return rc;
badkey:
{
struct ssi_aead_ctx *ctx = crypto_aead_ctx(tfm);
int rc = 0;
-
+
if (keylen < 3)
return -EINVAL;
unsigned int authsize)
{
struct ssi_aead_ctx *ctx = crypto_aead_ctx(authenc);
-
+
CHECK_AND_RETURN_UPON_FIPS_ERROR();
/* Unsupported auth. sizes */
if ((authsize == 0) ||
- (authsize >crypto_aead_maxauthsize(authenc))) {
+ (authsize > crypto_aead_maxauthsize(authenc))) {
return -ENOTSUPP;
}
}
#endif /*SSI_CC_HAS_AES_CCM*/
-static inline void
+static inline void
ssi_aead_create_assoc_desc(
- struct aead_request *areq,
+ struct aead_request *areq,
unsigned int flow_mode,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
struct crypto_aead *tfm = crypto_aead_reqtfm(areq);
switch (assoc_dma_type) {
case SSI_DMA_BUF_DLLI:
SSI_LOG_DEBUG("ASSOC buffer type DLLI\n");
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- sg_dma_address(areq->src),
- areq->assoclen, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], flow_mode);
- if (ctx->auth_mode == DRV_HASH_XCBC_MAC && (areq_ctx->cryptlen > 0) )
- HW_DESC_SET_DIN_NOT_LAST_INDICATION(&desc[idx]);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, sg_dma_address(areq->src),
+ areq->assoclen, NS_BIT); set_flow_mode(&desc[idx],
+ flow_mode);
+ if ((ctx->auth_mode == DRV_HASH_XCBC_MAC) &&
+ (areq_ctx->cryptlen > 0))
+ set_din_not_last_indication(&desc[idx]);
break;
case SSI_DMA_BUF_MLLI:
SSI_LOG_DEBUG("ASSOC buffer type MLLI\n");
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_MLLI,
- areq_ctx->assoc.sram_addr,
- areq_ctx->assoc.mlli_nents,
- NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], flow_mode);
- if (ctx->auth_mode == DRV_HASH_XCBC_MAC && (areq_ctx->cryptlen > 0) )
- HW_DESC_SET_DIN_NOT_LAST_INDICATION(&desc[idx]);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_MLLI, areq_ctx->assoc.sram_addr,
+ areq_ctx->assoc.mlli_nents, NS_BIT);
+ set_flow_mode(&desc[idx], flow_mode);
+ if ((ctx->auth_mode == DRV_HASH_XCBC_MAC) &&
+ (areq_ctx->cryptlen > 0))
+ set_din_not_last_indication(&desc[idx]);
break;
case SSI_DMA_BUF_NULL:
default:
static inline void
ssi_aead_process_authenc_data_desc(
- struct aead_request *areq,
+ struct aead_request *areq,
unsigned int flow_mode,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size,
int direct)
{
(direct == DRV_CRYPTO_DIRECTION_ENCRYPT) ?
areq_ctx->dstSgl : areq_ctx->srcSgl;
- unsigned int offset =
+ unsigned int offset =
(direct == DRV_CRYPTO_DIRECTION_ENCRYPT) ?
areq_ctx->dstOffset : areq_ctx->srcOffset;
SSI_LOG_DEBUG("AUTHENC: SRC/DST buffer type DLLI\n");
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- (sg_dma_address(cipher)+ offset), areq_ctx->cryptlen,
- NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], flow_mode);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI,
+ (sg_dma_address(cipher) + offset),
+ areq_ctx->cryptlen, NS_BIT);
+ set_flow_mode(&desc[idx], flow_mode);
break;
}
case SSI_DMA_BUF_MLLI:
{
/* DOUBLE-PASS flow (as default)
* assoc. + iv + data -compact in one table
- * if assoclen is ZERO only IV perform */
+ * if assoclen is ZERO only IV perform
+ */
ssi_sram_addr_t mlli_addr = areq_ctx->assoc.sram_addr;
- uint32_t mlli_nents = areq_ctx->assoc.mlli_nents;
+ u32 mlli_nents = areq_ctx->assoc.mlli_nents;
- if (likely(areq_ctx->is_single_pass == true)) {
- if (direct == DRV_CRYPTO_DIRECTION_ENCRYPT){
+ if (likely(areq_ctx->is_single_pass)) {
+ if (direct == DRV_CRYPTO_DIRECTION_ENCRYPT) {
mlli_addr = areq_ctx->dst.sram_addr;
mlli_nents = areq_ctx->dst.mlli_nents;
} else {
}
SSI_LOG_DEBUG("AUTHENC: SRC/DST buffer type MLLI\n");
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_MLLI,
- mlli_addr, mlli_nents, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], flow_mode);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_MLLI, mlli_addr, mlli_nents,
+ NS_BIT);
+ set_flow_mode(&desc[idx], flow_mode);
break;
}
case SSI_DMA_BUF_NULL:
static inline void
ssi_aead_process_cipher_data_desc(
- struct aead_request *areq,
+ struct aead_request *areq,
unsigned int flow_mode,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
unsigned int idx = *seq_size;
switch (data_dma_type) {
case SSI_DMA_BUF_DLLI:
SSI_LOG_DEBUG("CIPHER: SRC/DST buffer type DLLI\n");
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- (sg_dma_address(areq_ctx->srcSgl)+areq_ctx->srcOffset),
- areq_ctx->cryptlen, NS_BIT);
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- (sg_dma_address(areq_ctx->dstSgl)+areq_ctx->dstOffset),
- areq_ctx->cryptlen, NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], flow_mode);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI,
+ (sg_dma_address(areq_ctx->srcSgl) +
+ areq_ctx->srcOffset), areq_ctx->cryptlen, NS_BIT);
+ set_dout_dlli(&desc[idx],
+ (sg_dma_address(areq_ctx->dstSgl) +
+ areq_ctx->dstOffset),
+ areq_ctx->cryptlen, NS_BIT, 0);
+ set_flow_mode(&desc[idx], flow_mode);
break;
case SSI_DMA_BUF_MLLI:
SSI_LOG_DEBUG("CIPHER: SRC/DST buffer type MLLI\n");
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_MLLI,
- areq_ctx->src.sram_addr,
- areq_ctx->src.mlli_nents, NS_BIT);
- HW_DESC_SET_DOUT_MLLI(&desc[idx],
- areq_ctx->dst.sram_addr,
- areq_ctx->dst.mlli_nents, NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], flow_mode);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_MLLI, areq_ctx->src.sram_addr,
+ areq_ctx->src.mlli_nents, NS_BIT);
+ set_dout_mlli(&desc[idx], areq_ctx->dst.sram_addr,
+ areq_ctx->dst.mlli_nents, NS_BIT, 0);
+ set_flow_mode(&desc[idx], flow_mode);
break;
case SSI_DMA_BUF_NULL:
default:
static inline void ssi_aead_process_digest_result_desc(
struct aead_request *req,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
/* Get final ICV result */
if (direct == DRV_CRYPTO_DIRECTION_ENCRYPT) {
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], req_ctx->icv_dma_addr,
- ctx->authsize, NS_BIT, 1);
- HW_DESC_SET_QUEUE_LAST_IND(&desc[idx]);
+ hw_desc_init(&desc[idx]);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
+ set_dout_dlli(&desc[idx], req_ctx->icv_dma_addr, ctx->authsize,
+ NS_BIT, 1);
+ set_queue_last_ind(&desc[idx]);
if (ctx->auth_mode == DRV_HASH_XCBC_MAC) {
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_XCBC_MAC);
+ set_aes_not_hash_mode(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_XCBC_MAC);
} else {
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx],
- HASH_DIGEST_RESULT_LITTLE_ENDIAN);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hash_mode);
+ set_cipher_config0(&desc[idx],
+ HASH_DIGEST_RESULT_LITTLE_ENDIAN);
+ set_cipher_mode(&desc[idx], hash_mode);
}
} else { /*Decrypt*/
/* Get ICV out from hardware */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], req_ctx->mac_buf_dma_addr,
- ctx->authsize, NS_BIT, 1);
- HW_DESC_SET_QUEUE_LAST_IND(&desc[idx]);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], HASH_DIGEST_RESULT_LITTLE_ENDIAN);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_DISABLED);
+ hw_desc_init(&desc[idx]);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_dout_dlli(&desc[idx], req_ctx->mac_buf_dma_addr,
+ ctx->authsize, NS_BIT, 1);
+ set_queue_last_ind(&desc[idx]);
+ set_cipher_config0(&desc[idx],
+ HASH_DIGEST_RESULT_LITTLE_ENDIAN);
+ set_cipher_config1(&desc[idx], HASH_PADDING_DISABLED);
if (ctx->auth_mode == DRV_HASH_XCBC_MAC) {
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_XCBC_MAC);
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_XCBC_MAC);
+ set_aes_not_hash_mode(&desc[idx]);
} else {
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hash_mode);
+ set_cipher_mode(&desc[idx], hash_mode);
}
}
static inline void ssi_aead_setup_cipher_desc(
struct aead_request *req,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
int direct = req_ctx->gen_ctx.op_type;
/* Setup cipher state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], direct);
- HW_DESC_SET_FLOW_MODE(&desc[idx], ctx->flow_mode);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- req_ctx->gen_ctx.iv_dma_addr, hw_iv_size, NS_BIT);
- if (ctx->cipher_mode == DRV_CIPHER_CTR) {
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE1);
- } else {
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
- }
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->cipher_mode);
+ hw_desc_init(&desc[idx]);
+ set_cipher_config0(&desc[idx], direct);
+ set_flow_mode(&desc[idx], ctx->flow_mode);
+ set_din_type(&desc[idx], DMA_DLLI, req_ctx->gen_ctx.iv_dma_addr,
+ hw_iv_size, NS_BIT);
+ if (ctx->cipher_mode == DRV_CIPHER_CTR)
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE1);
+ else
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
+ set_cipher_mode(&desc[idx], ctx->cipher_mode);
idx++;
/* Setup enc. key */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], direct);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], ctx->flow_mode);
+ hw_desc_init(&desc[idx]);
+ set_cipher_config0(&desc[idx], direct);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
+ set_flow_mode(&desc[idx], ctx->flow_mode);
if (ctx->flow_mode == S_DIN_to_AES) {
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, ctx->enckey_dma_addr,
- ((ctx->enc_keylen == 24) ?
- CC_AES_KEY_SIZE_MAX : ctx->enc_keylen), NS_BIT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], ctx->enc_keylen);
+ set_din_type(&desc[idx], DMA_DLLI, ctx->enckey_dma_addr,
+ ((ctx->enc_keylen == 24) ? CC_AES_KEY_SIZE_MAX :
+ ctx->enc_keylen), NS_BIT);
+ set_key_size_aes(&desc[idx], ctx->enc_keylen);
} else {
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, ctx->enckey_dma_addr,
- ctx->enc_keylen, NS_BIT);
- HW_DESC_SET_KEY_SIZE_DES(&desc[idx], ctx->enc_keylen);
+ set_din_type(&desc[idx], DMA_DLLI, ctx->enckey_dma_addr,
+ ctx->enc_keylen, NS_BIT);
+ set_key_size_des(&desc[idx], ctx->enc_keylen);
}
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->cipher_mode);
+ set_cipher_mode(&desc[idx], ctx->cipher_mode);
idx++;
*seq_size = idx;
static inline void ssi_aead_process_cipher(
struct aead_request *req,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size,
unsigned int data_flow_mode)
{
ssi_aead_process_cipher_data_desc(req, data_flow_mode, desc, &idx);
if (direct == DRV_CRYPTO_DIRECTION_ENCRYPT) {
/* We must wait for DMA to write all cipher */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_NO_DMA(&desc[idx], 0, 0xfffff0);
- HW_DESC_SET_DOUT_NO_DMA(&desc[idx], 0, 0, 1);
+ hw_desc_init(&desc[idx]);
+ set_din_no_dma(&desc[idx], 0, 0xfffff0);
+ set_dout_no_dma(&desc[idx], 0, 0, 1);
idx++;
}
static inline void ssi_aead_hmac_setup_digest_desc(
struct aead_request *req,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct ssi_aead_ctx *ctx = crypto_aead_ctx(tfm);
unsigned int hash_mode = (ctx->auth_mode == DRV_HASH_SHA1) ?
DRV_HASH_HW_SHA1 : DRV_HASH_HW_SHA256;
- unsigned int digest_size = (ctx->auth_mode == DRV_HASH_SHA1) ?
+ unsigned int digest_size = (ctx->auth_mode == DRV_HASH_SHA1) ?
CC_SHA1_DIGEST_SIZE : CC_SHA256_DIGEST_SIZE;
unsigned int idx = *seq_size;
/* Loading hash ipad xor key state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hash_mode);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- ctx->auth_state.hmac.ipad_opad_dma_addr,
- digest_size, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hash_mode);
+ set_din_type(&desc[idx], DMA_DLLI,
+ ctx->auth_state.hmac.ipad_opad_dma_addr, digest_size,
+ NS_BIT);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
idx++;
/* Load init. digest len (64 bytes) */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hash_mode);
- HW_DESC_SET_DIN_SRAM(&desc[idx],
- ssi_ahash_get_initial_digest_len_sram_addr(ctx->drvdata, hash_mode),
- HASH_LEN_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hash_mode);
+ set_din_sram(&desc[idx],
+ ssi_ahash_get_initial_digest_len_sram_addr(ctx->drvdata,
+ hash_mode),
+ HASH_LEN_SIZE);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
*seq_size = idx;
static inline void ssi_aead_xcbc_setup_digest_desc(
struct aead_request *req,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
unsigned int idx = *seq_size;
/* Loading MAC state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0, CC_AES_BLOCK_SIZE);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_XCBC_MAC);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], CC_AES_128_BIT_KEY_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
+ hw_desc_init(&desc[idx]);
+ set_din_const(&desc[idx], 0, CC_AES_BLOCK_SIZE);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_XCBC_MAC);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_key_size_aes(&desc[idx], CC_AES_128_BIT_KEY_SIZE);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_aes_not_hash_mode(&desc[idx]);
idx++;
/* Setup XCBC MAC K1 */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- ctx->auth_state.xcbc.xcbc_keys_dma_addr,
- AES_KEYSIZE_128, NS_BIT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_XCBC_MAC);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], CC_AES_128_BIT_KEY_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI,
+ ctx->auth_state.xcbc.xcbc_keys_dma_addr,
+ AES_KEYSIZE_128, NS_BIT);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_XCBC_MAC);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_key_size_aes(&desc[idx], CC_AES_128_BIT_KEY_SIZE);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_aes_not_hash_mode(&desc[idx]);
idx++;
/* Setup XCBC MAC K2 */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- (ctx->auth_state.xcbc.xcbc_keys_dma_addr +
- AES_KEYSIZE_128),
- AES_KEYSIZE_128, NS_BIT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE1);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_XCBC_MAC);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], CC_AES_128_BIT_KEY_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI,
+ (ctx->auth_state.xcbc.xcbc_keys_dma_addr +
+ AES_KEYSIZE_128), AES_KEYSIZE_128, NS_BIT);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE1);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_XCBC_MAC);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_key_size_aes(&desc[idx], CC_AES_128_BIT_KEY_SIZE);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_aes_not_hash_mode(&desc[idx]);
idx++;
/* Setup XCBC MAC K3 */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- (ctx->auth_state.xcbc.xcbc_keys_dma_addr +
- 2 * AES_KEYSIZE_128),
- AES_KEYSIZE_128, NS_BIT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE2);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_XCBC_MAC);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], CC_AES_128_BIT_KEY_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI,
+ (ctx->auth_state.xcbc.xcbc_keys_dma_addr +
+ 2 * AES_KEYSIZE_128), AES_KEYSIZE_128, NS_BIT);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE2);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_XCBC_MAC);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_key_size_aes(&desc[idx], CC_AES_128_BIT_KEY_SIZE);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_aes_not_hash_mode(&desc[idx]);
idx++;
*seq_size = idx;
static inline void ssi_aead_process_digest_header_desc(
struct aead_request *req,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
unsigned int idx = *seq_size;
static inline void ssi_aead_process_digest_scheme_desc(
struct aead_request *req,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct ssi_aead_handle *aead_handle = ctx->drvdata->aead_handle;
unsigned int hash_mode = (ctx->auth_mode == DRV_HASH_SHA1) ?
DRV_HASH_HW_SHA1 : DRV_HASH_HW_SHA256;
- unsigned int digest_size = (ctx->auth_mode == DRV_HASH_SHA1) ?
+ unsigned int digest_size = (ctx->auth_mode == DRV_HASH_SHA1) ?
CC_SHA1_DIGEST_SIZE : CC_SHA256_DIGEST_SIZE;
unsigned int idx = *seq_size;
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hash_mode);
- HW_DESC_SET_DOUT_SRAM(&desc[idx], aead_handle->sram_workspace_addr,
- HASH_LEN_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE1);
- HW_DESC_SET_CIPHER_DO(&desc[idx], DO_PAD);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hash_mode);
+ set_dout_sram(&desc[idx], aead_handle->sram_workspace_addr,
+ HASH_LEN_SIZE);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE1);
+ set_cipher_do(&desc[idx], DO_PAD);
idx++;
/* Get final ICV result */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DOUT_SRAM(&desc[idx], aead_handle->sram_workspace_addr,
- digest_size);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], HASH_DIGEST_RESULT_LITTLE_ENDIAN);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hash_mode);
+ hw_desc_init(&desc[idx]);
+ set_dout_sram(&desc[idx], aead_handle->sram_workspace_addr,
+ digest_size);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
+ set_cipher_config0(&desc[idx], HASH_DIGEST_RESULT_LITTLE_ENDIAN);
+ set_cipher_mode(&desc[idx], hash_mode);
idx++;
/* Loading hash opad xor key state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hash_mode);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- (ctx->auth_state.hmac.ipad_opad_dma_addr + digest_size),
- digest_size, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hash_mode);
+ set_din_type(&desc[idx], DMA_DLLI,
+ (ctx->auth_state.hmac.ipad_opad_dma_addr + digest_size),
+ digest_size, NS_BIT);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
idx++;
/* Load init. digest len (64 bytes) */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hash_mode);
- HW_DESC_SET_DIN_SRAM(&desc[idx],
- ssi_ahash_get_initial_digest_len_sram_addr(ctx->drvdata, hash_mode),
- HASH_LEN_SIZE);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_ENABLED);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hash_mode);
+ set_din_sram(&desc[idx],
+ ssi_ahash_get_initial_digest_len_sram_addr(ctx->drvdata,
+ hash_mode),
+ HASH_LEN_SIZE);
+ set_cipher_config1(&desc[idx], HASH_PADDING_ENABLED);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
/* Perform HASH update */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_SRAM(&desc[idx], aead_handle->sram_workspace_addr,
- digest_size);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_HASH);
+ hw_desc_init(&desc[idx]);
+ set_din_sram(&desc[idx], aead_handle->sram_workspace_addr,
+ digest_size);
+ set_flow_mode(&desc[idx], DIN_HASH);
idx++;
*seq_size = idx;
static inline void ssi_aead_load_mlli_to_sram(
struct aead_request *req,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
struct aead_req_ctx *req_ctx = aead_request_ctx(req);
if (unlikely(
(req_ctx->assoc_buff_type == SSI_DMA_BUF_MLLI) ||
(req_ctx->data_buff_type == SSI_DMA_BUF_MLLI) ||
- (req_ctx->is_single_pass == false))) {
+ !req_ctx->is_single_pass)) {
SSI_LOG_DEBUG("Copy-to-sram: mlli_dma=%08x, mlli_size=%u\n",
(unsigned int)ctx->drvdata->mlli_sram_addr,
req_ctx->mlli_params.mlli_len);
/* Copy MLLI table host-to-sram */
- HW_DESC_INIT(&desc[*seq_size]);
- HW_DESC_SET_DIN_TYPE(&desc[*seq_size], DMA_DLLI,
- req_ctx->mlli_params.mlli_dma_addr,
- req_ctx->mlli_params.mlli_len, NS_BIT);
- HW_DESC_SET_DOUT_SRAM(&desc[*seq_size],
- ctx->drvdata->mlli_sram_addr,
- req_ctx->mlli_params.mlli_len);
- HW_DESC_SET_FLOW_MODE(&desc[*seq_size], BYPASS);
+ hw_desc_init(&desc[*seq_size]);
+ set_din_type(&desc[*seq_size], DMA_DLLI,
+ req_ctx->mlli_params.mlli_dma_addr,
+ req_ctx->mlli_params.mlli_len, NS_BIT);
+ set_dout_sram(&desc[*seq_size],
+ ctx->drvdata->mlli_sram_addr,
+ req_ctx->mlli_params.mlli_len);
+ set_flow_mode(&desc[*seq_size], BYPASS);
(*seq_size)++;
}
}
-static inline enum FlowMode ssi_aead_get_data_flow_mode(
+static inline enum cc_flow_mode ssi_aead_get_data_flow_mode(
enum drv_crypto_direction direct,
- enum FlowMode setup_flow_mode,
+ enum cc_flow_mode setup_flow_mode,
bool is_single_pass)
{
- enum FlowMode data_flow_mode;
+ enum cc_flow_mode data_flow_mode;
if (direct == DRV_CRYPTO_DIRECTION_ENCRYPT) {
if (setup_flow_mode == S_DIN_to_AES)
static inline void ssi_aead_hmac_authenc(
struct aead_request *req,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
unsigned int data_flow_mode = ssi_aead_get_data_flow_mode(
direct, ctx->flow_mode, req_ctx->is_single_pass);
- if (req_ctx->is_single_pass == true) {
+ if (req_ctx->is_single_pass) {
/**
* Single-pass flow
*/
return;
}
- /**
+ /**
* Double-pass flow
- * Fallback for unsupported single-pass modes,
- * i.e. using assoc. data of non-word-multiple */
+ * Fallback for unsupported single-pass modes,
+ * i.e. using assoc. data of non-word-multiple
+ */
if (direct == DRV_CRYPTO_DIRECTION_ENCRYPT) {
/* encrypt first.. */
ssi_aead_process_cipher(req, desc, seq_size, data_flow_mode);
/* decrypt after.. */
ssi_aead_process_cipher(req, desc, seq_size, data_flow_mode);
/* read the digest result with setting the completion bit
- must be after the cipher operation */
+ * must be after the cipher operation
+ */
ssi_aead_process_digest_result_desc(req, desc, seq_size);
}
}
static inline void
ssi_aead_xcbc_authenc(
struct aead_request *req,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
unsigned int data_flow_mode = ssi_aead_get_data_flow_mode(
direct, ctx->flow_mode, req_ctx->is_single_pass);
- if (req_ctx->is_single_pass == true) {
+ if (req_ctx->is_single_pass) {
/**
* Single-pass flow
*/
return;
}
- /**
+ /**
* Double-pass flow
- * Fallback for unsupported single-pass modes,
- * i.e. using assoc. data of non-word-multiple */
+ * Fallback for unsupported single-pass modes,
+ * i.e. using assoc. data of non-word-multiple
+ */
if (direct == DRV_CRYPTO_DIRECTION_ENCRYPT) {
/* encrypt first.. */
ssi_aead_process_cipher(req, desc, seq_size, data_flow_mode);
/* decrypt after..*/
ssi_aead_process_cipher(req, desc, seq_size, data_flow_mode);
/* read the digest result with setting the completion bit
- must be after the cipher operation */
+ * must be after the cipher operation
+ */
ssi_aead_process_digest_result_desc(req, desc, seq_size);
}
}
goto data_size_err;
if (ctx->cipher_mode == DRV_CIPHER_CCM)
break;
- if (ctx->cipher_mode == DRV_CIPHER_GCTR)
- {
- if (areq_ctx->plaintext_authenticate_only == true)
- areq_ctx->is_single_pass = false;
+ if (ctx->cipher_mode == DRV_CIPHER_GCTR) {
+ if (areq_ctx->plaintext_authenticate_only)
+ areq_ctx->is_single_pass = false;
break;
}
- if (!IS_ALIGNED(assoclen, sizeof(uint32_t)))
+ if (!IS_ALIGNED(assoclen, sizeof(u32)))
areq_ctx->is_single_pass = false;
if ((ctx->cipher_mode == DRV_CIPHER_CTR) &&
- !IS_ALIGNED(cipherlen, sizeof(uint32_t)))
+ !IS_ALIGNED(cipherlen, sizeof(u32)))
areq_ctx->is_single_pass = false;
break;
}
#if SSI_CC_HAS_AES_CCM
-static unsigned int format_ccm_a0(uint8_t *pA0Buff, uint32_t headerSize)
+static unsigned int format_ccm_a0(u8 *pA0Buff, u32 headerSize)
{
unsigned int len = 0;
- if ( headerSize == 0 ) {
+
+ if (headerSize == 0)
return 0;
- }
- if ( headerSize < ((1UL << 16) - (1UL << 8) )) {
+
+ if (headerSize < ((1UL << 16) - (1UL << 8))) {
len = 2;
pA0Buff[0] = (headerSize >> 8) & 0xFF;
static inline int ssi_aead_ccm(
struct aead_request *req,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
unsigned int cipher_flow_mode;
dma_addr_t mac_result;
-
if (req_ctx->gen_ctx.op_type == DRV_CRYPTO_DIRECTION_DECRYPT) {
cipher_flow_mode = AES_to_HASH_and_DOUT;
mac_result = req_ctx->mac_buf_dma_addr;
}
/* load key */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_CTR);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, ctx->enckey_dma_addr,
- ((ctx->enc_keylen == 24) ?
- CC_AES_KEY_SIZE_MAX : ctx->enc_keylen),
- NS_BIT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], ctx->enc_keylen);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_CTR);
+ set_din_type(&desc[idx], DMA_DLLI, ctx->enckey_dma_addr,
+ ((ctx->enc_keylen == 24) ? CC_AES_KEY_SIZE_MAX :
+ ctx->enc_keylen), NS_BIT);
+ set_key_size_aes(&desc[idx], ctx->enc_keylen);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
/* load ctr state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_CTR);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], ctx->enc_keylen);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- req_ctx->gen_ctx.iv_dma_addr,
- AES_BLOCK_SIZE, NS_BIT);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE1);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_CTR);
+ set_key_size_aes(&desc[idx], ctx->enc_keylen);
+ set_din_type(&desc[idx], DMA_DLLI,
+ req_ctx->gen_ctx.iv_dma_addr, AES_BLOCK_SIZE, NS_BIT);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE1);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
/* load MAC key */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_CBC_MAC);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, ctx->enckey_dma_addr,
- ((ctx->enc_keylen == 24) ?
- CC_AES_KEY_SIZE_MAX : ctx->enc_keylen),
- NS_BIT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], ctx->enc_keylen);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_CBC_MAC);
+ set_din_type(&desc[idx], DMA_DLLI, ctx->enckey_dma_addr,
+ ((ctx->enc_keylen == 24) ? CC_AES_KEY_SIZE_MAX :
+ ctx->enc_keylen), NS_BIT);
+ set_key_size_aes(&desc[idx], ctx->enc_keylen);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_aes_not_hash_mode(&desc[idx]);
idx++;
/* load MAC state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_CBC_MAC);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], ctx->enc_keylen);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- req_ctx->mac_buf_dma_addr,
- AES_BLOCK_SIZE, NS_BIT);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_CBC_MAC);
+ set_key_size_aes(&desc[idx], ctx->enc_keylen);
+ set_din_type(&desc[idx], DMA_DLLI, req_ctx->mac_buf_dma_addr,
+ AES_BLOCK_SIZE, NS_BIT);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_aes_not_hash_mode(&desc[idx]);
idx++;
-
/* process assoc data */
if (req->assoclen > 0) {
ssi_aead_create_assoc_desc(req, DIN_HASH, desc, &idx);
} else {
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- sg_dma_address(&req_ctx->ccm_adata_sg),
- AES_BLOCK_SIZE + req_ctx->ccm_hdr_size,
- NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_HASH);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI,
+ sg_dma_address(&req_ctx->ccm_adata_sg),
+ AES_BLOCK_SIZE + req_ctx->ccm_hdr_size, NS_BIT);
+ set_flow_mode(&desc[idx], DIN_HASH);
idx++;
}
/* process the cipher */
- if (req_ctx->cryptlen != 0) {
+ if (req_ctx->cryptlen != 0)
ssi_aead_process_cipher_data_desc(req, cipher_flow_mode, desc, &idx);
- }
/* Read temporal MAC */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_CBC_MAC);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], req_ctx->mac_buf_dma_addr,
- ctx->authsize, NS_BIT, 0);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], HASH_DIGEST_RESULT_LITTLE_ENDIAN);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_CBC_MAC);
+ set_dout_dlli(&desc[idx], req_ctx->mac_buf_dma_addr, ctx->authsize,
+ NS_BIT, 0);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
+ set_cipher_config0(&desc[idx], HASH_DIGEST_RESULT_LITTLE_ENDIAN);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_aes_not_hash_mode(&desc[idx]);
idx++;
/* load AES-CTR state (for last MAC calculation)*/
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_CTR);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- req_ctx->ccm_iv0_dma_addr ,
- AES_BLOCK_SIZE, NS_BIT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], ctx->enc_keylen);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE1);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_CTR);
+ set_cipher_config0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
+ set_din_type(&desc[idx], DMA_DLLI, req_ctx->ccm_iv0_dma_addr,
+ AES_BLOCK_SIZE, NS_BIT);
+ set_key_size_aes(&desc[idx], ctx->enc_keylen);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE1);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_NO_DMA(&desc[idx], 0, 0xfffff0);
- HW_DESC_SET_DOUT_NO_DMA(&desc[idx], 0, 0, 1);
+ hw_desc_init(&desc[idx]);
+ set_din_no_dma(&desc[idx], 0, 0xfffff0);
+ set_dout_no_dma(&desc[idx], 0, 0, 1);
idx++;
/* encrypt the "T" value and store MAC in mac_state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- req_ctx->mac_buf_dma_addr , ctx->authsize, NS_BIT);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], mac_result , ctx->authsize, NS_BIT, 1);
- HW_DESC_SET_QUEUE_LAST_IND(&desc[idx]);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_AES_DOUT);
- idx++;
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, req_ctx->mac_buf_dma_addr,
+ ctx->authsize, NS_BIT);
+ set_dout_dlli(&desc[idx], mac_result, ctx->authsize, NS_BIT, 1);
+ set_queue_last_ind(&desc[idx]);
+ set_flow_mode(&desc[idx], DIN_AES_DOUT);
+ idx++;
*seq_size = idx;
return 0;
}
-static int config_ccm_adata(struct aead_request *req) {
+static int config_ccm_adata(struct aead_request *req)
+{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct ssi_aead_ctx *ctx = crypto_aead_ctx(tfm);
struct aead_req_ctx *req_ctx = aead_request_ctx(req);
/* Note: The code assume that req->iv[0] already contains the value of L' of RFC3610 */
unsigned int l = lp + 1; /* This is L' of RFC 3610. */
unsigned int m = ctx->authsize; /* This is M' of RFC 3610. */
- uint8_t *b0 = req_ctx->ccm_config + CCM_B0_OFFSET;
- uint8_t *a0 = req_ctx->ccm_config + CCM_A0_OFFSET;
- uint8_t *ctr_count_0 = req_ctx->ccm_config + CCM_CTR_COUNT_0_OFFSET;
- unsigned int cryptlen = (req_ctx->gen_ctx.op_type ==
- DRV_CRYPTO_DIRECTION_ENCRYPT) ?
- req->cryptlen :
+ u8 *b0 = req_ctx->ccm_config + CCM_B0_OFFSET;
+ u8 *a0 = req_ctx->ccm_config + CCM_A0_OFFSET;
+ u8 *ctr_count_0 = req_ctx->ccm_config + CCM_CTR_COUNT_0_OFFSET;
+ unsigned int cryptlen = (req_ctx->gen_ctx.op_type ==
+ DRV_CRYPTO_DIRECTION_ENCRYPT) ?
+ req->cryptlen :
(req->cryptlen - ctx->authsize);
int rc;
+
memset(req_ctx->mac_buf, 0, AES_BLOCK_SIZE);
- memset(req_ctx->ccm_config, 0, AES_BLOCK_SIZE*3);
+ memset(req_ctx->ccm_config, 0, AES_BLOCK_SIZE * 3);
/* taken from crypto/ccm.c */
/* 2 <= L <= 8, so 1 <= L' <= 7. */
if (2 > l || l > 8) {
- SSI_LOG_ERR("illegal iv value %X\n",req->iv[0]);
+ SSI_LOG_ERR("illegal iv value %X\n", req->iv[0]);
return -EINVAL;
}
memcpy(b0, req->iv, AES_BLOCK_SIZE);
*b0 |= (8 * ((m - 2) / 2));
if (req->assoclen > 0)
*b0 |= 64; /* Enable bit 6 if Adata exists. */
-
+
rc = set_msg_len(b0 + 16 - l, cryptlen, l); /* Write L'. */
- if (rc != 0) {
+ if (rc != 0)
return rc;
- }
/* END of "taken from crypto/ccm.c" */
-
+
/* l(a) - size of associated data. */
- req_ctx->ccm_hdr_size = format_ccm_a0 (a0, req->assoclen);
+ req_ctx->ccm_hdr_size = format_ccm_a0(a0, req->assoclen);
memset(req->iv + 15 - req->iv[0], 0, req->iv[0] + 1);
- req->iv [15] = 1;
+ req->iv[15] = 1;
- memcpy(ctr_count_0, req->iv, AES_BLOCK_SIZE) ;
+ memcpy(ctr_count_0, req->iv, AES_BLOCK_SIZE);
ctr_count_0[15] = 0;
return 0;
/* In RFC 4309 there is an 11-bytes nonce+IV part, that we build here. */
memcpy(areq_ctx->ctr_iv + CCM_BLOCK_NONCE_OFFSET, ctx->ctr_nonce, CCM_BLOCK_NONCE_SIZE);
memcpy(areq_ctx->ctr_iv + CCM_BLOCK_IV_OFFSET, req->iv, CCM_BLOCK_IV_SIZE);
- req->iv = areq_ctx->ctr_iv;
+ req->iv = areq_ctx->ctr_iv;
req->assoclen -= CCM_BLOCK_IV_SIZE;
}
#endif /*SSI_CC_HAS_AES_CCM*/
static inline void ssi_aead_gcm_setup_ghash_desc(
struct aead_request *req,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
unsigned int idx = *seq_size;
/* load key to AES*/
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_ECB);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, ctx->enckey_dma_addr,
- ctx->enc_keylen, NS_BIT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], ctx->enc_keylen);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_ECB);
+ set_cipher_config0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
+ set_din_type(&desc[idx], DMA_DLLI, ctx->enckey_dma_addr,
+ ctx->enc_keylen, NS_BIT);
+ set_key_size_aes(&desc[idx], ctx->enc_keylen);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
/* process one zero block to generate hkey */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0x0, AES_BLOCK_SIZE);
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- req_ctx->hkey_dma_addr,
- AES_BLOCK_SIZE,
- NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_AES_DOUT);
+ hw_desc_init(&desc[idx]);
+ set_din_const(&desc[idx], 0x0, AES_BLOCK_SIZE);
+ set_dout_dlli(&desc[idx], req_ctx->hkey_dma_addr, AES_BLOCK_SIZE,
+ NS_BIT, 0);
+ set_flow_mode(&desc[idx], DIN_AES_DOUT);
idx++;
/* Memory Barrier */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_NO_DMA(&desc[idx], 0, 0xfffff0);
- HW_DESC_SET_DOUT_NO_DMA(&desc[idx], 0, 0, 1);
+ hw_desc_init(&desc[idx]);
+ set_din_no_dma(&desc[idx], 0, 0xfffff0);
+ set_dout_no_dma(&desc[idx], 0, 0, 1);
idx++;
/* Load GHASH subkey */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- req_ctx->hkey_dma_addr,
- AES_BLOCK_SIZE, NS_BIT);
- HW_DESC_SET_DOUT_NO_DMA(&desc[idx], 0, 0, 1);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_HASH_HW_GHASH);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_ENABLED);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, req_ctx->hkey_dma_addr,
+ AES_BLOCK_SIZE, NS_BIT);
+ set_dout_no_dma(&desc[idx], 0, 0, 1);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_aes_not_hash_mode(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_HASH_HW_GHASH);
+ set_cipher_config1(&desc[idx], HASH_PADDING_ENABLED);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
/* Configure Hash Engine to work with GHASH.
- Since it was not possible to extend HASH submodes to add GHASH,
- The following command is necessary in order to select GHASH (according to HW designers)*/
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_NO_DMA(&desc[idx], 0, 0xfffff0);
- HW_DESC_SET_DOUT_NO_DMA(&desc[idx], 0, 0, 1);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_HASH_HW_GHASH);
- HW_DESC_SET_CIPHER_DO(&desc[idx], 1); //1=AES_SK RKEK
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_ENABLED);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ * Since it was not possible to extend HASH submodes to add GHASH,
+ * The following command is necessary in order to
+ * select GHASH (according to HW designers)
+ */
+ hw_desc_init(&desc[idx]);
+ set_din_no_dma(&desc[idx], 0, 0xfffff0);
+ set_dout_no_dma(&desc[idx], 0, 0, 1);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_aes_not_hash_mode(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_HASH_HW_GHASH);
+ set_cipher_do(&desc[idx], 1); //1=AES_SK RKEK
+ set_cipher_config0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
+ set_cipher_config1(&desc[idx], HASH_PADDING_ENABLED);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
/* Load GHASH initial STATE (which is 0). (for any hash there is an initial state) */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0x0, AES_BLOCK_SIZE);
- HW_DESC_SET_DOUT_NO_DMA(&desc[idx], 0, 0, 1);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_HASH_HW_GHASH);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_ENABLED);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_din_const(&desc[idx], 0x0, AES_BLOCK_SIZE);
+ set_dout_no_dma(&desc[idx], 0, 0, 1);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_aes_not_hash_mode(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_HASH_HW_GHASH);
+ set_cipher_config1(&desc[idx], HASH_PADDING_ENABLED);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
idx++;
*seq_size = idx;
static inline void ssi_aead_gcm_setup_gctr_desc(
struct aead_request *req,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
unsigned int idx = *seq_size;
/* load key to AES*/
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_GCTR);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, ctx->enckey_dma_addr,
- ctx->enc_keylen, NS_BIT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], ctx->enc_keylen);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_GCTR);
+ set_cipher_config0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
+ set_din_type(&desc[idx], DMA_DLLI, ctx->enckey_dma_addr,
+ ctx->enc_keylen, NS_BIT);
+ set_key_size_aes(&desc[idx], ctx->enc_keylen);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
- if ((req_ctx->cryptlen != 0) && (req_ctx->plaintext_authenticate_only==false)){
+ if ((req_ctx->cryptlen != 0) && (!req_ctx->plaintext_authenticate_only)) {
/* load AES/CTR initial CTR value inc by 2*/
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_GCTR);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], ctx->enc_keylen);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- req_ctx->gcm_iv_inc2_dma_addr,
- AES_BLOCK_SIZE, NS_BIT);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE1);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_GCTR);
+ set_key_size_aes(&desc[idx], ctx->enc_keylen);
+ set_din_type(&desc[idx], DMA_DLLI,
+ req_ctx->gcm_iv_inc2_dma_addr, AES_BLOCK_SIZE,
+ NS_BIT);
+ set_cipher_config0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE1);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
}
static inline void ssi_aead_process_gcm_result_desc(
struct aead_request *req,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct ssi_aead_ctx *ctx = crypto_aead_ctx(tfm);
struct aead_req_ctx *req_ctx = aead_request_ctx(req);
- dma_addr_t mac_result;
+ dma_addr_t mac_result;
unsigned int idx = *seq_size;
if (req_ctx->gen_ctx.op_type == DRV_CRYPTO_DIRECTION_DECRYPT) {
}
/* process(ghash) gcm_block_len */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- req_ctx->gcm_block_len_dma_addr,
- AES_BLOCK_SIZE, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_HASH);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, req_ctx->gcm_block_len_dma_addr,
+ AES_BLOCK_SIZE, NS_BIT);
+ set_flow_mode(&desc[idx], DIN_HASH);
idx++;
/* Store GHASH state after GHASH(Associated Data + Cipher +LenBlock) */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_HASH_HW_GHASH);
- HW_DESC_SET_DIN_NO_DMA(&desc[idx], 0, 0xfffff0);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], req_ctx->mac_buf_dma_addr,
- AES_BLOCK_SIZE, NS_BIT, 0);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_HASH_HW_GHASH);
+ set_din_no_dma(&desc[idx], 0, 0xfffff0);
+ set_dout_dlli(&desc[idx], req_ctx->mac_buf_dma_addr, AES_BLOCK_SIZE,
+ NS_BIT, 0);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_aes_not_hash_mode(&desc[idx]);
- idx++;
+ idx++;
/* load AES/CTR initial CTR value inc by 1*/
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_GCTR);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], ctx->enc_keylen);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- req_ctx->gcm_iv_inc1_dma_addr,
- AES_BLOCK_SIZE, NS_BIT);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE1);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_GCTR);
+ set_key_size_aes(&desc[idx], ctx->enc_keylen);
+ set_din_type(&desc[idx], DMA_DLLI, req_ctx->gcm_iv_inc1_dma_addr,
+ AES_BLOCK_SIZE, NS_BIT);
+ set_cipher_config0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE1);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
/* Memory Barrier */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_NO_DMA(&desc[idx], 0, 0xfffff0);
- HW_DESC_SET_DOUT_NO_DMA(&desc[idx], 0, 0, 1);
+ hw_desc_init(&desc[idx]);
+ set_din_no_dma(&desc[idx], 0, 0xfffff0);
+ set_dout_no_dma(&desc[idx], 0, 0, 1);
idx++;
/* process GCTR on stored GHASH and store MAC in mac_state*/
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_GCTR);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- req_ctx->mac_buf_dma_addr,
- AES_BLOCK_SIZE, NS_BIT);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], mac_result, ctx->authsize, NS_BIT, 1);
- HW_DESC_SET_QUEUE_LAST_IND(&desc[idx]);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_AES_DOUT);
- idx++;
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_GCTR);
+ set_din_type(&desc[idx], DMA_DLLI, req_ctx->mac_buf_dma_addr,
+ AES_BLOCK_SIZE, NS_BIT);
+ set_dout_dlli(&desc[idx], mac_result, ctx->authsize, NS_BIT, 1);
+ set_queue_last_ind(&desc[idx]);
+ set_flow_mode(&desc[idx], DIN_AES_DOUT);
+ idx++;
*seq_size = idx;
}
static inline int ssi_aead_gcm(
struct aead_request *req,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
struct aead_req_ctx *req_ctx = aead_request_ctx(req);
cipher_flow_mode = AES_to_HASH_and_DOUT;
}
-
//in RFC4543 no data to encrypt. just copy data from src to dest.
- if (req_ctx->plaintext_authenticate_only==true){
+ if (req_ctx->plaintext_authenticate_only) {
ssi_aead_process_cipher_data_desc(req, BYPASS, desc, seq_size);
ssi_aead_gcm_setup_ghash_desc(req, desc, seq_size);
/* process(ghash) assoc data */
ssi_aead_gcm_setup_gctr_desc(req, desc, seq_size);
/* process(gctr+ghash) */
if (req_ctx->cryptlen != 0)
- ssi_aead_process_cipher_data_desc(req, cipher_flow_mode, desc, seq_size);
+ ssi_aead_process_cipher_data_desc(req, cipher_flow_mode, desc, seq_size);
ssi_aead_process_gcm_result_desc(req, desc, seq_size);
idx = *seq_size;
#ifdef CC_DEBUG
static inline void ssi_aead_dump_gcm(
- const char* title,
+ const char *title,
struct aead_request *req)
{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
if (ctx->cipher_mode != DRV_CIPHER_GCTR)
return;
- if (title != NULL) {
+ if (title) {
SSI_LOG_DEBUG("----------------------------------------------------------------------------------");
SSI_LOG_DEBUG("%s\n", title);
}
- SSI_LOG_DEBUG("cipher_mode %d, authsize %d, enc_keylen %d, assoclen %d, cryptlen %d \n", \
- ctx->cipher_mode, ctx->authsize, ctx->enc_keylen, req->assoclen, req_ctx->cryptlen );
+ SSI_LOG_DEBUG("cipher_mode %d, authsize %d, enc_keylen %d, assoclen %d, cryptlen %d\n", \
+ ctx->cipher_mode, ctx->authsize, ctx->enc_keylen, req->assoclen, req_ctx->cryptlen);
- if ( ctx->enckey != NULL ) {
- dump_byte_array("mac key",ctx->enckey, 16);
- }
+ if (ctx->enckey)
+ dump_byte_array("mac key", ctx->enckey, 16);
- dump_byte_array("req->iv",req->iv, AES_BLOCK_SIZE);
+ dump_byte_array("req->iv", req->iv, AES_BLOCK_SIZE);
- dump_byte_array("gcm_iv_inc1",req_ctx->gcm_iv_inc1, AES_BLOCK_SIZE);
+ dump_byte_array("gcm_iv_inc1", req_ctx->gcm_iv_inc1, AES_BLOCK_SIZE);
- dump_byte_array("gcm_iv_inc2",req_ctx->gcm_iv_inc2, AES_BLOCK_SIZE);
+ dump_byte_array("gcm_iv_inc2", req_ctx->gcm_iv_inc2, AES_BLOCK_SIZE);
- dump_byte_array("hkey",req_ctx->hkey, AES_BLOCK_SIZE);
+ dump_byte_array("hkey", req_ctx->hkey, AES_BLOCK_SIZE);
- dump_byte_array("mac_buf",req_ctx->mac_buf, AES_BLOCK_SIZE);
+ dump_byte_array("mac_buf", req_ctx->mac_buf, AES_BLOCK_SIZE);
- dump_byte_array("gcm_len_block",req_ctx->gcm_len_block.lenA, AES_BLOCK_SIZE);
+ dump_byte_array("gcm_len_block", req_ctx->gcm_len_block.lenA, AES_BLOCK_SIZE);
- if (req->src!=NULL && req->cryptlen) {
- dump_byte_array("req->src",sg_virt(req->src), req->cryptlen+req->assoclen);
- }
+ if (req->src && req->cryptlen)
+ dump_byte_array("req->src", sg_virt(req->src), req->cryptlen + req->assoclen);
- if (req->dst!=NULL) {
- dump_byte_array("req->dst",sg_virt(req->dst), req->cryptlen+ctx->authsize+req->assoclen);
- }
+ if (req->dst)
+ dump_byte_array("req->dst", sg_virt(req->dst), req->cryptlen + ctx->authsize + req->assoclen);
}
#endif
-static int config_gcm_context(struct aead_request *req) {
+static int config_gcm_context(struct aead_request *req)
+{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct ssi_aead_ctx *ctx = crypto_aead_ctx(tfm);
struct aead_req_ctx *req_ctx = aead_request_ctx(req);
-
- unsigned int cryptlen = (req_ctx->gen_ctx.op_type ==
- DRV_CRYPTO_DIRECTION_ENCRYPT) ?
- req->cryptlen :
+
+ unsigned int cryptlen = (req_ctx->gen_ctx.op_type ==
+ DRV_CRYPTO_DIRECTION_ENCRYPT) ?
+ req->cryptlen :
(req->cryptlen - ctx->authsize);
__be32 counter = cpu_to_be32(2);
- SSI_LOG_DEBUG("config_gcm_context() cryptlen = %d, req->assoclen = %d ctx->authsize = %d \n", cryptlen, req->assoclen, ctx->authsize);
+ SSI_LOG_DEBUG("config_gcm_context() cryptlen = %d, req->assoclen = %d ctx->authsize = %d\n", cryptlen, req->assoclen, ctx->authsize);
memset(req_ctx->hkey, 0, AES_BLOCK_SIZE);
memcpy(req->iv + 12, &counter, 4);
memcpy(req_ctx->gcm_iv_inc1, req->iv, 16);
-
- if (req_ctx->plaintext_authenticate_only == false)
- {
+ if (!req_ctx->plaintext_authenticate_only) {
__be64 temp64;
+
temp64 = cpu_to_be64(req->assoclen * 8);
- memcpy ( &req_ctx->gcm_len_block.lenA , &temp64, sizeof(temp64) );
+ memcpy(&req_ctx->gcm_len_block.lenA, &temp64, sizeof(temp64));
temp64 = cpu_to_be64(cryptlen * 8);
- memcpy ( &req_ctx->gcm_len_block.lenC , &temp64, 8 );
- }
- else { //rfc4543=> all data(AAD,IV,Plain) are considered additional data that is nothing is encrypted.
+ memcpy(&req_ctx->gcm_len_block.lenC, &temp64, 8);
+ } else { //rfc4543=> all data(AAD,IV,Plain) are considered additional data that is nothing is encrypted.
__be64 temp64;
- temp64 = cpu_to_be64((req->assoclen+GCM_BLOCK_RFC4_IV_SIZE+cryptlen) * 8);
- memcpy ( &req_ctx->gcm_len_block.lenA , &temp64, sizeof(temp64) );
+
+ temp64 = cpu_to_be64((req->assoclen + GCM_BLOCK_RFC4_IV_SIZE + cryptlen) * 8);
+ memcpy(&req_ctx->gcm_len_block.lenA, &temp64, sizeof(temp64));
temp64 = 0;
- memcpy ( &req_ctx->gcm_len_block.lenC , &temp64, 8 );
+ memcpy(&req_ctx->gcm_len_block.lenC, &temp64, 8);
}
return 0;
memcpy(areq_ctx->ctr_iv + GCM_BLOCK_RFC4_NONCE_OFFSET, ctx->ctr_nonce, GCM_BLOCK_RFC4_NONCE_SIZE);
memcpy(areq_ctx->ctr_iv + GCM_BLOCK_RFC4_IV_OFFSET, req->iv, GCM_BLOCK_RFC4_IV_SIZE);
- req->iv = areq_ctx->ctr_iv;
+ req->iv = areq_ctx->ctr_iv;
req->assoclen -= GCM_BLOCK_RFC4_IV_SIZE;
}
-
#endif /*SSI_CC_HAS_AES_GCM*/
static int ssi_aead_process(struct aead_request *req, enum drv_crypto_direction direct)
{
int rc = 0;
int seq_len = 0;
- HwDesc_s desc[MAX_AEAD_PROCESS_SEQ];
+ struct cc_hw_desc desc[MAX_AEAD_PROCESS_SEQ];
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct ssi_aead_ctx *ctx = crypto_aead_ctx(tfm);
struct aead_req_ctx *areq_ctx = aead_request_ctx(req);
struct device *dev = &ctx->drvdata->plat_dev->dev;
struct ssi_crypto_req ssi_req = {};
- DECL_CYCLE_COUNT_RESOURCES;
-
SSI_LOG_DEBUG("%s context=%p req=%p iv=%p src=%p src_ofs=%d dst=%p dst_ofs=%d cryptolen=%d\n",
- ((direct==DRV_CRYPTO_DIRECTION_ENCRYPT)?"Encrypt":"Decrypt"), ctx, req, req->iv,
+ ((direct == DRV_CRYPTO_DIRECTION_ENCRYPT) ? "Encrypt" : "Decrypt"), ctx, req, req->iv,
sg_virt(req->src), req->src->offset, sg_virt(req->dst), req->dst->offset, req->cryptlen);
CHECK_AND_RETURN_UPON_FIPS_ERROR();
/* STAT_PHASE_0: Init and sanity checks */
- START_CYCLE_COUNT();
-
+
/* Check data length according to mode */
if (unlikely(validate_data_size(ctx, direct, req) != 0)) {
SSI_LOG_ERR("Unsupported crypt/assoc len %d/%d.\n",
ssi_req.user_cb = (void *)ssi_aead_complete;
ssi_req.user_arg = (void *)req;
-#ifdef ENABLE_CYCLE_COUNT
- ssi_req.op_type = (direct == DRV_CRYPTO_DIRECTION_DECRYPT) ?
- STAT_OP_TYPE_DECODE : STAT_OP_TYPE_ENCODE;
-#endif
/* Setup request context */
areq_ctx->gen_ctx.op_type = direct;
areq_ctx->req_authsize = ctx->authsize;
areq_ctx->cipher_mode = ctx->cipher_mode;
- END_CYCLE_COUNT(ssi_req.op_type, STAT_PHASE_0);
-
/* STAT_PHASE_1: Map buffers */
- START_CYCLE_COUNT();
-
+
if (ctx->cipher_mode == DRV_CIPHER_CTR) {
/* Build CTR IV - Copy nonce from last 4 bytes in
- * CTR key to first 4 bytes in CTR IV */
+ * CTR key to first 4 bytes in CTR IV
+ */
memcpy(areq_ctx->ctr_iv, ctx->ctr_nonce, CTR_RFC3686_NONCE_SIZE);
- if (areq_ctx->backup_giv == NULL) /*User none-generated IV*/
+ if (!areq_ctx->backup_giv) /*User none-generated IV*/
memcpy(areq_ctx->ctr_iv + CTR_RFC3686_NONCE_SIZE,
req->iv, CTR_RFC3686_IV_SIZE);
/* Initialize counter portion of counter block */
/* Replace with counter iv */
req->iv = areq_ctx->ctr_iv;
areq_ctx->hw_iv_size = CTR_RFC3686_BLOCK_SIZE;
- } else if ((ctx->cipher_mode == DRV_CIPHER_CCM) ||
- (ctx->cipher_mode == DRV_CIPHER_GCTR) ) {
+ } else if ((ctx->cipher_mode == DRV_CIPHER_CCM) ||
+ (ctx->cipher_mode == DRV_CIPHER_GCTR)) {
areq_ctx->hw_iv_size = AES_BLOCK_SIZE;
if (areq_ctx->ctr_iv != req->iv) {
memcpy(areq_ctx->ctr_iv, req->iv, crypto_aead_ivsize(tfm));
rc = config_ccm_adata(req);
if (unlikely(rc != 0)) {
SSI_LOG_ERR("config_ccm_adata() returned with a failure %d!", rc);
- goto exit;
+ goto exit;
}
} else {
- areq_ctx->ccm_hdr_size = ccm_header_size_null;
+ areq_ctx->ccm_hdr_size = ccm_header_size_null;
}
#else
- areq_ctx->ccm_hdr_size = ccm_header_size_null;
+ areq_ctx->ccm_hdr_size = ccm_header_size_null;
#endif /*SSI_CC_HAS_AES_CCM*/
-#if SSI_CC_HAS_AES_GCM
+#if SSI_CC_HAS_AES_GCM
if (ctx->cipher_mode == DRV_CIPHER_GCTR) {
rc = config_gcm_context(req);
if (unlikely(rc != 0)) {
SSI_LOG_ERR("config_gcm_context() returned with a failure %d!", rc);
- goto exit;
+ goto exit;
}
- }
+ }
#endif /*SSI_CC_HAS_AES_GCM*/
rc = ssi_buffer_mgr_map_aead_request(ctx->drvdata, req);
}
/* do we need to generate IV? */
- if (areq_ctx->backup_giv != NULL) {
-
+ if (areq_ctx->backup_giv) {
/* set the DMA mapped IV address*/
if (ctx->cipher_mode == DRV_CIPHER_CTR) {
ssi_req.ivgen_dma_addr[0] = areq_ctx->gen_ctx.iv_dma_addr + CTR_RFC3686_NONCE_SIZE;
ssi_req.ivgen_dma_addr_len = 1;
} else if (ctx->cipher_mode == DRV_CIPHER_CCM) {
/* In ccm, the IV needs to exist both inside B0 and inside the counter.
- It is also copied to iv_dma_addr for other reasons (like returning
- it to the user).
- So, using 3 (identical) IV outputs. */
+ * It is also copied to iv_dma_addr for other reasons (like returning
+ * it to the user).
+ * So, using 3 (identical) IV outputs.
+ */
ssi_req.ivgen_dma_addr[0] = areq_ctx->gen_ctx.iv_dma_addr + CCM_BLOCK_IV_OFFSET;
ssi_req.ivgen_dma_addr[1] = sg_dma_address(&areq_ctx->ccm_adata_sg) + CCM_B0_OFFSET + CCM_BLOCK_IV_OFFSET;
ssi_req.ivgen_dma_addr[2] = sg_dma_address(&areq_ctx->ccm_adata_sg) + CCM_CTR_COUNT_0_OFFSET + CCM_BLOCK_IV_OFFSET;
ssi_req.ivgen_size = crypto_aead_ivsize(tfm);
}
- END_CYCLE_COUNT(ssi_req.op_type, STAT_PHASE_1);
-
/* STAT_PHASE_2: Create sequence */
- START_CYCLE_COUNT();
/* Load MLLI tables to SRAM if necessary */
ssi_aead_load_mlli_to_sram(req, desc, &seq_len);
case DRV_HASH_XCBC_MAC:
ssi_aead_xcbc_authenc(req, desc, &seq_len);
break;
-#if ( SSI_CC_HAS_AES_CCM || SSI_CC_HAS_AES_GCM )
+#if (SSI_CC_HAS_AES_CCM || SSI_CC_HAS_AES_GCM)
case DRV_HASH_NULL:
#if SSI_CC_HAS_AES_CCM
- if (ctx->cipher_mode == DRV_CIPHER_CCM) {
+ if (ctx->cipher_mode == DRV_CIPHER_CCM)
ssi_aead_ccm(req, desc, &seq_len);
- }
#endif /*SSI_CC_HAS_AES_CCM*/
#if SSI_CC_HAS_AES_GCM
- if (ctx->cipher_mode == DRV_CIPHER_GCTR) {
+ if (ctx->cipher_mode == DRV_CIPHER_GCTR)
ssi_aead_gcm(req, desc, &seq_len);
- }
#endif /*SSI_CC_HAS_AES_GCM*/
break;
#endif
- default:
+ default:
SSI_LOG_ERR("Unsupported authenc (%d)\n", ctx->auth_mode);
ssi_buffer_mgr_unmap_aead_request(dev, req);
rc = -ENOTSUPP;
goto exit;
}
- END_CYCLE_COUNT(ssi_req.op_type, STAT_PHASE_2);
-
/* STAT_PHASE_3: Lock HW and push sequence */
- START_CYCLE_COUNT();
rc = send_request(ctx->drvdata, &ssi_req, desc, seq_len, 1);
ssi_buffer_mgr_unmap_aead_request(dev, req);
}
-
- END_CYCLE_COUNT(ssi_req.op_type, STAT_PHASE_3);
exit:
return rc;
}
int rc = -EINVAL;
if (!valid_assoclen(req)) {
- SSI_LOG_ERR("invalid Assoclen:%u\n", req->assoclen );
+ SSI_LOG_ERR("invalid Assoclen:%u\n", req->assoclen);
goto out;
}
areq_ctx->backup_iv = req->iv;
areq_ctx->backup_giv = NULL;
areq_ctx->is_gcm4543 = true;
-
+
ssi_rfc4309_ccm_process(req);
-
+
rc = ssi_aead_process(req, DRV_CRYPTO_DIRECTION_ENCRYPT);
if (rc != -EINPROGRESS)
req->iv = areq_ctx->backup_iv;
req->iv = areq_ctx->backup_iv;
return rc;
-
}
#if SSI_CC_HAS_AES_CCM
/* No generated IV required */
areq_ctx->backup_iv = req->iv;
areq_ctx->backup_giv = NULL;
-
+
areq_ctx->is_gcm4543 = true;
ssi_rfc4309_ccm_process(req);
-
+
rc = ssi_aead_process(req, DRV_CRYPTO_DIRECTION_DECRYPT);
if (rc != -EINPROGRESS)
req->iv = areq_ctx->backup_iv;
{
struct ssi_aead_ctx *ctx = crypto_aead_ctx(tfm);
int rc = 0;
-
- SSI_LOG_DEBUG("ssi_rfc4106_gcm_setkey() keylen %d, key %p \n", keylen, key );
+
+ SSI_LOG_DEBUG("ssi_rfc4106_gcm_setkey() keylen %d, key %p\n", keylen, key);
if (keylen < 4)
return -EINVAL;
{
struct ssi_aead_ctx *ctx = crypto_aead_ctx(tfm);
int rc = 0;
-
- SSI_LOG_DEBUG("ssi_rfc4543_gcm_setkey() keylen %d, key %p \n", keylen, key );
+
+ SSI_LOG_DEBUG("ssi_rfc4543_gcm_setkey() keylen %d, key %p\n", keylen, key);
if (keylen < 4)
return -EINVAL;
static int ssi_rfc4106_gcm_setauthsize(struct crypto_aead *authenc,
unsigned int authsize)
{
- SSI_LOG_DEBUG("ssi_rfc4106_gcm_setauthsize() authsize %d \n", authsize );
-
- switch (authsize) {
- case 8:
- case 12:
- case 16:
- break;
- default:
- return -EINVAL;
- }
-
- return ssi_aead_setauthsize(authenc, authsize);
+ SSI_LOG_DEBUG("ssi_rfc4106_gcm_setauthsize() authsize %d\n", authsize);
+
+ switch (authsize) {
+ case 8:
+ case 12:
+ case 16:
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return ssi_aead_setauthsize(authenc, authsize);
}
static int ssi_rfc4543_gcm_setauthsize(struct crypto_aead *authenc,
- unsigned int authsize)
+ unsigned int authsize)
{
- SSI_LOG_DEBUG("ssi_rfc4543_gcm_setauthsize() authsize %d \n", authsize );
+ SSI_LOG_DEBUG("ssi_rfc4543_gcm_setauthsize() authsize %d\n", authsize);
if (authsize != 16)
return -EINVAL;
/* Very similar to ssi_aead_encrypt() above. */
struct aead_req_ctx *areq_ctx = aead_request_ctx(req);
- int rc = -EINVAL;
+ int rc = -EINVAL;
if (!valid_assoclen(req)) {
SSI_LOG_ERR("invalid Assoclen:%u\n", req->assoclen);
/* No generated IV required */
areq_ctx->backup_iv = req->iv;
areq_ctx->backup_giv = NULL;
-
+
areq_ctx->plaintext_authenticate_only = false;
ssi_rfc4_gcm_process(req);
struct aead_req_ctx *areq_ctx = aead_request_ctx(req);
int rc;
-
+
//plaintext is not encryped with rfc4543
areq_ctx->plaintext_authenticate_only = true;
/* No generated IV required */
areq_ctx->backup_iv = req->iv;
areq_ctx->backup_giv = NULL;
-
+
ssi_rfc4_gcm_process(req);
areq_ctx->is_gcm4543 = true;
/* Very similar to ssi_aead_decrypt() above. */
struct aead_req_ctx *areq_ctx = aead_request_ctx(req);
- int rc = -EINVAL;
+ int rc = -EINVAL;
if (!valid_assoclen(req)) {
SSI_LOG_ERR("invalid Assoclen:%u\n", req->assoclen);
/* No generated IV required */
areq_ctx->backup_iv = req->iv;
areq_ctx->backup_giv = NULL;
-
+
areq_ctx->plaintext_authenticate_only = false;
ssi_rfc4_gcm_process(req);
/* No generated IV required */
areq_ctx->backup_iv = req->iv;
areq_ctx->backup_giv = NULL;
-
+
ssi_rfc4_gcm_process(req);
areq_ctx->is_gcm4543 = true;
.cipher_mode = DRV_CIPHER_GCTR,
.flow_mode = S_DIN_to_AES,
.auth_mode = DRV_HASH_NULL,
- },
+ },
#endif /*SSI_CC_HAS_AES_GCM*/
};
struct ssi_aead_handle *aead_handle =
(struct ssi_aead_handle *)drvdata->aead_handle;
- if (aead_handle != NULL) {
+ if (aead_handle) {
/* Remove registered algs */
list_for_each_entry_safe(t_alg, n, &aead_handle->aead_list, entry) {
crypto_unregister_aead(&t_alg->aead_alg);
int alg;
aead_handle = kmalloc(sizeof(struct ssi_aead_handle), GFP_KERNEL);
- if (aead_handle == NULL) {
+ if (!aead_handle) {
rc = -ENOMEM;
goto fail0;
}
fail0:
return rc;
}
-
-
-
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
/* \file ssi_aead.h
- ARM CryptoCell AEAD Crypto API
+ * ARM CryptoCell AEAD Crypto API
*/
#ifndef __SSI_AEAD_H__
#include <crypto/algapi.h>
#include <crypto/ctr.h>
-
/* mac_cmp - HW writes 8 B but all bytes hold the same value */
#define ICV_CMP_SIZE 8
-#define CCM_CONFIG_BUF_SIZE (AES_BLOCK_SIZE*3)
+#define CCM_CONFIG_BUF_SIZE (AES_BLOCK_SIZE * 3)
#define MAX_MAC_SIZE MAX(SHA256_DIGEST_SIZE, AES_BLOCK_SIZE)
-
/* defines for AES GCM configuration buffer */
#define GCM_BLOCK_LEN_SIZE 8
-#define GCM_BLOCK_RFC4_IV_OFFSET 4
-#define GCM_BLOCK_RFC4_IV_SIZE 8 /* IV size for rfc's */
-#define GCM_BLOCK_RFC4_NONCE_OFFSET 0
-#define GCM_BLOCK_RFC4_NONCE_SIZE 4
-
-
+#define GCM_BLOCK_RFC4_IV_OFFSET 4
+#define GCM_BLOCK_RFC4_IV_SIZE 8 /* IV size for rfc's */
+#define GCM_BLOCK_RFC4_NONCE_OFFSET 0
+#define GCM_BLOCK_RFC4_NONCE_SIZE 4
/* Offsets into AES CCM configuration buffer */
#define CCM_B0_OFFSET 0
ccm_header_size_zero = 0,
ccm_header_size_2 = 2,
ccm_header_size_6 = 6,
- ccm_header_size_max = INT32_MAX
+ ccm_header_size_max = S32_MAX
};
struct aead_req_ctx {
/* Allocate cache line although only 4 bytes are needed to
- * assure next field falls @ cache line
- * Used for both: digest HW compare and CCM/GCM MAC value */
- uint8_t mac_buf[MAX_MAC_SIZE] ____cacheline_aligned;
- uint8_t ctr_iv[AES_BLOCK_SIZE] ____cacheline_aligned;
-
- //used in gcm
- uint8_t gcm_iv_inc1[AES_BLOCK_SIZE] ____cacheline_aligned;
- uint8_t gcm_iv_inc2[AES_BLOCK_SIZE] ____cacheline_aligned;
- uint8_t hkey[AES_BLOCK_SIZE] ____cacheline_aligned;
+ * assure next field falls @ cache line
+ * Used for both: digest HW compare and CCM/GCM MAC value
+ */
+ u8 mac_buf[MAX_MAC_SIZE] ____cacheline_aligned;
+ u8 ctr_iv[AES_BLOCK_SIZE] ____cacheline_aligned;
+
+ //used in gcm
+ u8 gcm_iv_inc1[AES_BLOCK_SIZE] ____cacheline_aligned;
+ u8 gcm_iv_inc2[AES_BLOCK_SIZE] ____cacheline_aligned;
+ u8 hkey[AES_BLOCK_SIZE] ____cacheline_aligned;
struct {
- uint8_t lenA[GCM_BLOCK_LEN_SIZE] ____cacheline_aligned;
- uint8_t lenC[GCM_BLOCK_LEN_SIZE] ;
+ u8 lenA[GCM_BLOCK_LEN_SIZE] ____cacheline_aligned;
+ u8 lenC[GCM_BLOCK_LEN_SIZE];
} gcm_len_block;
- uint8_t ccm_config[CCM_CONFIG_BUF_SIZE] ____cacheline_aligned;
+ u8 ccm_config[CCM_CONFIG_BUF_SIZE] ____cacheline_aligned;
unsigned int hw_iv_size ____cacheline_aligned; /*HW actual size input*/
- uint8_t backup_mac[MAX_MAC_SIZE]; /*used to prevent cache coherence problem*/
- uint8_t *backup_iv; /*store iv for generated IV flow*/
- uint8_t *backup_giv; /*store iv for rfc3686(ctr) flow*/
+ u8 backup_mac[MAX_MAC_SIZE]; /*used to prevent cache coherence problem*/
+ u8 *backup_iv; /*store iv for generated IV flow*/
+ u8 *backup_giv; /*store iv for rfc3686(ctr) flow*/
dma_addr_t mac_buf_dma_addr; /* internal ICV DMA buffer */
dma_addr_t ccm_iv0_dma_addr; /* buffer for internal ccm configurations */
dma_addr_t icv_dma_addr; /* Phys. address of ICV */
- //used in gcm
+ //used in gcm
dma_addr_t gcm_iv_inc1_dma_addr; /* buffer for internal gcm configurations */
dma_addr_t gcm_iv_inc2_dma_addr; /* buffer for internal gcm configurations */
dma_addr_t hkey_dma_addr; /* Phys. address of hkey */
dma_addr_t gcm_block_len_dma_addr; /* Phys. address of gcm block len */
bool is_gcm4543;
- uint8_t *icv_virt_addr; /* Virt. address of ICV */
+ u8 *icv_virt_addr; /* Virt. address of ICV */
struct async_gen_req_ctx gen_ctx;
struct ssi_mlli assoc;
struct ssi_mlli src;
struct ssi_mlli dst;
- struct scatterlist* srcSgl;
- struct scatterlist* dstSgl;
+ struct scatterlist *srcSgl;
+ struct scatterlist *dstSgl;
unsigned int srcOffset;
unsigned int dstOffset;
enum ssi_req_dma_buf_type assoc_buff_type;
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
#include "ssi_hash.h"
#include "ssi_aead.h"
-#define LLI_MAX_NUM_OF_DATA_ENTRIES 128
-#define LLI_MAX_NUM_OF_ASSOC_DATA_ENTRIES 4
-#define MLLI_TABLE_MIN_ALIGNMENT 4 /*Force the MLLI table to be align to uint32 */
-#define MAX_NUM_OF_BUFFERS_IN_MLLI 4
-#define MAX_NUM_OF_TOTAL_MLLI_ENTRIES (2*LLI_MAX_NUM_OF_DATA_ENTRIES + \
- LLI_MAX_NUM_OF_ASSOC_DATA_ENTRIES )
-
#ifdef CC_DEBUG
-#define DUMP_SGL(sg) \
- while (sg) { \
- SSI_LOG_DEBUG("page=%lu offset=%u length=%u (dma_len=%u) " \
- "dma_addr=%08x\n", (sg)->page_link, (sg)->offset, \
- (sg)->length, sg_dma_len(sg), (sg)->dma_address); \
- (sg) = sg_next(sg); \
- }
-#define DUMP_MLLI_TABLE(mlli_p, nents) \
- do { \
- SSI_LOG_DEBUG("mlli=%pK nents=%u\n", (mlli_p), (nents)); \
- while((nents)--) { \
- SSI_LOG_DEBUG("addr=0x%08X size=0x%08X\n", \
- (mlli_p)[LLI_WORD0_OFFSET], \
- (mlli_p)[LLI_WORD1_OFFSET]); \
- (mlli_p) += LLI_ENTRY_WORD_SIZE; \
- } \
- } while (0)
#define GET_DMA_BUFFER_TYPE(buff_type) ( \
((buff_type) == SSI_DMA_BUF_NULL) ? "BUF_NULL" : \
((buff_type) == SSI_DMA_BUF_DLLI) ? "BUF_DLLI" : \
((buff_type) == SSI_DMA_BUF_MLLI) ? "BUF_MLLI" : "BUF_INVALID")
#else
-#define DX_BUFFER_MGR_DUMP_SGL(sg)
-#define DX_BUFFER_MGR_DUMP_MLLI_TABLE(mlli_p, nents)
#define GET_DMA_BUFFER_TYPE(buff_type)
#endif
-
enum dma_buffer_type {
DMA_NULL_TYPE = -1,
DMA_SGL_TYPE = 1,
int total_data_len[MAX_NUM_OF_BUFFERS_IN_MLLI];
enum dma_buffer_type type[MAX_NUM_OF_BUFFERS_IN_MLLI];
bool is_last[MAX_NUM_OF_BUFFERS_IN_MLLI];
- uint32_t * mlli_nents[MAX_NUM_OF_BUFFERS_IN_MLLI];
+ u32 *mlli_nents[MAX_NUM_OF_BUFFERS_IN_MLLI];
};
-#ifdef CC_DMA_48BIT_SIM
-dma_addr_t ssi_buff_mgr_update_dma_addr(dma_addr_t orig_addr, uint32_t data_len)
-{
- dma_addr_t tmp_dma_addr;
-#ifdef CC_DMA_48BIT_SIM_FULL
- /* With this code all addresses will be switched to 48 bits. */
- /* The if condition protects from double expention */
- if((((orig_addr >> 16) & 0xFFFF) != 0xFFFF) &&
- (data_len <= CC_MAX_MLLI_ENTRY_SIZE)) {
-#else
- if((!(((orig_addr >> 16) & 0xFF) % 2)) &&
- (data_len <= CC_MAX_MLLI_ENTRY_SIZE)) {
-#endif
- tmp_dma_addr = ((orig_addr<<16) | 0xFFFF0000 |
- (orig_addr & UINT16_MAX));
- SSI_LOG_DEBUG("MAP DMA: orig address=0x%llX "
- "dma_address=0x%llX\n",
- orig_addr, tmp_dma_addr);
- return tmp_dma_addr;
- }
- return orig_addr;
-}
-
-dma_addr_t ssi_buff_mgr_restore_dma_addr(dma_addr_t orig_addr)
-{
- dma_addr_t tmp_dma_addr;
-#ifdef CC_DMA_48BIT_SIM_FULL
- /* With this code all addresses will be restored from 48 bits. */
- /* The if condition protects from double restoring */
- if((orig_addr >> 32) & 0xFFFF ) {
-#else
- if(((orig_addr >> 32) & 0xFFFF) &&
- !(((orig_addr >> 32) & 0xFF) % 2) ) {
-#endif
- /*return high 16 bits*/
- tmp_dma_addr = ((orig_addr >> 16));
- /*clean the 0xFFFF in the lower bits (set in the add expansion)*/
- tmp_dma_addr &= 0xFFFF0000;
- /* Set the original 16 bits */
- tmp_dma_addr |= (orig_addr & UINT16_MAX);
- SSI_LOG_DEBUG("Release DMA: orig address=0x%llX "
- "dma_address=0x%llX\n",
- orig_addr, tmp_dma_addr);
- return tmp_dma_addr;
- }
- return orig_addr;
-}
-#endif
/**
* ssi_buffer_mgr_get_sgl_nents() - Get scatterlist number of entries.
- *
+ *
* @sg_list: SG list
* @nbytes: [IN] Total SGL data bytes.
- * @lbytes: [OUT] Returns the amount of bytes at the last entry
+ * @lbytes: [OUT] Returns the amount of bytes at the last entry
*/
static unsigned int ssi_buffer_mgr_get_sgl_nents(
- struct scatterlist *sg_list, unsigned int nbytes, uint32_t *lbytes, bool *is_chained)
+ struct scatterlist *sg_list, unsigned int nbytes, u32 *lbytes, bool *is_chained)
{
unsigned int nents = 0;
+
while (nbytes != 0) {
if (sg_is_chain(sg_list)) {
- SSI_LOG_ERR("Unexpected chanined entry "
- "in sg (entry =0x%X) \n", nents);
+ SSI_LOG_ERR("Unexpected chained entry "
+ "in sg (entry =0x%X)\n", nents);
BUG();
}
if (sg_list->length != 0) {
nents++;
/* get the number of bytes in the last entry */
*lbytes = nbytes;
- nbytes -= ( sg_list->length > nbytes ) ? nbytes : sg_list->length;
+ nbytes -= (sg_list->length > nbytes) ? nbytes : sg_list->length;
sg_list = sg_next(sg_list);
} else {
sg_list = (struct scatterlist *)sg_page(sg_list);
- if (is_chained != NULL) {
+ if (is_chained)
*is_chained = true;
- }
}
}
- SSI_LOG_DEBUG("nents %d last bytes %d\n",nents, *lbytes);
+ SSI_LOG_DEBUG("nents %d last bytes %d\n", nents, *lbytes);
return nents;
}
/**
* ssi_buffer_mgr_zero_sgl() - Zero scatter scatter list data.
- *
+ *
* @sgl:
*/
-void ssi_buffer_mgr_zero_sgl(struct scatterlist *sgl, uint32_t data_len)
+void ssi_buffer_mgr_zero_sgl(struct scatterlist *sgl, u32 data_len)
{
struct scatterlist *current_sg = sgl;
int sg_index = 0;
while (sg_index <= data_len) {
- if (current_sg == NULL) {
+ if (!current_sg) {
/* reached the end of the sgl --> just return back */
return;
}
/**
* ssi_buffer_mgr_copy_scatterlist_portion() - Copy scatter list data,
* from to_skip to end, to dest and vice versa
- *
+ *
* @dest:
* @sg:
* @to_skip:
*/
void ssi_buffer_mgr_copy_scatterlist_portion(
u8 *dest, struct scatterlist *sg,
- uint32_t to_skip, uint32_t end,
+ u32 to_skip, u32 end,
enum ssi_sg_cpy_direct direct)
{
- uint32_t nents, lbytes;
+ u32 nents, lbytes;
nents = ssi_buffer_mgr_get_sgl_nents(sg, end, &lbytes, NULL);
sg_copy_buffer(sg, nents, (void *)dest, (end - to_skip + 1), to_skip,
}
static inline int ssi_buffer_mgr_render_buff_to_mlli(
- dma_addr_t buff_dma, uint32_t buff_size, uint32_t *curr_nents,
- uint32_t **mlli_entry_pp)
+ dma_addr_t buff_dma, u32 buff_size, u32 *curr_nents,
+ u32 **mlli_entry_pp)
{
- uint32_t *mlli_entry_p = *mlli_entry_pp;
- uint32_t new_nents;;
+ u32 *mlli_entry_p = *mlli_entry_pp;
+ u32 new_nents;;
/* Verify there is no memory overflow*/
- new_nents = (*curr_nents + buff_size/CC_MAX_MLLI_ENTRY_SIZE + 1);
- if (new_nents > MAX_NUM_OF_TOTAL_MLLI_ENTRIES ) {
+ new_nents = (*curr_nents + buff_size / CC_MAX_MLLI_ENTRY_SIZE + 1);
+ if (new_nents > MAX_NUM_OF_TOTAL_MLLI_ENTRIES)
return -ENOMEM;
- }
/*handle buffer longer than 64 kbytes */
- while (buff_size > CC_MAX_MLLI_ENTRY_SIZE ) {
- SSI_UPDATE_DMA_ADDR_TO_48BIT(buff_dma, CC_MAX_MLLI_ENTRY_SIZE);
- LLI_SET_ADDR(mlli_entry_p,buff_dma);
- LLI_SET_SIZE(mlli_entry_p, CC_MAX_MLLI_ENTRY_SIZE);
- SSI_LOG_DEBUG("entry[%d]: single_buff=0x%08X size=%08X\n",*curr_nents,
+ while (buff_size > CC_MAX_MLLI_ENTRY_SIZE) {
+ cc_lli_set_addr(mlli_entry_p, buff_dma);
+ cc_lli_set_size(mlli_entry_p, CC_MAX_MLLI_ENTRY_SIZE);
+ SSI_LOG_DEBUG("entry[%d]: single_buff=0x%08X size=%08X\n", *curr_nents,
mlli_entry_p[LLI_WORD0_OFFSET],
mlli_entry_p[LLI_WORD1_OFFSET]);
- SSI_RESTORE_DMA_ADDR_TO_48BIT(buff_dma);
buff_dma += CC_MAX_MLLI_ENTRY_SIZE;
buff_size -= CC_MAX_MLLI_ENTRY_SIZE;
mlli_entry_p = mlli_entry_p + 2;
(*curr_nents)++;
}
/*Last entry */
- SSI_UPDATE_DMA_ADDR_TO_48BIT(buff_dma, buff_size);
- LLI_SET_ADDR(mlli_entry_p,buff_dma);
- LLI_SET_SIZE(mlli_entry_p, buff_size);
- SSI_LOG_DEBUG("entry[%d]: single_buff=0x%08X size=%08X\n",*curr_nents,
+ cc_lli_set_addr(mlli_entry_p, buff_dma);
+ cc_lli_set_size(mlli_entry_p, buff_size);
+ SSI_LOG_DEBUG("entry[%d]: single_buff=0x%08X size=%08X\n", *curr_nents,
mlli_entry_p[LLI_WORD0_OFFSET],
mlli_entry_p[LLI_WORD1_OFFSET]);
mlli_entry_p = mlli_entry_p + 2;
return 0;
}
-
static inline int ssi_buffer_mgr_render_scatterlist_to_mlli(
- struct scatterlist *sgl, uint32_t sgl_data_len, uint32_t sglOffset, uint32_t *curr_nents,
- uint32_t **mlli_entry_pp)
+ struct scatterlist *sgl, u32 sgl_data_len, u32 sglOffset, u32 *curr_nents,
+ u32 **mlli_entry_pp)
{
struct scatterlist *curr_sgl = sgl;
- uint32_t *mlli_entry_p = *mlli_entry_pp;
- int32_t rc = 0;
+ u32 *mlli_entry_p = *mlli_entry_pp;
+ s32 rc = 0;
- for ( ; (curr_sgl != NULL) && (sgl_data_len != 0);
+ for ( ; (curr_sgl) && (sgl_data_len != 0);
curr_sgl = sg_next(curr_sgl)) {
- uint32_t entry_data_len =
+ u32 entry_data_len =
(sgl_data_len > sg_dma_len(curr_sgl) - sglOffset) ?
- sg_dma_len(curr_sgl) - sglOffset : sgl_data_len ;
+ sg_dma_len(curr_sgl) - sglOffset : sgl_data_len;
sgl_data_len -= entry_data_len;
rc = ssi_buffer_mgr_render_buff_to_mlli(
sg_dma_address(curr_sgl) + sglOffset, entry_data_len, curr_nents,
&mlli_entry_p);
- if(rc != 0) {
+ if (rc != 0)
return rc;
- }
- sglOffset=0;
+
+ sglOffset = 0;
}
*mlli_entry_pp = mlli_entry_p;
return 0;
struct buffer_array *sg_data,
struct mlli_params *mlli_params)
{
- uint32_t *mlli_p;
- uint32_t total_nents = 0,prev_total_nents = 0;
+ u32 *mlli_p;
+ u32 total_nents = 0, prev_total_nents = 0;
int rc = 0, i;
SSI_LOG_DEBUG("NUM of SG's = %d\n", sg_data->num_of_buffers);
mlli_params->mlli_virt_addr = dma_pool_alloc(
mlli_params->curr_pool, GFP_KERNEL,
&(mlli_params->mlli_dma_addr));
- if (unlikely(mlli_params->mlli_virt_addr == NULL)) {
+ if (unlikely(!mlli_params->mlli_virt_addr)) {
SSI_LOG_ERR("dma_pool_alloc() failed\n");
- rc =-ENOMEM;
+ rc = -ENOMEM;
goto build_mlli_exit;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(mlli_params->mlli_dma_addr,
- (MAX_NUM_OF_TOTAL_MLLI_ENTRIES*
- LLI_ENTRY_BYTE_SIZE));
/* Point to start of MLLI */
- mlli_p = (uint32_t *)mlli_params->mlli_virt_addr;
+ mlli_p = (u32 *)mlli_params->mlli_virt_addr;
/* go over all SG's and link it to one MLLI table */
for (i = 0; i < sg_data->num_of_buffers; i++) {
if (sg_data->type[i] == DMA_SGL_TYPE)
rc = ssi_buffer_mgr_render_scatterlist_to_mlli(
- sg_data->entry[i].sgl,
+ sg_data->entry[i].sgl,
sg_data->total_data_len[i], sg_data->offset[i], &total_nents,
&mlli_p);
else /*DMA_BUFF_TYPE*/
sg_data->entry[i].buffer_dma,
sg_data->total_data_len[i], &total_nents,
&mlli_p);
- if(rc != 0) {
+ if (rc != 0)
return rc;
- }
/* set last bit in the current table */
- if (sg_data->mlli_nents[i] != NULL) {
- /*Calculate the current MLLI table length for the
- length field in the descriptor*/
- *(sg_data->mlli_nents[i]) +=
+ if (sg_data->mlli_nents[i]) {
+ /*Calculate the current MLLI table length for the
+ *length field in the descriptor
+ */
+ *(sg_data->mlli_nents[i]) +=
(total_nents - prev_total_nents);
prev_total_nents = total_nents;
}
static inline void ssi_buffer_mgr_add_buffer_entry(
struct buffer_array *sgl_data,
dma_addr_t buffer_dma, unsigned int buffer_len,
- bool is_last_entry, uint32_t *mlli_nents)
+ bool is_last_entry, u32 *mlli_nents)
{
unsigned int index = sgl_data->num_of_buffers;
sgl_data->type[index] = DMA_BUFF_TYPE;
sgl_data->is_last[index] = is_last_entry;
sgl_data->mlli_nents[index] = mlli_nents;
- if (sgl_data->mlli_nents[index] != NULL)
+ if (sgl_data->mlli_nents[index])
*sgl_data->mlli_nents[index] = 0;
sgl_data->num_of_buffers++;
}
unsigned int data_len,
unsigned int data_offset,
bool is_last_table,
- uint32_t *mlli_nents)
+ u32 *mlli_nents)
{
unsigned int index = sgl_data->num_of_buffers;
sgl_data->type[index] = DMA_SGL_TYPE;
sgl_data->is_last[index] = is_last_table;
sgl_data->mlli_nents[index] = mlli_nents;
- if (sgl_data->mlli_nents[index] != NULL)
+ if (sgl_data->mlli_nents[index])
*sgl_data->mlli_nents[index] = 0;
sgl_data->num_of_buffers++;
}
static int
-ssi_buffer_mgr_dma_map_sg(struct device *dev, struct scatterlist *sg, uint32_t nents,
+ssi_buffer_mgr_dma_map_sg(struct device *dev, struct scatterlist *sg, u32 nents,
enum dma_data_direction direction)
{
- uint32_t i , j;
+ u32 i, j;
struct scatterlist *l_sg = sg;
+
for (i = 0; i < nents; i++) {
- if (l_sg == NULL) {
+ if (!l_sg)
break;
- }
- if (unlikely(dma_map_sg(dev, l_sg, 1, direction) != 1)){
+ if (unlikely(dma_map_sg(dev, l_sg, 1, direction) != 1)) {
SSI_LOG_ERR("dma_map_page() sg buffer failed\n");
goto err;
}
err:
/* Restore mapped parts */
for (j = 0; j < i; j++) {
- if (sg == NULL) {
+ if (!sg)
break;
- }
- dma_unmap_sg(dev,sg,1,direction);
+ dma_unmap_sg(dev, sg, 1, direction);
sg = sg_next(sg);
}
return 0;
static int ssi_buffer_mgr_map_scatterlist(
struct device *dev, struct scatterlist *sg,
unsigned int nbytes, int direction,
- uint32_t *nents, uint32_t max_sg_nents,
- uint32_t *lbytes, uint32_t *mapped_nents)
+ u32 *nents, u32 max_sg_nents,
+ u32 *lbytes, u32 *mapped_nents)
{
bool is_chained = false;
if (unlikely(dma_map_sg(dev, sg, 1, direction) != 1)) {
SSI_LOG_ERR("dma_map_sg() single buffer failed\n");
return -ENOMEM;
- }
+ }
SSI_LOG_DEBUG("Mapped sg: dma_address=0x%llX "
- "page_link=0x%08lX addr=%pK offset=%u "
+ "page=%p addr=%pK offset=%u "
"length=%u\n",
- (unsigned long long)sg_dma_address(sg),
- sg->page_link,
- sg_virt(sg),
+ (unsigned long long)sg_dma_address(sg),
+ sg_page(sg),
+ sg_virt(sg),
sg->offset, sg->length);
*lbytes = nbytes;
*nents = 1;
*mapped_nents = 1;
- SSI_UPDATE_DMA_ADDR_TO_48BIT(sg_dma_address(sg), sg_dma_len(sg));
} else { /*sg_is_last*/
- *nents = ssi_buffer_mgr_get_sgl_nents(sg, nbytes, lbytes,
+ *nents = ssi_buffer_mgr_get_sgl_nents(sg, nbytes, lbytes,
&is_chained);
if (*nents > max_sg_nents) {
*nents = 0;
}
if (!is_chained) {
/* In case of mmu the number of mapped nents might
- be changed from the original sgl nents */
+ * be changed from the original sgl nents
+ */
*mapped_nents = dma_map_sg(dev, sg, *nents, direction);
- if (unlikely(*mapped_nents == 0)){
+ if (unlikely(*mapped_nents == 0)) {
*nents = 0;
SSI_LOG_ERR("dma_map_sg() sg buffer failed\n");
return -ENOMEM;
}
} else {
/*In this case the driver maps entry by entry so it
- must have the same nents before and after map */
+ * must have the same nents before and after map
+ */
*mapped_nents = ssi_buffer_mgr_dma_map_sg(dev,
sg,
*nents,
direction);
- if (unlikely(*mapped_nents != *nents)){
+ if (unlikely(*mapped_nents != *nents)) {
*nents = *mapped_nents;
SSI_LOG_ERR("dma_map_sg() sg buffer failed\n");
return -ENOMEM;
static inline int
ssi_aead_handle_config_buf(struct device *dev,
struct aead_req_ctx *areq_ctx,
- uint8_t* config_data,
+ u8 *config_data,
struct buffer_array *sg_data,
unsigned int assoclen)
{
- SSI_LOG_DEBUG(" handle additional data config set to DLLI \n");
+ SSI_LOG_DEBUG(" handle additional data config set to DLLI\n");
/* create sg for the current buffer */
sg_init_one(&areq_ctx->ccm_adata_sg, config_data, AES_BLOCK_SIZE + areq_ctx->ccm_hdr_size);
- if (unlikely(dma_map_sg(dev, &areq_ctx->ccm_adata_sg, 1,
+ if (unlikely(dma_map_sg(dev, &areq_ctx->ccm_adata_sg, 1,
DMA_TO_DEVICE) != 1)) {
SSI_LOG_ERR("dma_map_sg() "
"config buffer failed\n");
return -ENOMEM;
}
SSI_LOG_DEBUG("Mapped curr_buff: dma_address=0x%llX "
- "page_link=0x%08lX addr=%pK "
+ "page=%p addr=%pK "
"offset=%u length=%u\n",
- (unsigned long long)sg_dma_address(&areq_ctx->ccm_adata_sg),
- areq_ctx->ccm_adata_sg.page_link,
+ (unsigned long long)sg_dma_address(&areq_ctx->ccm_adata_sg),
+ sg_page(&areq_ctx->ccm_adata_sg),
sg_virt(&areq_ctx->ccm_adata_sg),
- areq_ctx->ccm_adata_sg.offset,
+ areq_ctx->ccm_adata_sg.offset,
areq_ctx->ccm_adata_sg.length);
/* prepare for case of MLLI */
if (assoclen > 0) {
- ssi_buffer_mgr_add_scatterlist_entry(sg_data, 1,
+ ssi_buffer_mgr_add_scatterlist_entry(sg_data, 1,
&areq_ctx->ccm_adata_sg,
- (AES_BLOCK_SIZE +
+ (AES_BLOCK_SIZE +
areq_ctx->ccm_hdr_size), 0,
false, NULL);
}
return 0;
}
-
static inline int ssi_ahash_handle_curr_buf(struct device *dev,
struct ahash_req_ctx *areq_ctx,
- uint8_t* curr_buff,
- uint32_t curr_buff_cnt,
+ u8 *curr_buff,
+ u32 curr_buff_cnt,
struct buffer_array *sg_data)
{
- SSI_LOG_DEBUG(" handle curr buff %x set to DLLI \n", curr_buff_cnt);
+ SSI_LOG_DEBUG(" handle curr buff %x set to DLLI\n", curr_buff_cnt);
/* create sg for the current buffer */
- sg_init_one(areq_ctx->buff_sg,curr_buff, curr_buff_cnt);
+ sg_init_one(areq_ctx->buff_sg, curr_buff, curr_buff_cnt);
if (unlikely(dma_map_sg(dev, areq_ctx->buff_sg, 1,
DMA_TO_DEVICE) != 1)) {
SSI_LOG_ERR("dma_map_sg() "
return -ENOMEM;
}
SSI_LOG_DEBUG("Mapped curr_buff: dma_address=0x%llX "
- "page_link=0x%08lX addr=%pK "
+ "page=%p addr=%pK "
"offset=%u length=%u\n",
- (unsigned long long)sg_dma_address(areq_ctx->buff_sg),
- areq_ctx->buff_sg->page_link,
+ (unsigned long long)sg_dma_address(areq_ctx->buff_sg),
+ sg_page(areq_ctx->buff_sg),
sg_virt(areq_ctx->buff_sg),
- areq_ctx->buff_sg->offset,
+ areq_ctx->buff_sg->offset,
areq_ctx->buff_sg->length);
areq_ctx->data_dma_buf_type = SSI_DMA_BUF_DLLI;
areq_ctx->curr_sg = areq_ctx->buff_sg;
struct blkcipher_req_ctx *req_ctx = (struct blkcipher_req_ctx *)ctx;
if (likely(req_ctx->gen_ctx.iv_dma_addr != 0)) {
- SSI_LOG_DEBUG("Unmapped iv: iv_dma_addr=0x%llX iv_size=%u\n",
+ SSI_LOG_DEBUG("Unmapped iv: iv_dma_addr=0x%llX iv_size=%u\n",
(unsigned long long)req_ctx->gen_ctx.iv_dma_addr,
ivsize);
- SSI_RESTORE_DMA_ADDR_TO_48BIT(req_ctx->gen_ctx.iv_dma_addr);
- dma_unmap_single(dev, req_ctx->gen_ctx.iv_dma_addr,
- ivsize,
+ dma_unmap_single(dev, req_ctx->gen_ctx.iv_dma_addr,
+ ivsize,
req_ctx->is_giv ? DMA_BIDIRECTIONAL :
DMA_TO_DEVICE);
}
/* Release pool */
if (req_ctx->dma_buf_type == SSI_DMA_BUF_MLLI) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(req_ctx->mlli_params.mlli_dma_addr);
dma_pool_free(req_ctx->mlli_params.curr_pool,
req_ctx->mlli_params.mlli_virt_addr,
req_ctx->mlli_params.mlli_dma_addr);
}
- SSI_RESTORE_DMA_ADDR_TO_48BIT(sg_dma_address(src));
dma_unmap_sg(dev, src, req_ctx->in_nents,
DMA_BIDIRECTIONAL);
- SSI_LOG_DEBUG("Unmapped req->src=%pK\n",
+ SSI_LOG_DEBUG("Unmapped req->src=%pK\n",
sg_virt(src));
if (src != dst) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(sg_dma_address(dst));
- dma_unmap_sg(dev, dst, req_ctx->out_nents,
+ dma_unmap_sg(dev, dst, req_ctx->out_nents,
DMA_BIDIRECTIONAL);
SSI_LOG_DEBUG("Unmapped req->dst=%pK\n",
sg_virt(dst));
struct scatterlist *dst)
{
struct blkcipher_req_ctx *req_ctx = (struct blkcipher_req_ctx *)ctx;
- struct mlli_params *mlli_params = &req_ctx->mlli_params;
+ struct mlli_params *mlli_params = &req_ctx->mlli_params;
struct buff_mgr_handle *buff_mgr = drvdata->buff_mgr_handle;
struct device *dev = &drvdata->plat_dev->dev;
struct buffer_array sg_data;
- uint32_t dummy = 0;
+ u32 dummy = 0;
int rc = 0;
- uint32_t mapped_nents = 0;
+ u32 mapped_nents = 0;
req_ctx->dma_buf_type = SSI_DMA_BUF_DLLI;
mlli_params->curr_pool = NULL;
sg_data.num_of_buffers = 0;
/* Map IV buffer */
- if (likely(ivsize != 0) ) {
- dump_byte_array("iv", (uint8_t *)info, ivsize);
- req_ctx->gen_ctx.iv_dma_addr =
- dma_map_single(dev, (void *)info,
- ivsize,
- req_ctx->is_giv ? DMA_BIDIRECTIONAL:
+ if (likely(ivsize != 0)) {
+ dump_byte_array("iv", (u8 *)info, ivsize);
+ req_ctx->gen_ctx.iv_dma_addr =
+ dma_map_single(dev, (void *)info,
+ ivsize,
+ req_ctx->is_giv ? DMA_BIDIRECTIONAL :
DMA_TO_DEVICE);
- if (unlikely(dma_mapping_error(dev,
+ if (unlikely(dma_mapping_error(dev,
req_ctx->gen_ctx.iv_dma_addr))) {
SSI_LOG_ERR("Mapping iv %u B at va=%pK "
"for DMA failed\n", ivsize, info);
return -ENOMEM;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(req_ctx->gen_ctx.iv_dma_addr,
- ivsize);
SSI_LOG_DEBUG("Mapped iv %u B at va=%pK to dma=0x%llX\n",
ivsize, info,
(unsigned long long)req_ctx->gen_ctx.iv_dma_addr);
- } else
+ } else {
req_ctx->gen_ctx.iv_dma_addr = 0;
-
+ }
+
/* Map the src SGL */
rc = ssi_buffer_mgr_map_scatterlist(dev, src,
nbytes, DMA_BIDIRECTIONAL, &req_ctx->in_nents,
} else {
/* Map the dst sg */
if (unlikely(ssi_buffer_mgr_map_scatterlist(
- dev,dst, nbytes,
+ dev, dst, nbytes,
DMA_BIDIRECTIONAL, &req_ctx->out_nents,
LLI_MAX_NUM_OF_DATA_ENTRIES, &dummy,
&mapped_nents))){
&req_ctx->in_mlli_nents);
ssi_buffer_mgr_add_scatterlist_entry(&sg_data,
req_ctx->out_nents, dst,
- nbytes, 0, true,
+ nbytes, 0, true,
&req_ctx->out_mlli_nents);
}
}
-
+
if (unlikely(req_ctx->dma_buf_type == SSI_DMA_BUF_MLLI)) {
mlli_params->curr_pool = buff_mgr->mlli_buffs_pool;
rc = ssi_buffer_mgr_generate_mlli(dev, &sg_data, mlli_params);
- if (unlikely(rc!= 0))
+ if (unlikely(rc != 0))
goto ablkcipher_exit;
-
}
SSI_LOG_DEBUG("areq_ctx->dma_buf_type = %s\n",
struct aead_req_ctx *areq_ctx = aead_request_ctx(req);
unsigned int hw_iv_size = areq_ctx->hw_iv_size;
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
- uint32_t dummy;
+ struct ssi_drvdata *drvdata = dev_get_drvdata(dev);
+ u32 dummy;
bool chained;
- uint32_t size_to_unmap = 0;
+ u32 size_to_unmap = 0;
if (areq_ctx->mac_buf_dma_addr != 0) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(areq_ctx->mac_buf_dma_addr);
- dma_unmap_single(dev, areq_ctx->mac_buf_dma_addr,
+ dma_unmap_single(dev, areq_ctx->mac_buf_dma_addr,
MAX_MAC_SIZE, DMA_BIDIRECTIONAL);
}
#if SSI_CC_HAS_AES_GCM
if (areq_ctx->cipher_mode == DRV_CIPHER_GCTR) {
if (areq_ctx->hkey_dma_addr != 0) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(areq_ctx->hkey_dma_addr);
dma_unmap_single(dev, areq_ctx->hkey_dma_addr,
AES_BLOCK_SIZE, DMA_BIDIRECTIONAL);
}
-
+
if (areq_ctx->gcm_block_len_dma_addr != 0) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(areq_ctx->gcm_block_len_dma_addr);
dma_unmap_single(dev, areq_ctx->gcm_block_len_dma_addr,
AES_BLOCK_SIZE, DMA_TO_DEVICE);
}
-
+
if (areq_ctx->gcm_iv_inc1_dma_addr != 0) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(areq_ctx->gcm_iv_inc1_dma_addr);
- dma_unmap_single(dev, areq_ctx->gcm_iv_inc1_dma_addr,
+ dma_unmap_single(dev, areq_ctx->gcm_iv_inc1_dma_addr,
AES_BLOCK_SIZE, DMA_TO_DEVICE);
}
-
+
if (areq_ctx->gcm_iv_inc2_dma_addr != 0) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(areq_ctx->gcm_iv_inc2_dma_addr);
- dma_unmap_single(dev, areq_ctx->gcm_iv_inc2_dma_addr,
+ dma_unmap_single(dev, areq_ctx->gcm_iv_inc2_dma_addr,
AES_BLOCK_SIZE, DMA_TO_DEVICE);
}
}
if (areq_ctx->ccm_hdr_size != ccm_header_size_null) {
if (areq_ctx->ccm_iv0_dma_addr != 0) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(areq_ctx->ccm_iv0_dma_addr);
- dma_unmap_single(dev, areq_ctx->ccm_iv0_dma_addr,
+ dma_unmap_single(dev, areq_ctx->ccm_iv0_dma_addr,
AES_BLOCK_SIZE, DMA_TO_DEVICE);
}
dma_unmap_sg(dev, &areq_ctx->ccm_adata_sg, 1, DMA_TO_DEVICE);
}
if (areq_ctx->gen_ctx.iv_dma_addr != 0) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(areq_ctx->gen_ctx.iv_dma_addr);
dma_unmap_single(dev, areq_ctx->gen_ctx.iv_dma_addr,
hw_iv_size, DMA_BIDIRECTIONAL);
}
- /*In case a pool was set, a table was
- allocated and should be released */
- if (areq_ctx->mlli_params.curr_pool != NULL) {
- SSI_LOG_DEBUG("free MLLI buffer: dma=0x%08llX virt=%pK\n",
+ /*In case a pool was set, a table was
+ *allocated and should be released
+ */
+ if (areq_ctx->mlli_params.curr_pool) {
+ SSI_LOG_DEBUG("free MLLI buffer: dma=0x%08llX virt=%pK\n",
(unsigned long long)areq_ctx->mlli_params.mlli_dma_addr,
areq_ctx->mlli_params.mlli_virt_addr);
- SSI_RESTORE_DMA_ADDR_TO_48BIT(areq_ctx->mlli_params.mlli_dma_addr);
dma_pool_free(areq_ctx->mlli_params.curr_pool,
areq_ctx->mlli_params.mlli_virt_addr,
areq_ctx->mlli_params.mlli_dma_addr);
}
- SSI_LOG_DEBUG("Unmapping src sgl: req->src=%pK areq_ctx->src.nents=%u areq_ctx->assoc.nents=%u assoclen:%u cryptlen=%u\n", sg_virt(req->src),areq_ctx->src.nents,areq_ctx->assoc.nents,req->assoclen,req->cryptlen);
- SSI_RESTORE_DMA_ADDR_TO_48BIT(sg_dma_address(req->src));
- size_to_unmap = req->assoclen+req->cryptlen;
- if(areq_ctx->gen_ctx.op_type == DRV_CRYPTO_DIRECTION_ENCRYPT){
+ SSI_LOG_DEBUG("Unmapping src sgl: req->src=%pK areq_ctx->src.nents=%u areq_ctx->assoc.nents=%u assoclen:%u cryptlen=%u\n", sg_virt(req->src), areq_ctx->src.nents, areq_ctx->assoc.nents, req->assoclen, req->cryptlen);
+ size_to_unmap = req->assoclen + req->cryptlen;
+ if (areq_ctx->gen_ctx.op_type == DRV_CRYPTO_DIRECTION_ENCRYPT)
size_to_unmap += areq_ctx->req_authsize;
- }
if (areq_ctx->is_gcm4543)
size_to_unmap += crypto_aead_ivsize(tfm);
- dma_unmap_sg(dev, req->src, ssi_buffer_mgr_get_sgl_nents(req->src,size_to_unmap,&dummy,&chained) , DMA_BIDIRECTIONAL);
+ dma_unmap_sg(dev, req->src, ssi_buffer_mgr_get_sgl_nents(req->src, size_to_unmap, &dummy, &chained), DMA_BIDIRECTIONAL);
if (unlikely(req->src != req->dst)) {
- SSI_LOG_DEBUG("Unmapping dst sgl: req->dst=%pK\n",
+ SSI_LOG_DEBUG("Unmapping dst sgl: req->dst=%pK\n",
sg_virt(req->dst));
- SSI_RESTORE_DMA_ADDR_TO_48BIT(sg_dma_address(req->dst));
- dma_unmap_sg(dev, req->dst, ssi_buffer_mgr_get_sgl_nents(req->dst,size_to_unmap,&dummy,&chained),
+ dma_unmap_sg(dev, req->dst, ssi_buffer_mgr_get_sgl_nents(req->dst, size_to_unmap, &dummy, &chained),
DMA_BIDIRECTIONAL);
}
-#if DX_HAS_ACP
- if ((areq_ctx->gen_ctx.op_type == DRV_CRYPTO_DIRECTION_DECRYPT) &&
+ if (drvdata->coherent &&
+ (areq_ctx->gen_ctx.op_type == DRV_CRYPTO_DIRECTION_DECRYPT) &&
likely(req->src == req->dst))
{
- uint32_t size_to_skip = req->assoclen;
- if (areq_ctx->is_gcm4543) {
+ u32 size_to_skip = req->assoclen;
+
+ if (areq_ctx->is_gcm4543)
size_to_skip += crypto_aead_ivsize(tfm);
- }
+
/* copy mac to a temporary location to deal with possible
- data memory overriding that caused by cache coherence problem. */
+ * data memory overriding that caused by cache coherence problem.
+ */
ssi_buffer_mgr_copy_scatterlist_portion(
areq_ctx->backup_mac, req->src,
- size_to_skip+ req->cryptlen - areq_ctx->req_authsize,
- size_to_skip+ req->cryptlen, SSI_SG_FROM_BUF);
+ size_to_skip + req->cryptlen - areq_ctx->req_authsize,
+ size_to_skip + req->cryptlen, SSI_SG_FROM_BUF);
}
-#endif
}
static inline int ssi_buffer_mgr_get_aead_icv_nents(
struct scatterlist *sgl,
unsigned int sgl_nents,
unsigned int authsize,
- uint32_t last_entry_data_size,
+ u32 last_entry_data_size,
bool *is_icv_fragmented)
{
unsigned int icv_max_size = 0;
unsigned int icv_required_size = authsize > last_entry_data_size ? (authsize - last_entry_data_size) : authsize;
unsigned int nents;
unsigned int i;
-
+
if (sgl_nents < MAX_ICV_NENTS_SUPPORTED) {
*is_icv_fragmented = false;
return 0;
}
-
- for( i = 0 ; i < (sgl_nents - MAX_ICV_NENTS_SUPPORTED) ; i++) {
- if (sgl == NULL) {
+
+ for (i = 0 ; i < (sgl_nents - MAX_ICV_NENTS_SUPPORTED) ; i++) {
+ if (!sgl)
break;
- }
sgl = sg_next(sgl);
}
- if (sgl != NULL) {
+ if (sgl)
icv_max_size = sgl->length;
- }
if (last_entry_data_size > authsize) {
nents = 0; /* ICV attached to data in last entry (not fragmented!) */
struct device *dev = &drvdata->plat_dev->dev;
int rc = 0;
- if (unlikely(req->iv == NULL)) {
+ if (unlikely(!req->iv)) {
areq_ctx->gen_ctx.iv_dma_addr = 0;
goto chain_iv_exit;
}
SSI_LOG_ERR("Mapping iv %u B at va=%pK for DMA failed\n",
hw_iv_size, req->iv);
rc = -ENOMEM;
- goto chain_iv_exit;
+ goto chain_iv_exit;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(areq_ctx->gen_ctx.iv_dma_addr, hw_iv_size);
SSI_LOG_DEBUG("Mapped iv %u B at va=%pK to dma=0x%llX\n",
- hw_iv_size, req->iv,
+ hw_iv_size, req->iv,
(unsigned long long)areq_ctx->gen_ctx.iv_dma_addr);
- if (do_chain == true && areq_ctx->plaintext_authenticate_only == true){ // TODO: what about CTR?? ask Ron
+ if (do_chain && areq_ctx->plaintext_authenticate_only) { // TODO: what about CTR?? ask Ron
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
unsigned int iv_size_to_authenc = crypto_aead_ivsize(tfm);
unsigned int iv_ofs = GCM_BLOCK_RFC4_IV_OFFSET;
{
struct aead_req_ctx *areq_ctx = aead_request_ctx(req);
int rc = 0;
- uint32_t mapped_nents = 0;
+ u32 mapped_nents = 0;
struct scatterlist *current_sg = req->src;
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
unsigned int sg_index = 0;
- uint32_t size_of_assoc = req->assoclen;
+ u32 size_of_assoc = req->assoclen;
- if (areq_ctx->is_gcm4543) {
+ if (areq_ctx->is_gcm4543)
size_of_assoc += crypto_aead_ivsize(tfm);
- }
- if (sg_data == NULL) {
+ if (!sg_data) {
rc = -EINVAL;
goto chain_assoc_exit;
}
//it is assumed that if we reach here , the sgl is already mapped
sg_index = current_sg->length;
if (sg_index > size_of_assoc) { //the first entry in the scatter list contains all the associated data
- mapped_nents++;
- }
- else{
+ mapped_nents++;
+ } else {
while (sg_index <= size_of_assoc) {
current_sg = sg_next(current_sg);
//if have reached the end of the sgl, then this is unexpected
- if (current_sg == NULL) {
- SSI_LOG_ERR("reached end of sg list. unexpected \n");
+ if (!current_sg) {
+ SSI_LOG_ERR("reached end of sg list. unexpected\n");
BUG();
}
sg_index += current_sg->length;
areq_ctx->assoc.nents = mapped_nents;
/* in CCM case we have additional entry for
- * ccm header configurations */
+ * ccm header configurations
+ */
if (areq_ctx->ccm_hdr_size != ccm_header_size_null) {
if (unlikely((mapped_nents + 1) >
LLI_MAX_NUM_OF_ASSOC_DATA_ENTRIES)) {
-
SSI_LOG_ERR("CCM case.Too many fragments. "
"Current %d max %d\n",
(areq_ctx->assoc.nents + 1),
else
areq_ctx->assoc_buff_type = SSI_DMA_BUF_MLLI;
- if (unlikely((do_chain == true) ||
+ if (unlikely((do_chain) ||
(areq_ctx->assoc_buff_type == SSI_DMA_BUF_MLLI))) {
-
SSI_LOG_DEBUG("Chain assoc: buff_type=%s nents=%u\n",
GET_DMA_BUFFER_TYPE(areq_ctx->assoc_buff_type),
areq_ctx->assoc.nents);
static inline void ssi_buffer_mgr_prepare_aead_data_dlli(
struct aead_request *req,
- uint32_t *src_last_bytes, uint32_t *dst_last_bytes)
+ u32 *src_last_bytes, u32 *dst_last_bytes)
{
struct aead_req_ctx *areq_ctx = aead_request_ctx(req);
enum drv_crypto_direction direct = areq_ctx->gen_ctx.op_type;
if (likely(req->src == req->dst)) {
/*INPLACE*/
areq_ctx->icv_dma_addr = sg_dma_address(
- areq_ctx->srcSgl)+
+ areq_ctx->srcSgl) +
(*src_last_bytes - authsize);
areq_ctx->icv_virt_addr = sg_virt(
areq_ctx->srcSgl) +
areq_ctx->dstSgl) +
(*dst_last_bytes - authsize);
areq_ctx->icv_virt_addr = sg_virt(
- areq_ctx->dstSgl)+
+ areq_ctx->dstSgl) +
(*dst_last_bytes - authsize);
}
}
struct ssi_drvdata *drvdata,
struct aead_request *req,
struct buffer_array *sg_data,
- uint32_t *src_last_bytes, uint32_t *dst_last_bytes,
+ u32 *src_last_bytes, u32 *dst_last_bytes,
bool is_last_table)
{
struct aead_req_ctx *areq_ctx = aead_request_ctx(req);
/*INPLACE*/
ssi_buffer_mgr_add_scatterlist_entry(sg_data,
areq_ctx->src.nents, areq_ctx->srcSgl,
- areq_ctx->cryptlen,areq_ctx->srcOffset, is_last_table,
+ areq_ctx->cryptlen, areq_ctx->srcOffset, is_last_table,
&areq_ctx->src.mlli_nents);
icv_nents = ssi_buffer_mgr_get_aead_icv_nents(areq_ctx->srcSgl,
goto prepare_data_mlli_exit;
}
- if (unlikely(areq_ctx->is_icv_fragmented == true)) {
+ if (unlikely(areq_ctx->is_icv_fragmented)) {
/* Backup happens only when ICV is fragmented, ICV
- verification is made by CPU compare in order to simplify
- MAC verification upon request completion */
+ * verification is made by CPU compare in order to simplify
+ * MAC verification upon request completion
+ */
if (direct == DRV_CRYPTO_DIRECTION_DECRYPT) {
-#if !DX_HAS_ACP
- /* In ACP platform we already copying ICV
- for any INPLACE-DECRYPT operation, hence
- we must neglect this code. */
- uint32_t size_to_skip = req->assoclen;
- if (areq_ctx->is_gcm4543) {
- size_to_skip += crypto_aead_ivsize(tfm);
+ if (!drvdata->coherent) {
+ /* In coherent platforms (e.g. ACP)
+ * already copying ICV for any
+ * INPLACE-DECRYPT operation, hence
+ * we must neglect this code.
+ */
+ u32 skip = req->assoclen;
+
+ if (areq_ctx->is_gcm4543)
+ skip += crypto_aead_ivsize(tfm);
+
+ ssi_buffer_mgr_copy_scatterlist_portion(
+ areq_ctx->backup_mac, req->src,
+ (skip + req->cryptlen -
+ areq_ctx->req_authsize),
+ skip + req->cryptlen,
+ SSI_SG_TO_BUF);
}
- ssi_buffer_mgr_copy_scatterlist_portion(
- areq_ctx->backup_mac, req->src,
- size_to_skip+ req->cryptlen - areq_ctx->req_authsize,
- size_to_skip+ req->cryptlen, SSI_SG_TO_BUF);
-#endif
areq_ctx->icv_virt_addr = areq_ctx->backup_mac;
} else {
areq_ctx->icv_virt_addr = areq_ctx->mac_buf;
&areq_ctx->srcSgl[areq_ctx->src.nents - 1]) +
(*src_last_bytes - authsize);
areq_ctx->icv_virt_addr = sg_virt(
- &areq_ctx->srcSgl[areq_ctx->src.nents - 1]) +
+ &areq_ctx->srcSgl[areq_ctx->src.nents - 1]) +
(*src_last_bytes - authsize);
}
/*NON-INPLACE and DECRYPT*/
ssi_buffer_mgr_add_scatterlist_entry(sg_data,
areq_ctx->src.nents, areq_ctx->srcSgl,
- areq_ctx->cryptlen, areq_ctx->srcOffset,is_last_table,
+ areq_ctx->cryptlen, areq_ctx->srcOffset, is_last_table,
&areq_ctx->src.mlli_nents);
ssi_buffer_mgr_add_scatterlist_entry(sg_data,
areq_ctx->dst.nents, areq_ctx->dstSgl,
- areq_ctx->cryptlen,areq_ctx->dstOffset, is_last_table,
+ areq_ctx->cryptlen, areq_ctx->dstOffset, is_last_table,
&areq_ctx->dst.mlli_nents);
icv_nents = ssi_buffer_mgr_get_aead_icv_nents(areq_ctx->srcSgl,
goto prepare_data_mlli_exit;
}
- if (unlikely(areq_ctx->is_icv_fragmented == true)) {
+ if (unlikely(areq_ctx->is_icv_fragmented)) {
/* Backup happens only when ICV is fragmented, ICV
- verification is made by CPU compare in order to simplify
- MAC verification upon request completion */
- uint32_t size_to_skip = req->assoclen;
- if (areq_ctx->is_gcm4543) {
+ * verification is made by CPU compare in order to simplify
+ * MAC verification upon request completion
+ */
+ u32 size_to_skip = req->assoclen;
+
+ if (areq_ctx->is_gcm4543)
size_to_skip += crypto_aead_ivsize(tfm);
- }
+
ssi_buffer_mgr_copy_scatterlist_portion(
areq_ctx->backup_mac, req->src,
- size_to_skip+ req->cryptlen - areq_ctx->req_authsize,
- size_to_skip+ req->cryptlen, SSI_SG_TO_BUF);
+ size_to_skip + req->cryptlen - areq_ctx->req_authsize,
+ size_to_skip + req->cryptlen, SSI_SG_TO_BUF);
areq_ctx->icv_virt_addr = areq_ctx->backup_mac;
} else { /* Contig. ICV */
/*Should hanlde if the sg is not contig.*/
/*NON-INPLACE and ENCRYPT*/
ssi_buffer_mgr_add_scatterlist_entry(sg_data,
areq_ctx->dst.nents, areq_ctx->dstSgl,
- areq_ctx->cryptlen,areq_ctx->dstOffset, is_last_table,
+ areq_ctx->cryptlen, areq_ctx->dstOffset, is_last_table,
&areq_ctx->dst.mlli_nents);
ssi_buffer_mgr_add_scatterlist_entry(sg_data,
areq_ctx->src.nents, areq_ctx->srcSgl,
- areq_ctx->cryptlen, areq_ctx->srcOffset,is_last_table,
+ areq_ctx->cryptlen, areq_ctx->srcOffset, is_last_table,
&areq_ctx->src.mlli_nents);
icv_nents = ssi_buffer_mgr_get_aead_icv_nents(areq_ctx->dstSgl,
goto prepare_data_mlli_exit;
}
- if (likely(areq_ctx->is_icv_fragmented == false)) {
+ if (likely(!areq_ctx->is_icv_fragmented)) {
/* Contig. ICV */
areq_ctx->icv_dma_addr = sg_dma_address(
&areq_ctx->dstSgl[areq_ctx->dst.nents - 1]) +
unsigned int authsize = areq_ctx->req_authsize;
int src_last_bytes = 0, dst_last_bytes = 0;
int rc = 0;
- uint32_t src_mapped_nents = 0, dst_mapped_nents = 0;
- uint32_t offset = 0;
- unsigned int size_for_map = req->assoclen +req->cryptlen; /*non-inplace mode*/
+ u32 src_mapped_nents = 0, dst_mapped_nents = 0;
+ u32 offset = 0;
+ unsigned int size_for_map = req->assoclen + req->cryptlen; /*non-inplace mode*/
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
- uint32_t sg_index = 0;
+ u32 sg_index = 0;
bool chained = false;
bool is_gcm4543 = areq_ctx->is_gcm4543;
- uint32_t size_to_skip = req->assoclen;
- if (is_gcm4543) {
+ u32 size_to_skip = req->assoclen;
+
+ if (is_gcm4543)
size_to_skip += crypto_aead_ivsize(tfm);
- }
+
offset = size_to_skip;
- if (sg_data == NULL) {
+ if (!sg_data) {
rc = -EINVAL;
goto chain_data_exit;
}
areq_ctx->srcSgl = req->src;
areq_ctx->dstSgl = req->dst;
- if (is_gcm4543) {
+ if (is_gcm4543)
size_for_map += crypto_aead_ivsize(tfm);
- }
- size_for_map += (direct == DRV_CRYPTO_DIRECTION_ENCRYPT) ? authsize:0;
- src_mapped_nents = ssi_buffer_mgr_get_sgl_nents(req->src,size_for_map,&src_last_bytes, &chained);
+ size_for_map += (direct == DRV_CRYPTO_DIRECTION_ENCRYPT) ? authsize : 0;
+ src_mapped_nents = ssi_buffer_mgr_get_sgl_nents(req->src, size_for_map, &src_last_bytes, &chained);
sg_index = areq_ctx->srcSgl->length;
//check where the data starts
while (sg_index <= size_to_skip) {
offset -= areq_ctx->srcSgl->length;
areq_ctx->srcSgl = sg_next(areq_ctx->srcSgl);
//if have reached the end of the sgl, then this is unexpected
- if (areq_ctx->srcSgl == NULL) {
- SSI_LOG_ERR("reached end of sg list. unexpected \n");
+ if (!areq_ctx->srcSgl) {
+ SSI_LOG_ERR("reached end of sg list. unexpected\n");
BUG();
}
sg_index += areq_ctx->srcSgl->length;
areq_ctx->src.nents = src_mapped_nents;
- areq_ctx->srcOffset = offset;
+ areq_ctx->srcOffset = offset;
if (req->src != req->dst) {
- size_for_map = req->assoclen +req->cryptlen;
+ size_for_map = req->assoclen + req->cryptlen;
size_for_map += (direct == DRV_CRYPTO_DIRECTION_ENCRYPT) ? authsize : 0;
- if (is_gcm4543) {
+ if (is_gcm4543)
size_for_map += crypto_aead_ivsize(tfm);
- }
rc = ssi_buffer_mgr_map_scatterlist(dev, req->dst, size_for_map,
DMA_BIDIRECTIONAL, &(areq_ctx->dst.nents),
&dst_mapped_nents);
if (unlikely(rc != 0)) {
rc = -ENOMEM;
- goto chain_data_exit;
+ goto chain_data_exit;
}
}
- dst_mapped_nents = ssi_buffer_mgr_get_sgl_nents(req->dst,size_for_map,&dst_last_bytes, &chained);
+ dst_mapped_nents = ssi_buffer_mgr_get_sgl_nents(req->dst, size_for_map, &dst_last_bytes, &chained);
sg_index = areq_ctx->dstSgl->length;
offset = size_to_skip;
//check where the data starts
while (sg_index <= size_to_skip) {
-
offset -= areq_ctx->dstSgl->length;
areq_ctx->dstSgl = sg_next(areq_ctx->dstSgl);
//if have reached the end of the sgl, then this is unexpected
- if (areq_ctx->dstSgl == NULL) {
- SSI_LOG_ERR("reached end of sg list. unexpected \n");
+ if (!areq_ctx->dstSgl) {
+ SSI_LOG_ERR("reached end of sg list. unexpected\n");
BUG();
}
sg_index += areq_ctx->dstSgl->length;
areq_ctx->dstOffset = offset;
if ((src_mapped_nents > 1) ||
(dst_mapped_nents > 1) ||
- (do_chain == true)) {
+ do_chain) {
areq_ctx->data_buff_type = SSI_DMA_BUF_MLLI;
rc = ssi_buffer_mgr_prepare_aead_data_mlli(drvdata, req, sg_data,
&src_last_bytes, &dst_last_bytes, is_last_table);
return rc;
}
-static void ssi_buffer_mgr_update_aead_mlli_nents( struct ssi_drvdata *drvdata,
+static void ssi_buffer_mgr_update_aead_mlli_nents(struct ssi_drvdata *drvdata,
struct aead_request *req)
{
struct aead_req_ctx *areq_ctx = aead_request_ctx(req);
- uint32_t curr_mlli_size = 0;
-
+ u32 curr_mlli_size = 0;
+
if (areq_ctx->assoc_buff_type == SSI_DMA_BUF_MLLI) {
areq_ctx->assoc.sram_addr = drvdata->mlli_sram_addr;
- curr_mlli_size = areq_ctx->assoc.mlli_nents *
+ curr_mlli_size = areq_ctx->assoc.mlli_nents *
LLI_ENTRY_BYTE_SIZE;
}
areq_ctx->src.sram_addr = drvdata->mlli_sram_addr +
curr_mlli_size;
areq_ctx->dst.sram_addr = areq_ctx->src.sram_addr;
- if (areq_ctx->is_single_pass == false)
- areq_ctx->assoc.mlli_nents +=
+ if (!areq_ctx->is_single_pass)
+ areq_ctx->assoc.mlli_nents +=
areq_ctx->src.mlli_nents;
} else {
- if (areq_ctx->gen_ctx.op_type ==
+ if (areq_ctx->gen_ctx.op_type ==
DRV_CRYPTO_DIRECTION_DECRYPT) {
- areq_ctx->src.sram_addr =
+ areq_ctx->src.sram_addr =
drvdata->mlli_sram_addr +
curr_mlli_size;
- areq_ctx->dst.sram_addr =
- areq_ctx->src.sram_addr +
- areq_ctx->src.mlli_nents *
+ areq_ctx->dst.sram_addr =
+ areq_ctx->src.sram_addr +
+ areq_ctx->src.mlli_nents *
LLI_ENTRY_BYTE_SIZE;
- if (areq_ctx->is_single_pass == false)
- areq_ctx->assoc.mlli_nents +=
+ if (!areq_ctx->is_single_pass)
+ areq_ctx->assoc.mlli_nents +=
areq_ctx->src.mlli_nents;
} else {
- areq_ctx->dst.sram_addr =
+ areq_ctx->dst.sram_addr =
drvdata->mlli_sram_addr +
curr_mlli_size;
- areq_ctx->src.sram_addr =
+ areq_ctx->src.sram_addr =
areq_ctx->dst.sram_addr +
- areq_ctx->dst.mlli_nents *
+ areq_ctx->dst.mlli_nents *
LLI_ENTRY_BYTE_SIZE;
- if (areq_ctx->is_single_pass == false)
- areq_ctx->assoc.mlli_nents +=
+ if (!areq_ctx->is_single_pass)
+ areq_ctx->assoc.mlli_nents +=
areq_ctx->dst.mlli_nents;
}
}
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
bool is_gcm4543 = areq_ctx->is_gcm4543;
- uint32_t mapped_nents = 0;
- uint32_t dummy = 0; /*used for the assoc data fragments */
- uint32_t size_to_map = 0;
+ u32 mapped_nents = 0;
+ u32 dummy = 0; /*used for the assoc data fragments */
+ u32 size_to_map = 0;
mlli_params->curr_pool = NULL;
sg_data.num_of_buffers = 0;
-#if DX_HAS_ACP
- if ((areq_ctx->gen_ctx.op_type == DRV_CRYPTO_DIRECTION_DECRYPT) &&
+ if (drvdata->coherent &&
+ (areq_ctx->gen_ctx.op_type == DRV_CRYPTO_DIRECTION_DECRYPT) &&
likely(req->src == req->dst))
{
- uint32_t size_to_skip = req->assoclen;
- if (is_gcm4543) {
+ u32 size_to_skip = req->assoclen;
+
+ if (is_gcm4543)
size_to_skip += crypto_aead_ivsize(tfm);
- }
+
/* copy mac to a temporary location to deal with possible
- data memory overriding that caused by cache coherence problem. */
+ * data memory overriding that caused by cache coherence problem.
+ */
ssi_buffer_mgr_copy_scatterlist_portion(
areq_ctx->backup_mac, req->src,
- size_to_skip+ req->cryptlen - areq_ctx->req_authsize,
- size_to_skip+ req->cryptlen, SSI_SG_TO_BUF);
+ size_to_skip + req->cryptlen - areq_ctx->req_authsize,
+ size_to_skip + req->cryptlen, SSI_SG_TO_BUF);
}
-#endif
/* cacluate the size for cipher remove ICV in decrypt*/
- areq_ctx->cryptlen = (areq_ctx->gen_ctx.op_type ==
- DRV_CRYPTO_DIRECTION_ENCRYPT) ?
+ areq_ctx->cryptlen = (areq_ctx->gen_ctx.op_type ==
+ DRV_CRYPTO_DIRECTION_ENCRYPT) ?
req->cryptlen :
(req->cryptlen - authsize);
rc = -ENOMEM;
goto aead_map_failure;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(areq_ctx->mac_buf_dma_addr, MAX_MAC_SIZE);
if (areq_ctx->ccm_hdr_size != ccm_header_size_null) {
areq_ctx->ccm_iv0_dma_addr = dma_map_single(dev,
rc = -ENOMEM;
goto aead_map_failure;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(areq_ctx->ccm_iv0_dma_addr,
- AES_BLOCK_SIZE);
if (ssi_aead_handle_config_buf(dev, areq_ctx,
areq_ctx->ccm_config, &sg_data, req->assoclen) != 0) {
rc = -ENOMEM;
rc = -ENOMEM;
goto aead_map_failure;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(areq_ctx->hkey_dma_addr, AES_BLOCK_SIZE);
areq_ctx->gcm_block_len_dma_addr = dma_map_single(dev,
&areq_ctx->gcm_len_block, AES_BLOCK_SIZE, DMA_TO_DEVICE);
rc = -ENOMEM;
goto aead_map_failure;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(areq_ctx->gcm_block_len_dma_addr, AES_BLOCK_SIZE);
areq_ctx->gcm_iv_inc1_dma_addr = dma_map_single(dev,
areq_ctx->gcm_iv_inc1,
rc = -ENOMEM;
goto aead_map_failure;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(areq_ctx->gcm_iv_inc1_dma_addr,
- AES_BLOCK_SIZE);
areq_ctx->gcm_iv_inc2_dma_addr = dma_map_single(dev,
areq_ctx->gcm_iv_inc2,
rc = -ENOMEM;
goto aead_map_failure;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(areq_ctx->gcm_iv_inc2_dma_addr,
- AES_BLOCK_SIZE);
}
#endif /*SSI_CC_HAS_AES_GCM*/
size_to_map = req->cryptlen + req->assoclen;
- if (areq_ctx->gen_ctx.op_type == DRV_CRYPTO_DIRECTION_ENCRYPT) {
+ if (areq_ctx->gen_ctx.op_type == DRV_CRYPTO_DIRECTION_ENCRYPT)
size_to_map += authsize;
- }
+
if (is_gcm4543)
size_to_map += crypto_aead_ivsize(tfm);
rc = ssi_buffer_mgr_map_scatterlist(dev, req->src,
size_to_map, DMA_BIDIRECTIONAL, &(areq_ctx->src.nents),
- LLI_MAX_NUM_OF_ASSOC_DATA_ENTRIES+LLI_MAX_NUM_OF_DATA_ENTRIES, &dummy, &mapped_nents);
+ LLI_MAX_NUM_OF_ASSOC_DATA_ENTRIES + LLI_MAX_NUM_OF_DATA_ENTRIES, &dummy, &mapped_nents);
if (unlikely(rc != 0)) {
rc = -ENOMEM;
- goto aead_map_failure;
+ goto aead_map_failure;
}
- if (likely(areq_ctx->is_single_pass == true)) {
+ if (likely(areq_ctx->is_single_pass)) {
/*
- * Create MLLI table for:
- * (1) Assoc. data
- * (2) Src/Dst SGLs
- * Note: IV is contg. buffer (not an SGL)
- */
+ * Create MLLI table for:
+ * (1) Assoc. data
+ * (2) Src/Dst SGLs
+ * Note: IV is contg. buffer (not an SGL)
+ */
rc = ssi_buffer_mgr_aead_chain_assoc(drvdata, req, &sg_data, true, false);
if (unlikely(rc != 0))
goto aead_map_failure;
goto aead_map_failure;
} else { /* DOUBLE-PASS flow */
/*
- * Prepare MLLI table(s) in this order:
- *
- * If ENCRYPT/DECRYPT (inplace):
- * (1) MLLI table for assoc
- * (2) IV entry (chained right after end of assoc)
- * (3) MLLI for src/dst (inplace operation)
- *
- * If ENCRYPT (non-inplace)
- * (1) MLLI table for assoc
- * (2) IV entry (chained right after end of assoc)
- * (3) MLLI for dst
- * (4) MLLI for src
- *
- * If DECRYPT (non-inplace)
- * (1) MLLI table for assoc
- * (2) IV entry (chained right after end of assoc)
- * (3) MLLI for src
- * (4) MLLI for dst
- */
+ * Prepare MLLI table(s) in this order:
+ *
+ * If ENCRYPT/DECRYPT (inplace):
+ * (1) MLLI table for assoc
+ * (2) IV entry (chained right after end of assoc)
+ * (3) MLLI for src/dst (inplace operation)
+ *
+ * If ENCRYPT (non-inplace)
+ * (1) MLLI table for assoc
+ * (2) IV entry (chained right after end of assoc)
+ * (3) MLLI for dst
+ * (4) MLLI for src
+ *
+ * If DECRYPT (non-inplace)
+ * (1) MLLI table for assoc
+ * (2) IV entry (chained right after end of assoc)
+ * (3) MLLI for src
+ * (4) MLLI for dst
+ */
rc = ssi_buffer_mgr_aead_chain_assoc(drvdata, req, &sg_data, false, true);
if (unlikely(rc != 0))
goto aead_map_failure;
if (unlikely(
(areq_ctx->assoc_buff_type == SSI_DMA_BUF_MLLI) ||
(areq_ctx->data_buff_type == SSI_DMA_BUF_MLLI))) {
-
mlli_params->curr_pool = buff_mgr->mlli_buffs_pool;
rc = ssi_buffer_mgr_generate_mlli(dev, &sg_data, mlli_params);
- if (unlikely(rc != 0)) {
+ if (unlikely(rc != 0))
goto aead_map_failure;
- }
ssi_buffer_mgr_update_aead_mlli_nents(drvdata, req);
- SSI_LOG_DEBUG("assoc params mn %d\n",areq_ctx->assoc.mlli_nents);
- SSI_LOG_DEBUG("src params mn %d\n",areq_ctx->src.mlli_nents);
- SSI_LOG_DEBUG("dst params mn %d\n",areq_ctx->dst.mlli_nents);
+ SSI_LOG_DEBUG("assoc params mn %d\n", areq_ctx->assoc.mlli_nents);
+ SSI_LOG_DEBUG("src params mn %d\n", areq_ctx->src.mlli_nents);
+ SSI_LOG_DEBUG("dst params mn %d\n", areq_ctx->dst.mlli_nents);
}
return 0;
{
struct ahash_req_ctx *areq_ctx = (struct ahash_req_ctx *)ctx;
struct device *dev = &drvdata->plat_dev->dev;
- uint8_t* curr_buff = areq_ctx->buff_index ? areq_ctx->buff1 :
+ u8 *curr_buff = areq_ctx->buff_index ? areq_ctx->buff1 :
areq_ctx->buff0;
- uint32_t *curr_buff_cnt = areq_ctx->buff_index ? &areq_ctx->buff1_cnt :
+ u32 *curr_buff_cnt = areq_ctx->buff_index ? &areq_ctx->buff1_cnt :
&areq_ctx->buff0_cnt;
- struct mlli_params *mlli_params = &areq_ctx->mlli_params;
+ struct mlli_params *mlli_params = &areq_ctx->mlli_params;
struct buffer_array sg_data;
struct buff_mgr_handle *buff_mgr = drvdata->buff_mgr_handle;
- uint32_t dummy = 0;
- uint32_t mapped_nents = 0;
+ u32 dummy = 0;
+ u32 mapped_nents = 0;
SSI_LOG_DEBUG(" final params : curr_buff=%pK "
"curr_buff_cnt=0x%X nbytes = 0x%X "
/* nothing to do */
return 0;
}
-
+
/*TODO: copy data in case that buffer is enough for operation */
/* map the previous buffer */
- if (*curr_buff_cnt != 0 ) {
+ if (*curr_buff_cnt != 0) {
if (ssi_ahash_handle_curr_buf(dev, areq_ctx, curr_buff,
*curr_buff_cnt, &sg_data) != 0) {
return -ENOMEM;
}
if (src && (nbytes > 0) && do_update) {
- if ( unlikely( ssi_buffer_mgr_map_scatterlist( dev,src,
+ if (unlikely(ssi_buffer_mgr_map_scatterlist(dev, src,
nbytes,
DMA_TO_DEVICE,
&areq_ctx->in_nents,
&dummy, &mapped_nents))){
goto unmap_curr_buff;
}
- if ( src && (mapped_nents == 1)
- && (areq_ctx->data_dma_buf_type == SSI_DMA_BUF_NULL) ) {
- memcpy(areq_ctx->buff_sg,src,
+ if (src && (mapped_nents == 1)
+ && (areq_ctx->data_dma_buf_type == SSI_DMA_BUF_NULL)) {
+ memcpy(areq_ctx->buff_sg, src,
sizeof(struct scatterlist));
areq_ctx->buff_sg->length = nbytes;
areq_ctx->curr_sg = areq_ctx->buff_sg;
} else {
areq_ctx->data_dma_buf_type = SSI_DMA_BUF_MLLI;
}
-
}
/*build mlli */
}
}
/* change the buffer index for the unmap function */
- areq_ctx->buff_index = (areq_ctx->buff_index^1);
+ areq_ctx->buff_index = (areq_ctx->buff_index ^ 1);
SSI_LOG_DEBUG("areq_ctx->data_dma_buf_type = %s\n",
GET_DMA_BUFFER_TYPE(areq_ctx->data_dma_buf_type));
return 0;
dma_unmap_sg(dev, src, areq_ctx->in_nents, DMA_TO_DEVICE);
unmap_curr_buff:
- if (*curr_buff_cnt != 0 ) {
+ if (*curr_buff_cnt != 0)
dma_unmap_sg(dev, areq_ctx->buff_sg, 1, DMA_TO_DEVICE);
- }
+
return -ENOMEM;
}
{
struct ahash_req_ctx *areq_ctx = (struct ahash_req_ctx *)ctx;
struct device *dev = &drvdata->plat_dev->dev;
- uint8_t* curr_buff = areq_ctx->buff_index ? areq_ctx->buff1 :
+ u8 *curr_buff = areq_ctx->buff_index ? areq_ctx->buff1 :
areq_ctx->buff0;
- uint32_t *curr_buff_cnt = areq_ctx->buff_index ? &areq_ctx->buff1_cnt :
+ u32 *curr_buff_cnt = areq_ctx->buff_index ? &areq_ctx->buff1_cnt :
&areq_ctx->buff0_cnt;
- uint8_t* next_buff = areq_ctx->buff_index ? areq_ctx->buff0 :
+ u8 *next_buff = areq_ctx->buff_index ? areq_ctx->buff0 :
areq_ctx->buff1;
- uint32_t *next_buff_cnt = areq_ctx->buff_index ? &areq_ctx->buff0_cnt :
+ u32 *next_buff_cnt = areq_ctx->buff_index ? &areq_ctx->buff0_cnt :
&areq_ctx->buff1_cnt;
- struct mlli_params *mlli_params = &areq_ctx->mlli_params;
+ struct mlli_params *mlli_params = &areq_ctx->mlli_params;
unsigned int update_data_len;
- uint32_t total_in_len = nbytes + *curr_buff_cnt;
+ u32 total_in_len = nbytes + *curr_buff_cnt;
struct buffer_array sg_data;
struct buff_mgr_handle *buff_mgr = drvdata->buff_mgr_handle;
unsigned int swap_index = 0;
- uint32_t dummy = 0;
- uint32_t mapped_nents = 0;
-
+ u32 dummy = 0;
+ u32 mapped_nents = 0;
+
SSI_LOG_DEBUG(" update params : curr_buff=%pK "
"curr_buff_cnt=0x%X nbytes=0x%X "
- "src=%pK curr_index=%u \n",
+ "src=%pK curr_index=%u\n",
curr_buff, *curr_buff_cnt, nbytes,
src, areq_ctx->buff_index);
/* Init the type of the dma buffer */
"*curr_buff_cnt=0x%X copy_to=%pK\n",
curr_buff, *curr_buff_cnt,
&curr_buff[*curr_buff_cnt]);
- areq_ctx->in_nents =
+ areq_ctx->in_nents =
ssi_buffer_mgr_get_sgl_nents(src,
nbytes,
&dummy, NULL);
sg_copy_to_buffer(src, areq_ctx->in_nents,
- &curr_buff[*curr_buff_cnt], nbytes);
+ &curr_buff[*curr_buff_cnt], nbytes);
*curr_buff_cnt += nbytes;
return 1;
}
/* Copy the new residue to next buffer */
if (*next_buff_cnt != 0) {
SSI_LOG_DEBUG(" handle residue: next buff %pK skip data %u"
- " residue %u \n", next_buff,
+ " residue %u\n", next_buff,
(update_data_len - *curr_buff_cnt),
*next_buff_cnt);
ssi_buffer_mgr_copy_scatterlist_portion(next_buff, src,
- (update_data_len -*curr_buff_cnt),
- nbytes,SSI_SG_TO_BUF);
+ (update_data_len - *curr_buff_cnt),
+ nbytes, SSI_SG_TO_BUF);
/* change the buffer index for next operation */
swap_index = 1;
}
/* change the buffer index for next operation */
swap_index = 1;
}
-
- if ( update_data_len > *curr_buff_cnt ) {
- if ( unlikely( ssi_buffer_mgr_map_scatterlist( dev,src,
- (update_data_len -*curr_buff_cnt),
+
+ if (update_data_len > *curr_buff_cnt) {
+ if (unlikely(ssi_buffer_mgr_map_scatterlist(dev, src,
+ (update_data_len - *curr_buff_cnt),
DMA_TO_DEVICE,
&areq_ctx->in_nents,
LLI_MAX_NUM_OF_DATA_ENTRIES,
&dummy, &mapped_nents))){
goto unmap_curr_buff;
}
- if ( (mapped_nents == 1)
- && (areq_ctx->data_dma_buf_type == SSI_DMA_BUF_NULL) ) {
+ if ((mapped_nents == 1)
+ && (areq_ctx->data_dma_buf_type == SSI_DMA_BUF_NULL)) {
/* only one entry in the SG and no previous data */
- memcpy(areq_ctx->buff_sg,src,
+ memcpy(areq_ctx->buff_sg, src,
sizeof(struct scatterlist));
areq_ctx->buff_sg->length = update_data_len;
areq_ctx->data_dma_buf_type = SSI_DMA_BUF_DLLI;
mlli_params) != 0)) {
goto fail_unmap_din;
}
-
}
- areq_ctx->buff_index = (areq_ctx->buff_index^swap_index);
+ areq_ctx->buff_index = (areq_ctx->buff_index ^ swap_index);
return 0;
dma_unmap_sg(dev, src, areq_ctx->in_nents, DMA_TO_DEVICE);
unmap_curr_buff:
- if (*curr_buff_cnt != 0 ) {
+ if (*curr_buff_cnt != 0)
dma_unmap_sg(dev, areq_ctx->buff_sg, 1, DMA_TO_DEVICE);
- }
+
return -ENOMEM;
}
struct device *dev, void *ctx, struct scatterlist *src, bool do_revert)
{
struct ahash_req_ctx *areq_ctx = (struct ahash_req_ctx *)ctx;
- uint32_t *prev_len = areq_ctx->buff_index ? &areq_ctx->buff0_cnt :
+ u32 *prev_len = areq_ctx->buff_index ? &areq_ctx->buff0_cnt :
&areq_ctx->buff1_cnt;
- /*In case a pool was set, a table was
- allocated and should be released */
- if (areq_ctx->mlli_params.curr_pool != NULL) {
- SSI_LOG_DEBUG("free MLLI buffer: dma=0x%llX virt=%pK\n",
+ /*In case a pool was set, a table was
+ *allocated and should be released
+ */
+ if (areq_ctx->mlli_params.curr_pool) {
+ SSI_LOG_DEBUG("free MLLI buffer: dma=0x%llX virt=%pK\n",
(unsigned long long)areq_ctx->mlli_params.mlli_dma_addr,
areq_ctx->mlli_params.mlli_virt_addr);
- SSI_RESTORE_DMA_ADDR_TO_48BIT(areq_ctx->mlli_params.mlli_dma_addr);
dma_pool_free(areq_ctx->mlli_params.curr_pool,
areq_ctx->mlli_params.mlli_virt_addr,
areq_ctx->mlli_params.mlli_dma_addr);
}
-
+
if ((src) && likely(areq_ctx->in_nents != 0)) {
SSI_LOG_DEBUG("Unmapped sg src: virt=%pK dma=0x%llX len=0x%X\n",
sg_virt(src),
- (unsigned long long)sg_dma_address(src),
+ (unsigned long long)sg_dma_address(src),
sg_dma_len(src));
- SSI_RESTORE_DMA_ADDR_TO_48BIT(sg_dma_address(src));
- dma_unmap_sg(dev, src,
+ dma_unmap_sg(dev, src,
areq_ctx->in_nents, DMA_TO_DEVICE);
}
if (*prev_len != 0) {
SSI_LOG_DEBUG("Unmapped buffer: areq_ctx->buff_sg=%pK"
- "dma=0x%llX len 0x%X\n",
+ " dma=0x%llX len 0x%X\n",
sg_virt(areq_ctx->buff_sg),
- (unsigned long long)sg_dma_address(areq_ctx->buff_sg),
+ (unsigned long long)sg_dma_address(areq_ctx->buff_sg),
sg_dma_len(areq_ctx->buff_sg));
dma_unmap_sg(dev, areq_ctx->buff_sg, 1, DMA_TO_DEVICE);
if (!do_revert) {
buff_mgr_handle = (struct buff_mgr_handle *)
kmalloc(sizeof(struct buff_mgr_handle), GFP_KERNEL);
- if (buff_mgr_handle == NULL)
+ if (!buff_mgr_handle)
return -ENOMEM;
drvdata->buff_mgr_handle = buff_mgr_handle;
buff_mgr_handle->mlli_buffs_pool = dma_pool_create(
"dx_single_mlli_tables", dev,
- MAX_NUM_OF_TOTAL_MLLI_ENTRIES *
+ MAX_NUM_OF_TOTAL_MLLI_ENTRIES *
LLI_ENTRY_BYTE_SIZE,
MLLI_TABLE_MIN_ALIGNMENT, 0);
- if (unlikely(buff_mgr_handle->mlli_buffs_pool == NULL))
+ if (unlikely(!buff_mgr_handle->mlli_buffs_pool))
goto error;
return 0;
{
struct buff_mgr_handle *buff_mgr_handle = drvdata->buff_mgr_handle;
- if (buff_mgr_handle != NULL) {
+ if (buff_mgr_handle) {
dma_pool_destroy(buff_mgr_handle->mlli_buffs_pool);
kfree(drvdata->buff_mgr_handle);
drvdata->buff_mgr_handle = NULL;
-
}
return 0;
}
-
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
/* \file buffer_mgr.h
- Buffer Manager
+ * Buffer Manager
*/
#ifndef __SSI_BUFFER_MGR_H__
#include "ssi_config.h"
#include "ssi_driver.h"
-
enum ssi_req_dma_buf_type {
SSI_DMA_BUF_NULL = 0,
SSI_DMA_BUF_DLLI,
struct mlli_params {
struct dma_pool *curr_pool;
- uint8_t *mlli_virt_addr;
+ u8 *mlli_virt_addr;
dma_addr_t mlli_dma_addr;
- uint32_t mlli_len;
+ u32 mlli_len;
};
int ssi_buffer_mgr_init(struct ssi_drvdata *drvdata);
struct scatterlist *dst);
void ssi_buffer_mgr_unmap_blkcipher_request(
- struct device *dev,
+ struct device *dev,
void *ctx,
unsigned int ivsize,
struct scatterlist *src,
void ssi_buffer_mgr_unmap_hash_request(struct device *dev, void *ctx, struct scatterlist *src, bool do_revert);
-void ssi_buffer_mgr_copy_scatterlist_portion(u8 *dest, struct scatterlist *sg, uint32_t to_skip, uint32_t end, enum ssi_sg_cpy_direct direct);
-
-void ssi_buffer_mgr_zero_sgl(struct scatterlist *sgl, uint32_t data_len);
-
-
-#ifdef CC_DMA_48BIT_SIM
-dma_addr_t ssi_buff_mgr_update_dma_addr(dma_addr_t orig_addr, uint32_t data_len);
-dma_addr_t ssi_buff_mgr_restore_dma_addr(dma_addr_t orig_addr);
-
-#define SSI_UPDATE_DMA_ADDR_TO_48BIT(addr,size) addr = \
- ssi_buff_mgr_update_dma_addr(addr,size)
-#define SSI_RESTORE_DMA_ADDR_TO_48BIT(addr) addr = \
- ssi_buff_mgr_restore_dma_addr(addr)
-#else
-
-#define SSI_UPDATE_DMA_ADDR_TO_48BIT(addr,size) addr = addr
-#define SSI_RESTORE_DMA_ADDR_TO_48BIT(addr) addr = addr
+void ssi_buffer_mgr_copy_scatterlist_portion(u8 *dest, struct scatterlist *sg, u32 to_skip, u32 end, enum ssi_sg_cpy_direct direct);
-#endif
+void ssi_buffer_mgr_zero_sgl(struct scatterlist *sgl, u32 data_len);
#endif /*__BUFFER_MGR_H__*/
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
#define MAX_ABLKCIPHER_SEQ_LEN 6
#define template_ablkcipher template_u.ablkcipher
-#define template_sblkcipher template_u.blkcipher
#define SSI_MIN_AES_XTS_SIZE 0x10
#define SSI_MAX_AES_XTS_SIZE 0x2000
};
struct cc_user_key_info {
- uint8_t *key;
+ u8 *key;
dma_addr_t key_dma_addr;
};
+
struct cc_hw_key_info {
- enum HwCryptoKey key1_slot;
- enum HwCryptoKey key2_slot;
+ enum cc_hw_crypto_key key1_slot;
+ enum cc_hw_crypto_key key2_slot;
};
struct ssi_ablkcipher_ctx {
static void ssi_ablkcipher_complete(struct device *dev, void *ssi_req, void __iomem *cc_base);
-
-static int validate_keys_sizes(struct ssi_ablkcipher_ctx *ctx_p, uint32_t size) {
- switch (ctx_p->flow_mode){
+static int validate_keys_sizes(struct ssi_ablkcipher_ctx *ctx_p, u32 size) {
+ switch (ctx_p->flow_mode) {
case S_DIN_to_AES:
- switch (size){
+ switch (size) {
case CC_AES_128_BIT_KEY_SIZE:
case CC_AES_192_BIT_KEY_SIZE:
if (likely((ctx_p->cipher_mode != DRV_CIPHER_XTS) &&
break;
case CC_AES_256_BIT_KEY_SIZE:
return 0;
- case (CC_AES_192_BIT_KEY_SIZE*2):
- case (CC_AES_256_BIT_KEY_SIZE*2):
+ case (CC_AES_192_BIT_KEY_SIZE * 2):
+ case (CC_AES_256_BIT_KEY_SIZE * 2):
if (likely((ctx_p->cipher_mode == DRV_CIPHER_XTS) ||
(ctx_p->cipher_mode == DRV_CIPHER_ESSIV) ||
(ctx_p->cipher_mode == DRV_CIPHER_BITLOCKER)))
#endif
default:
break;
-
}
return -EINVAL;
}
-
static int validate_data_size(struct ssi_ablkcipher_ctx *ctx_p, unsigned int size) {
- switch (ctx_p->flow_mode){
+ switch (ctx_p->flow_mode) {
case S_DIN_to_AES:
- switch (ctx_p->cipher_mode){
+ switch (ctx_p->cipher_mode) {
case DRV_CIPHER_XTS:
if ((size >= SSI_MIN_AES_XTS_SIZE) &&
- (size <= SSI_MAX_AES_XTS_SIZE) &&
+ (size <= SSI_MAX_AES_XTS_SIZE) &&
IS_ALIGNED(size, AES_BLOCK_SIZE))
return 0;
break;
#endif /*SSI_CC_HAS_MULTI2*/
default:
break;
-
}
return -EINVAL;
}
{
struct ssi_crypto_alg *ssi_alg = container_of(tfm->__crt_alg, struct ssi_crypto_alg, crypto_alg);
- if ((ssi_alg->crypto_alg.cra_flags & CRYPTO_ALG_TYPE_MASK) == CRYPTO_ALG_TYPE_ABLKCIPHER) {
+ if ((ssi_alg->crypto_alg.cra_flags & CRYPTO_ALG_TYPE_MASK) == CRYPTO_ALG_TYPE_ABLKCIPHER)
return ssi_alg->crypto_alg.cra_ablkcipher.max_keysize;
- }
- if ((ssi_alg->crypto_alg.cra_flags & CRYPTO_ALG_TYPE_MASK) == CRYPTO_ALG_TYPE_BLKCIPHER) {
+ if ((ssi_alg->crypto_alg.cra_flags & CRYPTO_ALG_TYPE_MASK) == CRYPTO_ALG_TYPE_BLKCIPHER)
return ssi_alg->crypto_alg.cra_blkcipher.max_keysize;
- }
return 0;
}
int rc = 0;
unsigned int max_key_buf_size = get_max_keysize(tfm);
- SSI_LOG_DEBUG("Initializing context @%p for %s\n", ctx_p,
+ SSI_LOG_DEBUG("Initializing context @%p for %s\n", ctx_p,
crypto_tfm_alg_name(tfm));
CHECK_AND_RETURN_UPON_FIPS_ERROR();
dev = &ctx_p->drvdata->plat_dev->dev;
/* Allocate key buffer, cache line aligned */
- ctx_p->user.key = kmalloc(max_key_buf_size, GFP_KERNEL|GFP_DMA);
+ ctx_p->user.key = kmalloc(max_key_buf_size, GFP_KERNEL | GFP_DMA);
if (!ctx_p->user.key) {
SSI_LOG_ERR("Allocating key buffer in context failed\n");
rc = -ENOMEM;
max_key_buf_size, ctx_p->user.key);
return -ENOMEM;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(ctx_p->user.key_dma_addr, max_key_buf_size);
SSI_LOG_DEBUG("Mapped key %u B at va=%pK to dma=0x%llX\n",
max_key_buf_size, ctx_p->user.key,
(unsigned long long)ctx_p->user.key_dma_addr);
}
/* Unmap key buffer */
- SSI_RESTORE_DMA_ADDR_TO_48BIT(ctx_p->user.key_dma_addr);
dma_unmap_single(dev, ctx_p->user.key_dma_addr, max_key_buf_size,
DMA_TO_DEVICE);
- SSI_LOG_DEBUG("Unmapped key buffer key_dma_addr=0x%llX\n",
+ SSI_LOG_DEBUG("Unmapped key buffer key_dma_addr=0x%llX\n",
(unsigned long long)ctx_p->user.key_dma_addr);
/* Free key buffer in context */
SSI_LOG_DEBUG("Free key buffer in context. key=@%p\n", ctx_p->user.key);
}
+struct tdes_keys {
+ u8 key1[DES_KEY_SIZE];
+ u8 key2[DES_KEY_SIZE];
+ u8 key3[DES_KEY_SIZE];
+};
-typedef struct tdes_keys{
- u8 key1[DES_KEY_SIZE];
- u8 key2[DES_KEY_SIZE];
- u8 key3[DES_KEY_SIZE];
-}tdes_keys_t;
-
-static const u8 zero_buff[] = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
- 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
- 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
- 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0};
+static const u8 zero_buff[] = { 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
+ 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
+ 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
+ 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0};
/* The function verifies that tdes keys are not weak.*/
static int ssi_fips_verify_3des_keys(const u8 *key, unsigned int keylen)
{
#ifdef CCREE_FIPS_SUPPORT
- tdes_keys_t *tdes_key = (tdes_keys_t*)key;
+ struct tdes_keys *tdes_key = (struct tdes_keys *)key;
/* verify key1 != key2 and key3 != key2*/
- if (unlikely( (memcmp((u8*)tdes_key->key1, (u8*)tdes_key->key2, sizeof(tdes_key->key1)) == 0) ||
- (memcmp((u8*)tdes_key->key3, (u8*)tdes_key->key2, sizeof(tdes_key->key3)) == 0) )) {
- return -ENOEXEC;
- }
+ if (unlikely((memcmp((u8 *)tdes_key->key1, (u8 *)tdes_key->key2, sizeof(tdes_key->key1)) == 0) ||
+ (memcmp((u8 *)tdes_key->key3, (u8 *)tdes_key->key2, sizeof(tdes_key->key3)) == 0))) {
+ return -ENOEXEC;
+ }
#endif /* CCREE_FIPS_SUPPORT */
- return 0;
+ return 0;
}
/* The function verifies that xts keys are not weak.*/
static int ssi_fips_verify_xts_keys(const u8 *key, unsigned int keylen)
{
#ifdef CCREE_FIPS_SUPPORT
- /* Weak key is define as key that its first half (128/256 lsb) equals its second half (128/256 msb) */
- int singleKeySize = keylen >> 1;
+ /* Weak key is define as key that its first half (128/256 lsb) equals its second half (128/256 msb) */
+ int singleKeySize = keylen >> 1;
- if (unlikely(memcmp(key, &key[singleKeySize], singleKeySize) == 0)) {
+ if (unlikely(memcmp(key, &key[singleKeySize], singleKeySize) == 0))
return -ENOEXEC;
- }
#endif /* CCREE_FIPS_SUPPORT */
- return 0;
+ return 0;
}
-static enum HwCryptoKey hw_key_to_cc_hw_key(int slot_num)
+static enum cc_hw_crypto_key hw_key_to_cc_hw_key(int slot_num)
{
switch (slot_num) {
case 0:
return END_OF_KEYS;
}
-static int ssi_blkcipher_setkey(struct crypto_tfm *tfm,
- const u8 *key,
+static int ssi_blkcipher_setkey(struct crypto_tfm *tfm,
+ const u8 *key,
unsigned int keylen)
{
struct ssi_ablkcipher_ctx *ctx_p = crypto_tfm_ctx(tfm);
struct device *dev = &ctx_p->drvdata->plat_dev->dev;
u32 tmp[DES_EXPKEY_WORDS];
unsigned int max_key_buf_size = get_max_keysize(tfm);
- DECL_CYCLE_COUNT_RESOURCES;
SSI_LOG_DEBUG("Setting key in context @%p for %s. keylen=%u\n",
ctx_p, crypto_tfm_alg_name(tfm), keylen);
- dump_byte_array("key", (uint8_t *)key, keylen);
+ dump_byte_array("key", (u8 *)key, keylen);
CHECK_AND_RETURN_UPON_FIPS_ERROR();
SSI_LOG_DEBUG("ssi_blkcipher_setkey: after FIPS check");
-
+
/* STAT_PHASE_0: Init and sanity checks */
- START_CYCLE_COUNT();
#if SSI_CC_HAS_MULTI2
/*last byte of key buffer is round number and should not be a part of key size*/
- if (ctx_p->flow_mode == S_DIN_to_MULTI2) {
- keylen -=1;
- }
+ if (ctx_p->flow_mode == S_DIN_to_MULTI2)
+ keylen -= 1;
#endif /*SSI_CC_HAS_MULTI2*/
- if (unlikely(validate_keys_sizes(ctx_p,keylen) != 0)) {
+ if (unlikely(validate_keys_sizes(ctx_p, keylen) != 0)) {
SSI_LOG_ERR("Unsupported key size %d.\n", keylen);
crypto_tfm_set_flags(tfm, CRYPTO_TFM_RES_BAD_KEY_LEN);
return -EINVAL;
if (ssi_is_hw_key(tfm)) {
/* setting HW key slots */
- struct arm_hw_key_info *hki = (struct arm_hw_key_info*)key;
+ struct arm_hw_key_info *hki = (struct arm_hw_key_info *)key;
if (unlikely(ctx_p->flow_mode != S_DIN_to_AES)) {
SSI_LOG_ERR("HW key not supported for non-AES flows\n");
}
ctx_p->keylen = keylen;
- END_CYCLE_COUNT(STAT_OP_TYPE_SETKEY, STAT_PHASE_0);
SSI_LOG_DEBUG("ssi_blkcipher_setkey: ssi_is_hw_key ret 0");
return 0;
return -EINVAL;
}
}
- if ((ctx_p->cipher_mode == DRV_CIPHER_XTS) &&
+ if ((ctx_p->cipher_mode == DRV_CIPHER_XTS) &&
ssi_fips_verify_xts_keys(key, keylen) != 0) {
SSI_LOG_DEBUG("ssi_blkcipher_setkey: weak XTS key");
return -EINVAL;
}
- if ((ctx_p->flow_mode == S_DIN_to_DES) &&
- (keylen == DES3_EDE_KEY_SIZE) &&
+ if ((ctx_p->flow_mode == S_DIN_to_DES) &&
+ (keylen == DES3_EDE_KEY_SIZE) &&
ssi_fips_verify_3des_keys(key, keylen) != 0) {
SSI_LOG_DEBUG("ssi_blkcipher_setkey: weak 3DES key");
return -EINVAL;
}
-
- END_CYCLE_COUNT(STAT_OP_TYPE_SETKEY, STAT_PHASE_0);
-
/* STAT_PHASE_1: Copy key to ctx */
- START_CYCLE_COUNT();
- SSI_RESTORE_DMA_ADDR_TO_48BIT(ctx_p->user.key_dma_addr);
- dma_sync_single_for_cpu(dev, ctx_p->user.key_dma_addr,
+ dma_sync_single_for_cpu(dev, ctx_p->user.key_dma_addr,
max_key_buf_size, DMA_TO_DEVICE);
-#if SSI_CC_HAS_MULTI2
+
if (ctx_p->flow_mode == S_DIN_to_MULTI2) {
+#if SSI_CC_HAS_MULTI2
memcpy(ctx_p->user.key, key, CC_MULTI2_SYSTEM_N_DATA_KEY_SIZE);
ctx_p->key_round_number = key[CC_MULTI2_SYSTEM_N_DATA_KEY_SIZE];
if (ctx_p->key_round_number < CC_MULTI2_MIN_NUM_ROUNDS ||
crypto_tfm_set_flags(tfm, CRYPTO_TFM_RES_BAD_KEY_LEN);
SSI_LOG_DEBUG("ssi_blkcipher_setkey: SSI_CC_HAS_MULTI2 einval");
return -EINVAL;
- }
- } else
#endif /*SSI_CC_HAS_MULTI2*/
- {
+ } else {
memcpy(ctx_p->user.key, key, keylen);
if (keylen == 24)
memset(ctx_p->user.key + 24, 0, CC_AES_KEY_SIZE_MAX - 24);
int key_len = keylen >> 1;
int err;
SHASH_DESC_ON_STACK(desc, ctx_p->shash_tfm);
+
desc->tfm = ctx_p->shash_tfm;
err = crypto_shash_digest(desc, ctx_p->user.key, key_len, ctx_p->user.key + key_len);
}
}
}
- dma_sync_single_for_device(dev, ctx_p->user.key_dma_addr,
+ dma_sync_single_for_device(dev, ctx_p->user.key_dma_addr,
max_key_buf_size, DMA_TO_DEVICE);
- SSI_UPDATE_DMA_ADDR_TO_48BIT(ctx_p->user.key_dma_addr ,max_key_buf_size);
ctx_p->keylen = keylen;
-
- END_CYCLE_COUNT(STAT_OP_TYPE_SETKEY, STAT_PHASE_1);
SSI_LOG_DEBUG("ssi_blkcipher_setkey: return safely");
return 0;
struct blkcipher_req_ctx *req_ctx,
unsigned int ivsize,
unsigned int nbytes,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
struct ssi_ablkcipher_ctx *ctx_p = crypto_tfm_ctx(tfm);
case DRV_CIPHER_CTR:
case DRV_CIPHER_OFB:
/* Load cipher state */
- HW_DESC_INIT(&desc[*seq_size]);
- HW_DESC_SET_DIN_TYPE(&desc[*seq_size], DMA_DLLI,
- iv_dma_addr, ivsize,
- NS_BIT);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[*seq_size], direction);
- HW_DESC_SET_FLOW_MODE(&desc[*seq_size], flow_mode);
- HW_DESC_SET_CIPHER_MODE(&desc[*seq_size], cipher_mode);
- if ((cipher_mode == DRV_CIPHER_CTR) ||
- (cipher_mode == DRV_CIPHER_OFB) ) {
- HW_DESC_SET_SETUP_MODE(&desc[*seq_size],
- SETUP_LOAD_STATE1);
+ hw_desc_init(&desc[*seq_size]);
+ set_din_type(&desc[*seq_size], DMA_DLLI, iv_dma_addr, ivsize,
+ NS_BIT);
+ set_cipher_config0(&desc[*seq_size], direction);
+ set_flow_mode(&desc[*seq_size], flow_mode);
+ set_cipher_mode(&desc[*seq_size], cipher_mode);
+ if ((cipher_mode == DRV_CIPHER_CTR) ||
+ (cipher_mode == DRV_CIPHER_OFB)) {
+ set_setup_mode(&desc[*seq_size], SETUP_LOAD_STATE1);
} else {
- HW_DESC_SET_SETUP_MODE(&desc[*seq_size],
- SETUP_LOAD_STATE0);
+ set_setup_mode(&desc[*seq_size], SETUP_LOAD_STATE0);
}
(*seq_size)++;
/*FALLTHROUGH*/
case DRV_CIPHER_ECB:
/* Load key */
- HW_DESC_INIT(&desc[*seq_size]);
- HW_DESC_SET_CIPHER_MODE(&desc[*seq_size], cipher_mode);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[*seq_size], direction);
+ hw_desc_init(&desc[*seq_size]);
+ set_cipher_mode(&desc[*seq_size], cipher_mode);
+ set_cipher_config0(&desc[*seq_size], direction);
if (flow_mode == S_DIN_to_AES) {
-
if (ssi_is_hw_key(tfm)) {
- HW_DESC_SET_HW_CRYPTO_KEY(&desc[*seq_size], ctx_p->hw.key1_slot);
+ set_hw_crypto_key(&desc[*seq_size],
+ ctx_p->hw.key1_slot);
} else {
- HW_DESC_SET_DIN_TYPE(&desc[*seq_size], DMA_DLLI,
- key_dma_addr,
- ((key_len == 24) ? AES_MAX_KEY_SIZE : key_len),
- NS_BIT);
+ set_din_type(&desc[*seq_size], DMA_DLLI,
+ key_dma_addr, ((key_len == 24) ?
+ AES_MAX_KEY_SIZE :
+ key_len), NS_BIT);
}
- HW_DESC_SET_KEY_SIZE_AES(&desc[*seq_size], key_len);
+ set_key_size_aes(&desc[*seq_size], key_len);
} else {
/*des*/
- HW_DESC_SET_DIN_TYPE(&desc[*seq_size], DMA_DLLI,
- key_dma_addr, key_len,
- NS_BIT);
- HW_DESC_SET_KEY_SIZE_DES(&desc[*seq_size], key_len);
+ set_din_type(&desc[*seq_size], DMA_DLLI, key_dma_addr,
+ key_len, NS_BIT);
+ set_key_size_des(&desc[*seq_size], key_len);
}
- HW_DESC_SET_FLOW_MODE(&desc[*seq_size], flow_mode);
- HW_DESC_SET_SETUP_MODE(&desc[*seq_size], SETUP_LOAD_KEY0);
+ set_flow_mode(&desc[*seq_size], flow_mode);
+ set_setup_mode(&desc[*seq_size], SETUP_LOAD_KEY0);
(*seq_size)++;
break;
case DRV_CIPHER_XTS:
case DRV_CIPHER_ESSIV:
case DRV_CIPHER_BITLOCKER:
/* Load AES key */
- HW_DESC_INIT(&desc[*seq_size]);
- HW_DESC_SET_CIPHER_MODE(&desc[*seq_size], cipher_mode);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[*seq_size], direction);
+ hw_desc_init(&desc[*seq_size]);
+ set_cipher_mode(&desc[*seq_size], cipher_mode);
+ set_cipher_config0(&desc[*seq_size], direction);
if (ssi_is_hw_key(tfm)) {
- HW_DESC_SET_HW_CRYPTO_KEY(&desc[*seq_size], ctx_p->hw.key1_slot);
+ set_hw_crypto_key(&desc[*seq_size],
+ ctx_p->hw.key1_slot);
} else {
- HW_DESC_SET_DIN_TYPE(&desc[*seq_size], DMA_DLLI,
- key_dma_addr, key_len/2,
- NS_BIT);
+ set_din_type(&desc[*seq_size], DMA_DLLI, key_dma_addr,
+ (key_len / 2), NS_BIT);
}
- HW_DESC_SET_KEY_SIZE_AES(&desc[*seq_size], key_len/2);
- HW_DESC_SET_FLOW_MODE(&desc[*seq_size], flow_mode);
- HW_DESC_SET_SETUP_MODE(&desc[*seq_size], SETUP_LOAD_KEY0);
+ set_key_size_aes(&desc[*seq_size], (key_len / 2));
+ set_flow_mode(&desc[*seq_size], flow_mode);
+ set_setup_mode(&desc[*seq_size], SETUP_LOAD_KEY0);
(*seq_size)++;
/* load XEX key */
- HW_DESC_INIT(&desc[*seq_size]);
- HW_DESC_SET_CIPHER_MODE(&desc[*seq_size], cipher_mode);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[*seq_size], direction);
+ hw_desc_init(&desc[*seq_size]);
+ set_cipher_mode(&desc[*seq_size], cipher_mode);
+ set_cipher_config0(&desc[*seq_size], direction);
if (ssi_is_hw_key(tfm)) {
- HW_DESC_SET_HW_CRYPTO_KEY(&desc[*seq_size], ctx_p->hw.key2_slot);
+ set_hw_crypto_key(&desc[*seq_size],
+ ctx_p->hw.key2_slot);
} else {
- HW_DESC_SET_DIN_TYPE(&desc[*seq_size], DMA_DLLI,
- (key_dma_addr+key_len/2), key_len/2,
- NS_BIT);
+ set_din_type(&desc[*seq_size], DMA_DLLI,
+ (key_dma_addr + (key_len / 2)),
+ (key_len / 2), NS_BIT);
}
- HW_DESC_SET_XEX_DATA_UNIT_SIZE(&desc[*seq_size], du_size);
- HW_DESC_SET_FLOW_MODE(&desc[*seq_size], S_DIN_to_AES2);
- HW_DESC_SET_KEY_SIZE_AES(&desc[*seq_size], key_len/2);
- HW_DESC_SET_SETUP_MODE(&desc[*seq_size], SETUP_LOAD_XEX_KEY);
+ set_xex_data_unit_size(&desc[*seq_size], du_size);
+ set_flow_mode(&desc[*seq_size], S_DIN_to_AES2);
+ set_key_size_aes(&desc[*seq_size], (key_len / 2));
+ set_setup_mode(&desc[*seq_size], SETUP_LOAD_XEX_KEY);
(*seq_size)++;
-
+
/* Set state */
- HW_DESC_INIT(&desc[*seq_size]);
- HW_DESC_SET_SETUP_MODE(&desc[*seq_size], SETUP_LOAD_STATE1);
- HW_DESC_SET_CIPHER_MODE(&desc[*seq_size], cipher_mode);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[*seq_size], direction);
- HW_DESC_SET_KEY_SIZE_AES(&desc[*seq_size], key_len/2);
- HW_DESC_SET_FLOW_MODE(&desc[*seq_size], flow_mode);
- HW_DESC_SET_DIN_TYPE(&desc[*seq_size], DMA_DLLI,
- iv_dma_addr, CC_AES_BLOCK_SIZE,
- NS_BIT);
+ hw_desc_init(&desc[*seq_size]);
+ set_setup_mode(&desc[*seq_size], SETUP_LOAD_STATE1);
+ set_cipher_mode(&desc[*seq_size], cipher_mode);
+ set_cipher_config0(&desc[*seq_size], direction);
+ set_key_size_aes(&desc[*seq_size], (key_len / 2));
+ set_flow_mode(&desc[*seq_size], flow_mode);
+ set_din_type(&desc[*seq_size], DMA_DLLI, iv_dma_addr,
+ CC_AES_BLOCK_SIZE, NS_BIT);
(*seq_size)++;
break;
default:
struct crypto_tfm *tfm,
struct blkcipher_req_ctx *req_ctx,
unsigned int ivsize,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
struct ssi_ablkcipher_ctx *ctx_p = crypto_tfm_ctx(tfm);
-
+
int direction = req_ctx->gen_ctx.op_type;
/* Load system key */
- HW_DESC_INIT(&desc[*seq_size]);
- HW_DESC_SET_CIPHER_MODE(&desc[*seq_size], ctx_p->cipher_mode);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[*seq_size], direction);
- HW_DESC_SET_DIN_TYPE(&desc[*seq_size], DMA_DLLI, ctx_p->user.key_dma_addr,
- CC_MULTI2_SYSTEM_KEY_SIZE,
- NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[*seq_size], ctx_p->flow_mode);
- HW_DESC_SET_SETUP_MODE(&desc[*seq_size], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[*seq_size]);
+ set_cipher_mode(&desc[*seq_size], ctx_p->cipher_mode);
+ set_cipher_config0(&desc[*seq_size], direction);
+ set_din_type(&desc[*seq_size], DMA_DLLI, ctx_p->user.key_dma_addr,
+ CC_MULTI2_SYSTEM_KEY_SIZE, NS_BIT);
+ set_flow_mode(&desc[*seq_size], ctx_p->flow_mode);
+ set_setup_mode(&desc[*seq_size], SETUP_LOAD_KEY0);
(*seq_size)++;
/* load data key */
- HW_DESC_INIT(&desc[*seq_size]);
- HW_DESC_SET_DIN_TYPE(&desc[*seq_size], DMA_DLLI,
- (ctx_p->user.key_dma_addr +
- CC_MULTI2_SYSTEM_KEY_SIZE),
- CC_MULTI2_DATA_KEY_SIZE, NS_BIT);
- HW_DESC_SET_MULTI2_NUM_ROUNDS(&desc[*seq_size],
- ctx_p->key_round_number);
- HW_DESC_SET_FLOW_MODE(&desc[*seq_size], ctx_p->flow_mode);
- HW_DESC_SET_CIPHER_MODE(&desc[*seq_size], ctx_p->cipher_mode);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[*seq_size], direction);
- HW_DESC_SET_SETUP_MODE(&desc[*seq_size], SETUP_LOAD_STATE0 );
+ hw_desc_init(&desc[*seq_size]);
+ set_din_type(&desc[*seq_size], DMA_DLLI,
+ (ctx_p->user.key_dma_addr + CC_MULTI2_SYSTEM_KEY_SIZE),
+ CC_MULTI2_DATA_KEY_SIZE, NS_BIT);
+ set_multi2_num_rounds(&desc[*seq_size], ctx_p->key_round_number);
+ set_flow_mode(&desc[*seq_size], ctx_p->flow_mode);
+ set_cipher_mode(&desc[*seq_size], ctx_p->cipher_mode);
+ set_cipher_config0(&desc[*seq_size], direction);
+ set_setup_mode(&desc[*seq_size], SETUP_LOAD_STATE0);
(*seq_size)++;
-
-
+
/* Set state */
- HW_DESC_INIT(&desc[*seq_size]);
- HW_DESC_SET_DIN_TYPE(&desc[*seq_size], DMA_DLLI,
- req_ctx->gen_ctx.iv_dma_addr,
- ivsize, NS_BIT);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[*seq_size], direction);
- HW_DESC_SET_FLOW_MODE(&desc[*seq_size], ctx_p->flow_mode);
- HW_DESC_SET_CIPHER_MODE(&desc[*seq_size], ctx_p->cipher_mode);
- HW_DESC_SET_SETUP_MODE(&desc[*seq_size], SETUP_LOAD_STATE1);
+ hw_desc_init(&desc[*seq_size]);
+ set_din_type(&desc[*seq_size], DMA_DLLI, req_ctx->gen_ctx.iv_dma_addr,
+ ivsize, NS_BIT);
+ set_cipher_config0(&desc[*seq_size], direction);
+ set_flow_mode(&desc[*seq_size], ctx_p->flow_mode);
+ set_cipher_mode(&desc[*seq_size], ctx_p->cipher_mode);
+ set_setup_mode(&desc[*seq_size], SETUP_LOAD_STATE1);
(*seq_size)++;
-
}
#endif /*SSI_CC_HAS_MULTI2*/
struct scatterlist *dst, struct scatterlist *src,
unsigned int nbytes,
void *areq,
- HwDesc_s desc[],
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
struct ssi_ablkcipher_ctx *ctx_p = crypto_tfm_ctx(tfm);
return;
}
/* Process */
- if (likely(req_ctx->dma_buf_type == SSI_DMA_BUF_DLLI)){
+ if (likely(req_ctx->dma_buf_type == SSI_DMA_BUF_DLLI)) {
SSI_LOG_DEBUG(" data params addr 0x%llX length 0x%X \n",
(unsigned long long)sg_dma_address(src),
nbytes);
SSI_LOG_DEBUG(" data params addr 0x%llX length 0x%X \n",
(unsigned long long)sg_dma_address(dst),
nbytes);
- HW_DESC_INIT(&desc[*seq_size]);
- HW_DESC_SET_DIN_TYPE(&desc[*seq_size], DMA_DLLI,
- sg_dma_address(src),
- nbytes, NS_BIT);
- HW_DESC_SET_DOUT_DLLI(&desc[*seq_size],
- sg_dma_address(dst),
- nbytes,
- NS_BIT, (areq == NULL)? 0:1);
- if (areq != NULL) {
- HW_DESC_SET_QUEUE_LAST_IND(&desc[*seq_size]);
- }
- HW_DESC_SET_FLOW_MODE(&desc[*seq_size], flow_mode);
+ hw_desc_init(&desc[*seq_size]);
+ set_din_type(&desc[*seq_size], DMA_DLLI, sg_dma_address(src),
+ nbytes, NS_BIT);
+ set_dout_dlli(&desc[*seq_size], sg_dma_address(dst),
+ nbytes, NS_BIT, (!areq ? 0 : 1));
+ if (areq)
+ set_queue_last_ind(&desc[*seq_size]);
+
+ set_flow_mode(&desc[*seq_size], flow_mode);
(*seq_size)++;
} else {
/* bypass */
(unsigned long long)req_ctx->mlli_params.mlli_dma_addr,
req_ctx->mlli_params.mlli_len,
(unsigned int)ctx_p->drvdata->mlli_sram_addr);
- HW_DESC_INIT(&desc[*seq_size]);
- HW_DESC_SET_DIN_TYPE(&desc[*seq_size], DMA_DLLI,
- req_ctx->mlli_params.mlli_dma_addr,
- req_ctx->mlli_params.mlli_len,
- NS_BIT);
- HW_DESC_SET_DOUT_SRAM(&desc[*seq_size],
- ctx_p->drvdata->mlli_sram_addr,
- req_ctx->mlli_params.mlli_len);
- HW_DESC_SET_FLOW_MODE(&desc[*seq_size], BYPASS);
+ hw_desc_init(&desc[*seq_size]);
+ set_din_type(&desc[*seq_size], DMA_DLLI,
+ req_ctx->mlli_params.mlli_dma_addr,
+ req_ctx->mlli_params.mlli_len, NS_BIT);
+ set_dout_sram(&desc[*seq_size],
+ ctx_p->drvdata->mlli_sram_addr,
+ req_ctx->mlli_params.mlli_len);
+ set_flow_mode(&desc[*seq_size], BYPASS);
(*seq_size)++;
- HW_DESC_INIT(&desc[*seq_size]);
- HW_DESC_SET_DIN_TYPE(&desc[*seq_size], DMA_MLLI,
- ctx_p->drvdata->mlli_sram_addr,
- req_ctx->in_mlli_nents, NS_BIT);
+ hw_desc_init(&desc[*seq_size]);
+ set_din_type(&desc[*seq_size], DMA_MLLI,
+ ctx_p->drvdata->mlli_sram_addr,
+ req_ctx->in_mlli_nents, NS_BIT);
if (req_ctx->out_nents == 0) {
SSI_LOG_DEBUG(" din/dout params addr 0x%08X "
"addr 0x%08X\n",
(unsigned int)ctx_p->drvdata->mlli_sram_addr,
(unsigned int)ctx_p->drvdata->mlli_sram_addr);
- HW_DESC_SET_DOUT_MLLI(&desc[*seq_size],
- ctx_p->drvdata->mlli_sram_addr,
- req_ctx->in_mlli_nents,
- NS_BIT,(areq == NULL)? 0:1);
+ set_dout_mlli(&desc[*seq_size],
+ ctx_p->drvdata->mlli_sram_addr,
+ req_ctx->in_mlli_nents, NS_BIT,
+ (!areq ? 0 : 1));
} else {
SSI_LOG_DEBUG(" din/dout params "
"addr 0x%08X addr 0x%08X\n",
(unsigned int)ctx_p->drvdata->mlli_sram_addr,
- (unsigned int)ctx_p->drvdata->mlli_sram_addr +
- (uint32_t)LLI_ENTRY_BYTE_SIZE *
+ (unsigned int)ctx_p->drvdata->mlli_sram_addr +
+ (u32)LLI_ENTRY_BYTE_SIZE *
req_ctx->in_nents);
- HW_DESC_SET_DOUT_MLLI(&desc[*seq_size],
- (ctx_p->drvdata->mlli_sram_addr +
- LLI_ENTRY_BYTE_SIZE *
- req_ctx->in_mlli_nents),
- req_ctx->out_mlli_nents, NS_BIT,(areq == NULL)? 0:1);
+ set_dout_mlli(&desc[*seq_size],
+ (ctx_p->drvdata->mlli_sram_addr +
+ (LLI_ENTRY_BYTE_SIZE *
+ req_ctx->in_mlli_nents)),
+ req_ctx->out_mlli_nents, NS_BIT,
+ (!areq ? 0 : 1));
}
- if (areq != NULL) {
- HW_DESC_SET_QUEUE_LAST_IND(&desc[*seq_size]);
- }
- HW_DESC_SET_FLOW_MODE(&desc[*seq_size], flow_mode);
+ if (areq)
+ set_queue_last_ind(&desc[*seq_size]);
+
+ set_flow_mode(&desc[*seq_size], flow_mode);
(*seq_size)++;
}
}
static int ssi_blkcipher_complete(struct device *dev,
- struct ssi_ablkcipher_ctx *ctx_p,
- struct blkcipher_req_ctx *req_ctx,
- struct scatterlist *dst, struct scatterlist *src,
- void *info, //req info
- unsigned int ivsize,
- void *areq,
- void __iomem *cc_base)
+ struct ssi_ablkcipher_ctx *ctx_p,
+ struct blkcipher_req_ctx *req_ctx,
+ struct scatterlist *dst,
+ struct scatterlist *src,
+ unsigned int ivsize,
+ void *areq,
+ void __iomem *cc_base)
{
int completion_error = 0;
- uint32_t inflight_counter;
- DECL_CYCLE_COUNT_RESOURCES;
+ u32 inflight_counter;
- START_CYCLE_COUNT();
ssi_buffer_mgr_unmap_blkcipher_request(dev, req_ctx, ivsize, src, dst);
- info = req_ctx->backup_info;
- END_CYCLE_COUNT(STAT_OP_TYPE_GENERIC, STAT_PHASE_4);
-
/*Set the inflight couter value to local variable*/
inflight_counter = ctx_p->drvdata->inflight_counter;
/*Decrease the inflight counter*/
- if(ctx_p->flow_mode == BYPASS && ctx_p->drvdata->inflight_counter > 0)
+ if (ctx_p->flow_mode == BYPASS && ctx_p->drvdata->inflight_counter > 0)
ctx_p->drvdata->inflight_counter--;
- if(areq){
+ if (areq) {
ablkcipher_request_complete(areq, completion_error);
return 0;
}
unsigned int nbytes,
void *info, //req info
unsigned int ivsize,
- void *areq,
+ void *areq,
enum drv_crypto_direction direction)
{
struct ssi_ablkcipher_ctx *ctx_p = crypto_tfm_ctx(tfm);
struct device *dev = &ctx_p->drvdata->plat_dev->dev;
- HwDesc_s desc[MAX_ABLKCIPHER_SEQ_LEN];
+ struct cc_hw_desc desc[MAX_ABLKCIPHER_SEQ_LEN];
struct ssi_crypto_req ssi_req = {};
- int rc, seq_len = 0,cts_restore_flag = 0;
- DECL_CYCLE_COUNT_RESOURCES;
+ int rc, seq_len = 0, cts_restore_flag = 0;
SSI_LOG_DEBUG("%s areq=%p info=%p nbytes=%d\n",
- ((direction==DRV_CRYPTO_DIRECTION_ENCRYPT)?"Encrypt":"Decrypt"),
+ ((direction == DRV_CRYPTO_DIRECTION_ENCRYPT) ? "Encrypt" : "Decrypt"),
areq, info, nbytes);
CHECK_AND_RETURN_UPON_FIPS_ERROR();
/* STAT_PHASE_0: Init and sanity checks */
- START_CYCLE_COUNT();
-
+
/* TODO: check data length according to mode */
if (unlikely(validate_data_size(ctx_p, nbytes))) {
SSI_LOG_ERR("Unsupported data size %d.\n", nbytes);
/* No data to process is valid */
return 0;
}
- /*For CTS in case of data size aligned to 16 use CBC mode*/
- if (((nbytes % AES_BLOCK_SIZE) == 0) && (ctx_p->cipher_mode == DRV_CIPHER_CBC_CTS)){
-
+ /*For CTS in case of data size aligned to 16 use CBC mode*/
+ if (((nbytes % AES_BLOCK_SIZE) == 0) && (ctx_p->cipher_mode == DRV_CIPHER_CBC_CTS)) {
ctx_p->cipher_mode = DRV_CIPHER_CBC;
cts_restore_flag = 1;
}
/* Setup request context */
req_ctx->gen_ctx.op_type = direction;
-
- END_CYCLE_COUNT(ssi_req.op_type, STAT_PHASE_0);
/* STAT_PHASE_1: Map buffers */
- START_CYCLE_COUNT();
-
+
rc = ssi_buffer_mgr_map_blkcipher_request(ctx_p->drvdata, req_ctx, ivsize, nbytes, info, src, dst);
if (unlikely(rc != 0)) {
SSI_LOG_ERR("map_request() failed\n");
goto exit_process;
}
- END_CYCLE_COUNT(ssi_req.op_type, STAT_PHASE_1);
-
/* STAT_PHASE_2: Create sequence */
- START_CYCLE_COUNT();
/* Setup processing */
#if SSI_CC_HAS_MULTI2
- if (ctx_p->flow_mode == S_DIN_to_MULTI2) {
- ssi_blkcipher_create_multi2_setup_desc(tfm,
- req_ctx,
- ivsize,
- desc,
- &seq_len);
- } else
+ if (ctx_p->flow_mode == S_DIN_to_MULTI2)
+ ssi_blkcipher_create_multi2_setup_desc(tfm, req_ctx, ivsize,
+ desc, &seq_len);
+ else
#endif /*SSI_CC_HAS_MULTI2*/
- {
- ssi_blkcipher_create_setup_desc(tfm,
- req_ctx,
- ivsize,
- nbytes,
- desc,
- &seq_len);
- }
+ ssi_blkcipher_create_setup_desc(tfm, req_ctx, ivsize, nbytes,
+ desc, &seq_len);
/* Data processing */
ssi_blkcipher_create_data_desc(tfm,
- req_ctx,
+ req_ctx,
dst, src,
nbytes,
areq,
desc, &seq_len);
/* do we need to generate IV? */
- if (req_ctx->is_giv == true) {
+ if (req_ctx->is_giv) {
ssi_req.ivgen_dma_addr[0] = req_ctx->gen_ctx.iv_dma_addr;
ssi_req.ivgen_dma_addr_len = 1;
/* set the IV size (8/16 B long)*/
ssi_req.ivgen_size = ivsize;
}
- END_CYCLE_COUNT(ssi_req.op_type, STAT_PHASE_2);
/* STAT_PHASE_3: Lock HW and push sequence */
- START_CYCLE_COUNT();
-
- rc = send_request(ctx_p->drvdata, &ssi_req, desc, seq_len, (areq == NULL)? 0:1);
- if(areq != NULL) {
+
+ rc = send_request(ctx_p->drvdata, &ssi_req, desc, seq_len, (!areq) ? 0 : 1);
+ if (areq) {
if (unlikely(rc != -EINPROGRESS)) {
/* Failed to send the request or request completed synchronously */
ssi_buffer_mgr_unmap_blkcipher_request(dev, req_ctx, ivsize, src, dst);
}
- END_CYCLE_COUNT(ssi_req.op_type, STAT_PHASE_3);
} else {
if (rc != 0) {
ssi_buffer_mgr_unmap_blkcipher_request(dev, req_ctx, ivsize, src, dst);
- END_CYCLE_COUNT(ssi_req.op_type, STAT_PHASE_3);
} else {
- END_CYCLE_COUNT(ssi_req.op_type, STAT_PHASE_3);
- rc = ssi_blkcipher_complete(dev, ctx_p, req_ctx, dst, src, info, ivsize, NULL, ctx_p->drvdata->cc_base);
- }
+ rc = ssi_blkcipher_complete(dev, ctx_p, req_ctx, dst,
+ src, ivsize, NULL,
+ ctx_p->drvdata->cc_base);
+ }
}
exit_process:
if (cts_restore_flag != 0)
ctx_p->cipher_mode = DRV_CIPHER_CBC_CTS;
-
+
return rc;
}
CHECK_AND_RETURN_VOID_UPON_FIPS_ERROR();
- ssi_blkcipher_complete(dev, ctx_p, req_ctx, areq->dst, areq->src, areq->info, ivsize, areq, cc_base);
-}
-
-
-
-static int ssi_sblkcipher_init(struct crypto_tfm *tfm)
-{
- struct ssi_ablkcipher_ctx *ctx_p = crypto_tfm_ctx(tfm);
-
- /* Allocate sync ctx buffer */
- ctx_p->sync_ctx = kmalloc(sizeof(struct blkcipher_req_ctx), GFP_KERNEL|GFP_DMA);
- if (!ctx_p->sync_ctx) {
- SSI_LOG_ERR("Allocating sync ctx buffer in context failed\n");
- return -ENOMEM;
- }
- SSI_LOG_DEBUG("Allocated sync ctx buffer in context ctx_p->sync_ctx=@%p\n",
- ctx_p->sync_ctx);
-
- return ssi_blkcipher_init(tfm);
-}
-
-
-static void ssi_sblkcipher_exit(struct crypto_tfm *tfm)
-{
- struct ssi_ablkcipher_ctx *ctx_p = crypto_tfm_ctx(tfm);
-
- kfree(ctx_p->sync_ctx);
- SSI_LOG_DEBUG("Free sync ctx buffer in context ctx_p->sync_ctx=@%p\n", ctx_p->sync_ctx);
-
- ssi_blkcipher_exit(tfm);
-}
-
-#ifdef SYNC_ALGS
-static int ssi_sblkcipher_encrypt(struct blkcipher_desc *desc,
- struct scatterlist *dst, struct scatterlist *src,
- unsigned int nbytes)
-{
- struct crypto_blkcipher *blk_tfm = desc->tfm;
- struct crypto_tfm *tfm = crypto_blkcipher_tfm(blk_tfm);
- struct ssi_ablkcipher_ctx *ctx_p = crypto_tfm_ctx(tfm);
- struct blkcipher_req_ctx *req_ctx = ctx_p->sync_ctx;
- unsigned int ivsize = crypto_blkcipher_ivsize(blk_tfm);
-
- req_ctx->backup_info = desc->info;
- req_ctx->is_giv = false;
-
- return ssi_blkcipher_process(tfm, req_ctx, dst, src, nbytes, desc->info, ivsize, NULL, DRV_CRYPTO_DIRECTION_ENCRYPT);
-}
-
-static int ssi_sblkcipher_decrypt(struct blkcipher_desc *desc,
- struct scatterlist *dst, struct scatterlist *src,
- unsigned int nbytes)
-{
- struct crypto_blkcipher *blk_tfm = desc->tfm;
- struct crypto_tfm *tfm = crypto_blkcipher_tfm(blk_tfm);
- struct ssi_ablkcipher_ctx *ctx_p = crypto_tfm_ctx(tfm);
- struct blkcipher_req_ctx *req_ctx = ctx_p->sync_ctx;
- unsigned int ivsize = crypto_blkcipher_ivsize(blk_tfm);
-
- req_ctx->backup_info = desc->info;
- req_ctx->is_giv = false;
-
- return ssi_blkcipher_process(tfm, req_ctx, dst, src, nbytes, desc->info, ivsize, NULL, DRV_CRYPTO_DIRECTION_DECRYPT);
+ ssi_blkcipher_complete(dev, ctx_p, req_ctx, areq->dst, areq->src,
+ ivsize, areq, cc_base);
}
-#endif
/* Async wrap functions */
static int ssi_ablkcipher_init(struct crypto_tfm *tfm)
{
struct ablkcipher_tfm *ablktfm = &tfm->crt_ablkcipher;
-
+
ablktfm->reqsize = sizeof(struct blkcipher_req_ctx);
return ssi_blkcipher_init(tfm);
}
-
-static int ssi_ablkcipher_setkey(struct crypto_ablkcipher *tfm,
- const u8 *key,
+static int ssi_ablkcipher_setkey(struct crypto_ablkcipher *tfm,
+ const u8 *key,
unsigned int keylen)
{
return ssi_blkcipher_setkey(crypto_ablkcipher_tfm(tfm), key, keylen);
return ssi_blkcipher_process(tfm, req_ctx, req->dst, req->src, req->nbytes, req->info, ivsize, (void *)req, DRV_CRYPTO_DIRECTION_DECRYPT);
}
-
/* DX Block cipher alg */
static struct ssi_alg_template blkcipher_algs[] = {
/* Async template */
},
.cipher_mode = DRV_CIPHER_XTS,
.flow_mode = S_DIN_to_AES,
- .synchronous = false,
},
{
.name = "xts(aes)",
},
.cipher_mode = DRV_CIPHER_XTS,
.flow_mode = S_DIN_to_AES,
- .synchronous = false,
},
{
.name = "xts(aes)",
},
.cipher_mode = DRV_CIPHER_XTS,
.flow_mode = S_DIN_to_AES,
- .synchronous = false,
},
#endif /*SSI_CC_HAS_AES_XTS*/
#if SSI_CC_HAS_AES_ESSIV
},
.cipher_mode = DRV_CIPHER_ESSIV,
.flow_mode = S_DIN_to_AES,
- .synchronous = false,
},
{
.name = "essiv(aes)",
},
.cipher_mode = DRV_CIPHER_ESSIV,
.flow_mode = S_DIN_to_AES,
- .synchronous = false,
},
{
.name = "essiv(aes)",
},
.cipher_mode = DRV_CIPHER_ESSIV,
.flow_mode = S_DIN_to_AES,
- .synchronous = false,
},
#endif /*SSI_CC_HAS_AES_ESSIV*/
#if SSI_CC_HAS_AES_BITLOCKER
},
.cipher_mode = DRV_CIPHER_BITLOCKER,
.flow_mode = S_DIN_to_AES,
- .synchronous = false,
},
{
.name = "bitlocker(aes)",
},
.cipher_mode = DRV_CIPHER_BITLOCKER,
.flow_mode = S_DIN_to_AES,
- .synchronous = false,
},
{
.name = "bitlocker(aes)",
},
.cipher_mode = DRV_CIPHER_BITLOCKER,
.flow_mode = S_DIN_to_AES,
- .synchronous = false,
},
#endif /*SSI_CC_HAS_AES_BITLOCKER*/
{
},
.cipher_mode = DRV_CIPHER_ECB,
.flow_mode = S_DIN_to_AES,
- .synchronous = false,
},
{
.name = "cbc(aes)",
.min_keysize = AES_MIN_KEY_SIZE,
.max_keysize = AES_MAX_KEY_SIZE,
.ivsize = AES_BLOCK_SIZE,
- },
+ },
.cipher_mode = DRV_CIPHER_CBC,
.flow_mode = S_DIN_to_AES,
- .synchronous = false,
},
{
.name = "ofb(aes)",
},
.cipher_mode = DRV_CIPHER_OFB,
.flow_mode = S_DIN_to_AES,
- .synchronous = false,
},
#if SSI_CC_HAS_AES_CTS
{
},
.cipher_mode = DRV_CIPHER_CBC_CTS,
.flow_mode = S_DIN_to_AES,
- .synchronous = false,
},
#endif
{
},
.cipher_mode = DRV_CIPHER_CTR,
.flow_mode = S_DIN_to_AES,
- .synchronous = false,
},
{
.name = "cbc(des3_ede)",
},
.cipher_mode = DRV_CIPHER_CBC,
.flow_mode = S_DIN_to_DES,
- .synchronous = false,
},
{
.name = "ecb(des3_ede)",
},
.cipher_mode = DRV_CIPHER_ECB,
.flow_mode = S_DIN_to_DES,
- .synchronous = false,
},
{
.name = "cbc(des)",
},
.cipher_mode = DRV_CIPHER_CBC,
.flow_mode = S_DIN_to_DES,
- .synchronous = false,
},
{
.name = "ecb(des)",
},
.cipher_mode = DRV_CIPHER_ECB,
.flow_mode = S_DIN_to_DES,
- .synchronous = false,
},
#if SSI_CC_HAS_MULTI2
{
},
.cipher_mode = DRV_MULTI2_CBC,
.flow_mode = S_DIN_to_MULTI2,
- .synchronous = false,
},
{
.name = "ofb(multi2)",
},
.cipher_mode = DRV_MULTI2_OFB,
.flow_mode = S_DIN_to_MULTI2,
- .synchronous = false,
},
#endif /*SSI_CC_HAS_MULTI2*/
};
-static
+static
struct ssi_crypto_alg *ssi_ablkcipher_create_alg(struct ssi_alg_template *template)
{
struct ssi_crypto_alg *t_alg;
alg->cra_blocksize = template->blocksize;
alg->cra_alignmask = 0;
alg->cra_ctxsize = sizeof(struct ssi_ablkcipher_ctx);
-
- alg->cra_init = template->synchronous? ssi_sblkcipher_init:ssi_ablkcipher_init;
- alg->cra_exit = template->synchronous? ssi_sblkcipher_exit:ssi_blkcipher_exit;
- alg->cra_type = template->synchronous? &crypto_blkcipher_type:&crypto_ablkcipher_type;
- if(template->synchronous) {
- alg->cra_blkcipher = template->template_sblkcipher;
- alg->cra_flags = CRYPTO_ALG_KERN_DRIVER_ONLY |
- template->type;
- } else {
- alg->cra_ablkcipher = template->template_ablkcipher;
- alg->cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_KERN_DRIVER_ONLY |
+
+ alg->cra_init = ssi_ablkcipher_init;
+ alg->cra_exit = ssi_blkcipher_exit;
+ alg->cra_type = &crypto_ablkcipher_type;
+ alg->cra_ablkcipher = template->template_ablkcipher;
+ alg->cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_KERN_DRIVER_ONLY |
template->type;
- }
t_alg->cipher_mode = template->cipher_mode;
t_alg->flow_mode = template->flow_mode;
int ssi_ablkcipher_free(struct ssi_drvdata *drvdata)
{
struct ssi_crypto_alg *t_alg, *n;
- struct ssi_blkcipher_handle *blkcipher_handle =
+ struct ssi_blkcipher_handle *blkcipher_handle =
drvdata->blkcipher_handle;
struct device *dev;
+
dev = &drvdata->plat_dev->dev;
- if (blkcipher_handle != NULL) {
+ if (blkcipher_handle) {
/* Remove registered algs */
list_for_each_entry_safe(t_alg, n,
&blkcipher_handle->blkcipher_alg_list,
return 0;
}
-
-
int ssi_ablkcipher_alloc(struct ssi_drvdata *drvdata)
{
struct ssi_blkcipher_handle *ablkcipher_handle;
ablkcipher_handle = kmalloc(sizeof(struct ssi_blkcipher_handle),
GFP_KERNEL);
- if (ablkcipher_handle == NULL)
+ if (!ablkcipher_handle)
return -ENOMEM;
drvdata->blkcipher_handle = ablkcipher_handle;
kfree(t_alg);
goto fail0;
} else {
- list_add_tail(&t_alg->entry,
+ list_add_tail(&t_alg->entry,
&ablkcipher_handle->blkcipher_alg_list);
- SSI_LOG_DEBUG("Registered %s\n",
+ SSI_LOG_DEBUG("Registered %s\n",
t_alg->crypto_alg.cra_driver_name);
}
}
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
/* \file ssi_cipher.h
- ARM CryptoCell Cipher Crypto API
+ * ARM CryptoCell Cipher Crypto API
*/
#ifndef __SSI_CIPHER_H__
#include "ssi_driver.h"
#include "ssi_buffer_mgr.h"
-
/* Crypto cipher flags */
#define CC_CRYPTO_CIPHER_KEY_KFDE0 (1 << 0)
#define CC_CRYPTO_CIPHER_KEY_KFDE1 (1 << 1)
#define CC_CRYPTO_CIPHER_KEY_KFDE_MASK (CC_CRYPTO_CIPHER_KEY_KFDE0 | CC_CRYPTO_CIPHER_KEY_KFDE1 | CC_CRYPTO_CIPHER_KEY_KFDE2 | CC_CRYPTO_CIPHER_KEY_KFDE3)
-
struct blkcipher_req_ctx {
struct async_gen_req_ctx gen_ctx;
enum ssi_req_dma_buf_type dma_buf_type;
- uint32_t in_nents;
- uint32_t in_mlli_nents;
- uint32_t out_nents;
- uint32_t out_mlli_nents;
- uint8_t *backup_info; /*store iv for generated IV flow*/
+ u32 in_nents;
+ u32 in_mlli_nents;
+ u32 out_nents;
+ u32 out_mlli_nents;
+ u8 *backup_info; /*store iv for generated IV flow*/
bool is_giv;
struct mlli_params mlli_params;
};
-
-
int ssi_ablkcipher_alloc(struct ssi_drvdata *drvdata);
int ssi_ablkcipher_free(struct ssi_drvdata *drvdata);
CRYPTO_ALG_BULK_DU_4096)
#endif /* CRYPTO_ALG_BULK_MASK */
-
#ifdef CRYPTO_TFM_REQ_HW_KEY
static inline bool ssi_is_hw_key(struct crypto_tfm *tfm)
return (crypto_tfm_get_flags(tfm) & CRYPTO_TFM_REQ_HW_KEY);
}
-#else
+#else
struct arm_hw_key_info {
int hw_key1;
#endif /* CRYPTO_TFM_REQ_HW_KEY */
-
#endif /*__SSI_CIPHER_H__*/
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
/* \file ssi_config.h
- Definitions for ARM CryptoCell Linux Crypto Driver
+ * Definitions for ARM CryptoCell Linux Crypto Driver
*/
#ifndef __SSI_CONFIG_H__
#include <linux/version.h>
-#define DISABLE_COHERENT_DMA_OPS
//#define FLUSH_CACHE_ALL
//#define COMPLETION_DELAY
//#define DX_DUMP_DESCS
// #define DX_DUMP_BYTES
// #define CC_DEBUG
#define ENABLE_CC_SYSFS /* Enable sysfs interface for debugging REE driver */
-//#define ENABLE_CC_CYCLE_COUNT
//#define DX_IRQ_DELAY 100000
#define DMA_BIT_MASK_LEN 48 /* was 32 bit, but for juno's sake it was enlarged to 48 bit */
-#if defined ENABLE_CC_CYCLE_COUNT && defined ENABLE_CC_SYSFS
-#define CC_CYCLE_COUNT
-#endif
-
-
-#if defined (CONFIG_ARM64) // TODO currently only this mode was test on Juno (which is ARM64), need to enable coherent also.
-#define DISABLE_COHERENT_DMA_OPS
-#endif
-
-/* Define the CryptoCell DMA cache coherency signals configuration */
-#if defined (DISABLE_COHERENT_DMA_OPS)
- /* Software Controlled Cache Coherency (SCCC) */
- #define SSI_CACHE_PARAMS (0x000)
- /* CC attached to NONE-ACP such as HPP/ACE/AMBA4.
- * The customer is responsible to enable/disable this feature
- * according to his platform type. */
- #define DX_HAS_ACP 0
-#else
- #define SSI_CACHE_PARAMS (0xEEE)
- /* CC attached to ACP */
- #define DX_HAS_ACP 1
-#endif
-
#endif /*__DX_CONFIG_H__*/
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
#include <linux/sched.h>
#include <linux/random.h>
#include <linux/of.h>
+#include <linux/clk.h>
+#include <linux/of_address.h>
#include "ssi_config.h"
#include "ssi_driver.h"
#include "ssi_pm.h"
#include "ssi_fips_local.h"
-
#ifdef DX_DUMP_BYTES
-void dump_byte_array(const char *name, const uint8_t *the_array, unsigned long size)
+void dump_byte_array(const char *name, const u8 *the_array, unsigned long size)
{
- int i , line_offset = 0, ret = 0;
- const uint8_t *cur_byte;
+ int i, line_offset = 0, ret = 0;
+ const u8 *cur_byte;
char line_buf[80];
- if (the_array == NULL) {
+ if (!the_array) {
SSI_LOG_ERR("cannot dump_byte_array - NULL pointer\n");
return;
}
ret = snprintf(line_buf, sizeof(line_buf), "%s[%lu]: ",
name, size);
if (ret < 0) {
- SSI_LOG_ERR("snprintf returned %d . aborting buffer array dump\n",ret);
+ SSI_LOG_ERR("snprintf returned %d . aborting buffer array dump\n", ret);
return;
}
line_offset = ret;
- for (i = 0 , cur_byte = the_array;
+ for (i = 0, cur_byte = the_array;
(i < size) && (line_offset < sizeof(line_buf)); i++, cur_byte++) {
ret = snprintf(line_buf + line_offset,
sizeof(line_buf) - line_offset,
"0x%02X ", *cur_byte);
if (ret < 0) {
- SSI_LOG_ERR("snprintf returned %d . aborting buffer array dump\n",ret);
+ SSI_LOG_ERR("snprintf returned %d . aborting buffer array dump\n", ret);
return;
}
line_offset += ret;
{
struct ssi_drvdata *drvdata = (struct ssi_drvdata *)dev_id;
void __iomem *cc_base = drvdata->cc_base;
- uint32_t irr;
- uint32_t imr;
- DECL_CYCLE_COUNT_RESOURCES;
+ u32 irr;
+ u32 imr;
/* STAT_OP_TYPE_GENERIC STAT_PHASE_0: Interrupt */
- START_CYCLE_COUNT();
/* read the interrupt status */
irr = CC_HAL_READ_REGISTER(CC_REG_OFFSET(HOST_RGF, HOST_IRR));
#endif
/* AXI error interrupt */
if (unlikely((irr & SSI_AXI_ERR_IRQ_MASK) != 0)) {
- uint32_t axi_err;
-
+ u32 axi_err;
+
/* Read the AXI error ID */
axi_err = CC_HAL_READ_REGISTER(CC_REG_OFFSET(CRY_KERNEL, AXIM_MON_ERR));
SSI_LOG_DEBUG("AXI completion error: axim_mon_err=0x%08X\n", axi_err);
-
+
irr &= ~SSI_AXI_ERR_IRQ_MASK;
}
/* Just warning */
}
- END_CYCLE_COUNT(STAT_OP_TYPE_GENERIC, STAT_PHASE_0);
- START_CYCLE_COUNT_AT(drvdata->isr_exit_cycles);
-
return IRQ_HANDLED;
}
int init_cc_regs(struct ssi_drvdata *drvdata, bool is_probe)
{
- unsigned int val;
+ unsigned int val, cache_params;
void __iomem *cc_base = drvdata->cc_base;
/* Unmask all AXI interrupt sources AXI_CFG1 register */
/* Unmask relevant interrupt cause */
val = (~(SSI_COMP_IRQ_MASK | SSI_AXI_ERR_IRQ_MASK | SSI_GPR0_IRQ_MASK));
CC_HAL_WRITE_REGISTER(CC_REG_OFFSET(HOST_RGF, HOST_IMR), val);
-
+
#ifdef DX_HOST_IRQ_TIMER_INIT_VAL_REG_OFFSET
#ifdef DX_IRQ_DELAY
/* Set CC IRQ delay */
}
#endif
+ cache_params = (drvdata->coherent ? CC_COHERENT_CACHE_PARAMS : 0x0);
+
val = CC_HAL_READ_REGISTER(CC_REG_OFFSET(CRY_KERNEL, AXIM_CACHE_PARAMS));
- if (is_probe == true) {
+
+ if (is_probe)
SSI_LOG_INFO("Cache params previous: 0x%08X\n", val);
- }
- CC_HAL_WRITE_REGISTER(CC_REG_OFFSET(CRY_KERNEL, AXIM_CACHE_PARAMS), SSI_CACHE_PARAMS);
+
+ CC_HAL_WRITE_REGISTER(CC_REG_OFFSET(CRY_KERNEL, AXIM_CACHE_PARAMS),
+ cache_params);
val = CC_HAL_READ_REGISTER(CC_REG_OFFSET(CRY_KERNEL, AXIM_CACHE_PARAMS));
- if (is_probe == true) {
- SSI_LOG_INFO("Cache params current: 0x%08X (expected: 0x%08X)\n", val, SSI_CACHE_PARAMS);
- }
+
+ if (is_probe)
+ SSI_LOG_INFO("Cache params current: 0x%08X (expect: 0x%08X)\n",
+ val, cache_params);
return 0;
}
void __iomem *cc_base = NULL;
bool irq_registered = false;
struct ssi_drvdata *new_drvdata = kzalloc(sizeof(struct ssi_drvdata), GFP_KERNEL);
- uint32_t signature_val;
+ struct device *dev = &plat_dev->dev;
+ struct device_node *np = dev->of_node;
+ u32 signature_val;
int rc = 0;
- if (unlikely(new_drvdata == NULL)) {
+ if (unlikely(!new_drvdata)) {
SSI_LOG_ERR("Failed to allocate drvdata");
rc = -ENOMEM;
goto init_cc_res_err;
}
+ new_drvdata->clk = of_clk_get(np, 0);
+ new_drvdata->coherent = of_dma_is_coherent(np);
+
/*Initialize inflight counter used in dx_ablkcipher_secure_complete used for count of BYSPASS blocks operations*/
new_drvdata->inflight_counter = 0;
/* Get device resources */
/* First CC registers space */
new_drvdata->res_mem = platform_get_resource(plat_dev, IORESOURCE_MEM, 0);
- if (unlikely(new_drvdata->res_mem == NULL)) {
+ if (unlikely(!new_drvdata->res_mem)) {
SSI_LOG_ERR("Failed getting IO memory resource\n");
rc = -ENODEV;
goto init_cc_res_err;
(unsigned long long)new_drvdata->res_mem->end);
/* Map registers space */
req_mem_cc_regs = request_mem_region(new_drvdata->res_mem->start, resource_size(new_drvdata->res_mem), "arm_cc7x_regs");
- if (unlikely(req_mem_cc_regs == NULL)) {
+ if (unlikely(!req_mem_cc_regs)) {
SSI_LOG_ERR("Couldn't allocate registers memory region at "
"0x%08X\n", (unsigned int)new_drvdata->res_mem->start);
rc = -EBUSY;
goto init_cc_res_err;
}
cc_base = ioremap(new_drvdata->res_mem->start, resource_size(new_drvdata->res_mem));
- if (unlikely(cc_base == NULL)) {
+ if (unlikely(!cc_base)) {
SSI_LOG_ERR("ioremap[CC](0x%08X,0x%08X) failed\n",
(unsigned int)new_drvdata->res_mem->start, (unsigned int)resource_size(new_drvdata->res_mem));
rc = -ENOMEM;
}
SSI_LOG_DEBUG("CC registers mapped from %pa to 0x%p\n", &new_drvdata->res_mem->start, cc_base);
new_drvdata->cc_base = cc_base;
-
/* Then IRQ */
new_drvdata->res_irq = platform_get_resource(plat_dev, IORESOURCE_IRQ, 0);
- if (unlikely(new_drvdata->res_irq == NULL)) {
+ if (unlikely(!new_drvdata->res_irq)) {
SSI_LOG_ERR("Failed getting IRQ resource\n");
rc = -ENODEV;
goto init_cc_res_err;
new_drvdata->plat_dev = plat_dev;
- if(new_drvdata->plat_dev->dev.dma_mask == NULL)
+ rc = cc_clk_on(new_drvdata);
+ if (rc)
+ goto init_cc_res_err;
+
+ if (!new_drvdata->plat_dev->dev.dma_mask)
{
- new_drvdata->plat_dev->dev.dma_mask = & new_drvdata->plat_dev->dev.coherent_dma_mask;
+ new_drvdata->plat_dev->dev.dma_mask = &new_drvdata->plat_dev->dev.coherent_dma_mask;
}
if (!new_drvdata->plat_dev->dev.coherent_dma_mask)
{
signature_val = CC_HAL_READ_REGISTER(CC_REG_OFFSET(HOST_RGF, HOST_SIGNATURE));
if (signature_val != DX_DEV_SIGNATURE) {
SSI_LOG_ERR("Invalid CC signature: SIGNATURE=0x%08X != expected=0x%08X\n",
- signature_val, (uint32_t)DX_DEV_SIGNATURE);
+ signature_val, (u32)DX_DEV_SIGNATURE);
rc = -EINVAL;
goto init_cc_res_err;
}
init_cc_res_err:
SSI_LOG_ERR("Freeing CC HW resources!\n");
-
- if (new_drvdata != NULL) {
+
+ if (new_drvdata) {
ssi_aead_free(new_drvdata);
ssi_hash_free(new_drvdata);
ssi_ablkcipher_free(new_drvdata);
#ifdef ENABLE_CC_SYSFS
ssi_sysfs_fini();
#endif
-
- if (req_mem_cc_regs != NULL) {
+
+ if (req_mem_cc_regs) {
if (irq_registered) {
free_irq(new_drvdata->res_irq->start, new_drvdata);
new_drvdata->res_irq = NULL;
void fini_cc_regs(struct ssi_drvdata *drvdata)
{
/* Mask all interrupts */
- WRITE_REGISTER(drvdata->cc_base +
+ WRITE_REGISTER(drvdata->cc_base +
CC_REG_OFFSET(HOST_RGF, HOST_IMR), 0xFFFFFFFF);
-
}
static void cleanup_cc_resources(struct platform_device *plat_dev)
struct ssi_drvdata *drvdata =
(struct ssi_drvdata *)dev_get_drvdata(&plat_dev->dev);
- ssi_aead_free(drvdata);
- ssi_hash_free(drvdata);
- ssi_ablkcipher_free(drvdata);
+ ssi_aead_free(drvdata);
+ ssi_hash_free(drvdata);
+ ssi_ablkcipher_free(drvdata);
ssi_ivgen_fini(drvdata);
ssi_power_mgr_fini(drvdata);
ssi_buffer_mgr_fini(drvdata);
ssi_sysfs_fini();
#endif
- /* Mask all interrupts */
- WRITE_REGISTER(drvdata->cc_base + CC_REG_OFFSET(HOST_RGF, HOST_IMR),
- 0xFFFFFFFF);
+ fini_cc_regs(drvdata);
+ cc_clk_off(drvdata);
free_irq(drvdata->res_irq->start, drvdata);
drvdata->res_irq = NULL;
- fini_cc_regs(drvdata);
-
- if (drvdata->cc_base != NULL) {
+ if (drvdata->cc_base) {
iounmap(drvdata->cc_base);
release_mem_region(drvdata->res_mem->start,
resource_size(drvdata->res_mem));
dev_set_drvdata(&plat_dev->dev, NULL);
}
+int cc_clk_on(struct ssi_drvdata *drvdata)
+{
+ struct clk *clk = drvdata->clk;
+ int rc;
+
+ if (IS_ERR(clk))
+ /* Not all devices have a clock associated with CCREE */
+ return 0;
+
+ rc = clk_prepare_enable(clk);
+ if (rc)
+ return rc;
+
+ return 0;
+}
+
+void cc_clk_off(struct ssi_drvdata *drvdata)
+{
+ struct clk *clk = drvdata->clk;
+
+ if (IS_ERR(clk))
+ /* Not all devices have a clock associated with CCREE */
+ return;
+
+ clk_disable_unprepare(clk);
+}
+
static int cc7x_probe(struct platform_device *plat_dev)
{
int rc;
#if defined(CONFIG_ARM) && defined(CC_DEBUG)
- uint32_t ctr, cacheline_size;
+ u32 ctr, cacheline_size;
asm volatile("mrc p15, 0, %0, c0, c0, 1" : "=r" (ctr));
cacheline_size = 4 << ((ctr >> 16) & 0xf);
asm volatile("mrc p15, 0, %0, c0, c0, 0" : "=r" (ctr));
SSI_LOG_DEBUG("Main ID register (MIDR): Implementer 0x%02X, Arch 0x%01X,"
" Part 0x%03X, Rev r%dp%d\n",
- (ctr>>24), (ctr>>16)&0xF, (ctr>>4)&0xFFF, (ctr>>20)&0xF, ctr&0xF);
+ (ctr >> 24), (ctr >> 16) & 0xF, (ctr >> 4) & 0xFFF, (ctr >> 20) & 0xF, ctr & 0xF);
#endif
/* Map registers space */
static int cc7x_remove(struct platform_device *plat_dev)
{
SSI_LOG_DEBUG("Releasing cc7x resources...\n");
-
+
cleanup_cc_resources(plat_dev);
SSI_LOG(KERN_INFO, "ARM cc7x_ree device terminated\n");
-#ifdef ENABLE_CYCLE_COUNT
- display_all_stat_db();
-#endif
-
+
return 0;
}
-#if defined (CONFIG_PM_RUNTIME) || defined (CONFIG_PM_SLEEP)
+
+#if defined(CONFIG_PM_RUNTIME) || defined(CONFIG_PM_SLEEP)
static struct dev_pm_ops arm_cc7x_driver_pm = {
SET_RUNTIME_PM_OPS(ssi_power_mgr_runtime_suspend, ssi_power_mgr_runtime_resume, NULL)
};
#endif
-#if defined (CONFIG_PM_RUNTIME) || defined (CONFIG_PM_SLEEP)
+#if defined(CONFIG_PM_RUNTIME) || defined(CONFIG_PM_SLEEP)
#define DX_DRIVER_RUNTIME_PM (&arm_cc7x_driver_pm)
#else
#define DX_DRIVER_RUNTIME_PM NULL
#endif
-
#ifdef CONFIG_OF
static const struct of_device_id arm_cc7x_dev_of_match[] = {
{.compatible = "arm,cryptocell-712-ree"},
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
/* \file ssi_driver.h
- ARM CryptoCell Linux Crypto Driver
+ * ARM CryptoCell Linux Crypto Driver
*/
#ifndef __SSI_DRIVER_H__
#include <crypto/authenc.h>
#include <crypto/hash.h>
#include <linux/version.h>
-
-#ifndef INT32_MAX /* Missing in Linux kernel */
-#define INT32_MAX 0x7FFFFFFFL
-#endif
+#include <linux/clk.h>
/* Registers definitions from shared/hw/ree_include */
#include "dx_reg_base_host.h"
#include "dx_host.h"
-#define DX_CC_HOST_VIRT /* must be defined before including dx_cc_regs.h */
-#include "cc_hw_queue_defs.h"
#include "cc_regs.h"
#include "dx_reg_common.h"
#include "cc_hal.h"
-#include "ssi_sram_mgr.h"
#define CC_SUPPORT_SHA DX_DEV_SHA_MAX
#include "cc_crypto_ctx.h"
#include "ssi_sysfs.h"
#include "hash_defs.h"
#include "ssi_fips_local.h"
+#include "cc_hw_queue_defs.h"
+#include "ssi_sram_mgr.h"
#define DRV_MODULE_VERSION "3.0"
#define SSI_DEV_NAME_STR "cc715ree"
+#define CC_COHERENT_CACHE_PARAMS 0xEEE
+
#define SSI_CC_HAS_AES_CCM 1
#define SSI_CC_HAS_AES_GCM 1
#define SSI_CC_HAS_AES_XTS 1
/* Definitions for HW descriptors DIN/DOUT fields */
#define NS_BIT 1
#define AXI_ID 0
-/* AXI_ID is not actually the AXI ID of the transaction but the value of AXI_ID
- field in the HW descriptor. The DMA engine +8 that value. */
+/* AXI_ID is not actually the AXI ID of the transaction but the value of AXI_ID
+ * field in the HW descriptor. The DMA engine +8 that value.
+ */
/* Logging macros */
#define SSI_LOG(level, format, ...) \
- printk(level "cc715ree::%s: " format , __func__, ##__VA_ARGS__)
+ printk(level "cc715ree::%s: " format, __func__, ##__VA_ARGS__)
#define SSI_LOG_ERR(format, ...) SSI_LOG(KERN_ERR, format, ##__VA_ARGS__)
#define SSI_LOG_WARNING(format, ...) SSI_LOG(KERN_WARNING, format, ##__VA_ARGS__)
#define SSI_LOG_NOTICE(format, ...) SSI_LOG(KERN_NOTICE, format, ##__VA_ARGS__)
#define MIN(a, b) (((a) < (b)) ? (a) : (b))
#define MAX(a, b) (((a) > (b)) ? (a) : (b))
-#define SSI_MAX_IVGEN_DMA_ADDRESSES 3
+#define SSI_MAX_IVGEN_DMA_ADDRESSES 3
struct ssi_crypto_req {
void (*user_cb)(struct device *dev, void *req, void __iomem *cc_base);
void *user_arg;
- dma_addr_t ivgen_dma_addr[SSI_MAX_IVGEN_DMA_ADDRESSES]; /* For the first 'ivgen_dma_addr_len' addresses of this array,
- generated IV would be placed in it by send_request().
- Same generated IV for all addresses! */
+ dma_addr_t ivgen_dma_addr[SSI_MAX_IVGEN_DMA_ADDRESSES];
+ /* For the first 'ivgen_dma_addr_len' addresses of this array,
+ * generated IV would be placed in it by send_request().
+ * Same generated IV for all addresses!
+ */
unsigned int ivgen_dma_addr_len; /* Amount of 'ivgen_dma_addr' elements to be filled. */
unsigned int ivgen_size; /* The generated IV size required, 8/16 B allowed. */
struct completion seq_compl; /* request completion */
-#ifdef ENABLE_CYCLE_COUNT
- enum stat_op op_type;
- cycles_t submit_cycle;
- bool is_monitored_p;
-#endif
};
/**
struct resource *res_mem;
struct resource *res_irq;
void __iomem *cc_base;
-#ifdef DX_BASE_ENV_REGS
- void __iomem *env_base; /* ARM CryptoCell development FPGAs only */
-#endif
unsigned int irq;
- uint32_t irq_mask;
- uint32_t fw_ver;
+ u32 irq_mask;
+ u32 fw_ver;
/* Calibration time of start/stop
- * monitor descriptors */
- uint32_t monitor_null_cycles;
+ * monitor descriptors
+ */
+ u32 monitor_null_cycles;
struct platform_device *plat_dev;
ssi_sram_addr_t mlli_sram_addr;
struct completion icache_setup_completion;
void *fips_handle;
void *ivgen_handle;
void *sram_mgr_handle;
-
-#ifdef ENABLE_CYCLE_COUNT
- cycles_t isr_exit_cycles; /* Save for isr-to-tasklet latency */
-#endif
- uint32_t inflight_counter;
-
+ u32 inflight_counter;
+ struct clk *clk;
+ bool coherent;
};
struct ssi_crypto_alg {
int cipher_mode;
int flow_mode; /* Note: currently, refers to the cipher mode only. */
int auth_mode;
- bool synchronous;
struct ssi_drvdata *drvdata;
};
};
#ifdef DX_DUMP_BYTES
-void dump_byte_array(const char *name, const uint8_t *the_array, unsigned long size);
+void dump_byte_array(const char *name, const u8 *the_array, unsigned long size);
#else
#define dump_byte_array(name, array, size) do { \
} while (0);
#endif
-#ifdef ENABLE_CYCLE_COUNT
-#define DECL_CYCLE_COUNT_RESOURCES cycles_t _last_cycles_read
-#define START_CYCLE_COUNT() do { _last_cycles_read = get_cycles(); } while (0)
-#define END_CYCLE_COUNT(_stat_op_type, _stat_phase) update_host_stat(_stat_op_type, _stat_phase, get_cycles() - _last_cycles_read)
-#define GET_START_CYCLE_COUNT() _last_cycles_read
-#define START_CYCLE_COUNT_AT(_var) do { _var = get_cycles(); } while(0)
-#define END_CYCLE_COUNT_AT(_var, _stat_op_type, _stat_phase) update_host_stat(_stat_op_type, _stat_phase, get_cycles() - _var)
-#else
-#define DECL_CYCLE_COUNT_RESOURCES
-#define START_CYCLE_COUNT() do { } while (0)
-#define END_CYCLE_COUNT(_stat_op_type, _stat_phase) do { } while (0)
-#define GET_START_CYCLE_COUNT() 0
-#define START_CYCLE_COUNT_AT(_var) do { } while (0)
-#define END_CYCLE_COUNT_AT(_var, _stat_op_type, _stat_phase) do { } while (0)
-#endif /*ENABLE_CYCLE_COUNT*/
-
int init_cc_regs(struct ssi_drvdata *drvdata, bool is_probe);
void fini_cc_regs(struct ssi_drvdata *drvdata);
+int cc_clk_on(struct ssi_drvdata *drvdata);
+void cc_clk_off(struct ssi_drvdata *drvdata);
#endif /*__SSI_DRIVER_H__*/
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
-
/**************************************************************
-This file defines the driver FIPS APIs *
-***************************************************************/
+ * This file defines the driver FIPS APIs *
+ **************************************************************/
#include <linux/module.h>
#include "ssi_fips.h"
-
-extern int ssi_fips_ext_get_state(ssi_fips_state_t *p_state);
-extern int ssi_fips_ext_get_error(ssi_fips_error_t *p_err);
+extern int ssi_fips_ext_get_state(enum cc_fips_state_t *p_state);
+extern int ssi_fips_ext_get_error(enum cc_fips_error *p_err);
/*
-This function returns the REE FIPS state.
-It should be called by kernel module.
-*/
-int ssi_fips_get_state(ssi_fips_state_t *p_state)
+ * This function returns the REE FIPS state.
+ * It should be called by kernel module.
+ */
+int ssi_fips_get_state(enum cc_fips_state_t *p_state)
{
- int rc = 0;
+ int rc = 0;
- if (p_state == NULL) {
+ if (!p_state)
return -EINVAL;
- }
rc = ssi_fips_ext_get_state(p_state);
EXPORT_SYMBOL(ssi_fips_get_state);
/*
-This function returns the REE FIPS error.
-It should be called by kernel module.
-*/
-int ssi_fips_get_error(ssi_fips_error_t *p_err)
+ * This function returns the REE FIPS error.
+ * It should be called by kernel module.
+ */
+int ssi_fips_get_error(enum cc_fips_error *p_err)
{
- int rc = 0;
+ int rc = 0;
- if (p_err == NULL) {
+ if (!p_err)
return -EINVAL;
- }
rc = ssi_fips_ext_get_error(p_err);
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
#ifndef __SSI_FIPS_H__
#define __SSI_FIPS_H__
+/*!
+ * @file
+ * @brief This file contains FIPS related defintions and APIs.
+ */
-#ifndef INT32_MAX /* Missing in Linux kernel */
-#define INT32_MAX 0x7FFFFFFFL
-#endif
-
-
-/*!
-@file
-@brief This file contains FIPS related defintions and APIs.
-*/
-
-typedef enum ssi_fips_state {
- CC_FIPS_STATE_NOT_SUPPORTED = 0,
- CC_FIPS_STATE_SUPPORTED,
- CC_FIPS_STATE_ERROR,
- CC_FIPS_STATE_RESERVE32B = INT32_MAX
-} ssi_fips_state_t;
-
+enum cc_fips_state {
+ CC_FIPS_STATE_NOT_SUPPORTED = 0,
+ CC_FIPS_STATE_SUPPORTED,
+ CC_FIPS_STATE_ERROR,
+ CC_FIPS_STATE_RESERVE32B = S32_MAX
+};
-typedef enum ssi_fips_error {
+enum cc_fips_error {
CC_REE_FIPS_ERROR_OK = 0,
CC_REE_FIPS_ERROR_GENERAL,
CC_REE_FIPS_ERROR_FROM_TEE,
CC_REE_FIPS_ERROR_HMAC_SHA256_PUT,
CC_REE_FIPS_ERROR_HMAC_SHA512_PUT,
CC_REE_FIPS_ERROR_ROM_CHECKSUM,
- CC_REE_FIPS_ERROR_RESERVE32B = INT32_MAX
-} ssi_fips_error_t;
-
-
+ CC_REE_FIPS_ERROR_RESERVE32B = S32_MAX
+};
-int ssi_fips_get_state(ssi_fips_state_t *p_state);
-int ssi_fips_get_error(ssi_fips_error_t *p_err);
+int ssi_fips_get_state(enum cc_fips_state *p_state);
+int ssi_fips_get_error(enum cc_fips_error *p_err);
#endif /*__SSI_FIPS_H__*/
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
/*
-The test vectors were taken from:
-
-* AES
-NIST Special Publication 800-38A 2001 Edition
-Recommendation for Block Cipher Modes of Operation
-http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf
-Appendix F: Example Vectors for Modes of Operation of the AES
-
-* AES CTS
-Advanced Encryption Standard (AES) Encryption for Kerberos 5
-February 2005
-https://tools.ietf.org/html/rfc3962#appendix-B
-B. Sample Test Vectors
-
-* AES XTS
-http://csrc.nist.gov/groups/STM/cavp/#08
-http://csrc.nist.gov/groups/STM/cavp/documents/aes/XTSTestVectors.zip
-
-* AES CMAC
-http://csrc.nist.gov/groups/STM/cavp/index.html#07
-http://csrc.nist.gov/groups/STM/cavp/documents/mac/cmactestvectors.zip
-
-* AES-CCM
-http://csrc.nist.gov/groups/STM/cavp/#07
-http://csrc.nist.gov/groups/STM/cavp/documents/mac/ccmtestvectors.zip
-
-* AES-GCM
-http://csrc.nist.gov/groups/STM/cavp/documents/mac/gcmtestvectors.zip
-
-* Triple-DES
-NIST Special Publication 800-67 January 2012
-Recommendation for the Triple Data Encryption Algorithm (TDEA) Block Cipher
-http://csrc.nist.gov/publications/nistpubs/800-67-Rev1/SP-800-67-Rev1.pdf
-APPENDIX B: EXAMPLE OF TDEA FORWARD AND INVERSE CIPHER OPERATIONS
-and
-http://csrc.nist.gov/groups/STM/cavp/#01
-http://csrc.nist.gov/groups/STM/cavp/documents/des/tdesmct_intermediate.zip
-
-* HASH
-http://csrc.nist.gov/groups/STM/cavp/#03
-http://csrc.nist.gov/groups/STM/cavp/documents/shs/shabytetestvectors.zip
-
-* HMAC
-http://csrc.nist.gov/groups/STM/cavp/#07
-http://csrc.nist.gov/groups/STM/cavp/documents/mac/hmactestvectors.zip
-
-*/
+ * The test vectors were taken from:
+ *
+ * * AES
+ * NIST Special Publication 800-38A 2001 Edition
+ * Recommendation for Block Cipher Modes of Operation
+ * http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf
+ * Appendix F: Example Vectors for Modes of Operation of the AES
+ *
+ * * AES CTS
+ * Advanced Encryption Standard (AES) Encryption for Kerberos 5
+ * February 2005
+ * https://tools.ietf.org/html/rfc3962#appendix-B
+ * B. Sample Test Vectors
+ *
+ * * AES XTS
+ * http://csrc.nist.gov/groups/STM/cavp/#08
+ * http://csrc.nist.gov/groups/STM/cavp/documents/aes/XTSTestVectors.zip
+ *
+ * * AES CMAC
+ * http://csrc.nist.gov/groups/STM/cavp/index.html#07
+ * http://csrc.nist.gov/groups/STM/cavp/documents/mac/cmactestvectors.zip
+ *
+ * * AES-CCM
+ * http://csrc.nist.gov/groups/STM/cavp/#07
+ * http://csrc.nist.gov/groups/STM/cavp/documents/mac/ccmtestvectors.zip
+ *
+ * * AES-GCM
+ * http://csrc.nist.gov/groups/STM/cavp/documents/mac/gcmtestvectors.zip
+ *
+ * * Triple-DES
+ * NIST Special Publication 800-67 January 2012
+ * Recommendation for the Triple Data Encryption Algorithm (TDEA) Block Cipher
+ * http://csrc.nist.gov/publications/nistpubs/800-67-Rev1/SP-800-67-Rev1.pdf
+ * APPENDIX B: EXAMPLE OF TDEA FORWARD AND INVERSE CIPHER OPERATIONS
+ * and
+ * http://csrc.nist.gov/groups/STM/cavp/#01
+ * http://csrc.nist.gov/groups/STM/cavp/documents/des/tdesmct_intermediate.zip
+ *
+ * * HASH
+ * http://csrc.nist.gov/groups/STM/cavp/#03
+ * http://csrc.nist.gov/groups/STM/cavp/documents/shs/shabytetestvectors.zip
+ *
+ * * HMAC
+ * http://csrc.nist.gov/groups/STM/cavp/#07
+ * http://csrc.nist.gov/groups/STM/cavp/documents/mac/hmactestvectors.zip
+ */
/* NIST AES */
#define AES_128_BIT_KEY_SIZE 16
#define NIST_AES_CBC_IV { 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f }
#define NIST_AES_128_CBC_CIPHER { 0x76, 0x49, 0xab, 0xac, 0x81, 0x19, 0xb2, 0x46, 0xce, 0xe9, 0x8e, 0x9b, 0x12, 0xe9, 0x19, 0x7d }
-#define NIST_AES_192_CBC_CIPHER { 0x4f, 0x02, 0x1d, 0xb2, 0x43, 0xbc, 0x63, 0x3d, 0x71, 0x78, 0x18, 0x3a, 0x9f, 0xa0, 0x71, 0xe8 }
-#define NIST_AES_256_CBC_CIPHER { 0xf5, 0x8c, 0x4c, 0x04, 0xd6, 0xe5, 0xf1, 0xba, 0x77, 0x9e, 0xab, 0xfb, 0x5f, 0x7b, 0xfb, 0xd6 }
+#define NIST_AES_192_CBC_CIPHER { 0x4f, 0x02, 0x1d, 0xb2, 0x43, 0xbc, 0x63, 0x3d, 0x71, 0x78, 0x18, 0x3a, 0x9f, 0xa0, 0x71, 0xe8 }
+#define NIST_AES_256_CBC_CIPHER { 0xf5, 0x8c, 0x4c, 0x04, 0xd6, 0xe5, 0xf1, 0xba, 0x77, 0x9e, 0xab, 0xfb, 0x5f, 0x7b, 0xfb, 0xd6 }
#define NIST_AES_OFB_IV { 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f }
#define NIST_AES_128_OFB_CIPHER { 0x3b, 0x3f, 0xd9, 0x2e, 0xb7, 0x2d, 0xad, 0x20, 0x33, 0x34, 0x49, 0xf8, 0xe8, 0x3c, 0xfb, 0x4a }
-#define NIST_AES_192_OFB_CIPHER { 0xcd, 0xc8, 0x0d, 0x6f, 0xdd, 0xf1, 0x8c, 0xab, 0x34, 0xc2, 0x59, 0x09, 0xc9, 0x9a, 0x41, 0x74 }
+#define NIST_AES_192_OFB_CIPHER { 0xcd, 0xc8, 0x0d, 0x6f, 0xdd, 0xf1, 0x8c, 0xab, 0x34, 0xc2, 0x59, 0x09, 0xc9, 0x9a, 0x41, 0x74 }
#define NIST_AES_256_OFB_CIPHER { 0xdc, 0x7e, 0x84, 0xbf, 0xda, 0x79, 0x16, 0x4b, 0x7e, 0xcd, 0x84, 0x86, 0x98, 0x5d, 0x38, 0x60 }
#define NIST_AES_CTR_IV { 0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7, 0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff }
#define NIST_AES_128_CTR_CIPHER { 0x87, 0x4d, 0x61, 0x91, 0xb6, 0x20, 0xe3, 0x26, 0x1b, 0xef, 0x68, 0x64, 0x99, 0x0d, 0xb6, 0xce }
-#define NIST_AES_192_CTR_CIPHER { 0x1a, 0xbc, 0x93, 0x24, 0x17, 0x52, 0x1c, 0xa2, 0x4f, 0x2b, 0x04, 0x59, 0xfe, 0x7e, 0x6e, 0x0b }
-#define NIST_AES_256_CTR_CIPHER { 0x60, 0x1e, 0xc3, 0x13, 0x77, 0x57, 0x89, 0xa5, 0xb7, 0xa7, 0xf5, 0x04, 0xbb, 0xf3, 0xd2, 0x28 }
-
+#define NIST_AES_192_CTR_CIPHER { 0x1a, 0xbc, 0x93, 0x24, 0x17, 0x52, 0x1c, 0xa2, 0x4f, 0x2b, 0x04, 0x59, 0xfe, 0x7e, 0x6e, 0x0b }
+#define NIST_AES_256_CTR_CIPHER { 0x60, 0x1e, 0xc3, 0x13, 0x77, 0x57, 0x89, 0xa5, 0xb7, 0xa7, 0xf5, 0x04, 0xbb, 0xf3, 0xd2, 0x28 }
#define RFC3962_AES_128_KEY { 0x63, 0x68, 0x69, 0x63, 0x6b, 0x65, 0x6e, 0x20, 0x74, 0x65, 0x72, 0x69, 0x79, 0x61, 0x6b, 0x69 }
#define RFC3962_AES_VECTOR_SIZE 17
#define RFC3962_AES_CBC_CTS_IV { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 }
#define RFC3962_AES_128_CBC_CTS_CIPHER { 0xc6, 0x35, 0x35, 0x68, 0xf2, 0xbf, 0x8c, 0xb4, 0xd8, 0xa5, 0x80, 0x36, 0x2d, 0xa7, 0xff, 0x7f, 0x97 }
-
#define NIST_AES_256_XTS_KEY { 0xa1, 0xb9, 0x0c, 0xba, 0x3f, 0x06, 0xac, 0x35, 0x3b, 0x2c, 0x34, 0x38, 0x76, 0x08, 0x17, 0x62, \
0x09, 0x09, 0x23, 0x02, 0x6e, 0x91, 0x77, 0x18, 0x15, 0xf2, 0x9d, 0xab, 0x01, 0x93, 0x2f, 0x2f }
#define NIST_AES_256_XTS_IV { 0x4f, 0xae, 0xf7, 0x11, 0x7c, 0xda, 0x59, 0xc6, 0x6e, 0x4b, 0x92, 0x01, 0x3e, 0x76, 0x8a, 0xd5 }
#define NIST_AES_256_XTS_VECTOR_SIZE 16
-#define NIST_AES_256_XTS_PLAIN { 0xeb, 0xab, 0xce, 0x95, 0xb1, 0x4d, 0x3c, 0x8d, 0x6f, 0xb3, 0x50, 0x39, 0x07, 0x90, 0x31, 0x1c }
-#define NIST_AES_256_XTS_CIPHER { 0x77, 0x8a, 0xe8, 0xb4, 0x3c, 0xb9, 0x8d, 0x5a, 0x82, 0x50, 0x81, 0xd5, 0xbe, 0x47, 0x1c, 0x63 }
+#define NIST_AES_256_XTS_PLAIN { 0xeb, 0xab, 0xce, 0x95, 0xb1, 0x4d, 0x3c, 0x8d, 0x6f, 0xb3, 0x50, 0x39, 0x07, 0x90, 0x31, 0x1c }
+#define NIST_AES_256_XTS_CIPHER { 0x77, 0x8a, 0xe8, 0xb4, 0x3c, 0xb9, 0x8d, 0x5a, 0x82, 0x50, 0x81, 0xd5, 0xbe, 0x47, 0x1c, 0x63 }
#define NIST_AES_512_XTS_KEY { 0x1e, 0xa6, 0x61, 0xc5, 0x8d, 0x94, 0x3a, 0x0e, 0x48, 0x01, 0xe4, 0x2f, 0x4b, 0x09, 0x47, 0x14, \
0x9e, 0x7f, 0x9f, 0x8e, 0x3e, 0x68, 0xd0, 0xc7, 0x50, 0x52, 0x10, 0xbd, 0x31, 0x1a, 0x0e, 0x7c, \
#define NIST_AES_512_XTS_IV { 0xad, 0xf8, 0xd9, 0x26, 0x27, 0x46, 0x4a, 0xd2, 0xf0, 0x42, 0x8e, 0x84, 0xa9, 0xf8, 0x75, 0x64, }
#define NIST_AES_512_XTS_VECTOR_SIZE 32
#define NIST_AES_512_XTS_PLAIN { 0x2e, 0xed, 0xea, 0x52, 0xcd, 0x82, 0x15, 0xe1, 0xac, 0xc6, 0x47, 0xe8, 0x10, 0xbb, 0xc3, 0x64, \
- 0x2e, 0x87, 0x28, 0x7f, 0x8d, 0x2e, 0x57, 0xe3, 0x6c, 0x0a, 0x24, 0xfb, 0xc1, 0x2a, 0x20, 0x2e }
+ 0x2e, 0x87, 0x28, 0x7f, 0x8d, 0x2e, 0x57, 0xe3, 0x6c, 0x0a, 0x24, 0xfb, 0xc1, 0x2a, 0x20, 0x2e }
#define NIST_AES_512_XTS_CIPHER { 0xcb, 0xaa, 0xd0, 0xe2, 0xf6, 0xce, 0xa3, 0xf5, 0x0b, 0x37, 0xf9, 0x34, 0xd4, 0x6a, 0x9b, 0x13, \
- 0x0b, 0x9d, 0x54, 0xf0, 0x7e, 0x34, 0xf3, 0x6a, 0xf7, 0x93, 0xe8, 0x6f, 0x73, 0xc6, 0xd7, 0xdb }
-
+ 0x0b, 0x9d, 0x54, 0xf0, 0x7e, 0x34, 0xf3, 0x6a, 0xf7, 0x93, 0xe8, 0x6f, 0x73, 0xc6, 0xd7, 0xdb }
/* NIST AES-CMAC */
#define NIST_AES_128_CMAC_KEY { 0x67, 0x08, 0xc9, 0x88, 0x7b, 0x84, 0x70, 0x84, 0xf1, 0x23, 0xd3, 0xdd, 0x9c, 0x3a, 0x81, 0x36 }
#define NIST_AES_256_CMAC_VECTOR_SIZE 16
#define NIST_AES_256_CMAC_OUTPUT_SIZE 10
-
/* NIST TDES */
#define TDES_NUM_OF_KEYS 3
#define NIST_TDES_VECTOR_SIZE 8
#define NIST_TDES_IV_SIZE 8
-#define NIST_TDES_ECB_IV { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 }
+#define NIST_TDES_ECB_IV { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 }
#define NIST_TDES_ECB3_KEY { 0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef, \
0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef, 0x01, \
0x45, 0x67, 0x89, 0xab, 0xcd, 0xef, 0x01, 0x23 }
-#define NIST_TDES_ECB3_PLAIN_DATA { 0x54, 0x68, 0x65, 0x20, 0x71, 0x75, 0x66, 0x63 }
-#define NIST_TDES_ECB3_CIPHER { 0xa8, 0x26, 0xfd, 0x8c, 0xe5, 0x3b, 0x85, 0x5f }
+#define NIST_TDES_ECB3_PLAIN_DATA { 0x54, 0x68, 0x65, 0x20, 0x71, 0x75, 0x66, 0x63 }
+#define NIST_TDES_ECB3_CIPHER { 0xa8, 0x26, 0xfd, 0x8c, 0xe5, 0x3b, 0x85, 0x5f }
-#define NIST_TDES_CBC3_IV { 0xf8, 0xee, 0xe1, 0x35, 0x9c, 0x6e, 0x54, 0x40 }
+#define NIST_TDES_CBC3_IV { 0xf8, 0xee, 0xe1, 0x35, 0x9c, 0x6e, 0x54, 0x40 }
#define NIST_TDES_CBC3_KEY { 0xe9, 0xda, 0x37, 0xf8, 0xdc, 0x97, 0x6d, 0x5b, \
0xb6, 0x8c, 0x04, 0xe3, 0xec, 0x98, 0x20, 0x15, \
0xf4, 0x0e, 0x08, 0xb5, 0x97, 0x29, 0xf2, 0x8f }
-#define NIST_TDES_CBC3_PLAIN_DATA { 0x3b, 0xb7, 0xa7, 0xdb, 0xa3, 0xd5, 0x92, 0x91 }
-#define NIST_TDES_CBC3_CIPHER { 0x5b, 0x84, 0x24, 0xd2, 0x39, 0x3e, 0x55, 0xa2 }
-
+#define NIST_TDES_CBC3_PLAIN_DATA { 0x3b, 0xb7, 0xa7, 0xdb, 0xa3, 0xd5, 0x92, 0x91 }
+#define NIST_TDES_CBC3_CIPHER { 0x5b, 0x84, 0x24, 0xd2, 0x39, 0x3e, 0x55, 0xa2 }
/* NIST AES-CCM */
#define NIST_AESCCM_128_BIT_KEY_SIZE 16
#define NIST_AESCCM_256_CIPHER { 0xcc, 0x17, 0xbf, 0x87, 0x94, 0xc8, 0x43, 0x45, 0x7d, 0x89, 0x93, 0x91, 0x89, 0x8e, 0xd2, 0x2a }
#define NIST_AESCCM_256_MAC { 0x6f, 0x9d, 0x28, 0xfc, 0xb6, 0x42, 0x34, 0xe1, 0xcd, 0x79, 0x3c, 0x41, 0x44, 0xf1, 0xda, 0x50 }
-
/* NIST AES-GCM */
#define NIST_AESGCM_128_BIT_KEY_SIZE 16
#define NIST_AESGCM_192_BIT_KEY_SIZE 24
#define NIST_AESGCM_256_CIPHER { 0x42, 0x6e, 0x0e, 0xfc, 0x69, 0x3b, 0x7b, 0xe1, 0xf3, 0x01, 0x8d, 0xb7, 0xdd, 0xbb, 0x7e, 0x4d }
#define NIST_AESGCM_256_MAC { 0xee, 0x82, 0x57, 0x79, 0x5b, 0xe6, 0xa1, 0x16, 0x4d, 0x7e, 0x1d, 0x2d, 0x6c, 0xac, 0x77, 0xa7 }
-
/* NIST HASH */
#define NIST_SHA_MSG_SIZE 16
0x8f, 0x2b, 0xa9, 0x1c, 0x3a, 0x9f, 0x0c, 0x16, 0x53, 0xc4, 0xbf, 0x0a, 0xda, 0x35, 0x64, 0x55, \
0xea, 0x36, 0xfd, 0x31, 0xf8, 0xe7, 0x3e, 0x39, 0x51, 0xca, 0xd4, 0xeb, 0xba, 0x8c, 0x6e, 0x04 }
-
/* NIST HMAC */
#define NIST_HMAC_MSG_SIZE 128
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
/**************************************************************
-This file defines the driver FIPS functions that should be
-implemented by the driver user. Current implementation is sample code only.
-***************************************************************/
+ * This file defines the driver FIPS functions that should be
+ * implemented by the driver user. Current implementation is sample code only.
+ ***************************************************************/
#include <linux/module.h>
#include "ssi_fips_local.h"
#include "ssi_driver.h"
-
static bool tee_error;
module_param(tee_error, bool, 0644);
MODULE_PARM_DESC(tee_error, "Simulate TEE library failure flag: 0 - no error (default), 1 - TEE error occured ");
-static ssi_fips_state_t fips_state = CC_FIPS_STATE_NOT_SUPPORTED;
-static ssi_fips_error_t fips_error = CC_REE_FIPS_ERROR_OK;
+static enum cc_fips_state_t fips_state = CC_FIPS_STATE_NOT_SUPPORTED;
+static enum cc_fips_error fips_error = CC_REE_FIPS_ERROR_OK;
/*
-This function returns the FIPS REE state.
-The function should be implemented by the driver user, depends on where .
-the state value is stored.
-The reference code uses global variable.
-*/
-int ssi_fips_ext_get_state(ssi_fips_state_t *p_state)
+ * This function returns the FIPS REE state.
+ * The function should be implemented by the driver user, depends on where
+ * the state value is stored.
+ * The reference code uses global variable.
+ */
+int ssi_fips_ext_get_state(enum cc_fips_state_t *p_state)
{
- int rc = 0;
+ int rc = 0;
- if (p_state == NULL) {
+ if (!p_state)
return -EINVAL;
- }
*p_state = fips_state;
}
/*
-This function returns the FIPS REE error.
-The function should be implemented by the driver user, depends on where .
-the error value is stored.
-The reference code uses global variable.
-*/
-int ssi_fips_ext_get_error(ssi_fips_error_t *p_err)
+ * This function returns the FIPS REE error.
+ * The function should be implemented by the driver user, depends on where
+ * the error value is stored.
+ * The reference code uses global variable.
+ */
+int ssi_fips_ext_get_error(enum cc_fips_error *p_err)
{
- int rc = 0;
+ int rc = 0;
- if (p_err == NULL) {
+ if (!p_err)
return -EINVAL;
- }
*p_err = fips_error;
}
/*
-This function sets the FIPS REE state.
-The function should be implemented by the driver user, depends on where .
-the state value is stored.
-The reference code uses global variable.
-*/
-int ssi_fips_ext_set_state(ssi_fips_state_t state)
+ * This function sets the FIPS REE state.
+ * The function should be implemented by the driver user, depends on where
+ * the state value is stored.
+ * The reference code uses global variable.
+ */
+int ssi_fips_ext_set_state(enum cc_fips_state_t state)
{
fips_state = state;
return 0;
}
/*
-This function sets the FIPS REE error.
-The function should be implemented by the driver user, depends on where .
-the error value is stored.
-The reference code uses global variable.
-*/
-int ssi_fips_ext_set_error(ssi_fips_error_t err)
+ * This function sets the FIPS REE error.
+ * The function should be implemented by the driver user, depends on where
+ * the error value is stored.
+ * The reference code uses global variable.
+ */
+int ssi_fips_ext_set_error(enum cc_fips_error err)
{
fips_error = err;
return 0;
}
-
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
/**************************************************************
-This file defines the driver FIPS Low Level implmentaion functions,
-that executes the KAT.
-***************************************************************/
+ * This file defines the driver FIPS Low Level implmentaion functions,
+ * that executes the KAT.
+ ***************************************************************/
#include <linux/kernel.h>
#include "ssi_driver.h"
#include "ssi_hash.h"
#include "ssi_request_mgr.h"
-
-static const uint32_t digest_len_init[] = {
+static const u32 digest_len_init[] = {
0x00000040, 0x00000000, 0x00000000, 0x00000000 };
-static const uint32_t sha1_init[] = {
+static const u32 sha1_init[] = {
SHA1_H4, SHA1_H3, SHA1_H2, SHA1_H1, SHA1_H0 };
-static const uint32_t sha256_init[] = {
+static const u32 sha256_init[] = {
SHA256_H7, SHA256_H6, SHA256_H5, SHA256_H4,
SHA256_H3, SHA256_H2, SHA256_H1, SHA256_H0 };
#if (CC_SUPPORT_SHA > 256)
-static const uint32_t digest_len_sha512_init[] = {
+static const u32 digest_len_sha512_init[] = {
0x00000080, 0x00000000, 0x00000000, 0x00000000 };
-static const uint64_t sha512_init[] = {
+static const u64 sha512_init[] = {
SHA512_H7, SHA512_H6, SHA512_H5, SHA512_H4,
SHA512_H3, SHA512_H2, SHA512_H1, SHA512_H0 };
#endif
-
#define NIST_CIPHER_AES_MAX_VECTOR_SIZE 32
struct fips_cipher_ctx {
- uint8_t iv[CC_AES_IV_SIZE];
- uint8_t key[AES_512_BIT_KEY_SIZE];
- uint8_t din[NIST_CIPHER_AES_MAX_VECTOR_SIZE];
- uint8_t dout[NIST_CIPHER_AES_MAX_VECTOR_SIZE];
+ u8 iv[CC_AES_IV_SIZE];
+ u8 key[AES_512_BIT_KEY_SIZE];
+ u8 din[NIST_CIPHER_AES_MAX_VECTOR_SIZE];
+ u8 dout[NIST_CIPHER_AES_MAX_VECTOR_SIZE];
};
typedef struct _FipsCipherData {
- uint8_t isAes;
- uint8_t key[AES_512_BIT_KEY_SIZE];
+ u8 isAes;
+ u8 key[AES_512_BIT_KEY_SIZE];
size_t keySize;
- uint8_t iv[CC_AES_IV_SIZE];
+ u8 iv[CC_AES_IV_SIZE];
enum drv_crypto_direction direction;
enum drv_cipher_mode oprMode;
- uint8_t dataIn[NIST_CIPHER_AES_MAX_VECTOR_SIZE];
- uint8_t dataOut[NIST_CIPHER_AES_MAX_VECTOR_SIZE];
+ u8 dataIn[NIST_CIPHER_AES_MAX_VECTOR_SIZE];
+ u8 dataOut[NIST_CIPHER_AES_MAX_VECTOR_SIZE];
size_t dataInSize;
} FipsCipherData;
-
struct fips_cmac_ctx {
- uint8_t key[AES_256_BIT_KEY_SIZE];
- uint8_t din[NIST_CIPHER_AES_MAX_VECTOR_SIZE];
- uint8_t mac_res[CC_DIGEST_SIZE_MAX];
+ u8 key[AES_256_BIT_KEY_SIZE];
+ u8 din[NIST_CIPHER_AES_MAX_VECTOR_SIZE];
+ u8 mac_res[CC_DIGEST_SIZE_MAX];
};
typedef struct _FipsCmacData {
enum drv_crypto_direction direction;
- uint8_t key[AES_256_BIT_KEY_SIZE];
+ u8 key[AES_256_BIT_KEY_SIZE];
size_t key_size;
- uint8_t data_in[NIST_CIPHER_AES_MAX_VECTOR_SIZE];
+ u8 data_in[NIST_CIPHER_AES_MAX_VECTOR_SIZE];
size_t data_in_size;
- uint8_t mac_res[CC_DIGEST_SIZE_MAX];
+ u8 mac_res[CC_DIGEST_SIZE_MAX];
size_t mac_res_size;
} FipsCmacData;
-
struct fips_hash_ctx {
- uint8_t initial_digest[CC_DIGEST_SIZE_MAX];
- uint8_t din[NIST_SHA_MSG_SIZE];
- uint8_t mac_res[CC_DIGEST_SIZE_MAX];
+ u8 initial_digest[CC_DIGEST_SIZE_MAX];
+ u8 din[NIST_SHA_MSG_SIZE];
+ u8 mac_res[CC_DIGEST_SIZE_MAX];
};
typedef struct _FipsHashData {
enum drv_hash_mode hash_mode;
- uint8_t data_in[NIST_SHA_MSG_SIZE];
+ u8 data_in[NIST_SHA_MSG_SIZE];
size_t data_in_size;
- uint8_t mac_res[CC_DIGEST_SIZE_MAX];
+ u8 mac_res[CC_DIGEST_SIZE_MAX];
} FipsHashData;
-
/* note that the hmac key length must be equal or less than block size (block size is 64 up to sha256 and 128 for sha384/512) */
struct fips_hmac_ctx {
- uint8_t initial_digest[CC_DIGEST_SIZE_MAX];
- uint8_t key[CC_HMAC_BLOCK_SIZE_MAX];
- uint8_t k0[CC_HMAC_BLOCK_SIZE_MAX];
- uint8_t digest_bytes_len[HASH_LEN_SIZE];
- uint8_t tmp_digest[CC_DIGEST_SIZE_MAX];
- uint8_t din[NIST_HMAC_MSG_SIZE];
- uint8_t mac_res[CC_DIGEST_SIZE_MAX];
+ u8 initial_digest[CC_DIGEST_SIZE_MAX];
+ u8 key[CC_HMAC_BLOCK_SIZE_MAX];
+ u8 k0[CC_HMAC_BLOCK_SIZE_MAX];
+ u8 digest_bytes_len[HASH_LEN_SIZE];
+ u8 tmp_digest[CC_DIGEST_SIZE_MAX];
+ u8 din[NIST_HMAC_MSG_SIZE];
+ u8 mac_res[CC_DIGEST_SIZE_MAX];
};
typedef struct _FipsHmacData {
enum drv_hash_mode hash_mode;
- uint8_t key[CC_HMAC_BLOCK_SIZE_MAX];
+ u8 key[CC_HMAC_BLOCK_SIZE_MAX];
size_t key_size;
- uint8_t data_in[NIST_HMAC_MSG_SIZE];
+ u8 data_in[NIST_HMAC_MSG_SIZE];
size_t data_in_size;
- uint8_t mac_res[CC_DIGEST_SIZE_MAX];
+ u8 mac_res[CC_DIGEST_SIZE_MAX];
} FipsHmacData;
-
#define FIPS_CCM_B0_A0_ADATA_SIZE (NIST_AESCCM_IV_SIZE + NIST_AESCCM_IV_SIZE + NIST_AESCCM_ADATA_SIZE)
struct fips_ccm_ctx {
- uint8_t b0_a0_adata[FIPS_CCM_B0_A0_ADATA_SIZE];
- uint8_t iv[NIST_AESCCM_IV_SIZE];
- uint8_t ctr_cnt_0[NIST_AESCCM_IV_SIZE];
- uint8_t key[CC_AES_KEY_SIZE_MAX];
- uint8_t din[NIST_AESCCM_TEXT_SIZE];
- uint8_t dout[NIST_AESCCM_TEXT_SIZE];
- uint8_t mac_res[NIST_AESCCM_TAG_SIZE];
+ u8 b0_a0_adata[FIPS_CCM_B0_A0_ADATA_SIZE];
+ u8 iv[NIST_AESCCM_IV_SIZE];
+ u8 ctr_cnt_0[NIST_AESCCM_IV_SIZE];
+ u8 key[CC_AES_KEY_SIZE_MAX];
+ u8 din[NIST_AESCCM_TEXT_SIZE];
+ u8 dout[NIST_AESCCM_TEXT_SIZE];
+ u8 mac_res[NIST_AESCCM_TAG_SIZE];
};
typedef struct _FipsCcmData {
enum drv_crypto_direction direction;
- uint8_t key[CC_AES_KEY_SIZE_MAX];
+ u8 key[CC_AES_KEY_SIZE_MAX];
size_t keySize;
- uint8_t nonce[NIST_AESCCM_NONCE_SIZE];
- uint8_t adata[NIST_AESCCM_ADATA_SIZE];
+ u8 nonce[NIST_AESCCM_NONCE_SIZE];
+ u8 adata[NIST_AESCCM_ADATA_SIZE];
size_t adataSize;
- uint8_t dataIn[NIST_AESCCM_TEXT_SIZE];
+ u8 dataIn[NIST_AESCCM_TEXT_SIZE];
size_t dataInSize;
- uint8_t dataOut[NIST_AESCCM_TEXT_SIZE];
- uint8_t tagSize;
- uint8_t macResOut[NIST_AESCCM_TAG_SIZE];
+ u8 dataOut[NIST_AESCCM_TEXT_SIZE];
+ u8 tagSize;
+ u8 macResOut[NIST_AESCCM_TAG_SIZE];
} FipsCcmData;
-
struct fips_gcm_ctx {
- uint8_t adata[NIST_AESGCM_ADATA_SIZE];
- uint8_t key[CC_AES_KEY_SIZE_MAX];
- uint8_t hkey[CC_AES_KEY_SIZE_MAX];
- uint8_t din[NIST_AESGCM_TEXT_SIZE];
- uint8_t dout[NIST_AESGCM_TEXT_SIZE];
- uint8_t mac_res[NIST_AESGCM_TAG_SIZE];
- uint8_t len_block[AES_BLOCK_SIZE];
- uint8_t iv_inc1[AES_BLOCK_SIZE];
- uint8_t iv_inc2[AES_BLOCK_SIZE];
+ u8 adata[NIST_AESGCM_ADATA_SIZE];
+ u8 key[CC_AES_KEY_SIZE_MAX];
+ u8 hkey[CC_AES_KEY_SIZE_MAX];
+ u8 din[NIST_AESGCM_TEXT_SIZE];
+ u8 dout[NIST_AESGCM_TEXT_SIZE];
+ u8 mac_res[NIST_AESGCM_TAG_SIZE];
+ u8 len_block[AES_BLOCK_SIZE];
+ u8 iv_inc1[AES_BLOCK_SIZE];
+ u8 iv_inc2[AES_BLOCK_SIZE];
};
typedef struct _FipsGcmData {
enum drv_crypto_direction direction;
- uint8_t key[CC_AES_KEY_SIZE_MAX];
+ u8 key[CC_AES_KEY_SIZE_MAX];
size_t keySize;
- uint8_t iv[NIST_AESGCM_IV_SIZE];
- uint8_t adata[NIST_AESGCM_ADATA_SIZE];
+ u8 iv[NIST_AESGCM_IV_SIZE];
+ u8 adata[NIST_AESGCM_ADATA_SIZE];
size_t adataSize;
- uint8_t dataIn[NIST_AESGCM_TEXT_SIZE];
+ u8 dataIn[NIST_AESGCM_TEXT_SIZE];
size_t dataInSize;
- uint8_t dataOut[NIST_AESGCM_TEXT_SIZE];
- uint8_t tagSize;
- uint8_t macResOut[NIST_AESGCM_TAG_SIZE];
+ u8 dataOut[NIST_AESGCM_TEXT_SIZE];
+ u8 tagSize;
+ u8 macResOut[NIST_AESGCM_TAG_SIZE];
} FipsGcmData;
-
typedef union _fips_ctx {
struct fips_cipher_ctx cipher;
struct fips_cmac_ctx cmac;
struct fips_gcm_ctx gcm;
} fips_ctx;
-
/* test data tables */
static const FipsCipherData FipsCipherDataTable[] = {
/* AES */
{ 1, NIST_AES_256_XTS_KEY, CC_AES_256_BIT_KEY_SIZE, NIST_AES_256_XTS_IV, DRV_CRYPTO_DIRECTION_ENCRYPT, DRV_CIPHER_XTS, NIST_AES_256_XTS_PLAIN, NIST_AES_256_XTS_CIPHER, NIST_AES_256_XTS_VECTOR_SIZE },
{ 1, NIST_AES_256_XTS_KEY, CC_AES_256_BIT_KEY_SIZE, NIST_AES_256_XTS_IV, DRV_CRYPTO_DIRECTION_DECRYPT, DRV_CIPHER_XTS, NIST_AES_256_XTS_CIPHER, NIST_AES_256_XTS_PLAIN, NIST_AES_256_XTS_VECTOR_SIZE },
#if (CC_SUPPORT_SHA > 256)
- { 1, NIST_AES_512_XTS_KEY, 2*CC_AES_256_BIT_KEY_SIZE, NIST_AES_512_XTS_IV, DRV_CRYPTO_DIRECTION_ENCRYPT, DRV_CIPHER_XTS, NIST_AES_512_XTS_PLAIN, NIST_AES_512_XTS_CIPHER, NIST_AES_512_XTS_VECTOR_SIZE },
- { 1, NIST_AES_512_XTS_KEY, 2*CC_AES_256_BIT_KEY_SIZE, NIST_AES_512_XTS_IV, DRV_CRYPTO_DIRECTION_DECRYPT, DRV_CIPHER_XTS, NIST_AES_512_XTS_CIPHER, NIST_AES_512_XTS_PLAIN, NIST_AES_512_XTS_VECTOR_SIZE },
+ { 1, NIST_AES_512_XTS_KEY, 2 * CC_AES_256_BIT_KEY_SIZE, NIST_AES_512_XTS_IV, DRV_CRYPTO_DIRECTION_ENCRYPT, DRV_CIPHER_XTS, NIST_AES_512_XTS_PLAIN, NIST_AES_512_XTS_CIPHER, NIST_AES_512_XTS_VECTOR_SIZE },
+ { 1, NIST_AES_512_XTS_KEY, 2 * CC_AES_256_BIT_KEY_SIZE, NIST_AES_512_XTS_IV, DRV_CRYPTO_DIRECTION_DECRYPT, DRV_CIPHER_XTS, NIST_AES_512_XTS_CIPHER, NIST_AES_512_XTS_PLAIN, NIST_AES_512_XTS_VECTOR_SIZE },
#endif
/* DES */
{ 0, NIST_TDES_ECB3_KEY, CC_DRV_DES_TRIPLE_KEY_SIZE, NIST_TDES_ECB_IV, DRV_CRYPTO_DIRECTION_ENCRYPT, DRV_CIPHER_ECB, NIST_TDES_ECB3_PLAIN_DATA, NIST_TDES_ECB3_CIPHER, NIST_TDES_VECTOR_SIZE },
{ 0, NIST_TDES_CBC3_KEY, CC_DRV_DES_TRIPLE_KEY_SIZE, NIST_TDES_CBC3_IV, DRV_CRYPTO_DIRECTION_ENCRYPT, DRV_CIPHER_CBC, NIST_TDES_CBC3_PLAIN_DATA, NIST_TDES_CBC3_CIPHER, NIST_TDES_VECTOR_SIZE },
{ 0, NIST_TDES_CBC3_KEY, CC_DRV_DES_TRIPLE_KEY_SIZE, NIST_TDES_CBC3_IV, DRV_CRYPTO_DIRECTION_DECRYPT, DRV_CIPHER_CBC, NIST_TDES_CBC3_CIPHER, NIST_TDES_CBC3_PLAIN_DATA, NIST_TDES_VECTOR_SIZE },
};
+
#define FIPS_CIPHER_NUM_OF_TESTS (sizeof(FipsCipherDataTable) / sizeof(FipsCipherData))
static const FipsCmacData FipsCmacDataTable[] = {
{ DRV_CRYPTO_DIRECTION_ENCRYPT, NIST_AES_192_CMAC_KEY, AES_192_BIT_KEY_SIZE, NIST_AES_192_CMAC_PLAIN_DATA, NIST_AES_192_CMAC_VECTOR_SIZE, NIST_AES_192_CMAC_MAC, NIST_AES_192_CMAC_OUTPUT_SIZE },
{ DRV_CRYPTO_DIRECTION_ENCRYPT, NIST_AES_256_CMAC_KEY, AES_256_BIT_KEY_SIZE, NIST_AES_256_CMAC_PLAIN_DATA, NIST_AES_256_CMAC_VECTOR_SIZE, NIST_AES_256_CMAC_MAC, NIST_AES_256_CMAC_OUTPUT_SIZE },
};
+
#define FIPS_CMAC_NUM_OF_TESTS (sizeof(FipsCmacDataTable) / sizeof(FipsCmacData))
static const FipsHashData FipsHashDataTable[] = {
- { DRV_HASH_SHA1, NIST_SHA_1_MSG, NIST_SHA_MSG_SIZE, NIST_SHA_1_MD },
- { DRV_HASH_SHA256, NIST_SHA_256_MSG, NIST_SHA_MSG_SIZE, NIST_SHA_256_MD },
+ { DRV_HASH_SHA1, NIST_SHA_1_MSG, NIST_SHA_MSG_SIZE, NIST_SHA_1_MD },
+ { DRV_HASH_SHA256, NIST_SHA_256_MSG, NIST_SHA_MSG_SIZE, NIST_SHA_256_MD },
#if (CC_SUPPORT_SHA > 256)
// { DRV_HASH_SHA512, NIST_SHA_512_MSG, NIST_SHA_MSG_SIZE, NIST_SHA_512_MD },
#endif
};
+
#define FIPS_HASH_NUM_OF_TESTS (sizeof(FipsHashDataTable) / sizeof(FipsHashData))
static const FipsHmacData FipsHmacDataTable[] = {
- { DRV_HASH_SHA1, NIST_HMAC_SHA1_KEY, NIST_HMAC_SHA1_KEY_SIZE, NIST_HMAC_SHA1_MSG, NIST_HMAC_MSG_SIZE, NIST_HMAC_SHA1_MD },
- { DRV_HASH_SHA256, NIST_HMAC_SHA256_KEY, NIST_HMAC_SHA256_KEY_SIZE, NIST_HMAC_SHA256_MSG, NIST_HMAC_MSG_SIZE, NIST_HMAC_SHA256_MD },
+ { DRV_HASH_SHA1, NIST_HMAC_SHA1_KEY, NIST_HMAC_SHA1_KEY_SIZE, NIST_HMAC_SHA1_MSG, NIST_HMAC_MSG_SIZE, NIST_HMAC_SHA1_MD },
+ { DRV_HASH_SHA256, NIST_HMAC_SHA256_KEY, NIST_HMAC_SHA256_KEY_SIZE, NIST_HMAC_SHA256_MSG, NIST_HMAC_MSG_SIZE, NIST_HMAC_SHA256_MD },
#if (CC_SUPPORT_SHA > 256)
// { DRV_HASH_SHA512, NIST_HMAC_SHA512_KEY, NIST_HMAC_SHA512_KEY_SIZE, NIST_HMAC_SHA512_MSG, NIST_HMAC_MSG_SIZE, NIST_HMAC_SHA512_MD },
#endif
};
+
#define FIPS_HMAC_NUM_OF_TESTS (sizeof(FipsHmacDataTable) / sizeof(FipsHmacData))
static const FipsCcmData FipsCcmDataTable[] = {
- { DRV_CRYPTO_DIRECTION_ENCRYPT, NIST_AESCCM_128_KEY, NIST_AESCCM_128_BIT_KEY_SIZE, NIST_AESCCM_128_NONCE, NIST_AESCCM_128_ADATA, NIST_AESCCM_ADATA_SIZE, NIST_AESCCM_128_PLAIN_TEXT, NIST_AESCCM_TEXT_SIZE, NIST_AESCCM_128_CIPHER, NIST_AESCCM_TAG_SIZE, NIST_AESCCM_128_MAC },
- { DRV_CRYPTO_DIRECTION_DECRYPT, NIST_AESCCM_128_KEY, NIST_AESCCM_128_BIT_KEY_SIZE, NIST_AESCCM_128_NONCE, NIST_AESCCM_128_ADATA, NIST_AESCCM_ADATA_SIZE, NIST_AESCCM_128_CIPHER, NIST_AESCCM_TEXT_SIZE, NIST_AESCCM_128_PLAIN_TEXT, NIST_AESCCM_TAG_SIZE, NIST_AESCCM_128_MAC },
- { DRV_CRYPTO_DIRECTION_ENCRYPT, NIST_AESCCM_192_KEY, NIST_AESCCM_192_BIT_KEY_SIZE, NIST_AESCCM_192_NONCE, NIST_AESCCM_192_ADATA, NIST_AESCCM_ADATA_SIZE, NIST_AESCCM_192_PLAIN_TEXT, NIST_AESCCM_TEXT_SIZE, NIST_AESCCM_192_CIPHER, NIST_AESCCM_TAG_SIZE, NIST_AESCCM_192_MAC },
- { DRV_CRYPTO_DIRECTION_DECRYPT, NIST_AESCCM_192_KEY, NIST_AESCCM_192_BIT_KEY_SIZE, NIST_AESCCM_192_NONCE, NIST_AESCCM_192_ADATA, NIST_AESCCM_ADATA_SIZE, NIST_AESCCM_192_CIPHER, NIST_AESCCM_TEXT_SIZE, NIST_AESCCM_192_PLAIN_TEXT, NIST_AESCCM_TAG_SIZE, NIST_AESCCM_192_MAC },
- { DRV_CRYPTO_DIRECTION_ENCRYPT, NIST_AESCCM_256_KEY, NIST_AESCCM_256_BIT_KEY_SIZE, NIST_AESCCM_256_NONCE, NIST_AESCCM_256_ADATA, NIST_AESCCM_ADATA_SIZE, NIST_AESCCM_256_PLAIN_TEXT, NIST_AESCCM_TEXT_SIZE, NIST_AESCCM_256_CIPHER, NIST_AESCCM_TAG_SIZE, NIST_AESCCM_256_MAC },
- { DRV_CRYPTO_DIRECTION_DECRYPT, NIST_AESCCM_256_KEY, NIST_AESCCM_256_BIT_KEY_SIZE, NIST_AESCCM_256_NONCE, NIST_AESCCM_256_ADATA, NIST_AESCCM_ADATA_SIZE, NIST_AESCCM_256_CIPHER, NIST_AESCCM_TEXT_SIZE, NIST_AESCCM_256_PLAIN_TEXT, NIST_AESCCM_TAG_SIZE, NIST_AESCCM_256_MAC },
+ { DRV_CRYPTO_DIRECTION_ENCRYPT, NIST_AESCCM_128_KEY, NIST_AESCCM_128_BIT_KEY_SIZE, NIST_AESCCM_128_NONCE, NIST_AESCCM_128_ADATA, NIST_AESCCM_ADATA_SIZE, NIST_AESCCM_128_PLAIN_TEXT, NIST_AESCCM_TEXT_SIZE, NIST_AESCCM_128_CIPHER, NIST_AESCCM_TAG_SIZE, NIST_AESCCM_128_MAC },
+ { DRV_CRYPTO_DIRECTION_DECRYPT, NIST_AESCCM_128_KEY, NIST_AESCCM_128_BIT_KEY_SIZE, NIST_AESCCM_128_NONCE, NIST_AESCCM_128_ADATA, NIST_AESCCM_ADATA_SIZE, NIST_AESCCM_128_CIPHER, NIST_AESCCM_TEXT_SIZE, NIST_AESCCM_128_PLAIN_TEXT, NIST_AESCCM_TAG_SIZE, NIST_AESCCM_128_MAC },
+ { DRV_CRYPTO_DIRECTION_ENCRYPT, NIST_AESCCM_192_KEY, NIST_AESCCM_192_BIT_KEY_SIZE, NIST_AESCCM_192_NONCE, NIST_AESCCM_192_ADATA, NIST_AESCCM_ADATA_SIZE, NIST_AESCCM_192_PLAIN_TEXT, NIST_AESCCM_TEXT_SIZE, NIST_AESCCM_192_CIPHER, NIST_AESCCM_TAG_SIZE, NIST_AESCCM_192_MAC },
+ { DRV_CRYPTO_DIRECTION_DECRYPT, NIST_AESCCM_192_KEY, NIST_AESCCM_192_BIT_KEY_SIZE, NIST_AESCCM_192_NONCE, NIST_AESCCM_192_ADATA, NIST_AESCCM_ADATA_SIZE, NIST_AESCCM_192_CIPHER, NIST_AESCCM_TEXT_SIZE, NIST_AESCCM_192_PLAIN_TEXT, NIST_AESCCM_TAG_SIZE, NIST_AESCCM_192_MAC },
+ { DRV_CRYPTO_DIRECTION_ENCRYPT, NIST_AESCCM_256_KEY, NIST_AESCCM_256_BIT_KEY_SIZE, NIST_AESCCM_256_NONCE, NIST_AESCCM_256_ADATA, NIST_AESCCM_ADATA_SIZE, NIST_AESCCM_256_PLAIN_TEXT, NIST_AESCCM_TEXT_SIZE, NIST_AESCCM_256_CIPHER, NIST_AESCCM_TAG_SIZE, NIST_AESCCM_256_MAC },
+ { DRV_CRYPTO_DIRECTION_DECRYPT, NIST_AESCCM_256_KEY, NIST_AESCCM_256_BIT_KEY_SIZE, NIST_AESCCM_256_NONCE, NIST_AESCCM_256_ADATA, NIST_AESCCM_ADATA_SIZE, NIST_AESCCM_256_CIPHER, NIST_AESCCM_TEXT_SIZE, NIST_AESCCM_256_PLAIN_TEXT, NIST_AESCCM_TAG_SIZE, NIST_AESCCM_256_MAC },
};
+
#define FIPS_CCM_NUM_OF_TESTS (sizeof(FipsCcmDataTable) / sizeof(FipsCcmData))
static const FipsGcmData FipsGcmDataTable[] = {
- { DRV_CRYPTO_DIRECTION_ENCRYPT, NIST_AESGCM_128_KEY, NIST_AESGCM_128_BIT_KEY_SIZE, NIST_AESGCM_128_IV, NIST_AESGCM_128_ADATA, NIST_AESGCM_ADATA_SIZE, NIST_AESGCM_128_PLAIN_TEXT, NIST_AESGCM_TEXT_SIZE, NIST_AESGCM_128_CIPHER, NIST_AESGCM_TAG_SIZE, NIST_AESGCM_128_MAC },
- { DRV_CRYPTO_DIRECTION_DECRYPT, NIST_AESGCM_128_KEY, NIST_AESGCM_128_BIT_KEY_SIZE, NIST_AESGCM_128_IV, NIST_AESGCM_128_ADATA, NIST_AESGCM_ADATA_SIZE, NIST_AESGCM_128_CIPHER, NIST_AESGCM_TEXT_SIZE, NIST_AESGCM_128_PLAIN_TEXT, NIST_AESGCM_TAG_SIZE, NIST_AESGCM_128_MAC },
- { DRV_CRYPTO_DIRECTION_ENCRYPT, NIST_AESGCM_192_KEY, NIST_AESGCM_192_BIT_KEY_SIZE, NIST_AESGCM_192_IV, NIST_AESGCM_192_ADATA, NIST_AESGCM_ADATA_SIZE, NIST_AESGCM_192_PLAIN_TEXT, NIST_AESGCM_TEXT_SIZE, NIST_AESGCM_192_CIPHER, NIST_AESGCM_TAG_SIZE, NIST_AESGCM_192_MAC },
- { DRV_CRYPTO_DIRECTION_DECRYPT, NIST_AESGCM_192_KEY, NIST_AESGCM_192_BIT_KEY_SIZE, NIST_AESGCM_192_IV, NIST_AESGCM_192_ADATA, NIST_AESGCM_ADATA_SIZE, NIST_AESGCM_192_CIPHER, NIST_AESGCM_TEXT_SIZE, NIST_AESGCM_192_PLAIN_TEXT, NIST_AESGCM_TAG_SIZE, NIST_AESGCM_192_MAC },
- { DRV_CRYPTO_DIRECTION_ENCRYPT, NIST_AESGCM_256_KEY, NIST_AESGCM_256_BIT_KEY_SIZE, NIST_AESGCM_256_IV, NIST_AESGCM_256_ADATA, NIST_AESGCM_ADATA_SIZE, NIST_AESGCM_256_PLAIN_TEXT, NIST_AESGCM_TEXT_SIZE, NIST_AESGCM_256_CIPHER, NIST_AESGCM_TAG_SIZE, NIST_AESGCM_256_MAC },
- { DRV_CRYPTO_DIRECTION_DECRYPT, NIST_AESGCM_256_KEY, NIST_AESGCM_256_BIT_KEY_SIZE, NIST_AESGCM_256_IV, NIST_AESGCM_256_ADATA, NIST_AESGCM_ADATA_SIZE, NIST_AESGCM_256_CIPHER, NIST_AESGCM_TEXT_SIZE, NIST_AESGCM_256_PLAIN_TEXT, NIST_AESGCM_TAG_SIZE, NIST_AESGCM_256_MAC },
+ { DRV_CRYPTO_DIRECTION_ENCRYPT, NIST_AESGCM_128_KEY, NIST_AESGCM_128_BIT_KEY_SIZE, NIST_AESGCM_128_IV, NIST_AESGCM_128_ADATA, NIST_AESGCM_ADATA_SIZE, NIST_AESGCM_128_PLAIN_TEXT, NIST_AESGCM_TEXT_SIZE, NIST_AESGCM_128_CIPHER, NIST_AESGCM_TAG_SIZE, NIST_AESGCM_128_MAC },
+ { DRV_CRYPTO_DIRECTION_DECRYPT, NIST_AESGCM_128_KEY, NIST_AESGCM_128_BIT_KEY_SIZE, NIST_AESGCM_128_IV, NIST_AESGCM_128_ADATA, NIST_AESGCM_ADATA_SIZE, NIST_AESGCM_128_CIPHER, NIST_AESGCM_TEXT_SIZE, NIST_AESGCM_128_PLAIN_TEXT, NIST_AESGCM_TAG_SIZE, NIST_AESGCM_128_MAC },
+ { DRV_CRYPTO_DIRECTION_ENCRYPT, NIST_AESGCM_192_KEY, NIST_AESGCM_192_BIT_KEY_SIZE, NIST_AESGCM_192_IV, NIST_AESGCM_192_ADATA, NIST_AESGCM_ADATA_SIZE, NIST_AESGCM_192_PLAIN_TEXT, NIST_AESGCM_TEXT_SIZE, NIST_AESGCM_192_CIPHER, NIST_AESGCM_TAG_SIZE, NIST_AESGCM_192_MAC },
+ { DRV_CRYPTO_DIRECTION_DECRYPT, NIST_AESGCM_192_KEY, NIST_AESGCM_192_BIT_KEY_SIZE, NIST_AESGCM_192_IV, NIST_AESGCM_192_ADATA, NIST_AESGCM_ADATA_SIZE, NIST_AESGCM_192_CIPHER, NIST_AESGCM_TEXT_SIZE, NIST_AESGCM_192_PLAIN_TEXT, NIST_AESGCM_TAG_SIZE, NIST_AESGCM_192_MAC },
+ { DRV_CRYPTO_DIRECTION_ENCRYPT, NIST_AESGCM_256_KEY, NIST_AESGCM_256_BIT_KEY_SIZE, NIST_AESGCM_256_IV, NIST_AESGCM_256_ADATA, NIST_AESGCM_ADATA_SIZE, NIST_AESGCM_256_PLAIN_TEXT, NIST_AESGCM_TEXT_SIZE, NIST_AESGCM_256_CIPHER, NIST_AESGCM_TAG_SIZE, NIST_AESGCM_256_MAC },
+ { DRV_CRYPTO_DIRECTION_DECRYPT, NIST_AESGCM_256_KEY, NIST_AESGCM_256_BIT_KEY_SIZE, NIST_AESGCM_256_IV, NIST_AESGCM_256_ADATA, NIST_AESGCM_ADATA_SIZE, NIST_AESGCM_256_CIPHER, NIST_AESGCM_TEXT_SIZE, NIST_AESGCM_256_PLAIN_TEXT, NIST_AESGCM_TAG_SIZE, NIST_AESGCM_256_MAC },
};
-#define FIPS_GCM_NUM_OF_TESTS (sizeof(FipsGcmDataTable) / sizeof(FipsGcmData))
+#define FIPS_GCM_NUM_OF_TESTS (sizeof(FipsGcmDataTable) / sizeof(FipsGcmData))
-static inline ssi_fips_error_t
+static inline enum cc_fips_error
FIPS_CipherToFipsError(enum drv_cipher_mode mode, bool is_aes)
{
switch (mode)
{
case DRV_CIPHER_ECB:
- return is_aes ? CC_REE_FIPS_ERROR_AES_ECB_PUT : CC_REE_FIPS_ERROR_DES_ECB_PUT ;
+ return is_aes ? CC_REE_FIPS_ERROR_AES_ECB_PUT : CC_REE_FIPS_ERROR_DES_ECB_PUT;
case DRV_CIPHER_CBC:
- return is_aes ? CC_REE_FIPS_ERROR_AES_CBC_PUT : CC_REE_FIPS_ERROR_DES_CBC_PUT ;
+ return is_aes ? CC_REE_FIPS_ERROR_AES_CBC_PUT : CC_REE_FIPS_ERROR_DES_CBC_PUT;
case DRV_CIPHER_OFB:
return CC_REE_FIPS_ERROR_AES_OFB_PUT;
case DRV_CIPHER_CTR:
return CC_REE_FIPS_ERROR_GENERAL;
}
-
-static inline int
+static inline int
ssi_cipher_fips_run_test(struct ssi_drvdata *drvdata,
bool is_aes,
int cipher_mode,
int rc;
struct ssi_crypto_req ssi_req = {0};
- HwDesc_s desc[FIPS_CIPHER_MAX_SEQ_LEN];
+ struct cc_hw_desc desc[FIPS_CIPHER_MAX_SEQ_LEN];
int idx = 0;
int s_flow_mode = is_aes ? S_DIN_to_AES : S_DIN_to_DES;
case DRV_CIPHER_CTR:
case DRV_CIPHER_OFB:
/* Load cipher state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- iv_dma_addr, iv_len, NS_BIT);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], direction);
- HW_DESC_SET_FLOW_MODE(&desc[idx], s_flow_mode);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], cipher_mode);
- if ((cipher_mode == DRV_CIPHER_CTR) ||
- (cipher_mode == DRV_CIPHER_OFB) ) {
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE1);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI,
+ iv_dma_addr, iv_len, NS_BIT);
+ set_cipher_config0(&desc[idx], direction);
+ set_flow_mode(&desc[idx], s_flow_mode);
+ set_cipher_mode(&desc[idx], cipher_mode);
+ if ((cipher_mode == DRV_CIPHER_CTR) ||
+ (cipher_mode == DRV_CIPHER_OFB)) {
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE1);
} else {
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
}
idx++;
/*FALLTHROUGH*/
case DRV_CIPHER_ECB:
/* Load key */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], cipher_mode);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], direction);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], cipher_mode);
+ set_cipher_config0(&desc[idx], direction);
if (is_aes) {
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- key_dma_addr,
- ((key_len == 24) ? AES_MAX_KEY_SIZE : key_len),
- NS_BIT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], key_len);
+ set_din_type(&desc[idx], DMA_DLLI, key_dma_addr,
+ ((key_len == 24) ? AES_MAX_KEY_SIZE :
+ key_len), NS_BIT);
+ set_key_size_aes(&desc[idx], key_len);
} else {/*des*/
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- key_dma_addr, key_len,
- NS_BIT);
- HW_DESC_SET_KEY_SIZE_DES(&desc[idx], key_len);
+ set_din_type(&desc[idx], DMA_DLLI, key_dma_addr,
+ key_len, NS_BIT);
+ set_key_size_des(&desc[idx], key_len);
}
- HW_DESC_SET_FLOW_MODE(&desc[idx], s_flow_mode);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ set_flow_mode(&desc[idx], s_flow_mode);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
break;
case DRV_CIPHER_XTS:
/* Load AES key */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], cipher_mode);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], direction);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- key_dma_addr, key_len/2, NS_BIT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], key_len/2);
- HW_DESC_SET_FLOW_MODE(&desc[idx], s_flow_mode);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], cipher_mode);
+ set_cipher_config0(&desc[idx], direction);
+ set_din_type(&desc[idx], DMA_DLLI, key_dma_addr, (key_len / 2),
+ NS_BIT);
+ set_key_size_aes(&desc[idx], (key_len / 2));
+ set_flow_mode(&desc[idx], s_flow_mode);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
/* load XEX key */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], cipher_mode);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], direction);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- (key_dma_addr+key_len/2), key_len/2, NS_BIT);
- HW_DESC_SET_XEX_DATA_UNIT_SIZE(&desc[idx], data_size);
- HW_DESC_SET_FLOW_MODE(&desc[idx], s_flow_mode);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], key_len/2);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_XEX_KEY);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], cipher_mode);
+ set_cipher_config0(&desc[idx], direction);
+ set_din_type(&desc[idx], DMA_DLLI,
+ (key_dma_addr + (key_len / 2)),
+ (key_len / 2), NS_BIT);
+ set_xex_data_unit_size(&desc[idx], data_size);
+ set_flow_mode(&desc[idx], s_flow_mode);
+ set_key_size_aes(&desc[idx], (key_len / 2));
+ set_setup_mode(&desc[idx], SETUP_LOAD_XEX_KEY);
idx++;
/* Set state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE1);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], cipher_mode);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], direction);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], key_len/2);
- HW_DESC_SET_FLOW_MODE(&desc[idx], s_flow_mode);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- iv_dma_addr, CC_AES_BLOCK_SIZE, NS_BIT);
+ hw_desc_init(&desc[idx]);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE1);
+ set_cipher_mode(&desc[idx], cipher_mode);
+ set_cipher_config0(&desc[idx], direction);
+ set_key_size_aes(&desc[idx], (key_len / 2));
+ set_flow_mode(&desc[idx], s_flow_mode);
+ set_din_type(&desc[idx], DMA_DLLI, iv_dma_addr,
+ CC_AES_BLOCK_SIZE, NS_BIT);
idx++;
break;
default:
}
/* create data descriptor */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, din_dma_addr, data_size, NS_BIT);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], dout_dma_addr, data_size, NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], is_aes ? DIN_AES_DOUT : DIN_DES_DOUT);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, din_dma_addr, data_size, NS_BIT);
+ set_dout_dlli(&desc[idx], dout_dma_addr, data_size, NS_BIT, 0);
+ set_flow_mode(&desc[idx], is_aes ? DIN_AES_DOUT : DIN_DES_DOUT);
idx++;
/* perform the operation - Lock HW and push sequence */
return rc;
}
-
-ssi_fips_error_t
+enum cc_fips_error
ssi_cipher_fips_power_up_tests(struct ssi_drvdata *drvdata, void *cpu_addr_buffer, dma_addr_t dma_coherent_buffer)
{
- ssi_fips_error_t error = CC_REE_FIPS_ERROR_OK;
+ enum cc_fips_error error = CC_REE_FIPS_ERROR_OK;
size_t i;
struct fips_cipher_ctx *virt_ctx = (struct fips_cipher_ctx *)cpu_addr_buffer;
for (i = 0; i < FIPS_CIPHER_NUM_OF_TESTS; ++i)
{
- FipsCipherData *cipherData = (FipsCipherData*)&FipsCipherDataTable[i];
+ FipsCipherData *cipherData = (FipsCipherData *)&FipsCipherDataTable[i];
int rc = 0;
- size_t iv_size = cipherData->isAes ? NIST_AES_IV_SIZE : NIST_TDES_IV_SIZE ;
+ size_t iv_size = cipherData->isAes ? NIST_AES_IV_SIZE : NIST_TDES_IV_SIZE;
memset(cpu_addr_buffer, 0, sizeof(struct fips_cipher_ctx));
return error;
}
-
-static inline int
+static inline int
ssi_cmac_fips_run_test(struct ssi_drvdata *drvdata,
dma_addr_t key_dma_addr,
size_t key_len,
int rc;
struct ssi_crypto_req ssi_req = {0};
- HwDesc_s desc[FIPS_CMAC_MAX_SEQ_LEN];
+ struct cc_hw_desc desc[FIPS_CMAC_MAX_SEQ_LEN];
int idx = 0;
/* Setup CMAC Key */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, key_dma_addr,
- ((key_len == 24) ? AES_MAX_KEY_SIZE : key_len), NS_BIT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_CMAC);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], key_len);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, key_dma_addr,
+ ((key_len == 24) ? AES_MAX_KEY_SIZE : key_len), NS_BIT);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_CMAC);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_key_size_aes(&desc[idx], key_len);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
/* Load MAC state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, digest_dma_addr, CC_AES_BLOCK_SIZE, NS_BIT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_CMAC);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], key_len);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, digest_dma_addr, CC_AES_BLOCK_SIZE,
+ NS_BIT);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_CMAC);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_key_size_aes(&desc[idx], key_len);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
-
//ssi_hash_create_data_desc(state, ctx, DIN_AES_DOUT, desc, false, &idx);
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- din_dma_addr,
- din_len, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_AES_DOUT);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, din_dma_addr, din_len, NS_BIT);
+ set_flow_mode(&desc[idx], DIN_AES_DOUT);
idx++;
-
+
/* Get final MAC result */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], digest_dma_addr, CC_AES_BLOCK_SIZE, NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_AES_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_CMAC);
+ hw_desc_init(&desc[idx]);
+ set_dout_dlli(&desc[idx], digest_dma_addr, CC_AES_BLOCK_SIZE, NS_BIT,
+ 0);
+ set_flow_mode(&desc[idx], S_AES_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_CMAC);
idx++;
/* perform the operation - Lock HW and push sequence */
return rc;
}
-ssi_fips_error_t
+enum cc_fips_error
ssi_cmac_fips_power_up_tests(struct ssi_drvdata *drvdata, void *cpu_addr_buffer, dma_addr_t dma_coherent_buffer)
{
- ssi_fips_error_t error = CC_REE_FIPS_ERROR_OK;
+ enum cc_fips_error error = CC_REE_FIPS_ERROR_OK;
size_t i;
struct fips_cmac_ctx *virt_ctx = (struct fips_cmac_ctx *)cpu_addr_buffer;
for (i = 0; i < FIPS_CMAC_NUM_OF_TESTS; ++i)
{
- FipsCmacData *cmac_data = (FipsCmacData*)&FipsCmacDataTable[i];
+ FipsCmacData *cmac_data = (FipsCmacData *)&FipsCmacDataTable[i];
int rc = 0;
memset(cpu_addr_buffer, 0, sizeof(struct fips_cmac_ctx));
return error;
}
-
-static inline ssi_fips_error_t
+static inline enum cc_fips_error
FIPS_HashToFipsError(enum drv_hash_mode hash_mode)
{
switch (hash_mode) {
return CC_REE_FIPS_ERROR_GENERAL;
}
-static inline int
+static inline int
ssi_hash_fips_run_test(struct ssi_drvdata *drvdata,
dma_addr_t initial_digest_dma_addr,
dma_addr_t din_dma_addr,
int rc;
struct ssi_crypto_req ssi_req = {0};
- HwDesc_s desc[FIPS_HASH_MAX_SEQ_LEN];
+ struct cc_hw_desc desc[FIPS_HASH_MAX_SEQ_LEN];
int idx = 0;
/* Load initial digest */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hw_mode);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, initial_digest_dma_addr, inter_digestsize, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hw_mode);
+ set_din_type(&desc[idx], DMA_DLLI, initial_digest_dma_addr,
+ inter_digestsize, NS_BIT);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
idx++;
/* Load the hash current length */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hw_mode);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0, HASH_LEN_SIZE);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_ENABLED);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hw_mode);
+ set_din_const(&desc[idx], 0, HASH_LEN_SIZE);
+ set_cipher_config1(&desc[idx], HASH_PADDING_ENABLED);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
/* data descriptor */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, din_dma_addr, data_in_size, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_HASH);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, din_dma_addr, data_in_size, NS_BIT);
+ set_flow_mode(&desc[idx], DIN_HASH);
idx++;
/* Get final MAC result */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hw_mode);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], mac_res_dma_addr, digest_size, NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_DISABLED);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hw_mode);
+ set_dout_dlli(&desc[idx], mac_res_dma_addr, digest_size, NS_BIT, 0);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
+ set_cipher_config1(&desc[idx], HASH_PADDING_DISABLED);
if (unlikely((hash_mode == DRV_HASH_MD5) ||
(hash_mode == DRV_HASH_SHA384) ||
(hash_mode == DRV_HASH_SHA512))) {
- HW_DESC_SET_BYTES_SWAP(&desc[idx], 1);
+ set_bytes_swap(&desc[idx], 1);
} else {
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], HASH_DIGEST_RESULT_LITTLE_ENDIAN);
+ set_cipher_config0(&desc[idx],
+ HASH_DIGEST_RESULT_LITTLE_ENDIAN);
}
idx++;
return rc;
}
-ssi_fips_error_t
+enum cc_fips_error
ssi_hash_fips_power_up_tests(struct ssi_drvdata *drvdata, void *cpu_addr_buffer, dma_addr_t dma_coherent_buffer)
{
- ssi_fips_error_t error = CC_REE_FIPS_ERROR_OK;
+ enum cc_fips_error error = CC_REE_FIPS_ERROR_OK;
size_t i;
struct fips_hash_ctx *virt_ctx = (struct fips_hash_ctx *)cpu_addr_buffer;
for (i = 0; i < FIPS_HASH_NUM_OF_TESTS; ++i)
{
- FipsHashData *hash_data = (FipsHashData*)&FipsHashDataTable[i];
+ FipsHashData *hash_data = (FipsHashData *)&FipsHashDataTable[i];
int rc = 0;
enum drv_hash_hw_mode hw_mode = 0;
int digest_size = 0;
digest_size = CC_SHA1_DIGEST_SIZE;
inter_digestsize = CC_SHA1_DIGEST_SIZE;
/* copy the initial digest into the allocated cache coherent buffer */
- memcpy(virt_ctx->initial_digest, (void*)sha1_init, CC_SHA1_DIGEST_SIZE);
+ memcpy(virt_ctx->initial_digest, (void *)sha1_init, CC_SHA1_DIGEST_SIZE);
break;
case DRV_HASH_SHA256:
hw_mode = DRV_HASH_HW_SHA256;
digest_size = CC_SHA256_DIGEST_SIZE;
inter_digestsize = CC_SHA256_DIGEST_SIZE;
- memcpy(virt_ctx->initial_digest, (void*)sha256_init, CC_SHA256_DIGEST_SIZE);
+ memcpy(virt_ctx->initial_digest, (void *)sha256_init, CC_SHA256_DIGEST_SIZE);
break;
#if (CC_SUPPORT_SHA > 256)
case DRV_HASH_SHA512:
hw_mode = DRV_HASH_HW_SHA512;
digest_size = CC_SHA512_DIGEST_SIZE;
inter_digestsize = CC_SHA512_DIGEST_SIZE;
- memcpy(virt_ctx->initial_digest, (void*)sha512_init, CC_SHA512_DIGEST_SIZE);
+ memcpy(virt_ctx->initial_digest, (void *)sha512_init, CC_SHA512_DIGEST_SIZE);
break;
#endif
default:
FIPS_LOG("ssi_hash_fips_run_test %d returned error - rc = %d \n", i, rc);
error = FIPS_HashToFipsError(hash_data->hash_mode);
break;
- }
+ }
/* compare actual mac result to expected */
if (memcmp(virt_ctx->mac_res, hash_data->mac_res, digest_size) != 0)
error = FIPS_HashToFipsError(hash_data->hash_mode);
break;
- }
+ }
}
return error;
}
-
-static inline ssi_fips_error_t
+static inline enum cc_fips_error
FIPS_HmacToFipsError(enum drv_hash_mode hash_mode)
{
switch (hash_mode) {
return CC_REE_FIPS_ERROR_GENERAL;
}
-static inline int
+static inline int
ssi_hmac_fips_run_test(struct ssi_drvdata *drvdata,
dma_addr_t initial_digest_dma_addr,
dma_addr_t key_dma_addr,
dma_addr_t digest_bytes_len_dma_addr)
{
/* The implemented flow is not the same as the one implemented in ssi_hash.c (setkey + digest flows).
- In this flow, there is no need to store and reload some of the intermidiate results. */
+ * In this flow, there is no need to store and reload some of the intermidiate results.
+ */
/* max number of descriptors used for the flow */
#define FIPS_HMAC_MAX_SEQ_LEN 12
int rc;
struct ssi_crypto_req ssi_req = {0};
- HwDesc_s desc[FIPS_HMAC_MAX_SEQ_LEN];
+ struct cc_hw_desc desc[FIPS_HMAC_MAX_SEQ_LEN];
int idx = 0;
int i;
/* calc the hash opad first and ipad only afterwards (unlike the flow in ssi_hash.c) */
unsigned int hmacPadConst[2] = { HMAC_OPAD_CONST, HMAC_IPAD_CONST };
// assume (key_size <= block_size)
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, key_dma_addr, key_size, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], BYPASS);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], k0_dma_addr, key_size, NS_BIT, 0);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, key_dma_addr, key_size, NS_BIT);
+ set_flow_mode(&desc[idx], BYPASS);
+ set_dout_dlli(&desc[idx], k0_dma_addr, key_size, NS_BIT, 0);
idx++;
// if needed, append Key with zeros to create K0
if ((block_size - key_size) != 0) {
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0, (block_size - key_size));
- HW_DESC_SET_FLOW_MODE(&desc[idx], BYPASS);
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- (k0_dma_addr + key_size), (block_size - key_size),
- NS_BIT, 0);
+ hw_desc_init(&desc[idx]);
+ set_din_const(&desc[idx], 0, (block_size - key_size));
+ set_flow_mode(&desc[idx], BYPASS);
+ set_dout_dlli(&desc[idx], (k0_dma_addr + key_size),
+ (block_size - key_size), NS_BIT, 0);
idx++;
}
/* calc derived HMAC key */
for (i = 0; i < 2; i++) {
/* Load hash initial state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hw_mode);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, initial_digest_dma_addr, inter_digestsize, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hw_mode);
+ set_din_type(&desc[idx], DMA_DLLI, initial_digest_dma_addr,
+ inter_digestsize, NS_BIT);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
idx++;
-
/* Load the hash current length*/
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hw_mode);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0, HASH_LEN_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hw_mode);
+ set_din_const(&desc[idx], 0, HASH_LEN_SIZE);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
/* Prepare opad/ipad key */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_XOR_VAL(&desc[idx], hmacPadConst[i]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hw_mode);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE1);
+ hw_desc_init(&desc[idx]);
+ set_xor_val(&desc[idx], hmacPadConst[i]);
+ set_cipher_mode(&desc[idx], hw_mode);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE1);
idx++;
/* Perform HASH update */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- k0_dma_addr,
- block_size, NS_BIT);
- HW_DESC_SET_CIPHER_MODE(&desc[idx],hw_mode);
- HW_DESC_SET_XOR_ACTIVE(&desc[idx]);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_HASH);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, k0_dma_addr, block_size,
+ NS_BIT);
+ set_cipher_mode(&desc[idx], hw_mode);
+ set_xor_active(&desc[idx]);
+ set_flow_mode(&desc[idx], DIN_HASH);
idx++;
if (i == 0) {
/* First iteration - calc H(K0^opad) into tmp_digest_dma_addr */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hw_mode);
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- tmp_digest_dma_addr,
- inter_digestsize,
- NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hw_mode);
+ set_dout_dlli(&desc[idx], tmp_digest_dma_addr,
+ inter_digestsize, NS_BIT, 0);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
idx++;
// is this needed?? or continue with current descriptors??
}
/* data descriptor */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- din_dma_addr, data_in_size,
- NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_HASH);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, din_dma_addr, data_in_size, NS_BIT);
+ set_flow_mode(&desc[idx], DIN_HASH);
idx++;
/* HW last hash block padding (aka. "DO_PAD") */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hw_mode);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], k0_dma_addr, HASH_LEN_SIZE, NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE1);
- HW_DESC_SET_CIPHER_DO(&desc[idx], DO_PAD);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hw_mode);
+ set_dout_dlli(&desc[idx], k0_dma_addr, HASH_LEN_SIZE, NS_BIT, 0);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE1);
+ set_cipher_do(&desc[idx], DO_PAD);
idx++;
/* store the hash digest result in the context */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hw_mode);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], k0_dma_addr, digest_size, NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hw_mode);
+ set_dout_dlli(&desc[idx], k0_dma_addr, digest_size, NS_BIT, 0);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
if (unlikely((hash_mode == DRV_HASH_MD5) ||
(hash_mode == DRV_HASH_SHA384) ||
(hash_mode == DRV_HASH_SHA512))) {
- HW_DESC_SET_BYTES_SWAP(&desc[idx], 1);
+ set_bytes_swap(&desc[idx], 1);
} else {
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], HASH_DIGEST_RESULT_LITTLE_ENDIAN);
+ set_cipher_config0(&desc[idx],
+ HASH_DIGEST_RESULT_LITTLE_ENDIAN);
}
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
idx++;
/* at this point:
- tmp_digest = H(o_key_pad)
- k0 = H(i_key_pad || m)
- */
+ * tmp_digest = H(o_key_pad)
+ * k0 = H(i_key_pad || m)
+ */
/* Loading hash opad xor key state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hw_mode);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, tmp_digest_dma_addr, inter_digestsize, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hw_mode);
+ set_din_type(&desc[idx], DMA_DLLI, tmp_digest_dma_addr,
+ inter_digestsize, NS_BIT);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
idx++;
/* Load the hash current length */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hw_mode);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, digest_bytes_len_dma_addr, HASH_LEN_SIZE, NS_BIT);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_ENABLED);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hw_mode);
+ set_din_type(&desc[idx], DMA_DLLI, digest_bytes_len_dma_addr,
+ HASH_LEN_SIZE, NS_BIT);
+ set_cipher_config1(&desc[idx], HASH_PADDING_ENABLED);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
/* Memory Barrier: wait for IPAD/OPAD axi write to complete */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_NO_DMA(&desc[idx], 0, 0xfffff0);
- HW_DESC_SET_DOUT_NO_DMA(&desc[idx], 0, 0, 1);
+ hw_desc_init(&desc[idx]);
+ set_din_no_dma(&desc[idx], 0, 0xfffff0);
+ set_dout_no_dma(&desc[idx], 0, 0, 1);
idx++;
/* Perform HASH update */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, k0_dma_addr, digest_size, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_HASH);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, k0_dma_addr, digest_size, NS_BIT);
+ set_flow_mode(&desc[idx], DIN_HASH);
idx++;
-
/* Get final MAC result */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], hw_mode);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], mac_res_dma_addr, digest_size, NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_DISABLED);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], hw_mode);
+ set_dout_dlli(&desc[idx], mac_res_dma_addr, digest_size, NS_BIT, 0);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
+ set_cipher_config1(&desc[idx], HASH_PADDING_DISABLED);
if (unlikely((hash_mode == DRV_HASH_MD5) ||
(hash_mode == DRV_HASH_SHA384) ||
(hash_mode == DRV_HASH_SHA512))) {
- HW_DESC_SET_BYTES_SWAP(&desc[idx], 1);
+ set_bytes_swap(&desc[idx], 1);
} else {
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], HASH_DIGEST_RESULT_LITTLE_ENDIAN);
+ set_cipher_config0(&desc[idx],
+ HASH_DIGEST_RESULT_LITTLE_ENDIAN);
}
idx++;
return rc;
}
-ssi_fips_error_t
+enum cc_fips_error
ssi_hmac_fips_power_up_tests(struct ssi_drvdata *drvdata, void *cpu_addr_buffer, dma_addr_t dma_coherent_buffer)
{
- ssi_fips_error_t error = CC_REE_FIPS_ERROR_OK;
+ enum cc_fips_error error = CC_REE_FIPS_ERROR_OK;
size_t i;
struct fips_hmac_ctx *virt_ctx = (struct fips_hmac_ctx *)cpu_addr_buffer;
for (i = 0; i < FIPS_HMAC_NUM_OF_TESTS; ++i)
{
- FipsHmacData *hmac_data = (FipsHmacData*)&FipsHmacDataTable[i];
+ FipsHmacData *hmac_data = (FipsHmacData *)&FipsHmacDataTable[i];
int rc = 0;
enum drv_hash_hw_mode hw_mode = 0;
int digest_size = 0;
digest_size = CC_SHA1_DIGEST_SIZE;
block_size = CC_SHA1_BLOCK_SIZE;
inter_digestsize = CC_SHA1_DIGEST_SIZE;
- memcpy(virt_ctx->initial_digest, (void*)sha1_init, CC_SHA1_DIGEST_SIZE);
+ memcpy(virt_ctx->initial_digest, (void *)sha1_init, CC_SHA1_DIGEST_SIZE);
memcpy(virt_ctx->digest_bytes_len, digest_len_init, HASH_LEN_SIZE);
break;
case DRV_HASH_SHA256:
digest_size = CC_SHA256_DIGEST_SIZE;
block_size = CC_SHA256_BLOCK_SIZE;
inter_digestsize = CC_SHA256_DIGEST_SIZE;
- memcpy(virt_ctx->initial_digest, (void*)sha256_init, CC_SHA256_DIGEST_SIZE);
+ memcpy(virt_ctx->initial_digest, (void *)sha256_init, CC_SHA256_DIGEST_SIZE);
memcpy(virt_ctx->digest_bytes_len, digest_len_init, HASH_LEN_SIZE);
break;
#if (CC_SUPPORT_SHA > 256)
digest_size = CC_SHA512_DIGEST_SIZE;
block_size = CC_SHA512_BLOCK_SIZE;
inter_digestsize = CC_SHA512_DIGEST_SIZE;
- memcpy(virt_ctx->initial_digest, (void*)sha512_init, CC_SHA512_DIGEST_SIZE);
+ memcpy(virt_ctx->initial_digest, (void *)sha512_init, CC_SHA512_DIGEST_SIZE);
memcpy(virt_ctx->digest_bytes_len, digest_len_sha512_init, HASH_LEN_SIZE);
break;
#endif
return error;
}
-
-static inline int
+static inline int
ssi_ccm_fips_run_test(struct ssi_drvdata *drvdata,
enum drv_crypto_direction direction,
dma_addr_t key_dma_addr,
int rc;
struct ssi_crypto_req ssi_req = {0};
- HwDesc_s desc[FIPS_CCM_MAX_SEQ_LEN];
+ struct cc_hw_desc desc[FIPS_CCM_MAX_SEQ_LEN];
unsigned int idx = 0;
unsigned int cipher_flow_mode;
}
/* load key */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_CTR);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, key_dma_addr,
- ((key_size == NIST_AESCCM_192_BIT_KEY_SIZE) ? CC_AES_KEY_SIZE_MAX : key_size),
- NS_BIT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], key_size);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_CTR);
+ set_din_type(&desc[idx], DMA_DLLI, key_dma_addr,
+ ((key_size == NIST_AESCCM_192_BIT_KEY_SIZE) ?
+ CC_AES_KEY_SIZE_MAX : key_size), NS_BIT)
+ set_key_size_aes(&desc[idx], key_size);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
/* load ctr state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_CTR);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], key_size);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- iv_dma_addr, AES_BLOCK_SIZE,
- NS_BIT);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE1);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_CTR);
+ set_key_size_aes(&desc[idx], key_size);
+ set_din_type(&desc[idx], DMA_DLLI, iv_dma_addr, AES_BLOCK_SIZE,
+ NS_BIT);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE1);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
/* load MAC key */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_CBC_MAC);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, key_dma_addr,
- ((key_size == NIST_AESCCM_192_BIT_KEY_SIZE) ? CC_AES_KEY_SIZE_MAX : key_size),
- NS_BIT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], key_size);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_CBC_MAC);
+ set_din_type(&desc[idx], DMA_DLLI, key_dma_addr,
+ ((key_size == NIST_AESCCM_192_BIT_KEY_SIZE) ?
+ CC_AES_KEY_SIZE_MAX : key_size), NS_BIT);
+ set_key_size_aes(&desc[idx], key_size);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_aes_not_hash_mode(&desc[idx]);
idx++;
/* load MAC state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_CBC_MAC);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], key_size);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, mac_res_dma_addr, NIST_AESCCM_TAG_SIZE, NS_BIT);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_CBC_MAC);
+ set_key_size_aes(&desc[idx], key_size);
+ set_din_type(&desc[idx], DMA_DLLI, mac_res_dma_addr,
+ NIST_AESCCM_TAG_SIZE, NS_BIT);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_aes_not_hash_mode(&desc[idx]);
idx++;
/* prcess assoc data */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, b0_a0_adata_dma_addr, b0_a0_adata_size, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_HASH);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, b0_a0_adata_dma_addr,
+ b0_a0_adata_size, NS_BIT);
+ set_flow_mode(&desc[idx], DIN_HASH);
idx++;
/* process the cipher */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, din_dma_addr, din_size, NS_BIT);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], dout_dma_addr, din_size, NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], cipher_flow_mode);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, din_dma_addr, din_size, NS_BIT);
+ set_dout_dlli(&desc[idx], dout_dma_addr, din_size, NS_BIT, 0);
+ set_flow_mode(&desc[idx], cipher_flow_mode);
idx++;
/* Read temporal MAC */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_CBC_MAC);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], mac_res_dma_addr, NIST_AESCCM_TAG_SIZE, NS_BIT, 0);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], HASH_DIGEST_RESULT_LITTLE_ENDIAN);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_CBC_MAC);
+ set_dout_dlli(&desc[idx], mac_res_dma_addr, NIST_AESCCM_TAG_SIZE,
+ NS_BIT, 0);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
+ set_cipher_config0(&desc[idx], HASH_DIGEST_RESULT_LITTLE_ENDIAN);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_aes_not_hash_mode(&desc[idx]);
idx++;
/* load AES-CTR state (for last MAC calculation)*/
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_CTR);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- ctr_cnt_0_dma_addr,
- AES_BLOCK_SIZE, NS_BIT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], key_size);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE1);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_CTR);
+ set_cipher_config0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
+ set_din_type(&desc[idx], DMA_DLLI, ctr_cnt_0_dma_addr, AES_BLOCK_SIZE,
+ NS_BIT);
+ set_key_size_aes(&desc[idx], key_size);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE1);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
/* Memory Barrier */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_NO_DMA(&desc[idx], 0, 0xfffff0);
- HW_DESC_SET_DOUT_NO_DMA(&desc[idx], 0, 0, 1);
+ hw_desc_init(&desc[idx]);
+ set_din_no_dma(&desc[idx], 0, 0xfffff0);
+ set_dout_no_dma(&desc[idx], 0, 0, 1);
idx++;
/* encrypt the "T" value and store MAC inplace */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, mac_res_dma_addr, NIST_AESCCM_TAG_SIZE, NS_BIT);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], mac_res_dma_addr, NIST_AESCCM_TAG_SIZE, NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_AES_DOUT);
- idx++;
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, mac_res_dma_addr,
+ NIST_AESCCM_TAG_SIZE, NS_BIT);
+ set_dout_dlli(&desc[idx], mac_res_dma_addr, NIST_AESCCM_TAG_SIZE,
+ NS_BIT, 0);
+ set_flow_mode(&desc[idx], DIN_AES_DOUT);
+ idx++;
/* perform the operation - Lock HW and push sequence */
BUG_ON(idx > FIPS_CCM_MAX_SEQ_LEN);
return rc;
}
-ssi_fips_error_t
+enum cc_fips_error
ssi_ccm_fips_power_up_tests(struct ssi_drvdata *drvdata, void *cpu_addr_buffer, dma_addr_t dma_coherent_buffer)
{
- ssi_fips_error_t error = CC_REE_FIPS_ERROR_OK;
+ enum cc_fips_error error = CC_REE_FIPS_ERROR_OK;
size_t i;
struct fips_ccm_ctx *virt_ctx = (struct fips_ccm_ctx *)cpu_addr_buffer;
for (i = 0; i < FIPS_CCM_NUM_OF_TESTS; ++i)
{
- FipsCcmData *ccmData = (FipsCcmData*)&FipsCcmDataTable[i];
+ FipsCcmData *ccmData = (FipsCcmData *)&FipsCcmDataTable[i];
int rc = 0;
memset(cpu_addr_buffer, 0, sizeof(struct fips_ccm_ctx));
{
/* build B0 -- B0, nonce, l(m) */
__be16 data = cpu_to_be16(NIST_AESCCM_TEXT_SIZE);
+
virt_ctx->b0_a0_adata[0] = NIST_AESCCM_B0_VAL;
memcpy(virt_ctx->b0_a0_adata + 1, ccmData->nonce, NIST_AESCCM_NONCE_SIZE);
memcpy(virt_ctx->b0_a0_adata + 14, (u8 *)&data, sizeof(__be16));
if (memcmp(virt_ctx->dout, ccmData->dataOut, ccmData->dataInSize) != 0)
{
FIPS_LOG("dout comparison error %d - size=%d \n", i, ccmData->dataInSize);
- error = CC_REE_FIPS_ERROR_AESCCM_PUT;
+ error = CC_REE_FIPS_ERROR_AESCCM_PUT;
break;
- }
+ }
/* compare actual mac result to expected */
if (memcmp(virt_ctx->mac_res, ccmData->macResOut, ccmData->tagSize) != 0)
return error;
}
-
static inline int
ssi_gcm_fips_run_test(struct ssi_drvdata *drvdata,
enum drv_crypto_direction direction,
int rc;
struct ssi_crypto_req ssi_req = {0};
- HwDesc_s desc[FIPS_GCM_MAX_SEQ_LEN];
+ struct cc_hw_desc desc[FIPS_GCM_MAX_SEQ_LEN];
unsigned int idx = 0;
unsigned int cipher_flow_mode;
// ssi_aead_gcm_setup_ghash_desc(req, desc, seq_size);
///////////////////////////////// 1 ////////////////////////////////////
- /* load key to AES*/
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_ECB);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
- HW_DESC_SET_DIN_TYPE(&desc[idx],
- DMA_DLLI, key_dma_addr, key_size,
- NS_BIT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], key_size);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ /* load key to AES */
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_ECB);
+ set_cipher_config0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
+ set_din_type(&desc[idx], DMA_DLLI, key_dma_addr, key_size, NS_BIT);
+ set_key_size_aes(&desc[idx], key_size);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
/* process one zero block to generate hkey */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0x0, AES_BLOCK_SIZE);
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- hkey_dma_addr, AES_BLOCK_SIZE,
- NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_AES_DOUT);
+ hw_desc_init(&desc[idx]);
+ set_din_const(&desc[idx], 0x0, AES_BLOCK_SIZE);
+ set_dout_dlli(&desc[idx], hkey_dma_addr, AES_BLOCK_SIZE, NS_BIT, 0);
+ set_flow_mode(&desc[idx], DIN_AES_DOUT);
idx++;
/* Memory Barrier */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_NO_DMA(&desc[idx], 0, 0xfffff0);
- HW_DESC_SET_DOUT_NO_DMA(&desc[idx], 0, 0, 1);
+ hw_desc_init(&desc[idx]);
+ set_din_no_dma(&desc[idx], 0, 0xfffff0);
+ set_dout_no_dma(&desc[idx], 0, 0, 1);
idx++;
/* Load GHASH subkey */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- hkey_dma_addr, AES_BLOCK_SIZE,
- NS_BIT);
- HW_DESC_SET_DOUT_NO_DMA(&desc[idx], 0, 0, 1);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_HASH_HW_GHASH);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_ENABLED);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, hkey_dma_addr, AES_BLOCK_SIZE,
+ NS_BIT);
+ set_dout_no_dma(&desc[idx], 0, 0, 1);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_aes_not_hash_mode(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_HASH_HW_GHASH);
+ set_cipher_config1(&desc[idx], HASH_PADDING_ENABLED);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
/* Configure Hash Engine to work with GHASH.
- Since it was not possible to extend HASH submodes to add GHASH,
- The following command is necessary in order to select GHASH (according to HW designers)*/
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_NO_DMA(&desc[idx], 0, 0xfffff0);
- HW_DESC_SET_DOUT_NO_DMA(&desc[idx], 0, 0, 1);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_HASH_HW_GHASH);
- HW_DESC_SET_CIPHER_DO(&desc[idx], 1); //1=AES_SK RKEK
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_ENABLED);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ * Since it was not possible to extend HASH submodes to add GHASH,
+ * The following command is necessary in order to
+ * select GHASH (according to HW designers)
+ */
+ hw_desc_init(&desc[idx]);
+ set_din_no_dma(&desc[idx], 0, 0xfffff0);
+ set_dout_no_dma(&desc[idx], 0, 0, 1);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_aes_not_hash_mode(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_HASH_HW_GHASH);
+ set_cipher_do(&desc[idx], 1); //1=AES_SK RKEK
+ set_cipher_config0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
+ set_cipher_config1(&desc[idx], HASH_PADDING_ENABLED);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
/* Load GHASH initial STATE (which is 0). (for any hash there is an initial state) */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0x0, AES_BLOCK_SIZE);
- HW_DESC_SET_DOUT_NO_DMA(&desc[idx], 0, 0, 1);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_HASH_HW_GHASH);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_ENABLED);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_din_const(&desc[idx], 0x0, AES_BLOCK_SIZE);
+ set_dout_no_dma(&desc[idx], 0, 0, 1);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_aes_not_hash_mode(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_HASH_HW_GHASH);
+ set_cipher_config1(&desc[idx], HASH_PADDING_ENABLED);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
idx++;
-
-
///////////////////////////////// 2 ////////////////////////////////////
/* prcess(ghash) assoc data */
// if (req->assoclen > 0)
// ssi_aead_create_assoc_desc(req, DIN_HASH, desc, seq_size);
///////////////////////////////// 2 ////////////////////////////////////
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- adata_dma_addr, adata_size,
- NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_HASH);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, adata_dma_addr, adata_size, NS_BIT);
+ set_flow_mode(&desc[idx], DIN_HASH);
idx++;
-
///////////////////////////////// 3 ////////////////////////////////////
// ssi_aead_gcm_setup_gctr_desc(req, desc, seq_size);
///////////////////////////////// 3 ////////////////////////////////////
/* load key to AES*/
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_GCTR);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- key_dma_addr, key_size,
- NS_BIT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], key_size);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_GCTR);
+ set_cipher_config0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
+ set_din_type(&desc[idx], DMA_DLLI, key_dma_addr, key_size, NS_BIT);
+ set_key_size_aes(&desc[idx], key_size);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
/* load AES/CTR initial CTR value inc by 2*/
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_GCTR);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], key_size);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- iv_inc2_dma_addr, AES_BLOCK_SIZE,
- NS_BIT);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE1);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_GCTR);
+ set_key_size_aes(&desc[idx], key_size);
+ set_din_type(&desc[idx], DMA_DLLI, iv_inc2_dma_addr, AES_BLOCK_SIZE,
+ NS_BIT);
+ set_cipher_config0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE1);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
-
///////////////////////////////// 4 ////////////////////////////////////
/* process(gctr+ghash) */
// if (req_ctx->cryptlen != 0)
-// ssi_aead_process_cipher_data_desc(req, cipher_flow_mode, desc, seq_size);
+// ssi_aead_process_cipher_data_desc(req, cipher_flow_mode, desc, seq_size);
///////////////////////////////// 4 ////////////////////////////////////
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- din_dma_addr, din_size,
- NS_BIT);
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- dout_dma_addr, din_size,
- NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], cipher_flow_mode);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, din_dma_addr, din_size, NS_BIT);
+ set_dout_dlli(&desc[idx], dout_dma_addr, din_size, NS_BIT, 0);
+ set_flow_mode(&desc[idx], cipher_flow_mode);
idx++;
-
///////////////////////////////// 5 ////////////////////////////////////
// ssi_aead_process_gcm_result_desc(req, desc, seq_size);
///////////////////////////////// 5 ////////////////////////////////////
/* prcess(ghash) gcm_block_len */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- block_len_dma_addr, AES_BLOCK_SIZE,
- NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_HASH);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, block_len_dma_addr, AES_BLOCK_SIZE,
+ NS_BIT);
+ set_flow_mode(&desc[idx], DIN_HASH);
idx++;
/* Store GHASH state after GHASH(Associated Data + Cipher +LenBlock) */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_HASH_HW_GHASH);
- HW_DESC_SET_DIN_NO_DMA(&desc[idx], 0, 0xfffff0);
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- mac_res_dma_addr, AES_BLOCK_SIZE,
- NS_BIT, 0);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_AES_NOT_HASH_MODE(&desc[idx]);
- idx++;
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_HASH_HW_GHASH);
+ set_din_no_dma(&desc[idx], 0, 0xfffff0);
+ set_dout_dlli(&desc[idx], mac_res_dma_addr, AES_BLOCK_SIZE, NS_BIT, 0);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_aes_not_hash_mode(&desc[idx]);
+ idx++;
/* load AES/CTR initial CTR value inc by 1*/
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_GCTR);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], key_size);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- iv_inc1_dma_addr, AES_BLOCK_SIZE,
- NS_BIT);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE1);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_GCTR);
+ set_key_size_aes(&desc[idx], key_size);
+ set_din_type(&desc[idx], DMA_DLLI, iv_inc1_dma_addr, AES_BLOCK_SIZE,
+ NS_BIT);
+ set_cipher_config0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE1);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
/* Memory Barrier */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_NO_DMA(&desc[idx], 0, 0xfffff0);
- HW_DESC_SET_DOUT_NO_DMA(&desc[idx], 0, 0, 1);
+ hw_desc_init(&desc[idx]);
+ set_din_no_dma(&desc[idx], 0, 0xfffff0);
+ set_dout_no_dma(&desc[idx], 0, 0, 1);
idx++;
/* process GCTR on stored GHASH and store MAC inplace */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_GCTR);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- mac_res_dma_addr, AES_BLOCK_SIZE,
- NS_BIT);
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- mac_res_dma_addr, AES_BLOCK_SIZE,
- NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_AES_DOUT);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_GCTR);
+ set_din_type(&desc[idx], DMA_DLLI, mac_res_dma_addr, AES_BLOCK_SIZE,
+ NS_BIT);
+ set_dout_dlli(&desc[idx], mac_res_dma_addr, AES_BLOCK_SIZE, NS_BIT, 0);
+ set_flow_mode(&desc[idx], DIN_AES_DOUT);
idx++;
/* perform the operation - Lock HW and push sequence */
return rc;
}
-ssi_fips_error_t
+enum cc_fips_error
ssi_gcm_fips_power_up_tests(struct ssi_drvdata *drvdata, void *cpu_addr_buffer, dma_addr_t dma_coherent_buffer)
{
- ssi_fips_error_t error = CC_REE_FIPS_ERROR_OK;
+ enum cc_fips_error error = CC_REE_FIPS_ERROR_OK;
size_t i;
struct fips_gcm_ctx *virt_ctx = (struct fips_gcm_ctx *)cpu_addr_buffer;
for (i = 0; i < FIPS_GCM_NUM_OF_TESTS; ++i)
{
- FipsGcmData *gcmData = (FipsGcmData*)&FipsGcmDataTable[i];
+ FipsGcmData *gcmData = (FipsGcmData *)&FipsGcmDataTable[i];
int rc = 0;
memset(cpu_addr_buffer, 0, sizeof(struct fips_gcm_ctx));
/* len_block */
{
__be64 len_bits;
+
len_bits = cpu_to_be64(gcmData->adataSize * 8);
memcpy(virt_ctx->len_block, &len_bits, sizeof(len_bits));
len_bits = cpu_to_be64(gcmData->dataInSize * 8);
/* iv_inc1, iv_inc2 */
{
__be32 counter = cpu_to_be32(1);
+
memcpy(virt_ctx->iv_inc1, gcmData->iv, NIST_AESGCM_IV_SIZE);
memcpy(virt_ctx->iv_inc1 + NIST_AESGCM_IV_SIZE, &counter, sizeof(counter));
counter = cpu_to_be32(2);
return error;
}
-
size_t ssi_fips_max_mem_alloc_size(void)
{
FIPS_DBG("sizeof(struct fips_cipher_ctx) %d \n", sizeof(struct fips_cipher_ctx));
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
/**************************************************************
-This file defines the driver FIPS internal function, used by the driver itself.
-***************************************************************/
+ * This file defines the driver FIPS internal function, used by the driver itself.
+ ***************************************************************/
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/platform_device.h>
#include "ssi_driver.h"
#include "cc_hal.h"
-
#define FIPS_POWER_UP_TEST_CIPHER 1
#define FIPS_POWER_UP_TEST_CMAC 1
#define FIPS_POWER_UP_TEST_HASH 1
#endif
};
-
-extern int ssi_fips_get_state(ssi_fips_state_t *p_state);
-extern int ssi_fips_get_error(ssi_fips_error_t *p_err);
-extern int ssi_fips_ext_set_state(ssi_fips_state_t state);
-extern int ssi_fips_ext_set_error(ssi_fips_error_t err);
+extern int ssi_fips_get_state(enum cc_fips_state_t *p_state);
+extern int ssi_fips_get_error(enum cc_fips_error *p_err);
+extern int ssi_fips_ext_set_state(enum cc_fips_state_t state);
+extern int ssi_fips_ext_set_error(enum cc_fips_error err);
/* FIPS power-up tests */
-extern ssi_fips_error_t ssi_cipher_fips_power_up_tests(struct ssi_drvdata *drvdata, void *cpu_addr_buffer, dma_addr_t dma_coherent_buffer);
-extern ssi_fips_error_t ssi_cmac_fips_power_up_tests(struct ssi_drvdata *drvdata, void *cpu_addr_buffer, dma_addr_t dma_coherent_buffer);
-extern ssi_fips_error_t ssi_hash_fips_power_up_tests(struct ssi_drvdata *drvdata, void *cpu_addr_buffer, dma_addr_t dma_coherent_buffer);
-extern ssi_fips_error_t ssi_hmac_fips_power_up_tests(struct ssi_drvdata *drvdata, void *cpu_addr_buffer, dma_addr_t dma_coherent_buffer);
-extern ssi_fips_error_t ssi_ccm_fips_power_up_tests(struct ssi_drvdata *drvdata, void *cpu_addr_buffer, dma_addr_t dma_coherent_buffer);
-extern ssi_fips_error_t ssi_gcm_fips_power_up_tests(struct ssi_drvdata *drvdata, void *cpu_addr_buffer, dma_addr_t dma_coherent_buffer);
+extern enum cc_fips_error ssi_cipher_fips_power_up_tests(struct ssi_drvdata *drvdata, void *cpu_addr_buffer, dma_addr_t dma_coherent_buffer);
+extern enum cc_fips_error ssi_cmac_fips_power_up_tests(struct ssi_drvdata *drvdata, void *cpu_addr_buffer, dma_addr_t dma_coherent_buffer);
+extern enum cc_fips_error ssi_hash_fips_power_up_tests(struct ssi_drvdata *drvdata, void *cpu_addr_buffer, dma_addr_t dma_coherent_buffer);
+extern enum cc_fips_error ssi_hmac_fips_power_up_tests(struct ssi_drvdata *drvdata, void *cpu_addr_buffer, dma_addr_t dma_coherent_buffer);
+extern enum cc_fips_error ssi_ccm_fips_power_up_tests(struct ssi_drvdata *drvdata, void *cpu_addr_buffer, dma_addr_t dma_coherent_buffer);
+extern enum cc_fips_error ssi_gcm_fips_power_up_tests(struct ssi_drvdata *drvdata, void *cpu_addr_buffer, dma_addr_t dma_coherent_buffer);
extern size_t ssi_fips_max_mem_alloc_size(void);
-
/* The function called once at driver entry point to check whether TEE FIPS error occured.*/
static enum ssi_fips_error ssi_fips_get_tee_error(struct ssi_drvdata *drvdata)
{
- uint32_t regVal;
+ u32 regVal;
void __iomem *cc_base = drvdata->cc_base;
regVal = CC_HAL_READ_REGISTER(CC_REG_OFFSET(HOST_RGF, GPR_HOST));
- if (regVal == (CC_FIPS_SYNC_TEE_STATUS | CC_FIPS_SYNC_MODULE_OK)) {
+ if (regVal == (CC_FIPS_SYNC_TEE_STATUS | CC_FIPS_SYNC_MODULE_OK))
return CC_REE_FIPS_ERROR_OK;
- }
+
return CC_REE_FIPS_ERROR_FROM_TEE;
}
-
-/*
- This function should push the FIPS REE library status towards the TEE library.
- By writing the error state to HOST_GPR0 register. The function is called from .
- driver entry point so no need to protect by mutex.
-*/
-static void ssi_fips_update_tee_upon_ree_status(struct ssi_drvdata *drvdata, ssi_fips_error_t err)
+/*
+ * This function should push the FIPS REE library status towards the TEE library.
+ * By writing the error state to HOST_GPR0 register. The function is called from
+ * driver entry point so no need to protect by mutex.
+ */
+static void ssi_fips_update_tee_upon_ree_status(struct ssi_drvdata *drvdata, enum cc_fips_error err)
{
void __iomem *cc_base = drvdata->cc_base;
- if (err == CC_REE_FIPS_ERROR_OK) {
- CC_HAL_WRITE_REGISTER(CC_REG_OFFSET(HOST_RGF, HOST_GPR0), (CC_FIPS_SYNC_REE_STATUS|CC_FIPS_SYNC_MODULE_OK));
- } else {
- CC_HAL_WRITE_REGISTER(CC_REG_OFFSET(HOST_RGF, HOST_GPR0), (CC_FIPS_SYNC_REE_STATUS|CC_FIPS_SYNC_MODULE_ERROR));
- }
-}
-
+ if (err == CC_REE_FIPS_ERROR_OK)
+ CC_HAL_WRITE_REGISTER(CC_REG_OFFSET(HOST_RGF, HOST_GPR0), (CC_FIPS_SYNC_REE_STATUS | CC_FIPS_SYNC_MODULE_OK));
+ else
+ CC_HAL_WRITE_REGISTER(CC_REG_OFFSET(HOST_RGF, HOST_GPR0), (CC_FIPS_SYNC_REE_STATUS | CC_FIPS_SYNC_MODULE_ERROR));
+}
void ssi_fips_fini(struct ssi_drvdata *drvdata)
{
struct ssi_fips_handle *fips_h = drvdata->fips_handle;
- if (fips_h == NULL)
+ if (!fips_h)
return; /* Not allocated */
#ifdef COMP_IN_WQ
- if (fips_h->workq != NULL) {
+ if (fips_h->workq) {
flush_workqueue(fips_h->workq);
destroy_workqueue(fips_h->workq);
}
void fips_handler(struct ssi_drvdata *drvdata)
{
- struct ssi_fips_handle *fips_handle_ptr =
+ struct ssi_fips_handle *fips_handle_ptr =
drvdata->fips_handle;
#ifdef COMP_IN_WQ
queue_delayed_work(fips_handle_ptr->workq, &fips_handle_ptr->fipswork, 0);
#endif
}
-
-
#ifdef COMP_IN_WQ
static void fips_wq_handler(struct work_struct *work)
{
{
struct ssi_drvdata *drvdata = (struct ssi_drvdata *)devarg;
void __iomem *cc_base = drvdata->cc_base;
- uint32_t irq;
- uint32_t teeFipsError = 0;
+ u32 irq;
+ u32 teeFipsError = 0;
irq = (drvdata->irq & (SSI_GPR0_IRQ_MASK));
if (irq & SSI_GPR0_IRQ_MASK) {
teeFipsError = CC_HAL_READ_REGISTER(CC_REG_OFFSET(HOST_RGF, GPR_HOST));
- if (teeFipsError != (CC_FIPS_SYNC_TEE_STATUS | CC_FIPS_SYNC_MODULE_OK)) {
+ if (teeFipsError != (CC_FIPS_SYNC_TEE_STATUS | CC_FIPS_SYNC_MODULE_OK))
ssi_fips_set_error(drvdata, CC_REE_FIPS_ERROR_FROM_TEE);
- }
}
/* after verifing that there is nothing to do, Unmask AXI completion interrupt */
- CC_HAL_WRITE_REGISTER(CC_REG_OFFSET(HOST_RGF, HOST_IMR),
+ CC_HAL_WRITE_REGISTER(CC_REG_OFFSET(HOST_RGF, HOST_IMR),
CC_HAL_READ_REGISTER(
CC_REG_OFFSET(HOST_RGF, HOST_IMR)) & ~irq);
}
-
-ssi_fips_error_t cc_fips_run_power_up_tests(struct ssi_drvdata *drvdata)
+enum cc_fips_error cc_fips_run_power_up_tests(struct ssi_drvdata *drvdata)
{
- ssi_fips_error_t fips_error = CC_REE_FIPS_ERROR_OK;
- void * cpu_addr_buffer = NULL;
+ enum cc_fips_error fips_error = CC_REE_FIPS_ERROR_OK;
+ void *cpu_addr_buffer = NULL;
dma_addr_t dma_handle;
size_t alloc_buff_size = ssi_fips_max_mem_alloc_size();
struct device *dev = &drvdata->plat_dev->dev;
// the dma_handle is the returned phy address - use it in the HW descriptor
FIPS_DBG("dma_alloc_coherent \n");
cpu_addr_buffer = dma_alloc_coherent(dev, alloc_buff_size, &dma_handle, GFP_KERNEL);
- if (cpu_addr_buffer == NULL) {
+ if (!cpu_addr_buffer)
return CC_REE_FIPS_ERROR_GENERAL;
- }
+
FIPS_DBG("allocated coherent buffer - addr 0x%08X , size = %d \n", (size_t)cpu_addr_buffer, alloc_buff_size);
#if FIPS_POWER_UP_TEST_CIPHER
return fips_error;
}
-
-
-/* The function checks if FIPS supported and FIPS error exists.*
-* It should be used in every driver API.*/
+/* The function checks if FIPS supported and FIPS error exists.*
+ * It should be used in every driver API.
+ */
int ssi_fips_check_fips_error(void)
{
- ssi_fips_state_t fips_state;
+ enum cc_fips_state_t fips_state;
if (ssi_fips_get_state(&fips_state) != 0) {
FIPS_LOG("ssi_fips_get_state FAILED, returning.. \n");
return 0;
}
-
-/* The function sets the REE FIPS state.*
-* It should be used while driver is being loaded .*/
-int ssi_fips_set_state(ssi_fips_state_t state)
+/* The function sets the REE FIPS state.*
+ * It should be used while driver is being loaded.
+ */
+int ssi_fips_set_state(enum cc_fips_state_t state)
{
return ssi_fips_ext_set_state(state);
}
-/* The function sets the REE FIPS error, and pushes the error to TEE library. *
-* It should be used when any of the KAT tests fails .*/
-int ssi_fips_set_error(struct ssi_drvdata *p_drvdata, ssi_fips_error_t err)
+/* The function sets the REE FIPS error, and pushes the error to TEE library. *
+ * It should be used when any of the KAT tests fails.
+ */
+int ssi_fips_set_error(struct ssi_drvdata *p_drvdata, enum cc_fips_error err)
{
int rc = 0;
- ssi_fips_error_t current_err;
+ enum cc_fips_error current_err;
- FIPS_LOG("ssi_fips_set_error - fips_error = %d \n", err);
+ FIPS_LOG("ssi_fips_set_error - fips_error = %d \n", err);
// setting no error is not allowed
- if (err == CC_REE_FIPS_ERROR_OK) {
- return -ENOEXEC;
- }
- // If error exists, do not set new error
- if (ssi_fips_get_error(¤t_err) != 0) {
- return -ENOEXEC;
- }
- if (current_err != CC_REE_FIPS_ERROR_OK) {
- return -ENOEXEC;
- }
- // set REE internal error and state
+ if (err == CC_REE_FIPS_ERROR_OK)
+ return -ENOEXEC;
+
+ // If error exists, do not set new error
+ if (ssi_fips_get_error(¤t_err) != 0)
+ return -ENOEXEC;
+
+ if (current_err != CC_REE_FIPS_ERROR_OK)
+ return -ENOEXEC;
+
+ // set REE internal error and state
rc = ssi_fips_ext_set_error(err);
- if (rc != 0) {
- return -ENOEXEC;
- }
+ if (rc != 0)
+ return -ENOEXEC;
+
rc = ssi_fips_ext_set_state(CC_FIPS_STATE_ERROR);
- if (rc != 0) {
- return -ENOEXEC;
- }
+ if (rc != 0)
+ return -ENOEXEC;
- // push error towards TEE libraray, if it's not TEE error
- if (err != CC_REE_FIPS_ERROR_FROM_TEE) {
+ // push error towards TEE libraray, if it's not TEE error
+ if (err != CC_REE_FIPS_ERROR_FROM_TEE)
ssi_fips_update_tee_upon_ree_status(p_drvdata, err);
- }
+
return rc;
}
-
/* The function called once at driver entry point .*/
int ssi_fips_init(struct ssi_drvdata *p_drvdata)
{
- ssi_fips_error_t rc = CC_REE_FIPS_ERROR_OK;
+ enum cc_fips_error rc = CC_REE_FIPS_ERROR_OK;
struct ssi_fips_handle *fips_h;
FIPS_DBG("CC FIPS code .. (fips=%d) \n", ssi_fips_support);
- fips_h = kzalloc(sizeof(struct ssi_fips_handle),GFP_KERNEL);
- if (fips_h == NULL) {
+ fips_h = kzalloc(sizeof(struct ssi_fips_handle), GFP_KERNEL);
+ if (!fips_h) {
ssi_fips_set_error(p_drvdata, CC_REE_FIPS_ERROR_GENERAL);
return -ENOMEM;
}
#ifdef COMP_IN_WQ
SSI_LOG_DEBUG("Initializing fips workqueue\n");
fips_h->workq = create_singlethread_workqueue("arm_cc7x_fips_wq");
- if (unlikely(fips_h->workq == NULL)) {
+ if (unlikely(!fips_h->workq)) {
SSI_LOG_ERR("Failed creating fips work queue\n");
ssi_fips_set_error(p_drvdata, CC_REE_FIPS_ERROR_GENERAL);
rc = -ENOMEM;
#endif
/* init fips driver data */
- rc = ssi_fips_set_state((ssi_fips_support == 0)? CC_FIPS_STATE_NOT_SUPPORTED : CC_FIPS_STATE_SUPPORTED);
+ rc = ssi_fips_set_state((ssi_fips_support == 0) ? CC_FIPS_STATE_NOT_SUPPORTED : CC_FIPS_STATE_SUPPORTED);
if (unlikely(rc != 0)) {
ssi_fips_set_error(p_drvdata, CC_REE_FIPS_ERROR_GENERAL);
rc = -EAGAIN;
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
#ifndef __SSI_FIPS_LOCAL_H__
#define __SSI_FIPS_LOCAL_H__
-
#ifdef CONFIG_CCX7REE_FIPS_SUPPORT
#include "ssi_fips.h"
struct ssi_drvdata;
-// IG - how to make 1 file for TEE and REE
-typedef enum CC_FipsSyncStatus{
- CC_FIPS_SYNC_MODULE_OK = 0x0,
- CC_FIPS_SYNC_MODULE_ERROR = 0x1,
- CC_FIPS_SYNC_REE_STATUS = 0x4,
- CC_FIPS_SYNC_TEE_STATUS = 0x8,
- CC_FIPS_SYNC_STATUS_RESERVE32B = INT32_MAX
-}CCFipsSyncStatus_t;
-
-
#define CHECK_AND_RETURN_UPON_FIPS_ERROR() {\
- if (ssi_fips_check_fips_error() != 0) {\
- return -ENOEXEC;\
- }\
+ if (ssi_fips_check_fips_error() != 0) {\
+ return -ENOEXEC;\
+ } \
}
+
#define CHECK_AND_RETURN_VOID_UPON_FIPS_ERROR() {\
- if (ssi_fips_check_fips_error() != 0) {\
- return;\
- }\
+ if (ssi_fips_check_fips_error() != 0) {\
+ return;\
+ } \
}
+
#define SSI_FIPS_INIT(p_drvData) (ssi_fips_init(p_drvData))
#define SSI_FIPS_FINI(p_drvData) (ssi_fips_fini(p_drvData))
int ssi_fips_init(struct ssi_drvdata *p_drvdata);
void ssi_fips_fini(struct ssi_drvdata *drvdata);
int ssi_fips_check_fips_error(void);
-int ssi_fips_set_error(struct ssi_drvdata *p_drvdata, ssi_fips_error_t err);
+int ssi_fips_set_error(struct ssi_drvdata *p_drvdata, enum cc_fips_error err);
void fips_handler(struct ssi_drvdata *drvdata);
#else /* CONFIG_CC7XXREE_FIPS_SUPPORT */
#endif /* CONFIG_CC7XXREE_FIPS_SUPPORT */
-
#endif /*__SSI_FIPS_LOCAL_H__*/
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
struct completion init_comp;
};
-static const uint32_t digest_len_init[] = {
+static const u32 digest_len_init[] = {
0x00000040, 0x00000000, 0x00000000, 0x00000000 };
-static const uint32_t md5_init[] = {
+static const u32 md5_init[] = {
SHA1_H3, SHA1_H2, SHA1_H1, SHA1_H0 };
-static const uint32_t sha1_init[] = {
+static const u32 sha1_init[] = {
SHA1_H4, SHA1_H3, SHA1_H2, SHA1_H1, SHA1_H0 };
-static const uint32_t sha224_init[] = {
+static const u32 sha224_init[] = {
SHA224_H7, SHA224_H6, SHA224_H5, SHA224_H4,
SHA224_H3, SHA224_H2, SHA224_H1, SHA224_H0 };
-static const uint32_t sha256_init[] = {
+static const u32 sha256_init[] = {
SHA256_H7, SHA256_H6, SHA256_H5, SHA256_H4,
SHA256_H3, SHA256_H2, SHA256_H1, SHA256_H0 };
#if (DX_DEV_SHA_MAX > 256)
-static const uint32_t digest_len_sha512_init[] = {
+static const u32 digest_len_sha512_init[] = {
0x00000080, 0x00000000, 0x00000000, 0x00000000 };
-static const uint64_t sha384_init[] = {
+static const u64 sha384_init[] = {
SHA384_H7, SHA384_H6, SHA384_H5, SHA384_H4,
SHA384_H3, SHA384_H2, SHA384_H1, SHA384_H0 };
-static const uint64_t sha512_init[] = {
+static const u64 sha512_init[] = {
SHA512_H7, SHA512_H6, SHA512_H5, SHA512_H4,
SHA512_H3, SHA512_H2, SHA512_H1, SHA512_H0 };
#endif
static void ssi_hash_create_xcbc_setup(
- struct ahash_request *areq,
- HwDesc_s desc[],
+ struct ahash_request *areq,
+ struct cc_hw_desc desc[],
unsigned int *seq_size);
-static void ssi_hash_create_cmac_setup(struct ahash_request *areq,
- HwDesc_s desc[],
+static void ssi_hash_create_cmac_setup(struct ahash_request *areq,
+ struct cc_hw_desc desc[],
unsigned int *seq_size);
struct ssi_hash_alg {
struct list_head entry;
- bool synchronize;
int hash_mode;
int hw_mode;
int inter_digestsize;
struct ssi_drvdata *drvdata;
- union {
- struct ahash_alg ahash_alg;
- struct shash_alg shash_alg;
- };
+ struct ahash_alg ahash_alg;
};
-
struct hash_key_req_ctx {
- uint32_t keylen;
+ u32 keylen;
dma_addr_t key_dma_addr;
};
/* hash per-session context */
struct ssi_hash_ctx {
struct ssi_drvdata *drvdata;
- /* holds the origin digest; the digest after "setkey" if HMAC,*
- the initial digest if HASH. */
- uint8_t digest_buff[SSI_MAX_HASH_DIGEST_SIZE] ____cacheline_aligned;
- uint8_t opad_tmp_keys_buff[SSI_MAX_HASH_OPAD_TMP_KEYS_SIZE] ____cacheline_aligned;
+ /* holds the origin digest; the digest after "setkey" if HMAC,*
+ * the initial digest if HASH.
+ */
+ u8 digest_buff[SSI_MAX_HASH_DIGEST_SIZE] ____cacheline_aligned;
+ u8 opad_tmp_keys_buff[SSI_MAX_HASH_OPAD_TMP_KEYS_SIZE] ____cacheline_aligned;
+
dma_addr_t opad_tmp_keys_dma_addr ____cacheline_aligned;
dma_addr_t digest_buff_dma_addr;
/* use for hmac with key large then mode block size */
bool is_hmac;
};
-static const struct crypto_type crypto_shash_type;
-
static void ssi_hash_create_data_desc(
struct ahash_req_ctx *areq_ctx,
- struct ssi_hash_ctx *ctx,
- unsigned int flow_mode,HwDesc_s desc[],
+ struct ssi_hash_ctx *ctx,
+ unsigned int flow_mode, struct cc_hw_desc desc[],
bool is_not_last_data,
unsigned int *seq_size);
-static inline void ssi_set_hash_endianity(uint32_t mode, HwDesc_s *desc)
+static inline void ssi_set_hash_endianity(u32 mode, struct cc_hw_desc *desc)
{
if (unlikely((mode == DRV_HASH_MD5) ||
(mode == DRV_HASH_SHA384) ||
(mode == DRV_HASH_SHA512))) {
- HW_DESC_SET_BYTES_SWAP(desc, 1);
+ set_bytes_swap(desc, 1);
} else {
- HW_DESC_SET_CIPHER_CONFIG0(desc, HASH_DIGEST_RESULT_LITTLE_ENDIAN);
+ set_cipher_config0(desc, HASH_DIGEST_RESULT_LITTLE_ENDIAN);
}
}
-static int ssi_hash_map_result(struct device *dev,
- struct ahash_req_ctx *state,
+static int ssi_hash_map_result(struct device *dev,
+ struct ahash_req_ctx *state,
unsigned int digestsize)
{
- state->digest_result_dma_addr =
+ state->digest_result_dma_addr =
dma_map_single(dev, (void *)state->digest_result_buff,
digestsize,
DMA_BIDIRECTIONAL);
digestsize);
return -ENOMEM;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(state->digest_result_dma_addr,
- digestsize);
SSI_LOG_DEBUG("Mapped digest result buffer %u B "
"at va=%pK to dma=0x%llX\n",
digestsize, state->digest_result_buff,
return 0;
}
-static int ssi_hash_map_request(struct device *dev,
- struct ahash_req_ctx *state,
+static int ssi_hash_map_request(struct device *dev,
+ struct ahash_req_ctx *state,
struct ssi_hash_ctx *ctx)
{
bool is_hmac = ctx->is_hmac;
ssi_sram_addr_t larval_digest_addr = ssi_ahash_get_larval_digest_sram_addr(
ctx->drvdata, ctx->hash_mode);
struct ssi_crypto_req ssi_req = {};
- HwDesc_s desc;
+ struct cc_hw_desc desc;
int rc = -ENOMEM;
- state->buff0 = kzalloc(SSI_MAX_HASH_BLCK_SIZE ,GFP_KERNEL|GFP_DMA);
+ state->buff0 = kzalloc(SSI_MAX_HASH_BLCK_SIZE, GFP_KERNEL | GFP_DMA);
if (!state->buff0) {
SSI_LOG_ERR("Allocating buff0 in context failed\n");
goto fail0;
}
- state->buff1 = kzalloc(SSI_MAX_HASH_BLCK_SIZE ,GFP_KERNEL|GFP_DMA);
+ state->buff1 = kzalloc(SSI_MAX_HASH_BLCK_SIZE, GFP_KERNEL | GFP_DMA);
if (!state->buff1) {
SSI_LOG_ERR("Allocating buff1 in context failed\n");
goto fail_buff0;
}
- state->digest_result_buff = kzalloc(SSI_MAX_HASH_DIGEST_SIZE ,GFP_KERNEL|GFP_DMA);
+ state->digest_result_buff = kzalloc(SSI_MAX_HASH_DIGEST_SIZE, GFP_KERNEL | GFP_DMA);
if (!state->digest_result_buff) {
SSI_LOG_ERR("Allocating digest_result_buff in context failed\n");
goto fail_buff1;
}
- state->digest_buff = kzalloc(ctx->inter_digestsize, GFP_KERNEL|GFP_DMA);
+ state->digest_buff = kzalloc(ctx->inter_digestsize, GFP_KERNEL | GFP_DMA);
if (!state->digest_buff) {
SSI_LOG_ERR("Allocating digest-buffer in context failed\n");
goto fail_digest_result_buff;
SSI_LOG_DEBUG("Allocated digest-buffer in context ctx->digest_buff=@%p\n", state->digest_buff);
if (ctx->hw_mode != DRV_CIPHER_XCBC_MAC) {
- state->digest_bytes_len = kzalloc(HASH_LEN_SIZE, GFP_KERNEL|GFP_DMA);
+ state->digest_bytes_len = kzalloc(HASH_LEN_SIZE, GFP_KERNEL | GFP_DMA);
if (!state->digest_bytes_len) {
SSI_LOG_ERR("Allocating digest-bytes-len in context failed\n");
goto fail1;
state->digest_bytes_len = NULL;
}
- state->opad_digest_buff = kzalloc(ctx->inter_digestsize, GFP_KERNEL|GFP_DMA);
+ state->opad_digest_buff = kzalloc(ctx->inter_digestsize, GFP_KERNEL | GFP_DMA);
if (!state->opad_digest_buff) {
SSI_LOG_ERR("Allocating opad-digest-buffer in context failed\n");
goto fail2;
ctx->inter_digestsize, state->digest_buff);
goto fail3;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(state->digest_buff_dma_addr,
- ctx->inter_digestsize);
SSI_LOG_DEBUG("Mapped digest %d B at va=%pK to dma=0x%llX\n",
ctx->inter_digestsize, state->digest_buff,
(unsigned long long)state->digest_buff_dma_addr);
if (is_hmac) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(ctx->digest_buff_dma_addr);
dma_sync_single_for_cpu(dev, ctx->digest_buff_dma_addr, ctx->inter_digestsize, DMA_BIDIRECTIONAL);
- SSI_UPDATE_DMA_ADDR_TO_48BIT(ctx->digest_buff_dma_addr,
- ctx->inter_digestsize);
if ((ctx->hw_mode == DRV_CIPHER_XCBC_MAC) || (ctx->hw_mode == DRV_CIPHER_CMAC)) {
memset(state->digest_buff, 0, ctx->inter_digestsize);
} else { /*sha*/
memcpy(state->digest_buff, ctx->digest_buff, ctx->inter_digestsize);
#if (DX_DEV_SHA_MAX > 256)
- if (unlikely((ctx->hash_mode == DRV_HASH_SHA512) || (ctx->hash_mode == DRV_HASH_SHA384))) {
+ if (unlikely((ctx->hash_mode == DRV_HASH_SHA512) || (ctx->hash_mode == DRV_HASH_SHA384)))
memcpy(state->digest_bytes_len, digest_len_sha512_init, HASH_LEN_SIZE);
- } else {
+ else
memcpy(state->digest_bytes_len, digest_len_init, HASH_LEN_SIZE);
- }
#else
memcpy(state->digest_bytes_len, digest_len_init, HASH_LEN_SIZE);
#endif
}
- SSI_RESTORE_DMA_ADDR_TO_48BIT(state->digest_buff_dma_addr);
dma_sync_single_for_device(dev, state->digest_buff_dma_addr, ctx->inter_digestsize, DMA_BIDIRECTIONAL);
- SSI_UPDATE_DMA_ADDR_TO_48BIT(state->digest_buff_dma_addr,
- ctx->inter_digestsize);
if (ctx->hash_mode != DRV_HASH_NULL) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(ctx->opad_tmp_keys_dma_addr);
dma_sync_single_for_cpu(dev, ctx->opad_tmp_keys_dma_addr, ctx->inter_digestsize, DMA_BIDIRECTIONAL);
memcpy(state->opad_digest_buff, ctx->opad_tmp_keys_buff, ctx->inter_digestsize);
- SSI_UPDATE_DMA_ADDR_TO_48BIT(ctx->opad_tmp_keys_dma_addr,
- ctx->inter_digestsize);
- }
+ }
} else { /*hash*/
/* Copy the initial digests if hash flow. The SRAM contains the
- initial digests in the expected order for all SHA* */
- HW_DESC_INIT(&desc);
- HW_DESC_SET_DIN_SRAM(&desc, larval_digest_addr, ctx->inter_digestsize);
- HW_DESC_SET_DOUT_DLLI(&desc, state->digest_buff_dma_addr, ctx->inter_digestsize, NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc, BYPASS);
+ * initial digests in the expected order for all SHA*
+ */
+ hw_desc_init(&desc);
+ set_din_sram(&desc, larval_digest_addr, ctx->inter_digestsize);
+ set_dout_dlli(&desc, state->digest_buff_dma_addr,
+ ctx->inter_digestsize, NS_BIT, 0);
+ set_flow_mode(&desc, BYPASS);
rc = send_request(ctx->drvdata, &ssi_req, &desc, 1, 0);
if (unlikely(rc != 0)) {
HASH_LEN_SIZE, state->digest_bytes_len);
goto fail4;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(state->digest_bytes_len_dma_addr,
- HASH_LEN_SIZE);
SSI_LOG_DEBUG("Mapped digest len %u B at va=%pK to dma=0x%llX\n",
HASH_LEN_SIZE, state->digest_bytes_len,
(unsigned long long)state->digest_bytes_len_dma_addr);
ctx->inter_digestsize, state->opad_digest_buff);
goto fail5;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(state->opad_digest_dma_addr,
- ctx->inter_digestsize);
SSI_LOG_DEBUG("Mapped opad digest %d B at va=%pK to dma=0x%llX\n",
ctx->inter_digestsize, state->opad_digest_buff,
(unsigned long long)state->opad_digest_dma_addr);
fail5:
if (state->digest_bytes_len_dma_addr != 0) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(state->digest_bytes_len_dma_addr);
dma_unmap_single(dev, state->digest_bytes_len_dma_addr, HASH_LEN_SIZE, DMA_BIDIRECTIONAL);
state->digest_bytes_len_dma_addr = 0;
}
fail4:
if (state->digest_buff_dma_addr != 0) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(state->digest_buff_dma_addr);
dma_unmap_single(dev, state->digest_buff_dma_addr, ctx->inter_digestsize, DMA_BIDIRECTIONAL);
state->digest_buff_dma_addr = 0;
}
fail1:
kfree(state->digest_buff);
fail_digest_result_buff:
- if (state->digest_result_buff != NULL) {
+ if (state->digest_result_buff) {
kfree(state->digest_result_buff);
state->digest_result_buff = NULL;
}
fail_buff1:
- if (state->buff1 != NULL) {
+ if (state->buff1) {
kfree(state->buff1);
state->buff1 = NULL;
}
fail_buff0:
- if (state->buff0 != NULL) {
+ if (state->buff0) {
kfree(state->buff0);
state->buff0 = NULL;
}
return rc;
}
-static void ssi_hash_unmap_request(struct device *dev,
- struct ahash_req_ctx *state,
+static void ssi_hash_unmap_request(struct device *dev,
+ struct ahash_req_ctx *state,
struct ssi_hash_ctx *ctx)
{
if (state->digest_buff_dma_addr != 0) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(state->digest_buff_dma_addr);
dma_unmap_single(dev, state->digest_buff_dma_addr,
ctx->inter_digestsize, DMA_BIDIRECTIONAL);
SSI_LOG_DEBUG("Unmapped digest-buffer: digest_buff_dma_addr=0x%llX\n",
state->digest_buff_dma_addr = 0;
}
if (state->digest_bytes_len_dma_addr != 0) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(state->digest_bytes_len_dma_addr);
dma_unmap_single(dev, state->digest_bytes_len_dma_addr,
HASH_LEN_SIZE, DMA_BIDIRECTIONAL);
SSI_LOG_DEBUG("Unmapped digest-bytes-len buffer: digest_bytes_len_dma_addr=0x%llX\n",
state->digest_bytes_len_dma_addr = 0;
}
if (state->opad_digest_dma_addr != 0) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(state->opad_digest_dma_addr);
dma_unmap_single(dev, state->opad_digest_dma_addr,
ctx->inter_digestsize, DMA_BIDIRECTIONAL);
SSI_LOG_DEBUG("Unmapped opad-digest: opad_digest_dma_addr=0x%llX\n",
kfree(state->buff0);
}
-static void ssi_hash_unmap_result(struct device *dev,
- struct ahash_req_ctx *state,
+static void ssi_hash_unmap_result(struct device *dev,
+ struct ahash_req_ctx *state,
unsigned int digestsize, u8 *result)
{
if (state->digest_result_dma_addr != 0) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(state->digest_result_dma_addr);
dma_unmap_single(dev,
state->digest_result_dma_addr,
digestsize,
- DMA_BIDIRECTIONAL);
+ DMA_BIDIRECTIONAL);
SSI_LOG_DEBUG("unmpa digest result buffer "
"va (%pK) pa (%llx) len %u\n",
- state->digest_result_buff,
+ state->digest_result_buff,
(unsigned long long)state->digest_result_dma_addr,
digestsize);
memcpy(result,
struct ahash_req_ctx *state = ahash_request_ctx(req);
struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
struct ssi_hash_ctx *ctx = crypto_ahash_ctx(tfm);
- uint32_t digestsize = crypto_ahash_digestsize(tfm);
-
+ u32 digestsize = crypto_ahash_digestsize(tfm);
+
SSI_LOG_DEBUG("req=%pK\n", req);
ssi_buffer_mgr_unmap_hash_request(dev, state, req->src, false);
struct ahash_req_ctx *state = ahash_request_ctx(req);
struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
struct ssi_hash_ctx *ctx = crypto_ahash_ctx(tfm);
- uint32_t digestsize = crypto_ahash_digestsize(tfm);
-
+ u32 digestsize = crypto_ahash_digestsize(tfm);
+
SSI_LOG_DEBUG("req=%pK\n", req);
ssi_buffer_mgr_unmap_hash_request(dev, state, req->src, false);
req->base.complete(&req->base, 0);
}
-static int ssi_hash_digest(struct ahash_req_ctx *state,
- struct ssi_hash_ctx *ctx,
- unsigned int digestsize,
- struct scatterlist *src,
- unsigned int nbytes, u8 *result,
+static int ssi_hash_digest(struct ahash_req_ctx *state,
+ struct ssi_hash_ctx *ctx,
+ unsigned int digestsize,
+ struct scatterlist *src,
+ unsigned int nbytes, u8 *result,
void *async_req)
{
struct device *dev = &ctx->drvdata->plat_dev->dev;
bool is_hmac = ctx->is_hmac;
struct ssi_crypto_req ssi_req = {};
- HwDesc_s desc[SSI_MAX_AHASH_SEQ_LEN];
+ struct cc_hw_desc desc[SSI_MAX_AHASH_SEQ_LEN];
ssi_sram_addr_t larval_digest_addr = ssi_ahash_get_larval_digest_sram_addr(
ctx->drvdata, ctx->hash_mode);
int idx = 0;
int rc = 0;
-
- SSI_LOG_DEBUG("===== %s-digest (%d) ====\n", is_hmac?"hmac":"hash", nbytes);
+ SSI_LOG_DEBUG("===== %s-digest (%d) ====\n", is_hmac ? "hmac" : "hash", nbytes);
CHECK_AND_RETURN_UPON_FIPS_ERROR();
/* Setup DX request structure */
ssi_req.user_cb = (void *)ssi_hash_digest_complete;
ssi_req.user_arg = (void *)async_req;
-#ifdef ENABLE_CYCLE_COUNT
- ssi_req.op_type = STAT_OP_TYPE_ENCODE; /* Use "Encode" stats */
-#endif
}
/* If HMAC then load hash IPAD xor key, if HASH then load initial digest */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
if (is_hmac) {
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr, ctx->inter_digestsize, NS_BIT);
+ set_din_type(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr,
+ ctx->inter_digestsize, NS_BIT);
} else {
- HW_DESC_SET_DIN_SRAM(&desc[idx], larval_digest_addr, ctx->inter_digestsize);
+ set_din_sram(&desc[idx], larval_digest_addr,
+ ctx->inter_digestsize);
}
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
idx++;
/* Load the hash current length */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
if (is_hmac) {
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, state->digest_bytes_len_dma_addr, HASH_LEN_SIZE, NS_BIT);
+ set_din_type(&desc[idx], DMA_DLLI,
+ state->digest_bytes_len_dma_addr, HASH_LEN_SIZE,
+ NS_BIT);
} else {
- HW_DESC_SET_DIN_CONST(&desc[idx], 0, HASH_LEN_SIZE);
- if (likely(nbytes != 0)) {
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_ENABLED);
- } else {
- HW_DESC_SET_CIPHER_DO(&desc[idx], DO_PAD);
- }
- }
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ set_din_const(&desc[idx], 0, HASH_LEN_SIZE);
+ if (likely(nbytes != 0))
+ set_cipher_config1(&desc[idx], HASH_PADDING_ENABLED);
+ else
+ set_cipher_do(&desc[idx], DO_PAD);
+ }
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
ssi_hash_create_data_desc(state, ctx, DIN_HASH, desc, false, &idx);
if (is_hmac) {
/* HW last hash block padding (aka. "DO_PAD") */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], state->digest_buff_dma_addr, HASH_LEN_SIZE, NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE1);
- HW_DESC_SET_CIPHER_DO(&desc[idx], DO_PAD);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_dout_dlli(&desc[idx], state->digest_buff_dma_addr,
+ HASH_LEN_SIZE, NS_BIT, 0);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE1);
+ set_cipher_do(&desc[idx], DO_PAD);
idx++;
/* store the hash digest result in the context */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], state->digest_buff_dma_addr, digestsize, NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_dout_dlli(&desc[idx], state->digest_buff_dma_addr,
+ digestsize, NS_BIT, 0);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
ssi_set_hash_endianity(ctx->hash_mode, &desc[idx]);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
idx++;
/* Loading hash opad xor key state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, state->opad_digest_dma_addr, ctx->inter_digestsize, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_din_type(&desc[idx], DMA_DLLI, state->opad_digest_dma_addr,
+ ctx->inter_digestsize, NS_BIT);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
idx++;
/* Load the hash current length */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DIN_SRAM(&desc[idx], ssi_ahash_get_initial_digest_len_sram_addr(ctx->drvdata, ctx->hash_mode), HASH_LEN_SIZE);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_ENABLED);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_din_sram(&desc[idx],
+ ssi_ahash_get_initial_digest_len_sram_addr(
+ctx->drvdata, ctx->hash_mode), HASH_LEN_SIZE);
+ set_cipher_config1(&desc[idx], HASH_PADDING_ENABLED);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
/* Memory Barrier: wait for IPAD/OPAD axi write to complete */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_NO_DMA(&desc[idx], 0, 0xfffff0);
- HW_DESC_SET_DOUT_NO_DMA(&desc[idx], 0, 0, 1);
+ hw_desc_init(&desc[idx]);
+ set_din_no_dma(&desc[idx], 0, 0xfffff0);
+ set_dout_no_dma(&desc[idx], 0, 0, 1);
idx++;
/* Perform HASH update */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr, digestsize, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_HASH);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr,
+ digestsize, NS_BIT);
+ set_flow_mode(&desc[idx], DIN_HASH);
idx++;
}
/* Get final MAC result */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], state->digest_result_dma_addr, digestsize, NS_BIT, async_req? 1:0); /*TODO*/
- if (async_req) {
- HW_DESC_SET_QUEUE_LAST_IND(&desc[idx]);
- }
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_DISABLED);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ /* TODO */
+ set_dout_dlli(&desc[idx], state->digest_result_dma_addr, digestsize,
+ NS_BIT, (async_req ? 1 : 0));
+ if (async_req)
+ set_queue_last_ind(&desc[idx]);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
+ set_cipher_config1(&desc[idx], HASH_PADDING_DISABLED);
ssi_set_hash_endianity(ctx->hash_mode, &desc[idx]);
idx++;
SSI_LOG_ERR("send_request() failed (rc=%d)\n", rc);
ssi_buffer_mgr_unmap_hash_request(dev, state, src, true);
} else {
- ssi_buffer_mgr_unmap_hash_request(dev, state, src, false);
+ ssi_buffer_mgr_unmap_hash_request(dev, state, src, false);
}
ssi_hash_unmap_result(dev, state, digestsize, result);
ssi_hash_unmap_request(dev, state, ctx);
return rc;
}
-static int ssi_hash_update(struct ahash_req_ctx *state,
- struct ssi_hash_ctx *ctx,
- unsigned int block_size,
- struct scatterlist *src,
- unsigned int nbytes,
+static int ssi_hash_update(struct ahash_req_ctx *state,
+ struct ssi_hash_ctx *ctx,
+ unsigned int block_size,
+ struct scatterlist *src,
+ unsigned int nbytes,
void *async_req)
{
struct device *dev = &ctx->drvdata->plat_dev->dev;
struct ssi_crypto_req ssi_req = {};
- HwDesc_s desc[SSI_MAX_AHASH_SEQ_LEN];
- uint32_t idx = 0;
+ struct cc_hw_desc desc[SSI_MAX_AHASH_SEQ_LEN];
+ u32 idx = 0;
int rc;
SSI_LOG_DEBUG("===== %s-update (%d) ====\n", ctx->is_hmac ?
- "hmac":"hash", nbytes);
+ "hmac" : "hash", nbytes);
CHECK_AND_RETURN_UPON_FIPS_ERROR();
if (nbytes == 0) {
/* Setup DX request structure */
ssi_req.user_cb = (void *)ssi_hash_update_complete;
ssi_req.user_arg = async_req;
-#ifdef ENABLE_CYCLE_COUNT
- ssi_req.op_type = STAT_OP_TYPE_ENCODE; /* Use "Encode" stats */
-#endif
}
/* Restore hash digest */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr, ctx->inter_digestsize, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_din_type(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr,
+ ctx->inter_digestsize, NS_BIT);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
idx++;
/* Restore hash current length */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, state->digest_bytes_len_dma_addr, HASH_LEN_SIZE, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_din_type(&desc[idx], DMA_DLLI, state->digest_bytes_len_dma_addr,
+ HASH_LEN_SIZE, NS_BIT);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
ssi_hash_create_data_desc(state, ctx, DIN_HASH, desc, false, &idx);
/* store the hash digest result in context */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], state->digest_buff_dma_addr, ctx->inter_digestsize, NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_dout_dlli(&desc[idx], state->digest_buff_dma_addr,
+ ctx->inter_digestsize, NS_BIT, 0);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
idx++;
/* store current hash length in context */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], state->digest_bytes_len_dma_addr, HASH_LEN_SIZE, NS_BIT, async_req? 1:0);
- if (async_req) {
- HW_DESC_SET_QUEUE_LAST_IND(&desc[idx]);
- }
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE1);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_dout_dlli(&desc[idx], state->digest_bytes_len_dma_addr,
+ HASH_LEN_SIZE, NS_BIT, (async_req ? 1 : 0));
+ if (async_req)
+ set_queue_last_ind(&desc[idx]);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE1);
idx++;
if (async_req) {
return rc;
}
-static int ssi_hash_finup(struct ahash_req_ctx *state,
- struct ssi_hash_ctx *ctx,
- unsigned int digestsize,
- struct scatterlist *src,
- unsigned int nbytes,
- u8 *result,
+static int ssi_hash_finup(struct ahash_req_ctx *state,
+ struct ssi_hash_ctx *ctx,
+ unsigned int digestsize,
+ struct scatterlist *src,
+ unsigned int nbytes,
+ u8 *result,
void *async_req)
{
struct device *dev = &ctx->drvdata->plat_dev->dev;
bool is_hmac = ctx->is_hmac;
struct ssi_crypto_req ssi_req = {};
- HwDesc_s desc[SSI_MAX_AHASH_SEQ_LEN];
+ struct cc_hw_desc desc[SSI_MAX_AHASH_SEQ_LEN];
int idx = 0;
int rc;
- SSI_LOG_DEBUG("===== %s-finup (%d) ====\n", is_hmac?"hmac":"hash", nbytes);
+ SSI_LOG_DEBUG("===== %s-finup (%d) ====\n", is_hmac ? "hmac" : "hash", nbytes);
CHECK_AND_RETURN_UPON_FIPS_ERROR();
- if (unlikely(ssi_buffer_mgr_map_hash_request_final(ctx->drvdata, state, src , nbytes, 1) != 0)) {
+ if (unlikely(ssi_buffer_mgr_map_hash_request_final(ctx->drvdata, state, src, nbytes, 1) != 0)) {
SSI_LOG_ERR("map_ahash_request_final() failed\n");
return -ENOMEM;
}
/* Setup DX request structure */
ssi_req.user_cb = (void *)ssi_hash_complete;
ssi_req.user_arg = async_req;
-#ifdef ENABLE_CYCLE_COUNT
- ssi_req.op_type = STAT_OP_TYPE_ENCODE; /* Use "Encode" stats */
-#endif
}
/* Restore hash digest */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr, ctx->inter_digestsize, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_din_type(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr,
+ ctx->inter_digestsize, NS_BIT);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
idx++;
/* Restore hash current length */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_ENABLED);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, state->digest_bytes_len_dma_addr, HASH_LEN_SIZE, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_cipher_config1(&desc[idx], HASH_PADDING_ENABLED);
+ set_din_type(&desc[idx], DMA_DLLI, state->digest_bytes_len_dma_addr,
+ HASH_LEN_SIZE, NS_BIT);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
ssi_hash_create_data_desc(state, ctx, DIN_HASH, desc, false, &idx);
if (is_hmac) {
/* Store the hash digest result in the context */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], state->digest_buff_dma_addr, digestsize, NS_BIT, 0);
- ssi_set_hash_endianity(ctx->hash_mode,&desc[idx]);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_dout_dlli(&desc[idx], state->digest_buff_dma_addr,
+ digestsize, NS_BIT, 0);
+ ssi_set_hash_endianity(ctx->hash_mode, &desc[idx]);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
idx++;
/* Loading hash OPAD xor key state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, state->opad_digest_dma_addr, ctx->inter_digestsize, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_din_type(&desc[idx], DMA_DLLI, state->opad_digest_dma_addr,
+ ctx->inter_digestsize, NS_BIT);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
idx++;
/* Load the hash current length */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DIN_SRAM(&desc[idx], ssi_ahash_get_initial_digest_len_sram_addr(ctx->drvdata, ctx->hash_mode), HASH_LEN_SIZE);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_ENABLED);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_din_sram(&desc[idx],
+ ssi_ahash_get_initial_digest_len_sram_addr(
+ctx->drvdata, ctx->hash_mode), HASH_LEN_SIZE);
+ set_cipher_config1(&desc[idx], HASH_PADDING_ENABLED);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
/* Memory Barrier: wait for IPAD/OPAD axi write to complete */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_NO_DMA(&desc[idx], 0, 0xfffff0);
- HW_DESC_SET_DOUT_NO_DMA(&desc[idx], 0, 0, 1);
+ hw_desc_init(&desc[idx]);
+ set_din_no_dma(&desc[idx], 0, 0xfffff0);
+ set_dout_no_dma(&desc[idx], 0, 0, 1);
idx++;
/* Perform HASH update on last digest */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr, digestsize, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_HASH);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr,
+ digestsize, NS_BIT);
+ set_flow_mode(&desc[idx], DIN_HASH);
idx++;
}
/* Get final MAC result */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], state->digest_result_dma_addr, digestsize, NS_BIT, async_req? 1:0); /*TODO*/
- if (async_req) {
- HW_DESC_SET_QUEUE_LAST_IND(&desc[idx]);
- }
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_DISABLED);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
- ssi_set_hash_endianity(ctx->hash_mode,&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
+ hw_desc_init(&desc[idx]);
+ /* TODO */
+ set_dout_dlli(&desc[idx], state->digest_result_dma_addr, digestsize,
+ NS_BIT, (async_req ? 1 : 0));
+ if (async_req)
+ set_queue_last_ind(&desc[idx]);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_cipher_config1(&desc[idx], HASH_PADDING_DISABLED);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
+ ssi_set_hash_endianity(ctx->hash_mode, &desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
idx++;
if (async_req) {
return rc;
}
-static int ssi_hash_final(struct ahash_req_ctx *state,
- struct ssi_hash_ctx *ctx,
- unsigned int digestsize,
- struct scatterlist *src,
- unsigned int nbytes,
- u8 *result,
+static int ssi_hash_final(struct ahash_req_ctx *state,
+ struct ssi_hash_ctx *ctx,
+ unsigned int digestsize,
+ struct scatterlist *src,
+ unsigned int nbytes,
+ u8 *result,
void *async_req)
{
struct device *dev = &ctx->drvdata->plat_dev->dev;
bool is_hmac = ctx->is_hmac;
struct ssi_crypto_req ssi_req = {};
- HwDesc_s desc[SSI_MAX_AHASH_SEQ_LEN];
+ struct cc_hw_desc desc[SSI_MAX_AHASH_SEQ_LEN];
int idx = 0;
int rc;
- SSI_LOG_DEBUG("===== %s-final (%d) ====\n", is_hmac?"hmac":"hash", nbytes);
+ SSI_LOG_DEBUG("===== %s-final (%d) ====\n", is_hmac ? "hmac" : "hash", nbytes);
CHECK_AND_RETURN_UPON_FIPS_ERROR();
/* Setup DX request structure */
ssi_req.user_cb = (void *)ssi_hash_complete;
ssi_req.user_arg = async_req;
-#ifdef ENABLE_CYCLE_COUNT
- ssi_req.op_type = STAT_OP_TYPE_ENCODE; /* Use "Encode" stats */
-#endif
}
/* Restore hash digest */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr, ctx->inter_digestsize, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_din_type(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr,
+ ctx->inter_digestsize, NS_BIT);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
idx++;
/* Restore hash current length */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_DISABLED);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, state->digest_bytes_len_dma_addr, HASH_LEN_SIZE, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_cipher_config1(&desc[idx], HASH_PADDING_DISABLED);
+ set_din_type(&desc[idx], DMA_DLLI, state->digest_bytes_len_dma_addr,
+ HASH_LEN_SIZE, NS_BIT);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
ssi_hash_create_data_desc(state, ctx, DIN_HASH, desc, false, &idx);
/* "DO-PAD" must be enabled only when writing current length to HW */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_DO(&desc[idx], DO_PAD);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], state->digest_bytes_len_dma_addr, HASH_LEN_SIZE, NS_BIT, 0);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE1);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
+ hw_desc_init(&desc[idx]);
+ set_cipher_do(&desc[idx], DO_PAD);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_dout_dlli(&desc[idx], state->digest_bytes_len_dma_addr,
+ HASH_LEN_SIZE, NS_BIT, 0);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE1);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
idx++;
if (is_hmac) {
/* Store the hash digest result in the context */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], state->digest_buff_dma_addr, digestsize, NS_BIT, 0);
- ssi_set_hash_endianity(ctx->hash_mode,&desc[idx]);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_dout_dlli(&desc[idx], state->digest_buff_dma_addr,
+ digestsize, NS_BIT, 0);
+ ssi_set_hash_endianity(ctx->hash_mode, &desc[idx]);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
idx++;
/* Loading hash OPAD xor key state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, state->opad_digest_dma_addr, ctx->inter_digestsize, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_din_type(&desc[idx], DMA_DLLI, state->opad_digest_dma_addr,
+ ctx->inter_digestsize, NS_BIT);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
idx++;
/* Load the hash current length */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DIN_SRAM(&desc[idx], ssi_ahash_get_initial_digest_len_sram_addr(ctx->drvdata, ctx->hash_mode), HASH_LEN_SIZE);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_ENABLED);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_din_sram(&desc[idx],
+ ssi_ahash_get_initial_digest_len_sram_addr(
+ctx->drvdata, ctx->hash_mode), HASH_LEN_SIZE);
+ set_cipher_config1(&desc[idx], HASH_PADDING_ENABLED);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
/* Memory Barrier: wait for IPAD/OPAD axi write to complete */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_NO_DMA(&desc[idx], 0, 0xfffff0);
- HW_DESC_SET_DOUT_NO_DMA(&desc[idx], 0, 0, 1);
+ hw_desc_init(&desc[idx]);
+ set_din_no_dma(&desc[idx], 0, 0xfffff0);
+ set_dout_no_dma(&desc[idx], 0, 0, 1);
idx++;
/* Perform HASH update on last digest */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr, digestsize, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_HASH);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr,
+ digestsize, NS_BIT);
+ set_flow_mode(&desc[idx], DIN_HASH);
idx++;
}
/* Get final MAC result */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], state->digest_result_dma_addr, digestsize, NS_BIT, async_req? 1:0);
- if (async_req) {
- HW_DESC_SET_QUEUE_LAST_IND(&desc[idx]);
- }
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_DISABLED);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
- ssi_set_hash_endianity(ctx->hash_mode,&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
+ hw_desc_init(&desc[idx]);
+ set_dout_dlli(&desc[idx], state->digest_result_dma_addr, digestsize,
+ NS_BIT, (async_req ? 1 : 0));
+ if (async_req)
+ set_queue_last_ind(&desc[idx]);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_cipher_config1(&desc[idx], HASH_PADDING_DISABLED);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
+ ssi_set_hash_endianity(ctx->hash_mode, &desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
idx++;
if (async_req) {
static int ssi_hash_init(struct ahash_req_ctx *state, struct ssi_hash_ctx *ctx)
{
struct device *dev = &ctx->drvdata->plat_dev->dev;
- state->xcbc_count = 0;
- CHECK_AND_RETURN_UPON_FIPS_ERROR();
- ssi_hash_map_request(dev, state, ctx);
-
- return 0;
-}
+ state->xcbc_count = 0;
-#ifdef EXPORT_FIXED
-static int ssi_hash_export(struct ssi_hash_ctx *ctx, void *out)
-{
CHECK_AND_RETURN_UPON_FIPS_ERROR();
- memcpy(out, ctx, sizeof(struct ssi_hash_ctx));
- return 0;
-}
+ ssi_hash_map_request(dev, state, ctx);
-static int ssi_hash_import(struct ssi_hash_ctx *ctx, const void *in)
-{
- CHECK_AND_RETURN_UPON_FIPS_ERROR();
- memcpy(ctx, in, sizeof(struct ssi_hash_ctx));
return 0;
}
-#endif
static int ssi_hash_setkey(void *hash,
- const u8 *key,
- unsigned int keylen,
+ const u8 *key,
+ unsigned int keylen,
bool synchronize)
{
unsigned int hmacPadConst[2] = { HMAC_IPAD_CONST, HMAC_OPAD_CONST };
int blocksize = 0;
int digestsize = 0;
int i, idx = 0, rc = 0;
- HwDesc_s desc[SSI_MAX_AHASH_SEQ_LEN];
+ struct cc_hw_desc desc[SSI_MAX_AHASH_SEQ_LEN];
ssi_sram_addr_t larval_addr;
SSI_LOG_DEBUG("ssi_hash_setkey: start keylen: %d", keylen);
-
+
CHECK_AND_RETURN_UPON_FIPS_ERROR();
- if (synchronize) {
- ctx = crypto_shash_ctx(((struct crypto_shash *)hash));
- blocksize = crypto_tfm_alg_blocksize(&((struct crypto_shash *)hash)->base);
- digestsize = crypto_shash_digestsize(((struct crypto_shash *)hash));
- } else {
- ctx = crypto_ahash_ctx(((struct crypto_ahash *)hash));
- blocksize = crypto_tfm_alg_blocksize(&((struct crypto_ahash *)hash)->base);
- digestsize = crypto_ahash_digestsize(((struct crypto_ahash *)hash));
- }
-
+ ctx = crypto_ahash_ctx(((struct crypto_ahash *)hash));
+ blocksize = crypto_tfm_alg_blocksize(&((struct crypto_ahash *)hash)->base);
+ digestsize = crypto_ahash_digestsize(((struct crypto_ahash *)hash));
+
larval_addr = ssi_ahash_get_larval_digest_sram_addr(
ctx->drvdata, ctx->hash_mode);
/* The keylen value distinguishes HASH in case keylen is ZERO bytes,
- any NON-ZERO value utilizes HMAC flow */
+ * any NON-ZERO value utilizes HMAC flow
+ */
ctx->key_params.keylen = keylen;
ctx->key_params.key_dma_addr = 0;
ctx->is_hmac = true;
" DMA failed\n", key, keylen);
return -ENOMEM;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(ctx->key_params.key_dma_addr, keylen);
SSI_LOG_DEBUG("mapping key-buffer: key_dma_addr=0x%llX "
"keylen=%u\n",
(unsigned long long)ctx->key_params.key_dma_addr,
if (keylen > blocksize) {
/* Load hash initial state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DIN_SRAM(&desc[idx], larval_addr,
- ctx->inter_digestsize);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_din_sram(&desc[idx], larval_addr,
+ ctx->inter_digestsize);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
idx++;
-
+
/* Load the hash current length*/
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0, HASH_LEN_SIZE);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_ENABLED);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_din_const(&desc[idx], 0, HASH_LEN_SIZE);
+ set_cipher_config1(&desc[idx], HASH_PADDING_ENABLED);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
-
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- ctx->key_params.key_dma_addr,
- keylen, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_HASH);
+
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI,
+ ctx->key_params.key_dma_addr, keylen,
+ NS_BIT);
+ set_flow_mode(&desc[idx], DIN_HASH);
idx++;
-
+
/* Get hashed key */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], ctx->opad_tmp_keys_dma_addr,
- digestsize, NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
- HW_DESC_SET_CIPHER_CONFIG1(&desc[idx], HASH_PADDING_DISABLED);
- ssi_set_hash_endianity(ctx->hash_mode,&desc[idx]);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_dout_dlli(&desc[idx], ctx->opad_tmp_keys_dma_addr,
+ digestsize, NS_BIT, 0);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
+ set_cipher_config1(&desc[idx], HASH_PADDING_DISABLED);
+ ssi_set_hash_endianity(ctx->hash_mode, &desc[idx]);
idx++;
-
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0, (blocksize - digestsize));
- HW_DESC_SET_FLOW_MODE(&desc[idx], BYPASS);
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- (ctx->opad_tmp_keys_dma_addr + digestsize),
- (blocksize - digestsize),
- NS_BIT, 0);
+
+ hw_desc_init(&desc[idx]);
+ set_din_const(&desc[idx], 0, (blocksize - digestsize));
+ set_flow_mode(&desc[idx], BYPASS);
+ set_dout_dlli(&desc[idx], (ctx->opad_tmp_keys_dma_addr +
+ digestsize),
+ (blocksize - digestsize), NS_BIT, 0);
idx++;
} else {
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- ctx->key_params.key_dma_addr,
- keylen, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], BYPASS);
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- (ctx->opad_tmp_keys_dma_addr),
- keylen, NS_BIT, 0);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI,
+ ctx->key_params.key_dma_addr, keylen,
+ NS_BIT);
+ set_flow_mode(&desc[idx], BYPASS);
+ set_dout_dlli(&desc[idx], ctx->opad_tmp_keys_dma_addr,
+ keylen, NS_BIT, 0);
idx++;
if ((blocksize - keylen) != 0) {
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0, (blocksize - keylen));
- HW_DESC_SET_FLOW_MODE(&desc[idx], BYPASS);
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- (ctx->opad_tmp_keys_dma_addr + keylen),
- (blocksize - keylen),
- NS_BIT, 0);
+ hw_desc_init(&desc[idx]);
+ set_din_const(&desc[idx], 0,
+ (blocksize - keylen));
+ set_flow_mode(&desc[idx], BYPASS);
+ set_dout_dlli(&desc[idx],
+ (ctx->opad_tmp_keys_dma_addr +
+ keylen), (blocksize - keylen),
+ NS_BIT, 0);
idx++;
}
}
} else {
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0, blocksize);
- HW_DESC_SET_FLOW_MODE(&desc[idx], BYPASS);
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- (ctx->opad_tmp_keys_dma_addr),
- blocksize,
- NS_BIT, 0);
+ hw_desc_init(&desc[idx]);
+ set_din_const(&desc[idx], 0, blocksize);
+ set_flow_mode(&desc[idx], BYPASS);
+ set_dout_dlli(&desc[idx], (ctx->opad_tmp_keys_dma_addr),
+ blocksize, NS_BIT, 0);
idx++;
}
/* calc derived HMAC key */
for (idx = 0, i = 0; i < 2; i++) {
/* Load hash initial state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DIN_SRAM(&desc[idx], larval_addr,
- ctx->inter_digestsize);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_din_sram(&desc[idx], larval_addr, ctx->inter_digestsize);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
idx++;
/* Load the hash current length*/
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0, HASH_LEN_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_din_const(&desc[idx], 0, HASH_LEN_SIZE);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
/* Prepare ipad key */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_XOR_VAL(&desc[idx], hmacPadConst[i]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_HASH);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE1);
+ hw_desc_init(&desc[idx]);
+ set_xor_val(&desc[idx], hmacPadConst[i]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_flow_mode(&desc[idx], S_DIN_to_HASH);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE1);
idx++;
/* Perform HASH update */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- ctx->opad_tmp_keys_dma_addr,
- blocksize, NS_BIT);
- HW_DESC_SET_CIPHER_MODE(&desc[idx],ctx->hw_mode);
- HW_DESC_SET_XOR_ACTIVE(&desc[idx]);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_HASH);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, ctx->opad_tmp_keys_dma_addr,
+ blocksize, NS_BIT);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_xor_active(&desc[idx]);
+ set_flow_mode(&desc[idx], DIN_HASH);
idx++;
/* Get the IPAD/OPAD xor key (Note, IPAD is the initial digest of the first HASH "update" state) */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
if (i > 0) /* Not first iteration */
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- ctx->opad_tmp_keys_dma_addr,
- ctx->inter_digestsize,
- NS_BIT, 0);
+ set_dout_dlli(&desc[idx], ctx->opad_tmp_keys_dma_addr,
+ ctx->inter_digestsize, NS_BIT, 0);
else /* First iteration */
- HW_DESC_SET_DOUT_DLLI(&desc[idx],
- ctx->digest_buff_dma_addr,
- ctx->inter_digestsize,
- NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_HASH_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
+ set_dout_dlli(&desc[idx], ctx->digest_buff_dma_addr,
+ ctx->inter_digestsize, NS_BIT, 0);
+ set_flow_mode(&desc[idx], S_HASH_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
idx++;
}
rc = send_request(ctx->drvdata, &ssi_req, desc, idx, 0);
out:
- if (rc != 0) {
- if (synchronize) {
- crypto_shash_set_flags((struct crypto_shash *)hash, CRYPTO_TFM_RES_BAD_KEY_LEN);
- } else {
- crypto_ahash_set_flags((struct crypto_ahash *)hash, CRYPTO_TFM_RES_BAD_KEY_LEN);
- }
- }
+ if (rc)
+ crypto_ahash_set_flags((struct crypto_ahash *)hash, CRYPTO_TFM_RES_BAD_KEY_LEN);
if (ctx->key_params.key_dma_addr) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(ctx->key_params.key_dma_addr);
dma_unmap_single(&ctx->drvdata->plat_dev->dev,
ctx->key_params.key_dma_addr,
ctx->key_params.keylen, DMA_TO_DEVICE);
return rc;
}
-
static int ssi_xcbc_setkey(struct crypto_ahash *ahash,
const u8 *key, unsigned int keylen)
{
struct ssi_crypto_req ssi_req = {};
struct ssi_hash_ctx *ctx = crypto_ahash_ctx(ahash);
int idx = 0, rc = 0;
- HwDesc_s desc[SSI_MAX_AHASH_SEQ_LEN];
+ struct cc_hw_desc desc[SSI_MAX_AHASH_SEQ_LEN];
SSI_LOG_DEBUG("===== setkey (%d) ====\n", keylen);
CHECK_AND_RETURN_UPON_FIPS_ERROR();
" DMA failed\n", key, keylen);
return -ENOMEM;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(ctx->key_params.key_dma_addr, keylen);
SSI_LOG_DEBUG("mapping key-buffer: key_dma_addr=0x%llX "
"keylen=%u\n",
(unsigned long long)ctx->key_params.key_dma_addr,
ctx->key_params.keylen);
-
+
ctx->is_hmac = true;
/* 1. Load the AES key */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, ctx->key_params.key_dma_addr, keylen, NS_BIT);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_ECB);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], keylen);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, ctx->key_params.key_dma_addr,
+ keylen, NS_BIT);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_ECB);
+ set_cipher_config0(&desc[idx], DRV_CRYPTO_DIRECTION_ENCRYPT);
+ set_key_size_aes(&desc[idx], keylen);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0x01010101, CC_AES_128_BIT_KEY_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_AES_DOUT);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], (ctx->opad_tmp_keys_dma_addr +
- XCBC_MAC_K1_OFFSET),
+ hw_desc_init(&desc[idx]);
+ set_din_const(&desc[idx], 0x01010101, CC_AES_128_BIT_KEY_SIZE);
+ set_flow_mode(&desc[idx], DIN_AES_DOUT);
+ set_dout_dlli(&desc[idx], (ctx->opad_tmp_keys_dma_addr +
+ XCBC_MAC_K1_OFFSET),
CC_AES_128_BIT_KEY_SIZE, NS_BIT, 0);
idx++;
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0x02020202, CC_AES_128_BIT_KEY_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_AES_DOUT);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], (ctx->opad_tmp_keys_dma_addr +
- XCBC_MAC_K2_OFFSET),
+ hw_desc_init(&desc[idx]);
+ set_din_const(&desc[idx], 0x02020202, CC_AES_128_BIT_KEY_SIZE);
+ set_flow_mode(&desc[idx], DIN_AES_DOUT);
+ set_dout_dlli(&desc[idx], (ctx->opad_tmp_keys_dma_addr +
+ XCBC_MAC_K2_OFFSET),
CC_AES_128_BIT_KEY_SIZE, NS_BIT, 0);
idx++;
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0x03030303, CC_AES_128_BIT_KEY_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_AES_DOUT);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], (ctx->opad_tmp_keys_dma_addr +
+ hw_desc_init(&desc[idx]);
+ set_din_const(&desc[idx], 0x03030303, CC_AES_128_BIT_KEY_SIZE);
+ set_flow_mode(&desc[idx], DIN_AES_DOUT);
+ set_dout_dlli(&desc[idx], (ctx->opad_tmp_keys_dma_addr +
XCBC_MAC_K3_OFFSET),
CC_AES_128_BIT_KEY_SIZE, NS_BIT, 0);
idx++;
if (rc != 0)
crypto_ahash_set_flags(ahash, CRYPTO_TFM_RES_BAD_KEY_LEN);
- SSI_RESTORE_DMA_ADDR_TO_48BIT(ctx->key_params.key_dma_addr);
dma_unmap_single(&ctx->drvdata->plat_dev->dev,
ctx->key_params.key_dma_addr,
ctx->key_params.keylen, DMA_TO_DEVICE);
return rc;
}
+
#if SSI_CC_HAS_CMAC
static int ssi_cmac_setkey(struct crypto_ahash *ahash,
const u8 *key, unsigned int keylen)
{
struct ssi_hash_ctx *ctx = crypto_ahash_ctx(ahash);
- DECL_CYCLE_COUNT_RESOURCES;
+
SSI_LOG_DEBUG("===== setkey (%d) ====\n", keylen);
CHECK_AND_RETURN_UPON_FIPS_ERROR();
ctx->key_params.keylen = keylen;
/* STAT_PHASE_1: Copy key to ctx */
- START_CYCLE_COUNT();
-
- SSI_RESTORE_DMA_ADDR_TO_48BIT(ctx->opad_tmp_keys_dma_addr);
+
dma_sync_single_for_cpu(&ctx->drvdata->plat_dev->dev,
- ctx->opad_tmp_keys_dma_addr,
+ ctx->opad_tmp_keys_dma_addr,
keylen, DMA_TO_DEVICE);
memcpy(ctx->opad_tmp_keys_buff, key, keylen);
if (keylen == 24)
memset(ctx->opad_tmp_keys_buff + 24, 0, CC_AES_KEY_SIZE_MAX - 24);
-
+
dma_sync_single_for_device(&ctx->drvdata->plat_dev->dev,
- ctx->opad_tmp_keys_dma_addr,
+ ctx->opad_tmp_keys_dma_addr,
keylen, DMA_TO_DEVICE);
- SSI_UPDATE_DMA_ADDR_TO_48BIT(ctx->opad_tmp_keys_dma_addr, keylen);
-
+
ctx->key_params.keylen = keylen;
-
- END_CYCLE_COUNT(STAT_OP_TYPE_SETKEY, STAT_PHASE_1);
return 0;
}
struct device *dev = &ctx->drvdata->plat_dev->dev;
if (ctx->digest_buff_dma_addr != 0) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(ctx->digest_buff_dma_addr);
dma_unmap_single(dev, ctx->digest_buff_dma_addr,
sizeof(ctx->digest_buff), DMA_BIDIRECTIONAL);
SSI_LOG_DEBUG("Unmapped digest-buffer: "
ctx->digest_buff_dma_addr = 0;
}
if (ctx->opad_tmp_keys_dma_addr != 0) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(ctx->opad_tmp_keys_dma_addr);
dma_unmap_single(dev, ctx->opad_tmp_keys_dma_addr,
sizeof(ctx->opad_tmp_keys_buff),
DMA_BIDIRECTIONAL);
}
ctx->key_params.keylen = 0;
-
}
-
static int ssi_hash_alloc_ctx(struct ssi_hash_ctx *ctx)
{
struct device *dev = &ctx->drvdata->plat_dev->dev;
sizeof(ctx->digest_buff), ctx->digest_buff);
goto fail;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(ctx->digest_buff_dma_addr,
- sizeof(ctx->digest_buff));
SSI_LOG_DEBUG("Mapped digest %zu B at va=%pK to dma=0x%llX\n",
sizeof(ctx->digest_buff), ctx->digest_buff,
(unsigned long long)ctx->digest_buff_dma_addr);
ctx->opad_tmp_keys_buff);
goto fail;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(ctx->opad_tmp_keys_dma_addr,
- sizeof(ctx->opad_tmp_keys_buff));
SSI_LOG_DEBUG("Mapped opad_tmp_keys %zu B at va=%pK to dma=0x%llX\n",
sizeof(ctx->opad_tmp_keys_buff), ctx->opad_tmp_keys_buff,
(unsigned long long)ctx->opad_tmp_keys_dma_addr);
return -ENOMEM;
}
-static int ssi_shash_cra_init(struct crypto_tfm *tfm)
-{
- struct ssi_hash_ctx *ctx = crypto_tfm_ctx(tfm);
- struct shash_alg * shash_alg =
- container_of(tfm->__crt_alg, struct shash_alg, base);
- struct ssi_hash_alg *ssi_alg =
- container_of(shash_alg, struct ssi_hash_alg, shash_alg);
-
- CHECK_AND_RETURN_UPON_FIPS_ERROR();
- ctx->hash_mode = ssi_alg->hash_mode;
- ctx->hw_mode = ssi_alg->hw_mode;
- ctx->inter_digestsize = ssi_alg->inter_digestsize;
- ctx->drvdata = ssi_alg->drvdata;
-
- return ssi_hash_alloc_ctx(ctx);
-}
-
static int ssi_ahash_cra_init(struct crypto_tfm *tfm)
{
struct ssi_hash_ctx *ctx = crypto_tfm_ctx(tfm);
- struct hash_alg_common * hash_alg_common =
+ struct hash_alg_common *hash_alg_common =
container_of(tfm->__crt_alg, struct hash_alg_common, base);
- struct ahash_alg *ahash_alg =
+ struct ahash_alg *ahash_alg =
container_of(hash_alg_common, struct ahash_alg, halg);
struct ssi_hash_alg *ssi_alg =
container_of(ahash_alg, struct ssi_hash_alg, ahash_alg);
-
CHECK_AND_RETURN_UPON_FIPS_ERROR();
crypto_ahash_set_reqsize(__crypto_ahash_cast(tfm),
sizeof(struct ahash_req_ctx));
struct device *dev = &ctx->drvdata->plat_dev->dev;
unsigned int block_size = crypto_tfm_alg_blocksize(&tfm->base);
struct ssi_crypto_req ssi_req = {};
- HwDesc_s desc[SSI_MAX_AHASH_SEQ_LEN];
+ struct cc_hw_desc desc[SSI_MAX_AHASH_SEQ_LEN];
int rc;
- uint32_t idx = 0;
+ u32 idx = 0;
CHECK_AND_RETURN_UPON_FIPS_ERROR();
if (req->nbytes == 0) {
return -ENOMEM;
}
- if (ctx->hw_mode == DRV_CIPHER_XCBC_MAC) {
+ if (ctx->hw_mode == DRV_CIPHER_XCBC_MAC)
ssi_hash_create_xcbc_setup(req, desc, &idx);
- } else {
+ else
ssi_hash_create_cmac_setup(req, desc, &idx);
- }
-
+
ssi_hash_create_data_desc(state, ctx, DIN_AES_DOUT, desc, true, &idx);
/* store the hash digest result in context */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], state->digest_buff_dma_addr, ctx->inter_digestsize, NS_BIT, 1);
- HW_DESC_SET_QUEUE_LAST_IND(&desc[idx]);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_AES_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_dout_dlli(&desc[idx], state->digest_buff_dma_addr,
+ ctx->inter_digestsize, NS_BIT, 1);
+ set_queue_last_ind(&desc[idx]);
+ set_flow_mode(&desc[idx], S_AES_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
idx++;
/* Setup DX request structure */
ssi_req.user_cb = (void *)ssi_hash_update_complete;
ssi_req.user_arg = (void *)req;
-#ifdef ENABLE_CYCLE_COUNT
- ssi_req.op_type = STAT_OP_TYPE_ENCODE; /* Use "Encode" stats */
-#endif
rc = send_request(ctx->drvdata, &ssi_req, desc, idx, 1);
if (unlikely(rc != -EINPROGRESS)) {
struct ssi_hash_ctx *ctx = crypto_ahash_ctx(tfm);
struct device *dev = &ctx->drvdata->plat_dev->dev;
struct ssi_crypto_req ssi_req = {};
- HwDesc_s desc[SSI_MAX_AHASH_SEQ_LEN];
+ struct cc_hw_desc desc[SSI_MAX_AHASH_SEQ_LEN];
int idx = 0;
int rc = 0;
- uint32_t keySize, keyLen;
- uint32_t digestsize = crypto_ahash_digestsize(tfm);
+ u32 keySize, keyLen;
+ u32 digestsize = crypto_ahash_digestsize(tfm);
- uint32_t rem_cnt = state->buff_index ? state->buff1_cnt :
+ u32 rem_cnt = state->buff_index ? state->buff1_cnt :
state->buff0_cnt;
-
CHECK_AND_RETURN_UPON_FIPS_ERROR();
if (ctx->hw_mode == DRV_CIPHER_XCBC_MAC) {
/* Setup DX request structure */
ssi_req.user_cb = (void *)ssi_hash_complete;
ssi_req.user_arg = (void *)req;
-#ifdef ENABLE_CYCLE_COUNT
- ssi_req.op_type = STAT_OP_TYPE_ENCODE; /* Use "Encode" stats */
-#endif
if (state->xcbc_count && (rem_cnt == 0)) {
/* Load key for ECB decryption */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_ECB);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DRV_CRYPTO_DIRECTION_DECRYPT);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- (ctx->opad_tmp_keys_dma_addr +
- XCBC_MAC_K1_OFFSET),
- keySize, NS_BIT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], keyLen);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_ECB);
+ set_cipher_config0(&desc[idx], DRV_CRYPTO_DIRECTION_DECRYPT);
+ set_din_type(&desc[idx], DMA_DLLI,
+ (ctx->opad_tmp_keys_dma_addr +
+ XCBC_MAC_K1_OFFSET), keySize, NS_BIT);
+ set_key_size_aes(&desc[idx], keyLen);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
idx++;
-
/* Initiate decryption of block state to previous block_state-XOR-M[n] */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr, CC_AES_BLOCK_SIZE, NS_BIT);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], state->digest_buff_dma_addr, CC_AES_BLOCK_SIZE, NS_BIT,0);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_AES_DOUT);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr,
+ CC_AES_BLOCK_SIZE, NS_BIT);
+ set_dout_dlli(&desc[idx], state->digest_buff_dma_addr,
+ CC_AES_BLOCK_SIZE, NS_BIT, 0);
+ set_flow_mode(&desc[idx], DIN_AES_DOUT);
idx++;
/* Memory Barrier: wait for axi write to complete */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_NO_DMA(&desc[idx], 0, 0xfffff0);
- HW_DESC_SET_DOUT_NO_DMA(&desc[idx], 0, 0, 1);
+ hw_desc_init(&desc[idx]);
+ set_din_no_dma(&desc[idx], 0, 0xfffff0);
+ set_dout_no_dma(&desc[idx], 0, 0, 1);
idx++;
}
-
- if (ctx->hw_mode == DRV_CIPHER_XCBC_MAC) {
+
+ if (ctx->hw_mode == DRV_CIPHER_XCBC_MAC)
ssi_hash_create_xcbc_setup(req, desc, &idx);
- } else {
+ else
ssi_hash_create_cmac_setup(req, desc, &idx);
- }
if (state->xcbc_count == 0) {
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], keyLen);
- HW_DESC_SET_CMAC_SIZE0_MODE(&desc[idx]);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_key_size_aes(&desc[idx], keyLen);
+ set_cmac_size0_mode(&desc[idx]);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
} else if (rem_cnt > 0) {
ssi_hash_create_data_desc(state, ctx, DIN_AES_DOUT, desc, false, &idx);
} else {
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_CONST(&desc[idx], 0x00, CC_AES_BLOCK_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[idx], DIN_AES_DOUT);
+ hw_desc_init(&desc[idx]);
+ set_din_const(&desc[idx], 0x00, CC_AES_BLOCK_SIZE);
+ set_flow_mode(&desc[idx], DIN_AES_DOUT);
idx++;
}
-
+
/* Get final MAC result */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], state->digest_result_dma_addr, digestsize, NS_BIT, 1); /*TODO*/
- HW_DESC_SET_QUEUE_LAST_IND(&desc[idx]);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_AES_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
+ hw_desc_init(&desc[idx]);
+ /* TODO */
+ set_dout_dlli(&desc[idx], state->digest_result_dma_addr,
+ digestsize, NS_BIT, 1);
+ set_queue_last_ind(&desc[idx]);
+ set_flow_mode(&desc[idx], S_AES_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
idx++;
rc = send_request(ctx->drvdata, &ssi_req, desc, idx, 1);
struct ssi_hash_ctx *ctx = crypto_ahash_ctx(tfm);
struct device *dev = &ctx->drvdata->plat_dev->dev;
struct ssi_crypto_req ssi_req = {};
- HwDesc_s desc[SSI_MAX_AHASH_SEQ_LEN];
+ struct cc_hw_desc desc[SSI_MAX_AHASH_SEQ_LEN];
int idx = 0;
int rc = 0;
- uint32_t key_len = 0;
- uint32_t digestsize = crypto_ahash_digestsize(tfm);
+ u32 key_len = 0;
+ u32 digestsize = crypto_ahash_digestsize(tfm);
SSI_LOG_DEBUG("===== finup xcbc(%d) ====\n", req->nbytes);
CHECK_AND_RETURN_UPON_FIPS_ERROR();
SSI_LOG_DEBUG("No data to update. Call to fdx_mac_final \n");
return ssi_mac_final(req);
}
-
+
if (unlikely(ssi_buffer_mgr_map_hash_request_final(ctx->drvdata, state, req->src, req->nbytes, 1) != 0)) {
SSI_LOG_ERR("map_ahash_request_final() failed\n");
return -ENOMEM;
/* Setup DX request structure */
ssi_req.user_cb = (void *)ssi_hash_complete;
ssi_req.user_arg = (void *)req;
-#ifdef ENABLE_CYCLE_COUNT
- ssi_req.op_type = STAT_OP_TYPE_ENCODE; /* Use "Encode" stats */
-#endif
if (ctx->hw_mode == DRV_CIPHER_XCBC_MAC) {
key_len = CC_AES_128_BIT_KEY_SIZE;
}
if (req->nbytes == 0) {
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], key_len);
- HW_DESC_SET_CMAC_SIZE0_MODE(&desc[idx]);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_key_size_aes(&desc[idx], key_len);
+ set_cmac_size0_mode(&desc[idx]);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
} else {
ssi_hash_create_data_desc(state, ctx, DIN_AES_DOUT, desc, false, &idx);
}
-
+
/* Get final MAC result */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], state->digest_result_dma_addr, digestsize, NS_BIT, 1); /*TODO*/
- HW_DESC_SET_QUEUE_LAST_IND(&desc[idx]);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_AES_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
+ hw_desc_init(&desc[idx]);
+ /* TODO */
+ set_dout_dlli(&desc[idx], state->digest_result_dma_addr,
+ digestsize, NS_BIT, 1);
+ set_queue_last_ind(&desc[idx]);
+ set_flow_mode(&desc[idx], S_AES_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
idx++;
rc = send_request(ctx->drvdata, &ssi_req, desc, idx, 1);
struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
struct ssi_hash_ctx *ctx = crypto_ahash_ctx(tfm);
struct device *dev = &ctx->drvdata->plat_dev->dev;
- uint32_t digestsize = crypto_ahash_digestsize(tfm);
+ u32 digestsize = crypto_ahash_digestsize(tfm);
struct ssi_crypto_req ssi_req = {};
- HwDesc_s desc[SSI_MAX_AHASH_SEQ_LEN];
- uint32_t keyLen;
+ struct cc_hw_desc desc[SSI_MAX_AHASH_SEQ_LEN];
+ u32 keyLen;
int idx = 0;
int rc;
SSI_LOG_DEBUG("===== -digest mac (%d) ====\n", req->nbytes);
CHECK_AND_RETURN_UPON_FIPS_ERROR();
-
+
if (unlikely(ssi_hash_map_request(dev, state, ctx) != 0)) {
SSI_LOG_ERR("map_ahash_source() failed\n");
return -ENOMEM;
SSI_LOG_ERR("map_ahash_request_final() failed\n");
return -ENOMEM;
}
-
+
/* Setup DX request structure */
ssi_req.user_cb = (void *)ssi_hash_digest_complete;
ssi_req.user_arg = (void *)req;
-#ifdef ENABLE_CYCLE_COUNT
- ssi_req.op_type = STAT_OP_TYPE_ENCODE; /* Use "Encode" stats */
-#endif
-
if (ctx->hw_mode == DRV_CIPHER_XCBC_MAC) {
keyLen = CC_AES_128_BIT_KEY_SIZE;
ssi_hash_create_xcbc_setup(req, desc, &idx);
}
if (req->nbytes == 0) {
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], keyLen);
- HW_DESC_SET_CMAC_SIZE0_MODE(&desc[idx]);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
+ set_key_size_aes(&desc[idx], keyLen);
+ set_cmac_size0_mode(&desc[idx]);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
} else {
ssi_hash_create_data_desc(state, ctx, DIN_AES_DOUT, desc, false, &idx);
}
-
+
/* Get final MAC result */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DOUT_DLLI(&desc[idx], state->digest_result_dma_addr, CC_AES_BLOCK_SIZE, NS_BIT,1);
- HW_DESC_SET_QUEUE_LAST_IND(&desc[idx]);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_AES_to_DOUT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_WRITE_STATE0);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx],DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], ctx->hw_mode);
+ hw_desc_init(&desc[idx]);
+ set_dout_dlli(&desc[idx], state->digest_result_dma_addr,
+ CC_AES_BLOCK_SIZE, NS_BIT, 1);
+ set_queue_last_ind(&desc[idx]);
+ set_flow_mode(&desc[idx], S_AES_to_DOUT);
+ set_setup_mode(&desc[idx], SETUP_WRITE_STATE0);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_cipher_mode(&desc[idx], ctx->hw_mode);
idx++;
rc = send_request(ctx->drvdata, &ssi_req, desc, idx, 1);
return rc;
}
-//shash wrap functions
-#ifdef SYNC_ALGS
-static int ssi_shash_digest(struct shash_desc *desc,
- const u8 *data, unsigned int len, u8 *out)
-{
- struct ahash_req_ctx *state = shash_desc_ctx(desc);
- struct crypto_shash *tfm = desc->tfm;
- struct ssi_hash_ctx *ctx = crypto_shash_ctx(tfm);
- uint32_t digestsize = crypto_shash_digestsize(tfm);
- struct scatterlist src;
-
- if (len == 0) {
- return ssi_hash_digest(state, ctx, digestsize, NULL, 0, out, NULL);
- }
-
- /* sg_init_one may crash when len is 0 (depends on kernel configuration) */
- sg_init_one(&src, (const void *)data, len);
-
- return ssi_hash_digest(state, ctx, digestsize, &src, len, out, NULL);
-}
-
-static int ssi_shash_update(struct shash_desc *desc,
- const u8 *data, unsigned int len)
-{
- struct ahash_req_ctx *state = shash_desc_ctx(desc);
- struct crypto_shash *tfm = desc->tfm;
- struct ssi_hash_ctx *ctx = crypto_shash_ctx(tfm);
- uint32_t blocksize = crypto_tfm_alg_blocksize(&tfm->base);
- struct scatterlist src;
-
- sg_init_one(&src, (const void *)data, len);
-
- return ssi_hash_update(state, ctx, blocksize, &src, len, NULL);
-}
-
-static int ssi_shash_finup(struct shash_desc *desc,
- const u8 *data, unsigned int len, u8 *out)
-{
- struct ahash_req_ctx *state = shash_desc_ctx(desc);
- struct crypto_shash *tfm = desc->tfm;
- struct ssi_hash_ctx *ctx = crypto_shash_ctx(tfm);
- uint32_t digestsize = crypto_shash_digestsize(tfm);
- struct scatterlist src;
-
- sg_init_one(&src, (const void *)data, len);
-
- return ssi_hash_finup(state, ctx, digestsize, &src, len, out, NULL);
-}
-
-static int ssi_shash_final(struct shash_desc *desc, u8 *out)
-{
- struct ahash_req_ctx *state = shash_desc_ctx(desc);
- struct crypto_shash *tfm = desc->tfm;
- struct ssi_hash_ctx *ctx = crypto_shash_ctx(tfm);
- uint32_t digestsize = crypto_shash_digestsize(tfm);
-
- return ssi_hash_final(state, ctx, digestsize, NULL, 0, out, NULL);
-}
-
-static int ssi_shash_init(struct shash_desc *desc)
-{
- struct ahash_req_ctx *state = shash_desc_ctx(desc);
- struct crypto_shash *tfm = desc->tfm;
- struct ssi_hash_ctx *ctx = crypto_shash_ctx(tfm);
-
- return ssi_hash_init(state, ctx);
-}
-
-#ifdef EXPORT_FIXED
-static int ssi_shash_export(struct shash_desc *desc, void *out)
-{
- struct crypto_shash *tfm = desc->tfm;
- struct ssi_hash_ctx *ctx = crypto_shash_ctx(tfm);
-
- return ssi_hash_export(ctx, out);
-}
-
-static int ssi_shash_import(struct shash_desc *desc, const void *in)
-{
- struct crypto_shash *tfm = desc->tfm;
- struct ssi_hash_ctx *ctx = crypto_shash_ctx(tfm);
-
- return ssi_hash_import(ctx, in);
-}
-#endif
-
-static int ssi_shash_setkey(struct crypto_shash *tfm,
- const u8 *key, unsigned int keylen)
-{
- return ssi_hash_setkey((void *) tfm, key, keylen, true);
-}
-
-#endif /* SYNC_ALGS */
-
//ahash wrap functions
static int ssi_ahash_digest(struct ahash_request *req)
{
struct ahash_req_ctx *state = ahash_request_ctx(req);
struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
struct ssi_hash_ctx *ctx = crypto_ahash_ctx(tfm);
- uint32_t digestsize = crypto_ahash_digestsize(tfm);
-
+ u32 digestsize = crypto_ahash_digestsize(tfm);
+
return ssi_hash_digest(state, ctx, digestsize, req->src, req->nbytes, req->result, (void *)req);
}
struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
struct ssi_hash_ctx *ctx = crypto_ahash_ctx(tfm);
unsigned int block_size = crypto_tfm_alg_blocksize(&tfm->base);
-
+
return ssi_hash_update(state, ctx, block_size, req->src, req->nbytes, (void *)req);
}
struct ahash_req_ctx *state = ahash_request_ctx(req);
struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
struct ssi_hash_ctx *ctx = crypto_ahash_ctx(tfm);
- uint32_t digestsize = crypto_ahash_digestsize(tfm);
-
+ u32 digestsize = crypto_ahash_digestsize(tfm);
+
return ssi_hash_finup(state, ctx, digestsize, req->src, req->nbytes, req->result, (void *)req);
}
struct ahash_req_ctx *state = ahash_request_ctx(req);
struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
struct ssi_hash_ctx *ctx = crypto_ahash_ctx(tfm);
- uint32_t digestsize = crypto_ahash_digestsize(tfm);
-
+ u32 digestsize = crypto_ahash_digestsize(tfm);
+
return ssi_hash_final(state, ctx, digestsize, req->src, req->nbytes, req->result, (void *)req);
}
{
struct ahash_req_ctx *state = ahash_request_ctx(req);
struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
- struct ssi_hash_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct ssi_hash_ctx *ctx = crypto_ahash_ctx(tfm);
SSI_LOG_DEBUG("===== init (%d) ====\n", req->nbytes);
return ssi_hash_init(state, ctx);
}
-#ifdef EXPORT_FIXED
static int ssi_ahash_export(struct ahash_request *req, void *out)
{
struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
struct ssi_hash_ctx *ctx = crypto_ahash_ctx(ahash);
-
- return ssi_hash_export(ctx, out);
+ struct device *dev = &ctx->drvdata->plat_dev->dev;
+ struct ahash_req_ctx *state = ahash_request_ctx(req);
+ u8 *curr_buff = state->buff_index ? state->buff1 : state->buff0;
+ u32 curr_buff_cnt = state->buff_index ? state->buff1_cnt :
+ state->buff0_cnt;
+ const u32 tmp = CC_EXPORT_MAGIC;
+
+ CHECK_AND_RETURN_UPON_FIPS_ERROR();
+
+ memcpy(out, &tmp, sizeof(u32));
+ out += sizeof(u32);
+
+ dma_sync_single_for_cpu(dev, state->digest_buff_dma_addr,
+ ctx->inter_digestsize, DMA_BIDIRECTIONAL);
+ memcpy(out, state->digest_buff, ctx->inter_digestsize);
+ out += ctx->inter_digestsize;
+
+ if (state->digest_bytes_len_dma_addr) {
+ dma_sync_single_for_cpu(dev, state->digest_bytes_len_dma_addr,
+ HASH_LEN_SIZE, DMA_BIDIRECTIONAL);
+ memcpy(out, state->digest_bytes_len, HASH_LEN_SIZE);
+ } else {
+ /* Poison the unused exported digest len field. */
+ memset(out, 0x5F, HASH_LEN_SIZE);
+ }
+ out += HASH_LEN_SIZE;
+
+ memcpy(out, &curr_buff_cnt, sizeof(u32));
+ out += sizeof(u32);
+
+ memcpy(out, curr_buff, curr_buff_cnt);
+
+ /* No sync for device ineeded since we did not change the data,
+ * we only copy it
+ */
+
+ return 0;
}
static int ssi_ahash_import(struct ahash_request *req, const void *in)
{
struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
struct ssi_hash_ctx *ctx = crypto_ahash_ctx(ahash);
-
- return ssi_hash_import(ctx, in);
+ struct device *dev = &ctx->drvdata->plat_dev->dev;
+ struct ahash_req_ctx *state = ahash_request_ctx(req);
+ u32 tmp;
+ int rc;
+
+ CHECK_AND_RETURN_UPON_FIPS_ERROR();
+
+ memcpy(&tmp, in, sizeof(u32));
+ if (tmp != CC_EXPORT_MAGIC) {
+ rc = -EINVAL;
+ goto out;
+ }
+ in += sizeof(u32);
+
+ rc = ssi_hash_init(state, ctx);
+ if (rc)
+ goto out;
+
+ dma_sync_single_for_cpu(dev, state->digest_buff_dma_addr,
+ ctx->inter_digestsize, DMA_BIDIRECTIONAL);
+ memcpy(state->digest_buff, in, ctx->inter_digestsize);
+ in += ctx->inter_digestsize;
+
+ if (state->digest_bytes_len_dma_addr) {
+ dma_sync_single_for_cpu(dev, state->digest_bytes_len_dma_addr,
+ HASH_LEN_SIZE, DMA_BIDIRECTIONAL);
+ memcpy(state->digest_bytes_len, in, HASH_LEN_SIZE);
+ }
+ in += HASH_LEN_SIZE;
+
+ dma_sync_single_for_device(dev, state->digest_buff_dma_addr,
+ ctx->inter_digestsize, DMA_BIDIRECTIONAL);
+
+ if (state->digest_bytes_len_dma_addr)
+ dma_sync_single_for_device(dev,
+ state->digest_bytes_len_dma_addr,
+ HASH_LEN_SIZE, DMA_BIDIRECTIONAL);
+
+ state->buff_index = 0;
+
+ /* Sanity check the data as much as possible */
+ memcpy(&tmp, in, sizeof(u32));
+ if (tmp > SSI_MAX_HASH_BLCK_SIZE) {
+ rc = -EINVAL;
+ goto out;
+ }
+ in += sizeof(u32);
+
+ state->buff0_cnt = tmp;
+ memcpy(state->buff0, in, state->buff0_cnt);
+
+out:
+ return rc;
}
-#endif
static int ssi_ahash_setkey(struct crypto_ahash *ahash,
const u8 *key, unsigned int keylen)
-{
- return ssi_hash_setkey((void *) ahash, key, keylen, false);
+{
+ return ssi_hash_setkey((void *)ahash, key, keylen, false);
}
struct ssi_hash_template {
char name[CRYPTO_MAX_ALG_NAME];
char driver_name[CRYPTO_MAX_ALG_NAME];
- char hmac_name[CRYPTO_MAX_ALG_NAME];
- char hmac_driver_name[CRYPTO_MAX_ALG_NAME];
+ char mac_name[CRYPTO_MAX_ALG_NAME];
+ char mac_driver_name[CRYPTO_MAX_ALG_NAME];
unsigned int blocksize;
bool synchronize;
- union {
- struct ahash_alg template_ahash;
- struct shash_alg template_shash;
- };
+ struct ahash_alg template_ahash;
int hash_mode;
int hw_mode;
int inter_digestsize;
struct ssi_drvdata *drvdata;
};
+#define CC_STATE_SIZE(_x) \
+ ((_x) + HASH_LEN_SIZE + SSI_MAX_HASH_BLCK_SIZE + (2 * sizeof(u32)))
+
/* hash descriptors */
static struct ssi_hash_template driver_hash[] = {
//Asynchronize hash template
{
.name = "sha1",
.driver_name = "sha1-dx",
- .hmac_name = "hmac(sha1)",
- .hmac_driver_name = "hmac-sha1-dx",
+ .mac_name = "hmac(sha1)",
+ .mac_driver_name = "hmac-sha1-dx",
.blocksize = SHA1_BLOCK_SIZE,
.synchronize = false,
- {
- .template_ahash = {
- .init = ssi_ahash_init,
- .update = ssi_ahash_update,
- .final = ssi_ahash_final,
- .finup = ssi_ahash_finup,
- .digest = ssi_ahash_digest,
-#ifdef EXPORT_FIXED
- .export = ssi_ahash_export,
- .import = ssi_ahash_import,
-#endif
- .setkey = ssi_ahash_setkey,
- .halg = {
- .digestsize = SHA1_DIGEST_SIZE,
- .statesize = sizeof(struct sha1_state),
- },
- },
+ .template_ahash = {
+ .init = ssi_ahash_init,
+ .update = ssi_ahash_update,
+ .final = ssi_ahash_final,
+ .finup = ssi_ahash_finup,
+ .digest = ssi_ahash_digest,
+ .export = ssi_ahash_export,
+ .import = ssi_ahash_import,
+ .setkey = ssi_ahash_setkey,
+ .halg = {
+ .digestsize = SHA1_DIGEST_SIZE,
+ .statesize = CC_STATE_SIZE(SHA1_DIGEST_SIZE),
+ },
},
.hash_mode = DRV_HASH_SHA1,
.hw_mode = DRV_HASH_HW_SHA1,
{
.name = "sha256",
.driver_name = "sha256-dx",
- .hmac_name = "hmac(sha256)",
- .hmac_driver_name = "hmac-sha256-dx",
+ .mac_name = "hmac(sha256)",
+ .mac_driver_name = "hmac-sha256-dx",
.blocksize = SHA256_BLOCK_SIZE,
- .synchronize = false,
- {
- .template_ahash = {
- .init = ssi_ahash_init,
- .update = ssi_ahash_update,
- .final = ssi_ahash_final,
- .finup = ssi_ahash_finup,
- .digest = ssi_ahash_digest,
-#ifdef EXPORT_FIXED
- .export = ssi_ahash_export,
- .import = ssi_ahash_import,
-#endif
- .setkey = ssi_ahash_setkey,
- .halg = {
- .digestsize = SHA256_DIGEST_SIZE,
- .statesize = sizeof(struct sha256_state),
- },
- },
+ .template_ahash = {
+ .init = ssi_ahash_init,
+ .update = ssi_ahash_update,
+ .final = ssi_ahash_final,
+ .finup = ssi_ahash_finup,
+ .digest = ssi_ahash_digest,
+ .export = ssi_ahash_export,
+ .import = ssi_ahash_import,
+ .setkey = ssi_ahash_setkey,
+ .halg = {
+ .digestsize = SHA256_DIGEST_SIZE,
+ .statesize = CC_STATE_SIZE(SHA256_DIGEST_SIZE)
+ },
},
.hash_mode = DRV_HASH_SHA256,
.hw_mode = DRV_HASH_HW_SHA256,
{
.name = "sha224",
.driver_name = "sha224-dx",
- .hmac_name = "hmac(sha224)",
- .hmac_driver_name = "hmac-sha224-dx",
+ .mac_name = "hmac(sha224)",
+ .mac_driver_name = "hmac-sha224-dx",
.blocksize = SHA224_BLOCK_SIZE,
- .synchronize = false,
- {
- .template_ahash = {
- .init = ssi_ahash_init,
- .update = ssi_ahash_update,
- .final = ssi_ahash_final,
- .finup = ssi_ahash_finup,
- .digest = ssi_ahash_digest,
-#ifdef EXPORT_FIXED
- .export = ssi_ahash_export,
- .import = ssi_ahash_import,
-#endif
- .setkey = ssi_ahash_setkey,
- .halg = {
- .digestsize = SHA224_DIGEST_SIZE,
- .statesize = sizeof(struct sha256_state),
- },
- },
+ .template_ahash = {
+ .init = ssi_ahash_init,
+ .update = ssi_ahash_update,
+ .final = ssi_ahash_final,
+ .finup = ssi_ahash_finup,
+ .digest = ssi_ahash_digest,
+ .export = ssi_ahash_export,
+ .import = ssi_ahash_import,
+ .setkey = ssi_ahash_setkey,
+ .halg = {
+ .digestsize = SHA224_DIGEST_SIZE,
+ .statesize = CC_STATE_SIZE(SHA224_DIGEST_SIZE),
+ },
},
.hash_mode = DRV_HASH_SHA224,
.hw_mode = DRV_HASH_HW_SHA256,
{
.name = "sha384",
.driver_name = "sha384-dx",
- .hmac_name = "hmac(sha384)",
- .hmac_driver_name = "hmac-sha384-dx",
+ .mac_name = "hmac(sha384)",
+ .mac_driver_name = "hmac-sha384-dx",
.blocksize = SHA384_BLOCK_SIZE,
- .synchronize = false,
- {
- .template_ahash = {
- .init = ssi_ahash_init,
- .update = ssi_ahash_update,
- .final = ssi_ahash_final,
- .finup = ssi_ahash_finup,
- .digest = ssi_ahash_digest,
-#ifdef EXPORT_FIXED
- .export = ssi_ahash_export,
- .import = ssi_ahash_import,
-#endif
- .setkey = ssi_ahash_setkey,
- .halg = {
- .digestsize = SHA384_DIGEST_SIZE,
- .statesize = sizeof(struct sha512_state),
- },
- },
+ .template_ahash = {
+ .init = ssi_ahash_init,
+ .update = ssi_ahash_update,
+ .final = ssi_ahash_final,
+ .finup = ssi_ahash_finup,
+ .digest = ssi_ahash_digest,
+ .export = ssi_ahash_export,
+ .import = ssi_ahash_import,
+ .setkey = ssi_ahash_setkey,
+ .halg = {
+ .digestsize = SHA384_DIGEST_SIZE,
+ .statesize = CC_STATE_SIZE(SHA384_DIGEST_SIZE),
+ },
},
.hash_mode = DRV_HASH_SHA384,
.hw_mode = DRV_HASH_HW_SHA512,
{
.name = "sha512",
.driver_name = "sha512-dx",
- .hmac_name = "hmac(sha512)",
- .hmac_driver_name = "hmac-sha512-dx",
+ .mac_name = "hmac(sha512)",
+ .mac_driver_name = "hmac-sha512-dx",
.blocksize = SHA512_BLOCK_SIZE,
- .synchronize = false,
- {
- .template_ahash = {
- .init = ssi_ahash_init,
- .update = ssi_ahash_update,
- .final = ssi_ahash_final,
- .finup = ssi_ahash_finup,
- .digest = ssi_ahash_digest,
-#ifdef EXPORT_FIXED
- .export = ssi_ahash_export,
- .import = ssi_ahash_import,
-#endif
- .setkey = ssi_ahash_setkey,
- .halg = {
- .digestsize = SHA512_DIGEST_SIZE,
- .statesize = sizeof(struct sha512_state),
- },
- },
+ .template_ahash = {
+ .init = ssi_ahash_init,
+ .update = ssi_ahash_update,
+ .final = ssi_ahash_final,
+ .finup = ssi_ahash_finup,
+ .digest = ssi_ahash_digest,
+ .export = ssi_ahash_export,
+ .import = ssi_ahash_import,
+ .setkey = ssi_ahash_setkey,
+ .halg = {
+ .digestsize = SHA512_DIGEST_SIZE,
+ .statesize = CC_STATE_SIZE(SHA512_DIGEST_SIZE),
+ },
},
.hash_mode = DRV_HASH_SHA512,
.hw_mode = DRV_HASH_HW_SHA512,
{
.name = "md5",
.driver_name = "md5-dx",
- .hmac_name = "hmac(md5)",
- .hmac_driver_name = "hmac-md5-dx",
+ .mac_name = "hmac(md5)",
+ .mac_driver_name = "hmac-md5-dx",
.blocksize = MD5_HMAC_BLOCK_SIZE,
- .synchronize = false,
- {
- .template_ahash = {
- .init = ssi_ahash_init,
- .update = ssi_ahash_update,
- .final = ssi_ahash_final,
- .finup = ssi_ahash_finup,
- .digest = ssi_ahash_digest,
-#ifdef EXPORT_FIXED
- .export = ssi_ahash_export,
- .import = ssi_ahash_import,
-#endif
- .setkey = ssi_ahash_setkey,
- .halg = {
- .digestsize = MD5_DIGEST_SIZE,
- .statesize = sizeof(struct md5_state),
- },
- },
+ .template_ahash = {
+ .init = ssi_ahash_init,
+ .update = ssi_ahash_update,
+ .final = ssi_ahash_final,
+ .finup = ssi_ahash_finup,
+ .digest = ssi_ahash_digest,
+ .export = ssi_ahash_export,
+ .import = ssi_ahash_import,
+ .setkey = ssi_ahash_setkey,
+ .halg = {
+ .digestsize = MD5_DIGEST_SIZE,
+ .statesize = CC_STATE_SIZE(MD5_DIGEST_SIZE),
+ },
},
.hash_mode = DRV_HASH_MD5,
.hw_mode = DRV_HASH_HW_MD5,
.inter_digestsize = MD5_DIGEST_SIZE,
},
{
- .name = "xcbc(aes)",
- .driver_name = "xcbc-aes-dx",
+ .mac_name = "xcbc(aes)",
+ .mac_driver_name = "xcbc-aes-dx",
.blocksize = AES_BLOCK_SIZE,
- .synchronize = false,
- {
- .template_ahash = {
- .init = ssi_ahash_init,
- .update = ssi_mac_update,
- .final = ssi_mac_final,
- .finup = ssi_mac_finup,
- .digest = ssi_mac_digest,
- .setkey = ssi_xcbc_setkey,
-#ifdef EXPORT_FIXED
- .export = ssi_ahash_export,
- .import = ssi_ahash_import,
-#endif
- .halg = {
- .digestsize = AES_BLOCK_SIZE,
- .statesize = sizeof(struct aeshash_state),
- },
- },
+ .template_ahash = {
+ .init = ssi_ahash_init,
+ .update = ssi_mac_update,
+ .final = ssi_mac_final,
+ .finup = ssi_mac_finup,
+ .digest = ssi_mac_digest,
+ .setkey = ssi_xcbc_setkey,
+ .export = ssi_ahash_export,
+ .import = ssi_ahash_import,
+ .halg = {
+ .digestsize = AES_BLOCK_SIZE,
+ .statesize = CC_STATE_SIZE(AES_BLOCK_SIZE),
+ },
},
.hash_mode = DRV_HASH_NULL,
.hw_mode = DRV_CIPHER_XCBC_MAC,
},
#if SSI_CC_HAS_CMAC
{
- .name = "cmac(aes)",
- .driver_name = "cmac-aes-dx",
+ .mac_name = "cmac(aes)",
+ .mac_driver_name = "cmac-aes-dx",
.blocksize = AES_BLOCK_SIZE,
- .synchronize = false,
- {
- .template_ahash = {
- .init = ssi_ahash_init,
- .update = ssi_mac_update,
- .final = ssi_mac_final,
- .finup = ssi_mac_finup,
- .digest = ssi_mac_digest,
- .setkey = ssi_cmac_setkey,
-#ifdef EXPORT_FIXED
- .export = ssi_ahash_export,
- .import = ssi_ahash_import,
-#endif
- .halg = {
- .digestsize = AES_BLOCK_SIZE,
- .statesize = sizeof(struct aeshash_state),
- },
- },
+ .template_ahash = {
+ .init = ssi_ahash_init,
+ .update = ssi_mac_update,
+ .final = ssi_mac_final,
+ .finup = ssi_mac_finup,
+ .digest = ssi_mac_digest,
+ .setkey = ssi_cmac_setkey,
+ .export = ssi_ahash_export,
+ .import = ssi_ahash_import,
+ .halg = {
+ .digestsize = AES_BLOCK_SIZE,
+ .statesize = CC_STATE_SIZE(AES_BLOCK_SIZE),
+ },
},
.hash_mode = DRV_HASH_NULL,
.hw_mode = DRV_CIPHER_CMAC,
.inter_digestsize = AES_BLOCK_SIZE,
},
#endif
-
+
};
static struct ssi_hash_alg *
{
struct ssi_hash_alg *t_crypto_alg;
struct crypto_alg *alg;
+ struct ahash_alg *halg;
t_crypto_alg = kzalloc(sizeof(struct ssi_hash_alg), GFP_KERNEL);
if (!t_crypto_alg) {
return ERR_PTR(-ENOMEM);
}
- t_crypto_alg->synchronize = template->synchronize;
- if (template->synchronize) {
- struct shash_alg *halg;
- t_crypto_alg->shash_alg = template->template_shash;
- halg = &t_crypto_alg->shash_alg;
- alg = &halg->base;
- if (!keyed) halg->setkey = NULL;
- } else {
- struct ahash_alg *halg;
- t_crypto_alg->ahash_alg = template->template_ahash;
- halg = &t_crypto_alg->ahash_alg;
- alg = &halg->halg.base;
- if (!keyed) halg->setkey = NULL;
- }
+ t_crypto_alg->ahash_alg = template->template_ahash;
+ halg = &t_crypto_alg->ahash_alg;
+ alg = &halg->halg.base;
if (keyed) {
snprintf(alg->cra_name, CRYPTO_MAX_ALG_NAME, "%s",
- template->hmac_name);
+ template->mac_name);
snprintf(alg->cra_driver_name, CRYPTO_MAX_ALG_NAME, "%s",
- template->hmac_driver_name);
+ template->mac_driver_name);
} else {
+ halg->setkey = NULL;
snprintf(alg->cra_name, CRYPTO_MAX_ALG_NAME, "%s",
template->name);
snprintf(alg->cra_driver_name, CRYPTO_MAX_ALG_NAME, "%s",
alg->cra_blocksize = template->blocksize;
alg->cra_alignmask = 0;
alg->cra_exit = ssi_hash_cra_exit;
-
- if (template->synchronize) {
- alg->cra_init = ssi_shash_cra_init;
- alg->cra_flags = CRYPTO_ALG_TYPE_SHASH |
- CRYPTO_ALG_KERN_DRIVER_ONLY;
- alg->cra_type = &crypto_shash_type;
- } else {
- alg->cra_init = ssi_ahash_cra_init;
- alg->cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_TYPE_AHASH |
+
+ alg->cra_init = ssi_ahash_cra_init;
+ alg->cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_TYPE_AHASH |
CRYPTO_ALG_KERN_DRIVER_ONLY;
- alg->cra_type = &crypto_ahash_type;
- }
+ alg->cra_type = &crypto_ahash_type;
t_crypto_alg->hash_mode = template->hash_mode;
t_crypto_alg->hw_mode = template->hw_mode;
struct ssi_hash_handle *hash_handle = drvdata->hash_handle;
ssi_sram_addr_t sram_buff_ofs = hash_handle->digest_len_sram_addr;
unsigned int larval_seq_len = 0;
- HwDesc_s larval_seq[CC_DIGEST_SIZE_MAX/sizeof(uint32_t)];
+ struct cc_hw_desc larval_seq[CC_DIGEST_SIZE_MAX / sizeof(u32)];
int rc = 0;
#if (DX_DEV_SHA_MAX > 256)
int i;
#if (DX_DEV_SHA_MAX > 256)
/* We are forced to swap each double-word larval before copying to sram */
for (i = 0; i < ARRAY_SIZE(sha384_init); i++) {
- const uint32_t const0 = ((uint32_t *)((uint64_t *)&sha384_init[i]))[1];
- const uint32_t const1 = ((uint32_t *)((uint64_t *)&sha384_init[i]))[0];
+ const u32 const0 = ((u32 *)((u64 *)&sha384_init[i]))[1];
+ const u32 const1 = ((u32 *)((u64 *)&sha384_init[i]))[0];
ssi_sram_mgr_const2sram_desc(&const0, sram_buff_ofs, 1,
larval_seq, &larval_seq_len);
- sram_buff_ofs += sizeof(uint32_t);
+ sram_buff_ofs += sizeof(u32);
ssi_sram_mgr_const2sram_desc(&const1, sram_buff_ofs, 1,
larval_seq, &larval_seq_len);
- sram_buff_ofs += sizeof(uint32_t);
+ sram_buff_ofs += sizeof(u32);
}
rc = send_request_init(drvdata, larval_seq, larval_seq_len);
if (unlikely(rc != 0)) {
larval_seq_len = 0;
for (i = 0; i < ARRAY_SIZE(sha512_init); i++) {
- const uint32_t const0 = ((uint32_t *)((uint64_t *)&sha512_init[i]))[1];
- const uint32_t const1 = ((uint32_t *)((uint64_t *)&sha512_init[i]))[0];
+ const u32 const0 = ((u32 *)((u64 *)&sha512_init[i]))[1];
+ const u32 const1 = ((u32 *)((u64 *)&sha512_init[i]))[0];
ssi_sram_mgr_const2sram_desc(&const0, sram_buff_ofs, 1,
larval_seq, &larval_seq_len);
- sram_buff_ofs += sizeof(uint32_t);
+ sram_buff_ofs += sizeof(u32);
ssi_sram_mgr_const2sram_desc(&const1, sram_buff_ofs, 1,
larval_seq, &larval_seq_len);
- sram_buff_ofs += sizeof(uint32_t);
+ sram_buff_ofs += sizeof(u32);
}
rc = send_request_init(drvdata, larval_seq, larval_seq_len);
if (unlikely(rc != 0)) {
{
struct ssi_hash_handle *hash_handle;
ssi_sram_addr_t sram_buff;
- uint32_t sram_size_to_alloc;
+ u32 sram_size_to_alloc;
int rc = 0;
int alg;
hash_handle = kzalloc(sizeof(struct ssi_hash_handle), GFP_KERNEL);
- if (hash_handle == NULL) {
+ if (!hash_handle) {
SSI_LOG_ERR("kzalloc failed to allocate %zu B\n",
sizeof(struct ssi_hash_handle));
rc = -ENOMEM;
sizeof(sha1_init) +
sizeof(sha224_init) +
sizeof(sha256_init);
-
+
sram_buff = ssi_sram_mgr_alloc(drvdata, sram_size_to_alloc);
if (sram_buff == NULL_SRAM_ADDR) {
SSI_LOG_ERR("SRAM pool exhausted\n");
/* ahash registration */
for (alg = 0; alg < ARRAY_SIZE(driver_hash); alg++) {
struct ssi_hash_alg *t_alg;
-
+ int hw_mode = driver_hash[alg].hw_mode;
+
/* register hmac version */
+ t_alg = ssi_hash_create_alg(&driver_hash[alg], true);
+ if (IS_ERR(t_alg)) {
+ rc = PTR_ERR(t_alg);
+ SSI_LOG_ERR("%s alg allocation failed\n",
+ driver_hash[alg].driver_name);
+ goto fail;
+ }
+ t_alg->drvdata = drvdata;
- if ((((struct ssi_hash_template)driver_hash[alg]).hw_mode != DRV_CIPHER_XCBC_MAC) &&
- (((struct ssi_hash_template)driver_hash[alg]).hw_mode != DRV_CIPHER_CMAC)) {
- t_alg = ssi_hash_create_alg(&driver_hash[alg], true);
- if (IS_ERR(t_alg)) {
- rc = PTR_ERR(t_alg);
- SSI_LOG_ERR("%s alg allocation failed\n",
- driver_hash[alg].driver_name);
- goto fail;
- }
- t_alg->drvdata = drvdata;
-
- if (t_alg->synchronize) {
- rc = crypto_register_shash(&t_alg->shash_alg);
- if (unlikely(rc != 0)) {
- SSI_LOG_ERR("%s alg registration failed\n",
- t_alg->shash_alg.base.cra_driver_name);
- kfree(t_alg);
- goto fail;
- } else
- list_add_tail(&t_alg->entry, &hash_handle->hash_list);
- } else {
- rc = crypto_register_ahash(&t_alg->ahash_alg);
- if (unlikely(rc != 0)) {
- SSI_LOG_ERR("%s alg registration failed\n",
- t_alg->ahash_alg.halg.base.cra_driver_name);
- kfree(t_alg);
- goto fail;
- } else
- list_add_tail(&t_alg->entry, &hash_handle->hash_list);
- }
+ rc = crypto_register_ahash(&t_alg->ahash_alg);
+ if (unlikely(rc)) {
+ SSI_LOG_ERR("%s alg registration failed\n",
+ driver_hash[alg].driver_name);
+ kfree(t_alg);
+ goto fail;
+ } else {
+ list_add_tail(&t_alg->entry,
+ &hash_handle->hash_list);
}
+ if ((hw_mode == DRV_CIPHER_XCBC_MAC) ||
+ (hw_mode == DRV_CIPHER_CMAC))
+ continue;
+
/* register hash version */
t_alg = ssi_hash_create_alg(&driver_hash[alg], false);
if (IS_ERR(t_alg)) {
goto fail;
}
t_alg->drvdata = drvdata;
-
- if (t_alg->synchronize) {
- rc = crypto_register_shash(&t_alg->shash_alg);
- if (unlikely(rc != 0)) {
- SSI_LOG_ERR("%s alg registration failed\n",
- t_alg->shash_alg.base.cra_driver_name);
- kfree(t_alg);
- goto fail;
- } else
- list_add_tail(&t_alg->entry, &hash_handle->hash_list);
-
+
+ rc = crypto_register_ahash(&t_alg->ahash_alg);
+ if (unlikely(rc)) {
+ SSI_LOG_ERR("%s alg registration failed\n",
+ driver_hash[alg].driver_name);
+ kfree(t_alg);
+ goto fail;
} else {
- rc = crypto_register_ahash(&t_alg->ahash_alg);
- if (unlikely(rc != 0)) {
- SSI_LOG_ERR("%s alg registration failed\n",
- t_alg->ahash_alg.halg.base.cra_driver_name);
- kfree(t_alg);
- goto fail;
- } else
- list_add_tail(&t_alg->entry, &hash_handle->hash_list);
+ list_add_tail(&t_alg->entry, &hash_handle->hash_list);
}
}
fail:
- if (drvdata->hash_handle != NULL) {
+ if (drvdata->hash_handle) {
kfree(drvdata->hash_handle);
drvdata->hash_handle = NULL;
}
struct ssi_hash_alg *t_hash_alg, *hash_n;
struct ssi_hash_handle *hash_handle = drvdata->hash_handle;
- if (hash_handle != NULL) {
-
+ if (hash_handle) {
list_for_each_entry_safe(t_hash_alg, hash_n, &hash_handle->hash_list, entry) {
- if (t_hash_alg->synchronize) {
- crypto_unregister_shash(&t_hash_alg->shash_alg);
- } else {
- crypto_unregister_ahash(&t_hash_alg->ahash_alg);
- }
+ crypto_unregister_ahash(&t_hash_alg->ahash_alg);
list_del(&t_hash_alg->entry);
kfree(t_hash_alg);
}
-
+
kfree(hash_handle);
drvdata->hash_handle = NULL;
}
return 0;
}
-static void ssi_hash_create_xcbc_setup(struct ahash_request *areq,
- HwDesc_s desc[],
+static void ssi_hash_create_xcbc_setup(struct ahash_request *areq,
+ struct cc_hw_desc desc[],
unsigned int *seq_size) {
unsigned int idx = *seq_size;
struct ahash_req_ctx *state = ahash_request_ctx(areq);
struct ssi_hash_ctx *ctx = crypto_ahash_ctx(tfm);
/* Setup XCBC MAC K1 */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, (ctx->opad_tmp_keys_dma_addr
- + XCBC_MAC_K1_OFFSET),
- CC_AES_128_BIT_KEY_SIZE, NS_BIT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_XCBC_MAC);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], CC_AES_128_BIT_KEY_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, (ctx->opad_tmp_keys_dma_addr +
+ XCBC_MAC_K1_OFFSET),
+ CC_AES_128_BIT_KEY_SIZE, NS_BIT);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_XCBC_MAC);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_key_size_aes(&desc[idx], CC_AES_128_BIT_KEY_SIZE);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
/* Setup XCBC MAC K2 */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, (ctx->opad_tmp_keys_dma_addr
- + XCBC_MAC_K2_OFFSET),
- CC_AES_128_BIT_KEY_SIZE, NS_BIT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE1);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_XCBC_MAC);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], CC_AES_128_BIT_KEY_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, (ctx->opad_tmp_keys_dma_addr +
+ XCBC_MAC_K2_OFFSET),
+ CC_AES_128_BIT_KEY_SIZE, NS_BIT);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE1);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_XCBC_MAC);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_key_size_aes(&desc[idx], CC_AES_128_BIT_KEY_SIZE);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
/* Setup XCBC MAC K3 */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, (ctx->opad_tmp_keys_dma_addr
- + XCBC_MAC_K3_OFFSET),
- CC_AES_128_BIT_KEY_SIZE, NS_BIT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE2);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_XCBC_MAC);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], CC_AES_128_BIT_KEY_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, (ctx->opad_tmp_keys_dma_addr +
+ XCBC_MAC_K3_OFFSET),
+ CC_AES_128_BIT_KEY_SIZE, NS_BIT);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE2);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_XCBC_MAC);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_key_size_aes(&desc[idx], CC_AES_128_BIT_KEY_SIZE);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
/* Loading MAC state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr, CC_AES_BLOCK_SIZE, NS_BIT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_XCBC_MAC);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], CC_AES_128_BIT_KEY_SIZE);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr,
+ CC_AES_BLOCK_SIZE, NS_BIT);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_XCBC_MAC);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_key_size_aes(&desc[idx], CC_AES_128_BIT_KEY_SIZE);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
*seq_size = idx;
}
-static void ssi_hash_create_cmac_setup(struct ahash_request *areq,
- HwDesc_s desc[],
+static void ssi_hash_create_cmac_setup(struct ahash_request *areq,
+ struct cc_hw_desc desc[],
unsigned int *seq_size)
{
unsigned int idx = *seq_size;
struct ssi_hash_ctx *ctx = crypto_ahash_ctx(tfm);
/* Setup CMAC Key */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, ctx->opad_tmp_keys_dma_addr,
- ((ctx->key_params.keylen == 24) ? AES_MAX_KEY_SIZE : ctx->key_params.keylen), NS_BIT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_KEY0);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_CMAC);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], ctx->key_params.keylen);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, ctx->opad_tmp_keys_dma_addr,
+ ((ctx->key_params.keylen == 24) ? AES_MAX_KEY_SIZE :
+ ctx->key_params.keylen), NS_BIT);
+ set_setup_mode(&desc[idx], SETUP_LOAD_KEY0);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_CMAC);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_key_size_aes(&desc[idx], ctx->key_params.keylen);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
/* Load MAC state */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr, CC_AES_BLOCK_SIZE, NS_BIT);
- HW_DESC_SET_SETUP_MODE(&desc[idx], SETUP_LOAD_STATE0);
- HW_DESC_SET_CIPHER_MODE(&desc[idx], DRV_CIPHER_CMAC);
- HW_DESC_SET_CIPHER_CONFIG0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_KEY_SIZE_AES(&desc[idx], ctx->key_params.keylen);
- HW_DESC_SET_FLOW_MODE(&desc[idx], S_DIN_to_AES);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI, state->digest_buff_dma_addr,
+ CC_AES_BLOCK_SIZE, NS_BIT);
+ set_setup_mode(&desc[idx], SETUP_LOAD_STATE0);
+ set_cipher_mode(&desc[idx], DRV_CIPHER_CMAC);
+ set_cipher_config0(&desc[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_key_size_aes(&desc[idx], ctx->key_params.keylen);
+ set_flow_mode(&desc[idx], S_DIN_to_AES);
idx++;
*seq_size = idx;
}
static void ssi_hash_create_data_desc(struct ahash_req_ctx *areq_ctx,
struct ssi_hash_ctx *ctx,
unsigned int flow_mode,
- HwDesc_s desc[],
- bool is_not_last_data,
+ struct cc_hw_desc desc[],
+ bool is_not_last_data,
unsigned int *seq_size)
{
unsigned int idx = *seq_size;
if (likely(areq_ctx->data_dma_buf_type == SSI_DMA_BUF_DLLI)) {
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- sg_dma_address(areq_ctx->curr_sg),
- areq_ctx->curr_sg->length, NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], flow_mode);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI,
+ sg_dma_address(areq_ctx->curr_sg),
+ areq_ctx->curr_sg->length, NS_BIT);
+ set_flow_mode(&desc[idx], flow_mode);
idx++;
} else {
if (areq_ctx->data_dma_buf_type == SSI_DMA_BUF_NULL) {
return;
}
/* bypass */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_DLLI,
- areq_ctx->mlli_params.mlli_dma_addr,
- areq_ctx->mlli_params.mlli_len,
- NS_BIT);
- HW_DESC_SET_DOUT_SRAM(&desc[idx],
- ctx->drvdata->mlli_sram_addr,
- areq_ctx->mlli_params.mlli_len);
- HW_DESC_SET_FLOW_MODE(&desc[idx], BYPASS);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_DLLI,
+ areq_ctx->mlli_params.mlli_dma_addr,
+ areq_ctx->mlli_params.mlli_len, NS_BIT);
+ set_dout_sram(&desc[idx], ctx->drvdata->mlli_sram_addr,
+ areq_ctx->mlli_params.mlli_len);
+ set_flow_mode(&desc[idx], BYPASS);
idx++;
/* process */
- HW_DESC_INIT(&desc[idx]);
- HW_DESC_SET_DIN_TYPE(&desc[idx], DMA_MLLI,
- ctx->drvdata->mlli_sram_addr,
- areq_ctx->mlli_nents,
- NS_BIT);
- HW_DESC_SET_FLOW_MODE(&desc[idx], flow_mode);
+ hw_desc_init(&desc[idx]);
+ set_din_type(&desc[idx], DMA_MLLI,
+ ctx->drvdata->mlli_sram_addr,
+ areq_ctx->mlli_nents, NS_BIT);
+ set_flow_mode(&desc[idx], flow_mode);
idx++;
}
- if (is_not_last_data) {
- HW_DESC_SET_DIN_NOT_LAST_INDICATION(&desc[idx-1]);
- }
+ if (is_not_last_data)
+ set_din_not_last_indication(&desc[(idx - 1)]);
/* return updated desc sequence size */
*seq_size = idx;
}
/*!
- * Gets the address of the initial digest in SRAM
+ * Gets the address of the initial digest in SRAM
* according to the given hash mode
- *
+ *
* \param drvdata
* \param mode The Hash mode. Supported modes: MD5/SHA1/SHA224/SHA256
- *
- * \return uint32_t The address of the inital digest in SRAM
+ *
+ * \return u32 The address of the inital digest in SRAM
*/
-ssi_sram_addr_t ssi_ahash_get_larval_digest_sram_addr(void *drvdata, uint32_t mode)
+ssi_sram_addr_t ssi_ahash_get_larval_digest_sram_addr(void *drvdata, u32 mode)
{
struct ssi_drvdata *_drvdata = (struct ssi_drvdata *)drvdata;
struct ssi_hash_handle *hash_handle = _drvdata->hash_handle;
}
ssi_sram_addr_t
-ssi_ahash_get_initial_digest_len_sram_addr(void *drvdata, uint32_t mode)
+ssi_ahash_get_initial_digest_len_sram_addr(void *drvdata, u32 mode)
{
struct ssi_drvdata *_drvdata = (struct ssi_drvdata *)drvdata;
struct ssi_hash_handle *hash_handle = _drvdata->hash_handle;
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
/* \file ssi_hash.h
- ARM CryptoCell Hash Crypto API
+ * ARM CryptoCell Hash Crypto API
*/
#ifndef __SSI_HASH_H__
#define XCBC_MAC_K2_OFFSET 16
#define XCBC_MAC_K3_OFFSET 32
+#define CC_EXPORT_MAGIC 0xC2EE1070U
+
// this struct was taken from drivers/crypto/nx/nx-aes-xcbc.c and it is used for xcbc/cmac statesize
struct aeshash_state {
u8 state[AES_BLOCK_SIZE];
/* ahash state */
struct ahash_req_ctx {
- uint8_t* buff0;
- uint8_t* buff1;
- uint8_t* digest_result_buff;
+ u8 *buff0;
+ u8 *buff1;
+ u8 *digest_result_buff;
struct async_gen_req_ctx gen_ctx;
enum ssi_req_dma_buf_type data_dma_buf_type;
- uint8_t *digest_buff;
- uint8_t *opad_digest_buff;
- uint8_t *digest_bytes_len;
+ u8 *digest_buff;
+ u8 *opad_digest_buff;
+ u8 *digest_bytes_len;
dma_addr_t opad_digest_dma_addr;
dma_addr_t digest_buff_dma_addr;
dma_addr_t digest_bytes_len_dma_addr;
dma_addr_t digest_result_dma_addr;
- uint32_t buff0_cnt;
- uint32_t buff1_cnt;
- uint32_t buff_index;
- uint32_t xcbc_count; /* count xcbc update operatations */
+ u32 buff0_cnt;
+ u32 buff1_cnt;
+ u32 buff_index;
+ u32 xcbc_count; /* count xcbc update operatations */
struct scatterlist buff_sg[2];
struct scatterlist *curr_sg;
- uint32_t in_nents;
- uint32_t mlli_nents;
- struct mlli_params mlli_params;
+ u32 in_nents;
+ u32 mlli_nents;
+ struct mlli_params mlli_params;
};
int ssi_hash_alloc(struct ssi_drvdata *drvdata);
/*!
* Gets the initial digest length
- *
- * \param drvdata
+ *
+ * \param drvdata
* \param mode The Hash mode. Supported modes: MD5/SHA1/SHA224/SHA256/SHA384/SHA512
- *
- * \return uint32_t returns the address of the initial digest length in SRAM
+ *
+ * \return u32 returns the address of the initial digest length in SRAM
*/
ssi_sram_addr_t
-ssi_ahash_get_initial_digest_len_sram_addr(void *drvdata, uint32_t mode);
+ssi_ahash_get_initial_digest_len_sram_addr(void *drvdata, u32 mode);
/*!
- * Gets the address of the initial digest in SRAM
+ * Gets the address of the initial digest in SRAM
* according to the given hash mode
- *
- * \param drvdata
+ *
+ * \param drvdata
* \param mode The Hash mode. Supported modes: MD5/SHA1/SHA224/SHA256/SHA384/SHA512
- *
- * \return uint32_t The address of the inital digest in SRAM
+ *
+ * \return u32 The address of the inital digest in SRAM
*/
-ssi_sram_addr_t ssi_ahash_get_larval_digest_sram_addr(void *drvdata, uint32_t mode);
+ssi_sram_addr_t ssi_ahash_get_larval_digest_sram_addr(void *drvdata, u32 mode);
#endif /*__SSI_HASH_H__*/
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
/* The max. size of pool *MUST* be <= SRAM total size */
#define SSI_IVPOOL_SIZE 1024
/* The first 32B fraction of pool are dedicated to the
- next encryption "key" & "IV" for pool regeneration */
+ * next encryption "key" & "IV" for pool regeneration
+ */
#define SSI_IVPOOL_META_SIZE (CC_AES_IV_SIZE + AES_KEYSIZE_128)
#define SSI_IVPOOL_GEN_SEQ_LEN 4
/**
- * struct ssi_ivgen_ctx -IV pool generation context
- * @pool: the start address of the iv-pool resides in internal RAM
+ * struct ssi_ivgen_ctx -IV pool generation context
+ * @pool: the start address of the iv-pool resides in internal RAM
* @ctr_key_dma: address of pool's encryption key material in internal RAM
* @ctr_iv_dma: address of pool's counter iv in internal RAM
* @next_iv_ofs: the offset to the next available IV in pool
ssi_sram_addr_t pool;
ssi_sram_addr_t ctr_key;
ssi_sram_addr_t ctr_iv;
- uint32_t next_iv_ofs;
- uint8_t *pool_meta;
+ u32 next_iv_ofs;
+ u8 *pool_meta;
dma_addr_t pool_meta_dma;
};
/*!
- * Generates SSI_IVPOOL_SIZE of random bytes by
+ * Generates SSI_IVPOOL_SIZE of random bytes by
* encrypting 0's using AES128-CTR.
- *
+ *
* \param ivgen iv-pool context
* \param iv_seq IN/OUT array to the descriptors sequence
- * \param iv_seq_len IN/OUT pointer to the sequence length
+ * \param iv_seq_len IN/OUT pointer to the sequence length
*/
static int ssi_ivgen_generate_pool(
struct ssi_ivgen_ctx *ivgen_ctx,
- HwDesc_s iv_seq[],
+ struct cc_hw_desc iv_seq[],
unsigned int *iv_seq_len)
{
unsigned int idx = *iv_seq_len;
- if ( (*iv_seq_len + SSI_IVPOOL_GEN_SEQ_LEN) > SSI_IVPOOL_SEQ_LEN) {
+ if ((*iv_seq_len + SSI_IVPOOL_GEN_SEQ_LEN) > SSI_IVPOOL_SEQ_LEN) {
/* The sequence will be longer than allowed */
return -EINVAL;
}
/* Setup key */
- HW_DESC_INIT(&iv_seq[idx]);
- HW_DESC_SET_DIN_SRAM(&iv_seq[idx], ivgen_ctx->ctr_key, AES_KEYSIZE_128);
- HW_DESC_SET_SETUP_MODE(&iv_seq[idx], SETUP_LOAD_KEY0);
- HW_DESC_SET_CIPHER_CONFIG0(&iv_seq[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_FLOW_MODE(&iv_seq[idx], S_DIN_to_AES);
- HW_DESC_SET_KEY_SIZE_AES(&iv_seq[idx], CC_AES_128_BIT_KEY_SIZE);
- HW_DESC_SET_CIPHER_MODE(&iv_seq[idx], DRV_CIPHER_CTR);
+ hw_desc_init(&iv_seq[idx]);
+ set_din_sram(&iv_seq[idx], ivgen_ctx->ctr_key, AES_KEYSIZE_128);
+ set_setup_mode(&iv_seq[idx], SETUP_LOAD_KEY0);
+ set_cipher_config0(&iv_seq[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_flow_mode(&iv_seq[idx], S_DIN_to_AES);
+ set_key_size_aes(&iv_seq[idx], CC_AES_128_BIT_KEY_SIZE);
+ set_cipher_mode(&iv_seq[idx], DRV_CIPHER_CTR);
idx++;
/* Setup cipher state */
- HW_DESC_INIT(&iv_seq[idx]);
- HW_DESC_SET_DIN_SRAM(&iv_seq[idx], ivgen_ctx->ctr_iv, CC_AES_IV_SIZE);
- HW_DESC_SET_CIPHER_CONFIG0(&iv_seq[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
- HW_DESC_SET_FLOW_MODE(&iv_seq[idx], S_DIN_to_AES);
- HW_DESC_SET_SETUP_MODE(&iv_seq[idx], SETUP_LOAD_STATE1);
- HW_DESC_SET_KEY_SIZE_AES(&iv_seq[idx], CC_AES_128_BIT_KEY_SIZE);
- HW_DESC_SET_CIPHER_MODE(&iv_seq[idx], DRV_CIPHER_CTR);
+ hw_desc_init(&iv_seq[idx]);
+ set_din_sram(&iv_seq[idx], ivgen_ctx->ctr_iv, CC_AES_IV_SIZE);
+ set_cipher_config0(&iv_seq[idx], DESC_DIRECTION_ENCRYPT_ENCRYPT);
+ set_flow_mode(&iv_seq[idx], S_DIN_to_AES);
+ set_setup_mode(&iv_seq[idx], SETUP_LOAD_STATE1);
+ set_key_size_aes(&iv_seq[idx], CC_AES_128_BIT_KEY_SIZE);
+ set_cipher_mode(&iv_seq[idx], DRV_CIPHER_CTR);
idx++;
/* Perform dummy encrypt to skip first block */
- HW_DESC_INIT(&iv_seq[idx]);
- HW_DESC_SET_DIN_CONST(&iv_seq[idx], 0, CC_AES_IV_SIZE);
- HW_DESC_SET_DOUT_SRAM(&iv_seq[idx], ivgen_ctx->pool, CC_AES_IV_SIZE);
- HW_DESC_SET_FLOW_MODE(&iv_seq[idx], DIN_AES_DOUT);
+ hw_desc_init(&iv_seq[idx]);
+ set_din_const(&iv_seq[idx], 0, CC_AES_IV_SIZE);
+ set_dout_sram(&iv_seq[idx], ivgen_ctx->pool, CC_AES_IV_SIZE);
+ set_flow_mode(&iv_seq[idx], DIN_AES_DOUT);
idx++;
/* Generate IV pool */
- HW_DESC_INIT(&iv_seq[idx]);
- HW_DESC_SET_DIN_CONST(&iv_seq[idx], 0, SSI_IVPOOL_SIZE);
- HW_DESC_SET_DOUT_SRAM(&iv_seq[idx], ivgen_ctx->pool, SSI_IVPOOL_SIZE);
- HW_DESC_SET_FLOW_MODE(&iv_seq[idx], DIN_AES_DOUT);
+ hw_desc_init(&iv_seq[idx]);
+ set_din_const(&iv_seq[idx], 0, SSI_IVPOOL_SIZE);
+ set_dout_sram(&iv_seq[idx], ivgen_ctx->pool, SSI_IVPOOL_SIZE);
+ set_flow_mode(&iv_seq[idx], DIN_AES_DOUT);
idx++;
*iv_seq_len = idx; /* Update sequence length */
}
/*!
- * Generates the initial pool in SRAM.
- * This function should be invoked when resuming DX driver.
- *
- * \param drvdata
- *
+ * Generates the initial pool in SRAM.
+ * This function should be invoked when resuming DX driver.
+ *
+ * \param drvdata
+ *
* \return int Zero for success, negative value otherwise.
*/
int ssi_ivgen_init_sram_pool(struct ssi_drvdata *drvdata)
{
struct ssi_ivgen_ctx *ivgen_ctx = drvdata->ivgen_handle;
- HwDesc_s iv_seq[SSI_IVPOOL_SEQ_LEN];
+ struct cc_hw_desc iv_seq[SSI_IVPOOL_SEQ_LEN];
unsigned int iv_seq_len = 0;
int rc;
ivgen_ctx->ctr_iv = ivgen_ctx->pool + AES_KEYSIZE_128;
/* Copy initial enc. key and IV to SRAM at a single descriptor */
- HW_DESC_INIT(&iv_seq[iv_seq_len]);
- HW_DESC_SET_DIN_TYPE(&iv_seq[iv_seq_len], DMA_DLLI,
- ivgen_ctx->pool_meta_dma, SSI_IVPOOL_META_SIZE,
- NS_BIT);
- HW_DESC_SET_DOUT_SRAM(&iv_seq[iv_seq_len], ivgen_ctx->pool,
- SSI_IVPOOL_META_SIZE);
- HW_DESC_SET_FLOW_MODE(&iv_seq[iv_seq_len], BYPASS);
+ hw_desc_init(&iv_seq[iv_seq_len]);
+ set_din_type(&iv_seq[iv_seq_len], DMA_DLLI, ivgen_ctx->pool_meta_dma,
+ SSI_IVPOOL_META_SIZE, NS_BIT);
+ set_dout_sram(&iv_seq[iv_seq_len], ivgen_ctx->pool,
+ SSI_IVPOOL_META_SIZE);
+ set_flow_mode(&iv_seq[iv_seq_len], BYPASS);
iv_seq_len++;
/* Generate initial pool */
rc = ssi_ivgen_generate_pool(ivgen_ctx, iv_seq, &iv_seq_len);
- if (unlikely(rc != 0)) {
+ if (unlikely(rc != 0))
return rc;
- }
+
/* Fire-and-forget */
return send_request_init(drvdata, iv_seq, iv_seq_len);
}
/*!
* Free iv-pool and ivgen context.
- *
- * \param drvdata
+ *
+ * \param drvdata
*/
void ssi_ivgen_fini(struct ssi_drvdata *drvdata)
{
struct ssi_ivgen_ctx *ivgen_ctx = drvdata->ivgen_handle;
struct device *device = &(drvdata->plat_dev->dev);
- if (ivgen_ctx == NULL)
+ if (!ivgen_ctx)
return;
- if (ivgen_ctx->pool_meta != NULL) {
+ if (ivgen_ctx->pool_meta) {
memset(ivgen_ctx->pool_meta, 0, SSI_IVPOOL_META_SIZE);
- SSI_RESTORE_DMA_ADDR_TO_48BIT(ivgen_ctx->pool_meta_dma);
dma_free_coherent(device, SSI_IVPOOL_META_SIZE,
ivgen_ctx->pool_meta, ivgen_ctx->pool_meta_dma);
}
}
/*!
- * Allocates iv-pool and maps resources.
- * This function generates the first IV pool.
- *
+ * Allocates iv-pool and maps resources.
+ * This function generates the first IV pool.
+ *
* \param drvdata Driver's private context
- *
+ *
* \return int Zero for success, negative value otherwise.
*/
int ssi_ivgen_init(struct ssi_drvdata *drvdata)
rc = -ENOMEM;
goto out;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(ivgen_ctx->pool_meta_dma,
- SSI_IVPOOL_META_SIZE);
/* Allocate IV pool in SRAM */
ivgen_ctx->pool = ssi_sram_mgr_alloc(drvdata, SSI_IVPOOL_SIZE);
if (ivgen_ctx->pool == NULL_SRAM_ADDR) {
/*!
* Acquires 16 Bytes IV from the iv-pool
- *
+ *
* \param drvdata Driver private context
* \param iv_out_dma Array of physical IV out addresses
* \param iv_out_dma_len Length of iv_out_dma array (additional elements of iv_out_dma array are ignore)
- * \param iv_out_size May be 8 or 16 bytes long
+ * \param iv_out_size May be 8 or 16 bytes long
* \param iv_seq IN/OUT array to the descriptors sequence
- * \param iv_seq_len IN/OUT pointer to the sequence length
- *
- * \return int Zero for success, negative value otherwise.
+ * \param iv_seq_len IN/OUT pointer to the sequence length
+ *
+ * \return int Zero for success, negative value otherwise.
*/
int ssi_ivgen_getiv(
struct ssi_drvdata *drvdata,
dma_addr_t iv_out_dma[],
unsigned int iv_out_dma_len,
unsigned int iv_out_size,
- HwDesc_s iv_seq[],
+ struct cc_hw_desc iv_seq[],
unsigned int *iv_seq_len)
{
struct ssi_ivgen_ctx *ivgen_ctx = drvdata->ivgen_handle;
(iv_out_size != CTR_RFC3686_IV_SIZE)) {
return -EINVAL;
}
- if ( (iv_out_dma_len + 1) > SSI_IVPOOL_SEQ_LEN) {
+ if ((iv_out_dma_len + 1) > SSI_IVPOOL_SEQ_LEN) {
/* The sequence will be longer than allowed */
return -EINVAL;
}
//check that number of generated IV is limited to max dma address iv buffer size
- if ( iv_out_dma_len > SSI_MAX_IVGEN_DMA_ADDRESSES) {
+ if (iv_out_dma_len > SSI_MAX_IVGEN_DMA_ADDRESSES) {
/* The sequence will be longer than allowed */
return -EINVAL;
}
for (t = 0; t < iv_out_dma_len; t++) {
/* Acquire IV from pool */
- HW_DESC_INIT(&iv_seq[idx]);
- HW_DESC_SET_DIN_SRAM(&iv_seq[idx],
- ivgen_ctx->pool + ivgen_ctx->next_iv_ofs,
- iv_out_size);
- HW_DESC_SET_DOUT_DLLI(&iv_seq[idx], iv_out_dma[t],
- iv_out_size, NS_BIT, 0);
- HW_DESC_SET_FLOW_MODE(&iv_seq[idx], BYPASS);
+ hw_desc_init(&iv_seq[idx]);
+ set_din_sram(&iv_seq[idx], (ivgen_ctx->pool +
+ ivgen_ctx->next_iv_ofs),
+ iv_out_size);
+ set_dout_dlli(&iv_seq[idx], iv_out_dma[t], iv_out_size,
+ NS_BIT, 0);
+ set_flow_mode(&iv_seq[idx], BYPASS);
idx++;
}
/* Bypass operation is proceeded by crypto sequence, hence must
- * assure bypass-write-transaction by a memory barrier */
- HW_DESC_INIT(&iv_seq[idx]);
- HW_DESC_SET_DIN_NO_DMA(&iv_seq[idx], 0, 0xfffff0);
- HW_DESC_SET_DOUT_NO_DMA(&iv_seq[idx], 0, 0, 1);
+ * assure bypass-write-transaction by a memory barrier
+ */
+ hw_desc_init(&iv_seq[idx]);
+ set_din_no_dma(&iv_seq[idx], 0, 0xfffff0);
+ set_dout_no_dma(&iv_seq[idx], 0, 0, 1);
idx++;
*iv_seq_len = idx; /* update seq length */
return 0;
}
-
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
#include "cc_hw_queue_defs.h"
-
#define SSI_IVPOOL_SEQ_LEN 8
/*!
- * Allocates iv-pool and maps resources.
- * This function generates the first IV pool.
- *
+ * Allocates iv-pool and maps resources.
+ * This function generates the first IV pool.
+ *
* \param drvdata Driver's private context
- *
+ *
* \return int Zero for success, negative value otherwise.
*/
int ssi_ivgen_init(struct ssi_drvdata *drvdata);
/*!
* Free iv-pool and ivgen context.
- *
- * \param drvdata
+ *
+ * \param drvdata
*/
void ssi_ivgen_fini(struct ssi_drvdata *drvdata);
/*!
- * Generates the initial pool in SRAM.
- * This function should be invoked when resuming DX driver.
- *
- * \param drvdata
- *
+ * Generates the initial pool in SRAM.
+ * This function should be invoked when resuming DX driver.
+ *
+ * \param drvdata
+ *
* \return int Zero for success, negative value otherwise.
*/
int ssi_ivgen_init_sram_pool(struct ssi_drvdata *drvdata);
/*!
* Acquires 16 Bytes IV from the iv-pool
- *
+ *
* \param drvdata Driver private context
* \param iv_out_dma Array of physical IV out addresses
* \param iv_out_dma_len Length of iv_out_dma array (additional elements of iv_out_dma array are ignore)
- * \param iv_out_size May be 8 or 16 bytes long
+ * \param iv_out_size May be 8 or 16 bytes long
* \param iv_seq IN/OUT array to the descriptors sequence
- * \param iv_seq_len IN/OUT pointer to the sequence length
- *
- * \return int Zero for success, negative value otherwise.
+ * \param iv_seq_len IN/OUT pointer to the sequence length
+ *
+ * \return int Zero for success, negative value otherwise.
*/
int ssi_ivgen_getiv(
struct ssi_drvdata *drvdata,
dma_addr_t iv_out_dma[],
unsigned int iv_out_dma_len,
unsigned int iv_out_size,
- HwDesc_s iv_seq[],
+ struct cc_hw_desc iv_seq[],
unsigned int *iv_seq_len);
#endif /*__SSI_IVGEN_H__*/
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
-
#include "ssi_config.h"
#include <linux/kernel.h>
#include <linux/platform_device.h>
#include "ssi_ivgen.h"
#include "ssi_hash.h"
#include "ssi_pm.h"
-#include "ssi_pm_ext.h"
-
-#if defined (CONFIG_PM_RUNTIME) || defined (CONFIG_PM_SLEEP)
+#if defined(CONFIG_PM_RUNTIME) || defined(CONFIG_PM_SLEEP)
#define POWER_DOWN_ENABLE 0x01
#define POWER_DOWN_DISABLE 0x00
-
int ssi_power_mgr_runtime_suspend(struct device *dev)
{
struct ssi_drvdata *drvdata =
return rc;
}
fini_cc_regs(drvdata);
-
- /* Specific HW suspend code */
- ssi_pm_ext_hw_suspend(dev);
+ cc_clk_off(drvdata);
return 0;
}
SSI_LOG_DEBUG("ssi_power_mgr_runtime_resume , unset HOST_POWER_DOWN_EN\n");
WRITE_REGISTER(drvdata->cc_base + CC_REG_OFFSET(HOST_RGF, HOST_POWER_DOWN_EN), POWER_DOWN_DISABLE);
- /* Specific HW resume code */
- ssi_pm_ext_hw_resume(dev);
+
+ rc = cc_clk_on(drvdata);
+ if (rc) {
+ SSI_LOG_ERR("failed getting clock back on. We're toast.\n");
+ return rc;
+ }
rc = init_cc_regs(drvdata, false);
- if (rc !=0) {
- SSI_LOG_ERR("init_cc_regs (%x)\n",rc);
+ if (rc != 0) {
+ SSI_LOG_ERR("init_cc_regs (%x)\n", rc);
return rc;
}
rc = ssi_request_mgr_runtime_resume_queue(drvdata);
- if (rc !=0) {
- SSI_LOG_ERR("ssi_request_mgr_runtime_resume_queue (%x)\n",rc);
+ if (rc != 0) {
+ SSI_LOG_ERR("ssi_request_mgr_runtime_resume_queue (%x)\n", rc);
return rc;
}
/* must be after the queue resuming as it uses the HW queue*/
ssi_hash_init_sram_digest_consts(drvdata);
-
+
ssi_ivgen_init_sram_pool(drvdata);
return 0;
}
(struct ssi_drvdata *)dev_get_drvdata(dev))) {
pm_runtime_mark_last_busy(dev);
rc = pm_runtime_put_autosuspend(dev);
- }
- else {
+ } else {
/* Something wrong happens*/
BUG();
}
return rc;
-
}
#endif
-
-
int ssi_power_mgr_init(struct ssi_drvdata *drvdata)
{
int rc = 0;
-#if defined (CONFIG_PM_RUNTIME) || defined (CONFIG_PM_SLEEP)
+#if defined(CONFIG_PM_RUNTIME) || defined(CONFIG_PM_SLEEP)
struct platform_device *plat_dev = drvdata->plat_dev;
/* must be before the enabling to avoid resdundent suspending */
- pm_runtime_set_autosuspend_delay(&plat_dev->dev,SSI_SUSPEND_TIMEOUT);
+ pm_runtime_set_autosuspend_delay(&plat_dev->dev, SSI_SUSPEND_TIMEOUT);
pm_runtime_use_autosuspend(&plat_dev->dev);
/* activate the PM module */
rc = pm_runtime_set_active(&plat_dev->dev);
void ssi_power_mgr_fini(struct ssi_drvdata *drvdata)
{
-#if defined (CONFIG_PM_RUNTIME) || defined (CONFIG_PM_SLEEP)
+#if defined(CONFIG_PM_RUNTIME) || defined(CONFIG_PM_SLEEP)
struct platform_device *plat_dev = drvdata->plat_dev;
pm_runtime_disable(&plat_dev->dev);
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
/* \file ssi_pm.h
- */
+ */
#ifndef __SSI_POWER_MGR_H__
#define __SSI_POWER_MGR_H__
-
#include "ssi_config.h"
#include "ssi_driver.h"
-
#define SSI_SUSPEND_TIMEOUT 3000
-
int ssi_power_mgr_init(struct ssi_drvdata *drvdata);
void ssi_power_mgr_fini(struct ssi_drvdata *drvdata);
-#if defined (CONFIG_PM_RUNTIME) || defined (CONFIG_PM_SLEEP)
+#if defined(CONFIG_PM_RUNTIME) || defined(CONFIG_PM_SLEEP)
int ssi_power_mgr_runtime_suspend(struct device *dev);
int ssi_power_mgr_runtime_resume(struct device *dev);
+++ /dev/null
-/*
- * Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, see <http://www.gnu.org/licenses/>.
- */
-
-
-#include "ssi_config.h"
-#include <linux/kernel.h>
-#include <linux/platform_device.h>
-#include <linux/interrupt.h>
-#include <crypto/ctr.h>
-#include <linux/pm_runtime.h>
-#include "ssi_driver.h"
-#include "ssi_sram_mgr.h"
-#include "ssi_pm_ext.h"
-
-/*
-This function should suspend the HW (if possiable), It should be implemented by
-the driver user.
-The reference code clears the internal SRAM to imitate lose of state.
-*/
-void ssi_pm_ext_hw_suspend(struct device *dev)
-{
- struct ssi_drvdata *drvdata =
- (struct ssi_drvdata *)dev_get_drvdata(dev);
- unsigned int val;
- void __iomem *cc_base = drvdata->cc_base;
- unsigned int sram_addr = 0;
-
- CC_HAL_WRITE_REGISTER(CC_REG_OFFSET(HOST_RGF, SRAM_ADDR), sram_addr);
-
- for (;sram_addr < SSI_CC_SRAM_SIZE ; sram_addr+=4) {
- CC_HAL_WRITE_REGISTER(CC_REG_OFFSET(HOST_RGF, SRAM_DATA), 0x0);
-
- do {
- val = CC_HAL_READ_REGISTER(CC_REG_OFFSET(HOST_RGF, SRAM_DATA_READY));
- } while (!(val &0x1));
- }
-}
-
-/*
-This function should resume the HW (if possiable).It should be implemented by
-the driver user.
-*/
-void ssi_pm_ext_hw_resume(struct device *dev)
-{
- return;
-}
-
+++ /dev/null
-/*
- * Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, see <http://www.gnu.org/licenses/>.
- */
-
-/* \file ssi_pm_ext.h
- */
-
-#ifndef __PM_EXT_H__
-#define __PM_EXT_H__
-
-
-#include "ssi_config.h"
-#include "ssi_driver.h"
-
-void ssi_pm_ext_hw_suspend(struct device *dev);
-
-void ssi_pm_ext_hw_resume(struct device *dev);
-
-
-#endif /*__POWER_MGR_H__*/
-
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
#define SSI_MAX_POLL_ITER 10
-#define AXIM_MON_BASE_OFFSET CC_REG_OFFSET(CRY_KERNEL, AXIM_MON_COMP)
-
-#ifdef CC_CYCLE_COUNT
-
-#define MONITOR_CNTR_BIT 0
-
-/**
- * Monitor descriptor.
- * Used to measure CC performance.
- */
-#define INIT_CC_MONITOR_DESC(desc_p) \
-do { \
- HW_DESC_INIT(desc_p); \
- HW_DESC_SET_DIN_MONITOR_CNTR(desc_p); \
-} while (0)
-
-/**
- * Try adding monitor descriptor BEFORE enqueuing sequence.
- */
-#define CC_CYCLE_DESC_HEAD(cc_base_addr, desc_p, lock_p, is_monitored_p) \
-do { \
- if (!test_and_set_bit(MONITOR_CNTR_BIT, (lock_p))) { \
- enqueue_seq((cc_base_addr), (desc_p), 1); \
- *(is_monitored_p) = true; \
- } else { \
- *(is_monitored_p) = false; \
- } \
-} while (0)
-
-/**
- * If CC_CYCLE_DESC_HEAD was successfully added:
- * 1. Add memory barrier descriptor to ensure last AXI transaction.
- * 2. Add monitor descriptor to sequence tail AFTER enqueuing sequence.
- */
-#define CC_CYCLE_DESC_TAIL(cc_base_addr, desc_p, is_monitored) \
-do { \
- if ((is_monitored) == true) { \
- HwDesc_s barrier_desc; \
- HW_DESC_INIT(&barrier_desc); \
- HW_DESC_SET_DIN_NO_DMA(&barrier_desc, 0, 0xfffff0); \
- HW_DESC_SET_DOUT_NO_DMA(&barrier_desc, 0, 0, 1); \
- enqueue_seq((cc_base_addr), &barrier_desc, 1); \
- enqueue_seq((cc_base_addr), (desc_p), 1); \
- } \
-} while (0)
-
-/**
- * Try reading CC monitor counter value upon sequence complete.
- * Can only succeed if the lock_p is taken by the owner of the given request.
- */
-#define END_CC_MONITOR_COUNT(cc_base_addr, stat_op_type, stat_phase, monitor_null_cycles, lock_p, is_monitored) \
-do { \
- uint32_t elapsed_cycles; \
- if ((is_monitored) == true) { \
- elapsed_cycles = READ_REGISTER((cc_base_addr) + CC_REG_OFFSET(CRY_KERNEL, DSCRPTR_MEASURE_CNTR)); \
- clear_bit(MONITOR_CNTR_BIT, (lock_p)); \
- if (elapsed_cycles > 0) \
- update_cc_stat(stat_op_type, stat_phase, (elapsed_cycles - monitor_null_cycles)); \
- } \
-} while (0)
-
-#else /*CC_CYCLE_COUNT*/
-
-#define INIT_CC_MONITOR_DESC(desc_p) do { } while (0)
-#define CC_CYCLE_DESC_HEAD(cc_base_addr, desc_p, lock_p, is_monitored_p) do { } while (0)
-#define CC_CYCLE_DESC_TAIL(cc_base_addr, desc_p, is_monitored) do { } while (0)
-#define END_CC_MONITOR_COUNT(cc_base_addr, stat_op_type, stat_phase, monitor_null_cycles, lock_p, is_monitored) do { } while (0)
-#endif /*CC_CYCLE_COUNT*/
-
-
struct ssi_request_mgr_handle {
/* Request manager resources */
unsigned int hw_queue_size; /* HW capability */
unsigned int min_free_hw_slots;
unsigned int max_used_sw_slots;
struct ssi_crypto_req req_queue[MAX_REQUEST_QUEUE_SIZE];
- uint32_t req_queue_head;
- uint32_t req_queue_tail;
- uint32_t axi_completed;
- uint32_t q_free_slots;
+ u32 req_queue_head;
+ u32 req_queue_tail;
+ u32 axi_completed;
+ u32 q_free_slots;
spinlock_t hw_lock;
- HwDesc_s compl_desc;
- uint8_t *dummy_comp_buff;
+ struct cc_hw_desc compl_desc;
+ u8 *dummy_comp_buff;
dma_addr_t dummy_comp_buff_dma;
- HwDesc_s monitor_desc;
+ struct cc_hw_desc monitor_desc;
+
volatile unsigned long monitor_lock;
#ifdef COMP_IN_WQ
struct workqueue_struct *workq;
#else
struct tasklet_struct comptask;
#endif
-#if defined (CONFIG_PM_RUNTIME) || defined (CONFIG_PM_SLEEP)
+#if defined(CONFIG_PM_RUNTIME) || defined(CONFIG_PM_SLEEP)
bool is_runtime_suspended;
#endif
};
{
struct ssi_request_mgr_handle *req_mgr_h = drvdata->request_mgr_handle;
- if (req_mgr_h == NULL)
+ if (!req_mgr_h)
return; /* Not allocated */
if (req_mgr_h->dummy_comp_buff_dma != 0) {
- SSI_RESTORE_DMA_ADDR_TO_48BIT(req_mgr_h->dummy_comp_buff_dma);
dma_free_coherent(&drvdata->plat_dev->dev,
- sizeof(uint32_t), req_mgr_h->dummy_comp_buff,
+ sizeof(u32), req_mgr_h->dummy_comp_buff,
req_mgr_h->dummy_comp_buff_dma);
}
SSI_LOG_DEBUG("max_used_hw_slots=%d\n", (req_mgr_h->hw_queue_size -
- req_mgr_h->min_free_hw_slots) );
+ req_mgr_h->min_free_hw_slots));
SSI_LOG_DEBUG("max_used_sw_slots=%d\n", req_mgr_h->max_used_sw_slots);
#ifdef COMP_IN_WQ
int request_mgr_init(struct ssi_drvdata *drvdata)
{
-#ifdef CC_CYCLE_COUNT
- HwDesc_s monitor_desc[2];
- struct ssi_crypto_req monitor_req = {0};
-#endif
struct ssi_request_mgr_handle *req_mgr_h;
int rc = 0;
- req_mgr_h = kzalloc(sizeof(struct ssi_request_mgr_handle),GFP_KERNEL);
- if (req_mgr_h == NULL) {
+ req_mgr_h = kzalloc(sizeof(struct ssi_request_mgr_handle), GFP_KERNEL);
+ if (!req_mgr_h) {
rc = -ENOMEM;
goto req_mgr_init_err;
}
#ifdef COMP_IN_WQ
SSI_LOG_DEBUG("Initializing completion workqueue\n");
req_mgr_h->workq = create_singlethread_workqueue("arm_cc7x_wq");
- if (unlikely(req_mgr_h->workq == NULL)) {
+ if (unlikely(!req_mgr_h->workq)) {
SSI_LOG_ERR("Failed creating work queue\n");
rc = -ENOMEM;
goto req_mgr_init_err;
req_mgr_h->min_free_hw_slots = req_mgr_h->hw_queue_size;
req_mgr_h->max_used_sw_slots = 0;
-
/* Allocate DMA word for "dummy" completion descriptor use */
req_mgr_h->dummy_comp_buff = dma_alloc_coherent(&drvdata->plat_dev->dev,
- sizeof(uint32_t), &req_mgr_h->dummy_comp_buff_dma, GFP_KERNEL);
+ sizeof(u32), &req_mgr_h->dummy_comp_buff_dma, GFP_KERNEL);
if (!req_mgr_h->dummy_comp_buff) {
SSI_LOG_ERR("Not enough memory to allocate DMA (%zu) dropped "
- "buffer\n", sizeof(uint32_t));
+ "buffer\n", sizeof(u32));
rc = -ENOMEM;
goto req_mgr_init_err;
}
- SSI_UPDATE_DMA_ADDR_TO_48BIT(req_mgr_h->dummy_comp_buff_dma,
- sizeof(uint32_t));
/* Init. "dummy" completion descriptor */
- HW_DESC_INIT(&req_mgr_h->compl_desc);
- HW_DESC_SET_DIN_CONST(&req_mgr_h->compl_desc, 0, sizeof(uint32_t));
- HW_DESC_SET_DOUT_DLLI(&req_mgr_h->compl_desc,
- req_mgr_h->dummy_comp_buff_dma,
- sizeof(uint32_t), NS_BIT, 1);
- HW_DESC_SET_FLOW_MODE(&req_mgr_h->compl_desc, BYPASS);
- HW_DESC_SET_QUEUE_LAST_IND(&req_mgr_h->compl_desc);
-
-#ifdef CC_CYCLE_COUNT
- /* For CC-HW cycle performance trace */
- INIT_CC_MONITOR_DESC(&req_mgr_h->monitor_desc);
- set_bit(MONITOR_CNTR_BIT, &req_mgr_h->monitor_lock);
- monitor_desc[0] = req_mgr_h->monitor_desc;
- monitor_desc[1] = req_mgr_h->monitor_desc;
-
- rc = send_request(drvdata, &monitor_req, monitor_desc, 2, 0);
- if (unlikely(rc != 0))
- goto req_mgr_init_err;
-
- drvdata->monitor_null_cycles = READ_REGISTER(drvdata->cc_base +
- CC_REG_OFFSET(CRY_KERNEL, DSCRPTR_MEASURE_CNTR));
- SSI_LOG_ERR("Calibration time=0x%08x\n", drvdata->monitor_null_cycles);
-
- clear_bit(MONITOR_CNTR_BIT, &req_mgr_h->monitor_lock);
-#endif
+ hw_desc_init(&req_mgr_h->compl_desc);
+ set_din_const(&req_mgr_h->compl_desc, 0, sizeof(u32));
+ set_dout_dlli(&req_mgr_h->compl_desc, req_mgr_h->dummy_comp_buff_dma,
+ sizeof(u32), NS_BIT, 1);
+ set_flow_mode(&req_mgr_h->compl_desc, BYPASS);
+ set_queue_last_ind(&req_mgr_h->compl_desc);
return 0;
static inline void enqueue_seq(
void __iomem *cc_base,
- HwDesc_s seq[], unsigned int seq_len)
+ struct cc_hw_desc seq[], unsigned int seq_len)
{
int i;
for (i = 0; i < seq_len; i++) {
- writel_relaxed(seq[i].word[0], (volatile void __iomem *)(cc_base+CC_REG_OFFSET(CRY_KERNEL, DSCRPTR_QUEUE_WORD0)));
- writel_relaxed(seq[i].word[1], (volatile void __iomem *)(cc_base+CC_REG_OFFSET(CRY_KERNEL, DSCRPTR_QUEUE_WORD0)));
- writel_relaxed(seq[i].word[2], (volatile void __iomem *)(cc_base+CC_REG_OFFSET(CRY_KERNEL, DSCRPTR_QUEUE_WORD0)));
- writel_relaxed(seq[i].word[3], (volatile void __iomem *)(cc_base+CC_REG_OFFSET(CRY_KERNEL, DSCRPTR_QUEUE_WORD0)));
- writel_relaxed(seq[i].word[4], (volatile void __iomem *)(cc_base+CC_REG_OFFSET(CRY_KERNEL, DSCRPTR_QUEUE_WORD0)));
+ writel_relaxed(seq[i].word[0], (volatile void __iomem *)(cc_base + CC_REG_OFFSET(CRY_KERNEL, DSCRPTR_QUEUE_WORD0)));
+ writel_relaxed(seq[i].word[1], (volatile void __iomem *)(cc_base + CC_REG_OFFSET(CRY_KERNEL, DSCRPTR_QUEUE_WORD0)));
+ writel_relaxed(seq[i].word[2], (volatile void __iomem *)(cc_base + CC_REG_OFFSET(CRY_KERNEL, DSCRPTR_QUEUE_WORD0)));
+ writel_relaxed(seq[i].word[3], (volatile void __iomem *)(cc_base + CC_REG_OFFSET(CRY_KERNEL, DSCRPTR_QUEUE_WORD0)));
+ writel_relaxed(seq[i].word[4], (volatile void __iomem *)(cc_base + CC_REG_OFFSET(CRY_KERNEL, DSCRPTR_QUEUE_WORD0)));
wmb();
- writel_relaxed(seq[i].word[5], (volatile void __iomem *)(cc_base+CC_REG_OFFSET(CRY_KERNEL, DSCRPTR_QUEUE_WORD0)));
+ writel_relaxed(seq[i].word[5], (volatile void __iomem *)(cc_base + CC_REG_OFFSET(CRY_KERNEL, DSCRPTR_QUEUE_WORD0)));
#ifdef DX_DUMP_DESCS
SSI_LOG_DEBUG("desc[%02d]: 0x%08X 0x%08X 0x%08X 0x%08X 0x%08X 0x%08X\n", i,
seq[i].word[0], seq[i].word[1], seq[i].word[2], seq[i].word[3], seq[i].word[4], seq[i].word[5]);
}
/*!
- * Completion will take place if and only if user requested completion
- * by setting "is_dout = 0" in send_request().
- *
- * \param dev
+ * Completion will take place if and only if user requested completion
+ * by setting "is_dout = 0" in send_request().
+ *
+ * \param dev
* \param dx_compl_h The completion event to signal
*/
static void request_mgr_complete(struct device *dev, void *dx_compl_h, void __iomem *cc_base)
{
struct completion *this_compl = dx_compl_h;
+
complete(this_compl);
}
-
static inline int request_mgr_queues_status_check(
struct ssi_request_mgr_handle *req_mgr_h,
void __iomem *cc_base,
unsigned int total_seq_len)
{
unsigned long poll_queue;
-
- /* SW queue is checked only once as it will not
- be chaned during the poll becasue the spinlock_bh
- is held by the thread */
+
+ /* SW queue is checked only once as it will not
+ * be chaned during the poll becasue the spinlock_bh
+ * is held by the thread
+ */
if (unlikely(((req_mgr_h->req_queue_head + 1) &
- (MAX_REQUEST_QUEUE_SIZE - 1)) ==
+ (MAX_REQUEST_QUEUE_SIZE - 1)) ==
req_mgr_h->req_queue_tail)) {
- SSI_LOG_ERR("SW FIFO is full. req_queue_head=%d sw_fifo_len=%d\n",
+ SSI_LOG_ERR("SW FIFO is full. req_queue_head=%d sw_fifo_len=%d\n",
req_mgr_h->req_queue_head, MAX_REQUEST_QUEUE_SIZE);
return -EBUSY;
}
- if ((likely(req_mgr_h->q_free_slots >= total_seq_len)) ) {
+ if ((likely(req_mgr_h->q_free_slots >= total_seq_len)))
return 0;
- }
+
/* Wait for space in HW queue. Poll constant num of iterations. */
- for (poll_queue =0; poll_queue < SSI_MAX_POLL_ITER ; poll_queue ++) {
- req_mgr_h->q_free_slots =
+ for (poll_queue = 0; poll_queue < SSI_MAX_POLL_ITER ; poll_queue++) {
+ req_mgr_h->q_free_slots =
CC_HAL_READ_REGISTER(
CC_REG_OFFSET(CRY_KERNEL,
DSCRPTR_QUEUE_CONTENT));
- if (unlikely(req_mgr_h->q_free_slots <
+ if (unlikely(req_mgr_h->q_free_slots <
req_mgr_h->min_free_hw_slots)) {
req_mgr_h->min_free_hw_slots = req_mgr_h->q_free_slots;
}
- if (likely (req_mgr_h->q_free_slots >= total_seq_len)) {
+ if (likely(req_mgr_h->q_free_slots >= total_seq_len)) {
/* If there is enough place return */
return 0;
}
- SSI_LOG_DEBUG("HW FIFO is full. q_free_slots=%d total_seq_len=%d\n",
+ SSI_LOG_DEBUG("HW FIFO is full. q_free_slots=%d total_seq_len=%d\n",
req_mgr_h->q_free_slots, total_seq_len);
}
/* No room in the HW queue try again later */
SSI_LOG_DEBUG("HW FIFO full, timeout. req_queue_head=%d "
- "sw_fifo_len=%d q_free_slots=%d total_seq_len=%d\n",
+ "sw_fifo_len=%d q_free_slots=%d total_seq_len=%d\n",
req_mgr_h->req_queue_head,
MAX_REQUEST_QUEUE_SIZE,
req_mgr_h->q_free_slots,
/*!
* Enqueue caller request to crypto hardware.
- *
- * \param drvdata
+ *
+ * \param drvdata
* \param ssi_req The request to enqueue
* \param desc The crypto sequence
* \param len The crypto sequence length
- * \param is_dout If "true": completion is handled by the caller
- * If "false": this function adds a dummy descriptor completion
- * and waits upon completion signal.
- *
+ * \param is_dout If "true": completion is handled by the caller
+ * If "false": this function adds a dummy descriptor completion
+ * and waits upon completion signal.
+ *
* \return int Returns -EINPROGRESS if "is_dout=true"; "0" if "is_dout=false"
*/
int send_request(
struct ssi_drvdata *drvdata, struct ssi_crypto_req *ssi_req,
- HwDesc_s *desc, unsigned int len, bool is_dout)
+ struct cc_hw_desc *desc, unsigned int len, bool is_dout)
{
void __iomem *cc_base = drvdata->cc_base;
struct ssi_request_mgr_handle *req_mgr_h = drvdata->request_mgr_handle;
unsigned int used_sw_slots;
unsigned int iv_seq_len = 0;
unsigned int total_seq_len = len; /*initial sequence length*/
- HwDesc_s iv_seq[SSI_IVPOOL_SEQ_LEN];
+ struct cc_hw_desc iv_seq[SSI_IVPOOL_SEQ_LEN];
int rc;
unsigned int max_required_seq_len = (total_seq_len +
((ssi_req->ivgen_dma_addr_len == 0) ? 0 :
- SSI_IVPOOL_SEQ_LEN ) +
- ((is_dout == 0 )? 1 : 0));
- DECL_CYCLE_COUNT_RESOURCES;
+ SSI_IVPOOL_SEQ_LEN) +
+ ((is_dout == 0) ? 1 : 0));
-#if defined (CONFIG_PM_RUNTIME) || defined (CONFIG_PM_SLEEP)
+#if defined(CONFIG_PM_RUNTIME) || defined(CONFIG_PM_SLEEP)
rc = ssi_power_mgr_runtime_get(&drvdata->plat_dev->dev);
if (rc != 0) {
- SSI_LOG_ERR("ssi_power_mgr_runtime_get returned %x\n",rc);
+ SSI_LOG_ERR("ssi_power_mgr_runtime_get returned %x\n", rc);
return rc;
}
#endif
spin_lock_bh(&req_mgr_h->hw_lock);
/* Check if there is enough place in the SW/HW queues
- in case iv gen add the max size and in case of no dout add 1
- for the internal completion descriptor */
+ * in case iv gen add the max size and in case of no dout add 1
+ * for the internal completion descriptor
+ */
rc = request_mgr_queues_status_check(req_mgr_h,
cc_base,
max_required_seq_len);
- if (likely(rc == 0 ))
+ if (likely(rc == 0))
/* There is enough place in the queue */
break;
/* something wrong release the spinlock*/
spin_unlock_bh(&req_mgr_h->hw_lock);
if (rc != -EAGAIN) {
- /* Any error other than HW queue full
- (SW queue is full) */
-#if defined (CONFIG_PM_RUNTIME) || defined (CONFIG_PM_SLEEP)
+ /* Any error other than HW queue full
+ * (SW queue is full)
+ */
+#if defined(CONFIG_PM_RUNTIME) || defined(CONFIG_PM_SLEEP)
ssi_power_mgr_runtime_put_suspend(&drvdata->plat_dev->dev);
#endif
return rc;
} while (1);
/* Additional completion descriptor is needed incase caller did not
- enabled any DLLI/MLLI DOUT bit in the given sequence */
+ * enabled any DLLI/MLLI DOUT bit in the given sequence
+ */
if (!is_dout) {
init_completion(&ssi_req->seq_compl);
ssi_req->user_cb = request_mgr_complete;
if (unlikely(rc != 0)) {
SSI_LOG_ERR("Failed to generate IV (rc=%d)\n", rc);
spin_unlock_bh(&req_mgr_h->hw_lock);
-#if defined (CONFIG_PM_RUNTIME) || defined (CONFIG_PM_SLEEP)
+#if defined(CONFIG_PM_RUNTIME) || defined(CONFIG_PM_SLEEP)
ssi_power_mgr_runtime_put_suspend(&drvdata->plat_dev->dev);
#endif
return rc;
total_seq_len += iv_seq_len;
}
-
- used_sw_slots = ((req_mgr_h->req_queue_head - req_mgr_h->req_queue_tail) & (MAX_REQUEST_QUEUE_SIZE-1));
- if (unlikely(used_sw_slots > req_mgr_h->max_used_sw_slots)) {
+
+ used_sw_slots = ((req_mgr_h->req_queue_head - req_mgr_h->req_queue_tail) & (MAX_REQUEST_QUEUE_SIZE - 1));
+ if (unlikely(used_sw_slots > req_mgr_h->max_used_sw_slots))
req_mgr_h->max_used_sw_slots = used_sw_slots;
- }
-
- CC_CYCLE_DESC_HEAD(cc_base, &req_mgr_h->monitor_desc,
- &req_mgr_h->monitor_lock, &ssi_req->is_monitored_p);
/* Enqueue request - must be locked with HW lock*/
req_mgr_h->req_queue[req_mgr_h->req_queue_head] = *ssi_req;
- START_CYCLE_COUNT_AT(req_mgr_h->req_queue[req_mgr_h->req_queue_head].submit_cycle);
req_mgr_h->req_queue_head = (req_mgr_h->req_queue_head + 1) & (MAX_REQUEST_QUEUE_SIZE - 1);
/* TODO: Use circ_buf.h ? */
#endif
/* STAT_PHASE_4: Push sequence */
- START_CYCLE_COUNT();
enqueue_seq(cc_base, iv_seq, iv_seq_len);
enqueue_seq(cc_base, desc, len);
enqueue_seq(cc_base, &req_mgr_h->compl_desc, (is_dout ? 0 : 1));
- END_CYCLE_COUNT(ssi_req->op_type, STAT_PHASE_4);
-
- CC_CYCLE_DESC_TAIL(cc_base, &req_mgr_h->monitor_desc, ssi_req->is_monitored_p);
if (unlikely(req_mgr_h->q_free_slots < total_seq_len)) {
/*This means that there was a problem with the resume*/
if (!is_dout) {
/* Wait upon sequence completion.
- * Return "0" -Operation done successfully. */
- return wait_for_completion_interruptible(&ssi_req->seq_compl);
+ * Return "0" -Operation done successfully.
+ */
+ wait_for_completion(&ssi_req->seq_compl);
+ return 0;
} else {
/* Operation still in process */
return -EINPROGRESS;
}
}
-
/*!
* Enqueue caller request to crypto hardware during init process.
* assume this function is not called in middle of a flow,
* since we set QUEUE_LAST_IND flag in the last descriptor.
- *
- * \param drvdata
+ *
+ * \param drvdata
* \param desc The crypto sequence
* \param len The crypto sequence length
- *
+ *
* \return int Returns "0" upon success
*/
int send_request_init(
- struct ssi_drvdata *drvdata, HwDesc_s *desc, unsigned int len)
+ struct ssi_drvdata *drvdata, struct cc_hw_desc *desc, unsigned int len)
{
void __iomem *cc_base = drvdata->cc_base;
struct ssi_request_mgr_handle *req_mgr_h = drvdata->request_mgr_handle;
/* Wait for space in HW and SW FIFO. Poll for as much as FIFO_TIMEOUT. */
rc = request_mgr_queues_status_check(req_mgr_h, cc_base, total_seq_len);
- if (unlikely(rc != 0 )) {
+ if (unlikely(rc != 0))
return rc;
- }
- HW_DESC_SET_QUEUE_LAST_IND(&desc[len-1]);
+
+ set_queue_last_ind(&desc[(len - 1)]);
enqueue_seq(cc_base, desc, len);
return 0;
}
-
void complete_request(struct ssi_drvdata *drvdata)
{
- struct ssi_request_mgr_handle *request_mgr_handle =
+ struct ssi_request_mgr_handle *request_mgr_handle =
drvdata->request_mgr_handle;
#ifdef COMP_IN_WQ
queue_delayed_work(request_mgr_handle->workq, &request_mgr_handle->compwork, 0);
{
struct ssi_crypto_req *ssi_req;
struct platform_device *plat_dev = drvdata->plat_dev;
- struct ssi_request_mgr_handle * request_mgr_handle =
+ struct ssi_request_mgr_handle *request_mgr_handle =
drvdata->request_mgr_handle;
-#if defined (CONFIG_PM_RUNTIME) || defined (CONFIG_PM_SLEEP)
+#if defined(CONFIG_PM_RUNTIME) || defined(CONFIG_PM_SLEEP)
int rc = 0;
#endif
- DECL_CYCLE_COUNT_RESOURCES;
- while(request_mgr_handle->axi_completed) {
+ while (request_mgr_handle->axi_completed) {
request_mgr_handle->axi_completed--;
/* Dequeue request */
}
ssi_req = &request_mgr_handle->req_queue[request_mgr_handle->req_queue_tail];
- END_CYCLE_COUNT_AT(ssi_req->submit_cycle, ssi_req->op_type, STAT_PHASE_5); /* Seq. Comp. */
- END_CC_MONITOR_COUNT(drvdata->cc_base, ssi_req->op_type, STAT_PHASE_6,
- drvdata->monitor_null_cycles, &request_mgr_handle->monitor_lock, ssi_req->is_monitored_p);
#ifdef FLUSH_CACHE_ALL
flush_cache_all();
#ifdef COMPLETION_DELAY
/* Delay */
{
- uint32_t axi_err;
+ u32 axi_err;
int i;
+
SSI_LOG_INFO("Delay\n");
- for (i=0;i<1000000;i++) {
+ for (i = 0; i < 1000000; i++)
axi_err = READ_REGISTER(drvdata->cc_base + CC_REG_OFFSET(CRY_KERNEL, AXIM_MON_ERR));
- }
}
#endif /* COMPLETION_DELAY */
- if (likely(ssi_req->user_cb != NULL)) {
- START_CYCLE_COUNT();
+ if (likely(ssi_req->user_cb))
ssi_req->user_cb(&plat_dev->dev, ssi_req->user_arg, drvdata->cc_base);
- END_CYCLE_COUNT(STAT_OP_TYPE_GENERIC, STAT_PHASE_3);
- }
request_mgr_handle->req_queue_tail = (request_mgr_handle->req_queue_tail + 1) & (MAX_REQUEST_QUEUE_SIZE - 1);
SSI_LOG_DEBUG("Dequeue request tail=%u\n", request_mgr_handle->req_queue_tail);
SSI_LOG_DEBUG("Request completed. axi_completed=%d\n", request_mgr_handle->axi_completed);
-#if defined (CONFIG_PM_RUNTIME) || defined (CONFIG_PM_SLEEP)
+#if defined(CONFIG_PM_RUNTIME) || defined(CONFIG_PM_SLEEP)
rc = ssi_power_mgr_runtime_put_suspend(&plat_dev->dev);
- if (rc != 0) {
- SSI_LOG_ERR("Failed to set runtime suspension %d\n",rc);
- }
+ if (rc != 0)
+ SSI_LOG_ERR("Failed to set runtime suspension %d\n", rc);
#endif
}
}
+static inline u32 cc_axi_comp_count(void __iomem *cc_base)
+{
+ /* The CC_HAL_READ_REGISTER macro implictly requires and uses
+ * a base MMIO register address variable named cc_base.
+ */
+ return FIELD_GET(AXIM_MON_COMP_VALUE,
+ CC_HAL_READ_REGISTER(AXIM_MON_BASE_OFFSET));
+}
+
/* Deferred service handler, run as interrupt-fired tasklet */
static void comp_handler(unsigned long devarg)
{
struct ssi_drvdata *drvdata = (struct ssi_drvdata *)devarg;
void __iomem *cc_base = drvdata->cc_base;
- struct ssi_request_mgr_handle * request_mgr_handle =
+ struct ssi_request_mgr_handle *request_mgr_handle =
drvdata->request_mgr_handle;
- uint32_t irq;
-
- DECL_CYCLE_COUNT_RESOURCES;
-
- START_CYCLE_COUNT();
+ u32 irq;
irq = (drvdata->irq & SSI_COMP_IRQ_MASK);
if (irq & SSI_COMP_IRQ_MASK) {
/* To avoid the interrupt from firing as we unmask it, we clear it now */
CC_HAL_WRITE_REGISTER(CC_REG_OFFSET(HOST_RGF, HOST_ICR), SSI_COMP_IRQ_MASK);
-
+
/* Avoid race with above clear: Test completion counter once more */
- request_mgr_handle->axi_completed += CC_REG_FLD_GET(CRY_KERNEL, AXIM_MON_COMP, VALUE,
- CC_HAL_READ_REGISTER(AXIM_MON_BASE_OFFSET));
-
- /* ISR-to-Tasklet latency */
- if (request_mgr_handle->axi_completed) {
- /* Only if actually reflects ISR-to-completion-handling latency, i.e.,
- not duplicate as a result of interrupt after AXIM_MON_ERR clear, before end of loop */
- END_CYCLE_COUNT_AT(drvdata->isr_exit_cycles, STAT_OP_TYPE_GENERIC, STAT_PHASE_1);
- }
-
+ request_mgr_handle->axi_completed +=
+ cc_axi_comp_count(cc_base);
+
while (request_mgr_handle->axi_completed) {
do {
proc_completions(drvdata);
- /* At this point (after proc_completions()), request_mgr_handle->axi_completed is always 0.
- The following assignment was changed to = (previously was +=) to conform KW restrictions. */
- request_mgr_handle->axi_completed = CC_REG_FLD_GET(CRY_KERNEL, AXIM_MON_COMP, VALUE,
- CC_HAL_READ_REGISTER(AXIM_MON_BASE_OFFSET));
+ /* At this point (after proc_completions()),
+ * request_mgr_handle->axi_completed is 0.
+ */
+ request_mgr_handle->axi_completed =
+ cc_axi_comp_count(cc_base);
} while (request_mgr_handle->axi_completed > 0);
-
+
/* To avoid the interrupt from firing as we unmask it, we clear it now */
CC_HAL_WRITE_REGISTER(CC_REG_OFFSET(HOST_RGF, HOST_ICR), SSI_COMP_IRQ_MASK);
-
+
/* Avoid race with above clear: Test completion counter once more */
- request_mgr_handle->axi_completed += CC_REG_FLD_GET(CRY_KERNEL, AXIM_MON_COMP, VALUE,
- CC_HAL_READ_REGISTER(AXIM_MON_BASE_OFFSET));
+ request_mgr_handle->axi_completed +=
+ cc_axi_comp_count(cc_base);
}
-
}
/* after verifing that there is nothing to do, Unmask AXI completion interrupt */
- CC_HAL_WRITE_REGISTER(CC_REG_OFFSET(HOST_RGF, HOST_IMR),
+ CC_HAL_WRITE_REGISTER(CC_REG_OFFSET(HOST_RGF, HOST_IMR),
CC_HAL_READ_REGISTER(
CC_REG_OFFSET(HOST_RGF, HOST_IMR)) & ~irq);
- END_CYCLE_COUNT(STAT_OP_TYPE_GENERIC, STAT_PHASE_2);
}
/*
-resume the queue configuration - no need to take the lock as this happens inside
-the spin lock protection
-*/
-#if defined (CONFIG_PM_RUNTIME) || defined (CONFIG_PM_SLEEP)
+ * resume the queue configuration - no need to take the lock as this happens inside
+ * the spin lock protection
+ */
+#if defined(CONFIG_PM_RUNTIME) || defined(CONFIG_PM_SLEEP)
int ssi_request_mgr_runtime_resume_queue(struct ssi_drvdata *drvdata)
{
- struct ssi_request_mgr_handle * request_mgr_handle = drvdata->request_mgr_handle;
+ struct ssi_request_mgr_handle *request_mgr_handle = drvdata->request_mgr_handle;
spin_lock_bh(&request_mgr_handle->hw_lock);
request_mgr_handle->is_runtime_suspended = false;
spin_unlock_bh(&request_mgr_handle->hw_lock);
- return 0 ;
+ return 0;
}
/*
-suspend the queue configuration. Since it is used for the runtime suspend
-only verify that the queue can be suspended.
-*/
+ * suspend the queue configuration. Since it is used for the runtime suspend
+ * only verify that the queue can be suspended.
+ */
int ssi_request_mgr_runtime_suspend_queue(struct ssi_drvdata *drvdata)
{
- struct ssi_request_mgr_handle * request_mgr_handle =
+ struct ssi_request_mgr_handle *request_mgr_handle =
drvdata->request_mgr_handle;
-
+
/* lock the send_request */
spin_lock_bh(&request_mgr_handle->hw_lock);
- if (request_mgr_handle->req_queue_head !=
+ if (request_mgr_handle->req_queue_head !=
request_mgr_handle->req_queue_tail) {
spin_unlock_bh(&request_mgr_handle->hw_lock);
return -EBUSY;
bool ssi_request_mgr_is_queue_runtime_suspend(struct ssi_drvdata *drvdata)
{
- struct ssi_request_mgr_handle * request_mgr_handle =
+ struct ssi_request_mgr_handle *request_mgr_handle =
drvdata->request_mgr_handle;
- return request_mgr_handle->is_runtime_suspended;
+ return request_mgr_handle->is_runtime_suspended;
}
#endif
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
/* \file request_mgr.h
- Request Manager
+ * Request Manager
*/
#ifndef __REQUEST_MGR_H__
/*!
* Enqueue caller request to crypto hardware.
- *
- * \param drvdata
+ *
+ * \param drvdata
* \param ssi_req The request to enqueue
* \param desc The crypto sequence
* \param len The crypto sequence length
- * \param is_dout If "true": completion is handled by the caller
- * If "false": this function adds a dummy descriptor completion
- * and waits upon completion signal.
- *
+ * \param is_dout If "true": completion is handled by the caller
+ * If "false": this function adds a dummy descriptor completion
+ * and waits upon completion signal.
+ *
* \return int Returns -EINPROGRESS if "is_dout=ture"; "0" if "is_dout=false"
*/
int send_request(
struct ssi_drvdata *drvdata, struct ssi_crypto_req *ssi_req,
- HwDesc_s *desc, unsigned int len, bool is_dout);
+ struct cc_hw_desc *desc, unsigned int len, bool is_dout);
int send_request_init(
- struct ssi_drvdata *drvdata, HwDesc_s *desc, unsigned int len);
+ struct ssi_drvdata *drvdata, struct cc_hw_desc *desc, unsigned int len);
void complete_request(struct ssi_drvdata *drvdata);
void request_mgr_fini(struct ssi_drvdata *drvdata);
-#if defined (CONFIG_PM_RUNTIME) || defined (CONFIG_PM_SLEEP)
+#if defined(CONFIG_PM_RUNTIME) || defined(CONFIG_PM_SLEEP)
int ssi_request_mgr_runtime_resume_queue(struct ssi_drvdata *drvdata);
int ssi_request_mgr_runtime_suspend_queue(struct ssi_drvdata *drvdata);
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
#include "ssi_driver.h"
#include "ssi_sram_mgr.h"
-
/**
* struct ssi_sram_mgr_ctx -Internal RAM context manager
* @sram_free_offset: the offset to the non-allocated area
ssi_sram_addr_t sram_free_offset;
};
-
/**
* ssi_sram_mgr_fini() - Cleanup SRAM pool.
- *
+ *
* @drvdata: Associated device driver context
*/
void ssi_sram_mgr_fini(struct ssi_drvdata *drvdata)
struct ssi_sram_mgr_ctx *smgr_ctx = drvdata->sram_mgr_handle;
/* Free "this" context */
- if (smgr_ctx != NULL) {
+ if (smgr_ctx) {
memset(smgr_ctx, 0, sizeof(struct ssi_sram_mgr_ctx));
kfree(smgr_ctx);
}
}
/**
- * ssi_sram_mgr_init() - Initializes SRAM pool.
+ * ssi_sram_mgr_init() - Initializes SRAM pool.
* The pool starts right at the beginning of SRAM.
* Returns zero for success, negative value otherwise.
- *
+ *
* @drvdata: Associated device driver context
*/
int ssi_sram_mgr_init(struct ssi_drvdata *drvdata)
}
/*!
- * Allocated buffer from SRAM pool.
- * Note: Caller is responsible to free the LAST allocated buffer.
- * This function does not taking care of any fragmentation may occur
- * by the order of calls to alloc/free.
- *
- * \param drvdata
+ * Allocated buffer from SRAM pool.
+ * Note: Caller is responsible to free the LAST allocated buffer.
+ * This function does not taking care of any fragmentation may occur
+ * by the order of calls to alloc/free.
+ *
+ * \param drvdata
* \param size The requested bytes to allocate
*/
-ssi_sram_addr_t ssi_sram_mgr_alloc(struct ssi_drvdata *drvdata, uint32_t size)
+ssi_sram_addr_t ssi_sram_mgr_alloc(struct ssi_drvdata *drvdata, u32 size)
{
struct ssi_sram_mgr_ctx *smgr_ctx = drvdata->sram_mgr_handle;
ssi_sram_addr_t p;
size, smgr_ctx->sram_free_offset);
return NULL_SRAM_ADDR;
}
-
+
p = smgr_ctx->sram_free_offset;
smgr_ctx->sram_free_offset += size;
SSI_LOG_DEBUG("Allocated %u B @ %u\n", size, (unsigned int)p);
/**
* ssi_sram_mgr_const2sram_desc() - Create const descriptors sequence to
- * set values in given array into SRAM.
+ * set values in given array into SRAM.
* Note: each const value can't exceed word size.
- *
+ *
* @src: A pointer to array of words to set as consts.
* @dst: The target SRAM buffer to set into
* @nelements: The number of words in "src" array
* @seq_len: A pointer to the given IN/OUT sequence length
*/
void ssi_sram_mgr_const2sram_desc(
- const uint32_t *src, ssi_sram_addr_t dst,
+ const u32 *src, ssi_sram_addr_t dst,
unsigned int nelement,
- HwDesc_s *seq, unsigned int *seq_len)
+ struct cc_hw_desc *seq, unsigned int *seq_len)
{
- uint32_t i;
+ u32 i;
unsigned int idx = *seq_len;
for (i = 0; i < nelement; i++, idx++) {
- HW_DESC_INIT(&seq[idx]);
- HW_DESC_SET_DIN_CONST(&seq[idx], src[i], sizeof(uint32_t));
- HW_DESC_SET_DOUT_SRAM(&seq[idx], dst + (i * sizeof(uint32_t)), sizeof(uint32_t));
- HW_DESC_SET_FLOW_MODE(&seq[idx], BYPASS);
+ hw_desc_init(&seq[idx]);
+ set_din_const(&seq[idx], src[i], sizeof(u32));
+ set_dout_sram(&seq[idx], dst + (i * sizeof(u32)), sizeof(u32));
+ set_flow_mode(&seq[idx], BYPASS);
}
*seq_len = idx;
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
#ifndef __SSI_SRAM_MGR_H__
#define __SSI_SRAM_MGR_H__
-
#ifndef SSI_CC_SRAM_SIZE
#define SSI_CC_SRAM_SIZE 4096
#endif
* Address (offset) within CC internal SRAM
*/
-typedef uint64_t ssi_sram_addr_t;
+typedef u64 ssi_sram_addr_t;
#define NULL_SRAM_ADDR ((ssi_sram_addr_t)-1)
/*!
- * Initializes SRAM pool.
- * The first X bytes of SRAM are reserved for ROM usage, hence, pool
- * starts right after X bytes.
- *
- * \param drvdata
- *
+ * Initializes SRAM pool.
+ * The first X bytes of SRAM are reserved for ROM usage, hence, pool
+ * starts right after X bytes.
+ *
+ * \param drvdata
+ *
* \return int Zero for success, negative value otherwise.
*/
int ssi_sram_mgr_init(struct ssi_drvdata *drvdata);
/*!
* Uninits SRAM pool.
- *
- * \param drvdata
+ *
+ * \param drvdata
*/
void ssi_sram_mgr_fini(struct ssi_drvdata *drvdata);
/*!
- * Allocated buffer from SRAM pool.
- * Note: Caller is responsible to free the LAST allocated buffer.
- * This function does not taking care of any fragmentation may occur
- * by the order of calls to alloc/free.
- *
- * \param drvdata
+ * Allocated buffer from SRAM pool.
+ * Note: Caller is responsible to free the LAST allocated buffer.
+ * This function does not taking care of any fragmentation may occur
+ * by the order of calls to alloc/free.
+ *
+ * \param drvdata
* \param size The requested bytes to allocate
*/
-ssi_sram_addr_t ssi_sram_mgr_alloc(struct ssi_drvdata *drvdata, uint32_t size);
+ssi_sram_addr_t ssi_sram_mgr_alloc(struct ssi_drvdata *drvdata, u32 size);
/**
* ssi_sram_mgr_const2sram_desc() - Create const descriptors sequence to
- * set values in given array into SRAM.
+ * set values in given array into SRAM.
* Note: each const value can't exceed word size.
- *
+ *
* @src: A pointer to array of words to set as consts.
* @dst: The target SRAM buffer to set into
* @nelements: The number of words in "src" array
* @seq_len: A pointer to the given IN/OUT sequence length
*/
void ssi_sram_mgr_const2sram_desc(
- const uint32_t *src, ssi_sram_addr_t dst,
+ const u32 *src, ssi_sram_addr_t dst,
unsigned int nelement,
- HwDesc_s *seq, unsigned int *seq_len);
+ struct cc_hw_desc *seq, unsigned int *seq_len);
#endif /*__SSI_SRAM_MGR_H__*/
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
const char *stat_phase_name[MAX_STAT_PHASES];
};
-static struct stat_name stat_name_db[MAX_STAT_OP_TYPES] =
+static struct stat_name stat_name_db[MAX_STAT_OP_TYPES] =
{
{
/* STAT_OP_TYPE_NULL */
{
.op_type_name = "Encode",
.stat_phase_name[STAT_PHASE_0] = "Init and sanity checks",
- .stat_phase_name[STAT_PHASE_1] = "Map buffers",
- .stat_phase_name[STAT_PHASE_2] = "Create sequence",
+ .stat_phase_name[STAT_PHASE_1] = "Map buffers",
+ .stat_phase_name[STAT_PHASE_2] = "Create sequence",
.stat_phase_name[STAT_PHASE_3] = "Send Request",
.stat_phase_name[STAT_PHASE_4] = "HW-Q push",
.stat_phase_name[STAT_PHASE_5] = "Sequence completion",
},
{ .op_type_name = "Decode",
.stat_phase_name[STAT_PHASE_0] = "Init and sanity checks",
- .stat_phase_name[STAT_PHASE_1] = "Map buffers",
- .stat_phase_name[STAT_PHASE_2] = "Create sequence",
+ .stat_phase_name[STAT_PHASE_1] = "Map buffers",
+ .stat_phase_name[STAT_PHASE_2] = "Create sequence",
.stat_phase_name[STAT_PHASE_3] = "Send Request",
.stat_phase_name[STAT_PHASE_4] = "HW-Q push",
.stat_phase_name[STAT_PHASE_5] = "Sequence completion",
.stat_phase_name[STAT_PHASE_6] = "HW cycles",
},
- { .op_type_name = "Setkey",
+ { .op_type_name = "Setkey",
.stat_phase_name[STAT_PHASE_0] = "Init and sanity checks",
.stat_phase_name[STAT_PHASE_1] = "Copy key to ctx",
.stat_phase_name[STAT_PHASE_2] = "Create sequence",
};
/*
- * Structure used to create a directory
+ * Structure used to create a directory
* and its attributes in sysfs.
*/
struct sys_dir {
struct kobject *sys_dir_kobj;
struct attribute_group sys_dir_attr_group;
struct attribute **sys_dir_attr_list;
- uint32_t num_of_attrs;
+ u32 num_of_attrs;
struct ssi_drvdata *drvdata; /* Associated driver context */
};
static struct stat_item stat_host_db[MAX_STAT_OP_TYPES][MAX_STAT_PHASES];
static struct stat_item stat_cc_db[MAX_STAT_OP_TYPES][MAX_STAT_PHASES];
-
static void init_db(struct stat_item item[MAX_STAT_OP_TYPES][MAX_STAT_PHASES])
{
unsigned int i, j;
/* Clear db */
- for (i=0; i<MAX_STAT_OP_TYPES; i++) {
- for (j=0; j<MAX_STAT_PHASES; j++) {
+ for (i = 0; i < MAX_STAT_OP_TYPES; i++) {
+ for (j = 0; j < MAX_STAT_PHASES; j++) {
item[i][j].min = 0xFFFFFFFF;
item[i][j].max = 0;
item[i][j].sum = 0;
item->sum += result;
if (result < item->min)
item->min = result;
- if (result > item->max )
+ if (result > item->max)
item->max = result;
}
static void display_db(struct stat_item item[MAX_STAT_OP_TYPES][MAX_STAT_PHASES])
{
unsigned int i, j;
- uint64_t avg;
+ u64 avg;
- for (i=STAT_OP_TYPE_ENCODE; i<MAX_STAT_OP_TYPES; i++) {
- for (j=0; j<MAX_STAT_PHASES; j++) {
+ for (i = STAT_OP_TYPE_ENCODE; i < MAX_STAT_OP_TYPES; i++) {
+ for (j = 0; j < MAX_STAT_PHASES; j++) {
if (item[i][j].count > 0) {
- avg = (uint64_t)item[i][j].sum;
+ avg = (u64)item[i][j].sum;
do_div(avg, item[i][j].count);
- SSI_LOG_ERR("%s, %s: min=%d avg=%d max=%d sum=%lld count=%d\n",
- stat_name_db[i].op_type_name, stat_name_db[i].stat_phase_name[j],
+ SSI_LOG_ERR("%s, %s: min=%d avg=%d max=%d sum=%lld count=%d\n",
+ stat_name_db[i].op_type_name, stat_name_db[i].stat_phase_name[j],
item[i][j].min, (int)avg, item[i][j].max, (long long)item[i][j].sum, item[i][j].count);
}
}
}
}
-
/**************************************
* Attributes show functions section *
**************************************/
static ssize_t ssi_sys_stat_host_db_show(struct kobject *kobj,
struct kobj_attribute *attr, char *buf)
{
- int i, j ;
+ int i, j;
char line[512];
- uint32_t min_cyc, max_cyc;
- uint64_t avg;
- ssize_t buf_len, tmp_len=0;
+ u32 min_cyc, max_cyc;
+ u64 avg;
+ ssize_t buf_len, tmp_len = 0;
- buf_len = scnprintf(buf,PAGE_SIZE,
+ buf_len = scnprintf(buf, PAGE_SIZE,
"phase\t\t\t\t\t\t\tmin[cy]\tavg[cy]\tmax[cy]\t#samples\n");
- if ( buf_len <0 )/* scnprintf shouldn't return negative value according to its implementation*/
+ if (buf_len < 0)/* scnprintf shouldn't return negative value according to its implementation*/
return buf_len;
- for (i=STAT_OP_TYPE_ENCODE; i<MAX_STAT_OP_TYPES; i++) {
- for (j=0; j<MAX_STAT_PHASES-1; j++) {
+ for (i = STAT_OP_TYPE_ENCODE; i < MAX_STAT_OP_TYPES; i++) {
+ for (j = 0; j < MAX_STAT_PHASES - 1; j++) {
if (stat_host_db[i][j].count > 0) {
- avg = (uint64_t)stat_host_db[i][j].sum;
+ avg = (u64)stat_host_db[i][j].sum;
do_div(avg, stat_host_db[i][j].count);
min_cyc = stat_host_db[i][j].min;
max_cyc = stat_host_db[i][j].max;
} else {
avg = min_cyc = max_cyc = 0;
}
- tmp_len = scnprintf(line,512,
+ tmp_len = scnprintf(line, 512,
"%s::%s\t\t\t\t\t%6u\t%6u\t%6u\t%7u\n",
stat_name_db[i].op_type_name,
stat_name_db[i].stat_phase_name[j],
min_cyc, (unsigned int)avg, max_cyc,
stat_host_db[i][j].count);
- if ( tmp_len <0 )/* scnprintf shouldn't return negative value according to its implementation*/
+ if (tmp_len < 0)/* scnprintf shouldn't return negative value according to its implementation*/
return buf_len;
- if ( buf_len + tmp_len >= PAGE_SIZE)
+ if (buf_len + tmp_len >= PAGE_SIZE)
return buf_len;
buf_len += tmp_len;
- strncat(buf, line,512);
+ strncat(buf, line, 512);
}
}
return buf_len;
{
int i;
char line[256];
- uint32_t min_cyc, max_cyc;
- uint64_t avg;
- ssize_t buf_len,tmp_len=0;
+ u32 min_cyc, max_cyc;
+ u64 avg;
+ ssize_t buf_len, tmp_len = 0;
- buf_len = scnprintf(buf,PAGE_SIZE,
+ buf_len = scnprintf(buf, PAGE_SIZE,
"phase\tmin[cy]\tavg[cy]\tmax[cy]\t#samples\n");
- if ( buf_len <0 )/* scnprintf shouldn't return negative value according to its implementation*/
+ if (buf_len < 0)/* scnprintf shouldn't return negative value according to its implementation*/
return buf_len;
- for (i=STAT_OP_TYPE_ENCODE; i<MAX_STAT_OP_TYPES; i++) {
+ for (i = STAT_OP_TYPE_ENCODE; i < MAX_STAT_OP_TYPES; i++) {
if (stat_cc_db[i][STAT_PHASE_6].count > 0) {
- avg = (uint64_t)stat_cc_db[i][STAT_PHASE_6].sum;
+ avg = (u64)stat_cc_db[i][STAT_PHASE_6].sum;
do_div(avg, stat_cc_db[i][STAT_PHASE_6].count);
min_cyc = stat_cc_db[i][STAT_PHASE_6].min;
max_cyc = stat_cc_db[i][STAT_PHASE_6].max;
} else {
avg = min_cyc = max_cyc = 0;
}
- tmp_len = scnprintf(line,256,
+ tmp_len = scnprintf(line, 256,
"%s\t%6u\t%6u\t%6u\t%7u\n",
stat_name_db[i].op_type_name,
min_cyc,
max_cyc,
stat_cc_db[i][STAT_PHASE_6].count);
- if ( tmp_len < 0 )/* scnprintf shouldn't return negative value according to its implementation*/
+ if (tmp_len < 0)/* scnprintf shouldn't return negative value according to its implementation*/
return buf_len;
- if ( buf_len + tmp_len >= PAGE_SIZE)
+ if (buf_len + tmp_len >= PAGE_SIZE)
return buf_len;
buf_len += tmp_len;
- strncat(buf, line,256);
+ strncat(buf, line, 256);
}
return buf_len;
}
void display_all_stat_db(void)
{
- SSI_LOG_ERR("\n======= CYCLE COUNT STATS =======\n");
+ SSI_LOG_ERR("\n======= CYCLE COUNT STATS =======\n");
display_db(stat_host_db);
- SSI_LOG_ERR("\n======= CC HW CYCLE COUNT STATS =======\n");
+ SSI_LOG_ERR("\n======= CC HW CYCLE COUNT STATS =======\n");
display_db(stat_cc_db);
}
#endif /*CC_CYCLE_COUNT*/
-
-
static ssize_t ssi_sys_regdump_show(struct kobject *kobj,
struct kobj_attribute *attr, char *buf)
{
struct ssi_drvdata *drvdata = sys_get_drvdata();
- uint32_t register_value;
- void __iomem* cc_base = drvdata->cc_base;
+ u32 register_value;
+ void __iomem *cc_base = drvdata->cc_base;
int offset = 0;
register_value = CC_HAL_READ_REGISTER(CC_REG_OFFSET(HOST_RGF, HOST_SIGNATURE));
static ssize_t ssi_sys_help_show(struct kobject *kobj,
struct kobj_attribute *attr, char *buf)
{
- char* help_str[]={
+ char *help_str[] = {
"cat reg_dump ", "Print several of CC register values",
#if defined CC_CYCLE_COUNT
"cat stats_host ", "Print host statistics",
"echo <number> > stats_cc ", "Clear CC statistics database",
#endif
};
- int i=0, offset = 0;
+ int i = 0, offset = 0;
offset += scnprintf(buf + offset, PAGE_SIZE - offset, "Usage:\n");
- for ( i = 0; i < ARRAY_SIZE(help_str); i+=2) {
- offset += scnprintf(buf + offset, PAGE_SIZE - offset, "%s\t\t%s\n", help_str[i], help_str[i+1]);
- }
+ for (i = 0; i < ARRAY_SIZE(help_str); i += 2)
+ offset += scnprintf(buf + offset, PAGE_SIZE - offset, "%s\t\t%s\n", help_str[i], help_str[i + 1]);
+
return offset;
}
struct kobject *sys_dir_kobj;
struct attribute_group sys_dir_attr_group;
struct attribute **sys_dir_attr_list;
- uint32_t num_of_attrs;
+ u32 num_of_attrs;
struct ssi_drvdata *drvdata; /* Associated driver context */
};
{
/* TODO: supporting multiple SeP devices would require avoiding
* global "top_dir" and finding associated "top_dir" by traversing
- * up the tree to the kobject which matches one of the top_dir's */
+ * up the tree to the kobject which matches one of the top_dir's
+ */
return sys_top_dir.drvdata;
}
static int sys_init_dir(struct sys_dir *sys_dir, struct ssi_drvdata *drvdata,
struct kobject *parent_dir_kobj, const char *dir_name,
- struct kobj_attribute *attrs, uint32_t num_of_attrs)
+ struct kobj_attribute *attrs, u32 num_of_attrs)
{
int i;
kfree(sys_dir->sys_dir_attr_list);
- if (sys_dir->sys_dir_kobj != NULL)
+ if (sys_dir->sys_dir_kobj)
kobject_put(sys_dir->sys_dir_kobj);
}
/*
* Copyright (C) 2012-2017 ARM Limited or its affiliates.
- *
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
- *
+ *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
- *
+ *
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
/* \file ssi_sysfs.h
- ARM CryptoCell sysfs APIs
+ * ARM CryptoCell sysfs APIs
*/
#ifndef __SSI_SYSFS_H__
STAT_PHASE_6,
MAX_STAT_PHASES,
};
+
enum stat_op {
STAT_OP_TYPE_NULL = 0,
STAT_OP_TYPE_ENCODE,
retval = register_chrdev_region(MKDEV(COMEDI_MAJOR, 0),
COMEDI_NUM_MINORS, "comedi");
if (retval)
- return -EIO;
+ return retval;
+
cdev_init(&comedi_cdev, &comedi_fops);
comedi_cdev.owner = THIS_MODULE;
retval = kobject_set_name(&comedi_cdev.kobj, "comedi");
- if (retval) {
- unregister_chrdev_region(MKDEV(COMEDI_MAJOR, 0),
- COMEDI_NUM_MINORS);
- return retval;
- }
+ if (retval)
+ goto out_unregister_chrdev_region;
+
+ retval = cdev_add(&comedi_cdev, MKDEV(COMEDI_MAJOR, 0),
+ COMEDI_NUM_MINORS);
+ if (retval)
+ goto out_unregister_chrdev_region;
- if (cdev_add(&comedi_cdev, MKDEV(COMEDI_MAJOR, 0), COMEDI_NUM_MINORS)) {
- unregister_chrdev_region(MKDEV(COMEDI_MAJOR, 0),
- COMEDI_NUM_MINORS);
- return -EIO;
- }
comedi_class = class_create(THIS_MODULE, "comedi");
if (IS_ERR(comedi_class)) {
+ retval = PTR_ERR(comedi_class);
pr_err("failed to create class\n");
- cdev_del(&comedi_cdev);
- unregister_chrdev_region(MKDEV(COMEDI_MAJOR, 0),
- COMEDI_NUM_MINORS);
- return PTR_ERR(comedi_class);
+ goto out_cdev_del;
}
comedi_class->dev_groups = comedi_dev_groups;
dev = comedi_alloc_board_minor(NULL);
if (IS_ERR(dev)) {
- comedi_cleanup_board_minors();
- cdev_del(&comedi_cdev);
- unregister_chrdev_region(MKDEV(COMEDI_MAJOR, 0),
- COMEDI_NUM_MINORS);
- return PTR_ERR(dev);
+ retval = PTR_ERR(dev);
+ goto out_cleanup_board_minors;
}
/* comedi_alloc_board_minor() locked the mutex */
mutex_unlock(&dev->mutex);
comedi_proc_init();
return 0;
+
+out_cleanup_board_minors:
+ comedi_cleanup_board_minors();
+ class_destroy(comedi_class);
+out_cdev_del:
+ cdev_del(&comedi_cdev);
+out_unregister_chrdev_region:
+ unregister_chrdev_region(MKDEV(COMEDI_MAJOR, 0), COMEDI_NUM_MINORS);
+ return retval;
}
module_init(comedi_init);
/*
* ni_labpc ISA DMA support.
-*/
+ */
#ifndef _NI_LABPC_ISADMA_H
#define _NI_LABPC_ISADMA_H
/*
* ni_labpc register definitions.
-*/
+ */
#ifndef _NI_LABPC_REGS_H
#define _NI_LABPC_REGS_H
* running after the packet has been sent to the target DAC.
*/
val = 0x0F000000; /* Continue clock after target DAC data
- * (write to non-existent trimdac). */
+ * (write to non-existent trimdac).
+ */
val |= 0x00004000; /* Address the two main dual-DAC devices
- * (TSL's chip select enables target device). */
+ * (TSL's chip select enables target device).
+ */
val |= ((u32)(chan & 1) << 15); /* Address the DAC channel
- * within the device. */
+ * within the device.
+ */
val |= (u32)dacdata; /* Include DAC setpoint data. */
return s626_send_dac(dev, val);
}
return 0;
}
-
/* A board has been found, initialize it. */
static struct dgnc_board *dgnc_found_board(struct pci_dev *pdev, int id)
{
free_irq(brd->irq, brd);
}
-
/*
* As each timer expires, it determines (a) whether the "transmit"
* waiter needs to be woken up, and (b) whether the poller needs to
#define dgnc_jiffies_from_ms(a) (((a) * HZ) / 1000)
-/*
- * Define a local default termios struct. All ports will be created
- * with this termios initially. This is the same structure that is defined
- * as the default in tty_io.c with the same settings overridden as in serial.c
- *
- * In short, this should match the internal serial ports' defaults.
- */
-#define DEFAULT_IFLAGS (ICRNL | IXON)
-#define DEFAULT_OFLAGS (OPOST | ONLCR)
-#define DEFAULT_CFLAGS (B9600 | CS8 | CREAD | HUPCL | CLOCAL)
-#define DEFAULT_LFLAGS (ISIG | ICANON | ECHO | ECHOE | ECHOK | \
- ECHOCTL | ECHOKE | IEXTEN)
-
#ifndef _POSIX_VDISABLE
#define _POSIX_VDISABLE '\0'
#endif
.digi_term = "ansi" /* default terminal type */
};
-/*
- * Define a local default termios struct. All ports will be created
- * with this termios initially.
- *
- * This defines a raw port at 9600 baud, 8 data bits, no parity,
- * 1 stop bit.
- */
-static const struct ktermios default_termios = {
- .c_iflag = (DEFAULT_IFLAGS),
- .c_oflag = (DEFAULT_OFLAGS),
- .c_cflag = (DEFAULT_CFLAGS),
- .c_lflag = (DEFAULT_LFLAGS),
- .c_cc = INIT_C_CC,
- .c_line = 0,
-};
-
static int dgnc_tty_open(struct tty_struct *tty, struct file *file);
static void dgnc_tty_close(struct tty_struct *tty, struct file *file);
static int dgnc_block_til_ready(struct tty_struct *tty, struct file *file,
/* TTY Initialization/Cleanup Functions */
+static struct tty_driver *dgnc_tty_create(char *serial_name, uint maxports,
+ int major, int minor)
+{
+ int rc;
+ struct tty_driver *drv;
+
+ drv = tty_alloc_driver(maxports,
+ TTY_DRIVER_REAL_RAW |
+ TTY_DRIVER_DYNAMIC_DEV |
+ TTY_DRIVER_HARDWARE_BREAK);
+ if (IS_ERR(drv))
+ return drv;
+
+ drv->name = serial_name;
+ drv->name_base = 0;
+ drv->major = major;
+ drv->minor_start = minor;
+ drv->type = TTY_DRIVER_TYPE_SERIAL;
+ drv->subtype = SERIAL_TYPE_NORMAL;
+ drv->init_termios = tty_std_termios;
+ drv->init_termios.c_cflag = (B9600 | CS8 | CREAD | HUPCL | CLOCAL);
+ drv->init_termios.c_ispeed = 9600;
+ drv->init_termios.c_ospeed = 9600;
+ drv->driver_name = DRVSTR;
+ /*
+ * Entry points for driver. Called by the kernel from
+ * tty_io.c and n_tty.c.
+ */
+ tty_set_operations(drv, &dgnc_tty_ops);
+ rc = tty_register_driver(drv);
+ if (rc < 0) {
+ put_tty_driver(drv);
+ return ERR_PTR(rc);
+ }
+ return drv;
+}
+
+static void dgnc_tty_free(struct tty_driver *drv)
+{
+ tty_unregister_driver(drv);
+ put_tty_driver(drv);
+}
+
/**
* dgnc_tty_register() - Init the tty subsystem for this board.
*/
{
int rc;
- brd->serial_driver = tty_alloc_driver(brd->maxports,
- TTY_DRIVER_REAL_RAW |
- TTY_DRIVER_DYNAMIC_DEV |
- TTY_DRIVER_HARDWARE_BREAK);
- if (IS_ERR(brd->serial_driver))
- return PTR_ERR(brd->serial_driver);
-
snprintf(brd->serial_name, MAXTTYNAMELEN, "tty_dgnc_%d_",
brd->boardnum);
- brd->serial_driver->name = brd->serial_name;
- brd->serial_driver->name_base = 0;
- brd->serial_driver->major = 0;
- brd->serial_driver->minor_start = 0;
- brd->serial_driver->type = TTY_DRIVER_TYPE_SERIAL;
- brd->serial_driver->subtype = SERIAL_TYPE_NORMAL;
- brd->serial_driver->init_termios = default_termios;
- brd->serial_driver->driver_name = DRVSTR;
-
- /*
- * Entry points for driver. Called by the kernel from
- * tty_io.c and n_tty.c.
- */
- tty_set_operations(brd->serial_driver, &dgnc_tty_ops);
-
- rc = tty_register_driver(brd->serial_driver);
- if (rc < 0) {
- dev_dbg(&brd->pdev->dev,
- "Can't register tty device (%d)\n", rc);
- goto free_serial_driver;
+ brd->serial_driver = dgnc_tty_create(brd->serial_name,
+ brd->maxports, 0, 0);
+ if (IS_ERR(brd->serial_driver)) {
+ rc = PTR_ERR(brd->serial_driver);
+ dev_dbg(&brd->pdev->dev, "Can't register tty device (%d)\n",
+ rc);
+ return rc;
}
- /*
- * If we're doing transparent print, we have to do all of the above
- * again, separately so we don't get the LD confused about what major
- * we are when we get into the dgnc_tty_open() routine.
- */
- brd->print_driver = tty_alloc_driver(brd->maxports,
- TTY_DRIVER_REAL_RAW |
- TTY_DRIVER_DYNAMIC_DEV |
- TTY_DRIVER_HARDWARE_BREAK);
+ snprintf(brd->print_name, MAXTTYNAMELEN, "pr_dgnc_%d_", brd->boardnum);
+ brd->print_driver = dgnc_tty_create(brd->print_name, brd->maxports,
+ 0x80,
+ brd->serial_driver->major);
if (IS_ERR(brd->print_driver)) {
rc = PTR_ERR(brd->print_driver);
- goto unregister_serial_driver;
- }
-
- snprintf(brd->print_name, MAXTTYNAMELEN, "pr_dgnc_%d_", brd->boardnum);
-
- brd->print_driver->name = brd->print_name;
- brd->print_driver->name_base = 0;
- brd->print_driver->major = brd->serial_driver->major;
- brd->print_driver->minor_start = 0x80;
- brd->print_driver->type = TTY_DRIVER_TYPE_SERIAL;
- brd->print_driver->subtype = SERIAL_TYPE_NORMAL;
- brd->print_driver->init_termios = default_termios;
- brd->print_driver->driver_name = DRVSTR;
-
- /*
- * Entry points for driver. Called by the kernel from
- * tty_io.c and n_tty.c.
- */
- tty_set_operations(brd->print_driver, &dgnc_tty_ops);
-
- rc = tty_register_driver(brd->print_driver);
- if (rc < 0) {
dev_dbg(&brd->pdev->dev,
- "Can't register Transparent Print device(%d)\n",
- rc);
- goto free_print_driver;
+ "Can't register Transparent Print device(%d)\n", rc);
+ dgnc_tty_free(brd->serial_driver);
+ return rc;
}
-
return 0;
-
-free_print_driver:
- put_tty_driver(brd->print_driver);
-unregister_serial_driver:
- tty_unregister_driver(brd->serial_driver);
-free_serial_driver:
- put_tty_driver(brd->serial_driver);
-
- return rc;
}
void dgnc_tty_unregister(struct dgnc_board *brd)
{
- tty_unregister_driver(brd->print_driver);
- tty_unregister_driver(brd->serial_driver);
- put_tty_driver(brd->print_driver);
- put_tty_driver(brd->serial_driver);
+ dgnc_tty_free(brd->print_driver);
+ dgnc_tty_free(brd->serial_driver);
}
/**
for (num = 0; num < NUM_ENDPOINTS - 1; num++) {
p_ep_regs = &udc->p_regs->EP_REGS[num];
data = _nbu2ss_readl(&p_ep_regs->EP_PCKT_ADRS);
- buf_type = _nbu2ss_readl(&p_ep_regs->EP_CONTROL) & EPn_BUF_TYPE;
+ buf_type = _nbu2ss_readl(&p_ep_regs->EP_CONTROL) & EPN_BUF_TYPE;
if (buf_type == 0) {
/* Single Buffer */
- use_ram_size += (data & EPn_MPKT) / sizeof(u32);
+ use_ram_size += (data & EPN_MPKT) / sizeof(u32);
} else {
/* Double Buffer */
- use_ram_size += ((data & EPn_MPKT) / sizeof(u32)) * 2;
+ use_ram_size += ((data & EPN_MPKT) / sizeof(u32)) * 2;
}
if ((data >> 16) > last_ram_adr)
/* Bulk, Interrupt, ISO */
switch (ep->ep_type) {
case USB_ENDPOINT_XFER_BULK:
- data = EPn_BULK;
+ data = EPN_BULK;
break;
case USB_ENDPOINT_XFER_INT:
- data = EPn_BUF_SINGLE | EPn_INTERRUPT;
+ data = EPN_BUF_SINGLE | EPN_INTERRUPT;
break;
case USB_ENDPOINT_XFER_ISOC:
- data = EPn_ISO;
+ data = EPN_ISO;
break;
default:
if (ep->direct == USB_DIR_OUT) {
/*---------------------------------------------------------*/
/* OUT */
- data = EPn_EN | EPn_BCLR | EPn_DIR0;
+ data = EPN_EN | EPN_BCLR | EPN_DIR0;
_nbu2ss_bitset(&udc->p_regs->EP_REGS[num].EP_CONTROL, data);
- data = EPn_ONAK | EPn_OSTL_EN | EPn_OSTL;
+ data = EPN_ONAK | EPN_OSTL_EN | EPN_OSTL;
_nbu2ss_bitclr(&udc->p_regs->EP_REGS[num].EP_CONTROL, data);
- data = EPn_OUT_EN | EPn_OUT_END_EN;
+ data = EPN_OUT_EN | EPN_OUT_END_EN;
_nbu2ss_bitset(&udc->p_regs->EP_REGS[num].EP_INT_ENA, data);
} else {
/*---------------------------------------------------------*/
/* IN */
- data = EPn_EN | EPn_BCLR | EPn_AUTO;
+ data = EPN_EN | EPN_BCLR | EPN_AUTO;
_nbu2ss_bitset(&udc->p_regs->EP_REGS[num].EP_CONTROL, data);
- data = EPn_ISTL;
+ data = EPN_ISTL;
_nbu2ss_bitclr(&udc->p_regs->EP_REGS[num].EP_CONTROL, data);
- data = EPn_IN_EN | EPn_IN_END_EN;
+ data = EPN_IN_EN | EPN_IN_END_EN;
_nbu2ss_bitset(&udc->p_regs->EP_REGS[num].EP_INT_ENA, data);
}
if (ep->direct == USB_DIR_OUT) {
/*---------------------------------------------------------*/
/* OUT */
- data = EPn_ONAK | EPn_BCLR;
+ data = EPN_ONAK | EPN_BCLR;
_nbu2ss_bitset(&udc->p_regs->EP_REGS[num].EP_CONTROL, data);
- data = EPn_EN | EPn_DIR0;
+ data = EPN_EN | EPN_DIR0;
_nbu2ss_bitclr(&udc->p_regs->EP_REGS[num].EP_CONTROL, data);
- data = EPn_OUT_EN | EPn_OUT_END_EN;
+ data = EPN_OUT_EN | EPN_OUT_END_EN;
_nbu2ss_bitclr(&udc->p_regs->EP_REGS[num].EP_INT_ENA, data);
} else {
/*---------------------------------------------------------*/
/* IN */
- data = EPn_BCLR;
+ data = EPN_BCLR;
_nbu2ss_bitset(&udc->p_regs->EP_REGS[num].EP_CONTROL, data);
- data = EPn_EN | EPn_AUTO;
+ data = EPN_EN | EPN_AUTO;
_nbu2ss_bitclr(&udc->p_regs->EP_REGS[num].EP_CONTROL, data);
- data = EPn_IN_EN | EPn_IN_END_EN;
+ data = EPN_IN_EN | EPN_IN_END_EN;
_nbu2ss_bitclr(&udc->p_regs->EP_REGS[num].EP_INT_ENA, data);
}
/*---------------------------------------------------------*/
/* Transfer Direct */
- data = DCR1_EPn_DIR0;
+ data = DCR1_EPN_DIR0;
_nbu2ss_bitset(&udc->p_regs->EP_DCR[num].EP_DCR1, data);
/*---------------------------------------------------------*/
/* DMA Mode etc. */
- data = EPn_STOP_MODE | EPn_STOP_SET | EPn_DMAMODE0;
+ data = EPN_STOP_MODE | EPN_STOP_SET | EPN_DMAMODE0;
_nbu2ss_writel(&udc->p_regs->EP_REGS[num].EP_DMA_CTRL, data);
} else {
/*---------------------------------------------------------*/
/* IN */
- _nbu2ss_bitset(&udc->p_regs->EP_REGS[num].EP_CONTROL, EPn_AUTO);
+ _nbu2ss_bitset(&udc->p_regs->EP_REGS[num].EP_CONTROL, EPN_AUTO);
/*---------------------------------------------------------*/
/* DMA Mode etc. */
- data = EPn_BURST_SET | EPn_DMAMODE0;
+ data = EPN_BURST_SET | EPN_DMAMODE0;
_nbu2ss_writel(&udc->p_regs->EP_REGS[num].EP_DMA_CTRL, data);
}
}
/*---------------------------------------------------------*/
/* OUT */
_nbu2ss_writel(&preg->EP_DCR[num].EP_DCR2, 0);
- _nbu2ss_bitclr(&preg->EP_DCR[num].EP_DCR1, DCR1_EPn_DIR0);
+ _nbu2ss_bitclr(&preg->EP_DCR[num].EP_DCR1, DCR1_EPN_DIR0);
_nbu2ss_writel(&preg->EP_REGS[num].EP_DMA_CTRL, 0);
} else {
/*---------------------------------------------------------*/
/* IN */
- _nbu2ss_bitclr(&preg->EP_REGS[num].EP_CONTROL, EPn_AUTO);
+ _nbu2ss_bitclr(&preg->EP_REGS[num].EP_CONTROL, EPN_AUTO);
_nbu2ss_writel(&preg->EP_REGS[num].EP_DMA_CTRL, 0);
}
}
{
struct fc_regs *preg = udc->p_regs;
- _nbu2ss_bitclr(&preg->EP_DCR[ep->epnum - 1].EP_DCR1, DCR1_EPn_REQEN);
- mdelay(DMA_DISABLE_TIME); /* DCR1_EPn_REQEN Clear */
- _nbu2ss_bitclr(&preg->EP_REGS[ep->epnum - 1].EP_DMA_CTRL, EPn_DMA_EN);
+ _nbu2ss_bitclr(&preg->EP_DCR[ep->epnum - 1].EP_DCR1, DCR1_EPN_REQEN);
+ mdelay(DMA_DISABLE_TIME); /* DCR1_EPN_REQEN Clear */
+ _nbu2ss_bitclr(&preg->EP_REGS[ep->epnum - 1].EP_DMA_CTRL, EPN_DMA_EN);
}
/*-------------------------------------------------------------------------*/
} else {
num = epnum - 1;
- _nbu2ss_bitclr(&preg->EP_REGS[num].EP_CONTROL, EPn_AUTO);
+ _nbu2ss_bitclr(&preg->EP_REGS[num].EP_CONTROL, EPN_AUTO);
/* Writing of 1-4 bytes */
if (length)
_nbu2ss_writel(&preg->EP_REGS[num].EP_WRITE, data32);
- data = (((length) << 5) & EPn_DW) | EPn_DEND;
+ data = (((length) << 5) & EPN_DW) | EPN_DEND;
_nbu2ss_bitset(&preg->EP_REGS[num].EP_CONTROL, data);
- _nbu2ss_bitset(&preg->EP_REGS[num].EP_CONTROL, EPn_AUTO);
+ _nbu2ss_bitset(&preg->EP_REGS[num].EP_CONTROL, EPN_AUTO);
}
}
if (direct == USB_DIR_OUT)
memcpy(req->req.buf, ep->virt_buf,
req->req.actual & 0xfffffffc);
- } else
+ } else {
dma_unmap_single(udc->gadget.dev.parent,
req->req.dma, req->req.length,
(direct == USB_DIR_IN)
? DMA_TO_DEVICE
: DMA_FROM_DEVICE);
+ }
req->req.dma = DMA_ADDR_INVALID;
req->mapped = 0;
} else {
/*-------------------------------------------------------------------------*/
/* Endpoint 0 OUT Transfer (PIO, OverBytes) */
-static int EP0_out_OverBytes(struct nbu2ss_udc *udc, u8 *pBuf, u32 length)
+static int ep0_out_overbytes(struct nbu2ss_udc *udc, u8 *p_buf, u32 length)
{
u32 i;
- u32 iReadSize = 0;
- union usb_reg_access Temp32;
- union usb_reg_access *pBuf32 = (union usb_reg_access *)pBuf;
+ u32 i_read_size = 0;
+ union usb_reg_access temp_32;
+ union usb_reg_access *p_buf_32 = (union usb_reg_access *)p_buf;
if ((length > 0) && (length < sizeof(u32))) {
- Temp32.dw = _nbu2ss_readl(&udc->p_regs->EP0_READ);
+ temp_32.dw = _nbu2ss_readl(&udc->p_regs->EP0_READ);
for (i = 0 ; i < length ; i++)
- pBuf32->byte.DATA[i] = Temp32.byte.DATA[i];
- iReadSize += length;
+ p_buf_32->byte.DATA[i] = temp_32.byte.DATA[i];
+ i_read_size += length;
}
- return iReadSize;
+ return i_read_size;
}
/*-------------------------------------------------------------------------*/
/* Endpoint 0 IN Transfer (PIO) */
-static int EP0_in_PIO(struct nbu2ss_udc *udc, u8 *pBuf, u32 length)
+static int EP0_in_PIO(struct nbu2ss_udc *udc, u8 *p_buf, u32 length)
{
u32 i;
- u32 iMaxLength = EP0_PACKETSIZE;
- u32 iWordLength = 0;
- u32 iWriteLength = 0;
- union usb_reg_access *pBuf32 = (union usb_reg_access *)pBuf;
+ u32 i_max_length = EP0_PACKETSIZE;
+ u32 i_word_length = 0;
+ u32 i_write_length = 0;
+ union usb_reg_access *p_buf_32 = (union usb_reg_access *)p_buf;
/*------------------------------------------------------------*/
/* Transfer Length */
- if (iMaxLength < length)
- iWordLength = iMaxLength / sizeof(u32);
+ if (i_max_length < length)
+ i_word_length = i_max_length / sizeof(u32);
else
- iWordLength = length / sizeof(u32);
+ i_word_length = length / sizeof(u32);
/*------------------------------------------------------------*/
/* PIO */
- for (i = 0; i < iWordLength; i++) {
- _nbu2ss_writel(&udc->p_regs->EP0_WRITE, pBuf32->dw);
- pBuf32++;
- iWriteLength += sizeof(u32);
+ for (i = 0; i < i_word_length; i++) {
+ _nbu2ss_writel(&udc->p_regs->EP0_WRITE, p_buf_32->dw);
+ p_buf_32++;
+ i_write_length += sizeof(u32);
}
- return iWriteLength;
+ return i_write_length;
}
/*-------------------------------------------------------------------------*/
/* Endpoint 0 IN Transfer (PIO, OverBytes) */
-static int EP0_in_OverBytes(struct nbu2ss_udc *udc, u8 *pBuf, u32 iRemainSize)
+static int ep0_in_overbytes(struct nbu2ss_udc *udc,
+ u8 *p_buf,
+ u32 i_remain_size)
{
u32 i;
- union usb_reg_access Temp32;
- union usb_reg_access *pBuf32 = (union usb_reg_access *)pBuf;
+ union usb_reg_access temp_32;
+ union usb_reg_access *p_buf_32 = (union usb_reg_access *)p_buf;
- if ((iRemainSize > 0) && (iRemainSize < sizeof(u32))) {
- for (i = 0 ; i < iRemainSize ; i++)
- Temp32.byte.DATA[i] = pBuf32->byte.DATA[i];
- _nbu2ss_ep_in_end(udc, 0, Temp32.dw, iRemainSize);
+ if ((i_remain_size > 0) && (i_remain_size < sizeof(u32))) {
+ for (i = 0 ; i < i_remain_size ; i++)
+ temp_32.byte.DATA[i] = p_buf_32->byte.DATA[i];
+ _nbu2ss_ep_in_end(udc, 0, temp_32.dw, i_remain_size);
- return iRemainSize;
+ return i_remain_size;
}
return 0;
struct nbu2ss_req *req
)
{
- u8 *pBuffer; /* IN Data Buffer */
+ u8 *p_buffer; /* IN Data Buffer */
u32 data;
- u32 iRemainSize = 0;
+ u32 i_remain_size = 0;
int result = 0;
/*-------------------------------------------------------------*/
data &= ~(u32)EP0_INAK;
_nbu2ss_writel(&udc->p_regs->EP0_CONTROL, data);
- iRemainSize = req->req.length - req->req.actual;
- pBuffer = (u8 *)req->req.buf;
- pBuffer += req->req.actual;
+ i_remain_size = req->req.length - req->req.actual;
+ p_buffer = (u8 *)req->req.buf;
+ p_buffer += req->req.actual;
/*-------------------------------------------------------------*/
/* Data transfer */
- result = EP0_in_PIO(udc, pBuffer, iRemainSize);
+ result = EP0_in_PIO(udc, p_buffer, i_remain_size);
req->div_len = result;
- iRemainSize -= result;
+ i_remain_size -= result;
- if (iRemainSize == 0) {
+ if (i_remain_size == 0) {
EP0_send_NULL(udc, FALSE);
return result;
}
- if ((iRemainSize < sizeof(u32)) && (result != EP0_PACKETSIZE)) {
- pBuffer += result;
- result += EP0_in_OverBytes(udc, pBuffer, iRemainSize);
+ if ((i_remain_size < sizeof(u32)) && (result != EP0_PACKETSIZE)) {
+ p_buffer += result;
+ result += ep0_in_overbytes(udc, p_buffer, i_remain_size);
req->div_len = result;
}
struct nbu2ss_req *req
)
{
- u8 *pBuffer;
- u32 iRemainSize;
- u32 iRecvLength;
+ u8 *p_buffer;
+ u32 i_remain_size;
+ u32 i_recv_length;
int result = 0;
- int fRcvZero;
+ int f_rcv_zero;
/*-------------------------------------------------------------*/
/* Receive data confirmation */
- iRecvLength = _nbu2ss_readl(&udc->p_regs->EP0_LENGTH) & EP0_LDATA;
- if (iRecvLength != 0) {
- fRcvZero = 0;
+ i_recv_length = _nbu2ss_readl(&udc->p_regs->EP0_LENGTH) & EP0_LDATA;
+ if (i_recv_length != 0) {
+ f_rcv_zero = 0;
- iRemainSize = req->req.length - req->req.actual;
- pBuffer = (u8 *)req->req.buf;
- pBuffer += req->req.actual;
+ i_remain_size = req->req.length - req->req.actual;
+ p_buffer = (u8 *)req->req.buf;
+ p_buffer += req->req.actual;
- result = ep0_out_pio(udc, pBuffer
- , min(iRemainSize, iRecvLength));
+ result = ep0_out_pio(udc, p_buffer
+ , min(i_remain_size, i_recv_length));
if (result < 0)
return result;
req->req.actual += result;
- iRecvLength -= result;
+ i_recv_length -= result;
- if ((iRecvLength > 0) && (iRecvLength < sizeof(u32))) {
- pBuffer += result;
- iRemainSize -= result;
+ if ((i_recv_length > 0) && (i_recv_length < sizeof(u32))) {
+ p_buffer += result;
+ i_remain_size -= result;
- result = EP0_out_OverBytes(udc, pBuffer
- , min(iRemainSize, iRecvLength));
+ result = ep0_out_overbytes(udc, p_buffer
+ , min(i_remain_size, i_recv_length));
req->req.actual += result;
}
} else {
- fRcvZero = 1;
+ f_rcv_zero = 1;
}
/*-------------------------------------------------------------*/
return -EOVERFLOW;
}
- if (fRcvZero != 0) {
- iRemainSize = _nbu2ss_readl(&udc->p_regs->EP0_CONTROL);
- if (iRemainSize & EP0_ONAK) {
+ if (f_rcv_zero != 0) {
+ i_remain_size = _nbu2ss_readl(&udc->p_regs->EP0_CONTROL);
+ if (i_remain_size & EP0_ONAK) {
/*---------------------------------------------------*/
/* NACK release */
_nbu2ss_bitclr(&udc->p_regs->EP0_CONTROL, EP0_ONAK);
u32 length
)
{
- dma_addr_t pBuffer;
+ dma_addr_t p_buffer;
u32 mpkt;
u32 lmpkt;
u32 dmacnt;
return 1; /* DMA is forwarded */
req->dma_flag = TRUE;
- pBuffer = req->req.dma;
- pBuffer += req->req.actual;
+ p_buffer = req->req.dma;
+ p_buffer += req->req.actual;
/* DMA Address */
- _nbu2ss_writel(&preg->EP_DCR[num].EP_TADR, (u32)pBuffer);
+ _nbu2ss_writel(&preg->EP_DCR[num].EP_TADR, (u32)p_buffer);
/* Number of transfer packets */
- mpkt = _nbu2ss_readl(&preg->EP_REGS[num].EP_PCKT_ADRS) & EPn_MPKT;
+ mpkt = _nbu2ss_readl(&preg->EP_REGS[num].EP_PCKT_ADRS) & EPN_MPKT;
dmacnt = length / mpkt;
lmpkt = (length % mpkt) & ~(u32)0x03;
data = mpkt | (lmpkt << 16);
_nbu2ss_writel(&preg->EP_DCR[num].EP_DCR2, data);
- data = ((dmacnt & 0xff) << 16) | DCR1_EPn_DIR0 | DCR1_EPn_REQEN;
+ data = ((dmacnt & 0xff) << 16) | DCR1_EPN_DIR0 | DCR1_EPN_REQEN;
_nbu2ss_writel(&preg->EP_DCR[num].EP_DCR1, data);
if (burst == 0) {
_nbu2ss_writel(&preg->EP_REGS[num].EP_LEN_DCNT, 0);
- _nbu2ss_bitclr(&preg->EP_REGS[num].EP_DMA_CTRL, EPn_BURST_SET);
+ _nbu2ss_bitclr(&preg->EP_REGS[num].EP_DMA_CTRL, EPN_BURST_SET);
} else {
_nbu2ss_writel(&preg->EP_REGS[num].EP_LEN_DCNT
, (dmacnt << 16));
- _nbu2ss_bitset(&preg->EP_REGS[num].EP_DMA_CTRL, EPn_BURST_SET);
+ _nbu2ss_bitset(&preg->EP_REGS[num].EP_DMA_CTRL, EPN_BURST_SET);
}
- _nbu2ss_bitset(&preg->EP_REGS[num].EP_DMA_CTRL, EPn_DMA_EN);
+ _nbu2ss_bitset(&preg->EP_REGS[num].EP_DMA_CTRL, EPN_DMA_EN);
result = length & ~(u32)0x03;
req->div_len = result;
u32 length
)
{
- u8 *pBuffer;
+ u8 *p_buffer;
u32 i;
u32 data;
- u32 iWordLength;
- union usb_reg_access Temp32;
- union usb_reg_access *pBuf32;
+ u32 i_word_length;
+ union usb_reg_access temp_32;
+ union usb_reg_access *p_buf_32;
int result = 0;
struct fc_regs *preg = udc->p_regs;
if (length == 0)
return 0;
- pBuffer = (u8 *)req->req.buf;
- pBuf32 = (union usb_reg_access *)(pBuffer + req->req.actual);
+ p_buffer = (u8 *)req->req.buf;
+ p_buf_32 = (union usb_reg_access *)(p_buffer + req->req.actual);
- iWordLength = length / sizeof(u32);
- if (iWordLength > 0) {
+ i_word_length = length / sizeof(u32);
+ if (i_word_length > 0) {
/*---------------------------------------------------------*/
/* Copy of every four bytes */
- for (i = 0; i < iWordLength; i++) {
- pBuf32->dw =
+ for (i = 0; i < i_word_length; i++) {
+ p_buf_32->dw =
_nbu2ss_readl(&preg->EP_REGS[ep->epnum - 1].EP_READ);
- pBuf32++;
+ p_buf_32++;
}
- result = iWordLength * sizeof(u32);
+ result = i_word_length * sizeof(u32);
}
data = length - result;
if (data > 0) {
/*---------------------------------------------------------*/
/* Copy of fraction byte */
- Temp32.dw = _nbu2ss_readl(&preg->EP_REGS[ep->epnum - 1].EP_READ);
+ temp_32.dw =
+ _nbu2ss_readl(&preg->EP_REGS[ep->epnum - 1].EP_READ);
for (i = 0 ; i < data ; i++)
- pBuf32->byte.DATA[i] = Temp32.byte.DATA[i];
+ p_buf_32->byte.DATA[i] = temp_32.byte.DATA[i];
result += data;
}
)
{
u32 num;
- u32 iBufSize;
+ u32 i_buf_size;
int nret = 1;
if (ep->epnum == 0)
num = ep->epnum - 1;
- iBufSize = min((req->req.length - req->req.actual), data_size);
+ i_buf_size = min((req->req.length - req->req.actual), data_size);
if ((ep->ep_type != USB_ENDPOINT_XFER_INT) && (req->req.dma != 0) &&
- (iBufSize >= sizeof(u32))) {
- nret = _nbu2ss_out_dma(udc, req, num, iBufSize);
+ (i_buf_size >= sizeof(u32))) {
+ nret = _nbu2ss_out_dma(udc, req, num, i_buf_size);
} else {
- iBufSize = min_t(u32, iBufSize, ep->ep.maxpacket);
- nret = _nbu2ss_epn_out_pio(udc, ep, req, iBufSize);
+ i_buf_size = min_t(u32, i_buf_size, ep->ep.maxpacket);
+ nret = _nbu2ss_epn_out_pio(udc, ep, req, i_buf_size);
}
return nret;
)
{
u32 num;
- u32 iRecvLength;
+ u32 i_recv_length;
int result = 1;
struct fc_regs *preg = udc->p_regs;
/*-------------------------------------------------------------*/
/* Receive Length */
- iRecvLength
- = _nbu2ss_readl(&preg->EP_REGS[num].EP_LEN_DCNT) & EPn_LDATA;
+ i_recv_length
+ = _nbu2ss_readl(&preg->EP_REGS[num].EP_LEN_DCNT) & EPN_LDATA;
- if (iRecvLength != 0) {
- result = _nbu2ss_epn_out_data(udc, ep, req, iRecvLength);
- if (iRecvLength < ep->ep.maxpacket) {
- if (iRecvLength == result) {
+ if (i_recv_length != 0) {
+ result = _nbu2ss_epn_out_data(udc, ep, req, i_recv_length);
+ if (i_recv_length < ep->ep.maxpacket) {
+ if (i_recv_length == result) {
req->req.actual += result;
result = 0;
}
u32 length
)
{
- dma_addr_t pBuffer;
+ dma_addr_t p_buffer;
u32 mpkt; /* MaxPacketSize */
u32 lmpkt; /* Last Packet Data Size */
u32 dmacnt; /* IN Data Size */
- u32 iWriteLength;
+ u32 i_write_length;
u32 data;
int result = -EINVAL;
struct fc_regs *preg = udc->p_regs;
req->dma_flag = TRUE;
/* MAX Packet Size */
- mpkt = _nbu2ss_readl(&preg->EP_REGS[num].EP_PCKT_ADRS) & EPn_MPKT;
+ mpkt = _nbu2ss_readl(&preg->EP_REGS[num].EP_PCKT_ADRS) & EPN_MPKT;
if ((DMA_MAX_COUNT * mpkt) < length)
- iWriteLength = DMA_MAX_COUNT * mpkt;
+ i_write_length = DMA_MAX_COUNT * mpkt;
else
- iWriteLength = length;
+ i_write_length = length;
/*------------------------------------------------------------*/
/* Number of transmission packets */
- if (mpkt < iWriteLength) {
- dmacnt = iWriteLength / mpkt;
- lmpkt = (iWriteLength % mpkt) & ~(u32)0x3;
+ if (mpkt < i_write_length) {
+ dmacnt = i_write_length / mpkt;
+ lmpkt = (i_write_length % mpkt) & ~(u32)0x3;
if (lmpkt != 0)
dmacnt++;
else
} else {
dmacnt = 1;
- lmpkt = iWriteLength & ~(u32)0x3;
+ lmpkt = i_write_length & ~(u32)0x3;
}
/* Packet setting */
_nbu2ss_writel(&preg->EP_DCR[num].EP_DCR2, data);
/* Address setting */
- pBuffer = req->req.dma;
- pBuffer += req->req.actual;
- _nbu2ss_writel(&preg->EP_DCR[num].EP_TADR, (u32)pBuffer);
+ p_buffer = req->req.dma;
+ p_buffer += req->req.actual;
+ _nbu2ss_writel(&preg->EP_DCR[num].EP_TADR, (u32)p_buffer);
/* Packet and DMA setting */
- data = ((dmacnt & 0xff) << 16) | DCR1_EPn_REQEN;
+ data = ((dmacnt & 0xff) << 16) | DCR1_EPN_REQEN;
_nbu2ss_writel(&preg->EP_DCR[num].EP_DCR1, data);
/* Packet setting of EPC */
_nbu2ss_writel(&preg->EP_REGS[num].EP_LEN_DCNT, data);
/*DMA setting of EPC */
- _nbu2ss_bitset(&preg->EP_REGS[num].EP_DMA_CTRL, EPn_DMA_EN);
+ _nbu2ss_bitset(&preg->EP_REGS[num].EP_DMA_CTRL, EPN_DMA_EN);
- result = iWriteLength & ~(u32)0x3;
+ result = i_write_length & ~(u32)0x3;
req->div_len = result;
return result;
u32 length
)
{
- u8 *pBuffer;
+ u8 *p_buffer;
u32 i;
u32 data;
- u32 iWordLength;
- union usb_reg_access Temp32;
- union usb_reg_access *pBuf32 = NULL;
+ u32 i_word_length;
+ union usb_reg_access temp_32;
+ union usb_reg_access *p_buf_32 = NULL;
int result = 0;
struct fc_regs *preg = udc->p_regs;
return 1; /* DMA is forwarded */
if (length > 0) {
- pBuffer = (u8 *)req->req.buf;
- pBuf32 = (union usb_reg_access *)(pBuffer + req->req.actual);
+ p_buffer = (u8 *)req->req.buf;
+ p_buf_32 = (union usb_reg_access *)(p_buffer + req->req.actual);
- iWordLength = length / sizeof(u32);
- if (iWordLength > 0) {
- for (i = 0; i < iWordLength; i++) {
+ i_word_length = length / sizeof(u32);
+ if (i_word_length > 0) {
+ for (i = 0; i < i_word_length; i++) {
_nbu2ss_writel(
&preg->EP_REGS[ep->epnum - 1].EP_WRITE
- , pBuf32->dw
+ , p_buf_32->dw
);
- pBuf32++;
+ p_buf_32++;
}
- result = iWordLength * sizeof(u32);
+ result = i_word_length * sizeof(u32);
}
}
if (result != ep->ep.maxpacket) {
data = length - result;
- Temp32.dw = 0;
+ temp_32.dw = 0;
for (i = 0 ; i < data ; i++)
- Temp32.byte.DATA[i] = pBuf32->byte.DATA[i];
+ temp_32.byte.DATA[i] = p_buf_32->byte.DATA[i];
- _nbu2ss_ep_in_end(udc, ep->epnum, Temp32.dw, data);
+ _nbu2ss_ep_in_end(udc, ep->epnum, temp_32.dw, data);
result += data;
}
)
{
u32 num;
- u32 iBufSize;
+ u32 i_buf_size;
int result = 0;
u32 status;
/*-------------------------------------------------------------*/
/* State confirmation of FIFO */
if (req->req.actual == 0) {
- if ((status & EPn_IN_EMPTY) == 0)
+ if ((status & EPN_IN_EMPTY) == 0)
return 1; /* Not Empty */
} else {
- if ((status & EPn_IN_FULL) != 0)
+ if ((status & EPN_IN_FULL) != 0)
return 1; /* Not Empty */
}
/*-------------------------------------------------------------*/
/* Start transfer */
- iBufSize = req->req.length - req->req.actual;
- if (iBufSize > 0)
- result = _nbu2ss_epn_in_data(udc, ep, req, iBufSize);
+ i_buf_size = req->req.length - req->req.actual;
+ if (i_buf_size > 0)
+ result = _nbu2ss_epn_in_data(udc, ep, req, i_buf_size);
else if (req->req.length == 0)
_nbu2ss_zero_len_pkt(udc, ep->epnum);
}
} else {
- /* EPn */
+ /* EPN */
if (ep->direct == USB_DIR_OUT) {
/* OUT */
if (!bflag)
length = _nbu2ss_readl(
&ep->udc->p_regs->EP_REGS[ep->epnum - 1].EP_LEN_DCNT);
- length &= EPn_LDATA;
+ length &= EPN_LDATA;
if (length < ep->ep.maxpacket)
bflag = TRUE;
}
num = (ep_adrs & 0x7F) - 1;
if (ep_adrs & USB_DIR_IN)
- data = EPn_IPIDCLR;
+ data = EPN_IPIDCLR;
else
- data = EPn_BCLR | EPn_OPIDCLR;
+ data = EPN_BCLR | EPN_OPIDCLR;
_nbu2ss_bitset(&udc->p_regs->EP_REGS[num].EP_CONTROL, data);
}
ep->halted = TRUE;
if (ep_adrs & USB_DIR_IN)
- data = EPn_BCLR | EPn_ISTL;
+ data = EPN_BCLR | EPN_ISTL;
else
- data = EPn_OSTL_EN | EPn_OSTL;
+ data = EPN_OSTL_EN | EPN_OSTL;
_nbu2ss_bitset(&preg->EP_REGS[num].EP_CONTROL, data);
} else {
ep->stalled = FALSE;
if (ep_adrs & USB_DIR_IN) {
_nbu2ss_bitclr(&preg->EP_REGS[num].EP_CONTROL
- , EPn_ISTL);
+ , EPN_ISTL);
} else {
data =
_nbu2ss_readl(&preg->EP_REGS[num].EP_CONTROL);
- data &= ~EPn_OSTL;
- data |= EPn_OSTL_EN;
+ data &= ~EPN_OSTL;
+ data |= EPN_OSTL_EN;
_nbu2ss_writel(&preg->EP_REGS[num].EP_CONTROL
, data);
} else {
data = _nbu2ss_readl(&preg->EP_REGS[epnum - 1].EP_CONTROL);
- if ((data & EPn_EN) == 0)
+ if ((data & EPN_EN) == 0)
return -1;
if (ep_adrs & USB_ENDPOINT_DIR_MASK)
- bit_data = EPn_ISTL;
+ bit_data = EPN_ISTL;
else
- bit_data = EPn_OSTL;
+ bit_data = EPN_OSTL;
}
if ((data & bit_data) == 0)
regdata = _nbu2ss_readl(
&preg->EP_REGS[ep->epnum - 1].EP_STATUS);
- if ((regdata & EPn_IN_DATA) == 0)
+ if ((regdata & EPN_IN_DATA) == 0)
break;
mdelay(1);
/*-------------------------------------------------------------------------*/
static int std_req_set_configuration(struct nbu2ss_udc *udc)
{
- u32 ConfigValue = (u32)(udc->ctrl.wValue & 0x00ff);
+ u32 config_value = (u32)(udc->ctrl.wValue & 0x00ff);
if ((udc->ctrl.wIndex != 0x0000) ||
(udc->ctrl.wLength != 0x0000) ||
return -EINVAL;
}
- udc->curr_config = ConfigValue;
+ udc->curr_config = config_value;
- if (ConfigValue > 0) {
+ if (config_value > 0) {
_nbu2ss_bitset(&udc->p_regs->USB_CONTROL, CONF);
udc->devstate = USB_STATE_CONFIGURED;
status =
_nbu2ss_readl(&preg->EP_REGS[ep->epnum - 1].EP_STATUS);
- if ((status & EPn_IN_FULL) == 0) {
+ if ((status & EPN_IN_FULL) == 0) {
/*-----------------------------------------*/
/* 0 Length Packet */
req->zero = false;
}
ep_dmacnt = _nbu2ss_readl(&preg->EP_REGS[num].EP_LEN_DCNT)
- & EPn_DMACNT;
+ & EPN_DMACNT;
ep_dmacnt >>= 16;
for (i = 0; i < EPC_PLL_LOCK_COUNT; i++) {
dmacnt = _nbu2ss_readl(&preg->EP_DCR[num].EP_DCR1)
- & DCR1_EPn_DMACNT;
+ & DCR1_EPN_DMACNT;
dmacnt >>= 16;
if (ep_dmacnt == dmacnt)
break;
}
- _nbu2ss_bitclr(&preg->EP_DCR[num].EP_DCR1, DCR1_EPn_REQEN);
+ _nbu2ss_bitclr(&preg->EP_DCR[num].EP_DCR1, DCR1_EPN_REQEN);
if (dmacnt != 0) {
mpkt = ep->ep.maxpacket;
return;
}
- if (status & EPn_OUT_END_INT) {
- status &= ~EPn_OUT_INT;
+ if (status & EPN_OUT_END_INT) {
+ status &= ~EPN_OUT_INT;
_nbu2ss_epn_out_dma_int(udc, ep, req);
}
- if (status & EPn_OUT_INT)
+ if (status & EPN_OUT_INT)
_nbu2ss_epn_out_int(udc, ep, req);
- if (status & EPn_IN_END_INT) {
- status &= ~EPn_IN_INT;
+ if (status & EPN_IN_END_INT) {
+ status &= ~EPN_IN_INT;
_nbu2ss_epn_in_dma_int(udc, ep, req);
}
- if (status & EPn_IN_INT)
+ if (status & EPN_IN_INT)
_nbu2ss_epn_in_int(udc, ep, req);
}
_nbu2ss_bitset(&p->EP0_CONTROL, EP0_BCLR);
} else {
- /* EPn */
+ /* EPN */
_nbu2ss_ep_dma_abort(udc, ep);
- _nbu2ss_bitset(&p->EP_REGS[ep->epnum - 1].EP_CONTROL, EPn_BCLR);
+ _nbu2ss_bitset(&p->EP_REGS[ep->epnum - 1].EP_CONTROL, EPN_BCLR);
}
}
suspend_flag = 1;
}
- if (status & EPn_INT) {
+ if (status & EPN_INT) {
/* EP INT */
int_bit = status >> 8;
}
req = container_of(_req, struct nbu2ss_req, req);
- if (unlikely(!_req->complete || !_req->buf || !list_empty(&req->queue))) {
+ if (unlikely(!_req->complete ||
+ !_req->buf ||
+ !list_empty(&req->queue))) {
if (!_req->complete)
pr_err("udc: %s --- !_req->complete\n", __func__);
} else {
data = _nbu2ss_readl(&preg->EP_REGS[ep->epnum - 1].EP_LEN_DCNT)
- & EPn_LDATA;
+ & EPN_LDATA;
}
spin_unlock_irqrestore(&udc->lock, flags);
/*------- (0x001C) Setup Data 1 Register */
/*------- (0x0020) USB Interrupt Status Register */
-#define EPn_INT 0x00FFFF00
+#define EPN_INT 0x00FFFF00
#define EP15_INT BIT23
#define EP14_INT BIT22
#define EP13_INT BIT21
/*------- (0x0038) EP0 Read Register */
/*------- (0x003C) EP0 Write Register */
-/*------- (0x0040:) EPn Control Register */
-#define EPn_EN BIT31
-#define EPn_BUF_TYPE BIT30
-#define EPn_BUF_SINGLE BIT30
-
-#define EPn_DIR0 BIT26
-#define EPn_MODE (BIT25 + BIT24)
-#define EPn_BULK 0
-#define EPn_INTERRUPT BIT24
-#define EPn_ISO BIT25
-
-#define EPn_OVERSEL BIT17
-#define EPn_AUTO BIT16
-
-#define EPn_IPIDCLR BIT11
-#define EPn_OPIDCLR BIT10
-#define EPn_BCLR BIT09
-#define EPn_CBCLR BIT08
-#define EPn_DEND BIT07
-#define EPn_DW (BIT06 + BIT05)
-#define EPn_DW4 0
-#define EPn_DW3 (BIT06 + BIT05)
-#define EPn_DW2 BIT06
-#define EPn_DW1 BIT05
-
-#define EPn_OSTL_EN BIT04
-#define EPn_ISTL BIT03
-#define EPn_OSTL BIT02
-
-#define EPn_ONAK BIT00
-
-/*------- (0x0044:) EPn Status Register */
-#define EPn_ISO_PIDERR BIT29 /* R */
-#define EPn_OPID BIT28 /* R */
-#define EPn_OUT_NOTKN BIT27 /* R */
-#define EPn_ISO_OR BIT26 /* R */
-
-#define EPn_ISO_CRC BIT24 /* R */
-#define EPn_OUT_END_INT BIT23 /* RW */
-#define EPn_OUT_OR_INT BIT22 /* RW */
-#define EPn_OUT_NAK_ERR_INT BIT21 /* RW */
-#define EPn_OUT_STALL_INT BIT20 /* RW */
-#define EPn_OUT_INT BIT19 /* RW */
-#define EPn_OUT_NULL_INT BIT18 /* RW */
-#define EPn_OUT_FULL BIT17 /* R */
-#define EPn_OUT_EMPTY BIT16 /* R */
-
-#define EPn_IPID BIT10 /* R */
-#define EPn_IN_NOTKN BIT09 /* R */
-#define EPn_ISO_UR BIT08 /* R */
-#define EPn_IN_END_INT BIT07 /* RW */
-
-#define EPn_IN_NAK_ERR_INT BIT05 /* RW */
-#define EPn_IN_STALL_INT BIT04 /* RW */
-#define EPn_IN_INT BIT03 /* RW */
-#define EPn_IN_DATA BIT02 /* R */
-#define EPn_IN_FULL BIT01 /* R */
-#define EPn_IN_EMPTY BIT00 /* R */
-
-#define EPn_INT_EN \
- (EPn_OUT_END_INT | EPn_OUT_INT | EPn_IN_END_INT | EPn_IN_INT)
-
-/*------- (0x0048:) EPn Interrupt Enable Register */
-#define EPn_OUT_END_EN BIT23 /* RW */
-#define EPn_OUT_OR_EN BIT22 /* RW */
-#define EPn_OUT_NAK_ERR_EN BIT21 /* RW */
-#define EPn_OUT_STALL_EN BIT20 /* RW */
-#define EPn_OUT_EN BIT19 /* RW */
-#define EPn_OUT_NULL_EN BIT18 /* RW */
-
-#define EPn_IN_END_EN BIT07 /* RW */
-
-#define EPn_IN_NAK_ERR_EN BIT05 /* RW */
-#define EPn_IN_STALL_EN BIT04 /* RW */
-#define EPn_IN_EN BIT03 /* RW */
-
-/*------- (0x004C:) EPn Interrupt Enable Register */
-#define EPn_STOP_MODE BIT11
-#define EPn_DEND_SET BIT10
-#define EPn_BURST_SET BIT09
-#define EPn_STOP_SET BIT08
-
-#define EPn_DMA_EN BIT04
-
-#define EPn_DMAMODE0 BIT00
-
-/*------- (0x0050:) EPn MaxPacket & BaseAddress Register */
-#define EPn_BASEAD 0x1FFF0000
-#define EPn_MPKT 0x000007FF
-
-/*------- (0x0054:) EPn Length & DMA Count Register */
-#define EPn_DMACNT 0x01FF0000
-#define EPn_LDATA 0x000007FF
-
-/*------- (0x0058:) EPn Read Register */
-/*------- (0x005C:) EPn Write Register */
+/*------- (0x0040:) EPN Control Register */
+#define EPN_EN BIT31
+#define EPN_BUF_TYPE BIT30
+#define EPN_BUF_SINGLE BIT30
+
+#define EPN_DIR0 BIT26
+#define EPN_MODE (BIT25 + BIT24)
+#define EPN_BULK 0
+#define EPN_INTERRUPT BIT24
+#define EPN_ISO BIT25
+
+#define EPN_OVERSEL BIT17
+#define EPN_AUTO BIT16
+
+#define EPN_IPIDCLR BIT11
+#define EPN_OPIDCLR BIT10
+#define EPN_BCLR BIT09
+#define EPN_CBCLR BIT08
+#define EPN_DEND BIT07
+#define EPN_DW (BIT06 + BIT05)
+#define EPN_DW4 0
+#define EPN_DW3 (BIT06 + BIT05)
+#define EPN_DW2 BIT06
+#define EPN_DW1 BIT05
+
+#define EPN_OSTL_EN BIT04
+#define EPN_ISTL BIT03
+#define EPN_OSTL BIT02
+
+#define EPN_ONAK BIT00
+
+/*------- (0x0044:) EPN Status Register */
+#define EPN_ISO_PIDERR BIT29 /* R */
+#define EPN_OPID BIT28 /* R */
+#define EPN_OUT_NOTKN BIT27 /* R */
+#define EPN_ISO_OR BIT26 /* R */
+
+#define EPN_ISO_CRC BIT24 /* R */
+#define EPN_OUT_END_INT BIT23 /* RW */
+#define EPN_OUT_OR_INT BIT22 /* RW */
+#define EPN_OUT_NAK_ERR_INT BIT21 /* RW */
+#define EPN_OUT_STALL_INT BIT20 /* RW */
+#define EPN_OUT_INT BIT19 /* RW */
+#define EPN_OUT_NULL_INT BIT18 /* RW */
+#define EPN_OUT_FULL BIT17 /* R */
+#define EPN_OUT_EMPTY BIT16 /* R */
+
+#define EPN_IPID BIT10 /* R */
+#define EPN_IN_NOTKN BIT09 /* R */
+#define EPN_ISO_UR BIT08 /* R */
+#define EPN_IN_END_INT BIT07 /* RW */
+
+#define EPN_IN_NAK_ERR_INT BIT05 /* RW */
+#define EPN_IN_STALL_INT BIT04 /* RW */
+#define EPN_IN_INT BIT03 /* RW */
+#define EPN_IN_DATA BIT02 /* R */
+#define EPN_IN_FULL BIT01 /* R */
+#define EPN_IN_EMPTY BIT00 /* R */
+
+#define EPN_INT_EN \
+ (EPN_OUT_END_INT | EPN_OUT_INT | EPN_IN_END_INT | EPN_IN_INT)
+
+/*------- (0x0048:) EPN Interrupt Enable Register */
+#define EPN_OUT_END_EN BIT23 /* RW */
+#define EPN_OUT_OR_EN BIT22 /* RW */
+#define EPN_OUT_NAK_ERR_EN BIT21 /* RW */
+#define EPN_OUT_STALL_EN BIT20 /* RW */
+#define EPN_OUT_EN BIT19 /* RW */
+#define EPN_OUT_NULL_EN BIT18 /* RW */
+
+#define EPN_IN_END_EN BIT07 /* RW */
+
+#define EPN_IN_NAK_ERR_EN BIT05 /* RW */
+#define EPN_IN_STALL_EN BIT04 /* RW */
+#define EPN_IN_EN BIT03 /* RW */
+
+/*------- (0x004C:) EPN Interrupt Enable Register */
+#define EPN_STOP_MODE BIT11
+#define EPN_DEND_SET BIT10
+#define EPN_BURST_SET BIT09
+#define EPN_STOP_SET BIT08
+
+#define EPN_DMA_EN BIT04
+
+#define EPN_DMAMODE0 BIT00
+
+/*------- (0x0050:) EPN MaxPacket & BaseAddress Register */
+#define EPN_BASEAD 0x1FFF0000
+#define EPN_MPKT 0x000007FF
+
+/*------- (0x0054:) EPN Length & DMA Count Register */
+#define EPN_DMACNT 0x01FF0000
+#define EPN_LDATA 0x000007FF
+
+/*------- (0x0058:) EPN Read Register */
+/*------- (0x005C:) EPN Write Register */
/*------- (0x1000) AHBSCTR Register */
#define WAIT_MODE BIT00
#define EP_AVAILABLE 0xFFFF0000 /* R */
#define DMA_AVAILABLE 0x0000FFFF /* R */
-/*------- (0x1110:) EPnDCR1 Register */
-#define DCR1_EPn_DMACNT 0x00FF0000 /* RW */
+/*------- (0x1110:) EPNDCR1 Register */
+#define DCR1_EPN_DMACNT 0x00FF0000 /* RW */
-#define DCR1_EPn_DIR0 BIT01 /* RW */
-#define DCR1_EPn_REQEN BIT00 /* RW */
+#define DCR1_EPN_DIR0 BIT01 /* RW */
+#define DCR1_EPN_REQEN BIT00 /* RW */
-/*------- (0x1114:) EPnDCR2 Register */
-#define DCR2_EPn_LMPKT 0x07FF0000 /* RW */
+/*------- (0x1114:) EPNDCR2 Register */
+#define DCR2_EPN_LMPKT 0x07FF0000 /* RW */
-#define DCR2_EPn_MPKT 0x000007FF /* RW */
+#define DCR2_EPN_MPKT 0x000007FF /* RW */
-/*------- (0x1118:) EPnTADR Register */
-#define EPn_TADR 0xFFFFFFFF /* RW */
+/*------- (0x1118:) EPNTADR Register */
+#define EPN_TADR 0xFFFFFFFF /* RW */
/*===========================================================================*/
/* Struct */
u32 USB_ADDRESS; /* (0x0008) USB Address */
u32 UTMI_CHARACTER_1; /* (0x000C) UTMI Setting */
u32 TEST_CONTROL; /* (0x0010) TEST Control */
- u32 Reserved_14; /* (0x0014) Reserved */
+ u32 reserved_14; /* (0x0014) Reserved */
u32 SETUP_DATA0; /* (0x0018) Setup Data0 */
u32 SETUP_DATA1; /* (0x001C) Setup Data1 */
u32 USB_INT_STA; /* (0x0020) USB Interrupt Status */
struct ep_regs EP_REGS[REG_EP_NUM]; /* Endpoint Register */
- u8 Reserved220[0x1000 - 0x220]; /* (0x0220:0x0FFF) Reserved */
+ u8 reserved_220[0x1000 - 0x220]; /* (0x0220:0x0FFF) Reserved */
u32 AHBSCTR; /* (0x1000) AHBSCTR */
u32 AHBMCTR; /* (0x1004) AHBMCTR */
u32 EPCTR; /* (0x1010) EPCTR */
u32 USBF_EPTEST; /* (0x1014) USBF_EPTEST */
- u8 Reserved1018[0x20 - 0x18]; /* (0x1018:0x101F) Reserved */
+ u8 reserved_1018[0x20 - 0x18]; /* (0x1018:0x101F) Reserved */
u32 USBSSVER; /* (0x1020) USBSSVER */
u32 USBSSCONF; /* (0x1024) USBSSCONF */
- u8 Reserved1028[0x110 - 0x28]; /* (0x1028:0x110F) Reserved */
+ u8 reserved_1028[0x110 - 0x28]; /* (0x1028:0x110F) Reserved */
struct ep_dcr EP_DCR[REG_EP_NUM]; /* */
- u8 Reserved1200[0x1000 - 0x200]; /* Reserved */
+ u8 reserved_1200[0x1000 - 0x200]; /* Reserved */
} __aligned(32);
#define EP0_PACKETSIZE 64
#define EP_PACKETSIZE 1024
-/* EPn RAM SIZE */
+/* EPN RAM SIZE */
#define D_RAM_SIZE_CTRL 64
-/* EPn Bulk Endpoint Max Packet Size */
+/* EPN Bulk Endpoint Max Packet Size */
#define D_FS_RAM_SIZE_BULK 64
#define D_HS_RAM_SIZE_BULK 512
continue;
write_pos = &convert_buf[(y + j) * xres + x + i];
coeff = diffusing_matrix[i][j];
- if (-1 == coeff)
+ if (-1 == coeff) {
/* pixel itself */
*write_pos = pixel;
- else {
+ } else {
signed short p = *write_pos + error * coeff;
if (p > WHITE)
* OP0 OP1 CP0 CP1 CP2 CP3 CP4 MP0 MP1 MP2 MP3 MP4 MP5 CGM0 CGM1
* ON0 ON1 CN0 CN1 CN2 CN3 CN4 MN0 MN1 MN2 MN3 MN4 MN5 XXXX GC
*/
-#define CURVE(num, idx) curves[num * par->gamma.num_values + idx]
+#define CURVE(num, idx) curves[(num) * par->gamma.num_values + (idx)]
static int set_gamma(struct fbtft_par *par, u32 *curves)
{
unsigned long mask[] = {
src++;
}
tmp |= ((*src & 0x0100) ? 1 : 0);
- *(u64 *)dst = cpu_to_be64(tmp);
+ *(__be64 *)dst = cpu_to_be64(tmp);
dst += 8;
*dst++ = (u8)(*src++ & 0x00FF);
added++;
#include <linux/interrupt.h>
#include <linux/msi.h>
#include <linux/kthread.h>
+#include <linux/iommu.h>
#include "../../fsl-mc/include/mc.h"
-#include "../../fsl-mc/include/mc-sys.h"
#include "dpaa2-eth.h"
/* CREATE_TRACE_POINTS only needs to be defined once. Other dpa files
const char dpaa2_eth_drv_version[] = "0.1";
+static void *dpaa2_iova_to_virt(struct iommu_domain *domain,
+ dma_addr_t iova_addr)
+{
+ phys_addr_t phys_addr;
+
+ phys_addr = domain ? iommu_iova_to_phys(domain, iova_addr) : iova_addr;
+
+ return phys_to_virt(phys_addr);
+}
+
static void validate_rx_csum(struct dpaa2_eth_priv *priv,
u32 fd_status,
struct sk_buff *skb)
sgt = vaddr + dpaa2_fd_get_offset(fd);
for (i = 0; i < DPAA2_ETH_MAX_SG_ENTRIES; i++) {
addr = dpaa2_sg_get_addr(&sgt[i]);
+ sg_vaddr = dpaa2_iova_to_virt(priv->iommu_domain, addr);
dma_unmap_single(dev, addr, DPAA2_ETH_RX_BUF_SIZE,
DMA_FROM_DEVICE);
- sg_vaddr = phys_to_virt(addr);
skb_free_frag(sg_vaddr);
-
if (dpaa2_sg_is_final(&sgt[i]))
break;
}
/* Get the address and length from the S/G entry */
sg_addr = dpaa2_sg_get_addr(sge);
+ sg_vaddr = dpaa2_iova_to_virt(priv->iommu_domain, sg_addr);
dma_unmap_single(dev, sg_addr, DPAA2_ETH_RX_BUF_SIZE,
DMA_FROM_DEVICE);
- sg_vaddr = phys_to_virt(sg_addr);
sg_length = dpaa2_sg_get_len(sge);
if (i == 0) {
struct dpaa2_eth_drv_stats *percpu_extras;
struct device *dev = priv->net_dev->dev.parent;
struct dpaa2_fas *fas;
+ void *buf_data;
u32 status = 0;
/* Tracing point */
trace_dpaa2_rx_fd(priv->net_dev, fd);
+ vaddr = dpaa2_iova_to_virt(priv->iommu_domain, addr);
dma_unmap_single(dev, addr, DPAA2_ETH_RX_BUF_SIZE, DMA_FROM_DEVICE);
- vaddr = phys_to_virt(addr);
- prefetch(vaddr + priv->buf_layout.private_data_size);
- prefetch(vaddr + dpaa2_fd_get_offset(fd));
+ fas = dpaa2_get_fas(vaddr);
+ prefetch(fas);
+ buf_data = vaddr + dpaa2_fd_get_offset(fd);
+ prefetch(buf_data);
percpu_stats = this_cpu_ptr(priv->percpu_stats);
percpu_extras = this_cpu_ptr(priv->percpu_extras);
if (fd_format == dpaa2_fd_single) {
skb = build_linear_skb(priv, ch, fd, vaddr);
} else if (fd_format == dpaa2_fd_sg) {
- struct dpaa2_sg_entry *sgt =
- vaddr + dpaa2_fd_get_offset(fd);
- skb = build_frag_skb(priv, ch, sgt);
+ skb = build_frag_skb(priv, ch, buf_data);
skb_free_frag(vaddr);
percpu_extras->rx_sg_frames++;
percpu_extras->rx_sg_bytes += dpaa2_fd_get_len(fd);
/* Check if we need to validate the L4 csum */
if (likely(dpaa2_fd_get_frc(fd) & DPAA2_FD_FRC_FASV)) {
- fas = (struct dpaa2_fas *)
- (vaddr + priv->buf_layout.private_data_size);
status = le32_to_cpu(fas->status);
validate_rx_csum(priv, status, skb);
}
percpu_stats->rx_packets++;
percpu_stats->rx_bytes += dpaa2_fd_get_len(fd);
- if (priv->net_dev->features & NETIF_F_GRO)
- napi_gro_receive(napi, skb);
- else
- netif_receive_skb(skb);
+ napi_gro_receive(napi, skb);
return;
{
struct device *dev = priv->net_dev->dev.parent;
void *sgt_buf = NULL;
- void *hwa;
dma_addr_t addr;
int nr_frags = skb_shinfo(skb)->nr_frags;
struct dpaa2_sg_entry *sgt;
int num_sg;
int num_dma_bufs;
struct dpaa2_eth_swa *swa;
+ struct dpaa2_fas *fas;
/* Create and map scatterlist.
* We don't advertise NETIF_F_FRAGLIST, so skb_to_sgvec() will not have
sg_init_table(scl, nr_frags + 1);
num_sg = skb_to_sgvec(skb, scl, 0, skb->len);
- num_dma_bufs = dma_map_sg(dev, scl, num_sg, DMA_TO_DEVICE);
+ num_dma_bufs = dma_map_sg(dev, scl, num_sg, DMA_BIDIRECTIONAL);
if (unlikely(!num_dma_bufs)) {
err = -ENOMEM;
goto dma_map_sg_failed;
* on TX confirmation. We are clearing FAS (Frame Annotation Status)
* field from the hardware annotation area
*/
- hwa = sgt_buf + priv->buf_layout.private_data_size;
- memset(hwa + DPAA2_FAS_OFFSET, 0, DPAA2_FAS_SIZE);
+ fas = dpaa2_get_fas(sgt_buf);
+ memset(fas, 0, DPAA2_FAS_SIZE);
sgt = (struct dpaa2_sg_entry *)(sgt_buf + priv->tx_data_offset);
swa->num_dma_bufs = num_dma_bufs;
/* Separately map the SGT buffer */
- addr = dma_map_single(dev, sgt_buf, sgt_buf_size, DMA_TO_DEVICE);
+ addr = dma_map_single(dev, sgt_buf, sgt_buf_size, DMA_BIDIRECTIONAL);
if (unlikely(dma_mapping_error(dev, addr))) {
err = -ENOMEM;
goto dma_map_single_failed;
dma_map_single_failed:
kfree(sgt_buf);
sgt_buf_alloc_failed:
- dma_unmap_sg(dev, scl, num_sg, DMA_TO_DEVICE);
+ dma_unmap_sg(dev, scl, num_sg, DMA_BIDIRECTIONAL);
dma_map_sg_failed:
kfree(scl);
return err;
{
struct device *dev = priv->net_dev->dev.parent;
u8 *buffer_start;
- void *hwa;
+ struct dpaa2_fas *fas;
struct sk_buff **skbh;
dma_addr_t addr;
* on TX confirmation. We are clearing FAS (Frame Annotation Status)
* field from the hardware annotation area
*/
- hwa = buffer_start + priv->buf_layout.private_data_size;
- memset(hwa + DPAA2_FAS_OFFSET, 0, DPAA2_FAS_SIZE);
+ fas = dpaa2_get_fas(buffer_start);
+ memset(fas, 0, DPAA2_FAS_SIZE);
/* Store a backpointer to the skb at the beginning of the buffer
* (in the private data area) such that we can release it
addr = dma_map_single(dev, buffer_start,
skb_tail_pointer(skb) - buffer_start,
- DMA_TO_DEVICE);
+ DMA_BIDIRECTIONAL);
if (unlikely(dma_mapping_error(dev, addr)))
return -ENOMEM;
struct dpaa2_fas *fas;
fd_addr = dpaa2_fd_get_addr(fd);
- skbh = phys_to_virt(fd_addr);
+ skbh = dpaa2_iova_to_virt(priv->iommu_domain, fd_addr);
+ fas = dpaa2_get_fas(skbh);
if (fd_format == dpaa2_fd_single) {
skb = *skbh;
*/
dma_unmap_single(dev, fd_addr,
skb_tail_pointer(skb) - buffer_start,
- DMA_TO_DEVICE);
+ DMA_BIDIRECTIONAL);
} else if (fd_format == dpaa2_fd_sg) {
swa = (struct dpaa2_eth_swa *)skbh;
skb = swa->skb;
num_dma_bufs = swa->num_dma_bufs;
/* Unmap the scatterlist */
- dma_unmap_sg(dev, scl, num_sg, DMA_TO_DEVICE);
+ dma_unmap_sg(dev, scl, num_sg, DMA_BIDIRECTIONAL);
kfree(scl);
/* Unmap the SGT buffer */
unmap_size = priv->tx_data_offset +
sizeof(struct dpaa2_sg_entry) * (1 + num_dma_bufs);
- dma_unmap_single(dev, fd_addr, unmap_size, DMA_TO_DEVICE);
+ dma_unmap_single(dev, fd_addr, unmap_size, DMA_BIDIRECTIONAL);
} else {
/* Unsupported format, mark it as errored and give up */
if (status)
* buffer but before we free it. The caller function is responsible
* for checking the status value.
*/
- if (status && (dpaa2_fd_get_frc(fd) & DPAA2_FD_FRC_FASV)) {
- fas = (struct dpaa2_fas *)
- ((void *)skbh + priv->buf_layout.private_data_size);
+ if (status)
*status = le32_to_cpu(fas->status);
- }
/* Free SGT buffer kmalloc'ed on tx */
if (fd_format != dpaa2_fd_single)
dev_kfree_skb(skb);
}
-static int dpaa2_eth_tx(struct sk_buff *skb, struct net_device *net_dev)
+static netdev_tx_t dpaa2_eth_tx(struct sk_buff *skb, struct net_device *net_dev)
{
struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
struct dpaa2_fd fd;
struct rtnl_link_stats64 *percpu_stats;
struct dpaa2_eth_drv_stats *percpu_extras;
u32 status = 0;
+ u32 fd_errors;
+ bool has_fas_errors = false;
/* Tracing point */
trace_dpaa2_tx_conf_fd(priv->net_dev, fd);
percpu_extras->tx_conf_frames++;
percpu_extras->tx_conf_bytes += dpaa2_fd_get_len(fd);
- free_tx_fd(priv, fd, &status);
-
- if (unlikely(status & DPAA2_ETH_TXCONF_ERR_MASK)) {
- percpu_stats = this_cpu_ptr(priv->percpu_stats);
- /* Tx-conf logically pertains to the egress path. */
- percpu_stats->tx_errors++;
+ /* Check frame errors in the FD field */
+ fd_errors = dpaa2_fd_get_ctrl(fd) & DPAA2_FD_TX_ERR_MASK;
+ if (unlikely(fd_errors)) {
+ /* We only check error bits in the FAS field if corresponding
+ * FAERR bit is set in FD and the FAS field is marked as valid
+ */
+ has_fas_errors = (fd_errors & DPAA2_FD_CTRL_FAERR) &&
+ !!(dpaa2_fd_get_frc(fd) & DPAA2_FD_FRC_FASV);
+ if (net_ratelimit())
+ netdev_dbg(priv->net_dev, "TX frame FD error: %x08\n",
+ fd_errors);
}
+
+ free_tx_fd(priv, fd, has_fas_errors ? &status : NULL);
+
+ if (likely(!fd_errors))
+ return;
+
+ percpu_stats = this_cpu_ptr(priv->percpu_stats);
+ /* Tx-conf logically pertains to the egress path. */
+ percpu_stats->tx_errors++;
+
+ if (has_fas_errors && net_ratelimit())
+ netdev_dbg(priv->net_dev, "TX frame FAS error: %x08\n",
+ status & DPAA2_FAS_TX_ERR_MASK);
}
static int set_rx_csum(struct dpaa2_eth_priv *priv, bool enable)
int ret, i;
do {
- ret = dpaa2_io_service_acquire(NULL, priv->dpbp_attrs.bpid,
+ ret = dpaa2_io_service_acquire(NULL, priv->bpid,
buf_array, count);
if (ret < 0) {
netdev_err(priv->net_dev, "dpaa2_io_service_acquire() failed\n");
}
for (i = 0; i < ret; i++) {
/* Same logic as on regular Rx path */
+ vaddr = dpaa2_iova_to_virt(priv->iommu_domain,
+ buf_array[i]);
dma_unmap_single(dev, buf_array[i],
DPAA2_ETH_RX_BUF_SIZE,
DMA_FROM_DEVICE);
- vaddr = phys_to_virt(buf_array[i]);
skb_free_frag(vaddr);
}
} while (ret);
break;
/* Refill pool if appropriate */
- refill_pool(priv, ch, priv->dpbp_attrs.bpid);
+ refill_pool(priv, ch, priv->bpid);
store_cleaned = consume_frames(ch);
cleaned += store_cleaned;
netif_carrier_off(priv->net_dev);
}
- netdev_info(priv->net_dev, "Link Event: state %s",
+ netdev_info(priv->net_dev, "Link Event: state %s\n",
state.up ? "up" : "down");
return 0;
struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
int err;
- err = seed_pool(priv, priv->dpbp_attrs.bpid);
+ err = seed_pool(priv, priv->bpid);
if (err) {
/* Not much to do; the buffer pool, though not filled up,
* may still contain some buffers which would enable us
* to limp on.
*/
netdev_err(net_dev, "Buffer seeding failed for DPBP %d (bpid=%d)\n",
- priv->dpbp_dev->obj_desc.id, priv->dpbp_attrs.bpid);
+ priv->dpbp_dev->obj_desc.id, priv->bpid);
}
/* We'll only start the txqs when the link is actually ready; make sure
/** Fill in counters maintained by the GPP driver. These may be different from
* the hardware counters obtained by ethtool.
*/
-void dpaa2_eth_get_stats(struct net_device *net_dev,
- struct rtnl_link_stats64 *stats)
+static void dpaa2_eth_get_stats(struct net_device *net_dev,
+ struct rtnl_link_stats64 *stats)
{
struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
struct rtnl_link_stats64 *percpu_stats;
if (!channel) {
dev_info(dev,
"No affine channel for cpu %d and above\n", i);
+ err = -ENODEV;
goto err_alloc_ch;
}
/* Register the new context */
err = dpaa2_io_service_register(NULL, nctx);
if (err) {
- dev_info(dev, "No affine DPIO for cpu %d\n", i);
+ dev_dbg(dev, "No affine DPIO for cpu %d\n", i);
/* If no affine DPIO for this core, there's probably
- * none available for next cores either.
+ * none available for next cores either. Signal we want
+ * to retry later, in case the DPIO devices weren't
+ * probed yet.
*/
+ err = -EPROBE_DEFER;
goto err_service_reg;
}
err_alloc_ch:
if (cpumask_empty(&priv->dpio_cpumask)) {
dev_err(dev, "No cpu with an affine DPIO/DPCON\n");
- return -ENODEV;
+ return err;
}
dev_info(dev, "Cores %*pbl available for processing ingress traffic\n",
int err;
struct fsl_mc_device *dpbp_dev;
struct device *dev = priv->net_dev->dev.parent;
+ struct dpbp_attr dpbp_attrs;
err = fsl_mc_object_allocate(to_fsl_mc_device(dev), FSL_MC_POOL_DPBP,
&dpbp_dev);
goto err_open;
}
+ err = dpbp_reset(priv->mc_io, 0, dpbp_dev->mc_handle);
+ if (err) {
+ dev_err(dev, "dpbp_reset() failed\n");
+ goto err_reset;
+ }
+
err = dpbp_enable(priv->mc_io, 0, dpbp_dev->mc_handle);
if (err) {
dev_err(dev, "dpbp_enable() failed\n");
}
err = dpbp_get_attributes(priv->mc_io, 0, dpbp_dev->mc_handle,
- &priv->dpbp_attrs);
+ &dpbp_attrs);
if (err) {
dev_err(dev, "dpbp_get_attributes() failed\n");
goto err_get_attr;
}
+ priv->bpid = dpbp_attrs.bpid;
return 0;
err_get_attr:
dpbp_disable(priv->mc_io, 0, dpbp_dev->mc_handle);
err_enable:
+err_reset:
dpbp_close(priv->mc_io, 0, dpbp_dev->mc_handle);
err_open:
fsl_mc_object_free(dpbp_dev);
struct device *dev = &ls_dev->dev;
struct dpaa2_eth_priv *priv;
struct net_device *net_dev;
+ struct dpni_buffer_layout buf_layout = {0};
int err;
net_dev = dev_get_drvdata(dev);
priv = netdev_priv(net_dev);
- priv->dpni_id = ls_dev->obj_desc.id;
-
/* get a handle for the DPNI object */
- err = dpni_open(priv->mc_io, 0, priv->dpni_id, &priv->mc_token);
+ err = dpni_open(priv->mc_io, 0, ls_dev->obj_desc.id, &priv->mc_token);
if (err) {
dev_err(dev, "dpni_open() failed\n");
goto err_open;
/* Configure buffer layouts */
/* rx buffer */
- priv->buf_layout.pass_parser_result = true;
- priv->buf_layout.pass_frame_status = true;
- priv->buf_layout.private_data_size = DPAA2_ETH_SWA_SIZE;
- priv->buf_layout.data_align = DPAA2_ETH_RX_BUF_ALIGN;
- priv->buf_layout.options = DPNI_BUF_LAYOUT_OPT_PARSER_RESULT |
- DPNI_BUF_LAYOUT_OPT_FRAME_STATUS |
- DPNI_BUF_LAYOUT_OPT_PRIVATE_DATA_SIZE |
- DPNI_BUF_LAYOUT_OPT_DATA_ALIGN;
+ buf_layout.pass_parser_result = true;
+ buf_layout.pass_frame_status = true;
+ buf_layout.private_data_size = DPAA2_ETH_SWA_SIZE;
+ buf_layout.data_align = DPAA2_ETH_RX_BUF_ALIGN;
+ buf_layout.options = DPNI_BUF_LAYOUT_OPT_PARSER_RESULT |
+ DPNI_BUF_LAYOUT_OPT_FRAME_STATUS |
+ DPNI_BUF_LAYOUT_OPT_PRIVATE_DATA_SIZE |
+ DPNI_BUF_LAYOUT_OPT_DATA_ALIGN;
err = dpni_set_buffer_layout(priv->mc_io, 0, priv->mc_token,
- DPNI_QUEUE_RX, &priv->buf_layout);
+ DPNI_QUEUE_RX, &buf_layout);
if (err) {
dev_err(dev, "dpni_set_buffer_layout(RX) failed\n");
goto err_buf_layout;
}
/* tx buffer */
- priv->buf_layout.options = DPNI_BUF_LAYOUT_OPT_FRAME_STATUS |
- DPNI_BUF_LAYOUT_OPT_PRIVATE_DATA_SIZE;
+ buf_layout.options = DPNI_BUF_LAYOUT_OPT_FRAME_STATUS |
+ DPNI_BUF_LAYOUT_OPT_PRIVATE_DATA_SIZE;
err = dpni_set_buffer_layout(priv->mc_io, 0, priv->mc_token,
- DPNI_QUEUE_TX, &priv->buf_layout);
+ DPNI_QUEUE_TX, &buf_layout);
if (err) {
dev_err(dev, "dpni_set_buffer_layout(TX) failed\n");
goto err_buf_layout;
}
/* tx-confirm buffer */
- priv->buf_layout.options = DPNI_BUF_LAYOUT_OPT_FRAME_STATUS;
+ buf_layout.options = DPNI_BUF_LAYOUT_OPT_FRAME_STATUS;
err = dpni_set_buffer_layout(priv->mc_io, 0, priv->mc_token,
- DPNI_QUEUE_TX_CONFIRM, &priv->buf_layout);
+ DPNI_QUEUE_TX_CONFIRM, &buf_layout);
if (err) {
dev_err(dev, "dpni_set_buffer_layout(TX_CONF) failed\n");
goto err_buf_layout;
}
if ((priv->tx_data_offset % 64) != 0)
- dev_warn(dev, "Tx data offset (%d) not a multiple of 64B",
+ dev_warn(dev, "Tx data offset (%d) not a multiple of 64B\n",
priv->tx_data_offset);
/* Accommodate software annotation space (SWA) */
/* Set RX hash options
* flags is a combination of RXH_ bits
*/
-int dpaa2_eth_set_hash(struct net_device *net_dev, u64 flags)
+static int dpaa2_eth_set_hash(struct net_device *net_dev, u64 flags)
{
struct device *dev = net_dev->dev.parent;
struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
int err = 0;
if (!dpaa2_eth_hash_enabled(priv)) {
- dev_err(dev, "Hashing support is not enabled\n");
- return -EOPNOTSUPP;
+ dev_dbg(dev, "Hashing support is not enabled\n");
+ return 0;
}
memset(&cls_cfg, 0, sizeof(cls_cfg));
priv->rx_hash_fields |= hash_fields[i].rxnfc_field;
}
- dma_mem = kzalloc(DPAA2_CLASSIFIER_DMA_SIZE, GFP_DMA | GFP_KERNEL);
+ dma_mem = kzalloc(DPAA2_CLASSIFIER_DMA_SIZE, GFP_KERNEL);
if (!dma_mem)
return -ENOMEM;
err = dpni_prepare_key_cfg(&cls_cfg, dma_mem);
if (err) {
- dev_err(dev, "dpni_prepare_key_cfg error %d", err);
+ dev_err(dev, "dpni_prepare_key_cfg error %d\n", err);
goto err_prep_key;
}
memset(&dist_cfg, 0, sizeof(dist_cfg));
/* Prepare for setting the rx dist */
- dist_cfg.key_cfg_iova = dma_map_single(net_dev->dev.parent, dma_mem,
+ dist_cfg.key_cfg_iova = dma_map_single(dev, dma_mem,
DPAA2_CLASSIFIER_DMA_SIZE,
DMA_TO_DEVICE);
- if (dma_mapping_error(net_dev->dev.parent, dist_cfg.key_cfg_iova)) {
+ if (dma_mapping_error(dev, dist_cfg.key_cfg_iova)) {
dev_err(dev, "DMA mapping failed\n");
err = -ENOMEM;
goto err_dma_map;
dist_cfg.dist_mode = DPNI_DIST_MODE_HASH;
err = dpni_set_rx_tc_dist(priv->mc_io, 0, priv->mc_token, 0, &dist_cfg);
- dma_unmap_single(net_dev->dev.parent, dist_cfg.key_cfg_iova,
+ dma_unmap_single(dev, dist_cfg.key_cfg_iova,
DPAA2_CLASSIFIER_DMA_SIZE, DMA_TO_DEVICE);
if (err)
dev_err(dev, "dpni_set_rx_tc_dist() error %d\n", err);
netdev_err(net_dev, "Failed to configure hashing\n");
/* Configure handling of error frames */
- err_cfg.errors = DPAA2_ETH_RX_ERR_MASK;
+ err_cfg.errors = DPAA2_FAS_RX_ERR_MASK;
err_cfg.set_frame_annotation = 1;
err_cfg.error_action = DPNI_ERROR_ACTION_DISCARD;
err = dpni_set_errors_behavior(priv->mc_io, 0, priv->mc_token,
dpaa2_io_store_destroy(priv->channel[i]->store);
}
-static int netdev_init(struct net_device *net_dev)
+static int set_mac_addr(struct dpaa2_eth_priv *priv)
{
- int err;
+ struct net_device *net_dev = priv->net_dev;
struct device *dev = net_dev->dev.parent;
- struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
u8 mac_addr[ETH_ALEN], dpni_mac_addr[ETH_ALEN];
- u8 bcast_addr[ETH_ALEN];
-
- net_dev->netdev_ops = &dpaa2_eth_ops;
+ int err;
/* Get firmware address, if any */
err = dpni_get_port_mac_addr(priv->mc_io, 0, priv->mc_token, mac_addr);
err = dpni_get_primary_mac_addr(priv->mc_io, 0, priv->mc_token,
dpni_mac_addr);
if (err) {
- dev_err(dev, "dpni_get_primary_mac_addr() failed (%d)\n", err);
+ dev_err(dev, "dpni_get_primary_mac_addr() failed\n");
return err;
}
}
memcpy(net_dev->dev_addr, mac_addr, net_dev->addr_len);
} else if (is_zero_ether_addr(dpni_mac_addr)) {
- /* Fills in net_dev->dev_addr, as required by
- * register_netdevice()
+ /* No MAC address configured, fill in net_dev->dev_addr
+ * with a random one
*/
eth_hw_addr_random(net_dev);
- /* Make the user aware, without cluttering the boot log */
- dev_dbg_once(dev, " device(s) have all-zero hwaddr, replaced with random\n");
+ dev_dbg_once(dev, "device(s) have all-zero hwaddr, replaced with random\n");
+
err = dpni_set_primary_mac_addr(priv->mc_io, 0, priv->mc_token,
net_dev->dev_addr);
if (err) {
- dev_err(dev, "dpni_set_primary_mac_addr(): %d\n", err);
+ dev_err(dev, "dpni_set_primary_mac_addr() failed\n");
return err;
}
+
/* Override NET_ADDR_RANDOM set by eth_hw_addr_random(); for all
* practical purposes, this will be our "permanent" mac address,
* at least until the next reboot. This move will also permit
memcpy(net_dev->dev_addr, dpni_mac_addr, net_dev->addr_len);
}
- /* Explicitly add the broadcast address to the MAC filtering table;
- * the MC won't do that for us.
- */
+ return 0;
+}
+
+static int netdev_init(struct net_device *net_dev)
+{
+ struct device *dev = net_dev->dev.parent;
+ struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
+ u8 bcast_addr[ETH_ALEN];
+ u8 num_queues;
+ int err;
+
+ net_dev->netdev_ops = &dpaa2_eth_ops;
+
+ err = set_mac_addr(priv);
+ if (err)
+ return err;
+
+ /* Explicitly add the broadcast address to the MAC filtering table */
eth_broadcast_addr(bcast_addr);
err = dpni_add_mac_addr(priv->mc_io, 0, priv->mc_token, bcast_addr);
if (err) {
- dev_warn(dev, "dpni_add_mac_addr() failed (%d)\n", err);
- /* Won't return an error; at least, we'd have egress traffic */
+ dev_err(dev, "dpni_add_mac_addr() failed\n");
+ return err;
}
/* Reserve enough space to align buffer as per hardware requirement;
net_dev->min_mtu = 68;
net_dev->max_mtu = DPAA2_ETH_MAX_MTU;
+ /* Set actual number of queues in the net device */
+ num_queues = dpaa2_eth_queue_count(priv);
+ err = netif_set_real_num_tx_queues(net_dev, num_queues);
+ if (err) {
+ dev_err(dev, "netif_set_real_num_tx_queues() failed\n");
+ return err;
+ }
+ err = netif_set_real_num_rx_queues(net_dev, num_queues);
+ if (err) {
+ dev_err(dev, "netif_set_real_num_rx_queues() failed\n");
+ return err;
+ }
+
/* Our .ndo_init will be called herein */
err = register_netdev(net_dev);
if (err < 0) {
static irqreturn_t dpni_irq0_handler_thread(int irq_num, void *arg)
{
- u32 status, clear = 0;
+ u32 status = 0, clear = 0;
struct device *dev = (struct device *)arg;
struct fsl_mc_device *dpni_dev = to_fsl_mc_device(dev);
struct net_device *net_dev = dev_get_drvdata(dev);
err = dpni_get_irq_status(dpni_dev->mc_io, 0, dpni_dev->mc_handle,
DPNI_IRQ_INDEX, &status);
if (unlikely(err)) {
- netdev_err(net_dev, "Can't get irq status (err %d)", err);
+ netdev_err(net_dev, "Can't get irq status (err %d)\n", err);
clear = 0xffffffff;
goto out;
}
IRQF_NO_SUSPEND | IRQF_ONESHOT,
dev_name(&ls_dev->dev), &ls_dev->dev);
if (err < 0) {
- dev_err(&ls_dev->dev, "devm_request_threaded_irq(): %d", err);
+ dev_err(&ls_dev->dev, "devm_request_threaded_irq(): %d\n", err);
goto free_mc_irq;
}
err = dpni_set_irq_mask(ls_dev->mc_io, 0, ls_dev->mc_handle,
DPNI_IRQ_INDEX, DPNI_IRQ_EVENT_LINK_CHANGED);
if (err < 0) {
- dev_err(&ls_dev->dev, "dpni_set_irq_mask(): %d", err);
+ dev_err(&ls_dev->dev, "dpni_set_irq_mask(): %d\n", err);
goto free_irq;
}
err = dpni_set_irq_enable(ls_dev->mc_io, 0, ls_dev->mc_handle,
DPNI_IRQ_INDEX, 1);
if (err < 0) {
- dev_err(&ls_dev->dev, "dpni_set_irq_enable(): %d", err);
+ dev_err(&ls_dev->dev, "dpni_set_irq_enable(): %d\n", err);
goto free_irq;
}
priv = netdev_priv(net_dev);
priv->net_dev = net_dev;
+ priv->iommu_domain = iommu_get_domain_for_dev(dev);
+
/* Obtain a MC portal */
err = fsl_mc_portal_allocate(dpni_dev, FSL_MC_IO_ATOMIC_CONTEXT_PORTAL,
&priv->mc_io);
#define DPAA2_FD_FRC_FASWOV 0x0800
#define DPAA2_FD_FRC_FAICFDV 0x0400
+/* Error bits in FD CTRL */
+#define DPAA2_FD_CTRL_UFD 0x00000004
+#define DPAA2_FD_CTRL_SBE 0x00000008
+#define DPAA2_FD_CTRL_FSE 0x00000010
+#define DPAA2_FD_CTRL_FAERR 0x00000020
+
+#define DPAA2_FD_RX_ERR_MASK (DPAA2_FD_CTRL_SBE | \
+ DPAA2_FD_CTRL_FAERR)
+#define DPAA2_FD_TX_ERR_MASK (DPAA2_FD_CTRL_UFD | \
+ DPAA2_FD_CTRL_SBE | \
+ DPAA2_FD_CTRL_FSE | \
+ DPAA2_FD_CTRL_FAERR)
+
/* Annotation bits in FD CTRL */
#define DPAA2_FD_CTRL_ASAL 0x00020000 /* ASAL = 128 */
#define DPAA2_FD_CTRL_PTA 0x00800000
#define DPAA2_FAS_OFFSET 0
#define DPAA2_FAS_SIZE (sizeof(struct dpaa2_fas))
+/* Accessors for the hardware annotation fields that we use */
+#define dpaa2_get_hwa(buf_addr) \
+ ((void *)(buf_addr) + DPAA2_ETH_SWA_SIZE)
+#define dpaa2_get_fas(buf_addr) \
+ (struct dpaa2_fas *)(dpaa2_get_hwa(buf_addr) + DPAA2_FAS_OFFSET)
+
/* Error and status bits in the frame annotation status word */
/* Debug frame, otherwise supposed to be discarded */
#define DPAA2_FAS_DISC 0x80000000
/* L4 csum error */
#define DPAA2_FAS_L4CE 0x00000001
/* Possible errors on the ingress path */
-#define DPAA2_ETH_RX_ERR_MASK (DPAA2_FAS_KSE | \
+#define DPAA2_FAS_RX_ERR_MASK (DPAA2_FAS_KSE | \
DPAA2_FAS_EOFHE | \
DPAA2_FAS_MNLE | \
DPAA2_FAS_TIDE | \
DPAA2_FAS_L3CE | \
DPAA2_FAS_L4CE)
/* Tx errors */
-#define DPAA2_ETH_TXCONF_ERR_MASK (DPAA2_FAS_KSE | \
+#define DPAA2_FAS_TX_ERR_MASK (DPAA2_FAS_KSE | \
DPAA2_FAS_EOFHE | \
DPAA2_FAS_MNLE | \
DPAA2_FAS_TIDE)
u8 num_channels;
struct dpaa2_eth_channel *channel[DPAA2_ETH_MAX_DPCONS];
- int dpni_id;
struct dpni_attr dpni_attrs;
- /* Insofar as the MC is concerned, we're using one layout on all 3 types
- * of buffers (Rx, Tx, Tx-Conf).
- */
- struct dpni_buffer_layout buf_layout;
u16 tx_data_offset;
struct fsl_mc_device *dpbp_dev;
- struct dpbp_attr dpbp_attrs;
+ u16 bpid;
+ struct iommu_domain *iommu_domain;
u16 tx_qdid;
struct fsl_mc_io *mc_io;
extern const struct ethtool_ops dpaa2_ethtool_ops;
extern const char dpaa2_eth_drv_version[];
-int dpaa2_eth_set_hash(struct net_device *net_dev, u64 flags);
-
static int dpaa2_eth_queue_count(struct dpaa2_eth_priv *priv)
{
return priv->dpni_attrs.num_queues;
#include "dpaa2-eth.h"
/* To be kept in sync with DPNI statistics */
-char dpaa2_ethtool_stats[][ETH_GSTRING_LEN] = {
- "rx frames",
- "rx bytes",
- "rx mcast frames",
- "rx mcast bytes",
- "rx bcast frames",
- "rx bcast bytes",
- "tx frames",
- "tx bytes",
- "tx mcast frames",
- "tx mcast bytes",
- "tx bcast frames",
- "tx bcast bytes",
- "rx filtered frames",
- "rx discarded frames",
- "rx nobuffer discards",
- "tx discarded frames",
- "tx confirmed frames",
+static char dpaa2_ethtool_stats[][ETH_GSTRING_LEN] = {
+ "[hw] rx frames",
+ "[hw] rx bytes",
+ "[hw] rx mcast frames",
+ "[hw] rx mcast bytes",
+ "[hw] rx bcast frames",
+ "[hw] rx bcast bytes",
+ "[hw] tx frames",
+ "[hw] tx bytes",
+ "[hw] tx mcast frames",
+ "[hw] tx mcast bytes",
+ "[hw] tx bcast frames",
+ "[hw] tx bcast bytes",
+ "[hw] rx filtered frames",
+ "[hw] rx discarded frames",
+ "[hw] rx nobuffer discards",
+ "[hw] tx discarded frames",
+ "[hw] tx confirmed frames",
};
#define DPAA2_ETH_NUM_STATS ARRAY_SIZE(dpaa2_ethtool_stats)
-char dpaa2_ethtool_extras[][ETH_GSTRING_LEN] = {
+static char dpaa2_ethtool_extras[][ETH_GSTRING_LEN] = {
/* per-cpu stats */
- "tx conf frames",
- "tx conf bytes",
- "tx sg frames",
- "tx sg bytes",
- "rx sg frames",
- "rx sg bytes",
- "enqueue portal busy",
+ "[drv] tx conf frames",
+ "[drv] tx conf bytes",
+ "[drv] tx sg frames",
+ "[drv] tx sg bytes",
+ "[drv] rx sg frames",
+ "[drv] rx sg bytes",
+ "[drv] enqueue portal busy",
/* Channel stats */
- "dequeue portal busy",
- "channel pull errors",
- "cdan",
+ "[drv] dequeue portal busy",
+ "[drv] channel pull errors",
+ "[drv] cdan",
};
#define DPAA2_ETH_NUM_EXTRA_STATS ARRAY_SIZE(dpaa2_ethtool_extras)
err = dpni_get_link_state(priv->mc_io, 0, priv->mc_token, &state);
if (err) {
- netdev_err(net_dev, "ERROR %d getting link state", err);
+ netdev_err(net_dev, "ERROR %d getting link state\n", err);
goto out;
}
/* ethtool will be loud enough if we return an error; no point
* in putting our own error message on the console by default
*/
- netdev_dbg(net_dev, "ERROR %d setting link cfg", err);
+ netdev_dbg(net_dev, "ERROR %d setting link cfg\n", err);
return err;
}
err = dpni_get_statistics(priv->mc_io, 0, priv->mc_token,
j, &dpni_stats);
if (err != 0)
- netdev_warn(net_dev, "dpni_get_stats(%d) failed", j);
+ netdev_warn(net_dev, "dpni_get_stats(%d) failed\n", j);
switch (j) {
case 0:
num_cnt = sizeof(dpni_stats.page_0) / sizeof(u64);
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
-#include "../../fsl-mc/include/mc-sys.h"
-#include "../../fsl-mc/include/mc-cmd.h"
+#include <linux/kernel.h>
+#include <linux/errno.h>
+#include "../../fsl-mc/include/mc.h"
#include "dpni.h"
#include "dpni-cmd.h"
DPRCs can be defined statically and populated with objects
via a config file passed to the MC when firmware starts
- it. There is also a Linux user space tool called "restool"
- that can be used to create/destroy containers and objects
- dynamically.
+ it.
-DPAA2 Objects for an Ethernet Network Interface
-creates an MSI IRQ domain
-doing a 'device add' to expose the 'root' DPRC, in turn triggering
a bind of the root DPRC to the DPRC driver
+ The binding for the MC-bus device-tree node can be consulted here:
+ Documentation/devicetree/bindings/misc/fsl,qoriq-mc.txt
DPRC driver
-----------
mc-sys.o \
mc-io.o \
dprc.o \
- dpmng.o \
dprc-driver.o \
fsl-mc-allocator.o \
fsl-mc-msi.o \
#define DPBP_CMD_BASE_VERSION 1
#define DPBP_CMD_ID_OFFSET 4
-#define DPBP_CMD(id) ((id << DPBP_CMD_ID_OFFSET) | DPBP_CMD_BASE_VERSION)
+#define DPBP_CMD(id) (((id) << DPBP_CMD_ID_OFFSET) | DPBP_CMD_BASE_VERSION)
/* Command IDs */
#define DPBP_CMDID_CLOSE DPBP_CMD(0x800)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
-#include "../include/mc-sys.h"
-#include "../include/mc-cmd.h"
+#include <linux/kernel.h>
+#include "../include/mc.h"
#include "../include/dpbp.h"
#include "dpbp-cmd.h"
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
-#include "../include/mc-sys.h"
-#include "../include/mc-cmd.h"
+#include <linux/kernel.h>
+#include "../include/mc.h"
#include "../include/dpcon.h"
#include "dpcon-cmd.h"
/* Enable the generation of CDAN notifications */
if (ctx->is_cdan)
- qbman_swp_CDAN_set_context_enable(d->swp,
- (u16)ctx->id,
- ctx->qman64);
+ return qbman_swp_CDAN_set_context_enable(d->swp,
+ (u16)ctx->id,
+ ctx->qman64);
return 0;
}
EXPORT_SYMBOL(dpaa2_io_service_register);
* The size of the storage is "max_frames*sizeof(struct dpaa2_dq)".
* The 'dpaa2_io_store' returned is a DPIO service managed object.
*
- * Return pointer to dpaa2_io_store struct for successfuly created storage
+ * Return pointer to dpaa2_io_store struct for successfully created storage
* memory, or NULL on error.
*/
struct dpaa2_io_store *dpaa2_io_store_create(unsigned int max_frames,
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
-#include "../../include/mc-sys.h"
-#include "../../include/mc-cmd.h"
+#include <linux/kernel.h>
+#include "../../include/mc.h"
#include "dpio.h"
#include "dpio-cmd.h"
#define DPMCP_CMD_BASE_VERSION 1
#define DPMCP_CMD_ID_OFFSET 4
-#define DPMCP_CMD(id) ((id << DPMCP_CMD_ID_OFFSET) | DPMCP_CMD_BASE_VERSION)
+#define DPMCP_CMD(id) (((id) << DPMCP_CMD_ID_OFFSET) | DPMCP_CMD_BASE_VERSION)
/* Command IDs */
#define DPMCP_CMDID_CLOSE DPMCP_CMD(0x800)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
-#include "../include/mc-sys.h"
-#include "../include/mc-cmd.h"
+#include <linux/kernel.h>
+#include "../include/mc.h"
#include "dpmcp.h"
#include "dpmcp-cmd.h"
#define DPMNG_CMD_BASE_VERSION 1
#define DPMNG_CMD_ID_OFFSET 4
-#define DPMNG_CMD(id) ((id << DPMNG_CMD_ID_OFFSET) | DPMNG_CMD_BASE_VERSION)
+#define DPMNG_CMD(id) (((id) << DPMNG_CMD_ID_OFFSET) | DPMNG_CMD_BASE_VERSION)
/* Command IDs */
#define DPMNG_CMDID_GET_VERSION DPMNG_CMD(0x831)
+++ /dev/null
-/*
- * Copyright 2013-2016 Freescale Semiconductor Inc.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- * * Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * * Redistributions in binary form must reproduce the above copyright
- * notice, this list of conditions and the following disclaimer in the
- * documentation and/or other materials provided with the distribution.
- * * Neither the name of the above-listed copyright holders nor the
- * names of any contributors may be used to endorse or promote products
- * derived from this software without specific prior written permission.
- *
- * ALTERNATIVELY, this software may be distributed under the terms of the
- * GNU General Public License ("GPL") as published by the Free Software
- * Foundation, either version 2 of that License or (at your option) any
- * later version.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-#include "../include/mc-sys.h"
-#include "../include/mc-cmd.h"
-#include "../include/dpmng.h"
-
-#include "dpmng-cmd.h"
-
-/**
- * mc_get_version() - Retrieves the Management Complex firmware
- * version information
- * @mc_io: Pointer to opaque I/O object
- * @cmd_flags: Command flags; one or more of 'MC_CMD_FLAG_'
- * @mc_ver_info: Returned version information structure
- *
- * Return: '0' on Success; Error code otherwise.
- */
-int mc_get_version(struct fsl_mc_io *mc_io,
- u32 cmd_flags,
- struct mc_version *mc_ver_info)
-{
- struct mc_command cmd = { 0 };
- struct dpmng_rsp_get_version *rsp_params;
- int err;
-
- /* prepare command */
- cmd.header = mc_encode_cmd_header(DPMNG_CMDID_GET_VERSION,
- cmd_flags,
- 0);
-
- /* send command to mc*/
- err = mc_send_command(mc_io, &cmd);
- if (err)
- return err;
-
- /* retrieve response parameters */
- rsp_params = (struct dpmng_rsp_get_version *)cmd.params;
- mc_ver_info->revision = le32_to_cpu(rsp_params->revision);
- mc_ver_info->major = le32_to_cpu(rsp_params->version_major);
- mc_ver_info->minor = le32_to_cpu(rsp_params->version_minor);
-
- return 0;
-}
-EXPORT_SYMBOL(mc_get_version);
-
#define DPRC_CMD_BASE_VERSION 1
#define DPRC_CMD_ID_OFFSET 4
-#define DPRC_CMD(id) ((id << DPRC_CMD_ID_OFFSET) | DPRC_CMD_BASE_VERSION)
+#define DPRC_CMD(id) (((id) << DPRC_CMD_ID_OFFSET) | DPRC_CMD_BASE_VERSION)
/* Command IDs */
#define DPRC_CMDID_CLOSE DPRC_CMD(0x800)
#include <linux/slab.h>
#include <linux/interrupt.h>
#include <linux/msi.h>
-#include "../include/mc-bus.h"
-#include "../include/mc-sys.h"
+#include "../include/mc.h"
#include "dprc-cmd.h"
#include "fsl-mc-private.h"
#define FSL_MC_DPRC_DRIVER_NAME "fsl_mc_dprc"
-#define FSL_MC_DEVICE_MATCH(_mc_dev, _obj_desc) \
- (strcmp((_mc_dev)->obj_desc.type, (_obj_desc)->type) == 0 && \
- (_mc_dev)->obj_desc.id == (_obj_desc)->id)
-
-struct dprc_child_objs {
+struct fsl_mc_child_objs {
int child_count;
- struct dprc_obj_desc *child_array;
+ struct fsl_mc_obj_desc *child_array;
};
+static bool fsl_mc_device_match(struct fsl_mc_device *mc_dev,
+ struct fsl_mc_obj_desc *obj_desc)
+{
+ return mc_dev->obj_desc.id == obj_desc->id &&
+ !strcmp(mc_dev->obj_desc.type, obj_desc->type);
+
+}
+
static int __fsl_mc_device_remove_if_not_in_mc(struct device *dev, void *data)
{
int i;
- struct dprc_child_objs *objs;
+ struct fsl_mc_child_objs *objs;
struct fsl_mc_device *mc_dev;
WARN_ON(!dev);
objs = data;
for (i = 0; i < objs->child_count; i++) {
- struct dprc_obj_desc *obj_desc = &objs->child_array[i];
+ struct fsl_mc_obj_desc *obj_desc = &objs->child_array[i];
if (strlen(obj_desc->type) != 0 &&
- FSL_MC_DEVICE_MATCH(mc_dev, obj_desc))
+ fsl_mc_device_match(mc_dev, obj_desc))
break;
}
* been dynamically removed in the physical DPRC.
*/
static void dprc_remove_devices(struct fsl_mc_device *mc_bus_dev,
- struct dprc_obj_desc *obj_desc_array,
+ struct fsl_mc_obj_desc *obj_desc_array,
int num_child_objects_in_mc)
{
if (num_child_objects_in_mc != 0) {
* Remove child objects that are in the DPRC in Linux,
* but not in the MC:
*/
- struct dprc_child_objs objs;
+ struct fsl_mc_child_objs objs;
objs.child_count = num_child_objects_in_mc;
objs.child_array = obj_desc_array;
static int __fsl_mc_device_match(struct device *dev, void *data)
{
- struct dprc_obj_desc *obj_desc = data;
+ struct fsl_mc_obj_desc *obj_desc = data;
struct fsl_mc_device *mc_dev = to_fsl_mc_device(dev);
- return FSL_MC_DEVICE_MATCH(mc_dev, obj_desc);
+ return fsl_mc_device_match(mc_dev, obj_desc);
}
-static struct fsl_mc_device *fsl_mc_device_lookup(struct dprc_obj_desc
+static struct fsl_mc_device *fsl_mc_device_lookup(struct fsl_mc_obj_desc
*obj_desc,
struct fsl_mc_device
*mc_bus_dev)
* device is unbound from the corresponding device driver.
*/
static void check_plugged_state_change(struct fsl_mc_device *mc_dev,
- struct dprc_obj_desc *obj_desc)
+ struct fsl_mc_obj_desc *obj_desc)
{
int error;
u32 plugged_flag_at_mc =
- obj_desc->state & DPRC_OBJ_STATE_PLUGGED;
+ obj_desc->state & FSL_MC_OBJ_STATE_PLUGGED;
if (plugged_flag_at_mc !=
- (mc_dev->obj_desc.state & DPRC_OBJ_STATE_PLUGGED)) {
+ (mc_dev->obj_desc.state & FSL_MC_OBJ_STATE_PLUGGED)) {
if (plugged_flag_at_mc) {
- mc_dev->obj_desc.state |= DPRC_OBJ_STATE_PLUGGED;
+ mc_dev->obj_desc.state |= FSL_MC_OBJ_STATE_PLUGGED;
error = device_attach(&mc_dev->dev);
if (error < 0) {
dev_err(&mc_dev->dev,
error);
}
} else {
- mc_dev->obj_desc.state &= ~DPRC_OBJ_STATE_PLUGGED;
+ mc_dev->obj_desc.state &= ~FSL_MC_OBJ_STATE_PLUGGED;
device_release_driver(&mc_dev->dev);
}
}
* in the physical DPRC.
*/
static void dprc_add_new_devices(struct fsl_mc_device *mc_bus_dev,
- struct dprc_obj_desc *obj_desc_array,
+ struct fsl_mc_obj_desc *obj_desc_array,
int num_child_objects_in_mc)
{
int error;
for (i = 0; i < num_child_objects_in_mc; i++) {
struct fsl_mc_device *child_dev;
- struct dprc_obj_desc *obj_desc = &obj_desc_array[i];
+ struct fsl_mc_obj_desc *obj_desc = &obj_desc_array[i];
if (strlen(obj_desc->type) == 0)
continue;
* populated before they can get allocation requests from probe callbacks
* of the device drivers for the non-allocatable devices.
*/
-int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev,
- unsigned int *total_irq_count)
+static int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev,
+ unsigned int *total_irq_count)
{
int num_child_objects;
int dprc_get_obj_failures;
int error;
unsigned int irq_count = mc_bus_dev->obj_desc.irq_count;
- struct dprc_obj_desc *child_obj_desc_array = NULL;
+ struct fsl_mc_obj_desc *child_obj_desc_array = NULL;
error = dprc_get_obj_count(mc_bus_dev->mc_io,
0,
*/
dprc_get_obj_failures = 0;
for (i = 0; i < num_child_objects; i++) {
- struct dprc_obj_desc *obj_desc =
+ struct fsl_mc_obj_desc *obj_desc =
&child_obj_desc_array[i];
error = dprc_get_obj(mc_bus_dev->mc_io,
if ((strcmp(obj_desc->type, "dpseci") == 0) &&
(obj_desc->ver_major < 4))
obj_desc->flags |=
- DPRC_OBJ_FLAG_NO_MEM_SHAREABILITY;
+ FSL_MC_OBJ_FLAG_NO_MEM_SHAREABILITY;
irq_count += obj_desc->irq_count;
dev_dbg(&mc_bus_dev->dev,
return 0;
}
-EXPORT_SYMBOL_GPL(dprc_scan_objects);
/**
* dprc_scan_container - Scans a physical DPRC and synchronizes Linux bus state
* bus driver with the actual state of the MC by adding and removing
* devices as appropriate.
*/
-int dprc_scan_container(struct fsl_mc_device *mc_bus_dev)
+static int dprc_scan_container(struct fsl_mc_device *mc_bus_dev)
{
int error;
unsigned int irq_count;
fsl_mc_cleanup_all_resource_pools(mc_bus_dev);
return error;
}
-EXPORT_SYMBOL_GPL(dprc_scan_container);
/**
* dprc_irq0_handler - Regular ISR for DPRC interrupt 0
}
if (major_ver < DPRC_MIN_VER_MAJOR ||
- (major_ver == DPRC_MIN_VER_MAJOR &&
- minor_ver < DPRC_MIN_VER_MINOR)) {
+ (major_ver == DPRC_MIN_VER_MAJOR &&
+ minor_ver < DPRC_MIN_VER_MINOR)) {
dev_err(&mc_dev->dev,
"ERROR: DPRC version %d.%d not supported\n",
major_ver, minor_ver);
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
-#include "../include/mc-sys.h"
-#include "../include/mc-cmd.h"
-#include "../include/dprc.h"
+#include <linux/kernel.h>
+#include "../include/mc.h"
+#include "dprc.h"
#include "dprc-cmd.h"
u32 cmd_flags,
u16 token,
int obj_index,
- struct dprc_obj_desc *obj_desc)
+ struct fsl_mc_obj_desc *obj_desc)
{
struct mc_command cmd = { 0 };
struct dprc_cmd_get_obj *cmd_params;
#ifndef _FSL_DPRC_H
#define _FSL_DPRC_H
-#include "mc-cmd.h"
-
/*
* Data Path Resource Container API
* Contains DPRC API for managing and querying DPAA resources
*/
struct fsl_mc_io;
+struct fsl_mc_obj_desc;
int dprc_open(struct fsl_mc_io *mc_io,
u32 cmd_flags,
u32 cmd_flags,
u16 token);
-
/* IRQ */
/* IRQ index */
u16 token,
int *obj_count);
-/* Objects Attributes Flags */
-
-/* Opened state - Indicates that an object is open by at least one owner */
-#define DPRC_OBJ_STATE_OPEN 0x00000001
-/* Plugged state - Indicates that the object is plugged */
-#define DPRC_OBJ_STATE_PLUGGED 0x00000002
-
-/**
- * Shareability flag - Object flag indicating no memory shareability.
- * the object generates memory accesses that are non coherent with other
- * masters;
- * user is responsible for proper memory handling through IOMMU configuration.
- */
-#define DPRC_OBJ_FLAG_NO_MEM_SHAREABILITY 0x0001
-
-/**
- * struct dprc_obj_desc - Object descriptor, returned from dprc_get_obj()
- * @type: Type of object: NULL terminated string
- * @id: ID of logical object resource
- * @vendor: Object vendor identifier
- * @ver_major: Major version number
- * @ver_minor: Minor version number
- * @irq_count: Number of interrupts supported by the object
- * @region_count: Number of mappable regions supported by the object
- * @state: Object state: combination of DPRC_OBJ_STATE_ states
- * @label: Object label
- * @flags: Object's flags
- */
-struct dprc_obj_desc {
- char type[16];
- int id;
- u16 vendor;
- u16 ver_major;
- u16 ver_minor;
- u8 irq_count;
- u8 region_count;
- u32 state;
- char label[16];
- u16 flags;
-};
-
int dprc_get_obj(struct fsl_mc_io *mc_io,
u32 cmd_flags,
u16 token,
int obj_index,
- struct dprc_obj_desc *obj_desc);
+ struct fsl_mc_obj_desc *obj_desc);
int dprc_get_obj_desc(struct fsl_mc_io *mc_io,
u32 cmd_flags,
u16 token,
char *obj_type,
int obj_id,
- struct dprc_obj_desc *obj_desc);
+ struct fsl_mc_obj_desc *obj_desc);
int dprc_set_obj_irq(struct fsl_mc_io *mc_io,
u32 cmd_flags,
#include <linux/module.h>
#include <linux/msi.h>
-#include "../include/mc-bus.h"
-#include "../include/mc-sys.h"
+#include "../include/mc.h"
-#include "dpbp-cmd.h"
-#include "dpcon-cmd.h"
#include "fsl-mc-private.h"
-#define FSL_MC_IS_ALLOCATABLE(_obj_type) \
- (strcmp(_obj_type, "dpbp") == 0 || \
- strcmp(_obj_type, "dpmcp") == 0 || \
- strcmp(_obj_type, "dpcon") == 0)
+static bool __must_check fsl_mc_is_allocatable(const char *obj_type)
+{
+ return strcmp(obj_type, "dpbp") ||
+ strcmp(obj_type, "dpmcp") ||
+ strcmp(obj_type, "dpcon");
+}
/**
* fsl_mc_resource_pool_add_device - add allocatable object to a resource
if (WARN_ON(pool_type < 0 || pool_type >= FSL_MC_NUM_POOL_TYPES))
goto out;
- if (WARN_ON(!FSL_MC_IS_ALLOCATABLE(mc_dev->obj_desc.type)))
+ if (WARN_ON(!fsl_mc_is_allocatable(mc_dev->obj_desc.type)))
goto out;
if (WARN_ON(mc_dev->resource))
goto out;
struct fsl_mc_resource *resource;
int error = -EINVAL;
- if (WARN_ON(!FSL_MC_IS_ALLOCATABLE(mc_dev->obj_desc.type)))
+ if (WARN_ON(!fsl_mc_is_allocatable(mc_dev->obj_desc.type)))
goto out;
resource = mc_dev->resource;
struct fsl_mc_bus *mc_bus;
int error;
- if (WARN_ON(!FSL_MC_IS_ALLOCATABLE(mc_dev->obj_desc.type)))
+ if (WARN_ON(!fsl_mc_is_allocatable(mc_dev->obj_desc.type)))
return -EINVAL;
mc_bus_dev = to_fsl_mc_device(mc_dev->dev.parent);
{
int error;
- if (WARN_ON(!FSL_MC_IS_ALLOCATABLE(mc_dev->obj_desc.type)))
+ if (WARN_ON(!fsl_mc_is_allocatable(mc_dev->obj_desc.type)))
return -EINVAL;
if (mc_dev->resource) {
#include <linux/bitops.h>
#include <linux/msi.h>
#include <linux/dma-mapping.h>
-#include "../include/mc-bus.h"
-#include "../include/dpmng.h"
-#include "../include/mc-sys.h"
#include "fsl-mc-private.h"
#include "dprc-cmd.h"
+#include "dpmng-cmd.h"
/**
* Default DMA mask for devices on a fsl-mc bus
phys_addr_t start_phys_addr;
};
+/**
+ * struct mc_version
+ * @major: Major version number: incremented on API compatibility changes
+ * @minor: Minor version number: incremented on API additions (that are
+ * backward compatible); reset when major version is incremented
+ * @revision: Internal revision number: incremented on implementation changes
+ * and/or bug fixes that have no impact on API
+ */
+struct mc_version {
+ u32 major;
+ u32 minor;
+ u32 revision;
+};
+
/**
* fsl_mc_bus_match - device to driver matching callback
* @dev: the fsl-mc device to match against
* If the object is not 'plugged' don't match.
* Only exception is the root DPRC, which is a special case.
*/
- if ((mc_dev->obj_desc.state & DPRC_OBJ_STATE_PLUGGED) == 0 &&
+ if ((mc_dev->obj_desc.state & FSL_MC_OBJ_STATE_PLUGGED) == 0 &&
!fsl_mc_is_root_dprc(&mc_dev->dev))
goto out;
}
EXPORT_SYMBOL_GPL(fsl_mc_driver_unregister);
+/**
+ * mc_get_version() - Retrieves the Management Complex firmware
+ * version information
+ * @mc_io: Pointer to opaque I/O object
+ * @cmd_flags: Command flags; one or more of 'MC_CMD_FLAG_'
+ * @mc_ver_info: Returned version information structure
+ *
+ * Return: '0' on Success; Error code otherwise.
+ */
+static int mc_get_version(struct fsl_mc_io *mc_io,
+ u32 cmd_flags,
+ struct mc_version *mc_ver_info)
+{
+ struct mc_command cmd = { 0 };
+ struct dpmng_rsp_get_version *rsp_params;
+ int err;
+
+ /* prepare command */
+ cmd.header = mc_encode_cmd_header(DPMNG_CMDID_GET_VERSION,
+ cmd_flags,
+ 0);
+
+ /* send command to mc*/
+ err = mc_send_command(mc_io, &cmd);
+ if (err)
+ return err;
+
+ /* retrieve response parameters */
+ rsp_params = (struct dpmng_rsp_get_version *)cmd.params;
+ mc_ver_info->revision = le32_to_cpu(rsp_params->revision);
+ mc_ver_info->major = le32_to_cpu(rsp_params->version_major);
+ mc_ver_info->minor = le32_to_cpu(rsp_params->version_minor);
+
+ return 0;
+}
+
/**
* fsl_mc_get_root_dprc - function to traverse to the root dprc
*/
-void fsl_mc_get_root_dprc(struct device *dev,
- struct device **root_dprc_dev)
+static void fsl_mc_get_root_dprc(struct device *dev,
+ struct device **root_dprc_dev)
{
if (WARN_ON(!dev)) {
*root_dprc_dev = NULL;
*root_dprc_dev = (*root_dprc_dev)->parent;
}
}
-EXPORT_SYMBOL_GPL(fsl_mc_get_root_dprc);
static int get_dprc_attr(struct fsl_mc_io *mc_io,
int container_id, struct dprc_attributes *attr)
int i;
int error;
struct resource *regions;
- struct dprc_obj_desc *obj_desc = &mc_dev->obj_desc;
+ struct fsl_mc_obj_desc *obj_desc = &mc_dev->obj_desc;
struct device *parent_dev = mc_dev->dev.parent;
enum dprc_region_type mc_region_type;
static void fsl_mc_device_release(struct device *dev)
{
struct fsl_mc_device *mc_dev = to_fsl_mc_device(dev);
- struct fsl_mc_bus *mc_bus = NULL;
kfree(mc_dev->regions);
if (strcmp(mc_dev->obj_desc.type, "dprc") == 0)
- mc_bus = to_fsl_mc_bus(mc_dev);
-
- if (mc_bus)
- kfree(mc_bus);
+ kfree(to_fsl_mc_bus(mc_dev));
else
kfree(mc_dev);
}
/**
* Add a newly discovered fsl-mc device to be visible in Linux
*/
-int fsl_mc_device_add(struct dprc_obj_desc *obj_desc,
+int fsl_mc_device_add(struct fsl_mc_obj_desc *obj_desc,
struct fsl_mc_io *mc_io,
struct device *parent_dev,
struct fsl_mc_device **new_mc_dev)
}
/* Objects are coherent, unless 'no shareability' flag set. */
- if (!(obj_desc->flags & DPRC_OBJ_FLAG_NO_MEM_SHAREABILITY))
+ if (!(obj_desc->flags & FSL_MC_OBJ_FLAG_NO_MEM_SHAREABILITY))
arch_setup_dma_ops(&mc_dev->dev, 0, 0, NULL, true);
/*
error_cleanup_dev:
kfree(mc_dev->regions);
- if (mc_bus)
- kfree(mc_bus);
- else
- kfree(mc_dev);
+ kfree(mc_bus);
+ kfree(mc_dev);
return error;
}
const __be32 *cell;
ret = parse_mc_ranges(dev,
- &paddr_cells,
- &mc_addr_cells,
- &mc_size_cells,
- &ranges_start);
+ &paddr_cells,
+ &mc_addr_cells,
+ &mc_size_cells,
+ &ranges_start);
if (ret < 0)
return ret;
*/
static int fsl_mc_bus_probe(struct platform_device *pdev)
{
- struct dprc_obj_desc obj_desc;
+ struct fsl_mc_obj_desc obj_desc;
int error;
struct fsl_mc *mc;
struct fsl_mc_device *mc_bus_dev = NULL;
goto error_cleanup_mc_io;
}
- memset(&obj_desc, 0, sizeof(struct dprc_obj_desc));
+ memset(&obj_desc, 0, sizeof(struct fsl_mc_obj_desc));
error = dprc_get_api_version(mc_io, 0,
&obj_desc.ver_major,
&obj_desc.ver_minor);
#include <linux/irq.h>
#include <linux/irqdomain.h>
#include <linux/msi.h>
-#include "../include/mc-bus.h"
#include "fsl-mc-private.h"
/*
domain = msi_create_irq_domain(fwnode, info, parent);
if (domain)
- domain->bus_token = DOMAIN_BUS_FSL_MC_MSI;
+ irq_domain_update_bus_token(domain, DOMAIN_BUS_FSL_MC_MSI);
return domain;
}
#define _FSL_MC_PRIVATE_H_
#include "../include/mc.h"
-#include "../include/mc-bus.h"
+#include "dprc.h"
+#include <linux/mutex.h>
-int __must_check fsl_mc_device_add(struct dprc_obj_desc *obj_desc,
+/**
+ * Maximum number of total IRQs that can be pre-allocated for an MC bus'
+ * IRQ pool
+ */
+#define FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS 256
+
+/**
+ * struct fsl_mc_resource_pool - Pool of MC resources of a given
+ * type
+ * @type: type of resources in the pool
+ * @max_count: maximum number of resources in the pool
+ * @free_count: number of free resources in the pool
+ * @mutex: mutex to serialize access to the pool's free list
+ * @free_list: anchor node of list of free resources in the pool
+ * @mc_bus: pointer to the MC bus that owns this resource pool
+ */
+struct fsl_mc_resource_pool {
+ enum fsl_mc_pool_type type;
+ int max_count;
+ int free_count;
+ struct mutex mutex; /* serializes access to free_list */
+ struct list_head free_list;
+ struct fsl_mc_bus *mc_bus;
+};
+
+/**
+ * struct fsl_mc_bus - logical bus that corresponds to a physical DPRC
+ * @mc_dev: fsl-mc device for the bus device itself.
+ * @resource_pools: array of resource pools (one pool per resource type)
+ * for this MC bus. These resources represent allocatable entities
+ * from the physical DPRC.
+ * @irq_resources: Pointer to array of IRQ objects for the IRQ pool
+ * @scan_mutex: Serializes bus scanning
+ * @dprc_attr: DPRC attributes
+ */
+struct fsl_mc_bus {
+ struct fsl_mc_device mc_dev;
+ struct fsl_mc_resource_pool resource_pools[FSL_MC_NUM_POOL_TYPES];
+ struct fsl_mc_device_irq *irq_resources;
+ struct mutex scan_mutex; /* serializes bus scanning */
+ struct dprc_attributes dprc_attr;
+};
+
+#define to_fsl_mc_bus(_mc_dev) \
+ container_of(_mc_dev, struct fsl_mc_bus, mc_dev)
+
+int __must_check fsl_mc_device_add(struct fsl_mc_obj_desc *obj_desc,
struct fsl_mc_io *mc_io,
struct device *parent_dev,
struct fsl_mc_device **new_mc_dev);
void fsl_mc_allocator_driver_exit(void);
+void fsl_mc_init_all_resource_pools(struct fsl_mc_device *mc_bus_dev);
+
+void fsl_mc_cleanup_all_resource_pools(struct fsl_mc_device *mc_bus_dev);
+
int __must_check fsl_mc_resource_allocate(struct fsl_mc_bus *mc_bus,
enum fsl_mc_pool_type pool_type,
struct fsl_mc_resource
void its_fsl_mc_msi_cleanup(void);
+int fsl_mc_find_msi_domain(struct device *mc_platform_dev,
+ struct irq_domain **mc_msi_domain);
+
+int fsl_mc_populate_irq_pool(struct fsl_mc_bus *mc_bus,
+ unsigned int irq_count);
+
+void fsl_mc_cleanup_irq_pool(struct fsl_mc_bus *mc_bus);
+
int __must_check fsl_create_mc_io(struct device *dev,
phys_addr_t mc_portal_phys_addr,
u32 mc_portal_size,
void fsl_destroy_mc_io(struct fsl_mc_io *mc_io);
+bool fsl_mc_is_root_dprc(struct device *dev);
+
#endif /* _FSL_MC_PRIVATE_H_ */
#include <linux/msi.h>
#include <linux/of.h>
#include <linux/of_irq.h>
-#include "../include/mc-bus.h"
#include "fsl-mc-private.h"
static struct irq_chip its_msi_irq_chip = {
*/
#include <linux/io.h>
-#include "../include/mc-bus.h"
-#include "../include/mc-sys.h"
+#include "../include/mc.h"
#include "fsl-mc-private.h"
#include "dpmcp.h"
#include <linux/ioport.h>
#include <linux/device.h>
#include <linux/io.h>
-#include "../include/mc-sys.h"
-#include "../include/mc-cmd.h"
#include "../include/mc.h"
#include "dpmcp.h"
+++ /dev/null
-/*
- * Copyright 2013-2016 Freescale Semiconductor Inc.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- * * Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * * Redistributions in binary form must reproduce the above copyright
- * notice, this list of conditions and the following disclaimer in the
- * documentation and/or other materials provided with the distribution.
- * * Neither the name of the above-listed copyright holders nor the
- * names of any contributors may be used to endorse or promote products
- * derived from this software without specific prior written permission.
- *
- *
- * ALTERNATIVELY, this software may be distributed under the terms of the
- * GNU General Public License ("GPL") as published by the Free Software
- * Foundation, either version 2 of that License or (at your option) any
- * later version.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-#ifndef __FSL_DPMNG_H
-#define __FSL_DPMNG_H
-
-/*
- * Management Complex General API
- * Contains general API for the Management Complex firmware
- */
-
-struct fsl_mc_io;
-
-/**
- * Management Complex firmware version information
- */
-#define MC_VER_MAJOR 8
-#define MC_VER_MINOR 0
-
-/**
- * struct mc_version
- * @major: Major version number: incremented on API compatibility changes
- * @minor: Minor version number: incremented on API additions (that are
- * backward compatible); reset when major version is incremented
- * @revision: Internal revision number: incremented on implementation changes
- * and/or bug fixes that have no impact on API
- */
-struct mc_version {
- u32 major;
- u32 minor;
- u32 revision;
-};
-
-int mc_get_version(struct fsl_mc_io *mc_io,
- u32 cmd_flags,
- struct mc_version *mc_ver_info);
-
-#endif /* __FSL_DPMNG_H */
+++ /dev/null
-/*
- * Freescale Management Complex (MC) bus declarations
- *
- * Copyright (C) 2014-2016 Freescale Semiconductor, Inc.
- * Author: German Rivera <German.Rivera@freescale.com>
- *
- * This file is licensed under the terms of the GNU General Public
- * License version 2. This program is licensed "as is" without any
- * warranty of any kind, whether express or implied.
- */
-#ifndef _FSL_MC_MCBUS_H_
-#define _FSL_MC_MCBUS_H_
-
-#include "../include/mc.h"
-#include <linux/mutex.h>
-
-struct irq_domain;
-struct msi_domain_info;
-
-/**
- * Maximum number of total IRQs that can be pre-allocated for an MC bus'
- * IRQ pool
- */
-#define FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS 256
-
-#ifdef CONFIG_FSL_MC_BUS
-#define dev_is_fsl_mc(_dev) ((_dev)->bus == &fsl_mc_bus_type)
-#else
-/* If fsl-mc bus is not present device cannot belong to fsl-mc bus */
-#define dev_is_fsl_mc(_dev) (0)
-#endif
-
-/**
- * struct fsl_mc_resource_pool - Pool of MC resources of a given
- * type
- * @type: type of resources in the pool
- * @max_count: maximum number of resources in the pool
- * @free_count: number of free resources in the pool
- * @mutex: mutex to serialize access to the pool's free list
- * @free_list: anchor node of list of free resources in the pool
- * @mc_bus: pointer to the MC bus that owns this resource pool
- */
-struct fsl_mc_resource_pool {
- enum fsl_mc_pool_type type;
- int max_count;
- int free_count;
- struct mutex mutex; /* serializes access to free_list */
- struct list_head free_list;
- struct fsl_mc_bus *mc_bus;
-};
-
-/**
- * struct fsl_mc_bus - logical bus that corresponds to a physical DPRC
- * @mc_dev: fsl-mc device for the bus device itself.
- * @resource_pools: array of resource pools (one pool per resource type)
- * for this MC bus. These resources represent allocatable entities
- * from the physical DPRC.
- * @irq_resources: Pointer to array of IRQ objects for the IRQ pool
- * @scan_mutex: Serializes bus scanning
- * @dprc_attr: DPRC attributes
- */
-struct fsl_mc_bus {
- struct fsl_mc_device mc_dev;
- struct fsl_mc_resource_pool resource_pools[FSL_MC_NUM_POOL_TYPES];
- struct fsl_mc_device_irq *irq_resources;
- struct mutex scan_mutex; /* serializes bus scanning */
- struct dprc_attributes dprc_attr;
-};
-
-#define to_fsl_mc_bus(_mc_dev) \
- container_of(_mc_dev, struct fsl_mc_bus, mc_dev)
-
-int dprc_scan_container(struct fsl_mc_device *mc_bus_dev);
-
-int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev,
- unsigned int *total_irq_count);
-
-int __init dprc_driver_init(void);
-
-void dprc_driver_exit(void);
-
-int __init fsl_mc_allocator_driver_init(void);
-
-void fsl_mc_allocator_driver_exit(void);
-
-struct irq_domain *fsl_mc_msi_create_irq_domain(struct fwnode_handle *fwnode,
- struct msi_domain_info *info,
- struct irq_domain *parent);
-
-int fsl_mc_find_msi_domain(struct device *mc_platform_dev,
- struct irq_domain **mc_msi_domain);
-
-int fsl_mc_populate_irq_pool(struct fsl_mc_bus *mc_bus,
- unsigned int irq_count);
-
-void fsl_mc_cleanup_irq_pool(struct fsl_mc_bus *mc_bus);
-
-void fsl_mc_init_all_resource_pools(struct fsl_mc_device *mc_bus_dev);
-
-void fsl_mc_cleanup_all_resource_pools(struct fsl_mc_device *mc_bus_dev);
-
-bool fsl_mc_bus_exists(void);
-
-void fsl_mc_get_root_dprc(struct device *dev,
- struct device **root_dprc_dev);
-
-bool fsl_mc_is_root_dprc(struct device *dev);
-
-extern struct bus_type fsl_mc_bus_type;
-
-#endif /* _FSL_MC_MCBUS_H_ */
+++ /dev/null
-/*
- * Copyright 2013-2016 Freescale Semiconductor Inc.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- * * Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * * Redistributions in binary form must reproduce the above copyright
- * notice, this list of conditions and the following disclaimer in the
- * documentation and/or other materials provided with the distribution.
- * * Neither the name of the above-listed copyright holders nor the
- * names of any contributors may be used to endorse or promote products
- * derived from this software without specific prior written permission.
- *
- *
- * ALTERNATIVELY, this software may be distributed under the terms of the
- * GNU General Public License ("GPL") as published by the Free Software
- * Foundation, either version 2 of that License or (at your option) any
- * later version.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-#ifndef __FSL_MC_CMD_H
-#define __FSL_MC_CMD_H
-
-#define MC_CMD_NUM_OF_PARAMS 7
-
-struct mc_cmd_header {
- u8 src_id;
- u8 flags_hw;
- u8 status;
- u8 flags_sw;
- __le16 token;
- __le16 cmd_id;
-};
-
-struct mc_command {
- u64 header;
- u64 params[MC_CMD_NUM_OF_PARAMS];
-};
-
-struct mc_rsp_create {
- __le32 object_id;
-};
-
-struct mc_rsp_api_ver {
- __le16 major_ver;
- __le16 minor_ver;
-};
-
-enum mc_cmd_status {
- MC_CMD_STATUS_OK = 0x0, /* Completed successfully */
- MC_CMD_STATUS_READY = 0x1, /* Ready to be processed */
- MC_CMD_STATUS_AUTH_ERR = 0x3, /* Authentication error */
- MC_CMD_STATUS_NO_PRIVILEGE = 0x4, /* No privilege */
- MC_CMD_STATUS_DMA_ERR = 0x5, /* DMA or I/O error */
- MC_CMD_STATUS_CONFIG_ERR = 0x6, /* Configuration error */
- MC_CMD_STATUS_TIMEOUT = 0x7, /* Operation timed out */
- MC_CMD_STATUS_NO_RESOURCE = 0x8, /* No resources */
- MC_CMD_STATUS_NO_MEMORY = 0x9, /* No memory available */
- MC_CMD_STATUS_BUSY = 0xA, /* Device is busy */
- MC_CMD_STATUS_UNSUPPORTED_OP = 0xB, /* Unsupported operation */
- MC_CMD_STATUS_INVALID_STATE = 0xC /* Invalid state */
-};
-
-/*
- * MC command flags
- */
-
-/* High priority flag */
-#define MC_CMD_FLAG_PRI 0x80
-/* Command completion flag */
-#define MC_CMD_FLAG_INTR_DIS 0x01
-
-static inline u64 mc_encode_cmd_header(u16 cmd_id,
- u32 cmd_flags,
- u16 token)
-{
- u64 header = 0;
- struct mc_cmd_header *hdr = (struct mc_cmd_header *)&header;
-
- hdr->cmd_id = cpu_to_le16(cmd_id);
- hdr->token = cpu_to_le16(token);
- hdr->status = MC_CMD_STATUS_READY;
- if (cmd_flags & MC_CMD_FLAG_PRI)
- hdr->flags_hw = MC_CMD_FLAG_PRI;
- if (cmd_flags & MC_CMD_FLAG_INTR_DIS)
- hdr->flags_sw = MC_CMD_FLAG_INTR_DIS;
-
- return header;
-}
-
-static inline u16 mc_cmd_hdr_read_token(struct mc_command *cmd)
-{
- struct mc_cmd_header *hdr = (struct mc_cmd_header *)&cmd->header;
- u16 token = le16_to_cpu(hdr->token);
-
- return token;
-}
-
-static inline u32 mc_cmd_read_object_id(struct mc_command *cmd)
-{
- struct mc_rsp_create *rsp_params;
-
- rsp_params = (struct mc_rsp_create *)cmd->params;
- return le32_to_cpu(rsp_params->object_id);
-}
-
-static inline void mc_cmd_read_api_version(struct mc_command *cmd,
- u16 *major_ver,
- u16 *minor_ver)
-{
- struct mc_rsp_api_ver *rsp_params;
-
- rsp_params = (struct mc_rsp_api_ver *)cmd->params;
- *major_ver = le16_to_cpu(rsp_params->major_ver);
- *minor_ver = le16_to_cpu(rsp_params->minor_ver);
-}
-
-#endif /* __FSL_MC_CMD_H */
+++ /dev/null
-/*
- * Copyright 2013-2016 Freescale Semiconductor Inc.
- *
- * Interface of the I/O services to send MC commands to the MC hardware
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- * * Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * * Redistributions in binary form must reproduce the above copyright
- * notice, this list of conditions and the following disclaimer in the
- * documentation and/or other materials provided with the distribution.
- * * Neither the name of the above-listed copyright holders nor the
- * names of any contributors may be used to endorse or promote products
- * derived from this software without specific prior written permission.
- *
- *
- * ALTERNATIVELY, this software may be distributed under the terms of the
- * GNU General Public License ("GPL") as published by the Free Software
- * Foundation, either version 2 of that License or (at your option) any
- * later version.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef _FSL_MC_SYS_H
-#define _FSL_MC_SYS_H
-
-#include <linux/types.h>
-#include <linux/errno.h>
-#include <linux/mutex.h>
-#include <linux/spinlock.h>
-
-/**
- * Bit masks for a MC I/O object (struct fsl_mc_io) flags
- */
-#define FSL_MC_IO_ATOMIC_CONTEXT_PORTAL 0x0001
-
-struct fsl_mc_resource;
-struct mc_command;
-
-/**
- * struct fsl_mc_io - MC I/O object to be passed-in to mc_send_command()
- * @dev: device associated with this Mc I/O object
- * @flags: flags for mc_send_command()
- * @portal_size: MC command portal size in bytes
- * @portal_phys_addr: MC command portal physical address
- * @portal_virt_addr: MC command portal virtual address
- * @dpmcp_dev: pointer to the DPMCP device associated with the MC portal.
- *
- * Fields are only meaningful if the FSL_MC_IO_ATOMIC_CONTEXT_PORTAL flag is not
- * set:
- * @mutex: Mutex to serialize mc_send_command() calls that use the same MC
- * portal, if the fsl_mc_io object was created with the
- * FSL_MC_IO_ATOMIC_CONTEXT_PORTAL flag off. mc_send_command() calls for this
- * fsl_mc_io object must be made only from non-atomic context.
- *
- * Fields are only meaningful if the FSL_MC_IO_ATOMIC_CONTEXT_PORTAL flag is
- * set:
- * @spinlock: Spinlock to serialize mc_send_command() calls that use the same MC
- * portal, if the fsl_mc_io object was created with the
- * FSL_MC_IO_ATOMIC_CONTEXT_PORTAL flag on. mc_send_command() calls for this
- * fsl_mc_io object can be made from atomic or non-atomic context.
- */
-struct fsl_mc_io {
- struct device *dev;
- u16 flags;
- u16 portal_size;
- phys_addr_t portal_phys_addr;
- void __iomem *portal_virt_addr;
- struct fsl_mc_device *dpmcp_dev;
- union {
- /*
- * This field is only meaningful if the
- * FSL_MC_IO_ATOMIC_CONTEXT_PORTAL flag is not set
- */
- struct mutex mutex; /* serializes mc_send_command() */
-
- /*
- * This field is only meaningful if the
- * FSL_MC_IO_ATOMIC_CONTEXT_PORTAL flag is set
- */
- spinlock_t spinlock; /* serializes mc_send_command() */
- };
-};
-
-int mc_send_command(struct fsl_mc_io *mc_io, struct mc_command *cmd);
-
-#endif /* _FSL_MC_SYS_H */
#include <linux/device.h>
#include <linux/mod_devicetable.h>
#include <linux/interrupt.h>
-#include "../include/dprc.h"
#define FSL_MC_VENDOR_FREESCALE 0x1957
+struct irq_domain;
+struct msi_domain_info;
+
struct fsl_mc_device;
struct fsl_mc_io;
#define to_fsl_mc_irq(_mc_resource) \
container_of(_mc_resource, struct fsl_mc_device_irq, resource)
+/* Opened state - Indicates that an object is open by at least one owner */
+#define FSL_MC_OBJ_STATE_OPEN 0x00000001
+/* Plugged state - Indicates that the object is plugged */
+#define FSL_MC_OBJ_STATE_PLUGGED 0x00000002
+
+/**
+ * Shareability flag - Object flag indicating no memory shareability.
+ * the object generates memory accesses that are non coherent with other
+ * masters;
+ * user is responsible for proper memory handling through IOMMU configuration.
+ */
+#define FSL_MC_OBJ_FLAG_NO_MEM_SHAREABILITY 0x0001
+
+/**
+ * struct fsl_mc_obj_desc - Object descriptor
+ * @type: Type of object: NULL terminated string
+ * @id: ID of logical object resource
+ * @vendor: Object vendor identifier
+ * @ver_major: Major version number
+ * @ver_minor: Minor version number
+ * @irq_count: Number of interrupts supported by the object
+ * @region_count: Number of mappable regions supported by the object
+ * @state: Object state: combination of FSL_MC_OBJ_STATE_ states
+ * @label: Object label: NULL terminated string
+ * @flags: Object's flags
+ */
+struct fsl_mc_obj_desc {
+ char type[16];
+ int id;
+ u16 vendor;
+ u16 ver_major;
+ u16 ver_minor;
+ u8 irq_count;
+ u8 region_count;
+ u32 state;
+ char label[16];
+ u16 flags;
+};
+
/**
* Bit masks for a MC object device (struct fsl_mc_device) flags
*/
u16 icid;
u16 mc_handle;
struct fsl_mc_io *mc_io;
- struct dprc_obj_desc obj_desc;
+ struct fsl_mc_obj_desc obj_desc;
struct resource *regions;
struct fsl_mc_device_irq **irqs;
struct fsl_mc_resource *resource;
#define to_fsl_mc_device(_dev) \
container_of(_dev, struct fsl_mc_device, dev)
+#define MC_CMD_NUM_OF_PARAMS 7
+
+struct mc_cmd_header {
+ u8 src_id;
+ u8 flags_hw;
+ u8 status;
+ u8 flags_sw;
+ __le16 token;
+ __le16 cmd_id;
+};
+
+struct mc_command {
+ u64 header;
+ u64 params[MC_CMD_NUM_OF_PARAMS];
+};
+
+enum mc_cmd_status {
+ MC_CMD_STATUS_OK = 0x0, /* Completed successfully */
+ MC_CMD_STATUS_READY = 0x1, /* Ready to be processed */
+ MC_CMD_STATUS_AUTH_ERR = 0x3, /* Authentication error */
+ MC_CMD_STATUS_NO_PRIVILEGE = 0x4, /* No privilege */
+ MC_CMD_STATUS_DMA_ERR = 0x5, /* DMA or I/O error */
+ MC_CMD_STATUS_CONFIG_ERR = 0x6, /* Configuration error */
+ MC_CMD_STATUS_TIMEOUT = 0x7, /* Operation timed out */
+ MC_CMD_STATUS_NO_RESOURCE = 0x8, /* No resources */
+ MC_CMD_STATUS_NO_MEMORY = 0x9, /* No memory available */
+ MC_CMD_STATUS_BUSY = 0xA, /* Device is busy */
+ MC_CMD_STATUS_UNSUPPORTED_OP = 0xB, /* Unsupported operation */
+ MC_CMD_STATUS_INVALID_STATE = 0xC /* Invalid state */
+};
+
+/*
+ * MC command flags
+ */
+
+/* High priority flag */
+#define MC_CMD_FLAG_PRI 0x80
+/* Command completion flag */
+#define MC_CMD_FLAG_INTR_DIS 0x01
+
+static inline u64 mc_encode_cmd_header(u16 cmd_id,
+ u32 cmd_flags,
+ u16 token)
+{
+ u64 header = 0;
+ struct mc_cmd_header *hdr = (struct mc_cmd_header *)&header;
+
+ hdr->cmd_id = cpu_to_le16(cmd_id);
+ hdr->token = cpu_to_le16(token);
+ hdr->status = MC_CMD_STATUS_READY;
+ if (cmd_flags & MC_CMD_FLAG_PRI)
+ hdr->flags_hw = MC_CMD_FLAG_PRI;
+ if (cmd_flags & MC_CMD_FLAG_INTR_DIS)
+ hdr->flags_sw = MC_CMD_FLAG_INTR_DIS;
+
+ return header;
+}
+
+static inline u16 mc_cmd_hdr_read_token(struct mc_command *cmd)
+{
+ struct mc_cmd_header *hdr = (struct mc_cmd_header *)&cmd->header;
+ u16 token = le16_to_cpu(hdr->token);
+
+ return token;
+}
+
+struct mc_rsp_create {
+ __le32 object_id;
+};
+
+struct mc_rsp_api_ver {
+ __le16 major_ver;
+ __le16 minor_ver;
+};
+
+static inline u32 mc_cmd_read_object_id(struct mc_command *cmd)
+{
+ struct mc_rsp_create *rsp_params;
+
+ rsp_params = (struct mc_rsp_create *)cmd->params;
+ return le32_to_cpu(rsp_params->object_id);
+}
+
+static inline void mc_cmd_read_api_version(struct mc_command *cmd,
+ u16 *major_ver,
+ u16 *minor_ver)
+{
+ struct mc_rsp_api_ver *rsp_params;
+
+ rsp_params = (struct mc_rsp_api_ver *)cmd->params;
+ *major_ver = le16_to_cpu(rsp_params->major_ver);
+ *minor_ver = le16_to_cpu(rsp_params->minor_ver);
+}
+
+/**
+ * Bit masks for a MC I/O object (struct fsl_mc_io) flags
+ */
+#define FSL_MC_IO_ATOMIC_CONTEXT_PORTAL 0x0001
+
+/**
+ * struct fsl_mc_io - MC I/O object to be passed-in to mc_send_command()
+ * @dev: device associated with this Mc I/O object
+ * @flags: flags for mc_send_command()
+ * @portal_size: MC command portal size in bytes
+ * @portal_phys_addr: MC command portal physical address
+ * @portal_virt_addr: MC command portal virtual address
+ * @dpmcp_dev: pointer to the DPMCP device associated with the MC portal.
+ *
+ * Fields are only meaningful if the FSL_MC_IO_ATOMIC_CONTEXT_PORTAL flag is not
+ * set:
+ * @mutex: Mutex to serialize mc_send_command() calls that use the same MC
+ * portal, if the fsl_mc_io object was created with the
+ * FSL_MC_IO_ATOMIC_CONTEXT_PORTAL flag off. mc_send_command() calls for this
+ * fsl_mc_io object must be made only from non-atomic context.
+ *
+ * Fields are only meaningful if the FSL_MC_IO_ATOMIC_CONTEXT_PORTAL flag is
+ * set:
+ * @spinlock: Spinlock to serialize mc_send_command() calls that use the same MC
+ * portal, if the fsl_mc_io object was created with the
+ * FSL_MC_IO_ATOMIC_CONTEXT_PORTAL flag on. mc_send_command() calls for this
+ * fsl_mc_io object can be made from atomic or non-atomic context.
+ */
+struct fsl_mc_io {
+ struct device *dev;
+ u16 flags;
+ u16 portal_size;
+ phys_addr_t portal_phys_addr;
+ void __iomem *portal_virt_addr;
+ struct fsl_mc_device *dpmcp_dev;
+ union {
+ /*
+ * This field is only meaningful if the
+ * FSL_MC_IO_ATOMIC_CONTEXT_PORTAL flag is not set
+ */
+ struct mutex mutex; /* serializes mc_send_command() */
+
+ /*
+ * This field is only meaningful if the
+ * FSL_MC_IO_ATOMIC_CONTEXT_PORTAL flag is set
+ */
+ spinlock_t spinlock; /* serializes mc_send_command() */
+ };
+};
+
+int mc_send_command(struct fsl_mc_io *mc_io, struct mc_command *cmd);
+
+#ifdef CONFIG_FSL_MC_BUS
+#define dev_is_fsl_mc(_dev) ((_dev)->bus == &fsl_mc_bus_type)
+#else
+/* If fsl-mc bus is not present device cannot belong to fsl-mc bus */
+#define dev_is_fsl_mc(_dev) (0)
+#endif
+
/*
* module_fsl_mc_driver() - Helper macro for drivers that don't do
* anything special in module init/exit. This eliminates a lot of
void fsl_mc_object_free(struct fsl_mc_device *mc_adev);
+struct irq_domain *fsl_mc_msi_create_irq_domain(struct fwnode_handle *fwnode,
+ struct msi_domain_info *info,
+ struct irq_domain *parent);
+
int __must_check fsl_mc_allocate_irqs(struct fsl_mc_device *mc_dev);
void fsl_mc_free_irqs(struct fsl_mc_device *mc_dev);
+extern struct bus_type fsl_mc_bus_type;
+
#endif /* _FSL_MC_H_ */
sdu->cmd_evt = gdm_cpu_to_dev16(&udev->gdm_ed, LTE_TX_SDU);
if (nic_type == NIC_TYPE_ARP) {
send_len = len + SDU_PARAM_LEN;
- memcpy(sdu->data, data, len);
+ memcpy(sdu->data, data, len);
} else {
- send_len = len - ETH_HLEN;
- send_len += SDU_PARAM_LEN;
- memcpy(sdu->data, data + ETH_HLEN, len - ETH_HLEN);
+ send_len = len - ETH_HLEN;
+ send_len += SDU_PARAM_LEN;
+ memcpy(sdu->data, data + ETH_HLEN, len - ETH_HLEN);
}
sdu->len = gdm_cpu_to_dev16(&udev->gdm_ed, send_len);
will be called gb-usb.ko
endif # GREYBUS_BRIDGED_PHY
+
+config GREYBUS_ARCHE
+ tristate "Greybus Arche Platform driver"
+ depends on USB_HSIC_USB3613 || COMPILE_TEST
+ ---help---
+ Select this option if you have an Arche device.
+
+ To compile this code as a module, chose M here: the module
+ will be called gb-arche.ko
+
endif # GREYBUS
# Greybus Platform driver
gb-arche-y := arche-platform.o arche-apb-ctrl.o
-obj-$(CONFIG_USB_HSIC_USB3613) += gb-arche.o
+obj-$(CONFIG_GREYBUS_ARCHE) += gb-arche.o
#include "arche_platform.h"
+static void apb_bootret_deassert(struct device *dev);
+
struct arche_apb_ctrl_drvdata {
/* Control GPIO signals to and from AP <=> AP Bridges */
int resetn_gpio;
/* TODO: May have to send an event to SVC about this exit */
}
-void apb_bootret_assert(struct device *dev)
-{
- struct arche_apb_ctrl_drvdata *apb = dev_get_drvdata(dev);
-
- gpio_set_value(apb->boot_ret_gpio, 1);
-}
-
-void apb_bootret_deassert(struct device *dev)
+static void apb_bootret_deassert(struct device *dev)
{
struct arche_apb_ctrl_drvdata *apb = dev_get_drvdata(dev);
#include "arche_platform.h"
#include "greybus.h"
+#if IS_ENABLED(CONFIG_USB_HSIC_USB3613)
#include <linux/usb/usb3613.h>
+#else
+static inline int usb3613_hub_mode_ctrl(bool unused)
+{
+ return 0;
+}
+#endif
#define WD_COLDBOOT_PULSE_WIDTH_MS 30
WD_STATE_STANDBYBOOT_TRIG, /* As of now not used ?? */
WD_STATE_COLDBOOT_START, /* Cold boot process started */
WD_STATE_STANDBYBOOT_START, /* Not used */
- WD_STATE_TIMESYNC,
};
struct arche_platform_drvdata {
int wake_detect_irq;
spinlock_t wake_lock; /* Protect wake_detect_state */
struct mutex platform_state_mutex; /* Protect state */
- wait_queue_head_t wq; /* WQ for arche_pdata->state */
unsigned long wake_detect_start;
struct notifier_block pm_notifier;
struct device *dev;
- struct gb_timesync_svc *timesync_svc_pdata;
};
-static int arche_apb_bootret_assert(struct device *dev, void *data)
-{
- apb_bootret_assert(dev);
- return 0;
-}
-
-static int arche_apb_bootret_deassert(struct device *dev, void *data)
-{
- apb_bootret_deassert(dev);
- return 0;
-}
-
/* Requires calling context to hold arche_pdata->platform_state_mutex */
static void arche_platform_set_state(struct arche_platform_drvdata *arche_pdata,
enum arche_platform_state state)
arche_pdata->state = state;
}
-/*
- * arche_platform_change_state: Change the operational state
- *
- * This exported function allows external drivers to change the state
- * of the arche-platform driver.
- * Note that this function only supports transitions between two states
- * with limited functionality.
- *
- * - ARCHE_PLATFORM_STATE_TIME_SYNC:
- * Once set, allows timesync operations between SVC <=> AP and makes
- * sure that arche-platform driver ignores any subsequent events/pulses
- * from SVC over wake/detect.
- *
- * - ARCHE_PLATFORM_STATE_ACTIVE:
- * Puts back driver to active state, where any pulse from SVC on wake/detect
- * line would trigger either cold/standby boot.
- * Note: Transition request from this function does not trigger cold/standby
- * boot. It just puts back driver book keeping variable back to ACTIVE
- * state and restores the interrupt.
- *
- * Returns -ENODEV if device not found, -EAGAIN if the driver cannot currently
- * satisfy the requested state-transition or -EINVAL for all other
- * state-transition requests.
- */
-int arche_platform_change_state(enum arche_platform_state state,
- struct gb_timesync_svc *timesync_svc_pdata)
-{
- struct arche_platform_drvdata *arche_pdata;
- struct platform_device *pdev;
- struct device_node *np;
- int ret = -EAGAIN;
- unsigned long flags;
-
- np = of_find_compatible_node(NULL, NULL, "google,arche-platform");
- if (!np) {
- pr_err("google,arche-platform device node not found\n");
- return -ENODEV;
- }
-
- pdev = of_find_device_by_node(np);
- if (!pdev) {
- pr_err("arche-platform device not found\n");
- of_node_put(np);
- return -ENODEV;
- }
-
- arche_pdata = platform_get_drvdata(pdev);
-
- mutex_lock(&arche_pdata->platform_state_mutex);
- spin_lock_irqsave(&arche_pdata->wake_lock, flags);
-
- if (arche_pdata->state == state) {
- ret = 0;
- goto exit;
- }
-
- switch (state) {
- case ARCHE_PLATFORM_STATE_TIME_SYNC:
- if (arche_pdata->state != ARCHE_PLATFORM_STATE_ACTIVE) {
- ret = -EINVAL;
- goto exit;
- }
- if (arche_pdata->wake_detect_state != WD_STATE_IDLE) {
- dev_err(arche_pdata->dev,
- "driver busy with wake/detect line ops\n");
- goto exit;
- }
- device_for_each_child(arche_pdata->dev, NULL,
- arche_apb_bootret_assert);
- arche_pdata->wake_detect_state = WD_STATE_TIMESYNC;
- break;
- case ARCHE_PLATFORM_STATE_ACTIVE:
- if (arche_pdata->state != ARCHE_PLATFORM_STATE_TIME_SYNC) {
- ret = -EINVAL;
- goto exit;
- }
- device_for_each_child(arche_pdata->dev, NULL,
- arche_apb_bootret_deassert);
- arche_pdata->wake_detect_state = WD_STATE_IDLE;
- break;
- case ARCHE_PLATFORM_STATE_OFF:
- case ARCHE_PLATFORM_STATE_STANDBY:
- case ARCHE_PLATFORM_STATE_FW_FLASHING:
- dev_err(arche_pdata->dev, "busy, request to retry later\n");
- goto exit;
- default:
- ret = -EINVAL;
- dev_err(arche_pdata->dev,
- "invalid state transition request\n");
- goto exit;
- }
- arche_pdata->timesync_svc_pdata = timesync_svc_pdata;
- arche_platform_set_state(arche_pdata, state);
- if (state == ARCHE_PLATFORM_STATE_ACTIVE)
- wake_up(&arche_pdata->wq);
-
- ret = 0;
-exit:
- spin_unlock_irqrestore(&arche_pdata->wake_lock, flags);
- mutex_unlock(&arche_pdata->platform_state_mutex);
- put_device(&pdev->dev);
- of_node_put(np);
- return ret;
-}
-EXPORT_SYMBOL_GPL(arche_platform_change_state);
-
/* Requires arche_pdata->wake_lock is held by calling context */
static void arche_platform_set_wake_detect_state(
struct arche_platform_drvdata *arche_pdata,
spin_lock_irqsave(&arche_pdata->wake_lock, flags);
- if (arche_pdata->wake_detect_state == WD_STATE_TIMESYNC) {
- gb_timesync_irq(arche_pdata->timesync_svc_pdata);
- goto exit;
- }
-
if (gpio_get_value(arche_pdata->wake_detect_gpio)) {
/* wake/detect rising */
}
}
-exit:
spin_unlock_irqrestore(&arche_pdata->wake_lock, flags);
return IRQ_HANDLED;
struct arche_platform_drvdata *arche_pdata = platform_get_drvdata(pdev);
int ret = 0;
-retry:
mutex_lock(&arche_pdata->platform_state_mutex);
- if (arche_pdata->state == ARCHE_PLATFORM_STATE_TIME_SYNC) {
- mutex_unlock(&arche_pdata->platform_state_mutex);
- ret = wait_event_interruptible(
- arche_pdata->wq,
- arche_pdata->state != ARCHE_PLATFORM_STATE_TIME_SYNC);
- if (ret)
- return ret;
- goto retry;
- }
if (sysfs_streq(buf, "off")) {
if (arche_pdata->state == ARCHE_PLATFORM_STATE_OFF)
return sprintf(buf, "standby\n");
case ARCHE_PLATFORM_STATE_FW_FLASHING:
return sprintf(buf, "fw_flashing\n");
- case ARCHE_PLATFORM_STATE_TIME_SYNC:
- return sprintf(buf, "time_sync\n");
default:
return sprintf(buf, "unknown state\n");
}
spin_lock_init(&arche_pdata->wake_lock);
mutex_init(&arche_pdata->platform_state_mutex);
- init_waitqueue_head(&arche_pdata->wq);
arche_pdata->wake_detect_irq =
gpio_to_irq(arche_pdata->wake_detect_gpio);
goto err_device_remove;
}
- /* Register callback pointer */
- arche_platform_change_state_cb = arche_platform_change_state;
-
/* Explicitly power off if requested */
if (!of_property_read_bool(pdev->dev.of_node, "arche,init-off")) {
mutex_lock(&arche_pdata->platform_state_mutex);
return 0;
}
-static int arche_platform_suspend(struct device *dev)
+static __maybe_unused int arche_platform_suspend(struct device *dev)
{
/*
* If timing profile premits, we may shutdown bridge
return 0;
}
-static int arche_platform_resume(struct device *dev)
+static __maybe_unused int arche_platform_resume(struct device *dev)
{
/*
* Atleast for ES2 we have to meet the delay requirement between
ARCHE_PLATFORM_STATE_ACTIVE,
ARCHE_PLATFORM_STATE_STANDBY,
ARCHE_PLATFORM_STATE_FW_FLASHING,
- ARCHE_PLATFORM_STATE_TIME_SYNC,
};
-int arche_platform_change_state(enum arche_platform_state state,
- struct gb_timesync_svc *pdata);
-
-extern int (*arche_platform_change_state_cb)(enum arche_platform_state state,
- struct gb_timesync_svc *pdata);
int __init arche_apb_init(void);
void __exit arche_apb_exit(void);
int apb_ctrl_fw_flashing(struct device *dev);
int apb_ctrl_standby_boot(struct device *dev);
void apb_ctrl_poweroff(struct device *dev);
-void apb_bootret_assert(struct device *dev);
-void apb_bootret_deassert(struct device *dev);
#endif /* __ARCHE_PLATFORM_H */
light->channels_count = conf.channel_count;
light->name = kstrndup(conf.name, NAMES_MAX, GFP_KERNEL);
- light->channels = kzalloc(light->channels_count *
+ light->channels = kcalloc(light->channels_count,
sizeof(struct gb_channel), GFP_KERNEL);
if (!light->channels)
return -ENOMEM;
if (ret < 0)
goto out;
- glights->lights = kzalloc(glights->lights_count *
+ glights->lights = kcalloc(glights->lights_count,
sizeof(struct gb_light), GFP_KERNEL);
if (!glights->lights) {
ret = -ENOMEM;
if (ret < 0)
goto out;
- supplies->supply = kzalloc(supplies->supplies_count *
+ supplies->supply = kcalloc(supplies->supplies_count,
sizeof(struct gb_power_supply),
GFP_KERNEL);
return sprintf(buf, "%s\n", str);
}
-static IIO_DEVICE_ATTR(out_altvoltage0_out0_wavetype_available, S_IRUGO,
+static IIO_DEVICE_ATTR(out_altvoltage0_out0_wavetype_available, 0444,
ad9834_show_out0_wavetype_available, NULL, 0);
static
return sprintf(buf, "%s\n", str);
}
-static IIO_DEVICE_ATTR(out_altvoltage0_out1_wavetype_available, S_IRUGO,
+static IIO_DEVICE_ATTR(out_altvoltage0_out1_wavetype_available, 0444,
ad9834_show_out1_wavetype_available, NULL, 0);
/**
* see dds.h for further information
*/
-static IIO_DEV_ATTR_FREQ(0, 0, S_IWUSR, NULL, ad9834_write, AD9834_REG_FREQ0);
-static IIO_DEV_ATTR_FREQ(0, 1, S_IWUSR, NULL, ad9834_write, AD9834_REG_FREQ1);
-static IIO_DEV_ATTR_FREQSYMBOL(0, S_IWUSR, NULL, ad9834_write, AD9834_FSEL);
+static IIO_DEV_ATTR_FREQ(0, 0, 0200, NULL, ad9834_write, AD9834_REG_FREQ0);
+static IIO_DEV_ATTR_FREQ(0, 1, 0200, NULL, ad9834_write, AD9834_REG_FREQ1);
+static IIO_DEV_ATTR_FREQSYMBOL(0, 0200, NULL, ad9834_write, AD9834_FSEL);
static IIO_CONST_ATTR_FREQ_SCALE(0, "1"); /* 1Hz */
-static IIO_DEV_ATTR_PHASE(0, 0, S_IWUSR, NULL, ad9834_write, AD9834_REG_PHASE0);
-static IIO_DEV_ATTR_PHASE(0, 1, S_IWUSR, NULL, ad9834_write, AD9834_REG_PHASE1);
-static IIO_DEV_ATTR_PHASESYMBOL(0, S_IWUSR, NULL, ad9834_write, AD9834_PSEL);
+static IIO_DEV_ATTR_PHASE(0, 0, 0200, NULL, ad9834_write, AD9834_REG_PHASE0);
+static IIO_DEV_ATTR_PHASE(0, 1, 0200, NULL, ad9834_write, AD9834_REG_PHASE1);
+static IIO_DEV_ATTR_PHASESYMBOL(0, 0200, NULL, ad9834_write, AD9834_PSEL);
static IIO_CONST_ATTR_PHASE_SCALE(0, "0.0015339808"); /* 2PI/2^12 rad*/
-static IIO_DEV_ATTR_PINCONTROL_EN(0, S_IWUSR, NULL,
+static IIO_DEV_ATTR_PINCONTROL_EN(0, 0200, NULL,
ad9834_write, AD9834_PIN_SW);
-static IIO_DEV_ATTR_OUT_ENABLE(0, S_IWUSR, NULL, ad9834_write, AD9834_RESET);
-static IIO_DEV_ATTR_OUTY_ENABLE(0, 1, S_IWUSR, NULL,
+static IIO_DEV_ATTR_OUT_ENABLE(0, 0200, NULL, ad9834_write, AD9834_RESET);
+static IIO_DEV_ATTR_OUTY_ENABLE(0, 1, 0200, NULL,
ad9834_write, AD9834_OPBITEN);
static IIO_DEV_ATTR_OUT_WAVETYPE(0, 0, ad9834_store_wavetype, 0);
static IIO_DEV_ATTR_OUT_WAVETYPE(0, 1, ad9834_store_wavetype, 1);
#define IIO_DEV_ATTR_OUT_WAVETYPE(_channel, _output, _store, _addr) \
IIO_DEVICE_ATTR(out_altvoltage##_channel##_out##_output##_wavetype,\
- S_IWUSR, NULL, _store, _addr)
+ 0200, NULL, _store, _addr)
/**
* /sys/bus/iio/devices/.../out_altvoltageX_outY_wavetype_available
#
menu "Light sensors"
-config SENSORS_ISL29028
- tristate "Intersil ISL29028 Concurrent Light and Proximity Sensor"
- depends on I2C
- select REGMAP_I2C
- help
- Provides driver for the Intersil's ISL29028 device.
- This driver supports the sysfs interface to get the ALS, IR intensity,
- Proximity value via iio. The ISL29028 provides the concurrent sensing
- of ambient light and proximity.
-
config TSL2x7x
tristate "TAOS TSL/TMD2x71 and TSL/TMD2x72 Family of light and proximity sensors"
depends on I2C
# Makefile for industrial I/O Light sensors
#
-obj-$(CONFIG_SENSORS_ISL29028) += isl29028.o
-obj-$(CONFIG_TSL2x7x) += tsl2x7x_core.o
+obj-$(CONFIG_TSL2x7x) += tsl2x7x.o
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
* more details.
- *
- * You should have received a copy of the GNU General Public License along
- * with this program; if not, write to the Free Software Foundation, Inc.,
- * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#include <linux/kernel.h>
tsl2x7x_chip_on(indio_dev);
}
-static ssize_t tsl2x7x_power_state_show(struct device *dev,
+static ssize_t power_state_show(struct device *dev,
struct device_attribute *attr,
char *buf)
{
return snprintf(buf, PAGE_SIZE, "%d\n", chip->tsl2x7x_chip_status);
}
-static ssize_t tsl2x7x_power_state_store(struct device *dev,
+static ssize_t power_state_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t len)
{
return len;
}
-static ssize_t tsl2x7x_gain_available_show(struct device *dev,
+static ssize_t in_illuminance0_calibscale_available_show(struct device *dev,
struct device_attribute *attr,
char *buf)
{
return snprintf(buf, PAGE_SIZE, "%s\n", "1 8 16 120");
}
-static ssize_t tsl2x7x_prox_gain_available_show(struct device *dev,
+static ssize_t in_proximity0_calibscale_available_show(struct device *dev,
struct device_attribute *attr,
char *buf)
{
return snprintf(buf, PAGE_SIZE, "%s\n", "1 2 4 8");
}
-static ssize_t tsl2x7x_als_time_show(struct device *dev,
+static ssize_t in_illuminance0_integration_time_show(struct device *dev,
struct device_attribute *attr,
char *buf)
{
return snprintf(buf, PAGE_SIZE, "%d.%03d\n", y, z);
}
-static ssize_t tsl2x7x_als_time_store(struct device *dev,
+static ssize_t in_illuminance0_integration_time_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t len)
{
static IIO_CONST_ATTR(in_illuminance0_integration_time_available,
".00272 - .696");
-static ssize_t tsl2x7x_als_cal_target_show(struct device *dev,
+static ssize_t in_illuminance0_target_input_show(struct device *dev,
struct device_attribute *attr,
char *buf)
{
chip->tsl2x7x_settings.als_cal_target);
}
-static ssize_t tsl2x7x_als_cal_target_store(struct device *dev,
+static ssize_t in_illuminance0_target_input_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t len)
{
}
/* persistence settings */
-static ssize_t tsl2x7x_als_persistence_show(struct device *dev,
+static ssize_t in_intensity0_thresh_period_show(struct device *dev,
struct device_attribute *attr,
char *buf)
{
return snprintf(buf, PAGE_SIZE, "%d.%03d\n", y, z);
}
-static ssize_t tsl2x7x_als_persistence_store(struct device *dev,
+static ssize_t in_intensity0_thresh_period_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t len)
{
return IIO_VAL_INT_PLUS_MICRO;
}
-static ssize_t tsl2x7x_prox_persistence_show(struct device *dev,
+static ssize_t in_proximity0_thresh_period_show(struct device *dev,
struct device_attribute *attr,
char *buf)
{
return snprintf(buf, PAGE_SIZE, "%d.%03d\n", y, z);
}
-static ssize_t tsl2x7x_prox_persistence_store(struct device *dev,
+static ssize_t in_proximity0_thresh_period_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t len)
{
return IIO_VAL_INT_PLUS_MICRO;
}
-static ssize_t tsl2x7x_do_calibrate(struct device *dev,
+static ssize_t in_illuminance0_calibrate_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t len)
{
return len;
}
-static ssize_t tsl2x7x_luxtable_show(struct device *dev,
+static ssize_t in_illuminance0_lux_table_show(struct device *dev,
struct device_attribute *attr,
char *buf)
{
return offset;
}
-static ssize_t tsl2x7x_luxtable_store(struct device *dev,
+static ssize_t in_illuminance0_lux_table_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t len)
{
return len;
}
-static ssize_t tsl2x7x_do_prox_calibrate(struct device *dev,
+static ssize_t in_proximity0_calibrate_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t len)
{
return 0;
}
-static DEVICE_ATTR(power_state, S_IRUGO | S_IWUSR,
- tsl2x7x_power_state_show, tsl2x7x_power_state_store);
+static DEVICE_ATTR_RW(power_state);
-static DEVICE_ATTR(in_proximity0_calibscale_available, S_IRUGO,
- tsl2x7x_prox_gain_available_show, NULL);
+static DEVICE_ATTR_RO(in_proximity0_calibscale_available);
-static DEVICE_ATTR(in_illuminance0_calibscale_available, S_IRUGO,
- tsl2x7x_gain_available_show, NULL);
+static DEVICE_ATTR_RO(in_illuminance0_calibscale_available);
-static DEVICE_ATTR(in_illuminance0_integration_time, S_IRUGO | S_IWUSR,
- tsl2x7x_als_time_show, tsl2x7x_als_time_store);
+static DEVICE_ATTR_RW(in_illuminance0_integration_time);
-static DEVICE_ATTR(in_illuminance0_target_input, S_IRUGO | S_IWUSR,
- tsl2x7x_als_cal_target_show, tsl2x7x_als_cal_target_store);
+static DEVICE_ATTR_RW(in_illuminance0_target_input);
-static DEVICE_ATTR(in_illuminance0_calibrate, S_IWUSR, NULL,
- tsl2x7x_do_calibrate);
+static DEVICE_ATTR_WO(in_illuminance0_calibrate);
-static DEVICE_ATTR(in_proximity0_calibrate, S_IWUSR, NULL,
- tsl2x7x_do_prox_calibrate);
+static DEVICE_ATTR_WO(in_proximity0_calibrate);
-static DEVICE_ATTR(in_illuminance0_lux_table, S_IRUGO | S_IWUSR,
- tsl2x7x_luxtable_show, tsl2x7x_luxtable_store);
+static DEVICE_ATTR_RW(in_illuminance0_lux_table);
-static DEVICE_ATTR(in_intensity0_thresh_period, S_IRUGO | S_IWUSR,
- tsl2x7x_als_persistence_show, tsl2x7x_als_persistence_store);
+static DEVICE_ATTR_RW(in_intensity0_thresh_period);
-static DEVICE_ATTR(in_proximity0_thresh_period, S_IRUGO | S_IWUSR,
- tsl2x7x_prox_persistence_show, tsl2x7x_prox_persistence_store);
+static DEVICE_ATTR_RW(in_proximity0_thresh_period);
/* Use the default register values to identify the Taos device */
static int tsl2x7x_device_id(unsigned char *id, int target)
return ret;
}
-static int ade7753_spi_write_reg_16(struct device *dev,
- u8 reg_address,
- u16 value)
+static int ade7753_spi_write_reg_16(struct device *dev, u8 reg_address,
+ u16 value)
{
int ret;
struct iio_dev *indio_dev = dev_to_iio_dev(dev);
}
static int ade7753_spi_read_reg_8(struct device *dev,
- u8 reg_address,
- u8 *val)
+ u8 reg_address,
+ u8 *val)
{
struct iio_dev *indio_dev = dev_to_iio_dev(dev);
struct ade7753_state *st = iio_priv(indio_dev);
ret = spi_w8r8(st->us, ADE7753_READ_REG(reg_address));
if (ret < 0) {
dev_err(&st->us->dev, "problem when reading 8 bit register 0x%02X",
- reg_address);
+ reg_address);
return ret;
}
*val = ret;
}
static int ade7753_spi_read_reg_16(struct device *dev,
- u8 reg_address,
- u16 *val)
+ u8 reg_address,
+ u16 *val)
{
struct iio_dev *indio_dev = dev_to_iio_dev(dev);
struct ade7753_state *st = iio_priv(indio_dev);
}
static int ade7753_spi_read_reg_24(struct device *dev,
- u8 reg_address,
- u32 *val)
+ u8 reg_address,
+ u32 *val)
{
struct iio_dev *indio_dev = dev_to_iio_dev(dev);
struct ade7753_state *st = iio_priv(indio_dev);
ret = spi_sync_transfer(st->us, xfers, ARRAY_SIZE(xfers));
if (ret) {
dev_err(&st->us->dev, "problem when reading 24 bit register 0x%02X",
- reg_address);
+ reg_address);
goto error_ret;
}
*val = (st->rx[0] << 16) | (st->rx[1] << 8) | st->rx[2];
}
static ssize_t ade7753_read_8bit(struct device *dev,
- struct device_attribute *attr,
- char *buf)
+ struct device_attribute *attr,
+ char *buf)
{
int ret;
u8 val;
}
static ssize_t ade7753_read_16bit(struct device *dev,
- struct device_attribute *attr,
- char *buf)
+ struct device_attribute *attr,
+ char *buf)
{
int ret;
u16 val;
}
static ssize_t ade7753_read_24bit(struct device *dev,
- struct device_attribute *attr,
- char *buf)
+ struct device_attribute *attr,
+ char *buf)
{
int ret;
u32 val;
}
static ssize_t ade7753_write_8bit(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t len)
+ struct device_attribute *attr,
+ const char *buf,
+ size_t len)
{
struct iio_dev_attr *this_attr = to_iio_dev_attr(attr);
int ret;
}
static ssize_t ade7753_write_16bit(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t len)
+ struct device_attribute *attr,
+ const char *buf,
+ size_t len)
{
struct iio_dev_attr *this_attr = to_iio_dev_attr(attr);
int ret;
static IIO_DEV_ATTR_LAENERGY(ade7753_read_24bit, ADE7753_LAENERGY);
static IIO_DEV_ATTR_VAENERGY(ade7753_read_24bit, ADE7753_VAENERGY);
static IIO_DEV_ATTR_LVAENERGY(ade7753_read_24bit, ADE7753_LVAENERGY);
-static IIO_DEV_ATTR_CFDEN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CFDEN(0644,
ade7753_read_16bit,
ade7753_write_16bit,
ADE7753_CFDEN);
-static IIO_DEV_ATTR_CFNUM(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CFNUM(0644,
ade7753_read_8bit,
ade7753_write_8bit,
ADE7753_CFNUM);
static IIO_DEV_ATTR_CHKSUM(ade7753_read_8bit, ADE7753_CHKSUM);
-static IIO_DEV_ATTR_PHCAL(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_PHCAL(0644,
ade7753_read_16bit,
ade7753_write_16bit,
ADE7753_PHCAL);
-static IIO_DEV_ATTR_APOS(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_APOS(0644,
ade7753_read_16bit,
ade7753_write_16bit,
ADE7753_APOS);
-static IIO_DEV_ATTR_SAGCYC(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_SAGCYC(0644,
ade7753_read_8bit,
ade7753_write_8bit,
ADE7753_SAGCYC);
-static IIO_DEV_ATTR_SAGLVL(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_SAGLVL(0644,
ade7753_read_8bit,
ade7753_write_8bit,
ADE7753_SAGLVL);
-static IIO_DEV_ATTR_LINECYC(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_LINECYC(0644,
ade7753_read_8bit,
ade7753_write_8bit,
ADE7753_LINECYC);
-static IIO_DEV_ATTR_WDIV(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_WDIV(0644,
ade7753_read_8bit,
ade7753_write_8bit,
ADE7753_WDIV);
-static IIO_DEV_ATTR_IRMS(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_IRMS(0644,
ade7753_read_24bit,
NULL,
ADE7753_IRMS);
-static IIO_DEV_ATTR_VRMS(S_IRUGO,
+static IIO_DEV_ATTR_VRMS(0444,
ade7753_read_24bit,
NULL,
ADE7753_VRMS);
-static IIO_DEV_ATTR_IRMSOS(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_IRMSOS(0644,
ade7753_read_16bit,
ade7753_write_16bit,
ADE7753_IRMSOS);
-static IIO_DEV_ATTR_VRMSOS(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_VRMSOS(0644,
ade7753_read_16bit,
ade7753_write_16bit,
ADE7753_VRMSOS);
-static IIO_DEV_ATTR_WGAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_WGAIN(0644,
ade7753_read_16bit,
ade7753_write_16bit,
ADE7753_WGAIN);
-static IIO_DEV_ATTR_VAGAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_VAGAIN(0644,
ade7753_read_16bit,
ade7753_write_16bit,
ADE7753_VAGAIN);
-static IIO_DEV_ATTR_PGA_GAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_PGA_GAIN(0644,
ade7753_read_16bit,
ade7753_write_16bit,
ADE7753_GAIN);
-static IIO_DEV_ATTR_IPKLVL(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_IPKLVL(0644,
ade7753_read_8bit,
ade7753_write_8bit,
ADE7753_IPKLVL);
-static IIO_DEV_ATTR_VPKLVL(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_VPKLVL(0644,
ade7753_read_8bit,
ade7753_write_8bit,
ADE7753_VPKLVL);
-static IIO_DEV_ATTR_IPEAK(S_IRUGO,
+static IIO_DEV_ATTR_IPEAK(0444,
ade7753_read_24bit,
NULL,
ADE7753_IPEAK);
-static IIO_DEV_ATTR_VPEAK(S_IRUGO,
+static IIO_DEV_ATTR_VPEAK(0444,
ade7753_read_24bit,
NULL,
ADE7753_VPEAK);
-static IIO_DEV_ATTR_VPERIOD(S_IRUGO,
+static IIO_DEV_ATTR_VPERIOD(0444,
ade7753_read_16bit,
NULL,
ADE7753_PERIOD);
-static IIO_DEV_ATTR_CH_OFF(1, S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CH_OFF(1, 0644,
ade7753_read_8bit,
ade7753_write_8bit,
ADE7753_CH1OS);
-static IIO_DEV_ATTR_CH_OFF(2, S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CH_OFF(2, 0644,
ade7753_read_8bit,
ade7753_write_8bit,
ADE7753_CH2OS);
}
static ssize_t ade7753_read_frequency(struct device *dev,
- struct device_attribute *attr,
- char *buf)
+ struct device_attribute *attr,
+ char *buf)
{
int ret;
u16 t;
}
static ssize_t ade7753_write_frequency(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t len)
+ struct device_attribute *attr,
+ const char *buf,
+ size_t len)
{
struct iio_dev *indio_dev = dev_to_iio_dev(dev);
struct ade7753_state *st = iio_priv(indio_dev);
static IIO_CONST_ATTR(in_temp_offset, "-25 C");
static IIO_CONST_ATTR(in_temp_scale, "0.67 C");
-static IIO_DEV_ATTR_SAMP_FREQ(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_SAMP_FREQ(0644,
ade7753_read_frequency,
ade7753_write_frequency);
static IIO_DEV_ATTR_LAENERGY(ade7754_read_24bit, ADE7754_LAENERGY);
static IIO_DEV_ATTR_VAENERGY(ade7754_read_24bit, ADE7754_VAENERGY);
static IIO_DEV_ATTR_LVAENERGY(ade7754_read_24bit, ADE7754_LVAENERGY);
-static IIO_DEV_ATTR_VPEAK(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_VPEAK(0644,
ade7754_read_8bit,
ade7754_write_8bit,
ADE7754_VPEAK);
-static IIO_DEV_ATTR_IPEAK(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_IPEAK(0644,
ade7754_read_8bit,
ade7754_write_8bit,
ADE7754_VPEAK);
-static IIO_DEV_ATTR_APHCAL(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_APHCAL(0644,
ade7754_read_8bit,
ade7754_write_8bit,
ADE7754_APHCAL);
-static IIO_DEV_ATTR_BPHCAL(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_BPHCAL(0644,
ade7754_read_8bit,
ade7754_write_8bit,
ADE7754_BPHCAL);
-static IIO_DEV_ATTR_CPHCAL(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CPHCAL(0644,
ade7754_read_8bit,
ade7754_write_8bit,
ADE7754_CPHCAL);
-static IIO_DEV_ATTR_AAPOS(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_AAPOS(0644,
ade7754_read_16bit,
ade7754_write_16bit,
ADE7754_AAPOS);
-static IIO_DEV_ATTR_BAPOS(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_BAPOS(0644,
ade7754_read_16bit,
ade7754_write_16bit,
ADE7754_BAPOS);
-static IIO_DEV_ATTR_CAPOS(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CAPOS(0644,
ade7754_read_16bit,
ade7754_write_16bit,
ADE7754_CAPOS);
-static IIO_DEV_ATTR_WDIV(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_WDIV(0644,
ade7754_read_8bit,
ade7754_write_8bit,
ADE7754_WDIV);
-static IIO_DEV_ATTR_VADIV(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_VADIV(0644,
ade7754_read_8bit,
ade7754_write_8bit,
ADE7754_VADIV);
-static IIO_DEV_ATTR_CFNUM(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CFNUM(0644,
ade7754_read_16bit,
ade7754_write_16bit,
ADE7754_CFNUM);
-static IIO_DEV_ATTR_CFDEN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CFDEN(0644,
ade7754_read_16bit,
ade7754_write_16bit,
ADE7754_CFDEN);
-static IIO_DEV_ATTR_ACTIVE_POWER_A_GAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_ACTIVE_POWER_A_GAIN(0644,
ade7754_read_16bit,
ade7754_write_16bit,
ADE7754_AAPGAIN);
-static IIO_DEV_ATTR_ACTIVE_POWER_B_GAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_ACTIVE_POWER_B_GAIN(0644,
ade7754_read_16bit,
ade7754_write_16bit,
ADE7754_BAPGAIN);
-static IIO_DEV_ATTR_ACTIVE_POWER_C_GAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_ACTIVE_POWER_C_GAIN(0644,
ade7754_read_16bit,
ade7754_write_16bit,
ADE7754_CAPGAIN);
-static IIO_DEV_ATTR_AIRMS(S_IRUGO,
+static IIO_DEV_ATTR_AIRMS(0444,
ade7754_read_24bit,
NULL,
ADE7754_AIRMS);
-static IIO_DEV_ATTR_BIRMS(S_IRUGO,
+static IIO_DEV_ATTR_BIRMS(0444,
ade7754_read_24bit,
NULL,
ADE7754_BIRMS);
-static IIO_DEV_ATTR_CIRMS(S_IRUGO,
+static IIO_DEV_ATTR_CIRMS(0444,
ade7754_read_24bit,
NULL,
ADE7754_CIRMS);
-static IIO_DEV_ATTR_AVRMS(S_IRUGO,
+static IIO_DEV_ATTR_AVRMS(0444,
ade7754_read_24bit,
NULL,
ADE7754_AVRMS);
-static IIO_DEV_ATTR_BVRMS(S_IRUGO,
+static IIO_DEV_ATTR_BVRMS(0444,
ade7754_read_24bit,
NULL,
ADE7754_BVRMS);
-static IIO_DEV_ATTR_CVRMS(S_IRUGO,
+static IIO_DEV_ATTR_CVRMS(0444,
ade7754_read_24bit,
NULL,
ADE7754_CVRMS);
-static IIO_DEV_ATTR_AIRMSOS(S_IRUGO,
+static IIO_DEV_ATTR_AIRMSOS(0444,
ade7754_read_16bit,
ade7754_write_16bit,
ADE7754_AIRMSOS);
-static IIO_DEV_ATTR_BIRMSOS(S_IRUGO,
+static IIO_DEV_ATTR_BIRMSOS(0444,
ade7754_read_16bit,
ade7754_write_16bit,
ADE7754_BIRMSOS);
-static IIO_DEV_ATTR_CIRMSOS(S_IRUGO,
+static IIO_DEV_ATTR_CIRMSOS(0444,
ade7754_read_16bit,
ade7754_write_16bit,
ADE7754_CIRMSOS);
-static IIO_DEV_ATTR_AVRMSOS(S_IRUGO,
+static IIO_DEV_ATTR_AVRMSOS(0444,
ade7754_read_16bit,
ade7754_write_16bit,
ADE7754_AVRMSOS);
-static IIO_DEV_ATTR_BVRMSOS(S_IRUGO,
+static IIO_DEV_ATTR_BVRMSOS(0444,
ade7754_read_16bit,
ade7754_write_16bit,
ADE7754_BVRMSOS);
-static IIO_DEV_ATTR_CVRMSOS(S_IRUGO,
+static IIO_DEV_ATTR_CVRMSOS(0444,
ade7754_read_16bit,
ade7754_write_16bit,
ADE7754_CVRMSOS);
static IIO_CONST_ATTR(in_temp_offset, "129 C");
static IIO_CONST_ATTR(in_temp_scale, "4 C");
-static IIO_DEV_ATTR_SAMP_FREQ(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_SAMP_FREQ(0644,
ade7754_read_frequency,
ade7754_write_frequency);
return ret;
}
-static IIO_DEV_ATTR_VPEAK(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_VPEAK(0644,
ade7758_read_8bit,
ade7758_write_8bit,
ADE7758_VPEAK);
-static IIO_DEV_ATTR_IPEAK(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_IPEAK(0644,
ade7758_read_8bit,
ade7758_write_8bit,
ADE7758_VPEAK);
-static IIO_DEV_ATTR_APHCAL(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_APHCAL(0644,
ade7758_read_8bit,
ade7758_write_8bit,
ADE7758_APHCAL);
-static IIO_DEV_ATTR_BPHCAL(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_BPHCAL(0644,
ade7758_read_8bit,
ade7758_write_8bit,
ADE7758_BPHCAL);
-static IIO_DEV_ATTR_CPHCAL(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CPHCAL(0644,
ade7758_read_8bit,
ade7758_write_8bit,
ADE7758_CPHCAL);
-static IIO_DEV_ATTR_WDIV(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_WDIV(0644,
ade7758_read_8bit,
ade7758_write_8bit,
ADE7758_WDIV);
-static IIO_DEV_ATTR_VADIV(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_VADIV(0644,
ade7758_read_8bit,
ade7758_write_8bit,
ADE7758_VADIV);
-static IIO_DEV_ATTR_AIRMS(S_IRUGO,
+static IIO_DEV_ATTR_AIRMS(0444,
ade7758_read_24bit,
NULL,
ADE7758_AIRMS);
-static IIO_DEV_ATTR_BIRMS(S_IRUGO,
+static IIO_DEV_ATTR_BIRMS(0444,
ade7758_read_24bit,
NULL,
ADE7758_BIRMS);
-static IIO_DEV_ATTR_CIRMS(S_IRUGO,
+static IIO_DEV_ATTR_CIRMS(0444,
ade7758_read_24bit,
NULL,
ADE7758_CIRMS);
-static IIO_DEV_ATTR_AVRMS(S_IRUGO,
+static IIO_DEV_ATTR_AVRMS(0444,
ade7758_read_24bit,
NULL,
ADE7758_AVRMS);
-static IIO_DEV_ATTR_BVRMS(S_IRUGO,
+static IIO_DEV_ATTR_BVRMS(0444,
ade7758_read_24bit,
NULL,
ADE7758_BVRMS);
-static IIO_DEV_ATTR_CVRMS(S_IRUGO,
+static IIO_DEV_ATTR_CVRMS(0444,
ade7758_read_24bit,
NULL,
ADE7758_CVRMS);
-static IIO_DEV_ATTR_AIRMSOS(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_AIRMSOS(0644,
ade7758_read_16bit,
ade7758_write_16bit,
ADE7758_AIRMSOS);
-static IIO_DEV_ATTR_BIRMSOS(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_BIRMSOS(0644,
ade7758_read_16bit,
ade7758_write_16bit,
ADE7758_BIRMSOS);
-static IIO_DEV_ATTR_CIRMSOS(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CIRMSOS(0644,
ade7758_read_16bit,
ade7758_write_16bit,
ADE7758_CIRMSOS);
-static IIO_DEV_ATTR_AVRMSOS(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_AVRMSOS(0644,
ade7758_read_16bit,
ade7758_write_16bit,
ADE7758_AVRMSOS);
-static IIO_DEV_ATTR_BVRMSOS(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_BVRMSOS(0644,
ade7758_read_16bit,
ade7758_write_16bit,
ADE7758_BVRMSOS);
-static IIO_DEV_ATTR_CVRMSOS(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CVRMSOS(0644,
ade7758_read_16bit,
ade7758_write_16bit,
ADE7758_CVRMSOS);
-static IIO_DEV_ATTR_AIGAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_AIGAIN(0644,
ade7758_read_16bit,
ade7758_write_16bit,
ADE7758_AIGAIN);
-static IIO_DEV_ATTR_BIGAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_BIGAIN(0644,
ade7758_read_16bit,
ade7758_write_16bit,
ADE7758_BIGAIN);
-static IIO_DEV_ATTR_CIGAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CIGAIN(0644,
ade7758_read_16bit,
ade7758_write_16bit,
ADE7758_CIGAIN);
-static IIO_DEV_ATTR_AVRMSGAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_AVRMSGAIN(0644,
ade7758_read_16bit,
ade7758_write_16bit,
ADE7758_AVRMSGAIN);
-static IIO_DEV_ATTR_BVRMSGAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_BVRMSGAIN(0644,
ade7758_read_16bit,
ade7758_write_16bit,
ADE7758_BVRMSGAIN);
-static IIO_DEV_ATTR_CVRMSGAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CVRMSGAIN(0644,
ade7758_read_16bit,
ade7758_write_16bit,
ADE7758_CVRMSGAIN);
return st->write_reg_16(dev, ADE7854_CONFIG, val);
}
-static IIO_DEV_ATTR_AIGAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_AIGAIN(0644,
ade7854_read_24bit,
ade7854_write_24bit,
ADE7854_AIGAIN);
-static IIO_DEV_ATTR_BIGAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_BIGAIN(0644,
ade7854_read_24bit,
ade7854_write_24bit,
ADE7854_BIGAIN);
-static IIO_DEV_ATTR_CIGAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CIGAIN(0644,
ade7854_read_24bit,
ade7854_write_24bit,
ADE7854_CIGAIN);
-static IIO_DEV_ATTR_NIGAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_NIGAIN(0644,
ade7854_read_24bit,
ade7854_write_24bit,
ADE7854_NIGAIN);
-static IIO_DEV_ATTR_AVGAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_AVGAIN(0644,
ade7854_read_24bit,
ade7854_write_24bit,
ADE7854_AVGAIN);
-static IIO_DEV_ATTR_BVGAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_BVGAIN(0644,
ade7854_read_24bit,
ade7854_write_24bit,
ADE7854_BVGAIN);
-static IIO_DEV_ATTR_CVGAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CVGAIN(0644,
ade7854_read_24bit,
ade7854_write_24bit,
ADE7854_CVGAIN);
-static IIO_DEV_ATTR_APPARENT_POWER_A_GAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_APPARENT_POWER_A_GAIN(0644,
ade7854_read_24bit,
ade7854_write_24bit,
ADE7854_AVAGAIN);
-static IIO_DEV_ATTR_APPARENT_POWER_B_GAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_APPARENT_POWER_B_GAIN(0644,
ade7854_read_24bit,
ade7854_write_24bit,
ADE7854_BVAGAIN);
-static IIO_DEV_ATTR_APPARENT_POWER_C_GAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_APPARENT_POWER_C_GAIN(0644,
ade7854_read_24bit,
ade7854_write_24bit,
ADE7854_CVAGAIN);
-static IIO_DEV_ATTR_ACTIVE_POWER_A_OFFSET(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_ACTIVE_POWER_A_OFFSET(0644,
ade7854_read_24bit,
ade7854_write_24bit,
ADE7854_AWATTOS);
-static IIO_DEV_ATTR_ACTIVE_POWER_B_OFFSET(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_ACTIVE_POWER_B_OFFSET(0644,
ade7854_read_24bit,
ade7854_write_24bit,
ADE7854_BWATTOS);
-static IIO_DEV_ATTR_ACTIVE_POWER_C_OFFSET(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_ACTIVE_POWER_C_OFFSET(0644,
ade7854_read_24bit,
ade7854_write_24bit,
ADE7854_CWATTOS);
-static IIO_DEV_ATTR_REACTIVE_POWER_A_GAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_REACTIVE_POWER_A_GAIN(0644,
ade7854_read_24bit,
ade7854_write_24bit,
ADE7854_AVARGAIN);
-static IIO_DEV_ATTR_REACTIVE_POWER_B_GAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_REACTIVE_POWER_B_GAIN(0644,
ade7854_read_24bit,
ade7854_write_24bit,
ADE7854_BVARGAIN);
-static IIO_DEV_ATTR_REACTIVE_POWER_C_GAIN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_REACTIVE_POWER_C_GAIN(0644,
ade7854_read_24bit,
ade7854_write_24bit,
ADE7854_CVARGAIN);
-static IIO_DEV_ATTR_REACTIVE_POWER_A_OFFSET(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_REACTIVE_POWER_A_OFFSET(0644,
ade7854_read_24bit,
ade7854_write_24bit,
ADE7854_AVAROS);
-static IIO_DEV_ATTR_REACTIVE_POWER_B_OFFSET(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_REACTIVE_POWER_B_OFFSET(0644,
ade7854_read_24bit,
ade7854_write_24bit,
ADE7854_BVAROS);
-static IIO_DEV_ATTR_REACTIVE_POWER_C_OFFSET(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_REACTIVE_POWER_C_OFFSET(0644,
ade7854_read_24bit,
ade7854_write_24bit,
ADE7854_CVAROS);
-static IIO_DEV_ATTR_VPEAK(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_VPEAK(0644,
ade7854_read_32bit,
ade7854_write_32bit,
ADE7854_VPEAK);
-static IIO_DEV_ATTR_IPEAK(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_IPEAK(0644,
ade7854_read_32bit,
ade7854_write_32bit,
ADE7854_VPEAK);
-static IIO_DEV_ATTR_APHCAL(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_APHCAL(0644,
ade7854_read_16bit,
ade7854_write_16bit,
ADE7854_APHCAL);
-static IIO_DEV_ATTR_BPHCAL(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_BPHCAL(0644,
ade7854_read_16bit,
ade7854_write_16bit,
ADE7854_BPHCAL);
-static IIO_DEV_ATTR_CPHCAL(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CPHCAL(0644,
ade7854_read_16bit,
ade7854_write_16bit,
ADE7854_CPHCAL);
-static IIO_DEV_ATTR_CF1DEN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CF1DEN(0644,
ade7854_read_16bit,
ade7854_write_16bit,
ADE7854_CF1DEN);
-static IIO_DEV_ATTR_CF2DEN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CF2DEN(0644,
ade7854_read_16bit,
ade7854_write_16bit,
ADE7854_CF2DEN);
-static IIO_DEV_ATTR_CF3DEN(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CF3DEN(0644,
ade7854_read_16bit,
ade7854_write_16bit,
ADE7854_CF3DEN);
-static IIO_DEV_ATTR_LINECYC(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_LINECYC(0644,
ade7854_read_16bit,
ade7854_write_16bit,
ADE7854_LINECYC);
-static IIO_DEV_ATTR_SAGCYC(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_SAGCYC(0644,
ade7854_read_8bit,
ade7854_write_8bit,
ADE7854_SAGCYC);
-static IIO_DEV_ATTR_CFCYC(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_CFCYC(0644,
ade7854_read_8bit,
ade7854_write_8bit,
ADE7854_CFCYC);
-static IIO_DEV_ATTR_PEAKCYC(S_IWUSR | S_IRUGO,
+static IIO_DEV_ATTR_PEAKCYC(0644,
ade7854_read_8bit,
ade7854_write_8bit,
ADE7854_PEAKCYC);
ADE7854_ANGLE1);
static IIO_DEV_ATTR_ANGLE2(ade7854_read_24bit,
ADE7854_ANGLE2);
-static IIO_DEV_ATTR_AIRMS(S_IRUGO,
+static IIO_DEV_ATTR_AIRMS(0444,
ade7854_read_24bit,
NULL,
ADE7854_AIRMS);
-static IIO_DEV_ATTR_BIRMS(S_IRUGO,
+static IIO_DEV_ATTR_BIRMS(0444,
ade7854_read_24bit,
NULL,
ADE7854_BIRMS);
-static IIO_DEV_ATTR_CIRMS(S_IRUGO,
+static IIO_DEV_ATTR_CIRMS(0444,
ade7854_read_24bit,
NULL,
ADE7854_CIRMS);
-static IIO_DEV_ATTR_NIRMS(S_IRUGO,
+static IIO_DEV_ATTR_NIRMS(0444,
ade7854_read_24bit,
NULL,
ADE7854_NIRMS);
-static IIO_DEV_ATTR_AVRMS(S_IRUGO,
+static IIO_DEV_ATTR_AVRMS(0444,
ade7854_read_24bit,
NULL,
ADE7854_AVRMS);
-static IIO_DEV_ATTR_BVRMS(S_IRUGO,
+static IIO_DEV_ATTR_BVRMS(0444,
ade7854_read_24bit,
NULL,
ADE7854_BVRMS);
-static IIO_DEV_ATTR_CVRMS(S_IRUGO,
+static IIO_DEV_ATTR_CVRMS(0444,
ade7854_read_24bit,
NULL,
ADE7854_CVRMS);
-static IIO_DEV_ATTR_AIRMSOS(S_IRUGO,
+static IIO_DEV_ATTR_AIRMSOS(0444,
ade7854_read_16bit,
ade7854_write_16bit,
ADE7854_AIRMSOS);
-static IIO_DEV_ATTR_BIRMSOS(S_IRUGO,
+static IIO_DEV_ATTR_BIRMSOS(0444,
ade7854_read_16bit,
ade7854_write_16bit,
ADE7854_BIRMSOS);
-static IIO_DEV_ATTR_CIRMSOS(S_IRUGO,
+static IIO_DEV_ATTR_CIRMSOS(0444,
ade7854_read_16bit,
ade7854_write_16bit,
ADE7854_CIRMSOS);
-static IIO_DEV_ATTR_AVRMSOS(S_IRUGO,
+static IIO_DEV_ATTR_AVRMSOS(0444,
ade7854_read_16bit,
ade7854_write_16bit,
ADE7854_AVRMSOS);
-static IIO_DEV_ATTR_BVRMSOS(S_IRUGO,
+static IIO_DEV_ATTR_BVRMSOS(0444,
ade7854_read_16bit,
ade7854_write_16bit,
ADE7854_BVRMSOS);
-static IIO_DEV_ATTR_CVRMSOS(S_IRUGO,
+static IIO_DEV_ATTR_CVRMSOS(0444,
ade7854_read_16bit,
ade7854_write_16bit,
ADE7854_CVRMSOS);
unsigned char h_source_snap;
unsigned char h_command;
unsigned char h_vendor_id[3];
- unsigned short h_proto; /* packet type ID field */
+ __be16 h_proto; /* packet type ID field */
#define ETHER_PROTOCOL_TYPE_EAP 0x888e
#define ETHER_PROTOCOL_TYPE_IP 0x0800
#define ETHER_PROTOCOL_TYPE_ARP 0x0806
struct wpa_eapol_key {
unsigned char type;
- unsigned short key_info;
+ __be16 key_info;
unsigned short key_length;
unsigned char replay_counter[WPA_REPLAY_COUNTER_LEN];
unsigned char key_nonce[WPA_NONCE_LEN];
hdr = (struct hostif_hdr *)buffer;
DPRINTK(4, "size=%d\n", hdr->size);
- if (hdr->event < HIF_DATA_REQ || HIF_REQ_MAX < hdr->event) {
+ if (le16_to_cpu(hdr->event) < HIF_DATA_REQ ||
+ le16_to_cpu(hdr->event) > HIF_REQ_MAX) {
DPRINTK(1, "unknown event=%04X\n", hdr->event);
return 0;
}
hdr = (struct hostif_hdr *)p;
- if (hdr->event < HIF_DATA_REQ || HIF_REQ_MAX < hdr->event) {
+ if (le16_to_cpu(hdr->event) < HIF_DATA_REQ ||
+ le16_to_cpu(hdr->event) > HIF_REQ_MAX) {
DPRINTK(1, "unknown event=%04X\n", hdr->event);
return 0;
}
/* add event to hostt buffer */
- priv->hostt.buff[priv->hostt.qtail] = hdr->event;
+ priv->hostt.buff[priv->hostt.qtail] = le16_to_cpu(hdr->event);
priv->hostt.qtail = (priv->hostt.qtail + 1) % SME_EVENT_BUFF_SIZE;
DPRINTK(4, "event=%04X\n", hdr->event);
hdr = (struct hostif_hdr *)&rx_buffer->data[0];
rx_buffer->size = le16_to_cpu(hdr->size) + sizeof(hdr->size);
- event = hdr->event;
+ event = le16_to_cpu(hdr->event);
inc_rxqtail(priv);
ret = ks7010_sdio_writeb(priv, READ_STATUS, REG_STATUS_IDLE);
/* noise */
ap->noise = ap_info->noise;
/* capability */
- ap->capability = ap_info->capability;
+ ap->capability = le16_to_cpu(ap_info->capability);
/* rsn */
if ((ap_info->rsn_mode & RSN_MODE_WPA2) &&
(priv->wpa.version == IW_AUTH_WPA_VERSION_WPA2)) {
/* noise */
ap->noise = ap_info->noise;
/* capability */
- ap->capability = ap_info->capability;
+ ap->capability = le16_to_cpu(ap_info->capability);
/* channel */
ap->channel = ap_info->ch_info;
bp = ap_info->body;
- bsize = ap_info->body_size;
+ bsize = le16_to_cpu(ap_info->body_size);
offset = 0;
while (bsize > offset) {
break;
case LOCAL_GAIN:
memcpy(&priv->gain, priv->rxp, sizeof(priv->gain));
- DPRINTK(3, "TxMode=%d, RxMode=%d, TxGain=%d, RxGain=%d\n",
- priv->gain.TxMode, priv->gain.RxMode, priv->gain.TxGain,
- priv->gain.RxGain);
+ DPRINTK(3, "tx_mode=%d, rx_mode=%d, tx_gain=%d, rx_gain=%d\n",
+ priv->gain.tx_mode, priv->gain.rx_mode,
+ priv->gain.tx_gain, priv->gain.rx_gain);
break;
case LOCAL_EEPROM_SUM:
memcpy(&priv->eeprom_sum, priv->rxp, sizeof(priv->eeprom_sum));
wrqu.data.length += sizeof(associnfo_leader0) - 1;
pbuf += sizeof(associnfo_leader0) - 1;
- for (i = 0; i < assoc_req->reqIEs_size; i++)
+ for (i = 0; i < le16_to_cpu(assoc_req->req_ies_size); i++)
pbuf += sprintf(pbuf, "%02x", *(pb + i));
- wrqu.data.length += (assoc_req->reqIEs_size) * 2;
+ wrqu.data.length += (le16_to_cpu(assoc_req->req_ies_size)) * 2;
memcpy(pbuf, associnfo_leader1, sizeof(associnfo_leader1) - 1);
wrqu.data.length += sizeof(associnfo_leader1) - 1;
pbuf += sizeof(associnfo_leader1) - 1;
- pb += assoc_req->reqIEs_size;
- for (i = 0; i < assoc_resp->respIEs_size; i++)
+ pb += le16_to_cpu(assoc_req->req_ies_size);
+ for (i = 0; i < le16_to_cpu(assoc_resp->resp_ies_size); i++)
pbuf += sprintf(pbuf, "%02x", *(pb + i));
- wrqu.data.length += (assoc_resp->respIEs_size) * 2;
+ wrqu.data.length += (le16_to_cpu(assoc_resp->resp_ies_size)) * 2;
pbuf += sprintf(pbuf, ")");
wrqu.data.length += 1;
{
struct iw_statistics *wstats = &priv->wstats;
unsigned char rssi, signal, noise;
- unsigned char LinkSpeed;
- unsigned int TransmittedFrameCount, ReceivedFragmentCount;
- unsigned int FailedCount, FCSErrorCount;
+ unsigned char link_speed;
+ unsigned int transmitted_frame_count, received_fragment_count;
+ unsigned int failed_count, fcs_error_count;
DPRINTK(3, "\n");
rssi = get_BYTE(priv);
signal = get_BYTE(priv);
noise = get_BYTE(priv);
- LinkSpeed = get_BYTE(priv);
- TransmittedFrameCount = get_DWORD(priv);
- ReceivedFragmentCount = get_DWORD(priv);
- FailedCount = get_DWORD(priv);
- FCSErrorCount = get_DWORD(priv);
+ link_speed = get_BYTE(priv);
+ transmitted_frame_count = get_DWORD(priv);
+ received_fragment_count = get_DWORD(priv);
+ failed_count = get_DWORD(priv);
+ fcs_error_count = get_DWORD(priv);
DPRINTK(4, "phyinfo confirm rssi=%d signal=%d\n", rssi, signal);
- priv->current_rate = (LinkSpeed & RATE_MASK);
+ priv->current_rate = (link_speed & RATE_MASK);
wstats->qual.qual = signal;
wstats->qual.level = 256 - rssi;
wstats->qual.noise = 0; /* invalid noise value */
DPRINTK(3, "\n rssi=%u\n"
" signal=%u\n"
- " LinkSpeed=%ux500Kbps\n"
- " TransmittedFrameCount=%u\n"
- " ReceivedFragmentCount=%u\n"
- " FailedCount=%u\n"
- " FCSErrorCount=%u\n",
- rssi, signal, LinkSpeed, TransmittedFrameCount,
- ReceivedFragmentCount, FailedCount, FCSErrorCount);
-
+ " link_speed=%ux500Kbps\n"
+ " transmitted_frame_count=%u\n"
+ " received_fragment_count=%u\n"
+ " failed_count=%u\n"
+ " fcs_error_count=%u\n",
+ rssi, signal, link_speed, transmitted_frame_count,
+ received_fragment_count, failed_count, fcs_error_count);
/* wake_up_interruptible_all(&priv->confirm_wait); */
complete(&priv->confirm_wait);
}
static
void hostif_power_mgmt_request(struct ks_wlan_private *priv,
- unsigned long mode, unsigned long wake_up,
- unsigned long receiveDTIMs)
+ unsigned long mode, unsigned long wake_up,
+ unsigned long receive_dtims)
{
struct hostif_power_mgmt_request_t *pp;
- DPRINTK(3, "mode=%lu wake_up=%lu receiveDTIMs=%lu\n", mode, wake_up,
- receiveDTIMs);
+ DPRINTK(3, "mode=%lu wake_up=%lu receive_dtims=%lu\n", mode, wake_up,
+ receive_dtims);
pp = hostif_generic_request(sizeof(*pp), HIF_POWER_MGMT_REQ);
if (!pp)
pp->mode = cpu_to_le32((uint32_t)mode);
pp->wake_up = cpu_to_le32((uint32_t)wake_up);
- pp->receiveDTIMs = cpu_to_le32((uint32_t)receiveDTIMs);
+ pp->receive_dtims = cpu_to_le32((uint32_t)receive_dtims);
/* send to device request */
ps_confirm_wait_inc(priv);
static
void hostif_sme_set_wep(struct ks_wlan_private *priv, int type)
{
- u32 val;
+ __le32 val;
switch (type) {
case SME_WEP_INDEX_REQUEST:
}
struct wpa_suite_t {
- unsigned short size;
+ __le16 size;
unsigned char suite[4][CIPHER_ID_LEN];
} __packed;
struct rsn_mode_t {
- u32 rsn_mode;
- u16 rsn_capability;
+ __le32 rsn_mode;
+ __le16 rsn_capability;
} __packed;
static
{
struct wpa_suite_t wpa_suite;
struct rsn_mode_t rsn_mode;
- u32 val;
+ __le32 val;
memset(&wpa_suite, 0, sizeof(wpa_suite));
hostif_mib_set_request(priv, DOT11_RSN_CONFIG_UNICAST_CIPHER,
sizeof(wpa_suite.size) +
- CIPHER_ID_LEN * wpa_suite.size,
+ CIPHER_ID_LEN *
+ le16_to_cpu(wpa_suite.size),
MIB_VALUE_TYPE_OSTRING, &wpa_suite);
break;
case SME_RSN_MCAST_REQUEST:
hostif_mib_set_request(priv, DOT11_RSN_CONFIG_AUTH_SUITE,
sizeof(wpa_suite.size) +
- KEY_MGMT_ID_LEN * wpa_suite.size,
+ KEY_MGMT_ID_LEN *
+ le16_to_cpu(wpa_suite.size),
MIB_VALUE_TYPE_OSTRING, &wpa_suite);
break;
case SME_RSN_ENABLED_REQUEST:
int mc_count;
struct netdev_hw_addr *ha;
char set_address[NIC_MAX_MCAST_LIST * ETH_ALEN];
- unsigned long filter_type;
+ __le32 filter_type;
int i = 0;
DPRINTK(3, "\n");
static
void hostif_sme_power_mgmt_set(struct ks_wlan_private *priv)
{
- unsigned long mode, wake_up, receiveDTIMs;
+ unsigned long mode, wake_up, receive_dtims;
DPRINTK(3, "\n");
switch (priv->reg.power_mgmt) {
case POWER_MGMT_ACTIVE:
mode = POWER_ACTIVE;
wake_up = 0;
- receiveDTIMs = 0;
+ receive_dtims = 0;
break;
case POWER_MGMT_SAVE1:
if (priv->reg.operation_mode == MODE_INFRASTRUCTURE) {
mode = POWER_SAVE;
wake_up = 0;
- receiveDTIMs = 0;
+ receive_dtims = 0;
} else {
mode = POWER_ACTIVE;
wake_up = 0;
- receiveDTIMs = 0;
+ receive_dtims = 0;
}
break;
case POWER_MGMT_SAVE2:
if (priv->reg.operation_mode == MODE_INFRASTRUCTURE) {
mode = POWER_SAVE;
wake_up = 0;
- receiveDTIMs = 1;
+ receive_dtims = 1;
} else {
mode = POWER_ACTIVE;
wake_up = 0;
- receiveDTIMs = 0;
+ receive_dtims = 0;
}
break;
default:
mode = POWER_ACTIVE;
wake_up = 0;
- receiveDTIMs = 0;
+ receive_dtims = 0;
break;
}
- hostif_power_mgmt_request(priv, mode, wake_up, receiveDTIMs);
+ hostif_power_mgmt_request(priv, mode, wake_up, receive_dtims);
}
static
static
void hostif_sme_set_key(struct ks_wlan_private *priv, int type)
{
- u32 val;
+ __le32 val;
switch (type) {
case SME_SET_FLAG:
void hostif_sme_set_pmksa(struct ks_wlan_private *priv)
{
struct pmk_cache_t {
- u16 size;
+ __le16 size;
struct {
u8 bssid[ETH_ALEN];
u8 pmkid[IW_PMKID_LEN];
static
void hostif_sme_execute(struct ks_wlan_private *priv, int event)
{
- u32 val;
+ __le32 val;
DPRINTK(3, "event=%d\n", event);
switch (event) {
*/
struct hostif_hdr {
- u16 size;
- u16 event;
+ __le16 size;
+ __le16 event;
} __packed;
struct hostif_data_request_t {
struct hostif_hdr header;
- u16 auth_type;
+ __le16 auth_type;
#define TYPE_DATA 0x0000
#define TYPE_AUTH 0x0001
- u16 reserved;
+ __le16 reserved;
u8 data[0];
} __packed;
struct hostif_data_indication_t {
struct hostif_hdr header;
- u16 auth_type;
+ __le16 auth_type;
/* #define TYPE_DATA 0x0000 */
#define TYPE_PMK1 0x0001
#define TYPE_GMK1 0x0002
#define TYPE_GMK2 0x0003
- u16 reserved;
+ __le16 reserved;
u8 data[0];
} __packed;
struct hostif_mib_get_request_t {
struct hostif_hdr header;
- u32 mib_attribute;
+ __le32 mib_attribute;
} __packed;
struct hostif_mib_value_t {
- u16 size;
- u16 type;
+ __le16 size;
+ __le16 type;
#define MIB_VALUE_TYPE_NULL 0
#define MIB_VALUE_TYPE_INT 1
#define MIB_VALUE_TYPE_BOOL 2
struct hostif_mib_get_confirm_t {
struct hostif_hdr header;
- u32 mib_status;
+ __le32 mib_status;
#define MIB_SUCCESS 0
#define MIB_INVALID 1
#define MIB_READ_ONLY 2
#define MIB_WRITE_ONLY 3
- u32 mib_attribute;
+ __le32 mib_attribute;
struct hostif_mib_value_t mib_value;
} __packed;
struct hostif_mib_set_request_t {
struct hostif_hdr header;
- u32 mib_attribute;
+ __le32 mib_attribute;
struct hostif_mib_value_t mib_value;
} __packed;
struct hostif_mib_set_confirm_t {
struct hostif_hdr header;
- u32 mib_status;
- u32 mib_attribute;
+ __le32 mib_status;
+ __le32 mib_attribute;
} __packed;
struct hostif_power_mgmt_request_t {
struct hostif_hdr header;
- u32 mode;
+ __le32 mode;
#define POWER_ACTIVE 1
#define POWER_SAVE 2
- u32 wake_up;
+ __le32 wake_up;
#define SLEEP_FALSE 0
#define SLEEP_TRUE 1 /* not used */
- u32 receiveDTIMs;
+ __le32 receive_dtims;
#define DTIM_FALSE 0
#define DTIM_TRUE 1
} __packed;
struct hostif_power_mgmt_confirm_t {
struct hostif_hdr header;
- u16 result_code;
+ __le16 result_code;
} __packed;
struct hostif_start_request_t {
struct hostif_hdr header;
- u16 mode;
+ __le16 mode;
#define MODE_PSEUDO_ADHOC 0
#define MODE_INFRASTRUCTURE 1
#define MODE_AP 2 /* not used */
struct hostif_start_confirm_t {
struct hostif_hdr header;
- u16 result_code;
+ __le16 result_code;
} __packed;
#define SSID_MAX_SIZE 32
u8 rate_pad;
} __packed;
-struct FhParms_t {
- u16 dwellTime;
- u8 hopSet;
- u8 hopPattern;
- u8 hopIndex;
+struct fh_parms_t {
+ __le16 dwell_time;
+ u8 hop_set;
+ u8 hop_pattern;
+ u8 hop_index;
} __packed;
-struct DsParms_t {
+struct ds_parms_t {
u8 channel;
} __packed;
-struct CfParms_t {
+struct cf_parms_t {
u8 count;
u8 period;
- u16 maxDuration;
- u16 durRemaining;
+ __le16 max_duration;
+ __le16 dur_remaining;
} __packed;
-struct IbssParms_t {
- u16 atimWindow;
+struct ibss_parms_t {
+ __le16 atim_window;
} __packed;
struct rsn_t {
u8 body[RSN_BODY_SIZE];
} __packed;
-struct ErpParams_t {
+struct erp_params_t {
u8 erp_info;
} __packed;
u8 sq; /* +07 */
u8 noise; /* +08 */
u8 pad0; /* +09 */
- u16 beacon_period; /* +10 */
- u16 capability; /* +12 */
+ __le16 beacon_period; /* +10 */
+ __le16 capability; /* +12 */
#define BSS_CAP_ESS BIT(0)
#define BSS_CAP_IBSS BIT(1)
#define BSS_CAP_CF_POLABLE BIT(2)
u8 ch_info; /* +15 */
#define FRAME_TYPE_BEACON 0x80
#define FRAME_TYPE_PROBE_RESP 0x50
- u16 body_size; /* +16 */
+ __le16 body_size; /* +16 */
u8 body[1024]; /* +18 */
/* +1032 */
} __packed;
u8 sq; /* +07 */
u8 noise; /* +08 */
u8 pad0; /* +09 */
- u16 beacon_period; /* +10 */
- u16 capability; /* +12 */
+ __le16 beacon_period; /* +10 */
+ __le16 capability; /* +12 */
struct rate_set8_t rate_set; /* +14 */
- struct FhParms_t fh_parameter; /* +24 */
- struct DsParms_t ds_parameter; /* +29 */
- struct CfParms_t cf_parameter; /* +30 */
- struct IbssParms_t ibss_parameter; /* +36 */
- struct ErpParams_t erp_parameter; /* +38 */
+ struct fh_parms_t fh_parameter; /* +24 */
+ struct ds_parms_t ds_parameter; /* +29 */
+ struct cf_parms_t cf_parameter; /* +30 */
+ struct ibss_parms_t ibss_parameter; /* +36 */
+ struct erp_params_t erp_parameter; /* +38 */
u8 pad1; /* +39 */
struct rate_set8_t ext_rate_set; /* +40 */
u8 DTIM_period; /* +50 */
struct hostif_connect_indication_t {
struct hostif_hdr header;
- u16 connect_code;
+ __le16 connect_code;
#define RESULT_CONNECT 0
#define RESULT_DISCONNECT 1
struct link_ap_info_t link_ap_info;
struct hostif_stop_confirm_t {
struct hostif_hdr header;
- u16 result_code;
+ __le16 result_code;
} __packed;
/**
*/
struct hostif_ps_adhoc_set_request_t {
struct hostif_hdr header;
- u16 phy_type;
+ __le16 phy_type;
#define D_11B_ONLY_MODE 0
#define D_11G_ONLY_MODE 1
#define D_11BG_COMPATIBLE_MODE 2
#define D_11A_ONLY_MODE 3
- u16 cts_mode;
+ __le16 cts_mode;
#define CTS_MODE_FALSE 0
#define CTS_MODE_TRUE 1
- u16 channel;
+ __le16 channel;
struct rate_set16_t rate_set;
- u16 capability;
- u16 scan_type;
+ __le16 capability;
+ __le16 scan_type;
} __packed;
struct hostif_ps_adhoc_set_confirm_t {
struct hostif_hdr header;
- u16 result_code;
+ __le16 result_code;
} __packed;
/**
*/
struct hostif_infrastructure_set_request_t {
struct hostif_hdr header;
- u16 phy_type;
- u16 cts_mode;
+ __le16 phy_type;
+ __le16 cts_mode;
struct rate_set16_t rate_set;
struct ssid_t ssid;
- u16 capability;
- u16 beacon_lost_count;
- u16 auth_type;
+ __le16 capability;
+ __le16 beacon_lost_count;
+ __le16 auth_type;
#define AUTH_TYPE_OPEN_SYSTEM 0
#define AUTH_TYPE_SHARED_KEY 1
struct channel_list_t channel_list;
- u16 scan_type;
+ __le16 scan_type;
} __packed;
/**
*/
struct hostif_infrastructure_set2_request_t {
struct hostif_hdr header;
- u16 phy_type;
- u16 cts_mode;
+ __le16 phy_type;
+ __le16 cts_mode;
struct rate_set16_t rate_set;
struct ssid_t ssid;
- u16 capability;
- u16 beacon_lost_count;
- u16 auth_type;
+ __le16 capability;
+ __le16 beacon_lost_count;
+ __le16 auth_type;
#define AUTH_TYPE_OPEN_SYSTEM 0
#define AUTH_TYPE_SHARED_KEY 1
struct channel_list_t channel_list;
- u16 scan_type;
+ __le16 scan_type;
u8 bssid[ETH_ALEN];
} __packed;
struct hostif_infrastructure_set_confirm_t {
struct hostif_hdr header;
- u16 result_code;
+ __le16 result_code;
} __packed;
/**
*/
struct hostif_adhoc_set_request_t {
struct hostif_hdr header;
- u16 phy_type;
- u16 cts_mode;
- u16 channel;
+ __le16 phy_type;
+ __le16 cts_mode;
+ __le16 channel;
struct rate_set16_t rate_set;
struct ssid_t ssid;
- u16 capability;
- u16 scan_type;
+ __le16 capability;
+ __le16 scan_type;
} __packed;
/**
*/
struct hostif_adhoc_set2_request_t {
struct hostif_hdr header;
- u16 phy_type;
- u16 cts_mode;
- u16 reserved;
+ __le16 phy_type;
+ __le16 cts_mode;
+ __le16 reserved;
struct rate_set16_t rate_set;
struct ssid_t ssid;
- u16 capability;
- u16 scan_type;
+ __le16 capability;
+ __le16 scan_type;
struct channel_list_t channel_list;
u8 bssid[ETH_ALEN];
} __packed;
struct hostif_adhoc_set_confirm_t {
struct hostif_hdr header;
- u16 result_code;
+ __le16 result_code;
} __packed;
struct last_associate_t {
#define FRAME_TYPE_ASSOC_REQ 0x00
#define FRAME_TYPE_REASSOC_REQ 0x20
u8 pad;
- u16 capability;
- u16 listen_interval;
+ __le16 capability;
+ __le16 listen_interval;
u8 ap_address[6];
- u16 reqIEs_size;
+ __le16 req_ies_size;
} __packed;
struct association_response_t {
#define FRAME_TYPE_ASSOC_RESP 0x10
#define FRAME_TYPE_REASSOC_RESP 0x30
u8 pad;
- u16 capability;
- u16 status;
- u16 association_id;
- u16 respIEs_size;
+ __le16 capability;
+ __le16 status;
+ __le16 association_id;
+ __le16 resp_ies_size;
} __packed;
struct hostif_associate_indication_t {
struct hostif_hdr header;
struct association_request_t assoc_req;
struct association_response_t assoc_resp;
- /* followed by (reqIEs_size + respIEs_size) octets of data */
+ /* followed by (req_ies_size + resp_ies_size) octets of data */
/* reqIEs data *//* respIEs data */
} __packed;
#define ACTIVE_SCAN 0
#define PASSIVE_SCAN 1
u8 pad[3];
- u32 ch_time_min;
- u32 ch_time_max;
+ __le32 ch_time_min;
+ __le32 ch_time_max;
struct channel_list_t channel_list;
struct ssid_t ssid;
} __packed;
struct hostif_bss_scan_confirm_t {
struct hostif_hdr header;
- u16 result_code;
- u16 reserved;
+ __le16 result_code;
+ __le16 reserved;
} __packed;
struct hostif_phy_information_request_t {
struct hostif_hdr header;
- u16 type;
+ __le16 type;
#define NORMAL_TYPE 0
#define TIME_TYPE 1
- u16 time; /* unit 100ms */
+ __le16 time; /* unit 100ms */
} __packed;
struct hostif_phy_information_confirm_t {
u8 sq;
u8 noise;
u8 link_speed;
- u32 tx_frame;
- u32 rx_frame;
- u32 tx_error;
- u32 rx_error;
+ __le32 tx_frame;
+ __le32 rx_frame;
+ __le32 tx_error;
+ __le32 rx_error;
} __packed;
enum sleep_mode_type {
struct hostif_sleep_confirm_t {
struct hostif_hdr header;
- u16 result_code;
+ __le16 result_code;
} __packed;
struct hostif_mic_failure_request_t {
struct hostif_hdr header;
- u16 failure_count;
- u16 timer;
+ __le16 failure_count;
+ __le16 timer;
} __packed;
struct hostif_mic_failure_confirm_t {
struct hostif_hdr header;
- u16 result_code;
+ __le16 result_code;
} __packed;
#define BASIC_RATE 0x80
};
struct local_gain_t {
- u8 TxMode;
- u8 RxMode;
- u8 TxGain;
- u8 RxGain;
+ u8 tx_mode;
+ u8 rx_mode;
+ u8 tx_gain;
+ u8 rx_gain;
};
struct local_eeprom_sum_t {
netdev_info(dev, "SET_SLEEP_MODE %d\n", priv->sleep_mode);
hostif_sme_enqueue(priv, SME_SLEEP_REQUEST);
} else {
- netdev_err(dev, "SET_SLEEP_MODE %d errror\n", *uwrq);
+ netdev_err(dev, "SET_SLEEP_MODE %d error\n", *uwrq);
return -EINVAL;
}
return -EPERM;
/* for SLEEP MODE */
if (*uwrq >= 0 && *uwrq <= 0xFF) /* 0-255 */
- priv->gain.TxGain = (uint8_t)*uwrq;
+ priv->gain.tx_gain = (uint8_t)*uwrq;
else
return -EINVAL;
- if (priv->gain.TxGain < 0xFF)
- priv->gain.TxMode = 1;
+ if (priv->gain.tx_gain < 0xFF)
+ priv->gain.tx_mode = 1;
else
- priv->gain.TxMode = 0;
+ priv->gain.tx_mode = 0;
hostif_sme_enqueue(priv, SME_SET_GAIN);
return 0;
if (priv->sleep_mode == SLP_SLEEP)
return -EPERM;
/* for SLEEP MODE */
- *uwrq = priv->gain.TxGain;
+ *uwrq = priv->gain.tx_gain;
hostif_sme_enqueue(priv, SME_GET_GAIN);
return 0;
}
return -EPERM;
/* for SLEEP MODE */
if (*uwrq >= 0 && *uwrq <= 0xFF) /* 0-255 */
- priv->gain.RxGain = (uint8_t)*uwrq;
+ priv->gain.rx_gain = (uint8_t)*uwrq;
else
return -EINVAL;
- if (priv->gain.RxGain < 0xFF)
- priv->gain.RxMode = 1;
+ if (priv->gain.rx_gain < 0xFF)
+ priv->gain.rx_mode = 1;
else
- priv->gain.RxMode = 0;
+ priv->gain.rx_mode = 0;
hostif_sme_enqueue(priv, SME_SET_GAIN);
return 0;
if (priv->sleep_mode == SLP_SLEEP)
return -EPERM;
/* for SLEEP MODE */
- *uwrq = priv->gain.RxGain;
+ *uwrq = priv->gain.rx_gain;
hostif_sme_enqueue(priv, SME_GET_GAIN);
return 0;
}
memset(&srcaddr, 0, sizeof(srcaddr));
srcaddr.sin_family = AF_INET;
- srcaddr.sin_addr.s_addr = (__force u32)htonl(dev->ibd_ifip);
+ srcaddr.sin_addr.s_addr = htonl(dev->ibd_ifip);
memset(&dstaddr, 0, sizeof(dstaddr));
dstaddr.sin_family = AF_INET;
memset(&addr, 0, sizeof(addr));
addr.sin_family = AF_INET;
- addr.sin_addr.s_addr = (__force u32)htonl(dev->ibd_ifip);
+ addr.sin_addr.s_addr = htonl(dev->ibd_ifip);
addr.sin_port = htons(*kiblnd_tunables.kib_service);
/* Bind to failover device or port */
kiblnd_connd(void *arg)
{
spinlock_t *lock = &kiblnd_data.kib_connd_lock;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
unsigned long flags;
struct kib_conn *conn;
int timeout;
long id = (long)arg;
struct kib_sched_info *sched;
struct kib_conn *conn;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
unsigned long flags;
struct ib_wc wc;
int did_something;
{
rwlock_t *glock = &kiblnd_data.kib_global_lock;
struct kib_dev *dev;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
unsigned long flags;
int rc;
{
spinlock_t *connd_lock = &ksocknal_data.ksnd_connd_lock;
struct ksock_connreq *cr;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
int nloops = 0;
int cons_retry = 0;
int
ksocknal_reaper(void *arg)
{
- wait_queue_t wait;
+ wait_queue_entry_t wait;
struct ksock_conn *conn;
struct ksock_sched *sched;
struct list_head enomem_conns;
void libcfs_debug_dumplog(void)
{
- wait_queue_t wait;
+ wait_queue_entry_t wait;
struct task_struct *dumper;
/* we're being careful to ensure that the kernel thread is
memset(cfs_trace_data, 0, sizeof(cfs_trace_data));
for (i = 0; i < CFS_TCD_TYPE_MAX; i++) {
cfs_trace_data[i] =
- kmalloc(sizeof(union cfs_trace_data_union) *
- num_possible_cpus(), GFP_KERNEL);
+ kmalloc_array(num_possible_cpus(),
+ sizeof(union cfs_trace_data_union),
+ GFP_KERNEL);
if (!cfs_trace_data[i])
goto out;
}
} else {
tage = cfs_tage_alloc(GFP_ATOMIC);
if (unlikely(!tage)) {
- if ((!memory_pressure_get() ||
- in_interrupt()) && printk_ratelimit())
- pr_warn("cannot allocate a tage (%ld)\n",
- tcd->tcd_cur_pages);
+ if (!memory_pressure_get() || in_interrupt())
+ pr_warn_ratelimited("cannot allocate a tage (%ld)\n",
+ tcd->tcd_cur_pages);
return NULL;
}
}
* from here: this will lead to infinite recursion.
*/
- if (printk_ratelimit())
- pr_warn("debug daemon buffer overflowed; discarding 10%% of pages (%d of %ld)\n",
- pgcount + 1, tcd->tcd_cur_pages);
+ pr_warn_ratelimited("debug daemon buffer overflowed; discarding 10%% of pages (%d of %ld)\n",
+ pgcount + 1, tcd->tcd_cur_pages);
INIT_LIST_HEAD(&pc.pc_pages);
complete(&tctl->tctl_start);
while (1) {
- wait_queue_t __wait;
+ wait_queue_entry_t __wait;
pc.pc_want_daemon_pages = 0;
collect_pages(&pc);
{
int tms = *timeout_ms;
int wait;
- wait_queue_t wl;
+ wait_queue_entry_t wl;
unsigned long now;
if (!tms)
int rc;
/* Convert put fields to host byte order */
- hdr->msg.put.match_bits = le64_to_cpu(hdr->msg.put.match_bits);
- hdr->msg.put.ptl_index = le32_to_cpu(hdr->msg.put.ptl_index);
- hdr->msg.put.offset = le32_to_cpu(hdr->msg.put.offset);
+ le64_to_cpus(&hdr->msg.put.match_bits);
+ le32_to_cpus(&hdr->msg.put.ptl_index);
+ le32_to_cpus(&hdr->msg.put.offset);
info.mi_id.nid = hdr->src_nid;
info.mi_id.pid = hdr->src_pid;
int rc;
/* Convert get fields to host byte order */
- hdr->msg.get.match_bits = le64_to_cpu(hdr->msg.get.match_bits);
- hdr->msg.get.ptl_index = le32_to_cpu(hdr->msg.get.ptl_index);
- hdr->msg.get.sink_length = le32_to_cpu(hdr->msg.get.sink_length);
- hdr->msg.get.src_offset = le32_to_cpu(hdr->msg.get.src_offset);
+ le64_to_cpus(&hdr->msg.get.match_bits);
+ le32_to_cpus(&hdr->msg.get.ptl_index);
+ le32_to_cpus(&hdr->msg.get.sink_length);
+ le32_to_cpus(&hdr->msg.get.src_offset);
info.mi_id.nid = hdr->src_nid;
info.mi_id.pid = hdr->src_pid;
src.pid = hdr->src_pid;
/* Convert ack fields to host byte order */
- hdr->msg.ack.match_bits = le64_to_cpu(hdr->msg.ack.match_bits);
- hdr->msg.ack.mlength = le32_to_cpu(hdr->msg.ack.mlength);
+ le64_to_cpus(&hdr->msg.ack.match_bits);
+ le32_to_cpus(&hdr->msg.ack.mlength);
cpt = lnet_cpt_of_cookie(hdr->msg.ack.dst_wmd.wh_object_cookie);
lnet_res_lock(cpt);
/* convert common msg->hdr fields to host byteorder */
msg->msg_hdr.type = type;
msg->msg_hdr.src_nid = src_nid;
- msg->msg_hdr.src_pid = le32_to_cpu(msg->msg_hdr.src_pid);
+ le32_to_cpus(&msg->msg_hdr.src_pid);
msg->msg_hdr.dest_nid = dest_nid;
msg->msg_hdr.dest_pid = dest_pid;
msg->msg_hdr.payload_length = payload_length;
struct ifreq ifr;
int nob;
int rc;
- __u32 val;
+ __be32 val;
nob = strnlen(name, IFNAMSIZ);
if (nob == IFNAMSIZ) {
int
lnet_sock_accept(struct socket **newsockp, struct socket *sock)
{
- wait_queue_t wait;
+ wait_queue_entry_t wait;
struct socket *newsock;
int rc;
}
static int seq_fid_alloc_prep(struct lu_client_seq *seq,
- wait_queue_t *link)
+ wait_queue_entry_t *link)
{
if (seq->lcs_update) {
add_wait_queue(&seq->lcs_waitq, link);
int seq_client_alloc_fid(const struct lu_env *env,
struct lu_client_seq *seq, struct lu_fid *fid)
{
- wait_queue_t link;
+ wait_queue_entry_t link;
int rc;
LASSERT(seq);
return rc;
}
- CDEBUG(D_INFO, "%s: Switch to sequence [0x%16.16Lx]\n",
+ CDEBUG(D_INFO, "%s: Switch to sequence [0x%16.16llx]\n",
seq->lcs_name, seqnr);
seq->lcs_fid.f_oid = LUSTRE_FID_INIT_OID;
*fid = seq->lcs_fid;
mutex_unlock(&seq->lcs_mutex);
- CDEBUG(D_INFO, "%s: Allocated FID "DFID"\n", seq->lcs_name, PFID(fid));
+ CDEBUG(D_INFO, "%s: Allocated FID " DFID "\n", seq->lcs_name, PFID(fid));
return rc;
}
EXPORT_SYMBOL(seq_client_alloc_fid);
*/
void seq_client_flush(struct lu_client_seq *seq)
{
- wait_queue_t link;
+ wait_queue_entry_t link;
LASSERT(seq);
init_waitqueue_entry(&link, current);
continue;
LASSERTF(c_range->lsr_start <= n_range->lsr_start,
- "cur lsr_start "DRANGE" next lsr_start "DRANGE"\n",
+ "cur lsr_start " DRANGE " next lsr_start " DRANGE "\n",
PRANGE(c_range), PRANGE(n_range));
/* check merge possibility with next range */
f_curr->fce_range.lsr_end = new_start;
fld_cache_entry_add(cache, f_new, &f_curr->fce_list);
} else
- CERROR("NEW range ="DRANGE" curr = "DRANGE"\n",
+ CERROR("NEW range =" DRANGE " curr = " DRANGE "\n",
PRANGE(range), PRANGE(&f_curr->fce_range));
}
if (!prev)
prev = head;
- CDEBUG(D_INFO, "insert range "DRANGE"\n", PRANGE(&f_new->fce_range));
+ CDEBUG(D_INFO, "insert range " DRANGE "\n", PRANGE(&f_new->fce_range));
/* Add new entry to cache and lru list. */
fld_cache_entry_add(cache, f_new, prev);
out:
return 0;
}
-static struct file_operations fld_debugfs_cache_flush_fops = {
+static const struct file_operations fld_debugfs_cache_flush_fops = {
.owner = THIS_MODULE,
.open = simple_open,
.write = fld_debugfs_cache_flush_write,
* @{
*/
struct cl_page_list {
- unsigned pl_nr;
+ unsigned int pl_nr;
struct list_head pl_pages;
struct task_struct *pl_owner;
};
/**
* Number of pages owned by this IO. For invariant checking.
*/
- unsigned ci_owned_nr;
+ unsigned int ci_owned_nr;
};
/** @} cl_io */
struct lprocfs_vars {
const char *name;
- struct file_operations *fops;
+ const struct file_operations *fops;
void *data;
/**
* sysfs file mode.
struct dentry *ldebugfs_add_simple(struct dentry *root,
char *name,
void *data,
- struct file_operations *fops);
+ const struct file_operations *fops);
int ldebugfs_register_stats(struct dentry *parent,
const char *name,
void lprocfs_stats_collect(struct lprocfs_stats *stats, int idx,
struct lprocfs_counter *cnt);
-int lprocfs_single_release(struct inode *, struct file *);
-int lprocfs_seq_release(struct inode *, struct file *);
+int lprocfs_single_release(struct inode *inode, struct file *file);
+int lprocfs_seq_release(struct inode *inode, struct file *file);
/* write the name##_seq_show function, call LPROC_SEQ_FOPS_RO for read-only
* proc entries; otherwise, you will define name##_seq_write function also for
{ \
return single_open(file, name##_seq_show, inode->i_private); \
} \
-static struct file_operations name##_fops = { \
+static const struct file_operations name##_fops = { \
.owner = THIS_MODULE, \
.open = name##_single_open, \
.read = seq_read, \
{ \
return single_open(file, NULL, inode->i_private); \
} \
- static struct file_operations name##_##type##_fops = { \
+ static const struct file_operations name##_##type##_fops = { \
.open = name##_##type##_open, \
.write = name##_##type##_write, \
.release = lprocfs_single_release, \
* Version counter used to skip calls to lu_context_refill() when no
* keys were registered.
*/
- unsigned lc_version;
+ unsigned int lc_version;
/**
* Debugging cookie.
*/
- unsigned lc_cookie;
+ unsigned int lc_cookie;
};
/**
static inline int fid_set_id(struct lu_fid *fid, __u64 oid)
{
if (unlikely(fid_seq_is_igif(fid->f_seq))) {
- CERROR("bad IGIF, "DFID"\n", PFID(fid));
+ CERROR("bad IGIF, " DFID "\n", PFID(fid));
return -EBADF;
}
__u64 seq = ostid_seq(ostid);
if (ost_idx > 0xffff) {
- CERROR("bad ost_idx, "DOSTID" ost_idx:%u\n", POSTID(ostid),
+ CERROR("bad ost_idx, " DOSTID " ost_idx:%u\n", POSTID(ostid),
ost_idx);
return -EBADF;
}
static inline int fid_to_ostid(const struct lu_fid *fid, struct ost_id *ostid)
{
if (unlikely(fid_seq_is_igif(fid->f_seq))) {
- CERROR("bad IGIF, "DFID"\n", PFID(fid));
+ CERROR("bad IGIF, " DFID "\n", PFID(fid));
return -EBADF;
}
#define FID_NOBRACE_LEN 40
#define FID_LEN (FID_NOBRACE_LEN + 2)
#define DFID_NOBRACE "%#llx:0x%x:0x%x"
-#define DFID "["DFID_NOBRACE"]"
+#define DFID "[" DFID_NOBRACE "]"
#define PFID(fid) (unsigned long long)(fid)->f_seq, (fid)->f_oid, (fid)->f_ver
/* scanf input parse format for fids in DFID_NOBRACE format
static inline int lu_fid_diff(const struct lu_fid *fid1,
const struct lu_fid *fid2)
{
- LASSERTF(fid_seq(fid1) == fid_seq(fid2), "fid1:"DFID", fid2:"DFID"\n",
+ LASSERTF(fid_seq(fid1) == fid_seq(fid2), "fid1:" DFID ", fid2:" DFID "\n",
PFID(fid1), PFID(fid2));
if (fid_is_idif(fid1) && fid_is_idif(fid2))
sigmask(SIGALRM))
/**
- * wait_queue_t of Linux (version < 2.6.34) is a FIFO list for exclusively
+ * wait_queue_entry_t of Linux (version < 2.6.34) is a FIFO list for exclusively
* waiting threads, which is not always desirable because all threads will
* be waken up again and again, even user only needs a few of them to be
* active most time. This is not good for performance because cache can
*/
#define __l_wait_event(wq, condition, info, ret, l_add_wait) \
do { \
- wait_queue_t __wait; \
+ wait_queue_entry_t __wait; \
long __timeout = info->lwi_timeout; \
sigset_t __blocked; \
int __allow_intr = info->lwi_allow_intr; \
return atomic_read(&ns->ns_bref) == 0;
}
-void ldlm_namespace_move_to_active_locked(struct ldlm_namespace *,
- enum ldlm_side);
-void ldlm_namespace_move_to_inactive_locked(struct ldlm_namespace *,
- enum ldlm_side);
-struct ldlm_namespace *ldlm_namespace_first_locked(enum ldlm_side);
+void ldlm_namespace_move_to_active_locked(struct ldlm_namespace *ns,
+ enum ldlm_side client);
+void ldlm_namespace_move_to_inactive_locked(struct ldlm_namespace *ns,
+ enum ldlm_side client);
+struct ldlm_namespace *ldlm_namespace_first_locked(enum ldlm_side client);
/* ldlm_request.c */
/* Cancel lru flag, it indicates we cancel aged locks. */
int ldlm_fill_lvb(struct ldlm_lock *lock, struct req_capsule *pill,
enum req_location loc, void *data, int size);
struct ldlm_lock *
-ldlm_lock_create(struct ldlm_namespace *ns, const struct ldlm_res_id *,
+ldlm_lock_create(struct ldlm_namespace *ns, const struct ldlm_res_id *id,
enum ldlm_type type, enum ldlm_mode mode,
const struct ldlm_callback_suite *cbs,
void *data, __u32 lvb_len, enum lvb_type lvb_type);
-enum ldlm_error ldlm_lock_enqueue(struct ldlm_namespace *, struct ldlm_lock **,
- void *cookie, __u64 *flags);
-void ldlm_lock_addref_internal(struct ldlm_lock *, enum ldlm_mode mode);
-void ldlm_lock_addref_internal_nolock(struct ldlm_lock *, enum ldlm_mode mode);
-void ldlm_lock_decref_internal(struct ldlm_lock *, enum ldlm_mode mode);
-void ldlm_lock_decref_internal_nolock(struct ldlm_lock *, enum ldlm_mode mode);
+enum ldlm_error ldlm_lock_enqueue(struct ldlm_namespace *ns,
+ struct ldlm_lock **lock, void *cookie,
+ __u64 *flags);
+void ldlm_lock_addref_internal(struct ldlm_lock *lock, enum ldlm_mode mode);
+void ldlm_lock_addref_internal_nolock(struct ldlm_lock *lock,
+ enum ldlm_mode mode);
+void ldlm_lock_decref_internal(struct ldlm_lock *lock, enum ldlm_mode mode);
+void ldlm_lock_decref_internal_nolock(struct ldlm_lock *lock,
+ enum ldlm_mode mode);
int ldlm_run_ast_work(struct ldlm_namespace *ns, struct list_head *rpc_list,
enum ldlm_desc_ast_t ast_type);
int ldlm_lock_remove_from_lru_check(struct ldlm_lock *lock, time_t last_use);
*/
cli->cl_chunkbits = PAGE_SHIFT;
- if (!strcmp(name, LUSTRE_MDC_NAME)) {
+ if (!strcmp(name, LUSTRE_MDC_NAME))
cli->cl_max_rpcs_in_flight = OBD_MAX_RIF_DEFAULT;
- } else if (totalram_pages >> (20 - PAGE_SHIFT) <= 128 /* MB */) {
+ else if (totalram_pages >> (20 - PAGE_SHIFT) <= 128 /* MB */)
cli->cl_max_rpcs_in_flight = 2;
- } else if (totalram_pages >> (20 - PAGE_SHIFT) <= 256 /* MB */) {
+ else if (totalram_pages >> (20 - PAGE_SHIFT) <= 256 /* MB */)
cli->cl_max_rpcs_in_flight = 3;
- } else if (totalram_pages >> (20 - PAGE_SHIFT) <= 512 /* MB */) {
+ else if (totalram_pages >> (20 - PAGE_SHIFT) <= 512 /* MB */)
cli->cl_max_rpcs_in_flight = 4;
- } else {
+ else
cli->cl_max_rpcs_in_flight = OBD_MAX_RIF_DEFAULT;
- }
spin_lock_init(&cli->cl_mod_rpcs_lock);
spin_lock_init(&cli->cl_mod_rpcs_hist.oh_lock);
if (!ldlm_res_eq(&reply->lock_desc.l_resource.lr_name,
&lock->l_resource->lr_name)) {
- CDEBUG(D_INFO, "remote intent success, locking "DLDLMRES
- " instead of "DLDLMRES"\n",
+ CDEBUG(D_INFO, "remote intent success, locking " DLDLMRES
+ " instead of " DLDLMRES "\n",
PLDLMRES(&reply->lock_desc.l_resource),
PLDLMRES(lock->l_resource));
0, flags | LCF_BL_AST, opaque);
rc = ldlm_cli_cancel_list(&cancels, count, NULL, flags);
if (rc != ELDLM_OK)
- CERROR("canceling unused lock "DLDLMRES": rc = %d\n",
+ CERROR("canceling unused lock " DLDLMRES ": rc = %d\n",
PLDLMRES(res), rc);
LDLM_RESOURCE_DELREF(res);
LPROC_SEQ_FOPS_WR_ONLY(ldlm, dump_ns);
-LPROC_SEQ_FOPS_RW_TYPE(ldlm_rw, uint);
+static int ldlm_rw_uint_seq_show(struct seq_file *m, void *v)
+{
+ seq_printf(m, "%u\n", *(unsigned int *)m->private);
+ return 0;
+}
+
+static ssize_t
+ldlm_rw_uint_seq_write(struct file *file, const char __user *buffer,
+ size_t count, loff_t *off)
+{
+ struct seq_file *seq = file->private_data;
+
+ if (count == 0)
+ return 0;
+ return kstrtouint_from_user(buffer, count, 0,
+ (unsigned int *)seq->private);
+}
+
+LPROC_SEQ_FOPS(ldlm_rw_uint);
static struct lprocfs_vars ldlm_debugfs_list[] = {
{ "dump_namespaces", &ldlm_dump_ns_fops, NULL, 0222 },
struct ldlm_resource *res = cfs_hash_object(hs, hnode);
lock_res(res);
- CERROR("%s: namespace resource "DLDLMRES
+ CERROR("%s: namespace resource " DLDLMRES
" (%p) refcount nonzero (%d) after lock cleanup; forcing cleanup.\n",
ldlm_ns_name(ldlm_res_to_ns(res)), PLDLMRES(res), res,
atomic_read(&res->lr_refcount) - 1);
if (!((libcfs_debug | D_ERROR) & level))
return;
- CDEBUG(level, "--- Resource: "DLDLMRES" (%p) refcount = %d\n",
+ CDEBUG(level, "--- Resource: " DLDLMRES " (%p) refcount = %d\n",
PLDLMRES(res), res, atomic_read(&res->lr_refcount));
if (!list_empty(&res->lr_granted)) {
{
struct dentry *dentry;
- CDEBUG(D_INODE, "marking dentries for ino "DFID"(%p) invalid\n",
+ CDEBUG(D_INODE, "marking dentries for ino " DFID "(%p) invalid\n",
PFID(ll_inode2fid(inode)), inode);
spin_lock(&inode->i_lock);
if (it->it_lock_mode && inode) {
struct ll_sb_info *sbi = ll_i2sbi(inode);
- CDEBUG(D_DLMTRACE, "setting l_data to inode "DFID"(%p)\n",
+ CDEBUG(D_DLMTRACE, "setting l_data to inode " DFID "(%p)\n",
PFID(ll_inode2fid(inode)), inode);
ll_set_lock_data(sbi->ll_md_exp, inode, it, NULL);
}
struct md_op_data *op_data;
int rc;
- CDEBUG(D_VFSTRACE, "VFS Op:inode="DFID"(%p) pos/size %lu/%llu 32bit_api %d\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:inode=" DFID "(%p) pos/size %lu/%llu 32bit_api %d\n",
PFID(ll_inode2fid(inode)), inode, (unsigned long)pos,
i_size_read(inode), api32);
if (unlikely(lump->lum_magic != LMV_USER_MAGIC))
return -EINVAL;
- CDEBUG(D_VFSTRACE, "VFS Op:inode="DFID"(%p) name %s stripe_offset %d, stripe_count: %u\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:inode=" DFID "(%p) name %s stripe_offset %d, stripe_count: %u\n",
PFID(ll_inode2fid(parent)), parent, dirname,
(int)lump->lum_stripe_offset, lump->lum_stripe_count);
rc = md_getattr(sbi->ll_md_exp, op_data, &req);
ll_finish_md_op_data(op_data);
if (rc < 0) {
- CDEBUG(D_INFO, "md_getattr failed on inode "DFID": rc %d\n",
+ CDEBUG(D_INFO, "md_getattr failed on inode " DFID ": rc %d\n",
PFID(ll_inode2fid(inode)), rc);
goto out;
}
iput(inode);
if (rc != 0) {
CDEBUG(D_HSM, "Could not read file data version of "
- DFID" (rc = %d). Archive request (%#llx) could not be done.\n",
+ DFID " (rc = %d). Archive request (%#llx) could not be done.\n",
PFID(©->hc_hai.hai_fid), rc,
copy->hc_hai.hai_cookie);
hpk.hpk_flags |= HP_FLAG_RETRY;
if ((copy->hc_hai.hai_action == HSMA_ARCHIVE) &&
(copy->hc_data_version != data_version)) {
CDEBUG(D_HSM, "File data version mismatched. File content was changed during archiving. "
- DFID", start:%#llx current:%#llx\n",
+ DFID ", start:%#llx current:%#llx\n",
PFID(©->hc_hai.hai_fid),
copy->hc_data_version, data_version);
/* File was changed, send error to cdt. Do not ask for
struct obd_ioctl_data *data;
int rc = 0;
- CDEBUG(D_VFSTRACE, "VFS Op:inode="DFID"(%p), cmd=%#x\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:inode=" DFID "(%p), cmd=%#x\n",
PFID(ll_inode2fid(inode)), inode, cmd);
/* asm-ppc{,64} declares TCGETS, et. al. as type 't' not 'T' */
}
#if OBD_OCD_VERSION(2, 9, 50, 0) > LUSTRE_VERSION_CODE
- mode = data->ioc_type != 0 ? data->ioc_type : S_IRWXUGO;
+ mode = data->ioc_type != 0 ? data->ioc_type : 0777;
#else
mode = data->ioc_type;
#endif
struct ll_inode_info *lli = ll_i2info(inode);
int rc;
- CDEBUG(D_VFSTRACE, "VFS Op:inode="DFID"(%p)\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:inode=" DFID "(%p)\n",
PFID(ll_inode2fid(inode)), inode);
if (!is_root_inode(inode))
struct ll_file_data *fd;
int rc = 0;
- CDEBUG(D_VFSTRACE, "VFS Op:inode="DFID"(%p), flags %o\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:inode=" DFID "(%p), flags %o\n",
PFID(ll_inode2fid(inode)), inode, file->f_flags);
it = file->private_data; /* XXX: compat macro */
}
rc2 = ll_close_inode_openhandle(inode, och, 0, NULL);
if (rc2 < 0)
- CERROR("%s: error closing file "DFID": %d\n",
+ CERROR("%s: error closing file " DFID ": %d\n",
ll_get_fsname(inode->i_sb, NULL, 0),
PFID(&ll_i2info(inode)->lli_fid), rc2);
och = NULL; /* och has been freed in ll_close_inode_openhandle() */
int rc;
u16 refcheck;
- CDEBUG(D_INODE, "%s: Releasing file "DFID".\n",
+ CDEBUG(D_INODE, "%s: Releasing file " DFID ".\n",
ll_get_fsname(inode->i_sb, NULL, 0),
PFID(&ll_i2info(inode)->lli_fid));
struct ll_file_data *fd = LUSTRE_FPRIVATE(file);
int flags, rc;
- CDEBUG(D_VFSTRACE, "VFS Op:inode="DFID"(%p),cmd=%x\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:inode=" DFID "(%p),cmd=%x\n",
PFID(ll_inode2fid(inode)), inode, cmd);
ll_stats_ops_tally(ll_i2sbi(inode), LPROC_LL_IOCTL, 1);
retval = offset + ((origin == SEEK_END) ? i_size_read(inode) :
(origin == SEEK_CUR) ? file->f_pos : 0);
- CDEBUG(D_VFSTRACE, "VFS Op:inode="DFID"(%p), to=%llu=%#llx(%d)\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:inode=" DFID "(%p), to=%llu=%#llx(%d)\n",
PFID(ll_inode2fid(inode)), inode, retval, retval, origin);
ll_stats_ops_tally(ll_i2sbi(inode), LPROC_LL_LLSEEK, 1);
struct ptlrpc_request *req;
int rc, err;
- CDEBUG(D_VFSTRACE, "VFS Op:inode="DFID"(%p)\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:inode=" DFID "(%p)\n",
PFID(ll_inode2fid(inode)), inode);
ll_stats_ops_tally(ll_i2sbi(inode), LPROC_LL_FSYNC, 1);
int rc;
int rc2 = 0;
- CDEBUG(D_VFSTRACE, "VFS Op:inode="DFID" file_lock=%p\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:inode=" DFID " file_lock=%p\n",
PFID(ll_inode2fid(inode)), file_lock);
ll_stats_ops_tally(ll_i2sbi(inode), LPROC_LL_FLOCK, 1);
if (IS_ERR(op_data))
return PTR_ERR(op_data);
- CDEBUG(D_DLMTRACE, "inode="DFID", pid=%u, flags=%#llx, mode=%u, start=%llu, end=%llu\n",
+ CDEBUG(D_DLMTRACE, "inode=" DFID ", pid=%u, flags=%#llx, mode=%u, start=%llu, end=%llu\n",
PFID(ll_inode2fid(inode)), flock.l_flock.pid, flags,
einfo.ei_mode, flock.l_flock.start, flock.l_flock.end);
struct qstr qstr;
int rc;
- CDEBUG(D_VFSTRACE, "migrate %s under "DFID" to MDT%d\n",
+ CDEBUG(D_VFSTRACE, "migrate %s under " DFID " to MDT%d\n",
name, PFID(ll_inode2fid(parent)), mdtidx);
op_data = ll_prep_md_op_data(NULL, parent, NULL, name, namelen,
inode_lock(child_inode);
op_data->op_fid3 = *ll_inode2fid(child_inode);
if (!fid_is_sane(&op_data->op_fid3)) {
- CERROR("%s: migrate %s, but fid "DFID" is insane\n",
+ CERROR("%s: migrate %s, but fid " DFID " is insane\n",
ll_get_fsname(parent->i_sb, NULL, 0), name,
PFID(&op_data->op_fid3));
rc = -EINVAL;
goto out_unlock;
if (rc == mdtidx) {
- CDEBUG(D_INFO, "%s:"DFID" is already on MDT%d.\n", name,
+ CDEBUG(D_INFO, "%s: " DFID " is already on MDT%d.\n", name,
PFID(&op_data->op_fid3), mdtidx);
rc = 0;
goto out_unlock;
return 0;
fid = &ll_i2info(inode)->lli_fid;
- CDEBUG(D_INFO, "trying to match res "DFID" mode %s\n", PFID(fid),
+ CDEBUG(D_INFO, "trying to match res " DFID " mode %s\n", PFID(fid),
ldlm_lockname[mode]);
flags = LDLM_FL_BLOCK_GRANTED | LDLM_FL_CBPENDING | LDLM_FL_TEST_LOCK;
struct lu_fid *fid;
fid = &ll_i2info(inode)->lli_fid;
- CDEBUG(D_INFO, "trying to match res "DFID"\n", PFID(fid));
+ CDEBUG(D_INFO, "trying to match res " DFID "\n", PFID(fid));
return md_lock_match(ll_i2mdexp(inode), flags | LDLM_FL_BLOCK_GRANTED,
fid, LDLM_IBITS, &policy, mode, lockh);
return 0;
} else if (rc != 0) {
CDEBUG_LIMIT((rc == -EACCES || rc == -EIDRM) ? D_INFO : D_ERROR,
- "%s: revalidate FID "DFID" error: rc = %d\n",
+ "%s: revalidate FID " DFID " error: rc = %d\n",
ll_get_fsname(inode->i_sb, NULL, 0),
PFID(ll_inode2fid(inode)), rc);
}
struct obd_export *exp;
int rc = 0;
- CDEBUG(D_VFSTRACE, "VFS Op:inode="DFID"(%p),name=%pd\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:inode=" DFID "(%p),name=%pd\n",
PFID(ll_inode2fid(inode)), inode, dentry);
exp = ll_i2mdexp(inode);
return rc;
}
- CDEBUG(D_VFSTRACE, "VFS Op:inode="DFID"(%p), inode mode %x mask %o\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:inode=" DFID "(%p), inode mode %x mask %o\n",
PFID(ll_inode2fid(inode)), inode, inode->i_mode, mask);
/* squash fsuid/fsgid if needed */
}
/* -o localflock - only provides locally consistent flock locks */
-struct file_operations ll_file_operations = {
+const struct file_operations ll_file_operations = {
.read_iter = ll_file_read_iter,
.write_iter = ll_file_write_iter,
.unlocked_ioctl = ll_file_ioctl,
.flush = ll_flush
};
-struct file_operations ll_file_operations_flock = {
+const struct file_operations ll_file_operations_flock = {
.read_iter = ll_file_read_iter,
.write_iter = ll_file_write_iter,
.unlocked_ioctl = ll_file_ioctl,
};
/* These are for -o noflock - to return ENOSYS on flock calls */
-struct file_operations ll_file_operations_noflock = {
+const struct file_operations ll_file_operations_noflock = {
.read_iter = ll_file_read_iter,
.write_iter = ll_file_write_iter,
.unlocked_ioctl = ll_file_ioctl,
int lmmsize;
int rc;
- CDEBUG(D_INODE, DFID" LVB_READY=%d l_lvb_data=%p l_lvb_len=%d\n",
+ CDEBUG(D_INODE, DFID " LVB_READY=%d l_lvb_data=%p l_lvb_len=%d\n",
PFID(ll_inode2fid(inode)), ldlm_is_lvb_ready(lock),
lock->l_lvb_data, lock->l_lvb_len);
/* wait for IO to complete if it's still being used. */
if (wait_layout) {
- CDEBUG(D_INODE, "%s: "DFID"(%p) wait for layout reconf\n",
+ CDEBUG(D_INODE, "%s: " DFID "(%p) wait for layout reconf\n",
ll_get_fsname(inode->i_sb, NULL, 0),
PFID(&lli->lli_fid), inode);
if (rc == 0)
rc = -EAGAIN;
- CDEBUG(D_INODE, "%s: file="DFID" waiting layout return: %d.\n",
+ CDEBUG(D_INODE, "%s: file=" DFID " waiting layout return: %d.\n",
ll_get_fsname(inode->i_sb, NULL, 0),
PFID(&lli->lli_fid), rc);
}
it.it_op = IT_LAYOUT;
lockh.cookie = 0ULL;
- LDLM_DEBUG_NOLOCK("%s: requeue layout lock for file "DFID"(%p)",
+ LDLM_DEBUG_NOLOCK("%s: requeue layout lock for file " DFID "(%p)",
ll_get_fsname(inode->i_sb, NULL, 0),
PFID(&lli->lli_fid), inode);
static void cl_object_put_last(struct lu_env *env, struct cl_object *obj)
{
struct lu_object_header *header = obj->co_lu.lo_header;
- wait_queue_t waiter;
+ wait_queue_entry_t waiter;
if (unlikely(atomic_read(&header->loh_ref) != 1)) {
struct lu_site *site = obj->co_lu.lo_dev->ld_site;
struct ll_ra_info ll_ra_info;
unsigned int ll_namelen;
- struct file_operations *ll_fop;
+ const struct file_operations *ll_fop;
unsigned int ll_md_brw_pages; /* readdir pages per RPC */
struct inode *ll_iget(struct super_block *sb, ino_t hash,
struct lustre_md *lic);
int ll_test_inode_by_fid(struct inode *inode, void *opaque);
-int ll_md_blocking_ast(struct ldlm_lock *, struct ldlm_lock_desc *,
+int ll_md_blocking_ast(struct ldlm_lock *lock, struct ldlm_lock_desc *desc,
void *data, int flag);
struct dentry *ll_splice_alias(struct inode *inode, struct dentry *de);
void ll_update_times(struct ptlrpc_request *request, struct inode *inode);
/* llite/rw.c */
int ll_writepage(struct page *page, struct writeback_control *wbc);
-int ll_writepages(struct address_space *, struct writeback_control *wbc);
+int ll_writepages(struct address_space *mapping, struct writeback_control *wbc);
int ll_readpage(struct file *file, struct page *page);
void ll_readahead_init(struct inode *inode, struct ll_readahead_state *ras);
int vvp_io_write_commit(const struct lu_env *env, struct cl_io *io);
extern const struct address_space_operations ll_aops;
/* llite/file.c */
-extern struct file_operations ll_file_operations;
-extern struct file_operations ll_file_operations_flock;
-extern struct file_operations ll_file_operations_noflock;
+extern const struct file_operations ll_file_operations;
+extern const struct file_operations ll_file_operations_flock;
+extern const struct file_operations ll_file_operations_noflock;
extern const struct inode_operations ll_file_inode_operations;
int ll_have_md_lock(struct inode *inode, __u64 *bits,
enum ldlm_mode l_req_mode);
enum ldlm_mode mode);
int ll_file_open(struct inode *inode, struct file *file);
int ll_file_release(struct inode *inode, struct file *file);
-int ll_release_openhandle(struct inode *, struct lookup_intent *);
+int ll_release_openhandle(struct inode *inode, struct lookup_intent *it);
int ll_md_real_close(struct inode *inode, fmode_t fmode);
int ll_getattr(const struct path *path, struct kstat *stat,
u32 request_mask, unsigned int flags);
/* llite/dcache.c */
extern const struct dentry_operations ll_d_ops;
-void ll_intent_drop_lock(struct lookup_intent *);
-void ll_intent_release(struct lookup_intent *);
-void ll_invalidate_aliases(struct inode *);
+void ll_intent_drop_lock(struct lookup_intent *it);
+void ll_intent_release(struct lookup_intent *it);
+void ll_invalidate_aliases(struct inode *inode);
void ll_lookup_finish_locks(struct lookup_intent *it, struct inode *inode);
int ll_revalidate_it_finish(struct ptlrpc_request *request,
struct lookup_intent *it, struct inode *inode);
int ll_show_options(struct seq_file *seq, struct dentry *dentry);
void ll_dirty_page_discard_warn(struct page *page, int ioret);
int ll_prep_inode(struct inode **inode, struct ptlrpc_request *req,
- struct super_block *, struct lookup_intent *);
+ struct super_block *sb, struct lookup_intent *it);
int ll_obd_statfs(struct inode *inode, void __user *arg);
int ll_get_max_mdsize(struct ll_sb_info *sbi, int *max_mdsize);
int ll_get_default_mdsize(struct ll_sb_info *sbi, int *default_mdsize);
*/
if (it->it_remote_lock_mode) {
handle.cookie = it->it_remote_lock_handle;
- CDEBUG(D_DLMTRACE, "setting l_data to inode "DFID"%p for remote lock %#llx\n",
+ CDEBUG(D_DLMTRACE, "setting l_data to inode " DFID "%p for remote lock %#llx\n",
PFID(ll_inode2fid(inode)), inode,
handle.cookie);
md_set_lock_data(exp, &handle, inode, NULL);
handle.cookie = it->it_lock_handle;
- CDEBUG(D_DLMTRACE, "setting l_data to inode "DFID"%p for lock %#llx\n",
+ CDEBUG(D_DLMTRACE, "setting l_data to inode " DFID "%p for lock %#llx\n",
PFID(ll_inode2fid(inode)), inode, handle.cookie);
md_set_lock_data(exp, &handle, inode, &it->it_lock_bits);
goto out_lock_cn_cb;
}
if (!fid_is_sane(&sbi->ll_root_fid)) {
- CERROR("%s: Invalid root fid "DFID" during mount\n",
+ CERROR("%s: Invalid root fid " DFID " during mount\n",
sbi->ll_md_exp->exp_obd->obd_name,
PFID(&sbi->ll_root_fid));
err = -EINVAL;
goto out_lock_cn_cb;
}
- CDEBUG(D_SUPER, "rootfid "DFID"\n", PFID(&sbi->ll_root_fid));
+ CDEBUG(D_SUPER, "rootfid " DFID "\n", PFID(&sbi->ll_root_fid));
sb->s_op = &lustre_super_operations;
sb->s_xattr = ll_xattr_handlers;
ino = cl_fid_build_ino(fid, sbi->ll_flags & LL_SBI_32BIT_API);
inode = iget_locked(sb, ino);
if (!inode) {
- CERROR("%s: failed get simple inode "DFID": rc = -ENOENT\n",
+ CERROR("%s: failed get simple inode " DFID ": rc = -ENOENT\n",
ll_get_fsname(sb, NULL, 0), PFID(fid));
return ERR_PTR(-ENOENT);
}
inode->i_mode = (inode->i_mode & ~S_IFMT) |
(body->mbo_mode & S_IFMT);
- LASSERTF(S_ISDIR(inode->i_mode), "Not slave inode "DFID"\n",
+ LASSERTF(S_ISDIR(inode->i_mode), "Not slave inode " DFID "\n",
PFID(fid));
LTIME_S(inode->i_mtime) = 0;
LASSERT(lsm);
/* master object FID */
lli->lli_pfid = body->mbo_fid1;
- CDEBUG(D_INODE, "lli %p slave "DFID" master "DFID"\n",
+ CDEBUG(D_INODE, "lli %p slave " DFID " master " DFID "\n",
lli, PFID(fid), PFID(&lli->lli_pfid));
unlock_new_inode(inode);
}
int rc;
LASSERT(S_ISDIR(inode->i_mode));
- CDEBUG(D_INODE, "update lsm %p of "DFID"\n", lli->lli_lsm_md,
+ CDEBUG(D_INODE, "update lsm %p of " DFID "\n", lli->lli_lsm_md,
PFID(ll_inode2fid(inode)));
/* no striped information from request. */
* migration is done, the temporay MIGRATE layout has
* been removed
*/
- CDEBUG(D_INODE, DFID" finish migration.\n",
+ CDEBUG(D_INODE, DFID " finish migration.\n",
PFID(ll_inode2fid(inode)));
lmv_free_memmd(lli->lli_lsm_md);
lli->lli_lsm_md = NULL;
kfree(attr);
- CDEBUG(D_INODE, "Set lsm %p magic %x to "DFID"\n", lsm,
+ CDEBUG(D_INODE, "Set lsm %p magic %x to " DFID "\n", lsm,
lsm->lsm_md_magic, PFID(ll_inode2fid(inode)));
return 0;
}
struct lmv_stripe_md *old_lsm = lli->lli_lsm_md;
int idx;
- CERROR("%s: inode "DFID"(%p)'s lmv layout mismatch (%p)/(%p) magic:0x%x/0x%x stripe count: %d/%d master_mdt: %d/%d hash_type:0x%x/0x%x layout: 0x%x/0x%x pool:%s/%s\n",
+ CERROR("%s: inode " DFID "(%p)'s lmv layout mismatch (%p)/(%p) magic:0x%x/0x%x stripe count: %d/%d master_mdt: %d/%d hash_type:0x%x/0x%x layout: 0x%x/0x%x pool:%s/%s\n",
ll_get_fsname(inode->i_sb, NULL, 0), PFID(&lli->lli_fid),
inode, lsm, old_lsm,
lsm->lsm_md_magic, old_lsm->lsm_md_magic,
old_lsm->lsm_md_pool_name);
for (idx = 0; idx < old_lsm->lsm_md_stripe_count; idx++) {
- CERROR("%s: sub FIDs in old lsm idx %d, old: "DFID"\n",
+ CERROR("%s: sub FIDs in old lsm idx %d, old: " DFID "\n",
ll_get_fsname(inode->i_sb, NULL, 0), idx,
PFID(&old_lsm->lsm_md_oinfo[idx].lmo_fid));
}
for (idx = 0; idx < lsm->lsm_md_stripe_count; idx++) {
- CERROR("%s: sub FIDs in new lsm idx %d, new: "DFID"\n",
+ CERROR("%s: sub FIDs in new lsm idx %d, new: " DFID "\n",
ll_get_fsname(inode->i_sb, NULL, 0), idx,
PFID(&lsm->lsm_md_oinfo[idx].lmo_fid));
}
struct ll_inode_info *lli = ll_i2info(inode);
struct ll_sb_info *sbi = ll_i2sbi(inode);
- CDEBUG(D_VFSTRACE, "VFS Op:inode="DFID"(%p)\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:inode=" DFID "(%p)\n",
PFID(ll_inode2fid(inode)), inode);
if (S_ISDIR(inode->i_mode)) {
struct md_op_data *op_data = NULL;
int rc = 0;
- CDEBUG(D_VFSTRACE, "%s: setattr inode "DFID"(%p) from %llu to %llu, valid %x, hsm_import %d\n",
+ CDEBUG(D_VFSTRACE, "%s: setattr inode " DFID "(%p) from %llu to %llu, valid %x, hsm_import %d\n",
ll_get_fsname(inode->i_sb, NULL, 0), PFID(&lli->lli_fid), inode,
i_size_read(inode), attr->ia_size, attr->ia_valid, hsm_import);
* needs another check in addition to the VFS check above.
*/
if (attr->ia_size > ll_file_maxbytes(inode)) {
- CDEBUG(D_INODE, "file "DFID" too large %llu > %llu\n",
+ CDEBUG(D_INODE, "file " DFID " too large %llu > %llu\n",
PFID(&lli->lli_fid), attr->ia_size,
ll_file_maxbytes(inode));
return -EFBIG;
/* FID shouldn't be changed! */
if (fid_is_sane(&lli->lli_fid)) {
LASSERTF(lu_fid_eq(&lli->lli_fid, &body->mbo_fid1),
- "Trying to change FID "DFID" to the "DFID", inode "DFID"(%p)\n",
+ "Trying to change FID " DFID " to the " DFID ", inode " DFID "(%p)\n",
PFID(&lli->lli_fid), PFID(&body->mbo_fid1),
PFID(ll_inode2fid(inode)), inode);
} else {
struct ll_inode_info *lli = ll_i2info(inode);
int rc;
- CDEBUG(D_VFSTRACE, "VFS Op:inode="DFID"(%p)\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:inode=" DFID "(%p)\n",
PFID(&lli->lli_fid), inode);
/* Core attributes from the MDS first. This is a new inode, and
rc = md_getattr(sbi->ll_md_exp, op_data, &req);
ll_finish_md_op_data(op_data);
if (rc) {
- CERROR("%s: failure inode "DFID": rc = %d\n",
+ CERROR("%s: failure inode " DFID ": rc = %d\n",
sbi->ll_md_exp->exp_obd->obd_name,
PFID(ll_inode2fid(inode)), rc);
return -abs(rc);
if (!printed && ++count > 16) {
const struct dentry *de = vma->vm_file->f_path.dentry;
- CWARN("app(%s): the page %lu of file "DFID" is under heavy contention\n",
+ CWARN("app(%s): the page %lu of file " DFID " is under heavy contention\n",
current->comm, vmf->pgoff,
PFID(ll_inode2fid(de->d_inode)));
printed = true;
struct md_op_data *op_data;
int rc;
- CDEBUG(D_INFO, "searching inode for:(%lu,"DFID")\n", hash, PFID(fid));
+ CDEBUG(D_INFO, "searching inode for:(%lu," DFID ")\n", hash, PFID(fid));
inode = ilookup5(sb, hash, ll_test_inode_by_fid, (void *)fid);
if (inode)
rc = md_getattr(sbi->ll_md_exp, op_data, &req);
kfree(op_data);
if (rc) {
- CDEBUG(D_INFO, "can't get object attrs, fid "DFID", rc %d\n",
+ CDEBUG(D_INFO, "can't get object attrs, fid " DFID ", rc %d\n",
PFID(fid), rc);
return ERR_PTR(rc);
}
int fileid_len = sizeof(struct lustre_nfs_fid) / 4;
struct lustre_nfs_fid *nfs_fid = (void *)fh;
- CDEBUG(D_INFO, "%s: encoding for ("DFID") maxlen=%d minlen=%d\n",
+ CDEBUG(D_INFO, "%s: encoding for (" DFID ") maxlen=%d minlen=%d\n",
ll_get_fsname(inode->i_sb, NULL, 0),
PFID(ll_inode2fid(inode)), *plen, fileid_len);
sbi = ll_s2sbi(dir->i_sb);
- CDEBUG(D_INFO, "%s: getting parent for ("DFID")\n",
+ CDEBUG(D_INFO, "%s: getting parent for (" DFID ")\n",
ll_get_fsname(dir->i_sb, NULL, 0),
PFID(ll_inode2fid(dir)));
rc = md_getattr_name(sbi->ll_md_exp, op_data, &req);
ll_finish_md_op_data(op_data);
if (rc) {
- CERROR("%s: failure inode "DFID" get parent: rc = %d\n",
+ CERROR("%s: failure inode " DFID " get parent: rc = %d\n",
ll_get_fsname(dir->i_sb, NULL, 0),
PFID(ll_inode2fid(dir)), rc);
return rc;
#include "vvp_internal.h"
/* debugfs llite mount point registration */
-static struct file_operations ll_rw_extents_stats_fops;
-static struct file_operations ll_rw_extents_stats_pp_fops;
-static struct file_operations ll_rw_offset_stats_fops;
+static const struct file_operations ll_rw_extents_stats_fops;
+static const struct file_operations ll_rw_extents_stats_pp_fops;
+static const struct file_operations ll_rw_offset_stats_fops;
static ssize_t blocksize_show(struct kobject *kobj, struct attribute *attr,
char *buf)
inode->i_mode = (inode->i_mode & ~S_IFMT) | (body->mbo_mode & S_IFMT);
if (unlikely(inode->i_mode == 0)) {
- CERROR("Invalid inode "DFID" type\n", PFID(&lli->lli_fid));
+ CERROR("Invalid inode " DFID " type\n", PFID(&lli->lli_fid));
return -EINVAL;
}
}
} else if (!(inode->i_state & (I_FREEING | I_CLEAR))) {
rc = ll_update_inode(inode, md);
- CDEBUG(D_VFSTRACE, "got inode: "DFID"(%p): rc = %d\n",
+ CDEBUG(D_VFSTRACE, "got inode: " DFID "(%p): rc = %d\n",
PFID(&md->body->mbo_fid1), inode, rc);
if (rc) {
if (S_ISDIR(inode->i_mode))
if (!fid_res_name_eq(ll_inode2fid(inode),
&lock->l_resource->lr_name)) {
- LDLM_ERROR(lock, "data mismatch with object "DFID"(%p)",
+ LDLM_ERROR(lock, "data mismatch with object " DFID "(%p)",
PFID(ll_inode2fid(inode)), inode);
LBUG();
}
rc = ll_layout_conf(inode, &conf);
if (rc < 0)
CDEBUG(D_INODE, "cannot invalidate layout of "
- DFID": rc = %d\n",
+ DFID ": rc = %d\n",
PFID(ll_inode2fid(inode)), rc);
}
if ((bits & MDS_INODELOCK_UPDATE) && S_ISDIR(inode->i_mode)) {
struct ll_inode_info *lli = ll_i2info(inode);
- CDEBUG(D_INODE, "invalidating inode "DFID" lli = %p, pfid = "DFID"\n",
+ CDEBUG(D_INODE, "invalidating inode " DFID " lli = %p, pfid = " DFID "\n",
PFID(ll_inode2fid(inode)), lli,
PFID(&lli->lli_pfid));
* we have to invalidate the negative children
* on master inode
*/
- CDEBUG(D_INODE, "Invalidate s"DFID" m"DFID"\n",
+ CDEBUG(D_INODE, "Invalidate s" DFID " m" DFID "\n",
PFID(ll_inode2fid(inode)),
PFID(&lli->lli_pfid));
if (dentry->d_name.len > ll_i2sbi(parent)->ll_namelen)
return ERR_PTR(-ENAMETOOLONG);
- CDEBUG(D_VFSTRACE, "VFS Op:name=%pd, dir="DFID"(%p),intent=%s\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:name=%pd, dir=" DFID "(%p),intent=%s\n",
dentry, PFID(ll_inode2fid(parent)), parent, LL_IT2STR(it));
if (d_mountpoint(dentry))
struct lookup_intent *itp, it = { .it_op = IT_GETATTR };
struct dentry *de;
- CDEBUG(D_VFSTRACE, "VFS Op:name=%pd, dir="DFID"(%p),flags=%u\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:name=%pd, dir=" DFID "(%p),flags=%u\n",
dentry, PFID(ll_inode2fid(parent)), parent, flags);
/* Optimize away (CREATE && !OPEN). Let .create handle the race.
* together.
*/
static int ll_atomic_open(struct inode *dir, struct dentry *dentry,
- struct file *file, unsigned open_flags,
+ struct file *file, unsigned int open_flags,
umode_t mode, int *opened)
{
struct lookup_intent *it;
struct dentry *de;
int rc = 0;
- CDEBUG(D_VFSTRACE, "VFS Op:name=%pd, dir="DFID"(%p),file %p,open_flags %x,mode %x opened %d\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:name=%pd, dir=" DFID "(%p),file %p,open_flags %x,mode %x opened %d\n",
dentry, PFID(ll_inode2fid(dir)), dir, file, open_flags, mode,
*opened);
* lock on the inode. Since we finally have an inode pointer,
* stuff it in the lock.
*/
- CDEBUG(D_DLMTRACE, "setting l_ast_data to inode "DFID"(%p)\n",
+ CDEBUG(D_DLMTRACE, "setting l_ast_data to inode " DFID "(%p)\n",
PFID(ll_inode2fid(dir)), inode);
ll_set_lock_data(sbi->ll_md_exp, inode, it, NULL);
out:
struct inode *inode;
int rc = 0;
- CDEBUG(D_VFSTRACE, "VFS Op:name=%pd, dir="DFID"(%p), intent=%s\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:name=%pd, dir=" DFID "(%p), intent=%s\n",
dentry, PFID(ll_inode2fid(dir)), dir, LL_IT2STR(it));
rc = it_open_error(DISP_OPEN_CREATE, it);
LASSERT(body);
if (body->mbo_valid & OBD_MD_FLMTIME &&
body->mbo_mtime > LTIME_S(inode->i_mtime)) {
- CDEBUG(D_INODE, "setting fid "DFID" mtime from %lu to %llu\n",
+ CDEBUG(D_INODE, "setting fid " DFID " mtime from %lu to %llu\n",
PFID(ll_inode2fid(inode)), LTIME_S(inode->i_mtime),
body->mbo_mtime);
LTIME_S(inode->i_mtime) = body->mbo_mtime;
{
int err;
- CDEBUG(D_VFSTRACE, "VFS Op:name=%pd, dir="DFID"(%p) mode %o dev %x\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:name=%pd, dir=" DFID "(%p) mode %o dev %x\n",
dchild, PFID(ll_inode2fid(dir)), dir, mode,
old_encode_dev(rdev));
{
int rc;
- CDEBUG(D_VFSTRACE, "VFS Op:name=%pd, dir="DFID"(%p), flags=%u, excl=%d\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:name=%pd, dir=" DFID "(%p), flags=%u, excl=%d\n",
dentry, PFID(ll_inode2fid(dir)), dir, mode, want_excl);
rc = ll_mknod(dir, dentry, mode, 0);
{
int err;
- CDEBUG(D_VFSTRACE, "VFS Op:name=%pd, dir"DFID"(%p)\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:name=%pd, dir" DFID "(%p)\n",
dentry, PFID(ll_inode2fid(dir)), dir);
if (!IS_POSIXACL(dir) || !exp_connect_umask(ll_i2mdexp(dir)))
struct md_op_data *op_data;
int rc;
- CDEBUG(D_VFSTRACE, "VFS Op:name=%pd, dir="DFID"(%p)\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:name=%pd, dir=" DFID "(%p)\n",
dchild, PFID(ll_inode2fid(dir)), dir);
op_data = ll_prep_md_op_data(NULL, dir, NULL,
{
int err;
- CDEBUG(D_VFSTRACE, "VFS Op:name=%pd, dir="DFID"(%p),target=%.*s\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:name=%pd, dir=" DFID "(%p),target=%.*s\n",
dentry, PFID(ll_inode2fid(dir)), dir, 3000, oldname);
err = ll_new_node(dir, dentry, oldname, S_IFLNK | 0777,
struct md_op_data *op_data;
int err;
- CDEBUG(D_VFSTRACE, "VFS Op: inode="DFID"(%p), dir="DFID"(%p), target=%pd\n",
+ CDEBUG(D_VFSTRACE, "VFS Op: inode=" DFID "(%p), dir=" DFID "(%p), target=%pd\n",
PFID(ll_inode2fid(src)), src, PFID(ll_inode2fid(dir)), dir,
new_dentry);
return -EINVAL;
CDEBUG(D_VFSTRACE,
- "VFS Op:oldname=%pd, src_dir="DFID"(%p), newname=%pd, tgt_dir="DFID"(%p)\n",
+ "VFS Op:oldname=%pd, src_dir=" DFID "(%p), newname=%pd, tgt_dir=" DFID "(%p)\n",
src_dchild, PFID(ll_inode2fid(src)), src,
tgt_dchild, PFID(ll_inode2fid(tgt)), tgt);
if ((file_offset & ~PAGE_MASK) || (count & ~PAGE_MASK))
return -EINVAL;
- CDEBUG(D_VFSTRACE, "VFS Op:inode="DFID"(%p), size=%zd (max %lu), offset=%lld=%llx, pages %zd (max %lu)\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:inode=" DFID "(%p), size=%zd (max %lu), offset=%lld=%llx, pages %zd (max %lu)\n",
PFID(ll_inode2fid(inode)), inode, count, MAX_DIO_SIZE,
file_offset, file_offset, count >> PAGE_SHIFT,
MAX_DIO_SIZE >> PAGE_SHIFT);
}
static int ll_write_end(struct file *file, struct address_space *mapping,
- loff_t pos, unsigned len, unsigned copied,
+ loff_t pos, unsigned int len, unsigned int copied,
struct page *vmpage, void *fsdata)
{
struct ll_cl_context *lcc = fsdata;
}
CDEBUG(D_READA, "Handling (init) async glimpse: inode = "
- DFID", idx = %llu\n", PFID(&lli->lli_fid), index);
+ DFID ", idx = %llu\n", PFID(&lli->lli_fid), index);
cl_agl(inode);
lli->lli_agl_index = 0;
up_write(&lli->lli_glimpse_sem);
CDEBUG(D_READA, "Handled (init) async glimpse: inode= "
- DFID", idx = %llu, rc = %d\n",
+ DFID ", idx = %llu, rc = %d\n",
PFID(&lli->lli_fid), index, rc);
iput(inode);
sai->sai_in_readpage = 0;
if (IS_ERR(page)) {
rc = PTR_ERR(page);
- CDEBUG(D_READA, "error reading dir "DFID" at %llu/%llu: opendir_pid = %u: rc = %d\n",
+ CDEBUG(D_READA, "error reading dir " DFID " at %llu/%llu: opendir_pid = %u: rc = %d\n",
PFID(ll_inode2fid(dir)), pos, sai->sai_index,
lli->lli_opendir_pid, rc);
break;
if (sa_low_hit(sai)) {
rc = -EFAULT;
atomic_inc(&sbi->ll_sa_wrong);
- CDEBUG(D_READA, "Statahead for dir "DFID" hit ratio too low: hit/miss %llu/%llu, sent/replied %llu/%llu, stopping statahead thread: pid %d\n",
+ CDEBUG(D_READA, "Statahead for dir " DFID " hit ratio too low: hit/miss %llu/%llu, sent/replied %llu/%llu, stopping statahead thread: pid %d\n",
PFID(&lli->lli_fid), sai->sai_hit,
sai->sai_miss, sai->sai_sent,
sai->sai_replied, current_pid());
LASSERT(lli->lli_opendir_key == key);
LASSERT(lli->lli_opendir_pid);
- CDEBUG(D_READA, "deauthorize statahead for "DFID"\n",
+ CDEBUG(D_READA, "deauthorize statahead for " DFID "\n",
PFID(&lli->lli_fid));
spin_lock(&lli->lli_sa_lock);
struct ll_inode_info *lli = ll_i2info(dir);
rc = PTR_ERR(page);
- CERROR("%s: error reading dir "DFID" at %llu: opendir_pid = %u : rc = %d\n",
+ CERROR("%s: error reading dir " DFID " at %llu: opendir_pid = %u : rc = %d\n",
ll_get_fsname(dir->i_sb, NULL, 0),
PFID(ll_inode2fid(dir)), pos,
lli->lli_opendir_pid, rc);
} else if ((*dentryp)->d_inode != inode) {
/* revalidate, but inode is recreated */
CDEBUG(D_READA,
- "%s: stale dentry %pd inode "DFID", statahead inode "DFID"\n",
+ "%s: stale dentry %pd inode " DFID ", statahead inode " DFID "\n",
ll_get_fsname((*dentryp)->d_inode->i_sb,
NULL, 0),
*dentryp,
ll_finish_md_op_data(op_data);
if (rc) {
if (rc != -ENOENT)
- CERROR("%s: inode "DFID": rc = %d\n",
+ CERROR("%s: inode " DFID ": rc = %d\n",
ll_get_fsname(inode->i_sb, NULL, 0),
PFID(ll_inode2fid(inode)), rc);
goto failed;
LASSERT(symlen != 0);
if (body->mbo_eadatasize != symlen) {
- CERROR("%s: inode "DFID": symlink length %d not expected %d\n",
+ CERROR("%s: inode " DFID ": symlink length %d not expected %d\n",
ll_get_fsname(inode->i_sb, NULL, 0),
PFID(ll_inode2fid(inode)), body->mbo_eadatasize - 1,
symlen - 1);
#define PGC_DEPTH_SHIFT (32)
struct vvp_pgcache_id {
- unsigned vpi_bucket;
- unsigned vpi_depth;
+ unsigned int vpi_bucket;
+ unsigned int vpi_depth;
uint32_t vpi_index;
- unsigned vpi_curdep;
+ unsigned int vpi_curdep;
struct lu_object_header *vpi_obj;
};
io->ci_need_restart = vio->vui_layout_gen != gen;
if (io->ci_need_restart) {
CDEBUG(D_VFSTRACE,
- DFID" layout changed from %d to %d.\n",
+ DFID " layout changed from %d to %d.\n",
PFID(lu_object_fid(&obj->co_lu)),
vio->vui_layout_gen, gen);
/* today successful restore is the only possible case */
atomic_read(&obj->vob_mmap_cnt), inode);
if (inode) {
lli = ll_i2info(inode);
- (*p)(env, cookie, "%lu/%u %o %u %d %p "DFID,
+ (*p)(env, cookie, "%lu/%u %o %u %d %p " DFID,
inode->i_ino, inode->i_generation, inode->i_mode,
inode->i_nlink, atomic_read(&inode->i_count),
lli->lli_clob, PFID(&lli->lli_fid));
LASSERT(inode);
LASSERT(name);
- CDEBUG(D_VFSTRACE, "VFS Op:inode="DFID"(%p), xattr %s\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:inode=" DFID "(%p), xattr %s\n",
PFID(ll_inode2fid(inode)), inode, name);
if (!strcmp(name, "lov")) {
#endif
int rc;
- CDEBUG(D_VFSTRACE, "VFS Op:inode="DFID"(%p)\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:inode=" DFID "(%p)\n",
PFID(ll_inode2fid(inode)), inode);
ll_stats_ops_tally(ll_i2sbi(inode), LPROC_LL_GETXATTR, 1);
LASSERT(inode);
- CDEBUG(D_VFSTRACE, "VFS Op:inode="DFID"(%p)\n",
+ CDEBUG(D_VFSTRACE, "VFS Op:inode=" DFID "(%p)\n",
PFID(ll_inode2fid(inode)), inode);
ll_stats_ops_tally(ll_i2sbi(inode), LPROC_LL_LISTXATTR, 1);
if (rc < 0) {
CDEBUG(D_CACHE,
- "md_intent_lock failed with %d for fid "DFID"\n",
+ "md_intent_lock failed with %d for fid " DFID "\n",
rc, PFID(ll_inode2fid(inode)));
mutex_unlock(&lli->lli_xattrs_enq_lock);
return rc;
}
if (oit->it_status < 0) {
- CDEBUG(D_CACHE, "getxattr intent returned %d for fid "DFID"\n",
+ CDEBUG(D_CACHE, "getxattr intent returned %d for fid " DFID "\n",
oit->it_status, PFID(ll_inode2fid(inode)));
rc = oit->it_status;
/* xattr data is so large that we don't want to cache it */
return rc;
}
- CDEBUG(D_INODE, "FLD lookup got mds #%x for fid="DFID"\n",
+ CDEBUG(D_INODE, "FLD lookup got mds #%x for fid=" DFID "\n",
*mds, PFID(fid));
if (*mds >= lmv->desc.ld_tgt_count) {
lockh = (struct lustre_handle *)&it.it_lock_handle;
if (rc > 0 && !req) {
/* slave inode is still valid */
- CDEBUG(D_INODE, "slave "DFID" is still valid.\n",
+ CDEBUG(D_INODE, "slave " DFID " is still valid.\n",
PFID(&fid));
rc = 0;
} else {
if (IS_ERR(tgt))
return PTR_ERR(tgt);
- CDEBUG(D_INODE, "Try other stripes " DFID"\n",
+ CDEBUG(D_INODE, "Try other stripes " DFID "\n",
PFID(&oinfo->lmo_fid));
op_data->op_fid1 = oinfo->lmo_fid;
LASSERT(fid_is_sane(&op_data->op_fid1));
- CDEBUG(D_INODE, "INTENT LOCK '%s' for "DFID" '%*s' on "DFID"\n",
+ CDEBUG(D_INODE, "INTENT LOCK '%s' for " DFID " '%*s' on " DFID "\n",
LL_IT2STR(it), PFID(&op_data->op_fid2),
(int)op_data->op_namelen, op_data->op_name,
PFID(&op_data->op_fid1));
static int lmv_set_mdc_active(struct lmv_obd *lmv, const struct obd_uuid *uuid,
int activate)
{
- struct lmv_tgt_desc *uninitialized_var(tgt);
+ struct lmv_tgt_desc *tgt = NULL;
struct obd_device *obd;
u32 i;
int rc = 0;
*ptr = '/';
}
- CDEBUG(D_INFO, "%s: get path %s "DFID" rec: %llu ln: %u\n",
+ CDEBUG(D_INFO, "%s: get path %s " DFID " rec: %llu ln: %u\n",
tgt->ltd_exp->exp_obd->obd_name,
gf->gf_path, PFID(&gf->gf_fid), gf->gf_recno,
gf->gf_linkno);
}
if (!fid_is_sane(&gf->gf_fid)) {
- CERROR("%s: invalid FID "DFID": rc = %d\n",
+ CERROR("%s: invalid FID " DFID ": rc = %d\n",
tgt->ltd_exp->exp_obd->obd_name,
PFID(&gf->gf_fid), -EINVAL);
rc = -EINVAL;
if (rc)
return rc;
- CDEBUG(D_INODE, "CBDATA for "DFID"\n", PFID(fid));
+ CDEBUG(D_INODE, "CBDATA for " DFID "\n", PFID(fid));
/*
* With DNE every object can have two locks in different namespaces:
if (IS_ERR(tgt))
return PTR_ERR(tgt);
- CDEBUG(D_INODE, "CLOSE "DFID"\n", PFID(&op_data->op_fid1));
+ CDEBUG(D_INODE, "CLOSE " DFID "\n", PFID(&op_data->op_fid1));
return md_close(tgt->ltd_exp, op_data, mod, request);
}
if (IS_ERR(tgt))
return PTR_ERR(tgt);
- CDEBUG(D_INODE, "CREATE name '%.*s' on "DFID" -> mds #%x\n",
+ CDEBUG(D_INODE, "CREATE name '%.*s' on " DFID " -> mds #%x\n",
(int)op_data->op_namelen, op_data->op_name,
PFID(&op_data->op_fid1), op_data->op_mds);
CDEBUG(D_CONFIG, "Server doesn't support striped dirs\n");
}
- CDEBUG(D_INODE, "CREATE obj "DFID" -> mds #%x\n",
+ CDEBUG(D_INODE, "CREATE obj " DFID " -> mds #%x\n",
PFID(&op_data->op_fid1), op_data->op_mds);
op_data->op_flags |= MF_MDC_CANCEL_FID1;
if (rc == 0) {
if (!*request)
return rc;
- CDEBUG(D_INODE, "Created - "DFID"\n", PFID(&op_data->op_fid2));
+ CDEBUG(D_INODE, "Created - " DFID "\n", PFID(&op_data->op_fid2));
}
return rc;
}
if (rc)
return rc;
- CDEBUG(D_INODE, "ENQUEUE '%s' on "DFID"\n",
+ CDEBUG(D_INODE, "ENQUEUE '%s' on " DFID "\n",
LL_IT2STR(it), PFID(&op_data->op_fid1));
tgt = lmv_locate_mds(lmv, op_data, &op_data->op_fid1);
if (body->mbo_valid & OBD_MD_MDS) {
struct lu_fid rid = body->mbo_fid1;
- CDEBUG(D_INODE, "Request attrs for "DFID"\n",
+ CDEBUG(D_INODE, "Request attrs for " DFID "\n",
PFID(&rid));
tgt = lmv_find_target(lmv, &rid);
}
if (tgt->ltd_idx != op_tgt) {
- CDEBUG(D_INODE, "EARLY_CANCEL on "DFID"\n", PFID(fid));
+ CDEBUG(D_INODE, "EARLY_CANCEL on " DFID "\n", PFID(fid));
policy.l_inodebits.bits = bits;
rc = md_cancel_unused(tgt->ltd_exp, fid, &policy,
mode, LCF_ASYNC, NULL);
} else {
CDEBUG(D_INODE,
- "EARLY_CANCEL skip operation target %d on "DFID"\n",
+ "EARLY_CANCEL skip operation target %d on " DFID "\n",
op_tgt, PFID(fid));
op_data->op_flags |= flag;
rc = 0;
LASSERT(op_data->op_namelen != 0);
- CDEBUG(D_INODE, "LINK "DFID":%*s to "DFID"\n",
+ CDEBUG(D_INODE, "LINK " DFID ":%*s to " DFID "\n",
PFID(&op_data->op_fid2), (int)op_data->op_namelen,
op_data->op_name, PFID(&op_data->op_fid1));
LASSERT(oldlen != 0);
- CDEBUG(D_INODE, "RENAME %.*s in "DFID":%d to %.*s in "DFID":%d\n",
+ CDEBUG(D_INODE, "RENAME %.*s in " DFID ":%d to %.*s in " DFID ":%d\n",
(int)oldlen, old, PFID(&op_data->op_fid1),
op_data->op_mea1 ? op_data->op_mea1->lsm_md_stripe_count : 0,
(int)newlen, new, PFID(&op_data->op_fid2),
op_data->op_cap = cfs_curproc_cap_pack();
if (op_data->op_cli_flags & CLI_MIGRATE) {
- LASSERTF(fid_is_sane(&op_data->op_fid3), "invalid FID "DFID"\n",
+ LASSERTF(fid_is_sane(&op_data->op_fid3), "invalid FID " DFID "\n",
PFID(&op_data->op_fid3));
if (op_data->op_mea1) {
if (rc)
return rc;
- CDEBUG(D_INODE, "SETATTR for "DFID", valid 0x%x\n",
+ CDEBUG(D_INODE, "SETATTR for " DFID ", valid 0x%x\n",
PFID(&op_data->op_fid1), op_data->op_attr.ia_valid);
op_data->op_flags |= MF_MDC_CANCEL_FID1;
struct lu_fid master_fid = op_data->op_fid1;
struct obd_device *obd = exp->exp_obd;
__u64 hash_offset = offset;
+ __u32 ldp_flags;
struct page *min_ent_page = NULL;
struct page *ent_page = NULL;
struct lu_dirent *min_ent = NULL;
dp = kmap(ent_page);
memset(dp, 0, sizeof(*dp));
dp->ldp_hash_start = cpu_to_le64(offset);
- dp->ldp_flags |= LDF_COLLIDE;
+ ldp_flags = LDF_COLLIDE;
area = dp + 1;
left_bytes = PAGE_SIZE - sizeof(*dp);
ent_page = NULL;
} else {
if (ent == area)
- dp->ldp_flags |= LDF_EMPTY;
- dp->ldp_flags = cpu_to_le32(dp->ldp_flags);
+ ldp_flags |= LDF_EMPTY;
+ dp->ldp_flags |= cpu_to_le32(ldp_flags);
dp->ldp_hash_end = cpu_to_le64(hash_offset);
}
if (rc)
return rc;
- if (unlikely(lsm)) {
+ if (unlikely(lsm))
return lmv_read_striped_page(exp, op_data, cb_op, offset, ppage);
- }
tgt = lmv_find_target(lmv, &op_data->op_fid1);
if (IS_ERR(tgt))
if (likely(!(body->mbo_valid & OBD_MD_MDS)))
return rc;
- CDEBUG(D_INODE, "%s: try unlink to another MDT for "DFID"\n",
+ CDEBUG(D_INODE, "%s: try unlink to another MDT for " DFID "\n",
exp->exp_obd->obd_name, PFID(&body->mbo_fid1));
/* This is a remote object, try remote MDT, Note: it may
&lsm->lsm_md_oinfo[i].lmo_mds);
if (rc)
return rc;
- CDEBUG(D_INFO, "unpack fid #%d "DFID"\n", i,
+ CDEBUG(D_INFO, "unpack fid #%d " DFID "\n", i,
PFID(&lsm->lsm_md_oinfo[i].lmo_fid));
}
int tgt;
u32 i;
- CDEBUG(D_INODE, "Lock match for "DFID"\n", PFID(fid));
+ CDEBUG(D_INODE, "Lock match for " DFID "\n", PFID(fid));
/*
* With DNE every object can have two locks in different namespaces:
i < lmv->desc.ld_tgt_count;
i++, tgt = (tgt + 1) % lmv->desc.ld_tgt_count) {
if (tgt < 0) {
- CDEBUG(D_HA, "%s: "DFID" is inaccessible: rc = %d\n",
+ CDEBUG(D_HA, "%s: " DFID " is inaccessible: rc = %d\n",
obd->obd_name, PFID(fid), tgt);
tgt = 0;
}
return -EIO;
}
- if (oqctl->qc_cmd != Q_GETOQUOTA) {
+ if (oqctl->qc_cmd != Q_GETOQUOTA)
return obd_quotactl(tgt->ltd_exp, oqctl);
- }
for (i = 0; i < lmv->desc.ld_tgt_count; i++) {
int err;
for (i = 0; i < lsm->lsm_md_stripe_count; i++) {
struct inode *inode = lsm->lsm_md_oinfo[i].lmo_root;
- CDEBUG(D_INFO, ""DFID" size %llu, blocks %llu nlink %u, atime %lu ctime %lu, mtime %lu.\n",
+ CDEBUG(D_INFO, "" DFID " size %llu, blocks %llu nlink %u, atime %lu ctime %lu, mtime %lu.\n",
PFID(&lsm->lsm_md_oinfo[i].lmo_fid),
i_size_read(inode), (unsigned long long)inode->i_blocks,
inode->i_nlink, LTIME_S(inode->i_atime),
struct ost_lvb lti_lvb;
struct cl_2queue lti_cl2q;
struct cl_page_list lti_plist;
- wait_queue_t lti_waiter;
+ wait_queue_entry_t lti_waiter;
struct cl_attr lti_attr;
};
break;
case CIT_FAULT:
result = -EFAULT;
- CERROR("Page fault on a file without stripes: "DFID"\n",
+ CERROR("Page fault on a file without stripes: " DFID "\n",
PFID(lu_object_fid(&obj->co_lu)));
break;
}
assert_spin_locked(&lsm->lsm_lock);
LASSERT(lsm->lsm_lock_owner == current_pid());
- CDEBUG(D_INODE, "MDT ID "DOSTID" initial value: s=%llu m=%llu a=%llu c=%llu b=%llu\n",
+ CDEBUG(D_INODE, "MDT ID " DOSTID " initial value: s=%llu m=%llu a=%llu c=%llu b=%llu\n",
POSTID(&lsm->lsm_oi), lvb->lvb_size, lvb->lvb_mtime,
lvb->lvb_atime, lvb->lvb_ctime, lvb->lvb_blocks);
for (i = 0; i < lsm->lsm_stripe_count; i++) {
if (loi->loi_lvb.lvb_ctime > current_ctime)
current_ctime = loi->loi_lvb.lvb_ctime;
- CDEBUG(D_INODE, "MDT ID "DOSTID" on OST[%u]: s=%llu m=%llu a=%llu c=%llu b=%llu\n",
+ CDEBUG(D_INODE, "MDT ID " DOSTID " on OST[%u]: s=%llu m=%llu a=%llu c=%llu b=%llu\n",
POSTID(&lsm->lsm_oi), loi->loi_ost_idx,
loi->loi_lvb.lvb_size, loi->loi_lvb.lvb_mtime,
loi->loi_lvb.lvb_atime, loi->loi_lvb.lvb_ctime,
subhdr = cl_object_header(stripe);
oinfo = lov->lo_lsm->lsm_oinfo[idx];
- CDEBUG(D_INODE, DFID"@%p[%d] -> "DFID"@%p: ostid: "DOSTID
- " idx: %d gen: %d\n",
+ CDEBUG(D_INODE, DFID "@%p[%d] -> " DFID "@%p: ostid: " DOSTID " idx: %d gen: %d\n",
PFID(&subhdr->coh_lu.loh_fid), subhdr, idx,
PFID(&hdr->coh_lu.loh_fid), hdr, POSTID(&oinfo->loi_oi),
oinfo->loi_ost_idx, oinfo->loi_ost_gen);
struct lov_layout_raid0 *r0;
struct lu_site *site;
struct lu_site_bkt_data *bkt;
- wait_queue_t *waiter;
+ wait_queue_entry_t *waiter;
r0 = &lov->u.raid0;
LASSERT(r0->lo_sub[idx] == los);
LASSERT(0 <= llt && llt < ARRAY_SIZE(lov_dispatch));
- CDEBUG(D_INODE, DFID" from %s to %s\n",
+ CDEBUG(D_INODE, DFID " from %s to %s\n",
PFID(lu_object_fid(lov2lu(lov))),
llt2str(lov->lo_type), llt2str(llt));
out:
lov_conf_unlock(lov);
lov_lsm_put(lsm);
- CDEBUG(D_INODE, DFID" lo_layout_invalid=%d\n",
+ CDEBUG(D_INODE, DFID " lo_layout_invalid=%d\n",
PFID(lu_object_fid(lov2lu(lov))), lov->lo_layout_invalid);
return result;
}
struct ost_id oi;
lmm_oi_le_to_cpu(&oi, &lmm->lmm_oi);
- CDEBUG(level, "objid "DOSTID", magic 0x%08x, pattern %#x\n",
+ CDEBUG(level, "objid " DOSTID ", magic 0x%08x, pattern %#x\n",
POSTID(&oi), le32_to_cpu(lmm->lmm_magic),
le32_to_cpu(lmm->lmm_pattern));
CDEBUG(level, "stripe_size %u, stripe_count %u, layout_gen %u\n",
struct ost_id oi;
ostid_le_to_cpu(&lod->l_ost_oi, &oi);
- CDEBUG(level, "stripe %u idx %u subobj "DOSTID"\n", i,
+ CDEBUG(level, "stripe %u idx %u subobj " DOSTID "\n", i,
le32_to_cpu(lod->l_ost_idx), POSTID(&oi));
}
}
void lov_dump_lmm_v3(int level, struct lov_mds_md_v3 *lmm)
{
lov_dump_lmm_common(level, lmm);
- CDEBUG(level, "pool_name "LOV_POOLNAMEF"\n", lmm->lmm_pool_name);
+ CDEBUG(level, "pool_name " LOV_POOLNAMEF "\n", lmm->lmm_pool_name);
lov_dump_lmm_objects(level, lmm->lmm_objects,
le16_to_cpu(lmm->lmm_stripe_count));
}
return rc;
}
-static struct file_operations pool_proc_operations = {
+static const struct file_operations pool_proc_operations = {
.open = pool_proc_open,
.read = seq_read,
.llseek = seq_lseek,
poolname, new_pool,
&pool_proc_operations);
if (IS_ERR_OR_NULL(new_pool->pool_debugfs_entry)) {
- CWARN("Cannot add debugfs pool entry "LOV_POOLNAMEF"\n",
+ CWARN("Cannot add debugfs pool entry " LOV_POOLNAMEF "\n",
poolname);
new_pool->pool_debugfs_entry = NULL;
lov_pool_putref(new_pool);
goto out_err;
}
- CDEBUG(D_CONFIG, LOV_POOLNAMEF" is pool #%d\n",
+ CDEBUG(D_CONFIG, LOV_POOLNAMEF " is pool #%d\n",
poolname, lov->lov_pool_count);
return 0;
if (rc)
goto out;
- CDEBUG(D_CONFIG, "Added %s to "LOV_POOLNAMEF" as member %d\n",
+ CDEBUG(D_CONFIG, "Added %s to " LOV_POOLNAMEF " as member %d\n",
ostname, poolname, pool_tgt_count(pool));
out:
lov_ost_pool_remove(&pool->pool_obds, lov_idx);
- CDEBUG(D_CONFIG, "%s removed from "LOV_POOLNAMEF"\n", ostname,
+ CDEBUG(D_CONFIG, "%s removed from " LOV_POOLNAMEF "\n", ostname,
poolname);
out:
ptlrpc_req_finished(req);
resends++;
- CDEBUG(D_HA, "%s: resend:%d op:%d "DFID"/"DFID"\n",
+ CDEBUG(D_HA, "%s: resend:%d op:%d " DFID "/" DFID "\n",
obddev->obd_name, resends, it->it_op,
PFID(&op_data->op_fid1), PFID(&op_data->op_fid2));
OBD_FAIL_TIMEOUT(OBD_FAIL_MDC_ENQUEUE_PAUSE, obd_timeout);
}
- if (it->it_op & IT_CREAT) {
+ if (it->it_op & IT_CREAT)
/* XXX this belongs in ll_create_it */
- } else if (it->it_op == IT_OPEN) {
+ ;
+ else if (it->it_op == IT_OPEN)
LASSERT(!it_disposition(it, DISP_OPEN_CREATE));
- } else {
+ else
LASSERT(it->it_op & (IT_GETATTR | IT_LOOKUP | IT_LAYOUT));
- }
/* If we already have a matching lock, then cancel the new
* one. We have to set the data here instead of in
LASSERTF(fid_res_name_eq(&mdt_body->mbo_fid1,
&lock->l_resource->lr_name),
- "Lock res_id: "DLDLMRES", fid: "DFID"\n",
+ "Lock res_id: " DLDLMRES ", fid: " DFID "\n",
PLDLMRES(lock->l_resource), PFID(&mdt_body->mbo_fid1));
LDLM_LOCK_PUT(lock);
LASSERT(it);
- CDEBUG(D_DLMTRACE, "(name: %.*s,"DFID") in obj "DFID
+ CDEBUG(D_DLMTRACE, "(name: %.*s," DFID ") in obj " DFID
", intent: %s flags %#Lo\n", (int)op_data->op_namelen,
op_data->op_name, PFID(&op_data->op_fid2),
PFID(&op_data->op_fid1), ldlm_it2str(it->it_op),
ptlrpc_req_finished(req);
resends++;
- CDEBUG(D_HA, "%s: resend:%d create on "DFID"/"DFID"\n",
+ CDEBUG(D_HA, "%s: resend:%d create on " DFID "/" DFID "\n",
exp->exp_obd->obd_name, resends,
PFID(&op_data->op_fid1), PFID(&op_data->op_fid2));
*rootfid = body->mbo_fid1;
CDEBUG(D_NET,
- "root fid="DFID", last_committed=%llu\n",
+ "root fid=" DFID ", last_committed=%llu\n",
PFID(rootfid),
lustre_msg_get_last_committed(req->rq_repmsg));
out:
/* allocate a FID for volatile file */
rc = mdc_fid_alloc(NULL, exp, &op_data->op_fid2, op_data);
if (rc < 0) {
- CERROR("%s: "DFID" failed to allocate FID: %d\n",
+ CERROR("%s: " DFID " failed to allocate FID: %d\n",
obd->obd_name, PFID(&op_data->op_fid1), rc);
/* save the errcode and proceed to close */
saved_rc = rc;
/*
* TODO: repeat close after errors
*/
- CWARN("%s: close of FID "DFID" failed, file reference will be dropped when this client unmounts or is evicted\n",
+ CWARN("%s: close of FID " DFID " failed, file reference will be dropped when this client unmounts or is evicted\n",
obd->obd_name, PFID(&op_data->op_fid1));
rc = -ENOMEM;
goto out;
ptlrpc_req_finished(enq_req);
if (rc < 0) {
- CERROR("%s: "DFID" lock enqueue fails: rc = %d\n",
+ CERROR("%s: " DFID " lock enqueue fails: rc = %d\n",
exp->exp_obd->obd_name, PFID(&op_data->op_fid1), rc);
return rc;
}
rp_param.rp_hash64),
mdc_read_page_remote, &rp_param);
if (IS_ERR(page)) {
- CERROR("%s: read cache page: "DFID" at %llu: rc %ld\n",
+ CERROR("%s: read cache page: " DFID " at %llu: rc %ld\n",
exp->exp_obd->obd_name, PFID(&op_data->op_fid1),
rp_param.rp_off, PTR_ERR(page));
rc = PTR_ERR(page);
wait_on_page_locked(page);
(void)kmap(page);
if (!PageUptodate(page)) {
- CERROR("%s: page not updated: "DFID" at %llu: rc %d\n",
+ CERROR("%s: page not updated: " DFID " at %llu: rc %d\n",
exp->exp_obd->obd_name, PFID(&op_data->op_fid1),
rp_param.rp_off, -5);
goto fail;
if (!PageChecked(page))
SetPageChecked(page);
if (PageError(page)) {
- CERROR("%s: page error: "DFID" at %llu: rc %d\n",
+ CERROR("%s: page error: " DFID " at %llu: rc %d\n",
exp->exp_obd->obd_name, PFID(&op_data->op_fid1),
rp_param.rp_off, -5);
goto fail;
memcpy(key, KEY_FID2PATH, sizeof(KEY_FID2PATH));
memcpy(key + cfs_size_round(sizeof(KEY_FID2PATH)), gf, sizeof(*gf));
- CDEBUG(D_IOCTL, "path get "DFID" from %llu #%d\n",
+ CDEBUG(D_IOCTL, "path get " DFID " from %llu #%d\n",
PFID(&gf->gf_fid), gf->gf_recno, gf->gf_linkno);
if (!fid_is_sane(&gf->gf_fid)) {
/* We've given up the lock, prepare ourselves to update. */
LDLM_DEBUG(lock, "MGC cancel CB");
- CDEBUG(D_MGC, "Lock res "DLDLMRES" (%.8s)\n",
+ CDEBUG(D_MGC, "Lock res " DLDLMRES " (%.8s)\n",
PLDLMRES(lock->l_resource),
(char *)&lock->l_resource->lr_name.name[0]);
const struct lu_fid *fid;
fid = lu_object_fid(&descr->cld_obj->co_lu);
- (*printer)(env, cookie, DDESCR"@"DFID, PDESCR(descr), PFID(fid));
+ (*printer)(env, cookie, DDESCR "@" DFID, PDESCR(descr), PFID(fid));
}
EXPORT_SYMBOL(cl_lock_descr_print);
hdr = cl_object_header(o);
- CDEBUG(D_PAGE, "%lu@"DFID" %p %lx %d\n",
+ CDEBUG(D_PAGE, "%lu@" DFID " %p %lx %d\n",
idx, PFID(&hdr->coh_lu.loh_fid), vmpage, vmpage->private, type);
/* fast path. */
if (type == CPT_CACHEABLE) {
if (ostid_id(&cgl->lgl_oi) == ostid_id(&logid->lgl_oi) &&
ostid_seq(&cgl->lgl_oi) == ostid_seq(&logid->lgl_oi)) {
if (cgl->lgl_ogen != logid->lgl_ogen) {
- CERROR("%s: log "DOSTID" generation %x != %x\n",
+ CERROR("%s: log " DOSTID " generation %x != %x\n",
loghandle->lgh_ctxt->loc_obd->obd_name,
POSTID(&logid->lgl_oi), cgl->lgl_ogen,
logid->lgl_ogen);
rc = llog_open(env, cathandle->lgh_ctxt, &loghandle, logid, NULL,
LLOG_OPEN_EXISTS);
if (rc < 0) {
- CERROR("%s: error opening log id "DOSTID":%x: rc = %d\n",
+ CERROR("%s: error opening log id " DOSTID ":%x: rc = %d\n",
cathandle->lgh_ctxt->loc_obd->obd_name,
POSTID(&logid->lgl_oi), logid->lgl_ogen, rc);
return rc;
CERROR("invalid record in catalog\n");
return -EINVAL;
}
- CDEBUG(D_HA, "processing log "DOSTID":%x at index %u of catalog "
- DOSTID"\n", POSTID(&lir->lid_id.lgl_oi), lir->lid_id.lgl_ogen,
+ CDEBUG(D_HA, "processing log " DOSTID ":%x at index %u of catalog "
+ DOSTID "\n", POSTID(&lir->lid_id.lgl_oi), lir->lid_id.lgl_ogen,
rec->lrh_index, POSTID(&cat_llh->lgh_id.lgl_oi));
rc = llog_cat_id2handle(env, cat_llh, &llh, &lir->lid_id);
if (rc) {
- CERROR("%s: cannot find handle for llog "DOSTID": %d\n",
+ CERROR("%s: cannot find handle for llog " DOSTID ": %d\n",
cat_llh->lgh_ctxt->loc_obd->obd_name,
POSTID(&lir->lid_id.lgl_oi), rc);
return rc;
if (llh->llh_cat_idx > cat_llh->lgh_last_idx) {
struct llog_process_cat_data cd;
- CWARN("catlog "DOSTID" crosses index zero\n",
+ CWARN("catlog " DOSTID " crosses index zero\n",
POSTID(&cat_llh->lgh_id.lgl_oi));
cd.lpcd_first_idx = llh->llh_cat_idx;
static void print_llogd_body(struct llogd_body *d)
{
CDEBUG(D_OTHER, "llogd body: %p\n", d);
- CDEBUG(D_OTHER, "\tlgd_logid.lgl_oi: "DOSTID"\n",
+ CDEBUG(D_OTHER, "\tlgd_logid.lgl_oi: " DOSTID "\n",
POSTID(&d->lgd_logid.lgl_oi));
CDEBUG(D_OTHER, "\tlgd_logid.lgl_ogen: %#x\n", d->lgd_logid.lgl_ogen);
CDEBUG(D_OTHER, "\tlgd_ctxt_idx: %#x\n", d->lgd_ctxt_idx);
struct dentry *ldebugfs_add_simple(struct dentry *root,
char *name, void *data,
- struct file_operations *fops)
+ const struct file_operations *fops)
{
struct dentry *entry;
umode_t mode = 0;
EXPORT_SYMBOL_GPL(ldebugfs_register);
/* Generic callbacks */
-int lprocfs_rd_uint(struct seq_file *m, void *data)
-{
- seq_printf(m, "%u\n", *(unsigned int *)data);
- return 0;
-}
-EXPORT_SYMBOL(lprocfs_rd_uint);
-
-int lprocfs_wr_uint(struct file *file, const char __user *buffer,
- unsigned long count, void *data)
-{
- unsigned *p = data;
- char dummy[MAX_STRING_SIZE + 1], *end;
- unsigned long tmp;
-
- if (count >= sizeof(dummy))
- return -EINVAL;
-
- if (count == 0)
- return 0;
-
- if (copy_from_user(dummy, buffer, count))
- return -EFAULT;
-
- dummy[count] = '\0';
-
- tmp = simple_strtoul(dummy, &end, 0);
- if (dummy == end)
- return -EINVAL;
-
- *p = (unsigned int)tmp;
- return count;
-}
-EXPORT_SYMBOL(lprocfs_wr_uint);
-
static ssize_t uuid_show(struct kobject *kobj, struct attribute *attr,
char *buf)
{
bool first = true;
if (imp->imp_obd->obd_no_recov) {
- seq_printf(m, "no_recov");
+ seq_puts(m, "no_recov");
first = false;
}
imp->imp_connect_data.ocd_instance);
obd_connect_seq_flags2str(m, imp->imp_connect_data.ocd_connect_flags,
", ");
- seq_printf(m, " ]\n");
+ seq_puts(m, " ]\n");
obd_connect_data_seqprint(m, ocd);
- seq_printf(m, " import_flags: [ ");
+ seq_puts(m, " import_flags: [ ");
obd_import_flags2str(imp, m);
- seq_printf(m,
- " ]\n"
- " connection:\n"
- " failover_nids: [ ");
+ seq_puts(m,
+ " ]\n"
+ " connection:\n"
+ " failover_nids: [ ");
spin_lock(&imp->imp_lock);
j = 0;
list_for_each_entry(conn, &imp->imp_conn_list, oic_item) {
seq_printf(m, "current_state: %s\n",
ptlrpc_import_state_name(imp->imp_state));
- seq_printf(m, "state_history:\n");
+ seq_puts(m, "state_history:\n");
k = imp->imp_state_hist_idx;
for (j = 0; j < IMP_STATE_HIST_LEN; j++) {
struct import_state_hist *ish =
for (i = 0; i < AT_BINS; i++)
seq_printf(m, "%3u ", at->at_hist[i]);
- seq_printf(m, "\n");
+ seq_puts(m, "\n");
return 0;
}
EXPORT_SYMBOL(lprocfs_at_hist_helper);
flags = obd->u.cli.cl_import->imp_connect_data.ocd_connect_flags;
seq_printf(m, "flags=%#llx\n", flags);
obd_connect_seq_flags2str(m, flags, "\n");
- seq_printf(m, "\n");
+ seq_puts(m, "\n");
up_read(&obd->u.cli.cl_sem);
return 0;
}
lu_printer_t printer,
const struct lu_object_header *hdr)
{
- (*printer)(env, cookie, "header@%p[%#lx, %d, "DFID"%s%s%s]",
+ (*printer)(env, cookie, "header@%p[%#lx, %d, " DFID "%s%s%s]",
hdr, hdr->loh_flags, atomic_read(&hdr->loh_ref),
PFID(&hdr->loh_fid),
hlist_unhashed(&hdr->loh_hash) ? "" : " hash",
static struct lu_object *htable_lookup(struct lu_site *s,
struct cfs_hash_bd *bd,
const struct lu_fid *f,
- wait_queue_t *waiter,
+ wait_queue_entry_t *waiter,
__u64 *version)
{
struct lu_site_bkt_data *bkt;
struct lu_device *dev,
const struct lu_fid *f,
const struct lu_object_conf *conf,
- wait_queue_t *waiter)
+ wait_queue_entry_t *waiter)
{
struct lu_object *o;
struct lu_object *shadow;
{
struct lu_site_bkt_data *bkt;
struct lu_object *obj;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
while (1) {
obj = lu_object_find_try(env, dev, f, conf, &wait);
cache_size = cache_size / 100 * lu_cache_percent *
(PAGE_SIZE / 1024);
- for (bits = 1; (1 << bits) < cache_size; ++bits) {
+ for (bits = 1; (1 << bits) < cache_size; ++bits)
;
- }
return clamp_t(typeof(bits), bits, LU_SITE_BITS_MIN, bits_max);
}
{
struct echo_page *ep = cl2echo_page(slice);
- (*printer)(env, cookie, LUSTRE_ECHO_CLIENT_NAME"-page@%p %d vm@%p\n",
+ (*printer)(env, cookie, LUSTRE_ECHO_CLIENT_NAME "-page@%p %d vm@%p\n",
ep, mutex_is_locked(&ep->ep_lock),
slice->cpl_page->cp_vmpage);
return 0;
}
cl_echo_object_put(eco);
- CDEBUG(D_INFO, "oa oid "DOSTID"\n", POSTID(&oa->o_oi));
+ CDEBUG(D_INFO, "oa oid " DOSTID "\n", POSTID(&oa->o_oi));
failed:
if (created && rc)
struct lustre_handle conn = { 0 };
rc = class_connect(&conn, src, cluuid);
- if (rc == 0) {
+ if (rc == 0)
*exp = class_conn2export(&conn);
- }
return rc;
}
else if (ext->oe_start > tmp->oe_end)
n = &(*n)->rb_right;
else
- EASSERTF(0, tmp, EXTSTR"\n", EXTPARA(ext));
+ EASSERTF(0, tmp, EXTSTR "\n", EXTPARA(ext));
}
rb_link_node(&ext->oe_node, parent, n);
rb_insert_color(&ext->oe_node, &obj->oo_root);
/* grants has been allocated by caller */
LASSERTF(*grants >= chunksize + cli->cl_extent_tax,
"%u/%u/%u.\n", *grants, chunksize, cli->cl_extent_tax);
- LASSERTF((max_end - cur->oe_start) < max_pages, EXTSTR"\n",
+ LASSERTF((max_end - cur->oe_start) < max_pages, EXTSTR "\n",
EXTPARA(cur));
restart:
/* if covering by different locks, no chance to match */
if (olck->ols_dlmlock != ext->oe_dlmlock) {
EASSERTF(!overlapped(ext, cur), ext,
- EXTSTR"\n", EXTPARA(cur));
+ EXTSTR "\n", EXTPARA(cur));
ext = next_extent(ext);
continue;
*/
EASSERTF((ext->oe_start <= cur->oe_start &&
ext->oe_end >= cur->oe_end),
- ext, EXTSTR"\n", EXTPARA(cur));
+ ext, EXTSTR "\n", EXTPARA(cur));
if (ext->oe_state > OES_CACHE || ext->oe_fsync_wait) {
/* for simplicity, we wait for this extent to
struct brw_page *pga)
{
assert_spin_locked(&cli->cl_loi_list_lock);
- if (!(pga->flag & OBD_BRW_FROM_GRANT)) {
+ if (!(pga->flag & OBD_BRW_FROM_GRANT))
return;
- }
pga->flag &= ~OBD_BRW_FROM_GRANT;
atomic_long_dec(&obd_dirty_pages);
unsigned int erd_max_chunks;
};
-static inline unsigned osc_extent_chunks(const struct osc_extent *ext)
+static inline unsigned int osc_extent_chunks(const struct osc_extent *ext)
{
struct client_obd *cli = osc_cli(ext->oe_obj);
- unsigned ppc_bits = cli->cl_chunkbits - PAGE_SHIFT;
+ unsigned int ppc_bits = cli->cl_chunkbits - PAGE_SHIFT;
return (ext->oe_end >> ppc_bits) - (ext->oe_start >> ppc_bits) + 1;
}
return 1;
}
-static inline unsigned osc_max_write_chunks(const struct client_obd *cli)
+static inline unsigned int osc_max_write_chunks(const struct client_obd *cli)
{
/*
* LU-8135:
struct list_head oap_rpc_item;
u64 oap_obj_off;
- unsigned oap_page_off;
+ unsigned int oap_page_off;
enum async_flags oap_async_flags;
struct brw_page oap_brw_page;
msg = "changed in transit AND doesn't match the original - likely false positive due to mmap IO (bug 11742)"
;
- LCONSOLE_ERROR_MSG(0x132, "BAD WRITE CHECKSUM: %s: from %s inode "DFID
- " object "DOSTID" extent [%llu-%llu]\n",
+ LCONSOLE_ERROR_MSG(0x132, "BAD WRITE CHECKSUM: %s: from %s inode " DFID " object " DOSTID " extent [%llu-%llu]\n",
msg, libcfs_nid2str(peer->nid),
oa->o_valid & OBD_MD_FLFID ? oa->o_parent_seq : (__u64)0,
oa->o_valid & OBD_MD_FLFID ? oa->o_parent_oid : 0,
*/
CDEBUG((lustre_msg_get_flags(req->rq_reqmsg) & MSG_RESENT) ?
D_ADAPTTO : D_WARNING,
- "Reported service time %u > total measured time "
- CFS_DURATION_T"\n", service_time,
+ "Reported service time %u > total measured time " CFS_DURATION_T "\n",
+ service_time,
(long)(now - req->rq_sent));
return;
}
}
/* Sanity checks for a reconnected import. */
- if (!(imp->imp_replayable) != !(msg_flags & MSG_CONNECT_REPLAYABLE)) {
+ if (!(imp->imp_replayable) != !(msg_flags & MSG_CONNECT_REPLAYABLE))
CERROR("imp_replayable flag does not match server after reconnect. We should LBUG right here.\n");
- }
if (lustre_msg_get_last_committed(request->rq_repmsg) > 0 &&
lustre_msg_get_last_committed(request->rq_repmsg) <
- aa->pcaa_peer_committed) {
+ aa->pcaa_peer_committed)
CERROR("%s went back in time (transno %lld was previously committed, server now claims %lld)! See https://bugzilla.lustre.org/show_bug.cgi?id=9646\n",
obd2cli_tgt(imp->imp_obd), aa->pcaa_peer_committed,
lustre_msg_get_last_committed(request->rq_repmsg));
- }
finish:
ptlrpc_prepare_replay(imp);
rc = 0;
}
- if (imp->imp_state == LUSTRE_IMP_REPLAY_LOCKS) {
+ if (imp->imp_state == LUSTRE_IMP_REPLAY_LOCKS)
if (atomic_read(&imp->imp_replay_inflight) == 0) {
IMPORT_SET_STATE(imp, LUSTRE_IMP_REPLAY_WAIT);
rc = signal_completed_replay(imp);
if (rc)
goto out;
}
- }
- if (imp->imp_state == LUSTRE_IMP_REPLAY_WAIT) {
- if (atomic_read(&imp->imp_replay_inflight) == 0) {
+ if (imp->imp_state == LUSTRE_IMP_REPLAY_WAIT)
+ if (atomic_read(&imp->imp_replay_inflight) == 0)
IMPORT_SET_STATE(imp, LUSTRE_IMP_RECOVER);
- }
- }
if (imp->imp_state == LUSTRE_IMP_RECOVER) {
CDEBUG(D_HA, "reconnected to %s@%s\n",
field->rmf_name, offset, loc);
offset--;
- LASSERT(0 <= offset && offset < REQ_MAX_FIELD_NR);
+ LASSERT(offset < REQ_MAX_FIELD_NR);
return offset;
}
CDEBUG(D_RPCTRACE, "obdo: o_valid = %08x\n", valid);
if (valid & OBD_MD_FLID)
- CDEBUG(D_RPCTRACE, "obdo: id = "DOSTID"\n", POSTID(&oa->o_oi));
+ CDEBUG(D_RPCTRACE, "obdo: id = " DOSTID "\n", POSTID(&oa->o_oi));
if (valid & OBD_MD_FLFID)
CDEBUG(D_RPCTRACE, "obdo: o_parent_seq = %#llx\n",
oa->o_parent_seq);
void ptlrpc_set_add_new_req(struct ptlrpcd_ctl *pc,
struct ptlrpc_request *req);
int ptlrpc_expired_set(void *data);
-int ptlrpc_set_next_timeout(struct ptlrpc_request_set *);
+int ptlrpc_set_next_timeout(struct ptlrpc_request_set *set);
void ptlrpc_resend_req(struct ptlrpc_request *request);
void ptlrpc_set_bulk_mbits(struct ptlrpc_request *req);
void ptlrpc_assign_next_xid_nolock(struct ptlrpc_request *req);
int ptlrpc_init_portals(void);
void ptlrpc_exit_portals(void);
-void ptlrpc_request_handle_notconn(struct ptlrpc_request *);
+void ptlrpc_request_handle_notconn(struct ptlrpc_request *req);
void lustre_assert_wire_constants(void);
int ptlrpc_import_in_recovery(struct obd_import *imp);
int ptlrpc_set_import_discon(struct obd_import *imp, __u32 conn_cnt);
/* req_in handling should/must be fast */
if (ktime_get_real_seconds() - req->rq_arrival_time.tv_sec > 5)
- DEBUG_REQ(D_WARNING, req, "Slow req_in handling "CFS_DURATION_T"s",
+ DEBUG_REQ(D_WARNING, req, "Slow req_in handling " CFS_DURATION_T "s",
(long)(ktime_get_real_seconds() -
req->rq_arrival_time.tv_sec));
#include <linux/wait.h>
#include <linux/kobject.h>
#include "mostcore.h"
-#include "networking.h"
#define MEP_HDR_LEN 8
#define MDP_HDR_LEN 16
struct net_dev_context {
struct most_interface *iface;
- bool channels_opened;
bool is_mamac;
struct net_device *dev;
struct net_dev_channel rx;
struct net_dev_channel tx;
- struct completion mac_compl;
struct list_head list;
};
static struct list_head net_devices = LIST_HEAD_INIT(net_devices);
-static struct spinlock list_lock;
+static struct mutex probe_disc_mt; /* ch->linked = true, most_nd_open */
+static struct spinlock list_lock; /* list_head, ch->linked = false, dev_hold */
static struct most_aim aim;
static int skb_to_mamac(const struct sk_buff *skb, struct mbo *mbo)
static int most_nd_set_mac_address(struct net_device *dev, void *p)
{
- struct net_dev_context *nd = dev->ml_priv;
+ struct net_dev_context *nd = netdev_priv(dev);
int err = eth_mac_addr(dev, p);
if (err)
return err;
- BUG_ON(nd->dev != dev);
-
nd->is_mamac =
(dev->dev_addr[0] == 0 && dev->dev_addr[1] == 0 &&
dev->dev_addr[2] == 0 && dev->dev_addr[3] == 0);
return 0;
}
+static void on_netinfo(struct most_interface *iface,
+ unsigned char link_stat, unsigned char *mac_addr);
+
static int most_nd_open(struct net_device *dev)
{
- struct net_dev_context *nd = dev->ml_priv;
- long ret;
-
- netdev_info(dev, "open net device\n");
-
- BUG_ON(nd->dev != dev);
+ struct net_dev_context *nd = netdev_priv(dev);
+ int ret = 0;
- if (nd->channels_opened)
- return -EFAULT;
-
- BUG_ON(!nd->tx.linked || !nd->rx.linked);
+ mutex_lock(&probe_disc_mt);
if (most_start_channel(nd->iface, nd->rx.ch_id, &aim)) {
netdev_err(dev, "most_start_channel() failed\n");
- return -EBUSY;
+ ret = -EBUSY;
+ goto unlock;
}
if (most_start_channel(nd->iface, nd->tx.ch_id, &aim)) {
netdev_err(dev, "most_start_channel() failed\n");
most_stop_channel(nd->iface, nd->rx.ch_id, &aim);
- return -EBUSY;
+ ret = -EBUSY;
+ goto unlock;
}
- if (!is_valid_ether_addr(dev->dev_addr)) {
- nd->iface->request_netinfo(nd->iface, nd->tx.ch_id);
- ret = wait_for_completion_interruptible_timeout(
- &nd->mac_compl, msecs_to_jiffies(5000));
- if (!ret) {
- netdev_err(dev, "mac timeout\n");
- ret = -EBUSY;
- goto err;
- }
-
- if (ret < 0) {
- netdev_warn(dev, "mac waiting interrupted\n");
- goto err;
- }
- }
-
- nd->channels_opened = true;
+ netif_carrier_off(dev);
+ if (is_valid_ether_addr(dev->dev_addr))
+ netif_dormant_off(dev);
+ else
+ netif_dormant_on(dev);
netif_wake_queue(dev);
- return 0;
+ if (nd->iface->request_netinfo)
+ nd->iface->request_netinfo(nd->iface, nd->tx.ch_id, on_netinfo);
-err:
- most_stop_channel(nd->iface, nd->tx.ch_id, &aim);
- most_stop_channel(nd->iface, nd->rx.ch_id, &aim);
+unlock:
+ mutex_unlock(&probe_disc_mt);
return ret;
}
static int most_nd_stop(struct net_device *dev)
{
- struct net_dev_context *nd = dev->ml_priv;
-
- netdev_info(dev, "stop net device\n");
+ struct net_dev_context *nd = netdev_priv(dev);
- BUG_ON(nd->dev != dev);
netif_stop_queue(dev);
-
- if (nd->channels_opened) {
- most_stop_channel(nd->iface, nd->rx.ch_id, &aim);
- most_stop_channel(nd->iface, nd->tx.ch_id, &aim);
- nd->channels_opened = false;
- }
+ if (nd->iface->request_netinfo)
+ nd->iface->request_netinfo(nd->iface, nd->tx.ch_id, NULL);
+ most_stop_channel(nd->iface, nd->rx.ch_id, &aim);
+ most_stop_channel(nd->iface, nd->tx.ch_id, &aim);
return 0;
}
static netdev_tx_t most_nd_start_xmit(struct sk_buff *skb,
struct net_device *dev)
{
- struct net_dev_context *nd = dev->ml_priv;
+ struct net_dev_context *nd = netdev_priv(dev);
struct mbo *mbo;
int ret;
- BUG_ON(nd->dev != dev);
-
mbo = most_get_mbo(nd->iface, nd->tx.ch_id, &aim);
if (!mbo) {
dev->netdev_ops = &most_nd_ops;
}
-static void most_net_rm_netdev_safe(struct net_dev_context *nd)
+static struct net_dev_context *get_net_dev(struct most_interface *iface)
{
- if (!nd->dev)
- return;
-
- pr_info("remove net device %p\n", nd->dev);
+ struct net_dev_context *nd;
- unregister_netdev(nd->dev);
- free_netdev(nd->dev);
- nd->dev = NULL;
+ list_for_each_entry(nd, &net_devices, list)
+ if (nd->iface == iface)
+ return nd;
+ return NULL;
}
-static struct net_dev_context *get_net_dev_context(
- struct most_interface *iface)
+static struct net_dev_context *get_net_dev_hold(struct most_interface *iface)
{
- struct net_dev_context *nd, *tmp;
+ struct net_dev_context *nd;
unsigned long flags;
spin_lock_irqsave(&list_lock, flags);
- list_for_each_entry_safe(nd, tmp, &net_devices, list) {
- if (nd->iface == iface) {
- spin_unlock_irqrestore(&list_lock, flags);
- return nd;
- }
- }
+ nd = get_net_dev(iface);
+ if (nd && nd->rx.linked && nd->tx.linked)
+ dev_hold(nd->dev);
+ else
+ nd = NULL;
spin_unlock_irqrestore(&list_lock, flags);
- return NULL;
+ return nd;
}
static int aim_probe_channel(struct most_interface *iface, int channel_idx,
{
struct net_dev_context *nd;
struct net_dev_channel *ch;
+ struct net_device *dev;
unsigned long flags;
+ int ret = 0;
if (!iface)
return -EINVAL;
if (ccfg->data_type != MOST_CH_ASYNC)
return -EINVAL;
- nd = get_net_dev_context(iface);
-
+ mutex_lock(&probe_disc_mt);
+ nd = get_net_dev(iface);
if (!nd) {
- nd = kzalloc(sizeof(*nd), GFP_KERNEL);
- if (!nd)
- return -ENOMEM;
+ dev = alloc_netdev(sizeof(struct net_dev_context), "meth%d",
+ NET_NAME_UNKNOWN, most_nd_setup);
+ if (!dev) {
+ ret = -ENOMEM;
+ goto unlock;
+ }
- init_completion(&nd->mac_compl);
+ nd = netdev_priv(dev);
nd->iface = iface;
+ nd->dev = dev;
spin_lock_irqsave(&list_lock, flags);
list_add(&nd->list, &net_devices);
spin_unlock_irqrestore(&list_lock, flags);
- }
- ch = ccfg->direction == MOST_CH_TX ? &nd->tx : &nd->rx;
- if (ch->linked) {
- pr_err("only one channel per instance & direction allowed\n");
- return -EINVAL;
- }
-
- if (nd->tx.linked || nd->rx.linked) {
- struct net_device *dev =
- alloc_netdev(0, "meth%d", NET_NAME_UNKNOWN,
- most_nd_setup);
-
- if (!dev) {
- pr_err("no memory for net_device\n");
- return -ENOMEM;
+ ch = ccfg->direction == MOST_CH_TX ? &nd->tx : &nd->rx;
+ } else {
+ ch = ccfg->direction == MOST_CH_TX ? &nd->tx : &nd->rx;
+ if (ch->linked) {
+ pr_err("direction is allocated\n");
+ ret = -EINVAL;
+ goto unlock;
}
- nd->dev = dev;
- ch->ch_id = channel_idx;
- ch->linked = true;
-
- dev->ml_priv = nd;
- if (register_netdev(dev)) {
- pr_err("registering net device failed\n");
- ch->linked = false;
- free_netdev(dev);
- return -EINVAL;
+ if (register_netdev(nd->dev)) {
+ pr_err("register_netdev() failed\n");
+ ret = -EINVAL;
+ goto unlock;
}
}
-
ch->ch_id = channel_idx;
ch->linked = true;
- return 0;
+unlock:
+ mutex_unlock(&probe_disc_mt);
+ return ret;
}
static int aim_disconnect_channel(struct most_interface *iface,
struct net_dev_context *nd;
struct net_dev_channel *ch;
unsigned long flags;
+ int ret = 0;
- nd = get_net_dev_context(iface);
- if (!nd)
- return -EINVAL;
+ mutex_lock(&probe_disc_mt);
+ nd = get_net_dev(iface);
+ if (!nd) {
+ ret = -EINVAL;
+ goto unlock;
+ }
- if (nd->rx.linked && channel_idx == nd->rx.ch_id)
+ if (nd->rx.linked && channel_idx == nd->rx.ch_id) {
ch = &nd->rx;
- else if (nd->tx.linked && channel_idx == nd->tx.ch_id)
+ } else if (nd->tx.linked && channel_idx == nd->tx.ch_id) {
ch = &nd->tx;
- else
- return -EINVAL;
-
- ch->linked = false;
+ } else {
+ ret = -EINVAL;
+ goto unlock;
+ }
- /*
- * do not call most_stop_channel() here, because channels are
- * going to be closed in ndo_stop() after unregister_netdev()
- */
- most_net_rm_netdev_safe(nd);
+ if (nd->rx.linked && nd->tx.linked) {
+ spin_lock_irqsave(&list_lock, flags);
+ ch->linked = false;
+ spin_unlock_irqrestore(&list_lock, flags);
- if (!nd->rx.linked && !nd->tx.linked) {
+ /*
+ * do not call most_stop_channel() here, because channels are
+ * going to be closed in ndo_stop() after unregister_netdev()
+ */
+ unregister_netdev(nd->dev);
+ } else {
spin_lock_irqsave(&list_lock, flags);
list_del(&nd->list);
spin_unlock_irqrestore(&list_lock, flags);
- kfree(nd);
+
+ free_netdev(nd->dev);
}
- return 0;
+unlock:
+ mutex_unlock(&probe_disc_mt);
+ return ret;
}
static int aim_resume_tx_channel(struct most_interface *iface,
{
struct net_dev_context *nd;
- nd = get_net_dev_context(iface);
- if (!nd || !nd->channels_opened || nd->tx.ch_id != channel_idx)
+ nd = get_net_dev_hold(iface);
+ if (!nd)
return 0;
- if (!nd->dev)
- return 0;
+ if (nd->tx.ch_id != channel_idx)
+ goto put_nd;
netif_wake_queue(nd->dev);
+
+put_nd:
+ dev_put(nd->dev);
return 0;
}
struct sk_buff *skb;
struct net_device *dev;
unsigned int skb_len;
+ int ret = 0;
- nd = get_net_dev_context(mbo->ifp);
- if (!nd || !nd->channels_opened || nd->rx.ch_id != mbo->hdm_channel_id)
+ nd = get_net_dev_hold(mbo->ifp);
+ if (!nd)
return -EIO;
- dev = nd->dev;
- if (!dev) {
- pr_err_once("drop packet: missing net_device\n");
- return -EIO;
+ if (nd->rx.ch_id != mbo->hdm_channel_id) {
+ ret = -EIO;
+ goto put_nd;
}
+ dev = nd->dev;
+
if (nd->is_mamac) {
- if (!PMS_IS_MAMAC(buf, len))
- return -EIO;
+ if (!PMS_IS_MAMAC(buf, len)) {
+ ret = -EIO;
+ goto put_nd;
+ }
skb = dev_alloc_skb(len - MDP_HDR_LEN + 2 * ETH_ALEN + 2);
} else {
- if (!PMS_IS_MEP(buf, len))
- return -EIO;
+ if (!PMS_IS_MEP(buf, len)) {
+ ret = -EIO;
+ goto put_nd;
+ }
skb = dev_alloc_skb(len - MEP_HDR_LEN);
}
out:
most_put_mbo(mbo);
- return 0;
+
+put_nd:
+ dev_put(nd->dev);
+ return ret;
}
static struct most_aim aim = {
static int __init most_net_init(void)
{
- pr_info("most_net_init()\n");
spin_lock_init(&list_lock);
+ mutex_init(&probe_disc_mt);
return most_register_aim(&aim);
}
static void __exit most_net_exit(void)
{
- struct net_dev_context *nd, *tmp;
- unsigned long flags;
-
- spin_lock_irqsave(&list_lock, flags);
- list_for_each_entry_safe(nd, tmp, &net_devices, list) {
- list_del(&nd->list);
- spin_unlock_irqrestore(&list_lock, flags);
- /*
- * do not call most_stop_channel() here, because channels are
- * going to be closed in ndo_stop() after unregister_netdev()
- */
- most_net_rm_netdev_safe(nd);
- kfree(nd);
- spin_lock_irqsave(&list_lock, flags);
- }
- spin_unlock_irqrestore(&list_lock, flags);
-
most_deregister_aim(&aim);
- pr_info("most_net_exit()\n");
}
/**
- * most_deliver_netinfo - callback for HDM to be informed about HW's MAC
+ * on_netinfo - callback for HDM to be informed about HW's MAC
* @param iface - most interface instance
* @param link_stat - link status
* @param mac_addr - MAC address
*/
-void most_deliver_netinfo(struct most_interface *iface,
- unsigned char link_stat, unsigned char *mac_addr)
+static void on_netinfo(struct most_interface *iface,
+ unsigned char link_stat, unsigned char *mac_addr)
{
struct net_dev_context *nd;
struct net_device *dev;
const u8 *m = mac_addr;
- nd = get_net_dev_context(iface);
+ nd = get_net_dev_hold(iface);
if (!nd)
return;
dev = nd->dev;
- if (!dev)
- return;
+
+ if (link_stat)
+ netif_carrier_on(dev);
+ else
+ netif_carrier_off(dev);
if (m && is_valid_ether_addr(m)) {
if (!is_valid_ether_addr(dev->dev_addr)) {
netdev_info(dev, "set mac %02x-%02x-%02x-%02x-%02x-%02x\n",
m[0], m[1], m[2], m[3], m[4], m[5]);
ether_addr_copy(dev->dev_addr, m);
- complete(&nd->mac_compl);
+ netif_dormant_off(dev);
} else if (!ether_addr_equal(dev->dev_addr, m)) {
netdev_warn(dev, "reject mac %02x-%02x-%02x-%02x-%02x-%02x\n",
m[0], m[1], m[2], m[3], m[4], m[5]);
}
}
+
+ dev_put(nd->dev);
}
-EXPORT_SYMBOL(most_deliver_netinfo);
module_init(most_net_init);
module_exit(most_net_exit);
+++ /dev/null
-/*
- * Networking AIM - Networking Application Interface Module for MostCore
- *
- * Copyright (C) 2015, Microchip Technology Germany II GmbH & Co. KG
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * This file is licensed under GPLv2.
- */
-#ifndef _NETWORKING_H_
-#define _NETWORKING_H_
-
-#include "mostcore.h"
-
-void most_deliver_netinfo(struct most_interface *iface,
- unsigned char link_stat, unsigned char *mac_addr);
-
-#endif
config HDM_DIM2
tristate "DIM2 HDM"
- depends on AIM_NETWORK
depends on HAS_IOMEM
---help---
}
static void dim2_configure_cat(u8 cat_base, u8 ch_addr, u8 ch_type,
- bool read_not_write, bool sync_mfe)
+ bool read_not_write)
{
+ bool isoc_fce = ch_type == CAT_CT_VAL_ISOC;
+ bool sync_mfe = ch_type == CAT_CT_VAL_SYNC;
u16 const cat =
(read_not_write << CAT_RNW_BIT) |
(ch_type << CAT_CT_SHIFT) |
(ch_addr << CAT_CL_SHIFT) |
+ (isoc_fce << CAT_FCE_BIT) |
(sync_mfe << CAT_MFE_BIT) |
(false << CAT_MT_BIT) |
(true << CAT_CE_BIT);
static void dim2_configure_channel(
u8 ch_addr, u8 type, u8 is_tx, u16 dbr_address, u16 hw_buffer_size,
- u16 packet_length, bool sync_mfe)
+ u16 packet_length)
{
dim2_configure_cdt(ch_addr, dbr_address, hw_buffer_size, packet_length);
- dim2_configure_cat(MLB_CAT, ch_addr, type, is_tx ? 1 : 0, sync_mfe);
+ dim2_configure_cat(MLB_CAT, ch_addr, type, is_tx ? 1 : 0);
dim2_configure_adt(ch_addr);
- dim2_configure_cat(AHB_CAT, ch_addr, type, is_tx ? 0 : 1, sync_mfe);
+ dim2_configure_cat(AHB_CAT, ch_addr, type, is_tx ? 0 : 1);
/* unmask interrupt for used channel, enable mlb_sys_int[0] interrupt */
dimcb_io_write(&g.dim2->ACMR0,
channel_init(ch, ch_address / 2);
dim2_configure_channel(ch->addr, type, is_tx,
- ch->dbr_addr, ch->dbr_size, 0, false);
+ ch->dbr_addr, ch->dbr_size, 0);
return DIM_NO_ERROR;
}
isoc_init(ch, ch_address / 2, packet_length);
dim2_configure_channel(ch->addr, CAT_CT_VAL_ISOC, is_tx, ch->dbr_addr,
- ch->dbr_size, packet_length, false);
+ ch->dbr_size, packet_length);
return DIM_NO_ERROR;
}
dim2_clear_dbr(ch->dbr_addr, ch->dbr_size);
dim2_configure_channel(ch->addr, CAT_CT_VAL_SYNC, is_tx,
- ch->dbr_addr, ch->dbr_size, 0, true);
+ ch->dbr_addr, ch->dbr_size, 0);
return DIM_NO_ERROR;
}
#include <linux/kthread.h>
#include <mostcore.h>
-#include <networking.h>
#include "dim2_hal.h"
#include "dim2_hdm.h"
#include "dim2_errors.h"
unsigned char link_state;
int atx_idx;
struct medialb_bus bus;
+ void (*on_netinfo)(struct most_interface *,
+ unsigned char, unsigned char *);
};
#define iface_to_hdm(iface) container_of(iface, struct dim2_hdm, most_iface)
if (dev->deliver_netinfo) {
dev->deliver_netinfo--;
- most_deliver_netinfo(&dev->most_iface, dev->link_state,
- dev->mac_addrs);
+ if (dev->on_netinfo) {
+ dev->on_netinfo(&dev->most_iface,
+ dev->link_state,
+ dev->mac_addrs);
+ }
}
}
* Send a command to INIC which triggers retrieving of network info by means of
* "Message exchange over MDP/MEP". Return 0 on success, negative on failure.
*/
-static void request_netinfo(struct most_interface *most_iface, int ch_idx)
+static void request_netinfo(struct most_interface *most_iface, int ch_idx,
+ void (*on_netinfo)(struct most_interface *,
+ unsigned char, unsigned char *))
{
struct dim2_hdm *dev = iface_to_hdm(most_iface);
struct mbo *mbo;
u8 *data;
+ dev->on_netinfo = on_netinfo;
+ if (!on_netinfo)
+ return;
+
if (dev->atx_idx < 0) {
pr_err("Async Tx Not initialized\n");
return;
ADT1_CTRL_ASYNC_BD_MASK = DIM2_MASK(11),
ADT1_ISOC_SYNC_BD_MASK = DIM2_MASK(13),
+ CAT_FCE_BIT = 14,
CAT_MFE_BIT = 14,
CAT_MT_BIT = 13,
return 0;
}
-static void request_netinfo(struct most_interface *most_iface,
- int ch_idx)
-{
- pr_info("request_netinfo()\n");
-}
-
static void do_rx_work(struct hdm_i2c *dev)
{
struct mbo *mbo;
dev->most_iface.configure = configure_channel;
dev->most_iface.enqueue = enqueue;
dev->most_iface.poison_channel = poison_channel;
- dev->most_iface.request_netinfo = request_netinfo;
INIT_LIST_HEAD(&dev->rx.list);
mutex_init(&dev->rx.list_mutex);
config HDM_USB
tristate "USB HDM"
depends on USB && NET
- select AIM_NETWORK
+
---help---
Say Y here if you want to connect via USB to network tranceiver.
This device driver depends on the networking AIM.
#include <linux/etherdevice.h>
#include <linux/uaccess.h>
#include "mostcore.h"
-#include "networking.h"
#define USB_MTU 512
#define NO_ISOCHRONOUS_URB 0
struct mutex io_mutex;
struct timer_list link_stat_timer;
struct work_struct poll_work_obj;
+ void (*on_netinfo)(struct most_interface *, unsigned char,
+ unsigned char *);
};
#define to_mdev(d) container_of(d, struct most_dev, iface)
* polls for the NI state of the INIC every 2 seconds.
*
*/
-static void hdm_request_netinfo(struct most_interface *iface, int channel)
+static void hdm_request_netinfo(struct most_interface *iface, int channel,
+ void (*on_netinfo)(struct most_interface *,
+ unsigned char,
+ unsigned char *))
{
struct most_dev *mdev;
BUG_ON(!iface);
mdev = to_mdev(iface);
+ mdev->on_netinfo = on_netinfo;
+ if (!on_netinfo)
+ return;
+
mdev->link_stat_timer.expires = jiffies + HZ;
mod_timer(&mdev->link_stat_timer, mdev->link_stat_timer.expires);
}
hw_addr[4] = lo >> 8;
hw_addr[5] = lo;
- most_deliver_netinfo(&mdev->iface, link, hw_addr);
+ if (mdev->on_netinfo)
+ mdev->on_netinfo(&mdev->iface, link, hw_addr);
}
/**
* The callback returns a negative value on error, otherwise 0.
* @request_netinfo: triggers retrieving of network info from the HDM by
* means of "Message exchange over MDP/MEP"
+ * The call of the function request_netinfo with the parameter on_netinfo as
+ * NULL prohibits use of the previously obtained function pointer.
* @priv Private field used by mostcore to store context information.
*/
struct most_interface {
int (*enqueue)(struct most_interface *iface, int channel_idx,
struct mbo *mbo);
int (*poison_channel)(struct most_interface *iface, int channel_idx);
- void (*request_netinfo)(struct most_interface *iface, int channel_idx);
+ void (*request_netinfo)(struct most_interface *iface, int channel_idx,
+ void (*on_netinfo)(struct most_interface *iface,
+ unsigned char link_stat,
+ unsigned char *mac_addr));
void *priv;
};
status = cvmx_usb_initialize(dev, usb);
if (status) {
dev_dbg(dev, "USB initialization failed with %d\n", status);
- kfree(hcd);
+ usb_put_hcd(hcd);
return -1;
}
status = usb_add_hcd(hcd, irq, 0);
if (status) {
dev_dbg(dev, "USB add HCD failed with %d\n", status);
- kfree(hcd);
+ usb_put_hcd(hcd);
return -1;
}
device_wakeup_enable(hcd->self.controller);
if (status)
dev_dbg(dev, "USB shutdown failed with %d\n", status);
- kfree(hcd);
+ usb_put_hcd(hcd);
return 0;
}
interface = cvmx_helper_get_interface_num(ipd_port);
if (interface >= 0)
return interface;
- panic("Illegal ipd_port %d passed to INTERFACE\n", ipd_port);
+ panic("Illegal ipd_port %d passed to %s\n", ipd_port, __func__);
}
/**
cur_ch_offset = HAL_PRIME_CHNL_OFFSET_DONT_CARE;
- /* check if there is wps ie, */
- /* if there is wpsie in beacon, the hostapd will update beacon twice when stating hostapd, */
- /* and at first time the security ie (RSN/WPA IE) will not include in beacon. */
+ /* check if there is wps ie,
+ * if there is wpsie in beacon, the hostapd will update
+ * beacon twice when stating hostapd, and at first time the
+ * security ie (RSN/WPA IE) will not include in beacon.
+ */
if (!rtw_get_wps_ie(pnetwork->IEs + _FIXED_IE_LENGTH_, pnetwork->IELength - _FIXED_IE_LENGTH_, NULL, NULL))
pmlmeext->bstart_bss = true;
update_hw_ht_param(padapter);
}
- if (pmlmepriv->cur_network.join_res != true) { /* setting only at first time */
- /* WEP Key will be set before this function, do not clear CAM. */
+ /* setting only at first time */
+ if (!(pmlmepriv->cur_network.join_res)) {
+ /* WEP Key will be set before this function, do not
+ * clear CAM.
+ */
if ((psecuritypriv->dot11PrivacyAlgrthm != _WEP40_) &&
(psecuritypriv->dot11PrivacyAlgrthm != _WEP104_))
flush_all_cam_entry(padapter); /* clear CAM */
}
}
}
- /* TODO: need to judge the phy parameters on concurrent mode for single phy */
+ /* TODO: need to judge the phy parameters on concurrent
+ * mode for single phy
+ */
set_channel_bwmode(padapter, cur_channel, cur_ch_offset, cur_bwmode);
DBG_88E("CH =%d, BW =%d, offset =%d\n", cur_channel, cur_bwmode, cur_ch_offset);
}
break;
}
- if ((p == NULL) || (ie_len == 0))
+ if ((!p) || (ie_len == 0))
break;
}
if ((p) && !memcmp(p + 2, WMM_PARA_IE, 6)) {
pmlmepriv->qospriv.qos_option = 1;
- *(p + 8) |= BIT(7);/* QoS Info, support U-APSD */
+ /* QoS Info, support U-APSD */
+ *(p + 8) |= BIT(7);
- /* disable all ACM bits since the WMM admission control is not supported */
+ /* disable all ACM bits since the WMM
+ * admission control is not supported
+ */
*(p + 10) &= ~BIT(4); /* BE */
*(p + 14) &= ~BIT(4); /* BK */
*(p + 18) &= ~BIT(4); /* VI */
break;
}
- if ((p == NULL) || (ie_len == 0))
+ if ((!p) || (ie_len == 0))
break;
}
}
psta = rtw_get_stainfo(&padapter->stapriv, pbss_network->MacAddress);
if (!psta) {
psta = rtw_alloc_stainfo(&padapter->stapriv, pbss_network->MacAddress);
- if (psta == NULL)
+ if (!psta)
return _FAIL;
}
DBG_88E("%s\n", __func__);
pwps_ie_src = pmlmepriv->wps_beacon_ie;
- if (pwps_ie_src == NULL)
+ if (!pwps_ie_src)
return;
pwps_ie = rtw_get_wps_ie(ie+_FIXED_IE_LENGTH_, ielen-_FIXED_IE_LENGTH_, NULL, &wps_ielen);
- if (pwps_ie == NULL || wps_ielen == 0)
+ if (!pwps_ie || wps_ielen == 0)
return;
wps_offset = (uint)(pwps_ie-ie);
pmlmepriv->update_bcn = false;
pmlmeext->bstart_bss = false;
- /* reset and init security priv , this can refine with rtw_reset_securitypriv */
+ /* reset and init security priv , this can refine with
+ * rtw_reset_securitypriv
+ */
memset((unsigned char *)&padapter->securitypriv, 0, sizeof(struct security_priv));
padapter->securitypriv.ndisauthtype = Ndis802_11AuthModeOpen;
padapter->securitypriv.ndisencryptstatus = Ndis802_11WEPDisabled;
#include <rtw_mlme_ext.h>
/*
-Caller and the rtw_cmd_thread can protect cmd_q by spin_lock.
-No irqsave is necessary.
-*/
+ * Caller and the rtw_cmd_thread can protect cmd_q by spin_lock.
+ * No irqsave is necessary.
+ */
int rtw_init_cmd_priv(struct cmd_priv *pcmdpriv)
{
init_completion(&pcmdpriv->cmd_queue_comp);
init_completion(&pcmdpriv->terminate_cmdthread_comp);
- _rtw_init_queue(&(pcmdpriv->cmd_queue));
+ _rtw_init_queue(&pcmdpriv->cmd_queue);
return _SUCCESS;
}
/*
-Calling Context:
-
-rtw_enqueue_cmd can only be called between kernel thread,
-since only spin_lock is used.
-
-ISR/Call-Back functions can't call this sub-function.
-
-*/
+ * Calling Context:
+ *
+ * rtw_enqueue_cmd can only be called between kernel thread,
+ * since only spin_lock is used.
+ *
+ * ISR/Call-Back functions can't call this sub-function.
+ */
static int _rtw_enqueue_cmd(struct __queue *queue, struct cmd_obj *obj)
{
unsigned long irqL;
-
- if (obj == NULL)
+ if (!obj)
goto exit;
spin_lock_irqsave(&queue->lock, irqL);
exit:
-
return _SUCCESS;
}
int res = _FAIL;
struct adapter *padapter = pcmdpriv->padapter;
-
- if (cmd_obj == NULL)
+ if (!cmd_obj)
goto exit;
cmd_obj->padapter = padapter;
exit:
-
return res;
}
void rtw_free_cmd_obj(struct cmd_obj *pcmd)
{
-
if ((pcmd->cmdcode != _JoinBss_CMD_) && (pcmd->cmdcode != _CreateBss_CMD_)) {
/* free parmbuf in cmd_obj */
kfree(pcmd->parmbuf);
}
- if (pcmd->rsp != NULL) {
+ if (!pcmd->rsp) {
if (pcmd->rspsz != 0) {
/* free rsp in cmd_obj */
kfree(pcmd->rsp);
/* free cmd_obj */
kfree(pcmd);
-
}
int rtw_cmd_thread(void *context)
u8 (*cmd_hdl)(struct adapter *padapter, u8 *pbuf);
void (*pcmd_callback)(struct adapter *dev, struct cmd_obj *pcmd);
struct adapter *padapter = context;
- struct cmd_priv *pcmdpriv = &(padapter->cmdpriv);
+ struct cmd_priv *pcmdpriv = &padapter->cmdpriv;
allow_signal(SIGTERM);
pcmdpriv->cmdthd_running = true;
complete(&pcmdpriv->terminate_cmdthread_comp);
- RT_TRACE(_module_rtl871x_cmd_c_, _drv_info_, ("start r871x rtw_cmd_thread !!!!\n"));
+ RT_TRACE(_module_rtl871x_cmd_c_, _drv_info_,
+ ("start r871x %s !!!!\n", __func__));
while (1) {
if (wait_for_completion_interruptible(&pcmdpriv->cmd_queue_comp))
/* call callback function for post-processed */
if (pcmd->cmdcode < ARRAY_SIZE(rtw_cmd_callback)) {
pcmd_callback = rtw_cmd_callback[pcmd->cmdcode].callback;
- if (pcmd_callback == NULL) {
+ if (!pcmd_callback) {
RT_TRACE(_module_rtl871x_cmd_c_, _drv_info_, ("mlme_cmd_hdl(): pcmd_callback = 0x%p, cmdcode = 0x%x\n", pcmd_callback, pcmd->cmdcode));
rtw_free_cmd_obj(pcmd);
} else {
complete(&pcmdpriv->terminate_cmdthread_comp);
-
complete_and_exit(NULL, 0);
}
/*
-rtw_sitesurvey_cmd(~)
- ### NOTE:#### (!!!!)
- MUST TAKE CARE THAT BEFORE CALLING THIS FUNC, YOU SHOULD HAVE LOCKED pmlmepriv->lock
-*/
+ * rtw_sitesurvey_cmd(~)
+ * ### NOTE:#### (!!!!)
+ * MUST TAKE CARE THAT BEFORE CALLING THIS FUNC, YOU SHOULD HAVE
+ * LOCKED pmlmepriv->lock
+ */
u8 rtw_sitesurvey_cmd(struct adapter *padapter, struct ndis_802_11_ssid *ssid, int ssid_num,
struct rtw_ieee80211_channel *ch, int ch_num)
{
_clr_fwstate_(pmlmepriv, _FW_UNDER_SURVEY);
}
-
return res;
}
void rtw_readtssi_cmdrsp_callback(struct adapter *padapter, struct cmd_obj *pcmd)
{
-
kfree(pcmd->parmbuf);
kfree(pcmd);
}
struct wlan_bssid_ex *pdev_network = &padapter->registrypriv.dev_network;
u8 res = _SUCCESS;
-
LedControl8188eu(padapter, LED_CTL_START_TO_LINK);
if (pmlmepriv->assoc_ssid.SsidLength == 0)
res = rtw_enqueue_cmd(pcmdpriv, pcmd);
exit:
-
return res;
}
struct ht_priv *phtpriv = &pmlmepriv->htpriv;
enum ndis_802_11_network_infra ndis_network_mode = pnetwork->network.InfrastructureMode;
struct mlme_ext_priv *pmlmeext = &padapter->mlmeextpriv;
- struct mlme_ext_info *pmlmeinfo = &(pmlmeext->mlmext_info);
-
+ struct mlme_ext_info *pmlmeinfo = &pmlmeext->mlmext_info;
LedControl8188eu(padapter, LED_CTL_START_TO_LINK);
/* for IEs is fix buf size */
t_len = sizeof(struct wlan_bssid_ex);
-
/* for hidden ap to set fw_state here */
- if (!check_fwstate(pmlmepriv, WIFI_STATION_STATE|WIFI_ADHOC_STATE)) {
+ if (!check_fwstate(pmlmepriv, WIFI_STATION_STATE | WIFI_ADHOC_STATE)) {
switch (ndis_network_mode) {
case Ndis802_11IBSS:
set_fwstate(pmlmepriv, WIFI_ADHOC_STATE);
}
psecnetwork = (struct wlan_bssid_ex *)&psecuritypriv->sec_bss;
- if (psecnetwork == NULL) {
+ if (!psecnetwork) {
kfree(pcmd);
res = _FAIL;
- RT_TRACE(_module_rtl871x_cmd_c_, _drv_err_, ("rtw_joinbss_cmd :psecnetwork == NULL!!!\n"));
+ RT_TRACE(_module_rtl871x_cmd_c_, _drv_err_,
+ ("%s :psecnetwork == NULL!!!\n", __func__));
goto exit;
}
psecnetwork->IELength = rtw_restruct_sec_ie(padapter, &pnetwork->network.IEs[0], &psecnetwork->IEs[0], pnetwork->network.IELength);
-
pqospriv->qos_option = 0;
if (pregistrypriv->wmm_enable) {
exit:
-
return res;
}
struct cmd_priv *cmdpriv = &padapter->cmdpriv;
u8 res = _SUCCESS;
-
- RT_TRACE(_module_rtl871x_cmd_c_, _drv_notice_, ("+rtw_disassoc_cmd\n"));
+ RT_TRACE(_module_rtl871x_cmd_c_, _drv_notice_, ("+%s\n", __func__));
/* prepare cmd parameter */
param = kzalloc(sizeof(*param), GFP_KERNEL);
exit:
-
return res;
}
struct sta_info *sta = (struct sta_info *)psta;
u8 res = _SUCCESS;
-
if (!enqueue) {
clear_cam_entry(padapter, entry);
} else {
}
exit:
-
return res;
}
struct addBaReq_parm *paddbareq_parm;
u8 res = _SUCCESS;
-
ph2c = kzalloc(sizeof(struct cmd_obj), GFP_ATOMIC);
if (!ph2c) {
res = _FAIL;
exit:
-
return res;
}
struct cmd_priv *pcmdpriv = &padapter->cmdpriv;
u8 res = _SUCCESS;
-
ph2c = kzalloc(sizeof(struct cmd_obj), GFP_ATOMIC);
if (!ph2c) {
res = _FAIL;
init_h2fwcmd_w_parm_no_rsp(ph2c, pdrvextra_cmd_parm, _Set_Drv_Extra_CMD_);
-
/* rtw_enqueue_cmd(pcmdpriv, ph2c); */
res = rtw_enqueue_cmd(pcmdpriv, ph2c);
exit:
u8 res = _SUCCESS;
-
- RT_TRACE(_module_rtl871x_cmd_c_, _drv_notice_, ("+rtw_set_chplan_cmd\n"));
+ RT_TRACE(_module_rtl871x_cmd_c_, _drv_notice_, ("+%s\n", __func__));
/* check input parameter */
if (!rtw_is_channel_plan_valid(chplan)) {
exit:
-
return res;
}
u8 bEnterPS;
u8 bBusyTraffic = false, bTxBusyTraffic = false, bRxBusyTraffic = false;
u8 bHigherBusyTraffic = false, bHigherBusyRxTraffic = false, bHigherBusyTxTraffic = false;
- struct mlme_priv *pmlmepriv = &(padapter->mlmepriv);
+ struct mlme_priv *pmlmepriv = &padapter->mlmepriv;
/* */
/* Determine if our traffic is busy now */
struct mlme_priv *pmlmepriv;
padapter = (struct adapter *)pbuf;
- pmlmepriv = &(padapter->mlmepriv);
+ pmlmepriv = &padapter->mlmepriv;
#ifdef CONFIG_88EU_AP_MODE
if (check_fwstate(pmlmepriv, WIFI_AP_STATE) == true)
static void lps_ctrl_wk_hdl(struct adapter *padapter, u8 lps_ctrl_type)
{
struct pwrctrl_priv *pwrpriv = &padapter->pwrctrlpriv;
- struct mlme_priv *pmlmepriv = &(padapter->mlmepriv);
+ struct mlme_priv *pmlmepriv = &padapter->mlmepriv;
u8 mstatus;
-
if ((check_fwstate(pmlmepriv, WIFI_ADHOC_MASTER_STATE) == true) ||
(check_fwstate(pmlmepriv, WIFI_ADHOC_STATE) == true))
return;
default:
break;
}
-
}
u8 rtw_lps_ctrl_wk_cmd(struct adapter *padapter, u8 lps_ctrl_type, u8 enqueue)
exit:
-
return res;
}
res = rtw_enqueue_cmd(pcmdpriv, ph2c);
exit:
-
return res;
}
}
exit:
-
return res;
}
{
struct mlme_priv *pmlmepriv = &padapter->mlmepriv;
-
if (pcmd->res == H2C_DROPPED) {
/* TODO: cancel timer and do timeout handler directly... */
/* need to make timeout handlerOS independent */
/* free cmd */
rtw_free_cmd_obj(pcmd);
-
}
+
void rtw_disassoc_cmd_callback(struct adapter *padapter, struct cmd_obj *pcmd)
{
struct mlme_priv *pmlmepriv = &padapter->mlmepriv;
-
if (pcmd->res != H2C_SUCCESS) {
spin_lock_bh(&pmlmepriv->lock);
set_fwstate(pmlmepriv, _FW_LINKED);
{
struct mlme_priv *pmlmepriv = &padapter->mlmepriv;
-
if (pcmd->res == H2C_DROPPED) {
/* TODO: cancel timer and do timeout handler directly... */
/* need to make timeout handlerOS independent */
}
rtw_free_cmd_obj(pcmd);
-
}
void rtw_createbss_cmd_callback(struct adapter *padapter, struct cmd_obj *pcmd)
struct wlan_network *pwlan = NULL;
struct mlme_priv *pmlmepriv = &padapter->mlmepriv;
struct wlan_bssid_ex *pnetwork = (struct wlan_bssid_ex *)pcmd->parmbuf;
- struct wlan_network *tgt_network = &(pmlmepriv->cur_network);
-
+ struct wlan_network *tgt_network = &pmlmepriv->cur_network;
if (pcmd->res != H2C_SUCCESS) {
- RT_TRACE(_module_rtl871x_cmd_c_, _drv_err_, ("\n ********Error: rtw_createbss_cmd_callback Fail ************\n\n."));
+ RT_TRACE(_module_rtl871x_cmd_c_, _drv_err_,
+ ("\n **** Error: %s Fail ****\n\n.", __func__));
mod_timer(&pmlmepriv->assoc_timer,
jiffies + msecs_to_jiffies(1));
}
psta = rtw_get_stainfo(&padapter->stapriv, pnetwork->MacAddress);
if (!psta) {
psta = rtw_alloc_stainfo(&padapter->stapriv, pnetwork->MacAddress);
- if (psta == NULL) {
+ if (!psta) {
RT_TRACE(_module_rtl871x_cmd_c_, _drv_err_, ("\nCan't alloc sta_info when createbss_cmd_callback\n"));
goto createbss_cmd_fail;
}
rtw_indicate_connect(padapter);
} else {
pwlan = _rtw_alloc_network(pmlmepriv);
- spin_lock_bh(&(pmlmepriv->scanned_queue.lock));
- if (pwlan == NULL) {
+ spin_lock_bh(&pmlmepriv->scanned_queue.lock);
+ if (!pwlan) {
pwlan = rtw_get_oldest_wlan_network(&pmlmepriv->scanned_queue);
- if (pwlan == NULL) {
+ if (!pwlan) {
RT_TRACE(_module_rtl871x_cmd_c_, _drv_err_, ("\n Error: can't get pwlan in rtw_joinbss_event_callback\n"));
spin_unlock_bh(&pmlmepriv->scanned_queue.lock);
goto createbss_cmd_fail;
}
pwlan->last_scanned = jiffies;
} else {
- list_add_tail(&(pwlan->list), &pmlmepriv->scanned_queue.queue);
+ list_add_tail(&pwlan->list,
+ &pmlmepriv->scanned_queue.queue);
}
pnetwork->Length = get_wlan_bssid_ex_sz(pnetwork);
- memcpy(&(pwlan->network), pnetwork, pnetwork->Length);
+ memcpy(&pwlan->network, pnetwork, pnetwork->Length);
memcpy(&tgt_network->network, pnetwork, (get_wlan_bssid_ex_sz(pnetwork)));
_clr_fwstate_(pmlmepriv, _FW_UNDER_LINKING);
spin_unlock_bh(&pmlmepriv->scanned_queue.lock);
- /* we will set _FW_LINKED when there is one more sat to join us (rtw_stassoc_event_callback) */
+ /* we will set _FW_LINKED when there is one more sat to
+ * join us (rtw_stassoc_event_callback)
+ */
}
createbss_cmd_fail:
spin_unlock_bh(&pmlmepriv->lock);
rtw_free_cmd_obj(pcmd);
-
}
void rtw_setstaKey_cmdrsp_callback(struct adapter *padapter, struct cmd_obj *pcmd)
struct set_stakey_rsp *psetstakey_rsp = (struct set_stakey_rsp *)(pcmd->rsp);
struct sta_info *psta = rtw_get_stainfo(pstapriv, psetstakey_rsp->addr);
-
- if (psta == NULL) {
+ if (!psta) {
RT_TRACE(_module_rtl871x_cmd_c_, _drv_err_, ("\nERROR: %s => can't get sta_info\n\n", __func__));
goto exit;
}
struct set_assocsta_rsp *passocsta_rsp = (struct set_assocsta_rsp *)(pcmd->rsp);
struct sta_info *psta = rtw_get_stainfo(pstapriv, passocsta_parm->addr);
-
- if (psta == NULL) {
+ if (!psta) {
RT_TRACE(_module_rtl871x_cmd_c_, _drv_err_, ("\nERROR: %s => can't get sta_info\n\n", __func__));
goto exit;
}
exit:
rtw_free_cmd_obj(pcmd);
-
}
DBG_88E("rtw_macaddr_cfg MAC Address = %pM\n", (mac_addr));
}
-/* Baron adds to avoid FreeBSD warning */
-int ieee80211_is_empty_essid(const char *essid, int essid_len)
-{
- /* Single white space is for Linksys APs */
- if (essid_len == 1 && essid[0] == ' ')
- return 1;
-
- /* Otherwise, if the entire essid is 0, we assume it is hidden */
- while (essid_len) {
- essid_len--;
- if (essid[essid_len] != '\0')
- return 0;
- }
-
- return 1;
-}
-
-int ieee80211_get_hdrlen(u16 fc)
-{
- int hdrlen = 24;
-
- switch (WLAN_FC_GET_TYPE(fc)) {
- case RTW_IEEE80211_FTYPE_DATA:
- if (fc & RTW_IEEE80211_STYPE_QOS_DATA)
- hdrlen += 2;
- if ((fc & RTW_IEEE80211_FCTL_FROMDS) && (fc & RTW_IEEE80211_FCTL_TODS))
- hdrlen += 6; /* Addr4 */
- break;
- case RTW_IEEE80211_FTYPE_CTL:
- switch (WLAN_FC_GET_STYPE(fc)) {
- case RTW_IEEE80211_STYPE_CTS:
- case RTW_IEEE80211_STYPE_ACK:
- hdrlen = 10;
- break;
- default:
- hdrlen = 16;
- break;
- }
- break;
- }
-
- return hdrlen;
-}
-
static int rtw_get_cipher_info(struct wlan_network *pnetwork)
{
uint wpa_ielen;
sq_final = padapter->recvpriv.signal_qual;
/* the rssi value here is undecorated, and will be used for antenna diversity */
if (sq_smp != 101) /* from the right channel */
- rssi_final = (src->Rssi+dst->Rssi*4)/5;
+ rssi_final = (src->Rssi + dst->Rssi * 4) / 5;
else
rssi_final = rssi_ori;
} else {
RT_TRACE(_module_rtl871x_mlme_c_, _drv_info_, ("adhoc mode, fw_state:%x", get_fwstate(pmlmepriv)));
}
- /* s5. Cancle assoc_timer */
+ /* s5. Cancel assoc_timer */
del_timer_sync(&pmlmepriv->assoc_timer);
- RT_TRACE(_module_rtl871x_mlme_c_, _drv_info_, ("Cancle assoc_timer\n"));
+ RT_TRACE(_module_rtl871x_mlme_c_, _drv_info_, ("Cancel assoc_timer\n"));
} else {
RT_TRACE(_module_rtl871x_mlme_c_, _drv_err_, ("rtw_joinbss_event_callback err: fw_state:%x", get_fwstate(pmlmepriv)));
if (pqospriv->qos_option == 0) {
out_len = *pout_len;
- rtw_set_ie(out_ie+out_len, _VENDOR_SPECIFIC_IE_,
+ rtw_set_ie(out_ie + out_len, _VENDOR_SPECIFIC_IE_,
_WMM_IE_Length_, WMM_IE, pout_len);
pqospriv->qos_option = 1;
phtpriv = &psta->htpriv;
if ((phtpriv->ht_option) && (phtpriv->ampdu_enable)) {
- issued = (phtpriv->agg_enable_bitmap>>priority)&0x1;
- issued |= (phtpriv->candidate_tid_bitmap>>priority)&0x1;
+ issued = (phtpriv->agg_enable_bitmap >> priority) & 0x1;
+ issued |= (phtpriv->candidate_tid_bitmap >> priority) & 0x1;
if (issued == 0) {
DBG_88E("rtw_issue_addbareq_cmd, p=%d\n", priority);
}
/* icv_len included the mic code */
- datalen = precvframe->pkt->len-prxattrib->hdrlen -
- prxattrib->iv_len-prxattrib->icv_len-8;
+ datalen = precvframe->pkt->len-prxattrib->hdrlen - 8;
pframe = precvframe->pkt->data;
- payload = pframe+prxattrib->hdrlen+prxattrib->iv_len;
+ payload = pframe+prxattrib->hdrlen;
- RT_TRACE(_module_rtl871x_recv_c_, _drv_info_, ("\n prxattrib->iv_len=%d prxattrib->icv_len=%d\n", prxattrib->iv_len, prxattrib->icv_len));
rtw_seccalctkipmic(mickey, pframe, payload, datalen, &miccode[0],
(unsigned char)prxattrib->priority); /* care the length of the data */
default:
break;
}
+ if (res != _FAIL) {
+ memmove(precv_frame->pkt->data + precv_frame->attrib.iv_len, precv_frame->pkt->data, precv_frame->attrib.hdrlen);
+ skb_pull(precv_frame->pkt, precv_frame->attrib.iv_len);
+ skb_trim(precv_frame->pkt, precv_frame->pkt->len - precv_frame->attrib.icv_len);
+ }
} else if (prxattrib->bdecrypted == 1 && prxattrib->encrypt > 0 &&
- (psecuritypriv->busetkipkey == 1 || prxattrib->encrypt != _TKIP_))
- psecuritypriv->hw_decrypted = true;
+ (psecuritypriv->busetkipkey == 1 || prxattrib->encrypt != _TKIP_)) {
+ psecuritypriv->hw_decrypted = true;
+ }
if (res == _FAIL) {
rtw_free_recvframe(return_packet, &padapter->recvpriv.free_recv_queue);
if (auth_alg == 2) {
/* get ether_type */
- ptr = ptr + pfhdr->attrib.hdrlen + LLC_HEADER_SIZE + pfhdr->attrib.iv_len;
+ ptr = ptr + pfhdr->attrib.hdrlen + LLC_HEADER_SIZE;
memcpy(&be_tmp, ptr, 2);
ether_type = ntohs(be_tmp);
}
if (pattrib->privacy) {
+ struct sk_buff *skb = precv_frame->pkt;
+
RT_TRACE(_module_rtl871x_recv_c_, _drv_info_, ("validate_recv_data_frame:pattrib->privacy=%x\n", pattrib->privacy));
RT_TRACE(_module_rtl871x_recv_c_, _drv_info_, ("\n ^^^^^^^^^^^IS_MCAST(pattrib->ra(0x%02x))=%d^^^^^^^^^^^^^^^6\n", pattrib->ra[0], IS_MCAST(pattrib->ra)));
RT_TRACE(_module_rtl871x_recv_c_, _drv_info_, ("\n pattrib->encrypt=%d\n", pattrib->encrypt));
SET_ICE_IV_LEN(pattrib->iv_len, pattrib->icv_len, pattrib->encrypt);
+
+ if (pattrib->bdecrypted == 1 && pattrib->encrypt > 0) {
+ memmove(skb->data + pattrib->iv_len,
+ skb->data, pattrib->hdrlen);
+ skb_pull(skb, pattrib->iv_len);
+ skb_trim(skb, skb->len - pattrib->icv_len);
+ }
} else {
pattrib->encrypt = 0;
pattrib->iv_len = 0;
* Hence forward the frame to the monitor anyway to preserve the order
* in which frames were received.
*/
+
rtl88eu_mon_recv_hook(adapter->pmondev, precv_frame);
exit:
u8 *ptr = precvframe->pkt->data;
struct rx_pkt_attrib *pattrib = &precvframe->attrib;
- if (pattrib->encrypt)
- skb_trim(precvframe->pkt, precvframe->pkt->len - pattrib->icv_len);
-
- psnap = (struct ieee80211_snap_hdr *)(ptr+pattrib->hdrlen + pattrib->iv_len);
- psnap_type = ptr+pattrib->hdrlen + pattrib->iv_len+SNAP_SIZE;
+ psnap = (struct ieee80211_snap_hdr *)(ptr+pattrib->hdrlen);
+ psnap_type = ptr+pattrib->hdrlen + SNAP_SIZE;
/* convert hdr + possible LLC headers into Ethernet header */
if ((!memcmp(psnap, rtw_rfc1042_header, SNAP_SIZE) &&
(!memcmp(psnap_type, SNAP_ETH_TYPE_IPX, 2) == false) &&
bsnaphdr = false;
}
- rmv_len = pattrib->hdrlen + pattrib->iv_len + (bsnaphdr ? SNAP_SIZE : 0);
+ rmv_len = pattrib->hdrlen + (bsnaphdr ? SNAP_SIZE : 0);
len = precvframe->pkt->len - rmv_len;
- RT_TRACE(_module_rtl871x_recv_c_, _drv_info_,
- ("\n===pattrib->hdrlen: %x, pattrib->iv_len:%x===\n\n", pattrib->hdrlen, pattrib->iv_len));
-
memcpy(&be_tmp, ptr+rmv_len, 2);
eth_type = ntohs(be_tmp); /* pattrib->ether_type */
pattrib->eth_type = eth_type;
struct __queue *defrag_q)
{
struct list_head *plist, *phead;
- u8 wlanhdr_offset;
u8 curfragnum;
struct recv_frame *pfhdr, *pnfhdr;
struct recv_frame *prframe, *pnextrframe;
/* copy the 2nd~n fragment frame's payload to the first fragment */
/* get the 2nd~last fragment frame's payload */
- wlanhdr_offset = pnfhdr->attrib.hdrlen + pnfhdr->attrib.iv_len;
-
- skb_pull(pnextrframe->pkt, wlanhdr_offset);
-
- /* append to first fragment frame's tail (if privacy frame, pull the ICV) */
- skb_trim(prframe->pkt, prframe->pkt->len - pfhdr->attrib.icv_len);
+ skb_pull(pnextrframe->pkt, pnfhdr->attrib.hdrlen);
/* memcpy */
memcpy(skb_tail_pointer(pfhdr->pkt), pnfhdr->pkt->data,
skb_put(prframe->pkt, pnfhdr->pkt->len);
- pfhdr->attrib.icv_len = pnfhdr->attrib.icv_len;
+ pfhdr->attrib.icv_len = 0;
plist = plist->next;
}
nr_subframes = 0;
pattrib = &prframe->attrib;
- skb_pull(prframe->pkt, prframe->attrib.hdrlen);
-
- if (prframe->attrib.iv_len > 0)
- skb_pull(prframe->pkt, prframe->attrib.iv_len);
-
a_len = prframe->pkt->len;
pdata = prframe->pkt->data;
return retval;
}
-static int recv_func_prehandle(struct adapter *padapter,
- struct recv_frame *rframe)
-{
- int ret = _SUCCESS;
- struct __queue *pfree_recv_queue = &padapter->recvpriv.free_recv_queue;
-
- /* check the frame crtl field and decache */
- ret = validate_recv_frame(padapter, rframe);
- if (ret != _SUCCESS) {
- RT_TRACE(_module_rtl871x_recv_c_, _drv_info_, ("recv_func: validate_recv_frame fail! drop pkt\n"));
- rtw_free_recvframe(rframe, pfree_recv_queue);/* free this recv_frame */
- goto exit;
- }
-
-exit:
- return ret;
-}
-
static int recv_func_posthandle(struct adapter *padapter,
struct recv_frame *prframe)
{
struct rx_pkt_attrib *prxattrib = &rframe->attrib;
struct security_priv *psecuritypriv = &padapter->securitypriv;
struct mlme_priv *mlmepriv = &padapter->mlmepriv;
+ struct __queue *pfree_recv_queue = &padapter->recvpriv.free_recv_queue;
/* check if need to handle uc_swdec_pending_queue*/
if (check_fwstate(mlmepriv, WIFI_STATION_STATE) && psecuritypriv->busetkipkey) {
}
}
- ret = recv_func_prehandle(padapter, rframe);
-
- if (ret == _SUCCESS) {
+ /* check the frame crtl field and decache */
+ ret = validate_recv_frame(padapter, rframe);
+ if (ret != _SUCCESS) {
+ RT_TRACE(_module_rtl871x_recv_c_, _drv_info_, ("recv_func: validate_recv_frame fail! drop pkt\n"));
+ rtw_free_recvframe(rframe, pfree_recv_queue);/* free this recv_frame */
+ } else {
/* check if need to enqueue into uc_swdec_pending_queue*/
if (check_fwstate(mlmepriv, WIFI_STATION_STATE) &&
!IS_MCAST(prxattrib->ra) && prxattrib->encrypt > 0 &&
psta->keep_alive_trycnt = 0;
#endif /* CONFIG_88EU_AP_MODE */
-
}
u32 _rtw_init_sta_priv(struct sta_priv *pstapriv)
struct sta_info *psta;
s32 i;
-
pstapriv->pallocated_stainfo_buf = vzalloc(sizeof(struct sta_info) * NUM_STA + 4);
if (!pstapriv->pallocated_stainfo_buf)
pstapriv->max_num_sta = NUM_STA;
#endif
-
return _SUCCESS;
}
struct xmit_priv *pxmitpriv = &padapter->xmitpriv;
struct sta_priv *pstapriv = &padapter->stapriv;
-
if (!psta)
goto exit;
exit:
-
return _SUCCESS;
}
struct sta_priv *pstapriv = &padapter->stapriv;
struct sta_info *pbcmc_stainfo = rtw_get_bcmc_stainfo(padapter);
-
if (pstapriv->asoc_sta_count == 1)
return;
u8 *addr;
u8 bc_addr[ETH_ALEN] = {0xff, 0xff, 0xff, 0xff, 0xff, 0xff};
-
if (!hwaddr)
return NULL;
unsigned char bcast_addr[ETH_ALEN] = {0xff, 0xff, 0xff, 0xff, 0xff, 0xff};
struct sta_priv *pstapriv = &padapter->stapriv;
-
psta = rtw_alloc_stainfo(pstapriv, bcast_addr);
if (!psta) {
}
}
- if (0x00 == path_a_ok) {
+ if (path_a_ok == 0x00) {
ODM_RT_TRACE(dm_odm, ODM_COMP_CALIBRATION, ODM_DBG_LOUD,
("Path A IQK failed!!\n"));
}
}
}
- if (0x00 == path_b_ok) {
+ if (path_b_ok == 0x00) {
ODM_RT_TRACE(dm_odm, ODM_COMP_CALIBRATION, ODM_DBG_LOUD,
("Path B IQK failed!!\n"));
}
}
void rtl8188e_process_phy_info(struct adapter *padapter,
- struct recv_frame *precvframe)
+ struct recv_frame *precvframe)
{
/* Check RSSI */
process_rssi(padapter, precvframe);
#define MIN_FRAG_THRESHOLD 256U
#define MAX_FRAG_THRESHOLD 2346U
-/* Frame control field constants */
-#define RTW_IEEE80211_FCTL_VERS 0x0003
-#define RTW_IEEE80211_FCTL_FTYPE 0x000c
-#define RTW_IEEE80211_FCTL_STYPE 0x00f0
-#define RTW_IEEE80211_FCTL_TODS 0x0100
-#define RTW_IEEE80211_FCTL_FROMDS 0x0200
-#define RTW_IEEE80211_FCTL_MOREFRAGS 0x0400
-#define RTW_IEEE80211_FCTL_RETRY 0x0800
-#define RTW_IEEE80211_FCTL_PM 0x1000
-#define RTW_IEEE80211_FCTL_MOREDATA 0x2000
-#define RTW_IEEE80211_FCTL_PROTECTED 0x4000
-#define RTW_IEEE80211_FCTL_ORDER 0x8000
-#define RTW_IEEE80211_FCTL_CTL_EXT 0x0f00
-
-#define RTW_IEEE80211_FTYPE_MGMT 0x0000
-#define RTW_IEEE80211_FTYPE_CTL 0x0004
-#define RTW_IEEE80211_FTYPE_DATA 0x0008
-#define RTW_IEEE80211_FTYPE_EXT 0x000c
-
-/* management */
-#define RTW_IEEE80211_STYPE_ASSOC_REQ 0x0000
-#define RTW_IEEE80211_STYPE_ASSOC_RESP 0x0010
-#define RTW_IEEE80211_STYPE_REASSOC_REQ 0x0020
-#define RTW_IEEE80211_STYPE_REASSOC_RESP 0x0030
-#define RTW_IEEE80211_STYPE_PROBE_REQ 0x0040
-#define RTW_IEEE80211_STYPE_PROBE_RESP 0x0050
-#define RTW_IEEE80211_STYPE_BEACON 0x0080
-#define RTW_IEEE80211_STYPE_ATIM 0x0090
-#define RTW_IEEE80211_STYPE_DISASSOC 0x00A0
-#define RTW_IEEE80211_STYPE_AUTH 0x00B0
-#define RTW_IEEE80211_STYPE_DEAUTH 0x00C0
-#define RTW_IEEE80211_STYPE_ACTION 0x00D0
-
-/* control */
-#define RTW_IEEE80211_STYPE_CTL_EXT 0x0060
-#define RTW_IEEE80211_STYPE_BACK_REQ 0x0080
-#define RTW_IEEE80211_STYPE_BACK 0x0090
-#define RTW_IEEE80211_STYPE_PSPOLL 0x00A0
-#define RTW_IEEE80211_STYPE_RTS 0x00B0
-#define RTW_IEEE80211_STYPE_CTS 0x00C0
-#define RTW_IEEE80211_STYPE_ACK 0x00D0
-#define RTW_IEEE80211_STYPE_CFEND 0x00E0
-#define RTW_IEEE80211_STYPE_CFENDACK 0x00F0
-
-/* data */
-#define RTW_IEEE80211_STYPE_DATA 0x0000
-#define RTW_IEEE80211_STYPE_DATA_CFACK 0x0010
-#define RTW_IEEE80211_STYPE_DATA_CFPOLL 0x0020
-#define RTW_IEEE80211_STYPE_DATA_CFACKPOLL 0x0030
-#define RTW_IEEE80211_STYPE_NULLFUNC 0x0040
-#define RTW_IEEE80211_STYPE_CFACK 0x0050
-#define RTW_IEEE80211_STYPE_CFPOLL 0x0060
-#define RTW_IEEE80211_STYPE_CFACKPOLL 0x0070
-#define RTW_IEEE80211_STYPE_QOS_DATA 0x0080
-#define RTW_IEEE80211_STYPE_QOS_DATA_CFACK 0x0090
-#define RTW_IEEE80211_STYPE_QOS_DATA_CFPOLL 0x00A0
-#define RTW_IEEE80211_STYPE_QOS_DATA_CFACKPOLL 0x00B0
-#define RTW_IEEE80211_STYPE_QOS_NULLFUNC 0x00C0
-#define RTW_IEEE80211_STYPE_QOS_CFACK 0x00D0
-#define RTW_IEEE80211_STYPE_QOS_CFPOLL 0x00E0
-#define RTW_IEEE80211_STYPE_QOS_CFACKPOLL 0x00F0
-
/* sequence control field */
#define RTW_IEEE80211_SCTL_FRAG 0x000F
#define RTW_IEEE80211_SCTL_SEQ 0xFFF0
#define SNAP_SIZE sizeof(struct ieee80211_snap_hdr)
-#define WLAN_FC_GET_TYPE(fc) ((fc) & RTW_IEEE80211_FCTL_FTYPE)
-#define WLAN_FC_GET_STYPE(fc) ((fc) & RTW_IEEE80211_FCTL_STYPE)
-
#define WLAN_QC_GET_TID(qc) ((qc) & 0x0f)
#define WLAN_GET_SEQ_FRAG(seq) ((seq) & RTW_IEEE80211_SCTL_FRAG)
#define IEEE80211_DATA_HDR3_LEN 24
#define IEEE80211_DATA_HDR4_LEN 30
-
-#define IEEE80211_STATMASK_SIGNAL (1<<0)
-#define IEEE80211_STATMASK_RSSI (1<<1)
-#define IEEE80211_STATMASK_NOISE (1<<2)
-#define IEEE80211_STATMASK_RATE (1<<3)
-#define IEEE80211_STATMASK_WEMASK 0x7
-
-
#define IEEE80211_CCK_MODULATION (1<<0)
#define IEEE80211_OFDM_MODULATION (1<<1)
IEEE80211_OFDM_RATE_36MB_MASK | \
IEEE80211_OFDM_RATE_48MB_MASK | \
IEEE80211_OFDM_RATE_54MB_MASK)
-#define IEEE80211_DEFAULT_RATES_MASK \
- (IEEE80211_OFDM_DEFAULT_RATES_MASK | \
- IEEE80211_CCK_DEFAULT_RATES_MASK)
#define IEEE80211_NUM_OFDM_RATES 8
#define IEEE80211_NUM_CCK_RATES 4
#define WEP_KEYS 4
#define WEP_KEY_LEN 13
-#define BEACON_PROBE_SSID_ID_POSITION 12
-
-/* Management Frame Information Element Types */
-#define MFIE_TYPE_SSID 0
-#define MFIE_TYPE_RATES 1
-#define MFIE_TYPE_FH_SET 2
-#define MFIE_TYPE_DS_SET 3
-#define MFIE_TYPE_CF_SET 4
-#define MFIE_TYPE_TIM 5
-#define MFIE_TYPE_IBSS_SET 6
-#define MFIE_TYPE_CHALLENGE 16
-#define MFIE_TYPE_ERP 42
-#define MFIE_TYPE_RSN 48
-#define MFIE_TYPE_RATES_EX 50
-#define MFIE_TYPE_GENERIC 221
-
-#define IEEE80211_DEFAULT_TX_ESSID "Penguin"
-#define IEEE80211_DEFAULT_BASIC_RATE 10
-
/* SWEEP TABLE ENTRIES NUMBER*/
#define MAX_SWEEP_TAB_ENTRIES 42
#define MAX_SWEEP_TAB_ENTRIES_PER_PACKET 7
#define NETWORK_HAS_OFDM (1<<1)
#define NETWORK_HAS_CCK (1<<2)
-#define IEEE80211_DTIM_MBCAST 4
-#define IEEE80211_DTIM_UCAST 2
-#define IEEE80211_DTIM_VALID 1
-#define IEEE80211_DTIM_INVALID 0
-
-#define IEEE80211_PS_DISABLED 0
-#define IEEE80211_PS_UNICAST IEEE80211_DTIM_UCAST
-#define IEEE80211_PS_MBCAST IEEE80211_DTIM_MBCAST
#define IW_ESSID_MAX_SIZE 32
/*
join_res:
#define IEEE_G (1<<2)
#define IEEE_MODE_MASK (IEEE_A|IEEE_B|IEEE_G)
-/* Baron move to ieee80211.c */
-int ieee80211_is_empty_essid(const char *essid, int essid_len);
-int ieee80211_get_hdrlen(u16 fc);
-
/* Action category code */
enum rtw_ieee80211_category {
RTW_WLAN_CATEGORY_SPECTRUM_MGMT = 0,
netif_rx(skb);
}
+static void mon_recv_decrypted_recv(struct net_device *dev, const u8 *data,
+ int data_len)
+{
+ struct sk_buff *skb;
+ struct ieee80211_hdr *hdr;
+ int hdr_len;
+
+ skb = netdev_alloc_skb(dev, data_len);
+ if (!skb)
+ return;
+ memcpy(skb_put(skb, data_len), data, data_len);
+
+ /*
+ * Frame data is not encrypted. Strip off protection so
+ * userspace doesn't think that it is.
+ */
+
+ hdr = (struct ieee80211_hdr *)skb->data;
+ hdr_len = ieee80211_hdrlen(hdr->frame_control);
+
+ if (ieee80211_has_protected(hdr->frame_control))
+ hdr->frame_control &= ~cpu_to_le16(IEEE80211_FCTL_PROTECTED);
+
+ skb->ip_summed = CHECKSUM_UNNECESSARY;
+ skb->protocol = eth_type_trans(skb, dev);
+ netif_rx(skb);
+}
+
static void mon_recv_encrypted(struct net_device *dev, const u8 *data,
int data_len)
{
void rtl88eu_mon_recv_hook(struct net_device *dev, struct recv_frame *frame)
{
struct rx_pkt_attrib *attr;
- int iv_len, icv_len;
int data_len;
u8 *data;
data = frame->pkt->data;
data_len = frame->pkt->len;
- /* Broadcast and multicast frames don't have attr->{iv,icv}_len set */
- SET_ICE_IV_LEN(iv_len, icv_len, attr->encrypt);
-
if (attr->bdecrypted)
- mon_recv_decrypted(dev, data, data_len, iv_len, icv_len);
+ mon_recv_decrypted_recv(dev, data, data_len);
else
mon_recv_encrypted(dev, data, data_len);
}
};
/**
- * struct rt_dot11d_info * @CountryIeLen: value greater than 0 if @CountryIeBuf contains
- * valid country information element.
+ * struct rt_dot11d_info * @CountryIeLen: value greater than 0 if
+ * @CountryIeBuf contains valid country information element.
* @channel_map: holds channel values
* 0 - invalid,
* 1 - valid (active scan),
return 0;
}
-/* based on ipw2200 driver */
-static int _rtl92e_ioctl(struct net_device *dev, struct ifreq *rq, int cmd)
-{
- struct r8192_priv *priv = rtllib_priv(dev);
- struct iwreq *wrq = (struct iwreq *)rq;
- int ret = -1;
- struct rtllib_device *ieee = priv->rtllib;
- u32 key[4];
- const u8 broadcast_addr[ETH_ALEN] = {0xff, 0xff, 0xff, 0xff, 0xff, 0xff};
- struct iw_point *p = &wrq->u.data;
- struct ieee_param *ipw = NULL;
-
- mutex_lock(&priv->wx_mutex);
-
- switch (cmd) {
- case RTL_IOCTL_WPA_SUPPLICANT:
- if (p->length < sizeof(struct ieee_param) || !p->pointer) {
- ret = -EINVAL;
- goto out;
- }
-
- ipw = memdup_user(p->pointer, p->length);
- if (IS_ERR(ipw)) {
- ret = PTR_ERR(ipw);
- goto out;
- }
-
- if (ipw->cmd == IEEE_CMD_SET_ENCRYPTION) {
- if (ipw->u.crypt.set_tx) {
- if (strcmp(ipw->u.crypt.alg, "CCMP") == 0)
- ieee->pairwise_key_type = KEY_TYPE_CCMP;
- else if (strcmp(ipw->u.crypt.alg, "TKIP") == 0)
- ieee->pairwise_key_type = KEY_TYPE_TKIP;
- else if (strcmp(ipw->u.crypt.alg, "WEP") == 0) {
- if (ipw->u.crypt.key_len == 13)
- ieee->pairwise_key_type =
- KEY_TYPE_WEP104;
- else if (ipw->u.crypt.key_len == 5)
- ieee->pairwise_key_type =
- KEY_TYPE_WEP40;
- } else {
- ieee->pairwise_key_type = KEY_TYPE_NA;
- }
-
- if (ieee->pairwise_key_type) {
- if (is_zero_ether_addr(ieee->ap_mac_addr))
- ieee->iw_mode = IW_MODE_ADHOC;
- memcpy((u8 *)key, ipw->u.crypt.key, 16);
- rtl92e_enable_hw_security_config(dev);
- rtl92e_set_swcam(dev, 4,
- ipw->u.crypt.idx,
- ieee->pairwise_key_type,
- (u8 *)ieee->ap_mac_addr,
- 0, key, 0);
- rtl92e_set_key(dev, 4, ipw->u.crypt.idx,
- ieee->pairwise_key_type,
- (u8 *)ieee->ap_mac_addr,
- 0, key);
- if (ieee->iw_mode == IW_MODE_ADHOC) {
- rtl92e_set_swcam(dev,
- ipw->u.crypt.idx,
- ipw->u.crypt.idx,
- ieee->pairwise_key_type,
- (u8 *)ieee->ap_mac_addr,
- 0, key, 0);
- rtl92e_set_key(dev,
- ipw->u.crypt.idx,
- ipw->u.crypt.idx,
- ieee->pairwise_key_type,
- (u8 *)ieee->ap_mac_addr,
- 0, key);
- }
- }
- if ((ieee->pairwise_key_type ==
- KEY_TYPE_CCMP) &&
- ieee->pHTInfo->bCurrentHTSupport) {
- rtl92e_writeb(dev, 0x173, 1);
- }
-
- } else {
- memcpy((u8 *)key, ipw->u.crypt.key, 16);
- if (strcmp(ipw->u.crypt.alg, "CCMP") == 0)
- ieee->group_key_type = KEY_TYPE_CCMP;
- else if (strcmp(ipw->u.crypt.alg, "TKIP") == 0)
- ieee->group_key_type = KEY_TYPE_TKIP;
- else if (strcmp(ipw->u.crypt.alg, "WEP") == 0) {
- if (ipw->u.crypt.key_len == 13)
- ieee->group_key_type =
- KEY_TYPE_WEP104;
- else if (ipw->u.crypt.key_len == 5)
- ieee->group_key_type =
- KEY_TYPE_WEP40;
- } else
- ieee->group_key_type = KEY_TYPE_NA;
-
- if (ieee->group_key_type) {
- rtl92e_set_swcam(dev, ipw->u.crypt.idx,
- ipw->u.crypt.idx,
- ieee->group_key_type,
- broadcast_addr, 0, key,
- 0);
- rtl92e_set_key(dev, ipw->u.crypt.idx,
- ipw->u.crypt.idx,
- ieee->group_key_type,
- broadcast_addr, 0, key);
- }
- }
- }
-
- ret = rtllib_wpa_supplicant_ioctl(priv->rtllib, &wrq->u.data,
- 0);
- kfree(ipw);
- break;
- default:
- ret = -EOPNOTSUPP;
- break;
- }
-
-out:
- mutex_unlock(&priv->wx_mutex);
-
- return ret;
-}
-
-
static irqreturn_t _rtl92e_irq(int irq, void *netdev)
{
struct net_device *dev = netdev;
.ndo_open = _rtl92e_open,
.ndo_stop = _rtl92e_close,
.ndo_tx_timeout = _rtl92e_tx_timeout,
- .ndo_do_ioctl = _rtl92e_ioctl,
.ndo_set_rx_mode = _rtl92e_set_multicast,
.ndo_set_mac_address = _rtl92e_set_mac_adr,
.ndo_validate_addr = eth_validate_addr,
#define PHY_RSSI_SLID_WIN_MAX 100
-#define RTL_IOCTL_WPA_SUPPLICANT (SIOCIWFIRSTPRIV + 30)
-
#define TxBBGainTableLength 37
#define CCKTxBBGainTableLength 23
if (priv->rtllib->pHTInfo->IOTAction & HT_IOT_ACT_DISABLE_EDCA_TURBO)
goto dm_CheckEdcaTurbo_EXIT;
- {
- u8 *peername[11] = {
- "unknown", "realtek_90", "realtek_92se", "broadcom",
- "ralink", "atheros", "cisco", "marvell", "92u_softap",
- "self_softap"
- };
- static int wb_tmp;
-
- if (wb_tmp == 0) {
- netdev_info(dev,
- "%s():iot peer is %s, bssid: %pM\n",
- __func__, peername[pHTInfo->IOTPeer],
- priv->rtllib->current_network.bssid);
- wb_tmp = 1;
- }
- }
if (!priv->rtllib->bis_any_nonbepkts) {
curTxOkCnt = priv->stats.txbytesunicast - lastTxOkCnt;
curRxOkCnt = priv->stats.rxbytesunicast - lastRxOkCnt;
pHTInfo->CurSTAExtChnlOffset = HT_EXTCHNL_OFFSET_NO_EXT;
}
- pr_info("%s():pHTInfo->bCurBW40MHz:%x\n", __func__,
- pHTInfo->bCurBW40MHz);
+ netdev_dbg(ieee->dev, "%s():pHTInfo->bCurBW40MHz:%x\n", __func__,
+ pHTInfo->bCurBW40MHz);
pHTInfo->bSwBwInProgress = true;
#define MGMT_QUEUE_NUM 5
-#define IEEE_CMD_SET_WPA_PARAM 1
-#define IEEE_CMD_SET_WPA_IE 2
-#define IEEE_CMD_SET_ENCRYPTION 3
-#define IEEE_CMD_MLME 4
-
-#define IEEE_PARAM_WPA_ENABLED 1
-#define IEEE_PARAM_TKIP_COUNTERMEASURES 2
-#define IEEE_PARAM_DROP_UNENCRYPTED 3
-#define IEEE_PARAM_PRIVACY_INVOKED 4
-#define IEEE_PARAM_AUTH_ALGS 5
-#define IEEE_PARAM_IEEE_802_1X 6
-#define IEEE_PARAM_WPAX_SELECT 7
-
-#define IEEE_MLME_STA_DEAUTH 1
-#define IEEE_MLME_STA_DISASSOC 2
-
-
-#define IEEE_CRYPT_ERR_UNKNOWN_ALG 2
-#define IEEE_CRYPT_ERR_CRYPT_INIT_FAILED 4
-#define IEEE_CRYPT_ERR_KEY_SET_FAILED 5
-#define IEEE_CRYPT_ERR_CARD_CONF_FAILED 7
-#define IEEE_CRYPT_ALG_NAME_LEN 16
-
#define MAX_IE_LEN 0xff
-struct ieee_param {
- u32 cmd;
- u8 sta_addr[ETH_ALEN];
- union {
- struct {
- u8 name;
- u32 value;
- } wpa_param;
- struct {
- u32 len;
- u8 reserved[32];
- u8 data[0];
- } wpa_ie;
- struct {
- int command;
- int reason_code;
- } mlme;
- struct {
- u8 alg[IEEE_CRYPT_ALG_NAME_LEN];
- u8 set_tx;
- u32 err;
- u8 idx;
- u8 seq[8]; /* sequence counter (set: RX, get: TX) */
- u16 key_len;
- u8 key[0];
- } crypt;
- } u;
-};
-
#define msleep_interruptible_rsl msleep_interruptible
/* Maximum size for the MA-UNITDATA primitive, 802.11 standard section
struct sk_buff *rtllib_get_beacon(struct rtllib_device *ieee);
void rtllib_start_send_beacons(struct rtllib_device *ieee);
void rtllib_stop_send_beacons(struct rtllib_device *ieee);
-int rtllib_wpa_supplicant_ioctl(struct rtllib_device *ieee,
- struct iw_point *p, u8 is_mesh);
void notify_wx_assoc_event(struct rtllib_device *ieee);
void rtllib_ps_tx_ack(struct rtllib_device *ieee, short success);
return -1;
}
- if (rtllib_is_eapol_frame(ieee, skb, hdrlen))
- netdev_warn(ieee->dev, "RX: IEEE802.1X EAPOL frame!\n");
-
return 0;
}
associate_complete_wq);
struct rt_pwr_save_ctrl *pPSC = &(ieee->PowerSaveControl);
- netdev_info(ieee->dev, "Associated successfully\n");
+ netdev_info(ieee->dev, "Associated successfully with %pM\n",
+ ieee->current_network.bssid);
if (!ieee->is_silent_reset) {
netdev_info(ieee->dev, "normal associate\n");
notify_wx_assoc_event(ieee);
if (errcode) {
ieee->softmac_stats.rx_auth_rs_err++;
netdev_info(ieee->dev,
- "Authentication respose status code 0x%x", errcode);
+ "Authentication response status code 0x%x",
+ errcode);
rtllib_associate_abort(ieee);
return;
}
tasklet_kill(&ieee->ps_task);
}
-/********************************************************
- * Start of WPA code. *
- * this is stolen from the ipw2200 driver *
- ********************************************************/
-
-
-static int rtllib_wpa_enable(struct rtllib_device *ieee, int value)
-{
- /* This is called when wpa_supplicant loads and closes the driver
- * interface.
- */
- netdev_info(ieee->dev, "%s WPA\n", value ? "enabling" : "disabling");
- ieee->wpa_enabled = value;
- eth_zero_addr(ieee->ap_mac_addr);
- return 0;
-}
-
-
-static void rtllib_wpa_assoc_frame(struct rtllib_device *ieee, char *wpa_ie,
- int wpa_ie_len)
-{
- /* make sure WPA is enabled */
- rtllib_wpa_enable(ieee, 1);
-
- rtllib_disassociate(ieee);
-}
-
-
-static int rtllib_wpa_mlme(struct rtllib_device *ieee, int command, int reason)
-{
-
- int ret = 0;
-
- switch (command) {
- case IEEE_MLME_STA_DEAUTH:
- break;
-
- case IEEE_MLME_STA_DISASSOC:
- rtllib_disassociate(ieee);
- break;
-
- default:
- netdev_info(ieee->dev, "Unknown MLME request: %d\n", command);
- ret = -EOPNOTSUPP;
- }
-
- return ret;
-}
-
-
-static int rtllib_wpa_set_wpa_ie(struct rtllib_device *ieee,
- struct ieee_param *param, int plen)
-{
- u8 *buf;
-
- if (param->u.wpa_ie.len > MAX_WPA_IE_LEN ||
- (param->u.wpa_ie.len && param->u.wpa_ie.data == NULL))
- return -EINVAL;
-
- if (param->u.wpa_ie.len) {
- buf = kmemdup(param->u.wpa_ie.data, param->u.wpa_ie.len,
- GFP_KERNEL);
- if (buf == NULL)
- return -ENOMEM;
-
- kfree(ieee->wpa_ie);
- ieee->wpa_ie = buf;
- ieee->wpa_ie_len = param->u.wpa_ie.len;
- } else {
- kfree(ieee->wpa_ie);
- ieee->wpa_ie = NULL;
- ieee->wpa_ie_len = 0;
- }
-
- rtllib_wpa_assoc_frame(ieee, ieee->wpa_ie, ieee->wpa_ie_len);
- return 0;
-}
-
-#define AUTH_ALG_OPEN_SYSTEM 0x1
-#define AUTH_ALG_SHARED_KEY 0x2
-#define AUTH_ALG_LEAP 0x4
-static int rtllib_wpa_set_auth_algs(struct rtllib_device *ieee, int value)
-{
-
- struct rtllib_security sec = {
- .flags = SEC_AUTH_MODE,
- };
-
- if (value & AUTH_ALG_SHARED_KEY) {
- sec.auth_mode = WLAN_AUTH_SHARED_KEY;
- ieee->open_wep = 0;
- ieee->auth_mode = 1;
- } else if (value & AUTH_ALG_OPEN_SYSTEM) {
- sec.auth_mode = WLAN_AUTH_OPEN;
- ieee->open_wep = 1;
- ieee->auth_mode = 0;
- } else if (value & AUTH_ALG_LEAP) {
- sec.auth_mode = WLAN_AUTH_LEAP >> 6;
- ieee->open_wep = 1;
- ieee->auth_mode = 2;
- }
-
-
- if (ieee->set_security)
- ieee->set_security(ieee->dev, &sec);
-
- return 0;
-}
-
-static int rtllib_wpa_set_param(struct rtllib_device *ieee, u8 name, u32 value)
-{
- int ret = 0;
- unsigned long flags;
-
- switch (name) {
- case IEEE_PARAM_WPA_ENABLED:
- ret = rtllib_wpa_enable(ieee, value);
- break;
-
- case IEEE_PARAM_TKIP_COUNTERMEASURES:
- ieee->tkip_countermeasures = value;
- break;
-
- case IEEE_PARAM_DROP_UNENCRYPTED:
- {
- /* HACK:
- *
- * wpa_supplicant calls set_wpa_enabled when the driver
- * is loaded and unloaded, regardless of if WPA is being
- * used. No other calls are made which can be used to
- * determine if encryption will be used or not prior to
- * association being expected. If encryption is not being
- * used, drop_unencrypted is set to false, else true -- we
- * can use this to determine if the CAP_PRIVACY_ON bit should
- * be set.
- */
- struct rtllib_security sec = {
- .flags = SEC_ENABLED,
- .enabled = value,
- };
- ieee->drop_unencrypted = value;
- /* We only change SEC_LEVEL for open mode. Others
- * are set by ipw_wpa_set_encryption.
- */
- if (!value) {
- sec.flags |= SEC_LEVEL;
- sec.level = SEC_LEVEL_0;
- } else {
- sec.flags |= SEC_LEVEL;
- sec.level = SEC_LEVEL_1;
- }
- if (ieee->set_security)
- ieee->set_security(ieee->dev, &sec);
- break;
- }
-
- case IEEE_PARAM_PRIVACY_INVOKED:
- ieee->privacy_invoked = value;
- break;
-
- case IEEE_PARAM_AUTH_ALGS:
- ret = rtllib_wpa_set_auth_algs(ieee, value);
- break;
-
- case IEEE_PARAM_IEEE_802_1X:
- ieee->ieee802_1x = value;
- break;
- case IEEE_PARAM_WPAX_SELECT:
- spin_lock_irqsave(&ieee->wpax_suitlist_lock, flags);
- spin_unlock_irqrestore(&ieee->wpax_suitlist_lock, flags);
- break;
-
- default:
- netdev_info(ieee->dev, "Unknown WPA param: %d\n", name);
- ret = -EOPNOTSUPP;
- }
-
- return ret;
-}
-
-/* implementation borrowed from hostap driver */
-static int rtllib_wpa_set_encryption(struct rtllib_device *ieee,
- struct ieee_param *param, int param_len,
- u8 is_mesh)
-{
- int ret = 0;
- struct lib80211_crypto_ops *ops;
- struct lib80211_crypt_data **crypt;
-
- struct rtllib_security sec = {
- .flags = 0,
- };
-
- param->u.crypt.err = 0;
- param->u.crypt.alg[IEEE_CRYPT_ALG_NAME_LEN - 1] = '\0';
-
- if (param_len !=
- (int) ((char *) param->u.crypt.key - (char *) param) +
- param->u.crypt.key_len) {
- netdev_info(ieee->dev, "Len mismatch %d, %d\n", param_len,
- param->u.crypt.key_len);
- return -EINVAL;
- }
- if (is_broadcast_ether_addr(param->sta_addr)) {
- if (param->u.crypt.idx >= NUM_WEP_KEYS)
- return -EINVAL;
- crypt = &ieee->crypt_info.crypt[param->u.crypt.idx];
- } else {
- return -EINVAL;
- }
-
- if (strcmp(param->u.crypt.alg, "none") == 0) {
- if (crypt) {
- sec.enabled = 0;
- sec.level = SEC_LEVEL_0;
- sec.flags |= SEC_ENABLED | SEC_LEVEL;
- lib80211_crypt_delayed_deinit(&ieee->crypt_info, crypt);
- }
- goto done;
- }
- sec.enabled = 1;
- sec.flags |= SEC_ENABLED;
-
- /* IPW HW cannot build TKIP MIC, host decryption still needed. */
- if (!(ieee->host_encrypt || ieee->host_decrypt) &&
- strcmp(param->u.crypt.alg, "R-TKIP"))
- goto skip_host_crypt;
-
- ops = lib80211_get_crypto_ops(param->u.crypt.alg);
- if (ops == NULL && strcmp(param->u.crypt.alg, "R-WEP") == 0) {
- request_module("rtllib_crypt_wep");
- ops = lib80211_get_crypto_ops(param->u.crypt.alg);
- } else if (ops == NULL && strcmp(param->u.crypt.alg, "R-TKIP") == 0) {
- request_module("rtllib_crypt_tkip");
- ops = lib80211_get_crypto_ops(param->u.crypt.alg);
- } else if (ops == NULL && strcmp(param->u.crypt.alg, "R-CCMP") == 0) {
- request_module("rtllib_crypt_ccmp");
- ops = lib80211_get_crypto_ops(param->u.crypt.alg);
- }
- if (ops == NULL) {
- netdev_info(ieee->dev, "unknown crypto alg '%s'\n",
- param->u.crypt.alg);
- param->u.crypt.err = IEEE_CRYPT_ERR_UNKNOWN_ALG;
- ret = -EINVAL;
- goto done;
- }
- if (*crypt == NULL || (*crypt)->ops != ops) {
- struct lib80211_crypt_data *new_crypt;
-
- lib80211_crypt_delayed_deinit(&ieee->crypt_info, crypt);
-
- new_crypt = kzalloc(sizeof(*new_crypt), GFP_KERNEL);
- if (new_crypt == NULL) {
- ret = -ENOMEM;
- goto done;
- }
- new_crypt->ops = ops;
- if (new_crypt->ops && try_module_get(new_crypt->ops->owner))
- new_crypt->priv =
- new_crypt->ops->init(param->u.crypt.idx);
-
- if (new_crypt->priv == NULL) {
- kfree(new_crypt);
- param->u.crypt.err = IEEE_CRYPT_ERR_CRYPT_INIT_FAILED;
- ret = -EINVAL;
- goto done;
- }
-
- *crypt = new_crypt;
- }
-
- if (param->u.crypt.key_len > 0 && (*crypt)->ops->set_key &&
- (*crypt)->ops->set_key(param->u.crypt.key,
- param->u.crypt.key_len, param->u.crypt.seq,
- (*crypt)->priv) < 0) {
- netdev_info(ieee->dev, "key setting failed\n");
- param->u.crypt.err = IEEE_CRYPT_ERR_KEY_SET_FAILED;
- ret = -EINVAL;
- goto done;
- }
-
- skip_host_crypt:
- if (param->u.crypt.set_tx) {
- ieee->crypt_info.tx_keyidx = param->u.crypt.idx;
- sec.active_key = param->u.crypt.idx;
- sec.flags |= SEC_ACTIVE_KEY;
- } else
- sec.flags &= ~SEC_ACTIVE_KEY;
-
- memcpy(sec.keys[param->u.crypt.idx],
- param->u.crypt.key,
- param->u.crypt.key_len);
- sec.key_sizes[param->u.crypt.idx] = param->u.crypt.key_len;
- sec.flags |= (1 << param->u.crypt.idx);
-
- if (strcmp(param->u.crypt.alg, "R-WEP") == 0) {
- sec.flags |= SEC_LEVEL;
- sec.level = SEC_LEVEL_1;
- } else if (strcmp(param->u.crypt.alg, "R-TKIP") == 0) {
- sec.flags |= SEC_LEVEL;
- sec.level = SEC_LEVEL_2;
- } else if (strcmp(param->u.crypt.alg, "R-CCMP") == 0) {
- sec.flags |= SEC_LEVEL;
- sec.level = SEC_LEVEL_3;
- }
- done:
- if (ieee->set_security)
- ieee->set_security(ieee->dev, &sec);
-
- /* Do not reset port if card is in Managed mode since resetting will
- * generate new IEEE 802.11 authentication which may end up in looping
- * with IEEE 802.1X. If your hardware requires a reset after WEP
- * configuration (for example... Prism2), implement the reset_port in
- * the callbacks structures used to initialize the 802.11 stack.
- */
- if (ieee->reset_on_keychange &&
- ieee->iw_mode != IW_MODE_INFRA &&
- ieee->reset_port &&
- ieee->reset_port(ieee->dev)) {
- netdev_info(ieee->dev, "reset_port failed\n");
- param->u.crypt.err = IEEE_CRYPT_ERR_CARD_CONF_FAILED;
- return -EINVAL;
- }
-
- return ret;
-}
-
static inline struct sk_buff *
rtllib_disauth_skb(struct rtllib_network *beacon,
struct rtllib_device *ieee, u16 asRsn)
}
}
-int rtllib_wpa_supplicant_ioctl(struct rtllib_device *ieee, struct iw_point *p,
- u8 is_mesh)
-{
- struct ieee_param *param;
- int ret = 0;
-
- mutex_lock(&ieee->wx_mutex);
-
- if (p->length < sizeof(struct ieee_param) || !p->pointer) {
- ret = -EINVAL;
- goto out;
- }
-
- param = memdup_user(p->pointer, p->length);
- if (IS_ERR(param)) {
- ret = PTR_ERR(param);
- goto out;
- }
-
- switch (param->cmd) {
- case IEEE_CMD_SET_WPA_PARAM:
- ret = rtllib_wpa_set_param(ieee, param->u.wpa_param.name,
- param->u.wpa_param.value);
- break;
-
- case IEEE_CMD_SET_WPA_IE:
- ret = rtllib_wpa_set_wpa_ie(ieee, param, p->length);
- break;
-
- case IEEE_CMD_SET_ENCRYPTION:
- ret = rtllib_wpa_set_encryption(ieee, param, p->length, 0);
- break;
-
- case IEEE_CMD_MLME:
- ret = rtllib_wpa_mlme(ieee, param->u.mlme.command,
- param->u.mlme.reason_code);
- break;
-
- default:
- netdev_info(ieee->dev, "Unknown WPA supplicant request: %d\n",
- param->cmd);
- ret = -EOPNOTSUPP;
- break;
- }
-
- if (ret == 0 && copy_to_user(p->pointer, param, p->length))
- ret = -EFAULT;
-
- kfree(param);
-out:
- mutex_unlock(&ieee->wx_mutex);
-
- return ret;
-}
-EXPORT_SYMBOL(rtllib_wpa_supplicant_ioctl);
-
static void rtllib_MgntDisconnectIBSS(struct rtllib_device *rtllib)
{
u8 OpMode;
ret = -EINVAL;
goto done;
}
- netdev_info(dev, "alg name:%s\n", alg);
+ netdev_dbg(dev, "alg name:%s\n", alg);
ops = lib80211_get_crypto_ops(alg);
if (ops == NULL) {
struct sk_buff *frag, int hdr_len);
int ieee80211_xmit(struct sk_buff *skb, struct net_device *dev);
-void ieee80211_txb_free(struct ieee80211_txb *);
+void ieee80211_txb_free(struct ieee80211_txb *txb);
/* ieee80211_rx.c */
int ieee80211_register_crypto_ops(struct ieee80211_crypto_ops *ops);
int ieee80211_unregister_crypto_ops(struct ieee80211_crypto_ops *ops);
struct ieee80211_crypto_ops *ieee80211_get_crypto_ops(const char *name);
-void ieee80211_crypt_deinit_entries(struct ieee80211_device *, int);
-void ieee80211_crypt_deinit_handler(unsigned long);
+void ieee80211_crypt_deinit_entries(struct ieee80211_device *ieee, int force);
+void ieee80211_crypt_deinit_handler(unsigned long data);
void ieee80211_crypt_delayed_deinit(struct ieee80211_device *ieee,
struct ieee80211_crypt_data **crypt);
return lo | (((u16) hi) << 8);
}
-
-static inline u16 Mk16_le(u16 *v)
-{
- return le16_to_cpu(*v);
-}
-
-
static const u16 Sbox[256] = {
0xC6A5, 0xF884, 0xEE99, 0xF68D, 0xFF0D, 0xD6BD, 0xDEB1, 0x9154,
0x6050, 0x0203, 0xCEA9, 0x567D, 0xE719, 0xB562, 0x4DE6, 0xEC9A,
PPK[5] = TTAK[4] + IV16;
/* Step 2 - 96-bit bijective mixing using S-box */
- PPK[0] += _S_(PPK[5] ^ Mk16_le((u16 *) &TK[0]));
- PPK[1] += _S_(PPK[0] ^ Mk16_le((u16 *) &TK[2]));
- PPK[2] += _S_(PPK[1] ^ Mk16_le((u16 *) &TK[4]));
- PPK[3] += _S_(PPK[2] ^ Mk16_le((u16 *) &TK[6]));
- PPK[4] += _S_(PPK[3] ^ Mk16_le((u16 *) &TK[8]));
- PPK[5] += _S_(PPK[4] ^ Mk16_le((u16 *) &TK[10]));
-
- PPK[0] += RotR1(PPK[5] ^ Mk16_le((u16 *) &TK[12]));
- PPK[1] += RotR1(PPK[0] ^ Mk16_le((u16 *) &TK[14]));
+ PPK[0] += _S_(PPK[5] ^ le16_to_cpu(*(__le16 *)(&TK[0])));
+ PPK[1] += _S_(PPK[0] ^ le16_to_cpu(*(__le16 *)(&TK[2])));
+ PPK[2] += _S_(PPK[1] ^ le16_to_cpu(*(__le16 *)(&TK[4])));
+ PPK[3] += _S_(PPK[2] ^ le16_to_cpu(*(__le16 *)(&TK[6])));
+ PPK[4] += _S_(PPK[3] ^ le16_to_cpu(*(__le16 *)(&TK[8])));
+ PPK[5] += _S_(PPK[4] ^ le16_to_cpu(*(__le16 *)(&TK[10])));
+
+ PPK[0] += RotR1(PPK[5] ^ le16_to_cpu(*(__le16 *)(&TK[12])));
+ PPK[1] += RotR1(PPK[0] ^ le16_to_cpu(*(__le16 *)(&TK[14])));
PPK[2] += RotR1(PPK[1]);
PPK[3] += RotR1(PPK[2]);
PPK[4] += RotR1(PPK[3]);
WEPSeed[0] = Hi8(IV16);
WEPSeed[1] = (Hi8(IV16) | 0x20) & 0x7F;
WEPSeed[2] = Lo8(IV16);
- WEPSeed[3] = Lo8((PPK[5] ^ Mk16_le((u16 *) &TK[0])) >> 1);
+ WEPSeed[3] = Lo8((PPK[5] ^ le16_to_cpu(*(__le16 *)(&TK[0]))) >> 1);
#ifdef __BIG_ENDIAN
{
ieee80211_softmac_init(ieee);
ieee->pHTInfo = kzalloc(sizeof(RT_HIGH_THROUGHPUT), GFP_KERNEL);
- if (ieee->pHTInfo == NULL)
- {
+ if (ieee->pHTInfo == NULL) {
IEEE80211_DEBUG(IEEE80211_DL_ERR, "can't alloc memory for HTInfo\n");
goto failed;
}
spin_lock_irqsave(&(ieee->reorder_spinlock), flags);
IEEE80211_DEBUG(IEEE80211_DL_REORDER,"==================>%s()\n",__func__);
- if(pRxTs->RxTimeoutIndicateSeq != 0xffff)
- {
+ if(pRxTs->RxTimeoutIndicateSeq != 0xffff) {
// Indicate the pending packets sequentially according to SeqNum until meet the gap.
- while(!list_empty(&pRxTs->RxPendingPktList))
- {
+ while(!list_empty(&pRxTs->RxPendingPktList)) {
pReorderEntry = (PRX_REORDER_ENTRY)list_entry(pRxTs->RxPendingPktList.prev,RX_REORDER_ENTRY,List);
if(index == 0)
pRxTs->RxIndicateSeq = pReorderEntry->SeqNum;
if( SN_LESS(pReorderEntry->SeqNum, pRxTs->RxIndicateSeq) ||
- SN_EQUAL(pReorderEntry->SeqNum, pRxTs->RxIndicateSeq) )
- {
+ SN_EQUAL(pReorderEntry->SeqNum, pRxTs->RxIndicateSeq) ) {
list_del_init(&pReorderEntry->List);
if(SN_EQUAL(pReorderEntry->SeqNum, pRxTs->RxIndicateSeq))
index++;
list_add_tail(&pReorderEntry->List, &ieee->RxReorder_Unused_List);
- }
- else
- {
+ } else {
bPktInBuf = true;
break;
}
}
}
- if(index>0)
- {
+ if(index>0) {
// Set RxTimeoutIndicateSeq to 0xffff to indicate no pending packets in buffer now.
pRxTs->RxTimeoutIndicateSeq = 0xffff;
// Indicate packets
- if(index > REORDER_WIN_SIZE){
+ if(index > REORDER_WIN_SIZE) {
IEEE80211_DEBUG(IEEE80211_DL_ERR, "RxReorderIndicatePacket(): Rx Reorder buffer full!! \n");
spin_unlock_irqrestore(&(ieee->reorder_spinlock), flags);
return;
ieee80211_indicate_packets(ieee, ieee->stats_IndicateArray, index);
}
- if(bPktInBuf && (pRxTs->RxTimeoutIndicateSeq==0xffff))
- {
+ if(bPktInBuf && (pRxTs->RxTimeoutIndicateSeq==0xffff)) {
pRxTs->RxTimeoutIndicateSeq = pRxTs->RxIndicateSeq;
mod_timer(&pRxTs->RxPktPendingTimer,
jiffies + msecs_to_jiffies(ieee->pHTInfo->RxReorderPendingTime));
INIT_LIST_HEAD(&ieee->Tx_TS_Pending_List);
INIT_LIST_HEAD(&ieee->Tx_TS_Unused_List);
- for(count = 0; count < TOTAL_TS_NUM; count++)
- {
+ for(count = 0; count < TOTAL_TS_NUM; count++) {
//
pTxTS->num = count;
// The timers for the operation of Traffic Stream and Block Ack.
INIT_LIST_HEAD(&ieee->Rx_TS_Admit_List);
INIT_LIST_HEAD(&ieee->Rx_TS_Pending_List);
INIT_LIST_HEAD(&ieee->Rx_TS_Unused_List);
- for(count = 0; count < TOTAL_TS_NUM; count++)
- {
+ for(count = 0; count < TOTAL_TS_NUM; count++) {
pRxTS->num = count;
INIT_LIST_HEAD(&pRxTS->RxPendingPktList);
setup_timer(&pRxTS->TsCommonInfo.SetupTimer, TsSetupTimeOut,
// Initialize unused Rx Reorder List.
INIT_LIST_HEAD(&ieee->RxReorder_Unused_List);
//#ifdef TO_DO_LIST
- for(count = 0; count < REORDER_ENTRY_NUM; count++)
- {
+ for(count = 0; count < REORDER_ENTRY_NUM; count++) {
list_add_tail( &pRxReorderEntry->List,&ieee->RxReorder_Unused_List);
if(count == (REORDER_ENTRY_NUM-1))
break;
pRxReorderEntry = &ieee->RxReorderEntry[count+1];
}
//#endif
-
}
static void AdmitTS(struct ieee80211_device *ieee,
bool search_dir[4] = {0};
struct list_head *psearch_list; //FIXME
PTS_COMMON_INFO pRet = NULL;
- if(ieee->iw_mode == IW_MODE_MASTER) //ap mode
- {
- if(TxRxSelect == TX_DIR)
- {
+ if(ieee->iw_mode == IW_MODE_MASTER) { //ap mode
+ if(TxRxSelect == TX_DIR) {
search_dir[DIR_DOWN] = true;
search_dir[DIR_BI_DIR]= true;
- }
- else
- {
+ } else {
search_dir[DIR_UP] = true;
search_dir[DIR_BI_DIR]= true;
}
- }
- else if(ieee->iw_mode == IW_MODE_ADHOC)
- {
+ } else if(ieee->iw_mode == IW_MODE_ADHOC) {
if(TxRxSelect == TX_DIR)
search_dir[DIR_UP] = true;
else
search_dir[DIR_DOWN] = true;
- }
- else
- {
- if(TxRxSelect == TX_DIR)
- {
+ } else {
+ if(TxRxSelect == TX_DIR) {
search_dir[DIR_UP] = true;
search_dir[DIR_BI_DIR]= true;
search_dir[DIR_DIRECT]= true;
- }
- else
- {
+ } else {
search_dir[DIR_DOWN] = true;
search_dir[DIR_BI_DIR]= true;
search_dir[DIR_DIRECT]= true;
psearch_list = &ieee->Rx_TS_Admit_List;
//for(dir = DIR_UP; dir <= DIR_BI_DIR; dir++)
- for(dir = 0; dir <= DIR_BI_DIR; dir++)
- {
+ for(dir = 0; dir <= DIR_BI_DIR; dir++) {
if (!search_dir[dir])
continue;
list_for_each_entry(pRet, psearch_list, List){
// IEEE80211_DEBUG(IEEE80211_DL_TS, "ADD:%pM, TID:%d, dir:%d\n", pRet->Addr, pRet->TSpec.f.TSInfo.field.ucTSID, pRet->TSpec.f.TSInfo.field.ucDirection);
if (memcmp(pRet->Addr, Addr, 6) == 0)
if (pRet->TSpec.f.TSInfo.field.ucTSID == TID)
- if(pRet->TSpec.f.TSInfo.field.ucDirection == dir)
- {
+ if(pRet->TSpec.f.TSInfo.field.ucDirection == dir) {
// printk("Bingo! got it\n");
break;
}
-
}
if(&pRet->List != psearch_list)
break;
}
- if(&pRet->List != psearch_list){
+ if(&pRet->List != psearch_list)
return pRet ;
- }
else
return NULL;
}
// We do not build any TS for Broadcast or Multicast stream.
// So reject these kinds of search here.
//
- if (is_multicast_ether_addr(Addr))
- {
+ if (is_multicast_ether_addr(Addr)) {
IEEE80211_DEBUG(IEEE80211_DL_ERR, "get TS for Broadcast or Multicast\n");
return false;
}
- if (ieee->current_network.qos_data.supported == 0)
+ if (ieee->current_network.qos_data.supported == 0) {
UP = 0;
- else
- {
+ } else {
// In WMM case: we use 4 TID only
- if (!IsACValid(TID))
- {
+ if (!IsACValid(TID)) {
IEEE80211_DEBUG(IEEE80211_DL_ERR, " in %s(), TID(%d) is not valid\n", __func__, TID);
return false;
}
- switch (TID)
- {
+ switch (TID) {
case 0:
case 3:
UP = 0;
Addr,
UP,
TxRxSelect);
- if(*ppTS != NULL)
- {
+ if(*ppTS != NULL) {
return true;
- }
- else
- {
+ } else {
if (!bAddNewTs) {
IEEE80211_DEBUG(IEEE80211_DL_TS, "add new TS failed(tid:%d)\n", UP);
return false;
- }
- else
- {
+ } else {
//
// Create a new Traffic stream for current Tx/Rx
// This is for EDCA and WMM to add a new TS.
((TxRxSelect==TX_DIR)?DIR_DOWN:DIR_UP):
((TxRxSelect==TX_DIR)?DIR_UP:DIR_DOWN);
IEEE80211_DEBUG(IEEE80211_DL_TS, "to add Ts\n");
- if(!list_empty(pUnusedList))
- {
+ if(!list_empty(pUnusedList)) {
(*ppTS) = list_entry(pUnusedList->next, TS_COMMON_INFO, List);
list_del_init(&(*ppTS)->List);
- if(TxRxSelect==TX_DIR)
- {
+ if(TxRxSelect==TX_DIR) {
PTX_TS_RECORD tmp = container_of(*ppTS, TX_TS_RECORD, TsCommonInfo);
ResetTxTsEntry(tmp);
- }
- else{
+ } else {
PRX_TS_RECORD tmp = container_of(*ppTS, RX_TS_RECORD, TsCommonInfo);
ResetRxTsEntry(tmp);
}
// if there is DirectLink, we need to do additional operation here!!
return true;
- }
- else
- {
+ } else {
IEEE80211_DEBUG(IEEE80211_DL_ERR, "in function %s() There is not enough TS record to be used!!", __func__);
return false;
}
del_timer_sync(&pTs->InactTimer);
TsInitDelBA(ieee, pTs, TxRxSelect);
- if(TxRxSelect == RX_DIR)
- {
+ if(TxRxSelect == RX_DIR) {
//#ifdef TO_DO_LIST
PRX_REORDER_ENTRY pRxReorderEntry;
PRX_TS_RECORD pRxTS = (PRX_TS_RECORD)pTs;
if(timer_pending(&pRxTS->RxPktPendingTimer))
del_timer_sync(&pRxTS->RxPktPendingTimer);
- while(!list_empty(&pRxTS->RxPendingPktList))
- {
+ while(!list_empty(&pRxTS->RxPendingPktList)) {
spin_lock_irqsave(&(ieee->reorder_spinlock), flags);
//pRxReorderEntry = list_entry(&pRxTS->RxPendingPktList.prev,RX_REORDER_ENTRY,List);
pRxReorderEntry = (PRX_REORDER_ENTRY)list_entry(pRxTS->RxPendingPktList.prev,RX_REORDER_ENTRY,List);
{
int i = 0;
struct ieee80211_rxb *prxb = pRxReorderEntry->prxb;
- if (unlikely(!prxb))
- {
+ if (unlikely(!prxb)) {
spin_unlock_irqrestore(&(ieee->reorder_spinlock), flags);
return;
}
- for(i =0; i < prxb->nr_subframes; i++) {
+ for(i =0; i < prxb->nr_subframes; i++)
dev_kfree_skb(prxb->subframes[i]);
- }
+
kfree(prxb);
prxb = NULL;
}
}
//#endif
- }
- else
- {
+ } else {
PTX_TS_RECORD pTxTS = (PTX_TS_RECORD)pTs;
del_timer_sync(&pTxTS->TsAddBaTimer);
}
PTS_COMMON_INFO pTS, pTmpTS;
printk("===========>RemovePeerTS,%pM\n", Addr);
- list_for_each_entry_safe(pTS, pTmpTS, &ieee->Tx_TS_Pending_List, List)
- {
- if (memcmp(pTS->Addr, Addr, 6) == 0)
- {
+ list_for_each_entry_safe(pTS, pTmpTS, &ieee->Tx_TS_Pending_List, List) {
+ if (memcmp(pTS->Addr, Addr, 6) == 0) {
RemoveTsEntry(ieee, pTS, TX_DIR);
list_del_init(&pTS->List);
list_add_tail(&pTS->List, &ieee->Tx_TS_Unused_List);
}
}
- list_for_each_entry_safe(pTS, pTmpTS, &ieee->Tx_TS_Admit_List, List)
- {
- if (memcmp(pTS->Addr, Addr, 6) == 0)
- {
+ list_for_each_entry_safe(pTS, pTmpTS, &ieee->Tx_TS_Admit_List, List) {
+ if (memcmp(pTS->Addr, Addr, 6) == 0) {
printk("====>remove Tx_TS_admin_list\n");
RemoveTsEntry(ieee, pTS, TX_DIR);
list_del_init(&pTS->List);
}
}
- list_for_each_entry_safe(pTS, pTmpTS, &ieee->Rx_TS_Pending_List, List)
- {
- if (memcmp(pTS->Addr, Addr, 6) == 0)
- {
+ list_for_each_entry_safe(pTS, pTmpTS, &ieee->Rx_TS_Pending_List, List) {
+ if (memcmp(pTS->Addr, Addr, 6) == 0) {
RemoveTsEntry(ieee, pTS, RX_DIR);
list_del_init(&pTS->List);
list_add_tail(&pTS->List, &ieee->Rx_TS_Unused_List);
}
}
- list_for_each_entry_safe(pTS, pTmpTS, &ieee->Rx_TS_Admit_List, List)
- {
- if (memcmp(pTS->Addr, Addr, 6) == 0)
- {
+ list_for_each_entry_safe(pTS, pTmpTS, &ieee->Rx_TS_Admit_List, List) {
+ if (memcmp(pTS->Addr, Addr, 6) == 0) {
RemoveTsEntry(ieee, pTS, RX_DIR);
list_del_init(&pTS->List);
list_add_tail(&pTS->List, &ieee->Rx_TS_Unused_List);
{
PTS_COMMON_INFO pTS, pTmpTS;
- list_for_each_entry_safe(pTS, pTmpTS, &ieee->Tx_TS_Pending_List, List)
- {
+ list_for_each_entry_safe(pTS, pTmpTS, &ieee->Tx_TS_Pending_List, List) {
RemoveTsEntry(ieee, pTS, TX_DIR);
list_del_init(&pTS->List);
list_add_tail(&pTS->List, &ieee->Tx_TS_Unused_List);
}
- list_for_each_entry_safe(pTS, pTmpTS, &ieee->Tx_TS_Admit_List, List)
- {
+ list_for_each_entry_safe(pTS, pTmpTS, &ieee->Tx_TS_Admit_List, List) {
RemoveTsEntry(ieee, pTS, TX_DIR);
list_del_init(&pTS->List);
list_add_tail(&pTS->List, &ieee->Tx_TS_Unused_List);
}
- list_for_each_entry_safe(pTS, pTmpTS, &ieee->Rx_TS_Pending_List, List)
- {
+ list_for_each_entry_safe(pTS, pTmpTS, &ieee->Rx_TS_Pending_List, List) {
RemoveTsEntry(ieee, pTS, RX_DIR);
list_del_init(&pTS->List);
list_add_tail(&pTS->List, &ieee->Rx_TS_Unused_List);
}
- list_for_each_entry_safe(pTS, pTmpTS, &ieee->Rx_TS_Admit_List, List)
- {
+ list_for_each_entry_safe(pTS, pTmpTS, &ieee->Rx_TS_Admit_List, List) {
RemoveTsEntry(ieee, pTS, RX_DIR);
list_del_init(&pTS->List);
list_add_tail(&pTS->List, &ieee->Rx_TS_Unused_List);
void TsStartAddBaProcess(struct ieee80211_device *ieee, PTX_TS_RECORD pTxTS)
{
- if(!pTxTS->bAddBaReqInProgress)
- {
+ if(!pTxTS->bAddBaReqInProgress) {
pTxTS->bAddBaReqInProgress = true;
- if(pTxTS->bAddBaReqDelayed)
- {
+ if(pTxTS->bAddBaReqDelayed) {
IEEE80211_DEBUG(IEEE80211_DL_BA, "TsStartAddBaProcess(): Delayed Start ADDBA after 60 sec!!\n");
mod_timer(&pTxTS->TsAddBaTimer,
jiffies + msecs_to_jiffies(TS_ADDBA_DELAY));
- }
- else
- {
+ } else {
IEEE80211_DEBUG(IEEE80211_DL_BA,"TsStartAddBaProcess(): Immediately Start ADDBA now!!\n");
mod_timer(&pTxTS->TsAddBaTimer, jiffies+10); //set 10 ticks
}
- }
- else
+ } else {
IEEE80211_DEBUG(IEEE80211_DL_ERR, "%s()==>BA timer is already added\n", __func__);
+ }
}
#include <linux/proc_fs.h>
#include <linux/if_arp.h>
#include <linux/random.h>
-#include <asm/io.h>
+#include <linux/io.h>
#include "ieee80211/ieee80211.h"
#define RTL8192U
int write_nic_dword(struct net_device *dev, int x, u32 y);
void force_pci_posting(struct net_device *dev);
-void rtl8192_rtx_disable(struct net_device *);
-void rtl8192_rx_enable(struct net_device *);
-void rtl8192_tx_enable(struct net_device *);
+void rtl8192_rtx_disable(struct net_device *dev);
+void rtl8192_rx_enable(struct net_device *dev);
+void rtl8192_tx_enable(struct net_device *dev);
void rtl8192_disassociate(struct net_device *dev);
void rtl8185_set_rf_pins_enable(struct net_device *dev, u32 a);
* way, but there is no chance to set this as wep will not set
* group key in wext.
*/
- if (KEY_TYPE_WEP40 == ieee->pairwise_key_type ||
- KEY_TYPE_WEP104 == ieee->pairwise_key_type)
+ if (ieee->pairwise_key_type == KEY_TYPE_WEP40 ||
+ ieee->pairwise_key_type == KEY_TYPE_WEP104)
EnableHWSecurityConfig8192(dev);
}
/*update timing params*/
*/
encrypt = (network->capability & WLAN_CAPABILITY_PRIVACY) ||
(ieee->host_encrypt && crypt && crypt->ops &&
- (0 == strcmp(crypt->ops->name, "WEP")));
+ (strcmp(crypt->ops->name, "WEP") == 0));
/* simply judge */
if (encrypt && (wpa_ie_len == 0)) {
praddr = hdr->addr1;
/* Check if the received packet is acceptable. */
- bpacket_match_bssid = (IEEE80211_FTYPE_CTL != type) &&
+ bpacket_match_bssid = (type != IEEE80211_FTYPE_CTL) &&
(eqMacAddr(priv->ieee80211->current_network.bssid, (fc & IEEE80211_FCTL_TODS) ? hdr->addr1 : (fc & IEEE80211_FCTL_FROMDS) ? hdr->addr2 : hdr->addr3))
&& (!pstats->bHwError) && (!pstats->bCRC) && (!pstats->bICV);
bpacket_toself = bpacket_match_bssid &
struct ieee80211_device *ieee = priv->ieee80211;
SECR_value = SCR_TxEncEnable | SCR_RxDecEnable;
- if (((KEY_TYPE_WEP40 == ieee->pairwise_key_type) || (KEY_TYPE_WEP104 == ieee->pairwise_key_type)) && (priv->ieee80211->auth_mode != 2)) {
+ if (((ieee->pairwise_key_type == KEY_TYPE_WEP40) || (ieee->pairwise_key_type == KEY_TYPE_WEP104)) && (priv->ieee80211->auth_mode != 2)) {
SECR_value |= SCR_RxUseDK;
SECR_value |= SCR_TxUseDK;
} else if ((ieee->iw_mode == IW_MODE_ADHOC) && (ieee->pairwise_key_type & (KEY_TYPE_CCMP | KEY_TYPE_TKIP))) {
* Restore original EDCA according to the declaration of AP.
*/
if (priv->bcurrent_turbo_EDCA) {
+ u8 u1bAIFS;
+ u32 u4bAcParam, op_limit, cw_max, cw_min;
+
+ struct ieee80211_qos_parameters *qos_parameters = &priv->ieee80211->current_network.qos_data.parameters;
+ u8 mode = priv->ieee80211->mode;
+
+ /* For Each time updating EDCA parameter, reset EDCA turbo mode status. */
+ dm_init_edca_turbo(dev);
+
+ u1bAIFS = qos_parameters->aifs[0] * ((mode & (IEEE_G | IEEE_N_24G)) ? 9 : 20) + aSifsTime;
+
+ op_limit = (u32)le16_to_cpu(qos_parameters->tx_op_limit[0]);
+ cw_max = (u32)le16_to_cpu(qos_parameters->cw_max[0]);
+ cw_min = (u32)le16_to_cpu(qos_parameters->cw_min[0]);
+
+ op_limit <<= AC_PARAM_TXOP_LIMIT_OFFSET;
+ cw_max <<= AC_PARAM_ECW_MAX_OFFSET;
+ cw_min <<= AC_PARAM_ECW_MIN_OFFSET;
+ u1bAIFS <<= AC_PARAM_AIFS_OFFSET;
+
+ u4bAcParam = op_limit | cw_max | cw_min | u1bAIFS;
+ cpu_to_le32s(&u4bAcParam);
+
+ write_nic_dword(dev, EDCAPARA_BE, u4bAcParam);
+
+
+ /*
+ * Check ACM bit.
+ * If it is set, immediately set ACM control bit to downgrading AC for passing WMM testplan. Annie, 2005-12-13.
+ */
{
- u8 u1bAIFS;
- u32 u4bAcParam;
- struct ieee80211_qos_parameters *qos_parameters = &priv->ieee80211->current_network.qos_data.parameters;
- u8 mode = priv->ieee80211->mode;
-
- /* For Each time updating EDCA parameter, reset EDCA turbo mode status. */
- dm_init_edca_turbo(dev);
- u1bAIFS = qos_parameters->aifs[0] * ((mode&(IEEE_G|IEEE_N_24G)) ? 9 : 20) + aSifsTime;
- u4bAcParam = (((le16_to_cpu(qos_parameters->tx_op_limit[0])) << AC_PARAM_TXOP_LIMIT_OFFSET)|
- ((le16_to_cpu(qos_parameters->cw_max[0])) << AC_PARAM_ECW_MAX_OFFSET)|
- ((le16_to_cpu(qos_parameters->cw_min[0])) << AC_PARAM_ECW_MIN_OFFSET)|
- ((u32)u1bAIFS << AC_PARAM_AIFS_OFFSET));
- /*write_nic_dword(dev, WDCAPARA_ADD[i], u4bAcParam);*/
- write_nic_dword(dev, EDCAPARA_BE, u4bAcParam);
-
- /*
- * Check ACM bit.
- * If it is set, immediately set ACM control bit to downgrading AC for passing WMM testplan. Annie, 2005-12-13.
- */
- {
- /* TODO: Modified this part and try to set acm control in only 1 IO processing!! */
-
- PACI_AIFSN pAciAifsn = (PACI_AIFSN)&(qos_parameters->aifs[0]);
- u8 AcmCtrl;
-
- read_nic_byte(dev, AcmHwCtrl, &AcmCtrl);
-
- if (pAciAifsn->f.ACM) { /* ACM bit is 1. */
- AcmCtrl |= AcmHw_BeqEn;
- } else { /* ACM bit is 0. */
- AcmCtrl &= (~AcmHw_BeqEn);
- }
+ /* TODO: Modified this part and try to set acm control in only 1 IO processing!! */
- RT_TRACE(COMP_QOS, "SetHwReg8190pci(): [HW_VAR_ACM_CTRL] Write 0x%X\n", AcmCtrl);
- write_nic_byte(dev, AcmHwCtrl, AcmCtrl);
+ PACI_AIFSN pAciAifsn = (PACI_AIFSN)&(qos_parameters->aifs[0]);
+ u8 AcmCtrl;
+
+ read_nic_byte(dev, AcmHwCtrl, &AcmCtrl);
+
+ if (pAciAifsn->f.ACM) { /* ACM bit is 1. */
+ AcmCtrl |= AcmHw_BeqEn;
+ } else { /* ACM bit is 0. */
+ AcmCtrl &= (~AcmHw_BeqEn);
}
+
+ RT_TRACE(COMP_QOS, "SetHwReg8190pci(): [HW_VAR_ACM_CTRL] Write 0x%X\n", AcmCtrl);
+ write_nic_byte(dev, AcmHwCtrl, AcmCtrl);
}
priv->bcurrent_turbo_EDCA = false;
}
int r8712_generate_ie(struct registry_priv *pregistrypriv)
{
- int sz = 0, rateLen;
+ int sz = 0, rate_len;
struct wlan_bssid_ex *pdev_network = &pregistrypriv->dev_network;
u8 *ie = pdev_network->IEs;
pdev_network->Ssid.Ssid, &sz);
/*supported rates*/
set_supported_rate(pdev_network->rates, pregistrypriv->wireless_mode);
- rateLen = r8712_get_rateset_len(pdev_network->rates);
- if (rateLen > 8) {
+ rate_len = r8712_get_rateset_len(pdev_network->rates);
+ if (rate_len > 8) {
ie = r8712_set_ie(ie, _SUPPORTEDRATES_IE_, 8,
pdev_network->rates, &sz);
- ie = r8712_set_ie(ie, _EXT_SUPPORTEDRATES_IE_, (rateLen - 8),
+ ie = r8712_set_ie(ie, _EXT_SUPPORTEDRATES_IE_, (rate_len - 8),
(pdev_network->rates + 8), &sz);
- } else
+ } else {
ie = r8712_set_ie(ie, _SUPPORTEDRATES_IE_,
- rateLen, pdev_network->rates, &sz);
+ rate_len, pdev_network->rates, &sz);
+ }
/*DS parameter set*/
ie = r8712_set_ie(ie, _DSSET_IE_, 1,
(u8 *)&pdev_network->Configuration.DSConfig, &sz);
*/
static int wifi_test;
-module_param_string(ifname, ifname, sizeof(ifname), S_IRUGO | S_IWUSR);
+module_param_string(ifname, ifname, sizeof(ifname), 0644);
module_param(wifi_test, int, 0644);
module_param(initmac, charp, 0644);
module_param(video_mode, int, 0644);
*(__le16 *)(pbuf) &= (~cpu_to_le16(_FROM_DS_)); \
})
-#define get_tofr_ds(pframe) ((GetToDs(pframe) << 1) | GetFrDs(pframe))
-
+static inline unsigned char get_tofr_ds(unsigned char *pframe)
+{
+ return ((GetToDs(pframe) << 1) | GetFrDs(pframe));
+}
#define SetMFrag(pbuf) ({ \
*(__le16 *)(pbuf) |= cpu_to_le16(_MORE_FRAG_); \
/*
return the wlan_network with the matching addr
- Shall be calle under atomic context... to avoid possible racing condition...
+ Shall be called under atomic context... to avoid possible racing condition...
*/
struct wlan_network *_rtw_find_network(struct __queue *scanned_queue, u8 *addr)
{
/*
return the wlan_network with the matching addr
- Shall be calle under atomic context... to avoid possible racing condition...
+ Shall be called under atomic context... to avoid possible racing condition...
*/
struct wlan_network *rtw_find_network(struct __queue *scanned_queue, u8 *addr)
{
sq_final = ((u32)(src->PhyInfo.SignalQuality)+(u32)(dst->PhyInfo.SignalQuality)*4)/5;
rssi_final = (src->Rssi+dst->Rssi*4)/5;
} else {
- /* bss info not receving from the right channel, use the original RX signal infos */
+ /* bss info not receiving from the right channel, use the original RX signal infos */
ss_final = dst->PhyInfo.SignalStrength;
sq_final = dst->PhyInfo.SignalQuality;
rssi_final = dst->Rssi;
pnetwork->aid = 0;
pnetwork->join_res = 0;
- /* bss info not receving from the right channel */
+ /* bss info not receiving from the right channel */
if (pnetwork->network.PhyInfo.SignalQuality == 101)
pnetwork->network.PhyInfo.SignalQuality = 0;
} else {
pnetwork->last_scanned = jiffies;
- /* bss info not receving from the right channel */
+ /* bss info not receiving from the right channel */
if (pnetwork->network.PhyInfo.SignalQuality == 101)
pnetwork->network.PhyInfo.SignalQuality = 0;
pnetwork->last_scanned = jiffies;
- /* target.Reserved[0]== 1, means that scaned network is a bcn frame. */
+ /* target.Reserved[0]== 1, means that scanned network is a bcn frame. */
if ((pnetwork->network.IELength > target->IELength) && (target->Reserved[0] == 1))
update_ie = false;
/* Commented by Albert 2012/07/21 */
/* When doing the WPS, the wps_ie_len won't equal to 0 */
- /* And the Wi-Fi driver shouldn't allow the data packet to be tramsmitted. */
+ /* And the Wi-Fi driver shouldn't allow the data packet to be transmitted. */
if (padapter->securitypriv.wps_ie_len != 0) {
psta->ieee8021x_blocked = true;
padapter->securitypriv.wps_ie_len = 0;
/* for A-MPDU Rx reordering buffer control for bmc_sta & sta_info */
- /* if A-MPDU Rx is enabled, reseting rx_ordering_ctrl wstart_b(indicate_seq) to default value = 0xffff */
+ /* if A-MPDU Rx is enabled, resetting rx_ordering_ctrl wstart_b(indicate_seq) to default value = 0xffff */
/* todo: check if AP can send A-MPDU packets */
for (i = 0; i < 16 ; i++) {
/* preorder_ctrl = &precvpriv->recvreorder_ctrl[i]; */
rtw_update_ht_cap(padapter, cur_network->network.IEs, cur_network->network.IELength, (u8) cur_network->network.Configuration.DSConfig);
}
-/* Notes: the fucntion could be > passive_level (the same context as Rx tasklet) */
+/* Notes: the function could be > passive_level (the same context as Rx tasklet) */
/* pnetwork : returns from rtw_joinbss_event_callback */
/* ptarget_wlan: found from scanned_queue */
/* if join_res > 0, for (fw_state ==WIFI_STATION_STATE), we check if "ptarget_sta" & "ptarget_wlan" exist. */
}
- /* s5. Cancle assoc_timer */
+ /* s5. Cancel assoc_timer */
_cancel_timer(&pmlmepriv->assoc_timer, &timer_cancelled);
- RT_TRACE(_module_rtl871x_mlme_c_, _drv_info_, ("Cancle assoc_timer\n"));
+ RT_TRACE(_module_rtl871x_mlme_c_, _drv_info_, ("Cancel assoc_timer\n"));
} else{
RT_TRACE(_module_rtl871x_mlme_c_, _drv_err_, ("rtw_joinbss_event_callback err: fw_state:%x", get_fwstate(pmlmepriv)));
}
/*
-* _rtw_join_timeout_handler - Timeout/faliure handler for CMD JoinBss
+* _rtw_join_timeout_handler - Timeout/failure handler for CMD JoinBss
* @adapter: pointer to struct adapter structure
*/
void _rtw_join_timeout_handler (struct adapter *adapter)
}
/*
-* rtw_scan_timeout_handler - Timeout/Faliure handler for CMD SiteSurvey
+* rtw_scan_timeout_handler - Timeout/Failure handler for CMD SiteSurvey
* @adapter: pointer to struct adapter structure
*/
void rtw_scan_timeout_handler (struct adapter *adapter)
{
}
-/* the fucntion is at passive_level */
+/* the function is at passive_level */
void rtw_joinbss_reset(struct adapter *padapter)
{
u8 threshold;
}
}
-/* the fucntion is >= passive_level */
+/* the function is >= passive_level */
unsigned int rtw_restructure_ht_ie(struct adapter *padapter, u8 *in_ie, u8 *out_ie, uint in_len, uint *pout_len, u8 channel)
{
u32 ielen, out_len;
}
-/* the fucntion is > passive_level (in critical_section) */
+/* the function is > passive_level (in critical_section) */
void rtw_update_ht_cap(struct adapter *padapter, u8 *pie, uint ie_len, u8 channel)
{
u8 *p, max_ampdu_sz;
{
s32 wifiRssi = 0;
bool bWifiBusy = false, bWifiUnderBMode = false;
- static u8 nCCKLockCounter = 0;
+ static u8 nCCKLockCounter;
pBtCoexist->fBtcGet(pBtCoexist, BTC_GET_BL_WIFI_BUSY, &bWifiBusy);
pBtCoexist->fBtcGet(pBtCoexist, BTC_GET_S4_WIFI_RSSI, &wifiRssi);
static bool halbtc8723b1ant_IsWifiStatusChanged(PBTC_COEXIST pBtCoexist)
{
- static bool bPreWifiBusy = false, bPreUnder4way = false, bPreBtHsOn = false;
+ static bool bPreWifiBusy, bPreUnder4way, bPreBtHsOn;
bool bWifiBusy = false, bUnder4way = false, bBtHsOn = false;
bool bWifiConnected = false;
u32 wifiBw, wifiTrafficDir, faOfdm, faCck, wifiLinkStatus;
u8 wifiDot11Chnl, wifiHsChnl;
u32 fwVer = 0, btPatchVer = 0;
- static u8 PopReportIn10s = 0;
+ static u8 PopReportIn10s;
CL_SPRINTF(
cliBuf,
void EXhalbtc8723b1ant_Periodical(PBTC_COEXIST pBtCoexist)
{
- static u8 disVerInfoCnt = 0;
+ static u8 disVerInfoCnt;
u32 fwVer = 0, btPatchVer = 0;
BTC_PRINT(BTC_MSG_ALGORITHM, ALGO_TRACE, ("[BTCoex], ==========================Periodical ===========================\n"));
static bool halbtc8723b2ant_IsWifiStatusChanged(PBTC_COEXIST pBtCoexist)
{
- static bool bPreWifiBusy = false, bPreUnder4way = false, bPreBtHsOn = false;
+ static bool bPreWifiBusy, bPreUnder4way, bPreBtHsOn;
bool bWifiBusy = false, bUnder4way = false, bBtHsOn = false;
bool bWifiConnected = false;
void EXhalbtc8723b2ant_Periodical(PBTC_COEXIST pBtCoexist)
{
- static u8 disVerInfoCnt = 0;
+ static u8 disVerInfoCnt;
u32 fwVer = 0, btPatchVer = 0;
BTC_PRINT(BTC_MSG_ALGORITHM, ALGO_TRACE, ("[BTCoex], ==========================Periodical ===========================\n"));
static u8 halbtcoutsrc_GetWifiScanAPNum(struct adapter *padapter)
{
struct mlme_ext_priv *pmlmeext;
- static u8 scan_AP_num = 0;
+ static u8 scan_AP_num;
pmlmeext = &padapter->mlmeextpriv;
padapter->hal_data_sz = sizeof(struct hal_com_data);
padapter->HalData = vzalloc(padapter->hal_data_sz);
if (padapter->HalData == NULL) {
- DBG_8192C("cant not alloc memory for HAL DATA\n");
+ DBG_8192C("cannot alloc memory for HAL DATA\n");
return _FAIL;
}
}
/* Check input parameter. */
if (szStr == NULL || pu4bVal == NULL || pu4bMove == NULL) {
- DBG_871X("GetHexValueFromString(): Invalid inpur argumetns! szStr: %p, pu4bVal: %p, pu4bMove: %p\n", szStr, pu4bVal, pu4bMove);
+ DBG_871X("GetHexValueFromString(): Invalid input arguments! szStr: %p, pu4bVal: %p, pu4bMove: %p\n",
+ szStr, pu4bVal, pu4bMove);
return false;
}
#define AVG_THERMAL_NUM 8
#define IQK_Matrix_REG_NUM 8
-#define IQK_Matrix_Settings_NUM 14+24+21 /* Channels_2_4G_NUM + Channels_5G_20M_NUM + Channels_5G */
+#define IQK_Matrix_Settings_NUM (14 + 24 + 21) /* Channels_2_4G_NUM
+ * + Channels_5G_20M_NUM
+ * + Channels_5G
+ */
#define DM_Type_ByFW 0
#define DM_Type_ByDriver 1
if (!pDM_Odm->ForceEDCCA) {
if (pDM_Odm->RSSI_Min > pDM_Odm->AdapEn_RSSI)
- EDCCA_State = 1;
+ EDCCA_State = true;
else if (pDM_Odm->RSSI_Min < (pDM_Odm->AdapEn_RSSI - 5))
- EDCCA_State = 0;
+ EDCCA_State = false;
} else
- EDCCA_State = 1;
+ EDCCA_State = true;
if (
pDM_Odm->bLinked &&
)
);
- if (EDCCA_State == 1) {
+ if (EDCCA_State) {
Diff = IGI_target-(s8)IGI;
TH_L2H_dmc = pDM_Odm->TH_L2H_ini + Diff;
if (TH_L2H_dmc > 10)
{
PDM_ODM_T pDM_Odm = (PDM_ODM_T)pDM_VOID;
pDIG_T pDM_DigTable = &pDM_Odm->DM_DigTable;
- static bool bPaused = false;
+ static bool bPaused;
ODM_RT_TRACE(pDM_Odm, ODM_COMP_DIG, ODM_DBG_LOUD, ("odm_PauseDIG() =========>\n"));
#if DBG
#define ODM_RT_TRACE(pDM_Odm, comp, level, fmt)\
- if (\
- (comp & pDM_Odm->DebugComponents) &&\
- (level <= pDM_Odm->DebugLevel || level == ODM_DBG_SERIOUS)\
- ) {\
- RT_PRINTK fmt;\
- }
+ do {\
+ if (\
+ (comp & pDM_Odm->DebugComponents) &&\
+ (level <= pDM_Odm->DebugLevel ||\
+ level == ODM_DBG_SERIOUS)\
+ ) {\
+ RT_PRINTK fmt;\
+ } \
+ } while (0)
#define ODM_RT_TRACE_F(pDM_Odm, comp, level, fmt)\
- if (\
- (comp & pDM_Odm->DebugComponents) &&\
- (level <= pDM_Odm->DebugLevel)\
- ) {\
- RT_PRINTK fmt;\
- }
+ do {\
+ if (\
+ (comp & pDM_Odm->DebugComponents) &&\
+ (level <= pDM_Odm->DebugLevel)\
+ ) {\
+ RT_PRINTK fmt;\
+ } \
+ } while (0)
#define ODM_RT_ASSERT(pDM_Odm, expr, fmt)\
- if (!expr) {\
- DbgPrint("Assertion failed! %s at ......\n", #expr);\
- DbgPrint(\
- " ......%s,%s, line =%d\n",\
- __FILE__,\
- __func__,\
- __LINE__\
- );\
- RT_PRINTK fmt;\
- ASSERT(false);\
- }
+ do {\
+ if (!expr) {\
+ DbgPrint("Assertion failed! %s at ......\n", #expr);\
+ DbgPrint(\
+ " ......%s,%s, line =%d\n",\
+ __FILE__,\
+ __func__,\
+ __LINE__\
+ );\
+ RT_PRINTK fmt;\
+ ASSERT(false);\
+ } \
+ } while (0)
#define ODM_dbg_enter() { DbgPrint("==> %s\n", __func__); }
#define ODM_dbg_exit() { DbgPrint("<== %s\n", __func__); }
#define ODM_dbg_trace(str) { DbgPrint("%s:%s\n", __func__, str); }
#define ODM_PRINT_ADDR(pDM_Odm, comp, level, title_str, ptr)\
- if (\
- (comp & pDM_Odm->DebugComponents) &&\
- (level <= pDM_Odm->DebugLevel)\
- ) {\
- int __i;\
- u8 *__ptr = (u8 *)ptr;\
- DbgPrint("[ODM] ");\
- DbgPrint(title_str);\
- DbgPrint(" ");\
- for (__i = 0; __i < 6; __i++)\
- DbgPrint("%02X%s", __ptr[__i], (__i == 5) ? "" : "-");\
- DbgPrint("\n");\
- }
+ do {\
+ if (\
+ (comp & pDM_Odm->DebugComponents) &&\
+ (level <= pDM_Odm->DebugLevel)\
+ ) {\
+ int __i;\
+ u8 *__ptr = (u8 *)ptr;\
+ DbgPrint("[ODM] ");\
+ DbgPrint(title_str);\
+ DbgPrint(" ");\
+ for (__i = 0; __i < 6; __i++)\
+ DbgPrint("%02X%s", __ptr[__i], (__i == 5) ? "" : "-");\
+ DbgPrint("\n");\
+ } \
+ } while (0)
#else
#define ODM_RT_TRACE(pDM_Odm, comp, level, fmt) no_printk fmt
#define ODM_RT_TRACE_F(pDM_Odm, comp, level, fmt) no_printk fmt
efuse_addr = *pAddr;
if (efuse_addr >= efuse_max_available_len) {
- DBG_8192C("%s: addr(%d) over avaliable(%d)!!\n", __func__, efuse_addr, efuse_max_available_len);
+ DBG_8192C("%s: addr(%d) over available (%d)!!\n", __func__,
+ efuse_addr, efuse_max_available_len);
return false;
}
/* Config MAC */
/* */
rtStatus = phy_ConfigMACWithParaFile(Adapter, pszMACRegFile);
- if (rtStatus == _FAIL)
- {
+ if (rtStatus == _FAIL) {
ODM_ConfigMACWithHeaderFile(&pHalData->odmpriv);
rtStatus = _SUCCESS;
}
Adapter->registrypriv.RegEnableTxPowerLimit == 1 ||
(Adapter->registrypriv.RegEnableTxPowerLimit == 2 && pHalData->EEPROMRegulatory == 1)
) {
- if (PHY_ConfigRFWithPowerLimitTableParaFile(Adapter, pszRFTxPwrLmtFile) == _FAIL)
- {
+ if (PHY_ConfigRFWithPowerLimitTableParaFile(Adapter, pszRFTxPwrLmtFile) == _FAIL) {
if (HAL_STATUS_SUCCESS != ODM_ConfigRFWithHeaderFile(&pHalData->odmpriv, CONFIG_RF_TXPWR_LMT, (ODM_RF_RADIO_PATH_E)0))
rtStatus = _FAIL;
}
/* */
/* 1. Read PHY_REG.TXT BB INIT!! */
/* */
- if (phy_ConfigBBWithParaFile(Adapter, pszBBRegFile, CONFIG_BB_PHY_REG) == _FAIL)
- {
+ if (phy_ConfigBBWithParaFile(Adapter, pszBBRegFile, CONFIG_BB_PHY_REG) ==
+ _FAIL) {
if (HAL_STATUS_SUCCESS != ODM_ConfigBBWithHeaderFile(&pHalData->odmpriv, CONFIG_BB_PHY_REG))
rtStatus = _FAIL;
}
Adapter->registrypriv.RegEnableTxPowerByRate == 1 ||
(Adapter->registrypriv.RegEnableTxPowerByRate == 2 && pHalData->EEPROMRegulatory != 2)
) {
- if (phy_ConfigBBWithPgParaFile(Adapter, pszBBRegPgFile) == _FAIL)
- {
+ if (phy_ConfigBBWithPgParaFile(Adapter, pszBBRegPgFile) ==
+ _FAIL) {
if (HAL_STATUS_SUCCESS != ODM_ConfigBBWithHeaderFile(&pHalData->odmpriv, CONFIG_BB_PHY_REG_PG))
rtStatus = _FAIL;
}
/* */
/* 2. Read BB AGC table Initialization */
/* */
- if (phy_ConfigBBWithParaFile(Adapter, pszAGCTableFile, CONFIG_BB_AGC_TAB) == _FAIL)
- {
+ if (phy_ConfigBBWithParaFile(Adapter, pszAGCTableFile,
+ CONFIG_BB_AGC_TAB) == _FAIL) {
if (HAL_STATUS_SUCCESS != ODM_ConfigBBWithHeaderFile(&pHalData->odmpriv, CONFIG_BB_AGC_TAB))
rtStatus = _FAIL;
}
/*----Initialize RF fom connfiguration file----*/
switch (eRFPath) {
case RF_PATH_A:
- if (PHY_ConfigRFWithParaFile(Adapter, pszRadioAFile, eRFPath) == _FAIL)
- {
+ if (PHY_ConfigRFWithParaFile(Adapter, pszRadioAFile,
+ eRFPath) == _FAIL) {
if (HAL_STATUS_FAILURE == ODM_ConfigRFWithHeaderFile(&pHalData->odmpriv, CONFIG_RF_RADIO, (ODM_RF_RADIO_PATH_E)eRFPath))
rtStatus = _FAIL;
}
break;
case RF_PATH_B:
- if (PHY_ConfigRFWithParaFile(Adapter, pszRadioBFile, eRFPath) == _FAIL)
- {
+ if (PHY_ConfigRFWithParaFile(Adapter, pszRadioBFile,
+ eRFPath) == _FAIL) {
if (HAL_STATUS_FAILURE == ODM_ConfigRFWithHeaderFile(&pHalData->odmpriv, CONFIG_RF_RADIO, (ODM_RF_RADIO_PATH_E)eRFPath))
rtStatus = _FAIL;
}
/* 3 Configuration of Tx Power Tracking */
/* 3 ----------------------------------------------------------------- */
- if (PHY_ConfigRFWithTxPwrTrackParaFile(Adapter, pszTxPwrTrackFile) == _FAIL)
- {
+ if (PHY_ConfigRFWithTxPwrTrackParaFile(Adapter, pszTxPwrTrackFile) ==
+ _FAIL) {
ODM_ConfigRFWithTxPwrTrackHeaderFile(&pHalData->odmpriv);
}
u32 n = 0;
struct hal_com_data *pHalData = GET_HAL_DATA(padapter);
- while (pHalData->SdioTxOQTFreeSpace < agg_num)
- {
+ while (pHalData->SdioTxOQTFreeSpace < agg_num) {
if (
(padapter->bSurpriseRemoved == true) ||
(padapter->bDriverStopped == true)
struct xmit_priv *pxmitpriv = &padapter->xmitpriv;
struct dvobj_priv *pdvobjpriv = adapter_to_dvobj(padapter);
struct xmit_buf *pxmitbuf;
- struct adapter * pri_padapter = padapter;
+ struct adapter *pri_padapter = padapter;
s32 ret = 0;
u8 PageIdx = 0;
u32 deviceId;
pxmitbuf = NULL;
if (padapter->registrypriv.wifi_spec == 1) {
- for (idx = 0; idx<4; idx++)
+ for (idx = 0; idx < 4; idx++)
inx[idx] = pxmitpriv->wmm_para_seq[idx];
} else {
inx[0] = 0;
) {
if (pxmitbuf) {
/* pxmitbuf->priv_data will be NULL, and will crash here */
- if (pxmitbuf->len > 0 && pxmitbuf->priv_data) {
+ if (pxmitbuf->len > 0 &&
+ pxmitbuf->priv_data) {
struct xmit_frame *pframe;
- pframe = (struct xmit_frame*)pxmitbuf->priv_data;
+ pframe = (struct xmit_frame *)pxmitbuf->priv_data;
pframe->agg_num = k;
pxmitbuf->agg_num = k;
rtl8723b_update_txdesc(pframe, pframe->buf_addr);
if (pxmitbuf->len > 0) {
struct xmit_frame *pframe;
- pframe = (struct xmit_frame*)pxmitbuf->priv_data;
+ pframe = (struct xmit_frame *)pxmitbuf->priv_data;
pframe->agg_num = k;
pxmitbuf->agg_num = k;
rtl8723b_update_txdesc(pframe, pframe->buf_addr);
pxmitbuf->priv_data = NULL;
enqueue_pending_xmitbuf(pxmitpriv, pxmitbuf);
yield();
- }
- else
+ } else
rtw_free_xmitbuf(pxmitpriv, pxmitbuf);
pxmitbuf = NULL;
}
struct xmit_priv *pxmitpriv = &padapter->xmitpriv;
s32 err;
- if ((err = rtw_xmitframe_enqueue(padapter, pxmitframe)) != _SUCCESS) {
+ err = rtw_xmitframe_enqueue(padapter, pxmitframe);
+ if (err != _SUCCESS) {
rtw_free_xmitframe(pxmitpriv, pxmitframe);
pxmitpriv->tx_drop++;
#define DEFAULT_MAX_SCAN_AGE (15 * HZ)
#define DEFAULT_FTS 2346
-#define MAC_FMT "%02x:%02x:%02x:%02x:%02x:%02x"
-#define MAC_ARG(x) ((u8 *)(x))[0], ((u8 *)(x))[1], ((u8 *)(x))[2], ((u8 *)(x))[3], ((u8 *)(x))[4], ((u8 *)(x))[5]
-#define IP_FMT "%d.%d.%d.%d"
-#define IP_ARG(x) ((u8 *)(x))[0], ((u8 *)(x))[1], ((u8 *)(x))[2], ((u8 *)(x))[3]
+#define MAC_FMT "%pM"
+#define MAC_ARG(x) (x)
+#define IP_FMT "%pI4"
+#define IP_ARG(x) (x)
extern __inline int is_multicast_mac_addr(const u8 *addr)
{
}
#ifndef MAC_FMT
-#define MAC_FMT "%02x:%02x:%02x:%02x:%02x:%02x"
+#define MAC_FMT "%pM"
#endif
#ifndef MAC_ARG
-#define MAC_ARG(x) ((u8 *)(x))[0], ((u8 *)(x))[1], ((u8 *)(x))[2], ((u8 *)(x))[3], ((u8 *)(x))[4], ((u8 *)(x))[5]
+#define MAC_ARG(x) (x)
#endif
/* include <linux/smp_lock.h> */
#include <linux/netdevice.h>
#include <linux/skbuff.h>
- #include <asm/uaccess.h>
+ #include <linux/uaccess.h>
#include <asm/byteorder.h>
- #include <asm/atomic.h>
- #include <asm/io.h>
+ #include <linux/atomic.h>
+ #include <linux/io.h>
#include <linux/semaphore.h>
#include <linux/sem.h>
#include <linux/sched.h>
/* */
/* RF RL6052 Series API */
/* */
-void rtl8192c_RF_ChangeTxPath(struct adapter *Adapter,
- u16 DataRate);
-void rtl8192c_PHY_RF6052SetBandwidth(
- struct adapter * Adapter,
- enum CHANNEL_WIDTH Bandwidth);
-void rtl8192c_PHY_RF6052SetCckTxPower(
- struct adapter *Adapter,
- u8* pPowerlevel);
-void rtl8192c_PHY_RF6052SetOFDMTxPower(
- struct adapter *Adapter,
- u8* pPowerLevel,
- u8 Channel);
-int PHY_RF6052_Config8192C(struct adapter * Adapter );
+void rtl8192c_RF_ChangeTxPath(struct adapter *Adapter,
+ u16 DataRate);
+void rtl8192c_PHY_RF6052SetBandwidth(struct adapter *Adapter,
+ enum CHANNEL_WIDTH Bandwidth);
+void rtl8192c_PHY_RF6052SetCckTxPower(struct adapter *Adapter,
+ u8 *pPowerlevel);
+void rtl8192c_PHY_RF6052SetOFDMTxPower(struct adapter *Adapter,
+ u8 *pPowerLevel,
+ u8 Channel);
+int PHY_RF6052_Config8192C(struct adapter *Adapter);
/*--------------------------Exported Function prototype---------------------*/
#include <autoconf.h>
-
#define HAL_NAV_UPPER_UNIT_8723B 128 /* micro-second */
/* */
/* */
-/* 0x0000h ~ 0x00FFh System Configuration */
+/* 0x0000h ~ 0x00FFh System Configuration */
/* */
/* */
#define REG_RSV_CTRL_8723B 0x001C /* 3 Byte */
/* */
/* */
-/* 0x0100h ~ 0x01FFh MACTOP General Configuration */
+/* 0x0100h ~ 0x01FFh MACTOP General Configuration */
/* */
/* */
#define REG_C2HEVT_CMD_ID_8723B 0x01A0
/* */
/* */
-/* 0x0200h ~ 0x027Fh TXDMA Configuration */
+/* 0x0200h ~ 0x027Fh TXDMA Configuration */
/* */
/* */
/* */
/* */
-/* 0x0280h ~ 0x02FFh RXDMA Configuration */
+/* 0x0280h ~ 0x02FFh RXDMA Configuration */
/* */
/* */
#define REG_RXDMA_CONTROL_8723B 0x0286 /* Control the RX DMA. */
/* */
/* */
-/* 0x0300h ~ 0x03FFh PCIe */
+/* 0x0300h ~ 0x03FFh PCIe */
/* */
/* */
#define REG_PCIE_CTRL_REG_8723B 0x0300
/* */
/* */
-/* 0x0400h ~ 0x047Fh Protocol Configuration */
+/* 0x0400h ~ 0x047Fh Protocol Configuration */
/* */
/* */
#define REG_TXPKTBUF_BCNQ_BDNY_8723B 0x0424
/* */
/* */
-/* 0x0500h ~ 0x05FFh EDCA Configuration */
+/* 0x0500h ~ 0x05FFh EDCA Configuration */
/* */
/* */
#define REG_SECONDARY_CCA_CTRL_8723B 0x0577
/* */
/* */
-/* 0x0600h ~ 0x07FFh WMAC Configuration */
+/* 0x0600h ~ 0x07FFh WMAC Configuration */
/* */
/* */
-
/* */
/* SDIO Bus Specification */
/* */
/* */
#define SDIO_REG_HCPWM1_8723B 0x025 /* HCI Current Power Mode 1 */
-
/* */
-/* 8723 Regsiter Bit and Content definition */
+/* 8723 Register Bit and Content definition */
/* */
/* 2 HSISR */
/* */
/* */
-/* 0x0100h ~ 0x01FFh MACTOP General Configuration */
+/* 0x0100h ~ 0x01FFh MACTOP General Configuration */
/* */
/* */
-
/* */
/* */
-/* 0x0200h ~ 0x027Fh TXDMA Configuration */
+/* 0x0200h ~ 0x027Fh TXDMA Configuration */
/* */
/* */
/* */
/* */
-/* 0x0280h ~ 0x02FFh RXDMA Configuration */
+/* 0x0280h ~ 0x02FFh RXDMA Configuration */
/* */
/* */
#define BIT_USB_RXDMA_AGG_EN BIT(31)
/* */
/* */
-/* 0x0400h ~ 0x047Fh Protocol Configuration */
+/* 0x0400h ~ 0x047Fh Protocol Configuration */
/* */
/* */
/* */
/* */
-/* 0x0500h ~ 0x05FFh EDCA Configuration */
+/* 0x0500h ~ 0x05FFh EDCA Configuration */
/* */
/* */
/* */
/* */
-/* 0x0600h ~ 0x07FFh WMAC Configuration */
+/* 0x0600h ~ 0x07FFh WMAC Configuration */
/* */
/* */
#define EEPROM_RF_GAIN_OFFSET 0xC1
#define EEPROM_RF_GAIN_VAL 0x1F6
-
/* */
/* 8195 IMR/ISR bits (offset 0xB0, 8bits) */
/* */
#define IMR_BCNDMAINT3_8723B BIT23 /* Beacon DMA Interrupt 3 */
#define IMR_BCNDMAINT2_8723B BIT22 /* Beacon DMA Interrupt 2 */
#define IMR_BCNDMAINT1_8723B BIT21 /* Beacon DMA Interrupt 1 */
-#define IMR_BCNDOK7_8723B BIT20 /* Beacon Queue DMA OK Interrup 7 */
-#define IMR_BCNDOK6_8723B BIT19 /* Beacon Queue DMA OK Interrup 6 */
-#define IMR_BCNDOK5_8723B BIT18 /* Beacon Queue DMA OK Interrup 5 */
-#define IMR_BCNDOK4_8723B BIT17 /* Beacon Queue DMA OK Interrup 4 */
-#define IMR_BCNDOK3_8723B BIT16 /* Beacon Queue DMA OK Interrup 3 */
-#define IMR_BCNDOK2_8723B BIT15 /* Beacon Queue DMA OK Interrup 2 */
-#define IMR_BCNDOK1_8723B BIT14 /* Beacon Queue DMA OK Interrup 1 */
+#define IMR_BCNDOK7_8723B BIT20 /* Beacon Queue DMA OK Interrupt 7 */
+#define IMR_BCNDOK6_8723B BIT19 /* Beacon Queue DMA OK Interrupt 6 */
+#define IMR_BCNDOK5_8723B BIT18 /* Beacon Queue DMA OK Interrupt 5 */
+#define IMR_BCNDOK4_8723B BIT17 /* Beacon Queue DMA OK Interrupt 4 */
+#define IMR_BCNDOK3_8723B BIT16 /* Beacon Queue DMA OK Interrupt 3 */
+#define IMR_BCNDOK2_8723B BIT15 /* Beacon Queue DMA OK Interrupt 2 */
+#define IMR_BCNDOK1_8723B BIT14 /* Beacon Queue DMA OK Interrupt 1 */
#define IMR_ATIMEND_E_8723B BIT13 /* ATIM Window End Extension for Win7 */
#define IMR_TXERR_8723B BIT11 /* Tx Error Flag Interrupt Status, write 1 clear. */
#define IMR_RXERR_8723B BIT10 /* Rx Error Flag INT Status, Write 1 clear */
exit:
- kfree((u8 *)pwep);
+ kfree(pwep);
return ret;
}
ret = wpa_set_encryption(dev, param, param_len);
exit:
- kfree((u8 *)param);
+ kfree(param);
return ret;
}
if (copy_from_user(param, p->pointer, p->length))
{
- kfree((u8 *)param);
+ kfree(param);
ret = -EFAULT;
goto out;
}
if (ret == 0 && copy_to_user(p->pointer, param, p->length))
ret = -EFAULT;
- kfree((u8 *)param);
+ kfree(param);
out:
}
exit:
- kfree((u8 *)pwep);
+ kfree(pwep);
return ret;
if (copy_from_user(param, p->pointer, p->length))
{
- kfree((u8 *)param);
+ kfree(param);
ret = -EFAULT;
goto out;
}
ret = -EFAULT;
- kfree((u8 *)param);
+ kfree(param);
out:
DBG_871X(KERN_ERR "%s: (%d) addr = 0x%05x, val = 0x%x\n", __func__, *err, addr, v);
*err = 0;
- for (i = 0; i<SD_IO_TRY_CNT; i++)
+ for (i = 0; i < SD_IO_TRY_CNT; i++)
{
if (claim_needed) sdio_claim_host(func);
v = sdio_readl(func, addr, err);
}
}
- if (i ==SD_IO_TRY_CNT)
+ if (i == SD_IO_TRY_CNT)
DBG_871X(KERN_ERR "%s: FAIL!(%d) addr = 0x%05x, val = 0x%x, try_cnt =%d\n", __func__, *err, addr, v, i);
else
DBG_871X(KERN_ERR "%s: (%d) addr = 0x%05x, val = 0x%x, try_cnt =%d\n", __func__, *err, addr, v, i);
if (padapter->bSurpriseRemoved) {
/* DBG_871X(" %s (padapter->bSurpriseRemoved ||adapter->pwrctrlpriv.pnp_bstop_trx)!!!\n", __func__); */
- return ;
+ return;
}
func = psdio->func;
if (padapter->bSurpriseRemoved) {
/* DBG_871X(" %s (padapter->bSurpriseRemoved ||adapter->pwrctrlpriv.pnp_bstop_trx)!!!\n", __func__); */
- return ;
+ return;
}
func = psdio->func;
DBG_871X(KERN_ERR "%s: (%d) addr = 0x%05x val = 0x%08x\n", __func__, *err, addr, v);
*err = 0;
- for (i = 0; i<SD_IO_TRY_CNT; i++)
+ for (i = 0; i < SD_IO_TRY_CNT; i++)
{
if (claim_needed) sdio_claim_host(func);
sdio_writel(func, v, addr, err);
}
}
- if (i ==SD_IO_TRY_CNT)
+ if (i == SD_IO_TRY_CNT)
DBG_871X(KERN_ERR "%s: FAIL!(%d) addr = 0x%05x val = 0x%08x, try_cnt =%d\n", __func__, *err, addr, v, i);
else
DBG_871X(KERN_ERR "%s: (%d) addr = 0x%05x val = 0x%08x, try_cnt =%d\n", __func__, *err, addr, v, i);
func = psdio->func;
- if (unlikely((cnt == 1) || (cnt ==2)))
+ if (unlikely((cnt == 1) || (cnt == 2)))
{
int i;
u8 *pbuf = (u8 *)pdata;
*0 Success
*others Fail
*/
-s32 sd_read(struct intf_hdl * pintfhdl, u32 addr, u32 cnt, void *pdata)
+s32 sd_read(struct intf_hdl *pintfhdl, u32 addr, u32 cnt, void *pdata)
{
struct adapter *padapter;
struct dvobj_priv *psdiodev;
struct sdio_func *func;
u32 size;
- s32 err =-EPERM;
+ s32 err = -EPERM;
padapter = pintfhdl->padapter;
psdiodev = pintfhdl->pintf_dev;
}
func = psdio->func;
-/* size = sdio_align_size(func, cnt); */
+/* size = sdio_align_size(func, cnt); */
- if (unlikely((cnt == 1) || (cnt ==2)))
+ if (unlikely((cnt == 1) || (cnt == 2)))
{
int i;
u8 *pbuf = (u8 *)pdata;
PSDIO_DATA psdio;
struct sdio_func *func;
bool claim_needed;
- s32 err =-EPERM;
+ s32 err = -EPERM;
padapter = pintfhdl->padapter;
psdiodev = pintfhdl->pintf_dev;
*/
/*
- *Only these channels all allow active
- *scan on all world regulatory domains
+ * Only these channels all allow active
+ * scan on all world regulatory domains
*/
/* 2G chan 01 - chan 11 */
#define RTW_2GHZ_CH01_11 \
- REG_RULE(2412-10, 2462+10, 40, 0, 20, 0)
+ REG_RULE(2412 - 10, 2462 + 10, 40, 0, 20, 0)
/*
- *We enable active scan on these a case
- *by case basis by regulatory domain
+ * We enable active scan on these a case
+ * by case basis by regulatory domain
*/
/* 2G chan 12 - chan 13, PASSIV SCAN */
#define RTW_2GHZ_CH12_13 \
- REG_RULE(2467-10, 2472+10, 40, 0, 20, \
+ REG_RULE(2467 - 10, 2472 + 10, 40, 0, 20, \
NL80211_RRF_PASSIVE_SCAN)
/* 2G chan 14, PASSIVS SCAN, NO OFDM (B only) */
#define RTW_2GHZ_CH14 \
- REG_RULE(2484-10, 2484+10, 40, 0, 20, \
+ REG_RULE(2484 - 10, 2484 + 10, 40, 0, 20, \
NL80211_RRF_PASSIVE_SCAN | NL80211_RRF_NO_OFDM)
static const struct ieee80211_regdomain rtw_regdom_rd = {
.n_reg_rules = 3,
.alpha2 = "99",
.reg_rules = {
- RTW_2GHZ_CH01_11,
- RTW_2GHZ_CH12_13,
- }
+ RTW_2GHZ_CH01_11,
+ RTW_2GHZ_CH12_13,
+ }
};
static int rtw_ieee80211_channel_to_frequency(int chan, int band)
{
/* see 802.11 17.3.8.3.2 and Annex J
- * there are overlapping channel numbers in 5GHz and 2GHz bands */
+ * there are overlapping channel numbers in 5GHz and 2GHz bands
+ */
/* NL80211_BAND_2GHZ */
if (chan == 14)
u16 channel;
u32 freq;
- /* all channels disable */
+ /* all channels disable */
for (i = 0; i < NUM_NL80211_BANDS; i++) {
sband = wiphy->bands[i];
}
}
- /* channels apply by channel plans. */
+ /* channels apply by channel plans. */
for (i = 0; i < max_chan_nums; i++) {
channel = channel_set[i].ChannelNum;
freq =
ch = ieee80211_get_channel(wiphy, freq);
if (ch) {
- if (channel_set[i].ScanType == SCAN_PASSIVE) {
+ if (channel_set[i].ScanType == SCAN_PASSIVE)
ch->flags = IEEE80211_CHAN_NO_IR;
- }
- else {
+ else
ch->flags = 0;
- }
}
}
}
}
static void _rtw_regd_init_wiphy(struct rtw_regulatory *reg,
- struct wiphy *wiphy,
- void (*reg_notifier) (struct wiphy * wiphy,
- struct regulatory_request *
- request))
+ struct wiphy *wiphy,
+ void (*reg_notifier)(struct wiphy *wiphy,
+ struct
+ regulatory_request *
+ request))
{
const struct ieee80211_regdomain *regd;
}
int rtw_regd_init(struct adapter *padapter,
- void (*reg_notifier) (struct wiphy * wiphy,
+ void (*reg_notifier)(struct wiphy *wiphy,
struct regulatory_request *request))
{
- /* struct registry_priv *registrypriv = &padapter->registrypriv; */
struct wiphy *wiphy = padapter->rtw_wdev->wiphy;
+
_rtw_regd_init_wiphy(NULL, wiphy, reg_notifier);
return 0;
if (sendbytes > 8) {
memcpy(buf, inquiry_buf, 8);
- memcpy(buf + 8, inquiry_string, sendbytes - 8);
+ strncpy(buf + 8, inquiry_string, sendbytes - 8);
if (pro_formatter_flag) {
/* Additional Length */
buf[4] = 0x33;
rtsx_write_register(chip, SD_DCMPS_CTL, DCMPS_CHANGE, 0);
rtsx_write_register(chip, SD_VP_CTL, PHASE_CHANGE, 0);
- wait_timeout(10);
+ mdelay(10);
sd_reset_dcm(chip, tune_dir);
return STATUS_FAIL;
}
return STATUS_FAIL;
}
- wait_timeout(50);
+ mdelay(50);
}
if (chip->asic_code) {
reg = 0;
rtsx_read_register(chip, XD_CTL, ®);
if (reg & (XD_ECC1_ERROR | XD_ECC2_ERROR)) {
- wait_timeout(100);
+ mdelay(100);
if (detect_card_cd(chip,
XD_CARD) != STATUS_SUCCESS) {
}
sm750_set_current_gate(reg);
- }
+ }
}
unsigned int ddk750_get_vm_size(void)
sm750_set_current_gate(reg);
if (sm750_get_chip_type() != SM750LE) {
- /* set panel pll and graphic mode via mmio_88 */
+ /* set panel pll and graphic mode via mmio_88 */
reg = peek32(VGA_CONFIGURATION);
reg |= (VGA_CONFIGURATION_PLL | VGA_CONFIGURATION_MODE);
poke32(VGA_CONFIGURATION, reg);
* M = {1,...,255}
* N = {2,...,15}
*/
-unsigned int sm750_calc_pll_value(unsigned int request_orig, struct pll_value *pll)
+unsigned int sm750_calc_pll_value(unsigned int request_orig,
+ struct pll_value *pll)
{
/*
* as sm750 register definition,
#endif
};
-int dviInit(
- unsigned char edgeSelect,
- unsigned char busSelect,
- unsigned char dualEdgeClkSelect,
- unsigned char hsyncEnable,
- unsigned char vsyncEnable,
- unsigned char deskewEnable,
- unsigned char deskewSetting,
- unsigned char continuousSyncEnable,
- unsigned char pllFilterEnable,
- unsigned char pllFilterValue
- )
+int dviInit(unsigned char edgeSelect,
+ unsigned char busSelect,
+ unsigned char dualEdgeClkSelect,
+ unsigned char hsyncEnable,
+ unsigned char vsyncEnable,
+ unsigned char deskewEnable,
+ unsigned char deskewSetting,
+ unsigned char continuousSyncEnable,
+ unsigned char pllFilterEnable,
+ unsigned char pllFilterValue)
{
dvi_ctrl_device_t *pCurrentDviCtrl;
pCurrentDviCtrl = g_dcftSupportedDviController;
if (pCurrentDviCtrl->pfnInit) {
- return pCurrentDviCtrl->pfnInit(edgeSelect, busSelect, dualEdgeClkSelect, hsyncEnable,
- vsyncEnable, deskewEnable, deskewSetting, continuousSyncEnable,
- pllFilterEnable, pllFilterValue);
+ return pCurrentDviCtrl->pfnInit(edgeSelect,
+ busSelect,
+ dualEdgeClkSelect,
+ hsyncEnable,
+ vsyncEnable,
+ deskewEnable,
+ deskewSetting,
+ continuousSyncEnable,
+ pllFilterEnable,
+ pllFilterValue);
}
return -1; /* error */
}
/* dvi chip stuffs structros */
-typedef long (*PFN_DVICTRL_INIT)(
- unsigned char edgeSelect,
- unsigned char busSelect,
- unsigned char dualEdgeClkSelect,
- unsigned char hsyncEnable,
- unsigned char vsyncEnable,
- unsigned char deskewEnable,
- unsigned char deskewSetting,
- unsigned char continuousSyncEnable,
- unsigned char pllFilterEnable,
- unsigned char pllFilterValue);
+typedef long (*PFN_DVICTRL_INIT)(unsigned char edgeSelect,
+ unsigned char busSelect,
+ unsigned char dualEdgeClkSelect,
+ unsigned char hsyncEnable,
+ unsigned char vsyncEnable,
+ unsigned char deskewEnable,
+ unsigned char deskewSetting,
+ unsigned char continuousSyncEnable,
+ unsigned char pllFilterEnable,
+ unsigned char pllFilterValue);
typedef void (*PFN_DVICTRL_RESETCHIP)(void);
typedef char* (*PFN_DVICTRL_GETCHIPSTRING)(void);
#define DVI_CTRL_SII164
/* dvi functions prototype */
-int dviInit(
- unsigned char edgeSelect,
- unsigned char busSelect,
- unsigned char dualEdgeClkSelect,
- unsigned char hsyncEnable,
- unsigned char vsyncEnable,
- unsigned char deskewEnable,
- unsigned char deskewSetting,
- unsigned char continuousSyncEnable,
- unsigned char pllFilterEnable,
- unsigned char pllFilterValue
-);
+int dviInit(unsigned char edgeSelect,
+ unsigned char busSelect,
+ unsigned char dualEdgeClkSelect,
+ unsigned char hsyncEnable,
+ unsigned char vsyncEnable,
+ unsigned char deskewEnable,
+ unsigned char deskewSetting,
+ unsigned char continuousSyncEnable,
+ unsigned char pllFilterEnable,
+ unsigned char pllFilterValue);
#endif
#define MAX_HWI2C_FIFO 16
#define HWI2C_WAIT_TIMEOUT 0xF0000
-int sm750_hw_i2c_init(
-unsigned char bus_speed_mode
-)
+int sm750_hw_i2c_init(unsigned char bus_speed_mode)
{
unsigned int value;
* Return Value:
* Total number of bytes those are actually written.
*/
-static unsigned int hw_i2c_write_data(
- unsigned char addr,
- unsigned int length,
- unsigned char *buf
-)
+static unsigned int hw_i2c_write_data(unsigned char addr,
+ unsigned int length,
+ unsigned char *buf)
{
unsigned char count, i;
unsigned int total_bytes = 0;
* Return Value:
* Total number of actual bytes read from the slave device
*/
-static unsigned int hw_i2c_read_data(
- unsigned char addr,
- unsigned int length,
- unsigned char *buf
-)
+static unsigned int hw_i2c_read_data(unsigned char addr,
+ unsigned int length,
+ unsigned char *buf)
{
unsigned char count, i;
unsigned int total_bytes = 0;
* Return Value:
* Register value
*/
-unsigned char sm750_hw_i2c_read_reg(
- unsigned char addr,
- unsigned char reg
-)
+unsigned char sm750_hw_i2c_read_reg(unsigned char addr, unsigned char reg)
{
unsigned char value = 0xFF;
* 0 - Success
* -1 - Fail
*/
-int sm750_hw_i2c_write_reg(
- unsigned char addr,
- unsigned char reg,
- unsigned char data
-)
+int sm750_hw_i2c_write_reg(unsigned char addr,
+ unsigned char reg,
+ unsigned char data)
{
unsigned char value[2];
* 0 - Success
* -1 - Fail.
*/
-long sii164InitChip(
- unsigned char edgeSelect,
- unsigned char busSelect,
- unsigned char dualEdgeClkSelect,
- unsigned char hsyncEnable,
- unsigned char vsyncEnable,
- unsigned char deskewEnable,
- unsigned char deskewSetting,
- unsigned char continuousSyncEnable,
- unsigned char pllFilterEnable,
- unsigned char pllFilterValue
-)
+long sii164InitChip(unsigned char edgeSelect,
+ unsigned char busSelect,
+ unsigned char dualEdgeClkSelect,
+ unsigned char hsyncEnable,
+ unsigned char vsyncEnable,
+ unsigned char deskewEnable,
+ unsigned char deskewSetting,
+ unsigned char continuousSyncEnable,
+ unsigned char pllFilterEnable,
+ unsigned char pllFilterValue)
{
unsigned char config;
sii164SetPower(1);
}
-
/*
* sii164GetChipString
* This function returns a char string name of the current DVI Controller chip.
return gDviCtrlChipName;
}
-
/*
* sii164SetPower
* This function sets the power configuration of the DVI Controller Chip.
* Input:
* powerUp - Flag to set the power down or up
*/
-void sii164SetPower(
- unsigned char powerUp
-)
+void sii164SetPower(unsigned char powerUp)
{
unsigned char config;
}
}
-
/*
* sii164SelectHotPlugDetectionMode
* This function selects the mode of the hot plug detection.
*/
-static void sii164SelectHotPlugDetectionMode(
- sii164_hot_plug_mode_t hotPlugMode
-)
+static void sii164SelectHotPlugDetectionMode(sii164_hot_plug_mode_t hotPlugMode)
{
unsigned char detectReg;
- detectReg = i2cReadReg(SII164_I2C_ADDRESS, SII164_DETECT) & ~SII164_DETECT_MONITOR_SENSE_OUTPUT_FLAG;
+ detectReg = i2cReadReg(SII164_I2C_ADDRESS, SII164_DETECT) &
+ ~SII164_DETECT_MONITOR_SENSE_OUTPUT_FLAG;
switch (hotPlugMode) {
case SII164_HOTPLUG_DISABLE:
detectReg |= SII164_DETECT_MONITOR_SENSE_OUTPUT_HIGH;
*
* enableHotPlug - Enable (=1) / disable (=0) Hot Plug detection
*/
-void sii164EnableHotPlugDetection(
- unsigned char enableHotPlug
-)
+void sii164EnableHotPlugDetection(unsigned char enableHotPlug)
{
unsigned char detectReg;
{
unsigned char hotPlugValue;
- hotPlugValue = i2cReadReg(SII164_I2C_ADDRESS, SII164_DETECT) & SII164_DETECT_HOT_PLUG_STATUS_MASK;
+ hotPlugValue = i2cReadReg(SII164_I2C_ADDRESS, SII164_DETECT) &
+ SII164_DETECT_HOT_PLUG_STATUS_MASK;
if (hotPlugValue == SII164_DETECT_HOT_PLUG_STATUS_ON)
return 1;
else
{
unsigned char detectReg;
- detectReg = i2cReadReg(SII164_I2C_ADDRESS, SII164_DETECT) & SII164_DETECT_MONITOR_STATE_MASK;
+ detectReg = i2cReadReg(SII164_I2C_ADDRESS, SII164_DETECT) &
+ SII164_DETECT_MONITOR_STATE_MASK;
if (detectReg == SII164_DETECT_MONITOR_STATE_CHANGE)
return 1;
else
/* Clear the MDI interrupt */
detectReg = i2cReadReg(SII164_I2C_ADDRESS, SII164_DETECT);
- i2cWriteReg(SII164_I2C_ADDRESS, SII164_DETECT, detectReg | SII164_DETECT_MONITOR_STATE_CLEAR);
+ i2cWriteReg(SII164_I2C_ADDRESS, SII164_DETECT,
+ detectReg | SII164_DETECT_MONITOR_STATE_CLEAR);
}
#endif
/* Silicon Image SiI164 chip prototype */
-long sii164InitChip(
- unsigned char edgeSelect,
- unsigned char busSelect,
- unsigned char dualEdgeClkSelect,
- unsigned char hsyncEnable,
- unsigned char vsyncEnable,
- unsigned char deskewEnable,
- unsigned char deskewSetting,
- unsigned char continuousSyncEnable,
- unsigned char pllFilterEnable,
- unsigned char pllFilterValue
-);
+long sii164InitChip(unsigned char edgeSelect,
+ unsigned char busSelect,
+ unsigned char dualEdgeClkSelect,
+ unsigned char hsyncEnable,
+ unsigned char vsyncEnable,
+ unsigned char deskewEnable,
+ unsigned char deskewSetting,
+ unsigned char continuousSyncEnable,
+ unsigned char pllFilterEnable,
+ unsigned char pllFilterValue);
unsigned short sii164GetVendorID(void);
unsigned short sii164GetDeviceID(void);
* -1 - Fail to initialize the i2c
* 0 - Success
*/
-static long sm750le_i2c_init(unsigned char clk_gpio,
- unsigned char data_gpio)
+static long sm750le_i2c_init(unsigned char clk_gpio, unsigned char data_gpio)
{
int i;
* -1 - Fail to initialize the i2c
* 0 - Success
*/
-long sm750_sw_i2c_init(
- unsigned char clk_gpio,
- unsigned char data_gpio
-)
+long sm750_sw_i2c_init(unsigned char clk_gpio, unsigned char data_gpio)
{
int i;
* Return Value:
* Register value
*/
-unsigned char sm750_sw_i2c_read_reg(
- unsigned char addr,
- unsigned char reg
-)
+unsigned char sm750_sw_i2c_read_reg(unsigned char addr, unsigned char reg)
{
unsigned char data;
* 0 - Success
* -1 - Fail
*/
-long sm750_sw_i2c_write_reg(
- unsigned char addr,
- unsigned char reg,
- unsigned char data
-)
+long sm750_sw_i2c_write_reg(unsigned char addr,
+ unsigned char reg,
+ unsigned char data)
{
long ret = 0;
* -1 - Fail to initialize the i2c
* 0 - Success
*/
-long sm750_sw_i2c_init(
- unsigned char clk_gpio,
- unsigned char data_gpio
-);
+long sm750_sw_i2c_init(unsigned char clk_gpio, unsigned char data_gpio);
/*
* This function reads the slave device's register
* Return Value:
* Register value
*/
-unsigned char sm750_sw_i2c_read_reg(
- unsigned char addr,
- unsigned char reg
-);
+unsigned char sm750_sw_i2c_read_reg(unsigned char addr, unsigned char reg);
/*
* This function writes a value to the slave device's register
* 0 - Success
* -1 - Fail
*/
-long sm750_sw_i2c_write_reg(
- unsigned char addr,
- unsigned char reg,
- unsigned char data
-);
+long sm750_sw_i2c_write_reg(unsigned char addr,
+ unsigned char reg,
+ unsigned char data);
#endif /* _SWI2C_H_ */
static int g_noaccel;
static int g_nomtrr;
static const char *g_fbmode[] = {NULL, NULL};
-static const char *g_def_fbmode = "800x600-16@60";
+static const char *g_def_fbmode = "1024x768-32@60";
static char *g_settings;
static int g_dualview;
static char *g_option;
cursor = &crtc->cursor;
if (fbcursor->image.width > cursor->maxW ||
- fbcursor->image.height > cursor->maxH ||
- fbcursor->image.depth > 1) {
+ fbcursor->image.height > cursor->maxH ||
+ fbcursor->image.depth > 1) {
return -ENXIO;
}
sm750_hw_cursor_disable(cursor);
if (fbcursor->set & FB_CUR_SETSIZE)
sm750_hw_cursor_setSize(cursor,
- fbcursor->image.width,
- fbcursor->image.height);
+ fbcursor->image.width,
+ fbcursor->image.height);
if (fbcursor->set & FB_CUR_SETPOS)
sm750_hw_cursor_setPos(cursor,
- fbcursor->image.dx - info->var.xoffset,
- fbcursor->image.dy - info->var.yoffset);
+ fbcursor->image.dx - info->var.xoffset,
+ fbcursor->image.dy - info->var.yoffset);
if (fbcursor->set & FB_CUR_SETCMAP) {
/* get the 16bit color of kernel means */
u16 fg, bg;
fg = ((info->cmap.red[fbcursor->image.fg_color] & 0xf800)) |
- ((info->cmap.green[fbcursor->image.fg_color] & 0xfc00) >> 5) |
- ((info->cmap.blue[fbcursor->image.fg_color] & 0xf800) >> 11);
+ ((info->cmap.green[fbcursor->image.fg_color] & 0xfc00) >> 5) |
+ ((info->cmap.blue[fbcursor->image.fg_color] & 0xf800) >> 11);
bg = ((info->cmap.red[fbcursor->image.bg_color] & 0xf800)) |
- ((info->cmap.green[fbcursor->image.bg_color] & 0xfc00) >> 5) |
- ((info->cmap.blue[fbcursor->image.bg_color] & 0xf800) >> 11);
+ ((info->cmap.green[fbcursor->image.bg_color] & 0xfc00) >> 5) |
+ ((info->cmap.blue[fbcursor->image.bg_color] & 0xf800) >> 11);
sm750_hw_cursor_setColor(cursor, fg, bg);
}
if (fbcursor->set & (FB_CUR_SETSHAPE | FB_CUR_SETIMAGE)) {
sm750_hw_cursor_setData(cursor,
- fbcursor->rop,
- fbcursor->image.data,
- fbcursor->mask);
+ fbcursor->rop,
+ fbcursor->image.data,
+ fbcursor->mask);
}
if (fbcursor->enable)
rop = (region->rop != ROP_COPY) ? HW_ROP2_XOR : HW_ROP2_COPY;
/*
- * If not use spin_lock,system will die if user load driver
+ * If not use spin_lock, system will die if user load driver
* and immediately unload driver frequently (dual)
+ * since they fb_count could change during the lifetime of
+ * this lock, we are holding it for all cases.
*/
- if (sm750_dev->fb_count > 1)
- spin_lock(&sm750_dev->slock);
+ spin_lock(&sm750_dev->slock);
sm750_dev->accel.de_fillrect(&sm750_dev->accel,
base, pitch, Bpp,
region->dx, region->dy,
region->width, region->height,
color, rop);
- if (sm750_dev->fb_count > 1)
- spin_unlock(&sm750_dev->slock);
+ spin_unlock(&sm750_dev->slock);
}
static void lynxfb_ops_copyarea(struct fb_info *info,
/*
* If not use spin_lock, system will die if user load driver
* and immediately unload driver frequently (dual)
+ * since they fb_count could change during the lifetime of
+ * this lock, we are holding it for all cases.
*/
- if (sm750_dev->fb_count > 1)
- spin_lock(&sm750_dev->slock);
+ spin_lock(&sm750_dev->slock);
sm750_dev->accel.de_copyarea(&sm750_dev->accel,
base, pitch, region->sx, region->sy,
base, pitch, Bpp, region->dx, region->dy,
region->width, region->height,
HW_ROP2_COPY);
- if (sm750_dev->fb_count > 1)
- spin_unlock(&sm750_dev->slock);
+ spin_unlock(&sm750_dev->slock);
}
static void lynxfb_ops_imageblit(struct fb_info *info,
/*
* If not use spin_lock, system will die if user load driver
* and immediately unload driver frequently (dual)
+ * since they fb_count could change during the lifetime of
+ * this lock, we are holding it for all cases.
*/
- if (sm750_dev->fb_count > 1)
- spin_lock(&sm750_dev->slock);
+ spin_lock(&sm750_dev->slock);
sm750_dev->accel.de_imageblit(&sm750_dev->accel,
image->data, image->width >> 3, 0,
image->dx, image->dy,
image->width, image->height,
fgcol, bgcol, HW_ROP2_COPY);
- if (sm750_dev->fb_count > 1)
- spin_unlock(&sm750_dev->slock);
+ spin_unlock(&sm750_dev->slock);
}
static int lynxfb_ops_pan_display(struct fb_var_screeninfo *var,
int hw_sm750_deWait(void);
int hw_sm750le_deWait(void);
-int hw_sm750_output_setMode(struct lynxfb_output*, struct fb_var_screeninfo*,
- struct fb_fix_screeninfo*);
-int hw_sm750_crtc_checkMode(struct lynxfb_crtc*, struct fb_var_screeninfo*);
-int hw_sm750_crtc_setMode(struct lynxfb_crtc*, struct fb_var_screeninfo*,
- struct fb_fix_screeninfo*);
-int hw_sm750_setColReg(struct lynxfb_crtc*, ushort, ushort, ushort, ushort);
-int hw_sm750_setBLANK(struct lynxfb_output*, int);
-int hw_sm750le_setBLANK(struct lynxfb_output*, int);
+int hw_sm750_output_setMode(struct lynxfb_output *output,
+ struct fb_var_screeninfo *var,
+ struct fb_fix_screeninfo *fix);
+
+int hw_sm750_crtc_checkMode(struct lynxfb_crtc *crtc,
+ struct fb_var_screeninfo *var);
+
+int hw_sm750_crtc_setMode(struct lynxfb_crtc *crtc,
+ struct fb_var_screeninfo *var,
+ struct fb_fix_screeninfo *fix);
+
+int hw_sm750_setColReg(struct lynxfb_crtc *crtc, ushort index,
+ ushort red, ushort green, ushort blue);
+
+int hw_sm750_setBLANK(struct lynxfb_output *output, int blank);
+int hw_sm750le_setBLANK(struct lynxfb_output *output, int blank);
int hw_sm750_pan_display(struct lynxfb_crtc *crtc,
const struct fb_var_screeninfo *var,
const struct fb_info *info);
/* dpr1c */
reg = 0x3;
- clr = DE_STRETCH_FORMAT_PATTERN_XY | DE_STRETCH_FORMAT_PATTERN_Y_MASK |
- DE_STRETCH_FORMAT_PATTERN_X_MASK |
- DE_STRETCH_FORMAT_ADDRESSING_MASK |
- DE_STRETCH_FORMAT_SOURCE_HEIGHT_MASK;
+ clr = DE_STRETCH_FORMAT_PATTERN_XY |
+ DE_STRETCH_FORMAT_PATTERN_Y_MASK |
+ DE_STRETCH_FORMAT_PATTERN_X_MASK |
+ DE_STRETCH_FORMAT_ADDRESSING_MASK |
+ DE_STRETCH_FORMAT_SOURCE_HEIGHT_MASK;
/* DE_STRETCH bpp format need be initialized in setMode routine */
write_dpr(accel, DE_STRETCH_FORMAT,
}
int sm750_hw_fillrect(struct lynx_accel *accel,
- u32 base, u32 pitch, u32 Bpp,
- u32 x, u32 y, u32 width, u32 height,
- u32 color, u32 rop)
+ u32 base, u32 pitch, u32 Bpp,
+ u32 x, u32 y, u32 width, u32 height,
+ u32 color, u32 rop)
{
u32 deCtrl;
poke32(HWC_ADDRESS, 0);
}
-void sm750_hw_cursor_setSize(struct lynx_cursor *cursor,
- int w, int h)
+void sm750_hw_cursor_setSize(struct lynx_cursor *cursor, int w, int h)
{
cursor->w = w;
cursor->h = h;
}
-void sm750_hw_cursor_setPos(struct lynx_cursor *cursor,
- int x, int y)
+void sm750_hw_cursor_setPos(struct lynx_cursor *cursor, int x, int y)
{
u32 reg;
poke32(HWC_LOCATION, reg);
}
-void sm750_hw_cursor_setColor(struct lynx_cursor *cursor,
- u32 fg, u32 bg)
+void sm750_hw_cursor_setColor(struct lynx_cursor *cursor, u32 fg, u32 bg)
{
u32 reg = (fg << HWC_COLOR_12_2_RGB565_SHIFT) &
HWC_COLOR_12_2_RGB565_MASK;
poke32(HWC_COLOR_3, 0xffe0);
}
-void sm750_hw_cursor_setData(struct lynx_cursor *cursor,
- u16 rop, const u8 *pcol, const u8 *pmsk)
+void sm750_hw_cursor_setData(struct lynx_cursor *cursor, u16 rop,
+ const u8 *pcol, const u8 *pmsk)
{
int i, j, count, pitch, offset;
u8 color, mask, opr;
}
-void sm750_hw_cursor_setData2(struct lynx_cursor *cursor,
- u16 rop, const u8 *pcol, const u8 *pmsk)
+void sm750_hw_cursor_setData2(struct lynx_cursor *cursor, u16 rop,
+ const u8 *pcol, const u8 *pmsk)
{
int i, j, count, pitch, offset;
u8 color, mask;
kobjects.o \
selection.o \
serialio.o \
+ spk_ttyio.o \
synth.o \
thread.o \
varhandlers.o
goto oops;
if (ch == 8) {
u16 wch;
+
if (num == 0)
return -1;
wch = goto_buf[--num];
speakup_bs(vc);
} else {
u16 d = param->c;
+
speakup_con_write(vc, &d, 1);
}
break;
static int spk_serial_out(struct spk_synth *in_synth, const char ch);
static void spk_serial_send_xchar(char ch);
static void spk_serial_tiocmset(unsigned int set, unsigned int clear);
+static unsigned char spk_serial_in(void);
+static unsigned char spk_serial_in_nowait(void);
+static void spk_serial_flush_buffer(void);
struct spk_io_ops spk_serial_io_ops = {
.synth_out = spk_serial_out,
.send_xchar = spk_serial_send_xchar,
.tiocmset = spk_serial_tiocmset,
+ .synth_in = spk_serial_in,
+ .synth_in_nowait = spk_serial_in_nowait,
+ .flush_buffer = spk_serial_flush_buffer,
};
EXPORT_SYMBOL_GPL(spk_serial_io_ops);
outb(UART_MCR_DTR | UART_MCR_RTS | UART_MCR_OUT2,
speakup_info.port_tts + UART_MCR);
/* Turn on Interrupts */
- outb(UART_IER_MSI|UART_IER_RLSI|UART_IER_RDI,
- speakup_info.port_tts + UART_IER);
+ outb(UART_IER_MSI | UART_IER_RLSI | UART_IER_RDI,
+ speakup_info.port_tts + UART_IER);
inb(speakup_info.port_tts + UART_LSR);
inb(speakup_info.port_tts + UART_RX);
inb(speakup_info.port_tts + UART_IIR);
static void spk_serial_tiocmset(unsigned int set, unsigned int clear)
{
int old = inb(speakup_info.port_tts + UART_MCR);
+
outb((old & ~clear) | set, speakup_info.port_tts + UART_MCR);
}
}
while (spk_serial_tx_busy()) {
if (--tmout == 0) {
- pr_warn("%s: timed out (tx busy)\n", in_synth->long_name);
+ pr_warn("%s: timed out (tx busy)\n",
+ in_synth->long_name);
timeouts++;
return 0;
}
return 1;
}
-unsigned char spk_serial_in(void)
+static unsigned char spk_serial_in(void)
{
int tmout = SPK_SERIAL_TIMEOUT;
}
return inb_p(speakup_info.port_tts + UART_RX);
}
-EXPORT_SYMBOL_GPL(spk_serial_in);
-unsigned char spk_serial_in_nowait(void)
+static unsigned char spk_serial_in_nowait(void)
{
unsigned char lsr;
return 0;
return inb_p(speakup_info.port_tts + UART_RX);
}
-EXPORT_SYMBOL_GPL(spk_serial_in_nowait);
+
+static void spk_serial_flush_buffer(void)
+{
+ /* TODO: flush the UART 16550 buffer */
+}
static int spk_serial_out(struct spk_synth *in_synth, const char ch)
{
return 0;
}
-const char *spk_serial_synth_immediate(struct spk_synth *synth, const char *buff)
+const char *spk_serial_synth_immediate(struct spk_synth *synth,
+ const char *buff)
{
u_char ch;
#endif
#include <linux/serial_core.h>
+#include "spk_priv.h"
+
/*
* this is cut&paste from 8250.h. Get rid of the structure, the definitions
* and this whole broken driver.
};
/* countdown values for serial timeouts in us */
-#define SPK_SERIAL_TIMEOUT 100000
+#define SPK_SERIAL_TIMEOUT SPK_SYNTH_TIMEOUT
/* countdown values transmitter/dsr timeouts in us */
#define SPK_XMITR_TIMEOUT 100000
/* countdown values cts timeouts in us */
.trigger = 50,
.jiffies = 30,
.full = 40000,
+ .dev_name = SYNTH_DEFAULT_DEV,
.startup = SYNTH_START,
.checkval = SYNTH_CHECK,
.vars = vars,
- .io_ops = &spk_serial_io_ops,
+ .io_ops = &spk_ttyio_ops,
.probe = synth_probe,
- .release = spk_serial_release,
- .synth_immediate = spk_serial_synth_immediate,
+ .release = spk_ttyio_release,
+ .synth_immediate = spk_ttyio_synth_immediate,
.catch_up = spk_do_catch_up,
.flush = spk_synth_flush,
.is_alive = spk_synth_is_alive_restart,
{
int failed;
- failed = spk_serial_synth_probe(synth);
+ failed = spk_ttyio_synth_probe(synth);
if (failed == 0) {
synth->synth_immediate(synth, "\033=R\r");
mdelay(100);
}
module_param_named(ser, synth_acntsa.ser, int, 0444);
+module_param_named(dev, synth_acntsa.dev_name, charp, S_IRUGO);
module_param_named(start, synth_acntsa.startup, short, 0444);
MODULE_PARM_DESC(ser, "Set the serial port for the synthesizer (0-based).");
+MODULE_PARM_DESC(dev, "Set the device e.g. ttyUSB0, for the synthesizer.");
MODULE_PARM_DESC(start, "Start the synthesizer once it is loaded.");
module_spk_synth(synth_acntsa);
#include <linux/sched.h>
#include <linux/timer.h>
#include <linux/kthread.h>
+#include <linux/serial_reg.h> /* for UART_MCR* constants */
#include "spk_priv.h"
-#include "serialio.h"
#include "speakup.h"
#define DRV_VERSION "2.21"
.trigger = 50,
.jiffies = 50,
.full = 40000,
+ .dev_name = SYNTH_DEFAULT_DEV,
.startup = SYNTH_START,
.checkval = SYNTH_CHECK,
.vars = vars,
- .io_ops = &spk_serial_io_ops,
- .probe = spk_serial_synth_probe,
- .release = spk_serial_release,
- .synth_immediate = spk_serial_synth_immediate,
+ .io_ops = &spk_ttyio_ops,
+ .probe = spk_ttyio_synth_probe,
+ .release = spk_ttyio_release,
+ .synth_immediate = spk_ttyio_synth_immediate,
.catch_up = do_catch_up,
.flush = spk_synth_flush,
.is_alive = spk_synth_is_alive_restart,
}
module_param_named(ser, synth_apollo.ser, int, 0444);
+module_param_named(dev, synth_apollo.dev_name, charp, S_IRUGO);
module_param_named(start, synth_apollo.startup, short, 0444);
MODULE_PARM_DESC(ser, "Set the serial port for the synthesizer (0-based).");
+MODULE_PARM_DESC(dev, "Set the device e.g. ttyUSB0, for the synthesizer.");
MODULE_PARM_DESC(start, "Start the synthesizer once it is loaded.");
module_spk_synth(synth_apollo);
*/
#include "spk_priv.h"
#include "speakup.h"
-#include "serialio.h"
#define DRV_VERSION "2.11"
#define SYNTH_CLEAR 0x18 /* flush synth buffer */
.trigger = 50,
.jiffies = 30,
.full = 18000,
+ .dev_name = SYNTH_DEFAULT_DEV,
.startup = SYNTH_START,
.checkval = SYNTH_CHECK,
.vars = vars,
- .io_ops = &spk_serial_io_ops,
+ .io_ops = &spk_ttyio_ops,
.probe = synth_probe,
- .release = spk_serial_release,
- .synth_immediate = spk_serial_synth_immediate,
+ .release = spk_ttyio_release,
+ .synth_immediate = spk_ttyio_synth_immediate,
.catch_up = spk_do_catch_up,
.flush = synth_flush,
.is_alive = spk_synth_is_alive_restart,
static void synth_flush(struct spk_synth *synth)
{
+ synth->io_ops->flush_buffer();
synth->io_ops->send_xchar(SYNTH_CLEAR);
synth->io_ops->synth_out(synth, PROCSPEECH);
}
char synth_id[40] = "";
synth->synth_immediate(synth, "\x05[Q]");
- synth_id[test] = spk_serial_in();
+ synth_id[test] = synth->io_ops->synth_in();
if (synth_id[test] == 'A') {
do {
/* read version string from synth */
- synth_id[++test] = spk_serial_in();
+ synth_id[++test] = synth->io_ops->synth_in();
} while (synth_id[test] != '\n' && test < 32);
synth_id[++test] = 0x00;
}
{
int failed;
- failed = spk_serial_synth_probe(synth);
+ failed = spk_ttyio_synth_probe(synth);
if (failed == 0)
synth_version(synth);
synth->alive = !failed;
}
module_param_named(ser, synth_audptr.ser, int, 0444);
+module_param_named(dev, synth_audptr.dev_name, charp, S_IRUGO);
module_param_named(start, synth_audptr.startup, short, 0444);
MODULE_PARM_DESC(ser, "Set the serial port for the synthesizer (0-based).");
+MODULE_PARM_DESC(dev, "Set the device e.g. ttyUSB0, for the synthesizer.");
MODULE_PARM_DESC(start, "Start the synthesizer once it is loaded.");
module_spk_synth(synth_audptr);
.trigger = 50,
.jiffies = 50,
.full = 40000,
+ .dev_name = SYNTH_DEFAULT_DEV,
.startup = SYNTH_START,
.checkval = SYNTH_CHECK,
.vars = vars,
- .io_ops = &spk_serial_io_ops,
- .probe = spk_serial_synth_probe,
- .release = spk_serial_release,
- .synth_immediate = spk_serial_synth_immediate,
+ .io_ops = &spk_ttyio_ops,
+ .probe = spk_ttyio_synth_probe,
+ .release = spk_ttyio_release,
+ .synth_immediate = spk_ttyio_synth_immediate,
.catch_up = spk_do_catch_up,
.flush = spk_synth_flush,
.is_alive = spk_synth_is_alive_restart,
};
module_param_named(ser, synth_bns.ser, int, 0444);
+module_param_named(dev, synth_bns.dev_name, charp, S_IRUGO);
module_param_named(start, synth_bns.startup, short, 0444);
MODULE_PARM_DESC(ser, "Set the serial port for the synthesizer (0-based).");
+MODULE_PARM_DESC(dev, "Set the device e.g. ttyUSB0, for the synthesizer.");
MODULE_PARM_DESC(start, "Start the synthesizer once it is loaded.");
module_spk_synth(synth_bns);
#include <linux/kthread.h>
#include "spk_priv.h"
-#include "serialio.h"
#include "speakup.h"
#define DRV_VERSION "2.14"
#define SYNTH_CLEAR 0x03
#define PROCSPEECH 0x0b
-static unsigned char last_char;
-static inline u_char get_last_char(void)
-{
- u_char avail = inb_p(speakup_info.port_tts + UART_LSR) & UART_LSR_DR;
+static volatile unsigned char last_char;
- if (avail)
- last_char = inb_p(speakup_info.port_tts + UART_RX);
- return last_char;
+static void read_buff_add(u_char ch)
+{
+ last_char = ch;
}
static inline bool synth_full(void)
{
- return get_last_char() == 0x13;
+ return last_char == 0x13;
}
static void do_catch_up(struct spk_synth *synth);
.jiffies = 50,
.full = 40000,
.flags = SF_DEC,
+ .dev_name = SYNTH_DEFAULT_DEV,
.startup = SYNTH_START,
.checkval = SYNTH_CHECK,
.vars = vars,
- .io_ops = &spk_serial_io_ops,
- .probe = spk_serial_synth_probe,
- .release = spk_serial_release,
- .synth_immediate = spk_serial_synth_immediate,
+ .io_ops = &spk_ttyio_ops,
+ .probe = spk_ttyio_synth_probe,
+ .release = spk_ttyio_release,
+ .synth_immediate = spk_ttyio_synth_immediate,
.catch_up = do_catch_up,
.flush = synth_flush,
.is_alive = spk_synth_is_alive_restart,
.synth_adjust = NULL,
- .read_buff_add = NULL,
+ .read_buff_add = read_buff_add,
.get_index = NULL,
.indexing = {
.command = NULL,
static void synth_flush(struct spk_synth *synth)
{
in_escape = 0;
+ synth->io_ops->flush_buffer();
synth->synth_immediate(synth, "\033P;10z\033\\");
}
module_param_named(ser, synth_decext.ser, int, 0444);
+module_param_named(dev, synth_decext.dev_name, charp, S_IRUGO);
module_param_named(start, synth_decext.startup, short, 0444);
MODULE_PARM_DESC(ser, "Set the serial port for the synthesizer (0-based).");
+MODULE_PARM_DESC(dev, "Set the device e.g. ttyUSB0, for the synthesizer.");
MODULE_PARM_DESC(start, "Start the synthesizer once it is loaded.");
module_spk_synth(synth_decext);
#define CTRL_last_index 0x0b00 /* get last index spoken */
#define CTRL_io_priority 0x0c00 /* change i/o priority */
#define CTRL_free_mem 0x0d00 /* get free paragraphs on module */
-#define CTRL_get_lang 0x0e00 /* return bit mask of loaded languages */
+#define CTRL_get_lang 0x0e00 /* return bitmask of loaded languages */
#define CMD_test 0x2000 /* self-test request */
#define TEST_mask 0x0F00 /* isolate test field */
#define TEST_null 0x0000 /* no test requested */
#include <linux/kthread.h>
#include "speakup.h"
#include "spk_priv.h"
-#include "serialio.h"
#define DRV_VERSION "2.20"
#define SYNTH_CLEAR 0x03
static void do_catch_up(struct spk_synth *synth);
static void synth_flush(struct spk_synth *synth);
static void read_buff_add(u_char c);
-static unsigned char get_index(void);
+static unsigned char get_index(struct spk_synth *synth);
static int in_escape;
static int is_flushing;
.trigger = 50,
.jiffies = 50,
.full = 40000,
+ .dev_name = SYNTH_DEFAULT_DEV,
.startup = SYNTH_START,
.checkval = SYNTH_CHECK,
.vars = vars,
.default_pitch = ap_defaults,
.default_vol = g5_defaults,
- .io_ops = &spk_serial_io_ops,
- .probe = spk_serial_synth_probe,
- .release = spk_serial_release,
- .synth_immediate = spk_serial_synth_immediate,
+ .io_ops = &spk_ttyio_ops,
+ .probe = spk_ttyio_synth_probe,
+ .release = spk_ttyio_release,
+ .synth_immediate = spk_ttyio_synth_immediate,
.catch_up = do_catch_up,
.flush = synth_flush,
.is_alive = spk_synth_is_alive_restart,
static u_char lastind;
-static unsigned char get_index(void)
+static unsigned char get_index(struct spk_synth *synth)
{
u_char rv;
synth->io_ops->synth_out(synth, ']');
in_escape = 0;
is_flushing = 1;
+ synth->io_ops->flush_buffer();
synth->io_ops->synth_out(synth, SYNTH_CLEAR);
}
module_param_named(ser, synth_dectlk.ser, int, 0444);
+module_param_named(dev, synth_dectlk.dev_name, charp, S_IRUGO);
module_param_named(start, synth_dectlk.startup, short, 0444);
MODULE_PARM_DESC(ser, "Set the serial port for the synthesizer (0-based).");
+MODULE_PARM_DESC(dev, "Set the device e.g. ttyUSB0, for the synthesizer.");
MODULE_PARM_DESC(start, "Start the synthesizer once it is loaded.");
module_spk_synth(synth_dectlk);
.is_alive = spk_synth_is_alive_nop,
.synth_adjust = NULL,
.read_buff_add = NULL,
- .get_index = spk_serial_in_nowait,
+ .get_index = spk_synth_get_index,
.indexing = {
.command = "\x01%di",
.lowindex = 1,
* usec later.
*/
#define TTS_ALMOST_FULL 0x08 /* mask for AF bit: When set to 1,
- * indicates that less than 300 bytes
- * are available in the TTS input
- * buffer. AF is always 0 in the PCM,
- * TGN and CVSD modes.
- */
+ * indicates that less than 300 bytes
+ * are available in the TTS input
+ * buffer. AF is always 0 in the PCM,
+ * TGN and CVSD modes.
+ */
#define TTS_ALMOST_EMPTY 0x04 /* mask for AE bit: When set to 1,
* indicates that less than 300 bytes
* are remaining in DoubleTalk's input
.trigger = 50,
.jiffies = 50,
.full = 40000,
+ .dev_name = SYNTH_DEFAULT_DEV,
.startup = SYNTH_START,
.checkval = SYNTH_CHECK,
.vars = vars,
- .io_ops = &spk_serial_io_ops,
- .probe = spk_serial_synth_probe,
- .release = spk_serial_release,
- .synth_immediate = spk_serial_synth_immediate,
+ .io_ops = &spk_ttyio_ops,
+ .probe = spk_ttyio_synth_probe,
+ .release = spk_ttyio_release,
+ .synth_immediate = spk_ttyio_synth_immediate,
.catch_up = spk_do_catch_up,
.flush = spk_synth_flush,
.is_alive = spk_synth_is_alive_restart,
};
module_param_named(ser, synth_dummy.ser, int, 0444);
+module_param_named(dev, synth_dummy.dev_name, charp, S_IRUGO);
module_param_named(start, synth_dummy.startup, short, 0444);
MODULE_PARM_DESC(ser, "Set the serial port for the synthesizer (0-based).");
+MODULE_PARM_DESC(dev, "Set the device e.g. ttyUSB0, for the synthesizer.");
MODULE_PARM_DESC(start, "Start the synthesizer once it is loaded.");
module_spk_synth(synth_dummy);
*/
#include "speakup.h"
#include "spk_priv.h"
-#include "serialio.h"
#include "speakup_dtlk.h" /* local header file for LiteTalk values */
#define DRV_VERSION "2.11"
.trigger = 50,
.jiffies = 50,
.full = 40000,
+ .dev_name = SYNTH_DEFAULT_DEV,
.startup = SYNTH_START,
.checkval = SYNTH_CHECK,
.vars = vars,
- .io_ops = &spk_serial_io_ops,
+ .io_ops = &spk_ttyio_ops,
.probe = synth_probe,
- .release = spk_serial_release,
- .synth_immediate = spk_serial_synth_immediate,
+ .release = spk_ttyio_release,
+ .synth_immediate = spk_ttyio_synth_immediate,
.catch_up = spk_do_catch_up,
.flush = spk_synth_flush,
.is_alive = spk_synth_is_alive_restart,
.synth_adjust = NULL,
.read_buff_add = NULL,
- .get_index = spk_serial_in_nowait,
+ .get_index = spk_synth_get_index,
.indexing = {
.command = "\x01%di",
.lowindex = 1,
synth->synth_immediate(synth, "\x18\x01?");
for (i = 0; i < 50; i++) {
- buf[i] = spk_serial_in();
+ buf[i] = synth->io_ops->synth_in();
if (i > 2 && buf[i] == 0x7f)
break;
}
{
int failed = 0;
- failed = spk_serial_synth_probe(synth);
+ failed = spk_ttyio_synth_probe(synth);
if (failed == 0)
synth_interrogate(synth);
synth->alive = !failed;
}
module_param_named(ser, synth_ltlk.ser, int, 0444);
+module_param_named(dev, synth_ltlk.dev_name, charp, S_IRUGO);
module_param_named(start, synth_ltlk.startup, short, 0444);
MODULE_PARM_DESC(ser, "Set the serial port for the synthesizer (0-based).");
+MODULE_PARM_DESC(dev, "Set the device e.g. ttyUSB0, for the synthesizer.");
MODULE_PARM_DESC(start, "Start the synthesizer once it is loaded.");
module_spk_synth(synth_ltlk);
static int softsynth_probe(struct spk_synth *synth);
static void softsynth_release(void);
static int softsynth_is_alive(struct spk_synth *synth);
-static unsigned char get_index(void);
+static unsigned char get_index(struct spk_synth *synth);
static struct miscdevice synth_device, synthu_device;
static int init_pos;
return ret;
}
-static unsigned char get_index(void)
+static unsigned char get_index(struct spk_synth *synth)
{
int rv;
*/
#include "spk_priv.h"
#include "speakup.h"
-#include "serialio.h"
#define DRV_VERSION "2.11"
#define SYNTH_CLEAR 0x18
.trigger = 50,
.jiffies = 50,
.full = 40000,
+ .dev_name = SYNTH_DEFAULT_DEV,
.startup = SYNTH_START,
.checkval = SYNTH_CHECK,
.vars = vars,
- .io_ops = &spk_serial_io_ops,
- .probe = spk_serial_synth_probe,
- .release = spk_serial_release,
- .synth_immediate = spk_serial_synth_immediate,
+ .io_ops = &spk_ttyio_ops,
+ .probe = spk_ttyio_synth_probe,
+ .release = spk_ttyio_release,
+ .synth_immediate = spk_ttyio_synth_immediate,
.catch_up = spk_do_catch_up,
.flush = synth_flush,
.is_alive = spk_synth_is_alive_restart,
.synth_adjust = NULL,
.read_buff_add = NULL,
- .get_index = spk_serial_in_nowait,
+ .get_index = spk_synth_get_index,
.indexing = {
.command = "\x05[%c",
.lowindex = 1,
static void synth_flush(struct spk_synth *synth)
{
+ synth->io_ops->flush_buffer();
synth->io_ops->send_xchar(SYNTH_CLEAR);
}
module_param_named(ser, synth_spkout.ser, int, 0444);
+module_param_named(dev, synth_spkout.dev_name, charp, S_IRUGO);
module_param_named(start, synth_spkout.startup, short, 0444);
MODULE_PARM_DESC(ser, "Set the serial port for the synthesizer (0-based).");
+MODULE_PARM_DESC(dev, "Set the device e.g. ttyUSB0, for the synthesizer.");
MODULE_PARM_DESC(start, "Start the synthesizer once it is loaded.");
module_spk_synth(synth_spkout);
.trigger = 50,
.jiffies = 50,
.full = 40000,
+ .dev_name = SYNTH_DEFAULT_DEV,
.startup = SYNTH_START,
.checkval = SYNTH_CHECK,
.vars = vars,
- .io_ops = &spk_serial_io_ops,
- .probe = spk_serial_synth_probe,
- .release = spk_serial_release,
- .synth_immediate = spk_serial_synth_immediate,
+ .io_ops = &spk_ttyio_ops,
+ .probe = spk_ttyio_synth_probe,
+ .release = spk_ttyio_release,
+ .synth_immediate = spk_ttyio_synth_immediate,
.catch_up = spk_do_catch_up,
.flush = spk_synth_flush,
.is_alive = spk_synth_is_alive_restart,
};
module_param_named(ser, synth_txprt.ser, int, 0444);
+module_param_named(dev, synth_txprt.dev_name, charp, S_IRUGO);
module_param_named(start, synth_txprt.startup, short, 0444);
MODULE_PARM_DESC(ser, "Set the serial port for the synthesizer (0-based).");
+MODULE_PARM_DESC(dev, "Set the device e.g. ttyUSB0, for the synthesizer.");
MODULE_PARM_DESC(start, "Start the synthesizer once it is loaded.");
module_spk_synth(synth_txprt);
#endif
#define KT_SPKUP 15
+#define SPK_SYNTH_TIMEOUT 100000 /* in micro-seconds */
+#define SYNTH_DEFAULT_DEV "ttyS0"
+#define SYNTH_DEFAULT_SER 0
const struct old_serial_port *spk_serial_init(int index);
void spk_stop_serial_interrupt(void);
int spk_wait_for_xmitr(struct spk_synth *in_synth);
-unsigned char spk_serial_in(void);
-unsigned char spk_serial_in_nowait(void);
void spk_serial_release(void);
+void spk_ttyio_release(void);
void synth_buffer_skip_nonlatin1(void);
u16 synth_buffer_getc(void);
const char *buf, size_t count);
int spk_serial_synth_probe(struct spk_synth *synth);
+int spk_ttyio_synth_probe(struct spk_synth *synth);
const char *spk_serial_synth_immediate(struct spk_synth *synth, const char *buff);
+const char *spk_ttyio_synth_immediate(struct spk_synth *synth, const char *buff);
void spk_do_catch_up(struct spk_synth *synth);
void spk_synth_flush(struct spk_synth *synth);
+unsigned char spk_synth_get_index(struct spk_synth *synth);
int spk_synth_is_alive_nop(struct spk_synth *synth);
int spk_synth_is_alive_restart(struct spk_synth *synth);
__printf(1, 2)
extern struct var_t synth_time_vars[];
extern struct spk_io_ops spk_serial_io_ops;
+extern struct spk_io_ops spk_ttyio_ops;
#endif
--- /dev/null
+#include <linux/types.h>
+#include <linux/tty.h>
+#include <linux/tty_flip.h>
+#include <linux/slab.h>
+
+#include "speakup.h"
+#include "spk_types.h"
+#include "spk_priv.h"
+
+#define DEV_PREFIX_LP "lp"
+
+static const char * const lp_supported[] = { "acntsa", "bns", "dummy",
+ "txprt" };
+
+struct spk_ldisc_data {
+ char buf;
+ struct semaphore sem;
+ bool buf_free;
+};
+
+static struct spk_synth *spk_ttyio_synth;
+static struct tty_struct *speakup_tty;
+
+static int ser_to_dev(int ser, dev_t *dev_no)
+{
+ if (ser < 0 || ser > (255 - 64)) {
+ pr_err("speakup: Invalid ser param. Must be between 0 and 191 inclusive.\n");
+ return -EINVAL;
+ }
+
+ *dev_no = MKDEV(4, (64 + ser));
+ return 0;
+}
+
+static int get_dev_to_use(struct spk_synth *synth, dev_t *dev_no)
+{
+ /* use ser only when dev is not specified */
+ if (strcmp(synth->dev_name, SYNTH_DEFAULT_DEV) ||
+ synth->ser == SYNTH_DEFAULT_SER) {
+ /* for /dev/lp* check if synth is supported */
+ if (strncmp(synth->dev_name, DEV_PREFIX_LP,
+ strlen(DEV_PREFIX_LP)) == 0)
+ if (match_string(lp_supported, ARRAY_SIZE(lp_supported),
+ synth->name) < 0) {
+ int i;
+
+ pr_err("speakup: lp* is only supported on:");
+ for (i = 0; i < ARRAY_SIZE(lp_supported); i++)
+ pr_cont(" %s", lp_supported[i]);
+ pr_cont("\n");
+
+ return -ENOTSUPP;
+ }
+
+ return tty_dev_name_to_number(synth->dev_name, dev_no);
+ }
+
+ return ser_to_dev(synth->ser, dev_no);
+}
+
+static int spk_ttyio_ldisc_open(struct tty_struct *tty)
+{
+ struct spk_ldisc_data *ldisc_data;
+
+ if (tty->ops->write == NULL)
+ return -EOPNOTSUPP;
+ speakup_tty = tty;
+
+ ldisc_data = kmalloc(sizeof(struct spk_ldisc_data), GFP_KERNEL);
+ if (!ldisc_data) {
+ pr_err("speakup: Failed to allocate ldisc_data.\n");
+ return -ENOMEM;
+ }
+
+ sema_init(&ldisc_data->sem, 0);
+ ldisc_data->buf_free = true;
+ speakup_tty->disc_data = ldisc_data;
+
+ return 0;
+}
+
+static void spk_ttyio_ldisc_close(struct tty_struct *tty)
+{
+ kfree(speakup_tty->disc_data);
+ speakup_tty = NULL;
+}
+
+static int spk_ttyio_receive_buf2(struct tty_struct *tty,
+ const unsigned char *cp, char *fp, int count)
+{
+ struct spk_ldisc_data *ldisc_data = tty->disc_data;
+
+ if (spk_ttyio_synth->read_buff_add) {
+ int i;
+
+ for (i = 0; i < count; i++)
+ spk_ttyio_synth->read_buff_add(cp[i]);
+
+ return count;
+ }
+
+ if (!ldisc_data->buf_free)
+ /* ttyio_in will tty_schedule_flip */
+ return 0;
+
+ /* Make sure the consumer has read buf before we have seen
+ * buf_free == true and overwrite buf */
+ mb();
+
+ ldisc_data->buf = cp[0];
+ ldisc_data->buf_free = false;
+ up(&ldisc_data->sem);
+
+ return 1;
+}
+
+static struct tty_ldisc_ops spk_ttyio_ldisc_ops = {
+ .owner = THIS_MODULE,
+ .magic = TTY_LDISC_MAGIC,
+ .name = "speakup_ldisc",
+ .open = spk_ttyio_ldisc_open,
+ .close = spk_ttyio_ldisc_close,
+ .receive_buf2 = spk_ttyio_receive_buf2,
+};
+
+static int spk_ttyio_out(struct spk_synth *in_synth, const char ch);
+static void spk_ttyio_send_xchar(char ch);
+static void spk_ttyio_tiocmset(unsigned int set, unsigned int clear);
+static unsigned char spk_ttyio_in(void);
+static unsigned char spk_ttyio_in_nowait(void);
+static void spk_ttyio_flush_buffer(void);
+
+struct spk_io_ops spk_ttyio_ops = {
+ .synth_out = spk_ttyio_out,
+ .send_xchar = spk_ttyio_send_xchar,
+ .tiocmset = spk_ttyio_tiocmset,
+ .synth_in = spk_ttyio_in,
+ .synth_in_nowait = spk_ttyio_in_nowait,
+ .flush_buffer = spk_ttyio_flush_buffer,
+};
+EXPORT_SYMBOL_GPL(spk_ttyio_ops);
+
+static inline void get_termios(struct tty_struct *tty, struct ktermios *out_termios)
+{
+ down_read(&tty->termios_rwsem);
+ *out_termios = tty->termios;
+ up_read(&tty->termios_rwsem);
+}
+
+static int spk_ttyio_initialise_ldisc(struct spk_synth *synth)
+{
+ int ret = 0;
+ struct tty_struct *tty;
+ struct ktermios tmp_termios;
+ dev_t dev;
+
+ ret = tty_register_ldisc(N_SPEAKUP, &spk_ttyio_ldisc_ops);
+ if (ret) {
+ pr_err("Error registering line discipline.\n");
+ return ret;
+ }
+
+ ret = get_dev_to_use(synth, &dev);
+ if (ret)
+ return ret;
+
+ tty = tty_open_by_driver(dev, NULL, NULL);
+ if (IS_ERR(tty))
+ return PTR_ERR(tty);
+
+ if (tty->ops->open)
+ ret = tty->ops->open(tty, NULL);
+ else
+ ret = -ENODEV;
+
+ if (ret) {
+ tty_unlock(tty);
+ return ret;
+ }
+
+ clear_bit(TTY_HUPPED, &tty->flags);
+ /* ensure hardware flow control is enabled */
+ get_termios(tty, &tmp_termios);
+ if (!(tmp_termios.c_cflag & CRTSCTS)) {
+ tmp_termios.c_cflag |= CRTSCTS;
+ tty_set_termios(tty, &tmp_termios);
+ /*
+ * check c_cflag to see if it's updated as tty_set_termios may not return
+ * error even when no tty bits are changed by the request.
+ */
+ get_termios(tty, &tmp_termios);
+ if (!(tmp_termios.c_cflag & CRTSCTS))
+ pr_warn("speakup: Failed to set hardware flow control\n");
+ }
+
+ tty_unlock(tty);
+
+ ret = tty_set_ldisc(tty, N_SPEAKUP);
+
+ return ret;
+}
+
+static int spk_ttyio_out(struct spk_synth *in_synth, const char ch)
+{
+ if (in_synth->alive && speakup_tty && speakup_tty->ops->write) {
+ int ret = speakup_tty->ops->write(speakup_tty, &ch, 1);
+
+ if (ret == 0)
+ /* No room */
+ return 0;
+ if (ret < 0) {
+ pr_warn("%s: I/O error, deactivating speakup\n", in_synth->long_name);
+ /* No synth any more, so nobody will restart TTYs, and we thus
+ * need to do it ourselves. Now that there is no synth we can
+ * let application flood anyway
+ */
+ in_synth->alive = 0;
+ speakup_start_ttys();
+ return 0;
+ }
+ return 1;
+ }
+ return 0;
+}
+
+static void spk_ttyio_send_xchar(char ch)
+{
+ speakup_tty->ops->send_xchar(speakup_tty, ch);
+}
+
+static void spk_ttyio_tiocmset(unsigned int set, unsigned int clear)
+{
+ speakup_tty->ops->tiocmset(speakup_tty, set, clear);
+}
+
+static unsigned char ttyio_in(int timeout)
+{
+ struct spk_ldisc_data *ldisc_data = speakup_tty->disc_data;
+ char rv;
+
+ if (down_timeout(&ldisc_data->sem, usecs_to_jiffies(timeout)) == -ETIME) {
+ if (timeout)
+ pr_warn("spk_ttyio: timeout (%d) while waiting for input\n",
+ timeout);
+ return 0xff;
+ }
+
+ rv = ldisc_data->buf;
+ /* Make sure we have read buf before we set buf_free to let
+ * the producer overwrite it */
+ mb();
+ ldisc_data->buf_free = true;
+ /* Let TTY push more characters */
+ tty_schedule_flip(speakup_tty->port);
+
+ return rv;
+}
+
+static unsigned char spk_ttyio_in(void)
+{
+ return ttyio_in(SPK_SYNTH_TIMEOUT);
+}
+
+static unsigned char spk_ttyio_in_nowait(void)
+{
+ u8 rv = ttyio_in(0);
+
+ return (rv == 0xff) ? 0 : rv;
+}
+
+static void spk_ttyio_flush_buffer(void)
+{
+ if (speakup_tty->ops->flush_buffer)
+ speakup_tty->ops->flush_buffer(speakup_tty);
+}
+
+int spk_ttyio_synth_probe(struct spk_synth *synth)
+{
+ int rv = spk_ttyio_initialise_ldisc(synth);
+
+ if (rv)
+ return rv;
+
+ synth->alive = 1;
+ spk_ttyio_synth = synth;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(spk_ttyio_synth_probe);
+
+void spk_ttyio_release(void)
+{
+ if (!speakup_tty)
+ return;
+
+ tty_lock(speakup_tty);
+
+ if (speakup_tty->ops->close)
+ speakup_tty->ops->close(speakup_tty, NULL);
+
+ tty_ldisc_flush(speakup_tty);
+ tty_unlock(speakup_tty);
+ tty_ldisc_release(speakup_tty);
+}
+EXPORT_SYMBOL_GPL(spk_ttyio_release);
+
+const char *spk_ttyio_synth_immediate(struct spk_synth *synth, const char *buff)
+{
+ u_char ch;
+
+ while ((ch = *buff)) {
+ if (ch == '\n')
+ ch = synth->procspeech;
+ if (tty_write_room(speakup_tty) < 1 || !synth->io_ops->synth_out(synth, ch))
+ return buff;
+ buff++;
+ }
+ return NULL;
+}
+EXPORT_SYMBOL_GPL(spk_ttyio_synth_immediate);
int (*synth_out)(struct spk_synth *synth, const char ch);
void (*send_xchar)(char ch);
void (*tiocmset)(unsigned int set, unsigned int clear);
+ unsigned char (*synth_in)(void);
+ unsigned char (*synth_in_nowait)(void);
+ void (*flush_buffer)(void);
};
struct spk_synth {
int jiffies;
int full;
int ser;
+ char *dev_name;
short flags;
short startup;
const int checkval; /* for validating a proper synth module */
int (*is_alive)(struct spk_synth *synth);
int (*synth_adjust)(struct st_var_header *var);
void (*read_buff_add)(u_char);
- unsigned char (*get_index)(void);
+ unsigned char (*get_index)(struct spk_synth *synth);
struct synth_indexing indexing;
int alive;
struct attribute_group attributes;
void spk_synth_flush(struct spk_synth *synth)
{
+ synth->io_ops->flush_buffer();
synth->io_ops->synth_out(synth, synth->clear);
}
EXPORT_SYMBOL_GPL(spk_synth_flush);
+unsigned char spk_synth_get_index(struct spk_synth *synth)
+{
+ return synth->io_ops->synth_in_nowait();
+}
+EXPORT_SYMBOL_GPL(spk_synth_get_index);
+
int spk_synth_is_alive_nop(struct spk_synth *synth)
{
synth->alive = 1;
if (first)
first = 0;
else
- synth->get_index();
+ synth->get_index(synth);
index_count = 0;
sentence_count = sc;
}
void spk_get_index_count(int *linecount, int *sentcount)
{
- int ind = synth->get_index();
+ int ind = synth->get_index(synth);
if (ind) {
sentence_count = ind % 10;
mutex_unlock(&spk_mutex);
return -1;
}
- synths[i++] = in_synth;
- synths[i] = NULL;
+
if (in_synth->startup)
status = do_synth_init(in_synth);
+
+ if (!status) {
+ synths[i++] = in_synth;
+ synths[i] = NULL;
+ }
+
mutex_unlock(&spk_mutex);
return status;
}
}
/* packsym tells the FUSB302 chip that the next X bytes are payload */
buf[pos++] = FUSB302_TKN_PACKSYM | (len & 0x1F);
- buf[pos++] = msg->header & 0xFF;
- buf[pos++] = (msg->header >> 8) & 0xFF;
+ memcpy(&buf[pos], &msg->header, sizeof(msg->header));
+ pos += sizeof(msg->header);
len -= 2;
memcpy(&buf[pos], msg->payload, len);
The visorhba driver registers with visorbus as the function driver to
handle virtual scsi disk devices, specified using the
-SPAR_VHBA_CHANNEL_PROTOCOL_UUID type in the visorbus_register_visor_driver()
+VISOR_VHBA_CHANNEL_UUID type in the visorbus_register_visor_driver()
call. visorhba uses scsi_add_host() to expose a Linux block device
(e.g., /sys/block/) in the guest environment for each s-Par virtual device.
standard udev/systemd environments, as it includes the modules.alias
definition:
- "visorbus:"+SPAR_VHBA_CHANNEL_PROTOCOL_UUID_STR
+ "visorbus:"+VISOR_VHBA_CHANNEL_UUID_STR
i.e.:
The visornic driver registers with visorbus as the function driver to
handle virtual network devices, specified using the
-SPAR_VNIC_CHANNEL_PROTOCOL_UUID type in the visorbus_register_visor_driver()
+VISOR_VNIC_CHANNEL_UUID type in the visorbus_register_visor_driver()
call. visornic uses register_netdev() to expose a Linux device of class net
(e.g., /sys/class/net/) in the guest environment for each s-Par virtual
device.
standard udev/systemd environments, as it includes the modules.alias
definition:
- "visorbus:"+SPAR_VNIC_CHANNEL_PROTOCOL_UUID_STR
+ "visorbus:"+VISOR_VNIC_CHANNEL_UUID_STR
i.e.:
The visorinput driver registers with visorbus as the function driver to
handle human input devices, specified using the
-SPAR_KEYBOARD_CHANNEL_PROTOCOL_UUID and SPAR_MOUSE_CHANNEL_PROTOCOL_UUID
+VISOR_KEYBOARD_CHANNEL_UUID and VISOR_MOUSE_CHANNEL_UUID
types in the visorbus_register_visor_driver() call. visorinput uses
input_register_device() to expose devices of class input
(e.g., /sys/class/input/) for virtual keyboard and virtual mouse devices.
standard udev/systemd environments, as it includes the modules.alias
definition:
- "visorbus:"+SPAR_MOUSE_CHANNEL_PROTOCOL_UUID_STR
- "visorbus:"+SPAR_KEYBOARD_CHANNEL_PROTOCOL_UUID_STR
+ "visorbus:"+VISOR_MOUSE_CHANNEL_UUID_STR
+ "visorbus:"+VISOR_KEYBOARD_CHANNEL_UUID_STR
i.e.:
#define COVER(v, d) ((d) * DIV_ROUND_UP(v, d))
#endif
-#define ULTRA_CHANNEL_PROTOCOL_SIGNATURE SIGNATURE_32('E', 'C', 'N', 'L')
+#define VISOR_CHANNEL_SIGNATURE SIGNATURE_32('E', 'C', 'N', 'L')
enum channel_serverstate {
CHANNELSRV_UNINITIALIZED = 0, /* channel is in an undefined state */
/* access channel anytime */
};
-#define SPAR_CHANNEL_SERVER_READY(ch) \
+#define VISOR_CHANNEL_SERVER_READY(ch) \
(readl(&(ch)->srv_state) == CHANNELSRV_READY)
-#define ULTRA_VALID_CHANNELCLI_TRANSITION(o, n) \
+#define VISOR_VALID_CHANNELCLI_TRANSITION(o, n) \
(((((o) == CHANNELCLI_DETACHED) && ((n) == CHANNELCLI_DISABLED)) || \
(((o) == CHANNELCLI_ATTACHING) && ((n) == CHANNELCLI_DISABLED)) || \
(((o) == CHANNELCLI_ATTACHED) && ((n) == CHANNELCLI_DISABLED)) || \
(((o) == CHANNELCLI_BUSY) && ((n) == CHANNELCLI_OWNED)) || (0)) \
? (1) : (0))
-/* Values for ULTRA_CHANNEL_PROTOCOL.CliErrorBoot: */
+/* Values for VISORA_CHANNEL_PROTOCOL.CliErrorBoot: */
/* throttling invalid boot channel statetransition error due to client
* disabled
*/
-#define ULTRA_CLIERRORBOOT_THROTTLEMSG_DISABLED 0x01
+#define VISOR_CLIERRORBOOT_THROTTLEMSG_DISABLED 0x01
/* throttling invalid boot channel statetransition error due to client
* not attached
*/
-#define ULTRA_CLIERRORBOOT_THROTTLEMSG_NOTATTACHED 0x02
+#define VISOR_CLIERRORBOOT_THROTTLEMSG_NOTATTACHED 0x02
/* throttling invalid boot channel statetransition error due to busy channel */
-#define ULTRA_CLIERRORBOOT_THROTTLEMSG_BUSY 0x04
+#define VISOR_CLIERRORBOOT_THROTTLEMSG_BUSY 0x04
-/* Values for ULTRA_CHANNEL_PROTOCOL.Features: This define exists so
+/* Values for VISOR_CHANNEL_PROTOCOL.Features: This define exists so
* that windows guest can look at the FeatureFlags in the io channel,
* and configure the windows driver to use interrupts or not based on
* this setting. This flag is set in uislib after the
- * ULTRA_VHBA_init_channel is called. All feature bits for all
+ * VISOR_VHBA_init_channel is called. All feature bits for all
* channels should be defined here. The io channel feature bits are
* defined right here
*/
-#define ULTRA_IO_DRIVER_ENABLES_INTS (0x1ULL << 1)
-#define ULTRA_IO_CHANNEL_IS_POLLING (0x1ULL << 3)
-#define ULTRA_IO_IOVM_IS_OK_WITH_DRIVER_DISABLING_INTS (0x1ULL << 4)
-#define ULTRA_IO_DRIVER_DISABLES_INTS (0x1ULL << 5)
-#define ULTRA_IO_DRIVER_SUPPORTS_ENHANCED_RCVBUF_CHECKING (0x1ULL << 6)
+#define VISOR_DRIVER_ENABLES_INTS (0x1ULL << 1)
+#define VISOR_CHANNEL_IS_POLLING (0x1ULL << 3)
+#define VISOR_IOVM_OK_DRIVER_DISABLING_INTS (0x1ULL << 4)
+#define VISOR_DRIVER_DISABLES_INTS (0x1ULL << 5)
+#define VISOR_DRIVER_ENHANCED_RCVBUF_CHECKING (0x1ULL << 6)
/* Common Channel Header */
struct channel_header {
u8 recover_channel;
} __packed;
-#define ULTRA_CHANNEL_ENABLE_INTS (0x1ULL << 0)
+#define VISOR_CHANNEL_ENABLE_INTS (0x1ULL << 0)
/* Subheader for the Signal Type variation of the Common Channel */
struct signal_queue_header {
* is used to pass the EFI_DIAG_CAPTURE_PROTOCOL needed to log messages.
*/
static inline int
-spar_check_channel(struct channel_header *ch,
- uuid_le expected_uuid,
- char *chname,
- u64 expected_min_bytes,
- u32 expected_version,
- u64 expected_signature)
+visor_check_channel(struct channel_header *ch,
+ uuid_le expected_uuid,
+ char *chname,
+ u64 expected_min_bytes,
+ u32 expected_version,
+ u64 expected_signature)
{
if (uuid_le_cmp(expected_uuid, NULL_UUID_LE) != 0) {
/* caller wants us to verify type GUID */
*/
/* {414815ed-c58c-11da-95a9-00e08161165f} */
-#define SPAR_VHBA_CHANNEL_PROTOCOL_UUID \
+#define VISOR_VHBA_CHANNEL_UUID \
UUID_LE(0x414815ed, 0xc58c, 0x11da, \
0x95, 0xa9, 0x0, 0xe0, 0x81, 0x61, 0x16, 0x5f)
-static const uuid_le spar_vhba_channel_protocol_uuid =
- SPAR_VHBA_CHANNEL_PROTOCOL_UUID;
-#define SPAR_VHBA_CHANNEL_PROTOCOL_UUID_STR \
+static const uuid_le visor_vhba_channel_uuid = VISOR_VHBA_CHANNEL_UUID;
+#define VISOR_VHBA_CHANNEL_UUID_STR \
"414815ed-c58c-11da-95a9-00e08161165f"
/* {8cd5994d-c58e-11da-95a9-00e08161165f} */
-#define SPAR_VNIC_CHANNEL_PROTOCOL_UUID \
+#define VISOR_VNIC_CHANNEL_UUID \
UUID_LE(0x8cd5994d, 0xc58e, 0x11da, \
0x95, 0xa9, 0x0, 0xe0, 0x81, 0x61, 0x16, 0x5f)
-static const uuid_le spar_vnic_channel_protocol_uuid =
- SPAR_VNIC_CHANNEL_PROTOCOL_UUID;
-#define SPAR_VNIC_CHANNEL_PROTOCOL_UUID_STR \
+static const uuid_le visor_vnic_channel_uuid = VISOR_VNIC_CHANNEL_UUID;
+#define VISOR_VNIC_CHANNEL_UUID_STR \
"8cd5994d-c58e-11da-95a9-00e08161165f"
/* {72120008-4AAB-11DC-8530-444553544200} */
-#define SPAR_SIOVM_UUID \
+#define VISOR_SIOVM_UUID \
UUID_LE(0x72120008, 0x4AAB, 0x11DC, \
0x85, 0x30, 0x44, 0x45, 0x53, 0x54, 0x42, 0x00)
-static const uuid_le spar_siovm_uuid = SPAR_SIOVM_UUID;
+static const uuid_le visor_siovm_uuid = VISOR_SIOVM_UUID;
#endif
#include <linux/dma-direction.h>
#include "channel.h"
-#define ULTRA_VHBA_CHANNEL_PROTOCOL_SIGNATURE ULTRA_CHANNEL_PROTOCOL_SIGNATURE
-#define ULTRA_VNIC_CHANNEL_PROTOCOL_SIGNATURE ULTRA_CHANNEL_PROTOCOL_SIGNATURE
-#define ULTRA_VSWITCH_CHANNEL_PROTOCOL_SIGNATURE \
- ULTRA_CHANNEL_PROTOCOL_SIGNATURE
+#define VISOR_VHBA_CHANNEL_SIGNATURE VISOR_CHANNEL_SIGNATURE
+#define VISOR_VNIC_CHANNEL_SIGNATURE VISOR_CHANNEL_SIGNATURE
+#define VISOR_VSWITCH_CHANNEL_SIGNATURE VISOR_CHANNEL_SIGNATURE
/*
* Must increment these whenever you insert or delete fields within this channel
* usually add fields to the END of the channel struct without needing to
* increment this.
*/
-#define ULTRA_VHBA_CHANNEL_PROTOCOL_VERSIONID 2
-#define ULTRA_VNIC_CHANNEL_PROTOCOL_VERSIONID 2
-#define ULTRA_VSWITCH_CHANNEL_PROTOCOL_VERSIONID 1
-
-#define SPAR_VHBA_CHANNEL_OK_CLIENT(ch) \
- (spar_check_channel(ch, spar_vhba_channel_protocol_uuid, \
- "vhba", MIN_IO_CHANNEL_SIZE, \
- ULTRA_VHBA_CHANNEL_PROTOCOL_VERSIONID, \
- ULTRA_VHBA_CHANNEL_PROTOCOL_SIGNATURE))
-
-#define SPAR_VNIC_CHANNEL_OK_CLIENT(ch) \
- (spar_check_channel(ch, spar_vnic_channel_protocol_uuid, \
- "vnic", MIN_IO_CHANNEL_SIZE, \
- ULTRA_VNIC_CHANNEL_PROTOCOL_VERSIONID, \
- ULTRA_VNIC_CHANNEL_PROTOCOL_SIGNATURE))
+#define VISOR_VHBA_CHANNEL_VERSIONID 2
+#define VISOR_VNIC_CHANNEL_VERSIONID 2
+#define VISOR_VSWITCH_CHANNEL_VERSIONID 1
+
+#define VISOR_VHBA_CHANNEL_OK_CLIENT(ch) \
+ (visor_check_channel(ch, visor_vhba_channel_uuid, \
+ "vhba", MIN_IO_CHANNEL_SIZE, \
+ VISOR_VHBA_CHANNEL_VERSIONID, \
+ VISOR_VHBA_CHANNEL_SIGNATURE))
+
+#define VISOR_VNIC_CHANNEL_OK_CLIENT(ch) \
+ (visor_check_channel(ch, visor_vnic_channel_uuid, \
+ "vnic", MIN_IO_CHANNEL_SIZE, \
+ VISOR_VNIC_CHANNEL_VERSIONID, \
+ VISOR_VNIC_CHANNEL_SIGNATURE))
/*
* Everything necessary to handle SCSI & NIC traffic between Guest Partition and
* this header there is a large region of memory which contains the command and
* response queues as specified in cmd_q and rsp_q SIGNAL_QUEUE_HEADERS.
*/
-struct spar_io_channel_protocol {
+struct visor_io_channel {
struct channel_header channel_header;
struct signal_queue_header cmd_q;
struct signal_queue_header rsp_q;
#include "channel.h"
/* {2B3C2D10-7EF5-4ad8-B966-3448B7386B3D} */
-#define SPAR_CONTROLVM_CHANNEL_PROTOCOL_UUID \
+#define VISOR_CONTROLVM_CHANNEL_UUID \
UUID_LE(0x2b3c2d10, 0x7ef5, 0x4ad8, \
0xb9, 0x66, 0x34, 0x48, 0xb7, 0x38, 0x6b, 0x3d)
-#define ULTRA_CONTROLVM_CHANNEL_PROTOCOL_SIGNATURE \
- ULTRA_CHANNEL_PROTOCOL_SIGNATURE
+#define VISOR_CONTROLVM_CHANNEL_SIGNATURE VISOR_CHANNEL_SIGNATURE
#define CONTROLVM_MESSAGE_MAX 64
/* Must increment this whenever you insert or delete fields within
* software. Note that you can usually add fields to the END of the
* channel struct withOUT needing to increment this.
*/
-#define ULTRA_CONTROLVM_CHANNEL_PROTOCOL_VERSIONID 1
+#define VISOR_CONTROLVM_CHANNEL_VERSIONID 1
-#define SPAR_CONTROLVM_CHANNEL_OK_CLIENT(ch) \
- (spar_check_channel(ch, \
- SPAR_CONTROLVM_CHANNEL_PROTOCOL_UUID, \
- "controlvm", \
- sizeof(struct spar_controlvm_channel_protocol), \
- ULTRA_CONTROLVM_CHANNEL_PROTOCOL_VERSIONID, \
- ULTRA_CONTROLVM_CHANNEL_PROTOCOL_SIGNATURE))
+#define VISOR_CONTROLVM_CHANNEL_OK_CLIENT(ch) \
+ (visor_check_channel(ch, \
+ VISOR_CONTROLVM_CHANNEL_UUID, \
+ "controlvm", \
+ sizeof(struct visor_controlvm_channel), \
+ VISOR_CONTROLVM_CHANNEL_VERSIONID, \
+ VISOR_CONTROLVM_CHANNEL_SIGNATURE))
/* Defines for various channel queues */
#define CONTROLVM_QUEUE_REQUEST 0
/* Max num of messages stored during IOVM creation to be reused after crash */
#define CONTROLVM_CRASHMSG_MAX 2
-struct spar_segment_state {
+struct visor_segment_state {
/* Bit 0: May enter other states */
u16 enabled:1;
/* Bit 1: Assigned to active partition */
*/
} __packed;
-static const struct spar_segment_state segment_state_running = {
+static const struct visor_segment_state segment_state_running = {
1, 1, 1, 0, 1, 1, 1, 1
};
-static const struct spar_segment_state segment_state_paused = {
+static const struct visor_segment_state segment_state_paused = {
1, 1, 1, 0, 1, 1, 1, 0
};
-static const struct spar_segment_state segment_state_standby = {
+static const struct visor_segment_state segment_state_standby = {
1, 1, 0, 0, 1, 1, 1, 0
};
u8 reserved[3]; /* Natural alignment purposes */
} __packed;
-struct efi_spar_indication {
+struct efi_visor_indication {
u64 boot_to_fw_ui:1; /* Bit 0: Stop in uefi ui */
u64 clear_nvram:1; /* Bit 1: Clear NVRAM */
u64 clear_cmos:1; /* Bit 2: Clear CMOS */
u64 reserved:60; /* Natural alignment */
} __packed;
-enum ultra_chipset_feature {
- ULTRA_CHIPSET_FEATURE_REPLY = 0x00000001,
- ULTRA_CHIPSET_FEATURE_PARA_HOTPLUG = 0x00000002,
+enum visor_chipset_feature {
+ VISOR_CHIPSET_FEATURE_REPLY = 0x00000001,
+ VISOR_CHIPSET_FEATURE_PARA_HOTPLUG = 0x00000002,
};
/* This is the common structure that is at the beginning of every
/* for CONTROLVM_DEVICE_RECONFIGURE */
struct {
u32 bus_no;
- struct spar_segment_state state;
+ struct visor_segment_state state;
u8 reserved[2]; /* Natural alignment purposes */
} __packed bus_change_state; /* for CONTROLVM_BUS_CHANGESTATE */
struct {
u32 bus_no;
u32 dev_no;
- struct spar_segment_state state;
+ struct visor_segment_state state;
struct {
/* =1 if message is for a physical device */
u32 phys_device:1;
struct {
u32 bus_no;
u32 dev_no;
- struct spar_segment_state state;
+ struct visor_segment_state state;
u8 reserved[6]; /* Natural alignment purposes */
} __packed device_change_state_event;
/* for CONTROLVM_DEVICE_CHANGESTATE_EVENT */
u32 bus_count;
/* indicates the max number of switches */
u32 switch_count;
- enum ultra_chipset_feature features;
+ enum visor_chipset_feature features;
u32 platform_number; /* Platform Number */
} __packed init_chipset; /* for CONTROLVM_CHIPSET_INIT */
struct {
struct controlvm_message_packet cmd;
} __packed;
-struct spar_controlvm_channel_protocol {
+struct visor_controlvm_channel {
struct channel_header header;
u64 gp_controlvm; /* guest phys addr of this channel */
u64 gp_partition_tables;/* guest phys addr of partition tables */
u32 message_count; /* CONTROLVM_MESSAGE_MAX */
u64 gp_smbios_table; /* guest phys addr of SMBIOS tables */
u64 gp_physical_smbios_table; /* guest phys addr of SMBIOS table */
- /* ULTRA_MAX_GUESTS_PER_SERVICE */
+ /* VISOR_MAX_GUESTS_PER_SERVICE */
char gp_reserved[2688];
/* guest physical address of EFI firmware image base */
u32 installation_text_id; /* Id of string to display */
/* Number of remaining installation steps (for progress bars) */
u16 installation_remaining_steps;
- /* ULTRA_TOOL_ACTIONS Installation Action field */
+ /* VISOR_TOOL_ACTIONS Installation Action field */
u8 tool_action;
u8 reserved; /* alignment */
- struct efi_spar_indication efi_spar_ind;
- struct efi_spar_indication efi_spar_ind_supported;
+ struct efi_visor_indication efi_visor_ind;
u32 sp_reserved;
/* Force signals to begin on 128-byte cache line */
u8 reserved2[28];
* of total_length should equal PayloadBytes. The format of the strings at
* PayloadVmOffset will take different forms depending on the message.
*/
-struct spar_controlvm_parameters_header {
+struct visor_controlvm_parameters_header {
u32 total_length;
u32 header_length;
u32 connection_offset;
#include "channel.h"
/* {193b331b-c58f-11da-95a9-00e08161165f} */
-#define SPAR_VBUS_CHANNEL_PROTOCOL_UUID \
+#define VISOR_VBUS_CHANNEL_UUID \
UUID_LE(0x193b331b, 0xc58f, 0x11da, \
0x95, 0xa9, 0x0, 0xe0, 0x81, 0x61, 0x16, 0x5f)
-static const uuid_le spar_vbus_channel_protocol_uuid =
- SPAR_VBUS_CHANNEL_PROTOCOL_UUID;
+static const uuid_le visor_vbus_channel_uuid = VISOR_VBUS_CHANNEL_UUID;
-#define SPAR_VBUS_CHANNEL_PROTOCOL_SIGNATURE ULTRA_CHANNEL_PROTOCOL_SIGNATURE
+#define VISOR_VBUS_CHANNEL_SIGNATURE VISOR_CHANNEL_SIGNATURE
/* Must increment this whenever you insert or delete fields within this channel
* struct. Also increment whenever you change the meaning of fields within this
* usually add fields to the END of the channel struct withOUT needing to
* increment this.
*/
-#define SPAR_VBUS_CHANNEL_PROTOCOL_VERSIONID 1
+#define VISOR_VBUS_CHANNEL_VERSIONID 1
/*
* An array of this struct is present in the channel area for each vbus.
* It is filled in by the client side to provide info about the device
* and driver from the client's perspective.
*/
-struct ultra_vbus_deviceinfo {
+struct visor_vbus_deviceinfo {
u8 devtype[16]; /* short string identifying the device type */
u8 drvname[16]; /* driver .sys file name */
u8 infostrs[96]; /* kernel version */
u8 reserved[128]; /* pad size to 256 bytes */
} __packed;
-struct spar_vbus_headerinfo {
+struct visor_vbus_headerinfo {
u32 struct_bytes; /* size of this struct in bytes */
- u32 device_info_struct_bytes; /* sizeof(ULTRA_VBUS_DEVICEINFO) */
+ u32 device_info_struct_bytes; /* sizeof(VISOR_VBUS_DEVICEINFO) */
u32 dev_info_count; /* num of items in DevInfo member */
/* (this is the allocated size) */
u32 chp_info_offset; /* byte offset from beginning of this struct */
u8 reserved[104];
} __packed;
-struct spar_vbus_channel_protocol {
+struct visor_vbus_channel {
struct channel_header channel_header; /* initialized by server */
- struct spar_vbus_headerinfo hdr_info; /* initialized by server */
+ struct visor_vbus_headerinfo hdr_info; /* initialized by server */
/* the remainder of this channel is filled in by the client */
- struct ultra_vbus_deviceinfo chp_info;
+ struct visor_vbus_deviceinfo chp_info;
/* describes client chipset device and driver */
- struct ultra_vbus_deviceinfo bus_info;
+ struct visor_vbus_deviceinfo bus_info;
/* describes client bus device and driver */
- struct ultra_vbus_deviceinfo dev_info[0];
+ struct visor_vbus_deviceinfo dev_info[0];
/* describes client device and driver for each device on the bus */
} __packed;
};
/* filled in with info about parent chipset driver when we register with it */
-static struct ultra_vbus_deviceinfo chipset_driverinfo;
+static struct visor_vbus_deviceinfo chipset_driverinfo;
/* filled in with info about this driver, wrt it servicing client busses */
-static struct ultra_vbus_deviceinfo clientbus_driverinfo;
+static struct visor_vbus_deviceinfo clientbus_driverinfo;
/* list of visor_device structs, linked via .list_all */
static LIST_HEAD(list_all_bus_instances);
* /sys/kernel/debug/visorbus/visorbus<n>.
*/
/*
- * vbuschannel_print_devinfo() - format a struct ultra_vbus_deviceinfo
+ * vbuschannel_print_devinfo() - format a struct visor_vbus_deviceinfo
* and write it to a seq_file
- * @devinfo: the struct ultra_vbus_deviceinfo to format
+ * @devinfo: the struct visor_vbus_deviceinfo to format
* @seq: seq_file to write to
* @devix: the device index to be included in the output data, or -1 if no
* device index is to be included
* Reads @devInfo, and writes it in human-readable notation to @seq.
*/
static void
-vbuschannel_print_devinfo(struct ultra_vbus_deviceinfo *devinfo,
+vbuschannel_print_devinfo(struct visor_vbus_deviceinfo *devinfo,
struct seq_file *seq, int devix)
{
if (!isprint(devinfo->devtype[0]))
int i;
unsigned long off;
- struct ultra_vbus_deviceinfo dev_info;
+ struct visor_vbus_deviceinfo dev_info;
if (!channel)
return 0;
((vdev->name) ? (char *)(vdev->name) : ""),
vdev->chipset_bus_no);
if (visorchannel_read(channel,
- offsetof(struct spar_vbus_channel_protocol,
- chp_info),
+ offsetof(struct visor_vbus_channel, chp_info),
&dev_info, sizeof(dev_info)) >= 0)
vbuschannel_print_devinfo(&dev_info, seq, -1);
if (visorchannel_read(channel,
- offsetof(struct spar_vbus_channel_protocol,
- bus_info),
+ offsetof(struct visor_vbus_channel, bus_info),
&dev_info, sizeof(dev_info)) >= 0)
vbuschannel_print_devinfo(&dev_info, seq, -1);
- off = offsetof(struct spar_vbus_channel_protocol, dev_info);
+ off = offsetof(struct visor_vbus_channel, dev_info);
i = 0;
while (off + sizeof(dev_info) <= visorchannel_get_nbytes(channel)) {
if (visorchannel_read(channel, off, &dev_info,
static int
get_vbus_header_info(struct visorchannel *chan,
- struct spar_vbus_headerinfo *hdr_info)
+ struct visor_vbus_headerinfo *hdr_info)
{
int err;
- if (!spar_check_channel(visorchannel_get_header(chan),
- spar_vbus_channel_protocol_uuid,
- "vbus",
- sizeof(struct spar_vbus_channel_protocol),
- SPAR_VBUS_CHANNEL_PROTOCOL_VERSIONID,
- SPAR_VBUS_CHANNEL_PROTOCOL_SIGNATURE))
+ if (!visor_check_channel(visorchannel_get_header(chan),
+ visor_vbus_channel_uuid,
+ "vbus",
+ sizeof(struct visor_vbus_channel),
+ VISOR_VBUS_CHANNEL_VERSIONID,
+ VISOR_VBUS_CHANNEL_SIGNATURE))
return -EINVAL;
err = visorchannel_read(chan, sizeof(struct channel_header), hdr_info,
if (err < 0)
return err;
- if (hdr_info->struct_bytes < sizeof(struct spar_vbus_headerinfo))
+ if (hdr_info->struct_bytes < sizeof(struct visor_vbus_headerinfo))
return -EINVAL;
if (hdr_info->device_info_struct_bytes <
- sizeof(struct ultra_vbus_deviceinfo))
+ sizeof(struct visor_vbus_deviceinfo))
return -EINVAL;
return 0;
/*
* write_vbus_chp_info() - write the contents of <info> to the struct
- * spar_vbus_channel_protocol.chp_info
+ * visor_vbus_channel.chp_info
* @chan: indentifies the s-Par channel that will be updated
* @hdr_info: used to find appropriate channel offset to write data
* @info: contains the information to write
*/
static void
write_vbus_chp_info(struct visorchannel *chan,
- struct spar_vbus_headerinfo *hdr_info,
- struct ultra_vbus_deviceinfo *info)
+ struct visor_vbus_headerinfo *hdr_info,
+ struct visor_vbus_deviceinfo *info)
{
int off = sizeof(struct channel_header) + hdr_info->chp_info_offset;
/*
* write_vbus_bus_info() - write the contents of <info> to the struct
- * spar_vbus_channel_protocol.bus_info
+ * visor_vbus_channel.bus_info
* @chan: indentifies the s-Par channel that will be updated
* @hdr_info: used to find appropriate channel offset to write data
* @info: contains the information to write
*/
static void
write_vbus_bus_info(struct visorchannel *chan,
- struct spar_vbus_headerinfo *hdr_info,
- struct ultra_vbus_deviceinfo *info)
+ struct visor_vbus_headerinfo *hdr_info,
+ struct visor_vbus_deviceinfo *info)
{
int off = sizeof(struct channel_header) + hdr_info->bus_info_offset;
/*
* write_vbus_dev_info() - write the contents of <info> to the struct
- * spar_vbus_channel_protocol.dev_info[<devix>]
+ * visor_vbus_channel.dev_info[<devix>]
* @chan: indentifies the s-Par channel that will be updated
* @hdr_info: used to find appropriate channel offset to write data
* @info: contains the information to write
*/
static void
write_vbus_dev_info(struct visorchannel *chan,
- struct spar_vbus_headerinfo *hdr_info,
- struct ultra_vbus_deviceinfo *info, unsigned int devix)
+ struct visor_vbus_headerinfo *hdr_info,
+ struct visor_vbus_deviceinfo *info, unsigned int devix)
{
int off =
(sizeof(struct channel_header) + hdr_info->dev_info_offset) +
}
static void bus_device_info_init(
- struct ultra_vbus_deviceinfo *bus_device_info_ptr,
+ struct visor_vbus_deviceinfo *bus_device_info_ptr,
const char *dev_type, const char *drv_name)
{
- memset(bus_device_info_ptr, 0, sizeof(struct ultra_vbus_deviceinfo));
+ memset(bus_device_info_ptr, 0, sizeof(struct visor_vbus_deviceinfo));
snprintf(bus_device_info_ptr->devtype,
sizeof(bus_device_info_ptr->devtype),
"%s", (dev_type) ? dev_type : "unknownType");
struct visor_driver *visordrv;
u32 bus_no = visordev->chipset_bus_no;
u32 dev_no = visordev->chipset_dev_no;
- struct ultra_vbus_deviceinfo dev_info;
+ struct visor_vbus_deviceinfo dev_info;
const char *chan_type_name = NULL;
- struct spar_vbus_headerinfo *hdr_info;
+ struct visor_vbus_headerinfo *hdr_info;
if (!visordev->device.driver)
return;
bdev = visorbus_get_device_by_id(bus_no, BUS_ROOT_DEVICE, NULL);
if (!bdev)
return;
- hdr_info = (struct spar_vbus_headerinfo *)bdev->vbus_hdr_info;
+ hdr_info = (struct visor_vbus_headerinfo *)bdev->vbus_hdr_info;
if (!hdr_info)
return;
visordrv = to_visor_driver(visordev->device.driver);
EXPORT_SYMBOL_GPL(visorbus_register_visor_driver);
/*
- * create_bus_instance() - create a device instance for the visor bus itself
+ * visorbus_create_instance() - create a device instance for the visorbus itself
* @dev: struct visor_device indicating the bus instance
*
* Return: 0 for success, otherwise negative errno value indicating reason for
* failure
*/
static int
-create_bus_instance(struct visor_device *dev)
+visorbus_create_instance(struct visor_device *dev)
{
int id = dev->chipset_bus_no;
int err;
- struct spar_vbus_headerinfo *hdr_info;
+ struct visor_vbus_headerinfo *hdr_info;
hdr_info = kzalloc(sizeof(*hdr_info), GFP_KERNEL);
if (!hdr_info)
err_debugfs_dir:
debugfs_remove_recursive(dev->debugfs_dir);
kfree(hdr_info);
- dev_err(&dev->device, "create_bus_instance failed: %d\n", err);
+ dev_err(&dev->device, "visorbus_create_instance failed: %d\n", err);
return err;
}
/*
- * remove_bus_instance() - remove a device instance for the visor bus itself
+ * visorbus_remove_instance() - remove a device instance for the visorbus itself
* @dev: struct visor_device indentifying the bus to remove
*/
static void
-remove_bus_instance(struct visor_device *dev)
+visorbus_remove_instance(struct visor_device *dev)
{
/*
* Note that this will result in the release method for
}
/*
- * remove_all_visor_devices() - remove all child visor bus device instances
+ * remove_all_visor_devices() - remove all child visorbus device instances
*/
static void
remove_all_visor_devices(void)
}
int
-chipset_bus_create(struct visor_device *dev)
+visorchipset_bus_create(struct visor_device *dev)
{
int err;
- err = create_bus_instance(dev);
+ err = visorbus_create_instance(dev);
if (err < 0)
return err;
- bus_create_response(dev, err);
+ visorbus_create_response(dev, err);
return 0;
}
void
-chipset_bus_destroy(struct visor_device *dev)
+visorchipset_bus_destroy(struct visor_device *dev)
{
- remove_bus_instance(dev);
- bus_destroy_response(dev, 0);
+ visorbus_remove_instance(dev);
+ visorbus_destroy_response(dev, 0);
}
int
-chipset_device_create(struct visor_device *dev_info)
+visorchipset_device_create(struct visor_device *dev_info)
{
int err;
if (err < 0)
return err;
- device_create_response(dev_info, err);
+ visorbus_device_create_response(dev_info, err);
return 0;
}
void
-chipset_device_destroy(struct visor_device *dev_info)
+visorchipset_device_destroy(struct visor_device *dev_info)
{
remove_visor_device(dev_info);
- device_destroy_response(dev_info, 0);
+ visorbus_device_destroy_response(dev_info, 0);
}
/*
dev->pausing = false;
- device_pause_response(dev, status);
+ visorbus_device_pause_response(dev, status);
}
/*
* which will presumably want to send some sort of response to
* the initiator.
*/
- device_resume_response(dev, status);
+ visorbus_device_resume_response(dev, status);
}
/*
- * initiate_chipset_device_pause_resume() - start a pause or resume operation
- * for a visor device
+ * visorchipset_initiate_device_pause_resume() - start a pause or resume
+ * operation for a visor device
* @dev: struct visor_device identifying the device being paused or resumed
* @is_pause: true to indicate pause operation, false to indicate resume
*
* resume_state_change_complete().
*/
static int
-initiate_chipset_device_pause_resume(struct visor_device *dev, bool is_pause)
+visorchipset_initiate_device_pause_resume(struct visor_device *dev,
+ bool is_pause)
{
int err;
struct visor_driver *drv = NULL;
}
/**
- * chipset_device_pause() - start a pause operation for a visor device
+ * visorchipset_device_pause() - start a pause operation for a visor device
* @dev_info: struct visor_device identifying the device being paused
*
* Tell the subordinate function driver for a specific device to pause
* via a callback function; see pause_state_change_complete().
*/
int
-chipset_device_pause(struct visor_device *dev_info)
+visorchipset_device_pause(struct visor_device *dev_info)
{
int err;
- err = initiate_chipset_device_pause_resume(dev_info, true);
+ err = visorchipset_initiate_device_pause_resume(dev_info, true);
if (err < 0) {
dev_info->pausing = false;
}
/**
- * chipset_device_resume() - start a resume operation for a visor device
+ * visorchipset_device_resume() - start a resume operation for a visor device
* @dev_info: struct visor_device identifying the device being resumed
*
* Tell the subordinate function driver for a specific device to resume
* via a callback function; see resume_state_change_complete().
*/
int
-chipset_device_resume(struct visor_device *dev_info)
+visorchipset_device_resume(struct visor_device *dev_info)
{
int err;
- err = initiate_chipset_device_pause_resume(dev_info, false);
+ err = visorchipset_initiate_device_pause_resume(dev_info, false);
if (err < 0) {
dev_info->resuming = false;
struct visor_device *dev = list_entry(listentry,
struct visor_device,
list_all);
- remove_bus_instance(dev);
+ visorbus_remove_instance(dev);
}
bus_unregister(&visorbus_type);
* command line
*/
-int chipset_bus_create(struct visor_device *bus_info);
-void chipset_bus_destroy(struct visor_device *bus_info);
-int chipset_device_create(struct visor_device *dev_info);
-void chipset_device_destroy(struct visor_device *dev_info);
-int chipset_device_pause(struct visor_device *dev_info);
-int chipset_device_resume(struct visor_device *dev_info);
+int visorchipset_bus_create(struct visor_device *bus_info);
+void visorchipset_bus_destroy(struct visor_device *bus_info);
+int visorchipset_device_create(struct visor_device *dev_info);
+void visorchipset_device_destroy(struct visor_device *dev_info);
+int visorchipset_device_pause(struct visor_device *dev_info);
+int visorchipset_device_resume(struct visor_device *dev_info);
-void bus_create_response(struct visor_device *p, int response);
-void bus_destroy_response(struct visor_device *p, int response);
-void device_create_response(struct visor_device *p, int response);
-void device_destroy_response(struct visor_device *p, int response);
-void device_resume_response(struct visor_device *p, int response);
-void device_pause_response(struct visor_device *p, int response);
+void visorbus_create_response(struct visor_device *p, int response);
+void visorbus_destroy_response(struct visor_device *p, int response);
+void visorbus_device_create_response(struct visor_device *p, int response);
+void visorbus_device_destroy_response(struct visor_device *p, int response);
+void visorbus_device_resume_response(struct visor_device *p, int response);
+void visorbus_device_pause_response(struct visor_device *p, int response);
int visorbus_init(void);
void visorbus_exit(void);
#define MYDRVNAME "visorchannel"
-#define SPAR_CONSOLEVIDEO_CHANNEL_PROTOCOL_GUID \
+#define VISOR_CONSOLEVIDEO_CHANNEL_GUID \
UUID_LE(0x3cd6e705, 0xd6a2, 0x4aa5, \
0xad, 0x5c, 0x7b, 0x8, 0x88, 0x9d, 0xff, 0xe2)
-static const uuid_le spar_video_guid = SPAR_CONSOLEVIDEO_CHANNEL_PROTOCOL_GUID;
+static const uuid_le visor_video_guid = VISOR_CONSOLEVIDEO_CHANNEL_GUID;
struct visorchannel {
u64 physaddr;
* release later on.
*/
channel->requested = request_mem_region(physaddr, size, MYDRVNAME);
- if (!channel->requested && uuid_le_cmp(guid, spar_video_guid))
+ if (!channel->requested && uuid_le_cmp(guid, visor_video_guid))
/* we only care about errors if this is not the video channel */
goto err_destroy_channel;
channel->mapped = NULL;
channel->requested = request_mem_region(channel->physaddr,
channel_bytes, MYDRVNAME);
- if (!channel->requested && uuid_le_cmp(guid, spar_video_guid))
+ if (!channel->requested && uuid_le_cmp(guid, visor_video_guid))
/* we only care about errors if this is not the video channel */
goto err_destroy_channel;
#define MAX_CONTROLVM_PAYLOAD_BYTES (1024 * 128)
-#define UNISYS_SPAR_LEAF_ID 0x40000000
+#define UNISYS_VISOR_LEAF_ID 0x40000000
/* The s-Par leaf ID returns "UnisysSpar64" encoded across ebx, ecx, edx */
-#define UNISYS_SPAR_ID_EBX 0x73696e55
-#define UNISYS_SPAR_ID_ECX 0x70537379
-#define UNISYS_SPAR_ID_EDX 0x34367261
+#define UNISYS_VISOR_ID_EBX 0x73696e55
+#define UNISYS_VISOR_ID_ECX 0x70537379
+#define UNISYS_VISOR_ID_EDX 0x34367261
/*
* When the controlvm channel is idle for at least MIN_IDLE_SECONDS,
int err;
err = visorchannel_read(chipset_dev->controlvm_channel,
- offsetof(struct spar_controlvm_channel_protocol,
+ offsetof(struct visor_controlvm_channel,
tool_action),
&tool_action, sizeof(u8));
if (err)
if (kstrtou8(buf, 10, &tool_action))
return -EINVAL;
- err = visorchannel_write
- (chipset_dev->controlvm_channel,
- offsetof(struct spar_controlvm_channel_protocol,
- tool_action),
- &tool_action, sizeof(u8));
+ err = visorchannel_write(chipset_dev->controlvm_channel,
+ offsetof(struct visor_controlvm_channel,
+ tool_action),
+ &tool_action, sizeof(u8));
if (err)
return err;
struct device_attribute *attr,
char *buf)
{
- struct efi_spar_indication efi_spar_indication;
+ struct efi_visor_indication efi_visor_indication;
int err;
err = visorchannel_read(chipset_dev->controlvm_channel,
- offsetof(struct spar_controlvm_channel_protocol,
- efi_spar_ind),
- &efi_spar_indication,
- sizeof(struct efi_spar_indication));
+ offsetof(struct visor_controlvm_channel,
+ efi_visor_ind),
+ &efi_visor_indication,
+ sizeof(struct efi_visor_indication));
if (err)
return err;
- return sprintf(buf, "%u\n", efi_spar_indication.boot_to_tool);
+ return sprintf(buf, "%u\n", efi_visor_indication.boot_to_tool);
}
static ssize_t boottotool_store(struct device *dev,
const char *buf, size_t count)
{
int val, err;
- struct efi_spar_indication efi_spar_indication;
+ struct efi_visor_indication efi_visor_indication;
if (kstrtoint(buf, 10, &val))
return -EINVAL;
- efi_spar_indication.boot_to_tool = val;
- err = visorchannel_write
- (chipset_dev->controlvm_channel,
- offsetof(struct spar_controlvm_channel_protocol,
- efi_spar_ind), &(efi_spar_indication),
- sizeof(struct efi_spar_indication));
+ efi_visor_indication.boot_to_tool = val;
+ err = visorchannel_write(chipset_dev->controlvm_channel,
+ offsetof(struct visor_controlvm_channel,
+ efi_visor_ind),
+ &(efi_visor_indication),
+ sizeof(struct efi_visor_indication));
if (err)
return err;
int err;
err = visorchannel_read(chipset_dev->controlvm_channel,
- offsetof(struct spar_controlvm_channel_protocol,
+ offsetof(struct visor_controlvm_channel,
installation_error),
&error, sizeof(u32));
if (err)
if (kstrtou32(buf, 10, &error))
return -EINVAL;
- err = visorchannel_write
- (chipset_dev->controlvm_channel,
- offsetof(struct spar_controlvm_channel_protocol,
- installation_error),
- &error, sizeof(u32));
+ err = visorchannel_write(chipset_dev->controlvm_channel,
+ offsetof(struct visor_controlvm_channel,
+ installation_error),
+ &error, sizeof(u32));
if (err)
return err;
return count;
u32 text_id = 0;
int err;
- err = visorchannel_read
- (chipset_dev->controlvm_channel,
- offsetof(struct spar_controlvm_channel_protocol,
- installation_text_id),
- &text_id, sizeof(u32));
+ err = visorchannel_read(chipset_dev->controlvm_channel,
+ offsetof(struct visor_controlvm_channel,
+ installation_text_id),
+ &text_id, sizeof(u32));
if (err)
return err;
if (kstrtou32(buf, 10, &text_id))
return -EINVAL;
- err = visorchannel_write
- (chipset_dev->controlvm_channel,
- offsetof(struct spar_controlvm_channel_protocol,
- installation_text_id),
- &text_id, sizeof(u32));
+ err = visorchannel_write(chipset_dev->controlvm_channel,
+ offsetof(struct visor_controlvm_channel,
+ installation_text_id),
+ &text_id, sizeof(u32));
if (err)
return err;
return count;
int err;
err = visorchannel_read(chipset_dev->controlvm_channel,
- offsetof(struct spar_controlvm_channel_protocol,
+ offsetof(struct visor_controlvm_channel,
installation_remaining_steps),
&remaining_steps, sizeof(u16));
if (err)
if (kstrtou16(buf, 10, &remaining_steps))
return -EINVAL;
- err = visorchannel_write
- (chipset_dev->controlvm_channel,
- offsetof(struct spar_controlvm_channel_protocol,
- installation_remaining_steps),
- &remaining_steps, sizeof(u16));
+ err = visorchannel_write(chipset_dev->controlvm_channel,
+ offsetof(struct visor_controlvm_channel,
+ installation_remaining_steps),
+ &remaining_steps, sizeof(u16));
if (err)
return err;
return count;
static uuid_le
parser_id_get(struct parser_context *ctx)
{
- struct spar_controlvm_parameters_header *phdr = NULL;
+ struct visor_controlvm_parameters_header *phdr = NULL;
- phdr = (struct spar_controlvm_parameters_header *)(ctx->data);
+ phdr = (struct visor_controlvm_parameters_header *)(ctx->data);
return phdr->id;
}
static void *
parser_name_get(struct parser_context *ctx)
{
- struct spar_controlvm_parameters_header *phdr = NULL;
+ struct visor_controlvm_parameters_header *phdr = NULL;
- phdr = (struct spar_controlvm_parameters_header *)(ctx->data);
+ phdr = (struct visor_controlvm_parameters_header *)(ctx->data);
if (phdr->name_offset + phdr->name_length > ctx->param_bytes)
return NULL;
static int
controlvm_respond_chipset_init(struct controlvm_message_header *msg_hdr,
int response,
- enum ultra_chipset_feature features)
+ enum visor_chipset_feature features)
{
struct controlvm_message outmsg;
chipset_init(struct controlvm_message *inmsg)
{
static int chipset_inited;
- enum ultra_chipset_feature features = 0;
+ enum visor_chipset_feature features = 0;
int rc = CONTROLVM_RESP_SUCCESS;
int res = 0;
* also supports it).
*/
features = inmsg->cmd.init_chipset.features &
- ULTRA_CHIPSET_FEATURE_PARA_HOTPLUG;
+ VISOR_CHIPSET_FEATURE_PARA_HOTPLUG;
/*
* Set the "reply" bit so Command knows this is a
* features-aware driver.
*/
- features |= ULTRA_CHIPSET_FEATURE_REPLY;
+ features |= VISOR_CHIPSET_FEATURE_REPLY;
out_respond:
if (inmsg->hdr.flags.response_expected)
static int
controlvm_respond(struct controlvm_message_header *msg_hdr, int response,
- struct spar_segment_state *state)
+ struct visor_segment_state *state)
{
struct controlvm_message outmsg;
};
static int
-save_crash_message(struct controlvm_message *msg, enum crash_obj_type typ)
+save_crash_message(struct controlvm_message *msg, enum crash_obj_type cr_type)
{
u32 local_crash_msg_offset;
u16 local_crash_msg_count;
int err;
err = visorchannel_read(chipset_dev->controlvm_channel,
- offsetof(struct spar_controlvm_channel_protocol,
+ offsetof(struct visor_controlvm_channel,
saved_crash_message_count),
&local_crash_msg_count, sizeof(u16));
if (err) {
}
err = visorchannel_read(chipset_dev->controlvm_channel,
- offsetof(struct spar_controlvm_channel_protocol,
+ offsetof(struct visor_controlvm_channel,
saved_crash_message_offset),
&local_crash_msg_offset, sizeof(u32));
if (err) {
return err;
}
- switch (typ) {
+ switch (cr_type) {
case CRASH_DEV:
local_crash_msg_offset += sizeof(struct controlvm_message);
err = visorchannel_write(chipset_dev->controlvm_channel,
static int
device_changestate_responder(enum controlvm_id cmd_id,
struct visor_device *p, int response,
- struct spar_segment_state response_state)
+ struct visor_segment_state response_state)
{
struct controlvm_message outmsg;
u32 bus_no = p->chipset_bus_no;
}
static int
-bus_create(struct controlvm_message *inmsg)
+visorbus_create(struct controlvm_message *inmsg)
{
struct controlvm_message_packet *cmd = &inmsg->cmd;
struct controlvm_message_header *pmsg_hdr = NULL;
bus_info = visorbus_get_device_by_id(bus_no, BUS_ROOT_DEVICE, NULL);
if (bus_info && (bus_info->state.created == 1)) {
dev_err(&chipset_dev->acpi_device->dev,
- "failed bus_create: already exists\n");
+ "failed visorbus_create: already exists\n");
err = -EEXIST;
goto err_respond;
}
bus_info->chipset_bus_no = bus_no;
bus_info->chipset_dev_no = BUS_ROOT_DEVICE;
- if (uuid_le_cmp(cmd->create_bus.bus_inst_uuid, spar_siovm_uuid) == 0) {
+ if (uuid_le_cmp(cmd->create_bus.bus_inst_uuid, visor_siovm_uuid) == 0) {
err = save_crash_message(inmsg, CRASH_BUS);
if (err)
goto err_free_bus_info;
}
bus_info->visorchannel = visorchannel;
- /* Response will be handled by chipset_bus_create */
- err = chipset_bus_create(bus_info);
- /* If error chipset_bus_create didn't respond, need to respond here */
+ /* Response will be handled by visorchipset_bus_create */
+ err = visorchipset_bus_create(bus_info);
+ /* If visorchipset_bus_create didn't respond, need to respond here */
if (err)
goto err_destroy_channel;
}
static int
-bus_destroy(struct controlvm_message *inmsg)
+visorbus_destroy(struct controlvm_message *inmsg)
{
struct controlvm_message_packet *cmd = &inmsg->cmd;
struct controlvm_message_header *pmsg_hdr = NULL;
bus_info->pending_msg_hdr = pmsg_hdr;
}
- /* Response will be handled by chipset_bus_destroy */
- chipset_bus_destroy(bus_info);
+ /* Response will be handled by visorchipset_bus_destroy */
+ visorchipset_bus_destroy(bus_info);
return 0;
err_respond:
}
static int
-bus_configure(struct controlvm_message *inmsg,
- struct parser_context *parser_ctx)
+visorbus_configure(struct controlvm_message *inmsg,
+ struct parser_context *parser_ctx)
{
struct controlvm_message_packet *cmd = &inmsg->cmd;
u32 bus_no;
err_respond:
dev_err(&chipset_dev->acpi_device->dev,
- "bus_configured exited with err: %d\n", err);
+ "visorbus_configure exited with err: %d\n", err);
if (inmsg->hdr.flags.response_expected == 1)
controlvm_responder(inmsg->hdr.id, &inmsg->hdr, err);
return err;
}
static int
-my_device_create(struct controlvm_message *inmsg)
+visorbus_device_create(struct controlvm_message *inmsg)
{
struct controlvm_message_packet *cmd = &inmsg->cmd;
struct controlvm_message_header *pmsg_hdr = NULL;
dev_info->visorchannel = visorchannel;
dev_info->channel_type_guid = cmd->create_device.data_type_uuid;
if (uuid_le_cmp(cmd->create_device.data_type_uuid,
- spar_vhba_channel_protocol_uuid) == 0) {
+ visor_vhba_channel_uuid) == 0) {
err = save_crash_message(inmsg, CRASH_DEV);
if (err)
goto err_destroy_visorchannel;
sizeof(struct controlvm_message_header));
dev_info->pending_msg_hdr = pmsg_hdr;
}
- /* Chipset_device_create will send response */
- err = chipset_device_create(dev_info);
+ /* visorchipset_device_create will send response */
+ err = visorchipset_device_create(dev_info);
if (err)
goto err_destroy_visorchannel;
}
static int
-my_device_changestate(struct controlvm_message *inmsg)
+visorbus_device_changestate(struct controlvm_message *inmsg)
{
struct controlvm_message_packet *cmd = &inmsg->cmd;
struct controlvm_message_header *pmsg_hdr = NULL;
u32 bus_no = cmd->device_change_state.bus_no;
u32 dev_no = cmd->device_change_state.dev_no;
- struct spar_segment_state state = cmd->device_change_state.state;
+ struct visor_segment_state state = cmd->device_change_state.state;
struct visor_device *dev_info;
int err = 0;
if (state.alive == segment_state_running.alive &&
state.operating == segment_state_running.operating)
- /* Response will be sent from chipset_device_resume */
- err = chipset_device_resume(dev_info);
+ /* Response will be sent from visorchipset_device_resume */
+ err = visorchipset_device_resume(dev_info);
/* ServerNotReady / ServerLost / SegmentStateStandby */
else if (state.alive == segment_state_standby.alive &&
state.operating == segment_state_standby.operating)
/*
* technically this is standby case where server is lost.
- * Response will be sent from chipset_device_pause.
+ * Response will be sent from visorchipset_device_pause.
*/
- err = chipset_device_pause(dev_info);
+ err = visorchipset_device_pause(dev_info);
if (err)
goto err_respond;
}
static int
-my_device_destroy(struct controlvm_message *inmsg)
+visorbus_device_destroy(struct controlvm_message *inmsg)
{
struct controlvm_message_packet *cmd = &inmsg->cmd;
struct controlvm_message_header *pmsg_hdr = NULL;
dev_info->pending_msg_hdr = pmsg_hdr;
}
- chipset_device_destroy(dev_info);
+ visorchipset_device_destroy(dev_info);
return 0;
err_respond:
env_cmd, env_id, env_state, env_bus, env_dev, env_func, NULL
};
- sprintf(env_cmd, "SPAR_PARAHOTPLUG=1");
- sprintf(env_id, "SPAR_PARAHOTPLUG_ID=%d", req->id);
- sprintf(env_state, "SPAR_PARAHOTPLUG_STATE=%d",
+ sprintf(env_cmd, "VISOR_PARAHOTPLUG=1");
+ sprintf(env_id, "VISOR_PARAHOTPLUG_ID=%d", req->id);
+ sprintf(env_state, "VISOR_PARAHOTPLUG_STATE=%d",
cmd->device_change_state.state.active);
- sprintf(env_bus, "SPAR_PARAHOTPLUG_BUS=%d",
+ sprintf(env_bus, "VISOR_PARAHOTPLUG_BUS=%d",
cmd->device_change_state.bus_no);
- sprintf(env_dev, "SPAR_PARAHOTPLUG_DEVICE=%d",
+ sprintf(env_dev, "VISOR_PARAHOTPLUG_DEVICE=%d",
cmd->device_change_state.dev_no >> 3);
- sprintf(env_func, "SPAR_PARAHOTPLUG_FUNCTION=%d",
+ sprintf(env_func, "VISOR_PARAHOTPLUG_FUNCTION=%d",
cmd->device_change_state.dev_no & 0x7);
return kobject_uevent_env(&chipset_dev->acpi_device->dev.kobj,
/* get saved message count */
if (visorchannel_read(chipset_dev->controlvm_channel,
- offsetof(struct spar_controlvm_channel_protocol,
+ offsetof(struct visor_controlvm_channel,
saved_crash_message_count),
&local_crash_msg_count, sizeof(u16)) < 0) {
dev_err(&chipset_dev->acpi_device->dev,
/* get saved crash message offset */
if (visorchannel_read(chipset_dev->controlvm_channel,
- offsetof(struct spar_controlvm_channel_protocol,
+ offsetof(struct visor_controlvm_channel,
saved_crash_message_offset),
&local_crash_msg_offset, sizeof(u32)) < 0) {
dev_err(&chipset_dev->acpi_device->dev,
"no valid create_bus message\n");
return;
}
- bus_create(&local_crash_bus_msg);
+ visorbus_create(&local_crash_bus_msg);
/* reuse create device message for storage device */
if (!local_crash_dev_msg.cmd.create_device.channel_addr) {
"no valid create_device message\n");
return;
}
- my_device_create(&local_crash_dev_msg);
+ visorbus_device_create(&local_crash_dev_msg);
}
void
-bus_create_response(struct visor_device *bus_info, int response)
+visorbus_create_response(struct visor_device *bus_info, int response)
{
if (response >= 0)
bus_info->state.created = 1;
}
void
-bus_destroy_response(struct visor_device *bus_info, int response)
+visorbus_destroy_response(struct visor_device *bus_info, int response)
{
controlvm_responder(CONTROLVM_BUS_DESTROY, bus_info->pending_msg_hdr,
response);
}
void
-device_create_response(struct visor_device *dev_info, int response)
+visorbus_device_create_response(struct visor_device *dev_info, int response)
{
if (response >= 0)
dev_info->state.created = 1;
}
void
-device_destroy_response(struct visor_device *dev_info, int response)
+visorbus_device_destroy_response(struct visor_device *dev_info, int response)
{
controlvm_responder(CONTROLVM_DEVICE_DESTROY, dev_info->pending_msg_hdr,
response);
}
void
-device_pause_response(struct visor_device *dev_info,
- int response)
+visorbus_device_pause_response(struct visor_device *dev_info, int response)
{
device_changestate_responder(CONTROLVM_DEVICE_CHANGESTATE,
dev_info, response,
}
void
-device_resume_response(struct visor_device *dev_info, int response)
+visorbus_device_resume_response(struct visor_device *dev_info, int response)
{
device_changestate_responder(CONTROLVM_DEVICE_CHANGESTATE,
dev_info, response,
/* create parsing context if necessary */
local_addr = (inmsg.hdr.flags.test_message == 1);
- if (channel_addr == 0)
- return -EINVAL;
-
parm_addr = channel_addr + inmsg.hdr.payload_vm_offset;
parm_bytes = inmsg.hdr.payload_bytes;
err = chipset_init(&inmsg);
break;
case CONTROLVM_BUS_CREATE:
- err = bus_create(&inmsg);
+ err = visorbus_create(&inmsg);
break;
case CONTROLVM_BUS_DESTROY:
- err = bus_destroy(&inmsg);
+ err = visorbus_destroy(&inmsg);
break;
case CONTROLVM_BUS_CONFIGURE:
- err = bus_configure(&inmsg, parser_ctx);
+ err = visorbus_configure(&inmsg, parser_ctx);
break;
case CONTROLVM_DEVICE_CREATE:
- err = my_device_create(&inmsg);
+ err = visorbus_device_create(&inmsg);
break;
case CONTROLVM_DEVICE_CHANGESTATE:
if (cmd->device_change_state.flags.phys_device) {
* save the hdr and cmd structures for later use
* when sending back the response to Command
*/
- err = my_device_changestate(&inmsg);
+ err = visorbus_device_changestate(&inmsg);
break;
}
break;
case CONTROLVM_DEVICE_DESTROY:
- err = my_device_destroy(&inmsg);
+ err = visorbus_device_destroy(&inmsg);
break;
case CONTROLVM_DEVICE_CONFIGURE:
/* no op just send a respond that we passed */
/* parahotplug_worker */
parahotplug_process_list();
+/*
+ * The controlvm messages are sent in a bulk. If we start receiving messages, we
+ * want the polling to be fast. If we do not receive any message for
+ * MIN_IDLE_SECONDS, we can slow down the polling.
+ */
schedule_out:
if (time_after(jiffies, chipset_dev->most_recent_message_jiffies +
(HZ * MIN_IDLE_SECONDS))) {
{
int err = -ENODEV;
u64 addr;
- uuid_le uuid = SPAR_CONTROLVM_CHANNEL_PROTOCOL_UUID;
+ uuid_le uuid = VISOR_CONTROLVM_CHANNEL_UUID;
struct visorchannel *controlvm_channel;
chipset_dev = kzalloc(sizeof(*chipset_dev), GFP_KERNEL);
if (err < 0)
goto error_destroy_channel;
- if (!SPAR_CONTROLVM_CHANNEL_OK_CLIENT(
+ if (!VISOR_CONTROLVM_CHANNEL_OK_CLIENT(
visorchannel_get_header(controlvm_channel)))
goto error_delete_groups;
if (boot_cpu_has(X86_FEATURE_HYPERVISOR)) {
/* check the ID */
- cpuid(UNISYS_SPAR_LEAF_ID, &eax, &ebx, &ecx, &edx);
- return (ebx == UNISYS_SPAR_ID_EBX) &&
- (ecx == UNISYS_SPAR_ID_ECX) &&
- (edx == UNISYS_SPAR_ID_EDX);
+ cpuid(UNISYS_VISOR_LEAF_ID, &eax, &ebx, &ecx, &edx);
+ return (ebx == UNISYS_VISOR_ID_EBX) &&
+ (ecx == UNISYS_VISOR_ID_ECX) &&
+ (edx == UNISYS_VISOR_ID_EDX);
} else {
return 0;
}
/* GUIDS for HBA channel type supported by this driver */
static struct visor_channeltype_descriptor visorhba_channel_types[] = {
/* Note that the only channel type we expect to be reported by the
- * bus driver is the SPAR_VHBA channel.
+ * bus driver is the VISOR_VHBA channel.
*/
- { SPAR_VHBA_CHANNEL_PROTOCOL_UUID, "sparvhba" },
+ { VISOR_VHBA_CHANNEL_UUID, "sparvhba" },
{ NULL_UUID_LE, NULL }
};
MODULE_DEVICE_TABLE(visorbus, visorhba_channel_types);
-MODULE_ALIAS("visorbus:" SPAR_VHBA_CHANNEL_PROTOCOL_UUID_STR);
+MODULE_ALIAS("visorbus:" VISOR_VHBA_CHANNEL_UUID_STR);
struct visordisk_info {
u32 valid;
seq_printf(seq, "phys_flags_addr = 0x%016llx\n",
phys_flags_addr);
seq_printf(seq, "FeatureFlags = %llu\n",
- (__le64)readq(devdata->flags_addr));
+ (u64)readq(devdata->flags_addr));
}
seq_printf(seq, "acquire_failed_cnt = %llu\n",
devdata->acquire_failed_cnt);
if (!scsihost)
return -ENODEV;
- channel_offset = offsetof(struct spar_io_channel_protocol,
- vhba.max);
+ channel_offset = offsetof(struct visor_io_channel, vhba.max);
err = visorbus_read_channel(dev, channel_offset, &max,
sizeof(struct vhba_config_max));
if (err < 0)
goto err_scsi_remove_host;
}
devdata->debugfs_info =
- debugfs_create_file("info", S_IRUSR | S_IRGRP,
+ debugfs_create_file("info", 0440,
devdata->debugfs_dir, devdata,
&info_debugfs_fops);
if (!devdata->debugfs_info) {
devdata->serverchangingstate = false;
devdata->scsihost = scsihost;
- channel_offset = offsetof(struct spar_io_channel_protocol,
+ channel_offset = offsetof(struct visor_io_channel,
channel_header.features);
err = visorbus_read_channel(dev, channel_offset, &features, 8);
if (err)
goto err_debugfs_info;
- features |= ULTRA_IO_CHANNEL_IS_POLLING;
+ features |= VISOR_CHANNEL_IS_POLLING;
err = visorbus_write_channel(dev, channel_offset, &features, 8);
if (err)
goto err_debugfs_info;
debugfs_remove_recursive(devdata->debugfs_dir);
}
-/* This is used to tell the visor bus driver which types of visor devices
+/* This is used to tell the visorbus driver which types of visor devices
* we support, and what functions to call when a visor device that we support
* is attached or removed.
*/
#include <linux/types.h>
-/* Identifies mouse and keyboard activity which is specified by the firmware to
- * the host using the cmsimpleinput protocol. @ingroup coretypes
+/* These defines identify mouse and keyboard activity which is specified by the
+ * firmware to the host using the cmsimpleinput protocol. @ingroup coretypes
*/
-enum ultra_inputaction {
- inputaction_none = 0,
- inputaction_xy_motion = 1, /* only motion; arg1=x, arg2=y */
- inputaction_mouse_button_down = 2, /* arg1: 1=left,2=center,3=right */
- inputaction_mouse_button_up = 3, /* arg1: 1=left,2=center,3=right */
- inputaction_mouse_button_click = 4, /* arg1: 1=left,2=center,3=right */
- inputaction_mouse_button_dclick = 5, /* arg1: 1=left,2=center,
- * 3=right
- */
- inputaction_wheel_rotate_away = 6, /* arg1: wheel rotation away from
- * user
- */
- inputaction_wheel_rotate_toward = 7, /* arg1: wheel rotation toward
- * user
- */
- inputaction_set_max_xy = 8, /* set screen maxXY; arg1=x, arg2=y */
- inputaction_key_down = 64, /* arg1: scancode, as follows:
+#define INPUTACTION_XY_MOTION 1 /* only motion; arg1=x, arg2=y */
+#define INPUTACTION_MOUSE_BUTTON_DOWN 2 /* arg1: 1=left,2=center,3=right */
+#define INPUTACTION_MOUSE_BUTTON_UP 3 /* arg1: 1=left,2=center,3=right */
+#define INPUTACTION_MOUSE_BUTTON_CLICK 4 /* arg1: 1=left,2=center,3=right */
+#define INPUTACTION_MOUSE_BUTTON_DCLICK 5 /* arg1: 1=left,2=center,
+ * 3=right
+ */
+#define INPUTACTION_WHEEL_ROTATE_AWAY 6 /* arg1: wheel rotation away from
+ * user
+ */
+#define INPUTACTION_WHEEL_ROTATE_TOWARD 7 /* arg1: wheel rotation toward
+ * user
+ */
+#define INPUTACTION_KEY_DOWN 64 /* arg1: scancode, as follows:
* If arg1 <= 0xff, it's a 1-byte
* scancode and arg1 is that scancode.
* If arg1 > 0xff, it's a 2-byte
* high 8 bits. E.g., the right ALT key
* would appear as x'38e0'.
*/
- inputaction_key_up = 65, /* arg1: scancode (in same format as
+#define INPUTACTION_KEY_UP 65 /* arg1: scancode (in same format as
* inputaction_keyDown)
*/
- inputaction_set_locking_key_state = 66,
+#define INPUTACTION_SET_LOCKING_KEY_STATE 66
/* arg1: scancode (in same format
* as inputaction_keyDown);
* MUST refer to one of the
* in the LOCKED position
* (e.g., light is ON)
*/
- inputaction_key_down_up = 67, /* arg1: scancode (in same format
+#define INPUTACTION_KEY_DOWN_UP 67 /* arg1: scancode (in same format
* as inputaction_keyDown)
*/
- inputaction_last
-};
-struct ultra_inputactivity {
+struct visor_inputactivity {
u16 action;
u16 arg1;
u16 arg2;
u16 arg3;
} __packed;
-struct ultra_inputreport {
+struct visor_inputreport {
u64 seq_no;
- struct ultra_inputactivity activity;
+ struct visor_inputactivity activity;
} __packed;
#endif
#include "ultrainputreport.h"
/* Keyboard channel {c73416d0-b0b8-44af-b304-9d2ae99f1b3d} */
-#define SPAR_KEYBOARD_CHANNEL_PROTOCOL_UUID \
+#define VISOR_KEYBOARD_CHANNEL_UUID \
UUID_LE(0xc73416d0, 0xb0b8, 0x44af, \
0xb3, 0x4, 0x9d, 0x2a, 0xe9, 0x9f, 0x1b, 0x3d)
-#define SPAR_KEYBOARD_CHANNEL_PROTOCOL_UUID_STR "c73416d0-b0b8-44af-b304-9d2ae99f1b3d"
+#define VISOR_KEYBOARD_CHANNEL_UUID_STR "c73416d0-b0b8-44af-b304-9d2ae99f1b3d"
/* Mouse channel {addf07d4-94a9-46e2-81c3-61abcdbdbd87} */
-#define SPAR_MOUSE_CHANNEL_PROTOCOL_UUID \
+#define VISOR_MOUSE_CHANNEL_UUID \
UUID_LE(0xaddf07d4, 0x94a9, 0x46e2, \
0x81, 0xc3, 0x61, 0xab, 0xcd, 0xbd, 0xbd, 0x87)
-#define SPAR_MOUSE_CHANNEL_PROTOCOL_UUID_STR \
- "addf07d4-94a9-46e2-81c3-61abcdbdbd87"
+#define VISOR_MOUSE_CHANNEL_UUID_STR "addf07d4-94a9-46e2-81c3-61abcdbdbd87"
#define PIXELS_ACROSS_DEFAULT 800
#define PIXELS_DOWN_DEFAULT 600
unsigned char keycode_table[0];
};
-static const uuid_le spar_keyboard_channel_protocol_uuid =
- SPAR_KEYBOARD_CHANNEL_PROTOCOL_UUID;
-static const uuid_le spar_mouse_channel_protocol_uuid =
- SPAR_MOUSE_CHANNEL_PROTOCOL_UUID;
+static const uuid_le visor_keyboard_channel_uuid = VISOR_KEYBOARD_CHANNEL_UUID;
+static const uuid_le visor_mouse_channel_uuid = VISOR_MOUSE_CHANNEL_UUID;
/*
* Borrowed from drivers/input/keyboard/atakbd.c
enum visorinput_device_type devtype;
guid = visorchannel_get_uuid(dev->visorchannel);
- if (uuid_le_cmp(guid, spar_mouse_channel_protocol_uuid) == 0)
+ if (uuid_le_cmp(guid, visor_mouse_channel_uuid) == 0)
devtype = visorinput_mouse;
- else if (uuid_le_cmp(guid, spar_keyboard_channel_protocol_uuid) == 0)
+ else if (uuid_le_cmp(guid, visor_keyboard_channel_uuid) == 0)
devtype = visorinput_keyboard;
else
return -ENODEV;
static void
visorinput_channel_interrupt(struct visor_device *dev)
{
- struct ultra_inputreport r;
+ struct visor_inputreport r;
int scancode, keycode;
struct input_dev *visorinput_dev;
int xmotion, ymotion, button;
scancode = r.activity.arg1;
keycode = scancode_to_keycode(scancode);
switch (r.activity.action) {
- case inputaction_key_down:
+ case INPUTACTION_KEY_DOWN:
input_report_key(visorinput_dev, keycode, 1);
input_sync(visorinput_dev);
break;
- case inputaction_key_up:
+ case INPUTACTION_KEY_UP:
input_report_key(visorinput_dev, keycode, 0);
input_sync(visorinput_dev);
break;
- case inputaction_key_down_up:
+ case INPUTACTION_KEY_DOWN_UP:
input_report_key(visorinput_dev, keycode, 1);
input_sync(visorinput_dev);
input_report_key(visorinput_dev, keycode, 0);
input_sync(visorinput_dev);
break;
- case inputaction_set_locking_key_state:
+ case INPUTACTION_SET_LOCKING_KEY_STATE:
handle_locking_key(visorinput_dev, keycode,
r.activity.arg2);
break;
- case inputaction_xy_motion:
+ case INPUTACTION_XY_MOTION:
xmotion = r.activity.arg1;
ymotion = r.activity.arg2;
input_report_abs(visorinput_dev, ABS_X, xmotion);
input_report_abs(visorinput_dev, ABS_Y, ymotion);
input_sync(visorinput_dev);
break;
- case inputaction_mouse_button_down:
+ case INPUTACTION_MOUSE_BUTTON_DOWN:
button = calc_button(r.activity.arg1);
if (button < 0)
break;
input_report_key(visorinput_dev, button, 1);
input_sync(visorinput_dev);
break;
- case inputaction_mouse_button_up:
+ case INPUTACTION_MOUSE_BUTTON_UP:
button = calc_button(r.activity.arg1);
if (button < 0)
break;
input_report_key(visorinput_dev, button, 0);
input_sync(visorinput_dev);
break;
- case inputaction_mouse_button_click:
+ case INPUTACTION_MOUSE_BUTTON_CLICK:
button = calc_button(r.activity.arg1);
if (button < 0)
break;
input_report_key(visorinput_dev, button, 0);
input_sync(visorinput_dev);
break;
- case inputaction_mouse_button_dclick:
+ case INPUTACTION_MOUSE_BUTTON_DCLICK:
button = calc_button(r.activity.arg1);
if (button < 0)
break;
input_sync(visorinput_dev);
}
break;
- case inputaction_wheel_rotate_away:
+ case INPUTACTION_WHEEL_ROTATE_AWAY:
input_report_rel(visorinput_dev, REL_WHEEL, 1);
input_sync(visorinput_dev);
break;
- case inputaction_wheel_rotate_toward:
+ case INPUTACTION_WHEEL_ROTATE_TOWARD:
input_report_rel(visorinput_dev, REL_WHEEL, -1);
input_sync(visorinput_dev);
break;
/* GUIDS for all channel types supported by this driver. */
static struct visor_channeltype_descriptor visorinput_channel_types[] = {
- { SPAR_KEYBOARD_CHANNEL_PROTOCOL_UUID, "keyboard"},
- { SPAR_MOUSE_CHANNEL_PROTOCOL_UUID, "mouse"},
+ { VISOR_KEYBOARD_CHANNEL_UUID, "keyboard"},
+ { VISOR_MOUSE_CHANNEL_UUID, "mouse"},
{ NULL_UUID_LE, NULL }
};
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("s-Par human input driver for virtual keyboard/mouse");
-MODULE_ALIAS("visorbus:" SPAR_MOUSE_CHANNEL_PROTOCOL_UUID_STR);
-MODULE_ALIAS("visorbus:" SPAR_KEYBOARD_CHANNEL_PROTOCOL_UUID_STR);
+MODULE_ALIAS("visorbus:" VISOR_MOUSE_CHANNEL_UUID_STR);
+MODULE_ALIAS("visorbus:" VISOR_KEYBOARD_CHANNEL_UUID_STR);
/* GUIDS for director channel type supported by this driver. */
static struct visor_channeltype_descriptor visornic_channel_types[] = {
/* Note that the only channel type we expect to be reported by the
- * bus driver is the SPAR_VNIC channel.
+ * bus driver is the VISOR_VNIC channel.
*/
- { SPAR_VNIC_CHANNEL_PROTOCOL_UUID, "ultravnic" },
+ { VISOR_VNIC_CHANNEL_UUID, "ultravnic" },
{ NULL_UUID_LE, NULL }
};
MODULE_DEVICE_TABLE(visorbus, visornic_channel_types);
* must be added to scripts/mode/file2alias.c, etc., to get this working
* properly.
*/
-MODULE_ALIAS("visorbus:" SPAR_VNIC_CHANNEL_PROTOCOL_UUID_STR);
+MODULE_ALIAS("visorbus:" VISOR_VNIC_CHANNEL_UUID_STR);
struct chanstat {
unsigned long got_rcv;
/* Get MAC address from channel and read it into the device. */
netdev->addr_len = ETH_ALEN;
- channel_offset = offsetof(struct spar_io_channel_protocol,
- vnic.macaddr);
+ channel_offset = offsetof(struct visor_io_channel, vnic.macaddr);
err = visorbus_read_channel(dev, channel_offset, netdev->dev_addr,
ETH_ALEN);
if (err < 0) {
atomic_set(&devdata->usage, 1);
/* Setup rcv bufs */
- channel_offset = offsetof(struct spar_io_channel_protocol,
- vnic.num_rcv_bufs);
+ channel_offset = offsetof(struct visor_io_channel, vnic.num_rcv_bufs);
err = visorbus_read_channel(dev, channel_offset,
&devdata->num_rcv_bufs, 4);
if (err) {
devdata->server_change_state = false;
/*set the default mtu */
- channel_offset = offsetof(struct spar_io_channel_protocol,
- vnic.mtu);
+ channel_offset = offsetof(struct visor_io_channel, vnic.mtu);
err = visorbus_read_channel(dev, channel_offset, &netdev->mtu, 4);
if (err) {
dev_err(&dev->device,
*/
mod_timer(&devdata->irq_poll_timer, msecs_to_jiffies(2));
- channel_offset = offsetof(struct spar_io_channel_protocol,
+ channel_offset = offsetof(struct visor_io_channel,
channel_header.features);
err = visorbus_read_channel(dev, channel_offset, &features, 8);
if (err) {
goto cleanup_napi_add;
}
- features |= ULTRA_IO_CHANNEL_IS_POLLING;
- features |= ULTRA_IO_DRIVER_SUPPORTS_ENHANCED_RCVBUF_CHECKING;
+ features |= VISOR_CHANNEL_IS_POLLING;
+ features |= VISOR_DRIVER_ENHANCED_RCVBUF_CHECKING;
err = visorbus_write_channel(dev, channel_offset, &features, 8);
if (err) {
dev_err(&dev->device,
return 0;
}
-/* This is used to tell the visor bus driver which types of visor devices
+/* This is used to tell the visorbus driver which types of visor devices
* we support, and what functions to call when a visor device that we support
* is attached or removed.
*/
MAX_FRAGMENTS;
g_fragments_base = (char *)slot_mem + slot_mem_size;
- slot_mem_size += frag_mem_size;
g_free_fragments = g_fragments_base;
for (i = 0; i < (MAX_FRAGMENTS - 1); i++) {
}
static void
-cleaup_pagelistinfo(struct vchiq_pagelist_info *pagelistinfo)
+cleanup_pagelistinfo(struct vchiq_pagelist_info *pagelistinfo)
{
if (pagelistinfo->scatterlist_mapped) {
dma_unmap_sg(g_dev, pagelistinfo->scatterlist,
PAGE_SIZE));
size_t bytes = PAGE_SIZE - off;
+ if (!pg) {
+ cleanup_pagelistinfo(pagelistinfo);
+ return NULL;
+ }
+
if (bytes > length)
bytes = length;
pages[actual_pages] = pg;
} else {
down_read(&task->mm->mmap_sem);
actual_pages = get_user_pages(
- (unsigned long)buf & ~(PAGE_SIZE - 1),
+ (unsigned long)buf & PAGE_MASK,
num_pages,
(type == PAGELIST_READ) ? FOLL_WRITE : 0,
pages,
actual_pages--;
put_page(pages[actual_pages]);
}
- cleaup_pagelistinfo(pagelistinfo);
+ cleanup_pagelistinfo(pagelistinfo);
return NULL;
}
/* release user pages */
pagelistinfo->dma_dir);
if (dma_buffers == 0) {
- cleaup_pagelistinfo(pagelistinfo);
+ cleanup_pagelistinfo(pagelistinfo);
return NULL;
}
char *fragments;
if (down_interruptible(&g_free_fragments_sema) != 0) {
- cleaup_pagelistinfo(pagelistinfo);
+ cleanup_pagelistinfo(pagelistinfo);
return NULL;
}
free_pagelist(struct vchiq_pagelist_info *pagelistinfo,
int actual)
{
- unsigned int i;
PAGELIST_T *pagelist = pagelistinfo->pagelist;
struct page **pages = pagelistinfo->pages;
unsigned int num_pages = pagelistinfo->num_pages;
/* Need to mark all the pages dirty. */
if (pagelist->type != PAGELIST_WRITE &&
pagelistinfo->pages_need_release) {
+ unsigned int i;
+
for (i = 0; i < num_pages; i++)
set_page_dirty(pages[i]);
}
- cleaup_pagelistinfo(pagelistinfo);
+ cleanup_pagelistinfo(pagelistinfo);
}
struct page **pages;
u8 *kmapped_virt_ptr;
- /* Align virtAddr and endVirtAddr to 16 byte boundaries. */
+ /* Align virt_addr and end_virt_addr to 16 byte boundaries. */
virt_addr = (void *)((unsigned long)virt_addr & ~0x0fuL);
end_virt_addr = (void *)(((unsigned long)end_virt_addr + 15uL) &
if (only_nonzero && !service_ptr->service_use_count)
continue;
- if (service_ptr->srvstate != VCHIQ_SRVSTATE_FREE) {
- service_data[j].fourcc = service_ptr->base.fourcc;
- service_data[j].clientid = service_ptr->client_id;
- service_data[j++].use_count = service_ptr->
- service_use_count;
- }
+ if (service_ptr->srvstate == VCHIQ_SRVSTATE_FREE)
+ continue;
+
+ service_data[j].fourcc = service_ptr->base.fourcc;
+ service_data[j].clientid = service_ptr->client_id;
+ service_data[j++].use_count = service_ptr->service_use_count;
}
read_unlock_bh(&arm_state->susp_res_lock);
service = handle_to_service(handle);
if (service && (service->srvstate != VCHIQ_SRVSTATE_FREE) &&
(service->handle == handle)) {
- BUG_ON(service->ref_count == 0);
+ WARN_ON(service->ref_count == 0);
service->ref_count++;
} else
service = NULL;
spin_lock(&service_spinlock);
service = state->services[localport];
if (service && (service->srvstate != VCHIQ_SRVSTATE_FREE)) {
- BUG_ON(service->ref_count == 0);
+ WARN_ON(service->ref_count == 0);
service->ref_count++;
} else
service = NULL;
if (service && (service->srvstate != VCHIQ_SRVSTATE_FREE) &&
(service->handle == handle) &&
(service->instance == instance)) {
- BUG_ON(service->ref_count == 0);
+ WARN_ON(service->ref_count == 0);
service->ref_count++;
} else
service = NULL;
(service->srvstate == VCHIQ_SRVSTATE_CLOSED)) &&
(service->handle == handle) &&
(service->instance == instance)) {
- BUG_ON(service->ref_count == 0);
+ WARN_ON(service->ref_count == 0);
service->ref_count++;
} else
service = NULL;
if (srv && (srv->srvstate != VCHIQ_SRVSTATE_FREE) &&
(srv->instance == instance)) {
service = srv;
- BUG_ON(service->ref_count == 0);
+ WARN_ON(service->ref_count == 0);
service->ref_count++;
break;
}
lock_service(VCHIQ_SERVICE_T *service)
{
spin_lock(&service_spinlock);
- BUG_ON(!service || (service->ref_count == 0));
- if (service)
+ WARN_ON(!service);
+ if (service) {
+ WARN_ON(service->ref_count == 0);
service->ref_count++;
+ }
spin_unlock(&service_spinlock);
}
unlock_service(VCHIQ_SERVICE_T *service)
{
spin_lock(&service_spinlock);
- BUG_ON(!service || (service->ref_count == 0));
- if (service && service->ref_count) {
- service->ref_count--;
- if (!service->ref_count) {
- VCHIQ_STATE_T *state = service->state;
-
- BUG_ON(service->srvstate != VCHIQ_SRVSTATE_FREE);
- state->services[service->localport] = NULL;
- } else
- service = NULL;
+ if (!service) {
+ WARN(1, "%s: service is NULL\n", __func__);
+ goto unlock;
+ }
+ if (!service->ref_count) {
+ WARN(1, "%s: ref_count is zero\n", __func__);
+ goto unlock;
}
+ service->ref_count--;
+ if (!service->ref_count) {
+ VCHIQ_STATE_T *state = service->state;
+
+ WARN_ON(service->srvstate != VCHIQ_SRVSTATE_FREE);
+ state->services[service->localport] = NULL;
+ } else {
+ service = NULL;
+ }
+unlock:
spin_unlock(&service_spinlock);
if (service && service->userdata_term)
return NULL; /* No space available */
}
- BUG_ON(tx_pos ==
- (state->slot_queue_available * VCHIQ_SLOT_SIZE));
+ if (tx_pos == (state->slot_queue_available * VCHIQ_SLOT_SIZE)) {
+ pr_warn("%s: invalid tx_pos: %d\n", __func__, tx_pos);
+ return NULL;
+ }
slot_index = local->slot_queue[
SLOT_QUEUE_INDEX_FROM_POS(tx_pos) &
if (type == VCHIQ_MSG_DATA) {
int tx_end_index;
- BUG_ON(!service);
- BUG_ON((flags & (QMFLAGS_NO_MUTEX_LOCK |
- QMFLAGS_NO_MUTEX_UNLOCK)) != 0);
+ if (!service) {
+ WARN(1, "%s: service is NULL\n", __func__);
+ mutex_unlock(&state->slot_mutex);
+ return VCHIQ_ERROR;
+ }
+
+ WARN_ON((flags & (QMFLAGS_NO_MUTEX_LOCK |
+ QMFLAGS_NO_MUTEX_UNLOCK)) != 0);
if (service->closing) {
/* The service has been closed */
header, size, VCHIQ_MSG_SRCPORT(msgid),
VCHIQ_MSG_DSTPORT(msgid));
- BUG_ON(!service);
- BUG_ON((flags & (QMFLAGS_NO_MUTEX_LOCK |
- QMFLAGS_NO_MUTEX_UNLOCK)) != 0);
+ WARN_ON((flags & (QMFLAGS_NO_MUTEX_LOCK |
+ QMFLAGS_NO_MUTEX_UNLOCK)) != 0);
callback_result =
copy_message_data(copy_callback, context,
{
VCHIQ_STATE_T *state = service->state;
int resolved = 0;
- int rc;
while ((queue->process != queue->local_insert) &&
(queue->process != queue->remote_insert)) {
WARN_ON(!((int)(queue->local_insert - queue->process) > 0));
WARN_ON(!((int)(queue->remote_insert - queue->process) > 0));
- rc = mutex_lock_killable(&state->bulk_transfer_mutex);
- if (rc != 0)
+ if (mutex_lock_killable(&state->bulk_transfer_mutex))
break;
vchiq_transfer_bulk(bulk);
mutex_unlock(&service->bulk_mutex);
break;
}
-
- BUG_ON(queue->process == queue->local_insert);
- BUG_ON(queue->process != queue->remote_insert);
+ if (queue->process != queue->remote_insert) {
+ pr_err("%s: p %x != ri %x\n",
+ __func__,
+ queue->process,
+ queue->remote_insert);
+ mutex_unlock(&service->bulk_mutex);
+ goto bail_not_ready;
+ }
bulk = &queue->bulks[
BULK_INDEX(queue->remote_insert)];
vchiq_log_error(vchiq_core_log_level,
"Failed to send RESUME "
"message");
- BUG();
}
break;
VCHIQ_SHARED_STATE_T *remote;
VCHIQ_STATUS_T status;
char threadname[16];
- static int id;
int i;
vchiq_log_warning(vchiq_core_log_level,
"%s: slot_zero = %pK, is_master = %d",
__func__, slot_zero, is_master);
+ if (vchiq_states[0]) {
+ pr_err("%s: VCHIQ state already initialized\n", __func__);
+ return VCHIQ_ERROR;
+ }
+
/* Check the input configuration */
if (slot_zero->magic != VCHIQ_MAGIC) {
memset(state, 0, sizeof(VCHIQ_STATE_T));
- state->id = id++;
state->is_master = is_master;
/*
set_user_nice(state->sync_thread, -20);
wake_up_process(state->sync_thread);
- BUG_ON(state->id >= VCHIQ_MAX_STATES);
- vchiq_states[state->id] = state;
+ vchiq_states[0] = state;
/* Indicate readiness to the other side */
local->initialised = 1;
if (current == service->state->slot_handler_thread) {
status = vchiq_close_service_internal(service,
0/*!close_recvd*/);
- BUG_ON(status == VCHIQ_RETRY);
+ WARN_ON(status == VCHIQ_RETRY);
} else {
/* Mark the service for termination by the slot handler */
request_poll(service->state, service, VCHIQ_POLL_TERMINATE);
status = vchiq_close_service_internal(service,
0/*!close_recvd*/);
- BUG_ON(status == VCHIQ_RETRY);
+ WARN_ON(status == VCHIQ_RETRY);
} else {
/* Mark the service for removal by the slot handler */
request_poll(service->state, service, VCHIQ_POLL_REMOVE);
2, 2, 2, 2, 2, 2, 2, 2,
3, 3, 3, 3, 3, 3, 3, 3 };
-#define PIO2_CHANNEL0_BIT (1 << 0)
-#define PIO2_CHANNEL1_BIT (1 << 1)
-#define PIO2_CHANNEL2_BIT (1 << 2)
-#define PIO2_CHANNEL3_BIT (1 << 3)
-#define PIO2_CHANNEL4_BIT (1 << 4)
-#define PIO2_CHANNEL5_BIT (1 << 5)
-#define PIO2_CHANNEL6_BIT (1 << 6)
-#define PIO2_CHANNEL7_BIT (1 << 7)
-#define PIO2_CHANNEL8_BIT (1 << 0)
-#define PIO2_CHANNEL9_BIT (1 << 1)
-#define PIO2_CHANNEL10_BIT (1 << 2)
-#define PIO2_CHANNEL11_BIT (1 << 3)
-#define PIO2_CHANNEL12_BIT (1 << 4)
-#define PIO2_CHANNEL13_BIT (1 << 5)
-#define PIO2_CHANNEL14_BIT (1 << 6)
-#define PIO2_CHANNEL15_BIT (1 << 7)
-#define PIO2_CHANNEL16_BIT (1 << 0)
-#define PIO2_CHANNEL17_BIT (1 << 1)
-#define PIO2_CHANNEL18_BIT (1 << 2)
-#define PIO2_CHANNEL19_BIT (1 << 3)
-#define PIO2_CHANNEL20_BIT (1 << 4)
-#define PIO2_CHANNEL21_BIT (1 << 5)
-#define PIO2_CHANNEL22_BIT (1 << 6)
-#define PIO2_CHANNEL23_BIT (1 << 7)
-#define PIO2_CHANNEL24_BIT (1 << 0)
-#define PIO2_CHANNEL25_BIT (1 << 1)
-#define PIO2_CHANNEL26_BIT (1 << 2)
-#define PIO2_CHANNEL27_BIT (1 << 3)
-#define PIO2_CHANNEL28_BIT (1 << 4)
-#define PIO2_CHANNEL29_BIT (1 << 5)
-#define PIO2_CHANNEL30_BIT (1 << 6)
-#define PIO2_CHANNEL31_BIT (1 << 7)
+#define PIO2_CHANNEL0_BIT BIT(0)
+#define PIO2_CHANNEL1_BIT BIT(1)
+#define PIO2_CHANNEL2_BIT BIT(2)
+#define PIO2_CHANNEL3_BIT BIT(3)
+#define PIO2_CHANNEL4_BIT BIT(4)
+#define PIO2_CHANNEL5_BIT BIT(5)
+#define PIO2_CHANNEL6_BIT BIT(6)
+#define PIO2_CHANNEL7_BIT BIT(7)
+#define PIO2_CHANNEL8_BIT BIT(0)
+#define PIO2_CHANNEL9_BIT BIT(1)
+#define PIO2_CHANNEL10_BIT BIT(2)
+#define PIO2_CHANNEL11_BIT BIT(3)
+#define PIO2_CHANNEL12_BIT BIT(4)
+#define PIO2_CHANNEL13_BIT BIT(5)
+#define PIO2_CHANNEL14_BIT BIT(6)
+#define PIO2_CHANNEL15_BIT BIT(7)
+#define PIO2_CHANNEL16_BIT BIT(0)
+#define PIO2_CHANNEL17_BIT BIT(1)
+#define PIO2_CHANNEL18_BIT BIT(2)
+#define PIO2_CHANNEL19_BIT BIT(3)
+#define PIO2_CHANNEL20_BIT BIT(4)
+#define PIO2_CHANNEL21_BIT BIT(5)
+#define PIO2_CHANNEL22_BIT BIT(6)
+#define PIO2_CHANNEL23_BIT BIT(7)
+#define PIO2_CHANNEL24_BIT BIT(0)
+#define PIO2_CHANNEL25_BIT BIT(1)
+#define PIO2_CHANNEL26_BIT BIT(2)
+#define PIO2_CHANNEL27_BIT BIT(3)
+#define PIO2_CHANNEL28_BIT BIT(4)
+#define PIO2_CHANNEL29_BIT BIT(5)
+#define PIO2_CHANNEL30_BIT BIT(6)
+#define PIO2_CHANNEL31_BIT BIT(7)
static const int PIO2_CHANNEL_BIT[32] = { PIO2_CHANNEL0_BIT, PIO2_CHANNEL1_BIT,
PIO2_CHANNEL2_BIT, PIO2_CHANNEL3_BIT,
};
/* PIO2_REGS_INT_STAT_CNTR (0xc) */
-#define PIO2_COUNTER0 (1 << 0)
-#define PIO2_COUNTER1 (1 << 1)
-#define PIO2_COUNTER2 (1 << 2)
-#define PIO2_COUNTER3 (1 << 3)
-#define PIO2_COUNTER4 (1 << 4)
-#define PIO2_COUNTER5 (1 << 5)
+#define PIO2_COUNTER0 BIT(0)
+#define PIO2_COUNTER1 BIT(1)
+#define PIO2_COUNTER2 BIT(2)
+#define PIO2_COUNTER3 BIT(3)
+#define PIO2_COUNTER4 BIT(4)
+#define PIO2_COUNTER5 BIT(5)
static const int PIO2_COUNTER[6] = { PIO2_COUNTER0, PIO2_COUNTER1,
PIO2_COUNTER2, PIO2_COUNTER3,
/* PIO2_REGS_CTRL (0x18) */
#define PIO2_VME_INT_MASK 0x7
-#define PIO2_LED (1 << 6)
-#define PIO2_LOOP (1 << 7)
+#define PIO2_LED BIT(6)
+#define PIO2_LOOP BIT(7)
/* PIO2_REGS_VME_VECTOR (0x19) */
#define PIO2_VME_VECTOR_SPUR 0x0
{
unsigned short wRxBcnTSFOffst;
- wRxBcnTSFOffst = cwRXBCNTSFOff[byRxRate%MAX_RATE];
+ wRxBcnTSFOffst = cwRXBCNTSFOff[byRxRate % MAX_RATE];
qwTSF2 += (u64)wRxBcnTSFOffst;
struct vnt_private;
-void CARDvSetRSPINF(struct vnt_private *, u8);
-void CARDvUpdateBasicTopRate(struct vnt_private *);
-bool CARDbIsOFDMinBasicRate(struct vnt_private *);
-void CARDvSetLoopbackMode(struct vnt_private *, unsigned short wLoopbackMode);
-bool CARDbSoftwareReset(struct vnt_private *);
-void CARDvSetFirstNextTBTT(struct vnt_private *,
+void CARDvSetRSPINF(struct vnt_private *priv, u8 bb_type);
+void CARDvUpdateBasicTopRate(struct vnt_private *priv);
+bool CARDbIsOFDMinBasicRate(struct vnt_private *priv);
+void CARDvSetLoopbackMode(struct vnt_private *priv, unsigned short wLoopbackMode);
+bool CARDbSoftwareReset(struct vnt_private *priv);
+void CARDvSetFirstNextTBTT(struct vnt_private *priv,
unsigned short wBeaconInterval);
-void CARDvUpdateNextTBTT(struct vnt_private *, u64 qwTSF,
+void CARDvUpdateNextTBTT(struct vnt_private *priv, u64 qwTSF,
unsigned short wBeaconInterval);
-bool CARDbGetCurrentTSF(struct vnt_private *, u64 *pqwCurrTSF);
+bool CARDbGetCurrentTSF(struct vnt_private *priv, u64 *pqwCurrTSF);
u64 CARDqGetNextTBTT(u64 qwTSF, unsigned short wBeaconInterval);
u64 CARDqGetTSFOffset(unsigned char byRxRate, u64 qwTSF1, u64 qwTSF2);
-unsigned char CARDbyGetPktType(struct vnt_private *);
-void CARDvSafeResetTx(struct vnt_private *);
-void CARDvSafeResetRx(struct vnt_private *);
-bool CARDbRadioPowerOff(struct vnt_private *);
-bool CARDbRadioPowerOn(struct vnt_private *);
-bool CARDbSetPhyParameter(struct vnt_private *, u8);
-bool CARDbUpdateTSF(struct vnt_private *, unsigned char byRxRate,
+unsigned char CARDbyGetPktType(struct vnt_private *priv);
+void CARDvSafeResetTx(struct vnt_private *priv);
+void CARDvSafeResetRx(struct vnt_private *priv);
+bool CARDbRadioPowerOff(struct vnt_private *priv);
+bool CARDbRadioPowerOn(struct vnt_private *priv);
+bool CARDbSetPhyParameter(struct vnt_private *priv, u8 bb_type);
+bool CARDbUpdateTSF(struct vnt_private *priv, unsigned char byRxRate,
u64 qwBSSTimestamp);
-bool CARDbSetBeaconPeriod(struct vnt_private *, unsigned short wBeaconInterval);
+bool CARDbSetBeaconPeriod(struct vnt_private *priv, unsigned short wBeaconInterval);
#endif /* __CARD_H__ */
#include "card.h"
-void vnt_init_bands(struct vnt_private *);
+void vnt_init_bands(struct vnt_private *priv);
-bool set_channel(struct vnt_private *, struct ieee80211_channel *);
+bool set_channel(struct vnt_private *priv, struct ieee80211_channel *ch);
#endif /* _CHANNEL_H_ */
{
struct vnt_private *priv = pci_get_drvdata(pcid);
- if (priv == NULL)
+ if (!priv)
return;
device_free_info(priv);
}
priv->opts.tx_descs[0] * sizeof(struct vnt_tx_desc) +
priv->opts.tx_descs[1] * sizeof(struct vnt_tx_desc),
&priv->pool_dma, GFP_ATOMIC);
- if (vir_pool == NULL) {
+ if (!vir_pool) {
dev_err(&priv->pcid->dev, "allocate desc dma memory failed\n");
return false;
}
}
/* TODO: adhoc PS mode */
-
}
if (isr & ISR_BNTX) {
}
static void vnt_bss_info_changed(struct ieee80211_hw *hw,
- struct ieee80211_vif *vif, struct ieee80211_bss_conf *conf,
- u32 changed)
+ struct ieee80211_vif *vif,
+ struct ieee80211_bss_conf *conf, u32 changed)
{
struct vnt_private *priv = hw->priv;
}
static u64 vnt_prepare_multicast(struct ieee80211_hw *hw,
- struct netdev_hw_addr_list *mc_list)
+ struct netdev_hw_addr_list *mc_list)
{
struct vnt_private *priv = hw->priv;
struct netdev_hw_addr *ha;
}
static void vnt_configure(struct ieee80211_hw *hw,
- unsigned int changed_flags, unsigned int *total_flags, u64 multicast)
+ unsigned int changed_flags,
+ unsigned int *total_flags, u64 multicast)
{
struct vnt_private *priv = hw->priv;
u8 rx_mode = 0;
}
static int vnt_set_key(struct ieee80211_hw *hw, enum set_key_cmd cmd,
- struct ieee80211_vif *vif, struct ieee80211_sta *sta,
- struct ieee80211_key_conf *key)
+ struct ieee80211_vif *vif, struct ieee80211_sta *sta,
+ struct ieee80211_key_conf *key)
{
struct vnt_private *priv = hw->priv;
static int vt6655_resume(struct pci_dev *pcid)
{
-
pci_set_power_state(pcid, PCI_D0);
pci_enable_wake(pcid, PCI_D0, 0);
pci_restore_state(pcid);
#include "mac.h"
static int vnt_set_keymode(struct ieee80211_hw *hw, u8 *mac_addr,
- struct ieee80211_key_conf *key, u32 key_type, u32 mode,
- bool onfly_latch)
+ struct ieee80211_key_conf *key, u32 key_type,
+ u32 mode, bool onfly_latch)
{
struct vnt_private *priv = hw->priv;
u8 broadcast[6] = {0xff, 0xff, 0xff, 0xff, 0xff, 0xff};
#define MACvSetRFLE_LatchBase(iobase) \
MACvWordRegBitsOn(iobase, MAC_REG_SOFTPWRCTL, SOFTPWRCTL_RFLEOPT)
-bool MACbIsRegBitsOn(struct vnt_private *, unsigned char byRegOfs,
+bool MACbIsRegBitsOn(struct vnt_private *priv, unsigned char byRegOfs,
unsigned char byTestBits);
-bool MACbIsRegBitsOff(struct vnt_private *, unsigned char byRegOfs,
+bool MACbIsRegBitsOff(struct vnt_private *priv, unsigned char byRegOfs,
unsigned char byTestBits);
-bool MACbIsIntDisable(struct vnt_private *);
+bool MACbIsIntDisable(struct vnt_private *priv);
-void MACvSetShortRetryLimit(struct vnt_private *, unsigned char byRetryLimit);
+void MACvSetShortRetryLimit(struct vnt_private *priv, unsigned char byRetryLimit);
-void MACvSetLongRetryLimit(struct vnt_private *, unsigned char byRetryLimit);
-void MACvGetLongRetryLimit(struct vnt_private *,
+void MACvSetLongRetryLimit(struct vnt_private *priv, unsigned char byRetryLimit);
+void MACvGetLongRetryLimit(struct vnt_private *priv,
unsigned char *pbyRetryLimit);
-void MACvSetLoopbackMode(struct vnt_private *, unsigned char byLoopbackMode);
+void MACvSetLoopbackMode(struct vnt_private *priv, unsigned char byLoopbackMode);
-void MACvSaveContext(struct vnt_private *, unsigned char *pbyCxtBuf);
-void MACvRestoreContext(struct vnt_private *, unsigned char *pbyCxtBuf);
+void MACvSaveContext(struct vnt_private *priv, unsigned char *pbyCxtBuf);
+void MACvRestoreContext(struct vnt_private *priv, unsigned char *pbyCxtBuf);
-bool MACbSoftwareReset(struct vnt_private *);
-bool MACbSafeSoftwareReset(struct vnt_private *);
-bool MACbSafeRxOff(struct vnt_private *);
-bool MACbSafeTxOff(struct vnt_private *);
-bool MACbSafeStop(struct vnt_private *);
-bool MACbShutdown(struct vnt_private *);
-void MACvInitialize(struct vnt_private *);
-void MACvSetCurrRx0DescAddr(struct vnt_private *,
+bool MACbSoftwareReset(struct vnt_private *priv);
+bool MACbSafeSoftwareReset(struct vnt_private *priv);
+bool MACbSafeRxOff(struct vnt_private *priv);
+bool MACbSafeTxOff(struct vnt_private *priv);
+bool MACbSafeStop(struct vnt_private *priv);
+bool MACbShutdown(struct vnt_private *priv);
+void MACvInitialize(struct vnt_private *priv);
+void MACvSetCurrRx0DescAddr(struct vnt_private *priv,
u32 curr_desc_addr);
-void MACvSetCurrRx1DescAddr(struct vnt_private *,
+void MACvSetCurrRx1DescAddr(struct vnt_private *priv,
u32 curr_desc_addr);
-void MACvSetCurrTXDescAddr(int iTxType, struct vnt_private *,
+void MACvSetCurrTXDescAddr(int iTxType, struct vnt_private *priv,
u32 curr_desc_addr);
-void MACvSetCurrTx0DescAddrEx(struct vnt_private *,
+void MACvSetCurrTx0DescAddrEx(struct vnt_private *priv,
u32 curr_desc_addr);
-void MACvSetCurrAC0DescAddrEx(struct vnt_private *,
+void MACvSetCurrAC0DescAddrEx(struct vnt_private *priv,
u32 curr_desc_addr);
-void MACvSetCurrSyncDescAddrEx(struct vnt_private *,
+void MACvSetCurrSyncDescAddrEx(struct vnt_private *priv,
u32 curr_desc_addr);
-void MACvSetCurrATIMDescAddrEx(struct vnt_private *,
+void MACvSetCurrATIMDescAddrEx(struct vnt_private *priv,
u32 curr_desc_addr);
-void MACvTimer0MicroSDelay(struct vnt_private *, unsigned int uDelay);
-void MACvOneShotTimer1MicroSec(struct vnt_private *, unsigned int uDelayTime);
+void MACvTimer0MicroSDelay(struct vnt_private *priv, unsigned int uDelay);
+void MACvOneShotTimer1MicroSec(struct vnt_private *priv, unsigned int uDelayTime);
-void MACvSetMISCFifo(struct vnt_private *, unsigned short wOffset,
+void MACvSetMISCFifo(struct vnt_private *priv, unsigned short wOffset,
u32 dwData);
-bool MACbPSWakeup(struct vnt_private *);
+bool MACbPSWakeup(struct vnt_private *priv);
-void MACvSetKeyEntry(struct vnt_private *, unsigned short wKeyCtl,
+void MACvSetKeyEntry(struct vnt_private *priv, unsigned short wKeyCtl,
unsigned int uEntryIdx, unsigned int uKeyIdx,
unsigned char *pbyAddr, u32 *pdwKey,
unsigned char byLocalID);
-void MACvDisableKeyEntry(struct vnt_private *, unsigned int uEntryIdx);
+void MACvDisableKeyEntry(struct vnt_private *priv, unsigned int uEntryIdx);
#endif /* __MAC_H__ */
#define PS_FAST_INTERVAL 1 /* Fast power saving listen interval */
#define PS_MAX_INTERVAL 4 /* MAX power saving listen interval */
-void
-PSvDisablePowerSaving(
- struct vnt_private *
-);
+void PSvDisablePowerSaving(struct vnt_private *priv);
-void
-PSvEnablePowerSaving(
- struct vnt_private *,
- unsigned short wListenInterval
-);
+void PSvEnablePowerSaving(struct vnt_private *priv, unsigned short wListenInterval);
-bool
-PSbIsNextTBTTWakeUp(
- struct vnt_private *
-);
+bool PSbIsNextTBTTWakeUp(struct vnt_private *priv);
#endif /* __POWER_H__ */
/*--------------------- Export Functions --------------------------*/
-bool IFRFbWriteEmbedded(struct vnt_private *, unsigned long dwData);
-bool RFbSelectChannel(struct vnt_private *, unsigned char byRFType, u16);
+bool IFRFbWriteEmbedded(struct vnt_private *priv, unsigned long dwData);
+bool RFbSelectChannel(struct vnt_private *priv, unsigned char byRFType, u16 byChannel);
bool RFbInit(
- struct vnt_private *
+ struct vnt_private *priv
);
-bool RFvWriteWakeProgSyn(struct vnt_private *, unsigned char byRFType, u16);
-bool RFbSetPower(struct vnt_private *, unsigned int rate, u16);
+bool RFvWriteWakeProgSyn(struct vnt_private *priv, unsigned char byRFType, u16 uChannel);
+bool RFbSetPower(struct vnt_private *priv, unsigned int rate, u16 uCH);
bool RFbRawSetPower(
- struct vnt_private *,
+ struct vnt_private *priv,
unsigned char byPwr,
unsigned int rate
);
void
RFvRSSITodBm(
- struct vnt_private *,
+ struct vnt_private *priv,
unsigned char byCurrRSSI,
long *pldBm
);
/* {{ RobertYu: 20050104 */
-bool RFbAL7230SelectChannelPostProcess(struct vnt_private *, u16, u16);
+bool RFbAL7230SelectChannelPostProcess(struct vnt_private *priv, u16 byOldChannel, u16 byNewChannel);
/* }} RobertYu */
#endif /* __RF_H__ */
priv->sifs = C_SIFS_A;
priv->difs = C_SIFS_A + 2 * C_SLOT_SHORT;
max_min = 4;
- } else if (priv->packet_type == PK_TYPE_11B) {
- priv->slot = C_SLOT_LONG;
- priv->sifs = C_SIFS_BG;
- priv->difs = C_SIFS_BG + 2 * C_SLOT_LONG;
- max_min = 5;
- } else {/* PK_TYPE_11GA & PK_TYPE_11GB */
- bool ofdm_rate = false;
- unsigned int ii = 0;
-
+ } else {
priv->sifs = C_SIFS_BG;
- if (priv->short_slot_time)
+ if (priv->short_slot_time) {
priv->slot = C_SLOT_SHORT;
- else
+ max_min = 4;
+ } else {
priv->slot = C_SLOT_LONG;
-
- priv->difs = C_SIFS_BG + 2 * priv->slot;
-
- for (ii = RATE_54M; ii >= RATE_6M; ii--) {
- if (priv->basic_rates & ((u32)(0x1 << ii))) {
- ofdm_rate = true;
- break;
- }
+ max_min = 5;
}
- if (ofdm_rate)
- max_min = 4;
- else
- max_min = 5;
+ priv->difs = C_SIFS_BG + 2 * priv->slot;
}
priv->eifs = C_EIFS;
goto free_all;
}
+ if (vnt_key_init_table(priv))
+ goto free_all;
+
priv->int_interval = 1; /* bInterval is set to 1 */
vnt_int_start_interrupt(priv);
{
struct vnt_private *priv = hw->priv;
struct ieee80211_conf *conf = &hw->conf;
- u8 bb_type;
if (changed & IEEE80211_CONF_CHANGE_PS) {
if (conf->flags & IEEE80211_CONF_PS)
vnt_set_channel(priv, conf->chandef.chan->hw_value);
if (conf->chandef.chan->band == NL80211_BAND_5GHZ)
- bb_type = BB_TYPE_11A;
+ priv->bb_type = BB_TYPE_11A;
else
- bb_type = BB_TYPE_11G;
-
- if (priv->bb_type != bb_type) {
- priv->bb_type = bb_type;
-
- vnt_set_bss_mode(priv);
- }
+ priv->bb_type = BB_TYPE_11G;
}
if (changed & IEEE80211_CONF_CHANGE_POWER) {
priv->basic_rates = conf->basic_rates;
vnt_update_top_rates(priv);
+ vnt_set_bss_mode(priv);
dev_dbg(&priv->usb->dev, "basic rates %x\n", conf->basic_rates);
}
priv->short_slot_time = false;
vnt_set_short_slot_time(priv);
+ vnt_update_ifs(priv);
vnt_set_vga_gain_offset(priv, priv->bb_vga[0]);
vnt_update_pre_ed_threshold(priv, false);
}
{
struct vnt_private *priv = hw->priv;
- vnt_set_bss_mode(priv);
/* Set max sensitivity*/
vnt_update_pre_ed_threshold(priv, true);
}
}
static u32 vnt_get_rsvtime(struct vnt_private *priv, u8 pkt_type,
- u32 frame_length, u16 rate, int need_ack)
+ u32 frame_length, u16 rate, int need_ack)
{
u32 data_time, ack_time;
}
static __le16 vnt_rxtx_rsvtime_le16(struct vnt_private *priv, u8 pkt_type,
- u32 frame_length, u16 rate, int need_ack)
+ u32 frame_length, u16 rate, int need_ack)
{
return cpu_to_le16((u16)vnt_get_rsvtime(priv, pkt_type,
frame_length, rate, need_ack));
}
-static __le16 vnt_get_rtscts_rsvtime_le(struct vnt_private *priv,
- u8 rsv_type, u8 pkt_type, u32 frame_length, u16 current_rate)
+static __le16 vnt_get_rtscts_rsvtime_le(struct vnt_private *priv, u8 rsv_type,
+ u8 pkt_type, u32 frame_length,
+ u16 current_rate)
{
u32 rrv_time, rts_time, cts_time, ack_time, data_time;
- rrv_time = rts_time = cts_time = ack_time = data_time = 0;
+ rrv_time = 0;
+ rts_time = 0;
+ cts_time = 0;
+ ack_time = 0;
data_time = vnt_get_frame_time(priv->preamble_type, pkt_type,
frame_length, current_rate);
if (rsv_type == 0) {
- rts_time = vnt_get_frame_time(priv->preamble_type,
- pkt_type, 20, priv->top_cck_basic_rate);
- cts_time = ack_time = vnt_get_frame_time(priv->preamble_type,
- pkt_type, 14, priv->top_cck_basic_rate);
+ rts_time = vnt_get_frame_time(priv->preamble_type, pkt_type,
+ 20, priv->top_cck_basic_rate);
+ ack_time = vnt_get_frame_time(priv->preamble_type,
+ pkt_type, 14,
+ priv->top_cck_basic_rate);
+ cts_time = ack_time;
+
} else if (rsv_type == 1) {
- rts_time = vnt_get_frame_time(priv->preamble_type,
- pkt_type, 20, priv->top_cck_basic_rate);
+ rts_time = vnt_get_frame_time(priv->preamble_type, pkt_type,
+ 20, priv->top_cck_basic_rate);
cts_time = vnt_get_frame_time(priv->preamble_type, pkt_type,
14, priv->top_cck_basic_rate);
ack_time = vnt_get_frame_time(priv->preamble_type, pkt_type,
} else if (rsv_type == 2) {
rts_time = vnt_get_frame_time(priv->preamble_type, pkt_type,
20, priv->top_ofdm_basic_rate);
- cts_time = ack_time = vnt_get_frame_time(priv->preamble_type,
- pkt_type, 14, priv->top_ofdm_basic_rate);
+ ack_time = vnt_get_frame_time(priv->preamble_type,
+ pkt_type, 14,
+ priv->top_ofdm_basic_rate);
+ cts_time = ack_time;
+
} else if (rsv_type == 3) {
cts_time = vnt_get_frame_time(priv->preamble_type, pkt_type,
14, priv->top_cck_basic_rate);
return cpu_to_le16((u16)rrv_time);
}
-static __le16 vnt_get_duration_le(struct vnt_private *priv,
- u8 pkt_type, int need_ack)
+static __le16 vnt_get_duration_le(struct vnt_private *priv, u8 pkt_type,
+ int need_ack)
{
u32 ack_time = 0;
if (need_ack) {
if (pkt_type == PK_TYPE_11B)
ack_time = vnt_get_frame_time(priv->preamble_type,
- pkt_type, 14, priv->top_cck_basic_rate);
+ pkt_type, 14,
+ priv->top_cck_basic_rate);
else
ack_time = vnt_get_frame_time(priv->preamble_type,
- pkt_type, 14, priv->top_ofdm_basic_rate);
+ pkt_type, 14,
+ priv->top_ofdm_basic_rate);
return cpu_to_le16((u16)(priv->sifs + ack_time));
}
case RTSDUR_BA:
case RTSDUR_BA_F0:
case RTSDUR_BA_F1:
- cts_time = vnt_get_frame_time(priv->preamble_type,
- pkt_type, 14, priv->top_cck_basic_rate);
+ cts_time = vnt_get_frame_time(priv->preamble_type, pkt_type,
+ 14, priv->top_cck_basic_rate);
dur_time = cts_time + 2 * priv->sifs +
vnt_get_rsvtime(priv, pkt_type,
frame_length, rate, need_ack);
case RTSDUR_AA:
case RTSDUR_AA_F0:
case RTSDUR_AA_F1:
- cts_time = vnt_get_frame_time(priv->preamble_type,
- pkt_type, 14, priv->top_ofdm_basic_rate);
+ cts_time = vnt_get_frame_time(priv->preamble_type, pkt_type,
+ 14, priv->top_ofdm_basic_rate);
dur_time = cts_time + 2 * priv->sifs +
vnt_get_rsvtime(priv, pkt_type,
frame_length, rate, need_ack);
}
static u16 vnt_mac_hdr_pos(struct vnt_usb_send_context *tx_context,
- struct ieee80211_hdr *hdr)
+ struct ieee80211_hdr *hdr)
{
u8 *head = tx_context->data + offsetof(struct vnt_tx_buffer, fifo_head);
u8 *hdr_pos = (u8 *)hdr;
static u16 vnt_rxtx_datahead_g(struct vnt_usb_send_context *tx_context,
struct vnt_tx_datahead_g *buf)
{
-
struct vnt_private *priv = tx_context->priv;
struct ieee80211_hdr *hdr =
(struct ieee80211_hdr *)tx_context->skb->data;
/* Get SignalField,ServiceField,Length */
vnt_get_phy_field(priv, frame_len, rate, tx_context->pkt_type, &buf->a);
vnt_get_phy_field(priv, frame_len, priv->top_cck_basic_rate,
- PK_TYPE_11B, &buf->b);
+ PK_TYPE_11B, &buf->b);
/* Get Duration and TimeStamp */
if (ieee80211_is_pspoll(hdr->frame_control)) {
vnt_get_phy_field(priv, frame_len, rate, tx_context->pkt_type, &buf->a);
vnt_get_phy_field(priv, frame_len, priv->top_cck_basic_rate,
- PK_TYPE_11B, &buf->b);
+ PK_TYPE_11B, &buf->b);
/* Get Duration and TimeStamp */
buf->duration_a = vnt_get_duration_le(priv, tx_context->pkt_type,
}
static int vnt_fill_ieee80211_rts(struct vnt_usb_send_context *tx_context,
- struct ieee80211_rts *rts, __le16 duration)
+ struct ieee80211_rts *rts, __le16 duration)
{
struct ieee80211_hdr *hdr =
(struct ieee80211_hdr *)tx_context->skb->data;
u16 current_rate = tx_context->tx_rate;
u16 rts_frame_len = 20;
- vnt_get_phy_field(priv, rts_frame_len,
- priv->top_ofdm_basic_rate, tx_context->pkt_type, &buf->a);
+ vnt_get_phy_field(priv, rts_frame_len, priv->top_ofdm_basic_rate,
+ tx_context->pkt_type, &buf->a);
buf->duration = vnt_get_rtscts_duration_le(tx_context, RTSDUR_AA,
tx_context->pkt_type,
}
static u16 vnt_generate_tx_parameter(struct vnt_usb_send_context *tx_context,
- struct vnt_tx_buffer *tx_buffer,
- struct vnt_mic_hdr **mic_hdr, u32 need_mic,
- bool need_rts)
+ struct vnt_tx_buffer *tx_buffer,
+ struct vnt_mic_hdr **mic_hdr, u32 need_mic,
+ bool need_rts)
{
-
if (tx_context->pkt_type == PK_TYPE_11GB ||
tx_context->pkt_type == PK_TYPE_11GA) {
if (need_rts) {
}
static void vnt_fill_txkey(struct vnt_usb_send_context *tx_context,
- u8 *key_buffer, struct ieee80211_key_conf *tx_key, struct sk_buff *skb,
- u16 payload_len, struct vnt_mic_hdr *mic_hdr)
+ u8 *key_buffer, struct ieee80211_key_conf *tx_key,
+ struct sk_buff *skb, u16 payload_len,
+ struct vnt_mic_hdr *mic_hdr)
{
struct ieee80211_hdr *hdr = tx_context->hdr;
u64 pn64;
if (ieee80211_has_a4(hdr->frame_control))
ether_addr_copy(mic_hdr->addr4, hdr->addr4);
-
memcpy(key_buffer, tx_key->key, WLAN_KEY_LEN_CCMP);
break;
default:
break;
}
-
}
int vnt_tx_packet(struct vnt_private *priv, struct sk_buff *skb)
current_rate = rate->hw_value;
if (priv->current_rate != current_rate &&
- !(priv->hw->conf.flags & IEEE80211_CONF_OFFCHANNEL)) {
+ !(priv->hw->conf.flags & IEEE80211_CONF_OFFCHANNEL)) {
priv->current_rate = current_rate;
vnt_schedule_command(priv, WLAN_CMD_SETPOWER);
}
tx_key = info->control.hw_key;
if (tx_key->keylen > 0)
vnt_fill_txkey(tx_context, tx_buffer_head->tx_key,
- tx_key, skb, tx_body_size, mic_hdr);
+ tx_key, skb, tx_body_size, mic_hdr);
}
priv->seq_counter = (le16_to_cpu(hdr->seq_ctrl) &
return 0;
}
-static int vnt_beacon_xmit(struct vnt_private *priv,
- struct sk_buff *skb)
+static int vnt_beacon_xmit(struct vnt_private *priv, struct sk_buff *skb)
{
struct vnt_beacon_buffer *beacon_buffer;
struct vnt_tx_short_buf_head *short_head;
}
int vnt_beacon_enable(struct vnt_private *priv, struct ieee80211_vif *vif,
- struct ieee80211_bss_conf *conf)
+ struct ieee80211_bss_conf *conf)
{
vnt_mac_reg_bits_off(priv, MAC_REG_TCR, TCR_AUTOBCNTX);
netdev_err(vif->ndev, "Failed to set channel\n");
}
-static void handle_set_wfi_drv_handler(struct wilc_vif *vif,
- struct drv_handler *hif_drv_handler)
+static int handle_set_wfi_drv_handler(struct wilc_vif *vif,
+ struct drv_handler *hif_drv_handler)
{
int ret = 0;
struct wid wid;
+ u8 *currbyte, *buffer;
+ struct host_if_drv *hif_drv = NULL;
+
+ if (!vif->hif_drv)
+ return -EINVAL;
+
+ if (!hif_drv_handler)
+ return -EINVAL;
+
+ hif_drv = vif->hif_drv;
+
+ buffer = kzalloc(DRV_HANDLER_SIZE, GFP_KERNEL);
+ if (!buffer)
+ return -ENOMEM;
+
+ currbyte = buffer;
+ *currbyte = hif_drv->driver_handler_id & DRV_HANDLER_MASK;
+ currbyte++;
+ *currbyte = (u32)0 & DRV_HANDLER_MASK;
+ currbyte++;
+ *currbyte = (u32)0 & DRV_HANDLER_MASK;
+ currbyte++;
+ *currbyte = (u32)0 & DRV_HANDLER_MASK;
+ currbyte++;
+ *currbyte = (hif_drv_handler->name | (hif_drv_handler->mode << 1));
wid.id = (u16)WID_SET_DRV_HANDLER;
wid.type = WID_STR;
- wid.val = (s8 *)hif_drv_handler;
- wid.size = sizeof(*hif_drv_handler);
+ wid.val = (s8 *)buffer;
+ wid.size = DRV_HANDLER_SIZE;
ret = wilc_send_config_pkt(vif, SET_CFG, &wid, 1,
- hif_drv_handler->handler);
-
- if (!hif_drv_handler->handler)
- complete(&hif_driver_comp);
-
- if (ret)
+ hif_drv->driver_handler_id);
+ if (ret) {
netdev_err(vif->ndev, "Failed to set driver handler\n");
+ complete(&hif_driver_comp);
+ kfree(buffer);
+ return ret;
+ }
+ complete(&hif_driver_comp);
+ kfree(buffer);
+ return 0;
}
static void handle_set_operation_mode(struct wilc_vif *vif,
result = wilc_send_config_pkt(vif, SET_CFG, &wid, 1,
wilc_get_vif_idx(vif));
if (result)
- netdev_err(vif->ndev, "Failed to send dissconect\n");
+ netdev_err(vif->ndev, "Failed to send disconnect\n");
hif_drv->usr_conn_req.ssid_len = 0;
kfree(hif_drv->usr_conn_req.ssid);
pu8CurrByte += pstrStationParam->rates_len;
*pu8CurrByte++ = pstrStationParam->ht_supported;
- *pu8CurrByte++ = pstrStationParam->ht_capa_info & 0xFF;
- *pu8CurrByte++ = (pstrStationParam->ht_capa_info >> 8) & 0xFF;
-
- *pu8CurrByte++ = pstrStationParam->ht_ampdu_params;
- memcpy(pu8CurrByte, pstrStationParam->ht_supp_mcs_set,
- WILC_SUPP_MCS_SET_SIZE);
- pu8CurrByte += WILC_SUPP_MCS_SET_SIZE;
-
- *pu8CurrByte++ = pstrStationParam->ht_ext_params & 0xFF;
- *pu8CurrByte++ = (pstrStationParam->ht_ext_params >> 8) & 0xFF;
-
- *pu8CurrByte++ = pstrStationParam->ht_tx_bf_cap & 0xFF;
- *pu8CurrByte++ = (pstrStationParam->ht_tx_bf_cap >> 8) & 0xFF;
- *pu8CurrByte++ = (pstrStationParam->ht_tx_bf_cap >> 16) & 0xFF;
- *pu8CurrByte++ = (pstrStationParam->ht_tx_bf_cap >> 24) & 0xFF;
-
- *pu8CurrByte++ = pstrStationParam->ht_ante_sel;
+ memcpy(pu8CurrByte, &pstrStationParam->ht_capa,
+ sizeof(struct ieee80211_ht_cap));
+ pu8CurrByte += sizeof(struct ieee80211_ht_cap);
*pu8CurrByte++ = pstrStationParam->flags_mask & 0xFF;
*pu8CurrByte++ = (pstrStationParam->flags_mask >> 8) & 0xFF;
pu8CurrByte = wid.val;
*pu8CurrByte++ = (strHostIfSetMulti->enabled & 0xFF);
- *pu8CurrByte++ = 0;
- *pu8CurrByte++ = 0;
- *pu8CurrByte++ = 0;
+ *pu8CurrByte++ = ((strHostIfSetMulti->enabled >> 8) & 0xFF);
+ *pu8CurrByte++ = ((strHostIfSetMulti->enabled >> 16) & 0xFF);
+ *pu8CurrByte++ = ((strHostIfSetMulti->enabled >> 24) & 0xFF);
*pu8CurrByte++ = (strHostIfSetMulti->cnt & 0xFF);
*pu8CurrByte++ = ((strHostIfSetMulti->cnt >> 8) & 0xFF);
{
struct host_if_msg *msg;
struct wilc *wilc;
+ int ret = 0;
msg = container_of(work, struct host_if_msg, work);
wilc = msg->vif->wilc;
break;
case HOST_IF_MSG_SET_WFIDRV_HANDLER:
- handle_set_wfi_drv_handler(msg->vif, &msg->body.drv);
+ ret = handle_set_wfi_drv_handler(msg->vif, &msg->body.drv);
break;
case HOST_IF_MSG_SET_OPERATION_MODE:
break;
}
free_msg:
+ if (ret)
+ netdev_err(msg->vif->ndev, "Host cmd %d failed\n", msg->id);
kfree(msg);
complete(&hif_thread_comp);
}
return 0;
}
-int wilc_set_wfi_drv_handler(struct wilc_vif *vif, int index, u8 mac_idx)
+int wilc_set_wfi_drv_handler(struct wilc_vif *vif, int index, u8 mode,
+ u8 ifc_id)
{
int result = 0;
struct host_if_msg msg;
memset(&msg, 0, sizeof(struct host_if_msg));
msg.id = HOST_IF_MSG_SET_WFIDRV_HANDLER;
msg.body.drv.handler = index;
- msg.body.drv.mac_idx = mac_idx;
+ msg.body.drv.mode = mode;
+ msg.body.drv.name = ifc_id;
msg.vif = vif;
result = wilc_enqueue_cmd(&msg);
for (i = 0; i < wilc->vif_num; i++)
if (dev == wilc->vif[i]->ndev) {
wilc->vif[i]->hif_drv = hif_drv;
+ hif_drv->driver_handler_id = i + 1;
break;
}
del_timer_sync(&periodic_rssi);
del_timer_sync(&hif_drv->remain_on_ch_timer);
- wilc_set_wfi_drv_handler(vif, 0, 0);
+ wilc_set_wfi_drv_handler(vif, 0, 0, 0);
wait_for_completion(&hif_driver_comp);
if (hif_drv->usr_scan_req.scan_result) {
#ifndef HOST_INT_H
#define HOST_INT_H
-
+#include <linux/ieee80211.h>
#include "coreconfigurator.h"
#define IP_ALEN 4
#define ETH_ALEN 6
#define PMKID_LEN 16
#define WILC_MAX_NUM_PMKIDS 16
-#define WILC_SUPP_MCS_SET_SIZE 16
#define WILC_ADD_STA_LENGTH 40
#define SCAN_EVENT_DONE_ABORTED
#define NUM_CONCURRENT_IFC 2
+#define DRV_HANDLER_SIZE 5
+#define DRV_HANDLER_MASK 0x000000FF
struct rf_info {
u8 link_speed;
struct drv_handler {
u32 handler;
- u8 mac_idx;
+ u8 mode;
+ u8 name;
};
struct op_mode {
struct timer_list remain_on_ch_timer;
bool IFC_UP;
+ int driver_handler_id;
};
struct add_sta_param {
u8 rates_len;
const u8 *rates;
bool ht_supported;
- u16 ht_capa_info;
- u8 ht_ampdu_params;
- u8 ht_supp_mcs_set[16];
- u16 ht_ext_params;
- u32 ht_tx_bf_cap;
- u8 ht_ante_sel;
+ struct ieee80211_ht_cap ht_capa;
u16 flags_mask;
u16 flags_set;
};
void *user_arg);
int wilc_listen_state_expired(struct wilc_vif *vif, u32 session_id);
int wilc_frame_register(struct wilc_vif *vif, u16 frame_type, bool reg);
-int wilc_set_wfi_drv_handler(struct wilc_vif *vif, int index, u8 mac_idx);
+int wilc_set_wfi_drv_handler(struct wilc_vif *vif, int index, u8 mode,
+ u8 ifc_id);
int wilc_set_operation_mode(struct wilc_vif *vif, u32 mode);
int wilc_get_statistics(struct wilc_vif *vif, struct rf_info *stats);
void wilc_resolve_disconnect_aberration(struct wilc_vif *vif);
for (i = 0; i < wl->vif_num; i++) {
if (ndev == wl->vif[i]->ndev) {
- if (vif->iftype == AP_MODE) {
- wilc_set_wfi_drv_handler(vif,
- wilc_get_vif_idx(vif),
- 0);
- } else if (!wilc_wlan_get_num_conn_ifcs(wl)) {
- wilc_set_wfi_drv_handler(vif,
- wilc_get_vif_idx(vif),
- wl->open_ifcs);
- } else {
- if (memcmp(wl->vif[i ^ 1]->bssid,
- wl->vif[i ^ 1]->src_addr, 6))
- wilc_set_wfi_drv_handler(vif,
- wilc_get_vif_idx(vif),
- 0);
- else
- wilc_set_wfi_drv_handler(vif,
- wilc_get_vif_idx(vif),
- 1);
- }
+ wilc_set_wfi_drv_handler(vif, wilc_get_vif_idx(vif),
+ vif->iftype, vif->ifc_id);
wilc_set_operation_mode(vif, vif->iftype);
-
- wilc_get_mac_address(vif, mac_add);
- netdev_dbg(ndev, "Mac address: %pM\n", mac_add);
- memcpy(wl->vif[i]->src_addr, mac_add, ETH_ALEN);
-
break;
}
}
+ wilc_get_mac_address(vif, mac_add);
+ netdev_dbg(ndev, "Mac address: %pM\n", mac_add);
+ memcpy(wl->vif[i]->src_addr, mac_add, ETH_ALEN);
memcpy(ndev->dev_addr, wl->vif[i]->src_addr, ETH_ALEN);
vif = netdev_priv(ndev);
memset(vif, 0, sizeof(struct wilc_vif));
- if (i == 0)
+ if (i == 0) {
strcpy(ndev->name, "wlan%d");
- else
+ vif->ifc_id = 1;
+ } else {
strcpy(ndev->name, "p2p%d");
-
+ vif->ifc_id = 0;
+ }
vif->wilc = *wilc;
vif->ndev = ndev;
wl->vif[i] = vif;
static struct dentry *wilc_dir;
/*
- * --------------------------------------------------------------------------------
+ * ----------------------------------------------------------------------------
*/
#define DEBUG BIT(0)
#define INFO BIT(1)
EXPORT_SYMBOL_GPL(WILC_DEBUG_LEVEL);
/*
- * --------------------------------------------------------------------------------
+ * ----------------------------------------------------------------------------
*/
-static ssize_t wilc_debug_level_read(struct file *file, char __user *userbuf, size_t count, loff_t *ppos)
+static ssize_t wilc_debug_level_read(struct file *file, char __user *userbuf,
+ size_t count, loff_t *ppos)
{
char buf[128];
int res = 0;
if (*ppos > 0)
return 0;
- res = scnprintf(buf, sizeof(buf), "Debug Level: %x\n", atomic_read(&WILC_DEBUG_LEVEL));
+ res = scnprintf(buf, sizeof(buf), "Debug Level: %x\n",
+ atomic_read(&WILC_DEBUG_LEVEL));
return simple_read_from_buffer(userbuf, count, ppos, buf, res);
}
-static ssize_t wilc_debug_level_write(struct file *filp, const char __user *buf,
- size_t count, loff_t *ppos)
+static ssize_t wilc_debug_level_write(struct file *filp,
+ const char __user *buf, size_t count,
+ loff_t *ppos)
{
int flag = 0;
int ret;
return ret;
if (flag > DBG_LEVEL_ALL) {
- pr_info("%s, value (0x%08x) is out of range, stay previous flag (0x%08x)\n", __func__, flag, atomic_read(&WILC_DEBUG_LEVEL));
+ pr_info("%s, value (0x%08x) is out of range, stay previous flag (0x%08x)\n",
+ __func__, flag, atomic_read(&WILC_DEBUG_LEVEL));
return -EINVAL;
}
}
/*
- * --------------------------------------------------------------------------------
+ * ----------------------------------------------------------------------------
*/
#define FOPS(_open, _read, _write, _poll) { \
};
static struct wilc_debugfs_info_t debugfs_info[] = {
- { "wilc_debug_level", 0666, (DEBUG | ERR), FOPS(NULL, wilc_debug_level_read, wilc_debug_level_write, NULL), },
+ {
+ "wilc_debug_level",
+ 0666,
+ (DEBUG | ERR),
+ FOPS(NULL, wilc_debug_level_read, wilc_debug_level_write, NULL),
+ },
};
static int __init wilc_debugfs_init(void)
module_exit(wilc_debugfs_remove);
#endif
-
#define TCP_ACK_FILTER_LINK_SPEED_THRESH 54
#define DEFAULT_LINK_SPEED 72
-
#define IS_MANAGMEMENT 0x100
#define IS_MANAGMEMENT_CALLBACK 0x080
#define IS_MGMT_STATUS_SUCCES 0x040
.n_bitrates = ARRAY_SIZE(ieee80211_bitrates),
};
-
struct add_key_params {
u8 key_idx;
bool pairwise;
u8 *mac_addr;
};
+
static struct add_key_params g_add_gtk_key_params;
static struct wilc_wfi_key g_key_gtk_params;
static struct add_key_params g_add_ptk_key_params;
unsigned long now = jiffies;
int i, j;
-
for (i = 0; i < last_scanned_cnt; i++) {
if (time_after(now, last_scanned_shadow[i].time_scan +
(unsigned long)(SCAN_RESULT_EXPIRE))) {
memcpy(priv->au8AssociatedBss, pstrConnectInfo->bssid, ETH_ALEN);
-
for (i = 0; i < last_scanned_cnt; i++) {
if (memcmp(last_scanned_shadow[i].bssid,
pstrConnectInfo->bssid,
return -ENOMEM;
strHiddenNetwork.n_ssids = request->n_ssids;
-
for (i = 0; i < request->n_ssids; i++) {
if (request->ssids[i].ssid_len != 0) {
strHiddenNetwork.net_info[i].ssid = kmalloc(request->ssids[i].ssid_len, GFP_KERNEL);
priv->wilc_ptk[key_index]->seq = NULL;
}
-
-
if (!pairwise) {
if (params->cipher == WLAN_CIPHER_SUITE_TKIP)
u8gmode = ENCRYPT_ENABLED | WPA | TKIP;
else
u8pmode = priv->wilc_groupkey | AES;
-
if (params->key_len > 16 && params->cipher == WLAN_CIPHER_SUITE_TKIP) {
pu8TxMic = params->key + 24;
pu8RxMic = params->key + 16;
priv = wiphy_priv(wiphy);
-
if (!pairwise) {
key_params.key = priv->wilc_gtk[key_index]->key;
key_params.cipher = priv->wilc_gtk[key_index]->cipher;
vif = netdev_priv(priv->dev);
-
for (i = 0; i < priv->pmkid_list.numpmkid; i++) {
if (!memcmp(pmksa->bssid, priv->pmkid_list.pmkidlist[i].bssid,
ETH_ALEN)) {
}
}
-
if ((buff[P2P_PUB_ACTION_SUBTYPE] == GO_NEG_REQ || buff[P2P_PUB_ACTION_SUBTYPE] == GO_NEG_RSP) && (wilc_ie)) {
cfg80211_rx_mgmt(priv->wdev, s32Freq, 0, buff, size - 7, 0);
return;
{
struct p2p_mgmt_data *pv_data = priv;
-
kfree(pv_data->buff);
kfree(pv_data);
}
memcpy(mgmt_tx->buff, buf, len);
mgmt_tx->size = len;
-
if (ieee80211_is_probe_resp(mgmt->frame_control)) {
wilc_set_mac_chnl_num(vif, chan->hw_value);
curr_channel = chan->hw_value;
if (wilc_enable_ps)
wilc_set_power_mgmt(vif, enabled, timeout);
-
return 0;
}
if (wl->initialized) {
wilc_set_wfi_drv_handler(vif, wilc_get_vif_idx(vif),
- 0);
+ 0, vif->ifc_id);
wilc_set_operation_mode(vif, AP_MODE);
wilc_set_power_mgmt(vif, 0, 0);
}
strStaParams.ht_supported = false;
} else {
strStaParams.ht_supported = true;
- strStaParams.ht_capa_info = params->ht_capa->cap_info;
- strStaParams.ht_ampdu_params = params->ht_capa->ampdu_params_info;
- memcpy(strStaParams.ht_supp_mcs_set,
- ¶ms->ht_capa->mcs,
- WILC_SUPP_MCS_SET_SIZE);
- strStaParams.ht_ext_params = params->ht_capa->extended_ht_cap_info;
- strStaParams.ht_tx_bf_cap = params->ht_capa->tx_BF_cap_info;
- strStaParams.ht_ante_sel = params->ht_capa->antenna_selection_info;
+ strStaParams.ht_capa = *params->ht_capa;
}
strStaParams.flags_mask = params->sta_flags_mask;
strStaParams.ht_supported = false;
} else {
strStaParams.ht_supported = true;
- strStaParams.ht_capa_info = params->ht_capa->cap_info;
- strStaParams.ht_ampdu_params = params->ht_capa->ampdu_params_info;
- memcpy(strStaParams.ht_supp_mcs_set,
- ¶ms->ht_capa->mcs,
- WILC_SUPP_MCS_SET_SIZE);
- strStaParams.ht_ext_params = params->ht_capa->extended_ht_cap_info;
- strStaParams.ht_tx_bf_cap = params->ht_capa->tx_BF_cap_info;
- strStaParams.ht_ante_sel = params->ht_capa->antenna_selection_info;
+ strStaParams.ht_capa = *params->ht_capa;
}
strStaParams.flags_mask = params->sta_flags_mask;
priv = wiphy_priv(wiphy);
vif = netdev_priv(priv->wdev->netdev);
-
if (type == NL80211_IFTYPE_MONITOR) {
new_ifc = WILC_WFI_init_mon_interface(name, vif->ndev);
if (new_ifc) {
struct host_if_drv *hif_drv;
struct net_device *ndev;
u8 mode;
+ u8 ifc_id;
};
struct wilc {
WID_MODEL_NAME = 0x3027, /*Added for CAPI tool */
WID_MODEL_NUM = 0x3028, /*Added for CAPI tool */
WID_DEVICE_NAME = 0x3029, /*Added for CAPI tool */
- WID_SET_DRV_HANDLER = 0x3030,
+ WID_SET_DRV_HANDLER = 0x3079,
/* NMAC String WID list */
WID_11N_P_ACTION_REQ = 0x3080,
/*-------------------------------------------------------------*/
/* Commonly used basic types */
struct hfa384x_bytestr {
- u16 len;
+ __le16 len;
u8 data[0];
} __packed;
struct hfa384x_bytestr32 {
- u16 len;
+ __le16 len;
u8 data[32];
} __packed;
/*-- Configuration Record: HostScanRequest (data portion only) --*/
struct hfa384x_host_scan_request_data {
- u16 channel_list;
- u16 tx_rate;
+ __le16 channel_list;
+ __le16 tx_rate;
struct hfa384x_bytestr32 ssid;
} __packed;
/*-- Configuration Record: WPAData (data portion only) --*/
struct hfa384x_wpa_data {
- u16 datalen;
+ __le16 datalen;
u8 data[0]; /* max 80 */
} __packed;
/*-- Inquiry Frame, Diagnose: Host Scan Results & Subfields--*/
struct hfa384x_hscan_result_sub {
- u16 chid;
- u16 anl;
- u16 sl;
+ __le16 chid;
+ __le16 anl;
+ __le16 sl;
u8 bssid[WLAN_BSSID_LEN];
- u16 bcnint;
- u16 capinfo;
+ __le16 bcnint;
+ __le16 capinfo;
struct hfa384x_bytestr32 ssid;
u8 supprates[10]; /* 802.11 info element */
u16 proberesp_rate;
- u16 atim;
+ __le16 atim;
} __packed;
struct hfa384x_hscan_result {
if (result)
return result;
- hw->bufinfo.page = le16_to_cpu(hw->bufinfo.page);
- hw->bufinfo.offset = le16_to_cpu(hw->bufinfo.offset);
- hw->bufinfo.len = le16_to_cpu(hw->bufinfo.len);
+ le16_to_cpus(&hw->bufinfo.page);
+ le16_to_cpus(&hw->bufinfo.offset);
+ le16_to_cpus(&hw->bufinfo.len);
result = hfa384x_drvr_getconfig16(hw, HFA384x_RID_MAXLOADTIME,
&hw->dltimeout);
if (result)
return result;
- hw->dltimeout = le16_to_cpu(hw->dltimeout);
+ le16_to_cpus(&hw->dltimeout);
pr_debug("flashdl_enable\n");
HFA384x_TX_MACPORT_SET(0) | HFA384x_TX_STRUCTYPE_SET(1) |
HFA384x_TX_TXEX_SET(0) | HFA384x_TX_TXOK_SET(0);
#endif
- hw->txbuff.txfrm.desc.tx_control =
- cpu_to_le16(hw->txbuff.txfrm.desc.tx_control);
+ cpu_to_le16s(&hw->txbuff.txfrm.desc.tx_control);
/* copy the header over to the txdesc */
memcpy(&hw->txbuff.txfrm.desc.frame_control, p80211_hdr,
u16 fc;
/* Byte order convert once up front. */
- usbin->rxfrm.desc.status = le16_to_cpu(usbin->rxfrm.desc.status);
- usbin->rxfrm.desc.time = le32_to_cpu(usbin->rxfrm.desc.time);
+ le16_to_cpus(&usbin->rxfrm.desc.status);
+ le32_to_cpus(&usbin->rxfrm.desc.time);
/* Now handle frame based on port# */
switch (HFA384x_RXSTATUS_MACPORT_GET(usbin->rxfrm.desc.status)) {
static void hfa384x_usbin_info(struct wlandevice *wlandev,
union hfa384x_usbin *usbin)
{
- usbin->infofrm.info.framelen =
- le16_to_cpu(usbin->infofrm.info.framelen);
+ le16_to_cpus(&usbin->infofrm.info.framelen);
prism2sta_ev_info(wlandev, &usbin->infofrm.info);
}
HFA384x_PDR_NICID) {
memcpy(&nicid, &pda->rec[pda->nrec]->data.nicid,
sizeof(nicid));
- nicid.id = le16_to_cpu(nicid.id);
- nicid.variant = le16_to_cpu(nicid.variant);
- nicid.major = le16_to_cpu(nicid.major);
- nicid.minor = le16_to_cpu(nicid.minor);
+ le16_to_cpus(&nicid.id);
+ le16_to_cpus(&nicid.variant);
+ le16_to_cpus(&nicid.major);
+ le16_to_cpus(&nicid.minor);
}
if (le16_to_cpu(pda->rec[pda->nrec]->code) ==
HFA384x_PDR_MFISUPRANGE) {
memcpy(&rfid, &pda->rec[pda->nrec]->data.mfisuprange,
sizeof(rfid));
- rfid.id = le16_to_cpu(rfid.id);
- rfid.variant = le16_to_cpu(rfid.variant);
- rfid.bottom = le16_to_cpu(rfid.bottom);
- rfid.top = le16_to_cpu(rfid.top);
+ le16_to_cpus(&rfid.id);
+ le16_to_cpus(&rfid.variant);
+ le16_to_cpus(&rfid.bottom);
+ le16_to_cpus(&rfid.top);
}
if (le16_to_cpu(pda->rec[pda->nrec]->code) ==
HFA384x_PDR_CFISUPRANGE) {
memcpy(&macid, &pda->rec[pda->nrec]->data.cfisuprange,
sizeof(macid));
- macid.id = le16_to_cpu(macid.id);
- macid.variant = le16_to_cpu(macid.variant);
- macid.bottom = le16_to_cpu(macid.bottom);
- macid.top = le16_to_cpu(macid.top);
+ le16_to_cpus(&macid.id);
+ le16_to_cpus(&macid.variant);
+ le16_to_cpus(&macid.bottom);
+ le16_to_cpus(&macid.top);
}
(pda->nrec)++;
hw->ident_sta_fw.variant) >
HFA384x_FIRMWARE_VERSION(1, 5, 0)) {
if (msg->scantype.data != P80211ENUM_scantype_active)
- word = cpu_to_le16(msg->maxchanneltime.data);
+ word = msg->maxchanneltime.data;
else
word = 0;
goto exit;
}
if (word == HFA384x_PORTSTATUS_DISABLED) {
- u16 wordbuf[17];
+ __le16 wordbuf[17];
result = hfa384x_drvr_setconfig16(hw,
HFA384x_RID_CNFROAMINGMODE,
void prism2mgmt_bytestr2pstr(struct hfa384x_bytestr *bytestr,
struct p80211pstrd *pstr)
{
- pstr->len = (u8)(le16_to_cpu((u16)(bytestr->len)));
+ pstr->len = (u8)(le16_to_cpu(bytestr->len));
memcpy(pstr->data, bytestr->data, pstr->len);
}
}
/* get all the nic id fields in host byte order */
- hw->ident_nic.id = le16_to_cpu(hw->ident_nic.id);
- hw->ident_nic.variant = le16_to_cpu(hw->ident_nic.variant);
- hw->ident_nic.major = le16_to_cpu(hw->ident_nic.major);
- hw->ident_nic.minor = le16_to_cpu(hw->ident_nic.minor);
+ le16_to_cpus(&hw->ident_nic.id);
+ le16_to_cpus(&hw->ident_nic.variant);
+ le16_to_cpus(&hw->ident_nic.major);
+ le16_to_cpus(&hw->ident_nic.minor);
netdev_info(wlandev->netdev, "ident: nic h/w: id=0x%02x %d.%d.%d\n",
hw->ident_nic.id, hw->ident_nic.major,
}
/* get all the private fw id fields in host byte order */
- hw->ident_pri_fw.id = le16_to_cpu(hw->ident_pri_fw.id);
- hw->ident_pri_fw.variant = le16_to_cpu(hw->ident_pri_fw.variant);
- hw->ident_pri_fw.major = le16_to_cpu(hw->ident_pri_fw.major);
- hw->ident_pri_fw.minor = le16_to_cpu(hw->ident_pri_fw.minor);
+ le16_to_cpus(&hw->ident_pri_fw.id);
+ le16_to_cpus(&hw->ident_pri_fw.variant);
+ le16_to_cpus(&hw->ident_pri_fw.major);
+ le16_to_cpus(&hw->ident_pri_fw.minor);
netdev_info(wlandev->netdev, "ident: pri f/w: id=0x%02x %d.%d.%d\n",
hw->ident_pri_fw.id, hw->ident_pri_fw.major,
}
/* get all the station fw id fields in host byte order */
- hw->ident_sta_fw.id = le16_to_cpu(hw->ident_sta_fw.id);
- hw->ident_sta_fw.variant = le16_to_cpu(hw->ident_sta_fw.variant);
- hw->ident_sta_fw.major = le16_to_cpu(hw->ident_sta_fw.major);
- hw->ident_sta_fw.minor = le16_to_cpu(hw->ident_sta_fw.minor);
+ le16_to_cpus(&hw->ident_sta_fw.id);
+ le16_to_cpus(&hw->ident_sta_fw.variant);
+ le16_to_cpus(&hw->ident_sta_fw.major);
+ le16_to_cpus(&hw->ident_sta_fw.minor);
/* strip out the 'special' variant bits */
hw->mm_mods = hw->ident_sta_fw.variant & GENMASK(15, 14);
/* get all the Compatibility range, modem interface supplier
* fields in byte order
*/
- hw->cap_sup_mfi.role = le16_to_cpu(hw->cap_sup_mfi.role);
- hw->cap_sup_mfi.id = le16_to_cpu(hw->cap_sup_mfi.id);
- hw->cap_sup_mfi.variant = le16_to_cpu(hw->cap_sup_mfi.variant);
- hw->cap_sup_mfi.bottom = le16_to_cpu(hw->cap_sup_mfi.bottom);
- hw->cap_sup_mfi.top = le16_to_cpu(hw->cap_sup_mfi.top);
+ le16_to_cpus(&hw->cap_sup_mfi.role);
+ le16_to_cpus(&hw->cap_sup_mfi.id);
+ le16_to_cpus(&hw->cap_sup_mfi.variant);
+ le16_to_cpus(&hw->cap_sup_mfi.bottom);
+ le16_to_cpus(&hw->cap_sup_mfi.top);
netdev_info(wlandev->netdev,
"MFI:SUP:role=0x%02x:id=0x%02x:var=0x%02x:b/t=%d/%d\n",
/* get all the Compatibility range, controller interface supplier
* fields in byte order
*/
- hw->cap_sup_cfi.role = le16_to_cpu(hw->cap_sup_cfi.role);
- hw->cap_sup_cfi.id = le16_to_cpu(hw->cap_sup_cfi.id);
- hw->cap_sup_cfi.variant = le16_to_cpu(hw->cap_sup_cfi.variant);
- hw->cap_sup_cfi.bottom = le16_to_cpu(hw->cap_sup_cfi.bottom);
- hw->cap_sup_cfi.top = le16_to_cpu(hw->cap_sup_cfi.top);
+ le16_to_cpus(&hw->cap_sup_cfi.role);
+ le16_to_cpus(&hw->cap_sup_cfi.id);
+ le16_to_cpus(&hw->cap_sup_cfi.variant);
+ le16_to_cpus(&hw->cap_sup_cfi.bottom);
+ le16_to_cpus(&hw->cap_sup_cfi.top);
netdev_info(wlandev->netdev,
"CFI:SUP:role=0x%02x:id=0x%02x:var=0x%02x:b/t=%d/%d\n",
/* get all the Compatibility range, primary firmware supplier
* fields in byte order
*/
- hw->cap_sup_pri.role = le16_to_cpu(hw->cap_sup_pri.role);
- hw->cap_sup_pri.id = le16_to_cpu(hw->cap_sup_pri.id);
- hw->cap_sup_pri.variant = le16_to_cpu(hw->cap_sup_pri.variant);
- hw->cap_sup_pri.bottom = le16_to_cpu(hw->cap_sup_pri.bottom);
- hw->cap_sup_pri.top = le16_to_cpu(hw->cap_sup_pri.top);
+ le16_to_cpus(&hw->cap_sup_pri.role);
+ le16_to_cpus(&hw->cap_sup_pri.id);
+ le16_to_cpus(&hw->cap_sup_pri.variant);
+ le16_to_cpus(&hw->cap_sup_pri.bottom);
+ le16_to_cpus(&hw->cap_sup_pri.top);
netdev_info(wlandev->netdev,
"PRI:SUP:role=0x%02x:id=0x%02x:var=0x%02x:b/t=%d/%d\n",
/* get all the Compatibility range, station firmware supplier
* fields in byte order
*/
- hw->cap_sup_sta.role = le16_to_cpu(hw->cap_sup_sta.role);
- hw->cap_sup_sta.id = le16_to_cpu(hw->cap_sup_sta.id);
- hw->cap_sup_sta.variant = le16_to_cpu(hw->cap_sup_sta.variant);
- hw->cap_sup_sta.bottom = le16_to_cpu(hw->cap_sup_sta.bottom);
- hw->cap_sup_sta.top = le16_to_cpu(hw->cap_sup_sta.top);
+ le16_to_cpus(&hw->cap_sup_sta.role);
+ le16_to_cpus(&hw->cap_sup_sta.id);
+ le16_to_cpus(&hw->cap_sup_sta.variant);
+ le16_to_cpus(&hw->cap_sup_sta.bottom);
+ le16_to_cpus(&hw->cap_sup_sta.top);
if (hw->cap_sup_sta.id == 0x04) {
netdev_info(wlandev->netdev,
/* get all the Compatibility range, primary f/w actor, CFI supplier
* fields in byte order
*/
- hw->cap_act_pri_cfi.role = le16_to_cpu(hw->cap_act_pri_cfi.role);
- hw->cap_act_pri_cfi.id = le16_to_cpu(hw->cap_act_pri_cfi.id);
- hw->cap_act_pri_cfi.variant = le16_to_cpu(hw->cap_act_pri_cfi.variant);
- hw->cap_act_pri_cfi.bottom = le16_to_cpu(hw->cap_act_pri_cfi.bottom);
- hw->cap_act_pri_cfi.top = le16_to_cpu(hw->cap_act_pri_cfi.top);
+ le16_to_cpus(&hw->cap_act_pri_cfi.role);
+ le16_to_cpus(&hw->cap_act_pri_cfi.id);
+ le16_to_cpus(&hw->cap_act_pri_cfi.variant);
+ le16_to_cpus(&hw->cap_act_pri_cfi.bottom);
+ le16_to_cpus(&hw->cap_act_pri_cfi.top);
netdev_info(wlandev->netdev,
"PRI-CFI:ACT:role=0x%02x:id=0x%02x:var=0x%02x:b/t=%d/%d\n",
/* get all the Compatibility range, station f/w actor, CFI supplier
* fields in byte order
*/
- hw->cap_act_sta_cfi.role = le16_to_cpu(hw->cap_act_sta_cfi.role);
- hw->cap_act_sta_cfi.id = le16_to_cpu(hw->cap_act_sta_cfi.id);
- hw->cap_act_sta_cfi.variant = le16_to_cpu(hw->cap_act_sta_cfi.variant);
- hw->cap_act_sta_cfi.bottom = le16_to_cpu(hw->cap_act_sta_cfi.bottom);
- hw->cap_act_sta_cfi.top = le16_to_cpu(hw->cap_act_sta_cfi.top);
+ le16_to_cpus(&hw->cap_act_sta_cfi.role);
+ le16_to_cpus(&hw->cap_act_sta_cfi.id);
+ le16_to_cpus(&hw->cap_act_sta_cfi.variant);
+ le16_to_cpus(&hw->cap_act_sta_cfi.bottom);
+ le16_to_cpus(&hw->cap_act_sta_cfi.top);
netdev_info(wlandev->netdev,
"STA-CFI:ACT:role=0x%02x:id=0x%02x:var=0x%02x:b/t=%d/%d\n",
/* get all the Compatibility range, station f/w actor, MFI supplier
* fields in byte order
*/
- hw->cap_act_sta_mfi.role = le16_to_cpu(hw->cap_act_sta_mfi.role);
- hw->cap_act_sta_mfi.id = le16_to_cpu(hw->cap_act_sta_mfi.id);
- hw->cap_act_sta_mfi.variant = le16_to_cpu(hw->cap_act_sta_mfi.variant);
- hw->cap_act_sta_mfi.bottom = le16_to_cpu(hw->cap_act_sta_mfi.bottom);
- hw->cap_act_sta_mfi.top = le16_to_cpu(hw->cap_act_sta_mfi.top);
+ le16_to_cpus(&hw->cap_act_sta_mfi.role);
+ le16_to_cpus(&hw->cap_act_sta_mfi.id);
+ le16_to_cpus(&hw->cap_act_sta_mfi.variant);
+ le16_to_cpus(&hw->cap_act_sta_mfi.bottom);
+ le16_to_cpus(&hw->cap_act_sta_mfi.top);
netdev_info(wlandev->netdev,
"STA-MFI:ACT:role=0x%02x:id=0x%02x:var=0x%02x:b/t=%d/%d\n",
int i;
memcpy(&rec, &inf->info.assocstatus, sizeof(rec));
- rec.assocstatus = le16_to_cpu(rec.assocstatus);
- rec.reason = le16_to_cpu(rec.reason);
+ le16_to_cpus(&rec.assocstatus);
+ le16_to_cpus(&rec.reason);
/*
* Find the address in the list of authenticated stations.
void prism2sta_ev_info(struct wlandevice *wlandev,
struct hfa384x_inf_frame *inf)
{
- inf->infotype = le16_to_cpu(inf->infotype);
+ le16_to_cpus(&inf->infotype);
/* Dispatch */
switch (inf->infotype) {
case HFA384x_IT_HANDOVERADDR:
}
},
{0xFFFF, {
- 0x04, 0x1A, 0x04, 0x7E, 0x02, 0x1B, 0x05, 0x7E, /* ; C0-C7 */
- 0x01, 0x1A, 0x07, 0x7E, 0x00, 0x1A, 0x09, 0x7D, /* ; C8-CF */
- 0x7F, 0x19, 0x0B, 0x7D, 0x7E, 0x18, 0x0D, 0x7D, /* ; D0-D7 */
- 0x7D, 0x17, 0x10, 0x7C, 0x7D, 0x15, 0x12, 0x7C, /* ; D8-DF */
- 0x7C, 0x14, 0x14, 0x7C, 0x7C, 0x12, 0x15, 0x7D, /* ; E0-E7 */
- 0x7C, 0x10, 0x17, 0x7D, 0x7C, 0x0D, 0x18, 0x7F, /* ; EA-EF */
- 0x7D, 0x0B, 0x19, 0x7F, 0x7D, 0x09, 0x1A, 0x00, /* ; F0-F7 */
- 0x7D, 0x07, 0x1A, 0x02, 0x7E, 0x05, 0x1B, 0x02 /* ; F8-FF */
- }
+ 0x04, 0x1A, 0x04, 0x7E, 0x02, 0x1B, 0x05, 0x7E, /* ; C0-C7 */
+ 0x01, 0x1A, 0x07, 0x7E, 0x00, 0x1A, 0x09, 0x7D, /* ; C8-CF */
+ 0x7F, 0x19, 0x0B, 0x7D, 0x7E, 0x18, 0x0D, 0x7D, /* ; D0-D7 */
+ 0x7D, 0x17, 0x10, 0x7C, 0x7D, 0x15, 0x12, 0x7C, /* ; D8-DF */
+ 0x7C, 0x14, 0x14, 0x7C, 0x7C, 0x12, 0x15, 0x7D, /* ; E0-E7 */
+ 0x7C, 0x10, 0x17, 0x7D, 0x7C, 0x0D, 0x18, 0x7F, /* ; EA-EF */
+ 0x7D, 0x0B, 0x19, 0x7F, 0x7D, 0x09, 0x1A, 0x00, /* ; F0-F7 */
+ 0x7D, 0x07, 0x1A, 0x02, 0x7E, 0x05, 0x1B, 0x02 /* ; F8-FF */
+ }
}
};
*/
if (dump_payload)
goto after_immediate_data;
+ /*
+ * Check for underflow case where both EDTL and immediate data payload
+ * exceeds what is presented by CDB's TRANSFER LENGTH, and what has
+ * already been set in target_cmd_size_check() as se_cmd->data_length.
+ *
+ * For this special case, fail the command and dump the immediate data
+ * payload.
+ */
+ if (cmd->first_burst_len > cmd->se_cmd.data_length) {
+ cmd->sense_reason = TCM_INVALID_CDB_FIELD;
+ goto after_immediate_data;
+ }
immed_ret = iscsit_handle_immediate_data(cmd, hdr,
cmd->first_burst_len);
* always sleep waiting for RX/TX thread shutdown to complete
* within iscsit_close_connection().
*/
- if (!conn->conn_transport->rdma_shutdown)
+ if (!conn->conn_transport->rdma_shutdown) {
sleep = cmpxchg(&conn->tx_thread_active, true, false);
+ if (!sleep)
+ return;
+ }
atomic_set(&conn->conn_logout_remove, 0);
complete(&conn->conn_logout_comp);
{
int sleep = 1;
- if (!conn->conn_transport->rdma_shutdown)
+ if (!conn->conn_transport->rdma_shutdown) {
sleep = cmpxchg(&conn->tx_thread_active, true, false);
+ if (!sleep)
+ return;
+ }
atomic_set(&conn->conn_logout_remove, 0);
complete(&conn->conn_logout_comp);
return -EINVAL;
}
- ib_dev->ibd_bio_set = bioset_create(IBLOCK_BIO_POOL_SIZE, 0);
+ ib_dev->ibd_bio_set = bioset_create(IBLOCK_BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS);
if (!ib_dev->ibd_bio_set) {
pr_err("IBLOCK: Unable to create bioset\n");
goto out;
struct se_cmd *cmd = bio->bi_private;
struct iblock_req *ibr = cmd->priv;
- if (bio->bi_error) {
- pr_err("bio error: %p, err: %d\n", bio, bio->bi_error);
+ if (bio->bi_status) {
+ pr_err("bio error: %p, err: %d\n", bio, bio->bi_status);
/*
* Bump the ib_bio_err_cnt and release bio.
*/
{
struct se_cmd *cmd = bio->bi_private;
- if (bio->bi_error)
- pr_err("IBLOCK: cache flush failed: %d\n", bio->bi_error);
+ if (bio->bi_status)
+ pr_err("IBLOCK: cache flush failed: %d\n", bio->bi_status);
if (cmd) {
- if (bio->bi_error)
+ if (bio->bi_status)
target_complete_cmd(cmd, SAM_STAT_CHECK_CONDITION);
else
target_complete_cmd(cmd, SAM_STAT_GOOD);
void release_se_kmem_caches(void);
u32 scsi_get_new_index(scsi_index_t);
void transport_subsystem_check_init(void);
-void transport_cmd_finish_abort(struct se_cmd *, int);
+int transport_cmd_finish_abort(struct se_cmd *, int);
unsigned char *transport_dump_cmd_direction(struct se_cmd *);
void transport_dump_dev_state(struct se_device *, char *, int *);
void transport_dump_dev_info(struct se_device *, struct se_lun *,
}
static sense_reason_t pscsi_execute_cmd(struct se_cmd *cmd);
-static void pscsi_req_done(struct request *, int);
+static void pscsi_req_done(struct request *, blk_status_t);
/* pscsi_attach_hba():
*
goto fail;
}
- scsi_req_init(req);
-
if (sgl) {
ret = pscsi_map_sg(cmd, sgl, sgl_nents, req);
if (ret)
return 0;
}
-static void pscsi_req_done(struct request *req, int uptodate)
+static void pscsi_req_done(struct request *req, blk_status_t status)
{
struct se_cmd *cmd = req->end_io_data;
struct pscsi_plugin_task *pt = cmd->priv;
kfree(tmr);
}
-static void core_tmr_handle_tas_abort(struct se_cmd *cmd, int tas)
+static int core_tmr_handle_tas_abort(struct se_cmd *cmd, int tas)
{
unsigned long flags;
bool remove = true, send_tas;
transport_send_task_abort(cmd);
}
- transport_cmd_finish_abort(cmd, remove);
+ return transport_cmd_finish_abort(cmd, remove);
}
static int target_check_cdb_and_preempt(struct list_head *list,
cancel_work_sync(&se_cmd->work);
transport_wait_for_tasks(se_cmd);
- transport_cmd_finish_abort(se_cmd, true);
- target_put_sess_cmd(se_cmd);
+ if (!transport_cmd_finish_abort(se_cmd, true))
+ target_put_sess_cmd(se_cmd);
printk("ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for"
" ref_tag: %llu\n", ref_tag);
cancel_work_sync(&cmd->work);
transport_wait_for_tasks(cmd);
- transport_cmd_finish_abort(cmd, 1);
- target_put_sess_cmd(cmd);
+ if (!transport_cmd_finish_abort(cmd, 1))
+ target_put_sess_cmd(cmd);
}
}
cancel_work_sync(&cmd->work);
transport_wait_for_tasks(cmd);
- core_tmr_handle_tas_abort(cmd, tas);
- target_put_sess_cmd(cmd);
+ if (!core_tmr_handle_tas_abort(cmd, tas))
+ target_put_sess_cmd(cmd);
}
}
percpu_ref_put(&lun->lun_ref);
}
-void transport_cmd_finish_abort(struct se_cmd *cmd, int remove)
+int transport_cmd_finish_abort(struct se_cmd *cmd, int remove)
{
bool ack_kref = (cmd->se_cmd_flags & SCF_ACK_KREF);
+ int ret = 0;
if (cmd->se_cmd_flags & SCF_SE_LUN_CMD)
transport_lun_remove_cmd(cmd);
cmd->se_tfo->aborted_task(cmd);
if (transport_cmd_check_stop_to_fabric(cmd))
- return;
+ return 1;
if (remove && ack_kref)
- transport_put_cmd(cmd);
+ ret = transport_put_cmd(cmd);
+
+ return ret;
}
static void target_complete_failure_work(struct work_struct *work)
INT3400_THERMAL_MAXIMUM_UUID,
};
-static u8 *int3400_thermal_uuids[INT3400_THERMAL_MAXIMUM_UUID] = {
+static char *int3400_thermal_uuids[INT3400_THERMAL_MAXIMUM_UUID] = {
"42A441D6-AE6A-462b-A84B-4A8CE79027D3",
"3A95C389-E4B8-4629-A526-C52C88626BAE",
"97C68AE7-15FA-499c-B8C9-5DA81D606E0A",
}
for (j = 0; j < INT3400_THERMAL_MAXIMUM_UUID; j++) {
- u8 uuid[16];
+ guid_t guid;
- acpi_str_to_uuid(int3400_thermal_uuids[j], uuid);
- if (!strncmp(uuid, objb->buffer.pointer, 16)) {
+ guid_parse(int3400_thermal_uuids[j], &guid);
+ if (guid_equal((guid_t *)objb->buffer.pointer, &guid)) {
priv->uuid_bitmap |= (1 << j);
break;
}
return -EINVAL;
}
- pdev->dev.of_node = pdev->dev.parent->of_node;
-
mtherm->dev = &pdev->dev;
mtherm->rmap = dev_get_regmap(pdev->dev.parent, NULL);
if (!mtherm->rmap) {
return -ENODEV;
}
+ /*
+ * The reference taken to the parent's node which will be balanced on
+ * reprobe or on platform-device release.
+ */
+ device_set_of_node_from_dev(&pdev->dev, pdev->dev.parent);
+
mtherm->tz_device = devm_thermal_zone_of_sensor_register(&pdev->dev, 0,
mtherm, &max77620_thermal_ops);
if (IS_ERR(mtherm->tz_device)) {
menuconfig THUNDERBOLT
- tristate "Thunderbolt support for Apple devices"
+ tristate "Thunderbolt support"
depends on PCI
depends on X86 || COMPILE_TEST
select APPLE_PROPERTIES if EFI_STUB && X86
select CRC32
+ select CRYPTO
+ select CRYPTO_HASH
+ select NVMEM
help
- Cactus Ridge Thunderbolt Controller driver
- This driver is required if you want to hotplug Thunderbolt devices on
- Apple hardware.
-
- Device chaining is currently not supported.
+ Thunderbolt Controller driver. This driver is required if you
+ want to hotplug Thunderbolt devices on Apple hardware or on PCs
+ with Intel Falcon Ridge or newer.
To compile this driver a module, choose M here. The module will be
called thunderbolt.
obj-${CONFIG_THUNDERBOLT} := thunderbolt.o
thunderbolt-objs := nhi.o ctl.o tb.o switch.o cap.o path.o tunnel_pci.o eeprom.o
-
+thunderbolt-objs += domain.o dma_port.o icm.o
#include "tb.h"
+#define CAP_OFFSET_MAX 0xff
+#define VSE_CAP_OFFSET_MAX 0xffff
struct tb_cap_any {
union {
};
} __packed;
-static bool tb_cap_is_basic(struct tb_cap_any *cap)
-{
- /* basic.cap is u8. This checks only the lower 8 bit of cap. */
- return cap->basic.cap != 5;
-}
-
-static bool tb_cap_is_long(struct tb_cap_any *cap)
+/**
+ * tb_port_find_cap() - Find port capability
+ * @port: Port to find the capability for
+ * @cap: Capability to look
+ *
+ * Returns offset to start of capability or %-ENOENT if no such
+ * capability was found. Negative errno is returned if there was an
+ * error.
+ */
+int tb_port_find_cap(struct tb_port *port, enum tb_port_cap cap)
{
- return !tb_cap_is_basic(cap)
- && cap->extended_short.next == 0
- && cap->extended_short.length == 0;
-}
+ u32 offset;
-static enum tb_cap tb_cap(struct tb_cap_any *cap)
-{
- if (tb_cap_is_basic(cap))
- return cap->basic.cap;
+ /*
+ * DP out adapters claim to implement TMU capability but in
+ * reality they do not so we hard code the adapter specific
+ * capability offset here.
+ */
+ if (port->config.type == TB_TYPE_DP_HDMI_OUT)
+ offset = 0x39;
else
- /* extended_short/long have cap at the same offset. */
- return cap->extended_short.cap;
+ offset = 0x1;
+
+ do {
+ struct tb_cap_any header;
+ int ret;
+
+ ret = tb_port_read(port, &header, TB_CFG_PORT, offset, 1);
+ if (ret)
+ return ret;
+
+ if (header.basic.cap == cap)
+ return offset;
+
+ offset = header.basic.next;
+ } while (offset);
+
+ return -ENOENT;
}
-static u32 tb_cap_next(struct tb_cap_any *cap, u32 offset)
+static int tb_switch_find_cap(struct tb_switch *sw, enum tb_switch_cap cap)
{
- int next;
- if (offset == 1) {
- /*
- * The first pointer is part of the switch header and always
- * a simple pointer.
- */
- next = cap->basic.next;
- } else {
- /*
- * Somehow Intel decided to use 3 different types of capability
- * headers. It is not like anyone could have predicted that
- * single byte offsets are not enough...
- */
- if (tb_cap_is_basic(cap))
- next = cap->basic.next;
- else if (!tb_cap_is_long(cap))
- next = cap->extended_short.next;
- else
- next = cap->extended_long.next;
+ int offset = sw->config.first_cap_offset;
+
+ while (offset > 0 && offset < CAP_OFFSET_MAX) {
+ struct tb_cap_any header;
+ int ret;
+
+ ret = tb_sw_read(sw, &header, TB_CFG_SWITCH, offset, 1);
+ if (ret)
+ return ret;
+
+ if (header.basic.cap == cap)
+ return offset;
+
+ offset = header.basic.next;
}
- /*
- * "Hey, we could terminate some capability lists with a null offset
- * and others with a pointer to the last element." - "Great idea!"
- */
- if (next == offset)
- return 0;
- return next;
+
+ return -ENOENT;
}
/**
- * tb_find_cap() - find a capability
+ * tb_switch_find_vse_cap() - Find switch vendor specific capability
+ * @sw: Switch to find the capability for
+ * @vsec: Vendor specific capability to look
*
- * Return: Returns a positive offset if the capability was found and 0 if not.
- * Returns an error code on failure.
+ * Functions enumerates vendor specific capabilities (VSEC) of a switch
+ * and returns offset when capability matching @vsec is found. If no
+ * such capability is found returns %-ENOENT. In case of error returns
+ * negative errno.
*/
-int tb_find_cap(struct tb_port *port, enum tb_cfg_space space, enum tb_cap cap)
+int tb_switch_find_vse_cap(struct tb_switch *sw, enum tb_switch_vse_cap vsec)
{
- u32 offset = 1;
struct tb_cap_any header;
- int res;
- int retries = 10;
- while (retries--) {
- res = tb_port_read(port, &header, space, offset, 1);
- if (res) {
- /* Intel needs some help with linked lists. */
- if (space == TB_CFG_PORT && offset == 0xa
- && port->config.type == TB_TYPE_DP_HDMI_OUT) {
- offset = 0x39;
- continue;
- }
- return res;
- }
- if (offset != 1) {
- if (tb_cap(&header) == cap)
+ int offset;
+
+ offset = tb_switch_find_cap(sw, TB_SWITCH_CAP_VSE);
+ if (offset < 0)
+ return offset;
+
+ while (offset > 0 && offset < VSE_CAP_OFFSET_MAX) {
+ int ret;
+
+ ret = tb_sw_read(sw, &header, TB_CFG_SWITCH, offset, 2);
+ if (ret)
+ return ret;
+
+ /*
+ * Extended vendor specific capabilities come in two
+ * flavors: short and long. The latter is used when
+ * offset is over 0xff.
+ */
+ if (offset >= CAP_OFFSET_MAX) {
+ if (header.extended_long.vsec_id == vsec)
return offset;
- if (tb_cap_is_long(&header)) {
- /* tb_cap_extended_long is 2 dwords */
- res = tb_port_read(port, &header, space,
- offset, 2);
- if (res)
- return res;
- }
+ offset = header.extended_long.next;
+ } else {
+ if (header.extended_short.vsec_id == vsec)
+ return offset;
+ if (!header.extended_short.length)
+ return -ENOENT;
+ offset = header.extended_short.next;
}
- offset = tb_cap_next(&header, offset);
- if (!offset)
- return 0;
}
- tb_port_WARN(port,
- "run out of retries while looking for cap %#x in config space %d, last offset: %#x\n",
- cap, space, offset);
- return -EIO;
+
+ return -ENOENT;
}
*/
#include <linux/crc32.h>
+#include <linux/delay.h>
#include <linux/slab.h>
#include <linux/pci.h>
#include <linux/dmapool.h>
#include <linux/workqueue.h>
-#include <linux/kfifo.h>
#include "ctl.h"
-struct ctl_pkg {
- struct tb_ctl *ctl;
- void *buffer;
- struct ring_frame frame;
-};
-
-#define TB_CTL_RX_PKG_COUNT 10
+#define TB_CTL_RX_PKG_COUNT 10
+#define TB_CTL_RETRIES 4
/**
* struct tb_cfg - thunderbolt control channel
struct dma_pool *frame_pool;
struct ctl_pkg *rx_packets[TB_CTL_RX_PKG_COUNT];
- DECLARE_KFIFO(response_fifo, struct ctl_pkg*, 16);
- struct completion response_ready;
+ struct mutex request_queue_lock;
+ struct list_head request_queue;
+ bool running;
- hotplug_cb callback;
+ event_cb callback;
void *callback_data;
};
#define tb_ctl_info(ctl, format, arg...) \
dev_info(&(ctl)->nhi->pdev->dev, format, ## arg)
+#define tb_ctl_dbg(ctl, format, arg...) \
+ dev_dbg(&(ctl)->nhi->pdev->dev, format, ## arg)
-/* configuration packets definitions */
+static DECLARE_WAIT_QUEUE_HEAD(tb_cfg_request_cancel_queue);
+/* Serializes access to request kref_get/put */
+static DEFINE_MUTEX(tb_cfg_request_lock);
-enum tb_cfg_pkg_type {
- TB_CFG_PKG_READ = 1,
- TB_CFG_PKG_WRITE = 2,
- TB_CFG_PKG_ERROR = 3,
- TB_CFG_PKG_NOTIFY_ACK = 4,
- TB_CFG_PKG_EVENT = 5,
- TB_CFG_PKG_XDOMAIN_REQ = 6,
- TB_CFG_PKG_XDOMAIN_RESP = 7,
- TB_CFG_PKG_OVERRIDE = 8,
- TB_CFG_PKG_RESET = 9,
- TB_CFG_PKG_PREPARE_TO_SLEEP = 0xd,
-};
+/**
+ * tb_cfg_request_alloc() - Allocates a new config request
+ *
+ * This is refcounted object so when you are done with this, call
+ * tb_cfg_request_put() to it.
+ */
+struct tb_cfg_request *tb_cfg_request_alloc(void)
+{
+ struct tb_cfg_request *req;
-/* common header */
-struct tb_cfg_header {
- u32 route_hi:22;
- u32 unknown:10; /* highest order bit is set on replies */
- u32 route_lo;
-} __packed;
-
-/* additional header for read/write packets */
-struct tb_cfg_address {
- u32 offset:13; /* in dwords */
- u32 length:6; /* in dwords */
- u32 port:6;
- enum tb_cfg_space space:2;
- u32 seq:2; /* sequence number */
- u32 zero:3;
-} __packed;
-
-/* TB_CFG_PKG_READ, response for TB_CFG_PKG_WRITE */
-struct cfg_read_pkg {
- struct tb_cfg_header header;
- struct tb_cfg_address addr;
-} __packed;
-
-/* TB_CFG_PKG_WRITE, response for TB_CFG_PKG_READ */
-struct cfg_write_pkg {
- struct tb_cfg_header header;
- struct tb_cfg_address addr;
- u32 data[64]; /* maximum size, tb_cfg_address.length has 6 bits */
-} __packed;
-
-/* TB_CFG_PKG_ERROR */
-struct cfg_error_pkg {
- struct tb_cfg_header header;
- enum tb_cfg_error error:4;
- u32 zero1:4;
- u32 port:6;
- u32 zero2:2; /* Both should be zero, still they are different fields. */
- u32 zero3:16;
-} __packed;
-
-/* TB_CFG_PKG_EVENT */
-struct cfg_event_pkg {
- struct tb_cfg_header header;
- u32 port:6;
- u32 zero:25;
- bool unplug:1;
-} __packed;
-
-/* TB_CFG_PKG_RESET */
-struct cfg_reset_pkg {
- struct tb_cfg_header header;
-} __packed;
-
-/* TB_CFG_PKG_PREPARE_TO_SLEEP */
-struct cfg_pts_pkg {
- struct tb_cfg_header header;
- u32 data;
-} __packed;
+ req = kzalloc(sizeof(*req), GFP_KERNEL);
+ if (!req)
+ return NULL;
+ kref_init(&req->kref);
-/* utility functions */
+ return req;
+}
-static u64 get_route(struct tb_cfg_header header)
+/**
+ * tb_cfg_request_get() - Increase refcount of a request
+ * @req: Request whose refcount is increased
+ */
+void tb_cfg_request_get(struct tb_cfg_request *req)
{
- return (u64) header.route_hi << 32 | header.route_lo;
+ mutex_lock(&tb_cfg_request_lock);
+ kref_get(&req->kref);
+ mutex_unlock(&tb_cfg_request_lock);
}
-static struct tb_cfg_header make_header(u64 route)
+static void tb_cfg_request_destroy(struct kref *kref)
{
- struct tb_cfg_header header = {
- .route_hi = route >> 32,
- .route_lo = route,
- };
- /* check for overflow, route_hi is not 32 bits! */
- WARN_ON(get_route(header) != route);
- return header;
+ struct tb_cfg_request *req = container_of(kref, typeof(*req), kref);
+
+ kfree(req);
+}
+
+/**
+ * tb_cfg_request_put() - Decrease refcount and possibly release the request
+ * @req: Request whose refcount is decreased
+ *
+ * Call this function when you are done with the request. When refcount
+ * goes to %0 the object is released.
+ */
+void tb_cfg_request_put(struct tb_cfg_request *req)
+{
+ mutex_lock(&tb_cfg_request_lock);
+ kref_put(&req->kref, tb_cfg_request_destroy);
+ mutex_unlock(&tb_cfg_request_lock);
}
-static int check_header(struct ctl_pkg *pkg, u32 len, enum tb_cfg_pkg_type type,
- u64 route)
+static int tb_cfg_request_enqueue(struct tb_ctl *ctl,
+ struct tb_cfg_request *req)
+{
+ WARN_ON(test_bit(TB_CFG_REQUEST_ACTIVE, &req->flags));
+ WARN_ON(req->ctl);
+
+ mutex_lock(&ctl->request_queue_lock);
+ if (!ctl->running) {
+ mutex_unlock(&ctl->request_queue_lock);
+ return -ENOTCONN;
+ }
+ req->ctl = ctl;
+ list_add_tail(&req->list, &ctl->request_queue);
+ set_bit(TB_CFG_REQUEST_ACTIVE, &req->flags);
+ mutex_unlock(&ctl->request_queue_lock);
+ return 0;
+}
+
+static void tb_cfg_request_dequeue(struct tb_cfg_request *req)
+{
+ struct tb_ctl *ctl = req->ctl;
+
+ mutex_lock(&ctl->request_queue_lock);
+ list_del(&req->list);
+ clear_bit(TB_CFG_REQUEST_ACTIVE, &req->flags);
+ if (test_bit(TB_CFG_REQUEST_CANCELED, &req->flags))
+ wake_up(&tb_cfg_request_cancel_queue);
+ mutex_unlock(&ctl->request_queue_lock);
+}
+
+static bool tb_cfg_request_is_active(struct tb_cfg_request *req)
+{
+ return test_bit(TB_CFG_REQUEST_ACTIVE, &req->flags);
+}
+
+static struct tb_cfg_request *
+tb_cfg_request_find(struct tb_ctl *ctl, struct ctl_pkg *pkg)
+{
+ struct tb_cfg_request *req;
+ bool found = false;
+
+ mutex_lock(&pkg->ctl->request_queue_lock);
+ list_for_each_entry(req, &pkg->ctl->request_queue, list) {
+ tb_cfg_request_get(req);
+ if (req->match(req, pkg)) {
+ found = true;
+ break;
+ }
+ tb_cfg_request_put(req);
+ }
+ mutex_unlock(&pkg->ctl->request_queue_lock);
+
+ return found ? req : NULL;
+}
+
+/* utility functions */
+
+
+static int check_header(const struct ctl_pkg *pkg, u32 len,
+ enum tb_cfg_pkg_type type, u64 route)
{
struct tb_cfg_header *header = pkg->buffer;
if (WARN(header->unknown != 1 << 9,
"header->unknown is %#x\n", header->unknown))
return -EIO;
- if (WARN(route != get_route(*header),
+ if (WARN(route != tb_cfg_get_route(header),
"wrong route (expected %llx, got %llx)",
- route, get_route(*header)))
+ route, tb_cfg_get_route(header)))
return -EIO;
return 0;
}
if (WARN(length != addr.length, "wrong space (expected %x, got %x\n)",
length, addr.length))
return -EIO;
- if (WARN(addr.seq, "addr.seq is %#x\n", addr.seq))
- return -EIO;
/*
* We cannot check addr->port as it is set to the upstream port of the
* sender.
return 0;
}
-static struct tb_cfg_result decode_error(struct ctl_pkg *response)
+static struct tb_cfg_result decode_error(const struct ctl_pkg *response)
{
struct cfg_error_pkg *pkg = response->buffer;
struct tb_cfg_result res = { 0 };
- res.response_route = get_route(pkg->header);
+ res.response_route = tb_cfg_get_route(&pkg->header);
res.response_port = 0;
res.err = check_header(response, sizeof(*pkg), TB_CFG_PKG_ERROR,
- get_route(pkg->header));
+ tb_cfg_get_route(&pkg->header));
if (res.err)
return res;
}
-static struct tb_cfg_result parse_header(struct ctl_pkg *pkg, u32 len,
+static struct tb_cfg_result parse_header(const struct ctl_pkg *pkg, u32 len,
enum tb_cfg_pkg_type type, u64 route)
{
struct tb_cfg_header *header = pkg->buffer;
return decode_error(pkg);
res.response_port = 0; /* will be updated later for cfg_read/write */
- res.response_route = get_route(*header);
+ res.response_route = tb_cfg_get_route(header);
res.err = check_header(pkg, len, type, route);
return res;
}
}
}
-static void cpu_to_be32_array(__be32 *dst, u32 *src, size_t len)
+static void cpu_to_be32_array(__be32 *dst, const u32 *src, size_t len)
{
int i;
for (i = 0; i < len; i++)
dst[i] = be32_to_cpu(src[i]);
}
-static __be32 tb_crc(void *data, size_t len)
+static __be32 tb_crc(const void *data, size_t len)
{
return cpu_to_be32(~__crc32c_le(~0, data, len));
}
*
* Return: Returns 0 on success or an error code on failure.
*/
-static int tb_ctl_tx(struct tb_ctl *ctl, void *data, size_t len,
+static int tb_ctl_tx(struct tb_ctl *ctl, const void *data, size_t len,
enum tb_cfg_pkg_type type)
{
int res;
}
/**
- * tb_ctl_handle_plug_event() - acknowledge a plug event, invoke ctl->callback
+ * tb_ctl_handle_event() - acknowledge a plug event, invoke ctl->callback
*/
-static void tb_ctl_handle_plug_event(struct tb_ctl *ctl,
- struct ctl_pkg *response)
+static void tb_ctl_handle_event(struct tb_ctl *ctl, enum tb_cfg_pkg_type type,
+ struct ctl_pkg *pkg, size_t size)
{
- struct cfg_event_pkg *pkg = response->buffer;
- u64 route = get_route(pkg->header);
-
- if (check_header(response, sizeof(*pkg), TB_CFG_PKG_EVENT, route)) {
- tb_ctl_warn(ctl, "malformed TB_CFG_PKG_EVENT\n");
- return;
- }
-
- if (tb_cfg_error(ctl, route, pkg->port, TB_CFG_ERROR_ACK_PLUG_EVENT))
- tb_ctl_warn(ctl, "could not ack plug event on %llx:%x\n",
- route, pkg->port);
- WARN(pkg->zero, "pkg->zero is %#x\n", pkg->zero);
- ctl->callback(ctl->callback_data, route, pkg->port, pkg->unplug);
+ ctl->callback(ctl->callback_data, type, pkg->buffer, size);
}
static void tb_ctl_rx_submit(struct ctl_pkg *pkg)
*/
}
+static int tb_async_error(const struct ctl_pkg *pkg)
+{
+ const struct cfg_error_pkg *error = (const struct cfg_error_pkg *)pkg;
+
+ if (pkg->frame.eof != TB_CFG_PKG_ERROR)
+ return false;
+
+ switch (error->error) {
+ case TB_CFG_ERROR_LINK_ERROR:
+ case TB_CFG_ERROR_HEC_ERROR_DETECTED:
+ case TB_CFG_ERROR_FLOW_CONTROL_ERROR:
+ return true;
+
+ default:
+ return false;
+ }
+}
+
static void tb_ctl_rx_callback(struct tb_ring *ring, struct ring_frame *frame,
bool canceled)
{
struct ctl_pkg *pkg = container_of(frame, typeof(*pkg), frame);
+ struct tb_cfg_request *req;
+ __be32 crc32;
if (canceled)
return; /*
}
frame->size -= 4; /* remove checksum */
- if (*(__be32 *) (pkg->buffer + frame->size)
- != tb_crc(pkg->buffer, frame->size)) {
- tb_ctl_err(pkg->ctl,
- "RX: checksum mismatch, dropping packet\n");
- goto rx;
- }
+ crc32 = tb_crc(pkg->buffer, frame->size);
be32_to_cpu_array(pkg->buffer, pkg->buffer, frame->size / 4);
- if (frame->eof == TB_CFG_PKG_EVENT) {
- tb_ctl_handle_plug_event(pkg->ctl, pkg);
+ switch (frame->eof) {
+ case TB_CFG_PKG_READ:
+ case TB_CFG_PKG_WRITE:
+ case TB_CFG_PKG_ERROR:
+ case TB_CFG_PKG_OVERRIDE:
+ case TB_CFG_PKG_RESET:
+ if (*(__be32 *)(pkg->buffer + frame->size) != crc32) {
+ tb_ctl_err(pkg->ctl,
+ "RX: checksum mismatch, dropping packet\n");
+ goto rx;
+ }
+ if (tb_async_error(pkg)) {
+ tb_ctl_handle_event(pkg->ctl, frame->eof,
+ pkg, frame->size);
+ goto rx;
+ }
+ break;
+
+ case TB_CFG_PKG_EVENT:
+ if (*(__be32 *)(pkg->buffer + frame->size) != crc32) {
+ tb_ctl_err(pkg->ctl,
+ "RX: checksum mismatch, dropping packet\n");
+ goto rx;
+ }
+ /* Fall through */
+ case TB_CFG_PKG_ICM_EVENT:
+ tb_ctl_handle_event(pkg->ctl, frame->eof, pkg, frame->size);
goto rx;
+
+ default:
+ break;
}
- if (!kfifo_put(&pkg->ctl->response_fifo, pkg)) {
- tb_ctl_err(pkg->ctl, "RX: fifo is full\n");
- goto rx;
+
+ /*
+ * The received packet will be processed only if there is an
+ * active request and that the packet is what is expected. This
+ * prevents packets such as replies coming after timeout has
+ * triggered from messing with the active requests.
+ */
+ req = tb_cfg_request_find(pkg->ctl, pkg);
+ if (req) {
+ if (req->copy(req, pkg))
+ schedule_work(&req->work);
+ tb_cfg_request_put(req);
}
- complete(&pkg->ctl->response_ready);
- return;
+
rx:
tb_ctl_rx_submit(pkg);
}
+static void tb_cfg_request_work(struct work_struct *work)
+{
+ struct tb_cfg_request *req = container_of(work, typeof(*req), work);
+
+ if (!test_bit(TB_CFG_REQUEST_CANCELED, &req->flags))
+ req->callback(req->callback_data);
+
+ tb_cfg_request_dequeue(req);
+ tb_cfg_request_put(req);
+}
+
/**
- * tb_ctl_rx() - receive a packet from the control channel
+ * tb_cfg_request() - Start control request not waiting for it to complete
+ * @ctl: Control channel to use
+ * @req: Request to start
+ * @callback: Callback called when the request is completed
+ * @callback_data: Data to be passed to @callback
+ *
+ * This queues @req on the given control channel without waiting for it
+ * to complete. When the request completes @callback is called.
*/
-static struct tb_cfg_result tb_ctl_rx(struct tb_ctl *ctl, void *buffer,
- size_t length, int timeout_msec,
- u64 route, enum tb_cfg_pkg_type type)
+int tb_cfg_request(struct tb_ctl *ctl, struct tb_cfg_request *req,
+ void (*callback)(void *), void *callback_data)
{
- struct tb_cfg_result res;
- struct ctl_pkg *pkg;
+ int ret;
- if (!wait_for_completion_timeout(&ctl->response_ready,
- msecs_to_jiffies(timeout_msec))) {
- tb_ctl_WARN(ctl, "RX: timeout\n");
- return (struct tb_cfg_result) { .err = -ETIMEDOUT };
- }
- if (!kfifo_get(&ctl->response_fifo, &pkg)) {
- tb_ctl_WARN(ctl, "empty kfifo\n");
- return (struct tb_cfg_result) { .err = -EIO };
- }
+ req->flags = 0;
+ req->callback = callback;
+ req->callback_data = callback_data;
+ INIT_WORK(&req->work, tb_cfg_request_work);
+ INIT_LIST_HEAD(&req->list);
- res = parse_header(pkg, length, type, route);
- if (!res.err)
- memcpy(buffer, pkg->buffer, length);
- tb_ctl_rx_submit(pkg);
- return res;
+ tb_cfg_request_get(req);
+ ret = tb_cfg_request_enqueue(ctl, req);
+ if (ret)
+ goto err_put;
+
+ ret = tb_ctl_tx(ctl, req->request, req->request_size,
+ req->request_type);
+ if (ret)
+ goto err_dequeue;
+
+ if (!req->response)
+ schedule_work(&req->work);
+
+ return 0;
+
+err_dequeue:
+ tb_cfg_request_dequeue(req);
+err_put:
+ tb_cfg_request_put(req);
+
+ return ret;
+}
+
+/**
+ * tb_cfg_request_cancel() - Cancel a control request
+ * @req: Request to cancel
+ * @err: Error to assign to the request
+ *
+ * This function can be used to cancel ongoing request. It will wait
+ * until the request is not active anymore.
+ */
+void tb_cfg_request_cancel(struct tb_cfg_request *req, int err)
+{
+ set_bit(TB_CFG_REQUEST_CANCELED, &req->flags);
+ schedule_work(&req->work);
+ wait_event(tb_cfg_request_cancel_queue, !tb_cfg_request_is_active(req));
+ req->result.err = err;
+}
+
+static void tb_cfg_request_complete(void *data)
+{
+ complete(data);
}
+/**
+ * tb_cfg_request_sync() - Start control request and wait until it completes
+ * @ctl: Control channel to use
+ * @req: Request to start
+ * @timeout_msec: Timeout how long to wait @req to complete
+ *
+ * Starts a control request and waits until it completes. If timeout
+ * triggers the request is canceled before function returns. Note the
+ * caller needs to make sure only one message for given switch is active
+ * at a time.
+ */
+struct tb_cfg_result tb_cfg_request_sync(struct tb_ctl *ctl,
+ struct tb_cfg_request *req,
+ int timeout_msec)
+{
+ unsigned long timeout = msecs_to_jiffies(timeout_msec);
+ struct tb_cfg_result res = { 0 };
+ DECLARE_COMPLETION_ONSTACK(done);
+ int ret;
+
+ ret = tb_cfg_request(ctl, req, tb_cfg_request_complete, &done);
+ if (ret) {
+ res.err = ret;
+ return res;
+ }
+
+ if (!wait_for_completion_timeout(&done, timeout))
+ tb_cfg_request_cancel(req, -ETIMEDOUT);
+
+ flush_work(&req->work);
+
+ return req->result;
+}
/* public interface, alloc/start/stop/free */
*
* Return: Returns a pointer on success or NULL on failure.
*/
-struct tb_ctl *tb_ctl_alloc(struct tb_nhi *nhi, hotplug_cb cb, void *cb_data)
+struct tb_ctl *tb_ctl_alloc(struct tb_nhi *nhi, event_cb cb, void *cb_data)
{
int i;
struct tb_ctl *ctl = kzalloc(sizeof(*ctl), GFP_KERNEL);
ctl->callback = cb;
ctl->callback_data = cb_data;
- init_completion(&ctl->response_ready);
- INIT_KFIFO(ctl->response_fifo);
+ mutex_init(&ctl->request_queue_lock);
+ INIT_LIST_HEAD(&ctl->request_queue);
ctl->frame_pool = dma_pool_create("thunderbolt_ctl", &nhi->pdev->dev,
TB_FRAME_SIZE, 4, 0);
if (!ctl->frame_pool)
goto err;
- ctl->tx = ring_alloc_tx(nhi, 0, 10);
+ ctl->tx = ring_alloc_tx(nhi, 0, 10, RING_FLAG_NO_SUSPEND);
if (!ctl->tx)
goto err;
- ctl->rx = ring_alloc_rx(nhi, 0, 10);
+ ctl->rx = ring_alloc_rx(nhi, 0, 10, RING_FLAG_NO_SUSPEND);
if (!ctl->rx)
goto err;
void tb_ctl_free(struct tb_ctl *ctl)
{
int i;
+
+ if (!ctl)
+ return;
+
if (ctl->rx)
ring_free(ctl->rx);
if (ctl->tx)
ring_start(ctl->rx);
for (i = 0; i < TB_CTL_RX_PKG_COUNT; i++)
tb_ctl_rx_submit(ctl->rx_packets[i]);
+
+ ctl->running = true;
}
/**
*/
void tb_ctl_stop(struct tb_ctl *ctl)
{
+ mutex_lock(&ctl->request_queue_lock);
+ ctl->running = false;
+ mutex_unlock(&ctl->request_queue_lock);
+
ring_stop(ctl->rx);
ring_stop(ctl->tx);
- if (!kfifo_is_empty(&ctl->response_fifo))
- tb_ctl_WARN(ctl, "dangling response in response_fifo\n");
- kfifo_reset(&ctl->response_fifo);
+ if (!list_empty(&ctl->request_queue))
+ tb_ctl_WARN(ctl, "dangling request in request_queue\n");
+ INIT_LIST_HEAD(&ctl->request_queue);
tb_ctl_info(ctl, "control channel stopped\n");
}
enum tb_cfg_error error)
{
struct cfg_error_pkg pkg = {
- .header = make_header(route),
+ .header = tb_cfg_make_header(route),
.port = port,
.error = error,
};
return tb_ctl_tx(ctl, &pkg, sizeof(pkg), TB_CFG_PKG_ERROR);
}
+static bool tb_cfg_match(const struct tb_cfg_request *req,
+ const struct ctl_pkg *pkg)
+{
+ u64 route = tb_cfg_get_route(pkg->buffer) & ~BIT_ULL(63);
+
+ if (pkg->frame.eof == TB_CFG_PKG_ERROR)
+ return true;
+
+ if (pkg->frame.eof != req->response_type)
+ return false;
+ if (route != tb_cfg_get_route(req->request))
+ return false;
+ if (pkg->frame.size != req->response_size)
+ return false;
+
+ if (pkg->frame.eof == TB_CFG_PKG_READ ||
+ pkg->frame.eof == TB_CFG_PKG_WRITE) {
+ const struct cfg_read_pkg *req_hdr = req->request;
+ const struct cfg_read_pkg *res_hdr = pkg->buffer;
+
+ if (req_hdr->addr.seq != res_hdr->addr.seq)
+ return false;
+ }
+
+ return true;
+}
+
+static bool tb_cfg_copy(struct tb_cfg_request *req, const struct ctl_pkg *pkg)
+{
+ struct tb_cfg_result res;
+
+ /* Now make sure it is in expected format */
+ res = parse_header(pkg, req->response_size, req->response_type,
+ tb_cfg_get_route(req->request));
+ if (!res.err)
+ memcpy(req->response, pkg->buffer, req->response_size);
+
+ req->result = res;
+
+ /* Always complete when first response is received */
+ return true;
+}
+
/**
* tb_cfg_reset() - send a reset packet and wait for a response
*
struct tb_cfg_result tb_cfg_reset(struct tb_ctl *ctl, u64 route,
int timeout_msec)
{
- int err;
- struct cfg_reset_pkg request = { .header = make_header(route) };
+ struct cfg_reset_pkg request = { .header = tb_cfg_make_header(route) };
+ struct tb_cfg_result res = { 0 };
struct tb_cfg_header reply;
+ struct tb_cfg_request *req;
+
+ req = tb_cfg_request_alloc();
+ if (!req) {
+ res.err = -ENOMEM;
+ return res;
+ }
+
+ req->match = tb_cfg_match;
+ req->copy = tb_cfg_copy;
+ req->request = &request;
+ req->request_size = sizeof(request);
+ req->request_type = TB_CFG_PKG_RESET;
+ req->response = &reply;
+ req->response_size = sizeof(reply);
+ req->response_type = sizeof(TB_CFG_PKG_RESET);
+
+ res = tb_cfg_request_sync(ctl, req, timeout_msec);
- err = tb_ctl_tx(ctl, &request, sizeof(request), TB_CFG_PKG_RESET);
- if (err)
- return (struct tb_cfg_result) { .err = err };
+ tb_cfg_request_put(req);
- return tb_ctl_rx(ctl, &reply, sizeof(reply), timeout_msec, route,
- TB_CFG_PKG_RESET);
+ return res;
}
/**
{
struct tb_cfg_result res = { 0 };
struct cfg_read_pkg request = {
- .header = make_header(route),
+ .header = tb_cfg_make_header(route),
.addr = {
.port = port,
.space = space,
},
};
struct cfg_write_pkg reply;
+ int retries = 0;
- res.err = tb_ctl_tx(ctl, &request, sizeof(request), TB_CFG_PKG_READ);
- if (res.err)
- return res;
+ while (retries < TB_CTL_RETRIES) {
+ struct tb_cfg_request *req;
+
+ req = tb_cfg_request_alloc();
+ if (!req) {
+ res.err = -ENOMEM;
+ return res;
+ }
+
+ request.addr.seq = retries++;
+
+ req->match = tb_cfg_match;
+ req->copy = tb_cfg_copy;
+ req->request = &request;
+ req->request_size = sizeof(request);
+ req->request_type = TB_CFG_PKG_READ;
+ req->response = &reply;
+ req->response_size = 12 + 4 * length;
+ req->response_type = TB_CFG_PKG_READ;
+
+ res = tb_cfg_request_sync(ctl, req, timeout_msec);
+
+ tb_cfg_request_put(req);
+
+ if (res.err != -ETIMEDOUT)
+ break;
+
+ /* Wait a bit (arbitrary time) until we send a retry */
+ usleep_range(10, 100);
+ }
- res = tb_ctl_rx(ctl, &reply, 12 + 4 * length, timeout_msec, route,
- TB_CFG_PKG_READ);
if (res.err)
return res;
*
* Offset and length are in dwords.
*/
-struct tb_cfg_result tb_cfg_write_raw(struct tb_ctl *ctl, void *buffer,
+struct tb_cfg_result tb_cfg_write_raw(struct tb_ctl *ctl, const void *buffer,
u64 route, u32 port, enum tb_cfg_space space,
u32 offset, u32 length, int timeout_msec)
{
struct tb_cfg_result res = { 0 };
struct cfg_write_pkg request = {
- .header = make_header(route),
+ .header = tb_cfg_make_header(route),
.addr = {
.port = port,
.space = space,
},
};
struct cfg_read_pkg reply;
+ int retries = 0;
memcpy(&request.data, buffer, length * 4);
- res.err = tb_ctl_tx(ctl, &request, 12 + 4 * length, TB_CFG_PKG_WRITE);
- if (res.err)
- return res;
+ while (retries < TB_CTL_RETRIES) {
+ struct tb_cfg_request *req;
+
+ req = tb_cfg_request_alloc();
+ if (!req) {
+ res.err = -ENOMEM;
+ return res;
+ }
+
+ request.addr.seq = retries++;
+
+ req->match = tb_cfg_match;
+ req->copy = tb_cfg_copy;
+ req->request = &request;
+ req->request_size = 12 + 4 * length;
+ req->request_type = TB_CFG_PKG_WRITE;
+ req->response = &reply;
+ req->response_size = sizeof(reply);
+ req->response_type = TB_CFG_PKG_WRITE;
+
+ res = tb_cfg_request_sync(ctl, req, timeout_msec);
+
+ tb_cfg_request_put(req);
+
+ if (res.err != -ETIMEDOUT)
+ break;
+
+ /* Wait a bit (arbitrary time) until we send a retry */
+ usleep_range(10, 100);
+ }
- res = tb_ctl_rx(ctl, &reply, sizeof(reply), timeout_msec, route,
- TB_CFG_PKG_WRITE);
if (res.err)
return res;
{
struct tb_cfg_result res = tb_cfg_read_raw(ctl, buffer, route, port,
space, offset, length, TB_CFG_DEFAULT_TIMEOUT);
- if (res.err == 1) {
+ switch (res.err) {
+ case 0:
+ /* Success */
+ break;
+
+ case 1:
+ /* Thunderbolt error, tb_error holds the actual number */
tb_cfg_print_error(ctl, &res);
return -EIO;
+
+ case -ETIMEDOUT:
+ tb_ctl_warn(ctl, "timeout reading config space %u from %#x\n",
+ space, offset);
+ break;
+
+ default:
+ WARN(1, "tb_cfg_read: %d\n", res.err);
+ break;
}
- WARN(res.err, "tb_cfg_read: %d\n", res.err);
return res.err;
}
-int tb_cfg_write(struct tb_ctl *ctl, void *buffer, u64 route, u32 port,
+int tb_cfg_write(struct tb_ctl *ctl, const void *buffer, u64 route, u32 port,
enum tb_cfg_space space, u32 offset, u32 length)
{
struct tb_cfg_result res = tb_cfg_write_raw(ctl, buffer, route, port,
space, offset, length, TB_CFG_DEFAULT_TIMEOUT);
- if (res.err == 1) {
+ switch (res.err) {
+ case 0:
+ /* Success */
+ break;
+
+ case 1:
+ /* Thunderbolt error, tb_error holds the actual number */
tb_cfg_print_error(ctl, &res);
return -EIO;
+
+ case -ETIMEDOUT:
+ tb_ctl_warn(ctl, "timeout writing config space %u to %#x\n",
+ space, offset);
+ break;
+
+ default:
+ WARN(1, "tb_cfg_write: %d\n", res.err);
+ break;
}
- WARN(res.err, "tb_cfg_write: %d\n", res.err);
return res.err;
}
#ifndef _TB_CFG
#define _TB_CFG
+#include <linux/kref.h>
+
#include "nhi.h"
+#include "tb_msgs.h"
/* control channel */
struct tb_ctl;
-typedef void (*hotplug_cb)(void *data, u64 route, u8 port, bool unplug);
+typedef void (*event_cb)(void *data, enum tb_cfg_pkg_type type,
+ const void *buf, size_t size);
-struct tb_ctl *tb_ctl_alloc(struct tb_nhi *nhi, hotplug_cb cb, void *cb_data);
+struct tb_ctl *tb_ctl_alloc(struct tb_nhi *nhi, event_cb cb, void *cb_data);
void tb_ctl_start(struct tb_ctl *ctl);
void tb_ctl_stop(struct tb_ctl *ctl);
void tb_ctl_free(struct tb_ctl *ctl);
#define TB_CFG_DEFAULT_TIMEOUT 5000 /* msec */
-enum tb_cfg_space {
- TB_CFG_HOPS = 0,
- TB_CFG_PORT = 1,
- TB_CFG_SWITCH = 2,
- TB_CFG_COUNTERS = 3,
-};
-
-enum tb_cfg_error {
- TB_CFG_ERROR_PORT_NOT_CONNECTED = 0,
- TB_CFG_ERROR_INVALID_CONFIG_SPACE = 2,
- TB_CFG_ERROR_NO_SUCH_PORT = 4,
- TB_CFG_ERROR_ACK_PLUG_EVENT = 7, /* send as reply to TB_CFG_PKG_EVENT */
- TB_CFG_ERROR_LOOP = 8,
-};
-
struct tb_cfg_result {
u64 response_route;
u32 response_port; /*
enum tb_cfg_error tb_error; /* valid if err == 1 */
};
+struct ctl_pkg {
+ struct tb_ctl *ctl;
+ void *buffer;
+ struct ring_frame frame;
+};
+
+/**
+ * struct tb_cfg_request - Control channel request
+ * @kref: Reference count
+ * @ctl: Pointer to the control channel structure. Only set when the
+ * request is queued.
+ * @request_size: Size of the request packet (in bytes)
+ * @request_type: Type of the request packet
+ * @response: Response is stored here
+ * @response_size: Maximum size of one response packet
+ * @response_type: Expected type of the response packet
+ * @npackets: Number of packets expected to be returned with this request
+ * @match: Function used to match the incoming packet
+ * @copy: Function used to copy the incoming packet to @response
+ * @callback: Callback called when the request is finished successfully
+ * @callback_data: Data to be passed to @callback
+ * @flags: Flags for the request
+ * @work: Work item used to complete the request
+ * @result: Result after the request has been completed
+ * @list: Requests are queued using this field
+ *
+ * An arbitrary request over Thunderbolt control channel. For standard
+ * control channel message, one should use tb_cfg_read/write() and
+ * friends if possible.
+ */
+struct tb_cfg_request {
+ struct kref kref;
+ struct tb_ctl *ctl;
+ const void *request;
+ size_t request_size;
+ enum tb_cfg_pkg_type request_type;
+ void *response;
+ size_t response_size;
+ enum tb_cfg_pkg_type response_type;
+ size_t npackets;
+ bool (*match)(const struct tb_cfg_request *req,
+ const struct ctl_pkg *pkg);
+ bool (*copy)(struct tb_cfg_request *req, const struct ctl_pkg *pkg);
+ void (*callback)(void *callback_data);
+ void *callback_data;
+ unsigned long flags;
+ struct work_struct work;
+ struct tb_cfg_result result;
+ struct list_head list;
+};
+
+#define TB_CFG_REQUEST_ACTIVE 0
+#define TB_CFG_REQUEST_CANCELED 1
+
+struct tb_cfg_request *tb_cfg_request_alloc(void);
+void tb_cfg_request_get(struct tb_cfg_request *req);
+void tb_cfg_request_put(struct tb_cfg_request *req);
+int tb_cfg_request(struct tb_ctl *ctl, struct tb_cfg_request *req,
+ void (*callback)(void *), void *callback_data);
+void tb_cfg_request_cancel(struct tb_cfg_request *req, int err);
+struct tb_cfg_result tb_cfg_request_sync(struct tb_ctl *ctl,
+ struct tb_cfg_request *req, int timeout_msec);
+
+static inline u64 tb_cfg_get_route(const struct tb_cfg_header *header)
+{
+ return (u64) header->route_hi << 32 | header->route_lo;
+}
+
+static inline struct tb_cfg_header tb_cfg_make_header(u64 route)
+{
+ struct tb_cfg_header header = {
+ .route_hi = route >> 32,
+ .route_lo = route,
+ };
+ /* check for overflow, route_hi is not 32 bits! */
+ WARN_ON(tb_cfg_get_route(&header) != route);
+ return header;
+}
int tb_cfg_error(struct tb_ctl *ctl, u64 route, u32 port,
enum tb_cfg_error error);
u64 route, u32 port,
enum tb_cfg_space space, u32 offset,
u32 length, int timeout_msec);
-struct tb_cfg_result tb_cfg_write_raw(struct tb_ctl *ctl, void *buffer,
+struct tb_cfg_result tb_cfg_write_raw(struct tb_ctl *ctl, const void *buffer,
u64 route, u32 port,
enum tb_cfg_space space, u32 offset,
u32 length, int timeout_msec);
int tb_cfg_read(struct tb_ctl *ctl, void *buffer, u64 route, u32 port,
enum tb_cfg_space space, u32 offset, u32 length);
-int tb_cfg_write(struct tb_ctl *ctl, void *buffer, u64 route, u32 port,
+int tb_cfg_write(struct tb_ctl *ctl, const void *buffer, u64 route, u32 port,
enum tb_cfg_space space, u32 offset, u32 length);
int tb_cfg_get_upstream_port(struct tb_ctl *ctl, u64 route);
--- /dev/null
+/*
+ * Thunderbolt DMA configuration based mailbox support
+ *
+ * Copyright (C) 2017, Intel Corporation
+ * Authors: Michael Jamet <michael.jamet@intel.com>
+ * Mika Westerberg <mika.westerberg@linux.intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/delay.h>
+#include <linux/slab.h>
+
+#include "dma_port.h"
+#include "tb_regs.h"
+
+#define DMA_PORT_CAP 0x3e
+
+#define MAIL_DATA 1
+#define MAIL_DATA_DWORDS 16
+
+#define MAIL_IN 17
+#define MAIL_IN_CMD_SHIFT 28
+#define MAIL_IN_CMD_MASK GENMASK(31, 28)
+#define MAIL_IN_CMD_FLASH_WRITE 0x0
+#define MAIL_IN_CMD_FLASH_UPDATE_AUTH 0x1
+#define MAIL_IN_CMD_FLASH_READ 0x2
+#define MAIL_IN_CMD_POWER_CYCLE 0x4
+#define MAIL_IN_DWORDS_SHIFT 24
+#define MAIL_IN_DWORDS_MASK GENMASK(27, 24)
+#define MAIL_IN_ADDRESS_SHIFT 2
+#define MAIL_IN_ADDRESS_MASK GENMASK(23, 2)
+#define MAIL_IN_CSS BIT(1)
+#define MAIL_IN_OP_REQUEST BIT(0)
+
+#define MAIL_OUT 18
+#define MAIL_OUT_STATUS_RESPONSE BIT(29)
+#define MAIL_OUT_STATUS_CMD_SHIFT 4
+#define MAIL_OUT_STATUS_CMD_MASK GENMASK(7, 4)
+#define MAIL_OUT_STATUS_MASK GENMASK(3, 0)
+#define MAIL_OUT_STATUS_COMPLETED 0
+#define MAIL_OUT_STATUS_ERR_AUTH 1
+#define MAIL_OUT_STATUS_ERR_ACCESS 2
+
+#define DMA_PORT_TIMEOUT 5000 /* ms */
+#define DMA_PORT_RETRIES 3
+
+/**
+ * struct tb_dma_port - DMA control port
+ * @sw: Switch the DMA port belongs to
+ * @port: Switch port number where DMA capability is found
+ * @base: Start offset of the mailbox registers
+ * @buf: Temporary buffer to store a single block
+ */
+struct tb_dma_port {
+ struct tb_switch *sw;
+ u8 port;
+ u32 base;
+ u8 *buf;
+};
+
+/*
+ * When the switch is in safe mode it supports very little functionality
+ * so we don't validate that much here.
+ */
+static bool dma_port_match(const struct tb_cfg_request *req,
+ const struct ctl_pkg *pkg)
+{
+ u64 route = tb_cfg_get_route(pkg->buffer) & ~BIT_ULL(63);
+
+ if (pkg->frame.eof == TB_CFG_PKG_ERROR)
+ return true;
+ if (pkg->frame.eof != req->response_type)
+ return false;
+ if (route != tb_cfg_get_route(req->request))
+ return false;
+ if (pkg->frame.size != req->response_size)
+ return false;
+
+ return true;
+}
+
+static bool dma_port_copy(struct tb_cfg_request *req, const struct ctl_pkg *pkg)
+{
+ memcpy(req->response, pkg->buffer, req->response_size);
+ return true;
+}
+
+static int dma_port_read(struct tb_ctl *ctl, void *buffer, u64 route,
+ u32 port, u32 offset, u32 length, int timeout_msec)
+{
+ struct cfg_read_pkg request = {
+ .header = tb_cfg_make_header(route),
+ .addr = {
+ .seq = 1,
+ .port = port,
+ .space = TB_CFG_PORT,
+ .offset = offset,
+ .length = length,
+ },
+ };
+ struct tb_cfg_request *req;
+ struct cfg_write_pkg reply;
+ struct tb_cfg_result res;
+
+ req = tb_cfg_request_alloc();
+ if (!req)
+ return -ENOMEM;
+
+ req->match = dma_port_match;
+ req->copy = dma_port_copy;
+ req->request = &request;
+ req->request_size = sizeof(request);
+ req->request_type = TB_CFG_PKG_READ;
+ req->response = &reply;
+ req->response_size = 12 + 4 * length;
+ req->response_type = TB_CFG_PKG_READ;
+
+ res = tb_cfg_request_sync(ctl, req, timeout_msec);
+
+ tb_cfg_request_put(req);
+
+ if (res.err)
+ return res.err;
+
+ memcpy(buffer, &reply.data, 4 * length);
+ return 0;
+}
+
+static int dma_port_write(struct tb_ctl *ctl, const void *buffer, u64 route,
+ u32 port, u32 offset, u32 length, int timeout_msec)
+{
+ struct cfg_write_pkg request = {
+ .header = tb_cfg_make_header(route),
+ .addr = {
+ .seq = 1,
+ .port = port,
+ .space = TB_CFG_PORT,
+ .offset = offset,
+ .length = length,
+ },
+ };
+ struct tb_cfg_request *req;
+ struct cfg_read_pkg reply;
+ struct tb_cfg_result res;
+
+ memcpy(&request.data, buffer, length * 4);
+
+ req = tb_cfg_request_alloc();
+ if (!req)
+ return -ENOMEM;
+
+ req->match = dma_port_match;
+ req->copy = dma_port_copy;
+ req->request = &request;
+ req->request_size = 12 + 4 * length;
+ req->request_type = TB_CFG_PKG_WRITE;
+ req->response = &reply;
+ req->response_size = sizeof(reply);
+ req->response_type = TB_CFG_PKG_WRITE;
+
+ res = tb_cfg_request_sync(ctl, req, timeout_msec);
+
+ tb_cfg_request_put(req);
+
+ return res.err;
+}
+
+static int dma_find_port(struct tb_switch *sw)
+{
+ int port, ret;
+ u32 type;
+
+ /*
+ * The DMA (NHI) port is either 3 or 5 depending on the
+ * controller. Try both starting from 5 which is more common.
+ */
+ port = 5;
+ ret = dma_port_read(sw->tb->ctl, &type, tb_route(sw), port, 2, 1,
+ DMA_PORT_TIMEOUT);
+ if (!ret && (type & 0xffffff) == TB_TYPE_NHI)
+ return port;
+
+ port = 3;
+ ret = dma_port_read(sw->tb->ctl, &type, tb_route(sw), port, 2, 1,
+ DMA_PORT_TIMEOUT);
+ if (!ret && (type & 0xffffff) == TB_TYPE_NHI)
+ return port;
+
+ return -ENODEV;
+}
+
+/**
+ * dma_port_alloc() - Finds DMA control port from a switch pointed by route
+ * @sw: Switch from where find the DMA port
+ *
+ * Function checks if the switch NHI port supports DMA configuration
+ * based mailbox capability and if it does, allocates and initializes
+ * DMA port structure. Returns %NULL if the capabity was not found.
+ *
+ * The DMA control port is functional also when the switch is in safe
+ * mode.
+ */
+struct tb_dma_port *dma_port_alloc(struct tb_switch *sw)
+{
+ struct tb_dma_port *dma;
+ int port;
+
+ port = dma_find_port(sw);
+ if (port < 0)
+ return NULL;
+
+ dma = kzalloc(sizeof(*dma), GFP_KERNEL);
+ if (!dma)
+ return NULL;
+
+ dma->buf = kmalloc_array(MAIL_DATA_DWORDS, sizeof(u32), GFP_KERNEL);
+ if (!dma->buf) {
+ kfree(dma);
+ return NULL;
+ }
+
+ dma->sw = sw;
+ dma->port = port;
+ dma->base = DMA_PORT_CAP;
+
+ return dma;
+}
+
+/**
+ * dma_port_free() - Release DMA control port structure
+ * @dma: DMA control port
+ */
+void dma_port_free(struct tb_dma_port *dma)
+{
+ if (dma) {
+ kfree(dma->buf);
+ kfree(dma);
+ }
+}
+
+static int dma_port_wait_for_completion(struct tb_dma_port *dma,
+ unsigned int timeout)
+{
+ unsigned long end = jiffies + msecs_to_jiffies(timeout);
+ struct tb_switch *sw = dma->sw;
+
+ do {
+ int ret;
+ u32 in;
+
+ ret = dma_port_read(sw->tb->ctl, &in, tb_route(sw), dma->port,
+ dma->base + MAIL_IN, 1, 50);
+ if (ret) {
+ if (ret != -ETIMEDOUT)
+ return ret;
+ } else if (!(in & MAIL_IN_OP_REQUEST)) {
+ return 0;
+ }
+
+ usleep_range(50, 100);
+ } while (time_before(jiffies, end));
+
+ return -ETIMEDOUT;
+}
+
+static int status_to_errno(u32 status)
+{
+ switch (status & MAIL_OUT_STATUS_MASK) {
+ case MAIL_OUT_STATUS_COMPLETED:
+ return 0;
+ case MAIL_OUT_STATUS_ERR_AUTH:
+ return -EINVAL;
+ case MAIL_OUT_STATUS_ERR_ACCESS:
+ return -EACCES;
+ }
+
+ return -EIO;
+}
+
+static int dma_port_request(struct tb_dma_port *dma, u32 in,
+ unsigned int timeout)
+{
+ struct tb_switch *sw = dma->sw;
+ u32 out;
+ int ret;
+
+ ret = dma_port_write(sw->tb->ctl, &in, tb_route(sw), dma->port,
+ dma->base + MAIL_IN, 1, DMA_PORT_TIMEOUT);
+ if (ret)
+ return ret;
+
+ ret = dma_port_wait_for_completion(dma, timeout);
+ if (ret)
+ return ret;
+
+ ret = dma_port_read(sw->tb->ctl, &out, tb_route(sw), dma->port,
+ dma->base + MAIL_OUT, 1, DMA_PORT_TIMEOUT);
+ if (ret)
+ return ret;
+
+ return status_to_errno(out);
+}
+
+static int dma_port_flash_read_block(struct tb_dma_port *dma, u32 address,
+ void *buf, u32 size)
+{
+ struct tb_switch *sw = dma->sw;
+ u32 in, dwaddress, dwords;
+ int ret;
+
+ dwaddress = address / 4;
+ dwords = size / 4;
+
+ in = MAIL_IN_CMD_FLASH_READ << MAIL_IN_CMD_SHIFT;
+ if (dwords < MAIL_DATA_DWORDS)
+ in |= (dwords << MAIL_IN_DWORDS_SHIFT) & MAIL_IN_DWORDS_MASK;
+ in |= (dwaddress << MAIL_IN_ADDRESS_SHIFT) & MAIL_IN_ADDRESS_MASK;
+ in |= MAIL_IN_OP_REQUEST;
+
+ ret = dma_port_request(dma, in, DMA_PORT_TIMEOUT);
+ if (ret)
+ return ret;
+
+ return dma_port_read(sw->tb->ctl, buf, tb_route(sw), dma->port,
+ dma->base + MAIL_DATA, dwords, DMA_PORT_TIMEOUT);
+}
+
+static int dma_port_flash_write_block(struct tb_dma_port *dma, u32 address,
+ const void *buf, u32 size)
+{
+ struct tb_switch *sw = dma->sw;
+ u32 in, dwaddress, dwords;
+ int ret;
+
+ dwords = size / 4;
+
+ /* Write the block to MAIL_DATA registers */
+ ret = dma_port_write(sw->tb->ctl, buf, tb_route(sw), dma->port,
+ dma->base + MAIL_DATA, dwords, DMA_PORT_TIMEOUT);
+
+ in = MAIL_IN_CMD_FLASH_WRITE << MAIL_IN_CMD_SHIFT;
+
+ /* CSS header write is always done to the same magic address */
+ if (address >= DMA_PORT_CSS_ADDRESS) {
+ dwaddress = DMA_PORT_CSS_ADDRESS;
+ in |= MAIL_IN_CSS;
+ } else {
+ dwaddress = address / 4;
+ }
+
+ in |= ((dwords - 1) << MAIL_IN_DWORDS_SHIFT) & MAIL_IN_DWORDS_MASK;
+ in |= (dwaddress << MAIL_IN_ADDRESS_SHIFT) & MAIL_IN_ADDRESS_MASK;
+ in |= MAIL_IN_OP_REQUEST;
+
+ return dma_port_request(dma, in, DMA_PORT_TIMEOUT);
+}
+
+/**
+ * dma_port_flash_read() - Read from active flash region
+ * @dma: DMA control port
+ * @address: Address relative to the start of active region
+ * @buf: Buffer where the data is read
+ * @size: Size of the buffer
+ */
+int dma_port_flash_read(struct tb_dma_port *dma, unsigned int address,
+ void *buf, size_t size)
+{
+ unsigned int retries = DMA_PORT_RETRIES;
+ unsigned int offset;
+
+ offset = address & 3;
+ address = address & ~3;
+
+ do {
+ u32 nbytes = min_t(u32, size, MAIL_DATA_DWORDS * 4);
+ int ret;
+
+ ret = dma_port_flash_read_block(dma, address, dma->buf,
+ ALIGN(nbytes, 4));
+ if (ret) {
+ if (ret == -ETIMEDOUT) {
+ if (retries--)
+ continue;
+ ret = -EIO;
+ }
+ return ret;
+ }
+
+ memcpy(buf, dma->buf + offset, nbytes);
+
+ size -= nbytes;
+ address += nbytes;
+ buf += nbytes;
+ } while (size > 0);
+
+ return 0;
+}
+
+/**
+ * dma_port_flash_write() - Write to non-active flash region
+ * @dma: DMA control port
+ * @address: Address relative to the start of non-active region
+ * @buf: Data to write
+ * @size: Size of the buffer
+ *
+ * Writes block of data to the non-active flash region of the switch. If
+ * the address is given as %DMA_PORT_CSS_ADDRESS the block is written
+ * using CSS command.
+ */
+int dma_port_flash_write(struct tb_dma_port *dma, unsigned int address,
+ const void *buf, size_t size)
+{
+ unsigned int retries = DMA_PORT_RETRIES;
+ unsigned int offset;
+
+ if (address >= DMA_PORT_CSS_ADDRESS) {
+ offset = 0;
+ if (size > DMA_PORT_CSS_MAX_SIZE)
+ return -E2BIG;
+ } else {
+ offset = address & 3;
+ address = address & ~3;
+ }
+
+ do {
+ u32 nbytes = min_t(u32, size, MAIL_DATA_DWORDS * 4);
+ int ret;
+
+ memcpy(dma->buf + offset, buf, nbytes);
+
+ ret = dma_port_flash_write_block(dma, address, buf, nbytes);
+ if (ret) {
+ if (ret == -ETIMEDOUT) {
+ if (retries--)
+ continue;
+ ret = -EIO;
+ }
+ return ret;
+ }
+
+ size -= nbytes;
+ address += nbytes;
+ buf += nbytes;
+ } while (size > 0);
+
+ return 0;
+}
+
+/**
+ * dma_port_flash_update_auth() - Starts flash authenticate cycle
+ * @dma: DMA control port
+ *
+ * Starts the flash update authentication cycle. If the image in the
+ * non-active area was valid, the switch starts upgrade process where
+ * active and non-active area get swapped in the end. Caller should call
+ * dma_port_flash_update_auth_status() to get status of this command.
+ * This is because if the switch in question is root switch the
+ * thunderbolt host controller gets reset as well.
+ */
+int dma_port_flash_update_auth(struct tb_dma_port *dma)
+{
+ u32 in;
+
+ in = MAIL_IN_CMD_FLASH_UPDATE_AUTH << MAIL_IN_CMD_SHIFT;
+ in |= MAIL_IN_OP_REQUEST;
+
+ return dma_port_request(dma, in, 150);
+}
+
+/**
+ * dma_port_flash_update_auth_status() - Reads status of update auth command
+ * @dma: DMA control port
+ * @status: Status code of the operation
+ *
+ * The function checks if there is status available from the last update
+ * auth command. Returns %0 if there is no status and no further
+ * action is required. If there is status, %1 is returned instead and
+ * @status holds the failure code.
+ *
+ * Negative return means there was an error reading status from the
+ * switch.
+ */
+int dma_port_flash_update_auth_status(struct tb_dma_port *dma, u32 *status)
+{
+ struct tb_switch *sw = dma->sw;
+ u32 out, cmd;
+ int ret;
+
+ ret = dma_port_read(sw->tb->ctl, &out, tb_route(sw), dma->port,
+ dma->base + MAIL_OUT, 1, DMA_PORT_TIMEOUT);
+ if (ret)
+ return ret;
+
+ /* Check if the status relates to flash update auth */
+ cmd = (out & MAIL_OUT_STATUS_CMD_MASK) >> MAIL_OUT_STATUS_CMD_SHIFT;
+ if (cmd == MAIL_IN_CMD_FLASH_UPDATE_AUTH) {
+ if (status)
+ *status = out & MAIL_OUT_STATUS_MASK;
+
+ /* Reset is needed in any case */
+ return 1;
+ }
+
+ return 0;
+}
+
+/**
+ * dma_port_power_cycle() - Power cycles the switch
+ * @dma: DMA control port
+ *
+ * Triggers power cycle to the switch.
+ */
+int dma_port_power_cycle(struct tb_dma_port *dma)
+{
+ u32 in;
+
+ in = MAIL_IN_CMD_POWER_CYCLE << MAIL_IN_CMD_SHIFT;
+ in |= MAIL_IN_OP_REQUEST;
+
+ return dma_port_request(dma, in, 150);
+}
--- /dev/null
+/*
+ * Thunderbolt DMA configuration based mailbox support
+ *
+ * Copyright (C) 2017, Intel Corporation
+ * Authors: Michael Jamet <michael.jamet@intel.com>
+ * Mika Westerberg <mika.westerberg@linux.intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef DMA_PORT_H_
+#define DMA_PORT_H_
+
+#include "tb.h"
+
+struct tb_switch;
+struct tb_dma_port;
+
+#define DMA_PORT_CSS_ADDRESS 0x3fffff
+#define DMA_PORT_CSS_MAX_SIZE SZ_128
+
+struct tb_dma_port *dma_port_alloc(struct tb_switch *sw);
+void dma_port_free(struct tb_dma_port *dma);
+int dma_port_flash_read(struct tb_dma_port *dma, unsigned int address,
+ void *buf, size_t size);
+int dma_port_flash_update_auth(struct tb_dma_port *dma);
+int dma_port_flash_update_auth_status(struct tb_dma_port *dma, u32 *status);
+int dma_port_flash_write(struct tb_dma_port *dma, unsigned int address,
+ const void *buf, size_t size);
+int dma_port_power_cycle(struct tb_dma_port *dma);
+
+#endif
--- /dev/null
+/*
+ * Thunderbolt bus support
+ *
+ * Copyright (C) 2017, Intel Corporation
+ * Author: Mika Westerberg <mika.westerberg@linux.intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/device.h>
+#include <linux/idr.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/random.h>
+#include <crypto/hash.h>
+
+#include "tb.h"
+
+static DEFINE_IDA(tb_domain_ida);
+
+static const char * const tb_security_names[] = {
+ [TB_SECURITY_NONE] = "none",
+ [TB_SECURITY_USER] = "user",
+ [TB_SECURITY_SECURE] = "secure",
+ [TB_SECURITY_DPONLY] = "dponly",
+};
+
+static ssize_t security_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct tb *tb = container_of(dev, struct tb, dev);
+
+ return sprintf(buf, "%s\n", tb_security_names[tb->security_level]);
+}
+static DEVICE_ATTR_RO(security);
+
+static struct attribute *domain_attrs[] = {
+ &dev_attr_security.attr,
+ NULL,
+};
+
+static struct attribute_group domain_attr_group = {
+ .attrs = domain_attrs,
+};
+
+static const struct attribute_group *domain_attr_groups[] = {
+ &domain_attr_group,
+ NULL,
+};
+
+struct bus_type tb_bus_type = {
+ .name = "thunderbolt",
+};
+
+static void tb_domain_release(struct device *dev)
+{
+ struct tb *tb = container_of(dev, struct tb, dev);
+
+ tb_ctl_free(tb->ctl);
+ destroy_workqueue(tb->wq);
+ ida_simple_remove(&tb_domain_ida, tb->index);
+ mutex_destroy(&tb->lock);
+ kfree(tb);
+}
+
+struct device_type tb_domain_type = {
+ .name = "thunderbolt_domain",
+ .release = tb_domain_release,
+};
+
+/**
+ * tb_domain_alloc() - Allocate a domain
+ * @nhi: Pointer to the host controller
+ * @privsize: Size of the connection manager private data
+ *
+ * Allocates and initializes a new Thunderbolt domain. Connection
+ * managers are expected to call this and then fill in @cm_ops
+ * accordingly.
+ *
+ * Call tb_domain_put() to release the domain before it has been added
+ * to the system.
+ *
+ * Return: allocated domain structure on %NULL in case of error
+ */
+struct tb *tb_domain_alloc(struct tb_nhi *nhi, size_t privsize)
+{
+ struct tb *tb;
+
+ /*
+ * Make sure the structure sizes map with that the hardware
+ * expects because bit-fields are being used.
+ */
+ BUILD_BUG_ON(sizeof(struct tb_regs_switch_header) != 5 * 4);
+ BUILD_BUG_ON(sizeof(struct tb_regs_port_header) != 8 * 4);
+ BUILD_BUG_ON(sizeof(struct tb_regs_hop) != 2 * 4);
+
+ tb = kzalloc(sizeof(*tb) + privsize, GFP_KERNEL);
+ if (!tb)
+ return NULL;
+
+ tb->nhi = nhi;
+ mutex_init(&tb->lock);
+
+ tb->index = ida_simple_get(&tb_domain_ida, 0, 0, GFP_KERNEL);
+ if (tb->index < 0)
+ goto err_free;
+
+ tb->wq = alloc_ordered_workqueue("thunderbolt%d", 0, tb->index);
+ if (!tb->wq)
+ goto err_remove_ida;
+
+ tb->dev.parent = &nhi->pdev->dev;
+ tb->dev.bus = &tb_bus_type;
+ tb->dev.type = &tb_domain_type;
+ tb->dev.groups = domain_attr_groups;
+ dev_set_name(&tb->dev, "domain%d", tb->index);
+ device_initialize(&tb->dev);
+
+ return tb;
+
+err_remove_ida:
+ ida_simple_remove(&tb_domain_ida, tb->index);
+err_free:
+ kfree(tb);
+
+ return NULL;
+}
+
+static void tb_domain_event_cb(void *data, enum tb_cfg_pkg_type type,
+ const void *buf, size_t size)
+{
+ struct tb *tb = data;
+
+ if (!tb->cm_ops->handle_event) {
+ tb_warn(tb, "domain does not have event handler\n");
+ return;
+ }
+
+ tb->cm_ops->handle_event(tb, type, buf, size);
+}
+
+/**
+ * tb_domain_add() - Add domain to the system
+ * @tb: Domain to add
+ *
+ * Starts the domain and adds it to the system. Hotplugging devices will
+ * work after this has been returned successfully. In order to remove
+ * and release the domain after this function has been called, call
+ * tb_domain_remove().
+ *
+ * Return: %0 in case of success and negative errno in case of error
+ */
+int tb_domain_add(struct tb *tb)
+{
+ int ret;
+
+ if (WARN_ON(!tb->cm_ops))
+ return -EINVAL;
+
+ mutex_lock(&tb->lock);
+
+ tb->ctl = tb_ctl_alloc(tb->nhi, tb_domain_event_cb, tb);
+ if (!tb->ctl) {
+ ret = -ENOMEM;
+ goto err_unlock;
+ }
+
+ /*
+ * tb_schedule_hotplug_handler may be called as soon as the config
+ * channel is started. Thats why we have to hold the lock here.
+ */
+ tb_ctl_start(tb->ctl);
+
+ if (tb->cm_ops->driver_ready) {
+ ret = tb->cm_ops->driver_ready(tb);
+ if (ret)
+ goto err_ctl_stop;
+ }
+
+ ret = device_add(&tb->dev);
+ if (ret)
+ goto err_ctl_stop;
+
+ /* Start the domain */
+ if (tb->cm_ops->start) {
+ ret = tb->cm_ops->start(tb);
+ if (ret)
+ goto err_domain_del;
+ }
+
+ /* This starts event processing */
+ mutex_unlock(&tb->lock);
+
+ return 0;
+
+err_domain_del:
+ device_del(&tb->dev);
+err_ctl_stop:
+ tb_ctl_stop(tb->ctl);
+err_unlock:
+ mutex_unlock(&tb->lock);
+
+ return ret;
+}
+
+/**
+ * tb_domain_remove() - Removes and releases a domain
+ * @tb: Domain to remove
+ *
+ * Stops the domain, removes it from the system and releases all
+ * resources once the last reference has been released.
+ */
+void tb_domain_remove(struct tb *tb)
+{
+ mutex_lock(&tb->lock);
+ if (tb->cm_ops->stop)
+ tb->cm_ops->stop(tb);
+ /* Stop the domain control traffic */
+ tb_ctl_stop(tb->ctl);
+ mutex_unlock(&tb->lock);
+
+ flush_workqueue(tb->wq);
+ device_unregister(&tb->dev);
+}
+
+/**
+ * tb_domain_suspend_noirq() - Suspend a domain
+ * @tb: Domain to suspend
+ *
+ * Suspends all devices in the domain and stops the control channel.
+ */
+int tb_domain_suspend_noirq(struct tb *tb)
+{
+ int ret = 0;
+
+ /*
+ * The control channel interrupt is left enabled during suspend
+ * and taking the lock here prevents any events happening before
+ * we actually have stopped the domain and the control channel.
+ */
+ mutex_lock(&tb->lock);
+ if (tb->cm_ops->suspend_noirq)
+ ret = tb->cm_ops->suspend_noirq(tb);
+ if (!ret)
+ tb_ctl_stop(tb->ctl);
+ mutex_unlock(&tb->lock);
+
+ return ret;
+}
+
+/**
+ * tb_domain_resume_noirq() - Resume a domain
+ * @tb: Domain to resume
+ *
+ * Re-starts the control channel, and resumes all devices connected to
+ * the domain.
+ */
+int tb_domain_resume_noirq(struct tb *tb)
+{
+ int ret = 0;
+
+ mutex_lock(&tb->lock);
+ tb_ctl_start(tb->ctl);
+ if (tb->cm_ops->resume_noirq)
+ ret = tb->cm_ops->resume_noirq(tb);
+ mutex_unlock(&tb->lock);
+
+ return ret;
+}
+
+int tb_domain_suspend(struct tb *tb)
+{
+ int ret;
+
+ mutex_lock(&tb->lock);
+ if (tb->cm_ops->suspend) {
+ ret = tb->cm_ops->suspend(tb);
+ if (ret) {
+ mutex_unlock(&tb->lock);
+ return ret;
+ }
+ }
+ mutex_unlock(&tb->lock);
+ return 0;
+}
+
+void tb_domain_complete(struct tb *tb)
+{
+ mutex_lock(&tb->lock);
+ if (tb->cm_ops->complete)
+ tb->cm_ops->complete(tb);
+ mutex_unlock(&tb->lock);
+}
+
+/**
+ * tb_domain_approve_switch() - Approve switch
+ * @tb: Domain the switch belongs to
+ * @sw: Switch to approve
+ *
+ * This will approve switch by connection manager specific means. In
+ * case of success the connection manager will create tunnels for all
+ * supported protocols.
+ */
+int tb_domain_approve_switch(struct tb *tb, struct tb_switch *sw)
+{
+ struct tb_switch *parent_sw;
+
+ if (!tb->cm_ops->approve_switch)
+ return -EPERM;
+
+ /* The parent switch must be authorized before this one */
+ parent_sw = tb_to_switch(sw->dev.parent);
+ if (!parent_sw || !parent_sw->authorized)
+ return -EINVAL;
+
+ return tb->cm_ops->approve_switch(tb, sw);
+}
+
+/**
+ * tb_domain_approve_switch_key() - Approve switch and add key
+ * @tb: Domain the switch belongs to
+ * @sw: Switch to approve
+ *
+ * For switches that support secure connect, this function first adds
+ * key to the switch NVM using connection manager specific means. If
+ * adding the key is successful, the switch is approved and connected.
+ *
+ * Return: %0 on success and negative errno in case of failure.
+ */
+int tb_domain_approve_switch_key(struct tb *tb, struct tb_switch *sw)
+{
+ struct tb_switch *parent_sw;
+ int ret;
+
+ if (!tb->cm_ops->approve_switch || !tb->cm_ops->add_switch_key)
+ return -EPERM;
+
+ /* The parent switch must be authorized before this one */
+ parent_sw = tb_to_switch(sw->dev.parent);
+ if (!parent_sw || !parent_sw->authorized)
+ return -EINVAL;
+
+ ret = tb->cm_ops->add_switch_key(tb, sw);
+ if (ret)
+ return ret;
+
+ return tb->cm_ops->approve_switch(tb, sw);
+}
+
+/**
+ * tb_domain_challenge_switch_key() - Challenge and approve switch
+ * @tb: Domain the switch belongs to
+ * @sw: Switch to approve
+ *
+ * For switches that support secure connect, this function generates
+ * random challenge and sends it to the switch. The switch responds to
+ * this and if the response matches our random challenge, the switch is
+ * approved and connected.
+ *
+ * Return: %0 on success and negative errno in case of failure.
+ */
+int tb_domain_challenge_switch_key(struct tb *tb, struct tb_switch *sw)
+{
+ u8 challenge[TB_SWITCH_KEY_SIZE];
+ u8 response[TB_SWITCH_KEY_SIZE];
+ u8 hmac[TB_SWITCH_KEY_SIZE];
+ struct tb_switch *parent_sw;
+ struct crypto_shash *tfm;
+ struct shash_desc *shash;
+ int ret;
+
+ if (!tb->cm_ops->approve_switch || !tb->cm_ops->challenge_switch_key)
+ return -EPERM;
+
+ /* The parent switch must be authorized before this one */
+ parent_sw = tb_to_switch(sw->dev.parent);
+ if (!parent_sw || !parent_sw->authorized)
+ return -EINVAL;
+
+ get_random_bytes(challenge, sizeof(challenge));
+ ret = tb->cm_ops->challenge_switch_key(tb, sw, challenge, response);
+ if (ret)
+ return ret;
+
+ tfm = crypto_alloc_shash("hmac(sha256)", 0, 0);
+ if (IS_ERR(tfm))
+ return PTR_ERR(tfm);
+
+ ret = crypto_shash_setkey(tfm, sw->key, TB_SWITCH_KEY_SIZE);
+ if (ret)
+ goto err_free_tfm;
+
+ shash = kzalloc(sizeof(*shash) + crypto_shash_descsize(tfm),
+ GFP_KERNEL);
+ if (!shash) {
+ ret = -ENOMEM;
+ goto err_free_tfm;
+ }
+
+ shash->tfm = tfm;
+ shash->flags = CRYPTO_TFM_REQ_MAY_SLEEP;
+
+ memset(hmac, 0, sizeof(hmac));
+ ret = crypto_shash_digest(shash, challenge, sizeof(hmac), hmac);
+ if (ret)
+ goto err_free_shash;
+
+ /* The returned HMAC must match the one we calculated */
+ if (memcmp(response, hmac, sizeof(hmac))) {
+ ret = -EKEYREJECTED;
+ goto err_free_shash;
+ }
+
+ crypto_free_shash(tfm);
+ kfree(shash);
+
+ return tb->cm_ops->approve_switch(tb, sw);
+
+err_free_shash:
+ kfree(shash);
+err_free_tfm:
+ crypto_free_shash(tfm);
+
+ return ret;
+}
+
+/**
+ * tb_domain_disconnect_pcie_paths() - Disconnect all PCIe paths
+ * @tb: Domain whose PCIe paths to disconnect
+ *
+ * This needs to be called in preparation for NVM upgrade of the host
+ * controller. Makes sure all PCIe paths are disconnected.
+ *
+ * Return %0 on success and negative errno in case of error.
+ */
+int tb_domain_disconnect_pcie_paths(struct tb *tb)
+{
+ if (!tb->cm_ops->disconnect_pcie_paths)
+ return -EPERM;
+
+ return tb->cm_ops->disconnect_pcie_paths(tb);
+}
+
+int tb_domain_init(void)
+{
+ return bus_register(&tb_bus_type);
+}
+
+void tb_domain_exit(void)
+{
+ bus_unregister(&tb_bus_type);
+ ida_destroy(&tb_domain_ida);
+ tb_switch_exit();
+}
enum tb_drom_entry_type type:1;
} __packed;
+struct tb_drom_entry_generic {
+ struct tb_drom_entry_header header;
+ u8 data[0];
+} __packed;
+
struct tb_drom_entry_port {
/* BYTES 0-1 */
struct tb_drom_entry_header header;
if (res)
return res;
+ if (drom_offset == 0)
+ return -ENODEV;
+
/* read uid */
res = tb_eeprom_read_n(sw, drom_offset, data, 9);
if (res)
crc = tb_crc8(data + 1, 8);
if (crc != data[0]) {
- tb_sw_warn(sw, "uid crc8 missmatch (expected: %#x, got: %#x)\n",
+ tb_sw_warn(sw, "uid crc8 mismatch (expected: %#x, got: %#x)\n",
data[0], crc);
return -EIO;
}
return 0;
}
-static void tb_drom_parse_port_entry(struct tb_port *port,
- struct tb_drom_entry_port *entry)
+static int tb_drom_parse_entry_generic(struct tb_switch *sw,
+ struct tb_drom_entry_header *header)
{
- port->link_nr = entry->link_nr;
- if (entry->has_dual_link_port)
- port->dual_link_port =
- &port->sw->ports[entry->dual_link_port_nr];
+ const struct tb_drom_entry_generic *entry =
+ (const struct tb_drom_entry_generic *)header;
+
+ switch (header->index) {
+ case 1:
+ /* Length includes 2 bytes header so remove it before copy */
+ sw->vendor_name = kstrndup(entry->data,
+ header->len - sizeof(*header), GFP_KERNEL);
+ if (!sw->vendor_name)
+ return -ENOMEM;
+ break;
+
+ case 2:
+ sw->device_name = kstrndup(entry->data,
+ header->len - sizeof(*header), GFP_KERNEL);
+ if (!sw->device_name)
+ return -ENOMEM;
+ break;
+ }
+
+ return 0;
}
-static int tb_drom_parse_entry(struct tb_switch *sw,
- struct tb_drom_entry_header *header)
+static int tb_drom_parse_entry_port(struct tb_switch *sw,
+ struct tb_drom_entry_header *header)
{
struct tb_port *port;
int res;
enum tb_port_type type;
- if (header->type != TB_DROM_ENTRY_PORT)
- return 0;
-
port = &sw->ports[header->index];
port->disabled = header->port_disabled;
if (port->disabled)
header->len, sizeof(struct tb_drom_entry_port));
return -EIO;
}
- tb_drom_parse_port_entry(port, entry);
+ port->link_nr = entry->link_nr;
+ if (entry->has_dual_link_port)
+ port->dual_link_port =
+ &port->sw->ports[entry->dual_link_port_nr];
}
return 0;
}
struct tb_drom_header *header = (void *) sw->drom;
u16 pos = sizeof(*header);
u16 drom_size = header->data_len + TB_DROM_DATA_START;
+ int res;
while (pos < drom_size) {
struct tb_drom_entry_header *entry = (void *) (sw->drom + pos);
return -EIO;
}
- tb_drom_parse_entry(sw, entry);
+ switch (entry->type) {
+ case TB_DROM_ENTRY_GENERIC:
+ res = tb_drom_parse_entry_generic(sw, entry);
+ break;
+ case TB_DROM_ENTRY_PORT:
+ res = tb_drom_parse_entry_port(sw, entry);
+ break;
+ }
+ if (res)
+ return res;
pos += entry->len;
}
return -EINVAL;
}
+static int tb_drom_copy_nvm(struct tb_switch *sw, u16 *size)
+{
+ u32 drom_offset;
+ int ret;
+
+ if (!sw->dma_port)
+ return -ENODEV;
+
+ ret = tb_sw_read(sw, &drom_offset, TB_CFG_SWITCH,
+ sw->cap_plug_events + 12, 1);
+ if (ret)
+ return ret;
+
+ if (!drom_offset)
+ return -ENODEV;
+
+ ret = dma_port_flash_read(sw->dma_port, drom_offset + 14, size,
+ sizeof(*size));
+ if (ret)
+ return ret;
+
+ /* Size includes CRC8 + UID + CRC32 */
+ *size += 1 + 8 + 4;
+ sw->drom = kzalloc(*size, GFP_KERNEL);
+ if (!sw->drom)
+ return -ENOMEM;
+
+ ret = dma_port_flash_read(sw->dma_port, drom_offset, sw->drom, *size);
+ if (ret)
+ goto err_free;
+
+ /*
+ * Read UID from the minimal DROM because the one in NVM is just
+ * a placeholder.
+ */
+ tb_drom_read_uid_only(sw, &sw->uid);
+ return 0;
+
+err_free:
+ kfree(sw->drom);
+ sw->drom = NULL;
+ return ret;
+}
+
/**
* tb_drom_read - copy drom to sw->drom and parse it
*/
if (tb_drom_copy_efi(sw, &size) == 0)
goto parse;
+ /* Non-Apple hardware has the DROM as part of NVM */
+ if (tb_drom_copy_nvm(sw, &size) == 0)
+ goto parse;
+
/*
* The root switch contains only a dummy drom (header only,
* no entries). Hardcode the configuration here.
header->uid_crc8, crc);
goto err;
}
- sw->uid = header->uid;
+ if (!sw->uid)
+ sw->uid = header->uid;
+ sw->vendor = header->vendor_id;
+ sw->device = header->model_id;
crc = tb_crc32(sw->drom + TB_DROM_DATA_START, header->data_len);
if (crc != header->data_crc32) {
tb_sw_warn(sw,
- "drom data crc32 mismatch (expected: %#x, got: %#x), aborting\n",
+ "drom data crc32 mismatch (expected: %#x, got: %#x), continuing\n",
header->data_crc32, crc);
- goto err;
}
- if (header->device_rom_revision > 1)
+ if (header->device_rom_revision > 2)
tb_sw_warn(sw, "drom device_rom_revision %#x unknown\n",
header->device_rom_revision);
--- /dev/null
+/*
+ * Internal Thunderbolt Connection Manager. This is a firmware running on
+ * the Thunderbolt host controller performing most of the low-level
+ * handling.
+ *
+ * Copyright (C) 2017, Intel Corporation
+ * Authors: Michael Jamet <michael.jamet@intel.com>
+ * Mika Westerberg <mika.westerberg@linux.intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/delay.h>
+#include <linux/dmi.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/sizes.h>
+#include <linux/slab.h>
+#include <linux/workqueue.h>
+
+#include "ctl.h"
+#include "nhi_regs.h"
+#include "tb.h"
+
+#define PCIE2CIO_CMD 0x30
+#define PCIE2CIO_CMD_TIMEOUT BIT(31)
+#define PCIE2CIO_CMD_START BIT(30)
+#define PCIE2CIO_CMD_WRITE BIT(21)
+#define PCIE2CIO_CMD_CS_MASK GENMASK(20, 19)
+#define PCIE2CIO_CMD_CS_SHIFT 19
+#define PCIE2CIO_CMD_PORT_MASK GENMASK(18, 13)
+#define PCIE2CIO_CMD_PORT_SHIFT 13
+
+#define PCIE2CIO_WRDATA 0x34
+#define PCIE2CIO_RDDATA 0x38
+
+#define PHY_PORT_CS1 0x37
+#define PHY_PORT_CS1_LINK_DISABLE BIT(14)
+#define PHY_PORT_CS1_LINK_STATE_MASK GENMASK(29, 26)
+#define PHY_PORT_CS1_LINK_STATE_SHIFT 26
+
+#define ICM_TIMEOUT 5000 /* ms */
+#define ICM_MAX_LINK 4
+#define ICM_MAX_DEPTH 6
+
+/**
+ * struct icm - Internal connection manager private data
+ * @request_lock: Makes sure only one message is send to ICM at time
+ * @rescan_work: Work used to rescan the surviving switches after resume
+ * @upstream_port: Pointer to the PCIe upstream port this host
+ * controller is connected. This is only set for systems
+ * where ICM needs to be started manually
+ * @vnd_cap: Vendor defined capability where PCIe2CIO mailbox resides
+ * (only set when @upstream_port is not %NULL)
+ * @safe_mode: ICM is in safe mode
+ * @is_supported: Checks if we can support ICM on this controller
+ * @get_mode: Read and return the ICM firmware mode (optional)
+ * @get_route: Find a route string for given switch
+ * @device_connected: Handle device connected ICM message
+ * @device_disconnected: Handle device disconnected ICM message
+ */
+struct icm {
+ struct mutex request_lock;
+ struct delayed_work rescan_work;
+ struct pci_dev *upstream_port;
+ int vnd_cap;
+ bool safe_mode;
+ bool (*is_supported)(struct tb *tb);
+ int (*get_mode)(struct tb *tb);
+ int (*get_route)(struct tb *tb, u8 link, u8 depth, u64 *route);
+ void (*device_connected)(struct tb *tb,
+ const struct icm_pkg_header *hdr);
+ void (*device_disconnected)(struct tb *tb,
+ const struct icm_pkg_header *hdr);
+};
+
+struct icm_notification {
+ struct work_struct work;
+ struct icm_pkg_header *pkg;
+ struct tb *tb;
+};
+
+static inline struct tb *icm_to_tb(struct icm *icm)
+{
+ return ((void *)icm - sizeof(struct tb));
+}
+
+static inline u8 phy_port_from_route(u64 route, u8 depth)
+{
+ return tb_switch_phy_port_from_link(route >> ((depth - 1) * 8));
+}
+
+static inline u8 dual_link_from_link(u8 link)
+{
+ return link ? ((link - 1) ^ 0x01) + 1 : 0;
+}
+
+static inline u64 get_route(u32 route_hi, u32 route_lo)
+{
+ return (u64)route_hi << 32 | route_lo;
+}
+
+static inline bool is_apple(void)
+{
+ return dmi_match(DMI_BOARD_VENDOR, "Apple Inc.");
+}
+
+static bool icm_match(const struct tb_cfg_request *req,
+ const struct ctl_pkg *pkg)
+{
+ const struct icm_pkg_header *res_hdr = pkg->buffer;
+ const struct icm_pkg_header *req_hdr = req->request;
+
+ if (pkg->frame.eof != req->response_type)
+ return false;
+ if (res_hdr->code != req_hdr->code)
+ return false;
+
+ return true;
+}
+
+static bool icm_copy(struct tb_cfg_request *req, const struct ctl_pkg *pkg)
+{
+ const struct icm_pkg_header *hdr = pkg->buffer;
+
+ if (hdr->packet_id < req->npackets) {
+ size_t offset = hdr->packet_id * req->response_size;
+
+ memcpy(req->response + offset, pkg->buffer, req->response_size);
+ }
+
+ return hdr->packet_id == hdr->total_packets - 1;
+}
+
+static int icm_request(struct tb *tb, const void *request, size_t request_size,
+ void *response, size_t response_size, size_t npackets,
+ unsigned int timeout_msec)
+{
+ struct icm *icm = tb_priv(tb);
+ int retries = 3;
+
+ do {
+ struct tb_cfg_request *req;
+ struct tb_cfg_result res;
+
+ req = tb_cfg_request_alloc();
+ if (!req)
+ return -ENOMEM;
+
+ req->match = icm_match;
+ req->copy = icm_copy;
+ req->request = request;
+ req->request_size = request_size;
+ req->request_type = TB_CFG_PKG_ICM_CMD;
+ req->response = response;
+ req->npackets = npackets;
+ req->response_size = response_size;
+ req->response_type = TB_CFG_PKG_ICM_RESP;
+
+ mutex_lock(&icm->request_lock);
+ res = tb_cfg_request_sync(tb->ctl, req, timeout_msec);
+ mutex_unlock(&icm->request_lock);
+
+ tb_cfg_request_put(req);
+
+ if (res.err != -ETIMEDOUT)
+ return res.err == 1 ? -EIO : res.err;
+
+ usleep_range(20, 50);
+ } while (retries--);
+
+ return -ETIMEDOUT;
+}
+
+static bool icm_fr_is_supported(struct tb *tb)
+{
+ return !is_apple();
+}
+
+static inline int icm_fr_get_switch_index(u32 port)
+{
+ int index;
+
+ if ((port & ICM_PORT_TYPE_MASK) != TB_TYPE_PORT)
+ return 0;
+
+ index = port >> ICM_PORT_INDEX_SHIFT;
+ return index != 0xff ? index : 0;
+}
+
+static int icm_fr_get_route(struct tb *tb, u8 link, u8 depth, u64 *route)
+{
+ struct icm_fr_pkg_get_topology_response *switches, *sw;
+ struct icm_fr_pkg_get_topology request = {
+ .hdr = { .code = ICM_GET_TOPOLOGY },
+ };
+ size_t npackets = ICM_GET_TOPOLOGY_PACKETS;
+ int ret, index;
+ u8 i;
+
+ switches = kcalloc(npackets, sizeof(*switches), GFP_KERNEL);
+ if (!switches)
+ return -ENOMEM;
+
+ ret = icm_request(tb, &request, sizeof(request), switches,
+ sizeof(*switches), npackets, ICM_TIMEOUT);
+ if (ret)
+ goto err_free;
+
+ sw = &switches[0];
+ index = icm_fr_get_switch_index(sw->ports[link]);
+ if (!index) {
+ ret = -ENODEV;
+ goto err_free;
+ }
+
+ sw = &switches[index];
+ for (i = 1; i < depth; i++) {
+ unsigned int j;
+
+ if (!(sw->first_data & ICM_SWITCH_USED)) {
+ ret = -ENODEV;
+ goto err_free;
+ }
+
+ for (j = 0; j < ARRAY_SIZE(sw->ports); j++) {
+ index = icm_fr_get_switch_index(sw->ports[j]);
+ if (index > sw->switch_index) {
+ sw = &switches[index];
+ break;
+ }
+ }
+ }
+
+ *route = get_route(sw->route_hi, sw->route_lo);
+
+err_free:
+ kfree(switches);
+ return ret;
+}
+
+static int icm_fr_approve_switch(struct tb *tb, struct tb_switch *sw)
+{
+ struct icm_fr_pkg_approve_device request;
+ struct icm_fr_pkg_approve_device reply;
+ int ret;
+
+ memset(&request, 0, sizeof(request));
+ memcpy(&request.ep_uuid, sw->uuid, sizeof(request.ep_uuid));
+ request.hdr.code = ICM_APPROVE_DEVICE;
+ request.connection_id = sw->connection_id;
+ request.connection_key = sw->connection_key;
+
+ memset(&reply, 0, sizeof(reply));
+ /* Use larger timeout as establishing tunnels can take some time */
+ ret = icm_request(tb, &request, sizeof(request), &reply, sizeof(reply),
+ 1, 10000);
+ if (ret)
+ return ret;
+
+ if (reply.hdr.flags & ICM_FLAGS_ERROR) {
+ tb_warn(tb, "PCIe tunnel creation failed\n");
+ return -EIO;
+ }
+
+ return 0;
+}
+
+static int icm_fr_add_switch_key(struct tb *tb, struct tb_switch *sw)
+{
+ struct icm_fr_pkg_add_device_key request;
+ struct icm_fr_pkg_add_device_key_response reply;
+ int ret;
+
+ memset(&request, 0, sizeof(request));
+ memcpy(&request.ep_uuid, sw->uuid, sizeof(request.ep_uuid));
+ request.hdr.code = ICM_ADD_DEVICE_KEY;
+ request.connection_id = sw->connection_id;
+ request.connection_key = sw->connection_key;
+ memcpy(request.key, sw->key, TB_SWITCH_KEY_SIZE);
+
+ memset(&reply, 0, sizeof(reply));
+ ret = icm_request(tb, &request, sizeof(request), &reply, sizeof(reply),
+ 1, ICM_TIMEOUT);
+ if (ret)
+ return ret;
+
+ if (reply.hdr.flags & ICM_FLAGS_ERROR) {
+ tb_warn(tb, "Adding key to switch failed\n");
+ return -EIO;
+ }
+
+ return 0;
+}
+
+static int icm_fr_challenge_switch_key(struct tb *tb, struct tb_switch *sw,
+ const u8 *challenge, u8 *response)
+{
+ struct icm_fr_pkg_challenge_device request;
+ struct icm_fr_pkg_challenge_device_response reply;
+ int ret;
+
+ memset(&request, 0, sizeof(request));
+ memcpy(&request.ep_uuid, sw->uuid, sizeof(request.ep_uuid));
+ request.hdr.code = ICM_CHALLENGE_DEVICE;
+ request.connection_id = sw->connection_id;
+ request.connection_key = sw->connection_key;
+ memcpy(request.challenge, challenge, TB_SWITCH_KEY_SIZE);
+
+ memset(&reply, 0, sizeof(reply));
+ ret = icm_request(tb, &request, sizeof(request), &reply, sizeof(reply),
+ 1, ICM_TIMEOUT);
+ if (ret)
+ return ret;
+
+ if (reply.hdr.flags & ICM_FLAGS_ERROR)
+ return -EKEYREJECTED;
+ if (reply.hdr.flags & ICM_FLAGS_NO_KEY)
+ return -ENOKEY;
+
+ memcpy(response, reply.response, TB_SWITCH_KEY_SIZE);
+
+ return 0;
+}
+
+static void remove_switch(struct tb_switch *sw)
+{
+ struct tb_switch *parent_sw;
+
+ parent_sw = tb_to_switch(sw->dev.parent);
+ tb_port_at(tb_route(sw), parent_sw)->remote = NULL;
+ tb_switch_remove(sw);
+}
+
+static void
+icm_fr_device_connected(struct tb *tb, const struct icm_pkg_header *hdr)
+{
+ const struct icm_fr_event_device_connected *pkg =
+ (const struct icm_fr_event_device_connected *)hdr;
+ struct tb_switch *sw, *parent_sw;
+ struct icm *icm = tb_priv(tb);
+ bool authorized = false;
+ u8 link, depth;
+ u64 route;
+ int ret;
+
+ link = pkg->link_info & ICM_LINK_INFO_LINK_MASK;
+ depth = (pkg->link_info & ICM_LINK_INFO_DEPTH_MASK) >>
+ ICM_LINK_INFO_DEPTH_SHIFT;
+ authorized = pkg->link_info & ICM_LINK_INFO_APPROVED;
+
+ ret = icm->get_route(tb, link, depth, &route);
+ if (ret) {
+ tb_err(tb, "failed to find route string for switch at %u.%u\n",
+ link, depth);
+ return;
+ }
+
+ sw = tb_switch_find_by_uuid(tb, &pkg->ep_uuid);
+ if (sw) {
+ u8 phy_port, sw_phy_port;
+
+ parent_sw = tb_to_switch(sw->dev.parent);
+ sw_phy_port = phy_port_from_route(tb_route(sw), sw->depth);
+ phy_port = phy_port_from_route(route, depth);
+
+ /*
+ * On resume ICM will send us connected events for the
+ * devices that still are present. However, that
+ * information might have changed for example by the
+ * fact that a switch on a dual-link connection might
+ * have been enumerated using the other link now. Make
+ * sure our book keeping matches that.
+ */
+ if (sw->depth == depth && sw_phy_port == phy_port &&
+ !!sw->authorized == authorized) {
+ tb_port_at(tb_route(sw), parent_sw)->remote = NULL;
+ tb_port_at(route, parent_sw)->remote =
+ tb_upstream_port(sw);
+ sw->config.route_hi = upper_32_bits(route);
+ sw->config.route_lo = lower_32_bits(route);
+ sw->connection_id = pkg->connection_id;
+ sw->connection_key = pkg->connection_key;
+ sw->link = link;
+ sw->depth = depth;
+ sw->is_unplugged = false;
+ tb_switch_put(sw);
+ return;
+ }
+
+ /*
+ * User connected the same switch to another physical
+ * port or to another part of the topology. Remove the
+ * existing switch now before adding the new one.
+ */
+ remove_switch(sw);
+ tb_switch_put(sw);
+ }
+
+ /*
+ * If the switch was not found by UUID, look for a switch on
+ * same physical port (taking possible link aggregation into
+ * account) and depth. If we found one it is definitely a stale
+ * one so remove it first.
+ */
+ sw = tb_switch_find_by_link_depth(tb, link, depth);
+ if (!sw) {
+ u8 dual_link;
+
+ dual_link = dual_link_from_link(link);
+ if (dual_link)
+ sw = tb_switch_find_by_link_depth(tb, dual_link, depth);
+ }
+ if (sw) {
+ remove_switch(sw);
+ tb_switch_put(sw);
+ }
+
+ parent_sw = tb_switch_find_by_link_depth(tb, link, depth - 1);
+ if (!parent_sw) {
+ tb_err(tb, "failed to find parent switch for %u.%u\n",
+ link, depth);
+ return;
+ }
+
+ sw = tb_switch_alloc(tb, &parent_sw->dev, route);
+ if (!sw) {
+ tb_switch_put(parent_sw);
+ return;
+ }
+
+ sw->uuid = kmemdup(&pkg->ep_uuid, sizeof(pkg->ep_uuid), GFP_KERNEL);
+ sw->connection_id = pkg->connection_id;
+ sw->connection_key = pkg->connection_key;
+ sw->link = link;
+ sw->depth = depth;
+ sw->authorized = authorized;
+ sw->security_level = (pkg->hdr.flags & ICM_FLAGS_SLEVEL_MASK) >>
+ ICM_FLAGS_SLEVEL_SHIFT;
+
+ /* Link the two switches now */
+ tb_port_at(route, parent_sw)->remote = tb_upstream_port(sw);
+ tb_upstream_port(sw)->remote = tb_port_at(route, parent_sw);
+
+ ret = tb_switch_add(sw);
+ if (ret) {
+ tb_port_at(tb_route(sw), parent_sw)->remote = NULL;
+ tb_switch_put(sw);
+ }
+ tb_switch_put(parent_sw);
+}
+
+static void
+icm_fr_device_disconnected(struct tb *tb, const struct icm_pkg_header *hdr)
+{
+ const struct icm_fr_event_device_disconnected *pkg =
+ (const struct icm_fr_event_device_disconnected *)hdr;
+ struct tb_switch *sw;
+ u8 link, depth;
+
+ link = pkg->link_info & ICM_LINK_INFO_LINK_MASK;
+ depth = (pkg->link_info & ICM_LINK_INFO_DEPTH_MASK) >>
+ ICM_LINK_INFO_DEPTH_SHIFT;
+
+ if (link > ICM_MAX_LINK || depth > ICM_MAX_DEPTH) {
+ tb_warn(tb, "invalid topology %u.%u, ignoring\n", link, depth);
+ return;
+ }
+
+ sw = tb_switch_find_by_link_depth(tb, link, depth);
+ if (!sw) {
+ tb_warn(tb, "no switch exists at %u.%u, ignoring\n", link,
+ depth);
+ return;
+ }
+
+ remove_switch(sw);
+ tb_switch_put(sw);
+}
+
+static struct pci_dev *get_upstream_port(struct pci_dev *pdev)
+{
+ struct pci_dev *parent;
+
+ parent = pci_upstream_bridge(pdev);
+ while (parent) {
+ if (!pci_is_pcie(parent))
+ return NULL;
+ if (pci_pcie_type(parent) == PCI_EXP_TYPE_UPSTREAM)
+ break;
+ parent = pci_upstream_bridge(parent);
+ }
+
+ if (!parent)
+ return NULL;
+
+ switch (parent->device) {
+ case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_2C_BRIDGE:
+ case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_4C_BRIDGE:
+ case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_LP_BRIDGE:
+ case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_4C_BRIDGE:
+ case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_2C_BRIDGE:
+ return parent;
+ }
+
+ return NULL;
+}
+
+static bool icm_ar_is_supported(struct tb *tb)
+{
+ struct pci_dev *upstream_port;
+ struct icm *icm = tb_priv(tb);
+
+ /*
+ * Starting from Alpine Ridge we can use ICM on Apple machines
+ * as well. We just need to reset and re-enable it first.
+ */
+ if (!is_apple())
+ return true;
+
+ /*
+ * Find the upstream PCIe port in case we need to do reset
+ * through its vendor specific registers.
+ */
+ upstream_port = get_upstream_port(tb->nhi->pdev);
+ if (upstream_port) {
+ int cap;
+
+ cap = pci_find_ext_capability(upstream_port,
+ PCI_EXT_CAP_ID_VNDR);
+ if (cap > 0) {
+ icm->upstream_port = upstream_port;
+ icm->vnd_cap = cap;
+
+ return true;
+ }
+ }
+
+ return false;
+}
+
+static int icm_ar_get_mode(struct tb *tb)
+{
+ struct tb_nhi *nhi = tb->nhi;
+ int retries = 5;
+ u32 val;
+
+ do {
+ val = ioread32(nhi->iobase + REG_FW_STS);
+ if (val & REG_FW_STS_NVM_AUTH_DONE)
+ break;
+ msleep(30);
+ } while (--retries);
+
+ if (!retries) {
+ dev_err(&nhi->pdev->dev, "ICM firmware not authenticated\n");
+ return -ENODEV;
+ }
+
+ return nhi_mailbox_mode(nhi);
+}
+
+static int icm_ar_get_route(struct tb *tb, u8 link, u8 depth, u64 *route)
+{
+ struct icm_ar_pkg_get_route_response reply;
+ struct icm_ar_pkg_get_route request = {
+ .hdr = { .code = ICM_GET_ROUTE },
+ .link_info = depth << ICM_LINK_INFO_DEPTH_SHIFT | link,
+ };
+ int ret;
+
+ memset(&reply, 0, sizeof(reply));
+ ret = icm_request(tb, &request, sizeof(request), &reply, sizeof(reply),
+ 1, ICM_TIMEOUT);
+ if (ret)
+ return ret;
+
+ if (reply.hdr.flags & ICM_FLAGS_ERROR)
+ return -EIO;
+
+ *route = get_route(reply.route_hi, reply.route_lo);
+ return 0;
+}
+
+static void icm_handle_notification(struct work_struct *work)
+{
+ struct icm_notification *n = container_of(work, typeof(*n), work);
+ struct tb *tb = n->tb;
+ struct icm *icm = tb_priv(tb);
+
+ mutex_lock(&tb->lock);
+
+ switch (n->pkg->code) {
+ case ICM_EVENT_DEVICE_CONNECTED:
+ icm->device_connected(tb, n->pkg);
+ break;
+ case ICM_EVENT_DEVICE_DISCONNECTED:
+ icm->device_disconnected(tb, n->pkg);
+ break;
+ }
+
+ mutex_unlock(&tb->lock);
+
+ kfree(n->pkg);
+ kfree(n);
+}
+
+static void icm_handle_event(struct tb *tb, enum tb_cfg_pkg_type type,
+ const void *buf, size_t size)
+{
+ struct icm_notification *n;
+
+ n = kmalloc(sizeof(*n), GFP_KERNEL);
+ if (!n)
+ return;
+
+ INIT_WORK(&n->work, icm_handle_notification);
+ n->pkg = kmemdup(buf, size, GFP_KERNEL);
+ n->tb = tb;
+
+ queue_work(tb->wq, &n->work);
+}
+
+static int
+__icm_driver_ready(struct tb *tb, enum tb_security_level *security_level)
+{
+ struct icm_pkg_driver_ready_response reply;
+ struct icm_pkg_driver_ready request = {
+ .hdr.code = ICM_DRIVER_READY,
+ };
+ unsigned int retries = 10;
+ int ret;
+
+ memset(&reply, 0, sizeof(reply));
+ ret = icm_request(tb, &request, sizeof(request), &reply, sizeof(reply),
+ 1, ICM_TIMEOUT);
+ if (ret)
+ return ret;
+
+ if (security_level)
+ *security_level = reply.security_level & 0xf;
+
+ /*
+ * Hold on here until the switch config space is accessible so
+ * that we can read root switch config successfully.
+ */
+ do {
+ struct tb_cfg_result res;
+ u32 tmp;
+
+ res = tb_cfg_read_raw(tb->ctl, &tmp, 0, 0, TB_CFG_SWITCH,
+ 0, 1, 100);
+ if (!res.err)
+ return 0;
+
+ msleep(50);
+ } while (--retries);
+
+ return -ETIMEDOUT;
+}
+
+static int pci2cio_wait_completion(struct icm *icm, unsigned long timeout_msec)
+{
+ unsigned long end = jiffies + msecs_to_jiffies(timeout_msec);
+ u32 cmd;
+
+ do {
+ pci_read_config_dword(icm->upstream_port,
+ icm->vnd_cap + PCIE2CIO_CMD, &cmd);
+ if (!(cmd & PCIE2CIO_CMD_START)) {
+ if (cmd & PCIE2CIO_CMD_TIMEOUT)
+ break;
+ return 0;
+ }
+
+ msleep(50);
+ } while (time_before(jiffies, end));
+
+ return -ETIMEDOUT;
+}
+
+static int pcie2cio_read(struct icm *icm, enum tb_cfg_space cs,
+ unsigned int port, unsigned int index, u32 *data)
+{
+ struct pci_dev *pdev = icm->upstream_port;
+ int ret, vnd_cap = icm->vnd_cap;
+ u32 cmd;
+
+ cmd = index;
+ cmd |= (port << PCIE2CIO_CMD_PORT_SHIFT) & PCIE2CIO_CMD_PORT_MASK;
+ cmd |= (cs << PCIE2CIO_CMD_CS_SHIFT) & PCIE2CIO_CMD_CS_MASK;
+ cmd |= PCIE2CIO_CMD_START;
+ pci_write_config_dword(pdev, vnd_cap + PCIE2CIO_CMD, cmd);
+
+ ret = pci2cio_wait_completion(icm, 5000);
+ if (ret)
+ return ret;
+
+ pci_read_config_dword(pdev, vnd_cap + PCIE2CIO_RDDATA, data);
+ return 0;
+}
+
+static int pcie2cio_write(struct icm *icm, enum tb_cfg_space cs,
+ unsigned int port, unsigned int index, u32 data)
+{
+ struct pci_dev *pdev = icm->upstream_port;
+ int vnd_cap = icm->vnd_cap;
+ u32 cmd;
+
+ pci_write_config_dword(pdev, vnd_cap + PCIE2CIO_WRDATA, data);
+
+ cmd = index;
+ cmd |= (port << PCIE2CIO_CMD_PORT_SHIFT) & PCIE2CIO_CMD_PORT_MASK;
+ cmd |= (cs << PCIE2CIO_CMD_CS_SHIFT) & PCIE2CIO_CMD_CS_MASK;
+ cmd |= PCIE2CIO_CMD_WRITE | PCIE2CIO_CMD_START;
+ pci_write_config_dword(pdev, vnd_cap + PCIE2CIO_CMD, cmd);
+
+ return pci2cio_wait_completion(icm, 5000);
+}
+
+static int icm_firmware_reset(struct tb *tb, struct tb_nhi *nhi)
+{
+ struct icm *icm = tb_priv(tb);
+ u32 val;
+
+ /* Put ARC to wait for CIO reset event to happen */
+ val = ioread32(nhi->iobase + REG_FW_STS);
+ val |= REG_FW_STS_CIO_RESET_REQ;
+ iowrite32(val, nhi->iobase + REG_FW_STS);
+
+ /* Re-start ARC */
+ val = ioread32(nhi->iobase + REG_FW_STS);
+ val |= REG_FW_STS_ICM_EN_INVERT;
+ val |= REG_FW_STS_ICM_EN_CPU;
+ iowrite32(val, nhi->iobase + REG_FW_STS);
+
+ /* Trigger CIO reset now */
+ return pcie2cio_write(icm, TB_CFG_SWITCH, 0, 0x50, BIT(9));
+}
+
+static int icm_firmware_start(struct tb *tb, struct tb_nhi *nhi)
+{
+ unsigned int retries = 10;
+ int ret;
+ u32 val;
+
+ /* Check if the ICM firmware is already running */
+ val = ioread32(nhi->iobase + REG_FW_STS);
+ if (val & REG_FW_STS_ICM_EN)
+ return 0;
+
+ dev_info(&nhi->pdev->dev, "starting ICM firmware\n");
+
+ ret = icm_firmware_reset(tb, nhi);
+ if (ret)
+ return ret;
+
+ /* Wait until the ICM firmware tells us it is up and running */
+ do {
+ /* Check that the ICM firmware is running */
+ val = ioread32(nhi->iobase + REG_FW_STS);
+ if (val & REG_FW_STS_NVM_AUTH_DONE)
+ return 0;
+
+ msleep(300);
+ } while (--retries);
+
+ return -ETIMEDOUT;
+}
+
+static int icm_reset_phy_port(struct tb *tb, int phy_port)
+{
+ struct icm *icm = tb_priv(tb);
+ u32 state0, state1;
+ int port0, port1;
+ u32 val0, val1;
+ int ret;
+
+ if (!icm->upstream_port)
+ return 0;
+
+ if (phy_port) {
+ port0 = 3;
+ port1 = 4;
+ } else {
+ port0 = 1;
+ port1 = 2;
+ }
+
+ /*
+ * Read link status of both null ports belonging to a single
+ * physical port.
+ */
+ ret = pcie2cio_read(icm, TB_CFG_PORT, port0, PHY_PORT_CS1, &val0);
+ if (ret)
+ return ret;
+ ret = pcie2cio_read(icm, TB_CFG_PORT, port1, PHY_PORT_CS1, &val1);
+ if (ret)
+ return ret;
+
+ state0 = val0 & PHY_PORT_CS1_LINK_STATE_MASK;
+ state0 >>= PHY_PORT_CS1_LINK_STATE_SHIFT;
+ state1 = val1 & PHY_PORT_CS1_LINK_STATE_MASK;
+ state1 >>= PHY_PORT_CS1_LINK_STATE_SHIFT;
+
+ /* If they are both up we need to reset them now */
+ if (state0 != TB_PORT_UP || state1 != TB_PORT_UP)
+ return 0;
+
+ val0 |= PHY_PORT_CS1_LINK_DISABLE;
+ ret = pcie2cio_write(icm, TB_CFG_PORT, port0, PHY_PORT_CS1, val0);
+ if (ret)
+ return ret;
+
+ val1 |= PHY_PORT_CS1_LINK_DISABLE;
+ ret = pcie2cio_write(icm, TB_CFG_PORT, port1, PHY_PORT_CS1, val1);
+ if (ret)
+ return ret;
+
+ /* Wait a bit and then re-enable both ports */
+ usleep_range(10, 100);
+
+ ret = pcie2cio_read(icm, TB_CFG_PORT, port0, PHY_PORT_CS1, &val0);
+ if (ret)
+ return ret;
+ ret = pcie2cio_read(icm, TB_CFG_PORT, port1, PHY_PORT_CS1, &val1);
+ if (ret)
+ return ret;
+
+ val0 &= ~PHY_PORT_CS1_LINK_DISABLE;
+ ret = pcie2cio_write(icm, TB_CFG_PORT, port0, PHY_PORT_CS1, val0);
+ if (ret)
+ return ret;
+
+ val1 &= ~PHY_PORT_CS1_LINK_DISABLE;
+ return pcie2cio_write(icm, TB_CFG_PORT, port1, PHY_PORT_CS1, val1);
+}
+
+static int icm_firmware_init(struct tb *tb)
+{
+ struct icm *icm = tb_priv(tb);
+ struct tb_nhi *nhi = tb->nhi;
+ int ret;
+
+ ret = icm_firmware_start(tb, nhi);
+ if (ret) {
+ dev_err(&nhi->pdev->dev, "could not start ICM firmware\n");
+ return ret;
+ }
+
+ if (icm->get_mode) {
+ ret = icm->get_mode(tb);
+
+ switch (ret) {
+ case NHI_FW_SAFE_MODE:
+ icm->safe_mode = true;
+ break;
+
+ case NHI_FW_CM_MODE:
+ /* Ask ICM to accept all Thunderbolt devices */
+ nhi_mailbox_cmd(nhi, NHI_MAILBOX_ALLOW_ALL_DEVS, 0);
+ break;
+
+ default:
+ tb_err(tb, "ICM firmware is in wrong mode: %u\n", ret);
+ return -ENODEV;
+ }
+ }
+
+ /*
+ * Reset both physical ports if there is anything connected to
+ * them already.
+ */
+ ret = icm_reset_phy_port(tb, 0);
+ if (ret)
+ dev_warn(&nhi->pdev->dev, "failed to reset links on port0\n");
+ ret = icm_reset_phy_port(tb, 1);
+ if (ret)
+ dev_warn(&nhi->pdev->dev, "failed to reset links on port1\n");
+
+ return 0;
+}
+
+static int icm_driver_ready(struct tb *tb)
+{
+ struct icm *icm = tb_priv(tb);
+ int ret;
+
+ ret = icm_firmware_init(tb);
+ if (ret)
+ return ret;
+
+ if (icm->safe_mode) {
+ tb_info(tb, "Thunderbolt host controller is in safe mode.\n");
+ tb_info(tb, "You need to update NVM firmware of the controller before it can be used.\n");
+ tb_info(tb, "For latest updates check https://thunderbolttechnology.net/updates.\n");
+ return 0;
+ }
+
+ return __icm_driver_ready(tb, &tb->security_level);
+}
+
+static int icm_suspend(struct tb *tb)
+{
+ return nhi_mailbox_cmd(tb->nhi, NHI_MAILBOX_SAVE_DEVS, 0);
+}
+
+/*
+ * Mark all switches (except root switch) below this one unplugged. ICM
+ * firmware will send us an updated list of switches after we have send
+ * it driver ready command. If a switch is not in that list it will be
+ * removed when we perform rescan.
+ */
+static void icm_unplug_children(struct tb_switch *sw)
+{
+ unsigned int i;
+
+ if (tb_route(sw))
+ sw->is_unplugged = true;
+
+ for (i = 1; i <= sw->config.max_port_number; i++) {
+ struct tb_port *port = &sw->ports[i];
+
+ if (tb_is_upstream_port(port))
+ continue;
+ if (!port->remote)
+ continue;
+
+ icm_unplug_children(port->remote->sw);
+ }
+}
+
+static void icm_free_unplugged_children(struct tb_switch *sw)
+{
+ unsigned int i;
+
+ for (i = 1; i <= sw->config.max_port_number; i++) {
+ struct tb_port *port = &sw->ports[i];
+
+ if (tb_is_upstream_port(port))
+ continue;
+ if (!port->remote)
+ continue;
+
+ if (port->remote->sw->is_unplugged) {
+ tb_switch_remove(port->remote->sw);
+ port->remote = NULL;
+ } else {
+ icm_free_unplugged_children(port->remote->sw);
+ }
+ }
+}
+
+static void icm_rescan_work(struct work_struct *work)
+{
+ struct icm *icm = container_of(work, struct icm, rescan_work.work);
+ struct tb *tb = icm_to_tb(icm);
+
+ mutex_lock(&tb->lock);
+ if (tb->root_switch)
+ icm_free_unplugged_children(tb->root_switch);
+ mutex_unlock(&tb->lock);
+}
+
+static void icm_complete(struct tb *tb)
+{
+ struct icm *icm = tb_priv(tb);
+
+ if (tb->nhi->going_away)
+ return;
+
+ icm_unplug_children(tb->root_switch);
+
+ /*
+ * Now all existing children should be resumed, start events
+ * from ICM to get updated status.
+ */
+ __icm_driver_ready(tb, NULL);
+
+ /*
+ * We do not get notifications of devices that have been
+ * unplugged during suspend so schedule rescan to clean them up
+ * if any.
+ */
+ queue_delayed_work(tb->wq, &icm->rescan_work, msecs_to_jiffies(500));
+}
+
+static int icm_start(struct tb *tb)
+{
+ struct icm *icm = tb_priv(tb);
+ int ret;
+
+ if (icm->safe_mode)
+ tb->root_switch = tb_switch_alloc_safe_mode(tb, &tb->dev, 0);
+ else
+ tb->root_switch = tb_switch_alloc(tb, &tb->dev, 0);
+ if (!tb->root_switch)
+ return -ENODEV;
+
+ /*
+ * NVM upgrade has not been tested on Apple systems and they
+ * don't provide images publicly either. To be on the safe side
+ * prevent root switch NVM upgrade on Macs for now.
+ */
+ tb->root_switch->no_nvm_upgrade = is_apple();
+
+ ret = tb_switch_add(tb->root_switch);
+ if (ret)
+ tb_switch_put(tb->root_switch);
+
+ return ret;
+}
+
+static void icm_stop(struct tb *tb)
+{
+ struct icm *icm = tb_priv(tb);
+
+ cancel_delayed_work(&icm->rescan_work);
+ tb_switch_remove(tb->root_switch);
+ tb->root_switch = NULL;
+ nhi_mailbox_cmd(tb->nhi, NHI_MAILBOX_DRV_UNLOADS, 0);
+}
+
+static int icm_disconnect_pcie_paths(struct tb *tb)
+{
+ return nhi_mailbox_cmd(tb->nhi, NHI_MAILBOX_DISCONNECT_PCIE_PATHS, 0);
+}
+
+/* Falcon Ridge and Alpine Ridge */
+static const struct tb_cm_ops icm_fr_ops = {
+ .driver_ready = icm_driver_ready,
+ .start = icm_start,
+ .stop = icm_stop,
+ .suspend = icm_suspend,
+ .complete = icm_complete,
+ .handle_event = icm_handle_event,
+ .approve_switch = icm_fr_approve_switch,
+ .add_switch_key = icm_fr_add_switch_key,
+ .challenge_switch_key = icm_fr_challenge_switch_key,
+ .disconnect_pcie_paths = icm_disconnect_pcie_paths,
+};
+
+struct tb *icm_probe(struct tb_nhi *nhi)
+{
+ struct icm *icm;
+ struct tb *tb;
+
+ tb = tb_domain_alloc(nhi, sizeof(struct icm));
+ if (!tb)
+ return NULL;
+
+ icm = tb_priv(tb);
+ INIT_DELAYED_WORK(&icm->rescan_work, icm_rescan_work);
+ mutex_init(&icm->request_lock);
+
+ switch (nhi->pdev->device) {
+ case PCI_DEVICE_ID_INTEL_FALCON_RIDGE_2C_NHI:
+ case PCI_DEVICE_ID_INTEL_FALCON_RIDGE_4C_NHI:
+ icm->is_supported = icm_fr_is_supported;
+ icm->get_route = icm_fr_get_route;
+ icm->device_connected = icm_fr_device_connected;
+ icm->device_disconnected = icm_fr_device_disconnected;
+ tb->cm_ops = &icm_fr_ops;
+ break;
+
+ case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_2C_NHI:
+ case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_4C_NHI:
+ case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_LP_NHI:
+ case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_4C_NHI:
+ case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_2C_NHI:
+ icm->is_supported = icm_ar_is_supported;
+ icm->get_mode = icm_ar_get_mode;
+ icm->get_route = icm_ar_get_route;
+ icm->device_connected = icm_fr_device_connected;
+ icm->device_disconnected = icm_fr_device_disconnected;
+ tb->cm_ops = &icm_fr_ops;
+ break;
+ }
+
+ if (!icm->is_supported || !icm->is_supported(tb)) {
+ dev_dbg(&nhi->pdev->dev, "ICM not supported on this controller\n");
+ tb_domain_put(tb);
+ return NULL;
+ }
+
+ return tb;
+}
#include <linux/pci.h>
#include <linux/interrupt.h>
#include <linux/module.h>
-#include <linux/dmi.h>
+#include <linux/delay.h>
#include "nhi.h"
#include "nhi_regs.h"
#define RING_TYPE(ring) ((ring)->is_tx ? "TX ring" : "RX ring")
+/*
+ * Minimal number of vectors when we use MSI-X. Two for control channel
+ * Rx/Tx and the rest four are for cross domain DMA paths.
+ */
+#define MSIX_MIN_VECS 6
+#define MSIX_MAX_VECS 16
+
+#define NHI_MAILBOX_TIMEOUT 500 /* ms */
static int ring_interrupt_index(struct tb_ring *ring)
{
int bit = ring_interrupt_index(ring) & 31;
int mask = 1 << bit;
u32 old, new;
+
+ if (ring->irq > 0) {
+ u32 step, shift, ivr, misc;
+ void __iomem *ivr_base;
+ int index;
+
+ if (ring->is_tx)
+ index = ring->hop;
+ else
+ index = ring->hop + ring->nhi->hop_count;
+
+ /*
+ * Ask the hardware to clear interrupt status bits automatically
+ * since we already know which interrupt was triggered.
+ */
+ misc = ioread32(ring->nhi->iobase + REG_DMA_MISC);
+ if (!(misc & REG_DMA_MISC_INT_AUTO_CLEAR)) {
+ misc |= REG_DMA_MISC_INT_AUTO_CLEAR;
+ iowrite32(misc, ring->nhi->iobase + REG_DMA_MISC);
+ }
+
+ ivr_base = ring->nhi->iobase + REG_INT_VEC_ALLOC_BASE;
+ step = index / REG_INT_VEC_ALLOC_REGS * REG_INT_VEC_ALLOC_BITS;
+ shift = index % REG_INT_VEC_ALLOC_REGS * REG_INT_VEC_ALLOC_BITS;
+ ivr = ioread32(ivr_base + step);
+ ivr &= ~(REG_INT_VEC_ALLOC_MASK << shift);
+ if (active)
+ ivr |= ring->vector << shift;
+ iowrite32(ivr, ivr_base + step);
+ }
+
old = ioread32(ring->nhi->iobase + reg);
if (active)
new = old | mask;
return ret;
}
+static irqreturn_t ring_msix(int irq, void *data)
+{
+ struct tb_ring *ring = data;
+
+ schedule_work(&ring->work);
+ return IRQ_HANDLED;
+}
+
+static int ring_request_msix(struct tb_ring *ring, bool no_suspend)
+{
+ struct tb_nhi *nhi = ring->nhi;
+ unsigned long irqflags;
+ int ret;
+
+ if (!nhi->pdev->msix_enabled)
+ return 0;
+
+ ret = ida_simple_get(&nhi->msix_ida, 0, MSIX_MAX_VECS, GFP_KERNEL);
+ if (ret < 0)
+ return ret;
+
+ ring->vector = ret;
+
+ ring->irq = pci_irq_vector(ring->nhi->pdev, ring->vector);
+ if (ring->irq < 0)
+ return ring->irq;
+
+ irqflags = no_suspend ? IRQF_NO_SUSPEND : 0;
+ return request_irq(ring->irq, ring_msix, irqflags, "thunderbolt", ring);
+}
+
+static void ring_release_msix(struct tb_ring *ring)
+{
+ if (ring->irq <= 0)
+ return;
+
+ free_irq(ring->irq, ring);
+ ida_simple_remove(&ring->nhi->msix_ida, ring->vector);
+ ring->vector = 0;
+ ring->irq = 0;
+}
+
static struct tb_ring *ring_alloc(struct tb_nhi *nhi, u32 hop, int size,
- bool transmit)
+ bool transmit, unsigned int flags)
{
struct tb_ring *ring = NULL;
dev_info(&nhi->pdev->dev, "allocating %s ring %d of size %d\n",
ring->hop = hop;
ring->is_tx = transmit;
ring->size = size;
+ ring->flags = flags;
ring->head = 0;
ring->tail = 0;
ring->running = false;
+
+ if (ring_request_msix(ring, flags & RING_FLAG_NO_SUSPEND))
+ goto err;
+
ring->descriptors = dma_alloc_coherent(&ring->nhi->pdev->dev,
size * sizeof(*ring->descriptors),
&ring->descriptors_dma, GFP_KERNEL | __GFP_ZERO);
return NULL;
}
-struct tb_ring *ring_alloc_tx(struct tb_nhi *nhi, int hop, int size)
+struct tb_ring *ring_alloc_tx(struct tb_nhi *nhi, int hop, int size,
+ unsigned int flags)
{
- return ring_alloc(nhi, hop, size, true);
+ return ring_alloc(nhi, hop, size, true, flags);
}
-struct tb_ring *ring_alloc_rx(struct tb_nhi *nhi, int hop, int size)
+struct tb_ring *ring_alloc_rx(struct tb_nhi *nhi, int hop, int size,
+ unsigned int flags)
{
- return ring_alloc(nhi, hop, size, false);
+ return ring_alloc(nhi, hop, size, false, flags);
}
/**
{
mutex_lock(&ring->nhi->lock);
mutex_lock(&ring->lock);
+ if (ring->nhi->going_away)
+ goto err;
if (ring->running) {
dev_WARN(&ring->nhi->pdev->dev, "ring already started\n");
goto err;
mutex_lock(&ring->lock);
dev_info(&ring->nhi->pdev->dev, "stopping %s %d\n",
RING_TYPE(ring), ring->hop);
+ if (ring->nhi->going_away)
+ goto err;
if (!ring->running) {
dev_WARN(&ring->nhi->pdev->dev, "%s %d already stopped\n",
RING_TYPE(ring), ring->hop);
RING_TYPE(ring), ring->hop);
}
+ ring_release_msix(ring);
+
dma_free_coherent(&ring->nhi->pdev->dev,
ring->size * sizeof(*ring->descriptors),
ring->descriptors, ring->descriptors_dma);
mutex_unlock(&ring->nhi->lock);
/**
- * ring->work can no longer be scheduled (it is scheduled only by
- * nhi_interrupt_work and ring_stop). Wait for it to finish before
- * freeing the ring.
+ * ring->work can no longer be scheduled (it is scheduled only
+ * by nhi_interrupt_work, ring_stop and ring_msix). Wait for it
+ * to finish before freeing the ring.
*/
flush_work(&ring->work);
mutex_destroy(&ring->lock);
kfree(ring);
}
+/**
+ * nhi_mailbox_cmd() - Send a command through NHI mailbox
+ * @nhi: Pointer to the NHI structure
+ * @cmd: Command to send
+ * @data: Data to be send with the command
+ *
+ * Sends mailbox command to the firmware running on NHI. Returns %0 in
+ * case of success and negative errno in case of failure.
+ */
+int nhi_mailbox_cmd(struct tb_nhi *nhi, enum nhi_mailbox_cmd cmd, u32 data)
+{
+ ktime_t timeout;
+ u32 val;
+
+ iowrite32(data, nhi->iobase + REG_INMAIL_DATA);
+
+ val = ioread32(nhi->iobase + REG_INMAIL_CMD);
+ val &= ~(REG_INMAIL_CMD_MASK | REG_INMAIL_ERROR);
+ val |= REG_INMAIL_OP_REQUEST | cmd;
+ iowrite32(val, nhi->iobase + REG_INMAIL_CMD);
+
+ timeout = ktime_add_ms(ktime_get(), NHI_MAILBOX_TIMEOUT);
+ do {
+ val = ioread32(nhi->iobase + REG_INMAIL_CMD);
+ if (!(val & REG_INMAIL_OP_REQUEST))
+ break;
+ usleep_range(10, 20);
+ } while (ktime_before(ktime_get(), timeout));
+
+ if (val & REG_INMAIL_OP_REQUEST)
+ return -ETIMEDOUT;
+ if (val & REG_INMAIL_ERROR)
+ return -EIO;
+
+ return 0;
+}
+
+/**
+ * nhi_mailbox_mode() - Return current firmware operation mode
+ * @nhi: Pointer to the NHI structure
+ *
+ * The function reads current firmware operation mode using NHI mailbox
+ * registers and returns it to the caller.
+ */
+enum nhi_fw_mode nhi_mailbox_mode(struct tb_nhi *nhi)
+{
+ u32 val;
+
+ val = ioread32(nhi->iobase + REG_OUTMAIL_CMD);
+ val &= REG_OUTMAIL_CMD_OPMODE_MASK;
+ val >>= REG_OUTMAIL_CMD_OPMODE_SHIFT;
+
+ return (enum nhi_fw_mode)val;
+}
+
static void nhi_interrupt_work(struct work_struct *work)
{
struct tb_nhi *nhi = container_of(work, typeof(*nhi), interrupt_work);
{
struct pci_dev *pdev = to_pci_dev(dev);
struct tb *tb = pci_get_drvdata(pdev);
- thunderbolt_suspend(tb);
- return 0;
+
+ return tb_domain_suspend_noirq(tb);
}
static int nhi_resume_noirq(struct device *dev)
{
struct pci_dev *pdev = to_pci_dev(dev);
struct tb *tb = pci_get_drvdata(pdev);
- thunderbolt_resume(tb);
- return 0;
+
+ /*
+ * Check that the device is still there. It may be that the user
+ * unplugged last device which causes the host controller to go
+ * away on PCs.
+ */
+ if (!pci_device_is_present(pdev))
+ tb->nhi->going_away = true;
+
+ return tb_domain_resume_noirq(tb);
+}
+
+static int nhi_suspend(struct device *dev)
+{
+ struct pci_dev *pdev = to_pci_dev(dev);
+ struct tb *tb = pci_get_drvdata(pdev);
+
+ return tb_domain_suspend(tb);
+}
+
+static void nhi_complete(struct device *dev)
+{
+ struct pci_dev *pdev = to_pci_dev(dev);
+ struct tb *tb = pci_get_drvdata(pdev);
+
+ tb_domain_complete(tb);
}
static void nhi_shutdown(struct tb_nhi *nhi)
* We have to release the irq before calling flush_work. Otherwise an
* already executing IRQ handler could call schedule_work again.
*/
- devm_free_irq(&nhi->pdev->dev, nhi->pdev->irq, nhi);
- flush_work(&nhi->interrupt_work);
+ if (!nhi->pdev->msix_enabled) {
+ devm_free_irq(&nhi->pdev->dev, nhi->pdev->irq, nhi);
+ flush_work(&nhi->interrupt_work);
+ }
mutex_destroy(&nhi->lock);
+ ida_destroy(&nhi->msix_ida);
+}
+
+static int nhi_init_msi(struct tb_nhi *nhi)
+{
+ struct pci_dev *pdev = nhi->pdev;
+ int res, irq, nvec;
+
+ /* In case someone left them on. */
+ nhi_disable_interrupts(nhi);
+
+ ida_init(&nhi->msix_ida);
+
+ /*
+ * The NHI has 16 MSI-X vectors or a single MSI. We first try to
+ * get all MSI-X vectors and if we succeed, each ring will have
+ * one MSI-X. If for some reason that does not work out, we
+ * fallback to a single MSI.
+ */
+ nvec = pci_alloc_irq_vectors(pdev, MSIX_MIN_VECS, MSIX_MAX_VECS,
+ PCI_IRQ_MSIX);
+ if (nvec < 0) {
+ nvec = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
+ if (nvec < 0)
+ return nvec;
+
+ INIT_WORK(&nhi->interrupt_work, nhi_interrupt_work);
+
+ irq = pci_irq_vector(nhi->pdev, 0);
+ if (irq < 0)
+ return irq;
+
+ res = devm_request_irq(&pdev->dev, irq, nhi_msi,
+ IRQF_NO_SUSPEND, "thunderbolt", nhi);
+ if (res) {
+ dev_err(&pdev->dev, "request_irq failed, aborting\n");
+ return res;
+ }
+ }
+
+ return 0;
}
static int nhi_probe(struct pci_dev *pdev, const struct pci_device_id *id)
return res;
}
- res = pci_enable_msi(pdev);
- if (res) {
- dev_err(&pdev->dev, "cannot enable MSI, aborting\n");
- return res;
- }
-
res = pcim_iomap_regions(pdev, 1 << 0, "thunderbolt");
if (res) {
dev_err(&pdev->dev, "cannot obtain PCI resources, aborting\n");
if (nhi->hop_count != 12 && nhi->hop_count != 32)
dev_warn(&pdev->dev, "unexpected hop count: %d\n",
nhi->hop_count);
- INIT_WORK(&nhi->interrupt_work, nhi_interrupt_work);
nhi->tx_rings = devm_kcalloc(&pdev->dev, nhi->hop_count,
sizeof(*nhi->tx_rings), GFP_KERNEL);
if (!nhi->tx_rings || !nhi->rx_rings)
return -ENOMEM;
- nhi_disable_interrupts(nhi); /* In case someone left them on. */
- res = devm_request_irq(&pdev->dev, pdev->irq, nhi_msi,
- IRQF_NO_SUSPEND, /* must work during _noirq */
- "thunderbolt", nhi);
+ res = nhi_init_msi(nhi);
if (res) {
- dev_err(&pdev->dev, "request_irq failed, aborting\n");
+ dev_err(&pdev->dev, "cannot enable MSI, aborting\n");
return res;
}
/* magic value - clock related? */
iowrite32(3906250 / 10000, nhi->iobase + 0x38c00);
- dev_info(&nhi->pdev->dev, "NHI initialized, starting thunderbolt\n");
- tb = thunderbolt_alloc_and_start(nhi);
+ tb = icm_probe(nhi);
+ if (!tb)
+ tb = tb_probe(nhi);
if (!tb) {
+ dev_err(&nhi->pdev->dev,
+ "failed to determine connection manager, aborting\n");
+ return -ENODEV;
+ }
+
+ dev_info(&nhi->pdev->dev, "NHI initialized, starting thunderbolt\n");
+
+ res = tb_domain_add(tb);
+ if (res) {
/*
* At this point the RX/TX rings might already have been
* activated. Do a proper shutdown.
*/
+ tb_domain_put(tb);
nhi_shutdown(nhi);
return -EIO;
}
{
struct tb *tb = pci_get_drvdata(pdev);
struct tb_nhi *nhi = tb->nhi;
- thunderbolt_shutdown_and_free(tb);
+
+ tb_domain_remove(tb);
nhi_shutdown(nhi);
}
* pci-tunnels stay alive.
*/
.restore_noirq = nhi_resume_noirq,
+ .suspend = nhi_suspend,
+ .freeze = nhi_suspend,
+ .poweroff = nhi_suspend,
+ .complete = nhi_complete,
};
static struct pci_device_id nhi_ids[] = {
.device = PCI_DEVICE_ID_INTEL_FALCON_RIDGE_4C_NHI,
.subvendor = PCI_ANY_ID, .subdevice = PCI_ANY_ID,
},
+
+ /* Thunderbolt 3 */
+ { PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_2C_NHI) },
+ { PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_4C_NHI) },
+ { PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_USBONLY_NHI) },
+ { PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_LP_NHI) },
+ { PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_LP_USBONLY_NHI) },
+ { PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_2C_NHI) },
+ { PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_4C_NHI) },
+ { PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_USBONLY_NHI) },
+
{ 0,}
};
static int __init nhi_init(void)
{
- if (!dmi_match(DMI_BOARD_VENDOR, "Apple Inc."))
- return -ENOSYS;
- return pci_register_driver(&nhi_driver);
+ int ret;
+
+ ret = tb_domain_init();
+ if (ret)
+ return ret;
+ ret = pci_register_driver(&nhi_driver);
+ if (ret)
+ tb_domain_exit();
+ return ret;
}
static void __exit nhi_unload(void)
{
pci_unregister_driver(&nhi_driver);
+ tb_domain_exit();
}
module_init(nhi_init);
#ifndef DSL3510_H_
#define DSL3510_H_
+#include <linux/idr.h>
#include <linux/mutex.h>
#include <linux/workqueue.h>
/**
* struct tb_nhi - thunderbolt native host interface
+ * @lock: Must be held during ring creation/destruction. Is acquired by
+ * interrupt_work when dispatching interrupts to individual rings.
+ * @pdev: Pointer to the PCI device
+ * @iobase: MMIO space of the NHI
+ * @tx_rings: All Tx rings available on this host controller
+ * @rx_rings: All Rx rings available on this host controller
+ * @msix_ida: Used to allocate MSI-X vectors for rings
+ * @going_away: The host controller device is about to disappear so when
+ * this flag is set, avoid touching the hardware anymore.
+ * @interrupt_work: Work scheduled to handle ring interrupt when no
+ * MSI-X is used.
+ * @hop_count: Number of rings (end point hops) supported by NHI.
*/
struct tb_nhi {
- struct mutex lock; /*
- * Must be held during ring creation/destruction.
- * Is acquired by interrupt_work when dispatching
- * interrupts to individual rings.
- **/
+ struct mutex lock;
struct pci_dev *pdev;
void __iomem *iobase;
struct tb_ring **tx_rings;
struct tb_ring **rx_rings;
+ struct ida msix_ida;
+ bool going_away;
struct work_struct interrupt_work;
- u32 hop_count; /* Number of rings (end point hops) supported by NHI. */
+ u32 hop_count;
};
/**
* struct tb_ring - thunderbolt TX or RX ring associated with a NHI
+ * @lock: Lock serializing actions to this ring. Must be acquired after
+ * nhi->lock.
+ * @nhi: Pointer to the native host controller interface
+ * @size: Size of the ring
+ * @hop: Hop (DMA channel) associated with this ring
+ * @head: Head of the ring (write next descriptor here)
+ * @tail: Tail of the ring (complete next descriptor here)
+ * @descriptors: Allocated descriptors for this ring
+ * @queue: Queue holding frames to be transferred over this ring
+ * @in_flight: Queue holding frames that are currently in flight
+ * @work: Interrupt work structure
+ * @is_tx: Is the ring Tx or Rx
+ * @running: Is the ring running
+ * @irq: MSI-X irq number if the ring uses MSI-X. %0 otherwise.
+ * @vector: MSI-X vector number the ring uses (only set if @irq is > 0)
+ * @flags: Ring specific flags
*/
struct tb_ring {
- struct mutex lock; /* must be acquired after nhi->lock */
+ struct mutex lock;
struct tb_nhi *nhi;
int size;
int hop;
- int head; /* write next descriptor here */
- int tail; /* complete next descriptor here */
+ int head;
+ int tail;
struct ring_desc *descriptors;
dma_addr_t descriptors_dma;
struct list_head queue;
struct list_head in_flight;
struct work_struct work;
- bool is_tx:1; /* rx otherwise */
+ bool is_tx:1;
bool running:1;
+ int irq;
+ u8 vector;
+ unsigned int flags;
};
+/* Leave ring interrupt enabled on suspend */
+#define RING_FLAG_NO_SUSPEND BIT(0)
+
struct ring_frame;
typedef void (*ring_cb)(struct tb_ring*, struct ring_frame*, bool canceled);
#define TB_FRAME_SIZE 0x100 /* minimum size for ring_rx */
-struct tb_ring *ring_alloc_tx(struct tb_nhi *nhi, int hop, int size);
-struct tb_ring *ring_alloc_rx(struct tb_nhi *nhi, int hop, int size);
+struct tb_ring *ring_alloc_tx(struct tb_nhi *nhi, int hop, int size,
+ unsigned int flags);
+struct tb_ring *ring_alloc_rx(struct tb_nhi *nhi, int hop, int size,
+ unsigned int flags);
void ring_start(struct tb_ring *ring);
void ring_stop(struct tb_ring *ring);
void ring_free(struct tb_ring *ring);
return __ring_enqueue(ring, frame);
}
+enum nhi_fw_mode {
+ NHI_FW_SAFE_MODE,
+ NHI_FW_AUTH_MODE,
+ NHI_FW_EP_MODE,
+ NHI_FW_CM_MODE,
+};
+
+enum nhi_mailbox_cmd {
+ NHI_MAILBOX_SAVE_DEVS = 0x05,
+ NHI_MAILBOX_DISCONNECT_PCIE_PATHS = 0x06,
+ NHI_MAILBOX_DRV_UNLOADS = 0x07,
+ NHI_MAILBOX_ALLOW_ALL_DEVS = 0x23,
+};
+
+int nhi_mailbox_cmd(struct tb_nhi *nhi, enum nhi_mailbox_cmd cmd, u32 data);
+enum nhi_fw_mode nhi_mailbox_mode(struct tb_nhi *nhi);
+
+/*
+ * PCI IDs used in this driver from Win Ridge forward. There is no
+ * need for the PCI quirk anymore as we will use ICM also on Apple
+ * hardware.
+ */
+#define PCI_DEVICE_ID_INTEL_WIN_RIDGE_2C_NHI 0x157d
+#define PCI_DEVICE_ID_INTEL_WIN_RIDGE_2C_BRIDGE 0x157e
+#define PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_LP_NHI 0x15bf
+#define PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_LP_BRIDGE 0x15c0
+#define PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_4C_NHI 0x15d2
+#define PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_4C_BRIDGE 0x15d3
+#define PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_2C_NHI 0x15d9
+#define PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_2C_BRIDGE 0x15da
+#define PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_LP_USBONLY_NHI 0x15dc
+#define PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_USBONLY_NHI 0x15dd
+#define PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_USBONLY_NHI 0x15de
+
#endif
#define REG_RING_INTERRUPT_BASE 0x38200
#define RING_INTERRUPT_REG_COUNT(nhi) ((31 + 2 * nhi->hop_count) / 32)
+/* Interrupt Vector Allocation */
+#define REG_INT_VEC_ALLOC_BASE 0x38c40
+#define REG_INT_VEC_ALLOC_BITS 4
+#define REG_INT_VEC_ALLOC_MASK GENMASK(3, 0)
+#define REG_INT_VEC_ALLOC_REGS (32 / REG_INT_VEC_ALLOC_BITS)
+
/* The last 11 bits contain the number of hops supported by the NHI port. */
#define REG_HOP_COUNT 0x39640
+#define REG_DMA_MISC 0x39864
+#define REG_DMA_MISC_INT_AUTO_CLEAR BIT(2)
+
+#define REG_INMAIL_DATA 0x39900
+
+#define REG_INMAIL_CMD 0x39904
+#define REG_INMAIL_CMD_MASK GENMASK(7, 0)
+#define REG_INMAIL_ERROR BIT(30)
+#define REG_INMAIL_OP_REQUEST BIT(31)
+
+#define REG_OUTMAIL_CMD 0x3990c
+#define REG_OUTMAIL_CMD_OPMODE_SHIFT 8
+#define REG_OUTMAIL_CMD_OPMODE_MASK GENMASK(11, 8)
+
+#define REG_FW_STS 0x39944
+#define REG_FW_STS_NVM_AUTH_DONE BIT(31)
+#define REG_FW_STS_CIO_RESET_REQ BIT(30)
+#define REG_FW_STS_ICM_EN_CPU BIT(2)
+#define REG_FW_STS_ICM_EN_INVERT BIT(1)
+#define REG_FW_STS_ICM_EN BIT(0)
+
#endif
*/
#include <linux/delay.h>
+#include <linux/idr.h>
+#include <linux/nvmem-provider.h>
+#include <linux/sizes.h>
#include <linux/slab.h>
+#include <linux/vmalloc.h>
#include "tb.h"
+/* Switch authorization from userspace is serialized by this lock */
+static DEFINE_MUTEX(switch_lock);
+
+/* Switch NVM support */
+
+#define NVM_DEVID 0x05
+#define NVM_VERSION 0x08
+#define NVM_CSS 0x10
+#define NVM_FLASH_SIZE 0x45
+
+#define NVM_MIN_SIZE SZ_32K
+#define NVM_MAX_SIZE SZ_512K
+
+static DEFINE_IDA(nvm_ida);
+
+struct nvm_auth_status {
+ struct list_head list;
+ uuid_be uuid;
+ u32 status;
+};
+
+/*
+ * Hold NVM authentication failure status per switch This information
+ * needs to stay around even when the switch gets power cycled so we
+ * keep it separately.
+ */
+static LIST_HEAD(nvm_auth_status_cache);
+static DEFINE_MUTEX(nvm_auth_status_lock);
+
+static struct nvm_auth_status *__nvm_get_auth_status(const struct tb_switch *sw)
+{
+ struct nvm_auth_status *st;
+
+ list_for_each_entry(st, &nvm_auth_status_cache, list) {
+ if (!uuid_be_cmp(st->uuid, *sw->uuid))
+ return st;
+ }
+
+ return NULL;
+}
+
+static void nvm_get_auth_status(const struct tb_switch *sw, u32 *status)
+{
+ struct nvm_auth_status *st;
+
+ mutex_lock(&nvm_auth_status_lock);
+ st = __nvm_get_auth_status(sw);
+ mutex_unlock(&nvm_auth_status_lock);
+
+ *status = st ? st->status : 0;
+}
+
+static void nvm_set_auth_status(const struct tb_switch *sw, u32 status)
+{
+ struct nvm_auth_status *st;
+
+ if (WARN_ON(!sw->uuid))
+ return;
+
+ mutex_lock(&nvm_auth_status_lock);
+ st = __nvm_get_auth_status(sw);
+
+ if (!st) {
+ st = kzalloc(sizeof(*st), GFP_KERNEL);
+ if (!st)
+ goto unlock;
+
+ memcpy(&st->uuid, sw->uuid, sizeof(st->uuid));
+ INIT_LIST_HEAD(&st->list);
+ list_add_tail(&st->list, &nvm_auth_status_cache);
+ }
+
+ st->status = status;
+unlock:
+ mutex_unlock(&nvm_auth_status_lock);
+}
+
+static void nvm_clear_auth_status(const struct tb_switch *sw)
+{
+ struct nvm_auth_status *st;
+
+ mutex_lock(&nvm_auth_status_lock);
+ st = __nvm_get_auth_status(sw);
+ if (st) {
+ list_del(&st->list);
+ kfree(st);
+ }
+ mutex_unlock(&nvm_auth_status_lock);
+}
+
+static int nvm_validate_and_write(struct tb_switch *sw)
+{
+ unsigned int image_size, hdr_size;
+ const u8 *buf = sw->nvm->buf;
+ u16 ds_size;
+ int ret;
+
+ if (!buf)
+ return -EINVAL;
+
+ image_size = sw->nvm->buf_data_size;
+ if (image_size < NVM_MIN_SIZE || image_size > NVM_MAX_SIZE)
+ return -EINVAL;
+
+ /*
+ * FARB pointer must point inside the image and must at least
+ * contain parts of the digital section we will be reading here.
+ */
+ hdr_size = (*(u32 *)buf) & 0xffffff;
+ if (hdr_size + NVM_DEVID + 2 >= image_size)
+ return -EINVAL;
+
+ /* Digital section start should be aligned to 4k page */
+ if (!IS_ALIGNED(hdr_size, SZ_4K))
+ return -EINVAL;
+
+ /*
+ * Read digital section size and check that it also fits inside
+ * the image.
+ */
+ ds_size = *(u16 *)(buf + hdr_size);
+ if (ds_size >= image_size)
+ return -EINVAL;
+
+ if (!sw->safe_mode) {
+ u16 device_id;
+
+ /*
+ * Make sure the device ID in the image matches the one
+ * we read from the switch config space.
+ */
+ device_id = *(u16 *)(buf + hdr_size + NVM_DEVID);
+ if (device_id != sw->config.device_id)
+ return -EINVAL;
+
+ if (sw->generation < 3) {
+ /* Write CSS headers first */
+ ret = dma_port_flash_write(sw->dma_port,
+ DMA_PORT_CSS_ADDRESS, buf + NVM_CSS,
+ DMA_PORT_CSS_MAX_SIZE);
+ if (ret)
+ return ret;
+ }
+
+ /* Skip headers in the image */
+ buf += hdr_size;
+ image_size -= hdr_size;
+ }
+
+ return dma_port_flash_write(sw->dma_port, 0, buf, image_size);
+}
+
+static int nvm_authenticate_host(struct tb_switch *sw)
+{
+ int ret;
+
+ /*
+ * Root switch NVM upgrade requires that we disconnect the
+ * existing PCIe paths first (in case it is not in safe mode
+ * already).
+ */
+ if (!sw->safe_mode) {
+ ret = tb_domain_disconnect_pcie_paths(sw->tb);
+ if (ret)
+ return ret;
+ /*
+ * The host controller goes away pretty soon after this if
+ * everything goes well so getting timeout is expected.
+ */
+ ret = dma_port_flash_update_auth(sw->dma_port);
+ return ret == -ETIMEDOUT ? 0 : ret;
+ }
+
+ /*
+ * From safe mode we can get out by just power cycling the
+ * switch.
+ */
+ dma_port_power_cycle(sw->dma_port);
+ return 0;
+}
+
+static int nvm_authenticate_device(struct tb_switch *sw)
+{
+ int ret, retries = 10;
+
+ ret = dma_port_flash_update_auth(sw->dma_port);
+ if (ret && ret != -ETIMEDOUT)
+ return ret;
+
+ /*
+ * Poll here for the authentication status. It takes some time
+ * for the device to respond (we get timeout for a while). Once
+ * we get response the device needs to be power cycled in order
+ * to the new NVM to be taken into use.
+ */
+ do {
+ u32 status;
+
+ ret = dma_port_flash_update_auth_status(sw->dma_port, &status);
+ if (ret < 0 && ret != -ETIMEDOUT)
+ return ret;
+ if (ret > 0) {
+ if (status) {
+ tb_sw_warn(sw, "failed to authenticate NVM\n");
+ nvm_set_auth_status(sw, status);
+ }
+
+ tb_sw_info(sw, "power cycling the switch now\n");
+ dma_port_power_cycle(sw->dma_port);
+ return 0;
+ }
+
+ msleep(500);
+ } while (--retries);
+
+ return -ETIMEDOUT;
+}
+
+static int tb_switch_nvm_read(void *priv, unsigned int offset, void *val,
+ size_t bytes)
+{
+ struct tb_switch *sw = priv;
+
+ return dma_port_flash_read(sw->dma_port, offset, val, bytes);
+}
+
+static int tb_switch_nvm_write(void *priv, unsigned int offset, void *val,
+ size_t bytes)
+{
+ struct tb_switch *sw = priv;
+ int ret = 0;
+
+ if (mutex_lock_interruptible(&switch_lock))
+ return -ERESTARTSYS;
+
+ /*
+ * Since writing the NVM image might require some special steps,
+ * for example when CSS headers are written, we cache the image
+ * locally here and handle the special cases when the user asks
+ * us to authenticate the image.
+ */
+ if (!sw->nvm->buf) {
+ sw->nvm->buf = vmalloc(NVM_MAX_SIZE);
+ if (!sw->nvm->buf) {
+ ret = -ENOMEM;
+ goto unlock;
+ }
+ }
+
+ sw->nvm->buf_data_size = offset + bytes;
+ memcpy(sw->nvm->buf + offset, val, bytes);
+
+unlock:
+ mutex_unlock(&switch_lock);
+
+ return ret;
+}
+
+static struct nvmem_device *register_nvmem(struct tb_switch *sw, int id,
+ size_t size, bool active)
+{
+ struct nvmem_config config;
+
+ memset(&config, 0, sizeof(config));
+
+ if (active) {
+ config.name = "nvm_active";
+ config.reg_read = tb_switch_nvm_read;
+ } else {
+ config.name = "nvm_non_active";
+ config.reg_write = tb_switch_nvm_write;
+ }
+
+ config.id = id;
+ config.stride = 4;
+ config.word_size = 4;
+ config.size = size;
+ config.dev = &sw->dev;
+ config.owner = THIS_MODULE;
+ config.root_only = true;
+ config.priv = sw;
+
+ return nvmem_register(&config);
+}
+
+static int tb_switch_nvm_add(struct tb_switch *sw)
+{
+ struct nvmem_device *nvm_dev;
+ struct tb_switch_nvm *nvm;
+ u32 val;
+ int ret;
+
+ if (!sw->dma_port)
+ return 0;
+
+ nvm = kzalloc(sizeof(*nvm), GFP_KERNEL);
+ if (!nvm)
+ return -ENOMEM;
+
+ nvm->id = ida_simple_get(&nvm_ida, 0, 0, GFP_KERNEL);
+
+ /*
+ * If the switch is in safe-mode the only accessible portion of
+ * the NVM is the non-active one where userspace is expected to
+ * write new functional NVM.
+ */
+ if (!sw->safe_mode) {
+ u32 nvm_size, hdr_size;
+
+ ret = dma_port_flash_read(sw->dma_port, NVM_FLASH_SIZE, &val,
+ sizeof(val));
+ if (ret)
+ goto err_ida;
+
+ hdr_size = sw->generation < 3 ? SZ_8K : SZ_16K;
+ nvm_size = (SZ_1M << (val & 7)) / 8;
+ nvm_size = (nvm_size - hdr_size) / 2;
+
+ ret = dma_port_flash_read(sw->dma_port, NVM_VERSION, &val,
+ sizeof(val));
+ if (ret)
+ goto err_ida;
+
+ nvm->major = val >> 16;
+ nvm->minor = val >> 8;
+
+ nvm_dev = register_nvmem(sw, nvm->id, nvm_size, true);
+ if (IS_ERR(nvm_dev)) {
+ ret = PTR_ERR(nvm_dev);
+ goto err_ida;
+ }
+ nvm->active = nvm_dev;
+ }
+
+ nvm_dev = register_nvmem(sw, nvm->id, NVM_MAX_SIZE, false);
+ if (IS_ERR(nvm_dev)) {
+ ret = PTR_ERR(nvm_dev);
+ goto err_nvm_active;
+ }
+ nvm->non_active = nvm_dev;
+
+ mutex_lock(&switch_lock);
+ sw->nvm = nvm;
+ mutex_unlock(&switch_lock);
+
+ return 0;
+
+err_nvm_active:
+ if (nvm->active)
+ nvmem_unregister(nvm->active);
+err_ida:
+ ida_simple_remove(&nvm_ida, nvm->id);
+ kfree(nvm);
+
+ return ret;
+}
+
+static void tb_switch_nvm_remove(struct tb_switch *sw)
+{
+ struct tb_switch_nvm *nvm;
+
+ mutex_lock(&switch_lock);
+ nvm = sw->nvm;
+ sw->nvm = NULL;
+ mutex_unlock(&switch_lock);
+
+ if (!nvm)
+ return;
+
+ /* Remove authentication status in case the switch is unplugged */
+ if (!nvm->authenticating)
+ nvm_clear_auth_status(sw);
+
+ nvmem_unregister(nvm->non_active);
+ if (nvm->active)
+ nvmem_unregister(nvm->active);
+ ida_simple_remove(&nvm_ida, nvm->id);
+ vfree(nvm->buf);
+ kfree(nvm);
+}
+
/* port utility functions */
static const char *tb_port_type(struct tb_regs_port_header *port)
/* Port 0 is the switch itself and has no PHY. */
if (port->config.type == TB_TYPE_PORT && port->port != 0) {
- cap = tb_find_cap(port, TB_CFG_PORT, TB_CAP_PHY);
+ cap = tb_port_find_cap(port, TB_PORT_CAP_PHY);
if (cap > 0)
port->cap_phy = cap;
u32 data;
int res;
+ if (!sw->config.enabled)
+ return 0;
+
sw->config.plug_events_delay = 0xff;
res = tb_sw_write(sw, ((u32 *) &sw->config) + 4, TB_CFG_SWITCH, 4, 1);
if (res)
sw->cap_plug_events + 1, 1);
}
+static ssize_t authorized_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct tb_switch *sw = tb_to_switch(dev);
-/**
- * tb_switch_free() - free a tb_switch and all downstream switches
- */
-void tb_switch_free(struct tb_switch *sw)
+ return sprintf(buf, "%u\n", sw->authorized);
+}
+
+static int tb_switch_set_authorized(struct tb_switch *sw, unsigned int val)
{
- int i;
- /* port 0 is the switch itself and never has a remote */
- for (i = 1; i <= sw->config.max_port_number; i++) {
- if (tb_is_upstream_port(&sw->ports[i]))
- continue;
- if (sw->ports[i].remote)
- tb_switch_free(sw->ports[i].remote->sw);
- sw->ports[i].remote = NULL;
+ int ret = -EINVAL;
+
+ if (mutex_lock_interruptible(&switch_lock))
+ return -ERESTARTSYS;
+
+ if (sw->authorized)
+ goto unlock;
+
+ switch (val) {
+ /* Approve switch */
+ case 1:
+ if (sw->key)
+ ret = tb_domain_approve_switch_key(sw->tb, sw);
+ else
+ ret = tb_domain_approve_switch(sw->tb, sw);
+ break;
+
+ /* Challenge switch */
+ case 2:
+ if (sw->key)
+ ret = tb_domain_challenge_switch_key(sw->tb, sw);
+ break;
+
+ default:
+ break;
}
- if (!sw->is_unplugged)
- tb_plug_events_active(sw, false);
+ if (!ret) {
+ sw->authorized = val;
+ /* Notify status change to the userspace */
+ kobject_uevent(&sw->dev.kobj, KOBJ_CHANGE);
+ }
+
+unlock:
+ mutex_unlock(&switch_lock);
+ return ret;
+}
+
+static ssize_t authorized_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct tb_switch *sw = tb_to_switch(dev);
+ unsigned int val;
+ ssize_t ret;
+
+ ret = kstrtouint(buf, 0, &val);
+ if (ret)
+ return ret;
+ if (val > 2)
+ return -EINVAL;
+
+ ret = tb_switch_set_authorized(sw, val);
+
+ return ret ? ret : count;
+}
+static DEVICE_ATTR_RW(authorized);
+
+static ssize_t device_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct tb_switch *sw = tb_to_switch(dev);
+
+ return sprintf(buf, "%#x\n", sw->device);
+}
+static DEVICE_ATTR_RO(device);
+
+static ssize_t
+device_name_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct tb_switch *sw = tb_to_switch(dev);
+
+ return sprintf(buf, "%s\n", sw->device_name ? sw->device_name : "");
+}
+static DEVICE_ATTR_RO(device_name);
+
+static ssize_t key_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct tb_switch *sw = tb_to_switch(dev);
+ ssize_t ret;
+
+ if (mutex_lock_interruptible(&switch_lock))
+ return -ERESTARTSYS;
+
+ if (sw->key)
+ ret = sprintf(buf, "%*phN\n", TB_SWITCH_KEY_SIZE, sw->key);
+ else
+ ret = sprintf(buf, "\n");
+
+ mutex_unlock(&switch_lock);
+ return ret;
+}
+
+static ssize_t key_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct tb_switch *sw = tb_to_switch(dev);
+ u8 key[TB_SWITCH_KEY_SIZE];
+ ssize_t ret = count;
+
+ if (count < 64)
+ return -EINVAL;
+ if (hex2bin(key, buf, sizeof(key)))
+ return -EINVAL;
+
+ if (mutex_lock_interruptible(&switch_lock))
+ return -ERESTARTSYS;
+
+ if (sw->authorized) {
+ ret = -EBUSY;
+ } else {
+ kfree(sw->key);
+ sw->key = kmemdup(key, sizeof(key), GFP_KERNEL);
+ if (!sw->key)
+ ret = -ENOMEM;
+ }
+
+ mutex_unlock(&switch_lock);
+ return ret;
+}
+static DEVICE_ATTR_RW(key);
+
+static ssize_t nvm_authenticate_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct tb_switch *sw = tb_to_switch(dev);
+ u32 status;
+
+ nvm_get_auth_status(sw, &status);
+ return sprintf(buf, "%#x\n", status);
+}
+
+static ssize_t nvm_authenticate_store(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t count)
+{
+ struct tb_switch *sw = tb_to_switch(dev);
+ bool val;
+ int ret;
+
+ if (mutex_lock_interruptible(&switch_lock))
+ return -ERESTARTSYS;
+
+ /* If NVMem devices are not yet added */
+ if (!sw->nvm) {
+ ret = -EAGAIN;
+ goto exit_unlock;
+ }
+
+ ret = kstrtobool(buf, &val);
+ if (ret)
+ goto exit_unlock;
+
+ /* Always clear the authentication status */
+ nvm_clear_auth_status(sw);
+
+ if (val) {
+ ret = nvm_validate_and_write(sw);
+ if (ret)
+ goto exit_unlock;
+
+ sw->nvm->authenticating = true;
+
+ if (!tb_route(sw))
+ ret = nvm_authenticate_host(sw);
+ else
+ ret = nvm_authenticate_device(sw);
+ }
+
+exit_unlock:
+ mutex_unlock(&switch_lock);
+
+ if (ret)
+ return ret;
+ return count;
+}
+static DEVICE_ATTR_RW(nvm_authenticate);
+
+static ssize_t nvm_version_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct tb_switch *sw = tb_to_switch(dev);
+ int ret;
+
+ if (mutex_lock_interruptible(&switch_lock))
+ return -ERESTARTSYS;
+
+ if (sw->safe_mode)
+ ret = -ENODATA;
+ else if (!sw->nvm)
+ ret = -EAGAIN;
+ else
+ ret = sprintf(buf, "%x.%x\n", sw->nvm->major, sw->nvm->minor);
+
+ mutex_unlock(&switch_lock);
+
+ return ret;
+}
+static DEVICE_ATTR_RO(nvm_version);
+
+static ssize_t vendor_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct tb_switch *sw = tb_to_switch(dev);
+
+ return sprintf(buf, "%#x\n", sw->vendor);
+}
+static DEVICE_ATTR_RO(vendor);
+
+static ssize_t
+vendor_name_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct tb_switch *sw = tb_to_switch(dev);
+
+ return sprintf(buf, "%s\n", sw->vendor_name ? sw->vendor_name : "");
+}
+static DEVICE_ATTR_RO(vendor_name);
+
+static ssize_t unique_id_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct tb_switch *sw = tb_to_switch(dev);
+
+ return sprintf(buf, "%pUb\n", sw->uuid);
+}
+static DEVICE_ATTR_RO(unique_id);
+
+static struct attribute *switch_attrs[] = {
+ &dev_attr_authorized.attr,
+ &dev_attr_device.attr,
+ &dev_attr_device_name.attr,
+ &dev_attr_key.attr,
+ &dev_attr_nvm_authenticate.attr,
+ &dev_attr_nvm_version.attr,
+ &dev_attr_vendor.attr,
+ &dev_attr_vendor_name.attr,
+ &dev_attr_unique_id.attr,
+ NULL,
+};
+
+static umode_t switch_attr_is_visible(struct kobject *kobj,
+ struct attribute *attr, int n)
+{
+ struct device *dev = container_of(kobj, struct device, kobj);
+ struct tb_switch *sw = tb_to_switch(dev);
+
+ if (attr == &dev_attr_key.attr) {
+ if (tb_route(sw) &&
+ sw->tb->security_level == TB_SECURITY_SECURE &&
+ sw->security_level == TB_SECURITY_SECURE)
+ return attr->mode;
+ return 0;
+ } else if (attr == &dev_attr_nvm_authenticate.attr ||
+ attr == &dev_attr_nvm_version.attr) {
+ if (sw->dma_port)
+ return attr->mode;
+ return 0;
+ }
+
+ return sw->safe_mode ? 0 : attr->mode;
+}
+
+static struct attribute_group switch_group = {
+ .is_visible = switch_attr_is_visible,
+ .attrs = switch_attrs,
+};
+
+static const struct attribute_group *switch_groups[] = {
+ &switch_group,
+ NULL,
+};
+
+static void tb_switch_release(struct device *dev)
+{
+ struct tb_switch *sw = tb_to_switch(dev);
+
+ dma_port_free(sw->dma_port);
+
+ kfree(sw->uuid);
+ kfree(sw->device_name);
+ kfree(sw->vendor_name);
kfree(sw->ports);
kfree(sw->drom);
+ kfree(sw->key);
kfree(sw);
}
+struct device_type tb_switch_type = {
+ .name = "thunderbolt_device",
+ .release = tb_switch_release,
+};
+
+static int tb_switch_get_generation(struct tb_switch *sw)
+{
+ switch (sw->config.device_id) {
+ case PCI_DEVICE_ID_INTEL_LIGHT_RIDGE:
+ case PCI_DEVICE_ID_INTEL_EAGLE_RIDGE:
+ case PCI_DEVICE_ID_INTEL_LIGHT_PEAK:
+ case PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_2C:
+ case PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_4C:
+ case PCI_DEVICE_ID_INTEL_PORT_RIDGE:
+ case PCI_DEVICE_ID_INTEL_REDWOOD_RIDGE_2C_BRIDGE:
+ case PCI_DEVICE_ID_INTEL_REDWOOD_RIDGE_4C_BRIDGE:
+ return 1;
+
+ case PCI_DEVICE_ID_INTEL_WIN_RIDGE_2C_BRIDGE:
+ case PCI_DEVICE_ID_INTEL_FALCON_RIDGE_2C_BRIDGE:
+ case PCI_DEVICE_ID_INTEL_FALCON_RIDGE_4C_BRIDGE:
+ return 2;
+
+ case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_LP_BRIDGE:
+ case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_2C_BRIDGE:
+ case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_4C_BRIDGE:
+ case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_2C_BRIDGE:
+ case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_4C_BRIDGE:
+ return 3;
+
+ default:
+ /*
+ * For unknown switches assume generation to be 1 to be
+ * on the safe side.
+ */
+ tb_sw_warn(sw, "unsupported switch device id %#x\n",
+ sw->config.device_id);
+ return 1;
+ }
+}
+
/**
- * tb_switch_alloc() - allocate and initialize a switch
+ * tb_switch_alloc() - allocate a switch
+ * @tb: Pointer to the owning domain
+ * @parent: Parent device for this switch
+ * @route: Route string for this switch
*
- * Return: Returns a NULL on failure.
+ * Allocates and initializes a switch. Will not upload configuration to
+ * the switch. For that you need to call tb_switch_configure()
+ * separately. The returned switch should be released by calling
+ * tb_switch_put().
+ *
+ * Return: Pointer to the allocated switch or %NULL in case of failure
*/
-struct tb_switch *tb_switch_alloc(struct tb *tb, u64 route)
+struct tb_switch *tb_switch_alloc(struct tb *tb, struct device *parent,
+ u64 route)
{
int i;
int cap;
sw->tb = tb;
if (tb_cfg_read(tb->ctl, &sw->config, route, 0, TB_CFG_SWITCH, 0, 5))
- goto err;
- tb_info(tb,
- "initializing Switch at %#llx (depth: %d, up port: %d)\n",
- route, tb_route_length(route), upstream_port);
- tb_info(tb, "old switch config:\n");
+ goto err_free_sw_ports;
+
+ tb_info(tb, "current switch config:\n");
tb_dump_switch(tb, &sw->config);
/* configure switch */
sw->config.depth = tb_route_length(route);
sw->config.route_lo = route;
sw->config.route_hi = route >> 32;
- sw->config.enabled = 1;
- /* from here on we may use the tb_sw_* functions & macros */
-
- if (sw->config.vendor_id != 0x8086)
- tb_sw_warn(sw, "unknown switch vendor id %#x\n",
- sw->config.vendor_id);
-
- if (sw->config.device_id != PCI_DEVICE_ID_INTEL_LIGHT_RIDGE &&
- sw->config.device_id != PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_4C &&
- sw->config.device_id != PCI_DEVICE_ID_INTEL_PORT_RIDGE &&
- sw->config.device_id != PCI_DEVICE_ID_INTEL_FALCON_RIDGE_2C_BRIDGE &&
- sw->config.device_id != PCI_DEVICE_ID_INTEL_FALCON_RIDGE_4C_BRIDGE)
- tb_sw_warn(sw, "unsupported switch device id %#x\n",
- sw->config.device_id);
-
- /* upload configuration */
- if (tb_sw_write(sw, 1 + (u32 *) &sw->config, TB_CFG_SWITCH, 1, 3))
- goto err;
+ sw->config.enabled = 0;
/* initialize ports */
sw->ports = kcalloc(sw->config.max_port_number + 1, sizeof(*sw->ports),
GFP_KERNEL);
if (!sw->ports)
- goto err;
+ goto err_free_sw_ports;
for (i = 0; i <= sw->config.max_port_number; i++) {
/* minimum setup for tb_find_cap and tb_drom_read to work */
sw->ports[i].port = i;
}
- cap = tb_find_cap(&sw->ports[0], TB_CFG_SWITCH, TB_CAP_PLUG_EVENTS);
+ sw->generation = tb_switch_get_generation(sw);
+
+ cap = tb_switch_find_vse_cap(sw, TB_VSE_CAP_PLUG_EVENTS);
if (cap < 0) {
- tb_sw_warn(sw, "cannot find TB_CAP_PLUG_EVENTS aborting\n");
- goto err;
+ tb_sw_warn(sw, "cannot find TB_VSE_CAP_PLUG_EVENTS aborting\n");
+ goto err_free_sw_ports;
}
sw->cap_plug_events = cap;
- /* read drom */
- if (tb_drom_read(sw))
- tb_sw_warn(sw, "tb_eeprom_read_rom failed, continuing\n");
- tb_sw_info(sw, "uid: %#llx\n", sw->uid);
-
- for (i = 0; i <= sw->config.max_port_number; i++) {
- if (sw->ports[i].disabled) {
- tb_port_info(&sw->ports[i], "disabled by eeprom\n");
- continue;
- }
- if (tb_init_port(&sw->ports[i]))
- goto err;
- }
-
- /* TODO: I2C, IECS, link controller */
+ /* Root switch is always authorized */
+ if (!route)
+ sw->authorized = true;
- if (tb_plug_events_active(sw, true))
- goto err;
+ device_initialize(&sw->dev);
+ sw->dev.parent = parent;
+ sw->dev.bus = &tb_bus_type;
+ sw->dev.type = &tb_switch_type;
+ sw->dev.groups = switch_groups;
+ dev_set_name(&sw->dev, "%u-%llx", tb->index, tb_route(sw));
return sw;
-err:
+
+err_free_sw_ports:
kfree(sw->ports);
- kfree(sw->drom);
kfree(sw);
+
return NULL;
}
+/**
+ * tb_switch_alloc_safe_mode() - allocate a switch that is in safe mode
+ * @tb: Pointer to the owning domain
+ * @parent: Parent device for this switch
+ * @route: Route string for this switch
+ *
+ * This creates a switch in safe mode. This means the switch pretty much
+ * lacks all capabilities except DMA configuration port before it is
+ * flashed with a valid NVM firmware.
+ *
+ * The returned switch must be released by calling tb_switch_put().
+ *
+ * Return: Pointer to the allocated switch or %NULL in case of failure
+ */
+struct tb_switch *
+tb_switch_alloc_safe_mode(struct tb *tb, struct device *parent, u64 route)
+{
+ struct tb_switch *sw;
+
+ sw = kzalloc(sizeof(*sw), GFP_KERNEL);
+ if (!sw)
+ return NULL;
+
+ sw->tb = tb;
+ sw->config.depth = tb_route_length(route);
+ sw->config.route_hi = upper_32_bits(route);
+ sw->config.route_lo = lower_32_bits(route);
+ sw->safe_mode = true;
+
+ device_initialize(&sw->dev);
+ sw->dev.parent = parent;
+ sw->dev.bus = &tb_bus_type;
+ sw->dev.type = &tb_switch_type;
+ sw->dev.groups = switch_groups;
+ dev_set_name(&sw->dev, "%u-%llx", tb->index, tb_route(sw));
+
+ return sw;
+}
+
+/**
+ * tb_switch_configure() - Uploads configuration to the switch
+ * @sw: Switch to configure
+ *
+ * Call this function before the switch is added to the system. It will
+ * upload configuration to the switch and makes it available for the
+ * connection manager to use.
+ *
+ * Return: %0 in case of success and negative errno in case of failure
+ */
+int tb_switch_configure(struct tb_switch *sw)
+{
+ struct tb *tb = sw->tb;
+ u64 route;
+ int ret;
+
+ route = tb_route(sw);
+ tb_info(tb,
+ "initializing Switch at %#llx (depth: %d, up port: %d)\n",
+ route, tb_route_length(route), sw->config.upstream_port_number);
+
+ if (sw->config.vendor_id != PCI_VENDOR_ID_INTEL)
+ tb_sw_warn(sw, "unknown switch vendor id %#x\n",
+ sw->config.vendor_id);
+
+ sw->config.enabled = 1;
+
+ /* upload configuration */
+ ret = tb_sw_write(sw, 1 + (u32 *)&sw->config, TB_CFG_SWITCH, 1, 3);
+ if (ret)
+ return ret;
+
+ return tb_plug_events_active(sw, true);
+}
+
+static void tb_switch_set_uuid(struct tb_switch *sw)
+{
+ u32 uuid[4];
+ int cap;
+
+ if (sw->uuid)
+ return;
+
+ /*
+ * The newer controllers include fused UUID as part of link
+ * controller specific registers
+ */
+ cap = tb_switch_find_vse_cap(sw, TB_VSE_CAP_LINK_CONTROLLER);
+ if (cap > 0) {
+ tb_sw_read(sw, uuid, TB_CFG_SWITCH, cap + 3, 4);
+ } else {
+ /*
+ * ICM generates UUID based on UID and fills the upper
+ * two words with ones. This is not strictly following
+ * UUID format but we want to be compatible with it so
+ * we do the same here.
+ */
+ uuid[0] = sw->uid & 0xffffffff;
+ uuid[1] = (sw->uid >> 32) & 0xffffffff;
+ uuid[2] = 0xffffffff;
+ uuid[3] = 0xffffffff;
+ }
+
+ sw->uuid = kmemdup(uuid, sizeof(uuid), GFP_KERNEL);
+}
+
+static int tb_switch_add_dma_port(struct tb_switch *sw)
+{
+ u32 status;
+ int ret;
+
+ switch (sw->generation) {
+ case 3:
+ break;
+
+ case 2:
+ /* Only root switch can be upgraded */
+ if (tb_route(sw))
+ return 0;
+ break;
+
+ default:
+ /*
+ * DMA port is the only thing available when the switch
+ * is in safe mode.
+ */
+ if (!sw->safe_mode)
+ return 0;
+ break;
+ }
+
+ if (sw->no_nvm_upgrade)
+ return 0;
+
+ sw->dma_port = dma_port_alloc(sw);
+ if (!sw->dma_port)
+ return 0;
+
+ /*
+ * Check status of the previous flash authentication. If there
+ * is one we need to power cycle the switch in any case to make
+ * it functional again.
+ */
+ ret = dma_port_flash_update_auth_status(sw->dma_port, &status);
+ if (ret <= 0)
+ return ret;
+
+ if (status) {
+ tb_sw_info(sw, "switch flash authentication failed\n");
+ tb_switch_set_uuid(sw);
+ nvm_set_auth_status(sw, status);
+ }
+
+ tb_sw_info(sw, "power cycling the switch now\n");
+ dma_port_power_cycle(sw->dma_port);
+
+ /*
+ * We return error here which causes the switch adding failure.
+ * It should appear back after power cycle is complete.
+ */
+ return -ESHUTDOWN;
+}
+
+/**
+ * tb_switch_add() - Add a switch to the domain
+ * @sw: Switch to add
+ *
+ * This is the last step in adding switch to the domain. It will read
+ * identification information from DROM and initializes ports so that
+ * they can be used to connect other switches. The switch will be
+ * exposed to the userspace when this function successfully returns. To
+ * remove and release the switch, call tb_switch_remove().
+ *
+ * Return: %0 in case of success and negative errno in case of failure
+ */
+int tb_switch_add(struct tb_switch *sw)
+{
+ int i, ret;
+
+ /*
+ * Initialize DMA control port now before we read DROM. Recent
+ * host controllers have more complete DROM on NVM that includes
+ * vendor and model identification strings which we then expose
+ * to the userspace. NVM can be accessed through DMA
+ * configuration based mailbox.
+ */
+ ret = tb_switch_add_dma_port(sw);
+ if (ret)
+ return ret;
+
+ if (!sw->safe_mode) {
+ /* read drom */
+ ret = tb_drom_read(sw);
+ if (ret) {
+ tb_sw_warn(sw, "tb_eeprom_read_rom failed\n");
+ return ret;
+ }
+ tb_sw_info(sw, "uid: %#llx\n", sw->uid);
+
+ tb_switch_set_uuid(sw);
+
+ for (i = 0; i <= sw->config.max_port_number; i++) {
+ if (sw->ports[i].disabled) {
+ tb_port_info(&sw->ports[i], "disabled by eeprom\n");
+ continue;
+ }
+ ret = tb_init_port(&sw->ports[i]);
+ if (ret)
+ return ret;
+ }
+ }
+
+ ret = device_add(&sw->dev);
+ if (ret)
+ return ret;
+
+ ret = tb_switch_nvm_add(sw);
+ if (ret)
+ device_del(&sw->dev);
+
+ return ret;
+}
+
+/**
+ * tb_switch_remove() - Remove and release a switch
+ * @sw: Switch to remove
+ *
+ * This will remove the switch from the domain and release it after last
+ * reference count drops to zero. If there are switches connected below
+ * this switch, they will be removed as well.
+ */
+void tb_switch_remove(struct tb_switch *sw)
+{
+ int i;
+
+ /* port 0 is the switch itself and never has a remote */
+ for (i = 1; i <= sw->config.max_port_number; i++) {
+ if (tb_is_upstream_port(&sw->ports[i]))
+ continue;
+ if (sw->ports[i].remote)
+ tb_switch_remove(sw->ports[i].remote->sw);
+ sw->ports[i].remote = NULL;
+ }
+
+ if (!sw->is_unplugged)
+ tb_plug_events_active(sw, false);
+
+ tb_switch_nvm_remove(sw);
+ device_unregister(&sw->dev);
+}
+
/**
* tb_sw_set_unplugged() - set is_unplugged on switch and downstream switches
*/
int tb_switch_resume(struct tb_switch *sw)
{
int i, err;
- u64 uid;
tb_sw_info(sw, "resuming switch\n");
- err = tb_drom_read_uid_only(sw, &uid);
- if (err) {
- tb_sw_warn(sw, "uid read failed\n");
- return err;
- }
- if (sw != sw->tb->root_switch && sw->uid != uid) {
- tb_sw_info(sw,
- "changed while suspended (uid %#llx -> %#llx)\n",
- sw->uid, uid);
- return -ENODEV;
+ /*
+ * Check for UID of the connected switches except for root
+ * switch which we assume cannot be removed.
+ */
+ if (tb_route(sw)) {
+ u64 uid;
+
+ err = tb_drom_read_uid_only(sw, &uid);
+ if (err) {
+ tb_sw_warn(sw, "uid read failed\n");
+ return err;
+ }
+ if (sw->uid != uid) {
+ tb_sw_info(sw,
+ "changed while suspended (uid %#llx -> %#llx)\n",
+ sw->uid, uid);
+ return -ENODEV;
+ }
}
/* upload configuration */
* effect?
*/
}
+
+struct tb_sw_lookup {
+ struct tb *tb;
+ u8 link;
+ u8 depth;
+ const uuid_be *uuid;
+};
+
+static int tb_switch_match(struct device *dev, void *data)
+{
+ struct tb_switch *sw = tb_to_switch(dev);
+ struct tb_sw_lookup *lookup = data;
+
+ if (!sw)
+ return 0;
+ if (sw->tb != lookup->tb)
+ return 0;
+
+ if (lookup->uuid)
+ return !memcmp(sw->uuid, lookup->uuid, sizeof(*lookup->uuid));
+
+ /* Root switch is matched only by depth */
+ if (!lookup->depth)
+ return !sw->depth;
+
+ return sw->link == lookup->link && sw->depth == lookup->depth;
+}
+
+/**
+ * tb_switch_find_by_link_depth() - Find switch by link and depth
+ * @tb: Domain the switch belongs
+ * @link: Link number the switch is connected
+ * @depth: Depth of the switch in link
+ *
+ * Returned switch has reference count increased so the caller needs to
+ * call tb_switch_put() when done with the switch.
+ */
+struct tb_switch *tb_switch_find_by_link_depth(struct tb *tb, u8 link, u8 depth)
+{
+ struct tb_sw_lookup lookup;
+ struct device *dev;
+
+ memset(&lookup, 0, sizeof(lookup));
+ lookup.tb = tb;
+ lookup.link = link;
+ lookup.depth = depth;
+
+ dev = bus_find_device(&tb_bus_type, NULL, &lookup, tb_switch_match);
+ if (dev)
+ return tb_to_switch(dev);
+
+ return NULL;
+}
+
+/**
+ * tb_switch_find_by_link_depth() - Find switch by UUID
+ * @tb: Domain the switch belongs
+ * @uuid: UUID to look for
+ *
+ * Returned switch has reference count increased so the caller needs to
+ * call tb_switch_put() when done with the switch.
+ */
+struct tb_switch *tb_switch_find_by_uuid(struct tb *tb, const uuid_be *uuid)
+{
+ struct tb_sw_lookup lookup;
+ struct device *dev;
+
+ memset(&lookup, 0, sizeof(lookup));
+ lookup.tb = tb;
+ lookup.uuid = uuid;
+
+ dev = bus_find_device(&tb_bus_type, NULL, &lookup, tb_switch_match);
+ if (dev)
+ return tb_to_switch(dev);
+
+ return NULL;
+}
+
+void tb_switch_exit(void)
+{
+ ida_destroy(&nvm_ida);
+}
#include <linux/slab.h>
#include <linux/errno.h>
#include <linux/delay.h>
+#include <linux/dmi.h>
#include "tb.h"
#include "tb_regs.h"
#include "tunnel_pci.h"
+/**
+ * struct tb_cm - Simple Thunderbolt connection manager
+ * @tunnel_list: List of active tunnels
+ * @hotplug_active: tb_handle_hotplug will stop progressing plug
+ * events and exit if this is not set (it needs to
+ * acquire the lock one more time). Used to drain wq
+ * after cfg has been paused.
+ */
+struct tb_cm {
+ struct list_head tunnel_list;
+ bool hotplug_active;
+};
/* enumeration & hot plug handling */
tb_port_WARN(port, "port already has a remote!\n");
return;
}
- sw = tb_switch_alloc(port->sw->tb, tb_downstream_route(port));
+ sw = tb_switch_alloc(port->sw->tb, &port->sw->dev,
+ tb_downstream_route(port));
if (!sw)
return;
+
+ if (tb_switch_configure(sw)) {
+ tb_switch_put(sw);
+ return;
+ }
+
+ sw->authorized = true;
+
+ if (tb_switch_add(sw)) {
+ tb_switch_put(sw);
+ return;
+ }
+
port->remote = tb_upstream_port(sw);
tb_upstream_port(sw)->remote = port;
tb_scan_switch(sw);
*/
static void tb_free_invalid_tunnels(struct tb *tb)
{
+ struct tb_cm *tcm = tb_priv(tb);
struct tb_pci_tunnel *tunnel;
struct tb_pci_tunnel *n;
- list_for_each_entry_safe(tunnel, n, &tb->tunnel_list, list)
- {
+
+ list_for_each_entry_safe(tunnel, n, &tcm->tunnel_list, list) {
if (tb_pci_is_invalid(tunnel)) {
tb_pci_deactivate(tunnel);
+ list_del(&tunnel->list);
tb_pci_free(tunnel);
}
}
if (!port->remote)
continue;
if (port->remote->sw->is_unplugged) {
- tb_switch_free(port->remote->sw);
+ tb_switch_remove(port->remote->sw);
port->remote = NULL;
} else {
tb_free_unplugged_children(port->remote->sw);
continue;
if (sw->ports[i].config.type != TB_TYPE_PCIE_DOWN)
continue;
- cap = tb_find_cap(&sw->ports[i], TB_CFG_PORT, TB_CAP_PCIE);
- if (cap <= 0)
+ cap = tb_port_find_cap(&sw->ports[i], TB_PORT_CAP_ADAP);
+ if (cap < 0)
continue;
res = tb_port_read(&sw->ports[i], &data, TB_CFG_PORT, cap, 1);
if (res < 0)
struct tb_port *up_port;
struct tb_port *down_port;
struct tb_pci_tunnel *tunnel;
+ struct tb_cm *tcm = tb_priv(tb);
+
/* scan for pcie devices at depth 1*/
for (i = 1; i <= tb->root_switch->config.max_port_number; i++) {
if (tb_is_upstream_port(&tb->root_switch->ports[i]))
}
/* check whether port is already activated */
- cap = tb_find_cap(up_port, TB_CFG_PORT, TB_CAP_PCIE);
- if (cap <= 0)
+ cap = tb_port_find_cap(up_port, TB_PORT_CAP_ADAP);
+ if (cap < 0)
continue;
if (tb_port_read(up_port, &data, TB_CFG_PORT, cap, 1))
continue;
tb_pci_free(tunnel);
}
+ list_add(&tunnel->list, &tcm->tunnel_list);
}
}
{
struct tb_hotplug_event *ev = container_of(work, typeof(*ev), work);
struct tb *tb = ev->tb;
+ struct tb_cm *tcm = tb_priv(tb);
struct tb_switch *sw;
struct tb_port *port;
mutex_lock(&tb->lock);
- if (!tb->hotplug_active)
+ if (!tcm->hotplug_active)
goto out; /* during init, suspend or shutdown */
sw = get_switch_at_route(tb->root_switch, ev->route);
tb_port_info(port, "unplugged\n");
tb_sw_set_unplugged(port->remote->sw);
tb_free_invalid_tunnels(tb);
- tb_switch_free(port->remote->sw);
+ tb_switch_remove(port->remote->sw);
port->remote = NULL;
} else {
tb_port_info(port,
*
* Delegates to tb_handle_hotplug.
*/
-static void tb_schedule_hotplug_handler(void *data, u64 route, u8 port,
- bool unplug)
+static void tb_handle_event(struct tb *tb, enum tb_cfg_pkg_type type,
+ const void *buf, size_t size)
{
- struct tb *tb = data;
- struct tb_hotplug_event *ev = kmalloc(sizeof(*ev), GFP_KERNEL);
+ const struct cfg_event_pkg *pkg = buf;
+ struct tb_hotplug_event *ev;
+ u64 route;
+
+ if (type != TB_CFG_PKG_EVENT) {
+ tb_warn(tb, "unexpected event %#x, ignoring\n", type);
+ return;
+ }
+
+ route = tb_cfg_get_route(&pkg->header);
+
+ if (tb_cfg_error(tb->ctl, route, pkg->port,
+ TB_CFG_ERROR_ACK_PLUG_EVENT)) {
+ tb_warn(tb, "could not ack plug event on %llx:%x\n", route,
+ pkg->port);
+ }
+
+ ev = kmalloc(sizeof(*ev), GFP_KERNEL);
if (!ev)
return;
INIT_WORK(&ev->work, tb_handle_hotplug);
ev->tb = tb;
ev->route = route;
- ev->port = port;
- ev->unplug = unplug;
+ ev->port = pkg->port;
+ ev->unplug = pkg->unplug;
queue_work(tb->wq, &ev->work);
}
-/**
- * thunderbolt_shutdown_and_free() - shutdown everything
- *
- * Free all switches and the config channel.
- *
- * Used in the error path of thunderbolt_alloc_and_start.
- */
-void thunderbolt_shutdown_and_free(struct tb *tb)
+static void tb_stop(struct tb *tb)
{
+ struct tb_cm *tcm = tb_priv(tb);
struct tb_pci_tunnel *tunnel;
struct tb_pci_tunnel *n;
- mutex_lock(&tb->lock);
-
/* tunnels are only present after everything has been initialized */
- list_for_each_entry_safe(tunnel, n, &tb->tunnel_list, list) {
+ list_for_each_entry_safe(tunnel, n, &tcm->tunnel_list, list) {
tb_pci_deactivate(tunnel);
tb_pci_free(tunnel);
}
-
- if (tb->root_switch)
- tb_switch_free(tb->root_switch);
- tb->root_switch = NULL;
-
- if (tb->ctl) {
- tb_ctl_stop(tb->ctl);
- tb_ctl_free(tb->ctl);
- }
- tb->ctl = NULL;
- tb->hotplug_active = false; /* signal tb_handle_hotplug to quit */
-
- /* allow tb_handle_hotplug to acquire the lock */
- mutex_unlock(&tb->lock);
- if (tb->wq) {
- flush_workqueue(tb->wq);
- destroy_workqueue(tb->wq);
- tb->wq = NULL;
- }
- mutex_destroy(&tb->lock);
- kfree(tb);
+ tb_switch_remove(tb->root_switch);
+ tcm->hotplug_active = false; /* signal tb_handle_hotplug to quit */
}
-/**
- * thunderbolt_alloc_and_start() - setup the thunderbolt bus
- *
- * Allocates a tb_cfg control channel, initializes the root switch, enables
- * plug events and activates pci devices.
- *
- * Return: Returns NULL on error.
- */
-struct tb *thunderbolt_alloc_and_start(struct tb_nhi *nhi)
+static int tb_start(struct tb *tb)
{
- struct tb *tb;
-
- BUILD_BUG_ON(sizeof(struct tb_regs_switch_header) != 5 * 4);
- BUILD_BUG_ON(sizeof(struct tb_regs_port_header) != 8 * 4);
- BUILD_BUG_ON(sizeof(struct tb_regs_hop) != 2 * 4);
+ struct tb_cm *tcm = tb_priv(tb);
+ int ret;
- tb = kzalloc(sizeof(*tb), GFP_KERNEL);
- if (!tb)
- return NULL;
-
- tb->nhi = nhi;
- mutex_init(&tb->lock);
- mutex_lock(&tb->lock);
- INIT_LIST_HEAD(&tb->tunnel_list);
-
- tb->wq = alloc_ordered_workqueue("thunderbolt", 0);
- if (!tb->wq)
- goto err_locked;
+ tb->root_switch = tb_switch_alloc(tb, &tb->dev, 0);
+ if (!tb->root_switch)
+ return -ENOMEM;
- tb->ctl = tb_ctl_alloc(tb->nhi, tb_schedule_hotplug_handler, tb);
- if (!tb->ctl)
- goto err_locked;
/*
- * tb_schedule_hotplug_handler may be called as soon as the config
- * channel is started. Thats why we have to hold the lock here.
+ * ICM firmware upgrade needs running firmware and in native
+ * mode that is not available so disable firmware upgrade of the
+ * root switch.
*/
- tb_ctl_start(tb->ctl);
+ tb->root_switch->no_nvm_upgrade = true;
- tb->root_switch = tb_switch_alloc(tb, 0);
- if (!tb->root_switch)
- goto err_locked;
+ ret = tb_switch_configure(tb->root_switch);
+ if (ret) {
+ tb_switch_put(tb->root_switch);
+ return ret;
+ }
+
+ /* Announce the switch to the world */
+ ret = tb_switch_add(tb->root_switch);
+ if (ret) {
+ tb_switch_put(tb->root_switch);
+ return ret;
+ }
/* Full scan to discover devices added before the driver was loaded. */
tb_scan_switch(tb->root_switch);
tb_activate_pcie_devices(tb);
/* Allow tb_handle_hotplug to progress events */
- tb->hotplug_active = true;
- mutex_unlock(&tb->lock);
- return tb;
-
-err_locked:
- mutex_unlock(&tb->lock);
- thunderbolt_shutdown_and_free(tb);
- return NULL;
+ tcm->hotplug_active = true;
+ return 0;
}
-void thunderbolt_suspend(struct tb *tb)
+static int tb_suspend_noirq(struct tb *tb)
{
+ struct tb_cm *tcm = tb_priv(tb);
+
tb_info(tb, "suspending...\n");
- mutex_lock(&tb->lock);
tb_switch_suspend(tb->root_switch);
- tb_ctl_stop(tb->ctl);
- tb->hotplug_active = false; /* signal tb_handle_hotplug to quit */
- mutex_unlock(&tb->lock);
+ tcm->hotplug_active = false; /* signal tb_handle_hotplug to quit */
tb_info(tb, "suspend finished\n");
+
+ return 0;
}
-void thunderbolt_resume(struct tb *tb)
+static int tb_resume_noirq(struct tb *tb)
{
+ struct tb_cm *tcm = tb_priv(tb);
struct tb_pci_tunnel *tunnel, *n;
+
tb_info(tb, "resuming...\n");
- mutex_lock(&tb->lock);
- tb_ctl_start(tb->ctl);
/* remove any pci devices the firmware might have setup */
tb_switch_reset(tb, 0);
tb_switch_resume(tb->root_switch);
tb_free_invalid_tunnels(tb);
tb_free_unplugged_children(tb->root_switch);
- list_for_each_entry_safe(tunnel, n, &tb->tunnel_list, list)
+ list_for_each_entry_safe(tunnel, n, &tcm->tunnel_list, list)
tb_pci_restart(tunnel);
- if (!list_empty(&tb->tunnel_list)) {
+ if (!list_empty(&tcm->tunnel_list)) {
/*
* the pcie links need some time to get going.
* 100ms works for me...
msleep(100);
}
/* Allow tb_handle_hotplug to progress events */
- tb->hotplug_active = true;
- mutex_unlock(&tb->lock);
+ tcm->hotplug_active = true;
tb_info(tb, "resume finished\n");
+
+ return 0;
+}
+
+static const struct tb_cm_ops tb_cm_ops = {
+ .start = tb_start,
+ .stop = tb_stop,
+ .suspend_noirq = tb_suspend_noirq,
+ .resume_noirq = tb_resume_noirq,
+ .handle_event = tb_handle_event,
+};
+
+struct tb *tb_probe(struct tb_nhi *nhi)
+{
+ struct tb_cm *tcm;
+ struct tb *tb;
+
+ if (!dmi_match(DMI_BOARD_VENDOR, "Apple Inc."))
+ return NULL;
+
+ tb = tb_domain_alloc(nhi, sizeof(*tcm));
+ if (!tb)
+ return NULL;
+
+ tb->security_level = TB_SECURITY_NONE;
+ tb->cm_ops = &tb_cm_ops;
+
+ tcm = tb_priv(tb);
+ INIT_LIST_HEAD(&tcm->tunnel_list);
+
+ return tb;
}
#ifndef TB_H_
#define TB_H_
+#include <linux/nvmem-provider.h>
#include <linux/pci.h>
+#include <linux/uuid.h>
#include "tb_regs.h"
#include "ctl.h"
+#include "dma_port.h"
+
+/**
+ * struct tb_switch_nvm - Structure holding switch NVM information
+ * @major: Major version number of the active NVM portion
+ * @minor: Minor version number of the active NVM portion
+ * @id: Identifier used with both NVM portions
+ * @active: Active portion NVMem device
+ * @non_active: Non-active portion NVMem device
+ * @buf: Buffer where the NVM image is stored before it is written to
+ * the actual NVM flash device
+ * @buf_data_size: Number of bytes actually consumed by the new NVM
+ * image
+ * @authenticating: The switch is authenticating the new NVM
+ */
+struct tb_switch_nvm {
+ u8 major;
+ u8 minor;
+ int id;
+ struct nvmem_device *active;
+ struct nvmem_device *non_active;
+ void *buf;
+ size_t buf_data_size;
+ bool authenticating;
+};
+
+/**
+ * enum tb_security_level - Thunderbolt security level
+ * @TB_SECURITY_NONE: No security, legacy mode
+ * @TB_SECURITY_USER: User approval required at minimum
+ * @TB_SECURITY_SECURE: One time saved key required at minimum
+ * @TB_SECURITY_DPONLY: Only tunnel Display port (and USB)
+ */
+enum tb_security_level {
+ TB_SECURITY_NONE,
+ TB_SECURITY_USER,
+ TB_SECURITY_SECURE,
+ TB_SECURITY_DPONLY,
+};
+
+#define TB_SWITCH_KEY_SIZE 32
+/* Each physical port contains 2 links on modern controllers */
+#define TB_SWITCH_LINKS_PER_PHY_PORT 2
/**
* struct tb_switch - a thunderbolt switch
+ * @dev: Device for the switch
+ * @config: Switch configuration
+ * @ports: Ports in this switch
+ * @dma_port: If the switch has port supporting DMA configuration based
+ * mailbox this will hold the pointer to that (%NULL
+ * otherwise). If set it also means the switch has
+ * upgradeable NVM.
+ * @tb: Pointer to the domain the switch belongs to
+ * @uid: Unique ID of the switch
+ * @uuid: UUID of the switch (or %NULL if not supported)
+ * @vendor: Vendor ID of the switch
+ * @device: Device ID of the switch
+ * @vendor_name: Name of the vendor (or %NULL if not known)
+ * @device_name: Name of the device (or %NULL if not known)
+ * @generation: Switch Thunderbolt generation
+ * @cap_plug_events: Offset to the plug events capability (%0 if not found)
+ * @is_unplugged: The switch is going away
+ * @drom: DROM of the switch (%NULL if not found)
+ * @nvm: Pointer to the NVM if the switch has one (%NULL otherwise)
+ * @no_nvm_upgrade: Prevent NVM upgrade of this switch
+ * @safe_mode: The switch is in safe-mode
+ * @authorized: Whether the switch is authorized by user or policy
+ * @work: Work used to automatically authorize a switch
+ * @security_level: Switch supported security level
+ * @key: Contains the key used to challenge the device or %NULL if not
+ * supported. Size of the key is %TB_SWITCH_KEY_SIZE.
+ * @connection_id: Connection ID used with ICM messaging
+ * @connection_key: Connection key used with ICM messaging
+ * @link: Root switch link this switch is connected (ICM only)
+ * @depth: Depth in the chain this switch is connected (ICM only)
+ *
+ * When the switch is being added or removed to the domain (other
+ * switches) you need to have domain lock held. For switch authorization
+ * internal switch_lock is enough.
*/
struct tb_switch {
+ struct device dev;
struct tb_regs_switch_header config;
struct tb_port *ports;
+ struct tb_dma_port *dma_port;
struct tb *tb;
u64 uid;
- int cap_plug_events; /* offset, zero if not found */
- bool is_unplugged; /* unplugged, will go away */
+ uuid_be *uuid;
+ u16 vendor;
+ u16 device;
+ const char *vendor_name;
+ const char *device_name;
+ unsigned int generation;
+ int cap_plug_events;
+ bool is_unplugged;
u8 *drom;
+ struct tb_switch_nvm *nvm;
+ bool no_nvm_upgrade;
+ bool safe_mode;
+ unsigned int authorized;
+ struct work_struct work;
+ enum tb_security_level security_level;
+ u8 *key;
+ u8 connection_id;
+ u8 connection_key;
+ u8 link;
+ u8 depth;
};
/**
int path_length; /* number of hops */
};
+/**
+ * struct tb_cm_ops - Connection manager specific operations vector
+ * @driver_ready: Called right after control channel is started. Used by
+ * ICM to send driver ready message to the firmware.
+ * @start: Starts the domain
+ * @stop: Stops the domain
+ * @suspend_noirq: Connection manager specific suspend_noirq
+ * @resume_noirq: Connection manager specific resume_noirq
+ * @suspend: Connection manager specific suspend
+ * @complete: Connection manager specific complete
+ * @handle_event: Handle thunderbolt event
+ * @approve_switch: Approve switch
+ * @add_switch_key: Add key to switch
+ * @challenge_switch_key: Challenge switch using key
+ * @disconnect_pcie_paths: Disconnects PCIe paths before NVM update
+ */
+struct tb_cm_ops {
+ int (*driver_ready)(struct tb *tb);
+ int (*start)(struct tb *tb);
+ void (*stop)(struct tb *tb);
+ int (*suspend_noirq)(struct tb *tb);
+ int (*resume_noirq)(struct tb *tb);
+ int (*suspend)(struct tb *tb);
+ void (*complete)(struct tb *tb);
+ void (*handle_event)(struct tb *tb, enum tb_cfg_pkg_type,
+ const void *buf, size_t size);
+ int (*approve_switch)(struct tb *tb, struct tb_switch *sw);
+ int (*add_switch_key)(struct tb *tb, struct tb_switch *sw);
+ int (*challenge_switch_key)(struct tb *tb, struct tb_switch *sw,
+ const u8 *challenge, u8 *response);
+ int (*disconnect_pcie_paths)(struct tb *tb);
+};
/**
* struct tb - main thunderbolt bus structure
+ * @dev: Domain device
+ * @lock: Big lock. Must be held when accessing any struct
+ * tb_switch / struct tb_port.
+ * @nhi: Pointer to the NHI structure
+ * @ctl: Control channel for this domain
+ * @wq: Ordered workqueue for all domain specific work
+ * @root_switch: Root switch of this domain
+ * @cm_ops: Connection manager specific operations vector
+ * @index: Linux assigned domain number
+ * @security_level: Current security level
+ * @privdata: Private connection manager specific data
*/
struct tb {
- struct mutex lock; /*
- * Big lock. Must be held when accessing cfg or
- * any struct tb_switch / struct tb_port.
- */
+ struct device dev;
+ struct mutex lock;
struct tb_nhi *nhi;
struct tb_ctl *ctl;
- struct workqueue_struct *wq; /* ordered workqueue for plug events */
+ struct workqueue_struct *wq;
struct tb_switch *root_switch;
- struct list_head tunnel_list; /* list of active PCIe tunnels */
- bool hotplug_active; /*
- * tb_handle_hotplug will stop progressing plug
- * events and exit if this is not set (it needs to
- * acquire the lock one more time). Used to drain
- * wq after cfg has been paused.
- */
-
+ const struct tb_cm_ops *cm_ops;
+ int index;
+ enum tb_security_level security_level;
+ unsigned long privdata[0];
};
+static inline void *tb_priv(struct tb *tb)
+{
+ return (void *)tb->privdata;
+}
+
/* helper functions & macros */
/**
return ((u64) sw->config.route_hi) << 32 | sw->config.route_lo;
}
+static inline struct tb_port *tb_port_at(u64 route, struct tb_switch *sw)
+{
+ u8 port;
+
+ port = route >> (sw->config.depth * 8);
+ if (WARN_ON(port > sw->config.max_port_number))
+ return NULL;
+ return &sw->ports[port];
+}
+
static inline int tb_sw_read(struct tb_switch *sw, void *buffer,
enum tb_cfg_space space, u32 offset, u32 length)
{
length);
}
-static inline int tb_port_write(struct tb_port *port, void *buffer,
+static inline int tb_port_write(struct tb_port *port, const void *buffer,
enum tb_cfg_space space, u32 offset, u32 length)
{
return tb_cfg_write(port->sw->tb->ctl,
#define tb_port_info(port, fmt, arg...) \
__TB_PORT_PRINT(tb_info, port, fmt, ##arg)
+struct tb *icm_probe(struct tb_nhi *nhi);
+struct tb *tb_probe(struct tb_nhi *nhi);
+
+extern struct bus_type tb_bus_type;
+extern struct device_type tb_domain_type;
+extern struct device_type tb_switch_type;
+
+int tb_domain_init(void);
+void tb_domain_exit(void);
+void tb_switch_exit(void);
+
+struct tb *tb_domain_alloc(struct tb_nhi *nhi, size_t privsize);
+int tb_domain_add(struct tb *tb);
+void tb_domain_remove(struct tb *tb);
+int tb_domain_suspend_noirq(struct tb *tb);
+int tb_domain_resume_noirq(struct tb *tb);
+int tb_domain_suspend(struct tb *tb);
+void tb_domain_complete(struct tb *tb);
+int tb_domain_approve_switch(struct tb *tb, struct tb_switch *sw);
+int tb_domain_approve_switch_key(struct tb *tb, struct tb_switch *sw);
+int tb_domain_challenge_switch_key(struct tb *tb, struct tb_switch *sw);
+int tb_domain_disconnect_pcie_paths(struct tb *tb);
+
+static inline void tb_domain_put(struct tb *tb)
+{
+ put_device(&tb->dev);
+}
-struct tb *thunderbolt_alloc_and_start(struct tb_nhi *nhi);
-void thunderbolt_shutdown_and_free(struct tb *tb);
-void thunderbolt_suspend(struct tb *tb);
-void thunderbolt_resume(struct tb *tb);
-
-struct tb_switch *tb_switch_alloc(struct tb *tb, u64 route);
-void tb_switch_free(struct tb_switch *sw);
+struct tb_switch *tb_switch_alloc(struct tb *tb, struct device *parent,
+ u64 route);
+struct tb_switch *tb_switch_alloc_safe_mode(struct tb *tb,
+ struct device *parent, u64 route);
+int tb_switch_configure(struct tb_switch *sw);
+int tb_switch_add(struct tb_switch *sw);
+void tb_switch_remove(struct tb_switch *sw);
void tb_switch_suspend(struct tb_switch *sw);
int tb_switch_resume(struct tb_switch *sw);
int tb_switch_reset(struct tb *tb, u64 route);
void tb_sw_set_unplugged(struct tb_switch *sw);
struct tb_switch *get_switch_at_route(struct tb_switch *sw, u64 route);
+struct tb_switch *tb_switch_find_by_link_depth(struct tb *tb, u8 link,
+ u8 depth);
+struct tb_switch *tb_switch_find_by_uuid(struct tb *tb, const uuid_be *uuid);
+
+static inline unsigned int tb_switch_phy_port_from_link(unsigned int link)
+{
+ return (link - 1) / TB_SWITCH_LINKS_PER_PHY_PORT;
+}
+
+static inline void tb_switch_put(struct tb_switch *sw)
+{
+ put_device(&sw->dev);
+}
+
+static inline bool tb_is_switch(const struct device *dev)
+{
+ return dev->type == &tb_switch_type;
+}
+
+static inline struct tb_switch *tb_to_switch(struct device *dev)
+{
+ if (tb_is_switch(dev))
+ return container_of(dev, struct tb_switch, dev);
+ return NULL;
+}
int tb_wait_for_port(struct tb_port *port, bool wait_if_unplugged);
int tb_port_add_nfc_credits(struct tb_port *port, int credits);
int tb_port_clear_counter(struct tb_port *port, int counter);
-int tb_find_cap(struct tb_port *port, enum tb_cfg_space space, enum tb_cap cap);
+int tb_switch_find_vse_cap(struct tb_switch *sw, enum tb_switch_vse_cap vsec);
+int tb_port_find_cap(struct tb_port *port, enum tb_port_cap cap);
struct tb_path *tb_path_alloc(struct tb *tb, int num_hops);
void tb_path_free(struct tb_path *path);
--- /dev/null
+/*
+ * Thunderbolt control channel messages
+ *
+ * Copyright (C) 2014 Andreas Noever <andreas.noever@gmail.com>
+ * Copyright (C) 2017, Intel Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef _TB_MSGS
+#define _TB_MSGS
+
+#include <linux/types.h>
+#include <linux/uuid.h>
+
+enum tb_cfg_pkg_type {
+ TB_CFG_PKG_READ = 1,
+ TB_CFG_PKG_WRITE = 2,
+ TB_CFG_PKG_ERROR = 3,
+ TB_CFG_PKG_NOTIFY_ACK = 4,
+ TB_CFG_PKG_EVENT = 5,
+ TB_CFG_PKG_XDOMAIN_REQ = 6,
+ TB_CFG_PKG_XDOMAIN_RESP = 7,
+ TB_CFG_PKG_OVERRIDE = 8,
+ TB_CFG_PKG_RESET = 9,
+ TB_CFG_PKG_ICM_EVENT = 10,
+ TB_CFG_PKG_ICM_CMD = 11,
+ TB_CFG_PKG_ICM_RESP = 12,
+ TB_CFG_PKG_PREPARE_TO_SLEEP = 0xd,
+
+};
+
+enum tb_cfg_space {
+ TB_CFG_HOPS = 0,
+ TB_CFG_PORT = 1,
+ TB_CFG_SWITCH = 2,
+ TB_CFG_COUNTERS = 3,
+};
+
+enum tb_cfg_error {
+ TB_CFG_ERROR_PORT_NOT_CONNECTED = 0,
+ TB_CFG_ERROR_LINK_ERROR = 1,
+ TB_CFG_ERROR_INVALID_CONFIG_SPACE = 2,
+ TB_CFG_ERROR_NO_SUCH_PORT = 4,
+ TB_CFG_ERROR_ACK_PLUG_EVENT = 7, /* send as reply to TB_CFG_PKG_EVENT */
+ TB_CFG_ERROR_LOOP = 8,
+ TB_CFG_ERROR_HEC_ERROR_DETECTED = 12,
+ TB_CFG_ERROR_FLOW_CONTROL_ERROR = 13,
+};
+
+/* common header */
+struct tb_cfg_header {
+ u32 route_hi:22;
+ u32 unknown:10; /* highest order bit is set on replies */
+ u32 route_lo;
+} __packed;
+
+/* additional header for read/write packets */
+struct tb_cfg_address {
+ u32 offset:13; /* in dwords */
+ u32 length:6; /* in dwords */
+ u32 port:6;
+ enum tb_cfg_space space:2;
+ u32 seq:2; /* sequence number */
+ u32 zero:3;
+} __packed;
+
+/* TB_CFG_PKG_READ, response for TB_CFG_PKG_WRITE */
+struct cfg_read_pkg {
+ struct tb_cfg_header header;
+ struct tb_cfg_address addr;
+} __packed;
+
+/* TB_CFG_PKG_WRITE, response for TB_CFG_PKG_READ */
+struct cfg_write_pkg {
+ struct tb_cfg_header header;
+ struct tb_cfg_address addr;
+ u32 data[64]; /* maximum size, tb_cfg_address.length has 6 bits */
+} __packed;
+
+/* TB_CFG_PKG_ERROR */
+struct cfg_error_pkg {
+ struct tb_cfg_header header;
+ enum tb_cfg_error error:4;
+ u32 zero1:4;
+ u32 port:6;
+ u32 zero2:2; /* Both should be zero, still they are different fields. */
+ u32 zero3:16;
+} __packed;
+
+/* TB_CFG_PKG_EVENT */
+struct cfg_event_pkg {
+ struct tb_cfg_header header;
+ u32 port:6;
+ u32 zero:25;
+ bool unplug:1;
+} __packed;
+
+/* TB_CFG_PKG_RESET */
+struct cfg_reset_pkg {
+ struct tb_cfg_header header;
+} __packed;
+
+/* TB_CFG_PKG_PREPARE_TO_SLEEP */
+struct cfg_pts_pkg {
+ struct tb_cfg_header header;
+ u32 data;
+} __packed;
+
+/* ICM messages */
+
+enum icm_pkg_code {
+ ICM_GET_TOPOLOGY = 0x1,
+ ICM_DRIVER_READY = 0x3,
+ ICM_APPROVE_DEVICE = 0x4,
+ ICM_CHALLENGE_DEVICE = 0x5,
+ ICM_ADD_DEVICE_KEY = 0x6,
+ ICM_GET_ROUTE = 0xa,
+};
+
+enum icm_event_code {
+ ICM_EVENT_DEVICE_CONNECTED = 3,
+ ICM_EVENT_DEVICE_DISCONNECTED = 4,
+};
+
+struct icm_pkg_header {
+ u8 code;
+ u8 flags;
+ u8 packet_id;
+ u8 total_packets;
+} __packed;
+
+#define ICM_FLAGS_ERROR BIT(0)
+#define ICM_FLAGS_NO_KEY BIT(1)
+#define ICM_FLAGS_SLEVEL_SHIFT 3
+#define ICM_FLAGS_SLEVEL_MASK GENMASK(4, 3)
+
+struct icm_pkg_driver_ready {
+ struct icm_pkg_header hdr;
+} __packed;
+
+struct icm_pkg_driver_ready_response {
+ struct icm_pkg_header hdr;
+ u8 romver;
+ u8 ramver;
+ u16 security_level;
+} __packed;
+
+/* Falcon Ridge & Alpine Ridge common messages */
+
+struct icm_fr_pkg_get_topology {
+ struct icm_pkg_header hdr;
+} __packed;
+
+#define ICM_GET_TOPOLOGY_PACKETS 14
+
+struct icm_fr_pkg_get_topology_response {
+ struct icm_pkg_header hdr;
+ u32 route_lo;
+ u32 route_hi;
+ u8 first_data;
+ u8 second_data;
+ u8 drom_i2c_address_index;
+ u8 switch_index;
+ u32 reserved[2];
+ u32 ports[16];
+ u32 port_hop_info[16];
+} __packed;
+
+#define ICM_SWITCH_USED BIT(0)
+#define ICM_SWITCH_UPSTREAM_PORT_MASK GENMASK(7, 1)
+#define ICM_SWITCH_UPSTREAM_PORT_SHIFT 1
+
+#define ICM_PORT_TYPE_MASK GENMASK(23, 0)
+#define ICM_PORT_INDEX_SHIFT 24
+#define ICM_PORT_INDEX_MASK GENMASK(31, 24)
+
+struct icm_fr_event_device_connected {
+ struct icm_pkg_header hdr;
+ uuid_be ep_uuid;
+ u8 connection_key;
+ u8 connection_id;
+ u16 link_info;
+ u32 ep_name[55];
+} __packed;
+
+#define ICM_LINK_INFO_LINK_MASK 0x7
+#define ICM_LINK_INFO_DEPTH_SHIFT 4
+#define ICM_LINK_INFO_DEPTH_MASK GENMASK(7, 4)
+#define ICM_LINK_INFO_APPROVED BIT(8)
+
+struct icm_fr_pkg_approve_device {
+ struct icm_pkg_header hdr;
+ uuid_be ep_uuid;
+ u8 connection_key;
+ u8 connection_id;
+ u16 reserved;
+} __packed;
+
+struct icm_fr_event_device_disconnected {
+ struct icm_pkg_header hdr;
+ u16 reserved;
+ u16 link_info;
+} __packed;
+
+struct icm_fr_pkg_add_device_key {
+ struct icm_pkg_header hdr;
+ uuid_be ep_uuid;
+ u8 connection_key;
+ u8 connection_id;
+ u16 reserved;
+ u32 key[8];
+} __packed;
+
+struct icm_fr_pkg_add_device_key_response {
+ struct icm_pkg_header hdr;
+ uuid_be ep_uuid;
+ u8 connection_key;
+ u8 connection_id;
+ u16 reserved;
+} __packed;
+
+struct icm_fr_pkg_challenge_device {
+ struct icm_pkg_header hdr;
+ uuid_be ep_uuid;
+ u8 connection_key;
+ u8 connection_id;
+ u16 reserved;
+ u32 challenge[8];
+} __packed;
+
+struct icm_fr_pkg_challenge_device_response {
+ struct icm_pkg_header hdr;
+ uuid_be ep_uuid;
+ u8 connection_key;
+ u8 connection_id;
+ u16 reserved;
+ u32 challenge[8];
+ u32 response[8];
+} __packed;
+
+/* Alpine Ridge only messages */
+
+struct icm_ar_pkg_get_route {
+ struct icm_pkg_header hdr;
+ u16 reserved;
+ u16 link_info;
+} __packed;
+
+struct icm_ar_pkg_get_route_response {
+ struct icm_pkg_header hdr;
+ u16 reserved;
+ u16 link_info;
+ u32 route_hi;
+ u32 route_lo;
+} __packed;
+
+#endif
*/
#define TB_MAX_CONFIG_RW_LENGTH 60
-enum tb_cap {
- TB_CAP_PHY = 0x0001,
- TB_CAP_TIME1 = 0x0003,
- TB_CAP_PCIE = 0x0004,
- TB_CAP_I2C = 0x0005,
- TB_CAP_PLUG_EVENTS = 0x0105, /* also EEPROM */
- TB_CAP_TIME2 = 0x0305,
- TB_CAP_IECS = 0x0405,
- TB_CAP_LINK_CONTROLLER = 0x0605, /* also IECS */
+enum tb_switch_cap {
+ TB_SWITCH_CAP_VSE = 0x05,
+};
+
+enum tb_switch_vse_cap {
+ TB_VSE_CAP_PLUG_EVENTS = 0x01, /* also EEPROM */
+ TB_VSE_CAP_TIME2 = 0x03,
+ TB_VSE_CAP_IECS = 0x04,
+ TB_VSE_CAP_LINK_CONTROLLER = 0x06, /* also IECS */
+};
+
+enum tb_port_cap {
+ TB_PORT_CAP_PHY = 0x01,
+ TB_PORT_CAP_TIME1 = 0x03,
+ TB_PORT_CAP_ADAP = 0x04,
+ TB_PORT_CAP_VSE = 0x05,
};
enum tb_port_state {
u8 cap; /* if cap == 0x05 then we have a extended capability */
} __packed;
+/**
+ * struct tb_cap_extended_short - Switch extended short capability
+ * @next: Pointer to the next capability. If @next and @length are zero
+ * then we have a long cap.
+ * @cap: Base capability ID (see &enum tb_switch_cap)
+ * @vsec_id: Vendor specific capability ID (see &enum switch_vse_cap)
+ * @length: Length of this capability
+ */
struct tb_cap_extended_short {
- u8 next; /* if next and length are zero then we have a long cap */
- enum tb_cap cap:16;
+ u8 next;
+ u8 cap;
+ u8 vsec_id;
u8 length;
} __packed;
+/**
+ * struct tb_cap_extended_long - Switch extended long capability
+ * @zero1: This field should be zero
+ * @cap: Base capability ID (see &enum tb_switch_cap)
+ * @vsec_id: Vendor specific capability ID (see &enum switch_vse_cap)
+ * @zero2: This field should be zero
+ * @next: Pointer to the next capability
+ * @length: Length of this capability
+ */
struct tb_cap_extended_long {
u8 zero1;
- enum tb_cap cap:16;
+ u8 cap;
+ u8 vsec_id;
u8 zero2;
u16 next;
u16 length;
static int tb_pci_port_active(struct tb_port *port, bool active)
{
u32 word = active ? 0x80000000 : 0x0;
- int cap = tb_find_cap(port, TB_CFG_PORT, TB_CAP_PCIE);
- if (cap <= 0) {
- tb_port_warn(port, "TB_CAP_PCIE not found: %d\n", cap);
- return cap ? cap : -ENXIO;
+ int cap = tb_port_find_cap(port, TB_PORT_CAP_ADAP);
+ if (cap < 0) {
+ tb_port_warn(port, "TB_PORT_CAP_ADAP not found: %d\n", cap);
+ return cap;
}
return tb_port_write(port, &word, TB_CFG_PORT, cap, 1);
}
*/
int tb_pci_activate(struct tb_pci_tunnel *tunnel)
{
- int res;
if (tunnel->path_to_up->activated || tunnel->path_to_down->activated) {
tb_tunnel_WARN(tunnel,
"trying to activate an already activated tunnel\n");
return -EINVAL;
}
- res = tb_pci_restart(tunnel);
- if (res)
- return res;
-
- list_add(&tunnel->list, &tunnel->tb->tunnel_list);
- return 0;
+ return tb_pci_restart(tunnel);
}
tb_path_deactivate(tunnel->path_to_down);
if (tunnel->path_to_up->activated)
tb_path_deactivate(tunnel->path_to_up);
- list_del_init(&tunnel->list);
}
obj-$(CONFIG_TTY) += tty_io.o n_tty.o tty_ioctl.o tty_ldisc.o \
tty_buffer.o tty_port.o tty_mutex.o \
- tty_ldsem.o tty_baudrate.o tty_jobctrl.o
+ tty_ldsem.o tty_baudrate.o tty_jobctrl.o \
+ n_null.o
obj-$(CONFIG_LEGACY_PTYS) += pty.o
obj-$(CONFIG_UNIX98_PTYS) += pty.o
obj-$(CONFIG_AUDIT) += tty_audit.o
clear_bit(TTY_IO_ERROR, &tty->flags);
info->xmit.head = info->xmit.tail = 0;
- /*
- * Set up the tty->alt_speed kludge
- */
- if ((port->flags & ASYNC_SPD_MASK) == ASYNC_SPD_HI)
- tty->alt_speed = 57600;
- if ((port->flags & ASYNC_SPD_MASK) == ASYNC_SPD_VHI)
- tty->alt_speed = 115200;
- if ((port->flags & ASYNC_SPD_MASK) == ASYNC_SPD_SHI)
- tty->alt_speed = 230400;
- if ((port->flags & ASYNC_SPD_MASK) == ASYNC_SPD_WARP)
- tty->alt_speed = 460800;
-
/*
* and set the speed of the serial port
*/
check_and_exit:
if (tty_port_initialized(port)) {
if (change_spd) {
- if ((port->flags & ASYNC_SPD_MASK) == ASYNC_SPD_HI)
- tty->alt_speed = 57600;
- if ((port->flags & ASYNC_SPD_MASK) == ASYNC_SPD_VHI)
- tty->alt_speed = 115200;
- if ((port->flags & ASYNC_SPD_MASK) == ASYNC_SPD_SHI)
- tty->alt_speed = 230400;
- if ((port->flags & ASYNC_SPD_MASK) == ASYNC_SPD_WARP)
- tty->alt_speed = 460800;
+ /* warn about deprecation unless clearing */
+ if (new_serial.flags & ASYNC_SPD_MASK)
+ dev_warn_ratelimited(tty->dev, "use of SPD flags is deprecated\n");
change_speed(tty, state, NULL);
}
} else
cflag = tty->termios.c_cflag;
iflag = tty->termios.c_iflag;
- /*
- * Set up the tty->alt_speed kludge
- */
- if ((info->port.flags & ASYNC_SPD_MASK) == ASYNC_SPD_HI)
- tty->alt_speed = 57600;
- if ((info->port.flags & ASYNC_SPD_MASK) == ASYNC_SPD_VHI)
- tty->alt_speed = 115200;
- if ((info->port.flags & ASYNC_SPD_MASK) == ASYNC_SPD_SHI)
- tty->alt_speed = 230400;
- if ((info->port.flags & ASYNC_SPD_MASK) == ASYNC_SPD_WARP)
- tty->alt_speed = 460800;
-
card = info->card;
channel = info->line - card->first_line;
struct serial_struct __user *new_info)
{
struct serial_struct new_serial;
+ int old_flags;
int ret;
if (copy_from_user(&new_serial, new_info, sizeof(new_serial)))
return -EFAULT;
mutex_lock(&info->port.mutex);
+
+ old_flags = info->port.flags;
+
if (!capable(CAP_SYS_ADMIN)) {
if (new_serial.close_delay != info->port.close_delay ||
new_serial.baud_base != info->baud ||
check_and_exit:
if (tty_port_initialized(&info->port)) {
+ if ((new_serial.flags ^ old_flags) & ASYNC_SPD_MASK) {
+ /* warn about deprecation unless clearing */
+ if (new_serial.flags & ASYNC_SPD_MASK)
+ dev_warn_ratelimited(tty->dev, "use of SPD flags is deprecated\n");
+ }
cy_set_line_char(info, tty);
ret = 0;
} else {
config HVC_IUCV
bool "z/VM IUCV Hypervisor console support (VM only)"
- depends on S390
+ depends on S390 && NET
select HVC_DRIVER
select IUCV
default y
.attrs = hvcs_attrs,
};
-static ssize_t hvcs_rescan_show(struct device_driver *ddp, char *buf)
+static ssize_t rescan_show(struct device_driver *ddp, char *buf)
{
/* A 1 means it is updating, a 0 means it is done updating */
return snprintf(buf, PAGE_SIZE, "%d\n", hvcs_rescan_status);
}
-static ssize_t hvcs_rescan_store(struct device_driver *ddp, const char * buf,
+static ssize_t rescan_store(struct device_driver *ddp, const char * buf,
size_t count)
{
if ((simple_strtol(buf, NULL, 0) != 1)
return count;
}
-static DRIVER_ATTR(rescan,
- S_IRUGO | S_IWUSR, hvcs_rescan_show, hvcs_rescan_store);
+static DRIVER_ATTR_RW(rescan);
static void hvcs_kick(void)
{
free_irq(irq, hvcsd);
return;
} else if (hvcsd->port.count < 0) {
- printk(KERN_ERR "HVCS: vty-server@%X open_count: %d"
- " is missmanaged.\n",
+ printk(KERN_ERR "HVCS: vty-server@%X open_count: %d is mismanaged.\n",
hvcsd->vdev->unit_address, hvcsd->port.count);
}
gsm->io_error++;
}
+static int gsm_disconnect(struct gsm_mux *gsm)
+{
+ struct gsm_dlci *dlci = gsm->dlci[0];
+ struct gsm_control *gc;
+
+ if (!dlci)
+ return 0;
+
+ /* In theory disconnecting DLCI 0 is sufficient but for some
+ modems this is apparently not the case. */
+ gc = gsm_control_send(gsm, CMD_CLD, NULL, 0);
+ if (gc)
+ gsm_control_wait(gsm, gc);
+
+ del_timer_sync(&gsm->t2_timer);
+ /* Now we are sure T2 has stopped */
+
+ gsm_dlci_begin_close(dlci);
+ wait_event_interruptible(gsm->event,
+ dlci->state == DLCI_CLOSED);
+
+ if (signal_pending(current))
+ return -EINTR;
+
+ return 0;
+}
+
/**
* gsm_cleanup_mux - generic GSM protocol cleanup
* @gsm: our mux
int i;
struct gsm_dlci *dlci = gsm->dlci[0];
struct gsm_msg *txq, *ntxq;
- struct gsm_control *gc;
gsm->dead = 1;
if (i == MAX_MUX)
return;
- /* In theory disconnecting DLCI 0 is sufficient but for some
- modems this is apparently not the case. */
- if (dlci) {
- gc = gsm_control_send(gsm, CMD_CLD, NULL, 0);
- if (gc)
- gsm_control_wait(gsm, gc);
- }
del_timer_sync(&gsm->t2_timer);
/* Now we are sure T2 has stopped */
- if (dlci) {
+ if (dlci)
dlci->dead = 1;
- gsm_dlci_begin_close(dlci);
- wait_event_interruptible(gsm->event,
- dlci->state == DLCI_CLOSED);
- }
+
/* Free up any link layer users */
mutex_lock(&gsm->mutex);
for (i = 0; i < NUM_DLCI; i++)
*/
if (need_close || need_restart) {
- gsm_dlci_begin_close(gsm->dlci[0]);
- /* This will timeout if the link is down due to N2 expiring */
- wait_event_interruptible(gsm->event,
- gsm->dlci[0]->state == DLCI_CLOSED);
- if (signal_pending(current))
- return -EINTR;
+ int ret;
+
+ ret = gsm_disconnect(gsm);
+
+ if (ret)
+ return ret;
}
if (need_restart)
gsm_cleanup_mux(gsm);
--- /dev/null
+#include <linux/types.h>
+#include <linux/errno.h>
+#include <linux/tty.h>
+#include <linux/module.h>
+
+/*
+ * n_null.c - Null line discipline used in the failure path
+ *
+ * Copyright (C) Intel 2017
+ *
+ * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2
+ * as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ */
+
+static int n_null_open(struct tty_struct *tty)
+{
+ return 0;
+}
+
+static void n_null_close(struct tty_struct *tty)
+{
+}
+
+static ssize_t n_null_read(struct tty_struct *tty, struct file *file,
+ unsigned char __user * buf, size_t nr)
+{
+ return -EOPNOTSUPP;
+}
+
+static ssize_t n_null_write(struct tty_struct *tty, struct file *file,
+ const unsigned char *buf, size_t nr)
+{
+ return -EOPNOTSUPP;
+}
+
+static void n_null_receivebuf(struct tty_struct *tty,
+ const unsigned char *cp, char *fp,
+ int cnt)
+{
+}
+
+static struct tty_ldisc_ops null_ldisc = {
+ .owner = THIS_MODULE,
+ .magic = TTY_LDISC_MAGIC,
+ .name = "n_null",
+ .open = n_null_open,
+ .close = n_null_close,
+ .read = n_null_read,
+ .write = n_null_write,
+ .receive_buf = n_null_receivebuf
+};
+
+static int __init n_null_init(void)
+{
+ BUG_ON(tty_register_ldisc(N_NULL, &null_ldisc));
+ return 0;
+}
+
+static void __exit n_null_exit(void)
+{
+ tty_unregister_ldisc(N_NULL);
+}
+
+module_init(n_null_init);
+module_exit(n_null_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Alan Cox");
+MODULE_ALIAS_LDISC(N_NULL);
+MODULE_DESCRIPTION("Null ldisc driver");
#include <linux/slab.h>
#include <linux/mutex.h>
#include <linux/poll.h>
+#include <linux/mount.h>
+#include <linux/file.h>
+#include <linux/ioctl.h>
#undef TTY_DEBUG_HANGUP
#ifdef TTY_DEBUG_HANGUP
#ifdef CONFIG_UNIX98_PTYS
if (tty->driver == ptm_driver) {
mutex_lock(&devpts_mutex);
- if (tty->link->driver_data)
- devpts_pty_kill(tty->link->driver_data);
+ if (tty->link->driver_data) {
+ struct path *path = tty->link->driver_data;
+
+ devpts_pty_kill(path->dentry);
+ path_put(path);
+ kfree(path);
+ }
mutex_unlock(&devpts_mutex);
}
#endif
return retval;
}
+/**
+ * pty_open_peer - open the peer of a pty
+ * @tty: the peer of the pty being opened
+ *
+ * Open the cached dentry in tty->link, providing a safe way for userspace
+ * to get the slave end of a pty (where they have the master fd and cannot
+ * access or trust the mount namespace /dev/pts was mounted inside).
+ */
+static struct file *pty_open_peer(struct tty_struct *tty, int flags)
+{
+ if (tty->driver->subtype != PTY_TYPE_MASTER)
+ return ERR_PTR(-EIO);
+ return dentry_open(tty->link->driver_data, flags, current_cred());
+}
+
+static int pty_get_peer(struct tty_struct *tty, int flags)
+{
+ int fd = -1;
+ struct file *filp = NULL;
+ int retval = -EINVAL;
+
+ fd = get_unused_fd_flags(0);
+ if (fd < 0) {
+ retval = fd;
+ goto err;
+ }
+
+ filp = pty_open_peer(tty, flags);
+ if (IS_ERR(filp)) {
+ retval = PTR_ERR(filp);
+ goto err_put;
+ }
+
+ fd_install(fd, filp);
+ return fd;
+
+err_put:
+ put_unused_fd(fd);
+err:
+ return retval;
+}
+
static void pty_cleanup(struct tty_struct *tty)
{
tty_port_put(tty->port);
return -ENOIOCTLCMD;
}
+static long pty_bsd_compat_ioctl(struct tty_struct *tty,
+ unsigned int cmd, unsigned long arg)
+{
+ /*
+ * PTY ioctls don't require any special translation between 32-bit and
+ * 64-bit userspace, they are already compatible.
+ */
+ return pty_bsd_ioctl(tty, cmd, arg);
+}
+
static int legacy_count = CONFIG_LEGACY_PTY_COUNT;
/*
* not really modular, but the easiest way to keep compat with existing
.chars_in_buffer = pty_chars_in_buffer,
.unthrottle = pty_unthrottle,
.ioctl = pty_bsd_ioctl,
+ .compat_ioctl = pty_bsd_compat_ioctl,
.cleanup = pty_cleanup,
.resize = pty_resize,
.remove = pty_remove
return pty_get_pktmode(tty, (int __user *)arg);
case TIOCGPTN: /* Get PT Number */
return put_user(tty->index, (unsigned int __user *)arg);
+ case TIOCGPTPEER: /* Open the other end */
+ return pty_get_peer(tty, (int) arg);
case TIOCSIG: /* Send signal to other side of pty */
return pty_signal(tty, (int) arg);
}
return -ENOIOCTLCMD;
}
+static long pty_unix98_compat_ioctl(struct tty_struct *tty,
+ unsigned int cmd, unsigned long arg)
+{
+ /*
+ * PTY ioctls don't require any special translation between 32-bit and
+ * 64-bit userspace, they are already compatible.
+ */
+ return pty_unix98_ioctl(tty, cmd, arg);
+}
+
/**
* ptm_unix98_lookup - find a pty master
* @driver: ptm driver
.chars_in_buffer = pty_chars_in_buffer,
.unthrottle = pty_unthrottle,
.ioctl = pty_unix98_ioctl,
+ .compat_ioctl = pty_unix98_compat_ioctl,
.resize = pty_resize,
.cleanup = pty_cleanup
};
{
struct pts_fs_info *fsi;
struct tty_struct *tty;
+ struct path *pts_path;
struct dentry *dentry;
int retval;
int index;
retval = PTR_ERR(dentry);
goto err_release;
}
- tty->link->driver_data = dentry;
+ /* We need to cache a fake path for TIOCGPTPEER. */
+ pts_path = kmalloc(sizeof(struct path), GFP_KERNEL);
+ if (!pts_path)
+ goto err_release;
+ pts_path->mnt = filp->f_path.mnt;
+ pts_path->dentry = dentry;
+ path_get(pts_path);
+ tty->link->driver_data = pts_path;
retval = ptm_driver->ops->open(tty, filp);
if (retval)
- goto err_release;
+ goto err_path_put;
tty_debug_hangup(tty, "opening (count=%d)\n", tty->count);
tty_unlock(tty);
return 0;
+err_path_put:
+ path_put(pts_path);
+ kfree(pts_path);
err_release:
tty_unlock(tty);
// This will also put-ref the fsi
tty_port_set_initialized(&info->port, 1);
- /*
- * Set up the tty->alt_speed kludge
- */
- if ((info->flags & ROCKET_SPD_MASK) == ROCKET_SPD_HI)
- tty->alt_speed = 57600;
- if ((info->flags & ROCKET_SPD_MASK) == ROCKET_SPD_VHI)
- tty->alt_speed = 115200;
- if ((info->flags & ROCKET_SPD_MASK) == ROCKET_SPD_SHI)
- tty->alt_speed = 230400;
- if ((info->flags & ROCKET_SPD_MASK) == ROCKET_SPD_WARP)
- tty->alt_speed = 460800;
-
configure_r_port(tty, info, NULL);
if (C_BAUD(tty)) {
sSetDTR(cp);
return -EPERM;
}
info->flags = ((info->flags & ~ROCKET_USR_MASK) | (new_serial.flags & ROCKET_USR_MASK));
- configure_r_port(tty, info, NULL);
mutex_unlock(&info->port.mutex);
return 0;
}
+ if ((new_serial.flags ^ info->flags) & ROCKET_SPD_MASK) {
+ /* warn about deprecation, unless clearing */
+ if (new_serial.flags & ROCKET_SPD_MASK)
+ dev_warn_ratelimited(tty->dev, "use of SPD flags is deprecated\n");
+ }
+
info->flags = ((info->flags & ~ROCKET_FLAGS) | (new_serial.flags & ROCKET_FLAGS));
info->port.close_delay = new_serial.close_delay;
info->port.closing_wait = new_serial.closing_wait;
- if ((info->flags & ROCKET_SPD_MASK) == ROCKET_SPD_HI)
- tty->alt_speed = 57600;
- if ((info->flags & ROCKET_SPD_MASK) == ROCKET_SPD_VHI)
- tty->alt_speed = 115200;
- if ((info->flags & ROCKET_SPD_MASK) == ROCKET_SPD_SHI)
- tty->alt_speed = 230400;
- if ((info->flags & ROCKET_SPD_MASK) == ROCKET_SPD_WARP)
- tty->alt_speed = 460800;
mutex_unlock(&info->port.mutex);
configure_r_port(tty, info, NULL);
{
return of_device_modalias(dev, buf, PAGE_SIZE);
}
+DEVICE_ATTR_RO(modalias);
-static struct device_attribute serdev_device_attrs[] = {
- __ATTR_RO(modalias),
- __ATTR_NULL
+static struct attribute *serdev_device_attrs[] = {
+ &dev_attr_modalias.attr,
+ NULL,
};
+ATTRIBUTE_GROUPS(serdev_device);
static struct bus_type serdev_bus_type = {
.name = "serial",
.probe = serdev_drv_probe,
.remove = serdev_drv_remove,
.uevent = serdev_uevent,
- .dev_attrs = serdev_device_attrs,
+ .dev_groups = serdev_device_groups,
};
/**
/* tty_set_termios() return not checked as it is always 0 */
tty_set_termios(tty, &ktermios);
- return speed;
+ return ktermios.c_ospeed;
}
static void ttyport_set_flow_control(struct serdev_controller *ctrl, bool enable)
#define UART_CAP_HFIFO (1 << 14) /* UART has a "hidden" FIFO */
#define UART_CAP_RPM (1 << 15) /* Runtime PM is active while idle */
#define UART_CAP_IRDA (1 << 16) /* UART supports IrDA line discipline */
+#define UART_CAP_MINI (1 << 17) /* Mini UART on BCM283X family lacks:
+ * STOP PARITY EPAR SPAR WLEN5 WLEN6
+ */
#define UART_BUG_QUOT (1 << 0) /* UART has buggy quot LSB */
#define UART_BUG_TXEN (1 << 1) /* UART has buggy TX IIR status */
--- /dev/null
+/*
+ * Serial Port driver for Aspeed VUART device
+ *
+ * Copyright (C) 2016 Jeremy Kerr <jk@ozlabs.org>, IBM Corp.
+ * Copyright (C) 2006 Arnd Bergmann <arnd@arndb.de>, IBM Corp.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+#include <linux/device.h>
+#include <linux/module.h>
+#include <linux/of_address.h>
+#include <linux/of_irq.h>
+#include <linux/of_platform.h>
+#include <linux/clk.h>
+
+#include "8250.h"
+
+#define ASPEED_VUART_GCRA 0x20
+#define ASPEED_VUART_GCRA_VUART_EN BIT(0)
+#define ASPEED_VUART_GCRA_DISABLE_HOST_TX_DISCARD BIT(5)
+#define ASPEED_VUART_GCRB 0x24
+#define ASPEED_VUART_GCRB_HOST_SIRQ_MASK GENMASK(7, 4)
+#define ASPEED_VUART_GCRB_HOST_SIRQ_SHIFT 4
+#define ASPEED_VUART_ADDRL 0x28
+#define ASPEED_VUART_ADDRH 0x2c
+
+struct aspeed_vuart {
+ struct device *dev;
+ void __iomem *regs;
+ struct clk *clk;
+ int line;
+};
+
+/*
+ * The VUART is basically two UART 'front ends' connected by their FIFO
+ * (no actual serial line in between). One is on the BMC side (management
+ * controller) and one is on the host CPU side.
+ *
+ * It allows the BMC to provide to the host a "UART" that pipes into
+ * the BMC itself and can then be turned by the BMC into a network console
+ * of some sort for example.
+ *
+ * This driver is for the BMC side. The sysfs files allow the BMC
+ * userspace which owns the system configuration policy, to specify
+ * at what IO port and interrupt number the host side will appear
+ * to the host on the Host <-> BMC LPC bus. It could be different on a
+ * different system (though most of them use 3f8/4).
+ */
+
+static ssize_t lpc_address_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct aspeed_vuart *vuart = dev_get_drvdata(dev);
+ u16 addr;
+
+ addr = (readb(vuart->regs + ASPEED_VUART_ADDRH) << 8) |
+ (readb(vuart->regs + ASPEED_VUART_ADDRL));
+
+ return snprintf(buf, PAGE_SIZE - 1, "0x%x\n", addr);
+}
+
+static ssize_t lpc_address_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct aspeed_vuart *vuart = dev_get_drvdata(dev);
+ unsigned long val;
+ int err;
+
+ err = kstrtoul(buf, 0, &val);
+ if (err)
+ return err;
+
+ writeb(val >> 8, vuart->regs + ASPEED_VUART_ADDRH);
+ writeb(val >> 0, vuart->regs + ASPEED_VUART_ADDRL);
+
+ return count;
+}
+
+static DEVICE_ATTR_RW(lpc_address);
+
+static ssize_t sirq_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct aspeed_vuart *vuart = dev_get_drvdata(dev);
+ u8 reg;
+
+ reg = readb(vuart->regs + ASPEED_VUART_GCRB);
+ reg &= ASPEED_VUART_GCRB_HOST_SIRQ_MASK;
+ reg >>= ASPEED_VUART_GCRB_HOST_SIRQ_SHIFT;
+
+ return snprintf(buf, PAGE_SIZE - 1, "%u\n", reg);
+}
+
+static ssize_t sirq_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct aspeed_vuart *vuart = dev_get_drvdata(dev);
+ unsigned long val;
+ int err;
+ u8 reg;
+
+ err = kstrtoul(buf, 0, &val);
+ if (err)
+ return err;
+
+ val <<= ASPEED_VUART_GCRB_HOST_SIRQ_SHIFT;
+ val &= ASPEED_VUART_GCRB_HOST_SIRQ_MASK;
+
+ reg = readb(vuart->regs + ASPEED_VUART_GCRB);
+ reg &= ~ASPEED_VUART_GCRB_HOST_SIRQ_MASK;
+ reg |= val;
+ writeb(reg, vuart->regs + ASPEED_VUART_GCRB);
+
+ return count;
+}
+
+static DEVICE_ATTR_RW(sirq);
+
+static struct attribute *aspeed_vuart_attrs[] = {
+ &dev_attr_sirq.attr,
+ &dev_attr_lpc_address.attr,
+ NULL,
+};
+
+static const struct attribute_group aspeed_vuart_attr_group = {
+ .attrs = aspeed_vuart_attrs,
+};
+
+static void aspeed_vuart_set_enabled(struct aspeed_vuart *vuart, bool enabled)
+{
+ u8 reg = readb(vuart->regs + ASPEED_VUART_GCRA);
+
+ if (enabled)
+ reg |= ASPEED_VUART_GCRA_VUART_EN;
+ else
+ reg &= ~ASPEED_VUART_GCRA_VUART_EN;
+
+ writeb(reg, vuart->regs + ASPEED_VUART_GCRA);
+}
+
+static void aspeed_vuart_set_host_tx_discard(struct aspeed_vuart *vuart,
+ bool discard)
+{
+ u8 reg;
+
+ reg = readb(vuart->regs + ASPEED_VUART_GCRA);
+
+ /* If the DISABLE_HOST_TX_DISCARD bit is set, discard is disabled */
+ if (!discard)
+ reg |= ASPEED_VUART_GCRA_DISABLE_HOST_TX_DISCARD;
+ else
+ reg &= ~ASPEED_VUART_GCRA_DISABLE_HOST_TX_DISCARD;
+
+ writeb(reg, vuart->regs + ASPEED_VUART_GCRA);
+}
+
+static int aspeed_vuart_startup(struct uart_port *uart_port)
+{
+ struct uart_8250_port *uart_8250_port = up_to_u8250p(uart_port);
+ struct aspeed_vuart *vuart = uart_8250_port->port.private_data;
+ int rc;
+
+ rc = serial8250_do_startup(uart_port);
+ if (rc)
+ return rc;
+
+ aspeed_vuart_set_host_tx_discard(vuart, false);
+
+ return 0;
+}
+
+static void aspeed_vuart_shutdown(struct uart_port *uart_port)
+{
+ struct uart_8250_port *uart_8250_port = up_to_u8250p(uart_port);
+ struct aspeed_vuart *vuart = uart_8250_port->port.private_data;
+
+ aspeed_vuart_set_host_tx_discard(vuart, true);
+
+ serial8250_do_shutdown(uart_port);
+}
+
+static int aspeed_vuart_probe(struct platform_device *pdev)
+{
+ struct uart_8250_port port;
+ struct aspeed_vuart *vuart;
+ struct device_node *np;
+ struct resource *res;
+ u32 clk, prop;
+ int rc;
+
+ np = pdev->dev.of_node;
+
+ vuart = devm_kzalloc(&pdev->dev, sizeof(*vuart), GFP_KERNEL);
+ if (!vuart)
+ return -ENOMEM;
+
+ vuart->dev = &pdev->dev;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ vuart->regs = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(vuart->regs))
+ return PTR_ERR(vuart->regs);
+
+ memset(&port, 0, sizeof(port));
+ port.port.private_data = vuart;
+ port.port.membase = vuart->regs;
+ port.port.mapbase = res->start;
+ port.port.mapsize = resource_size(res);
+ port.port.startup = aspeed_vuart_startup;
+ port.port.shutdown = aspeed_vuart_shutdown;
+ port.port.dev = &pdev->dev;
+
+ rc = sysfs_create_group(&vuart->dev->kobj, &aspeed_vuart_attr_group);
+ if (rc < 0)
+ return rc;
+
+ if (of_property_read_u32(np, "clock-frequency", &clk)) {
+ vuart->clk = devm_clk_get(&pdev->dev, NULL);
+ if (IS_ERR(vuart->clk)) {
+ dev_warn(&pdev->dev,
+ "clk or clock-frequency not defined\n");
+ return PTR_ERR(vuart->clk);
+ }
+
+ rc = clk_prepare_enable(vuart->clk);
+ if (rc < 0)
+ return rc;
+
+ clk = clk_get_rate(vuart->clk);
+ }
+
+ /* If current-speed was set, then try not to change it. */
+ if (of_property_read_u32(np, "current-speed", &prop) == 0)
+ port.port.custom_divisor = clk / (16 * prop);
+
+ /* Check for shifted address mapping */
+ if (of_property_read_u32(np, "reg-offset", &prop) == 0)
+ port.port.mapbase += prop;
+
+ /* Check for registers offset within the devices address range */
+ if (of_property_read_u32(np, "reg-shift", &prop) == 0)
+ port.port.regshift = prop;
+
+ /* Check for fifo size */
+ if (of_property_read_u32(np, "fifo-size", &prop) == 0)
+ port.port.fifosize = prop;
+
+ /* Check for a fixed line number */
+ rc = of_alias_get_id(np, "serial");
+ if (rc >= 0)
+ port.port.line = rc;
+
+ port.port.irq = irq_of_parse_and_map(np, 0);
+ port.port.irqflags = IRQF_SHARED;
+ port.port.iotype = UPIO_MEM;
+ port.port.type = PORT_16550A;
+ port.port.uartclk = clk;
+ port.port.flags = UPF_SHARE_IRQ | UPF_BOOT_AUTOCONF
+ | UPF_FIXED_PORT | UPF_FIXED_TYPE | UPF_NO_THRE_TEST;
+
+ if (of_property_read_bool(np, "no-loopback-test"))
+ port.port.flags |= UPF_SKIP_TEST;
+
+ if (port.port.fifosize)
+ port.capabilities = UART_CAP_FIFO;
+
+ if (of_property_read_bool(np, "auto-flow-control"))
+ port.capabilities |= UART_CAP_AFE;
+
+ rc = serial8250_register_8250_port(&port);
+ if (rc < 0)
+ goto err_clk_disable;
+
+ vuart->line = rc;
+
+ aspeed_vuart_set_enabled(vuart, true);
+ aspeed_vuart_set_host_tx_discard(vuart, true);
+ platform_set_drvdata(pdev, vuart);
+
+ return 0;
+
+err_clk_disable:
+ clk_disable_unprepare(vuart->clk);
+ irq_dispose_mapping(port.port.irq);
+ return rc;
+}
+
+static int aspeed_vuart_remove(struct platform_device *pdev)
+{
+ struct aspeed_vuart *vuart = platform_get_drvdata(pdev);
+
+ aspeed_vuart_set_enabled(vuart, false);
+ serial8250_unregister_port(vuart->line);
+ sysfs_remove_group(&vuart->dev->kobj, &aspeed_vuart_attr_group);
+ clk_disable_unprepare(vuart->clk);
+
+ return 0;
+}
+
+static const struct of_device_id aspeed_vuart_table[] = {
+ { .compatible = "aspeed,ast2400-vuart" },
+ { .compatible = "aspeed,ast2500-vuart" },
+ { },
+};
+
+static struct platform_driver aspeed_vuart_driver = {
+ .driver = {
+ .name = "aspeed-vuart",
+ .of_match_table = aspeed_vuart_table,
+ },
+ .probe = aspeed_vuart_probe,
+ .remove = aspeed_vuart_remove,
+};
+
+module_platform_driver(aspeed_vuart_driver);
+
+MODULE_AUTHOR("Jeremy Kerr <jk@ozlabs.org>");
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("Driver for Aspeed VUART device");
/* initialize data */
spin_lock_init(&data->uart.port.lock);
- data->uart.capabilities = UART_CAP_FIFO;
+ data->uart.capabilities = UART_CAP_FIFO | UART_CAP_MINI;
data->uart.port.dev = &pdev->dev;
data->uart.port.regshift = 2;
data->uart.port.type = PORT_16550;
if (up->dl_write)
uart->dl_write = up->dl_write;
- if (uart->port.type != PORT_8250_CIR) {
- if (serial8250_isa_config != NULL)
- serial8250_isa_config(0, &uart->port,
- &uart->capabilities);
-
- ret = uart_add_one_port(&serial8250_reg,
- &uart->port);
- if (ret == 0)
- ret = uart->port.line;
- } else {
- dev_info(uart->port.dev,
- "skipping CIR port at 0x%lx / 0x%llx, IRQ %d\n",
- uart->port.iobase,
- (unsigned long long)uart->port.mapbase,
- uart->port.irq);
+ if (serial8250_isa_config != NULL)
+ serial8250_isa_config(0, &uart->port,
+ &uart->capabilities);
- ret = 0;
- }
+ ret = uart_add_one_port(&serial8250_reg, &uart->port);
+ if (ret == 0)
+ ret = uart->port.line;
}
mutex_unlock(&serial_mutex);
u8 __iomem *p;
int err;
- port->port.flags |= UPF_EXAR_EFR;
port->port.uartclk = baud * 16;
err = default_setup(priv, pcidev, idx, offset, port);
return default_setup(priv, pcidev, idx, offset, port);
}
-static void setup_gpio(u8 __iomem *p)
+static void setup_gpio(struct pci_dev *pcidev, u8 __iomem *p)
{
+ /*
+ * The Commtech adapters required the MPIOs to be driven low. The Exar
+ * devices will export them as GPIOs, so we pre-configure them safely
+ * as inputs.
+ */
+ u8 dir = pcidev->vendor == PCI_VENDOR_ID_EXAR ? 0xff : 0x00;
+
writeb(0x00, p + UART_EXAR_MPIOINT_7_0);
writeb(0x00, p + UART_EXAR_MPIOLVL_7_0);
writeb(0x00, p + UART_EXAR_MPIO3T_7_0);
writeb(0x00, p + UART_EXAR_MPIOINV_7_0);
- writeb(0x00, p + UART_EXAR_MPIOSEL_7_0);
+ writeb(dir, p + UART_EXAR_MPIOSEL_7_0);
writeb(0x00, p + UART_EXAR_MPIOOD_7_0);
writeb(0x00, p + UART_EXAR_MPIOINT_15_8);
writeb(0x00, p + UART_EXAR_MPIOLVL_15_8);
writeb(0x00, p + UART_EXAR_MPIO3T_15_8);
writeb(0x00, p + UART_EXAR_MPIOINV_15_8);
- writeb(0x00, p + UART_EXAR_MPIOSEL_15_8);
+ writeb(dir, p + UART_EXAR_MPIOSEL_15_8);
writeb(0x00, p + UART_EXAR_MPIOOD_15_8);
}
if (idx == 0) {
/* Setup Multipurpose Input/Output pins. */
- setup_gpio(p);
+ setup_gpio(pcidev, p);
port->port.private_data = xr17v35x_register_gpio(pcidev);
}
#include <linux/of_irq.h>
#include <linux/of_platform.h>
#include <linux/clk.h>
+#include <linux/reset.h>
#include "8250.h"
struct of_serial_info {
struct clk *clk;
+ struct reset_control *rst;
int type;
int line;
};
}
}
+ info->rst = devm_reset_control_get_optional_shared(&ofdev->dev, NULL);
+ if (IS_ERR(info->rst))
+ goto out;
+ ret = reset_control_deassert(info->rst);
+ if (ret)
+ goto out;
+
port->type = type;
port->uartclk = clk;
port->flags = UPF_SHARE_IRQ | UPF_BOOT_AUTOCONF | UPF_IOREMAP
serial8250_unregister_port(info->line);
+ reset_control_assert(info->rst);
if (info->clk)
clk_disable_unprepare(info->clk);
kfree(info);
up->lsr_saved_flags = 0;
up->msr_saved_flags = 0;
+ /* Disable DMA for console UART */
+ if (uart_console(port))
+ up->dma = NULL;
+
if (up->dma) {
ret = serial8250_request_dma(up);
if (ret) {
static void __dma_rx_complete(void *param)
{
- __dma_rx_do_complete(param);
- omap_8250_rx_dma(param);
+ struct uart_8250_port *p = param;
+ struct uart_8250_dma *dma = p->dma;
+ struct dma_tx_state state;
+ unsigned long flags;
+
+ spin_lock_irqsave(&p->port.lock, flags);
+
+ /*
+ * If the tx status is not DMA_COMPLETE, then this is a delayed
+ * completion callback. A previous RX timeout flush would have
+ * already pushed the data, so exit.
+ */
+ if (dmaengine_tx_status(dma->rxchan, dma->rx_cookie, &state) !=
+ DMA_COMPLETE) {
+ spin_unlock_irqrestore(&p->port.lock, flags);
+ return;
+ }
+ __dma_rx_do_complete(p);
+ omap_8250_rx_dma(p);
+
+ spin_unlock_irqrestore(&p->port.lock, flags);
}
static void omap_8250_rx_dma_flush(struct uart_8250_port *p)
if ((up->capabilities & UART_CAP_HFIFO) &&
(serial_in(up, UART_LSR) & BOTH_EMPTY) != BOTH_EMPTY)
break;
+ /* The BCM2835 MINI UART THRE bit is really a not-full bit. */
+ if ((up->capabilities & UART_CAP_MINI) &&
+ !(serial_in(up, UART_LSR) & UART_LSR_THRE))
+ break;
} while (--count > 0);
if (uart_circ_chars_pending(xmit) < WAKEUP_CHARS)
}
}
- if (port->irq) {
+ if (port->irq && !(up->port.flags & UPF_NO_THRE_TEST)) {
unsigned char iir1;
/*
* Test for UARTs that do not reassert THRE when the
unsigned long flags;
unsigned int baud, quot, frac = 0;
+ if (up->capabilities & UART_CAP_MINI) {
+ termios->c_cflag &= ~(CSTOPB | PARENB | PARODD | CMSPAR);
+ if ((termios->c_cflag & CSIZE) == CS5 ||
+ (termios->c_cflag & CSIZE) == CS6)
+ termios->c_cflag = (termios->c_cflag & ~CSIZE) | CS7;
+ }
cval = serial8250_compute_lcr(up, termios->c_cflag);
baud = serial8250_get_baud_rate(port, termios, old);
To compile this driver as a module, choose M here: the module
will be called 8250_accent.
+config SERIAL_8250_ASPEED_VUART
+ tristate "Aspeed Virtual UART"
+ depends on SERIAL_8250
+ depends on OF
+ help
+ If you want to use the virtual UART (VUART) device on Aspeed
+ BMC platforms, enable this option. This enables the 16550A-
+ compatible device on the local LPC bus, giving a UART device
+ with no physical RS232 connections.
+
config SERIAL_8250_BOCA
tristate "Support Boca cards"
depends on SERIAL_8250 != n && ISA && SERIAL_8250_MANY_PORTS
obj-$(CONFIG_SERIAL_8250_HP300) += 8250_hp300.o
obj-$(CONFIG_SERIAL_8250_CS) += serial_cs.o
obj-$(CONFIG_SERIAL_8250_ACORN) += 8250_acorn.o
+obj-$(CONFIG_SERIAL_8250_ASPEED_VUART) += 8250_aspeed_vuart.o
obj-$(CONFIG_SERIAL_8250_BCM2835AUX) += 8250_bcm2835aux.o
obj-$(CONFIG_SERIAL_8250_CONSOLE) += 8250_early.o
obj-$(CONFIG_SERIAL_8250_FOURPORT) += 8250_fourport.o
If unsure, say Y.
config SERIAL_ATMEL
- bool "AT91 / AT32 on-chip serial port support"
+ bool "AT91 on-chip serial port support"
depends on HAS_DMA
- depends on ARCH_AT91 || AVR32 || COMPILE_TEST
+ depends on ARCH_AT91 || COMPILE_TEST
select SERIAL_CORE
select SERIAL_MCTRL_GPIO if GPIOLIB
help
This enables the driver for the on-chip UARTs of the Atmel
- AT91 and AT32 processors.
+ AT91 processors.
config SERIAL_ATMEL_CONSOLE
- bool "Support for console on AT91 / AT32 serial port"
+ bool "Support for console on AT91 serial port"
depends on SERIAL_ATMEL=y
select SERIAL_CORE_CONSOLE
help
Say Y here if you wish to use an on-chip UART on a Atmel
- AT91 or AT32 processor as the system console (the system
+ AT91 processor as the system console (the system
console is the device which receives all kernel messages and
warnings and which allows logins in single user mode).
config SERIAL_ATMEL_PDC
- bool "Support DMA transfers on AT91 / AT32 serial port"
+ bool "Support DMA transfers on AT91 serial port"
depends on SERIAL_ATMEL
default y
help
Say Y here if you wish to use the PDC to do DMA transfers to
- and from the Atmel AT91 / AT32 serial port. In order to
+ and from the Atmel AT91 serial port. In order to
actually use DMA transfers, make sure that the use_dma_tx
and use_dma_rx members in the atmel_uart_data struct is set
appropriately for each port.
bool "Install as device ttyATn instead of ttySn"
depends on SERIAL_ATMEL=y
help
- Say Y here if you wish to have the internal AT91 / AT32 UARTs
+ Say Y here if you wish to have the internal AT91 UARTs
appear as /dev/ttyATn (major 204, minor starting at 154)
instead of the normal /dev/ttySn (major 4, minor starting at
64). This is necessary if you also want other UARTs, such as
and warnings and which allows logins in single user mode)
Otherwise, say 'N'.
+config SERIAL_OWL
+ bool "Actions Semi Owl serial port support"
+ depends on ARCH_ACTIONS || COMPILE_TEST
+ select SERIAL_CORE
+ help
+ This driver is for Actions Semiconductor S500/S900 SoC's UART.
+ Say 'Y' here if you wish to use the on-board serial port.
+ Otherwise, say 'N'.
+
+config SERIAL_OWL_CONSOLE
+ bool "Console on Actions Semi Owl serial port"
+ depends on SERIAL_OWL=y
+ select SERIAL_CORE_CONSOLE
+ select SERIAL_EARLYCON
+ default y
+ help
+ Say 'Y' here if you wish to use Actions Semiconductor S500/S900 UART
+ as the system console. Only earlycon is implemented currently.
+
endmenu
config SERIAL_MCTRL_GPIO
obj-$(CONFIG_SERIAL_MVEBU_UART) += mvebu-uart.o
obj-$(CONFIG_SERIAL_PIC32) += pic32_uart.o
obj-$(CONFIG_SERIAL_MPS2_UART) += mps2-uart.o
+obj-$(CONFIG_SERIAL_OWL) += owl-uart.o
# GPIOLIB helpers for modem control lines
obj-$(CONFIG_SERIAL_MCTRL_GPIO) += serial_mctrl_gpio.o
#define AMBA_CONSOLE NULL
#endif
+static DEFINE_MUTEX(amba_reg_lock);
static struct uart_driver amba_reg = {
.owner = THIS_MODULE,
.driver_name = "ttyAM",
amba_ports[i] = uap;
amba_set_drvdata(dev, uap);
+
+ mutex_lock(&amba_reg_lock);
+ if (!amba_reg.state) {
+ ret = uart_register_driver(&amba_reg);
+ if (ret < 0) {
+ mutex_unlock(&amba_reg_lock);
+ dev_err(uap->port.dev,
+ "Failed to register AMBA-PL010 driver\n");
+ return ret;
+ }
+ }
+ mutex_unlock(&amba_reg_lock);
+
ret = uart_add_one_port(&amba_reg, &uap->port);
if (ret)
amba_ports[i] = NULL;
{
struct uart_amba_port *uap = amba_get_drvdata(dev);
int i;
+ bool busy = false;
uart_remove_one_port(&amba_reg, &uap->port);
for (i = 0; i < ARRAY_SIZE(amba_ports); i++)
if (amba_ports[i] == uap)
amba_ports[i] = NULL;
+ else if (amba_ports[i])
+ busy = true;
+
+ if (!busy)
+ uart_unregister_driver(&amba_reg);
return 0;
}
static int __init pl010_init(void)
{
- int ret;
-
printk(KERN_INFO "Serial: AMBA driver\n");
- ret = uart_register_driver(&amba_reg);
- if (ret == 0) {
- ret = amba_driver_register(&pl010_driver);
- if (ret)
- uart_unregister_driver(&amba_reg);
- }
- return ret;
+ return amba_driver_register(&pl010_driver);
}
static void __exit pl010_exit(void)
{
amba_driver_unregister(&pl010_driver);
- uart_unregister_driver(&amba_reg);
}
module_init(pl010_init);
/*
- * Driver for Atmel AT91 / AT32 Serial ports
+ * Driver for Atmel AT91 Serial ports
* Copyright (C) 2003 Rick Bronson
*
* Based on drivers/char/serial_sa1100.c, by Deep Blue Solutions Ltd.
#include <linux/err.h>
#include <linux/irq.h>
#include <linux/suspend.h>
+#include <linux/mm.h>
#include <asm/io.h>
#include <asm/ioctls.h>
/*
* at91: 6 USARTs and one DBGU port (SAM9260)
- * avr32: 4
* samx7: 3 USARTs and 5 UARTs
*/
#define ATMEL_MAX_UART 8
__raw_writel(value, port->membase + reg);
}
-#ifdef CONFIG_AVR32
-
-/* AVR32 cannot handle 8 or 16bit I/O accesses but only 32bit I/O accesses */
-static inline u8 atmel_uart_read_char(struct uart_port *port)
-{
- return __raw_readl(port->membase + ATMEL_US_RHR);
-}
-
-static inline void atmel_uart_write_char(struct uart_port *port, u8 value)
-{
- __raw_writel(value, port->membase + ATMEL_US_THR);
-}
-
-#else
-
static inline u8 atmel_uart_read_char(struct uart_port *port)
{
return __raw_readb(port->membase + ATMEL_US_RHR);
__raw_writeb(value, port->membase + ATMEL_US_THR);
}
-#endif
-
#ifdef CONFIG_SERIAL_ATMEL_PDC
static bool atmel_use_pdc_rx(struct uart_port *port)
{
sg_set_page(&atmel_port->sg_tx,
virt_to_page(port->state->xmit.buf),
UART_XMIT_SIZE,
- (unsigned long)port->state->xmit.buf & ~PAGE_MASK);
+ offset_in_page(port->state->xmit.buf));
nent = dma_map_sg(port->dev,
&atmel_port->sg_tx,
1,
sg_set_page(&atmel_port->sg_rx,
virt_to_page(ring->buf),
sizeof(struct atmel_uart_char) * ATMEL_SERIAL_RINGSIZE,
- (unsigned long)ring->buf & ~PAGE_MASK);
+ offset_in_page(ring->buf));
nent = dma_map_sg(port->dev,
&atmel_port->sg_rx,
1,
struct platform_device *pdev)
{
struct device_node *np = pdev->dev.of_node;
- struct atmel_uart_data *pdata = dev_get_platdata(&pdev->dev);
-
- if (np) {
- /* DMA/PDC usage specification */
- if (of_property_read_bool(np, "atmel,use-dma-rx")) {
- if (of_property_read_bool(np, "dmas")) {
- atmel_port->use_dma_rx = true;
- atmel_port->use_pdc_rx = false;
- } else {
- atmel_port->use_dma_rx = false;
- atmel_port->use_pdc_rx = true;
- }
+
+ /* DMA/PDC usage specification */
+ if (of_property_read_bool(np, "atmel,use-dma-rx")) {
+ if (of_property_read_bool(np, "dmas")) {
+ atmel_port->use_dma_rx = true;
+ atmel_port->use_pdc_rx = false;
} else {
atmel_port->use_dma_rx = false;
- atmel_port->use_pdc_rx = false;
+ atmel_port->use_pdc_rx = true;
}
+ } else {
+ atmel_port->use_dma_rx = false;
+ atmel_port->use_pdc_rx = false;
+ }
- if (of_property_read_bool(np, "atmel,use-dma-tx")) {
- if (of_property_read_bool(np, "dmas")) {
- atmel_port->use_dma_tx = true;
- atmel_port->use_pdc_tx = false;
- } else {
- atmel_port->use_dma_tx = false;
- atmel_port->use_pdc_tx = true;
- }
+ if (of_property_read_bool(np, "atmel,use-dma-tx")) {
+ if (of_property_read_bool(np, "dmas")) {
+ atmel_port->use_dma_tx = true;
+ atmel_port->use_pdc_tx = false;
} else {
atmel_port->use_dma_tx = false;
- atmel_port->use_pdc_tx = false;
+ atmel_port->use_pdc_tx = true;
}
-
} else {
- atmel_port->use_pdc_rx = pdata->use_dma_rx;
- atmel_port->use_pdc_tx = pdata->use_dma_tx;
- atmel_port->use_dma_rx = false;
atmel_port->use_dma_tx = false;
+ atmel_port->use_pdc_tx = false;
}
-
}
static void atmel_init_rs485(struct uart_port *port,
struct platform_device *pdev)
{
struct device_node *np = pdev->dev.of_node;
- struct atmel_uart_data *pdata = dev_get_platdata(&pdev->dev);
-
- if (np) {
- struct serial_rs485 *rs485conf = &port->rs485;
- u32 rs485_delay[2];
- /* rs485 properties */
- if (of_property_read_u32_array(np, "rs485-rts-delay",
- rs485_delay, 2) == 0) {
- rs485conf->delay_rts_before_send = rs485_delay[0];
- rs485conf->delay_rts_after_send = rs485_delay[1];
- rs485conf->flags = 0;
- }
- if (of_get_property(np, "rs485-rx-during-tx", NULL))
- rs485conf->flags |= SER_RS485_RX_DURING_TX;
+ struct serial_rs485 *rs485conf = &port->rs485;
+ u32 rs485_delay[2];
- if (of_get_property(np, "linux,rs485-enabled-at-boot-time",
- NULL))
- rs485conf->flags |= SER_RS485_ENABLED;
- } else {
- port->rs485 = pdata->rs485;
+ /* rs485 properties */
+ if (of_property_read_u32_array(np, "rs485-rts-delay",
+ rs485_delay, 2) == 0) {
+ rs485conf->delay_rts_before_send = rs485_delay[0];
+ rs485conf->delay_rts_after_send = rs485_delay[1];
+ rs485conf->flags = 0;
}
+ if (of_get_property(np, "rs485-rx-during-tx", NULL))
+ rs485conf->flags |= SER_RS485_RX_DURING_TX;
+
+ if (of_get_property(np, "linux,rs485-enabled-at-boot-time", NULL))
+ rs485conf->flags |= SER_RS485_ENABLED;
}
static void atmel_set_ops(struct uart_port *port)
{
int ret;
struct uart_port *port = &atmel_port->uart;
- struct atmel_uart_data *pdata = dev_get_platdata(&pdev->dev);
atmel_init_property(atmel_port, pdev);
atmel_set_ops(port);
atmel_init_rs485(port, pdev);
port->iotype = UPIO_MEM;
- port->flags = UPF_BOOT_AUTOCONF;
+ port->flags = UPF_BOOT_AUTOCONF | UPF_IOREMAP;
port->ops = &atmel_pops;
port->fifosize = 1;
port->dev = &pdev->dev;
port->mapbase = pdev->resource[0].start;
port->irq = pdev->resource[1].start;
port->rs485_config = atmel_config_rs485;
+ port->membase = NULL;
memset(&atmel_port->rx_ring, 0, sizeof(atmel_port->rx_ring));
- if (pdata && pdata->regs) {
- /* Already mapped by setup code */
- port->membase = pdata->regs;
- } else {
- port->flags |= UPF_IOREMAP;
- port->membase = NULL;
- }
-
/* for console, the clock could already be configured */
if (!atmel_port->clk) {
atmel_port->clk = clk_get(&pdev->dev, "usart");
return 0;
}
-struct platform_device *atmel_default_console_device; /* the serial console device */
-
#ifdef CONFIG_SERIAL_ATMEL_CONSOLE
static void atmel_console_putchar(struct uart_port *port, int ch)
{
#define ATMEL_CONSOLE_DEVICE (&atmel_console)
-/*
- * Early console initialization (before VM subsystem initialized).
- */
-static int __init atmel_console_init(void)
-{
- int ret;
- if (atmel_default_console_device) {
- struct atmel_uart_data *pdata =
- dev_get_platdata(&atmel_default_console_device->dev);
- int id = pdata->num;
- struct atmel_uart_port *atmel_port = &atmel_ports[id];
-
- atmel_port->backup_imr = 0;
- atmel_port->uart.line = id;
-
- add_preferred_console(ATMEL_DEVICENAME, id, NULL);
- ret = atmel_init_port(atmel_port, atmel_default_console_device);
- if (ret)
- return ret;
- register_console(&atmel_console);
- }
-
- return 0;
-}
-
-console_initcall(atmel_console_init);
-
-/*
- * Late console initialization.
- */
-static int __init atmel_late_console_init(void)
-{
- if (atmel_default_console_device
- && !(atmel_console.flags & CON_ENABLED))
- register_console(&atmel_console);
-
- return 0;
-}
-
-core_initcall(atmel_late_console_init);
-
static inline bool atmel_is_console_port(struct uart_port *port)
{
return port->cons && port->cons->index == port->line;
{
struct atmel_uart_port *atmel_port;
struct device_node *np = pdev->dev.of_node;
- struct atmel_uart_data *pdata = dev_get_platdata(&pdev->dev);
void *data;
int ret = -ENODEV;
bool rs485_enabled;
BUILD_BUG_ON(ATMEL_SERIAL_RINGSIZE & (ATMEL_SERIAL_RINGSIZE - 1));
- if (np)
- ret = of_alias_get_id(np, "serial");
- else
- if (pdata)
- ret = pdata->num;
-
+ ret = of_alias_get_id(np, "serial");
if (ret < 0)
/* port id not found in platform data nor device-tree aliases:
* auto-enumerate it */
#define UARTBAUD_SBNS 0x00002000
#define UARTBAUD_SBR 0x00000000
#define UARTBAUD_SBR_MASK 0x1fff
+#define UARTBAUD_OSR_MASK 0x1f
+#define UARTBAUD_OSR_SHIFT 24
#define UARTSTAT_LBKDIF 0x80000000
#define UARTSTAT_RXEDGIF 0x40000000
#define DEV_NAME "ttyLP"
#define UART_NR 6
+/* IMX lpuart has four extra unused regs located at the beginning */
+#define IMX_REG_OFF 0x10
+
struct lpuart_port {
struct uart_port port;
struct clk *clk;
unsigned int txfifo_size;
unsigned int rxfifo_size;
- bool lpuart32;
bool lpuart_dma_tx_use;
bool lpuart_dma_rx_use;
wait_queue_head_t dma_wait;
};
+struct lpuart_soc_data {
+ char iotype;
+ u8 reg_off;
+};
+
+static const struct lpuart_soc_data vf_data = {
+ .iotype = UPIO_MEM,
+};
+
+static const struct lpuart_soc_data ls_data = {
+ .iotype = UPIO_MEM32BE,
+};
+
+static struct lpuart_soc_data imx_data = {
+ .iotype = UPIO_MEM32,
+ .reg_off = IMX_REG_OFF,
+};
+
static const struct of_device_id lpuart_dt_ids[] = {
- {
- .compatible = "fsl,vf610-lpuart",
- },
- {
- .compatible = "fsl,ls1021a-lpuart",
- },
+ { .compatible = "fsl,vf610-lpuart", .data = &vf_data, },
+ { .compatible = "fsl,ls1021a-lpuart", .data = &ls_data, },
+ { .compatible = "fsl,imx7ulp-lpuart", .data = &imx_data, },
{ /* sentinel */ }
};
MODULE_DEVICE_TABLE(of, lpuart_dt_ids);
/* Forward declare this for the dma callbacks*/
static void lpuart_dma_tx_complete(void *arg);
-static u32 lpuart32_read(void __iomem *addr)
+static inline u32 lpuart32_read(struct uart_port *port, u32 off)
{
- return ioread32be(addr);
+ switch (port->iotype) {
+ case UPIO_MEM32:
+ return readl(port->membase + off);
+ case UPIO_MEM32BE:
+ return ioread32be(port->membase + off);
+ default:
+ return 0;
+ }
}
-static void lpuart32_write(u32 val, void __iomem *addr)
+static inline void lpuart32_write(struct uart_port *port, u32 val,
+ u32 off)
{
- iowrite32be(val, addr);
+ switch (port->iotype) {
+ case UPIO_MEM32:
+ writel(val, port->membase + off);
+ break;
+ case UPIO_MEM32BE:
+ iowrite32be(val, port->membase + off);
+ break;
+ }
}
static void lpuart_stop_tx(struct uart_port *port)
{
unsigned long temp;
- temp = lpuart32_read(port->membase + UARTCTRL);
+ temp = lpuart32_read(port, UARTCTRL);
temp &= ~(UARTCTRL_TIE | UARTCTRL_TCIE);
- lpuart32_write(temp, port->membase + UARTCTRL);
+ lpuart32_write(port, temp, UARTCTRL);
}
static void lpuart_stop_rx(struct uart_port *port)
{
unsigned long temp;
- temp = lpuart32_read(port->membase + UARTCTRL);
- lpuart32_write(temp & ~UARTCTRL_RE, port->membase + UARTCTRL);
+ temp = lpuart32_read(port, UARTCTRL);
+ lpuart32_write(port, temp & ~UARTCTRL_RE, UARTCTRL);
}
static void lpuart_dma_tx(struct lpuart_port *sport)
struct circ_buf *xmit = &sport->port.state->xmit;
unsigned long txcnt;
- txcnt = lpuart32_read(sport->port.membase + UARTWATER);
+ txcnt = lpuart32_read(&sport->port, UARTWATER);
txcnt = txcnt >> UARTWATER_TXCNT_OFF;
txcnt &= UARTWATER_COUNT_MASK;
while (!uart_circ_empty(xmit) && (txcnt < sport->txfifo_size)) {
- lpuart32_write(xmit->buf[xmit->tail], sport->port.membase + UARTDATA);
+ lpuart32_write(&sport->port, xmit->buf[xmit->tail], UARTDATA);
xmit->tail = (xmit->tail + 1) & (UART_XMIT_SIZE - 1);
sport->port.icount.tx++;
- txcnt = lpuart32_read(sport->port.membase + UARTWATER);
+ txcnt = lpuart32_read(&sport->port, UARTWATER);
txcnt = txcnt >> UARTWATER_TXCNT_OFF;
txcnt &= UARTWATER_COUNT_MASK;
}
struct lpuart_port *sport = container_of(port, struct lpuart_port, port);
unsigned long temp;
- temp = lpuart32_read(port->membase + UARTCTRL);
- lpuart32_write(temp | UARTCTRL_TIE, port->membase + UARTCTRL);
+ temp = lpuart32_read(port, UARTCTRL);
+ lpuart32_write(port, temp | UARTCTRL_TIE, UARTCTRL);
- if (lpuart32_read(port->membase + UARTSTAT) & UARTSTAT_TDRE)
+ if (lpuart32_read(port, UARTSTAT) & UARTSTAT_TDRE)
lpuart32_transmit_buffer(sport);
}
static unsigned int lpuart32_tx_empty(struct uart_port *port)
{
- return (lpuart32_read(port->membase + UARTSTAT) & UARTSTAT_TC) ?
+ return (lpuart32_read(port, UARTSTAT) & UARTSTAT_TC) ?
TIOCSER_TEMT : 0;
}
spin_lock_irqsave(&sport->port.lock, flags);
if (sport->port.x_char) {
- if (sport->lpuart32)
- lpuart32_write(sport->port.x_char, sport->port.membase + UARTDATA);
+ if (sport->port.iotype & (UPIO_MEM32 | UPIO_MEM32BE))
+ lpuart32_write(&sport->port, sport->port.x_char, UARTDATA);
else
writeb(sport->port.x_char, sport->port.membase + UARTDR);
goto out;
}
if (uart_circ_empty(xmit) || uart_tx_stopped(&sport->port)) {
- if (sport->lpuart32)
+ if (sport->port.iotype & (UPIO_MEM32 | UPIO_MEM32BE))
lpuart32_stop_tx(&sport->port);
else
lpuart_stop_tx(&sport->port);
goto out;
}
- if (sport->lpuart32)
+ if (sport->port.iotype & (UPIO_MEM32 | UPIO_MEM32BE))
lpuart32_transmit_buffer(sport);
else
lpuart_transmit_buffer(sport);
spin_lock_irqsave(&sport->port.lock, flags);
- while (!(lpuart32_read(sport->port.membase + UARTFIFO) & UARTFIFO_RXEMPT)) {
+ while (!(lpuart32_read(&sport->port, UARTFIFO) & UARTFIFO_RXEMPT)) {
flg = TTY_NORMAL;
sport->port.icount.rx++;
/*
* to clear the FE, OR, NF, FE, PE flags,
* read STAT then read DATA reg
*/
- sr = lpuart32_read(sport->port.membase + UARTSTAT);
- rx = lpuart32_read(sport->port.membase + UARTDATA);
+ sr = lpuart32_read(&sport->port, UARTSTAT);
+ rx = lpuart32_read(&sport->port, UARTDATA);
rx &= 0x3ff;
if (uart_handle_sysrq_char(&sport->port, (unsigned char)rx))
struct lpuart_port *sport = dev_id;
unsigned long sts, rxcount;
- sts = lpuart32_read(sport->port.membase + UARTSTAT);
- rxcount = lpuart32_read(sport->port.membase + UARTWATER);
+ sts = lpuart32_read(&sport->port, UARTSTAT);
+ rxcount = lpuart32_read(&sport->port, UARTWATER);
rxcount = rxcount >> UARTWATER_RXCNT_OFF;
if (sts & UARTSTAT_RDRF || rxcount > 0)
lpuart32_rxint(irq, dev_id);
if ((sts & UARTSTAT_TDRE) &&
- !(lpuart32_read(sport->port.membase + UARTBAUD) & UARTBAUD_TDMAE))
+ !(lpuart32_read(&sport->port, UARTBAUD) & UARTBAUD_TDMAE))
lpuart_txint(irq, dev_id);
- lpuart32_write(sts, sport->port.membase + UARTSTAT);
+ lpuart32_write(&sport->port, sts, UARTSTAT);
return IRQ_HANDLED;
}
unsigned int temp = 0;
unsigned long reg;
- reg = lpuart32_read(port->membase + UARTMODIR);
+ reg = lpuart32_read(port, UARTMODIR);
if (reg & UARTMODIR_TXCTSE)
temp |= TIOCM_CTS;
{
unsigned long temp;
- temp = lpuart32_read(port->membase + UARTMODIR) &
+ temp = lpuart32_read(port, UARTMODIR) &
~(UARTMODIR_RXRTSE | UARTMODIR_TXCTSE);
if (mctrl & TIOCM_RTS)
if (mctrl & TIOCM_CTS)
temp |= UARTMODIR_TXCTSE;
- lpuart32_write(temp, port->membase + UARTMODIR);
+ lpuart32_write(port, temp, UARTMODIR);
}
static void lpuart_break_ctl(struct uart_port *port, int break_state)
{
unsigned long temp;
- temp = lpuart32_read(port->membase + UARTCTRL) & ~UARTCTRL_SBK;
+ temp = lpuart32_read(port, UARTCTRL) & ~UARTCTRL_SBK;
if (break_state != 0)
temp |= UARTCTRL_SBK;
- lpuart32_write(temp, port->membase + UARTCTRL);
+ lpuart32_write(port, temp, UARTCTRL);
}
static void lpuart_setup_watermark(struct lpuart_port *sport)
unsigned long val, ctrl;
unsigned long ctrl_saved;
- ctrl = lpuart32_read(sport->port.membase + UARTCTRL);
+ ctrl = lpuart32_read(&sport->port, UARTCTRL);
ctrl_saved = ctrl;
ctrl &= ~(UARTCTRL_TIE | UARTCTRL_TCIE | UARTCTRL_TE |
UARTCTRL_RIE | UARTCTRL_RE);
- lpuart32_write(ctrl, sport->port.membase + UARTCTRL);
+ lpuart32_write(&sport->port, ctrl, UARTCTRL);
/* enable FIFO mode */
- val = lpuart32_read(sport->port.membase + UARTFIFO);
+ val = lpuart32_read(&sport->port, UARTFIFO);
val |= UARTFIFO_TXFE | UARTFIFO_RXFE;
val |= UARTFIFO_TXFLUSH | UARTFIFO_RXFLUSH;
- lpuart32_write(val, sport->port.membase + UARTFIFO);
+ lpuart32_write(&sport->port, val, UARTFIFO);
/* set the watermark */
val = (0x1 << UARTWATER_RXWATER_OFF) | (0x0 << UARTWATER_TXWATER_OFF);
- lpuart32_write(val, sport->port.membase + UARTWATER);
+ lpuart32_write(&sport->port, val, UARTWATER);
/* Restore cr2 */
- lpuart32_write(ctrl_saved, sport->port.membase + UARTCTRL);
+ lpuart32_write(&sport->port, ctrl_saved, UARTCTRL);
}
static void rx_dma_timer_init(struct lpuart_port *sport)
unsigned long temp;
/* determine FIFO size */
- temp = lpuart32_read(sport->port.membase + UARTFIFO);
+ temp = lpuart32_read(&sport->port, UARTFIFO);
sport->txfifo_size = 0x1 << (((temp >> UARTFIFO_TXSIZE_OFF) &
UARTFIFO_FIFOSIZE_MASK) - 1);
lpuart32_setup_watermark(sport);
- temp = lpuart32_read(sport->port.membase + UARTCTRL);
+ temp = lpuart32_read(&sport->port, UARTCTRL);
temp |= (UARTCTRL_RIE | UARTCTRL_TIE | UARTCTRL_RE | UARTCTRL_TE);
temp |= UARTCTRL_ILIE;
- lpuart32_write(temp, sport->port.membase + UARTCTRL);
+ lpuart32_write(&sport->port, temp, UARTCTRL);
spin_unlock_irqrestore(&sport->port.lock, flags);
return 0;
spin_lock_irqsave(&port->lock, flags);
/* disable Rx/Tx and interrupts */
- temp = lpuart32_read(port->membase + UARTCTRL);
+ temp = lpuart32_read(port, UARTCTRL);
temp &= ~(UARTCTRL_TE | UARTCTRL_RE |
UARTCTRL_TIE | UARTCTRL_TCIE | UARTCTRL_RIE);
- lpuart32_write(temp, port->membase + UARTCTRL);
+ lpuart32_write(port, temp, UARTCTRL);
spin_unlock_irqrestore(&port->lock, flags);
spin_unlock_irqrestore(&sport->port.lock, flags);
}
+static void
+lpuart32_serial_setbrg(struct lpuart_port *sport, unsigned int baudrate)
+{
+ u32 sbr, osr, baud_diff, tmp_osr, tmp_sbr, tmp_diff, tmp;
+ u32 clk = sport->port.uartclk;
+
+ /*
+ * The idea is to use the best OSR (over-sampling rate) possible.
+ * Note, OSR is typically hard-set to 16 in other LPUART instantiations.
+ * Loop to find the best OSR value possible, one that generates minimum
+ * baud_diff iterate through the rest of the supported values of OSR.
+ *
+ * Calculation Formula:
+ * Baud Rate = baud clock / ((OSR+1) × SBR)
+ */
+ baud_diff = baudrate;
+ osr = 0;
+ sbr = 0;
+
+ for (tmp_osr = 4; tmp_osr <= 32; tmp_osr++) {
+ /* calculate the temporary sbr value */
+ tmp_sbr = (clk / (baudrate * tmp_osr));
+ if (tmp_sbr == 0)
+ tmp_sbr = 1;
+
+ /*
+ * calculate the baud rate difference based on the temporary
+ * osr and sbr values
+ */
+ tmp_diff = clk / (tmp_osr * tmp_sbr) - baudrate;
+
+ /* select best values between sbr and sbr+1 */
+ tmp = clk / (tmp_osr * (tmp_sbr + 1));
+ if (tmp_diff > (baudrate - tmp)) {
+ tmp_diff = baudrate - tmp;
+ tmp_sbr++;
+ }
+
+ if (tmp_diff <= baud_diff) {
+ baud_diff = tmp_diff;
+ osr = tmp_osr;
+ sbr = tmp_sbr;
+
+ if (!baud_diff)
+ break;
+ }
+ }
+
+ /* handle buadrate outside acceptable rate */
+ if (baud_diff > ((baudrate / 100) * 3))
+ dev_warn(sport->port.dev,
+ "unacceptable baud rate difference of more than 3%%\n");
+
+ tmp = lpuart32_read(&sport->port, UARTBAUD);
+
+ if ((osr > 3) && (osr < 8))
+ tmp |= UARTBAUD_BOTHEDGE;
+
+ tmp &= ~(UARTBAUD_OSR_MASK << UARTBAUD_OSR_SHIFT);
+ tmp |= (((osr-1) & UARTBAUD_OSR_MASK) << UARTBAUD_OSR_SHIFT);
+
+ tmp &= ~UARTBAUD_SBR_MASK;
+ tmp |= sbr & UARTBAUD_SBR_MASK;
+
+ tmp &= ~(UARTBAUD_TDMAE | UARTBAUD_RDMAE);
+
+ lpuart32_write(&sport->port, tmp, UARTBAUD);
+}
+
static void
lpuart32_set_termios(struct uart_port *port, struct ktermios *termios,
struct ktermios *old)
unsigned long ctrl, old_ctrl, bd, modem;
unsigned int baud;
unsigned int old_csize = old ? old->c_cflag & CSIZE : CS8;
- unsigned int sbr;
- ctrl = old_ctrl = lpuart32_read(sport->port.membase + UARTCTRL);
- bd = lpuart32_read(sport->port.membase + UARTBAUD);
- modem = lpuart32_read(sport->port.membase + UARTMODIR);
+ ctrl = old_ctrl = lpuart32_read(&sport->port, UARTCTRL);
+ bd = lpuart32_read(&sport->port, UARTBAUD);
+ modem = lpuart32_read(&sport->port, UARTMODIR);
/*
* only support CS8 and CS7, and for CS7 must enable PE.
* supported mode:
uart_update_timeout(port, termios->c_cflag, baud);
/* wait transmit engin complete */
- while (!(lpuart32_read(sport->port.membase + UARTSTAT) & UARTSTAT_TC))
+ while (!(lpuart32_read(&sport->port, UARTSTAT) & UARTSTAT_TC))
barrier();
/* disable transmit and receive */
- lpuart32_write(old_ctrl & ~(UARTCTRL_TE | UARTCTRL_RE),
- sport->port.membase + UARTCTRL);
+ lpuart32_write(&sport->port, old_ctrl & ~(UARTCTRL_TE | UARTCTRL_RE),
+ UARTCTRL);
- sbr = sport->port.uartclk / (16 * baud);
- bd &= ~UARTBAUD_SBR_MASK;
- bd |= sbr & UARTBAUD_SBR_MASK;
- bd |= UARTBAUD_BOTHEDGE;
- bd &= ~(UARTBAUD_TDMAE | UARTBAUD_RDMAE);
- lpuart32_write(bd, sport->port.membase + UARTBAUD);
- lpuart32_write(modem, sport->port.membase + UARTMODIR);
- lpuart32_write(ctrl, sport->port.membase + UARTCTRL);
+ lpuart32_serial_setbrg(sport, baud);
+ lpuart32_write(&sport->port, modem, UARTMODIR);
+ lpuart32_write(&sport->port, ctrl, UARTCTRL);
/* restore control register */
spin_unlock_irqrestore(&sport->port.lock, flags);
static void lpuart32_console_putchar(struct uart_port *port, int ch)
{
- while (!(lpuart32_read(port->membase + UARTSTAT) & UARTSTAT_TDRE))
+ while (!(lpuart32_read(port, UARTSTAT) & UARTSTAT_TDRE))
barrier();
- lpuart32_write(ch, port->membase + UARTDATA);
+ lpuart32_write(port, ch, UARTDATA);
}
static void
spin_lock_irqsave(&sport->port.lock, flags);
/* first save CR2 and then disable interrupts */
- cr = old_cr = lpuart32_read(sport->port.membase + UARTCTRL);
+ cr = old_cr = lpuart32_read(&sport->port, UARTCTRL);
cr |= (UARTCTRL_TE | UARTCTRL_RE);
cr &= ~(UARTCTRL_TIE | UARTCTRL_TCIE | UARTCTRL_RIE);
- lpuart32_write(cr, sport->port.membase + UARTCTRL);
+ lpuart32_write(&sport->port, cr, UARTCTRL);
uart_console_write(&sport->port, s, count, lpuart32_console_putchar);
/* wait for transmitter finish complete and restore CR2 */
- while (!(lpuart32_read(sport->port.membase + UARTSTAT) & UARTSTAT_TC))
+ while (!(lpuart32_read(&sport->port, UARTSTAT) & UARTSTAT_TC))
barrier();
- lpuart32_write(old_cr, sport->port.membase + UARTCTRL);
+ lpuart32_write(&sport->port, old_cr, UARTCTRL);
if (locked)
spin_unlock_irqrestore(&sport->port.lock, flags);
unsigned long cr, bd;
unsigned int sbr, uartclk, baud_raw;
- cr = lpuart32_read(sport->port.membase + UARTCTRL);
+ cr = lpuart32_read(&sport->port, UARTCTRL);
cr &= UARTCTRL_TE | UARTCTRL_RE;
if (!cr)
return;
/* ok, the port was enabled */
- cr = lpuart32_read(sport->port.membase + UARTCTRL);
+ cr = lpuart32_read(&sport->port, UARTCTRL);
*parity = 'n';
if (cr & UARTCTRL_PE) {
else
*bits = 8;
- bd = lpuart32_read(sport->port.membase + UARTBAUD);
+ bd = lpuart32_read(&sport->port, UARTBAUD);
bd &= UARTBAUD_SBR_MASK;
sbr = bd;
uartclk = clk_get_rate(sport->clk);
if (options)
uart_parse_options(options, &baud, &parity, &bits, &flow);
else
- if (sport->lpuart32)
+ if (sport->port.iotype & (UPIO_MEM32 | UPIO_MEM32BE))
lpuart32_console_get_options(sport, &baud, &parity, &bits);
else
lpuart_console_get_options(sport, &baud, &parity, &bits);
- if (sport->lpuart32)
+ if (sport->port.iotype & (UPIO_MEM32 | UPIO_MEM32BE))
lpuart32_setup_watermark(sport);
else
lpuart_setup_watermark(sport);
if (!device->port.membase)
return -ENODEV;
+ device->port.iotype = UPIO_MEM32BE;
device->con->write = lpuart32_early_write;
return 0;
}
+static int __init lpuart32_imx_early_console_setup(struct earlycon_device *device,
+ const char *opt)
+{
+ if (!device->port.membase)
+ return -ENODEV;
+
+ device->port.iotype = UPIO_MEM32;
+ device->port.membase += IMX_REG_OFF;
+ device->con->write = lpuart32_early_write;
+
+ return 0;
+}
OF_EARLYCON_DECLARE(lpuart, "fsl,vf610-lpuart", lpuart_early_console_setup);
OF_EARLYCON_DECLARE(lpuart32, "fsl,ls1021a-lpuart", lpuart32_early_console_setup);
+OF_EARLYCON_DECLARE(lpuart32, "fsl,imx7ulp-lpuart", lpuart32_imx_early_console_setup);
EARLYCON_DECLARE(lpuart, lpuart_early_console_setup);
EARLYCON_DECLARE(lpuart32, lpuart32_early_console_setup);
static int lpuart_probe(struct platform_device *pdev)
{
+ const struct of_device_id *of_id = of_match_device(lpuart_dt_ids,
+ &pdev->dev);
+ const struct lpuart_soc_data *sdata = of_id->data;
struct device_node *np = pdev->dev.of_node;
struct lpuart_port *sport;
struct resource *res;
return ret;
}
sport->port.line = ret;
- sport->lpuart32 = of_device_is_compatible(np, "fsl,ls1021a-lpuart");
-
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
sport->port.membase = devm_ioremap_resource(&pdev->dev, res);
if (IS_ERR(sport->port.membase))
return PTR_ERR(sport->port.membase);
+ sport->port.membase += sdata->reg_off;
sport->port.mapbase = res->start;
sport->port.dev = &pdev->dev;
sport->port.type = PORT_LPUART;
- sport->port.iotype = UPIO_MEM;
ret = platform_get_irq(pdev, 0);
if (ret < 0) {
dev_err(&pdev->dev, "cannot obtain irq\n");
return ret;
}
sport->port.irq = ret;
-
- if (sport->lpuart32)
+ sport->port.iotype = sdata->iotype;
+ if (sport->port.iotype & (UPIO_MEM32 | UPIO_MEM32BE))
sport->port.ops = &lpuart32_pops;
else
sport->port.ops = &lpuart_pops;
platform_set_drvdata(pdev, &sport->port);
- if (sport->lpuart32)
+ if (sport->port.iotype & (UPIO_MEM32 | UPIO_MEM32BE))
lpuart_reg.cons = LPUART32_CONSOLE;
else
lpuart_reg.cons = LPUART_CONSOLE;
struct lpuart_port *sport = dev_get_drvdata(dev);
unsigned long temp;
- if (sport->lpuart32) {
+ if (sport->port.iotype & (UPIO_MEM32 | UPIO_MEM32BE)) {
/* disable Rx/Tx and interrupts */
- temp = lpuart32_read(sport->port.membase + UARTCTRL);
+ temp = lpuart32_read(&sport->port, UARTCTRL);
temp &= ~(UARTCTRL_TE | UARTCTRL_TIE | UARTCTRL_TCIE);
- lpuart32_write(temp, sport->port.membase + UARTCTRL);
+ lpuart32_write(&sport->port, temp, UARTCTRL);
} else {
/* disable Rx/Tx and interrupts */
temp = readb(sport->port.membase + UARTCR2);
if (sport->port.suspended && !sport->port.irq_wake)
clk_prepare_enable(sport->clk);
- if (sport->lpuart32) {
+ if (sport->port.iotype & (UPIO_MEM32 | UPIO_MEM32BE)) {
lpuart32_setup_watermark(sport);
- temp = lpuart32_read(sport->port.membase + UARTCTRL);
+ temp = lpuart32_read(&sport->port, UARTCTRL);
temp |= (UARTCTRL_RIE | UARTCTRL_TIE | UARTCTRL_RE |
UARTCTRL_TE | UARTCTRL_ILIE);
- lpuart32_write(temp, sport->port.membase + UARTCTRL);
+ lpuart32_write(&sport->port, temp, UARTCTRL);
} else {
lpuart_setup_watermark(sport);
temp = readb(sport->port.membase + UARTCR2);
#define UART_NR 8
+/* RX DMA buffer periods */
+#define RX_DMA_PERIODS 4
+#define RX_BUF_SIZE (PAGE_SIZE)
+
+
/* i.MX21 type uart runs on all i.mx except i.MX1 and i.MX6q */
enum imx_uart_type {
IMX1_UART,
unsigned int have_rtscts:1;
unsigned int have_rtsgpio:1;
unsigned int dte_mode:1;
- unsigned int irda_inv_rx:1;
- unsigned int irda_inv_tx:1;
- unsigned short trcv_delay; /* transceiver delay */
struct clk *clk_ipg;
struct clk *clk_per;
const struct imx_uart_data *devdata;
struct dma_chan *dma_chan_rx, *dma_chan_tx;
struct scatterlist rx_sgl, tx_sgl[2];
void *rx_buf;
+ unsigned int rx_buf_size;
struct circ_buf rx_ring;
unsigned int rx_periods;
dma_cookie_t rx_cookie;
}
}
-#define RX_BUF_SIZE (PAGE_SIZE)
-
/*
* There are two kinds of RX DMA interrupts(such as in the MX6Q):
* [1] the RX DMA buffer is full.
}
}
-/* RX DMA buffer periods */
-#define RX_DMA_PERIODS 4
-
static int start_rx_dma(struct imx_port *sport)
{
struct scatterlist *sgl = &sport->rx_sgl;
sport->rx_ring.head = 0;
sport->rx_ring.tail = 0;
- sport->rx_periods = RX_DMA_PERIODS;
- sg_init_one(sgl, sport->rx_buf, RX_BUF_SIZE);
+ sg_init_one(sgl, sport->rx_buf, sport->rx_buf_size);
ret = dma_map_sg(dev, sgl, 1, DMA_FROM_DEVICE);
if (ret == 0) {
dev_err(dev, "DMA mapping error for RX.\n");
goto err;
}
- sport->rx_buf = kzalloc(PAGE_SIZE, GFP_KERNEL);
+ sport->rx_buf = kzalloc(sport->rx_buf_size, GFP_KERNEL);
if (!sport->rx_buf) {
ret = -ENOMEM;
goto err;
imx_enable_dma(sport);
temp = readl(sport->port.membase + UCR1);
- temp |= UCR1_RRDYEN | UCR1_RTSDEN | UCR1_UARTEN;
+ temp |= UCR1_RRDYEN | UCR1_UARTEN;
+ if (sport->have_rtscts)
+ temp |= UCR1_RTSDEN;
writel(temp, sport->port.membase + UCR1);
imx_enable_ms(&sport->port);
/*
- * If the serial port is opened for reading start RX DMA immediately
- * instead of waiting for RX FIFO interrupts. In our iMX53 the average
- * delay for the first reception dropped from approximately 35000
- * microseconds to 1000 microseconds.
+ * Start RX DMA immediately instead of waiting for RX FIFO interrupts.
+ * In our iMX53 the average delay for the first reception dropped from
+ * approximately 35000 microseconds to 1000 microseconds.
*/
if (sport->dma_is_enabled) {
- struct tty_struct *tty = sport->port.state->port.tty;
- struct tty_file_private *file_priv;
- int readcnt = 0;
-
- spin_lock(&tty->files_lock);
-
- if (!list_empty(&tty->tty_files))
- list_for_each_entry(file_priv, &tty->tty_files, list)
- if (!(file_priv->file->f_flags & O_WRONLY))
- readcnt++;
-
- spin_unlock(&tty->files_lock);
-
- if (readcnt > 0) {
- imx_disable_rx_int(sport);
- start_rx_dma(sport);
- }
+ imx_disable_rx_int(sport);
+ start_rx_dma(sport);
}
spin_unlock_irqrestore(&sport->port.lock, flags);
{
struct device_node *np = pdev->dev.of_node;
int ret;
+ u32 dma_buf_size[2];
sport->devdata = of_device_get_match_data(&pdev->dev);
if (!sport->devdata)
if (of_get_property(np, "rts-gpios", NULL))
sport->have_rtsgpio = 1;
+ if (!of_property_read_u32_array(np, "fsl,dma-size", dma_buf_size, 2)) {
+ sport->rx_buf_size = dma_buf_size[0] * dma_buf_size[1];
+ sport->rx_periods = dma_buf_size[1];
+ } else {
+ sport->rx_buf_size = RX_BUF_SIZE;
+ sport->rx_periods = RX_DMA_PERIODS;
+ }
+
return 0;
}
#else
writel(val, port->membase + AML_UART_MISC);
ret = request_irq(port->irq, meson_uart_interrupt, 0,
- meson_uart_type(port), port);
+ port->name, port);
return ret;
}
while (!meson_uart_tx_empty(port))
cpu_relax();
- val = readl(port->membase + AML_UART_REG5);
- val &= ~AML_UART_BAUD_MASK;
if (port->uartclk == 24000000) {
val = ((port->uartclk / 3) / baud) - 1;
val |= AML_UART_BAUD_XTAL;
if (cflags & CSTOPB)
val |= AML_UART_STOP_BIN_2SB;
else
- val &= ~AML_UART_STOP_BIN_1SB;
+ val |= AML_UART_STOP_BIN_1SB;
if (cflags & CRTSCTS)
val &= ~AML_UART_TWO_WIRE_EN;
return ret;
}
-static int meson_uart_res_size(struct uart_port *port)
-{
- struct platform_device *pdev = to_platform_device(port->dev);
- struct resource *res;
-
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- if (!res) {
- dev_err(port->dev, "cannot obtain I/O memory region");
- return -ENODEV;
- }
-
- return resource_size(res);
-}
-
static void meson_uart_release_port(struct uart_port *port)
{
- int size = meson_uart_res_size(port);
-
- if (port->flags & UPF_IOREMAP) {
- devm_release_mem_region(port->dev, port->mapbase, size);
- devm_iounmap(port->dev, port->membase);
- port->membase = NULL;
- }
+ devm_iounmap(port->dev, port->membase);
+ port->membase = NULL;
+ devm_release_mem_region(port->dev, port->mapbase, port->mapsize);
}
static int meson_uart_request_port(struct uart_port *port)
{
- int size = meson_uart_res_size(port);
-
- if (size < 0)
- return size;
-
- if (!devm_request_mem_region(port->dev, port->mapbase, size,
+ if (!devm_request_mem_region(port->dev, port->mapbase, port->mapsize,
dev_name(port->dev))) {
dev_err(port->dev, "Memory region busy\n");
return -EBUSY;
}
- if (port->flags & UPF_IOREMAP) {
- port->membase = devm_ioremap_nocache(port->dev,
- port->mapbase,
- size);
- if (port->membase == NULL)
- return -ENOMEM;
- }
+ port->membase = devm_ioremap_nocache(port->dev, port->mapbase,
+ port->mapsize);
+ if (!port->membase)
+ return -ENOMEM;
return 0;
}
};
#ifdef CONFIG_SERIAL_MESON_CONSOLE
+static void meson_uart_enable_tx_engine(struct uart_port *port)
+{
+ u32 val;
+
+ val = readl(port->membase + AML_UART_CONTROL);
+ val |= AML_UART_TX_EN;
+ writel(val, port->membase + AML_UART_CONTROL);
+}
static void meson_console_putchar(struct uart_port *port, int ch)
{
}
val = readl(port->membase + AML_UART_CONTROL);
- val |= AML_UART_TX_EN;
tmp = val & ~(AML_UART_TX_INT_EN | AML_UART_RX_INT_EN);
writel(tmp, port->membase + AML_UART_CONTROL);
if (!port || !port->membase)
return -ENODEV;
+ meson_uart_enable_tx_engine(port);
+
if (options)
uart_parse_options(options, &baud, &parity, &bits, &flow);
if (!device->port.membase)
return -ENODEV;
+ meson_uart_enable_tx_engine(&device->port);
device->con->write = meson_serial_early_console_write;
return 0;
}
+/* Legacy bindings, should be removed when no more used */
OF_EARLYCON_DECLARE(meson, "amlogic,meson-uart",
meson_serial_early_console_setup);
+/* Stable bindings */
+OF_EARLYCON_DECLARE(meson, "amlogic,meson-ao-uart",
+ meson_serial_early_console_setup);
#define MESON_SERIAL_CONSOLE (&meson_serial_console)
#else
.cons = MESON_SERIAL_CONSOLE,
};
+static inline struct clk *meson_uart_probe_clock(struct device *dev,
+ const char *id)
+{
+ struct clk *clk = NULL;
+ int ret;
+
+ clk = devm_clk_get(dev, id);
+ if (IS_ERR(clk))
+ return clk;
+
+ ret = clk_prepare_enable(clk);
+ if (ret) {
+ dev_err(dev, "couldn't enable clk\n");
+ return ERR_PTR(ret);
+ }
+
+ devm_add_action_or_reset(dev,
+ (void(*)(void *))clk_disable_unprepare,
+ clk);
+
+ return clk;
+}
+
+/*
+ * This function gets clocks in the legacy non-stable DT bindings.
+ * This code will be remove once all the platforms switch to the
+ * new DT bindings.
+ */
+static int meson_uart_probe_clocks_legacy(struct platform_device *pdev,
+ struct uart_port *port)
+{
+ struct clk *clk = NULL;
+
+ clk = meson_uart_probe_clock(&pdev->dev, NULL);
+ if (IS_ERR(clk))
+ return PTR_ERR(clk);
+
+ port->uartclk = clk_get_rate(clk);
+
+ return 0;
+}
+
+static int meson_uart_probe_clocks(struct platform_device *pdev,
+ struct uart_port *port)
+{
+ struct clk *clk_xtal = NULL;
+ struct clk *clk_pclk = NULL;
+ struct clk *clk_baud = NULL;
+
+ clk_pclk = meson_uart_probe_clock(&pdev->dev, "pclk");
+ if (IS_ERR(clk_pclk))
+ return PTR_ERR(clk_pclk);
+
+ clk_xtal = meson_uart_probe_clock(&pdev->dev, "xtal");
+ if (IS_ERR(clk_xtal))
+ return PTR_ERR(clk_xtal);
+
+ clk_baud = meson_uart_probe_clock(&pdev->dev, "baud");
+ if (IS_ERR(clk_baud))
+ return PTR_ERR(clk_baud);
+
+ port->uartclk = clk_get_rate(clk_baud);
+
+ return 0;
+}
+
static int meson_uart_probe(struct platform_device *pdev)
{
struct resource *res_mem, *res_irq;
struct uart_port *port;
- struct clk *clk;
int ret = 0;
if (pdev->dev.of_node)
if (!port)
return -ENOMEM;
- clk = clk_get(&pdev->dev, NULL);
- if (IS_ERR(clk))
- return PTR_ERR(clk);
+ /* Use legacy way until all platforms switch to new bindings */
+ if (of_device_is_compatible(pdev->dev.of_node, "amlogic,meson-uart"))
+ ret = meson_uart_probe_clocks_legacy(pdev, port);
+ else
+ ret = meson_uart_probe_clocks(pdev, port);
+
+ if (ret)
+ return ret;
- port->uartclk = clk_get_rate(clk);
port->iotype = UPIO_MEM;
port->mapbase = res_mem->start;
+ port->mapsize = resource_size(res_mem);
port->irq = res_irq->start;
- port->flags = UPF_BOOT_AUTOCONF | UPF_IOREMAP | UPF_LOW_LATENCY;
+ port->flags = UPF_BOOT_AUTOCONF | UPF_LOW_LATENCY;
port->dev = &pdev->dev;
port->line = pdev->id;
port->type = PORT_MESON;
return 0;
}
-
static const struct of_device_id meson_uart_dt_match[] = {
+ /* Legacy bindings, should be removed when no more used */
{ .compatible = "amlogic,meson-uart" },
+ /* Stable bindings */
+ { .compatible = "amlogic,meson6-uart" },
+ { .compatible = "amlogic,meson8-uart" },
+ { .compatible = "amlogic,meson8b-uart" },
+ { .compatible = "amlogic,meson-gx-uart" },
{ /* sentinel */ },
};
MODULE_DEVICE_TABLE(of, meson_uart_dt_match);
if (!dma_set_mask(pi->port.dev, 0xffffffff)) {
printk(KERN_ERR "MPSC: Inadequate DMA support\n");
rc = -ENXIO;
- } else if ((pi->dma_region = dma_alloc_noncoherent(pi->port.dev,
+ } else if ((pi->dma_region = dma_alloc_attrs(pi->port.dev,
MPSC_DMA_ALLOC_SIZE,
- &pi->dma_region_p, GFP_KERNEL))
+ &pi->dma_region_p, GFP_KERNEL,
+ DMA_ATTR_NON_CONSISTENT))
== NULL) {
printk(KERN_ERR "MPSC: Can't alloc Desc region\n");
rc = -ENOMEM;
pr_debug("mpsc_free_ring_mem[%d]: Freeing ring mem\n", pi->port.line);
if (pi->dma_region) {
- dma_free_noncoherent(pi->port.dev, MPSC_DMA_ALLOC_SIZE,
- pi->dma_region, pi->dma_region_p);
+ dma_free_attrs(pi->port.dev, MPSC_DMA_ALLOC_SIZE,
+ pi->dma_region, pi->dma_region_p,
+ DMA_ATTR_NON_CONSISTENT);
pi->dma_region = NULL;
pi->dma_region_p = (dma_addr_t)NULL;
}
--- /dev/null
+/*
+ * Actions Semi Owl family serial console
+ *
+ * Copyright 2013 Actions Semi Inc.
+ * Author: Actions Semi, Inc.
+ *
+ * Copyright (c) 2016-2017 Andreas Färber
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/console.h>
+#include <linux/delay.h>
+#include <linux/io.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+#include <linux/serial.h>
+#include <linux/serial_core.h>
+
+#define OWL_UART_CTL 0x000
+#define OWL_UART_TXDAT 0x008
+#define OWL_UART_STAT 0x00c
+
+#define OWL_UART_CTL_TRFS_TX BIT(14)
+#define OWL_UART_CTL_EN BIT(15)
+#define OWL_UART_CTL_RXIE BIT(18)
+#define OWL_UART_CTL_TXIE BIT(19)
+
+#define OWL_UART_STAT_RIP BIT(0)
+#define OWL_UART_STAT_TIP BIT(1)
+#define OWL_UART_STAT_TFFU BIT(6)
+#define OWL_UART_STAT_TRFL_MASK (0x1f << 11)
+#define OWL_UART_STAT_UTBB BIT(17)
+
+static inline void owl_uart_write(struct uart_port *port, u32 val, unsigned int off)
+{
+ writel(val, port->membase + off);
+}
+
+static inline u32 owl_uart_read(struct uart_port *port, unsigned int off)
+{
+ return readl(port->membase + off);
+}
+
+#ifdef CONFIG_SERIAL_OWL_CONSOLE
+
+static void owl_console_putchar(struct uart_port *port, int ch)
+{
+ if (!port->membase)
+ return;
+
+ while (owl_uart_read(port, OWL_UART_STAT) & OWL_UART_STAT_TFFU)
+ cpu_relax();
+
+ owl_uart_write(port, ch, OWL_UART_TXDAT);
+}
+
+static void owl_uart_port_write(struct uart_port *port, const char *s,
+ u_int count)
+{
+ u32 old_ctl, val;
+ unsigned long flags;
+ int locked;
+
+ local_irq_save(flags);
+
+ if (port->sysrq)
+ locked = 0;
+ else if (oops_in_progress)
+ locked = spin_trylock(&port->lock);
+ else {
+ spin_lock(&port->lock);
+ locked = 1;
+ }
+
+ old_ctl = owl_uart_read(port, OWL_UART_CTL);
+ val = old_ctl | OWL_UART_CTL_TRFS_TX;
+ /* disable IRQ */
+ val &= ~(OWL_UART_CTL_RXIE | OWL_UART_CTL_TXIE);
+ owl_uart_write(port, val, OWL_UART_CTL);
+
+ uart_console_write(port, s, count, owl_console_putchar);
+
+ /* wait until all contents have been sent out */
+ while (owl_uart_read(port, OWL_UART_STAT) & OWL_UART_STAT_TRFL_MASK)
+ cpu_relax();
+
+ /* clear IRQ pending */
+ val = owl_uart_read(port, OWL_UART_STAT);
+ val |= OWL_UART_STAT_TIP | OWL_UART_STAT_RIP;
+ owl_uart_write(port, val, OWL_UART_STAT);
+
+ owl_uart_write(port, old_ctl, OWL_UART_CTL);
+
+ if (locked)
+ spin_unlock(&port->lock);
+
+ local_irq_restore(flags);
+}
+
+static void owl_uart_early_console_write(struct console *co,
+ const char *s,
+ u_int count)
+{
+ struct earlycon_device *dev = co->data;
+
+ owl_uart_port_write(&dev->port, s, count);
+}
+
+static int __init
+owl_uart_early_console_setup(struct earlycon_device *device, const char *opt)
+{
+ if (!device->port.membase)
+ return -ENODEV;
+
+ device->con->write = owl_uart_early_console_write;
+
+ return 0;
+}
+OF_EARLYCON_DECLARE(owl, "actions,owl-uart",
+ owl_uart_early_console_setup);
+
+#endif /* CONFIG_SERIAL_OWL_CONSOLE */
sg_dma_len(sg) = priv->trigger_level;
sg_set_page(&priv->sg_rx, virt_to_page(priv->rx_buf_virt),
- sg_dma_len(sg), (unsigned long)priv->rx_buf_virt &
- ~PAGE_MASK);
+ sg_dma_len(sg), offset_in_page(priv->rx_buf_virt));
sg_dma_address(sg) = priv->rx_buf_dma;
clk = devm_clk_get(&pdev->dev, NULL);
if (IS_ERR(clk)) {
- if (PTR_ERR(clk) == -EPROBE_DEFER) {
- ret = -EPROBE_DEFER;
+ ret = PTR_ERR(clk);
+ if (ret == -EPROBE_DEFER)
goto err_out;
- }
+ uartclk = 0;
+ } else {
+ clk_prepare_enable(clk);
+ uartclk = clk_get_rate(clk);
+ }
+
+ if (!uartclk) {
dev_notice(&pdev->dev, "Using default clock frequency\n");
uartclk = s->chip->freq_std;
- } else
- uartclk = clk_get_rate(clk);
+ }
/* Check input frequency */
if ((uartclk < s->chip->freq_min) || (uartclk > s->chip->freq_max)) {
old_custom_divisor != uport->custom_divisor) {
/*
* If they're setting up a custom divisor or speed,
- * instead of clearing it, then bitch about it. No
- * need to rate-limit; it's CAP_SYS_ADMIN only.
+ * instead of clearing it, then bitch about it.
*/
if (uport->flags & UPF_SPD_MASK) {
- dev_notice(uport->dev,
+ dev_notice_ratelimited(uport->dev,
"%s sets custom speed on %s. This is deprecated.\n",
current->comm,
tty_name(port->tty));
chan = dma_request_slave_channel(port->dev,
dir == DMA_MEM_TO_DEV ? "tx" : "rx");
if (!chan) {
- dev_warn(port->dev,
- "dma_request_slave_channel_compat failed\n");
+ dev_warn(port->dev, "dma_request_slave_channel failed\n");
return NULL;
}
if (s->chan_rx)
sci_rx_dma_release(s, false);
}
-#else
+
+static void sci_flush_buffer(struct uart_port *port)
+{
+ /*
+ * In uart_flush_buffer(), the xmit circular buffer has just been
+ * cleared, so we have to reset tx_dma_len accordingly.
+ */
+ to_sci_port(port)->tx_dma_len = 0;
+}
+#else /* !CONFIG_SERIAL_SH_SCI_DMA */
static inline void sci_request_dma(struct uart_port *port)
{
}
static inline void sci_free_dma(struct uart_port *port)
{
}
-#endif
+
+#define sci_flush_buffer NULL
+#endif /* !CONFIG_SERIAL_SH_SCI_DMA */
static irqreturn_t sci_rx_interrupt(int irq, void *ptr)
{
.break_ctl = sci_break_ctl,
.startup = sci_startup,
.shutdown = sci_shutdown,
+ .flush_buffer = sci_flush_buffer,
.set_termios = sci_set_termios,
.pm = sci_pm,
.type = sci_type,
static const char banner[] __initconst = "SuperH (H)SCI(F) driver initialized";
+static DEFINE_MUTEX(sci_uart_registration_lock);
static struct uart_driver sci_uart_driver = {
.owner = THIS_MODULE,
.driver_name = "sci",
return -EINVAL;
}
+ mutex_lock(&sci_uart_registration_lock);
+ if (!sci_uart_driver.state) {
+ ret = uart_register_driver(&sci_uart_driver);
+ if (ret) {
+ mutex_unlock(&sci_uart_registration_lock);
+ return ret;
+ }
+ }
+ mutex_unlock(&sci_uart_registration_lock);
+
ret = sci_init_single(dev, sciport, index, p, false);
if (ret)
return ret;
static int __init sci_init(void)
{
- int ret;
-
pr_info("%s\n", banner);
- ret = uart_register_driver(&sci_uart_driver);
- if (likely(ret == 0)) {
- ret = platform_driver_register(&sci_driver);
- if (unlikely(ret))
- uart_unregister_driver(&sci_uart_driver);
- }
-
- return ret;
+ return platform_driver_register(&sci_driver);
}
static void __exit sci_exit(void)
{
platform_driver_unregister(&sci_driver);
- uart_unregister_driver(&sci_uart_driver);
+
+ if (sci_uart_driver.state)
+ uart_unregister_driver(&sci_uart_driver);
}
#ifdef CONFIG_SERIAL_SH_SCI_CONSOLE
return HRTIMER_RESTART;
}
-static struct of_device_id sirfsoc_uart_ids[] = {
+static const struct of_device_id sirfsoc_uart_ids[] = {
{ .compatible = "sirf,prima2-uart", .data = &sirfsoc_uart,},
{ .compatible = "sirf,atlas7-uart", .data = &sirfsoc_uart},
{ .compatible = "sirf,prima2-usp-uart", .data = &sirfsoc_usp},
* @pclk: APB clock
* @baud: Current baud rate
* @clk_rate_change_nb: Notifier block for clock changes
+ * @quirks: Flags for RXBS support.
*/
struct cdns_uart {
struct uart_port *port;
if (rc) {
dev_err(&pdev->dev,
"uart_add_one_port() failed; err=%i\n", rc);
- goto err_out_notif_unreg;
+ goto err_out_pm_disable;
}
return 0;
+err_out_pm_disable:
+ pm_runtime_disable(&pdev->dev);
+ pm_runtime_set_suspended(&pdev->dev);
+ pm_runtime_dont_use_autosuspend(&pdev->dev);
err_out_notif_unreg:
#ifdef CONFIG_COMMON_CLK
clk_notifier_unregister(cdns_uart_data->uartclk,
&cdns_uart_data->clk_rate_change_nb);
#endif
err_out_clk_disable:
- pm_runtime_disable(&pdev->dev);
- pm_runtime_set_suspended(&pdev->dev);
- pm_runtime_dont_use_autosuspend(&pdev->dev);
clk_disable_unprepare(cdns_uart_data->uartclk);
err_out_clk_dis_pclk:
clk_disable_unprepare(cdns_uart_data->pclk);
struct cond_wait {
struct cond_wait *next;
wait_queue_head_t q;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
unsigned int data;
};
static void init_cond_wait(struct cond_wait *w, unsigned int data);
return NULL;
}
+/**
+ * tty_dev_name_to_number - return dev_t for device name
+ * @name: user space name of device under /dev
+ * @number: pointer to dev_t that this function will populate
+ *
+ * This function converts device names like ttyS0 or ttyUSB1 into dev_t
+ * like (4, 64) or (188, 1). If no corresponding driver is registered then
+ * the function returns -ENODEV.
+ *
+ * Locking: this acquires tty_mutex to protect the tty_drivers list from
+ * being modified while we are traversing it, and makes sure to
+ * release it before exiting.
+ */
+int tty_dev_name_to_number(const char *name, dev_t *number)
+{
+ struct tty_driver *p;
+ int ret;
+ int index, prefix_length = 0;
+ const char *str;
+
+ for (str = name; *str && !isdigit(*str); str++)
+ ;
+
+ if (!*str)
+ return -EINVAL;
+
+ ret = kstrtoint(str, 10, &index);
+ if (ret)
+ return ret;
+
+ prefix_length = str - name;
+ mutex_lock(&tty_mutex);
+
+ list_for_each_entry(p, &tty_drivers, tty_drivers)
+ if (prefix_length == strlen(p->name) && strncmp(name,
+ p->name, prefix_length) == 0) {
+ if (index < p->num) {
+ *number = MKDEV(p->major, p->minor_start + index);
+ goto out;
+ }
+ }
+
+ /* if here then driver wasn't found */
+ ret = -ENODEV;
+out:
+ mutex_unlock(&tty_mutex);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(tty_dev_name_to_number);
+
#ifdef CONFIG_CONSOLE_POLL
/**
struct tty_struct *tty;
if (driver->ops->lookup)
- tty = driver->ops->lookup(driver, file, idx);
+ if (!file)
+ tty = ERR_PTR(-EIO);
+ else
+ tty = driver->ops->lookup(driver, file, idx);
else
tty = driver->ttys[idx];
struct tty_driver *console_driver = console_device(index);
if (console_driver) {
driver = tty_driver_kref_get(console_driver);
- if (driver) {
+ if (driver && filp) {
/* Don't let /dev/console block */
filp->f_flags |= O_NONBLOCK;
break;
* - concurrent tty driver removal w/ lookup
* - concurrent tty removal from driver table
*/
-static struct tty_struct *tty_open_by_driver(dev_t device, struct inode *inode,
+struct tty_struct *tty_open_by_driver(dev_t device, struct inode *inode,
struct file *filp)
{
struct tty_struct *tty;
tty_driver_kref_put(driver);
return tty;
}
+EXPORT_SYMBOL_GPL(tty_open_by_driver);
/**
* tty_open - open a tty device
tty_ldisc_debug(tty, "%p: closed\n", ld);
}
+/**
+ * tty_ldisc_failto - helper for ldisc failback
+ * @tty: tty to open the ldisc on
+ * @ld: ldisc we are trying to fail back to
+ *
+ * Helper to try and recover a tty when switching back to the old
+ * ldisc fails and we need something attached.
+ */
+
+static int tty_ldisc_failto(struct tty_struct *tty, int ld)
+{
+ struct tty_ldisc *disc = tty_ldisc_get(tty, ld);
+ int r;
+
+ if (IS_ERR(disc))
+ return PTR_ERR(disc);
+ tty->ldisc = disc;
+ tty_set_termios_ldisc(tty, ld);
+ if ((r = tty_ldisc_open(tty, disc)) < 0)
+ tty_ldisc_put(disc);
+ return r;
+}
+
/**
* tty_ldisc_restore - helper for tty ldisc change
* @tty: tty to recover
static void tty_ldisc_restore(struct tty_struct *tty, struct tty_ldisc *old)
{
- struct tty_ldisc *new_ldisc;
- int r;
-
/* There is an outstanding reference here so this is safe */
old = tty_ldisc_get(tty, old->ops->num);
WARN_ON(IS_ERR(old));
tty_set_termios_ldisc(tty, old->ops->num);
if (tty_ldisc_open(tty, old) < 0) {
tty_ldisc_put(old);
- /* This driver is always present */
- new_ldisc = tty_ldisc_get(tty, N_TTY);
- if (IS_ERR(new_ldisc))
- panic("n_tty: get");
- tty->ldisc = new_ldisc;
- tty_set_termios_ldisc(tty, N_TTY);
- r = tty_ldisc_open(tty, new_ldisc);
- if (r < 0)
- panic("Couldn't open N_TTY ldisc for "
- "%s --- error %d.",
- tty_name(tty), r);
+ /* The traditional behaviour is to fall back to N_TTY, we
+ want to avoid falling back to N_NULL unless we have no
+ choice to avoid the risk of breaking anything */
+ if (tty_ldisc_failto(tty, N_TTY) < 0 &&
+ tty_ldisc_failto(tty, N_NULL) < 0)
+ panic("Couldn't open N_NULL ldisc for %s.",
+ tty_name(tty));
}
}
tty_unlock(tty);
return retval;
}
+EXPORT_SYMBOL_GPL(tty_set_ldisc);
/**
* tty_ldisc_kill - teardown ldisc
tty_ldisc_debug(tty, "released\n");
}
+EXPORT_SYMBOL_GPL(tty_ldisc_release);
/**
* tty_ldisc_init - ldisc setup for new tty
{
int i;
unsigned short inbuf[E_TABSZ];
+ unsigned char ubuf[E_TABSZ];
- if (!access_ok(VERIFY_READ, arg, E_TABSZ))
+ if (copy_from_user(ubuf, arg, E_TABSZ))
return -EFAULT;
- for (i = 0; i < E_TABSZ ; i++) {
- unsigned char uc;
- __get_user(uc, arg+i);
- inbuf[i] = UNI_DIRECT_BASE | uc;
- }
+ for (i = 0; i < E_TABSZ ; i++)
+ inbuf[i] = UNI_DIRECT_BASE | ubuf[i];
console_lock();
memcpy(translations[USER_MAP], inbuf, sizeof(inbuf));
unsigned short *p = translations[USER_MAP];
unsigned char outbuf[E_TABSZ];
- if (!access_ok(VERIFY_WRITE, arg, E_TABSZ))
- return -EFAULT;
-
console_lock();
for (i = 0; i < E_TABSZ ; i++)
{
}
console_unlock();
- for (i = 0; i < E_TABSZ ; i++)
- __put_user(outbuf[i], arg+i);
- return 0;
+ return copy_to_user(arg, outbuf, sizeof(outbuf)) ? -EFAULT : 0;
}
int con_set_trans_new(ushort __user * arg)
{
- int i;
unsigned short inbuf[E_TABSZ];
- if (!access_ok(VERIFY_READ, arg, E_TABSZ*sizeof(unsigned short)))
+ if (copy_from_user(inbuf, arg, sizeof(inbuf)))
return -EFAULT;
- for (i = 0; i < E_TABSZ ; i++)
- __get_user(inbuf[i], arg+i);
-
console_lock();
memcpy(translations[USER_MAP], inbuf, sizeof(inbuf));
update_user_maps();
int con_get_trans_new(ushort __user * arg)
{
- int i;
unsigned short outbuf[E_TABSZ];
- if (!access_ok(VERIFY_WRITE, arg, E_TABSZ*sizeof(unsigned short)))
- return -EFAULT;
-
console_lock();
memcpy(outbuf, translations[USER_MAP], sizeof(outbuf));
console_unlock();
- for (i = 0; i < E_TABSZ ; i++)
- __put_user(outbuf[i], arg+i);
- return 0;
+ return copy_to_user(arg, outbuf, sizeof(outbuf)) ? -EFAULT : 0;
}
/*
if (!ct)
return 0;
- unilist = kmalloc_array(ct, sizeof(struct unipair), GFP_KERNEL);
- if (!unilist)
- return -ENOMEM;
-
- for (i = ct, plist = unilist; i; i--, plist++, list++) {
- __get_user(plist->unicode, &list->unicode);
- __get_user(plist->fontpos, &list->fontpos);
- }
+ unilist = memdup_user(list, ct * sizeof(struct unipair));
+ if (IS_ERR(unilist))
+ return PTR_ERR(unilist);
console_lock();
*/
int con_get_unimap(struct vc_data *vc, ushort ct, ushort __user *uct, struct unipair __user *list)
{
- int i, j, k;
+ int i, j, k, ret = 0;
ushort ect;
u16 **p1, *p2;
struct uni_pagedir *p;
- struct unipair *unilist, *plist;
+ struct unipair *unilist;
unilist = kmalloc_array(ct, sizeof(struct unipair), GFP_KERNEL);
if (!unilist)
}
}
console_unlock();
- for (i = min(ect, ct), plist = unilist; i; i--, list++, plist++) {
- __put_user(plist->unicode, &list->unicode);
- __put_user(plist->fontpos, &list->fontpos);
- }
- __put_user(ect, uct);
+ if (copy_to_user(list, unilist, min(ect, ct) * sizeof(struct unipair)))
+ ret = -EFAULT;
+ put_user(ect, uct);
kfree(unilist);
- return ((ect <= ct) ? 0 : -ENOMEM);
+ return ret ? ret : (ect <= ct) ? 0 : -ENOMEM;
}
/*
#if defined(CONFIG_X86) || defined(CONFIG_IA64) || defined(CONFIG_ALPHA) ||\
defined(CONFIG_MIPS) || defined(CONFIG_PPC) || defined(CONFIG_SPARC) ||\
defined(CONFIG_PARISC) || defined(CONFIG_SUPERH) ||\
- (defined(CONFIG_ARM) && defined(CONFIG_KEYBOARD_ATKBD) && !defined(CONFIG_ARCH_RPC)) ||\
- defined(CONFIG_AVR32)
+ (defined(CONFIG_ARM) && defined(CONFIG_KEYBOARD_ATKBD) && !defined(CONFIG_ARCH_RPC))
#define HW_RAW(dev) (test_bit(EV_MSC, dev->evbit) && test_bit(MSC_RAW, dev->mscbit) &&\
((dev)->id.bustype == BUS_I8042) && ((dev)->id.vendor == 0x0001) && ((dev)->id.product == 0x0001))
else if (_underline)
a = (a & 0xf0) | vc->vc_ulcolor;
else if (_intensity == 0)
- a = (a & 0xf0) | vc->vc_ulcolor;
+ a = (a & 0xf0) | vc->vc_halfcolor;
if (_reverse)
a = ((a) & 0x88) | ((((a) >> 4) | ((a) << 4)) & 0x77);
if (_blink)
* related to the kernel should not use this.
*/
data = vt_get_shift_state();
- ret = __put_user(data, p);
+ ret = put_user(data, p);
break;
case TIOCL_GETMOUSEREPORTING:
console_lock(); /* May be overkill */
data = mouse_reporting();
console_unlock();
- ret = __put_user(data, p);
+ ret = put_user(data, p);
break;
case TIOCL_SETVESABLANK:
console_lock();
break;
case TIOCL_GETKMSGREDIRECT:
data = vt_get_kmsg_redirect();
- ret = __put_user(data, p);
+ ret = put_user(data, p);
break;
case TIOCL_SETKMSGREDIRECT:
if (!capable(CAP_SYS_ADMIN)) {
if (copy_from_user(&tmp, user_ud, sizeof tmp))
return -EFAULT;
- if (tmp.entries)
- if (!access_ok(VERIFY_WRITE, tmp.entries,
- tmp.entry_ct*sizeof(struct unipair)))
- return -EFAULT;
switch (cmd) {
case PIO_UNIMAP:
if (!perm)
if (copy_from_user(&tmp, user_ud, sizeof tmp))
return -EFAULT;
tmp_entries = compat_ptr(tmp.entries);
- if (tmp_entries)
- if (!access_ok(VERIFY_WRITE, tmp_entries,
- tmp.entry_ct*sizeof(struct unipair)))
- return -EFAULT;
switch (cmd) {
case PIO_UNIMAP:
if (!perm)
return err;
}
- if (!pdev->irq) {
- dev_warn(&pdev->dev, "No IRQ assigned to device: "
- "no support for interrupts?\n");
- pci_disable_device(pdev);
- return -ENODEV;
- }
-
- if (!pci_intx_mask_supported(pdev)) {
+ if (pdev->irq && !pci_intx_mask_supported(pdev)) {
err = -ENODEV;
goto err_verify;
}
gdev->info.name = "uio_pci_generic";
gdev->info.version = DRIVER_VERSION;
- gdev->info.irq = pdev->irq;
- gdev->info.irq_flags = IRQF_SHARED;
- gdev->info.handler = irqhandler;
gdev->pdev = pdev;
+ if (pdev->irq) {
+ gdev->info.irq = pdev->irq;
+ gdev->info.irq_flags = IRQF_SHARED;
+ gdev->info.handler = irqhandler;
+ } else {
+ dev_warn(&pdev->dev, "No IRQ assigned to device: "
+ "no support for interrupts?\n");
+ }
err = uio_register_device(&pdev->dev, &gdev->info);
if (err)
{
ci_hdrc_gadget_destroy(ci);
ci_hdrc_host_destroy(ci);
- if (ci->is_otg)
+ if (ci->is_otg && ci->roles[CI_ROLE_GADGET])
ci_hdrc_otg_destroy(ci);
}
/* initialize role(s) before the interrupt is requested */
if (dr_mode == USB_DR_MODE_OTG || dr_mode == USB_DR_MODE_HOST) {
ret = ci_hdrc_host_init(ci);
- if (ret)
- dev_info(dev, "doesn't support host\n");
+ if (ret) {
+ if (ret == -ENXIO)
+ dev_info(dev, "doesn't support host\n");
+ else
+ goto deinit_phy;
+ }
}
if (dr_mode == USB_DR_MODE_OTG || dr_mode == USB_DR_MODE_PERIPHERAL) {
ret = ci_hdrc_gadget_init(ci);
- if (ret)
- dev_info(dev, "doesn't support gadget\n");
+ if (ret) {
+ if (ret == -ENXIO)
+ dev_info(dev, "doesn't support gadget\n");
+ else
+ goto deinit_host;
+ }
}
if (!ci->roles[CI_ROLE_HOST] && !ci->roles[CI_ROLE_GADGET]) {
dev_err(dev, "no supported roles\n");
ret = -ENODEV;
- goto deinit_phy;
+ goto deinit_gadget;
}
if (ci->is_otg && ci->roles[CI_ROLE_GADGET]) {
ret = ci_hdrc_otg_init(ci);
if (ret) {
dev_err(dev, "init otg fails, ret = %d\n", ret);
- goto stop;
+ goto deinit_gadget;
}
}
remove_debug:
dbg_remove_files(ci);
stop:
- ci_role_destroy(ci);
+ if (ci->is_otg && ci->roles[CI_ROLE_GADGET])
+ ci_hdrc_otg_destroy(ci);
+deinit_gadget:
+ ci_hdrc_gadget_destroy(ci);
+deinit_host:
+ ci_hdrc_host_destroy(ci);
deinit_phy:
ci_usb_phy_exit(ci);
ulpi_exit:
ktime_set(timer_sec, timer_nsec));
ci->enabled_otg_timer_bits |= (1 << t);
if ((ci->next_otg_timer == NUM_OTG_FSM_TIMERS) ||
- (ci->hr_timeouts[ci->next_otg_timer] >
+ ktime_after(ci->hr_timeouts[ci->next_otg_timer],
ci->hr_timeouts[t])) {
ci->next_otg_timer = t;
hrtimer_start_range_ns(&ci->otg_fsm_hrtimer,
for_each_set_bit(cur_timer, &enabled_timer_bits,
NUM_OTG_FSM_TIMERS) {
if ((next_timer == NUM_OTG_FSM_TIMERS) ||
- (ci->hr_timeouts[next_timer] <
+ ktime_before(ci->hr_timeouts[next_timer],
ci->hr_timeouts[cur_timer]))
next_timer = cur_timer;
}
now = ktime_get();
for_each_set_bit(cur_timer, &enabled_timer_bits, NUM_OTG_FSM_TIMERS) {
- if (now >= ci->hr_timeouts[cur_timer]) {
+ if (ktime_compare(now, ci->hr_timeouts[cur_timer]) >= 0) {
ci->enabled_otg_timer_bits &= ~(1 << cur_timer);
if (otg_timer_handlers[cur_timer])
ret = otg_timer_handlers[cur_timer](ci);
} else {
if ((next_timer == NUM_OTG_FSM_TIMERS) ||
- (ci->hr_timeouts[cur_timer] <
+ ktime_before(ci->hr_timeouts[cur_timer],
ci->hr_timeouts[next_timer]))
next_timer = cur_timer;
}
if (we < 0)
return usb_translate_errors(we);
- buf = kmalloc(count, GFP_KERNEL);
- if (!buf) {
- rv = -ENOMEM;
- goto outnl;
- }
-
- r = copy_from_user(buf, buffer, count);
- if (r > 0) {
- rv = -EFAULT;
- goto out_free_mem;
- }
+ buf = memdup_user(buffer, count);
+ if (IS_ERR(buf))
+ return PTR_ERR(buf);
/* concurrent writes and disconnect */
r = mutex_lock_interruptible(&desc->wlock);
usb_autopm_put_interface(desc->intf);
mutex_unlock(&desc->wlock);
-outnl:
- return rv < 0 ? rv : count;
+ return count;
out_free_mem_pm:
usb_autopm_put_interface(desc->intf);
case USBDEVFS_DROP_PRIVILEGES:
ret = proc_drop_privileges(ps, p);
break;
+ case USBDEVFS_GET_SPEED:
+ ret = ps->dev->speed;
+ break;
}
done:
static int hcd_pci_resume_noirq(struct device *dev)
{
- struct pci_dev *pci_dev = to_pci_dev(dev);
-
- powermac_set_asic(pci_dev, 1);
-
- /* Go back to D0 and disable remote wakeup */
- pci_back_from_sleep(pci_dev);
+ powermac_set_asic(to_pci_dev(dev), 1);
return 0;
}
#include <linux/module.h>
#include <linux/version.h>
#include <linux/kernel.h>
+#include <linux/sched/task_stack.h>
#include <linux/slab.h>
#include <linux/completion.h>
#include <linux/utsname.h>
static int register_root_hub(struct usb_hcd *hcd)
{
struct device *parent_dev = hcd->self.controller;
- struct device *sysdev = hcd->self.sysdev;
struct usb_device *usb_dev = hcd->self.root_hub;
const int devnum = 1;
int retval;
/* Did the HC die before the root hub was registered? */
if (HCD_DEAD(hcd))
usb_hc_died (hcd); /* This time clean up */
- usb_dev->dev.of_node = sysdev->of_node;
}
mutex_unlock(&usb_bus_idr_lock);
if (hcd->self.uses_pio_for_control)
return ret;
if (IS_ENABLED(CONFIG_HAS_DMA) && hcd->self.uses_dma) {
+ if (is_vmalloc_addr(urb->setup_packet)) {
+ WARN_ONCE(1, "setup packet is not dma capable\n");
+ return -EAGAIN;
+ } else if (object_is_on_stack(urb->setup_packet)) {
+ WARN_ONCE(1, "setup packet is on stack\n");
+ return -EAGAIN;
+ }
+
urb->setup_dma = dma_map_single(
hcd->self.sysdev,
urb->setup_packet,
} else if (is_vmalloc_addr(urb->transfer_buffer)) {
WARN_ONCE(1, "transfer buffer not dma capable\n");
ret = -EAGAIN;
+ } else if (object_is_on_stack(urb->transfer_buffer)) {
+ WARN_ONCE(1, "transfer buffer is on stack\n");
+ ret = -EAGAIN;
} else {
urb->transfer_dma = dma_map_single(
hcd->self.sysdev,
kref_put(&hub->kref, hub_release);
}
+static bool hub_descriptor_is_sane(struct usb_host_interface *desc)
+{
+ /* Some hubs have a subclass of 1, which AFAICT according to the */
+ /* specs is not defined, but it works */
+ if (desc->desc.bInterfaceSubClass != 0 &&
+ desc->desc.bInterfaceSubClass != 1)
+ return false;
+
+ /* Multiple endpoints? What kind of mutant ninja-hub is this? */
+ if (desc->desc.bNumEndpoints != 1)
+ return false;
+
+ /* If the first endpoint is not interrupt IN, we'd better punt! */
+ if (!usb_endpoint_is_int_in(&desc->endpoint[0].desc))
+ return false;
+
+ return true;
+}
+
static int hub_probe(struct usb_interface *intf, const struct usb_device_id *id)
{
struct usb_host_interface *desc;
- struct usb_endpoint_descriptor *endpoint;
struct usb_device *hdev;
struct usb_hub *hub;
}
#endif
- /* Some hubs have a subclass of 1, which AFAICT according to the */
- /* specs is not defined, but it works */
- if ((desc->desc.bInterfaceSubClass != 0) &&
- (desc->desc.bInterfaceSubClass != 1)) {
-descriptor_error:
+ if (!hub_descriptor_is_sane(desc)) {
dev_err(&intf->dev, "bad descriptor, ignoring hub\n");
return -EIO;
}
- /* Multiple endpoints? What kind of mutant ninja-hub is this? */
- if (desc->desc.bNumEndpoints != 1)
- goto descriptor_error;
-
- endpoint = &desc->endpoint[0].desc;
-
- /* If it's not an interrupt in endpoint, we'd better punt! */
- if (!usb_endpoint_is_int_in(endpoint))
- goto descriptor_error;
-
/* We found a hub */
dev_info(&intf->dev, "USB hub found\n");
if (id->driver_info & HUB_QUIRK_CHECK_PORT_AUTOSUSPEND)
hub->quirk_check_port_auto_suspend = 1;
- if (hub_configure(hub, endpoint) >= 0)
+ if (hub_configure(hub, &desc->endpoint[0].desc) >= 0)
return 0;
hub_disconnect(intf);
if (PMSG_IS_AUTO(msg))
goto err_ltm;
}
- if (usb_unlocked_disable_lpm(udev)) {
- dev_err(&udev->dev, "Failed to disable LPM before suspend\n.");
- status = -ENOMEM;
- if (PMSG_IS_AUTO(msg))
- goto err_lpm3;
- }
/* see 7.1.7.6 */
if (hub_is_superspeed(hub->hdev))
if (status) {
dev_dbg(&port_dev->dev, "can't suspend, status %d\n", status);
- /* Try to enable USB3 LPM and LTM again */
- usb_unlocked_enable_lpm(udev);
- err_lpm3:
+ /* Try to enable USB3 LTM again */
usb_enable_ltm(udev);
err_ltm:
/* Try to enable USB2 hardware LPM again */
if (udev->usb2_hw_lpm_capable == 1)
usb_set_usb2_hardware_lpm(udev, 1);
- /* Try to enable USB3 LTM and LPM */
+ /* Try to enable USB3 LTM */
usb_enable_ltm(udev);
- usb_unlocked_enable_lpm(udev);
}
usb_unlock_port(port_dev);
#include <linux/device.h>
#include <linux/leds.h>
#include <linux/module.h>
+#include <linux/of.h>
#include <linux/slab.h>
#include <linux/usb.h>
+#include <linux/usb/of.h>
struct usbport_trig_data {
struct led_classdev *led_cdev;
* Adding & removing ports
***************************************/
+/**
+ * usbport_trig_port_observed - Check if port should be observed
+ */
+static bool usbport_trig_port_observed(struct usbport_trig_data *usbport_data,
+ struct usb_device *usb_dev, int port1)
+{
+ struct device *dev = usbport_data->led_cdev->dev;
+ struct device_node *led_np = dev->of_node;
+ struct of_phandle_args args;
+ struct device_node *port_np;
+ int count, i;
+
+ if (!led_np)
+ return false;
+
+ /* Get node of port being added */
+ port_np = usb_of_get_child_node(usb_dev->dev.of_node, port1);
+ if (!port_np)
+ return false;
+
+ /* Amount of trigger sources for this LED */
+ count = of_count_phandle_with_args(led_np, "trigger-sources",
+ "#trigger-source-cells");
+ if (count < 0) {
+ dev_warn(dev, "Failed to get trigger sources for %s\n",
+ led_np->full_name);
+ return false;
+ }
+
+ /* Check list of sources for this specific port */
+ for (i = 0; i < count; i++) {
+ int err;
+
+ err = of_parse_phandle_with_args(led_np, "trigger-sources",
+ "#trigger-source-cells", i,
+ &args);
+ if (err) {
+ dev_err(dev, "Failed to get trigger source phandle at index %d: %d\n",
+ i, err);
+ continue;
+ }
+
+ of_node_put(args.np);
+
+ if (args.np == port_np)
+ return true;
+ }
+
+ return false;
+}
+
static int usbport_trig_add_port(struct usbport_trig_data *usbport_data,
struct usb_device *usb_dev,
const char *hub_name, int portnum)
port->data = usbport_data;
port->hub = usb_dev;
port->portnum = portnum;
+ port->observed = usbport_trig_port_observed(usbport_data, usb_dev,
+ portnum);
len = strlen(hub_name) + 8;
port->port_name = kzalloc(len, GFP_KERNEL);
if (err)
goto err_free;
usb_for_each_dev(usbport_data, usbport_trig_add_usb_dev_ports);
+ usbport_trig_update_count(usbport_data);
/* Notifications */
usbport_data->nb.notifier_call = usbport_trig_notify,
*
* Find the node from device tree according to its port number.
*
- * Return: On success, a pointer to the device node, %NULL on failure.
+ * Return: A pointer to the node with incremented refcount if found, or
+ * %NULL otherwise.
*/
struct device_node *usb_of_get_child_node(struct device_node *parent,
int portnum)
/* Blackmagic Design UltraStudio SDI */
{ USB_DEVICE(0x1edb, 0xbd4f), .driver_info = USB_QUIRK_NO_LPM },
+ /* Hauppauge HVR-950q */
+ { USB_DEVICE(0x2040, 0x7200), .driver_info =
+ USB_QUIRK_CONFIG_INTF_STRINGS },
+
/* INTEL VALUE SSD */
{ USB_DEVICE(0x8086, 0xf1a5), .driver_info = USB_QUIRK_RESET_RESUME },
*/
#define USB_ACPI_LOCATION_VALID (1 << 31)
+static struct acpi_device *usb_acpi_find_port(struct acpi_device *parent,
+ int raw)
+{
+ struct acpi_device *adev;
+
+ if (!parent)
+ return NULL;
+
+ list_for_each_entry(adev, &parent->children, node) {
+ if (acpi_device_adr(adev) == raw)
+ return adev;
+ }
+
+ return acpi_find_child_device(parent, raw, false);
+}
+
static struct acpi_device *usb_acpi_find_companion(struct device *dev)
{
struct usb_device *udev;
int raw;
raw = usb_hcd_find_raw_port_number(hcd, port1);
- adev = acpi_find_child_device(ACPI_COMPANION(&udev->dev),
- raw, false);
+
+ adev = usb_acpi_find_port(ACPI_COMPANION(&udev->dev),
+ raw);
+
if (!adev)
return NULL;
} else {
return NULL;
acpi_bus_get_device(parent_handle, &adev);
- adev = acpi_find_child_device(adev, port1, false);
+
+ adev = usb_acpi_find_port(adev, port1);
+
if (!adev)
return NULL;
}
usb_destroy_configuration(udev);
usb_release_bos_descriptor(udev);
+ of_node_put(dev->of_node);
usb_put_hcd(hcd);
kfree(udev->product);
kfree(udev->manufacturer);
dev->route = 0;
dev->dev.parent = bus->controller;
+ device_set_of_node_from_dev(&dev->dev, bus->sysdev);
dev_set_name(&dev->dev, "usb%d", bus->busnum);
root_hub = 1;
} else {
switch (dwc->desired_dr_role) {
case DWC3_GCTL_PRTCAP_HOST:
ret = dwc3_host_init(dwc);
- if (ret)
+ if (ret) {
dev_err(dwc->dev, "failed to initialize host\n");
+ } else {
+ if (dwc->usb2_phy)
+ otg_set_vbus(dwc->usb2_phy->otg, true);
+ if (dwc->usb2_generic_phy)
+ phy_set_mode(dwc->usb2_generic_phy, PHY_MODE_USB_HOST);
+
+ }
break;
case DWC3_GCTL_PRTCAP_DEVICE:
dwc3_event_buffers_setup(dwc);
+
+ if (dwc->usb2_phy)
+ otg_set_vbus(dwc->usb2_phy->otg, false);
+ if (dwc->usb2_generic_phy)
+ phy_set_mode(dwc->usb2_generic_phy, PHY_MODE_USB_DEVICE);
+
ret = dwc3_gadget_init(dwc);
if (ret)
dev_err(dwc->dev, "failed to initialize peripheral\n");
dwc3_writel(dwc->regs, DWC3_GCTL, reg);
}
+static int dwc3_core_get_phy(struct dwc3 *dwc);
+
/**
* dwc3_core_init - Low-level initialization of DWC3 Core
* @dwc: Pointer to our controller context structure
if (ret)
goto err0;
+ ret = dwc3_core_get_phy(dwc);
+ if (ret)
+ goto err0;
+
dwc3_core_setup_global_control(dwc);
dwc3_core_num_eps(dwc);
dwc3_writel(dwc->regs, DWC3_GUCTL2, reg);
}
- /*
- * Enable hardware control of sending remote wakeup in HS when
- * the device is in the L1 state.
- */
- if (dwc->revision >= DWC3_REVISION_290A) {
+ if (dwc->revision >= DWC3_REVISION_250A) {
reg = dwc3_readl(dwc->regs, DWC3_GUCTL1);
- reg |= DWC3_GUCTL1_DEV_L1_EXIT_BY_HW;
+
+ /*
+ * Enable hardware control of sending remote wakeup
+ * in HS when the device is in the L1 state.
+ */
+ if (dwc->revision >= DWC3_REVISION_290A)
+ reg |= DWC3_GUCTL1_DEV_L1_EXIT_BY_HW;
+
+ if (dwc->dis_tx_ipgap_linecheck_quirk)
+ reg |= DWC3_GUCTL1_TX_IPGAP_LINECHECK_DIS;
+
dwc3_writel(dwc->regs, DWC3_GUCTL1, reg);
}
switch (dwc->dr_mode) {
case USB_DR_MODE_PERIPHERAL:
dwc3_set_prtcap(dwc, DWC3_GCTL_PRTCAP_DEVICE);
+
+ if (dwc->usb2_phy)
+ otg_set_vbus(dwc->usb2_phy->otg, false);
+ if (dwc->usb2_generic_phy)
+ phy_set_mode(dwc->usb2_generic_phy, PHY_MODE_USB_DEVICE);
+
ret = dwc3_gadget_init(dwc);
if (ret) {
if (ret != -EPROBE_DEFER)
break;
case USB_DR_MODE_HOST:
dwc3_set_prtcap(dwc, DWC3_GCTL_PRTCAP_HOST);
+
+ if (dwc->usb2_phy)
+ otg_set_vbus(dwc->usb2_phy->otg, true);
+ if (dwc->usb2_generic_phy)
+ phy_set_mode(dwc->usb2_generic_phy, PHY_MODE_USB_HOST);
+
ret = dwc3_host_init(dwc);
if (ret) {
if (ret != -EPROBE_DEFER)
"snps,dis-u2-freeclk-exists-quirk");
dwc->dis_del_phy_power_chg_quirk = device_property_read_bool(dev,
"snps,dis-del-phy-power-chg-quirk");
+ dwc->dis_tx_ipgap_linecheck_quirk = device_property_read_bool(dev,
+ "snps,dis-tx-ipgap-linecheck-quirk");
dwc->tx_de_emphasis_quirk = device_property_read_bool(dev,
"snps,tx_de_emphasis_quirk");
platform_set_drvdata(pdev, dwc);
dwc3_cache_hwparams(dwc);
- ret = dwc3_core_get_phy(dwc);
- if (ret)
- goto err0;
-
spin_lock_init(&dwc->lock);
pm_runtime_set_active(dev);
-/**
+/*
* core.h - DesignWare USB3 DRD Core Header
*
* Copyright (C) 2010-2011 Texas Instruments Incorporated - http://www.ti.com
#define DWC3_GCTL_DSBLCLKGTNG BIT(0)
/* Global User Control 1 Register */
+#define DWC3_GUCTL1_TX_IPGAP_LINECHECK_DIS BIT(28)
#define DWC3_GUCTL1_DEV_L1_EXIT_BY_HW BIT(24)
/* Global USB2 PHY Configuration Register */
* @trb_pool_dma: dma address of @trb_pool
* @trb_enqueue: enqueue 'pointer' into TRB array
* @trb_dequeue: dequeue 'pointer' into TRB array
- * @desc: usb_endpoint_descriptor pointer
* @dwc: pointer to DWC controller
* @saved_state: ep state saved during hibernation
* @flags: endpoint flags (wedged, stalled, ...)
* @bpl: DW0-3
* @bph: DW4-7
* @size: DW8-B
- * @trl: DWC-F
+ * @ctrl: DWC-F
*/
struct dwc3_trb {
u32 bpl;
} __packed;
/**
- * dwc3_hwparams - copy of HWPARAMS registers
- * @hwparams0 - GHWPARAMS0
- * @hwparams1 - GHWPARAMS1
- * @hwparams2 - GHWPARAMS2
- * @hwparams3 - GHWPARAMS3
- * @hwparams4 - GHWPARAMS4
- * @hwparams5 - GHWPARAMS5
- * @hwparams6 - GHWPARAMS6
- * @hwparams7 - GHWPARAMS7
- * @hwparams8 - GHWPARAMS8
+ * struct dwc3_hwparams - copy of HWPARAMS registers
+ * @hwparams0: GHWPARAMS0
+ * @hwparams1: GHWPARAMS1
+ * @hwparams2: GHWPARAMS2
+ * @hwparams3: GHWPARAMS3
+ * @hwparams4: GHWPARAMS4
+ * @hwparams5: GHWPARAMS5
+ * @hwparams6: GHWPARAMS6
+ * @hwparams7: GHWPARAMS7
+ * @hwparams8: GHWPARAMS8
*/
struct dwc3_hwparams {
u32 hwparams0;
* @unaligned: true for OUT endpoints with length not divisible by maxp
* @direction: IN or OUT direction flag
* @mapped: true when request has been dma-mapped
- * @queued: true when request has been queued to HW
+ * @started: request is started
+ * @zero: wants a ZLP
*/
struct dwc3_request {
struct usb_request request;
/**
* struct dwc3 - representation of our controller
- * @drd_work - workqueue used for role swapping
+ * @drd_work: workqueue used for role swapping
* @ep0_trb: trb which is used for the ctrl_req
+ * @bounce: address of bounce buffer
+ * @scratchbuf: address of scratch buffer
* @setup_buf: used while precessing STD USB requests
- * @ep0_trb: dma address of ep0_trb
+ * @ep0_trb_addr: dma address of @ep0_trb
+ * @bounce_addr: dma address of @bounce
* @ep0_usb_req: dummy req used while handling STD USB requests
* @scratch_addr: dma address of scratchbuf
* @ep0_in_setup: one control transfer is completed and enter setup phase
* @lock: for synchronizing
* @dev: pointer to our struct device
+ * @sysdev: pointer to the DMA-capable device
* @xhci: pointer to our xHCI child
- * @event_buffer_list: a list of event buffers
+ * @xhci_resources: struct resources for our @xhci child
+ * @ev_buf: struct dwc3_event_buffer pointer
+ * @eps: endpoint array
* @gadget: device side representation of the peripheral controller
* @gadget_driver: pointer to the gadget driver
* @regs: base address for our registers
* @usb2_generic_phy: pointer to USB2 PHY
* @usb3_generic_phy: pointer to USB3 PHY
* @ulpi: pointer to ulpi interface
- * @dcfg: saved contents of DCFG register
- * @gctl: saved contents of GCTL register
* @isoch_delay: wValue from Set Isochronous Delay request;
* @u2sel: parameter from Set SEL request.
* @u2pel: parameter from Set SEL request.
* @pending_events: true when we have pending IRQs to be handled
* @pullups_connected: true when Run/Stop bit is set
* @setup_packet_pending: true when there's a Setup Packet in FIFO. Workaround
- * @start_config_issued: true when StartConfig command has been issued
* @three_stage_setup: set if we perform a three phase setup
* @usb3_lpm_capable: set if hadrware supports Link Power Management
* @disable_scramble_quirk: set if we enable the disable scramble quirk
* @dis_u2_susphy_quirk: set if we disable usb2 suspend phy
* @dis_enblslpm_quirk: set if we clear enblslpm in GUSB2PHYCFG,
* disabling the suspend signal to the PHY.
+ * @dis_rxdet_inp3_quirk: set if we disable Rx.Detect in P3
* @dis_u2_freeclk_exists_quirk : set if we clear u2_freeclk_exists
* in GUSB2PHYCFG, specify that USB2 PHY doesn't
* provide a free-running PHY clock.
* @dis_del_phy_power_chg_quirk: set if we disable delay phy power
* change quirk.
+ * @dis_tx_ipgap_linecheck_quirk: set if we disable u2mac linestate
+ * check during HS transmit.
* @tx_de_emphasis_quirk: set if we enable Tx de-emphasis quirk
* @tx_de_emphasis: Tx de-emphasis value
* 0 - -6dB de-emphasis
unsigned dis_rxdet_inp3_quirk:1;
unsigned dis_u2_freeclk_exists_quirk:1;
unsigned dis_del_phy_power_chg_quirk:1;
+ unsigned dis_tx_ipgap_linecheck_quirk:1;
unsigned tx_de_emphasis_quirk:1;
unsigned tx_de_emphasis:2;
* @event: the event code
*/
static inline const char *
-dwc3_gadget_event_string(const struct dwc3_event_devt *event)
+dwc3_gadget_event_string(char *str, const struct dwc3_event_devt *event)
{
- static char str[256];
enum dwc3_link_state state = event->event_info & DWC3_LINK_STATE_MASK;
switch (event->type) {
return str;
}
+static inline void dwc3_decode_get_status(__u8 t, __u16 i, __u16 l, char *str)
+{
+ switch (t & USB_RECIP_MASK) {
+ case USB_RECIP_INTERFACE:
+ sprintf(str, "Get Interface Status(Intf = %d, Length = %d)",
+ i, l);
+ break;
+ case USB_RECIP_ENDPOINT:
+ sprintf(str, "Get Endpoint Status(ep%d%s)",
+ i & ~USB_DIR_IN,
+ i & USB_DIR_IN ? "in" : "out");
+ break;
+ }
+}
+
+static inline void dwc3_decode_set_clear_feature(__u8 t, __u8 b, __u16 v,
+ __u16 i, char *str)
+{
+ switch (t & USB_RECIP_MASK) {
+ case USB_RECIP_DEVICE:
+ sprintf(str, "%s Device Feature(%s%s)",
+ b == USB_REQ_CLEAR_FEATURE ? "Clear" : "Set",
+ ({char *s;
+ switch (v) {
+ case USB_DEVICE_SELF_POWERED:
+ s = "Self Powered";
+ break;
+ case USB_DEVICE_REMOTE_WAKEUP:
+ s = "Remote Wakeup";
+ break;
+ case USB_DEVICE_TEST_MODE:
+ s = "Test Mode";
+ break;
+ default:
+ s = "UNKNOWN";
+ } s; }),
+ v == USB_DEVICE_TEST_MODE ?
+ ({ char *s;
+ switch (i) {
+ case TEST_J:
+ s = ": TEST_J";
+ break;
+ case TEST_K:
+ s = ": TEST_K";
+ break;
+ case TEST_SE0_NAK:
+ s = ": TEST_SE0_NAK";
+ break;
+ case TEST_PACKET:
+ s = ": TEST_PACKET";
+ break;
+ case TEST_FORCE_EN:
+ s = ": TEST_FORCE_EN";
+ break;
+ default:
+ s = ": UNKNOWN";
+ } s; }) : "");
+ break;
+ case USB_RECIP_INTERFACE:
+ sprintf(str, "%s Interface Feature(%s)",
+ b == USB_REQ_CLEAR_FEATURE ? "Clear" : "Set",
+ v == USB_INTRF_FUNC_SUSPEND ?
+ "Function Suspend" : "UNKNOWN");
+ break;
+ case USB_RECIP_ENDPOINT:
+ sprintf(str, "%s Endpoint Feature(%s ep%d%s)",
+ b == USB_REQ_CLEAR_FEATURE ? "Clear" : "Set",
+ v == USB_ENDPOINT_HALT ? "Halt" : "UNKNOWN",
+ i & ~USB_DIR_IN,
+ i & USB_DIR_IN ? "in" : "out");
+ break;
+ }
+}
+
+static inline void dwc3_decode_set_address(__u16 v, char *str)
+{
+ sprintf(str, "Set Address(Addr = %02x)", v);
+}
+
+static inline void dwc3_decode_get_set_descriptor(__u8 t, __u8 b, __u16 v,
+ __u16 i, __u16 l, char *str)
+{
+ sprintf(str, "%s %s Descriptor(Index = %d, Length = %d)",
+ b == USB_REQ_GET_DESCRIPTOR ? "Get" : "Set",
+ ({ char *s;
+ switch (v >> 8) {
+ case USB_DT_DEVICE:
+ s = "Device";
+ break;
+ case USB_DT_CONFIG:
+ s = "Configuration";
+ break;
+ case USB_DT_STRING:
+ s = "String";
+ break;
+ case USB_DT_INTERFACE:
+ s = "Interface";
+ break;
+ case USB_DT_ENDPOINT:
+ s = "Endpoint";
+ break;
+ case USB_DT_DEVICE_QUALIFIER:
+ s = "Device Qualifier";
+ break;
+ case USB_DT_OTHER_SPEED_CONFIG:
+ s = "Other Speed Config";
+ break;
+ case USB_DT_INTERFACE_POWER:
+ s = "Interface Power";
+ break;
+ case USB_DT_OTG:
+ s = "OTG";
+ break;
+ case USB_DT_DEBUG:
+ s = "Debug";
+ break;
+ case USB_DT_INTERFACE_ASSOCIATION:
+ s = "Interface Association";
+ break;
+ case USB_DT_BOS:
+ s = "BOS";
+ break;
+ case USB_DT_DEVICE_CAPABILITY:
+ s = "Device Capability";
+ break;
+ case USB_DT_PIPE_USAGE:
+ s = "Pipe Usage";
+ break;
+ case USB_DT_SS_ENDPOINT_COMP:
+ s = "SS Endpoint Companion";
+ break;
+ case USB_DT_SSP_ISOC_ENDPOINT_COMP:
+ s = "SSP Isochronous Endpoint Companion";
+ break;
+ default:
+ s = "UNKNOWN";
+ break;
+ } s; }), v & 0xff, l);
+}
+
+
+static inline void dwc3_decode_get_configuration(__u16 l, char *str)
+{
+ sprintf(str, "Get Configuration(Length = %d)", l);
+}
+
+static inline void dwc3_decode_set_configuration(__u8 v, char *str)
+{
+ sprintf(str, "Set Configuration(Config = %d)", v);
+}
+
+static inline void dwc3_decode_get_intf(__u16 i, __u16 l, char *str)
+{
+ sprintf(str, "Get Interface(Intf = %d, Length = %d)", i, l);
+}
+
+static inline void dwc3_decode_set_intf(__u8 v, __u16 i, char *str)
+{
+ sprintf(str, "Set Interface(Intf = %d, Alt.Setting = %d)", i, v);
+}
+
+static inline void dwc3_decode_synch_frame(__u16 i, __u16 l, char *str)
+{
+ sprintf(str, "Synch Frame(Endpoint = %d, Length = %d)", i, l);
+}
+
+static inline void dwc3_decode_set_sel(__u16 l, char *str)
+{
+ sprintf(str, "Set SEL(Length = %d)", l);
+}
+
+static inline void dwc3_decode_set_isoch_delay(__u8 v, char *str)
+{
+ sprintf(str, "Set Isochronous Delay(Delay = %d ns)", v);
+}
+
+/**
+ * dwc3_decode_ctrl - returns a string represetion of ctrl request
+ */
+static inline const char *dwc3_decode_ctrl(char *str, __u8 bRequestType,
+ __u8 bRequest, __u16 wValue, __u16 wIndex, __u16 wLength)
+{
+ switch (bRequest) {
+ case USB_REQ_GET_STATUS:
+ dwc3_decode_get_status(bRequestType, wIndex, wLength, str);
+ break;
+ case USB_REQ_CLEAR_FEATURE:
+ case USB_REQ_SET_FEATURE:
+ dwc3_decode_set_clear_feature(bRequestType, bRequest, wValue,
+ wIndex, str);
+ break;
+ case USB_REQ_SET_ADDRESS:
+ dwc3_decode_set_address(wValue, str);
+ break;
+ case USB_REQ_GET_DESCRIPTOR:
+ case USB_REQ_SET_DESCRIPTOR:
+ dwc3_decode_get_set_descriptor(bRequestType, bRequest, wValue,
+ wIndex, wLength, str);
+ break;
+ case USB_REQ_GET_CONFIGURATION:
+ dwc3_decode_get_configuration(wLength, str);
+ break;
+ case USB_REQ_SET_CONFIGURATION:
+ dwc3_decode_set_configuration(wValue, str);
+ break;
+ case USB_REQ_GET_INTERFACE:
+ dwc3_decode_get_intf(wIndex, wLength, str);
+ break;
+ case USB_REQ_SET_INTERFACE:
+ dwc3_decode_set_intf(wValue, wIndex, str);
+ break;
+ case USB_REQ_SYNCH_FRAME:
+ dwc3_decode_synch_frame(wIndex, wLength, str);
+ break;
+ case USB_REQ_SET_SEL:
+ dwc3_decode_set_sel(wLength, str);
+ break;
+ case USB_REQ_SET_ISOCH_DELAY:
+ dwc3_decode_set_isoch_delay(wValue, str);
+ break;
+ default:
+ sprintf(str, "%02x %02x %02x %02x %02x %02x %02x %02x",
+ bRequestType, bRequest,
+ cpu_to_le16(wValue) & 0xff,
+ cpu_to_le16(wValue) >> 8,
+ cpu_to_le16(wIndex) & 0xff,
+ cpu_to_le16(wIndex) >> 8,
+ cpu_to_le16(wLength) & 0xff,
+ cpu_to_le16(wLength) >> 8);
+ }
+
+ return str;
+}
+
/**
* dwc3_ep_event_string - returns event name
* @event: then event code
*/
static inline const char *
-dwc3_ep_event_string(const struct dwc3_event_depevt *event, u32 ep0state)
+dwc3_ep_event_string(char *str, const struct dwc3_event_depevt *event,
+ u32 ep0state)
{
u8 epnum = event->endpoint_number;
- static char str[256];
size_t len;
int status;
int ret;
}
}
-static inline const char *dwc3_decode_event(u32 event, u32 ep0state)
+static inline const char *dwc3_decode_event(char *str, u32 event, u32 ep0state)
{
const union dwc3_event evt = (union dwc3_event) event;
if (evt.type.is_devspec)
- return dwc3_gadget_event_string(&evt.devt);
+ return dwc3_gadget_event_string(str, &evt.devt);
else
- return dwc3_ep_event_string(&evt.depevt, ep0state);
+ return dwc3_ep_event_string(str, &evt.depevt, ep0state);
}
static inline const char *dwc3_ep_cmd_status_string(int status)
goto out;
}
- seq_printf(s, "enqueue pointer %d\n", dep->trb_enqueue);
- seq_printf(s, "dequeue pointer %d\n", dep->trb_dequeue);
- seq_printf(s, "\n--------------------------------------------------\n\n");
seq_printf(s, "buffer_addr,size,type,ioc,isp_imi,csp,chn,lst,hwo\n");
for (i = 0; i < DWC3_TRB_NUM; i++) {
struct dwc3_trb *trb = &dep->trb_pool[i];
unsigned int type = DWC3_TRBCTL_TYPE(trb->ctrl);
- seq_printf(s, "%08x%08x,%d,%s,%d,%d,%d,%d,%d,%d\n",
+ seq_printf(s, "%08x%08x,%d,%s,%d,%d,%d,%d,%d,%d %c%c\n",
trb->bph, trb->bpl, trb->size,
dwc3_trb_type_string(type),
!!(trb->ctrl & DWC3_TRB_CTRL_IOC),
!!(trb->ctrl & DWC3_TRB_CTRL_CSP),
!!(trb->ctrl & DWC3_TRB_CTRL_CHN),
!!(trb->ctrl & DWC3_TRB_CTRL_LST),
- !!(trb->ctrl & DWC3_TRB_CTRL_HWO));
+ !!(trb->ctrl & DWC3_TRB_CTRL_HWO),
+ dep->trb_enqueue == i ? 'E' : ' ',
+ dep->trb_dequeue == i ? 'D' : ' ');
}
out:
dev_err(dev, "couldn't get clock\n");
return -EINVAL;
}
- clk_prepare_enable(exynos->clk);
+ ret = clk_prepare_enable(exynos->clk);
+ if (ret)
+ return ret;
exynos->susp_clk = devm_clk_get(dev, "usbdrd30_susp_clk");
if (IS_ERR(exynos->susp_clk))
exynos->susp_clk = NULL;
- clk_prepare_enable(exynos->susp_clk);
+ ret = clk_prepare_enable(exynos->susp_clk);
+ if (ret)
+ goto susp_clk_err;
if (of_device_is_compatible(node, "samsung,exynos7-dwusb3")) {
exynos->axius_clk = devm_clk_get(dev, "usbdrd30_axius_clk");
ret = -ENODEV;
goto axius_clk_err;
}
- clk_prepare_enable(exynos->axius_clk);
+ ret = clk_prepare_enable(exynos->axius_clk);
+ if (ret)
+ goto axius_clk_err;
} else {
exynos->axius_clk = NULL;
}
clk_disable_unprepare(exynos->axius_clk);
axius_clk_err:
clk_disable_unprepare(exynos->susp_clk);
+susp_clk_err:
clk_disable_unprepare(exynos->clk);
return ret;
}
#define PCI_DEVICE_ID_INTEL_CNPLP 0x9dee
#define PCI_DEVICE_ID_INTEL_CNPH 0xa36e
-#define PCI_INTEL_BXT_DSM_UUID "732b85d5-b7a7-4a1b-9ba0-4bbd00ffd511"
+#define PCI_INTEL_BXT_DSM_GUID "732b85d5-b7a7-4a1b-9ba0-4bbd00ffd511"
#define PCI_INTEL_BXT_FUNC_PMU_PWR 4
#define PCI_INTEL_BXT_STATE_D0 0
#define PCI_INTEL_BXT_STATE_D3 3
* struct dwc3_pci - Driver private structure
* @dwc3: child dwc3 platform_device
* @pci: our link to PCI bus
- * @uuid: _DSM UUID
+ * @guid: _DSM GUID
* @has_dsm_for_pm: true for devices which need to run _DSM on runtime PM
*/
struct dwc3_pci {
struct platform_device *dwc3;
struct pci_dev *pci;
- u8 uuid[16];
+ guid_t guid;
unsigned int has_dsm_for_pm:1;
};
if (pdev->device == PCI_DEVICE_ID_INTEL_BXT ||
pdev->device == PCI_DEVICE_ID_INTEL_BXT_M) {
- acpi_str_to_uuid(PCI_INTEL_BXT_DSM_UUID, dwc->uuid);
+ guid_parse(PCI_INTEL_BXT_DSM_GUID, &dwc->guid);
dwc->has_dsm_for_pm = true;
}
}
device_init_wakeup(dev, true);
- device_set_run_wake(dev, true);
pci_set_drvdata(pci, dwc);
pm_runtime_put(dev);
tmp.type = ACPI_TYPE_INTEGER;
tmp.integer.value = param;
- obj = acpi_evaluate_dsm(ACPI_HANDLE(&dwc->pci->dev), dwc->uuid,
+ obj = acpi_evaluate_dsm(ACPI_HANDLE(&dwc->pci->dev), &dwc->guid,
1, PCI_INTEL_BXT_FUNC_PMU_PWR, &argv4);
if (!obj) {
dev_err(&dwc->pci->dev, "failed to evaluate _DSM\n");
{
struct dwc3_pci *dwc = dev_get_drvdata(dev);
- if (device_run_wake(dev))
+ if (device_can_wakeup(dev))
return dwc3_pci_dsm(dwc, PCI_INTEL_BXT_STATE_D3);
return -EBUSY;
dwc3_data->syscfg_reg_off = res->start;
- dev_vdbg(&pdev->dev, "glue-logic addr 0x%p, syscfg-reg offset 0x%x\n",
+ dev_vdbg(&pdev->dev, "glue-logic addr 0x%pK, syscfg-reg offset 0x%x\n",
dwc3_data->glue_base, dwc3_data->syscfg_reg_off);
dwc3_data->rstc_pwrdn =
-/**
+/*
* ep0.c - DesignWare USB3 DRD Controller Endpoint 0 Handling
*
* Copyright (C) 2010-2011 Texas Instruments Incorporated - http://www.ti.com
{
struct dwc3_ep *dep;
u32 recip;
+ u32 value;
u32 reg;
u16 usb_status = 0;
__le16 *response_pkt;
+ /* We don't support PTM_STATUS */
+ value = le16_to_cpu(ctrl->wValue);
+ if (value != 0)
+ return -EINVAL;
+
recip = ctrl->bRequestType & USB_RECIP_MASK;
switch (recip) {
case USB_RECIP_DEVICE:
-/**
+/*
* gadget.c - DesignWare USB3 DRD Controller Gadget Framework Link
*
* Copyright (C) 2010-2011 Texas Instruments Incorporated - http://www.ti.com
#include "io.h"
/**
- * dwc3_gadget_set_test_mode - Enables USB2 Test Modes
+ * dwc3_gadget_set_test_mode - enables usb2 test modes
* @dwc: pointer to our context structure
* @mode: the mode to set (J, K SE0 NAK, Force Enable)
*
- * Caller should take care of locking. This function will
- * return 0 on success or -EINVAL if wrong Test Selector
- * is passed
+ * Caller should take care of locking. This function will return 0 on
+ * success or -EINVAL if wrong Test Selector is passed.
*/
int dwc3_gadget_set_test_mode(struct dwc3 *dwc, int mode)
{
}
/**
- * dwc3_gadget_get_link_state - Gets current state of USB Link
+ * dwc3_gadget_get_link_state - gets current state of usb link
* @dwc: pointer to our context structure
*
* Caller should take care of locking. This function will
}
/**
- * dwc3_gadget_set_link_state - Sets USB Link to a particular State
+ * dwc3_gadget_set_link_state - sets usb link to a particular state
* @dwc: pointer to our context structure
* @state: the state to put link into
*
}
/**
- * dwc3_ep_inc_trb() - Increment a TRB index.
- * @index - Pointer to the TRB index to increment.
+ * dwc3_ep_inc_trb - increment a trb index.
+ * @index: Pointer to the TRB index to increment.
*
* The index should never point to the link TRB. After incrementing,
* if it is point to the link TRB, wrap around to the beginning. The
*index = 0;
}
+/**
+ * dwc3_ep_inc_enq - increment endpoint's enqueue pointer
+ * @dep: The endpoint whose enqueue pointer we're incrementing
+ */
static void dwc3_ep_inc_enq(struct dwc3_ep *dep)
{
dwc3_ep_inc_trb(&dep->trb_enqueue);
}
+/**
+ * dwc3_ep_inc_deq - increment endpoint's dequeue pointer
+ * @dep: The endpoint whose enqueue pointer we're incrementing
+ */
static void dwc3_ep_inc_deq(struct dwc3_ep *dep)
{
dwc3_ep_inc_trb(&dep->trb_dequeue);
}
+/**
+ * dwc3_gadget_giveback - call struct usb_request's ->complete callback
+ * @dep: The endpoint to whom the request belongs to
+ * @req: The request we're giving back
+ * @status: completion code for the request
+ *
+ * Must be called with controller's lock held and interrupts disabled. This
+ * function will unmap @req and call its ->complete() callback to notify upper
+ * layers that it has completed.
+ */
void dwc3_gadget_giveback(struct dwc3_ep *dep, struct dwc3_request *req,
int status)
{
pm_runtime_put(dwc->dev);
}
+/**
+ * dwc3_send_gadget_generic_command - issue a generic command for the controller
+ * @dwc: pointer to the controller context
+ * @cmd: the command to be issued
+ * @param: command parameter
+ *
+ * Caller should take care of locking. Issue @cmd with a given @param to @dwc
+ * and wait for its completion.
+ */
int dwc3_send_gadget_generic_command(struct dwc3 *dwc, unsigned cmd, u32 param)
{
u32 timeout = 500;
static int __dwc3_gadget_wakeup(struct dwc3 *dwc);
+/**
+ * dwc3_send_gadget_ep_cmd - issue an endpoint command
+ * @dep: the endpoint to which the command is going to be issued
+ * @cmd: the command to be issued
+ * @params: parameters to the command
+ *
+ * Caller should handle locking. This function will issue @cmd with given
+ * @params to @dep and wait for its completion.
+ */
int dwc3_send_gadget_ep_cmd(struct dwc3_ep *dep, unsigned cmd,
struct dwc3_gadget_ep_cmd_params *params)
{
static int dwc3_gadget_set_xfer_resource(struct dwc3 *dwc, struct dwc3_ep *dep);
/**
- * dwc3_gadget_start_config - Configure EP resources
+ * dwc3_gadget_start_config - configure ep resources
* @dwc: pointer to our controller context structure
* @dep: endpoint that is being enabled
*
- * The assignment of transfer resources cannot perfectly follow the
- * data book due to the fact that the controller driver does not have
- * all knowledge of the configuration in advance. It is given this
- * information piecemeal by the composite gadget framework after every
- * SET_CONFIGURATION and SET_INTERFACE. Trying to follow the databook
- * programming model in this scenario can cause errors. For two
- * reasons:
+ * Issue a %DWC3_DEPCMD_DEPSTARTCFG command to @dep. After the command's
+ * completion, it will set Transfer Resource for all available endpoints.
*
- * 1) The databook says to do DEPSTARTCFG for every SET_CONFIGURATION
- * and SET_INTERFACE (8.1.5). This is incorrect in the scenario of
- * multiple interfaces.
+ * The assignment of transfer resources cannot perfectly follow the data book
+ * due to the fact that the controller driver does not have all knowledge of the
+ * configuration in advance. It is given this information piecemeal by the
+ * composite gadget framework after every SET_CONFIGURATION and
+ * SET_INTERFACE. Trying to follow the databook programming model in this
+ * scenario can cause errors. For two reasons:
*
- * 2) The databook does not mention doing more DEPXFERCFG for new
+ * 1) The databook says to do %DWC3_DEPCMD_DEPSTARTCFG for every
+ * %USB_REQ_SET_CONFIGURATION and %USB_REQ_SET_INTERFACE (8.1.5). This is
+ * incorrect in the scenario of multiple interfaces.
+ *
+ * 2) The databook does not mention doing more %DWC3_DEPCMD_DEPXFERCFG for new
* endpoint on alt setting (8.1.6).
*
* The following simplified method is used instead:
*
- * All hardware endpoints can be assigned a transfer resource and this
- * setting will stay persistent until either a core reset or
- * hibernation. So whenever we do a DEPSTARTCFG(0) we can go ahead and
- * do DEPXFERCFG for every hardware endpoint as well. We are
+ * All hardware endpoints can be assigned a transfer resource and this setting
+ * will stay persistent until either a core reset or hibernation. So whenever we
+ * do a %DWC3_DEPCMD_DEPSTARTCFG(0) we can go ahead and do
+ * %DWC3_DEPCMD_DEPXFERCFG for every hardware endpoint as well. We are
* guaranteed that there are as many transfer resources as endpoints.
*
- * This function is called for each endpoint when it is being enabled
- * but is triggered only when called for EP0-out, which always happens
- * first, and which should only happen in one of the above conditions.
+ * This function is called for each endpoint when it is being enabled but is
+ * triggered only when called for EP0-out, which always happens first, and which
+ * should only happen in one of the above conditions.
*/
static int dwc3_gadget_start_config(struct dwc3 *dwc, struct dwc3_ep *dep)
{
}
/**
- * __dwc3_gadget_ep_enable - Initializes a HW endpoint
+ * __dwc3_gadget_ep_enable - initializes a hw endpoint
* @dep: endpoint to be initialized
- * @desc: USB Endpoint Descriptor
+ * @modify: if true, modify existing endpoint configuration
+ * @restore: if true, restore endpoint configuration from scratch buffer
*
- * Caller should take care of locking
+ * Caller should take care of locking. Execute all necessary commands to
+ * initialize a HW endpoint so it can be used by a gadget driver.
*/
static int __dwc3_gadget_ep_enable(struct dwc3_ep *dep,
bool modify, bool restore)
}
/**
- * __dwc3_gadget_ep_disable - Disables a HW endpoint
+ * __dwc3_gadget_ep_disable - disables a hw endpoint
* @dep: the endpoint to disable
*
- * This function also removes requests which are currently processed ny the
- * hardware and those which are not yet scheduled.
+ * This function undoes what __dwc3_gadget_ep_enable did and also removes
+ * requests which are currently being processed by the hardware and those which
+ * are not yet scheduled.
+ *
* Caller should take care of locking.
*/
static int __dwc3_gadget_ep_disable(struct dwc3_ep *dep)
}
/**
- * dwc3_ep_prev_trb() - Returns the previous TRB in the ring
+ * dwc3_ep_prev_trb - returns the previous TRB in the ring
* @dep: The endpoint with the TRB ring
* @index: The index of the current TRB in the ring
*
static u32 dwc3_calc_trbs_left(struct dwc3_ep *dep)
{
struct dwc3_trb *tmp;
- struct dwc3 *dwc = dep->dwc;
u8 trbs_left;
/*
*/
if (dep->trb_enqueue == dep->trb_dequeue) {
tmp = dwc3_ep_prev_trb(dep, dep->trb_enqueue);
- if (dev_WARN_ONCE(dwc->dev, tmp->ctrl & DWC3_TRB_CTRL_HWO,
- "%s No TRBS left\n", dep->name))
+ if (tmp->ctrl & DWC3_TRB_CTRL_HWO)
return 0;
return DWC3_TRB_NUM - 1;
}
list_for_each_entry_safe(req, n, &dep->pending_list, list) {
+ struct dwc3 *dwc = dep->dwc;
+ int ret;
+
+ ret = usb_gadget_map_request_by_dev(dwc->sysdev, &req->request,
+ dep->direction);
+ if (ret)
+ return;
+
+ req->sg = req->request.sg;
+ req->num_pending_sgs = req->request.num_mapped_sgs;
+
if (req->num_pending_sgs > 0)
dwc3_prepare_one_trb_sg(dep, req);
else
static int __dwc3_gadget_ep_queue(struct dwc3_ep *dep, struct dwc3_request *req)
{
struct dwc3 *dwc = dep->dwc;
- int ret;
+ int ret = 0;
if (!dep->endpoint.desc) {
dev_err(dwc->dev, "%s: can't queue to disabled endpoint\n",
return -ESHUTDOWN;
}
- if (WARN(req->dep != dep, "request %p belongs to '%s'\n",
- &req->request, req->dep->name)) {
- dev_err(dwc->dev, "%s: request %p belongs to '%s'\n",
- dep->name, &req->request, req->dep->name);
+ if (WARN(req->dep != dep, "request %pK belongs to '%s'\n",
+ &req->request, req->dep->name))
return -EINVAL;
- }
pm_runtime_get(dwc->dev);
trace_dwc3_ep_queue(req);
- ret = usb_gadget_map_request_by_dev(dwc->sysdev, &req->request,
- dep->direction);
- if (ret)
- return ret;
-
- req->sg = req->request.sg;
- req->num_pending_sgs = req->request.num_mapped_sgs;
-
list_add_tail(&req->list, &dep->pending_list);
/*
}
goto out1;
}
- dev_err(dwc->dev, "request %p was not queued to %s\n",
+ dev_err(dwc->dev, "request %pK was not queued to %s\n",
request, ep->name);
ret = -EINVAL;
goto out0;
static irqreturn_t dwc3_thread_interrupt(int irq, void *_dwc);
/**
- * dwc3_gadget_setup_nump - Calculate and initialize NUMP field of DCFG
- * dwc: pointer to our context structure
+ * dwc3_gadget_setup_nump - calculate and initialize NUMP field of %DWC3_DCFG
+ * @dwc: pointer to our context structure
*
* The following looks like complex but it's actually very simple. In order to
* calculate the number of packets we can burst at once on OUT transfers, we're
dwc3_writel(dwc->regs, DWC3_DEV_IMOD(0), 0);
}
- reg = dwc3_readl(dwc->regs, DWC3_DCFG);
- reg &= ~(DWC3_DCFG_SPEED_MASK);
-
- /**
- * WORKAROUND: DWC3 revision < 2.20a have an issue
- * which would cause metastability state on Run/Stop
- * bit if we try to force the IP to USB2-only mode.
- *
- * Because of that, we cannot configure the IP to any
- * speed other than the SuperSpeed
- *
- * Refers to:
- *
- * STAR#9000525659: Clock Domain Crossing on DCTL in
- * USB 2.0 Mode
- */
- if (dwc->revision < DWC3_REVISION_220A) {
- reg |= DWC3_DCFG_SUPERSPEED;
- } else {
- switch (dwc->maximum_speed) {
- case USB_SPEED_LOW:
- reg |= DWC3_DCFG_LOWSPEED;
- break;
- case USB_SPEED_FULL:
- reg |= DWC3_DCFG_FULLSPEED;
- break;
- case USB_SPEED_HIGH:
- reg |= DWC3_DCFG_HIGHSPEED;
- break;
- case USB_SPEED_SUPER_PLUS:
- reg |= DWC3_DCFG_SUPERSPEED_PLUS;
- break;
- default:
- dev_err(dwc->dev, "invalid dwc->maximum_speed (%d)\n",
- dwc->maximum_speed);
- /* fall through */
- case USB_SPEED_SUPER:
- reg |= DWC3_DCFG_SUPERSPEED;
- break;
- }
- }
- dwc3_writel(dwc->regs, DWC3_DCFG, reg);
-
/*
* We are telling dwc3 that we want to use DCFG.NUMP as ACK TP's NUMP
* field instead of letting dwc3 itself calculate that automatically.
return 0;
}
+static void dwc3_gadget_set_speed(struct usb_gadget *g,
+ enum usb_device_speed speed)
+{
+ struct dwc3 *dwc = gadget_to_dwc(g);
+ unsigned long flags;
+ u32 reg;
+
+ spin_lock_irqsave(&dwc->lock, flags);
+ reg = dwc3_readl(dwc->regs, DWC3_DCFG);
+ reg &= ~(DWC3_DCFG_SPEED_MASK);
+
+ /*
+ * WORKAROUND: DWC3 revision < 2.20a have an issue
+ * which would cause metastability state on Run/Stop
+ * bit if we try to force the IP to USB2-only mode.
+ *
+ * Because of that, we cannot configure the IP to any
+ * speed other than the SuperSpeed
+ *
+ * Refers to:
+ *
+ * STAR#9000525659: Clock Domain Crossing on DCTL in
+ * USB 2.0 Mode
+ */
+ if (dwc->revision < DWC3_REVISION_220A) {
+ reg |= DWC3_DCFG_SUPERSPEED;
+ } else {
+ switch (speed) {
+ case USB_SPEED_LOW:
+ reg |= DWC3_DCFG_LOWSPEED;
+ break;
+ case USB_SPEED_FULL:
+ reg |= DWC3_DCFG_FULLSPEED;
+ break;
+ case USB_SPEED_HIGH:
+ reg |= DWC3_DCFG_HIGHSPEED;
+ break;
+ case USB_SPEED_SUPER:
+ reg |= DWC3_DCFG_SUPERSPEED;
+ break;
+ case USB_SPEED_SUPER_PLUS:
+ reg |= DWC3_DCFG_SUPERSPEED_PLUS;
+ break;
+ default:
+ dev_err(dwc->dev, "invalid speed (%d)\n", speed);
+
+ if (dwc->revision & DWC3_REVISION_IS_DWC31)
+ reg |= DWC3_DCFG_SUPERSPEED_PLUS;
+ else
+ reg |= DWC3_DCFG_SUPERSPEED;
+ }
+ }
+ dwc3_writel(dwc->regs, DWC3_DCFG, reg);
+
+ spin_unlock_irqrestore(&dwc->lock, flags);
+}
+
static const struct usb_gadget_ops dwc3_gadget_ops = {
.get_frame = dwc3_gadget_get_frame,
.wakeup = dwc3_gadget_wakeup,
.pullup = dwc3_gadget_pullup,
.udc_start = dwc3_gadget_start,
.udc_stop = dwc3_gadget_stop,
+ .udc_set_speed = dwc3_gadget_set_speed,
};
/* -------------------------------------------------------------------------- */
-static int dwc3_gadget_init_endpoints(struct dwc3 *dwc, u8 num)
+static int dwc3_gadget_init_endpoints(struct dwc3 *dwc, u8 total)
{
struct dwc3_ep *dep;
u8 epnum;
INIT_LIST_HEAD(&dwc->gadget.ep_list);
- for (epnum = 0; epnum < num; epnum++) {
+ for (epnum = 0; epnum < total; epnum++) {
bool direction = epnum & 1;
+ u8 num = epnum >> 1;
dep = kzalloc(sizeof(*dep), GFP_KERNEL);
if (!dep)
dep->regs = dwc->regs + DWC3_DEP_BASE(epnum);
dwc->eps[epnum] = dep;
- snprintf(dep->name, sizeof(dep->name), "ep%d%s", epnum >> 1,
+ snprintf(dep->name, sizeof(dep->name), "ep%u%s", num,
direction ? "in" : "out");
dep->endpoint.name = dep->name;
spin_lock_init(&dep->lock);
- if (epnum == 0 || epnum == 1) {
+ if (num == 0) {
usb_ep_set_maxpacket_limit(&dep->endpoint, 512);
dep->endpoint.maxburst = 1;
dep->endpoint.ops = &dwc3_gadget_ep0_ops;
- if (!epnum)
+ if (!direction)
dwc->gadget.ep0 = &dep->endpoint;
} else if (direction) {
int mdwidth;
+ int kbytes;
int size;
int ret;
- int num;
mdwidth = DWC3_MDWIDTH(dwc->hwparams.hwparams0);
/* MDWIDTH is represented in bits, we need it in bytes */
mdwidth /= 8;
- size = dwc3_readl(dwc->regs, DWC3_GTXFIFOSIZ(epnum >> 1));
+ size = dwc3_readl(dwc->regs, DWC3_GTXFIFOSIZ(num));
size = DWC3_GTXFIFOSIZ_TXFDEF(size);
/* FIFO Depth is in MDWDITH bytes. Multiply */
size *= mdwidth;
- num = size / 1024;
- if (num == 0)
- num = 1;
+ kbytes = size / 1024;
+ if (kbytes == 0)
+ kbytes = 1;
/*
- * FIFO sizes account an extra MDWIDTH * (num + 1) bytes for
+ * FIFO sizes account an extra MDWIDTH * (kbytes + 1) bytes for
* internal overhead. We don't really know how these are used,
* but documentation say it exists.
*/
- size -= mdwidth * (num + 1);
- size /= num;
+ size -= mdwidth * (kbytes + 1);
+ size /= kbytes;
usb_ep_set_maxpacket_limit(&dep->endpoint, size);
return ret;
}
- if (epnum == 0 || epnum == 1) {
+ if (num == 0) {
dep->endpoint.caps.type_control = true;
} else {
dep->endpoint.caps.type_iso = true;
{
unsigned int is_ss = evtinfo & BIT(4);
- /**
+ /*
* WORKAROUND: DWC3 revison 2.20a with hibernation support
* have a known issue which can cause USB CV TD.9.23 to fail
* randomly.
{
trace_dwc3_event(event->raw, dwc);
- /* Endpoint IRQ, handle it and return early */
- if (event->type.is_devspec == 0) {
- /* depevt */
- return dwc3_endpoint_interrupt(dwc, &event->depevt);
- }
-
- switch (event->type.type) {
- case DWC3_EVENT_TYPE_DEV:
+ if (!event->type.is_devspec)
+ dwc3_endpoint_interrupt(dwc, &event->depevt);
+ else if (event->type.type == DWC3_EVENT_TYPE_DEV)
dwc3_gadget_interrupt(dwc, &event->devt);
- break;
- /* REVISIT what to do with Carkit and I2C events ? */
- default:
+ else
dev_err(dwc->dev, "UNKNOWN IRQ type %d\n", event->raw);
- }
}
static irqreturn_t dwc3_process_event_buf(struct dwc3_event_buffer *evt)
}
/**
- * dwc3_gadget_init - Initializes gadget related registers
+ * dwc3_gadget_init - initializes gadget related registers
* @dwc: pointer to our controller context structure
*
* Returns 0 on success otherwise negative errno.
-/**
+/*
* gadget.h - DesignWare USB3 DRD Gadget Header
*
* Copyright (C) 2010-2011 Texas Instruments Incorporated - http://www.ti.com
#define to_dwc3_request(r) (container_of(r, struct dwc3_request, request))
+/**
+ * next_request - gets the next request on the given list
+ * @list: the request list to operate on
+ *
+ * Caller should take care of locking. This function return %NULL or the first
+ * request available on @list.
+ */
static inline struct dwc3_request *next_request(struct list_head *list)
{
return list_first_entry_or_null(list, struct dwc3_request, list);
}
+/**
+ * dwc3_gadget_move_started_request - move @req to the started_list
+ * @req: the request to be moved
+ *
+ * Caller should take care of locking. This function will move @req from its
+ * current list to the endpoint's started_list.
+ */
static inline void dwc3_gadget_move_started_request(struct dwc3_request *req)
{
struct dwc3_ep *dep = req->dep;
/**
* dwc3_gadget_ep_get_transfer_index - Gets transfer index from HW
- * @dwc: DesignWare USB3 Pointer
- * @number: DWC endpoint number
+ * @dep: dwc3 endpoint
*
- * Caller should take care of locking
+ * Caller should take care of locking. Returns the transfer resource
+ * index for a given endpoint.
*/
static inline u32 dwc3_gadget_ep_get_transfer_index(struct dwc3_ep *dep)
{
TP_STRUCT__entry(
__field(u32, event)
__field(u32, ep0state)
+ __dynamic_array(char, str, DWC3_MSG_MAX)
),
TP_fast_assign(
__entry->event = event;
__entry->ep0state = dwc->ep0state;
),
TP_printk("event (%08x): %s", __entry->event,
- dwc3_decode_event(__entry->event, __entry->ep0state))
+ dwc3_decode_event(__get_str(str), __entry->event,
+ __entry->ep0state))
);
DEFINE_EVENT(dwc3_log_event, dwc3_event,
__field(__u16, wValue)
__field(__u16, wIndex)
__field(__u16, wLength)
+ __dynamic_array(char, str, DWC3_MSG_MAX)
),
TP_fast_assign(
__entry->bRequestType = ctrl->bRequestType;
__entry->wIndex = le16_to_cpu(ctrl->wIndex);
__entry->wLength = le16_to_cpu(ctrl->wLength);
),
- TP_printk("bRequestType %02x bRequest %02x wValue %04x wIndex %04x wLength %d",
- __entry->bRequestType, __entry->bRequest,
- __entry->wValue, __entry->wIndex,
- __entry->wLength
+ TP_printk("%s", dwc3_decode_ctrl(__get_str(str), __entry->bRequestType,
+ __entry->bRequest, __entry->wValue,
+ __entry->wIndex, __entry->wLength)
)
);
TP_PROTO(struct dwc3_request *req),
TP_ARGS(req),
TP_STRUCT__entry(
- __dynamic_array(char, name, DWC3_MSG_MAX)
+ __string(name, req->dep->name)
__field(struct dwc3_request *, req)
__field(unsigned, actual)
__field(unsigned, length)
__field(int, no_interrupt)
),
TP_fast_assign(
- snprintf(__get_str(name), DWC3_MSG_MAX, "%s", req->dep->name);
+ __assign_str(name, req->dep->name);
__entry->req = req;
__entry->actual = req->request.actual;
__entry->length = req->request.length;
struct dwc3_gadget_ep_cmd_params *params, int cmd_status),
TP_ARGS(dep, cmd, params, cmd_status),
TP_STRUCT__entry(
- __dynamic_array(char, name, DWC3_MSG_MAX)
+ __string(name, dep->name)
__field(unsigned int, cmd)
__field(u32, param0)
__field(u32, param1)
__field(int, cmd_status)
),
TP_fast_assign(
- snprintf(__get_str(name), DWC3_MSG_MAX, "%s", dep->name);
+ __assign_str(name, dep->name);
__entry->cmd = cmd;
__entry->param0 = params->param0;
__entry->param1 = params->param1;
TP_PROTO(struct dwc3_ep *dep, struct dwc3_trb *trb),
TP_ARGS(dep, trb),
TP_STRUCT__entry(
- __dynamic_array(char, name, DWC3_MSG_MAX)
+ __string(name, dep->name)
__field(struct dwc3_trb *, trb)
__field(u32, allocated)
__field(u32, queued)
__field(u32, type)
),
TP_fast_assign(
- snprintf(__get_str(name), DWC3_MSG_MAX, "%s", dep->name);
+ __assign_str(name, dep->name);
__entry->trb = trb;
__entry->allocated = dep->allocated_requests;
__entry->queued = dep->queued_requests;
TP_PROTO(struct dwc3_ep *dep),
TP_ARGS(dep),
TP_STRUCT__entry(
- __dynamic_array(char, name, DWC3_MSG_MAX)
+ __string(name, dep->name)
__field(unsigned, maxpacket)
__field(unsigned, maxpacket_limit)
__field(unsigned, max_streams)
__field(u8, trb_dequeue)
),
TP_fast_assign(
- snprintf(__get_str(name), DWC3_MSG_MAX, "%s", dep->name);
+ __assign_str(name, dep->name);
__entry->maxpacket = dep->endpoint.maxpacket;
__entry->maxpacket_limit = dep->endpoint.maxpacket_limit;
__entry->max_streams = dep->endpoint.max_streams;
u32 reg;
int ret;
+ reg = dwc3_readl(dwc->regs, DWC3_GUSB2PHYCFG(0));
+ if (reg & DWC3_GUSB2PHYCFG_SUSPHY) {
+ reg &= ~DWC3_GUSB2PHYCFG_SUSPHY;
+ dwc3_writel(dwc->regs, DWC3_GUSB2PHYCFG(0), reg);
+ }
+
reg = DWC3_GUSB2PHYACC_NEWREGREQ | DWC3_ULPI_ADDR(addr);
dwc3_writel(dwc->regs, DWC3_GUSB2PHYACC(0), reg);
struct dwc3 *dwc = dev_get_drvdata(dev);
u32 reg;
+ reg = dwc3_readl(dwc->regs, DWC3_GUSB2PHYCFG(0));
+ if (reg & DWC3_GUSB2PHYCFG_SUSPHY) {
+ reg &= ~DWC3_GUSB2PHYCFG_SUSPHY;
+ dwc3_writel(dwc->regs, DWC3_GUSB2PHYCFG(0), reg);
+ }
+
reg = DWC3_GUSB2PHYACC_NEWREGREQ | DWC3_ULPI_ADDR(addr);
reg |= DWC3_GUSB2PHYACC_WRITE | val;
dwc3_writel(dwc->regs, DWC3_GUSB2PHYACC(0), reg);
static struct xdbc_state xdbc;
static bool early_console_keep;
-#define XDBC_TRACE
#ifdef XDBC_TRACE
#define xdbc_trace trace_printk
#else
don't have this kind of hardware (except maybe inside Linux PDAs).
For more information, see <http://www.linux-usb.org/gadget> and
- the kernel DocBook documentation for this API.
+ the kernel documentation for this API.
if USB_GADGET
config USB_U_ETHER
tristate
+config USB_U_AUDIO
+ tristate
+
config USB_F_SERIAL
tristate
config USB_F_UAC1
tristate
+config USB_F_UAC1_LEGACY
+ tristate
+
config USB_F_UAC2
tristate
depends on SND
select USB_LIBCOMPOSITE
select SND_PCM
+ select USB_U_AUDIO
select USB_F_UAC1
help
This Audio function implements 1 AudioControl interface,
1 AudioStreaming Interface each for USB-OUT and USB-IN.
- This driver requires a real Audio codec to be present
- on the device.
+ This driver doesn't expect any real Audio codec to be present
+ on the device - the audio streams are simply sinked to and
+ sourced from a virtual ALSA sound card created. The user-space
+ application may choose to do whatever it wants with the data
+ received from the USB Host and choose to provide whatever it
+ wants as audio data to the USB Host.
+
+config USB_CONFIGFS_F_UAC1_LEGACY
+ bool "Audio Class 1.0 (legacy implementation)"
+ depends on USB_CONFIGFS
+ depends on SND
+ select USB_LIBCOMPOSITE
+ select SND_PCM
+ select USB_F_UAC1_LEGACY
+ help
+ This Audio function implements 1 AudioControl interface,
+ 1 AudioStreaming Interface each for USB-OUT and USB-IN.
+ This is a legacy driver and requires a real Audio codec
+ to be present on the device.
config USB_CONFIGFS_F_UAC2
bool "Audio Class 2.0"
depends on SND
select USB_LIBCOMPOSITE
select SND_PCM
+ select USB_U_AUDIO
select USB_F_UAC2
help
This Audio function is compatible with USB Audio Class
static int bos_desc(struct usb_composite_dev *cdev)
{
struct usb_ext_cap_descriptor *usb_ext;
- struct usb_ss_cap_descriptor *ss_cap;
struct usb_dcd_config_params dcd_config_params;
struct usb_bos_descriptor *bos = cdev->req->buf;
* The Superspeed USB Capability descriptor shall be implemented by all
* SuperSpeed devices.
*/
- ss_cap = cdev->req->buf + le16_to_cpu(bos->wTotalLength);
- bos->bNumDeviceCaps++;
- le16_add_cpu(&bos->wTotalLength, USB_DT_USB_SS_CAP_SIZE);
- ss_cap->bLength = USB_DT_USB_SS_CAP_SIZE;
- ss_cap->bDescriptorType = USB_DT_DEVICE_CAPABILITY;
- ss_cap->bDevCapabilityType = USB_SS_CAP_TYPE;
- ss_cap->bmAttributes = 0; /* LTM is not supported yet */
- ss_cap->wSpeedSupported = cpu_to_le16(USB_LOW_SPEED_OPERATION |
- USB_FULL_SPEED_OPERATION |
- USB_HIGH_SPEED_OPERATION |
- USB_5GBPS_OPERATION);
- ss_cap->bFunctionalitySupport = USB_LOW_SPEED_OPERATION;
-
- /* Get Controller configuration */
- if (cdev->gadget->ops->get_config_params)
- cdev->gadget->ops->get_config_params(&dcd_config_params);
- else {
- dcd_config_params.bU1devExitLat = USB_DEFAULT_U1_DEV_EXIT_LAT;
- dcd_config_params.bU2DevExitLat =
- cpu_to_le16(USB_DEFAULT_U2_DEV_EXIT_LAT);
+ if (gadget_is_superspeed(cdev->gadget)) {
+ struct usb_ss_cap_descriptor *ss_cap;
+
+ ss_cap = cdev->req->buf + le16_to_cpu(bos->wTotalLength);
+ bos->bNumDeviceCaps++;
+ le16_add_cpu(&bos->wTotalLength, USB_DT_USB_SS_CAP_SIZE);
+ ss_cap->bLength = USB_DT_USB_SS_CAP_SIZE;
+ ss_cap->bDescriptorType = USB_DT_DEVICE_CAPABILITY;
+ ss_cap->bDevCapabilityType = USB_SS_CAP_TYPE;
+ ss_cap->bmAttributes = 0; /* LTM is not supported yet */
+ ss_cap->wSpeedSupported = cpu_to_le16(USB_LOW_SPEED_OPERATION |
+ USB_FULL_SPEED_OPERATION |
+ USB_HIGH_SPEED_OPERATION |
+ USB_5GBPS_OPERATION);
+ ss_cap->bFunctionalitySupport = USB_LOW_SPEED_OPERATION;
+
+ /* Get Controller configuration */
+ if (cdev->gadget->ops->get_config_params) {
+ cdev->gadget->ops->get_config_params(
+ &dcd_config_params);
+ } else {
+ dcd_config_params.bU1devExitLat =
+ USB_DEFAULT_U1_DEV_EXIT_LAT;
+ dcd_config_params.bU2DevExitLat =
+ cpu_to_le16(USB_DEFAULT_U2_DEV_EXIT_LAT);
+ }
+ ss_cap->bU1devExitLat = dcd_config_params.bU1devExitLat;
+ ss_cap->bU2DevExitLat = dcd_config_params.bU2DevExitLat;
}
- ss_cap->bU1devExitLat = dcd_config_params.bU1devExitLat;
- ss_cap->bU2DevExitLat = dcd_config_params.bU2DevExitLat;
/* The SuperSpeedPlus USB Device Capability descriptor */
if (gadget_is_superspeed_plus(cdev->gadget)) {
cdev->desc.bcdUSB = cpu_to_le16(0x0210);
}
} else {
- cdev->desc.bcdUSB = cpu_to_le16(0x0200);
+ if (gadget->lpm_capable)
+ cdev->desc.bcdUSB = cpu_to_le16(0x0201);
+ else
+ cdev->desc.bcdUSB = cpu_to_le16(0x0200);
}
value = min(w_length, (u16) sizeof cdev->desc);
value = min(w_length, (u16) value);
break;
case USB_DT_BOS:
- if (gadget_is_superspeed(gadget)) {
+ if (gadget_is_superspeed(gadget) ||
+ gadget->lpm_capable) {
value = bos_desc(cdev);
value = min(w_length, (u16) value);
}
static ssize_t os_desc_use_show(struct config_item *item, char *page)
{
- return sprintf(page, "%d",
+ return sprintf(page, "%d\n",
os_desc_item_to_gadget_info(item)->use_os_desc);
}
static ssize_t os_desc_b_vendor_code_show(struct config_item *item, char *page)
{
- return sprintf(page, "%d",
+ return sprintf(page, "0x%02x\n",
os_desc_item_to_gadget_info(item)->b_vendor_code);
}
static ssize_t os_desc_qw_sign_show(struct config_item *item, char *page)
{
struct gadget_info *gi = os_desc_item_to_gadget_info(item);
+ int res;
+
+ res = utf16s_to_utf8s((wchar_t *) gi->qw_sign, OS_STRING_QW_SIGN_LEN,
+ UTF16_LITTLE_ENDIAN, page, PAGE_SIZE - 1);
+ page[res++] = '\n';
- memcpy(page, gi->qw_sign, OS_STRING_QW_SIGN_LEN);
- return OS_STRING_QW_SIGN_LEN;
+ return res;
}
static ssize_t os_desc_qw_sign_store(struct config_item *item, const char *page,
static ssize_t ext_prop_type_show(struct config_item *item, char *page)
{
- return sprintf(page, "%d", to_usb_os_desc_ext_prop(item)->type);
+ return sprintf(page, "%d\n", to_usb_os_desc_ext_prop(item)->type);
}
static ssize_t ext_prop_type_store(struct config_item *item,
obj-$(CONFIG_USB_F_MASS_STORAGE)+= usb_f_mass_storage.o
usb_f_fs-y := f_fs.o
obj-$(CONFIG_USB_F_FS) += usb_f_fs.o
-usb_f_uac1-y := f_uac1.o u_uac1.o
+obj-$(CONFIG_USB_U_AUDIO) += u_audio.o
+usb_f_uac1-y := f_uac1.o
obj-$(CONFIG_USB_F_UAC1) += usb_f_uac1.o
+usb_f_uac1_legacy-y := f_uac1_legacy.o u_uac1_legacy.o
+obj-$(CONFIG_USB_F_UAC1_LEGACY) += usb_f_uac1_legacy.o
usb_f_uac2-y := f_uac2.o
obj-$(CONFIG_USB_F_UAC2) += usb_f_uac2.o
usb_f_uvc-y := f_uvc.o uvc_queue.o uvc_v4l2.o uvc_video.o uvc_configfs.o
struct ffs_epfile {
/* Protects ep->ep and ep->req. */
struct mutex mutex;
- wait_queue_head_t wait;
struct ffs_data *ffs;
struct ffs_ep *ep; /* P: ffs->eps_lock */
if (file->f_flags & O_NONBLOCK)
return -EAGAIN;
- ret = wait_event_interruptible(epfile->wait, (ep = epfile->ep));
+ ret = wait_event_interruptible(
+ epfile->ffs->wait, (ep = epfile->ep));
if (ret)
return -EINTR;
}
unsigned long value)
{
struct ffs_epfile *epfile = file->private_data;
+ struct ffs_ep *ep;
int ret;
ENTER();
if (WARN_ON(epfile->ffs->state != FFS_ACTIVE))
return -ENODEV;
+ /* Wait for endpoint to be enabled */
+ ep = epfile->ep;
+ if (!ep) {
+ if (file->f_flags & O_NONBLOCK)
+ return -EAGAIN;
+
+ ret = wait_event_interruptible(
+ epfile->ffs->wait, (ep = epfile->ep));
+ if (ret)
+ return -EINTR;
+ }
+
spin_lock_irq(&epfile->ffs->eps_lock);
- if (likely(epfile->ep)) {
- switch (code) {
- case FUNCTIONFS_FIFO_STATUS:
- ret = usb_ep_fifo_status(epfile->ep->ep);
- break;
- case FUNCTIONFS_FIFO_FLUSH:
- usb_ep_fifo_flush(epfile->ep->ep);
- ret = 0;
- break;
- case FUNCTIONFS_CLEAR_HALT:
- ret = usb_ep_clear_halt(epfile->ep->ep);
- break;
- case FUNCTIONFS_ENDPOINT_REVMAP:
- ret = epfile->ep->num;
- break;
- case FUNCTIONFS_ENDPOINT_DESC:
- {
- int desc_idx;
- struct usb_endpoint_descriptor *desc;
- switch (epfile->ffs->gadget->speed) {
- case USB_SPEED_SUPER:
- desc_idx = 2;
- break;
- case USB_SPEED_HIGH:
- desc_idx = 1;
- break;
- default:
- desc_idx = 0;
- }
- desc = epfile->ep->descs[desc_idx];
+ /* In the meantime, endpoint got disabled or changed. */
+ if (epfile->ep != ep) {
+ spin_unlock_irq(&epfile->ffs->eps_lock);
+ return -ESHUTDOWN;
+ }
- spin_unlock_irq(&epfile->ffs->eps_lock);
- ret = copy_to_user((void *)value, desc, desc->bLength);
- if (ret)
- ret = -EFAULT;
- return ret;
- }
+ switch (code) {
+ case FUNCTIONFS_FIFO_STATUS:
+ ret = usb_ep_fifo_status(epfile->ep->ep);
+ break;
+ case FUNCTIONFS_FIFO_FLUSH:
+ usb_ep_fifo_flush(epfile->ep->ep);
+ ret = 0;
+ break;
+ case FUNCTIONFS_CLEAR_HALT:
+ ret = usb_ep_clear_halt(epfile->ep->ep);
+ break;
+ case FUNCTIONFS_ENDPOINT_REVMAP:
+ ret = epfile->ep->num;
+ break;
+ case FUNCTIONFS_ENDPOINT_DESC:
+ {
+ int desc_idx;
+ struct usb_endpoint_descriptor *desc;
+
+ switch (epfile->ffs->gadget->speed) {
+ case USB_SPEED_SUPER:
+ desc_idx = 2;
+ break;
+ case USB_SPEED_HIGH:
+ desc_idx = 1;
+ break;
default:
- ret = -ENOTTY;
+ desc_idx = 0;
}
- } else {
- ret = -ENODEV;
+ desc = epfile->ep->descs[desc_idx];
+
+ spin_unlock_irq(&epfile->ffs->eps_lock);
+ ret = copy_to_user((void *)value, desc, desc->bLength);
+ if (ret)
+ ret = -EFAULT;
+ return ret;
+ }
+ default:
+ ret = -ENOTTY;
}
spin_unlock_irq(&epfile->ffs->eps_lock);
pr_info("%s(): freeing\n", __func__);
ffs_data_clear(ffs);
BUG_ON(waitqueue_active(&ffs->ev.waitq) ||
- waitqueue_active(&ffs->ep0req_completion.wait));
+ waitqueue_active(&ffs->ep0req_completion.wait) ||
+ waitqueue_active(&ffs->wait));
kfree(ffs->dev_name);
kfree(ffs);
}
mutex_init(&ffs->mutex);
spin_lock_init(&ffs->eps_lock);
init_waitqueue_head(&ffs->ev.waitq);
+ init_waitqueue_head(&ffs->wait);
init_completion(&ffs->ep0req_completion);
/* XXX REVISIT need to update it in some places, or do we? */
for (i = 1; i <= count; ++i, ++epfile) {
epfile->ffs = ffs;
mutex_init(&epfile->mutex);
- init_waitqueue_head(&epfile->wait);
if (ffs->user_flags & FUNCTIONFS_VIRTUAL_ADDR)
sprintf(epfile->name, "ep%02x", ffs->eps_addrmap[i]);
else
ENTER();
for (; count; --count, ++epfile) {
- BUG_ON(mutex_is_locked(&epfile->mutex) ||
- waitqueue_active(&epfile->wait));
+ BUG_ON(mutex_is_locked(&epfile->mutex));
if (epfile->dentry) {
d_delete(epfile->dentry);
dput(epfile->dentry);
break;
}
- wake_up(&epfile->wait);
-
++ep;
++epfile;
}
+
+ wake_up_interruptible(&ffs->wait);
spin_unlock_irqrestore(&func->ffs->eps_lock, flags);
return ret;
struct usb_gadget *gadget;
struct usb_composite_dev *cdev;
struct fsg_dev *fsg, *new_fsg;
+ wait_queue_head_t io_wait;
wait_queue_head_t fsg_wait;
/* filesem protects: backing files in use */
struct rw_semaphore filesem;
- /* lock protects: state, all the req_busy's */
+ /* lock protects: state and thread_task */
spinlock_t lock;
struct usb_ep *ep0; /* Copy of gadget->ep0 */
unsigned int running:1;
unsigned int sysfs:1;
- int thread_wakeup_needed;
struct completion thread_notifier;
struct task_struct *thread_task;
static int exception_in_progress(struct fsg_common *common)
{
- return common->state > FSG_STATE_IDLE;
+ return common->state > FSG_STATE_NORMAL;
}
/* Make bulk-out requests be divisible by the maxpacket size */
/* These routines may be called in process context or in_irq */
-/* Caller must hold fsg->lock */
-static void wakeup_thread(struct fsg_common *common)
-{
- /*
- * Ensure the reading of thread_wakeup_needed
- * and the writing of bh->state are completed
- */
- smp_mb();
- /* Tell the main thread that something has happened */
- common->thread_wakeup_needed = 1;
- if (common->thread_task)
- wake_up_process(common->thread_task);
-}
-
static void raise_exception(struct fsg_common *common, enum fsg_state new_state)
{
unsigned long flags;
if (req->status == -ECONNRESET) /* Request was cancelled */
usb_ep_fifo_flush(ep);
- /* Hold the lock while we update the request and buffer states */
- smp_wmb();
- spin_lock(&common->lock);
- bh->inreq_busy = 0;
- bh->state = BUF_STATE_EMPTY;
- wakeup_thread(common);
- spin_unlock(&common->lock);
+ /* Synchronize with the smp_load_acquire() in sleep_thread() */
+ smp_store_release(&bh->state, BUF_STATE_EMPTY);
+ wake_up(&common->io_wait);
}
static void bulk_out_complete(struct usb_ep *ep, struct usb_request *req)
if (req->status == -ECONNRESET) /* Request was cancelled */
usb_ep_fifo_flush(ep);
- /* Hold the lock while we update the request and buffer states */
- smp_wmb();
- spin_lock(&common->lock);
- bh->outreq_busy = 0;
- bh->state = BUF_STATE_FULL;
- wakeup_thread(common);
- spin_unlock(&common->lock);
+ /* Synchronize with the smp_load_acquire() in sleep_thread() */
+ smp_store_release(&bh->state, BUF_STATE_FULL);
+ wake_up(&common->io_wait);
}
static int _fsg_common_get_max_lun(struct fsg_common *common)
* and reinitialize our state.
*/
DBG(fsg, "bulk reset request\n");
- raise_exception(fsg->common, FSG_STATE_RESET);
+ raise_exception(fsg->common, FSG_STATE_PROTOCOL_RESET);
return USB_GADGET_DELAYED_STATUS;
case US_BULK_GET_MAX_LUN:
/* All the following routines run in process context */
/* Use this for bulk or interrupt transfers, not ep0 */
-static void start_transfer(struct fsg_dev *fsg, struct usb_ep *ep,
- struct usb_request *req, int *pbusy,
- enum fsg_buffer_state *state)
+static int start_transfer(struct fsg_dev *fsg, struct usb_ep *ep,
+ struct usb_request *req)
{
int rc;
if (ep == fsg->bulk_in)
dump_msg(fsg, "bulk-in", req->buf, req->length);
- spin_lock_irq(&fsg->common->lock);
- *pbusy = 1;
- *state = BUF_STATE_BUSY;
- spin_unlock_irq(&fsg->common->lock);
-
rc = usb_ep_queue(ep, req, GFP_KERNEL);
- if (rc == 0)
- return; /* All good, we're done */
-
- *pbusy = 0;
- *state = BUF_STATE_EMPTY;
+ if (rc) {
- /* We can't do much more than wait for a reset */
+ /* We can't do much more than wait for a reset */
+ req->status = rc;
- /*
- * Note: currently the net2280 driver fails zero-length
- * submissions if DMA is enabled.
- */
- if (rc != -ESHUTDOWN && !(rc == -EOPNOTSUPP && req->length == 0))
- WARNING(fsg, "error in submission: %s --> %d\n", ep->name, rc);
+ /*
+ * Note: currently the net2280 driver fails zero-length
+ * submissions if DMA is enabled.
+ */
+ if (rc != -ESHUTDOWN &&
+ !(rc == -EOPNOTSUPP && req->length == 0))
+ WARNING(fsg, "error in submission: %s --> %d\n",
+ ep->name, rc);
+ }
+ return rc;
}
static bool start_in_transfer(struct fsg_common *common, struct fsg_buffhd *bh)
{
if (!fsg_is_set(common))
return false;
- start_transfer(common->fsg, common->fsg->bulk_in,
- bh->inreq, &bh->inreq_busy, &bh->state);
+ bh->state = BUF_STATE_SENDING;
+ if (start_transfer(common->fsg, common->fsg->bulk_in, bh->inreq))
+ bh->state = BUF_STATE_EMPTY;
return true;
}
{
if (!fsg_is_set(common))
return false;
- start_transfer(common->fsg, common->fsg->bulk_out,
- bh->outreq, &bh->outreq_busy, &bh->state);
+ bh->state = BUF_STATE_RECEIVING;
+ if (start_transfer(common->fsg, common->fsg->bulk_out, bh->outreq))
+ bh->state = BUF_STATE_FULL;
return true;
}
-static int sleep_thread(struct fsg_common *common, bool can_freeze)
+static int sleep_thread(struct fsg_common *common, bool can_freeze,
+ struct fsg_buffhd *bh)
{
- int rc = 0;
-
- /* Wait until a signal arrives or we are woken up */
- for (;;) {
- if (can_freeze)
- try_to_freeze();
- set_current_state(TASK_INTERRUPTIBLE);
- if (signal_pending(current)) {
- rc = -EINTR;
- break;
- }
- if (common->thread_wakeup_needed)
- break;
- schedule();
- }
- __set_current_state(TASK_RUNNING);
- common->thread_wakeup_needed = 0;
+ int rc;
- /*
- * Ensure the writing of thread_wakeup_needed
- * and the reading of bh->state are completed
- */
- smp_mb();
- return rc;
+ /* Wait until a signal arrives or bh is no longer busy */
+ if (can_freeze)
+ /*
+ * synchronize with the smp_store_release(&bh->state) in
+ * bulk_in_complete() or bulk_out_complete()
+ */
+ rc = wait_event_freezable(common->io_wait,
+ bh && smp_load_acquire(&bh->state) >=
+ BUF_STATE_EMPTY);
+ else
+ rc = wait_event_interruptible(common->io_wait,
+ bh && smp_load_acquire(&bh->state) >=
+ BUF_STATE_EMPTY);
+ return rc ? -EINTR : 0;
}
/* Wait for the next buffer to become available */
bh = common->next_buffhd_to_fill;
- while (bh->state != BUF_STATE_EMPTY) {
- rc = sleep_thread(common, false);
- if (rc)
- return rc;
- }
+ rc = sleep_thread(common, false, bh);
+ if (rc)
+ return rc;
/*
* If we were asked to read past the end of file,
bh = common->next_buffhd_to_drain;
if (bh->state == BUF_STATE_EMPTY && !get_some_more)
break; /* We stopped early */
- if (bh->state == BUF_STATE_FULL) {
- smp_rmb();
- common->next_buffhd_to_drain = bh->next;
- bh->state = BUF_STATE_EMPTY;
-
- /* Did something go wrong with the transfer? */
- if (bh->outreq->status != 0) {
- curlun->sense_data = SS_COMMUNICATION_FAILURE;
- curlun->sense_data_info =
+
+ /* Wait for the data to be received */
+ rc = sleep_thread(common, false, bh);
+ if (rc)
+ return rc;
+
+ common->next_buffhd_to_drain = bh->next;
+ bh->state = BUF_STATE_EMPTY;
+
+ /* Did something go wrong with the transfer? */
+ if (bh->outreq->status != 0) {
+ curlun->sense_data = SS_COMMUNICATION_FAILURE;
+ curlun->sense_data_info =
file_offset >> curlun->blkbits;
- curlun->info_valid = 1;
- break;
- }
+ curlun->info_valid = 1;
+ break;
+ }
- amount = bh->outreq->actual;
- if (curlun->file_length - file_offset < amount) {
- LERROR(curlun,
- "write %u @ %llu beyond end %llu\n",
+ amount = bh->outreq->actual;
+ if (curlun->file_length - file_offset < amount) {
+ LERROR(curlun, "write %u @ %llu beyond end %llu\n",
amount, (unsigned long long)file_offset,
(unsigned long long)curlun->file_length);
- amount = curlun->file_length - file_offset;
- }
+ amount = curlun->file_length - file_offset;
+ }
- /* Don't accept excess data. The spec doesn't say
- * what to do in this case. We'll ignore the error.
- */
- amount = min(amount, bh->bulk_out_intended_length);
-
- /* Don't write a partial block */
- amount = round_down(amount, curlun->blksize);
- if (amount == 0)
- goto empty_write;
-
- /* Perform the write */
- file_offset_tmp = file_offset;
- nwritten = vfs_write(curlun->filp,
- (char __user *)bh->buf,
- amount, &file_offset_tmp);
- VLDBG(curlun, "file write %u @ %llu -> %d\n", amount,
- (unsigned long long)file_offset, (int)nwritten);
- if (signal_pending(current))
- return -EINTR; /* Interrupted! */
-
- if (nwritten < 0) {
- LDBG(curlun, "error in file write: %d\n",
- (int)nwritten);
- nwritten = 0;
- } else if (nwritten < amount) {
- LDBG(curlun, "partial file write: %d/%u\n",
- (int)nwritten, amount);
- nwritten = round_down(nwritten, curlun->blksize);
- }
- file_offset += nwritten;
- amount_left_to_write -= nwritten;
- common->residue -= nwritten;
+ /*
+ * Don't accept excess data. The spec doesn't say
+ * what to do in this case. We'll ignore the error.
+ */
+ amount = min(amount, bh->bulk_out_intended_length);
- /* If an error occurred, report it and its position */
- if (nwritten < amount) {
- curlun->sense_data = SS_WRITE_ERROR;
- curlun->sense_data_info =
+ /* Don't write a partial block */
+ amount = round_down(amount, curlun->blksize);
+ if (amount == 0)
+ goto empty_write;
+
+ /* Perform the write */
+ file_offset_tmp = file_offset;
+ nwritten = vfs_write(curlun->filp, (char __user *)bh->buf,
+ amount, &file_offset_tmp);
+ VLDBG(curlun, "file write %u @ %llu -> %d\n", amount,
+ (unsigned long long)file_offset, (int)nwritten);
+ if (signal_pending(current))
+ return -EINTR; /* Interrupted! */
+
+ if (nwritten < 0) {
+ LDBG(curlun, "error in file write: %d\n",
+ (int) nwritten);
+ nwritten = 0;
+ } else if (nwritten < amount) {
+ LDBG(curlun, "partial file write: %d/%u\n",
+ (int) nwritten, amount);
+ nwritten = round_down(nwritten, curlun->blksize);
+ }
+ file_offset += nwritten;
+ amount_left_to_write -= nwritten;
+ common->residue -= nwritten;
+
+ /* If an error occurred, report it and its position */
+ if (nwritten < amount) {
+ curlun->sense_data = SS_WRITE_ERROR;
+ curlun->sense_data_info =
file_offset >> curlun->blkbits;
- curlun->info_valid = 1;
- break;
- }
+ curlun->info_valid = 1;
+ break;
+ }
empty_write:
- /* Did the host decide to stop early? */
- if (bh->outreq->actual < bh->bulk_out_intended_length) {
- common->short_packet_received = 1;
- break;
- }
- continue;
+ /* Did the host decide to stop early? */
+ if (bh->outreq->actual < bh->bulk_out_intended_length) {
+ common->short_packet_received = 1;
+ break;
}
-
- /* Wait for something to happen */
- rc = sleep_thread(common, false);
- if (rc)
- return rc;
}
return -EIO; /* No default reply */
static int throw_away_data(struct fsg_common *common)
{
- struct fsg_buffhd *bh;
+ struct fsg_buffhd *bh, *bh2;
u32 amount;
int rc;
bh->state != BUF_STATE_EMPTY || common->usb_amount_left > 0;
bh = common->next_buffhd_to_drain) {
- /* Throw away the data in a filled buffer */
- if (bh->state == BUF_STATE_FULL) {
- smp_rmb();
- bh->state = BUF_STATE_EMPTY;
- common->next_buffhd_to_drain = bh->next;
-
- /* A short packet or an error ends everything */
- if (bh->outreq->actual < bh->bulk_out_intended_length ||
- bh->outreq->status != 0) {
- raise_exception(common,
- FSG_STATE_ABORT_BULK_OUT);
- return -EINTR;
- }
- continue;
- }
-
/* Try to submit another request if we need one */
- bh = common->next_buffhd_to_fill;
- if (bh->state == BUF_STATE_EMPTY
- && common->usb_amount_left > 0) {
+ bh2 = common->next_buffhd_to_fill;
+ if (bh2->state == BUF_STATE_EMPTY &&
+ common->usb_amount_left > 0) {
amount = min(common->usb_amount_left, FSG_BUFLEN);
/*
* equal to the buffer size, which is divisible by
* the bulk-out maxpacket size.
*/
- set_bulk_out_req_length(common, bh, amount);
- if (!start_out_transfer(common, bh))
+ set_bulk_out_req_length(common, bh2, amount);
+ if (!start_out_transfer(common, bh2))
/* Dunno what to do if common->fsg is NULL */
return -EIO;
- common->next_buffhd_to_fill = bh->next;
+ common->next_buffhd_to_fill = bh2->next;
common->usb_amount_left -= amount;
continue;
}
- /* Otherwise wait for something to happen */
- rc = sleep_thread(common, true);
+ /* Wait for the data to be received */
+ rc = sleep_thread(common, false, bh);
if (rc)
return rc;
+
+ /* Throw away the data in a filled buffer */
+ bh->state = BUF_STATE_EMPTY;
+ common->next_buffhd_to_drain = bh->next;
+
+ /* A short packet or an error ends everything */
+ if (bh->outreq->actual < bh->bulk_out_intended_length ||
+ bh->outreq->status != 0) {
+ raise_exception(common, FSG_STATE_ABORT_BULK_OUT);
+ return -EINTR;
+ }
}
return 0;
}
return rc;
}
-static int send_status(struct fsg_common *common)
+static void send_status(struct fsg_common *common)
{
struct fsg_lun *curlun = common->curlun;
struct fsg_buffhd *bh;
/* Wait for the next buffer to become available */
bh = common->next_buffhd_to_fill;
- while (bh->state != BUF_STATE_EMPTY) {
- rc = sleep_thread(common, true);
- if (rc)
- return rc;
- }
+ rc = sleep_thread(common, false, bh);
+ if (rc)
+ return;
if (curlun) {
sd = curlun->sense_data;
bh->inreq->zero = 0;
if (!start_in_transfer(common, bh))
/* Don't know what to do if common->fsg is NULL */
- return -EIO;
+ return;
common->next_buffhd_to_fill = bh->next;
- return 0;
+ return;
}
/* Wait for the next buffer to become available for data or status */
bh = common->next_buffhd_to_fill;
common->next_buffhd_to_drain = bh;
- while (bh->state != BUF_STATE_EMPTY) {
- rc = sleep_thread(common, true);
- if (rc)
- return rc;
- }
+ rc = sleep_thread(common, false, bh);
+ if (rc)
+ return rc;
+
common->phase_error = 0;
common->short_packet_received = 0;
/* Wait for the next buffer to become available */
bh = common->next_buffhd_to_fill;
- while (bh->state != BUF_STATE_EMPTY) {
- rc = sleep_thread(common, true);
- if (rc)
- return rc;
- }
+ rc = sleep_thread(common, true, bh);
+ if (rc)
+ return rc;
/* Queue a request to read a Bulk-only CBW */
set_bulk_out_req_length(common, bh, US_BULK_CB_WRAP_LEN);
*/
/* Wait for the CBW to arrive */
- while (bh->state != BUF_STATE_FULL) {
- rc = sleep_thread(common, true);
- if (rc)
- return rc;
- }
- smp_rmb();
+ rc = sleep_thread(common, true, bh);
+ if (rc)
+ return rc;
+
rc = fsg_is_set(common) ? received_cbw(common->fsg, bh) : -EIO;
bh->state = BUF_STATE_EMPTY;
if (!sig)
break;
if (sig != SIGUSR1) {
+ spin_lock_irq(&common->lock);
if (common->state < FSG_STATE_EXIT)
DBG(common, "Main thread exiting on signal\n");
- raise_exception(common, FSG_STATE_EXIT);
+ common->state = FSG_STATE_EXIT;
+ spin_unlock_irq(&common->lock);
}
}
if (likely(common->fsg)) {
for (i = 0; i < common->fsg_num_buffers; ++i) {
bh = &common->buffhds[i];
- if (bh->inreq_busy)
+ if (bh->state == BUF_STATE_SENDING)
usb_ep_dequeue(common->fsg->bulk_in, bh->inreq);
- if (bh->outreq_busy)
+ if (bh->state == BUF_STATE_RECEIVING)
usb_ep_dequeue(common->fsg->bulk_out,
bh->outreq);
- }
- /* Wait until everything is idle */
- for (;;) {
- int num_active = 0;
- for (i = 0; i < common->fsg_num_buffers; ++i) {
- bh = &common->buffhds[i];
- num_active += bh->inreq_busy + bh->outreq_busy;
- }
- if (num_active == 0)
- break;
- if (sleep_thread(common, true))
+ /* Wait for a transfer to become idle */
+ if (sleep_thread(common, false, bh))
return;
}
common->next_buffhd_to_drain = &common->buffhds[0];
exception_req_tag = common->exception_req_tag;
old_state = common->state;
+ common->state = FSG_STATE_NORMAL;
- if (old_state == FSG_STATE_ABORT_BULK_OUT)
- common->state = FSG_STATE_STATUS_PHASE;
- else {
+ if (old_state != FSG_STATE_ABORT_BULK_OUT) {
for (i = 0; i < ARRAY_SIZE(common->luns); ++i) {
curlun = common->luns[i];
if (!curlun)
curlun->sense_data_info = 0;
curlun->info_valid = 0;
}
- common->state = FSG_STATE_IDLE;
}
spin_unlock_irq(&common->lock);
/* Carry out any extra actions required for the exception */
switch (old_state) {
+ case FSG_STATE_NORMAL:
+ break;
+
case FSG_STATE_ABORT_BULK_OUT:
send_status(common);
- spin_lock_irq(&common->lock);
- if (common->state == FSG_STATE_STATUS_PHASE)
- common->state = FSG_STATE_IDLE;
- spin_unlock_irq(&common->lock);
break;
- case FSG_STATE_RESET:
+ case FSG_STATE_PROTOCOL_RESET:
/*
* In case we were forced against our will to halt a
* bulk endpoint, clear the halt now. (The SuperH UDC
break;
case FSG_STATE_EXIT:
- case FSG_STATE_TERMINATED:
do_set_interface(common, NULL); /* Free resources */
spin_lock_irq(&common->lock);
common->state = FSG_STATE_TERMINATED; /* Stop the thread */
spin_unlock_irq(&common->lock);
break;
- case FSG_STATE_INTERFACE_CHANGE:
- case FSG_STATE_DISCONNECT:
- case FSG_STATE_COMMAND_PHASE:
- case FSG_STATE_DATA_PHASE:
- case FSG_STATE_STATUS_PHASE:
- case FSG_STATE_IDLE:
+ case FSG_STATE_TERMINATED:
break;
}
}
}
if (!common->running) {
- sleep_thread(common, true);
+ sleep_thread(common, true, NULL);
continue;
}
- if (get_next_command(common))
+ if (get_next_command(common) || exception_in_progress(common))
continue;
-
- spin_lock_irq(&common->lock);
- if (!exception_in_progress(common))
- common->state = FSG_STATE_DATA_PHASE;
- spin_unlock_irq(&common->lock);
-
- if (do_scsi_command(common) || finish_reply(common))
+ if (do_scsi_command(common) || exception_in_progress(common))
continue;
-
- spin_lock_irq(&common->lock);
- if (!exception_in_progress(common))
- common->state = FSG_STATE_STATUS_PHASE;
- spin_unlock_irq(&common->lock);
-
- if (send_status(common))
+ if (finish_reply(common) || exception_in_progress(common))
continue;
-
- spin_lock_irq(&common->lock);
- if (!exception_in_progress(common))
- common->state = FSG_STATE_IDLE;
- spin_unlock_irq(&common->lock);
+ send_status(common);
}
spin_lock_irq(&common->lock);
spin_lock_init(&common->lock);
kref_init(&common->ref);
init_completion(&common->thread_notifier);
+ init_waitqueue_head(&common->io_wait);
init_waitqueue_head(&common->fsg_wait);
common->state = FSG_STATE_TERMINATED;
memset(common->luns, 0, sizeof(common->luns));
if (common->state != FSG_STATE_TERMINATED) {
raise_exception(common, FSG_STATE_EXIT);
wait_for_completion(&common->thread_notifier);
- common->thread_task = NULL;
}
for (i = 0; i < ARRAY_SIZE(common->luns); ++i) {
}
if (!common->thread_task) {
- common->state = FSG_STATE_IDLE;
+ common->state = FSG_STATE_NORMAL;
common->thread_task =
kthread_create(fsg_main_thread, common, "file-storage");
if (IS_ERR(common->thread_task)) {
- int ret = PTR_ERR(common->thread_task);
+ ret = PTR_ERR(common->thread_task);
common->thread_task = NULL;
common->state = FSG_STATE_TERMINATED;
return ret;
/*
- * f_audio.c -- USB Audio class function driver
- *
- * Copyright (C) 2008 Bryan Wu <cooloney@kernel.org>
- * Copyright (C) 2008 Analog Devices, Inc
+ * f_uac1.c -- USB Audio Class 1.0 Function (using u_audio API)
*
- * Enter bugs at http://blackfin.uclinux.org/
+ * Copyright (C) 2016 Ruslan Bilovol <ruslan.bilovol@gmail.com>
*
- * Licensed under the GPL-2 or later.
+ * This driver doesn't expect any real Audio codec to be present
+ * on the device - the audio streams are simply sinked to and
+ * sourced from a virtual ALSA sound card created.
+ *
+ * This file is based on f_uac1.c which is
+ * Copyright (C) 2008 Bryan Wu <cooloney@kernel.org>
+ * Copyright (C) 2008 Analog Devices, Inc
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
*/
-#include <linux/slab.h>
-#include <linux/kernel.h>
+#include <linux/usb/audio.h>
#include <linux/module.h>
-#include <linux/device.h>
-#include <linux/atomic.h>
+#include "u_audio.h"
#include "u_uac1.h"
-static int generic_set_cmd(struct usb_audio_control *con, u8 cmd, int value);
-static int generic_get_cmd(struct usb_audio_control *con, u8 cmd);
+struct f_uac1 {
+ struct g_audio g_audio;
+ u8 ac_intf, as_in_intf, as_out_intf;
+ u8 ac_alt, as_in_alt, as_out_alt; /* needed for get_alt() */
+};
+
+static inline struct f_uac1 *func_to_uac1(struct usb_function *f)
+{
+ return container_of(f, struct f_uac1, g_audio.func);
+}
/*
* DESCRIPTORS ... most are static, but strings and full
*/
/*
- * We have two interfaces- AudioControl and AudioStreaming
- * TODO: only supcard playback currently
+ * We have three interfaces - one AudioControl and two AudioStreaming
+ *
+ * The driver implements a simple UAC_1 topology.
+ * USB-OUT -> IT_1 -> OT_2 -> ALSA_Capture
+ * ALSA_Playback -> IT_3 -> OT_4 -> USB-IN
*/
-#define F_AUDIO_AC_INTERFACE 0
-#define F_AUDIO_AS_INTERFACE 1
-#define F_AUDIO_NUM_INTERFACES 1
+#define F_AUDIO_AC_INTERFACE 0
+#define F_AUDIO_AS_OUT_INTERFACE 1
+#define F_AUDIO_AS_IN_INTERFACE 2
+/* Number of streaming interfaces */
+#define F_AUDIO_NUM_INTERFACES 2
/* B.3.1 Standard AC Interface Descriptor */
static struct usb_interface_descriptor ac_interface_desc = {
* The number of AudioStreaming and MIDIStreaming interfaces
* in the Audio Interface Collection
*/
-DECLARE_UAC_AC_HEADER_DESCRIPTOR(1);
+DECLARE_UAC_AC_HEADER_DESCRIPTOR(2);
#define UAC_DT_AC_HEADER_LENGTH UAC_DT_AC_HEADER_SIZE(F_AUDIO_NUM_INTERFACES)
-/* 1 input terminal, 1 output terminal and 1 feature unit */
-#define UAC_DT_TOTAL_LENGTH (UAC_DT_AC_HEADER_LENGTH + UAC_DT_INPUT_TERMINAL_SIZE \
- + UAC_DT_OUTPUT_TERMINAL_SIZE + UAC_DT_FEATURE_UNIT_SIZE(0))
+/* 2 input terminals and 2 output terminals */
+#define UAC_DT_TOTAL_LENGTH (UAC_DT_AC_HEADER_LENGTH \
+ + 2*UAC_DT_INPUT_TERMINAL_SIZE + 2*UAC_DT_OUTPUT_TERMINAL_SIZE)
/* B.3.2 Class-Specific AC Interface Descriptor */
-static struct uac1_ac_header_descriptor_1 ac_header_desc = {
+static struct uac1_ac_header_descriptor_2 ac_header_desc = {
.bLength = UAC_DT_AC_HEADER_LENGTH,
.bDescriptorType = USB_DT_CS_INTERFACE,
.bDescriptorSubtype = UAC_HEADER,
- .bcdADC = __constant_cpu_to_le16(0x0100),
- .wTotalLength = __constant_cpu_to_le16(UAC_DT_TOTAL_LENGTH),
+ .bcdADC = cpu_to_le16(0x0100),
+ .wTotalLength = cpu_to_le16(UAC_DT_TOTAL_LENGTH),
.bInCollection = F_AUDIO_NUM_INTERFACES,
.baInterfaceNr = {
- /* Interface number of the first AudioStream interface */
+ /* Interface number of the AudioStream interfaces */
[0] = 1,
+ [1] = 2,
}
};
-#define INPUT_TERMINAL_ID 1
-static struct uac_input_terminal_descriptor input_terminal_desc = {
+#define USB_OUT_IT_ID 1
+static struct uac_input_terminal_descriptor usb_out_it_desc = {
.bLength = UAC_DT_INPUT_TERMINAL_SIZE,
.bDescriptorType = USB_DT_CS_INTERFACE,
.bDescriptorSubtype = UAC_INPUT_TERMINAL,
- .bTerminalID = INPUT_TERMINAL_ID,
+ .bTerminalID = USB_OUT_IT_ID,
.wTerminalType = UAC_TERMINAL_STREAMING,
.bAssocTerminal = 0,
.wChannelConfig = 0x3,
};
-DECLARE_UAC_FEATURE_UNIT_DESCRIPTOR(0);
-
-#define FEATURE_UNIT_ID 2
-static struct uac_feature_unit_descriptor_0 feature_unit_desc = {
- .bLength = UAC_DT_FEATURE_UNIT_SIZE(0),
+#define IO_OUT_OT_ID 2
+static struct uac1_output_terminal_descriptor io_out_ot_desc = {
+ .bLength = UAC_DT_OUTPUT_TERMINAL_SIZE,
.bDescriptorType = USB_DT_CS_INTERFACE,
- .bDescriptorSubtype = UAC_FEATURE_UNIT,
- .bUnitID = FEATURE_UNIT_ID,
- .bSourceID = INPUT_TERMINAL_ID,
- .bControlSize = 2,
- .bmaControls[0] = (UAC_FU_MUTE | UAC_FU_VOLUME),
+ .bDescriptorSubtype = UAC_OUTPUT_TERMINAL,
+ .bTerminalID = IO_OUT_OT_ID,
+ .wTerminalType = UAC_OUTPUT_TERMINAL_SPEAKER,
+ .bAssocTerminal = 0,
+ .bSourceID = USB_OUT_IT_ID,
};
-static struct usb_audio_control mute_control = {
- .list = LIST_HEAD_INIT(mute_control.list),
- .name = "Mute Control",
- .type = UAC_FU_MUTE,
- /* Todo: add real Mute control code */
- .set = generic_set_cmd,
- .get = generic_get_cmd,
+#define IO_IN_IT_ID 3
+static struct uac_input_terminal_descriptor io_in_it_desc = {
+ .bLength = UAC_DT_INPUT_TERMINAL_SIZE,
+ .bDescriptorType = USB_DT_CS_INTERFACE,
+ .bDescriptorSubtype = UAC_INPUT_TERMINAL,
+ .bTerminalID = IO_IN_IT_ID,
+ .wTerminalType = UAC_INPUT_TERMINAL_MICROPHONE,
+ .bAssocTerminal = 0,
+ .wChannelConfig = 0x3,
};
-static struct usb_audio_control volume_control = {
- .list = LIST_HEAD_INIT(volume_control.list),
- .name = "Volume Control",
- .type = UAC_FU_VOLUME,
- /* Todo: add real Volume control code */
- .set = generic_set_cmd,
- .get = generic_get_cmd,
+#define USB_IN_OT_ID 4
+static struct uac1_output_terminal_descriptor usb_in_ot_desc = {
+ .bLength = UAC_DT_OUTPUT_TERMINAL_SIZE,
+ .bDescriptorType = USB_DT_CS_INTERFACE,
+ .bDescriptorSubtype = UAC_OUTPUT_TERMINAL,
+ .bTerminalID = USB_IN_OT_ID,
+ .wTerminalType = UAC_TERMINAL_STREAMING,
+ .bAssocTerminal = 0,
+ .bSourceID = IO_IN_IT_ID,
};
-static struct usb_audio_control_selector feature_unit = {
- .list = LIST_HEAD_INIT(feature_unit.list),
- .id = FEATURE_UNIT_ID,
- .name = "Mute & Volume Control",
- .type = UAC_FEATURE_UNIT,
- .desc = (struct usb_descriptor_header *)&feature_unit_desc,
+/* B.4.1 Standard AS Interface Descriptor */
+static struct usb_interface_descriptor as_out_interface_alt_0_desc = {
+ .bLength = USB_DT_INTERFACE_SIZE,
+ .bDescriptorType = USB_DT_INTERFACE,
+ .bAlternateSetting = 0,
+ .bNumEndpoints = 0,
+ .bInterfaceClass = USB_CLASS_AUDIO,
+ .bInterfaceSubClass = USB_SUBCLASS_AUDIOSTREAMING,
};
-#define OUTPUT_TERMINAL_ID 3
-static struct uac1_output_terminal_descriptor output_terminal_desc = {
- .bLength = UAC_DT_OUTPUT_TERMINAL_SIZE,
- .bDescriptorType = USB_DT_CS_INTERFACE,
- .bDescriptorSubtype = UAC_OUTPUT_TERMINAL,
- .bTerminalID = OUTPUT_TERMINAL_ID,
- .wTerminalType = UAC_OUTPUT_TERMINAL_SPEAKER,
- .bAssocTerminal = FEATURE_UNIT_ID,
- .bSourceID = FEATURE_UNIT_ID,
+static struct usb_interface_descriptor as_out_interface_alt_1_desc = {
+ .bLength = USB_DT_INTERFACE_SIZE,
+ .bDescriptorType = USB_DT_INTERFACE,
+ .bAlternateSetting = 1,
+ .bNumEndpoints = 1,
+ .bInterfaceClass = USB_CLASS_AUDIO,
+ .bInterfaceSubClass = USB_SUBCLASS_AUDIOSTREAMING,
};
-/* B.4.1 Standard AS Interface Descriptor */
-static struct usb_interface_descriptor as_interface_alt_0_desc = {
+static struct usb_interface_descriptor as_in_interface_alt_0_desc = {
.bLength = USB_DT_INTERFACE_SIZE,
.bDescriptorType = USB_DT_INTERFACE,
.bAlternateSetting = 0,
.bInterfaceSubClass = USB_SUBCLASS_AUDIOSTREAMING,
};
-static struct usb_interface_descriptor as_interface_alt_1_desc = {
+static struct usb_interface_descriptor as_in_interface_alt_1_desc = {
.bLength = USB_DT_INTERFACE_SIZE,
.bDescriptorType = USB_DT_INTERFACE,
.bAlternateSetting = 1,
};
/* B.4.2 Class-Specific AS Interface Descriptor */
-static struct uac1_as_header_descriptor as_header_desc = {
+static struct uac1_as_header_descriptor as_out_header_desc = {
.bLength = UAC_DT_AS_HEADER_SIZE,
.bDescriptorType = USB_DT_CS_INTERFACE,
.bDescriptorSubtype = UAC_AS_GENERAL,
- .bTerminalLink = INPUT_TERMINAL_ID,
+ .bTerminalLink = USB_OUT_IT_ID,
+ .bDelay = 1,
+ .wFormatTag = UAC_FORMAT_TYPE_I_PCM,
+};
+
+static struct uac1_as_header_descriptor as_in_header_desc = {
+ .bLength = UAC_DT_AS_HEADER_SIZE,
+ .bDescriptorType = USB_DT_CS_INTERFACE,
+ .bDescriptorSubtype = UAC_AS_GENERAL,
+ .bTerminalLink = USB_IN_OT_ID,
.bDelay = 1,
.wFormatTag = UAC_FORMAT_TYPE_I_PCM,
};
DECLARE_UAC_FORMAT_TYPE_I_DISCRETE_DESC(1);
-static struct uac_format_type_i_discrete_descriptor_1 as_type_i_desc = {
+static struct uac_format_type_i_discrete_descriptor_1 as_out_type_i_desc = {
.bLength = UAC_FORMAT_TYPE_I_DISCRETE_DESC_SIZE(1),
.bDescriptorType = USB_DT_CS_INTERFACE,
.bDescriptorSubtype = UAC_FORMAT_TYPE,
.bLength = UAC_ISO_ENDPOINT_DESC_SIZE,
.bDescriptorType = USB_DT_CS_ENDPOINT,
.bDescriptorSubtype = UAC_EP_GENERAL,
- .bmAttributes = 1,
+ .bmAttributes = 1,
.bLockDelayUnits = 1,
- .wLockDelay = __constant_cpu_to_le16(1),
+ .wLockDelay = cpu_to_le16(1),
+};
+
+static struct uac_format_type_i_discrete_descriptor_1 as_in_type_i_desc = {
+ .bLength = UAC_FORMAT_TYPE_I_DISCRETE_DESC_SIZE(1),
+ .bDescriptorType = USB_DT_CS_INTERFACE,
+ .bDescriptorSubtype = UAC_FORMAT_TYPE,
+ .bFormatType = UAC_FORMAT_TYPE_I,
+ .bSubframeSize = 2,
+ .bBitResolution = 16,
+ .bSamFreqType = 1,
+};
+
+/* Standard ISO OUT Endpoint Descriptor */
+static struct usb_endpoint_descriptor as_in_ep_desc = {
+ .bLength = USB_DT_ENDPOINT_AUDIO_SIZE,
+ .bDescriptorType = USB_DT_ENDPOINT,
+ .bEndpointAddress = USB_DIR_IN,
+ .bmAttributes = USB_ENDPOINT_SYNC_ASYNC
+ | USB_ENDPOINT_XFER_ISOC,
+ .wMaxPacketSize = cpu_to_le16(UAC1_OUT_EP_MAX_PACKET_SIZE),
+ .bInterval = 4,
+};
+
+/* Class-specific AS ISO OUT Endpoint Descriptor */
+static struct uac_iso_endpoint_descriptor as_iso_in_desc = {
+ .bLength = UAC_ISO_ENDPOINT_DESC_SIZE,
+ .bDescriptorType = USB_DT_CS_ENDPOINT,
+ .bDescriptorSubtype = UAC_EP_GENERAL,
+ .bmAttributes = 1,
+ .bLockDelayUnits = 0,
+ .wLockDelay = 0,
};
static struct usb_descriptor_header *f_audio_desc[] = {
(struct usb_descriptor_header *)&ac_interface_desc,
(struct usb_descriptor_header *)&ac_header_desc,
- (struct usb_descriptor_header *)&input_terminal_desc,
- (struct usb_descriptor_header *)&output_terminal_desc,
- (struct usb_descriptor_header *)&feature_unit_desc,
+ (struct usb_descriptor_header *)&usb_out_it_desc,
+ (struct usb_descriptor_header *)&io_out_ot_desc,
+ (struct usb_descriptor_header *)&io_in_it_desc,
+ (struct usb_descriptor_header *)&usb_in_ot_desc,
- (struct usb_descriptor_header *)&as_interface_alt_0_desc,
- (struct usb_descriptor_header *)&as_interface_alt_1_desc,
- (struct usb_descriptor_header *)&as_header_desc,
+ (struct usb_descriptor_header *)&as_out_interface_alt_0_desc,
+ (struct usb_descriptor_header *)&as_out_interface_alt_1_desc,
+ (struct usb_descriptor_header *)&as_out_header_desc,
- (struct usb_descriptor_header *)&as_type_i_desc,
+ (struct usb_descriptor_header *)&as_out_type_i_desc,
(struct usb_descriptor_header *)&as_out_ep_desc,
(struct usb_descriptor_header *)&as_iso_out_desc,
+
+ (struct usb_descriptor_header *)&as_in_interface_alt_0_desc,
+ (struct usb_descriptor_header *)&as_in_interface_alt_1_desc,
+ (struct usb_descriptor_header *)&as_in_header_desc,
+
+ (struct usb_descriptor_header *)&as_in_type_i_desc,
+
+ (struct usb_descriptor_header *)&as_in_ep_desc,
+ (struct usb_descriptor_header *)&as_iso_in_desc,
NULL,
};
enum {
STR_AC_IF,
- STR_INPUT_TERMINAL,
- STR_INPUT_TERMINAL_CH_NAMES,
- STR_FEAT_DESC_0,
- STR_OUTPUT_TERMINAL,
- STR_AS_IF_ALT0,
- STR_AS_IF_ALT1,
+ STR_USB_OUT_IT,
+ STR_USB_OUT_IT_CH_NAMES,
+ STR_IO_OUT_OT,
+ STR_IO_IN_IT,
+ STR_IO_IN_IT_CH_NAMES,
+ STR_USB_IN_OT,
+ STR_AS_OUT_IF_ALT0,
+ STR_AS_OUT_IF_ALT1,
+ STR_AS_IN_IF_ALT0,
+ STR_AS_IN_IF_ALT1,
};
static struct usb_string strings_uac1[] = {
[STR_AC_IF].s = "AC Interface",
- [STR_INPUT_TERMINAL].s = "Input terminal",
- [STR_INPUT_TERMINAL_CH_NAMES].s = "Channels",
- [STR_FEAT_DESC_0].s = "Volume control & mute",
- [STR_OUTPUT_TERMINAL].s = "Output terminal",
- [STR_AS_IF_ALT0].s = "AS Interface",
- [STR_AS_IF_ALT1].s = "AS Interface",
+ [STR_USB_OUT_IT].s = "Playback Input terminal",
+ [STR_USB_OUT_IT_CH_NAMES].s = "Playback Channels",
+ [STR_IO_OUT_OT].s = "Playback Output terminal",
+ [STR_IO_IN_IT].s = "Capture Input terminal",
+ [STR_IO_IN_IT_CH_NAMES].s = "Capture Channels",
+ [STR_USB_IN_OT].s = "Capture Output terminal",
+ [STR_AS_OUT_IF_ALT0].s = "Playback Inactive",
+ [STR_AS_OUT_IF_ALT1].s = "Playback Active",
+ [STR_AS_IN_IF_ALT0].s = "Capture Inactive",
+ [STR_AS_IN_IF_ALT1].s = "Capture Active",
{ },
};
* This function is an ALSA sound card following USB Audio Class Spec 1.0.
*/
-/*-------------------------------------------------------------------------*/
-struct f_audio_buf {
- u8 *buf;
- int actual;
- struct list_head list;
-};
-
-static struct f_audio_buf *f_audio_buffer_alloc(int buf_size)
-{
- struct f_audio_buf *copy_buf;
-
- copy_buf = kzalloc(sizeof *copy_buf, GFP_ATOMIC);
- if (!copy_buf)
- return ERR_PTR(-ENOMEM);
-
- copy_buf->buf = kzalloc(buf_size, GFP_ATOMIC);
- if (!copy_buf->buf) {
- kfree(copy_buf);
- return ERR_PTR(-ENOMEM);
- }
-
- return copy_buf;
-}
-
-static void f_audio_buffer_free(struct f_audio_buf *audio_buf)
-{
- kfree(audio_buf->buf);
- kfree(audio_buf);
-}
-/*-------------------------------------------------------------------------*/
-
-struct f_audio {
- struct gaudio card;
-
- /* endpoints handle full and/or high speeds */
- struct usb_ep *out_ep;
-
- spinlock_t lock;
- struct f_audio_buf *copy_buf;
- struct work_struct playback_work;
- struct list_head play_queue;
-
- /* Control Set command */
- struct list_head cs;
- u8 set_cmd;
- struct usb_audio_control *set_con;
-};
-
-static inline struct f_audio *func_to_audio(struct usb_function *f)
-{
- return container_of(f, struct f_audio, card.func);
-}
-
-/*-------------------------------------------------------------------------*/
-
-static void f_audio_playback_work(struct work_struct *data)
-{
- struct f_audio *audio = container_of(data, struct f_audio,
- playback_work);
- struct f_audio_buf *play_buf;
-
- spin_lock_irq(&audio->lock);
- if (list_empty(&audio->play_queue)) {
- spin_unlock_irq(&audio->lock);
- return;
- }
- play_buf = list_first_entry(&audio->play_queue,
- struct f_audio_buf, list);
- list_del(&play_buf->list);
- spin_unlock_irq(&audio->lock);
-
- u_audio_playback(&audio->card, play_buf->buf, play_buf->actual);
- f_audio_buffer_free(play_buf);
-}
-
-static int f_audio_out_ep_complete(struct usb_ep *ep, struct usb_request *req)
-{
- struct f_audio *audio = req->context;
- struct usb_composite_dev *cdev = audio->card.func.config->cdev;
- struct f_audio_buf *copy_buf = audio->copy_buf;
- struct f_uac1_opts *opts;
- int audio_buf_size;
- int err;
-
- opts = container_of(audio->card.func.fi, struct f_uac1_opts,
- func_inst);
- audio_buf_size = opts->audio_buf_size;
-
- if (!copy_buf)
- return -EINVAL;
-
- /* Copy buffer is full, add it to the play_queue */
- if (audio_buf_size - copy_buf->actual < req->actual) {
- list_add_tail(©_buf->list, &audio->play_queue);
- schedule_work(&audio->playback_work);
- copy_buf = f_audio_buffer_alloc(audio_buf_size);
- if (IS_ERR(copy_buf))
- return -ENOMEM;
- }
-
- memcpy(copy_buf->buf + copy_buf->actual, req->buf, req->actual);
- copy_buf->actual += req->actual;
- audio->copy_buf = copy_buf;
-
- err = usb_ep_queue(ep, req, GFP_ATOMIC);
- if (err)
- ERROR(cdev, "%s queue req: %d\n", ep->name, err);
-
- return 0;
-
-}
-
-static void f_audio_complete(struct usb_ep *ep, struct usb_request *req)
-{
- struct f_audio *audio = req->context;
- int status = req->status;
- u32 data = 0;
- struct usb_ep *out_ep = audio->out_ep;
-
- switch (status) {
-
- case 0: /* normal completion? */
- if (ep == out_ep)
- f_audio_out_ep_complete(ep, req);
- else if (audio->set_con) {
- memcpy(&data, req->buf, req->length);
- audio->set_con->set(audio->set_con, audio->set_cmd,
- le16_to_cpu(data));
- audio->set_con = NULL;
- }
- break;
- default:
- break;
- }
-}
-
-static int audio_set_intf_req(struct usb_function *f,
- const struct usb_ctrlrequest *ctrl)
-{
- struct f_audio *audio = func_to_audio(f);
- struct usb_composite_dev *cdev = f->config->cdev;
- struct usb_request *req = cdev->req;
- u8 id = ((le16_to_cpu(ctrl->wIndex) >> 8) & 0xFF);
- u16 len = le16_to_cpu(ctrl->wLength);
- u16 w_value = le16_to_cpu(ctrl->wValue);
- u8 con_sel = (w_value >> 8) & 0xFF;
- u8 cmd = (ctrl->bRequest & 0x0F);
- struct usb_audio_control_selector *cs;
- struct usb_audio_control *con;
-
- DBG(cdev, "bRequest 0x%x, w_value 0x%04x, len %d, entity %d\n",
- ctrl->bRequest, w_value, len, id);
-
- list_for_each_entry(cs, &audio->cs, list) {
- if (cs->id == id) {
- list_for_each_entry(con, &cs->control, list) {
- if (con->type == con_sel) {
- audio->set_con = con;
- break;
- }
- }
- break;
- }
- }
-
- audio->set_cmd = cmd;
- req->context = audio;
- req->complete = f_audio_complete;
-
- return len;
-}
-
-static int audio_get_intf_req(struct usb_function *f,
- const struct usb_ctrlrequest *ctrl)
-{
- struct f_audio *audio = func_to_audio(f);
- struct usb_composite_dev *cdev = f->config->cdev;
- struct usb_request *req = cdev->req;
- int value = -EOPNOTSUPP;
- u8 id = ((le16_to_cpu(ctrl->wIndex) >> 8) & 0xFF);
- u16 len = le16_to_cpu(ctrl->wLength);
- u16 w_value = le16_to_cpu(ctrl->wValue);
- u8 con_sel = (w_value >> 8) & 0xFF;
- u8 cmd = (ctrl->bRequest & 0x0F);
- struct usb_audio_control_selector *cs;
- struct usb_audio_control *con;
-
- DBG(cdev, "bRequest 0x%x, w_value 0x%04x, len %d, entity %d\n",
- ctrl->bRequest, w_value, len, id);
-
- list_for_each_entry(cs, &audio->cs, list) {
- if (cs->id == id) {
- list_for_each_entry(con, &cs->control, list) {
- if (con->type == con_sel && con->get) {
- value = con->get(con, cmd);
- break;
- }
- }
- break;
- }
- }
-
- req->context = audio;
- req->complete = f_audio_complete;
- len = min_t(size_t, sizeof(value), len);
- memcpy(req->buf, &value, len);
-
- return len;
-}
-
static int audio_set_endpoint_req(struct usb_function *f,
const struct usb_ctrlrequest *ctrl)
{
* activation uses set_alt().
*/
switch (ctrl->bRequestType) {
- case USB_DIR_OUT | USB_TYPE_CLASS | USB_RECIP_INTERFACE:
- value = audio_set_intf_req(f, ctrl);
- break;
-
- case USB_DIR_IN | USB_TYPE_CLASS | USB_RECIP_INTERFACE:
- value = audio_get_intf_req(f, ctrl);
- break;
-
case USB_DIR_OUT | USB_TYPE_CLASS | USB_RECIP_ENDPOINT:
value = audio_set_endpoint_req(f, ctrl);
break;
static int f_audio_set_alt(struct usb_function *f, unsigned intf, unsigned alt)
{
- struct f_audio *audio = func_to_audio(f);
struct usb_composite_dev *cdev = f->config->cdev;
- struct usb_ep *out_ep = audio->out_ep;
- struct usb_request *req;
- struct f_uac1_opts *opts;
- int req_buf_size, req_count, audio_buf_size;
- int i = 0, err = 0;
-
- DBG(cdev, "intf %d, alt %d\n", intf, alt);
+ struct usb_gadget *gadget = cdev->gadget;
+ struct device *dev = &gadget->dev;
+ struct f_uac1 *uac1 = func_to_uac1(f);
+ int ret = 0;
+
+ /* No i/f has more than 2 alt settings */
+ if (alt > 1) {
+ dev_err(dev, "%s:%d Error!\n", __func__, __LINE__);
+ return -EINVAL;
+ }
- opts = container_of(f->fi, struct f_uac1_opts, func_inst);
- req_buf_size = opts->req_buf_size;
- req_count = opts->req_count;
- audio_buf_size = opts->audio_buf_size;
-
- if (intf == 1) {
- if (alt == 1) {
- err = config_ep_by_speed(cdev->gadget, f, out_ep);
- if (err)
- return err;
-
- usb_ep_enable(out_ep);
- audio->copy_buf = f_audio_buffer_alloc(audio_buf_size);
- if (IS_ERR(audio->copy_buf))
- return -ENOMEM;
-
- /*
- * allocate a bunch of read buffers
- * and queue them all at once.
- */
- for (i = 0; i < req_count && err == 0; i++) {
- req = usb_ep_alloc_request(out_ep, GFP_ATOMIC);
- if (req) {
- req->buf = kzalloc(req_buf_size,
- GFP_ATOMIC);
- if (req->buf) {
- req->length = req_buf_size;
- req->context = audio;
- req->complete =
- f_audio_complete;
- err = usb_ep_queue(out_ep,
- req, GFP_ATOMIC);
- if (err)
- ERROR(cdev,
- "%s queue req: %d\n",
- out_ep->name, err);
- } else
- err = -ENOMEM;
- } else
- err = -ENOMEM;
- }
-
- } else {
- struct f_audio_buf *copy_buf = audio->copy_buf;
- if (copy_buf) {
- list_add_tail(©_buf->list,
- &audio->play_queue);
- schedule_work(&audio->playback_work);
- }
+ if (intf == uac1->ac_intf) {
+ /* Control I/f has only 1 AltSetting - 0 */
+ if (alt) {
+ dev_err(dev, "%s:%d Error!\n", __func__, __LINE__);
+ return -EINVAL;
}
+ return 0;
}
- return err;
+ if (intf == uac1->as_out_intf) {
+ uac1->as_out_alt = alt;
+
+ if (alt)
+ ret = u_audio_start_capture(&uac1->g_audio);
+ else
+ u_audio_stop_capture(&uac1->g_audio);
+ } else if (intf == uac1->as_in_intf) {
+ uac1->as_in_alt = alt;
+
+ if (alt)
+ ret = u_audio_start_playback(&uac1->g_audio);
+ else
+ u_audio_stop_playback(&uac1->g_audio);
+ } else {
+ dev_err(dev, "%s:%d Error!\n", __func__, __LINE__);
+ return -EINVAL;
+ }
+
+ return ret;
}
-static void f_audio_disable(struct usb_function *f)
+static int f_audio_get_alt(struct usb_function *f, unsigned intf)
{
- return;
+ struct usb_composite_dev *cdev = f->config->cdev;
+ struct usb_gadget *gadget = cdev->gadget;
+ struct device *dev = &gadget->dev;
+ struct f_uac1 *uac1 = func_to_uac1(f);
+
+ if (intf == uac1->ac_intf)
+ return uac1->ac_alt;
+ else if (intf == uac1->as_out_intf)
+ return uac1->as_out_alt;
+ else if (intf == uac1->as_in_intf)
+ return uac1->as_in_alt;
+ else
+ dev_err(dev, "%s:%d Invalid Interface %d!\n",
+ __func__, __LINE__, intf);
+
+ return -EINVAL;
}
-/*-------------------------------------------------------------------------*/
-static void f_audio_build_desc(struct f_audio *audio)
+static void f_audio_disable(struct usb_function *f)
{
- struct gaudio *card = &audio->card;
- u8 *sam_freq;
- int rate;
+ struct f_uac1 *uac1 = func_to_uac1(f);
- /* Set channel numbers */
- input_terminal_desc.bNrChannels = u_audio_get_playback_channels(card);
- as_type_i_desc.bNrChannels = u_audio_get_playback_channels(card);
+ uac1->as_out_alt = 0;
+ uac1->as_in_alt = 0;
- /* Set sample rates */
- rate = u_audio_get_playback_rate(card);
- sam_freq = as_type_i_desc.tSamFreq[0];
- memcpy(sam_freq, &rate, 3);
-
- /* Todo: Set Sample bits and other parameters */
-
- return;
+ u_audio_stop_capture(&uac1->g_audio);
}
+/*-------------------------------------------------------------------------*/
+
/* audio function driver setup/binding */
-static int
-f_audio_bind(struct usb_configuration *c, struct usb_function *f)
+static int f_audio_bind(struct usb_configuration *c, struct usb_function *f)
{
- struct usb_composite_dev *cdev = c->cdev;
- struct f_audio *audio = func_to_audio(f);
- struct usb_string *us;
- int status;
- struct usb_ep *ep = NULL;
- struct f_uac1_opts *audio_opts;
+ struct usb_composite_dev *cdev = c->cdev;
+ struct usb_gadget *gadget = cdev->gadget;
+ struct f_uac1 *uac1 = func_to_uac1(f);
+ struct g_audio *audio = func_to_g_audio(f);
+ struct f_uac1_opts *audio_opts;
+ struct usb_ep *ep = NULL;
+ struct usb_string *us;
+ u8 *sam_freq;
+ int rate;
+ int status;
audio_opts = container_of(f->fi, struct f_uac1_opts, func_inst);
- audio->card.gadget = c->cdev->gadget;
- /* set up ASLA audio devices */
- if (!audio_opts->bound) {
- status = gaudio_setup(&audio->card);
- if (status < 0)
- return status;
- audio_opts->bound = true;
- }
+
us = usb_gstrings_attach(cdev, uac1_strings, ARRAY_SIZE(strings_uac1));
if (IS_ERR(us))
return PTR_ERR(us);
ac_interface_desc.iInterface = us[STR_AC_IF].id;
- input_terminal_desc.iTerminal = us[STR_INPUT_TERMINAL].id;
- input_terminal_desc.iChannelNames = us[STR_INPUT_TERMINAL_CH_NAMES].id;
- feature_unit_desc.iFeature = us[STR_FEAT_DESC_0].id;
- output_terminal_desc.iTerminal = us[STR_OUTPUT_TERMINAL].id;
- as_interface_alt_0_desc.iInterface = us[STR_AS_IF_ALT0].id;
- as_interface_alt_1_desc.iInterface = us[STR_AS_IF_ALT1].id;
+ usb_out_it_desc.iTerminal = us[STR_USB_OUT_IT].id;
+ usb_out_it_desc.iChannelNames = us[STR_USB_OUT_IT_CH_NAMES].id;
+ io_out_ot_desc.iTerminal = us[STR_IO_OUT_OT].id;
+ as_out_interface_alt_0_desc.iInterface = us[STR_AS_OUT_IF_ALT0].id;
+ as_out_interface_alt_1_desc.iInterface = us[STR_AS_OUT_IF_ALT1].id;
+ io_in_it_desc.iTerminal = us[STR_IO_IN_IT].id;
+ io_in_it_desc.iChannelNames = us[STR_IO_IN_IT_CH_NAMES].id;
+ usb_in_ot_desc.iTerminal = us[STR_USB_IN_OT].id;
+ as_in_interface_alt_0_desc.iInterface = us[STR_AS_IN_IF_ALT0].id;
+ as_in_interface_alt_1_desc.iInterface = us[STR_AS_IN_IF_ALT1].id;
+ /* Set channel numbers */
+ usb_out_it_desc.bNrChannels = num_channels(audio_opts->c_chmask);
+ usb_out_it_desc.wChannelConfig = cpu_to_le16(audio_opts->c_chmask);
+ as_out_type_i_desc.bNrChannels = num_channels(audio_opts->c_chmask);
+ as_out_type_i_desc.bSubframeSize = audio_opts->c_ssize;
+ as_out_type_i_desc.bBitResolution = audio_opts->c_ssize * 8;
+ io_in_it_desc.bNrChannels = num_channels(audio_opts->p_chmask);
+ io_in_it_desc.wChannelConfig = cpu_to_le16(audio_opts->p_chmask);
+ as_in_type_i_desc.bNrChannels = num_channels(audio_opts->p_chmask);
+ as_in_type_i_desc.bSubframeSize = audio_opts->p_ssize;
+ as_in_type_i_desc.bBitResolution = audio_opts->p_ssize * 8;
- f_audio_build_desc(audio);
+ /* Set sample rates */
+ rate = audio_opts->c_srate;
+ sam_freq = as_out_type_i_desc.tSamFreq[0];
+ memcpy(sam_freq, &rate, 3);
+ rate = audio_opts->p_srate;
+ sam_freq = as_in_type_i_desc.tSamFreq[0];
+ memcpy(sam_freq, &rate, 3);
/* allocate instance-specific interface IDs, and patch descriptors */
status = usb_interface_id(c, f);
if (status < 0)
goto fail;
ac_interface_desc.bInterfaceNumber = status;
+ uac1->ac_intf = status;
+ uac1->ac_alt = 0;
status = usb_interface_id(c, f);
if (status < 0)
goto fail;
- as_interface_alt_0_desc.bInterfaceNumber = status;
- as_interface_alt_1_desc.bInterfaceNumber = status;
+ as_out_interface_alt_0_desc.bInterfaceNumber = status;
+ as_out_interface_alt_1_desc.bInterfaceNumber = status;
+ uac1->as_out_intf = status;
+ uac1->as_out_alt = 0;
+
+ status = usb_interface_id(c, f);
+ if (status < 0)
+ goto fail;
+ as_in_interface_alt_0_desc.bInterfaceNumber = status;
+ as_in_interface_alt_1_desc.bInterfaceNumber = status;
+ uac1->as_in_intf = status;
+ uac1->as_in_alt = 0;
+
+ audio->gadget = gadget;
status = -ENODEV;
audio->out_ep = ep;
audio->out_ep->desc = &as_out_ep_desc;
- status = -ENOMEM;
+ ep = usb_ep_autoconfig(cdev->gadget, &as_in_ep_desc);
+ if (!ep)
+ goto fail;
+ audio->in_ep = ep;
+ audio->in_ep->desc = &as_in_ep_desc;
/* copy descriptors, and track endpoint copies */
status = usb_assign_descriptors(f, f_audio_desc, f_audio_desc, NULL,
NULL);
if (status)
goto fail;
+
+ audio->out_ep_maxpsize = as_out_ep_desc.wMaxPacketSize;
+ audio->in_ep_maxpsize = as_in_ep_desc.wMaxPacketSize;
+ audio->params.c_chmask = audio_opts->c_chmask;
+ audio->params.c_srate = audio_opts->c_srate;
+ audio->params.c_ssize = audio_opts->c_ssize;
+ audio->params.p_chmask = audio_opts->p_chmask;
+ audio->params.p_srate = audio_opts->p_srate;
+ audio->params.p_ssize = audio_opts->p_ssize;
+ audio->params.req_number = audio_opts->req_number;
+
+ status = g_audio_setup(audio, "UAC1_PCM", "UAC1_Gadget");
+ if (status)
+ goto err_card_register;
+
return 0;
+err_card_register:
+ usb_free_all_descriptors(f);
fail:
- gaudio_cleanup(&audio->card);
return status;
}
/*-------------------------------------------------------------------------*/
-static int generic_set_cmd(struct usb_audio_control *con, u8 cmd, int value)
-{
- con->data[cmd] = value;
-
- return 0;
-}
-
-static int generic_get_cmd(struct usb_audio_control *con, u8 cmd)
-{
- return con->data[cmd];
-}
-
-/* Todo: add more control selecotor dynamically */
-static int control_selector_init(struct f_audio *audio)
-{
- INIT_LIST_HEAD(&audio->cs);
- list_add(&feature_unit.list, &audio->cs);
-
- INIT_LIST_HEAD(&feature_unit.control);
- list_add(&mute_control.list, &feature_unit.control);
- list_add(&volume_control.list, &feature_unit.control);
-
- volume_control.data[UAC__CUR] = 0xffc0;
- volume_control.data[UAC__MIN] = 0xe3a0;
- volume_control.data[UAC__MAX] = 0xfff0;
- volume_control.data[UAC__RES] = 0x0030;
-
- return 0;
-}
-
static inline struct f_uac1_opts *to_f_uac1_opts(struct config_item *item)
{
return container_of(to_config_group(item), struct f_uac1_opts,
.release = f_uac1_attr_release,
};
-#define UAC1_INT_ATTRIBUTE(name) \
-static ssize_t f_uac1_opts_##name##_show(struct config_item *item, \
- char *page) \
+#define UAC1_ATTRIBUTE(name) \
+static ssize_t f_uac1_opts_##name##_show( \
+ struct config_item *item, \
+ char *page) \
{ \
struct f_uac1_opts *opts = to_f_uac1_opts(item); \
int result; \
return result; \
} \
\
-static ssize_t f_uac1_opts_##name##_store(struct config_item *item, \
+static ssize_t f_uac1_opts_##name##_store( \
+ struct config_item *item, \
const char *page, size_t len) \
{ \
struct f_uac1_opts *opts = to_f_uac1_opts(item); \
\
CONFIGFS_ATTR(f_uac1_opts_, name)
-UAC1_INT_ATTRIBUTE(req_buf_size);
-UAC1_INT_ATTRIBUTE(req_count);
-UAC1_INT_ATTRIBUTE(audio_buf_size);
-
-#define UAC1_STR_ATTRIBUTE(name) \
-static ssize_t f_uac1_opts_##name##_show(struct config_item *item, \
- char *page) \
-{ \
- struct f_uac1_opts *opts = to_f_uac1_opts(item); \
- int result; \
- \
- mutex_lock(&opts->lock); \
- result = sprintf(page, "%s\n", opts->name); \
- mutex_unlock(&opts->lock); \
- \
- return result; \
-} \
- \
-static ssize_t f_uac1_opts_##name##_store(struct config_item *item, \
- const char *page, size_t len) \
-{ \
- struct f_uac1_opts *opts = to_f_uac1_opts(item); \
- int ret = -EBUSY; \
- char *tmp; \
- \
- mutex_lock(&opts->lock); \
- if (opts->refcnt) \
- goto end; \
- \
- tmp = kstrndup(page, len, GFP_KERNEL); \
- if (tmp) { \
- ret = -ENOMEM; \
- goto end; \
- } \
- if (opts->name##_alloc) \
- kfree(opts->name); \
- opts->name##_alloc = true; \
- opts->name = tmp; \
- ret = len; \
- \
-end: \
- mutex_unlock(&opts->lock); \
- return ret; \
-} \
- \
-CONFIGFS_ATTR(f_uac1_opts_, name)
-
-UAC1_STR_ATTRIBUTE(fn_play);
-UAC1_STR_ATTRIBUTE(fn_cap);
-UAC1_STR_ATTRIBUTE(fn_cntl);
+UAC1_ATTRIBUTE(c_chmask);
+UAC1_ATTRIBUTE(c_srate);
+UAC1_ATTRIBUTE(c_ssize);
+UAC1_ATTRIBUTE(p_chmask);
+UAC1_ATTRIBUTE(p_srate);
+UAC1_ATTRIBUTE(p_ssize);
+UAC1_ATTRIBUTE(req_number);
static struct configfs_attribute *f_uac1_attrs[] = {
- &f_uac1_opts_attr_req_buf_size,
- &f_uac1_opts_attr_req_count,
- &f_uac1_opts_attr_audio_buf_size,
- &f_uac1_opts_attr_fn_play,
- &f_uac1_opts_attr_fn_cap,
- &f_uac1_opts_attr_fn_cntl,
+ &f_uac1_opts_attr_c_chmask,
+ &f_uac1_opts_attr_c_srate,
+ &f_uac1_opts_attr_c_ssize,
+ &f_uac1_opts_attr_p_chmask,
+ &f_uac1_opts_attr_p_srate,
+ &f_uac1_opts_attr_p_ssize,
+ &f_uac1_opts_attr_req_number,
NULL,
};
struct f_uac1_opts *opts;
opts = container_of(f, struct f_uac1_opts, func_inst);
- if (opts->fn_play_alloc)
- kfree(opts->fn_play);
- if (opts->fn_cap_alloc)
- kfree(opts->fn_cap);
- if (opts->fn_cntl_alloc)
- kfree(opts->fn_cntl);
kfree(opts);
}
config_group_init_type_name(&opts->func_inst.group, "",
&f_uac1_func_type);
- opts->req_buf_size = UAC1_OUT_EP_MAX_PACKET_SIZE;
- opts->req_count = UAC1_REQ_COUNT;
- opts->audio_buf_size = UAC1_AUDIO_BUF_SIZE;
- opts->fn_play = FILE_PCM_PLAYBACK;
- opts->fn_cap = FILE_PCM_CAPTURE;
- opts->fn_cntl = FILE_CONTROL;
+ opts->c_chmask = UAC1_DEF_CCHMASK;
+ opts->c_srate = UAC1_DEF_CSRATE;
+ opts->c_ssize = UAC1_DEF_CSSIZE;
+ opts->p_chmask = UAC1_DEF_PCHMASK;
+ opts->p_srate = UAC1_DEF_PSRATE;
+ opts->p_ssize = UAC1_DEF_PSSIZE;
+ opts->req_number = UAC1_DEF_REQ_NUM;
return &opts->func_inst;
}
static void f_audio_free(struct usb_function *f)
{
- struct f_audio *audio = func_to_audio(f);
+ struct g_audio *audio;
struct f_uac1_opts *opts;
- gaudio_cleanup(&audio->card);
+ audio = func_to_g_audio(f);
opts = container_of(f->fi, struct f_uac1_opts, func_inst);
kfree(audio);
mutex_lock(&opts->lock);
static void f_audio_unbind(struct usb_configuration *c, struct usb_function *f)
{
+ struct g_audio *audio = func_to_g_audio(f);
+
+ g_audio_cleanup(audio);
usb_free_all_descriptors(f);
+
+ audio->gadget = NULL;
}
static struct usb_function *f_audio_alloc(struct usb_function_instance *fi)
{
- struct f_audio *audio;
+ struct f_uac1 *uac1;
struct f_uac1_opts *opts;
/* allocate and initialize one new instance */
- audio = kzalloc(sizeof(*audio), GFP_KERNEL);
- if (!audio)
+ uac1 = kzalloc(sizeof(*uac1), GFP_KERNEL);
+ if (!uac1)
return ERR_PTR(-ENOMEM);
- audio->card.func.name = "g_audio";
-
opts = container_of(fi, struct f_uac1_opts, func_inst);
mutex_lock(&opts->lock);
++opts->refcnt;
mutex_unlock(&opts->lock);
- INIT_LIST_HEAD(&audio->play_queue);
- spin_lock_init(&audio->lock);
-
- audio->card.func.bind = f_audio_bind;
- audio->card.func.unbind = f_audio_unbind;
- audio->card.func.set_alt = f_audio_set_alt;
- audio->card.func.setup = f_audio_setup;
- audio->card.func.disable = f_audio_disable;
- audio->card.func.free_func = f_audio_free;
-
- control_selector_init(audio);
- INIT_WORK(&audio->playback_work, f_audio_playback_work);
+ uac1->g_audio.func.name = "uac1_func";
+ uac1->g_audio.func.bind = f_audio_bind;
+ uac1->g_audio.func.unbind = f_audio_unbind;
+ uac1->g_audio.func.set_alt = f_audio_set_alt;
+ uac1->g_audio.func.get_alt = f_audio_get_alt;
+ uac1->g_audio.func.setup = f_audio_setup;
+ uac1->g_audio.func.disable = f_audio_disable;
+ uac1->g_audio.func.free_func = f_audio_free;
- return &audio->card.func;
+ return &uac1->g_audio.func;
}
DECLARE_USB_FUNCTION_INIT(uac1, f_audio_alloc_inst, f_audio_alloc);
MODULE_LICENSE("GPL");
-MODULE_AUTHOR("Bryan Wu");
+MODULE_AUTHOR("Ruslan Bilovol");
--- /dev/null
+/*
+ * f_audio.c -- USB Audio class function driver
+ *
+ * Copyright (C) 2008 Bryan Wu <cooloney@kernel.org>
+ * Copyright (C) 2008 Analog Devices, Inc
+ *
+ * Enter bugs at http://blackfin.uclinux.org/
+ *
+ * Licensed under the GPL-2 or later.
+ */
+
+#include <linux/slab.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/device.h>
+#include <linux/atomic.h>
+
+#include "u_uac1_legacy.h"
+
+static int generic_set_cmd(struct usb_audio_control *con, u8 cmd, int value);
+static int generic_get_cmd(struct usb_audio_control *con, u8 cmd);
+
+/*
+ * DESCRIPTORS ... most are static, but strings and full
+ * configuration descriptors are built on demand.
+ */
+
+/*
+ * We have two interfaces- AudioControl and AudioStreaming
+ * TODO: only supcard playback currently
+ */
+#define F_AUDIO_AC_INTERFACE 0
+#define F_AUDIO_AS_INTERFACE 1
+#define F_AUDIO_NUM_INTERFACES 1
+
+/* B.3.1 Standard AC Interface Descriptor */
+static struct usb_interface_descriptor ac_interface_desc = {
+ .bLength = USB_DT_INTERFACE_SIZE,
+ .bDescriptorType = USB_DT_INTERFACE,
+ .bNumEndpoints = 0,
+ .bInterfaceClass = USB_CLASS_AUDIO,
+ .bInterfaceSubClass = USB_SUBCLASS_AUDIOCONTROL,
+};
+
+/*
+ * The number of AudioStreaming and MIDIStreaming interfaces
+ * in the Audio Interface Collection
+ */
+DECLARE_UAC_AC_HEADER_DESCRIPTOR(1);
+
+#define UAC_DT_AC_HEADER_LENGTH UAC_DT_AC_HEADER_SIZE(F_AUDIO_NUM_INTERFACES)
+/* 1 input terminal, 1 output terminal and 1 feature unit */
+#define UAC_DT_TOTAL_LENGTH (UAC_DT_AC_HEADER_LENGTH + UAC_DT_INPUT_TERMINAL_SIZE \
+ + UAC_DT_OUTPUT_TERMINAL_SIZE + UAC_DT_FEATURE_UNIT_SIZE(0))
+/* B.3.2 Class-Specific AC Interface Descriptor */
+static struct uac1_ac_header_descriptor_1 ac_header_desc = {
+ .bLength = UAC_DT_AC_HEADER_LENGTH,
+ .bDescriptorType = USB_DT_CS_INTERFACE,
+ .bDescriptorSubtype = UAC_HEADER,
+ .bcdADC = __constant_cpu_to_le16(0x0100),
+ .wTotalLength = __constant_cpu_to_le16(UAC_DT_TOTAL_LENGTH),
+ .bInCollection = F_AUDIO_NUM_INTERFACES,
+ .baInterfaceNr = {
+ /* Interface number of the first AudioStream interface */
+ [0] = 1,
+ }
+};
+
+#define INPUT_TERMINAL_ID 1
+static struct uac_input_terminal_descriptor input_terminal_desc = {
+ .bLength = UAC_DT_INPUT_TERMINAL_SIZE,
+ .bDescriptorType = USB_DT_CS_INTERFACE,
+ .bDescriptorSubtype = UAC_INPUT_TERMINAL,
+ .bTerminalID = INPUT_TERMINAL_ID,
+ .wTerminalType = UAC_TERMINAL_STREAMING,
+ .bAssocTerminal = 0,
+ .wChannelConfig = 0x3,
+};
+
+DECLARE_UAC_FEATURE_UNIT_DESCRIPTOR(0);
+
+#define FEATURE_UNIT_ID 2
+static struct uac_feature_unit_descriptor_0 feature_unit_desc = {
+ .bLength = UAC_DT_FEATURE_UNIT_SIZE(0),
+ .bDescriptorType = USB_DT_CS_INTERFACE,
+ .bDescriptorSubtype = UAC_FEATURE_UNIT,
+ .bUnitID = FEATURE_UNIT_ID,
+ .bSourceID = INPUT_TERMINAL_ID,
+ .bControlSize = 2,
+ .bmaControls[0] = (UAC_FU_MUTE | UAC_FU_VOLUME),
+};
+
+static struct usb_audio_control mute_control = {
+ .list = LIST_HEAD_INIT(mute_control.list),
+ .name = "Mute Control",
+ .type = UAC_FU_MUTE,
+ /* Todo: add real Mute control code */
+ .set = generic_set_cmd,
+ .get = generic_get_cmd,
+};
+
+static struct usb_audio_control volume_control = {
+ .list = LIST_HEAD_INIT(volume_control.list),
+ .name = "Volume Control",
+ .type = UAC_FU_VOLUME,
+ /* Todo: add real Volume control code */
+ .set = generic_set_cmd,
+ .get = generic_get_cmd,
+};
+
+static struct usb_audio_control_selector feature_unit = {
+ .list = LIST_HEAD_INIT(feature_unit.list),
+ .id = FEATURE_UNIT_ID,
+ .name = "Mute & Volume Control",
+ .type = UAC_FEATURE_UNIT,
+ .desc = (struct usb_descriptor_header *)&feature_unit_desc,
+};
+
+#define OUTPUT_TERMINAL_ID 3
+static struct uac1_output_terminal_descriptor output_terminal_desc = {
+ .bLength = UAC_DT_OUTPUT_TERMINAL_SIZE,
+ .bDescriptorType = USB_DT_CS_INTERFACE,
+ .bDescriptorSubtype = UAC_OUTPUT_TERMINAL,
+ .bTerminalID = OUTPUT_TERMINAL_ID,
+ .wTerminalType = UAC_OUTPUT_TERMINAL_SPEAKER,
+ .bAssocTerminal = FEATURE_UNIT_ID,
+ .bSourceID = FEATURE_UNIT_ID,
+};
+
+/* B.4.1 Standard AS Interface Descriptor */
+static struct usb_interface_descriptor as_interface_alt_0_desc = {
+ .bLength = USB_DT_INTERFACE_SIZE,
+ .bDescriptorType = USB_DT_INTERFACE,
+ .bAlternateSetting = 0,
+ .bNumEndpoints = 0,
+ .bInterfaceClass = USB_CLASS_AUDIO,
+ .bInterfaceSubClass = USB_SUBCLASS_AUDIOSTREAMING,
+};
+
+static struct usb_interface_descriptor as_interface_alt_1_desc = {
+ .bLength = USB_DT_INTERFACE_SIZE,
+ .bDescriptorType = USB_DT_INTERFACE,
+ .bAlternateSetting = 1,
+ .bNumEndpoints = 1,
+ .bInterfaceClass = USB_CLASS_AUDIO,
+ .bInterfaceSubClass = USB_SUBCLASS_AUDIOSTREAMING,
+};
+
+/* B.4.2 Class-Specific AS Interface Descriptor */
+static struct uac1_as_header_descriptor as_header_desc = {
+ .bLength = UAC_DT_AS_HEADER_SIZE,
+ .bDescriptorType = USB_DT_CS_INTERFACE,
+ .bDescriptorSubtype = UAC_AS_GENERAL,
+ .bTerminalLink = INPUT_TERMINAL_ID,
+ .bDelay = 1,
+ .wFormatTag = UAC_FORMAT_TYPE_I_PCM,
+};
+
+DECLARE_UAC_FORMAT_TYPE_I_DISCRETE_DESC(1);
+
+static struct uac_format_type_i_discrete_descriptor_1 as_type_i_desc = {
+ .bLength = UAC_FORMAT_TYPE_I_DISCRETE_DESC_SIZE(1),
+ .bDescriptorType = USB_DT_CS_INTERFACE,
+ .bDescriptorSubtype = UAC_FORMAT_TYPE,
+ .bFormatType = UAC_FORMAT_TYPE_I,
+ .bSubframeSize = 2,
+ .bBitResolution = 16,
+ .bSamFreqType = 1,
+};
+
+/* Standard ISO OUT Endpoint Descriptor */
+static struct usb_endpoint_descriptor as_out_ep_desc = {
+ .bLength = USB_DT_ENDPOINT_AUDIO_SIZE,
+ .bDescriptorType = USB_DT_ENDPOINT,
+ .bEndpointAddress = USB_DIR_OUT,
+ .bmAttributes = USB_ENDPOINT_SYNC_ADAPTIVE
+ | USB_ENDPOINT_XFER_ISOC,
+ .wMaxPacketSize = cpu_to_le16(UAC1_OUT_EP_MAX_PACKET_SIZE),
+ .bInterval = 4,
+};
+
+/* Class-specific AS ISO OUT Endpoint Descriptor */
+static struct uac_iso_endpoint_descriptor as_iso_out_desc = {
+ .bLength = UAC_ISO_ENDPOINT_DESC_SIZE,
+ .bDescriptorType = USB_DT_CS_ENDPOINT,
+ .bDescriptorSubtype = UAC_EP_GENERAL,
+ .bmAttributes = 1,
+ .bLockDelayUnits = 1,
+ .wLockDelay = __constant_cpu_to_le16(1),
+};
+
+static struct usb_descriptor_header *f_audio_desc[] = {
+ (struct usb_descriptor_header *)&ac_interface_desc,
+ (struct usb_descriptor_header *)&ac_header_desc,
+
+ (struct usb_descriptor_header *)&input_terminal_desc,
+ (struct usb_descriptor_header *)&output_terminal_desc,
+ (struct usb_descriptor_header *)&feature_unit_desc,
+
+ (struct usb_descriptor_header *)&as_interface_alt_0_desc,
+ (struct usb_descriptor_header *)&as_interface_alt_1_desc,
+ (struct usb_descriptor_header *)&as_header_desc,
+
+ (struct usb_descriptor_header *)&as_type_i_desc,
+
+ (struct usb_descriptor_header *)&as_out_ep_desc,
+ (struct usb_descriptor_header *)&as_iso_out_desc,
+ NULL,
+};
+
+enum {
+ STR_AC_IF,
+ STR_INPUT_TERMINAL,
+ STR_INPUT_TERMINAL_CH_NAMES,
+ STR_FEAT_DESC_0,
+ STR_OUTPUT_TERMINAL,
+ STR_AS_IF_ALT0,
+ STR_AS_IF_ALT1,
+};
+
+static struct usb_string strings_uac1[] = {
+ [STR_AC_IF].s = "AC Interface",
+ [STR_INPUT_TERMINAL].s = "Input terminal",
+ [STR_INPUT_TERMINAL_CH_NAMES].s = "Channels",
+ [STR_FEAT_DESC_0].s = "Volume control & mute",
+ [STR_OUTPUT_TERMINAL].s = "Output terminal",
+ [STR_AS_IF_ALT0].s = "AS Interface",
+ [STR_AS_IF_ALT1].s = "AS Interface",
+ { },
+};
+
+static struct usb_gadget_strings str_uac1 = {
+ .language = 0x0409, /* en-us */
+ .strings = strings_uac1,
+};
+
+static struct usb_gadget_strings *uac1_strings[] = {
+ &str_uac1,
+ NULL,
+};
+
+/*
+ * This function is an ALSA sound card following USB Audio Class Spec 1.0.
+ */
+
+/*-------------------------------------------------------------------------*/
+struct f_audio_buf {
+ u8 *buf;
+ int actual;
+ struct list_head list;
+};
+
+static struct f_audio_buf *f_audio_buffer_alloc(int buf_size)
+{
+ struct f_audio_buf *copy_buf;
+
+ copy_buf = kzalloc(sizeof *copy_buf, GFP_ATOMIC);
+ if (!copy_buf)
+ return ERR_PTR(-ENOMEM);
+
+ copy_buf->buf = kzalloc(buf_size, GFP_ATOMIC);
+ if (!copy_buf->buf) {
+ kfree(copy_buf);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ return copy_buf;
+}
+
+static void f_audio_buffer_free(struct f_audio_buf *audio_buf)
+{
+ kfree(audio_buf->buf);
+ kfree(audio_buf);
+}
+/*-------------------------------------------------------------------------*/
+
+struct f_audio {
+ struct gaudio card;
+
+ u8 ac_intf, ac_alt;
+ u8 as_intf, as_alt;
+
+ /* endpoints handle full and/or high speeds */
+ struct usb_ep *out_ep;
+
+ spinlock_t lock;
+ struct f_audio_buf *copy_buf;
+ struct work_struct playback_work;
+ struct list_head play_queue;
+
+ /* Control Set command */
+ struct list_head cs;
+ u8 set_cmd;
+ struct usb_audio_control *set_con;
+};
+
+static inline struct f_audio *func_to_audio(struct usb_function *f)
+{
+ return container_of(f, struct f_audio, card.func);
+}
+
+/*-------------------------------------------------------------------------*/
+
+static void f_audio_playback_work(struct work_struct *data)
+{
+ struct f_audio *audio = container_of(data, struct f_audio,
+ playback_work);
+ struct f_audio_buf *play_buf;
+
+ spin_lock_irq(&audio->lock);
+ if (list_empty(&audio->play_queue)) {
+ spin_unlock_irq(&audio->lock);
+ return;
+ }
+ play_buf = list_first_entry(&audio->play_queue,
+ struct f_audio_buf, list);
+ list_del(&play_buf->list);
+ spin_unlock_irq(&audio->lock);
+
+ u_audio_playback(&audio->card, play_buf->buf, play_buf->actual);
+ f_audio_buffer_free(play_buf);
+}
+
+static int f_audio_out_ep_complete(struct usb_ep *ep, struct usb_request *req)
+{
+ struct f_audio *audio = req->context;
+ struct usb_composite_dev *cdev = audio->card.func.config->cdev;
+ struct f_audio_buf *copy_buf = audio->copy_buf;
+ struct f_uac1_legacy_opts *opts;
+ int audio_buf_size;
+ int err;
+
+ opts = container_of(audio->card.func.fi, struct f_uac1_legacy_opts,
+ func_inst);
+ audio_buf_size = opts->audio_buf_size;
+
+ if (!copy_buf)
+ return -EINVAL;
+
+ /* Copy buffer is full, add it to the play_queue */
+ if (audio_buf_size - copy_buf->actual < req->actual) {
+ list_add_tail(©_buf->list, &audio->play_queue);
+ schedule_work(&audio->playback_work);
+ copy_buf = f_audio_buffer_alloc(audio_buf_size);
+ if (IS_ERR(copy_buf))
+ return -ENOMEM;
+ }
+
+ memcpy(copy_buf->buf + copy_buf->actual, req->buf, req->actual);
+ copy_buf->actual += req->actual;
+ audio->copy_buf = copy_buf;
+
+ err = usb_ep_queue(ep, req, GFP_ATOMIC);
+ if (err)
+ ERROR(cdev, "%s queue req: %d\n", ep->name, err);
+
+ return 0;
+
+}
+
+static void f_audio_complete(struct usb_ep *ep, struct usb_request *req)
+{
+ struct f_audio *audio = req->context;
+ int status = req->status;
+ u32 data = 0;
+ struct usb_ep *out_ep = audio->out_ep;
+
+ switch (status) {
+
+ case 0: /* normal completion? */
+ if (ep == out_ep)
+ f_audio_out_ep_complete(ep, req);
+ else if (audio->set_con) {
+ memcpy(&data, req->buf, req->length);
+ audio->set_con->set(audio->set_con, audio->set_cmd,
+ le16_to_cpu(data));
+ audio->set_con = NULL;
+ }
+ break;
+ default:
+ break;
+ }
+}
+
+static int audio_set_intf_req(struct usb_function *f,
+ const struct usb_ctrlrequest *ctrl)
+{
+ struct f_audio *audio = func_to_audio(f);
+ struct usb_composite_dev *cdev = f->config->cdev;
+ struct usb_request *req = cdev->req;
+ u8 id = ((le16_to_cpu(ctrl->wIndex) >> 8) & 0xFF);
+ u16 len = le16_to_cpu(ctrl->wLength);
+ u16 w_value = le16_to_cpu(ctrl->wValue);
+ u8 con_sel = (w_value >> 8) & 0xFF;
+ u8 cmd = (ctrl->bRequest & 0x0F);
+ struct usb_audio_control_selector *cs;
+ struct usb_audio_control *con;
+
+ DBG(cdev, "bRequest 0x%x, w_value 0x%04x, len %d, entity %d\n",
+ ctrl->bRequest, w_value, len, id);
+
+ list_for_each_entry(cs, &audio->cs, list) {
+ if (cs->id == id) {
+ list_for_each_entry(con, &cs->control, list) {
+ if (con->type == con_sel) {
+ audio->set_con = con;
+ break;
+ }
+ }
+ break;
+ }
+ }
+
+ audio->set_cmd = cmd;
+ req->context = audio;
+ req->complete = f_audio_complete;
+
+ return len;
+}
+
+static int audio_get_intf_req(struct usb_function *f,
+ const struct usb_ctrlrequest *ctrl)
+{
+ struct f_audio *audio = func_to_audio(f);
+ struct usb_composite_dev *cdev = f->config->cdev;
+ struct usb_request *req = cdev->req;
+ int value = -EOPNOTSUPP;
+ u8 id = ((le16_to_cpu(ctrl->wIndex) >> 8) & 0xFF);
+ u16 len = le16_to_cpu(ctrl->wLength);
+ u16 w_value = le16_to_cpu(ctrl->wValue);
+ u8 con_sel = (w_value >> 8) & 0xFF;
+ u8 cmd = (ctrl->bRequest & 0x0F);
+ struct usb_audio_control_selector *cs;
+ struct usb_audio_control *con;
+
+ DBG(cdev, "bRequest 0x%x, w_value 0x%04x, len %d, entity %d\n",
+ ctrl->bRequest, w_value, len, id);
+
+ list_for_each_entry(cs, &audio->cs, list) {
+ if (cs->id == id) {
+ list_for_each_entry(con, &cs->control, list) {
+ if (con->type == con_sel && con->get) {
+ value = con->get(con, cmd);
+ break;
+ }
+ }
+ break;
+ }
+ }
+
+ req->context = audio;
+ req->complete = f_audio_complete;
+ len = min_t(size_t, sizeof(value), len);
+ memcpy(req->buf, &value, len);
+
+ return len;
+}
+
+static int audio_set_endpoint_req(struct usb_function *f,
+ const struct usb_ctrlrequest *ctrl)
+{
+ struct usb_composite_dev *cdev = f->config->cdev;
+ int value = -EOPNOTSUPP;
+ u16 ep = le16_to_cpu(ctrl->wIndex);
+ u16 len = le16_to_cpu(ctrl->wLength);
+ u16 w_value = le16_to_cpu(ctrl->wValue);
+
+ DBG(cdev, "bRequest 0x%x, w_value 0x%04x, len %d, endpoint %d\n",
+ ctrl->bRequest, w_value, len, ep);
+
+ switch (ctrl->bRequest) {
+ case UAC_SET_CUR:
+ value = len;
+ break;
+
+ case UAC_SET_MIN:
+ break;
+
+ case UAC_SET_MAX:
+ break;
+
+ case UAC_SET_RES:
+ break;
+
+ case UAC_SET_MEM:
+ break;
+
+ default:
+ break;
+ }
+
+ return value;
+}
+
+static int audio_get_endpoint_req(struct usb_function *f,
+ const struct usb_ctrlrequest *ctrl)
+{
+ struct usb_composite_dev *cdev = f->config->cdev;
+ int value = -EOPNOTSUPP;
+ u8 ep = ((le16_to_cpu(ctrl->wIndex) >> 8) & 0xFF);
+ u16 len = le16_to_cpu(ctrl->wLength);
+ u16 w_value = le16_to_cpu(ctrl->wValue);
+
+ DBG(cdev, "bRequest 0x%x, w_value 0x%04x, len %d, endpoint %d\n",
+ ctrl->bRequest, w_value, len, ep);
+
+ switch (ctrl->bRequest) {
+ case UAC_GET_CUR:
+ case UAC_GET_MIN:
+ case UAC_GET_MAX:
+ case UAC_GET_RES:
+ value = len;
+ break;
+ case UAC_GET_MEM:
+ break;
+ default:
+ break;
+ }
+
+ return value;
+}
+
+static int
+f_audio_setup(struct usb_function *f, const struct usb_ctrlrequest *ctrl)
+{
+ struct usb_composite_dev *cdev = f->config->cdev;
+ struct usb_request *req = cdev->req;
+ int value = -EOPNOTSUPP;
+ u16 w_index = le16_to_cpu(ctrl->wIndex);
+ u16 w_value = le16_to_cpu(ctrl->wValue);
+ u16 w_length = le16_to_cpu(ctrl->wLength);
+
+ /* composite driver infrastructure handles everything; interface
+ * activation uses set_alt().
+ */
+ switch (ctrl->bRequestType) {
+ case USB_DIR_OUT | USB_TYPE_CLASS | USB_RECIP_INTERFACE:
+ value = audio_set_intf_req(f, ctrl);
+ break;
+
+ case USB_DIR_IN | USB_TYPE_CLASS | USB_RECIP_INTERFACE:
+ value = audio_get_intf_req(f, ctrl);
+ break;
+
+ case USB_DIR_OUT | USB_TYPE_CLASS | USB_RECIP_ENDPOINT:
+ value = audio_set_endpoint_req(f, ctrl);
+ break;
+
+ case USB_DIR_IN | USB_TYPE_CLASS | USB_RECIP_ENDPOINT:
+ value = audio_get_endpoint_req(f, ctrl);
+ break;
+
+ default:
+ ERROR(cdev, "invalid control req%02x.%02x v%04x i%04x l%d\n",
+ ctrl->bRequestType, ctrl->bRequest,
+ w_value, w_index, w_length);
+ }
+
+ /* respond with data transfer or status phase? */
+ if (value >= 0) {
+ DBG(cdev, "audio req%02x.%02x v%04x i%04x l%d\n",
+ ctrl->bRequestType, ctrl->bRequest,
+ w_value, w_index, w_length);
+ req->zero = 0;
+ req->length = value;
+ value = usb_ep_queue(cdev->gadget->ep0, req, GFP_ATOMIC);
+ if (value < 0)
+ ERROR(cdev, "audio response on err %d\n", value);
+ }
+
+ /* device either stalls (value < 0) or reports success */
+ return value;
+}
+
+static int f_audio_set_alt(struct usb_function *f, unsigned intf, unsigned alt)
+{
+ struct f_audio *audio = func_to_audio(f);
+ struct usb_composite_dev *cdev = f->config->cdev;
+ struct usb_ep *out_ep = audio->out_ep;
+ struct usb_request *req;
+ struct f_uac1_legacy_opts *opts;
+ int req_buf_size, req_count, audio_buf_size;
+ int i = 0, err = 0;
+
+ DBG(cdev, "intf %d, alt %d\n", intf, alt);
+
+ opts = container_of(f->fi, struct f_uac1_legacy_opts, func_inst);
+ req_buf_size = opts->req_buf_size;
+ req_count = opts->req_count;
+ audio_buf_size = opts->audio_buf_size;
+
+ /* No i/f has more than 2 alt settings */
+ if (alt > 1) {
+ ERROR(cdev, "%s:%d Error!\n", __func__, __LINE__);
+ return -EINVAL;
+ }
+
+ if (intf == audio->ac_intf) {
+ /* Control I/f has only 1 AltSetting - 0 */
+ if (alt) {
+ ERROR(cdev, "%s:%d Error!\n", __func__, __LINE__);
+ return -EINVAL;
+ }
+ return 0;
+ } else if (intf == audio->as_intf) {
+ if (alt == 1) {
+ err = config_ep_by_speed(cdev->gadget, f, out_ep);
+ if (err)
+ return err;
+
+ usb_ep_enable(out_ep);
+ audio->copy_buf = f_audio_buffer_alloc(audio_buf_size);
+ if (IS_ERR(audio->copy_buf))
+ return -ENOMEM;
+
+ /*
+ * allocate a bunch of read buffers
+ * and queue them all at once.
+ */
+ for (i = 0; i < req_count && err == 0; i++) {
+ req = usb_ep_alloc_request(out_ep, GFP_ATOMIC);
+ if (req) {
+ req->buf = kzalloc(req_buf_size,
+ GFP_ATOMIC);
+ if (req->buf) {
+ req->length = req_buf_size;
+ req->context = audio;
+ req->complete =
+ f_audio_complete;
+ err = usb_ep_queue(out_ep,
+ req, GFP_ATOMIC);
+ if (err)
+ ERROR(cdev,
+ "%s queue req: %d\n",
+ out_ep->name, err);
+ } else
+ err = -ENOMEM;
+ } else
+ err = -ENOMEM;
+ }
+
+ } else {
+ struct f_audio_buf *copy_buf = audio->copy_buf;
+ if (copy_buf) {
+ list_add_tail(©_buf->list,
+ &audio->play_queue);
+ schedule_work(&audio->playback_work);
+ }
+ }
+ audio->as_alt = alt;
+ }
+
+ return err;
+}
+
+static int f_audio_get_alt(struct usb_function *f, unsigned intf)
+{
+ struct f_audio *audio = func_to_audio(f);
+ struct usb_composite_dev *cdev = f->config->cdev;
+
+ if (intf == audio->ac_intf)
+ return audio->ac_alt;
+ else if (intf == audio->as_intf)
+ return audio->as_alt;
+ else
+ ERROR(cdev, "%s:%d Invalid Interface %d!\n",
+ __func__, __LINE__, intf);
+
+ return -EINVAL;
+}
+
+static void f_audio_disable(struct usb_function *f)
+{
+ return;
+}
+
+/*-------------------------------------------------------------------------*/
+
+static void f_audio_build_desc(struct f_audio *audio)
+{
+ struct gaudio *card = &audio->card;
+ u8 *sam_freq;
+ int rate;
+
+ /* Set channel numbers */
+ input_terminal_desc.bNrChannels = u_audio_get_playback_channels(card);
+ as_type_i_desc.bNrChannels = u_audio_get_playback_channels(card);
+
+ /* Set sample rates */
+ rate = u_audio_get_playback_rate(card);
+ sam_freq = as_type_i_desc.tSamFreq[0];
+ memcpy(sam_freq, &rate, 3);
+
+ /* Todo: Set Sample bits and other parameters */
+
+ return;
+}
+
+/* audio function driver setup/binding */
+static int
+f_audio_bind(struct usb_configuration *c, struct usb_function *f)
+{
+ struct usb_composite_dev *cdev = c->cdev;
+ struct f_audio *audio = func_to_audio(f);
+ struct usb_string *us;
+ int status;
+ struct usb_ep *ep = NULL;
+ struct f_uac1_legacy_opts *audio_opts;
+
+ audio_opts = container_of(f->fi, struct f_uac1_legacy_opts, func_inst);
+ audio->card.gadget = c->cdev->gadget;
+ /* set up ASLA audio devices */
+ if (!audio_opts->bound) {
+ status = gaudio_setup(&audio->card);
+ if (status < 0)
+ return status;
+ audio_opts->bound = true;
+ }
+ us = usb_gstrings_attach(cdev, uac1_strings, ARRAY_SIZE(strings_uac1));
+ if (IS_ERR(us))
+ return PTR_ERR(us);
+ ac_interface_desc.iInterface = us[STR_AC_IF].id;
+ input_terminal_desc.iTerminal = us[STR_INPUT_TERMINAL].id;
+ input_terminal_desc.iChannelNames = us[STR_INPUT_TERMINAL_CH_NAMES].id;
+ feature_unit_desc.iFeature = us[STR_FEAT_DESC_0].id;
+ output_terminal_desc.iTerminal = us[STR_OUTPUT_TERMINAL].id;
+ as_interface_alt_0_desc.iInterface = us[STR_AS_IF_ALT0].id;
+ as_interface_alt_1_desc.iInterface = us[STR_AS_IF_ALT1].id;
+
+
+ f_audio_build_desc(audio);
+
+ /* allocate instance-specific interface IDs, and patch descriptors */
+ status = usb_interface_id(c, f);
+ if (status < 0)
+ goto fail;
+ ac_interface_desc.bInterfaceNumber = status;
+ audio->ac_intf = status;
+ audio->ac_alt = 0;
+
+ status = usb_interface_id(c, f);
+ if (status < 0)
+ goto fail;
+ as_interface_alt_0_desc.bInterfaceNumber = status;
+ as_interface_alt_1_desc.bInterfaceNumber = status;
+ audio->as_intf = status;
+ audio->as_alt = 0;
+
+ status = -ENODEV;
+
+ /* allocate instance-specific endpoints */
+ ep = usb_ep_autoconfig(cdev->gadget, &as_out_ep_desc);
+ if (!ep)
+ goto fail;
+ audio->out_ep = ep;
+ audio->out_ep->desc = &as_out_ep_desc;
+
+ status = -ENOMEM;
+
+ /* copy descriptors, and track endpoint copies */
+ status = usb_assign_descriptors(f, f_audio_desc, f_audio_desc, NULL,
+ NULL);
+ if (status)
+ goto fail;
+ return 0;
+
+fail:
+ gaudio_cleanup(&audio->card);
+ return status;
+}
+
+/*-------------------------------------------------------------------------*/
+
+static int generic_set_cmd(struct usb_audio_control *con, u8 cmd, int value)
+{
+ con->data[cmd] = value;
+
+ return 0;
+}
+
+static int generic_get_cmd(struct usb_audio_control *con, u8 cmd)
+{
+ return con->data[cmd];
+}
+
+/* Todo: add more control selecotor dynamically */
+static int control_selector_init(struct f_audio *audio)
+{
+ INIT_LIST_HEAD(&audio->cs);
+ list_add(&feature_unit.list, &audio->cs);
+
+ INIT_LIST_HEAD(&feature_unit.control);
+ list_add(&mute_control.list, &feature_unit.control);
+ list_add(&volume_control.list, &feature_unit.control);
+
+ volume_control.data[UAC__CUR] = 0xffc0;
+ volume_control.data[UAC__MIN] = 0xe3a0;
+ volume_control.data[UAC__MAX] = 0xfff0;
+ volume_control.data[UAC__RES] = 0x0030;
+
+ return 0;
+}
+
+static inline
+struct f_uac1_legacy_opts *to_f_uac1_opts(struct config_item *item)
+{
+ return container_of(to_config_group(item), struct f_uac1_legacy_opts,
+ func_inst.group);
+}
+
+static void f_uac1_attr_release(struct config_item *item)
+{
+ struct f_uac1_legacy_opts *opts = to_f_uac1_opts(item);
+
+ usb_put_function_instance(&opts->func_inst);
+}
+
+static struct configfs_item_operations f_uac1_item_ops = {
+ .release = f_uac1_attr_release,
+};
+
+#define UAC1_INT_ATTRIBUTE(name) \
+static ssize_t f_uac1_opts_##name##_show(struct config_item *item, \
+ char *page) \
+{ \
+ struct f_uac1_legacy_opts *opts = to_f_uac1_opts(item); \
+ int result; \
+ \
+ mutex_lock(&opts->lock); \
+ result = sprintf(page, "%u\n", opts->name); \
+ mutex_unlock(&opts->lock); \
+ \
+ return result; \
+} \
+ \
+static ssize_t f_uac1_opts_##name##_store(struct config_item *item, \
+ const char *page, size_t len) \
+{ \
+ struct f_uac1_legacy_opts *opts = to_f_uac1_opts(item); \
+ int ret; \
+ u32 num; \
+ \
+ mutex_lock(&opts->lock); \
+ if (opts->refcnt) { \
+ ret = -EBUSY; \
+ goto end; \
+ } \
+ \
+ ret = kstrtou32(page, 0, &num); \
+ if (ret) \
+ goto end; \
+ \
+ opts->name = num; \
+ ret = len; \
+ \
+end: \
+ mutex_unlock(&opts->lock); \
+ return ret; \
+} \
+ \
+CONFIGFS_ATTR(f_uac1_opts_, name)
+
+UAC1_INT_ATTRIBUTE(req_buf_size);
+UAC1_INT_ATTRIBUTE(req_count);
+UAC1_INT_ATTRIBUTE(audio_buf_size);
+
+#define UAC1_STR_ATTRIBUTE(name) \
+static ssize_t f_uac1_opts_##name##_show(struct config_item *item, \
+ char *page) \
+{ \
+ struct f_uac1_legacy_opts *opts = to_f_uac1_opts(item); \
+ int result; \
+ \
+ mutex_lock(&opts->lock); \
+ result = sprintf(page, "%s\n", opts->name); \
+ mutex_unlock(&opts->lock); \
+ \
+ return result; \
+} \
+ \
+static ssize_t f_uac1_opts_##name##_store(struct config_item *item, \
+ const char *page, size_t len) \
+{ \
+ struct f_uac1_legacy_opts *opts = to_f_uac1_opts(item); \
+ int ret = -EBUSY; \
+ char *tmp; \
+ \
+ mutex_lock(&opts->lock); \
+ if (opts->refcnt) \
+ goto end; \
+ \
+ tmp = kstrndup(page, len, GFP_KERNEL); \
+ if (tmp) { \
+ ret = -ENOMEM; \
+ goto end; \
+ } \
+ if (opts->name##_alloc) \
+ kfree(opts->name); \
+ opts->name##_alloc = true; \
+ opts->name = tmp; \
+ ret = len; \
+ \
+end: \
+ mutex_unlock(&opts->lock); \
+ return ret; \
+} \
+ \
+CONFIGFS_ATTR(f_uac1_opts_, name)
+
+UAC1_STR_ATTRIBUTE(fn_play);
+UAC1_STR_ATTRIBUTE(fn_cap);
+UAC1_STR_ATTRIBUTE(fn_cntl);
+
+static struct configfs_attribute *f_uac1_attrs[] = {
+ &f_uac1_opts_attr_req_buf_size,
+ &f_uac1_opts_attr_req_count,
+ &f_uac1_opts_attr_audio_buf_size,
+ &f_uac1_opts_attr_fn_play,
+ &f_uac1_opts_attr_fn_cap,
+ &f_uac1_opts_attr_fn_cntl,
+ NULL,
+};
+
+static struct config_item_type f_uac1_func_type = {
+ .ct_item_ops = &f_uac1_item_ops,
+ .ct_attrs = f_uac1_attrs,
+ .ct_owner = THIS_MODULE,
+};
+
+static void f_audio_free_inst(struct usb_function_instance *f)
+{
+ struct f_uac1_legacy_opts *opts;
+
+ opts = container_of(f, struct f_uac1_legacy_opts, func_inst);
+ if (opts->fn_play_alloc)
+ kfree(opts->fn_play);
+ if (opts->fn_cap_alloc)
+ kfree(opts->fn_cap);
+ if (opts->fn_cntl_alloc)
+ kfree(opts->fn_cntl);
+ kfree(opts);
+}
+
+static struct usb_function_instance *f_audio_alloc_inst(void)
+{
+ struct f_uac1_legacy_opts *opts;
+
+ opts = kzalloc(sizeof(*opts), GFP_KERNEL);
+ if (!opts)
+ return ERR_PTR(-ENOMEM);
+
+ mutex_init(&opts->lock);
+ opts->func_inst.free_func_inst = f_audio_free_inst;
+
+ config_group_init_type_name(&opts->func_inst.group, "",
+ &f_uac1_func_type);
+
+ opts->req_buf_size = UAC1_OUT_EP_MAX_PACKET_SIZE;
+ opts->req_count = UAC1_REQ_COUNT;
+ opts->audio_buf_size = UAC1_AUDIO_BUF_SIZE;
+ opts->fn_play = FILE_PCM_PLAYBACK;
+ opts->fn_cap = FILE_PCM_CAPTURE;
+ opts->fn_cntl = FILE_CONTROL;
+ return &opts->func_inst;
+}
+
+static void f_audio_free(struct usb_function *f)
+{
+ struct f_audio *audio = func_to_audio(f);
+ struct f_uac1_legacy_opts *opts;
+
+ gaudio_cleanup(&audio->card);
+ opts = container_of(f->fi, struct f_uac1_legacy_opts, func_inst);
+ kfree(audio);
+ mutex_lock(&opts->lock);
+ --opts->refcnt;
+ mutex_unlock(&opts->lock);
+}
+
+static void f_audio_unbind(struct usb_configuration *c, struct usb_function *f)
+{
+ usb_free_all_descriptors(f);
+}
+
+static struct usb_function *f_audio_alloc(struct usb_function_instance *fi)
+{
+ struct f_audio *audio;
+ struct f_uac1_legacy_opts *opts;
+
+ /* allocate and initialize one new instance */
+ audio = kzalloc(sizeof(*audio), GFP_KERNEL);
+ if (!audio)
+ return ERR_PTR(-ENOMEM);
+
+ audio->card.func.name = "g_audio";
+
+ opts = container_of(fi, struct f_uac1_legacy_opts, func_inst);
+ mutex_lock(&opts->lock);
+ ++opts->refcnt;
+ mutex_unlock(&opts->lock);
+ INIT_LIST_HEAD(&audio->play_queue);
+ spin_lock_init(&audio->lock);
+
+ audio->card.func.bind = f_audio_bind;
+ audio->card.func.unbind = f_audio_unbind;
+ audio->card.func.set_alt = f_audio_set_alt;
+ audio->card.func.get_alt = f_audio_get_alt;
+ audio->card.func.setup = f_audio_setup;
+ audio->card.func.disable = f_audio_disable;
+ audio->card.func.free_func = f_audio_free;
+
+ control_selector_init(audio);
+
+ INIT_WORK(&audio->playback_work, f_audio_playback_work);
+
+ return &audio->card.func;
+}
+
+DECLARE_USB_FUNCTION_INIT(uac1_legacy, f_audio_alloc_inst, f_audio_alloc);
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Bryan Wu");
#include <linux/usb/audio.h>
#include <linux/usb/audio-v2.h>
-#include <linux/platform_device.h>
#include <linux/module.h>
-#include <sound/core.h>
-#include <sound/pcm.h>
-#include <sound/pcm_params.h>
-
+#include "u_audio.h"
#include "u_uac2.h"
/*
#define UNFLW_CTRL 8
#define OVFLW_CTRL 10
-static const char *uac2_name = "snd_uac2";
-
-struct uac2_req {
- struct uac2_rtd_params *pp; /* parent param */
- struct usb_request *req;
-};
-
-struct uac2_rtd_params {
- struct snd_uac2_chip *uac2; /* parent chip */
- bool ep_enabled; /* if the ep is enabled */
- /* Size of the ring buffer */
- size_t dma_bytes;
- unsigned char *dma_area;
-
- struct snd_pcm_substream *ss;
-
- /* Ring buffer */
- ssize_t hw_ptr;
-
- void *rbuf;
-
- size_t period_size;
-
- unsigned max_psize;
- struct uac2_req *ureq;
-
- spinlock_t lock;
-};
-
-struct snd_uac2_chip {
- struct platform_device pdev;
- struct platform_driver pdrv;
-
- struct uac2_rtd_params p_prm;
- struct uac2_rtd_params c_prm;
-
- struct snd_card *card;
- struct snd_pcm *pcm;
-
- /* timekeeping for the playback endpoint */
- unsigned int p_interval;
- unsigned int p_residue;
-
- /* pre-calculated values for playback iso completion */
- unsigned int p_pktsize;
- unsigned int p_pktsize_residue;
- unsigned int p_framesize;
+struct f_uac2 {
+ struct g_audio g_audio;
+ u8 ac_intf, as_in_intf, as_out_intf;
+ u8 ac_alt, as_in_alt, as_out_alt; /* needed for get_alt() */
};
-#define BUFF_SIZE_MAX (PAGE_SIZE * 16)
-#define PRD_SIZE_MAX PAGE_SIZE
-#define MIN_PERIODS 4
-
-static struct snd_pcm_hardware uac2_pcm_hardware = {
- .info = SNDRV_PCM_INFO_INTERLEAVED | SNDRV_PCM_INFO_BLOCK_TRANSFER
- | SNDRV_PCM_INFO_MMAP | SNDRV_PCM_INFO_MMAP_VALID
- | SNDRV_PCM_INFO_PAUSE | SNDRV_PCM_INFO_RESUME,
- .rates = SNDRV_PCM_RATE_CONTINUOUS,
- .periods_max = BUFF_SIZE_MAX / PRD_SIZE_MAX,
- .buffer_bytes_max = BUFF_SIZE_MAX,
- .period_bytes_max = PRD_SIZE_MAX,
- .periods_min = MIN_PERIODS,
-};
-
-struct audio_dev {
- u8 ac_intf, ac_alt;
- u8 as_out_intf, as_out_alt;
- u8 as_in_intf, as_in_alt;
-
- struct usb_ep *in_ep, *out_ep;
- struct usb_function func;
-
- /* The ALSA Sound Card it represents on the USB-Client side */
- struct snd_uac2_chip uac2;
-};
-
-static inline
-struct audio_dev *func_to_agdev(struct usb_function *f)
+static inline struct f_uac2 *func_to_uac2(struct usb_function *f)
{
- return container_of(f, struct audio_dev, func);
+ return container_of(f, struct f_uac2, g_audio.func);
}
static inline
-struct audio_dev *uac2_to_agdev(struct snd_uac2_chip *u)
-{
- return container_of(u, struct audio_dev, uac2);
-}
-
-static inline
-struct snd_uac2_chip *pdev_to_uac2(struct platform_device *p)
-{
- return container_of(p, struct snd_uac2_chip, pdev);
-}
-
-static inline
-struct f_uac2_opts *agdev_to_uac2_opts(struct audio_dev *agdev)
+struct f_uac2_opts *g_audio_to_uac2_opts(struct g_audio *agdev)
{
return container_of(agdev->func.fi, struct f_uac2_opts, func_inst);
}
-static inline
-uint num_channels(uint chanmask)
-{
- uint num = 0;
-
- while (chanmask) {
- num += (chanmask & 1);
- chanmask >>= 1;
- }
-
- return num;
-}
-
-static void
-agdev_iso_complete(struct usb_ep *ep, struct usb_request *req)
-{
- unsigned pending;
- unsigned long flags;
- unsigned int hw_ptr;
- bool update_alsa = false;
- int status = req->status;
- struct uac2_req *ur = req->context;
- struct snd_pcm_substream *substream;
- struct uac2_rtd_params *prm = ur->pp;
- struct snd_uac2_chip *uac2 = prm->uac2;
-
- /* i/f shutting down */
- if (!prm->ep_enabled || req->status == -ESHUTDOWN)
- return;
-
- /*
- * We can't really do much about bad xfers.
- * Afterall, the ISOCH xfers could fail legitimately.
- */
- if (status)
- pr_debug("%s: iso_complete status(%d) %d/%d\n",
- __func__, status, req->actual, req->length);
-
- substream = prm->ss;
-
- /* Do nothing if ALSA isn't active */
- if (!substream)
- goto exit;
-
- spin_lock_irqsave(&prm->lock, flags);
-
- if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK) {
- /*
- * For each IN packet, take the quotient of the current data
- * rate and the endpoint's interval as the base packet size.
- * If there is a residue from this division, add it to the
- * residue accumulator.
- */
- req->length = uac2->p_pktsize;
- uac2->p_residue += uac2->p_pktsize_residue;
-
- /*
- * Whenever there are more bytes in the accumulator than we
- * need to add one more sample frame, increase this packet's
- * size and decrease the accumulator.
- */
- if (uac2->p_residue / uac2->p_interval >= uac2->p_framesize) {
- req->length += uac2->p_framesize;
- uac2->p_residue -= uac2->p_framesize *
- uac2->p_interval;
- }
-
- req->actual = req->length;
- }
-
- pending = prm->hw_ptr % prm->period_size;
- pending += req->actual;
- if (pending >= prm->period_size)
- update_alsa = true;
-
- hw_ptr = prm->hw_ptr;
- prm->hw_ptr = (prm->hw_ptr + req->actual) % prm->dma_bytes;
-
- spin_unlock_irqrestore(&prm->lock, flags);
-
- /* Pack USB load in ALSA ring buffer */
- pending = prm->dma_bytes - hw_ptr;
-
- if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK) {
- if (unlikely(pending < req->actual)) {
- memcpy(req->buf, prm->dma_area + hw_ptr, pending);
- memcpy(req->buf + pending, prm->dma_area,
- req->actual - pending);
- } else {
- memcpy(req->buf, prm->dma_area + hw_ptr, req->actual);
- }
- } else {
- if (unlikely(pending < req->actual)) {
- memcpy(prm->dma_area + hw_ptr, req->buf, pending);
- memcpy(prm->dma_area, req->buf + pending,
- req->actual - pending);
- } else {
- memcpy(prm->dma_area + hw_ptr, req->buf, req->actual);
- }
- }
-
-exit:
- if (usb_ep_queue(ep, req, GFP_ATOMIC))
- dev_err(&uac2->pdev.dev, "%d Error!\n", __LINE__);
-
- if (update_alsa)
- snd_pcm_period_elapsed(substream);
-
- return;
-}
-
-static int
-uac2_pcm_trigger(struct snd_pcm_substream *substream, int cmd)
-{
- struct snd_uac2_chip *uac2 = snd_pcm_substream_chip(substream);
- struct audio_dev *agdev = uac2_to_agdev(uac2);
- struct f_uac2_opts *uac2_opts = agdev_to_uac2_opts(agdev);
- struct uac2_rtd_params *prm;
- unsigned long flags;
- int err = 0;
-
- if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK)
- prm = &uac2->p_prm;
- else
- prm = &uac2->c_prm;
-
- spin_lock_irqsave(&prm->lock, flags);
-
- /* Reset */
- prm->hw_ptr = 0;
-
- switch (cmd) {
- case SNDRV_PCM_TRIGGER_START:
- case SNDRV_PCM_TRIGGER_RESUME:
- prm->ss = substream;
- break;
- case SNDRV_PCM_TRIGGER_STOP:
- case SNDRV_PCM_TRIGGER_SUSPEND:
- prm->ss = NULL;
- break;
- default:
- err = -EINVAL;
- }
-
- spin_unlock_irqrestore(&prm->lock, flags);
-
- /* Clear buffer after Play stops */
- if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK && !prm->ss)
- memset(prm->rbuf, 0, prm->max_psize * uac2_opts->req_number);
-
- return err;
-}
-
-static snd_pcm_uframes_t uac2_pcm_pointer(struct snd_pcm_substream *substream)
-{
- struct snd_uac2_chip *uac2 = snd_pcm_substream_chip(substream);
- struct uac2_rtd_params *prm;
-
- if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK)
- prm = &uac2->p_prm;
- else
- prm = &uac2->c_prm;
-
- return bytes_to_frames(substream->runtime, prm->hw_ptr);
-}
-
-static int uac2_pcm_hw_params(struct snd_pcm_substream *substream,
- struct snd_pcm_hw_params *hw_params)
-{
- struct snd_uac2_chip *uac2 = snd_pcm_substream_chip(substream);
- struct uac2_rtd_params *prm;
- int err;
-
- if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK)
- prm = &uac2->p_prm;
- else
- prm = &uac2->c_prm;
-
- err = snd_pcm_lib_malloc_pages(substream,
- params_buffer_bytes(hw_params));
- if (err >= 0) {
- prm->dma_bytes = substream->runtime->dma_bytes;
- prm->dma_area = substream->runtime->dma_area;
- prm->period_size = params_period_bytes(hw_params);
- }
-
- return err;
-}
-
-static int uac2_pcm_hw_free(struct snd_pcm_substream *substream)
-{
- struct snd_uac2_chip *uac2 = snd_pcm_substream_chip(substream);
- struct uac2_rtd_params *prm;
-
- if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK)
- prm = &uac2->p_prm;
- else
- prm = &uac2->c_prm;
-
- prm->dma_area = NULL;
- prm->dma_bytes = 0;
- prm->period_size = 0;
-
- return snd_pcm_lib_free_pages(substream);
-}
-
-static int uac2_pcm_open(struct snd_pcm_substream *substream)
-{
- struct snd_uac2_chip *uac2 = snd_pcm_substream_chip(substream);
- struct snd_pcm_runtime *runtime = substream->runtime;
- struct audio_dev *audio_dev;
- struct f_uac2_opts *opts;
- int p_ssize, c_ssize;
- int p_srate, c_srate;
- int p_chmask, c_chmask;
-
- audio_dev = uac2_to_agdev(uac2);
- opts = container_of(audio_dev->func.fi, struct f_uac2_opts, func_inst);
- p_ssize = opts->p_ssize;
- c_ssize = opts->c_ssize;
- p_srate = opts->p_srate;
- c_srate = opts->c_srate;
- p_chmask = opts->p_chmask;
- c_chmask = opts->c_chmask;
- uac2->p_residue = 0;
-
- runtime->hw = uac2_pcm_hardware;
-
- if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK) {
- spin_lock_init(&uac2->p_prm.lock);
- runtime->hw.rate_min = p_srate;
- switch (p_ssize) {
- case 3:
- runtime->hw.formats = SNDRV_PCM_FMTBIT_S24_3LE;
- break;
- case 4:
- runtime->hw.formats = SNDRV_PCM_FMTBIT_S32_LE;
- break;
- default:
- runtime->hw.formats = SNDRV_PCM_FMTBIT_S16_LE;
- break;
- }
- runtime->hw.channels_min = num_channels(p_chmask);
- runtime->hw.period_bytes_min = 2 * uac2->p_prm.max_psize
- / runtime->hw.periods_min;
- } else {
- spin_lock_init(&uac2->c_prm.lock);
- runtime->hw.rate_min = c_srate;
- switch (c_ssize) {
- case 3:
- runtime->hw.formats = SNDRV_PCM_FMTBIT_S24_3LE;
- break;
- case 4:
- runtime->hw.formats = SNDRV_PCM_FMTBIT_S32_LE;
- break;
- default:
- runtime->hw.formats = SNDRV_PCM_FMTBIT_S16_LE;
- break;
- }
- runtime->hw.channels_min = num_channels(c_chmask);
- runtime->hw.period_bytes_min = 2 * uac2->c_prm.max_psize
- / runtime->hw.periods_min;
- }
-
- runtime->hw.rate_max = runtime->hw.rate_min;
- runtime->hw.channels_max = runtime->hw.channels_min;
-
- snd_pcm_hw_constraint_integer(runtime, SNDRV_PCM_HW_PARAM_PERIODS);
-
- return 0;
-}
-
-/* ALSA cries without these function pointers */
-static int uac2_pcm_null(struct snd_pcm_substream *substream)
-{
- return 0;
-}
-
-static struct snd_pcm_ops uac2_pcm_ops = {
- .open = uac2_pcm_open,
- .close = uac2_pcm_null,
- .ioctl = snd_pcm_lib_ioctl,
- .hw_params = uac2_pcm_hw_params,
- .hw_free = uac2_pcm_hw_free,
- .trigger = uac2_pcm_trigger,
- .pointer = uac2_pcm_pointer,
- .prepare = uac2_pcm_null,
-};
-
-static int snd_uac2_probe(struct platform_device *pdev)
-{
- struct snd_uac2_chip *uac2 = pdev_to_uac2(pdev);
- struct snd_card *card;
- struct snd_pcm *pcm;
- struct audio_dev *audio_dev;
- struct f_uac2_opts *opts;
- int err;
- int p_chmask, c_chmask;
-
- audio_dev = uac2_to_agdev(uac2);
- opts = container_of(audio_dev->func.fi, struct f_uac2_opts, func_inst);
- p_chmask = opts->p_chmask;
- c_chmask = opts->c_chmask;
-
- /* Choose any slot, with no id */
- err = snd_card_new(&pdev->dev, -1, NULL, THIS_MODULE, 0, &card);
- if (err < 0)
- return err;
-
- uac2->card = card;
-
- /*
- * Create first PCM device
- * Create a substream only for non-zero channel streams
- */
- err = snd_pcm_new(uac2->card, "UAC2 PCM", 0,
- p_chmask ? 1 : 0, c_chmask ? 1 : 0, &pcm);
- if (err < 0)
- goto snd_fail;
-
- strcpy(pcm->name, "UAC2 PCM");
- pcm->private_data = uac2;
-
- uac2->pcm = pcm;
-
- snd_pcm_set_ops(pcm, SNDRV_PCM_STREAM_PLAYBACK, &uac2_pcm_ops);
- snd_pcm_set_ops(pcm, SNDRV_PCM_STREAM_CAPTURE, &uac2_pcm_ops);
-
- strcpy(card->driver, "UAC2_Gadget");
- strcpy(card->shortname, "UAC2_Gadget");
- sprintf(card->longname, "UAC2_Gadget %i", pdev->id);
-
- snd_pcm_lib_preallocate_pages_for_all(pcm, SNDRV_DMA_TYPE_CONTINUOUS,
- snd_dma_continuous_data(GFP_KERNEL), 0, BUFF_SIZE_MAX);
-
- err = snd_card_register(card);
- if (!err) {
- platform_set_drvdata(pdev, card);
- return 0;
- }
-
-snd_fail:
- snd_card_free(card);
-
- uac2->pcm = NULL;
- uac2->card = NULL;
-
- return err;
-}
-
-static int snd_uac2_remove(struct platform_device *pdev)
-{
- struct snd_card *card = platform_get_drvdata(pdev);
-
- if (card)
- return snd_card_free(card);
-
- return 0;
-}
-
-static void snd_uac2_release(struct device *dev)
-{
- dev_dbg(dev, "releasing '%s'\n", dev_name(dev));
-}
-
-static int alsa_uac2_init(struct audio_dev *agdev)
-{
- struct snd_uac2_chip *uac2 = &agdev->uac2;
- int err;
-
- uac2->pdrv.probe = snd_uac2_probe;
- uac2->pdrv.remove = snd_uac2_remove;
- uac2->pdrv.driver.name = uac2_name;
-
- uac2->pdev.id = 0;
- uac2->pdev.name = uac2_name;
- uac2->pdev.dev.release = snd_uac2_release;
-
- /* Register snd_uac2 driver */
- err = platform_driver_register(&uac2->pdrv);
- if (err)
- return err;
-
- /* Register snd_uac2 device */
- err = platform_device_register(&uac2->pdev);
- if (err)
- platform_driver_unregister(&uac2->pdrv);
-
- return err;
-}
-
-static void alsa_uac2_exit(struct audio_dev *agdev)
-{
- struct snd_uac2_chip *uac2 = &agdev->uac2;
-
- platform_driver_unregister(&uac2->pdrv);
- platform_device_unregister(&uac2->pdev);
-}
-
-
/* --------- USB Function Interface ------------- */
enum {
__u32 dRES;
} __packed;
-static inline void
-free_ep(struct uac2_rtd_params *prm, struct usb_ep *ep)
-{
- struct snd_uac2_chip *uac2 = prm->uac2;
- struct audio_dev *agdev = uac2_to_agdev(uac2);
- struct f_uac2_opts *uac2_opts = agdev_to_uac2_opts(agdev);
- int i;
-
- if (!prm->ep_enabled)
- return;
-
- prm->ep_enabled = false;
-
- for (i = 0; i < uac2_opts->req_number; i++) {
- if (prm->ureq[i].req) {
- usb_ep_dequeue(ep, prm->ureq[i].req);
- usb_ep_free_request(ep, prm->ureq[i].req);
- prm->ureq[i].req = NULL;
- }
- }
-
- if (usb_ep_disable(ep))
- dev_err(&uac2->pdev.dev,
- "%s:%d Error!\n", __func__, __LINE__);
-}
-
static void set_ep_max_packet_size(const struct f_uac2_opts *uac2_opts,
struct usb_endpoint_descriptor *ep_desc,
unsigned int factor, bool is_playback)
static int
afunc_bind(struct usb_configuration *cfg, struct usb_function *fn)
{
- struct audio_dev *agdev = func_to_agdev(fn);
- struct snd_uac2_chip *uac2 = &agdev->uac2;
+ struct f_uac2 *uac2 = func_to_uac2(fn);
+ struct g_audio *agdev = func_to_g_audio(fn);
struct usb_composite_dev *cdev = cfg->cdev;
struct usb_gadget *gadget = cdev->gadget;
- struct device *dev = &uac2->pdev.dev;
- struct uac2_rtd_params *prm;
+ struct device *dev = &gadget->dev;
struct f_uac2_opts *uac2_opts;
struct usb_string *us;
int ret;
return ret;
}
std_ac_if_desc.bInterfaceNumber = ret;
- agdev->ac_intf = ret;
- agdev->ac_alt = 0;
+ uac2->ac_intf = ret;
+ uac2->ac_alt = 0;
ret = usb_interface_id(cfg, fn);
if (ret < 0) {
}
std_as_out_if0_desc.bInterfaceNumber = ret;
std_as_out_if1_desc.bInterfaceNumber = ret;
- agdev->as_out_intf = ret;
- agdev->as_out_alt = 0;
+ uac2->as_out_intf = ret;
+ uac2->as_out_alt = 0;
ret = usb_interface_id(cfg, fn);
if (ret < 0) {
}
std_as_in_if0_desc.bInterfaceNumber = ret;
std_as_in_if1_desc.bInterfaceNumber = ret;
- agdev->as_in_intf = ret;
- agdev->as_in_alt = 0;
+ uac2->as_in_intf = ret;
+ uac2->as_in_alt = 0;
+
+ /* Calculate wMaxPacketSize according to audio bandwidth */
+ set_ep_max_packet_size(uac2_opts, &fs_epin_desc, 1000, true);
+ set_ep_max_packet_size(uac2_opts, &fs_epout_desc, 1000, false);
+ set_ep_max_packet_size(uac2_opts, &hs_epin_desc, 8000, true);
+ set_ep_max_packet_size(uac2_opts, &hs_epout_desc, 8000, false);
agdev->out_ep = usb_ep_autoconfig(gadget, &fs_epout_desc);
if (!agdev->out_ep) {
return ret;
}
- uac2->p_prm.uac2 = uac2;
- uac2->c_prm.uac2 = uac2;
-
- /* Calculate wMaxPacketSize according to audio bandwidth */
- set_ep_max_packet_size(uac2_opts, &fs_epin_desc, 1000, true);
- set_ep_max_packet_size(uac2_opts, &fs_epout_desc, 1000, false);
- set_ep_max_packet_size(uac2_opts, &hs_epin_desc, 8000, true);
- set_ep_max_packet_size(uac2_opts, &hs_epout_desc, 8000, false);
+ agdev->in_ep_maxpsize = max(fs_epin_desc.wMaxPacketSize,
+ hs_epin_desc.wMaxPacketSize);
+ agdev->out_ep_maxpsize = max(fs_epout_desc.wMaxPacketSize,
+ hs_epout_desc.wMaxPacketSize);
hs_epout_desc.bEndpointAddress = fs_epout_desc.bEndpointAddress;
hs_epin_desc.bEndpointAddress = fs_epin_desc.bEndpointAddress;
if (ret)
return ret;
- prm = &agdev->uac2.c_prm;
- prm->max_psize = hs_epout_desc.wMaxPacketSize;
- prm->ureq = kcalloc(uac2_opts->req_number, sizeof(struct uac2_req),
- GFP_KERNEL);
- if (!prm->ureq) {
- ret = -ENOMEM;
- goto err_free_descs;
- }
- prm->rbuf = kcalloc(uac2_opts->req_number, prm->max_psize, GFP_KERNEL);
- if (!prm->rbuf) {
- prm->max_psize = 0;
- ret = -ENOMEM;
- goto err_free_descs;
- }
-
- prm = &agdev->uac2.p_prm;
- prm->max_psize = hs_epin_desc.wMaxPacketSize;
- prm->ureq = kcalloc(uac2_opts->req_number, sizeof(struct uac2_req),
- GFP_KERNEL);
- if (!prm->ureq) {
- ret = -ENOMEM;
- goto err_free_descs;
- }
- prm->rbuf = kcalloc(uac2_opts->req_number, prm->max_psize, GFP_KERNEL);
- if (!prm->rbuf) {
- prm->max_psize = 0;
- ret = -ENOMEM;
- goto err_no_memory;
- }
+ agdev->gadget = gadget;
- ret = alsa_uac2_init(agdev);
+ agdev->params.p_chmask = uac2_opts->p_chmask;
+ agdev->params.p_srate = uac2_opts->p_srate;
+ agdev->params.p_ssize = uac2_opts->p_ssize;
+ agdev->params.c_chmask = uac2_opts->c_chmask;
+ agdev->params.c_srate = uac2_opts->c_srate;
+ agdev->params.c_ssize = uac2_opts->c_ssize;
+ agdev->params.req_number = uac2_opts->req_number;
+ ret = g_audio_setup(agdev, "UAC2 PCM", "UAC2_Gadget");
if (ret)
- goto err_no_memory;
+ goto err_free_descs;
return 0;
-err_no_memory:
- kfree(agdev->uac2.p_prm.ureq);
- kfree(agdev->uac2.c_prm.ureq);
- kfree(agdev->uac2.p_prm.rbuf);
- kfree(agdev->uac2.c_prm.rbuf);
err_free_descs:
usb_free_all_descriptors(fn);
+ agdev->gadget = NULL;
return ret;
}
afunc_set_alt(struct usb_function *fn, unsigned intf, unsigned alt)
{
struct usb_composite_dev *cdev = fn->config->cdev;
- struct audio_dev *agdev = func_to_agdev(fn);
- struct f_uac2_opts *opts = agdev_to_uac2_opts(agdev);
- struct snd_uac2_chip *uac2 = &agdev->uac2;
+ struct f_uac2 *uac2 = func_to_uac2(fn);
struct usb_gadget *gadget = cdev->gadget;
- struct device *dev = &uac2->pdev.dev;
- struct usb_request *req;
- struct usb_ep *ep;
- struct uac2_rtd_params *prm;
- int req_len, i;
+ struct device *dev = &gadget->dev;
+ int ret = 0;
/* No i/f has more than 2 alt settings */
if (alt > 1) {
return -EINVAL;
}
- if (intf == agdev->ac_intf) {
+ if (intf == uac2->ac_intf) {
/* Control I/f has only 1 AltSetting - 0 */
if (alt) {
dev_err(dev, "%s:%d Error!\n", __func__, __LINE__);
return 0;
}
- if (intf == agdev->as_out_intf) {
- ep = agdev->out_ep;
- prm = &uac2->c_prm;
- config_ep_by_speed(gadget, fn, ep);
- agdev->as_out_alt = alt;
- req_len = prm->max_psize;
- } else if (intf == agdev->as_in_intf) {
- unsigned int factor, rate;
- struct usb_endpoint_descriptor *ep_desc;
-
- ep = agdev->in_ep;
- prm = &uac2->p_prm;
- config_ep_by_speed(gadget, fn, ep);
- agdev->as_in_alt = alt;
-
- /* pre-calculate the playback endpoint's interval */
- if (gadget->speed == USB_SPEED_FULL) {
- ep_desc = &fs_epin_desc;
- factor = 1000;
- } else {
- ep_desc = &hs_epin_desc;
- factor = 8000;
- }
-
- /* pre-compute some values for iso_complete() */
- uac2->p_framesize = opts->p_ssize *
- num_channels(opts->p_chmask);
- rate = opts->p_srate * uac2->p_framesize;
- uac2->p_interval = factor / (1 << (ep_desc->bInterval - 1));
- uac2->p_pktsize = min_t(unsigned int, rate / uac2->p_interval,
- prm->max_psize);
+ if (intf == uac2->as_out_intf) {
+ uac2->as_out_alt = alt;
- if (uac2->p_pktsize < prm->max_psize)
- uac2->p_pktsize_residue = rate % uac2->p_interval;
+ if (alt)
+ ret = u_audio_start_capture(&uac2->g_audio);
else
- uac2->p_pktsize_residue = 0;
+ u_audio_stop_capture(&uac2->g_audio);
+ } else if (intf == uac2->as_in_intf) {
+ uac2->as_in_alt = alt;
- req_len = uac2->p_pktsize;
- uac2->p_residue = 0;
+ if (alt)
+ ret = u_audio_start_playback(&uac2->g_audio);
+ else
+ u_audio_stop_playback(&uac2->g_audio);
} else {
dev_err(dev, "%s:%d Error!\n", __func__, __LINE__);
return -EINVAL;
}
- if (alt == 0) {
- free_ep(prm, ep);
- return 0;
- }
-
- prm->ep_enabled = true;
- usb_ep_enable(ep);
-
- for (i = 0; i < opts->req_number; i++) {
- if (!prm->ureq[i].req) {
- req = usb_ep_alloc_request(ep, GFP_ATOMIC);
- if (req == NULL)
- return -ENOMEM;
-
- prm->ureq[i].req = req;
- prm->ureq[i].pp = prm;
-
- req->zero = 0;
- req->context = &prm->ureq[i];
- req->length = req_len;
- req->complete = agdev_iso_complete;
- req->buf = prm->rbuf + i * prm->max_psize;
- }
-
- if (usb_ep_queue(ep, prm->ureq[i].req, GFP_ATOMIC))
- dev_err(dev, "%s:%d Error!\n", __func__, __LINE__);
- }
-
- return 0;
+ return ret;
}
static int
afunc_get_alt(struct usb_function *fn, unsigned intf)
{
- struct audio_dev *agdev = func_to_agdev(fn);
- struct snd_uac2_chip *uac2 = &agdev->uac2;
-
- if (intf == agdev->ac_intf)
- return agdev->ac_alt;
- else if (intf == agdev->as_out_intf)
- return agdev->as_out_alt;
- else if (intf == agdev->as_in_intf)
- return agdev->as_in_alt;
+ struct f_uac2 *uac2 = func_to_uac2(fn);
+ struct g_audio *agdev = func_to_g_audio(fn);
+
+ if (intf == uac2->ac_intf)
+ return uac2->ac_alt;
+ else if (intf == uac2->as_out_intf)
+ return uac2->as_out_alt;
+ else if (intf == uac2->as_in_intf)
+ return uac2->as_in_alt;
else
- dev_err(&uac2->pdev.dev,
+ dev_err(&agdev->gadget->dev,
"%s:%d Invalid Interface %d!\n",
__func__, __LINE__, intf);
static void
afunc_disable(struct usb_function *fn)
{
- struct audio_dev *agdev = func_to_agdev(fn);
- struct snd_uac2_chip *uac2 = &agdev->uac2;
-
- free_ep(&uac2->p_prm, agdev->in_ep);
- agdev->as_in_alt = 0;
+ struct f_uac2 *uac2 = func_to_uac2(fn);
- free_ep(&uac2->c_prm, agdev->out_ep);
- agdev->as_out_alt = 0;
+ uac2->as_in_alt = 0;
+ uac2->as_out_alt = 0;
+ u_audio_stop_capture(&uac2->g_audio);
+ u_audio_stop_playback(&uac2->g_audio);
}
static int
in_rq_cur(struct usb_function *fn, const struct usb_ctrlrequest *cr)
{
struct usb_request *req = fn->config->cdev->req;
- struct audio_dev *agdev = func_to_agdev(fn);
- struct snd_uac2_chip *uac2 = &agdev->uac2;
+ struct g_audio *agdev = func_to_g_audio(fn);
struct f_uac2_opts *opts;
u16 w_length = le16_to_cpu(cr->wLength);
u16 w_index = le16_to_cpu(cr->wIndex);
int value = -EOPNOTSUPP;
int p_srate, c_srate;
- opts = agdev_to_uac2_opts(agdev);
+ opts = g_audio_to_uac2_opts(agdev);
p_srate = opts->p_srate;
c_srate = opts->c_srate;
*(u8 *)req->buf = 1;
value = min_t(unsigned, w_length, 1);
} else {
- dev_err(&uac2->pdev.dev,
+ dev_err(&agdev->gadget->dev,
"%s:%d control_selector=%d TODO!\n",
__func__, __LINE__, control_selector);
}
in_rq_range(struct usb_function *fn, const struct usb_ctrlrequest *cr)
{
struct usb_request *req = fn->config->cdev->req;
- struct audio_dev *agdev = func_to_agdev(fn);
- struct snd_uac2_chip *uac2 = &agdev->uac2;
+ struct g_audio *agdev = func_to_g_audio(fn);
struct f_uac2_opts *opts;
u16 w_length = le16_to_cpu(cr->wLength);
u16 w_index = le16_to_cpu(cr->wIndex);
int value = -EOPNOTSUPP;
int p_srate, c_srate;
- opts = agdev_to_uac2_opts(agdev);
+ opts = g_audio_to_uac2_opts(agdev);
p_srate = opts->p_srate;
c_srate = opts->c_srate;
value = min_t(unsigned, w_length, sizeof r);
memcpy(req->buf, &r, value);
} else {
- dev_err(&uac2->pdev.dev,
+ dev_err(&agdev->gadget->dev,
"%s:%d control_selector=%d TODO!\n",
__func__, __LINE__, control_selector);
}
static int
setup_rq_inf(struct usb_function *fn, const struct usb_ctrlrequest *cr)
{
- struct audio_dev *agdev = func_to_agdev(fn);
- struct snd_uac2_chip *uac2 = &agdev->uac2;
+ struct f_uac2 *uac2 = func_to_uac2(fn);
+ struct g_audio *agdev = func_to_g_audio(fn);
u16 w_index = le16_to_cpu(cr->wIndex);
u8 intf = w_index & 0xff;
- if (intf != agdev->ac_intf) {
- dev_err(&uac2->pdev.dev,
+ if (intf != uac2->ac_intf) {
+ dev_err(&agdev->gadget->dev,
"%s:%d Error!\n", __func__, __LINE__);
return -EOPNOTSUPP;
}
afunc_setup(struct usb_function *fn, const struct usb_ctrlrequest *cr)
{
struct usb_composite_dev *cdev = fn->config->cdev;
- struct audio_dev *agdev = func_to_agdev(fn);
- struct snd_uac2_chip *uac2 = &agdev->uac2;
+ struct g_audio *agdev = func_to_g_audio(fn);
struct usb_request *req = cdev->req;
u16 w_length = le16_to_cpu(cr->wLength);
int value = -EOPNOTSUPP;
if ((cr->bRequestType & USB_RECIP_MASK) == USB_RECIP_INTERFACE)
value = setup_rq_inf(fn, cr);
else
- dev_err(&uac2->pdev.dev, "%s:%d Error!\n", __func__, __LINE__);
+ dev_err(&agdev->gadget->dev, "%s:%d Error!\n",
+ __func__, __LINE__);
if (value >= 0) {
req->length = value;
req->zero = value < w_length;
value = usb_ep_queue(cdev->gadget->ep0, req, GFP_ATOMIC);
if (value < 0) {
- dev_err(&uac2->pdev.dev,
+ dev_err(&agdev->gadget->dev,
"%s:%d Error!\n", __func__, __LINE__);
req->status = 0;
}
static void afunc_free(struct usb_function *f)
{
- struct audio_dev *agdev;
+ struct g_audio *agdev;
struct f_uac2_opts *opts;
- agdev = func_to_agdev(f);
+ agdev = func_to_g_audio(f);
opts = container_of(f->fi, struct f_uac2_opts, func_inst);
kfree(agdev);
mutex_lock(&opts->lock);
static void afunc_unbind(struct usb_configuration *c, struct usb_function *f)
{
- struct audio_dev *agdev = func_to_agdev(f);
- struct uac2_rtd_params *prm;
+ struct g_audio *agdev = func_to_g_audio(f);
- alsa_uac2_exit(agdev);
-
- prm = &agdev->uac2.p_prm;
- kfree(prm->rbuf);
-
- prm = &agdev->uac2.c_prm;
- kfree(prm->rbuf);
- kfree(prm->ureq);
+ g_audio_cleanup(agdev);
usb_free_all_descriptors(f);
+
+ agdev->gadget = NULL;
}
static struct usb_function *afunc_alloc(struct usb_function_instance *fi)
{
- struct audio_dev *agdev;
+ struct f_uac2 *uac2;
struct f_uac2_opts *opts;
- agdev = kzalloc(sizeof(*agdev), GFP_KERNEL);
- if (agdev == NULL)
+ uac2 = kzalloc(sizeof(*uac2), GFP_KERNEL);
+ if (uac2 == NULL)
return ERR_PTR(-ENOMEM);
opts = container_of(fi, struct f_uac2_opts, func_inst);
++opts->refcnt;
mutex_unlock(&opts->lock);
- agdev->func.name = "uac2_func";
- agdev->func.bind = afunc_bind;
- agdev->func.unbind = afunc_unbind;
- agdev->func.set_alt = afunc_set_alt;
- agdev->func.get_alt = afunc_get_alt;
- agdev->func.disable = afunc_disable;
- agdev->func.setup = afunc_setup;
- agdev->func.free_func = afunc_free;
+ uac2->g_audio.func.name = "uac2_func";
+ uac2->g_audio.func.bind = afunc_bind;
+ uac2->g_audio.func.unbind = afunc_unbind;
+ uac2->g_audio.func.set_alt = afunc_set_alt;
+ uac2->g_audio.func.get_alt = afunc_get_alt;
+ uac2->g_audio.func.disable = afunc_disable;
+ uac2->g_audio.func.setup = afunc_setup;
+ uac2->g_audio.func.free_func = afunc_free;
- return &agdev->func;
+ return &uac2->g_audio.func;
}
DECLARE_USB_FUNCTION_INIT(uac2, afunc_alloc_inst, afunc_alloc);
#define FSG_MAX_LUNS 16
enum fsg_buffer_state {
+ BUF_STATE_SENDING = -2,
+ BUF_STATE_RECEIVING,
BUF_STATE_EMPTY = 0,
- BUF_STATE_FULL,
- BUF_STATE_BUSY
+ BUF_STATE_FULL
};
struct fsg_buffhd {
unsigned int bulk_out_intended_length;
struct usb_request *inreq;
- int inreq_busy;
struct usb_request *outreq;
- int outreq_busy;
};
enum fsg_state {
- /* This one isn't used anywhere */
- FSG_STATE_COMMAND_PHASE = -10,
- FSG_STATE_DATA_PHASE,
- FSG_STATE_STATUS_PHASE,
-
- FSG_STATE_IDLE = 0,
+ FSG_STATE_NORMAL,
FSG_STATE_ABORT_BULK_OUT,
- FSG_STATE_RESET,
- FSG_STATE_INTERFACE_CHANGE,
+ FSG_STATE_PROTOCOL_RESET,
FSG_STATE_CONFIG_CHANGE,
- FSG_STATE_DISCONNECT,
FSG_STATE_EXIT,
FSG_STATE_TERMINATED
};
--- /dev/null
+/*
+ * u_audio.c -- interface to USB gadget "ALSA sound card" utilities
+ *
+ * Copyright (C) 2016
+ * Author: Ruslan Bilovol <ruslan.bilovol@gmail.com>
+ *
+ * Sound card implementation was cut-and-pasted with changes
+ * from f_uac2.c and has:
+ * Copyright (C) 2011
+ * Yadwinder Singh (yadi.brar01@gmail.com)
+ * Jaswinder Singh (jaswinder.singh@linaro.org)
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/module.h>
+#include <sound/core.h>
+#include <sound/pcm.h>
+#include <sound/pcm_params.h>
+
+#include "u_audio.h"
+
+#define BUFF_SIZE_MAX (PAGE_SIZE * 16)
+#define PRD_SIZE_MAX PAGE_SIZE
+#define MIN_PERIODS 4
+
+struct uac_req {
+ struct uac_rtd_params *pp; /* parent param */
+ struct usb_request *req;
+};
+
+/* Runtime data params for one stream */
+struct uac_rtd_params {
+ struct snd_uac_chip *uac; /* parent chip */
+ bool ep_enabled; /* if the ep is enabled */
+ /* Size of the ring buffer */
+ size_t dma_bytes;
+ unsigned char *dma_area;
+
+ struct snd_pcm_substream *ss;
+
+ /* Ring buffer */
+ ssize_t hw_ptr;
+
+ void *rbuf;
+
+ size_t period_size;
+
+ unsigned max_psize; /* MaxPacketSize of endpoint */
+ struct uac_req *ureq;
+
+ spinlock_t lock;
+};
+
+struct snd_uac_chip {
+ struct g_audio *audio_dev;
+
+ struct uac_rtd_params p_prm;
+ struct uac_rtd_params c_prm;
+
+ struct snd_card *card;
+ struct snd_pcm *pcm;
+
+ /* timekeeping for the playback endpoint */
+ unsigned int p_interval;
+ unsigned int p_residue;
+
+ /* pre-calculated values for playback iso completion */
+ unsigned int p_pktsize;
+ unsigned int p_pktsize_residue;
+ unsigned int p_framesize;
+};
+
+static struct snd_pcm_hardware uac_pcm_hardware = {
+ .info = SNDRV_PCM_INFO_INTERLEAVED | SNDRV_PCM_INFO_BLOCK_TRANSFER
+ | SNDRV_PCM_INFO_MMAP | SNDRV_PCM_INFO_MMAP_VALID
+ | SNDRV_PCM_INFO_PAUSE | SNDRV_PCM_INFO_RESUME,
+ .rates = SNDRV_PCM_RATE_CONTINUOUS,
+ .periods_max = BUFF_SIZE_MAX / PRD_SIZE_MAX,
+ .buffer_bytes_max = BUFF_SIZE_MAX,
+ .period_bytes_max = PRD_SIZE_MAX,
+ .periods_min = MIN_PERIODS,
+};
+
+static void u_audio_iso_complete(struct usb_ep *ep, struct usb_request *req)
+{
+ unsigned pending;
+ unsigned long flags;
+ unsigned int hw_ptr;
+ bool update_alsa = false;
+ int status = req->status;
+ struct uac_req *ur = req->context;
+ struct snd_pcm_substream *substream;
+ struct uac_rtd_params *prm = ur->pp;
+ struct snd_uac_chip *uac = prm->uac;
+
+ /* i/f shutting down */
+ if (!prm->ep_enabled || req->status == -ESHUTDOWN)
+ return;
+
+ /*
+ * We can't really do much about bad xfers.
+ * Afterall, the ISOCH xfers could fail legitimately.
+ */
+ if (status)
+ pr_debug("%s: iso_complete status(%d) %d/%d\n",
+ __func__, status, req->actual, req->length);
+
+ substream = prm->ss;
+
+ /* Do nothing if ALSA isn't active */
+ if (!substream)
+ goto exit;
+
+ spin_lock_irqsave(&prm->lock, flags);
+
+ if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK) {
+ /*
+ * For each IN packet, take the quotient of the current data
+ * rate and the endpoint's interval as the base packet size.
+ * If there is a residue from this division, add it to the
+ * residue accumulator.
+ */
+ req->length = uac->p_pktsize;
+ uac->p_residue += uac->p_pktsize_residue;
+
+ /*
+ * Whenever there are more bytes in the accumulator than we
+ * need to add one more sample frame, increase this packet's
+ * size and decrease the accumulator.
+ */
+ if (uac->p_residue / uac->p_interval >= uac->p_framesize) {
+ req->length += uac->p_framesize;
+ uac->p_residue -= uac->p_framesize *
+ uac->p_interval;
+ }
+
+ req->actual = req->length;
+ }
+
+ pending = prm->hw_ptr % prm->period_size;
+ pending += req->actual;
+ if (pending >= prm->period_size)
+ update_alsa = true;
+
+ hw_ptr = prm->hw_ptr;
+ prm->hw_ptr = (prm->hw_ptr + req->actual) % prm->dma_bytes;
+
+ spin_unlock_irqrestore(&prm->lock, flags);
+
+ /* Pack USB load in ALSA ring buffer */
+ pending = prm->dma_bytes - hw_ptr;
+
+ if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK) {
+ if (unlikely(pending < req->actual)) {
+ memcpy(req->buf, prm->dma_area + hw_ptr, pending);
+ memcpy(req->buf + pending, prm->dma_area,
+ req->actual - pending);
+ } else {
+ memcpy(req->buf, prm->dma_area + hw_ptr, req->actual);
+ }
+ } else {
+ if (unlikely(pending < req->actual)) {
+ memcpy(prm->dma_area + hw_ptr, req->buf, pending);
+ memcpy(prm->dma_area, req->buf + pending,
+ req->actual - pending);
+ } else {
+ memcpy(prm->dma_area + hw_ptr, req->buf, req->actual);
+ }
+ }
+
+exit:
+ if (usb_ep_queue(ep, req, GFP_ATOMIC))
+ dev_err(uac->card->dev, "%d Error!\n", __LINE__);
+
+ if (update_alsa)
+ snd_pcm_period_elapsed(substream);
+}
+
+static int uac_pcm_trigger(struct snd_pcm_substream *substream, int cmd)
+{
+ struct snd_uac_chip *uac = snd_pcm_substream_chip(substream);
+ struct uac_rtd_params *prm;
+ struct g_audio *audio_dev;
+ struct uac_params *params;
+ unsigned long flags;
+ int err = 0;
+
+ audio_dev = uac->audio_dev;
+ params = &audio_dev->params;
+
+ if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK)
+ prm = &uac->p_prm;
+ else
+ prm = &uac->c_prm;
+
+ spin_lock_irqsave(&prm->lock, flags);
+
+ /* Reset */
+ prm->hw_ptr = 0;
+
+ switch (cmd) {
+ case SNDRV_PCM_TRIGGER_START:
+ case SNDRV_PCM_TRIGGER_RESUME:
+ prm->ss = substream;
+ break;
+ case SNDRV_PCM_TRIGGER_STOP:
+ case SNDRV_PCM_TRIGGER_SUSPEND:
+ prm->ss = NULL;
+ break;
+ default:
+ err = -EINVAL;
+ }
+
+ spin_unlock_irqrestore(&prm->lock, flags);
+
+ /* Clear buffer after Play stops */
+ if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK && !prm->ss)
+ memset(prm->rbuf, 0, prm->max_psize * params->req_number);
+
+ return err;
+}
+
+static snd_pcm_uframes_t uac_pcm_pointer(struct snd_pcm_substream *substream)
+{
+ struct snd_uac_chip *uac = snd_pcm_substream_chip(substream);
+ struct uac_rtd_params *prm;
+
+ if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK)
+ prm = &uac->p_prm;
+ else
+ prm = &uac->c_prm;
+
+ return bytes_to_frames(substream->runtime, prm->hw_ptr);
+}
+
+static int uac_pcm_hw_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *hw_params)
+{
+ struct snd_uac_chip *uac = snd_pcm_substream_chip(substream);
+ struct uac_rtd_params *prm;
+ int err;
+
+ if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK)
+ prm = &uac->p_prm;
+ else
+ prm = &uac->c_prm;
+
+ err = snd_pcm_lib_malloc_pages(substream,
+ params_buffer_bytes(hw_params));
+ if (err >= 0) {
+ prm->dma_bytes = substream->runtime->dma_bytes;
+ prm->dma_area = substream->runtime->dma_area;
+ prm->period_size = params_period_bytes(hw_params);
+ }
+
+ return err;
+}
+
+static int uac_pcm_hw_free(struct snd_pcm_substream *substream)
+{
+ struct snd_uac_chip *uac = snd_pcm_substream_chip(substream);
+ struct uac_rtd_params *prm;
+
+ if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK)
+ prm = &uac->p_prm;
+ else
+ prm = &uac->c_prm;
+
+ prm->dma_area = NULL;
+ prm->dma_bytes = 0;
+ prm->period_size = 0;
+
+ return snd_pcm_lib_free_pages(substream);
+}
+
+static int uac_pcm_open(struct snd_pcm_substream *substream)
+{
+ struct snd_uac_chip *uac = snd_pcm_substream_chip(substream);
+ struct snd_pcm_runtime *runtime = substream->runtime;
+ struct g_audio *audio_dev;
+ struct uac_params *params;
+ int p_ssize, c_ssize;
+ int p_srate, c_srate;
+ int p_chmask, c_chmask;
+
+ audio_dev = uac->audio_dev;
+ params = &audio_dev->params;
+ p_ssize = params->p_ssize;
+ c_ssize = params->c_ssize;
+ p_srate = params->p_srate;
+ c_srate = params->c_srate;
+ p_chmask = params->p_chmask;
+ c_chmask = params->c_chmask;
+ uac->p_residue = 0;
+
+ runtime->hw = uac_pcm_hardware;
+
+ if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK) {
+ spin_lock_init(&uac->p_prm.lock);
+ runtime->hw.rate_min = p_srate;
+ switch (p_ssize) {
+ case 3:
+ runtime->hw.formats = SNDRV_PCM_FMTBIT_S24_3LE;
+ break;
+ case 4:
+ runtime->hw.formats = SNDRV_PCM_FMTBIT_S32_LE;
+ break;
+ default:
+ runtime->hw.formats = SNDRV_PCM_FMTBIT_S16_LE;
+ break;
+ }
+ runtime->hw.channels_min = num_channels(p_chmask);
+ runtime->hw.period_bytes_min = 2 * uac->p_prm.max_psize
+ / runtime->hw.periods_min;
+ } else {
+ spin_lock_init(&uac->c_prm.lock);
+ runtime->hw.rate_min = c_srate;
+ switch (c_ssize) {
+ case 3:
+ runtime->hw.formats = SNDRV_PCM_FMTBIT_S24_3LE;
+ break;
+ case 4:
+ runtime->hw.formats = SNDRV_PCM_FMTBIT_S32_LE;
+ break;
+ default:
+ runtime->hw.formats = SNDRV_PCM_FMTBIT_S16_LE;
+ break;
+ }
+ runtime->hw.channels_min = num_channels(c_chmask);
+ runtime->hw.period_bytes_min = 2 * uac->c_prm.max_psize
+ / runtime->hw.periods_min;
+ }
+
+ runtime->hw.rate_max = runtime->hw.rate_min;
+ runtime->hw.channels_max = runtime->hw.channels_min;
+
+ snd_pcm_hw_constraint_integer(runtime, SNDRV_PCM_HW_PARAM_PERIODS);
+
+ return 0;
+}
+
+/* ALSA cries without these function pointers */
+static int uac_pcm_null(struct snd_pcm_substream *substream)
+{
+ return 0;
+}
+
+static struct snd_pcm_ops uac_pcm_ops = {
+ .open = uac_pcm_open,
+ .close = uac_pcm_null,
+ .ioctl = snd_pcm_lib_ioctl,
+ .hw_params = uac_pcm_hw_params,
+ .hw_free = uac_pcm_hw_free,
+ .trigger = uac_pcm_trigger,
+ .pointer = uac_pcm_pointer,
+ .prepare = uac_pcm_null,
+};
+
+static inline void free_ep(struct uac_rtd_params *prm, struct usb_ep *ep)
+{
+ struct snd_uac_chip *uac = prm->uac;
+ struct g_audio *audio_dev;
+ struct uac_params *params;
+ int i;
+
+ if (!prm->ep_enabled)
+ return;
+
+ prm->ep_enabled = false;
+
+ audio_dev = uac->audio_dev;
+ params = &audio_dev->params;
+
+ for (i = 0; i < params->req_number; i++) {
+ if (prm->ureq[i].req) {
+ usb_ep_dequeue(ep, prm->ureq[i].req);
+ usb_ep_free_request(ep, prm->ureq[i].req);
+ prm->ureq[i].req = NULL;
+ }
+ }
+
+ if (usb_ep_disable(ep))
+ dev_err(uac->card->dev, "%s:%d Error!\n", __func__, __LINE__);
+}
+
+
+int u_audio_start_capture(struct g_audio *audio_dev)
+{
+ struct snd_uac_chip *uac = audio_dev->uac;
+ struct usb_gadget *gadget = audio_dev->gadget;
+ struct device *dev = &gadget->dev;
+ struct usb_request *req;
+ struct usb_ep *ep;
+ struct uac_rtd_params *prm;
+ struct uac_params *params = &audio_dev->params;
+ int req_len, i;
+
+ ep = audio_dev->out_ep;
+ prm = &uac->c_prm;
+ config_ep_by_speed(gadget, &audio_dev->func, ep);
+ req_len = prm->max_psize;
+
+ prm->ep_enabled = true;
+ usb_ep_enable(ep);
+
+ for (i = 0; i < params->req_number; i++) {
+ if (!prm->ureq[i].req) {
+ req = usb_ep_alloc_request(ep, GFP_ATOMIC);
+ if (req == NULL)
+ return -ENOMEM;
+
+ prm->ureq[i].req = req;
+ prm->ureq[i].pp = prm;
+
+ req->zero = 0;
+ req->context = &prm->ureq[i];
+ req->length = req_len;
+ req->complete = u_audio_iso_complete;
+ req->buf = prm->rbuf + i * prm->max_psize;
+ }
+
+ if (usb_ep_queue(ep, prm->ureq[i].req, GFP_ATOMIC))
+ dev_err(dev, "%s:%d Error!\n", __func__, __LINE__);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(u_audio_start_capture);
+
+void u_audio_stop_capture(struct g_audio *audio_dev)
+{
+ struct snd_uac_chip *uac = audio_dev->uac;
+
+ free_ep(&uac->c_prm, audio_dev->out_ep);
+}
+EXPORT_SYMBOL_GPL(u_audio_stop_capture);
+
+int u_audio_start_playback(struct g_audio *audio_dev)
+{
+ struct snd_uac_chip *uac = audio_dev->uac;
+ struct usb_gadget *gadget = audio_dev->gadget;
+ struct device *dev = &gadget->dev;
+ struct usb_request *req;
+ struct usb_ep *ep;
+ struct uac_rtd_params *prm;
+ struct uac_params *params = &audio_dev->params;
+ unsigned int factor, rate;
+ const struct usb_endpoint_descriptor *ep_desc;
+ int req_len, i;
+
+ ep = audio_dev->in_ep;
+ prm = &uac->p_prm;
+ config_ep_by_speed(gadget, &audio_dev->func, ep);
+
+ ep_desc = ep->desc;
+
+ /* pre-calculate the playback endpoint's interval */
+ if (gadget->speed == USB_SPEED_FULL)
+ factor = 1000;
+ else
+ factor = 8000;
+
+ /* pre-compute some values for iso_complete() */
+ uac->p_framesize = params->p_ssize *
+ num_channels(params->p_chmask);
+ rate = params->p_srate * uac->p_framesize;
+ uac->p_interval = factor / (1 << (ep_desc->bInterval - 1));
+ uac->p_pktsize = min_t(unsigned int, rate / uac->p_interval,
+ prm->max_psize);
+
+ if (uac->p_pktsize < prm->max_psize)
+ uac->p_pktsize_residue = rate % uac->p_interval;
+ else
+ uac->p_pktsize_residue = 0;
+
+ req_len = uac->p_pktsize;
+ uac->p_residue = 0;
+
+ prm->ep_enabled = true;
+ usb_ep_enable(ep);
+
+ for (i = 0; i < params->req_number; i++) {
+ if (!prm->ureq[i].req) {
+ req = usb_ep_alloc_request(ep, GFP_ATOMIC);
+ if (req == NULL)
+ return -ENOMEM;
+
+ prm->ureq[i].req = req;
+ prm->ureq[i].pp = prm;
+
+ req->zero = 0;
+ req->context = &prm->ureq[i];
+ req->length = req_len;
+ req->complete = u_audio_iso_complete;
+ req->buf = prm->rbuf + i * prm->max_psize;
+ }
+
+ if (usb_ep_queue(ep, prm->ureq[i].req, GFP_ATOMIC))
+ dev_err(dev, "%s:%d Error!\n", __func__, __LINE__);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(u_audio_start_playback);
+
+void u_audio_stop_playback(struct g_audio *audio_dev)
+{
+ struct snd_uac_chip *uac = audio_dev->uac;
+
+ free_ep(&uac->p_prm, audio_dev->in_ep);
+}
+EXPORT_SYMBOL_GPL(u_audio_stop_playback);
+
+int g_audio_setup(struct g_audio *g_audio, const char *pcm_name,
+ const char *card_name)
+{
+ struct snd_uac_chip *uac;
+ struct snd_card *card;
+ struct snd_pcm *pcm;
+ struct uac_params *params;
+ int p_chmask, c_chmask;
+ int err;
+
+ if (!g_audio)
+ return -EINVAL;
+
+ uac = kzalloc(sizeof(*uac), GFP_KERNEL);
+ if (!uac)
+ return -ENOMEM;
+ g_audio->uac = uac;
+ uac->audio_dev = g_audio;
+
+ params = &g_audio->params;
+ p_chmask = params->p_chmask;
+ c_chmask = params->c_chmask;
+
+ if (c_chmask) {
+ struct uac_rtd_params *prm = &uac->c_prm;
+
+ uac->c_prm.uac = uac;
+ prm->max_psize = g_audio->out_ep_maxpsize;
+
+ prm->ureq = kcalloc(params->req_number, sizeof(struct uac_req),
+ GFP_KERNEL);
+ if (!prm->ureq) {
+ err = -ENOMEM;
+ goto fail;
+ }
+
+ prm->rbuf = kcalloc(params->req_number, prm->max_psize,
+ GFP_KERNEL);
+ if (!prm->rbuf) {
+ prm->max_psize = 0;
+ err = -ENOMEM;
+ goto fail;
+ }
+ }
+
+ if (p_chmask) {
+ struct uac_rtd_params *prm = &uac->p_prm;
+
+ uac->p_prm.uac = uac;
+ prm->max_psize = g_audio->in_ep_maxpsize;
+
+ prm->ureq = kcalloc(params->req_number, sizeof(struct uac_req),
+ GFP_KERNEL);
+ if (!prm->ureq) {
+ err = -ENOMEM;
+ goto fail;
+ }
+
+ prm->rbuf = kcalloc(params->req_number, prm->max_psize,
+ GFP_KERNEL);
+ if (!prm->rbuf) {
+ prm->max_psize = 0;
+ err = -ENOMEM;
+ goto fail;
+ }
+ }
+
+ /* Choose any slot, with no id */
+ err = snd_card_new(&g_audio->gadget->dev,
+ -1, NULL, THIS_MODULE, 0, &card);
+ if (err < 0)
+ goto fail;
+
+ uac->card = card;
+
+ /*
+ * Create first PCM device
+ * Create a substream only for non-zero channel streams
+ */
+ err = snd_pcm_new(uac->card, pcm_name, 0,
+ p_chmask ? 1 : 0, c_chmask ? 1 : 0, &pcm);
+ if (err < 0)
+ goto snd_fail;
+
+ strcpy(pcm->name, pcm_name);
+ pcm->private_data = uac;
+ uac->pcm = pcm;
+
+ snd_pcm_set_ops(pcm, SNDRV_PCM_STREAM_PLAYBACK, &uac_pcm_ops);
+ snd_pcm_set_ops(pcm, SNDRV_PCM_STREAM_CAPTURE, &uac_pcm_ops);
+
+ strcpy(card->driver, card_name);
+ strcpy(card->shortname, card_name);
+ sprintf(card->longname, "%s %i", card_name, card->dev->id);
+
+ snd_pcm_lib_preallocate_pages_for_all(pcm, SNDRV_DMA_TYPE_CONTINUOUS,
+ snd_dma_continuous_data(GFP_KERNEL), 0, BUFF_SIZE_MAX);
+
+ err = snd_card_register(card);
+
+ if (!err)
+ return 0;
+
+snd_fail:
+ snd_card_free(card);
+fail:
+ kfree(uac->p_prm.ureq);
+ kfree(uac->c_prm.ureq);
+ kfree(uac->p_prm.rbuf);
+ kfree(uac->c_prm.rbuf);
+ kfree(uac);
+
+ return err;
+}
+EXPORT_SYMBOL_GPL(g_audio_setup);
+
+void g_audio_cleanup(struct g_audio *g_audio)
+{
+ struct snd_uac_chip *uac;
+ struct snd_card *card;
+
+ if (!g_audio || !g_audio->uac)
+ return;
+
+ uac = g_audio->uac;
+ card = uac->card;
+ if (card)
+ snd_card_free(card);
+
+ kfree(uac->p_prm.ureq);
+ kfree(uac->c_prm.ureq);
+ kfree(uac->p_prm.rbuf);
+ kfree(uac->c_prm.rbuf);
+ kfree(uac);
+}
+EXPORT_SYMBOL_GPL(g_audio_cleanup);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("USB gadget \"ALSA sound card\" utilities");
+MODULE_AUTHOR("Ruslan Bilovol");
--- /dev/null
+/*
+ * u_audio.h -- interface to USB gadget "ALSA sound card" utilities
+ *
+ * Copyright (C) 2016
+ * Author: Ruslan Bilovol <ruslan.bilovol@gmail.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#ifndef __U_AUDIO_H
+#define __U_AUDIO_H
+
+#include <linux/usb/composite.h>
+
+struct uac_params {
+ /* playback */
+ int p_chmask; /* channel mask */
+ int p_srate; /* rate in Hz */
+ int p_ssize; /* sample size */
+
+ /* capture */
+ int c_chmask; /* channel mask */
+ int c_srate; /* rate in Hz */
+ int c_ssize; /* sample size */
+
+ int req_number; /* number of preallocated requests */
+};
+
+struct g_audio {
+ struct usb_function func;
+ struct usb_gadget *gadget;
+
+ struct usb_ep *in_ep;
+ struct usb_ep *out_ep;
+
+ /* Max packet size for all in_ep possible speeds */
+ unsigned int in_ep_maxpsize;
+ /* Max packet size for all out_ep possible speeds */
+ unsigned int out_ep_maxpsize;
+
+ /* The ALSA Sound Card it represents on the USB-Client side */
+ struct snd_uac_chip *uac;
+
+ struct uac_params params;
+};
+
+static inline struct g_audio *func_to_g_audio(struct usb_function *f)
+{
+ return container_of(f, struct g_audio, func);
+}
+
+static inline uint num_channels(uint chanmask)
+{
+ uint num = 0;
+
+ while (chanmask) {
+ num += (chanmask & 1);
+ chanmask >>= 1;
+ }
+
+ return num;
+}
+
+/*
+ * g_audio_setup - initialize one virtual ALSA sound card
+ * @g_audio: struct with filled params, in_ep_maxpsize, out_ep_maxpsize
+ * @pcm_name: the id string for a PCM instance of this sound card
+ * @card_name: name of this soundcard
+ *
+ * This sets up the single virtual ALSA sound card that may be exported by a
+ * gadget driver using this framework.
+ *
+ * Context: may sleep
+ *
+ * Returns zero on success, or a negative error on failure.
+ */
+int g_audio_setup(struct g_audio *g_audio, const char *pcm_name,
+ const char *card_name);
+void g_audio_cleanup(struct g_audio *g_audio);
+
+int u_audio_start_capture(struct g_audio *g_audio);
+void u_audio_stop_capture(struct g_audio *g_audio);
+int u_audio_start_playback(struct g_audio *g_audio);
+void u_audio_stop_playback(struct g_audio *g_audio);
+
+#endif /* __U_AUDIO_H */
#define FFS_FL_CALL_CLOSED_CALLBACK 0
#define FFS_FL_BOUND 1
+ /* For waking up blocked threads when function is enabled. */
+ wait_queue_head_t wait;
+
/* Active function */
struct ffs_function *func;
/*
- * u_uac1.h -- interface to USB gadget "ALSA AUDIO" utilities
+ * u_uac1.h - Utility definitions for UAC1 function
*
- * Copyright (C) 2008 Bryan Wu <cooloney@kernel.org>
- * Copyright (C) 2008 Analog Devices, Inc
+ * Copyright (C) 2016 Ruslan Bilovol <ruslan.bilovol@gmail.com>
*
- * Enter bugs at http://blackfin.uclinux.org/
- *
- * Licensed under the GPL-2 or later.
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
*/
-#ifndef __U_AUDIO_H
-#define __U_AUDIO_H
+#ifndef __U_UAC1_H
+#define __U_UAC1_H
-#include <linux/device.h>
-#include <linux/err.h>
-#include <linux/usb/audio.h>
#include <linux/usb/composite.h>
-#include <sound/core.h>
-#include <sound/pcm.h>
-#include <sound/pcm_params.h>
-
-#define FILE_PCM_PLAYBACK "/dev/snd/pcmC0D0p"
-#define FILE_PCM_CAPTURE "/dev/snd/pcmC0D0c"
-#define FILE_CONTROL "/dev/snd/controlC0"
-
#define UAC1_OUT_EP_MAX_PACKET_SIZE 200
-#define UAC1_REQ_COUNT 256
-#define UAC1_AUDIO_BUF_SIZE 48000
-
-/*
- * This represents the USB side of an audio card device, managed by a USB
- * function which provides control and stream interfaces.
- */
-
-struct gaudio_snd_dev {
- struct gaudio *card;
- struct file *filp;
- struct snd_pcm_substream *substream;
- int access;
- int format;
- int channels;
- int rate;
-};
-
-struct gaudio {
- struct usb_function func;
- struct usb_gadget *gadget;
+#define UAC1_DEF_CCHMASK 0x3
+#define UAC1_DEF_CSRATE 48000
+#define UAC1_DEF_CSSIZE 2
+#define UAC1_DEF_PCHMASK 0x3
+#define UAC1_DEF_PSRATE 48000
+#define UAC1_DEF_PSSIZE 2
+#define UAC1_DEF_REQ_NUM 2
- /* ALSA sound device interfaces */
- struct gaudio_snd_dev control;
- struct gaudio_snd_dev playback;
- struct gaudio_snd_dev capture;
-
- /* TODO */
-};
struct f_uac1_opts {
struct usb_function_instance func_inst;
- int req_buf_size;
- int req_count;
- int audio_buf_size;
- char *fn_play;
- char *fn_cap;
- char *fn_cntl;
+ int c_chmask;
+ int c_srate;
+ int c_ssize;
+ int p_chmask;
+ int p_srate;
+ int p_ssize;
+ int req_number;
unsigned bound:1;
- unsigned fn_play_alloc:1;
- unsigned fn_cap_alloc:1;
- unsigned fn_cntl_alloc:1;
+
struct mutex lock;
int refcnt;
};
-int gaudio_setup(struct gaudio *card);
-void gaudio_cleanup(struct gaudio *the_card);
-
-size_t u_audio_playback(struct gaudio *card, void *buf, size_t count);
-int u_audio_get_playback_channels(struct gaudio *card);
-int u_audio_get_playback_rate(struct gaudio *card);
-
-#endif /* __U_AUDIO_H */
+#endif /* __U_UAC1_H */
#include <linux/random.h>
#include <linux/syscalls.h>
-#include "u_uac1.h"
+#include "u_uac1_legacy.h"
/*
* This component encapsulates the ALSA devices for USB audio gadget
{
struct snd_pcm_file *pcm_file;
struct gaudio_snd_dev *snd;
- struct f_uac1_opts *opts;
+ struct f_uac1_legacy_opts *opts;
char *fn_play, *fn_cap, *fn_cntl;
- opts = container_of(card->func.fi, struct f_uac1_opts, func_inst);
+ opts = container_of(card->func.fi, struct f_uac1_legacy_opts,
+ func_inst);
fn_play = opts->fn_play;
fn_cap = opts->fn_cap;
fn_cntl = opts->fn_cntl;
--- /dev/null
+/*
+ * u_uac1.h -- interface to USB gadget "ALSA AUDIO" utilities
+ *
+ * Copyright (C) 2008 Bryan Wu <cooloney@kernel.org>
+ * Copyright (C) 2008 Analog Devices, Inc
+ *
+ * Enter bugs at http://blackfin.uclinux.org/
+ *
+ * Licensed under the GPL-2 or later.
+ */
+
+#ifndef __U_UAC1_LEGACY_H
+#define __U_UAC1_LEGACY_H
+
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/usb/audio.h>
+#include <linux/usb/composite.h>
+
+#include <sound/core.h>
+#include <sound/pcm.h>
+#include <sound/pcm_params.h>
+
+#define FILE_PCM_PLAYBACK "/dev/snd/pcmC0D0p"
+#define FILE_PCM_CAPTURE "/dev/snd/pcmC0D0c"
+#define FILE_CONTROL "/dev/snd/controlC0"
+
+#define UAC1_OUT_EP_MAX_PACKET_SIZE 200
+#define UAC1_REQ_COUNT 256
+#define UAC1_AUDIO_BUF_SIZE 48000
+
+/*
+ * This represents the USB side of an audio card device, managed by a USB
+ * function which provides control and stream interfaces.
+ */
+
+struct gaudio_snd_dev {
+ struct gaudio *card;
+ struct file *filp;
+ struct snd_pcm_substream *substream;
+ int access;
+ int format;
+ int channels;
+ int rate;
+};
+
+struct gaudio {
+ struct usb_function func;
+ struct usb_gadget *gadget;
+
+ /* ALSA sound device interfaces */
+ struct gaudio_snd_dev control;
+ struct gaudio_snd_dev playback;
+ struct gaudio_snd_dev capture;
+
+ /* TODO */
+};
+
+struct f_uac1_legacy_opts {
+ struct usb_function_instance func_inst;
+ int req_buf_size;
+ int req_count;
+ int audio_buf_size;
+ char *fn_play;
+ char *fn_cap;
+ char *fn_cntl;
+ unsigned bound:1;
+ unsigned fn_play_alloc:1;
+ unsigned fn_cap_alloc:1;
+ unsigned fn_cntl_alloc:1;
+ struct mutex lock;
+ int refcnt;
+};
+
+int gaudio_setup(struct gaudio *card);
+void gaudio_cleanup(struct gaudio *the_card);
+
+size_t u_audio_playback(struct gaudio *card, void *buf, size_t count);
+int u_audio_get_playback_channels(struct gaudio *card);
+int u_audio_get_playback_rate(struct gaudio *card);
+
+#endif /* __U_UAC1_LEGACY_H */
depends on SND
select USB_LIBCOMPOSITE
select SND_PCM
- select USB_F_UAC1 if GADGET_UAC1
+ select USB_F_UAC1 if (GADGET_UAC1 && !GADGET_UAC1_LEGACY)
+ select USB_F_UAC1_LEGACY if (GADGET_UAC1 && GADGET_UAC1_LEGACY)
select USB_F_UAC2 if !GADGET_UAC1
+ select USB_U_AUDIO if (USB_F_UAC2 || USB_F_UAC1)
help
This Gadget Audio driver is compatible with USB Audio Class
specification 2.0. It implements 1 AudioControl interface,
dynamically linked module called "g_audio".
config GADGET_UAC1
- bool "UAC 1.0 (Legacy)"
+ bool "UAC 1.0"
depends on USB_AUDIO
help
- If you instead want older UAC Spec-1.0 driver that also has audio
+ If you instead want older USB Audio Class specification 1.0 support
+ with similar driver capabilities.
+
+config GADGET_UAC1_LEGACY
+ bool "UAC 1.0 (Legacy)"
+ depends on GADGET_UAC1
+ help
+ If you instead want legacy UAC Spec-1.0 driver that also has audio
paths hardwired to the Audio codec chip on-board and doesn't work
without one.
module_param(c_ssize, uint, S_IRUGO);
MODULE_PARM_DESC(c_ssize, "Capture Sample Size(bytes)");
#else
+#ifndef CONFIG_GADGET_UAC1_LEGACY
#include "u_uac1.h"
+/* Playback(USB-IN) Default Stereo - Fl/Fr */
+static int p_chmask = UAC1_DEF_PCHMASK;
+module_param(p_chmask, uint, S_IRUGO);
+MODULE_PARM_DESC(p_chmask, "Playback Channel Mask");
+
+/* Playback Default 48 KHz */
+static int p_srate = UAC1_DEF_PSRATE;
+module_param(p_srate, uint, S_IRUGO);
+MODULE_PARM_DESC(p_srate, "Playback Sampling Rate");
+
+/* Playback Default 16bits/sample */
+static int p_ssize = UAC1_DEF_PSSIZE;
+module_param(p_ssize, uint, S_IRUGO);
+MODULE_PARM_DESC(p_ssize, "Playback Sample Size(bytes)");
+
+/* Capture(USB-OUT) Default Stereo - Fl/Fr */
+static int c_chmask = UAC1_DEF_CCHMASK;
+module_param(c_chmask, uint, S_IRUGO);
+MODULE_PARM_DESC(c_chmask, "Capture Channel Mask");
+
+/* Capture Default 48 KHz */
+static int c_srate = UAC1_DEF_CSRATE;
+module_param(c_srate, uint, S_IRUGO);
+MODULE_PARM_DESC(c_srate, "Capture Sampling Rate");
+
+/* Capture Default 16bits/sample */
+static int c_ssize = UAC1_DEF_CSSIZE;
+module_param(c_ssize, uint, S_IRUGO);
+MODULE_PARM_DESC(c_ssize, "Capture Sample Size(bytes)");
+#else /* CONFIG_GADGET_UAC1_LEGACY */
+#include "u_uac1_legacy.h"
+
static char *fn_play = FILE_PCM_PLAYBACK;
module_param(fn_play, charp, S_IRUGO);
MODULE_PARM_DESC(fn_play, "Playback PCM device file name");
static int audio_buf_size = UAC1_AUDIO_BUF_SIZE;
module_param(audio_buf_size, int, S_IRUGO);
MODULE_PARM_DESC(audio_buf_size, "Audio buffer size");
+#endif /* CONFIG_GADGET_UAC1_LEGACY */
#endif
/* string IDs are assigned dynamically */
/* .bcdUSB = DYNAMIC */
-#ifdef CONFIG_GADGET_UAC1
+#ifdef CONFIG_GADGET_UAC1_LEGACY
.bDeviceClass = USB_CLASS_PER_INTERFACE,
.bDeviceSubClass = 0,
.bDeviceProtocol = 0,
#ifndef CONFIG_GADGET_UAC1
struct f_uac2_opts *uac2_opts;
#else
+#ifndef CONFIG_GADGET_UAC1_LEGACY
struct f_uac1_opts *uac1_opts;
+#else
+ struct f_uac1_legacy_opts *uac1_opts;
+#endif
#endif
int status;
if (IS_ERR(fi_uac2))
return PTR_ERR(fi_uac2);
#else
+#ifndef CONFIG_GADGET_UAC1_LEGACY
fi_uac1 = usb_get_function_instance("uac1");
+#else
+ fi_uac1 = usb_get_function_instance("uac1_legacy");
+#endif
if (IS_ERR(fi_uac1))
return PTR_ERR(fi_uac1);
#endif
uac2_opts->c_ssize = c_ssize;
uac2_opts->req_number = UAC2_DEF_REQ_NUM;
#else
+#ifndef CONFIG_GADGET_UAC1_LEGACY
uac1_opts = container_of(fi_uac1, struct f_uac1_opts, func_inst);
+ uac1_opts->p_chmask = p_chmask;
+ uac1_opts->p_srate = p_srate;
+ uac1_opts->p_ssize = p_ssize;
+ uac1_opts->c_chmask = c_chmask;
+ uac1_opts->c_srate = c_srate;
+ uac1_opts->c_ssize = c_ssize;
+ uac1_opts->req_number = UAC1_DEF_REQ_NUM;
+#else /* CONFIG_GADGET_UAC1_LEGACY */
+ uac1_opts = container_of(fi_uac1, struct f_uac1_legacy_opts, func_inst);
uac1_opts->fn_play = fn_play;
uac1_opts->fn_cap = fn_cap;
uac1_opts->fn_cntl = fn_cntl;
uac1_opts->req_buf_size = req_buf_size;
uac1_opts->req_count = req_count;
uac1_opts->audio_buf_size = audio_buf_size;
+#endif /* CONFIG_GADGET_UAC1_LEGACY */
#endif
status = usb_string_ids_tab(cdev, strings_dev);
usb_composite_overwrite_options(cdev, &coverwrite);
dev_info(&cdev->gadget->dev,
DRIVER_DESC ", version: " DRIVER_VERSION "\n");
- set_bit(0, &msg_registered);
return 0;
fail_otg_desc:
static int __init msg_init(void)
{
- return usb_composite_probe(&msg_driver);
+ int ret;
+
+ ret = usb_composite_probe(&msg_driver);
+ set_bit(0, &msg_registered);
+
+ return ret;
}
module_init(msg_init);
config USB_ATMEL_USBA
tristate "Atmel USBA"
- depends on ((AVR32 && !OF) || ARCH_AT91)
+ depends on ARCH_AT91
help
USBA is the integrated high-speed USB Device controller on
the AT32AP700x, some AT91SAM9 and AT91CAP9 processors from Atmel.
controller, which support super speed USB peripheral.
config USB_SNP_CORE
- depends on USB_AMD5536UDC
+ depends on (USB_AMD5536UDC || USB_SNP_UDC_PLAT)
tristate
help
This enables core driver support for Synopsys USB 2.0 Device
This IP is different to the High Speed OTG IP that can be enabled
by selecting USB_DWC2 or USB_DWC3 options.
+config USB_SNP_UDC_PLAT
+ tristate "Synopsys USB 2.0 Device controller"
+ depends on (USB_GADGET && OF)
+ select USB_GADGET_DUALSPEED
+ select USB_SNP_CORE
+ default ARCH_BCM_IPROC
+ help
+ This adds Platform Device support for Synopsys Designware core
+ AHB subsystem USB2.0 Device Controller (UDC).
+
+ This driver works with UDCs integrated into Broadcom's Northstar2
+ and Cygnus SoCs.
+
+ If unsure, say N.
#
# Controllers available in both integrated and discrete versions
#
obj-$(CONFIG_USB_DUMMY_HCD) += dummy_hcd.o
obj-$(CONFIG_USB_NET2272) += net2272.o
obj-$(CONFIG_USB_NET2280) += net2280.o
-obj-$(CONFIG_USB_SNP_CORE) += amd5536udc.o
+obj-$(CONFIG_USB_SNP_CORE) += snps_udc_core.o
obj-$(CONFIG_USB_AMD5536UDC) += amd5536udc_pci.o
obj-$(CONFIG_USB_PXA25X) += pxa25x_udc.o
obj-$(CONFIG_USB_PXA27X) += pxa27x_udc.o
obj-$(CONFIG_USB_MV_U3D) += mv_u3d_core.o
obj-$(CONFIG_USB_GR_UDC) += gr_udc.o
obj-$(CONFIG_USB_GADGET_XILINX) += udc-xilinx.o
+obj-$(CONFIG_USB_SNP_UDC_PLAT) += snps_udc_plat.o
obj-$(CONFIG_USB_BDC_UDC) += bdc/
/* debug control */
/* #define UDC_VERBOSE */
+#include <linux/extcon.h>
#include <linux/usb/ch9.h>
#include <linux/usb/gadget.h>
#define UDC_HSA0_REV 1
#define UDC_HSB1_REV 2
+/* Broadcom chip rev. */
+#define UDC_BCM_REV 10
+
/*
* SETUP usb commands
* needed, because some SETUP's are handled in hw, but must be passed to
#define UDC_DEVCTL_BRLEN_MASK 0x00ff0000
#define UDC_DEVCTL_BRLEN_OFS 16
+#define UDC_DEVCTL_SRX_FLUSH 14
#define UDC_DEVCTL_CSR_DONE 13
#define UDC_DEVCTL_DEVNAK 12
#define UDC_DEVCTL_SD 10
u16 cur_config;
u16 cur_intf;
u16 cur_alt;
+
+ /* for platform device and extcon support */
+ struct device *dev;
+ struct phy *udc_phy;
+ struct extcon_dev *edev;
+ struct extcon_specific_cable_nb extcon_nb;
+ struct notifier_block nb;
+ struct delayed_work drd_work;
+ struct workqueue_struct *drd_wq;
+ u32 conn_type;
};
#define to_amd5536_udc(g) (container_of((g), struct udc, gadget))
int udc_mask_unused_interrupts(struct udc *dev);
irqreturn_t udc_irq(int irq, void *pdev);
void gadget_release(struct device *pdev);
+void empty_req_queue(struct udc_ep *ep);
void udc_basic_init(struct udc *dev);
void free_dma_pools(struct udc *dev);
int init_dma_pools(struct udc *dev);
/* debug macros ------------------------------------------------------------*/
-#define DBG(udc , args...) dev_dbg(&(udc)->pdev->dev, args)
+#define DBG(udc , args...) dev_dbg(udc->dev, args)
#ifdef UDC_VERBOSE
#define VDBG DBG
dev->phys_addr = resource;
dev->irq = pdev->irq;
dev->pdev = pdev;
+ dev->dev = &pdev->dev;
/* general probing */
if (udc_probe(dev)) {
spin_lock_irq(&udc->lock);
for (i = 0; i < inode->i_size / 4; i++)
- data[i] = usba_io_readl(udc->regs + i * 4);
+ data[i] = readl_relaxed(udc->regs + i * 4);
spin_unlock_irq(&udc->lock);
file->private_data = data;
if (crq->wLength != cpu_to_le16(sizeof(status)))
goto stall;
ep->state = DATA_STAGE_IN;
- usba_io_writew(status, ep->fifo);
+ writew_relaxed(status, ep->fifo);
usba_ep_writel(ep, SET_STA, USBA_TX_PK_RDY);
break;
}
#define USBA_REMOTE_WAKE_UP (1 << 10)
#define USBA_PULLD_DIS (1 << 11)
-#if defined(CONFIG_AVR32)
-#define USBA_ENABLE_MASK USBA_EN_USBA
-#define USBA_DISABLE_MASK 0
-#elif defined(CONFIG_ARCH_AT91)
#define USBA_ENABLE_MASK (USBA_EN_USBA | USBA_PULLD_DIS)
#define USBA_DISABLE_MASK USBA_DETACH
-#endif /* CONFIG_ARCH_AT91 */
/* Bitfields in FNUM */
#define USBA_MICRO_FRAME_NUM_OFFSET 0
| USBA_BF(name, value))
/* Register access macros */
-#ifdef CONFIG_AVR32
-#define usba_io_readl __raw_readl
-#define usba_io_writel __raw_writel
-#define usba_io_writew __raw_writew
-#else
-#define usba_io_readl readl_relaxed
-#define usba_io_writel writel_relaxed
-#define usba_io_writew writew_relaxed
-#endif
-
#define usba_readl(udc, reg) \
- usba_io_readl((udc)->regs + USBA_##reg)
+ readl_relaxed((udc)->regs + USBA_##reg)
#define usba_writel(udc, reg, value) \
- usba_io_writel((value), (udc)->regs + USBA_##reg)
+ writel_relaxed((value), (udc)->regs + USBA_##reg)
#define usba_ep_readl(ep, reg) \
- usba_io_readl((ep)->ep_regs + USBA_EPT_##reg)
+ readl_relaxed((ep)->ep_regs + USBA_EPT_##reg)
#define usba_ep_writel(ep, reg, value) \
- usba_io_writel((value), (ep)->ep_regs + USBA_EPT_##reg)
+ writel_relaxed((value), (ep)->ep_regs + USBA_EPT_##reg)
#define usba_dma_readl(ep, reg) \
- usba_io_readl((ep)->dma_regs + USBA_DMA_##reg)
+ readl_relaxed((ep)->dma_regs + USBA_DMA_##reg)
#define usba_dma_writel(ep, reg, value) \
- usba_io_writel((value), (ep)->dma_regs + USBA_DMA_##reg)
+ writel_relaxed((value), (ep)->dma_regs + USBA_DMA_##reg)
/* Calculate base address for a given endpoint or DMA controller */
#define USBA_EPT_BASE(x) (0x100 + (x) * 0x20)
bdc->dev = dev;
dev_dbg(bdc->dev, "bdc->regs: %p irq=%d\n", bdc->regs, bdc->irq);
- temp = bdc_readl(bdc->regs, BDC_BDCSC);
+ temp = bdc_readl(bdc->regs, BDC_BDCCAP1);
if ((temp & BDC_P64) &&
!dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64))) {
dev_dbg(bdc->dev, "Using 64-bit address\n");
#include <linux/list.h>
#include <linux/err.h>
#include <linux/dma-mapping.h>
+#include <linux/sched/task_stack.h>
#include <linux/workqueue.h>
#include <linux/usb/ch9.h>
goto out;
ret = ep->ops->disable(ep);
- if (ret) {
- ret = ret;
+ if (ret)
goto out;
- }
ep->enabled = false;
req->num_mapped_sgs = mapped;
} else {
+ if (is_vmalloc_addr(req->buf)) {
+ dev_err(dev, "buffer is not dma capable\n");
+ return -EFAULT;
+ } else if (object_is_on_stack(req->buf)) {
+ dev_err(dev, "buffer is on stack\n");
+ return -EFAULT;
+ }
+
req->dma = dma_map_single(dev, req->buf, req->length,
is_in ? DMA_TO_DEVICE : DMA_FROM_DEVICE);
udc->gadget->ops->udc_stop(udc->gadget);
}
+/**
+ * usb_gadget_udc_set_speed - tells usb device controller speed supported by
+ * current driver
+ * @udc: The device we want to set maximum speed
+ * @speed: The maximum speed to allowed to run
+ *
+ * This call is issued by the UDC Class driver before calling
+ * usb_gadget_udc_start() in order to make sure that we don't try to
+ * connect on speeds the gadget driver doesn't support.
+ */
+static inline void usb_gadget_udc_set_speed(struct usb_udc *udc,
+ enum usb_device_speed speed)
+{
+ if (udc->gadget->ops->udc_set_speed)
+ udc->gadget->ops->udc_set_speed(udc->gadget, speed);
+}
+
/**
* usb_udc_release - release the usb_udc struct
* @dev: the dev member within usb_udc
udc->dev.driver = &driver->driver;
udc->gadget->dev.driver = &driver->driver;
+ if (driver->max_speed < udc->gadget->max_speed)
+ usb_gadget_udc_set_speed(udc, driver->max_speed);
+
ret = driver->bind(udc->gadget, driver);
if (ret)
goto err1;
}
static DEVICE_ATTR_RO(state);
+static ssize_t function_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct usb_udc *udc = container_of(dev, struct usb_udc, dev);
+ struct usb_gadget_driver *drv = udc->driver;
+
+ if (!drv || !drv->function)
+ return 0;
+ return scnprintf(buf, PAGE_SIZE, "%s\n", drv->function);
+}
+static DEVICE_ATTR_RO(function);
+
#define USB_UDC_SPEED_ATTR(name, param) \
ssize_t name##_show(struct device *dev, \
struct device_attribute *attr, char *buf) \
&dev_attr_srp.attr,
&dev_attr_soft_connect.attr,
&dev_attr_state.attr,
+ &dev_attr_function.attr,
&dev_attr_current_speed.attr,
&dev_attr_maximum_speed.attr,
unsigned long flags;
dum = gadget_dev_to_dummy(&_gadget->dev);
-
- if (value && dum->driver) {
- if (mod_data.is_super_speed)
- dum->gadget.speed = dum->driver->max_speed;
- else if (mod_data.is_high_speed)
- dum->gadget.speed = min_t(u8, USB_SPEED_HIGH,
- dum->driver->max_speed);
- else
- dum->gadget.speed = USB_SPEED_FULL;
- dummy_udc_update_ep0(dum);
-
- if (dum->gadget.speed < dum->driver->max_speed)
- dev_dbg(udc_dev(dum), "This device can perform faster"
- " if you connect it to a %s port...\n",
- usb_speed_string(dum->driver->max_speed));
- }
dum_hcd = gadget_to_dummy_hcd(_gadget);
spin_lock_irqsave(&dum->lock, flags);
return 0;
}
+static void dummy_udc_set_speed(struct usb_gadget *_gadget,
+ enum usb_device_speed speed)
+{
+ struct dummy *dum;
+
+ dum = gadget_dev_to_dummy(&_gadget->dev);
+
+ if (mod_data.is_super_speed)
+ dum->gadget.speed = min_t(u8, USB_SPEED_SUPER, speed);
+ else if (mod_data.is_high_speed)
+ dum->gadget.speed = min_t(u8, USB_SPEED_HIGH, speed);
+ else
+ dum->gadget.speed = USB_SPEED_FULL;
+
+ dummy_udc_update_ep0(dum);
+
+ if (dum->gadget.speed < speed)
+ dev_dbg(udc_dev(dum), "This device can perform faster"
+ " if you connect it to a %s port...\n",
+ usb_speed_string(speed));
+}
+
static int dummy_udc_start(struct usb_gadget *g,
struct usb_gadget_driver *driver);
static int dummy_udc_stop(struct usb_gadget *g);
.pullup = dummy_pullup,
.udc_start = dummy_udc_start,
.udc_stop = dummy_udc_stop,
+ .udc_set_speed = dummy_udc_set_speed,
};
/*-------------------------------------------------------------------------*/
.fifo_flush = mv_ep_fifo_flush, /* flush fifo */
};
-static void udc_clock_enable(struct mv_udc *udc)
+static int udc_clock_enable(struct mv_udc *udc)
{
- clk_prepare_enable(udc->clk);
+ return clk_prepare_enable(udc->clk);
}
static void udc_clock_disable(struct mv_udc *udc)
return 0;
dev_dbg(&udc->dev->dev, "enable udc\n");
- udc_clock_enable(udc);
+ retval = udc_clock_enable(udc);
+ if (retval)
+ return retval;
+
if (udc->pdata->phy_init) {
retval = udc->pdata->phy_init(udc->phy_regs);
if (retval) {
BUG_ON(dev->driver);
/* then clean up the resources we allocated during probe() */
- net2280_led_shutdown(dev);
if (dev->requests) {
int i;
for (i = 1; i < 5; i++) {
free_irq(pdev->irq, dev);
if (dev->quirks & PLX_PCIE)
pci_disable_msi(pdev);
- if (dev->regs)
+ if (dev->regs) {
+ net2280_led_shutdown(dev);
iounmap(dev->regs);
+ }
if (dev->region)
release_mem_region(pci_resource_start(pdev, 0),
pci_resource_len(pdev, 0));
*/
#include <linux/delay.h>
+#include <linux/dma-mapping.h>
#include <linux/err.h>
#include <linux/extcon.h>
#include <linux/interrupt.h>
#define USB3_AXI_INT_ENA 0x00c
#define USB3_DMA_INT_STA 0x010
#define USB3_DMA_INT_ENA 0x014
+#define USB3_DMA_CH0_CON(n) (0x030 + ((n) - 1) * 0x10) /* n = 1 to 4 */
+#define USB3_DMA_CH0_PRD_ADR(n) (0x034 + ((n) - 1) * 0x10) /* n = 1 to 4 */
#define USB3_USB_COM_CON 0x200
#define USB3_USB20_CON 0x204
#define USB3_USB30_CON 0x208
/* AXI_INT_ENA and AXI_INT_STA */
#define AXI_INT_DMAINT BIT(31)
#define AXI_INT_EPCINT BIT(30)
+/* PRD's n = from 1 to 4 */
+#define AXI_INT_PRDEN_CLR_STA_SHIFT(n) (16 + (n) - 1)
+#define AXI_INT_PRDERR_STA_SHIFT(n) (0 + (n) - 1)
+#define AXI_INT_PRDEN_CLR_STA(n) (1 << AXI_INT_PRDEN_CLR_STA_SHIFT(n))
+#define AXI_INT_PRDERR_STA(n) (1 << AXI_INT_PRDERR_STA_SHIFT(n))
+
+/* DMA_INT_ENA and DMA_INT_STA */
+#define DMA_INT(n) BIT(n)
+
+/* DMA_CH0_CONn */
+#define DMA_CON_PIPE_DIR BIT(15) /* 1: In Transfer */
+#define DMA_CON_PIPE_NO_SHIFT 8
+#define DMA_CON_PIPE_NO_MASK GENMASK(12, DMA_CON_PIPE_NO_SHIFT)
+#define DMA_COM_PIPE_NO(n) (((n) << DMA_CON_PIPE_NO_SHIFT) & \
+ DMA_CON_PIPE_NO_MASK)
+#define DMA_CON_PRD_EN BIT(0)
/* LCLKSEL */
#define LCLKSEL_LSEL BIT(18)
#define USB3_EP0_BUF_SIZE 8
#define USB3_MAX_NUM_PIPES 30
#define USB3_WAIT_US 3
+#define USB3_DMA_NUM_SETTING_AREA 4
+/*
+ * To avoid double-meaning of "0" (xferred 65536 bytes or received zlp if
+ * buffer size is 65536), this driver uses the maximum size per a entry is
+ * 32768 bytes.
+ */
+#define USB3_DMA_MAX_XFER_SIZE 32768
+#define USB3_DMA_PRD_SIZE 4096
struct renesas_usb3;
+
+/* Physical Region Descriptor Table */
+struct renesas_usb3_prd {
+ u32 word1;
+#define USB3_PRD1_E BIT(30) /* the end of chain */
+#define USB3_PRD1_U BIT(29) /* completion of transfer */
+#define USB3_PRD1_D BIT(28) /* Error occurred */
+#define USB3_PRD1_INT BIT(27) /* Interrupt occurred */
+#define USB3_PRD1_LST BIT(26) /* Last Packet */
+#define USB3_PRD1_B_INC BIT(24)
+#define USB3_PRD1_MPS_8 0
+#define USB3_PRD1_MPS_16 BIT(21)
+#define USB3_PRD1_MPS_32 BIT(22)
+#define USB3_PRD1_MPS_64 (BIT(22) | BIT(21))
+#define USB3_PRD1_MPS_512 BIT(23)
+#define USB3_PRD1_MPS_1024 (BIT(23) | BIT(21))
+#define USB3_PRD1_MPS_RESERVED (BIT(23) | BIT(22) | BIT(21))
+#define USB3_PRD1_SIZE_MASK GENMASK(15, 0)
+
+ u32 bap;
+};
+#define USB3_DMA_NUM_PRD_ENTRIES (USB3_DMA_PRD_SIZE / \
+ sizeof(struct renesas_usb3_prd))
+#define USB3_DMA_MAX_XFER_SIZE_ALL_PRDS (USB3_DMA_PRD_SIZE / \
+ sizeof(struct renesas_usb3_prd) * \
+ USB3_DMA_MAX_XFER_SIZE)
+
+struct renesas_usb3_dma {
+ struct renesas_usb3_prd *prd;
+ dma_addr_t prd_dma;
+ int num; /* Setting area number (from 1 to 4) */
+ bool used;
+};
+
struct renesas_usb3_request {
struct usb_request req;
struct list_head queue;
struct renesas_usb3_ep {
struct usb_ep ep;
struct renesas_usb3 *usb3;
+ struct renesas_usb3_dma *dma;
int num;
char ep_name[USB3_EP_NAME_SIZE];
struct list_head queue;
struct renesas_usb3_ep *usb3_ep;
int num_usb3_eps;
+ struct renesas_usb3_dma dma[USB3_DMA_NUM_SETTING_AREA];
+
spinlock_t lock;
int disabled_count;
(i) < (usb3)->num_usb3_eps; \
(i)++, usb3_ep = usb3_get_ep(usb3, (i)))
+#define usb3_get_dma(usb3, i) (&(usb3)->dma[i])
+#define usb3_for_each_dma(usb3, dma, i) \
+ for ((i) = 0, dma = usb3_get_dma((usb3), (i)); \
+ (i) < USB3_DMA_NUM_SETTING_AREA; \
+ (i)++, dma = usb3_get_dma((usb3), (i)))
+
static const char udc_name[] = "renesas_usb3";
+static bool use_dma = 1;
+module_param(use_dma, bool, 0644);
+MODULE_PARM_DESC(use_dma, "use dedicated DMAC");
+
static void usb3_write(struct renesas_usb3 *usb3, u32 data, u32 offs)
{
iowrite32(data, usb3->reg + offs);
usb3_p0_xfer(usb3_ep, usb3_req);
}
+static void usb3_enable_dma_pipen(struct renesas_usb3 *usb3)
+{
+ usb3_set_bit(usb3, PN_CON_DATAIF_EN, USB3_PN_CON);
+}
+
+static void usb3_disable_dma_pipen(struct renesas_usb3 *usb3)
+{
+ usb3_clear_bit(usb3, PN_CON_DATAIF_EN, USB3_PN_CON);
+}
+
+static void usb3_enable_dma_irq(struct renesas_usb3 *usb3, int num)
+{
+ usb3_set_bit(usb3, DMA_INT(num), USB3_DMA_INT_ENA);
+}
+
+static void usb3_disable_dma_irq(struct renesas_usb3 *usb3, int num)
+{
+ usb3_clear_bit(usb3, DMA_INT(num), USB3_DMA_INT_ENA);
+}
+
+static u32 usb3_dma_mps_to_prd_word1(struct renesas_usb3_ep *usb3_ep)
+{
+ switch (usb3_ep->ep.maxpacket) {
+ case 8:
+ return USB3_PRD1_MPS_8;
+ case 16:
+ return USB3_PRD1_MPS_16;
+ case 32:
+ return USB3_PRD1_MPS_32;
+ case 64:
+ return USB3_PRD1_MPS_64;
+ case 512:
+ return USB3_PRD1_MPS_512;
+ case 1024:
+ return USB3_PRD1_MPS_1024;
+ default:
+ return USB3_PRD1_MPS_RESERVED;
+ }
+}
+
+static bool usb3_dma_get_setting_area(struct renesas_usb3_ep *usb3_ep,
+ struct renesas_usb3_request *usb3_req)
+{
+ struct renesas_usb3 *usb3 = usb3_ep_to_usb3(usb3_ep);
+ struct renesas_usb3_dma *dma;
+ int i;
+ bool ret = false;
+
+ if (usb3_req->req.length > USB3_DMA_MAX_XFER_SIZE_ALL_PRDS) {
+ dev_dbg(usb3_to_dev(usb3), "%s: the length is too big (%d)\n",
+ __func__, usb3_req->req.length);
+ return false;
+ }
+
+ /* The driver doesn't handle zero-length packet via dmac */
+ if (!usb3_req->req.length)
+ return false;
+
+ if (usb3_dma_mps_to_prd_word1(usb3_ep) == USB3_PRD1_MPS_RESERVED)
+ return false;
+
+ usb3_for_each_dma(usb3, dma, i) {
+ if (dma->used)
+ continue;
+
+ if (usb_gadget_map_request(&usb3->gadget, &usb3_req->req,
+ usb3_ep->dir_in) < 0)
+ break;
+
+ dma->used = true;
+ usb3_ep->dma = dma;
+ ret = true;
+ break;
+ }
+
+ return ret;
+}
+
+static void usb3_dma_put_setting_area(struct renesas_usb3_ep *usb3_ep,
+ struct renesas_usb3_request *usb3_req)
+{
+ struct renesas_usb3 *usb3 = usb3_ep_to_usb3(usb3_ep);
+ int i;
+ struct renesas_usb3_dma *dma;
+
+ usb3_for_each_dma(usb3, dma, i) {
+ if (usb3_ep->dma == dma) {
+ usb_gadget_unmap_request(&usb3->gadget, &usb3_req->req,
+ usb3_ep->dir_in);
+ dma->used = false;
+ usb3_ep->dma = NULL;
+ break;
+ }
+ }
+}
+
+static void usb3_dma_fill_prd(struct renesas_usb3_ep *usb3_ep,
+ struct renesas_usb3_request *usb3_req)
+{
+ struct renesas_usb3_prd *cur_prd = usb3_ep->dma->prd;
+ u32 remain = usb3_req->req.length;
+ u32 dma = usb3_req->req.dma;
+ u32 len;
+ int i = 0;
+
+ do {
+ len = min_t(u32, remain, USB3_DMA_MAX_XFER_SIZE) &
+ USB3_PRD1_SIZE_MASK;
+ cur_prd->word1 = usb3_dma_mps_to_prd_word1(usb3_ep) |
+ USB3_PRD1_B_INC | len;
+ cur_prd->bap = dma;
+ remain -= len;
+ dma += len;
+ if (!remain || (i + 1) < USB3_DMA_NUM_PRD_ENTRIES)
+ break;
+
+ cur_prd++;
+ i++;
+ } while (1);
+
+ cur_prd->word1 |= USB3_PRD1_E | USB3_PRD1_INT;
+ if (usb3_ep->dir_in)
+ cur_prd->word1 |= USB3_PRD1_LST;
+}
+
+static void usb3_dma_kick_prd(struct renesas_usb3_ep *usb3_ep)
+{
+ struct renesas_usb3_dma *dma = usb3_ep->dma;
+ struct renesas_usb3 *usb3 = usb3_ep_to_usb3(usb3_ep);
+ u32 dma_con = DMA_COM_PIPE_NO(usb3_ep->num) | DMA_CON_PRD_EN;
+
+ if (usb3_ep->dir_in)
+ dma_con |= DMA_CON_PIPE_DIR;
+
+ wmb(); /* prd entries should be in system memory here */
+
+ usb3_write(usb3, 1 << usb3_ep->num, USB3_DMA_INT_STA);
+ usb3_write(usb3, AXI_INT_PRDEN_CLR_STA(dma->num) |
+ AXI_INT_PRDERR_STA(dma->num), USB3_AXI_INT_STA);
+
+ usb3_write(usb3, dma->prd_dma, USB3_DMA_CH0_PRD_ADR(dma->num));
+ usb3_write(usb3, dma_con, USB3_DMA_CH0_CON(dma->num));
+ usb3_enable_dma_irq(usb3, usb3_ep->num);
+}
+
+static void usb3_dma_stop_prd(struct renesas_usb3_ep *usb3_ep)
+{
+ struct renesas_usb3 *usb3 = usb3_ep_to_usb3(usb3_ep);
+ struct renesas_usb3_dma *dma = usb3_ep->dma;
+
+ usb3_disable_dma_irq(usb3, usb3_ep->num);
+ usb3_write(usb3, 0, USB3_DMA_CH0_CON(dma->num));
+}
+
+static int usb3_dma_update_status(struct renesas_usb3_ep *usb3_ep,
+ struct renesas_usb3_request *usb3_req)
+{
+ struct renesas_usb3_prd *cur_prd = usb3_ep->dma->prd;
+ struct usb_request *req = &usb3_req->req;
+ u32 remain, len;
+ int i = 0;
+ int status = 0;
+
+ rmb(); /* The controller updated prd entries */
+
+ do {
+ if (cur_prd->word1 & USB3_PRD1_D)
+ status = -EIO;
+ if (cur_prd->word1 & USB3_PRD1_E)
+ len = req->length % USB3_DMA_MAX_XFER_SIZE;
+ else
+ len = USB3_DMA_MAX_XFER_SIZE;
+ remain = cur_prd->word1 & USB3_PRD1_SIZE_MASK;
+ req->actual += len - remain;
+
+ if (cur_prd->word1 & USB3_PRD1_E ||
+ (i + 1) < USB3_DMA_NUM_PRD_ENTRIES)
+ break;
+
+ cur_prd++;
+ i++;
+ } while (1);
+
+ return status;
+}
+
+static bool usb3_dma_try_start(struct renesas_usb3_ep *usb3_ep,
+ struct renesas_usb3_request *usb3_req)
+{
+ struct renesas_usb3 *usb3 = usb3_ep_to_usb3(usb3_ep);
+
+ if (!use_dma)
+ return false;
+
+ if (usb3_dma_get_setting_area(usb3_ep, usb3_req)) {
+ usb3_pn_stop(usb3);
+ usb3_enable_dma_pipen(usb3);
+ usb3_dma_fill_prd(usb3_ep, usb3_req);
+ usb3_dma_kick_prd(usb3_ep);
+ usb3_pn_start(usb3);
+ return true;
+ }
+
+ return false;
+}
+
+static int usb3_dma_try_stop(struct renesas_usb3_ep *usb3_ep,
+ struct renesas_usb3_request *usb3_req)
+{
+ struct renesas_usb3 *usb3 = usb3_ep_to_usb3(usb3_ep);
+ unsigned long flags;
+ int status = 0;
+
+ spin_lock_irqsave(&usb3->lock, flags);
+ if (!usb3_ep->dma)
+ goto out;
+
+ if (!usb3_pn_change(usb3, usb3_ep->num))
+ usb3_disable_dma_pipen(usb3);
+ usb3_dma_stop_prd(usb3_ep);
+ status = usb3_dma_update_status(usb3_ep, usb3_req);
+ usb3_dma_put_setting_area(usb3_ep, usb3_req);
+
+out:
+ spin_unlock_irqrestore(&usb3->lock, flags);
+ return status;
+}
+
+static int renesas_usb3_dma_free_prd(struct renesas_usb3 *usb3,
+ struct device *dev)
+{
+ int i;
+ struct renesas_usb3_dma *dma;
+
+ usb3_for_each_dma(usb3, dma, i) {
+ if (dma->prd) {
+ dma_free_coherent(dev, USB3_DMA_MAX_XFER_SIZE,
+ dma->prd, dma->prd_dma);
+ dma->prd = NULL;
+ }
+ }
+
+ return 0;
+}
+
+static int renesas_usb3_dma_alloc_prd(struct renesas_usb3 *usb3,
+ struct device *dev)
+{
+ int i;
+ struct renesas_usb3_dma *dma;
+
+ if (!use_dma)
+ return 0;
+
+ usb3_for_each_dma(usb3, dma, i) {
+ dma->prd = dma_alloc_coherent(dev, USB3_DMA_PRD_SIZE,
+ &dma->prd_dma, GFP_KERNEL);
+ if (!dma->prd) {
+ renesas_usb3_dma_free_prd(usb3, dev);
+ return -ENOMEM;
+ }
+ dma->num = i + 1;
+ }
+
+ return 0;
+}
+
static void usb3_start_pipen(struct renesas_usb3_ep *usb3_ep,
struct renesas_usb3_request *usb3_req)
{
goto out;
usb3_ep->started = true;
+
+ if (usb3_dma_try_start(usb3_ep, usb3_req))
+ goto out;
+
usb3_pn_start(usb3);
if (usb3_ep->dir_in) {
}
}
+static void usb3_irq_dma_int(struct renesas_usb3 *usb3, u32 dma_sta)
+{
+ struct renesas_usb3_ep *usb3_ep;
+ struct renesas_usb3_request *usb3_req;
+ int i, status;
+
+ for (i = 0; i < usb3->num_usb3_eps; i++) {
+ if (!(dma_sta & DMA_INT(i)))
+ continue;
+
+ usb3_ep = usb3_get_ep(usb3, i);
+ if (!(usb3_read(usb3, USB3_AXI_INT_STA) &
+ AXI_INT_PRDEN_CLR_STA(usb3_ep->dma->num)))
+ continue;
+
+ usb3_req = usb3_get_request(usb3_ep);
+ status = usb3_dma_try_stop(usb3_ep, usb3_req);
+ usb3_request_done_pipen(usb3, usb3_ep, usb3_req, status);
+ }
+}
+
+static void usb3_irq_dma(struct renesas_usb3 *usb3)
+{
+ u32 dma_sta = usb3_read(usb3, USB3_DMA_INT_STA);
+
+ dma_sta &= usb3_read(usb3, USB3_DMA_INT_ENA);
+ if (dma_sta) {
+ usb3_write(usb3, dma_sta, USB3_DMA_INT_STA);
+ usb3_irq_dma_int(usb3, dma_sta);
+ }
+}
+
static irqreturn_t renesas_usb3_irq(int irq, void *_usb3)
{
struct renesas_usb3 *usb3 = _usb3;
irqreturn_t ret = IRQ_NONE;
u32 axi_int_sta = usb3_read(usb3, USB3_AXI_INT_STA);
+ if (axi_int_sta & AXI_INT_DMAINT) {
+ usb3_irq_dma(usb3);
+ ret = IRQ_HANDLED;
+ }
+
if (axi_int_sta & AXI_INT_EPCINT) {
usb3_irq_epc(usb3);
ret = IRQ_HANDLED;
usb3_req = usb3_get_request(usb3_ep);
if (!usb3_req)
break;
+ usb3_dma_try_stop(usb3_ep, usb3_req);
usb3_request_done(usb3_ep, usb3_req, -ESHUTDOWN);
} while (1);
dev_dbg(usb3_to_dev(usb3), "ep_dequeue: ep%2d, %u\n", usb3_ep->num,
_req->length);
+ usb3_dma_try_stop(usb3_ep, usb3_req);
usb3_request_done_pipen(usb3, usb3_ep, usb3_req, -ECONNRESET);
return 0;
device_remove_file(&pdev->dev, &dev_attr_role);
usb_del_gadget_udc(&usb3->gadget);
+ renesas_usb3_dma_free_prd(usb3, &pdev->dev);
__renesas_usb3_ep_free_request(usb3->ep0_req);
if (!usb3->ep0_req)
return -ENOMEM;
+ ret = renesas_usb3_dma_alloc_prd(usb3, &pdev->dev);
+ if (ret < 0)
+ goto err_alloc_prd;
+
ret = usb_add_gadget_udc(&pdev->dev, &usb3->gadget);
if (ret < 0)
goto err_add_udc;
usb_del_gadget_udc(&usb3->gadget);
err_add_udc:
+ renesas_usb3_dma_free_prd(usb3, &pdev->dev);
+
+err_alloc_prd:
__renesas_usb3_ep_free_request(usb3->ep0_req);
return ret;
#include "amd5536udc.h"
static void udc_tasklet_disconnect(unsigned long);
-static void empty_req_queue(struct udc_ep *);
static void udc_setup_endpoints(struct udc *dev);
static void udc_soft_reset(struct udc *dev);
static struct udc_request *udc_alloc_bna_dummy(struct udc_ep *ep);
if (use_dma && use_dma_ppb && !use_dma_ppb_du) {
DBG(dev, "DMA mode = PPBNDU (packet per buffer "
"WITHOUT desc. update)\n");
- dev_info(&dev->pdev->dev, "DMA mode (%s)\n", "PPBNDU");
+ dev_info(dev->dev, "DMA mode (%s)\n", "PPBNDU");
} else if (use_dma && use_dma_ppb && use_dma_ppb_du) {
DBG(dev, "DMA mode = PPBDU (packet per buffer "
"WITH desc. update)\n");
- dev_info(&dev->pdev->dev, "DMA mode (%s)\n", "PPBDU");
+ dev_info(dev->dev, "DMA mode (%s)\n", "PPBDU");
}
if (use_dma && use_dma_bufferfill_mode) {
DBG(dev, "DMA mode = BF (buffer fill mode)\n");
- dev_info(&dev->pdev->dev, "DMA mode (%s)\n", "BF");
+ dev_info(dev->dev, "DMA mode (%s)\n", "BF");
}
if (!use_dma)
- dev_info(&dev->pdev->dev, "FIFO mode\n");
+ dev_info(dev->dev, "FIFO mode\n");
DBG(dev, "-------------------------------------------------------\n");
}
}
/* Empty request queue of an endpoint; caller holds spinlock */
-static void empty_req_queue(struct udc_ep *ep)
+void empty_req_queue(struct udc_ep *ep)
{
struct udc_request *req;
complete_req(ep, req, -ESHUTDOWN);
}
}
+EXPORT_SYMBOL_GPL(empty_req_queue);
/* Dequeues a request packet, called by gadget driver */
static int udc_dequeue(struct usb_ep *usbep, struct usb_request *usbreq)
/* Bringup after Connect event, initial bringup to be ready for ep0 events */
static void usb_connect(struct udc *dev)
{
+ /* Return if already connected */
+ if (dev->connected)
+ return;
- dev_info(&dev->pdev->dev, "USB Connect\n");
+ dev_info(dev->dev, "USB Connect\n");
dev->connected = 1;
*/
static void usb_disconnect(struct udc *dev)
{
+ /* Return if already disconnected */
+ if (!dev->connected)
+ return;
- dev_info(&dev->pdev->dev, "USB Disconnect\n");
+ dev_info(dev->dev, "USB Disconnect\n");
dev->connected = 0;
/* device int. status reset */
writel(UDC_DEV_MSK_DISABLE, &dev->regs->irqsts);
- spin_lock_irqsave(&udc_irq_spinlock, flags);
- writel(AMD_BIT(UDC_DEVCFG_SOFTRESET), &dev->regs->cfg);
- readl(&dev->regs->cfg);
- spin_unlock_irqrestore(&udc_irq_spinlock, flags);
-
+ /* Don't do this for Broadcom UDC since this is a reserved
+ * bit.
+ */
+ if (dev->chiprev != UDC_BCM_REV) {
+ spin_lock_irqsave(&udc_irq_spinlock, flags);
+ writel(AMD_BIT(UDC_DEVCFG_SOFTRESET), &dev->regs->cfg);
+ readl(&dev->regs->cfg);
+ spin_unlock_irqrestore(&udc_irq_spinlock, flags);
+ }
}
/* RDE timer callback to set RDE bit */
}
/* HE event ? */
if (tmp & AMD_BIT(UDC_EPSTS_HE)) {
- dev_err(&dev->pdev->dev, "HE ep%dout occurred\n", ep->num);
+ dev_err(dev->dev, "HE ep%dout occurred\n", ep->num);
/* clear HE */
writel(tmp | AMD_BIT(UDC_EPSTS_HE), &ep->regs->sts);
if (use_dma) {
/* BNA ? */
if (epsts & AMD_BIT(UDC_EPSTS_BNA)) {
- dev_err(&dev->pdev->dev,
+ dev_err(dev->dev,
"BNA ep%din occurred - DESPTR = %08lx\n",
ep->num,
(unsigned long) readl(&ep->regs->desptr));
}
/* HE event ? */
if (epsts & AMD_BIT(UDC_EPSTS_HE)) {
- dev_err(&dev->pdev->dev,
+ dev_err(dev->dev,
"HE ep%dn occurred - DESPTR = %08lx\n",
ep->num, (unsigned long) readl(&ep->regs->desptr));
/* link up all endpoints */
udc_setup_endpoints(dev);
- dev_info(&dev->pdev->dev, "Connect: %s\n",
+ dev_info(dev->dev, "Connect: %s\n",
usb_speed_string(dev->gadget.speed));
/* init ep 0 */
}
/* DMA setup */
- dev->data_requests = dma_pool_create("data_requests", NULL,
+ dev->data_requests = dma_pool_create("data_requests", dev->dev,
sizeof(struct udc_data_dma), 0, 0);
if (!dev->data_requests) {
DBG(dev, "can't get request data pool\n");
dev->ep[UDC_EP0IN_IX].dma = &dev->regs->ctl;
/* dma desc for setup data */
- dev->stp_requests = dma_pool_create("setup requests", NULL,
+ dev->stp_requests = dma_pool_create("setup requests", dev->dev,
sizeof(struct udc_stp_dma), 0, 0);
if (!dev->stp_requests) {
DBG(dev, "can't get stp request pool\n");
/* init registers, interrupts, ... */
startup_registers(dev);
- dev_info(&dev->pdev->dev, "%s\n", mod_desc);
+ dev_info(dev->dev, "%s\n", mod_desc);
snprintf(tmp, sizeof(tmp), "%d", dev->irq);
- dev_info(&dev->pdev->dev,
- "irq %s, pci mem %08lx, chip rev %02x(Geode5536 %s)\n",
- tmp, dev->phys_addr, dev->chiprev,
- (dev->chiprev == UDC_HSA0_REV) ? "A0" : "B1");
- strcpy(tmp, UDC_DRIVER_VERSION_STRING);
- if (dev->chiprev == UDC_HSA0_REV) {
- dev_err(&dev->pdev->dev, "chip revision is A0; too old\n");
- retval = -ENODEV;
- goto finished;
+
+ /* Print this device info for AMD chips only*/
+ if (dev->chiprev == UDC_HSA0_REV ||
+ dev->chiprev == UDC_HSB1_REV) {
+ dev_info(dev->dev, "irq %s, pci mem %08lx, chip rev %02x(Geode5536 %s)\n",
+ tmp, dev->phys_addr, dev->chiprev,
+ (dev->chiprev == UDC_HSA0_REV) ?
+ "A0" : "B1");
+ strcpy(tmp, UDC_DRIVER_VERSION_STRING);
+ if (dev->chiprev == UDC_HSA0_REV) {
+ dev_err(dev->dev, "chip revision is A0; too old\n");
+ retval = -ENODEV;
+ goto finished;
+ }
+ dev_info(dev->dev,
+ "driver version: %s(for Geode5536 B1)\n", tmp);
}
- dev_info(&dev->pdev->dev,
- "driver version: %s(for Geode5536 B1)\n", tmp);
+
udc = dev;
- retval = usb_add_gadget_udc_release(&udc->pdev->dev, &dev->gadget,
+ retval = usb_add_gadget_udc_release(udc->dev, &dev->gadget,
gadget_release);
if (retval)
goto finished;
--- /dev/null
+/*
+ * snps_udc_plat.c - Synopsys UDC Platform Driver
+ *
+ * Copyright (C) 2016 Broadcom
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation version 2.
+ *
+ * This program is distributed "as is" WITHOUT ANY WARRANTY of any
+ * kind, whether express or implied; without even the implied warranty
+ * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/extcon.h>
+#include <linux/of_address.h>
+#include <linux/of_irq.h>
+#include <linux/of_gpio.h>
+#include <linux/platform_device.h>
+#include <linux/phy/phy.h>
+#include <linux/module.h>
+#include <linux/dmapool.h>
+#include <linux/interrupt.h>
+#include <linux/moduleparam.h>
+#include "amd5536udc.h"
+
+/* description */
+#define UDC_MOD_DESCRIPTION "Synopsys UDC platform driver"
+
+void start_udc(struct udc *udc)
+{
+ if (udc->driver) {
+ dev_info(udc->dev, "Connecting...\n");
+ udc_enable_dev_setup_interrupts(udc);
+ udc_basic_init(udc);
+ udc->connected = 1;
+ }
+}
+
+void stop_udc(struct udc *udc)
+{
+ int tmp;
+ u32 reg;
+
+ spin_lock(&udc->lock);
+
+ /* Flush the receieve fifo */
+ reg = readl(&udc->regs->ctl);
+ reg |= AMD_BIT(UDC_DEVCTL_SRX_FLUSH);
+ writel(reg, &udc->regs->ctl);
+
+ reg = readl(&udc->regs->ctl);
+ reg &= ~(AMD_BIT(UDC_DEVCTL_SRX_FLUSH));
+ writel(reg, &udc->regs->ctl);
+ dev_dbg(udc->dev, "ep rx queue flushed\n");
+
+ /* Mask interrupts. Required more so when the
+ * UDC is connected to a DRD phy.
+ */
+ udc_mask_unused_interrupts(udc);
+
+ /* Disconnect gadget driver */
+ if (udc->driver) {
+ spin_unlock(&udc->lock);
+ udc->driver->disconnect(&udc->gadget);
+ spin_lock(&udc->lock);
+
+ /* empty queues */
+ for (tmp = 0; tmp < UDC_EP_NUM; tmp++)
+ empty_req_queue(&udc->ep[tmp]);
+ }
+ udc->connected = 0;
+
+ spin_unlock(&udc->lock);
+ dev_info(udc->dev, "Device disconnected\n");
+}
+
+void udc_drd_work(struct work_struct *work)
+{
+ struct udc *udc;
+
+ udc = container_of(to_delayed_work(work),
+ struct udc, drd_work);
+
+ if (udc->conn_type) {
+ dev_dbg(udc->dev, "idle -> device\n");
+ start_udc(udc);
+ } else {
+ dev_dbg(udc->dev, "device -> idle\n");
+ stop_udc(udc);
+ }
+}
+
+static int usbd_connect_notify(struct notifier_block *self,
+ unsigned long event, void *ptr)
+{
+ struct udc *udc = container_of(self, struct udc, nb);
+
+ dev_dbg(udc->dev, "%s: event: %lu\n", __func__, event);
+
+ udc->conn_type = event;
+
+ schedule_delayed_work(&udc->drd_work, 0);
+
+ return NOTIFY_OK;
+}
+
+static int udc_plat_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct resource *res;
+ struct udc *udc;
+ int ret;
+
+ udc = devm_kzalloc(dev, sizeof(*udc), GFP_KERNEL);
+ if (!udc)
+ return -ENOMEM;
+
+ spin_lock_init(&udc->lock);
+ udc->dev = dev;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ udc->virt_addr = devm_ioremap_resource(dev, res);
+ if (IS_ERR(udc->regs))
+ return PTR_ERR(udc->regs);
+
+ /* udc csr registers base */
+ udc->csr = udc->virt_addr + UDC_CSR_ADDR;
+
+ /* dev registers base */
+ udc->regs = udc->virt_addr + UDC_DEVCFG_ADDR;
+
+ /* ep registers base */
+ udc->ep_regs = udc->virt_addr + UDC_EPREGS_ADDR;
+
+ /* fifo's base */
+ udc->rxfifo = (u32 __iomem *)(udc->virt_addr + UDC_RXFIFO_ADDR);
+ udc->txfifo = (u32 __iomem *)(udc->virt_addr + UDC_TXFIFO_ADDR);
+
+ udc->phys_addr = (unsigned long)res->start;
+
+ udc->irq = irq_of_parse_and_map(dev->of_node, 0);
+ if (udc->irq <= 0) {
+ dev_err(dev, "Can't parse and map interrupt\n");
+ return -EINVAL;
+ }
+
+ udc->udc_phy = devm_of_phy_get_by_index(dev, dev->of_node, 0);
+ if (IS_ERR(udc->udc_phy)) {
+ dev_err(dev, "Failed to obtain phy from device tree\n");
+ return PTR_ERR(udc->udc_phy);
+ }
+
+ ret = phy_init(udc->udc_phy);
+ if (ret) {
+ dev_err(dev, "UDC phy init failed");
+ return ret;
+ }
+
+ ret = phy_power_on(udc->udc_phy);
+ if (ret) {
+ dev_err(dev, "UDC phy power on failed");
+ phy_exit(udc->udc_phy);
+ return ret;
+ }
+
+ /* Register for extcon if supported */
+ if (of_get_property(dev->of_node, "extcon", NULL)) {
+ udc->edev = extcon_get_edev_by_phandle(dev, 0);
+ if (IS_ERR(udc->edev)) {
+ if (PTR_ERR(udc->edev) == -EPROBE_DEFER)
+ return -EPROBE_DEFER;
+ dev_err(dev, "Invalid or missing extcon\n");
+ ret = PTR_ERR(udc->edev);
+ goto exit_phy;
+ }
+
+ udc->nb.notifier_call = usbd_connect_notify;
+ ret = extcon_register_notifier(udc->edev, EXTCON_USB,
+ &udc->nb);
+ if (ret < 0) {
+ dev_err(dev, "Can't register extcon device\n");
+ goto exit_phy;
+ }
+
+ ret = extcon_get_cable_state_(udc->edev, EXTCON_USB);
+ if (ret < 0) {
+ dev_err(dev, "Can't get cable state\n");
+ goto exit_extcon;
+ } else if (ret) {
+ udc->conn_type = ret;
+ }
+ INIT_DELAYED_WORK(&udc->drd_work, udc_drd_work);
+ }
+
+ /* init dma pools */
+ if (use_dma) {
+ ret = init_dma_pools(udc);
+ if (ret != 0)
+ goto exit_extcon;
+ }
+
+ ret = devm_request_irq(dev, udc->irq, udc_irq, IRQF_SHARED,
+ "snps-udc", udc);
+ if (ret < 0) {
+ dev_err(dev, "Request irq %d failed for UDC\n", udc->irq);
+ goto exit_dma;
+ }
+
+ platform_set_drvdata(pdev, udc);
+ udc->chiprev = UDC_BCM_REV;
+
+ if (udc_probe(udc)) {
+ ret = -ENODEV;
+ goto exit_dma;
+ }
+ dev_info(dev, "Synopsys UDC platform driver probe successful\n");
+
+ return 0;
+
+exit_dma:
+ if (use_dma)
+ free_dma_pools(udc);
+exit_extcon:
+ if (udc->edev)
+ extcon_unregister_notifier(udc->edev, EXTCON_USB, &udc->nb);
+exit_phy:
+ if (udc->udc_phy) {
+ phy_power_off(udc->udc_phy);
+ phy_exit(udc->udc_phy);
+ }
+ return ret;
+}
+
+static int udc_plat_remove(struct platform_device *pdev)
+{
+ struct udc *dev;
+
+ dev = platform_get_drvdata(pdev);
+
+ usb_del_gadget_udc(&dev->gadget);
+ /* gadget driver must not be registered */
+ if (WARN_ON(dev->driver))
+ return 0;
+
+ /* dma pool cleanup */
+ free_dma_pools(dev);
+
+ udc_remove(dev);
+
+ platform_set_drvdata(pdev, NULL);
+
+ if (dev->drd_wq) {
+ flush_workqueue(dev->drd_wq);
+ destroy_workqueue(dev->drd_wq);
+ }
+
+ phy_power_off(dev->udc_phy);
+ phy_exit(dev->udc_phy);
+ extcon_unregister_notifier(dev->edev, EXTCON_USB, &dev->nb);
+
+ dev_info(&pdev->dev, "Synopsys UDC platform driver removed\n");
+
+ return 0;
+}
+
+#ifdef CONFIG_PM_SLEEP
+static int udc_plat_suspend(struct device *dev)
+{
+ struct udc *udc;
+
+ udc = dev_get_drvdata(dev);
+ stop_udc(udc);
+
+ if (extcon_get_cable_state_(udc->edev, EXTCON_USB) > 0) {
+ dev_dbg(udc->dev, "device -> idle\n");
+ stop_udc(udc);
+ }
+ phy_power_off(udc->udc_phy);
+ phy_exit(udc->udc_phy);
+
+ return 0;
+}
+
+static int udc_plat_resume(struct device *dev)
+{
+ struct udc *udc;
+ int ret;
+
+ udc = dev_get_drvdata(dev);
+
+ ret = phy_init(udc->udc_phy);
+ if (ret) {
+ dev_err(udc->dev, "UDC phy init failure");
+ return ret;
+ }
+
+ ret = phy_power_on(udc->udc_phy);
+ if (ret) {
+ dev_err(udc->dev, "UDC phy power on failure");
+ phy_exit(udc->udc_phy);
+ return ret;
+ }
+
+ if (extcon_get_cable_state_(udc->edev, EXTCON_USB) > 0) {
+ dev_dbg(udc->dev, "idle -> device\n");
+ start_udc(udc);
+ }
+
+ return 0;
+}
+static const struct dev_pm_ops udc_plat_pm_ops = {
+ .suspend = udc_plat_suspend,
+ .resume = udc_plat_resume,
+};
+#endif
+
+#if defined(CONFIG_OF)
+static const struct of_device_id of_udc_match[] = {
+ { .compatible = "brcm,ns2-udc", },
+ { .compatible = "brcm,cygnus-udc", },
+ { .compatible = "brcm,iproc-udc", },
+ { }
+};
+MODULE_DEVICE_TABLE(of, of_udc_match);
+#endif
+
+static struct platform_driver udc_plat_driver = {
+ .probe = udc_plat_probe,
+ .remove = udc_plat_remove,
+ .driver = {
+ .name = "snps-udc-plat",
+ .of_match_table = of_match_ptr(of_udc_match),
+#ifdef CONFIG_PM_SLEEP
+ .pm = &udc_plat_pm_ops,
+#endif
+ },
+};
+module_platform_driver(udc_plat_driver);
+
+MODULE_DESCRIPTION(UDC_MOD_DESCRIPTION);
+MODULE_AUTHOR("Broadcom");
+MODULE_LICENSE("GPL v2");
break;
}
if (&req->usb_req != _req) {
- spin_unlock_irqrestore(&ep->udc->lock, flags);
+ spin_unlock_irqrestore(&udc->lock, flags);
return -EINVAL;
}
xudc_done(ep, req, -ECONNRESET);
config USB_OHCI_HCD_OMAP3
tristate "OHCI support for OMAP3 and later chips"
depends on (ARCH_OMAP3 || ARCH_OMAP4 || SOC_OMAP5)
+ select USB_OHCI_HCD_PLATFORM
default y
- ---help---
+ help
+ This option is deprecated now and the driver was removed, use
+ USB_OHCI_HCD_PLATFORM instead.
+
Enables support for the on-chip OHCI controller on
OMAP3 and later chips.
config USB_UHCI_PLATFORM
bool
- default y if ARCH_VT8500
+ default y if (ARCH_VT8500 || ARCH_ASPEED)
+
+config USB_UHCI_ASPEED
+ bool
+ default y if ARCH_ASPEED
config USB_UHCI_BIG_ENDIAN_MMIO
bool
obj-$(CONFIG_USB_OHCI_HCD_PLATFORM) += ohci-platform.o
obj-$(CONFIG_USB_OHCI_EXYNOS) += ohci-exynos.o
obj-$(CONFIG_USB_OHCI_HCD_OMAP1) += ohci-omap.o
-obj-$(CONFIG_USB_OHCI_HCD_OMAP3) += ohci-omap3.o
obj-$(CONFIG_USB_OHCI_HCD_SPEAR) += ohci-spear.o
obj-$(CONFIG_USB_OHCI_HCD_STI) += ohci-st.o
obj-$(CONFIG_USB_OHCI_HCD_AT91) += ohci-at91.o
struct exynos_ehci_hcd *exynos_ehci = to_exynos_ehci(hcd);
int ret;
- clk_prepare_enable(exynos_ehci->clk);
+ ret = clk_prepare_enable(exynos_ehci->clk);
+ if (ret)
+ return ret;
ret = exynos_ehci_phy_enable(dev);
if (ret) {
addr |= epnum << 8;
addr |= dev->devnum;
stream->ps.usecs = HS_USECS_ISO(maxp);
- think_time = dev->tt ? dev->tt->think_time : 0;
+ think_time = dev->tt->think_time;
stream->ps.tt_usecs = NS_TO_US(think_time + usb_calc_bus_time(
dev->speed, is_input, 1, maxp));
hs_transfers = max(1u, (maxp + 187) / 188);
*/
now = ktime_get();
for_each_set_bit(e, &events, EHCI_HRTIMER_NUM_EVENTS) {
- if (now >= ehci->hr_timeouts[e])
+ if (ktime_compare(now, ehci->hr_timeouts[e]) >= 0)
event_handlers[e](ehci);
else
ehci_enable_event(ehci, e, false);
*/
now = ktime_get();
for_each_set_bit(e, &events, FOTG210_HRTIMER_NUM_EVENTS) {
- if (now >= fotg210->hr_timeouts[e])
+ if (ktime_compare(now, fotg210->hr_timeouts[e]) >= 0)
event_handlers[e](fotg210);
else
fotg210_enable_event(fotg210, e, false);
+++ /dev/null
-/*
- * ohci-omap3.c - driver for OHCI on OMAP3 and later processors
- *
- * Bus Glue for OMAP3 USBHOST 3 port OHCI controller
- * This controller is also used in later OMAPs and AM35x chips
- *
- * Copyright (C) 2007-2010 Texas Instruments, Inc.
- * Author: Vikram Pandita <vikram.pandita@ti.com>
- * Author: Anand Gadiyar <gadiyar@ti.com>
- * Author: Keshava Munegowda <keshava_mgowda@ti.com>
- *
- * Based on ehci-omap.c and some other ohci glue layers
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
- *
- * TODO (last updated Feb 27, 2011):
- * - add kernel-doc
- */
-
-#include <linux/dma-mapping.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <linux/of.h>
-#include <linux/usb/otg.h>
-#include <linux/platform_device.h>
-#include <linux/pm_runtime.h>
-#include <linux/usb.h>
-#include <linux/usb/hcd.h>
-
-#include "ohci.h"
-
-#define DRIVER_DESC "OHCI OMAP3 driver"
-
-static const char hcd_name[] = "ohci-omap3";
-static struct hc_driver __read_mostly ohci_omap3_hc_driver;
-
-/*
- * configure so an HC device and id are always provided
- * always called with process context; sleeping is OK
- */
-
-/**
- * ohci_hcd_omap3_probe - initialize OMAP-based HCDs
- *
- * Allocates basic resources for this USB host controller, and
- * then invokes the start() method for the HCD associated with it
- * through the hotplug entry's driver_data.
- */
-static int ohci_hcd_omap3_probe(struct platform_device *pdev)
-{
- struct device *dev = &pdev->dev;
- struct ohci_hcd *ohci;
- struct usb_hcd *hcd = NULL;
- void __iomem *regs = NULL;
- struct resource *res;
- int ret;
- int irq;
-
- if (usb_disabled())
- return -ENODEV;
-
- if (!dev->parent) {
- dev_err(dev, "Missing parent device\n");
- return -ENODEV;
- }
-
- irq = platform_get_irq(pdev, 0);
- if (irq < 0) {
- dev_err(dev, "OHCI irq failed\n");
- return -ENODEV;
- }
-
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- if (!res) {
- dev_err(dev, "UHH OHCI get resource failed\n");
- return -ENOMEM;
- }
-
- regs = ioremap(res->start, resource_size(res));
- if (!regs) {
- dev_err(dev, "UHH OHCI ioremap failed\n");
- return -ENOMEM;
- }
-
- /*
- * Right now device-tree probed devices don't get dma_mask set.
- * Since shared usb code relies on it, set it here for now.
- * Once we have dma capability bindings this can go away.
- */
- ret = dma_coerce_mask_and_coherent(dev, DMA_BIT_MASK(32));
- if (ret)
- goto err_io;
-
- ret = -ENODEV;
- hcd = usb_create_hcd(&ohci_omap3_hc_driver, dev,
- dev_name(dev));
- if (!hcd) {
- dev_err(dev, "usb_create_hcd failed\n");
- goto err_io;
- }
-
- hcd->rsrc_start = res->start;
- hcd->rsrc_len = resource_size(res);
- hcd->regs = regs;
-
- pm_runtime_enable(dev);
- pm_runtime_get_sync(dev);
-
- ohci = hcd_to_ohci(hcd);
- /*
- * RemoteWakeupConnected has to be set explicitly before
- * calling ohci_run. The reset value of RWC is 0.
- */
- ohci->hc_control = OHCI_CTRL_RWC;
-
- ret = usb_add_hcd(hcd, irq, 0);
- if (ret) {
- dev_dbg(dev, "failed to add hcd with err %d\n", ret);
- goto err_add_hcd;
- }
- device_wakeup_enable(hcd->self.controller);
-
- return 0;
-
-err_add_hcd:
- pm_runtime_put_sync(dev);
- usb_put_hcd(hcd);
-
-err_io:
- iounmap(regs);
-
- return ret;
-}
-
-/*
- * may be called without controller electrically present
- * may be called with controller, bus, and devices active
- */
-
-/**
- * ohci_hcd_omap3_remove - shutdown processing for OHCI HCDs
- * @pdev: USB Host Controller being removed
- *
- * Reverses the effect of ohci_hcd_omap3_probe(), first invoking
- * the HCD's stop() method. It is always called from a thread
- * context, normally "rmmod", "apmd", or something similar.
- */
-static int ohci_hcd_omap3_remove(struct platform_device *pdev)
-{
- struct device *dev = &pdev->dev;
- struct usb_hcd *hcd = dev_get_drvdata(dev);
-
- iounmap(hcd->regs);
- usb_remove_hcd(hcd);
- pm_runtime_put_sync(dev);
- pm_runtime_disable(dev);
- usb_put_hcd(hcd);
- return 0;
-}
-
-static const struct of_device_id omap_ohci_dt_ids[] = {
- { .compatible = "ti,ohci-omap3" },
- { }
-};
-
-MODULE_DEVICE_TABLE(of, omap_ohci_dt_ids);
-
-static struct platform_driver ohci_hcd_omap3_driver = {
- .probe = ohci_hcd_omap3_probe,
- .remove = ohci_hcd_omap3_remove,
- .shutdown = usb_hcd_platform_shutdown,
- .driver = {
- .name = "ohci-omap3",
- .of_match_table = omap_ohci_dt_ids,
- },
-};
-
-static int __init ohci_omap3_init(void)
-{
- if (usb_disabled())
- return -ENODEV;
-
- pr_info("%s: " DRIVER_DESC "\n", hcd_name);
-
- ohci_init_driver(&ohci_omap3_hc_driver, NULL);
- return platform_driver_register(&ohci_hcd_omap3_driver);
-}
-module_init(ohci_omap3_init);
-
-static void __exit ohci_omap3_cleanup(void)
-{
- platform_driver_unregister(&ohci_hcd_omap3_driver);
-}
-module_exit(ohci_omap3_cleanup);
-
-MODULE_DESCRIPTION(DRIVER_DESC);
-MODULE_ALIAS("platform:ohci-omap3");
-MODULE_AUTHOR("Anand Gadiyar <gadiyar@ti.com>");
-MODULE_LICENSE("GPL");
#include <linux/err.h>
#include <linux/phy/phy.h>
#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
#include <linux/reset.h>
#include <linux/usb/ohci_pdriver.h>
#include <linux/usb.h>
if (of_property_read_bool(dev->dev.of_node, "no-big-frame-no"))
ohci->flags |= OHCI_QUIRK_FRAME_NO;
+ if (of_property_read_bool(dev->dev.of_node,
+ "remote-wakeup-connected"))
+ ohci->hc_control = OHCI_CTRL_RWC;
+
of_property_read_u32(dev->dev.of_node, "num-ports",
&ohci->num_ports);
}
#endif
+ pm_runtime_set_active(&dev->dev);
+ pm_runtime_enable(&dev->dev);
if (pdata->power_on) {
err = pdata->power_on(dev);
if (err < 0)
if (pdata->power_off)
pdata->power_off(dev);
err_reset:
+ pm_runtime_disable(&dev->dev);
while (--rst >= 0)
reset_control_assert(priv->resets[rst]);
err_put_clks:
struct ohci_platform_priv *priv = hcd_to_ohci_priv(hcd);
int clk, rst;
+ pm_runtime_get_sync(&dev->dev);
usb_remove_hcd(hcd);
if (pdata->power_off)
usb_put_hcd(hcd);
+ pm_runtime_put_sync(&dev->dev);
+ pm_runtime_disable(&dev->dev);
+
if (pdata == &ohci_platform_defaults)
dev->dev.platform_data = NULL;
static const struct of_device_id ohci_platform_ids[] = {
{ .compatible = "generic-ohci", },
{ .compatible = "cavium,octeon-6335-ohci", },
+ { .compatible = "ti,ohci-omap3", },
{ }
};
MODULE_DEVICE_TABLE(of, ohci_platform_ids);
static int pxa27x_start_hc(struct pxa27x_ohci *pxa_ohci, struct device *dev)
{
- int retval = 0;
+ int retval;
struct pxaohci_platform_data *inf;
uint32_t uhchr;
struct usb_hcd *hcd = dev_get_drvdata(dev);
inf = dev_get_platdata(dev);
- clk_prepare_enable(pxa_ohci->clk);
+ retval = clk_prepare_enable(pxa_ohci->clk);
+ if (retval)
+ return retval;
pxa27x_reset_hc(pxa_ohci);
if (inf->init)
retval = inf->init(dev);
- if (retval < 0)
+ if (retval < 0) {
+ clk_disable_unprepare(pxa_ohci->clk);
return retval;
+ }
if (cpu_is_pxa3xx())
pxa3xx_u2d_start_hc(&hcd->self);
static int resume_detect_interrupts_are_broken(struct uhci_hcd *uhci)
{
- /* If we have to ignore overcurrent events then almost by definition
- * we can't depend on resume-detect interrupts. */
- if (ignore_oc)
+ /*
+ * If we have to ignore overcurrent events then almost by definition
+ * we can't depend on resume-detect interrupts.
+ *
+ * Those interrupts also don't seem to work on ASpeed SoCs.
+ */
+ if (ignore_oc || uhci_is_aspeed(uhci))
return 1;
return uhci->resume_detect_interrupts_are_broken ?
{
uhci->is_stopped = 0;
+ /*
+ * Clear stale status bits on Aspeed as we get a stale HCH
+ * which causes problems later on
+ */
+ if (uhci_is_aspeed(uhci))
+ uhci_writew(uhci, uhci_readw(uhci, USBSTS), USBSTS);
+
/* Mark it configured and running with a 64-byte max packet.
* All interrupts are enabled, even though RESUME won't do anything.
*/
/* USB port status and control registers */
#define USBPORTSC1 16
#define USBPORTSC2 18
+#define USBPORTSC3 20
+#define USBPORTSC4 22
#define USBPORTSC_CCS 0x0001 /* Current Connect Status
* ("device present") */
#define USBPORTSC_CSC 0x0002 /* Connect Status Change */
unsigned int wait_for_hp:1; /* Wait for HP port reset */
unsigned int big_endian_mmio:1; /* Big endian registers */
unsigned int big_endian_desc:1; /* Big endian descriptors */
+ unsigned int is_aspeed:1; /* Aspeed impl. workarounds */
/* Support for port suspend/resume/reset */
unsigned long port_c_suspend; /* Bit-arrays of ports */
#define PCI_VENDOR_ID_GENESYS 0x17a0
#define PCI_DEVICE_ID_GL880S_UHCI 0x8083
+/* Aspeed SoC needs some quirks */
+static inline bool uhci_is_aspeed(const struct uhci_hcd *uhci)
+{
+ return IS_ENABLED(CONFIG_USB_UHCI_ASPEED) && uhci->is_aspeed;
+}
+
/*
* Functions used to access controller registers. The UCHI spec says that host
* controller I/O registers are mapped into PCI I/O space. For non-PCI hosts
#define uhci_big_endian_mmio(u) 0
#endif
+static inline int uhci_aspeed_reg(unsigned int reg)
+{
+ switch (reg) {
+ case USBCMD:
+ return 00;
+ case USBSTS:
+ return 0x04;
+ case USBINTR:
+ return 0x08;
+ case USBFRNUM:
+ return 0x80;
+ case USBFLBASEADD:
+ return 0x0c;
+ case USBSOF:
+ return 0x84;
+ case USBPORTSC1:
+ return 0x88;
+ case USBPORTSC2:
+ return 0x8c;
+ case USBPORTSC3:
+ return 0x90;
+ case USBPORTSC4:
+ return 0x94;
+ default:
+ pr_warn("UHCI: Unsupported register 0x%02x on Aspeed\n", reg);
+ /* Return an unimplemented register */
+ return 0x10;
+ }
+}
+
static inline u32 uhci_readl(const struct uhci_hcd *uhci, int reg)
{
if (uhci_has_pci_registers(uhci))
return inl(uhci->io_addr + reg);
+ else if (uhci_is_aspeed(uhci))
+ return readl(uhci->regs + uhci_aspeed_reg(reg));
#ifdef CONFIG_USB_UHCI_BIG_ENDIAN_MMIO
else if (uhci_big_endian_mmio(uhci))
return readl_be(uhci->regs + reg);
{
if (uhci_has_pci_registers(uhci))
outl(val, uhci->io_addr + reg);
+ else if (uhci_is_aspeed(uhci))
+ writel(val, uhci->regs + uhci_aspeed_reg(reg));
#ifdef CONFIG_USB_UHCI_BIG_ENDIAN_MMIO
else if (uhci_big_endian_mmio(uhci))
writel_be(val, uhci->regs + reg);
{
if (uhci_has_pci_registers(uhci))
return inw(uhci->io_addr + reg);
+ else if (uhci_is_aspeed(uhci))
+ return readl(uhci->regs + uhci_aspeed_reg(reg));
#ifdef CONFIG_USB_UHCI_BIG_ENDIAN_MMIO
else if (uhci_big_endian_mmio(uhci))
return readw_be(uhci->regs + reg);
{
if (uhci_has_pci_registers(uhci))
outw(val, uhci->io_addr + reg);
+ else if (uhci_is_aspeed(uhci))
+ writel(val, uhci->regs + uhci_aspeed_reg(reg));
#ifdef CONFIG_USB_UHCI_BIG_ENDIAN_MMIO
else if (uhci_big_endian_mmio(uhci))
writew_be(val, uhci->regs + reg);
{
if (uhci_has_pci_registers(uhci))
return inb(uhci->io_addr + reg);
+ else if (uhci_is_aspeed(uhci))
+ return readl(uhci->regs + uhci_aspeed_reg(reg));
#ifdef CONFIG_USB_UHCI_BIG_ENDIAN_MMIO
else if (uhci_big_endian_mmio(uhci))
return readb_be(uhci->regs + reg);
{
if (uhci_has_pci_registers(uhci))
outb(val, uhci->io_addr + reg);
+ else if (uhci_is_aspeed(uhci))
+ writel(val, uhci->regs + uhci_aspeed_reg(reg));
#ifdef CONFIG_USB_UHCI_BIG_ENDIAN_MMIO
else if (uhci_big_endian_mmio(uhci))
writeb_be(val, uhci->regs + reg);
/* Intel controllers use non-PME wakeup signalling */
if (to_pci_dev(uhci_dev(uhci))->vendor == PCI_VENDOR_ID_INTEL)
- device_set_run_wake(uhci_dev(uhci), 1);
+ device_set_wakeup_capable(uhci_dev(uhci), true);
/* Set up pointers to PCI-specific functions */
uhci->reset_hc = uhci_pci_reset_hc;
{
struct uhci_hcd *uhci = hcd_to_uhci(hcd);
- uhci->rh_numports = uhci_count_ports(hcd);
+ /* Probe number of ports if not already provided by DT */
+ if (!uhci->rh_numports)
+ uhci->rh_numports = uhci_count_ports(hcd);
/* Set up pointers to to generic functions */
uhci->reset_hc = uhci_generic_reset_hc;
static int uhci_hcd_platform_probe(struct platform_device *pdev)
{
+ struct device_node *np = pdev->dev.of_node;
struct usb_hcd *hcd;
struct uhci_hcd *uhci;
struct resource *res;
uhci->regs = hcd->regs;
+ /* Grab some things from the device-tree */
+ if (np) {
+ u32 num_ports;
+
+ if (of_property_read_u32(np, "#ports", &num_ports) == 0) {
+ uhci->rh_numports = num_ports;
+ dev_info(&pdev->dev,
+ "Detected %d ports from device-tree\n",
+ num_ports);
+ }
+ if (of_device_is_compatible(np, "aspeed,ast2400-uhci") ||
+ of_device_is_compatible(np, "aspeed,ast2500-uhci")) {
+ uhci->is_aspeed = 1;
+ dev_info(&pdev->dev,
+ "Enabled Aspeed implementation workarounds\n");
+ }
+ }
ret = usb_add_hcd(hcd, pdev->resource[1].start, IRQF_SHARED);
if (ret)
goto err_rmr;
return NULL;
}
-void xhci_free_or_cache_endpoint_ring(struct xhci_hcd *xhci,
+void xhci_free_endpoint_ring(struct xhci_hcd *xhci,
struct xhci_virt_device *virt_dev,
unsigned int ep_index)
{
- int rings_cached;
-
- rings_cached = virt_dev->num_rings_cached;
- if (rings_cached < XHCI_MAX_RINGS_CACHED) {
- virt_dev->ring_cache[rings_cached] =
- virt_dev->eps[ep_index].ring;
- virt_dev->num_rings_cached++;
- xhci_dbg(xhci, "Cached old ring, "
- "%d ring%s cached\n",
- virt_dev->num_rings_cached,
- (virt_dev->num_rings_cached > 1) ? "s" : "");
- } else {
- xhci_ring_free(xhci, virt_dev->eps[ep_index].ring);
- xhci_dbg(xhci, "Ring cache full (%d rings), "
- "freeing ring\n",
- virt_dev->num_rings_cached);
- }
+ xhci_ring_free(xhci, virt_dev->eps[ep_index].ring);
virt_dev->eps[ep_index].ring = NULL;
}
-/* Zero an endpoint ring (except for link TRBs) and move the enqueue and dequeue
- * pointers to the beginning of the ring.
- */
-static void xhci_reinit_cached_ring(struct xhci_hcd *xhci,
- struct xhci_ring *ring, unsigned int cycle_state,
- enum xhci_ring_type type)
-{
- struct xhci_segment *seg = ring->first_seg;
- int i;
-
- do {
- memset(seg->trbs, 0,
- sizeof(union xhci_trb)*TRBS_PER_SEGMENT);
- if (cycle_state == 0) {
- for (i = 0; i < TRBS_PER_SEGMENT; i++)
- seg->trbs[i].link.control |=
- cpu_to_le32(TRB_CYCLE);
- }
- /* All endpoint rings have link TRBs */
- xhci_link_segments(xhci, seg, seg->next, type);
- seg = seg->next;
- } while (seg != ring->first_seg);
- ring->type = type;
- xhci_initialize_ring_info(ring, cycle_state);
- /* td list should be empty since all URBs have been cancelled,
- * but just in case...
- */
- INIT_LIST_HEAD(&ring->td_list);
-}
-
/*
* Expand an existing ring.
- * Look for a cached ring or allocate a new ring which has same segment numbers
- * and link the two rings.
+ * Allocate a new ring which has same segment numbers and link the two rings.
*/
int xhci_ring_expansion(struct xhci_hcd *xhci, struct xhci_ring *ring,
unsigned int num_trbs, gfp_t flags)
/* If necessary, update the number of active TTs on this root port */
xhci_update_tt_active_eps(xhci, dev, old_active_eps);
- if (dev->ring_cache) {
- for (i = 0; i < dev->num_rings_cached; i++)
- xhci_ring_free(xhci, dev->ring_cache[i]);
- kfree(dev->ring_cache);
- }
-
if (dev->in_ctx)
xhci_free_container_ctx(xhci, dev->in_ctx);
if (dev->out_ctx)
if (!dev->eps[0].ring)
goto fail;
- /* Allocate pointers to the ring cache */
- dev->ring_cache = kzalloc(
- sizeof(struct xhci_ring *)*XHCI_MAX_RINGS_CACHED,
- flags);
- if (!dev->ring_cache)
- goto fail;
- dev->num_rings_cached = 0;
-
dev->udev = udev;
/* Point to output device context in dcbaa. */
/* Set up the endpoint ring */
virt_dev->eps[ep_index].new_ring =
xhci_ring_alloc(xhci, 2, 1, ring_type, max_packet, mem_flags);
- if (!virt_dev->eps[ep_index].new_ring) {
- /* Attempt to use the ring cache */
- if (virt_dev->num_rings_cached == 0)
- return -ENOMEM;
- virt_dev->num_rings_cached--;
- virt_dev->eps[ep_index].new_ring =
- virt_dev->ring_cache[virt_dev->num_rings_cached];
- virt_dev->ring_cache[virt_dev->num_rings_cached] = NULL;
- xhci_reinit_cached_ring(xhci, virt_dev->eps[ep_index].new_ring,
- 1, ring_type);
- }
+ if (!virt_dev->eps[ep_index].new_ring)
+ return -ENOMEM;
+
virt_dev->eps[ep_index].skip = false;
ep_ring = virt_dev->eps[ep_index].new_ring;
#ifdef CONFIG_ACPI
static void xhci_pme_acpi_rtd3_enable(struct pci_dev *dev)
{
- static const u8 intel_dsm_uuid[] = {
- 0xb7, 0x0c, 0x34, 0xac, 0x01, 0xe9, 0xbf, 0x45,
- 0xb7, 0xe6, 0x2b, 0x34, 0xec, 0x93, 0x1e, 0x23,
- };
+ static const guid_t intel_dsm_guid =
+ GUID_INIT(0xac340cb7, 0xe901, 0x45bf,
+ 0xb7, 0xe6, 0x2b, 0x34, 0xec, 0x93, 0x1e, 0x23);
union acpi_object *obj;
- obj = acpi_evaluate_dsm(ACPI_HANDLE(&dev->dev), intel_dsm_uuid, 3, 1,
+ obj = acpi_evaluate_dsm(ACPI_HANDLE(&dev->dev), &intel_dsm_guid, 3, 1,
NULL);
ACPI_FREE(obj);
}
return NULL;
}
+
+/*
+ * Get the hw dequeue pointer xHC stopped on, either directly from the
+ * endpoint context, or if streams are in use from the stream context.
+ * The returned hw_dequeue contains the lowest four bits with cycle state
+ * and possbile stream context type.
+ */
+static u64 xhci_get_hw_deq(struct xhci_hcd *xhci, struct xhci_virt_device *vdev,
+ unsigned int ep_index, unsigned int stream_id)
+{
+ struct xhci_ep_ctx *ep_ctx;
+ struct xhci_stream_ctx *st_ctx;
+ struct xhci_virt_ep *ep;
+
+ ep = &vdev->eps[ep_index];
+
+ if (ep->ep_state & EP_HAS_STREAMS) {
+ st_ctx = &ep->stream_info->stream_ctx_array[stream_id];
+ return le64_to_cpu(st_ctx->stream_ring);
+ }
+ ep_ctx = xhci_get_ep_ctx(xhci, vdev->out_ctx, ep_index);
+ return le64_to_cpu(ep_ctx->deq);
+}
+
/*
* Move the xHC's endpoint ring dequeue pointer past cur_td.
* Record the new state of the xHC's endpoint ring dequeue segment,
- * dequeue pointer, and new consumer cycle state in state.
+ * dequeue pointer, stream id, and new consumer cycle state in state.
* Update our internal representation of the ring's dequeue pointer.
*
* We do this in three jumps:
stream_id);
return;
}
-
/* Dig out the cycle state saved by the xHC during the stop ep cmd */
xhci_dbg_trace(xhci, trace_xhci_dbg_cancel_urb,
"Finding endpoint context");
- /* 4.6.9 the css flag is written to the stream context for streams */
- if (ep->ep_state & EP_HAS_STREAMS) {
- struct xhci_stream_ctx *ctx =
- &ep->stream_info->stream_ctx_array[stream_id];
- hw_dequeue = le64_to_cpu(ctx->stream_ring);
- } else {
- struct xhci_ep_ctx *ep_ctx
- = xhci_get_ep_ctx(xhci, dev->out_ctx, ep_index);
- hw_dequeue = le64_to_cpu(ep_ctx->deq);
- }
+ hw_dequeue = xhci_get_hw_deq(xhci, dev, ep_index, stream_id);
new_seg = ep_ring->deq_seg;
new_deq = ep_ring->dequeue;
state->new_cycle_state = hw_dequeue & 0x1;
+ state->stream_id = stream_id;
/*
* We want to find the pointer, segment and cycle state of the new trb
struct xhci_td *last_unlinked_td;
struct xhci_ep_ctx *ep_ctx;
struct xhci_virt_device *vdev;
-
+ u64 hw_deq;
struct xhci_dequeue_state deq_state;
if (unlikely(TRB_TO_SUSPEND_PORT(le32_to_cpu(trb->generic.field[3])))) {
if (list_empty(&ep->cancelled_td_list)) {
xhci_stop_watchdog_timer_in_irq(xhci, ep);
- ep->stopped_td = NULL;
ring_doorbell_for_active_rings(xhci, slot_id, ep_index);
return;
}
* If we stopped on the TD we need to cancel, then we have to
* move the xHC endpoint ring dequeue pointer past this TD.
*/
- if (cur_td == ep->stopped_td)
+ hw_deq = xhci_get_hw_deq(xhci, vdev, ep_index,
+ cur_td->urb->stream_id);
+ hw_deq &= ~0xf;
+
+ if (trb_in_td(xhci, cur_td->start_seg, cur_td->first_trb,
+ cur_td->last_trb, hw_deq, false)) {
xhci_find_new_dequeue_state(xhci, slot_id, ep_index,
- cur_td->urb->stream_id,
- cur_td, &deq_state);
- else
+ cur_td->urb->stream_id,
+ cur_td, &deq_state);
+ } else {
td_to_noop(xhci, ep_ring, cur_td, false);
+ }
+
remove_finished_td:
/*
* The event handler won't see a completion for this TD anymore,
/* If necessary, queue a Set Transfer Ring Dequeue Pointer command */
if (deq_state.new_deq_ptr && deq_state.new_deq_seg) {
xhci_queue_new_dequeue_state(xhci, slot_id, ep_index,
- ep->stopped_td->urb->stream_id, &deq_state);
+ &deq_state);
xhci_ring_cmd_db(xhci);
} else {
/* Otherwise ring the doorbell(s) to restart queued transfers */
ring_doorbell_for_active_rings(xhci, slot_id, ep_index);
}
- ep->stopped_td = NULL;
-
/*
* Drop the lock and complete the URBs in the cancelled TD list.
* New TDs to be cancelled might be added to the end of the list before
static void xhci_cleanup_halted_endpoint(struct xhci_hcd *xhci,
unsigned int slot_id, unsigned int ep_index,
unsigned int stream_id,
- struct xhci_td *td, union xhci_trb *ep_trb)
+ struct xhci_td *td, union xhci_trb *ep_trb,
+ enum xhci_ep_reset_type reset_type)
{
struct xhci_virt_ep *ep = &xhci->devs[slot_id]->eps[ep_index];
struct xhci_command *command;
return;
ep->ep_state |= EP_HALTED;
- ep->stopped_stream = stream_id;
- xhci_queue_reset_ep(xhci, command, slot_id, ep_index);
- xhci_cleanup_stalled_ring(xhci, ep_index, td);
+ xhci_queue_reset_ep(xhci, command, slot_id, ep_index, reset_type);
- ep->stopped_stream = 0;
+ if (reset_type == EP_HARD_RESET)
+ xhci_cleanup_stalled_ring(xhci, ep_index, stream_id, td);
xhci_ring_cmd_db(xhci);
}
static int finish_td(struct xhci_hcd *xhci, struct xhci_td *td,
union xhci_trb *ep_trb, struct xhci_transfer_event *event,
- struct xhci_virt_ep *ep, int *status, bool skip)
+ struct xhci_virt_ep *ep, int *status)
{
struct xhci_virt_device *xdev;
struct xhci_ep_ctx *ep_ctx;
ep_ctx = xhci_get_ep_ctx(xhci, xdev->out_ctx, ep_index);
trb_comp_code = GET_COMP_CODE(le32_to_cpu(event->transfer_len));
- if (skip)
- goto td_cleanup;
-
if (trb_comp_code == COMP_STOPPED_LENGTH_INVALID ||
trb_comp_code == COMP_STOPPED ||
trb_comp_code == COMP_STOPPED_SHORT_PACKET) {
* stopped TDs. A stopped TD may be restarted, so don't update
* the ring dequeue pointer or take this TD off any lists yet.
*/
- ep->stopped_td = td;
return 0;
}
if (trb_comp_code == COMP_STALL_ERROR ||
* The class driver clears the device side halt later.
*/
xhci_cleanup_halted_endpoint(xhci, slot_id, ep_index,
- ep_ring->stream_id, td, ep_trb);
+ ep_ring->stream_id, td, ep_trb,
+ EP_HARD_RESET);
} else {
/* Update ring dequeue pointer */
while (ep_ring->dequeue != td->last_trb)
inc_deq(xhci, ep_ring);
}
-td_cleanup:
return xhci_td_cleanup(xhci, td, ep_ring, status);
}
td->urb->actual_length = requested;
finish_td:
- return finish_td(xhci, td, ep_trb, event, ep, status, false);
+ return finish_td(xhci, td, ep_trb, event, ep, status);
}
/*
td->urb->actual_length += frame->actual_length;
- return finish_td(xhci, td, ep_trb, event, ep, status, false);
+ return finish_td(xhci, td, ep_trb, event, ep, status);
}
static int skip_isoc_td(struct xhci_hcd *xhci, struct xhci_td *td,
inc_deq(xhci, ep_ring);
inc_deq(xhci, ep_ring);
- return finish_td(xhci, td, NULL, event, ep, status, true);
+ return xhci_td_cleanup(xhci, td, ep_ring, status);
}
/*
remaining);
td->urb->actual_length = 0;
}
- return finish_td(xhci, td, ep_trb, event, ep, status, false);
+ return finish_td(xhci, td, ep_trb, event, ep, status);
}
/*
bool handling_skipped_tds = false;
slot_id = TRB_TO_SLOT_ID(le32_to_cpu(event->flags));
+ ep_index = TRB_TO_EP_ID(le32_to_cpu(event->flags)) - 1;
+ trb_comp_code = GET_COMP_CODE(le32_to_cpu(event->transfer_len));
+ ep_trb_dma = le64_to_cpu(event->buffer);
+
xdev = xhci->devs[slot_id];
if (!xdev) {
xhci_err(xhci, "ERROR Transfer event pointed to bad slot %u\n",
slot_id);
- xhci_err(xhci, "@%016llx %08x %08x %08x %08x\n",
- (unsigned long long) xhci_trb_virt_to_dma(
- xhci->event_ring->deq_seg,
- xhci->event_ring->dequeue),
- lower_32_bits(le64_to_cpu(event->buffer)),
- upper_32_bits(le64_to_cpu(event->buffer)),
- le32_to_cpu(event->transfer_len),
- le32_to_cpu(event->flags));
- return -ENODEV;
- }
-
- /* Endpoint ID is 1 based, our index is zero based */
- ep_index = TRB_TO_EP_ID(le32_to_cpu(event->flags)) - 1;
+ goto err_out;
+ }
+
ep = &xdev->eps[ep_index];
- ep_ring = xhci_dma_to_transfer_ring(ep, le64_to_cpu(event->buffer));
+ ep_ring = xhci_dma_to_transfer_ring(ep, ep_trb_dma);
ep_ctx = xhci_get_ep_ctx(xhci, xdev->out_ctx, ep_index);
- if (!ep_ring || GET_EP_CTX_STATE(ep_ctx) == EP_STATE_DISABLED) {
+
+ if (GET_EP_CTX_STATE(ep_ctx) == EP_STATE_DISABLED) {
xhci_err(xhci,
- "ERROR Transfer event for disabled endpoint slot %u ep %u or incorrect stream ring\n",
+ "ERROR Transfer event for disabled endpoint slot %u ep %u\n",
slot_id, ep_index);
- xhci_err(xhci, "@%016llx %08x %08x %08x %08x\n",
- (unsigned long long) xhci_trb_virt_to_dma(
- xhci->event_ring->deq_seg,
- xhci->event_ring->dequeue),
- lower_32_bits(le64_to_cpu(event->buffer)),
- upper_32_bits(le64_to_cpu(event->buffer)),
- le32_to_cpu(event->transfer_len),
- le32_to_cpu(event->flags));
- return -ENODEV;
+ goto err_out;
+ }
+
+ /* Some transfer events don't always point to a trb, see xhci 4.17.4 */
+ if (!ep_ring) {
+ switch (trb_comp_code) {
+ case COMP_STALL_ERROR:
+ case COMP_USB_TRANSACTION_ERROR:
+ case COMP_INVALID_STREAM_TYPE_ERROR:
+ case COMP_INVALID_STREAM_ID_ERROR:
+ xhci_cleanup_halted_endpoint(xhci, slot_id, ep_index, 0,
+ NULL, NULL, EP_SOFT_RESET);
+ goto cleanup;
+ case COMP_RING_UNDERRUN:
+ case COMP_RING_OVERRUN:
+ goto cleanup;
+ default:
+ xhci_err(xhci, "ERROR Transfer event for unknown stream ring slot %u ep %u\n",
+ slot_id, ep_index);
+ goto err_out;
+ }
}
/* Count current td numbers if ep->skip is set */
td_num++;
}
- ep_trb_dma = le64_to_cpu(event->buffer);
- trb_comp_code = GET_COMP_CODE(le32_to_cpu(event->transfer_len));
/* Look for common error cases */
switch (trb_comp_code) {
/* Skip codes that require special handling depending on
slot_id, ep_index);
case COMP_SHORT_PACKET:
break;
+ /* Completion codes for endpoint stopped state */
case COMP_STOPPED:
xhci_dbg(xhci, "Stopped on Transfer TRB for slot %u ep %u\n",
slot_id, ep_index);
"Stopped with short packet transfer detected for slot %u ep %u\n",
slot_id, ep_index);
break;
+ /* Completion codes for endpoint halted state */
case COMP_STALL_ERROR:
xhci_dbg(xhci, "Stalled endpoint for slot %u ep %u\n", slot_id,
ep_index);
ep->ep_state |= EP_HALTED;
status = -EPIPE;
break;
- case COMP_TRB_ERROR:
- xhci_warn(xhci,
- "WARN: TRB error for slot %u ep %u on endpoint\n",
- slot_id, ep_index);
- status = -EILSEQ;
- break;
case COMP_SPLIT_TRANSACTION_ERROR:
case COMP_USB_TRANSACTION_ERROR:
xhci_dbg(xhci, "Transfer error for slot %u ep %u on endpoint\n",
slot_id, ep_index);
status = -EOVERFLOW;
break;
+ /* Completion codes for endpoint error state */
+ case COMP_TRB_ERROR:
+ xhci_warn(xhci,
+ "WARN: TRB error for slot %u ep %u on endpoint\n",
+ slot_id, ep_index);
+ status = -EILSEQ;
+ break;
+ /* completion codes not indicating endpoint state change */
case COMP_DATA_BUFFER_ERROR:
xhci_warn(xhci,
"WARN: HC couldn't access mem fast enough for slot %u ep %u\n",
TRB_TO_SLOT_ID(le32_to_cpu(event->flags)),
ep_index);
goto cleanup;
- case COMP_INCOMPATIBLE_DEVICE_ERROR:
- xhci_warn(xhci,
- "WARN: detect an incompatible device for slot %u ep %u",
- slot_id, ep_index);
- status = -EPROTO;
- break;
case COMP_MISSED_SERVICE_ERROR:
/*
* When encounter missed service error, one or more isoc tds
"No Ping response error for slot %u ep %u, Skip one Isoc TD\n",
slot_id, ep_index);
goto cleanup;
+
+ case COMP_INCOMPATIBLE_DEVICE_ERROR:
+ /* needs disable slot command to recover */
+ xhci_warn(xhci,
+ "WARN: detect an incompatible device for slot %u ep %u",
+ slot_id, ep_index);
+ status = -EPROTO;
+ break;
default:
if (xhci_is_vendor_info_code(xhci, trb_comp_code)) {
status = 0;
} while (handling_skipped_tds);
return 0;
+
+err_out:
+ xhci_err(xhci, "@%016llx %08x %08x %08x %08x\n",
+ (unsigned long long) xhci_trb_virt_to_dma(
+ xhci->event_ring->deq_seg,
+ xhci->event_ring->dequeue),
+ lower_32_bits(le64_to_cpu(event->buffer)),
+ upper_32_bits(le64_to_cpu(event->buffer)),
+ le32_to_cpu(event->transfer_len),
+ le32_to_cpu(event->flags));
+ return -ENODEV;
}
/*
/* Set Transfer Ring Dequeue Pointer command */
void xhci_queue_new_dequeue_state(struct xhci_hcd *xhci,
unsigned int slot_id, unsigned int ep_index,
- unsigned int stream_id,
struct xhci_dequeue_state *deq_state)
{
dma_addr_t addr;
u32 trb_slot_id = SLOT_ID_FOR_TRB(slot_id);
u32 trb_ep_index = EP_ID_FOR_TRB(ep_index);
- u32 trb_stream_id = STREAM_ID_FOR_TRB(stream_id);
+ u32 trb_stream_id = STREAM_ID_FOR_TRB(deq_state->stream_id);
u32 trb_sct = 0;
u32 type = TRB_TYPE(TRB_SET_DEQ);
struct xhci_virt_ep *ep;
ep->queued_deq_seg = deq_state->new_deq_seg;
ep->queued_deq_ptr = deq_state->new_deq_ptr;
- if (stream_id)
+ if (deq_state->stream_id)
trb_sct = SCT_FOR_TRB(SCT_PRI_TR);
ret = queue_command(xhci, cmd,
lower_32_bits(addr) | trb_sct | deq_state->new_cycle_state,
}
int xhci_queue_reset_ep(struct xhci_hcd *xhci, struct xhci_command *cmd,
- int slot_id, unsigned int ep_index)
+ int slot_id, unsigned int ep_index,
+ enum xhci_ep_reset_type reset_type)
{
u32 trb_slot_id = SLOT_ID_FOR_TRB(slot_id);
u32 trb_ep_index = EP_ID_FOR_TRB(ep_index);
u32 type = TRB_TYPE(TRB_RESET_EP);
+ if (reset_type == EP_SOFT_RESET)
+ type |= TRB_TSP;
+
return queue_command(xhci, cmd, 0, 0, 0,
trb_slot_id | trb_ep_index | type, false);
}
if (xhci->quirks & XHCI_MTK_HOST) {
ret = xhci_mtk_add_ep_quirk(hcd, udev, ep);
if (ret < 0) {
- xhci_free_or_cache_endpoint_ring(xhci,
- virt_dev, ep_index);
+ xhci_free_endpoint_ring(xhci, virt_dev, ep_index);
return ret;
}
}
for (i = 1; i < 31; i++) {
if ((le32_to_cpu(ctrl_ctx->drop_flags) & (1 << (i + 1))) &&
!(le32_to_cpu(ctrl_ctx->add_flags) & (1 << (i + 1)))) {
- xhci_free_or_cache_endpoint_ring(xhci, virt_dev, i);
+ xhci_free_endpoint_ring(xhci, virt_dev, i);
xhci_check_bw_drop_ep_streams(xhci, virt_dev, i);
}
}
xhci_zero_in_ctx(xhci, virt_dev);
/*
* Install any rings for completely new endpoints or changed endpoints,
- * and free or cache any old rings from changed endpoints.
+ * and free any old rings from changed endpoints.
*/
for (i = 1; i < 31; i++) {
if (!virt_dev->eps[i].new_ring)
continue;
- /* Only cache or free the old ring if it exists.
+ /* Only free the old ring if it exists.
* It may not if this is the first add of an endpoint.
*/
if (virt_dev->eps[i].ring) {
- xhci_free_or_cache_endpoint_ring(xhci, virt_dev, i);
+ xhci_free_endpoint_ring(xhci, virt_dev, i);
}
xhci_check_bw_drop_ep_streams(xhci, virt_dev, i);
virt_dev->eps[i].ring = virt_dev->eps[i].new_ring;
added_ctxs, added_ctxs);
}
-void xhci_cleanup_stalled_ring(struct xhci_hcd *xhci,
- unsigned int ep_index, struct xhci_td *td)
+void xhci_cleanup_stalled_ring(struct xhci_hcd *xhci, unsigned int ep_index,
+ unsigned int stream_id, struct xhci_td *td)
{
struct xhci_dequeue_state deq_state;
struct xhci_virt_ep *ep;
* or it will attempt to resend it on the next doorbell ring.
*/
xhci_find_new_dequeue_state(xhci, udev->slot_id,
- ep_index, ep->stopped_stream, td, &deq_state);
+ ep_index, stream_id, td, &deq_state);
if (!deq_state.new_deq_ptr || !deq_state.new_deq_seg)
return;
xhci_dbg_trace(xhci, trace_xhci_dbg_reset_ep,
"Queueing new dequeue state");
xhci_queue_new_dequeue_state(xhci, udev->slot_id,
- ep_index, ep->stopped_stream, &deq_state);
+ ep_index, &deq_state);
} else {
/* Better hope no one uses the input context between now and the
* reset endpoint completion!
*
* Wait for the Reset Device command to finish. Remove all structures
* associated with the endpoints that were disabled. Clear the input device
- * structure? Cache the rings? Reset the control endpoint 0 max packet size?
+ * structure? Reset the control endpoint 0 max packet size?
*
* If the virt_dev to be reset does not exist or does not match the udev,
* it means the device is lost, possibly due to the xHC restore error and
spin_unlock_irqrestore(&xhci->lock, flags);
}
- /* Everything but endpoint 0 is disabled, so free or cache the rings. */
+ /* Everything but endpoint 0 is disabled, so free the rings. */
last_freed_endpoint = 1;
for (i = 1; i < 31; i++) {
struct xhci_virt_ep *ep = &virt_dev->eps[i];
}
if (ep->ring) {
- xhci_free_or_cache_endpoint_ring(xhci, virt_dev, i);
+ xhci_free_endpoint_ring(xhci, virt_dev, i);
last_freed_endpoint = i;
}
if (!list_empty(&virt_dev->eps[i].bw_endpoint_list))
#define EP_GETTING_NO_STREAMS (1 << 5)
/* ---- Related to URB cancellation ---- */
struct list_head cancelled_td_list;
- struct xhci_td *stopped_td;
- unsigned int stopped_stream;
/* Watchdog timer for stop endpoint command to cancel URBs */
struct timer_list stop_cmd_timer;
struct xhci_hcd *xhci;
struct xhci_container_ctx *out_ctx;
/* Used for addressing devices and configuration changes */
struct xhci_container_ctx *in_ctx;
- /* Rings saved to ensure old alt settings can be re-instated */
- struct xhci_ring **ring_cache;
- int num_rings_cached;
-#define XHCI_MAX_RINGS_CACHED 31
struct xhci_virt_ep eps[31];
u8 fake_port;
u8 real_port;
/* Stop Ring - Transfer State Preserve */
#define TRB_TSP (1<<9)
+enum xhci_ep_reset_type {
+ EP_HARD_RESET,
+ EP_SOFT_RESET,
+};
+
/* Force Event */
#define TRB_TO_VF_INTR_TARGET(p) (((p) & (0x3ff << 22)) >> 22)
#define TRB_TO_VF_ID(p) (((p) & (0xff << 16)) >> 16)
struct xhci_segment *new_deq_seg;
union xhci_trb *new_deq_ptr;
int new_cycle_state;
+ unsigned int stream_id;
};
enum xhci_ring_type {
void xhci_ring_free(struct xhci_hcd *xhci, struct xhci_ring *ring);
int xhci_ring_expansion(struct xhci_hcd *xhci, struct xhci_ring *ring,
unsigned int num_trbs, gfp_t flags);
-void xhci_free_or_cache_endpoint_ring(struct xhci_hcd *xhci,
+void xhci_free_endpoint_ring(struct xhci_hcd *xhci,
struct xhci_virt_device *virt_dev,
unsigned int ep_index);
struct xhci_stream_info *xhci_alloc_stream_info(struct xhci_hcd *xhci,
int xhci_queue_evaluate_context(struct xhci_hcd *xhci, struct xhci_command *cmd,
dma_addr_t in_ctx_ptr, u32 slot_id, bool command_must_succeed);
int xhci_queue_reset_ep(struct xhci_hcd *xhci, struct xhci_command *cmd,
- int slot_id, unsigned int ep_index);
+ int slot_id, unsigned int ep_index,
+ enum xhci_ep_reset_type reset_type);
int xhci_queue_reset_device(struct xhci_hcd *xhci, struct xhci_command *cmd,
u32 slot_id);
void xhci_find_new_dequeue_state(struct xhci_hcd *xhci,
struct xhci_dequeue_state *state);
void xhci_queue_new_dequeue_state(struct xhci_hcd *xhci,
unsigned int slot_id, unsigned int ep_index,
- unsigned int stream_id,
struct xhci_dequeue_state *deq_state);
-void xhci_cleanup_stalled_ring(struct xhci_hcd *xhci,
- unsigned int ep_index, struct xhci_td *td);
+void xhci_cleanup_stalled_ring(struct xhci_hcd *xhci, unsigned int ep_index,
+ unsigned int stream_id, struct xhci_td *td);
void xhci_stop_endpoint_command_watchdog(unsigned long arg);
void xhci_handle_command_timeout(struct work_struct *work);
To compile this driver as a module, choose M here: the
module will be called chaoskey.
-
-config UCSI
- tristate "USB Type-C Connector System Software Interface driver"
- depends on ACPI
- help
- UCSI driver is meant to be used as a convenience tool for desktop and
- server systems that are not equipped to handle USB in device mode. It
- will always select USB host role for the USB Type-C ports on systems
- that provide UCSI interface.
-
- USB Type-C Connector System Software Interface (UCSI) is a
- specification for an interface that allows the Operating System to
- control the USB Type-C ports on a system. Things the need controlling
- include the USB Data Role (host or device), and when USB Power
- Delivery is supported, the Power Role (source or sink). With USB
- Type-C connectors, when two dual role capable devices are attached
- together, the data role is selected randomly. Therefore it is
- important to give the OS a way to select the role. Otherwise the user
- would have to unplug and replug in order in order to attempt to swap
- the data and power roles.
-
- The UCSI specification can be downloaded from:
- http://www.intel.com/content/www/us/en/io/universal-serial-bus/usb-type-c-ucsi-spec.html
-
- To compile the driver as a module, choose M here: the module will be
- called ucsi.
obj-$(CONFIG_USB_HSIC_USB3503) += usb3503.o
obj-$(CONFIG_USB_HSIC_USB4604) += usb4604.o
obj-$(CONFIG_USB_CHAOSKEY) += chaoskey.o
-obj-$(CONFIG_UCSI) += ucsi.o
obj-$(CONFIG_USB_SISUSBVGA) += sisusbvga/
obj-$(CONFIG_USB_LINK_LAYER_TEST) += lvstest.o
case USB_DEVICE_ID_CODEMERCS_IOWPV2:
case USB_DEVICE_ID_CODEMERCS_IOW40:
/* IOW24 and IOW40 use a synchronous call */
- buf = kmalloc(count, GFP_KERNEL);
- if (!buf) {
- retval = -ENOMEM;
- goto exit;
- }
- if (copy_from_user(buf, user_buffer, count)) {
- retval = -EFAULT;
- kfree(buf);
+ buf = memdup_user(user_buffer, count);
+ if (IS_ERR(buf)) {
+ retval = PTR_ERR(buf);
goto exit;
}
retval = usb_set_report(dev->interface, 2, 0, buf, count);
+++ /dev/null
-/*
- * USB Type-C Connector System Software Interface driver
- *
- * Copyright (C) 2016, Intel Corporation
- * Author: Heikki Krogerus <heikki.krogerus@linux.intel.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/platform_device.h>
-#include <linux/module.h>
-#include <linux/delay.h>
-#include <linux/acpi.h>
-
-#include "ucsi.h"
-
-/* Double the time defined by MIN_TIME_TO_RESPOND_WITH_BUSY */
-#define UCSI_TIMEOUT_MS 20
-
-enum ucsi_status {
- UCSI_IDLE = 0,
- UCSI_BUSY,
- UCSI_ERROR,
-};
-
-struct ucsi_connector {
- int num;
- struct ucsi *ucsi;
- struct work_struct work;
- struct ucsi_connector_capability cap;
-};
-
-struct ucsi {
- struct device *dev;
- struct ucsi_data __iomem *data;
-
- enum ucsi_status status;
- struct completion complete;
- struct ucsi_capability cap;
- struct ucsi_connector *connector;
-
- /* device lock */
- spinlock_t dev_lock;
-
- /* PPM Communication lock */
- struct mutex ppm_lock;
-
- /* PPM communication flags */
- unsigned long flags;
-#define EVENT_PENDING 0
-#define COMMAND_PENDING 1
-};
-
-static int ucsi_acpi_cmd(struct ucsi *ucsi, struct ucsi_control *ctrl)
-{
- uuid_le uuid = UUID_LE(0x6f8398c2, 0x7ca4, 0x11e4,
- 0xad, 0x36, 0x63, 0x10, 0x42, 0xb5, 0x00, 0x8f);
- union acpi_object *obj;
-
- ucsi->data->ctrl.raw_cmd = ctrl->raw_cmd;
-
- obj = acpi_evaluate_dsm(ACPI_HANDLE(ucsi->dev), uuid.b, 1, 1, NULL);
- if (!obj) {
- dev_err(ucsi->dev, "%s: failed to evaluate _DSM\n", __func__);
- return -EIO;
- }
-
- ACPI_FREE(obj);
- return 0;
-}
-
-static void ucsi_acpi_notify(acpi_handle handle, u32 event, void *data)
-{
- struct ucsi *ucsi = data;
- struct ucsi_cci *cci;
-
- spin_lock(&ucsi->dev_lock);
-
- ucsi->status = UCSI_IDLE;
- cci = &ucsi->data->cci;
-
- /*
- * REVISIT: This is not documented behavior, but all known PPMs ACK
- * asynchronous events by sending notification with cleared CCI.
- */
- if (!ucsi->data->raw_cci) {
- if (test_bit(EVENT_PENDING, &ucsi->flags))
- complete(&ucsi->complete);
- else
- dev_WARN(ucsi->dev, "spurious notification\n");
- goto out_unlock;
- }
-
- if (test_bit(COMMAND_PENDING, &ucsi->flags)) {
- if (cci->busy) {
- ucsi->status = UCSI_BUSY;
- complete(&ucsi->complete);
-
- goto out_unlock;
- } else if (cci->ack_complete || cci->cmd_complete) {
- /* Error Indication is only valid with commands */
- if (cci->error && cci->cmd_complete)
- ucsi->status = UCSI_ERROR;
-
- ucsi->data->ctrl.raw_cmd = 0;
- complete(&ucsi->complete);
- }
- }
-
- if (cci->connector_change) {
- struct ucsi_connector *con;
-
- /*
- * This is workaround for buggy PPMs that create asynchronous
- * event notifications before OPM has enabled them.
- */
- if (!ucsi->connector)
- goto out_unlock;
-
- con = ucsi->connector + (cci->connector_change - 1);
-
- /*
- * PPM will not clear the connector specific bit in Connector
- * Change Indication field of CCI until the driver has ACK it,
- * and the driver can not ACK it before it has been processed.
- * The PPM will not generate new events before the first has
- * been acknowledged, even if they are for an other connector.
- * So only one event at a time.
- */
- if (!test_and_set_bit(EVENT_PENDING, &ucsi->flags))
- schedule_work(&con->work);
- }
-out_unlock:
- spin_unlock(&ucsi->dev_lock);
-}
-
-static int ucsi_ack(struct ucsi *ucsi, u8 cmd)
-{
- struct ucsi_control ctrl;
- int ret;
-
- ctrl.cmd.cmd = UCSI_ACK_CC_CI;
- ctrl.cmd.length = 0;
- ctrl.cmd.data = cmd;
- ret = ucsi_acpi_cmd(ucsi, &ctrl);
- if (ret)
- return ret;
-
- /* Waiting for ACK also with ACK CMD for now */
- ret = wait_for_completion_timeout(&ucsi->complete,
- msecs_to_jiffies(UCSI_TIMEOUT_MS));
- if (!ret)
- return -ETIMEDOUT;
- return 0;
-}
-
-static int ucsi_run_cmd(struct ucsi *ucsi, struct ucsi_control *ctrl,
- void *data, size_t size)
-{
- u16 err_value = 0;
- int ret;
-
- set_bit(COMMAND_PENDING, &ucsi->flags);
-
- ret = ucsi_acpi_cmd(ucsi, ctrl);
- if (ret)
- goto err_clear_flag;
-
- ret = wait_for_completion_timeout(&ucsi->complete,
- msecs_to_jiffies(UCSI_TIMEOUT_MS));
- if (!ret) {
- ret = -ETIMEDOUT;
- goto err_clear_flag;
- }
-
- switch (ucsi->status) {
- case UCSI_IDLE:
- if (data)
- memcpy(data, ucsi->data->message_in, size);
-
- ret = ucsi_ack(ucsi, UCSI_ACK_CMD);
- break;
- case UCSI_BUSY:
- /* The caller decides whether to cancel or not */
- ret = -EBUSY;
- goto err_clear_flag;
- case UCSI_ERROR:
- ret = ucsi_ack(ucsi, UCSI_ACK_CMD);
- if (ret)
- goto err_clear_flag;
-
- ctrl->cmd.cmd = UCSI_GET_ERROR_STATUS;
- ctrl->cmd.length = 0;
- ctrl->cmd.data = 0;
- ret = ucsi_acpi_cmd(ucsi, ctrl);
- if (ret)
- goto err_clear_flag;
-
- ret = wait_for_completion_timeout(&ucsi->complete,
- msecs_to_jiffies(UCSI_TIMEOUT_MS));
- if (!ret) {
- ret = -ETIMEDOUT;
- goto err_clear_flag;
- }
-
- memcpy(&err_value, ucsi->data->message_in, sizeof(err_value));
-
- /* Something has really gone wrong */
- if (WARN_ON(ucsi->status == UCSI_ERROR)) {
- ret = -ENODEV;
- goto err_clear_flag;
- }
-
- ret = ucsi_ack(ucsi, UCSI_ACK_CMD);
- if (ret)
- goto err_clear_flag;
-
- switch (err_value) {
- case UCSI_ERROR_INCOMPATIBLE_PARTNER:
- ret = -EOPNOTSUPP;
- break;
- case UCSI_ERROR_CC_COMMUNICATION_ERR:
- ret = -ECOMM;
- break;
- case UCSI_ERROR_CONTRACT_NEGOTIATION_FAIL:
- ret = -EIO;
- break;
- case UCSI_ERROR_DEAD_BATTERY:
- dev_warn(ucsi->dev, "Dead battery condition!\n");
- ret = -EPERM;
- break;
- /* The following mean a bug in this driver */
- case UCSI_ERROR_INVALID_CON_NUM:
- case UCSI_ERROR_UNREGONIZED_CMD:
- case UCSI_ERROR_INVALID_CMD_ARGUMENT:
- default:
- dev_warn(ucsi->dev,
- "%s: possible UCSI driver bug - error %hu\n",
- __func__, err_value);
- ret = -EINVAL;
- break;
- }
- break;
- }
- ctrl->raw_cmd = 0;
-err_clear_flag:
- clear_bit(COMMAND_PENDING, &ucsi->flags);
- return ret;
-}
-
-static void ucsi_connector_change(struct work_struct *work)
-{
- struct ucsi_connector *con = container_of(work, struct ucsi_connector,
- work);
- struct ucsi_connector_status constat;
- struct ucsi *ucsi = con->ucsi;
- struct ucsi_control ctrl;
- int ret;
-
- mutex_lock(&ucsi->ppm_lock);
-
- ctrl.cmd.cmd = UCSI_GET_CONNECTOR_STATUS;
- ctrl.cmd.length = 0;
- ctrl.cmd.data = con->num;
- ret = ucsi_run_cmd(con->ucsi, &ctrl, &constat, sizeof(constat));
- if (ret) {
- dev_err(ucsi->dev, "%s: failed to read connector status (%d)\n",
- __func__, ret);
- goto out_ack_event;
- }
-
- /* Ignoring disconnections and Alternate Modes */
- if (!constat.connected || !(constat.change &
- (UCSI_CONSTAT_PARTNER_CHANGE | UCSI_CONSTAT_CONNECT_CHANGE)) ||
- constat.partner_flags & UCSI_CONSTAT_PARTNER_FLAG_ALT_MODE)
- goto out_ack_event;
-
- /* If the partner got USB Host role, attempting swap */
- if (constat.partner_type & UCSI_CONSTAT_PARTNER_TYPE_DFP) {
- ctrl.uor.cmd = UCSI_SET_UOR;
- ctrl.uor.con_num = con->num;
- ctrl.uor.role = UCSI_UOR_ROLE_DFP;
-
- ret = ucsi_run_cmd(con->ucsi, &ctrl, NULL, 0);
- if (ret)
- dev_err(ucsi->dev, "%s: failed to swap role (%d)\n",
- __func__, ret);
- }
-out_ack_event:
- ucsi_ack(ucsi, UCSI_ACK_EVENT);
- clear_bit(EVENT_PENDING, &ucsi->flags);
- mutex_unlock(&ucsi->ppm_lock);
-}
-
-static int ucsi_reset_ppm(struct ucsi *ucsi)
-{
- int timeout = UCSI_TIMEOUT_MS;
- struct ucsi_control ctrl;
- int ret;
-
- memset(&ctrl, 0, sizeof(ctrl));
- ctrl.cmd.cmd = UCSI_PPM_RESET;
- ret = ucsi_acpi_cmd(ucsi, &ctrl);
- if (ret)
- return ret;
-
- /* There is no quarantee the PPM will ever set the RESET_COMPLETE bit */
- while (!ucsi->data->cci.reset_complete && timeout--)
- usleep_range(1000, 2000);
- return 0;
-}
-
-static int ucsi_init(struct ucsi *ucsi)
-{
- struct ucsi_connector *con;
- struct ucsi_control ctrl;
- int ret;
- int i;
-
- init_completion(&ucsi->complete);
- spin_lock_init(&ucsi->dev_lock);
- mutex_init(&ucsi->ppm_lock);
-
- /* Reset the PPM */
- ret = ucsi_reset_ppm(ucsi);
- if (ret)
- return ret;
-
- /*
- * REVISIT: Executing second reset to WA an issue seen on some of the
- * Broxton based platforms, where the first reset puts the PPM into a
- * state where it's unable to recognise some of the commands.
- */
- ret = ucsi_reset_ppm(ucsi);
- if (ret)
- return ret;
-
- mutex_lock(&ucsi->ppm_lock);
-
- /* Enable basic notifications */
- ctrl.cmd.cmd = UCSI_SET_NOTIFICATION_ENABLE;
- ctrl.cmd.length = 0;
- ctrl.cmd.data = UCSI_ENABLE_NTFY_CMD_COMPLETE | UCSI_ENABLE_NTFY_ERROR;
- ret = ucsi_run_cmd(ucsi, &ctrl, NULL, 0);
- if (ret)
- goto err_reset;
-
- /* Get PPM capabilities */
- ctrl.cmd.cmd = UCSI_GET_CAPABILITY;
- ret = ucsi_run_cmd(ucsi, &ctrl, &ucsi->cap, sizeof(ucsi->cap));
- if (ret)
- goto err_reset;
-
- if (!ucsi->cap.num_connectors) {
- ret = -ENODEV;
- goto err_reset;
- }
-
- ucsi->connector = devm_kcalloc(ucsi->dev, ucsi->cap.num_connectors,
- sizeof(*ucsi->connector), GFP_KERNEL);
- if (!ucsi->connector) {
- ret = -ENOMEM;
- goto err_reset;
- }
-
- for (i = 1, con = ucsi->connector; i < ucsi->cap.num_connectors + 1;
- i++, con++) {
- /* Get connector capability */
- ctrl.cmd.cmd = UCSI_GET_CONNECTOR_CAPABILITY;
- ctrl.cmd.data = i;
- ret = ucsi_run_cmd(ucsi, &ctrl, &con->cap, sizeof(con->cap));
- if (ret)
- goto err_reset;
-
- con->num = i;
- con->ucsi = ucsi;
- INIT_WORK(&con->work, ucsi_connector_change);
- }
-
- /* Enable all notifications */
- ctrl.cmd.cmd = UCSI_SET_NOTIFICATION_ENABLE;
- ctrl.cmd.data = UCSI_ENABLE_NTFY_ALL;
- ret = ucsi_run_cmd(ucsi, &ctrl, NULL, 0);
- if (ret < 0)
- goto err_reset;
-
- mutex_unlock(&ucsi->ppm_lock);
- return 0;
-err_reset:
- ucsi_reset_ppm(ucsi);
- mutex_unlock(&ucsi->ppm_lock);
- return ret;
-}
-
-static int ucsi_acpi_probe(struct platform_device *pdev)
-{
- struct resource *res;
- acpi_status status;
- struct ucsi *ucsi;
- int ret;
-
- ucsi = devm_kzalloc(&pdev->dev, sizeof(*ucsi), GFP_KERNEL);
- if (!ucsi)
- return -ENOMEM;
-
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- if (!res) {
- dev_err(&pdev->dev, "missing memory resource\n");
- return -ENODEV;
- }
-
- /*
- * NOTE: ACPI has claimed the memory region as it's also an Operation
- * Region. It's not possible to request it in the driver.
- */
- ucsi->data = devm_ioremap(&pdev->dev, res->start, resource_size(res));
- if (!ucsi->data)
- return -ENOMEM;
-
- ucsi->dev = &pdev->dev;
-
- status = acpi_install_notify_handler(ACPI_HANDLE(&pdev->dev),
- ACPI_ALL_NOTIFY,
- ucsi_acpi_notify, ucsi);
- if (ACPI_FAILURE(status))
- return -ENODEV;
-
- ret = ucsi_init(ucsi);
- if (ret) {
- acpi_remove_notify_handler(ACPI_HANDLE(&pdev->dev),
- ACPI_ALL_NOTIFY,
- ucsi_acpi_notify);
- return ret;
- }
-
- platform_set_drvdata(pdev, ucsi);
- return 0;
-}
-
-static int ucsi_acpi_remove(struct platform_device *pdev)
-{
- struct ucsi *ucsi = platform_get_drvdata(pdev);
-
- acpi_remove_notify_handler(ACPI_HANDLE(&pdev->dev),
- ACPI_ALL_NOTIFY, ucsi_acpi_notify);
-
- /* Make sure there are no events in the middle of being processed */
- if (wait_on_bit_timeout(&ucsi->flags, EVENT_PENDING,
- TASK_UNINTERRUPTIBLE,
- msecs_to_jiffies(UCSI_TIMEOUT_MS)))
- dev_WARN(ucsi->dev, "%s: Events still pending\n", __func__);
-
- ucsi_reset_ppm(ucsi);
- return 0;
-}
-
-static const struct acpi_device_id ucsi_acpi_match[] = {
- { "PNP0CA0", 0 },
- { },
-};
-MODULE_DEVICE_TABLE(acpi, ucsi_acpi_match);
-
-static struct platform_driver ucsi_acpi_platform_driver = {
- .driver = {
- .name = "ucsi_acpi",
- .acpi_match_table = ACPI_PTR(ucsi_acpi_match),
- },
- .probe = ucsi_acpi_probe,
- .remove = ucsi_acpi_remove,
-};
-
-module_platform_driver(ucsi_acpi_platform_driver);
-
-MODULE_AUTHOR("Heikki Krogerus <heikki.krogerus@linux.intel.com>");
-MODULE_LICENSE("GPL v2");
-MODULE_DESCRIPTION("USB Type-C System Software Interface (UCSI) driver");
MODULE_DEVICE_TABLE(usb, id_table);
/* the different text display modes the device is capable of */
-static char *display_textmodes[] = {"raw", "hex", "ascii", NULL};
+static const char *display_textmodes[] = {"raw", "hex", "ascii"};
struct usb_sevsegdev {
struct usb_device *udev;
buf[0] = 0;
- for (i = 0; display_textmodes[i]; i++) {
+ for (i = 0; i < ARRAY_SIZE(display_textmodes); i++) {
if (mydev->textmode == i) {
strcat(buf, " [");
strcat(buf, display_textmodes[i]);
struct usb_sevsegdev *mydev = usb_get_intfdata(intf);
int i;
- for (i = 0; display_textmodes[i]; i++) {
- if (sysfs_streq(display_textmodes[i], buf)) {
- mydev->textmode = i;
- update_display_visual(mydev, GFP_KERNEL);
- return count;
- }
- }
+ i = sysfs_match_string(display_textmodes, buf);
+ if (i < 0)
+ return i;
- return -EINVAL;
+ mydev->textmode = i;
+ update_display_visual(mydev, GFP_KERNEL);
+ return count;
}
static DEVICE_ATTR(textmode, S_IRUGO | S_IWUSR, show_attr_textmode, set_attr_textmode);
static inline struct mtu3_request *next_request(struct mtu3_ep *mep)
{
- struct list_head *queue = &mep->req_list;
-
- if (list_empty(queue))
- return NULL;
-
- return list_first_entry(queue, struct mtu3_request, list);
+ return list_first_entry_or_null(&mep->req_list, struct mtu3_request,
+ list);
}
static inline void mtu3_writel(void __iomem *base, u32 offset, u32 data)
{
struct platform_device *pdev = to_platform_device(dev);
struct ssusb_mtk *ssusb = platform_get_drvdata(pdev);
+ int ret;
dev_dbg(dev, "%s\n", __func__);
return 0;
ssusb_wakeup_disable(ssusb);
- clk_prepare_enable(ssusb->sys_clk);
- clk_prepare_enable(ssusb->ref_clk);
- ssusb_phy_power_on(ssusb);
+ ret = clk_prepare_enable(ssusb->sys_clk);
+ if (ret)
+ goto err_sys_clk;
+
+ ret = clk_prepare_enable(ssusb->ref_clk);
+ if (ret)
+ goto err_ref_clk;
+
+ ret = ssusb_phy_power_on(ssusb);
+ if (ret)
+ goto err_power_on;
+
ssusb_host_enable(ssusb);
return 0;
+
+err_power_on:
+ clk_disable_unprepare(ssusb->ref_clk);
+err_ref_clk:
+ clk_disable_unprepare(ssusb->sys_clk);
+err_sys_clk:
+ return ret;
}
static const struct dev_pm_ops mtu3_pm_ops = {
musb->io.ep_select = musb_flat_ep_select;
}
+ if (musb->io.quirks & MUSB_G_NO_SKB_RESERVE)
+ musb->g.quirk_avoids_skb_reserve = 1;
+
/* At least tusb6010 has its own offsets */
if (musb->ops->ep_offset)
musb->io.ep_offset = musb->ops->ep_offset;
*/
struct musb_platform_ops {
+#define MUSB_G_NO_SKB_RESERVE BIT(9)
#define MUSB_DA8XX BIT(8)
#define MUSB_PRESERVE_SESSION BIT(7)
#define MUSB_DMA_UX500 BIT(6)
musb_dma->status = MUSB_DMA_STATUS_FREE;
musb_dma->max_len = SZ_4M;
- dc = dma_request_slave_channel(dev->parent, str);
- if (!dc) {
- dev_err(dev, "Failed to request %s.\n", str);
- ret = -EPROBE_DEFER;
+ dc = dma_request_chan(dev->parent, str);
+ if (IS_ERR(dc)) {
+ ret = PTR_ERR(dc);
+ if (ret != -EPROBE_DEFER)
+ dev_err(dev, "Failed to request %s: %d.\n",
+ str, ret);
goto err;
}
+
cppi41_channel->dc = dc;
}
return 0;
struct dma_channel *channel = hw_ep->rx_channel;
void __iomem *epio = hw_ep->regs;
dma_addr_t *buf;
- u32 length, res;
+ u32 length;
u16 val;
buf = (void *)urb->iso_frame_desc[qh->iso_idx].offset +
val |= MUSB_RXCSR_DMAENAB;
musb_writew(hw_ep->regs, MUSB_RXCSR, val);
- res = dma->channel_program(channel, qh->maxpacket, 0,
+ return dma->channel_program(channel, qh->maxpacket, 0,
(u32)buf, length);
-
- return res;
}
#else
static inline int musb_rx_dma_iso_cppi41(struct dma_controller *dma,
| TUSB_INT_SRC_ID_STATUS_CHNG))
idle_timeout = tusb_otg_ints(musb, int_src, tbase);
- /* TX dma callback must be handled here, RX dma callback is
- * handled in tusb_omap_dma_cb.
+ /*
+ * Just clear the DMA interrupt if it comes as the completion for both
+ * TX and RX is handled by the DMA callback in tusb6010_omap
*/
if ((int_src & TUSB_INT_SRC_TXRX_DMA_DONE)) {
u32 dma_src = musb_readl(tbase, TUSB_DMA_INT_SRC);
- u32 real_dma_src = musb_readl(tbase, TUSB_DMA_INT_MASK);
dev_dbg(musb->controller, "DMA IRQ %08x\n", dma_src);
- real_dma_src = ~real_dma_src & dma_src;
- if (tusb_dma_omap(musb) && real_dma_src) {
- int tx_source = (real_dma_src & 0xffff);
- int i;
-
- for (i = 1; i <= 15; i++) {
- if (tx_source & (1 << i)) {
- dev_dbg(musb->controller, "completing ep%i %s\n", i, "tx");
- musb_dma_completion(musb, i, 1);
- }
- }
- }
musb_writel(tbase, TUSB_DMA_INT_CLEAR, dma_src);
}
}
static const struct musb_platform_ops tusb_ops = {
- .quirks = MUSB_DMA_TUSB_OMAP | MUSB_IN_TUSB,
+ .quirks = MUSB_DMA_TUSB_OMAP | MUSB_IN_TUSB |
+ MUSB_G_NO_SKB_RESERVE,
.init = tusb_musb_init,
.exit = tusb_musb_exit,
#include <linux/platform_device.h>
#include <linux/dma-mapping.h>
#include <linux/slab.h>
-#include <linux/omap-dma.h>
+#include <linux/dmaengine.h>
#include "musb_core.h"
#include "tusb6010.h"
#define MAX_DMAREQ 5 /* REVISIT: Really 6, but req5 not OK */
-#define OMAP24XX_DMA_EXT_DMAREQ0 2
-#define OMAP24XX_DMA_EXT_DMAREQ1 3
-#define OMAP242X_DMA_EXT_DMAREQ2 14
-#define OMAP242X_DMA_EXT_DMAREQ3 15
-#define OMAP242X_DMA_EXT_DMAREQ4 16
-#define OMAP242X_DMA_EXT_DMAREQ5 64
+struct tusb_dma_data {
+ s8 dmareq;
+ struct dma_chan *chan;
+};
struct tusb_omap_dma_ch {
struct musb *musb;
u8 tx;
struct musb_hw_ep *hw_ep;
- int ch;
- s8 dmareq;
- s8 sync_dev;
+ struct tusb_dma_data *dma_data;
struct tusb_omap_dma *tusb_dma;
struct dma_controller controller;
void __iomem *tbase;
- int ch;
- s8 dmareq;
- s8 sync_dev;
+ struct tusb_dma_data dma_pool[MAX_DMAREQ];
unsigned multichannel:1;
};
* See also musb_dma_completion in plat_uds.c and musb_g_[tx|rx]() in
* musb_gadget.c.
*/
-static void tusb_omap_dma_cb(int lch, u16 ch_status, void *data)
+static void tusb_omap_dma_cb(void *data)
{
struct dma_channel *channel = (struct dma_channel *)data;
struct tusb_omap_dma_ch *chdat = to_chdat(channel);
void __iomem *ep_conf = hw_ep->conf;
void __iomem *mbase = musb->mregs;
unsigned long remaining, flags, pio;
- int ch;
spin_lock_irqsave(&musb->lock, flags);
- if (tusb_dma->multichannel)
- ch = chdat->ch;
- else
- ch = tusb_dma->ch;
-
- if (ch_status != OMAP_DMA_BLOCK_IRQ)
- printk(KERN_ERR "TUSB DMA error status: %i\n", ch_status);
-
- dev_dbg(musb->controller, "ep%i %s dma callback ch: %i status: %x\n",
- chdat->epnum, chdat->tx ? "tx" : "rx",
- ch, ch_status);
+ dev_dbg(musb->controller, "ep%i %s dma callback\n",
+ chdat->epnum, chdat->tx ? "tx" : "rx");
if (chdat->tx)
remaining = musb_readl(ep_conf, TUSB_EP_TX_OFFSET);
/* HW issue #10: XFR_SIZE may get corrupt on DMA (both async & sync) */
if (unlikely(remaining > chdat->transfer_len)) {
- dev_dbg(musb->controller, "Corrupt %s dma ch%i XFR_SIZE: 0x%08lx\n",
- chdat->tx ? "tx" : "rx", chdat->ch,
- remaining);
+ dev_dbg(musb->controller, "Corrupt %s XFR_SIZE: 0x%08lx\n",
+ chdat->tx ? "tx" : "rx", remaining);
remaining = 0;
}
channel->status = MUSB_DMA_STATUS_FREE;
- /* Handle only RX callbacks here. TX callbacks must be handled based
- * on the TUSB DMA status interrupt.
- * REVISIT: Use both TUSB DMA status interrupt and OMAP DMA callback
- * interrupt for RX and TX.
- */
- if (!chdat->tx)
- musb_dma_completion(musb, chdat->epnum, chdat->tx);
+ musb_dma_completion(musb, chdat->epnum, chdat->tx);
/* We must terminate short tx transfers manually by setting TXPKTRDY.
* REVISIT: This same problem may occur with other MUSB dma as well.
struct musb_hw_ep *hw_ep = chdat->hw_ep;
void __iomem *mbase = musb->mregs;
void __iomem *ep_conf = hw_ep->conf;
- dma_addr_t fifo = hw_ep->fifo_sync;
- struct omap_dma_channel_params dma_params;
+ dma_addr_t fifo_addr = hw_ep->fifo_sync;
u32 dma_remaining;
- int src_burst, dst_burst;
u16 csr;
u32 psize;
- int ch;
- s8 dmareq;
- s8 sync_dev;
+ struct tusb_dma_data *dma_data;
+ struct dma_async_tx_descriptor *dma_desc;
+ struct dma_slave_config dma_cfg;
+ enum dma_transfer_direction dma_dir;
+ u32 port_window;
+ int ret;
if (unlikely(dma_addr & 0x1) || (len < 32) || (len > packet_sz))
return false;
dma_remaining = TUSB_EP_CONFIG_XFR_SIZE(dma_remaining);
if (dma_remaining) {
- dev_dbg(musb->controller, "Busy %s dma ch%i, not using: %08x\n",
- chdat->tx ? "tx" : "rx", chdat->ch,
- dma_remaining);
+ dev_dbg(musb->controller, "Busy %s dma, not using: %08x\n",
+ chdat->tx ? "tx" : "rx", dma_remaining);
return false;
}
else
chdat->transfer_packet_sz = packet_sz;
- if (tusb_dma->multichannel) {
- ch = chdat->ch;
- dmareq = chdat->dmareq;
- sync_dev = chdat->sync_dev;
- } else {
+ dma_data = chdat->dma_data;
+ if (!tusb_dma->multichannel) {
if (tusb_omap_use_shared_dmareq(chdat) != 0) {
dev_dbg(musb->controller, "could not get dma for ep%i\n", chdat->epnum);
return false;
}
- if (tusb_dma->ch < 0) {
+ if (dma_data->dmareq < 0) {
/* REVISIT: This should get blocked earlier, happens
* with MSC ErrorRecoveryTest
*/
WARN_ON(1);
return false;
}
-
- ch = tusb_dma->ch;
- dmareq = tusb_dma->dmareq;
- sync_dev = tusb_dma->sync_dev;
- omap_set_dma_callback(ch, tusb_omap_dma_cb, channel);
}
chdat->packet_sz = packet_sz;
channel->status = MUSB_DMA_STATUS_BUSY;
/* Since we're recycling dma areas, we need to clean or invalidate */
- if (chdat->tx)
+ if (chdat->tx) {
+ dma_dir = DMA_MEM_TO_DEV;
dma_map_single(dev, phys_to_virt(dma_addr), len,
DMA_TO_DEVICE);
- else
+ } else {
+ dma_dir = DMA_DEV_TO_MEM;
dma_map_single(dev, phys_to_virt(dma_addr), len,
DMA_FROM_DEVICE);
+ }
+
+ memset(&dma_cfg, 0, sizeof(dma_cfg));
/* Use 16-bit transfer if dma_addr is not 32-bit aligned */
if ((dma_addr & 0x3) == 0) {
- dma_params.data_type = OMAP_DMA_DATA_TYPE_S32;
- dma_params.elem_count = 8; /* Elements in frame */
+ dma_cfg.src_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES;
+ dma_cfg.dst_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES;
+ port_window = 8;
} else {
- dma_params.data_type = OMAP_DMA_DATA_TYPE_S16;
- dma_params.elem_count = 16; /* Elements in frame */
- fifo = hw_ep->fifo_async;
- }
+ dma_cfg.src_addr_width = DMA_SLAVE_BUSWIDTH_2_BYTES;
+ dma_cfg.dst_addr_width = DMA_SLAVE_BUSWIDTH_2_BYTES;
+ port_window = 16;
- dma_params.frame_count = chdat->transfer_len / 32; /* Burst sz frame */
+ fifo_addr = hw_ep->fifo_async;
+ }
- dev_dbg(musb->controller, "ep%i %s dma ch%i dma: %pad len: %u(%u) packet_sz: %i(%i)\n",
- chdat->epnum, chdat->tx ? "tx" : "rx",
- ch, &dma_addr, chdat->transfer_len, len,
- chdat->transfer_packet_sz, packet_sz);
+ dev_dbg(musb->controller,
+ "ep%i %s dma: %pad len: %u(%u) packet_sz: %i(%i)\n",
+ chdat->epnum, chdat->tx ? "tx" : "rx", &dma_addr,
+ chdat->transfer_len, len, chdat->transfer_packet_sz, packet_sz);
+
+ dma_cfg.src_addr = fifo_addr;
+ dma_cfg.dst_addr = fifo_addr;
+ dma_cfg.src_port_window_size = port_window;
+ dma_cfg.src_maxburst = port_window;
+ dma_cfg.dst_port_window_size = port_window;
+ dma_cfg.dst_maxburst = port_window;
+
+ ret = dmaengine_slave_config(dma_data->chan, &dma_cfg);
+ if (ret) {
+ dev_err(musb->controller, "DMA slave config failed: %d\n", ret);
+ return false;
+ }
- /*
- * Prepare omap DMA for transfer
- */
- if (chdat->tx) {
- dma_params.src_amode = OMAP_DMA_AMODE_POST_INC;
- dma_params.src_start = (unsigned long)dma_addr;
- dma_params.src_ei = 0;
- dma_params.src_fi = 0;
-
- dma_params.dst_amode = OMAP_DMA_AMODE_DOUBLE_IDX;
- dma_params.dst_start = (unsigned long)fifo;
- dma_params.dst_ei = 1;
- dma_params.dst_fi = -31; /* Loop 32 byte window */
-
- dma_params.trigger = sync_dev;
- dma_params.sync_mode = OMAP_DMA_SYNC_FRAME;
- dma_params.src_or_dst_synch = 0; /* Dest sync */
-
- src_burst = OMAP_DMA_DATA_BURST_16; /* 16x32 read */
- dst_burst = OMAP_DMA_DATA_BURST_8; /* 8x32 write */
- } else {
- dma_params.src_amode = OMAP_DMA_AMODE_DOUBLE_IDX;
- dma_params.src_start = (unsigned long)fifo;
- dma_params.src_ei = 1;
- dma_params.src_fi = -31; /* Loop 32 byte window */
-
- dma_params.dst_amode = OMAP_DMA_AMODE_POST_INC;
- dma_params.dst_start = (unsigned long)dma_addr;
- dma_params.dst_ei = 0;
- dma_params.dst_fi = 0;
-
- dma_params.trigger = sync_dev;
- dma_params.sync_mode = OMAP_DMA_SYNC_FRAME;
- dma_params.src_or_dst_synch = 1; /* Source sync */
-
- src_burst = OMAP_DMA_DATA_BURST_8; /* 8x32 read */
- dst_burst = OMAP_DMA_DATA_BURST_16; /* 16x32 write */
+ dma_desc = dmaengine_prep_slave_single(dma_data->chan, dma_addr,
+ chdat->transfer_len, dma_dir,
+ DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
+ if (!dma_desc) {
+ dev_err(musb->controller, "DMA prep_slave_single failed\n");
+ return false;
}
- dev_dbg(musb->controller, "ep%i %s using %i-bit %s dma from 0x%08lx to 0x%08lx\n",
+ dma_desc->callback = tusb_omap_dma_cb;
+ dma_desc->callback_param = channel;
+ dmaengine_submit(dma_desc);
+
+ dev_dbg(musb->controller,
+ "ep%i %s using %i-bit %s dma from %pad to %pad\n",
chdat->epnum, chdat->tx ? "tx" : "rx",
- (dma_params.data_type == OMAP_DMA_DATA_TYPE_S32) ? 32 : 16,
+ dma_cfg.src_addr_width * 8,
((dma_addr & 0x3) == 0) ? "sync" : "async",
- dma_params.src_start, dma_params.dst_start);
-
- omap_set_dma_params(ch, &dma_params);
- omap_set_dma_src_burst_mode(ch, src_burst);
- omap_set_dma_dest_burst_mode(ch, dst_burst);
- omap_set_dma_write_mode(ch, OMAP_DMA_WRITE_LAST_NON_POSTED);
+ (dma_dir == DMA_MEM_TO_DEV) ? &dma_addr : &fifo_addr,
+ (dma_dir == DMA_MEM_TO_DEV) ? &fifo_addr : &dma_addr);
/*
* Prepare MUSB for DMA transfer
*/
+ musb_ep_select(mbase, chdat->epnum);
if (chdat->tx) {
- musb_ep_select(mbase, chdat->epnum);
csr = musb_readw(hw_ep->regs, MUSB_TXCSR);
csr |= (MUSB_TXCSR_AUTOSET | MUSB_TXCSR_DMAENAB
| MUSB_TXCSR_DMAMODE | MUSB_TXCSR_MODE);
csr &= ~MUSB_TXCSR_P_UNDERRUN;
musb_writew(hw_ep->regs, MUSB_TXCSR, csr);
} else {
- musb_ep_select(mbase, chdat->epnum);
csr = musb_readw(hw_ep->regs, MUSB_RXCSR);
csr |= MUSB_RXCSR_DMAENAB;
csr &= ~(MUSB_RXCSR_AUTOCLEAR | MUSB_RXCSR_DMAMODE);
csr | MUSB_RXCSR_P_WZC_BITS);
}
- /*
- * Start DMA transfer
- */
- omap_start_dma(ch);
+ /* Start DMA transfer */
+ dma_async_issue_pending(dma_data->chan);
if (chdat->tx) {
/* Send transfer_packet_sz packets at a time */
static int tusb_omap_dma_abort(struct dma_channel *channel)
{
struct tusb_omap_dma_ch *chdat = to_chdat(channel);
- struct tusb_omap_dma *tusb_dma = chdat->tusb_dma;
- if (!tusb_dma->multichannel) {
- if (tusb_dma->ch >= 0) {
- omap_stop_dma(tusb_dma->ch);
- omap_free_dma(tusb_dma->ch);
- tusb_dma->ch = -1;
- }
-
- tusb_dma->dmareq = -1;
- tusb_dma->sync_dev = -1;
- }
+ if (chdat->dma_data)
+ dmaengine_terminate_all(chdat->dma_data->chan);
channel->status = MUSB_DMA_STATUS_FREE;
u32 reg = musb_readl(chdat->tbase, TUSB_DMA_EP_MAP);
int i, dmareq_nr = -1;
- const int sync_dev[6] = {
- OMAP24XX_DMA_EXT_DMAREQ0,
- OMAP24XX_DMA_EXT_DMAREQ1,
- OMAP242X_DMA_EXT_DMAREQ2,
- OMAP242X_DMA_EXT_DMAREQ3,
- OMAP242X_DMA_EXT_DMAREQ4,
- OMAP242X_DMA_EXT_DMAREQ5,
- };
-
for (i = 0; i < MAX_DMAREQ; i++) {
int cur = (reg & (0xf << (i * 5))) >> (i * 5);
if (cur == 0) {
reg |= ((1 << 4) << (dmareq_nr * 5));
musb_writel(chdat->tbase, TUSB_DMA_EP_MAP, reg);
- chdat->dmareq = dmareq_nr;
- chdat->sync_dev = sync_dev[chdat->dmareq];
+ chdat->dma_data = &chdat->tusb_dma->dma_pool[dmareq_nr];
return 0;
}
{
u32 reg;
- if (!chdat || chdat->dmareq < 0)
+ if (!chdat || !chdat->dma_data || chdat->dma_data->dmareq < 0)
return;
reg = musb_readl(chdat->tbase, TUSB_DMA_EP_MAP);
- reg &= ~(0x1f << (chdat->dmareq * 5));
+ reg &= ~(0x1f << (chdat->dma_data->dmareq * 5));
musb_writel(chdat->tbase, TUSB_DMA_EP_MAP, reg);
- chdat->dmareq = -1;
- chdat->sync_dev = -1;
+ chdat->dma_data = NULL;
}
static struct dma_channel *dma_channel_pool[MAX_DMAREQ];
u8 tx)
{
int ret, i;
- const char *dev_name;
struct tusb_omap_dma *tusb_dma;
struct musb *musb;
- void __iomem *tbase;
struct dma_channel *channel = NULL;
struct tusb_omap_dma_ch *chdat = NULL;
- u32 reg;
+ struct tusb_dma_data *dma_data = NULL;
tusb_dma = container_of(c, struct tusb_omap_dma, controller);
musb = tusb_dma->controller.musb;
- tbase = musb->ctrl_base;
-
- reg = musb_readl(tbase, TUSB_DMA_INT_MASK);
- if (tx)
- reg &= ~(1 << hw_ep->epnum);
- else
- reg &= ~(1 << (hw_ep->epnum + 15));
- musb_writel(tbase, TUSB_DMA_INT_MASK, reg);
/* REVISIT: Why does dmareq5 not work? */
if (hw_ep->epnum == 0) {
if (!channel)
return NULL;
- if (tx) {
- chdat->tx = 1;
- dev_name = "TUSB transmit";
- } else {
- chdat->tx = 0;
- dev_name = "TUSB receive";
- }
-
chdat->musb = tusb_dma->controller.musb;
chdat->tbase = tusb_dma->tbase;
chdat->hw_ep = hw_ep;
chdat->epnum = hw_ep->epnum;
- chdat->dmareq = -1;
chdat->completed_len = 0;
chdat->tusb_dma = tusb_dma;
+ if (tx)
+ chdat->tx = 1;
+ else
+ chdat->tx = 0;
channel->max_len = 0x7fffffff;
channel->desired_mode = 0;
channel->actual_len = 0;
- if (tusb_dma->multichannel) {
- ret = tusb_omap_dma_allocate_dmareq(chdat);
- if (ret != 0)
- goto free_dmareq;
-
- ret = omap_request_dma(chdat->sync_dev, dev_name,
- tusb_omap_dma_cb, channel, &chdat->ch);
- if (ret != 0)
- goto free_dmareq;
- } else if (tusb_dma->ch == -1) {
- tusb_dma->dmareq = 0;
- tusb_dma->sync_dev = OMAP24XX_DMA_EXT_DMAREQ0;
-
- /* Callback data gets set later in the shared dmareq case */
- ret = omap_request_dma(tusb_dma->sync_dev, "TUSB shared",
- tusb_omap_dma_cb, NULL, &tusb_dma->ch);
- if (ret != 0)
- goto free_dmareq;
-
- chdat->dmareq = -1;
- chdat->ch = -1;
+ if (!chdat->dma_data) {
+ if (tusb_dma->multichannel) {
+ ret = tusb_omap_dma_allocate_dmareq(chdat);
+ if (ret != 0)
+ goto free_dmareq;
+ } else {
+ chdat->dma_data = &tusb_dma->dma_pool[0];
+ }
}
- dev_dbg(musb->controller, "ep%i %s dma: %s dma%i dmareq%i sync%i\n",
+ dma_data = chdat->dma_data;
+
+ dev_dbg(musb->controller, "ep%i %s dma: %s dmareq%i\n",
chdat->epnum,
chdat->tx ? "tx" : "rx",
- chdat->ch >= 0 ? "dedicated" : "shared",
- chdat->ch >= 0 ? chdat->ch : tusb_dma->ch,
- chdat->dmareq >= 0 ? chdat->dmareq : tusb_dma->dmareq,
- chdat->sync_dev >= 0 ? chdat->sync_dev : tusb_dma->sync_dev);
+ tusb_dma->multichannel ? "shared" : "dedicated",
+ dma_data->dmareq);
return channel;
{
struct tusb_omap_dma_ch *chdat = to_chdat(channel);
struct musb *musb = chdat->musb;
- void __iomem *tbase = musb->ctrl_base;
- u32 reg;
-
- dev_dbg(musb->controller, "ep%i ch%i\n", chdat->epnum, chdat->ch);
-
- reg = musb_readl(tbase, TUSB_DMA_INT_MASK);
- if (chdat->tx)
- reg |= (1 << chdat->epnum);
- else
- reg |= (1 << (chdat->epnum + 15));
- musb_writel(tbase, TUSB_DMA_INT_MASK, reg);
- reg = musb_readl(tbase, TUSB_DMA_INT_CLEAR);
- if (chdat->tx)
- reg |= (1 << chdat->epnum);
- else
- reg |= (1 << (chdat->epnum + 15));
- musb_writel(tbase, TUSB_DMA_INT_CLEAR, reg);
+ dev_dbg(musb->controller, "Release for ep%i\n", chdat->epnum);
channel->status = MUSB_DMA_STATUS_UNKNOWN;
- if (chdat->ch >= 0) {
- omap_stop_dma(chdat->ch);
- omap_free_dma(chdat->ch);
- chdat->ch = -1;
- }
-
- if (chdat->dmareq >= 0)
- tusb_omap_dma_free_dmareq(chdat);
+ dmaengine_terminate_sync(chdat->dma_data->chan);
+ tusb_omap_dma_free_dmareq(chdat);
channel = NULL;
}
kfree(ch->private_data);
kfree(ch);
}
- }
- if (tusb_dma && !tusb_dma->multichannel && tusb_dma->ch >= 0)
- omap_free_dma(tusb_dma->ch);
+ /* Free up the DMA channels */
+ if (tusb_dma && tusb_dma->dma_pool[i].chan)
+ dma_release_channel(tusb_dma->dma_pool[i].chan);
+ }
kfree(tusb_dma);
}
EXPORT_SYMBOL_GPL(tusb_dma_controller_destroy);
+static int tusb_omap_allocate_dma_pool(struct tusb_omap_dma *tusb_dma)
+{
+ struct musb *musb = tusb_dma->controller.musb;
+ int i;
+ int ret = 0;
+
+ for (i = 0; i < MAX_DMAREQ; i++) {
+ struct tusb_dma_data *dma_data = &tusb_dma->dma_pool[i];
+
+ /*
+ * Request DMA channels:
+ * - one channel in case of non multichannel mode
+ * - MAX_DMAREQ number of channels in multichannel mode
+ */
+ if (i == 0 || tusb_dma->multichannel) {
+ char ch_name[8];
+
+ sprintf(ch_name, "dmareq%d", i);
+ dma_data->chan = dma_request_chan(musb->controller,
+ ch_name);
+ if (IS_ERR(dma_data->chan)) {
+ dev_err(musb->controller,
+ "Failed to request %s\n", ch_name);
+ ret = PTR_ERR(dma_data->chan);
+ goto dma_error;
+ }
+
+ dma_data->dmareq = i;
+ } else {
+ dma_data->dmareq = -1;
+ }
+ }
+
+ return 0;
+
+dma_error:
+ for (; i >= 0; i--) {
+ struct tusb_dma_data *dma_data = &tusb_dma->dma_pool[i];
+
+ if (dma_data->dmareq >= 0)
+ dma_release_channel(dma_data->chan);
+ }
+
+ return ret;
+}
+
struct dma_controller *
tusb_dma_controller_create(struct musb *musb, void __iomem *base)
{
tusb_dma->controller.musb = musb;
tusb_dma->tbase = musb->ctrl_base;
- tusb_dma->ch = -1;
- tusb_dma->dmareq = -1;
- tusb_dma->sync_dev = -1;
-
tusb_dma->controller.channel_alloc = tusb_omap_dma_allocate;
tusb_dma->controller.channel_release = tusb_omap_dma_release;
tusb_dma->controller.channel_program = tusb_omap_dma_program;
ch->private_data = chdat;
}
+ if (tusb_omap_allocate_dma_pool(tusb_dma))
+ goto cleanup;
+
return &tusb_dma->controller;
cleanup:
menu "USB Physical Layer drivers"
config USB_PHY
+ select EXTCON
def_bool n
#
config TAHVO_USB
tristate "Tahvo USB transceiver driver"
- depends on MFD_RETU && EXTCON
+ depends on MFD_RETU
depends on USB_GADGET || !USB_GADGET # if USB_GADGET=m, this can't be 'y'
select USB_PHY
help
depends on (USB || USB_GADGET) && (ARCH_QCOM || COMPILE_TEST)
depends on USB_GADGET || !USB_GADGET # if USB_GADGET=m, this can't be 'y'
depends on RESET_CONTROLLER
- depends on EXTCON
select USB_PHY
help
Enable this to support the USB OTG transceiver on Qualcomm chips. It
config USB_QCOM_8X16_PHY
tristate "Qualcomm APQ8016/MSM8916 on-chip USB PHY controller support"
depends on ARCH_QCOM || COMPILE_TEST
- depends on RESET_CONTROLLER && EXTCON
+ depends on RESET_CONTROLLER
select USB_PHY
select USB_ULPI_VIEWPORT
help
void (*setup_gpio)(enum usb_otg_state state);
};
-/**
- * struct msm_usb_cable - structure for exteternal connector cable
- * state tracking
- * @nb: hold event notification callback
- * @conn: used for notification registration
- */
-struct msm_usb_cable {
- struct notifier_block nb;
- struct extcon_dev *extcon;
-};
-
/**
* struct msm_otg: OTG driver data. Shared by HCD and DCD.
* @otg: USB OTG Transceiver structure.
bool manual_pullup;
- struct msm_usb_cable vbus;
- struct msm_usb_cable id;
-
struct gpio_desc *switch_gpio;
struct notifier_block reboot;
};
static int msm_otg_vbus_notifier(struct notifier_block *nb, unsigned long event,
void *ptr)
{
- struct msm_usb_cable *vbus = container_of(nb, struct msm_usb_cable, nb);
- struct msm_otg *motg = container_of(vbus, struct msm_otg, vbus);
+ struct usb_phy *usb_phy = container_of(nb, struct usb_phy, vbus_nb);
+ struct msm_otg *motg = container_of(usb_phy, struct msm_otg, phy);
if (event)
set_bit(B_SESS_VLD, &motg->inputs);
static int msm_otg_id_notifier(struct notifier_block *nb, unsigned long event,
void *ptr)
{
- struct msm_usb_cable *id = container_of(nb, struct msm_usb_cable, nb);
- struct msm_otg *motg = container_of(id, struct msm_otg, id);
+ struct usb_phy *usb_phy = container_of(nb, struct usb_phy, id_nb);
+ struct msm_otg *motg = container_of(usb_phy, struct msm_otg, phy);
if (event)
clear_bit(ID, &motg->inputs);
static int msm_otg_read_dt(struct platform_device *pdev, struct msm_otg *motg)
{
struct msm_otg_platform_data *pdata;
- struct extcon_dev *ext_id, *ext_vbus;
struct device_node *node = pdev->dev.of_node;
struct property *prop;
int len, ret, words;
if (IS_ERR(motg->switch_gpio))
return PTR_ERR(motg->switch_gpio);
- ext_id = ERR_PTR(-ENODEV);
- ext_vbus = ERR_PTR(-ENODEV);
- if (of_property_read_bool(node, "extcon")) {
-
- /* Each one of them is not mandatory */
- ext_vbus = extcon_get_edev_by_phandle(&pdev->dev, 0);
- if (IS_ERR(ext_vbus) && PTR_ERR(ext_vbus) != -ENODEV)
- return PTR_ERR(ext_vbus);
-
- ext_id = extcon_get_edev_by_phandle(&pdev->dev, 1);
- if (IS_ERR(ext_id) && PTR_ERR(ext_id) != -ENODEV)
- return PTR_ERR(ext_id);
- }
-
- if (!IS_ERR(ext_vbus)) {
- motg->vbus.extcon = ext_vbus;
- motg->vbus.nb.notifier_call = msm_otg_vbus_notifier;
- ret = devm_extcon_register_notifier(&pdev->dev, ext_vbus,
- EXTCON_USB, &motg->vbus.nb);
- if (ret < 0) {
- dev_err(&pdev->dev, "register VBUS notifier failed\n");
- return ret;
- }
-
- ret = extcon_get_state(ext_vbus, EXTCON_USB);
- if (ret)
- set_bit(B_SESS_VLD, &motg->inputs);
- else
- clear_bit(B_SESS_VLD, &motg->inputs);
- }
-
- if (!IS_ERR(ext_id)) {
- motg->id.extcon = ext_id;
- motg->id.nb.notifier_call = msm_otg_id_notifier;
- ret = devm_extcon_register_notifier(&pdev->dev, ext_id,
- EXTCON_USB_HOST, &motg->id.nb);
- if (ret < 0) {
- dev_err(&pdev->dev, "register ID notifier failed\n");
- return ret;
- }
-
- ret = extcon_get_state(ext_id, EXTCON_USB_HOST);
- if (ret)
- clear_bit(ID, &motg->inputs);
- else
- set_bit(ID, &motg->inputs);
- }
-
prop = of_find_property(node, "qcom,phy-init-sequence", &len);
if (!prop || !len)
return 0;
phy->init = msm_phy_init;
phy->notify_disconnect = msm_phy_notify_disconnect;
phy->type = USB_PHY_TYPE_USB2;
+ phy->vbus_nb.notifier_call = msm_otg_vbus_notifier;
+ phy->id_nb.notifier_call = msm_otg_id_notifier;
phy->io_ops = &msm_otg_io_ops;
goto disable_ldo;
}
+ ret = extcon_get_state(phy->edev, EXTCON_USB);
+ if (ret)
+ set_bit(B_SESS_VLD, &motg->inputs);
+ else
+ clear_bit(B_SESS_VLD, &motg->inputs);
+
+ ret = extcon_get_state(phy->id_edev, EXTCON_USB_HOST);
+ if (ret)
+ clear_bit(ID, &motg->inputs);
+ else
+ set_bit(ID, &motg->inputs);
+
platform_set_drvdata(pdev, motg);
device_init_wakeup(&pdev->dev, 1);
struct reset_control *phy_reset;
- struct extcon_dev *vbus_edev;
- struct notifier_block vbus_notify;
-
struct gpio_desc *switch_gpio;
struct notifier_block reboot_notify;
};
static int phy_8x16_vbus_notify(struct notifier_block *nb, unsigned long event,
void *ptr)
{
- struct phy_8x16 *qphy = container_of(nb, struct phy_8x16, vbus_notify);
+ struct usb_phy *usb_phy = container_of(nb, struct usb_phy, vbus_nb);
+ struct phy_8x16 *qphy = container_of(usb_phy, struct phy_8x16, phy);
if (event)
phy_8x16_vbus_on(qphy);
val = ULPI_PWR_OTG_COMP_DISABLE;
usb_phy_io_write(phy, val, ULPI_SET(ULPI_PWR_CLK_MNG_REG));
- state = extcon_get_state(qphy->vbus_edev, EXTCON_USB);
+ state = extcon_get_state(qphy->phy.edev, EXTCON_USB);
if (state)
phy_8x16_vbus_on(qphy);
else
phy->io_priv = qphy->regs + HSPHY_ULPI_VIEWPORT;
phy->io_ops = &ulpi_viewport_access_ops;
phy->type = USB_PHY_TYPE_USB2;
+ phy->vbus_nb.notifier_call = phy_8x16_vbus_notify;
+ phy->id_nb.notifier_call = NULL;
ret = phy_8x16_read_devicetree(qphy);
if (ret < 0)
return ret;
- qphy->vbus_edev = extcon_get_edev_by_phandle(phy->dev, 0);
- if (IS_ERR(qphy->vbus_edev))
- return PTR_ERR(qphy->vbus_edev);
-
ret = clk_set_rate(qphy->core_clk, INT_MAX);
if (ret < 0)
dev_dbg(phy->dev, "Can't boost core clock\n");
if (WARN_ON(ret))
goto off_clks;
- qphy->vbus_notify.notifier_call = phy_8x16_vbus_notify;
- ret = devm_extcon_register_notifier(&pdev->dev, qphy->vbus_edev,
- EXTCON_USB, &qphy->vbus_notify);
- if (ret < 0)
- goto off_power;
-
ret = usb_add_phy_dev(&qphy->phy);
if (ret)
goto off_power;
return *phy == match_data;
}
+static int usb_add_extcon(struct usb_phy *x)
+{
+ int ret;
+
+ if (of_property_read_bool(x->dev->of_node, "extcon")) {
+ x->edev = extcon_get_edev_by_phandle(x->dev, 0);
+ if (IS_ERR(x->edev))
+ return PTR_ERR(x->edev);
+
+ x->id_edev = extcon_get_edev_by_phandle(x->dev, 1);
+ if (IS_ERR(x->id_edev)) {
+ x->id_edev = NULL;
+ dev_info(x->dev, "No separate ID extcon device\n");
+ }
+
+ if (x->vbus_nb.notifier_call) {
+ ret = devm_extcon_register_notifier(x->dev, x->edev,
+ EXTCON_USB,
+ &x->vbus_nb);
+ if (ret < 0) {
+ dev_err(x->dev,
+ "register VBUS notifier failed\n");
+ return ret;
+ }
+ }
+
+ if (x->id_nb.notifier_call) {
+ struct extcon_dev *id_ext;
+
+ if (x->id_edev)
+ id_ext = x->id_edev;
+ else
+ id_ext = x->edev;
+
+ ret = devm_extcon_register_notifier(x->dev, id_ext,
+ EXTCON_USB_HOST,
+ &x->id_nb);
+ if (ret < 0) {
+ dev_err(x->dev,
+ "register ID notifier failed\n");
+ return ret;
+ }
+ }
+ }
+
+ return 0;
+}
+
/**
* devm_usb_get_phy - find the USB PHY
* @dev - device that requests this phy
return -EINVAL;
}
+ ret = usb_add_extcon(x);
+ if (ret)
+ return ret;
+
ATOMIC_INIT_NOTIFIER_HEAD(&x->notifier);
spin_lock_irqsave(&phy_lock, flags);
{
struct usb_phy_bind *phy_bind;
unsigned long flags;
+ int ret;
if (!x->dev) {
dev_err(x->dev, "no device provided for PHY\n");
return -EINVAL;
}
+ ret = usb_add_extcon(x);
+ if (ret)
+ return ret;
+
ATOMIC_INIT_NOTIFIER_HEAD(&x->notifier);
spin_lock_irqsave(&phy_lock, flags);
{ USB_DEVICE(0x10C4, 0x8977) }, /* CEL MeshWorks DevKit Device */
{ USB_DEVICE(0x10C4, 0x8998) }, /* KCF Technologies PRN */
{ USB_DEVICE(0x10C4, 0x8A2A) }, /* HubZ dual ZigBee and Z-Wave dongle */
+ { USB_DEVICE(0x10C4, 0x8A5E) }, /* CEL EM3588 ZigBee USB Stick Long Range */
{ USB_DEVICE(0x10C4, 0xEA60) }, /* Silicon Labs factory default */
{ USB_DEVICE(0x10C4, 0xEA61) }, /* Silicon Labs factory default */
{ USB_DEVICE(0x10C4, 0xEA70) }, /* Silicon Labs factory default */
int div_okay = 1;
int baud;
- /*
- * The logic involved in setting the baudrate can be cleanly split into
- * 3 steps.
- * 1. Standard baud rates are set in tty->termios->c_cflag
- * 2. If these are not enough, you can set any speed using alt_speed as
- * follows:
- * - set tty->termios->c_cflag speed to B38400
- * - set your real speed in tty->alt_speed; it gets ignored when
- * alt_speed==0, (or)
- * - call TIOCSSERIAL ioctl with (struct serial_struct) set as
- * follows:
- * flags & ASYNC_SPD_MASK == ASYNC_SPD_[HI, VHI, SHI, WARP],
- * this just sets alt_speed to (HI: 57600, VHI: 115200,
- * SHI: 230400, WARP: 460800)
- * ** Steps 1, 2 are done courtesy of tty_get_baud_rate
- * 3. You can also set baud rate by setting custom divisor as follows
- * - set tty->termios->c_cflag speed to B38400
- * - call TIOCSSERIAL ioctl with (struct serial_struct) set as
- * follows:
- * o flags & ASYNC_SPD_MASK == ASYNC_SPD_CUST
- * o custom_divisor set to baud_base / your_new_baudrate
- * ** Step 3 is done courtesy of code borrowed from serial.c
- * I should really spend some time and separate + move this common
- * code to serial.c, it is replicated in nearly every serial driver
- * you see.
- */
-
- /* 1. Get the baud rate from the tty settings, this observes
- alt_speed hack */
-
baud = tty_get_baud_rate(tty);
dev_dbg(dev, "%s - tty_get_baud_rate reports speed %d\n", __func__, baud);
- /* 2. Observe async-compatible custom_divisor hack, update baudrate
- if needed */
-
+ /*
+ * Observe deprecated async-compatible custom_divisor hack, update
+ * baudrate if needed.
+ */
if (baud == 38400 &&
((priv->flags & ASYNC_SPD_MASK) == ASYNC_SPD_CUST) &&
(priv->custom_divisor)) {
__func__, priv->custom_divisor, baud);
}
- /* 3. Convert baudrate to device-specific divisor */
-
if (!baud)
baud = 9600;
switch (priv->chip_type) {
/* Do error checking and permission checking */
if (!capable(CAP_SYS_ADMIN)) {
- if (((new_serial.flags & ~ASYNC_USR_MASK) !=
- (priv->flags & ~ASYNC_USR_MASK))) {
+ if ((new_serial.flags ^ priv->flags) & ~ASYNC_USR_MASK) {
mutex_unlock(&priv->cfg_lock);
return -EPERM;
}
check_and_exit:
write_latency_timer(port);
- if ((old_priv.flags & ASYNC_SPD_MASK) !=
- (priv->flags & ASYNC_SPD_MASK)) {
- if ((priv->flags & ASYNC_SPD_MASK) == ASYNC_SPD_HI)
- tty->alt_speed = 57600;
- else if ((priv->flags & ASYNC_SPD_MASK) == ASYNC_SPD_VHI)
- tty->alt_speed = 115200;
- else if ((priv->flags & ASYNC_SPD_MASK) == ASYNC_SPD_SHI)
- tty->alt_speed = 230400;
- else if ((priv->flags & ASYNC_SPD_MASK) == ASYNC_SPD_WARP)
- tty->alt_speed = 460800;
- else
- tty->alt_speed = 0;
- }
- if (((old_priv.flags & ASYNC_SPD_MASK) !=
- (priv->flags & ASYNC_SPD_MASK)) ||
- (((priv->flags & ASYNC_SPD_MASK) == ASYNC_SPD_CUST) &&
- (old_priv.custom_divisor != priv->custom_divisor))) {
+ if ((priv->flags ^ old_priv.flags) & ASYNC_SPD_MASK ||
+ ((priv->flags & ASYNC_SPD_MASK) == ASYNC_SPD_CUST &&
+ priv->custom_divisor != old_priv.custom_divisor)) {
+
+ /* warn about deprecation unless clearing */
+ if (priv->flags & ASYNC_SPD_MASK)
+ dev_warn_ratelimited(&port->dev, "use of SPD flags is deprecated\n");
+
change_speed(tty, port);
mutex_unlock(&priv->cfg_lock);
}
.write = usb_wwan_write,
.write_room = usb_wwan_write_room,
.chars_in_buffer = usb_wwan_chars_in_buffer,
+ .tiocmget = usb_wwan_tiocmget,
+ .tiocmset = usb_wwan_tiocmset,
.attach = qc_attach,
.release = qc_release,
.port_probe = usb_wwan_port_probe,
static speed_t upd78f0730_get_baud_rate(struct tty_struct *tty)
{
const speed_t baud_rate = tty_get_baud_rate(tty);
- const speed_t supported[] = {
+ static const speed_t supported[] = {
0, 2400, 4800, 9600, 19200, 38400, 57600, 115200, 153600
};
int i;
static int upd78f0730_open(struct tty_struct *tty, struct usb_serial_port *port)
{
- struct upd78f0730_open_close request = {
+ static const struct upd78f0730_open_close request = {
.opcode = UPD78F0730_CMD_OPEN_CLOSE,
.state = UPD78F0730_PORT_OPEN
};
static void upd78f0730_close(struct usb_serial_port *port)
{
- struct upd78f0730_open_close request = {
+ static const struct upd78f0730_open_close request = {
.opcode = UPD78F0730_CMD_OPEN_CLOSE,
.state = UPD78F0730_PORT_CLOSE
};
}
}
+static int setup_port_bulk_in(struct usb_serial_port *port,
+ struct usb_endpoint_descriptor *epd)
+{
+ struct usb_serial_driver *type = port->serial->type;
+ struct usb_device *udev = port->serial->dev;
+ int buffer_size;
+ int i;
+
+ buffer_size = max_t(int, type->bulk_in_size, usb_endpoint_maxp(epd));
+ port->bulk_in_size = buffer_size;
+ port->bulk_in_endpointAddress = epd->bEndpointAddress;
+
+ for (i = 0; i < ARRAY_SIZE(port->read_urbs); ++i) {
+ set_bit(i, &port->read_urbs_free);
+ port->read_urbs[i] = usb_alloc_urb(0, GFP_KERNEL);
+ if (!port->read_urbs[i])
+ return -ENOMEM;
+ port->bulk_in_buffers[i] = kmalloc(buffer_size, GFP_KERNEL);
+ if (!port->bulk_in_buffers[i])
+ return -ENOMEM;
+ usb_fill_bulk_urb(port->read_urbs[i], udev,
+ usb_rcvbulkpipe(udev, epd->bEndpointAddress),
+ port->bulk_in_buffers[i], buffer_size,
+ type->read_bulk_callback, port);
+ }
+
+ port->read_urb = port->read_urbs[0];
+ port->bulk_in_buffer = port->bulk_in_buffers[0];
+
+ return 0;
+}
+
+static int setup_port_bulk_out(struct usb_serial_port *port,
+ struct usb_endpoint_descriptor *epd)
+{
+ struct usb_serial_driver *type = port->serial->type;
+ struct usb_device *udev = port->serial->dev;
+ int buffer_size;
+ int i;
+
+ if (kfifo_alloc(&port->write_fifo, PAGE_SIZE, GFP_KERNEL))
+ return -ENOMEM;
+ if (type->bulk_out_size)
+ buffer_size = type->bulk_out_size;
+ else
+ buffer_size = usb_endpoint_maxp(epd);
+ port->bulk_out_size = buffer_size;
+ port->bulk_out_endpointAddress = epd->bEndpointAddress;
+
+ for (i = 0; i < ARRAY_SIZE(port->write_urbs); ++i) {
+ set_bit(i, &port->write_urbs_free);
+ port->write_urbs[i] = usb_alloc_urb(0, GFP_KERNEL);
+ if (!port->write_urbs[i])
+ return -ENOMEM;
+ port->bulk_out_buffers[i] = kmalloc(buffer_size, GFP_KERNEL);
+ if (!port->bulk_out_buffers[i])
+ return -ENOMEM;
+ usb_fill_bulk_urb(port->write_urbs[i], udev,
+ usb_sndbulkpipe(udev, epd->bEndpointAddress),
+ port->bulk_out_buffers[i], buffer_size,
+ type->write_bulk_callback, port);
+ }
+
+ port->write_urb = port->write_urbs[0];
+ port->bulk_out_buffer = port->bulk_out_buffers[0];
+
+ return 0;
+}
+
+static int setup_port_interrupt_in(struct usb_serial_port *port,
+ struct usb_endpoint_descriptor *epd)
+{
+ struct usb_serial_driver *type = port->serial->type;
+ struct usb_device *udev = port->serial->dev;
+ int buffer_size;
+
+ port->interrupt_in_urb = usb_alloc_urb(0, GFP_KERNEL);
+ if (!port->interrupt_in_urb)
+ return -ENOMEM;
+ buffer_size = usb_endpoint_maxp(epd);
+ port->interrupt_in_endpointAddress = epd->bEndpointAddress;
+ port->interrupt_in_buffer = kmalloc(buffer_size, GFP_KERNEL);
+ if (!port->interrupt_in_buffer)
+ return -ENOMEM;
+ usb_fill_int_urb(port->interrupt_in_urb, udev,
+ usb_rcvintpipe(udev, epd->bEndpointAddress),
+ port->interrupt_in_buffer, buffer_size,
+ type->read_int_callback, port,
+ epd->bInterval);
+
+ return 0;
+}
+
+static int setup_port_interrupt_out(struct usb_serial_port *port,
+ struct usb_endpoint_descriptor *epd)
+{
+ struct usb_serial_driver *type = port->serial->type;
+ struct usb_device *udev = port->serial->dev;
+ int buffer_size;
+
+ port->interrupt_out_urb = usb_alloc_urb(0, GFP_KERNEL);
+ if (!port->interrupt_out_urb)
+ return -ENOMEM;
+ buffer_size = usb_endpoint_maxp(epd);
+ port->interrupt_out_size = buffer_size;
+ port->interrupt_out_endpointAddress = epd->bEndpointAddress;
+ port->interrupt_out_buffer = kmalloc(buffer_size, GFP_KERNEL);
+ if (!port->interrupt_out_buffer)
+ return -ENOMEM;
+ usb_fill_int_urb(port->interrupt_out_urb, udev,
+ usb_sndintpipe(udev, epd->bEndpointAddress),
+ port->interrupt_out_buffer, buffer_size,
+ type->write_int_callback, port,
+ epd->bInterval);
+
+ return 0;
+}
+
static int usb_serial_probe(struct usb_interface *interface,
const struct usb_device_id *id)
{
struct usb_device *dev = interface_to_usbdev(interface);
struct usb_serial *serial = NULL;
struct usb_serial_port *port;
- struct usb_endpoint_descriptor *endpoint;
struct usb_serial_endpoints *epds;
struct usb_serial_driver *type = NULL;
int retval;
- int buffer_size;
int i;
- int j;
int num_ports = 0;
unsigned char max_endpoints;
dev_dbg(ddev, "setting up %d port structure(s)\n", max_endpoints);
for (i = 0; i < max_endpoints; ++i) {
port = kzalloc(sizeof(struct usb_serial_port), GFP_KERNEL);
- if (!port)
- goto probe_error;
+ if (!port) {
+ retval = -ENOMEM;
+ goto err_free_epds;
+ }
tty_port_init(&port->port);
port->port.ops = &serial_port_ops;
port->serial = serial;
/* set up the endpoint information */
for (i = 0; i < epds->num_bulk_in; ++i) {
- endpoint = epds->bulk_in[i];
- port = serial->port[i];
- buffer_size = max_t(int, serial->type->bulk_in_size,
- usb_endpoint_maxp(endpoint));
- port->bulk_in_size = buffer_size;
- port->bulk_in_endpointAddress = endpoint->bEndpointAddress;
-
- for (j = 0; j < ARRAY_SIZE(port->read_urbs); ++j) {
- set_bit(j, &port->read_urbs_free);
- port->read_urbs[j] = usb_alloc_urb(0, GFP_KERNEL);
- if (!port->read_urbs[j])
- goto probe_error;
- port->bulk_in_buffers[j] = kmalloc(buffer_size,
- GFP_KERNEL);
- if (!port->bulk_in_buffers[j])
- goto probe_error;
- usb_fill_bulk_urb(port->read_urbs[j], dev,
- usb_rcvbulkpipe(dev,
- endpoint->bEndpointAddress),
- port->bulk_in_buffers[j], buffer_size,
- serial->type->read_bulk_callback,
- port);
- }
-
- port->read_urb = port->read_urbs[0];
- port->bulk_in_buffer = port->bulk_in_buffers[0];
+ retval = setup_port_bulk_in(serial->port[i], epds->bulk_in[i]);
+ if (retval)
+ goto err_free_epds;
}
for (i = 0; i < epds->num_bulk_out; ++i) {
- endpoint = epds->bulk_out[i];
- port = serial->port[i];
- if (kfifo_alloc(&port->write_fifo, PAGE_SIZE, GFP_KERNEL))
- goto probe_error;
- buffer_size = serial->type->bulk_out_size;
- if (!buffer_size)
- buffer_size = usb_endpoint_maxp(endpoint);
- port->bulk_out_size = buffer_size;
- port->bulk_out_endpointAddress = endpoint->bEndpointAddress;
-
- for (j = 0; j < ARRAY_SIZE(port->write_urbs); ++j) {
- set_bit(j, &port->write_urbs_free);
- port->write_urbs[j] = usb_alloc_urb(0, GFP_KERNEL);
- if (!port->write_urbs[j])
- goto probe_error;
- port->bulk_out_buffers[j] = kmalloc(buffer_size,
- GFP_KERNEL);
- if (!port->bulk_out_buffers[j])
- goto probe_error;
- usb_fill_bulk_urb(port->write_urbs[j], dev,
- usb_sndbulkpipe(dev,
- endpoint->bEndpointAddress),
- port->bulk_out_buffers[j], buffer_size,
- serial->type->write_bulk_callback,
- port);
- }
-
- port->write_urb = port->write_urbs[0];
- port->bulk_out_buffer = port->bulk_out_buffers[0];
+ retval = setup_port_bulk_out(serial->port[i],
+ epds->bulk_out[i]);
+ if (retval)
+ goto err_free_epds;
}
if (serial->type->read_int_callback) {
for (i = 0; i < epds->num_interrupt_in; ++i) {
- endpoint = epds->interrupt_in[i];
- port = serial->port[i];
- port->interrupt_in_urb = usb_alloc_urb(0, GFP_KERNEL);
- if (!port->interrupt_in_urb)
- goto probe_error;
- buffer_size = usb_endpoint_maxp(endpoint);
- port->interrupt_in_endpointAddress =
- endpoint->bEndpointAddress;
- port->interrupt_in_buffer = kmalloc(buffer_size,
- GFP_KERNEL);
- if (!port->interrupt_in_buffer)
- goto probe_error;
- usb_fill_int_urb(port->interrupt_in_urb, dev,
- usb_rcvintpipe(dev,
- endpoint->bEndpointAddress),
- port->interrupt_in_buffer, buffer_size,
- serial->type->read_int_callback, port,
- endpoint->bInterval);
+ retval = setup_port_interrupt_in(serial->port[i],
+ epds->interrupt_in[i]);
+ if (retval)
+ goto err_free_epds;
}
} else if (epds->num_interrupt_in) {
dev_dbg(ddev, "The device claims to support interrupt in transfers, but read_int_callback is not defined\n");
if (serial->type->write_int_callback) {
for (i = 0; i < epds->num_interrupt_out; ++i) {
- endpoint = epds->interrupt_out[i];
- port = serial->port[i];
- port->interrupt_out_urb = usb_alloc_urb(0, GFP_KERNEL);
- if (!port->interrupt_out_urb)
- goto probe_error;
- buffer_size = usb_endpoint_maxp(endpoint);
- port->interrupt_out_size = buffer_size;
- port->interrupt_out_endpointAddress =
- endpoint->bEndpointAddress;
- port->interrupt_out_buffer = kmalloc(buffer_size,
- GFP_KERNEL);
- if (!port->interrupt_out_buffer)
- goto probe_error;
- usb_fill_int_urb(port->interrupt_out_urb, dev,
- usb_sndintpipe(dev,
- endpoint->bEndpointAddress),
- port->interrupt_out_buffer, buffer_size,
- serial->type->write_int_callback, port,
- endpoint->bInterval);
+ retval = setup_port_interrupt_out(serial->port[i],
+ epds->interrupt_out[i]);
+ if (retval)
+ goto err_free_epds;
}
} else if (epds->num_interrupt_out) {
dev_dbg(ddev, "The device claims to support interrupt out transfers, but write_int_callback is not defined\n");
if (type->attach) {
retval = type->attach(serial);
if (retval < 0)
- goto probe_error;
+ goto err_free_epds;
serial->attached = 1;
if (retval > 0) {
/* quietly accept this device, but don't bind to a
serial->attached = 1;
}
- if (allocate_minors(serial, num_ports)) {
+ retval = allocate_minors(serial, num_ports);
+ if (retval) {
dev_err(ddev, "No more free serial minor numbers\n");
- goto probe_error;
+ goto err_free_epds;
}
/* register all of the individual ports with the driver core */
module_put(type->driver.owner);
return 0;
-probe_error:
- retval = -EIO;
err_free_epds:
kfree(epds);
err_put_serial:
#define REG_HW_TRAP1 0xFF89
/* SRB Status */
-#define SS_SUCCESS 0x00 /* No Sense */
-#define SS_NOT_READY 0x02
-#define SS_MEDIUM_ERR 0x03
-#define SS_HW_ERR 0x04
-#define SS_ILLEGAL_REQUEST 0x05
-#define SS_UNIT_ATTENTION 0x06
+#define SS_SUCCESS 0x000000 /* No Sense */
+#define SS_NOT_READY 0x023A00 /* Medium not present */
+#define SS_MEDIUM_ERR 0x031100 /* Unrecovered read error */
+#define SS_HW_ERR 0x040800 /* Communication failure */
+#define SS_ILLEGAL_REQUEST 0x052000 /* Invalid command */
+#define SS_UNIT_ATTENTION 0x062900 /* Reset occurred */
/* ENE Load FW Pattern */
#define SD_INIT1_PATTERN 1
return USB_STOR_TRANSPORT_GOOD;
}
-static int sd_scsi_test_unit_ready(struct us_data *us, struct scsi_cmnd *srb)
+static int do_scsi_request_sense(struct us_data *us, struct scsi_cmnd *srb)
{
struct ene_ub6250_info *info = (struct ene_ub6250_info *) us->extra;
+ unsigned char buf[18];
- if (info->SD_Status.Insert && info->SD_Status.Ready)
- return USB_STOR_TRANSPORT_GOOD;
- else {
- ene_sd_init(us);
- return USB_STOR_TRANSPORT_GOOD;
- }
+ memset(buf, 0, 18);
+ buf[0] = 0x70; /* Current error */
+ buf[2] = info->SrbStatus >> 16; /* Sense key */
+ buf[7] = 10; /* Additional length */
+ buf[12] = info->SrbStatus >> 8; /* ASC */
+ buf[13] = info->SrbStatus; /* ASCQ */
+ usb_stor_set_xfer_buf(buf, sizeof(buf), srb);
return USB_STOR_TRANSPORT_GOOD;
}
-static int sd_scsi_inquiry(struct us_data *us, struct scsi_cmnd *srb)
+static int do_scsi_inquiry(struct us_data *us, struct scsi_cmnd *srb)
{
unsigned char data_ptr[36] = {
- 0x00, 0x80, 0x02, 0x00, 0x1F, 0x00, 0x00, 0x00, 0x55,
+ 0x00, 0x00, 0x02, 0x00, 0x1F, 0x00, 0x00, 0x00, 0x55,
0x53, 0x42, 0x32, 0x2E, 0x30, 0x20, 0x20, 0x43, 0x61,
0x72, 0x64, 0x52, 0x65, 0x61, 0x64, 0x65, 0x72, 0x20,
0x20, 0x20, 0x20, 0x20, 0x20, 0x30, 0x31, 0x30, 0x30 };
return USB_STOR_TRANSPORT_GOOD;
}
+static int sd_scsi_test_unit_ready(struct us_data *us, struct scsi_cmnd *srb)
+{
+ struct ene_ub6250_info *info = (struct ene_ub6250_info *) us->extra;
+
+ if (info->SD_Status.Insert && info->SD_Status.Ready)
+ return USB_STOR_TRANSPORT_GOOD;
+ else {
+ ene_sd_init(us);
+ return USB_STOR_TRANSPORT_GOOD;
+ }
+
+ return USB_STOR_TRANSPORT_GOOD;
+}
+
static int sd_scsi_mode_sense(struct us_data *us, struct scsi_cmnd *srb)
{
struct ene_ub6250_info *info = (struct ene_ub6250_info *) us->extra;
return USB_STOR_TRANSPORT_GOOD;
}
-static int ms_scsi_inquiry(struct us_data *us, struct scsi_cmnd *srb)
-{
- /* pr_info("MS_SCSI_Inquiry\n"); */
- unsigned char data_ptr[36] = {
- 0x00, 0x80, 0x02, 0x00, 0x1F, 0x00, 0x00, 0x00, 0x55,
- 0x53, 0x42, 0x32, 0x2E, 0x30, 0x20, 0x20, 0x43, 0x61,
- 0x72, 0x64, 0x52, 0x65, 0x61, 0x64, 0x65, 0x72, 0x20,
- 0x20, 0x20, 0x20, 0x20, 0x20, 0x30, 0x31, 0x30, 0x30};
-
- usb_stor_set_xfer_buf(data_ptr, 36, srb);
- return USB_STOR_TRANSPORT_GOOD;
-}
-
static int ms_scsi_mode_sense(struct us_data *us, struct scsi_cmnd *srb)
{
struct ene_ub6250_info *info = (struct ene_ub6250_info *) us->extra;
bcb->CDB[0] = 0xEF;
result = ene_send_scsi_cmd(us, FDIR_WRITE, buf, 0);
+ if (us->srb != NULL)
+ scsi_set_resid(us->srb, 0);
info->BIN_FLAG = flag;
kfree(buf);
int result;
struct ene_ub6250_info *info = (struct ene_ub6250_info *)us->extra;
- info->SrbStatus = SS_SUCCESS;
switch (srb->cmnd[0]) {
case TEST_UNIT_READY:
result = sd_scsi_test_unit_ready(us, srb);
break; /* 0x00 */
+ case REQUEST_SENSE:
+ result = do_scsi_request_sense(us, srb);
+ break; /* 0x03 */
case INQUIRY:
- result = sd_scsi_inquiry(us, srb);
+ result = do_scsi_inquiry(us, srb);
break; /* 0x12 */
case MODE_SENSE:
result = sd_scsi_mode_sense(us, srb);
result = USB_STOR_TRANSPORT_FAILED;
break;
}
+ if (result == USB_STOR_TRANSPORT_GOOD)
+ info->SrbStatus = SS_SUCCESS;
return result;
}
{
int result;
struct ene_ub6250_info *info = (struct ene_ub6250_info *)us->extra;
- info->SrbStatus = SS_SUCCESS;
+
switch (srb->cmnd[0]) {
case TEST_UNIT_READY:
result = ms_scsi_test_unit_ready(us, srb);
break; /* 0x00 */
+ case REQUEST_SENSE:
+ result = do_scsi_request_sense(us, srb);
+ break; /* 0x03 */
case INQUIRY:
- result = ms_scsi_inquiry(us, srb);
+ result = do_scsi_inquiry(us, srb);
break; /* 0x12 */
case MODE_SENSE:
result = ms_scsi_mode_sense(us, srb);
result = USB_STOR_TRANSPORT_FAILED;
break;
}
+ if (result == USB_STOR_TRANSPORT_GOOD)
+ info->SrbStatus = SS_SUCCESS;
return result;
}
static int ene_transport(struct scsi_cmnd *srb, struct us_data *us)
{
- int result = 0;
+ int result = USB_STOR_XFER_GOOD;
struct ene_ub6250_info *info = (struct ene_ub6250_info *)(us->extra);
/*US_DEBUG(usb_stor_show_command(us, srb)); */
scsi_set_resid(srb, 0);
- if (unlikely(!(info->SD_Status.Ready || info->MS_Status.Ready))) {
+ if (unlikely(!(info->SD_Status.Ready || info->MS_Status.Ready)))
result = ene_init(us);
- } else {
+ if (result == USB_STOR_XFER_GOOD) {
+ result = USB_STOR_TRANSPORT_ERROR;
if (info->SD_Status.Ready)
result = sd_scsi_irp(us, srb);
if (info->MS_Status.Ready)
result = ms_scsi_irp(us, srb);
}
- return 0;
+ return result;
}
static struct scsi_host_template ene_ub6250_host_template;
To compile this driver as module, choose M here: the module will be
called typec_wcove
+source "drivers/usb/typec/ucsi/Kconfig"
+
endmenu
obj-$(CONFIG_TYPEC) += typec.o
obj-$(CONFIG_TYPEC_WCOVE) += typec_wcove.o
+obj-$(CONFIG_TYPEC_UCSI) += ucsi/
#include <linux/device.h>
#include <linux/module.h>
+#include <linux/mutex.h>
#include <linux/slab.h>
#include <linux/usb/typec.h>
enum typec_role pwr_role;
enum typec_role vconn_role;
enum typec_pwr_opmode pwr_opmode;
+ enum typec_port_type port_type;
+ struct mutex port_type_lock;
const struct typec_capability *cap;
};
}
static void typec_init_modes(struct typec_altmode *alt,
- struct typec_mode_desc *desc, bool is_port)
+ const struct typec_mode_desc *desc, bool is_port)
{
int i;
};
static struct typec_altmode *
-typec_register_altmode(struct device *parent, struct typec_altmode_desc *desc)
+typec_register_altmode(struct device *parent,
+ const struct typec_altmode_desc *desc)
{
struct typec_altmode *alt;
int ret;
*/
struct typec_altmode *
typec_partner_register_altmode(struct typec_partner *partner,
- struct typec_altmode_desc *desc)
+ const struct typec_altmode_desc *desc)
{
return typec_register_altmode(&partner->dev, desc);
}
*/
struct typec_altmode *
typec_plug_register_altmode(struct typec_plug *plug,
- struct typec_altmode_desc *desc)
+ const struct typec_altmode_desc *desc)
{
return typec_register_altmode(&plug->dev, desc);
}
[TYPEC_HOST] = "host",
};
+static const char * const typec_port_types[] = {
+ [TYPEC_PORT_DFP] = "source",
+ [TYPEC_PORT_UFP] = "sink",
+ [TYPEC_PORT_DRP] = "dual",
+};
+
+static const char * const typec_port_types_drp[] = {
+ [TYPEC_PORT_DFP] = "dual [source] sink",
+ [TYPEC_PORT_UFP] = "dual source [sink]",
+ [TYPEC_PORT_DRP] = "[dual] source sink",
+};
+
static ssize_t
preferred_role_store(struct device *dev, struct device_attribute *attr,
const char *buf, size_t size)
struct typec_port *port = to_typec_port(dev);
int ret;
- if (port->cap->type != TYPEC_PORT_DRP) {
- dev_dbg(dev, "data role swap only supported with DRP ports\n");
- return -EOPNOTSUPP;
- }
-
if (!port->cap->dr_set) {
dev_dbg(dev, "data role swapping not supported\n");
return -EOPNOTSUPP;
if (ret < 0)
return ret;
+ mutex_lock(&port->port_type_lock);
+ if (port->port_type != TYPEC_PORT_DRP) {
+ dev_dbg(dev, "port type fixed at \"%s\"",
+ typec_port_types[port->port_type]);
+ ret = -EOPNOTSUPP;
+ goto unlock_and_ret;
+ }
+
ret = port->cap->dr_set(port->cap, ret);
if (ret)
- return ret;
+ goto unlock_and_ret;
- return size;
+ ret = size;
+unlock_and_ret:
+ mutex_unlock(&port->port_type_lock);
+ return ret;
}
static ssize_t data_role_show(struct device *dev,
const char *buf, size_t size)
{
struct typec_port *port = to_typec_port(dev);
- int ret = size;
+ int ret;
if (!port->cap->pd_revision) {
dev_dbg(dev, "USB Power Delivery not supported\n");
if (ret < 0)
return ret;
+ mutex_lock(&port->port_type_lock);
+ if (port->port_type != TYPEC_PORT_DRP) {
+ dev_dbg(dev, "port type fixed at \"%s\"",
+ typec_port_types[port->port_type]);
+ ret = -EOPNOTSUPP;
+ goto unlock_and_ret;
+ }
+
ret = port->cap->pr_set(port->cap, ret);
if (ret)
- return ret;
+ goto unlock_and_ret;
- return size;
+ ret = size;
+unlock_and_ret:
+ mutex_unlock(&port->port_type_lock);
+ return ret;
}
static ssize_t power_role_show(struct device *dev,
}
static DEVICE_ATTR_RW(power_role);
+static ssize_t
+port_type_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t size)
+{
+ struct typec_port *port = to_typec_port(dev);
+ int ret;
+ enum typec_port_type type;
+
+ if (!port->cap->port_type_set || port->cap->type != TYPEC_PORT_DRP) {
+ dev_dbg(dev, "changing port type not supported\n");
+ return -EOPNOTSUPP;
+ }
+
+ ret = sysfs_match_string(typec_port_types, buf);
+ if (ret < 0)
+ return ret;
+
+ type = ret;
+ mutex_lock(&port->port_type_lock);
+
+ if (port->port_type == type) {
+ ret = size;
+ goto unlock_and_ret;
+ }
+
+ ret = port->cap->port_type_set(port->cap, type);
+ if (ret)
+ goto unlock_and_ret;
+
+ port->port_type = type;
+ ret = size;
+
+unlock_and_ret:
+ mutex_unlock(&port->port_type_lock);
+ return ret;
+}
+
+static ssize_t
+port_type_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct typec_port *port = to_typec_port(dev);
+
+ if (port->cap->type == TYPEC_PORT_DRP)
+ return sprintf(buf, "%s\n",
+ typec_port_types_drp[port->port_type]);
+
+ return sprintf(buf, "[%s]\n", typec_port_types[port->cap->type]);
+}
+static DEVICE_ATTR_RW(port_type);
+
static const char * const typec_pwr_opmodes[] = {
[TYPEC_PWR_MODE_USB] = "default",
[TYPEC_PWR_MODE_1_5A] = "1.5A",
&dev_attr_usb_power_delivery_revision.attr,
&dev_attr_usb_typec_revision.attr,
&dev_attr_vconn_source.attr,
+ &dev_attr_port_type.attr,
NULL,
};
ATTRIBUTE_GROUPS(typec);
}
EXPORT_SYMBOL_GPL(typec_set_vconn_role);
+static int partner_match(struct device *dev, void *data)
+{
+ return is_typec_partner(dev);
+}
+
/**
* typec_set_pwr_opmode - Report changed power operation mode
* @port: The USB Type-C Port where the mode was changed
void typec_set_pwr_opmode(struct typec_port *port,
enum typec_pwr_opmode opmode)
{
+ struct device *partner_dev;
+
if (port->pwr_opmode == opmode)
return;
port->pwr_opmode = opmode;
sysfs_notify(&port->dev.kobj, NULL, "power_operation_mode");
kobject_uevent(&port->dev.kobj, KOBJ_CHANGE);
+
+ partner_dev = device_find_child(&port->dev, NULL, partner_match);
+ if (partner_dev) {
+ struct typec_partner *partner = to_typec_partner(partner_dev);
+
+ if (opmode == TYPEC_PWR_MODE_PD && !partner->usb_pd) {
+ partner->usb_pd = 1;
+ sysfs_notify(&partner_dev->kobj, NULL,
+ "supports_usb_power_delivery");
+ }
+ put_device(partner_dev);
+ }
}
EXPORT_SYMBOL_GPL(typec_set_pwr_opmode);
*/
struct typec_altmode *
typec_port_register_altmode(struct typec_port *port,
- struct typec_altmode_desc *desc)
+ const struct typec_altmode_desc *desc)
{
return typec_register_altmode(&port->dev, desc);
}
port->id = id;
port->cap = cap;
+ port->port_type = cap->type;
+ mutex_init(&port->port_type_lock);
port->prefer_role = cap->prefer_role;
port->dev.class = typec_class;
WCOVE_ROLE_DEVICE,
};
-static uuid_le uuid = UUID_LE(0x482383f0, 0x2876, 0x4e49,
- 0x86, 0x85, 0xdb, 0x66, 0x21, 0x1a, 0xf0, 0x37);
+static guid_t guid = GUID_INIT(0x482383f0, 0x2876, 0x4e49,
+ 0x86, 0x85, 0xdb, 0x66, 0x21, 0x1a, 0xf0, 0x37);
static int wcove_typec_func(struct wcove_typec *wcove,
enum wcove_typec_func func, int param)
tmp.type = ACPI_TYPE_INTEGER;
tmp.integer.value = param;
- obj = acpi_evaluate_dsm(ACPI_HANDLE(wcove->dev), uuid.b, 1, func,
+ obj = acpi_evaluate_dsm(ACPI_HANDLE(wcove->dev), &guid, 1, func,
&argv4);
if (!obj) {
dev_err(wcove->dev, "%s: failed to evaluate _DSM\n", __func__);
if (ret)
return ret;
- if (!acpi_check_dsm(ACPI_HANDLE(&pdev->dev), uuid.b, 0, 0x1f)) {
+ if (!acpi_check_dsm(ACPI_HANDLE(&pdev->dev), &guid, 0, 0x1f)) {
dev_err(&pdev->dev, "Missing _DSM functions\n");
return -ENODEV;
}
--- /dev/null
+config TYPEC_UCSI
+ tristate "USB Type-C Connector System Software Interface driver"
+ depends on !CPU_BIG_ENDIAN
+ select TYPEC
+ help
+ USB Type-C Connector System Software Interface (UCSI) is a
+ specification for an interface that allows the operating system to
+ control the USB Type-C ports. On UCSI system the USB Type-C ports
+ function autonomously by default, but in order to get the status of
+ the ports and support basic operations like role swapping, the driver
+ is required. UCSI is available on most of the new Intel based systems
+ that are equipped with Embedded Controller and USB Type-C ports.
+
+ UCSI specification does not define the interface method, so depending
+ on the platform, ACPI, PCI, I2C, etc. may be used. Therefore this
+ driver only provides the core part, and separate drivers are needed
+ for every supported interface method.
+
+ The UCSI specification can be downloaded from:
+ http://www.intel.com/content/www/us/en/io/universal-serial-bus/usb-type-c-ucsi-spec.html
+
+ To compile the driver as a module, choose M here: the module will be
+ called typec_ucsi.
+
+if TYPEC_UCSI
+
+config UCSI_ACPI
+ tristate "UCSI ACPI Interface Driver"
+ depends on ACPI
+ help
+ This driver enables UCSI support on platforms that expose UCSI
+ interface as ACPI device. On new Intel Atom based platforms starting
+ from Broxton SoCs and Core platforms stating from Skylake, UCSI is an
+ ACPI enumerated device.
+
+ To compile the driver as a module, choose M here: the module will be
+ called ucsi_acpi
+
+endif
--- /dev/null
+CFLAGS_trace.o := -I$(src)
+
+obj-$(CONFIG_TYPEC_UCSI) += typec_ucsi.o
+
+typec_ucsi-y := ucsi.o
+
+typec_ucsi-$(CONFIG_FTRACE) += trace.o
+
+obj-$(CONFIG_UCSI_ACPI) += ucsi_acpi.o
--- /dev/null
+#ifndef __UCSI_DEBUG_H
+#define __UCSI_DEBUG_H
+
+#include "ucsi.h"
+
+static const char * const ucsi_cmd_strs[] = {
+ [0] = "Unknown command",
+ [UCSI_PPM_RESET] = "PPM_RESET",
+ [UCSI_CANCEL] = "CANCEL",
+ [UCSI_CONNECTOR_RESET] = "CONNECTOR_RESET",
+ [UCSI_ACK_CC_CI] = "ACK_CC_CI",
+ [UCSI_SET_NOTIFICATION_ENABLE] = "SET_NOTIFICATION_ENABLE",
+ [UCSI_GET_CAPABILITY] = "GET_CAPABILITY",
+ [UCSI_GET_CONNECTOR_CAPABILITY] = "GET_CONNECTOR_CAPABILITY",
+ [UCSI_SET_UOM] = "SET_UOM",
+ [UCSI_SET_UOR] = "SET_UOR",
+ [UCSI_SET_PDM] = "SET_PDM",
+ [UCSI_SET_PDR] = "SET_PDR",
+ [UCSI_GET_ALTERNATE_MODES] = "GET_ALTERNATE_MODES",
+ [UCSI_GET_CAM_SUPPORTED] = "GET_CAM_SUPPORTED",
+ [UCSI_GET_CURRENT_CAM] = "GET_CURRENT_CAM",
+ [UCSI_SET_NEW_CAM] = "SET_NEW_CAM",
+ [UCSI_GET_PDOS] = "GET_PDOS",
+ [UCSI_GET_CABLE_PROPERTY] = "GET_CABLE_PROPERTY",
+ [UCSI_GET_CONNECTOR_STATUS] = "GET_CONNECTOR_STATUS",
+ [UCSI_GET_ERROR_STATUS] = "GET_ERROR_STATUS",
+};
+
+static inline const char *ucsi_cmd_str(u64 raw_cmd)
+{
+ u8 cmd = raw_cmd & GENMASK(7, 0);
+
+ return ucsi_cmd_strs[(cmd >= ARRAY_SIZE(ucsi_cmd_strs)) ? 0 : cmd];
+}
+
+static const char * const ucsi_ack_strs[] = {
+ [0] = "",
+ [UCSI_ACK_EVENT] = "event",
+ [UCSI_ACK_CMD] = "command",
+};
+
+static inline const char *ucsi_ack_str(u8 ack)
+{
+ return ucsi_ack_strs[(ack >= ARRAY_SIZE(ucsi_ack_strs)) ? 0 : ack];
+}
+
+static inline const char *ucsi_cci_str(u32 cci)
+{
+ if (cci & GENMASK(7, 0)) {
+ if (cci & BIT(29))
+ return "Event pending (ACK completed)";
+ if (cci & BIT(31))
+ return "Event pending (command completed)";
+ return "Connector Change";
+ }
+ if (cci & BIT(29))
+ return "ACK completed";
+ if (cci & BIT(31))
+ return "Command completed";
+
+ return "";
+}
+
+#endif /* __UCSI_DEBUG_H */
--- /dev/null
+#define CREATE_TRACE_POINTS
+#include "trace.h"
--- /dev/null
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM ucsi
+
+#if !defined(__UCSI_TRACE_H) || defined(TRACE_HEADER_MULTI_READ)
+#define __UCSI_TRACE_H
+
+#include <linux/tracepoint.h>
+#include "ucsi.h"
+#include "debug.h"
+
+DECLARE_EVENT_CLASS(ucsi_log_ack,
+ TP_PROTO(u8 ack),
+ TP_ARGS(ack),
+ TP_STRUCT__entry(
+ __field(u8, ack)
+ ),
+ TP_fast_assign(
+ __entry->ack = ack;
+ ),
+ TP_printk("ACK %s", ucsi_ack_str(__entry->ack))
+);
+
+DEFINE_EVENT(ucsi_log_ack, ucsi_ack,
+ TP_PROTO(u8 ack),
+ TP_ARGS(ack)
+);
+
+DECLARE_EVENT_CLASS(ucsi_log_control,
+ TP_PROTO(struct ucsi_control *ctrl),
+ TP_ARGS(ctrl),
+ TP_STRUCT__entry(
+ __field(u64, ctrl)
+ ),
+ TP_fast_assign(
+ __entry->ctrl = ctrl->raw_cmd;
+ ),
+ TP_printk("control=%08llx (%s)", __entry->ctrl,
+ ucsi_cmd_str(__entry->ctrl))
+);
+
+DEFINE_EVENT(ucsi_log_control, ucsi_command,
+ TP_PROTO(struct ucsi_control *ctrl),
+ TP_ARGS(ctrl)
+);
+
+DECLARE_EVENT_CLASS(ucsi_log_command,
+ TP_PROTO(struct ucsi_control *ctrl, int ret),
+ TP_ARGS(ctrl, ret),
+ TP_STRUCT__entry(
+ __field(u64, ctrl)
+ __field(int, ret)
+ ),
+ TP_fast_assign(
+ __entry->ctrl = ctrl->raw_cmd;
+ __entry->ret = ret;
+ ),
+ TP_printk("%s -> %s (err=%d)", ucsi_cmd_str(__entry->ctrl),
+ __entry->ret < 0 ? "FAIL" : "OK",
+ __entry->ret < 0 ? __entry->ret : 0)
+);
+
+DEFINE_EVENT(ucsi_log_command, ucsi_run_command,
+ TP_PROTO(struct ucsi_control *ctrl, int ret),
+ TP_ARGS(ctrl, ret)
+);
+
+DEFINE_EVENT(ucsi_log_command, ucsi_reset_ppm,
+ TP_PROTO(struct ucsi_control *ctrl, int ret),
+ TP_ARGS(ctrl, ret)
+);
+
+DECLARE_EVENT_CLASS(ucsi_log_cci,
+ TP_PROTO(u32 cci),
+ TP_ARGS(cci),
+ TP_STRUCT__entry(
+ __field(u32, cci)
+ ),
+ TP_fast_assign(
+ __entry->cci = cci;
+ ),
+ TP_printk("CCI=%08x %s", __entry->cci, ucsi_cci_str(__entry->cci))
+);
+
+DEFINE_EVENT(ucsi_log_cci, ucsi_notify,
+ TP_PROTO(u32 cci),
+ TP_ARGS(cci)
+);
+
+DECLARE_EVENT_CLASS(ucsi_log_connector_status,
+ TP_PROTO(int port, struct ucsi_connector_status *status),
+ TP_ARGS(port, status),
+ TP_STRUCT__entry(
+ __field(int, port)
+ __field(u16, change)
+ __field(u8, opmode)
+ __field(u8, connected)
+ __field(u8, pwr_dir)
+ __field(u8, partner_flags)
+ __field(u8, partner_type)
+ __field(u32, request_data_obj)
+ __field(u8, bc_status)
+ ),
+ TP_fast_assign(
+ __entry->port = port - 1;
+ __entry->change = status->change;
+ __entry->opmode = status->pwr_op_mode;
+ __entry->connected = status->connected;
+ __entry->pwr_dir = status->pwr_dir;
+ __entry->partner_flags = status->partner_flags;
+ __entry->partner_type = status->partner_type;
+ __entry->request_data_obj = status->request_data_obj;
+ __entry->bc_status = status->bc_status;
+ ),
+ TP_printk("port%d status: change=%04x, opmode=%x, connected=%d, "
+ "sourcing=%d, partner_flags=%x, partner_type=%x, "
+ "request_data_obj=%08x, BC status=%x", __entry->port,
+ __entry->change, __entry->opmode, __entry->connected,
+ __entry->pwr_dir, __entry->partner_flags, __entry->partner_type,
+ __entry->request_data_obj, __entry->bc_status)
+);
+
+DEFINE_EVENT(ucsi_log_connector_status, ucsi_connector_change,
+ TP_PROTO(int port, struct ucsi_connector_status *status),
+ TP_ARGS(port, status)
+);
+
+DEFINE_EVENT(ucsi_log_connector_status, ucsi_register_port,
+ TP_PROTO(int port, struct ucsi_connector_status *status),
+ TP_ARGS(port, status)
+);
+
+#endif /* __UCSI_TRACE_H */
+
+/* This part must be outside protection */
+
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH .
+
+#undef TRACE_INCLUDE_FILE
+#define TRACE_INCLUDE_FILE trace
+
+#include <trace/define_trace.h>
--- /dev/null
+/*
+ * USB Type-C Connector System Software Interface driver
+ *
+ * Copyright (C) 2017, Intel Corporation
+ * Author: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/completion.h>
+#include <linux/property.h>
+#include <linux/device.h>
+#include <linux/module.h>
+#include <linux/delay.h>
+#include <linux/slab.h>
+#include <linux/usb/typec.h>
+
+#include "ucsi.h"
+#include "trace.h"
+
+#define to_ucsi_connector(_cap_) container_of(_cap_, struct ucsi_connector, \
+ typec_cap)
+
+/*
+ * UCSI_TIMEOUT_MS - PPM communication timeout
+ *
+ * Ideally we could use MIN_TIME_TO_RESPOND_WITH_BUSY (which is defined in UCSI
+ * specification) here as reference, but unfortunately we can't. It is very
+ * difficult to estimate the time it takes for the system to process the command
+ * before it is actually passed to the PPM.
+ */
+#define UCSI_TIMEOUT_MS 1000
+
+/*
+ * UCSI_SWAP_TIMEOUT_MS - Timeout for role swap requests
+ *
+ * 5 seconds is close to the time it takes for CapsCounter to reach 0, so even
+ * if the PPM does not generate Connector Change events before that with
+ * partners that do not support USB Power Delivery, this should still work.
+ */
+#define UCSI_SWAP_TIMEOUT_MS 5000
+
+enum ucsi_status {
+ UCSI_IDLE = 0,
+ UCSI_BUSY,
+ UCSI_ERROR,
+};
+
+struct ucsi_connector {
+ int num;
+
+ struct ucsi *ucsi;
+ struct work_struct work;
+ struct completion complete;
+
+ struct typec_port *port;
+ struct typec_partner *partner;
+
+ struct typec_capability typec_cap;
+
+ struct ucsi_connector_status status;
+ struct ucsi_connector_capability cap;
+};
+
+struct ucsi {
+ struct device *dev;
+ struct ucsi_ppm *ppm;
+
+ enum ucsi_status status;
+ struct completion complete;
+ struct ucsi_capability cap;
+ struct ucsi_connector *connector;
+
+ struct work_struct work;
+
+ /* PPM Communication lock */
+ struct mutex ppm_lock;
+
+ /* PPM communication flags */
+ unsigned long flags;
+#define EVENT_PENDING 0
+#define COMMAND_PENDING 1
+#define ACK_PENDING 2
+};
+
+static inline int ucsi_sync(struct ucsi *ucsi)
+{
+ if (ucsi->ppm && ucsi->ppm->sync)
+ return ucsi->ppm->sync(ucsi->ppm);
+ return 0;
+}
+
+static int ucsi_command(struct ucsi *ucsi, struct ucsi_control *ctrl)
+{
+ int ret;
+
+ trace_ucsi_command(ctrl);
+
+ set_bit(COMMAND_PENDING, &ucsi->flags);
+
+ ret = ucsi->ppm->cmd(ucsi->ppm, ctrl);
+ if (ret)
+ goto err_clear_flag;
+
+ if (!wait_for_completion_timeout(&ucsi->complete,
+ msecs_to_jiffies(UCSI_TIMEOUT_MS))) {
+ dev_warn(ucsi->dev, "PPM NOT RESPONDING\n");
+ ret = -ETIMEDOUT;
+ }
+
+err_clear_flag:
+ clear_bit(COMMAND_PENDING, &ucsi->flags);
+
+ return ret;
+}
+
+static int ucsi_ack(struct ucsi *ucsi, u8 ack)
+{
+ struct ucsi_control ctrl;
+ int ret;
+
+ trace_ucsi_ack(ack);
+
+ set_bit(ACK_PENDING, &ucsi->flags);
+
+ UCSI_CMD_ACK(ctrl, ack);
+ ret = ucsi->ppm->cmd(ucsi->ppm, &ctrl);
+ if (ret)
+ goto out_clear_bit;
+
+ /* Waiting for ACK with ACK CMD, but not with EVENT for now */
+ if (ack == UCSI_ACK_EVENT)
+ goto out_clear_bit;
+
+ if (!wait_for_completion_timeout(&ucsi->complete,
+ msecs_to_jiffies(UCSI_TIMEOUT_MS)))
+ ret = -ETIMEDOUT;
+
+out_clear_bit:
+ clear_bit(ACK_PENDING, &ucsi->flags);
+
+ if (ret)
+ dev_err(ucsi->dev, "%s: failed\n", __func__);
+
+ return ret;
+}
+
+static int ucsi_run_command(struct ucsi *ucsi, struct ucsi_control *ctrl,
+ void *data, size_t size)
+{
+ struct ucsi_control _ctrl;
+ u8 data_length;
+ u16 error;
+ int ret;
+
+ ret = ucsi_command(ucsi, ctrl);
+ if (ret)
+ goto err;
+
+ switch (ucsi->status) {
+ case UCSI_IDLE:
+ ret = ucsi_sync(ucsi);
+ if (ret)
+ dev_warn(ucsi->dev, "%s: sync failed\n", __func__);
+
+ if (data)
+ memcpy(data, ucsi->ppm->data->message_in, size);
+
+ data_length = ucsi->ppm->data->cci.data_length;
+
+ ret = ucsi_ack(ucsi, UCSI_ACK_CMD);
+ if (!ret)
+ ret = data_length;
+ break;
+ case UCSI_BUSY:
+ /* The caller decides whether to cancel or not */
+ ret = -EBUSY;
+ break;
+ case UCSI_ERROR:
+ ret = ucsi_ack(ucsi, UCSI_ACK_CMD);
+ if (ret)
+ break;
+
+ _ctrl.raw_cmd = 0;
+ _ctrl.cmd.cmd = UCSI_GET_ERROR_STATUS;
+ ret = ucsi_command(ucsi, &_ctrl);
+ if (ret) {
+ dev_err(ucsi->dev, "reading error failed!\n");
+ break;
+ }
+
+ memcpy(&error, ucsi->ppm->data->message_in, sizeof(error));
+
+ /* Something has really gone wrong */
+ if (WARN_ON(ucsi->status == UCSI_ERROR)) {
+ ret = -ENODEV;
+ break;
+ }
+
+ ret = ucsi_ack(ucsi, UCSI_ACK_CMD);
+ if (ret)
+ break;
+
+ switch (error) {
+ case UCSI_ERROR_INCOMPATIBLE_PARTNER:
+ ret = -EOPNOTSUPP;
+ break;
+ case UCSI_ERROR_CC_COMMUNICATION_ERR:
+ ret = -ECOMM;
+ break;
+ case UCSI_ERROR_CONTRACT_NEGOTIATION_FAIL:
+ ret = -EPROTO;
+ break;
+ case UCSI_ERROR_DEAD_BATTERY:
+ dev_warn(ucsi->dev, "Dead battery condition!\n");
+ ret = -EPERM;
+ break;
+ /* The following mean a bug in this driver */
+ case UCSI_ERROR_INVALID_CON_NUM:
+ case UCSI_ERROR_UNREGONIZED_CMD:
+ case UCSI_ERROR_INVALID_CMD_ARGUMENT:
+ dev_warn(ucsi->dev,
+ "%s: possible UCSI driver bug - error 0x%x\n",
+ __func__, error);
+ ret = -EINVAL;
+ break;
+ default:
+ dev_warn(ucsi->dev,
+ "%s: error without status\n", __func__);
+ ret = -EIO;
+ break;
+ }
+ break;
+ }
+
+err:
+ trace_ucsi_run_command(ctrl, ret);
+
+ return ret;
+}
+
+/* -------------------------------------------------------------------------- */
+
+static void ucsi_pwr_opmode_change(struct ucsi_connector *con)
+{
+ switch (con->status.pwr_op_mode) {
+ case UCSI_CONSTAT_PWR_OPMODE_PD:
+ typec_set_pwr_opmode(con->port, TYPEC_PWR_MODE_PD);
+ break;
+ case UCSI_CONSTAT_PWR_OPMODE_TYPEC1_5:
+ typec_set_pwr_opmode(con->port, TYPEC_PWR_MODE_1_5A);
+ break;
+ case UCSI_CONSTAT_PWR_OPMODE_TYPEC3_0:
+ typec_set_pwr_opmode(con->port, TYPEC_PWR_MODE_3_0A);
+ break;
+ default:
+ typec_set_pwr_opmode(con->port, TYPEC_PWR_MODE_USB);
+ break;
+ }
+}
+
+static int ucsi_register_partner(struct ucsi_connector *con)
+{
+ struct typec_partner_desc partner;
+
+ if (con->partner)
+ return 0;
+
+ memset(&partner, 0, sizeof(partner));
+
+ switch (con->status.partner_type) {
+ case UCSI_CONSTAT_PARTNER_TYPE_DEBUG:
+ partner.accessory = TYPEC_ACCESSORY_DEBUG;
+ break;
+ case UCSI_CONSTAT_PARTNER_TYPE_AUDIO:
+ partner.accessory = TYPEC_ACCESSORY_AUDIO;
+ break;
+ default:
+ break;
+ }
+
+ partner.usb_pd = con->status.pwr_op_mode == UCSI_CONSTAT_PWR_OPMODE_PD;
+
+ con->partner = typec_register_partner(con->port, &partner);
+ if (!con->partner) {
+ dev_err(con->ucsi->dev, "con%d: failed to register partner\n",
+ con->num);
+ return -ENODEV;
+ }
+
+ return 0;
+}
+
+static void ucsi_unregister_partner(struct ucsi_connector *con)
+{
+ typec_unregister_partner(con->partner);
+ con->partner = NULL;
+}
+
+static void ucsi_connector_change(struct work_struct *work)
+{
+ struct ucsi_connector *con = container_of(work, struct ucsi_connector,
+ work);
+ struct ucsi *ucsi = con->ucsi;
+ struct ucsi_control ctrl;
+ int ret;
+
+ mutex_lock(&ucsi->ppm_lock);
+
+ UCSI_CMD_GET_CONNECTOR_STATUS(ctrl, con->num);
+ ret = ucsi_run_command(ucsi, &ctrl, &con->status, sizeof(con->status));
+ if (ret < 0) {
+ dev_err(ucsi->dev, "%s: GET_CONNECTOR_STATUS failed (%d)\n",
+ __func__, ret);
+ goto out_unlock;
+ }
+
+ if (con->status.change & UCSI_CONSTAT_POWER_OPMODE_CHANGE)
+ ucsi_pwr_opmode_change(con);
+
+ if (con->status.change & UCSI_CONSTAT_POWER_DIR_CHANGE) {
+ typec_set_pwr_role(con->port, con->status.pwr_dir);
+
+ /* Complete pending power role swap */
+ if (!completion_done(&con->complete))
+ complete(&con->complete);
+ }
+
+ if (con->status.change & UCSI_CONSTAT_PARTNER_CHANGE) {
+ switch (con->status.partner_type) {
+ case UCSI_CONSTAT_PARTNER_TYPE_UFP:
+ typec_set_data_role(con->port, TYPEC_HOST);
+ break;
+ case UCSI_CONSTAT_PARTNER_TYPE_DFP:
+ typec_set_data_role(con->port, TYPEC_DEVICE);
+ break;
+ default:
+ break;
+ }
+
+ /* Complete pending data role swap */
+ if (!completion_done(&con->complete))
+ complete(&con->complete);
+ }
+
+ if (con->status.change & UCSI_CONSTAT_CONNECT_CHANGE) {
+ if (con->status.connected)
+ ucsi_register_partner(con);
+ else
+ ucsi_unregister_partner(con);
+ }
+
+ ret = ucsi_ack(ucsi, UCSI_ACK_EVENT);
+ if (ret)
+ dev_err(ucsi->dev, "%s: ACK failed (%d)", __func__, ret);
+
+ trace_ucsi_connector_change(con->num, &con->status);
+
+out_unlock:
+ clear_bit(EVENT_PENDING, &ucsi->flags);
+ mutex_unlock(&ucsi->ppm_lock);
+}
+
+/**
+ * ucsi_notify - PPM notification handler
+ * @ucsi: Source UCSI Interface for the notifications
+ *
+ * Handle notifications from PPM of @ucsi.
+ */
+void ucsi_notify(struct ucsi *ucsi)
+{
+ struct ucsi_cci *cci;
+
+ /* There is no requirement to sync here, but no harm either. */
+ ucsi_sync(ucsi);
+
+ cci = &ucsi->ppm->data->cci;
+
+ if (cci->error)
+ ucsi->status = UCSI_ERROR;
+ else if (cci->busy)
+ ucsi->status = UCSI_BUSY;
+ else
+ ucsi->status = UCSI_IDLE;
+
+ if (cci->cmd_complete && test_bit(COMMAND_PENDING, &ucsi->flags)) {
+ complete(&ucsi->complete);
+ } else if (cci->ack_complete && test_bit(ACK_PENDING, &ucsi->flags)) {
+ complete(&ucsi->complete);
+ } else if (cci->connector_change) {
+ struct ucsi_connector *con;
+
+ con = &ucsi->connector[cci->connector_change - 1];
+
+ if (!test_and_set_bit(EVENT_PENDING, &ucsi->flags))
+ schedule_work(&con->work);
+ }
+
+ trace_ucsi_notify(ucsi->ppm->data->raw_cci);
+}
+EXPORT_SYMBOL_GPL(ucsi_notify);
+
+/* -------------------------------------------------------------------------- */
+
+static int ucsi_reset_connector(struct ucsi_connector *con, bool hard)
+{
+ struct ucsi_control ctrl;
+
+ UCSI_CMD_CONNECTOR_RESET(ctrl, con, hard);
+
+ return ucsi_run_command(con->ucsi, &ctrl, NULL, 0);
+}
+
+static int ucsi_reset_ppm(struct ucsi *ucsi)
+{
+ struct ucsi_control ctrl;
+ unsigned long tmo;
+ int ret;
+
+ ctrl.raw_cmd = 0;
+ ctrl.cmd.cmd = UCSI_PPM_RESET;
+ trace_ucsi_command(&ctrl);
+ ret = ucsi->ppm->cmd(ucsi->ppm, &ctrl);
+ if (ret)
+ goto err;
+
+ tmo = jiffies + msecs_to_jiffies(UCSI_TIMEOUT_MS);
+
+ do {
+ /* Here sync is critical. */
+ ret = ucsi_sync(ucsi);
+ if (ret)
+ goto err;
+
+ if (ucsi->ppm->data->cci.reset_complete)
+ break;
+
+ /* If the PPM is still doing something else, reset it again. */
+ if (ucsi->ppm->data->raw_cci) {
+ dev_warn_ratelimited(ucsi->dev,
+ "Failed to reset PPM! Trying again..\n");
+
+ trace_ucsi_command(&ctrl);
+ ret = ucsi->ppm->cmd(ucsi->ppm, &ctrl);
+ if (ret)
+ goto err;
+ }
+
+ /* Letting the PPM settle down. */
+ msleep(20);
+
+ ret = -ETIMEDOUT;
+ } while (time_is_after_jiffies(tmo));
+
+err:
+ trace_ucsi_reset_ppm(&ctrl, ret);
+
+ return ret;
+}
+
+static int ucsi_role_cmd(struct ucsi_connector *con, struct ucsi_control *ctrl)
+{
+ int ret;
+
+ ret = ucsi_run_command(con->ucsi, ctrl, NULL, 0);
+ if (ret == -ETIMEDOUT) {
+ struct ucsi_control c;
+
+ /* PPM most likely stopped responding. Resetting everything. */
+ ucsi_reset_ppm(con->ucsi);
+
+ UCSI_CMD_SET_NTFY_ENABLE(c, UCSI_ENABLE_NTFY_ALL);
+ ucsi_run_command(con->ucsi, &c, NULL, 0);
+
+ ucsi_reset_connector(con, true);
+ }
+
+ return ret;
+}
+
+static int
+ucsi_dr_swap(const struct typec_capability *cap, enum typec_data_role role)
+{
+ struct ucsi_connector *con = to_ucsi_connector(cap);
+ struct ucsi_control ctrl;
+ int ret = 0;
+
+ if (!con->partner)
+ return -ENOTCONN;
+
+ mutex_lock(&con->ucsi->ppm_lock);
+
+ if ((con->status.partner_type == UCSI_CONSTAT_PARTNER_TYPE_DFP &&
+ role == TYPEC_DEVICE) ||
+ (con->status.partner_type == UCSI_CONSTAT_PARTNER_TYPE_UFP &&
+ role == TYPEC_HOST))
+ goto out_unlock;
+
+ UCSI_CMD_SET_UOR(ctrl, con, role);
+ ret = ucsi_role_cmd(con, &ctrl);
+ if (ret < 0)
+ goto out_unlock;
+
+ mutex_unlock(&con->ucsi->ppm_lock);
+
+ if (!wait_for_completion_timeout(&con->complete,
+ msecs_to_jiffies(UCSI_SWAP_TIMEOUT_MS)))
+ return -ETIMEDOUT;
+
+ return 0;
+
+out_unlock:
+ mutex_unlock(&con->ucsi->ppm_lock);
+
+ return ret;
+}
+
+static int
+ucsi_pr_swap(const struct typec_capability *cap, enum typec_role role)
+{
+ struct ucsi_connector *con = to_ucsi_connector(cap);
+ struct ucsi_control ctrl;
+ int ret = 0;
+
+ if (!con->partner)
+ return -ENOTCONN;
+
+ mutex_lock(&con->ucsi->ppm_lock);
+
+ if (con->status.pwr_dir == role)
+ goto out_unlock;
+
+ UCSI_CMD_SET_PDR(ctrl, con, role);
+ ret = ucsi_role_cmd(con, &ctrl);
+ if (ret < 0)
+ goto out_unlock;
+
+ mutex_unlock(&con->ucsi->ppm_lock);
+
+ if (!wait_for_completion_timeout(&con->complete,
+ msecs_to_jiffies(UCSI_SWAP_TIMEOUT_MS)))
+ return -ETIMEDOUT;
+
+ mutex_lock(&con->ucsi->ppm_lock);
+
+ /* Something has gone wrong while swapping the role */
+ if (con->status.pwr_op_mode != UCSI_CONSTAT_PWR_OPMODE_PD) {
+ ucsi_reset_connector(con, true);
+ ret = -EPROTO;
+ }
+
+out_unlock:
+ mutex_unlock(&con->ucsi->ppm_lock);
+
+ return ret;
+}
+
+static struct fwnode_handle *ucsi_find_fwnode(struct ucsi_connector *con)
+{
+ struct fwnode_handle *fwnode;
+ int i = 1;
+
+ device_for_each_child_node(con->ucsi->dev, fwnode)
+ if (i++ == con->num)
+ return fwnode;
+ return NULL;
+}
+
+static int ucsi_register_port(struct ucsi *ucsi, int index)
+{
+ struct ucsi_connector *con = &ucsi->connector[index];
+ struct typec_capability *cap = &con->typec_cap;
+ enum typec_accessory *accessory = cap->accessory;
+ struct ucsi_control ctrl;
+ int ret;
+
+ INIT_WORK(&con->work, ucsi_connector_change);
+ init_completion(&con->complete);
+ con->num = index + 1;
+ con->ucsi = ucsi;
+
+ /* Get connector capability */
+ UCSI_CMD_GET_CONNECTOR_CAPABILITY(ctrl, con->num);
+ ret = ucsi_run_command(ucsi, &ctrl, &con->cap, sizeof(con->cap));
+ if (ret < 0)
+ return ret;
+
+ if (con->cap.op_mode & UCSI_CONCAP_OPMODE_DRP)
+ cap->type = TYPEC_PORT_DRP;
+ else if (con->cap.op_mode & UCSI_CONCAP_OPMODE_DFP)
+ cap->type = TYPEC_PORT_DFP;
+ else if (con->cap.op_mode & UCSI_CONCAP_OPMODE_UFP)
+ cap->type = TYPEC_PORT_UFP;
+
+ cap->revision = ucsi->cap.typec_version;
+ cap->pd_revision = ucsi->cap.pd_version;
+ cap->prefer_role = TYPEC_NO_PREFERRED_ROLE;
+
+ if (con->cap.op_mode & UCSI_CONCAP_OPMODE_AUDIO_ACCESSORY)
+ *accessory++ = TYPEC_ACCESSORY_AUDIO;
+ if (con->cap.op_mode & UCSI_CONCAP_OPMODE_DEBUG_ACCESSORY)
+ *accessory = TYPEC_ACCESSORY_DEBUG;
+
+ cap->fwnode = ucsi_find_fwnode(con);
+ cap->dr_set = ucsi_dr_swap;
+ cap->pr_set = ucsi_pr_swap;
+
+ /* Register the connector */
+ con->port = typec_register_port(ucsi->dev, cap);
+ if (!con->port)
+ return -ENODEV;
+
+ /* Get the status */
+ UCSI_CMD_GET_CONNECTOR_STATUS(ctrl, con->num);
+ ret = ucsi_run_command(ucsi, &ctrl, &con->status, sizeof(con->status));
+ if (ret < 0) {
+ dev_err(ucsi->dev, "con%d: failed to get status\n", con->num);
+ return 0;
+ }
+
+ ucsi_pwr_opmode_change(con);
+ typec_set_pwr_role(con->port, con->status.pwr_dir);
+
+ switch (con->status.partner_type) {
+ case UCSI_CONSTAT_PARTNER_TYPE_UFP:
+ typec_set_data_role(con->port, TYPEC_HOST);
+ break;
+ case UCSI_CONSTAT_PARTNER_TYPE_DFP:
+ typec_set_data_role(con->port, TYPEC_DEVICE);
+ break;
+ default:
+ break;
+ }
+
+ /* Check if there is already something connected */
+ if (con->status.connected)
+ ucsi_register_partner(con);
+
+ trace_ucsi_register_port(con->num, &con->status);
+
+ return 0;
+}
+
+static void ucsi_init(struct work_struct *work)
+{
+ struct ucsi *ucsi = container_of(work, struct ucsi, work);
+ struct ucsi_connector *con;
+ struct ucsi_control ctrl;
+ int ret;
+ int i;
+
+ mutex_lock(&ucsi->ppm_lock);
+
+ /* Reset the PPM */
+ ret = ucsi_reset_ppm(ucsi);
+ if (ret) {
+ dev_err(ucsi->dev, "failed to reset PPM!\n");
+ goto err;
+ }
+
+ /* Enable basic notifications */
+ UCSI_CMD_SET_NTFY_ENABLE(ctrl, UCSI_ENABLE_NTFY_CMD_COMPLETE |
+ UCSI_ENABLE_NTFY_ERROR);
+ ret = ucsi_run_command(ucsi, &ctrl, NULL, 0);
+ if (ret < 0)
+ goto err_reset;
+
+ /* Get PPM capabilities */
+ UCSI_CMD_GET_CAPABILITY(ctrl);
+ ret = ucsi_run_command(ucsi, &ctrl, &ucsi->cap, sizeof(ucsi->cap));
+ if (ret < 0)
+ goto err_reset;
+
+ if (!ucsi->cap.num_connectors) {
+ ret = -ENODEV;
+ goto err_reset;
+ }
+
+ /* Allocate the connectors. Released in ucsi_unregister_ppm() */
+ ucsi->connector = kcalloc(ucsi->cap.num_connectors + 1,
+ sizeof(*ucsi->connector), GFP_KERNEL);
+ if (!ucsi->connector) {
+ ret = -ENOMEM;
+ goto err_reset;
+ }
+
+ /* Register all connectors */
+ for (i = 0; i < ucsi->cap.num_connectors; i++) {
+ ret = ucsi_register_port(ucsi, i);
+ if (ret)
+ goto err_unregister;
+ }
+
+ /* Enable all notifications */
+ UCSI_CMD_SET_NTFY_ENABLE(ctrl, UCSI_ENABLE_NTFY_ALL);
+ ret = ucsi_run_command(ucsi, &ctrl, NULL, 0);
+ if (ret < 0)
+ goto err_unregister;
+
+ mutex_unlock(&ucsi->ppm_lock);
+
+ return;
+
+err_unregister:
+ for (con = ucsi->connector; con->port; con++) {
+ ucsi_unregister_partner(con);
+ typec_unregister_port(con->port);
+ con->port = NULL;
+ }
+
+err_reset:
+ ucsi_reset_ppm(ucsi);
+err:
+ mutex_unlock(&ucsi->ppm_lock);
+ dev_err(ucsi->dev, "PPM init failed (%d)\n", ret);
+}
+
+/**
+ * ucsi_register_ppm - Register UCSI PPM Interface
+ * @dev: Device interface to the PPM
+ * @ppm: The PPM interface
+ *
+ * Allocates UCSI instance, associates it with @ppm and returns it to the
+ * caller, and schedules initialization of the interface.
+ */
+struct ucsi *ucsi_register_ppm(struct device *dev, struct ucsi_ppm *ppm)
+{
+ struct ucsi *ucsi;
+
+ ucsi = kzalloc(sizeof(*ucsi), GFP_KERNEL);
+ if (!ucsi)
+ return ERR_PTR(-ENOMEM);
+
+ INIT_WORK(&ucsi->work, ucsi_init);
+ init_completion(&ucsi->complete);
+ mutex_init(&ucsi->ppm_lock);
+
+ ucsi->dev = dev;
+ ucsi->ppm = ppm;
+
+ /*
+ * Communication with the PPM takes a lot of time. It is not reasonable
+ * to initialize the driver here. Using a work for now.
+ */
+ queue_work(system_long_wq, &ucsi->work);
+
+ return ucsi;
+}
+EXPORT_SYMBOL_GPL(ucsi_register_ppm);
+
+/**
+ * ucsi_unregister_ppm - Unregister UCSI PPM Interface
+ * @ucsi: struct ucsi associated with the PPM
+ *
+ * Unregister UCSI PPM that was created with ucsi_register().
+ */
+void ucsi_unregister_ppm(struct ucsi *ucsi)
+{
+ struct ucsi_control ctrl;
+ int i;
+
+ /* Make sure that we are not in the middle of driver initialization */
+ cancel_work_sync(&ucsi->work);
+
+ mutex_lock(&ucsi->ppm_lock);
+
+ /* Disable everything except command complete notification */
+ UCSI_CMD_SET_NTFY_ENABLE(ctrl, UCSI_ENABLE_NTFY_CMD_COMPLETE)
+ ucsi_run_command(ucsi, &ctrl, NULL, 0);
+
+ mutex_unlock(&ucsi->ppm_lock);
+
+ for (i = 0; i < ucsi->cap.num_connectors; i++) {
+ cancel_work_sync(&ucsi->connector[i].work);
+ ucsi_unregister_partner(&ucsi->connector[i]);
+ typec_unregister_port(ucsi->connector[i].port);
+ }
+
+ ucsi_reset_ppm(ucsi);
+
+ kfree(ucsi->connector);
+ kfree(ucsi);
+}
+EXPORT_SYMBOL_GPL(ucsi_unregister_ppm);
+
+MODULE_AUTHOR("Heikki Krogerus <heikki.krogerus@linux.intel.com>");
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("USB Type-C Connector System Software Interface driver");
+#ifndef __DRIVER_USB_TYPEC_UCSI_H
+#define __DRIVER_USB_TYPEC_UCSI_H
+
+#include <linux/bitops.h>
#include <linux/types.h>
/* -------------------------------------------------------------------------- */
/* Command Status and Connector Change Indication (CCI) data structure */
struct ucsi_cci {
- unsigned int RESERVED1:1;
- unsigned int connector_change:7;
+ u8:1; /* reserved */
+ u8 connector_change:7;
u8 data_length;
- unsigned int RESERVED9:9;
- unsigned int not_supported:1;
- unsigned int cancel_complete:1;
- unsigned int reset_complete:1;
- unsigned int busy:1;
- unsigned int ack_complete:1;
- unsigned int error:1;
- unsigned int cmd_complete:1;
+ u16:9; /* reserved */
+ u16 not_supported:1;
+ u16 cancel_complete:1;
+ u16 reset_complete:1;
+ u16 busy:1;
+ u16 ack_complete:1;
+ u16 error:1;
+ u16 cmd_complete:1;
} __packed;
/* Default fields in CONTROL data structure */
u64 data:48;
} __packed;
+/* ACK Command structure */
+struct ucsi_ack_cmd {
+ u8 cmd;
+ u8 length;
+ u8 cci_ack:1;
+ u8 cmd_ack:1;
+ u8:6; /* reserved */
+} __packed;
+
+/* Connector Reset Command structure */
+struct ucsi_con_rst {
+ u8 cmd;
+ u8 length;
+ u8 con_num:7;
+ u8 hard_reset:1;
+} __packed;
+
/* Set USB Operation Mode Command structure */
struct ucsi_uor_cmd {
u8 cmd;
u8 length;
- u64 con_num:7;
- u64 role:3;
+ u16 con_num:7;
+ u16 role:3;
#define UCSI_UOR_ROLE_DFP BIT(0)
#define UCSI_UOR_ROLE_UFP BIT(1)
#define UCSI_UOR_ROLE_DRP BIT(2)
- u64 data:38;
+ u16:6; /* reserved */
} __packed;
struct ucsi_control {
u64 raw_cmd;
struct ucsi_command cmd;
struct ucsi_uor_cmd uor;
+ struct ucsi_ack_cmd ack;
+ struct ucsi_con_rst con_rst;
};
};
-struct ucsi_data {
- u16 version;
- u16 RESERVED;
- union {
- u32 raw_cci;
- struct ucsi_cci cci;
- };
- struct ucsi_control ctrl;
- u32 message_in[4];
- u32 message_out[4];
-} __packed;
+#define __UCSI_CMD(_ctrl_, _cmd_) \
+{ \
+ (_ctrl_).raw_cmd = 0; \
+ (_ctrl_).cmd.cmd = _cmd_; \
+}
+
+/* Helper for preparing ucsi_control for CONNECTOR_RESET command. */
+#define UCSI_CMD_CONNECTOR_RESET(_ctrl_, _con_, _hard_) \
+{ \
+ __UCSI_CMD(_ctrl_, UCSI_CONNECTOR_RESET) \
+ (_ctrl_).con_rst.con_num = (_con_)->num; \
+ (_ctrl_).con_rst.hard_reset = _hard_; \
+}
+
+/* Helper for preparing ucsi_control for ACK_CC_CI command. */
+#define UCSI_CMD_ACK(_ctrl_, _ack_) \
+{ \
+ __UCSI_CMD(_ctrl_, UCSI_ACK_CC_CI) \
+ (_ctrl_).ack.cci_ack = ((_ack_) == UCSI_ACK_EVENT); \
+ (_ctrl_).ack.cmd_ack = ((_ack_) == UCSI_ACK_CMD); \
+}
+
+/* Helper for preparing ucsi_control for SET_NOTIFY_ENABLE command. */
+#define UCSI_CMD_SET_NTFY_ENABLE(_ctrl_, _ntfys_) \
+{ \
+ __UCSI_CMD(_ctrl_, UCSI_SET_NOTIFICATION_ENABLE) \
+ (_ctrl_).cmd.data = _ntfys_; \
+}
+
+/* Helper for preparing ucsi_control for GET_CAPABILITY command. */
+#define UCSI_CMD_GET_CAPABILITY(_ctrl_) \
+{ \
+ __UCSI_CMD(_ctrl_, UCSI_GET_CAPABILITY) \
+}
+
+/* Helper for preparing ucsi_control for GET_CONNECTOR_CAPABILITY command. */
+#define UCSI_CMD_GET_CONNECTOR_CAPABILITY(_ctrl_, _con_) \
+{ \
+ __UCSI_CMD(_ctrl_, UCSI_GET_CONNECTOR_CAPABILITY) \
+ (_ctrl_).cmd.data = _con_; \
+}
+
+/* Helper for preparing ucsi_control for GET_CONNECTOR_STATUS command. */
+#define UCSI_CMD_GET_CONNECTOR_STATUS(_ctrl_, _con_) \
+{ \
+ __UCSI_CMD(_ctrl_, UCSI_GET_CONNECTOR_STATUS) \
+ (_ctrl_).cmd.data = _con_; \
+}
+
+#define __UCSI_ROLE(_ctrl_, _cmd_, _con_num_) \
+{ \
+ __UCSI_CMD(_ctrl_, _cmd_) \
+ (_ctrl_).uor.con_num = _con_num_; \
+ (_ctrl_).uor.role = UCSI_UOR_ROLE_DRP; \
+}
+
+/* Helper for preparing ucsi_control for SET_UOR command. */
+#define UCSI_CMD_SET_UOR(_ctrl_, _con_, _role_) \
+{ \
+ __UCSI_ROLE(_ctrl_, UCSI_SET_UOR, (_con_)->num) \
+ (_ctrl_).uor.role |= (_role_) == TYPEC_HOST ? UCSI_UOR_ROLE_DFP : \
+ UCSI_UOR_ROLE_UFP; \
+}
+
+/* Helper for preparing ucsi_control for SET_PDR command. */
+#define UCSI_CMD_SET_PDR(_ctrl_, _con_, _role_) \
+{ \
+ __UCSI_ROLE(_ctrl_, UCSI_SET_PDR, (_con_)->num) \
+ (_ctrl_).uor.role |= (_role_) == TYPEC_SOURCE ? UCSI_UOR_ROLE_DFP : \
+ UCSI_UOR_ROLE_UFP; \
+}
/* Commands */
#define UCSI_PPM_RESET 0x01
#define UCSI_GET_CONNECTOR_CAPABILITY 0x07
#define UCSI_SET_UOM 0x08
#define UCSI_SET_UOR 0x09
-#define UCSI_SET_PDM 0x0A
-#define UCSI_SET_PDR 0x0B
-#define UCSI_GET_ALTERNATE_MODES 0x0C
-#define UCSI_GET_CAM_SUPPORTED 0x0D
-#define UCSI_GET_CURRENT_CAM 0x0E
-#define UCSI_SET_NEW_CAM 0x0F
+#define UCSI_SET_PDM 0x0a
+#define UCSI_SET_PDR 0x0b
+#define UCSI_GET_ALTERNATE_MODES 0x0c
+#define UCSI_GET_CAM_SUPPORTED 0x0d
+#define UCSI_GET_CURRENT_CAM 0x0e
+#define UCSI_SET_NEW_CAM 0x0f
#define UCSI_GET_PDOS 0x10
#define UCSI_GET_CABLE_PROPERTY 0x11
#define UCSI_GET_CONNECTOR_STATUS 0x12
#define UCSI_CAP_ATTR_POWER_AC_SUPPLY BIT(8)
#define UCSI_CAP_ATTR_POWER_OTHER BIT(10)
#define UCSI_CAP_ATTR_POWER_VBUS BIT(14)
- u8 num_connectors;
+ u32 num_connectors:8;
u32 features:24;
#define UCSI_CAP_SET_UOM BIT(0)
#define UCSI_CAP_SET_PDM BIT(1)
#define UCSI_CAP_EXT_SUPPLY_NOTIFICATIONS BIT(6)
#define UCSI_CAP_PD_RESET BIT(7)
u8 num_alt_modes;
- u8 RESERVED;
+ u8 reserved;
u16 bc_version;
u16 pd_version;
u16 typec_version;
#define UCSI_CONCAP_OPMODE_ALT_MODE BIT(7)
u8 provider:1;
u8 consumer:1;
+ u8:6; /* reserved */
+} __packed;
+
+struct ucsi_altmode {
+ u16 svid;
+ u32 mid;
} __packed;
/* Data structure filled by PPM in response to GET_CABLE_PROPERTY command. */
#define UCSI_CABLE_PROPERTY_PLUG_TYPE_C 2
#define UCSI_CABLE_PROPERTY_PLUG_OTHER 3
u8 mode_support:1;
- u8 RESERVED_2:2;
+ u8:2; /* reserved */
u8 latency:4;
- u8 RESERVED_4:4;
+ u8:4; /* reserved */
} __packed;
/* Data structure filled by PPM in response to GET_CONNECTOR_STATUS command. */
#define UCSI_CONSTAT_PWR_OPMODE_DEFAULT 1
#define UCSI_CONSTAT_PWR_OPMODE_BC 2
#define UCSI_CONSTAT_PWR_OPMODE_PD 3
-#define UCSI_CONSTAT_PWR_OPMODE_TYPEC1_3 4
+#define UCSI_CONSTAT_PWR_OPMODE_TYPEC1_5 4
#define UCSI_CONSTAT_PWR_OPMODE_TYPEC3_0 5
u16 connected:1;
u16 pwr_dir:1;
u16 partner_type:3;
#define UCSI_CONSTAT_PARTNER_TYPE_DFP 1
#define UCSI_CONSTAT_PARTNER_TYPE_UFP 2
-#define UCSI_CONSTAT_PARTNER_TYPE_CABLE_NO_UFP 3 /* Powered Cable */
+#define UCSI_CONSTAT_PARTNER_TYPE_CABLE 3 /* Powered Cable */
#define UCSI_CONSTAT_PARTNER_TYPE_CABLE_AND_UFP 4 /* Powered Cable */
#define UCSI_CONSTAT_PARTNER_TYPE_DEBUG 5
#define UCSI_CONSTAT_PARTNER_TYPE_AUDIO 6
u8 provider_cap_limit_reason:4;
#define UCSI_CONSTAT_CAP_PWR_LOWERED 0
#define UCSI_CONSTAT_CAP_PWR_BUDGET_LIMIT 1
- u8 RESERVED:2;
+ u8:2; /* reserved */
} __packed;
/* -------------------------------------------------------------------------- */
+struct ucsi;
+
+struct ucsi_data {
+ u16 version;
+ u16 reserved;
+ union {
+ u32 raw_cci;
+ struct ucsi_cci cci;
+ };
+ struct ucsi_control ctrl;
+ u32 message_in[4];
+ u32 message_out[4];
+} __packed;
+
+/*
+ * struct ucsi_ppm - Interface to UCSI Platform Policy Manager
+ * @data: memory location to the UCSI data structures
+ * @cmd: UCSI command execution routine
+ * @sync: Refresh UCSI mailbox (the data structures)
+ */
+struct ucsi_ppm {
+ struct ucsi_data *data;
+ int (*cmd)(struct ucsi_ppm *, struct ucsi_control *);
+ int (*sync)(struct ucsi_ppm *);
+};
+
+struct ucsi *ucsi_register_ppm(struct device *dev, struct ucsi_ppm *ppm);
+void ucsi_unregister_ppm(struct ucsi *ucsi);
+void ucsi_notify(struct ucsi *ucsi);
+
+#endif /* __DRIVER_USB_TYPEC_UCSI_H */
--- /dev/null
+/*
+ * UCSI ACPI driver
+ *
+ * Copyright (C) 2017, Intel Corporation
+ * Author: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/platform_device.h>
+#include <linux/module.h>
+#include <linux/acpi.h>
+
+#include "ucsi.h"
+
+#define UCSI_DSM_UUID "6f8398c2-7ca4-11e4-ad36-631042b5008f"
+#define UCSI_DSM_FUNC_WRITE 1
+#define UCSI_DSM_FUNC_READ 2
+
+struct ucsi_acpi {
+ struct device *dev;
+ struct ucsi *ucsi;
+ struct ucsi_ppm ppm;
+ guid_t guid;
+};
+
+static int ucsi_acpi_dsm(struct ucsi_acpi *ua, int func)
+{
+ union acpi_object *obj;
+
+ obj = acpi_evaluate_dsm(ACPI_HANDLE(ua->dev), &ua->guid, 1, func,
+ NULL);
+ if (!obj) {
+ dev_err(ua->dev, "%s: failed to evaluate _DSM %d\n",
+ __func__, func);
+ return -EIO;
+ }
+
+ ACPI_FREE(obj);
+ return 0;
+}
+
+static int ucsi_acpi_cmd(struct ucsi_ppm *ppm, struct ucsi_control *ctrl)
+{
+ struct ucsi_acpi *ua = container_of(ppm, struct ucsi_acpi, ppm);
+
+ ppm->data->ctrl.raw_cmd = ctrl->raw_cmd;
+
+ return ucsi_acpi_dsm(ua, UCSI_DSM_FUNC_WRITE);
+}
+
+static int ucsi_acpi_sync(struct ucsi_ppm *ppm)
+{
+ struct ucsi_acpi *ua = container_of(ppm, struct ucsi_acpi, ppm);
+
+ return ucsi_acpi_dsm(ua, UCSI_DSM_FUNC_READ);
+}
+
+static void ucsi_acpi_notify(acpi_handle handle, u32 event, void *data)
+{
+ struct ucsi_acpi *ua = data;
+
+ ucsi_notify(ua->ucsi);
+}
+
+static int ucsi_acpi_probe(struct platform_device *pdev)
+{
+ struct ucsi_acpi *ua;
+ struct resource *res;
+ acpi_status status;
+ int ret;
+
+ ua = devm_kzalloc(&pdev->dev, sizeof(*ua), GFP_KERNEL);
+ if (!ua)
+ return -ENOMEM;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!res) {
+ dev_err(&pdev->dev, "missing memory resource\n");
+ return -ENODEV;
+ }
+
+ /*
+ * NOTE: The memory region for the data structures is used also in an
+ * operation region, which means ACPI has already reserved it. Therefore
+ * it can not be requested here, and we can not use
+ * devm_ioremap_resource().
+ */
+ ua->ppm.data = devm_ioremap(&pdev->dev, res->start, resource_size(res));
+ if (!ua->ppm.data)
+ return -ENOMEM;
+
+ if (!ua->ppm.data->version)
+ return -ENODEV;
+
+ ret = guid_parse(UCSI_DSM_UUID, &ua->guid);
+ if (ret)
+ return ret;
+
+ ua->ppm.cmd = ucsi_acpi_cmd;
+ ua->ppm.sync = ucsi_acpi_sync;
+ ua->dev = &pdev->dev;
+
+ status = acpi_install_notify_handler(ACPI_HANDLE(&pdev->dev),
+ ACPI_DEVICE_NOTIFY,
+ ucsi_acpi_notify, ua);
+ if (ACPI_FAILURE(status)) {
+ dev_err(&pdev->dev, "failed to install notify handler\n");
+ return -ENODEV;
+ }
+
+ ua->ucsi = ucsi_register_ppm(&pdev->dev, &ua->ppm);
+ if (IS_ERR(ua->ucsi)) {
+ acpi_remove_notify_handler(ACPI_HANDLE(&pdev->dev),
+ ACPI_DEVICE_NOTIFY,
+ ucsi_acpi_notify);
+ return PTR_ERR(ua->ucsi);
+ }
+
+ platform_set_drvdata(pdev, ua);
+
+ return 0;
+}
+
+static int ucsi_acpi_remove(struct platform_device *pdev)
+{
+ struct ucsi_acpi *ua = platform_get_drvdata(pdev);
+
+ ucsi_unregister_ppm(ua->ucsi);
+
+ acpi_remove_notify_handler(ACPI_HANDLE(&pdev->dev), ACPI_DEVICE_NOTIFY,
+ ucsi_acpi_notify);
+
+ return 0;
+}
+
+static const struct acpi_device_id ucsi_acpi_match[] = {
+ { "PNP0CA0", 0 },
+ { },
+};
+MODULE_DEVICE_TABLE(acpi, ucsi_acpi_match);
+
+static struct platform_driver ucsi_acpi_platform_driver = {
+ .driver = {
+ .name = "ucsi_acpi",
+ .acpi_match_table = ACPI_PTR(ucsi_acpi_match),
+ },
+ .probe = ucsi_acpi_probe,
+ .remove = ucsi_acpi_remove,
+};
+
+module_platform_driver(ucsi_acpi_platform_driver);
+
+MODULE_AUTHOR("Heikki Krogerus <heikki.krogerus@linux.intel.com>");
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("UCSI ACPI driver");
return ret;
}
-static ssize_t show_match_busid(struct device_driver *drv, char *buf)
+static ssize_t match_busid_show(struct device_driver *drv, char *buf)
{
int i;
char *out = buf;
return out - buf;
}
-static ssize_t store_match_busid(struct device_driver *dev, const char *buf,
+static ssize_t match_busid_store(struct device_driver *dev, const char *buf,
size_t count)
{
int len;
return -EINVAL;
}
-static DRIVER_ATTR(match_busid, S_IRUSR | S_IWUSR, show_match_busid,
- store_match_busid);
+static DRIVER_ATTR_RW(match_busid);
static ssize_t rebind_store(struct device_driver *dev, const char *buf,
size_t count)
kmem_cache_free(stub_priv_cache, priv);
kfree(urb->transfer_buffer);
+ urb->transfer_buffer = NULL;
+
kfree(urb->setup_packet);
+ urb->setup_packet = NULL;
+
usb_free_urb(urb);
}
}
struct urb *urb = priv->urb;
kfree(urb->setup_packet);
+ urb->setup_packet = NULL;
+
kfree(urb->transfer_buffer);
+ urb->transfer_buffer = NULL;
+
list_del(&priv->list);
kmem_cache_free(stub_priv_cache, priv);
usb_free_urb(urb);
unsigned long unlink_seqnum;
};
+enum hub_speed {
+ HUB_SPEED_HIGH = 0,
+ HUB_SPEED_SUPER,
+};
+
/* Number of supported ports. Value has an upperbound of USB_MAXCHILDREN */
#ifdef CONFIG_USBIP_VHCI_HC_PORTS
#define VHCI_HC_PORTS CONFIG_USBIP_VHCI_HC_PORTS
#define VHCI_HC_PORTS 8
#endif
+/* Each VHCI has 2 hubs (USB2 and USB3), each has VHCI_HC_PORTS ports */
+#define VHCI_PORTS (VHCI_HC_PORTS*2)
+
#ifdef CONFIG_USBIP_VHCI_NR_HCS
#define VHCI_NR_HCS CONFIG_USBIP_VHCI_NR_HCS
#else
#define MAX_STATUS_NAME 16
-/* for usb_bus.hcpriv */
-struct vhci_hcd {
+struct vhci {
spinlock_t lock;
+ struct platform_device *pdev;
+
+ struct vhci_hcd *vhci_hcd_hs;
+ struct vhci_hcd *vhci_hcd_ss;
+};
+
+/* for usb_hcd.hcd_priv[0] */
+struct vhci_hcd {
+ struct vhci *vhci;
+
u32 port_status[VHCI_HC_PORTS];
unsigned resuming:1;
};
extern int vhci_num_controllers;
-extern struct platform_device **vhci_pdevs;
+extern struct vhci *vhcis;
extern struct attribute_group vhci_attr_group;
/* vhci_hcd.c */
static inline int port_to_pdev_nr(__u32 port)
{
- return port / VHCI_HC_PORTS;
+ return port / VHCI_PORTS;
}
-static inline struct vhci_hcd *hcd_to_vhci(struct usb_hcd *hcd)
+static inline struct vhci_hcd *hcd_to_vhci_hcd(struct usb_hcd *hcd)
{
return (struct vhci_hcd *) (hcd->hcd_priv);
}
return (hcd)->self.bus_name;
}
-static inline struct usb_hcd *vhci_to_hcd(struct vhci_hcd *vhci)
+static inline struct usb_hcd *vhci_hcd_to_hcd(struct vhci_hcd *vhci_hcd)
{
- return container_of((void *) vhci, struct usb_hcd, hcd_priv);
+ return container_of((void *) vhci_hcd, struct usb_hcd, hcd_priv);
}
-static inline struct vhci_hcd *vdev_to_vhci(struct vhci_device *vdev)
+static inline struct vhci_hcd *vdev_to_vhci_hcd(struct vhci_device *vdev)
{
- return container_of(
- (void *)(vdev - vdev->rhport), struct vhci_hcd, vdev);
+ return container_of((void *)(vdev - vdev->rhport), struct vhci_hcd, vdev);
}
#endif /* __USBIP_VHCI_H */
static const char driver_desc[] = "USB/IP Virtual Host Controller";
int vhci_num_controllers = VHCI_NR_HCS;
-
-struct platform_device **vhci_pdevs;
+struct vhci *vhcis;
static const char * const bit_desc[] = {
"CONNECTION", /*0*/
"SUSPEND", /*2*/
"OVER_CURRENT", /*3*/
"RESET", /*4*/
- "R5", /*5*/
+ "L1", /*5*/
"R6", /*6*/
"R7", /*7*/
"POWER", /*8*/
"C_SUSPEND", /*18*/
"C_OVER_CURRENT", /*19*/
"C_RESET", /*20*/
- "R21", /*21*/
+ "C_L1", /*21*/
"R22", /*22*/
"R23", /*23*/
"R24", /*24*/
"R31", /*31*/
};
-static void dump_port_status_diff(u32 prev_status, u32 new_status)
+static const char * const bit_desc_ss[] = {
+ "CONNECTION", /*0*/
+ "ENABLE", /*1*/
+ "SUSPEND", /*2*/
+ "OVER_CURRENT", /*3*/
+ "RESET", /*4*/
+ "L1", /*5*/
+ "R6", /*6*/
+ "R7", /*7*/
+ "R8", /*8*/
+ "POWER", /*9*/
+ "HIGHSPEED", /*10*/
+ "PORT_TEST", /*11*/
+ "INDICATOR", /*12*/
+ "R13", /*13*/
+ "R14", /*14*/
+ "R15", /*15*/
+ "C_CONNECTION", /*16*/
+ "C_ENABLE", /*17*/
+ "C_SUSPEND", /*18*/
+ "C_OVER_CURRENT", /*19*/
+ "C_RESET", /*20*/
+ "C_BH_RESET", /*21*/
+ "C_LINK_STATE", /*22*/
+ "C_CONFIG_ERROR", /*23*/
+ "R24", /*24*/
+ "R25", /*25*/
+ "R26", /*26*/
+ "R27", /*27*/
+ "R28", /*28*/
+ "R29", /*29*/
+ "R30", /*30*/
+ "R31", /*31*/
+};
+
+static void dump_port_status_diff(u32 prev_status, u32 new_status, bool usb3)
{
int i = 0;
u32 bit = 1;
+ const char * const *desc = bit_desc;
+
+ if (usb3)
+ desc = bit_desc_ss;
pr_debug("status prev -> new: %08x -> %08x\n", prev_status, new_status);
while (bit) {
else
change = ' ';
- if (prev || new)
- pr_debug(" %c%s\n", change, bit_desc[i]);
+ if (prev || new) {
+ pr_debug(" %c%s\n", change, desc[i]);
+
+ if (bit == 1) /* USB_PORT_STAT_CONNECTION */
+ pr_debug(" %c%s\n", change, "USB_PORT_STAT_SPEED_5GBPS");
+ }
bit <<= 1;
i++;
}
void rh_port_connect(struct vhci_device *vdev, enum usb_device_speed speed)
{
- struct vhci_hcd *vhci = vdev_to_vhci(vdev);
+ struct vhci_hcd *vhci_hcd = vdev_to_vhci_hcd(vdev);
+ struct vhci *vhci = vhci_hcd->vhci;
int rhport = vdev->rhport;
u32 status;
unsigned long flags;
spin_lock_irqsave(&vhci->lock, flags);
- status = vhci->port_status[rhport];
+ status = vhci_hcd->port_status[rhport];
status |= USB_PORT_STAT_CONNECTION | (1 << USB_PORT_FEAT_C_CONNECTION);
break;
}
- vhci->port_status[rhport] = status;
+ vhci_hcd->port_status[rhport] = status;
spin_unlock_irqrestore(&vhci->lock, flags);
- usb_hcd_poll_rh_status(vhci_to_hcd(vhci));
+ usb_hcd_poll_rh_status(vhci_hcd_to_hcd(vhci_hcd));
}
static void rh_port_disconnect(struct vhci_device *vdev)
{
- struct vhci_hcd *vhci = vdev_to_vhci(vdev);
+ struct vhci_hcd *vhci_hcd = vdev_to_vhci_hcd(vdev);
+ struct vhci *vhci = vhci_hcd->vhci;
int rhport = vdev->rhport;
u32 status;
unsigned long flags;
spin_lock_irqsave(&vhci->lock, flags);
- status = vhci->port_status[rhport];
+ status = vhci_hcd->port_status[rhport];
status &= ~USB_PORT_STAT_CONNECTION;
status |= (1 << USB_PORT_FEAT_C_CONNECTION);
- vhci->port_status[rhport] = status;
+ vhci_hcd->port_status[rhport] = status;
spin_unlock_irqrestore(&vhci->lock, flags);
- usb_hcd_poll_rh_status(vhci_to_hcd(vhci));
+ usb_hcd_poll_rh_status(vhci_hcd_to_hcd(vhci_hcd));
}
#define PORT_C_MASK \
*/
static int vhci_hub_status(struct usb_hcd *hcd, char *buf)
{
- struct vhci_hcd *vhci;
- int retval;
+ struct vhci_hcd *vhci_hcd = hcd_to_vhci_hcd(hcd);
+ struct vhci *vhci = vhci_hcd->vhci;
+ int retval = DIV_ROUND_UP(VHCI_HC_PORTS + 1, 8);
int rhport;
int changed = 0;
unsigned long flags;
- retval = DIV_ROUND_UP(VHCI_HC_PORTS + 1, 8);
memset(buf, 0, retval);
- vhci = hcd_to_vhci(hcd);
-
spin_lock_irqsave(&vhci->lock, flags);
if (!HCD_HW_ACCESSIBLE(hcd)) {
usbip_dbg_vhci_rh("hw accessible flag not on?\n");
/* check pseudo status register for each port */
for (rhport = 0; rhport < VHCI_HC_PORTS; rhport++) {
- if ((vhci->port_status[rhport] & PORT_C_MASK)) {
+ if ((vhci_hcd->port_status[rhport] & PORT_C_MASK)) {
/* The status of a port has been changed, */
usbip_dbg_vhci_rh("port %d status changed\n", rhport);
return changed ? retval : 0;
}
+/* usb 3.0 root hub device descriptor */
+static struct {
+ struct usb_bos_descriptor bos;
+ struct usb_ss_cap_descriptor ss_cap;
+} __packed usb3_bos_desc = {
+
+ .bos = {
+ .bLength = USB_DT_BOS_SIZE,
+ .bDescriptorType = USB_DT_BOS,
+ .wTotalLength = cpu_to_le16(sizeof(usb3_bos_desc)),
+ .bNumDeviceCaps = 1,
+ },
+ .ss_cap = {
+ .bLength = USB_DT_USB_SS_CAP_SIZE,
+ .bDescriptorType = USB_DT_DEVICE_CAPABILITY,
+ .bDevCapabilityType = USB_SS_CAP_TYPE,
+ .wSpeedSupported = cpu_to_le16(USB_5GBPS_OPERATION),
+ .bFunctionalitySupport = ilog2(USB_5GBPS_OPERATION),
+ },
+};
+
+static inline void
+ss_hub_descriptor(struct usb_hub_descriptor *desc)
+{
+ memset(desc, 0, sizeof *desc);
+ desc->bDescriptorType = USB_DT_SS_HUB;
+ desc->bDescLength = 12;
+ desc->wHubCharacteristics = cpu_to_le16(
+ HUB_CHAR_INDV_PORT_LPSM | HUB_CHAR_COMMON_OCPM);
+ desc->bNbrPorts = VHCI_HC_PORTS;
+ desc->u.ss.bHubHdrDecLat = 0x04; /* Worst case: 0.4 micro sec*/
+ desc->u.ss.DeviceRemovable = 0xffff;
+}
+
static inline void hub_descriptor(struct usb_hub_descriptor *desc)
{
int width;
static int vhci_hub_control(struct usb_hcd *hcd, u16 typeReq, u16 wValue,
u16 wIndex, char *buf, u16 wLength)
{
- struct vhci_hcd *dum;
+ struct vhci_hcd *vhci_hcd;
+ struct vhci *vhci;
int retval = 0;
int rhport;
unsigned long flags;
/*
* NOTE:
- * wIndex shows the port number and begins from 1.
+ * wIndex (bits 0-7) shows the port number and begins from 1?
*/
+ wIndex = ((__u8)(wIndex & 0x00ff));
usbip_dbg_vhci_rh("typeReq %x wValue %x wIndex %x\n", typeReq, wValue,
wIndex);
+
if (wIndex > VHCI_HC_PORTS)
pr_err("invalid port number %d\n", wIndex);
- rhport = ((__u8)(wIndex & 0x00ff)) - 1;
+ rhport = wIndex - 1;
- dum = hcd_to_vhci(hcd);
+ vhci_hcd = hcd_to_vhci_hcd(hcd);
+ vhci = vhci_hcd->vhci;
- spin_lock_irqsave(&dum->lock, flags);
+ spin_lock_irqsave(&vhci->lock, flags);
/* store old status and compare now and old later */
if (usbip_dbg_flag_vhci_rh) {
- memcpy(prev_port_status, dum->port_status,
+ memcpy(prev_port_status, vhci_hcd->port_status,
sizeof(prev_port_status));
}
case ClearPortFeature:
switch (wValue) {
case USB_PORT_FEAT_SUSPEND:
- if (dum->port_status[rhport] & USB_PORT_STAT_SUSPEND) {
+ if (hcd->speed == HCD_USB3) {
+ pr_err(" ClearPortFeature: USB_PORT_FEAT_SUSPEND req not "
+ "supported for USB 3.0 roothub\n");
+ goto error;
+ }
+ usbip_dbg_vhci_rh(
+ " ClearPortFeature: USB_PORT_FEAT_SUSPEND\n");
+ if (vhci_hcd->port_status[rhport] & USB_PORT_STAT_SUSPEND) {
/* 20msec signaling */
- dum->resuming = 1;
- dum->re_timeout =
- jiffies + msecs_to_jiffies(20);
+ vhci_hcd->resuming = 1;
+ vhci_hcd->re_timeout = jiffies + msecs_to_jiffies(20);
}
break;
case USB_PORT_FEAT_POWER:
usbip_dbg_vhci_rh(
" ClearPortFeature: USB_PORT_FEAT_POWER\n");
- dum->port_status[rhport] = 0;
- dum->resuming = 0;
- break;
- case USB_PORT_FEAT_C_RESET:
- usbip_dbg_vhci_rh(
- " ClearPortFeature: USB_PORT_FEAT_C_RESET\n");
- switch (dum->vdev[rhport].speed) {
- case USB_SPEED_HIGH:
- dum->port_status[rhport] |=
- USB_PORT_STAT_HIGH_SPEED;
- break;
- case USB_SPEED_LOW:
- dum->port_status[rhport] |=
- USB_PORT_STAT_LOW_SPEED;
- break;
- default:
- break;
- }
+ if (hcd->speed == HCD_USB3)
+ vhci_hcd->port_status[rhport] &= ~USB_SS_PORT_STAT_POWER;
+ else
+ vhci_hcd->port_status[rhport] &= ~USB_PORT_STAT_POWER;
break;
default:
usbip_dbg_vhci_rh(" ClearPortFeature: default %x\n",
wValue);
- dum->port_status[rhport] &= ~(1 << wValue);
+ vhci_hcd->port_status[rhport] &= ~(1 << wValue);
break;
}
break;
case GetHubDescriptor:
usbip_dbg_vhci_rh(" GetHubDescriptor\n");
- hub_descriptor((struct usb_hub_descriptor *) buf);
+ if (hcd->speed == HCD_USB3 &&
+ (wLength < USB_DT_SS_HUB_SIZE ||
+ wValue != (USB_DT_SS_HUB << 8))) {
+ pr_err("Wrong hub descriptor type for USB 3.0 roothub.\n");
+ goto error;
+ }
+ if (hcd->speed == HCD_USB3)
+ ss_hub_descriptor((struct usb_hub_descriptor *) buf);
+ else
+ hub_descriptor((struct usb_hub_descriptor *) buf);
+ break;
+ case DeviceRequest | USB_REQ_GET_DESCRIPTOR:
+ if (hcd->speed != HCD_USB3)
+ goto error;
+
+ if ((wValue >> 8) != USB_DT_BOS)
+ goto error;
+
+ memcpy(buf, &usb3_bos_desc, sizeof(usb3_bos_desc));
+ retval = sizeof(usb3_bos_desc);
break;
case GetHubStatus:
usbip_dbg_vhci_rh(" GetHubStatus\n");
break;
case GetPortStatus:
usbip_dbg_vhci_rh(" GetPortStatus port %x\n", wIndex);
- if (wIndex > VHCI_HC_PORTS || wIndex < 1) {
+ if (wIndex < 1) {
pr_err("invalid port number %d\n", wIndex);
retval = -EPIPE;
}
/* whoever resets or resumes must GetPortStatus to
* complete it!!
*/
- if (dum->resuming && time_after(jiffies, dum->re_timeout)) {
- dum->port_status[rhport] |=
- (1 << USB_PORT_FEAT_C_SUSPEND);
- dum->port_status[rhport] &=
- ~(1 << USB_PORT_FEAT_SUSPEND);
- dum->resuming = 0;
- dum->re_timeout = 0;
+ if (vhci_hcd->resuming && time_after(jiffies, vhci_hcd->re_timeout)) {
+ vhci_hcd->port_status[rhport] |= (1 << USB_PORT_FEAT_C_SUSPEND);
+ vhci_hcd->port_status[rhport] &= ~(1 << USB_PORT_FEAT_SUSPEND);
+ vhci_hcd->resuming = 0;
+ vhci_hcd->re_timeout = 0;
}
- if ((dum->port_status[rhport] & (1 << USB_PORT_FEAT_RESET)) !=
- 0 && time_after(jiffies, dum->re_timeout)) {
- dum->port_status[rhport] |=
- (1 << USB_PORT_FEAT_C_RESET);
- dum->port_status[rhport] &=
- ~(1 << USB_PORT_FEAT_RESET);
- dum->re_timeout = 0;
+ if ((vhci_hcd->port_status[rhport] & (1 << USB_PORT_FEAT_RESET)) !=
+ 0 && time_after(jiffies, vhci_hcd->re_timeout)) {
+ vhci_hcd->port_status[rhport] |= (1 << USB_PORT_FEAT_C_RESET);
+ vhci_hcd->port_status[rhport] &= ~(1 << USB_PORT_FEAT_RESET);
+ vhci_hcd->re_timeout = 0;
- if (dum->vdev[rhport].ud.status ==
+ if (vhci_hcd->vdev[rhport].ud.status ==
VDEV_ST_NOTASSIGNED) {
usbip_dbg_vhci_rh(
" enable rhport %d (status %u)\n",
rhport,
- dum->vdev[rhport].ud.status);
- dum->port_status[rhport] |=
+ vhci_hcd->vdev[rhport].ud.status);
+ vhci_hcd->port_status[rhport] |=
USB_PORT_STAT_ENABLE;
}
+
+ if (hcd->speed < HCD_USB3) {
+ switch (vhci_hcd->vdev[rhport].speed) {
+ case USB_SPEED_HIGH:
+ vhci_hcd->port_status[rhport] |=
+ USB_PORT_STAT_HIGH_SPEED;
+ break;
+ case USB_SPEED_LOW:
+ vhci_hcd->port_status[rhport] |=
+ USB_PORT_STAT_LOW_SPEED;
+ break;
+ default:
+ pr_err("vhci_device speed not set\n");
+ break;
+ }
+ }
}
- ((__le16 *) buf)[0] = cpu_to_le16(dum->port_status[rhport]);
+ ((__le16 *) buf)[0] = cpu_to_le16(vhci_hcd->port_status[rhport]);
((__le16 *) buf)[1] =
- cpu_to_le16(dum->port_status[rhport] >> 16);
+ cpu_to_le16(vhci_hcd->port_status[rhport] >> 16);
usbip_dbg_vhci_rh(" GetPortStatus bye %x %x\n", ((u16 *)buf)[0],
((u16 *)buf)[1]);
break;
case SetPortFeature:
switch (wValue) {
+ case USB_PORT_FEAT_LINK_STATE:
+ usbip_dbg_vhci_rh(
+ " SetPortFeature: USB_PORT_FEAT_LINK_STATE\n");
+ if (hcd->speed != HCD_USB3) {
+ pr_err("USB_PORT_FEAT_LINK_STATE req not "
+ "supported for USB 2.0 roothub\n");
+ goto error;
+ }
+ /*
+ * Since this is dummy we don't have an actual link so
+ * there is nothing to do for the SET_LINK_STATE cmd
+ */
+ break;
+ case USB_PORT_FEAT_U1_TIMEOUT:
+ usbip_dbg_vhci_rh(
+ " SetPortFeature: USB_PORT_FEAT_U1_TIMEOUT\n");
+ case USB_PORT_FEAT_U2_TIMEOUT:
+ usbip_dbg_vhci_rh(
+ " SetPortFeature: USB_PORT_FEAT_U2_TIMEOUT\n");
+ /* TODO: add suspend/resume support! */
+ if (hcd->speed != HCD_USB3) {
+ pr_err("USB_PORT_FEAT_U1/2_TIMEOUT req not "
+ "supported for USB 2.0 roothub\n");
+ goto error;
+ }
+ break;
case USB_PORT_FEAT_SUSPEND:
usbip_dbg_vhci_rh(
" SetPortFeature: USB_PORT_FEAT_SUSPEND\n");
+ /* Applicable only for USB2.0 hub */
+ if (hcd->speed == HCD_USB3) {
+ pr_err("USB_PORT_FEAT_SUSPEND req not "
+ "supported for USB 3.0 roothub\n");
+ goto error;
+ }
+
+ vhci_hcd->port_status[rhport] |= USB_PORT_STAT_SUSPEND;
break;
+ case USB_PORT_FEAT_POWER:
+ usbip_dbg_vhci_rh(
+ " SetPortFeature: USB_PORT_FEAT_POWER\n");
+ if (hcd->speed == HCD_USB3)
+ vhci_hcd->port_status[rhport] |= USB_SS_PORT_STAT_POWER;
+ else
+ vhci_hcd->port_status[rhport] |= USB_PORT_STAT_POWER;
+ break;
+ case USB_PORT_FEAT_BH_PORT_RESET:
+ usbip_dbg_vhci_rh(
+ " SetPortFeature: USB_PORT_FEAT_BH_PORT_RESET\n");
+ /* Applicable only for USB3.0 hub */
+ if (hcd->speed != HCD_USB3) {
+ pr_err("USB_PORT_FEAT_BH_PORT_RESET req not "
+ "supported for USB 2.0 roothub\n");
+ goto error;
+ }
+ /* FALLS THROUGH */
case USB_PORT_FEAT_RESET:
usbip_dbg_vhci_rh(
" SetPortFeature: USB_PORT_FEAT_RESET\n");
- /* if it's already running, disconnect first */
- if (dum->port_status[rhport] & USB_PORT_STAT_ENABLE) {
- dum->port_status[rhport] &=
- ~(USB_PORT_STAT_ENABLE |
- USB_PORT_STAT_LOW_SPEED |
- USB_PORT_STAT_HIGH_SPEED);
- /* FIXME test that code path! */
+ /* if it's already enabled, disable */
+ if (hcd->speed == HCD_USB3) {
+ vhci_hcd->port_status[rhport] = 0;
+ vhci_hcd->port_status[rhport] =
+ (USB_SS_PORT_STAT_POWER |
+ USB_PORT_STAT_CONNECTION |
+ USB_PORT_STAT_RESET);
+ } else if (vhci_hcd->port_status[rhport] & USB_PORT_STAT_ENABLE) {
+ vhci_hcd->port_status[rhport] &= ~(USB_PORT_STAT_ENABLE
+ | USB_PORT_STAT_LOW_SPEED
+ | USB_PORT_STAT_HIGH_SPEED);
}
+
/* 50msec reset signaling */
- dum->re_timeout = jiffies + msecs_to_jiffies(50);
+ vhci_hcd->re_timeout = jiffies + msecs_to_jiffies(50);
- /* FALLTHROUGH */
+ /* FALLS THROUGH */
default:
usbip_dbg_vhci_rh(" SetPortFeature: default %d\n",
wValue);
- dum->port_status[rhport] |= (1 << wValue);
- break;
+ if (hcd->speed == HCD_USB3) {
+ if ((vhci_hcd->port_status[rhport] &
+ USB_SS_PORT_STAT_POWER) != 0) {
+ vhci_hcd->port_status[rhport] |= (1 << wValue);
+ }
+ } else
+ if ((vhci_hcd->port_status[rhport] &
+ USB_PORT_STAT_POWER) != 0) {
+ vhci_hcd->port_status[rhport] |= (1 << wValue);
+ }
+ }
+ break;
+ case GetPortErrorCount:
+ usbip_dbg_vhci_rh(" GetPortErrorCount\n");
+ if (hcd->speed != HCD_USB3) {
+ pr_err("GetPortErrorCount req not "
+ "supported for USB 2.0 roothub\n");
+ goto error;
+ }
+ /* We'll always return 0 since this is a dummy hub */
+ *(__le32 *) buf = cpu_to_le32(0);
+ break;
+ case SetHubDepth:
+ usbip_dbg_vhci_rh(" SetHubDepth\n");
+ if (hcd->speed != HCD_USB3) {
+ pr_err("SetHubDepth req not supported for "
+ "USB 2.0 roothub\n");
+ goto error;
}
break;
-
default:
- pr_err("default: no such request\n");
-
+ pr_err("default hub control req: %04x v%04x i%04x l%d\n",
+ typeReq, wValue, wIndex, wLength);
+error:
/* "protocol stall" on error */
retval = -EPIPE;
}
/* Only dump valid port status */
if (rhport >= 0) {
dump_port_status_diff(prev_port_status[rhport],
- dum->port_status[rhport]);
+ vhci_hcd->port_status[rhport],
+ hcd->speed == HCD_USB3);
}
}
usbip_dbg_vhci_rh(" bye\n");
- spin_unlock_irqrestore(&dum->lock, flags);
+ spin_unlock_irqrestore(&vhci->lock, flags);
+
+ if ((vhci_hcd->port_status[rhport] & PORT_C_MASK) != 0)
+ usb_hcd_poll_rh_status(hcd);
return retval;
}
static void vhci_tx_urb(struct urb *urb, struct vhci_device *vdev)
{
struct vhci_priv *priv;
- struct vhci_hcd *vhci;
+ struct vhci_hcd *vhci_hcd;
unsigned long flags;
if (!vdev) {
pr_err("could not get virtual device");
return;
}
- vhci = vdev_to_vhci(vdev);
+ vhci_hcd = vdev_to_vhci_hcd(vdev);
priv = kzalloc(sizeof(struct vhci_priv), GFP_ATOMIC);
if (!priv) {
spin_lock_irqsave(&vdev->priv_lock, flags);
- priv->seqnum = atomic_inc_return(&vhci->seqnum);
+ priv->seqnum = atomic_inc_return(&vhci_hcd->seqnum);
if (priv->seqnum == 0xffff)
dev_info(&urb->dev->dev, "seqnum max\n");
spin_unlock_irqrestore(&vdev->priv_lock, flags);
}
-static int vhci_urb_enqueue(struct usb_hcd *hcd, struct urb *urb,
- gfp_t mem_flags)
+static int vhci_urb_enqueue(struct usb_hcd *hcd, struct urb *urb, gfp_t mem_flags)
{
- struct vhci_hcd *vhci = hcd_to_vhci(hcd);
+ struct vhci_hcd *vhci_hcd = hcd_to_vhci_hcd(hcd);
+ struct vhci *vhci = vhci_hcd->vhci;
struct device *dev = &urb->dev->dev;
u8 portnum = urb->dev->portnum;
int ret = 0;
pr_err("invalid port number %d\n", portnum);
return -ENODEV;
}
- vdev = &vhci->vdev[portnum-1];
+ vdev = &vhci_hcd->vdev[portnum-1];
/* patch to usb_sg_init() is in 2.5.60 */
BUG_ON(!urb->transfer_buffer && urb->transfer_buffer_length);
*/
static int vhci_urb_dequeue(struct usb_hcd *hcd, struct urb *urb, int status)
{
- struct vhci_hcd *vhci = hcd_to_vhci(hcd);
+ struct vhci_hcd *vhci_hcd = hcd_to_vhci_hcd(hcd);
+ struct vhci *vhci = vhci_hcd->vhci;
struct vhci_priv *priv;
struct vhci_device *vdev;
unsigned long flags;
usb_hcd_unlink_urb_from_ep(hcd, urb);
spin_unlock_irqrestore(&vhci->lock, flags);
- usb_hcd_giveback_urb(vhci_to_hcd(vhci), urb, urb->status);
+ usb_hcd_giveback_urb(hcd, urb, urb->status);
spin_lock_irqsave(&vhci->lock, flags);
} else {
return -ENOMEM;
}
- unlink->seqnum = atomic_inc_return(&vhci->seqnum);
+ unlink->seqnum = atomic_inc_return(&vhci_hcd->seqnum);
if (unlink->seqnum == 0xffff)
pr_info("seqnum max\n");
static void vhci_device_unlink_cleanup(struct vhci_device *vdev)
{
- struct vhci_hcd *vhci = vdev_to_vhci(vdev);
- struct usb_hcd *hcd = vhci_to_hcd(vhci);
+ struct vhci_hcd *vhci_hcd = vdev_to_vhci_hcd(vdev);
+ struct usb_hcd *hcd = vhci_hcd_to_hcd(vhci_hcd);
+ struct vhci *vhci = vhci_hcd->vhci;
struct vhci_unlink *unlink, *tmp;
unsigned long flags;
pr_info("disconnect device\n");
}
-
static void vhci_device_reset(struct usbip_device *ud)
{
struct vhci_device *vdev = container_of(ud, struct vhci_device, ud);
return val;
}
+static int vhci_setup(struct usb_hcd *hcd)
+{
+ struct vhci *vhci = *((void **)dev_get_platdata(hcd->self.controller));
+ hcd->self.sg_tablesize = ~0;
+ if (usb_hcd_is_primary_hcd(hcd)) {
+ vhci->vhci_hcd_hs = hcd_to_vhci_hcd(hcd);
+ vhci->vhci_hcd_hs->vhci = vhci;
+ /*
+ * Mark the first roothub as being USB 2.0.
+ * The USB 3.0 roothub will be registered later by
+ * vhci_hcd_probe()
+ */
+ hcd->speed = HCD_USB2;
+ hcd->self.root_hub->speed = USB_SPEED_HIGH;
+ } else {
+ vhci->vhci_hcd_ss = hcd_to_vhci_hcd(hcd);
+ vhci->vhci_hcd_ss->vhci = vhci;
+ hcd->speed = HCD_USB3;
+ hcd->self.root_hub->speed = USB_SPEED_SUPER;
+ }
+ return 0;
+}
+
static int vhci_start(struct usb_hcd *hcd)
{
- struct vhci_hcd *vhci = hcd_to_vhci(hcd);
+ struct vhci_hcd *vhci_hcd = hcd_to_vhci_hcd(hcd);
int id, rhport;
- int err = 0;
+ int err;
usbip_dbg_vhci_hc("enter vhci_start\n");
+ if (usb_hcd_is_primary_hcd(hcd))
+ spin_lock_init(&vhci_hcd->vhci->lock);
+
/* initialize private data of usb_hcd */
for (rhport = 0; rhport < VHCI_HC_PORTS; rhport++) {
- struct vhci_device *vdev = &vhci->vdev[rhport];
+ struct vhci_device *vdev = &vhci_hcd->vdev[rhport];
vhci_device_init(vdev);
vdev->rhport = rhport;
}
- atomic_set(&vhci->seqnum, 0);
- spin_lock_init(&vhci->lock);
+ atomic_set(&vhci_hcd->seqnum, 0);
hcd->power_budget = 0; /* no limit */
hcd->uses_new_polling = 1;
+#ifdef CONFIG_USB_OTG
+ hcd->self.otg_port = 1;
+#endif
+
id = hcd_name_to_id(hcd_name(hcd));
if (id < 0) {
pr_err("invalid vhci name %s\n", hcd_name(hcd));
}
/* vhci_hcd is now ready to be controlled through sysfs */
- if (id == 0) {
+ if (id == 0 && usb_hcd_is_primary_hcd(hcd)) {
err = vhci_init_attr_group();
if (err) {
pr_err("init attr group\n");
static void vhci_stop(struct usb_hcd *hcd)
{
- struct vhci_hcd *vhci = hcd_to_vhci(hcd);
+ struct vhci_hcd *vhci_hcd = hcd_to_vhci_hcd(hcd);
int id, rhport;
usbip_dbg_vhci_hc("stop VHCI controller\n");
/* 1. remove the userland interface of vhci_hcd */
id = hcd_name_to_id(hcd_name(hcd));
- if (id == 0) {
+ if (id == 0 && usb_hcd_is_primary_hcd(hcd)) {
sysfs_remove_group(&hcd_dev(hcd)->kobj, &vhci_attr_group);
vhci_finish_attr_group();
}
/* 2. shutdown all the ports of vhci_hcd */
for (rhport = 0; rhport < VHCI_HC_PORTS; rhport++) {
- struct vhci_device *vdev = &vhci->vdev[rhport];
+ struct vhci_device *vdev = &vhci_hcd->vdev[rhport];
usbip_event_add(&vdev->ud, VDEV_EVENT_REMOVED);
usbip_stop_eh(&vdev->ud);
/* FIXME: suspend/resume */
static int vhci_bus_suspend(struct usb_hcd *hcd)
{
- struct vhci_hcd *vhci = hcd_to_vhci(hcd);
+ struct vhci *vhci = *((void **)dev_get_platdata(hcd->self.controller));
unsigned long flags;
dev_dbg(&hcd->self.root_hub->dev, "%s\n", __func__);
static int vhci_bus_resume(struct usb_hcd *hcd)
{
- struct vhci_hcd *vhci = hcd_to_vhci(hcd);
+ struct vhci *vhci = *((void **)dev_get_platdata(hcd->self.controller));
int rc = 0;
unsigned long flags;
#define vhci_bus_resume NULL
#endif
+/* Change a group of bulk endpoints to support multiple stream IDs */
+static int vhci_alloc_streams(struct usb_hcd *hcd, struct usb_device *udev,
+ struct usb_host_endpoint **eps, unsigned int num_eps,
+ unsigned int num_streams, gfp_t mem_flags)
+{
+ dev_dbg(&hcd->self.root_hub->dev, "vhci_alloc_streams not implemented\n");
+ return 0;
+}
+
+/* Reverts a group of bulk endpoints back to not using stream IDs. */
+static int vhci_free_streams(struct usb_hcd *hcd, struct usb_device *udev,
+ struct usb_host_endpoint **eps, unsigned int num_eps,
+ gfp_t mem_flags)
+{
+ dev_dbg(&hcd->self.root_hub->dev, "vhci_free_streams not implemented\n");
+ return 0;
+}
+
static struct hc_driver vhci_hc_driver = {
.description = driver_name,
.product_desc = driver_desc,
.hcd_priv_size = sizeof(struct vhci_hcd),
- .flags = HCD_USB2,
+ .flags = HCD_USB3 | HCD_SHARED,
+ .reset = vhci_setup,
.start = vhci_start,
.stop = vhci_stop,
.hub_control = vhci_hub_control,
.bus_suspend = vhci_bus_suspend,
.bus_resume = vhci_bus_resume,
+
+ .alloc_streams = vhci_alloc_streams,
+ .free_streams = vhci_free_streams,
};
static int vhci_hcd_probe(struct platform_device *pdev)
{
- struct usb_hcd *hcd;
+ struct vhci *vhci = *((void **)dev_get_platdata(&pdev->dev));
+ struct usb_hcd *hcd_hs;
+ struct usb_hcd *hcd_ss;
int ret;
usbip_dbg_vhci_hc("name %s id %d\n", pdev->name, pdev->id);
* Allocate and initialize hcd.
* Our private data is also allocated automatically.
*/
- hcd = usb_create_hcd(&vhci_hc_driver, &pdev->dev, dev_name(&pdev->dev));
- if (!hcd) {
- pr_err("create hcd failed\n");
+ hcd_hs = usb_create_hcd(&vhci_hc_driver, &pdev->dev, dev_name(&pdev->dev));
+ if (!hcd_hs) {
+ pr_err("create primary hcd failed\n");
return -ENOMEM;
}
- hcd->has_tt = 1;
+ hcd_hs->has_tt = 1;
/*
* Finish generic HCD structure initialization and register.
* Call the driver's reset() and start() routines.
*/
- ret = usb_add_hcd(hcd, 0, 0);
+ ret = usb_add_hcd(hcd_hs, 0, 0);
if (ret != 0) {
- pr_err("usb_add_hcd failed %d\n", ret);
- usb_put_hcd(hcd);
- return ret;
+ pr_err("usb_add_hcd hs failed %d\n", ret);
+ goto put_usb2_hcd;
+ }
+
+ hcd_ss = usb_create_shared_hcd(&vhci_hc_driver, &pdev->dev,
+ dev_name(&pdev->dev), hcd_hs);
+ if (!hcd_ss) {
+ ret = -ENOMEM;
+ pr_err("create shared hcd failed\n");
+ goto remove_usb2_hcd;
+ }
+
+ ret = usb_add_hcd(hcd_ss, 0, 0);
+ if (ret) {
+ pr_err("usb_add_hcd ss failed %d\n", ret);
+ goto put_usb3_hcd;
}
usbip_dbg_vhci_hc("bye\n");
return 0;
+
+put_usb3_hcd:
+ usb_put_hcd(hcd_ss);
+remove_usb2_hcd:
+ usb_remove_hcd(hcd_hs);
+put_usb2_hcd:
+ usb_put_hcd(hcd_hs);
+ vhci->vhci_hcd_hs = NULL;
+ vhci->vhci_hcd_ss = NULL;
+ return ret;
}
static int vhci_hcd_remove(struct platform_device *pdev)
{
- struct usb_hcd *hcd;
-
- hcd = platform_get_drvdata(pdev);
- if (!hcd)
- return 0;
+ struct vhci *vhci = *((void **)dev_get_platdata(&pdev->dev));
/*
* Disconnects the root hub,
* then reverses the effects of usb_add_hcd(),
* invoking the HCD's stop() methods.
*/
- usb_remove_hcd(hcd);
- usb_put_hcd(hcd);
+ usb_remove_hcd(vhci_hcd_to_hcd(vhci->vhci_hcd_ss));
+ usb_put_hcd(vhci_hcd_to_hcd(vhci->vhci_hcd_ss));
+
+ usb_remove_hcd(vhci_hcd_to_hcd(vhci->vhci_hcd_hs));
+ usb_put_hcd(vhci_hcd_to_hcd(vhci->vhci_hcd_hs));
+
+ vhci->vhci_hcd_hs = NULL;
+ vhci->vhci_hcd_ss = NULL;
return 0;
}
static int vhci_hcd_suspend(struct platform_device *pdev, pm_message_t state)
{
struct usb_hcd *hcd;
- struct vhci_hcd *vhci;
+ struct vhci *vhci;
int rhport;
int connected = 0;
int ret = 0;
unsigned long flags;
+ dev_dbg(&pdev->dev, "%s\n", __func__);
+
hcd = platform_get_drvdata(pdev);
if (!hcd)
return 0;
- vhci = hcd_to_vhci(hcd);
+
+ vhci = *((void **)dev_get_platdata(hcd->self.controller));
spin_lock_irqsave(&vhci->lock, flags);
- for (rhport = 0; rhport < VHCI_HC_PORTS; rhport++)
- if (vhci->port_status[rhport] & USB_PORT_STAT_CONNECTION)
+ for (rhport = 0; rhport < VHCI_HC_PORTS; rhport++) {
+ if (vhci->vhci_hcd_hs->port_status[rhport] &
+ USB_PORT_STAT_CONNECTION)
connected += 1;
+ if (vhci->vhci_hcd_ss->port_status[rhport] &
+ USB_PORT_STAT_CONNECTION)
+ connected += 1;
+ }
+
spin_unlock_irqrestore(&vhci->lock, flags);
if (connected > 0) {
},
};
-static int add_platform_device(int id)
-{
- struct platform_device *pdev;
- int dev_nr;
-
- if (id == 0)
- dev_nr = -1;
- else
- dev_nr = id;
-
- pdev = platform_device_register_simple(driver_name, dev_nr, NULL, 0);
- if (IS_ERR(pdev))
- return PTR_ERR(pdev);
-
- *(vhci_pdevs + id) = pdev;
- return 0;
-}
-
static void del_platform_devices(void)
{
struct platform_device *pdev;
int i;
for (i = 0; i < vhci_num_controllers; i++) {
- pdev = *(vhci_pdevs + i);
+ pdev = vhcis[i].pdev;
if (pdev != NULL)
platform_device_unregister(pdev);
- *(vhci_pdevs + i) = NULL;
+ vhcis[i].pdev = NULL;
}
sysfs_remove_link(&platform_bus.kobj, driver_name);
}
if (vhci_num_controllers < 1)
vhci_num_controllers = 1;
- vhci_pdevs = kcalloc(vhci_num_controllers, sizeof(void *), GFP_KERNEL);
- if (vhci_pdevs == NULL)
+ vhcis = kcalloc(vhci_num_controllers, sizeof(struct vhci), GFP_KERNEL);
+ if (vhcis == NULL)
return -ENOMEM;
+ for (i = 0; i < vhci_num_controllers; i++) {
+ vhcis[i].pdev = platform_device_alloc(driver_name, i);
+ if (!vhcis[i].pdev) {
+ i--;
+ while (i >= 0)
+ platform_device_put(vhcis[i--].pdev);
+ ret = -ENOMEM;
+ goto err_device_alloc;
+ }
+ }
+ for (i = 0; i < vhci_num_controllers; i++) {
+ void *vhci = &vhcis[i];
+ ret = platform_device_add_data(vhcis[i].pdev, &vhci, sizeof(void *));
+ if (ret)
+ goto err_driver_register;
+ }
+
ret = platform_driver_register(&vhci_driver);
if (ret)
goto err_driver_register;
for (i = 0; i < vhci_num_controllers; i++) {
- ret = add_platform_device(i);
- if (ret)
- goto err_platform_device_register;
+ ret = platform_device_add(vhcis[i].pdev);
+ if (ret < 0) {
+ i--;
+ while (i >= 0)
+ platform_device_del(vhcis[i--].pdev);
+ goto err_add_hcd;
+ }
}
pr_info(DRIVER_DESC " v" USBIP_VERSION "\n");
return ret;
-err_platform_device_register:
- del_platform_devices();
+err_add_hcd:
platform_driver_unregister(&vhci_driver);
err_driver_register:
- kfree(vhci_pdevs);
+ for (i = 0; i < vhci_num_controllers; i++)
+ platform_device_put(vhcis[i].pdev);
+err_device_alloc:
+ kfree(vhcis);
return ret;
}
{
del_platform_devices();
platform_driver_unregister(&vhci_driver);
- kfree(vhci_pdevs);
+ kfree(vhcis);
}
module_init(vhci_hcd_init);
static void vhci_recv_ret_submit(struct vhci_device *vdev,
struct usbip_header *pdu)
{
- struct vhci_hcd *vhci = vdev_to_vhci(vdev);
+ struct vhci_hcd *vhci_hcd = vdev_to_vhci_hcd(vdev);
+ struct vhci *vhci = vhci_hcd->vhci;
struct usbip_device *ud = &vdev->ud;
struct urb *urb;
unsigned long flags;
if (!urb) {
pr_err("cannot find a urb of seqnum %u\n", pdu->base.seqnum);
pr_info("max seqnum %d\n",
- atomic_read(&vhci->seqnum));
+ atomic_read(&vhci_hcd->seqnum));
usbip_event_add(ud, VDEV_EVENT_ERROR_TCP);
return;
}
usbip_dbg_vhci_rx("now giveback urb %p\n", urb);
spin_lock_irqsave(&vhci->lock, flags);
- usb_hcd_unlink_urb_from_ep(vhci_to_hcd(vhci), urb);
+ usb_hcd_unlink_urb_from_ep(vhci_hcd_to_hcd(vhci_hcd), urb);
spin_unlock_irqrestore(&vhci->lock, flags);
- usb_hcd_giveback_urb(vhci_to_hcd(vhci), urb, urb->status);
+ usb_hcd_giveback_urb(vhci_hcd_to_hcd(vhci_hcd), urb, urb->status);
usbip_dbg_vhci_rx("Leave\n");
}
static void vhci_recv_ret_unlink(struct vhci_device *vdev,
struct usbip_header *pdu)
{
- struct vhci_hcd *vhci = vdev_to_vhci(vdev);
+ struct vhci_hcd *vhci_hcd = vdev_to_vhci_hcd(vdev);
+ struct vhci *vhci = vhci_hcd->vhci;
struct vhci_unlink *unlink;
struct urb *urb;
unsigned long flags;
pr_info("urb->status %d\n", urb->status);
spin_lock_irqsave(&vhci->lock, flags);
- usb_hcd_unlink_urb_from_ep(vhci_to_hcd(vhci), urb);
+ usb_hcd_unlink_urb_from_ep(vhci_hcd_to_hcd(vhci_hcd), urb);
spin_unlock_irqrestore(&vhci->lock, flags);
- usb_hcd_giveback_urb(vhci_to_hcd(vhci), urb, urb->status);
+ usb_hcd_giveback_urb(vhci_hcd_to_hcd(vhci_hcd), urb, urb->status);
}
kfree(unlink);
/* TODO: refine locking ?*/
+/*
+ * output example:
+ * hub port sta spd dev socket local_busid
+ * hs 0000 004 000 00000000 c5a7bb80 1-2.3
+ * ................................................
+ * ss 0008 004 000 00000000 d8cee980 2-3.4
+ * ................................................
+ *
+ * IP address can be retrieved from a socket pointer address by looking
+ * up /proc/net/{tcp,tcp6}. Also, a userland program may remember a
+ * port number and its peer IP address.
+ */
+static void port_show_vhci(char **out, int hub, int port, struct vhci_device *vdev)
+{
+ if (hub == HUB_SPEED_HIGH)
+ *out += sprintf(*out, "hs %04u %03u ",
+ port, vdev->ud.status);
+ else /* hub == HUB_SPEED_SUPER */
+ *out += sprintf(*out, "ss %04u %03u ",
+ port, vdev->ud.status);
+
+ if (vdev->ud.status == VDEV_ST_USED) {
+ *out += sprintf(*out, "%03u %08x ",
+ vdev->speed, vdev->devid);
+ *out += sprintf(*out, "%16p %s",
+ vdev->ud.tcp_socket,
+ dev_name(&vdev->udev->dev));
+
+ } else {
+ *out += sprintf(*out, "000 00000000 ");
+ *out += sprintf(*out, "0000000000000000 0-0");
+ }
+
+ *out += sprintf(*out, "\n");
+}
+
/* Sysfs entry to show port status */
static ssize_t status_show_vhci(int pdev_nr, char *out)
{
- struct platform_device *pdev = *(vhci_pdevs + pdev_nr);
- struct vhci_hcd *vhci;
+ struct platform_device *pdev = vhcis[pdev_nr].pdev;
+ struct vhci *vhci;
+ struct usb_hcd *hcd;
+ struct vhci_hcd *vhci_hcd;
char *s = out;
- int i = 0;
+ int i;
unsigned long flags;
if (!pdev || !out) {
return 0;
}
- vhci = hcd_to_vhci(platform_get_drvdata(pdev));
+ hcd = platform_get_drvdata(pdev);
+ vhci_hcd = hcd_to_vhci_hcd(hcd);
+ vhci = vhci_hcd->vhci;
spin_lock_irqsave(&vhci->lock, flags);
- /*
- * output example:
- * port sta spd dev socket local_busid
- * 0000 004 000 00000000 c5a7bb80 1-2.3
- * 0001 004 000 00000000 d8cee980 2-3.4
- *
- * IP address can be retrieved from a socket pointer address by looking
- * up /proc/net/{tcp,tcp6}. Also, a userland program may remember a
- * port number and its peer IP address.
- */
for (i = 0; i < VHCI_HC_PORTS; i++) {
- struct vhci_device *vdev = &vhci->vdev[i];
+ struct vhci_device *vdev = &vhci->vhci_hcd_hs->vdev[i];
spin_lock(&vdev->ud.lock);
- out += sprintf(out, "%04u %03u ",
- (pdev_nr * VHCI_HC_PORTS) + i,
- vdev->ud.status);
-
- if (vdev->ud.status == VDEV_ST_USED) {
- out += sprintf(out, "%03u %08x ",
- vdev->speed, vdev->devid);
- out += sprintf(out, "%16p %s",
- vdev->ud.tcp_socket,
- dev_name(&vdev->udev->dev));
-
- } else {
- out += sprintf(out, "000 00000000 ");
- out += sprintf(out, "0000000000000000 0-0");
- }
+ port_show_vhci(&out, HUB_SPEED_HIGH,
+ pdev_nr * VHCI_PORTS + i, vdev);
+ spin_unlock(&vdev->ud.lock);
+ }
- out += sprintf(out, "\n");
+ for (i = 0; i < VHCI_HC_PORTS; i++) {
+ struct vhci_device *vdev = &vhci->vhci_hcd_ss->vdev[i];
+
+ spin_lock(&vdev->ud.lock);
+ port_show_vhci(&out, HUB_SPEED_SUPER,
+ pdev_nr * VHCI_PORTS + VHCI_HC_PORTS + i, vdev);
spin_unlock(&vdev->ud.lock);
}
int i = 0;
for (i = 0; i < VHCI_HC_PORTS; i++) {
- out += sprintf(out, "%04u %03u ",
- (pdev_nr * VHCI_HC_PORTS) + i,
+ out += sprintf(out, "hs %04u %03u ",
+ (pdev_nr * VHCI_PORTS) + i,
+ VDEV_ST_NOTASSIGNED);
+ out += sprintf(out, "000 00000000 0000000000000000 0-0");
+ out += sprintf(out, "\n");
+ }
+
+ for (i = 0; i < VHCI_HC_PORTS; i++) {
+ out += sprintf(out, "ss %04u %03u ",
+ (pdev_nr * VHCI_PORTS) + VHCI_HC_PORTS + i,
VDEV_ST_NOTASSIGNED);
out += sprintf(out, "000 00000000 0000000000000000 0-0");
out += sprintf(out, "\n");
int pdev_nr;
out += sprintf(out,
- "port sta spd dev socket local_busid\n");
+ "hub port sta spd dev socket local_busid\n");
pdev_nr = status_name_to_id(attr->attr.name);
if (pdev_nr < 0)
{
char *s = out;
- out += sprintf(out, "%d\n", VHCI_HC_PORTS * vhci_num_controllers);
+ /*
+ * Half the ports are for SPEED_HIGH and half for SPEED_SUPER, thus the * 2.
+ */
+ out += sprintf(out, "%d\n", VHCI_PORTS * vhci_num_controllers);
return out - s;
}
static DEVICE_ATTR_RO(nports);
/* Sysfs entry to shutdown a virtual connection */
-static int vhci_port_disconnect(struct vhci_hcd *vhci, __u32 rhport)
+static int vhci_port_disconnect(struct vhci_hcd *vhci_hcd, __u32 rhport)
{
- struct vhci_device *vdev = &vhci->vdev[rhport];
+ struct vhci_device *vdev = &vhci_hcd->vdev[rhport];
+ struct vhci *vhci = vhci_hcd->vhci;
unsigned long flags;
usbip_dbg_vhci_sysfs("enter\n");
{
__u32 port = 0, pdev_nr = 0, rhport = 0;
struct usb_hcd *hcd;
+ struct vhci_hcd *vhci_hcd;
int ret;
if (kstrtoint(buf, 10, &port) < 0)
if (!valid_port(pdev_nr, rhport))
return -EINVAL;
- hcd = platform_get_drvdata(*(vhci_pdevs + pdev_nr));
+ hcd = platform_get_drvdata(vhcis[pdev_nr].pdev);
if (hcd == NULL) {
dev_err(dev, "port is not ready %u\n", port);
return -EAGAIN;
}
- ret = vhci_port_disconnect(hcd_to_vhci(hcd), rhport);
+ usbip_dbg_vhci_sysfs("rhport %d\n", rhport);
+
+ if ((port / VHCI_HC_PORTS) % 2)
+ vhci_hcd = hcd_to_vhci_hcd(hcd)->vhci->vhci_hcd_ss;
+ else
+ vhci_hcd = hcd_to_vhci_hcd(hcd)->vhci->vhci_hcd_hs;
+
+ ret = vhci_port_disconnect(vhci_hcd, rhport);
if (ret < 0)
return -EINVAL;
case USB_SPEED_FULL:
case USB_SPEED_HIGH:
case USB_SPEED_WIRELESS:
+ case USB_SPEED_SUPER:
break;
default:
pr_err("Failed attach request for unsupported USB speed: %s\n",
int sockfd = 0;
__u32 port = 0, pdev_nr = 0, rhport = 0, devid = 0, speed = 0;
struct usb_hcd *hcd;
- struct vhci_hcd *vhci;
+ struct vhci_hcd *vhci_hcd;
struct vhci_device *vdev;
+ struct vhci *vhci;
int err;
unsigned long flags;
if (!valid_args(pdev_nr, rhport, speed))
return -EINVAL;
- hcd = platform_get_drvdata(*(vhci_pdevs + pdev_nr));
+ hcd = platform_get_drvdata(vhcis[pdev_nr].pdev);
if (hcd == NULL) {
dev_err(dev, "port %d is not ready\n", port);
return -EAGAIN;
}
- vhci = hcd_to_vhci(hcd);
- vdev = &vhci->vdev[rhport];
+
+ vhci_hcd = hcd_to_vhci_hcd(hcd);
+ vhci = vhci_hcd->vhci;
+
+ if (speed == USB_SPEED_SUPER)
+ vdev = &vhci->vhci_hcd_ss->vdev[rhport];
+ else
+ vdev = &vhci->vhci_hcd_hs->vdev[rhport];
/* Extract socket from fd. */
socket = sockfd_lookup(sockfd, &err);
beacon_timeout_ms = bt;
return size;
}
+static CLASS_ATTR_RW(beacon_timeout_ms);
-static struct class_attribute uwb_class_attrs[] = {
- __ATTR(beacon_timeout_ms, S_IWUSR | S_IRUGO,
- beacon_timeout_ms_show, beacon_timeout_ms_store),
- __ATTR_NULL,
+static struct attribute *uwb_class_attrs[] = {
+ &class_attr_beacon_timeout_ms.attr,
+ NULL,
};
+ATTRIBUTE_GROUPS(uwb_class);
/** Device model classes */
struct class uwb_rc_class = {
.name = "uwb_rc",
- .class_attrs = uwb_class_attrs,
+ .class_groups = uwb_class_groups,
};
dev_err(i1480->dev, "MPI-READ: command execution failed: %d\n",
reply->bResultCode);
result = -EIO;
+ goto out;
}
for (cnt = 0; cnt < size; cnt++) {
if (reply->data[cnt].page != (srcaddr + cnt) >> 8)
queue_work(vfio_irqfd_cleanup_wq, &virqfd->shutdown);
}
-static int virqfd_wakeup(wait_queue_t *wait, unsigned mode, int sync, void *key)
+static int virqfd_wakeup(wait_queue_entry_t *wait, unsigned mode, int sync, void *key)
{
struct virqfd *virqfd = container_of(wait, struct virqfd, wait);
unsigned long flags = (unsigned long)key;
add_wait_queue(wqh, &poll->wait);
}
-static int vhost_poll_wakeup(wait_queue_t *wait, unsigned mode, int sync,
+static int vhost_poll_wakeup(wait_queue_entry_t *wait, unsigned mode, int sync,
void *key)
{
struct vhost_poll *poll = container_of(wait, struct vhost_poll, wait);
struct vhost_poll {
poll_table table;
wait_queue_head_t *wqh;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
struct vhost_work work;
unsigned long mask;
struct vhost_dev *dev;
};
static struct miscdevice vhost_vsock_misc = {
- .minor = MISC_DYNAMIC_MINOR,
+ .minor = VHOST_VSOCK_MINOR,
.name = "vhost-vsock",
.fops = &vhost_vsock_fops,
};
MODULE_LICENSE("GPL v2");
MODULE_AUTHOR("Asias He");
MODULE_DESCRIPTION("vhost transport for vsock ");
+MODULE_ALIAS_MISCDEV(VHOST_VSOCK_MINOR);
+MODULE_ALIAS("devname:vhost-vsock");
#include <asm/io.h>
-#include "../w1.h"
-#include "../w1_int.h"
+#include <linux/w1.h>
#define DS1WM_CMD 0x00 /* R/W 4 bits command */
#include <linux/delay.h>
#include <asm/delay.h>
-#include "../w1.h"
-#include "../w1_int.h"
+#include <linux/w1.h>
/**
* Allow the active pullup to be disabled, default is enabled.
*/
static int ds2482_active_pullup = 1;
module_param_named(active_pullup, ds2482_active_pullup, int, 0644);
+MODULE_PARM_DESC(active_pullup, "Active pullup (apply to all buses): " \
+ "0-disable, 1-enable (default)");
/**
* The DS2482 registers - there are 3 registers that are addressed by a read
#define DS2482_REG_STS_PPD 0x02
#define DS2482_REG_STS_1WB 0x01
-
-static int ds2482_probe(struct i2c_client *client,
- const struct i2c_device_id *id);
-static int ds2482_remove(struct i2c_client *client);
-
-
-/**
- * Driver data (common to all clients)
- */
-static const struct i2c_device_id ds2482_id[] = {
- { "ds2482", 0 },
- { }
-};
-MODULE_DEVICE_TABLE(i2c, ds2482_id);
-
-static struct i2c_driver ds2482_driver = {
- .driver = {
- .name = "ds2482",
- },
- .probe = ds2482_probe,
- .remove = ds2482_remove,
- .id_table = ds2482_id,
-};
-
/*
* Client data (each client gets its own)
*/
return 0;
}
+/**
+ * Driver data (common to all clients)
+ */
+static const struct i2c_device_id ds2482_id[] = {
+ { "ds2482", 0 },
+ { }
+};
+MODULE_DEVICE_TABLE(i2c, ds2482_id);
+
+static struct i2c_driver ds2482_driver = {
+ .driver = {
+ .name = "ds2482",
+ },
+ .probe = ds2482_probe,
+ .remove = ds2482_remove,
+ .id_table = ds2482_id,
+};
module_i2c_driver(ds2482_driver);
-MODULE_PARM_DESC(active_pullup, "Active pullup (apply to all buses): " \
- "0-disable, 1-enable (default)");
MODULE_AUTHOR("Ben Gardner <bgardner@wabtec.com>");
MODULE_DESCRIPTION("DS2482 driver");
MODULE_LICENSE("GPL");
#include <linux/usb.h>
#include <linux/slab.h>
-#include "../w1_int.h"
-#include "../w1.h"
+#include <linux/w1.h>
/* USB Standard */
/* USB Control request vendor type */
u8 reserved2;
};
-static struct usb_device_id ds_id_table [] = {
- { USB_DEVICE(0x04fa, 0x2490) },
- { },
-};
-MODULE_DEVICE_TABLE(usb, ds_id_table);
-
-static int ds_probe(struct usb_interface *, const struct usb_device_id *);
-static void ds_disconnect(struct usb_interface *);
-
-static int ds_send_control(struct ds_device *, u16, u16);
-static int ds_send_control_cmd(struct ds_device *, u16, u16);
-
static LIST_HEAD(ds_devices);
static DEFINE_MUTEX(ds_mutex);
-static struct usb_driver ds_driver = {
- .name = "DS9490R",
- .probe = ds_probe,
- .disconnect = ds_disconnect,
- .id_table = ds_id_table,
-};
-
static int ds_send_control_cmd(struct ds_device *dev, u16 value, u16 index)
{
int err;
kfree(dev);
}
+static struct usb_device_id ds_id_table [] = {
+ { USB_DEVICE(0x04fa, 0x2490) },
+ { },
+};
+MODULE_DEVICE_TABLE(usb, ds_id_table);
+
+static struct usb_driver ds_driver = {
+ .name = "DS9490R",
+ .probe = ds_probe,
+ .disconnect = ds_disconnect,
+ .id_table = ds_id_table,
+};
module_usb_driver(ds_driver);
-MODULE_LICENSE("GPL");
MODULE_AUTHOR("Evgeniy Polyakov <zbr@ioremap.net>");
MODULE_DESCRIPTION("DS2490 USB <-> W1 bus master driver (DS9490*)");
+MODULE_LICENSE("GPL");
#include <linux/pci_ids.h>
#include <linux/pci.h>
-#include "../w1.h"
-#include "../w1_int.h"
-
-MODULE_LICENSE("GPL");
-MODULE_AUTHOR("Evgeniy Polyakov <zbr@ioremap.net>");
-MODULE_DESCRIPTION("Driver for transport(Dallas 1-wire protocol) over VGA DDC(matrox gpio).");
-
-static struct pci_device_id matrox_w1_tbl[] = {
- { PCI_DEVICE(PCI_VENDOR_ID_MATROX, PCI_DEVICE_ID_MATROX_G400) },
- { },
-};
-MODULE_DEVICE_TABLE(pci, matrox_w1_tbl);
-
-static int matrox_w1_probe(struct pci_dev *, const struct pci_device_id *);
-static void matrox_w1_remove(struct pci_dev *);
-
-static struct pci_driver matrox_w1_pci_driver = {
- .name = "matrox_w1",
- .id_table = matrox_w1_tbl,
- .probe = matrox_w1_probe,
- .remove = matrox_w1_remove,
-};
+#include <linux/w1.h>
/*
* Matrox G400 DDC registers.
struct w1_bus_master *bus_master;
};
-static u8 matrox_w1_read_ddc_bit(void *);
-static void matrox_w1_write_ddc_bit(void *, u8);
-
/*
* These functions read and write DDC Data bit.
*
}
kfree(dev);
}
+
+static struct pci_device_id matrox_w1_tbl[] = {
+ { PCI_DEVICE(PCI_VENDOR_ID_MATROX, PCI_DEVICE_ID_MATROX_G400) },
+ { },
+};
+MODULE_DEVICE_TABLE(pci, matrox_w1_tbl);
+
+static struct pci_driver matrox_w1_pci_driver = {
+ .name = "matrox_w1",
+ .id_table = matrox_w1_tbl,
+ .probe = matrox_w1_probe,
+ .remove = matrox_w1_remove,
+};
module_pci_driver(matrox_w1_pci_driver);
+
+MODULE_AUTHOR("Evgeniy Polyakov <zbr@ioremap.net>");
+MODULE_DESCRIPTION("Driver for transport(Dallas 1-wire protocol) over VGA DDC(matrox gpio).");
+MODULE_LICENSE("GPL");
#include <linux/module.h>
#include <linux/platform_device.h>
-#include "../w1.h"
-#include "../w1_int.h"
+#include <linux/w1.h>
/*
* MXC W1 Register offsets
#include <linux/pm_runtime.h>
#include <linux/of.h>
-#include "../w1.h"
-#include "../w1_int.h"
+#include <linux/w1.h>
#define MOD_NAME "OMAP_HDQ:"
#define OMAP_HDQ_MAX_USER 4
static DECLARE_WAIT_QUEUE_HEAD(hdq_wait_queue);
+
static int w1_id;
+module_param(w1_id, int, S_IRUSR);
+MODULE_PARM_DESC(w1_id, "1-wire id for the slave detection in HDQ mode");
struct hdq_data {
struct device *dev;
};
-static int omap_hdq_probe(struct platform_device *pdev);
-static int omap_hdq_remove(struct platform_device *pdev);
-
-static const struct of_device_id omap_hdq_dt_ids[] = {
- { .compatible = "ti,omap3-1w" },
- { .compatible = "ti,am4372-hdq" },
- {}
-};
-MODULE_DEVICE_TABLE(of, omap_hdq_dt_ids);
-
-static struct platform_driver omap_hdq_driver = {
- .probe = omap_hdq_probe,
- .remove = omap_hdq_remove,
- .driver = {
- .name = "omap_hdq",
- .of_match_table = omap_hdq_dt_ids,
- },
-};
-
-static u8 omap_w1_read_byte(void *_hdq);
-static void omap_w1_write_byte(void *_hdq, u8 byte);
-static u8 omap_w1_reset_bus(void *_hdq);
-
-
-static struct w1_bus_master omap_w1_master = {
- .read_byte = omap_w1_read_byte,
- .write_byte = omap_w1_write_byte,
- .reset_bus = omap_w1_reset_bus,
-};
-
/* HDQ register I/O routines */
static inline u8 hdq_reg_in(struct hdq_data *hdq_data, u32 offset)
{
}
}
+static struct w1_bus_master omap_w1_master = {
+ .read_byte = omap_w1_read_byte,
+ .write_byte = omap_w1_write_byte,
+ .reset_bus = omap_w1_reset_bus,
+};
+
static int omap_hdq_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
return 0;
}
-module_platform_driver(omap_hdq_driver);
+static const struct of_device_id omap_hdq_dt_ids[] = {
+ { .compatible = "ti,omap3-1w" },
+ { .compatible = "ti,am4372-hdq" },
+ {}
+};
+MODULE_DEVICE_TABLE(of, omap_hdq_dt_ids);
-module_param(w1_id, int, S_IRUSR);
-MODULE_PARM_DESC(w1_id, "1-wire id for the slave detection in HDQ mode");
+static struct platform_driver omap_hdq_driver = {
+ .probe = omap_hdq_probe,
+ .remove = omap_hdq_remove,
+ .driver = {
+ .name = "omap_hdq",
+ .of_match_table = omap_hdq_dt_ids,
+ },
+};
+module_platform_driver(omap_hdq_driver);
MODULE_AUTHOR("Texas Instruments");
MODULE_DESCRIPTION("HDQ-1W driver Library");
#include <linux/of.h>
#include <linux/delay.h>
-#include "../w1.h"
-#include "../w1_int.h"
+#include <linux/w1.h>
static u8 w1_gpio_set_pullup(void *data, int delay)
{
#include <linux/mutex.h>
#include <linux/power/bq27xxx_battery.h>
-#include "../w1.h"
-#include "../w1_int.h"
-#include "../w1_family.h"
+#include <linux/w1.h>
+
+#define W1_FAMILY_BQ27000 0x01
#define HDQ_CMD_READ (0)
#define HDQ_CMD_WRITE (1<<7)
static int F_ID;
+module_param(F_ID, int, S_IRUSR);
+MODULE_PARM_DESC(F_ID, "1-wire slave FID for BQ device");
static int w1_bq27000_read(struct device *dev, unsigned int reg)
{
w1_unregister_family(&w1_bq27000_family);
}
-
module_init(w1_bq27000_init);
module_exit(w1_bq27000_exit);
-module_param(F_ID, int, S_IRUSR);
-MODULE_PARM_DESC(F_ID, "1-wire slave FID for BQ device");
-MODULE_ALIAS("w1-family-" __stringify(W1_FAMILY_BQ27000));
-MODULE_LICENSE("GPL");
MODULE_AUTHOR("Texas Instruments Ltd");
MODULE_DESCRIPTION("HDQ/1-wire slave driver bq27000 battery monitor chip");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("w1-family-" __stringify(W1_FAMILY_BQ27000));
#include <linux/string.h>
#include <linux/types.h>
-#include "../w1.h"
-#include "../w1_family.h"
+#include <linux/w1.h>
+
+#define W1_FAMILY_DS2405 0x05
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Maciej S. Szmigiero <mail@maciej.szmigiero.name>");
#include <linux/slab.h>
#include <linux/crc16.h>
-#include "../w1.h"
-#include "../w1_int.h"
-#include "../w1_family.h"
+#include <linux/w1.h>
-MODULE_LICENSE("GPL");
-MODULE_AUTHOR("Scott Alfter <scott@alfter.us>");
-MODULE_DESCRIPTION("w1 family 12 driver for DS2406 2 Pin IO");
+#define W1_FAMILY_DS2406 0x12
#define W1_F12_FUNC_READ_STATUS 0xAA
#define W1_F12_FUNC_WRITE_STATUS 0x55
.fops = &w1_f12_fops,
};
module_w1_family(w1_family_12);
+
+MODULE_AUTHOR("Scott Alfter <scott@alfter.us>");
+MODULE_DESCRIPTION("w1 family 12 driver for DS2406 2 Pin IO");
+MODULE_LICENSE("GPL");
#include <linux/delay.h>
#include <linux/slab.h>
-#include "../w1.h"
-#include "../w1_int.h"
-#include "../w1_family.h"
-
-MODULE_LICENSE("GPL");
-MODULE_AUTHOR("Jean-Francois Dagenais <dagenaisj@sonatest.com>");
-MODULE_DESCRIPTION("w1 family 29 driver for DS2408 8 Pin IO");
-MODULE_ALIAS("w1-family-" __stringify(W1_FAMILY_DS2408));
+#include <linux/w1.h>
+#define W1_FAMILY_DS2408 0x29
#define W1_F29_RETRIES 3
.fops = &w1_f29_fops,
};
module_w1_family(w1_family_29);
+
+MODULE_AUTHOR("Jean-Francois Dagenais <dagenaisj@sonatest.com>");
+MODULE_DESCRIPTION("w1 family 29 driver for DS2408 8 Pin IO");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("w1-family-" __stringify(W1_FAMILY_DS2408));
#include <linux/delay.h>
#include <linux/slab.h>
-#include "../w1.h"
-#include "../w1_int.h"
-#include "../w1_family.h"
+#include <linux/w1.h>
-MODULE_LICENSE("GPL");
-MODULE_AUTHOR("Mariusz Bialonczyk <manio@skyboo.net>");
-MODULE_DESCRIPTION("w1 family 3a driver for DS2413 2 Pin IO");
-MODULE_ALIAS("w1-family-" __stringify(W1_FAMILY_DS2413));
+#define W1_FAMILY_DS2413 0x3A
#define W1_F3A_RETRIES 3
#define W1_F3A_FUNC_PIO_ACCESS_READ 0xF5
.fops = &w1_f3a_fops,
};
module_w1_family(w1_family_3a);
+
+MODULE_AUTHOR("Mariusz Bialonczyk <manio@skyboo.net>");
+MODULE_DESCRIPTION("w1 family 3a driver for DS2413 2 Pin IO");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("w1-family-" __stringify(W1_FAMILY_DS2413));
#include <linux/delay.h>
#include <linux/crc16.h>
-#include "../w1.h"
-#include "../w1_int.h"
-#include "../w1_family.h"
+#include <linux/w1.h>
+
+#define W1_COUNTER_DS2423 0x1D
#define CRC16_VALID 0xb001
#define CRC16_INIT 0
};
module_w1_family(w1_family_1d);
-MODULE_LICENSE("GPL");
MODULE_AUTHOR("Mika Laitio <lamikr@pilppa.org>");
MODULE_DESCRIPTION("w1 family 1d driver for DS2423, 4 counters and 4kb ram");
+MODULE_LICENSE("GPL");
MODULE_ALIAS("w1-family-" __stringify(W1_COUNTER_DS2423));
#include <linux/types.h>
#include <linux/delay.h>
-#include "../w1.h"
-#include "../w1_int.h"
-#include "../w1_family.h"
+#include <linux/w1.h>
+
+#define W1_EEPROM_DS2431 0x2D
#define W1_F2D_EEPROM_SIZE 128
#define W1_F2D_PAGE_COUNT 4
};
module_w1_family(w1_family_2d);
-MODULE_LICENSE("GPL");
MODULE_AUTHOR("Bernhard Weirich <bernhard.weirich@riedel.net>");
MODULE_DESCRIPTION("w1 family 2d driver for DS2431, 1kb EEPROM");
+MODULE_LICENSE("GPL");
MODULE_ALIAS("w1-family-" __stringify(W1_EEPROM_DS2431));
#endif
-#include "../w1.h"
-#include "../w1_int.h"
-#include "../w1_family.h"
+#include <linux/w1.h>
-MODULE_LICENSE("GPL");
-MODULE_AUTHOR("Ben Gardner <bgardner@wabtec.com>");
-MODULE_DESCRIPTION("w1 family 23 driver for DS2433, 4kb EEPROM");
-MODULE_ALIAS("w1-family-" __stringify(W1_EEPROM_DS2433));
+#define W1_EEPROM_DS2433 0x23
#define W1_EEPROM_SIZE 512
#define W1_PAGE_COUNT 16
.fops = &w1_f23_fops,
};
module_w1_family(w1_family_23);
+
+MODULE_AUTHOR("Ben Gardner <bgardner@wabtec.com>");
+MODULE_DESCRIPTION("w1 family 23 driver for DS2433, 4kb EEPROM");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("w1-family-" __stringify(W1_EEPROM_DS2433));
#include <linux/types.h>
#include <linux/delay.h>
-#include "../w1.h"
-#include "../w1_family.h"
+#include <linux/w1.h>
+
+#define W1_FAMILY_DS2438 0x26
#define W1_DS2438_RETRIES 3
#include <linux/idr.h>
#include <linux/gfp.h>
-#include "../w1.h"
-#include "../w1_int.h"
-#include "../w1_family.h"
+#include <linux/w1.h>
+
#include "w1_ds2760.h"
+#define W1_FAMILY_DS2760 0x30
+
static int w1_ds2760_io(struct device *dev, char *buf, int addr, size_t count,
int io)
{
{
return w1_ds2760_io(dev, buf, addr, count, 0);
}
+EXPORT_SYMBOL(w1_ds2760_read);
int w1_ds2760_write(struct device *dev, char *buf, int addr, size_t count)
{
return w1_ds2760_io(dev, buf, addr, count, 1);
}
+EXPORT_SYMBOL(w1_ds2760_write);
static int w1_ds2760_eeprom_cmd(struct device *dev, int addr, int cmd)
{
{
return w1_ds2760_eeprom_cmd(dev, addr, W1_DS2760_COPY_DATA);
}
+EXPORT_SYMBOL(w1_ds2760_store_eeprom);
int w1_ds2760_recall_eeprom(struct device *dev, int addr)
{
return w1_ds2760_eeprom_cmd(dev, addr, W1_DS2760_RECALL_DATA);
}
+EXPORT_SYMBOL(w1_ds2760_recall_eeprom);
static ssize_t w1_slave_read(struct file *filp, struct kobject *kobj,
struct bin_attribute *bin_attr, char *buf,
};
module_w1_family(w1_ds2760_family);
-EXPORT_SYMBOL(w1_ds2760_read);
-EXPORT_SYMBOL(w1_ds2760_write);
-EXPORT_SYMBOL(w1_ds2760_store_eeprom);
-EXPORT_SYMBOL(w1_ds2760_recall_eeprom);
-
-MODULE_LICENSE("GPL");
MODULE_AUTHOR("Szabolcs Gyurko <szabolcs.gyurko@tlt.hu>");
MODULE_DESCRIPTION("1-wire Driver Dallas 2760 battery monitor chip");
+MODULE_LICENSE("GPL");
MODULE_ALIAS("w1-family-" __stringify(W1_FAMILY_DS2760));
#include <linux/mutex.h>
#include <linux/idr.h>
-#include "../w1.h"
-#include "../w1_int.h"
-#include "../w1_family.h"
+#include <linux/w1.h>
+
#include "w1_ds2780.h"
+#define W1_FAMILY_DS2780 0x32
+
static int w1_ds2780_do_io(struct device *dev, char *buf, int addr,
size_t count, int io)
{
};
module_w1_family(w1_ds2780_family);
-MODULE_LICENSE("GPL");
MODULE_AUTHOR("Clifton Barnes <cabarnes@indesign-llc.com>");
MODULE_DESCRIPTION("1-wire Driver for Maxim/Dallas DS2780 Stand-Alone Fuel Gauge IC");
+MODULE_LICENSE("GPL");
MODULE_ALIAS("w1-family-" __stringify(W1_FAMILY_DS2780));
#include <linux/platform_device.h>
#include <linux/mutex.h>
-#include "../w1.h"
-#include "../w1_int.h"
-#include "../w1_family.h"
+#include <linux/w1.h>
+
#include "w1_ds2781.h"
+#define W1_FAMILY_DS2781 0x3D
+
static int w1_ds2781_do_io(struct device *dev, char *buf, int addr,
size_t count, int io)
{
};
module_w1_family(w1_ds2781_family);
-MODULE_LICENSE("GPL");
MODULE_AUTHOR("Renata Sayakhova <renata@oktetlabs.ru>");
MODULE_DESCRIPTION("1-wire Driver for Maxim/Dallas DS2781 Stand-Alone Fuel Gauge IC");
+MODULE_LICENSE("GPL");
MODULE_ALIAS("w1-family-" __stringify(W1_FAMILY_DS2781));
#define CRC16_INIT 0
#define CRC16_VALID 0xb001
-#include "../w1.h"
-#include "../w1_int.h"
-#include "../w1_family.h"
+#include <linux/w1.h>
-MODULE_LICENSE("GPL");
-MODULE_AUTHOR("Markus Franke <franke.m@sebakmt.com>, <franm@hrz.tu-chemnitz.de>");
-MODULE_DESCRIPTION("w1 family 1C driver for DS28E04, 4kb EEPROM and PIO");
-MODULE_ALIAS("w1-family-" __stringify(W1_FAMILY_DS28E04));
+#define W1_FAMILY_DS28E04 0x1C
/* Allow the strong pullup to be disabled, but default to enabled.
* If it was disabled a parasite powered device might not get the required
.fops = &w1_f1C_fops,
};
module_w1_family(w1_family_1C);
+
+MODULE_AUTHOR("Markus Franke <franke.m@sebakmt.com>, <franm@hrz.tu-chemnitz.de>");
+MODULE_DESCRIPTION("w1 family 1C driver for DS28E04, 4kb EEPROM and PIO");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("w1-family-" __stringify(W1_FAMILY_DS28E04));
#include <linux/device.h>
#include <linux/types.h>
-#include "../w1.h"
-#include "../w1_int.h"
-#include "../w1_family.h"
+#include <linux/w1.h>
-MODULE_LICENSE("GPL");
-MODULE_AUTHOR("Evgeniy Polyakov <zbr@ioremap.net>");
-MODULE_DESCRIPTION("Driver for 1-wire Dallas network protocol, 64bit memory family.");
-MODULE_ALIAS("w1-family-" __stringify(W1_FAMILY_SMEM_01));
-MODULE_ALIAS("w1-family-" __stringify(W1_FAMILY_SMEM_81));
+#define W1_FAMILY_SMEM_01 0x01
+#define W1_FAMILY_SMEM_81 0x81
static struct w1_family w1_smem_family_01 = {
.fid = W1_FAMILY_SMEM_01,
module_init(w1_smem_init);
module_exit(w1_smem_fini);
+
+MODULE_AUTHOR("Evgeniy Polyakov <zbr@ioremap.net>");
+MODULE_DESCRIPTION("Driver for 1-wire Dallas network protocol, 64bit memory family.");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("w1-family-" __stringify(W1_FAMILY_SMEM_01));
+MODULE_ALIAS("w1-family-" __stringify(W1_FAMILY_SMEM_81));
#include <linux/slab.h>
#include <linux/delay.h>
-#include "../w1.h"
-#include "../w1_int.h"
-#include "../w1_family.h"
+#include <linux/w1.h>
-MODULE_LICENSE("GPL");
-MODULE_AUTHOR("Evgeniy Polyakov <zbr@ioremap.net>");
-MODULE_DESCRIPTION("Driver for 1-wire Dallas network protocol, temperature family.");
-MODULE_ALIAS("w1-family-" __stringify(W1_THERM_DS18S20));
-MODULE_ALIAS("w1-family-" __stringify(W1_THERM_DS1822));
-MODULE_ALIAS("w1-family-" __stringify(W1_THERM_DS18B20));
-MODULE_ALIAS("w1-family-" __stringify(W1_THERM_DS1825));
-MODULE_ALIAS("w1-family-" __stringify(W1_THERM_DS28EA00));
+#define W1_THERM_DS18S20 0x10
+#define W1_THERM_DS1822 0x22
+#define W1_THERM_DS18B20 0x28
+#define W1_THERM_DS1825 0x3B
+#define W1_THERM_DS28EA00 0x42
/* Allow the strong pullup to be disabled, but default to enabled.
* If it was disabled a parasite powered device might not get the require
module_init(w1_therm_init);
module_exit(w1_therm_fini);
+
+MODULE_AUTHOR("Evgeniy Polyakov <zbr@ioremap.net>");
+MODULE_DESCRIPTION("Driver for 1-wire Dallas network protocol, temperature family.");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("w1-family-" __stringify(W1_THERM_DS18S20));
+MODULE_ALIAS("w1-family-" __stringify(W1_THERM_DS1822));
+MODULE_ALIAS("w1-family-" __stringify(W1_THERM_DS18B20));
+MODULE_ALIAS("w1-family-" __stringify(W1_THERM_DS1825));
+MODULE_ALIAS("w1-family-" __stringify(W1_THERM_DS28EA00));
#include <linux/atomic.h>
-#include "w1.h"
-#include "w1_int.h"
-#include "w1_family.h"
+#include "w1_internal.h"
#include "w1_netlink.h"
-MODULE_LICENSE("GPL");
-MODULE_AUTHOR("Evgeniy Polyakov <zbr@ioremap.net>");
-MODULE_DESCRIPTION("Driver for 1-wire Dallas network protocol.");
+#define W1_FAMILY_DEFAULT 0
static int w1_timeout = 10;
-static int w1_timeout_us = 0;
-int w1_max_slave_count = 64;
-int w1_max_slave_ttl = 10;
-
module_param_named(timeout, w1_timeout, int, 0);
MODULE_PARM_DESC(timeout, "time in seconds between automatic slave searches");
+
+static int w1_timeout_us = 0;
module_param_named(timeout_us, w1_timeout_us, int, 0);
MODULE_PARM_DESC(timeout_us,
"time in microseconds between automatic slave searches");
+
/* A search stops when w1_max_slave_count devices have been found in that
* search. The next search will start over and detect the same set of devices
* on a static 1-wire bus. Memory is not allocated based on this number, just
* device on the network and w1_max_slave_count is set to 1, the device id can
* be read directly skipping the normal slower search process.
*/
+int w1_max_slave_count = 64;
module_param_named(max_slave_count, w1_max_slave_count, int, 0);
MODULE_PARM_DESC(max_slave_count,
"maximum number of slaves detected in a search");
+
+int w1_max_slave_ttl = 10;
module_param_named(slave_ttl, w1_max_slave_ttl, int, 0);
MODULE_PARM_DESC(slave_ttl,
"Number of searches not seeing a slave before it will be removed");
module_init(w1_init);
module_exit(w1_fini);
+
+MODULE_AUTHOR("Evgeniy Polyakov <zbr@ioremap.net>");
+MODULE_DESCRIPTION("Driver for 1-wire Dallas network protocol.");
+MODULE_LICENSE("GPL");
#include <linux/delay.h>
#include <linux/export.h>
-#include "w1_family.h"
-#include "w1.h"
+#include "w1_internal.h"
DEFINE_SPINLOCK(w1_flock);
static LIST_HEAD(w1_families);
return ret;
}
+EXPORT_SYMBOL(w1_register_family);
/**
* w1_unregister_family() - unregister a device family driver
flush_signals(current);
}
}
+EXPORT_SYMBOL(w1_unregister_family);
/*
* Should be called under w1_flock held.
atomic_inc(&f->refcnt);
smp_mb__after_atomic();
}
-
-EXPORT_SYMBOL(w1_unregister_family);
-EXPORT_SYMBOL(w1_register_family);
+++ /dev/null
-/*
- * Copyright (c) 2004 Evgeniy Polyakov <zbr@ioremap.net>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- */
-
-#ifndef __W1_FAMILY_H
-#define __W1_FAMILY_H
-
-#include <linux/types.h>
-#include <linux/device.h>
-#include <linux/atomic.h>
-
-#define W1_FAMILY_DEFAULT 0
-#define W1_FAMILY_BQ27000 0x01
-#define W1_FAMILY_SMEM_01 0x01
-#define W1_FAMILY_SMEM_81 0x81
-#define W1_FAMILY_DS2405 0x05
-#define W1_THERM_DS18S20 0x10
-#define W1_FAMILY_DS28E04 0x1C
-#define W1_COUNTER_DS2423 0x1D
-#define W1_THERM_DS1822 0x22
-#define W1_EEPROM_DS2433 0x23
-#define W1_FAMILY_DS2438 0x26
-#define W1_THERM_DS18B20 0x28
-#define W1_FAMILY_DS2408 0x29
-#define W1_EEPROM_DS2431 0x2D
-#define W1_FAMILY_DS2760 0x30
-#define W1_FAMILY_DS2780 0x32
-#define W1_FAMILY_DS2413 0x3A
-#define W1_FAMILY_DS2406 0x12
-#define W1_THERM_DS1825 0x3B
-#define W1_FAMILY_DS2781 0x3D
-#define W1_THERM_DS28EA00 0x42
-
-#define MAXNAMELEN 32
-
-struct w1_slave;
-
-/**
- * struct w1_family_ops - operations for a family type
- * @add_slave: add_slave
- * @remove_slave: remove_slave
- * @groups: sysfs group
- */
-struct w1_family_ops
-{
- int (* add_slave)(struct w1_slave *);
- void (* remove_slave)(struct w1_slave *);
- const struct attribute_group **groups;
-};
-
-/**
- * struct w1_family - reference counted family structure.
- * @family_entry: family linked list
- * @fid: 8 bit family identifier
- * @fops: operations for this family
- * @refcnt: reference counter
- */
-struct w1_family
-{
- struct list_head family_entry;
- u8 fid;
-
- struct w1_family_ops *fops;
-
- atomic_t refcnt;
-};
-
-extern spinlock_t w1_flock;
-
-void w1_family_put(struct w1_family *);
-void __w1_family_get(struct w1_family *);
-struct w1_family * w1_family_registered(u8);
-void w1_unregister_family(struct w1_family *);
-int w1_register_family(struct w1_family *);
-
-/**
- * module_w1_driver() - Helper macro for registering a 1-Wire families
- * @__w1_family: w1_family struct
- *
- * Helper macro for 1-Wire families which do not do anything special in module
- * init/exit. This eliminates a lot of boilerplate. Each module may only
- * use this macro once, and calling it replaces module_init() and module_exit()
- */
-#define module_w1_family(__w1_family) \
- module_driver(__w1_family, w1_register_family, \
- w1_unregister_family)
-
-#endif /* __W1_FAMILY_H */
#include <linux/export.h>
#include <linux/moduleparam.h>
-#include "w1.h"
+#include "w1_internal.h"
#include "w1_netlink.h"
-#include "w1_int.h"
static int w1_search_count = -1; /* Default is continual scan */
module_param_named(search_count, w1_search_count, int, 0);
return retval;
}
+EXPORT_SYMBOL(w1_add_master_device);
void __w1_remove_master_device(struct w1_master *dev)
{
__w1_remove_master_device(found);
}
-
-EXPORT_SYMBOL(w1_add_master_device);
EXPORT_SYMBOL(w1_remove_master_device);
+++ /dev/null
-/*
- * Copyright (c) 2004 Evgeniy Polyakov <zbr@ioremap.net>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- */
-
-#ifndef __W1_INT_H
-#define __W1_INT_H
-
-#include <linux/kernel.h>
-#include <linux/device.h>
-
-#include "w1.h"
-
-int w1_add_master_device(struct w1_bus_master *);
-void w1_remove_master_device(struct w1_bus_master *);
-void __w1_remove_master_device(struct w1_master *);
-
-#endif /* __W1_INT_H */
--- /dev/null
+/*
+ * Copyright (c) 2004 Evgeniy Polyakov <zbr@ioremap.net>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __W1_H
+#define __W1_H
+
+#include <linux/w1.h>
+
+#include <linux/completion.h>
+#include <linux/mutex.h>
+
+#define W1_SLAVE_ACTIVE 0
+#define W1_SLAVE_DETACH 1
+
+/**
+ * struct w1_async_cmd - execute callback from the w1_process kthread
+ * @async_entry: link entry
+ * @cb: callback function, must list_del and destroy this list before
+ * returning
+ *
+ * When inserted into the w1_master async_list, w1_process will execute
+ * the callback. Embed this into the structure with the command details.
+ */
+struct w1_async_cmd {
+ struct list_head async_entry;
+ void (*cb)(struct w1_master *dev, struct w1_async_cmd *async_cmd);
+};
+
+int w1_create_master_attributes(struct w1_master *master);
+void w1_destroy_master_attributes(struct w1_master *master);
+void w1_search(struct w1_master *dev, u8 search_type,
+ w1_slave_found_callback cb);
+void w1_search_devices(struct w1_master *dev, u8 search_type,
+ w1_slave_found_callback cb);
+/* call w1_unref_slave to release the reference counts w1_search_slave added */
+struct w1_slave *w1_search_slave(struct w1_reg_num *id);
+/*
+ * decrements the reference on sl->master and sl, and cleans up if zero
+ * returns the reference count after it has been decremented
+ */
+int w1_unref_slave(struct w1_slave *sl);
+void w1_slave_found(struct w1_master *dev, u64 rn);
+void w1_search_process_cb(struct w1_master *dev, u8 search_type,
+ w1_slave_found_callback cb);
+struct w1_slave *w1_slave_search_device(struct w1_master *dev,
+ struct w1_reg_num *rn);
+struct w1_master *w1_search_master_id(u32 id);
+
+/* Disconnect and reconnect devices in the given family. Used for finding
+ * unclaimed devices after a family has been registered or releasing devices
+ * after a family has been unregistered. Set attach to 1 when a new family
+ * has just been registered, to 0 when it has been unregistered.
+ */
+void w1_reconnect_slaves(struct w1_family *f, int attach);
+int w1_attach_slave_device(struct w1_master *dev, struct w1_reg_num *rn);
+/* 0 success, otherwise EBUSY */
+int w1_slave_detach(struct w1_slave *sl);
+
+void __w1_remove_master_device(struct w1_master *dev);
+
+void w1_family_put(struct w1_family *f);
+void __w1_family_get(struct w1_family *f);
+struct w1_family *w1_family_registered(u8 fid);
+
+extern struct device_driver w1_master_driver;
+extern struct device w1_master_device;
+extern int w1_max_slave_count;
+extern int w1_max_slave_ttl;
+extern struct list_head w1_masters;
+extern struct mutex w1_mlock;
+extern spinlock_t w1_flock;
+
+int w1_process_callbacks(struct w1_master *dev);
+int w1_process(void *data);
+
+#endif /* __W1_H */
#include <linux/moduleparam.h>
#include <linux/module.h>
-#include "w1.h"
+#include "w1_internal.h"
static int w1_delay_parm = 1;
module_param_named(delay_coef, w1_delay_parm, int, 0);
#include <linux/netlink.h>
#include <linux/connector.h>
-#include "w1.h"
+#include "w1_internal.h"
#include "w1_netlink.h"
#if defined(CONFIG_W1_CON) && (defined(CONFIG_CONNECTOR) || (defined(CONFIG_CONNECTOR_MODULE) && defined(CONFIG_W1_MODULE)))
#include <asm/types.h>
#include <linux/connector.h>
-#include "w1.h"
+#include "w1_internal.h"
/**
* enum w1_cn_msg_flags - bitfield flags for struct cn_msg.flags
bool force)
{
unsigned tcpu = cpumask_first_and(dest, cpu_online_mask);
+ int ret = rebind_irq_to_cpu(data->irq, tcpu);
- return rebind_irq_to_cpu(data->irq, tcpu);
+ if (!ret)
+ irq_data_update_effective_affinity(data, cpumask_of(tcpu));
+
+ return ret;
}
static void enable_dynirq(struct irq_data *data)
{
switch (system_state) {
case SYSTEM_BOOTING:
+ case SYSTEM_SCHEDULING:
orderly_poweroff(true);
break;
case SYSTEM_RUNNING:
if (ret)
goto deregister;
+ pr_info("/dev/mcelog registered by Xen\n");
+
return 0;
deregister:
return xen_tmem_new_pool(uuid_private, 0, pagesize);
}
-static int tmem_cleancache_init_shared_fs(char *uuid, size_t pagesize)
+static int tmem_cleancache_init_shared_fs(uuid_t *uuid, size_t pagesize)
{
struct tmem_pool_uuid shared_uuid;
- shared_uuid.uuid_lo = *(u64 *)uuid;
- shared_uuid.uuid_hi = *(u64 *)(&uuid[8]);
+ shared_uuid.uuid_lo = *(u64 *)&uuid->b[0];
+ shared_uuid.uuid_hi = *(u64 *)&uuid->b[8];
return xen_tmem_new_pool(shared_uuid, TMEM_POOL_SHARED, pagesize);
}
{
struct sockaddr_rxrpc srx;
struct afs_server *server;
- struct uuid_v1 *r;
+ struct afs_uuid *r;
unsigned loop;
__be32 *b;
int ret;
}
_debug("unmarshall UUID");
- call->request = kmalloc(sizeof(struct uuid_v1), GFP_KERNEL);
+ call->request = kmalloc(sizeof(struct afs_uuid), GFP_KERNEL);
if (!call->request)
return -ENOMEM;
static void SRXAFSCB_ProbeUuid(struct work_struct *work)
{
struct afs_call *call = container_of(work, struct afs_call, work);
- struct uuid_v1 *r = call->request;
+ struct afs_uuid *r = call->request;
struct {
__be32 match;
*/
static int afs_deliver_cb_probe_uuid(struct afs_call *call)
{
- struct uuid_v1 *r;
+ struct afs_uuid *r;
unsigned loop;
__be32 *b;
int ret;
}
_debug("unmarshall UUID");
- call->request = kmalloc(sizeof(struct uuid_v1), GFP_KERNEL);
+ call->request = kmalloc(sizeof(struct afs_uuid), GFP_KERNEL);
if (!call->request)
return -ENOMEM;
b = call->buffer;
r = call->request;
- r->time_low = b[0];
- r->time_mid = htons(ntohl(b[1]));
- r->time_hi_and_version = htons(ntohl(b[2]));
+ r->time_low = ntohl(b[0]);
+ r->time_mid = ntohl(b[1]);
+ r->time_hi_and_version = ntohl(b[2]);
r->clock_seq_hi_and_reserved = ntohl(b[3]);
r->clock_seq_low = ntohl(b[4]);
unsigned mtu; /* MTU of interface */
};
+struct afs_uuid {
+ __be32 time_low; /* low part of timestamp */
+ __be16 time_mid; /* mid part of timestamp */
+ __be16 time_hi_and_version; /* high part of timestamp and version */
+ __u8 clock_seq_hi_and_reserved; /* clock seq hi and variant */
+ __u8 clock_seq_low; /* clock seq low */
+ __u8 node[6]; /* spatially unique node ID (MAC addr) */
+};
+
/*****************************************************************************/
/*
* cache.c
* main.c
*/
extern struct workqueue_struct *afs_wq;
-extern struct uuid_v1 afs_uuid;
+extern struct afs_uuid afs_uuid;
/*
* misc.c
module_param(rootcell, charp, 0);
MODULE_PARM_DESC(rootcell, "root AFS cell name and VL server IP addr list");
-struct uuid_v1 afs_uuid;
+struct afs_uuid afs_uuid;
struct workqueue_struct *afs_wq;
/*
ssize_t ret;
/* enforce forwards compatibility on users */
- if (unlikely(iocb->aio_reserved1 || iocb->aio_reserved2)) {
+ if (unlikely(iocb->aio_reserved2)) {
pr_debug("EINVAL: reserve field set\n");
return -EINVAL;
}
req->common.ki_pos = iocb->aio_offset;
req->common.ki_complete = aio_complete;
req->common.ki_flags = iocb_flags(req->common.ki_filp);
+ req->common.ki_hint = file_write_hint(file);
if (iocb->aio_flags & IOCB_FLAG_RESFD) {
/*
req->common.ki_flags |= IOCB_EVENTFD;
}
+ ret = kiocb_set_rw_flags(&req->common, iocb->aio_rw_flags);
+ if (unlikely(ret)) {
+ pr_debug("EINVAL: aio_rw_flags\n");
+ goto out_put_req;
+ }
+
+ if ((req->common.ki_flags & IOCB_NOWAIT) &&
+ !(req->common.ki_flags & IOCB_DIRECT)) {
+ ret = -EOPNOTSUPP;
+ goto out_put_req;
+ }
+
ret = put_user(KIOCB_KEY, &user_iocb->aio_key);
if (unlikely(ret)) {
pr_debug("EFAULT: aio_key\n");
struct autofs_wait_queue {
wait_queue_head_t queue;
struct autofs_wait_queue *next;
- autofs_wqt_t wait_queue_token;
+ autofs_wqt_t wait_queue_entry_token;
/* We use the following to see what we are waiting for */
struct qstr name;
u32 dev;
int status;
token = (autofs_wqt_t) param->fail.token;
- status = param->fail.status ? param->fail.status : -ENOENT;
+ status = param->fail.status < 0 ? param->fail.status : -ENOENT;
return autofs4_wait_release(sbi, token, status);
}
size_t pktsz;
pr_debug("wait id = 0x%08lx, name = %.*s, type=%d\n",
- (unsigned long) wq->wait_queue_token,
+ (unsigned long) wq->wait_queue_entry_token,
wq->name.len, wq->name.name, type);
memset(&pkt, 0, sizeof(pkt)); /* For security reasons */
pktsz = sizeof(*mp);
- mp->wait_queue_token = wq->wait_queue_token;
+ mp->wait_queue_entry_token = wq->wait_queue_entry_token;
mp->len = wq->name.len;
memcpy(mp->name, wq->name.name, wq->name.len);
mp->name[wq->name.len] = '\0';
pktsz = sizeof(*ep);
- ep->wait_queue_token = wq->wait_queue_token;
+ ep->wait_queue_entry_token = wq->wait_queue_entry_token;
ep->len = wq->name.len;
memcpy(ep->name, wq->name.name, wq->name.len);
ep->name[wq->name.len] = '\0';
pktsz = sizeof(*packet);
- packet->wait_queue_token = wq->wait_queue_token;
+ packet->wait_queue_entry_token = wq->wait_queue_entry_token;
packet->len = wq->name.len;
memcpy(packet->name, wq->name.name, wq->name.len);
packet->name[wq->name.len] = '\0';
return -ENOMEM;
}
- wq->wait_queue_token = autofs4_next_wait_queue;
+ wq->wait_queue_entry_token = autofs4_next_wait_queue;
if (++autofs4_next_wait_queue == 0)
autofs4_next_wait_queue = 1;
wq->next = sbi->queues;
}
pr_debug("new wait id = 0x%08lx, name = %.*s, nfy=%d\n",
- (unsigned long) wq->wait_queue_token, wq->name.len,
+ (unsigned long) wq->wait_queue_entry_token, wq->name.len,
wq->name.name, notify);
/*
} else {
wq->wait_ctr++;
pr_debug("existing wait id = 0x%08lx, name = %.*s, nfy=%d\n",
- (unsigned long) wq->wait_queue_token, wq->name.len,
+ (unsigned long) wq->wait_queue_entry_token, wq->name.len,
wq->name.name, notify);
mutex_unlock(&sbi->wq_mutex);
kfree(qstr.name);
}
-int autofs4_wait_release(struct autofs_sb_info *sbi, autofs_wqt_t wait_queue_token, int status)
+int autofs4_wait_release(struct autofs_sb_info *sbi, autofs_wqt_t wait_queue_entry_token, int status)
{
struct autofs_wait_queue *wq, **wql;
mutex_lock(&sbi->wq_mutex);
for (wql = &sbi->queues; (wq = *wql) != NULL; wql = &wq->next) {
- if (wq->wait_queue_token == wait_queue_token)
+ if (wq->wait_queue_entry_token == wait_queue_entry_token)
break;
}
bio_init(&bio, vecs, nr_pages);
bio.bi_bdev = bdev;
bio.bi_iter.bi_sector = pos >> 9;
+ bio.bi_write_hint = iocb->ki_hint;
bio.bi_private = current;
bio.bi_end_io = blkdev_bio_end_io_simple;
if (vecs != inline_vecs)
kfree(vecs);
- if (unlikely(bio.bi_error))
- return bio.bi_error;
+ if (unlikely(bio.bi_status))
+ ret = blk_status_to_errno(bio.bi_status);
+
+ bio_uninit(&bio);
+
return ret;
}
bool should_dirty = dio->should_dirty;
if (dio->multi_bio && !atomic_dec_and_test(&dio->ref)) {
- if (bio->bi_error && !dio->bio.bi_error)
- dio->bio.bi_error = bio->bi_error;
+ if (bio->bi_status && !dio->bio.bi_status)
+ dio->bio.bi_status = bio->bi_status;
} else {
if (!dio->is_sync) {
struct kiocb *iocb = dio->iocb;
- ssize_t ret = dio->bio.bi_error;
+ ssize_t ret;
- if (likely(!ret)) {
+ if (likely(!dio->bio.bi_status)) {
ret = dio->size;
iocb->ki_pos += ret;
+ } else {
+ ret = blk_status_to_errno(dio->bio.bi_status);
}
dio->iocb->ki_complete(iocb, ret, 0);
bool is_read = (iov_iter_rw(iter) == READ), is_sync;
loff_t pos = iocb->ki_pos;
blk_qc_t qc = BLK_QC_T_NONE;
- int ret;
+ int ret = 0;
if ((pos | iov_iter_alignment(iter)) &
(bdev_logical_block_size(bdev) - 1))
for (;;) {
bio->bi_bdev = bdev;
bio->bi_iter.bi_sector = pos >> 9;
+ bio->bi_write_hint = iocb->ki_hint;
bio->bi_private = dio;
bio->bi_end_io = blkdev_bio_end_io;
ret = bio_iov_iter_get_pages(bio, iter);
if (unlikely(ret)) {
- bio->bi_error = ret;
+ bio->bi_status = BLK_STS_IOERR;
bio_endio(bio);
break;
}
}
__set_current_state(TASK_RUNNING);
- ret = dio->bio.bi_error;
+ if (!ret)
+ ret = blk_status_to_errno(dio->bio.bi_status);
if (likely(!ret))
ret = dio->size;
static __init int blkdev_init(void)
{
- blkdev_dio_pool = bioset_create(4, offsetof(struct blkdev_dio, bio));
+ blkdev_dio_pool = bioset_create(4, offsetof(struct blkdev_dio, bio), BIOSET_NEED_BVECS);
if (!blkdev_dio_pool)
return -ENOMEM;
return 0;
* The original bio may be split to several sub-bios, this is
* done during endio of sub-bios
*/
- int (*subio_endio)(struct inode *, struct btrfs_io_bio *, int);
+ blk_status_t (*subio_endio)(struct inode *, struct btrfs_io_bio *,
+ blk_status_t);
};
/*
/* mutex is not held! This is not save if IO is not yet completed
* on umount */
iodone_w_error = 0;
- if (bp->bi_error)
+ if (bp->bi_status)
iodone_w_error = 1;
BUG_ON(NULL == block);
if ((dev_state->state->print_mask &
BTRFSIC_PRINT_MASK_END_IO_BIO_BH))
pr_info("bio_end_io(err=%d) for %c @%llu (%s/%llu/%d)\n",
- bp->bi_error,
+ bp->bi_status,
btrfsic_get_block_type(dev_state->state, block),
block->logical_bytenr, dev_state->name,
block->dev_bytenr, block->mirror_num);
unsigned long index;
int ret;
- if (bio->bi_error)
+ if (bio->bi_status)
cb->errors = 1;
/* if there are more bios still pending for this compressed
struct page *page;
unsigned long index;
- if (bio->bi_error)
+ if (bio->bi_status)
cb->errors = 1;
/* if there are more bios still pending for this compressed
cb->start,
cb->start + cb->len - 1,
NULL,
- bio->bi_error ? 0 : 1);
+ bio->bi_status ? 0 : 1);
cb->compressed_pages[0]->mapping = NULL;
end_compressed_writeback(inode, cb);
* This also checksums the file bytes and gets things ready for
* the end io hooks.
*/
-int btrfs_submit_compressed_write(struct inode *inode, u64 start,
+blk_status_t btrfs_submit_compressed_write(struct inode *inode, u64 start,
unsigned long len, u64 disk_start,
unsigned long compressed_len,
struct page **compressed_pages,
struct page *page;
u64 first_byte = disk_start;
struct block_device *bdev;
- int ret;
+ blk_status_t ret;
int skip_sum = BTRFS_I(inode)->flags & BTRFS_INODE_NODATASUM;
WARN_ON(start & ((u64)PAGE_SIZE - 1));
cb = kmalloc(compressed_bio_size(fs_info, compressed_len), GFP_NOFS);
if (!cb)
- return -ENOMEM;
+ return BLK_STS_RESOURCE;
refcount_set(&cb->pending_bios, 0);
cb->errors = 0;
cb->inode = inode;
bio = compressed_bio_alloc(bdev, first_byte, GFP_NOFS);
if (!bio) {
kfree(cb);
- return -ENOMEM;
+ return BLK_STS_RESOURCE;
}
bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
bio->bi_private = cb;
/* create and submit bios for the compressed pages */
bytes_left = compressed_len;
for (pg_index = 0; pg_index < cb->nr_pages; pg_index++) {
+ int submit = 0;
+
page = compressed_pages[pg_index];
page->mapping = inode->i_mapping;
if (bio->bi_iter.bi_size)
- ret = io_tree->ops->merge_bio_hook(page, 0,
+ submit = io_tree->ops->merge_bio_hook(page, 0,
PAGE_SIZE,
bio, 0);
- else
- ret = 0;
page->mapping = NULL;
- if (ret || bio_add_page(bio, page, PAGE_SIZE, 0) <
+ if (submit || bio_add_page(bio, page, PAGE_SIZE, 0) <
PAGE_SIZE) {
bio_get(bio);
ret = btrfs_map_bio(fs_info, bio, 0, 1);
if (ret) {
- bio->bi_error = ret;
+ bio->bi_status = ret;
bio_endio(bio);
}
ret = btrfs_map_bio(fs_info, bio, 0, 1);
if (ret) {
- bio->bi_error = ret;
+ bio->bi_status = ret;
bio_endio(bio);
}
* After the compressed pages are read, we copy the bytes into the
* bio we were passed and then call the bio end_io calls
*/
-int btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,
+blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,
int mirror_num, unsigned long bio_flags)
{
struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
u64 em_len;
u64 em_start;
struct extent_map *em;
- int ret = -ENOMEM;
+ blk_status_t ret = BLK_STS_RESOURCE;
int faili = 0;
u32 *sums;
PAGE_SIZE);
read_unlock(&em_tree->lock);
if (!em)
- return -EIO;
+ return BLK_STS_IOERR;
compressed_len = em->block_len;
cb = kmalloc(compressed_bio_size(fs_info, compressed_len), GFP_NOFS);
__GFP_HIGHMEM);
if (!cb->compressed_pages[pg_index]) {
faili = pg_index - 1;
- ret = -ENOMEM;
+ ret = BLK_STS_RESOURCE;
goto fail2;
}
}
refcount_set(&cb->pending_bios, 1);
for (pg_index = 0; pg_index < nr_pages; pg_index++) {
+ int submit = 0;
+
page = cb->compressed_pages[pg_index];
page->mapping = inode->i_mapping;
page->index = em_start >> PAGE_SHIFT;
if (comp_bio->bi_iter.bi_size)
- ret = tree->ops->merge_bio_hook(page, 0,
+ submit = tree->ops->merge_bio_hook(page, 0,
PAGE_SIZE,
comp_bio, 0);
- else
- ret = 0;
page->mapping = NULL;
- if (ret || bio_add_page(comp_bio, page, PAGE_SIZE, 0) <
+ if (submit || bio_add_page(comp_bio, page, PAGE_SIZE, 0) <
PAGE_SIZE) {
bio_get(comp_bio);
ret = btrfs_map_bio(fs_info, comp_bio, mirror_num, 0);
if (ret) {
- comp_bio->bi_error = ret;
+ comp_bio->bi_status = ret;
bio_endio(comp_bio);
}
ret = btrfs_map_bio(fs_info, comp_bio, mirror_num, 0);
if (ret) {
- comp_bio->bi_error = ret;
+ comp_bio->bi_status = ret;
bio_endio(comp_bio);
}
unsigned long total_out, u64 disk_start,
struct bio *bio);
-int btrfs_submit_compressed_write(struct inode *inode, u64 start,
+blk_status_t btrfs_submit_compressed_write(struct inode *inode, u64 start,
unsigned long len, u64 disk_start,
unsigned long compressed_len,
struct page **compressed_pages,
unsigned long nr_pages);
-int btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,
+blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,
int mirror_num, unsigned long bio_flags);
enum btrfs_compression_type {
struct btrfs_dio_private;
int btrfs_del_csums(struct btrfs_trans_handle *trans,
struct btrfs_fs_info *fs_info, u64 bytenr, u64 len);
-int btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, u32 *dst);
-int btrfs_lookup_bio_sums_dio(struct inode *inode, struct bio *bio,
+blk_status_t btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, u32 *dst);
+blk_status_t btrfs_lookup_bio_sums_dio(struct inode *inode, struct bio *bio,
u64 logical_offset);
int btrfs_insert_file_extent(struct btrfs_trans_handle *trans,
struct btrfs_root *root,
int btrfs_csum_file_blocks(struct btrfs_trans_handle *trans,
struct btrfs_root *root,
struct btrfs_ordered_sum *sums);
-int btrfs_csum_one_bio(struct inode *inode, struct bio *bio,
+blk_status_t btrfs_csum_one_bio(struct inode *inode, struct bio *bio,
u64 file_start, int contig);
int btrfs_lookup_csums_range(struct btrfs_root *root, u64 start, u64 end,
struct list_head *list, int search_commit);
bio_end_io_t *end_io;
void *private;
struct btrfs_fs_info *info;
- int error;
+ blk_status_t status;
enum btrfs_wq_endio_type metadata;
struct list_head list;
struct btrfs_work work;
*/
u64 bio_offset;
struct btrfs_work work;
- int error;
+ blk_status_t status;
};
/*
btrfs_work_func_t func;
fs_info = end_io_wq->info;
- end_io_wq->error = bio->bi_error;
+ end_io_wq->status = bio->bi_status;
if (bio_op(bio) == REQ_OP_WRITE) {
if (end_io_wq->metadata == BTRFS_WQ_ENDIO_METADATA) {
btrfs_queue_work(wq, &end_io_wq->work);
}
-int btrfs_bio_wq_end_io(struct btrfs_fs_info *info, struct bio *bio,
+blk_status_t btrfs_bio_wq_end_io(struct btrfs_fs_info *info, struct bio *bio,
enum btrfs_wq_endio_type metadata)
{
struct btrfs_end_io_wq *end_io_wq;
end_io_wq = kmem_cache_alloc(btrfs_end_io_wq_cache, GFP_NOFS);
if (!end_io_wq)
- return -ENOMEM;
+ return BLK_STS_RESOURCE;
end_io_wq->private = bio->bi_private;
end_io_wq->end_io = bio->bi_end_io;
end_io_wq->info = info;
- end_io_wq->error = 0;
+ end_io_wq->status = 0;
end_io_wq->bio = bio;
end_io_wq->metadata = metadata;
static void run_one_async_start(struct btrfs_work *work)
{
struct async_submit_bio *async;
- int ret;
+ blk_status_t ret;
async = container_of(work, struct async_submit_bio, work);
ret = async->submit_bio_start(async->inode, async->bio,
async->mirror_num, async->bio_flags,
async->bio_offset);
if (ret)
- async->error = ret;
+ async->status = ret;
}
static void run_one_async_done(struct btrfs_work *work)
wake_up(&fs_info->async_submit_wait);
/* If an error occurred we just want to clean up the bio and move on */
- if (async->error) {
- async->bio->bi_error = async->error;
+ if (async->status) {
+ async->bio->bi_status = async->status;
bio_endio(async->bio);
return;
}
kfree(async);
}
-int btrfs_wq_submit_bio(struct btrfs_fs_info *fs_info, struct inode *inode,
- struct bio *bio, int mirror_num,
- unsigned long bio_flags,
- u64 bio_offset,
- extent_submit_bio_hook_t *submit_bio_start,
- extent_submit_bio_hook_t *submit_bio_done)
+blk_status_t btrfs_wq_submit_bio(struct btrfs_fs_info *fs_info,
+ struct inode *inode, struct bio *bio, int mirror_num,
+ unsigned long bio_flags, u64 bio_offset,
+ extent_submit_bio_hook_t *submit_bio_start,
+ extent_submit_bio_hook_t *submit_bio_done)
{
struct async_submit_bio *async;
async = kmalloc(sizeof(*async), GFP_NOFS);
if (!async)
- return -ENOMEM;
+ return BLK_STS_RESOURCE;
async->inode = inode;
async->bio = bio;
async->bio_flags = bio_flags;
async->bio_offset = bio_offset;
- async->error = 0;
+ async->status = 0;
atomic_inc(&fs_info->nr_async_submits);
return 0;
}
-static int btree_csum_one_bio(struct bio *bio)
+static blk_status_t btree_csum_one_bio(struct bio *bio)
{
struct bio_vec *bvec;
struct btrfs_root *root;
break;
}
- return ret;
+ return errno_to_blk_status(ret);
}
-static int __btree_submit_bio_start(struct inode *inode, struct bio *bio,
- int mirror_num, unsigned long bio_flags,
- u64 bio_offset)
+static blk_status_t __btree_submit_bio_start(struct inode *inode,
+ struct bio *bio, int mirror_num, unsigned long bio_flags,
+ u64 bio_offset)
{
/*
* when we're called for a write, we're already in the async
return btree_csum_one_bio(bio);
}
-static int __btree_submit_bio_done(struct inode *inode, struct bio *bio,
- int mirror_num, unsigned long bio_flags,
- u64 bio_offset)
+static blk_status_t __btree_submit_bio_done(struct inode *inode,
+ struct bio *bio, int mirror_num, unsigned long bio_flags,
+ u64 bio_offset)
{
- int ret;
+ blk_status_t ret;
/*
* when we're called for a write, we're already in the async
*/
ret = btrfs_map_bio(btrfs_sb(inode->i_sb), bio, mirror_num, 1);
if (ret) {
- bio->bi_error = ret;
+ bio->bi_status = ret;
bio_endio(bio);
}
return ret;
return 1;
}
-static int btree_submit_bio_hook(struct inode *inode, struct bio *bio,
+static blk_status_t btree_submit_bio_hook(struct inode *inode, struct bio *bio,
int mirror_num, unsigned long bio_flags,
u64 bio_offset)
{
struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
int async = check_async_write(bio_flags);
- int ret;
+ blk_status_t ret;
if (bio_op(bio) != REQ_OP_WRITE) {
/*
return 0;
out_w_error:
- bio->bi_error = ret;
+ bio->bi_status = ret;
bio_endio(bio);
return ret;
}
end_io_wq = container_of(work, struct btrfs_end_io_wq, work);
bio = end_io_wq->bio;
- bio->bi_error = end_io_wq->error;
+ bio->bi_status = end_io_wq->status;
bio->bi_private = end_io_wq->private;
bio->bi_end_io = end_io_wq->end_io;
kmem_cache_free(btrfs_end_io_wq_cache, end_io_wq);
* any device where the flush fails with eopnotsupp are flagged as not-barrier
* capable
*/
-static int write_dev_flush(struct btrfs_device *device, int wait)
+static blk_status_t write_dev_flush(struct btrfs_device *device, int wait)
{
struct request_queue *q = bdev_get_queue(device->bdev);
struct bio *bio;
- int ret = 0;
+ blk_status_t ret = 0;
if (!test_bit(QUEUE_FLAG_WC, &q->queue_flags))
return 0;
wait_for_completion(&device->flush_wait);
- if (bio->bi_error) {
- ret = bio->bi_error;
+ if (bio->bi_status) {
+ ret = bio->bi_status;
btrfs_dev_stat_inc_and_print(device,
BTRFS_DEV_STAT_FLUSH_ERRS);
}
device->flush_bio = NULL;
bio = btrfs_io_bio_alloc(GFP_NOFS, 0);
if (!bio)
- return -ENOMEM;
+ return BLK_STS_RESOURCE;
bio->bi_end_io = btrfs_end_empty_barrier;
bio->bi_bdev = device->bdev;
struct btrfs_device *dev;
int errors_send = 0;
int errors_wait = 0;
- int ret;
+ blk_status_t ret;
/* send down all the barriers */
head = &info->fs_devices->devices;
int btrfs_read_buffer(struct extent_buffer *buf, u64 parent_transid);
u32 btrfs_csum_data(const char *data, u32 seed, size_t len);
void btrfs_csum_final(u32 crc, u8 *result);
-int btrfs_bio_wq_end_io(struct btrfs_fs_info *info, struct bio *bio,
+blk_status_t btrfs_bio_wq_end_io(struct btrfs_fs_info *info, struct bio *bio,
enum btrfs_wq_endio_type metadata);
-int btrfs_wq_submit_bio(struct btrfs_fs_info *fs_info, struct inode *inode,
- struct bio *bio, int mirror_num,
- unsigned long bio_flags, u64 bio_offset,
- extent_submit_bio_hook_t *submit_bio_start,
- extent_submit_bio_hook_t *submit_bio_done);
+blk_status_t btrfs_wq_submit_bio(struct btrfs_fs_info *fs_info,
+ struct inode *inode, struct bio *bio, int mirror_num,
+ unsigned long bio_flags, u64 bio_offset,
+ extent_submit_bio_hook_t *submit_bio_start,
+ extent_submit_bio_hook_t *submit_bio_done);
unsigned long btrfs_async_submit_limit(struct btrfs_fs_info *info);
int btrfs_write_tree_block(struct extent_buffer *buf);
int btrfs_wait_tree_block_writeback(struct extent_buffer *buf);
goto free_state_cache;
btrfs_bioset = bioset_create(BIO_POOL_SIZE,
- offsetof(struct btrfs_io_bio, bio));
+ offsetof(struct btrfs_io_bio, bio),
+ BIOSET_NEED_BVECS);
if (!btrfs_bioset)
goto free_buffer_cache;
struct extent_io_tree *tree = &BTRFS_I(inode)->io_tree;
struct bio *bio;
int read_mode = 0;
+ blk_status_t status;
int ret;
BUG_ON(bio_op(failed_bio) == REQ_OP_WRITE);
"Repair Read Error: submitting new read[%#x] to this_mirror=%d, in_validation=%d",
read_mode, failrec->this_mirror, failrec->in_validation);
- ret = tree->ops->submit_bio_hook(inode, bio, failrec->this_mirror,
+ status = tree->ops->submit_bio_hook(inode, bio, failrec->this_mirror,
failrec->bio_flags, 0);
- if (ret) {
+ if (status) {
free_io_failure(BTRFS_I(inode), failrec);
bio_put(bio);
+ ret = blk_status_to_errno(status);
}
return ret;
*/
static void end_bio_extent_writepage(struct bio *bio)
{
+ int error = blk_status_to_errno(bio->bi_status);
struct bio_vec *bvec;
u64 start;
u64 end;
start = page_offset(page);
end = start + bvec->bv_offset + bvec->bv_len - 1;
- end_extent_writepage(page, bio->bi_error, start, end);
+ end_extent_writepage(page, error, start, end);
end_page_writeback(page);
}
static void end_bio_extent_readpage(struct bio *bio)
{
struct bio_vec *bvec;
- int uptodate = !bio->bi_error;
+ int uptodate = !bio->bi_status;
struct btrfs_io_bio *io_bio = btrfs_io_bio(bio);
struct extent_io_tree *tree;
u64 offset = 0;
btrfs_debug(fs_info,
"end_bio_extent_readpage: bi_sector=%llu, err=%d, mirror=%u",
- (u64)bio->bi_iter.bi_sector, bio->bi_error,
+ (u64)bio->bi_iter.bi_sector, bio->bi_status,
io_bio->mirror_num);
tree = &BTRFS_I(inode)->io_tree;
ret = bio_readpage_error(bio, offset, page,
start, end, mirror);
if (ret == 0) {
- uptodate = !bio->bi_error;
+ uptodate = !bio->bi_status;
offset += len;
continue;
}
endio_readpage_release_extent(tree, extent_start, extent_len,
uptodate);
if (io_bio->end_io)
- io_bio->end_io(io_bio, bio->bi_error);
+ io_bio->end_io(io_bio, blk_status_to_errno(bio->bi_status));
bio_put(bio);
}
static int __must_check submit_one_bio(struct bio *bio, int mirror_num,
unsigned long bio_flags)
{
- int ret = 0;
+ blk_status_t ret = 0;
struct bio_vec *bvec = bio->bi_io_vec + bio->bi_vcnt - 1;
struct page *page = bvec->bv_page;
struct extent_io_tree *tree = bio->bi_private;
btrfsic_submit_bio(bio);
bio_put(bio);
- return ret;
+ return blk_status_to_errno(ret);
}
static int merge_bio(struct extent_io_tree *tree, struct page *page,
bio_add_page(bio, page, page_size, offset);
bio->bi_end_io = end_io_func;
bio->bi_private = tree;
+ bio->bi_write_hint = page->mapping->host->i_write_hint;
bio_set_op_attrs(bio, op, op_flags);
if (wbc) {
wbc_init_bio(wbc, bio);
BUG_ON(!eb);
done = atomic_dec_and_test(&eb->io_pages);
- if (bio->bi_error ||
+ if (bio->bi_status ||
test_bit(EXTENT_BUFFER_WRITE_ERR, &eb->bflags)) {
ClearPageUptodate(page);
set_btree_ioerr(page);
struct btrfs_io_bio;
struct io_failure_record;
-typedef int (extent_submit_bio_hook_t)(struct inode *inode, struct bio *bio,
- int mirror_num, unsigned long bio_flags,
- u64 bio_offset);
+typedef blk_status_t (extent_submit_bio_hook_t)(struct inode *inode,
+ struct bio *bio, int mirror_num, unsigned long bio_flags,
+ u64 bio_offset);
struct extent_io_ops {
/*
* The following callbacks must be allways defined, the function
kfree(bio->csum_allocated);
}
-static int __btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio,
+static blk_status_t __btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio,
u64 logical_offset, u32 *dst, int dio)
{
struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
path = btrfs_alloc_path();
if (!path)
- return -ENOMEM;
+ return BLK_STS_RESOURCE;
nblocks = bio->bi_iter.bi_size >> inode->i_sb->s_blocksize_bits;
if (!dst) {
csum_size, GFP_NOFS);
if (!btrfs_bio->csum_allocated) {
btrfs_free_path(path);
- return -ENOMEM;
+ return BLK_STS_RESOURCE;
}
btrfs_bio->csum = btrfs_bio->csum_allocated;
btrfs_bio->end_io = btrfs_io_bio_endio_readpage;
return 0;
}
-int btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, u32 *dst)
+blk_status_t btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, u32 *dst)
{
return __btrfs_lookup_bio_sums(inode, bio, 0, dst, 0);
}
-int btrfs_lookup_bio_sums_dio(struct inode *inode, struct bio *bio, u64 offset)
+blk_status_t btrfs_lookup_bio_sums_dio(struct inode *inode, struct bio *bio, u64 offset)
{
return __btrfs_lookup_bio_sums(inode, bio, offset, NULL, 1);
}
return ret;
}
-int btrfs_csum_one_bio(struct inode *inode, struct bio *bio,
+blk_status_t btrfs_csum_one_bio(struct inode *inode, struct bio *bio,
u64 file_start, int contig)
{
struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
sums = kzalloc(btrfs_ordered_sum_size(fs_info, bio->bi_iter.bi_size),
GFP_NOFS);
if (!sums)
- return -ENOMEM;
+ return BLK_STS_RESOURCE;
sums->len = bio->bi_iter.bi_size;
INIT_LIST_HEAD(&sums->list);
ssize_t num_written = 0;
bool sync = (file->f_flags & O_DSYNC) || IS_SYNC(file->f_mapping->host);
ssize_t err;
- loff_t pos;
- size_t count;
+ loff_t pos = iocb->ki_pos;
+ size_t count = iov_iter_count(from);
loff_t oldsize;
int clean_page = 0;
- inode_lock(inode);
+ if ((iocb->ki_flags & IOCB_NOWAIT) &&
+ (iocb->ki_flags & IOCB_DIRECT)) {
+ /* Don't sleep on inode rwsem */
+ if (!inode_trylock(inode))
+ return -EAGAIN;
+ /*
+ * We will allocate space in case nodatacow is not set,
+ * so bail
+ */
+ if (!(BTRFS_I(inode)->flags & (BTRFS_INODE_NODATACOW |
+ BTRFS_INODE_PREALLOC)) ||
+ check_can_nocow(BTRFS_I(inode), pos, &count) <= 0) {
+ inode_unlock(inode);
+ return -EAGAIN;
+ }
+ } else
+ inode_lock(inode);
+
err = generic_write_checks(iocb, from);
if (err <= 0) {
inode_unlock(inode);
*/
update_time_for_write(inode);
- pos = iocb->ki_pos;
- count = iov_iter_count(from);
start_pos = round_down(pos, fs_info->sectorsize);
oldsize = i_size_read(inode);
if (start_pos > oldsize) {
return offset;
}
+static int btrfs_file_open(struct inode *inode, struct file *filp)
+{
+ filp->f_mode |= FMODE_AIO_NOWAIT;
+ return generic_file_open(inode, filp);
+}
+
const struct file_operations btrfs_file_operations = {
.llseek = btrfs_file_llseek,
.read_iter = generic_file_read_iter,
.splice_read = generic_file_splice_read,
.write_iter = btrfs_file_write_iter,
.mmap = btrfs_file_mmap,
- .open = generic_file_open,
+ .open = btrfs_file_open,
.release = btrfs_release_file,
.fsync = btrfs_sync_file,
.fallocate = btrfs_fallocate,
NULL, EXTENT_LOCKED | EXTENT_DELALLOC,
PAGE_UNLOCK | PAGE_CLEAR_DIRTY |
PAGE_SET_WRITEBACK);
- ret = btrfs_submit_compressed_write(inode,
+ if (btrfs_submit_compressed_write(inode,
async_extent->start,
async_extent->ram_size,
ins.objectid,
ins.offset, async_extent->pages,
- async_extent->nr_pages);
- if (ret) {
+ async_extent->nr_pages)) {
struct extent_io_tree *tree = &BTRFS_I(inode)->io_tree;
struct page *p = async_extent->pages[0];
const u64 start = async_extent->start;
* At IO completion time the cums attached on the ordered extent record
* are inserted into the btree
*/
-static int __btrfs_submit_bio_start(struct inode *inode, struct bio *bio,
- int mirror_num, unsigned long bio_flags,
- u64 bio_offset)
+static blk_status_t __btrfs_submit_bio_start(struct inode *inode,
+ struct bio *bio, int mirror_num, unsigned long bio_flags,
+ u64 bio_offset)
{
- int ret = 0;
+ blk_status_t ret = 0;
ret = btrfs_csum_one_bio(inode, bio, 0, 0);
BUG_ON(ret); /* -ENOMEM */
* At IO completion time the cums attached on the ordered extent record
* are inserted into the btree
*/
-static int __btrfs_submit_bio_done(struct inode *inode, struct bio *bio,
- int mirror_num, unsigned long bio_flags,
- u64 bio_offset)
+static blk_status_t __btrfs_submit_bio_done(struct inode *inode,
+ struct bio *bio, int mirror_num, unsigned long bio_flags,
+ u64 bio_offset)
{
struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
- int ret;
+ blk_status_t ret;
ret = btrfs_map_bio(fs_info, bio, mirror_num, 1);
if (ret) {
- bio->bi_error = ret;
+ bio->bi_status = ret;
bio_endio(bio);
}
return ret;
* extent_io.c submission hook. This does the right thing for csum calculation
* on write, or reading the csums from the tree before a read
*/
-static int btrfs_submit_bio_hook(struct inode *inode, struct bio *bio,
+static blk_status_t btrfs_submit_bio_hook(struct inode *inode, struct bio *bio,
int mirror_num, unsigned long bio_flags,
u64 bio_offset)
{
struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
struct btrfs_root *root = BTRFS_I(inode)->root;
enum btrfs_wq_endio_type metadata = BTRFS_WQ_ENDIO_DATA;
- int ret = 0;
+ blk_status_t ret = 0;
int skip_sum;
int async = !atomic_read(&BTRFS_I(inode)->sync_writers);
ret = btrfs_map_bio(fs_info, bio, mirror_num, 0);
out:
- if (ret < 0) {
- bio->bi_error = ret;
+ if (ret) {
+ bio->bi_status = ret;
bio_endio(bio);
}
return ret;
struct bio_vec *bvec;
int i;
- if (bio->bi_error)
+ if (bio->bi_status)
goto end;
ASSERT(bio->bi_vcnt == 1);
int ret;
int i;
- if (bio->bi_error)
+ if (bio->bi_status)
goto end;
uptodate = 1;
bio_put(bio);
}
-static int __btrfs_subio_endio_read(struct inode *inode,
- struct btrfs_io_bio *io_bio, int err)
+static blk_status_t __btrfs_subio_endio_read(struct inode *inode,
+ struct btrfs_io_bio *io_bio, blk_status_t err)
{
struct btrfs_fs_info *fs_info;
struct bio_vec *bvec;
io_bio->mirror_num,
btrfs_retry_endio, &done);
if (ret) {
- err = ret;
+ err = errno_to_blk_status(ret);
goto next;
}
return err;
}
-static int btrfs_subio_endio_read(struct inode *inode,
- struct btrfs_io_bio *io_bio, int err)
+static blk_status_t btrfs_subio_endio_read(struct inode *inode,
+ struct btrfs_io_bio *io_bio, blk_status_t err)
{
bool skip_csum = BTRFS_I(inode)->flags & BTRFS_INODE_NODATASUM;
struct inode *inode = dip->inode;
struct bio *dio_bio;
struct btrfs_io_bio *io_bio = btrfs_io_bio(bio);
- int err = bio->bi_error;
+ blk_status_t err = bio->bi_status;
if (dip->flags & BTRFS_DIO_ORIG_BIO_SUBMITTED)
err = btrfs_subio_endio_read(inode, io_bio, err);
kfree(dip);
- dio_bio->bi_error = bio->bi_error;
- dio_end_io(dio_bio, bio->bi_error);
+ dio_bio->bi_status = bio->bi_status;
+ dio_end_io(dio_bio);
if (io_bio->end_io)
- io_bio->end_io(io_bio, err);
+ io_bio->end_io(io_bio, blk_status_to_errno(err));
bio_put(bio);
}
struct bio *dio_bio = dip->dio_bio;
__endio_write_update_ordered(dip->inode, dip->logical_offset,
- dip->bytes, !bio->bi_error);
+ dip->bytes, !bio->bi_status);
kfree(dip);
- dio_bio->bi_error = bio->bi_error;
- dio_end_io(dio_bio, bio->bi_error);
+ dio_bio->bi_status = bio->bi_status;
+ dio_end_io(dio_bio);
bio_put(bio);
}
-static int __btrfs_submit_bio_start_direct_io(struct inode *inode,
+static blk_status_t __btrfs_submit_bio_start_direct_io(struct inode *inode,
struct bio *bio, int mirror_num,
unsigned long bio_flags, u64 offset)
{
- int ret;
+ blk_status_t ret;
ret = btrfs_csum_one_bio(inode, bio, offset, 1);
BUG_ON(ret); /* -ENOMEM */
return 0;
static void btrfs_end_dio_bio(struct bio *bio)
{
struct btrfs_dio_private *dip = bio->bi_private;
- int err = bio->bi_error;
+ blk_status_t err = bio->bi_status;
if (err)
btrfs_warn(BTRFS_I(dip->inode)->root->fs_info,
if (dip->errors) {
bio_io_error(dip->orig_bio);
} else {
- dip->dio_bio->bi_error = 0;
+ dip->dio_bio->bi_status = 0;
bio_endio(dip->orig_bio);
}
out:
return bio;
}
-static inline int btrfs_lookup_and_bind_dio_csum(struct inode *inode,
+static inline blk_status_t btrfs_lookup_and_bind_dio_csum(struct inode *inode,
struct btrfs_dio_private *dip,
struct bio *bio,
u64 file_offset)
{
struct btrfs_io_bio *io_bio = btrfs_io_bio(bio);
struct btrfs_io_bio *orig_io_bio = btrfs_io_bio(dip->orig_bio);
- int ret;
+ blk_status_t ret;
/*
* We load all the csum data we need when we submit
struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
struct btrfs_dio_private *dip = bio->bi_private;
bool write = bio_op(bio) == REQ_OP_WRITE;
- int ret;
+ blk_status_t ret;
if (async_submit)
async_submit = !atomic_read(&BTRFS_I(inode)->sync_writers);
* callbacks - they require an allocated dip and a clone of dio_bio.
*/
if (io_bio && dip) {
- io_bio->bi_error = -EIO;
+ io_bio->bi_status = BLK_STS_IOERR;
bio_endio(io_bio);
/*
* The end io callbacks free our dip, do the final put on io_bio
unlock_extent(&BTRFS_I(inode)->io_tree, file_offset,
file_offset + dio_bio->bi_iter.bi_size - 1);
- dio_bio->bi_error = -EIO;
+ dio_bio->bi_status = BLK_STS_IOERR;
/*
* Releases and cleans up our dio_bio, no need to bio_put()
* nor bio_endio()/bio_io_error() against dio_bio.
*/
- dio_end_io(dio_bio, ret);
+ dio_end_io(dio_bio);
}
if (io_bio)
bio_put(io_bio);
dio_data.overwrite = 1;
inode_unlock(inode);
relock = true;
+ } else if (iocb->ki_flags & IOCB_NOWAIT) {
+ ret = -EAGAIN;
+ goto out;
}
ret = btrfs_delalloc_reserve_space(inode, offset, count);
if (ret)
* this frees the rbio and runs through all the bios in the
* bio_list and calls end_io on them
*/
-static void rbio_orig_end_io(struct btrfs_raid_bio *rbio, int err)
+static void rbio_orig_end_io(struct btrfs_raid_bio *rbio, blk_status_t err)
{
struct bio *cur = bio_list_get(&rbio->bio_list);
struct bio *next;
while (cur) {
next = cur->bi_next;
cur->bi_next = NULL;
- cur->bi_error = err;
+ cur->bi_status = err;
bio_endio(cur);
cur = next;
}
static void raid_write_end_io(struct bio *bio)
{
struct btrfs_raid_bio *rbio = bio->bi_private;
- int err = bio->bi_error;
+ blk_status_t err = bio->bi_status;
int max_errors;
if (err)
max_errors = (rbio->operation == BTRFS_RBIO_PARITY_SCRUB) ?
0 : rbio->bbio->max_errors;
if (atomic_read(&rbio->error) > max_errors)
- err = -EIO;
+ err = BLK_STS_IOERR;
rbio_orig_end_io(rbio, err);
}
* devices or if they are not contiguous
*/
if (last_end == disk_start && stripe->dev->bdev &&
- !last->bi_error &&
+ !last->bi_status &&
last->bi_bdev == stripe->dev->bdev) {
ret = bio_add_page(last, page, PAGE_SIZE, 0);
if (ret == PAGE_SIZE)
{
struct btrfs_raid_bio *rbio = bio->bi_private;
- if (bio->bi_error)
+ if (bio->bi_status)
fail_bio_stripe(rbio, bio);
else
set_bio_pages_uptodate(bio);
* we only read stripe pages off the disk, set them
* up to date if there were no errors
*/
- if (bio->bi_error)
+ if (bio->bi_status)
fail_bio_stripe(rbio, bio);
else
set_bio_pages_uptodate(bio);
{
struct btrfs_raid_bio *rbio = bio->bi_private;
- if (bio->bi_error)
+ if (bio->bi_status)
fail_bio_stripe(rbio, bio);
else
set_bio_pages_uptodate(bio);
struct scrub_ctx *sctx;
struct btrfs_device *dev;
struct bio *bio;
- int err;
+ blk_status_t status;
u64 logical;
u64 physical;
#if SCRUB_PAGES_PER_WR_BIO >= SCRUB_PAGES_PER_RD_BIO
struct scrub_bio_ret {
struct completion event;
- int error;
+ blk_status_t status;
};
static void scrub_bio_wait_endio(struct bio *bio)
{
struct scrub_bio_ret *ret = bio->bi_private;
- ret->error = bio->bi_error;
+ ret->status = bio->bi_status;
complete(&ret->event);
}
int ret;
init_completion(&done.event);
- done.error = 0;
+ done.status = 0;
bio->bi_iter.bi_sector = page->logical >> 9;
bio->bi_private = &done;
bio->bi_end_io = scrub_bio_wait_endio;
return ret;
wait_for_completion(&done.event);
- if (done.error)
+ if (done.status)
return -EIO;
return 0;
bio->bi_bdev = sbio->dev->bdev;
bio->bi_iter.bi_sector = sbio->physical >> 9;
bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
- sbio->err = 0;
+ sbio->status = 0;
} else if (sbio->physical + sbio->page_count * PAGE_SIZE !=
spage->physical_for_dev_replace ||
sbio->logical + sbio->page_count * PAGE_SIZE !=
struct scrub_bio *sbio = bio->bi_private;
struct btrfs_fs_info *fs_info = sbio->dev->fs_info;
- sbio->err = bio->bi_error;
+ sbio->status = bio->bi_status;
sbio->bio = bio;
btrfs_init_work(&sbio->work, btrfs_scrubwrc_helper,
int i;
WARN_ON(sbio->page_count > SCRUB_PAGES_PER_WR_BIO);
- if (sbio->err) {
+ if (sbio->status) {
struct btrfs_dev_replace *dev_replace =
&sbio->sctx->fs_info->dev_replace;
bio->bi_bdev = sbio->dev->bdev;
bio->bi_iter.bi_sector = sbio->physical >> 9;
bio_set_op_attrs(bio, REQ_OP_READ, 0);
- sbio->err = 0;
+ sbio->status = 0;
} else if (sbio->physical + sbio->page_count * PAGE_SIZE !=
spage->physical ||
sbio->logical + sbio->page_count * PAGE_SIZE !=
struct scrub_block *sblock = bio->bi_private;
struct btrfs_fs_info *fs_info = sblock->sctx->fs_info;
- if (bio->bi_error)
+ if (bio->bi_status)
sblock->no_io_error_seen = 0;
bio_put(bio);
struct scrub_bio *sbio = bio->bi_private;
struct btrfs_fs_info *fs_info = sbio->dev->fs_info;
- sbio->err = bio->bi_error;
+ sbio->status = bio->bi_status;
sbio->bio = bio;
btrfs_queue_work(fs_info->scrub_workers, &sbio->work);
int i;
BUG_ON(sbio->page_count > SCRUB_PAGES_PER_RD_BIO);
- if (sbio->err) {
+ if (sbio->status) {
for (i = 0; i < sbio->page_count; i++) {
struct scrub_page *spage = sbio->pagev[i];
struct scrub_parity *sparity = (struct scrub_parity *)bio->bi_private;
struct btrfs_fs_info *fs_info = sparity->sctx->fs_info;
- if (bio->bi_error)
+ if (bio->bi_status)
bitmap_or(sparity->ebitmap, sparity->ebitmap, sparity->dbitmap,
sparity->nsectors);
struct btrfs_bio *bbio = bio->bi_private;
int is_orig_bio = 0;
- if (bio->bi_error) {
+ if (bio->bi_status) {
atomic_inc(&bbio->error);
- if (bio->bi_error == -EIO || bio->bi_error == -EREMOTEIO) {
+ if (bio->bi_status == BLK_STS_IOERR ||
+ bio->bi_status == BLK_STS_TARGET) {
unsigned int stripe_index =
btrfs_io_bio(bio)->stripe_index;
struct btrfs_device *dev;
* beyond the tolerance of the btrfs bio
*/
if (atomic_read(&bbio->error) > bbio->max_errors) {
- bio->bi_error = -EIO;
+ bio->bi_status = BLK_STS_IOERR;
} else {
/*
* this bio is actually up to date, we didn't
* go over the max number of errors
*/
- bio->bi_error = 0;
+ bio->bi_status = 0;
}
btrfs_end_bbio(bbio, bio);
btrfs_io_bio(bio)->mirror_num = bbio->mirror_num;
bio->bi_iter.bi_sector = logical >> 9;
- bio->bi_error = -EIO;
+ bio->bi_status = BLK_STS_IOERR;
btrfs_end_bbio(bbio, bio);
}
}
static int fsync_buffers_list(spinlock_t *lock, struct list_head *list);
static int submit_bh_wbc(int op, int op_flags, struct buffer_head *bh,
- struct writeback_control *wbc);
+ enum rw_hint hint, struct writeback_control *wbc);
#define BH_ENTRY(list) list_entry((list), struct buffer_head, b_assoc_buffers)
do {
struct buffer_head *next = bh->b_this_page;
if (buffer_async_write(bh)) {
- submit_bh_wbc(REQ_OP_WRITE, write_flags, bh, wbc);
+ submit_bh_wbc(REQ_OP_WRITE, write_flags, bh,
+ inode->i_write_hint, wbc);
nr_underway++;
}
bh = next;
struct buffer_head *next = bh->b_this_page;
if (buffer_async_write(bh)) {
clear_buffer_dirty(bh);
- submit_bh_wbc(REQ_OP_WRITE, write_flags, bh, wbc);
+ submit_bh_wbc(REQ_OP_WRITE, write_flags, bh,
+ inode->i_write_hint, wbc);
nr_underway++;
}
bh = next;
if (unlikely(bio_flagged(bio, BIO_QUIET)))
set_bit(BH_Quiet, &bh->b_state);
- bh->b_end_io(bh, !bio->bi_error);
+ bh->b_end_io(bh, !bio->bi_status);
bio_put(bio);
}
}
static int submit_bh_wbc(int op, int op_flags, struct buffer_head *bh,
- struct writeback_control *wbc)
+ enum rw_hint write_hint, struct writeback_control *wbc)
{
struct bio *bio;
bio->bi_iter.bi_sector = bh->b_blocknr * (bh->b_size >> 9);
bio->bi_bdev = bh->b_bdev;
+ bio->bi_write_hint = write_hint;
bio_add_page(bio, bh->b_page, bh->b_size, bh_offset(bh));
BUG_ON(bio->bi_iter.bi_size != bh->b_size);
int submit_bh(int op, int op_flags, struct buffer_head *bh)
{
- return submit_bh_wbc(op, op_flags, bh, NULL);
+ return submit_bh_wbc(op, op_flags, bh, 0, NULL);
}
EXPORT_SYMBOL(submit_bh);
#include <linux/fscache-cache.h>
#include <linux/timer.h>
-#include <linux/wait.h>
+#include <linux/wait_bit.h>
#include <linux/cred.h>
#include <linux/workqueue.h>
#include <linux/security.h>
* backing file read tracking
*/
struct cachefiles_one_read {
- wait_queue_t monitor; /* link into monitored waitqueue */
+ wait_queue_entry_t monitor; /* link into monitored waitqueue */
struct page *back_page; /* backing file page we're waiting for */
struct page *netfs_page; /* netfs page we're going to fill */
struct fscache_retrieval *op; /* retrieval op covering this */
wait_queue_head_t *wq;
signed long timeout = 60 * HZ;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
bool requeue;
/* if the object we're waiting for is queued for processing,
* - we use this to detect read completion of backing pages
* - the caller holds the waitqueue lock
*/
-static int cachefiles_read_waiter(wait_queue_t *wait, unsigned mode,
+static int cachefiles_read_waiter(wait_queue_entry_t *wait, unsigned mode,
int sync, void *_key)
{
struct cachefiles_one_read *monitor =
}
/* remove from the waitqueue */
- list_del(&wait->task_list);
+ list_del(&wait->entry);
/* move onto the action list and queue for FS-Cache thread pool */
ASSERT(monitor->op);
if (!is_sync_kiocb(iocb))
ctx->iocb = iocb;
- if (to->type & ITER_IOVEC)
+ if (to->type == ITER_IOVEC)
ctx->should_dirty = true;
rc = setup_aio_ctx_iter(ctx, to, READ);
#include <linux/pagemap.h>
#include <linux/freezer.h>
#include <linux/sched/signal.h>
+#include <linux/wait_bit.h>
#include <asm/div64.h>
#include "cifsfs.h"
if (!pages) {
pages = vmalloc(max_pages * sizeof(struct page *));
- if (!bv) {
+ if (!pages) {
kvfree(bv);
return -ENOMEM;
}
struct cifs_fid *fid, __u16 search_flags,
struct cifs_search_info *srch_inf)
{
- return CIFSFindFirst(xid, tcon, path, cifs_sb,
- &fid->netfid, search_flags, srch_inf, true);
+ int rc;
+
+ rc = CIFSFindFirst(xid, tcon, path, cifs_sb,
+ &fid->netfid, search_flags, srch_inf, true);
+ if (rc)
+ cifs_dbg(FYI, "find first failed=%d\n", rc);
+ return rc;
}
static int
rc = SMB2_open(xid, &oparms, utf16_path, &oplock, NULL, NULL);
kfree(utf16_path);
if (rc) {
- cifs_dbg(VFS, "open dir failed\n");
+ cifs_dbg(FYI, "open dir failed rc=%d\n", rc);
return rc;
}
rc = SMB2_query_directory(xid, tcon, fid->persistent_fid,
fid->volatile_fid, 0, srch_inf);
if (rc) {
- cifs_dbg(VFS, "query directory failed\n");
+ cifs_dbg(FYI, "query directory failed rc=%d\n", rc);
SMB2_close(xid, tcon, fid->persistent_fid, fid->volatile_fid);
}
return rc;
sg = init_sg(rqst, sign);
if (!sg) {
- cifs_dbg(VFS, "%s: Failed to init sg %d", __func__, rc);
+ cifs_dbg(VFS, "%s: Failed to init sg", __func__);
+ rc = -ENOMEM;
goto free_req;
}
iv = kzalloc(iv_len, GFP_KERNEL);
if (!iv) {
cifs_dbg(VFS, "%s: Failed to alloc IV", __func__);
+ rc = -ENOMEM;
goto free_sg;
}
iv[0] = 3;
pcreatetime = (__u64 *)value;
*pcreatetime = CIFS_I(inode)->createtime;
return sizeof(__u64);
-
- return rc;
}
COMPATIBLE_IOCTL(TIOCCBRK)
COMPATIBLE_IOCTL(TIOCGSID)
COMPATIBLE_IOCTL(TIOCGICOUNT)
-COMPATIBLE_IOCTL(TIOCGPKT)
-COMPATIBLE_IOCTL(TIOCGPTLCK)
COMPATIBLE_IOCTL(TIOCGEXCL)
/* Little t */
COMPATIBLE_IOCTL(TIOCGETD)
COMPATIBLE_IOCTL(TIOCMBIC)
COMPATIBLE_IOCTL(TIOCMBIS)
COMPATIBLE_IOCTL(TIOCMSET)
-COMPATIBLE_IOCTL(TIOCPKT)
COMPATIBLE_IOCTL(TIOCNOTTY)
COMPATIBLE_IOCTL(TIOCSTI)
COMPATIBLE_IOCTL(TIOCOUTQ)
COMPATIBLE_IOCTL(TIOCSPGRP)
COMPATIBLE_IOCTL(TIOCGPGRP)
-COMPATIBLE_IOCTL(TIOCGPTN)
-COMPATIBLE_IOCTL(TIOCSPTLCK)
COMPATIBLE_IOCTL(TIOCSERGETLSR)
-COMPATIBLE_IOCTL(TIOCSIG)
#ifdef TIOCSRS485
COMPATIBLE_IOCTL(TIOCSRS485)
#endif
goto errout;
}
err = submit_bio_wait(bio);
- if ((err == 0) && bio->bi_error)
+ if (err == 0 && bio->bi_status)
err = -EIO;
bio_put(bio);
if (err)
};
struct wait_exceptional_entry_queue {
- wait_queue_t wait;
+ wait_queue_entry_t wait;
struct exceptional_entry_key key;
};
return wait_table + hash;
}
-static int wake_exceptional_entry_func(wait_queue_t *wait, unsigned int mode,
+static int wake_exceptional_entry_func(wait_queue_entry_t *wait, unsigned int mode,
int sync, void *keyp)
{
struct exceptional_entry_key *key = keyp;
if (ret < 0)
goto out;
}
+ start_index = indices[pvec.nr - 1] + 1;
}
out:
put_dax(dax_dev);
* 2 as published by the Free Software Foundation.
*
* debugfs is for people to use instead of /proc or /sys.
- * See Documentation/DocBook/filesystems for more details.
+ * See Documentation/filesystems/ for more details.
*
*/
* 2 as published by the Free Software Foundation.
*
* debugfs is for people to use instead of /proc or /sys.
- * See Documentation/DocBook/kernel-api for more details.
+ * See ./Documentation/core-api/kernel-api.rst for more details.
*
*/
dio_complete(dio, 0, true);
}
-static int dio_bio_complete(struct dio *dio, struct bio *bio);
+static blk_status_t dio_bio_complete(struct dio *dio, struct bio *bio);
/*
* Asynchronous IO callback.
/**
* dio_end_io - handle the end io action for the given bio
* @bio: The direct io bio thats being completed
- * @error: Error if there was one
*
* This is meant to be called by any filesystem that uses their own dio_submit_t
* so that the DIO specific endio actions are dealt with after the filesystem
* has done it's completion work.
*/
-void dio_end_io(struct bio *bio, int error)
+void dio_end_io(struct bio *bio)
{
struct dio *dio = bio->bi_private;
else
bio->bi_end_io = dio_bio_end_io;
+ bio->bi_write_hint = dio->iocb->ki_hint;
+
sdio->bio = bio;
sdio->logical_offset_in_bio = sdio->cur_page_fs_offset;
}
/*
* Process one completed BIO. No locks are held.
*/
-static int dio_bio_complete(struct dio *dio, struct bio *bio)
+static blk_status_t dio_bio_complete(struct dio *dio, struct bio *bio)
{
struct bio_vec *bvec;
unsigned i;
- int err;
+ blk_status_t err = bio->bi_status;
- if (bio->bi_error)
- dio->io_error = -EIO;
+ if (err) {
+ if (err == BLK_STS_AGAIN && (bio->bi_opf & REQ_NOWAIT))
+ dio->io_error = -EAGAIN;
+ else
+ dio->io_error = -EIO;
+ }
if (dio->is_async && dio->op == REQ_OP_READ && dio->should_dirty) {
- err = bio->bi_error;
bio_check_pages_dirty(bio); /* transfers ownership */
} else {
bio_for_each_segment_all(bvec, bio, i) {
set_page_dirty_lock(page);
put_page(page);
}
- err = bio->bi_error;
bio_put(bio);
}
return err;
bio = dio->bio_list;
dio->bio_list = bio->bi_private;
spin_unlock_irqrestore(&dio->bio_lock, flags);
- ret2 = dio_bio_complete(dio, bio);
+ ret2 = blk_status_to_errno(dio_bio_complete(dio, bio));
if (ret == 0)
ret = ret2;
}
if (iov_iter_rw(iter) == WRITE) {
dio->op = REQ_OP_WRITE;
dio->op_flags = REQ_SYNC | REQ_IDLE;
+ if (iocb->ki_flags & IOCB_NOWAIT)
+ dio->op_flags |= REQ_NOWAIT;
} else {
dio->op = REQ_OP_READ;
}
* This is used to atomically remove a wait queue entry from the eventfd wait
* queue head, and read/reset the counter value.
*/
-int eventfd_ctx_remove_wait_queue(struct eventfd_ctx *ctx, wait_queue_t *wait,
+int eventfd_ctx_remove_wait_queue(struct eventfd_ctx *ctx, wait_queue_entry_t *wait,
__u64 *cnt)
{
unsigned long flags;
*
* Returns %0 if successful, or the following error codes:
*
- * -EAGAIN : The operation would have blocked but @no_wait was non-zero.
- * -ERESTARTSYS : A signal interrupted the wait operation.
+ * - -EAGAIN : The operation would have blocked but @no_wait was non-zero.
+ * - -ERESTARTSYS : A signal interrupted the wait operation.
*
* If @no_wait is zero, the function might sleep until the eventfd internal
* counter becomes greater than zero.
* Wait queue item that will be linked to the target file wait
* queue head.
*/
- wait_queue_t wait;
+ wait_queue_entry_t wait;
/* The wait queue head that linked the "wait" wait queue item */
wait_queue_head_t *whead;
return !list_empty(p);
}
-static inline struct eppoll_entry *ep_pwq_from_wait(wait_queue_t *p)
+static inline struct eppoll_entry *ep_pwq_from_wait(wait_queue_entry_t *p)
{
return container_of(p, struct eppoll_entry, wait);
}
/* Get the "struct epitem" from a wait queue pointer */
-static inline struct epitem *ep_item_from_wait(wait_queue_t *p)
+static inline struct epitem *ep_item_from_wait(wait_queue_entry_t *p)
{
return container_of(p, struct eppoll_entry, wait)->base;
}
* mechanism. It is called by the stored file descriptors when they
* have events to report.
*/
-static int ep_poll_callback(wait_queue_t *wait, unsigned mode, int sync, void *key)
+static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, void *key)
{
int pwake = 0;
unsigned long flags;
* can't use __remove_wait_queue(). whead->lock is held by
* the caller.
*/
- list_del_init(&wait->task_list);
+ list_del_init(&wait->entry);
}
spin_lock_irqsave(&ep->lock, flags);
int res = 0, eavail, timed_out = 0;
unsigned long flags;
u64 slack = 0;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
ktime_t expires, *to = NULL;
if (timeout > 0) {
if (write) {
unsigned long size = bprm->vma->vm_end - bprm->vma->vm_start;
+ unsigned long ptr_size;
struct rlimit *rlim;
+ /*
+ * Since the stack will hold pointers to the strings, we
+ * must account for them as well.
+ *
+ * The size calculation is the entire vma while each arg page is
+ * built, so each time we get here it's calculating how far it
+ * is currently (rather than each call being just the newly
+ * added size from the arg page). As a result, we need to
+ * always add the entire size of the pointers, so that on the
+ * last call to get_arg_page() we'll actually have the entire
+ * correct size.
+ */
+ ptr_size = (bprm->argc + bprm->envc) * sizeof(void *);
+ if (ptr_size > ULONG_MAX - size)
+ goto fail;
+ size += ptr_size;
+
acct_arg_size(bprm, size / PAGE_SIZE);
/*
* to work from.
*/
rlim = current->signal->rlim;
- if (size > ACCESS_ONCE(rlim[RLIMIT_STACK].rlim_cur) / 4) {
- put_page(page);
- return NULL;
- }
+ if (size > READ_ONCE(rlim[RLIMIT_STACK].rlim_cur) / 4)
+ goto fail;
}
return page;
+
+fail:
+ put_page(page);
+ return NULL;
}
static void put_arg_page(struct page *page)
struct inode *inode = file_inode(iocb->ki_filp);
ssize_t ret;
- inode_lock_shared(inode);
+ if (!inode_trylock_shared(inode)) {
+ if (iocb->ki_flags & IOCB_NOWAIT)
+ return -EAGAIN;
+ inode_lock_shared(inode);
+ }
/*
* Recheck under inode lock - at this point we are sure it cannot
* change anymore
struct inode *inode = file_inode(iocb->ki_filp);
ssize_t ret;
- inode_lock(inode);
+ if (!inode_trylock(inode)) {
+ if (iocb->ki_flags & IOCB_NOWAIT)
+ return -EAGAIN;
+ inode_lock(inode);
+ }
ret = ext4_write_checks(iocb, from);
if (ret <= 0)
goto out;
return ext4_dax_write_iter(iocb, from);
#endif
- inode_lock(inode);
+ if (!inode_trylock(inode)) {
+ if (iocb->ki_flags & IOCB_NOWAIT)
+ return -EAGAIN;
+ inode_lock(inode);
+ }
+
ret = ext4_write_checks(iocb, from);
if (ret <= 0)
goto out;
iocb->private = &overwrite;
/* Check whether we do a DIO overwrite or not */
- if (o_direct && ext4_should_dioread_nolock(inode) && !unaligned_aio &&
- ext4_overwrite_io(inode, iocb->ki_pos, iov_iter_count(from)))
- overwrite = 1;
+ if (o_direct && !unaligned_aio) {
+ if (ext4_overwrite_io(inode, iocb->ki_pos, iov_iter_count(from))) {
+ if (ext4_should_dioread_nolock(inode))
+ overwrite = 1;
+ } else if (iocb->ki_flags & IOCB_NOWAIT) {
+ ret = -EAGAIN;
+ goto out;
+ }
+ }
ret = __generic_file_write_iter(iocb, from);
inode_unlock(inode);
if (ret < 0)
return ret;
}
+
+ /* Set the flags to support nowait AIO */
+ filp->f_mode |= FMODE_AIO_NOWAIT;
+
return dquot_file_open(inode, filp);
}
}
#endif
- if (bio->bi_error) {
+ if (bio->bi_status) {
SetPageError(page);
mapping_set_error(page->mapping, -EIO);
}
continue;
}
clear_buffer_async_write(bh);
- if (bio->bi_error)
+ if (bio->bi_status)
buffer_io_error(bh);
} while ((bh = bh->b_this_page) != head);
bit_spin_unlock(BH_Uptodate_Lock, &head->b_state);
bdevname(bio->bi_bdev, b),
(long long) bio->bi_iter.bi_sector,
(unsigned) bio_sectors(bio),
- bio->bi_error)) {
+ bio->bi_status)) {
ext4_finish_bio(bio);
bio_put(bio);
return;
}
bio->bi_end_io = NULL;
- if (bio->bi_error) {
+ if (bio->bi_status) {
struct inode *inode = io_end->inode;
ext4_warning(inode->i_sb, "I/O error %d writing to inode %lu "
"(offset %llu size %ld starting block %llu)",
- bio->bi_error, inode->i_ino,
+ bio->bi_status, inode->i_ino,
(unsigned long long) io_end->offset,
(long) io_end->size,
(unsigned long long)
bi_sector >> (inode->i_blkbits - 9));
- mapping_set_error(inode->i_mapping, bio->bi_error);
+ mapping_set_error(inode->i_mapping,
+ blk_status_to_errno(bio->bi_status));
}
if (io_end->flag & EXT4_IO_END_UNWRITTEN) {
if (bio) {
int io_op_flags = io->io_wbc->sync_mode == WB_SYNC_ALL ?
REQ_SYNC : 0;
+ io->io_bio->bi_write_hint = io->io_end->inode->i_write_hint;
bio_set_op_attrs(io->io_bio, REQ_OP_WRITE, io_op_flags);
submit_bio(io->io_bio);
}
ret = io_submit_init_bio(io, bh);
if (ret)
return ret;
+ io->io_bio->bi_write_hint = inode->i_write_hint;
}
ret = bio_add_page(io->io_bio, page, bh->b_size, bh_offset(bh));
if (ret != bh->b_size)
int i;
if (ext4_bio_encrypted(bio)) {
- if (bio->bi_error) {
+ if (bio->bi_status) {
fscrypt_release_ctx(bio->bi_private);
} else {
fscrypt_decrypt_bio_pages(bio->bi_private, bio);
bio_for_each_segment_all(bv, bio, i) {
struct page *page = bv->bv_page;
- if (!bio->bi_error) {
+ if (!bio->bi_status) {
SetPageUptodate(page);
} else {
ClearPageUptodate(page);
sb->s_qcop = &ext4_qctl_operations;
sb->s_quota_types = QTYPE_MASK_USR | QTYPE_MASK_GRP | QTYPE_MASK_PRJ;
#endif
- memcpy(sb->s_uuid, es->s_uuid, sizeof(es->s_uuid));
+ memcpy(&sb->s_uuid, es->s_uuid, sizeof(es->s_uuid));
INIT_LIST_HEAD(&sbi->s_orphan); /* unlinked but open files */
mutex_init(&sbi->s_orphan_lock);
#ifdef CONFIG_F2FS_FAULT_INJECTION
if (time_to_inject(F2FS_P_SB(bio->bi_io_vec->bv_page), FAULT_IO)) {
f2fs_show_injection_info(FAULT_IO);
- bio->bi_error = -EIO;
+ bio->bi_status = BLK_STS_IOERR;
}
#endif
if (f2fs_bio_encrypted(bio)) {
- if (bio->bi_error) {
+ if (bio->bi_status) {
fscrypt_release_ctx(bio->bi_private);
} else {
fscrypt_decrypt_bio_pages(bio->bi_private, bio);
bio_for_each_segment_all(bvec, bio, i) {
struct page *page = bvec->bv_page;
- if (!bio->bi_error) {
+ if (!bio->bi_status) {
if (!PageUptodate(page))
SetPageUptodate(page);
} else {
unlock_page(page);
mempool_free(page, sbi->write_io_dummy);
- if (unlikely(bio->bi_error))
+ if (unlikely(bio->bi_status))
f2fs_stop_checkpoint(sbi, true);
continue;
}
fscrypt_pullback_bio_page(&page, true);
- if (unlikely(bio->bi_error)) {
+ if (unlikely(bio->bi_status)) {
mapping_set_error(page->mapping, -EIO);
f2fs_stop_checkpoint(sbi, true);
}
{
struct discard_cmd *dc = (struct discard_cmd *)bio->bi_private;
- dc->error = bio->bi_error;
+ dc->error = blk_status_to_errno(bio->bi_status);
dc->state = D_DONE;
complete(&dc->wait);
bio_put(bio);
sb->s_time_gran = 1;
sb->s_flags = (sb->s_flags & ~MS_POSIXACL) |
(test_opt(sbi, POSIX_ACL) ? MS_POSIXACL : 0);
- memcpy(sb->s_uuid, raw_super->uuid, sizeof(raw_super->uuid));
+ memcpy(&sb->s_uuid, raw_super->uuid, sizeof(raw_super->uuid));
/* init f2fs-specific super block info */
sbi->valid_super_block = valid_super_block;
}
#endif
+static bool rw_hint_valid(enum rw_hint hint)
+{
+ switch (hint) {
+ case RWF_WRITE_LIFE_NOT_SET:
+ case RWH_WRITE_LIFE_NONE:
+ case RWH_WRITE_LIFE_SHORT:
+ case RWH_WRITE_LIFE_MEDIUM:
+ case RWH_WRITE_LIFE_LONG:
+ case RWH_WRITE_LIFE_EXTREME:
+ return true;
+ default:
+ return false;
+ }
+}
+
+static long fcntl_rw_hint(struct file *file, unsigned int cmd,
+ unsigned long arg)
+{
+ struct inode *inode = file_inode(file);
+ u64 *argp = (u64 __user *)arg;
+ enum rw_hint hint;
+ u64 h;
+
+ switch (cmd) {
+ case F_GET_FILE_RW_HINT:
+ h = file_write_hint(file);
+ if (copy_to_user(argp, &h, sizeof(*argp)))
+ return -EFAULT;
+ return 0;
+ case F_SET_FILE_RW_HINT:
+ if (copy_from_user(&h, argp, sizeof(h)))
+ return -EFAULT;
+ hint = (enum rw_hint) h;
+ if (!rw_hint_valid(hint))
+ return -EINVAL;
+
+ spin_lock(&file->f_lock);
+ file->f_write_hint = hint;
+ spin_unlock(&file->f_lock);
+ return 0;
+ case F_GET_RW_HINT:
+ h = inode->i_write_hint;
+ if (copy_to_user(argp, &h, sizeof(*argp)))
+ return -EFAULT;
+ return 0;
+ case F_SET_RW_HINT:
+ if (copy_from_user(&h, argp, sizeof(h)))
+ return -EFAULT;
+ hint = (enum rw_hint) h;
+ if (!rw_hint_valid(hint))
+ return -EINVAL;
+
+ inode_lock(inode);
+ inode->i_write_hint = hint;
+ inode_unlock(inode);
+ return 0;
+ default:
+ return -EINVAL;
+ }
+}
+
static long do_fcntl(int fd, unsigned int cmd, unsigned long arg,
struct file *filp)
{
case F_GET_SEALS:
err = shmem_fcntl(filp, cmd, arg);
break;
+ case F_GET_RW_HINT:
+ case F_SET_RW_HINT:
+ case F_GET_FILE_RW_HINT:
+ case F_SET_FILE_RW_HINT:
+ err = fcntl_rw_hint(filp, cmd, arg);
+ break;
default:
break;
}
}
/**
- * __mark_inode_dirty - internal function
- * @inode: inode to mark
- * @flags: what kind of dirty (i.e. I_DIRTY_SYNC)
- * Mark an inode as dirty. Callers should use mark_inode_dirty or
- * mark_inode_dirty_sync.
+ * __mark_inode_dirty - internal function
+ *
+ * @inode: inode to mark
+ * @flags: what kind of dirty (i.e. I_DIRTY_SYNC)
+ *
+ * Mark an inode as dirty. Callers should use mark_inode_dirty or
+ * mark_inode_dirty_sync.
*
* Put the inode on the super block's dirty list.
*
void pin_kill(struct fs_pin *p)
{
- wait_queue_t wait;
+ wait_queue_entry_t wait;
if (!p) {
rcu_read_unlock();
rcu_read_unlock();
schedule();
rcu_read_lock();
- if (likely(list_empty(&wait.task_list)))
+ if (likely(list_empty(&wait.entry)))
break;
/* OK, we know p couldn't have been freed yet */
spin_lock_irq(&p->wait.lock);
atomic_t sd_log_in_flight;
struct bio *sd_log_bio;
wait_queue_head_t sd_log_flush_wait;
- int sd_log_error;
atomic_t sd_reserving_log;
wait_queue_head_t sd_reserving_log_wait;
*/
static void gfs2_end_log_write_bh(struct gfs2_sbd *sdp, struct bio_vec *bvec,
- int error)
+ blk_status_t error)
{
struct buffer_head *bh, *next;
struct page *page = bvec->bv_page;
struct page *page;
int i;
- if (bio->bi_error) {
- sdp->sd_log_error = bio->bi_error;
- fs_err(sdp, "Error %d writing to log\n", bio->bi_error);
- }
+ if (bio->bi_status)
+ fs_err(sdp, "Error %d writing to log\n", bio->bi_status);
bio_for_each_segment_all(bvec, bio, i) {
page = bvec->bv_page;
if (page_has_buffers(page))
- gfs2_end_log_write_bh(sdp, bvec, bio->bi_error);
+ gfs2_end_log_write_bh(sdp, bvec, bio->bi_status);
else
mempool_free(page, gfs2_page_pool);
}
do {
struct buffer_head *next = bh->b_this_page;
len -= bh->b_size;
- bh->b_end_io(bh, !bio->bi_error);
+ bh->b_end_io(bh, !bio->bi_status);
bh = next;
} while (bh && len);
}
{
struct page *page = bio->bi_private;
- if (!bio->bi_error)
+ if (!bio->bi_status)
SetPageUptodate(page);
else
- pr_warn("error %d reading superblock\n", bio->bi_error);
+ pr_warn("error %d reading superblock\n", bio->bi_status);
unlock_page(page);
}
memcpy(sb->sb_lockproto, str->sb_lockproto, GFS2_LOCKNAME_LEN);
memcpy(sb->sb_locktable, str->sb_locktable, GFS2_LOCKNAME_LEN);
- memcpy(s->s_uuid, str->sb_uuid, 16);
+ memcpy(&s->s_uuid, str->sb_uuid, 16);
}
/**
return snprintf(buf, PAGE_SIZE, "%s\n", sdp->sd_fsname);
}
-static int gfs2_uuid_valid(const u8 *uuid)
-{
- int i;
-
- for (i = 0; i < 16; i++) {
- if (uuid[i])
- return 1;
- }
- return 0;
-}
-
static ssize_t uuid_show(struct gfs2_sbd *sdp, char *buf)
{
struct super_block *s = sdp->sd_vfs;
- const u8 *uuid = s->s_uuid;
+
buf[0] = '\0';
- if (!gfs2_uuid_valid(uuid))
+ if (uuid_is_null(&s->s_uuid))
return 0;
- return snprintf(buf, PAGE_SIZE, "%pUB\n", uuid);
+ return snprintf(buf, PAGE_SIZE, "%pUB\n", &s->s_uuid);
}
static ssize_t freeze_show(struct gfs2_sbd *sdp, char *buf)
{
struct gfs2_sbd *sdp = container_of(kobj, struct gfs2_sbd, sd_kobj);
struct super_block *s = sdp->sd_vfs;
- const u8 *uuid = s->s_uuid;
add_uevent_var(env, "LOCKTABLE=%s", sdp->sd_table_name);
add_uevent_var(env, "LOCKPROTO=%s", sdp->sd_proto_name);
if (!test_bit(SDF_NOJOURNALID, &sdp->sd_flags))
add_uevent_var(env, "JOURNALID=%d", sdp->sd_lockstruct.ls_jid);
- if (gfs2_uuid_valid(uuid))
- add_uevent_var(env, "UUID=%pUB", uuid);
+ if (!uuid_is_null(&s->s_uuid))
+ add_uevent_var(env, "UUID=%pUB", &s->s_uuid);
return 0;
}
i_gid_write(inode, 0);
atomic_set(&inode->i_writecount, 0);
inode->i_size = 0;
+ inode->i_write_hint = WRITE_LIFE_NOT_SET;
inode->i_blocks = 0;
inode->i_bytes = 0;
inode->i_generation = 0;
wait_queue_head_t *wq;
DEFINE_WAIT_BIT(wait, &inode->i_state, __I_NEW);
wq = bit_waitqueue(&inode->i_state, __I_NEW);
- prepare_to_wait(wq, &wait.wait, TASK_UNINTERRUPTIBLE);
+ prepare_to_wait(wq, &wait.wq_entry, TASK_UNINTERRUPTIBLE);
spin_unlock(&inode->i_lock);
spin_unlock(&inode_hash_lock);
schedule();
- finish_wait(wq, &wait.wait);
+ finish_wait(wq, &wait.wq_entry);
spin_lock(&inode_hash_lock);
}
DEFINE_WAIT_BIT(q, &inode->i_state, __I_DIO_WAKEUP);
do {
- prepare_to_wait(wq, &q.wait, TASK_UNINTERRUPTIBLE);
+ prepare_to_wait(wq, &q.wq_entry, TASK_UNINTERRUPTIBLE);
if (atomic_read(&inode->i_dio_count))
schedule();
} while (atomic_read(&inode->i_dio_count));
- finish_wait(wq, &q.wait);
+ finish_wait(wq, &q.wq_entry);
}
/**
struct iomap_dio *dio = bio->bi_private;
bool should_dirty = (dio->flags & IOMAP_DIO_DIRTY);
- if (bio->bi_error)
- iomap_dio_set_error(dio, bio->bi_error);
+ if (bio->bi_status)
+ iomap_dio_set_error(dio, blk_status_to_errno(bio->bi_status));
if (atomic_dec_and_test(&dio->ref)) {
if (is_sync_kiocb(dio->iocb)) {
bio->bi_bdev = iomap->bdev;
bio->bi_iter.bi_sector =
iomap->blkno + ((pos - iomap->offset) >> 9);
+ bio->bi_write_hint = dio->iocb->ki_hint;
bio->bi_private = dio;
bio->bi_end_io = iomap_dio_bio_end_io;
flags |= IOMAP_WRITE;
}
+ if (iocb->ki_flags & IOCB_NOWAIT) {
+ if (filemap_range_has_page(mapping, start, end)) {
+ ret = -EAGAIN;
+ goto out_free_dio;
+ }
+ flags |= IOMAP_NOWAIT;
+ }
+
ret = filemap_write_and_wait_range(mapping, start, end);
if (ret)
goto out_free_dio;
wait_queue_head_t *wq;
DEFINE_WAIT_BIT(wait, &jinode->i_flags, __JI_COMMIT_RUNNING);
wq = bit_waitqueue(&jinode->i_flags, __JI_COMMIT_RUNNING);
- prepare_to_wait(wq, &wait.wait, TASK_UNINTERRUPTIBLE);
+ prepare_to_wait(wq, &wait.wq_entry, TASK_UNINTERRUPTIBLE);
spin_unlock(&journal->j_list_lock);
schedule();
- finish_wait(wq, &wait.wait);
+ finish_wait(wq, &wait.wq_entry);
goto restart;
}
return handle;
}
-/**
- * handle_t *jbd2_journal_start() - Obtain a new handle.
- * @journal: Journal to start transaction on.
- * @nblocks: number of block buffer we might modify
- *
- * We make sure that the transaction can guarantee at least nblocks of
- * modified buffers in the log. We block until the log can guarantee
- * that much space. Additionally, if rsv_blocks > 0, we also create another
- * handle with rsv_blocks reserved blocks in the journal. This handle is
- * is stored in h_rsv_handle. It is not attached to any particular transaction
- * and thus doesn't block transaction commit. If the caller uses this reserved
- * handle, it has to set h_rsv_handle to NULL as otherwise jbd2_journal_stop()
- * on the parent handle will dispose the reserved one. Reserved handle has to
- * be converted to a normal handle using jbd2_journal_start_reserved() before
- * it can be used.
- *
- * Return a pointer to a newly allocated handle, or an ERR_PTR() value
- * on failure.
- */
handle_t *jbd2__journal_start(journal_t *journal, int nblocks, int rsv_blocks,
gfp_t gfp_mask, unsigned int type,
unsigned int line_no)
EXPORT_SYMBOL(jbd2__journal_start);
+/**
+ * handle_t *jbd2_journal_start() - Obtain a new handle.
+ * @journal: Journal to start transaction on.
+ * @nblocks: number of block buffer we might modify
+ *
+ * We make sure that the transaction can guarantee at least nblocks of
+ * modified buffers in the log. We block until the log can guarantee
+ * that much space. Additionally, if rsv_blocks > 0, we also create another
+ * handle with rsv_blocks reserved blocks in the journal. This handle is
+ * is stored in h_rsv_handle. It is not attached to any particular transaction
+ * and thus doesn't block transaction commit. If the caller uses this reserved
+ * handle, it has to set h_rsv_handle to NULL as otherwise jbd2_journal_stop()
+ * on the parent handle will dispose the reserved one. Reserved handle has to
+ * be converted to a normal handle using jbd2_journal_start_reserved() before
+ * it can be used.
+ *
+ * Return a pointer to a newly allocated handle, or an ERR_PTR() value
+ * on failure.
+ */
handle_t *jbd2_journal_start(journal_t *journal, int nblocks)
{
return jbd2__journal_start(journal, nblocks, 0, GFP_NOFS, 0, 0);
* @handle: transaction to add buffer modifications to
* @bh: bh to be used for metadata writes
*
- * Returns an error code or 0 on success.
+ * Returns: error code or 0 on success.
*
* In full data journalling mode the buffer may be of type BJ_AsyncData,
- * because we're write()ing a buffer which is also part of a shared mapping.
+ * because we're ``write()ing`` a buffer which is also part of a shared mapping.
*/
int jbd2_journal_get_write_access(handle_t *handle, struct buffer_head *bh)
bp->l_flag |= lbmDONE;
- if (bio->bi_error) {
+ if (bio->bi_status) {
bp->l_flag |= lbmERROR;
jfs_err("lbmIODone: I/O error in JFS log");
{
struct page *page = bio->bi_private;
- if (bio->bi_error) {
+ if (bio->bi_status) {
printk(KERN_ERR "metapage_read_end_io: I/O error\n");
SetPageError(page);
}
BUG_ON(!PagePrivate(page));
- if (bio->bi_error) {
+ if (bio->bi_status) {
printk(KERN_ERR "metapage_write_end_io: I/O error\n");
SetPageError(page);
}
bio_for_each_segment_all(bv, bio, i) {
struct page *page = bv->bv_page;
- page_endio(page, op_is_write(bio_op(bio)), bio->bi_error);
+ page_endio(page, op_is_write(bio_op(bio)),
+ blk_status_to_errno(bio->bi_status));
}
bio_put(bio);
*
* So an mpage read of the first 16 blocks of an ext2 file will cause I/O to be
* submitted in the following order:
+ *
* 12 0 1 2 3 4 5 6 7 8 9 10 11 13 14 15 16
*
* because the indirect block has to be read to get the mappings of blocks
goto confused;
wbc_init_bio(wbc, bio);
+ bio->bi_write_hint = inode->i_write_hint;
}
/*
* The worst of all namespace operations - renaming directory. "Perverted"
* doesn't even start to describe it. Somebody in UCB had a heck of a trip...
* Problems:
+ *
* a) we can get into loop creation.
* b) race potential - two innocent renames can create a loop together.
* That's where 4.4 screws up. Current fix: serialization on
{
struct parallel_io *par = bio->bi_private;
- if (bio->bi_error) {
+ if (bio->bi_status) {
struct nfs_pgio_header *header = par->data;
if (!header->pnfs_error)
struct parallel_io *par = bio->bi_private;
struct nfs_pgio_header *header = par->data;
- if (bio->bi_error) {
+ if (bio->bi_status) {
if (!header->pnfs_error)
header->pnfs_error = -EIO;
pnfs_set_lo_fail(header->lseg);
* A single slot, so highest used slotid is either 0 or -1
*/
nfs4_free_slot(tbl, slot);
- nfs4_slot_tbl_drain_complete(tbl);
spin_unlock(&tbl->slot_tbl_lock);
}
}
EXPORT_SYMBOL_GPL(nfs_link);
-static void
-nfs_complete_rename(struct rpc_task *task, struct nfs_renamedata *data)
-{
- struct dentry *old_dentry = data->old_dentry;
- struct dentry *new_dentry = data->new_dentry;
- struct inode *old_inode = d_inode(old_dentry);
- struct inode *new_inode = d_inode(new_dentry);
-
- nfs_mark_for_revalidate(old_inode);
-
- switch (task->tk_status) {
- case 0:
- if (new_inode != NULL)
- nfs_drop_nlink(new_inode);
- d_move(old_dentry, new_dentry);
- nfs_set_verifier(new_dentry,
- nfs_save_change_attribute(data->new_dir));
- break;
- case -ENOENT:
- nfs_dentry_handle_enoent(old_dentry);
- }
-}
-
/*
* RENAME
* FIXME: Some nfsds, like the Linux user space nfsd, may generate a
{
struct inode *old_inode = d_inode(old_dentry);
struct inode *new_inode = d_inode(new_dentry);
- struct dentry *dentry = NULL;
+ struct dentry *dentry = NULL, *rehash = NULL;
struct rpc_task *task;
int error = -EBUSY;
* To prevent any new references to the target during the
* rename, we unhash the dentry in advance.
*/
- if (!d_unhashed(new_dentry))
+ if (!d_unhashed(new_dentry)) {
d_drop(new_dentry);
+ rehash = new_dentry;
+ }
if (d_count(new_dentry) > 2) {
int err;
goto out;
new_dentry = dentry;
+ rehash = NULL;
new_inode = NULL;
}
}
if (new_inode != NULL)
NFS_PROTO(new_inode)->return_delegation(new_inode);
- task = nfs_async_rename(old_dir, new_dir, old_dentry, new_dentry,
- nfs_complete_rename);
+ task = nfs_async_rename(old_dir, new_dir, old_dentry, new_dentry, NULL);
if (IS_ERR(task)) {
error = PTR_ERR(task);
goto out;
if (error == 0)
error = task->tk_status;
rpc_put_task(task);
+ nfs_mark_for_revalidate(old_inode);
out:
+ if (rehash)
+ d_rehash(rehash);
trace_nfs_rename_exit(old_dir, old_dentry,
new_dir, new_dentry, error);
+ if (!error) {
+ if (new_inode != NULL)
+ nfs_drop_nlink(new_inode);
+ /*
+ * The d_move() should be here instead of in an async RPC completion
+ * handler because we need the proper locks to move the dentry. If
+ * we're interrupted by a signal, the async RPC completion handler
+ * should mark the directories for revalidation.
+ */
+ d_move(old_dentry, new_dentry);
+ nfs_set_verifier(new_dentry,
+ nfs_save_change_attribute(new_dir));
+ } else if (error == -ENOENT)
+ nfs_dentry_handle_enoent(old_dentry);
+
/* new dentry created? */
if (dentry)
dput(dentry);
#include <linux/security.h>
#include <linux/crc32.h>
#include <linux/nfs_page.h>
+#include <linux/wait_bit.h>
#define NFS_MS_MASK (MS_RDONLY|MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_SYNCHRONOUS)
/* Except MODE, it seems harmless of setting twice. */
if (opendata->o_arg.createmode != NFS4_CREATE_EXCLUSIVE &&
- attrset[1] & FATTR4_WORD1_MODE)
+ (attrset[1] & FATTR4_WORD1_MODE ||
+ attrset[2] & FATTR4_WORD2_MODE_UMASK))
sattr->ia_valid &= ~ATTR_MODE;
if (attrset[2] & FATTR4_WORD2_SECURITY_LABEL)
};
static int
-nfs4_wake_lock_waiter(wait_queue_t *wait, unsigned int mode, int flags, void *key)
+nfs4_wake_lock_waiter(wait_queue_entry_t *wait, unsigned int mode, int flags, void *key)
{
int ret;
struct cb_notify_lock_args *cbnl = key;
.inode = state->inode,
.owner = &owner,
.notified = false };
- wait_queue_t wait;
+ wait_queue_entry_t wait;
/* Don't bother with waitqueue if we don't expect a callback */
if (!test_bit(NFS_STATE_MAY_NOTIFY_LOCK, &state->flags))
size_t max_pages = max_response_pages(server);
dprintk("--> %s\n", __func__);
+ nfs4_sequence_free_slot(&lgp->res.seq_res);
nfs4_free_pages(lgp->args.layout.pages, max_pages);
pnfs_put_layout_hdr(NFS_I(inode)->layout);
put_nfs_open_context(lgp->args.ctx);
/* if layoutp->len is 0, nfs4_layoutget_prepare called rpc_exit */
if (status == 0 && lgp->res.layoutp->len)
lseg = pnfs_layout_process(lgp);
- nfs4_sequence_free_slot(&lgp->res.seq_res);
rpc_put_task(task);
dprintk("<-- %s status=%d\n", __func__, status);
if (status)
put_rpccred(cred);
switch (status) {
case 0:
+ case -EINTR:
+ case -ERESTARTSYS:
break;
case -ETIMEDOUT:
if (clnt->cl_softrtry)
u8 *buf, *d, type, assoc;
int error;
+ if (WARN_ON_ONCE(!blk_queue_scsi_passthrough(q)))
+ return -EINVAL;
+
buf = kzalloc(bufflen, GFP_KERNEL);
if (!buf)
return -ENOMEM;
goto out_free_buf;
}
req = scsi_req(rq);
- scsi_req_init(rq);
error = blk_rq_map_kern(q, rq, buf, bufflen, GFP_KERNEL);
if (error)
#endif
static inline int
-uuid_parse(char **mesg, char *buf, unsigned char **puuid)
+nfsd_uuid_parse(char **mesg, char *buf, unsigned char **puuid)
{
int len;
if (strcmp(buf, "fsloc") == 0)
err = fsloc_parse(&mesg, buf, &exp.ex_fslocs);
else if (strcmp(buf, "uuid") == 0)
- err = uuid_parse(&mesg, buf, &exp.ex_uuid);
+ err = nfsd_uuid_parse(&mesg, buf, &exp.ex_uuid);
else if (strcmp(buf, "secinfo") == 0)
err = secinfo_parse(&mesg, buf, &exp);
else
{
struct nilfs_segment_buffer *segbuf = bio->bi_private;
- if (bio->bi_error)
+ if (bio->bi_status)
atomic_inc(&segbuf->sb_err);
bio_put(bio);
}
struct nilfs_segctor_wait_request {
- wait_queue_t wq;
+ wait_queue_entry_t wq;
__u32 seq;
int err;
atomic_t done;
unsigned long flags;
spin_lock_irqsave(&sci->sc_wait_request.lock, flags);
- list_for_each_entry_safe(wrq, n, &sci->sc_wait_request.task_list,
- wq.task_list) {
+ list_for_each_entry_safe(wrq, n, &sci->sc_wait_request.head, wq.entry) {
if (!atomic_read(&wrq->done) &&
nilfs_cnt32_ge(sci->sc_seq_done, wrq->seq)) {
wrq->err = err;
{
struct o2hb_bio_wait_ctxt *wc = bio->bi_private;
- if (bio->bi_error) {
- mlog(ML_ERROR, "IO Error %d\n", bio->bi_error);
- wc->wc_error = bio->bi_error;
+ if (bio->bi_status) {
+ mlog(ML_ERROR, "IO Error %d\n", bio->bi_status);
+ wc->wc_error = blk_status_to_errno(bio->bi_status);
}
o2hb_bio_wait_dec(wc, 1);
struct ocfs2_lock_res *lockres;
lockres = &OCFS2_I(inode)->ip_inode_lockres;
+ /* had_lock means that the currect process already takes the cluster
+ * lock previously. If had_lock is 1, we have nothing to do here, and
+ * it will get unlocked where we got the lock.
+ */
if (!had_lock) {
ocfs2_remove_holder(lockres, oh);
ocfs2_inode_unlock(inode, ex);
cbits = le32_to_cpu(di->id2.i_super.s_clustersize_bits);
bbits = le32_to_cpu(di->id2.i_super.s_blocksize_bits);
sb->s_maxbytes = ocfs2_max_file_offset(bbits, cbits);
- memcpy(sb->s_uuid, di->id2.i_super.s_uuid,
+ memcpy(&sb->s_uuid, di->id2.i_super.s_uuid,
sizeof(di->id2.i_super.s_uuid));
osb->osb_dx_mask = (1 << (cbits - bbits)) - 1;
void *buffer,
size_t buffer_size)
{
- int ret;
+ int ret, had_lock;
struct buffer_head *di_bh = NULL;
+ struct ocfs2_lock_holder oh;
- ret = ocfs2_inode_lock(inode, &di_bh, 0);
- if (ret < 0) {
- mlog_errno(ret);
- return ret;
+ had_lock = ocfs2_inode_lock_tracker(inode, &di_bh, 0, &oh);
+ if (had_lock < 0) {
+ mlog_errno(had_lock);
+ return had_lock;
}
down_read(&OCFS2_I(inode)->ip_xattr_sem);
ret = ocfs2_xattr_get_nolock(inode, di_bh, name_index,
name, buffer, buffer_size);
up_read(&OCFS2_I(inode)->ip_xattr_sem);
- ocfs2_inode_unlock(inode, 0);
+ ocfs2_inode_unlock_tracker(inode, 0, &oh, had_lock);
brelse(di_bh);
{
struct buffer_head *di_bh = NULL;
struct ocfs2_dinode *di;
- int ret, credits, ref_meta = 0, ref_credits = 0;
+ int ret, credits, had_lock, ref_meta = 0, ref_credits = 0;
struct ocfs2_super *osb = OCFS2_SB(inode->i_sb);
struct inode *tl_inode = osb->osb_tl_inode;
struct ocfs2_xattr_set_ctxt ctxt = { NULL, NULL, NULL, };
struct ocfs2_refcount_tree *ref_tree = NULL;
+ struct ocfs2_lock_holder oh;
struct ocfs2_xattr_info xi = {
.xi_name_index = name_index,
return -ENOMEM;
}
- ret = ocfs2_inode_lock(inode, &di_bh, 1);
- if (ret < 0) {
+ had_lock = ocfs2_inode_lock_tracker(inode, &di_bh, 1, &oh);
+ if (had_lock < 0) {
+ ret = had_lock;
mlog_errno(ret);
goto cleanup_nolock;
}
if (ret)
mlog_errno(ret);
}
- ocfs2_inode_unlock(inode, 1);
+ ocfs2_inode_unlock_tracker(inode, 1, &oh, had_lock);
cleanup_nolock:
brelse(di_bh);
brelse(xbs.xattr_bh);
likely(f->f_op->write || f->f_op->write_iter))
f->f_mode |= FMODE_CAN_WRITE;
+ f->f_write_hint = WRITE_LIFE_NOT_SET;
f->f_flags &= ~(O_CREAT | O_EXCL | O_NOCTTY | O_TRUNC);
file_ra_state_init(&f->f_ra, f->f_mapping->host->i_mapping);
spin_lock(&m->q.lock);
if (m->c != -1) {
for (;;) {
- if (likely(list_empty(&wait.task_list)))
- __add_wait_queue_tail(&m->q, &wait);
+ if (likely(list_empty(&wait.entry)))
+ __add_wait_queue_entry_tail(&m->q, &wait);
set_current_state(TASK_UNINTERRUPTIBLE);
if (m->c == -1)
do {
long n = left, t;
- if (likely(list_empty(&wait.task_list)))
- __add_wait_queue_tail_exclusive(&m->q, &wait);
+ if (likely(list_empty(&wait.entry)))
+ __add_wait_queue_entry_tail_exclusive(&m->q, &wait);
set_current_state(TASK_INTERRUPTIBLE);
if (m->c > 0)
left = -EINTR;
} while (left > 0);
- if (!list_empty(&wait.task_list))
- list_del(&wait.task_list);
+ if (!list_empty(&wait.entry))
+ list_del(&wait.entry);
else if (left <= 0 && waitqueue_active(&m->q))
__wake_up_locked_key(&m->q, TASK_INTERRUPTIBLE, NULL);
__set_current_state(TASK_RUNNING);
return err;
}
-static struct ovl_fh *ovl_encode_fh(struct dentry *lower, uuid_be *uuid)
+static struct ovl_fh *ovl_encode_fh(struct dentry *lower, uuid_t *uuid)
{
struct ovl_fh *fh;
int fh_type, fh_len, dwords;
struct dentry *upper)
{
struct super_block *sb = lower->d_sb;
- uuid_be *uuid = (uuid_be *) &sb->s_uuid;
const struct ovl_fh *fh = NULL;
int err;
* up and a pure upper inode.
*/
if (sb->s_export_op && sb->s_export_op->fh_to_dentry &&
- uuid_be_cmp(*uuid, NULL_UUID_BE)) {
- fh = ovl_encode_fh(lower, uuid);
+ !uuid_is_null(&sb->s_uuid)) {
+ fh = ovl_encode_fh(lower, &sb->s_uuid);
if (IS_ERR(fh))
return PTR_ERR(fh);
}
.link = link
};
- upper = lookup_one_len(dentry->d_name.name, upperdir,
- dentry->d_name.len);
- err = PTR_ERR(upper);
- if (IS_ERR(upper))
- goto out;
-
err = security_inode_copy_up(dentry, &new_creds);
if (err < 0)
- goto out1;
+ goto out;
if (new_creds)
old_creds = override_creds(new_creds);
}
if (err)
- goto out2;
+ goto out;
if (S_ISREG(stat->mode)) {
struct path upperpath;
/*
* Store identifier of lower inode in upper inode xattr to
* allow lookup of the copy up origin inode.
+ *
+ * Don't set origin when we are breaking the association with a lower
+ * hard link.
*/
- err = ovl_set_origin(dentry, lowerpath->dentry, temp);
- if (err)
+ if (S_ISDIR(stat->mode) || stat->nlink == 1) {
+ err = ovl_set_origin(dentry, lowerpath->dentry, temp);
+ if (err)
+ goto out_cleanup;
+ }
+
+ upper = lookup_one_len(dentry->d_name.name, upperdir,
+ dentry->d_name.len);
+ if (IS_ERR(upper)) {
+ err = PTR_ERR(upper);
+ upper = NULL;
goto out_cleanup;
+ }
if (tmpfile)
err = ovl_do_link(temp, udir, upper, true);
/* Restore timestamps on parent (best effort) */
ovl_set_timestamps(upperdir, pstat);
-out2:
+out:
dput(temp);
-out1:
dput(upper);
-out:
return err;
out_cleanup:
if (!tmpfile)
ovl_cleanup(wdir, temp);
- goto out2;
+ goto out;
}
/*
* Make sure that the stored uuid matches the uuid of the lower
* layer where file handle will be decoded.
*/
- if (uuid_be_cmp(fh->uuid, *(uuid_be *) &mnt->mnt_sb->s_uuid))
+ if (!uuid_equal(&fh->uuid, &mnt->mnt_sb->s_uuid))
goto out;
origin = exportfs_decode_fh(mnt, (struct fid *)fh->fid,
u8 len; /* size of this header + size of fid */
u8 flags; /* OVL_FH_FLAG_* */
u8 type; /* fid_type of fid */
- uuid_be uuid; /* uuid of filesystem */
+ uuid_t uuid; /* uuid of filesystem */
u8 fid[0]; /* file identifier */
} __packed;
struct kiocb kiocb;
ssize_t ret;
- if (flags & ~(RWF_HIPRI | RWF_DSYNC | RWF_SYNC))
- return -EOPNOTSUPP;
-
init_sync_kiocb(&kiocb, filp);
- if (flags & RWF_HIPRI)
- kiocb.ki_flags |= IOCB_HIPRI;
- if (flags & RWF_DSYNC)
- kiocb.ki_flags |= IOCB_DSYNC;
- if (flags & RWF_SYNC)
- kiocb.ki_flags |= (IOCB_DSYNC | IOCB_SYNC);
+ ret = kiocb_set_rw_flags(&kiocb, flags);
+ if (ret)
+ return ret;
kiocb.ki_pos = *ppos;
if (type == READ)
static void queue_log_writer(struct super_block *s)
{
- wait_queue_t wait;
+ wait_queue_entry_t wait;
struct reiserfs_journal *journal = SB_JOURNAL(s);
set_bit(J_WRITERS_QUEUED, &journal->j_state);
return table->entry++;
}
-static int __pollwake(wait_queue_t *wait, unsigned mode, int sync, void *key)
+static int __pollwake(wait_queue_entry_t *wait, unsigned mode, int sync, void *key)
{
struct poll_wqueues *pwq = wait->private;
DECLARE_WAITQUEUE(dummy_wait, pwq->polling_task);
return default_wake_function(&dummy_wait, mode, sync, key);
}
-static int pollwake(wait_queue_t *wait, unsigned mode, int sync, void *key)
+static int pollwake(wait_queue_entry_t *wait, unsigned mode, int sync, void *key)
{
struct poll_table_entry *entry;
if (likely(!waitqueue_active(wqh)))
return;
- /* wait_queue_t->func(POLLFREE) should do remove_wait_queue() */
+ /* wait_queue_entry_t->func(POLLFREE) should do remove_wait_queue() */
wake_up_poll(wqh, POLLHUP | POLLFREE);
}
/*
* allocate new block and move data
*/
- switch (fs32_to_cpu(sb, usb1->fs_optim)) {
- case UFS_OPTSPACE:
+ if (fs32_to_cpu(sb, usb1->fs_optim) == UFS_OPTSPACE) {
request = newcount;
- if (uspi->s_minfree < 5 || uspi->cs_total.cs_nffree
- > uspi->s_dsize * uspi->s_minfree / (2 * 100))
- break;
- usb1->fs_optim = cpu_to_fs32(sb, UFS_OPTTIME);
- break;
- default:
- usb1->fs_optim = cpu_to_fs32(sb, UFS_OPTTIME);
-
- case UFS_OPTTIME:
+ if (uspi->cs_total.cs_nffree < uspi->s_space_to_time)
+ usb1->fs_optim = cpu_to_fs32(sb, UFS_OPTTIME);
+ } else {
request = uspi->s_fpb;
- if (uspi->cs_total.cs_nffree < uspi->s_dsize *
- (uspi->s_minfree - 2) / 100)
- break;
- usb1->fs_optim = cpu_to_fs32(sb, UFS_OPTTIME);
- break;
+ if (uspi->cs_total.cs_nffree > uspi->s_time_to_space)
+ usb1->fs_optim = cpu_to_fs32(sb, UFS_OPTSPACE);
}
result = ufs_alloc_fragments (inode, cgno, goal, request, err);
if (result) {
*/
inode->i_mode = mode = fs16_to_cpu(sb, ufs_inode->ui_mode);
set_nlink(inode, fs16_to_cpu(sb, ufs_inode->ui_nlink));
- if (inode->i_nlink == 0) {
- ufs_error (sb, "ufs_read_inode", "inode %lu has zero nlink\n", inode->i_ino);
- return -1;
- }
+ if (inode->i_nlink == 0)
+ return -ESTALE;
/*
* Linux now has 32-bit uid and gid, so we can support EFT.
i_gid_write(inode, ufs_get_inode_gid(sb, ufs_inode));
inode->i_size = fs64_to_cpu(sb, ufs_inode->ui_size);
- inode->i_atime.tv_sec = fs32_to_cpu(sb, ufs_inode->ui_atime.tv_sec);
- inode->i_ctime.tv_sec = fs32_to_cpu(sb, ufs_inode->ui_ctime.tv_sec);
- inode->i_mtime.tv_sec = fs32_to_cpu(sb, ufs_inode->ui_mtime.tv_sec);
+ inode->i_atime.tv_sec = (signed)fs32_to_cpu(sb, ufs_inode->ui_atime.tv_sec);
+ inode->i_ctime.tv_sec = (signed)fs32_to_cpu(sb, ufs_inode->ui_ctime.tv_sec);
+ inode->i_mtime.tv_sec = (signed)fs32_to_cpu(sb, ufs_inode->ui_mtime.tv_sec);
inode->i_mtime.tv_nsec = 0;
inode->i_atime.tv_nsec = 0;
inode->i_ctime.tv_nsec = 0;
*/
inode->i_mode = mode = fs16_to_cpu(sb, ufs2_inode->ui_mode);
set_nlink(inode, fs16_to_cpu(sb, ufs2_inode->ui_nlink));
- if (inode->i_nlink == 0) {
- ufs_error (sb, "ufs_read_inode", "inode %lu has zero nlink\n", inode->i_ino);
- return -1;
- }
+ if (inode->i_nlink == 0)
+ return -ESTALE;
/*
* Linux now has 32-bit uid and gid, so we can support EFT.
struct ufs_sb_private_info *uspi = UFS_SB(sb)->s_uspi;
struct buffer_head * bh;
struct inode *inode;
- int err;
+ int err = -EIO;
UFSD("ENTER, ino %lu\n", ino);
err = ufs1_read_inode(inode,
ufs_inode + ufs_inotofsbo(inode->i_ino));
}
-
+ brelse(bh);
if (err)
goto bad_inode;
+
inode->i_version++;
ufsi->i_lastfrag =
(inode->i_size + uspi->s_fsize - 1) >> uspi->s_fshift;
ufs_set_inode_ops(inode);
- brelse(bh);
-
UFSD("EXIT\n");
unlock_new_inode(inode);
return inode;
bad_inode:
iget_failed(inode);
- return ERR_PTR(-EIO);
+ return ERR_PTR(err);
}
static void ufs1_update_inode(struct inode *inode, struct ufs_inode *ufs_inode)
uspi->s_root_blocks = mul_u64_u32_div(uspi->s_dsize,
uspi->s_minfree, 100);
+ if (uspi->s_minfree <= 5) {
+ uspi->s_time_to_space = ~0ULL;
+ uspi->s_space_to_time = 0;
+ usb1->fs_optim = cpu_to_fs32(sb, UFS_OPTSPACE);
+ } else {
+ uspi->s_time_to_space = (uspi->s_root_blocks / 2) + 1;
+ uspi->s_space_to_time = mul_u64_u32_div(uspi->s_dsize,
+ uspi->s_minfree - 2, 100) - 1;
+ }
/*
* Compute another frequently used values
__s32 fs_magic; /* filesystem magic */
unsigned int s_dirblksize;
__u64 s_root_blocks;
+ __u64 s_time_to_space;
+ __u64 s_space_to_time;
};
/*
struct userfaultfd_wait_queue {
struct uffd_msg msg;
- wait_queue_t wq;
+ wait_queue_entry_t wq;
struct userfaultfd_ctx *ctx;
bool waken;
};
unsigned long len;
};
-static int userfaultfd_wake_function(wait_queue_t *wq, unsigned mode,
+static int userfaultfd_wake_function(wait_queue_entry_t *wq, unsigned mode,
int wake_flags, void *key)
{
struct userfaultfd_wake_range *range = key;
* wouldn't be enough, the smp_mb__before_spinlock is
* enough to avoid an explicit smp_mb() here.
*/
- list_del_init(&wq->task_list);
+ list_del_init(&wq->entry);
out:
return ret;
}
* and it's fine not to block on the spinlock. The uwq on this
* kernel stack can be released after the list_del_init.
*/
- if (!list_empty_careful(&uwq.wq.task_list)) {
+ if (!list_empty_careful(&uwq.wq.entry)) {
spin_lock(&ctx->fault_pending_wqh.lock);
/*
* No need of list_del_init(), the uwq on the stack
* will be freed shortly anyway.
*/
- list_del(&uwq.wq.task_list);
+ list_del(&uwq.wq.entry);
spin_unlock(&ctx->fault_pending_wqh.lock);
}
static inline struct userfaultfd_wait_queue *find_userfault_in(
wait_queue_head_t *wqh)
{
- wait_queue_t *wq;
+ wait_queue_entry_t *wq;
struct userfaultfd_wait_queue *uwq;
VM_BUG_ON(!spin_is_locked(&wqh->lock));
if (!waitqueue_active(wqh))
goto out;
/* walk in reverse to provide FIFO behavior to read userfaults */
- wq = list_last_entry(&wqh->task_list, typeof(*wq), task_list);
+ wq = list_last_entry(&wqh->head, typeof(*wq), entry);
uwq = container_of(wq, struct userfaultfd_wait_queue, wq);
out:
return uwq;
* changes __remove_wait_queue() to use
* list_del_init() in turn breaking the
* !list_empty_careful() check in
- * handle_userfault(). The uwq->wq.task_list
+ * handle_userfault(). The uwq->wq.head list
* must never be empty at any time during the
* refile, or the waitqueue could disappear
* from under us. The "wait_queue_head_t"
* parameter of __remove_wait_queue() is unused
* anyway.
*/
- list_del(&uwq->wq.task_list);
+ list_del(&uwq->wq.entry);
__add_wait_queue(&ctx->fault_wqh, &uwq->wq);
write_seqcount_end(&ctx->refile_seq);
fork_nctx = (struct userfaultfd_ctx *)
(unsigned long)
uwq->msg.arg.reserved.reserved1;
- list_move(&uwq->wq.task_list, &fork_event);
+ list_move(&uwq->wq.entry, &fork_event);
spin_unlock(&ctx->event_wqh.lock);
ret = 0;
break;
if (!list_empty(&fork_event)) {
uwq = list_first_entry(&fork_event,
typeof(*uwq),
- wq.task_list);
- list_del(&uwq->wq.task_list);
+ wq.entry);
+ list_del(&uwq->wq.entry);
__add_wait_queue(&ctx->event_wqh, &uwq->wq);
userfaultfd_event_complete(ctx, uwq);
}
static void userfaultfd_show_fdinfo(struct seq_file *m, struct file *f)
{
struct userfaultfd_ctx *ctx = f->private_data;
- wait_queue_t *wq;
+ wait_queue_entry_t *wq;
struct userfaultfd_wait_queue *uwq;
unsigned long pending = 0, total = 0;
spin_lock(&ctx->fault_pending_wqh.lock);
- list_for_each_entry(wq, &ctx->fault_pending_wqh.task_list, task_list) {
+ list_for_each_entry(wq, &ctx->fault_pending_wqh.head, entry) {
uwq = container_of(wq, struct userfaultfd_wait_queue, wq);
pending++;
total++;
}
- list_for_each_entry(wq, &ctx->fault_wqh.task_list, task_list) {
+ list_for_each_entry(wq, &ctx->fault_wqh.head, entry) {
uwq = container_of(wq, struct userfaultfd_wait_queue, wq);
total++;
}
xfs_sysfs.o \
xfs_trans.o \
xfs_xattr.o \
- kmem.o \
- uuid.o
+ kmem.o
# low-level transaction/log code
xfs-y += xfs_log.o \
+++ /dev/null
-/*
- * Copyright (c) 2000-2003,2005 Silicon Graphics, Inc.
- * All Rights Reserved.
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it would be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
- */
-#include <xfs.h>
-
-/* IRIX interpretation of an uuid_t */
-typedef struct {
- __be32 uu_timelow;
- __be16 uu_timemid;
- __be16 uu_timehi;
- __be16 uu_clockseq;
- __be16 uu_node[3];
-} xfs_uu_t;
-
-/*
- * uuid_getnodeuniq - obtain the node unique fields of a UUID.
- *
- * This is not in any way a standard or condoned UUID function;
- * it just something that's needed for user-level file handles.
- */
-void
-uuid_getnodeuniq(uuid_t *uuid, int fsid [2])
-{
- xfs_uu_t *uup = (xfs_uu_t *)uuid;
-
- fsid[0] = (be16_to_cpu(uup->uu_clockseq) << 16) |
- be16_to_cpu(uup->uu_timemid);
- fsid[1] = be32_to_cpu(uup->uu_timelow);
-}
-
-int
-uuid_is_nil(uuid_t *uuid)
-{
- int i;
- char *cp = (char *)uuid;
-
- if (uuid == NULL)
- return 0;
- /* implied check of version number here... */
- for (i = 0; i < sizeof *uuid; i++)
- if (*cp++) return 0; /* not nil */
- return 1; /* is nil */
-}
-
-int
-uuid_equal(uuid_t *uuid1, uuid_t *uuid2)
-{
- return memcmp(uuid1, uuid2, sizeof(uuid_t)) ? 0 : 1;
-}
+++ /dev/null
-/*
- * Copyright (c) 2000-2003,2005 Silicon Graphics, Inc.
- * All Rights Reserved.
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it would be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
- */
-#ifndef __XFS_SUPPORT_UUID_H__
-#define __XFS_SUPPORT_UUID_H__
-
-typedef struct {
- unsigned char __u_bits[16];
-} uuid_t;
-
-extern int uuid_is_nil(uuid_t *uuid);
-extern int uuid_equal(uuid_t *uuid1, uuid_t *uuid2);
-extern void uuid_getnodeuniq(uuid_t *uuid, int fsid [2]);
-
-static inline void
-uuid_copy(uuid_t *dst, uuid_t *src)
-{
- memcpy(dst, src, sizeof(uuid_t));
-}
-
-#endif /* __XFS_SUPPORT_UUID_H__ */
struct xfs_inode *ip = XFS_I(ioend->io_inode);
xfs_off_t offset = ioend->io_offset;
size_t size = ioend->io_size;
- int error = ioend->io_bio->bi_error;
+ int error;
/*
* Just clean up the in-memory strutures if the fs has been shut down.
/*
* Clean up any COW blocks on an I/O error.
*/
+ error = blk_status_to_errno(ioend->io_bio->bi_status);
if (unlikely(error)) {
switch (ioend->io_type) {
case XFS_IO_COW:
else if (ioend->io_append_trans)
queue_work(mp->m_data_workqueue, &ioend->io_work);
else
- xfs_destroy_ioend(ioend, bio->bi_error);
+ xfs_destroy_ioend(ioend, blk_status_to_errno(bio->bi_status));
}
STATIC int
* time.
*/
if (status) {
- ioend->io_bio->bi_error = status;
+ ioend->io_bio->bi_status = errno_to_blk_status(status);
bio_endio(ioend->io_bio);
return status;
}
+ ioend->io_bio->bi_write_hint = ioend->io_inode->i_write_hint;
submit_bio(ioend->io_bio);
return 0;
}
bio_chain(ioend->io_bio, new);
bio_get(ioend->io_bio); /* for xfs_destroy_ioend */
ioend->io_bio->bi_opf = REQ_OP_WRITE | wbc_to_write_flags(wbc);
+ ioend->io_bio->bi_write_hint = ioend->io_inode->i_write_hint;
submit_bio(ioend->io_bio);
ioend->io_bio = new;
}
* The swap code (ab-)uses ->bmap to get a block mapping and then
* bypasseѕ the file system for actual I/O. We really can't allow
* that on reflinks inodes, so we have to skip out here. And yes,
- * 0 is the magic code for a bmap error..
+ * 0 is the magic code for a bmap error.
+ *
+ * Since we don't pass back blockdev info, we can't return bmap
+ * information for rt files either.
*/
- if (xfs_is_reflink_inode(ip))
+ if (xfs_is_reflink_inode(ip) || XFS_IS_REALTIME_INODE(ip))
return 0;
filemap_write_and_wait(mapping);
* don't overwrite existing errors - otherwise we can lose errors on
* buffers that require multiple bios to complete.
*/
- if (bio->bi_error)
- cmpxchg(&bp->b_io_error, 0, bio->bi_error);
+ if (bio->bi_status) {
+ int error = blk_status_to_errno(bio->bi_status);
+
+ cmpxchg(&bp->b_io_error, 0, error);
+ }
if (!bp->b_error && xfs_buf_is_vmapped(bp) && (bp->b_flags & XBF_READ))
invalidate_kernel_vmap_range(bp->b_addr, xfs_buf_vmap_len(bp));
if (!count)
return 0; /* skip atime */
- xfs_ilock(ip, XFS_IOLOCK_SHARED);
+ if (!xfs_ilock_nowait(ip, XFS_IOLOCK_SHARED)) {
+ if (iocb->ki_flags & IOCB_NOWAIT)
+ return -EAGAIN;
+ xfs_ilock(ip, XFS_IOLOCK_SHARED);
+ }
ret = dax_iomap_rw(iocb, to, &xfs_iomap_ops);
xfs_iunlock(ip, XFS_IOLOCK_SHARED);
iolock = XFS_IOLOCK_SHARED;
}
- xfs_ilock(ip, iolock);
+ if (!xfs_ilock_nowait(ip, iolock)) {
+ if (iocb->ki_flags & IOCB_NOWAIT)
+ return -EAGAIN;
+ xfs_ilock(ip, iolock);
+ }
ret = xfs_file_aio_write_checks(iocb, from, &iolock);
if (ret)
* otherwise demote the lock if we had to take the exclusive lock
* for other reasons in xfs_file_aio_write_checks.
*/
- if (unaligned_io)
- inode_dio_wait(inode);
- else if (iolock == XFS_IOLOCK_EXCL) {
+ if (unaligned_io) {
+ /* If we are going to wait for other DIO to finish, bail */
+ if (iocb->ki_flags & IOCB_NOWAIT) {
+ if (atomic_read(&inode->i_dio_count))
+ return -EAGAIN;
+ } else {
+ inode_dio_wait(inode);
+ }
+ } else if (iolock == XFS_IOLOCK_EXCL) {
xfs_ilock_demote(ip, XFS_IOLOCK_EXCL);
iolock = XFS_IOLOCK_SHARED;
}
size_t count;
loff_t pos;
- xfs_ilock(ip, iolock);
+ if (!xfs_ilock_nowait(ip, iolock)) {
+ if (iocb->ki_flags & IOCB_NOWAIT)
+ return -EAGAIN;
+ xfs_ilock(ip, iolock);
+ }
+
ret = xfs_file_aio_write_checks(iocb, from, &iolock);
if (ret)
goto out;
return -EFBIG;
if (XFS_FORCED_SHUTDOWN(XFS_M(inode->i_sb)))
return -EIO;
+ file->f_mode |= FMODE_AIO_NOWAIT;
return 0;
}
DEFINE_WAIT_BIT(wait, &ip->i_flags, __XFS_INEW_BIT);
do {
- prepare_to_wait(wq, &wait.wait, TASK_UNINTERRUPTIBLE);
+ prepare_to_wait(wq, &wait.wq_entry, TASK_UNINTERRUPTIBLE);
if (!xfs_iflags_test(ip, XFS_INEW))
break;
schedule();
} while (true);
- finish_wait(wq, &wait.wait);
+ finish_wait(wq, &wait.wq_entry);
}
/*
DEFINE_WAIT_BIT(wait, &ip->i_flags, __XFS_IFLOCK_BIT);
do {
- prepare_to_wait_exclusive(wq, &wait.wait, TASK_UNINTERRUPTIBLE);
+ prepare_to_wait_exclusive(wq, &wait.wq_entry, TASK_UNINTERRUPTIBLE);
if (xfs_isiflocked(ip))
io_schedule();
} while (!xfs_iflock_nowait(ip));
- finish_wait(wq, &wait.wait);
+ finish_wait(wq, &wait.wq_entry);
}
STATIC uint
xfs_iunpin(ip);
do {
- prepare_to_wait(wq, &wait.wait, TASK_UNINTERRUPTIBLE);
+ prepare_to_wait(wq, &wait.wq_entry, TASK_UNINTERRUPTIBLE);
if (xfs_ipincount(ip))
io_schedule();
} while (xfs_ipincount(ip));
- finish_wait(wq, &wait.wait);
+ finish_wait(wq, &wait.wq_entry);
}
void
in_f->ilf_dsize = in_f32->ilf_dsize;
in_f->ilf_ino = in_f32->ilf_ino;
/* copy biggest field of ilf_u */
- memcpy(in_f->ilf_u.ilfu_uuid.__u_bits,
- in_f32->ilf_u.ilfu_uuid.__u_bits,
- sizeof(uuid_t));
+ uuid_copy(&in_f->ilf_u.ilfu_uuid, &in_f32->ilf_u.ilfu_uuid);
in_f->ilf_blkno = in_f32->ilf_blkno;
in_f->ilf_len = in_f32->ilf_len;
in_f->ilf_boffset = in_f32->ilf_boffset;
in_f->ilf_dsize = in_f64->ilf_dsize;
in_f->ilf_ino = in_f64->ilf_ino;
/* copy biggest field of ilf_u */
- memcpy(in_f->ilf_u.ilfu_uuid.__u_bits,
- in_f64->ilf_u.ilfu_uuid.__u_bits,
- sizeof(uuid_t));
+ uuid_copy(&in_f->ilf_u.ilfu_uuid, &in_f64->ilf_u.ilfu_uuid);
in_f->ilf_blkno = in_f64->ilf_blkno;
in_f->ilf_len = in_f64->ilf_len;
in_f->ilf_boffset = in_f64->ilf_boffset;
lockmode = xfs_ilock_data_map_shared(ip);
}
+ if ((flags & IOMAP_NOWAIT) && !(ip->i_df.if_flags & XFS_IFEXTENTS)) {
+ error = -EAGAIN;
+ goto out_unlock;
+ }
+
ASSERT(offset <= mp->m_super->s_maxbytes);
if ((xfs_fsize_t)offset + length > mp->m_super->s_maxbytes)
length = mp->m_super->s_maxbytes - offset;
if ((flags & (IOMAP_WRITE | IOMAP_ZERO)) && xfs_is_reflink_inode(ip)) {
if (flags & IOMAP_DIRECT) {
+ /*
+ * A reflinked inode will result in CoW alloc.
+ * FIXME: It could still overwrite on unshared extents
+ * and not need allocation.
+ */
+ if (flags & IOMAP_NOWAIT) {
+ error = -EAGAIN;
+ goto out_unlock;
+ }
/* may drop and re-acquire the ilock */
error = xfs_reflink_allocate_cow(ip, &imap, &shared,
&lockmode);
}
if ((flags & IOMAP_WRITE) && imap_needs_alloc(inode, &imap, nimaps)) {
+ /*
+ * If nowait is set bail since we are going to make
+ * allocations.
+ */
+ if (flags & IOMAP_NOWAIT) {
+ error = -EAGAIN;
+ goto out_unlock;
+ }
/*
* We cap the maximum length we map here to MAX_WRITEBACK_PAGES
* pages to keep the chunks of work done where somewhat symmetric
#define __XFS_LINUX__
#include <linux/types.h>
+#include <linux/uuid.h>
/*
* Kernel specific type declarations for XFS
#include "kmem.h"
#include "mrlock.h"
-#include "uuid.h"
#include <linux/semaphore.h>
#include <linux/mm.h>
{
ASSERT(head->h_magicno == cpu_to_be32(XLOG_HEADER_MAGIC_NUM));
- if (uuid_is_nil(&head->h_fs_uuid)) {
+ if (uuid_is_null(&head->h_fs_uuid)) {
/*
* IRIX doesn't write the h_fs_uuid or h_fmt fields. If
- * h_fs_uuid is nil, we assume this log was last mounted
+ * h_fs_uuid is null, we assume this log was last mounted
* by IRIX and continue.
*/
- xfs_warn(mp, "nil uuid in log - IRIX style log");
+ xfs_warn(mp, "null uuid in log - IRIX style log");
} else if (unlikely(!uuid_equal(&mp->m_sb.sb_uuid, &head->h_fs_uuid))) {
xfs_warn(mp, "log has mismatched uuid - can't recover");
xlog_header_check_dump(mp, head);
int hole, i;
/* Publish UUID in struct super_block */
- BUILD_BUG_ON(sizeof(mp->m_super->s_uuid) != sizeof(uuid_t));
- memcpy(&mp->m_super->s_uuid, uuid, sizeof(uuid_t));
+ uuid_copy(&mp->m_super->s_uuid, uuid);
if (mp->m_flags & XFS_MOUNT_NOUUID)
return 0;
- if (uuid_is_nil(uuid)) {
- xfs_warn(mp, "Filesystem has nil UUID - can't mount");
+ if (uuid_is_null(uuid)) {
+ xfs_warn(mp, "Filesystem has null UUID - can't mount");
return -EINVAL;
}
mutex_lock(&xfs_uuid_table_mutex);
for (i = 0, hole = -1; i < xfs_uuid_table_size; i++) {
- if (uuid_is_nil(&xfs_uuid_table[i])) {
+ if (uuid_is_null(&xfs_uuid_table[i])) {
hole = i;
continue;
}
mutex_lock(&xfs_uuid_table_mutex);
for (i = 0; i < xfs_uuid_table_size; i++) {
- if (uuid_is_nil(&xfs_uuid_table[i]))
+ if (uuid_is_null(&xfs_uuid_table[i]))
continue;
if (!uuid_equal(uuid, &xfs_uuid_table[i]))
continue;
* Copies the low order bits of the timestamp and the randomly
* set "sequence" number out of a UUID.
*/
- uuid_getnodeuniq(&sbp->sb_uuid, mp->m_fixedfsid);
+ mp->m_fixedfsid[0] =
+ (get_unaligned_be16(&sbp->sb_uuid.b[8]) << 16) |
+ get_unaligned_be16(&sbp->sb_uuid.b[4]);
+ mp->m_fixedfsid[1] = get_unaligned_be32(&sbp->sb_uuid.b[0]);
mp->m_dmevmask = 0; /* not persistent; set after each mount */
xfs_init_zones(void)
{
xfs_ioend_bioset = bioset_create(4 * MAX_BUF_PER_PAGE,
- offsetof(struct xfs_ioend, io_inline_bio));
+ offsetof(struct xfs_ioend, io_inline_bio),
+ BIOSET_NEED_BVECS);
if (!xfs_ioend_bioset)
goto out;
bool acpi_bay_match(acpi_handle handle);
bool acpi_dock_match(acpi_handle handle);
-bool acpi_check_dsm(acpi_handle handle, const u8 *uuid, u64 rev, u64 funcs);
-union acpi_object *acpi_evaluate_dsm(acpi_handle handle, const u8 *uuid,
+bool acpi_check_dsm(acpi_handle handle, const guid_t *guid, u64 rev, u64 funcs);
+union acpi_object *acpi_evaluate_dsm(acpi_handle handle, const guid_t *guid,
u64 rev, u64 func, union acpi_object *argv4);
static inline union acpi_object *
-acpi_evaluate_dsm_typed(acpi_handle handle, const u8 *uuid, u64 rev, u64 func,
- union acpi_object *argv4, acpi_object_type type)
+acpi_evaluate_dsm_typed(acpi_handle handle, const guid_t *guid, u64 rev,
+ u64 func, union acpi_object *argv4,
+ acpi_object_type type)
{
union acpi_object *obj;
- obj = acpi_evaluate_dsm(handle, uuid, rev, func, argv4);
+ obj = acpi_evaluate_dsm(handle, guid, rev, func, argv4);
if (obj && obj->type != type) {
ACPI_FREE(obj);
obj = NULL;
u32 of_compatible_ok:1;
u32 coherent_dma:1;
u32 cca_seen:1;
- u32 reserved:20;
+ u32 spi_i2c_slave:1;
+ u32 reserved:19;
};
/* File System */
/* Wakeup Management */
struct acpi_device_wakeup_flags {
u8 valid:1; /* Can successfully enable wakeup? */
- u8 run_wake:1; /* Run-Wake GPE devices */
u8 notifier_present:1; /* Wake-up notify handler has been installed */
u8 enabled:1; /* Enabled for wakeup */
};
struct acpi_device_wakeup_context {
- struct work_struct work;
+ void (*func)(struct acpi_device_wakeup_context *context);
struct device *dev;
};
#endif
#ifdef CONFIG_PM
+void acpi_pm_wakeup_event(struct device *dev);
acpi_status acpi_add_pm_notifier(struct acpi_device *adev, struct device *dev,
- void (*work_func)(struct work_struct *work));
+ void (*func)(struct acpi_device_wakeup_context *context));
acpi_status acpi_remove_pm_notifier(struct acpi_device *adev);
+bool acpi_pm_device_can_wakeup(struct device *dev);
int acpi_pm_device_sleep_state(struct device *, int *, int);
-int acpi_pm_device_run_wake(struct device *, bool);
+int acpi_pm_set_device_wakeup(struct device *dev, bool enable);
#else
+static inline void acpi_pm_wakeup_event(struct device *dev)
+{
+}
static inline acpi_status acpi_add_pm_notifier(struct acpi_device *adev,
struct device *dev,
- void (*work_func)(struct work_struct *work))
+ void (*func)(struct acpi_device_wakeup_context *context))
{
return AE_SUPPORT;
}
{
return AE_SUPPORT;
}
+static inline bool acpi_pm_device_can_wakeup(struct device *dev)
+{
+ return false;
+}
static inline int acpi_pm_device_sleep_state(struct device *d, int *p, int m)
{
if (p)
return (m >= ACPI_STATE_D0 && m <= ACPI_STATE_D3_COLD) ?
m : ACPI_STATE_D0;
}
-static inline int acpi_pm_device_run_wake(struct device *dev, bool enable)
-{
- return -ENODEV;
-}
-#endif
-
-#ifdef CONFIG_PM_SLEEP
-int acpi_pm_device_sleep_wake(struct device *, bool);
-#else
-static inline int acpi_pm_device_sleep_wake(struct device *dev, bool enable)
+static inline int acpi_pm_set_device_wakeup(struct device *dev, bool enable)
{
return -ENODEV;
}
/* Current ACPICA subsystem version in YYYYMMDD format */
-#define ACPI_CA_VERSION 0x20170303
+#define ACPI_CA_VERSION 0x20170531
#include <acpi/acconfig.h>
#include <acpi/actypes.h>
u8 type_specific;
};
+struct acpi_resource_label {
+ u16 string_length;
+ char *string_ptr;
+};
+
struct acpi_resource_source {
u8 index;
u16 string_length;
u64 address;
};
+/* Generic Address Space Access Sizes */
+#define ACPI_ACCESS_SIZE_UNDEFINED 0
+#define ACPI_ACCESS_SIZE_BYTE 1
+#define ACPI_ACCESS_SIZE_WORD 2
+#define ACPI_ACCESS_SIZE_DWORD 3
+#define ACPI_ACCESS_SIZE_QWORD 4
+
struct acpi_resource_gpio {
u8 revision_id;
u8 connection_type;
#define ACPI_UART_CLEAR_TO_SEND (1<<6)
#define ACPI_UART_REQUEST_TO_SEND (1<<7)
+struct acpi_resource_pin_function {
+ u8 revision_id;
+ u8 pin_config;
+ u8 sharable; /* For values, see Interrupt Attributes above */
+ u16 function_number;
+ u16 pin_table_length;
+ u16 vendor_length;
+ struct acpi_resource_source resource_source;
+ u16 *pin_table;
+ u8 *vendor_data;
+};
+
+struct acpi_resource_pin_config {
+ u8 revision_id;
+ u8 producer_consumer; /* For values, see Producer/Consumer above */
+ u8 sharable; /* For values, see Interrupt Attributes above */
+ u8 pin_config_type;
+ u32 pin_config_value;
+ u16 pin_table_length;
+ u16 vendor_length;
+ struct acpi_resource_source resource_source;
+ u16 *pin_table;
+ u8 *vendor_data;
+};
+
+/* Values for pin_config_type field above */
+
+#define ACPI_PIN_CONFIG_DEFAULT 0
+#define ACPI_PIN_CONFIG_BIAS_PULL_UP 1
+#define ACPI_PIN_CONFIG_BIAS_PULL_DOWN 2
+#define ACPI_PIN_CONFIG_BIAS_DEFAULT 3
+#define ACPI_PIN_CONFIG_BIAS_DISABLE 4
+#define ACPI_PIN_CONFIG_BIAS_HIGH_IMPEDANCE 5
+#define ACPI_PIN_CONFIG_BIAS_BUS_HOLD 6
+#define ACPI_PIN_CONFIG_DRIVE_OPEN_DRAIN 7
+#define ACPI_PIN_CONFIG_DRIVE_OPEN_SOURCE 8
+#define ACPI_PIN_CONFIG_DRIVE_PUSH_PULL 9
+#define ACPI_PIN_CONFIG_DRIVE_STRENGTH 10
+#define ACPI_PIN_CONFIG_SLEW_RATE 11
+#define ACPI_PIN_CONFIG_INPUT_DEBOUNCE 12
+#define ACPI_PIN_CONFIG_INPUT_SCHMITT_TRIGGER 13
+
+struct acpi_resource_pin_group {
+ u8 revision_id;
+ u8 producer_consumer; /* For values, see Producer/Consumer above */
+ u16 pin_table_length;
+ u16 vendor_length;
+ u16 *pin_table;
+ struct acpi_resource_label resource_label;
+ u8 *vendor_data;
+};
+
+struct acpi_resource_pin_group_function {
+ u8 revision_id;
+ u8 producer_consumer; /* For values, see Producer/Consumer above */
+ u8 sharable; /* For values, see Interrupt Attributes above */
+ u16 function_number;
+ u16 vendor_length;
+ struct acpi_resource_source resource_source;
+ struct acpi_resource_label resource_source_label;
+ u8 *vendor_data;
+};
+
+struct acpi_resource_pin_group_config {
+ u8 revision_id;
+ u8 producer_consumer; /* For values, see Producer/Consumer above */
+ u8 sharable; /* For values, see Interrupt Attributes above */
+ u8 pin_config_type; /* For values, see pin_config_type above */
+ u32 pin_config_value;
+ u16 vendor_length;
+ struct acpi_resource_source resource_source;
+ struct acpi_resource_label resource_source_label;
+ u8 *vendor_data;
+};
+
/* ACPI_RESOURCE_TYPEs */
#define ACPI_RESOURCE_TYPE_IRQ 0
#define ACPI_RESOURCE_TYPE_GPIO 17 /* ACPI 5.0 */
#define ACPI_RESOURCE_TYPE_FIXED_DMA 18 /* ACPI 5.0 */
#define ACPI_RESOURCE_TYPE_SERIAL_BUS 19 /* ACPI 5.0 */
-#define ACPI_RESOURCE_TYPE_MAX 19
+#define ACPI_RESOURCE_TYPE_PIN_FUNCTION 20 /* ACPI 6.2 */
+#define ACPI_RESOURCE_TYPE_PIN_CONFIG 21 /* ACPI 6.2 */
+#define ACPI_RESOURCE_TYPE_PIN_GROUP 22 /* ACPI 6.2 */
+#define ACPI_RESOURCE_TYPE_PIN_GROUP_FUNCTION 23 /* ACPI 6.2 */
+#define ACPI_RESOURCE_TYPE_PIN_GROUP_CONFIG 24 /* ACPI 6.2 */
+#define ACPI_RESOURCE_TYPE_MAX 24
/* Master union for resource descriptors */
struct acpi_resource_spi_serialbus spi_serial_bus;
struct acpi_resource_uart_serialbus uart_serial_bus;
struct acpi_resource_common_serialbus common_serial_bus;
+ struct acpi_resource_pin_function pin_function;
+ struct acpi_resource_pin_config pin_config;
+ struct acpi_resource_pin_group pin_group;
+ struct acpi_resource_pin_group_function pin_group_function;
+ struct acpi_resource_pin_group_config pin_group_config;
/* Common fields */
#define ACPI_SIG_ECDT "ECDT" /* Embedded Controller Boot Resources Table */
#define ACPI_SIG_EINJ "EINJ" /* Error Injection table */
#define ACPI_SIG_ERST "ERST" /* Error Record Serialization Table */
+#define ACPI_SIG_HMAT "HMAT" /* Heterogeneous Memory Attributes Table */
#define ACPI_SIG_HEST "HEST" /* Hardware Error Source Table */
#define ACPI_SIG_MADT "APIC" /* Multiple APIC Description Table */
#define ACPI_SIG_MSCT "MSCT" /* Maximum System Characteristics Table */
+#define ACPI_SIG_PPTT "PPTT" /* Processor Properties Topology Table */
#define ACPI_SIG_SBST "SBST" /* Smart Battery Specification Table */
#define ACPI_SIG_SLIT "SLIT" /* System Locality Distance Information Table */
#define ACPI_SIG_SRAT "SRAT" /* System Resource Affinity Table */
ACPI_HEST_TYPE_AER_BRIDGE = 8,
ACPI_HEST_TYPE_GENERIC_ERROR = 9,
ACPI_HEST_TYPE_GENERIC_ERROR_V2 = 10,
- ACPI_HEST_TYPE_RESERVED = 11 /* 11 and greater are reserved */
+ ACPI_HEST_TYPE_IA32_DEFERRED_CHECK = 11,
+ ACPI_HEST_TYPE_RESERVED = 12 /* 12 and greater are reserved */
};
/*
#define ACPI_HEST_FIRMWARE_FIRST (1)
#define ACPI_HEST_GLOBAL (1<<1)
+#define ACPI_HEST_GHES_ASSIST (1<<2)
/*
* Macros to access the bus/segment numbers in Bus field above:
ACPI_HEST_NOTIFY_SEA = 8, /* ACPI 6.1 */
ACPI_HEST_NOTIFY_SEI = 9, /* ACPI 6.1 */
ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
- ACPI_HEST_NOTIFY_RESERVED = 11 /* 11 and greater are reserved */
+ ACPI_HEST_NOTIFY_SOFTWARE_DELEGATED = 11, /* ACPI 6.2 */
+ ACPI_HEST_NOTIFY_RESERVED = 12 /* 12 and greater are reserved */
};
/* Values for config_write_enable bitfield above */
struct acpi_hest_ia_machine_check {
struct acpi_hest_header header;
u16 reserved1;
- u8 flags;
+ u8 flags; /* See flags ACPI_HEST_GLOBAL, etc. above */
u8 enabled;
u32 records_to_preallocate;
u32 max_sections_per_record;
struct acpi_hest_ia_corrected {
struct acpi_hest_header header;
u16 reserved1;
- u8 flags;
+ u8 flags; /* See flags ACPI_HEST_GLOBAL, etc. above */
u8 enabled;
u32 records_to_preallocate;
u32 max_sections_per_record;
#define ACPI_HEST_GEN_VALID_FRU_STRING (1<<1)
#define ACPI_HEST_GEN_VALID_TIMESTAMP (1<<2)
+/* 11: IA32 Deferred Machine Check Exception (ACPI 6.2) */
+
+struct acpi_hest_ia_deferred_check {
+ struct acpi_hest_header header;
+ u16 reserved1;
+ u8 flags; /* See flags ACPI_HEST_GLOBAL, etc. above */
+ u8 enabled;
+ u32 records_to_preallocate;
+ u32 max_sections_per_record;
+ struct acpi_hest_notify notify;
+ u8 num_hardware_banks;
+ u8 reserved2[3];
+};
+
+/*******************************************************************************
+ *
+ * HMAT - Heterogeneous Memory Attributes Table (ACPI 6.2)
+ * Version 1
+ *
+ ******************************************************************************/
+
+struct acpi_table_hmat {
+ struct acpi_table_header header; /* Common ACPI table header */
+ u32 reserved;
+};
+
+/* Values for HMAT structure types */
+
+enum acpi_hmat_type {
+ ACPI_HMAT_TYPE_ADDRESS_RANGE = 0, /* Memory subystem address range */
+ ACPI_HMAT_TYPE_LOCALITY = 1, /* System locality latency and bandwidth information */
+ ACPI_HMAT_TYPE_CACHE = 2, /* Memory side cache information */
+ ACPI_HMAT_TYPE_RESERVED = 3 /* 3 and greater are reserved */
+};
+
+struct acpi_hmat_structure {
+ u16 type;
+ u16 reserved;
+ u32 length;
+};
+
+/*
+ * HMAT Structures, correspond to Type in struct acpi_hmat_structure
+ */
+
+/* 0: Memory subystem address range */
+
+struct acpi_hmat_address_range {
+ struct acpi_hmat_structure header;
+ u16 flags;
+ u16 reserved1;
+ u32 processor_PD; /* Processor proximity domain */
+ u32 memory_PD; /* Memory proximity domain */
+ u32 reserved2;
+ u64 physical_address_base; /* Physical address range base */
+ u64 physical_address_length; /* Physical address range length */
+};
+
+/* Masks for Flags field above */
+
+#define ACPI_HMAT_PROCESSOR_PD_VALID (1) /* 1: processor_PD field is valid */
+#define ACPI_HMAT_MEMORY_PD_VALID (1<<1) /* 1: memory_PD field is valid */
+#define ACPI_HMAT_RESERVATION_HINT (1<<2) /* 1: Reservation hint */
+
+/* 1: System locality latency and bandwidth information */
+
+struct acpi_hmat_locality {
+ struct acpi_hmat_structure header;
+ u8 flags;
+ u8 data_type;
+ u16 reserved1;
+ u32 number_of_initiator_Pds;
+ u32 number_of_target_Pds;
+ u32 reserved2;
+ u64 entry_base_unit;
+};
+
+/* Masks for Flags field above */
+
+#define ACPI_HMAT_MEMORY_HIERARCHY (0x0F)
+
+/* Values for Memory Hierarchy flag */
+
+#define ACPI_HMAT_MEMORY 0
+#define ACPI_HMAT_LAST_LEVEL_CACHE 1
+#define ACPI_HMAT_1ST_LEVEL_CACHE 2
+#define ACPI_HMAT_2ND_LEVEL_CACHE 3
+#define ACPI_HMAT_3RD_LEVEL_CACHE 4
+
+/* Values for data_type field above */
+
+#define ACPI_HMAT_ACCESS_LATENCY 0
+#define ACPI_HMAT_READ_LATENCY 1
+#define ACPI_HMAT_WRITE_LATENCY 2
+#define ACPI_HMAT_ACCESS_BANDWIDTH 3
+#define ACPI_HMAT_READ_BANDWIDTH 4
+#define ACPI_HMAT_WRITE_BANDWIDTH 5
+
+/* 2: Memory side cache information */
+
+struct acpi_hmat_cache {
+ struct acpi_hmat_structure header;
+ u32 memory_PD;
+ u32 reserved1;
+ u64 cache_size;
+ u32 cache_attributes;
+ u16 reserved2;
+ u16 number_of_SMBIOShandles;
+};
+
+/* Masks for cache_attributes field above */
+
+#define ACPI_HMAT_TOTAL_CACHE_LEVEL (0x0000000F)
+#define ACPI_HMAT_CACHE_LEVEL (0x000000F0)
+#define ACPI_HMAT_CACHE_ASSOCIATIVITY (0x00000F00)
+#define ACPI_HMAT_WRITE_POLICY (0x0000F000)
+#define ACPI_HMAT_CACHE_LINE_SIZE (0xFFFF0000)
+
+/* Values for cache associativity flag */
+
+#define ACPI_HMAT_CA_NONE (0)
+#define ACPI_HMAT_CA_DIRECT_MAPPED (1)
+#define ACPI_HMAT_CA_COMPLEX_CACHE_INDEXING (2)
+
+/* Values for write policy flag */
+
+#define ACPI_HMAT_CP_NONE (0)
+#define ACPI_HMAT_CP_WB (1)
+#define ACPI_HMAT_CP_WT (2)
+
/*******************************************************************************
*
* MADT - Multiple APIC Description Table
/* Values for PCATCompat flag */
-#define ACPI_MADT_DUAL_PIC 0
-#define ACPI_MADT_MULTIPLE_APIC 1
+#define ACPI_MADT_DUAL_PIC 1
+#define ACPI_MADT_MULTIPLE_APIC 0
/* Values for MADT subtable type in struct acpi_subtable_header */
u64 hint_address[1]; /* Variable length */
};
+/*******************************************************************************
+ *
+ * PPTT - Processor Properties Topology Table (ACPI 6.2)
+ * Version 1
+ *
+ ******************************************************************************/
+
+struct acpi_table_pptt {
+ struct acpi_table_header header; /* Common ACPI table header */
+};
+
+/* Values for Type field above */
+
+enum acpi_pptt_type {
+ ACPI_PPTT_TYPE_PROCESSOR = 0,
+ ACPI_PPTT_TYPE_CACHE = 1,
+ ACPI_PPTT_TYPE_ID = 2,
+ ACPI_PPTT_TYPE_RESERVED = 3
+};
+
+/* 0: Processor Hierarchy Node Structure */
+
+struct acpi_pptt_processor {
+ struct acpi_subtable_header header;
+ u16 reserved;
+ u32 flags;
+ u32 parent;
+ u32 acpi_processor_id;
+ u32 number_of_priv_resources;
+};
+
+/* Flags */
+
+#define ACPI_PPTT_PHYSICAL_PACKAGE (1) /* Physical package */
+#define ACPI_PPTT_ACPI_PROCESSOR_ID_VALID (2) /* ACPI Processor ID valid */
+
+/* 1: Cache Type Structure */
+
+struct acpi_pptt_cache {
+ struct acpi_subtable_header header;
+ u16 reserved;
+ u32 flags;
+ u32 next_level_of_cache;
+ u32 size;
+ u32 number_of_sets;
+ u8 associativity;
+ u8 attributes;
+ u16 line_size;
+};
+
+/* Flags */
+
+#define ACPI_PPTT_SIZE_PROPERTY_VALID (1) /* Physical property valid */
+#define ACPI_PPTT_NUMBER_OF_SETS_VALID (1<<1) /* Number of sets valid */
+#define ACPI_PPTT_ASSOCIATIVITY_VALID (1<<2) /* Associativity valid */
+#define ACPI_PPTT_ALLOCATION_TYPE_VALID (1<<3) /* Allocation type valid */
+#define ACPI_PPTT_CACHE_TYPE_VALID (1<<4) /* Cache type valid */
+#define ACPI_PPTT_WRITE_POLICY_VALID (1<<5) /* Write policy valid */
+#define ACPI_PPTT_LINE_SIZE_VALID (1<<6) /* Line size valid */
+
+/* Masks for Attributes */
+
+#define ACPI_PPTT_MASK_ALLOCATION_TYPE (0x03) /* Allocation type */
+#define ACPI_PPTT_MASK_CACHE_TYPE (0x0C) /* Cache type */
+#define ACPI_PPTT_MASK_WRITE_POLICY (0x10) /* Write policy */
+
+/* 2: ID Structure */
+
+struct acpi_pptt_id {
+ struct acpi_subtable_header header;
+ u16 reserved;
+ u32 vendor_id;
+ u64 level1_id;
+ u64 level2_id;
+ u16 major_rev;
+ u16 minor_rev;
+ u16 spin_rev;
+};
+
/*******************************************************************************
*
* SBST - Smart Battery Specification Table
ACPI_SRAT_TYPE_MEMORY_AFFINITY = 1,
ACPI_SRAT_TYPE_X2APIC_CPU_AFFINITY = 2,
ACPI_SRAT_TYPE_GICC_AFFINITY = 3,
- ACPI_SRAT_TYPE_RESERVED = 4 /* 4 and greater are reserved */
+ ACPI_SRAT_TYPE_GIC_ITS_AFFINITY = 4, /* ACPI 6.2 */
+ ACPI_SRAT_TYPE_RESERVED = 5 /* 5 and greater are reserved */
};
/*
#define ACPI_SRAT_GICC_ENABLED (1) /* 00: Use affinity structure */
+/* 4: GCC ITS Affinity (ACPI 6.2) */
+
+struct acpi_srat_gic_its_affinity {
+ struct acpi_subtable_header header;
+ u32 proximity_domain;
+ u16 reserved;
+ u32 its_id;
+};
+
/* Reset to default packing */
#pragma pack()
#define ACPI_SIG_WDAT "WDAT" /* Watchdog Action Table */
#define ACPI_SIG_WDDT "WDDT" /* Watchdog Timer Description Table */
#define ACPI_SIG_WDRT "WDRT" /* Watchdog Resource Table */
+#define ACPI_SIG_WSMT "WSMT" /* Windows SMM Security Migrations Table */
#define ACPI_SIG_XXXX "XXXX" /* Intermediate AML header for ASL/ASL+ converter */
#ifdef ACPI_UNDEFINED_TABLES
* Version 2
*
* Conforms to "TCG ACPI Specification, Family 1.2 and 2.0",
- * December 19, 2014
+ * Version 1.2, Revision 8
+ * February 27, 2017
*
* NOTE: There are two versions of the table with the same signature --
* the client version and the server version. The common platform_class
* Version 4
*
* Conforms to "TCG ACPI Specification, Family 1.2 and 2.0",
- * December 19, 2014
+ * Version 1.2, Revision 8
+ * February 27, 2017
*
******************************************************************************/
#define ACPI_TPM2_MEMORY_MAPPED 6
#define ACPI_TPM2_COMMAND_BUFFER 7
#define ACPI_TPM2_COMMAND_BUFFER_WITH_START_METHOD 8
-#define ACPI_TPM2_COMMAND_BUFFER_WITH_SMC 11
+#define ACPI_TPM2_COMMAND_BUFFER_WITH_ARM_SMC 11 /* V1.2 Rev 8 */
+
+/* Trailer appears after any start_method subtables */
+
+struct acpi_tpm2_trailer {
+ u32 minimum_log_length; /* Minimum length for the event log area */
+ u64 log_address; /* Address of the event log area */
+};
+
+/*
+ * Subtables (start_method-specific)
+ */
+
+/* 11: Start Method for ARM SMC (V1.2 Rev 8) */
+
+struct acpi_tpm2_arm_smc {
+ u32 global_interrupt;
+ u8 interrupt_flags;
+ u8 operation_flags;
+ u16 reserved;
+ u32 function_id;
+};
+
+/* Values for interrupt_flags above */
+
+#define ACPI_TPM2_INTERRUPT_SUPPORT (1)
+
+/* Values for operation_flags above */
+
+#define ACPI_TPM2_IDLE_SUPPORT (1)
/*******************************************************************************
*
u8 units;
};
+/*******************************************************************************
+ *
+ * WSMT - Windows SMM Security Migrations Table
+ * Version 1
+ *
+ * Conforms to "Windows SMM Security Migrations Table",
+ * Version 1.0, April 18, 2016
+ *
+ ******************************************************************************/
+
+struct acpi_table_wsmt {
+ struct acpi_table_header header; /* Common ACPI table header */
+ u32 protection_flags;
+};
+
+/* Flags for protection_flags field above */
+
+#define ACPI_WSMT_FIXED_COMM_BUFFERS (1)
+#define ACPI_WSMT_COMM_BUFFER_NESTED_PTR_PROTECTION (2)
+#define ACPI_WSMT_SYSTEM_RESOURCE_PROTECTION (4)
+
/* Reset to default packing */
#pragma pack()
u32 image_offset_y;
};
+/* Flags for Status field above */
+
+#define ACPI_BGRT_DISPLAYED (1)
+#define ACPI_BGRT_ORIENTATION_OFFSET (3 << 1)
+
/*******************************************************************************
*
* DRTM - Dynamic Root of Trust for Measurement table
/*******************************************************************************
*
* PCCT - Platform Communications Channel Table (ACPI 5.0)
- * Version 1
+ * Version 2 (ACPI 6.2)
*
******************************************************************************/
ACPI_PCCT_TYPE_GENERIC_SUBSPACE = 0,
ACPI_PCCT_TYPE_HW_REDUCED_SUBSPACE = 1,
ACPI_PCCT_TYPE_HW_REDUCED_SUBSPACE_TYPE2 = 2, /* ACPI 6.1 */
- ACPI_PCCT_TYPE_RESERVED = 3 /* 3 and greater are reserved */
+ ACPI_PCCT_TYPE_EXT_PCC_MASTER_SUBSPACE = 3, /* ACPI 6.2 */
+ ACPI_PCCT_TYPE_EXT_PCC_SLAVE_SUBSPACE = 4, /* ACPI 6.2 */
+ ACPI_PCCT_TYPE_RESERVED = 5 /* 5 and greater are reserved */
};
/*
struct acpi_pcct_hw_reduced {
struct acpi_subtable_header header;
- u32 doorbell_interrupt;
+ u32 platform_interrupt;
u8 flags;
u8 reserved;
u64 base_address;
struct acpi_pcct_hw_reduced_type2 {
struct acpi_subtable_header header;
- u32 doorbell_interrupt;
+ u32 platform_interrupt;
u8 flags;
u8 reserved;
u64 base_address;
u32 latency;
u32 max_access_rate;
u16 min_turnaround_time;
- struct acpi_generic_address doorbell_ack_register;
+ struct acpi_generic_address platform_ack_register;
u64 ack_preserve_mask;
u64 ack_write_mask;
};
+/* 3: Extended PCC Master Subspace Type 3 (ACPI 6.2) */
+
+struct acpi_pcct_ext_pcc_master {
+ struct acpi_subtable_header header;
+ u32 platform_interrupt;
+ u8 flags;
+ u8 reserved1;
+ u64 base_address;
+ u32 length;
+ struct acpi_generic_address doorbell_register;
+ u64 preserve_mask;
+ u64 write_mask;
+ u32 latency;
+ u32 max_access_rate;
+ u32 min_turnaround_time;
+ struct acpi_generic_address platform_ack_register;
+ u64 ack_preserve_mask;
+ u64 ack_set_mask;
+ u64 reserved2;
+ struct acpi_generic_address cmd_complete_register;
+ u64 cmd_complete_mask;
+ struct acpi_generic_address cmd_update_register;
+ u64 cmd_update_preserve_mask;
+ u64 cmd_update_set_mask;
+ struct acpi_generic_address error_status_register;
+ u64 error_status_mask;
+};
+
+/* 4: Extended PCC Slave Subspace Type 4 (ACPI 6.2) */
+
+struct acpi_pcct_ext_pcc_slave {
+ struct acpi_subtable_header header;
+ u32 platform_interrupt;
+ u8 flags;
+ u8 reserved1;
+ u64 base_address;
+ u32 length;
+ struct acpi_generic_address doorbell_register;
+ u64 preserve_mask;
+ u64 write_mask;
+ u32 latency;
+ u32 max_access_rate;
+ u32 min_turnaround_time;
+ struct acpi_generic_address platform_ack_register;
+ u64 ack_preserve_mask;
+ u64 ack_set_mask;
+ u64 reserved2;
+ struct acpi_generic_address cmd_complete_register;
+ u64 cmd_complete_mask;
+ struct acpi_generic_address cmd_update_register;
+ u64 cmd_update_preserve_mask;
+ u64 cmd_update_set_mask;
+ struct acpi_generic_address error_status_register;
+ u64 error_status_mask;
+};
+
/* Values for doorbell flags above */
#define ACPI_PCCT_INTERRUPT_POLARITY (1)
u16 status;
};
+/* Extended PCC Subspace Shared Memory Region (ACPI 6.2) */
+
+struct acpi_pcct_ext_pcc_shared_memory {
+ u32 signature;
+ u32 flags;
+ u32 length;
+ u32 command;
+};
+
/*******************************************************************************
*
* PMTT - Platform Memory Topology Table (ACPI 5.0)
/* acpisrc:struct_defs -- for acpisrc conversion */
/*
- * ACPI_MACHINE_WIDTH must be specified in an OS- or compiler-dependent header
- * and must be either 32 or 64. 16-bit ACPICA is no longer supported, as of
- * 12/2006.
+ * ACPI_MACHINE_WIDTH must be specified in an OS- or compiler-dependent
+ * header and must be either 32 or 64. 16-bit ACPICA is no longer
+ * supported, as of 12/2006.
*/
#ifndef ACPI_MACHINE_WIDTH
#error ACPI_MACHINE_WIDTH not defined
* s64 64-bit (8 byte) signed value
*
* COMPILER_DEPENDENT_UINT64/s64 - These types are defined in the
- * compiler-dependent header(s) and were introduced because there is no common
- * 64-bit integer type across the various compilation models, as shown in
- * the table below.
+ * compiler-dependent header(s) and were introduced because there is no
+ * common 64-bit integer type across the various compilation models, as
+ * shown in the table below.
*
* Datatype LP64 ILP64 LLP64 ILP32 LP32 16bit
* char 8 8 8 8 8 8
* 2) These types represent the native word size of the target mode of the
* processor, and may be 16-bit, 32-bit, or 64-bit as required. They are
* usually used for memory allocation, efficient loop counters, and array
- * indexes. The types are similar to the size_t type in the C library and are
- * required because there is no C type that consistently represents the native
- * data width. acpi_size is needed because there is no guarantee that a
- * kernel-level C library is present.
+ * indexes. The types are similar to the size_t type in the C library and
+ * are required because there is no C type that consistently represents the
+ * native data width. acpi_size is needed because there is no guarantee
+ * that a kernel-level C library is present.
*
* acpi_size 16/32/64-bit unsigned value
* acpi_native_int 16/32/64-bit signed value
/*
* In the case of the Itanium Processor Family (IPF), the hardware does not
- * support misaligned memory transfers. Set the MISALIGNMENT_NOT_SUPPORTED flag
- * to indicate that special precautions must be taken to avoid alignment faults.
- * (IA64 or ia64 is currently used by existing compilers to indicate IPF.)
+ * support misaligned memory transfers. Set the MISALIGNMENT_NOT_SUPPORTED
+ * flag to indicate that special precautions must be taken to avoid alignment
+ * faults. (IA64 or ia64 is currently used by existing compilers to indicate
+ * IPF.)
*
* Note: EM64T and other X86-64 processors support misaligned transfers,
* so there is no need to define this flag.
#endif
/*
- * Some compilers complain about unused variables. Sometimes we don't want to
- * use all the variables (for example, _acpi_module_name). This allows us
+ * Some compilers complain about unused variables. Sometimes we don't want
+ * to use all the variables (for example, _acpi_module_name). This allows us
* to tell the compiler in a per-variable manner that a variable
* is unused
*/
#endif
/*
- * All ACPICA external functions that are available to the rest of the kernel
- * are tagged with thes macros which can be defined as appropriate for the host.
+ * All ACPICA external functions that are available to the rest of the
+ * kernel are tagged with these macros which can be defined as appropriate
+ * for the host.
*
* Notes:
* ACPI_EXPORT_SYMBOL_INIT is used for initialization and termination
/******************************************************************************
*
- * ACPI Specification constants (Do not change unless the specification changes)
+ * ACPI Specification constants (Do not change unless the specification
+ * changes)
*
*****************************************************************************/
#define ACPI_DO_NOT_WAIT 0
/*
- * Obsolete: Acpi integer width. In ACPI version 1 (1996), integers are 32 bits.
- * In ACPI version 2 (2000) and later, integers are 64 bits. Note that this
- * pertains to the ACPI integer type only, not to other integers used in the
- * implementation of the ACPICA subsystem.
+ * Obsolete: Acpi integer width. In ACPI version 1 (1996), integers are
+ * 32 bits. In ACPI version 2 (2000) and later, integers are max 64 bits.
+ * Note that this pertains to the ACPI integer type only, not to other
+ * integers used in the implementation of the ACPICA subsystem.
*
* 01/2010: This type is obsolete and has been removed from the entire ACPICA
* code base. It remains here for compatibility with device drivers that use
#define ACPI_NOTIFY_LOCALITY_UPDATE (u8) 0x0B
#define ACPI_NOTIFY_SHUTDOWN_REQUEST (u8) 0x0C
#define ACPI_NOTIFY_AFFINITY_UPDATE (u8) 0x0D
+#define ACPI_NOTIFY_MEMORY_UPDATE (u8) 0x0E
-#define ACPI_GENERIC_NOTIFY_MAX 0x0D
+#define ACPI_GENERIC_NOTIFY_MAX 0x0E
#define ACPI_SPECIFIC_NOTIFY_MAX 0x84
/*
/*
* These are object types that do not map directly to the ACPI
- * object_type() operator. They are used for various internal purposes only.
- * If new predefined ACPI_TYPEs are added (via the ACPI specification), these
- * internal types must move upwards. (There is code that depends on these
- * values being contiguous with the external types above.)
+ * object_type() operator. They are used for various internal purposes
+ * only. If new predefined ACPI_TYPEs are added (via the ACPI
+ * specification), these internal types must move upwards. (There
+ * is code that depends on these values being contiguous with the
+ * external types above.)
*/
#define ACPI_TYPE_LOCAL_REGION_FIELD 0x11
#define ACPI_TYPE_LOCAL_BANK_FIELD 0x12
* | | | | +-- Type of dispatch:to method, handler, notify, or none
* | | | +----- Interrupt type: edge or level triggered
* | | +------- Is a Wake GPE
- * | +--------- Is GPE masked by the software GPE masking machanism
+ * | +--------- Is GPE masked by the software GPE masking mechanism
* +------------ <Reserved>
*/
#define ACPI_GPE_DISPATCH_NONE (u8) 0x00
*/
/*
- * Note: Type == ACPI_TYPE_ANY (0) is used to indicate a NULL package element
- * or an unresolved named reference.
+ * Note: Type == ACPI_TYPE_ANY (0) is used to indicate a NULL package
+ * element or an unresolved named reference.
*/
union acpi_object {
acpi_object_type type; /* See definition of acpi_ns_type for values */
/*
* Structure returned from acpi_get_object_info.
- * Optimized for both 32- and 64-bit builds
+ * Optimized for both 32-bit and 64-bit builds.
*/
struct acpi_device_info {
u32 info_size; /* Size of info, including ID strings */
#define UUID_PERSISTENT_VIRTUAL_DISK "5cea02c9-4d07-69d3-269f-4496fbe096f9"
#define UUID_PERSISTENT_VIRTUAL_CD "08018188-42cd-bb48-100f-5387d53ded3d"
+/* Processor Properties (ACPI 6.2) */
+
+#define UUID_CACHE_PROPERTIES "6DC63E77-257E-4E78-A973-A21F2796898D"
+#define UUID_PHYSICAL_PROPERTY "DDE4D59A-AA42-4349-B407-EA40F57D9FB7"
+
/* Miscellaneous */
#define UUID_PLATFORM_CAPABILITIES "0811b06e-4a27-44f9-8d60-3cbbc22e7b48"
#define ACPI_INIT_FUNCTION
#endif
+#ifndef ACPI_STRUCT_INIT
+#define ACPI_STRUCT_INIT(field, value) value
+#endif
+
#endif /* __ACENV_H__ */
* Use compiler specific <stdarg.h> is a good practice for even when
* -nostdinc is specified (i.e., ACPI_USE_STANDARD_HEADERS undefined.
*/
+#ifndef va_arg
+#ifdef ACPI_USE_BUILTIN_STDARG
+typedef __builtin_va_list va_list;
+#define va_start(v, l) __builtin_va_start(v, l)
+#define va_end(v) __builtin_va_end(v)
+#define va_arg(v, l) __builtin_va_arg(v, l)
+#define va_copy(d, s) __builtin_va_copy(d, s)
+#else
#include <stdarg.h>
+#endif
+#endif
#define ACPI_INLINE __inline__
* Use compiler specific <stdarg.h> is a good practice for even when
* -nostdinc is specified (i.e., ACPI_USE_STANDARD_HEADERS undefined.
*/
+#ifndef va_arg
#include <stdarg.h>
+#endif
/* Configuration specific to Intel 64-bit C compiler */
#define ACPI_MSG_BIOS_ERROR KERN_ERR "ACPI BIOS Error (bug): "
#define ACPI_MSG_BIOS_WARNING KERN_WARNING "ACPI BIOS Warning (bug): "
+#define ACPI_STRUCT_INIT(field, value) .field = value
+
#else /* !__KERNEL__ */
#define ACPI_USE_STANDARD_HEADERS
+++ /dev/null
-#ifndef _ASM_GENERIC_SIGINFO_H
-#define _ASM_GENERIC_SIGINFO_H
-
-#include <uapi/asm-generic/siginfo.h>
-
-#define __SI_MASK 0xffff0000u
-#define __SI_KILL (0 << 16)
-#define __SI_TIMER (1 << 16)
-#define __SI_POLL (2 << 16)
-#define __SI_FAULT (3 << 16)
-#define __SI_CHLD (4 << 16)
-#define __SI_RT (5 << 16)
-#define __SI_MESGQ (6 << 16)
-#define __SI_SYS (7 << 16)
-#define __SI_CODE(T,N) ((T) | ((N) & 0xffff))
-
-struct siginfo;
-void do_schedule_next_timer(struct siginfo *info);
-
-extern int copy_siginfo_to_user(struct siginfo __user *to, const struct siginfo *from);
-
-#endif
KEEP(*(__##name##_of_table)) \
KEEP(*(__##name##_of_table_end))
-#define CLKSRC_OF_TABLES() OF_TABLE(CONFIG_CLKSRC_OF, clksrc)
-#define CLKEVT_OF_TABLES() OF_TABLE(CONFIG_CLKEVT_OF, clkevt)
+#define TIMER_OF_TABLES() OF_TABLE(CONFIG_TIMER_OF, timer)
#define IRQCHIP_OF_MATCH_TABLE() OF_TABLE(CONFIG_IRQCHIP, irqchip)
#define CLK_OF_TABLES() OF_TABLE(CONFIG_COMMON_CLK, clk)
#define IOMMU_OF_TABLES() OF_TABLE(CONFIG_OF_IOMMU, iommu)
MEM_DISCARD(init.rodata) \
CLK_OF_TABLES() \
RESERVEDMEM_OF_TABLES() \
- CLKSRC_OF_TABLES() \
- CLKEVT_OF_TABLES() \
+ TIMER_OF_TABLES() \
IOMMU_OF_TABLES() \
CPU_METHOD_OF_TABLES() \
CPUIDLE_METHOD_OF_TABLES() \
KERNEL_DTB() \
IRQCHIP_OF_MATCH_TABLE() \
ACPI_PROBE_TABLE(irqchip) \
- ACPI_PROBE_TABLE(clksrc) \
+ ACPI_PROBE_TABLE(timer) \
+ ACPI_PROBE_TABLE(iort) \
EARLYCON_TABLE()
#define INIT_TEXT \
#ifndef _DT_BINDINGS_CLK_SUN50I_A64_H_
#define _DT_BINDINGS_CLK_SUN50I_A64_H_
+#define CLK_PLL_PERIPH0 11
+
#define CLK_BUS_MIPI_DSI 28
#define CLK_BUS_CE 29
#define CLK_BUS_DMA 30
#ifndef _DT_BINDINGS_CLK_SUN8I_H3_H_
#define _DT_BINDINGS_CLK_SUN8I_H3_H_
+#define CLK_PLL_PERIPH0 9
+
#define CLK_CPUX 14
#define CLK_BUS_CE 20
--- /dev/null
+/*
+ * This header provides constants for the MVEBU ICU driver.
+ */
+
+#ifndef _DT_BINDINGS_INTERRUPT_CONTROLLER_MVEBU_ICU_H
+#define _DT_BINDINGS_INTERRUPT_CONTROLLER_MVEBU_ICU_H
+
+/* interrupt specifier cell 0 */
+
+#define ICU_GRP_NSR 0x0
+#define ICU_GRP_SR 0x1
+#define ICU_GRP_SEI 0x4
+#define ICU_GRP_REI 0x5
+
+#endif
--- /dev/null
+/*
+ * This header provides constants for most Multiplexer bindings.
+ *
+ * Most Multiplexer bindings specify an idle state. In most cases, the
+ * the multiplexer can be left as is when idle, and in some cases it can
+ * disconnect the input/output and leave the multiplexer in a high
+ * impedance state.
+ */
+
+#ifndef _DT_BINDINGS_MUX_MUX_H
+#define _DT_BINDINGS_MUX_MUX_H
+
+#define MUX_IDLE_AS_IS (-1)
+#define MUX_IDLE_DISCONNECT (-2)
+
+#endif
#include <linux/resource_ext.h>
#include <linux/device.h>
#include <linux/property.h>
+#include <linux/uuid.h>
#ifndef _LINUX
#define _LINUX
struct acpi_buffer ret; /* free by caller if success */
};
-acpi_status acpi_str_to_uuid(char *str, u8 *uuid);
acpi_status acpi_run_osc(acpi_handle handle, struct acpi_osc_context *context);
/* Indexes into _OSC Capabilities Buffer (DWORDs 2 & 3 are device-specific) */
}
static inline union acpi_object *acpi_evaluate_dsm(acpi_handle handle,
- const u8 *uuid,
+ const guid_t *guid,
int rev, int func,
union acpi_object *argv4)
{
--- /dev/null
+/*
+ * include/linux/arch_topology.h - arch specific cpu topology information
+ */
+#ifndef _LINUX_ARCH_TOPOLOGY_H_
+#define _LINUX_ARCH_TOPOLOGY_H_
+
+void topology_normalize_cpu_scale(void);
+
+struct device_node;
+int topology_parse_cpu_capacity(struct device_node *cpu_node, int cpu);
+
+struct sched_domain;
+unsigned long topology_get_cpu_scale(struct sched_domain *sd, int cpu);
+
+void topology_set_cpu_scale(unsigned int cpu, unsigned long capacity);
+
+#endif /* _LINUX_ARCH_TOPOLOGY_H_ */
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
* Hardware documentation available from http://www.t13.org/
*
#ifndef __BCM47XX_NVRAM_H
#define __BCM47XX_NVRAM_H
+#include <linux/errno.h>
#include <linux/types.h>
#include <linux/kernel.h>
#include <linux/vmalloc.h>
/*
* will die
*/
-#define bio_to_phys(bio) (page_to_phys(bio_page((bio))) + (unsigned long) bio_offset((bio)))
#define bvec_to_phys(bv) (page_to_phys((bv)->bv_page) + (unsigned long) (bv)->bv_offset)
/*
return bio_split(bio, sectors, gfp, bs);
}
-extern struct bio_set *bioset_create(unsigned int, unsigned int);
-extern struct bio_set *bioset_create_nobvec(unsigned int, unsigned int);
+extern struct bio_set *bioset_create(unsigned int, unsigned int, int flags);
+enum {
+ BIOSET_NEED_BVECS = BIT(0),
+ BIOSET_NEED_RESCUER = BIT(1),
+};
extern void bioset_free(struct bio_set *);
extern mempool_t *biovec_create_pool(int pool_entries);
return bio_alloc_bioset(gfp_mask, nr_iovecs, fs_bio_set);
}
-static inline struct bio *bio_clone(struct bio *bio, gfp_t gfp_mask)
-{
- return bio_clone_bioset(bio, gfp_mask, fs_bio_set);
-}
-
static inline struct bio *bio_kmalloc(gfp_t gfp_mask, unsigned int nr_iovecs)
{
return bio_alloc_bioset(gfp_mask, nr_iovecs, NULL);
static inline void bio_io_error(struct bio *bio)
{
- bio->bi_error = -EIO;
+ bio->bi_status = BLK_STS_IOERR;
+ bio_endio(bio);
+}
+
+static inline void bio_wouldblock_error(struct bio *bio)
+{
+ bio->bi_status = BLK_STS_AGAIN;
bio_endio(bio);
}
extern void bio_init(struct bio *bio, struct bio_vec *table,
unsigned short max_vecs);
+extern void bio_uninit(struct bio *);
extern void bio_reset(struct bio *);
void bio_chain(struct bio *, struct bio *);
struct blk_mq_ctx **ctxs;
unsigned int nr_ctx;
- wait_queue_t dispatch_wait;
+ wait_queue_entry_t dispatch_wait;
atomic_t wait_index;
struct blk_mq_tags *tags;
struct blk_mq_tags *sched_tags;
- struct srcu_struct queue_rq_srcu;
-
unsigned long queued;
unsigned long run;
#define BLK_MQ_MAX_DISPATCH_ORDER 7
struct dentry *debugfs_dir;
struct dentry *sched_debugfs_dir;
#endif
+
+ /* Must be the last member - see also blk_mq_hw_ctx_size(). */
+ struct srcu_struct queue_rq_srcu[0];
};
struct blk_mq_tag_set {
bool last;
};
-typedef int (queue_rq_fn)(struct blk_mq_hw_ctx *, const struct blk_mq_queue_data *);
+typedef blk_status_t (queue_rq_fn)(struct blk_mq_hw_ctx *,
+ const struct blk_mq_queue_data *);
typedef enum blk_eh_timer_return (timeout_fn)(struct request *, bool);
typedef int (init_hctx_fn)(struct blk_mq_hw_ctx *, void *, unsigned int);
typedef void (exit_hctx_fn)(struct blk_mq_hw_ctx *, unsigned int);
init_request_fn *init_request;
exit_request_fn *exit_request;
reinit_request_fn *reinit_request;
+ /* Called from inside blk_get_request() */
+ void (*initialize_rq_fn)(struct request *rq);
map_queues_fn *map_queues;
};
enum {
- BLK_MQ_RQ_QUEUE_OK = 0, /* queued fine */
- BLK_MQ_RQ_QUEUE_BUSY = 1, /* requeue IO for later */
- BLK_MQ_RQ_QUEUE_ERROR = 2, /* end IO with error */
-
BLK_MQ_F_SHOULD_MERGE = 1 << 0,
BLK_MQ_F_TAG_SHARED = 1 << 1,
BLK_MQ_F_SG_MERGE = 1 << 2,
BLK_MQ_REQ_INTERNAL = (1 << 2), /* allocate internal/sched tag */
};
-struct request *blk_mq_alloc_request(struct request_queue *q, int rw,
+struct request *blk_mq_alloc_request(struct request_queue *q, unsigned int op,
unsigned int flags);
-struct request *blk_mq_alloc_request_hctx(struct request_queue *q, int op,
- unsigned int flags, unsigned int hctx_idx);
+struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
+ unsigned int op, unsigned int flags, unsigned int hctx_idx);
struct request *blk_mq_tag_to_rq(struct blk_mq_tags *tags, unsigned int tag);
enum {
int blk_mq_request_started(struct request *rq);
void blk_mq_start_request(struct request *rq);
-void blk_mq_end_request(struct request *rq, int error);
-void __blk_mq_end_request(struct request *rq, int error);
+void blk_mq_end_request(struct request *rq, blk_status_t error);
+void __blk_mq_end_request(struct request *rq, blk_status_t error);
void blk_mq_requeue_request(struct request *rq, bool kick_requeue_list);
void blk_mq_add_to_requeue_list(struct request *rq, bool at_head,
void blk_mq_start_hw_queues(struct request_queue *q);
void blk_mq_start_stopped_hw_queue(struct blk_mq_hw_ctx *hctx, bool async);
void blk_mq_start_stopped_hw_queues(struct request_queue *q, bool async);
+void blk_mq_quiesce_queue(struct request_queue *q);
+void blk_mq_unquiesce_queue(struct request_queue *q);
void blk_mq_delay_run_hw_queue(struct blk_mq_hw_ctx *hctx, unsigned long msecs);
void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async);
void blk_mq_run_hw_queues(struct request_queue *q, bool async);
int blk_mq_map_queues(struct blk_mq_tag_set *set);
void blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, int nr_hw_queues);
+void blk_mq_quiesce_queue_nowait(struct request_queue *q);
+
/*
* Driver command data is immediately after the request. So subtract request
* size to get back to the original request, add request size to get the PDU.
struct cgroup_subsys_state;
typedef void (bio_end_io_t) (struct bio *);
+/*
+ * Block error status values. See block/blk-core:blk_errors for the details.
+ */
+typedef u8 __bitwise blk_status_t;
+#define BLK_STS_OK 0
+#define BLK_STS_NOTSUPP ((__force blk_status_t)1)
+#define BLK_STS_TIMEOUT ((__force blk_status_t)2)
+#define BLK_STS_NOSPC ((__force blk_status_t)3)
+#define BLK_STS_TRANSPORT ((__force blk_status_t)4)
+#define BLK_STS_TARGET ((__force blk_status_t)5)
+#define BLK_STS_NEXUS ((__force blk_status_t)6)
+#define BLK_STS_MEDIUM ((__force blk_status_t)7)
+#define BLK_STS_PROTECTION ((__force blk_status_t)8)
+#define BLK_STS_RESOURCE ((__force blk_status_t)9)
+#define BLK_STS_IOERR ((__force blk_status_t)10)
+
+/* hack for device mapper, don't use elsewhere: */
+#define BLK_STS_DM_REQUEUE ((__force blk_status_t)11)
+
+#define BLK_STS_AGAIN ((__force blk_status_t)12)
+
struct blk_issue_stat {
u64 stat;
};
struct bio {
struct bio *bi_next; /* request queue link */
struct block_device *bi_bdev;
- int bi_error;
+ blk_status_t bi_status;
unsigned int bi_opf; /* bottom bits req flags,
* top bits REQ_OP. Use
* accessors.
*/
unsigned short bi_flags; /* status, etc and bvec pool number */
unsigned short bi_ioprio;
+ unsigned short bi_write_hint;
struct bvec_iter bi_iter;
/* command specific flags for REQ_OP_WRITE_ZEROES: */
__REQ_NOUNMAP, /* do not free blocks when zeroing */
+ __REQ_NOWAIT, /* Don't wait if request will block */
__REQ_NR_BITS, /* stops here */
};
#define REQ_BACKGROUND (1ULL << __REQ_BACKGROUND)
#define REQ_NOUNMAP (1ULL << __REQ_NOUNMAP)
+#define REQ_NOWAIT (1ULL << __REQ_NOWAIT)
#define REQ_FAILFAST_MASK \
(REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT | REQ_FAILFAST_DRIVER)
*/
#define BLKCG_MAX_POLS 3
-typedef void (rq_end_io_fn)(struct request *, int);
+typedef void (rq_end_io_fn)(struct request *, blk_status_t);
#define BLK_RL_SYNCFULL (1U << 0)
#define BLK_RL_ASYNCFULL (1U << 1)
unsigned int extra_len; /* length of alignment and padding */
+ unsigned short write_hint;
+
unsigned long deadline;
struct list_head timeout_list;
int nr_rqs[2]; /* # allocated [a]sync rqs */
int nr_rqs_elvpriv; /* # allocated rqs w/ elvpriv */
+ atomic_t shared_hctx_restart;
+
struct blk_queue_stats *stats;
struct rq_wb *rq_wb;
rq_timed_out_fn *rq_timed_out_fn;
dma_drain_needed_fn *dma_drain_needed;
lld_busy_fn *lld_busy_fn;
+ /* Called just after a request is allocated */
init_rq_fn *init_rq_fn;
+ /* Called just before a request is freed */
exit_rq_fn *exit_rq_fn;
+ /* Called from inside blk_get_request() */
+ void (*initialize_rq_fn)(struct request *rq);
const struct blk_mq_ops *mq_ops;
void *rq_alloc_data;
struct work_struct release_work;
+
+#define BLK_MAX_WRITE_HINTS 5
+ u64 write_hints[BLK_MAX_WRITE_HINTS];
};
#define QUEUE_FLAG_QUEUED 1 /* uses generic tag queueing */
#define QUEUE_FLAG_STATS 27 /* track rq completion times */
#define QUEUE_FLAG_POLL_STATS 28 /* collecting stats for hybrid polling */
#define QUEUE_FLAG_REGISTERED 29 /* queue has been registered to a disk */
+#define QUEUE_FLAG_SCSI_PASSTHROUGH 30 /* queue supports SCSI commands */
+#define QUEUE_FLAG_QUIESCED 31 /* queue has been quiesced */
#define QUEUE_FLAG_DEFAULT ((1 << QUEUE_FLAG_IO_STAT) | \
(1 << QUEUE_FLAG_STACKABLE) | \
(1 << QUEUE_FLAG_SAME_COMP) | \
(1 << QUEUE_FLAG_POLL))
+/*
+ * @q->queue_lock is set while a queue is being initialized. Since we know
+ * that no other threads access the queue object before @q->queue_lock has
+ * been set, it is safe to manipulate queue flags without holding the
+ * queue_lock if @q->queue_lock == NULL. See also blk_alloc_queue_node() and
+ * blk_init_allocated_queue().
+ */
static inline void queue_lockdep_assert_held(struct request_queue *q)
{
if (q->queue_lock)
#define blk_queue_secure_erase(q) \
(test_bit(QUEUE_FLAG_SECERASE, &(q)->queue_flags))
#define blk_queue_dax(q) test_bit(QUEUE_FLAG_DAX, &(q)->queue_flags)
+#define blk_queue_scsi_passthrough(q) \
+ test_bit(QUEUE_FLAG_SCSI_PASSTHROUGH, &(q)->queue_flags)
#define blk_noretry_request(rq) \
((rq)->cmd_flags & (REQ_FAILFAST_DEV|REQ_FAILFAST_TRANSPORT| \
REQ_FAILFAST_DRIVER))
+#define blk_queue_quiesced(q) test_bit(QUEUE_FLAG_QUIESCED, &(q)->queue_flags)
static inline bool blk_account_rq(struct request *rq)
{
static inline bool blk_write_same_mergeable(struct bio *a, struct bio *b)
{
- if (bio_data(a) == bio_data(b))
+ if (bio_page(a) == bio_page(b) &&
+ bio_offset(a) == bio_offset(b))
return true;
return false;
#define BLK_DEFAULT_SG_TIMEOUT (60 * HZ)
#define BLK_MIN_SG_TIMEOUT (7 * HZ)
-#ifdef CONFIG_BOUNCE
-extern int init_emergency_isa_pool(void);
-extern void blk_queue_bounce(struct request_queue *q, struct bio **bio);
-#else
-static inline int init_emergency_isa_pool(void)
-{
- return 0;
-}
-static inline void blk_queue_bounce(struct request_queue *q, struct bio **bio)
-{
-}
-#endif /* CONFIG_MMU */
-
struct rq_map_data {
struct page **pages;
int page_order;
extern void blk_init_request_from_bio(struct request *req, struct bio *bio);
extern void blk_put_request(struct request *);
extern void __blk_put_request(struct request_queue *, struct request *);
-extern struct request *blk_get_request(struct request_queue *, int, gfp_t);
+extern struct request *blk_get_request(struct request_queue *, unsigned int op,
+ gfp_t gfp_mask);
extern void blk_requeue_request(struct request_queue *, struct request *);
extern int blk_lld_busy(struct request_queue *q);
extern int blk_rq_prep_clone(struct request *rq, struct request *rq_src,
int (*bio_ctr)(struct bio *, struct bio *, void *),
void *data);
extern void blk_rq_unprep_clone(struct request *rq);
-extern int blk_insert_cloned_request(struct request_queue *q,
+extern blk_status_t blk_insert_cloned_request(struct request_queue *q,
struct request *rq);
extern int blk_rq_append_bio(struct request *rq, struct bio *bio);
extern void blk_delay_queue(struct request_queue *, unsigned long);
-extern void blk_queue_split(struct request_queue *, struct bio **,
- struct bio_set *);
+extern void blk_queue_split(struct request_queue *, struct bio **);
extern void blk_recount_segments(struct request_queue *, struct bio *);
extern int scsi_verify_blk_ioctl(struct block_device *, unsigned int);
extern int scsi_cmd_blk_ioctl(struct block_device *, fmode_t,
extern void __blk_run_queue_uncond(struct request_queue *q);
extern void blk_run_queue(struct request_queue *);
extern void blk_run_queue_async(struct request_queue *q);
-extern void blk_mq_quiesce_queue(struct request_queue *q);
extern int blk_rq_map_user(struct request_queue *, struct request *,
struct rq_map_data *, void __user *, unsigned long,
gfp_t);
extern void blk_execute_rq_nowait(struct request_queue *, struct gendisk *,
struct request *, int, rq_end_io_fn *);
+int blk_status_to_errno(blk_status_t status);
+blk_status_t errno_to_blk_status(int errno);
+
bool blk_mq_poll(struct request_queue *q, blk_qc_t cookie);
static inline struct request_queue *bdev_get_queue(struct block_device *bdev)
* blk_end_request() for parts of the original function.
* This prevents code duplication in drivers.
*/
-extern bool blk_update_request(struct request *rq, int error,
+extern bool blk_update_request(struct request *rq, blk_status_t error,
unsigned int nr_bytes);
-extern void blk_finish_request(struct request *rq, int error);
-extern bool blk_end_request(struct request *rq, int error,
+extern void blk_finish_request(struct request *rq, blk_status_t error);
+extern bool blk_end_request(struct request *rq, blk_status_t error,
unsigned int nr_bytes);
-extern void blk_end_request_all(struct request *rq, int error);
-extern bool __blk_end_request(struct request *rq, int error,
+extern void blk_end_request_all(struct request *rq, blk_status_t error);
+extern bool __blk_end_request(struct request *rq, blk_status_t error,
unsigned int nr_bytes);
-extern void __blk_end_request_all(struct request *rq, int error);
-extern bool __blk_end_request_cur(struct request *rq, int error);
+extern void __blk_end_request_all(struct request *rq, blk_status_t error);
+extern bool __blk_end_request_cur(struct request *rq, blk_status_t error);
extern void blk_complete_request(struct request *);
extern void __blk_complete_request(struct request *);
#define blkdev_entry_to_request(entry) list_entry((entry), struct request, queuelist)
-static inline unsigned long queue_bounce_pfn(struct request_queue *q)
-{
- return q->limits.bounce_pfn;
-}
-
static inline unsigned long queue_segment_boundary(struct request_queue *q)
{
return q->limits.seg_boundary_mask;
const char *disk_name;
};
-typedef int (integrity_processing_fn) (struct blk_integrity_iter *);
+typedef blk_status_t (integrity_processing_fn) (struct blk_integrity_iter *);
struct blk_integrity_profile {
integrity_processing_fn *generate_fn;
struct cleancache_ops {
int (*init_fs)(size_t);
- int (*init_shared_fs)(char *uuid, size_t);
+ int (*init_shared_fs)(uuid_t *uuid, size_t);
int (*get_page)(int, struct cleancache_filekey,
pgoff_t, struct page *);
void (*put_page)(int, struct cleancache_filekey,
#endif /* !CONFIG_GENERIC_CLOCKEVENTS */
-#define CLOCKEVENT_OF_DECLARE(name, compat, fn) \
- OF_DECLARE_1_RET(clkevt, name, compat, fn)
-
-#ifdef CONFIG_CLKEVT_PROBE
-extern int clockevent_probe(void);
-#else
-static inline int clockevent_probe(void) { return 0; }
-#endif
-
#endif /* _LINUX_CLOCKCHIPS_H */
void (*suspend)(struct clocksource *cs);
void (*resume)(struct clocksource *cs);
void (*mark_unstable)(struct clocksource *cs);
+ void (*tick_stable)(struct clocksource *cs);
/* private: */
#ifdef CONFIG_CLOCKSOURCE_WATCHDOG
extern int clocksource_i8253_init(void);
+#define TIMER_OF_DECLARE(name, compat, fn) \
+ OF_DECLARE_1_RET(timer, name, compat, fn)
+
#define CLOCKSOURCE_OF_DECLARE(name, compat, fn) \
- OF_DECLARE_1_RET(clksrc, name, compat, fn)
+ TIMER_OF_DECLARE(name, compat, fn)
-#ifdef CONFIG_CLKSRC_PROBE
-extern void clocksource_probe(void);
+#ifdef CONFIG_TIMER_PROBE
+extern void timer_probe(void);
#else
-static inline void clocksource_probe(void) {}
+static inline void timer_probe(void) {}
#endif
-#define CLOCKSOURCE_ACPI_DECLARE(name, table_id, fn) \
- ACPI_DECLARE_PROBE_ENTRY(clksrc, name, table_id, 0, NULL, 0, fn)
+#define TIMER_ACPI_DECLARE(name, table_id, fn) \
+ ACPI_DECLARE_PROBE_ENTRY(timer, name, table_id, 0, NULL, 0, fn)
#endif /* _LINUX_CLOCKSOURCE_H */
struct compat_timeval it_value;
};
+struct itimerval;
+int get_compat_itimerval(struct itimerval *, const struct compat_itimerval __user *);
+int put_compat_itimerval(struct compat_itimerval __user *, const struct itimerval *);
+
struct compat_tms {
compat_clock_t tms_utime;
compat_clock_t tms_stime;
compat_int_t:32; compat_int_t:32; compat_int_t:32;
};
+struct timex;
+int compat_get_timex(struct timex *, const struct compat_timex __user *);
+int compat_put_timex(struct compat_timex __user *, const struct timex *);
+
#define _COMPAT_NSIG_WORDS (_COMPAT_NSIG / _COMPAT_NSIG_BPW)
typedef struct {
# define __release(x) __context__(x,-1)
# define __cond_lock(x,c) ((c) ? ({ __acquire(x); 1; }) : 0)
# define __percpu __attribute__((noderef, address_space(3)))
-#ifdef CONFIG_SPARSE_RCU_POINTER
# define __rcu __attribute__((noderef, address_space(4)))
-#else /* CONFIG_SPARSE_RCU_POINTER */
-# define __rcu
-#endif /* CONFIG_SPARSE_RCU_POINTER */
# define __private __attribute__((noderef))
extern void __chk_user_ptr(const volatile void __user *);
extern void __chk_io_ptr(const volatile void __iomem *);
#endif
#ifdef CONFIG_OF
-extern struct coresight_platform_data *of_get_coresight_platform_data(
- struct device *dev, struct device_node *node);
+extern int of_coresight_get_cpu(const struct device_node *node);
+extern struct coresight_platform_data *
+of_get_coresight_platform_data(struct device *dev,
+ const struct device_node *node);
#else
+static inline int of_coresight_get_cpu(const struct device_node *node)
+{ return 0; }
static inline struct coresight_platform_data *of_get_coresight_platform_data(
- struct device *dev, struct device_node *node) { return NULL; }
+ struct device *dev, const struct device_node *node) { return NULL; }
#endif
#ifdef CONFIG_PID_NS
extern struct bus_type cpu_subsys;
#ifdef CONFIG_HOTPLUG_CPU
-/* Stop CPUs going up and down. */
-
-extern void cpu_hotplug_begin(void);
-extern void cpu_hotplug_done(void);
-extern void get_online_cpus(void);
-extern void put_online_cpus(void);
+extern void cpus_write_lock(void);
+extern void cpus_write_unlock(void);
+extern void cpus_read_lock(void);
+extern void cpus_read_unlock(void);
+extern void lockdep_assert_cpus_held(void);
extern void cpu_hotplug_disable(void);
extern void cpu_hotplug_enable(void);
void clear_tasks_mm_cpumask(int cpu);
int cpu_down(unsigned int cpu);
-#else /* CONFIG_HOTPLUG_CPU */
-
-static inline void cpu_hotplug_begin(void) {}
-static inline void cpu_hotplug_done(void) {}
-#define get_online_cpus() do { } while (0)
-#define put_online_cpus() do { } while (0)
-#define cpu_hotplug_disable() do { } while (0)
-#define cpu_hotplug_enable() do { } while (0)
-#endif /* CONFIG_HOTPLUG_CPU */
+#else /* CONFIG_HOTPLUG_CPU */
+
+static inline void cpus_write_lock(void) { }
+static inline void cpus_write_unlock(void) { }
+static inline void cpus_read_lock(void) { }
+static inline void cpus_read_unlock(void) { }
+static inline void lockdep_assert_cpus_held(void) { }
+static inline void cpu_hotplug_disable(void) { }
+static inline void cpu_hotplug_enable(void) { }
+#endif /* !CONFIG_HOTPLUG_CPU */
+
+/* Wrappers which go away once all code is converted */
+static inline void cpu_hotplug_begin(void) { cpus_write_lock(); }
+static inline void cpu_hotplug_done(void) { cpus_write_unlock(); }
+static inline void get_online_cpus(void) { cpus_read_lock(); }
+static inline void put_online_cpus(void) { cpus_read_unlock(); }
#ifdef CONFIG_PM_SLEEP_SMP
extern int freeze_secondary_cpus(int primary);
}
#endif
+extern unsigned int arch_freq_get_on_cpu(int cpu);
+
/* the following are really really optional */
extern struct freq_attr cpufreq_freq_attr_scaling_available_freqs;
extern struct freq_attr cpufreq_freq_attr_scaling_boost_freqs;
CPUHP_XEN_EVTCHN_PREPARE,
CPUHP_ARM_SHMOBILE_SCU_PREPARE,
CPUHP_SH_SH3X_PREPARE,
- CPUHP_BLK_MQ_PREPARE,
CPUHP_NET_FLOW_PREPARE,
CPUHP_TOPOLOGY_PREPARE,
CPUHP_NET_IUCV_PREPARE,
CPUHP_AP_ONLINE_IDLE,
CPUHP_AP_SMPBOOT_THREADS,
CPUHP_AP_X86_VDSO_VMA_ONLINE,
+ CPUHP_AP_IRQ_AFFINITY_ONLINE,
CPUHP_AP_PERF_ONLINE,
CPUHP_AP_PERF_X86_ONLINE,
CPUHP_AP_PERF_X86_UNCORE_ONLINE,
int (*startup)(unsigned int cpu),
int (*teardown)(unsigned int cpu), bool multi_instance);
+int __cpuhp_setup_state_cpuslocked(enum cpuhp_state state, const char *name,
+ bool invoke,
+ int (*startup)(unsigned int cpu),
+ int (*teardown)(unsigned int cpu),
+ bool multi_instance);
/**
* cpuhp_setup_state - Setup hotplug state callbacks with calling the callbacks
* @state: The state for which the calls are installed
return __cpuhp_setup_state(state, name, true, startup, teardown, false);
}
+static inline int cpuhp_setup_state_cpuslocked(enum cpuhp_state state,
+ const char *name,
+ int (*startup)(unsigned int cpu),
+ int (*teardown)(unsigned int cpu))
+{
+ return __cpuhp_setup_state_cpuslocked(state, name, true, startup,
+ teardown, false);
+}
+
/**
* cpuhp_setup_state_nocalls - Setup hotplug state callbacks without calling the
* callbacks
false);
}
+static inline int cpuhp_setup_state_nocalls_cpuslocked(enum cpuhp_state state,
+ const char *name,
+ int (*startup)(unsigned int cpu),
+ int (*teardown)(unsigned int cpu))
+{
+ return __cpuhp_setup_state_cpuslocked(state, name, false, startup,
+ teardown, false);
+}
+
/**
* cpuhp_setup_state_multi - Add callbacks for multi state
* @state: The state for which the calls are installed
int __cpuhp_state_add_instance(enum cpuhp_state state, struct hlist_node *node,
bool invoke);
+int __cpuhp_state_add_instance_cpuslocked(enum cpuhp_state state,
+ struct hlist_node *node, bool invoke);
/**
* cpuhp_state_add_instance - Add an instance for a state and invoke startup
return __cpuhp_state_add_instance(state, node, false);
}
+static inline int
+cpuhp_state_add_instance_nocalls_cpuslocked(enum cpuhp_state state,
+ struct hlist_node *node)
+{
+ return __cpuhp_state_add_instance_cpuslocked(state, node, false);
+}
+
void __cpuhp_remove_state(enum cpuhp_state state, bool invoke);
+void __cpuhp_remove_state_cpuslocked(enum cpuhp_state state, bool invoke);
/**
* cpuhp_remove_state - Remove hotplug state callbacks and invoke the teardown
__cpuhp_remove_state(state, false);
}
+static inline void cpuhp_remove_state_nocalls_cpuslocked(enum cpuhp_state state)
+{
+ __cpuhp_remove_state_cpuslocked(state, false);
+}
+
/**
* cpuhp_remove_multi_state - Remove hotplug multi state callback
* @state: The state for which the calls are removed
(cpu) = cpumask_next_zero((cpu), (mask)), \
(cpu) < nr_cpu_ids;)
+extern int cpumask_next_wrap(int n, const struct cpumask *mask, int start, bool wrap);
+
+/**
+ * for_each_cpu_wrap - iterate over every cpu in a mask, starting at a specified location
+ * @cpu: the (optionally unsigned) integer iterator
+ * @mask: the cpumask poiter
+ * @start: the start location
+ *
+ * The implementation does not assume any bit in @mask is set (including @start).
+ *
+ * After the loop, cpu is >= nr_cpu_ids.
+ */
+#define for_each_cpu_wrap(cpu, mask, start) \
+ for ((cpu) = cpumask_next_wrap((start)-1, (mask), (start), false); \
+ (cpu) < nr_cpumask_bits; \
+ (cpu) = cpumask_next_wrap((cpu), (mask), (start), true))
+
/**
* for_each_cpu_and - iterate over every cpu in both masks
* @cpu: the (optionally unsigned) integer iterator
set_bit(cpumask_check(cpu), cpumask_bits(dstp));
}
+static inline void __cpumask_set_cpu(unsigned int cpu, struct cpumask *dstp)
+{
+ __set_bit(cpumask_check(cpu), cpumask_bits(dstp));
+}
+
+
/**
* cpumask_clear_cpu - clear a cpu in a cpumask
* @cpu: cpu number (< nr_cpu_ids)
clear_bit(cpumask_check(cpu), cpumask_bits(dstp));
}
+static inline void __cpumask_clear_cpu(int cpu, struct cpumask *dstp)
+{
+ __clear_bit(cpumask_check(cpu), cpumask_bits(dstp));
+}
+
/**
* cpumask_test_cpu - test for a cpu in a cpumask
* @cpu: cpu number (< nr_cpu_ids)
--- /dev/null
+#ifndef _LINUX_CRC4_H
+#define _LINUX_CRC4_H
+
+#include <linux/types.h>
+
+extern uint8_t crc4(uint8_t c, uint64_t x, int bits);
+
+#endif /* _LINUX_CRC4_H */
-/* Credentials management - see Documentation/security/credentials.txt
+/* Credentials management - see Documentation/security/credentials.rst
*
* Copyright (C) 2008 Red Hat, Inc. All Rights Reserved.
* Written by David Howells (dhowells@redhat.com)
* 2 as published by the Free Software Foundation.
*
* debugfs is for people to use instead of /proc or /sys.
- * See Documentation/DocBook/filesystems for more details.
+ * See Documentation/filesystems/ for more details.
*/
#ifndef _DEBUGFS_H_
* 2 : The target wants to push back the io
*/
typedef int (*dm_endio_fn) (struct dm_target *ti,
- struct bio *bio, int error);
+ struct bio *bio, blk_status_t *error);
typedef int (*dm_request_endio_fn) (struct dm_target *ti,
- struct request *clone, int error,
+ struct request *clone, blk_status_t error,
union map_info *map_context);
typedef void (*dm_presuspend_fn) (struct dm_target *ti);
* @name: The name of the bus.
* @dev_name: Used for subsystems to enumerate devices like ("foo%u", dev->id).
* @dev_root: Default device to use as the parent.
- * @dev_attrs: Default attributes of the devices on the bus.
* @bus_groups: Default attributes of the bus.
* @dev_groups: Default attributes of the devices on the bus.
* @drv_groups: Default attributes of the device drivers on the bus.
const char *name;
const char *dev_name;
struct device *dev_root;
- struct device_attribute *dev_attrs; /* use dev_groups instead */
const struct attribute_group **bus_groups;
const struct attribute_group **dev_groups;
const struct attribute_group **drv_groups;
* struct class - device classes
* @name: Name of the class.
* @owner: The module owner.
- * @class_attrs: Default attributes of this class.
* @class_groups: Default attributes of this class.
* @dev_groups: Default attributes of the devices that belong to the class.
* @dev_kobj: The kobject that represents this class and links it into the hierarchy.
const char *name;
struct module *owner;
- struct class_attribute *class_attrs;
const struct attribute_group **class_groups;
const struct attribute_group **dev_groups;
struct kobject *dev_kobj;
const char *buf, size_t count);
};
-#define CLASS_ATTR(_name, _mode, _show, _store) \
- struct class_attribute class_attr_##_name = __ATTR(_name, _mode, _show, _store)
#define CLASS_ATTR_RW(_name) \
struct class_attribute class_attr_##_name = __ATTR_RW(_name)
#define CLASS_ATTR_RO(_name) \
*
* @offline_disabled: If set, the device is permanently online.
* @offline: Set after successful invocation of bus type's .offline().
+ * @of_node_reused: Set if the device-tree node is shared with an ancestor
+ * device.
*
* At the lowest level, every device in a Linux system is represented by an
* instance of struct device. The device structure contains the information
bool offline_disabled:1;
bool offline:1;
+ bool of_node_reused:1;
};
static inline struct device *kobj_to_dev(struct kobject *kobj)
extern int device_online(struct device *dev);
extern void set_primary_fwnode(struct device *dev, struct fwnode_handle *fwnode);
extern void set_secondary_fwnode(struct device *dev, struct fwnode_handle *fwnode);
+void device_set_of_node_from_dev(struct device *dev, const struct device *dev2);
static inline int dev_num_vf(struct device *dev)
{
#define EFI_CAPSULE_POPULATE_SYSTEM_TABLE 0x00020000
#define EFI_CAPSULE_INITIATE_RESET 0x00040000
+struct capsule_info {
+ efi_capsule_header_t header;
+ int reset_type;
+ long index;
+ size_t count;
+ size_t total_size;
+ phys_addr_t *pages;
+ size_t page_bytes_remain;
+};
+
+int __efi_capsule_setup_info(struct capsule_info *cap_info);
+
/*
* Allocation types for calls to boottime->allocate_pages.
*/
size_t size, int *reset);
extern int efi_capsule_update(efi_capsule_header_t *capsule,
- struct page **pages);
+ phys_addr_t *pages);
#ifdef CONFIG_EFI_RUNTIME_MAP
int efi_runtime_map_init(struct kobject *);
int (*request_merge)(struct request_queue *q, struct request **, struct bio *);
void (*request_merged)(struct request_queue *, struct request *, enum elv_merge);
void (*requests_merged)(struct request_queue *, struct request *, struct request *);
- struct request *(*get_request)(struct request_queue *, unsigned int, struct blk_mq_alloc_data *);
- void (*put_request)(struct request *);
+ void (*limit_depth)(unsigned int, struct blk_mq_alloc_data *);
+ void (*prepare_request)(struct request *, struct bio *bio);
+ void (*finish_request)(struct request *);
void (*insert_requests)(struct blk_mq_hw_ctx *, struct list_head *, bool);
struct request *(*dispatch_request)(struct blk_mq_hw_ctx *);
bool (*has_work)(struct blk_mq_hw_ctx *);
void (*requeue_request)(struct request *);
struct request *(*former_request)(struct request_queue *, struct request *);
struct request *(*next_request)(struct request_queue *, struct request *);
- int (*get_rq_priv)(struct request_queue *, struct request *, struct bio *);
- void (*put_rq_priv)(struct request_queue *, struct request *);
void (*init_icq)(struct io_cq *);
void (*exit_icq)(struct io_cq *);
};
struct eventfd_ctx *eventfd_ctx_fileget(struct file *file);
__u64 eventfd_signal(struct eventfd_ctx *ctx, __u64 n);
ssize_t eventfd_ctx_read(struct eventfd_ctx *ctx, int no_wait, __u64 *cnt);
-int eventfd_ctx_remove_wait_queue(struct eventfd_ctx *ctx, wait_queue_t *wait,
+int eventfd_ctx_remove_wait_queue(struct eventfd_ctx *ctx, wait_queue_entry_t *wait,
__u64 *cnt);
#else /* CONFIG_EVENTFD */
}
static inline int eventfd_ctx_remove_wait_queue(struct eventfd_ctx *ctx,
- wait_queue_t *wait, __u64 *cnt)
+ wait_queue_entry_t *wait, __u64 *cnt)
{
return -ENOSYS;
}
#define _LINUX_FS_H
#include <linux/linkage.h>
-#include <linux/wait.h>
+#include <linux/wait_bit.h>
#include <linux/kdev_t.h>
#include <linux/dcache.h>
#include <linux/path.h>
#include <linux/rwsem.h>
#include <linux/capability.h>
#include <linux/semaphore.h>
+#include <linux/fcntl.h>
#include <linux/fiemap.h>
#include <linux/rculist_bl.h>
#include <linux/atomic.h>
#include <linux/percpu-rwsem.h>
#include <linux/workqueue.h>
#include <linux/delayed_call.h>
+#include <linux/uuid.h>
#include <asm/byteorder.h>
#include <uapi/linux/fs.h>
/* File was opened by fanotify and shouldn't generate fanotify events */
#define FMODE_NONOTIFY ((__force fmode_t)0x4000000)
+/* File is capable of returning -EAGAIN if AIO will block */
+#define FMODE_AIO_NOWAIT ((__force fmode_t)0x8000000)
+
/*
* Flag for rw_copy_check_uvector and compat_rw_copy_check_uvector
* that indicates that they should check the contents of the iovec are
struct address_space;
struct writeback_control;
+/*
+ * Write life time hint values.
+ */
+enum rw_hint {
+ WRITE_LIFE_NOT_SET = 0,
+ WRITE_LIFE_NONE = RWH_WRITE_LIFE_NONE,
+ WRITE_LIFE_SHORT = RWH_WRITE_LIFE_SHORT,
+ WRITE_LIFE_MEDIUM = RWH_WRITE_LIFE_MEDIUM,
+ WRITE_LIFE_LONG = RWH_WRITE_LIFE_LONG,
+ WRITE_LIFE_EXTREME = RWH_WRITE_LIFE_EXTREME,
+};
+
#define IOCB_EVENTFD (1 << 0)
#define IOCB_APPEND (1 << 1)
#define IOCB_DIRECT (1 << 2)
#define IOCB_DSYNC (1 << 4)
#define IOCB_SYNC (1 << 5)
#define IOCB_WRITE (1 << 6)
+#define IOCB_NOWAIT (1 << 7)
struct kiocb {
struct file *ki_filp;
void (*ki_complete)(struct kiocb *iocb, long ret, long ret2);
void *private;
int ki_flags;
+ enum rw_hint ki_hint;
};
static inline bool is_sync_kiocb(struct kiocb *kiocb)
return kiocb->ki_complete == NULL;
}
-static inline int iocb_flags(struct file *file);
-
-static inline void init_sync_kiocb(struct kiocb *kiocb, struct file *filp)
-{
- *kiocb = (struct kiocb) {
- .ki_filp = filp,
- .ki_flags = iocb_flags(filp),
- };
-}
-
/*
* "descriptor" for what we're up to with a read.
* This allows us to use the same read code yet
spinlock_t i_lock; /* i_blocks, i_bytes, maybe i_size */
unsigned short i_bytes;
unsigned int i_blkbits;
+ enum rw_hint i_write_hint;
blkcnt_t i_blocks;
#ifdef __NEED_I_SIZE_ORDERED
* Must not be taken from IRQ context.
*/
spinlock_t f_lock;
+ enum rw_hint f_write_hint;
atomic_long_t f_count;
unsigned int f_flags;
fmode_t f_mode;
#define OFFT_OFFSET_MAX INT_LIMIT(off_t)
#endif
-#include <linux/fcntl.h>
-
extern void send_sigio(struct fown_struct *fown, int fd, int band);
/*
struct sb_writers s_writers;
- char s_id[32]; /* Informational name */
- u8 s_uuid[16]; /* UUID */
+ char s_id[32]; /* Informational name */
+ uuid_t s_uuid; /* UUID */
void *s_fs_info; /* Filesystem private info */
unsigned int s_max_links;
return !uid_valid(inode->i_uid) || !gid_valid(inode->i_gid);
}
+static inline enum rw_hint file_write_hint(struct file *file)
+{
+ if (file->f_write_hint != WRITE_LIFE_NOT_SET)
+ return file->f_write_hint;
+
+ return file_inode(file)->i_write_hint;
+}
+
+static inline int iocb_flags(struct file *file);
+
+static inline void init_sync_kiocb(struct kiocb *kiocb, struct file *filp)
+{
+ *kiocb = (struct kiocb) {
+ .ki_filp = filp,
+ .ki_flags = iocb_flags(filp),
+ .ki_hint = file_write_hint(filp),
+ };
+}
+
/*
* Inode state bits. Protected by inode->i_lock
*
extern void filemap_fdatawait_keep_errors(struct address_space *);
extern int filemap_fdatawait_range(struct address_space *, loff_t lstart,
loff_t lend);
+extern bool filemap_range_has_page(struct address_space *, loff_t lstart,
+ loff_t lend);
extern int filemap_write_and_wait(struct address_space *mapping);
extern int filemap_write_and_wait_range(struct address_space *mapping,
loff_t lstart, loff_t lend);
DIO_SKIP_DIO_COUNT = 0x08,
};
-void dio_end_io(struct bio *bio, int error);
+void dio_end_io(struct bio *bio);
ssize_t __blockdev_direct_IO(struct kiocb *iocb, struct inode *inode,
struct block_device *bdev, struct iov_iter *iter,
return res;
}
+static inline int kiocb_set_rw_flags(struct kiocb *ki, int flags)
+{
+ if (unlikely(flags & ~RWF_SUPPORTED))
+ return -EOPNOTSUPP;
+
+ if (flags & RWF_NOWAIT) {
+ if (!(ki->ki_filp->f_mode & FMODE_AIO_NOWAIT))
+ return -EOPNOTSUPP;
+ ki->ki_flags |= IOCB_NOWAIT;
+ }
+ if (flags & RWF_HIPRI)
+ ki->ki_flags |= IOCB_HIPRI;
+ if (flags & RWF_DSYNC)
+ ki->ki_flags |= IOCB_DSYNC;
+ if (flags & RWF_SYNC)
+ ki->ki_flags |= (IOCB_DSYNC | IOCB_SYNC);
+ return 0;
+}
+
static inline ino_t parent_ino(struct dentry *dentry)
{
ino_t res;
struct device dev;
u8 engine_type;
u8 version;
+ u8 unit;
+ struct fsi_slave *slave;
+ uint32_t addr;
+ uint32_t size;
};
+extern int fsi_device_read(struct fsi_device *dev, uint32_t addr,
+ void *val, size_t size);
+extern int fsi_device_write(struct fsi_device *dev, uint32_t addr,
+ const void *val, size_t size);
+extern int fsi_device_peek(struct fsi_device *dev, void *val);
+
struct fsi_device_id {
u8 engine_type;
u8 version;
#define FSI_DEVICE_VERSIONED(t, v) \
.engine_type = (t), .version = (v),
-
struct fsi_driver {
struct device_driver drv;
const struct fsi_device_id *id_table;
#define to_fsi_dev(devp) container_of(devp, struct fsi_device, dev)
#define to_fsi_drv(drvp) container_of(drvp, struct fsi_driver, drv)
+extern int fsi_driver_register(struct fsi_driver *fsi_drv);
+extern void fsi_driver_unregister(struct fsi_driver *fsi_drv);
+
+/* module_fsi_driver() - Helper macro for drivers that don't do
+ * anything special in module init/exit. This eliminates a lot of
+ * boilerplate. Each module may only use this macro once, and
+ * calling it replaces module_init() and module_exit()
+ */
+#define module_fsi_driver(__fsi_driver) \
+ module_driver(__fsi_driver, fsi_driver_register, \
+ fsi_driver_unregister)
+
+/* direct slave API */
+extern int fsi_slave_claim_range(struct fsi_slave *slave,
+ uint32_t addr, uint32_t size);
+extern void fsi_slave_release_range(struct fsi_slave *slave,
+ uint32_t addr, uint32_t size);
+extern int fsi_slave_read(struct fsi_slave *slave, uint32_t addr,
+ void *val, size_t size);
+extern int fsi_slave_write(struct fsi_slave *slave, uint32_t addr,
+ const void *val, size_t size);
+
+
+
extern struct bus_type fsi_bus_type;
#endif /* LINUX_FSI_H */
return NULL;
}
-static inline int blk_part_pack_uuid(const u8 *uuid_str, u8 *to)
-{
- uuid_be_to_bin(uuid_str, (uuid_be *)to);
- return 0;
-}
-
static inline int disk_max_parts(struct gendisk *disk)
{
if (disk->flags & GENHD_FL_EXT_DEVT)
dev_t devt = MKDEV(0, 0);
return devt;
}
-
-static inline int blk_part_pack_uuid(const u8 *uuid_str, u8 *to)
-{
- return -EINVAL;
-}
#endif /* CONFIG_BLOCK */
#endif /* _LINUX_GENHD_H */
/**
* hash_for_each_possible_rcu - iterate over all possible objects hashing to the
* same bucket in an rcu enabled hashtable
- * in a rcu enabled hashtable
* @name: hashtable to iterate
* @obj: the type * to use as a loop cursor for each entry
* @member: the name of the hlist_node within the struct
atomic_t user_requested_state;
int poll_interval;
int raw_hystersis;
+ int latency_ms;
struct iio_trigger *trigger;
int timestamp_ns_scale;
struct hid_sensor_hub_attribute_info poll;
struct hid_sensor_hub_attribute_info report_state;
struct hid_sensor_hub_attribute_info power_state;
struct hid_sensor_hub_attribute_info sensitivity;
+ struct hid_sensor_hub_attribute_info report_latency;
struct work_struct work;
};
int64_t hid_sensor_convert_timestamp(struct hid_sensor_common *st,
int64_t raw_value);
+bool hid_sensor_batch_mode_supported(struct hid_sensor_common *st);
+int hid_sensor_set_report_latency(struct hid_sensor_common *st, int latency);
+int hid_sensor_get_report_latency(struct hid_sensor_common *st);
#endif
#define HID_USAGE_SENSOR_ORIENT_TILT_Z 0x200481
#define HID_USAGE_SENSOR_DEVICE_ORIENTATION 0x20008A
+#define HID_USAGE_SENSOR_RELATIVE_ORIENTATION 0x20008E
+#define HID_USAGE_SENSOR_GEOMAGNETIC_ORIENTATION 0x2000C1
#define HID_USAGE_SENSOR_ORIENT_ROTATION_MATRIX 0x200482
#define HID_USAGE_SENSOR_ORIENT_QUATERNION 0x200483
#define HID_USAGE_SENSOR_ORIENT_MAGN_FLUX 0x200484
#define HID_USAGE_SENSOR_PROP_REPORT_STATE 0x200316
#define HID_USAGE_SENSOR_PROY_POWER_STATE 0x200319
+/* Batch mode selectors */
+#define HID_USAGE_SENSOR_PROP_REPORT_LATENCY 0x20031B
+
/* Per data field properties */
#define HID_USAGE_SENSOR_DATA_MOD_NONE 0x00
#define HID_USAGE_SENSOR_DATA_MOD_CHANGE_SENSITIVITY_ABS 0x1000
}
/* Precise sleep: */
-extern long hrtimer_nanosleep(struct timespec64 *rqtp,
- struct timespec __user *rmtp,
+
+extern int nanosleep_copyout(struct restart_block *, struct timespec *);
+extern long hrtimer_nanosleep(const struct timespec64 *rqtp,
const enum hrtimer_mode mode,
const clockid_t clockid);
-extern long hrtimer_nanosleep_restart(struct restart_block *restart_block);
extern void hrtimer_init_sleeper(struct hrtimer_sleeper *sl,
struct task_struct *tsk);
+++ /dev/null
-/*
- * twl4030_madc.h - Header for TWL4030 MADC
- *
- * Copyright (C) 2011 Texas Instruments Incorporated - http://www.ti.com/
- * J Keerthy <j-keerthy@ti.com>
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * version 2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- * General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
- * 02110-1301 USA
- *
- */
-
-#ifndef _TWL4030_MADC_H
-#define _TWL4030_MADC_H
-
-struct twl4030_madc_conversion_method {
- u8 sel;
- u8 avg;
- u8 rbase;
- u8 ctrl;
-};
-
-#define TWL4030_MADC_MAX_CHANNELS 16
-
-
-/*
- * twl4030_madc_request- madc request packet for channel conversion
- * @channels: 16 bit bitmap for individual channels
- * @do_avgP: sample the input channel for 4 consecutive cycles
- * @method: RT, SW1, SW2
- * @type: Polling or interrupt based method
- * @raw: Return raw value, do not convert it
- */
-
-struct twl4030_madc_request {
- unsigned long channels;
- bool do_avg;
- u16 method;
- u16 type;
- bool active;
- bool result_pending;
- bool raw;
- int rbuf[TWL4030_MADC_MAX_CHANNELS];
- void (*func_cb)(int len, int channels, int *buf);
-};
-
-enum conversion_methods {
- TWL4030_MADC_RT,
- TWL4030_MADC_SW1,
- TWL4030_MADC_SW2,
- TWL4030_MADC_NUM_METHODS
-};
-
-enum sample_type {
- TWL4030_MADC_WAIT,
- TWL4030_MADC_IRQ_ONESHOT,
- TWL4030_MADC_IRQ_REARM
-};
-
-#define TWL4030_MADC_CTRL1 0x00
-#define TWL4030_MADC_CTRL2 0x01
-
-#define TWL4030_MADC_RTSELECT_LSB 0x02
-#define TWL4030_MADC_SW1SELECT_LSB 0x06
-#define TWL4030_MADC_SW2SELECT_LSB 0x0A
-
-#define TWL4030_MADC_RTAVERAGE_LSB 0x04
-#define TWL4030_MADC_SW1AVERAGE_LSB 0x08
-#define TWL4030_MADC_SW2AVERAGE_LSB 0x0C
-
-#define TWL4030_MADC_CTRL_SW1 0x12
-#define TWL4030_MADC_CTRL_SW2 0x13
-
-#define TWL4030_MADC_RTCH0_LSB 0x17
-#define TWL4030_MADC_GPCH0_LSB 0x37
-
-#define TWL4030_MADC_MADCON (1 << 0) /* MADC power on */
-#define TWL4030_MADC_BUSY (1 << 0) /* MADC busy */
-/* MADC conversion completion */
-#define TWL4030_MADC_EOC_SW (1 << 1)
-/* MADC SWx start conversion */
-#define TWL4030_MADC_SW_START (1 << 5)
-#define TWL4030_MADC_ADCIN0 (1 << 0)
-#define TWL4030_MADC_ADCIN1 (1 << 1)
-#define TWL4030_MADC_ADCIN2 (1 << 2)
-#define TWL4030_MADC_ADCIN3 (1 << 3)
-#define TWL4030_MADC_ADCIN4 (1 << 4)
-#define TWL4030_MADC_ADCIN5 (1 << 5)
-#define TWL4030_MADC_ADCIN6 (1 << 6)
-#define TWL4030_MADC_ADCIN7 (1 << 7)
-#define TWL4030_MADC_ADCIN8 (1 << 8)
-#define TWL4030_MADC_ADCIN9 (1 << 9)
-#define TWL4030_MADC_ADCIN10 (1 << 10)
-#define TWL4030_MADC_ADCIN11 (1 << 11)
-#define TWL4030_MADC_ADCIN12 (1 << 12)
-#define TWL4030_MADC_ADCIN13 (1 << 13)
-#define TWL4030_MADC_ADCIN14 (1 << 14)
-#define TWL4030_MADC_ADCIN15 (1 << 15)
-
-/* Fixed channels */
-#define TWL4030_MADC_BTEMP TWL4030_MADC_ADCIN1
-#define TWL4030_MADC_VBUS TWL4030_MADC_ADCIN8
-#define TWL4030_MADC_VBKB TWL4030_MADC_ADCIN9
-#define TWL4030_MADC_ICHG TWL4030_MADC_ADCIN10
-#define TWL4030_MADC_VCHG TWL4030_MADC_ADCIN11
-#define TWL4030_MADC_VBAT TWL4030_MADC_ADCIN12
-
-/* Step size and prescaler ratio */
-#define TEMP_STEP_SIZE 147
-#define TEMP_PSR_R 100
-#define CURR_STEP_SIZE 147
-#define CURR_PSR_R1 44
-#define CURR_PSR_R2 88
-
-#define TWL4030_BCI_BCICTL1 0x23
-#define TWL4030_BCI_CGAIN 0x020
-#define TWL4030_BCI_MESBAT (1 << 1)
-#define TWL4030_BCI_TYPEN (1 << 4)
-#define TWL4030_BCI_ITHEN (1 << 3)
-
-#define REG_BCICTL2 0x024
-#define TWL4030_BCI_ITHSENS 0x007
-
-/* Register and bits for GPBR1 register */
-#define TWL4030_REG_GPBR1 0x0c
-#define TWL4030_GPBR1_MADC_HFCLK_EN (1 << 7)
-
-struct twl4030_madc_user_parms {
- int channel;
- int average;
- int status;
- u16 result;
-};
-
-int twl4030_madc_conversion(struct twl4030_madc_request *conv);
-int twl4030_get_madc_conversion(int channel_no);
-#endif
void (*init_dev)(ide_drive_t *);
void (*set_pio_mode)(struct hwif_s *, ide_drive_t *);
void (*set_dma_mode)(struct hwif_s *, ide_drive_t *);
- int (*reset_poll)(ide_drive_t *);
+ blk_status_t (*reset_poll)(ide_drive_t *);
void (*pre_reset)(ide_drive_t *);
void (*resetproc)(ide_drive_t *);
void (*maskproc)(ide_drive_t *, int);
extern int ide_vlb_clk;
extern int ide_pci_clk;
-int ide_end_rq(ide_drive_t *, struct request *, int, unsigned int);
+int ide_end_rq(ide_drive_t *, struct request *, blk_status_t, unsigned int);
void ide_kill_rq(ide_drive_t *, struct request *);
void __ide_set_handler(ide_drive_t *, ide_handler_t *, unsigned int);
const struct ide_devset *setting, int arg);
void ide_complete_cmd(ide_drive_t *, struct ide_cmd *, u8, u8);
-int ide_complete_rq(ide_drive_t *, int, unsigned int);
+int ide_complete_rq(ide_drive_t *, blk_status_t, unsigned int);
void ide_tf_readback(ide_drive_t *drive, struct ide_cmd *cmd);
void ide_tf_dump(const char *, struct ide_cmd *);
int iio_convert_raw_to_processed(struct iio_channel *chan, int raw,
int *processed, unsigned int scale);
+/**
+ * iio_get_channel_ext_info_count() - get number of ext_info attributes
+ * connected to the channel.
+ * @chan: The channel being queried
+ *
+ * Returns the number of ext_info attributes
+ */
+unsigned int iio_get_channel_ext_info_count(struct iio_channel *chan);
+
+/**
+ * iio_read_channel_ext_info() - read ext_info attribute from a given channel
+ * @chan: The channel being queried.
+ * @attr: The ext_info attribute to read.
+ * @buf: Where to store the attribute value. Assumed to hold
+ * at least PAGE_SIZE bytes.
+ *
+ * Returns the number of bytes written to buf (perhaps w/o zero termination;
+ * it need not even be a string), or an error code.
+ */
+ssize_t iio_read_channel_ext_info(struct iio_channel *chan,
+ const char *attr, char *buf);
+
+/**
+ * iio_write_channel_ext_info() - write ext_info attribute from a given channel
+ * @chan: The channel being queried.
+ * @attr: The ext_info attribute to read.
+ * @buf: The new attribute value. Strings needs to be zero-
+ * terminated, but the terminator should not be included
+ * in the below len.
+ * @len: The size of the new attribute value.
+ *
+ * Returns the number of accepted bytes, which should be the same as len.
+ * An error code can also be returned.
+ */
+ssize_t iio_write_channel_ext_info(struct iio_channel *chan, const char *attr,
+ const char *buf, size_t len);
+
#endif
#define INDIO_BUFFER_SOFTWARE 0x04
#define INDIO_BUFFER_HARDWARE 0x08
#define INDIO_EVENT_TRIGGERED 0x10
+#define INDIO_HARDWARE_TRIGGERED 0x20
#define INDIO_ALL_BUFFER_MODES \
(INDIO_BUFFER_TRIGGERED | INDIO_BUFFER_HARDWARE | INDIO_BUFFER_SOFTWARE)
+#define INDIO_ALL_TRIGGERED_MODES \
+ (INDIO_BUFFER_TRIGGERED \
+ | INDIO_EVENT_TRIGGERED \
+ | INDIO_HARDWARE_TRIGGERED)
+
#define INDIO_MAX_RAW_ELEMENTS 4
struct iio_trigger; /* forward declaration */
#define _STM32_TIMER_TRIGGER_H_
#define TIM1_TRGO "tim1_trgo"
+#define TIM1_TRGO2 "tim1_trgo2"
#define TIM1_CH1 "tim1_ch1"
#define TIM1_CH2 "tim1_ch2"
#define TIM1_CH3 "tim1_ch3"
#define TIM7_TRGO "tim7_trgo"
#define TIM8_TRGO "tim8_trgo"
+#define TIM8_TRGO2 "tim8_trgo2"
#define TIM8_CH1 "tim8_ch1"
#define TIM8_CH2 "tim8_ch2"
#define TIM8_CH3 "tim8_ch3"
}
#endif
+#ifdef CONFIG_IRQ_TIMINGS
+void irq_timings_enable(void);
+void irq_timings_disable(void);
+u64 irq_timings_next_event(u64 now);
+#endif
+
struct seq_file;
int show_interrupts(struct seq_file *p, void *v);
int arch_show_interrupts(struct seq_file *p, int prec);
#define IOMAP_REPORT (1 << 2) /* report extent status, e.g. FIEMAP */
#define IOMAP_FAULT (1 << 3) /* mapping for page fault */
#define IOMAP_DIRECT (1 << 4) /* direct I/O */
+#define IOMAP_NOWAIT (1 << 5) /* Don't wait for writeback */
struct iomap_ops {
/*
#include <linux/topology.h>
#include <linux/wait.h>
#include <linux/io.h>
+#include <linux/slab.h>
#include <asm/irq.h>
#include <asm/ptrace.h>
* @affinity: IRQ affinity on SMP. If this is an IPI
* related irq, then this is the mask of the
* CPUs to which an IPI can be sent.
+ * @effective_affinity: The effective IRQ affinity on SMP as some irq
+ * chips do not allow multi CPU destinations.
+ * A subset of @affinity.
* @msi_desc: MSI descriptor
* @ipi_offset: Offset of first IPI target cpu in @affinity. Optional.
*/
void *handler_data;
struct msi_desc *msi_desc;
cpumask_var_t affinity;
+#ifdef CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK
+ cpumask_var_t effective_affinity;
+#endif
#ifdef CONFIG_GENERIC_IRQ_IPI
unsigned int ipi_offset;
#endif
* IRQD_WAKEUP_ARMED - Wakeup mode armed
* IRQD_FORWARDED_TO_VCPU - The interrupt is forwarded to a VCPU
* IRQD_AFFINITY_MANAGED - Affinity is auto-managed by the kernel
+ * IRQD_IRQ_STARTED - Startup state of the interrupt
+ * IRQD_MANAGED_SHUTDOWN - Interrupt was shutdown due to empty affinity
+ * mask. Applies only to affinity managed irqs.
+ * IRQD_SINGLE_TARGET - IRQ allows only a single affinity target
*/
enum {
IRQD_TRIGGER_MASK = 0xf,
IRQD_WAKEUP_ARMED = (1 << 19),
IRQD_FORWARDED_TO_VCPU = (1 << 20),
IRQD_AFFINITY_MANAGED = (1 << 21),
+ IRQD_IRQ_STARTED = (1 << 22),
+ IRQD_MANAGED_SHUTDOWN = (1 << 23),
+ IRQD_SINGLE_TARGET = (1 << 24),
};
#define __irqd_to_state(d) ACCESS_PRIVATE((d)->common, state_use_accessors)
return __irqd_to_state(d) & IRQD_LEVEL;
}
+/*
+ * Must only be called of irqchip.irq_set_affinity() or low level
+ * hieararchy domain allocation functions.
+ */
+static inline void irqd_set_single_target(struct irq_data *d)
+{
+ __irqd_to_state(d) |= IRQD_SINGLE_TARGET;
+}
+
+static inline bool irqd_is_single_target(struct irq_data *d)
+{
+ return __irqd_to_state(d) & IRQD_SINGLE_TARGET;
+}
+
static inline bool irqd_is_wakeup_set(struct irq_data *d)
{
return __irqd_to_state(d) & IRQD_WAKEUP_STATE;
__irqd_to_state(d) &= ~IRQD_ACTIVATED;
}
+static inline bool irqd_is_started(struct irq_data *d)
+{
+ return __irqd_to_state(d) & IRQD_IRQ_STARTED;
+}
+
+static inline bool irqd_is_managed_and_shutdown(struct irq_data *d)
+{
+ return __irqd_to_state(d) & IRQD_MANAGED_SHUTDOWN;
+}
+
#undef __irqd_to_state
static inline irq_hw_number_t irqd_to_hwirq(struct irq_data *d)
const struct cpumask *cpumask, bool force);
extern int irq_set_vcpu_affinity(unsigned int irq, void *vcpu_info);
+#if defined(CONFIG_SMP) && defined(CONFIG_GENERIC_IRQ_MIGRATION)
extern void irq_migrate_all_off_this_cpu(void);
+extern int irq_affinity_online_cpu(unsigned int cpu);
+#else
+# define irq_affinity_online_cpu NULL
+#endif
#if defined(CONFIG_SMP) && defined(CONFIG_GENERIC_PENDING_IRQ)
void irq_move_irq(struct irq_data *data);
void irq_move_masked_irq(struct irq_data *data);
+void irq_force_complete_move(struct irq_desc *desc);
#else
static inline void irq_move_irq(struct irq_data *data) { }
static inline void irq_move_masked_irq(struct irq_data *data) { }
+static inline void irq_force_complete_move(struct irq_desc *desc) { }
#endif
extern int no_irq_affinity;
return d->common->affinity;
}
+#ifdef CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK
+static inline
+struct cpumask *irq_data_get_effective_affinity_mask(struct irq_data *d)
+{
+ return d->common->effective_affinity;
+}
+static inline void irq_data_update_effective_affinity(struct irq_data *d,
+ const struct cpumask *m)
+{
+ cpumask_copy(d->common->effective_affinity, m);
+}
+#else
+static inline void irq_data_update_effective_affinity(struct irq_data *d,
+ const struct cpumask *m)
+{
+}
+static inline
+struct cpumask *irq_data_get_effective_affinity_mask(struct irq_data *d)
+{
+ return d->common->affinity;
+}
+#endif
+
unsigned int arch_dynirq_lower_bound(unsigned int from);
int __irq_alloc_descs(int irq, unsigned int from, unsigned int cnt, int node,
void irq_remove_generic_chip(struct irq_chip_generic *gc, u32 msk,
unsigned int clr, unsigned int set);
+struct irq_chip_generic *
+devm_irq_alloc_generic_chip(struct device *dev, const char *name, int num_ct,
+ unsigned int irq_base, void __iomem *reg_base,
+ irq_flow_handler_t handler);
+int devm_irq_setup_generic_chip(struct device *dev, struct irq_chip_generic *gc,
+ u32 msk, enum irq_gc_flags flags,
+ unsigned int clr, unsigned int set);
+
struct irq_chip_generic *irq_get_domain_generic_chip(struct irq_domain *d, unsigned int hw_irq);
int __irq_alloc_domain_generic_chips(struct irq_domain *d, int irqs_per_chip,
handler, clr, set, flags); \
})
+static inline void irq_free_generic_chip(struct irq_chip_generic *gc)
+{
+ kfree(gc);
+}
+
+static inline void irq_destroy_generic_chip(struct irq_chip_generic *gc,
+ u32 msk, unsigned int clr,
+ unsigned int set)
+{
+ irq_remove_generic_chip(gc, msk, clr, set);
+ irq_free_generic_chip(gc);
+}
+
static inline struct irq_chip_type *irq_data_get_chip_type(struct irq_data *d)
{
return container_of(d->chip, struct irq_chip_type, chip);
* @rcu: rcu head for delayed free
* @kobj: kobject used to represent this struct in sysfs
* @dir: /proc/irq/ procfs entry
+ * @debugfs_file: dentry for the debugfs file
* @name: flow handler name for /proc/interrupts output
*/
struct irq_desc {
#ifdef CONFIG_PROC_FS
struct proc_dir_entry *dir;
#endif
+#ifdef CONFIG_GENERIC_IRQ_DEBUGFS
+ struct dentry *debugfs_file;
+#endif
#ifdef CONFIG_SPARSE_IRQ
struct rcu_head rcu;
struct kobject kobj;
* @host_data: private data pointer for use by owner. Not touched by irq_domain
* core code.
* @flags: host per irq_domain flags
+ * @mapcount: The number of mapped interrupts
*
* Optional elements
* @of_node: Pointer to device tree nodes associated with the irq_domain. Used
* setting up one or more generic chips for interrupt controllers
* drivers using the generic chip library which uses this pointer.
* @parent: Pointer to parent irq_domain to support hierarchy irq_domains
+ * @debugfs_file: dentry for the domain debugfs file
*
* Revmap data, used internally by irq_domain
* @revmap_direct_max_irq: The largest hwirq that can be set for controllers that
const struct irq_domain_ops *ops;
void *host_data;
unsigned int flags;
+ unsigned int mapcount;
/* Optional data */
struct fwnode_handle *fwnode;
#ifdef CONFIG_IRQ_DOMAIN_HIERARCHY
struct irq_domain *parent;
#endif
+#ifdef CONFIG_GENERIC_IRQ_DEBUGFS
+ struct dentry *debugfs_file;
+#endif
/* reverse map data. The linear map gets appended to the irq_domain */
irq_hw_number_t hwirq_max;
/* Irq domain is hierarchical */
IRQ_DOMAIN_FLAG_HIERARCHY = (1 << 0),
- /* Core calls alloc/free recursive through the domain hierarchy. */
- IRQ_DOMAIN_FLAG_AUTO_RECURSIVE = (1 << 1),
+ /* Irq domain name was allocated in __irq_domain_add() */
+ IRQ_DOMAIN_NAME_ALLOCATED = (1 << 6),
/* Irq domain is an IPI domain with virq per cpu */
IRQ_DOMAIN_FLAG_IPI_PER_CPU = (1 << 2),
}
#ifdef CONFIG_IRQ_DOMAIN
-struct fwnode_handle *irq_domain_alloc_fwnode(void *data);
+struct fwnode_handle *__irq_domain_alloc_fwnode(unsigned int type, int id,
+ const char *name, void *data);
+
+enum {
+ IRQCHIP_FWNODE_REAL,
+ IRQCHIP_FWNODE_NAMED,
+ IRQCHIP_FWNODE_NAMED_ID,
+};
+
+static inline
+struct fwnode_handle *irq_domain_alloc_named_fwnode(const char *name)
+{
+ return __irq_domain_alloc_fwnode(IRQCHIP_FWNODE_NAMED, 0, name, NULL);
+}
+
+static inline
+struct fwnode_handle *irq_domain_alloc_named_id_fwnode(const char *name, int id)
+{
+ return __irq_domain_alloc_fwnode(IRQCHIP_FWNODE_NAMED_ID, id, name,
+ NULL);
+}
+
+static inline struct fwnode_handle *irq_domain_alloc_fwnode(void *data)
+{
+ return __irq_domain_alloc_fwnode(IRQCHIP_FWNODE_REAL, 0, NULL, data);
+}
+
void irq_domain_free_fwnode(struct fwnode_handle *fwnode);
struct irq_domain *__irq_domain_add(struct fwnode_handle *fwnode, int size,
irq_hw_number_t hwirq_max, int direct_max,
return fwnode && fwnode->type == FWNODE_IRQCHIP;
}
+extern void irq_domain_update_bus_token(struct irq_domain *domain,
+ enum irq_domain_bus_token bus_token);
+
static inline
struct irq_domain *irq_find_matching_fwnode(struct fwnode_handle *fwnode,
enum irq_domain_bus_token bus_token)
NULL);
}
-extern int irq_domain_alloc_irqs_recursive(struct irq_domain *domain,
+extern int irq_domain_alloc_irqs_hierarchy(struct irq_domain *domain,
unsigned int irq_base,
unsigned int nr_irqs, void *arg);
extern int irq_domain_set_hwirq_and_chip(struct irq_domain *domain,
extern bool early_boot_irqs_disabled;
-/* Values used for system_state */
+/*
+ * Values used for system_state. Ordering of the states must not be changed
+ * as code checks for <, <=, >, >= STATE.
+ */
extern enum system_states {
SYSTEM_BOOTING,
+ SYSTEM_SCHEDULING,
SYSTEM_RUNNING,
SYSTEM_HALT,
SYSTEM_POWER_OFF,
* 2 of the License, or (at your option) any later version.
*
*
- * See Documentation/security/keys.txt for information on keys/keyrings.
+ * See Documentation/security/keys/core.rst for information on keys/keyrings.
*/
#ifndef _LINUX_KEY_H
int kobject_uevent(struct kobject *kobj, enum kobject_action action);
int kobject_uevent_env(struct kobject *kobj, enum kobject_action action,
char *envp[]);
+int kobject_synth_uevent(struct kobject *kobj, const char *buf, size_t count);
__printf(2, 3)
int add_uevent_var(struct kobj_uevent_env *env, const char *format, ...);
-int kobject_action_type(const char *buf, size_t count,
- enum kobject_action *type);
-
#endif /* _KOBJECT_H_ */
struct kvm_kernel_irqfd {
/* Used for MSI fast-path */
struct kvm *kvm;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
/* Update side is protected by irqfds.lock */
struct kvm_kernel_irq_routing_entry irq_entry;
seqcount_t irq_entry_sc;
*
*
* libata documentation is available via 'make {ps|pdf}docs',
- * as Documentation/DocBook/libata.*
+ * as Documentation/driver-api/libata.rst
*
*/
#define llist_for_each(pos, node) \
for ((pos) = (node); pos; (pos) = (pos)->next)
+/**
+ * llist_for_each_safe - iterate over some deleted entries of a lock-less list
+ * safe against removal of list entry
+ * @pos: the &struct llist_node to use as a loop cursor
+ * @n: another &struct llist_node to use as temporary storage
+ * @node: the first entry of deleted list entries
+ *
+ * In general, some entries of the lock-less list can be traversed
+ * safely only after being deleted from list, so start with an entry
+ * instead of list head.
+ *
+ * If being used on entries deleted from lock-less list directly, the
+ * traverse order is from the newest to the oldest added entry. If
+ * you want to traverse from the oldest to the newest, you must
+ * reverse the order by yourself before traversing.
+ */
+#define llist_for_each_safe(pos, n, node) \
+ for ((pos) = (node); (pos) && ((n) = (pos)->next, true); (pos) = (n))
+
/**
* llist_for_each_entry - iterate over some deleted entries of lock-less list of given type
* @pos: the type * to use as a loop cursor.
#include <linux/rculist.h>
/**
+ * union security_list_options - Linux Security Module hook function list
+ *
* Security hooks for program execution operations.
*
* @bprm_set_creds:
* @value will be set to the allocated attribute value.
* @len will be set to the length of the value.
* Returns 0 if @name and @value have been successfully set,
- * -EOPNOTSUPP if no security attribute is needed, or
- * -ENOMEM on memory allocation failure.
+ * -EOPNOTSUPP if no security attribute is needed, or
+ * -ENOMEM on memory allocation failure.
* @inode_create:
* Check permission to create a regular file.
* @dir contains inode structure of the parent of the new file.
* process @tsk. Note that this hook is sometimes called from interrupt.
* Note that the fown_struct, @fown, is never outside the context of a
* struct file, so the file structure (and associated security information)
- * can always be obtained:
- * container_of(fown, struct file, f_owner)
+ * can always be obtained: container_of(fown, struct file, f_owner)
* @tsk contains the structure of task receiving signal.
* @fown contains the file owner information.
* @sig is the signal that will be sent. When 0, kernel sends SIGIO.
* to receive an open file descriptor via socket IPC.
* @file contains the file structure being received.
* Return 0 if permission is granted.
- * @file_open
+ * @file_open:
* Save open-time permission checking state for later use upon
* file_permission, and recheck access if anything has changed
* since inode_permission.
* @sma contains the semaphore structure. May be NULL.
* @cmd contains the operation to be performed.
* Return 0 if permission is granted.
- * @sem_semop
+ * @sem_semop:
* Check permissions before performing operations on members of the
* semaphore set @sma. If the @alter flag is nonzero, the semaphore set
* may be modified.
* @alter contains the flag indicating whether changes are to be made.
* Return 0 if permission is granted.
*
- * @binder_set_context_mgr
+ * @binder_set_context_mgr:
* Check whether @mgr is allowed to be the binder context manager.
* @mgr contains the task_struct for the task being registered.
* Return 0 if permission is granted.
- * @binder_transaction
+ * @binder_transaction:
* Check whether @from is allowed to invoke a binder transaction call
* to @to.
* @from contains the task_struct for the sending task.
* @to contains the task_struct for the receiving task.
- * @binder_transfer_binder
+ * @binder_transfer_binder:
* Check whether @from is allowed to transfer a binder reference to @to.
* @from contains the task_struct for the sending task.
* @to contains the task_struct for the receiving task.
- * @binder_transfer_file
+ * @binder_transfer_file:
* Check whether @from is allowed to transfer @file to @to.
* @from contains the task_struct for the sending task.
* @file contains the struct file being transferred.
* @cred contains the credentials to use.
* @ns contains the user namespace we want the capability in
* @cap contains the capability <include/linux/capability.h>.
- * @audit: Whether to write an audit message or not
+ * @audit contains whether to write an audit message or not
* Return 0 if the capability is granted for @tsk.
* @syslog:
* Check permission before accessing the kernel message ring or changing
* @inode we wish to get the security context of.
* @ctx is a pointer in which to place the allocated security context.
* @ctxlen points to the place to put the length of @ctx.
- * This is the main security structure.
*/
-
union security_list_options {
int (*binder_set_context_mgr)(struct task_struct *mgr);
int (*binder_transaction)(struct task_struct *from,
#define AXP806_BUS_ADDR_EXT 0xfe
#define AXP806_REG_ADDR_EXT 0xff
+#define AXP803_POLYPHASE_CTRL 0x14
+#define AXP803_FLDO1_V_OUT 0x1c
+#define AXP803_FLDO2_V_OUT 0x1d
+#define AXP803_DCDC1_V_OUT 0x20
+#define AXP803_DCDC2_V_OUT 0x21
+#define AXP803_DCDC3_V_OUT 0x22
+#define AXP803_DCDC4_V_OUT 0x23
+#define AXP803_DCDC5_V_OUT 0x24
+#define AXP803_DCDC6_V_OUT 0x25
+#define AXP803_DCDC_FREQ_CTRL 0x3b
+
/* Interrupt */
#define AXP152_IRQ1_EN 0x40
#define AXP152_IRQ2_EN 0x41
AXP809_REG_ID_MAX,
};
+enum {
+ AXP803_DCDC1 = 0,
+ AXP803_DCDC2,
+ AXP803_DCDC3,
+ AXP803_DCDC4,
+ AXP803_DCDC5,
+ AXP803_DCDC6,
+ AXP803_DC1SW,
+ AXP803_ALDO1,
+ AXP803_ALDO2,
+ AXP803_ALDO3,
+ AXP803_DLDO1,
+ AXP803_DLDO2,
+ AXP803_DLDO3,
+ AXP803_DLDO4,
+ AXP803_ELDO1,
+ AXP803_ELDO2,
+ AXP803_ELDO3,
+ AXP803_FLDO1,
+ AXP803_FLDO2,
+ AXP803_RTC_LDO,
+ AXP803_LDO_IO0,
+ AXP803_LDO_IO1,
+ AXP803_REG_ID_MAX,
+};
+
/* IRQs */
enum {
AXP152_IRQ_LDO0IN_CONNECT = 1,
TPS65917_REG_SMPS3,
TPS65917_REG_SMPS4,
TPS65917_REG_SMPS5,
+ TPS65917_REG_SMPS12,
/* LDO regulators */
TPS65917_REG_LDO1,
TPS65917_REG_LDO2,
TPS65917_EXTERNAL_REQSTR_ID_SMPS3,
TPS65917_EXTERNAL_REQSTR_ID_SMPS4,
TPS65917_EXTERNAL_REQSTR_ID_SMPS5,
+ TPS65917_EXTERNAL_REQSTR_ID_SMPS12,
TPS65917_EXTERNAL_REQSTR_ID_LDO1,
TPS65917_EXTERNAL_REQSTR_ID_LDO2,
TPS65917_EXTERNAL_REQSTR_ID_LDO3,
#define TIM_CR1_DIR BIT(4) /* Counter Direction */
#define TIM_CR1_ARPE BIT(7) /* Auto-reload Preload Ena */
#define TIM_CR2_MMS (BIT(4) | BIT(5) | BIT(6)) /* Master mode selection */
+#define TIM_CR2_MMS2 GENMASK(23, 20) /* Master mode selection 2 */
#define TIM_SMCR_SMS (BIT(0) | BIT(1) | BIT(2)) /* Slave mode selection */
#define TIM_SMCR_TS (BIT(4) | BIT(5) | BIT(6)) /* Trigger selection */
#define TIM_DIER_UIE BIT(0) /* Update interrupt */
#define MAX_TIM_PSC 0xFFFF
#define TIM_CR2_MMS_SHIFT 4
+#define TIM_CR2_MMS2_SHIFT 20
#define TIM_SMCR_TS_SHIFT 4
#define TIM_BDTR_BKF_MASK 0xF
#define TIM_BDTR_BKF_SHIFT 16
#define tmio_ioread16(addr) readw(addr)
#define tmio_ioread16_rep(r, b, l) readsw(r, b, l)
#define tmio_ioread32(addr) \
- (((u32) readw((addr))) | (((u32) readw((addr) + 2)) << 16))
+ (((u32)readw((addr))) | (((u32)readw((addr) + 2)) << 16))
#define tmio_iowrite8(val, addr) writeb((val), (addr))
#define tmio_iowrite16(val, addr) writew((val), (addr))
#define tmio_iowrite16_rep(r, b, l) writesw(r, b, l)
#define tmio_iowrite32(val, addr) \
do { \
- writew((val), (addr)); \
- writew((val) >> 16, (addr) + 2); \
+ writew((val), (addr)); \
+ writew((val) >> 16, (addr) + 2); \
} while (0)
#define CNF_CMD 0x04
} while (0)
/* tmio MMC platform flags */
-#define TMIO_MMC_WRPROTECT_DISABLE (1 << 0)
+#define TMIO_MMC_WRPROTECT_DISABLE BIT(0)
/*
* Some controllers can support a 2-byte block size when the bus width
* is configured in 4-bit mode.
*/
-#define TMIO_MMC_BLKSZ_2BYTES (1 << 1)
+#define TMIO_MMC_BLKSZ_2BYTES BIT(1)
/*
* Some controllers can support SDIO IRQ signalling.
*/
-#define TMIO_MMC_SDIO_IRQ (1 << 2)
+#define TMIO_MMC_SDIO_IRQ BIT(2)
-/* Some features are only available or tested on RCar Gen2 or later */
-#define TMIO_MMC_MIN_RCAR2 (1 << 3)
+/* Some features are only available or tested on R-Car Gen2 or later */
+#define TMIO_MMC_MIN_RCAR2 BIT(3)
/*
* Some controllers require waiting for the SD bus to become
* idle before writing to some registers.
*/
-#define TMIO_MMC_HAS_IDLE_WAIT (1 << 4)
+#define TMIO_MMC_HAS_IDLE_WAIT BIT(4)
/*
* A GPIO is used for card hotplug detection. We need an extra flag for this,
* because 0 is a valid GPIO number too, and requiring users to specify
* cd_gpio < 0 to disable GPIO hotplug would break backwards compatibility.
*/
-#define TMIO_MMC_USE_GPIO_CD (1 << 5)
+#define TMIO_MMC_USE_GPIO_CD BIT(5)
/*
* Some controllers doesn't have over 0x100 register.
* it is used to checking accessibility of
* CTL_SD_CARD_CLK_CTL / CTL_CLK_AND_WAIT_CTL
*/
-#define TMIO_MMC_HAVE_HIGH_REG (1 << 6)
+#define TMIO_MMC_HAVE_HIGH_REG BIT(6)
/*
* Some controllers have CMD12 automatically
* issue/non-issue register
*/
-#define TMIO_MMC_HAVE_CMD12_CTRL (1 << 7)
+#define TMIO_MMC_HAVE_CMD12_CTRL BIT(7)
/* Controller has some SDIO status bits which must be 1 */
-#define TMIO_MMC_SDIO_STATUS_SETBITS (1 << 8)
+#define TMIO_MMC_SDIO_STATUS_SETBITS BIT(8)
/*
* Some controllers have a 32-bit wide data port register
*/
-#define TMIO_MMC_32BIT_DATA_PORT (1 << 9)
+#define TMIO_MMC_32BIT_DATA_PORT BIT(9)
/*
* Some controllers allows to set SDx actual clock
*/
-#define TMIO_MMC_CLK_ACTUAL (1 << 10)
+#define TMIO_MMC_CLK_ACTUAL BIT(10)
int tmio_core_mmc_enable(void __iomem *cnf, int shift, unsigned long base);
int tmio_core_mmc_resume(void __iomem *cnf, int shift, unsigned long base);
struct tmio_fb_data {
int (*lcd_set_power)(struct platform_device *fb_dev,
- bool on);
+ bool on);
int (*lcd_mode)(struct platform_device *fb_dev,
- const struct fb_videomode *mode);
+ const struct fb_videomode *mode);
int num_modes;
struct fb_videomode *modes;
int width;
};
-
#endif
bool en_ck32k_xtal;
bool en_dev_slp;
bool pm_off;
- struct tps65910_sleep_keepon_data *slp_keepon;
+ struct tps65910_sleep_keepon_data slp_keepon;
bool en_gpio_sleep[TPS6591X_MAX_NUM_GPIO];
unsigned long regulator_ext_sleep_control[TPS65910_NUM_REGS];
struct regulator_init_data *tps65910_pmic_init_data[TPS65910_NUM_REGS];
#define VHOST_NET_MINOR 238
#define UHID_MINOR 239
#define USERIO_MINOR 240
+#define VHOST_VSOCK_MINOR 241
#define MISC_DYNAMIC_MINOR 255
struct device;
#include <asm/page.h>
+#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH
+#include <asm/tlbbatch.h>
+#endif
+
#define USE_SPLIT_PTE_PTLOCKS (NR_CPUS >= CONFIG_SPLIT_PTLOCK_CPUS)
#define USE_SPLIT_PMD_PTLOCKS (USE_SPLIT_PTE_PTLOCKS && \
IS_ENABLED(CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK))
struct tlbflush_unmap_batch {
#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH
/*
- * Each bit set is a CPU that potentially has a TLB entry for one of
- * the PFNs being flushed. See set_tlb_ubc_flush_pending().
+ * The arch code makes the following promise: generic code can modify a
+ * PTE, then call arch_tlbbatch_add_mm() (which internally provides all
+ * needed barriers), then call arch_tlbbatch_flush(), and the entries
+ * will be flushed on all CPUs by the time that arch_tlbbatch_flush()
+ * returns.
*/
- struct cpumask cpumask;
+ struct arch_tlbflush_unmap_batch arch;
- /* True if any bit in cpumask is set */
+ /* True if a flush is needed. */
bool flush_required;
/*
struct mmc_part part[MMC_NUM_PHY_PARTITION]; /* physical partitions */
unsigned int nr_parts;
- struct mmc_queue_req *mqrq; /* Shared queue structure */
unsigned int bouncesz; /* Bounce buffer size */
- int qdepth; /* Shared queue depth */
};
static inline bool mmc_large_sector(struct mmc_card *card)
int (*get_cd)(struct mmc_host *host);
void (*enable_sdio_irq)(struct mmc_host *host, int enable);
+ void (*ack_sdio_irq)(struct mmc_host *host);
/* optional callback for HC quirks */
void (*init_card)(struct mmc_host *host, struct mmc_card *card);
*/
struct mmc_slot {
int cd_irq;
+ bool cd_wake_enabled;
void *handler_priv;
};
#define MMC_CAP_UHS_SDR50 (1 << 18) /* Host supports UHS SDR50 mode */
#define MMC_CAP_UHS_SDR104 (1 << 19) /* Host supports UHS SDR104 mode */
#define MMC_CAP_UHS_DDR50 (1 << 20) /* Host supports UHS DDR50 mode */
+#define MMC_CAP_NO_BOUNCE_BUFF (1 << 21) /* Disable bounce buffers on host */
#define MMC_CAP_DRIVER_TYPE_A (1 << 23) /* Host supports Driver Type A */
#define MMC_CAP_DRIVER_TYPE_C (1 << 24) /* Host supports Driver Type C */
#define MMC_CAP_DRIVER_TYPE_D (1 << 25) /* Host supports Driver Type D */
+#define MMC_CAP_CD_WAKE (1 << 28) /* Enable card detect wake */
#define MMC_CAP_CMD_DURING_TFR (1 << 29) /* Commands during data transfer */
#define MMC_CAP_CMD23 (1 << 30) /* CMD23 supported. */
#define MMC_CAP_HW_RESET (1 << 31) /* Hardware reset */
#define MMC_CAP2_HS200_1_2V_SDR (1 << 6) /* can support */
#define MMC_CAP2_HS200 (MMC_CAP2_HS200_1_8V_SDR | \
MMC_CAP2_HS200_1_2V_SDR)
-#define MMC_CAP2_HC_ERASE_SZ (1 << 9) /* High-capacity erase size */
#define MMC_CAP2_CD_ACTIVE_HIGH (1 << 10) /* Card-detect signal active high */
#define MMC_CAP2_RO_ACTIVE_HIGH (1 << 11) /* Write-protect signal active high */
#define MMC_CAP2_PACKED_RD (1 << 12) /* Allow packed read */
unsigned int sdio_irqs;
struct task_struct *sdio_irq_thread;
+ struct delayed_work sdio_irq_work;
bool sdio_irq_pending;
atomic_t sdio_irq_thread_abort;
}
void sdio_run_irqs(struct mmc_host *host);
+void sdio_signal_irq(struct mmc_host *host);
#ifdef CONFIG_REGULATOR
int mmc_regulator_get_ocrmask(struct regulator *supply);
hwparam_ioport, /* Module parameter configures an I/O port */
hwparam_iomem, /* Module parameter configures an I/O mem address */
hwparam_ioport_or_iomem, /* Module parameter could be either, depending on other option */
- hwparam_irq, /* Module parameter configures an I/O port */
+ hwparam_irq, /* Module parameter configures an IRQ */
hwparam_dma, /* Module parameter configures a DMA channel */
hwparam_dma_addr, /* Module parameter configures a DMA buffer address */
hwparam_other, /* Module parameter configures some other value */
* Minimum amount of bit errors per @ecc_step_ds guaranteed
* to be correctable. If unknown, set to zero.
* @ecc_step_ds: [INTERN] ECC step required by the @ecc_strength_ds,
- * also from the datasheet. It is the recommended ECC step
+ * also from the datasheet. It is the recommended ECC step
* size, if known; if unknown, set to zero.
* @onfi_timing_mode_default: [INTERN] default ONFI timing mode. This field is
* set to the actually used ONFI mode if the chip is
* raisins, and once those are gone this will be removed.
*
* Returns:
- * MUTEX_TRYLOCK_FAILED - trylock failed,
- * MUTEX_TRYLOCK_SUCCESS - lock acquired,
- * MUTEX_TRYLOCK_RECURSIVE - we already owned the lock.
+ * - MUTEX_TRYLOCK_FAILED - trylock failed,
+ * - MUTEX_TRYLOCK_SUCCESS - lock acquired,
+ * - MUTEX_TRYLOCK_RECURSIVE - we already owned the lock.
*/
static inline /* __deprecated */ __must_check enum mutex_trylock_recursive_enum
mutex_trylock_recursive(struct mutex *lock)
--- /dev/null
+/*
+ * mux/consumer.h - definitions for the multiplexer consumer interface
+ *
+ * Copyright (C) 2017 Axentia Technologies AB
+ *
+ * Author: Peter Rosin <peda@axentia.se>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef _LINUX_MUX_CONSUMER_H
+#define _LINUX_MUX_CONSUMER_H
+
+struct device;
+struct mux_control;
+
+unsigned int mux_control_states(struct mux_control *mux);
+int __must_check mux_control_select(struct mux_control *mux,
+ unsigned int state);
+int __must_check mux_control_try_select(struct mux_control *mux,
+ unsigned int state);
+int mux_control_deselect(struct mux_control *mux);
+
+struct mux_control *mux_control_get(struct device *dev, const char *mux_name);
+void mux_control_put(struct mux_control *mux);
+
+struct mux_control *devm_mux_control_get(struct device *dev,
+ const char *mux_name);
+
+#endif /* _LINUX_MUX_CONSUMER_H */
--- /dev/null
+/*
+ * mux/driver.h - definitions for the multiplexer driver interface
+ *
+ * Copyright (C) 2017 Axentia Technologies AB
+ *
+ * Author: Peter Rosin <peda@axentia.se>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef _LINUX_MUX_DRIVER_H
+#define _LINUX_MUX_DRIVER_H
+
+#include <dt-bindings/mux/mux.h>
+#include <linux/device.h>
+#include <linux/semaphore.h>
+
+struct mux_chip;
+struct mux_control;
+
+/**
+ * struct mux_control_ops - Mux controller operations for a mux chip.
+ * @set: Set the state of the given mux controller.
+ */
+struct mux_control_ops {
+ int (*set)(struct mux_control *mux, int state);
+};
+
+/**
+ * struct mux_control - Represents a mux controller.
+ * @lock: Protects the mux controller state.
+ * @chip: The mux chip that is handling this mux controller.
+ * @cached_state: The current mux controller state, or -1 if none.
+ * @states: The number of mux controller states.
+ * @idle_state: The mux controller state to use when inactive, or one
+ * of MUX_IDLE_AS_IS and MUX_IDLE_DISCONNECT.
+ *
+ * Mux drivers may only change @states and @idle_state, and may only do so
+ * between allocation and registration of the mux controller. Specifically,
+ * @cached_state is internal to the mux core and should never be written by
+ * mux drivers.
+ */
+struct mux_control {
+ struct semaphore lock; /* protects the state of the mux */
+
+ struct mux_chip *chip;
+ int cached_state;
+
+ unsigned int states;
+ int idle_state;
+};
+
+/**
+ * struct mux_chip - Represents a chip holding mux controllers.
+ * @controllers: Number of mux controllers handled by the chip.
+ * @mux: Array of mux controllers that are handled.
+ * @dev: Device structure.
+ * @id: Used to identify the device internally.
+ * @ops: Mux controller operations.
+ */
+struct mux_chip {
+ unsigned int controllers;
+ struct mux_control *mux;
+ struct device dev;
+ int id;
+
+ const struct mux_control_ops *ops;
+};
+
+#define to_mux_chip(x) container_of((x), struct mux_chip, dev)
+
+/**
+ * mux_chip_priv() - Get the extra memory reserved by mux_chip_alloc().
+ * @mux_chip: The mux-chip to get the private memory from.
+ *
+ * Return: Pointer to the private memory reserved by the allocator.
+ */
+static inline void *mux_chip_priv(struct mux_chip *mux_chip)
+{
+ return &mux_chip->mux[mux_chip->controllers];
+}
+
+struct mux_chip *mux_chip_alloc(struct device *dev,
+ unsigned int controllers, size_t sizeof_priv);
+int mux_chip_register(struct mux_chip *mux_chip);
+void mux_chip_unregister(struct mux_chip *mux_chip);
+void mux_chip_free(struct mux_chip *mux_chip);
+
+struct mux_chip *devm_mux_chip_alloc(struct device *dev,
+ unsigned int controllers,
+ size_t sizeof_priv);
+int devm_mux_chip_register(struct device *dev, struct mux_chip *mux_chip);
+
+/**
+ * mux_control_get_index() - Get the index of the given mux controller
+ * @mux: The mux-control to get the index for.
+ *
+ * Return: The index of the mux controller within the mux chip the mux
+ * controller is a part of.
+ */
+static inline unsigned int mux_control_get_index(struct mux_control *mux)
+{
+ return mux - mux->chip->mux;
+}
+
+#endif /* _LINUX_MUX_DRIVER_H */
/**
* struct net_device - The DEVICE structure.
- * Actually, this whole structure is a big mistake. It mixes I/O
- * data with strictly "high-level" data, and it has to know about
- * almost every data structure used in the INET module.
+ *
+ * Actually, this whole structure is a big mistake. It mixes I/O
+ * data with strictly "high-level" data, and it has to know about
+ * almost every data structure used in the INET module.
*
* @name: This is the first field of the "visible" part of this structure
* (i.e. as seen by users in the "Space.c" file). It is the name
- * of the interface.
+ * of the interface.
*
* @name_hlist: Device name hash chain, please keep it close to name[]
* @ifalias: SNMP alias
};
-#define FCNVME_ASSOC_HOSTID_LEN 16
#define FCNVME_ASSOC_HOSTNQN_LEN 256
#define FCNVME_ASSOC_SUBNQN_LEN 256
__be16 cntlid;
__be16 sqsize;
__be32 rsvd52;
- u8 hostid[FCNVME_ASSOC_HOSTID_LEN];
+ uuid_t hostid;
u8 hostnqn[FCNVME_ASSOC_HOSTNQN_LEN];
u8 subnqn[FCNVME_ASSOC_SUBNQN_LEN];
u8 rsvd632[384];
#define _LINUX_NVME_H
#include <linux/types.h>
+#include <linux/uuid.h>
/* NQN names in commands fields specified one size */
#define NVMF_NQN_FIELD_LEN 256
NVMF_RDMA_CMS_RDMA_CM = 1, /* Sockets based endpoint addressing */
};
-#define NVMF_AQ_DEPTH 32
+#define NVME_AQ_DEPTH 32
enum {
NVME_REG_CAP = 0x0000, /* Controller Capabilities */
NVME_REG_ACQ = 0x0030, /* Admin CQ Base Address */
NVME_REG_CMBLOC = 0x0038, /* Controller Memory Buffer Location */
NVME_REG_CMBSZ = 0x003c, /* Controller Memory Buffer Size */
+ NVME_REG_DBS = 0x1000, /* SQ 0 Tail Doorbell */
};
#define NVME_CAP_MQES(cap) ((cap) & 0xffff)
__u8 tnvmcap[16];
__u8 unvmcap[16];
__le32 rpmbs;
- __u8 rsvd316[4];
+ __le16 edstt;
+ __u8 dsto;
+ __u8 fwug;
__le16 kas;
- __u8 rsvd322[190];
+ __le16 hctma;
+ __le16 mntmt;
+ __le16 mxtmt;
+ __le32 sanicap;
+ __u8 rsvd332[180];
__u8 sqes;
__u8 cqes;
__le16 maxcmd;
NVME_CTRL_ONCS_WRITE_ZEROES = 1 << 3,
NVME_CTRL_VWC_PRESENT = 1 << 0,
NVME_CTRL_OACS_SEC_SUPP = 1 << 0,
+ NVME_CTRL_OACS_DIRECTIVES = 1 << 5,
NVME_CTRL_OACS_DBBUF_SUPP = 1 << 7,
};
__le16 nabsn;
__le16 nabo;
__le16 nabspf;
- __u16 rsvd46;
+ __le16 noiob;
__u8 nvmcap[16];
__u8 rsvd64[40];
__u8 nguid[16];
NVME_ID_CNS_NS = 0x00,
NVME_ID_CNS_CTRL = 0x01,
NVME_ID_CNS_NS_ACTIVE_LIST = 0x02,
+ NVME_ID_CNS_NS_DESC_LIST = 0x03,
NVME_ID_CNS_NS_PRESENT_LIST = 0x10,
NVME_ID_CNS_NS_PRESENT = 0x11,
NVME_ID_CNS_CTRL_NS_LIST = 0x12,
NVME_ID_CNS_CTRL_LIST = 0x13,
};
+enum {
+ NVME_DIR_IDENTIFY = 0x00,
+ NVME_DIR_STREAMS = 0x01,
+ NVME_DIR_SND_ID_OP_ENABLE = 0x01,
+ NVME_DIR_SND_ST_OP_REL_ID = 0x01,
+ NVME_DIR_SND_ST_OP_REL_RSC = 0x02,
+ NVME_DIR_RCV_ID_OP_PARAM = 0x01,
+ NVME_DIR_RCV_ST_OP_PARAM = 0x01,
+ NVME_DIR_RCV_ST_OP_STATUS = 0x02,
+ NVME_DIR_RCV_ST_OP_RESOURCE = 0x03,
+ NVME_DIR_ENDIR = 0x01,
+};
+
enum {
NVME_NS_FEAT_THIN = 1 << 0,
NVME_NS_FLBAS_LBA_MASK = 0xf,
NVME_NS_DPS_PI_TYPE3 = 3,
};
+struct nvme_ns_id_desc {
+ __u8 nidt;
+ __u8 nidl;
+ __le16 reserved;
+};
+
+#define NVME_NIDT_EUI64_LEN 8
+#define NVME_NIDT_NGUID_LEN 16
+#define NVME_NIDT_UUID_LEN 16
+
+enum {
+ NVME_NIDT_EUI64 = 0x01,
+ NVME_NIDT_NGUID = 0x02,
+ NVME_NIDT_UUID = 0x03,
+};
+
struct nvme_smart_log {
__u8 critical_warning;
__u8 temperature[2];
NVME_RW_PRINFO_PRCHK_APP = 1 << 11,
NVME_RW_PRINFO_PRCHK_GUARD = 1 << 12,
NVME_RW_PRINFO_PRACT = 1 << 13,
+ NVME_RW_DTYPE_STREAMS = 1 << 4,
};
struct nvme_dsm_cmd {
__le64 entries[32];
};
+enum {
+ NVME_HOST_MEM_ENABLE = (1 << 0),
+ NVME_HOST_MEM_RETURN = (1 << 1),
+};
+
/* Admin commands */
enum nvme_admin_opcode {
nvme_admin_download_fw = 0x11,
nvme_admin_ns_attach = 0x15,
nvme_admin_keep_alive = 0x18,
+ nvme_admin_directive_send = 0x19,
+ nvme_admin_directive_recv = 0x1a,
nvme_admin_dbbuf = 0x7C,
nvme_admin_format_nvm = 0x80,
nvme_admin_security_send = 0x81,
__u32 rsvd11[5];
};
+#define NVME_IDENTIFY_DATA_SIZE 4096
+
struct nvme_features {
__u8 opcode;
__u8 flags;
union nvme_data_ptr dptr;
__le32 fid;
__le32 dword11;
- __u32 rsvd12[4];
+ __le32 dword12;
+ __le32 dword13;
+ __le32 dword14;
+ __le32 dword15;
+};
+
+struct nvme_host_mem_buf_desc {
+ __le64 addr;
+ __le32 size;
+ __u32 rsvd;
};
struct nvme_create_cq {
__u32 rsvd14[2];
};
+struct nvme_directive_cmd {
+ __u8 opcode;
+ __u8 flags;
+ __u16 command_id;
+ __le32 nsid;
+ __u64 rsvd2[2];
+ union nvme_data_ptr dptr;
+ __le32 numd;
+ __u8 doper;
+ __u8 dtype;
+ __le16 dspec;
+ __u8 endir;
+ __u8 tdtype;
+ __u16 rsvd15;
+
+ __u32 rsvd16[3];
+};
+
/*
* Fabrics subcommands.
*/
};
struct nvmf_connect_data {
- __u8 hostid[16];
+ uuid_t hostid;
__le16 cntlid;
char resv4[238];
char subsysnqn[NVMF_NQN_FIELD_LEN];
__u32 rsvd12[6];
};
+struct streams_directive_params {
+ __u16 msl;
+ __u16 nssa;
+ __u16 nsso;
+ __u8 rsvd[10];
+ __u32 sws;
+ __u16 sgs;
+ __u16 nsa;
+ __u16 nso;
+ __u8 rsvd2[6];
+};
+
struct nvme_command {
union {
struct nvme_common_command common;
struct nvmf_property_set_command prop_set;
struct nvmf_property_get_command prop_get;
struct nvme_dbbuf dbbuf;
+ struct nvme_directive_cmd directive;
};
};
#define NVME_VS(major, minor, tertiary) \
(((major) << 16) | ((minor) << 8) | (tertiary))
+#define NVME_MAJOR(ver) ((ver) >> 16)
+#define NVME_MINOR(ver) (((ver) >> 8) & 0xff)
+#define NVME_TERTIARY(ver) ((ver) & 0xff)
+
#endif /* _LINUX_NVME_H */
extern struct padata_instance *padata_alloc_possible(
struct workqueue_struct *wq);
-extern struct padata_instance *padata_alloc(struct workqueue_struct *wq,
- const struct cpumask *pcpumask,
- const struct cpumask *cbcpumask);
extern void padata_free(struct padata_instance *pinst);
extern int padata_do_parallel(struct padata_instance *pinst,
struct padata_priv *padata, int cb_cpu);
/*
* Add an arbitrary waiter to a page's wait queue
*/
-extern void add_page_wait_queue(struct page *page, wait_queue_t *waiter);
+extern void add_page_wait_queue(struct page *page, wait_queue_entry_t *waiter);
/*
* Fault everything in given userspace address range in.
static inline void acpiphp_check_host_bridge(struct acpi_device *adev) { }
#endif
-extern const u8 pci_acpi_dsm_uuid[];
+extern const guid_t pci_acpi_dsm_guid;
#define DEVICE_LABEL_DSM 0x07
#define RESET_DELAY_DSM 0x08
#define FUNCTION_DELAY_DSM 0x09
u8 pm_cap; /* PM capability offset */
unsigned int pme_support:5; /* Bitmask of states from which PME#
can be generated */
- unsigned int pme_interrupt:1;
unsigned int pme_poll:1; /* Poll device's PME status bit */
unsigned int d1_support:1; /* Low power state D1 is supported */
unsigned int d2_support:1; /* Low power state D2 is supported */
unsigned int irq_managed:1;
unsigned int has_secondary_link:1;
unsigned int non_compliant_bars:1; /* broken BARs; ignore them */
+ unsigned int is_probed:1; /* device probing in progress */
pci_dev_flags_t dev_flags;
atomic_t enable_cnt; /* pci_enable_device has been called */
pci_power_t pci_choose_state(struct pci_dev *dev, pm_message_t state);
bool pci_pme_capable(struct pci_dev *dev, pci_power_t state);
void pci_pme_active(struct pci_dev *dev, bool enable);
-int __pci_enable_wake(struct pci_dev *dev, pci_power_t state,
- bool runtime, bool enable);
+int pci_enable_wake(struct pci_dev *dev, pci_power_t state, bool enable);
int pci_wake_from_d3(struct pci_dev *dev, bool enable);
int pci_prepare_to_sleep(struct pci_dev *dev);
int pci_back_from_sleep(struct pci_dev *dev);
void pci_d3cold_enable(struct pci_dev *dev);
void pci_d3cold_disable(struct pci_dev *dev);
-static inline int pci_enable_wake(struct pci_dev *dev, pci_power_t state,
- bool enable)
-{
- return __pci_enable_wake(dev, state, false, enable);
-}
-
/* PCI Virtual Channel */
int pci_save_vc_state(struct pci_dev *dev);
void pci_restore_vc_state(struct pci_dev *dev);
struct list_head sched_cb_entry;
int sched_cb_usage;
+
+ int online;
};
struct perf_output_handle {
#include <linux/mtd/nand.h>
#include <linux/mtd/partitions.h>
-#include <linux/serial.h>
/* Compact Flash */
struct at91_cf_data {
bool need_reset_workaround;
};
- /* Serial */
-struct atmel_uart_data {
- int num; /* port num */
- short use_dma_tx; /* use transmit DMA? */
- short use_dma_rx; /* use receive DMA? */
- void __iomem *regs; /* virt. base address, if any */
- struct serial_rs485 rs485; /* rs485 settings */
-};
-
/* FIXME: this needs a better location, but gets stuff building again */
extern int at91_suspend_entering_slow_clock(void);
struct mtk_chip_config {
u32 tx_mlsb;
u32 rx_mlsb;
+ u32 cs_pol;
+ u32 sample_sel;
};
#endif
unsigned int idle_notification:1;
unsigned int request_pending:1;
unsigned int deferred_resume:1;
- unsigned int run_wake:1;
unsigned int runtime_auto:1;
bool ignore_children:1;
unsigned int no_callbacks:1;
void dev_pm_opp_put_prop_name(struct opp_table *opp_table);
struct opp_table *dev_pm_opp_set_regulators(struct device *dev, const char * const names[], unsigned int count);
void dev_pm_opp_put_regulators(struct opp_table *opp_table);
+struct opp_table *dev_pm_opp_set_clkname(struct device *dev, const char * name);
+void dev_pm_opp_put_clkname(struct opp_table *opp_table);
struct opp_table *dev_pm_opp_register_set_opp_helper(struct device *dev, int (*set_opp)(struct dev_pm_set_opp_data *data));
void dev_pm_opp_register_put_opp_helper(struct opp_table *opp_table);
int dev_pm_opp_set_rate(struct device *dev, unsigned long target_freq);
static inline void dev_pm_opp_put_regulators(struct opp_table *opp_table) {}
+static inline struct opp_table *dev_pm_opp_set_clkname(struct device *dev, const char * name)
+{
+ return ERR_PTR(-ENOTSUPP);
+}
+
+static inline void dev_pm_opp_put_clkname(struct opp_table *opp_table) {}
+
static inline int dev_pm_opp_set_rate(struct device *dev, unsigned long target_freq)
{
return -ENOTSUPP;
atomic_add_unless(&dev->power.usage_count, -1, 0);
}
-static inline bool device_run_wake(struct device *dev)
-{
- return dev->power.run_wake;
-}
-
-static inline void device_set_run_wake(struct device *dev, bool enable)
-{
- dev->power.run_wake = enable;
-}
-
static inline bool pm_runtime_suspended(struct device *dev)
{
return dev->power.runtime_status == RPM_SUSPENDED
static inline void pm_suspend_ignore_children(struct device *dev, bool enable) {}
static inline void pm_runtime_get_noresume(struct device *dev) {}
static inline void pm_runtime_put_noidle(struct device *dev) {}
-static inline bool device_run_wake(struct device *dev) { return false; }
-static inline void device_set_run_wake(struct device *dev, bool enable) {}
static inline bool pm_runtime_suspended(struct device *dev) { return false; }
static inline bool pm_runtime_active(struct device *dev) { return true; }
static inline bool pm_runtime_status_suspended(struct device *dev) { return false; }
struct poll_table_entry {
struct file *filp;
unsigned long key;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
wait_queue_head_t *wait_address;
};
* @clock_gettime: Read the current time
* @clock_getres: Get the clock resolution
* @clock_settime: Set the current time value
- * @timer_create: Create a new timer
- * @timer_delete: Remove a previously created timer
- * @timer_gettime: Get remaining time and interval of a timer
- * @timer_settime: Set a timer's initial expiration and interval
- * @fasync: Optional character device fasync method
- * @mmap: Optional character device mmap method
* @open: Optional character device open method
* @release: Optional character device release method
* @ioctl: Optional character device ioctl method
int (*clock_settime)(struct posix_clock *pc,
const struct timespec64 *ts);
- int (*timer_create) (struct posix_clock *pc, struct k_itimer *kit);
-
- int (*timer_delete) (struct posix_clock *pc, struct k_itimer *kit);
-
- void (*timer_gettime)(struct posix_clock *pc,
- struct k_itimer *kit, struct itimerspec64 *tsp);
-
- int (*timer_settime)(struct posix_clock *pc,
- struct k_itimer *kit, int flags,
- struct itimerspec64 *tsp, struct itimerspec64 *old);
/*
* Optional character device methods:
*/
- int (*fasync) (struct posix_clock *pc,
- int fd, struct file *file, int on);
-
long (*ioctl) (struct posix_clock *pc,
unsigned int cmd, unsigned long arg);
- int (*mmap) (struct posix_clock *pc,
- struct vm_area_struct *vma);
-
int (*open) (struct posix_clock *pc, fmode_t f_mode);
uint (*poll) (struct posix_clock *pc,
#include <linux/timex.h>
#include <linux/alarmtimer.h>
+struct siginfo;
struct cpu_timer_list {
struct list_head entry;
#define FD_TO_CLOCKID(fd) ((~(clockid_t) (fd) << 3) | CLOCKFD)
#define CLOCKID_TO_FD(clk) ((unsigned int) ~((clk) >> 3))
-/* POSIX.1b interval timer structure. */
-struct k_itimer {
- struct list_head list; /* free/ allocate list */
- struct hlist_node t_hash;
- spinlock_t it_lock;
- clockid_t it_clock; /* which timer type */
- timer_t it_id; /* timer id */
- int it_overrun; /* overrun on pending signal */
- int it_overrun_last; /* overrun on last delivered signal */
- int it_requeue_pending; /* waiting to requeue this timer */
#define REQUEUE_PENDING 1
- int it_sigev_notify; /* notify word of sigevent struct */
- struct signal_struct *it_signal;
+
+/**
+ * struct k_itimer - POSIX.1b interval timer structure.
+ * @list: List head for binding the timer to signals->posix_timers
+ * @t_hash: Entry in the posix timer hash table
+ * @it_lock: Lock protecting the timer
+ * @kclock: Pointer to the k_clock struct handling this timer
+ * @it_clock: The posix timer clock id
+ * @it_id: The posix timer id for identifying the timer
+ * @it_active: Marker that timer is active
+ * @it_overrun: The overrun counter for pending signals
+ * @it_overrun_last: The overrun at the time of the last delivered signal
+ * @it_requeue_pending: Indicator that timer waits for being requeued on
+ * signal delivery
+ * @it_sigev_notify: The notify word of sigevent struct for signal delivery
+ * @it_interval: The interval for periodic timers
+ * @it_signal: Pointer to the creators signal struct
+ * @it_pid: The pid of the process/task targeted by the signal
+ * @it_process: The task to wakeup on clock_nanosleep (CPU timers)
+ * @sigq: Pointer to preallocated sigqueue
+ * @it: Union representing the various posix timer type
+ * internals. Also used for rcu freeing the timer.
+ */
+struct k_itimer {
+ struct list_head list;
+ struct hlist_node t_hash;
+ spinlock_t it_lock;
+ const struct k_clock *kclock;
+ clockid_t it_clock;
+ timer_t it_id;
+ int it_active;
+ int it_overrun;
+ int it_overrun_last;
+ int it_requeue_pending;
+ int it_sigev_notify;
+ ktime_t it_interval;
+ struct signal_struct *it_signal;
union {
- struct pid *it_pid; /* pid of process to send signal to */
- struct task_struct *it_process; /* for clock_nanosleep */
+ struct pid *it_pid;
+ struct task_struct *it_process;
};
- struct sigqueue *sigq; /* signal queue entry. */
+ struct sigqueue *sigq;
union {
struct {
- struct hrtimer timer;
- ktime_t interval;
+ struct hrtimer timer;
} real;
- struct cpu_timer_list cpu;
+ struct cpu_timer_list cpu;
struct {
- unsigned int clock;
- unsigned int node;
- unsigned long incr;
- unsigned long expires;
- } mmtimer;
- struct {
- struct alarm alarmtimer;
- ktime_t interval;
+ struct alarm alarmtimer;
} alarm;
- struct rcu_head rcu;
+ struct rcu_head rcu;
} it;
};
-struct k_clock {
- int (*clock_getres) (const clockid_t which_clock, struct timespec64 *tp);
- int (*clock_set) (const clockid_t which_clock,
- const struct timespec64 *tp);
- int (*clock_get) (const clockid_t which_clock, struct timespec64 *tp);
- int (*clock_adj) (const clockid_t which_clock, struct timex *tx);
- int (*timer_create) (struct k_itimer *timer);
- int (*nsleep) (const clockid_t which_clock, int flags,
- struct timespec64 *, struct timespec __user *);
- long (*nsleep_restart) (struct restart_block *restart_block);
- int (*timer_set) (struct k_itimer *timr, int flags,
- struct itimerspec64 *new_setting,
- struct itimerspec64 *old_setting);
- int (*timer_del) (struct k_itimer *timr);
-#define TIMER_RETRY 1
- void (*timer_get) (struct k_itimer *timr,
- struct itimerspec64 *cur_setting);
-};
-
-extern struct k_clock clock_posix_cpu;
-extern struct k_clock clock_posix_dynamic;
-
-void posix_timers_register_clock(const clockid_t clock_id, struct k_clock *new_clock);
-
-/* function to call to trigger timer event */
-int posix_timer_event(struct k_itimer *timr, int si_private);
-
-void posix_cpu_timer_schedule(struct k_itimer *timer);
-
void run_posix_cpu_timers(struct task_struct *task);
void posix_cpu_timers_exit(struct task_struct *task);
void posix_cpu_timers_exit_group(struct task_struct *task);
void set_process_cpu_timer(struct task_struct *task, unsigned int clock_idx,
u64 *newval, u64 *oldval);
-long clock_nanosleep_restart(struct restart_block *restart_block);
-
void update_rlimit_cpu(struct task_struct *task, unsigned long rlim_new);
+void posixtimer_rearm(struct siginfo *info);
+
#endif
#define current_pt_regs() task_pt_regs(current)
#endif
-#ifndef ptrace_signal_deliver
-#define ptrace_signal_deliver() ((void)0)
-#endif
-
/*
* unlike current_pt_regs(), this one is equal to task_pt_regs(current)
* on *all* architectures; the only reason to have a per-arch definition
LPSS_BSW_SSP,
LPSS_SPT_SSP,
LPSS_BXT_SSP,
+ LPSS_CNL_SSP,
};
struct ssp_device {
* unlimited scalability while maintaining a constant level of contention
* on the root node.
*
+ * This seemingly RCU-private file must be available to SRCU users
+ * because the size of the TREE SRCU srcu_struct structure depends
+ * on these definitions.
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
/*
* RCU segmented callback lists
*
+ * This seemingly RCU-private file must be available to SRCU users
+ * because the size of the TREE SRCU srcu_struct structure depends
+ * on these definitions.
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
#define __LINUX_RCUPDATE_H
#include <linux/types.h>
-#include <linux/cache.h>
-#include <linux/spinlock.h>
-#include <linux/threads.h>
-#include <linux/cpumask.h>
-#include <linux/seqlock.h>
-#include <linux/lockdep.h>
-#include <linux/debugobjects.h>
-#include <linux/bug.h>
#include <linux/compiler.h>
-#include <linux/ktime.h>
+#include <linux/atomic.h>
#include <linux/irqflags.h>
+#include <linux/preempt.h>
+#include <linux/bottom_half.h>
+#include <linux/lockdep.h>
+#include <asm/processor.h>
+#include <linux/cpumask.h>
-#include <asm/barrier.h>
-
-#ifndef CONFIG_TINY_RCU
-extern int rcu_expedited; /* for sysctl */
-extern int rcu_normal; /* also for sysctl */
-#endif /* #ifndef CONFIG_TINY_RCU */
-
-#ifdef CONFIG_TINY_RCU
-/* Tiny RCU doesn't expedite, as its purpose in life is instead to be tiny. */
-static inline bool rcu_gp_is_normal(void) /* Internal RCU use. */
-{
- return true;
-}
-static inline bool rcu_gp_is_expedited(void) /* Internal RCU use. */
-{
- return false;
-}
-
-static inline void rcu_expedite_gp(void)
-{
-}
-
-static inline void rcu_unexpedite_gp(void)
-{
-}
-#else /* #ifdef CONFIG_TINY_RCU */
-bool rcu_gp_is_normal(void); /* Internal RCU use. */
-bool rcu_gp_is_expedited(void); /* Internal RCU use. */
-void rcu_expedite_gp(void);
-void rcu_unexpedite_gp(void);
-#endif /* #else #ifdef CONFIG_TINY_RCU */
-
-enum rcutorture_type {
- RCU_FLAVOR,
- RCU_BH_FLAVOR,
- RCU_SCHED_FLAVOR,
- RCU_TASKS_FLAVOR,
- SRCU_FLAVOR,
- INVALID_RCU_FLAVOR
-};
-
-#if defined(CONFIG_TREE_RCU) || defined(CONFIG_PREEMPT_RCU)
-void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags,
- unsigned long *gpnum, unsigned long *completed);
-void rcutorture_record_test_transition(void);
-void rcutorture_record_progress(unsigned long vernum);
-void do_trace_rcu_torture_read(const char *rcutorturename,
- struct rcu_head *rhp,
- unsigned long secs,
- unsigned long c_old,
- unsigned long c);
-bool rcu_irq_enter_disabled(void);
-#else
-static inline void rcutorture_get_gp_data(enum rcutorture_type test_type,
- int *flags,
- unsigned long *gpnum,
- unsigned long *completed)
-{
- *flags = 0;
- *gpnum = 0;
- *completed = 0;
-}
-static inline void rcutorture_record_test_transition(void)
-{
-}
-static inline void rcutorture_record_progress(unsigned long vernum)
-{
-}
-static inline bool rcu_irq_enter_disabled(void)
-{
- return false;
-}
-#ifdef CONFIG_RCU_TRACE
-void do_trace_rcu_torture_read(const char *rcutorturename,
- struct rcu_head *rhp,
- unsigned long secs,
- unsigned long c_old,
- unsigned long c);
-#else
-#define do_trace_rcu_torture_read(rcutorturename, rhp, secs, c_old, c) \
- do { } while (0)
-#endif
-#endif
-
-#define UINT_CMP_GE(a, b) (UINT_MAX / 2 >= (a) - (b))
-#define UINT_CMP_LT(a, b) (UINT_MAX / 2 < (a) - (b))
#define ULONG_CMP_GE(a, b) (ULONG_MAX / 2 >= (a) - (b))
#define ULONG_CMP_LT(a, b) (ULONG_MAX / 2 < (a) - (b))
#define ulong2long(a) (*(long *)(&(a)))
/* Exported common interfaces */
#ifdef CONFIG_PREEMPT_RCU
-
-/**
- * call_rcu() - Queue an RCU callback for invocation after a grace period.
- * @head: structure to be used for queueing the RCU updates.
- * @func: actual callback function to be invoked after the grace period
- *
- * The callback function will be invoked some time after a full grace
- * period elapses, in other words after all pre-existing RCU read-side
- * critical sections have completed. However, the callback function
- * might well execute concurrently with RCU read-side critical sections
- * that started after call_rcu() was invoked. RCU read-side critical
- * sections are delimited by rcu_read_lock() and rcu_read_unlock(),
- * and may be nested.
- *
- * Note that all CPUs must agree that the grace period extended beyond
- * all pre-existing RCU read-side critical section. On systems with more
- * than one CPU, this means that when "func()" is invoked, each CPU is
- * guaranteed to have executed a full memory barrier since the end of its
- * last RCU read-side critical section whose beginning preceded the call
- * to call_rcu(). It also means that each CPU executing an RCU read-side
- * critical section that continues beyond the start of "func()" must have
- * executed a memory barrier after the call_rcu() but before the beginning
- * of that RCU read-side critical section. Note that these guarantees
- * include CPUs that are offline, idle, or executing in user mode, as
- * well as CPUs that are executing in the kernel.
- *
- * Furthermore, if CPU A invoked call_rcu() and CPU B invoked the
- * resulting RCU callback function "func()", then both CPU A and CPU B are
- * guaranteed to execute a full memory barrier during the time interval
- * between the call to call_rcu() and the invocation of "func()" -- even
- * if CPU A and CPU B are the same CPU (but again only if the system has
- * more than one CPU).
- */
-void call_rcu(struct rcu_head *head,
- rcu_callback_t func);
-
+void call_rcu(struct rcu_head *head, rcu_callback_t func);
#else /* #ifdef CONFIG_PREEMPT_RCU */
-
-/* In classic RCU, call_rcu() is just call_rcu_sched(). */
#define call_rcu call_rcu_sched
-
#endif /* #else #ifdef CONFIG_PREEMPT_RCU */
-/**
- * call_rcu_bh() - Queue an RCU for invocation after a quicker grace period.
- * @head: structure to be used for queueing the RCU updates.
- * @func: actual callback function to be invoked after the grace period
- *
- * The callback function will be invoked some time after a full grace
- * period elapses, in other words after all currently executing RCU
- * read-side critical sections have completed. call_rcu_bh() assumes
- * that the read-side critical sections end on completion of a softirq
- * handler. This means that read-side critical sections in process
- * context must not be interrupted by softirqs. This interface is to be
- * used when most of the read-side critical sections are in softirq context.
- * RCU read-side critical sections are delimited by :
- * - rcu_read_lock() and rcu_read_unlock(), if in interrupt context.
- * OR
- * - rcu_read_lock_bh() and rcu_read_unlock_bh(), if in process context.
- * These may be nested.
- *
- * See the description of call_rcu() for more detailed information on
- * memory ordering guarantees.
- */
-void call_rcu_bh(struct rcu_head *head,
- rcu_callback_t func);
-
-/**
- * call_rcu_sched() - Queue an RCU for invocation after sched grace period.
- * @head: structure to be used for queueing the RCU updates.
- * @func: actual callback function to be invoked after the grace period
- *
- * The callback function will be invoked some time after a full grace
- * period elapses, in other words after all currently executing RCU
- * read-side critical sections have completed. call_rcu_sched() assumes
- * that the read-side critical sections end on enabling of preemption
- * or on voluntary preemption.
- * RCU read-side critical sections are delimited by :
- * - rcu_read_lock_sched() and rcu_read_unlock_sched(),
- * OR
- * anything that disables preemption.
- * These may be nested.
- *
- * See the description of call_rcu() for more detailed information on
- * memory ordering guarantees.
- */
-void call_rcu_sched(struct rcu_head *head,
- rcu_callback_t func);
-
+void call_rcu_bh(struct rcu_head *head, rcu_callback_t func);
+void call_rcu_sched(struct rcu_head *head, rcu_callback_t func);
void synchronize_sched(void);
-
-/**
- * call_rcu_tasks() - Queue an RCU for invocation task-based grace period
- * @head: structure to be used for queueing the RCU updates.
- * @func: actual callback function to be invoked after the grace period
- *
- * The callback function will be invoked some time after a full grace
- * period elapses, in other words after all currently executing RCU
- * read-side critical sections have completed. call_rcu_tasks() assumes
- * that the read-side critical sections end at a voluntary context
- * switch (not a preemption!), entry into idle, or transition to usermode
- * execution. As such, there are no read-side primitives analogous to
- * rcu_read_lock() and rcu_read_unlock() because this primitive is intended
- * to determine that all tasks have passed through a safe state, not so
- * much for data-strcuture synchronization.
- *
- * See the description of call_rcu() for more detailed information on
- * memory ordering guarantees.
- */
void call_rcu_tasks(struct rcu_head *head, rcu_callback_t func);
void synchronize_rcu_tasks(void);
void rcu_barrier_tasks(void);
void rcu_report_dead(unsigned int cpu);
void rcu_cpu_starting(unsigned int cpu);
-#ifndef CONFIG_TINY_RCU
-void rcu_end_inkernel_boot(void);
-#else /* #ifndef CONFIG_TINY_RCU */
-static inline void rcu_end_inkernel_boot(void) { }
-#endif /* #ifndef CONFIG_TINY_RCU */
-
#ifdef CONFIG_RCU_STALL_COMMON
void rcu_sysrq_start(void);
void rcu_sysrq_end(void);
#else /* #ifdef CONFIG_RCU_STALL_COMMON */
-static inline void rcu_sysrq_start(void)
-{
-}
-static inline void rcu_sysrq_end(void)
-{
-}
+static inline void rcu_sysrq_start(void) { }
+static inline void rcu_sysrq_end(void) { }
#endif /* #else #ifdef CONFIG_RCU_STALL_COMMON */
#ifdef CONFIG_NO_HZ_FULL
#ifdef CONFIG_RCU_NOCB_CPU
void rcu_init_nohz(void);
#else /* #ifdef CONFIG_RCU_NOCB_CPU */
-static inline void rcu_init_nohz(void)
-{
-}
+static inline void rcu_init_nohz(void) { }
#endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */
/**
rcu_note_voluntary_context_switch(current); \
} while (0)
-#if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE) || defined(CONFIG_SMP)
-bool __rcu_is_watching(void);
-#endif /* #if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE) || defined(CONFIG_SMP) */
-
/*
* Infrastructure to implement the synchronize_() primitives in
* TREE_RCU and rcu_barrier_() primitives in TINY_RCU.
#error "Unknown RCU implementation specified to kernel configuration"
#endif
-#define RCU_SCHEDULER_INACTIVE 0
-#define RCU_SCHEDULER_INIT 1
-#define RCU_SCHEDULER_RUNNING 2
-
/*
* init_rcu_head_on_stack()/destroy_rcu_head_on_stack() are needed for dynamic
* initialization and destruction of rcu_head on the stack. rcu_head structures
void init_rcu_head_on_stack(struct rcu_head *head);
void destroy_rcu_head_on_stack(struct rcu_head *head);
#else /* !CONFIG_DEBUG_OBJECTS_RCU_HEAD */
-static inline void init_rcu_head(struct rcu_head *head)
-{
-}
-
-static inline void destroy_rcu_head(struct rcu_head *head)
-{
-}
-
-static inline void init_rcu_head_on_stack(struct rcu_head *head)
-{
-}
-
-static inline void destroy_rcu_head_on_stack(struct rcu_head *head)
-{
-}
+static inline void init_rcu_head(struct rcu_head *head) { }
+static inline void destroy_rcu_head(struct rcu_head *head) { }
+static inline void init_rcu_head_on_stack(struct rcu_head *head) { }
+static inline void destroy_rcu_head_on_stack(struct rcu_head *head) { }
#endif /* #else !CONFIG_DEBUG_OBJECTS_RCU_HEAD */
#if defined(CONFIG_HOTPLUG_CPU) && defined(CONFIG_PROVE_RCU)
bool rcu_lockdep_current_cpu_online(void);
#else /* #if defined(CONFIG_HOTPLUG_CPU) && defined(CONFIG_PROVE_RCU) */
-static inline bool rcu_lockdep_current_cpu_online(void)
-{
- return true;
-}
+static inline bool rcu_lockdep_current_cpu_online(void) { return true; }
#endif /* #else #if defined(CONFIG_HOTPLUG_CPU) && defined(CONFIG_PROVE_RCU) */
#ifdef CONFIG_DEBUG_LOCK_ALLOC
extern struct lockdep_map rcu_sched_lock_map;
extern struct lockdep_map rcu_callback_map;
int debug_lockdep_rcu_enabled(void);
-
int rcu_read_lock_held(void);
int rcu_read_lock_bh_held(void);
-
-/**
- * rcu_read_lock_sched_held() - might we be in RCU-sched read-side critical section?
- *
- * If CONFIG_DEBUG_LOCK_ALLOC is selected, returns nonzero iff in an
- * RCU-sched read-side critical section. In absence of
- * CONFIG_DEBUG_LOCK_ALLOC, this assumes we are in an RCU-sched read-side
- * critical section unless it can prove otherwise.
- */
int rcu_read_lock_sched_held(void);
#else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
"Illegal context switch in RCU read-side critical section");
}
#else /* #ifdef CONFIG_PROVE_RCU */
-static inline void rcu_preempt_sleep_check(void)
-{
-}
+static inline void rcu_preempt_sleep_check(void) { }
#endif /* #else #ifdef CONFIG_PROVE_RCU */
#define rcu_sleep_check() \
#define kfree_rcu(ptr, rcu_head) \
__kfree_rcu(&((ptr)->rcu_head), offsetof(typeof(*(ptr)), rcu_head))
-#ifdef CONFIG_TINY_RCU
-static inline int rcu_needs_cpu(u64 basemono, u64 *nextevt)
-{
- *nextevt = KTIME_MAX;
- return 0;
-}
-#endif /* #ifdef CONFIG_TINY_RCU */
-
-#if defined(CONFIG_RCU_NOCB_CPU_ALL)
-static inline bool rcu_is_nocb_cpu(int cpu) { return true; }
-#elif defined(CONFIG_RCU_NOCB_CPU)
-bool rcu_is_nocb_cpu(int cpu);
-#else
-static inline bool rcu_is_nocb_cpu(int cpu) { return false; }
-#endif
-
-
-/* Only for use by adaptive-ticks code. */
-#ifdef CONFIG_NO_HZ_FULL_SYSIDLE
-bool rcu_sys_is_idle(void);
-void rcu_sysidle_force_exit(void);
-#else /* #ifdef CONFIG_NO_HZ_FULL_SYSIDLE */
-
-static inline bool rcu_sys_is_idle(void)
-{
- return false;
-}
-
-static inline void rcu_sysidle_force_exit(void)
-{
-}
-
-#endif /* #else #ifdef CONFIG_NO_HZ_FULL_SYSIDLE */
-
-
-/*
- * Dump the ftrace buffer, but only one time per callsite per boot.
- */
-#define rcu_ftrace_dump(oops_dump_mode) \
-do { \
- static atomic_t ___rfd_beenhere = ATOMIC_INIT(0); \
- \
- if (!atomic_read(&___rfd_beenhere) && \
- !atomic_xchg(&___rfd_beenhere, 1)) \
- ftrace_dump(oops_dump_mode); \
-} while (0)
/*
* Place this after a lock-acquisition primitive to guarantee that
#ifndef __LINUX_TINY_H
#define __LINUX_TINY_H
-#include <linux/cache.h>
+#include <linux/ktime.h>
struct rcu_dynticks;
static inline int rcu_dynticks_snap(struct rcu_dynticks *rdtp)
return 0;
}
-static inline bool rcu_eqs_special_set(int cpu)
-{
- return false; /* Never flag non-existent other CPUs! */
-}
+/* Never flag non-existent other CPUs! */
+static inline bool rcu_eqs_special_set(int cpu) { return false; }
static inline unsigned long get_state_synchronize_rcu(void)
{
rcu_note_voluntary_context_switch_lite(current); \
} while (0)
-/*
- * Take advantage of the fact that there is only one CPU, which
- * allows us to ignore virtualization-based context switches.
- */
-static inline void rcu_virt_note_context_switch(int cpu)
-{
-}
-
-/*
- * Return the number of grace periods started.
- */
-static inline unsigned long rcu_batches_started(void)
-{
- return 0;
-}
-
-/*
- * Return the number of bottom-half grace periods started.
- */
-static inline unsigned long rcu_batches_started_bh(void)
-{
- return 0;
-}
-
-/*
- * Return the number of sched grace periods started.
- */
-static inline unsigned long rcu_batches_started_sched(void)
-{
- return 0;
-}
-
-/*
- * Return the number of grace periods completed.
- */
-static inline unsigned long rcu_batches_completed(void)
-{
- return 0;
-}
-
-/*
- * Return the number of bottom-half grace periods completed.
- */
-static inline unsigned long rcu_batches_completed_bh(void)
-{
- return 0;
-}
-
-/*
- * Return the number of sched grace periods completed.
- */
-static inline unsigned long rcu_batches_completed_sched(void)
+static inline int rcu_needs_cpu(u64 basemono, u64 *nextevt)
{
+ *nextevt = KTIME_MAX;
return 0;
}
/*
- * Return the number of expedited grace periods completed.
- */
-static inline unsigned long rcu_exp_batches_completed(void)
-{
- return 0;
-}
-
-/*
- * Return the number of expedited sched grace periods completed.
+ * Take advantage of the fact that there is only one CPU, which
+ * allows us to ignore virtualization-based context switches.
*/
-static inline unsigned long rcu_exp_batches_completed_sched(void)
-{
- return 0;
-}
-
-static inline void rcu_force_quiescent_state(void)
-{
-}
-
-static inline void rcu_bh_force_quiescent_state(void)
-{
-}
-
-static inline void rcu_sched_force_quiescent_state(void)
-{
-}
-
-static inline void show_rcu_gp_kthreads(void)
-{
-}
-
-static inline void rcu_cpu_stall_reset(void)
-{
-}
-
-static inline void rcu_idle_enter(void)
-{
-}
-
-static inline void rcu_idle_exit(void)
-{
-}
-
-static inline void rcu_irq_enter(void)
-{
-}
-
-static inline void rcu_irq_exit_irqson(void)
-{
-}
-
-static inline void rcu_irq_enter_irqson(void)
-{
-}
-
-static inline void rcu_irq_exit(void)
-{
-}
-
-static inline void exit_rcu(void)
-{
-}
+static inline void rcu_virt_note_context_switch(int cpu) { }
+static inline void rcu_cpu_stall_reset(void) { }
+static inline void rcu_idle_enter(void) { }
+static inline void rcu_idle_exit(void) { }
+static inline void rcu_irq_enter(void) { }
+static inline bool rcu_irq_enter_disabled(void) { return false; }
+static inline void rcu_irq_exit_irqson(void) { }
+static inline void rcu_irq_enter_irqson(void) { }
+static inline void rcu_irq_exit(void) { }
+static inline void exit_rcu(void) { }
#if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_SRCU)
extern int rcu_scheduler_active __read_mostly;
void rcu_scheduler_starting(void);
#else /* #if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_SRCU) */
-static inline void rcu_scheduler_starting(void)
-{
-}
+static inline void rcu_scheduler_starting(void) { }
#endif /* #else #if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_SRCU) */
+static inline void rcu_end_inkernel_boot(void) { }
+static inline bool rcu_is_watching(void) { return true; }
-#if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE)
-
-static inline bool rcu_is_watching(void)
-{
- return __rcu_is_watching();
-}
-
-#else /* defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE) */
-
-static inline bool rcu_is_watching(void)
-{
- return true;
-}
-
-#endif /* #else defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE) */
-
-static inline void rcu_request_urgent_qs_task(struct task_struct *t)
-{
-}
-
-static inline void rcu_all_qs(void)
-{
- barrier(); /* Avoid RCU read-side critical sections leaking across. */
-}
+/* Avoid RCU read-side critical sections leaking across. */
+static inline void rcu_all_qs(void) { barrier(); }
/* RCUtree hotplug events */
#define rcutree_prepare_cpu NULL
unsigned long get_state_synchronize_sched(void);
void cond_synchronize_sched(unsigned long oldstate);
-extern unsigned long rcutorture_testseq;
-extern unsigned long rcutorture_vernum;
-unsigned long rcu_batches_started(void);
-unsigned long rcu_batches_started_bh(void);
-unsigned long rcu_batches_started_sched(void);
-unsigned long rcu_batches_completed(void);
-unsigned long rcu_batches_completed_bh(void);
-unsigned long rcu_batches_completed_sched(void);
-unsigned long rcu_exp_batches_completed(void);
-unsigned long rcu_exp_batches_completed_sched(void);
-void show_rcu_gp_kthreads(void);
-
-void rcu_force_quiescent_state(void);
-void rcu_bh_force_quiescent_state(void);
-void rcu_sched_force_quiescent_state(void);
-
void rcu_idle_enter(void);
void rcu_idle_exit(void);
void rcu_irq_enter(void);
void rcu_irq_exit(void);
void rcu_irq_enter_irqson(void);
void rcu_irq_exit_irqson(void);
+bool rcu_irq_enter_disabled(void);
void exit_rcu(void);
void rcu_scheduler_starting(void);
extern int rcu_scheduler_active __read_mostly;
-
+void rcu_end_inkernel_boot(void);
bool rcu_is_watching(void);
-void rcu_request_urgent_qs_task(struct task_struct *t);
-
void rcu_all_qs(void);
/* RCUtree hotplug events */
return atomic_read(&r->refs);
}
+#ifdef CONFIG_REFCOUNT_FULL
extern __must_check bool refcount_add_not_zero(unsigned int i, refcount_t *r);
extern void refcount_add(unsigned int i, refcount_t *r);
extern void refcount_inc(refcount_t *r);
extern __must_check bool refcount_sub_and_test(unsigned int i, refcount_t *r);
-extern void refcount_sub(unsigned int i, refcount_t *r);
extern __must_check bool refcount_dec_and_test(refcount_t *r);
extern void refcount_dec(refcount_t *r);
+#else
+static inline __must_check bool refcount_add_not_zero(unsigned int i, refcount_t *r)
+{
+ return atomic_add_unless(&r->refs, i, 0);
+}
+
+static inline void refcount_add(unsigned int i, refcount_t *r)
+{
+ atomic_add(i, &r->refs);
+}
+
+static inline __must_check bool refcount_inc_not_zero(refcount_t *r)
+{
+ return atomic_add_unless(&r->refs, 1, 0);
+}
+
+static inline void refcount_inc(refcount_t *r)
+{
+ atomic_inc(&r->refs);
+}
+
+static inline __must_check bool refcount_sub_and_test(unsigned int i, refcount_t *r)
+{
+ return atomic_sub_and_test(i, &r->refs);
+}
+
+static inline __must_check bool refcount_dec_and_test(refcount_t *r)
+{
+ return atomic_dec_and_test(&r->refs);
+}
+
+static inline void refcount_dec(refcount_t *r)
+{
+ atomic_dec(&r->refs);
+}
+#endif /* CONFIG_REFCOUNT_FULL */
extern __must_check bool refcount_dec_if_one(refcount_t *r);
extern __must_check bool refcount_dec_not_one(refcount_t *r);
const struct regmap_config *config,
struct lock_class_key *lock_key,
const char *lock_name);
+struct regmap *__regmap_init_w1(struct device *w1_dev,
+ const struct regmap_config *config,
+ struct lock_class_key *lock_key,
+ const char *lock_name);
struct regmap *__regmap_init_mmio_clk(struct device *dev, const char *clk_id,
void __iomem *regs,
const struct regmap_config *config,
const struct regmap_config *config,
struct lock_class_key *lock_key,
const char *lock_name);
+struct regmap *__devm_regmap_init_w1(struct device *w1_dev,
+ const struct regmap_config *config,
+ struct lock_class_key *lock_key,
+ const char *lock_name);
struct regmap *__devm_regmap_init_mmio_clk(struct device *dev,
const char *clk_id,
void __iomem *regs,
__regmap_lockdep_wrapper(__regmap_init_spmi_ext, #config, \
dev, config)
+/**
+ * regmap_init_w1() - Initialise register map
+ *
+ * @w1_dev: Device that will be interacted with
+ * @config: Configuration for register map
+ *
+ * The return value will be an ERR_PTR() on error or a valid pointer to
+ * a struct regmap.
+ */
+#define regmap_init_w1(w1_dev, config) \
+ __regmap_lockdep_wrapper(__regmap_init_w1, #config, \
+ w1_dev, config)
+
/**
* regmap_init_mmio_clk() - Initialise register map with register clock
*
__regmap_lockdep_wrapper(__devm_regmap_init_spmi_ext, #config, \
dev, config)
+/**
+ * devm_regmap_init_w1() - Initialise managed register map
+ *
+ * @w1_dev: Device that will be interacted with
+ * @config: Configuration for register map
+ *
+ * The return value will be an ERR_PTR() on error or a valid pointer
+ * to a struct regmap. The regmap will be automatically freed by the
+ * device management code.
+ */
+#define devm_regmap_init_w1(w1_dev, config) \
+ __regmap_lockdep_wrapper(__devm_regmap_init_w1, #config, \
+ w1_dev, config)
/**
* devm_regmap_init_mmio_clk() - Initialise managed register map with clock
*
*
* @status_base: Base status register address.
* @mask_base: Base mask register address.
+ * @mask_writeonly: Base mask register is write only.
* @unmask_base: Base unmask register address. for chips who have
* separate mask and unmask registers
* @ack_base: Base ack address. If zero then the chip is clear on read.
unsigned int wake_base;
unsigned int type_base;
unsigned int irq_reg_stride;
+ bool mask_writeonly:1;
bool init_ack_masked:1;
bool mask_invert:1;
bool use_ack:1;
* @ramp_delay: Time to settle down after voltage change (unit: uV/us)
* @settling_time: Time to settle down after voltage change when voltage
* change is non-linear (unit: microseconds).
+ * @settling_time_up: Time to settle down after voltage increase when voltage
+ * change is non-linear (unit: microseconds).
+ * @settling_time_down : Time to settle down after voltage decrease when
+ * voltage change is non-linear (unit: microseconds).
* @active_discharge: Enable/disable active discharge. The enum
* regulator_active_discharge values are used for
* initialisation.
unsigned int ramp_delay;
unsigned int settling_time;
+ unsigned int settling_time_up;
+ unsigned int settling_time_down;
unsigned int enable_time;
unsigned int active_discharge;
struct compat_timespec;
struct pollfd;
+enum timespec_type {
+ TT_NONE = 0,
+ TT_NATIVE = 1,
+#ifdef CONFIG_COMPAT
+ TT_COMPAT = 2,
+#endif
+};
+
/*
* System call restart block.
*/
/* For nanosleep */
struct {
clockid_t clockid;
- struct timespec __user *rmtp;
+ enum timespec_type type;
+ union {
+ struct timespec __user *rmtp;
#ifdef CONFIG_COMPAT
- struct compat_timespec __user *compat_rmtp;
+ struct compat_timespec __user *compat_rmtp;
#endif
+ };
u64 expires;
} nanosleep;
/* For poll */
int line;
void *magic;
#endif
+#ifdef CONFIG_DEBUG_LOCK_ALLOC
+ struct lockdep_map dep_map;
+#endif
};
struct rt_mutex_waiter;
#ifdef CONFIG_DEBUG_RT_MUTEXES
# define __DEBUG_RT_MUTEX_INITIALIZER(mutexname) \
, .name = #mutexname, .file = __FILE__, .line = __LINE__
-# define rt_mutex_init(mutex) __rt_mutex_init(mutex, __func__)
+
+# define rt_mutex_init(mutex) \
+do { \
+ static struct lock_class_key __key; \
+ __rt_mutex_init(mutex, __func__, &__key); \
+} while (0)
+
extern void rt_mutex_debug_task_free(struct task_struct *tsk);
#else
# define __DEBUG_RT_MUTEX_INITIALIZER(mutexname)
-# define rt_mutex_init(mutex) __rt_mutex_init(mutex, NULL)
+# define rt_mutex_init(mutex) __rt_mutex_init(mutex, NULL, NULL)
# define rt_mutex_debug_task_free(t) do { } while (0)
#endif
+#ifdef CONFIG_DEBUG_LOCK_ALLOC
+#define __DEP_MAP_RT_MUTEX_INITIALIZER(mutexname) \
+ , .dep_map = { .name = #mutexname }
+#else
+#define __DEP_MAP_RT_MUTEX_INITIALIZER(mutexname)
+#endif
+
#define __RT_MUTEX_INITIALIZER(mutexname) \
{ .wait_lock = __RAW_SPIN_LOCK_UNLOCKED(mutexname.wait_lock) \
, .waiters = RB_ROOT \
, .owner = NULL \
- __DEBUG_RT_MUTEX_INITIALIZER(mutexname)}
+ __DEBUG_RT_MUTEX_INITIALIZER(mutexname) \
+ __DEP_MAP_RT_MUTEX_INITIALIZER(mutexname)}
#define DEFINE_RT_MUTEX(mutexname) \
struct rt_mutex mutexname = __RT_MUTEX_INITIALIZER(mutexname)
return lock->owner != NULL;
}
-extern void __rt_mutex_init(struct rt_mutex *lock, const char *name);
+extern void __rt_mutex_init(struct rt_mutex *lock, const char *name, struct lock_class_key *key);
extern void rt_mutex_destroy(struct rt_mutex *lock);
extern void rt_mutex_lock(struct rt_mutex *lock);
const void *buf, size_t buflen, off_t skip);
size_t sg_pcopy_to_buffer(struct scatterlist *sgl, unsigned int nents,
void *buf, size_t buflen, off_t skip);
+size_t sg_zero_buffer(struct scatterlist *sgl, unsigned int nents,
+ size_t buflen, off_t skip);
/*
* Maximum number of entries that will be allocated in one piece, if
u64 dl_runtime; /* Maximum runtime for each instance */
u64 dl_deadline; /* Relative deadline of each instance */
u64 dl_period; /* Separation of two instances (period) */
- u64 dl_bw; /* dl_runtime / dl_deadline */
+ u64 dl_bw; /* dl_runtime / dl_period */
+ u64 dl_density; /* dl_runtime / dl_deadline */
/*
* Actual scheduling parameters. Initialized with the values above,
*
* @dl_yielded tells if task gave up the CPU before consuming
* all its available runtime during the last job.
+ *
+ * @dl_non_contending tells if the task is inactive while still
+ * contributing to the active utilization. In other words, it
+ * indicates if the inactive timer has been armed and its handler
+ * has not been executed yet. This flag is useful to avoid race
+ * conditions between the inactive timer handler and the wakeup
+ * code.
*/
int dl_throttled;
int dl_boosted;
int dl_yielded;
+ int dl_non_contending;
/*
* Bandwidth enforcement timer. Each -deadline task has its
* own bandwidth to be enforced, thus we need one timer per task.
*/
struct hrtimer dl_timer;
+
+ /*
+ * Inactive timer, responsible for decreasing the active utilization
+ * at the "0-lag time". When a -deadline task blocks, it contributes
+ * to GRUB's active utilization until the "0-lag time", hence a
+ * timer is needed to decrease the active utilization at the correct
+ * time.
+ */
+ struct hrtimer inactive_timer;
};
union rcu_special {
* current.
* task_xid_nr_ns() : id seen from the ns specified;
*
- * set_task_vxid() : assigns a virtual id to a task;
- *
* see also pid_nr() etc in include/linux/pid.h
*/
pid_t __task_pid_nr_ns(struct task_struct *task, enum pid_type type, struct pid_namespace *ns);
#define tsk_used_math(p) ((p)->flags & PF_USED_MATH)
#define used_math() tsk_used_math(current)
+static inline bool is_percpu_thread(void)
+{
+#ifdef CONFIG_SMP
+ return (current->flags & PF_NO_SETAFFINITY) &&
+ (current->nr_cpus_allowed == 1);
+#else
+ return true;
+#endif
+}
+
/* Per-process atomic flags. */
#define PFA_NO_NEW_PRIVS 0 /* May not gain new privileges. */
#define PFA_SPREAD_PAGE 1 /* Spread page cache over cpuset */
extern void sched_clock_init(void);
#ifndef CONFIG_HAVE_UNSTABLE_SCHED_CLOCK
-static inline void sched_clock_init_late(void)
-{
-}
-
static inline void sched_clock_tick(void)
{
}
{
}
-static inline void sched_clock_idle_wakeup_event(u64 delta_ns)
+static inline void sched_clock_idle_wakeup_event(void)
{
}
return sched_clock();
}
#else
-extern void sched_clock_init_late(void);
extern int sched_clock_stable(void);
extern void clear_sched_clock_stable(void);
*/
extern u64 __sched_clock_offset;
-
extern void sched_clock_tick(void);
+extern void sched_clock_tick_stable(void);
extern void sched_clock_idle_sleep_event(void);
-extern void sched_clock_idle_wakeup_event(u64 delta_ns);
+extern void sched_clock_idle_wakeup_event(void);
/*
* As outlined in clock.c, provides a fast, high resolution, nanosecond
#endif
#ifdef CONFIG_NO_HZ_COMMON
-void calc_load_enter_idle(void);
-void calc_load_exit_idle(void);
+void calc_load_nohz_start(void);
+void calc_load_nohz_stop(void);
#else
-static inline void calc_load_enter_idle(void) { }
-static inline void calc_load_exit_idle(void) { }
+static inline void calc_load_nohz_start(void) { }
+static inline void calc_load_nohz_stop(void) { }
#endif /* CONFIG_NO_HZ_COMMON */
#if defined(CONFIG_NO_HZ_COMMON) && defined(CONFIG_SMP)
}
struct task_struct *task_rcu_dereference(struct task_struct **ptask);
-struct task_struct *try_get_task_struct(struct task_struct **ptask);
-
#ifdef CONFIG_ARCH_WANTS_DYNAMIC_TASK_STRUCT
extern int arch_task_struct_size __read_mostly;
#define UPF_NO_TXEN_TEST ((__force upf_t) (1 << 15))
#define UPF_MAGIC_MULTIPLIER ((__force upf_t) ASYNC_MAGIC_MULTIPLIER /* 16 */ )
+#define UPF_NO_THRE_TEST ((__force upf_t) (1 << 19))
/* Port has hardware-assisted h/w flow control */
#define UPF_AUTO_CTS ((__force upf_t) (1 << 20))
#define UPF_AUTO_RTS ((__force upf_t) (1 << 21))
#include <linux/bug.h>
#include <linux/signal_types.h>
+#include <linux/string.h>
struct task_struct;
/* for sysctl */
extern int print_fatal_signals;
-#ifndef HAVE_ARCH_COPY_SIGINFO
-
-#include <linux/string.h>
-
static inline void copy_siginfo(struct siginfo *to, struct siginfo *from)
{
if (from->si_code < 0)
memcpy(to, from, __ARCH_SI_PREAMBLE_SIZE + sizeof(from->_sifields._sigchld));
}
-#endif
+int copy_siginfo_to_user(struct siginfo __user *to, const struct siginfo *from);
/*
* Define some primitives to manipulate sigset_t.
* @offset: the offset within the fragment (starting at the
* fragment's own offset)
* @size: the number of bytes to map
- * @dir: the direction of the mapping (%PCI_DMA_*)
+ * @dir: the direction of the mapping (``PCI_DMA_*``)
*
* Maps the page associated with @frag to @device.
*/
int red_left_pad; /* Left redzone padding size */
#ifdef CONFIG_SYSFS
struct kobject kobj; /* For sysfs */
+ struct work_struct kobj_remove_work;
#endif
#ifdef CONFIG_MEMCG
struct memcg_cache_params memcg_params;
#ifndef __SPI_SH_MSIOF_H__
#define __SPI_SH_MSIOF_H__
+enum {
+ MSIOF_SPI_MASTER,
+ MSIOF_SPI_SLAVE,
+};
+
struct sh_msiof_spi_info {
int tx_fifo_override;
int rx_fifo_override;
u16 num_chipselect;
+ int mode;
unsigned int dma_tx_id;
unsigned int dma_rx_id;
u32 dtdl;
struct dma_chan;
struct property_entry;
-struct spi_master;
+struct spi_controller;
struct spi_transfer;
struct spi_flash_read_message;
/*
- * INTERFACES between SPI master-side drivers and SPI infrastructure.
- * (There's no SPI slave support for Linux yet...)
+ * INTERFACES between SPI master-side drivers and SPI slave protocol handlers,
+ * and SPI infrastructure.
*/
extern struct bus_type spi_bus_type;
void spi_statistics_add_transfer_stats(struct spi_statistics *stats,
struct spi_transfer *xfer,
- struct spi_master *master);
+ struct spi_controller *ctlr);
#define SPI_STATISTICS_ADD_TO_FIELD(stats, field, count) \
do { \
SPI_STATISTICS_ADD_TO_FIELD(stats, field, 1)
/**
- * struct spi_device - Master side proxy for an SPI slave device
+ * struct spi_device - Controller side proxy for an SPI slave device
* @dev: Driver model representation of the device.
- * @master: SPI controller used with the device.
+ * @controller: SPI controller used with the device.
+ * @master: Copy of controller, for backwards compatibility.
* @max_speed_hz: Maximum clock rate to be used with this chip
* (on this board); may be changed by the device's driver.
* The spi_transfer.speed_hz can override this for each transfer.
- * @chip_select: Chipselect, distinguishing chips handled by @master.
+ * @chip_select: Chipselect, distinguishing chips handled by @controller.
* @mode: The spi mode defines how data is clocked out and in.
* This may be changed by the device's driver.
* The "active low" default for chipselect mode can be overridden
*/
struct spi_device {
struct device dev;
- struct spi_master *master;
+ struct spi_controller *controller;
+ struct spi_controller *master; /* compatibility layer */
u32 max_speed_hz;
u8 chip_select;
u8 bits_per_word;
put_device(&spi->dev);
}
-/* ctldata is for the bus_master driver's runtime state */
+/* ctldata is for the bus_controller driver's runtime state */
static inline void *spi_get_ctldata(struct spi_device *spi)
{
return spi->controller_state;
spi_unregister_driver)
/**
- * struct spi_master - interface to SPI master controller
+ * struct spi_controller - interface to SPI master or slave controller
* @dev: device interface to this driver
- * @list: link with the global spi_master list
+ * @list: link with the global spi_controller list
* @bus_num: board-specific (and often SOC-specific) identifier for a
* given SPI controller.
* @num_chipselect: chipselects are used to distinguish individual
* @min_speed_hz: Lowest supported transfer speed
* @max_speed_hz: Highest supported transfer speed
* @flags: other constraints relevant to this driver
+ * @slave: indicates that this is an SPI slave controller
* @max_transfer_size: function that returns the max transfer size for
* a &spi_device; may be %NULL, so the default %SIZE_MAX will be used.
* @max_message_size: function that returns the max message size for
* the device whose settings are being modified.
* @transfer: adds a message to the controller's transfer queue.
* @cleanup: frees controller-specific state
- * @can_dma: determine whether this master supports DMA
- * @queued: whether this master is providing an internal message queue
+ * @can_dma: determine whether this controller supports DMA
+ * @queued: whether this controller is providing an internal message queue
* @kworker: thread struct for message pump
* @kworker_task: pointer to task for message pump kworker thread
* @pump_messages: work struct for scheduling work to the message pump
* @handle_err: the subsystem calls the driver to handle an error that occurs
* in the generic implementation of transfer_one_message().
* @unprepare_message: undo any work done by prepare_message().
+ * @slave_abort: abort the ongoing transfer request on an SPI slave controller
* @spi_flash_read: to support spi-controller hardwares that provide
* accelerated interface to read from flash devices.
* @spi_flash_can_dma: analogous to can_dma() interface, but for
* @cs_gpios: Array of GPIOs to use as chip select lines; one per CS
* number. Any individual value may be -ENOENT for CS lines that
* are not GPIOs (driven by the SPI controller itself).
- * @statistics: statistics for the spi_master
+ * @statistics: statistics for the spi_controller
* @dma_tx: DMA transmit channel
* @dma_rx: DMA receive channel
* @dummy_rx: dummy receive buffer for full-duplex devices
* what Linux expects, this optional hook can be used to translate
* between the two.
*
- * Each SPI master controller can communicate with one or more @spi_device
+ * Each SPI controller can communicate with one or more @spi_device
* children. These make a small bus, sharing MOSI, MISO and SCK signals
* but not chip select signals. Each device may be configured to use a
* different clock rate, since those shared signals are ignored unless
* an SPI slave device. For each such message it queues, it calls the
* message's completion function when the transaction completes.
*/
-struct spi_master {
+struct spi_controller {
struct device dev;
struct list_head list;
/* other constraints relevant to this driver */
u16 flags;
-#define SPI_MASTER_HALF_DUPLEX BIT(0) /* can't do full duplex */
-#define SPI_MASTER_NO_RX BIT(1) /* can't do buffer read */
-#define SPI_MASTER_NO_TX BIT(2) /* can't do buffer write */
-#define SPI_MASTER_MUST_RX BIT(3) /* requires rx */
-#define SPI_MASTER_MUST_TX BIT(4) /* requires tx */
-#define SPI_MASTER_GPIO_SS BIT(5) /* GPIO CS must select slave */
+#define SPI_CONTROLLER_HALF_DUPLEX BIT(0) /* can't do full duplex */
+#define SPI_CONTROLLER_NO_RX BIT(1) /* can't do buffer read */
+#define SPI_CONTROLLER_NO_TX BIT(2) /* can't do buffer write */
+#define SPI_CONTROLLER_MUST_RX BIT(3) /* requires rx */
+#define SPI_CONTROLLER_MUST_TX BIT(4) /* requires tx */
+
+#define SPI_MASTER_GPIO_SS BIT(5) /* GPIO CS must select slave */
+
+ /* flag indicating this is an SPI slave controller */
+ bool slave;
/*
* on some hardware transfer / message size may be constrained
* any other request management
* + To a given spi_device, message queueing is pure fifo
*
- * + The master's main job is to process its message queue,
- * selecting a chip then transferring data
+ * + The controller's main job is to process its message queue,
+ * selecting a chip (for masters), then transferring data
* + If there are multiple spi_device children, the i/o queue
* arbitration algorithm is unspecified (round robin, fifo,
* priority, reservations, preemption, etc)
int (*transfer)(struct spi_device *spi,
struct spi_message *mesg);
- /* called on release() to free memory provided by spi_master */
+ /* called on release() to free memory provided by spi_controller */
void (*cleanup)(struct spi_device *spi);
/*
* not modify or store xfer and dma_tx and dma_rx must be set
* while the device is prepared.
*/
- bool (*can_dma)(struct spi_master *master,
+ bool (*can_dma)(struct spi_controller *ctlr,
struct spi_device *spi,
struct spi_transfer *xfer);
/*
* These hooks are for drivers that want to use the generic
- * master transfer queueing mechanism. If these are used, the
+ * controller transfer queueing mechanism. If these are used, the
* transfer() function above must NOT be specified by the driver.
* Over time we expect SPI drivers to be phased over to this API.
*/
struct completion xfer_completion;
size_t max_dma_len;
- int (*prepare_transfer_hardware)(struct spi_master *master);
- int (*transfer_one_message)(struct spi_master *master,
+ int (*prepare_transfer_hardware)(struct spi_controller *ctlr);
+ int (*transfer_one_message)(struct spi_controller *ctlr,
struct spi_message *mesg);
- int (*unprepare_transfer_hardware)(struct spi_master *master);
- int (*prepare_message)(struct spi_master *master,
+ int (*unprepare_transfer_hardware)(struct spi_controller *ctlr);
+ int (*prepare_message)(struct spi_controller *ctlr,
struct spi_message *message);
- int (*unprepare_message)(struct spi_master *master,
+ int (*unprepare_message)(struct spi_controller *ctlr,
struct spi_message *message);
+ int (*slave_abort)(struct spi_controller *ctlr);
int (*spi_flash_read)(struct spi_device *spi,
struct spi_flash_read_message *msg);
bool (*spi_flash_can_dma)(struct spi_device *spi,
* of transfer_one_message() provied by the core.
*/
void (*set_cs)(struct spi_device *spi, bool enable);
- int (*transfer_one)(struct spi_master *master, struct spi_device *spi,
+ int (*transfer_one)(struct spi_controller *ctlr, struct spi_device *spi,
struct spi_transfer *transfer);
- void (*handle_err)(struct spi_master *master,
+ void (*handle_err)(struct spi_controller *ctlr,
struct spi_message *message);
/* gpio chip select */
void *dummy_rx;
void *dummy_tx;
- int (*fw_translate_cs)(struct spi_master *master, unsigned cs);
+ int (*fw_translate_cs)(struct spi_controller *ctlr, unsigned cs);
};
-static inline void *spi_master_get_devdata(struct spi_master *master)
+static inline void *spi_controller_get_devdata(struct spi_controller *ctlr)
{
- return dev_get_drvdata(&master->dev);
+ return dev_get_drvdata(&ctlr->dev);
}
-static inline void spi_master_set_devdata(struct spi_master *master, void *data)
+static inline void spi_controller_set_devdata(struct spi_controller *ctlr,
+ void *data)
{
- dev_set_drvdata(&master->dev, data);
+ dev_set_drvdata(&ctlr->dev, data);
}
-static inline struct spi_master *spi_master_get(struct spi_master *master)
+static inline struct spi_controller *spi_controller_get(struct spi_controller *ctlr)
{
- if (!master || !get_device(&master->dev))
+ if (!ctlr || !get_device(&ctlr->dev))
return NULL;
- return master;
+ return ctlr;
+}
+
+static inline void spi_controller_put(struct spi_controller *ctlr)
+{
+ if (ctlr)
+ put_device(&ctlr->dev);
}
-static inline void spi_master_put(struct spi_master *master)
+static inline bool spi_controller_is_slave(struct spi_controller *ctlr)
{
- if (master)
- put_device(&master->dev);
+ return IS_ENABLED(CONFIG_SPI_SLAVE) && ctlr->slave;
}
/* PM calls that need to be issued by the driver */
-extern int spi_master_suspend(struct spi_master *master);
-extern int spi_master_resume(struct spi_master *master);
+extern int spi_controller_suspend(struct spi_controller *ctlr);
+extern int spi_controller_resume(struct spi_controller *ctlr);
/* Calls the driver make to interact with the message queue */
-extern struct spi_message *spi_get_next_queued_message(struct spi_master *master);
-extern void spi_finalize_current_message(struct spi_master *master);
-extern void spi_finalize_current_transfer(struct spi_master *master);
+extern struct spi_message *spi_get_next_queued_message(struct spi_controller *ctlr);
+extern void spi_finalize_current_message(struct spi_controller *ctlr);
+extern void spi_finalize_current_transfer(struct spi_controller *ctlr);
-/* the spi driver core manages memory for the spi_master classdev */
-extern struct spi_master *
-spi_alloc_master(struct device *host, unsigned size);
+/* the spi driver core manages memory for the spi_controller classdev */
+extern struct spi_controller *__spi_alloc_controller(struct device *host,
+ unsigned int size, bool slave);
-extern int spi_register_master(struct spi_master *master);
-extern int devm_spi_register_master(struct device *dev,
- struct spi_master *master);
-extern void spi_unregister_master(struct spi_master *master);
+static inline struct spi_controller *spi_alloc_master(struct device *host,
+ unsigned int size)
+{
+ return __spi_alloc_controller(host, size, false);
+}
-extern struct spi_master *spi_busnum_to_master(u16 busnum);
+static inline struct spi_controller *spi_alloc_slave(struct device *host,
+ unsigned int size)
+{
+ if (!IS_ENABLED(CONFIG_SPI_SLAVE))
+ return NULL;
+
+ return __spi_alloc_controller(host, size, true);
+}
+
+extern int spi_register_controller(struct spi_controller *ctlr);
+extern int devm_spi_register_controller(struct device *dev,
+ struct spi_controller *ctlr);
+extern void spi_unregister_controller(struct spi_controller *ctlr);
+
+extern struct spi_controller *spi_busnum_to_master(u16 busnum);
/*
* SPI resource management while processing a SPI message
*/
-typedef void (*spi_res_release_t)(struct spi_master *master,
+typedef void (*spi_res_release_t)(struct spi_controller *ctlr,
struct spi_message *msg,
void *res);
extern void spi_res_add(struct spi_message *message, void *res);
extern void spi_res_free(void *res);
-extern void spi_res_release(struct spi_master *master,
+extern void spi_res_release(struct spi_controller *ctlr,
struct spi_message *message);
/*---------------------------------------------------------------------------*/
/* for optional use by whatever driver currently owns the
* spi_message ... between calls to spi_async and then later
- * complete(), that's the spi_master controller driver.
+ * complete(), that's the spi_controller controller driver.
*/
struct list_head queue;
void *state;
extern int spi_async(struct spi_device *spi, struct spi_message *message);
extern int spi_async_locked(struct spi_device *spi,
struct spi_message *message);
+extern int spi_slave_abort(struct spi_device *spi);
static inline size_t
spi_max_message_size(struct spi_device *spi)
{
- struct spi_master *master = spi->master;
- if (!master->max_message_size)
+ struct spi_controller *ctlr = spi->controller;
+
+ if (!ctlr->max_message_size)
return SIZE_MAX;
- return master->max_message_size(spi);
+ return ctlr->max_message_size(spi);
}
static inline size_t
spi_max_transfer_size(struct spi_device *spi)
{
- struct spi_master *master = spi->master;
+ struct spi_controller *ctlr = spi->controller;
size_t tr_max = SIZE_MAX;
size_t msg_max = spi_max_message_size(spi);
- if (master->max_transfer_size)
- tr_max = master->max_transfer_size(spi);
+ if (ctlr->max_transfer_size)
+ tr_max = ctlr->max_transfer_size(spi);
/* transfer size limit must not be greater than messsage size limit */
return min(tr_max, msg_max);
/* SPI transfer replacement methods which make use of spi_res */
struct spi_replaced_transfers;
-typedef void (*spi_replaced_release_t)(struct spi_master *master,
+typedef void (*spi_replaced_release_t)(struct spi_controller *ctlr,
struct spi_message *msg,
struct spi_replaced_transfers *res);
/**
/* SPI transfer transformation methods */
-extern int spi_split_transfers_maxsize(struct spi_master *master,
+extern int spi_split_transfers_maxsize(struct spi_controller *ctlr,
struct spi_message *msg,
size_t maxsize,
gfp_t gfp);
extern int spi_sync(struct spi_device *spi, struct spi_message *message);
extern int spi_sync_locked(struct spi_device *spi, struct spi_message *message);
-extern int spi_bus_lock(struct spi_master *master);
-extern int spi_bus_unlock(struct spi_master *master);
+extern int spi_bus_lock(struct spi_controller *ctlr);
+extern int spi_bus_unlock(struct spi_controller *ctlr);
/**
* spi_sync_transfer - synchronous SPI data transfer
/* SPI core interface for flash read support */
static inline bool spi_flash_read_supported(struct spi_device *spi)
{
- return spi->master->spi_flash_read &&
- (!spi->master->flash_read_supported ||
- spi->master->flash_read_supported(spi));
+ return spi->controller->spi_flash_read &&
+ (!spi->controller->flash_read_supported ||
+ spi->controller->flash_read_supported(spi));
}
int spi_flash_read(struct spi_device *spi,
* @irq: Initializes spi_device.irq; depends on how the board is wired.
* @max_speed_hz: Initializes spi_device.max_speed_hz; based on limits
* from the chip datasheet and board-specific signal quality issues.
- * @bus_num: Identifies which spi_master parents the spi_device; unused
+ * @bus_num: Identifies which spi_controller parents the spi_device; unused
* by spi_new_device(), and otherwise depends on board wiring.
* @chip_select: Initializes spi_device.chip_select; depends on how
* the board is wired.
/* bus_num is board specific and matches the bus_num of some
- * spi_master that will probably be registered later.
+ * spi_controller that will probably be registered later.
*
* chip_select reflects how this chip is wired to that master;
* it's less than num_chipselect.
/* If you're hotplugging an adapter with devices (parport, usb, etc)
* use spi_new_device() to describe each device. You can also call
* spi_unregister_device() to start making that device vanish, but
- * normally that would be handled by spi_unregister_master().
+ * normally that would be handled by spi_unregister_controller().
*
* You can also use spi_alloc_device() and spi_add_device() to use a two
* stage registration sequence for each spi_device. This gives the caller
* be defined using the board info.
*/
extern struct spi_device *
-spi_alloc_device(struct spi_master *master);
+spi_alloc_device(struct spi_controller *ctlr);
extern int
spi_add_device(struct spi_device *spi);
extern struct spi_device *
-spi_new_device(struct spi_master *, struct spi_board_info *);
+spi_new_device(struct spi_controller *, struct spi_board_info *);
extern void spi_unregister_device(struct spi_device *spi);
spi_get_device_id(const struct spi_device *sdev);
static inline bool
-spi_transfer_is_last(struct spi_master *master, struct spi_transfer *xfer)
+spi_transfer_is_last(struct spi_controller *ctlr, struct spi_transfer *xfer)
{
- return list_is_last(&xfer->transfer_list, &master->cur_msg->transfers);
+ return list_is_last(&xfer->transfer_list, &ctlr->cur_msg->transfers);
}
+
+/* Compatibility layer */
+#define spi_master spi_controller
+
+#define SPI_MASTER_HALF_DUPLEX SPI_CONTROLLER_HALF_DUPLEX
+#define SPI_MASTER_NO_RX SPI_CONTROLLER_NO_RX
+#define SPI_MASTER_NO_TX SPI_CONTROLLER_NO_TX
+#define SPI_MASTER_MUST_RX SPI_CONTROLLER_MUST_RX
+#define SPI_MASTER_MUST_TX SPI_CONTROLLER_MUST_TX
+
+#define spi_master_get_devdata(_ctlr) spi_controller_get_devdata(_ctlr)
+#define spi_master_set_devdata(_ctlr, _data) \
+ spi_controller_set_devdata(_ctlr, _data)
+#define spi_master_get(_ctlr) spi_controller_get(_ctlr)
+#define spi_master_put(_ctlr) spi_controller_put(_ctlr)
+#define spi_master_suspend(_ctlr) spi_controller_suspend(_ctlr)
+#define spi_master_resume(_ctlr) spi_controller_resume(_ctlr)
+
+#define spi_register_master(_ctlr) spi_register_controller(_ctlr)
+#define devm_spi_register_master(_dev, _ctlr) \
+ devm_spi_register_controller(_dev, _ctlr)
+#define spi_unregister_master(_ctlr) spi_unregister_controller(_ctlr)
+
#endif /* __LINUX_SPI_H */
raw_spin_trylock_irqsave(spinlock_check(lock), flags); \
})
+/**
+ * spin_unlock_wait - Interpose between successive critical sections
+ * @lock: the spinlock whose critical sections are to be interposed.
+ *
+ * Semantically this is equivalent to a spin_lock() immediately
+ * followed by a spin_unlock(). However, most architectures have
+ * more efficient implementations in which the spin_unlock_wait()
+ * cannot block concurrent lock acquisition, and in some cases
+ * where spin_unlock_wait() does not write to the lock variable.
+ * Nevertheless, spin_unlock_wait() can have high overhead, so if
+ * you feel the need to use it, please check to see if there is
+ * a better way to get your job done.
+ *
+ * The ordering guarantees provided by spin_unlock_wait() are:
+ *
+ * 1. All accesses preceding the spin_unlock_wait() happen before
+ * any accesses in later critical sections for this same lock.
+ * 2. All accesses following the spin_unlock_wait() happen after
+ * any accesses in earlier critical sections for this same lock.
+ */
static __always_inline void spin_unlock_wait(spinlock_t *lock)
{
raw_spin_unlock_wait(&lock->rlock);
struct gen_pool;
#ifdef CONFIG_SRAM_EXEC
-int sram_exec_copy(struct gen_pool *pool, void *dst, void *src, size_t size);
+void *sram_exec_copy(struct gen_pool *pool, void *dst, void *src, size_t size);
#else
-static inline int sram_exec_copy(struct gen_pool *pool, void *dst, void *src,
- size_t size)
+static inline void *sram_exec_copy(struct gen_pool *pool, void *dst, void *src,
+ size_t size)
{
- return -ENODEV;
+ return NULL;
}
#endif /* CONFIG_SRAM_EXEC */
#endif /* __LINUX_SRAM_H__ */
#include <linux/srcutiny.h>
#elif defined(CONFIG_TREE_SRCU)
#include <linux/srcutree.h>
-#elif defined(CONFIG_CLASSIC_SRCU)
-#include <linux/srcuclassic.h>
-#else
+#elif defined(CONFIG_SRCU)
#error "Unknown SRCU implementation specified to kernel configuration"
+#else
+/* Dummy definition for things like notifiers. Actual use gets link error. */
+struct srcu_struct { };
#endif
-/**
- * call_srcu() - Queue a callback for invocation after an SRCU grace period
- * @sp: srcu_struct in queue the callback
- * @head: structure to be used for queueing the SRCU callback.
- * @func: function to be invoked after the SRCU grace period
- *
- * The callback function will be invoked some time after a full SRCU
- * grace period elapses, in other words after all pre-existing SRCU
- * read-side critical sections have completed. However, the callback
- * function might well execute concurrently with other SRCU read-side
- * critical sections that started after call_srcu() was invoked. SRCU
- * read-side critical sections are delimited by srcu_read_lock() and
- * srcu_read_unlock(), and may be nested.
- *
- * The callback will be invoked from process context, but must nevertheless
- * be fast and must not block.
- */
void call_srcu(struct srcu_struct *sp, struct rcu_head *head,
void (*func)(struct rcu_head *head));
-
void cleanup_srcu_struct(struct srcu_struct *sp);
int __srcu_read_lock(struct srcu_struct *sp) __acquires(sp);
void __srcu_read_unlock(struct srcu_struct *sp, int idx) __releases(sp);
+++ /dev/null
-/*
- * Sleepable Read-Copy Update mechanism for mutual exclusion,
- * classic v4.11 variant.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, you can access it online at
- * http://www.gnu.org/licenses/gpl-2.0.html.
- *
- * Copyright (C) IBM Corporation, 2017
- *
- * Author: Paul McKenney <paulmck@us.ibm.com>
- */
-
-#ifndef _LINUX_SRCU_CLASSIC_H
-#define _LINUX_SRCU_CLASSIC_H
-
-struct srcu_array {
- unsigned long lock_count[2];
- unsigned long unlock_count[2];
-};
-
-struct rcu_batch {
- struct rcu_head *head, **tail;
-};
-
-#define RCU_BATCH_INIT(name) { NULL, &(name.head) }
-
-struct srcu_struct {
- unsigned long completed;
- struct srcu_array __percpu *per_cpu_ref;
- spinlock_t queue_lock; /* protect ->batch_queue, ->running */
- bool running;
- /* callbacks just queued */
- struct rcu_batch batch_queue;
- /* callbacks try to do the first check_zero */
- struct rcu_batch batch_check0;
- /* callbacks done with the first check_zero and the flip */
- struct rcu_batch batch_check1;
- struct rcu_batch batch_done;
- struct delayed_work work;
-#ifdef CONFIG_DEBUG_LOCK_ALLOC
- struct lockdep_map dep_map;
-#endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
-};
-
-void process_srcu(struct work_struct *work);
-
-#define __SRCU_STRUCT_INIT(name) \
- { \
- .completed = -300, \
- .per_cpu_ref = &name##_srcu_array, \
- .queue_lock = __SPIN_LOCK_UNLOCKED(name.queue_lock), \
- .running = false, \
- .batch_queue = RCU_BATCH_INIT(name.batch_queue), \
- .batch_check0 = RCU_BATCH_INIT(name.batch_check0), \
- .batch_check1 = RCU_BATCH_INIT(name.batch_check1), \
- .batch_done = RCU_BATCH_INIT(name.batch_done), \
- .work = __DELAYED_WORK_INITIALIZER(name.work, process_srcu, 0),\
- __SRCU_DEP_MAP_INIT(name) \
- }
-
-/*
- * Define and initialize a srcu struct at build time.
- * Do -not- call init_srcu_struct() nor cleanup_srcu_struct() on it.
- *
- * Note that although DEFINE_STATIC_SRCU() hides the name from other
- * files, the per-CPU variable rules nevertheless require that the
- * chosen name be globally unique. These rules also prohibit use of
- * DEFINE_STATIC_SRCU() within a function. If these rules are too
- * restrictive, declare the srcu_struct manually. For example, in
- * each file:
- *
- * static struct srcu_struct my_srcu;
- *
- * Then, before the first use of each my_srcu, manually initialize it:
- *
- * init_srcu_struct(&my_srcu);
- *
- * See include/linux/percpu-defs.h for the rules on per-CPU variables.
- */
-#define __DEFINE_SRCU(name, is_static) \
- static DEFINE_PER_CPU(struct srcu_array, name##_srcu_array);\
- is_static struct srcu_struct name = __SRCU_STRUCT_INIT(name)
-#define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static */)
-#define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, static)
-
-void synchronize_srcu_expedited(struct srcu_struct *sp);
-void srcu_barrier(struct srcu_struct *sp);
-unsigned long srcu_batches_completed(struct srcu_struct *sp);
-
-static inline void srcutorture_get_gp_data(enum rcutorture_type test_type,
- struct srcu_struct *sp, int *flags,
- unsigned long *gpnum,
- unsigned long *completed)
-{
- if (test_type != SRCU_FLAVOR)
- return;
- *flags = 0;
- *completed = sp->completed;
- *gpnum = *completed;
- if (sp->batch_queue.head || sp->batch_check0.head || sp->batch_check0.head)
- (*gpnum)++;
-}
-
-#endif
#include <linux/swait.h>
struct srcu_struct {
- int srcu_lock_nesting[2]; /* srcu_read_lock() nesting depth. */
+ short srcu_lock_nesting[2]; /* srcu_read_lock() nesting depth. */
+ short srcu_idx; /* Current reader array element. */
+ u8 srcu_gp_running; /* GP workqueue running? */
+ u8 srcu_gp_waiting; /* GP waiting for readers? */
struct swait_queue_head srcu_wq;
/* Last srcu_read_unlock() wakes GP. */
- unsigned long srcu_gp_seq; /* GP seq # for callback tagging. */
- struct rcu_segcblist srcu_cblist;
- /* Pending SRCU callbacks. */
- int srcu_idx; /* Current reader array element. */
- bool srcu_gp_running; /* GP workqueue running? */
- bool srcu_gp_waiting; /* GP waiting for readers? */
+ struct rcu_head *srcu_cb_head; /* Pending callbacks: Head. */
+ struct rcu_head **srcu_cb_tail; /* Pending callbacks: Tail. */
struct work_struct srcu_work; /* For driving grace periods. */
#ifdef CONFIG_DEBUG_LOCK_ALLOC
struct lockdep_map dep_map;
#define __SRCU_STRUCT_INIT(name) \
{ \
.srcu_wq = __SWAIT_QUEUE_HEAD_INITIALIZER(name.srcu_wq), \
- .srcu_cblist = RCU_SEGCBLIST_INITIALIZER(name.srcu_cblist), \
+ .srcu_cb_tail = &name.srcu_cb_head, \
.srcu_work = __WORK_INITIALIZER(name.srcu_work, srcu_drive_gp), \
__SRCU_DEP_MAP_INIT(name) \
}
void synchronize_srcu(struct srcu_struct *sp);
-static inline void synchronize_srcu_expedited(struct srcu_struct *sp)
+/*
+ * Counts the new reader in the appropriate per-CPU element of the
+ * srcu_struct. Can be invoked from irq/bh handlers, but the matching
+ * __srcu_read_unlock() must be in the same handler instance. Returns an
+ * index that must be passed to the matching srcu_read_unlock().
+ */
+static inline int __srcu_read_lock(struct srcu_struct *sp)
{
- synchronize_srcu(sp);
-}
+ int idx;
-static inline void srcu_barrier(struct srcu_struct *sp)
-{
- synchronize_srcu(sp);
+ idx = READ_ONCE(sp->srcu_idx);
+ WRITE_ONCE(sp->srcu_lock_nesting[idx], sp->srcu_lock_nesting[idx] + 1);
+ return idx;
}
-static inline unsigned long srcu_batches_completed(struct srcu_struct *sp)
+static inline void synchronize_srcu_expedited(struct srcu_struct *sp)
{
- return 0;
+ synchronize_srcu(sp);
}
-static inline void srcutorture_get_gp_data(enum rcutorture_type test_type,
- struct srcu_struct *sp, int *flags,
- unsigned long *gpnum,
- unsigned long *completed)
+static inline void srcu_barrier(struct srcu_struct *sp)
{
- if (test_type != SRCU_FLAVOR)
- return;
- *flags = 0;
- *completed = sp->srcu_gp_seq;
- *gpnum = *completed;
+ synchronize_srcu(sp);
}
#endif
unsigned long srcu_unlock_count[2]; /* Unlocks per CPU. */
/* Update-side state. */
- spinlock_t lock ____cacheline_internodealigned_in_smp;
+ raw_spinlock_t __private lock ____cacheline_internodealigned_in_smp;
struct rcu_segcblist srcu_cblist; /* List of callbacks.*/
unsigned long srcu_gp_seq_needed; /* Furthest future GP needed. */
unsigned long srcu_gp_seq_needed_exp; /* Furthest future exp GP. */
* Node in SRCU combining tree, similar in function to rcu_data.
*/
struct srcu_node {
- spinlock_t lock;
+ raw_spinlock_t __private lock;
unsigned long srcu_have_cbs[4]; /* GP seq for children */
/* having CBs, but only */
/* is > ->srcu_gq_seq. */
struct srcu_node *level[RCU_NUM_LVLS + 1];
/* First node at each level. */
struct mutex srcu_cb_mutex; /* Serialize CB preparation. */
- spinlock_t gp_lock; /* protect ->srcu_cblist */
+ raw_spinlock_t __private lock; /* Protect counters */
struct mutex srcu_gp_mutex; /* Serialize GP work. */
unsigned int srcu_idx; /* Current rdr array element. */
unsigned long srcu_gp_seq; /* Grace-period seq #. */
#define __SRCU_STRUCT_INIT(name) \
{ \
.sda = &name##_srcu_data, \
- .gp_lock = __SPIN_LOCK_UNLOCKED(name.gp_lock), \
+ .lock = __RAW_SPIN_LOCK_UNLOCKED(name.lock), \
.srcu_gp_seq_needed = 0 - 1, \
__SRCU_DEP_MAP_INIT(name) \
}
void synchronize_srcu_expedited(struct srcu_struct *sp);
void srcu_barrier(struct srcu_struct *sp);
-unsigned long srcu_batches_completed(struct srcu_struct *sp);
-
-void srcutorture_get_gp_data(enum rcutorture_type test_type,
- struct srcu_struct *sp, int *flags,
- unsigned long *gpnum, unsigned long *completed);
#endif
* @fn() runs.
*
* This can be thought of as a very heavy write lock, equivalent to
- * grabbing every spinlock in the kernel. */
+ * grabbing every spinlock in the kernel.
+ *
+ * Protects against CPU hotplug.
+ */
int stop_machine(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus);
+/**
+ * stop_machine_cpuslocked: freeze the machine on all CPUs and run this function
+ * @fn: the function to run
+ * @data: the data ptr for the @fn()
+ * @cpus: the cpus to run the @fn() on (NULL = any online cpu)
+ *
+ * Same as above. Must be called from with in a cpus_read_lock() protected
+ * region. Avoids nested calls to cpus_read_lock().
+ */
+int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus);
+
int stop_machine_from_inactive_cpu(cpu_stop_fn_t fn, void *data,
const struct cpumask *cpus);
#else /* CONFIG_SMP || CONFIG_HOTPLUG_CPU */
-static inline int stop_machine(cpu_stop_fn_t fn, void *data,
- const struct cpumask *cpus)
+static inline int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
+ const struct cpumask *cpus)
{
unsigned long flags;
int ret;
return ret;
}
+static inline int stop_machine(cpu_stop_fn_t fn, void *data,
+ const struct cpumask *cpus)
+{
+ return stop_machine_cpuslocked(fn, data, cpus);
+}
+
static inline int stop_machine_from_inactive_cpu(cpu_stop_fn_t fn, void *data,
const struct cpumask *cpus)
{
#include <linux/ktime.h>
#include <linux/sunrpc/types.h>
#include <linux/spinlock.h>
-#include <linux/wait.h>
+#include <linux/wait_bit.h>
#include <linux/workqueue.h>
#include <linux/sunrpc/xdr.h>
int superhyway_add_devices(struct superhyway_bus *bus, struct superhyway_device **devices, int nr_devices);
/* drivers/sh/superhyway/superhyway-sysfs.c */
-extern struct device_attribute superhyway_dev_attrs[];
+extern const struct attribute_group *superhyway_dev_groups[];
#endif /* __LINUX_SUPERHYWAY_H */
struct platform_freeze_ops {
int (*begin)(void);
int (*prepare)(void);
+ void (*wake)(void);
+ void (*sync)(void);
void (*restore)(void);
void (*end)(void);
};
extern bool pm_wakeup_pending(void);
extern void pm_system_wakeup(void);
-extern void pm_wakeup_clear(void);
+extern void pm_system_cancel_wakeup(void);
+extern void pm_wakeup_clear(bool reset);
extern void pm_system_irq_wakeup(unsigned int irq_number);
extern bool pm_get_wakeup_count(unsigned int *count, bool block);
extern bool pm_save_wakeup_count(unsigned int count);
static inline bool pm_wakeup_pending(void) { return false; }
static inline void pm_system_wakeup(void) {}
-static inline void pm_wakeup_clear(void) {}
+static inline void pm_wakeup_clear(bool reset) {}
static inline void pm_system_irq_wakeup(unsigned int irq_number) {}
static inline void lock_system_sleep(void) {}
}
static inline struct kernfs_node *sysfs_get_dirent(struct kernfs_node *parent,
- const unsigned char *name)
+ const char *name)
{
return kernfs_find_and_get(parent, name);
}
*/
struct tk_read_base {
struct clocksource *clock;
- u64 (*read)(struct clocksource *cs);
u64 mask;
u64 cycle_last;
u32 mult;
* @clock_was_set_seq: The sequence number of clock was set events
* @cs_was_changed_seq: The sequence number of clocksource change events
* @next_leap_ktime: CLOCK_MONOTONIC time value of a pending leap-second
- * @raw_time: Monotonic raw base time in timespec64 format
+ * @raw_sec: CLOCK_MONOTONIC_RAW time in seconds
* @cycle_interval: Number of clock cycles in one NTP interval
* @xtime_interval: Number of clock shifted nano seconds in one NTP
* interval.
* @xtime_remainder: Shifted nano seconds left over when rounding
* @cycle_interval
- * @raw_interval: Raw nano seconds accumulated per NTP interval.
+ * @raw_interval: Shifted raw nano seconds accumulated per NTP interval.
* @ntp_error: Difference between accumulated time and NTP time in ntp
* shifted nano seconds.
* @ntp_error_shift: Shift conversion between clock shifted nano seconds and
unsigned int clock_was_set_seq;
u8 cs_was_changed_seq;
ktime_t next_leap_ktime;
- struct timespec64 raw_time;
+ u64 raw_sec;
/* The following members are for timekeeping internal use */
u64 cycle_interval;
u64 xtime_interval;
s64 xtime_remainder;
- u32 raw_interval;
+ u64 raw_interval;
/* The ntp_tick_length() value currently being used.
* This cached copy ensures we consistently apply the tick
* length for an entire tick, as ntp_tick_length may change
struct tty_struct *link;
struct fasync_struct *fasync;
- int alt_speed; /* For magic substitution of 38400 bps */
wait_queue_head_t write_wait;
wait_queue_head_t read_wait;
struct work_struct hangup_work;
/* tty_io.c */
extern int __init tty_init(void);
extern const char *tty_name(const struct tty_struct *tty);
+extern struct tty_struct *tty_open_by_driver(dev_t device, struct inode *inode,
+ struct file *filp);
+extern int tty_dev_name_to_number(const char *name, dev_t *number);
#else
static inline void tty_kref_put(struct tty_struct *tty)
{ }
{ return 0; }
static inline const char *tty_name(const struct tty_struct *tty)
{ return "(none)"; }
+static inline struct tty_struct *tty_open_by_driver(dev_t device,
+ struct inode *inode, struct file *filp)
+{ return NULL; }
+static inline int tty_dev_name_to_number(const char *name, dev_t *number)
+{ return -ENOTSUPP; }
#endif
extern struct ktermios tty_std_termios;
int (*udc_start)(struct usb_gadget *,
struct usb_gadget_driver *);
int (*udc_stop)(struct usb_gadget *);
+ void (*udc_set_speed)(struct usb_gadget *, enum usb_device_speed);
struct usb_ep *(*match_ep)(struct usb_gadget *,
struct usb_endpoint_descriptor *,
struct usb_ss_ep_comp_descriptor *);
* @deactivated: True if gadget is deactivated - in deactivated state it cannot
* be connected.
* @connected: True if gadget is connected.
+ * @lpm_capable: If the gadget max_speed is FULL or HIGH, this flag
+ * indicates that it supports LPM as per the LPM ECN & errata.
*
* Gadgets have a mostly-portable "gadget driver" implementing device
* functions, handling all usb configurations and interfaces. Gadget
unsigned is_selfpowered:1;
unsigned deactivated:1;
unsigned connected:1;
+ unsigned lpm_capable:1;
};
#define work_to_gadget(w) (container_of((w), struct usb_gadget, work))
((USB_DIR_IN|USB_TYPE_STANDARD|USB_RECIP_INTERFACE)<<8)
#define EndpointRequest \
- ((USB_DIR_IN|USB_TYPE_STANDARD|USB_RECIP_INTERFACE)<<8)
+ ((USB_DIR_IN|USB_TYPE_STANDARD|USB_RECIP_ENDPOINT)<<8)
#define EndpointOutRequest \
- ((USB_DIR_OUT|USB_TYPE_STANDARD|USB_RECIP_INTERFACE)<<8)
+ ((USB_DIR_OUT|USB_TYPE_STANDARD|USB_RECIP_ENDPOINT)<<8)
/* class requests from the USB 2.0 hub spec, table 11-15 */
#define HUB_CLASS_REQ(dir, type, request) ((((dir) | (type)) << 8) | (request))
#ifndef __LINUX_USB_PHY_H
#define __LINUX_USB_PHY_H
+#include <linux/extcon.h>
#include <linux/notifier.h>
#include <linux/usb.h>
struct usb_phy_io_ops *io_ops;
void __iomem *io_priv;
+ /* to support extcon device */
+ struct extcon_dev *edev;
+ struct extcon_dev *id_edev;
+ struct notifier_block vbus_nb;
+ struct notifier_block id_nb;
+
/* for notification of usb_phy_events */
struct atomic_notifier_head notifier;
struct typec_altmode
*typec_partner_register_altmode(struct typec_partner *partner,
- struct typec_altmode_desc *desc);
+ const struct typec_altmode_desc *desc);
struct typec_altmode
*typec_plug_register_altmode(struct typec_plug *plug,
- struct typec_altmode_desc *desc);
+ const struct typec_altmode_desc *desc);
struct typec_altmode
*typec_port_register_altmode(struct typec_port *port,
- struct typec_altmode_desc *desc);
+ const struct typec_altmode_desc *desc);
void typec_unregister_altmode(struct typec_altmode *altmode);
struct typec_port *typec_altmode2port(struct typec_altmode *alt);
* @pr_set: Set Power Role
* @vconn_set: Set VCONN Role
* @activate_mode: Enter/exit given Alternate Mode
+ * @port_type_set: Set port type
*
* Static capabilities of a single USB Type-C port.
*/
int (*activate_mode)(const struct typec_capability *,
int mode, int activate);
+ int (*port_type_set)(const struct typec_capability *,
+ enum typec_port_type);
+
};
/* Specific to try_role(). Indicates the user want's to clear the preference. */
#include <uapi/linux/uuid.h>
-/*
- * V1 (time-based) UUID definition [RFC 4122].
- * - the timestamp is a 60-bit value, split 32/16/12, and goes in 100ns
- * increments since midnight 15th October 1582
- * - add AFS_UUID_TO_UNIX_TIME to convert unix time in 100ns units to UUID
- * time
- * - the clock sequence is a 14-bit counter to avoid duplicate times
- */
-struct uuid_v1 {
- __be32 time_low; /* low part of timestamp */
- __be16 time_mid; /* mid part of timestamp */
- __be16 time_hi_and_version; /* high part of timestamp and version */
-#define UUID_TO_UNIX_TIME 0x01b21dd213814000ULL
-#define UUID_TIMEHI_MASK 0x0fff
-#define UUID_VERSION_TIME 0x1000 /* time-based UUID */
-#define UUID_VERSION_NAME 0x3000 /* name-based UUID */
-#define UUID_VERSION_RANDOM 0x4000 /* (pseudo-)random generated UUID */
- u8 clock_seq_hi_and_reserved; /* clock seq hi and variant */
-#define UUID_CLOCKHI_MASK 0x3f
-#define UUID_VARIANT_STD 0x80
- u8 clock_seq_low; /* clock seq low */
- u8 node[6]; /* spatially unique node ID (MAC addr) */
-};
+typedef struct {
+ __u8 b[16];
+} uuid_t;
+
+#define UUID_INIT(a, b, c, d0, d1, d2, d3, d4, d5, d6, d7) \
+((uuid_t) \
+{{ ((a) >> 24) & 0xff, ((a) >> 16) & 0xff, ((a) >> 8) & 0xff, (a) & 0xff, \
+ ((b) >> 8) & 0xff, (b) & 0xff, \
+ ((c) >> 8) & 0xff, (c) & 0xff, \
+ (d0), (d1), (d2), (d3), (d4), (d5), (d6), (d7) }})
/*
* The length of a UUID string ("aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee")
*/
#define UUID_STRING_LEN 36
-static inline int uuid_le_cmp(const uuid_le u1, const uuid_le u2)
+extern const guid_t guid_null;
+extern const uuid_t uuid_null;
+
+static inline bool guid_equal(const guid_t *u1, const guid_t *u2)
+{
+ return memcmp(u1, u2, sizeof(guid_t)) == 0;
+}
+
+static inline void guid_copy(guid_t *dst, const guid_t *src)
+{
+ memcpy(dst, src, sizeof(guid_t));
+}
+
+static inline bool guid_is_null(const guid_t *guid)
+{
+ return guid_equal(guid, &guid_null);
+}
+
+static inline bool uuid_equal(const uuid_t *u1, const uuid_t *u2)
+{
+ return memcmp(u1, u2, sizeof(uuid_t)) == 0;
+}
+
+static inline void uuid_copy(uuid_t *dst, const uuid_t *src)
{
- return memcmp(&u1, &u2, sizeof(uuid_le));
+ memcpy(dst, src, sizeof(uuid_t));
}
-static inline int uuid_be_cmp(const uuid_be u1, const uuid_be u2)
+static inline bool uuid_is_null(const uuid_t *uuid)
{
- return memcmp(&u1, &u2, sizeof(uuid_be));
+ return uuid_equal(uuid, &uuid_null);
}
void generate_random_uuid(unsigned char uuid[16]);
-extern void uuid_le_gen(uuid_le *u);
-extern void uuid_be_gen(uuid_be *u);
+extern void guid_gen(guid_t *u);
+extern void uuid_gen(uuid_t *u);
bool __must_check uuid_is_valid(const char *uuid);
-extern const u8 uuid_le_index[16];
-extern const u8 uuid_be_index[16];
+extern const u8 guid_index[16];
+extern const u8 uuid_index[16];
+
+int guid_parse(const char *uuid, guid_t *u);
+int uuid_parse(const char *uuid, uuid_t *u);
+
+/* backwards compatibility, don't use in new code */
+typedef uuid_t uuid_be;
+#define UUID_BE(a, _b, c, d0, d1, d2, d3, d4, d5, d6, d7) \
+ UUID_INIT(a, _b, c, d0, d1, d2, d3, d4, d5, d6, d7)
+#define NULL_UUID_BE \
+ UUID_BE(0x00000000, 0x0000, 0x0000, 0x00, 0x00, 0x00, 0x00, \
+ 0x00, 0x00, 0x00, 0x00)
-int uuid_le_to_bin(const char *uuid, uuid_le *u);
-int uuid_be_to_bin(const char *uuid, uuid_be *u);
+#define uuid_le_gen(u) guid_gen(u)
+#define uuid_be_gen(u) uuid_gen(u)
+#define uuid_le_to_bin(guid, u) guid_parse(guid, u)
+#define uuid_be_to_bin(uuid, u) uuid_parse(uuid, u)
+
+static inline int uuid_le_cmp(const guid_t u1, const guid_t u2)
+{
+ return memcmp(&u1, &u2, sizeof(guid_t));
+}
+
+static inline int uuid_be_cmp(const uuid_t u1, const uuid_t u2)
+{
+ return memcmp(&u1, &u2, sizeof(uuid_t));
+}
#endif
void (*thread)(void *, void *);
void *data;
struct work_struct inject;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
poll_table pt;
struct work_struct shutdown;
struct virqfd **pvirqfd;
#endif
#endif
#ifdef CONFIG_DEBUG_TLBFLUSH
-#ifdef CONFIG_SMP
NR_TLB_REMOTE_FLUSH, /* cpu tried to flush others' tlbs */
NR_TLB_REMOTE_FLUSH_RECEIVED,/* cpu received ipi for flush */
-#endif /* CONFIG_SMP */
NR_TLB_LOCAL_FLUSH_ALL,
NR_TLB_LOCAL_FLUSH_ONE,
#endif /* CONFIG_DEBUG_TLBFLUSH */
* GNU General Public License for more details.
*/
-#ifndef __W1_H
-#define __W1_H
+#ifndef __LINUX_W1_H
+#define __LINUX_W1_H
+
+#include <linux/device.h>
/**
* struct w1_reg_num - broken out slave device id
* @id: along with family is the unique device id
* @crc: checksum of the other bytes
*/
-struct w1_reg_num
-{
+struct w1_reg_num {
#if defined(__LITTLE_ENDIAN_BITFIELD)
__u64 family:8,
id:48,
#ifdef __KERNEL__
-#include <linux/completion.h>
-#include <linux/device.h>
-#include <linux/mutex.h>
-
-#include "w1_family.h"
-
#define W1_MAXNAMELEN 32
#define W1_SEARCH 0xF0
#define W1_MATCH_ROM 0x55
#define W1_RESUME_CMD 0xA5
-#define W1_SLAVE_ACTIVE 0
-#define W1_SLAVE_DETACH 1
-
/**
* struct w1_slave - holds a single slave device on the bus
*
* @dev: kernel device identifier
*
*/
-struct w1_slave
-{
+struct w1_slave {
struct module *owner;
unsigned char name[W1_MAXNAMELEN];
struct list_head w1_slave_entry;
typedef void (*w1_slave_found_callback)(struct w1_master *, u64);
-
/**
* struct w1_bus_master - operations available on a bus master
*
* reset_bus.
*
*/
-struct w1_bus_master
-{
+struct w1_bus_master {
void *data;
u8 (*read_bit)(void *);
* @bus_master: io operations available
* @seq: sequence number used for netlink broadcasts
*/
-struct w1_master
-{
+struct w1_master {
struct list_head w1_master_entry;
struct module *owner;
unsigned char name[W1_MAXNAMELEN];
u32 seq;
};
+int w1_add_master_device(struct w1_bus_master *master);
+void w1_remove_master_device(struct w1_bus_master *master);
+
/**
- * struct w1_async_cmd - execute callback from the w1_process kthread
- * @async_entry: link entry
- * @cb: callback function, must list_del and destroy this list before
- * returning
- *
- * When inserted into the w1_master async_list, w1_process will execute
- * the callback. Embed this into the structure with the command details.
+ * struct w1_family_ops - operations for a family type
+ * @add_slave: add_slave
+ * @remove_slave: remove_slave
+ * @groups: sysfs group
*/
-struct w1_async_cmd {
- struct list_head async_entry;
- void (*cb)(struct w1_master *dev, struct w1_async_cmd *async_cmd);
+struct w1_family_ops {
+ int (*add_slave)(struct w1_slave *sl);
+ void (*remove_slave)(struct w1_slave *sl);
+ const struct attribute_group **groups;
};
-int w1_create_master_attributes(struct w1_master *);
-void w1_destroy_master_attributes(struct w1_master *master);
-void w1_search(struct w1_master *dev, u8 search_type, w1_slave_found_callback cb);
-void w1_search_devices(struct w1_master *dev, u8 search_type, w1_slave_found_callback cb);
-/* call w1_unref_slave to release the reference counts w1_search_slave added */
-struct w1_slave *w1_search_slave(struct w1_reg_num *id);
-/* decrements the reference on sl->master and sl, and cleans up if zero
- * returns the reference count after it has been decremented */
-int w1_unref_slave(struct w1_slave *sl);
-void w1_slave_found(struct w1_master *dev, u64 rn);
-void w1_search_process_cb(struct w1_master *dev, u8 search_type,
- w1_slave_found_callback cb);
-struct w1_slave *w1_slave_search_device(struct w1_master *dev,
- struct w1_reg_num *rn);
-struct w1_master *w1_search_master_id(u32 id);
-
-/* Disconnect and reconnect devices in the given family. Used for finding
- * unclaimed devices after a family has been registered or releasing devices
- * after a family has been unregistered. Set attach to 1 when a new family
- * has just been registered, to 0 when it has been unregistered.
+/**
+ * struct w1_family - reference counted family structure.
+ * @family_entry: family linked list
+ * @fid: 8 bit family identifier
+ * @fops: operations for this family
+ * @refcnt: reference counter
+ */
+struct w1_family {
+ struct list_head family_entry;
+ u8 fid;
+
+ struct w1_family_ops *fops;
+
+ atomic_t refcnt;
+};
+
+int w1_register_family(struct w1_family *family);
+void w1_unregister_family(struct w1_family *family);
+
+/**
+ * module_w1_driver() - Helper macro for registering a 1-Wire families
+ * @__w1_family: w1_family struct
+ *
+ * Helper macro for 1-Wire families which do not do anything special in module
+ * init/exit. This eliminates a lot of boilerplate. Each module may only
+ * use this macro once, and calling it replaces module_init() and module_exit()
*/
-void w1_reconnect_slaves(struct w1_family *f, int attach);
-int w1_attach_slave_device(struct w1_master *dev, struct w1_reg_num *rn);
-/* 0 success, otherwise EBUSY */
-int w1_slave_detach(struct w1_slave *sl);
+#define module_w1_family(__w1_family) \
+ module_driver(__w1_family, w1_register_family, \
+ w1_unregister_family)
u8 w1_triplet(struct w1_master *dev, int bdir);
void w1_write_8(struct w1_master *, u8);
return container_of(dev, struct w1_master, dev);
}
-extern struct device_driver w1_master_driver;
-extern struct device w1_master_device;
-extern int w1_max_slave_count;
-extern int w1_max_slave_ttl;
-extern struct list_head w1_masters;
-extern struct mutex w1_mlock;
-
-extern int w1_process_callbacks(struct w1_master *dev);
-extern int w1_process(void *);
-
#endif /* __KERNEL__ */
-#endif /* __W1_H */
+#endif /* __LINUX_W1_H */
#include <asm/current.h>
#include <uapi/linux/wait.h>
-typedef struct __wait_queue wait_queue_t;
-typedef int (*wait_queue_func_t)(wait_queue_t *wait, unsigned mode, int flags, void *key);
-int default_wake_function(wait_queue_t *wait, unsigned mode, int flags, void *key);
+typedef struct wait_queue_entry wait_queue_entry_t;
-/* __wait_queue::flags */
+typedef int (*wait_queue_func_t)(struct wait_queue_entry *wq_entry, unsigned mode, int flags, void *key);
+int default_wake_function(struct wait_queue_entry *wq_entry, unsigned mode, int flags, void *key);
+
+/* wait_queue_entry::flags */
#define WQ_FLAG_EXCLUSIVE 0x01
#define WQ_FLAG_WOKEN 0x02
-struct __wait_queue {
+/*
+ * A single wait-queue entry structure:
+ */
+struct wait_queue_entry {
unsigned int flags;
void *private;
wait_queue_func_t func;
- struct list_head task_list;
-};
-
-struct wait_bit_key {
- void *flags;
- int bit_nr;
-#define WAIT_ATOMIC_T_BIT_NR -1
- unsigned long timeout;
+ struct list_head entry;
};
-struct wait_bit_queue {
- struct wait_bit_key key;
- wait_queue_t wait;
-};
-
-struct __wait_queue_head {
+struct wait_queue_head {
spinlock_t lock;
- struct list_head task_list;
+ struct list_head head;
};
-typedef struct __wait_queue_head wait_queue_head_t;
+typedef struct wait_queue_head wait_queue_head_t;
struct task_struct;
* Macros for declaration and initialisaton of the datatypes
*/
-#define __WAITQUEUE_INITIALIZER(name, tsk) { \
- .private = tsk, \
- .func = default_wake_function, \
- .task_list = { NULL, NULL } }
+#define __WAITQUEUE_INITIALIZER(name, tsk) { \
+ .private = tsk, \
+ .func = default_wake_function, \
+ .entry = { NULL, NULL } }
-#define DECLARE_WAITQUEUE(name, tsk) \
- wait_queue_t name = __WAITQUEUE_INITIALIZER(name, tsk)
+#define DECLARE_WAITQUEUE(name, tsk) \
+ struct wait_queue_entry name = __WAITQUEUE_INITIALIZER(name, tsk)
-#define __WAIT_QUEUE_HEAD_INITIALIZER(name) { \
- .lock = __SPIN_LOCK_UNLOCKED(name.lock), \
- .task_list = { &(name).task_list, &(name).task_list } }
+#define __WAIT_QUEUE_HEAD_INITIALIZER(name) { \
+ .lock = __SPIN_LOCK_UNLOCKED(name.lock), \
+ .head = { &(name).head, &(name).head } }
#define DECLARE_WAIT_QUEUE_HEAD(name) \
- wait_queue_head_t name = __WAIT_QUEUE_HEAD_INITIALIZER(name)
-
-#define __WAIT_BIT_KEY_INITIALIZER(word, bit) \
- { .flags = word, .bit_nr = bit, }
+ struct wait_queue_head name = __WAIT_QUEUE_HEAD_INITIALIZER(name)
-#define __WAIT_ATOMIC_T_KEY_INITIALIZER(p) \
- { .flags = p, .bit_nr = WAIT_ATOMIC_T_BIT_NR, }
+extern void __init_waitqueue_head(struct wait_queue_head *wq_head, const char *name, struct lock_class_key *);
-extern void __init_waitqueue_head(wait_queue_head_t *q, const char *name, struct lock_class_key *);
-
-#define init_waitqueue_head(q) \
- do { \
- static struct lock_class_key __key; \
- \
- __init_waitqueue_head((q), #q, &__key); \
+#define init_waitqueue_head(wq_head) \
+ do { \
+ static struct lock_class_key __key; \
+ \
+ __init_waitqueue_head((wq_head), #wq_head, &__key); \
} while (0)
#ifdef CONFIG_LOCKDEP
# define __WAIT_QUEUE_HEAD_INIT_ONSTACK(name) \
({ init_waitqueue_head(&name); name; })
# define DECLARE_WAIT_QUEUE_HEAD_ONSTACK(name) \
- wait_queue_head_t name = __WAIT_QUEUE_HEAD_INIT_ONSTACK(name)
+ struct wait_queue_head name = __WAIT_QUEUE_HEAD_INIT_ONSTACK(name)
#else
# define DECLARE_WAIT_QUEUE_HEAD_ONSTACK(name) DECLARE_WAIT_QUEUE_HEAD(name)
#endif
-static inline void init_waitqueue_entry(wait_queue_t *q, struct task_struct *p)
+static inline void init_waitqueue_entry(struct wait_queue_entry *wq_entry, struct task_struct *p)
{
- q->flags = 0;
- q->private = p;
- q->func = default_wake_function;
+ wq_entry->flags = 0;
+ wq_entry->private = p;
+ wq_entry->func = default_wake_function;
}
static inline void
-init_waitqueue_func_entry(wait_queue_t *q, wait_queue_func_t func)
+init_waitqueue_func_entry(struct wait_queue_entry *wq_entry, wait_queue_func_t func)
{
- q->flags = 0;
- q->private = NULL;
- q->func = func;
+ wq_entry->flags = 0;
+ wq_entry->private = NULL;
+ wq_entry->func = func;
}
/**
* waitqueue_active -- locklessly test for waiters on the queue
- * @q: the waitqueue to test for waiters
+ * @wq_head: the waitqueue to test for waiters
*
* returns true if the wait list is not empty
*
* NOTE: this function is lockless and requires care, incorrect usage _will_
* lead to sporadic and non-obvious failure.
*
- * Use either while holding wait_queue_head_t::lock or when used for wakeups
+ * Use either while holding wait_queue_head::lock or when used for wakeups
* with an extra smp_mb() like:
*
* CPU0 - waker CPU1 - waiter
*
* for (;;) {
- * @cond = true; prepare_to_wait(&wq, &wait, state);
+ * @cond = true; prepare_to_wait(&wq_head, &wait, state);
* smp_mb(); // smp_mb() from set_current_state()
- * if (waitqueue_active(wq)) if (@cond)
- * wake_up(wq); break;
+ * if (waitqueue_active(wq_head)) if (@cond)
+ * wake_up(wq_head); break;
* schedule();
* }
- * finish_wait(&wq, &wait);
+ * finish_wait(&wq_head, &wait);
*
* Because without the explicit smp_mb() it's possible for the
* waitqueue_active() load to get hoisted over the @cond store such that we'll
* Also note that this 'optimization' trades a spin_lock() for an smp_mb(),
* which (when the lock is uncontended) are of roughly equal cost.
*/
-static inline int waitqueue_active(wait_queue_head_t *q)
+static inline int waitqueue_active(struct wait_queue_head *wq_head)
{
- return !list_empty(&q->task_list);
+ return !list_empty(&wq_head->head);
}
/**
* wq_has_sleeper - check if there are any waiting processes
- * @wq: wait queue head
+ * @wq_head: wait queue head
*
- * Returns true if wq has waiting processes
+ * Returns true if wq_head has waiting processes
*
* Please refer to the comment for waitqueue_active.
*/
-static inline bool wq_has_sleeper(wait_queue_head_t *wq)
+static inline bool wq_has_sleeper(struct wait_queue_head *wq_head)
{
/*
* We need to be sure we are in sync with the
* waiting side.
*/
smp_mb();
- return waitqueue_active(wq);
+ return waitqueue_active(wq_head);
}
-extern void add_wait_queue(wait_queue_head_t *q, wait_queue_t *wait);
-extern void add_wait_queue_exclusive(wait_queue_head_t *q, wait_queue_t *wait);
-extern void remove_wait_queue(wait_queue_head_t *q, wait_queue_t *wait);
+extern void add_wait_queue(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry);
+extern void add_wait_queue_exclusive(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry);
+extern void remove_wait_queue(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry);
-static inline void __add_wait_queue(wait_queue_head_t *head, wait_queue_t *new)
+static inline void __add_wait_queue(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry)
{
- list_add(&new->task_list, &head->task_list);
+ list_add(&wq_entry->entry, &wq_head->head);
}
/*
* Used for wake-one threads:
*/
static inline void
-__add_wait_queue_exclusive(wait_queue_head_t *q, wait_queue_t *wait)
+__add_wait_queue_exclusive(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry)
{
- wait->flags |= WQ_FLAG_EXCLUSIVE;
- __add_wait_queue(q, wait);
+ wq_entry->flags |= WQ_FLAG_EXCLUSIVE;
+ __add_wait_queue(wq_head, wq_entry);
}
-static inline void __add_wait_queue_tail(wait_queue_head_t *head,
- wait_queue_t *new)
+static inline void __add_wait_queue_entry_tail(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry)
{
- list_add_tail(&new->task_list, &head->task_list);
+ list_add_tail(&wq_entry->entry, &wq_head->head);
}
static inline void
-__add_wait_queue_tail_exclusive(wait_queue_head_t *q, wait_queue_t *wait)
+__add_wait_queue_entry_tail_exclusive(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry)
{
- wait->flags |= WQ_FLAG_EXCLUSIVE;
- __add_wait_queue_tail(q, wait);
+ wq_entry->flags |= WQ_FLAG_EXCLUSIVE;
+ __add_wait_queue_entry_tail(wq_head, wq_entry);
}
static inline void
-__remove_wait_queue(wait_queue_head_t *head, wait_queue_t *old)
+__remove_wait_queue(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry)
{
- list_del(&old->task_list);
+ list_del(&wq_entry->entry);
}
-typedef int wait_bit_action_f(struct wait_bit_key *, int mode);
-void __wake_up(wait_queue_head_t *q, unsigned int mode, int nr, void *key);
-void __wake_up_locked_key(wait_queue_head_t *q, unsigned int mode, void *key);
-void __wake_up_sync_key(wait_queue_head_t *q, unsigned int mode, int nr, void *key);
-void __wake_up_locked(wait_queue_head_t *q, unsigned int mode, int nr);
-void __wake_up_sync(wait_queue_head_t *q, unsigned int mode, int nr);
-void __wake_up_bit(wait_queue_head_t *, void *, int);
-int __wait_on_bit(wait_queue_head_t *, struct wait_bit_queue *, wait_bit_action_f *, unsigned);
-int __wait_on_bit_lock(wait_queue_head_t *, struct wait_bit_queue *, wait_bit_action_f *, unsigned);
-void wake_up_bit(void *, int);
-void wake_up_atomic_t(atomic_t *);
-int out_of_line_wait_on_bit(void *, int, wait_bit_action_f *, unsigned);
-int out_of_line_wait_on_bit_timeout(void *, int, wait_bit_action_f *, unsigned, unsigned long);
-int out_of_line_wait_on_bit_lock(void *, int, wait_bit_action_f *, unsigned);
-int out_of_line_wait_on_atomic_t(atomic_t *, int (*)(atomic_t *), unsigned);
-wait_queue_head_t *bit_waitqueue(void *, int);
+void __wake_up(struct wait_queue_head *wq_head, unsigned int mode, int nr, void *key);
+void __wake_up_locked_key(struct wait_queue_head *wq_head, unsigned int mode, void *key);
+void __wake_up_sync_key(struct wait_queue_head *wq_head, unsigned int mode, int nr, void *key);
+void __wake_up_locked(struct wait_queue_head *wq_head, unsigned int mode, int nr);
+void __wake_up_sync(struct wait_queue_head *wq_head, unsigned int mode, int nr);
#define wake_up(x) __wake_up(x, TASK_NORMAL, 1, NULL)
#define wake_up_nr(x, nr) __wake_up(x, TASK_NORMAL, nr, NULL)
/*
* Wakeup macros to be used to report events to the targets.
*/
-#define wake_up_poll(x, m) \
+#define wake_up_poll(x, m) \
__wake_up(x, TASK_NORMAL, 1, (void *) (m))
-#define wake_up_locked_poll(x, m) \
+#define wake_up_locked_poll(x, m) \
__wake_up_locked_key((x), TASK_NORMAL, (void *) (m))
-#define wake_up_interruptible_poll(x, m) \
+#define wake_up_interruptible_poll(x, m) \
__wake_up(x, TASK_INTERRUPTIBLE, 1, (void *) (m))
-#define wake_up_interruptible_sync_poll(x, m) \
+#define wake_up_interruptible_sync_poll(x, m) \
__wake_up_sync_key((x), TASK_INTERRUPTIBLE, 1, (void *) (m))
-#define ___wait_cond_timeout(condition) \
-({ \
- bool __cond = (condition); \
- if (__cond && !__ret) \
- __ret = 1; \
- __cond || !__ret; \
+#define ___wait_cond_timeout(condition) \
+({ \
+ bool __cond = (condition); \
+ if (__cond && !__ret) \
+ __ret = 1; \
+ __cond || !__ret; \
})
-#define ___wait_is_interruptible(state) \
- (!__builtin_constant_p(state) || \
- state == TASK_INTERRUPTIBLE || state == TASK_KILLABLE) \
+#define ___wait_is_interruptible(state) \
+ (!__builtin_constant_p(state) || \
+ state == TASK_INTERRUPTIBLE || state == TASK_KILLABLE) \
-extern void init_wait_entry(wait_queue_t *__wait, int flags);
+extern void init_wait_entry(struct wait_queue_entry *wq_entry, int flags);
/*
* The below macro ___wait_event() has an explicit shadow of the __ret
* otherwise.
*/
-#define ___wait_event(wq, condition, state, exclusive, ret, cmd) \
-({ \
- __label__ __out; \
- wait_queue_t __wait; \
- long __ret = ret; /* explicit shadow */ \
- \
- init_wait_entry(&__wait, exclusive ? WQ_FLAG_EXCLUSIVE : 0); \
- for (;;) { \
- long __int = prepare_to_wait_event(&wq, &__wait, state);\
- \
- if (condition) \
- break; \
- \
- if (___wait_is_interruptible(state) && __int) { \
- __ret = __int; \
- goto __out; \
- } \
- \
- cmd; \
- } \
- finish_wait(&wq, &__wait); \
-__out: __ret; \
+#define ___wait_event(wq_head, condition, state, exclusive, ret, cmd) \
+({ \
+ __label__ __out; \
+ struct wait_queue_entry __wq_entry; \
+ long __ret = ret; /* explicit shadow */ \
+ \
+ init_wait_entry(&__wq_entry, exclusive ? WQ_FLAG_EXCLUSIVE : 0); \
+ for (;;) { \
+ long __int = prepare_to_wait_event(&wq_head, &__wq_entry, state);\
+ \
+ if (condition) \
+ break; \
+ \
+ if (___wait_is_interruptible(state) && __int) { \
+ __ret = __int; \
+ goto __out; \
+ } \
+ \
+ cmd; \
+ } \
+ finish_wait(&wq_head, &__wq_entry); \
+__out: __ret; \
})
-#define __wait_event(wq, condition) \
- (void)___wait_event(wq, condition, TASK_UNINTERRUPTIBLE, 0, 0, \
+#define __wait_event(wq_head, condition) \
+ (void)___wait_event(wq_head, condition, TASK_UNINTERRUPTIBLE, 0, 0, \
schedule())
/**
* wait_event - sleep until a condition gets true
- * @wq: the waitqueue to wait on
+ * @wq_head: the waitqueue to wait on
* @condition: a C expression for the event to wait for
*
* The process is put to sleep (TASK_UNINTERRUPTIBLE) until the
* @condition evaluates to true. The @condition is checked each time
- * the waitqueue @wq is woken up.
+ * the waitqueue @wq_head is woken up.
*
* wake_up() has to be called after changing any variable that could
* change the result of the wait condition.
*/
-#define wait_event(wq, condition) \
-do { \
- might_sleep(); \
- if (condition) \
- break; \
- __wait_event(wq, condition); \
+#define wait_event(wq_head, condition) \
+do { \
+ might_sleep(); \
+ if (condition) \
+ break; \
+ __wait_event(wq_head, condition); \
} while (0)
-#define __io_wait_event(wq, condition) \
- (void)___wait_event(wq, condition, TASK_UNINTERRUPTIBLE, 0, 0, \
+#define __io_wait_event(wq_head, condition) \
+ (void)___wait_event(wq_head, condition, TASK_UNINTERRUPTIBLE, 0, 0, \
io_schedule())
/*
* io_wait_event() -- like wait_event() but with io_schedule()
*/
-#define io_wait_event(wq, condition) \
-do { \
- might_sleep(); \
- if (condition) \
- break; \
- __io_wait_event(wq, condition); \
+#define io_wait_event(wq_head, condition) \
+do { \
+ might_sleep(); \
+ if (condition) \
+ break; \
+ __io_wait_event(wq_head, condition); \
} while (0)
-#define __wait_event_freezable(wq, condition) \
- ___wait_event(wq, condition, TASK_INTERRUPTIBLE, 0, 0, \
+#define __wait_event_freezable(wq_head, condition) \
+ ___wait_event(wq_head, condition, TASK_INTERRUPTIBLE, 0, 0, \
schedule(); try_to_freeze())
/**
* wait_event_freezable - sleep (or freeze) until a condition gets true
- * @wq: the waitqueue to wait on
+ * @wq_head: the waitqueue to wait on
* @condition: a C expression for the event to wait for
*
* The process is put to sleep (TASK_INTERRUPTIBLE -- so as not to contribute
* to system load) until the @condition evaluates to true. The
- * @condition is checked each time the waitqueue @wq is woken up.
+ * @condition is checked each time the waitqueue @wq_head is woken up.
*
* wake_up() has to be called after changing any variable that could
* change the result of the wait condition.
*/
-#define wait_event_freezable(wq, condition) \
-({ \
- int __ret = 0; \
- might_sleep(); \
- if (!(condition)) \
- __ret = __wait_event_freezable(wq, condition); \
- __ret; \
+#define wait_event_freezable(wq_head, condition) \
+({ \
+ int __ret = 0; \
+ might_sleep(); \
+ if (!(condition)) \
+ __ret = __wait_event_freezable(wq_head, condition); \
+ __ret; \
})
-#define __wait_event_timeout(wq, condition, timeout) \
- ___wait_event(wq, ___wait_cond_timeout(condition), \
- TASK_UNINTERRUPTIBLE, 0, timeout, \
+#define __wait_event_timeout(wq_head, condition, timeout) \
+ ___wait_event(wq_head, ___wait_cond_timeout(condition), \
+ TASK_UNINTERRUPTIBLE, 0, timeout, \
__ret = schedule_timeout(__ret))
/**
* wait_event_timeout - sleep until a condition gets true or a timeout elapses
- * @wq: the waitqueue to wait on
+ * @wq_head: the waitqueue to wait on
* @condition: a C expression for the event to wait for
* @timeout: timeout, in jiffies
*
* The process is put to sleep (TASK_UNINTERRUPTIBLE) until the
* @condition evaluates to true. The @condition is checked each time
- * the waitqueue @wq is woken up.
+ * the waitqueue @wq_head is woken up.
*
* wake_up() has to be called after changing any variable that could
* change the result of the wait condition.
* or the remaining jiffies (at least 1) if the @condition evaluated
* to %true before the @timeout elapsed.
*/
-#define wait_event_timeout(wq, condition, timeout) \
-({ \
- long __ret = timeout; \
- might_sleep(); \
- if (!___wait_cond_timeout(condition)) \
- __ret = __wait_event_timeout(wq, condition, timeout); \
- __ret; \
+#define wait_event_timeout(wq_head, condition, timeout) \
+({ \
+ long __ret = timeout; \
+ might_sleep(); \
+ if (!___wait_cond_timeout(condition)) \
+ __ret = __wait_event_timeout(wq_head, condition, timeout); \
+ __ret; \
})
-#define __wait_event_freezable_timeout(wq, condition, timeout) \
- ___wait_event(wq, ___wait_cond_timeout(condition), \
- TASK_INTERRUPTIBLE, 0, timeout, \
+#define __wait_event_freezable_timeout(wq_head, condition, timeout) \
+ ___wait_event(wq_head, ___wait_cond_timeout(condition), \
+ TASK_INTERRUPTIBLE, 0, timeout, \
__ret = schedule_timeout(__ret); try_to_freeze())
/*
* like wait_event_timeout() -- except it uses TASK_INTERRUPTIBLE to avoid
* increasing load and is freezable.
*/
-#define wait_event_freezable_timeout(wq, condition, timeout) \
-({ \
- long __ret = timeout; \
- might_sleep(); \
- if (!___wait_cond_timeout(condition)) \
- __ret = __wait_event_freezable_timeout(wq, condition, timeout); \
- __ret; \
+#define wait_event_freezable_timeout(wq_head, condition, timeout) \
+({ \
+ long __ret = timeout; \
+ might_sleep(); \
+ if (!___wait_cond_timeout(condition)) \
+ __ret = __wait_event_freezable_timeout(wq_head, condition, timeout); \
+ __ret; \
})
-#define __wait_event_exclusive_cmd(wq, condition, cmd1, cmd2) \
- (void)___wait_event(wq, condition, TASK_UNINTERRUPTIBLE, 1, 0, \
+#define __wait_event_exclusive_cmd(wq_head, condition, cmd1, cmd2) \
+ (void)___wait_event(wq_head, condition, TASK_UNINTERRUPTIBLE, 1, 0, \
cmd1; schedule(); cmd2)
/*
* Just like wait_event_cmd(), except it sets exclusive flag
*/
-#define wait_event_exclusive_cmd(wq, condition, cmd1, cmd2) \
-do { \
- if (condition) \
- break; \
- __wait_event_exclusive_cmd(wq, condition, cmd1, cmd2); \
+#define wait_event_exclusive_cmd(wq_head, condition, cmd1, cmd2) \
+do { \
+ if (condition) \
+ break; \
+ __wait_event_exclusive_cmd(wq_head, condition, cmd1, cmd2); \
} while (0)
-#define __wait_event_cmd(wq, condition, cmd1, cmd2) \
- (void)___wait_event(wq, condition, TASK_UNINTERRUPTIBLE, 0, 0, \
+#define __wait_event_cmd(wq_head, condition, cmd1, cmd2) \
+ (void)___wait_event(wq_head, condition, TASK_UNINTERRUPTIBLE, 0, 0, \
cmd1; schedule(); cmd2)
/**
* wait_event_cmd - sleep until a condition gets true
- * @wq: the waitqueue to wait on
+ * @wq_head: the waitqueue to wait on
* @condition: a C expression for the event to wait for
* @cmd1: the command will be executed before sleep
* @cmd2: the command will be executed after sleep
*
* The process is put to sleep (TASK_UNINTERRUPTIBLE) until the
* @condition evaluates to true. The @condition is checked each time
- * the waitqueue @wq is woken up.
+ * the waitqueue @wq_head is woken up.
*
* wake_up() has to be called after changing any variable that could
* change the result of the wait condition.
*/
-#define wait_event_cmd(wq, condition, cmd1, cmd2) \
-do { \
- if (condition) \
- break; \
- __wait_event_cmd(wq, condition, cmd1, cmd2); \
+#define wait_event_cmd(wq_head, condition, cmd1, cmd2) \
+do { \
+ if (condition) \
+ break; \
+ __wait_event_cmd(wq_head, condition, cmd1, cmd2); \
} while (0)
-#define __wait_event_interruptible(wq, condition) \
- ___wait_event(wq, condition, TASK_INTERRUPTIBLE, 0, 0, \
+#define __wait_event_interruptible(wq_head, condition) \
+ ___wait_event(wq_head, condition, TASK_INTERRUPTIBLE, 0, 0, \
schedule())
/**
* wait_event_interruptible - sleep until a condition gets true
- * @wq: the waitqueue to wait on
+ * @wq_head: the waitqueue to wait on
* @condition: a C expression for the event to wait for
*
* The process is put to sleep (TASK_INTERRUPTIBLE) until the
* @condition evaluates to true or a signal is received.
- * The @condition is checked each time the waitqueue @wq is woken up.
+ * The @condition is checked each time the waitqueue @wq_head is woken up.
*
* wake_up() has to be called after changing any variable that could
* change the result of the wait condition.
* The function will return -ERESTARTSYS if it was interrupted by a
* signal and 0 if @condition evaluated to true.
*/
-#define wait_event_interruptible(wq, condition) \
-({ \
- int __ret = 0; \
- might_sleep(); \
- if (!(condition)) \
- __ret = __wait_event_interruptible(wq, condition); \
- __ret; \
+#define wait_event_interruptible(wq_head, condition) \
+({ \
+ int __ret = 0; \
+ might_sleep(); \
+ if (!(condition)) \
+ __ret = __wait_event_interruptible(wq_head, condition); \
+ __ret; \
})
-#define __wait_event_interruptible_timeout(wq, condition, timeout) \
- ___wait_event(wq, ___wait_cond_timeout(condition), \
- TASK_INTERRUPTIBLE, 0, timeout, \
+#define __wait_event_interruptible_timeout(wq_head, condition, timeout) \
+ ___wait_event(wq_head, ___wait_cond_timeout(condition), \
+ TASK_INTERRUPTIBLE, 0, timeout, \
__ret = schedule_timeout(__ret))
/**
* wait_event_interruptible_timeout - sleep until a condition gets true or a timeout elapses
- * @wq: the waitqueue to wait on
+ * @wq_head: the waitqueue to wait on
* @condition: a C expression for the event to wait for
* @timeout: timeout, in jiffies
*
* The process is put to sleep (TASK_INTERRUPTIBLE) until the
* @condition evaluates to true or a signal is received.
- * The @condition is checked each time the waitqueue @wq is woken up.
+ * The @condition is checked each time the waitqueue @wq_head is woken up.
*
* wake_up() has to be called after changing any variable that could
* change the result of the wait condition.
* to %true before the @timeout elapsed, or -%ERESTARTSYS if it was
* interrupted by a signal.
*/
-#define wait_event_interruptible_timeout(wq, condition, timeout) \
-({ \
- long __ret = timeout; \
- might_sleep(); \
- if (!___wait_cond_timeout(condition)) \
- __ret = __wait_event_interruptible_timeout(wq, \
- condition, timeout); \
- __ret; \
+#define wait_event_interruptible_timeout(wq_head, condition, timeout) \
+({ \
+ long __ret = timeout; \
+ might_sleep(); \
+ if (!___wait_cond_timeout(condition)) \
+ __ret = __wait_event_interruptible_timeout(wq_head, \
+ condition, timeout); \
+ __ret; \
})
-#define __wait_event_hrtimeout(wq, condition, timeout, state) \
-({ \
- int __ret = 0; \
- struct hrtimer_sleeper __t; \
- \
- hrtimer_init_on_stack(&__t.timer, CLOCK_MONOTONIC, \
- HRTIMER_MODE_REL); \
- hrtimer_init_sleeper(&__t, current); \
- if ((timeout) != KTIME_MAX) \
- hrtimer_start_range_ns(&__t.timer, timeout, \
- current->timer_slack_ns, \
- HRTIMER_MODE_REL); \
- \
- __ret = ___wait_event(wq, condition, state, 0, 0, \
- if (!__t.task) { \
- __ret = -ETIME; \
- break; \
- } \
- schedule()); \
- \
- hrtimer_cancel(&__t.timer); \
- destroy_hrtimer_on_stack(&__t.timer); \
- __ret; \
+#define __wait_event_hrtimeout(wq_head, condition, timeout, state) \
+({ \
+ int __ret = 0; \
+ struct hrtimer_sleeper __t; \
+ \
+ hrtimer_init_on_stack(&__t.timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL); \
+ hrtimer_init_sleeper(&__t, current); \
+ if ((timeout) != KTIME_MAX) \
+ hrtimer_start_range_ns(&__t.timer, timeout, \
+ current->timer_slack_ns, \
+ HRTIMER_MODE_REL); \
+ \
+ __ret = ___wait_event(wq_head, condition, state, 0, 0, \
+ if (!__t.task) { \
+ __ret = -ETIME; \
+ break; \
+ } \
+ schedule()); \
+ \
+ hrtimer_cancel(&__t.timer); \
+ destroy_hrtimer_on_stack(&__t.timer); \
+ __ret; \
})
/**
* wait_event_hrtimeout - sleep until a condition gets true or a timeout elapses
- * @wq: the waitqueue to wait on
+ * @wq_head: the waitqueue to wait on
* @condition: a C expression for the event to wait for
* @timeout: timeout, as a ktime_t
*
* The process is put to sleep (TASK_UNINTERRUPTIBLE) until the
* @condition evaluates to true or a signal is received.
- * The @condition is checked each time the waitqueue @wq is woken up.
+ * The @condition is checked each time the waitqueue @wq_head is woken up.
*
* wake_up() has to be called after changing any variable that could
* change the result of the wait condition.
* The function returns 0 if @condition became true, or -ETIME if the timeout
* elapsed.
*/
-#define wait_event_hrtimeout(wq, condition, timeout) \
-({ \
- int __ret = 0; \
- might_sleep(); \
- if (!(condition)) \
- __ret = __wait_event_hrtimeout(wq, condition, timeout, \
- TASK_UNINTERRUPTIBLE); \
- __ret; \
+#define wait_event_hrtimeout(wq_head, condition, timeout) \
+({ \
+ int __ret = 0; \
+ might_sleep(); \
+ if (!(condition)) \
+ __ret = __wait_event_hrtimeout(wq_head, condition, timeout, \
+ TASK_UNINTERRUPTIBLE); \
+ __ret; \
})
/**
* wait_event_interruptible_hrtimeout - sleep until a condition gets true or a timeout elapses
- * @wq: the waitqueue to wait on
+ * @wq_head: the waitqueue to wait on
* @condition: a C expression for the event to wait for
* @timeout: timeout, as a ktime_t
*
* The process is put to sleep (TASK_INTERRUPTIBLE) until the
* @condition evaluates to true or a signal is received.
- * The @condition is checked each time the waitqueue @wq is woken up.
+ * The @condition is checked each time the waitqueue @wq_head is woken up.
*
* wake_up() has to be called after changing any variable that could
* change the result of the wait condition.
* The function returns 0 if @condition became true, -ERESTARTSYS if it was
* interrupted by a signal, or -ETIME if the timeout elapsed.
*/
-#define wait_event_interruptible_hrtimeout(wq, condition, timeout) \
-({ \
- long __ret = 0; \
- might_sleep(); \
- if (!(condition)) \
- __ret = __wait_event_hrtimeout(wq, condition, timeout, \
- TASK_INTERRUPTIBLE); \
- __ret; \
+#define wait_event_interruptible_hrtimeout(wq, condition, timeout) \
+({ \
+ long __ret = 0; \
+ might_sleep(); \
+ if (!(condition)) \
+ __ret = __wait_event_hrtimeout(wq, condition, timeout, \
+ TASK_INTERRUPTIBLE); \
+ __ret; \
})
-#define __wait_event_interruptible_exclusive(wq, condition) \
- ___wait_event(wq, condition, TASK_INTERRUPTIBLE, 1, 0, \
+#define __wait_event_interruptible_exclusive(wq, condition) \
+ ___wait_event(wq, condition, TASK_INTERRUPTIBLE, 1, 0, \
schedule())
-#define wait_event_interruptible_exclusive(wq, condition) \
-({ \
- int __ret = 0; \
- might_sleep(); \
- if (!(condition)) \
- __ret = __wait_event_interruptible_exclusive(wq, condition);\
- __ret; \
+#define wait_event_interruptible_exclusive(wq, condition) \
+({ \
+ int __ret = 0; \
+ might_sleep(); \
+ if (!(condition)) \
+ __ret = __wait_event_interruptible_exclusive(wq, condition); \
+ __ret; \
})
-#define __wait_event_killable_exclusive(wq, condition) \
- ___wait_event(wq, condition, TASK_KILLABLE, 1, 0, \
+#define __wait_event_killable_exclusive(wq, condition) \
+ ___wait_event(wq, condition, TASK_KILLABLE, 1, 0, \
schedule())
-#define wait_event_killable_exclusive(wq, condition) \
-({ \
- int __ret = 0; \
- might_sleep(); \
- if (!(condition)) \
- __ret = __wait_event_killable_exclusive(wq, condition); \
- __ret; \
+#define wait_event_killable_exclusive(wq, condition) \
+({ \
+ int __ret = 0; \
+ might_sleep(); \
+ if (!(condition)) \
+ __ret = __wait_event_killable_exclusive(wq, condition); \
+ __ret; \
})
-#define __wait_event_freezable_exclusive(wq, condition) \
- ___wait_event(wq, condition, TASK_INTERRUPTIBLE, 1, 0, \
+#define __wait_event_freezable_exclusive(wq, condition) \
+ ___wait_event(wq, condition, TASK_INTERRUPTIBLE, 1, 0, \
schedule(); try_to_freeze())
-#define wait_event_freezable_exclusive(wq, condition) \
-({ \
- int __ret = 0; \
- might_sleep(); \
- if (!(condition)) \
- __ret = __wait_event_freezable_exclusive(wq, condition);\
- __ret; \
+#define wait_event_freezable_exclusive(wq, condition) \
+({ \
+ int __ret = 0; \
+ might_sleep(); \
+ if (!(condition)) \
+ __ret = __wait_event_freezable_exclusive(wq, condition); \
+ __ret; \
})
-extern int do_wait_intr(wait_queue_head_t *, wait_queue_t *);
-extern int do_wait_intr_irq(wait_queue_head_t *, wait_queue_t *);
-
-#define __wait_event_interruptible_locked(wq, condition, exclusive, fn) \
-({ \
- int __ret; \
- DEFINE_WAIT(__wait); \
- if (exclusive) \
- __wait.flags |= WQ_FLAG_EXCLUSIVE; \
- do { \
- __ret = fn(&(wq), &__wait); \
- if (__ret) \
- break; \
- } while (!(condition)); \
- __remove_wait_queue(&(wq), &__wait); \
- __set_current_state(TASK_RUNNING); \
- __ret; \
+extern int do_wait_intr(wait_queue_head_t *, wait_queue_entry_t *);
+extern int do_wait_intr_irq(wait_queue_head_t *, wait_queue_entry_t *);
+
+#define __wait_event_interruptible_locked(wq, condition, exclusive, fn) \
+({ \
+ int __ret; \
+ DEFINE_WAIT(__wait); \
+ if (exclusive) \
+ __wait.flags |= WQ_FLAG_EXCLUSIVE; \
+ do { \
+ __ret = fn(&(wq), &__wait); \
+ if (__ret) \
+ break; \
+ } while (!(condition)); \
+ __remove_wait_queue(&(wq), &__wait); \
+ __set_current_state(TASK_RUNNING); \
+ __ret; \
})
* The function will return -ERESTARTSYS if it was interrupted by a
* signal and 0 if @condition evaluated to true.
*/
-#define wait_event_interruptible_locked(wq, condition) \
- ((condition) \
+#define wait_event_interruptible_locked(wq, condition) \
+ ((condition) \
? 0 : __wait_event_interruptible_locked(wq, condition, 0, do_wait_intr))
/**
* The function will return -ERESTARTSYS if it was interrupted by a
* signal and 0 if @condition evaluated to true.
*/
-#define wait_event_interruptible_locked_irq(wq, condition) \
- ((condition) \
+#define wait_event_interruptible_locked_irq(wq, condition) \
+ ((condition) \
? 0 : __wait_event_interruptible_locked(wq, condition, 0, do_wait_intr_irq))
/**
* The function will return -ERESTARTSYS if it was interrupted by a
* signal and 0 if @condition evaluated to true.
*/
-#define wait_event_interruptible_exclusive_locked(wq, condition) \
- ((condition) \
+#define wait_event_interruptible_exclusive_locked(wq, condition) \
+ ((condition) \
? 0 : __wait_event_interruptible_locked(wq, condition, 1, do_wait_intr))
/**
* The function will return -ERESTARTSYS if it was interrupted by a
* signal and 0 if @condition evaluated to true.
*/
-#define wait_event_interruptible_exclusive_locked_irq(wq, condition) \
- ((condition) \
+#define wait_event_interruptible_exclusive_locked_irq(wq, condition) \
+ ((condition) \
? 0 : __wait_event_interruptible_locked(wq, condition, 1, do_wait_intr_irq))
-#define __wait_event_killable(wq, condition) \
+#define __wait_event_killable(wq, condition) \
___wait_event(wq, condition, TASK_KILLABLE, 0, 0, schedule())
/**
* The function will return -ERESTARTSYS if it was interrupted by a
* signal and 0 if @condition evaluated to true.
*/
-#define wait_event_killable(wq, condition) \
-({ \
- int __ret = 0; \
- might_sleep(); \
- if (!(condition)) \
- __ret = __wait_event_killable(wq, condition); \
- __ret; \
+#define wait_event_killable(wq_head, condition) \
+({ \
+ int __ret = 0; \
+ might_sleep(); \
+ if (!(condition)) \
+ __ret = __wait_event_killable(wq_head, condition); \
+ __ret; \
})
-#define __wait_event_lock_irq(wq, condition, lock, cmd) \
- (void)___wait_event(wq, condition, TASK_UNINTERRUPTIBLE, 0, 0, \
- spin_unlock_irq(&lock); \
- cmd; \
- schedule(); \
+#define __wait_event_lock_irq(wq_head, condition, lock, cmd) \
+ (void)___wait_event(wq_head, condition, TASK_UNINTERRUPTIBLE, 0, 0, \
+ spin_unlock_irq(&lock); \
+ cmd; \
+ schedule(); \
spin_lock_irq(&lock))
/**
* condition is checked under the lock. This
* is expected to be called with the lock
* taken.
- * @wq: the waitqueue to wait on
+ * @wq_head: the waitqueue to wait on
* @condition: a C expression for the event to wait for
* @lock: a locked spinlock_t, which will be released before cmd
* and schedule() and reacquired afterwards.
*
* The process is put to sleep (TASK_UNINTERRUPTIBLE) until the
* @condition evaluates to true. The @condition is checked each time
- * the waitqueue @wq is woken up.
+ * the waitqueue @wq_head is woken up.
*
* wake_up() has to be called after changing any variable that could
* change the result of the wait condition.
* dropped before invoking the cmd and going to sleep and is reacquired
* afterwards.
*/
-#define wait_event_lock_irq_cmd(wq, condition, lock, cmd) \
-do { \
- if (condition) \
- break; \
- __wait_event_lock_irq(wq, condition, lock, cmd); \
+#define wait_event_lock_irq_cmd(wq_head, condition, lock, cmd) \
+do { \
+ if (condition) \
+ break; \
+ __wait_event_lock_irq(wq_head, condition, lock, cmd); \
} while (0)
/**
* condition is checked under the lock. This
* is expected to be called with the lock
* taken.
- * @wq: the waitqueue to wait on
+ * @wq_head: the waitqueue to wait on
* @condition: a C expression for the event to wait for
* @lock: a locked spinlock_t, which will be released before schedule()
* and reacquired afterwards.
*
* The process is put to sleep (TASK_UNINTERRUPTIBLE) until the
* @condition evaluates to true. The @condition is checked each time
- * the waitqueue @wq is woken up.
+ * the waitqueue @wq_head is woken up.
*
* wake_up() has to be called after changing any variable that could
* change the result of the wait condition.
* This is supposed to be called while holding the lock. The lock is
* dropped before going to sleep and is reacquired afterwards.
*/
-#define wait_event_lock_irq(wq, condition, lock) \
-do { \
- if (condition) \
- break; \
- __wait_event_lock_irq(wq, condition, lock, ); \
+#define wait_event_lock_irq(wq_head, condition, lock) \
+do { \
+ if (condition) \
+ break; \
+ __wait_event_lock_irq(wq_head, condition, lock, ); \
} while (0)
-#define __wait_event_interruptible_lock_irq(wq, condition, lock, cmd) \
- ___wait_event(wq, condition, TASK_INTERRUPTIBLE, 0, 0, \
- spin_unlock_irq(&lock); \
- cmd; \
- schedule(); \
+#define __wait_event_interruptible_lock_irq(wq_head, condition, lock, cmd) \
+ ___wait_event(wq_head, condition, TASK_INTERRUPTIBLE, 0, 0, \
+ spin_unlock_irq(&lock); \
+ cmd; \
+ schedule(); \
spin_lock_irq(&lock))
/**
* wait_event_interruptible_lock_irq_cmd - sleep until a condition gets true.
* The condition is checked under the lock. This is expected to
* be called with the lock taken.
- * @wq: the waitqueue to wait on
+ * @wq_head: the waitqueue to wait on
* @condition: a C expression for the event to wait for
* @lock: a locked spinlock_t, which will be released before cmd and
* schedule() and reacquired afterwards.
*
* The process is put to sleep (TASK_INTERRUPTIBLE) until the
* @condition evaluates to true or a signal is received. The @condition is
- * checked each time the waitqueue @wq is woken up.
+ * checked each time the waitqueue @wq_head is woken up.
*
* wake_up() has to be called after changing any variable that could
* change the result of the wait condition.
* The macro will return -ERESTARTSYS if it was interrupted by a signal
* and 0 if @condition evaluated to true.
*/
-#define wait_event_interruptible_lock_irq_cmd(wq, condition, lock, cmd) \
-({ \
- int __ret = 0; \
- if (!(condition)) \
- __ret = __wait_event_interruptible_lock_irq(wq, \
- condition, lock, cmd); \
- __ret; \
+#define wait_event_interruptible_lock_irq_cmd(wq_head, condition, lock, cmd) \
+({ \
+ int __ret = 0; \
+ if (!(condition)) \
+ __ret = __wait_event_interruptible_lock_irq(wq_head, \
+ condition, lock, cmd); \
+ __ret; \
})
/**
* wait_event_interruptible_lock_irq - sleep until a condition gets true.
* The condition is checked under the lock. This is expected
* to be called with the lock taken.
- * @wq: the waitqueue to wait on
+ * @wq_head: the waitqueue to wait on
* @condition: a C expression for the event to wait for
* @lock: a locked spinlock_t, which will be released before schedule()
* and reacquired afterwards.
*
* The process is put to sleep (TASK_INTERRUPTIBLE) until the
* @condition evaluates to true or signal is received. The @condition is
- * checked each time the waitqueue @wq is woken up.
+ * checked each time the waitqueue @wq_head is woken up.
*
* wake_up() has to be called after changing any variable that could
* change the result of the wait condition.
* The macro will return -ERESTARTSYS if it was interrupted by a signal
* and 0 if @condition evaluated to true.
*/
-#define wait_event_interruptible_lock_irq(wq, condition, lock) \
-({ \
- int __ret = 0; \
- if (!(condition)) \
- __ret = __wait_event_interruptible_lock_irq(wq, \
- condition, lock,); \
- __ret; \
+#define wait_event_interruptible_lock_irq(wq_head, condition, lock) \
+({ \
+ int __ret = 0; \
+ if (!(condition)) \
+ __ret = __wait_event_interruptible_lock_irq(wq_head, \
+ condition, lock,); \
+ __ret; \
})
-#define __wait_event_interruptible_lock_irq_timeout(wq, condition, \
- lock, timeout) \
- ___wait_event(wq, ___wait_cond_timeout(condition), \
- TASK_INTERRUPTIBLE, 0, timeout, \
- spin_unlock_irq(&lock); \
- __ret = schedule_timeout(__ret); \
+#define __wait_event_interruptible_lock_irq_timeout(wq_head, condition, \
+ lock, timeout) \
+ ___wait_event(wq_head, ___wait_cond_timeout(condition), \
+ TASK_INTERRUPTIBLE, 0, timeout, \
+ spin_unlock_irq(&lock); \
+ __ret = schedule_timeout(__ret); \
spin_lock_irq(&lock));
/**
* wait_event_interruptible_lock_irq_timeout - sleep until a condition gets
* true or a timeout elapses. The condition is checked under
* the lock. This is expected to be called with the lock taken.
- * @wq: the waitqueue to wait on
+ * @wq_head: the waitqueue to wait on
* @condition: a C expression for the event to wait for
* @lock: a locked spinlock_t, which will be released before schedule()
* and reacquired afterwards.
*
* The process is put to sleep (TASK_INTERRUPTIBLE) until the
* @condition evaluates to true or signal is received. The @condition is
- * checked each time the waitqueue @wq is woken up.
+ * checked each time the waitqueue @wq_head is woken up.
*
* wake_up() has to be called after changing any variable that could
* change the result of the wait condition.
* was interrupted by a signal, and the remaining jiffies otherwise
* if the condition evaluated to true before the timeout elapsed.
*/
-#define wait_event_interruptible_lock_irq_timeout(wq, condition, lock, \
- timeout) \
-({ \
- long __ret = timeout; \
- if (!___wait_cond_timeout(condition)) \
- __ret = __wait_event_interruptible_lock_irq_timeout( \
- wq, condition, lock, timeout); \
- __ret; \
+#define wait_event_interruptible_lock_irq_timeout(wq_head, condition, lock, \
+ timeout) \
+({ \
+ long __ret = timeout; \
+ if (!___wait_cond_timeout(condition)) \
+ __ret = __wait_event_interruptible_lock_irq_timeout( \
+ wq_head, condition, lock, timeout); \
+ __ret; \
})
/*
* Waitqueues which are removed from the waitqueue_head at wakeup time
*/
-void prepare_to_wait(wait_queue_head_t *q, wait_queue_t *wait, int state);
-void prepare_to_wait_exclusive(wait_queue_head_t *q, wait_queue_t *wait, int state);
-long prepare_to_wait_event(wait_queue_head_t *q, wait_queue_t *wait, int state);
-void finish_wait(wait_queue_head_t *q, wait_queue_t *wait);
-long wait_woken(wait_queue_t *wait, unsigned mode, long timeout);
-int woken_wake_function(wait_queue_t *wait, unsigned mode, int sync, void *key);
-int autoremove_wake_function(wait_queue_t *wait, unsigned mode, int sync, void *key);
-int wake_bit_function(wait_queue_t *wait, unsigned mode, int sync, void *key);
-
-#define DEFINE_WAIT_FUNC(name, function) \
- wait_queue_t name = { \
- .private = current, \
- .func = function, \
- .task_list = LIST_HEAD_INIT((name).task_list), \
+void prepare_to_wait(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry, int state);
+void prepare_to_wait_exclusive(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry, int state);
+long prepare_to_wait_event(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry, int state);
+void finish_wait(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry);
+long wait_woken(struct wait_queue_entry *wq_entry, unsigned mode, long timeout);
+int woken_wake_function(struct wait_queue_entry *wq_entry, unsigned mode, int sync, void *key);
+int autoremove_wake_function(struct wait_queue_entry *wq_entry, unsigned mode, int sync, void *key);
+
+#define DEFINE_WAIT_FUNC(name, function) \
+ struct wait_queue_entry name = { \
+ .private = current, \
+ .func = function, \
+ .entry = LIST_HEAD_INIT((name).entry), \
}
#define DEFINE_WAIT(name) DEFINE_WAIT_FUNC(name, autoremove_wake_function)
-#define DEFINE_WAIT_BIT(name, word, bit) \
- struct wait_bit_queue name = { \
- .key = __WAIT_BIT_KEY_INITIALIZER(word, bit), \
- .wait = { \
- .private = current, \
- .func = wake_bit_function, \
- .task_list = \
- LIST_HEAD_INIT((name).wait.task_list), \
- }, \
- }
-
-#define init_wait(wait) \
- do { \
- (wait)->private = current; \
- (wait)->func = autoremove_wake_function; \
- INIT_LIST_HEAD(&(wait)->task_list); \
- (wait)->flags = 0; \
+#define init_wait(wait) \
+ do { \
+ (wait)->private = current; \
+ (wait)->func = autoremove_wake_function; \
+ INIT_LIST_HEAD(&(wait)->entry); \
+ (wait)->flags = 0; \
} while (0)
-
-extern int bit_wait(struct wait_bit_key *, int);
-extern int bit_wait_io(struct wait_bit_key *, int);
-extern int bit_wait_timeout(struct wait_bit_key *, int);
-extern int bit_wait_io_timeout(struct wait_bit_key *, int);
-
-/**
- * wait_on_bit - wait for a bit to be cleared
- * @word: the word being waited on, a kernel virtual address
- * @bit: the bit of the word being waited on
- * @mode: the task state to sleep in
- *
- * There is a standard hashed waitqueue table for generic use. This
- * is the part of the hashtable's accessor API that waits on a bit.
- * For instance, if one were to have waiters on a bitflag, one would
- * call wait_on_bit() in threads waiting for the bit to clear.
- * One uses wait_on_bit() where one is waiting for the bit to clear,
- * but has no intention of setting it.
- * Returned value will be zero if the bit was cleared, or non-zero
- * if the process received a signal and the mode permitted wakeup
- * on that signal.
- */
-static inline int
-wait_on_bit(unsigned long *word, int bit, unsigned mode)
-{
- might_sleep();
- if (!test_bit(bit, word))
- return 0;
- return out_of_line_wait_on_bit(word, bit,
- bit_wait,
- mode);
-}
-
-/**
- * wait_on_bit_io - wait for a bit to be cleared
- * @word: the word being waited on, a kernel virtual address
- * @bit: the bit of the word being waited on
- * @mode: the task state to sleep in
- *
- * Use the standard hashed waitqueue table to wait for a bit
- * to be cleared. This is similar to wait_on_bit(), but calls
- * io_schedule() instead of schedule() for the actual waiting.
- *
- * Returned value will be zero if the bit was cleared, or non-zero
- * if the process received a signal and the mode permitted wakeup
- * on that signal.
- */
-static inline int
-wait_on_bit_io(unsigned long *word, int bit, unsigned mode)
-{
- might_sleep();
- if (!test_bit(bit, word))
- return 0;
- return out_of_line_wait_on_bit(word, bit,
- bit_wait_io,
- mode);
-}
-
-/**
- * wait_on_bit_timeout - wait for a bit to be cleared or a timeout elapses
- * @word: the word being waited on, a kernel virtual address
- * @bit: the bit of the word being waited on
- * @mode: the task state to sleep in
- * @timeout: timeout, in jiffies
- *
- * Use the standard hashed waitqueue table to wait for a bit
- * to be cleared. This is similar to wait_on_bit(), except also takes a
- * timeout parameter.
- *
- * Returned value will be zero if the bit was cleared before the
- * @timeout elapsed, or non-zero if the @timeout elapsed or process
- * received a signal and the mode permitted wakeup on that signal.
- */
-static inline int
-wait_on_bit_timeout(unsigned long *word, int bit, unsigned mode,
- unsigned long timeout)
-{
- might_sleep();
- if (!test_bit(bit, word))
- return 0;
- return out_of_line_wait_on_bit_timeout(word, bit,
- bit_wait_timeout,
- mode, timeout);
-}
-
-/**
- * wait_on_bit_action - wait for a bit to be cleared
- * @word: the word being waited on, a kernel virtual address
- * @bit: the bit of the word being waited on
- * @action: the function used to sleep, which may take special actions
- * @mode: the task state to sleep in
- *
- * Use the standard hashed waitqueue table to wait for a bit
- * to be cleared, and allow the waiting action to be specified.
- * This is like wait_on_bit() but allows fine control of how the waiting
- * is done.
- *
- * Returned value will be zero if the bit was cleared, or non-zero
- * if the process received a signal and the mode permitted wakeup
- * on that signal.
- */
-static inline int
-wait_on_bit_action(unsigned long *word, int bit, wait_bit_action_f *action,
- unsigned mode)
-{
- might_sleep();
- if (!test_bit(bit, word))
- return 0;
- return out_of_line_wait_on_bit(word, bit, action, mode);
-}
-
-/**
- * wait_on_bit_lock - wait for a bit to be cleared, when wanting to set it
- * @word: the word being waited on, a kernel virtual address
- * @bit: the bit of the word being waited on
- * @mode: the task state to sleep in
- *
- * There is a standard hashed waitqueue table for generic use. This
- * is the part of the hashtable's accessor API that waits on a bit
- * when one intends to set it, for instance, trying to lock bitflags.
- * For instance, if one were to have waiters trying to set bitflag
- * and waiting for it to clear before setting it, one would call
- * wait_on_bit() in threads waiting to be able to set the bit.
- * One uses wait_on_bit_lock() where one is waiting for the bit to
- * clear with the intention of setting it, and when done, clearing it.
- *
- * Returns zero if the bit was (eventually) found to be clear and was
- * set. Returns non-zero if a signal was delivered to the process and
- * the @mode allows that signal to wake the process.
- */
-static inline int
-wait_on_bit_lock(unsigned long *word, int bit, unsigned mode)
-{
- might_sleep();
- if (!test_and_set_bit(bit, word))
- return 0;
- return out_of_line_wait_on_bit_lock(word, bit, bit_wait, mode);
-}
-
-/**
- * wait_on_bit_lock_io - wait for a bit to be cleared, when wanting to set it
- * @word: the word being waited on, a kernel virtual address
- * @bit: the bit of the word being waited on
- * @mode: the task state to sleep in
- *
- * Use the standard hashed waitqueue table to wait for a bit
- * to be cleared and then to atomically set it. This is similar
- * to wait_on_bit(), but calls io_schedule() instead of schedule()
- * for the actual waiting.
- *
- * Returns zero if the bit was (eventually) found to be clear and was
- * set. Returns non-zero if a signal was delivered to the process and
- * the @mode allows that signal to wake the process.
- */
-static inline int
-wait_on_bit_lock_io(unsigned long *word, int bit, unsigned mode)
-{
- might_sleep();
- if (!test_and_set_bit(bit, word))
- return 0;
- return out_of_line_wait_on_bit_lock(word, bit, bit_wait_io, mode);
-}
-
-/**
- * wait_on_bit_lock_action - wait for a bit to be cleared, when wanting to set it
- * @word: the word being waited on, a kernel virtual address
- * @bit: the bit of the word being waited on
- * @action: the function used to sleep, which may take special actions
- * @mode: the task state to sleep in
- *
- * Use the standard hashed waitqueue table to wait for a bit
- * to be cleared and then to set it, and allow the waiting action
- * to be specified.
- * This is like wait_on_bit() but allows fine control of how the waiting
- * is done.
- *
- * Returns zero if the bit was (eventually) found to be clear and was
- * set. Returns non-zero if a signal was delivered to the process and
- * the @mode allows that signal to wake the process.
- */
-static inline int
-wait_on_bit_lock_action(unsigned long *word, int bit, wait_bit_action_f *action,
- unsigned mode)
-{
- might_sleep();
- if (!test_and_set_bit(bit, word))
- return 0;
- return out_of_line_wait_on_bit_lock(word, bit, action, mode);
-}
-
-/**
- * wait_on_atomic_t - Wait for an atomic_t to become 0
- * @val: The atomic value being waited on, a kernel virtual address
- * @action: the function used to sleep, which may take special actions
- * @mode: the task state to sleep in
- *
- * Wait for an atomic_t to become 0. We abuse the bit-wait waitqueue table for
- * the purpose of getting a waitqueue, but we set the key to a bit number
- * outside of the target 'word'.
- */
-static inline
-int wait_on_atomic_t(atomic_t *val, int (*action)(atomic_t *), unsigned mode)
-{
- might_sleep();
- if (atomic_read(val) == 0)
- return 0;
- return out_of_line_wait_on_atomic_t(val, action, mode);
-}
-
#endif /* _LINUX_WAIT_H */
--- /dev/null
+#ifndef _LINUX_WAIT_BIT_H
+#define _LINUX_WAIT_BIT_H
+
+/*
+ * Linux wait-bit related types and methods:
+ */
+#include <linux/wait.h>
+
+struct wait_bit_key {
+ void *flags;
+ int bit_nr;
+#define WAIT_ATOMIC_T_BIT_NR -1
+ unsigned long timeout;
+};
+
+struct wait_bit_queue_entry {
+ struct wait_bit_key key;
+ struct wait_queue_entry wq_entry;
+};
+
+#define __WAIT_BIT_KEY_INITIALIZER(word, bit) \
+ { .flags = word, .bit_nr = bit, }
+
+#define __WAIT_ATOMIC_T_KEY_INITIALIZER(p) \
+ { .flags = p, .bit_nr = WAIT_ATOMIC_T_BIT_NR, }
+
+typedef int wait_bit_action_f(struct wait_bit_key *key, int mode);
+void __wake_up_bit(struct wait_queue_head *wq_head, void *word, int bit);
+int __wait_on_bit(struct wait_queue_head *wq_head, struct wait_bit_queue_entry *wbq_entry, wait_bit_action_f *action, unsigned int mode);
+int __wait_on_bit_lock(struct wait_queue_head *wq_head, struct wait_bit_queue_entry *wbq_entry, wait_bit_action_f *action, unsigned int mode);
+void wake_up_bit(void *word, int bit);
+void wake_up_atomic_t(atomic_t *p);
+int out_of_line_wait_on_bit(void *word, int, wait_bit_action_f *action, unsigned int mode);
+int out_of_line_wait_on_bit_timeout(void *word, int, wait_bit_action_f *action, unsigned int mode, unsigned long timeout);
+int out_of_line_wait_on_bit_lock(void *word, int, wait_bit_action_f *action, unsigned int mode);
+int out_of_line_wait_on_atomic_t(atomic_t *p, int (*)(atomic_t *), unsigned int mode);
+struct wait_queue_head *bit_waitqueue(void *word, int bit);
+extern void __init wait_bit_init(void);
+
+int wake_bit_function(struct wait_queue_entry *wq_entry, unsigned mode, int sync, void *key);
+
+#define DEFINE_WAIT_BIT(name, word, bit) \
+ struct wait_bit_queue_entry name = { \
+ .key = __WAIT_BIT_KEY_INITIALIZER(word, bit), \
+ .wq_entry = { \
+ .private = current, \
+ .func = wake_bit_function, \
+ .entry = \
+ LIST_HEAD_INIT((name).wq_entry.entry), \
+ }, \
+ }
+
+extern int bit_wait(struct wait_bit_key *key, int bit);
+extern int bit_wait_io(struct wait_bit_key *key, int bit);
+extern int bit_wait_timeout(struct wait_bit_key *key, int bit);
+extern int bit_wait_io_timeout(struct wait_bit_key *key, int bit);
+
+/**
+ * wait_on_bit - wait for a bit to be cleared
+ * @word: the word being waited on, a kernel virtual address
+ * @bit: the bit of the word being waited on
+ * @mode: the task state to sleep in
+ *
+ * There is a standard hashed waitqueue table for generic use. This
+ * is the part of the hashtable's accessor API that waits on a bit.
+ * For instance, if one were to have waiters on a bitflag, one would
+ * call wait_on_bit() in threads waiting for the bit to clear.
+ * One uses wait_on_bit() where one is waiting for the bit to clear,
+ * but has no intention of setting it.
+ * Returned value will be zero if the bit was cleared, or non-zero
+ * if the process received a signal and the mode permitted wakeup
+ * on that signal.
+ */
+static inline int
+wait_on_bit(unsigned long *word, int bit, unsigned mode)
+{
+ might_sleep();
+ if (!test_bit(bit, word))
+ return 0;
+ return out_of_line_wait_on_bit(word, bit,
+ bit_wait,
+ mode);
+}
+
+/**
+ * wait_on_bit_io - wait for a bit to be cleared
+ * @word: the word being waited on, a kernel virtual address
+ * @bit: the bit of the word being waited on
+ * @mode: the task state to sleep in
+ *
+ * Use the standard hashed waitqueue table to wait for a bit
+ * to be cleared. This is similar to wait_on_bit(), but calls
+ * io_schedule() instead of schedule() for the actual waiting.
+ *
+ * Returned value will be zero if the bit was cleared, or non-zero
+ * if the process received a signal and the mode permitted wakeup
+ * on that signal.
+ */
+static inline int
+wait_on_bit_io(unsigned long *word, int bit, unsigned mode)
+{
+ might_sleep();
+ if (!test_bit(bit, word))
+ return 0;
+ return out_of_line_wait_on_bit(word, bit,
+ bit_wait_io,
+ mode);
+}
+
+/**
+ * wait_on_bit_timeout - wait for a bit to be cleared or a timeout elapses
+ * @word: the word being waited on, a kernel virtual address
+ * @bit: the bit of the word being waited on
+ * @mode: the task state to sleep in
+ * @timeout: timeout, in jiffies
+ *
+ * Use the standard hashed waitqueue table to wait for a bit
+ * to be cleared. This is similar to wait_on_bit(), except also takes a
+ * timeout parameter.
+ *
+ * Returned value will be zero if the bit was cleared before the
+ * @timeout elapsed, or non-zero if the @timeout elapsed or process
+ * received a signal and the mode permitted wakeup on that signal.
+ */
+static inline int
+wait_on_bit_timeout(unsigned long *word, int bit, unsigned mode,
+ unsigned long timeout)
+{
+ might_sleep();
+ if (!test_bit(bit, word))
+ return 0;
+ return out_of_line_wait_on_bit_timeout(word, bit,
+ bit_wait_timeout,
+ mode, timeout);
+}
+
+/**
+ * wait_on_bit_action - wait for a bit to be cleared
+ * @word: the word being waited on, a kernel virtual address
+ * @bit: the bit of the word being waited on
+ * @action: the function used to sleep, which may take special actions
+ * @mode: the task state to sleep in
+ *
+ * Use the standard hashed waitqueue table to wait for a bit
+ * to be cleared, and allow the waiting action to be specified.
+ * This is like wait_on_bit() but allows fine control of how the waiting
+ * is done.
+ *
+ * Returned value will be zero if the bit was cleared, or non-zero
+ * if the process received a signal and the mode permitted wakeup
+ * on that signal.
+ */
+static inline int
+wait_on_bit_action(unsigned long *word, int bit, wait_bit_action_f *action,
+ unsigned mode)
+{
+ might_sleep();
+ if (!test_bit(bit, word))
+ return 0;
+ return out_of_line_wait_on_bit(word, bit, action, mode);
+}
+
+/**
+ * wait_on_bit_lock - wait for a bit to be cleared, when wanting to set it
+ * @word: the word being waited on, a kernel virtual address
+ * @bit: the bit of the word being waited on
+ * @mode: the task state to sleep in
+ *
+ * There is a standard hashed waitqueue table for generic use. This
+ * is the part of the hashtable's accessor API that waits on a bit
+ * when one intends to set it, for instance, trying to lock bitflags.
+ * For instance, if one were to have waiters trying to set bitflag
+ * and waiting for it to clear before setting it, one would call
+ * wait_on_bit() in threads waiting to be able to set the bit.
+ * One uses wait_on_bit_lock() where one is waiting for the bit to
+ * clear with the intention of setting it, and when done, clearing it.
+ *
+ * Returns zero if the bit was (eventually) found to be clear and was
+ * set. Returns non-zero if a signal was delivered to the process and
+ * the @mode allows that signal to wake the process.
+ */
+static inline int
+wait_on_bit_lock(unsigned long *word, int bit, unsigned mode)
+{
+ might_sleep();
+ if (!test_and_set_bit(bit, word))
+ return 0;
+ return out_of_line_wait_on_bit_lock(word, bit, bit_wait, mode);
+}
+
+/**
+ * wait_on_bit_lock_io - wait for a bit to be cleared, when wanting to set it
+ * @word: the word being waited on, a kernel virtual address
+ * @bit: the bit of the word being waited on
+ * @mode: the task state to sleep in
+ *
+ * Use the standard hashed waitqueue table to wait for a bit
+ * to be cleared and then to atomically set it. This is similar
+ * to wait_on_bit(), but calls io_schedule() instead of schedule()
+ * for the actual waiting.
+ *
+ * Returns zero if the bit was (eventually) found to be clear and was
+ * set. Returns non-zero if a signal was delivered to the process and
+ * the @mode allows that signal to wake the process.
+ */
+static inline int
+wait_on_bit_lock_io(unsigned long *word, int bit, unsigned mode)
+{
+ might_sleep();
+ if (!test_and_set_bit(bit, word))
+ return 0;
+ return out_of_line_wait_on_bit_lock(word, bit, bit_wait_io, mode);
+}
+
+/**
+ * wait_on_bit_lock_action - wait for a bit to be cleared, when wanting to set it
+ * @word: the word being waited on, a kernel virtual address
+ * @bit: the bit of the word being waited on
+ * @action: the function used to sleep, which may take special actions
+ * @mode: the task state to sleep in
+ *
+ * Use the standard hashed waitqueue table to wait for a bit
+ * to be cleared and then to set it, and allow the waiting action
+ * to be specified.
+ * This is like wait_on_bit() but allows fine control of how the waiting
+ * is done.
+ *
+ * Returns zero if the bit was (eventually) found to be clear and was
+ * set. Returns non-zero if a signal was delivered to the process and
+ * the @mode allows that signal to wake the process.
+ */
+static inline int
+wait_on_bit_lock_action(unsigned long *word, int bit, wait_bit_action_f *action,
+ unsigned mode)
+{
+ might_sleep();
+ if (!test_and_set_bit(bit, word))
+ return 0;
+ return out_of_line_wait_on_bit_lock(word, bit, action, mode);
+}
+
+/**
+ * wait_on_atomic_t - Wait for an atomic_t to become 0
+ * @val: The atomic value being waited on, a kernel virtual address
+ * @action: the function used to sleep, which may take special actions
+ * @mode: the task state to sleep in
+ *
+ * Wait for an atomic_t to become 0. We abuse the bit-wait waitqueue table for
+ * the purpose of getting a waitqueue, but we set the key to a bit number
+ * outside of the target 'word'.
+ */
+static inline
+int wait_on_atomic_t(atomic_t *val, int (*action)(atomic_t *), unsigned mode)
+{
+ might_sleep();
+ if (atomic_read(val) == 0)
+ return 0;
+ return out_of_line_wait_on_atomic_t(val, action, mode);
+}
+
+#endif /* _LINUX_WAIT_BIT_H */
#define UNIX_GC_CANDIDATE 0
#define UNIX_GC_MAYBE_CYCLE 1
struct socket_wq peer_wq;
- wait_queue_t peer_wake;
+ wait_queue_entry_t peer_wake;
};
static inline struct unix_sock *unix_sk(const struct sock *sk)
* The purpose of the skwq_has_sleeper and sock_poll_wait is to wrap the memory
* barrier call. They were added due to the race found within the tcp code.
*
- * Consider following tcp code paths:
+ * Consider following tcp code paths::
*
- * CPU1 CPU2
- *
- * sys_select receive packet
+ * CPU1 CPU2
+ * sys_select receive packet
* ... ...
* __add_wait_queue update tp->rcv_nxt
* ... ...
* @tsflags: timestamping flags to use
* @tx_flags: completed with instructions for time stamping
*
- * Note : callers should take care of initial *tx_flags value (usually 0)
+ * Note: callers should take care of initial ``*tx_flags`` value (usually 0)
*/
static inline void sock_tx_timestamp(const struct sock *sk, __u16 tsflags,
__u8 *tx_flags)
struct net;
#ifdef CONFIG_WEXT_CORE
-int wext_handle_ioctl(struct net *net, struct ifreq *ifr, unsigned int cmd,
+int wext_handle_ioctl(struct net *net, struct iwreq *iwr, unsigned int cmd,
void __user *arg);
int compat_wext_handle_ioctl(struct net *net, unsigned int cmd,
unsigned long arg);
struct iw_statistics *get_wireless_stats(struct net_device *dev);
int call_commit_handler(struct net_device *dev);
#else
-static inline int wext_handle_ioctl(struct net *net, struct ifreq *ifr, unsigned int cmd,
+static inline int wext_handle_ioctl(struct net *net, struct iwreq *iwr, unsigned int cmd,
void __user *arg)
{
return -EINVAL;
}
#endif
-#ifdef CONFIG_XFRM_OFFLOAD
void __net_init xfrm_dev_init(void);
+
+#ifdef CONFIG_XFRM_OFFLOAD
int validate_xmit_xfrm(struct sk_buff *skb, netdev_features_t features);
int xfrm_dev_state_add(struct net *net, struct xfrm_state *x,
struct xfrm_user_offload *xuo);
}
}
#else
-static inline void __net_init xfrm_dev_init(void)
-{
-}
-
static inline int validate_xmit_xfrm(struct sk_buff *skb, netdev_features_t features)
{
return 0;
osd_req_done_fn *async_done;
void *async_private;
- int async_error;
+ blk_status_t async_error;
int req_errors;
};
extern void scsi_kunmap_atomic_sg(void *virt);
extern int scsi_init_io(struct scsi_cmnd *cmd);
+extern void scsi_initialize_rq(struct request *rq);
extern int scsi_dma_map(struct scsi_cmnd *cmd);
extern void scsi_dma_unmap(struct scsi_cmnd *cmd);
kfree(req->cmd);
}
-void scsi_req_init(struct request *);
+void scsi_req_init(struct scsi_request *req);
#endif /* _SCSI_SCSI_REQUEST_H */
--- /dev/null
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM fsi
+
+#if !defined(_TRACE_FSI_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_FSI_H
+
+#include <linux/tracepoint.h>
+
+TRACE_EVENT(fsi_master_read,
+ TP_PROTO(const struct fsi_master *master, int link, int id,
+ uint32_t addr, size_t size),
+ TP_ARGS(master, link, id, addr, size),
+ TP_STRUCT__entry(
+ __field(int, master_idx)
+ __field(int, link)
+ __field(int, id)
+ __field(__u32, addr)
+ __field(size_t, size)
+ ),
+ TP_fast_assign(
+ __entry->master_idx = master->idx;
+ __entry->link = link;
+ __entry->id = id;
+ __entry->addr = addr;
+ __entry->size = size;
+ ),
+ TP_printk("fsi%d:%02d:%02d %08x[%zd]",
+ __entry->master_idx,
+ __entry->link,
+ __entry->id,
+ __entry->addr,
+ __entry->size
+ )
+);
+
+TRACE_EVENT(fsi_master_write,
+ TP_PROTO(const struct fsi_master *master, int link, int id,
+ uint32_t addr, size_t size, const void *data),
+ TP_ARGS(master, link, id, addr, size, data),
+ TP_STRUCT__entry(
+ __field(int, master_idx)
+ __field(int, link)
+ __field(int, id)
+ __field(__u32, addr)
+ __field(size_t, size)
+ __field(__u32, data)
+ ),
+ TP_fast_assign(
+ __entry->master_idx = master->idx;
+ __entry->link = link;
+ __entry->id = id;
+ __entry->addr = addr;
+ __entry->size = size;
+ __entry->data = 0;
+ memcpy(&__entry->data, data, size);
+ ),
+ TP_printk("fsi%d:%02d:%02d %08x[%zd] <= {%*ph}",
+ __entry->master_idx,
+ __entry->link,
+ __entry->id,
+ __entry->addr,
+ __entry->size,
+ (int)__entry->size, &__entry->data
+ )
+);
+
+TRACE_EVENT(fsi_master_rw_result,
+ TP_PROTO(const struct fsi_master *master, int link, int id,
+ uint32_t addr, size_t size,
+ bool write, const void *data, int ret),
+ TP_ARGS(master, link, id, addr, size, write, data, ret),
+ TP_STRUCT__entry(
+ __field(int, master_idx)
+ __field(int, link)
+ __field(int, id)
+ __field(__u32, addr)
+ __field(size_t, size)
+ __field(bool, write)
+ __field(__u32, data)
+ __field(int, ret)
+ ),
+ TP_fast_assign(
+ __entry->master_idx = master->idx;
+ __entry->link = link;
+ __entry->id = id;
+ __entry->addr = addr;
+ __entry->size = size;
+ __entry->write = write;
+ __entry->data = 0;
+ __entry->ret = ret;
+ if (__entry->write || !__entry->ret)
+ memcpy(&__entry->data, data, size);
+ ),
+ TP_printk("fsi%d:%02d:%02d %08x[%zd] %s {%*ph} ret %d",
+ __entry->master_idx,
+ __entry->link,
+ __entry->id,
+ __entry->addr,
+ __entry->size,
+ __entry->write ? "<=" : "=>",
+ (int)__entry->size, &__entry->data,
+ __entry->ret
+ )
+);
+
+TRACE_EVENT(fsi_master_break,
+ TP_PROTO(const struct fsi_master *master, int link),
+ TP_ARGS(master, link),
+ TP_STRUCT__entry(
+ __field(int, master_idx)
+ __field(int, link)
+ ),
+ TP_fast_assign(
+ __entry->master_idx = master->idx;
+ __entry->link = link;
+ ),
+ TP_printk("fsi%d:%d",
+ __entry->master_idx,
+ __entry->link
+ )
+);
+
+
+#endif /* _TRACE_FSI_H */
+
+#include <trace/define_trace.h>
--- /dev/null
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM fsi_master_gpio
+
+#if !defined(_TRACE_FSI_MASTER_GPIO_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_FSI_MASTER_GPIO_H
+
+#include <linux/tracepoint.h>
+
+TRACE_EVENT(fsi_master_gpio_in,
+ TP_PROTO(const struct fsi_master_gpio *master, int bits, uint64_t msg),
+ TP_ARGS(master, bits, msg),
+ TP_STRUCT__entry(
+ __field(int, master_idx)
+ __field(int, bits)
+ __field(uint64_t, msg)
+ ),
+ TP_fast_assign(
+ __entry->master_idx = master->master.idx;
+ __entry->bits = bits;
+ __entry->msg = msg & ((1ull<<bits) - 1);
+ ),
+ TP_printk("fsi-gpio%d => %0*llx[%d]",
+ __entry->master_idx,
+ (__entry->bits + 3) / 4,
+ __entry->msg,
+ __entry->bits
+ )
+);
+
+TRACE_EVENT(fsi_master_gpio_out,
+ TP_PROTO(const struct fsi_master_gpio *master, int bits, uint64_t msg),
+ TP_ARGS(master, bits, msg),
+ TP_STRUCT__entry(
+ __field(int, master_idx)
+ __field(int, bits)
+ __field(uint64_t, msg)
+ ),
+ TP_fast_assign(
+ __entry->master_idx = master->master.idx;
+ __entry->bits = bits;
+ __entry->msg = msg & ((1ull<<bits) - 1);
+ ),
+ TP_printk("fsi-gpio%d <= %0*llx[%d]",
+ __entry->master_idx,
+ (__entry->bits + 3) / 4,
+ __entry->msg,
+ __entry->bits
+ )
+);
+
+TRACE_EVENT(fsi_master_gpio_break,
+ TP_PROTO(const struct fsi_master_gpio *master),
+ TP_ARGS(master),
+ TP_STRUCT__entry(
+ __field(int, master_idx)
+ ),
+ TP_fast_assign(
+ __entry->master_idx = master->master.idx;
+ ),
+ TP_printk("fsi-gpio%d ----break---",
+ __entry->master_idx
+ )
+);
+
+#endif /* _TRACE_FSI_MASTER_GPIO_H */
+
+#include <trace/define_trace.h>
* "OnlineQ": _rcu_barrier() found online CPU with callbacks.
* "OnlineNQ": _rcu_barrier() found online CPU, no callbacks.
* "IRQ": An rcu_barrier_callback() callback posted on remote CPU.
+ * "IRQNQ": An rcu_barrier_callback() callback found no callbacks.
* "CB": An rcu_barrier_callback() invoked a callback, not the last.
* "LastCB": An rcu_barrier_callback() invoked the last callback.
* "Inc2": _rcu_barrier() piggyback check counter incremented.
#include <linux/ktime.h>
#include <linux/tracepoint.h>
-DECLARE_EVENT_CLASS(spi_master,
+DECLARE_EVENT_CLASS(spi_controller,
- TP_PROTO(struct spi_master *master),
+ TP_PROTO(struct spi_controller *controller),
- TP_ARGS(master),
+ TP_ARGS(controller),
TP_STRUCT__entry(
__field( int, bus_num )
),
TP_fast_assign(
- __entry->bus_num = master->bus_num;
+ __entry->bus_num = controller->bus_num;
),
TP_printk("spi%d", (int)__entry->bus_num)
);
-DEFINE_EVENT(spi_master, spi_master_idle,
+DEFINE_EVENT(spi_controller, spi_controller_idle,
- TP_PROTO(struct spi_master *master),
+ TP_PROTO(struct spi_controller *controller),
- TP_ARGS(master)
+ TP_ARGS(controller)
);
-DEFINE_EVENT(spi_master, spi_master_busy,
+DEFINE_EVENT(spi_controller, spi_controller_busy,
- TP_PROTO(struct spi_master *master),
+ TP_PROTO(struct spi_controller *controller),
- TP_ARGS(master)
+ TP_ARGS(controller)
);
),
TP_fast_assign(
- __entry->bus_num = msg->spi->master->bus_num;
+ __entry->bus_num = msg->spi->controller->bus_num;
__entry->chip_select = msg->spi->chip_select;
__entry->msg = msg;
),
),
TP_fast_assign(
- __entry->bus_num = msg->spi->master->bus_num;
+ __entry->bus_num = msg->spi->controller->bus_num;
__entry->chip_select = msg->spi->chip_select;
__entry->msg = msg;
__entry->frame = msg->frame_length;
),
TP_fast_assign(
- __entry->bus_num = msg->spi->master->bus_num;
+ __entry->bus_num = msg->spi->controller->bus_num;
__entry->chip_select = msg->spi->chip_select;
__entry->xfer = xfer;
__entry->len = xfer->len;
#define TIOCGPKT _IOR('T', 0x38, int) /* Get packet mode state */
#define TIOCGPTLCK _IOR('T', 0x39, int) /* Get Pty lock state */
#define TIOCGEXCL _IOR('T', 0x40, int) /* Get exclusive mode state */
+#define TIOCGPTPEER _IOR('T', 0x41, int) /* Safely open the slave */
#define FIONCLEX 0x5450
#define FIOCLEX 0x5451
#define si_arch _sifields._sigsys._arch
#endif
-#ifndef __KERNEL__
+#ifdef __KERNEL__
+#define __SI_MASK 0xffff0000u
+#define __SI_KILL (0 << 16)
+#define __SI_TIMER (1 << 16)
+#define __SI_POLL (2 << 16)
+#define __SI_FAULT (3 << 16)
+#define __SI_CHLD (4 << 16)
+#define __SI_RT (5 << 16)
+#define __SI_MESGQ (6 << 16)
+#define __SI_SYS (7 << 16)
+#define __SI_CODE(T,N) ((T) | ((N) & 0xffff))
+#else /* __KERNEL__ */
#define __SI_KILL 0
#define __SI_TIMER 0
#define __SI_POLL 0
#define __SI_MESGQ 0
#define __SI_SYS 0
#define __SI_CODE(T,N) (N)
-#endif
+#endif /* __KERNEL__ */
/*
* si_code values
#define N_TXTADDR(x) (N_MAGIC(x) == QMAGIC ? PAGE_SIZE : 0)
#endif
-/* Address of data segment in memory after it is loaded.
- Note that it is up to you to define SEGMENT_SIZE
- on machines not listed here. */
-#if defined(vax) || defined(hp300) || defined(pyr)
-#define SEGMENT_SIZE page_size
-#endif
-#ifdef sony
-#define SEGMENT_SIZE 0x2000
-#endif /* Sony. */
-#ifdef is68k
-#define SEGMENT_SIZE 0x20000
-#endif
-#if defined(m68k) && defined(PORTAR)
-#define PAGE_SIZE 0x400
-#define SEGMENT_SIZE PAGE_SIZE
-#endif
-
-#ifdef linux
+/* Address of data segment in memory after it is loaded. */
#ifndef __KERNEL__
#include <unistd.h>
#endif
#endif
#endif
#endif
-#endif
#define _N_SEGMENT_ROUND(x) ALIGN(x, SEGMENT_SIZE)
unsigned int r_extern:1;
/* Four bits that aren't used, but when writing an object file
it is desirable to clear them. */
-#ifdef NS32K
- unsigned r_bsr:1;
- unsigned r_disp:1;
- unsigned r_pad:2;
-#else
unsigned int r_pad:4;
-#endif
};
#endif /* no N_RELOCATION_INFO_DECLARED. */
struct iocb {
/* these are internal to the kernel/libc. */
__u64 aio_data; /* data to be returned in event's data */
- __u32 PADDED(aio_key, aio_reserved1);
+ __u32 PADDED(aio_key, aio_rw_flags);
/* the kernel sets aio_key to the req # */
/* common fields */
#define AUTOFS_MIN_PROTO_VERSION AUTOFS_PROTO_VERSION
/*
- * The wait_queue_token (autofs_wqt_t) is part of a structure which is passed
+ * The wait_queue_entry_token (autofs_wqt_t) is part of a structure which is passed
* back to the kernel via ioctl from userspace. On architectures where 32- and
* 64-bit userspace binaries can be executed it's important that the size of
* autofs_wqt_t stays constant between 32- and 64-bit Linux kernels so that we
struct autofs_packet_missing {
struct autofs_packet_hdr hdr;
- autofs_wqt_t wait_queue_token;
+ autofs_wqt_t wait_queue_entry_token;
int len;
char name[NAME_MAX+1];
};
/* v4 multi expire (via pipe) */
struct autofs_packet_expire_multi {
struct autofs_packet_hdr hdr;
- autofs_wqt_t wait_queue_token;
+ autofs_wqt_t wait_queue_entry_token;
int len;
char name[NAME_MAX+1];
};
/* autofs v5 common packet struct */
struct autofs_v5_packet {
struct autofs_packet_hdr hdr;
- autofs_wqt_t wait_queue_token;
+ autofs_wqt_t wait_queue_entry_token;
__u32 dev;
__u64 ino;
__u32 uid;
#define DM_DEV_SET_GEOMETRY _IOWR(DM_IOCTL, DM_DEV_SET_GEOMETRY_CMD, struct dm_ioctl)
#define DM_VERSION_MAJOR 4
-#define DM_VERSION_MINOR 35
+#define DM_VERSION_MINOR 36
#define DM_VERSION_PATCHLEVEL 0
-#define DM_VERSION_EXTRA "-ioctl (2016-06-23)"
+#define DM_VERSION_EXTRA "-ioctl (2017-06-09)"
/* Status bits */
#define DM_READONLY_FLAG (1 << 0) /* In/Out */
#define F_SEAL_WRITE 0x0008 /* prevent writes */
/* (1U << 31) is reserved for signed error codes */
+/*
+ * Set/Get write life time hints. {GET,SET}_RW_HINT operate on the
+ * underlying inode, while {GET,SET}_FILE_RW_HINT operate only on
+ * the specific file.
+ */
+#define F_GET_RW_HINT (F_LINUX_SPECIFIC_BASE + 11)
+#define F_SET_RW_HINT (F_LINUX_SPECIFIC_BASE + 12)
+#define F_GET_FILE_RW_HINT (F_LINUX_SPECIFIC_BASE + 13)
+#define F_SET_FILE_RW_HINT (F_LINUX_SPECIFIC_BASE + 14)
+
+/*
+ * Valid hint values for F_{GET,SET}_RW_HINT. 0 is "not set", or can be
+ * used to clear any hints previously set.
+ */
+#define RWF_WRITE_LIFE_NOT_SET 0
+#define RWH_WRITE_LIFE_NONE 1
+#define RWH_WRITE_LIFE_SHORT 2
+#define RWH_WRITE_LIFE_MEDIUM 3
+#define RWH_WRITE_LIFE_LONG 4
+#define RWH_WRITE_LIFE_EXTREME 5
+
/*
* Types of directory notifications that may be requested.
*/
#define RWF_HIPRI 0x00000001 /* high priority request, poll if possible */
#define RWF_DSYNC 0x00000002 /* per-IO O_DSYNC */
#define RWF_SYNC 0x00000004 /* per-IO O_SYNC */
+#define RWF_NOWAIT 0x00000008 /* per-IO, return -EAGAIN if operation would block */
+
+#define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC |\
+ RWF_NOWAIT)
#endif /* _UAPI_LINUX_FS_H */
LO_FLAGS_AUTOCLEAR = 4,
LO_FLAGS_PARTSCAN = 8,
LO_FLAGS_DIRECT_IO = 16,
+ LO_FLAGS_BLOCKSIZE = 32,
};
#include <asm/posix_types.h> /* for __kernel_old_dev_t */
__u64 lo_init[2];
};
+#define LO_INFO_BLOCKSIZE(l) (l)->lo_init[0]
+
/*
* Loop filter types
*/
#define NBD_FLAG_HAS_FLAGS (1 << 0) /* nbd-server supports flags */
#define NBD_FLAG_READ_ONLY (1 << 1) /* device is read-only */
#define NBD_FLAG_SEND_FLUSH (1 << 2) /* can flush writeback cache */
+#define NBD_FLAG_SEND_FUA (1 << 3) /* send FUA (forced unit access) */
/* there is a gap here to match userspace */
#define NBD_FLAG_SEND_TRIM (1 << 5) /* send trim/discard */
#define NBD_FLAG_CAN_MULTI_CONN (1 << 8) /* Server supports multiple connections per export. */
+/* values for cmd flags in the upper 16 bits of request type */
+#define NBD_CMD_FLAG_FUA (1 << 16) /* FUA (forced unit access) op */
+
/* These are client behavior specific flags. */
#define NBD_CFLAG_DESTROY_ON_DISCONNECT (1 << 0) /* delete the nbd device on
disconnect. */
* For the sched_{set,get}attr() calls
*/
#define SCHED_FLAG_RESET_ON_FORK 0x01
+#define SCHED_FLAG_RECLAIM 0x02
#endif /* _UAPI_LINUX_SCHED_H */
#define SER_RS485_RTS_AFTER_SEND (1 << 2) /* Logical level for
RTS pin after sent*/
#define SER_RS485_RX_DURING_TX (1 << 4)
+#define SER_RS485_TERMINATE_BUS (1 << 5) /* Enable bus
+ termination
+ (if supported) */
__u32 delay_rts_before_send; /* Delay before send (milliseconds) */
__u32 delay_rts_after_send; /* Delay after send (milliseconds) */
__u32 padding[5]; /* Memory is cheap, new structs
/* Parisc type numbers. */
#define PORT_MUX 48
-/* Atmel AT91 / AT32 SoC */
+/* Atmel AT91 SoC */
#define PORT_ATMEL 49
/* Macintosh Zilog type numbers */
#define CLOCK_BOOTTIME 7
#define CLOCK_REALTIME_ALARM 8
#define CLOCK_BOOTTIME_ALARM 9
-#define CLOCK_SGI_CYCLE 10 /* Hardware specific */
+/*
+ * The driver implementing this got removed. The clock ID is kept as a
+ * place holder. Do not reuse!
+ */
+#define CLOCK_SGI_CYCLE 10
#define CLOCK_TAI 11
#define MAX_CLOCKS 16
#define N_TRACESINK 23 /* Trace data routing for MIPI P1149.7 */
#define N_TRACEROUTER 24 /* Trace data routing for MIPI P1149.7 */
#define N_NCI 25 /* NFC NCI UART */
+#define N_SPEAKUP 26 /* Speakup communication with synths */
+#define N_NULL 27 /* Null ldisc used for error handling */
#endif /* _UAPI_LINUX_TTY_H */
#define FUNCTIONFS_INTERFACE_REVMAP _IO('g', 128)
/*
- * Returns real bEndpointAddress of an endpoint. If function is not
- * active returns -ENODEV.
+ * Returns real bEndpointAddress of an endpoint. If endpoint shuts down
+ * during the call, returns -ESHUTDOWN.
*/
#define FUNCTIONFS_ENDPOINT_REVMAP _IO('g', 129)
/*
- * Returns endpoint descriptor. If function is not active returns -ENODEV.
+ * Returns endpoint descriptor. If endpoint shuts down during the call,
+ * returns -ESHUTDOWN.
*/
#define FUNCTIONFS_ENDPOINT_DESC _IOR('g', 130, \
struct usb_endpoint_descriptor)
unsigned char eps[0];
};
+/*
+ * USB_SPEED_* values returned by USBDEVFS_GET_SPEED are defined in
+ * linux/usb/ch9.h
+ */
+
#define USBDEVFS_CONTROL _IOWR('U', 0, struct usbdevfs_ctrltransfer)
#define USBDEVFS_CONTROL32 _IOWR('U', 0, struct usbdevfs_ctrltransfer32)
#define USBDEVFS_BULK _IOWR('U', 2, struct usbdevfs_bulktransfer)
#define USBDEVFS_ALLOC_STREAMS _IOR('U', 28, struct usbdevfs_streams)
#define USBDEVFS_FREE_STREAMS _IOR('U', 29, struct usbdevfs_streams)
#define USBDEVFS_DROP_PRIVILEGES _IOW('U', 30, __u32)
+#define USBDEVFS_GET_SPEED _IO('U', 31)
#endif /* _UAPI_LINUX_USBDEVICE_FS_H */
typedef struct {
__u8 b[16];
-} uuid_le;
+} guid_t;
-typedef struct {
- __u8 b[16];
-} uuid_be;
-
-#define UUID_LE(a, b, c, d0, d1, d2, d3, d4, d5, d6, d7) \
-((uuid_le) \
+#define GUID_INIT(a, b, c, d0, d1, d2, d3, d4, d5, d6, d7) \
+((guid_t) \
{{ (a) & 0xff, ((a) >> 8) & 0xff, ((a) >> 16) & 0xff, ((a) >> 24) & 0xff, \
(b) & 0xff, ((b) >> 8) & 0xff, \
(c) & 0xff, ((c) >> 8) & 0xff, \
(d0), (d1), (d2), (d3), (d4), (d5), (d6), (d7) }})
-#define UUID_BE(a, b, c, d0, d1, d2, d3, d4, d5, d6, d7) \
-((uuid_be) \
-{{ ((a) >> 24) & 0xff, ((a) >> 16) & 0xff, ((a) >> 8) & 0xff, (a) & 0xff, \
- ((b) >> 8) & 0xff, (b) & 0xff, \
- ((c) >> 8) & 0xff, (c) & 0xff, \
- (d0), (d1), (d2), (d3), (d4), (d5), (d6), (d7) }})
-
+/* backwards compatibility, don't use in new code */
+typedef guid_t uuid_le;
+#define UUID_LE(a, b, c, d0, d1, d2, d3, d4, d5, d6, d7) \
+ GUID_INIT(a, b, c, d0, d1, d2, d3, d4, d5, d6, d7)
#define NULL_UUID_LE \
UUID_LE(0x00000000, 0x0000, 0x0000, 0x00, 0x00, 0x00, 0x00, \
- 0x00, 0x00, 0x00, 0x00)
-
-#define NULL_UUID_BE \
- UUID_BE(0x00000000, 0x0000, 0x0000, 0x00, 0x00, 0x00, 0x00, \
- 0x00, 0x00, 0x00, 0x00)
-
+ 0x00, 0x00, 0x00, 0x00)
#endif /* _UAPI_LINUX_UUID_H_ */
endmenu # "CPU/Task time and stats accounting"
-menu "RCU Subsystem"
-
-config TREE_RCU
- bool
- default y if !PREEMPT && SMP
- help
- This option selects the RCU implementation that is
- designed for very large SMP system with hundreds or
- thousands of CPUs. It also scales down nicely to
- smaller systems.
-
-config PREEMPT_RCU
- bool
- default y if PREEMPT
- help
- This option selects the RCU implementation that is
- designed for very large SMP systems with hundreds or
- thousands of CPUs, but for which real-time response
- is also required. It also scales down nicely to
- smaller systems.
-
- Select this option if you are unsure.
-
-config TINY_RCU
- bool
- default y if !PREEMPT && !SMP
- help
- This option selects the RCU implementation that is
- designed for UP systems from which real-time response
- is not required. This option greatly reduces the
- memory footprint of RCU.
-
-config RCU_EXPERT
- bool "Make expert-level adjustments to RCU configuration"
- default n
- help
- This option needs to be enabled if you wish to make
- expert-level adjustments to RCU configuration. By default,
- no such adjustments can be made, which has the often-beneficial
- side-effect of preventing "make oldconfig" from asking you all
- sorts of detailed questions about how you would like numerous
- obscure RCU options to be set up.
-
- Say Y if you need to make expert-level adjustments to RCU.
-
- Say N if you are unsure.
-
-config SRCU
- bool
- default y
- help
- This option selects the sleepable version of RCU. This version
- permits arbitrary sleeping or blocking within RCU read-side critical
- sections.
-
-config CLASSIC_SRCU
- bool "Use v4.11 classic SRCU implementation"
- default n
- depends on RCU_EXPERT && SRCU
- help
- This option selects the traditional well-tested classic SRCU
- implementation from v4.11, as might be desired for enterprise
- Linux distributions. Without this option, the shiny new
- Tiny SRCU and Tree SRCU implementations are used instead.
- At some point, it is hoped that Tiny SRCU and Tree SRCU
- will accumulate enough test time and confidence to allow
- Classic SRCU to be dropped entirely.
-
- Say Y if you need a rock-solid SRCU.
-
- Say N if you would like help test Tree SRCU.
-
-config TINY_SRCU
- bool
- default y if SRCU && TINY_RCU && !CLASSIC_SRCU
- help
- This option selects the single-CPU non-preemptible version of SRCU.
-
-config TREE_SRCU
- bool
- default y if SRCU && !TINY_RCU && !CLASSIC_SRCU
- help
- This option selects the full-fledged version of SRCU.
-
-config TASKS_RCU
- bool
- default n
- select SRCU
- help
- This option enables a task-based RCU implementation that uses
- only voluntary context switch (not preemption!), idle, and
- user-mode execution as quiescent states.
-
-config RCU_STALL_COMMON
- def_bool ( TREE_RCU || PREEMPT_RCU || RCU_TRACE )
- help
- This option enables RCU CPU stall code that is common between
- the TINY and TREE variants of RCU. The purpose is to allow
- the tiny variants to disable RCU CPU stall warnings, while
- making these warnings mandatory for the tree variants.
-
-config RCU_NEED_SEGCBLIST
- def_bool ( TREE_RCU || PREEMPT_RCU || TINY_SRCU || TREE_SRCU )
-
-config CONTEXT_TRACKING
- bool
-
-config CONTEXT_TRACKING_FORCE
- bool "Force context tracking"
- depends on CONTEXT_TRACKING
- default y if !NO_HZ_FULL
- help
- The major pre-requirement for full dynticks to work is to
- support the context tracking subsystem. But there are also
- other dependencies to provide in order to make the full
- dynticks working.
-
- This option stands for testing when an arch implements the
- context tracking backend but doesn't yet fullfill all the
- requirements to make the full dynticks feature working.
- Without the full dynticks, there is no way to test the support
- for context tracking and the subsystems that rely on it: RCU
- userspace extended quiescent state and tickless cputime
- accounting. This option copes with the absence of the full
- dynticks subsystem by forcing the context tracking on all
- CPUs in the system.
-
- Say Y only if you're working on the development of an
- architecture backend for the context tracking.
-
- Say N otherwise, this option brings an overhead that you
- don't want in production.
-
-
-config RCU_FANOUT
- int "Tree-based hierarchical RCU fanout value"
- range 2 64 if 64BIT
- range 2 32 if !64BIT
- depends on (TREE_RCU || PREEMPT_RCU) && RCU_EXPERT
- default 64 if 64BIT
- default 32 if !64BIT
- help
- This option controls the fanout of hierarchical implementations
- of RCU, allowing RCU to work efficiently on machines with
- large numbers of CPUs. This value must be at least the fourth
- root of NR_CPUS, which allows NR_CPUS to be insanely large.
- The default value of RCU_FANOUT should be used for production
- systems, but if you are stress-testing the RCU implementation
- itself, small RCU_FANOUT values allow you to test large-system
- code paths on small(er) systems.
-
- Select a specific number if testing RCU itself.
- Take the default if unsure.
-
-config RCU_FANOUT_LEAF
- int "Tree-based hierarchical RCU leaf-level fanout value"
- range 2 64 if 64BIT
- range 2 32 if !64BIT
- depends on (TREE_RCU || PREEMPT_RCU) && RCU_EXPERT
- default 16
- help
- This option controls the leaf-level fanout of hierarchical
- implementations of RCU, and allows trading off cache misses
- against lock contention. Systems that synchronize their
- scheduling-clock interrupts for energy-efficiency reasons will
- want the default because the smaller leaf-level fanout keeps
- lock contention levels acceptably low. Very large systems
- (hundreds or thousands of CPUs) will instead want to set this
- value to the maximum value possible in order to reduce the
- number of cache misses incurred during RCU's grace-period
- initialization. These systems tend to run CPU-bound, and thus
- are not helped by synchronized interrupts, and thus tend to
- skew them, which reduces lock contention enough that large
- leaf-level fanouts work well. That said, setting leaf-level
- fanout to a large number will likely cause problematic
- lock contention on the leaf-level rcu_node structures unless
- you boot with the skew_tick kernel parameter.
-
- Select a specific number if testing RCU itself.
-
- Select the maximum permissible value for large systems, but
- please understand that you may also need to set the skew_tick
- kernel boot parameter to avoid contention on the rcu_node
- structure's locks.
-
- Take the default if unsure.
-
-config RCU_FAST_NO_HZ
- bool "Accelerate last non-dyntick-idle CPU's grace periods"
- depends on NO_HZ_COMMON && SMP && RCU_EXPERT
- default n
- help
- This option permits CPUs to enter dynticks-idle state even if
- they have RCU callbacks queued, and prevents RCU from waking
- these CPUs up more than roughly once every four jiffies (by
- default, you can adjust this using the rcutree.rcu_idle_gp_delay
- parameter), thus improving energy efficiency. On the other
- hand, this option increases the duration of RCU grace periods,
- for example, slowing down synchronize_rcu().
-
- Say Y if energy efficiency is critically important, and you
- don't care about increased grace-period durations.
-
- Say N if you are unsure.
-
-config TREE_RCU_TRACE
- def_bool RCU_TRACE && ( TREE_RCU || PREEMPT_RCU )
- select DEBUG_FS
- help
- This option provides tracing for the TREE_RCU and
- PREEMPT_RCU implementations, permitting Makefile to
- trivially select kernel/rcutree_trace.c.
-
-config RCU_BOOST
- bool "Enable RCU priority boosting"
- depends on RT_MUTEXES && PREEMPT_RCU && RCU_EXPERT
- default n
- help
- This option boosts the priority of preempted RCU readers that
- block the current preemptible RCU grace period for too long.
- This option also prevents heavy loads from blocking RCU
- callback invocation for all flavors of RCU.
-
- Say Y here if you are working with real-time apps or heavy loads
- Say N here if you are unsure.
-
-config RCU_KTHREAD_PRIO
- int "Real-time priority to use for RCU worker threads"
- range 1 99 if RCU_BOOST
- range 0 99 if !RCU_BOOST
- default 1 if RCU_BOOST
- default 0 if !RCU_BOOST
- depends on RCU_EXPERT
- help
- This option specifies the SCHED_FIFO priority value that will be
- assigned to the rcuc/n and rcub/n threads and is also the value
- used for RCU_BOOST (if enabled). If you are working with a
- real-time application that has one or more CPU-bound threads
- running at a real-time priority level, you should set
- RCU_KTHREAD_PRIO to a priority higher than the highest-priority
- real-time CPU-bound application thread. The default RCU_KTHREAD_PRIO
- value of 1 is appropriate in the common case, which is real-time
- applications that do not have any CPU-bound threads.
-
- Some real-time applications might not have a single real-time
- thread that saturates a given CPU, but instead might have
- multiple real-time threads that, taken together, fully utilize
- that CPU. In this case, you should set RCU_KTHREAD_PRIO to
- a priority higher than the lowest-priority thread that is
- conspiring to prevent the CPU from running any non-real-time
- tasks. For example, if one thread at priority 10 and another
- thread at priority 5 are between themselves fully consuming
- the CPU time on a given CPU, then RCU_KTHREAD_PRIO should be
- set to priority 6 or higher.
-
- Specify the real-time priority, or take the default if unsure.
-
-config RCU_BOOST_DELAY
- int "Milliseconds to delay boosting after RCU grace-period start"
- range 0 3000
- depends on RCU_BOOST
- default 500
- help
- This option specifies the time to wait after the beginning of
- a given grace period before priority-boosting preempted RCU
- readers blocking that grace period. Note that any RCU reader
- blocking an expedited RCU grace period is boosted immediately.
-
- Accept the default if unsure.
-
-config RCU_NOCB_CPU
- bool "Offload RCU callback processing from boot-selected CPUs"
- depends on TREE_RCU || PREEMPT_RCU
- depends on RCU_EXPERT || NO_HZ_FULL
- default n
- help
- Use this option to reduce OS jitter for aggressive HPC or
- real-time workloads. It can also be used to offload RCU
- callback invocation to energy-efficient CPUs in battery-powered
- asymmetric multiprocessors.
-
- This option offloads callback invocation from the set of
- CPUs specified at boot time by the rcu_nocbs parameter.
- For each such CPU, a kthread ("rcuox/N") will be created to
- invoke callbacks, where the "N" is the CPU being offloaded,
- and where the "x" is "b" for RCU-bh, "p" for RCU-preempt, and
- "s" for RCU-sched. Nothing prevents this kthread from running
- on the specified CPUs, but (1) the kthreads may be preempted
- between each callback, and (2) affinity or cgroups can be used
- to force the kthreads to run on whatever set of CPUs is desired.
-
- Say Y here if you want to help to debug reduced OS jitter.
- Say N here if you are unsure.
-
-choice
- prompt "Build-forced no-CBs CPUs"
- default RCU_NOCB_CPU_NONE
- depends on RCU_NOCB_CPU
- help
- This option allows no-CBs CPUs (whose RCU callbacks are invoked
- from kthreads rather than from softirq context) to be specified
- at build time. Additional no-CBs CPUs may be specified by
- the rcu_nocbs= boot parameter.
-
-config RCU_NOCB_CPU_NONE
- bool "No build_forced no-CBs CPUs"
- help
- This option does not force any of the CPUs to be no-CBs CPUs.
- Only CPUs designated by the rcu_nocbs= boot parameter will be
- no-CBs CPUs, whose RCU callbacks will be invoked by per-CPU
- kthreads whose names begin with "rcuo". All other CPUs will
- invoke their own RCU callbacks in softirq context.
-
- Select this option if you want to choose no-CBs CPUs at
- boot time, for example, to allow testing of different no-CBs
- configurations without having to rebuild the kernel each time.
-
-config RCU_NOCB_CPU_ZERO
- bool "CPU 0 is a build_forced no-CBs CPU"
- help
- This option forces CPU 0 to be a no-CBs CPU, so that its RCU
- callbacks are invoked by a per-CPU kthread whose name begins
- with "rcuo". Additional CPUs may be designated as no-CBs
- CPUs using the rcu_nocbs= boot parameter will be no-CBs CPUs.
- All other CPUs will invoke their own RCU callbacks in softirq
- context.
-
- Select this if CPU 0 needs to be a no-CBs CPU for real-time
- or energy-efficiency reasons, but the real reason it exists
- is to ensure that randconfig testing covers mixed systems.
-
-config RCU_NOCB_CPU_ALL
- bool "All CPUs are build_forced no-CBs CPUs"
- help
- This option forces all CPUs to be no-CBs CPUs. The rcu_nocbs=
- boot parameter will be ignored. All CPUs' RCU callbacks will
- be executed in the context of per-CPU rcuo kthreads created for
- this purpose. Assuming that the kthreads whose names start with
- "rcuo" are bound to "housekeeping" CPUs, this reduces OS jitter
- on the remaining CPUs, but might decrease memory locality during
- RCU-callback invocation, thus potentially degrading throughput.
-
- Select this if all CPUs need to be no-CBs CPUs for real-time
- or energy-efficiency reasons.
-
-endchoice
-
-endmenu # "RCU Subsystem"
+source "kernel/rcu/Kconfig"
config BUILD_BIN2C
bool
config CPUSETS
bool "Cpuset controller"
+ depends on SMP
help
This option will let you create and manage CPUSETs which
allow dynamically partitioning a system into sets of CPUs and
static noinline void __ref rest_init(void)
{
+ struct task_struct *tsk;
int pid;
rcu_scheduler_starting();
* the init task will end up wanting to create kthreads, which, if
* we schedule it before we create kthreadd, will OOPS.
*/
- kernel_thread(kernel_init, NULL, CLONE_FS);
+ pid = kernel_thread(kernel_init, NULL, CLONE_FS);
+ /*
+ * Pin init on the boot CPU. Task migration is not properly working
+ * until sched_init_smp() has been run. It will set the allowed
+ * CPUs for init to the non isolated CPUs.
+ */
+ rcu_read_lock();
+ tsk = find_task_by_pid_ns(pid, &init_pid_ns);
+ set_cpus_allowed_ptr(tsk, cpumask_of(smp_processor_id()));
+ rcu_read_unlock();
+
numa_default_policy();
pid = kernel_thread(kthreadd, NULL, CLONE_FS | CLONE_FILES);
rcu_read_lock();
kthreadd_task = find_task_by_pid_ns(pid, &init_pid_ns);
rcu_read_unlock();
+
+ /*
+ * Enable might_sleep() and smp_processor_id() checks.
+ * They cannot be enabled earlier because with CONFIG_PRREMPT=y
+ * kernel_thread() would trigger might_sleep() splats. With
+ * CONFIG_PREEMPT_VOLUNTARY=y the init task might have scheduled
+ * already, but it's stuck on the kthreadd_done completion.
+ */
+ system_state = SYSTEM_SCHEDULING;
+
complete(&kthreadd_done);
/*
* init can allocate pages on any node
*/
set_mems_allowed(node_states[N_MEMORY]);
- /*
- * init can run on any cpu.
- */
- set_cpus_allowed_ptr(current, cpu_all_mask);
cad_pid = task_pid(current);
ktime_t uninitialized_var(calltime), delta, rettime;
/* 1) run (and print duration) */
- if (initcall_debug && system_state == SYSTEM_BOOTING) {
+ if (initcall_debug && system_state < SYSTEM_RUNNING) {
pr_debug("calling %lli_%pF @ %i\n",
(long long)entry->cookie,
entry->func, task_pid_nr(current));
calltime = ktime_get();
}
entry->func(entry->data, entry->cookie);
- if (initcall_debug && system_state == SYSTEM_BOOTING) {
+ if (initcall_debug && system_state < SYSTEM_RUNNING) {
rettime = ktime_get();
delta = ktime_sub(rettime, calltime);
pr_debug("initcall %lli_%pF returned 0 after %lld usecs\n",
{
ktime_t uninitialized_var(starttime), delta, endtime;
- if (initcall_debug && system_state == SYSTEM_BOOTING) {
+ if (initcall_debug && system_state < SYSTEM_RUNNING) {
pr_debug("async_waiting @ %i\n", task_pid_nr(current));
starttime = ktime_get();
}
wait_event(async_done, lowest_in_progress(domain) >= cookie);
- if (initcall_debug && system_state == SYSTEM_BOOTING) {
+ if (initcall_debug && system_state < SYSTEM_RUNNING) {
endtime = ktime_get();
delta = ktime_sub(endtime, starttime);
if (err)
return err;
+ if (is_pointer_value(env, insn->src_reg)) {
+ verbose("R%d leaks addr into mem\n", insn->src_reg);
+ return -EACCES;
+ }
+
/* check whether atomic_add can read the memory */
err = check_mem_access(env, insn->dst_reg, insn->off,
BPF_SIZE(insn->code), BPF_READ, -1);
#include <linux/uaccess.h>
-static int compat_get_timex(struct timex *txc, struct compat_timex __user *utp)
+int compat_get_timex(struct timex *txc, const struct compat_timex __user *utp)
{
- memset(txc, 0, sizeof(struct timex));
-
- if (!access_ok(VERIFY_READ, utp, sizeof(struct compat_timex)) ||
- __get_user(txc->modes, &utp->modes) ||
- __get_user(txc->offset, &utp->offset) ||
- __get_user(txc->freq, &utp->freq) ||
- __get_user(txc->maxerror, &utp->maxerror) ||
- __get_user(txc->esterror, &utp->esterror) ||
- __get_user(txc->status, &utp->status) ||
- __get_user(txc->constant, &utp->constant) ||
- __get_user(txc->precision, &utp->precision) ||
- __get_user(txc->tolerance, &utp->tolerance) ||
- __get_user(txc->time.tv_sec, &utp->time.tv_sec) ||
- __get_user(txc->time.tv_usec, &utp->time.tv_usec) ||
- __get_user(txc->tick, &utp->tick) ||
- __get_user(txc->ppsfreq, &utp->ppsfreq) ||
- __get_user(txc->jitter, &utp->jitter) ||
- __get_user(txc->shift, &utp->shift) ||
- __get_user(txc->stabil, &utp->stabil) ||
- __get_user(txc->jitcnt, &utp->jitcnt) ||
- __get_user(txc->calcnt, &utp->calcnt) ||
- __get_user(txc->errcnt, &utp->errcnt) ||
- __get_user(txc->stbcnt, &utp->stbcnt))
- return -EFAULT;
+ struct compat_timex tx32;
- return 0;
-}
-
-static int compat_put_timex(struct compat_timex __user *utp, struct timex *txc)
-{
- if (!access_ok(VERIFY_WRITE, utp, sizeof(struct compat_timex)) ||
- __put_user(txc->modes, &utp->modes) ||
- __put_user(txc->offset, &utp->offset) ||
- __put_user(txc->freq, &utp->freq) ||
- __put_user(txc->maxerror, &utp->maxerror) ||
- __put_user(txc->esterror, &utp->esterror) ||
- __put_user(txc->status, &utp->status) ||
- __put_user(txc->constant, &utp->constant) ||
- __put_user(txc->precision, &utp->precision) ||
- __put_user(txc->tolerance, &utp->tolerance) ||
- __put_user(txc->time.tv_sec, &utp->time.tv_sec) ||
- __put_user(txc->time.tv_usec, &utp->time.tv_usec) ||
- __put_user(txc->tick, &utp->tick) ||
- __put_user(txc->ppsfreq, &utp->ppsfreq) ||
- __put_user(txc->jitter, &utp->jitter) ||
- __put_user(txc->shift, &utp->shift) ||
- __put_user(txc->stabil, &utp->stabil) ||
- __put_user(txc->jitcnt, &utp->jitcnt) ||
- __put_user(txc->calcnt, &utp->calcnt) ||
- __put_user(txc->errcnt, &utp->errcnt) ||
- __put_user(txc->stbcnt, &utp->stbcnt) ||
- __put_user(txc->tai, &utp->tai))
+ if (copy_from_user(&tx32, utp, sizeof(struct compat_timex)))
return -EFAULT;
- return 0;
-}
-COMPAT_SYSCALL_DEFINE2(gettimeofday, struct compat_timeval __user *, tv,
- struct timezone __user *, tz)
-{
- if (tv) {
- struct timeval ktv;
- do_gettimeofday(&ktv);
- if (compat_put_timeval(&ktv, tv))
- return -EFAULT;
- }
- if (tz) {
- if (copy_to_user(tz, &sys_tz, sizeof(sys_tz)))
- return -EFAULT;
- }
+ txc->modes = tx32.modes;
+ txc->offset = tx32.offset;
+ txc->freq = tx32.freq;
+ txc->maxerror = tx32.maxerror;
+ txc->esterror = tx32.esterror;
+ txc->status = tx32.status;
+ txc->constant = tx32.constant;
+ txc->precision = tx32.precision;
+ txc->tolerance = tx32.tolerance;
+ txc->time.tv_sec = tx32.time.tv_sec;
+ txc->time.tv_usec = tx32.time.tv_usec;
+ txc->tick = tx32.tick;
+ txc->ppsfreq = tx32.ppsfreq;
+ txc->jitter = tx32.jitter;
+ txc->shift = tx32.shift;
+ txc->stabil = tx32.stabil;
+ txc->jitcnt = tx32.jitcnt;
+ txc->calcnt = tx32.calcnt;
+ txc->errcnt = tx32.errcnt;
+ txc->stbcnt = tx32.stbcnt;
return 0;
}
-COMPAT_SYSCALL_DEFINE2(settimeofday, struct compat_timeval __user *, tv,
- struct timezone __user *, tz)
-{
- struct timespec64 new_ts;
- struct timeval user_tv;
- struct timezone new_tz;
-
- if (tv) {
- if (compat_get_timeval(&user_tv, tv))
- return -EFAULT;
- new_ts.tv_sec = user_tv.tv_sec;
- new_ts.tv_nsec = user_tv.tv_usec * NSEC_PER_USEC;
- }
- if (tz) {
- if (copy_from_user(&new_tz, tz, sizeof(*tz)))
- return -EFAULT;
- }
-
- return do_sys_settimeofday64(tv ? &new_ts : NULL, tz ? &new_tz : NULL);
+int compat_put_timex(struct compat_timex __user *utp, const struct timex *txc)
+{
+ struct compat_timex tx32;
+
+ memset(&tx32, 0, sizeof(struct compat_timex));
+ tx32.modes = txc->modes;
+ tx32.offset = txc->offset;
+ tx32.freq = txc->freq;
+ tx32.maxerror = txc->maxerror;
+ tx32.esterror = txc->esterror;
+ tx32.status = txc->status;
+ tx32.constant = txc->constant;
+ tx32.precision = txc->precision;
+ tx32.tolerance = txc->tolerance;
+ tx32.time.tv_sec = txc->time.tv_sec;
+ tx32.time.tv_usec = txc->time.tv_usec;
+ tx32.tick = txc->tick;
+ tx32.ppsfreq = txc->ppsfreq;
+ tx32.jitter = txc->jitter;
+ tx32.shift = txc->shift;
+ tx32.stabil = txc->stabil;
+ tx32.jitcnt = txc->jitcnt;
+ tx32.calcnt = txc->calcnt;
+ tx32.errcnt = txc->errcnt;
+ tx32.stbcnt = txc->stbcnt;
+ tx32.tai = txc->tai;
+ if (copy_to_user(utp, &tx32, sizeof(struct compat_timex)))
+ return -EFAULT;
+ return 0;
}
static int __compat_get_timeval(struct timeval *tv, const struct compat_timeval __user *ctv)
return 0;
}
-static long compat_nanosleep_restart(struct restart_block *restart)
-{
- struct compat_timespec __user *rmtp;
- struct timespec rmt;
- mm_segment_t oldfs;
- long ret;
-
- restart->nanosleep.rmtp = (struct timespec __user *) &rmt;
- oldfs = get_fs();
- set_fs(KERNEL_DS);
- ret = hrtimer_nanosleep_restart(restart);
- set_fs(oldfs);
-
- if (ret == -ERESTART_RESTARTBLOCK) {
- rmtp = restart->nanosleep.compat_rmtp;
-
- if (rmtp && compat_put_timespec(&rmt, rmtp))
- return -EFAULT;
- }
-
- return ret;
-}
-
-COMPAT_SYSCALL_DEFINE2(nanosleep, struct compat_timespec __user *, rqtp,
- struct compat_timespec __user *, rmtp)
+int get_compat_itimerval(struct itimerval *o, const struct compat_itimerval __user *i)
{
- struct timespec tu, rmt;
- struct timespec64 tu64;
- mm_segment_t oldfs;
- long ret;
+ struct compat_itimerval v32;
- if (compat_get_timespec(&tu, rqtp))
+ if (copy_from_user(&v32, i, sizeof(struct compat_itimerval)))
return -EFAULT;
-
- tu64 = timespec_to_timespec64(tu);
- if (!timespec64_valid(&tu64))
- return -EINVAL;
-
- oldfs = get_fs();
- set_fs(KERNEL_DS);
- ret = hrtimer_nanosleep(&tu64,
- rmtp ? (struct timespec __user *)&rmt : NULL,
- HRTIMER_MODE_REL, CLOCK_MONOTONIC);
- set_fs(oldfs);
-
- /*
- * hrtimer_nanosleep() can only return 0 or
- * -ERESTART_RESTARTBLOCK here because:
- *
- * - we call it with HRTIMER_MODE_REL and therefor exclude the
- * -ERESTARTNOHAND return path.
- *
- * - we supply the rmtp argument from the task stack (due to
- * the necessary compat conversion. So the update cannot
- * fail, which excludes the -EFAULT return path as well. If
- * it fails nevertheless we have a bigger problem and wont
- * reach this place anymore.
- *
- * - if the return value is 0, we do not have to update rmtp
- * because there is no remaining time.
- *
- * We check for -ERESTART_RESTARTBLOCK nevertheless if the
- * core implementation decides to return random nonsense.
- */
- if (ret == -ERESTART_RESTARTBLOCK) {
- struct restart_block *restart = ¤t->restart_block;
-
- restart->fn = compat_nanosleep_restart;
- restart->nanosleep.compat_rmtp = rmtp;
-
- if (rmtp && compat_put_timespec(&rmt, rmtp))
- return -EFAULT;
- }
- return ret;
-}
-
-static inline long get_compat_itimerval(struct itimerval *o,
- struct compat_itimerval __user *i)
-{
- return (!access_ok(VERIFY_READ, i, sizeof(*i)) ||
- (__get_user(o->it_interval.tv_sec, &i->it_interval.tv_sec) |
- __get_user(o->it_interval.tv_usec, &i->it_interval.tv_usec) |
- __get_user(o->it_value.tv_sec, &i->it_value.tv_sec) |
- __get_user(o->it_value.tv_usec, &i->it_value.tv_usec)));
-}
-
-static inline long put_compat_itimerval(struct compat_itimerval __user *o,
- struct itimerval *i)
-{
- return (!access_ok(VERIFY_WRITE, o, sizeof(*o)) ||
- (__put_user(i->it_interval.tv_sec, &o->it_interval.tv_sec) |
- __put_user(i->it_interval.tv_usec, &o->it_interval.tv_usec) |
- __put_user(i->it_value.tv_sec, &o->it_value.tv_sec) |
- __put_user(i->it_value.tv_usec, &o->it_value.tv_usec)));
-}
-
-asmlinkage long sys_ni_posix_timers(void);
-
-COMPAT_SYSCALL_DEFINE2(getitimer, int, which,
- struct compat_itimerval __user *, it)
-{
- struct itimerval kit;
- int error;
-
- if (!IS_ENABLED(CONFIG_POSIX_TIMERS))
- return sys_ni_posix_timers();
-
- error = do_getitimer(which, &kit);
- if (!error && put_compat_itimerval(it, &kit))
- error = -EFAULT;
- return error;
+ o->it_interval.tv_sec = v32.it_interval.tv_sec;
+ o->it_interval.tv_usec = v32.it_interval.tv_usec;
+ o->it_value.tv_sec = v32.it_value.tv_sec;
+ o->it_value.tv_usec = v32.it_value.tv_usec;
+ return 0;
}
-COMPAT_SYSCALL_DEFINE3(setitimer, int, which,
- struct compat_itimerval __user *, in,
- struct compat_itimerval __user *, out)
+int put_compat_itimerval(struct compat_itimerval __user *o, const struct itimerval *i)
{
- struct itimerval kin, kout;
- int error;
-
- if (!IS_ENABLED(CONFIG_POSIX_TIMERS))
- return sys_ni_posix_timers();
+ struct compat_itimerval v32;
- if (in) {
- if (get_compat_itimerval(&kin, in))
- return -EFAULT;
- } else
- memset(&kin, 0, sizeof(kin));
-
- error = do_setitimer(which, &kin, out ? &kout : NULL);
- if (error || !out)
- return error;
- if (put_compat_itimerval(out, &kout))
- return -EFAULT;
- return 0;
+ v32.it_interval.tv_sec = i->it_interval.tv_sec;
+ v32.it_interval.tv_usec = i->it_interval.tv_usec;
+ v32.it_value.tv_sec = i->it_value.tv_sec;
+ v32.it_value.tv_usec = i->it_value.tv_usec;
+ return copy_to_user(o, &v32, sizeof(struct compat_itimerval)) ? -EFAULT : 0;
}
static compat_clock_t clock_t_to_compat_clock_t(clock_t x)
return 0;
}
-COMPAT_SYSCALL_DEFINE3(timer_create, clockid_t, which_clock,
- struct compat_sigevent __user *, timer_event_spec,
- timer_t __user *, created_timer_id)
-{
- struct sigevent __user *event = NULL;
-
- if (timer_event_spec) {
- struct sigevent kevent;
-
- event = compat_alloc_user_space(sizeof(*event));
- if (get_compat_sigevent(&kevent, timer_event_spec) ||
- copy_to_user(event, &kevent, sizeof(*event)))
- return -EFAULT;
- }
-
- return sys_timer_create(which_clock, event, created_timer_id);
-}
-
-COMPAT_SYSCALL_DEFINE4(timer_settime, timer_t, timer_id, int, flags,
- struct compat_itimerspec __user *, new,
- struct compat_itimerspec __user *, old)
-{
- long err;
- mm_segment_t oldfs;
- struct itimerspec newts, oldts;
-
- if (!new)
- return -EINVAL;
- if (get_compat_itimerspec(&newts, new))
- return -EFAULT;
- oldfs = get_fs();
- set_fs(KERNEL_DS);
- err = sys_timer_settime(timer_id, flags,
- (struct itimerspec __user *) &newts,
- (struct itimerspec __user *) &oldts);
- set_fs(oldfs);
- if (!err && old && put_compat_itimerspec(old, &oldts))
- return -EFAULT;
- return err;
-}
-
-COMPAT_SYSCALL_DEFINE2(timer_gettime, timer_t, timer_id,
- struct compat_itimerspec __user *, setting)
-{
- long err;
- mm_segment_t oldfs;
- struct itimerspec ts;
-
- oldfs = get_fs();
- set_fs(KERNEL_DS);
- err = sys_timer_gettime(timer_id,
- (struct itimerspec __user *) &ts);
- set_fs(oldfs);
- if (!err && put_compat_itimerspec(setting, &ts))
- return -EFAULT;
- return err;
-}
-
-COMPAT_SYSCALL_DEFINE2(clock_settime, clockid_t, which_clock,
- struct compat_timespec __user *, tp)
-{
- long err;
- mm_segment_t oldfs;
- struct timespec ts;
-
- if (compat_get_timespec(&ts, tp))
- return -EFAULT;
- oldfs = get_fs();
- set_fs(KERNEL_DS);
- err = sys_clock_settime(which_clock,
- (struct timespec __user *) &ts);
- set_fs(oldfs);
- return err;
-}
-
-COMPAT_SYSCALL_DEFINE2(clock_gettime, clockid_t, which_clock,
- struct compat_timespec __user *, tp)
-{
- long err;
- mm_segment_t oldfs;
- struct timespec ts;
-
- oldfs = get_fs();
- set_fs(KERNEL_DS);
- err = sys_clock_gettime(which_clock,
- (struct timespec __user *) &ts);
- set_fs(oldfs);
- if (!err && compat_put_timespec(&ts, tp))
- return -EFAULT;
- return err;
-}
-
-COMPAT_SYSCALL_DEFINE2(clock_adjtime, clockid_t, which_clock,
- struct compat_timex __user *, utp)
-{
- struct timex txc;
- mm_segment_t oldfs;
- int err, ret;
-
- err = compat_get_timex(&txc, utp);
- if (err)
- return err;
-
- oldfs = get_fs();
- set_fs(KERNEL_DS);
- ret = sys_clock_adjtime(which_clock, (struct timex __user *) &txc);
- set_fs(oldfs);
-
- err = compat_put_timex(utp, &txc);
- if (err)
- return err;
-
- return ret;
-}
-
-COMPAT_SYSCALL_DEFINE2(clock_getres, clockid_t, which_clock,
- struct compat_timespec __user *, tp)
-{
- long err;
- mm_segment_t oldfs;
- struct timespec ts;
-
- oldfs = get_fs();
- set_fs(KERNEL_DS);
- err = sys_clock_getres(which_clock,
- (struct timespec __user *) &ts);
- set_fs(oldfs);
- if (!err && tp && compat_put_timespec(&ts, tp))
- return -EFAULT;
- return err;
-}
-
-static long compat_clock_nanosleep_restart(struct restart_block *restart)
-{
- long err;
- mm_segment_t oldfs;
- struct timespec tu;
- struct compat_timespec __user *rmtp = restart->nanosleep.compat_rmtp;
-
- restart->nanosleep.rmtp = (struct timespec __user *) &tu;
- oldfs = get_fs();
- set_fs(KERNEL_DS);
- err = clock_nanosleep_restart(restart);
- set_fs(oldfs);
-
- if ((err == -ERESTART_RESTARTBLOCK) && rmtp &&
- compat_put_timespec(&tu, rmtp))
- return -EFAULT;
-
- if (err == -ERESTART_RESTARTBLOCK) {
- restart->fn = compat_clock_nanosleep_restart;
- restart->nanosleep.compat_rmtp = rmtp;
- }
- return err;
-}
-
-COMPAT_SYSCALL_DEFINE4(clock_nanosleep, clockid_t, which_clock, int, flags,
- struct compat_timespec __user *, rqtp,
- struct compat_timespec __user *, rmtp)
-{
- long err;
- mm_segment_t oldfs;
- struct timespec in, out;
- struct restart_block *restart;
-
- if (compat_get_timespec(&in, rqtp))
- return -EFAULT;
-
- oldfs = get_fs();
- set_fs(KERNEL_DS);
- err = sys_clock_nanosleep(which_clock, flags,
- (struct timespec __user *) &in,
- (struct timespec __user *) &out);
- set_fs(oldfs);
-
- if ((err == -ERESTART_RESTARTBLOCK) && rmtp &&
- compat_put_timespec(&out, rmtp))
- return -EFAULT;
-
- if (err == -ERESTART_RESTARTBLOCK) {
- restart = ¤t->restart_block;
- restart->fn = compat_clock_nanosleep_restart;
- restart->nanosleep.compat_rmtp = rmtp;
- }
- return err;
-}
-
/*
* We currently only need the following fields from the sigevent
* structure: sigev_value, sigev_signo, sig_notify and (sometimes
return ret;
}
-#ifdef __ARCH_WANT_COMPAT_SYS_TIME
-
-/* compat_time_t is a 32 bit "long" and needs to get converted. */
-
-COMPAT_SYSCALL_DEFINE1(time, compat_time_t __user *, tloc)
-{
- compat_time_t i;
- struct timeval tv;
-
- do_gettimeofday(&tv);
- i = tv.tv_sec;
-
- if (tloc) {
- if (put_user(i,tloc))
- return -EFAULT;
- }
- force_successful_syscall_return();
- return i;
-}
-
-COMPAT_SYSCALL_DEFINE1(stime, compat_time_t __user *, tptr)
-{
- struct timespec tv;
- int err;
-
- if (get_user(tv.tv_sec, tptr))
- return -EFAULT;
-
- tv.tv_nsec = 0;
-
- err = security_settime(&tv, NULL);
- if (err)
- return err;
-
- do_settimeofday(&tv);
- return 0;
-}
-
-#endif /* __ARCH_WANT_COMPAT_SYS_TIME */
-
-COMPAT_SYSCALL_DEFINE1(adjtimex, struct compat_timex __user *, utp)
-{
- struct timex txc;
- int err, ret;
-
- err = compat_get_timex(&txc, utp);
- if (err)
- return err;
-
- ret = do_adjtimex(&txc);
-
- err = compat_put_timex(utp, &txc);
- if (err)
- return err;
-
- return ret;
-}
-
#ifdef CONFIG_NUMA
COMPAT_SYSCALL_DEFINE6(move_pages, pid_t, pid, compat_ulong_t, nr_pages,
compat_uptr_t __user *, pages32,
# KEEP ALPHABETICALLY SORTED
# CONFIG_DEVKMEM is not set
# CONFIG_DEVMEM is not set
+# CONFIG_FHANDLE is not set
# CONFIG_INET_LRO is not set
-# CONFIG_MODULES is not set
+# CONFIG_NFSD is not set
+# CONFIG_NFS_FS is not set
# CONFIG_OABI_COMPAT is not set
# CONFIG_SYSVIPC is not set
+# CONFIG_USELIB is not set
CONFIG_ANDROID=y
CONFIG_ANDROID_BINDER_IPC=y
CONFIG_ANDROID_LOW_MEMORY_KILLER=y
CONFIG_AUDIT=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_CGROUPS=y
+CONFIG_CGROUP_BPF=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_DEBUG=y
CONFIG_CGROUP_FREEZER=y
CONFIG_FB=y
CONFIG_HARDENED_USERCOPY=y
CONFIG_HIGH_RES_TIMERS=y
+CONFIG_IKCONFIG=y
+CONFIG_IKCONFIG_PROC=y
CONFIG_INET6_AH=y
CONFIG_INET6_ESP=y
CONFIG_INET6_IPCOMP=y
CONFIG_IP_NF_TARGET_NETMAP=y
CONFIG_IP_NF_TARGET_REDIRECT=y
CONFIG_IP_NF_TARGET_REJECT=y
+CONFIG_MODULES=y
+CONFIG_MODULE_UNLOAD=y
+CONFIG_MODVERSIONS=y
CONFIG_NET=y
CONFIG_NETDEVICES=y
CONFIG_NETFILTER=y
# CONFIG_NF_CONNTRACK_SIP is not set
# CONFIG_PM_WAKELOCKS_GC is not set
# CONFIG_VT is not set
+CONFIG_ARM64_SW_TTBR0_PAN=y
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_BLK_DEV_DM=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_SIZE=8192
+CONFIG_CC_STACKPROTECTOR_STRONG=y
CONFIG_COMPACTION=y
-CONFIG_STRICT_KERNEL_RWX=y
+CONFIG_CPU_SW_DOMAIN_PAN=y
CONFIG_DM_CRYPT=y
CONFIG_DM_UEVENT=y
CONFIG_DM_VERITY=y
CONFIG_SMARTJOYPLUS_FF=y
CONFIG_SND=y
CONFIG_SOUND=y
+CONFIG_STRICT_KERNEL_RWX=y
CONFIG_SUSPEND_TIME=y
CONFIG_TABLET_USB_ACECAD=y
CONFIG_TABLET_USB_AIPTEK=y
#include <linux/smpboot.h>
#include <linux/relay.h>
#include <linux/slab.h>
+#include <linux/percpu-rwsem.h>
#include <trace/events/power.h>
#define CREATE_TRACE_POINTS
static DEFINE_PER_CPU(struct cpuhp_cpu_state, cpuhp_state);
+#if defined(CONFIG_LOCKDEP) && defined(CONFIG_SMP)
+static struct lock_class_key cpuhp_state_key;
+static struct lockdep_map cpuhp_state_lock_map =
+ STATIC_LOCKDEP_MAP_INIT("cpuhp_state", &cpuhp_state_key);
+#endif
+
/**
* cpuhp_step - Hotplug state machine step
* @name: Name of the step
mutex_unlock(&cpu_add_remove_lock);
}
-/* If set, cpu_up and cpu_down will return -EBUSY and do nothing.
+/*
+ * If set, cpu_up and cpu_down will return -EBUSY and do nothing.
* Should always be manipulated under cpu_add_remove_lock
*/
static int cpu_hotplug_disabled;
#ifdef CONFIG_HOTPLUG_CPU
-static struct {
- struct task_struct *active_writer;
- /* wait queue to wake up the active_writer */
- wait_queue_head_t wq;
- /* verifies that no writer will get active while readers are active */
- struct mutex lock;
- /*
- * Also blocks the new readers during
- * an ongoing cpu hotplug operation.
- */
- atomic_t refcount;
-
-#ifdef CONFIG_DEBUG_LOCK_ALLOC
- struct lockdep_map dep_map;
-#endif
-} cpu_hotplug = {
- .active_writer = NULL,
- .wq = __WAIT_QUEUE_HEAD_INITIALIZER(cpu_hotplug.wq),
- .lock = __MUTEX_INITIALIZER(cpu_hotplug.lock),
-#ifdef CONFIG_DEBUG_LOCK_ALLOC
- .dep_map = STATIC_LOCKDEP_MAP_INIT("cpu_hotplug.dep_map", &cpu_hotplug.dep_map),
-#endif
-};
-
-/* Lockdep annotations for get/put_online_cpus() and cpu_hotplug_begin/end() */
-#define cpuhp_lock_acquire_read() lock_map_acquire_read(&cpu_hotplug.dep_map)
-#define cpuhp_lock_acquire_tryread() \
- lock_map_acquire_tryread(&cpu_hotplug.dep_map)
-#define cpuhp_lock_acquire() lock_map_acquire(&cpu_hotplug.dep_map)
-#define cpuhp_lock_release() lock_map_release(&cpu_hotplug.dep_map)
+DEFINE_STATIC_PERCPU_RWSEM(cpu_hotplug_lock);
-
-void get_online_cpus(void)
+void cpus_read_lock(void)
{
- might_sleep();
- if (cpu_hotplug.active_writer == current)
- return;
- cpuhp_lock_acquire_read();
- mutex_lock(&cpu_hotplug.lock);
- atomic_inc(&cpu_hotplug.refcount);
- mutex_unlock(&cpu_hotplug.lock);
+ percpu_down_read(&cpu_hotplug_lock);
}
-EXPORT_SYMBOL_GPL(get_online_cpus);
+EXPORT_SYMBOL_GPL(cpus_read_lock);
-void put_online_cpus(void)
+void cpus_read_unlock(void)
{
- int refcount;
-
- if (cpu_hotplug.active_writer == current)
- return;
-
- refcount = atomic_dec_return(&cpu_hotplug.refcount);
- if (WARN_ON(refcount < 0)) /* try to fix things up */
- atomic_inc(&cpu_hotplug.refcount);
-
- if (refcount <= 0 && waitqueue_active(&cpu_hotplug.wq))
- wake_up(&cpu_hotplug.wq);
-
- cpuhp_lock_release();
-
+ percpu_up_read(&cpu_hotplug_lock);
}
-EXPORT_SYMBOL_GPL(put_online_cpus);
+EXPORT_SYMBOL_GPL(cpus_read_unlock);
-/*
- * This ensures that the hotplug operation can begin only when the
- * refcount goes to zero.
- *
- * Note that during a cpu-hotplug operation, the new readers, if any,
- * will be blocked by the cpu_hotplug.lock
- *
- * Since cpu_hotplug_begin() is always called after invoking
- * cpu_maps_update_begin(), we can be sure that only one writer is active.
- *
- * Note that theoretically, there is a possibility of a livelock:
- * - Refcount goes to zero, last reader wakes up the sleeping
- * writer.
- * - Last reader unlocks the cpu_hotplug.lock.
- * - A new reader arrives at this moment, bumps up the refcount.
- * - The writer acquires the cpu_hotplug.lock finds the refcount
- * non zero and goes to sleep again.
- *
- * However, this is very difficult to achieve in practice since
- * get_online_cpus() not an api which is called all that often.
- *
- */
-void cpu_hotplug_begin(void)
+void cpus_write_lock(void)
{
- DEFINE_WAIT(wait);
-
- cpu_hotplug.active_writer = current;
- cpuhp_lock_acquire();
+ percpu_down_write(&cpu_hotplug_lock);
+}
- for (;;) {
- mutex_lock(&cpu_hotplug.lock);
- prepare_to_wait(&cpu_hotplug.wq, &wait, TASK_UNINTERRUPTIBLE);
- if (likely(!atomic_read(&cpu_hotplug.refcount)))
- break;
- mutex_unlock(&cpu_hotplug.lock);
- schedule();
- }
- finish_wait(&cpu_hotplug.wq, &wait);
+void cpus_write_unlock(void)
+{
+ percpu_up_write(&cpu_hotplug_lock);
}
-void cpu_hotplug_done(void)
+void lockdep_assert_cpus_held(void)
{
- cpu_hotplug.active_writer = NULL;
- mutex_unlock(&cpu_hotplug.lock);
- cpuhp_lock_release();
+ percpu_rwsem_assert_held(&cpu_hotplug_lock);
}
/*
EXPORT_SYMBOL_GPL(cpu_hotplug_enable);
#endif /* CONFIG_HOTPLUG_CPU */
-/* Notifier wrappers for transitioning to state machine */
-
static int bringup_wait_for_ap(unsigned int cpu)
{
struct cpuhp_cpu_state *st = per_cpu_ptr(&cpuhp_state, cpu);
st->should_run = false;
+ lock_map_acquire(&cpuhp_state_lock_map);
/* Single callback invocation for [un]install ? */
if (st->single) {
if (st->cb_state < CPUHP_AP_ONLINE) {
else if (st->state > st->target)
ret = cpuhp_ap_offline(cpu, st);
}
+ lock_map_release(&cpuhp_state_lock_map);
st->result = ret;
complete(&st->done);
}
if (!cpu_online(cpu))
return 0;
+ lock_map_acquire(&cpuhp_state_lock_map);
+ lock_map_release(&cpuhp_state_lock_map);
+
/*
* If we are up and running, use the hotplug thread. For early calls
* we invoke the thread function directly.
enum cpuhp_state state = st->state;
trace_cpuhp_enter(cpu, st->target, state, cpuhp_kick_ap_work);
+ lock_map_acquire(&cpuhp_state_lock_map);
+ lock_map_release(&cpuhp_state_lock_map);
__cpuhp_kick_ap_work(st);
wait_for_completion(&st->done);
trace_cpuhp_exit(cpu, st->state, state, st->result);
rcu_read_unlock();
}
-static inline void check_for_tasks(int dead_cpu)
-{
- struct task_struct *g, *p;
-
- read_lock(&tasklist_lock);
- for_each_process_thread(g, p) {
- if (!p->on_rq)
- continue;
- /*
- * We do the check with unlocked task_rq(p)->lock.
- * Order the reading to do not warn about a task,
- * which was running on this cpu in the past, and
- * it's just been woken on another cpu.
- */
- rmb();
- if (task_cpu(p) != dead_cpu)
- continue;
-
- pr_warn("Task %s (pid=%d) is on cpu %d (state=%ld, flags=%x)\n",
- p->comm, task_pid_nr(p), dead_cpu, p->state, p->flags);
- }
- read_unlock(&tasklist_lock);
-}
-
/* Take this CPU down. */
static int take_cpu_down(void *_param)
{
/*
* So now all preempt/rcu users must observe !cpu_active().
*/
- err = stop_machine(take_cpu_down, NULL, cpumask_of(cpu));
+ err = stop_machine_cpuslocked(take_cpu_down, NULL, cpumask_of(cpu));
if (err) {
/* CPU refused to die */
irq_unlock_sparse();
if (!cpu_present(cpu))
return -EINVAL;
- cpu_hotplug_begin();
+ cpus_write_lock();
cpuhp_tasks_frozen = tasks_frozen;
}
out:
- cpu_hotplug_done();
+ cpus_write_unlock();
return ret;
}
struct task_struct *idle;
int ret = 0;
- cpu_hotplug_begin();
+ cpus_write_lock();
if (!cpu_present(cpu)) {
ret = -EINVAL;
target = min((int)target, CPUHP_BRINGUP_CPU);
ret = cpuhp_up_callbacks(cpu, st, target);
out:
- cpu_hotplug_done();
+ cpus_write_unlock();
return ret;
}
.startup.single = smpboot_unpark_threads,
.teardown.single = NULL,
},
+ [CPUHP_AP_IRQ_AFFINITY_ONLINE] = {
+ .name = "irq/affinity:online",
+ .startup.single = irq_affinity_online_cpu,
+ .teardown.single = NULL,
+ },
[CPUHP_AP_PERF_ONLINE] = {
.name = "perf:online",
.startup.single = perf_event_init_cpu,
}
}
-int __cpuhp_state_add_instance(enum cpuhp_state state, struct hlist_node *node,
- bool invoke)
+int __cpuhp_state_add_instance_cpuslocked(enum cpuhp_state state,
+ struct hlist_node *node,
+ bool invoke)
{
struct cpuhp_step *sp;
int cpu;
int ret;
+ lockdep_assert_cpus_held();
+
sp = cpuhp_get_step(state);
if (sp->multi_instance == false)
return -EINVAL;
- get_online_cpus();
mutex_lock(&cpuhp_state_mutex);
if (!invoke || !sp->startup.multi)
hlist_add_head(node, &sp->list);
unlock:
mutex_unlock(&cpuhp_state_mutex);
- put_online_cpus();
+ return ret;
+}
+
+int __cpuhp_state_add_instance(enum cpuhp_state state, struct hlist_node *node,
+ bool invoke)
+{
+ int ret;
+
+ cpus_read_lock();
+ ret = __cpuhp_state_add_instance_cpuslocked(state, node, invoke);
+ cpus_read_unlock();
return ret;
}
EXPORT_SYMBOL_GPL(__cpuhp_state_add_instance);
/**
- * __cpuhp_setup_state - Setup the callbacks for an hotplug machine state
+ * __cpuhp_setup_state_cpuslocked - Setup the callbacks for an hotplug machine state
* @state: The state to setup
* @invoke: If true, the startup function is invoked for cpus where
* cpu state >= @state
* @multi_instance: State is set up for multiple instances which get
* added afterwards.
*
+ * The caller needs to hold cpus read locked while calling this function.
* Returns:
* On success:
* Positive state number if @state is CPUHP_AP_ONLINE_DYN
* 0 for all other states
* On failure: proper (negative) error code
*/
-int __cpuhp_setup_state(enum cpuhp_state state,
- const char *name, bool invoke,
- int (*startup)(unsigned int cpu),
- int (*teardown)(unsigned int cpu),
- bool multi_instance)
+int __cpuhp_setup_state_cpuslocked(enum cpuhp_state state,
+ const char *name, bool invoke,
+ int (*startup)(unsigned int cpu),
+ int (*teardown)(unsigned int cpu),
+ bool multi_instance)
{
int cpu, ret = 0;
bool dynstate;
+ lockdep_assert_cpus_held();
+
if (cpuhp_cb_check(state) || !name)
return -EINVAL;
- get_online_cpus();
mutex_lock(&cpuhp_state_mutex);
ret = cpuhp_store_callbacks(state, name, startup, teardown,
}
out:
mutex_unlock(&cpuhp_state_mutex);
- put_online_cpus();
/*
* If the requested state is CPUHP_AP_ONLINE_DYN, return the
* dynamically allocated state in case of success.
return state;
return ret;
}
+EXPORT_SYMBOL(__cpuhp_setup_state_cpuslocked);
+
+int __cpuhp_setup_state(enum cpuhp_state state,
+ const char *name, bool invoke,
+ int (*startup)(unsigned int cpu),
+ int (*teardown)(unsigned int cpu),
+ bool multi_instance)
+{
+ int ret;
+
+ cpus_read_lock();
+ ret = __cpuhp_setup_state_cpuslocked(state, name, invoke, startup,
+ teardown, multi_instance);
+ cpus_read_unlock();
+ return ret;
+}
EXPORT_SYMBOL(__cpuhp_setup_state);
int __cpuhp_state_remove_instance(enum cpuhp_state state,
if (!sp->multi_instance)
return -EINVAL;
- get_online_cpus();
+ cpus_read_lock();
mutex_lock(&cpuhp_state_mutex);
if (!invoke || !cpuhp_get_teardown_cb(state))
remove:
hlist_del(node);
mutex_unlock(&cpuhp_state_mutex);
- put_online_cpus();
+ cpus_read_unlock();
return 0;
}
EXPORT_SYMBOL_GPL(__cpuhp_state_remove_instance);
/**
- * __cpuhp_remove_state - Remove the callbacks for an hotplug machine state
+ * __cpuhp_remove_state_cpuslocked - Remove the callbacks for an hotplug machine state
* @state: The state to remove
* @invoke: If true, the teardown function is invoked for cpus where
* cpu state >= @state
*
+ * The caller needs to hold cpus read locked while calling this function.
* The teardown callback is currently not allowed to fail. Think
* about module removal!
*/
-void __cpuhp_remove_state(enum cpuhp_state state, bool invoke)
+void __cpuhp_remove_state_cpuslocked(enum cpuhp_state state, bool invoke)
{
struct cpuhp_step *sp = cpuhp_get_step(state);
int cpu;
BUG_ON(cpuhp_cb_check(state));
- get_online_cpus();
+ lockdep_assert_cpus_held();
mutex_lock(&cpuhp_state_mutex);
if (sp->multi_instance) {
remove:
cpuhp_store_callbacks(state, NULL, NULL, NULL, false);
mutex_unlock(&cpuhp_state_mutex);
- put_online_cpus();
+}
+EXPORT_SYMBOL(__cpuhp_remove_state_cpuslocked);
+
+void __cpuhp_remove_state(enum cpuhp_state state, bool invoke)
+{
+ cpus_read_lock();
+ __cpuhp_remove_state_cpuslocked(state, invoke);
+ cpus_read_unlock();
}
EXPORT_SYMBOL(__cpuhp_remove_state);
NULL
};
-static struct attribute_group cpuhp_cpu_attr_group = {
+static const struct attribute_group cpuhp_cpu_attr_group = {
.attrs = cpuhp_cpu_attrs,
.name = "hotplug",
NULL
NULL
};
-static struct attribute_group cpuhp_cpu_root_attr_group = {
+static const struct attribute_group cpuhp_cpu_root_attr_group = {
.attrs = cpuhp_cpu_root_attrs,
.name = "hotplug",
NULL
-/* Task credentials management - see Documentation/security/credentials.txt
+/* Task credentials management - see Documentation/security/credentials.rst
*
* Copyright (C) 2008 Red Hat, Inc. All Rights Reserved.
* Written by David Howells (dhowells@redhat.com)
static LIST_HEAD(pmus);
static DEFINE_MUTEX(pmus_lock);
static struct srcu_struct pmus_srcu;
+static cpumask_var_t perf_online_mask;
/*
* perf event paranoia level:
return 0;
}
-static inline u64 perf_cgroup_event_cgrp_time(struct perf_event *event)
-{
- return 0;
-}
-
static inline void update_cgrp_time_from_event(struct perf_event *event)
{
}
if (perf_paranoid_cpu() && !capable(CAP_SYS_ADMIN))
return ERR_PTR(-EACCES);
- /*
- * We could be clever and allow to attach a event to an
- * offline CPU and activate it when the CPU comes up, but
- * that's for later.
- */
- if (!cpu_online(cpu))
- return ERR_PTR(-ENODEV);
-
cpuctx = per_cpu_ptr(pmu->pmu_cpu_context, cpu);
ctx = &cpuctx->ctx;
get_ctx(ctx);
__output_copy(handle, values, n * sizeof(u64));
}
-/*
- * XXX PERF_FORMAT_GROUP vs inherited events seems difficult.
- */
static void perf_output_read_group(struct perf_output_handle *handle,
struct perf_event *event,
u64 enabled, u64 running)
#define PERF_FORMAT_TOTAL_TIMES (PERF_FORMAT_TOTAL_TIME_ENABLED|\
PERF_FORMAT_TOTAL_TIME_RUNNING)
+/*
+ * XXX PERF_SAMPLE_READ vs inherited events seems difficult.
+ *
+ * The problem is that its both hard and excessively expensive to iterate the
+ * child list, not to mention that its impossible to IPI the children running
+ * on another CPU, from interrupt/NMI context.
+ */
static void perf_output_read(struct perf_output_handle *handle,
struct perf_event *event)
{
int err = 0;
mutex_lock(&swhash->hlist_mutex);
- if (!swevent_hlist_deref(swhash) && cpu_online(cpu)) {
+ if (!swevent_hlist_deref(swhash) &&
+ cpumask_test_cpu(cpu, perf_online_mask)) {
struct swevent_hlist *hlist;
hlist = kzalloc(sizeof(*hlist), GFP_KERNEL);
{
int err, cpu, failed_cpu;
- get_online_cpus();
+ mutex_lock(&pmus_lock);
for_each_possible_cpu(cpu) {
err = swevent_hlist_get_cpu(cpu);
if (err) {
goto fail;
}
}
- put_online_cpus();
-
+ mutex_unlock(&pmus_lock);
return 0;
fail:
for_each_possible_cpu(cpu) {
break;
swevent_hlist_put_cpu(cpu);
}
-
- put_online_cpus();
+ mutex_unlock(&pmus_lock);
return err;
}
pmu->hrtimer_interval_ms = timer;
/* update all cpuctx for this PMU */
- get_online_cpus();
+ cpus_read_lock();
for_each_online_cpu(cpu) {
struct perf_cpu_context *cpuctx;
cpuctx = per_cpu_ptr(pmu->pmu_cpu_context, cpu);
cpu_function_call(cpu,
(remote_function_f)perf_mux_hrtimer_restart, cpuctx);
}
- put_online_cpus();
+ cpus_read_unlock();
mutex_unlock(&mux_interval_mutex);
return count;
lockdep_set_class(&cpuctx->ctx.mutex, &cpuctx_mutex);
lockdep_set_class(&cpuctx->ctx.lock, &cpuctx_lock);
cpuctx->ctx.pmu = pmu;
+ cpuctx->online = cpumask_test_cpu(cpu, perf_online_mask);
__perf_mux_hrtimer_init(cpuctx, cpu);
}
static struct pmu *perf_init_event(struct perf_event *event)
{
- struct pmu *pmu = NULL;
+ struct pmu *pmu;
int idx;
int ret;
local64_set(&hwc->period_left, hwc->sample_period);
/*
- * we currently do not support PERF_FORMAT_GROUP on inherited events
+ * We currently do not support PERF_SAMPLE_READ on inherited events.
+ * See perf_output_read().
*/
- if (attr->inherit && (attr->read_format & PERF_FORMAT_GROUP))
+ if (attr->inherit && (attr->sample_type & PERF_SAMPLE_READ))
goto err_ns;
if (!has_branch_stack(event))
}
pmu = perf_init_event(event);
- if (!pmu)
- goto err_ns;
- else if (IS_ERR(pmu)) {
+ if (IS_ERR(pmu)) {
err = PTR_ERR(pmu);
goto err_ns;
}
event->addr_filters_offs = kcalloc(pmu->nr_addr_filters,
sizeof(unsigned long),
GFP_KERNEL);
- if (!event->addr_filters_offs)
+ if (!event->addr_filters_offs) {
+ err = -ENOMEM;
goto err_per_task;
+ }
/* force hw sync on the address filters */
event->addr_filters_gen = 1;
goto err_task;
}
- get_online_cpus();
-
if (task) {
err = mutex_lock_interruptible(&task->signal->cred_guard_mutex);
if (err)
- goto err_cpus;
+ goto err_task;
/*
* Reuse ptrace permission checks for now.
goto err_locked;
}
+ if (!task) {
+ /*
+ * Check if the @cpu we're creating an event for is online.
+ *
+ * We use the perf_cpu_context::ctx::mutex to serialize against
+ * the hotplug notifiers. See perf_event_{init,exit}_cpu().
+ */
+ struct perf_cpu_context *cpuctx =
+ container_of(ctx, struct perf_cpu_context, ctx);
+
+ if (!cpuctx->online) {
+ err = -ENODEV;
+ goto err_locked;
+ }
+ }
+
+
/*
* Must be under the same ctx::mutex as perf_install_in_context(),
* because we need to serialize with concurrent event creation.
put_task_struct(task);
}
- put_online_cpus();
-
mutex_lock(¤t->perf_event_mutex);
list_add_tail(&event->owner_entry, ¤t->perf_event_list);
mutex_unlock(¤t->perf_event_mutex);
err_cred:
if (task)
mutex_unlock(&task->signal->cred_guard_mutex);
-err_cpus:
- put_online_cpus();
err_task:
if (task)
put_task_struct(task);
goto err_unlock;
}
+ if (!task) {
+ /*
+ * Check if the @cpu we're creating an event for is online.
+ *
+ * We use the perf_cpu_context::ctx::mutex to serialize against
+ * the hotplug notifiers. See perf_event_{init,exit}_cpu().
+ */
+ struct perf_cpu_context *cpuctx =
+ container_of(ctx, struct perf_cpu_context, ctx);
+ if (!cpuctx->online) {
+ err = -ENODEV;
+ goto err_unlock;
+ }
+ }
+
if (!exclusive_event_installable(event, ctx)) {
err = -EBUSY;
goto err_unlock;
struct swevent_htable *swhash;
int cpu;
+ zalloc_cpumask_var(&perf_online_mask, GFP_KERNEL);
+
for_each_possible_cpu(cpu) {
swhash = &per_cpu(swevent_htable, cpu);
mutex_init(&swhash->hlist_mutex);
}
}
-int perf_event_init_cpu(unsigned int cpu)
+void perf_swevent_init_cpu(unsigned int cpu)
{
struct swevent_htable *swhash = &per_cpu(swevent_htable, cpu);
rcu_assign_pointer(swhash->swevent_hlist, hlist);
}
mutex_unlock(&swhash->hlist_mutex);
- return 0;
}
#if defined CONFIG_HOTPLUG_CPU || defined CONFIG_KEXEC_CORE
static void perf_event_exit_cpu_context(int cpu)
{
+ struct perf_cpu_context *cpuctx;
struct perf_event_context *ctx;
struct pmu *pmu;
- int idx;
- idx = srcu_read_lock(&pmus_srcu);
- list_for_each_entry_rcu(pmu, &pmus, entry) {
- ctx = &per_cpu_ptr(pmu->pmu_cpu_context, cpu)->ctx;
+ mutex_lock(&pmus_lock);
+ list_for_each_entry(pmu, &pmus, entry) {
+ cpuctx = per_cpu_ptr(pmu->pmu_cpu_context, cpu);
+ ctx = &cpuctx->ctx;
mutex_lock(&ctx->mutex);
smp_call_function_single(cpu, __perf_event_exit_context, ctx, 1);
+ cpuctx->online = 0;
mutex_unlock(&ctx->mutex);
}
- srcu_read_unlock(&pmus_srcu, idx);
+ cpumask_clear_cpu(cpu, perf_online_mask);
+ mutex_unlock(&pmus_lock);
}
#else
#endif
+int perf_event_init_cpu(unsigned int cpu)
+{
+ struct perf_cpu_context *cpuctx;
+ struct perf_event_context *ctx;
+ struct pmu *pmu;
+
+ perf_swevent_init_cpu(cpu);
+
+ mutex_lock(&pmus_lock);
+ cpumask_set_cpu(cpu, perf_online_mask);
+ list_for_each_entry(pmu, &pmus, entry) {
+ cpuctx = per_cpu_ptr(pmu->pmu_cpu_context, cpu);
+ ctx = &cpuctx->ctx;
+
+ mutex_lock(&ctx->mutex);
+ cpuctx->online = 1;
+ mutex_unlock(&ctx->mutex);
+ }
+ mutex_unlock(&pmus_lock);
+
+ return 0;
+}
+
int perf_event_exit_cpu(unsigned int cpu)
{
perf_event_exit_cpu_context(cpu);
int ret = -ENOMEM, max_order = 0;
if (!has_aux(event))
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
if (event->pmu->capabilities & PERF_PMU_CAP_AUX_NO_SG) {
/*
rcu_read_unlock();
}
-struct task_struct *try_get_task_struct(struct task_struct **ptask)
-{
- struct task_struct *task;
-
- rcu_read_lock();
- task = task_rcu_dereference(ptask);
- if (task)
- get_task_struct(task);
- rcu_read_unlock();
-
- return task;
-}
-
/*
* Determine if a process group is "orphaned", according to the POSIX
* definition in 2.2.2.52. Orphaned process groups are not to be affected
int __user *wo_stat;
struct rusage __user *wo_rusage;
- wait_queue_t child_wait;
+ wait_queue_entry_t child_wait;
int notask_error;
};
return 0;
}
-static int child_wait_callback(wait_queue_t *wait, unsigned mode,
+static int child_wait_callback(wait_queue_entry_t *wait, unsigned mode,
int sync, void *key)
{
struct wait_opts *wo = container_of(wait, struct wait_opts,
addr < (unsigned long)_etext)
return 1;
- if (system_state == SYSTEM_BOOTING &&
+ if (system_state < SYSTEM_RUNNING &&
init_kernel_text(addr))
return 1;
return 0;
* @requeue_pi_key: the requeue_pi target futex key
* @bitset: bitset for the optional bitmasked wakeup
*
- * We use this hashed waitqueue, instead of a normal wait_queue_t, so
+ * We use this hashed waitqueue, instead of a normal wait_queue_entry_t, so
* we can wake only the relevant ones (hashed queues may be shared).
*
* A futex_q has a woken state, just like tasks have TASK_RUNNING.
*
* Return: a negative error code or 0
*
- * The key words are stored in *key on success.
+ * The key words are stored in @key on success.
*
* For shared mappings, it's (page->index, file_inode(vma->vm_file),
* offset_within_page). For private mappings, it's (uaddr, current->mm).
* @set_waiters: force setting the FUTEX_WAITERS bit (1) or not (0)
*
* Return:
- * 0 - ready to wait;
- * 1 - acquired the lock;
- * <0 - error
+ * - 0 - ready to wait;
+ * - 1 - acquired the lock;
+ * - <0 - error
*
* The hb->lock and futex_key refs shall be held by the caller.
*/
* hb1 and hb2 must be held by the caller.
*
* Return:
- * 0 - failed to acquire the lock atomically;
- * >0 - acquired the lock, return value is vpid of the top_waiter
- * <0 - error
+ * - 0 - failed to acquire the lock atomically;
+ * - >0 - acquired the lock, return value is vpid of the top_waiter
+ * - <0 - error
*/
static int futex_proxy_trylock_atomic(u32 __user *pifutex,
struct futex_hash_bucket *hb1,
* uaddr2 atomically on behalf of the top waiter.
*
* Return:
- * >=0 - on success, the number of tasks requeued or woken;
- * <0 - on error
+ * - >=0 - on success, the number of tasks requeued or woken;
+ * - <0 - on error
*/
static int futex_requeue(u32 __user *uaddr1, unsigned int flags,
u32 __user *uaddr2, int nr_wake, int nr_requeue,
* be paired with exactly one earlier call to queue_me().
*
* Return:
- * 1 - if the futex_q was still queued (and we removed unqueued it);
- * 0 - if the futex_q was already removed by the waking thread
+ * - 1 - if the futex_q was still queued (and we removed unqueued it);
+ * - 0 - if the futex_q was already removed by the waking thread
*/
static int unqueue_me(struct futex_q *q)
{
* acquire the lock. Must be called with the hb lock held.
*
* Return:
- * 1 - success, lock taken;
- * 0 - success, lock not taken;
- * <0 - on error (-EFAULT)
+ * - 1 - success, lock taken;
+ * - 0 - success, lock not taken;
+ * - <0 - on error (-EFAULT)
*/
static int fixup_owner(u32 __user *uaddr, struct futex_q *q, int locked)
{
* with no q.key reference on failure.
*
* Return:
- * 0 - uaddr contains val and hb has been locked;
- * <1 - -EFAULT or -EWOULDBLOCK (uaddr does not contain val) and hb is unlocked
+ * - 0 - uaddr contains val and hb has been locked;
+ * - <1 - -EFAULT or -EWOULDBLOCK (uaddr does not contain val) and hb is unlocked
*/
static int futex_wait_setup(u32 __user *uaddr, u32 val, unsigned int flags,
struct futex_q *q, struct futex_hash_bucket **hb)
* called with the hb lock held.
*
* Return:
- * 0 = no early wakeup detected;
- * <0 = -ETIMEDOUT or -ERESTARTNOINTR
+ * - 0 = no early wakeup detected;
+ * - <0 = -ETIMEDOUT or -ERESTARTNOINTR
*/
static inline
int handle_early_requeue_pi_wakeup(struct futex_hash_bucket *hb,
* If 4 or 7, we cleanup and return with -ETIMEDOUT.
*
* Return:
- * 0 - On success;
- * <0 - On error
+ * - 0 - On success;
+ * - <0 - On error
*/
static int futex_wait_requeue_pi(u32 __user *uaddr, unsigned int flags,
u32 val, ktime_t *abs_time, u32 bitset,
config GENERIC_IRQ_SHOW_LEVEL
bool
+# Supports effective affinity mask
+config GENERIC_IRQ_EFFECTIVE_AFF_MASK
+ bool
+
# Facility to allocate a hardware interrupt. This is legacy support
# and should not be used in new code. Use irq domains instead.
config GENERIC_IRQ_LEGACY_ALLOC_HWIRQ
config HANDLE_DOMAIN_IRQ
bool
+config IRQ_TIMINGS
+ bool
+
config IRQ_DOMAIN_DEBUG
bool "Expose hardware/virtual IRQ mapping via debugfs"
depends on IRQ_DOMAIN && DEBUG_FS
If you don't know what to do here, say N.
+config GENERIC_IRQ_DEBUGFS
+ bool "Expose irq internals in debugfs"
+ depends on DEBUG_FS
+ default n
+ ---help---
+
+ Exposes internal state information through debugfs. Mostly for
+ developers and debugging of hard to diagnose interrupt problems.
+
+ If you don't know what to do here, say N.
+
endmenu
obj-y := irqdesc.o handle.o manage.o spurious.o resend.o chip.o dummychip.o devres.o
+obj-$(CONFIG_IRQ_TIMINGS) += timings.o
obj-$(CONFIG_GENERIC_IRQ_CHIP) += generic-chip.o
obj-$(CONFIG_GENERIC_IRQ_PROBE) += autoprobe.o
obj-$(CONFIG_IRQ_DOMAIN) += irqdomain.o
obj-$(CONFIG_GENERIC_MSI_IRQ) += msi.o
obj-$(CONFIG_GENERIC_IRQ_IPI) += ipi.o
obj-$(CONFIG_SMP) += affinity.o
+obj-$(CONFIG_GENERIC_IRQ_DEBUGFS) += debugfs.o
-
+/*
+ * Copyright (C) 2016 Thomas Gleixner.
+ * Copyright (C) 2016-2017 Christoph Hellwig.
+ */
#include <linux/interrupt.h>
#include <linux/kernel.h>
#include <linux/slab.h>
}
}
-static int get_nodes_in_cpumask(const struct cpumask *mask, nodemask_t *nodemsk)
+static cpumask_var_t *alloc_node_to_present_cpumask(void)
+{
+ cpumask_var_t *masks;
+ int node;
+
+ masks = kcalloc(nr_node_ids, sizeof(cpumask_var_t), GFP_KERNEL);
+ if (!masks)
+ return NULL;
+
+ for (node = 0; node < nr_node_ids; node++) {
+ if (!zalloc_cpumask_var(&masks[node], GFP_KERNEL))
+ goto out_unwind;
+ }
+
+ return masks;
+
+out_unwind:
+ while (--node >= 0)
+ free_cpumask_var(masks[node]);
+ kfree(masks);
+ return NULL;
+}
+
+static void free_node_to_present_cpumask(cpumask_var_t *masks)
+{
+ int node;
+
+ for (node = 0; node < nr_node_ids; node++)
+ free_cpumask_var(masks[node]);
+ kfree(masks);
+}
+
+static void build_node_to_present_cpumask(cpumask_var_t *masks)
+{
+ int cpu;
+
+ for_each_present_cpu(cpu)
+ cpumask_set_cpu(cpu, masks[cpu_to_node(cpu)]);
+}
+
+static int get_nodes_in_cpumask(cpumask_var_t *node_to_present_cpumask,
+ const struct cpumask *mask, nodemask_t *nodemsk)
{
int n, nodes = 0;
/* Calculate the number of nodes in the supplied affinity mask */
- for_each_online_node(n) {
- if (cpumask_intersects(mask, cpumask_of_node(n))) {
+ for_each_node(n) {
+ if (cpumask_intersects(mask, node_to_present_cpumask[n])) {
node_set(n, *nodemsk);
nodes++;
}
int last_affv = affv + affd->pre_vectors;
nodemask_t nodemsk = NODE_MASK_NONE;
struct cpumask *masks;
- cpumask_var_t nmsk;
+ cpumask_var_t nmsk, *node_to_present_cpumask;
if (!zalloc_cpumask_var(&nmsk, GFP_KERNEL))
return NULL;
if (!masks)
goto out;
+ node_to_present_cpumask = alloc_node_to_present_cpumask();
+ if (!node_to_present_cpumask)
+ goto out;
+
/* Fill out vectors at the beginning that don't need affinity */
for (curvec = 0; curvec < affd->pre_vectors; curvec++)
cpumask_copy(masks + curvec, irq_default_affinity);
/* Stabilize the cpumasks */
get_online_cpus();
- nodes = get_nodes_in_cpumask(cpu_online_mask, &nodemsk);
+ build_node_to_present_cpumask(node_to_present_cpumask);
+ nodes = get_nodes_in_cpumask(node_to_present_cpumask, cpu_present_mask,
+ &nodemsk);
/*
* If the number of nodes in the mask is greater than or equal the
*/
if (affv <= nodes) {
for_each_node_mask(n, nodemsk) {
- cpumask_copy(masks + curvec, cpumask_of_node(n));
+ cpumask_copy(masks + curvec,
+ node_to_present_cpumask[n]);
if (++curvec == last_affv)
break;
}
vecs_per_node = (affv - (curvec - affd->pre_vectors)) / nodes;
/* Get the cpus on this node which are in the mask */
- cpumask_and(nmsk, cpu_online_mask, cpumask_of_node(n));
+ cpumask_and(nmsk, cpu_present_mask, node_to_present_cpumask[n]);
/* Calculate the number of cpus per vector */
ncpus = cpumask_weight(nmsk);
/* Fill out vectors at the end that don't need affinity */
for (; curvec < nvecs; curvec++)
cpumask_copy(masks + curvec, irq_default_affinity);
+ free_node_to_present_cpumask(node_to_present_cpumask);
out:
free_cpumask_var(nmsk);
return masks;
{
int resv = affd->pre_vectors + affd->post_vectors;
int vecs = maxvec - resv;
- int cpus;
+ int ret;
- /* Stabilize the cpumasks */
get_online_cpus();
- cpus = cpumask_weight(cpu_online_mask);
+ ret = min_t(int, cpumask_weight(cpu_present_mask), vecs) + resv;
put_online_cpus();
-
- return min(cpus, vecs) + resv;
+ return ret;
}
if (desc->irq_data.chip->irq_set_type)
desc->irq_data.chip->irq_set_type(&desc->irq_data,
IRQ_TYPE_PROBE);
- irq_startup(desc, false);
+ irq_startup(desc, IRQ_NORESEND, IRQ_START_FORCE);
}
raw_spin_unlock_irq(&desc->lock);
}
raw_spin_lock_irq(&desc->lock);
if (!desc->action && irq_settings_can_probe(desc)) {
desc->istate |= IRQS_AUTODETECT | IRQS_WAITING;
- if (irq_startup(desc, false))
+ if (irq_startup(desc, IRQ_NORESEND, IRQ_START_FORCE))
desc->istate |= IRQS_PENDING;
}
raw_spin_unlock_irq(&desc->lock);
* This file contains the core interrupt handling code, for irq-chip
* based architectures.
*
- * Detailed information is available in Documentation/DocBook/genericirq
+ * Detailed information is available in Documentation/core-api/genericirq.rst
*/
#include <linux/irq.h>
irqd_set(&desc->irq_data, IRQD_IRQ_MASKED);
}
-int irq_startup(struct irq_desc *desc, bool resend)
+static void irq_state_clr_started(struct irq_desc *desc)
{
- int ret = 0;
+ irqd_clear(&desc->irq_data, IRQD_IRQ_STARTED);
+}
- irq_state_clr_disabled(desc);
- desc->depth = 0;
+static void irq_state_set_started(struct irq_desc *desc)
+{
+ irqd_set(&desc->irq_data, IRQD_IRQ_STARTED);
+}
+
+enum {
+ IRQ_STARTUP_NORMAL,
+ IRQ_STARTUP_MANAGED,
+ IRQ_STARTUP_ABORT,
+};
+
+#ifdef CONFIG_SMP
+static int
+__irq_startup_managed(struct irq_desc *desc, struct cpumask *aff, bool force)
+{
+ struct irq_data *d = irq_desc_get_irq_data(desc);
+
+ if (!irqd_affinity_is_managed(d))
+ return IRQ_STARTUP_NORMAL;
+
+ irqd_clr_managed_shutdown(d);
- irq_domain_activate_irq(&desc->irq_data);
- if (desc->irq_data.chip->irq_startup) {
- ret = desc->irq_data.chip->irq_startup(&desc->irq_data);
+ if (cpumask_any_and(aff, cpu_online_mask) > nr_cpu_ids) {
+ /*
+ * Catch code which fiddles with enable_irq() on a managed
+ * and potentially shutdown IRQ. Chained interrupt
+ * installment or irq auto probing should not happen on
+ * managed irqs either. Emit a warning, break the affinity
+ * and start it up as a normal interrupt.
+ */
+ if (WARN_ON_ONCE(force))
+ return IRQ_STARTUP_NORMAL;
+ /*
+ * The interrupt was requested, but there is no online CPU
+ * in it's affinity mask. Put it into managed shutdown
+ * state and let the cpu hotplug mechanism start it up once
+ * a CPU in the mask becomes available.
+ */
+ irqd_set_managed_shutdown(d);
+ return IRQ_STARTUP_ABORT;
+ }
+ return IRQ_STARTUP_MANAGED;
+}
+#else
+static int
+__irq_startup_managed(struct irq_desc *desc, struct cpumask *aff, bool force)
+{
+ return IRQ_STARTUP_NORMAL;
+}
+#endif
+
+static int __irq_startup(struct irq_desc *desc)
+{
+ struct irq_data *d = irq_desc_get_irq_data(desc);
+ int ret = 0;
+
+ irq_domain_activate_irq(d);
+ if (d->chip->irq_startup) {
+ ret = d->chip->irq_startup(d);
+ irq_state_clr_disabled(desc);
irq_state_clr_masked(desc);
} else {
irq_enable(desc);
}
+ irq_state_set_started(desc);
+ return ret;
+}
+
+int irq_startup(struct irq_desc *desc, bool resend, bool force)
+{
+ struct irq_data *d = irq_desc_get_irq_data(desc);
+ struct cpumask *aff = irq_data_get_affinity_mask(d);
+ int ret = 0;
+
+ desc->depth = 0;
+
+ if (irqd_is_started(d)) {
+ irq_enable(desc);
+ } else {
+ switch (__irq_startup_managed(desc, aff, force)) {
+ case IRQ_STARTUP_NORMAL:
+ ret = __irq_startup(desc);
+ irq_setup_affinity(desc);
+ break;
+ case IRQ_STARTUP_MANAGED:
+ ret = __irq_startup(desc);
+ irq_set_affinity_locked(d, aff, false);
+ break;
+ case IRQ_STARTUP_ABORT:
+ return 0;
+ }
+ }
if (resend)
check_irq_resend(desc);
+
return ret;
}
+static void __irq_disable(struct irq_desc *desc, bool mask);
+
void irq_shutdown(struct irq_desc *desc)
{
- irq_state_set_disabled(desc);
- desc->depth = 1;
- if (desc->irq_data.chip->irq_shutdown)
- desc->irq_data.chip->irq_shutdown(&desc->irq_data);
- else if (desc->irq_data.chip->irq_disable)
- desc->irq_data.chip->irq_disable(&desc->irq_data);
- else
- desc->irq_data.chip->irq_mask(&desc->irq_data);
+ if (irqd_is_started(&desc->irq_data)) {
+ desc->depth = 1;
+ if (desc->irq_data.chip->irq_shutdown) {
+ desc->irq_data.chip->irq_shutdown(&desc->irq_data);
+ irq_state_set_disabled(desc);
+ irq_state_set_masked(desc);
+ } else {
+ __irq_disable(desc, true);
+ }
+ irq_state_clr_started(desc);
+ }
+ /*
+ * This must be called even if the interrupt was never started up,
+ * because the activation can happen before the interrupt is
+ * available for request/startup. It has it's own state tracking so
+ * it's safe to call it unconditionally.
+ */
irq_domain_deactivate_irq(&desc->irq_data);
- irq_state_set_masked(desc);
}
void irq_enable(struct irq_desc *desc)
{
- irq_state_clr_disabled(desc);
- if (desc->irq_data.chip->irq_enable)
- desc->irq_data.chip->irq_enable(&desc->irq_data);
- else
- desc->irq_data.chip->irq_unmask(&desc->irq_data);
- irq_state_clr_masked(desc);
+ if (!irqd_irq_disabled(&desc->irq_data)) {
+ unmask_irq(desc);
+ } else {
+ irq_state_clr_disabled(desc);
+ if (desc->irq_data.chip->irq_enable) {
+ desc->irq_data.chip->irq_enable(&desc->irq_data);
+ irq_state_clr_masked(desc);
+ } else {
+ unmask_irq(desc);
+ }
+ }
+}
+
+static void __irq_disable(struct irq_desc *desc, bool mask)
+{
+ if (irqd_irq_disabled(&desc->irq_data)) {
+ if (mask)
+ mask_irq(desc);
+ } else {
+ irq_state_set_disabled(desc);
+ if (desc->irq_data.chip->irq_disable) {
+ desc->irq_data.chip->irq_disable(&desc->irq_data);
+ irq_state_set_masked(desc);
+ } else if (mask) {
+ mask_irq(desc);
+ }
+ }
}
/**
*/
void irq_disable(struct irq_desc *desc)
{
- irq_state_set_disabled(desc);
- if (desc->irq_data.chip->irq_disable) {
- desc->irq_data.chip->irq_disable(&desc->irq_data);
- irq_state_set_masked(desc);
- } else if (irq_settings_disable_unlazy(desc)) {
- mask_irq(desc);
- }
+ __irq_disable(desc, irq_settings_disable_unlazy(desc));
}
void irq_percpu_enable(struct irq_desc *desc, unsigned int cpu)
static inline void mask_ack_irq(struct irq_desc *desc)
{
- if (desc->irq_data.chip->irq_mask_ack)
+ if (desc->irq_data.chip->irq_mask_ack) {
desc->irq_data.chip->irq_mask_ack(&desc->irq_data);
- else {
- desc->irq_data.chip->irq_mask(&desc->irq_data);
+ irq_state_set_masked(desc);
+ } else {
+ mask_irq(desc);
if (desc->irq_data.chip->irq_ack)
desc->irq_data.chip->irq_ack(&desc->irq_data);
}
- irq_state_set_masked(desc);
}
void mask_irq(struct irq_desc *desc)
{
+ if (irqd_irq_masked(&desc->irq_data))
+ return;
+
if (desc->irq_data.chip->irq_mask) {
desc->irq_data.chip->irq_mask(&desc->irq_data);
irq_state_set_masked(desc);
void unmask_irq(struct irq_desc *desc)
{
+ if (!irqd_irq_masked(&desc->irq_data))
+ return;
+
if (desc->irq_data.chip->irq_unmask) {
desc->irq_data.chip->irq_unmask(&desc->irq_data);
irq_state_clr_masked(desc);
if (chip->flags & IRQCHIP_EOI_THREADED)
chip->irq_eoi(&desc->irq_data);
- if (chip->irq_unmask) {
- chip->irq_unmask(&desc->irq_data);
- irq_state_clr_masked(desc);
- }
+ unmask_irq(desc);
}
/*
irq_settings_set_norequest(desc);
irq_settings_set_nothread(desc);
desc->action = &chained_action;
- irq_startup(desc, true);
+ irq_startup(desc, IRQ_RESEND, IRQ_START_FORCE);
}
}
if (!desc)
return;
+
+ /*
+ * Warn when a driver sets the no autoenable flag on an already
+ * active interrupt.
+ */
+ WARN_ON_ONCE(!desc->depth && (set & _IRQ_NOAUTOEN));
+
irq_settings_clr_and_set(desc, clr, set);
irqd_clear(&desc->irq_data, IRQD_NO_BALANCING | IRQD_PER_CPU |
#include "internals.h"
+/* For !GENERIC_IRQ_EFFECTIVE_AFF_MASK this looks at general affinity mask */
+static inline bool irq_needs_fixup(struct irq_data *d)
+{
+ const struct cpumask *m = irq_data_get_effective_affinity_mask(d);
+
+ return cpumask_test_cpu(smp_processor_id(), m);
+}
+
static bool migrate_one_irq(struct irq_desc *desc)
{
struct irq_data *d = irq_desc_get_irq_data(desc);
- const struct cpumask *affinity = d->common->affinity;
- struct irq_chip *c;
- bool ret = false;
+ struct irq_chip *chip = irq_data_get_irq_chip(d);
+ bool maskchip = !irq_can_move_pcntxt(d) && !irqd_irq_masked(d);
+ const struct cpumask *affinity;
+ bool brokeaff = false;
+ int err;
/*
- * If this is a per-CPU interrupt, or the affinity does not
- * include this CPU, then we have nothing to do.
+ * IRQ chip might be already torn down, but the irq descriptor is
+ * still in the radix tree. Also if the chip has no affinity setter,
+ * nothing can be done here.
*/
- if (irqd_is_per_cpu(d) ||
- !cpumask_test_cpu(smp_processor_id(), affinity))
+ if (!chip || !chip->irq_set_affinity) {
+ pr_debug("IRQ %u: Unable to migrate away\n", d->irq);
return false;
+ }
+
+ /*
+ * No move required, if:
+ * - Interrupt is per cpu
+ * - Interrupt is not started
+ * - Affinity mask does not include this CPU.
+ *
+ * Note: Do not check desc->action as this might be a chained
+ * interrupt.
+ */
+ if (irqd_is_per_cpu(d) || !irqd_is_started(d) || !irq_needs_fixup(d)) {
+ /*
+ * If an irq move is pending, abort it if the dying CPU is
+ * the sole target.
+ */
+ irq_fixup_move_pending(desc, false);
+ return false;
+ }
+
+ /*
+ * Complete an eventually pending irq move cleanup. If this
+ * interrupt was moved in hard irq context, then the vectors need
+ * to be cleaned up. It can't wait until this interrupt actually
+ * happens and this CPU was involved.
+ */
+ irq_force_complete_move(desc);
+
+ /*
+ * If there is a setaffinity pending, then try to reuse the pending
+ * mask, so the last change of the affinity does not get lost. If
+ * there is no move pending or the pending mask does not contain
+ * any online CPU, use the current affinity mask.
+ */
+ if (irq_fixup_move_pending(desc, true))
+ affinity = irq_desc_get_pending_mask(desc);
+ else
+ affinity = irq_data_get_affinity_mask(d);
+
+ /* Mask the chip for interrupts which cannot move in process context */
+ if (maskchip && chip->irq_mask)
+ chip->irq_mask(d);
if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
+ /*
+ * If the interrupt is managed, then shut it down and leave
+ * the affinity untouched.
+ */
+ if (irqd_affinity_is_managed(d)) {
+ irqd_set_managed_shutdown(d);
+ irq_shutdown(desc);
+ return false;
+ }
affinity = cpu_online_mask;
- ret = true;
+ brokeaff = true;
}
- c = irq_data_get_irq_chip(d);
- if (!c->irq_set_affinity) {
- pr_debug("IRQ%u: unable to set affinity\n", d->irq);
- } else {
- int r = irq_do_set_affinity(d, affinity, false);
- if (r)
- pr_warn_ratelimited("IRQ%u: set affinity failed(%d).\n",
- d->irq, r);
+ err = irq_do_set_affinity(d, affinity, true);
+ if (err) {
+ pr_warn_ratelimited("IRQ%u: set affinity failed(%d).\n",
+ d->irq, err);
+ brokeaff = false;
}
- return ret;
+ if (maskchip && chip->irq_unmask)
+ chip->irq_unmask(d);
+
+ return brokeaff;
}
/**
*/
void irq_migrate_all_off_this_cpu(void)
{
- unsigned int irq;
struct irq_desc *desc;
- unsigned long flags;
-
- local_irq_save(flags);
+ unsigned int irq;
for_each_active_irq(irq) {
bool affinity_broken;
affinity_broken = migrate_one_irq(desc);
raw_spin_unlock(&desc->lock);
- if (affinity_broken)
- pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
+ if (affinity_broken) {
+ pr_warn_ratelimited("IRQ %u: no longer affine to CPU%u\n",
irq, smp_processor_id());
+ }
+ }
+}
+
+static void irq_restore_affinity_of_irq(struct irq_desc *desc, unsigned int cpu)
+{
+ struct irq_data *data = irq_desc_get_irq_data(desc);
+ const struct cpumask *affinity = irq_data_get_affinity_mask(data);
+
+ if (!irqd_affinity_is_managed(data) || !desc->action ||
+ !irq_data_get_irq_chip(data) || !cpumask_test_cpu(cpu, affinity))
+ return;
+
+ if (irqd_is_managed_and_shutdown(data)) {
+ irq_startup(desc, IRQ_RESEND, IRQ_START_COND);
+ return;
+ }
+
+ /*
+ * If the interrupt can only be directed to a single target
+ * CPU then it is already assigned to a CPU in the affinity
+ * mask. No point in trying to move it around.
+ */
+ if (!irqd_is_single_target(data))
+ irq_set_affinity_locked(data, affinity, false);
+}
+
+/**
+ * irq_affinity_online_cpu - Restore affinity for managed interrupts
+ * @cpu: Upcoming CPU for which interrupts should be restored
+ */
+int irq_affinity_online_cpu(unsigned int cpu)
+{
+ struct irq_desc *desc;
+ unsigned int irq;
+
+ irq_lock_sparse();
+ for_each_active_irq(irq) {
+ desc = irq_to_desc(irq);
+ raw_spin_lock_irq(&desc->lock);
+ irq_restore_affinity_of_irq(desc, cpu);
+ raw_spin_unlock_irq(&desc->lock);
}
+ irq_unlock_sparse();
- local_irq_restore(flags);
+ return 0;
}
--- /dev/null
+/*
+ * Copyright 2017 Thomas Gleixner <tglx@linutronix.de>
+ *
+ * This file is licensed under the GPL V2.
+ */
+#include <linux/irqdomain.h>
+#include <linux/irq.h>
+
+#include "internals.h"
+
+static struct dentry *irq_dir;
+
+struct irq_bit_descr {
+ unsigned int mask;
+ char *name;
+};
+#define BIT_MASK_DESCR(m) { .mask = m, .name = #m }
+
+static void irq_debug_show_bits(struct seq_file *m, int ind, unsigned int state,
+ const struct irq_bit_descr *sd, int size)
+{
+ int i;
+
+ for (i = 0; i < size; i++, sd++) {
+ if (state & sd->mask)
+ seq_printf(m, "%*s%s\n", ind + 12, "", sd->name);
+ }
+}
+
+#ifdef CONFIG_SMP
+static void irq_debug_show_masks(struct seq_file *m, struct irq_desc *desc)
+{
+ struct irq_data *data = irq_desc_get_irq_data(desc);
+ struct cpumask *msk;
+
+ msk = irq_data_get_affinity_mask(data);
+ seq_printf(m, "affinity: %*pbl\n", cpumask_pr_args(msk));
+#ifdef CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK
+ msk = irq_data_get_effective_affinity_mask(data);
+ seq_printf(m, "effectiv: %*pbl\n", cpumask_pr_args(msk));
+#endif
+#ifdef CONFIG_GENERIC_PENDING_IRQ
+ msk = desc->pending_mask;
+ seq_printf(m, "pending: %*pbl\n", cpumask_pr_args(msk));
+#endif
+}
+#else
+static void irq_debug_show_masks(struct seq_file *m, struct irq_desc *desc) { }
+#endif
+
+static const struct irq_bit_descr irqchip_flags[] = {
+ BIT_MASK_DESCR(IRQCHIP_SET_TYPE_MASKED),
+ BIT_MASK_DESCR(IRQCHIP_EOI_IF_HANDLED),
+ BIT_MASK_DESCR(IRQCHIP_MASK_ON_SUSPEND),
+ BIT_MASK_DESCR(IRQCHIP_ONOFFLINE_ENABLED),
+ BIT_MASK_DESCR(IRQCHIP_SKIP_SET_WAKE),
+ BIT_MASK_DESCR(IRQCHIP_ONESHOT_SAFE),
+ BIT_MASK_DESCR(IRQCHIP_EOI_THREADED),
+};
+
+static void
+irq_debug_show_chip(struct seq_file *m, struct irq_data *data, int ind)
+{
+ struct irq_chip *chip = data->chip;
+
+ if (!chip) {
+ seq_printf(m, "chip: None\n");
+ return;
+ }
+ seq_printf(m, "%*schip: %s\n", ind, "", chip->name);
+ seq_printf(m, "%*sflags: 0x%lx\n", ind + 1, "", chip->flags);
+ irq_debug_show_bits(m, ind, chip->flags, irqchip_flags,
+ ARRAY_SIZE(irqchip_flags));
+}
+
+static void
+irq_debug_show_data(struct seq_file *m, struct irq_data *data, int ind)
+{
+ seq_printf(m, "%*sdomain: %s\n", ind, "",
+ data->domain ? data->domain->name : "");
+ seq_printf(m, "%*shwirq: 0x%lx\n", ind + 1, "", data->hwirq);
+ irq_debug_show_chip(m, data, ind + 1);
+#ifdef CONFIG_IRQ_DOMAIN_HIERARCHY
+ if (!data->parent_data)
+ return;
+ seq_printf(m, "%*sparent:\n", ind + 1, "");
+ irq_debug_show_data(m, data->parent_data, ind + 4);
+#endif
+}
+
+static const struct irq_bit_descr irqdata_states[] = {
+ BIT_MASK_DESCR(IRQ_TYPE_EDGE_RISING),
+ BIT_MASK_DESCR(IRQ_TYPE_EDGE_FALLING),
+ BIT_MASK_DESCR(IRQ_TYPE_LEVEL_HIGH),
+ BIT_MASK_DESCR(IRQ_TYPE_LEVEL_LOW),
+ BIT_MASK_DESCR(IRQD_LEVEL),
+
+ BIT_MASK_DESCR(IRQD_ACTIVATED),
+ BIT_MASK_DESCR(IRQD_IRQ_STARTED),
+ BIT_MASK_DESCR(IRQD_IRQ_DISABLED),
+ BIT_MASK_DESCR(IRQD_IRQ_MASKED),
+ BIT_MASK_DESCR(IRQD_IRQ_INPROGRESS),
+
+ BIT_MASK_DESCR(IRQD_PER_CPU),
+ BIT_MASK_DESCR(IRQD_NO_BALANCING),
+
+ BIT_MASK_DESCR(IRQD_SINGLE_TARGET),
+ BIT_MASK_DESCR(IRQD_MOVE_PCNTXT),
+ BIT_MASK_DESCR(IRQD_AFFINITY_SET),
+ BIT_MASK_DESCR(IRQD_SETAFFINITY_PENDING),
+ BIT_MASK_DESCR(IRQD_AFFINITY_MANAGED),
+ BIT_MASK_DESCR(IRQD_MANAGED_SHUTDOWN),
+
+ BIT_MASK_DESCR(IRQD_FORWARDED_TO_VCPU),
+
+ BIT_MASK_DESCR(IRQD_WAKEUP_STATE),
+ BIT_MASK_DESCR(IRQD_WAKEUP_ARMED),
+};
+
+static const struct irq_bit_descr irqdesc_states[] = {
+ BIT_MASK_DESCR(_IRQ_NOPROBE),
+ BIT_MASK_DESCR(_IRQ_NOREQUEST),
+ BIT_MASK_DESCR(_IRQ_NOTHREAD),
+ BIT_MASK_DESCR(_IRQ_NOAUTOEN),
+ BIT_MASK_DESCR(_IRQ_NESTED_THREAD),
+ BIT_MASK_DESCR(_IRQ_PER_CPU_DEVID),
+ BIT_MASK_DESCR(_IRQ_IS_POLLED),
+ BIT_MASK_DESCR(_IRQ_DISABLE_UNLAZY),
+};
+
+static const struct irq_bit_descr irqdesc_istates[] = {
+ BIT_MASK_DESCR(IRQS_AUTODETECT),
+ BIT_MASK_DESCR(IRQS_SPURIOUS_DISABLED),
+ BIT_MASK_DESCR(IRQS_POLL_INPROGRESS),
+ BIT_MASK_DESCR(IRQS_ONESHOT),
+ BIT_MASK_DESCR(IRQS_REPLAY),
+ BIT_MASK_DESCR(IRQS_WAITING),
+ BIT_MASK_DESCR(IRQS_PENDING),
+ BIT_MASK_DESCR(IRQS_SUSPENDED),
+};
+
+
+static int irq_debug_show(struct seq_file *m, void *p)
+{
+ struct irq_desc *desc = m->private;
+ struct irq_data *data;
+
+ raw_spin_lock_irq(&desc->lock);
+ data = irq_desc_get_irq_data(desc);
+ seq_printf(m, "handler: %pf\n", desc->handle_irq);
+ seq_printf(m, "status: 0x%08x\n", desc->status_use_accessors);
+ irq_debug_show_bits(m, 0, desc->status_use_accessors, irqdesc_states,
+ ARRAY_SIZE(irqdesc_states));
+ seq_printf(m, "istate: 0x%08x\n", desc->istate);
+ irq_debug_show_bits(m, 0, desc->istate, irqdesc_istates,
+ ARRAY_SIZE(irqdesc_istates));
+ seq_printf(m, "ddepth: %u\n", desc->depth);
+ seq_printf(m, "wdepth: %u\n", desc->wake_depth);
+ seq_printf(m, "dstate: 0x%08x\n", irqd_get(data));
+ irq_debug_show_bits(m, 0, irqd_get(data), irqdata_states,
+ ARRAY_SIZE(irqdata_states));
+ seq_printf(m, "node: %d\n", irq_data_get_node(data));
+ irq_debug_show_masks(m, desc);
+ irq_debug_show_data(m, data, 0);
+ raw_spin_unlock_irq(&desc->lock);
+ return 0;
+}
+
+static int irq_debug_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, irq_debug_show, inode->i_private);
+}
+
+static const struct file_operations dfs_irq_ops = {
+ .open = irq_debug_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+void irq_add_debugfs_entry(unsigned int irq, struct irq_desc *desc)
+{
+ char name [10];
+
+ if (!irq_dir || !desc || desc->debugfs_file)
+ return;
+
+ sprintf(name, "%d", irq);
+ desc->debugfs_file = debugfs_create_file(name, 0444, irq_dir, desc,
+ &dfs_irq_ops);
+}
+
+static int __init irq_debugfs_init(void)
+{
+ struct dentry *root_dir;
+ int irq;
+
+ root_dir = debugfs_create_dir("irq", NULL);
+ if (!root_dir)
+ return -ENOMEM;
+
+ irq_domain_debugfs_init(root_dir);
+
+ irq_dir = debugfs_create_dir("irqs", root_dir);
+
+ irq_lock_sparse();
+ for_each_active_irq(irq)
+ irq_add_debugfs_entry(irq, irq_to_desc(irq));
+ irq_unlock_sparse();
+
+ return 0;
+}
+__initcall(irq_debugfs_init);
#include <linux/gfp.h>
#include <linux/irq.h>
+#include "internals.h"
+
/*
* Device resource management aware IRQ request/free implementation.
*/
return base;
}
EXPORT_SYMBOL_GPL(__devm_irq_alloc_descs);
+
+#ifdef CONFIG_GENERIC_IRQ_CHIP
+/**
+ * devm_irq_alloc_generic_chip - Allocate and initialize a generic chip
+ * for a managed device
+ * @dev: Device to allocate the generic chip for
+ * @name: Name of the irq chip
+ * @num_ct: Number of irq_chip_type instances associated with this
+ * @irq_base: Interrupt base nr for this chip
+ * @reg_base: Register base address (virtual)
+ * @handler: Default flow handler associated with this chip
+ *
+ * Returns an initialized irq_chip_generic structure. The chip defaults
+ * to the primary (index 0) irq_chip_type and @handler
+ */
+struct irq_chip_generic *
+devm_irq_alloc_generic_chip(struct device *dev, const char *name, int num_ct,
+ unsigned int irq_base, void __iomem *reg_base,
+ irq_flow_handler_t handler)
+{
+ struct irq_chip_generic *gc;
+ unsigned long sz = sizeof(*gc) + num_ct * sizeof(struct irq_chip_type);
+
+ gc = devm_kzalloc(dev, sz, GFP_KERNEL);
+ if (gc)
+ irq_init_generic_chip(gc, name, num_ct,
+ irq_base, reg_base, handler);
+
+ return gc;
+}
+EXPORT_SYMBOL_GPL(devm_irq_alloc_generic_chip);
+
+struct irq_generic_chip_devres {
+ struct irq_chip_generic *gc;
+ u32 msk;
+ unsigned int clr;
+ unsigned int set;
+};
+
+static void devm_irq_remove_generic_chip(struct device *dev, void *res)
+{
+ struct irq_generic_chip_devres *this = res;
+
+ irq_remove_generic_chip(this->gc, this->msk, this->clr, this->set);
+}
+
+/**
+ * devm_irq_setup_generic_chip - Setup a range of interrupts with a generic
+ * chip for a managed device
+ *
+ * @dev: Device to setup the generic chip for
+ * @gc: Generic irq chip holding all data
+ * @msk: Bitmask holding the irqs to initialize relative to gc->irq_base
+ * @flags: Flags for initialization
+ * @clr: IRQ_* bits to clear
+ * @set: IRQ_* bits to set
+ *
+ * Set up max. 32 interrupts starting from gc->irq_base. Note, this
+ * initializes all interrupts to the primary irq_chip_type and its
+ * associated handler.
+ */
+int devm_irq_setup_generic_chip(struct device *dev, struct irq_chip_generic *gc,
+ u32 msk, enum irq_gc_flags flags,
+ unsigned int clr, unsigned int set)
+{
+ struct irq_generic_chip_devres *dr;
+
+ dr = devres_alloc(devm_irq_remove_generic_chip,
+ sizeof(*dr), GFP_KERNEL);
+ if (!dr)
+ return -ENOMEM;
+
+ irq_setup_generic_chip(gc, msk, flags, clr, set);
+
+ dr->gc = gc;
+ dr->msk = msk;
+ dr->clr = clr;
+ dr->set = set;
+ devres_add(dev, dr);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(devm_irq_setup_generic_chip);
+#endif /* CONFIG_GENERIC_IRQ_CHIP */
iowrite32be(val, addr);
}
-static void
-irq_init_generic_chip(struct irq_chip_generic *gc, const char *name,
- int num_ct, unsigned int irq_base,
- void __iomem *reg_base, irq_flow_handler_t handler)
+void irq_init_generic_chip(struct irq_chip_generic *gc, const char *name,
+ int num_ct, unsigned int irq_base,
+ void __iomem *reg_base, irq_flow_handler_t handler)
{
raw_spin_lock_init(&gc->lock);
gc->num_ct = num_ct;
*
* This file contains the core interrupt handling code.
*
- * Detailed information is available in Documentation/DocBook/genericirq
+ * Detailed information is available in Documentation/core-api/genericirq.rst
*
*/
unsigned int irq = desc->irq_data.irq;
struct irqaction *action;
+ record_irq_time(desc);
+
for_each_action_of_desc(desc, action) {
irqreturn_t res;
#include <linux/irqdesc.h>
#include <linux/kernel_stat.h>
#include <linux/pm_runtime.h>
+#include <linux/sched/clock.h>
#ifdef CONFIG_SPARSE_IRQ
# define IRQ_BITMAP_BITS (NR_IRQS + 8196)
IRQS_WAITING = 0x00000080,
IRQS_PENDING = 0x00000200,
IRQS_SUSPENDED = 0x00000800,
+ IRQS_TIMINGS = 0x00001000,
};
#include "debug.h"
extern void __disable_irq(struct irq_desc *desc);
extern void __enable_irq(struct irq_desc *desc);
-extern int irq_startup(struct irq_desc *desc, bool resend);
+#define IRQ_RESEND true
+#define IRQ_NORESEND false
+
+#define IRQ_START_FORCE true
+#define IRQ_START_COND false
+
+extern int irq_startup(struct irq_desc *desc, bool resend, bool force);
+
extern void irq_shutdown(struct irq_desc *desc);
extern void irq_enable(struct irq_desc *desc);
extern void irq_disable(struct irq_desc *desc);
extern bool irq_can_set_affinity_usr(unsigned int irq);
-extern int irq_select_affinity_usr(unsigned int irq, struct cpumask *mask);
+extern int irq_select_affinity_usr(unsigned int irq);
extern void irq_set_thread_affinity(struct irq_desc *desc);
extern int irq_do_set_affinity(struct irq_data *data,
const struct cpumask *dest, bool force);
+#ifdef CONFIG_SMP
+extern int irq_setup_affinity(struct irq_desc *desc);
+#else
+static inline int irq_setup_affinity(struct irq_desc *desc) { return 0; }
+#endif
+
/* Inline functions for support of irq chips on slow busses */
static inline void chip_bus_lock(struct irq_desc *desc)
{
#define __irqd_to_state(d) ACCESS_PRIVATE((d)->common, state_use_accessors)
+static inline unsigned int irqd_get(struct irq_data *d)
+{
+ return __irqd_to_state(d);
+}
+
/*
* Manipulation functions for irq_data.state
*/
__irqd_to_state(d) &= ~IRQD_SETAFFINITY_PENDING;
}
+static inline void irqd_set_managed_shutdown(struct irq_data *d)
+{
+ __irqd_to_state(d) |= IRQD_MANAGED_SHUTDOWN;
+}
+
+static inline void irqd_clr_managed_shutdown(struct irq_data *d)
+{
+ __irqd_to_state(d) &= ~IRQD_MANAGED_SHUTDOWN;
+}
+
static inline void irqd_clear(struct irq_data *d, unsigned int mask)
{
__irqd_to_state(d) &= ~mask;
static inline void
irq_pm_remove_action(struct irq_desc *desc, struct irqaction *action) { }
#endif
+
+#ifdef CONFIG_IRQ_TIMINGS
+
+#define IRQ_TIMINGS_SHIFT 5
+#define IRQ_TIMINGS_SIZE (1 << IRQ_TIMINGS_SHIFT)
+#define IRQ_TIMINGS_MASK (IRQ_TIMINGS_SIZE - 1)
+
+/**
+ * struct irq_timings - irq timings storing structure
+ * @values: a circular buffer of u64 encoded <timestamp,irq> values
+ * @count: the number of elements in the array
+ */
+struct irq_timings {
+ u64 values[IRQ_TIMINGS_SIZE];
+ int count;
+};
+
+DECLARE_PER_CPU(struct irq_timings, irq_timings);
+
+extern void irq_timings_free(int irq);
+extern int irq_timings_alloc(int irq);
+
+static inline void irq_remove_timings(struct irq_desc *desc)
+{
+ desc->istate &= ~IRQS_TIMINGS;
+
+ irq_timings_free(irq_desc_get_irq(desc));
+}
+
+static inline void irq_setup_timings(struct irq_desc *desc, struct irqaction *act)
+{
+ int irq = irq_desc_get_irq(desc);
+ int ret;
+
+ /*
+ * We don't need the measurement because the idle code already
+ * knows the next expiry event.
+ */
+ if (act->flags & __IRQF_TIMER)
+ return;
+
+ /*
+ * In case the timing allocation fails, we just want to warn,
+ * not fail, so letting the system boot anyway.
+ */
+ ret = irq_timings_alloc(irq);
+ if (ret) {
+ pr_warn("Failed to allocate irq timing stats for irq%d (%d)",
+ irq, ret);
+ return;
+ }
+
+ desc->istate |= IRQS_TIMINGS;
+}
+
+extern void irq_timings_enable(void);
+extern void irq_timings_disable(void);
+
+DECLARE_STATIC_KEY_FALSE(irq_timing_enabled);
+
+/*
+ * The interrupt number and the timestamp are encoded into a single
+ * u64 variable to optimize the size.
+ * 48 bit time stamp and 16 bit IRQ number is way sufficient.
+ * Who cares an IRQ after 78 hours of idle time?
+ */
+static inline u64 irq_timing_encode(u64 timestamp, int irq)
+{
+ return (timestamp << 16) | irq;
+}
+
+static inline int irq_timing_decode(u64 value, u64 *timestamp)
+{
+ *timestamp = value >> 16;
+ return value & U16_MAX;
+}
+
+/*
+ * The function record_irq_time is only called in one place in the
+ * interrupts handler. We want this function always inline so the code
+ * inside is embedded in the function and the static key branching
+ * code can act at the higher level. Without the explicit
+ * __always_inline we can end up with a function call and a small
+ * overhead in the hotpath for nothing.
+ */
+static __always_inline void record_irq_time(struct irq_desc *desc)
+{
+ if (!static_branch_likely(&irq_timing_enabled))
+ return;
+
+ if (desc->istate & IRQS_TIMINGS) {
+ struct irq_timings *timings = this_cpu_ptr(&irq_timings);
+
+ timings->values[timings->count & IRQ_TIMINGS_MASK] =
+ irq_timing_encode(local_clock(),
+ irq_desc_get_irq(desc));
+
+ timings->count++;
+ }
+}
+#else
+static inline void irq_remove_timings(struct irq_desc *desc) {}
+static inline void irq_setup_timings(struct irq_desc *desc,
+ struct irqaction *act) {};
+static inline void record_irq_time(struct irq_desc *desc) {}
+#endif /* CONFIG_IRQ_TIMINGS */
+
+
+#ifdef CONFIG_GENERIC_IRQ_CHIP
+void irq_init_generic_chip(struct irq_chip_generic *gc, const char *name,
+ int num_ct, unsigned int irq_base,
+ void __iomem *reg_base, irq_flow_handler_t handler);
+#else
+static inline void
+irq_init_generic_chip(struct irq_chip_generic *gc, const char *name,
+ int num_ct, unsigned int irq_base,
+ void __iomem *reg_base, irq_flow_handler_t handler) { }
+#endif /* CONFIG_GENERIC_IRQ_CHIP */
+
+#ifdef CONFIG_GENERIC_PENDING_IRQ
+static inline bool irq_can_move_pcntxt(struct irq_data *data)
+{
+ return irqd_can_move_in_process_context(data);
+}
+static inline bool irq_move_pending(struct irq_data *data)
+{
+ return irqd_is_setaffinity_pending(data);
+}
+static inline void
+irq_copy_pending(struct irq_desc *desc, const struct cpumask *mask)
+{
+ cpumask_copy(desc->pending_mask, mask);
+}
+static inline void
+irq_get_pending(struct cpumask *mask, struct irq_desc *desc)
+{
+ cpumask_copy(mask, desc->pending_mask);
+}
+static inline struct cpumask *irq_desc_get_pending_mask(struct irq_desc *desc)
+{
+ return desc->pending_mask;
+}
+bool irq_fixup_move_pending(struct irq_desc *desc, bool force_clear);
+#else /* CONFIG_GENERIC_PENDING_IRQ */
+static inline bool irq_can_move_pcntxt(struct irq_data *data)
+{
+ return true;
+}
+static inline bool irq_move_pending(struct irq_data *data)
+{
+ return false;
+}
+static inline void
+irq_copy_pending(struct irq_desc *desc, const struct cpumask *mask)
+{
+}
+static inline void
+irq_get_pending(struct cpumask *mask, struct irq_desc *desc)
+{
+}
+static inline struct cpumask *irq_desc_get_pending_mask(struct irq_desc *desc)
+{
+ return NULL;
+}
+static inline bool irq_fixup_move_pending(struct irq_desc *desc, bool fclear)
+{
+ return false;
+}
+#endif /* !CONFIG_GENERIC_PENDING_IRQ */
+
+#ifdef CONFIG_GENERIC_IRQ_DEBUGFS
+#include <linux/debugfs.h>
+
+void irq_add_debugfs_entry(unsigned int irq, struct irq_desc *desc);
+static inline void irq_remove_debugfs_entry(struct irq_desc *desc)
+{
+ debugfs_remove(desc->debugfs_file);
+}
+# ifdef CONFIG_IRQ_DOMAIN
+void irq_domain_debugfs_init(struct dentry *root);
+# else
+static inline void irq_domain_debugfs_init(struct dentry *root);
+# endif
+#else /* CONFIG_GENERIC_IRQ_DEBUGFS */
+static inline void irq_add_debugfs_entry(unsigned int irq, struct irq_desc *d)
+{
+}
+static inline void irq_remove_debugfs_entry(struct irq_desc *d)
+{
+}
+#endif /* CONFIG_GENERIC_IRQ_DEBUGFS */
*
* This file contains the interrupt descriptor management code
*
- * Detailed information is available in Documentation/DocBook/genericirq
+ * Detailed information is available in Documentation/core-api/genericirq.rst
*
*/
#include <linux/irq.h>
#endif
#ifdef CONFIG_SMP
-static int alloc_masks(struct irq_desc *desc, gfp_t gfp, int node)
+static int alloc_masks(struct irq_desc *desc, int node)
{
if (!zalloc_cpumask_var_node(&desc->irq_common_data.affinity,
- gfp, node))
+ GFP_KERNEL, node))
return -ENOMEM;
+#ifdef CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK
+ if (!zalloc_cpumask_var_node(&desc->irq_common_data.effective_affinity,
+ GFP_KERNEL, node)) {
+ free_cpumask_var(desc->irq_common_data.affinity);
+ return -ENOMEM;
+ }
+#endif
+
#ifdef CONFIG_GENERIC_PENDING_IRQ
- if (!zalloc_cpumask_var_node(&desc->pending_mask, gfp, node)) {
+ if (!zalloc_cpumask_var_node(&desc->pending_mask, GFP_KERNEL, node)) {
+#ifdef CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK
+ free_cpumask_var(desc->irq_common_data.effective_affinity);
+#endif
free_cpumask_var(desc->irq_common_data.affinity);
return -ENOMEM;
}
#else
static inline int
-alloc_masks(struct irq_desc *desc, gfp_t gfp, int node) { return 0; }
+alloc_masks(struct irq_desc *desc, int node) { return 0; }
static inline void
desc_smp_init(struct irq_desc *desc, int node, const struct cpumask *affinity) { }
#endif
desc->irq_data.chip_data = NULL;
irq_settings_clr_and_set(desc, ~0, _IRQ_DEFAULT_INIT_FLAGS);
irqd_set(&desc->irq_data, IRQD_IRQ_DISABLED);
+ irqd_set(&desc->irq_data, IRQD_IRQ_MASKED);
desc->handle_irq = handle_bad_irq;
desc->depth = 1;
desc->irq_count = 0;
free_cpumask_var(desc->pending_mask);
#endif
free_cpumask_var(desc->irq_common_data.affinity);
+#ifdef CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK
+ free_cpumask_var(desc->irq_common_data.effective_affinity);
+#endif
}
#else
static inline void free_masks(struct irq_desc *desc) { }
struct module *owner)
{
struct irq_desc *desc;
- gfp_t gfp = GFP_KERNEL;
- desc = kzalloc_node(sizeof(*desc), gfp, node);
+ desc = kzalloc_node(sizeof(*desc), GFP_KERNEL, node);
if (!desc)
return NULL;
/* allocate based on nr_cpu_ids */
if (!desc->kstat_irqs)
goto err_desc;
- if (alloc_masks(desc, gfp, node))
+ if (alloc_masks(desc, node))
goto err_kstat;
raw_spin_lock_init(&desc->lock);
{
struct irq_desc *desc = irq_to_desc(irq);
+ irq_remove_debugfs_entry(desc);
unregister_irq_proc(irq, desc);
/*
/* Let arch update nr_irqs and return the nr of preallocated irqs */
initcnt = arch_probe_nr_irqs();
- printk(KERN_INFO "NR_IRQS:%d nr_irqs:%d %d\n", NR_IRQS, nr_irqs, initcnt);
+ printk(KERN_INFO "NR_IRQS: %d, nr_irqs: %d, preallocated irqs: %d\n",
+ NR_IRQS, nr_irqs, initcnt);
if (WARN_ON(nr_irqs > IRQ_BITMAP_BITS))
nr_irqs = IRQ_BITMAP_BITS;
init_irq_default_affinity();
- printk(KERN_INFO "NR_IRQS:%d\n", NR_IRQS);
+ printk(KERN_INFO "NR_IRQS: %d\n", NR_IRQS);
desc = irq_desc;
count = ARRAY_SIZE(irq_desc);
for (i = 0; i < count; i++) {
desc[i].kstat_irqs = alloc_percpu(unsigned int);
- alloc_masks(&desc[i], GFP_KERNEL, node);
+ alloc_masks(&desc[i], node);
raw_spin_lock_init(&desc[i].lock);
lockdep_set_class(&desc[i].lock, &irq_desc_lock_class);
desc_set_defaults(i, &desc[i], node, NULL, NULL);
static void irq_domain_check_hierarchy(struct irq_domain *domain);
struct irqchip_fwid {
- struct fwnode_handle fwnode;
- char *name;
+ struct fwnode_handle fwnode;
+ unsigned int type;
+ char *name;
void *data;
};
+#ifdef CONFIG_GENERIC_IRQ_DEBUGFS
+static void debugfs_add_domain_dir(struct irq_domain *d);
+static void debugfs_remove_domain_dir(struct irq_domain *d);
+#else
+static inline void debugfs_add_domain_dir(struct irq_domain *d) { }
+static inline void debugfs_remove_domain_dir(struct irq_domain *d) { }
+#endif
+
/**
* irq_domain_alloc_fwnode - Allocate a fwnode_handle suitable for
* identifying an irq domain
- * @data: optional user-provided data
+ * @type: Type of irqchip_fwnode. See linux/irqdomain.h
+ * @name: Optional user provided domain name
+ * @id: Optional user provided id if name != NULL
+ * @data: Optional user-provided data
*
- * Allocate a struct device_node, and return a poiner to the embedded
+ * Allocate a struct irqchip_fwid, and return a poiner to the embedded
* fwnode_handle (or NULL on failure).
+ *
+ * Note: The types IRQCHIP_FWNODE_NAMED and IRQCHIP_FWNODE_NAMED_ID are
+ * solely to transport name information to irqdomain creation code. The
+ * node is not stored. For other types the pointer is kept in the irq
+ * domain struct.
*/
-struct fwnode_handle *irq_domain_alloc_fwnode(void *data)
+struct fwnode_handle *__irq_domain_alloc_fwnode(unsigned int type, int id,
+ const char *name, void *data)
{
struct irqchip_fwid *fwid;
- char *name;
+ char *n;
fwid = kzalloc(sizeof(*fwid), GFP_KERNEL);
- name = kasprintf(GFP_KERNEL, "irqchip@%p", data);
- if (!fwid || !name) {
+ switch (type) {
+ case IRQCHIP_FWNODE_NAMED:
+ n = kasprintf(GFP_KERNEL, "%s", name);
+ break;
+ case IRQCHIP_FWNODE_NAMED_ID:
+ n = kasprintf(GFP_KERNEL, "%s-%d", name, id);
+ break;
+ default:
+ n = kasprintf(GFP_KERNEL, "irqchip@%p", data);
+ break;
+ }
+
+ if (!fwid || !n) {
kfree(fwid);
- kfree(name);
+ kfree(n);
return NULL;
}
- fwid->name = name;
+ fwid->type = type;
+ fwid->name = n;
fwid->data = data;
fwid->fwnode.type = FWNODE_IRQCHIP;
return &fwid->fwnode;
}
-EXPORT_SYMBOL_GPL(irq_domain_alloc_fwnode);
+EXPORT_SYMBOL_GPL(__irq_domain_alloc_fwnode);
/**
* irq_domain_free_fwnode - Free a non-OF-backed fwnode_handle
void *host_data)
{
struct device_node *of_node = to_of_node(fwnode);
+ struct irqchip_fwid *fwid;
struct irq_domain *domain;
+ static atomic_t unknown_domains;
+
domain = kzalloc_node(sizeof(*domain) + (sizeof(unsigned int) * size),
GFP_KERNEL, of_node_to_nid(of_node));
if (WARN_ON(!domain))
return NULL;
+ if (fwnode && is_fwnode_irqchip(fwnode)) {
+ fwid = container_of(fwnode, struct irqchip_fwid, fwnode);
+
+ switch (fwid->type) {
+ case IRQCHIP_FWNODE_NAMED:
+ case IRQCHIP_FWNODE_NAMED_ID:
+ domain->name = kstrdup(fwid->name, GFP_KERNEL);
+ if (!domain->name) {
+ kfree(domain);
+ return NULL;
+ }
+ domain->flags |= IRQ_DOMAIN_NAME_ALLOCATED;
+ break;
+ default:
+ domain->fwnode = fwnode;
+ domain->name = fwid->name;
+ break;
+ }
+ } else if (of_node) {
+ char *name;
+
+ /*
+ * DT paths contain '/', which debugfs is legitimately
+ * unhappy about. Replace them with ':', which does
+ * the trick and is not as offensive as '\'...
+ */
+ name = kstrdup(of_node_full_name(of_node), GFP_KERNEL);
+ if (!name) {
+ kfree(domain);
+ return NULL;
+ }
+
+ strreplace(name, '/', ':');
+
+ domain->name = name;
+ domain->fwnode = fwnode;
+ domain->flags |= IRQ_DOMAIN_NAME_ALLOCATED;
+ }
+
+ if (!domain->name) {
+ if (fwnode) {
+ pr_err("Invalid fwnode type (%d) for irqdomain\n",
+ fwnode->type);
+ }
+ domain->name = kasprintf(GFP_KERNEL, "unknown-%d",
+ atomic_inc_return(&unknown_domains));
+ if (!domain->name) {
+ kfree(domain);
+ return NULL;
+ }
+ domain->flags |= IRQ_DOMAIN_NAME_ALLOCATED;
+ }
+
of_node_get(of_node);
/* Fill structure */
INIT_RADIX_TREE(&domain->revmap_tree, GFP_KERNEL);
domain->ops = ops;
domain->host_data = host_data;
- domain->fwnode = fwnode;
domain->hwirq_max = hwirq_max;
domain->revmap_size = size;
domain->revmap_direct_max_irq = direct_max;
irq_domain_check_hierarchy(domain);
mutex_lock(&irq_domain_mutex);
+ debugfs_add_domain_dir(domain);
list_add(&domain->link, &irq_domain_list);
mutex_unlock(&irq_domain_mutex);
void irq_domain_remove(struct irq_domain *domain)
{
mutex_lock(&irq_domain_mutex);
+ debugfs_remove_domain_dir(domain);
WARN_ON(!radix_tree_empty(&domain->revmap_tree));
pr_debug("Removed domain %s\n", domain->name);
of_node_put(irq_domain_get_of_node(domain));
+ if (domain->flags & IRQ_DOMAIN_NAME_ALLOCATED)
+ kfree(domain->name);
kfree(domain);
}
EXPORT_SYMBOL_GPL(irq_domain_remove);
+void irq_domain_update_bus_token(struct irq_domain *domain,
+ enum irq_domain_bus_token bus_token)
+{
+ char *name;
+
+ if (domain->bus_token == bus_token)
+ return;
+
+ mutex_lock(&irq_domain_mutex);
+
+ domain->bus_token = bus_token;
+
+ name = kasprintf(GFP_KERNEL, "%s-%d", domain->name, bus_token);
+ if (!name) {
+ mutex_unlock(&irq_domain_mutex);
+ return;
+ }
+
+ debugfs_remove_domain_dir(domain);
+
+ if (domain->flags & IRQ_DOMAIN_NAME_ALLOCATED)
+ kfree(domain->name);
+ else
+ domain->flags |= IRQ_DOMAIN_NAME_ALLOCATED;
+
+ domain->name = name;
+ debugfs_add_domain_dir(domain);
+
+ mutex_unlock(&irq_domain_mutex);
+}
+
/**
* irq_domain_add_simple() - Register an irq_domain and optionally map a range of irqs
* @of_node: pointer to interrupt controller's device tree node.
irq_data->domain = NULL;
irq_data->hwirq = 0;
+ domain->mapcount--;
/* Clear reverse map for this hwirq */
if (hwirq < domain->revmap_size) {
domain->name = irq_data->chip->name;
}
+ domain->mapcount++;
if (hwirq < domain->revmap_size) {
domain->linear_revmap[hwirq] = virq;
} else {
EXPORT_SYMBOL_GPL(irq_find_mapping);
#ifdef CONFIG_IRQ_DOMAIN_DEBUG
+static void virq_debug_show_one(struct seq_file *m, struct irq_desc *desc)
+{
+ struct irq_domain *domain;
+ struct irq_data *data;
+
+ domain = desc->irq_data.domain;
+ data = &desc->irq_data;
+
+ while (domain) {
+ unsigned int irq = data->irq;
+ unsigned long hwirq = data->hwirq;
+ struct irq_chip *chip;
+ bool direct;
+
+ if (data == &desc->irq_data)
+ seq_printf(m, "%5d ", irq);
+ else
+ seq_printf(m, "%5d+ ", irq);
+ seq_printf(m, "0x%05lx ", hwirq);
+
+ chip = irq_data_get_irq_chip(data);
+ seq_printf(m, "%-15s ", (chip && chip->name) ? chip->name : "none");
+
+ seq_printf(m, data ? "0x%p " : " %p ",
+ irq_data_get_irq_chip_data(data));
+
+ seq_printf(m, " %c ", (desc->action && desc->action->handler) ? '*' : ' ');
+ direct = (irq == hwirq) && (irq < domain->revmap_direct_max_irq);
+ seq_printf(m, "%6s%-8s ",
+ (hwirq < domain->revmap_size) ? "LINEAR" : "RADIX",
+ direct ? "(DIRECT)" : "");
+ seq_printf(m, "%s\n", domain->name);
+#ifdef CONFIG_IRQ_DOMAIN_HIERARCHY
+ domain = domain->parent;
+ data = data->parent_data;
+#else
+ domain = NULL;
+#endif
+ }
+}
+
static int virq_debug_show(struct seq_file *m, void *private)
{
unsigned long flags;
struct irq_desc *desc;
struct irq_domain *domain;
struct radix_tree_iter iter;
- void *data, **slot;
+ void **slot;
int i;
seq_printf(m, " %-16s %-6s %-10s %-10s %s\n",
mutex_lock(&irq_domain_mutex);
list_for_each_entry(domain, &irq_domain_list, link) {
struct device_node *of_node;
+ const char *name;
+
int count = 0;
+
of_node = irq_domain_get_of_node(domain);
+ if (of_node)
+ name = of_node_full_name(of_node);
+ else if (is_fwnode_irqchip(domain->fwnode))
+ name = container_of(domain->fwnode, struct irqchip_fwid,
+ fwnode)->name;
+ else
+ name = "";
+
radix_tree_for_each_slot(slot, &domain->revmap_tree, &iter, 0)
count++;
seq_printf(m, "%c%-16s %6u %10u %10u %s\n",
domain == irq_default_domain ? '*' : ' ', domain->name,
domain->revmap_size + count, domain->revmap_size,
domain->revmap_direct_max_irq,
- of_node ? of_node_full_name(of_node) : "");
+ name);
}
mutex_unlock(&irq_domain_mutex);
continue;
raw_spin_lock_irqsave(&desc->lock, flags);
- domain = desc->irq_data.domain;
-
- if (domain) {
- struct irq_chip *chip;
- int hwirq = desc->irq_data.hwirq;
- bool direct;
-
- seq_printf(m, "%5d ", i);
- seq_printf(m, "0x%05x ", hwirq);
-
- chip = irq_desc_get_chip(desc);
- seq_printf(m, "%-15s ", (chip && chip->name) ? chip->name : "none");
-
- data = irq_desc_get_chip_data(desc);
- seq_printf(m, data ? "0x%p " : " %p ", data);
-
- seq_printf(m, " %c ", (desc->action && desc->action->handler) ? '*' : ' ');
- direct = (i == hwirq) && (i < domain->revmap_direct_max_irq);
- seq_printf(m, "%6s%-8s ",
- (hwirq < domain->revmap_size) ? "LINEAR" : "RADIX",
- direct ? "(DIRECT)" : "");
- seq_printf(m, "%s\n", desc->irq_data.domain->name);
- }
-
+ virq_debug_show_one(m, desc);
raw_spin_unlock_irqrestore(&desc->lock, flags);
}
struct irq_domain *domain = data->domain;
irq_hw_number_t hwirq = data->hwirq;
+ domain->mapcount++;
if (hwirq < domain->revmap_size) {
domain->linear_revmap[hwirq] = virq;
} else {
struct irq_domain *domain = data->domain;
irq_hw_number_t hwirq = data->hwirq;
+ domain->mapcount--;
if (hwirq < domain->revmap_size) {
domain->linear_revmap[hwirq] = 0;
} else {
irq_domain_free_irqs_common(domain, virq, nr_irqs);
}
-static bool irq_domain_is_auto_recursive(struct irq_domain *domain)
-{
- return domain->flags & IRQ_DOMAIN_FLAG_AUTO_RECURSIVE;
-}
-
-static void irq_domain_free_irqs_recursive(struct irq_domain *domain,
+static void irq_domain_free_irqs_hierarchy(struct irq_domain *domain,
unsigned int irq_base,
unsigned int nr_irqs)
{
domain->ops->free(domain, irq_base, nr_irqs);
- if (irq_domain_is_auto_recursive(domain)) {
- BUG_ON(!domain->parent);
- irq_domain_free_irqs_recursive(domain->parent, irq_base,
- nr_irqs);
- }
}
-int irq_domain_alloc_irqs_recursive(struct irq_domain *domain,
+int irq_domain_alloc_irqs_hierarchy(struct irq_domain *domain,
unsigned int irq_base,
unsigned int nr_irqs, void *arg)
{
- int ret = 0;
- struct irq_domain *parent = domain->parent;
- bool recursive = irq_domain_is_auto_recursive(domain);
-
- BUG_ON(recursive && !parent);
- if (recursive)
- ret = irq_domain_alloc_irqs_recursive(parent, irq_base,
- nr_irqs, arg);
- if (ret < 0)
- return ret;
-
- ret = domain->ops->alloc(domain, irq_base, nr_irqs, arg);
- if (ret < 0 && recursive)
- irq_domain_free_irqs_recursive(parent, irq_base, nr_irqs);
-
- return ret;
+ return domain->ops->alloc(domain, irq_base, nr_irqs, arg);
}
/**
}
mutex_lock(&irq_domain_mutex);
- ret = irq_domain_alloc_irqs_recursive(domain, virq, nr_irqs, arg);
+ ret = irq_domain_alloc_irqs_hierarchy(domain, virq, nr_irqs, arg);
if (ret < 0) {
mutex_unlock(&irq_domain_mutex);
goto out_free_irq_data;
mutex_lock(&irq_domain_mutex);
for (i = 0; i < nr_irqs; i++)
irq_domain_remove_irq(virq + i);
- irq_domain_free_irqs_recursive(data->domain, virq, nr_irqs);
+ irq_domain_free_irqs_hierarchy(data->domain, virq, nr_irqs);
mutex_unlock(&irq_domain_mutex);
irq_domain_free_irq_data(virq, nr_irqs);
unsigned int irq_base, unsigned int nr_irqs,
void *arg)
{
- /* irq_domain_alloc_irqs_recursive() has called parent's alloc() */
- if (irq_domain_is_auto_recursive(domain))
- return 0;
+ if (!domain->parent)
+ return -ENOSYS;
- domain = domain->parent;
- if (domain)
- return irq_domain_alloc_irqs_recursive(domain, irq_base,
- nr_irqs, arg);
- return -ENOSYS;
+ return irq_domain_alloc_irqs_hierarchy(domain->parent, irq_base,
+ nr_irqs, arg);
}
EXPORT_SYMBOL_GPL(irq_domain_alloc_irqs_parent);
void irq_domain_free_irqs_parent(struct irq_domain *domain,
unsigned int irq_base, unsigned int nr_irqs)
{
- /* irq_domain_free_irqs_recursive() will call parent's free */
- if (!irq_domain_is_auto_recursive(domain) && domain->parent)
- irq_domain_free_irqs_recursive(domain->parent, irq_base,
- nr_irqs);
+ if (!domain->parent)
+ return;
+
+ irq_domain_free_irqs_hierarchy(domain->parent, irq_base, nr_irqs);
}
EXPORT_SYMBOL_GPL(irq_domain_free_irqs_parent);
{
}
#endif /* CONFIG_IRQ_DOMAIN_HIERARCHY */
+
+#ifdef CONFIG_GENERIC_IRQ_DEBUGFS
+static struct dentry *domain_dir;
+
+static void
+irq_domain_debug_show_one(struct seq_file *m, struct irq_domain *d, int ind)
+{
+ seq_printf(m, "%*sname: %s\n", ind, "", d->name);
+ seq_printf(m, "%*ssize: %u\n", ind + 1, "",
+ d->revmap_size + d->revmap_direct_max_irq);
+ seq_printf(m, "%*smapped: %u\n", ind + 1, "", d->mapcount);
+ seq_printf(m, "%*sflags: 0x%08x\n", ind +1 , "", d->flags);
+#ifdef CONFIG_IRQ_DOMAIN_HIERARCHY
+ if (!d->parent)
+ return;
+ seq_printf(m, "%*sparent: %s\n", ind + 1, "", d->parent->name);
+ irq_domain_debug_show_one(m, d->parent, ind + 4);
+#endif
+}
+
+static int irq_domain_debug_show(struct seq_file *m, void *p)
+{
+ struct irq_domain *d = m->private;
+
+ /* Default domain? Might be NULL */
+ if (!d) {
+ if (!irq_default_domain)
+ return 0;
+ d = irq_default_domain;
+ }
+ irq_domain_debug_show_one(m, d, 0);
+ return 0;
+}
+
+static int irq_domain_debug_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, irq_domain_debug_show, inode->i_private);
+}
+
+static const struct file_operations dfs_domain_ops = {
+ .open = irq_domain_debug_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static void debugfs_add_domain_dir(struct irq_domain *d)
+{
+ if (!d->name || !domain_dir || d->debugfs_file)
+ return;
+ d->debugfs_file = debugfs_create_file(d->name, 0444, domain_dir, d,
+ &dfs_domain_ops);
+}
+
+static void debugfs_remove_domain_dir(struct irq_domain *d)
+{
+ if (d->debugfs_file)
+ debugfs_remove(d->debugfs_file);
+}
+
+void __init irq_domain_debugfs_init(struct dentry *root)
+{
+ struct irq_domain *d;
+
+ domain_dir = debugfs_create_dir("domains", root);
+ if (!domain_dir)
+ return;
+
+ debugfs_create_file("default", 0444, domain_dir, NULL, &dfs_domain_ops);
+ mutex_lock(&irq_domain_mutex);
+ list_for_each_entry(d, &irq_domain_list, link)
+ debugfs_add_domain_dir(d);
+ mutex_unlock(&irq_domain_mutex);
+}
+#endif
set_bit(IRQTF_AFFINITY, &action->thread_flags);
}
-#ifdef CONFIG_GENERIC_PENDING_IRQ
-static inline bool irq_can_move_pcntxt(struct irq_data *data)
-{
- return irqd_can_move_in_process_context(data);
-}
-static inline bool irq_move_pending(struct irq_data *data)
-{
- return irqd_is_setaffinity_pending(data);
-}
-static inline void
-irq_copy_pending(struct irq_desc *desc, const struct cpumask *mask)
-{
- cpumask_copy(desc->pending_mask, mask);
-}
-static inline void
-irq_get_pending(struct cpumask *mask, struct irq_desc *desc)
-{
- cpumask_copy(mask, desc->pending_mask);
-}
-#else
-static inline bool irq_can_move_pcntxt(struct irq_data *data) { return true; }
-static inline bool irq_move_pending(struct irq_data *data) { return false; }
-static inline void
-irq_copy_pending(struct irq_desc *desc, const struct cpumask *mask) { }
-static inline void
-irq_get_pending(struct cpumask *mask, struct irq_desc *desc) { }
-#endif
-
int irq_do_set_affinity(struct irq_data *data, const struct cpumask *mask,
bool force)
{
/*
* Generic version of the affinity autoselector.
*/
-static int setup_affinity(struct irq_desc *desc, struct cpumask *mask)
+int irq_setup_affinity(struct irq_desc *desc)
{
struct cpumask *set = irq_default_affinity;
- int node = irq_desc_get_node(desc);
+ int ret, node = irq_desc_get_node(desc);
+ static DEFINE_RAW_SPINLOCK(mask_lock);
+ static struct cpumask mask;
/* Excludes PER_CPU and NO_BALANCE interrupts */
if (!__irq_can_set_affinity(desc))
return 0;
+ raw_spin_lock(&mask_lock);
/*
* Preserve the managed affinity setting and a userspace affinity
* setup, but make sure that one of the targets is online.
irqd_clear(&desc->irq_data, IRQD_AFFINITY_SET);
}
- cpumask_and(mask, cpu_online_mask, set);
+ cpumask_and(&mask, cpu_online_mask, set);
if (node != NUMA_NO_NODE) {
const struct cpumask *nodemask = cpumask_of_node(node);
/* make sure at least one of the cpus in nodemask is online */
- if (cpumask_intersects(mask, nodemask))
- cpumask_and(mask, mask, nodemask);
+ if (cpumask_intersects(&mask, nodemask))
+ cpumask_and(&mask, &mask, nodemask);
}
- irq_do_set_affinity(&desc->irq_data, mask, false);
- return 0;
+ ret = irq_do_set_affinity(&desc->irq_data, &mask, false);
+ raw_spin_unlock(&mask_lock);
+ return ret;
}
#else
/* Wrapper for ALPHA specific affinity selector magic */
-static inline int setup_affinity(struct irq_desc *d, struct cpumask *mask)
+int irq_setup_affinity(struct irq_desc *desc)
{
- return irq_select_affinity(irq_desc_get_irq(d));
+ return irq_select_affinity(irq_desc_get_irq(desc));
}
#endif
/*
- * Called when affinity is set via /proc/irq
+ * Called when a bogus affinity is set via /proc/irq
*/
-int irq_select_affinity_usr(unsigned int irq, struct cpumask *mask)
+int irq_select_affinity_usr(unsigned int irq)
{
struct irq_desc *desc = irq_to_desc(irq);
unsigned long flags;
int ret;
raw_spin_lock_irqsave(&desc->lock, flags);
- ret = setup_affinity(desc, mask);
+ ret = irq_setup_affinity(desc);
raw_spin_unlock_irqrestore(&desc->lock, flags);
return ret;
}
-
-#else
-static inline int
-setup_affinity(struct irq_desc *desc, struct cpumask *mask)
-{
- return 0;
-}
#endif
/**
goto err_out;
/* Prevent probing on this irq: */
irq_settings_set_noprobe(desc);
- irq_enable(desc);
- check_irq_resend(desc);
- /* fall-through */
+ /*
+ * Call irq_startup() not irq_enable() here because the
+ * interrupt might be marked NOAUTOEN. So irq_startup()
+ * needs to be invoked when it gets enabled the first
+ * time. If it was already started up, then irq_startup()
+ * will invoke irq_enable() under the hood.
+ */
+ irq_startup(desc, IRQ_RESEND, IRQ_START_COND);
+ break;
}
default:
desc->depth--;
struct irqaction *old, **old_ptr;
unsigned long flags, thread_mask = 0;
int ret, nested, shared = 0;
- cpumask_var_t mask;
if (!desc)
return -EINVAL;
}
}
- if (!alloc_cpumask_var(&mask, GFP_KERNEL)) {
- ret = -ENOMEM;
- goto out_thread;
- }
-
/*
* Drivers are often written to work w/o knowledge about the
* underlying irq chip implementation, so a request for a
*/
if (thread_mask == ~0UL) {
ret = -EBUSY;
- goto out_mask;
+ goto out_unlock;
}
/*
* The thread_mask for the action is or'ed to
pr_err("Threaded irq requested with handler=NULL and !ONESHOT for irq %d\n",
irq);
ret = -EINVAL;
- goto out_mask;
+ goto out_unlock;
}
if (!shared) {
if (ret) {
pr_err("Failed to request resources for %s (irq %d) on irqchip %s\n",
new->name, irq, desc->irq_data.chip->name);
- goto out_mask;
+ goto out_unlock;
}
init_waitqueue_head(&desc->wait_for_threads);
if (ret) {
irq_release_resources(desc);
- goto out_mask;
+ goto out_unlock;
}
}
if (new->flags & IRQF_ONESHOT)
desc->istate |= IRQS_ONESHOT;
- if (irq_settings_can_autoenable(desc))
- irq_startup(desc, true);
- else
- /* Undo nested disables: */
- desc->depth = 1;
-
/* Exclude IRQ from balancing if requested */
if (new->flags & IRQF_NOBALANCING) {
irq_settings_set_no_balancing(desc);
irqd_set(&desc->irq_data, IRQD_NO_BALANCING);
}
- /* Set default affinity mask once everything is setup */
- setup_affinity(desc, mask);
+ if (irq_settings_can_autoenable(desc)) {
+ irq_startup(desc, IRQ_RESEND, IRQ_START_COND);
+ } else {
+ /*
+ * Shared interrupts do not go well with disabling
+ * auto enable. The sharing interrupt might request
+ * it while it's still disabled and then wait for
+ * interrupts forever.
+ */
+ WARN_ON_ONCE(new->flags & IRQF_SHARED);
+ /* Undo nested disables: */
+ desc->depth = 1;
+ }
} else if (new->flags & IRQF_TRIGGER_MASK) {
unsigned int nmsk = new->flags & IRQF_TRIGGER_MASK;
raw_spin_unlock_irqrestore(&desc->lock, flags);
+ irq_setup_timings(desc, new);
+
/*
* Strictly no need to wake it up, but hung_task complains
* when no hard interrupt wakes the thread up.
wake_up_process(new->secondary->thread);
register_irq_proc(irq, desc);
+ irq_add_debugfs_entry(irq, desc);
new->dir = NULL;
register_handler_proc(irq, new);
- free_cpumask_var(mask);
-
return 0;
mismatch:
}
ret = -EBUSY;
-out_mask:
+out_unlock:
raw_spin_unlock_irqrestore(&desc->lock, flags);
- free_cpumask_var(mask);
out_thread:
if (new->thread) {
irq_settings_clr_disable_unlazy(desc);
irq_shutdown(desc);
irq_release_resources(desc);
+ irq_remove_timings(desc);
}
#ifdef CONFIG_SMP
#include "internals.h"
+/**
+ * irq_fixup_move_pending - Cleanup irq move pending from a dying CPU
+ * @desc: Interrupt descpriptor to clean up
+ * @force_clear: If set clear the move pending bit unconditionally.
+ * If not set, clear it only when the dying CPU is the
+ * last one in the pending mask.
+ *
+ * Returns true if the pending bit was set and the pending mask contains an
+ * online CPU other than the dying CPU.
+ */
+bool irq_fixup_move_pending(struct irq_desc *desc, bool force_clear)
+{
+ struct irq_data *data = irq_desc_get_irq_data(desc);
+
+ if (!irqd_is_setaffinity_pending(data))
+ return false;
+
+ /*
+ * The outgoing CPU might be the last online target in a pending
+ * interrupt move. If that's the case clear the pending move bit.
+ */
+ if (cpumask_any_and(desc->pending_mask, cpu_online_mask) >= nr_cpu_ids) {
+ irqd_clr_move_pending(data);
+ return false;
+ }
+ if (force_clear)
+ irqd_clr_move_pending(data);
+ return true;
+}
+
void irq_move_masked_irq(struct irq_data *idata)
{
struct irq_desc *desc = irq_data_to_desc(idata);
struct msi_domain_info *info,
struct irq_domain *parent)
{
+ struct irq_domain *domain;
+
if (info->flags & MSI_FLAG_USE_DEF_DOM_OPS)
msi_domain_update_dom_ops(info);
if (info->flags & MSI_FLAG_USE_DEF_CHIP_OPS)
msi_domain_update_chip_ops(info);
- return irq_domain_create_hierarchy(parent, IRQ_DOMAIN_FLAG_MSI, 0,
- fwnode, &msi_domain_ops, info);
+ domain = irq_domain_create_hierarchy(parent, IRQ_DOMAIN_FLAG_MSI, 0,
+ fwnode, &msi_domain_ops, info);
+
+ if (domain && !domain->name && info->chip)
+ domain->name = info->chip->name;
+
+ return domain;
}
int msi_domain_prepare_irqs(struct irq_domain *domain, struct device *dev,
ops->set_desc(arg, desc);
/* Assumes the domain mutex is held! */
- ret = irq_domain_alloc_irqs_recursive(domain, virq, 1, arg);
+ ret = irq_domain_alloc_irqs_hierarchy(domain, virq, 1, arg);
if (ret)
break;
#ifdef CONFIG_SMP
-static int show_irq_affinity(int type, struct seq_file *m, void *v)
+enum {
+ AFFINITY,
+ AFFINITY_LIST,
+ EFFECTIVE,
+ EFFECTIVE_LIST,
+};
+
+static int show_irq_affinity(int type, struct seq_file *m)
{
struct irq_desc *desc = irq_to_desc((long)m->private);
- const struct cpumask *mask = desc->irq_common_data.affinity;
+ const struct cpumask *mask;
+ switch (type) {
+ case AFFINITY:
+ case AFFINITY_LIST:
+ mask = desc->irq_common_data.affinity;
#ifdef CONFIG_GENERIC_PENDING_IRQ
- if (irqd_is_setaffinity_pending(&desc->irq_data))
- mask = desc->pending_mask;
+ if (irqd_is_setaffinity_pending(&desc->irq_data))
+ mask = desc->pending_mask;
#endif
- if (type)
+ break;
+ case EFFECTIVE:
+ case EFFECTIVE_LIST:
+#ifdef CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK
+ mask = desc->irq_common_data.effective_affinity;
+ break;
+#else
+ return -EINVAL;
+#endif
+ };
+
+ switch (type) {
+ case AFFINITY_LIST:
+ case EFFECTIVE_LIST:
seq_printf(m, "%*pbl\n", cpumask_pr_args(mask));
- else
+ break;
+ case AFFINITY:
+ case EFFECTIVE:
seq_printf(m, "%*pb\n", cpumask_pr_args(mask));
+ break;
+ }
return 0;
}
int no_irq_affinity;
static int irq_affinity_proc_show(struct seq_file *m, void *v)
{
- return show_irq_affinity(0, m, v);
+ return show_irq_affinity(AFFINITY, m);
}
static int irq_affinity_list_proc_show(struct seq_file *m, void *v)
{
- return show_irq_affinity(1, m, v);
+ return show_irq_affinity(AFFINITY_LIST, m);
}
* one online CPU still has to be targeted.
*/
if (!cpumask_intersects(new_value, cpu_online_mask)) {
- /* Special case for empty set - allow the architecture
- code to set default SMP affinity. */
- err = irq_select_affinity_usr(irq, new_value) ? -EINVAL : count;
+ /*
+ * Special case for empty set - allow the architecture code
+ * to set default SMP affinity.
+ */
+ err = irq_select_affinity_usr(irq) ? -EINVAL : count;
} else {
irq_set_affinity(irq, new_value);
err = count;
.write = irq_affinity_list_proc_write,
};
+#ifdef CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK
+static int irq_effective_aff_proc_show(struct seq_file *m, void *v)
+{
+ return show_irq_affinity(EFFECTIVE, m);
+}
+
+static int irq_effective_aff_list_proc_show(struct seq_file *m, void *v)
+{
+ return show_irq_affinity(EFFECTIVE_LIST, m);
+}
+
+static int irq_effective_aff_proc_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, irq_effective_aff_proc_show, PDE_DATA(inode));
+}
+
+static int irq_effective_aff_list_proc_open(struct inode *inode,
+ struct file *file)
+{
+ return single_open(file, irq_effective_aff_list_proc_show,
+ PDE_DATA(inode));
+}
+
+static const struct file_operations irq_effective_aff_proc_fops = {
+ .open = irq_effective_aff_proc_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static const struct file_operations irq_effective_aff_list_proc_fops = {
+ .open = irq_effective_aff_list_proc_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+#endif
+
static int default_affinity_show(struct seq_file *m, void *v)
{
seq_printf(m, "%*pb\n", cpumask_pr_args(irq_default_affinity));
void register_irq_proc(unsigned int irq, struct irq_desc *desc)
{
static DEFINE_MUTEX(register_lock);
+ void __maybe_unused *irqp = (void *)(unsigned long) irq;
char name [MAX_NAMELEN];
if (!root_irq_dir || (desc->irq_data.chip == &no_irq_chip))
#ifdef CONFIG_SMP
/* create /proc/irq/<irq>/smp_affinity */
proc_create_data("smp_affinity", 0644, desc->dir,
- &irq_affinity_proc_fops, (void *)(long)irq);
+ &irq_affinity_proc_fops, irqp);
/* create /proc/irq/<irq>/affinity_hint */
proc_create_data("affinity_hint", 0444, desc->dir,
- &irq_affinity_hint_proc_fops, (void *)(long)irq);
+ &irq_affinity_hint_proc_fops, irqp);
/* create /proc/irq/<irq>/smp_affinity_list */
proc_create_data("smp_affinity_list", 0644, desc->dir,
- &irq_affinity_list_proc_fops, (void *)(long)irq);
+ &irq_affinity_list_proc_fops, irqp);
proc_create_data("node", 0444, desc->dir,
- &irq_node_proc_fops, (void *)(long)irq);
+ &irq_node_proc_fops, irqp);
+# ifdef CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK
+ proc_create_data("effective_affinity", 0444, desc->dir,
+ &irq_effective_aff_proc_fops, irqp);
+ proc_create_data("effective_affinity_list", 0444, desc->dir,
+ &irq_effective_aff_list_proc_fops, irqp);
+# endif
#endif
-
proc_create_data("spurious", 0444, desc->dir,
&irq_spurious_proc_fops, (void *)(long)irq);
remove_proc_entry("affinity_hint", desc->dir);
remove_proc_entry("smp_affinity_list", desc->dir);
remove_proc_entry("node", desc->dir);
+# ifdef CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK
+ remove_proc_entry("effective_affinity", desc->dir);
+ remove_proc_entry("effective_affinity_list", desc->dir);
+# endif
#endif
remove_proc_entry("spurious", desc->dir);
--- /dev/null
+/*
+ * linux/kernel/irq/timings.c
+ *
+ * Copyright (C) 2016, Linaro Ltd - Daniel Lezcano <daniel.lezcano@linaro.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ */
+#include <linux/kernel.h>
+#include <linux/percpu.h>
+#include <linux/slab.h>
+#include <linux/static_key.h>
+#include <linux/interrupt.h>
+#include <linux/idr.h>
+#include <linux/irq.h>
+#include <linux/math64.h>
+
+#include <trace/events/irq.h>
+
+#include "internals.h"
+
+DEFINE_STATIC_KEY_FALSE(irq_timing_enabled);
+
+DEFINE_PER_CPU(struct irq_timings, irq_timings);
+
+struct irqt_stat {
+ u64 next_evt;
+ u64 last_ts;
+ u64 variance;
+ u32 avg;
+ u32 nr_samples;
+ int anomalies;
+ int valid;
+};
+
+static DEFINE_IDR(irqt_stats);
+
+void irq_timings_enable(void)
+{
+ static_branch_enable(&irq_timing_enabled);
+}
+
+void irq_timings_disable(void)
+{
+ static_branch_disable(&irq_timing_enabled);
+}
+
+/**
+ * irqs_update - update the irq timing statistics with a new timestamp
+ *
+ * @irqs: an irqt_stat struct pointer
+ * @ts: the new timestamp
+ *
+ * The statistics are computed online, in other words, the code is
+ * designed to compute the statistics on a stream of values rather
+ * than doing multiple passes on the values to compute the average,
+ * then the variance. The integer division introduces a loss of
+ * precision but with an acceptable error margin regarding the results
+ * we would have with the double floating precision: we are dealing
+ * with nanosec, so big numbers, consequently the mantisse is
+ * negligeable, especially when converting the time in usec
+ * afterwards.
+ *
+ * The computation happens at idle time. When the CPU is not idle, the
+ * interrupts' timestamps are stored in the circular buffer, when the
+ * CPU goes idle and this routine is called, all the buffer's values
+ * are injected in the statistical model continuying to extend the
+ * statistics from the previous busy-idle cycle.
+ *
+ * The observations showed a device will trigger a burst of periodic
+ * interrupts followed by one or two peaks of longer time, for
+ * instance when a SD card device flushes its cache, then the periodic
+ * intervals occur again. A one second inactivity period resets the
+ * stats, that gives us the certitude the statistical values won't
+ * exceed 1x10^9, thus the computation won't overflow.
+ *
+ * Basically, the purpose of the algorithm is to watch the periodic
+ * interrupts and eliminate the peaks.
+ *
+ * An interrupt is considered periodically stable if the interval of
+ * its occurences follow the normal distribution, thus the values
+ * comply with:
+ *
+ * avg - 3 x stddev < value < avg + 3 x stddev
+ *
+ * Which can be simplified to:
+ *
+ * -3 x stddev < value - avg < 3 x stddev
+ *
+ * abs(value - avg) < 3 x stddev
+ *
+ * In order to save a costly square root computation, we use the
+ * variance. For the record, stddev = sqrt(variance). The equation
+ * above becomes:
+ *
+ * abs(value - avg) < 3 x sqrt(variance)
+ *
+ * And finally we square it:
+ *
+ * (value - avg) ^ 2 < (3 x sqrt(variance)) ^ 2
+ *
+ * (value - avg) x (value - avg) < 9 x variance
+ *
+ * Statistically speaking, any values out of this interval is
+ * considered as an anomaly and is discarded. However, a normal
+ * distribution appears when the number of samples is 30 (it is the
+ * rule of thumb in statistics, cf. "30 samples" on Internet). When
+ * there are three consecutive anomalies, the statistics are resetted.
+ *
+ */
+static void irqs_update(struct irqt_stat *irqs, u64 ts)
+{
+ u64 old_ts = irqs->last_ts;
+ u64 variance = 0;
+ u64 interval;
+ s64 diff;
+
+ /*
+ * The timestamps are absolute time values, we need to compute
+ * the timing interval between two interrupts.
+ */
+ irqs->last_ts = ts;
+
+ /*
+ * The interval type is u64 in order to deal with the same
+ * type in our computation, that prevent mindfuck issues with
+ * overflow, sign and division.
+ */
+ interval = ts - old_ts;
+
+ /*
+ * The interrupt triggered more than one second apart, that
+ * ends the sequence as predictible for our purpose. In this
+ * case, assume we have the beginning of a sequence and the
+ * timestamp is the first value. As it is impossible to
+ * predict anything at this point, return.
+ *
+ * Note the first timestamp of the sequence will always fall
+ * in this test because the old_ts is zero. That is what we
+ * want as we need another timestamp to compute an interval.
+ */
+ if (interval >= NSEC_PER_SEC) {
+ memset(irqs, 0, sizeof(*irqs));
+ irqs->last_ts = ts;
+ return;
+ }
+
+ /*
+ * Pre-compute the delta with the average as the result is
+ * used several times in this function.
+ */
+ diff = interval - irqs->avg;
+
+ /*
+ * Increment the number of samples.
+ */
+ irqs->nr_samples++;
+
+ /*
+ * Online variance divided by the number of elements if there
+ * is more than one sample. Normally the formula is division
+ * by nr_samples - 1 but we assume the number of element will be
+ * more than 32 and dividing by 32 instead of 31 is enough
+ * precise.
+ */
+ if (likely(irqs->nr_samples > 1))
+ variance = irqs->variance >> IRQ_TIMINGS_SHIFT;
+
+ /*
+ * The rule of thumb in statistics for the normal distribution
+ * is having at least 30 samples in order to have the model to
+ * apply. Values outside the interval are considered as an
+ * anomaly.
+ */
+ if ((irqs->nr_samples >= 30) && ((diff * diff) > (9 * variance))) {
+ /*
+ * After three consecutive anomalies, we reset the
+ * stats as it is no longer stable enough.
+ */
+ if (irqs->anomalies++ >= 3) {
+ memset(irqs, 0, sizeof(*irqs));
+ irqs->last_ts = ts;
+ return;
+ }
+ } else {
+ /*
+ * The anomalies must be consecutives, so at this
+ * point, we reset the anomalies counter.
+ */
+ irqs->anomalies = 0;
+ }
+
+ /*
+ * The interrupt is considered stable enough to try to predict
+ * the next event on it.
+ */
+ irqs->valid = 1;
+
+ /*
+ * Online average algorithm:
+ *
+ * new_average = average + ((value - average) / count)
+ *
+ * The variance computation depends on the new average
+ * to be computed here first.
+ *
+ */
+ irqs->avg = irqs->avg + (diff >> IRQ_TIMINGS_SHIFT);
+
+ /*
+ * Online variance algorithm:
+ *
+ * new_variance = variance + (value - average) x (value - new_average)
+ *
+ * Warning: irqs->avg is updated with the line above, hence
+ * 'interval - irqs->avg' is no longer equal to 'diff'
+ */
+ irqs->variance = irqs->variance + (diff * (interval - irqs->avg));
+
+ /*
+ * Update the next event
+ */
+ irqs->next_evt = ts + irqs->avg;
+}
+
+/**
+ * irq_timings_next_event - Return when the next event is supposed to arrive
+ *
+ * During the last busy cycle, the number of interrupts is incremented
+ * and stored in the irq_timings structure. This information is
+ * necessary to:
+ *
+ * - know if the index in the table wrapped up:
+ *
+ * If more than the array size interrupts happened during the
+ * last busy/idle cycle, the index wrapped up and we have to
+ * begin with the next element in the array which is the last one
+ * in the sequence, otherwise it is a the index 0.
+ *
+ * - have an indication of the interrupts activity on this CPU
+ * (eg. irq/sec)
+ *
+ * The values are 'consumed' after inserting in the statistical model,
+ * thus the count is reinitialized.
+ *
+ * The array of values **must** be browsed in the time direction, the
+ * timestamp must increase between an element and the next one.
+ *
+ * Returns a nanosec time based estimation of the earliest interrupt,
+ * U64_MAX otherwise.
+ */
+u64 irq_timings_next_event(u64 now)
+{
+ struct irq_timings *irqts = this_cpu_ptr(&irq_timings);
+ struct irqt_stat *irqs;
+ struct irqt_stat __percpu *s;
+ u64 ts, next_evt = U64_MAX;
+ int i, irq = 0;
+
+ /*
+ * This function must be called with the local irq disabled in
+ * order to prevent the timings circular buffer to be updated
+ * while we are reading it.
+ */
+ WARN_ON_ONCE(!irqs_disabled());
+
+ /*
+ * Number of elements in the circular buffer: If it happens it
+ * was flushed before, then the number of elements could be
+ * smaller than IRQ_TIMINGS_SIZE, so the count is used,
+ * otherwise the array size is used as we wrapped. The index
+ * begins from zero when we did not wrap. That could be done
+ * in a nicer way with the proper circular array structure
+ * type but with the cost of extra computation in the
+ * interrupt handler hot path. We choose efficiency.
+ *
+ * Inject measured irq/timestamp to the statistical model
+ * while decrementing the counter because we consume the data
+ * from our circular buffer.
+ */
+ for (i = irqts->count & IRQ_TIMINGS_MASK,
+ irqts->count = min(IRQ_TIMINGS_SIZE, irqts->count);
+ irqts->count > 0; irqts->count--, i = (i + 1) & IRQ_TIMINGS_MASK) {
+
+ irq = irq_timing_decode(irqts->values[i], &ts);
+
+ s = idr_find(&irqt_stats, irq);
+ if (s) {
+ irqs = this_cpu_ptr(s);
+ irqs_update(irqs, ts);
+ }
+ }
+
+ /*
+ * Look in the list of interrupts' statistics, the earliest
+ * next event.
+ */
+ idr_for_each_entry(&irqt_stats, s, i) {
+
+ irqs = this_cpu_ptr(s);
+
+ if (!irqs->valid)
+ continue;
+
+ if (irqs->next_evt <= now) {
+ irq = i;
+ next_evt = now;
+
+ /*
+ * This interrupt mustn't use in the future
+ * until new events occur and update the
+ * statistics.
+ */
+ irqs->valid = 0;
+ break;
+ }
+
+ if (irqs->next_evt < next_evt) {
+ irq = i;
+ next_evt = irqs->next_evt;
+ }
+ }
+
+ return next_evt;
+}
+
+void irq_timings_free(int irq)
+{
+ struct irqt_stat __percpu *s;
+
+ s = idr_find(&irqt_stats, irq);
+ if (s) {
+ free_percpu(s);
+ idr_remove(&irqt_stats, irq);
+ }
+}
+
+int irq_timings_alloc(int irq)
+{
+ struct irqt_stat __percpu *s;
+ int id;
+
+ /*
+ * Some platforms can have the same private interrupt per cpu,
+ * so this function may be be called several times with the
+ * same interrupt number. Just bail out in case the per cpu
+ * stat structure is already allocated.
+ */
+ s = idr_find(&irqt_stats, irq);
+ if (s)
+ return 0;
+
+ s = alloc_percpu(*s);
+ if (!s)
+ return -ENOMEM;
+
+ idr_preload(GFP_KERNEL);
+ id = idr_alloc(&irqt_stats, s, irq, irq + 1, GFP_NOWAIT);
+ idr_preload_end();
+
+ if (id < 0) {
+ free_percpu(s);
+ return id;
+ }
+
+ return 0;
+}
#include <linux/static_key.h>
#include <linux/jump_label_ratelimit.h>
#include <linux/bug.h>
+#include <linux/cpu.h>
#ifdef HAVE_JUMP_LABEL
return;
}
+ cpus_read_lock();
jump_label_lock();
if (atomic_read(&key->enabled) == 0) {
atomic_set(&key->enabled, -1);
atomic_inc(&key->enabled);
}
jump_label_unlock();
+ cpus_read_unlock();
}
EXPORT_SYMBOL_GPL(static_key_slow_inc);
static void __static_key_slow_dec(struct static_key *key,
unsigned long rate_limit, struct delayed_work *work)
{
+ cpus_read_lock();
/*
* The negative count check is valid even when a negative
* key->enabled is in use by static_key_slow_inc(); a
if (!atomic_dec_and_mutex_lock(&key->enabled, &jump_label_mutex)) {
WARN(atomic_read(&key->enabled) < 0,
"jump label: negative count!\n");
+ cpus_read_unlock();
return;
}
jump_label_update(key);
}
jump_label_unlock();
+ cpus_read_unlock();
}
static void jump_label_update_timeout(struct work_struct *work)
if (static_key_initialized)
return;
+ cpus_read_lock();
jump_label_lock();
jump_label_sort_entries(iter_start, iter_stop);
}
static_key_initialized = true;
jump_label_unlock();
+ cpus_read_unlock();
}
#ifdef CONFIG_MODULES
struct module *mod = data;
int ret = 0;
+ cpus_read_lock();
+ jump_label_lock();
+
switch (val) {
case MODULE_STATE_COMING:
- jump_label_lock();
ret = jump_label_add_module(mod);
if (ret) {
WARN(1, "Failed to allocatote memory: jump_label may not work properly.\n");
jump_label_del_module(mod);
}
- jump_label_unlock();
break;
case MODULE_STATE_GOING:
- jump_label_lock();
jump_label_del_module(mod);
- jump_label_unlock();
break;
case MODULE_STATE_LIVE:
- jump_label_lock();
jump_label_invalidate_module_init(mod);
- jump_label_unlock();
break;
}
+ jump_label_unlock();
+ cpus_read_unlock();
+
return notifier_from_errno(ret);
}
#include <linux/syscore_ops.h>
#include <linux/compiler.h>
#include <linux/hugetlb.h>
+#include <linux/frame.h>
#include <asm/page.h>
#include <asm/sections.h>
* only when panic_cpu holds the current CPU number; this is the only CPU
* which processes crash_kexec routines.
*/
-void __crash_kexec(struct pt_regs *regs)
+void __noclone __crash_kexec(struct pt_regs *regs)
{
/* Take the kexec_mutex here to prevent sys_kexec_load
* running on one cpu from replacing the crash kernel
mutex_unlock(&kexec_mutex);
}
}
+STACK_FRAME_NON_STANDARD(__crash_kexec);
void crash_kexec(struct pt_regs *regs)
{
*/
static void do_optimize_kprobes(void)
{
- /* Optimization never be done when disarmed */
- if (kprobes_all_disarmed || !kprobes_allow_optimization ||
- list_empty(&optimizing_list))
- return;
-
/*
* The optimization/unoptimization refers online_cpus via
* stop_machine() and cpu-hotplug modifies online_cpus.
* This combination can cause a deadlock (cpu-hotplug try to lock
* text_mutex but stop_machine can not be done because online_cpus
* has been changed)
- * To avoid this deadlock, we need to call get_online_cpus()
+ * To avoid this deadlock, caller must have locked cpu hotplug
* for preventing cpu-hotplug outside of text_mutex locking.
*/
- get_online_cpus();
+ lockdep_assert_cpus_held();
+
+ /* Optimization never be done when disarmed */
+ if (kprobes_all_disarmed || !kprobes_allow_optimization ||
+ list_empty(&optimizing_list))
+ return;
+
mutex_lock(&text_mutex);
arch_optimize_kprobes(&optimizing_list);
mutex_unlock(&text_mutex);
- put_online_cpus();
}
/*
{
struct optimized_kprobe *op, *tmp;
+ /* See comment in do_optimize_kprobes() */
+ lockdep_assert_cpus_held();
+
/* Unoptimization must be done anytime */
if (list_empty(&unoptimizing_list))
return;
- /* Ditto to do_optimize_kprobes */
- get_online_cpus();
mutex_lock(&text_mutex);
arch_unoptimize_kprobes(&unoptimizing_list, &freeing_list);
/* Loop free_list for disarming */
list_del_init(&op->list);
}
mutex_unlock(&text_mutex);
- put_online_cpus();
}
/* Reclaim all kprobes on the free_list */
static void kprobe_optimizer(struct work_struct *work)
{
mutex_lock(&kprobe_mutex);
+ cpus_read_lock();
/* Lock modules while optimizing kprobes */
mutex_lock(&module_mutex);
do_free_cleaned_kprobes();
mutex_unlock(&module_mutex);
+ cpus_read_unlock();
mutex_unlock(&kprobe_mutex);
/* Step 5: Kick optimizer again if needed */
/* Short cut to direct unoptimizing */
static void force_unoptimize_kprobe(struct optimized_kprobe *op)
{
- get_online_cpus();
+ lockdep_assert_cpus_held();
arch_unoptimize_kprobe(op);
- put_online_cpus();
if (kprobe_disabled(&op->kp))
arch_disarm_kprobe(&op->kp);
}
return;
/* For preparing optimization, jump_label_text_reserved() is called */
+ cpus_read_lock();
jump_label_lock();
mutex_lock(&text_mutex);
out:
mutex_unlock(&text_mutex);
jump_label_unlock();
+ cpus_read_unlock();
}
#ifdef CONFIG_SYSCTL
if (kprobes_allow_optimization)
goto out;
+ cpus_read_lock();
kprobes_allow_optimization = true;
for (i = 0; i < KPROBE_TABLE_SIZE; i++) {
head = &kprobe_table[i];
if (!kprobe_disabled(p))
optimize_kprobe(p);
}
+ cpus_read_unlock();
printk(KERN_INFO "Kprobes globally optimized\n");
out:
mutex_unlock(&kprobe_mutex);
return;
}
+ cpus_read_lock();
kprobes_allow_optimization = false;
for (i = 0; i < KPROBE_TABLE_SIZE; i++) {
head = &kprobe_table[i];
unoptimize_kprobe(p, false);
}
}
+ cpus_read_unlock();
mutex_unlock(&kprobe_mutex);
/* Wait for unoptimizing completion */
arm_kprobe_ftrace(kp);
return;
}
- /*
- * Here, since __arm_kprobe() doesn't use stop_machine(),
- * this doesn't cause deadlock on text_mutex. So, we don't
- * need get_online_cpus().
- */
+ cpus_read_lock();
mutex_lock(&text_mutex);
__arm_kprobe(kp);
mutex_unlock(&text_mutex);
+ cpus_read_unlock();
}
/* Disarm a kprobe with text_mutex */
disarm_kprobe_ftrace(kp);
return;
}
- /* Ditto */
+
+ cpus_read_lock();
mutex_lock(&text_mutex);
__disarm_kprobe(kp, reopt);
mutex_unlock(&text_mutex);
+ cpus_read_unlock();
}
/*
int ret = 0;
struct kprobe *ap = orig_p;
+ cpus_read_lock();
+
/* For preparing optimization, jump_label_text_reserved() is called */
jump_label_lock();
- /*
- * Get online CPUs to avoid text_mutex deadlock.with stop machine,
- * which is invoked by unoptimize_kprobe() in add_new_kprobe()
- */
- get_online_cpus();
mutex_lock(&text_mutex);
if (!kprobe_aggrprobe(orig_p)) {
out:
mutex_unlock(&text_mutex);
- put_online_cpus();
jump_label_unlock();
+ cpus_read_unlock();
if (ret == 0 && kprobe_disabled(ap) && !kprobe_disabled(p)) {
ap->flags &= ~KPROBE_FLAG_DISABLED;
goto out;
}
- mutex_lock(&text_mutex); /* Avoiding text modification */
+ cpus_read_lock();
+ /* Prevent text modification */
+ mutex_lock(&text_mutex);
ret = prepare_kprobe(p);
mutex_unlock(&text_mutex);
+ cpus_read_unlock();
if (ret)
goto out;
/* Try to optimize kprobe */
try_to_optimize_kprobe(p);
-
out:
mutex_unlock(&kprobe_mutex);
ops = container_of(fops, struct klp_ops, fops);
- rcu_read_lock();
+ /*
+ * A variant of synchronize_sched() is used to allow patching functions
+ * where RCU is not watching, see klp_synchronize_transition().
+ */
+ preempt_disable_notrace();
func = list_first_or_null_rcu(&ops->func_stack, struct klp_func,
stack_node);
klp_arch_set_pc(regs, (unsigned long)func->new_func);
unlock:
- rcu_read_unlock();
+ preempt_enable_notrace();
}
/*
}
static DECLARE_DELAYED_WORK(klp_transition_work, klp_transition_work_fn);
+/*
+ * This function is just a stub to implement a hard force
+ * of synchronize_sched(). This requires synchronizing
+ * tasks even in userspace and idle.
+ */
+static void klp_sync(struct work_struct *work)
+{
+}
+
+/*
+ * We allow to patch also functions where RCU is not watching,
+ * e.g. before user_exit(). We can not rely on the RCU infrastructure
+ * to do the synchronization. Instead hard force the sched synchronization.
+ *
+ * This approach allows to use RCU functions for manipulating func_stack
+ * safely.
+ */
+static void klp_synchronize_transition(void)
+{
+ schedule_on_each_cpu(klp_sync);
+}
+
/*
* The transition to the target patch state is complete. Clean up the data
* structures.
* func->transition gets cleared, the handler may choose a
* removed function.
*/
- synchronize_rcu();
+ klp_synchronize_transition();
}
if (klp_transition_patch->immediate)
/* Prevent klp_ftrace_handler() from seeing KLP_UNDEFINED state */
if (klp_target_state == KLP_PATCHED)
- synchronize_rcu();
+ klp_synchronize_transition();
read_lock(&tasklist_lock);
for_each_process_thread(g, task) {
*/
void klp_update_patch_state(struct task_struct *task)
{
- rcu_read_lock();
+ /*
+ * A variant of synchronize_sched() is used to allow patching functions
+ * where RCU is not watching, see klp_synchronize_transition().
+ */
+ preempt_disable_notrace();
/*
* This test_and_clear_tsk_thread_flag() call also serves as a read
if (test_and_clear_tsk_thread_flag(task, TIF_PATCH_PENDING))
task->patch_state = READ_ONCE(klp_target_state);
- rcu_read_unlock();
+ preempt_enable_notrace();
}
/*
clear_tsk_thread_flag(idle_task(cpu), TIF_PATCH_PENDING);
/* Let any remaining calls to klp_update_patch_state() complete */
- synchronize_rcu();
+ klp_synchronize_transition();
klp_start_transition();
}
if (debug_locks_silent)
return 0;
- printk("\n");
+ pr_warn("\n");
pr_warn("======================================================\n");
pr_warn("WARNING: possible circular locking dependency detected\n");
print_kernel_ident();
pr_warn("------------------------------------------------------\n");
- printk("%s/%d is trying to acquire lock:\n",
+ pr_warn("%s/%d is trying to acquire lock:\n",
curr->comm, task_pid_nr(curr));
print_lock(check_src);
- printk("\nbut task is already holding lock:\n");
+ pr_warn("\nbut task is already holding lock:\n");
print_lock(check_tgt);
- printk("\nwhich lock already depends on the new lock.\n\n");
- printk("\nthe existing dependency chain (in reverse order) is:\n");
+ pr_warn("\nwhich lock already depends on the new lock.\n\n");
+ pr_warn("\nthe existing dependency chain (in reverse order) is:\n");
print_circular_bug_entry(entry, depth);
if (!debug_locks_off_graph_unlock() || debug_locks_silent)
return 0;
- printk("\n");
+ pr_warn("\n");
pr_warn("=====================================================\n");
pr_warn("WARNING: %s-safe -> %s-unsafe lock order detected\n",
irqclass, irqclass);
print_kernel_ident();
pr_warn("-----------------------------------------------------\n");
- printk("%s/%d [HC%u[%lu]:SC%u[%lu]:HE%u:SE%u] is trying to acquire:\n",
+ pr_warn("%s/%d [HC%u[%lu]:SC%u[%lu]:HE%u:SE%u] is trying to acquire:\n",
curr->comm, task_pid_nr(curr),
curr->hardirq_context, hardirq_count() >> HARDIRQ_SHIFT,
curr->softirq_context, softirq_count() >> SOFTIRQ_SHIFT,
curr->softirqs_enabled);
print_lock(next);
- printk("\nand this task is already holding:\n");
+ pr_warn("\nand this task is already holding:\n");
print_lock(prev);
- printk("which would create a new lock dependency:\n");
+ pr_warn("which would create a new lock dependency:\n");
print_lock_name(hlock_class(prev));
- printk(KERN_CONT " ->");
+ pr_cont(" ->");
print_lock_name(hlock_class(next));
- printk(KERN_CONT "\n");
+ pr_cont("\n");
- printk("\nbut this new dependency connects a %s-irq-safe lock:\n",
+ pr_warn("\nbut this new dependency connects a %s-irq-safe lock:\n",
irqclass);
print_lock_name(backwards_entry->class);
- printk("\n... which became %s-irq-safe at:\n", irqclass);
+ pr_warn("\n... which became %s-irq-safe at:\n", irqclass);
print_stack_trace(backwards_entry->class->usage_traces + bit1, 1);
- printk("\nto a %s-irq-unsafe lock:\n", irqclass);
+ pr_warn("\nto a %s-irq-unsafe lock:\n", irqclass);
print_lock_name(forwards_entry->class);
- printk("\n... which became %s-irq-unsafe at:\n", irqclass);
- printk("...");
+ pr_warn("\n... which became %s-irq-unsafe at:\n", irqclass);
+ pr_warn("...");
print_stack_trace(forwards_entry->class->usage_traces + bit2, 1);
- printk("\nother info that might help us debug this:\n\n");
+ pr_warn("\nother info that might help us debug this:\n\n");
print_irq_lock_scenario(backwards_entry, forwards_entry,
hlock_class(prev), hlock_class(next));
lockdep_print_held_locks(curr);
- printk("\nthe dependencies between %s-irq-safe lock and the holding lock:\n", irqclass);
+ pr_warn("\nthe dependencies between %s-irq-safe lock and the holding lock:\n", irqclass);
if (!save_trace(&prev_root->trace))
return 0;
print_shortest_lock_dependencies(backwards_entry, prev_root);
- printk("\nthe dependencies between the lock to be acquired");
- printk(" and %s-irq-unsafe lock:\n", irqclass);
+ pr_warn("\nthe dependencies between the lock to be acquired");
+ pr_warn(" and %s-irq-unsafe lock:\n", irqclass);
if (!save_trace(&next_root->trace))
return 0;
print_shortest_lock_dependencies(forwards_entry, next_root);
- printk("\nstack backtrace:\n");
+ pr_warn("\nstack backtrace:\n");
dump_stack();
return 0;
if (!debug_locks_off_graph_unlock() || debug_locks_silent)
return 0;
- printk("\n");
+ pr_warn("\n");
pr_warn("============================================\n");
pr_warn("WARNING: possible recursive locking detected\n");
print_kernel_ident();
pr_warn("--------------------------------------------\n");
- printk("%s/%d is trying to acquire lock:\n",
+ pr_warn("%s/%d is trying to acquire lock:\n",
curr->comm, task_pid_nr(curr));
print_lock(next);
- printk("\nbut task is already holding lock:\n");
+ pr_warn("\nbut task is already holding lock:\n");
print_lock(prev);
- printk("\nother info that might help us debug this:\n");
+ pr_warn("\nother info that might help us debug this:\n");
print_deadlock_scenario(next, prev);
lockdep_print_held_locks(curr);
- printk("\nstack backtrace:\n");
+ pr_warn("\nstack backtrace:\n");
dump_stack();
return 0;
struct held_lock *hlock_next,
struct lock_chain *chain)
{
- printk("\n");
+ pr_warn("\n");
pr_warn("============================\n");
pr_warn("WARNING: chain_key collision\n");
print_kernel_ident();
pr_warn("----------------------------\n");
- printk("%s/%d: ", current->comm, task_pid_nr(current));
- printk("Hash chain already cached but the contents don't match!\n");
+ pr_warn("%s/%d: ", current->comm, task_pid_nr(current));
+ pr_warn("Hash chain already cached but the contents don't match!\n");
- printk("Held locks:");
+ pr_warn("Held locks:");
print_chain_keys_held_locks(curr, hlock_next);
- printk("Locks in cached chain:");
+ pr_warn("Locks in cached chain:");
print_chain_keys_chain(chain);
- printk("\nstack backtrace:\n");
+ pr_warn("\nstack backtrace:\n");
dump_stack();
}
#endif
if (!debug_locks_off_graph_unlock() || debug_locks_silent)
return 0;
- printk("\n");
+ pr_warn("\n");
pr_warn("================================\n");
pr_warn("WARNING: inconsistent lock state\n");
print_kernel_ident();
pr_warn("--------------------------------\n");
- printk("inconsistent {%s} -> {%s} usage.\n",
+ pr_warn("inconsistent {%s} -> {%s} usage.\n",
usage_str[prev_bit], usage_str[new_bit]);
- printk("%s/%d [HC%u[%lu]:SC%u[%lu]:HE%u:SE%u] takes:\n",
+ pr_warn("%s/%d [HC%u[%lu]:SC%u[%lu]:HE%u:SE%u] takes:\n",
curr->comm, task_pid_nr(curr),
trace_hardirq_context(curr), hardirq_count() >> HARDIRQ_SHIFT,
trace_softirq_context(curr), softirq_count() >> SOFTIRQ_SHIFT,
trace_softirqs_enabled(curr));
print_lock(this);
- printk("{%s} state was registered at:\n", usage_str[prev_bit]);
+ pr_warn("{%s} state was registered at:\n", usage_str[prev_bit]);
print_stack_trace(hlock_class(this)->usage_traces + prev_bit, 1);
print_irqtrace_events(curr);
- printk("\nother info that might help us debug this:\n");
+ pr_warn("\nother info that might help us debug this:\n");
print_usage_bug_scenario(this);
lockdep_print_held_locks(curr);
- printk("\nstack backtrace:\n");
+ pr_warn("\nstack backtrace:\n");
dump_stack();
return 0;
if (!debug_locks_off_graph_unlock() || debug_locks_silent)
return 0;
- printk("\n");
+ pr_warn("\n");
pr_warn("========================================================\n");
pr_warn("WARNING: possible irq lock inversion dependency detected\n");
print_kernel_ident();
pr_warn("--------------------------------------------------------\n");
- printk("%s/%d just changed the state of lock:\n",
+ pr_warn("%s/%d just changed the state of lock:\n",
curr->comm, task_pid_nr(curr));
print_lock(this);
if (forwards)
- printk("but this lock took another, %s-unsafe lock in the past:\n", irqclass);
+ pr_warn("but this lock took another, %s-unsafe lock in the past:\n", irqclass);
else
- printk("but this lock was taken by another, %s-safe lock in the past:\n", irqclass);
+ pr_warn("but this lock was taken by another, %s-safe lock in the past:\n", irqclass);
print_lock_name(other->class);
- printk("\n\nand interrupts could create inverse lock ordering between them.\n\n");
+ pr_warn("\n\nand interrupts could create inverse lock ordering between them.\n\n");
- printk("\nother info that might help us debug this:\n");
+ pr_warn("\nother info that might help us debug this:\n");
/* Find a middle lock (if one exists) */
depth = get_lock_depth(other);
do {
if (depth == 0 && (entry != root)) {
- printk("lockdep:%s bad path found in chain graph\n", __func__);
+ pr_warn("lockdep:%s bad path found in chain graph\n", __func__);
break;
}
middle = entry;
lockdep_print_held_locks(curr);
- printk("\nthe shortest dependencies between 2nd lock and 1st lock:\n");
+ pr_warn("\nthe shortest dependencies between 2nd lock and 1st lock:\n");
if (!save_trace(&root->trace))
return 0;
print_shortest_lock_dependencies(other, root);
- printk("\nstack backtrace:\n");
+ pr_warn("\nstack backtrace:\n");
dump_stack();
return 0;
if (debug_locks_silent)
return 0;
- printk("\n");
+ pr_warn("\n");
pr_warn("==================================\n");
pr_warn("WARNING: Nested lock was not taken\n");
print_kernel_ident();
pr_warn("----------------------------------\n");
- printk("%s/%d is trying to lock:\n", curr->comm, task_pid_nr(curr));
+ pr_warn("%s/%d is trying to lock:\n", curr->comm, task_pid_nr(curr));
print_lock(hlock);
- printk("\nbut this task is not holding:\n");
- printk("%s\n", hlock->nest_lock->name);
+ pr_warn("\nbut this task is not holding:\n");
+ pr_warn("%s\n", hlock->nest_lock->name);
- printk("\nstack backtrace:\n");
+ pr_warn("\nstack backtrace:\n");
dump_stack();
- printk("\nother info that might help us debug this:\n");
+ pr_warn("\nother info that might help us debug this:\n");
lockdep_print_held_locks(curr);
- printk("\nstack backtrace:\n");
+ pr_warn("\nstack backtrace:\n");
dump_stack();
return 0;
if (debug_locks_silent)
return 0;
- printk("\n");
+ pr_warn("\n");
pr_warn("=====================================\n");
pr_warn("WARNING: bad unlock balance detected!\n");
print_kernel_ident();
pr_warn("-------------------------------------\n");
- printk("%s/%d is trying to release lock (",
+ pr_warn("%s/%d is trying to release lock (",
curr->comm, task_pid_nr(curr));
print_lockdep_cache(lock);
- printk(KERN_CONT ") at:\n");
+ pr_cont(") at:\n");
print_ip_sym(ip);
- printk("but there are no more locks to release!\n");
- printk("\nother info that might help us debug this:\n");
+ pr_warn("but there are no more locks to release!\n");
+ pr_warn("\nother info that might help us debug this:\n");
lockdep_print_held_locks(curr);
- printk("\nstack backtrace:\n");
+ pr_warn("\nstack backtrace:\n");
dump_stack();
return 0;
if (debug_locks_silent)
return 0;
- printk("\n");
+ pr_warn("\n");
pr_warn("=================================\n");
pr_warn("WARNING: bad contention detected!\n");
print_kernel_ident();
pr_warn("---------------------------------\n");
- printk("%s/%d is trying to contend lock (",
+ pr_warn("%s/%d is trying to contend lock (",
curr->comm, task_pid_nr(curr));
print_lockdep_cache(lock);
- printk(KERN_CONT ") at:\n");
+ pr_cont(") at:\n");
print_ip_sym(ip);
- printk("but there are no locks held!\n");
- printk("\nother info that might help us debug this:\n");
+ pr_warn("but there are no locks held!\n");
+ pr_warn("\nother info that might help us debug this:\n");
lockdep_print_held_locks(curr);
- printk("\nstack backtrace:\n");
+ pr_warn("\nstack backtrace:\n");
dump_stack();
return 0;
if (debug_locks_silent)
return;
- printk("\n");
+ pr_warn("\n");
pr_warn("=========================\n");
pr_warn("WARNING: held lock freed!\n");
print_kernel_ident();
pr_warn("-------------------------\n");
- printk("%s/%d is freeing memory %p-%p, with a lock still held there!\n",
+ pr_warn("%s/%d is freeing memory %p-%p, with a lock still held there!\n",
curr->comm, task_pid_nr(curr), mem_from, mem_to-1);
print_lock(hlock);
lockdep_print_held_locks(curr);
- printk("\nstack backtrace:\n");
+ pr_warn("\nstack backtrace:\n");
dump_stack();
}
if (debug_locks_silent)
return;
- printk("\n");
+ pr_warn("\n");
pr_warn("====================================\n");
pr_warn("WARNING: %s/%d still has locks held!\n",
current->comm, task_pid_nr(current));
print_kernel_ident();
pr_warn("------------------------------------\n");
lockdep_print_held_locks(current);
- printk("\nstack backtrace:\n");
+ pr_warn("\nstack backtrace:\n");
dump_stack();
}
int unlock = 1;
if (unlikely(!debug_locks)) {
- printk("INFO: lockdep is turned off.\n");
+ pr_warn("INFO: lockdep is turned off.\n");
return;
}
- printk("\nShowing all locks held in the system:\n");
+ pr_warn("\nShowing all locks held in the system:\n");
/*
* Here we try to get the tasklist_lock as hard as possible,
retry:
if (!read_trylock(&tasklist_lock)) {
if (count == 10)
- printk("hm, tasklist_lock locked, retrying... ");
+ pr_warn("hm, tasklist_lock locked, retrying... ");
if (count) {
count--;
- printk(" #%d", 10-count);
+ pr_cont(" #%d", 10-count);
mdelay(200);
goto retry;
}
- printk(" ignoring it.\n");
+ pr_cont(" ignoring it.\n");
unlock = 0;
} else {
if (count != 10)
- printk(KERN_CONT " locked it.\n");
+ pr_cont(" locked it.\n");
}
do_each_thread(g, p) {
unlock = 1;
} while_each_thread(g, p);
- printk("\n");
+ pr_warn("\n");
pr_warn("=============================================\n\n");
if (unlock)
if (unlikely(curr->lockdep_depth)) {
if (!debug_locks_off())
return;
- printk("\n");
+ pr_warn("\n");
pr_warn("================================================\n");
pr_warn("WARNING: lock held when returning to user space!\n");
print_kernel_ident();
pr_warn("------------------------------------------------\n");
- printk("%s/%d is leaving the kernel with locks still held!\n",
+ pr_warn("%s/%d is leaving the kernel with locks still held!\n",
curr->comm, curr->pid);
lockdep_print_held_locks(curr);
}
{
struct task_struct *curr = current;
-#ifndef CONFIG_PROVE_RCU_REPEATEDLY
- if (!debug_locks_off())
- return;
-#endif /* #ifdef CONFIG_PROVE_RCU_REPEATEDLY */
/* Note: the following can be executed concurrently, so be careful. */
- printk("\n");
+ pr_warn("\n");
pr_warn("=============================\n");
pr_warn("WARNING: suspicious RCU usage\n");
print_kernel_ident();
pr_warn("-----------------------------\n");
- printk("%s:%d %s!\n", file, line, s);
- printk("\nother info that might help us debug this:\n\n");
- printk("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
+ pr_warn("%s:%d %s!\n", file, line, s);
+ pr_warn("\nother info that might help us debug this:\n\n");
+ pr_warn("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
!rcu_lockdep_current_cpu_online()
? "RCU used illegally from offline CPU!\n"
: !rcu_is_watching()
* rcu_read_lock_bh() and so on from extended quiescent states.
*/
if (!rcu_is_watching())
- printk("RCU used illegally from extended quiescent state!\n");
+ pr_warn("RCU used illegally from extended quiescent state!\n");
lockdep_print_held_locks(curr);
- printk("\nstack backtrace:\n");
+ pr_warn("\nstack backtrace:\n");
dump_stack();
}
EXPORT_SYMBOL_GPL(lockdep_rcu_suspicious);
* (or statically defined) before it can be locked. memset()-ing
* the mutex to 0 is not allowed.
*
- * ( The CONFIG_DEBUG_MUTEXES .config option turns on debugging
- * checks that will enforce the restrictions and will also do
- * deadlock debugging. )
+ * (The CONFIG_DEBUG_MUTEXES .config option turns on debugging
+ * checks that will enforce the restrictions and will also do
+ * deadlock debugging)
*
* This function is similar to (but not equivalent to) down().
*/
memset(waiter, 0x22, sizeof(*waiter));
}
-void debug_rt_mutex_init(struct rt_mutex *lock, const char *name)
+void debug_rt_mutex_init(struct rt_mutex *lock, const char *name, struct lock_class_key *key)
{
/*
* Make sure we are not reinitializing a held lock:
*/
debug_check_no_locks_freed((void *)lock, sizeof(*lock));
lock->name = name;
+
+#ifdef CONFIG_DEBUG_LOCK_ALLOC
+ lockdep_init_map(&lock->dep_map, name, key, 0);
+#endif
}
extern void debug_rt_mutex_init_waiter(struct rt_mutex_waiter *waiter);
extern void debug_rt_mutex_free_waiter(struct rt_mutex_waiter *waiter);
-extern void debug_rt_mutex_init(struct rt_mutex *lock, const char *name);
+extern void debug_rt_mutex_init(struct rt_mutex *lock, const char *name, struct lock_class_key *key);
extern void debug_rt_mutex_lock(struct rt_mutex *lock);
extern void debug_rt_mutex_unlock(struct rt_mutex *lock);
extern void debug_rt_mutex_proxy_lock(struct rt_mutex *lock,
{
might_sleep();
+ mutex_acquire(&lock->dep_map, 0, 0, _RET_IP_);
rt_mutex_fastlock(lock, TASK_UNINTERRUPTIBLE, rt_mutex_slowlock);
}
EXPORT_SYMBOL_GPL(rt_mutex_lock);
*/
int __sched rt_mutex_lock_interruptible(struct rt_mutex *lock)
{
+ int ret;
+
might_sleep();
- return rt_mutex_fastlock(lock, TASK_INTERRUPTIBLE, rt_mutex_slowlock);
+ mutex_acquire(&lock->dep_map, 0, 0, _RET_IP_);
+ ret = rt_mutex_fastlock(lock, TASK_INTERRUPTIBLE, rt_mutex_slowlock);
+ if (ret)
+ mutex_release(&lock->dep_map, 1, _RET_IP_);
+
+ return ret;
}
EXPORT_SYMBOL_GPL(rt_mutex_lock_interruptible);
int
rt_mutex_timed_lock(struct rt_mutex *lock, struct hrtimer_sleeper *timeout)
{
+ int ret;
+
might_sleep();
- return rt_mutex_timed_fastlock(lock, TASK_INTERRUPTIBLE, timeout,
+ mutex_acquire(&lock->dep_map, 0, 0, _RET_IP_);
+ ret = rt_mutex_timed_fastlock(lock, TASK_INTERRUPTIBLE, timeout,
RT_MUTEX_MIN_CHAINWALK,
rt_mutex_slowlock);
+ if (ret)
+ mutex_release(&lock->dep_map, 1, _RET_IP_);
+
+ return ret;
}
EXPORT_SYMBOL_GPL(rt_mutex_timed_lock);
*/
int __sched rt_mutex_trylock(struct rt_mutex *lock)
{
+ int ret;
+
if (WARN_ON_ONCE(in_irq() || in_nmi() || in_serving_softirq()))
return 0;
- return rt_mutex_fasttrylock(lock, rt_mutex_slowtrylock);
+ ret = rt_mutex_fasttrylock(lock, rt_mutex_slowtrylock);
+ if (ret)
+ mutex_acquire(&lock->dep_map, 0, 1, _RET_IP_);
+
+ return ret;
}
EXPORT_SYMBOL_GPL(rt_mutex_trylock);
*/
void __sched rt_mutex_unlock(struct rt_mutex *lock)
{
+ mutex_release(&lock->dep_map, 1, _RET_IP_);
rt_mutex_fastunlock(lock, rt_mutex_slowunlock);
}
EXPORT_SYMBOL_GPL(rt_mutex_unlock);
lock->magic = NULL;
#endif
}
-
EXPORT_SYMBOL_GPL(rt_mutex_destroy);
/**
*
* Initializing of a locked rt lock is not allowed
*/
-void __rt_mutex_init(struct rt_mutex *lock, const char *name)
+void __rt_mutex_init(struct rt_mutex *lock, const char *name,
+ struct lock_class_key *key)
{
lock->owner = NULL;
raw_spin_lock_init(&lock->wait_lock);
lock->waiters = RB_ROOT;
lock->waiters_leftmost = NULL;
- debug_rt_mutex_init(lock, name);
+ if (name && key)
+ debug_rt_mutex_init(lock, name, key);
}
EXPORT_SYMBOL_GPL(__rt_mutex_init);
void rt_mutex_init_proxy_locked(struct rt_mutex *lock,
struct task_struct *proxy_owner)
{
- __rt_mutex_init(lock, NULL);
+ __rt_mutex_init(lock, NULL, NULL);
debug_rt_mutex_proxy_lock(lock, proxy_owner);
rt_mutex_set_owner(lock, proxy_owner);
}
#define debug_rt_mutex_proxy_lock(l,p) do { } while (0)
#define debug_rt_mutex_proxy_unlock(l) do { } while (0)
#define debug_rt_mutex_unlock(l) do { } while (0)
-#define debug_rt_mutex_init(m, n) do { } while (0)
+#define debug_rt_mutex_init(m, n, k) do { } while (0)
#define debug_rt_mutex_deadlock(d, a ,l) do { } while (0)
#define debug_rt_mutex_print_deadlock(w) do { } while (0)
#define debug_rt_mutex_reset_waiter(w) do { } while (0)
struct module_kobject *mk,
const char *buffer, size_t count)
{
- enum kobject_action action;
-
- if (kobject_action_type(buffer, count, &action) == 0)
- kobject_uevent(&mk->kobj, action);
+ kobject_synth_uevent(&mk->kobj, buffer, count);
return count;
}
.release = padata_sysfs_release,
};
-/**
- * padata_alloc_possible - Allocate and initialize padata instance.
- * Use the cpu_possible_mask for serial and
- * parallel workers.
- *
- * @wq: workqueue to use for the allocated padata instance
- */
-struct padata_instance *padata_alloc_possible(struct workqueue_struct *wq)
-{
- return padata_alloc(wq, cpu_possible_mask, cpu_possible_mask);
-}
-EXPORT_SYMBOL(padata_alloc_possible);
-
/**
* padata_alloc - allocate and initialize a padata instance and specify
* cpumasks for serial and parallel workers.
* @wq: workqueue to use for the allocated padata instance
* @pcpumask: cpumask that will be used for padata parallelization
* @cbcpumask: cpumask that will be used for padata serialization
+ *
+ * Must be called from a cpus_read_lock() protected region
*/
-struct padata_instance *padata_alloc(struct workqueue_struct *wq,
- const struct cpumask *pcpumask,
- const struct cpumask *cbcpumask)
+static struct padata_instance *padata_alloc(struct workqueue_struct *wq,
+ const struct cpumask *pcpumask,
+ const struct cpumask *cbcpumask)
{
struct padata_instance *pinst;
struct parallel_data *pd = NULL;
if (!pinst)
goto err;
- get_online_cpus();
if (!alloc_cpumask_var(&pinst->cpumask.pcpu, GFP_KERNEL))
goto err_free_inst;
if (!alloc_cpumask_var(&pinst->cpumask.cbcpu, GFP_KERNEL)) {
pinst->flags = 0;
- put_online_cpus();
-
BLOCKING_INIT_NOTIFIER_HEAD(&pinst->cpumask_change_notifier);
kobject_init(&pinst->kobj, &padata_attr_type);
mutex_init(&pinst->lock);
#ifdef CONFIG_HOTPLUG_CPU
- cpuhp_state_add_instance_nocalls(hp_online, &pinst->node);
+ cpuhp_state_add_instance_nocalls_cpuslocked(hp_online, &pinst->node);
#endif
return pinst;
free_cpumask_var(pinst->cpumask.cbcpu);
err_free_inst:
kfree(pinst);
- put_online_cpus();
err:
return NULL;
}
+/**
+ * padata_alloc_possible - Allocate and initialize padata instance.
+ * Use the cpu_possible_mask for serial and
+ * parallel workers.
+ *
+ * @wq: workqueue to use for the allocated padata instance
+ *
+ * Must be called from a cpus_read_lock() protected region
+ */
+struct padata_instance *padata_alloc_possible(struct workqueue_struct *wq)
+{
+ lockdep_assert_cpus_held();
+ return padata_alloc(wq, cpu_possible_mask, cpu_possible_mask);
+}
+EXPORT_SYMBOL(padata_alloc_possible);
+
/**
* padata_free - free a padata instance
*
};
-static struct attribute_group attr_group = {
+static const struct attribute_group attr_group = {
.attrs = g,
};
if (!pm_freezing)
atomic_inc(&system_freezing_cnt);
- pm_wakeup_clear();
+ pm_wakeup_clear(true);
pr_info("Freezing user space processes ... ");
pm_freezing = true;
error = try_to_freeze_tasks(true);
#include <asm/pgtable.h>
#include <asm/tlbflush.h>
#include <asm/io.h>
-#ifdef CONFIG_STRICT_KERNEL_RWX
+#ifdef CONFIG_ARCH_HAS_SET_MEMORY
#include <asm/set_memory.h>
#endif
#include "power.h"
-#ifdef CONFIG_STRICT_KERNEL_RWX
+#if defined(CONFIG_STRICT_KERNEL_RWX) && defined(CONFIG_ARCH_HAS_SET_MEMORY)
static bool hibernate_restore_protection;
static bool hibernate_restore_protection_active;
static inline void hibernate_restore_protection_end(void) {}
static inline void hibernate_restore_protect_page(void *page_address) {}
static inline void hibernate_restore_unprotect_page(void *page_address) {}
-#endif /* CONFIG_STRICT_KERNEL_RWX */
+#endif /* CONFIG_STRICT_KERNEL_RWX && CONFIG_ARCH_HAS_SET_MEMORY */
static int swsusp_page_is_free(struct page *);
static void swsusp_set_page_forbidden(struct page *);
* also be located in the high memory, because of the way in which
* copy_data_pages() works.
*/
-static int swsusp_alloc(struct memory_bitmap *orig_bm,
- struct memory_bitmap *copy_bm,
+static int swsusp_alloc(struct memory_bitmap *copy_bm,
unsigned int nr_pages, unsigned int nr_highmem)
{
if (nr_highmem > 0) {
return -ENOMEM;
}
- if (swsusp_alloc(&orig_bm, ©_bm, nr_pages, nr_highmem)) {
+ if (swsusp_alloc(©_bm, nr_pages, nr_highmem)) {
printk(KERN_ERR "PM: Memory allocation failed\n");
return -ENOMEM;
}
static void freeze_enter(void)
{
+ trace_suspend_resume(TPS("machine_suspend"), PM_SUSPEND_FREEZE, true);
+
spin_lock_irq(&suspend_freeze_lock);
if (pm_wakeup_pending())
goto out;
/* Push all the CPUs into the idle loop. */
wake_up_all_idle_cpus();
- pr_debug("PM: suspend-to-idle\n");
/* Make the current CPU wait so it can enter the idle loop too. */
wait_event(suspend_freeze_wait_head,
suspend_freeze_state == FREEZE_STATE_WAKE);
- pr_debug("PM: resume from suspend-to-idle\n");
cpuidle_pause();
put_online_cpus();
out:
suspend_freeze_state = FREEZE_STATE_NONE;
spin_unlock_irq(&suspend_freeze_lock);
+
+ trace_suspend_resume(TPS("machine_suspend"), PM_SUSPEND_FREEZE, false);
+}
+
+static void s2idle_loop(void)
+{
+ pr_debug("PM: suspend-to-idle\n");
+
+ do {
+ freeze_enter();
+
+ if (freeze_ops && freeze_ops->wake)
+ freeze_ops->wake();
+
+ dpm_resume_noirq(PMSG_RESUME);
+ if (freeze_ops && freeze_ops->sync)
+ freeze_ops->sync();
+
+ if (pm_wakeup_pending())
+ break;
+
+ pm_wakeup_clear(false);
+ } while (!dpm_suspend_noirq(PMSG_SUSPEND));
+
+ pr_debug("PM: resume from suspend-to-idle\n");
}
void freeze_wake(void)
* all the devices are suspended.
*/
if (state == PM_SUSPEND_FREEZE) {
- trace_suspend_resume(TPS("machine_suspend"), state, true);
- freeze_enter();
- trace_suspend_resume(TPS("machine_suspend"), state, false);
- goto Platform_wake;
+ s2idle_loop();
+ goto Platform_early_resume;
}
error = disable_nonboot_cpus();
struct hib_bio_batch {
atomic_t count;
wait_queue_head_t wait;
- int error;
+ blk_status_t error;
};
static void hib_init_batch(struct hib_bio_batch *hb)
{
atomic_set(&hb->count, 0);
init_waitqueue_head(&hb->wait);
- hb->error = 0;
+ hb->error = BLK_STS_OK;
}
static void hib_end_io(struct bio *bio)
struct hib_bio_batch *hb = bio->bi_private;
struct page *page = bio->bi_io_vec[0].bv_page;
- if (bio->bi_error) {
+ if (bio->bi_status) {
printk(KERN_ALERT "Read-error on swap-device (%u:%u:%Lu)\n",
imajor(bio->bi_bdev->bd_inode),
iminor(bio->bi_bdev->bd_inode),
flush_icache_range((unsigned long)page_address(page),
(unsigned long)page_address(page) + PAGE_SIZE);
- if (bio->bi_error && !hb->error)
- hb->error = bio->bi_error;
+ if (bio->bi_status && !hb->error)
+ hb->error = bio->bi_status;
if (atomic_dec_and_test(&hb->count))
wake_up(&hb->wait);
return error;
}
-static int hib_wait_io(struct hib_bio_batch *hb)
+static blk_status_t hib_wait_io(struct hib_bio_batch *hb)
{
wait_event(hb->wait, atomic_read(&hb->count) == 0);
- return hb->error;
+ return blk_status_to_errno(hb->error);
}
/*
unsigned long long k;
unsigned long timeout;
- if ((boot_delay == 0 || system_state != SYSTEM_BOOTING)
+ if ((boot_delay == 0 || system_state >= SYSTEM_RUNNING)
|| suppress_message_printing(level)) {
return;
}
--- /dev/null
+#
+# RCU-related configuration options
+#
+
+menu "RCU Subsystem"
+
+config TREE_RCU
+ bool
+ default y if !PREEMPT && SMP
+ help
+ This option selects the RCU implementation that is
+ designed for very large SMP system with hundreds or
+ thousands of CPUs. It also scales down nicely to
+ smaller systems.
+
+config PREEMPT_RCU
+ bool
+ default y if PREEMPT
+ help
+ This option selects the RCU implementation that is
+ designed for very large SMP systems with hundreds or
+ thousands of CPUs, but for which real-time response
+ is also required. It also scales down nicely to
+ smaller systems.
+
+ Select this option if you are unsure.
+
+config TINY_RCU
+ bool
+ default y if !PREEMPT && !SMP
+ help
+ This option selects the RCU implementation that is
+ designed for UP systems from which real-time response
+ is not required. This option greatly reduces the
+ memory footprint of RCU.
+
+config RCU_EXPERT
+ bool "Make expert-level adjustments to RCU configuration"
+ default n
+ help
+ This option needs to be enabled if you wish to make
+ expert-level adjustments to RCU configuration. By default,
+ no such adjustments can be made, which has the often-beneficial
+ side-effect of preventing "make oldconfig" from asking you all
+ sorts of detailed questions about how you would like numerous
+ obscure RCU options to be set up.
+
+ Say Y if you need to make expert-level adjustments to RCU.
+
+ Say N if you are unsure.
+
+config SRCU
+ bool
+ help
+ This option selects the sleepable version of RCU. This version
+ permits arbitrary sleeping or blocking within RCU read-side critical
+ sections.
+
+config TINY_SRCU
+ bool
+ default y if SRCU && TINY_RCU
+ help
+ This option selects the single-CPU non-preemptible version of SRCU.
+
+config TREE_SRCU
+ bool
+ default y if SRCU && !TINY_RCU
+ help
+ This option selects the full-fledged version of SRCU.
+
+config TASKS_RCU
+ bool
+ default n
+ select SRCU
+ help
+ This option enables a task-based RCU implementation that uses
+ only voluntary context switch (not preemption!), idle, and
+ user-mode execution as quiescent states.
+
+config RCU_STALL_COMMON
+ def_bool ( TREE_RCU || PREEMPT_RCU )
+ help
+ This option enables RCU CPU stall code that is common between
+ the TINY and TREE variants of RCU. The purpose is to allow
+ the tiny variants to disable RCU CPU stall warnings, while
+ making these warnings mandatory for the tree variants.
+
+config RCU_NEED_SEGCBLIST
+ def_bool ( TREE_RCU || PREEMPT_RCU || TREE_SRCU )
+
+config CONTEXT_TRACKING
+ bool
+
+config CONTEXT_TRACKING_FORCE
+ bool "Force context tracking"
+ depends on CONTEXT_TRACKING
+ default y if !NO_HZ_FULL
+ help
+ The major pre-requirement for full dynticks to work is to
+ support the context tracking subsystem. But there are also
+ other dependencies to provide in order to make the full
+ dynticks working.
+
+ This option stands for testing when an arch implements the
+ context tracking backend but doesn't yet fullfill all the
+ requirements to make the full dynticks feature working.
+ Without the full dynticks, there is no way to test the support
+ for context tracking and the subsystems that rely on it: RCU
+ userspace extended quiescent state and tickless cputime
+ accounting. This option copes with the absence of the full
+ dynticks subsystem by forcing the context tracking on all
+ CPUs in the system.
+
+ Say Y only if you're working on the development of an
+ architecture backend for the context tracking.
+
+ Say N otherwise, this option brings an overhead that you
+ don't want in production.
+
+
+config RCU_FANOUT
+ int "Tree-based hierarchical RCU fanout value"
+ range 2 64 if 64BIT
+ range 2 32 if !64BIT
+ depends on (TREE_RCU || PREEMPT_RCU) && RCU_EXPERT
+ default 64 if 64BIT
+ default 32 if !64BIT
+ help
+ This option controls the fanout of hierarchical implementations
+ of RCU, allowing RCU to work efficiently on machines with
+ large numbers of CPUs. This value must be at least the fourth
+ root of NR_CPUS, which allows NR_CPUS to be insanely large.
+ The default value of RCU_FANOUT should be used for production
+ systems, but if you are stress-testing the RCU implementation
+ itself, small RCU_FANOUT values allow you to test large-system
+ code paths on small(er) systems.
+
+ Select a specific number if testing RCU itself.
+ Take the default if unsure.
+
+config RCU_FANOUT_LEAF
+ int "Tree-based hierarchical RCU leaf-level fanout value"
+ range 2 64 if 64BIT
+ range 2 32 if !64BIT
+ depends on (TREE_RCU || PREEMPT_RCU) && RCU_EXPERT
+ default 16
+ help
+ This option controls the leaf-level fanout of hierarchical
+ implementations of RCU, and allows trading off cache misses
+ against lock contention. Systems that synchronize their
+ scheduling-clock interrupts for energy-efficiency reasons will
+ want the default because the smaller leaf-level fanout keeps
+ lock contention levels acceptably low. Very large systems
+ (hundreds or thousands of CPUs) will instead want to set this
+ value to the maximum value possible in order to reduce the
+ number of cache misses incurred during RCU's grace-period
+ initialization. These systems tend to run CPU-bound, and thus
+ are not helped by synchronized interrupts, and thus tend to
+ skew them, which reduces lock contention enough that large
+ leaf-level fanouts work well. That said, setting leaf-level
+ fanout to a large number will likely cause problematic
+ lock contention on the leaf-level rcu_node structures unless
+ you boot with the skew_tick kernel parameter.
+
+ Select a specific number if testing RCU itself.
+
+ Select the maximum permissible value for large systems, but
+ please understand that you may also need to set the skew_tick
+ kernel boot parameter to avoid contention on the rcu_node
+ structure's locks.
+
+ Take the default if unsure.
+
+config RCU_FAST_NO_HZ
+ bool "Accelerate last non-dyntick-idle CPU's grace periods"
+ depends on NO_HZ_COMMON && SMP && RCU_EXPERT
+ default n
+ help
+ This option permits CPUs to enter dynticks-idle state even if
+ they have RCU callbacks queued, and prevents RCU from waking
+ these CPUs up more than roughly once every four jiffies (by
+ default, you can adjust this using the rcutree.rcu_idle_gp_delay
+ parameter), thus improving energy efficiency. On the other
+ hand, this option increases the duration of RCU grace periods,
+ for example, slowing down synchronize_rcu().
+
+ Say Y if energy efficiency is critically important, and you
+ don't care about increased grace-period durations.
+
+ Say N if you are unsure.
+
+config RCU_BOOST
+ bool "Enable RCU priority boosting"
+ depends on RT_MUTEXES && PREEMPT_RCU && RCU_EXPERT
+ default n
+ help
+ This option boosts the priority of preempted RCU readers that
+ block the current preemptible RCU grace period for too long.
+ This option also prevents heavy loads from blocking RCU
+ callback invocation for all flavors of RCU.
+
+ Say Y here if you are working with real-time apps or heavy loads
+ Say N here if you are unsure.
+
+config RCU_BOOST_DELAY
+ int "Milliseconds to delay boosting after RCU grace-period start"
+ range 0 3000
+ depends on RCU_BOOST
+ default 500
+ help
+ This option specifies the time to wait after the beginning of
+ a given grace period before priority-boosting preempted RCU
+ readers blocking that grace period. Note that any RCU reader
+ blocking an expedited RCU grace period is boosted immediately.
+
+ Accept the default if unsure.
+
+config RCU_NOCB_CPU
+ bool "Offload RCU callback processing from boot-selected CPUs"
+ depends on TREE_RCU || PREEMPT_RCU
+ depends on RCU_EXPERT || NO_HZ_FULL
+ default n
+ help
+ Use this option to reduce OS jitter for aggressive HPC or
+ real-time workloads. It can also be used to offload RCU
+ callback invocation to energy-efficient CPUs in battery-powered
+ asymmetric multiprocessors.
+
+ This option offloads callback invocation from the set of
+ CPUs specified at boot time by the rcu_nocbs parameter.
+ For each such CPU, a kthread ("rcuox/N") will be created to
+ invoke callbacks, where the "N" is the CPU being offloaded,
+ and where the "x" is "b" for RCU-bh, "p" for RCU-preempt, and
+ "s" for RCU-sched. Nothing prevents this kthread from running
+ on the specified CPUs, but (1) the kthreads may be preempted
+ between each callback, and (2) affinity or cgroups can be used
+ to force the kthreads to run on whatever set of CPUs is desired.
+
+ Say Y here if you want to help to debug reduced OS jitter.
+ Say N here if you are unsure.
+
+endmenu # "RCU Subsystem"
--- /dev/null
+#
+# RCU-related debugging configuration options
+#
+
+menu "RCU Debugging"
+
+config PROVE_RCU
+ def_bool PROVE_LOCKING
+
+config TORTURE_TEST
+ tristate
+ default n
+
+config RCU_PERF_TEST
+ tristate "performance tests for RCU"
+ depends on DEBUG_KERNEL
+ select TORTURE_TEST
+ select SRCU
+ select TASKS_RCU
+ default n
+ help
+ This option provides a kernel module that runs performance
+ tests on the RCU infrastructure. The kernel module may be built
+ after the fact on the running kernel to be tested, if desired.
+
+ Say Y here if you want RCU performance tests to be built into
+ the kernel.
+ Say M if you want the RCU performance tests to build as a module.
+ Say N if you are unsure.
+
+config RCU_TORTURE_TEST
+ tristate "torture tests for RCU"
+ depends on DEBUG_KERNEL
+ select TORTURE_TEST
+ select SRCU
+ select TASKS_RCU
+ default n
+ help
+ This option provides a kernel module that runs torture tests
+ on the RCU infrastructure. The kernel module may be built
+ after the fact on the running kernel to be tested, if desired.
+
+ Say Y here if you want RCU torture tests to be built into
+ the kernel.
+ Say M if you want the RCU torture tests to build as a module.
+ Say N if you are unsure.
+
+config RCU_CPU_STALL_TIMEOUT
+ int "RCU CPU stall timeout in seconds"
+ depends on RCU_STALL_COMMON
+ range 3 300
+ default 21
+ help
+ If a given RCU grace period extends more than the specified
+ number of seconds, a CPU stall warning is printed. If the
+ RCU grace period persists, additional CPU stall warnings are
+ printed at more widely spaced intervals.
+
+config RCU_TRACE
+ bool "Enable tracing for RCU"
+ depends on DEBUG_KERNEL
+ default y if TREE_RCU
+ select TRACE_CLOCK
+ help
+ This option enables additional tracepoints for ftrace-style
+ event tracing.
+
+ Say Y here if you want to enable RCU tracing
+ Say N if you are unsure.
+
+config RCU_EQS_DEBUG
+ bool "Provide debugging asserts for adding NO_HZ support to an arch"
+ depends on DEBUG_KERNEL
+ help
+ This option provides consistency checks in RCU's handling of
+ NO_HZ. These checks have proven quite helpful in detecting
+ bugs in arch-specific NO_HZ code.
+
+ Say N here if you need ultimate kernel/user switch latencies
+ Say Y if you are unsure
+
+endmenu # "RCU Debugging"
KCOV_INSTRUMENT := n
obj-y += update.o sync.o
-obj-$(CONFIG_CLASSIC_SRCU) += srcu.o
obj-$(CONFIG_TREE_SRCU) += srcutree.o
obj-$(CONFIG_TINY_SRCU) += srcutiny.o
obj-$(CONFIG_RCU_TORTURE_TEST) += rcutorture.o
obj-$(CONFIG_RCU_PERF_TEST) += rcuperf.o
obj-$(CONFIG_TREE_RCU) += tree.o
obj-$(CONFIG_PREEMPT_RCU) += tree.o
-obj-$(CONFIG_TREE_RCU_TRACE) += tree_trace.o
obj-$(CONFIG_TINY_RCU) += tiny.o
obj-$(CONFIG_RCU_NEED_SEGCBLIST) += rcu_segcblist.o
*/
#define TPS(x) tracepoint_string(x)
+/*
+ * Dump the ftrace buffer, but only one time per callsite per boot.
+ */
+#define rcu_ftrace_dump(oops_dump_mode) \
+do { \
+ static atomic_t ___rfd_beenhere = ATOMIC_INIT(0); \
+ \
+ if (!atomic_read(&___rfd_beenhere) && \
+ !atomic_xchg(&___rfd_beenhere, 1)) \
+ ftrace_dump(oops_dump_mode); \
+} while (0)
+
void rcu_early_boot_tests(void);
void rcu_test_sync_prims(void);
cpu <= rnp->grphi; \
cpu = cpumask_next((cpu), cpu_possible_mask))
+/*
+ * Wrappers for the rcu_node::lock acquire and release.
+ *
+ * Because the rcu_nodes form a tree, the tree traversal locking will observe
+ * different lock values, this in turn means that an UNLOCK of one level
+ * followed by a LOCK of another level does not imply a full memory barrier;
+ * and most importantly transitivity is lost.
+ *
+ * In order to restore full ordering between tree levels, augment the regular
+ * lock acquire functions with smp_mb__after_unlock_lock().
+ *
+ * As ->lock of struct rcu_node is a __private field, therefore one should use
+ * these wrappers rather than directly call raw_spin_{lock,unlock}* on ->lock.
+ */
+#define raw_spin_lock_rcu_node(p) \
+do { \
+ raw_spin_lock(&ACCESS_PRIVATE(p, lock)); \
+ smp_mb__after_unlock_lock(); \
+} while (0)
+
+#define raw_spin_unlock_rcu_node(p) raw_spin_unlock(&ACCESS_PRIVATE(p, lock))
+
+#define raw_spin_lock_irq_rcu_node(p) \
+do { \
+ raw_spin_lock_irq(&ACCESS_PRIVATE(p, lock)); \
+ smp_mb__after_unlock_lock(); \
+} while (0)
+
+#define raw_spin_unlock_irq_rcu_node(p) \
+ raw_spin_unlock_irq(&ACCESS_PRIVATE(p, lock))
+
+#define raw_spin_lock_irqsave_rcu_node(p, flags) \
+do { \
+ raw_spin_lock_irqsave(&ACCESS_PRIVATE(p, lock), flags); \
+ smp_mb__after_unlock_lock(); \
+} while (0)
+
+#define raw_spin_unlock_irqrestore_rcu_node(p, flags) \
+ raw_spin_unlock_irqrestore(&ACCESS_PRIVATE(p, lock), flags) \
+
+#define raw_spin_trylock_rcu_node(p) \
+({ \
+ bool ___locked = raw_spin_trylock(&ACCESS_PRIVATE(p, lock)); \
+ \
+ if (___locked) \
+ smp_mb__after_unlock_lock(); \
+ ___locked; \
+})
+
#endif /* #if defined(SRCU) || !defined(TINY_RCU) */
+#ifdef CONFIG_TINY_RCU
+/* Tiny RCU doesn't expedite, as its purpose in life is instead to be tiny. */
+static inline bool rcu_gp_is_normal(void) /* Internal RCU use. */
+{
+ return true;
+}
+static inline bool rcu_gp_is_expedited(void) /* Internal RCU use. */
+{
+ return false;
+}
+
+static inline void rcu_expedite_gp(void)
+{
+}
+
+static inline void rcu_unexpedite_gp(void)
+{
+}
+#else /* #ifdef CONFIG_TINY_RCU */
+bool rcu_gp_is_normal(void); /* Internal RCU use. */
+bool rcu_gp_is_expedited(void); /* Internal RCU use. */
+void rcu_expedite_gp(void);
+void rcu_unexpedite_gp(void);
+void rcupdate_announce_bootup_oddness(void);
+#endif /* #else #ifdef CONFIG_TINY_RCU */
+
+#define RCU_SCHEDULER_INACTIVE 0
+#define RCU_SCHEDULER_INIT 1
+#define RCU_SCHEDULER_RUNNING 2
+
+#ifdef CONFIG_TINY_RCU
+static inline void rcu_request_urgent_qs_task(struct task_struct *t) { }
+#else /* #ifdef CONFIG_TINY_RCU */
+void rcu_request_urgent_qs_task(struct task_struct *t);
+#endif /* #else #ifdef CONFIG_TINY_RCU */
+
+enum rcutorture_type {
+ RCU_FLAVOR,
+ RCU_BH_FLAVOR,
+ RCU_SCHED_FLAVOR,
+ RCU_TASKS_FLAVOR,
+ SRCU_FLAVOR,
+ INVALID_RCU_FLAVOR
+};
+
+#if defined(CONFIG_TREE_RCU) || defined(CONFIG_PREEMPT_RCU)
+void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags,
+ unsigned long *gpnum, unsigned long *completed);
+void rcutorture_record_test_transition(void);
+void rcutorture_record_progress(unsigned long vernum);
+void do_trace_rcu_torture_read(const char *rcutorturename,
+ struct rcu_head *rhp,
+ unsigned long secs,
+ unsigned long c_old,
+ unsigned long c);
+#else
+static inline void rcutorture_get_gp_data(enum rcutorture_type test_type,
+ int *flags,
+ unsigned long *gpnum,
+ unsigned long *completed)
+{
+ *flags = 0;
+ *gpnum = 0;
+ *completed = 0;
+}
+static inline void rcutorture_record_test_transition(void)
+{
+}
+static inline void rcutorture_record_progress(unsigned long vernum)
+{
+}
+#ifdef CONFIG_RCU_TRACE
+void do_trace_rcu_torture_read(const char *rcutorturename,
+ struct rcu_head *rhp,
+ unsigned long secs,
+ unsigned long c_old,
+ unsigned long c);
+#else
+#define do_trace_rcu_torture_read(rcutorturename, rhp, secs, c_old, c) \
+ do { } while (0)
+#endif
+#endif
+
+#ifdef CONFIG_TINY_SRCU
+
+static inline void srcutorture_get_gp_data(enum rcutorture_type test_type,
+ struct srcu_struct *sp, int *flags,
+ unsigned long *gpnum,
+ unsigned long *completed)
+{
+ if (test_type != SRCU_FLAVOR)
+ return;
+ *flags = 0;
+ *completed = sp->srcu_idx;
+ *gpnum = *completed;
+}
+
+#elif defined(CONFIG_TREE_SRCU)
+
+void srcutorture_get_gp_data(enum rcutorture_type test_type,
+ struct srcu_struct *sp, int *flags,
+ unsigned long *gpnum, unsigned long *completed);
+
+#endif
+
+#ifdef CONFIG_TINY_RCU
+
+/*
+ * Return the number of grace periods started.
+ */
+static inline unsigned long rcu_batches_started(void)
+{
+ return 0;
+}
+
+/*
+ * Return the number of bottom-half grace periods started.
+ */
+static inline unsigned long rcu_batches_started_bh(void)
+{
+ return 0;
+}
+
+/*
+ * Return the number of sched grace periods started.
+ */
+static inline unsigned long rcu_batches_started_sched(void)
+{
+ return 0;
+}
+
+/*
+ * Return the number of grace periods completed.
+ */
+static inline unsigned long rcu_batches_completed(void)
+{
+ return 0;
+}
+
+/*
+ * Return the number of bottom-half grace periods completed.
+ */
+static inline unsigned long rcu_batches_completed_bh(void)
+{
+ return 0;
+}
+
+/*
+ * Return the number of sched grace periods completed.
+ */
+static inline unsigned long rcu_batches_completed_sched(void)
+{
+ return 0;
+}
+
+/*
+ * Return the number of expedited grace periods completed.
+ */
+static inline unsigned long rcu_exp_batches_completed(void)
+{
+ return 0;
+}
+
+/*
+ * Return the number of expedited sched grace periods completed.
+ */
+static inline unsigned long rcu_exp_batches_completed_sched(void)
+{
+ return 0;
+}
+
+static inline unsigned long srcu_batches_completed(struct srcu_struct *sp)
+{
+ return 0;
+}
+
+static inline void rcu_force_quiescent_state(void)
+{
+}
+
+static inline void rcu_bh_force_quiescent_state(void)
+{
+}
+
+static inline void rcu_sched_force_quiescent_state(void)
+{
+}
+
+static inline void show_rcu_gp_kthreads(void)
+{
+}
+
+#else /* #ifdef CONFIG_TINY_RCU */
+extern unsigned long rcutorture_testseq;
+extern unsigned long rcutorture_vernum;
+unsigned long rcu_batches_started(void);
+unsigned long rcu_batches_started_bh(void);
+unsigned long rcu_batches_started_sched(void);
+unsigned long rcu_batches_completed(void);
+unsigned long rcu_batches_completed_bh(void);
+unsigned long rcu_batches_completed_sched(void);
+unsigned long rcu_exp_batches_completed(void);
+unsigned long rcu_exp_batches_completed_sched(void);
+unsigned long srcu_batches_completed(struct srcu_struct *sp);
+void show_rcu_gp_kthreads(void);
+void rcu_force_quiescent_state(void);
+void rcu_bh_force_quiescent_state(void);
+void rcu_sched_force_quiescent_state(void);
+#endif /* #else #ifdef CONFIG_TINY_RCU */
+
+#ifdef CONFIG_RCU_NOCB_CPU
+bool rcu_is_nocb_cpu(int cpu);
+#else
+static inline bool rcu_is_nocb_cpu(int cpu) { return false; }
+#endif
+
#endif /* __LINUX_RCU_H */
#include <linux/torture.h>
#include <linux/vmalloc.h>
+#include "rcu.h"
+
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Paul E. McKenney <paulmck@linux.vnet.ibm.com>");
#define VERBOSE_PERFOUT_ERRSTRING(s) \
do { if (verbose) pr_alert("%s" PERF_FLAG "!!! %s\n", perf_type, s); } while (0)
+torture_param(bool, gp_async, false, "Use asynchronous GP wait primitives");
+torture_param(int, gp_async_max, 1000, "Max # outstanding waits per reader");
torture_param(bool, gp_exp, false, "Use expedited GP wait primitives");
torture_param(int, holdoff, 10, "Holdoff time before test start (s)");
-torture_param(int, nreaders, -1, "Number of RCU reader threads");
+torture_param(int, nreaders, 0, "Number of RCU reader threads");
torture_param(int, nwriters, -1, "Number of RCU updater threads");
-torture_param(bool, shutdown, false, "Shutdown at end of performance tests.");
+torture_param(bool, shutdown, !IS_ENABLED(MODULE),
+ "Shutdown at end of performance tests.");
torture_param(bool, verbose, true, "Enable verbose debugging printk()s");
+torture_param(int, writer_holdoff, 0, "Holdoff (us) between GPs, zero to disable");
static char *perf_type = "rcu";
module_param(perf_type, charp, 0444);
static u64 t_rcu_perf_writer_finished;
static unsigned long b_rcu_perf_writer_started;
static unsigned long b_rcu_perf_writer_finished;
+static DEFINE_PER_CPU(atomic_t, n_async_inflight);
static int rcu_perf_writer_state;
#define RTWS_INIT 0
-#define RTWS_EXP_SYNC 1
-#define RTWS_SYNC 2
-#define RTWS_IDLE 2
-#define RTWS_STOPPING 3
+#define RTWS_ASYNC 1
+#define RTWS_BARRIER 2
+#define RTWS_EXP_SYNC 3
+#define RTWS_SYNC 4
+#define RTWS_IDLE 5
+#define RTWS_STOPPING 6
#define MAX_MEAS 10000
#define MIN_MEAS 100
unsigned long (*started)(void);
unsigned long (*completed)(void);
unsigned long (*exp_completed)(void);
+ void (*async)(struct rcu_head *head, rcu_callback_t func);
+ void (*gp_barrier)(void);
void (*sync)(void);
void (*exp_sync)(void);
const char *name;
.started = rcu_batches_started,
.completed = rcu_batches_completed,
.exp_completed = rcu_exp_batches_completed,
+ .async = call_rcu,
+ .gp_barrier = rcu_barrier,
.sync = synchronize_rcu,
.exp_sync = synchronize_rcu_expedited,
.name = "rcu"
.started = rcu_batches_started_bh,
.completed = rcu_batches_completed_bh,
.exp_completed = rcu_exp_batches_completed_sched,
+ .async = call_rcu_bh,
+ .gp_barrier = rcu_barrier_bh,
.sync = synchronize_rcu_bh,
.exp_sync = synchronize_rcu_bh_expedited,
.name = "rcu_bh"
return srcu_batches_completed(srcu_ctlp);
}
+static void srcu_call_rcu(struct rcu_head *head, rcu_callback_t func)
+{
+ call_srcu(srcu_ctlp, head, func);
+}
+
+static void srcu_rcu_barrier(void)
+{
+ srcu_barrier(srcu_ctlp);
+}
+
static void srcu_perf_synchronize(void)
{
synchronize_srcu(srcu_ctlp);
.started = NULL,
.completed = srcu_perf_completed,
.exp_completed = srcu_perf_completed,
+ .async = srcu_call_rcu,
+ .gp_barrier = srcu_rcu_barrier,
.sync = srcu_perf_synchronize,
.exp_sync = srcu_perf_synchronize_expedited,
.name = "srcu"
};
+static struct srcu_struct srcud;
+
+static void srcu_sync_perf_init(void)
+{
+ srcu_ctlp = &srcud;
+ init_srcu_struct(srcu_ctlp);
+}
+
+static void srcu_sync_perf_cleanup(void)
+{
+ cleanup_srcu_struct(srcu_ctlp);
+}
+
+static struct rcu_perf_ops srcud_ops = {
+ .ptype = SRCU_FLAVOR,
+ .init = srcu_sync_perf_init,
+ .cleanup = srcu_sync_perf_cleanup,
+ .readlock = srcu_perf_read_lock,
+ .readunlock = srcu_perf_read_unlock,
+ .started = NULL,
+ .completed = srcu_perf_completed,
+ .exp_completed = srcu_perf_completed,
+ .async = srcu_call_rcu,
+ .gp_barrier = srcu_rcu_barrier,
+ .sync = srcu_perf_synchronize,
+ .exp_sync = srcu_perf_synchronize_expedited,
+ .name = "srcud"
+};
+
/*
* Definitions for sched perf testing.
*/
.started = rcu_batches_started_sched,
.completed = rcu_batches_completed_sched,
.exp_completed = rcu_exp_batches_completed_sched,
+ .async = call_rcu_sched,
+ .gp_barrier = rcu_barrier_sched,
.sync = synchronize_sched,
.exp_sync = synchronize_sched_expedited,
.name = "sched"
.readunlock = tasks_perf_read_unlock,
.started = rcu_no_completed,
.completed = rcu_no_completed,
+ .async = call_rcu_tasks,
+ .gp_barrier = rcu_barrier_tasks,
.sync = synchronize_rcu_tasks,
.exp_sync = synchronize_rcu_tasks,
.name = "tasks"
return 0;
}
+/*
+ * Callback function for asynchronous grace periods from rcu_perf_writer().
+ */
+static void rcu_perf_async_cb(struct rcu_head *rhp)
+{
+ atomic_dec(this_cpu_ptr(&n_async_inflight));
+ kfree(rhp);
+}
+
/*
* RCU perf writer kthread. Repeatedly does a grace period.
*/
int i = 0;
int i_max;
long me = (long)arg;
+ struct rcu_head *rhp = NULL;
struct sched_param sp;
bool started = false, done = false, alldone = false;
u64 t;
}
do {
+ if (writer_holdoff)
+ udelay(writer_holdoff);
wdp = &wdpp[i];
*wdp = ktime_get_mono_fast_ns();
- if (gp_exp) {
+ if (gp_async) {
+retry:
+ if (!rhp)
+ rhp = kmalloc(sizeof(*rhp), GFP_KERNEL);
+ if (rhp && atomic_read(this_cpu_ptr(&n_async_inflight)) < gp_async_max) {
+ rcu_perf_writer_state = RTWS_ASYNC;
+ atomic_inc(this_cpu_ptr(&n_async_inflight));
+ cur_ops->async(rhp, rcu_perf_async_cb);
+ rhp = NULL;
+ } else if (!kthread_should_stop()) {
+ rcu_perf_writer_state = RTWS_BARRIER;
+ cur_ops->gp_barrier();
+ goto retry;
+ } else {
+ kfree(rhp); /* Because we are stopping. */
+ }
+ } else if (gp_exp) {
rcu_perf_writer_state = RTWS_EXP_SYNC;
cur_ops->exp_sync();
} else {
i++;
rcu_perf_wait_shutdown();
} while (!torture_must_stop());
+ if (gp_async) {
+ rcu_perf_writer_state = RTWS_BARRIER;
+ cur_ops->gp_barrier();
+ }
rcu_perf_writer_state = RTWS_STOPPING;
writer_n_durations[me] = i_max;
torture_kthread_stopping("rcu_perf_writer");
u64 *wdp;
u64 *wdpp;
+ /*
+ * Would like warning at start, but everything is expedited
+ * during the mid-boot phase, so have to wait till the end.
+ */
+ if (rcu_gp_is_expedited() && !rcu_gp_is_normal() && !gp_exp)
+ VERBOSE_PERFOUT_ERRSTRING("All grace periods expedited, no normal ones to measure!");
+ if (rcu_gp_is_normal() && gp_exp)
+ VERBOSE_PERFOUT_ERRSTRING("All grace periods normal, no expedited ones to measure!");
+ if (gp_exp && gp_async)
+ VERBOSE_PERFOUT_ERRSTRING("No expedited async GPs, so went with async!");
+
if (torture_cleanup_begin())
return;
long i;
int firsterr = 0;
static struct rcu_perf_ops *perf_ops[] = {
- &rcu_ops, &rcu_bh_ops, &srcu_ops, &sched_ops,
+ &rcu_ops, &rcu_bh_ops, &srcu_ops, &srcud_ops, &sched_ops,
RCUPERF_TASKS_OPS
};
firsterr = -ENOMEM;
goto unwind;
}
- if (rcu_gp_is_expedited() && !rcu_gp_is_normal() && !gp_exp) {
- VERBOSE_PERFOUT_ERRSTRING("All grace periods expedited, no normal ones to measure!");
- firsterr = -EINVAL;
- goto unwind;
- }
- if (rcu_gp_is_normal() && gp_exp) {
- VERBOSE_PERFOUT_ERRSTRING("All grace periods normal, no expedited ones to measure!");
- firsterr = -EINVAL;
- goto unwind;
- }
for (i = 0; i < nrealwriters; i++) {
writer_durations[i] =
kcalloc(MAX_MEAS, sizeof(*writer_durations[i]),
#include <linux/torture.h>
#include <linux/vmalloc.h>
+#include "rcu.h"
+
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Paul E. McKenney <paulmck@us.ibm.com> and Josh Triplett <josh@joshtriplett.org>");
int __maybe_unused cpu;
int idx;
-#if defined(CONFIG_TREE_SRCU) || defined(CONFIG_CLASSIC_SRCU)
#ifdef CONFIG_TREE_SRCU
idx = srcu_ctlp->srcu_idx & 0x1;
-#else /* #ifdef CONFIG_TREE_SRCU */
- idx = srcu_ctlp->completed & 0x1;
-#endif /* #else #ifdef CONFIG_TREE_SRCU */
pr_alert("%s%s Tree SRCU per-CPU(idx=%d):",
torture_type, TORTURE_FLAG, idx);
for_each_possible_cpu(cpu) {
unsigned long l0, l1;
unsigned long u0, u1;
long c0, c1;
-#ifdef CONFIG_TREE_SRCU
struct srcu_data *counts;
counts = per_cpu_ptr(srcu_ctlp->sda, cpu);
u0 = counts->srcu_unlock_count[!idx];
u1 = counts->srcu_unlock_count[idx];
-#else /* #ifdef CONFIG_TREE_SRCU */
- struct srcu_array *counts;
-
- counts = per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu);
- u0 = counts->unlock_count[!idx];
- u1 = counts->unlock_count[idx];
-#endif /* #else #ifdef CONFIG_TREE_SRCU */
/*
* Make sure that a lock is always counted if the corresponding
*/
smp_rmb();
-#ifdef CONFIG_TREE_SRCU
l0 = counts->srcu_lock_count[!idx];
l1 = counts->srcu_lock_count[idx];
-#else /* #ifdef CONFIG_TREE_SRCU */
- l0 = counts->lock_count[!idx];
- l1 = counts->lock_count[idx];
-#endif /* #else #ifdef CONFIG_TREE_SRCU */
c0 = l0 - u0;
c1 = l1 - u1;
pr_cont("\n");
#elif defined(CONFIG_TINY_SRCU)
idx = READ_ONCE(srcu_ctlp->srcu_idx) & 0x1;
- pr_alert("%s%s Tiny SRCU per-CPU(idx=%d): (%d,%d)\n",
+ pr_alert("%s%s Tiny SRCU per-CPU(idx=%d): (%hd,%hd)\n",
torture_type, TORTURE_FLAG, idx,
READ_ONCE(srcu_ctlp->srcu_lock_nesting[!idx]),
READ_ONCE(srcu_ctlp->srcu_lock_nesting[idx]));
+++ /dev/null
-/*
- * Sleepable Read-Copy Update mechanism for mutual exclusion.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, you can access it online at
- * http://www.gnu.org/licenses/gpl-2.0.html.
- *
- * Copyright (C) IBM Corporation, 2006
- * Copyright (C) Fujitsu, 2012
- *
- * Author: Paul McKenney <paulmck@us.ibm.com>
- * Lai Jiangshan <laijs@cn.fujitsu.com>
- *
- * For detailed explanation of Read-Copy Update mechanism see -
- * Documentation/RCU/ *.txt
- *
- */
-
-#include <linux/export.h>
-#include <linux/mutex.h>
-#include <linux/percpu.h>
-#include <linux/preempt.h>
-#include <linux/rcupdate_wait.h>
-#include <linux/sched.h>
-#include <linux/smp.h>
-#include <linux/delay.h>
-#include <linux/srcu.h>
-
-#include "rcu.h"
-
-/*
- * Initialize an rcu_batch structure to empty.
- */
-static inline void rcu_batch_init(struct rcu_batch *b)
-{
- b->head = NULL;
- b->tail = &b->head;
-}
-
-/*
- * Enqueue a callback onto the tail of the specified rcu_batch structure.
- */
-static inline void rcu_batch_queue(struct rcu_batch *b, struct rcu_head *head)
-{
- *b->tail = head;
- b->tail = &head->next;
-}
-
-/*
- * Is the specified rcu_batch structure empty?
- */
-static inline bool rcu_batch_empty(struct rcu_batch *b)
-{
- return b->tail == &b->head;
-}
-
-/*
- * Remove the callback at the head of the specified rcu_batch structure
- * and return a pointer to it, or return NULL if the structure is empty.
- */
-static inline struct rcu_head *rcu_batch_dequeue(struct rcu_batch *b)
-{
- struct rcu_head *head;
-
- if (rcu_batch_empty(b))
- return NULL;
-
- head = b->head;
- b->head = head->next;
- if (b->tail == &head->next)
- rcu_batch_init(b);
-
- return head;
-}
-
-/*
- * Move all callbacks from the rcu_batch structure specified by "from" to
- * the structure specified by "to".
- */
-static inline void rcu_batch_move(struct rcu_batch *to, struct rcu_batch *from)
-{
- if (!rcu_batch_empty(from)) {
- *to->tail = from->head;
- to->tail = from->tail;
- rcu_batch_init(from);
- }
-}
-
-static int init_srcu_struct_fields(struct srcu_struct *sp)
-{
- sp->completed = 0;
- spin_lock_init(&sp->queue_lock);
- sp->running = false;
- rcu_batch_init(&sp->batch_queue);
- rcu_batch_init(&sp->batch_check0);
- rcu_batch_init(&sp->batch_check1);
- rcu_batch_init(&sp->batch_done);
- INIT_DELAYED_WORK(&sp->work, process_srcu);
- sp->per_cpu_ref = alloc_percpu(struct srcu_array);
- return sp->per_cpu_ref ? 0 : -ENOMEM;
-}
-
-#ifdef CONFIG_DEBUG_LOCK_ALLOC
-
-int __init_srcu_struct(struct srcu_struct *sp, const char *name,
- struct lock_class_key *key)
-{
- /* Don't re-initialize a lock while it is held. */
- debug_check_no_locks_freed((void *)sp, sizeof(*sp));
- lockdep_init_map(&sp->dep_map, name, key, 0);
- return init_srcu_struct_fields(sp);
-}
-EXPORT_SYMBOL_GPL(__init_srcu_struct);
-
-#else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
-
-/**
- * init_srcu_struct - initialize a sleep-RCU structure
- * @sp: structure to initialize.
- *
- * Must invoke this on a given srcu_struct before passing that srcu_struct
- * to any other function. Each srcu_struct represents a separate domain
- * of SRCU protection.
- */
-int init_srcu_struct(struct srcu_struct *sp)
-{
- return init_srcu_struct_fields(sp);
-}
-EXPORT_SYMBOL_GPL(init_srcu_struct);
-
-#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
-
-/*
- * Returns approximate total of the readers' ->lock_count[] values for the
- * rank of per-CPU counters specified by idx.
- */
-static unsigned long srcu_readers_lock_idx(struct srcu_struct *sp, int idx)
-{
- int cpu;
- unsigned long sum = 0;
-
- for_each_possible_cpu(cpu) {
- struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
-
- sum += READ_ONCE(cpuc->lock_count[idx]);
- }
- return sum;
-}
-
-/*
- * Returns approximate total of the readers' ->unlock_count[] values for the
- * rank of per-CPU counters specified by idx.
- */
-static unsigned long srcu_readers_unlock_idx(struct srcu_struct *sp, int idx)
-{
- int cpu;
- unsigned long sum = 0;
-
- for_each_possible_cpu(cpu) {
- struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
-
- sum += READ_ONCE(cpuc->unlock_count[idx]);
- }
- return sum;
-}
-
-/*
- * Return true if the number of pre-existing readers is determined to
- * be zero.
- */
-static bool srcu_readers_active_idx_check(struct srcu_struct *sp, int idx)
-{
- unsigned long unlocks;
-
- unlocks = srcu_readers_unlock_idx(sp, idx);
-
- /*
- * Make sure that a lock is always counted if the corresponding unlock
- * is counted. Needs to be a smp_mb() as the read side may contain a
- * read from a variable that is written to before the synchronize_srcu()
- * in the write side. In this case smp_mb()s A and B act like the store
- * buffering pattern.
- *
- * This smp_mb() also pairs with smp_mb() C to prevent accesses after the
- * synchronize_srcu() from being executed before the grace period ends.
- */
- smp_mb(); /* A */
-
- /*
- * If the locks are the same as the unlocks, then there must have
- * been no readers on this index at some time in between. This does not
- * mean that there are no more readers, as one could have read the
- * current index but not have incremented the lock counter yet.
- *
- * Possible bug: There is no guarantee that there haven't been ULONG_MAX
- * increments of ->lock_count[] since the unlocks were counted, meaning
- * that this could return true even if there are still active readers.
- * Since there are no memory barriers around srcu_flip(), the CPU is not
- * required to increment ->completed before running
- * srcu_readers_unlock_idx(), which means that there could be an
- * arbitrarily large number of critical sections that execute after
- * srcu_readers_unlock_idx() but use the old value of ->completed.
- */
- return srcu_readers_lock_idx(sp, idx) == unlocks;
-}
-
-/**
- * srcu_readers_active - returns true if there are readers. and false
- * otherwise
- * @sp: which srcu_struct to count active readers (holding srcu_read_lock).
- *
- * Note that this is not an atomic primitive, and can therefore suffer
- * severe errors when invoked on an active srcu_struct. That said, it
- * can be useful as an error check at cleanup time.
- */
-static bool srcu_readers_active(struct srcu_struct *sp)
-{
- int cpu;
- unsigned long sum = 0;
-
- for_each_possible_cpu(cpu) {
- struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
-
- sum += READ_ONCE(cpuc->lock_count[0]);
- sum += READ_ONCE(cpuc->lock_count[1]);
- sum -= READ_ONCE(cpuc->unlock_count[0]);
- sum -= READ_ONCE(cpuc->unlock_count[1]);
- }
- return sum;
-}
-
-/**
- * cleanup_srcu_struct - deconstruct a sleep-RCU structure
- * @sp: structure to clean up.
- *
- * Must invoke this only after you are finished using a given srcu_struct
- * that was initialized via init_srcu_struct(). This code does some
- * probabalistic checking, spotting late uses of srcu_read_lock(),
- * synchronize_srcu(), synchronize_srcu_expedited(), and call_srcu().
- * If any such late uses are detected, the per-CPU memory associated with
- * the srcu_struct is simply leaked and WARN_ON() is invoked. If the
- * caller frees the srcu_struct itself, a use-after-free crash will likely
- * ensue, but at least there will be a warning printed.
- */
-void cleanup_srcu_struct(struct srcu_struct *sp)
-{
- if (WARN_ON(srcu_readers_active(sp)))
- return; /* Leakage unless caller handles error. */
- free_percpu(sp->per_cpu_ref);
- sp->per_cpu_ref = NULL;
-}
-EXPORT_SYMBOL_GPL(cleanup_srcu_struct);
-
-/*
- * Counts the new reader in the appropriate per-CPU element of the
- * srcu_struct.
- * Returns an index that must be passed to the matching srcu_read_unlock().
- */
-int __srcu_read_lock(struct srcu_struct *sp)
-{
- int idx;
-
- idx = READ_ONCE(sp->completed) & 0x1;
- this_cpu_inc(sp->per_cpu_ref->lock_count[idx]);
- smp_mb(); /* B */ /* Avoid leaking the critical section. */
- return idx;
-}
-EXPORT_SYMBOL_GPL(__srcu_read_lock);
-
-/*
- * Removes the count for the old reader from the appropriate per-CPU
- * element of the srcu_struct. Note that this may well be a different
- * CPU than that which was incremented by the corresponding srcu_read_lock().
- */
-void __srcu_read_unlock(struct srcu_struct *sp, int idx)
-{
- smp_mb(); /* C */ /* Avoid leaking the critical section. */
- this_cpu_inc(sp->per_cpu_ref->unlock_count[idx]);
-}
-EXPORT_SYMBOL_GPL(__srcu_read_unlock);
-
-/*
- * We use an adaptive strategy for synchronize_srcu() and especially for
- * synchronize_srcu_expedited(). We spin for a fixed time period
- * (defined below) to allow SRCU readers to exit their read-side critical
- * sections. If there are still some readers after 10 microseconds,
- * we repeatedly block for 1-millisecond time periods. This approach
- * has done well in testing, so there is no need for a config parameter.
- */
-#define SRCU_RETRY_CHECK_DELAY 5
-#define SYNCHRONIZE_SRCU_TRYCOUNT 2
-#define SYNCHRONIZE_SRCU_EXP_TRYCOUNT 12
-
-/*
- * @@@ Wait until all pre-existing readers complete. Such readers
- * will have used the index specified by "idx".
- * the caller should ensures the ->completed is not changed while checking
- * and idx = (->completed & 1) ^ 1
- */
-static bool try_check_zero(struct srcu_struct *sp, int idx, int trycount)
-{
- for (;;) {
- if (srcu_readers_active_idx_check(sp, idx))
- return true;
- if (--trycount <= 0)
- return false;
- udelay(SRCU_RETRY_CHECK_DELAY);
- }
-}
-
-/*
- * Increment the ->completed counter so that future SRCU readers will
- * use the other rank of the ->(un)lock_count[] arrays. This allows
- * us to wait for pre-existing readers in a starvation-free manner.
- */
-static void srcu_flip(struct srcu_struct *sp)
-{
- WRITE_ONCE(sp->completed, sp->completed + 1);
-
- /*
- * Ensure that if the updater misses an __srcu_read_unlock()
- * increment, that task's next __srcu_read_lock() will see the
- * above counter update. Note that both this memory barrier
- * and the one in srcu_readers_active_idx_check() provide the
- * guarantee for __srcu_read_lock().
- */
- smp_mb(); /* D */ /* Pairs with C. */
-}
-
-/*
- * Enqueue an SRCU callback on the specified srcu_struct structure,
- * initiating grace-period processing if it is not already running.
- *
- * Note that all CPUs must agree that the grace period extended beyond
- * all pre-existing SRCU read-side critical section. On systems with
- * more than one CPU, this means that when "func()" is invoked, each CPU
- * is guaranteed to have executed a full memory barrier since the end of
- * its last corresponding SRCU read-side critical section whose beginning
- * preceded the call to call_rcu(). It also means that each CPU executing
- * an SRCU read-side critical section that continues beyond the start of
- * "func()" must have executed a memory barrier after the call_rcu()
- * but before the beginning of that SRCU read-side critical section.
- * Note that these guarantees include CPUs that are offline, idle, or
- * executing in user mode, as well as CPUs that are executing in the kernel.
- *
- * Furthermore, if CPU A invoked call_rcu() and CPU B invoked the
- * resulting SRCU callback function "func()", then both CPU A and CPU
- * B are guaranteed to execute a full memory barrier during the time
- * interval between the call to call_rcu() and the invocation of "func()".
- * This guarantee applies even if CPU A and CPU B are the same CPU (but
- * again only if the system has more than one CPU).
- *
- * Of course, these guarantees apply only for invocations of call_srcu(),
- * srcu_read_lock(), and srcu_read_unlock() that are all passed the same
- * srcu_struct structure.
- */
-void call_srcu(struct srcu_struct *sp, struct rcu_head *head,
- rcu_callback_t func)
-{
- unsigned long flags;
-
- head->next = NULL;
- head->func = func;
- spin_lock_irqsave(&sp->queue_lock, flags);
- smp_mb__after_unlock_lock(); /* Caller's prior accesses before GP. */
- rcu_batch_queue(&sp->batch_queue, head);
- if (!sp->running) {
- sp->running = true;
- queue_delayed_work(system_power_efficient_wq, &sp->work, 0);
- }
- spin_unlock_irqrestore(&sp->queue_lock, flags);
-}
-EXPORT_SYMBOL_GPL(call_srcu);
-
-static void srcu_advance_batches(struct srcu_struct *sp, int trycount);
-static void srcu_reschedule(struct srcu_struct *sp);
-
-/*
- * Helper function for synchronize_srcu() and synchronize_srcu_expedited().
- */
-static void __synchronize_srcu(struct srcu_struct *sp, int trycount)
-{
- struct rcu_synchronize rcu;
- struct rcu_head *head = &rcu.head;
- bool done = false;
-
- RCU_LOCKDEP_WARN(lock_is_held(&sp->dep_map) ||
- lock_is_held(&rcu_bh_lock_map) ||
- lock_is_held(&rcu_lock_map) ||
- lock_is_held(&rcu_sched_lock_map),
- "Illegal synchronize_srcu() in same-type SRCU (or in RCU) read-side critical section");
-
- might_sleep();
- init_completion(&rcu.completion);
-
- head->next = NULL;
- head->func = wakeme_after_rcu;
- spin_lock_irq(&sp->queue_lock);
- smp_mb__after_unlock_lock(); /* Caller's prior accesses before GP. */
- if (!sp->running) {
- /* steal the processing owner */
- sp->running = true;
- rcu_batch_queue(&sp->batch_check0, head);
- spin_unlock_irq(&sp->queue_lock);
-
- srcu_advance_batches(sp, trycount);
- if (!rcu_batch_empty(&sp->batch_done)) {
- BUG_ON(sp->batch_done.head != head);
- rcu_batch_dequeue(&sp->batch_done);
- done = true;
- }
- /* give the processing owner to work_struct */
- srcu_reschedule(sp);
- } else {
- rcu_batch_queue(&sp->batch_queue, head);
- spin_unlock_irq(&sp->queue_lock);
- }
-
- if (!done) {
- wait_for_completion(&rcu.completion);
- smp_mb(); /* Caller's later accesses after GP. */
- }
-
-}
-
-/**
- * synchronize_srcu - wait for prior SRCU read-side critical-section completion
- * @sp: srcu_struct with which to synchronize.
- *
- * Wait for the count to drain to zero of both indexes. To avoid the
- * possible starvation of synchronize_srcu(), it waits for the count of
- * the index=((->completed & 1) ^ 1) to drain to zero at first,
- * and then flip the completed and wait for the count of the other index.
- *
- * Can block; must be called from process context.
- *
- * Note that it is illegal to call synchronize_srcu() from the corresponding
- * SRCU read-side critical section; doing so will result in deadlock.
- * However, it is perfectly legal to call synchronize_srcu() on one
- * srcu_struct from some other srcu_struct's read-side critical section,
- * as long as the resulting graph of srcu_structs is acyclic.
- *
- * There are memory-ordering constraints implied by synchronize_srcu().
- * On systems with more than one CPU, when synchronize_srcu() returns,
- * each CPU is guaranteed to have executed a full memory barrier since
- * the end of its last corresponding SRCU-sched read-side critical section
- * whose beginning preceded the call to synchronize_srcu(). In addition,
- * each CPU having an SRCU read-side critical section that extends beyond
- * the return from synchronize_srcu() is guaranteed to have executed a
- * full memory barrier after the beginning of synchronize_srcu() and before
- * the beginning of that SRCU read-side critical section. Note that these
- * guarantees include CPUs that are offline, idle, or executing in user mode,
- * as well as CPUs that are executing in the kernel.
- *
- * Furthermore, if CPU A invoked synchronize_srcu(), which returned
- * to its caller on CPU B, then both CPU A and CPU B are guaranteed
- * to have executed a full memory barrier during the execution of
- * synchronize_srcu(). This guarantee applies even if CPU A and CPU B
- * are the same CPU, but again only if the system has more than one CPU.
- *
- * Of course, these memory-ordering guarantees apply only when
- * synchronize_srcu(), srcu_read_lock(), and srcu_read_unlock() are
- * passed the same srcu_struct structure.
- */
-void synchronize_srcu(struct srcu_struct *sp)
-{
- __synchronize_srcu(sp, (rcu_gp_is_expedited() && !rcu_gp_is_normal())
- ? SYNCHRONIZE_SRCU_EXP_TRYCOUNT
- : SYNCHRONIZE_SRCU_TRYCOUNT);
-}
-EXPORT_SYMBOL_GPL(synchronize_srcu);
-
-/**
- * synchronize_srcu_expedited - Brute-force SRCU grace period
- * @sp: srcu_struct with which to synchronize.
- *
- * Wait for an SRCU grace period to elapse, but be more aggressive about
- * spinning rather than blocking when waiting.
- *
- * Note that synchronize_srcu_expedited() has the same deadlock and
- * memory-ordering properties as does synchronize_srcu().
- */
-void synchronize_srcu_expedited(struct srcu_struct *sp)
-{
- __synchronize_srcu(sp, SYNCHRONIZE_SRCU_EXP_TRYCOUNT);
-}
-EXPORT_SYMBOL_GPL(synchronize_srcu_expedited);
-
-/**
- * srcu_barrier - Wait until all in-flight call_srcu() callbacks complete.
- * @sp: srcu_struct on which to wait for in-flight callbacks.
- */
-void srcu_barrier(struct srcu_struct *sp)
-{
- synchronize_srcu(sp);
-}
-EXPORT_SYMBOL_GPL(srcu_barrier);
-
-/**
- * srcu_batches_completed - return batches completed.
- * @sp: srcu_struct on which to report batch completion.
- *
- * Report the number of batches, correlated with, but not necessarily
- * precisely the same as, the number of grace periods that have elapsed.
- */
-unsigned long srcu_batches_completed(struct srcu_struct *sp)
-{
- return sp->completed;
-}
-EXPORT_SYMBOL_GPL(srcu_batches_completed);
-
-#define SRCU_CALLBACK_BATCH 10
-#define SRCU_INTERVAL 1
-
-/*
- * Move any new SRCU callbacks to the first stage of the SRCU grace
- * period pipeline.
- */
-static void srcu_collect_new(struct srcu_struct *sp)
-{
- if (!rcu_batch_empty(&sp->batch_queue)) {
- spin_lock_irq(&sp->queue_lock);
- rcu_batch_move(&sp->batch_check0, &sp->batch_queue);
- spin_unlock_irq(&sp->queue_lock);
- }
-}
-
-/*
- * Core SRCU state machine. Advance callbacks from ->batch_check0 to
- * ->batch_check1 and then to ->batch_done as readers drain.
- */
-static void srcu_advance_batches(struct srcu_struct *sp, int trycount)
-{
- int idx = 1 ^ (sp->completed & 1);
-
- /*
- * Because readers might be delayed for an extended period after
- * fetching ->completed for their index, at any point in time there
- * might well be readers using both idx=0 and idx=1. We therefore
- * need to wait for readers to clear from both index values before
- * invoking a callback.
- */
-
- if (rcu_batch_empty(&sp->batch_check0) &&
- rcu_batch_empty(&sp->batch_check1))
- return; /* no callbacks need to be advanced */
-
- if (!try_check_zero(sp, idx, trycount))
- return; /* failed to advance, will try after SRCU_INTERVAL */
-
- /*
- * The callbacks in ->batch_check1 have already done with their
- * first zero check and flip back when they were enqueued on
- * ->batch_check0 in a previous invocation of srcu_advance_batches().
- * (Presumably try_check_zero() returned false during that
- * invocation, leaving the callbacks stranded on ->batch_check1.)
- * They are therefore ready to invoke, so move them to ->batch_done.
- */
- rcu_batch_move(&sp->batch_done, &sp->batch_check1);
-
- if (rcu_batch_empty(&sp->batch_check0))
- return; /* no callbacks need to be advanced */
- srcu_flip(sp);
-
- /*
- * The callbacks in ->batch_check0 just finished their
- * first check zero and flip, so move them to ->batch_check1
- * for future checking on the other idx.
- */
- rcu_batch_move(&sp->batch_check1, &sp->batch_check0);
-
- /*
- * SRCU read-side critical sections are normally short, so check
- * at least twice in quick succession after a flip.
- */
- trycount = trycount < 2 ? 2 : trycount;
- if (!try_check_zero(sp, idx^1, trycount))
- return; /* failed to advance, will try after SRCU_INTERVAL */
-
- /*
- * The callbacks in ->batch_check1 have now waited for all
- * pre-existing readers using both idx values. They are therefore
- * ready to invoke, so move them to ->batch_done.
- */
- rcu_batch_move(&sp->batch_done, &sp->batch_check1);
-}
-
-/*
- * Invoke a limited number of SRCU callbacks that have passed through
- * their grace period. If there are more to do, SRCU will reschedule
- * the workqueue. Note that needed memory barriers have been executed
- * in this task's context by srcu_readers_active_idx_check().
- */
-static void srcu_invoke_callbacks(struct srcu_struct *sp)
-{
- int i;
- struct rcu_head *head;
-
- for (i = 0; i < SRCU_CALLBACK_BATCH; i++) {
- head = rcu_batch_dequeue(&sp->batch_done);
- if (!head)
- break;
- local_bh_disable();
- head->func(head);
- local_bh_enable();
- }
-}
-
-/*
- * Finished one round of SRCU grace period. Start another if there are
- * more SRCU callbacks queued, otherwise put SRCU into not-running state.
- */
-static void srcu_reschedule(struct srcu_struct *sp)
-{
- bool pending = true;
-
- if (rcu_batch_empty(&sp->batch_done) &&
- rcu_batch_empty(&sp->batch_check1) &&
- rcu_batch_empty(&sp->batch_check0) &&
- rcu_batch_empty(&sp->batch_queue)) {
- spin_lock_irq(&sp->queue_lock);
- if (rcu_batch_empty(&sp->batch_done) &&
- rcu_batch_empty(&sp->batch_check1) &&
- rcu_batch_empty(&sp->batch_check0) &&
- rcu_batch_empty(&sp->batch_queue)) {
- sp->running = false;
- pending = false;
- }
- spin_unlock_irq(&sp->queue_lock);
- }
-
- if (pending)
- queue_delayed_work(system_power_efficient_wq,
- &sp->work, SRCU_INTERVAL);
-}
-
-/*
- * This is the work-queue function that handles SRCU grace periods.
- */
-void process_srcu(struct work_struct *work)
-{
- struct srcu_struct *sp;
-
- sp = container_of(work, struct srcu_struct, work.work);
-
- srcu_collect_new(sp);
- srcu_advance_batches(sp, 1);
- srcu_invoke_callbacks(sp);
- srcu_reschedule(sp);
-}
-EXPORT_SYMBOL_GPL(process_srcu);
sp->srcu_lock_nesting[0] = 0;
sp->srcu_lock_nesting[1] = 0;
init_swait_queue_head(&sp->srcu_wq);
- sp->srcu_gp_seq = 0;
- rcu_segcblist_init(&sp->srcu_cblist);
+ sp->srcu_cb_head = NULL;
+ sp->srcu_cb_tail = &sp->srcu_cb_head;
sp->srcu_gp_running = false;
sp->srcu_gp_waiting = false;
sp->srcu_idx = 0;
{
WARN_ON(sp->srcu_lock_nesting[0] || sp->srcu_lock_nesting[1]);
flush_work(&sp->srcu_work);
- WARN_ON(rcu_seq_state(sp->srcu_gp_seq));
WARN_ON(sp->srcu_gp_running);
WARN_ON(sp->srcu_gp_waiting);
- WARN_ON(!rcu_segcblist_empty(&sp->srcu_cblist));
+ WARN_ON(sp->srcu_cb_head);
+ WARN_ON(&sp->srcu_cb_head != sp->srcu_cb_tail);
}
EXPORT_SYMBOL_GPL(cleanup_srcu_struct);
-/*
- * Counts the new reader in the appropriate per-CPU element of the
- * srcu_struct. Can be invoked from irq/bh handlers, but the matching
- * __srcu_read_unlock() must be in the same handler instance. Returns an
- * index that must be passed to the matching srcu_read_unlock().
- */
-int __srcu_read_lock(struct srcu_struct *sp)
-{
- int idx;
-
- idx = READ_ONCE(sp->srcu_idx);
- WRITE_ONCE(sp->srcu_lock_nesting[idx], sp->srcu_lock_nesting[idx] + 1);
- return idx;
-}
-EXPORT_SYMBOL_GPL(__srcu_read_lock);
-
/*
* Removes the count for the old reader from the appropriate element of
* the srcu_struct.
void srcu_drive_gp(struct work_struct *wp)
{
int idx;
- struct rcu_cblist ready_cbs;
- struct srcu_struct *sp;
+ struct rcu_head *lh;
struct rcu_head *rhp;
+ struct srcu_struct *sp;
sp = container_of(wp, struct srcu_struct, srcu_work);
- if (sp->srcu_gp_running || rcu_segcblist_empty(&sp->srcu_cblist))
+ if (sp->srcu_gp_running || !READ_ONCE(sp->srcu_cb_head))
return; /* Already running or nothing to do. */
- /* Tag recently arrived callbacks and wait for readers. */
+ /* Remove recently arrived callbacks and wait for readers. */
WRITE_ONCE(sp->srcu_gp_running, true);
- rcu_segcblist_accelerate(&sp->srcu_cblist,
- rcu_seq_snap(&sp->srcu_gp_seq));
- rcu_seq_start(&sp->srcu_gp_seq);
+ local_irq_disable();
+ lh = sp->srcu_cb_head;
+ sp->srcu_cb_head = NULL;
+ sp->srcu_cb_tail = &sp->srcu_cb_head;
+ local_irq_enable();
idx = sp->srcu_idx;
WRITE_ONCE(sp->srcu_idx, !sp->srcu_idx);
WRITE_ONCE(sp->srcu_gp_waiting, true); /* srcu_read_unlock() wakes! */
swait_event(sp->srcu_wq, !READ_ONCE(sp->srcu_lock_nesting[idx]));
WRITE_ONCE(sp->srcu_gp_waiting, false); /* srcu_read_unlock() cheap. */
- rcu_seq_end(&sp->srcu_gp_seq);
-
- /* Update callback list based on GP, and invoke ready callbacks. */
- rcu_segcblist_advance(&sp->srcu_cblist,
- rcu_seq_current(&sp->srcu_gp_seq));
- if (rcu_segcblist_ready_cbs(&sp->srcu_cblist)) {
- rcu_cblist_init(&ready_cbs);
- local_irq_disable();
- rcu_segcblist_extract_done_cbs(&sp->srcu_cblist, &ready_cbs);
- local_irq_enable();
- rhp = rcu_cblist_dequeue(&ready_cbs);
- for (; rhp != NULL; rhp = rcu_cblist_dequeue(&ready_cbs)) {
- local_bh_disable();
- rhp->func(rhp);
- local_bh_enable();
- }
- local_irq_disable();
- rcu_segcblist_insert_count(&sp->srcu_cblist, &ready_cbs);
- local_irq_enable();
+
+ /* Invoke the callbacks we removed above. */
+ while (lh) {
+ rhp = lh;
+ lh = lh->next;
+ local_bh_disable();
+ rhp->func(rhp);
+ local_bh_enable();
}
- WRITE_ONCE(sp->srcu_gp_running, false);
/*
- * If more callbacks, reschedule ourselves. This can race with
- * a call_srcu() at interrupt level, but the ->srcu_gp_running
- * checks will straighten that out.
+ * Enable rescheduling, and if there are more callbacks,
+ * reschedule ourselves. This can race with a call_srcu()
+ * at interrupt level, but the ->srcu_gp_running checks will
+ * straighten that out.
*/
- if (!rcu_segcblist_empty(&sp->srcu_cblist))
+ WRITE_ONCE(sp->srcu_gp_running, false);
+ if (READ_ONCE(sp->srcu_cb_head))
schedule_work(&sp->srcu_work);
}
EXPORT_SYMBOL_GPL(srcu_drive_gp);
* Enqueue an SRCU callback on the specified srcu_struct structure,
* initiating grace-period processing if it is not already running.
*/
-void call_srcu(struct srcu_struct *sp, struct rcu_head *head,
+void call_srcu(struct srcu_struct *sp, struct rcu_head *rhp,
rcu_callback_t func)
{
unsigned long flags;
- head->func = func;
+ rhp->func = func;
+ rhp->next = NULL;
local_irq_save(flags);
- rcu_segcblist_enqueue(&sp->srcu_cblist, head, false);
+ *sp->srcu_cb_tail = rhp;
+ sp->srcu_cb_tail = &rhp->next;
local_irq_restore(flags);
if (!READ_ONCE(sp->srcu_gp_running))
schedule_work(&sp->srcu_work);
#include "rcu.h"
#include "rcu_segcblist.h"
-ulong exp_holdoff = 25 * 1000; /* Holdoff (ns) for auto-expediting. */
+/* Holdoff in nanoseconds for auto-expediting. */
+#define DEFAULT_SRCU_EXP_HOLDOFF (25 * 1000)
+static ulong exp_holdoff = DEFAULT_SRCU_EXP_HOLDOFF;
module_param(exp_holdoff, ulong, 0444);
+/* Overflow-check frequency. N bits roughly says every 2**N grace periods. */
+static ulong counter_wrap_check = (ULONG_MAX >> 2);
+module_param(counter_wrap_check, ulong, 0444);
+
static void srcu_invoke_callbacks(struct work_struct *work);
static void srcu_reschedule(struct srcu_struct *sp, unsigned long delay);
/* Each pass through this loop initializes one srcu_node structure. */
rcu_for_each_node_breadth_first(sp, snp) {
- spin_lock_init(&snp->lock);
+ raw_spin_lock_init(&ACCESS_PRIVATE(snp, lock));
WARN_ON_ONCE(ARRAY_SIZE(snp->srcu_have_cbs) !=
ARRAY_SIZE(snp->srcu_data_have_cbs));
for (i = 0; i < ARRAY_SIZE(snp->srcu_have_cbs); i++) {
snp_first = sp->level[level];
for_each_possible_cpu(cpu) {
sdp = per_cpu_ptr(sp->sda, cpu);
- spin_lock_init(&sdp->lock);
+ raw_spin_lock_init(&ACCESS_PRIVATE(sdp, lock));
rcu_segcblist_init(&sdp->srcu_cblist);
sdp->srcu_cblist_invoking = false;
sdp->srcu_gp_seq_needed = sp->srcu_gp_seq;
/* Don't re-initialize a lock while it is held. */
debug_check_no_locks_freed((void *)sp, sizeof(*sp));
lockdep_init_map(&sp->dep_map, name, key, 0);
- spin_lock_init(&sp->gp_lock);
+ raw_spin_lock_init(&ACCESS_PRIVATE(sp, lock));
return init_srcu_struct_fields(sp, false);
}
EXPORT_SYMBOL_GPL(__init_srcu_struct);
*/
int init_srcu_struct(struct srcu_struct *sp)
{
- spin_lock_init(&sp->gp_lock);
+ raw_spin_lock_init(&ACCESS_PRIVATE(sp, lock));
return init_srcu_struct_fields(sp, false);
}
EXPORT_SYMBOL_GPL(init_srcu_struct);
* First-use initialization of statically allocated srcu_struct
* structure. Wiring up the combining tree is more than can be
* done with compile-time initialization, so this check is added
- * to each update-side SRCU primitive. Use ->gp_lock, which -is-
+ * to each update-side SRCU primitive. Use sp->lock, which -is-
* compile-time initialized, to resolve races involving multiple
* CPUs trying to garner first-use privileges.
*/
/* The smp_load_acquire() pairs with the smp_store_release(). */
if (!rcu_seq_state(smp_load_acquire(&sp->srcu_gp_seq_needed))) /*^^^*/
return; /* Already initialized. */
- spin_lock_irqsave(&sp->gp_lock, flags);
+ raw_spin_lock_irqsave_rcu_node(sp, flags);
if (!rcu_seq_state(sp->srcu_gp_seq_needed)) {
- spin_unlock_irqrestore(&sp->gp_lock, flags);
+ raw_spin_unlock_irqrestore_rcu_node(sp, flags);
return;
}
init_srcu_struct_fields(sp, true);
- spin_unlock_irqrestore(&sp->gp_lock, flags);
+ raw_spin_unlock_irqrestore_rcu_node(sp, flags);
}
/*
* not mean that there are no more readers, as one could have read
* the current index but not have incremented the lock counter yet.
*
- * Possible bug: There is no guarantee that there haven't been
- * ULONG_MAX increments of ->srcu_lock_count[] since the unlocks were
- * counted, meaning that this could return true even if there are
- * still active readers. Since there are no memory barriers around
- * srcu_flip(), the CPU is not required to increment ->srcu_idx
- * before running srcu_readers_unlock_idx(), which means that there
- * could be an arbitrarily large number of critical sections that
- * execute after srcu_readers_unlock_idx() but use the old value
- * of ->srcu_idx.
+ * So suppose that the updater is preempted here for so long
+ * that more than ULONG_MAX non-nested readers come and go in
+ * the meantime. It turns out that this cannot result in overflow
+ * because if a reader modifies its unlock count after we read it
+ * above, then that reader's next load of ->srcu_idx is guaranteed
+ * to get the new value, which will cause it to operate on the
+ * other bank of counters, where it cannot contribute to the
+ * overflow of these counters. This means that there is a maximum
+ * of 2*NR_CPUS increments, which cannot overflow given current
+ * systems, especially not on 64-bit systems.
+ *
+ * OK, how about nesting? This does impose a limit on nesting
+ * of floor(ULONG_MAX/NR_CPUS/2), which should be sufficient,
+ * especially on 64-bit systems.
*/
return srcu_readers_lock_idx(sp, idx) == unlocks;
}
struct srcu_data *sdp = this_cpu_ptr(sp->sda);
int state;
- RCU_LOCKDEP_WARN(!lockdep_is_held(&sp->gp_lock),
- "Invoked srcu_gp_start() without ->gp_lock!");
+ lockdep_assert_held(&sp->lock);
WARN_ON_ONCE(ULONG_CMP_GE(sp->srcu_gp_seq, sp->srcu_gp_seq_needed));
rcu_segcblist_advance(&sdp->srcu_cblist,
rcu_seq_current(&sp->srcu_gp_seq));
{
unsigned long cbdelay;
bool cbs;
+ int cpu;
+ unsigned long flags;
unsigned long gpseq;
int idx;
int idxnext;
unsigned long mask;
+ struct srcu_data *sdp;
struct srcu_node *snp;
/* Prevent more than one additional grace period. */
mutex_lock(&sp->srcu_cb_mutex);
/* End the current grace period. */
- spin_lock_irq(&sp->gp_lock);
+ raw_spin_lock_irq_rcu_node(sp);
idx = rcu_seq_state(sp->srcu_gp_seq);
WARN_ON_ONCE(idx != SRCU_STATE_SCAN2);
cbdelay = srcu_get_delay(sp);
gpseq = rcu_seq_current(&sp->srcu_gp_seq);
if (ULONG_CMP_LT(sp->srcu_gp_seq_needed_exp, gpseq))
sp->srcu_gp_seq_needed_exp = gpseq;
- spin_unlock_irq(&sp->gp_lock);
+ raw_spin_unlock_irq_rcu_node(sp);
mutex_unlock(&sp->srcu_gp_mutex);
/* A new grace period can start at this point. But only one. */
idx = rcu_seq_ctr(gpseq) % ARRAY_SIZE(snp->srcu_have_cbs);
idxnext = (idx + 1) % ARRAY_SIZE(snp->srcu_have_cbs);
rcu_for_each_node_breadth_first(sp, snp) {
- spin_lock_irq(&snp->lock);
+ raw_spin_lock_irq_rcu_node(snp);
cbs = false;
if (snp >= sp->level[rcu_num_lvls - 1])
cbs = snp->srcu_have_cbs[idx] == gpseq;
snp->srcu_gp_seq_needed_exp = gpseq;
mask = snp->srcu_data_have_cbs[idx];
snp->srcu_data_have_cbs[idx] = 0;
- spin_unlock_irq(&snp->lock);
- if (cbs) {
- smp_mb(); /* GP end before CB invocation. */
+ raw_spin_unlock_irq_rcu_node(snp);
+ if (cbs)
srcu_schedule_cbs_snp(sp, snp, mask, cbdelay);
- }
+
+ /* Occasionally prevent srcu_data counter wrap. */
+ if (!(gpseq & counter_wrap_check))
+ for (cpu = snp->grplo; cpu <= snp->grphi; cpu++) {
+ sdp = per_cpu_ptr(sp->sda, cpu);
+ raw_spin_lock_irqsave_rcu_node(sdp, flags);
+ if (ULONG_CMP_GE(gpseq,
+ sdp->srcu_gp_seq_needed + 100))
+ sdp->srcu_gp_seq_needed = gpseq;
+ raw_spin_unlock_irqrestore_rcu_node(sdp, flags);
+ }
}
/* Callback initiation done, allow grace periods after next. */
mutex_unlock(&sp->srcu_cb_mutex);
/* Start a new grace period if needed. */
- spin_lock_irq(&sp->gp_lock);
+ raw_spin_lock_irq_rcu_node(sp);
gpseq = rcu_seq_current(&sp->srcu_gp_seq);
if (!rcu_seq_state(gpseq) &&
ULONG_CMP_LT(gpseq, sp->srcu_gp_seq_needed)) {
srcu_gp_start(sp);
- spin_unlock_irq(&sp->gp_lock);
+ raw_spin_unlock_irq_rcu_node(sp);
/* Throttle expedited grace periods: Should be rare! */
srcu_reschedule(sp, rcu_seq_ctr(gpseq) & 0x3ff
? 0 : SRCU_INTERVAL);
} else {
- spin_unlock_irq(&sp->gp_lock);
+ raw_spin_unlock_irq_rcu_node(sp);
}
}
if (rcu_seq_done(&sp->srcu_gp_seq, s) ||
ULONG_CMP_GE(READ_ONCE(snp->srcu_gp_seq_needed_exp), s))
return;
- spin_lock_irqsave(&snp->lock, flags);
+ raw_spin_lock_irqsave_rcu_node(snp, flags);
if (ULONG_CMP_GE(snp->srcu_gp_seq_needed_exp, s)) {
- spin_unlock_irqrestore(&snp->lock, flags);
+ raw_spin_unlock_irqrestore_rcu_node(snp, flags);
return;
}
WRITE_ONCE(snp->srcu_gp_seq_needed_exp, s);
- spin_unlock_irqrestore(&snp->lock, flags);
+ raw_spin_unlock_irqrestore_rcu_node(snp, flags);
}
- spin_lock_irqsave(&sp->gp_lock, flags);
+ raw_spin_lock_irqsave_rcu_node(sp, flags);
if (!ULONG_CMP_LT(sp->srcu_gp_seq_needed_exp, s))
sp->srcu_gp_seq_needed_exp = s;
- spin_unlock_irqrestore(&sp->gp_lock, flags);
+ raw_spin_unlock_irqrestore_rcu_node(sp, flags);
}
/*
for (; snp != NULL; snp = snp->srcu_parent) {
if (rcu_seq_done(&sp->srcu_gp_seq, s) && snp != sdp->mynode)
return; /* GP already done and CBs recorded. */
- spin_lock_irqsave(&snp->lock, flags);
+ raw_spin_lock_irqsave_rcu_node(snp, flags);
if (ULONG_CMP_GE(snp->srcu_have_cbs[idx], s)) {
snp_seq = snp->srcu_have_cbs[idx];
if (snp == sdp->mynode && snp_seq == s)
snp->srcu_data_have_cbs[idx] |= sdp->grpmask;
- spin_unlock_irqrestore(&snp->lock, flags);
+ raw_spin_unlock_irqrestore_rcu_node(snp, flags);
if (snp == sdp->mynode && snp_seq != s) {
- smp_mb(); /* CBs after GP! */
srcu_schedule_cbs_sdp(sdp, do_norm
? SRCU_INTERVAL
: 0);
snp->srcu_data_have_cbs[idx] |= sdp->grpmask;
if (!do_norm && ULONG_CMP_LT(snp->srcu_gp_seq_needed_exp, s))
snp->srcu_gp_seq_needed_exp = s;
- spin_unlock_irqrestore(&snp->lock, flags);
+ raw_spin_unlock_irqrestore_rcu_node(snp, flags);
}
/* Top of tree, must ensure the grace period will be started. */
- spin_lock_irqsave(&sp->gp_lock, flags);
+ raw_spin_lock_irqsave_rcu_node(sp, flags);
if (ULONG_CMP_LT(sp->srcu_gp_seq_needed, s)) {
/*
* Record need for grace period s. Pair with load
queue_delayed_work(system_power_efficient_wq, &sp->work,
srcu_get_delay(sp));
}
- spin_unlock_irqrestore(&sp->gp_lock, flags);
+ raw_spin_unlock_irqrestore_rcu_node(sp, flags);
}
/*
*/
static void srcu_flip(struct srcu_struct *sp)
{
+ /*
+ * Ensure that if this updater saw a given reader's increment
+ * from __srcu_read_lock(), that reader was using an old value
+ * of ->srcu_idx. Also ensure that if a given reader sees the
+ * new value of ->srcu_idx, this updater's earlier scans cannot
+ * have seen that reader's increments (which is OK, because this
+ * grace period need not wait on that reader).
+ */
+ smp_mb(); /* E */ /* Pairs with B and C. */
+
WRITE_ONCE(sp->srcu_idx, sp->srcu_idx + 1);
/*
return true; /* With reasonable probability, idle! */
}
+/*
+ * SRCU callback function to leak a callback.
+ */
+static void srcu_leak_callback(struct rcu_head *rhp)
+{
+}
+
/*
* Enqueue an SRCU callback on the srcu_data structure associated with
* the current CPU and the specified srcu_struct structure, initiating
struct srcu_data *sdp;
check_init_srcu_struct(sp);
+ if (debug_rcu_head_queue(rhp)) {
+ /* Probable double call_srcu(), so leak the callback. */
+ WRITE_ONCE(rhp->func, srcu_leak_callback);
+ WARN_ONCE(1, "call_srcu(): Leaked duplicate callback\n");
+ return;
+ }
rhp->func = func;
local_irq_save(flags);
sdp = this_cpu_ptr(sp->sda);
- spin_lock(&sdp->lock);
+ raw_spin_lock_rcu_node(sdp);
rcu_segcblist_enqueue(&sdp->srcu_cblist, rhp, false);
rcu_segcblist_advance(&sdp->srcu_cblist,
rcu_seq_current(&sp->srcu_gp_seq));
sdp->srcu_gp_seq_needed_exp = s;
needexp = true;
}
- spin_unlock_irqrestore(&sdp->lock, flags);
+ raw_spin_unlock_irqrestore_rcu_node(sdp, flags);
if (needgp)
srcu_funnel_gp_start(sp, sdp, s, do_norm);
else if (needexp)
srcu_funnel_exp_start(sp, sdp->mynode, s);
}
+/**
+ * call_srcu() - Queue a callback for invocation after an SRCU grace period
+ * @sp: srcu_struct in queue the callback
+ * @head: structure to be used for queueing the SRCU callback.
+ * @func: function to be invoked after the SRCU grace period
+ *
+ * The callback function will be invoked some time after a full SRCU
+ * grace period elapses, in other words after all pre-existing SRCU
+ * read-side critical sections have completed. However, the callback
+ * function might well execute concurrently with other SRCU read-side
+ * critical sections that started after call_srcu() was invoked. SRCU
+ * read-side critical sections are delimited by srcu_read_lock() and
+ * srcu_read_unlock(), and may be nested.
+ *
+ * The callback will be invoked from process context, but must nevertheless
+ * be fast and must not block.
+ */
void call_srcu(struct srcu_struct *sp, struct rcu_head *rhp,
rcu_callback_t func)
{
*/
for_each_possible_cpu(cpu) {
sdp = per_cpu_ptr(sp->sda, cpu);
- spin_lock_irq(&sdp->lock);
+ raw_spin_lock_irq_rcu_node(sdp);
atomic_inc(&sp->srcu_barrier_cpu_cnt);
sdp->srcu_barrier_head.func = srcu_barrier_cb;
+ debug_rcu_head_queue(&sdp->srcu_barrier_head);
if (!rcu_segcblist_entrain(&sdp->srcu_cblist,
- &sdp->srcu_barrier_head, 0))
+ &sdp->srcu_barrier_head, 0)) {
+ debug_rcu_head_unqueue(&sdp->srcu_barrier_head);
atomic_dec(&sp->srcu_barrier_cpu_cnt);
- spin_unlock_irq(&sdp->lock);
+ }
+ raw_spin_unlock_irq_rcu_node(sdp);
}
/* Remove the initial count, at which point reaching zero can happen. */
*/
idx = rcu_seq_state(smp_load_acquire(&sp->srcu_gp_seq)); /* ^^^ */
if (idx == SRCU_STATE_IDLE) {
- spin_lock_irq(&sp->gp_lock);
+ raw_spin_lock_irq_rcu_node(sp);
if (ULONG_CMP_GE(sp->srcu_gp_seq, sp->srcu_gp_seq_needed)) {
WARN_ON_ONCE(rcu_seq_state(sp->srcu_gp_seq));
- spin_unlock_irq(&sp->gp_lock);
+ raw_spin_unlock_irq_rcu_node(sp);
mutex_unlock(&sp->srcu_gp_mutex);
return;
}
idx = rcu_seq_state(READ_ONCE(sp->srcu_gp_seq));
if (idx == SRCU_STATE_IDLE)
srcu_gp_start(sp);
- spin_unlock_irq(&sp->gp_lock);
+ raw_spin_unlock_irq_rcu_node(sp);
if (idx != SRCU_STATE_IDLE) {
mutex_unlock(&sp->srcu_gp_mutex);
return; /* Someone else started the grace period. */
sdp = container_of(work, struct srcu_data, work.work);
sp = sdp->sp;
rcu_cblist_init(&ready_cbs);
- spin_lock_irq(&sdp->lock);
- smp_mb(); /* Old grace periods before callback invocation! */
+ raw_spin_lock_irq_rcu_node(sdp);
rcu_segcblist_advance(&sdp->srcu_cblist,
rcu_seq_current(&sp->srcu_gp_seq));
if (sdp->srcu_cblist_invoking ||
!rcu_segcblist_ready_cbs(&sdp->srcu_cblist)) {
- spin_unlock_irq(&sdp->lock);
+ raw_spin_unlock_irq_rcu_node(sdp);
return; /* Someone else on the job or nothing to do. */
}
/* We are on the job! Extract and invoke ready callbacks. */
sdp->srcu_cblist_invoking = true;
rcu_segcblist_extract_done_cbs(&sdp->srcu_cblist, &ready_cbs);
- spin_unlock_irq(&sdp->lock);
+ raw_spin_unlock_irq_rcu_node(sdp);
rhp = rcu_cblist_dequeue(&ready_cbs);
for (; rhp != NULL; rhp = rcu_cblist_dequeue(&ready_cbs)) {
+ debug_rcu_head_unqueue(rhp);
local_bh_disable();
rhp->func(rhp);
local_bh_enable();
* Update counts, accelerate new callbacks, and if needed,
* schedule another round of callback invocation.
*/
- spin_lock_irq(&sdp->lock);
+ raw_spin_lock_irq_rcu_node(sdp);
rcu_segcblist_insert_count(&sdp->srcu_cblist, &ready_cbs);
(void)rcu_segcblist_accelerate(&sdp->srcu_cblist,
rcu_seq_snap(&sp->srcu_gp_seq));
sdp->srcu_cblist_invoking = false;
more = rcu_segcblist_ready_cbs(&sdp->srcu_cblist);
- spin_unlock_irq(&sdp->lock);
+ raw_spin_unlock_irq_rcu_node(sdp);
if (more)
srcu_schedule_cbs_sdp(sdp, 0);
}
{
bool pushgp = true;
- spin_lock_irq(&sp->gp_lock);
+ raw_spin_lock_irq_rcu_node(sp);
if (ULONG_CMP_GE(sp->srcu_gp_seq, sp->srcu_gp_seq_needed)) {
if (!WARN_ON_ONCE(rcu_seq_state(sp->srcu_gp_seq))) {
/* All requests fulfilled, time to go idle. */
/* Outstanding request and no GP. Start one. */
srcu_gp_start(sp);
}
- spin_unlock_irq(&sp->gp_lock);
+ raw_spin_unlock_irq_rcu_node(sp);
if (pushgp)
queue_delayed_work(system_power_efficient_wq, &sp->work, delay);
*gpnum = rcu_seq_ctr(sp->srcu_gp_seq_needed);
}
EXPORT_SYMBOL_GPL(srcutorture_get_gp_data);
+
+static int __init srcu_bootup_announce(void)
+{
+ pr_info("Hierarchical SRCU implementation.\n");
+ if (exp_holdoff != DEFAULT_SRCU_EXP_HOLDOFF)
+ pr_info("\tNon-default auto-expedite holdoff of %lu ns.\n", exp_holdoff);
+ return 0;
+}
+early_initcall(srcu_bootup_announce);
#include <linux/time.h>
#include <linux/cpu.h>
#include <linux/prefetch.h>
-#include <linux/trace_events.h>
#include "rcu.h"
-/* Forward declarations for tiny_plugin.h. */
-struct rcu_ctrlblk;
-static void __call_rcu(struct rcu_head *head,
- rcu_callback_t func,
- struct rcu_ctrlblk *rcp);
+/* Global control variables for rcupdate callback mechanism. */
+struct rcu_ctrlblk {
+ struct rcu_head *rcucblist; /* List of pending callbacks (CBs). */
+ struct rcu_head **donetail; /* ->next pointer of last "done" CB. */
+ struct rcu_head **curtail; /* ->next pointer of last CB. */
+};
+
+/* Definition for rcupdate control block. */
+static struct rcu_ctrlblk rcu_sched_ctrlblk = {
+ .donetail = &rcu_sched_ctrlblk.rcucblist,
+ .curtail = &rcu_sched_ctrlblk.rcucblist,
+};
+
+static struct rcu_ctrlblk rcu_bh_ctrlblk = {
+ .donetail = &rcu_bh_ctrlblk.rcucblist,
+ .curtail = &rcu_bh_ctrlblk.rcucblist,
+};
#include "tiny_plugin.h"
}
EXPORT_SYMBOL(rcu_barrier_sched);
-#if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE)
-
-/*
- * Test whether RCU thinks that the current CPU is idle.
- */
-bool notrace __rcu_is_watching(void)
-{
- return true;
-}
-EXPORT_SYMBOL(__rcu_is_watching);
-
-#endif /* defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE) */
-
/*
* Helper function for rcu_sched_qs() and rcu_bh_qs().
* Also irqs are disabled to avoid confusion due to interrupt handlers
*/
static int rcu_qsctr_help(struct rcu_ctrlblk *rcp)
{
- RCU_TRACE(reset_cpu_stall_ticks(rcp);)
if (rcp->donetail != rcp->curtail) {
rcp->donetail = rcp->curtail;
return 1;
*/
void rcu_check_callbacks(int user)
{
- RCU_TRACE(check_cpu_stalls();)
if (user)
rcu_sched_qs();
else if (!in_softirq())
*/
static void __rcu_process_callbacks(struct rcu_ctrlblk *rcp)
{
- const char *rn = NULL;
struct rcu_head *next, *list;
unsigned long flags;
- RCU_TRACE(int cb_count = 0;)
/* Move the ready-to-invoke callbacks to a local list. */
local_irq_save(flags);
local_irq_restore(flags);
return;
}
- RCU_TRACE(trace_rcu_batch_start(rcp->name, 0, rcp->qlen, -1);)
list = rcp->rcucblist;
rcp->rcucblist = *rcp->donetail;
*rcp->donetail = NULL;
local_irq_restore(flags);
/* Invoke the callbacks on the local list. */
- RCU_TRACE(rn = rcp->name;)
while (list) {
next = list->next;
prefetch(next);
debug_rcu_head_unqueue(list);
local_bh_disable();
- __rcu_reclaim(rn, list);
+ __rcu_reclaim("", list);
local_bh_enable();
list = next;
- RCU_TRACE(cb_count++;)
}
- RCU_TRACE(rcu_trace_sub_qlen(rcp, cb_count);)
- RCU_TRACE(trace_rcu_batch_end(rcp->name,
- cb_count, 0, need_resched(),
- is_idle_task(current),
- false));
}
static __latent_entropy void rcu_process_callbacks(struct softirq_action *unused)
local_irq_save(flags);
*rcp->curtail = head;
rcp->curtail = &head->next;
- RCU_TRACE(rcp->qlen++;)
local_irq_restore(flags);
if (unlikely(is_idle_task(current))) {
void __init rcu_init(void)
{
open_softirq(RCU_SOFTIRQ, rcu_process_callbacks);
- RCU_TRACE(reset_cpu_stall_ticks(&rcu_sched_ctrlblk);)
- RCU_TRACE(reset_cpu_stall_ticks(&rcu_bh_ctrlblk);)
-
rcu_early_boot_tests();
}
* Author: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
*/
-#include <linux/kthread.h>
-#include <linux/init.h>
-#include <linux/debugfs.h>
-#include <linux/seq_file.h>
-
-/* Global control variables for rcupdate callback mechanism. */
-struct rcu_ctrlblk {
- struct rcu_head *rcucblist; /* List of pending callbacks (CBs). */
- struct rcu_head **donetail; /* ->next pointer of last "done" CB. */
- struct rcu_head **curtail; /* ->next pointer of last CB. */
- RCU_TRACE(long qlen); /* Number of pending CBs. */
- RCU_TRACE(unsigned long gp_start); /* Start time for stalls. */
- RCU_TRACE(unsigned long ticks_this_gp); /* Statistic for stalls. */
- RCU_TRACE(unsigned long jiffies_stall); /* Jiffies at next stall. */
- RCU_TRACE(const char *name); /* Name of RCU type. */
-};
-
-/* Definition for rcupdate control block. */
-static struct rcu_ctrlblk rcu_sched_ctrlblk = {
- .donetail = &rcu_sched_ctrlblk.rcucblist,
- .curtail = &rcu_sched_ctrlblk.rcucblist,
- RCU_TRACE(.name = "rcu_sched")
-};
-
-static struct rcu_ctrlblk rcu_bh_ctrlblk = {
- .donetail = &rcu_bh_ctrlblk.rcucblist,
- .curtail = &rcu_bh_ctrlblk.rcucblist,
- RCU_TRACE(.name = "rcu_bh")
-};
-
#if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_SRCU)
#include <linux/kernel_stat.h>
}
#endif /* #if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_SRCU) */
-
-#ifdef CONFIG_RCU_TRACE
-
-static void rcu_trace_sub_qlen(struct rcu_ctrlblk *rcp, int n)
-{
- unsigned long flags;
-
- local_irq_save(flags);
- rcp->qlen -= n;
- local_irq_restore(flags);
-}
-
-/*
- * Dump statistics for TINY_RCU, such as they are.
- */
-static int show_tiny_stats(struct seq_file *m, void *unused)
-{
- seq_printf(m, "rcu_sched: qlen: %ld\n", rcu_sched_ctrlblk.qlen);
- seq_printf(m, "rcu_bh: qlen: %ld\n", rcu_bh_ctrlblk.qlen);
- return 0;
-}
-
-static int show_tiny_stats_open(struct inode *inode, struct file *file)
-{
- return single_open(file, show_tiny_stats, NULL);
-}
-
-static const struct file_operations show_tiny_stats_fops = {
- .owner = THIS_MODULE,
- .open = show_tiny_stats_open,
- .read = seq_read,
- .llseek = seq_lseek,
- .release = single_release,
-};
-
-static struct dentry *rcudir;
-
-static int __init rcutiny_trace_init(void)
-{
- struct dentry *retval;
-
- rcudir = debugfs_create_dir("rcu", NULL);
- if (!rcudir)
- goto free_out;
- retval = debugfs_create_file("rcudata", 0444, rcudir,
- NULL, &show_tiny_stats_fops);
- if (!retval)
- goto free_out;
- return 0;
-free_out:
- debugfs_remove_recursive(rcudir);
- return 1;
-}
-device_initcall(rcutiny_trace_init);
-
-static void check_cpu_stall(struct rcu_ctrlblk *rcp)
-{
- unsigned long j;
- unsigned long js;
-
- if (rcu_cpu_stall_suppress)
- return;
- rcp->ticks_this_gp++;
- j = jiffies;
- js = READ_ONCE(rcp->jiffies_stall);
- if (rcp->rcucblist && ULONG_CMP_GE(j, js)) {
- pr_err("INFO: %s stall on CPU (%lu ticks this GP) idle=%llx (t=%lu jiffies q=%ld)\n",
- rcp->name, rcp->ticks_this_gp, DYNTICK_TASK_EXIT_IDLE,
- jiffies - rcp->gp_start, rcp->qlen);
- dump_stack();
- WRITE_ONCE(rcp->jiffies_stall,
- jiffies + 3 * rcu_jiffies_till_stall_check() + 3);
- } else if (ULONG_CMP_GE(j, js)) {
- WRITE_ONCE(rcp->jiffies_stall,
- jiffies + rcu_jiffies_till_stall_check());
- }
-}
-
-static void reset_cpu_stall_ticks(struct rcu_ctrlblk *rcp)
-{
- rcp->ticks_this_gp = 0;
- rcp->gp_start = jiffies;
- WRITE_ONCE(rcp->jiffies_stall,
- jiffies + rcu_jiffies_till_stall_check());
-}
-
-static void check_cpu_stalls(void)
-{
- RCU_TRACE(check_cpu_stall(&rcu_bh_ctrlblk);)
- RCU_TRACE(check_cpu_stall(&rcu_sched_ctrlblk);)
-}
-
-#endif /* #ifdef CONFIG_RCU_TRACE */
static void sync_sched_exp_online_cleanup(int cpu);
/* rcuc/rcub kthread realtime priority */
-#ifdef CONFIG_RCU_KTHREAD_PRIO
-static int kthread_prio = CONFIG_RCU_KTHREAD_PRIO;
-#else /* #ifdef CONFIG_RCU_KTHREAD_PRIO */
static int kthread_prio = IS_ENABLED(CONFIG_RCU_BOOST) ? 1 : 0;
-#endif /* #else #ifdef CONFIG_RCU_KTHREAD_PRIO */
module_param(kthread_prio, int, 0644);
/* Delay in jiffies for grace-period initialization delays, debug only. */
-#ifdef CONFIG_RCU_TORTURE_TEST_SLOW_PREINIT
-static int gp_preinit_delay = CONFIG_RCU_TORTURE_TEST_SLOW_PREINIT_DELAY;
-module_param(gp_preinit_delay, int, 0644);
-#else /* #ifdef CONFIG_RCU_TORTURE_TEST_SLOW_PREINIT */
-static const int gp_preinit_delay;
-#endif /* #else #ifdef CONFIG_RCU_TORTURE_TEST_SLOW_PREINIT */
-
-#ifdef CONFIG_RCU_TORTURE_TEST_SLOW_INIT
-static int gp_init_delay = CONFIG_RCU_TORTURE_TEST_SLOW_INIT_DELAY;
-module_param(gp_init_delay, int, 0644);
-#else /* #ifdef CONFIG_RCU_TORTURE_TEST_SLOW_INIT */
-static const int gp_init_delay;
-#endif /* #else #ifdef CONFIG_RCU_TORTURE_TEST_SLOW_INIT */
-
-#ifdef CONFIG_RCU_TORTURE_TEST_SLOW_CLEANUP
-static int gp_cleanup_delay = CONFIG_RCU_TORTURE_TEST_SLOW_CLEANUP_DELAY;
-module_param(gp_cleanup_delay, int, 0644);
-#else /* #ifdef CONFIG_RCU_TORTURE_TEST_SLOW_CLEANUP */
-static const int gp_cleanup_delay;
-#endif /* #else #ifdef CONFIG_RCU_TORTURE_TEST_SLOW_CLEANUP */
+static int gp_preinit_delay;
+module_param(gp_preinit_delay, int, 0444);
+static int gp_init_delay;
+module_param(gp_init_delay, int, 0444);
+static int gp_cleanup_delay;
+module_param(gp_cleanup_delay, int, 0444);
/*
* Number of grace periods between delays, normalized by the duration of
*/
void rcu_sched_qs(void)
{
+ RCU_LOCKDEP_WARN(preemptible(), "rcu_sched_qs() invoked with preemption enabled!!!");
if (!__this_cpu_read(rcu_sched_data.cpu_no_qs.s))
return;
trace_rcu_grace_period(TPS("rcu_sched"),
void rcu_bh_qs(void)
{
+ RCU_LOCKDEP_WARN(preemptible(), "rcu_bh_qs() invoked with preemption enabled!!!");
if (__this_cpu_read(rcu_bh_data.cpu_no_qs.s)) {
trace_rcu_grace_period(TPS("rcu_bh"),
__this_cpu_read(rcu_bh_data.gpnum),
static DEFINE_PER_CPU(struct rcu_dynticks, rcu_dynticks) = {
.dynticks_nesting = DYNTICK_TASK_EXIT_IDLE,
.dynticks = ATOMIC_INIT(RCU_DYNTICK_CTRL_CTR),
-#ifdef CONFIG_NO_HZ_FULL_SYSIDLE
- .dynticks_idle_nesting = DYNTICK_TASK_NEST_VALUE,
- .dynticks_idle = ATOMIC_INIT(1),
-#endif /* #ifdef CONFIG_NO_HZ_FULL_SYSIDLE */
};
/*
barrier(); /* Avoid RCU read-side critical sections leaking down. */
trace_rcu_utilization(TPS("Start context switch"));
rcu_sched_qs();
- rcu_preempt_note_context_switch();
+ rcu_preempt_note_context_switch(preempt);
/* Load rcu_urgent_qs before other flags. */
if (!smp_load_acquire(this_cpu_ptr(&rcu_dynticks.rcu_urgent_qs)))
goto out;
}
EXPORT_SYMBOL_GPL(rcu_all_qs);
-static long blimit = 10; /* Maximum callbacks per rcu_do_batch. */
-static long qhimark = 10000; /* If this many pending, ignore blimit. */
-static long qlowmark = 100; /* Once only this many pending, use blimit. */
+#define DEFAULT_RCU_BLIMIT 10 /* Maximum callbacks per rcu_do_batch. */
+static long blimit = DEFAULT_RCU_BLIMIT;
+#define DEFAULT_RCU_QHIMARK 10000 /* If this many pending, ignore blimit. */
+static long qhimark = DEFAULT_RCU_QHIMARK;
+#define DEFAULT_RCU_QLOMARK 100 /* Once only this many pending, use blimit. */
+static long qlowmark = DEFAULT_RCU_QLOMARK;
module_param(blimit, long, 0444);
module_param(qhimark, long, 0444);
static bool rcu_start_gp_advanced(struct rcu_state *rsp, struct rcu_node *rnp,
struct rcu_data *rdp);
-static void force_qs_rnp(struct rcu_state *rsp,
- int (*f)(struct rcu_data *rsp, bool *isidle,
- unsigned long *maxj),
- bool *isidle, unsigned long *maxj);
+static void force_qs_rnp(struct rcu_state *rsp, int (*f)(struct rcu_data *rsp));
static void force_quiescent_state(struct rcu_state *rsp);
static int rcu_pending(void);
int idx = (READ_ONCE(rnp->completed) + 1) & 0x1;
int *fp = &rnp->need_future_gp[idx];
+ RCU_LOCKDEP_WARN(!irqs_disabled(), "rcu_future_needs_gp() invoked with irqs enabled!!!");
return READ_ONCE(*fp);
}
static bool
cpu_needs_another_gp(struct rcu_state *rsp, struct rcu_data *rdp)
{
+ RCU_LOCKDEP_WARN(!irqs_disabled(), "cpu_needs_another_gp() invoked with irqs enabled!!!");
if (rcu_gp_in_progress(rsp))
return false; /* No, a grace period is already in progress. */
if (rcu_future_needs_gp(rsp))
struct rcu_data *rdp;
struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
+ RCU_LOCKDEP_WARN(!irqs_disabled(), "rcu_eqs_enter_common() invoked with irqs enabled!!!");
trace_rcu_dyntick(TPS("Start"), rdtp->dynticks_nesting, 0);
if (IS_ENABLED(CONFIG_RCU_EQS_DEBUG) &&
!user && !is_idle_task(current)) {
local_irq_save(flags);
rcu_eqs_enter(false);
- rcu_sysidle_enter(0);
local_irq_restore(flags);
}
EXPORT_SYMBOL_GPL(rcu_idle_enter);
trace_rcu_dyntick(TPS("--="), rdtp->dynticks_nesting, rdtp->dynticks_nesting - 1);
rdtp->dynticks_nesting--;
}
- rcu_sysidle_enter(1);
}
/*
struct rcu_dynticks *rdtp;
long long oldval;
+ RCU_LOCKDEP_WARN(!irqs_disabled(), "rcu_eqs_exit() invoked with irqs enabled!!!");
rdtp = this_cpu_ptr(&rcu_dynticks);
oldval = rdtp->dynticks_nesting;
WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && oldval < 0);
local_irq_save(flags);
rcu_eqs_exit(false);
- rcu_sysidle_exit(0);
local_irq_restore(flags);
}
EXPORT_SYMBOL_GPL(rcu_idle_exit);
trace_rcu_dyntick(TPS("++="), oldval, rdtp->dynticks_nesting);
else
rcu_eqs_exit_common(oldval, true);
- rcu_sysidle_exit(1);
}
/*
rcu_dynticks_eqs_enter();
}
-/**
- * __rcu_is_watching - are RCU read-side critical sections safe?
- *
- * Return true if RCU is watching the running CPU, which means that
- * this CPU can safely enter RCU read-side critical sections. Unlike
- * rcu_is_watching(), the caller of __rcu_is_watching() must have at
- * least disabled preemption.
- */
-bool notrace __rcu_is_watching(void)
-{
- return !rcu_dynticks_curr_cpu_in_eqs();
-}
-
/**
* rcu_is_watching - see if RCU thinks that the current CPU is idle
*
- * If the current CPU is in its idle loop and is neither in an interrupt
+ * Return true if RCU is watching the running CPU, which means that this
+ * CPU can safely enter RCU read-side critical sections. In other words,
+ * if the current CPU is in its idle loop and is neither in an interrupt
* or NMI handler, return true.
*/
bool notrace rcu_is_watching(void)
bool ret;
preempt_disable_notrace();
- ret = __rcu_is_watching();
+ ret = !rcu_dynticks_curr_cpu_in_eqs();
preempt_enable_notrace();
return ret;
}
* credit them with an implicit quiescent state. Return 1 if this CPU
* is in dynticks idle mode, which is an extended quiescent state.
*/
-static int dyntick_save_progress_counter(struct rcu_data *rdp,
- bool *isidle, unsigned long *maxj)
+static int dyntick_save_progress_counter(struct rcu_data *rdp)
{
rdp->dynticks_snap = rcu_dynticks_snap(rdp->dynticks);
- rcu_sysidle_check_cpu(rdp, isidle, maxj);
if (rcu_dynticks_in_eqs(rdp->dynticks_snap)) {
trace_rcu_fqs(rdp->rsp->name, rdp->gpnum, rdp->cpu, TPS("dti"));
if (ULONG_CMP_LT(READ_ONCE(rdp->gpnum) + ULONG_MAX / 4,
* idle state since the last call to dyntick_save_progress_counter()
* for this same CPU, or by virtue of having been offline.
*/
-static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
- bool *isidle, unsigned long *maxj)
+static int rcu_implicit_dynticks_qs(struct rcu_data *rdp)
{
unsigned long jtsq;
bool *rnhqp;
static unsigned long rcu_cbs_completed(struct rcu_state *rsp,
struct rcu_node *rnp)
{
+ lockdep_assert_held(&rnp->lock);
+
/*
* If RCU is idle, we just wait for the next grace period.
* But we can only be sure that RCU is idle if we are looking
bool ret = false;
struct rcu_node *rnp_root = rcu_get_root(rdp->rsp);
+ lockdep_assert_held(&rnp->lock);
+
/*
* Pick up grace-period number for new callbacks. If this
* grace period is already marked as needed, return to the caller.
{
bool ret = false;
+ lockdep_assert_held(&rnp->lock);
+
/* If no pending (not yet ready to invoke) callbacks, nothing to do. */
if (!rcu_segcblist_pend_cbs(&rdp->cblist))
return false;
static bool rcu_advance_cbs(struct rcu_state *rsp, struct rcu_node *rnp,
struct rcu_data *rdp)
{
+ lockdep_assert_held(&rnp->lock);
+
/* If no pending (not yet ready to invoke) callbacks, nothing to do. */
if (!rcu_segcblist_pend_cbs(&rdp->cblist))
return false;
bool ret;
bool need_gp;
+ lockdep_assert_held(&rnp->lock);
+
/* Handle the ends of any preceding grace periods first. */
if (rdp->completed == rnp->completed &&
!unlikely(READ_ONCE(rdp->gpwrap))) {
*/
static void rcu_gp_fqs(struct rcu_state *rsp, bool first_time)
{
- bool isidle = false;
- unsigned long maxj;
struct rcu_node *rnp = rcu_get_root(rsp);
WRITE_ONCE(rsp->gp_activity, jiffies);
rsp->n_force_qs++;
if (first_time) {
/* Collect dyntick-idle snapshots. */
- if (is_sysidle_rcu_state(rsp)) {
- isidle = true;
- maxj = jiffies - ULONG_MAX / 4;
- }
- force_qs_rnp(rsp, dyntick_save_progress_counter,
- &isidle, &maxj);
- rcu_sysidle_report_gp(rsp, isidle, maxj);
+ force_qs_rnp(rsp, dyntick_save_progress_counter);
} else {
/* Handle dyntick-idle and offline CPUs. */
- isidle = true;
- force_qs_rnp(rsp, rcu_implicit_dynticks_qs, &isidle, &maxj);
+ force_qs_rnp(rsp, rcu_implicit_dynticks_qs);
}
/* Clear flag to prevent immediate re-entry. */
if (READ_ONCE(rsp->gp_flags) & RCU_GP_FLAG_FQS) {
rcu_start_gp_advanced(struct rcu_state *rsp, struct rcu_node *rnp,
struct rcu_data *rdp)
{
+ lockdep_assert_held(&rnp->lock);
if (!rsp->gp_kthread || !cpu_needs_another_gp(rsp, rdp)) {
/*
* Either we have not yet spawned the grace-period
static void rcu_report_qs_rsp(struct rcu_state *rsp, unsigned long flags)
__releases(rcu_get_root(rsp)->lock)
{
+ lockdep_assert_held(&rcu_get_root(rsp)->lock);
WARN_ON_ONCE(!rcu_gp_in_progress(rsp));
WRITE_ONCE(rsp->gp_flags, READ_ONCE(rsp->gp_flags) | RCU_GP_FLAG_FQS);
raw_spin_unlock_irqrestore_rcu_node(rcu_get_root(rsp), flags);
unsigned long oldmask = 0;
struct rcu_node *rnp_c;
+ lockdep_assert_held(&rnp->lock);
+
/* Walk up the rcu_node hierarchy. */
for (;;) {
if (!(rnp->qsmask & mask) || rnp->gpnum != gps) {
unsigned long mask;
struct rcu_node *rnp_p;
+ lockdep_assert_held(&rnp->lock);
if (rcu_state_p == &rcu_sched_state || rsp != rcu_state_p ||
rnp->qsmask != 0 || rcu_preempt_blocked_readers_cgp(rnp)) {
raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
rcu_send_cbs_to_orphanage(int cpu, struct rcu_state *rsp,
struct rcu_node *rnp, struct rcu_data *rdp)
{
+ lockdep_assert_held(&rsp->orphan_lock);
+
/* No-CBs CPUs do not have orphanable callbacks. */
if (!IS_ENABLED(CONFIG_HOTPLUG_CPU) || rcu_is_nocb_cpu(rdp->cpu))
return;
{
struct rcu_data *rdp = raw_cpu_ptr(rsp->rda);
+ lockdep_assert_held(&rsp->orphan_lock);
+
/* No-CBs CPUs are handled specially. */
if (!IS_ENABLED(CONFIG_HOTPLUG_CPU) ||
rcu_nocb_adopt_orphan_cbs(rsp, rdp, flags))
long mask;
struct rcu_node *rnp = rnp_leaf;
+ lockdep_assert_held(&rnp->lock);
if (!IS_ENABLED(CONFIG_HOTPLUG_CPU) ||
rnp->qsmaskinit || rcu_preempt_has_tasks(rnp))
return;
*
* The caller must have suppressed start of new grace periods.
*/
-static void force_qs_rnp(struct rcu_state *rsp,
- int (*f)(struct rcu_data *rsp, bool *isidle,
- unsigned long *maxj),
- bool *isidle, unsigned long *maxj)
+static void force_qs_rnp(struct rcu_state *rsp, int (*f)(struct rcu_data *rsp))
{
int cpu;
unsigned long flags;
for_each_leaf_node_possible_cpu(rnp, cpu) {
unsigned long bit = leaf_node_cpu_bit(rnp, cpu);
if ((rnp->qsmask & bit) != 0) {
- if (f(per_cpu_ptr(rsp->rda, cpu), isidle, maxj))
+ if (f(per_cpu_ptr(rsp->rda, cpu)))
mask |= bit;
}
}
WARN_ON_ONCE((unsigned long)head & (sizeof(void *) - 1));
if (debug_rcu_head_queue(head)) {
- /* Probable double call_rcu(), so leak the callback. */
+ /*
+ * Probable double call_rcu(), so leak the callback.
+ * Use rcu:rcu_callback trace event to find the previous
+ * time callback was passed to __call_rcu().
+ */
+ WARN_ONCE(1, "__call_rcu(): Double-freed CB %p->%pF()!!!\n",
+ head, head->func);
WRITE_ONCE(head->func, rcu_leak_callback);
- WARN_ONCE(1, "__call_rcu(): Leaked duplicate callback\n");
return;
}
head->func = func;
local_irq_restore(flags);
}
-/*
- * Queue an RCU-sched callback for invocation after a grace period.
+/**
+ * call_rcu_sched() - Queue an RCU for invocation after sched grace period.
+ * @head: structure to be used for queueing the RCU updates.
+ * @func: actual callback function to be invoked after the grace period
+ *
+ * The callback function will be invoked some time after a full grace
+ * period elapses, in other words after all currently executing RCU
+ * read-side critical sections have completed. call_rcu_sched() assumes
+ * that the read-side critical sections end on enabling of preemption
+ * or on voluntary preemption.
+ * RCU read-side critical sections are delimited by :
+ * - rcu_read_lock_sched() and rcu_read_unlock_sched(), OR
+ * - anything that disables preemption.
+ *
+ * These may be nested.
+ *
+ * See the description of call_rcu() for more detailed information on
+ * memory ordering guarantees.
*/
void call_rcu_sched(struct rcu_head *head, rcu_callback_t func)
{
}
EXPORT_SYMBOL_GPL(call_rcu_sched);
-/*
- * Queue an RCU callback for invocation after a quicker grace period.
+/**
+ * call_rcu_bh() - Queue an RCU for invocation after a quicker grace period.
+ * @head: structure to be used for queueing the RCU updates.
+ * @func: actual callback function to be invoked after the grace period
+ *
+ * The callback function will be invoked some time after a full grace
+ * period elapses, in other words after all currently executing RCU
+ * read-side critical sections have completed. call_rcu_bh() assumes
+ * that the read-side critical sections end on completion of a softirq
+ * handler. This means that read-side critical sections in process
+ * context must not be interrupted by softirqs. This interface is to be
+ * used when most of the read-side critical sections are in softirq context.
+ * RCU read-side critical sections are delimited by :
+ * - rcu_read_lock() and rcu_read_unlock(), if in interrupt context.
+ * OR
+ * - rcu_read_lock_bh() and rcu_read_unlock_bh(), if in process context.
+ * These may be nested.
+ *
+ * See the description of call_rcu() for more detailed information on
+ * memory ordering guarantees.
*/
void call_rcu_bh(struct rcu_head *head, rcu_callback_t func)
{
* to have executed a full memory barrier during the execution of
* synchronize_sched() -- even if CPU A and CPU B are the same CPU (but
* again only if the system has more than one CPU).
- *
- * This primitive provides the guarantees made by the (now removed)
- * synchronize_kernel() API. In contrast, synchronize_rcu() only
- * guarantees that rcu_read_lock() sections will have completed.
- * In "classic RCU", these two guarantees happen to be one and
- * the same, but can differ in realtime RCU implementations.
*/
void synchronize_sched(void)
{
struct rcu_data *rdp = raw_cpu_ptr(rsp->rda);
_rcu_barrier_trace(rsp, "IRQ", -1, rsp->barrier_sequence);
- atomic_inc(&rsp->barrier_cpu_count);
- rsp->call(&rdp->barrier_head, rcu_barrier_callback);
+ rdp->barrier_head.func = rcu_barrier_callback;
+ debug_rcu_head_queue(&rdp->barrier_head);
+ if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head, 0)) {
+ atomic_inc(&rsp->barrier_cpu_count);
+ } else {
+ debug_rcu_head_unqueue(&rdp->barrier_head);
+ _rcu_barrier_trace(rsp, "IRQNQ", -1, rsp->barrier_sequence);
+ }
}
/*
long mask;
struct rcu_node *rnp = rnp_leaf;
+ lockdep_assert_held(&rnp->lock);
for (;;) {
mask = rnp->grpmask;
rnp = rnp->parent;
!init_nocb_callback_list(rdp))
rcu_segcblist_init(&rdp->cblist); /* Re-enable callbacks. */
rdp->dynticks->dynticks_nesting = DYNTICK_TASK_EXIT_IDLE;
- rcu_sysidle_init_percpu_data(rdp->dynticks);
rcu_dynticks_eqs_online();
raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */
bool rcu_need_heavy_qs; /* GP old, need heavy quiescent state. */
unsigned long rcu_qs_ctr; /* Light universal quiescent state ctr. */
bool rcu_urgent_qs; /* GP old need light quiescent state. */
-#ifdef CONFIG_NO_HZ_FULL_SYSIDLE
- long long dynticks_idle_nesting;
- /* irq/process nesting level from idle. */
- atomic_t dynticks_idle; /* Even value for idle, else odd. */
- /* "Idle" excludes userspace execution. */
- unsigned long dynticks_idle_jiffies;
- /* End of last non-NMI non-idle period. */
-#endif /* #ifdef CONFIG_NO_HZ_FULL_SYSIDLE */
#ifdef CONFIG_RCU_FAST_NO_HZ
bool all_lazy; /* Are all CPU's CBs lazy? */
unsigned long nonlazy_posted;
/* Number of tasks boosted for expedited GP. */
unsigned long n_normal_boosts;
/* Number of tasks boosted for normal GP. */
- unsigned long n_balk_blkd_tasks;
- /* Refused to boost: no blocked tasks. */
- unsigned long n_balk_exp_gp_tasks;
- /* Refused to boost: nothing blocking GP. */
- unsigned long n_balk_boost_tasks;
- /* Refused to boost: already boosting. */
- unsigned long n_balk_notblocked;
- /* Refused to boost: RCU RS CS still running. */
- unsigned long n_balk_notyet;
- /* Refused to boost: not yet time. */
- unsigned long n_balk_nos;
- /* Refused to boost: not sure why, though. */
- /* This can happen due to race conditions. */
#ifdef CONFIG_RCU_NOCB_CPU
struct swait_queue_head nocb_gp_wq[2];
/* Place for rcu_nocb_kthread() to wait GP. */
};
/* Values for nocb_defer_wakeup field in struct rcu_data. */
-#define RCU_NOGP_WAKE_NOT 0
-#define RCU_NOGP_WAKE 1
-#define RCU_NOGP_WAKE_FORCE 2
+#define RCU_NOCB_WAKE_NOT 0
+#define RCU_NOCB_WAKE 1
+#define RCU_NOCB_WAKE_FORCE 2
#define RCU_JIFFIES_TILL_FORCE_QS (1 + (HZ > 250) + (HZ > 500))
/* For jiffies_till_first_fqs and */
/* Forward declarations for rcutree_plugin.h */
static void rcu_bootup_announce(void);
-static void rcu_preempt_note_context_switch(void);
+static void rcu_preempt_note_context_switch(bool preempt);
static int rcu_preempt_blocked_readers_cgp(struct rcu_node *rnp);
#ifdef CONFIG_HOTPLUG_CPU
static bool rcu_preempt_has_tasks(struct rcu_node *rnp);
#endif /* #ifdef CONFIG_RCU_NOCB_CPU */
static void __maybe_unused rcu_kick_nohz_cpu(int cpu);
static bool init_nocb_callback_list(struct rcu_data *rdp);
-static void rcu_sysidle_enter(int irq);
-static void rcu_sysidle_exit(int irq);
-static void rcu_sysidle_check_cpu(struct rcu_data *rdp, bool *isidle,
- unsigned long *maxj);
-static bool is_sysidle_rcu_state(struct rcu_state *rsp);
-static void rcu_sysidle_report_gp(struct rcu_state *rsp, int isidle,
- unsigned long maxj);
static void rcu_bind_gp_kthread(void);
-static void rcu_sysidle_init_percpu_data(struct rcu_dynticks *rdtp);
static bool rcu_nohz_full_cpu(struct rcu_state *rsp);
static void rcu_dynticks_task_enter(void);
static void rcu_dynticks_task_exit(void);
#endif /* #else #ifdef CONFIG_SRCU */
#endif /* #ifndef RCU_TREE_NONCORE */
-
-#ifdef CONFIG_RCU_TRACE
-/* Read out queue lengths for tracing. */
-static inline void rcu_nocb_q_lengths(struct rcu_data *rdp, long *ql, long *qll)
-{
-#ifdef CONFIG_RCU_NOCB_CPU
- *ql = atomic_long_read(&rdp->nocb_q_count);
- *qll = atomic_long_read(&rdp->nocb_q_count_lazy);
-#else /* #ifdef CONFIG_RCU_NOCB_CPU */
- *ql = 0;
- *qll = 0;
-#endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */
-}
-#endif /* #ifdef CONFIG_RCU_TRACE */
-
-/*
- * Wrappers for the rcu_node::lock acquire and release.
- *
- * Because the rcu_nodes form a tree, the tree traversal locking will observe
- * different lock values, this in turn means that an UNLOCK of one level
- * followed by a LOCK of another level does not imply a full memory barrier;
- * and most importantly transitivity is lost.
- *
- * In order to restore full ordering between tree levels, augment the regular
- * lock acquire functions with smp_mb__after_unlock_lock().
- *
- * As ->lock of struct rcu_node is a __private field, therefore one should use
- * these wrappers rather than directly call raw_spin_{lock,unlock}* on ->lock.
- */
-static inline void raw_spin_lock_rcu_node(struct rcu_node *rnp)
-{
- raw_spin_lock(&ACCESS_PRIVATE(rnp, lock));
- smp_mb__after_unlock_lock();
-}
-
-static inline void raw_spin_unlock_rcu_node(struct rcu_node *rnp)
-{
- raw_spin_unlock(&ACCESS_PRIVATE(rnp, lock));
-}
-
-static inline void raw_spin_lock_irq_rcu_node(struct rcu_node *rnp)
-{
- raw_spin_lock_irq(&ACCESS_PRIVATE(rnp, lock));
- smp_mb__after_unlock_lock();
-}
-
-static inline void raw_spin_unlock_irq_rcu_node(struct rcu_node *rnp)
-{
- raw_spin_unlock_irq(&ACCESS_PRIVATE(rnp, lock));
-}
-
-#define raw_spin_lock_irqsave_rcu_node(rnp, flags) \
-do { \
- typecheck(unsigned long, flags); \
- raw_spin_lock_irqsave(&ACCESS_PRIVATE(rnp, lock), flags); \
- smp_mb__after_unlock_lock(); \
-} while (0)
-
-#define raw_spin_unlock_irqrestore_rcu_node(rnp, flags) \
-do { \
- typecheck(unsigned long, flags); \
- raw_spin_unlock_irqrestore(&ACCESS_PRIVATE(rnp, lock), flags); \
-} while (0)
-
-static inline bool raw_spin_trylock_rcu_node(struct rcu_node *rnp)
-{
- bool locked = raw_spin_trylock(&ACCESS_PRIVATE(rnp, lock));
-
- if (locked)
- smp_mb__after_unlock_lock();
- return locked;
-}
*
* Caller must hold the rcu_state's exp_mutex.
*/
-static int sync_rcu_preempt_exp_done(struct rcu_node *rnp)
+static bool sync_rcu_preempt_exp_done(struct rcu_node *rnp)
{
return rnp->exp_tasks == NULL &&
READ_ONCE(rnp->expmask) == 0;
static void __init rcu_bootup_announce_oddness(void)
{
if (IS_ENABLED(CONFIG_RCU_TRACE))
- pr_info("\tRCU debugfs-based tracing is enabled.\n");
+ pr_info("\tRCU event tracing is enabled.\n");
if ((IS_ENABLED(CONFIG_64BIT) && RCU_FANOUT != 64) ||
(!IS_ENABLED(CONFIG_64BIT) && RCU_FANOUT != 32))
pr_info("\tCONFIG_RCU_FANOUT set to non-default value of %d\n",
pr_info("\tBoot-time adjustment of leaf fanout to %d.\n", rcu_fanout_leaf);
if (nr_cpu_ids != NR_CPUS)
pr_info("\tRCU restricting CPUs from NR_CPUS=%d to nr_cpu_ids=%d.\n", NR_CPUS, nr_cpu_ids);
- if (IS_ENABLED(CONFIG_RCU_BOOST))
- pr_info("\tRCU kthread priority: %d.\n", kthread_prio);
+#ifdef CONFIG_RCU_BOOST
+ pr_info("\tRCU priority boosting: priority %d delay %d ms.\n", kthread_prio, CONFIG_RCU_BOOST_DELAY);
+#endif
+ if (blimit != DEFAULT_RCU_BLIMIT)
+ pr_info("\tBoot-time adjustment of callback invocation limit to %ld.\n", blimit);
+ if (qhimark != DEFAULT_RCU_QHIMARK)
+ pr_info("\tBoot-time adjustment of callback high-water mark to %ld.\n", qhimark);
+ if (qlowmark != DEFAULT_RCU_QLOMARK)
+ pr_info("\tBoot-time adjustment of callback low-water mark to %ld.\n", qlowmark);
+ if (jiffies_till_first_fqs != ULONG_MAX)
+ pr_info("\tBoot-time adjustment of first FQS scan delay to %ld jiffies.\n", jiffies_till_first_fqs);
+ if (jiffies_till_next_fqs != ULONG_MAX)
+ pr_info("\tBoot-time adjustment of subsequent FQS scan delay to %ld jiffies.\n", jiffies_till_next_fqs);
+ if (rcu_kick_kthreads)
+ pr_info("\tKick kthreads if too-long grace period.\n");
+ if (IS_ENABLED(CONFIG_DEBUG_OBJECTS_RCU_HEAD))
+ pr_info("\tRCU callback double-/use-after-free debug enabled.\n");
+ if (gp_preinit_delay)
+ pr_info("\tRCU debug GP pre-init slowdown %d jiffies.\n", gp_preinit_delay);
+ if (gp_init_delay)
+ pr_info("\tRCU debug GP init slowdown %d jiffies.\n", gp_init_delay);
+ if (gp_cleanup_delay)
+ pr_info("\tRCU debug GP init slowdown %d jiffies.\n", gp_cleanup_delay);
+ if (IS_ENABLED(CONFIG_RCU_EQS_DEBUG))
+ pr_info("\tRCU debug extended QS entry/exit.\n");
+ rcupdate_announce_bootup_oddness();
}
#ifdef CONFIG_PREEMPT_RCU
(rnp->expmask & rdp->grpmask ? RCU_EXP_BLKD : 0);
struct task_struct *t = current;
+ lockdep_assert_held(&rnp->lock);
+
/*
* Decide where to queue the newly blocked task. In theory,
* this could be an if-statement. In practice, when I tried
*/
static void rcu_preempt_qs(void)
{
+ RCU_LOCKDEP_WARN(preemptible(), "rcu_preempt_qs() invoked with preemption enabled!!!\n");
if (__this_cpu_read(rcu_data_p->cpu_no_qs.s)) {
trace_rcu_grace_period(TPS("rcu_preempt"),
__this_cpu_read(rcu_data_p->gpnum),
*
* Caller must disable interrupts.
*/
-static void rcu_preempt_note_context_switch(void)
+static void rcu_preempt_note_context_switch(bool preempt)
{
struct task_struct *t = current;
struct rcu_data *rdp;
struct rcu_node *rnp;
+ RCU_LOCKDEP_WARN(!irqs_disabled(), "rcu_preempt_note_context_switch() invoked with interrupts enabled!!!\n");
+ WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0);
if (t->rcu_read_lock_nesting > 0 &&
!t->rcu_read_unlock_special.b.blocked) {
*/
static void rcu_preempt_check_blocked_tasks(struct rcu_node *rnp)
{
+ RCU_LOCKDEP_WARN(preemptible(), "rcu_preempt_check_blocked_tasks() invoked with preemption enabled!!!\n");
WARN_ON_ONCE(rcu_preempt_blocked_readers_cgp(rnp));
if (rcu_preempt_has_tasks(rnp))
rnp->gp_tasks = rnp->blkd_tasks.next;
#endif /* #ifdef CONFIG_RCU_BOOST */
-/*
- * Queue a preemptible-RCU callback for invocation after a grace period.
+/**
+ * call_rcu() - Queue an RCU callback for invocation after a grace period.
+ * @head: structure to be used for queueing the RCU updates.
+ * @func: actual callback function to be invoked after the grace period
+ *
+ * The callback function will be invoked some time after a full grace
+ * period elapses, in other words after all pre-existing RCU read-side
+ * critical sections have completed. However, the callback function
+ * might well execute concurrently with RCU read-side critical sections
+ * that started after call_rcu() was invoked. RCU read-side critical
+ * sections are delimited by rcu_read_lock() and rcu_read_unlock(),
+ * and may be nested.
+ *
+ * Note that all CPUs must agree that the grace period extended beyond
+ * all pre-existing RCU read-side critical section. On systems with more
+ * than one CPU, this means that when "func()" is invoked, each CPU is
+ * guaranteed to have executed a full memory barrier since the end of its
+ * last RCU read-side critical section whose beginning preceded the call
+ * to call_rcu(). It also means that each CPU executing an RCU read-side
+ * critical section that continues beyond the start of "func()" must have
+ * executed a memory barrier after the call_rcu() but before the beginning
+ * of that RCU read-side critical section. Note that these guarantees
+ * include CPUs that are offline, idle, or executing in user mode, as
+ * well as CPUs that are executing in the kernel.
+ *
+ * Furthermore, if CPU A invoked call_rcu() and CPU B invoked the
+ * resulting RCU callback function "func()", then both CPU A and CPU B are
+ * guaranteed to execute a full memory barrier during the time interval
+ * between the call to call_rcu() and the invocation of "func()" -- even
+ * if CPU A and CPU B are the same CPU (but again only if the system has
+ * more than one CPU).
*/
void call_rcu(struct rcu_head *head, rcu_callback_t func)
{
* synchronize_rcu() was waiting. RCU read-side critical sections are
* delimited by rcu_read_lock() and rcu_read_unlock(), and may be nested.
*
- * See the description of synchronize_sched() for more detailed information
- * on memory ordering guarantees.
+ * See the description of synchronize_sched() for more detailed
+ * information on memory-ordering guarantees. However, please note
+ * that -only- the memory-ordering guarantees apply. For example,
+ * synchronize_rcu() is -not- guaranteed to wait on things like code
+ * protected by preempt_disable(), instead, synchronize_rcu() is -only-
+ * guaranteed to wait on RCU read-side critical sections, that is, sections
+ * of code protected by rcu_read_lock().
*/
void synchronize_rcu(void)
{
* Because preemptible RCU does not exist, we never have to check for
* CPUs being in quiescent states.
*/
-static void rcu_preempt_note_context_switch(void)
+static void rcu_preempt_note_context_switch(bool preempt)
{
}
#include "../locking/rtmutex_common.h"
-#ifdef CONFIG_RCU_TRACE
-
-static void rcu_initiate_boost_trace(struct rcu_node *rnp)
-{
- if (!rcu_preempt_has_tasks(rnp))
- rnp->n_balk_blkd_tasks++;
- else if (rnp->exp_tasks == NULL && rnp->gp_tasks == NULL)
- rnp->n_balk_exp_gp_tasks++;
- else if (rnp->gp_tasks != NULL && rnp->boost_tasks != NULL)
- rnp->n_balk_boost_tasks++;
- else if (rnp->gp_tasks != NULL && rnp->qsmask != 0)
- rnp->n_balk_notblocked++;
- else if (rnp->gp_tasks != NULL &&
- ULONG_CMP_LT(jiffies, rnp->boost_time))
- rnp->n_balk_notyet++;
- else
- rnp->n_balk_nos++;
-}
-
-#else /* #ifdef CONFIG_RCU_TRACE */
-
-static void rcu_initiate_boost_trace(struct rcu_node *rnp)
-{
-}
-
-#endif /* #else #ifdef CONFIG_RCU_TRACE */
-
static void rcu_wake_cond(struct task_struct *t, int status)
{
/*
{
struct task_struct *t;
+ lockdep_assert_held(&rnp->lock);
if (!rcu_preempt_blocked_readers_cgp(rnp) && rnp->exp_tasks == NULL) {
- rnp->n_balk_exp_gp_tasks++;
raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
return;
}
if (t)
rcu_wake_cond(t, rnp->boost_kthread_status);
} else {
- rcu_initiate_boost_trace(rnp);
raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
}
}
int rcu_needs_cpu(u64 basemono, u64 *nextevt)
{
*nextevt = KTIME_MAX;
- return IS_ENABLED(CONFIG_RCU_NOCB_CPU_ALL)
- ? 0 : rcu_cpu_has_callbacks(NULL);
+ return rcu_cpu_has_callbacks(NULL);
}
/*
struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
unsigned long dj;
- if (IS_ENABLED(CONFIG_RCU_NOCB_CPU_ALL)) {
- *nextevt = KTIME_MAX;
- return 0;
- }
+ RCU_LOCKDEP_WARN(!irqs_disabled(), "rcu_needs_cpu() invoked with irqs enabled!!!");
/* Snapshot to detect later posting of non-lazy callback. */
rdtp->nonlazy_posted_snap = rdtp->nonlazy_posted;
struct rcu_state *rsp;
int tne;
- if (IS_ENABLED(CONFIG_RCU_NOCB_CPU_ALL) ||
- rcu_is_nocb_cpu(smp_processor_id()))
+ RCU_LOCKDEP_WARN(!irqs_disabled(), "rcu_prepare_for_idle() invoked with irqs enabled!!!");
+ if (rcu_is_nocb_cpu(smp_processor_id()))
return;
/* Handle nohz enablement switches conservatively. */
*/
static void rcu_cleanup_after_idle(void)
{
- if (IS_ENABLED(CONFIG_RCU_NOCB_CPU_ALL) ||
- rcu_is_nocb_cpu(smp_processor_id()))
+ RCU_LOCKDEP_WARN(!irqs_disabled(), "rcu_cleanup_after_idle() invoked with irqs enabled!!!");
+ if (rcu_is_nocb_cpu(smp_processor_id()))
return;
if (rcu_try_advance_all_cbs())
invoke_rcu_core();
init_swait_queue_head(&rnp->nocb_gp_wq[1]);
}
-#ifndef CONFIG_RCU_NOCB_CPU_ALL
/* Is the specified CPU a no-CBs CPU? */
bool rcu_is_nocb_cpu(int cpu)
{
return cpumask_test_cpu(cpu, rcu_nocb_mask);
return false;
}
-#endif /* #ifndef CONFIG_RCU_NOCB_CPU_ALL */
/*
* Kick the leader kthread for this NOCB group.
if (READ_ONCE(rdp_leader->nocb_leader_sleep) || force) {
/* Prior smp_mb__after_atomic() orders against prior enqueue. */
WRITE_ONCE(rdp_leader->nocb_leader_sleep, false);
+ smp_mb(); /* ->nocb_leader_sleep before swake_up(). */
swake_up(&rdp_leader->nocb_wq);
}
}
trace_rcu_nocb_wake(rdp->rsp->name, rdp->cpu,
TPS("WakeEmpty"));
} else {
- WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOGP_WAKE);
+ WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOCB_WAKE);
/* Store ->nocb_defer_wakeup before ->rcu_urgent_qs. */
smp_store_release(this_cpu_ptr(&rcu_dynticks.rcu_urgent_qs), true);
trace_rcu_nocb_wake(rdp->rsp->name, rdp->cpu,
trace_rcu_nocb_wake(rdp->rsp->name, rdp->cpu,
TPS("WakeOvf"));
} else {
- WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOGP_WAKE_FORCE);
+ WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOCB_WAKE_FORCE);
/* Store ->nocb_defer_wakeup before ->rcu_urgent_qs. */
smp_store_release(this_cpu_ptr(&rcu_dynticks.rcu_urgent_qs), true);
trace_rcu_nocb_wake(rdp->rsp->name, rdp->cpu,
* nocb_gp_head, where they await a grace period.
*/
gotcbs = false;
+ smp_mb(); /* wakeup before ->nocb_head reads. */
for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_follower) {
rdp->nocb_gp_head = READ_ONCE(rdp->nocb_head);
if (!rdp->nocb_gp_head)
if (!rcu_nocb_need_deferred_wakeup(rdp))
return;
ndw = READ_ONCE(rdp->nocb_defer_wakeup);
- WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOGP_WAKE_NOT);
- wake_nocb_leader(rdp, ndw == RCU_NOGP_WAKE_FORCE);
+ WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOCB_WAKE_NOT);
+ wake_nocb_leader(rdp, ndw == RCU_NOCB_WAKE_FORCE);
trace_rcu_nocb_wake(rdp->rsp->name, rdp->cpu, TPS("DeferredWake"));
}
bool need_rcu_nocb_mask = true;
struct rcu_state *rsp;
-#ifdef CONFIG_RCU_NOCB_CPU_NONE
- need_rcu_nocb_mask = false;
-#endif /* #ifndef CONFIG_RCU_NOCB_CPU_NONE */
-
#if defined(CONFIG_NO_HZ_FULL)
if (tick_nohz_full_running && cpumask_weight(tick_nohz_full_mask))
need_rcu_nocb_mask = true;
if (!have_rcu_nocb_mask)
return;
-#ifdef CONFIG_RCU_NOCB_CPU_ZERO
- pr_info("\tOffload RCU callbacks from CPU 0\n");
- cpumask_set_cpu(0, rcu_nocb_mask);
-#endif /* #ifdef CONFIG_RCU_NOCB_CPU_ZERO */
-#ifdef CONFIG_RCU_NOCB_CPU_ALL
- pr_info("\tOffload RCU callbacks from all CPUs\n");
- cpumask_copy(rcu_nocb_mask, cpu_possible_mask);
-#endif /* #ifdef CONFIG_RCU_NOCB_CPU_ALL */
#if defined(CONFIG_NO_HZ_FULL)
if (tick_nohz_full_running)
cpumask_or(rcu_nocb_mask, rcu_nocb_mask, tick_nohz_full_mask);
#endif /* #ifdef CONFIG_NO_HZ_FULL */
}
-
-#ifdef CONFIG_NO_HZ_FULL_SYSIDLE
-
-static int full_sysidle_state; /* Current system-idle state. */
-#define RCU_SYSIDLE_NOT 0 /* Some CPU is not idle. */
-#define RCU_SYSIDLE_SHORT 1 /* All CPUs idle for brief period. */
-#define RCU_SYSIDLE_LONG 2 /* All CPUs idle for long enough. */
-#define RCU_SYSIDLE_FULL 3 /* All CPUs idle, ready for sysidle. */
-#define RCU_SYSIDLE_FULL_NOTED 4 /* Actually entered sysidle state. */
-
-/*
- * Invoked to note exit from irq or task transition to idle. Note that
- * usermode execution does -not- count as idle here! After all, we want
- * to detect full-system idle states, not RCU quiescent states and grace
- * periods. The caller must have disabled interrupts.
- */
-static void rcu_sysidle_enter(int irq)
-{
- unsigned long j;
- struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
-
- /* If there are no nohz_full= CPUs, no need to track this. */
- if (!tick_nohz_full_enabled())
- return;
-
- /* Adjust nesting, check for fully idle. */
- if (irq) {
- rdtp->dynticks_idle_nesting--;
- WARN_ON_ONCE(rdtp->dynticks_idle_nesting < 0);
- if (rdtp->dynticks_idle_nesting != 0)
- return; /* Still not fully idle. */
- } else {
- if ((rdtp->dynticks_idle_nesting & DYNTICK_TASK_NEST_MASK) ==
- DYNTICK_TASK_NEST_VALUE) {
- rdtp->dynticks_idle_nesting = 0;
- } else {
- rdtp->dynticks_idle_nesting -= DYNTICK_TASK_NEST_VALUE;
- WARN_ON_ONCE(rdtp->dynticks_idle_nesting < 0);
- return; /* Still not fully idle. */
- }
- }
-
- /* Record start of fully idle period. */
- j = jiffies;
- WRITE_ONCE(rdtp->dynticks_idle_jiffies, j);
- smp_mb__before_atomic();
- atomic_inc(&rdtp->dynticks_idle);
- smp_mb__after_atomic();
- WARN_ON_ONCE(atomic_read(&rdtp->dynticks_idle) & 0x1);
-}
-
-/*
- * Unconditionally force exit from full system-idle state. This is
- * invoked when a normal CPU exits idle, but must be called separately
- * for the timekeeping CPU (tick_do_timer_cpu). The reason for this
- * is that the timekeeping CPU is permitted to take scheduling-clock
- * interrupts while the system is in system-idle state, and of course
- * rcu_sysidle_exit() has no way of distinguishing a scheduling-clock
- * interrupt from any other type of interrupt.
- */
-void rcu_sysidle_force_exit(void)
-{
- int oldstate = READ_ONCE(full_sysidle_state);
- int newoldstate;
-
- /*
- * Each pass through the following loop attempts to exit full
- * system-idle state. If contention proves to be a problem,
- * a trylock-based contention tree could be used here.
- */
- while (oldstate > RCU_SYSIDLE_SHORT) {
- newoldstate = cmpxchg(&full_sysidle_state,
- oldstate, RCU_SYSIDLE_NOT);
- if (oldstate == newoldstate &&
- oldstate == RCU_SYSIDLE_FULL_NOTED) {
- rcu_kick_nohz_cpu(tick_do_timer_cpu);
- return; /* We cleared it, done! */
- }
- oldstate = newoldstate;
- }
- smp_mb(); /* Order initial oldstate fetch vs. later non-idle work. */
-}
-
-/*
- * Invoked to note entry to irq or task transition from idle. Note that
- * usermode execution does -not- count as idle here! The caller must
- * have disabled interrupts.
- */
-static void rcu_sysidle_exit(int irq)
-{
- struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
-
- /* If there are no nohz_full= CPUs, no need to track this. */
- if (!tick_nohz_full_enabled())
- return;
-
- /* Adjust nesting, check for already non-idle. */
- if (irq) {
- rdtp->dynticks_idle_nesting++;
- WARN_ON_ONCE(rdtp->dynticks_idle_nesting <= 0);
- if (rdtp->dynticks_idle_nesting != 1)
- return; /* Already non-idle. */
- } else {
- /*
- * Allow for irq misnesting. Yes, it really is possible
- * to enter an irq handler then never leave it, and maybe
- * also vice versa. Handle both possibilities.
- */
- if (rdtp->dynticks_idle_nesting & DYNTICK_TASK_NEST_MASK) {
- rdtp->dynticks_idle_nesting += DYNTICK_TASK_NEST_VALUE;
- WARN_ON_ONCE(rdtp->dynticks_idle_nesting <= 0);
- return; /* Already non-idle. */
- } else {
- rdtp->dynticks_idle_nesting = DYNTICK_TASK_EXIT_IDLE;
- }
- }
-
- /* Record end of idle period. */
- smp_mb__before_atomic();
- atomic_inc(&rdtp->dynticks_idle);
- smp_mb__after_atomic();
- WARN_ON_ONCE(!(atomic_read(&rdtp->dynticks_idle) & 0x1));
-
- /*
- * If we are the timekeeping CPU, we are permitted to be non-idle
- * during a system-idle state. This must be the case, because
- * the timekeeping CPU has to take scheduling-clock interrupts
- * during the time that the system is transitioning to full
- * system-idle state. This means that the timekeeping CPU must
- * invoke rcu_sysidle_force_exit() directly if it does anything
- * more than take a scheduling-clock interrupt.
- */
- if (smp_processor_id() == tick_do_timer_cpu)
- return;
-
- /* Update system-idle state: We are clearly no longer fully idle! */
- rcu_sysidle_force_exit();
-}
-
-/*
- * Check to see if the current CPU is idle. Note that usermode execution
- * does not count as idle. The caller must have disabled interrupts,
- * and must be running on tick_do_timer_cpu.
- */
-static void rcu_sysidle_check_cpu(struct rcu_data *rdp, bool *isidle,
- unsigned long *maxj)
-{
- int cur;
- unsigned long j;
- struct rcu_dynticks *rdtp = rdp->dynticks;
-
- /* If there are no nohz_full= CPUs, don't check system-wide idleness. */
- if (!tick_nohz_full_enabled())
- return;
-
- /*
- * If some other CPU has already reported non-idle, if this is
- * not the flavor of RCU that tracks sysidle state, or if this
- * is an offline or the timekeeping CPU, nothing to do.
- */
- if (!*isidle || rdp->rsp != rcu_state_p ||
- cpu_is_offline(rdp->cpu) || rdp->cpu == tick_do_timer_cpu)
- return;
- /* Verify affinity of current kthread. */
- WARN_ON_ONCE(smp_processor_id() != tick_do_timer_cpu);
-
- /* Pick up current idle and NMI-nesting counter and check. */
- cur = atomic_read(&rdtp->dynticks_idle);
- if (cur & 0x1) {
- *isidle = false; /* We are not idle! */
- return;
- }
- smp_mb(); /* Read counters before timestamps. */
-
- /* Pick up timestamps. */
- j = READ_ONCE(rdtp->dynticks_idle_jiffies);
- /* If this CPU entered idle more recently, update maxj timestamp. */
- if (ULONG_CMP_LT(*maxj, j))
- *maxj = j;
-}
-
-/*
- * Is this the flavor of RCU that is handling full-system idle?
- */
-static bool is_sysidle_rcu_state(struct rcu_state *rsp)
-{
- return rsp == rcu_state_p;
-}
-
-/*
- * Return a delay in jiffies based on the number of CPUs, rcu_node
- * leaf fanout, and jiffies tick rate. The idea is to allow larger
- * systems more time to transition to full-idle state in order to
- * avoid the cache thrashing that otherwise occur on the state variable.
- * Really small systems (less than a couple of tens of CPUs) should
- * instead use a single global atomically incremented counter, and later
- * versions of this will automatically reconfigure themselves accordingly.
- */
-static unsigned long rcu_sysidle_delay(void)
-{
- if (nr_cpu_ids <= CONFIG_NO_HZ_FULL_SYSIDLE_SMALL)
- return 0;
- return DIV_ROUND_UP(nr_cpu_ids * HZ, rcu_fanout_leaf * 1000);
-}
-
-/*
- * Advance the full-system-idle state. This is invoked when all of
- * the non-timekeeping CPUs are idle.
- */
-static void rcu_sysidle(unsigned long j)
-{
- /* Check the current state. */
- switch (READ_ONCE(full_sysidle_state)) {
- case RCU_SYSIDLE_NOT:
-
- /* First time all are idle, so note a short idle period. */
- WRITE_ONCE(full_sysidle_state, RCU_SYSIDLE_SHORT);
- break;
-
- case RCU_SYSIDLE_SHORT:
-
- /*
- * Idle for a bit, time to advance to next state?
- * cmpxchg failure means race with non-idle, let them win.
- */
- if (ULONG_CMP_GE(jiffies, j + rcu_sysidle_delay()))
- (void)cmpxchg(&full_sysidle_state,
- RCU_SYSIDLE_SHORT, RCU_SYSIDLE_LONG);
- break;
-
- case RCU_SYSIDLE_LONG:
-
- /*
- * Do an additional check pass before advancing to full.
- * cmpxchg failure means race with non-idle, let them win.
- */
- if (ULONG_CMP_GE(jiffies, j + rcu_sysidle_delay()))
- (void)cmpxchg(&full_sysidle_state,
- RCU_SYSIDLE_LONG, RCU_SYSIDLE_FULL);
- break;
-
- default:
- break;
- }
-}
-
-/*
- * Found a non-idle non-timekeeping CPU, so kick the system-idle state
- * back to the beginning.
- */
-static void rcu_sysidle_cancel(void)
-{
- smp_mb();
- if (full_sysidle_state > RCU_SYSIDLE_SHORT)
- WRITE_ONCE(full_sysidle_state, RCU_SYSIDLE_NOT);
-}
-
-/*
- * Update the sysidle state based on the results of a force-quiescent-state
- * scan of the CPUs' dyntick-idle state.
- */
-static void rcu_sysidle_report(struct rcu_state *rsp, int isidle,
- unsigned long maxj, bool gpkt)
-{
- if (rsp != rcu_state_p)
- return; /* Wrong flavor, ignore. */
- if (gpkt && nr_cpu_ids <= CONFIG_NO_HZ_FULL_SYSIDLE_SMALL)
- return; /* Running state machine from timekeeping CPU. */
- if (isidle)
- rcu_sysidle(maxj); /* More idle! */
- else
- rcu_sysidle_cancel(); /* Idle is over. */
-}
-
-/*
- * Wrapper for rcu_sysidle_report() when called from the grace-period
- * kthread's context.
- */
-static void rcu_sysidle_report_gp(struct rcu_state *rsp, int isidle,
- unsigned long maxj)
-{
- /* If there are no nohz_full= CPUs, no need to track this. */
- if (!tick_nohz_full_enabled())
- return;
-
- rcu_sysidle_report(rsp, isidle, maxj, true);
-}
-
-/* Callback and function for forcing an RCU grace period. */
-struct rcu_sysidle_head {
- struct rcu_head rh;
- int inuse;
-};
-
-static void rcu_sysidle_cb(struct rcu_head *rhp)
-{
- struct rcu_sysidle_head *rshp;
-
- /*
- * The following memory barrier is needed to replace the
- * memory barriers that would normally be in the memory
- * allocator.
- */
- smp_mb(); /* grace period precedes setting inuse. */
-
- rshp = container_of(rhp, struct rcu_sysidle_head, rh);
- WRITE_ONCE(rshp->inuse, 0);
-}
-
-/*
- * Check to see if the system is fully idle, other than the timekeeping CPU.
- * The caller must have disabled interrupts. This is not intended to be
- * called unless tick_nohz_full_enabled().
- */
-bool rcu_sys_is_idle(void)
-{
- static struct rcu_sysidle_head rsh;
- int rss = READ_ONCE(full_sysidle_state);
-
- if (WARN_ON_ONCE(smp_processor_id() != tick_do_timer_cpu))
- return false;
-
- /* Handle small-system case by doing a full scan of CPUs. */
- if (nr_cpu_ids <= CONFIG_NO_HZ_FULL_SYSIDLE_SMALL) {
- int oldrss = rss - 1;
-
- /*
- * One pass to advance to each state up to _FULL.
- * Give up if any pass fails to advance the state.
- */
- while (rss < RCU_SYSIDLE_FULL && oldrss < rss) {
- int cpu;
- bool isidle = true;
- unsigned long maxj = jiffies - ULONG_MAX / 4;
- struct rcu_data *rdp;
-
- /* Scan all the CPUs looking for nonidle CPUs. */
- for_each_possible_cpu(cpu) {
- rdp = per_cpu_ptr(rcu_state_p->rda, cpu);
- rcu_sysidle_check_cpu(rdp, &isidle, &maxj);
- if (!isidle)
- break;
- }
- rcu_sysidle_report(rcu_state_p, isidle, maxj, false);
- oldrss = rss;
- rss = READ_ONCE(full_sysidle_state);
- }
- }
-
- /* If this is the first observation of an idle period, record it. */
- if (rss == RCU_SYSIDLE_FULL) {
- rss = cmpxchg(&full_sysidle_state,
- RCU_SYSIDLE_FULL, RCU_SYSIDLE_FULL_NOTED);
- return rss == RCU_SYSIDLE_FULL;
- }
-
- smp_mb(); /* ensure rss load happens before later caller actions. */
-
- /* If already fully idle, tell the caller (in case of races). */
- if (rss == RCU_SYSIDLE_FULL_NOTED)
- return true;
-
- /*
- * If we aren't there yet, and a grace period is not in flight,
- * initiate a grace period. Either way, tell the caller that
- * we are not there yet. We use an xchg() rather than an assignment
- * to make up for the memory barriers that would otherwise be
- * provided by the memory allocator.
- */
- if (nr_cpu_ids > CONFIG_NO_HZ_FULL_SYSIDLE_SMALL &&
- !rcu_gp_in_progress(rcu_state_p) &&
- !rsh.inuse && xchg(&rsh.inuse, 1) == 0)
- call_rcu(&rsh.rh, rcu_sysidle_cb);
- return false;
-}
-
-/*
- * Initialize dynticks sysidle state for CPUs coming online.
- */
-static void rcu_sysidle_init_percpu_data(struct rcu_dynticks *rdtp)
-{
- rdtp->dynticks_idle_nesting = DYNTICK_TASK_NEST_VALUE;
-}
-
-#else /* #ifdef CONFIG_NO_HZ_FULL_SYSIDLE */
-
-static void rcu_sysidle_enter(int irq)
-{
-}
-
-static void rcu_sysidle_exit(int irq)
-{
-}
-
-static void rcu_sysidle_check_cpu(struct rcu_data *rdp, bool *isidle,
- unsigned long *maxj)
-{
-}
-
-static bool is_sysidle_rcu_state(struct rcu_state *rsp)
-{
- return false;
-}
-
-static void rcu_sysidle_report_gp(struct rcu_state *rsp, int isidle,
- unsigned long maxj)
-{
-}
-
-static void rcu_sysidle_init_percpu_data(struct rcu_dynticks *rdtp)
-{
-}
-
-#endif /* #else #ifdef CONFIG_NO_HZ_FULL_SYSIDLE */
-
/*
* Is this CPU a NO_HZ_FULL CPU that should ignore RCU so that the
* grace-period kthread will do force_quiescent_state() processing?
if (!tick_nohz_full_enabled())
return;
-#ifdef CONFIG_NO_HZ_FULL_SYSIDLE
- cpu = tick_do_timer_cpu;
- if (cpu >= 0 && cpu < nr_cpu_ids)
- set_cpus_allowed_ptr(current, cpumask_of(cpu));
-#else /* #ifdef CONFIG_NO_HZ_FULL_SYSIDLE */
housekeeping_affine(current);
-#endif /* #else #ifdef CONFIG_NO_HZ_FULL_SYSIDLE */
}
/* Record the current task on dyntick-idle entry. */
+++ /dev/null
-/*
- * Read-Copy Update tracing for hierarchical implementation.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, you can access it online at
- * http://www.gnu.org/licenses/gpl-2.0.html.
- *
- * Copyright IBM Corporation, 2008
- * Author: Paul E. McKenney
- *
- * Papers: http://www.rdrop.com/users/paulmck/RCU
- *
- * For detailed explanation of Read-Copy Update mechanism see -
- * Documentation/RCU
- *
- */
-#include <linux/types.h>
-#include <linux/kernel.h>
-#include <linux/init.h>
-#include <linux/spinlock.h>
-#include <linux/smp.h>
-#include <linux/rcupdate.h>
-#include <linux/interrupt.h>
-#include <linux/sched.h>
-#include <linux/atomic.h>
-#include <linux/bitops.h>
-#include <linux/completion.h>
-#include <linux/percpu.h>
-#include <linux/notifier.h>
-#include <linux/cpu.h>
-#include <linux/mutex.h>
-#include <linux/debugfs.h>
-#include <linux/seq_file.h>
-#include <linux/prefetch.h>
-
-#define RCU_TREE_NONCORE
-#include "tree.h"
-#include "rcu.h"
-
-static int r_open(struct inode *inode, struct file *file,
- const struct seq_operations *op)
-{
- int ret = seq_open(file, op);
- if (!ret) {
- struct seq_file *m = (struct seq_file *)file->private_data;
- m->private = inode->i_private;
- }
- return ret;
-}
-
-static void *r_start(struct seq_file *m, loff_t *pos)
-{
- struct rcu_state *rsp = (struct rcu_state *)m->private;
- *pos = cpumask_next(*pos - 1, cpu_possible_mask);
- if ((*pos) < nr_cpu_ids)
- return per_cpu_ptr(rsp->rda, *pos);
- return NULL;
-}
-
-static void *r_next(struct seq_file *m, void *v, loff_t *pos)
-{
- (*pos)++;
- return r_start(m, pos);
-}
-
-static void r_stop(struct seq_file *m, void *v)
-{
-}
-
-static int show_rcubarrier(struct seq_file *m, void *v)
-{
- struct rcu_state *rsp = (struct rcu_state *)m->private;
- seq_printf(m, "bcc: %d bseq: %lu\n",
- atomic_read(&rsp->barrier_cpu_count),
- rsp->barrier_sequence);
- return 0;
-}
-
-static int rcubarrier_open(struct inode *inode, struct file *file)
-{
- return single_open(file, show_rcubarrier, inode->i_private);
-}
-
-static const struct file_operations rcubarrier_fops = {
- .owner = THIS_MODULE,
- .open = rcubarrier_open,
- .read = seq_read,
- .llseek = no_llseek,
- .release = single_release,
-};
-
-#ifdef CONFIG_RCU_BOOST
-
-static char convert_kthread_status(unsigned int kthread_status)
-{
- if (kthread_status > RCU_KTHREAD_MAX)
- return '?';
- return "SRWOY"[kthread_status];
-}
-
-#endif /* #ifdef CONFIG_RCU_BOOST */
-
-static void print_one_rcu_data(struct seq_file *m, struct rcu_data *rdp)
-{
- long ql, qll;
-
- if (!rdp->beenonline)
- return;
- seq_printf(m, "%3d%cc=%ld g=%ld cnq=%d/%d:%d",
- rdp->cpu,
- cpu_is_offline(rdp->cpu) ? '!' : ' ',
- ulong2long(rdp->completed), ulong2long(rdp->gpnum),
- rdp->cpu_no_qs.b.norm,
- rdp->rcu_qs_ctr_snap == per_cpu(rdp->dynticks->rcu_qs_ctr, rdp->cpu),
- rdp->core_needs_qs);
- seq_printf(m, " dt=%d/%llx/%d df=%lu",
- rcu_dynticks_snap(rdp->dynticks),
- rdp->dynticks->dynticks_nesting,
- rdp->dynticks->dynticks_nmi_nesting,
- rdp->dynticks_fqs);
- seq_printf(m, " of=%lu", rdp->offline_fqs);
- rcu_nocb_q_lengths(rdp, &ql, &qll);
- qll += rcu_segcblist_n_lazy_cbs(&rdp->cblist);
- ql += rcu_segcblist_n_cbs(&rdp->cblist);
- seq_printf(m, " ql=%ld/%ld qs=%c%c%c%c",
- qll, ql,
- ".N"[!rcu_segcblist_segempty(&rdp->cblist, RCU_NEXT_TAIL)],
- ".R"[!rcu_segcblist_segempty(&rdp->cblist,
- RCU_NEXT_READY_TAIL)],
- ".W"[!rcu_segcblist_segempty(&rdp->cblist, RCU_WAIT_TAIL)],
- ".D"[!rcu_segcblist_segempty(&rdp->cblist, RCU_DONE_TAIL)]);
-#ifdef CONFIG_RCU_BOOST
- seq_printf(m, " kt=%d/%c ktl=%x",
- per_cpu(rcu_cpu_has_work, rdp->cpu),
- convert_kthread_status(per_cpu(rcu_cpu_kthread_status,
- rdp->cpu)),
- per_cpu(rcu_cpu_kthread_loops, rdp->cpu) & 0xffff);
-#endif /* #ifdef CONFIG_RCU_BOOST */
- seq_printf(m, " b=%ld", rdp->blimit);
- seq_printf(m, " ci=%lu nci=%lu co=%lu ca=%lu\n",
- rdp->n_cbs_invoked, rdp->n_nocbs_invoked,
- rdp->n_cbs_orphaned, rdp->n_cbs_adopted);
-}
-
-static int show_rcudata(struct seq_file *m, void *v)
-{
- print_one_rcu_data(m, (struct rcu_data *)v);
- return 0;
-}
-
-static const struct seq_operations rcudate_op = {
- .start = r_start,
- .next = r_next,
- .stop = r_stop,
- .show = show_rcudata,
-};
-
-static int rcudata_open(struct inode *inode, struct file *file)
-{
- return r_open(inode, file, &rcudate_op);
-}
-
-static const struct file_operations rcudata_fops = {
- .owner = THIS_MODULE,
- .open = rcudata_open,
- .read = seq_read,
- .llseek = no_llseek,
- .release = seq_release,
-};
-
-static int show_rcuexp(struct seq_file *m, void *v)
-{
- int cpu;
- struct rcu_state *rsp = (struct rcu_state *)m->private;
- struct rcu_data *rdp;
- unsigned long s0 = 0, s1 = 0, s2 = 0, s3 = 0;
-
- for_each_possible_cpu(cpu) {
- rdp = per_cpu_ptr(rsp->rda, cpu);
- s0 += atomic_long_read(&rdp->exp_workdone0);
- s1 += atomic_long_read(&rdp->exp_workdone1);
- s2 += atomic_long_read(&rdp->exp_workdone2);
- s3 += atomic_long_read(&rdp->exp_workdone3);
- }
- seq_printf(m, "s=%lu wd0=%lu wd1=%lu wd2=%lu wd3=%lu enq=%d sc=%lu\n",
- rsp->expedited_sequence, s0, s1, s2, s3,
- atomic_read(&rsp->expedited_need_qs),
- rsp->expedited_sequence / 2);
- return 0;
-}
-
-static int rcuexp_open(struct inode *inode, struct file *file)
-{
- return single_open(file, show_rcuexp, inode->i_private);
-}
-
-static const struct file_operations rcuexp_fops = {
- .owner = THIS_MODULE,
- .open = rcuexp_open,
- .read = seq_read,
- .llseek = no_llseek,
- .release = single_release,
-};
-
-#ifdef CONFIG_RCU_BOOST
-
-static void print_one_rcu_node_boost(struct seq_file *m, struct rcu_node *rnp)
-{
- seq_printf(m, "%d:%d tasks=%c%c%c%c kt=%c ntb=%lu neb=%lu nnb=%lu ",
- rnp->grplo, rnp->grphi,
- "T."[list_empty(&rnp->blkd_tasks)],
- "N."[!rnp->gp_tasks],
- "E."[!rnp->exp_tasks],
- "B."[!rnp->boost_tasks],
- convert_kthread_status(rnp->boost_kthread_status),
- rnp->n_tasks_boosted, rnp->n_exp_boosts,
- rnp->n_normal_boosts);
- seq_printf(m, "j=%04x bt=%04x\n",
- (int)(jiffies & 0xffff),
- (int)(rnp->boost_time & 0xffff));
- seq_printf(m, " balk: nt=%lu egt=%lu bt=%lu nb=%lu ny=%lu nos=%lu\n",
- rnp->n_balk_blkd_tasks,
- rnp->n_balk_exp_gp_tasks,
- rnp->n_balk_boost_tasks,
- rnp->n_balk_notblocked,
- rnp->n_balk_notyet,
- rnp->n_balk_nos);
-}
-
-static int show_rcu_node_boost(struct seq_file *m, void *unused)
-{
- struct rcu_node *rnp;
-
- rcu_for_each_leaf_node(&rcu_preempt_state, rnp)
- print_one_rcu_node_boost(m, rnp);
- return 0;
-}
-
-static int rcu_node_boost_open(struct inode *inode, struct file *file)
-{
- return single_open(file, show_rcu_node_boost, NULL);
-}
-
-static const struct file_operations rcu_node_boost_fops = {
- .owner = THIS_MODULE,
- .open = rcu_node_boost_open,
- .read = seq_read,
- .llseek = no_llseek,
- .release = single_release,
-};
-
-#endif /* #ifdef CONFIG_RCU_BOOST */
-
-static void print_one_rcu_state(struct seq_file *m, struct rcu_state *rsp)
-{
- unsigned long gpnum;
- int level = 0;
- struct rcu_node *rnp;
-
- gpnum = rsp->gpnum;
- seq_printf(m, "c=%ld g=%ld s=%d jfq=%ld j=%x ",
- ulong2long(rsp->completed), ulong2long(gpnum),
- rsp->gp_state,
- (long)(rsp->jiffies_force_qs - jiffies),
- (int)(jiffies & 0xffff));
- seq_printf(m, "nfqs=%lu/nfqsng=%lu(%lu) fqlh=%lu oqlen=%ld/%ld\n",
- rsp->n_force_qs, rsp->n_force_qs_ngp,
- rsp->n_force_qs - rsp->n_force_qs_ngp,
- READ_ONCE(rsp->n_force_qs_lh),
- rsp->orphan_done.len_lazy,
- rsp->orphan_done.len);
- for (rnp = &rsp->node[0]; rnp - &rsp->node[0] < rcu_num_nodes; rnp++) {
- if (rnp->level != level) {
- seq_puts(m, "\n");
- level = rnp->level;
- }
- seq_printf(m, "%lx/%lx->%lx %c%c>%c %d:%d ^%d ",
- rnp->qsmask, rnp->qsmaskinit, rnp->qsmaskinitnext,
- ".G"[rnp->gp_tasks != NULL],
- ".E"[rnp->exp_tasks != NULL],
- ".T"[!list_empty(&rnp->blkd_tasks)],
- rnp->grplo, rnp->grphi, rnp->grpnum);
- }
- seq_puts(m, "\n");
-}
-
-static int show_rcuhier(struct seq_file *m, void *v)
-{
- struct rcu_state *rsp = (struct rcu_state *)m->private;
- print_one_rcu_state(m, rsp);
- return 0;
-}
-
-static int rcuhier_open(struct inode *inode, struct file *file)
-{
- return single_open(file, show_rcuhier, inode->i_private);
-}
-
-static const struct file_operations rcuhier_fops = {
- .owner = THIS_MODULE,
- .open = rcuhier_open,
- .read = seq_read,
- .llseek = no_llseek,
- .release = single_release,
-};
-
-static void show_one_rcugp(struct seq_file *m, struct rcu_state *rsp)
-{
- unsigned long flags;
- unsigned long completed;
- unsigned long gpnum;
- unsigned long gpage;
- unsigned long gpmax;
- struct rcu_node *rnp = &rsp->node[0];
-
- raw_spin_lock_irqsave_rcu_node(rnp, flags);
- completed = READ_ONCE(rsp->completed);
- gpnum = READ_ONCE(rsp->gpnum);
- if (completed == gpnum)
- gpage = 0;
- else
- gpage = jiffies - rsp->gp_start;
- gpmax = rsp->gp_max;
- raw_spin_unlock_irqrestore(&rnp->lock, flags);
- seq_printf(m, "completed=%ld gpnum=%ld age=%ld max=%ld\n",
- ulong2long(completed), ulong2long(gpnum), gpage, gpmax);
-}
-
-static int show_rcugp(struct seq_file *m, void *v)
-{
- struct rcu_state *rsp = (struct rcu_state *)m->private;
- show_one_rcugp(m, rsp);
- return 0;
-}
-
-static int rcugp_open(struct inode *inode, struct file *file)
-{
- return single_open(file, show_rcugp, inode->i_private);
-}
-
-static const struct file_operations rcugp_fops = {
- .owner = THIS_MODULE,
- .open = rcugp_open,
- .read = seq_read,
- .llseek = no_llseek,
- .release = single_release,
-};
-
-static void print_one_rcu_pending(struct seq_file *m, struct rcu_data *rdp)
-{
- if (!rdp->beenonline)
- return;
- seq_printf(m, "%3d%cnp=%ld ",
- rdp->cpu,
- cpu_is_offline(rdp->cpu) ? '!' : ' ',
- rdp->n_rcu_pending);
- seq_printf(m, "qsp=%ld rpq=%ld cbr=%ld cng=%ld ",
- rdp->n_rp_core_needs_qs,
- rdp->n_rp_report_qs,
- rdp->n_rp_cb_ready,
- rdp->n_rp_cpu_needs_gp);
- seq_printf(m, "gpc=%ld gps=%ld nn=%ld ndw%ld\n",
- rdp->n_rp_gp_completed,
- rdp->n_rp_gp_started,
- rdp->n_rp_nocb_defer_wakeup,
- rdp->n_rp_need_nothing);
-}
-
-static int show_rcu_pending(struct seq_file *m, void *v)
-{
- print_one_rcu_pending(m, (struct rcu_data *)v);
- return 0;
-}
-
-static const struct seq_operations rcu_pending_op = {
- .start = r_start,
- .next = r_next,
- .stop = r_stop,
- .show = show_rcu_pending,
-};
-
-static int rcu_pending_open(struct inode *inode, struct file *file)
-{
- return r_open(inode, file, &rcu_pending_op);
-}
-
-static const struct file_operations rcu_pending_fops = {
- .owner = THIS_MODULE,
- .open = rcu_pending_open,
- .read = seq_read,
- .llseek = no_llseek,
- .release = seq_release,
-};
-
-static int show_rcutorture(struct seq_file *m, void *unused)
-{
- seq_printf(m, "rcutorture test sequence: %lu %s\n",
- rcutorture_testseq >> 1,
- (rcutorture_testseq & 0x1) ? "(test in progress)" : "");
- seq_printf(m, "rcutorture update version number: %lu\n",
- rcutorture_vernum);
- return 0;
-}
-
-static int rcutorture_open(struct inode *inode, struct file *file)
-{
- return single_open(file, show_rcutorture, NULL);
-}
-
-static const struct file_operations rcutorture_fops = {
- .owner = THIS_MODULE,
- .open = rcutorture_open,
- .read = seq_read,
- .llseek = seq_lseek,
- .release = single_release,
-};
-
-static struct dentry *rcudir;
-
-static int __init rcutree_trace_init(void)
-{
- struct rcu_state *rsp;
- struct dentry *retval;
- struct dentry *rspdir;
-
- rcudir = debugfs_create_dir("rcu", NULL);
- if (!rcudir)
- goto free_out;
-
- for_each_rcu_flavor(rsp) {
- rspdir = debugfs_create_dir(rsp->name, rcudir);
- if (!rspdir)
- goto free_out;
-
- retval = debugfs_create_file("rcudata", 0444,
- rspdir, rsp, &rcudata_fops);
- if (!retval)
- goto free_out;
-
- retval = debugfs_create_file("rcuexp", 0444,
- rspdir, rsp, &rcuexp_fops);
- if (!retval)
- goto free_out;
-
- retval = debugfs_create_file("rcu_pending", 0444,
- rspdir, rsp, &rcu_pending_fops);
- if (!retval)
- goto free_out;
-
- retval = debugfs_create_file("rcubarrier", 0444,
- rspdir, rsp, &rcubarrier_fops);
- if (!retval)
- goto free_out;
-
-#ifdef CONFIG_RCU_BOOST
- if (rsp == &rcu_preempt_state) {
- retval = debugfs_create_file("rcuboost", 0444,
- rspdir, NULL, &rcu_node_boost_fops);
- if (!retval)
- goto free_out;
- }
-#endif
-
- retval = debugfs_create_file("rcugp", 0444,
- rspdir, rsp, &rcugp_fops);
- if (!retval)
- goto free_out;
-
- retval = debugfs_create_file("rcuhier", 0444,
- rspdir, rsp, &rcuhier_fops);
- if (!retval)
- goto free_out;
- }
-
- retval = debugfs_create_file("rcutorture", 0444, rcudir,
- NULL, &rcutorture_fops);
- if (!retval)
- goto free_out;
- return 0;
-free_out:
- debugfs_remove_recursive(rcudir);
- return 1;
-}
-device_initcall(rcutree_trace_init);
#define MODULE_PARAM_PREFIX "rcupdate."
#ifndef CONFIG_TINY_RCU
+extern int rcu_expedited; /* from sysctl */
module_param(rcu_expedited, int, 0);
+extern int rcu_normal; /* from sysctl */
module_param(rcu_normal, int, 0);
static int rcu_normal_after_boot;
module_param(rcu_normal_after_boot, int, 0);
struct rcu_synchronize *rs_array)
{
int i;
+ int j;
/* Initialize and register callbacks for each flavor specified. */
for (i = 0; i < n; i++) {
}
init_rcu_head_on_stack(&rs_array[i].head);
init_completion(&rs_array[i].completion);
- (crcu_array[i])(&rs_array[i].head, wakeme_after_rcu);
+ for (j = 0; j < i; j++)
+ if (crcu_array[j] == crcu_array[i])
+ break;
+ if (j == i)
+ (crcu_array[i])(&rs_array[i].head, wakeme_after_rcu);
}
/* Wait for all callbacks to be invoked. */
(crcu_array[i] == call_rcu ||
crcu_array[i] == call_rcu_bh))
continue;
- wait_for_completion(&rs_array[i].completion);
+ for (j = 0; j < i; j++)
+ if (crcu_array[j] == crcu_array[i])
+ break;
+ if (j == i)
+ wait_for_completion(&rs_array[i].completion);
destroy_rcu_head_on_stack(&rs_array[i].head);
}
}
DEFINE_SRCU(tasks_rcu_exit_srcu);
/* Control stall timeouts. Disable with <= 0, otherwise jiffies till stall. */
-static int rcu_task_stall_timeout __read_mostly = HZ * 60 * 10;
+#define RCU_TASK_STALL_TIMEOUT (HZ * 60 * 10)
+static int rcu_task_stall_timeout __read_mostly = RCU_TASK_STALL_TIMEOUT;
module_param(rcu_task_stall_timeout, int, 0644);
static void rcu_spawn_tasks_kthread(void);
static struct task_struct *rcu_tasks_kthread_ptr;
-/*
- * Post an RCU-tasks callback. First call must be from process context
- * after the scheduler if fully operational.
+/**
+ * call_rcu_tasks() - Queue an RCU for invocation task-based grace period
+ * @rhp: structure to be used for queueing the RCU updates.
+ * @func: actual callback function to be invoked after the grace period
+ *
+ * The callback function will be invoked some time after a full grace
+ * period elapses, in other words after all currently executing RCU
+ * read-side critical sections have completed. call_rcu_tasks() assumes
+ * that the read-side critical sections end at a voluntary context
+ * switch (not a preemption!), entry into idle, or transition to usermode
+ * execution. As such, there are no read-side primitives analogous to
+ * rcu_read_lock() and rcu_read_unlock() because this primitive is intended
+ * to determine that all tasks have passed through a safe state, not so
+ * much for data-strcuture synchronization.
+ *
+ * See the description of call_rcu() for more detailed information on
+ * memory ordering guarantees.
*/
void call_rcu_tasks(struct rcu_head *rhp, rcu_callback_t func)
{
#endif /* #ifdef CONFIG_TASKS_RCU */
+#ifndef CONFIG_TINY_RCU
+
+/*
+ * Print any non-default Tasks RCU settings.
+ */
+static void __init rcu_tasks_bootup_oddness(void)
+{
+#ifdef CONFIG_TASKS_RCU
+ if (rcu_task_stall_timeout != RCU_TASK_STALL_TIMEOUT)
+ pr_info("\tTasks-RCU CPU stall warnings timeout set to %d (rcu_task_stall_timeout).\n", rcu_task_stall_timeout);
+ else
+ pr_info("\tTasks RCU enabled.\n");
+#endif /* #ifdef CONFIG_TASKS_RCU */
+}
+
+#endif /* #ifndef CONFIG_TINY_RCU */
+
#ifdef CONFIG_PROVE_RCU
/*
#else
void rcu_early_boot_tests(void) {}
#endif /* CONFIG_PROVE_RCU */
+
+#ifndef CONFIG_TINY_RCU
+
+/*
+ * Print any significant non-default boot-time settings.
+ */
+void __init rcupdate_announce_bootup_oddness(void)
+{
+ if (rcu_normal)
+ pr_info("\tNo expedited grace period (rcu_normal).\n");
+ else if (rcu_normal_after_boot)
+ pr_info("\tNo expedited grace period (rcu_normal_after_boot).\n");
+ else if (rcu_expedited)
+ pr_info("\tAll grace periods are expedited (rcu_expedited).\n");
+ if (rcu_cpu_stall_suppress)
+ pr_info("\tRCU CPU stall warnings suppressed (rcu_cpu_stall_suppress).\n");
+ if (rcu_cpu_stall_timeout != CONFIG_RCU_CPU_STALL_TIMEOUT)
+ pr_info("\tRCU CPU stall warnings timeout set to %d (rcu_cpu_stall_timeout).\n", rcu_cpu_stall_timeout);
+ rcu_tasks_bootup_oddness();
+}
+
+#endif /* #ifndef CONFIG_TINY_RCU */
endif
obj-y += core.o loadavg.o clock.o cputime.o
-obj-y += idle_task.o fair.o rt.o deadline.o stop_task.o
-obj-y += wait.o swait.o completion.o idle.o
-obj-$(CONFIG_SMP) += cpupri.o cpudeadline.o topology.o
+obj-y += idle_task.o fair.o rt.o deadline.o
+obj-y += wait.o wait_bit.o swait.o completion.o idle.o
+obj-$(CONFIG_SMP) += cpupri.o cpudeadline.o topology.o stop_task.o
obj-$(CONFIG_SCHED_AUTOGROUP) += autogroup.o
obj-$(CONFIG_SCHEDSTATS) += stats.o
obj-$(CONFIG_SCHED_DEBUG) += debug.o
#include <linux/workqueue.h>
#include <linux/compiler.h>
#include <linux/tick.h>
+#include <linux/init.h>
/*
* Scheduler clock - returns current time in nanosec units.
return static_branch_likely(&__sched_clock_stable);
}
+static void __scd_stamp(struct sched_clock_data *scd)
+{
+ scd->tick_gtod = ktime_get_ns();
+ scd->tick_raw = sched_clock();
+}
+
static void __set_sched_clock_stable(void)
{
- struct sched_clock_data *scd = this_scd();
+ struct sched_clock_data *scd;
+ /*
+ * Since we're still unstable and the tick is already running, we have
+ * to disable IRQs in order to get a consistent scd->tick* reading.
+ */
+ local_irq_disable();
+ scd = this_scd();
/*
* Attempt to make the (initial) unstable->stable transition continuous.
*/
__sched_clock_offset = (scd->tick_gtod + __gtod_offset) - (scd->tick_raw);
+ local_irq_enable();
printk(KERN_INFO "sched_clock: Marking stable (%lld, %lld)->(%lld, %lld)\n",
scd->tick_gtod, __gtod_offset,
tick_dep_clear(TICK_DEP_BIT_CLOCK_UNSTABLE);
}
+/*
+ * If we ever get here, we're screwed, because we found out -- typically after
+ * the fact -- that TSC wasn't good. This means all our clocksources (including
+ * ktime) could have reported wrong values.
+ *
+ * What we do here is an attempt to fix up and continue sort of where we left
+ * off in a coherent manner.
+ *
+ * The only way to fully avoid random clock jumps is to boot with:
+ * "tsc=unstable".
+ */
static void __sched_clock_work(struct work_struct *work)
{
+ struct sched_clock_data *scd;
+ int cpu;
+
+ /* take a current timestamp and set 'now' */
+ preempt_disable();
+ scd = this_scd();
+ __scd_stamp(scd);
+ scd->clock = scd->tick_gtod + __gtod_offset;
+ preempt_enable();
+
+ /* clone to all CPUs */
+ for_each_possible_cpu(cpu)
+ per_cpu(sched_clock_data, cpu) = *scd;
+
+ printk(KERN_WARNING "TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'.\n");
+ printk(KERN_INFO "sched_clock: Marking unstable (%lld, %lld)<-(%lld, %lld)\n",
+ scd->tick_gtod, __gtod_offset,
+ scd->tick_raw, __sched_clock_offset);
+
static_branch_disable(&__sched_clock_stable);
}
static void __clear_sched_clock_stable(void)
{
- struct sched_clock_data *scd = this_scd();
-
- /*
- * Attempt to make the stable->unstable transition continuous.
- *
- * Trouble is, this is typically called from the TSC watchdog
- * timer, which is late per definition. This means the tick
- * values can already be screwy.
- *
- * Still do what we can.
- */
- __gtod_offset = (scd->tick_raw + __sched_clock_offset) - (scd->tick_gtod);
-
- printk(KERN_INFO "sched_clock: Marking unstable (%lld, %lld)<-(%lld, %lld)\n",
- scd->tick_gtod, __gtod_offset,
- scd->tick_raw, __sched_clock_offset);
+ if (!sched_clock_stable())
+ return;
tick_dep_set(TICK_DEP_BIT_CLOCK_UNSTABLE);
-
- if (sched_clock_stable())
- schedule_work(&sched_clock_work);
+ schedule_work(&sched_clock_work);
}
void clear_sched_clock_stable(void)
__clear_sched_clock_stable();
}
-void sched_clock_init_late(void)
+/*
+ * We run this as late_initcall() such that it runs after all built-in drivers,
+ * notably: acpi_processor and intel_idle, which can mark the TSC as unstable.
+ */
+static int __init sched_clock_init_late(void)
{
sched_clock_running = 2;
/*
if (__sched_clock_stable_early)
__set_sched_clock_stable();
+
+ return 0;
}
+late_initcall(sched_clock_init_late);
/*
* min, max except they take wrapping into account
{
struct sched_clock_data *scd;
+ if (sched_clock_stable())
+ return;
+
+ if (unlikely(!sched_clock_running))
+ return;
+
WARN_ON_ONCE(!irqs_disabled());
+ scd = this_scd();
+ __scd_stamp(scd);
+ sched_clock_local(scd);
+}
+
+void sched_clock_tick_stable(void)
+{
+ u64 gtod, clock;
+
+ if (!sched_clock_stable())
+ return;
+
/*
- * Update these values even if sched_clock_stable(), because it can
- * become unstable at any point in time at which point we need some
- * values to fall back on.
+ * Called under watchdog_lock.
*
- * XXX arguably we can skip this if we expose tsc_clocksource_reliable
+ * The watchdog just found this TSC to (still) be stable, so now is a
+ * good moment to update our __gtod_offset. Because once we find the
+ * TSC to be unstable, any computation will be computing crap.
*/
- scd = this_scd();
- scd->tick_raw = sched_clock();
- scd->tick_gtod = ktime_get_ns();
-
- if (!sched_clock_stable() && likely(sched_clock_running))
- sched_clock_local(scd);
+ local_irq_disable();
+ gtod = ktime_get_ns();
+ clock = sched_clock();
+ __gtod_offset = (clock + __sched_clock_offset) - gtod;
+ local_irq_enable();
}
/*
EXPORT_SYMBOL_GPL(sched_clock_idle_sleep_event);
/*
- * We just idled delta nanoseconds (called with irqs disabled):
+ * We just idled; resync with ktime.
*/
-void sched_clock_idle_wakeup_event(u64 delta_ns)
+void sched_clock_idle_wakeup_event(void)
{
- if (timekeeping_suspended)
+ unsigned long flags;
+
+ if (sched_clock_stable())
+ return;
+
+ if (unlikely(timekeeping_suspended))
return;
+ local_irq_save(flags);
sched_clock_tick();
- touch_softlockup_watchdog_sched();
+ local_irq_restore(flags);
}
EXPORT_SYMBOL_GPL(sched_clock_idle_wakeup_event);
if (!x->done) {
DECLARE_WAITQUEUE(wait, current);
- __add_wait_queue_tail_exclusive(&x->wait, &wait);
+ __add_wait_queue_entry_tail_exclusive(&x->wait, &wait);
do {
if (signal_pending_state(state, current)) {
timeout = -ERESTARTSYS;
#include <uapi/linux/sched/types.h>
#include <linux/sched/loadavg.h>
#include <linux/sched/hotplug.h>
+#include <linux/wait_bit.h>
#include <linux/cpuset.h>
#include <linux/delayacct.h>
#include <linux/init_task.h>
dequeue_task(rq, p, flags);
}
-void sched_set_stop_task(int cpu, struct task_struct *stop)
-{
- struct sched_param param = { .sched_priority = MAX_RT_PRIO - 1 };
- struct task_struct *old_stop = cpu_rq(cpu)->stop;
-
- if (stop) {
- /*
- * Make it appear like a SCHED_FIFO task, its something
- * userspace knows about and won't get confused about.
- *
- * Also, it will make PI more or less work without too
- * much confusion -- but then, stop work should not
- * rely on PI working anyway.
- */
- sched_setscheduler_nocheck(stop, SCHED_FIFO, ¶m);
-
- stop->sched_class = &stop_sched_class;
- }
-
- cpu_rq(cpu)->stop = stop;
-
- if (old_stop) {
- /*
- * Reset it back to a normal scheduling class so that
- * it can die in pieces.
- */
- old_stop->sched_class = &rt_sched_class;
- }
-}
-
/*
* __normal_prio - return the priority that is based on the static prio
*/
*avg += diff >> 3;
}
+void sched_set_stop_task(int cpu, struct task_struct *stop)
+{
+ struct sched_param param = { .sched_priority = MAX_RT_PRIO - 1 };
+ struct task_struct *old_stop = cpu_rq(cpu)->stop;
+
+ if (stop) {
+ /*
+ * Make it appear like a SCHED_FIFO task, its something
+ * userspace knows about and won't get confused about.
+ *
+ * Also, it will make PI more or less work without too
+ * much confusion -- but then, stop work should not
+ * rely on PI working anyway.
+ */
+ sched_setscheduler_nocheck(stop, SCHED_FIFO, ¶m);
+
+ stop->sched_class = &stop_sched_class;
+ }
+
+ cpu_rq(cpu)->stop = stop;
+
+ if (old_stop) {
+ /*
+ * Reset it back to a normal scheduling class so that
+ * it can die in pieces.
+ */
+ old_stop->sched_class = &rt_sched_class;
+ }
+}
+
#else
static inline int __set_cpus_allowed_ptr(struct task_struct *p,
{
struct rq *rq = this_rq();
struct llist_node *llist = llist_del_all(&rq->wake_list);
- struct task_struct *p;
+ struct task_struct *p, *t;
struct rq_flags rf;
if (!llist)
rq_lock_irqsave(rq, &rf);
update_rq_clock(rq);
- while (llist) {
- int wake_flags = 0;
-
- p = llist_entry(llist, struct task_struct, wake_entry);
- llist = llist_next(llist);
-
- if (p->sched_remote_wakeup)
- wake_flags = WF_MIGRATED;
-
- ttwu_do_activate(rq, p, wake_flags, &rf);
- }
+ llist_for_each_entry_safe(p, t, llist, wake_entry)
+ ttwu_do_activate(rq, p, p->sched_remote_wakeup ? WF_MIGRATED : 0, &rf);
rq_unlock_irqrestore(rq, &rf);
}
return try_to_wake_up(p, state, 0);
}
-/*
- * This function clears the sched_dl_entity static params.
- */
-void __dl_clear_params(struct task_struct *p)
-{
- struct sched_dl_entity *dl_se = &p->dl;
-
- dl_se->dl_runtime = 0;
- dl_se->dl_deadline = 0;
- dl_se->dl_period = 0;
- dl_se->flags = 0;
- dl_se->dl_bw = 0;
-
- dl_se->dl_throttled = 0;
- dl_se->dl_yielded = 0;
-}
-
/*
* Perform scheduler related setup for a newly forked process p.
* p is forked by current.
RB_CLEAR_NODE(&p->dl.rb_node);
init_dl_task_timer(&p->dl);
+ init_dl_inactive_task_timer(&p->dl);
__dl_clear_params(p);
INIT_LIST_HEAD(&p->rt.run_list);
unsigned long to_ratio(u64 period, u64 runtime)
{
if (runtime == RUNTIME_INF)
- return 1ULL << 20;
+ return BW_UNIT;
/*
* Doing this here saves a lot of checks in all
if (period == 0)
return 0;
- return div64_u64(runtime << 20, period);
-}
-
-#ifdef CONFIG_SMP
-inline struct dl_bw *dl_bw_of(int i)
-{
- RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held(),
- "sched RCU must be held");
- return &cpu_rq(i)->rd->dl_bw;
-}
-
-static inline int dl_bw_cpus(int i)
-{
- struct root_domain *rd = cpu_rq(i)->rd;
- int cpus = 0;
-
- RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held(),
- "sched RCU must be held");
- for_each_cpu_and(i, rd->span, cpu_active_mask)
- cpus++;
-
- return cpus;
-}
-#else
-inline struct dl_bw *dl_bw_of(int i)
-{
- return &cpu_rq(i)->dl.dl_bw;
-}
-
-static inline int dl_bw_cpus(int i)
-{
- return 1;
-}
-#endif
-
-/*
- * We must be sure that accepting a new task (or allowing changing the
- * parameters of an existing one) is consistent with the bandwidth
- * constraints. If yes, this function also accordingly updates the currently
- * allocated bandwidth to reflect the new situation.
- *
- * This function is called while holding p's rq->lock.
- *
- * XXX we should delay bw change until the task's 0-lag point, see
- * __setparam_dl().
- */
-static int dl_overflow(struct task_struct *p, int policy,
- const struct sched_attr *attr)
-{
-
- struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
- u64 period = attr->sched_period ?: attr->sched_deadline;
- u64 runtime = attr->sched_runtime;
- u64 new_bw = dl_policy(policy) ? to_ratio(period, runtime) : 0;
- int cpus, err = -1;
-
- /* !deadline task may carry old deadline bandwidth */
- if (new_bw == p->dl.dl_bw && task_has_dl_policy(p))
- return 0;
-
- /*
- * Either if a task, enters, leave, or stays -deadline but changes
- * its parameters, we may need to update accordingly the total
- * allocated bandwidth of the container.
- */
- raw_spin_lock(&dl_b->lock);
- cpus = dl_bw_cpus(task_cpu(p));
- if (dl_policy(policy) && !task_has_dl_policy(p) &&
- !__dl_overflow(dl_b, cpus, 0, new_bw)) {
- __dl_add(dl_b, new_bw);
- err = 0;
- } else if (dl_policy(policy) && task_has_dl_policy(p) &&
- !__dl_overflow(dl_b, cpus, p->dl.dl_bw, new_bw)) {
- __dl_clear(dl_b, p->dl.dl_bw);
- __dl_add(dl_b, new_bw);
- err = 0;
- } else if (!dl_policy(policy) && task_has_dl_policy(p)) {
- __dl_clear(dl_b, p->dl.dl_bw);
- err = 0;
- }
- raw_spin_unlock(&dl_b->lock);
-
- return err;
+ return div64_u64(runtime << BW_SHIFT, period);
}
-extern void init_dl_bw(struct dl_bw *dl_b);
-
/*
* wake_up_new_task - wake up a newly created task for the first time.
*
exception_exit(prev_state);
}
-int default_wake_function(wait_queue_t *curr, unsigned mode, int wake_flags,
+int default_wake_function(wait_queue_entry_t *curr, unsigned mode, int wake_flags,
void *key)
{
return try_to_wake_up(curr->private, mode, wake_flags);
return pid ? find_task_by_vpid(pid) : current;
}
-/*
- * This function initializes the sched_dl_entity of a newly becoming
- * SCHED_DEADLINE task.
- *
- * Only the static values are considered here, the actual runtime and the
- * absolute deadline will be properly calculated when the task is enqueued
- * for the first time with its new policy.
- */
-static void
-__setparam_dl(struct task_struct *p, const struct sched_attr *attr)
-{
- struct sched_dl_entity *dl_se = &p->dl;
-
- dl_se->dl_runtime = attr->sched_runtime;
- dl_se->dl_deadline = attr->sched_deadline;
- dl_se->dl_period = attr->sched_period ?: dl_se->dl_deadline;
- dl_se->flags = attr->sched_flags;
- dl_se->dl_bw = to_ratio(dl_se->dl_period, dl_se->dl_runtime);
-
- /*
- * Changing the parameters of a task is 'tricky' and we're not doing
- * the correct thing -- also see task_dead_dl() and switched_from_dl().
- *
- * What we SHOULD do is delay the bandwidth release until the 0-lag
- * point. This would include retaining the task_struct until that time
- * and change dl_overflow() to not immediately decrement the current
- * amount.
- *
- * Instead we retain the current runtime/deadline and let the new
- * parameters take effect after the current reservation period lapses.
- * This is safe (albeit pessimistic) because the 0-lag point is always
- * before the current scheduling deadline.
- *
- * We can still have temporary overloads because we do not delay the
- * change in bandwidth until that time; so admission control is
- * not on the safe side. It does however guarantee tasks will never
- * consume more than promised.
- */
-}
-
/*
* sched_setparam() passes in -1 for its policy, to let the functions
* it calls know not to change it.
p->sched_class = &fair_sched_class;
}
-static void
-__getparam_dl(struct task_struct *p, struct sched_attr *attr)
-{
- struct sched_dl_entity *dl_se = &p->dl;
-
- attr->sched_priority = p->rt_priority;
- attr->sched_runtime = dl_se->dl_runtime;
- attr->sched_deadline = dl_se->dl_deadline;
- attr->sched_period = dl_se->dl_period;
- attr->sched_flags = dl_se->flags;
-}
-
-/*
- * This function validates the new parameters of a -deadline task.
- * We ask for the deadline not being zero, and greater or equal
- * than the runtime, as well as the period of being zero or
- * greater than deadline. Furthermore, we have to be sure that
- * user parameters are above the internal resolution of 1us (we
- * check sched_runtime only since it is always the smaller one) and
- * below 2^63 ns (we have to check both sched_deadline and
- * sched_period, as the latter can be zero).
- */
-static bool
-__checkparam_dl(const struct sched_attr *attr)
-{
- /* deadline != 0 */
- if (attr->sched_deadline == 0)
- return false;
-
- /*
- * Since we truncate DL_SCALE bits, make sure we're at least
- * that big.
- */
- if (attr->sched_runtime < (1ULL << DL_SCALE))
- return false;
-
- /*
- * Since we use the MSB for wrap-around and sign issues, make
- * sure it's not set (mind that period can be equal to zero).
- */
- if (attr->sched_deadline & (1ULL << 63) ||
- attr->sched_period & (1ULL << 63))
- return false;
-
- /* runtime <= deadline <= period (if period != 0) */
- if ((attr->sched_period != 0 &&
- attr->sched_period < attr->sched_deadline) ||
- attr->sched_deadline < attr->sched_runtime)
- return false;
-
- return true;
-}
-
/*
* Check the target process has a UID that matches the current process's:
*/
return match;
}
-static bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr)
-{
- struct sched_dl_entity *dl_se = &p->dl;
-
- if (dl_se->dl_runtime != attr->sched_runtime ||
- dl_se->dl_deadline != attr->sched_deadline ||
- dl_se->dl_period != attr->sched_period ||
- dl_se->flags != attr->sched_flags)
- return true;
-
- return false;
-}
-
static int __sched_setscheduler(struct task_struct *p,
const struct sched_attr *attr,
bool user, bool pi)
int queue_flags = DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK;
struct rq *rq;
- /* May grab non-irq protected spin_locks: */
- BUG_ON(in_interrupt());
+ /* The pi code expects interrupts enabled */
+ BUG_ON(pi && in_interrupt());
recheck:
/* Double check policy once rq lock held: */
if (policy < 0) {
return -EINVAL;
}
- if (attr->sched_flags & ~(SCHED_FLAG_RESET_ON_FORK))
+ if (attr->sched_flags &
+ ~(SCHED_FLAG_RESET_ON_FORK | SCHED_FLAG_RECLAIM))
return -EINVAL;
/*
* of a SCHED_DEADLINE task) we need to check if enough bandwidth
* is available.
*/
- if ((dl_policy(policy) || dl_task(p)) && dl_overflow(p, policy, attr)) {
+ if ((dl_policy(policy) || dl_task(p)) && sched_dl_overflow(p, policy, attr)) {
task_rq_unlock(rq, p, &rf);
return -EBUSY;
}
#endif
}
+#ifdef CONFIG_SMP
+
int cpuset_cpumask_can_shrink(const struct cpumask *cur,
const struct cpumask *trial)
{
- int ret = 1, trial_cpus;
- struct dl_bw *cur_dl_b;
- unsigned long flags;
+ int ret = 1;
if (!cpumask_weight(cur))
return ret;
- rcu_read_lock_sched();
- cur_dl_b = dl_bw_of(cpumask_any(cur));
- trial_cpus = cpumask_weight(trial);
-
- raw_spin_lock_irqsave(&cur_dl_b->lock, flags);
- if (cur_dl_b->bw != -1 &&
- cur_dl_b->bw * trial_cpus < cur_dl_b->total_bw)
- ret = 0;
- raw_spin_unlock_irqrestore(&cur_dl_b->lock, flags);
- rcu_read_unlock_sched();
+ ret = dl_cpuset_cpumask_can_shrink(cur, trial);
return ret;
}
goto out;
}
-#ifdef CONFIG_SMP
if (dl_task(p) && !cpumask_intersects(task_rq(p)->rd->span,
- cs_cpus_allowed)) {
- unsigned int dest_cpu = cpumask_any_and(cpu_active_mask,
- cs_cpus_allowed);
- struct dl_bw *dl_b;
- bool overflow;
- int cpus;
- unsigned long flags;
-
- rcu_read_lock_sched();
- dl_b = dl_bw_of(dest_cpu);
- raw_spin_lock_irqsave(&dl_b->lock, flags);
- cpus = dl_bw_cpus(dest_cpu);
- overflow = __dl_overflow(dl_b, cpus, 0, p->dl.dl_bw);
- if (overflow)
- ret = -EBUSY;
- else {
- /*
- * We reserve space for this task in the destination
- * root_domain, as we can't fail after this point.
- * We will free resources in the source root_domain
- * later on (see set_cpus_allowed_dl()).
- */
- __dl_add(dl_b, p->dl.dl_bw);
- }
- raw_spin_unlock_irqrestore(&dl_b->lock, flags);
- rcu_read_unlock_sched();
+ cs_cpus_allowed))
+ ret = dl_task_can_attach(p, cs_cpus_allowed);
- }
-#endif
out:
return ret;
}
-#ifdef CONFIG_SMP
-
bool sched_smp_initialized __read_mostly;
#ifdef CONFIG_NUMA_BALANCING
static int cpuset_cpu_inactive(unsigned int cpu)
{
- unsigned long flags;
- struct dl_bw *dl_b;
- bool overflow;
- int cpus;
-
if (!cpuhp_tasks_frozen) {
- rcu_read_lock_sched();
- dl_b = dl_bw_of(cpu);
-
- raw_spin_lock_irqsave(&dl_b->lock, flags);
- cpus = dl_bw_cpus(cpu);
- overflow = __dl_overflow(dl_b, cpus, 0, 0);
- raw_spin_unlock_irqrestore(&dl_b->lock, flags);
-
- rcu_read_unlock_sched();
-
- if (overflow)
+ if (dl_cpu_busy(cpu))
return -EBUSY;
cpuset_update_active_cpus();
} else {
* users of this state to go away such that all new such users will
* observe it.
*
- * For CONFIG_PREEMPT we have preemptible RCU and its sync_rcu() might
- * not imply sync_sched(), so wait for both.
- *
* Do sync before park smpboot threads to take care the rcu boost case.
*/
- if (IS_ENABLED(CONFIG_PREEMPT))
- synchronize_rcu_mult(call_rcu, call_rcu_sched);
- else
- synchronize_rcu();
+ synchronize_rcu_mult(call_rcu, call_rcu_sched);
if (!sched_smp_initialized)
return 0;
cpumask_var_t non_isolated_cpus;
alloc_cpumask_var(&non_isolated_cpus, GFP_KERNEL);
- alloc_cpumask_var(&fallback_doms, GFP_KERNEL);
sched_init_numa();
* happen.
*/
mutex_lock(&sched_domains_mutex);
- init_sched_domains(cpu_active_mask);
+ sched_init_domains(cpu_active_mask);
cpumask_andnot(non_isolated_cpus, cpu_possible_mask, cpu_isolated_map);
if (cpumask_empty(non_isolated_cpus))
cpumask_set_cpu(smp_processor_id(), non_isolated_cpus);
init_sched_dl_class();
sched_init_smt();
- sched_clock_init_late();
sched_smp_initialized = true;
}
void __init sched_init_smp(void)
{
sched_init_granularity();
- sched_clock_init_late();
}
#endif /* CONFIG_SMP */
DECLARE_PER_CPU(cpumask_var_t, load_balance_mask);
DECLARE_PER_CPU(cpumask_var_t, select_idle_mask);
-#define WAIT_TABLE_BITS 8
-#define WAIT_TABLE_SIZE (1 << WAIT_TABLE_BITS)
-static wait_queue_head_t bit_wait_table[WAIT_TABLE_SIZE] __cacheline_aligned;
-
-wait_queue_head_t *bit_waitqueue(void *word, int bit)
-{
- const int shift = BITS_PER_LONG == 32 ? 5 : 6;
- unsigned long val = (unsigned long)word << shift | bit;
-
- return bit_wait_table + hash_long(val, WAIT_TABLE_BITS);
-}
-EXPORT_SYMBOL(bit_waitqueue);
-
void __init sched_init(void)
{
int i, j;
unsigned long alloc_size = 0, ptr;
sched_clock_init();
-
- for (i = 0; i < WAIT_TABLE_SIZE; i++)
- init_waitqueue_head(bit_wait_table + i);
+ wait_bit_init();
#ifdef CONFIG_FAIR_GROUP_SCHED
alloc_size += 2 * nr_cpu_ids * sizeof(void **);
calc_load_update = jiffies + LOAD_FREQ;
#ifdef CONFIG_SMP
- zalloc_cpumask_var(&sched_domains_tmpmask, GFP_NOWAIT);
/* May be allocated at isolcpus cmdline parse time */
if (cpu_isolated_map == NULL)
zalloc_cpumask_var(&cpu_isolated_map, GFP_NOWAIT);
if ((preempt_count_equals(preempt_offset) && !irqs_disabled() &&
!is_idle_task(current)) ||
- system_state != SYSTEM_RUNNING || oops_in_progress)
+ system_state == SYSTEM_BOOTING || system_state > SYSTEM_RUNNING ||
+ oops_in_progress)
return;
+
if (time_before(jiffies, prev_jiffy + HZ) && prev_jiffy)
return;
prev_jiffy = jiffies;
task_rq_unlock(rq, tsk, &rf);
}
-#endif /* CONFIG_CGROUP_SCHED */
-
-#ifdef CONFIG_RT_GROUP_SCHED
-/*
- * Ensure that the real time constraints are schedulable.
- */
-static DEFINE_MUTEX(rt_constraints_mutex);
-
-/* Must be called with tasklist_lock held */
-static inline int tg_has_rt_tasks(struct task_group *tg)
-{
- struct task_struct *g, *p;
-
- /*
- * Autogroups do not have RT tasks; see autogroup_create().
- */
- if (task_group_is_autogroup(tg))
- return 0;
-
- for_each_process_thread(g, p) {
- if (rt_task(p) && task_group(p) == tg)
- return 1;
- }
-
- return 0;
-}
-
-struct rt_schedulable_data {
- struct task_group *tg;
- u64 rt_period;
- u64 rt_runtime;
-};
-
-static int tg_rt_schedulable(struct task_group *tg, void *data)
-{
- struct rt_schedulable_data *d = data;
- struct task_group *child;
- unsigned long total, sum = 0;
- u64 period, runtime;
-
- period = ktime_to_ns(tg->rt_bandwidth.rt_period);
- runtime = tg->rt_bandwidth.rt_runtime;
-
- if (tg == d->tg) {
- period = d->rt_period;
- runtime = d->rt_runtime;
- }
-
- /*
- * Cannot have more runtime than the period.
- */
- if (runtime > period && runtime != RUNTIME_INF)
- return -EINVAL;
-
- /*
- * Ensure we don't starve existing RT tasks.
- */
- if (rt_bandwidth_enabled() && !runtime && tg_has_rt_tasks(tg))
- return -EBUSY;
-
- total = to_ratio(period, runtime);
-
- /*
- * Nobody can have more than the global setting allows.
- */
- if (total > to_ratio(global_rt_period(), global_rt_runtime()))
- return -EINVAL;
-
- /*
- * The sum of our children's runtime should not exceed our own.
- */
- list_for_each_entry_rcu(child, &tg->children, siblings) {
- period = ktime_to_ns(child->rt_bandwidth.rt_period);
- runtime = child->rt_bandwidth.rt_runtime;
-
- if (child == d->tg) {
- period = d->rt_period;
- runtime = d->rt_runtime;
- }
-
- sum += to_ratio(period, runtime);
- }
-
- if (sum > total)
- return -EINVAL;
-
- return 0;
-}
-
-static int __rt_schedulable(struct task_group *tg, u64 period, u64 runtime)
-{
- int ret;
-
- struct rt_schedulable_data data = {
- .tg = tg,
- .rt_period = period,
- .rt_runtime = runtime,
- };
-
- rcu_read_lock();
- ret = walk_tg_tree(tg_rt_schedulable, tg_nop, &data);
- rcu_read_unlock();
-
- return ret;
-}
-
-static int tg_set_rt_bandwidth(struct task_group *tg,
- u64 rt_period, u64 rt_runtime)
-{
- int i, err = 0;
-
- /*
- * Disallowing the root group RT runtime is BAD, it would disallow the
- * kernel creating (and or operating) RT threads.
- */
- if (tg == &root_task_group && rt_runtime == 0)
- return -EINVAL;
-
- /* No period doesn't make any sense. */
- if (rt_period == 0)
- return -EINVAL;
-
- mutex_lock(&rt_constraints_mutex);
- read_lock(&tasklist_lock);
- err = __rt_schedulable(tg, rt_period, rt_runtime);
- if (err)
- goto unlock;
-
- raw_spin_lock_irq(&tg->rt_bandwidth.rt_runtime_lock);
- tg->rt_bandwidth.rt_period = ns_to_ktime(rt_period);
- tg->rt_bandwidth.rt_runtime = rt_runtime;
-
- for_each_possible_cpu(i) {
- struct rt_rq *rt_rq = tg->rt_rq[i];
-
- raw_spin_lock(&rt_rq->rt_runtime_lock);
- rt_rq->rt_runtime = rt_runtime;
- raw_spin_unlock(&rt_rq->rt_runtime_lock);
- }
- raw_spin_unlock_irq(&tg->rt_bandwidth.rt_runtime_lock);
-unlock:
- read_unlock(&tasklist_lock);
- mutex_unlock(&rt_constraints_mutex);
-
- return err;
-}
-
-static int sched_group_set_rt_runtime(struct task_group *tg, long rt_runtime_us)
-{
- u64 rt_runtime, rt_period;
-
- rt_period = ktime_to_ns(tg->rt_bandwidth.rt_period);
- rt_runtime = (u64)rt_runtime_us * NSEC_PER_USEC;
- if (rt_runtime_us < 0)
- rt_runtime = RUNTIME_INF;
-
- return tg_set_rt_bandwidth(tg, rt_period, rt_runtime);
-}
-
-static long sched_group_rt_runtime(struct task_group *tg)
-{
- u64 rt_runtime_us;
-
- if (tg->rt_bandwidth.rt_runtime == RUNTIME_INF)
- return -1;
-
- rt_runtime_us = tg->rt_bandwidth.rt_runtime;
- do_div(rt_runtime_us, NSEC_PER_USEC);
- return rt_runtime_us;
-}
-
-static int sched_group_set_rt_period(struct task_group *tg, u64 rt_period_us)
-{
- u64 rt_runtime, rt_period;
-
- rt_period = rt_period_us * NSEC_PER_USEC;
- rt_runtime = tg->rt_bandwidth.rt_runtime;
-
- return tg_set_rt_bandwidth(tg, rt_period, rt_runtime);
-}
-
-static long sched_group_rt_period(struct task_group *tg)
-{
- u64 rt_period_us;
-
- rt_period_us = ktime_to_ns(tg->rt_bandwidth.rt_period);
- do_div(rt_period_us, NSEC_PER_USEC);
- return rt_period_us;
-}
-#endif /* CONFIG_RT_GROUP_SCHED */
-
-#ifdef CONFIG_RT_GROUP_SCHED
-static int sched_rt_global_constraints(void)
-{
- int ret = 0;
-
- mutex_lock(&rt_constraints_mutex);
- read_lock(&tasklist_lock);
- ret = __rt_schedulable(NULL, 0, 0);
- read_unlock(&tasklist_lock);
- mutex_unlock(&rt_constraints_mutex);
-
- return ret;
-}
-
-static int sched_rt_can_attach(struct task_group *tg, struct task_struct *tsk)
-{
- /* Don't accept realtime tasks when there is no way for them to run */
- if (rt_task(tsk) && tg->rt_bandwidth.rt_runtime == 0)
- return 0;
-
- return 1;
-}
-
-#else /* !CONFIG_RT_GROUP_SCHED */
-static int sched_rt_global_constraints(void)
-{
- unsigned long flags;
- int i;
-
- raw_spin_lock_irqsave(&def_rt_bandwidth.rt_runtime_lock, flags);
- for_each_possible_cpu(i) {
- struct rt_rq *rt_rq = &cpu_rq(i)->rt;
-
- raw_spin_lock(&rt_rq->rt_runtime_lock);
- rt_rq->rt_runtime = global_rt_runtime();
- raw_spin_unlock(&rt_rq->rt_runtime_lock);
- }
- raw_spin_unlock_irqrestore(&def_rt_bandwidth.rt_runtime_lock, flags);
-
- return 0;
-}
-#endif /* CONFIG_RT_GROUP_SCHED */
-
-static int sched_dl_global_validate(void)
-{
- u64 runtime = global_rt_runtime();
- u64 period = global_rt_period();
- u64 new_bw = to_ratio(period, runtime);
- struct dl_bw *dl_b;
- int cpu, ret = 0;
- unsigned long flags;
-
- /*
- * Here we want to check the bandwidth not being set to some
- * value smaller than the currently allocated bandwidth in
- * any of the root_domains.
- *
- * FIXME: Cycling on all the CPUs is overdoing, but simpler than
- * cycling on root_domains... Discussion on different/better
- * solutions is welcome!
- */
- for_each_possible_cpu(cpu) {
- rcu_read_lock_sched();
- dl_b = dl_bw_of(cpu);
-
- raw_spin_lock_irqsave(&dl_b->lock, flags);
- if (new_bw < dl_b->total_bw)
- ret = -EBUSY;
- raw_spin_unlock_irqrestore(&dl_b->lock, flags);
-
- rcu_read_unlock_sched();
-
- if (ret)
- break;
- }
-
- return ret;
-}
-
-static void sched_dl_do_global(void)
-{
- u64 new_bw = -1;
- struct dl_bw *dl_b;
- int cpu;
- unsigned long flags;
-
- def_dl_bandwidth.dl_period = global_rt_period();
- def_dl_bandwidth.dl_runtime = global_rt_runtime();
-
- if (global_rt_runtime() != RUNTIME_INF)
- new_bw = to_ratio(global_rt_period(), global_rt_runtime());
-
- /*
- * FIXME: As above...
- */
- for_each_possible_cpu(cpu) {
- rcu_read_lock_sched();
- dl_b = dl_bw_of(cpu);
-
- raw_spin_lock_irqsave(&dl_b->lock, flags);
- dl_b->bw = new_bw;
- raw_spin_unlock_irqrestore(&dl_b->lock, flags);
-
- rcu_read_unlock_sched();
- }
-}
-
-static int sched_rt_global_validate(void)
-{
- if (sysctl_sched_rt_period <= 0)
- return -EINVAL;
-
- if ((sysctl_sched_rt_runtime != RUNTIME_INF) &&
- (sysctl_sched_rt_runtime > sysctl_sched_rt_period))
- return -EINVAL;
-
- return 0;
-}
-
-static void sched_rt_do_global(void)
-{
- def_rt_bandwidth.rt_runtime = global_rt_runtime();
- def_rt_bandwidth.rt_period = ns_to_ktime(global_rt_period());
-}
-
-int sched_rt_handler(struct ctl_table *table, int write,
- void __user *buffer, size_t *lenp,
- loff_t *ppos)
-{
- int old_period, old_runtime;
- static DEFINE_MUTEX(mutex);
- int ret;
-
- mutex_lock(&mutex);
- old_period = sysctl_sched_rt_period;
- old_runtime = sysctl_sched_rt_runtime;
-
- ret = proc_dointvec(table, write, buffer, lenp, ppos);
-
- if (!ret && write) {
- ret = sched_rt_global_validate();
- if (ret)
- goto undo;
-
- ret = sched_dl_global_validate();
- if (ret)
- goto undo;
-
- ret = sched_rt_global_constraints();
- if (ret)
- goto undo;
-
- sched_rt_do_global();
- sched_dl_do_global();
- }
- if (0) {
-undo:
- sysctl_sched_rt_period = old_period;
- sysctl_sched_rt_runtime = old_runtime;
- }
- mutex_unlock(&mutex);
-
- return ret;
-}
-
-int sched_rr_handler(struct ctl_table *table, int write,
- void __user *buffer, size_t *lenp,
- loff_t *ppos)
-{
- int ret;
- static DEFINE_MUTEX(mutex);
-
- mutex_lock(&mutex);
- ret = proc_dointvec(table, write, buffer, lenp, ppos);
- /*
- * Make sure that internally we keep jiffies.
- * Also, writing zero resets the timeslice to default:
- */
- if (!ret && write) {
- sched_rr_timeslice =
- sysctl_sched_rr_timeslice <= 0 ? RR_TIMESLICE :
- msecs_to_jiffies(sysctl_sched_rr_timeslice);
- }
- mutex_unlock(&mutex);
- return ret;
-}
-
-#ifdef CONFIG_CGROUP_SCHED
static inline struct task_group *css_tg(struct cgroup_subsys_state *css)
{
* userspace. Once a task gets some ticks, the monotonicy code at
* 'update' will ensure things converge to the observed ratio.
*/
- if (stime == 0) {
- utime = rtime;
- goto update;
+ if (stime != 0) {
+ if (utime == 0)
+ stime = rtime;
+ else
+ stime = scale_stime(stime, rtime, stime + utime);
}
- if (utime == 0) {
- stime = rtime;
- goto update;
- }
-
- stime = scale_stime(stime, rtime, stime + utime);
-
-update:
/*
* Make sure stime doesn't go backwards; this preserves monotonicity
* for utime because rtime is monotonic.
#include "sched.h"
#include <linux/slab.h>
+#include <uapi/linux/sched/types.h>
struct dl_bandwidth def_dl_bandwidth;
return !RB_EMPTY_NODE(&dl_se->rb_node);
}
+#ifdef CONFIG_SMP
+static inline struct dl_bw *dl_bw_of(int i)
+{
+ RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held(),
+ "sched RCU must be held");
+ return &cpu_rq(i)->rd->dl_bw;
+}
+
+static inline int dl_bw_cpus(int i)
+{
+ struct root_domain *rd = cpu_rq(i)->rd;
+ int cpus = 0;
+
+ RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held(),
+ "sched RCU must be held");
+ for_each_cpu_and(i, rd->span, cpu_active_mask)
+ cpus++;
+
+ return cpus;
+}
+#else
+static inline struct dl_bw *dl_bw_of(int i)
+{
+ return &cpu_rq(i)->dl.dl_bw;
+}
+
+static inline int dl_bw_cpus(int i)
+{
+ return 1;
+}
+#endif
+
+static inline
+void add_running_bw(u64 dl_bw, struct dl_rq *dl_rq)
+{
+ u64 old = dl_rq->running_bw;
+
+ lockdep_assert_held(&(rq_of_dl_rq(dl_rq))->lock);
+ dl_rq->running_bw += dl_bw;
+ SCHED_WARN_ON(dl_rq->running_bw < old); /* overflow */
+ SCHED_WARN_ON(dl_rq->running_bw > dl_rq->this_bw);
+}
+
+static inline
+void sub_running_bw(u64 dl_bw, struct dl_rq *dl_rq)
+{
+ u64 old = dl_rq->running_bw;
+
+ lockdep_assert_held(&(rq_of_dl_rq(dl_rq))->lock);
+ dl_rq->running_bw -= dl_bw;
+ SCHED_WARN_ON(dl_rq->running_bw > old); /* underflow */
+ if (dl_rq->running_bw > old)
+ dl_rq->running_bw = 0;
+}
+
+static inline
+void add_rq_bw(u64 dl_bw, struct dl_rq *dl_rq)
+{
+ u64 old = dl_rq->this_bw;
+
+ lockdep_assert_held(&(rq_of_dl_rq(dl_rq))->lock);
+ dl_rq->this_bw += dl_bw;
+ SCHED_WARN_ON(dl_rq->this_bw < old); /* overflow */
+}
+
+static inline
+void sub_rq_bw(u64 dl_bw, struct dl_rq *dl_rq)
+{
+ u64 old = dl_rq->this_bw;
+
+ lockdep_assert_held(&(rq_of_dl_rq(dl_rq))->lock);
+ dl_rq->this_bw -= dl_bw;
+ SCHED_WARN_ON(dl_rq->this_bw > old); /* underflow */
+ if (dl_rq->this_bw > old)
+ dl_rq->this_bw = 0;
+ SCHED_WARN_ON(dl_rq->running_bw > dl_rq->this_bw);
+}
+
+void dl_change_utilization(struct task_struct *p, u64 new_bw)
+{
+ struct rq *rq;
+
+ if (task_on_rq_queued(p))
+ return;
+
+ rq = task_rq(p);
+ if (p->dl.dl_non_contending) {
+ sub_running_bw(p->dl.dl_bw, &rq->dl);
+ p->dl.dl_non_contending = 0;
+ /*
+ * If the timer handler is currently running and the
+ * timer cannot be cancelled, inactive_task_timer()
+ * will see that dl_not_contending is not set, and
+ * will not touch the rq's active utilization,
+ * so we are still safe.
+ */
+ if (hrtimer_try_to_cancel(&p->dl.inactive_timer) == 1)
+ put_task_struct(p);
+ }
+ sub_rq_bw(p->dl.dl_bw, &rq->dl);
+ add_rq_bw(new_bw, &rq->dl);
+}
+
+/*
+ * The utilization of a task cannot be immediately removed from
+ * the rq active utilization (running_bw) when the task blocks.
+ * Instead, we have to wait for the so called "0-lag time".
+ *
+ * If a task blocks before the "0-lag time", a timer (the inactive
+ * timer) is armed, and running_bw is decreased when the timer
+ * fires.
+ *
+ * If the task wakes up again before the inactive timer fires,
+ * the timer is cancelled, whereas if the task wakes up after the
+ * inactive timer fired (and running_bw has been decreased) the
+ * task's utilization has to be added to running_bw again.
+ * A flag in the deadline scheduling entity (dl_non_contending)
+ * is used to avoid race conditions between the inactive timer handler
+ * and task wakeups.
+ *
+ * The following diagram shows how running_bw is updated. A task is
+ * "ACTIVE" when its utilization contributes to running_bw; an
+ * "ACTIVE contending" task is in the TASK_RUNNING state, while an
+ * "ACTIVE non contending" task is a blocked task for which the "0-lag time"
+ * has not passed yet. An "INACTIVE" task is a task for which the "0-lag"
+ * time already passed, which does not contribute to running_bw anymore.
+ * +------------------+
+ * wakeup | ACTIVE |
+ * +------------------>+ contending |
+ * | add_running_bw | |
+ * | +----+------+------+
+ * | | ^
+ * | dequeue | |
+ * +--------+-------+ | |
+ * | | t >= 0-lag | | wakeup
+ * | INACTIVE |<---------------+ |
+ * | | sub_running_bw | |
+ * +--------+-------+ | |
+ * ^ | |
+ * | t < 0-lag | |
+ * | | |
+ * | V |
+ * | +----+------+------+
+ * | sub_running_bw | ACTIVE |
+ * +-------------------+ |
+ * inactive timer | non contending |
+ * fired +------------------+
+ *
+ * The task_non_contending() function is invoked when a task
+ * blocks, and checks if the 0-lag time already passed or
+ * not (in the first case, it directly updates running_bw;
+ * in the second case, it arms the inactive timer).
+ *
+ * The task_contending() function is invoked when a task wakes
+ * up, and checks if the task is still in the "ACTIVE non contending"
+ * state or not (in the second case, it updates running_bw).
+ */
+static void task_non_contending(struct task_struct *p)
+{
+ struct sched_dl_entity *dl_se = &p->dl;
+ struct hrtimer *timer = &dl_se->inactive_timer;
+ struct dl_rq *dl_rq = dl_rq_of_se(dl_se);
+ struct rq *rq = rq_of_dl_rq(dl_rq);
+ s64 zerolag_time;
+
+ /*
+ * If this is a non-deadline task that has been boosted,
+ * do nothing
+ */
+ if (dl_se->dl_runtime == 0)
+ return;
+
+ WARN_ON(hrtimer_active(&dl_se->inactive_timer));
+ WARN_ON(dl_se->dl_non_contending);
+
+ zerolag_time = dl_se->deadline -
+ div64_long((dl_se->runtime * dl_se->dl_period),
+ dl_se->dl_runtime);
+
+ /*
+ * Using relative times instead of the absolute "0-lag time"
+ * allows to simplify the code
+ */
+ zerolag_time -= rq_clock(rq);
+
+ /*
+ * If the "0-lag time" already passed, decrease the active
+ * utilization now, instead of starting a timer
+ */
+ if (zerolag_time < 0) {
+ if (dl_task(p))
+ sub_running_bw(dl_se->dl_bw, dl_rq);
+ if (!dl_task(p) || p->state == TASK_DEAD) {
+ struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
+
+ if (p->state == TASK_DEAD)
+ sub_rq_bw(p->dl.dl_bw, &rq->dl);
+ raw_spin_lock(&dl_b->lock);
+ __dl_clear(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
+ __dl_clear_params(p);
+ raw_spin_unlock(&dl_b->lock);
+ }
+
+ return;
+ }
+
+ dl_se->dl_non_contending = 1;
+ get_task_struct(p);
+ hrtimer_start(timer, ns_to_ktime(zerolag_time), HRTIMER_MODE_REL);
+}
+
+static void task_contending(struct sched_dl_entity *dl_se, int flags)
+{
+ struct dl_rq *dl_rq = dl_rq_of_se(dl_se);
+
+ /*
+ * If this is a non-deadline task that has been boosted,
+ * do nothing
+ */
+ if (dl_se->dl_runtime == 0)
+ return;
+
+ if (flags & ENQUEUE_MIGRATED)
+ add_rq_bw(dl_se->dl_bw, dl_rq);
+
+ if (dl_se->dl_non_contending) {
+ dl_se->dl_non_contending = 0;
+ /*
+ * If the timer handler is currently running and the
+ * timer cannot be cancelled, inactive_task_timer()
+ * will see that dl_not_contending is not set, and
+ * will not touch the rq's active utilization,
+ * so we are still safe.
+ */
+ if (hrtimer_try_to_cancel(&dl_se->inactive_timer) == 1)
+ put_task_struct(dl_task_of(dl_se));
+ } else {
+ /*
+ * Since "dl_non_contending" is not set, the
+ * task's utilization has already been removed from
+ * active utilization (either when the task blocked,
+ * when the "inactive timer" fired).
+ * So, add it back.
+ */
+ add_running_bw(dl_se->dl_bw, dl_rq);
+ }
+}
+
static inline int is_leftmost(struct task_struct *p, struct dl_rq *dl_rq)
{
struct sched_dl_entity *dl_se = &p->dl;
#else
init_dl_bw(&dl_rq->dl_bw);
#endif
+
+ dl_rq->running_bw = 0;
+ dl_rq->this_bw = 0;
+ init_dl_rq_bw_ratio(dl_rq);
}
#ifdef CONFIG_SMP
}
/*
- * When a -deadline entity is queued back on the runqueue, its runtime and
- * deadline might need updating.
+ * Revised wakeup rule [1]: For self-suspending tasks, rather then
+ * re-initializing task's runtime and deadline, the revised wakeup
+ * rule adjusts the task's runtime to avoid the task to overrun its
+ * density.
+ *
+ * Reasoning: a task may overrun the density if:
+ * runtime / (deadline - t) > dl_runtime / dl_deadline
+ *
+ * Therefore, runtime can be adjusted to:
+ * runtime = (dl_runtime / dl_deadline) * (deadline - t)
+ *
+ * In such way that runtime will be equal to the maximum density
+ * the task can use without breaking any rule.
+ *
+ * [1] Luca Abeni, Giuseppe Lipari, and Juri Lelli. 2015. Constant
+ * bandwidth server revisited. SIGBED Rev. 11, 4 (January 2015), 19-24.
+ */
+static void
+update_dl_revised_wakeup(struct sched_dl_entity *dl_se, struct rq *rq)
+{
+ u64 laxity = dl_se->deadline - rq_clock(rq);
+
+ /*
+ * If the task has deadline < period, and the deadline is in the past,
+ * it should already be throttled before this check.
+ *
+ * See update_dl_entity() comments for further details.
+ */
+ WARN_ON(dl_time_before(dl_se->deadline, rq_clock(rq)));
+
+ dl_se->runtime = (dl_se->dl_density * laxity) >> BW_SHIFT;
+}
+
+/*
+ * Regarding the deadline, a task with implicit deadline has a relative
+ * deadline == relative period. A task with constrained deadline has a
+ * relative deadline <= relative period.
+ *
+ * We support constrained deadline tasks. However, there are some restrictions
+ * applied only for tasks which do not have an implicit deadline. See
+ * update_dl_entity() to know more about such restrictions.
+ *
+ * The dl_is_implicit() returns true if the task has an implicit deadline.
+ */
+static inline bool dl_is_implicit(struct sched_dl_entity *dl_se)
+{
+ return dl_se->dl_deadline == dl_se->dl_period;
+}
+
+/*
+ * When a deadline entity is placed in the runqueue, its runtime and deadline
+ * might need to be updated. This is done by a CBS wake up rule. There are two
+ * different rules: 1) the original CBS; and 2) the Revisited CBS.
+ *
+ * When the task is starting a new period, the Original CBS is used. In this
+ * case, the runtime is replenished and a new absolute deadline is set.
+ *
+ * When a task is queued before the begin of the next period, using the
+ * remaining runtime and deadline could make the entity to overflow, see
+ * dl_entity_overflow() to find more about runtime overflow. When such case
+ * is detected, the runtime and deadline need to be updated.
+ *
+ * If the task has an implicit deadline, i.e., deadline == period, the Original
+ * CBS is applied. the runtime is replenished and a new absolute deadline is
+ * set, as in the previous cases.
+ *
+ * However, the Original CBS does not work properly for tasks with
+ * deadline < period, which are said to have a constrained deadline. By
+ * applying the Original CBS, a constrained deadline task would be able to run
+ * runtime/deadline in a period. With deadline < period, the task would
+ * overrun the runtime/period allowed bandwidth, breaking the admission test.
*
- * The policy here is that we update the deadline of the entity only if:
- * - the current deadline is in the past,
- * - using the remaining runtime with the current deadline would make
- * the entity exceed its bandwidth.
+ * In order to prevent this misbehave, the Revisited CBS is used for
+ * constrained deadline tasks when a runtime overflow is detected. In the
+ * Revisited CBS, rather than replenishing & setting a new absolute deadline,
+ * the remaining runtime of the task is reduced to avoid runtime overflow.
+ * Please refer to the comments update_dl_revised_wakeup() function to find
+ * more about the Revised CBS rule.
*/
static void update_dl_entity(struct sched_dl_entity *dl_se,
struct sched_dl_entity *pi_se)
if (dl_time_before(dl_se->deadline, rq_clock(rq)) ||
dl_entity_overflow(dl_se, pi_se, rq_clock(rq))) {
+
+ if (unlikely(!dl_is_implicit(dl_se) &&
+ !dl_time_before(dl_se->deadline, rq_clock(rq)) &&
+ !dl_se->dl_boosted)){
+ update_dl_revised_wakeup(dl_se, rq);
+ return;
+ }
+
dl_se->deadline = rq_clock(rq) + pi_se->dl_deadline;
dl_se->runtime = pi_se->dl_runtime;
}
* The task might have changed its scheduling policy to something
* different than SCHED_DEADLINE (through switched_from_dl()).
*/
- if (!dl_task(p)) {
- __dl_clear_params(p);
+ if (!dl_task(p))
goto unlock;
- }
/*
* The task might have been boosted by someone else and might be in the
if (unlikely(dl_se->dl_boosted || !start_dl_timer(p)))
return;
dl_se->dl_throttled = 1;
+ if (dl_se->runtime > 0)
+ dl_se->runtime = 0;
}
}
extern bool sched_rt_bandwidth_account(struct rt_rq *rt_rq);
+/*
+ * This function implements the GRUB accounting rule:
+ * according to the GRUB reclaiming algorithm, the runtime is
+ * not decreased as "dq = -dt", but as
+ * "dq = -max{u / Umax, (1 - Uinact - Uextra)} dt",
+ * where u is the utilization of the task, Umax is the maximum reclaimable
+ * utilization, Uinact is the (per-runqueue) inactive utilization, computed
+ * as the difference between the "total runqueue utilization" and the
+ * runqueue active utilization, and Uextra is the (per runqueue) extra
+ * reclaimable utilization.
+ * Since rq->dl.running_bw and rq->dl.this_bw contain utilizations
+ * multiplied by 2^BW_SHIFT, the result has to be shifted right by
+ * BW_SHIFT.
+ * Since rq->dl.bw_ratio contains 1 / Umax multipled by 2^RATIO_SHIFT,
+ * dl_bw is multiped by rq->dl.bw_ratio and shifted right by RATIO_SHIFT.
+ * Since delta is a 64 bit variable, to have an overflow its value
+ * should be larger than 2^(64 - 20 - 8), which is more than 64 seconds.
+ * So, overflow is not an issue here.
+ */
+u64 grub_reclaim(u64 delta, struct rq *rq, struct sched_dl_entity *dl_se)
+{
+ u64 u_inact = rq->dl.this_bw - rq->dl.running_bw; /* Utot - Uact */
+ u64 u_act;
+ u64 u_act_min = (dl_se->dl_bw * rq->dl.bw_ratio) >> RATIO_SHIFT;
+
+ /*
+ * Instead of computing max{u * bw_ratio, (1 - u_inact - u_extra)},
+ * we compare u_inact + rq->dl.extra_bw with
+ * 1 - (u * rq->dl.bw_ratio >> RATIO_SHIFT), because
+ * u_inact + rq->dl.extra_bw can be larger than
+ * 1 * (so, 1 - u_inact - rq->dl.extra_bw would be negative
+ * leading to wrong results)
+ */
+ if (u_inact + rq->dl.extra_bw > BW_UNIT - u_act_min)
+ u_act = u_act_min;
+ else
+ u_act = BW_UNIT - u_inact - rq->dl.extra_bw;
+
+ return (delta * u_act) >> BW_SHIFT;
+}
+
/*
* Update the current task's runtime statistics (provided it is still
* a -deadline task and has not been removed from the dl_rq).
sched_rt_avg_update(rq, delta_exec);
+ if (unlikely(dl_se->flags & SCHED_FLAG_RECLAIM))
+ delta_exec = grub_reclaim(delta_exec, rq, &curr->dl);
dl_se->runtime -= delta_exec;
throttle:
}
}
+static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
+{
+ struct sched_dl_entity *dl_se = container_of(timer,
+ struct sched_dl_entity,
+ inactive_timer);
+ struct task_struct *p = dl_task_of(dl_se);
+ struct rq_flags rf;
+ struct rq *rq;
+
+ rq = task_rq_lock(p, &rf);
+
+ if (!dl_task(p) || p->state == TASK_DEAD) {
+ struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
+
+ if (p->state == TASK_DEAD && dl_se->dl_non_contending) {
+ sub_running_bw(p->dl.dl_bw, dl_rq_of_se(&p->dl));
+ sub_rq_bw(p->dl.dl_bw, dl_rq_of_se(&p->dl));
+ dl_se->dl_non_contending = 0;
+ }
+
+ raw_spin_lock(&dl_b->lock);
+ __dl_clear(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
+ raw_spin_unlock(&dl_b->lock);
+ __dl_clear_params(p);
+
+ goto unlock;
+ }
+ if (dl_se->dl_non_contending == 0)
+ goto unlock;
+
+ sched_clock_tick();
+ update_rq_clock(rq);
+
+ sub_running_bw(dl_se->dl_bw, &rq->dl);
+ dl_se->dl_non_contending = 0;
+unlock:
+ task_rq_unlock(rq, p, &rf);
+ put_task_struct(p);
+
+ return HRTIMER_NORESTART;
+}
+
+void init_dl_inactive_task_timer(struct sched_dl_entity *dl_se)
+{
+ struct hrtimer *timer = &dl_se->inactive_timer;
+
+ hrtimer_init(timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+ timer->function = inactive_task_timer;
+}
+
#ifdef CONFIG_SMP
static void inc_dl_deadline(struct dl_rq *dl_rq, u64 deadline)
* parameters of the task might need updating. Otherwise,
* we want a replenishment of its runtime.
*/
- if (flags & ENQUEUE_WAKEUP)
+ if (flags & ENQUEUE_WAKEUP) {
+ task_contending(dl_se, flags);
update_dl_entity(dl_se, pi_se);
- else if (flags & ENQUEUE_REPLENISH)
+ } else if (flags & ENQUEUE_REPLENISH) {
replenish_dl_entity(dl_se, pi_se);
+ }
__enqueue_dl_entity(dl_se);
}
__dequeue_dl_entity(dl_se);
}
-static inline bool dl_is_constrained(struct sched_dl_entity *dl_se)
-{
- return dl_se->dl_deadline < dl_se->dl_period;
-}
-
static void enqueue_task_dl(struct rq *rq, struct task_struct *p, int flags)
{
struct task_struct *pi_task = rt_mutex_get_top_task(p);
* If that is the case, the task will be throttled and
* the replenishment timer will be set to the next period.
*/
- if (!p->dl.dl_throttled && dl_is_constrained(&p->dl))
+ if (!p->dl.dl_throttled && !dl_is_implicit(&p->dl))
dl_check_constrained_dl(&p->dl);
+ if (p->on_rq == TASK_ON_RQ_MIGRATING || flags & ENQUEUE_RESTORE) {
+ add_rq_bw(p->dl.dl_bw, &rq->dl);
+ add_running_bw(p->dl.dl_bw, &rq->dl);
+ }
+
/*
- * If p is throttled, we do nothing. In fact, if it exhausted
+ * If p is throttled, we do not enqueue it. In fact, if it exhausted
* its budget it needs a replenishment and, since it now is on
* its rq, the bandwidth timer callback (which clearly has not
* run yet) will take care of this.
+ * However, the active utilization does not depend on the fact
+ * that the task is on the runqueue or not (but depends on the
+ * task's state - in GRUB parlance, "inactive" vs "active contending").
+ * In other words, even if a task is throttled its utilization must
+ * be counted in the active utilization; hence, we need to call
+ * add_running_bw().
*/
- if (p->dl.dl_throttled && !(flags & ENQUEUE_REPLENISH))
+ if (p->dl.dl_throttled && !(flags & ENQUEUE_REPLENISH)) {
+ if (flags & ENQUEUE_WAKEUP)
+ task_contending(&p->dl, flags);
+
return;
+ }
enqueue_dl_entity(&p->dl, pi_se, flags);
{
update_curr_dl(rq);
__dequeue_task_dl(rq, p, flags);
+
+ if (p->on_rq == TASK_ON_RQ_MIGRATING || flags & DEQUEUE_SAVE) {
+ sub_running_bw(p->dl.dl_bw, &rq->dl);
+ sub_rq_bw(p->dl.dl_bw, &rq->dl);
+ }
+
+ /*
+ * This check allows to start the inactive timer (or to immediately
+ * decrease the active utilization, if needed) in two cases:
+ * when the task blocks and when it is terminating
+ * (p->state == TASK_DEAD). We can handle the two cases in the same
+ * way, because from GRUB's point of view the same thing is happening
+ * (the task moves from "active contending" to "active non contending"
+ * or "inactive")
+ */
+ if (flags & DEQUEUE_SLEEP)
+ task_non_contending(p);
}
/*
return cpu;
}
+static void migrate_task_rq_dl(struct task_struct *p)
+{
+ struct rq *rq;
+
+ if (p->state != TASK_WAKING)
+ return;
+
+ rq = task_rq(p);
+ /*
+ * Since p->state == TASK_WAKING, set_task_cpu() has been called
+ * from try_to_wake_up(). Hence, p->pi_lock is locked, but
+ * rq->lock is not... So, lock it
+ */
+ raw_spin_lock(&rq->lock);
+ if (p->dl.dl_non_contending) {
+ sub_running_bw(p->dl.dl_bw, &rq->dl);
+ p->dl.dl_non_contending = 0;
+ /*
+ * If the timer handler is currently running and the
+ * timer cannot be cancelled, inactive_task_timer()
+ * will see that dl_not_contending is not set, and
+ * will not touch the rq's active utilization,
+ * so we are still safe.
+ */
+ if (hrtimer_try_to_cancel(&p->dl.inactive_timer) == 1)
+ put_task_struct(p);
+ }
+ sub_rq_bw(p->dl.dl_bw, &rq->dl);
+ raw_spin_unlock(&rq->lock);
+}
+
static void check_preempt_equal_dl(struct rq *rq, struct task_struct *p)
{
/*
*/
}
-static void task_dead_dl(struct task_struct *p)
-{
- struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
-
- /*
- * Since we are TASK_DEAD we won't slip out of the domain!
- */
- raw_spin_lock_irq(&dl_b->lock);
- /* XXX we should retain the bw until 0-lag */
- dl_b->total_bw -= p->dl.dl_bw;
- raw_spin_unlock_irq(&dl_b->lock);
-}
-
static void set_curr_task_dl(struct rq *rq)
{
struct task_struct *p = rq->curr;
* then possible that next_task has migrated.
*/
task = pick_next_pushable_dl_task(rq);
- if (task_cpu(next_task) == rq->cpu && task == next_task) {
+ if (task == next_task) {
/*
* The task is still there. We don't try
* again, some other cpu will pull it when ready.
}
deactivate_task(rq, next_task, 0);
+ sub_running_bw(next_task->dl.dl_bw, &rq->dl);
+ sub_rq_bw(next_task->dl.dl_bw, &rq->dl);
set_task_cpu(next_task, later_rq->cpu);
+ add_rq_bw(next_task->dl.dl_bw, &later_rq->dl);
+ add_running_bw(next_task->dl.dl_bw, &later_rq->dl);
activate_task(later_rq, next_task, 0);
ret = 1;
resched = true;
deactivate_task(src_rq, p, 0);
+ sub_running_bw(p->dl.dl_bw, &src_rq->dl);
+ sub_rq_bw(p->dl.dl_bw, &src_rq->dl);
set_task_cpu(p, this_cpu);
+ add_rq_bw(p->dl.dl_bw, &this_rq->dl);
+ add_running_bw(p->dl.dl_bw, &this_rq->dl);
activate_task(this_rq, p, 0);
dmin = p->dl.deadline;
* until we complete the update.
*/
raw_spin_lock(&src_dl_b->lock);
- __dl_clear(src_dl_b, p->dl.dl_bw);
+ __dl_clear(src_dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
raw_spin_unlock(&src_dl_b->lock);
}
static void switched_from_dl(struct rq *rq, struct task_struct *p)
{
/*
- * Start the deadline timer; if we switch back to dl before this we'll
- * continue consuming our current CBS slice. If we stay outside of
- * SCHED_DEADLINE until the deadline passes, the timer will reset the
- * task.
+ * task_non_contending() can start the "inactive timer" (if the 0-lag
+ * time is in the future). If the task switches back to dl before
+ * the "inactive timer" fires, it can continue to consume its current
+ * runtime using its current deadline. If it stays outside of
+ * SCHED_DEADLINE until the 0-lag time passes, inactive_task_timer()
+ * will reset the task parameters.
*/
- if (!start_dl_timer(p))
- __dl_clear_params(p);
+ if (task_on_rq_queued(p) && p->dl.dl_runtime)
+ task_non_contending(p);
+
+ if (!task_on_rq_queued(p))
+ sub_rq_bw(p->dl.dl_bw, &rq->dl);
+
+ /*
+ * We cannot use inactive_task_timer() to invoke sub_running_bw()
+ * at the 0-lag time, because the task could have been migrated
+ * while SCHED_OTHER in the meanwhile.
+ */
+ if (p->dl.dl_non_contending)
+ p->dl.dl_non_contending = 0;
/*
* Since this might be the only -deadline task on the rq,
*/
static void switched_to_dl(struct rq *rq, struct task_struct *p)
{
+ if (hrtimer_try_to_cancel(&p->dl.inactive_timer) == 1)
+ put_task_struct(p);
/* If p is not queued we will update its parameters at next wakeup. */
- if (!task_on_rq_queued(p))
- return;
+ if (!task_on_rq_queued(p)) {
+ add_rq_bw(p->dl.dl_bw, &rq->dl);
+ return;
+ }
/*
* If p is boosted we already updated its params in
* rt_mutex_setprio()->enqueue_task(..., ENQUEUE_REPLENISH),
#ifdef CONFIG_SMP
.select_task_rq = select_task_rq_dl,
+ .migrate_task_rq = migrate_task_rq_dl,
.set_cpus_allowed = set_cpus_allowed_dl,
.rq_online = rq_online_dl,
.rq_offline = rq_offline_dl,
.set_curr_task = set_curr_task_dl,
.task_tick = task_tick_dl,
.task_fork = task_fork_dl,
- .task_dead = task_dead_dl,
.prio_changed = prio_changed_dl,
.switched_from = switched_from_dl,
.update_curr = update_curr_dl,
};
+int sched_dl_global_validate(void)
+{
+ u64 runtime = global_rt_runtime();
+ u64 period = global_rt_period();
+ u64 new_bw = to_ratio(period, runtime);
+ struct dl_bw *dl_b;
+ int cpu, ret = 0;
+ unsigned long flags;
+
+ /*
+ * Here we want to check the bandwidth not being set to some
+ * value smaller than the currently allocated bandwidth in
+ * any of the root_domains.
+ *
+ * FIXME: Cycling on all the CPUs is overdoing, but simpler than
+ * cycling on root_domains... Discussion on different/better
+ * solutions is welcome!
+ */
+ for_each_possible_cpu(cpu) {
+ rcu_read_lock_sched();
+ dl_b = dl_bw_of(cpu);
+
+ raw_spin_lock_irqsave(&dl_b->lock, flags);
+ if (new_bw < dl_b->total_bw)
+ ret = -EBUSY;
+ raw_spin_unlock_irqrestore(&dl_b->lock, flags);
+
+ rcu_read_unlock_sched();
+
+ if (ret)
+ break;
+ }
+
+ return ret;
+}
+
+void init_dl_rq_bw_ratio(struct dl_rq *dl_rq)
+{
+ if (global_rt_runtime() == RUNTIME_INF) {
+ dl_rq->bw_ratio = 1 << RATIO_SHIFT;
+ dl_rq->extra_bw = 1 << BW_SHIFT;
+ } else {
+ dl_rq->bw_ratio = to_ratio(global_rt_runtime(),
+ global_rt_period()) >> (BW_SHIFT - RATIO_SHIFT);
+ dl_rq->extra_bw = to_ratio(global_rt_period(),
+ global_rt_runtime());
+ }
+}
+
+void sched_dl_do_global(void)
+{
+ u64 new_bw = -1;
+ struct dl_bw *dl_b;
+ int cpu;
+ unsigned long flags;
+
+ def_dl_bandwidth.dl_period = global_rt_period();
+ def_dl_bandwidth.dl_runtime = global_rt_runtime();
+
+ if (global_rt_runtime() != RUNTIME_INF)
+ new_bw = to_ratio(global_rt_period(), global_rt_runtime());
+
+ /*
+ * FIXME: As above...
+ */
+ for_each_possible_cpu(cpu) {
+ rcu_read_lock_sched();
+ dl_b = dl_bw_of(cpu);
+
+ raw_spin_lock_irqsave(&dl_b->lock, flags);
+ dl_b->bw = new_bw;
+ raw_spin_unlock_irqrestore(&dl_b->lock, flags);
+
+ rcu_read_unlock_sched();
+ init_dl_rq_bw_ratio(&cpu_rq(cpu)->dl);
+ }
+}
+
+/*
+ * We must be sure that accepting a new task (or allowing changing the
+ * parameters of an existing one) is consistent with the bandwidth
+ * constraints. If yes, this function also accordingly updates the currently
+ * allocated bandwidth to reflect the new situation.
+ *
+ * This function is called while holding p's rq->lock.
+ */
+int sched_dl_overflow(struct task_struct *p, int policy,
+ const struct sched_attr *attr)
+{
+ struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
+ u64 period = attr->sched_period ?: attr->sched_deadline;
+ u64 runtime = attr->sched_runtime;
+ u64 new_bw = dl_policy(policy) ? to_ratio(period, runtime) : 0;
+ int cpus, err = -1;
+
+ /* !deadline task may carry old deadline bandwidth */
+ if (new_bw == p->dl.dl_bw && task_has_dl_policy(p))
+ return 0;
+
+ /*
+ * Either if a task, enters, leave, or stays -deadline but changes
+ * its parameters, we may need to update accordingly the total
+ * allocated bandwidth of the container.
+ */
+ raw_spin_lock(&dl_b->lock);
+ cpus = dl_bw_cpus(task_cpu(p));
+ if (dl_policy(policy) && !task_has_dl_policy(p) &&
+ !__dl_overflow(dl_b, cpus, 0, new_bw)) {
+ if (hrtimer_active(&p->dl.inactive_timer))
+ __dl_clear(dl_b, p->dl.dl_bw, cpus);
+ __dl_add(dl_b, new_bw, cpus);
+ err = 0;
+ } else if (dl_policy(policy) && task_has_dl_policy(p) &&
+ !__dl_overflow(dl_b, cpus, p->dl.dl_bw, new_bw)) {
+ /*
+ * XXX this is slightly incorrect: when the task
+ * utilization decreases, we should delay the total
+ * utilization change until the task's 0-lag point.
+ * But this would require to set the task's "inactive
+ * timer" when the task is not inactive.
+ */
+ __dl_clear(dl_b, p->dl.dl_bw, cpus);
+ __dl_add(dl_b, new_bw, cpus);
+ dl_change_utilization(p, new_bw);
+ err = 0;
+ } else if (!dl_policy(policy) && task_has_dl_policy(p)) {
+ /*
+ * Do not decrease the total deadline utilization here,
+ * switched_from_dl() will take care to do it at the correct
+ * (0-lag) time.
+ */
+ err = 0;
+ }
+ raw_spin_unlock(&dl_b->lock);
+
+ return err;
+}
+
+/*
+ * This function initializes the sched_dl_entity of a newly becoming
+ * SCHED_DEADLINE task.
+ *
+ * Only the static values are considered here, the actual runtime and the
+ * absolute deadline will be properly calculated when the task is enqueued
+ * for the first time with its new policy.
+ */
+void __setparam_dl(struct task_struct *p, const struct sched_attr *attr)
+{
+ struct sched_dl_entity *dl_se = &p->dl;
+
+ dl_se->dl_runtime = attr->sched_runtime;
+ dl_se->dl_deadline = attr->sched_deadline;
+ dl_se->dl_period = attr->sched_period ?: dl_se->dl_deadline;
+ dl_se->flags = attr->sched_flags;
+ dl_se->dl_bw = to_ratio(dl_se->dl_period, dl_se->dl_runtime);
+ dl_se->dl_density = to_ratio(dl_se->dl_deadline, dl_se->dl_runtime);
+}
+
+void __getparam_dl(struct task_struct *p, struct sched_attr *attr)
+{
+ struct sched_dl_entity *dl_se = &p->dl;
+
+ attr->sched_priority = p->rt_priority;
+ attr->sched_runtime = dl_se->dl_runtime;
+ attr->sched_deadline = dl_se->dl_deadline;
+ attr->sched_period = dl_se->dl_period;
+ attr->sched_flags = dl_se->flags;
+}
+
+/*
+ * This function validates the new parameters of a -deadline task.
+ * We ask for the deadline not being zero, and greater or equal
+ * than the runtime, as well as the period of being zero or
+ * greater than deadline. Furthermore, we have to be sure that
+ * user parameters are above the internal resolution of 1us (we
+ * check sched_runtime only since it is always the smaller one) and
+ * below 2^63 ns (we have to check both sched_deadline and
+ * sched_period, as the latter can be zero).
+ */
+bool __checkparam_dl(const struct sched_attr *attr)
+{
+ /* deadline != 0 */
+ if (attr->sched_deadline == 0)
+ return false;
+
+ /*
+ * Since we truncate DL_SCALE bits, make sure we're at least
+ * that big.
+ */
+ if (attr->sched_runtime < (1ULL << DL_SCALE))
+ return false;
+
+ /*
+ * Since we use the MSB for wrap-around and sign issues, make
+ * sure it's not set (mind that period can be equal to zero).
+ */
+ if (attr->sched_deadline & (1ULL << 63) ||
+ attr->sched_period & (1ULL << 63))
+ return false;
+
+ /* runtime <= deadline <= period (if period != 0) */
+ if ((attr->sched_period != 0 &&
+ attr->sched_period < attr->sched_deadline) ||
+ attr->sched_deadline < attr->sched_runtime)
+ return false;
+
+ return true;
+}
+
+/*
+ * This function clears the sched_dl_entity static params.
+ */
+void __dl_clear_params(struct task_struct *p)
+{
+ struct sched_dl_entity *dl_se = &p->dl;
+
+ dl_se->dl_runtime = 0;
+ dl_se->dl_deadline = 0;
+ dl_se->dl_period = 0;
+ dl_se->flags = 0;
+ dl_se->dl_bw = 0;
+ dl_se->dl_density = 0;
+
+ dl_se->dl_throttled = 0;
+ dl_se->dl_yielded = 0;
+ dl_se->dl_non_contending = 0;
+}
+
+bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr)
+{
+ struct sched_dl_entity *dl_se = &p->dl;
+
+ if (dl_se->dl_runtime != attr->sched_runtime ||
+ dl_se->dl_deadline != attr->sched_deadline ||
+ dl_se->dl_period != attr->sched_period ||
+ dl_se->flags != attr->sched_flags)
+ return true;
+
+ return false;
+}
+
+#ifdef CONFIG_SMP
+int dl_task_can_attach(struct task_struct *p, const struct cpumask *cs_cpus_allowed)
+{
+ unsigned int dest_cpu = cpumask_any_and(cpu_active_mask,
+ cs_cpus_allowed);
+ struct dl_bw *dl_b;
+ bool overflow;
+ int cpus, ret;
+ unsigned long flags;
+
+ rcu_read_lock_sched();
+ dl_b = dl_bw_of(dest_cpu);
+ raw_spin_lock_irqsave(&dl_b->lock, flags);
+ cpus = dl_bw_cpus(dest_cpu);
+ overflow = __dl_overflow(dl_b, cpus, 0, p->dl.dl_bw);
+ if (overflow)
+ ret = -EBUSY;
+ else {
+ /*
+ * We reserve space for this task in the destination
+ * root_domain, as we can't fail after this point.
+ * We will free resources in the source root_domain
+ * later on (see set_cpus_allowed_dl()).
+ */
+ __dl_add(dl_b, p->dl.dl_bw, cpus);
+ ret = 0;
+ }
+ raw_spin_unlock_irqrestore(&dl_b->lock, flags);
+ rcu_read_unlock_sched();
+ return ret;
+}
+
+int dl_cpuset_cpumask_can_shrink(const struct cpumask *cur,
+ const struct cpumask *trial)
+{
+ int ret = 1, trial_cpus;
+ struct dl_bw *cur_dl_b;
+ unsigned long flags;
+
+ rcu_read_lock_sched();
+ cur_dl_b = dl_bw_of(cpumask_any(cur));
+ trial_cpus = cpumask_weight(trial);
+
+ raw_spin_lock_irqsave(&cur_dl_b->lock, flags);
+ if (cur_dl_b->bw != -1 &&
+ cur_dl_b->bw * trial_cpus < cur_dl_b->total_bw)
+ ret = 0;
+ raw_spin_unlock_irqrestore(&cur_dl_b->lock, flags);
+ rcu_read_unlock_sched();
+ return ret;
+}
+
+bool dl_cpu_busy(unsigned int cpu)
+{
+ unsigned long flags;
+ struct dl_bw *dl_b;
+ bool overflow;
+ int cpus;
+
+ rcu_read_lock_sched();
+ dl_b = dl_bw_of(cpu);
+ raw_spin_lock_irqsave(&dl_b->lock, flags);
+ cpus = dl_bw_cpus(cpu);
+ overflow = __dl_overflow(dl_b, cpus, 0, 0);
+ raw_spin_unlock_irqrestore(&dl_b->lock, flags);
+ rcu_read_unlock_sched();
+ return overflow;
+}
+#endif
+
#ifdef CONFIG_SCHED_DEBUG
extern void print_dl_rq(struct seq_file *m, int cpu, struct dl_rq *dl_rq);
#define P(x) \
SEQ_printf(m, " .%-30s: %Ld\n", #x, (long long)(rt_rq->x))
+#define PU(x) \
+ SEQ_printf(m, " .%-30s: %lu\n", #x, (unsigned long)(rt_rq->x))
#define PN(x) \
SEQ_printf(m, " .%-30s: %Ld.%06ld\n", #x, SPLIT_NS(rt_rq->x))
- P(rt_nr_running);
+ PU(rt_nr_running);
+#ifdef CONFIG_SMP
+ PU(rt_nr_migratory);
+#endif
P(rt_throttled);
PN(rt_time);
PN(rt_runtime);
#undef PN
+#undef PU
#undef P
}
struct dl_bw *dl_bw;
SEQ_printf(m, "\ndl_rq[%d]:\n", cpu);
- SEQ_printf(m, " .%-30s: %ld\n", "dl_nr_running", dl_rq->dl_nr_running);
+
+#define PU(x) \
+ SEQ_printf(m, " .%-30s: %lu\n", #x, (unsigned long)(dl_rq->x))
+
+ PU(dl_nr_running);
#ifdef CONFIG_SMP
+ PU(dl_nr_migratory);
dl_bw = &cpu_rq(cpu)->rd->dl_bw;
#else
dl_bw = &dl_rq->dl_bw;
#endif
SEQ_printf(m, " .%-30s: %lld\n", "dl_bw->bw", dl_bw->bw);
SEQ_printf(m, " .%-30s: %lld\n", "dl_bw->total_bw", dl_bw->total_bw);
+
+#undef PU
}
extern __read_mostly int sched_clock_running;
}
/* Iterate thr' all leaf cfs_rq's on a runqueue */
-#define for_each_leaf_cfs_rq(rq, cfs_rq) \
- list_for_each_entry_rcu(cfs_rq, &rq->leaf_cfs_rq_list, leaf_cfs_rq_list)
+#define for_each_leaf_cfs_rq_safe(rq, cfs_rq, pos) \
+ list_for_each_entry_safe(cfs_rq, pos, &rq->leaf_cfs_rq_list, \
+ leaf_cfs_rq_list)
/* Do the two (enqueued) entities belong to the same group ? */
static inline struct cfs_rq *
{
}
-#define for_each_leaf_cfs_rq(rq, cfs_rq) \
- for (cfs_rq = &rq->cfs; cfs_rq; cfs_rq = NULL)
+#define for_each_leaf_cfs_rq_safe(rq, cfs_rq, pos) \
+ for (cfs_rq = &rq->cfs, pos = NULL; cfs_rq; cfs_rq = pos)
static inline struct sched_entity *parent_entity(struct sched_entity *se)
{
static unsigned long source_load(int cpu, int type);
static unsigned long target_load(int cpu, int type);
static unsigned long capacity_of(int cpu);
-static long effective_load(struct task_group *tg, int cpu, long wl, long wg);
/* Cached statistics for all CPUs within a node */
struct numa_stats {
return;
- down_read(&mm->mmap_sem);
+ if (!down_read_trylock(&mm->mmap_sem))
+ return;
vma = find_vma(mm, start);
if (!vma) {
reset_ptenuma_scan(p);
}
}
}
+
+/*
+ * Can a task be moved from prev_cpu to this_cpu without causing a load
+ * imbalance that would trigger the load balancer?
+ */
+static inline bool numa_wake_affine(struct sched_domain *sd,
+ struct task_struct *p, int this_cpu,
+ int prev_cpu, int sync)
+{
+ struct numa_stats prev_load, this_load;
+ s64 this_eff_load, prev_eff_load;
+
+ update_numa_stats(&prev_load, cpu_to_node(prev_cpu));
+ update_numa_stats(&this_load, cpu_to_node(this_cpu));
+
+ /*
+ * If sync wakeup then subtract the (maximum possible)
+ * effect of the currently running task from the load
+ * of the current CPU:
+ */
+ if (sync) {
+ unsigned long current_load = task_h_load(current);
+
+ if (this_load.load > current_load)
+ this_load.load -= current_load;
+ else
+ this_load.load = 0;
+ }
+
+ /*
+ * In low-load situations, where this_cpu's node is idle due to the
+ * sync cause above having dropped this_load.load to 0, move the task.
+ * Moving to an idle socket will not create a bad imbalance.
+ *
+ * Otherwise check if the nodes are near enough in load to allow this
+ * task to be woken on this_cpu's node.
+ */
+ if (this_load.load > 0) {
+ unsigned long task_load = task_h_load(p);
+
+ this_eff_load = 100;
+ this_eff_load *= prev_load.compute_capacity;
+
+ prev_eff_load = 100 + (sd->imbalance_pct - 100) / 2;
+ prev_eff_load *= this_load.compute_capacity;
+
+ this_eff_load *= this_load.load + task_load;
+ prev_eff_load *= prev_load.load - task_load;
+
+ return this_eff_load <= prev_eff_load;
+ }
+
+ return true;
+}
#else
static void task_tick_numa(struct rq *rq, struct task_struct *curr)
{
static inline void account_numa_dequeue(struct rq *rq, struct task_struct *p)
{
}
+
+#ifdef CONFIG_SMP
+static inline bool numa_wake_affine(struct sched_domain *sd,
+ struct task_struct *p, int this_cpu,
+ int prev_cpu, int sync)
+{
+ return true;
+}
+#endif /* !SMP */
#endif /* CONFIG_NUMA_BALANCING */
static void
/*
* Step 2: update *_avg.
*/
- sa->load_avg = div_u64(sa->load_sum, LOAD_AVG_MAX);
+ sa->load_avg = div_u64(sa->load_sum, LOAD_AVG_MAX - 1024 + sa->period_contrib);
if (cfs_rq) {
cfs_rq->runnable_load_avg =
- div_u64(cfs_rq->runnable_load_sum, LOAD_AVG_MAX);
+ div_u64(cfs_rq->runnable_load_sum, LOAD_AVG_MAX - 1024 + sa->period_contrib);
}
- sa->util_avg = sa->util_sum / LOAD_AVG_MAX;
+ sa->util_avg = sa->util_sum / (LOAD_AVG_MAX - 1024 + sa->period_contrib);
return 1;
}
* differential update where we store the last value we propagated. This in
* turn allows skipping updates if the differential is 'small'.
*
- * Updating tg's load_avg is necessary before update_cfs_share() (which is
- * done) and effective_load() (which is not done because it is too costly).
+ * Updating tg's load_avg is necessary before update_cfs_share().
*/
static inline void update_tg_load_avg(struct cfs_rq *cfs_rq, int force)
{
hrtimer_cancel(&cfs_b->slack_timer);
}
+/*
+ * Both these cpu hotplug callbacks race against unregister_fair_sched_group()
+ *
+ * The race is harmless, since modifying bandwidth settings of unhooked group
+ * bits doesn't do much.
+ */
+
+/* cpu online calback */
static void __maybe_unused update_runtime_enabled(struct rq *rq)
{
- struct cfs_rq *cfs_rq;
+ struct task_group *tg;
- for_each_leaf_cfs_rq(rq, cfs_rq) {
- struct cfs_bandwidth *cfs_b = &cfs_rq->tg->cfs_bandwidth;
+ lockdep_assert_held(&rq->lock);
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(tg, &task_groups, list) {
+ struct cfs_bandwidth *cfs_b = &tg->cfs_bandwidth;
+ struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of(rq)];
raw_spin_lock(&cfs_b->lock);
cfs_rq->runtime_enabled = cfs_b->quota != RUNTIME_INF;
raw_spin_unlock(&cfs_b->lock);
}
+ rcu_read_unlock();
}
+/* cpu offline callback */
static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq)
{
- struct cfs_rq *cfs_rq;
+ struct task_group *tg;
+
+ lockdep_assert_held(&rq->lock);
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(tg, &task_groups, list) {
+ struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of(rq)];
- for_each_leaf_cfs_rq(rq, cfs_rq) {
if (!cfs_rq->runtime_enabled)
continue;
if (cfs_rq_throttled(cfs_rq))
unthrottle_cfs_rq(cfs_rq);
}
+ rcu_read_unlock();
}
#else /* CONFIG_CFS_BANDWIDTH */
return 0;
}
-#ifdef CONFIG_FAIR_GROUP_SCHED
-/*
- * effective_load() calculates the load change as seen from the root_task_group
- *
- * Adding load to a group doesn't make a group heavier, but can cause movement
- * of group shares between cpus. Assuming the shares were perfectly aligned one
- * can calculate the shift in shares.
- *
- * Calculate the effective load difference if @wl is added (subtracted) to @tg
- * on this @cpu and results in a total addition (subtraction) of @wg to the
- * total group weight.
- *
- * Given a runqueue weight distribution (rw_i) we can compute a shares
- * distribution (s_i) using:
- *
- * s_i = rw_i / \Sum rw_j (1)
- *
- * Suppose we have 4 CPUs and our @tg is a direct child of the root group and
- * has 7 equal weight tasks, distributed as below (rw_i), with the resulting
- * shares distribution (s_i):
- *
- * rw_i = { 2, 4, 1, 0 }
- * s_i = { 2/7, 4/7, 1/7, 0 }
- *
- * As per wake_affine() we're interested in the load of two CPUs (the CPU the
- * task used to run on and the CPU the waker is running on), we need to
- * compute the effect of waking a task on either CPU and, in case of a sync
- * wakeup, compute the effect of the current task going to sleep.
- *
- * So for a change of @wl to the local @cpu with an overall group weight change
- * of @wl we can compute the new shares distribution (s'_i) using:
- *
- * s'_i = (rw_i + @wl) / (@wg + \Sum rw_j) (2)
- *
- * Suppose we're interested in CPUs 0 and 1, and want to compute the load
- * differences in waking a task to CPU 0. The additional task changes the
- * weight and shares distributions like:
- *
- * rw'_i = { 3, 4, 1, 0 }
- * s'_i = { 3/8, 4/8, 1/8, 0 }
- *
- * We can then compute the difference in effective weight by using:
- *
- * dw_i = S * (s'_i - s_i) (3)
- *
- * Where 'S' is the group weight as seen by its parent.
- *
- * Therefore the effective change in loads on CPU 0 would be 5/56 (3/8 - 2/7)
- * times the weight of the group. The effect on CPU 1 would be -4/56 (4/8 -
- * 4/7) times the weight of the group.
- */
-static long effective_load(struct task_group *tg, int cpu, long wl, long wg)
-{
- struct sched_entity *se = tg->se[cpu];
-
- if (!tg->parent) /* the trivial, non-cgroup case */
- return wl;
-
- for_each_sched_entity(se) {
- struct cfs_rq *cfs_rq = se->my_q;
- long W, w = cfs_rq_load_avg(cfs_rq);
-
- tg = cfs_rq->tg;
-
- /*
- * W = @wg + \Sum rw_j
- */
- W = wg + atomic_long_read(&tg->load_avg);
-
- /* Ensure \Sum rw_j >= rw_i */
- W -= cfs_rq->tg_load_avg_contrib;
- W += w;
-
- /*
- * w = rw_i + @wl
- */
- w += wl;
-
- /*
- * wl = S * s'_i; see (2)
- */
- if (W > 0 && w < W)
- wl = (w * (long)scale_load_down(tg->shares)) / W;
- else
- wl = scale_load_down(tg->shares);
-
- /*
- * Per the above, wl is the new se->load.weight value; since
- * those are clipped to [MIN_SHARES, ...) do so now. See
- * calc_cfs_shares().
- */
- if (wl < MIN_SHARES)
- wl = MIN_SHARES;
-
- /*
- * wl = dw_i = S * (s'_i - s_i); see (3)
- */
- wl -= se->avg.load_avg;
-
- /*
- * Recursively apply this logic to all parent groups to compute
- * the final effective load change on the root group. Since
- * only the @tg group gets extra weight, all parent groups can
- * only redistribute existing shares. @wl is the shift in shares
- * resulting from this level per the above.
- */
- wg = 0;
- }
-
- return wl;
-}
-#else
-
-static long effective_load(struct task_group *tg, int cpu, long wl, long wg)
-{
- return wl;
-}
-
-#endif
-
static void record_wakee(struct task_struct *p)
{
/*
static int wake_affine(struct sched_domain *sd, struct task_struct *p,
int prev_cpu, int sync)
{
- s64 this_load, load;
- s64 this_eff_load, prev_eff_load;
- int idx, this_cpu;
- struct task_group *tg;
- unsigned long weight;
- int balanced;
-
- idx = sd->wake_idx;
- this_cpu = smp_processor_id();
- load = source_load(prev_cpu, idx);
- this_load = target_load(this_cpu, idx);
-
- /*
- * If sync wakeup then subtract the (maximum possible)
- * effect of the currently running task from the load
- * of the current CPU:
- */
- if (sync) {
- tg = task_group(current);
- weight = current->se.avg.load_avg;
-
- this_load += effective_load(tg, this_cpu, -weight, -weight);
- load += effective_load(tg, prev_cpu, 0, -weight);
- }
-
- tg = task_group(p);
- weight = p->se.avg.load_avg;
+ int this_cpu = smp_processor_id();
+ bool affine = false;
/*
- * In low-load situations, where prev_cpu is idle and this_cpu is idle
- * due to the sync cause above having dropped this_load to 0, we'll
- * always have an imbalance, but there's really nothing you can do
- * about that, so that's good too.
- *
- * Otherwise check if either cpus are near enough in load to allow this
- * task to be woken on this_cpu.
+ * Common case: CPUs are in the same socket, and select_idle_sibling()
+ * will do its thing regardless of what we return:
*/
- this_eff_load = 100;
- this_eff_load *= capacity_of(prev_cpu);
-
- prev_eff_load = 100 + (sd->imbalance_pct - 100) / 2;
- prev_eff_load *= capacity_of(this_cpu);
-
- if (this_load > 0) {
- this_eff_load *= this_load +
- effective_load(tg, this_cpu, weight, weight);
-
- prev_eff_load *= load + effective_load(tg, prev_cpu, 0, weight);
- }
-
- balanced = this_eff_load <= prev_eff_load;
+ if (cpus_share_cache(prev_cpu, this_cpu))
+ affine = true;
+ else
+ affine = numa_wake_affine(sd, p, this_cpu, prev_cpu, sync);
schedstat_inc(p->se.statistics.nr_wakeups_affine_attempts);
+ if (affine) {
+ schedstat_inc(sd->ttwu_move_affine);
+ schedstat_inc(p->se.statistics.nr_wakeups_affine);
+ }
- if (!balanced)
- return 0;
-
- schedstat_inc(sd->ttwu_move_affine);
- schedstat_inc(p->se.statistics.nr_wakeups_affine);
-
- return 1;
+ return affine;
}
static inline int task_util(struct task_struct *p);
int i;
/* Skip over this group if it has no CPUs allowed */
- if (!cpumask_intersects(sched_group_cpus(group),
+ if (!cpumask_intersects(sched_group_span(group),
&p->cpus_allowed))
continue;
local_group = cpumask_test_cpu(this_cpu,
- sched_group_cpus(group));
+ sched_group_span(group));
/*
* Tally up the load of all CPUs in the group and find
runnable_load = 0;
max_spare_cap = 0;
- for_each_cpu(i, sched_group_cpus(group)) {
+ for_each_cpu(i, sched_group_span(group)) {
/* Bias balancing toward cpus of our domain */
if (local_group)
load = source_load(i, load_idx);
/* Check if we have any choice: */
if (group->group_weight == 1)
- return cpumask_first(sched_group_cpus(group));
+ return cpumask_first(sched_group_span(group));
/* Traverse only the allowed CPUs */
- for_each_cpu_and(i, sched_group_cpus(group), &p->cpus_allowed) {
+ for_each_cpu_and(i, sched_group_span(group), &p->cpus_allowed) {
if (idle_cpu(i)) {
struct rq *rq = cpu_rq(i);
struct cpuidle_state *idle = idle_get_state(rq);
return shallowest_idle_cpu != -1 ? shallowest_idle_cpu : least_loaded_cpu;
}
-/*
- * Implement a for_each_cpu() variant that starts the scan at a given cpu
- * (@start), and wraps around.
- *
- * This is used to scan for idle CPUs; such that not all CPUs looking for an
- * idle CPU find the same CPU. The down-side is that tasks tend to cycle
- * through the LLC domain.
- *
- * Especially tbench is found sensitive to this.
- */
-
-static int cpumask_next_wrap(int n, const struct cpumask *mask, int start, int *wrapped)
-{
- int next;
-
-again:
- next = find_next_bit(cpumask_bits(mask), nr_cpumask_bits, n+1);
-
- if (*wrapped) {
- if (next >= start)
- return nr_cpumask_bits;
- } else {
- if (next >= nr_cpumask_bits) {
- *wrapped = 1;
- n = -1;
- goto again;
- }
- }
-
- return next;
-}
-
-#define for_each_cpu_wrap(cpu, mask, start, wrap) \
- for ((wrap) = 0, (cpu) = (start)-1; \
- (cpu) = cpumask_next_wrap((cpu), (mask), (start), &(wrap)), \
- (cpu) < nr_cpumask_bits; )
-
#ifdef CONFIG_SCHED_SMT
static inline void set_idle_cores(int cpu, int val)
static int select_idle_core(struct task_struct *p, struct sched_domain *sd, int target)
{
struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_idle_mask);
- int core, cpu, wrap;
+ int core, cpu;
if (!static_branch_likely(&sched_smt_present))
return -1;
cpumask_and(cpus, sched_domain_span(sd), &p->cpus_allowed);
- for_each_cpu_wrap(core, cpus, target, wrap) {
+ for_each_cpu_wrap(core, cpus, target) {
bool idle = true;
for_each_cpu(cpu, cpu_smt_mask(core)) {
static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int target)
{
struct sched_domain *this_sd;
- u64 avg_cost, avg_idle = this_rq()->avg_idle;
+ u64 avg_cost, avg_idle;
u64 time, cost;
s64 delta;
- int cpu, wrap;
+ int cpu, nr = INT_MAX;
this_sd = rcu_dereference(*this_cpu_ptr(&sd_llc));
if (!this_sd)
return -1;
- avg_cost = this_sd->avg_scan_cost;
-
/*
* Due to large variance we need a large fuzz factor; hackbench in
* particularly is sensitive here.
*/
- if (sched_feat(SIS_AVG_CPU) && (avg_idle / 512) < avg_cost)
+ avg_idle = this_rq()->avg_idle / 512;
+ avg_cost = this_sd->avg_scan_cost + 1;
+
+ if (sched_feat(SIS_AVG_CPU) && avg_idle < avg_cost)
return -1;
+ if (sched_feat(SIS_PROP)) {
+ u64 span_avg = sd->span_weight * avg_idle;
+ if (span_avg > 4*avg_cost)
+ nr = div_u64(span_avg, avg_cost);
+ else
+ nr = 4;
+ }
+
time = local_clock();
- for_each_cpu_wrap(cpu, sched_domain_span(sd), target, wrap) {
+ for_each_cpu_wrap(cpu, sched_domain_span(sd), target) {
+ if (!--nr)
+ return -1;
if (!cpumask_test_cpu(cpu, &p->cpus_allowed))
continue;
if (idle_cpu(cpu))
if (affine_sd) {
sd = NULL; /* Prefer wake_affine over balance flags */
- if (cpu != prev_cpu && wake_affine(affine_sd, p, prev_cpu, sync))
+ if (cpu == prev_cpu)
+ goto pick_cpu;
+
+ if (wake_affine(affine_sd, p, prev_cpu, sync))
new_cpu = cpu;
}
if (!sd) {
+ pick_cpu:
if (sd_flag & SD_BALANCE_WAKE) /* XXX always ? */
new_cpu = select_idle_sibling(p, prev_cpu, new_cpu);
if (entity_is_task(se) && unlikely(task_of(se)->policy == SCHED_IDLE))
return;
- for_each_sched_entity(se)
+ for_each_sched_entity(se) {
+ if (SCHED_WARN_ON(!se->on_rq))
+ return;
cfs_rq_of(se)->last = se;
+ }
}
static void set_next_buddy(struct sched_entity *se)
if (entity_is_task(se) && unlikely(task_of(se)->policy == SCHED_IDLE))
return;
- for_each_sched_entity(se)
+ for_each_sched_entity(se) {
+ if (SCHED_WARN_ON(!se->on_rq))
+ return;
cfs_rq_of(se)->next = se;
+ }
}
static void set_skip_buddy(struct sched_entity *se)
if (dst_nid == p->numa_preferred_nid)
return 0;
+ /* Leaving a core idle is often worse than degrading locality. */
+ if (env->idle != CPU_NOT_IDLE)
+ return -1;
+
if (numa_group) {
src_faults = group_faults(p, src_nid);
dst_faults = group_faults(p, dst_nid);
}
#ifdef CONFIG_FAIR_GROUP_SCHED
+
+static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
+{
+ if (cfs_rq->load.weight)
+ return false;
+
+ if (cfs_rq->avg.load_sum)
+ return false;
+
+ if (cfs_rq->avg.util_sum)
+ return false;
+
+ if (cfs_rq->runnable_load_sum)
+ return false;
+
+ return true;
+}
+
static void update_blocked_averages(int cpu)
{
struct rq *rq = cpu_rq(cpu);
- struct cfs_rq *cfs_rq;
+ struct cfs_rq *cfs_rq, *pos;
struct rq_flags rf;
rq_lock_irqsave(rq, &rf);
* Iterates the task_group tree in a bottom up fashion, see
* list_add_leaf_cfs_rq() for details.
*/
- for_each_leaf_cfs_rq(rq, cfs_rq) {
+ for_each_leaf_cfs_rq_safe(rq, cfs_rq, pos) {
struct sched_entity *se;
/* throttled entities do not contribute to load */
se = cfs_rq->tg->se[cpu];
if (se && !skip_blocked_update(se))
update_load_avg(se, 0);
+
+ /*
+ * There can be a lot of idle CPU cgroups. Don't let fully
+ * decayed cfs_rqs linger on the list.
+ */
+ if (cfs_rq_is_decayed(cfs_rq))
+ list_del_leaf_cfs_rq(cfs_rq);
}
rq_unlock_irqrestore(rq, &rf);
}
* span the current group.
*/
- for_each_cpu(cpu, sched_group_cpus(sdg)) {
+ for_each_cpu(cpu, sched_group_span(sdg)) {
struct sched_group_capacity *sgc;
struct rq *rq = cpu_rq(cpu);
memset(sgs, 0, sizeof(*sgs));
- for_each_cpu_and(i, sched_group_cpus(group), env->cpus) {
+ for_each_cpu_and(i, sched_group_span(group), env->cpus) {
struct rq *rq = cpu_rq(i);
/* Bias balancing toward cpus of our domain */
struct sg_lb_stats *sgs = &tmp_sgs;
int local_group;
- local_group = cpumask_test_cpu(env->dst_cpu, sched_group_cpus(sg));
+ local_group = cpumask_test_cpu(env->dst_cpu, sched_group_span(sg));
if (local_group) {
sds->local = sg;
sgs = local;
unsigned long busiest_load = 0, busiest_capacity = 1;
int i;
- for_each_cpu_and(i, sched_group_cpus(group), env->cpus) {
+ for_each_cpu_and(i, sched_group_span(group), env->cpus) {
unsigned long capacity, wl;
enum fbq_type rt;
static int should_we_balance(struct lb_env *env)
{
struct sched_group *sg = env->sd->groups;
- struct cpumask *sg_cpus, *sg_mask;
int cpu, balance_cpu = -1;
/*
if (env->idle == CPU_NEWLY_IDLE)
return 1;
- sg_cpus = sched_group_cpus(sg);
- sg_mask = sched_group_mask(sg);
/* Try to find first idle cpu */
- for_each_cpu_and(cpu, sg_cpus, env->cpus) {
- if (!cpumask_test_cpu(cpu, sg_mask) || !idle_cpu(cpu))
+ for_each_cpu_and(cpu, group_balance_mask(sg), env->cpus) {
+ if (!idle_cpu(cpu))
continue;
balance_cpu = cpu;
.sd = sd,
.dst_cpu = this_cpu,
.dst_rq = this_rq,
- .dst_grpmask = sched_group_cpus(sd->groups),
+ .dst_grpmask = sched_group_span(sd->groups),
.idle = idle,
.loop_break = sched_nr_migrate_break,
.cpus = cpus,
if (!cpu_active(cpu))
return;
+ /* Spare idle load balancing on CPUs that don't want to be disturbed: */
+ if (!is_housekeeping_cpu(cpu))
+ return;
+
if (test_bit(NOHZ_TICK_STOPPED, nohz_flags(cpu)))
return;
#ifdef CONFIG_SCHED_DEBUG
void print_cfs_stats(struct seq_file *m, int cpu)
{
- struct cfs_rq *cfs_rq;
+ struct cfs_rq *cfs_rq, *pos;
rcu_read_lock();
- for_each_leaf_cfs_rq(cpu_rq(cpu), cfs_rq)
+ for_each_leaf_cfs_rq_safe(cpu_rq(cpu), cfs_rq, pos)
print_cfs_rq(m, cpu, cfs_rq);
rcu_read_unlock();
}
* When doing wakeups, attempt to limit superfluous scans of the LLC domain.
*/
SCHED_FEAT(SIS_AVG_CPU, false)
+SCHED_FEAT(SIS_PROP, true)
/*
* Issue a WARN when we do multiple update_rq_clock() calls
SCHED_FEAT(RT_PUSH_IPI, true)
#endif
-SCHED_FEAT(FORCE_SD_OVERLAP, false)
SCHED_FEAT(RT_RUNTIME_SHARE, true)
SCHED_FEAT(LB_MIN, false)
SCHED_FEAT(ATTACH_AGE_LOAD, true)
*/
__current_set_polling();
+ quiet_vmstat();
tick_nohz_idle_enter();
while (!need_resched()) {
* load-average relies on per-cpu sampling from the tick, it is affected by
* NO_HZ.
*
- * The basic idea is to fold the nr_active delta into a global idle-delta upon
+ * The basic idea is to fold the nr_active delta into a global NO_HZ-delta upon
* entering NO_HZ state such that we can include this as an 'extra' cpu delta
* when we read the global state.
*
* - When we go NO_HZ idle during the window, we can negate our sample
* contribution, causing under-accounting.
*
- * We avoid this by keeping two idle-delta counters and flipping them
+ * We avoid this by keeping two NO_HZ-delta counters and flipping them
* when the window starts, thus separating old and new NO_HZ load.
*
* The only trick is the slight shift in index flip for read vs write.
* r:0 0 1 1 0 0 1 1 0
* w:0 1 1 0 0 1 1 0 0
*
- * This ensures we'll fold the old idle contribution in this window while
+ * This ensures we'll fold the old NO_HZ contribution in this window while
* accumlating the new one.
*
- * - When we wake up from NO_HZ idle during the window, we push up our
+ * - When we wake up from NO_HZ during the window, we push up our
* contribution, since we effectively move our sample point to a known
* busy state.
*
* This is solved by pushing the window forward, and thus skipping the
- * sample, for this cpu (effectively using the idle-delta for this cpu which
+ * sample, for this cpu (effectively using the NO_HZ-delta for this cpu which
* was in effect at the time the window opened). This also solves the issue
- * of having to deal with a cpu having been in NOHZ idle for multiple
- * LOAD_FREQ intervals.
+ * of having to deal with a cpu having been in NO_HZ for multiple LOAD_FREQ
+ * intervals.
*
* When making the ILB scale, we should try to pull this in as well.
*/
-static atomic_long_t calc_load_idle[2];
+static atomic_long_t calc_load_nohz[2];
static int calc_load_idx;
static inline int calc_load_write_idx(void)
/*
* If the folding window started, make sure we start writing in the
- * next idle-delta.
+ * next NO_HZ-delta.
*/
if (!time_before(jiffies, READ_ONCE(calc_load_update)))
idx++;
return calc_load_idx & 1;
}
-void calc_load_enter_idle(void)
+void calc_load_nohz_start(void)
{
struct rq *this_rq = this_rq();
long delta;
/*
- * We're going into NOHZ mode, if there's any pending delta, fold it
- * into the pending idle delta.
+ * We're going into NO_HZ mode, if there's any pending delta, fold it
+ * into the pending NO_HZ delta.
*/
delta = calc_load_fold_active(this_rq, 0);
if (delta) {
int idx = calc_load_write_idx();
- atomic_long_add(delta, &calc_load_idle[idx]);
+ atomic_long_add(delta, &calc_load_nohz[idx]);
}
}
-void calc_load_exit_idle(void)
+void calc_load_nohz_stop(void)
{
struct rq *this_rq = this_rq();
this_rq->calc_load_update += LOAD_FREQ;
}
-static long calc_load_fold_idle(void)
+static long calc_load_nohz_fold(void)
{
int idx = calc_load_read_idx();
long delta = 0;
- if (atomic_long_read(&calc_load_idle[idx]))
- delta = atomic_long_xchg(&calc_load_idle[idx], 0);
+ if (atomic_long_read(&calc_load_nohz[idx]))
+ delta = atomic_long_xchg(&calc_load_nohz[idx], 0);
return delta;
}
/*
* NO_HZ can leave us missing all per-cpu ticks calling
- * calc_load_account_active(), but since an idle CPU folds its delta into
- * calc_load_tasks_idle per calc_load_account_idle(), all we need to do is fold
- * in the pending idle delta if our idle period crossed a load cycle boundary.
+ * calc_load_fold_active(), but since a NO_HZ CPU folds its delta into
+ * calc_load_nohz per calc_load_nohz_start(), all we need to do is fold
+ * in the pending NO_HZ delta if our NO_HZ period crossed a load cycle boundary.
*
* Once we've updated the global active value, we need to apply the exponential
* weights adjusted to the number of cycles missed.
}
/*
- * Flip the idle index...
+ * Flip the NO_HZ index...
*
* Make sure we first write the new time then flip the index, so that
* calc_load_write_idx() will see the new time when it reads the new
}
#else /* !CONFIG_NO_HZ_COMMON */
-static inline long calc_load_fold_idle(void) { return 0; }
+static inline long calc_load_nohz_fold(void) { return 0; }
static inline void calc_global_nohz(void) { }
#endif /* CONFIG_NO_HZ_COMMON */
return;
/*
- * Fold the 'old' idle-delta to include all NO_HZ cpus.
+ * Fold the 'old' NO_HZ-delta to include all NO_HZ cpus.
*/
- delta = calc_load_fold_idle();
+ delta = calc_load_nohz_fold();
if (delta)
atomic_long_add(delta, &calc_load_tasks);
WRITE_ONCE(calc_load_update, sample_window + LOAD_FREQ);
/*
- * In case we idled for multiple LOAD_FREQ intervals, catch up in bulk.
+ * In case we went to NO_HZ for multiple LOAD_FREQ intervals
+ * catch up in bulk.
*/
calc_global_nohz();
}
int enqueue = 0;
struct rt_rq *rt_rq = sched_rt_period_rt_rq(rt_b, i);
struct rq *rq = rq_of_rt_rq(rt_rq);
+ int skip;
+
+ /*
+ * When span == cpu_online_mask, taking each rq->lock
+ * can be time-consuming. Try to avoid it when possible.
+ */
+ raw_spin_lock(&rt_rq->rt_runtime_lock);
+ skip = !rt_rq->rt_time && !rt_rq->rt_nr_running;
+ raw_spin_unlock(&rt_rq->rt_runtime_lock);
+ if (skip)
+ continue;
raw_spin_lock(&rq->lock);
if (rt_rq->rt_time) {
* pushing.
*/
task = pick_next_pushable_task(rq);
- if (task_cpu(next_task) == rq->cpu && task == next_task) {
+ if (task == next_task) {
/*
* The task hasn't migrated, and is still the next
* eligible task, but we failed to find a run-queue
.update_curr = update_curr_rt,
};
+#ifdef CONFIG_RT_GROUP_SCHED
+/*
+ * Ensure that the real time constraints are schedulable.
+ */
+static DEFINE_MUTEX(rt_constraints_mutex);
+
+/* Must be called with tasklist_lock held */
+static inline int tg_has_rt_tasks(struct task_group *tg)
+{
+ struct task_struct *g, *p;
+
+ /*
+ * Autogroups do not have RT tasks; see autogroup_create().
+ */
+ if (task_group_is_autogroup(tg))
+ return 0;
+
+ for_each_process_thread(g, p) {
+ if (rt_task(p) && task_group(p) == tg)
+ return 1;
+ }
+
+ return 0;
+}
+
+struct rt_schedulable_data {
+ struct task_group *tg;
+ u64 rt_period;
+ u64 rt_runtime;
+};
+
+static int tg_rt_schedulable(struct task_group *tg, void *data)
+{
+ struct rt_schedulable_data *d = data;
+ struct task_group *child;
+ unsigned long total, sum = 0;
+ u64 period, runtime;
+
+ period = ktime_to_ns(tg->rt_bandwidth.rt_period);
+ runtime = tg->rt_bandwidth.rt_runtime;
+
+ if (tg == d->tg) {
+ period = d->rt_period;
+ runtime = d->rt_runtime;
+ }
+
+ /*
+ * Cannot have more runtime than the period.
+ */
+ if (runtime > period && runtime != RUNTIME_INF)
+ return -EINVAL;
+
+ /*
+ * Ensure we don't starve existing RT tasks.
+ */
+ if (rt_bandwidth_enabled() && !runtime && tg_has_rt_tasks(tg))
+ return -EBUSY;
+
+ total = to_ratio(period, runtime);
+
+ /*
+ * Nobody can have more than the global setting allows.
+ */
+ if (total > to_ratio(global_rt_period(), global_rt_runtime()))
+ return -EINVAL;
+
+ /*
+ * The sum of our children's runtime should not exceed our own.
+ */
+ list_for_each_entry_rcu(child, &tg->children, siblings) {
+ period = ktime_to_ns(child->rt_bandwidth.rt_period);
+ runtime = child->rt_bandwidth.rt_runtime;
+
+ if (child == d->tg) {
+ period = d->rt_period;
+ runtime = d->rt_runtime;
+ }
+
+ sum += to_ratio(period, runtime);
+ }
+
+ if (sum > total)
+ return -EINVAL;
+
+ return 0;
+}
+
+static int __rt_schedulable(struct task_group *tg, u64 period, u64 runtime)
+{
+ int ret;
+
+ struct rt_schedulable_data data = {
+ .tg = tg,
+ .rt_period = period,
+ .rt_runtime = runtime,
+ };
+
+ rcu_read_lock();
+ ret = walk_tg_tree(tg_rt_schedulable, tg_nop, &data);
+ rcu_read_unlock();
+
+ return ret;
+}
+
+static int tg_set_rt_bandwidth(struct task_group *tg,
+ u64 rt_period, u64 rt_runtime)
+{
+ int i, err = 0;
+
+ /*
+ * Disallowing the root group RT runtime is BAD, it would disallow the
+ * kernel creating (and or operating) RT threads.
+ */
+ if (tg == &root_task_group && rt_runtime == 0)
+ return -EINVAL;
+
+ /* No period doesn't make any sense. */
+ if (rt_period == 0)
+ return -EINVAL;
+
+ mutex_lock(&rt_constraints_mutex);
+ read_lock(&tasklist_lock);
+ err = __rt_schedulable(tg, rt_period, rt_runtime);
+ if (err)
+ goto unlock;
+
+ raw_spin_lock_irq(&tg->rt_bandwidth.rt_runtime_lock);
+ tg->rt_bandwidth.rt_period = ns_to_ktime(rt_period);
+ tg->rt_bandwidth.rt_runtime = rt_runtime;
+
+ for_each_possible_cpu(i) {
+ struct rt_rq *rt_rq = tg->rt_rq[i];
+
+ raw_spin_lock(&rt_rq->rt_runtime_lock);
+ rt_rq->rt_runtime = rt_runtime;
+ raw_spin_unlock(&rt_rq->rt_runtime_lock);
+ }
+ raw_spin_unlock_irq(&tg->rt_bandwidth.rt_runtime_lock);
+unlock:
+ read_unlock(&tasklist_lock);
+ mutex_unlock(&rt_constraints_mutex);
+
+ return err;
+}
+
+int sched_group_set_rt_runtime(struct task_group *tg, long rt_runtime_us)
+{
+ u64 rt_runtime, rt_period;
+
+ rt_period = ktime_to_ns(tg->rt_bandwidth.rt_period);
+ rt_runtime = (u64)rt_runtime_us * NSEC_PER_USEC;
+ if (rt_runtime_us < 0)
+ rt_runtime = RUNTIME_INF;
+
+ return tg_set_rt_bandwidth(tg, rt_period, rt_runtime);
+}
+
+long sched_group_rt_runtime(struct task_group *tg)
+{
+ u64 rt_runtime_us;
+
+ if (tg->rt_bandwidth.rt_runtime == RUNTIME_INF)
+ return -1;
+
+ rt_runtime_us = tg->rt_bandwidth.rt_runtime;
+ do_div(rt_runtime_us, NSEC_PER_USEC);
+ return rt_runtime_us;
+}
+
+int sched_group_set_rt_period(struct task_group *tg, u64 rt_period_us)
+{
+ u64 rt_runtime, rt_period;
+
+ rt_period = rt_period_us * NSEC_PER_USEC;
+ rt_runtime = tg->rt_bandwidth.rt_runtime;
+
+ return tg_set_rt_bandwidth(tg, rt_period, rt_runtime);
+}
+
+long sched_group_rt_period(struct task_group *tg)
+{
+ u64 rt_period_us;
+
+ rt_period_us = ktime_to_ns(tg->rt_bandwidth.rt_period);
+ do_div(rt_period_us, NSEC_PER_USEC);
+ return rt_period_us;
+}
+
+static int sched_rt_global_constraints(void)
+{
+ int ret = 0;
+
+ mutex_lock(&rt_constraints_mutex);
+ read_lock(&tasklist_lock);
+ ret = __rt_schedulable(NULL, 0, 0);
+ read_unlock(&tasklist_lock);
+ mutex_unlock(&rt_constraints_mutex);
+
+ return ret;
+}
+
+int sched_rt_can_attach(struct task_group *tg, struct task_struct *tsk)
+{
+ /* Don't accept realtime tasks when there is no way for them to run */
+ if (rt_task(tsk) && tg->rt_bandwidth.rt_runtime == 0)
+ return 0;
+
+ return 1;
+}
+
+#else /* !CONFIG_RT_GROUP_SCHED */
+static int sched_rt_global_constraints(void)
+{
+ unsigned long flags;
+ int i;
+
+ raw_spin_lock_irqsave(&def_rt_bandwidth.rt_runtime_lock, flags);
+ for_each_possible_cpu(i) {
+ struct rt_rq *rt_rq = &cpu_rq(i)->rt;
+
+ raw_spin_lock(&rt_rq->rt_runtime_lock);
+ rt_rq->rt_runtime = global_rt_runtime();
+ raw_spin_unlock(&rt_rq->rt_runtime_lock);
+ }
+ raw_spin_unlock_irqrestore(&def_rt_bandwidth.rt_runtime_lock, flags);
+
+ return 0;
+}
+#endif /* CONFIG_RT_GROUP_SCHED */
+
+static int sched_rt_global_validate(void)
+{
+ if (sysctl_sched_rt_period <= 0)
+ return -EINVAL;
+
+ if ((sysctl_sched_rt_runtime != RUNTIME_INF) &&
+ (sysctl_sched_rt_runtime > sysctl_sched_rt_period))
+ return -EINVAL;
+
+ return 0;
+}
+
+static void sched_rt_do_global(void)
+{
+ def_rt_bandwidth.rt_runtime = global_rt_runtime();
+ def_rt_bandwidth.rt_period = ns_to_ktime(global_rt_period());
+}
+
+int sched_rt_handler(struct ctl_table *table, int write,
+ void __user *buffer, size_t *lenp,
+ loff_t *ppos)
+{
+ int old_period, old_runtime;
+ static DEFINE_MUTEX(mutex);
+ int ret;
+
+ mutex_lock(&mutex);
+ old_period = sysctl_sched_rt_period;
+ old_runtime = sysctl_sched_rt_runtime;
+
+ ret = proc_dointvec(table, write, buffer, lenp, ppos);
+
+ if (!ret && write) {
+ ret = sched_rt_global_validate();
+ if (ret)
+ goto undo;
+
+ ret = sched_dl_global_validate();
+ if (ret)
+ goto undo;
+
+ ret = sched_rt_global_constraints();
+ if (ret)
+ goto undo;
+
+ sched_rt_do_global();
+ sched_dl_do_global();
+ }
+ if (0) {
+undo:
+ sysctl_sched_rt_period = old_period;
+ sysctl_sched_rt_runtime = old_runtime;
+ }
+ mutex_unlock(&mutex);
+
+ return ret;
+}
+
+int sched_rr_handler(struct ctl_table *table, int write,
+ void __user *buffer, size_t *lenp,
+ loff_t *ppos)
+{
+ int ret;
+ static DEFINE_MUTEX(mutex);
+
+ mutex_lock(&mutex);
+ ret = proc_dointvec(table, write, buffer, lenp, ppos);
+ /*
+ * Make sure that internally we keep jiffies.
+ * Also, writing zero resets the timeslice to default:
+ */
+ if (!ret && write) {
+ sched_rr_timeslice =
+ sysctl_sched_rr_timeslice <= 0 ? RR_TIMESLICE :
+ msecs_to_jiffies(sysctl_sched_rr_timeslice);
+ }
+ mutex_unlock(&mutex);
+ return ret;
+}
+
#ifdef CONFIG_SCHED_DEBUG
extern void print_rt_rq(struct seq_file *m, int cpu, struct rt_rq *rt_rq);
#include "cpuacct.h"
#ifdef CONFIG_SCHED_DEBUG
-#define SCHED_WARN_ON(x) WARN_ONCE(x, #x)
+# define SCHED_WARN_ON(x) WARN_ONCE(x, #x)
#else
-#define SCHED_WARN_ON(x) ((void)(x))
+# define SCHED_WARN_ON(x) ({ (void)(x), 0; })
#endif
struct rq;
return sysctl_sched_rt_runtime >= 0;
}
-extern struct dl_bw *dl_bw_of(int i);
-
struct dl_bw {
raw_spinlock_t lock;
u64 bw, total_bw;
};
+static inline void __dl_update(struct dl_bw *dl_b, s64 bw);
+
static inline
-void __dl_clear(struct dl_bw *dl_b, u64 tsk_bw)
+void __dl_clear(struct dl_bw *dl_b, u64 tsk_bw, int cpus)
{
dl_b->total_bw -= tsk_bw;
+ __dl_update(dl_b, (s32)tsk_bw / cpus);
}
static inline
-void __dl_add(struct dl_bw *dl_b, u64 tsk_bw)
+void __dl_add(struct dl_bw *dl_b, u64 tsk_bw, int cpus)
{
dl_b->total_bw += tsk_bw;
+ __dl_update(dl_b, -((s32)tsk_bw / cpus));
}
static inline
dl_b->bw * cpus < dl_b->total_bw - old_bw + new_bw;
}
+void dl_change_utilization(struct task_struct *p, u64 new_bw);
extern void init_dl_bw(struct dl_bw *dl_b);
+extern int sched_dl_global_validate(void);
+extern void sched_dl_do_global(void);
+extern int sched_dl_overflow(struct task_struct *p, int policy,
+ const struct sched_attr *attr);
+extern void __setparam_dl(struct task_struct *p, const struct sched_attr *attr);
+extern void __getparam_dl(struct task_struct *p, struct sched_attr *attr);
+extern bool __checkparam_dl(const struct sched_attr *attr);
+extern void __dl_clear_params(struct task_struct *p);
+extern bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr);
+extern int dl_task_can_attach(struct task_struct *p,
+ const struct cpumask *cs_cpus_allowed);
+extern int dl_cpuset_cpumask_can_shrink(const struct cpumask *cur,
+ const struct cpumask *trial);
+extern bool dl_cpu_busy(unsigned int cpu);
#ifdef CONFIG_CGROUP_SCHED
extern void init_tg_rt_entry(struct task_group *tg, struct rt_rq *rt_rq,
struct sched_rt_entity *rt_se, int cpu,
struct sched_rt_entity *parent);
+extern int sched_group_set_rt_runtime(struct task_group *tg, long rt_runtime_us);
+extern int sched_group_set_rt_period(struct task_group *tg, u64 rt_period_us);
+extern long sched_group_rt_runtime(struct task_group *tg);
+extern long sched_group_rt_period(struct task_group *tg);
+extern int sched_rt_can_attach(struct task_group *tg, struct task_struct *tsk);
extern struct task_group *sched_create_group(struct task_group *parent);
extern void sched_online_group(struct task_group *tg,
#else
struct dl_bw dl_bw;
#endif
+ /*
+ * "Active utilization" for this runqueue: increased when a
+ * task wakes up (becomes TASK_RUNNING) and decreased when a
+ * task blocks
+ */
+ u64 running_bw;
+
+ /*
+ * Utilization of the tasks "assigned" to this runqueue (including
+ * the tasks that are in runqueue and the tasks that executed on this
+ * CPU and blocked). Increased when a task moves to this runqueue, and
+ * decreased when the task moves away (migrates, changes scheduling
+ * policy, or terminates).
+ * This is needed to compute the "inactive utilization" for the
+ * runqueue (inactive utilization = this_bw - running_bw).
+ */
+ u64 this_bw;
+ u64 extra_bw;
+
+ /*
+ * Inverse of the fraction of CPU utilization that can be reclaimed
+ * by the GRUB algorithm.
+ */
+ u64 bw_ratio;
};
#ifdef CONFIG_SMP
extern struct root_domain def_root_domain;
extern struct mutex sched_domains_mutex;
-extern cpumask_var_t fallback_doms;
-extern cpumask_var_t sched_domains_tmpmask;
extern void init_defrootdomain(void);
-extern int init_sched_domains(const struct cpumask *cpu_map);
+extern int sched_init_domains(const struct cpumask *cpu_map);
extern void rq_attach_root(struct rq *rq, struct root_domain *rd);
#endif /* CONFIG_SMP */
unsigned long next_update;
int imbalance; /* XXX unrelated to capacity but shared group state */
- unsigned long cpumask[0]; /* iteration mask */
+#ifdef CONFIG_SCHED_DEBUG
+ int id;
+#endif
+
+ unsigned long cpumask[0]; /* balance mask */
};
struct sched_group {
unsigned long cpumask[0];
};
-static inline struct cpumask *sched_group_cpus(struct sched_group *sg)
+static inline struct cpumask *sched_group_span(struct sched_group *sg)
{
return to_cpumask(sg->cpumask);
}
/*
- * cpumask masking which cpus in the group are allowed to iterate up the domain
- * tree.
+ * See build_balance_mask().
*/
-static inline struct cpumask *sched_group_mask(struct sched_group *sg)
+static inline struct cpumask *group_balance_mask(struct sched_group *sg)
{
return to_cpumask(sg->sgc->cpumask);
}
*/
static inline unsigned int group_first_cpu(struct sched_group *group)
{
- return cpumask_first(sched_group_cpus(group));
+ return cpumask_first(sched_group_span(group));
}
extern int group_balance_cpu(struct sched_group *sg);
curr->sched_class->set_curr_task(rq);
}
+#ifdef CONFIG_SMP
#define sched_class_highest (&stop_sched_class)
+#else
+#define sched_class_highest (&dl_sched_class)
+#endif
#define for_each_class(class) \
for (class = sched_class_highest; class; class = class->next)
extern struct dl_bandwidth def_dl_bandwidth;
extern void init_dl_bandwidth(struct dl_bandwidth *dl_b, u64 period, u64 runtime);
extern void init_dl_task_timer(struct sched_dl_entity *dl_se);
+extern void init_dl_inactive_task_timer(struct sched_dl_entity *dl_se);
+extern void init_dl_rq_bw_ratio(struct dl_rq *dl_rq);
+#define BW_SHIFT 20
+#define BW_UNIT (1 << BW_SHIFT)
+#define RATIO_SHIFT 8
unsigned long to_ratio(u64 period, u64 runtime);
extern void init_entity_runnable_average(struct sched_entity *se);
static inline void nohz_balance_exit_idle(unsigned int cpu) { }
#endif
+
+#ifdef CONFIG_SMP
+static inline
+void __dl_update(struct dl_bw *dl_b, s64 bw)
+{
+ struct root_domain *rd = container_of(dl_b, struct root_domain, dl_bw);
+ int i;
+
+ RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held(),
+ "sched RCU must be held");
+ for_each_cpu_and(i, rd->span, cpu_active_mask) {
+ struct rq *rq = cpu_rq(i);
+
+ rq->dl.extra_bw += bw;
+ }
+}
+#else
+static inline
+void __dl_update(struct dl_bw *dl_b, s64 bw)
+{
+ struct dl_rq *dl = container_of(dl_b, struct dl_rq, dl_bw);
+
+ dl->extra_bw += bw;
+}
+#endif
+
+
#ifdef CONFIG_IRQ_TIME_ACCOUNTING
struct irqtime {
u64 total;
/* Protected by sched_domains_mutex: */
cpumask_var_t sched_domains_tmpmask;
+cpumask_var_t sched_domains_tmpmask2;
#ifdef CONFIG_SCHED_DEBUG
cpumask_clear(groupmask);
- printk(KERN_DEBUG "%*s domain %d: ", level, "", level);
+ printk(KERN_DEBUG "%*s domain-%d: ", level, "", level);
if (!(sd->flags & SD_LOAD_BALANCE)) {
printk("does not load-balance\n");
return -1;
}
- printk(KERN_CONT "span %*pbl level %s\n",
+ printk(KERN_CONT "span=%*pbl level=%s\n",
cpumask_pr_args(sched_domain_span(sd)), sd->name);
if (!cpumask_test_cpu(cpu, sched_domain_span(sd))) {
printk(KERN_ERR "ERROR: domain->span does not contain "
"CPU%d\n", cpu);
}
- if (!cpumask_test_cpu(cpu, sched_group_cpus(group))) {
+ if (!cpumask_test_cpu(cpu, sched_group_span(group))) {
printk(KERN_ERR "ERROR: domain->groups does not contain"
" CPU%d\n", cpu);
}
break;
}
- if (!cpumask_weight(sched_group_cpus(group))) {
+ if (!cpumask_weight(sched_group_span(group))) {
printk(KERN_CONT "\n");
printk(KERN_ERR "ERROR: empty group\n");
break;
}
if (!(sd->flags & SD_OVERLAP) &&
- cpumask_intersects(groupmask, sched_group_cpus(group))) {
+ cpumask_intersects(groupmask, sched_group_span(group))) {
printk(KERN_CONT "\n");
printk(KERN_ERR "ERROR: repeated CPUs\n");
break;
}
- cpumask_or(groupmask, groupmask, sched_group_cpus(group));
+ cpumask_or(groupmask, groupmask, sched_group_span(group));
- printk(KERN_CONT " %*pbl",
- cpumask_pr_args(sched_group_cpus(group)));
- if (group->sgc->capacity != SCHED_CAPACITY_SCALE) {
- printk(KERN_CONT " (cpu_capacity = %lu)",
- group->sgc->capacity);
+ printk(KERN_CONT " %d:{ span=%*pbl",
+ group->sgc->id,
+ cpumask_pr_args(sched_group_span(group)));
+
+ if ((sd->flags & SD_OVERLAP) &&
+ !cpumask_equal(group_balance_mask(group), sched_group_span(group))) {
+ printk(KERN_CONT " mask=%*pbl",
+ cpumask_pr_args(group_balance_mask(group)));
+ }
+
+ if (group->sgc->capacity != SCHED_CAPACITY_SCALE)
+ printk(KERN_CONT " cap=%lu", group->sgc->capacity);
+
+ if (group == sd->groups && sd->child &&
+ !cpumask_equal(sched_domain_span(sd->child),
+ sched_group_span(group))) {
+ printk(KERN_ERR "ERROR: domain->groups does not match domain->child\n");
}
+ printk(KERN_CONT " }");
+
group = group->next;
+
+ if (group != sd->groups)
+ printk(KERN_CONT ",");
+
} while (group != sd->groups);
printk(KERN_CONT "\n");
return;
}
- printk(KERN_DEBUG "CPU%d attaching sched-domain:\n", cpu);
+ printk(KERN_DEBUG "CPU%d attaching sched-domain(s):\n", cpu);
for (;;) {
if (sched_domain_debug_one(sd, cpu, level, sched_domains_tmpmask))
};
/*
- * Build an iteration mask that can exclude certain CPUs from the upwards
- * domain traversal.
+ * Return the canonical balance CPU for this group, this is the first CPU
+ * of this group that's also in the balance mask.
*
- * Asymmetric node setups can result in situations where the domain tree is of
- * unequal depth, make sure to skip domains that already cover the entire
- * range.
+ * The balance mask are all those CPUs that could actually end up at this
+ * group. See build_balance_mask().
*
- * In that case build_sched_domains() will have terminated the iteration early
- * and our sibling sd spans will be empty. Domains should always include the
- * CPU they're built on, so check that.
+ * Also see should_we_balance().
*/
-static void build_group_mask(struct sched_domain *sd, struct sched_group *sg)
+int group_balance_cpu(struct sched_group *sg)
{
- const struct cpumask *span = sched_domain_span(sd);
+ return cpumask_first(group_balance_mask(sg));
+}
+
+
+/*
+ * NUMA topology (first read the regular topology blurb below)
+ *
+ * Given a node-distance table, for example:
+ *
+ * node 0 1 2 3
+ * 0: 10 20 30 20
+ * 1: 20 10 20 30
+ * 2: 30 20 10 20
+ * 3: 20 30 20 10
+ *
+ * which represents a 4 node ring topology like:
+ *
+ * 0 ----- 1
+ * | |
+ * | |
+ * | |
+ * 3 ----- 2
+ *
+ * We want to construct domains and groups to represent this. The way we go
+ * about doing this is to build the domains on 'hops'. For each NUMA level we
+ * construct the mask of all nodes reachable in @level hops.
+ *
+ * For the above NUMA topology that gives 3 levels:
+ *
+ * NUMA-2 0-3 0-3 0-3 0-3
+ * groups: {0-1,3},{1-3} {0-2},{0,2-3} {1-3},{0-1,3} {0,2-3},{0-2}
+ *
+ * NUMA-1 0-1,3 0-2 1-3 0,2-3
+ * groups: {0},{1},{3} {0},{1},{2} {1},{2},{3} {0},{2},{3}
+ *
+ * NUMA-0 0 1 2 3
+ *
+ *
+ * As can be seen; things don't nicely line up as with the regular topology.
+ * When we iterate a domain in child domain chunks some nodes can be
+ * represented multiple times -- hence the "overlap" naming for this part of
+ * the topology.
+ *
+ * In order to minimize this overlap, we only build enough groups to cover the
+ * domain. For instance Node-0 NUMA-2 would only get groups: 0-1,3 and 1-3.
+ *
+ * Because:
+ *
+ * - the first group of each domain is its child domain; this
+ * gets us the first 0-1,3
+ * - the only uncovered node is 2, who's child domain is 1-3.
+ *
+ * However, because of the overlap, computing a unique CPU for each group is
+ * more complicated. Consider for instance the groups of NODE-1 NUMA-2, both
+ * groups include the CPUs of Node-0, while those CPUs would not in fact ever
+ * end up at those groups (they would end up in group: 0-1,3).
+ *
+ * To correct this we have to introduce the group balance mask. This mask
+ * will contain those CPUs in the group that can reach this group given the
+ * (child) domain tree.
+ *
+ * With this we can once again compute balance_cpu and sched_group_capacity
+ * relations.
+ *
+ * XXX include words on how balance_cpu is unique and therefore can be
+ * used for sched_group_capacity links.
+ *
+ *
+ * Another 'interesting' topology is:
+ *
+ * node 0 1 2 3
+ * 0: 10 20 20 30
+ * 1: 20 10 20 20
+ * 2: 20 20 10 20
+ * 3: 30 20 20 10
+ *
+ * Which looks a little like:
+ *
+ * 0 ----- 1
+ * | / |
+ * | / |
+ * | / |
+ * 2 ----- 3
+ *
+ * This topology is asymmetric, nodes 1,2 are fully connected, but nodes 0,3
+ * are not.
+ *
+ * This leads to a few particularly weird cases where the sched_domain's are
+ * not of the same number for each cpu. Consider:
+ *
+ * NUMA-2 0-3 0-3
+ * groups: {0-2},{1-3} {1-3},{0-2}
+ *
+ * NUMA-1 0-2 0-3 0-3 1-3
+ *
+ * NUMA-0 0 1 2 3
+ *
+ */
+
+
+/*
+ * Build the balance mask; it contains only those CPUs that can arrive at this
+ * group and should be considered to continue balancing.
+ *
+ * We do this during the group creation pass, therefore the group information
+ * isn't complete yet, however since each group represents a (child) domain we
+ * can fully construct this using the sched_domain bits (which are already
+ * complete).
+ */
+static void
+build_balance_mask(struct sched_domain *sd, struct sched_group *sg, struct cpumask *mask)
+{
+ const struct cpumask *sg_span = sched_group_span(sg);
struct sd_data *sdd = sd->private;
struct sched_domain *sibling;
int i;
- for_each_cpu(i, span) {
+ cpumask_clear(mask);
+
+ for_each_cpu(i, sg_span) {
sibling = *per_cpu_ptr(sdd->sd, i);
- if (!cpumask_test_cpu(i, sched_domain_span(sibling)))
+
+ /*
+ * Can happen in the asymmetric case, where these siblings are
+ * unused. The mask will not be empty because those CPUs that
+ * do have the top domain _should_ span the domain.
+ */
+ if (!sibling->child)
continue;
- cpumask_set_cpu(i, sched_group_mask(sg));
+ /* If we would not end up here, we can't continue from here */
+ if (!cpumask_equal(sg_span, sched_domain_span(sibling->child)))
+ continue;
+
+ cpumask_set_cpu(i, mask);
}
+
+ /* We must not have empty masks here */
+ WARN_ON_ONCE(cpumask_empty(mask));
}
/*
- * Return the canonical balance CPU for this group, this is the first CPU
- * of this group that's also in the iteration mask.
+ * XXX: This creates per-node group entries; since the load-balancer will
+ * immediately access remote memory to construct this group's load-balance
+ * statistics having the groups node local is of dubious benefit.
*/
-int group_balance_cpu(struct sched_group *sg)
+static struct sched_group *
+build_group_from_child_sched_domain(struct sched_domain *sd, int cpu)
{
- return cpumask_first_and(sched_group_cpus(sg), sched_group_mask(sg));
+ struct sched_group *sg;
+ struct cpumask *sg_span;
+
+ sg = kzalloc_node(sizeof(struct sched_group) + cpumask_size(),
+ GFP_KERNEL, cpu_to_node(cpu));
+
+ if (!sg)
+ return NULL;
+
+ sg_span = sched_group_span(sg);
+ if (sd->child)
+ cpumask_copy(sg_span, sched_domain_span(sd->child));
+ else
+ cpumask_copy(sg_span, sched_domain_span(sd));
+
+ return sg;
+}
+
+static void init_overlap_sched_group(struct sched_domain *sd,
+ struct sched_group *sg)
+{
+ struct cpumask *mask = sched_domains_tmpmask2;
+ struct sd_data *sdd = sd->private;
+ struct cpumask *sg_span;
+ int cpu;
+
+ build_balance_mask(sd, sg, mask);
+ cpu = cpumask_first_and(sched_group_span(sg), mask);
+
+ sg->sgc = *per_cpu_ptr(sdd->sgc, cpu);
+ if (atomic_inc_return(&sg->sgc->ref) == 1)
+ cpumask_copy(group_balance_mask(sg), mask);
+ else
+ WARN_ON_ONCE(!cpumask_equal(group_balance_mask(sg), mask));
+
+ /*
+ * Initialize sgc->capacity such that even if we mess up the
+ * domains and no possible iteration will get us here, we won't
+ * die on a /0 trap.
+ */
+ sg_span = sched_group_span(sg);
+ sg->sgc->capacity = SCHED_CAPACITY_SCALE * cpumask_weight(sg_span);
+ sg->sgc->min_capacity = SCHED_CAPACITY_SCALE;
}
static int
build_overlap_sched_groups(struct sched_domain *sd, int cpu)
{
- struct sched_group *first = NULL, *last = NULL, *groups = NULL, *sg;
+ struct sched_group *first = NULL, *last = NULL, *sg;
const struct cpumask *span = sched_domain_span(sd);
struct cpumask *covered = sched_domains_tmpmask;
struct sd_data *sdd = sd->private;
cpumask_clear(covered);
- for_each_cpu(i, span) {
+ for_each_cpu_wrap(i, span, cpu) {
struct cpumask *sg_span;
if (cpumask_test_cpu(i, covered))
sibling = *per_cpu_ptr(sdd->sd, i);
- /* See the comment near build_group_mask(). */
+ /*
+ * Asymmetric node setups can result in situations where the
+ * domain tree is of unequal depth, make sure to skip domains
+ * that already cover the entire range.
+ *
+ * In that case build_sched_domains() will have terminated the
+ * iteration early and our sibling sd spans will be empty.
+ * Domains should always include the CPU they're built on, so
+ * check that.
+ */
if (!cpumask_test_cpu(i, sched_domain_span(sibling)))
continue;
- sg = kzalloc_node(sizeof(struct sched_group) + cpumask_size(),
- GFP_KERNEL, cpu_to_node(cpu));
-
+ sg = build_group_from_child_sched_domain(sibling, cpu);
if (!sg)
goto fail;
- sg_span = sched_group_cpus(sg);
- if (sibling->child)
- cpumask_copy(sg_span, sched_domain_span(sibling->child));
- else
- cpumask_set_cpu(i, sg_span);
-
+ sg_span = sched_group_span(sg);
cpumask_or(covered, covered, sg_span);
- sg->sgc = *per_cpu_ptr(sdd->sgc, i);
- if (atomic_inc_return(&sg->sgc->ref) == 1)
- build_group_mask(sd, sg);
-
- /*
- * Initialize sgc->capacity such that even if we mess up the
- * domains and no possible iteration will get us here, we won't
- * die on a /0 trap.
- */
- sg->sgc->capacity = SCHED_CAPACITY_SCALE * cpumask_weight(sg_span);
- sg->sgc->min_capacity = SCHED_CAPACITY_SCALE;
-
- /*
- * Make sure the first group of this domain contains the
- * canonical balance CPU. Otherwise the sched_domain iteration
- * breaks. See update_sg_lb_stats().
- */
- if ((!groups && cpumask_test_cpu(cpu, sg_span)) ||
- group_balance_cpu(sg) == cpu)
- groups = sg;
+ init_overlap_sched_group(sd, sg);
if (!first)
first = sg;
last = sg;
last->next = first;
}
- sd->groups = groups;
+ sd->groups = first;
return 0;
return -ENOMEM;
}
-static int get_group(int cpu, struct sd_data *sdd, struct sched_group **sg)
+
+/*
+ * Package topology (also see the load-balance blurb in fair.c)
+ *
+ * The scheduler builds a tree structure to represent a number of important
+ * topology features. By default (default_topology[]) these include:
+ *
+ * - Simultaneous multithreading (SMT)
+ * - Multi-Core Cache (MC)
+ * - Package (DIE)
+ *
+ * Where the last one more or less denotes everything up to a NUMA node.
+ *
+ * The tree consists of 3 primary data structures:
+ *
+ * sched_domain -> sched_group -> sched_group_capacity
+ * ^ ^ ^ ^
+ * `-' `-'
+ *
+ * The sched_domains are per-cpu and have a two way link (parent & child) and
+ * denote the ever growing mask of CPUs belonging to that level of topology.
+ *
+ * Each sched_domain has a circular (double) linked list of sched_group's, each
+ * denoting the domains of the level below (or individual CPUs in case of the
+ * first domain level). The sched_group linked by a sched_domain includes the
+ * CPU of that sched_domain [*].
+ *
+ * Take for instance a 2 threaded, 2 core, 2 cache cluster part:
+ *
+ * CPU 0 1 2 3 4 5 6 7
+ *
+ * DIE [ ]
+ * MC [ ] [ ]
+ * SMT [ ] [ ] [ ] [ ]
+ *
+ * - or -
+ *
+ * DIE 0-7 0-7 0-7 0-7 0-7 0-7 0-7 0-7
+ * MC 0-3 0-3 0-3 0-3 4-7 4-7 4-7 4-7
+ * SMT 0-1 0-1 2-3 2-3 4-5 4-5 6-7 6-7
+ *
+ * CPU 0 1 2 3 4 5 6 7
+ *
+ * One way to think about it is: sched_domain moves you up and down among these
+ * topology levels, while sched_group moves you sideways through it, at child
+ * domain granularity.
+ *
+ * sched_group_capacity ensures each unique sched_group has shared storage.
+ *
+ * There are two related construction problems, both require a CPU that
+ * uniquely identify each group (for a given domain):
+ *
+ * - The first is the balance_cpu (see should_we_balance() and the
+ * load-balance blub in fair.c); for each group we only want 1 CPU to
+ * continue balancing at a higher domain.
+ *
+ * - The second is the sched_group_capacity; we want all identical groups
+ * to share a single sched_group_capacity.
+ *
+ * Since these topologies are exclusive by construction. That is, its
+ * impossible for an SMT thread to belong to multiple cores, and cores to
+ * be part of multiple caches. There is a very clear and unique location
+ * for each CPU in the hierarchy.
+ *
+ * Therefore computing a unique CPU for each group is trivial (the iteration
+ * mask is redundant and set all 1s; all CPUs in a group will end up at _that_
+ * group), we can simply pick the first CPU in each group.
+ *
+ *
+ * [*] in other words, the first group of each domain is its child domain.
+ */
+
+static struct sched_group *get_group(int cpu, struct sd_data *sdd)
{
struct sched_domain *sd = *per_cpu_ptr(sdd->sd, cpu);
struct sched_domain *child = sd->child;
+ struct sched_group *sg;
if (child)
cpu = cpumask_first(sched_domain_span(child));
- if (sg) {
- *sg = *per_cpu_ptr(sdd->sg, cpu);
- (*sg)->sgc = *per_cpu_ptr(sdd->sgc, cpu);
+ sg = *per_cpu_ptr(sdd->sg, cpu);
+ sg->sgc = *per_cpu_ptr(sdd->sgc, cpu);
+
+ /* For claim_allocations: */
+ atomic_inc(&sg->ref);
+ atomic_inc(&sg->sgc->ref);
- /* For claim_allocations: */
- atomic_set(&(*sg)->sgc->ref, 1);
+ if (child) {
+ cpumask_copy(sched_group_span(sg), sched_domain_span(child));
+ cpumask_copy(group_balance_mask(sg), sched_group_span(sg));
+ } else {
+ cpumask_set_cpu(cpu, sched_group_span(sg));
+ cpumask_set_cpu(cpu, group_balance_mask(sg));
}
- return cpu;
+ sg->sgc->capacity = SCHED_CAPACITY_SCALE * cpumask_weight(sched_group_span(sg));
+ sg->sgc->min_capacity = SCHED_CAPACITY_SCALE;
+
+ return sg;
}
/*
struct cpumask *covered;
int i;
- get_group(cpu, sdd, &sd->groups);
- atomic_inc(&sd->groups->ref);
-
- if (cpu != cpumask_first(span))
- return 0;
-
lockdep_assert_held(&sched_domains_mutex);
covered = sched_domains_tmpmask;
cpumask_clear(covered);
- for_each_cpu(i, span) {
+ for_each_cpu_wrap(i, span, cpu) {
struct sched_group *sg;
- int group, j;
if (cpumask_test_cpu(i, covered))
continue;
- group = get_group(i, sdd, &sg);
- cpumask_setall(sched_group_mask(sg));
+ sg = get_group(i, sdd);
- for_each_cpu(j, span) {
- if (get_group(j, sdd, NULL) != group)
- continue;
-
- cpumask_set_cpu(j, covered);
- cpumask_set_cpu(j, sched_group_cpus(sg));
- }
+ cpumask_or(covered, covered, sched_group_span(sg));
if (!first)
first = sg;
last = sg;
}
last->next = first;
+ sd->groups = first;
return 0;
}
do {
int cpu, max_cpu = -1;
- sg->group_weight = cpumask_weight(sched_group_cpus(sg));
+ sg->group_weight = cpumask_weight(sched_group_span(sg));
if (!(sd->flags & SD_ASYM_PACKING))
goto next;
- for_each_cpu(cpu, sched_group_cpus(sg)) {
+ for_each_cpu(cpu, sched_group_span(sg)) {
if (max_cpu < 0)
max_cpu = cpu;
else if (sched_asym_prefer(cpu, max_cpu))
if (!sgc)
return -ENOMEM;
+#ifdef CONFIG_SCHED_DEBUG
+ sgc->id = j;
+#endif
+
*per_cpu_ptr(sdd->sgc, j) = sgc;
}
}
sd = build_sched_domain(tl, cpu_map, attr, sd, i);
if (tl == sched_domain_topology)
*per_cpu_ptr(d.sd, i) = sd;
- if (tl->flags & SDTL_OVERLAP || sched_feat(FORCE_SD_OVERLAP))
+ if (tl->flags & SDTL_OVERLAP)
sd->flags |= SD_OVERLAP;
if (cpumask_equal(cpu_map, sched_domain_span(sd)))
break;
* cpumask) fails, then fallback to a single sched domain,
* as determined by the single cpumask fallback_doms.
*/
-cpumask_var_t fallback_doms;
+static cpumask_var_t fallback_doms;
/*
* arch_update_cpu_topology lets virtualized architectures update the
* For now this just excludes isolated CPUs, but could be used to
* exclude other special cases in the future.
*/
-int init_sched_domains(const struct cpumask *cpu_map)
+int sched_init_domains(const struct cpumask *cpu_map)
{
int err;
+ zalloc_cpumask_var(&sched_domains_tmpmask, GFP_KERNEL);
+ zalloc_cpumask_var(&sched_domains_tmpmask2, GFP_KERNEL);
+ zalloc_cpumask_var(&fallback_doms, GFP_KERNEL);
+
arch_update_cpu_topology();
ndoms_cur = 1;
doms_cur = alloc_sched_domains(ndoms_cur);
#include <linux/hash.h>
#include <linux/kthread.h>
-void __init_waitqueue_head(wait_queue_head_t *q, const char *name, struct lock_class_key *key)
+void __init_waitqueue_head(struct wait_queue_head *wq_head, const char *name, struct lock_class_key *key)
{
- spin_lock_init(&q->lock);
- lockdep_set_class_and_name(&q->lock, key, name);
- INIT_LIST_HEAD(&q->task_list);
+ spin_lock_init(&wq_head->lock);
+ lockdep_set_class_and_name(&wq_head->lock, key, name);
+ INIT_LIST_HEAD(&wq_head->head);
}
EXPORT_SYMBOL(__init_waitqueue_head);
-void add_wait_queue(wait_queue_head_t *q, wait_queue_t *wait)
+void add_wait_queue(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry)
{
unsigned long flags;
- wait->flags &= ~WQ_FLAG_EXCLUSIVE;
- spin_lock_irqsave(&q->lock, flags);
- __add_wait_queue(q, wait);
- spin_unlock_irqrestore(&q->lock, flags);
+ wq_entry->flags &= ~WQ_FLAG_EXCLUSIVE;
+ spin_lock_irqsave(&wq_head->lock, flags);
+ __add_wait_queue_entry_tail(wq_head, wq_entry);
+ spin_unlock_irqrestore(&wq_head->lock, flags);
}
EXPORT_SYMBOL(add_wait_queue);
-void add_wait_queue_exclusive(wait_queue_head_t *q, wait_queue_t *wait)
+void add_wait_queue_exclusive(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry)
{
unsigned long flags;
- wait->flags |= WQ_FLAG_EXCLUSIVE;
- spin_lock_irqsave(&q->lock, flags);
- __add_wait_queue_tail(q, wait);
- spin_unlock_irqrestore(&q->lock, flags);
+ wq_entry->flags |= WQ_FLAG_EXCLUSIVE;
+ spin_lock_irqsave(&wq_head->lock, flags);
+ __add_wait_queue_entry_tail(wq_head, wq_entry);
+ spin_unlock_irqrestore(&wq_head->lock, flags);
}
EXPORT_SYMBOL(add_wait_queue_exclusive);
-void remove_wait_queue(wait_queue_head_t *q, wait_queue_t *wait)
+void remove_wait_queue(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry)
{
unsigned long flags;
- spin_lock_irqsave(&q->lock, flags);
- __remove_wait_queue(q, wait);
- spin_unlock_irqrestore(&q->lock, flags);
+ spin_lock_irqsave(&wq_head->lock, flags);
+ __remove_wait_queue(wq_head, wq_entry);
+ spin_unlock_irqrestore(&wq_head->lock, flags);
}
EXPORT_SYMBOL(remove_wait_queue);
* started to run but is not in state TASK_RUNNING. try_to_wake_up() returns
* zero in this (rare) case, and we handle it by continuing to scan the queue.
*/
-static void __wake_up_common(wait_queue_head_t *q, unsigned int mode,
+static void __wake_up_common(struct wait_queue_head *wq_head, unsigned int mode,
int nr_exclusive, int wake_flags, void *key)
{
- wait_queue_t *curr, *next;
+ wait_queue_entry_t *curr, *next;
- list_for_each_entry_safe(curr, next, &q->task_list, task_list) {
+ list_for_each_entry_safe(curr, next, &wq_head->head, entry) {
unsigned flags = curr->flags;
if (curr->func(curr, mode, wake_flags, key) &&
/**
* __wake_up - wake up threads blocked on a waitqueue.
- * @q: the waitqueue
+ * @wq_head: the waitqueue
* @mode: which threads
* @nr_exclusive: how many wake-one or wake-many threads to wake up
* @key: is directly passed to the wakeup function
* It may be assumed that this function implies a write memory barrier before
* changing the task state if and only if any tasks are woken up.
*/
-void __wake_up(wait_queue_head_t *q, unsigned int mode,
+void __wake_up(struct wait_queue_head *wq_head, unsigned int mode,
int nr_exclusive, void *key)
{
unsigned long flags;
- spin_lock_irqsave(&q->lock, flags);
- __wake_up_common(q, mode, nr_exclusive, 0, key);
- spin_unlock_irqrestore(&q->lock, flags);
+ spin_lock_irqsave(&wq_head->lock, flags);
+ __wake_up_common(wq_head, mode, nr_exclusive, 0, key);
+ spin_unlock_irqrestore(&wq_head->lock, flags);
}
EXPORT_SYMBOL(__wake_up);
/*
* Same as __wake_up but called with the spinlock in wait_queue_head_t held.
*/
-void __wake_up_locked(wait_queue_head_t *q, unsigned int mode, int nr)
+void __wake_up_locked(struct wait_queue_head *wq_head, unsigned int mode, int nr)
{
- __wake_up_common(q, mode, nr, 0, NULL);
+ __wake_up_common(wq_head, mode, nr, 0, NULL);
}
EXPORT_SYMBOL_GPL(__wake_up_locked);
-void __wake_up_locked_key(wait_queue_head_t *q, unsigned int mode, void *key)
+void __wake_up_locked_key(struct wait_queue_head *wq_head, unsigned int mode, void *key)
{
- __wake_up_common(q, mode, 1, 0, key);
+ __wake_up_common(wq_head, mode, 1, 0, key);
}
EXPORT_SYMBOL_GPL(__wake_up_locked_key);
/**
* __wake_up_sync_key - wake up threads blocked on a waitqueue.
- * @q: the waitqueue
+ * @wq_head: the waitqueue
* @mode: which threads
* @nr_exclusive: how many wake-one or wake-many threads to wake up
* @key: opaque value to be passed to wakeup targets
* It may be assumed that this function implies a write memory barrier before
* changing the task state if and only if any tasks are woken up.
*/
-void __wake_up_sync_key(wait_queue_head_t *q, unsigned int mode,
+void __wake_up_sync_key(struct wait_queue_head *wq_head, unsigned int mode,
int nr_exclusive, void *key)
{
unsigned long flags;
int wake_flags = 1; /* XXX WF_SYNC */
- if (unlikely(!q))
+ if (unlikely(!wq_head))
return;
if (unlikely(nr_exclusive != 1))
wake_flags = 0;
- spin_lock_irqsave(&q->lock, flags);
- __wake_up_common(q, mode, nr_exclusive, wake_flags, key);
- spin_unlock_irqrestore(&q->lock, flags);
+ spin_lock_irqsave(&wq_head->lock, flags);
+ __wake_up_common(wq_head, mode, nr_exclusive, wake_flags, key);
+ spin_unlock_irqrestore(&wq_head->lock, flags);
}
EXPORT_SYMBOL_GPL(__wake_up_sync_key);
/*
* __wake_up_sync - see __wake_up_sync_key()
*/
-void __wake_up_sync(wait_queue_head_t *q, unsigned int mode, int nr_exclusive)
+void __wake_up_sync(struct wait_queue_head *wq_head, unsigned int mode, int nr_exclusive)
{
- __wake_up_sync_key(q, mode, nr_exclusive, NULL);
+ __wake_up_sync_key(wq_head, mode, nr_exclusive, NULL);
}
EXPORT_SYMBOL_GPL(__wake_up_sync); /* For internal use only */
* loads to move into the critical region).
*/
void
-prepare_to_wait(wait_queue_head_t *q, wait_queue_t *wait, int state)
+prepare_to_wait(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry, int state)
{
unsigned long flags;
- wait->flags &= ~WQ_FLAG_EXCLUSIVE;
- spin_lock_irqsave(&q->lock, flags);
- if (list_empty(&wait->task_list))
- __add_wait_queue(q, wait);
+ wq_entry->flags &= ~WQ_FLAG_EXCLUSIVE;
+ spin_lock_irqsave(&wq_head->lock, flags);
+ if (list_empty(&wq_entry->entry))
+ __add_wait_queue(wq_head, wq_entry);
set_current_state(state);
- spin_unlock_irqrestore(&q->lock, flags);
+ spin_unlock_irqrestore(&wq_head->lock, flags);
}
EXPORT_SYMBOL(prepare_to_wait);
void
-prepare_to_wait_exclusive(wait_queue_head_t *q, wait_queue_t *wait, int state)
+prepare_to_wait_exclusive(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry, int state)
{
unsigned long flags;
- wait->flags |= WQ_FLAG_EXCLUSIVE;
- spin_lock_irqsave(&q->lock, flags);
- if (list_empty(&wait->task_list))
- __add_wait_queue_tail(q, wait);
+ wq_entry->flags |= WQ_FLAG_EXCLUSIVE;
+ spin_lock_irqsave(&wq_head->lock, flags);
+ if (list_empty(&wq_entry->entry))
+ __add_wait_queue_entry_tail(wq_head, wq_entry);
set_current_state(state);
- spin_unlock_irqrestore(&q->lock, flags);
+ spin_unlock_irqrestore(&wq_head->lock, flags);
}
EXPORT_SYMBOL(prepare_to_wait_exclusive);
-void init_wait_entry(wait_queue_t *wait, int flags)
+void init_wait_entry(struct wait_queue_entry *wq_entry, int flags)
{
- wait->flags = flags;
- wait->private = current;
- wait->func = autoremove_wake_function;
- INIT_LIST_HEAD(&wait->task_list);
+ wq_entry->flags = flags;
+ wq_entry->private = current;
+ wq_entry->func = autoremove_wake_function;
+ INIT_LIST_HEAD(&wq_entry->entry);
}
EXPORT_SYMBOL(init_wait_entry);
-long prepare_to_wait_event(wait_queue_head_t *q, wait_queue_t *wait, int state)
+long prepare_to_wait_event(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry, int state)
{
unsigned long flags;
long ret = 0;
- spin_lock_irqsave(&q->lock, flags);
+ spin_lock_irqsave(&wq_head->lock, flags);
if (unlikely(signal_pending_state(state, current))) {
/*
* Exclusive waiter must not fail if it was selected by wakeup,
*
* The caller will recheck the condition and return success if
* we were already woken up, we can not miss the event because
- * wakeup locks/unlocks the same q->lock.
+ * wakeup locks/unlocks the same wq_head->lock.
*
* But we need to ensure that set-condition + wakeup after that
* can't see us, it should wake up another exclusive waiter if
* we fail.
*/
- list_del_init(&wait->task_list);
+ list_del_init(&wq_entry->entry);
ret = -ERESTARTSYS;
} else {
- if (list_empty(&wait->task_list)) {
- if (wait->flags & WQ_FLAG_EXCLUSIVE)
- __add_wait_queue_tail(q, wait);
+ if (list_empty(&wq_entry->entry)) {
+ if (wq_entry->flags & WQ_FLAG_EXCLUSIVE)
+ __add_wait_queue_entry_tail(wq_head, wq_entry);
else
- __add_wait_queue(q, wait);
+ __add_wait_queue(wq_head, wq_entry);
}
set_current_state(state);
}
- spin_unlock_irqrestore(&q->lock, flags);
+ spin_unlock_irqrestore(&wq_head->lock, flags);
return ret;
}
* condition in the caller before they add the wait
* entry to the wake queue.
*/
-int do_wait_intr(wait_queue_head_t *wq, wait_queue_t *wait)
+int do_wait_intr(wait_queue_head_t *wq, wait_queue_entry_t *wait)
{
- if (likely(list_empty(&wait->task_list)))
- __add_wait_queue_tail(wq, wait);
+ if (likely(list_empty(&wait->entry)))
+ __add_wait_queue_entry_tail(wq, wait);
set_current_state(TASK_INTERRUPTIBLE);
if (signal_pending(current))
}
EXPORT_SYMBOL(do_wait_intr);
-int do_wait_intr_irq(wait_queue_head_t *wq, wait_queue_t *wait)
+int do_wait_intr_irq(wait_queue_head_t *wq, wait_queue_entry_t *wait)
{
- if (likely(list_empty(&wait->task_list)))
- __add_wait_queue_tail(wq, wait);
+ if (likely(list_empty(&wait->entry)))
+ __add_wait_queue_entry_tail(wq, wait);
set_current_state(TASK_INTERRUPTIBLE);
if (signal_pending(current))
/**
* finish_wait - clean up after waiting in a queue
- * @q: waitqueue waited on
- * @wait: wait descriptor
+ * @wq_head: waitqueue waited on
+ * @wq_entry: wait descriptor
*
* Sets current thread back to running state and removes
* the wait descriptor from the given waitqueue if still
* queued.
*/
-void finish_wait(wait_queue_head_t *q, wait_queue_t *wait)
+void finish_wait(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry)
{
unsigned long flags;
* have _one_ other CPU that looks at or modifies
* the list).
*/
- if (!list_empty_careful(&wait->task_list)) {
- spin_lock_irqsave(&q->lock, flags);
- list_del_init(&wait->task_list);
- spin_unlock_irqrestore(&q->lock, flags);
+ if (!list_empty_careful(&wq_entry->entry)) {
+ spin_lock_irqsave(&wq_head->lock, flags);
+ list_del_init(&wq_entry->entry);
+ spin_unlock_irqrestore(&wq_head->lock, flags);
}
}
EXPORT_SYMBOL(finish_wait);
-int autoremove_wake_function(wait_queue_t *wait, unsigned mode, int sync, void *key)
+int autoremove_wake_function(struct wait_queue_entry *wq_entry, unsigned mode, int sync, void *key)
{
- int ret = default_wake_function(wait, mode, sync, key);
+ int ret = default_wake_function(wq_entry, mode, sync, key);
if (ret)
- list_del_init(&wait->task_list);
+ list_del_init(&wq_entry->entry);
return ret;
}
EXPORT_SYMBOL(autoremove_wake_function);
/*
* DEFINE_WAIT_FUNC(wait, woken_wake_func);
*
- * add_wait_queue(&wq, &wait);
+ * add_wait_queue(&wq_head, &wait);
* for (;;) {
* if (condition)
* break;
*
* p->state = mode; condition = true;
* smp_mb(); // A smp_wmb(); // C
- * if (!wait->flags & WQ_FLAG_WOKEN) wait->flags |= WQ_FLAG_WOKEN;
+ * if (!wq_entry->flags & WQ_FLAG_WOKEN) wq_entry->flags |= WQ_FLAG_WOKEN;
* schedule() try_to_wake_up();
* p->state = TASK_RUNNING; ~~~~~~~~~~~~~~~~~~
- * wait->flags &= ~WQ_FLAG_WOKEN; condition = true;
+ * wq_entry->flags &= ~WQ_FLAG_WOKEN; condition = true;
* smp_mb() // B smp_wmb(); // C
- * wait->flags |= WQ_FLAG_WOKEN;
+ * wq_entry->flags |= WQ_FLAG_WOKEN;
* }
- * remove_wait_queue(&wq, &wait);
+ * remove_wait_queue(&wq_head, &wait);
*
*/
-long wait_woken(wait_queue_t *wait, unsigned mode, long timeout)
+long wait_woken(struct wait_queue_entry *wq_entry, unsigned mode, long timeout)
{
set_current_state(mode); /* A */
/*
* woken_wake_function() such that if we observe WQ_FLAG_WOKEN we must
* also observe all state before the wakeup.
*/
- if (!(wait->flags & WQ_FLAG_WOKEN) && !is_kthread_should_stop())
+ if (!(wq_entry->flags & WQ_FLAG_WOKEN) && !is_kthread_should_stop())
timeout = schedule_timeout(timeout);
__set_current_state(TASK_RUNNING);
* condition being true _OR_ WQ_FLAG_WOKEN such that we will not miss
* an event.
*/
- smp_store_mb(wait->flags, wait->flags & ~WQ_FLAG_WOKEN); /* B */
+ smp_store_mb(wq_entry->flags, wq_entry->flags & ~WQ_FLAG_WOKEN); /* B */
return timeout;
}
EXPORT_SYMBOL(wait_woken);
-int woken_wake_function(wait_queue_t *wait, unsigned mode, int sync, void *key)
+int woken_wake_function(struct wait_queue_entry *wq_entry, unsigned mode, int sync, void *key)
{
/*
* Although this function is called under waitqueue lock, LOCK
* and is paired with smp_store_mb() in wait_woken().
*/
smp_wmb(); /* C */
- wait->flags |= WQ_FLAG_WOKEN;
+ wq_entry->flags |= WQ_FLAG_WOKEN;
- return default_wake_function(wait, mode, sync, key);
+ return default_wake_function(wq_entry, mode, sync, key);
}
EXPORT_SYMBOL(woken_wake_function);
-
-int wake_bit_function(wait_queue_t *wait, unsigned mode, int sync, void *arg)
-{
- struct wait_bit_key *key = arg;
- struct wait_bit_queue *wait_bit
- = container_of(wait, struct wait_bit_queue, wait);
-
- if (wait_bit->key.flags != key->flags ||
- wait_bit->key.bit_nr != key->bit_nr ||
- test_bit(key->bit_nr, key->flags))
- return 0;
- else
- return autoremove_wake_function(wait, mode, sync, key);
-}
-EXPORT_SYMBOL(wake_bit_function);
-
-/*
- * To allow interruptible waiting and asynchronous (i.e. nonblocking)
- * waiting, the actions of __wait_on_bit() and __wait_on_bit_lock() are
- * permitted return codes. Nonzero return codes halt waiting and return.
- */
-int __sched
-__wait_on_bit(wait_queue_head_t *wq, struct wait_bit_queue *q,
- wait_bit_action_f *action, unsigned mode)
-{
- int ret = 0;
-
- do {
- prepare_to_wait(wq, &q->wait, mode);
- if (test_bit(q->key.bit_nr, q->key.flags))
- ret = (*action)(&q->key, mode);
- } while (test_bit(q->key.bit_nr, q->key.flags) && !ret);
- finish_wait(wq, &q->wait);
- return ret;
-}
-EXPORT_SYMBOL(__wait_on_bit);
-
-int __sched out_of_line_wait_on_bit(void *word, int bit,
- wait_bit_action_f *action, unsigned mode)
-{
- wait_queue_head_t *wq = bit_waitqueue(word, bit);
- DEFINE_WAIT_BIT(wait, word, bit);
-
- return __wait_on_bit(wq, &wait, action, mode);
-}
-EXPORT_SYMBOL(out_of_line_wait_on_bit);
-
-int __sched out_of_line_wait_on_bit_timeout(
- void *word, int bit, wait_bit_action_f *action,
- unsigned mode, unsigned long timeout)
-{
- wait_queue_head_t *wq = bit_waitqueue(word, bit);
- DEFINE_WAIT_BIT(wait, word, bit);
-
- wait.key.timeout = jiffies + timeout;
- return __wait_on_bit(wq, &wait, action, mode);
-}
-EXPORT_SYMBOL_GPL(out_of_line_wait_on_bit_timeout);
-
-int __sched
-__wait_on_bit_lock(wait_queue_head_t *wq, struct wait_bit_queue *q,
- wait_bit_action_f *action, unsigned mode)
-{
- int ret = 0;
-
- for (;;) {
- prepare_to_wait_exclusive(wq, &q->wait, mode);
- if (test_bit(q->key.bit_nr, q->key.flags)) {
- ret = action(&q->key, mode);
- /*
- * See the comment in prepare_to_wait_event().
- * finish_wait() does not necessarily takes wq->lock,
- * but test_and_set_bit() implies mb() which pairs with
- * smp_mb__after_atomic() before wake_up_page().
- */
- if (ret)
- finish_wait(wq, &q->wait);
- }
- if (!test_and_set_bit(q->key.bit_nr, q->key.flags)) {
- if (!ret)
- finish_wait(wq, &q->wait);
- return 0;
- } else if (ret) {
- return ret;
- }
- }
-}
-EXPORT_SYMBOL(__wait_on_bit_lock);
-
-int __sched out_of_line_wait_on_bit_lock(void *word, int bit,
- wait_bit_action_f *action, unsigned mode)
-{
- wait_queue_head_t *wq = bit_waitqueue(word, bit);
- DEFINE_WAIT_BIT(wait, word, bit);
-
- return __wait_on_bit_lock(wq, &wait, action, mode);
-}
-EXPORT_SYMBOL(out_of_line_wait_on_bit_lock);
-
-void __wake_up_bit(wait_queue_head_t *wq, void *word, int bit)
-{
- struct wait_bit_key key = __WAIT_BIT_KEY_INITIALIZER(word, bit);
- if (waitqueue_active(wq))
- __wake_up(wq, TASK_NORMAL, 1, &key);
-}
-EXPORT_SYMBOL(__wake_up_bit);
-
-/**
- * wake_up_bit - wake up a waiter on a bit
- * @word: the word being waited on, a kernel virtual address
- * @bit: the bit of the word being waited on
- *
- * There is a standard hashed waitqueue table for generic use. This
- * is the part of the hashtable's accessor API that wakes up waiters
- * on a bit. For instance, if one were to have waiters on a bitflag,
- * one would call wake_up_bit() after clearing the bit.
- *
- * In order for this to function properly, as it uses waitqueue_active()
- * internally, some kind of memory barrier must be done prior to calling
- * this. Typically, this will be smp_mb__after_atomic(), but in some
- * cases where bitflags are manipulated non-atomically under a lock, one
- * may need to use a less regular barrier, such fs/inode.c's smp_mb(),
- * because spin_unlock() does not guarantee a memory barrier.
- */
-void wake_up_bit(void *word, int bit)
-{
- __wake_up_bit(bit_waitqueue(word, bit), word, bit);
-}
-EXPORT_SYMBOL(wake_up_bit);
-
-/*
- * Manipulate the atomic_t address to produce a better bit waitqueue table hash
- * index (we're keying off bit -1, but that would produce a horrible hash
- * value).
- */
-static inline wait_queue_head_t *atomic_t_waitqueue(atomic_t *p)
-{
- if (BITS_PER_LONG == 64) {
- unsigned long q = (unsigned long)p;
- return bit_waitqueue((void *)(q & ~1), q & 1);
- }
- return bit_waitqueue(p, 0);
-}
-
-static int wake_atomic_t_function(wait_queue_t *wait, unsigned mode, int sync,
- void *arg)
-{
- struct wait_bit_key *key = arg;
- struct wait_bit_queue *wait_bit
- = container_of(wait, struct wait_bit_queue, wait);
- atomic_t *val = key->flags;
-
- if (wait_bit->key.flags != key->flags ||
- wait_bit->key.bit_nr != key->bit_nr ||
- atomic_read(val) != 0)
- return 0;
- return autoremove_wake_function(wait, mode, sync, key);
-}
-
-/*
- * To allow interruptible waiting and asynchronous (i.e. nonblocking) waiting,
- * the actions of __wait_on_atomic_t() are permitted return codes. Nonzero
- * return codes halt waiting and return.
- */
-static __sched
-int __wait_on_atomic_t(wait_queue_head_t *wq, struct wait_bit_queue *q,
- int (*action)(atomic_t *), unsigned mode)
-{
- atomic_t *val;
- int ret = 0;
-
- do {
- prepare_to_wait(wq, &q->wait, mode);
- val = q->key.flags;
- if (atomic_read(val) == 0)
- break;
- ret = (*action)(val);
- } while (!ret && atomic_read(val) != 0);
- finish_wait(wq, &q->wait);
- return ret;
-}
-
-#define DEFINE_WAIT_ATOMIC_T(name, p) \
- struct wait_bit_queue name = { \
- .key = __WAIT_ATOMIC_T_KEY_INITIALIZER(p), \
- .wait = { \
- .private = current, \
- .func = wake_atomic_t_function, \
- .task_list = \
- LIST_HEAD_INIT((name).wait.task_list), \
- }, \
- }
-
-__sched int out_of_line_wait_on_atomic_t(atomic_t *p, int (*action)(atomic_t *),
- unsigned mode)
-{
- wait_queue_head_t *wq = atomic_t_waitqueue(p);
- DEFINE_WAIT_ATOMIC_T(wait, p);
-
- return __wait_on_atomic_t(wq, &wait, action, mode);
-}
-EXPORT_SYMBOL(out_of_line_wait_on_atomic_t);
-
-/**
- * wake_up_atomic_t - Wake up a waiter on a atomic_t
- * @p: The atomic_t being waited on, a kernel virtual address
- *
- * Wake up anyone waiting for the atomic_t to go to zero.
- *
- * Abuse the bit-waker function and its waitqueue hash table set (the atomic_t
- * check is done by the waiter's wake function, not the by the waker itself).
- */
-void wake_up_atomic_t(atomic_t *p)
-{
- __wake_up_bit(atomic_t_waitqueue(p), p, WAIT_ATOMIC_T_BIT_NR);
-}
-EXPORT_SYMBOL(wake_up_atomic_t);
-
-__sched int bit_wait(struct wait_bit_key *word, int mode)
-{
- schedule();
- if (signal_pending_state(mode, current))
- return -EINTR;
- return 0;
-}
-EXPORT_SYMBOL(bit_wait);
-
-__sched int bit_wait_io(struct wait_bit_key *word, int mode)
-{
- io_schedule();
- if (signal_pending_state(mode, current))
- return -EINTR;
- return 0;
-}
-EXPORT_SYMBOL(bit_wait_io);
-
-__sched int bit_wait_timeout(struct wait_bit_key *word, int mode)
-{
- unsigned long now = READ_ONCE(jiffies);
- if (time_after_eq(now, word->timeout))
- return -EAGAIN;
- schedule_timeout(word->timeout - now);
- if (signal_pending_state(mode, current))
- return -EINTR;
- return 0;
-}
-EXPORT_SYMBOL_GPL(bit_wait_timeout);
-
-__sched int bit_wait_io_timeout(struct wait_bit_key *word, int mode)
-{
- unsigned long now = READ_ONCE(jiffies);
- if (time_after_eq(now, word->timeout))
- return -EAGAIN;
- io_schedule_timeout(word->timeout - now);
- if (signal_pending_state(mode, current))
- return -EINTR;
- return 0;
-}
-EXPORT_SYMBOL_GPL(bit_wait_io_timeout);
--- /dev/null
+/*
+ * The implementation of the wait_bit*() and related waiting APIs:
+ */
+#include <linux/wait_bit.h>
+#include <linux/sched/signal.h>
+#include <linux/sched/debug.h>
+#include <linux/hash.h>
+
+#define WAIT_TABLE_BITS 8
+#define WAIT_TABLE_SIZE (1 << WAIT_TABLE_BITS)
+
+static wait_queue_head_t bit_wait_table[WAIT_TABLE_SIZE] __cacheline_aligned;
+
+wait_queue_head_t *bit_waitqueue(void *word, int bit)
+{
+ const int shift = BITS_PER_LONG == 32 ? 5 : 6;
+ unsigned long val = (unsigned long)word << shift | bit;
+
+ return bit_wait_table + hash_long(val, WAIT_TABLE_BITS);
+}
+EXPORT_SYMBOL(bit_waitqueue);
+
+int wake_bit_function(struct wait_queue_entry *wq_entry, unsigned mode, int sync, void *arg)
+{
+ struct wait_bit_key *key = arg;
+ struct wait_bit_queue_entry *wait_bit = container_of(wq_entry, struct wait_bit_queue_entry, wq_entry);
+
+ if (wait_bit->key.flags != key->flags ||
+ wait_bit->key.bit_nr != key->bit_nr ||
+ test_bit(key->bit_nr, key->flags))
+ return 0;
+ else
+ return autoremove_wake_function(wq_entry, mode, sync, key);
+}
+EXPORT_SYMBOL(wake_bit_function);
+
+/*
+ * To allow interruptible waiting and asynchronous (i.e. nonblocking)
+ * waiting, the actions of __wait_on_bit() and __wait_on_bit_lock() are
+ * permitted return codes. Nonzero return codes halt waiting and return.
+ */
+int __sched
+__wait_on_bit(struct wait_queue_head *wq_head, struct wait_bit_queue_entry *wbq_entry,
+ wait_bit_action_f *action, unsigned mode)
+{
+ int ret = 0;
+
+ do {
+ prepare_to_wait(wq_head, &wbq_entry->wq_entry, mode);
+ if (test_bit(wbq_entry->key.bit_nr, wbq_entry->key.flags))
+ ret = (*action)(&wbq_entry->key, mode);
+ } while (test_bit(wbq_entry->key.bit_nr, wbq_entry->key.flags) && !ret);
+ finish_wait(wq_head, &wbq_entry->wq_entry);
+ return ret;
+}
+EXPORT_SYMBOL(__wait_on_bit);
+
+int __sched out_of_line_wait_on_bit(void *word, int bit,
+ wait_bit_action_f *action, unsigned mode)
+{
+ struct wait_queue_head *wq_head = bit_waitqueue(word, bit);
+ DEFINE_WAIT_BIT(wq_entry, word, bit);
+
+ return __wait_on_bit(wq_head, &wq_entry, action, mode);
+}
+EXPORT_SYMBOL(out_of_line_wait_on_bit);
+
+int __sched out_of_line_wait_on_bit_timeout(
+ void *word, int bit, wait_bit_action_f *action,
+ unsigned mode, unsigned long timeout)
+{
+ struct wait_queue_head *wq_head = bit_waitqueue(word, bit);
+ DEFINE_WAIT_BIT(wq_entry, word, bit);
+
+ wq_entry.key.timeout = jiffies + timeout;
+ return __wait_on_bit(wq_head, &wq_entry, action, mode);
+}
+EXPORT_SYMBOL_GPL(out_of_line_wait_on_bit_timeout);
+
+int __sched
+__wait_on_bit_lock(struct wait_queue_head *wq_head, struct wait_bit_queue_entry *wbq_entry,
+ wait_bit_action_f *action, unsigned mode)
+{
+ int ret = 0;
+
+ for (;;) {
+ prepare_to_wait_exclusive(wq_head, &wbq_entry->wq_entry, mode);
+ if (test_bit(wbq_entry->key.bit_nr, wbq_entry->key.flags)) {
+ ret = action(&wbq_entry->key, mode);
+ /*
+ * See the comment in prepare_to_wait_event().
+ * finish_wait() does not necessarily takes wwq_head->lock,
+ * but test_and_set_bit() implies mb() which pairs with
+ * smp_mb__after_atomic() before wake_up_page().
+ */
+ if (ret)
+ finish_wait(wq_head, &wbq_entry->wq_entry);
+ }
+ if (!test_and_set_bit(wbq_entry->key.bit_nr, wbq_entry->key.flags)) {
+ if (!ret)
+ finish_wait(wq_head, &wbq_entry->wq_entry);
+ return 0;
+ } else if (ret) {
+ return ret;
+ }
+ }
+}
+EXPORT_SYMBOL(__wait_on_bit_lock);
+
+int __sched out_of_line_wait_on_bit_lock(void *word, int bit,
+ wait_bit_action_f *action, unsigned mode)
+{
+ struct wait_queue_head *wq_head = bit_waitqueue(word, bit);
+ DEFINE_WAIT_BIT(wq_entry, word, bit);
+
+ return __wait_on_bit_lock(wq_head, &wq_entry, action, mode);
+}
+EXPORT_SYMBOL(out_of_line_wait_on_bit_lock);
+
+void __wake_up_bit(struct wait_queue_head *wq_head, void *word, int bit)
+{
+ struct wait_bit_key key = __WAIT_BIT_KEY_INITIALIZER(word, bit);
+ if (waitqueue_active(wq_head))
+ __wake_up(wq_head, TASK_NORMAL, 1, &key);
+}
+EXPORT_SYMBOL(__wake_up_bit);
+
+/**
+ * wake_up_bit - wake up a waiter on a bit
+ * @word: the word being waited on, a kernel virtual address
+ * @bit: the bit of the word being waited on
+ *
+ * There is a standard hashed waitqueue table for generic use. This
+ * is the part of the hashtable's accessor API that wakes up waiters
+ * on a bit. For instance, if one were to have waiters on a bitflag,
+ * one would call wake_up_bit() after clearing the bit.
+ *
+ * In order for this to function properly, as it uses waitqueue_active()
+ * internally, some kind of memory barrier must be done prior to calling
+ * this. Typically, this will be smp_mb__after_atomic(), but in some
+ * cases where bitflags are manipulated non-atomically under a lock, one
+ * may need to use a less regular barrier, such fs/inode.c's smp_mb(),
+ * because spin_unlock() does not guarantee a memory barrier.
+ */
+void wake_up_bit(void *word, int bit)
+{
+ __wake_up_bit(bit_waitqueue(word, bit), word, bit);
+}
+EXPORT_SYMBOL(wake_up_bit);
+
+/*
+ * Manipulate the atomic_t address to produce a better bit waitqueue table hash
+ * index (we're keying off bit -1, but that would produce a horrible hash
+ * value).
+ */
+static inline wait_queue_head_t *atomic_t_waitqueue(atomic_t *p)
+{
+ if (BITS_PER_LONG == 64) {
+ unsigned long q = (unsigned long)p;
+ return bit_waitqueue((void *)(q & ~1), q & 1);
+ }
+ return bit_waitqueue(p, 0);
+}
+
+static int wake_atomic_t_function(struct wait_queue_entry *wq_entry, unsigned mode, int sync,
+ void *arg)
+{
+ struct wait_bit_key *key = arg;
+ struct wait_bit_queue_entry *wait_bit = container_of(wq_entry, struct wait_bit_queue_entry, wq_entry);
+ atomic_t *val = key->flags;
+
+ if (wait_bit->key.flags != key->flags ||
+ wait_bit->key.bit_nr != key->bit_nr ||
+ atomic_read(val) != 0)
+ return 0;
+ return autoremove_wake_function(wq_entry, mode, sync, key);
+}
+
+/*
+ * To allow interruptible waiting and asynchronous (i.e. nonblocking) waiting,
+ * the actions of __wait_on_atomic_t() are permitted return codes. Nonzero
+ * return codes halt waiting and return.
+ */
+static __sched
+int __wait_on_atomic_t(struct wait_queue_head *wq_head, struct wait_bit_queue_entry *wbq_entry,
+ int (*action)(atomic_t *), unsigned mode)
+{
+ atomic_t *val;
+ int ret = 0;
+
+ do {
+ prepare_to_wait(wq_head, &wbq_entry->wq_entry, mode);
+ val = wbq_entry->key.flags;
+ if (atomic_read(val) == 0)
+ break;
+ ret = (*action)(val);
+ } while (!ret && atomic_read(val) != 0);
+ finish_wait(wq_head, &wbq_entry->wq_entry);
+ return ret;
+}
+
+#define DEFINE_WAIT_ATOMIC_T(name, p) \
+ struct wait_bit_queue_entry name = { \
+ .key = __WAIT_ATOMIC_T_KEY_INITIALIZER(p), \
+ .wq_entry = { \
+ .private = current, \
+ .func = wake_atomic_t_function, \
+ .entry = \
+ LIST_HEAD_INIT((name).wq_entry.entry), \
+ }, \
+ }
+
+__sched int out_of_line_wait_on_atomic_t(atomic_t *p, int (*action)(atomic_t *),
+ unsigned mode)
+{
+ struct wait_queue_head *wq_head = atomic_t_waitqueue(p);
+ DEFINE_WAIT_ATOMIC_T(wq_entry, p);
+
+ return __wait_on_atomic_t(wq_head, &wq_entry, action, mode);
+}
+EXPORT_SYMBOL(out_of_line_wait_on_atomic_t);
+
+/**
+ * wake_up_atomic_t - Wake up a waiter on a atomic_t
+ * @p: The atomic_t being waited on, a kernel virtual address
+ *
+ * Wake up anyone waiting for the atomic_t to go to zero.
+ *
+ * Abuse the bit-waker function and its waitqueue hash table set (the atomic_t
+ * check is done by the waiter's wake function, not the by the waker itself).
+ */
+void wake_up_atomic_t(atomic_t *p)
+{
+ __wake_up_bit(atomic_t_waitqueue(p), p, WAIT_ATOMIC_T_BIT_NR);
+}
+EXPORT_SYMBOL(wake_up_atomic_t);
+
+__sched int bit_wait(struct wait_bit_key *word, int mode)
+{
+ schedule();
+ if (signal_pending_state(mode, current))
+ return -EINTR;
+ return 0;
+}
+EXPORT_SYMBOL(bit_wait);
+
+__sched int bit_wait_io(struct wait_bit_key *word, int mode)
+{
+ io_schedule();
+ if (signal_pending_state(mode, current))
+ return -EINTR;
+ return 0;
+}
+EXPORT_SYMBOL(bit_wait_io);
+
+__sched int bit_wait_timeout(struct wait_bit_key *word, int mode)
+{
+ unsigned long now = READ_ONCE(jiffies);
+ if (time_after_eq(now, word->timeout))
+ return -EAGAIN;
+ schedule_timeout(word->timeout - now);
+ if (signal_pending_state(mode, current))
+ return -EINTR;
+ return 0;
+}
+EXPORT_SYMBOL_GPL(bit_wait_timeout);
+
+__sched int bit_wait_io_timeout(struct wait_bit_key *word, int mode)
+{
+ unsigned long now = READ_ONCE(jiffies);
+ if (time_after_eq(now, word->timeout))
+ return -EAGAIN;
+ io_schedule_timeout(word->timeout - now);
+ if (signal_pending_state(mode, current))
+ return -EINTR;
+ return 0;
+}
+EXPORT_SYMBOL_GPL(bit_wait_io_timeout);
+
+void __init wait_bit_init(void)
+{
+ int i;
+
+ for (i = 0; i < WAIT_TABLE_SIZE; i++)
+ init_waitqueue_head(bit_wait_table + i);
+}
#include <linux/compat.h>
#include <linux/cn_proc.h>
#include <linux/compiler.h>
+#include <linux/posix-timers.h>
#define CREATE_TRACE_POINTS
#include <trace/events/signal.h>
return !tsk->ptrace;
}
-static void collect_signal(int sig, struct sigpending *list, siginfo_t *info)
+static void collect_signal(int sig, struct sigpending *list, siginfo_t *info,
+ bool *resched_timer)
{
struct sigqueue *q, *first = NULL;
still_pending:
list_del_init(&first->list);
copy_siginfo(info, &first->info);
+
+ *resched_timer =
+ (first->flags & SIGQUEUE_PREALLOC) &&
+ (info->si_code == SI_TIMER) &&
+ (info->si_sys_private);
+
__sigqueue_free(first);
} else {
/*
}
static int __dequeue_signal(struct sigpending *pending, sigset_t *mask,
- siginfo_t *info)
+ siginfo_t *info, bool *resched_timer)
{
int sig = next_signal(pending, mask);
if (sig)
- collect_signal(sig, pending, info);
+ collect_signal(sig, pending, info, resched_timer);
return sig;
}
*/
int dequeue_signal(struct task_struct *tsk, sigset_t *mask, siginfo_t *info)
{
+ bool resched_timer = false;
int signr;
/* We only dequeue private signals from ourselves, we don't let
* signalfd steal them
*/
- signr = __dequeue_signal(&tsk->pending, mask, info);
+ signr = __dequeue_signal(&tsk->pending, mask, info, &resched_timer);
if (!signr) {
signr = __dequeue_signal(&tsk->signal->shared_pending,
- mask, info);
+ mask, info, &resched_timer);
#ifdef CONFIG_POSIX_TIMERS
/*
* itimer signal ?
current->jobctl |= JOBCTL_STOP_DEQUEUED;
}
#ifdef CONFIG_POSIX_TIMERS
- if ((info->si_code & __SI_MASK) == __SI_TIMER && info->si_sys_private) {
+ if (resched_timer) {
/*
* Release the siglock to ensure proper locking order
* of timer locks outside of siglocks. Note, we leave
* about to disable them again anyway.
*/
spin_unlock(&tsk->sighand->siglock);
- do_schedule_next_timer(info);
+ posixtimer_rearm(info);
spin_lock(&tsk->sighand->siglock);
}
#endif
static int ptrace_signal(int signr, siginfo_t *info)
{
- ptrace_signal_deliver();
/*
* We do not check sig_kernel_stop(signr) but set this marker
* unconditionally because we do not know whether debugger will
struct call_function_data {
struct call_single_data __percpu *csd;
cpumask_var_t cpumask;
+ cpumask_var_t cpumask_ipi;
};
static DEFINE_PER_CPU_SHARED_ALIGNED(struct call_function_data, cfd_data);
if (!zalloc_cpumask_var_node(&cfd->cpumask, GFP_KERNEL,
cpu_to_node(cpu)))
return -ENOMEM;
+ if (!zalloc_cpumask_var_node(&cfd->cpumask_ipi, GFP_KERNEL,
+ cpu_to_node(cpu))) {
+ free_cpumask_var(cfd->cpumask);
+ return -ENOMEM;
+ }
cfd->csd = alloc_percpu(struct call_single_data);
if (!cfd->csd) {
free_cpumask_var(cfd->cpumask);
+ free_cpumask_var(cfd->cpumask_ipi);
return -ENOMEM;
}
struct call_function_data *cfd = &per_cpu(cfd_data, cpu);
free_cpumask_var(cfd->cpumask);
+ free_cpumask_var(cfd->cpumask_ipi);
free_percpu(cfd->csd);
return 0;
}
cfd = this_cpu_ptr(&cfd_data);
cpumask_and(cfd->cpumask, mask, cpu_online_mask);
- cpumask_clear_cpu(this_cpu, cfd->cpumask);
+ __cpumask_clear_cpu(this_cpu, cfd->cpumask);
/* Some callers race with other cpus changing the passed mask */
if (unlikely(!cpumask_weight(cfd->cpumask)))
return;
+ cpumask_clear(cfd->cpumask_ipi);
for_each_cpu(cpu, cfd->cpumask) {
struct call_single_data *csd = per_cpu_ptr(cfd->csd, cpu);
csd->flags |= CSD_FLAG_SYNCHRONOUS;
csd->func = func;
csd->info = info;
- llist_add(&csd->llist, &per_cpu(call_single_queue, cpu));
+ if (llist_add(&csd->llist, &per_cpu(call_single_queue, cpu)))
+ __cpumask_set_cpu(cpu, cfd->cpumask_ipi);
}
/* Send a message to all CPUs in the map */
- arch_send_call_function_ipi_mask(cfd->cpumask);
+ arch_send_call_function_ipi_mask(cfd->cpumask_ipi);
if (wait) {
for_each_cpu(cpu, cfd->cpumask) {
}
early_initcall(cpu_stop_init);
-static int __stop_machine(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus)
+int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
+ const struct cpumask *cpus)
{
struct multi_stop_data msdata = {
.fn = fn,
.active_cpus = cpus,
};
+ lockdep_assert_cpus_held();
+
if (!stop_machine_initialized) {
/*
* Handle the case where stop_machine() is called
int ret;
/* No CPUs can come up or down during this. */
- get_online_cpus();
- ret = __stop_machine(fn, data, cpus);
- put_online_cpus();
+ cpus_read_lock();
+ ret = stop_machine_cpuslocked(fn, data, cpus);
+ cpus_read_unlock();
return ret;
}
EXPORT_SYMBOL_GPL(stop_machine);
/* Only supports reads */
if (oldval && oldlen) {
char buf[UUID_STRING_LEN + 1];
- uuid_be uuid;
+ uuid_t uuid;
result = kernel_read(file, 0, buf, sizeof(buf) - 1);
if (result < 0)
buf[result] = '\0';
result = -EIO;
- if (uuid_be_to_bin(buf, &uuid))
+ if (uuid_parse(buf, &uuid))
goto out;
if (oldlen > 16)
Note the boot CPU will still be kept outside the range to
handle the timekeeping duty.
-config NO_HZ_FULL_SYSIDLE
- bool "Detect full-system idle state for full dynticks system"
- depends on NO_HZ_FULL
- default n
- help
- At least one CPU must keep the scheduling-clock tick running for
- timekeeping purposes whenever there is a non-idle CPU, where
- "non-idle" also includes dynticks CPUs as long as they are
- running non-idle tasks. Because the underlying adaptive-tick
- support cannot distinguish between all CPUs being idle and
- all CPUs each running a single task in dynticks mode, the
- underlying support simply ensures that there is always a CPU
- handling the scheduling-clock tick, whether or not all CPUs
- are idle. This Kconfig option enables scalable detection of
- the all-CPUs-idle state, thus allowing the scheduling-clock
- tick to be disabled when all CPUs are idle. Note that scalable
- detection of the all-CPUs-idle state means that larger systems
- will be slower to declare the all-CPUs-idle state.
-
- Say Y if you would like to help debug all-CPUs-idle detection.
-
- Say N if you are unsure.
-
-config NO_HZ_FULL_SYSIDLE_SMALL
- int "Number of CPUs above which large-system approach is used"
- depends on NO_HZ_FULL_SYSIDLE
- range 1 NR_CPUS
- default 8
- help
- The full-system idle detection mechanism takes a lazy approach
- on large systems, as is required to attain decent scalability.
- However, on smaller systems, scalability is not anywhere near as
- large a concern as is energy efficiency. The sysidle subsystem
- therefore uses a fast but non-scalable algorithm for small
- systems and a lazier but scalable algorithm for large systems.
- This Kconfig parameter defines the number of CPUs in the largest
- system that will be considered to be "small".
-
- The default value will be fine in most cases. Battery-powered
- systems that (1) enable NO_HZ_FULL_SYSIDLE, (2) have larger
- numbers of CPUs, and (3) are suffering from battery-lifetime
- problems due to long sysidle latencies might wish to experiment
- with larger values for this Kconfig parameter. On the other
- hand, they might be even better served by disabling NO_HZ_FULL
- entirely, given that NO_HZ_FULL is intended for HPC and
- real-time workloads that at present do not tend to be run on
- battery-powered systems.
-
- Take the default if you are unsure.
-
config NO_HZ
bool "Old Idle dynticks config"
depends on !ARCH_USES_GETTIMEOFFSET && GENERIC_CLOCKEVENTS
#include <linux/posix-timers.h>
#include <linux/workqueue.h>
#include <linux/freezer.h>
+#include <linux/compat.h>
+
+#include "posix-timers.h"
#define CREATE_TRACE_POINTS
#include <trace/events/alarmtimer.h>
clockid_t base_clockid;
} alarm_bases[ALARM_NUMTYPE];
+#if defined(CONFIG_POSIX_TIMERS) || defined(CONFIG_RTC_CLASS)
/* freezer information to handle clock_nanosleep triggered wakeups */
static enum alarmtimer_type freezer_alarmtype;
static ktime_t freezer_expires;
static ktime_t freezer_delta;
static DEFINE_SPINLOCK(freezer_delta_lock);
+#endif
static struct wakeup_source *ws;
}
#endif
-static void alarmtimer_freezerset(ktime_t absexp, enum alarmtimer_type type)
-{
- struct alarm_base *base;
- unsigned long flags;
- ktime_t delta;
-
- switch(type) {
- case ALARM_REALTIME:
- base = &alarm_bases[ALARM_REALTIME];
- type = ALARM_REALTIME_FREEZER;
- break;
- case ALARM_BOOTTIME:
- base = &alarm_bases[ALARM_BOOTTIME];
- type = ALARM_BOOTTIME_FREEZER;
- break;
- default:
- WARN_ONCE(1, "Invalid alarm type: %d\n", type);
- return;
- }
-
- delta = ktime_sub(absexp, base->gettime());
-
- spin_lock_irqsave(&freezer_delta_lock, flags);
- if (!freezer_delta || (delta < freezer_delta)) {
- freezer_delta = delta;
- freezer_expires = absexp;
- freezer_alarmtype = type;
- }
- spin_unlock_irqrestore(&freezer_delta_lock, flags);
-}
-
-
/**
* alarm_init - Initialize an alarm structure
* @alarm: ptr to alarm to be initialized
}
EXPORT_SYMBOL_GPL(alarm_forward_now);
+#ifdef CONFIG_POSIX_TIMERS
+
+static void alarmtimer_freezerset(ktime_t absexp, enum alarmtimer_type type)
+{
+ struct alarm_base *base;
+ unsigned long flags;
+ ktime_t delta;
+
+ switch(type) {
+ case ALARM_REALTIME:
+ base = &alarm_bases[ALARM_REALTIME];
+ type = ALARM_REALTIME_FREEZER;
+ break;
+ case ALARM_BOOTTIME:
+ base = &alarm_bases[ALARM_BOOTTIME];
+ type = ALARM_BOOTTIME_FREEZER;
+ break;
+ default:
+ WARN_ONCE(1, "Invalid alarm type: %d\n", type);
+ return;
+ }
+
+ delta = ktime_sub(absexp, base->gettime());
+
+ spin_lock_irqsave(&freezer_delta_lock, flags);
+ if (!freezer_delta || (delta < freezer_delta)) {
+ freezer_delta = delta;
+ freezer_expires = absexp;
+ freezer_alarmtype = type;
+ }
+ spin_unlock_irqrestore(&freezer_delta_lock, flags);
+}
/**
* clock2alarm - helper that converts from clockid to alarmtypes
static enum alarmtimer_restart alarm_handle_timer(struct alarm *alarm,
ktime_t now)
{
- unsigned long flags;
struct k_itimer *ptr = container_of(alarm, struct k_itimer,
- it.alarm.alarmtimer);
+ it.alarm.alarmtimer);
enum alarmtimer_restart result = ALARMTIMER_NORESTART;
+ unsigned long flags;
+ int si_private = 0;
spin_lock_irqsave(&ptr->it_lock, flags);
- if ((ptr->it_sigev_notify & ~SIGEV_THREAD_ID) != SIGEV_NONE) {
- if (IS_ENABLED(CONFIG_POSIX_TIMERS) &&
- posix_timer_event(ptr, 0) != 0)
- ptr->it_overrun++;
- }
- /* Re-add periodic timers */
- if (ptr->it.alarm.interval) {
- ptr->it_overrun += alarm_forward(alarm, now,
- ptr->it.alarm.interval);
+ ptr->it_active = 0;
+ if (ptr->it_interval)
+ si_private = ++ptr->it_requeue_pending;
+
+ if (posix_timer_event(ptr, si_private) && ptr->it_interval) {
+ /*
+ * Handle ignored signals and rearm the timer. This will go
+ * away once we handle ignored signals proper.
+ */
+ ptr->it_overrun += alarm_forward_now(alarm, ptr->it_interval);
+ ++ptr->it_requeue_pending;
+ ptr->it_active = 1;
result = ALARMTIMER_RESTART;
}
spin_unlock_irqrestore(&ptr->it_lock, flags);
return result;
}
+/**
+ * alarm_timer_rearm - Posix timer callback for rearming timer
+ * @timr: Pointer to the posixtimer data struct
+ */
+static void alarm_timer_rearm(struct k_itimer *timr)
+{
+ struct alarm *alarm = &timr->it.alarm.alarmtimer;
+
+ timr->it_overrun += alarm_forward_now(alarm, timr->it_interval);
+ alarm_start(alarm, alarm->node.expires);
+}
+
+/**
+ * alarm_timer_forward - Posix timer callback for forwarding timer
+ * @timr: Pointer to the posixtimer data struct
+ * @now: Current time to forward the timer against
+ */
+static int alarm_timer_forward(struct k_itimer *timr, ktime_t now)
+{
+ struct alarm *alarm = &timr->it.alarm.alarmtimer;
+
+ return (int) alarm_forward(alarm, timr->it_interval, now);
+}
+
+/**
+ * alarm_timer_remaining - Posix timer callback to retrieve remaining time
+ * @timr: Pointer to the posixtimer data struct
+ * @now: Current time to calculate against
+ */
+static ktime_t alarm_timer_remaining(struct k_itimer *timr, ktime_t now)
+{
+ struct alarm *alarm = &timr->it.alarm.alarmtimer;
+
+ return ktime_sub(now, alarm->node.expires);
+}
+
+/**
+ * alarm_timer_try_to_cancel - Posix timer callback to cancel a timer
+ * @timr: Pointer to the posixtimer data struct
+ */
+static int alarm_timer_try_to_cancel(struct k_itimer *timr)
+{
+ return alarm_try_to_cancel(&timr->it.alarm.alarmtimer);
+}
+
+/**
+ * alarm_timer_arm - Posix timer callback to arm a timer
+ * @timr: Pointer to the posixtimer data struct
+ * @expires: The new expiry time
+ * @absolute: Expiry value is absolute time
+ * @sigev_none: Posix timer does not deliver signals
+ */
+static void alarm_timer_arm(struct k_itimer *timr, ktime_t expires,
+ bool absolute, bool sigev_none)
+{
+ struct alarm *alarm = &timr->it.alarm.alarmtimer;
+ struct alarm_base *base = &alarm_bases[alarm->type];
+
+ if (!absolute)
+ expires = ktime_add_safe(expires, base->gettime());
+ if (sigev_none)
+ alarm->node.expires = expires;
+ else
+ alarm_start(&timr->it.alarm.alarmtimer, expires);
+}
+
/**
* alarm_clock_getres - posix getres interface
* @which_clock: clockid
return 0;
}
-/**
- * alarm_timer_get - posix timer_get interface
- * @new_timer: k_itimer pointer
- * @cur_setting: itimerspec data to fill
- *
- * Copies out the current itimerspec data
- */
-static void alarm_timer_get(struct k_itimer *timr,
- struct itimerspec64 *cur_setting)
-{
- ktime_t relative_expiry_time =
- alarm_expires_remaining(&(timr->it.alarm.alarmtimer));
-
- if (ktime_to_ns(relative_expiry_time) > 0) {
- cur_setting->it_value = ktime_to_timespec64(relative_expiry_time);
- } else {
- cur_setting->it_value.tv_sec = 0;
- cur_setting->it_value.tv_nsec = 0;
- }
-
- cur_setting->it_interval = ktime_to_timespec64(timr->it.alarm.interval);
-}
-
-/**
- * alarm_timer_del - posix timer_del interface
- * @timr: k_itimer pointer to be deleted
- *
- * Cancels any programmed alarms for the given timer.
- */
-static int alarm_timer_del(struct k_itimer *timr)
-{
- if (!rtcdev)
- return -ENOTSUPP;
-
- if (alarm_try_to_cancel(&timr->it.alarm.alarmtimer) < 0)
- return TIMER_RETRY;
-
- return 0;
-}
-
-/**
- * alarm_timer_set - posix timer_set interface
- * @timr: k_itimer pointer to be deleted
- * @flags: timer flags
- * @new_setting: itimerspec to be used
- * @old_setting: itimerspec being replaced
- *
- * Sets the timer to new_setting, and starts the timer.
- */
-static int alarm_timer_set(struct k_itimer *timr, int flags,
- struct itimerspec64 *new_setting,
- struct itimerspec64 *old_setting)
-{
- ktime_t exp;
-
- if (!rtcdev)
- return -ENOTSUPP;
-
- if (flags & ~TIMER_ABSTIME)
- return -EINVAL;
-
- if (old_setting)
- alarm_timer_get(timr, old_setting);
-
- /* If the timer was already set, cancel it */
- if (alarm_try_to_cancel(&timr->it.alarm.alarmtimer) < 0)
- return TIMER_RETRY;
-
- /* start the timer */
- timr->it.alarm.interval = timespec64_to_ktime(new_setting->it_interval);
-
- /*
- * Rate limit to the tick as a hot fix to prevent DOS. Will be
- * mopped up later.
- */
- if (timr->it.alarm.interval < TICK_NSEC)
- timr->it.alarm.interval = TICK_NSEC;
-
- exp = timespec64_to_ktime(new_setting->it_value);
- /* Convert (if necessary) to absolute time */
- if (flags != TIMER_ABSTIME) {
- ktime_t now;
-
- now = alarm_bases[timr->it.alarm.alarmtimer.type].gettime();
- exp = ktime_add_safe(now, exp);
- }
-
- alarm_start(&timr->it.alarm.alarmtimer, exp);
- return 0;
-}
-
/**
* alarmtimer_nsleep_wakeup - Wakeup function for alarm_timer_nsleep
* @alarm: ptr to alarm that fired
*
* Sets the alarm timer and sleeps until it is fired or interrupted.
*/
-static int alarmtimer_do_nsleep(struct alarm *alarm, ktime_t absexp)
+static int alarmtimer_do_nsleep(struct alarm *alarm, ktime_t absexp,
+ enum alarmtimer_type type)
{
+ struct restart_block *restart;
alarm->data = (void *)current;
do {
set_current_state(TASK_INTERRUPTIBLE);
__set_current_state(TASK_RUNNING);
- return (alarm->data == NULL);
-}
-
-
-/**
- * update_rmtp - Update remaining timespec value
- * @exp: expiration time
- * @type: timer type
- * @rmtp: user pointer to remaining timepsec value
- *
- * Helper function that fills in rmtp value with time between
- * now and the exp value
- */
-static int update_rmtp(ktime_t exp, enum alarmtimer_type type,
- struct timespec __user *rmtp)
-{
- struct timespec rmt;
- ktime_t rem;
-
- rem = ktime_sub(exp, alarm_bases[type].gettime());
-
- if (rem <= 0)
+ if (!alarm->data)
return 0;
- rmt = ktime_to_timespec(rem);
- if (copy_to_user(rmtp, &rmt, sizeof(*rmtp)))
- return -EFAULT;
+ if (freezing(current))
+ alarmtimer_freezerset(absexp, type);
+ restart = ¤t->restart_block;
+ if (restart->nanosleep.type != TT_NONE) {
+ struct timespec rmt;
+ ktime_t rem;
+
+ rem = ktime_sub(absexp, alarm_bases[type].gettime());
- return 1;
+ if (rem <= 0)
+ return 0;
+ rmt = ktime_to_timespec(rem);
+ return nanosleep_copyout(restart, &rmt);
+ }
+ return -ERESTART_RESTARTBLOCK;
}
/**
static long __sched alarm_timer_nsleep_restart(struct restart_block *restart)
{
enum alarmtimer_type type = restart->nanosleep.clockid;
- ktime_t exp;
- struct timespec __user *rmtp;
+ ktime_t exp = restart->nanosleep.expires;
struct alarm alarm;
- int ret = 0;
- exp = restart->nanosleep.expires;
alarm_init(&alarm, type, alarmtimer_nsleep_wakeup);
- if (alarmtimer_do_nsleep(&alarm, exp))
- goto out;
-
- if (freezing(current))
- alarmtimer_freezerset(exp, type);
-
- rmtp = restart->nanosleep.rmtp;
- if (rmtp) {
- ret = update_rmtp(exp, type, rmtp);
- if (ret <= 0)
- goto out;
- }
-
-
- /* The other values in restart are already filled in */
- ret = -ERESTART_RESTARTBLOCK;
-out:
- return ret;
+ return alarmtimer_do_nsleep(&alarm, exp, type);
}
/**
* Handles clock_nanosleep calls against _ALARM clockids
*/
static int alarm_timer_nsleep(const clockid_t which_clock, int flags,
- struct timespec64 *tsreq,
- struct timespec __user *rmtp)
+ const struct timespec64 *tsreq)
{
enum alarmtimer_type type = clock2alarm(which_clock);
- struct restart_block *restart;
+ struct restart_block *restart = ¤t->restart_block;
struct alarm alarm;
ktime_t exp;
int ret = 0;
exp = ktime_add(now, exp);
}
- if (alarmtimer_do_nsleep(&alarm, exp))
- goto out;
-
- if (freezing(current))
- alarmtimer_freezerset(exp, type);
+ ret = alarmtimer_do_nsleep(&alarm, exp, type);
+ if (ret != -ERESTART_RESTARTBLOCK)
+ return ret;
/* abs timers don't set remaining time or restart */
- if (flags == TIMER_ABSTIME) {
- ret = -ERESTARTNOHAND;
- goto out;
- }
+ if (flags == TIMER_ABSTIME)
+ return -ERESTARTNOHAND;
- if (rmtp) {
- ret = update_rmtp(exp, type, rmtp);
- if (ret <= 0)
- goto out;
- }
-
- restart = ¤t->restart_block;
restart->fn = alarm_timer_nsleep_restart;
restart->nanosleep.clockid = type;
restart->nanosleep.expires = exp;
- restart->nanosleep.rmtp = rmtp;
- ret = -ERESTART_RESTARTBLOCK;
-
-out:
return ret;
}
+const struct k_clock alarm_clock = {
+ .clock_getres = alarm_clock_getres,
+ .clock_get = alarm_clock_get,
+ .timer_create = alarm_timer_create,
+ .timer_set = common_timer_set,
+ .timer_del = common_timer_del,
+ .timer_get = common_timer_get,
+ .timer_arm = alarm_timer_arm,
+ .timer_rearm = alarm_timer_rearm,
+ .timer_forward = alarm_timer_forward,
+ .timer_remaining = alarm_timer_remaining,
+ .timer_try_to_cancel = alarm_timer_try_to_cancel,
+ .nsleep = alarm_timer_nsleep,
+};
+#endif /* CONFIG_POSIX_TIMERS */
+
/* Suspend hook structures */
static const struct dev_pm_ops alarmtimer_pm_ops = {
struct platform_device *pdev;
int error = 0;
int i;
- struct k_clock alarm_clock = {
- .clock_getres = alarm_clock_getres,
- .clock_get = alarm_clock_get,
- .timer_create = alarm_timer_create,
- .timer_set = alarm_timer_set,
- .timer_del = alarm_timer_del,
- .timer_get = alarm_timer_get,
- .nsleep = alarm_timer_nsleep,
- };
alarmtimer_rtc_timer_init();
- if (IS_ENABLED(CONFIG_POSIX_TIMERS)) {
- posix_timers_register_clock(CLOCK_REALTIME_ALARM, &alarm_clock);
- posix_timers_register_clock(CLOCK_BOOTTIME_ALARM, &alarm_clock);
- }
-
/* Initialize alarm bases */
alarm_bases[ALARM_REALTIME].base_clockid = CLOCK_REALTIME;
alarm_bases[ALARM_REALTIME].gettime = &ktime_get_real;
continue;
}
+ if (cs == curr_clocksource && cs->tick_stable)
+ cs->tick_stable(cs);
+
if (!(cs->flags & CLOCK_SOURCE_VALID_FOR_HRES) &&
(cs->flags & CLOCK_SOURCE_IS_CONTINUOUS) &&
(watchdog->flags & CLOCK_SOURCE_IS_CONTINUOUS)) {
#include <linux/sched/debug.h>
#include <linux/timer.h>
#include <linux/freezer.h>
+#include <linux/compat.h>
#include <linux/uaccess.h>
}
EXPORT_SYMBOL_GPL(hrtimer_init_sleeper);
+int nanosleep_copyout(struct restart_block *restart, struct timespec *ts)
+{
+ switch(restart->nanosleep.type) {
+#ifdef CONFIG_COMPAT
+ case TT_COMPAT:
+ if (compat_put_timespec(ts, restart->nanosleep.compat_rmtp))
+ return -EFAULT;
+ break;
+#endif
+ case TT_NATIVE:
+ if (copy_to_user(restart->nanosleep.rmtp, ts, sizeof(struct timespec)))
+ return -EFAULT;
+ break;
+ default:
+ BUG();
+ }
+ return -ERESTART_RESTARTBLOCK;
+}
+
static int __sched do_nanosleep(struct hrtimer_sleeper *t, enum hrtimer_mode mode)
{
+ struct restart_block *restart;
+
hrtimer_init_sleeper(t, current);
do {
__set_current_state(TASK_RUNNING);
- return t->task == NULL;
-}
-
-static int update_rmtp(struct hrtimer *timer, struct timespec __user *rmtp)
-{
- struct timespec rmt;
- ktime_t rem;
-
- rem = hrtimer_expires_remaining(timer);
- if (rem <= 0)
+ if (!t->task)
return 0;
- rmt = ktime_to_timespec(rem);
- if (copy_to_user(rmtp, &rmt, sizeof(*rmtp)))
- return -EFAULT;
+ restart = ¤t->restart_block;
+ if (restart->nanosleep.type != TT_NONE) {
+ ktime_t rem = hrtimer_expires_remaining(&t->timer);
+ struct timespec rmt;
- return 1;
+ if (rem <= 0)
+ return 0;
+ rmt = ktime_to_timespec(rem);
+
+ return nanosleep_copyout(restart, &rmt);
+ }
+ return -ERESTART_RESTARTBLOCK;
}
-long __sched hrtimer_nanosleep_restart(struct restart_block *restart)
+static long __sched hrtimer_nanosleep_restart(struct restart_block *restart)
{
struct hrtimer_sleeper t;
- struct timespec __user *rmtp;
- int ret = 0;
+ int ret;
hrtimer_init_on_stack(&t.timer, restart->nanosleep.clockid,
HRTIMER_MODE_ABS);
hrtimer_set_expires_tv64(&t.timer, restart->nanosleep.expires);
- if (do_nanosleep(&t, HRTIMER_MODE_ABS))
- goto out;
-
- rmtp = restart->nanosleep.rmtp;
- if (rmtp) {
- ret = update_rmtp(&t.timer, rmtp);
- if (ret <= 0)
- goto out;
- }
-
- /* The other values in restart are already filled in */
- ret = -ERESTART_RESTARTBLOCK;
-out:
+ ret = do_nanosleep(&t, HRTIMER_MODE_ABS);
destroy_hrtimer_on_stack(&t.timer);
return ret;
}
-long hrtimer_nanosleep(struct timespec64 *rqtp, struct timespec __user *rmtp,
+long hrtimer_nanosleep(const struct timespec64 *rqtp,
const enum hrtimer_mode mode, const clockid_t clockid)
{
struct restart_block *restart;
hrtimer_init_on_stack(&t.timer, clockid, mode);
hrtimer_set_expires_range_ns(&t.timer, timespec64_to_ktime(*rqtp), slack);
- if (do_nanosleep(&t, mode))
+ ret = do_nanosleep(&t, mode);
+ if (ret != -ERESTART_RESTARTBLOCK)
goto out;
/* Absolute timers do not update the rmtp value and restart: */
goto out;
}
- if (rmtp) {
- ret = update_rmtp(&t.timer, rmtp);
- if (ret <= 0)
- goto out;
- }
-
restart = ¤t->restart_block;
restart->fn = hrtimer_nanosleep_restart;
restart->nanosleep.clockid = t.timer.base->clockid;
- restart->nanosleep.rmtp = rmtp;
restart->nanosleep.expires = hrtimer_get_expires_tv64(&t.timer);
-
- ret = -ERESTART_RESTARTBLOCK;
out:
destroy_hrtimer_on_stack(&t.timer);
return ret;
if (!timespec64_valid(&tu64))
return -EINVAL;
- return hrtimer_nanosleep(&tu64, rmtp, HRTIMER_MODE_REL, CLOCK_MONOTONIC);
+ current->restart_block.nanosleep.type = rmtp ? TT_NATIVE : TT_NONE;
+ current->restart_block.nanosleep.rmtp = rmtp;
+ return hrtimer_nanosleep(&tu64, HRTIMER_MODE_REL, CLOCK_MONOTONIC);
+}
+
+#ifdef CONFIG_COMPAT
+
+COMPAT_SYSCALL_DEFINE2(nanosleep, struct compat_timespec __user *, rqtp,
+ struct compat_timespec __user *, rmtp)
+{
+ struct timespec64 tu64;
+ struct timespec tu;
+
+ if (compat_get_timespec(&tu, rqtp))
+ return -EFAULT;
+
+ tu64 = timespec_to_timespec64(tu);
+ if (!timespec64_valid(&tu64))
+ return -EINVAL;
+
+ current->restart_block.nanosleep.type = rmtp ? TT_COMPAT : TT_NONE;
+ current->restart_block.nanosleep.compat_rmtp = rmtp;
+ return hrtimer_nanosleep(&tu64, HRTIMER_MODE_REL, CLOCK_MONOTONIC);
}
+#endif
/*
* Functions related to boot-time initialization:
#include <linux/posix-timers.h>
#include <linux/hrtimer.h>
#include <trace/events/timer.h>
+#include <linux/compat.h>
#include <linux/uaccess.h>
return error;
}
+#ifdef CONFIG_COMPAT
+COMPAT_SYSCALL_DEFINE2(getitimer, int, which,
+ struct compat_itimerval __user *, it)
+{
+ struct itimerval kit;
+ int error = do_getitimer(which, &kit);
+
+ if (!error && put_compat_itimerval(it, &kit))
+ error = -EFAULT;
+ return error;
+}
+#endif
+
/*
* The timer is automagically restarted, when interval != 0
u64 oval, nval, ointerval, ninterval;
struct cpu_itimer *it = &tsk->signal->it[clock_id];
- nval = timeval_to_ns(&value->it_value);
- ninterval = timeval_to_ns(&value->it_interval);
+ /*
+ * Use the to_ktime conversion because that clamps the maximum
+ * value to KTIME_MAX and avoid multiplication overflows.
+ */
+ nval = ktime_to_ns(timeval_to_ktime(value->it_value));
+ ninterval = ktime_to_ns(timeval_to_ktime(value->it_interval));
spin_lock_irq(&tsk->sighand->siglock);
return -EFAULT;
return 0;
}
+
+#ifdef CONFIG_COMPAT
+COMPAT_SYSCALL_DEFINE3(setitimer, int, which,
+ struct compat_itimerval __user *, in,
+ struct compat_itimerval __user *, out)
+{
+ struct itimerval kin, kout;
+ int error;
+
+ if (in) {
+ if (get_compat_itimerval(&kin, in))
+ return -EFAULT;
+ } else {
+ memset(&kin, 0, sizeof(kin));
+ }
+
+ error = do_setitimer(which, &kin, out ? &kout : NULL);
+ if (error || !out)
+ return error;
+ if (put_compat_itimerval(out, &kout))
+ return -EFAULT;
+ return 0;
+}
+#endif
#include <linux/syscalls.h>
#include <linux/uaccess.h>
+#include "posix-timers.h"
+
static void delete_clock(struct kref *kref);
/*
return result;
}
-static int posix_clock_fasync(int fd, struct file *fp, int on)
-{
- struct posix_clock *clk = get_posix_clock(fp);
- int err = 0;
-
- if (!clk)
- return -ENODEV;
-
- if (clk->ops.fasync)
- err = clk->ops.fasync(clk, fd, fp, on);
-
- put_posix_clock(clk);
-
- return err;
-}
-
-static int posix_clock_mmap(struct file *fp, struct vm_area_struct *vma)
-{
- struct posix_clock *clk = get_posix_clock(fp);
- int err = -ENODEV;
-
- if (!clk)
- return -ENODEV;
-
- if (clk->ops.mmap)
- err = clk->ops.mmap(clk, vma);
-
- put_posix_clock(clk);
-
- return err;
-}
-
static long posix_clock_ioctl(struct file *fp,
unsigned int cmd, unsigned long arg)
{
.unlocked_ioctl = posix_clock_ioctl,
.open = posix_clock_open,
.release = posix_clock_release,
- .fasync = posix_clock_fasync,
- .mmap = posix_clock_mmap,
#ifdef CONFIG_COMPAT
.compat_ioctl = posix_clock_compat_ioctl,
#endif
return err;
}
-static int pc_timer_create(struct k_itimer *kit)
-{
- clockid_t id = kit->it_clock;
- struct posix_clock_desc cd;
- int err;
-
- err = get_clock_desc(id, &cd);
- if (err)
- return err;
-
- if (cd.clk->ops.timer_create)
- err = cd.clk->ops.timer_create(cd.clk, kit);
- else
- err = -EOPNOTSUPP;
-
- put_clock_desc(&cd);
-
- return err;
-}
-
-static int pc_timer_delete(struct k_itimer *kit)
-{
- clockid_t id = kit->it_clock;
- struct posix_clock_desc cd;
- int err;
-
- err = get_clock_desc(id, &cd);
- if (err)
- return err;
-
- if (cd.clk->ops.timer_delete)
- err = cd.clk->ops.timer_delete(cd.clk, kit);
- else
- err = -EOPNOTSUPP;
-
- put_clock_desc(&cd);
-
- return err;
-}
-
-static void pc_timer_gettime(struct k_itimer *kit, struct itimerspec64 *ts)
-{
- clockid_t id = kit->it_clock;
- struct posix_clock_desc cd;
-
- if (get_clock_desc(id, &cd))
- return;
-
- if (cd.clk->ops.timer_gettime)
- cd.clk->ops.timer_gettime(cd.clk, kit, ts);
-
- put_clock_desc(&cd);
-}
-
-static int pc_timer_settime(struct k_itimer *kit, int flags,
- struct itimerspec64 *ts, struct itimerspec64 *old)
-{
- clockid_t id = kit->it_clock;
- struct posix_clock_desc cd;
- int err;
-
- err = get_clock_desc(id, &cd);
- if (err)
- return err;
-
- if (cd.clk->ops.timer_settime)
- err = cd.clk->ops.timer_settime(cd.clk, kit, flags, ts, old);
- else
- err = -EOPNOTSUPP;
-
- put_clock_desc(&cd);
-
- return err;
-}
-
-struct k_clock clock_posix_dynamic = {
+const struct k_clock clock_posix_dynamic = {
.clock_getres = pc_clock_getres,
.clock_set = pc_clock_settime,
.clock_get = pc_clock_gettime,
.clock_adj = pc_clock_adjtime,
- .timer_create = pc_timer_create,
- .timer_set = pc_timer_settime,
- .timer_del = pc_timer_delete,
- .timer_get = pc_timer_gettime,
};
#include <trace/events/timer.h>
#include <linux/tick.h>
#include <linux/workqueue.h>
+#include <linux/compat.h>
+
+#include "posix-timers.h"
+
+static void posix_cpu_timer_rearm(struct k_itimer *timer);
/*
* Called after updating RLIMIT_CPU to run cpu timer and update
if (CPUCLOCK_WHICH(new_timer->it_clock) >= CPUCLOCK_MAX)
return -EINVAL;
+ new_timer->kclock = &clock_posix_cpu;
+
INIT_LIST_HEAD(&new_timer->it.cpu.entry);
rcu_read_lock();
* reload the timer. But we need to keep it
* ticking in case the signal is deliverable next time.
*/
- posix_cpu_timer_schedule(timer);
+ posix_cpu_timer_rearm(timer);
+ ++timer->it_requeue_pending;
}
}
WARN_ON_ONCE(p == NULL);
- new_expires = timespec64_to_ns(&new->it_value);
+ /*
+ * Use the to_ktime conversion because that clamps the maximum
+ * value to KTIME_MAX and avoid multiplication overflows.
+ */
+ new_expires = ktime_to_ns(timespec64_to_ktime(new->it_value));
/*
* Protect against sighand release/switch in exit/exec and p->cpu_timers
*/
itp->it_interval = ns_to_timespec64(timer->it.cpu.incr);
- if (timer->it.cpu.expires == 0) { /* Timer not armed at all. */
- itp->it_value.tv_sec = itp->it_value.tv_nsec = 0;
+ if (!timer->it.cpu.expires)
return;
- }
/*
* Sample the clock to take the difference with the expiry time.
* Call the timer disarmed, nothing else to do.
*/
timer->it.cpu.expires = 0;
- itp->it_value = ns_to_timespec64(timer->it.cpu.expires);
return;
} else {
cpu_timer_sample_group(timer->it_clock, p, &now);
}
/*
- * This is called from the signal code (via do_schedule_next_timer)
+ * This is called from the signal code (via posixtimer_rearm)
* when the last timer signal was delivered and we have to reload the timer.
*/
-void posix_cpu_timer_schedule(struct k_itimer *timer)
+static void posix_cpu_timer_rearm(struct k_itimer *timer)
{
struct sighand_struct *sighand;
unsigned long flags;
cpu_clock_sample(timer->it_clock, p, &now);
bump_cpu_timer(timer, now);
if (unlikely(p->exit_state))
- goto out;
+ return;
/* Protect timer list r/w in arm_timer() */
sighand = lock_task_sighand(p, &flags);
if (!sighand)
- goto out;
+ return;
} else {
/*
* Protect arm_timer() and timer sampling in case of call to
* We can't even collect a sample any more.
*/
timer->it.cpu.expires = 0;
- goto out;
+ return;
} else if (unlikely(p->exit_state) && thread_group_empty(p)) {
- unlock_task_sighand(p, &flags);
- /* Optimizations: if the process is dying, no need to rearm */
- goto out;
+ /* If the process is dying, no need to rearm */
+ goto unlock;
}
cpu_timer_sample_group(timer->it_clock, p, &now);
bump_cpu_timer(timer, now);
*/
WARN_ON_ONCE(!irqs_disabled());
arm_timer(timer);
+unlock:
unlock_task_sighand(p, &flags);
-
-out:
- timer->it_overrun_last = timer->it_overrun;
- timer->it_overrun = -1;
- ++timer->it_requeue_pending;
}
/**
}
static int do_cpu_nanosleep(const clockid_t which_clock, int flags,
- struct timespec64 *rqtp, struct itimerspec64 *it)
+ const struct timespec64 *rqtp)
{
+ struct itimerspec64 it;
struct k_itimer timer;
+ u64 expires;
int error;
/*
timer.it_process = current;
if (!error) {
static struct itimerspec64 zero_it;
+ struct restart_block *restart;
- memset(it, 0, sizeof *it);
- it->it_value = *rqtp;
+ memset(&it, 0, sizeof(it));
+ it.it_value = *rqtp;
spin_lock_irq(&timer.it_lock);
- error = posix_cpu_timer_set(&timer, flags, it, NULL);
+ error = posix_cpu_timer_set(&timer, flags, &it, NULL);
if (error) {
spin_unlock_irq(&timer.it_lock);
return error;
/*
* We were interrupted by a signal.
*/
- *rqtp = ns_to_timespec64(timer.it.cpu.expires);
- error = posix_cpu_timer_set(&timer, 0, &zero_it, it);
+ expires = timer.it.cpu.expires;
+ error = posix_cpu_timer_set(&timer, 0, &zero_it, &it);
if (!error) {
/*
* Timer is now unarmed, deletion can not fail.
spin_unlock_irq(&timer.it_lock);
}
- if ((it->it_value.tv_sec | it->it_value.tv_nsec) == 0) {
+ if ((it.it_value.tv_sec | it.it_value.tv_nsec) == 0) {
/*
* It actually did fire already.
*/
}
error = -ERESTART_RESTARTBLOCK;
+ /*
+ * Report back to the user the time still remaining.
+ */
+ restart = ¤t->restart_block;
+ restart->nanosleep.expires = expires;
+ if (restart->nanosleep.type != TT_NONE) {
+ struct timespec ts;
+
+ ts = timespec64_to_timespec(it.it_value);
+ error = nanosleep_copyout(restart, &ts);
+ }
}
return error;
static long posix_cpu_nsleep_restart(struct restart_block *restart_block);
static int posix_cpu_nsleep(const clockid_t which_clock, int flags,
- struct timespec64 *rqtp, struct timespec __user *rmtp)
+ const struct timespec64 *rqtp)
{
struct restart_block *restart_block = ¤t->restart_block;
- struct itimerspec64 it;
- struct timespec ts;
int error;
/*
CPUCLOCK_PID(which_clock) == task_pid_vnr(current)))
return -EINVAL;
- error = do_cpu_nanosleep(which_clock, flags, rqtp, &it);
+ error = do_cpu_nanosleep(which_clock, flags, rqtp);
if (error == -ERESTART_RESTARTBLOCK) {
if (flags & TIMER_ABSTIME)
return -ERESTARTNOHAND;
- /*
- * Report back to the user the time still remaining.
- */
- ts = timespec64_to_timespec(it.it_value);
- if (rmtp && copy_to_user(rmtp, &ts, sizeof(*rmtp)))
- return -EFAULT;
restart_block->fn = posix_cpu_nsleep_restart;
restart_block->nanosleep.clockid = which_clock;
- restart_block->nanosleep.rmtp = rmtp;
- restart_block->nanosleep.expires = timespec64_to_ns(rqtp);
}
return error;
}
static long posix_cpu_nsleep_restart(struct restart_block *restart_block)
{
clockid_t which_clock = restart_block->nanosleep.clockid;
- struct itimerspec64 it;
struct timespec64 t;
- struct timespec tmp;
- int error;
t = ns_to_timespec64(restart_block->nanosleep.expires);
- error = do_cpu_nanosleep(which_clock, TIMER_ABSTIME, &t, &it);
-
- if (error == -ERESTART_RESTARTBLOCK) {
- struct timespec __user *rmtp = restart_block->nanosleep.rmtp;
- /*
- * Report back to the user the time still remaining.
- */
- tmp = timespec64_to_timespec(it.it_value);
- if (rmtp && copy_to_user(rmtp, &tmp, sizeof(*rmtp)))
- return -EFAULT;
-
- restart_block->nanosleep.expires = timespec64_to_ns(&t);
- }
- return error;
-
+ return do_cpu_nanosleep(which_clock, TIMER_ABSTIME, &t);
}
#define PROCESS_CLOCK MAKE_PROCESS_CPUCLOCK(0, CPUCLOCK_SCHED)
return posix_cpu_timer_create(timer);
}
static int process_cpu_nsleep(const clockid_t which_clock, int flags,
- struct timespec64 *rqtp,
- struct timespec __user *rmtp)
-{
- return posix_cpu_nsleep(PROCESS_CLOCK, flags, rqtp, rmtp);
-}
-static long process_cpu_nsleep_restart(struct restart_block *restart_block)
+ const struct timespec64 *rqtp)
{
- return -EINVAL;
+ return posix_cpu_nsleep(PROCESS_CLOCK, flags, rqtp);
}
static int thread_cpu_clock_getres(const clockid_t which_clock,
struct timespec64 *tp)
return posix_cpu_timer_create(timer);
}
-struct k_clock clock_posix_cpu = {
+const struct k_clock clock_posix_cpu = {
.clock_getres = posix_cpu_clock_getres,
.clock_set = posix_cpu_clock_set,
.clock_get = posix_cpu_clock_get,
.timer_create = posix_cpu_timer_create,
.nsleep = posix_cpu_nsleep,
- .nsleep_restart = posix_cpu_nsleep_restart,
.timer_set = posix_cpu_timer_set,
.timer_del = posix_cpu_timer_del,
.timer_get = posix_cpu_timer_get,
+ .timer_rearm = posix_cpu_timer_rearm,
};
-static __init int init_posix_cpu_timers(void)
-{
- struct k_clock process = {
- .clock_getres = process_cpu_clock_getres,
- .clock_get = process_cpu_clock_get,
- .timer_create = process_cpu_timer_create,
- .nsleep = process_cpu_nsleep,
- .nsleep_restart = process_cpu_nsleep_restart,
- };
- struct k_clock thread = {
- .clock_getres = thread_cpu_clock_getres,
- .clock_get = thread_cpu_clock_get,
- .timer_create = thread_cpu_timer_create,
- };
-
- posix_timers_register_clock(CLOCK_PROCESS_CPUTIME_ID, &process);
- posix_timers_register_clock(CLOCK_THREAD_CPUTIME_ID, &thread);
+const struct k_clock clock_process = {
+ .clock_getres = process_cpu_clock_getres,
+ .clock_get = process_cpu_clock_get,
+ .timer_create = process_cpu_timer_create,
+ .nsleep = process_cpu_nsleep,
+};
- return 0;
-}
-__initcall(init_posix_cpu_timers);
+const struct k_clock clock_thread = {
+ .clock_getres = thread_cpu_clock_getres,
+ .clock_get = thread_cpu_clock_get,
+ .timer_create = thread_cpu_timer_create,
+};
#include <linux/ktime.h>
#include <linux/timekeeping.h>
#include <linux/posix-timers.h>
+#include <linux/compat.h>
asmlinkage long sys_ni_posix_timers(void)
{
}
#define SYS_NI(name) SYSCALL_ALIAS(sys_##name, sys_ni_posix_timers)
+#define COMPAT_SYS_NI(name) SYSCALL_ALIAS(compat_sys_##name, sys_ni_posix_timers)
SYS_NI(timer_create);
SYS_NI(timer_gettime);
#ifdef __ARCH_WANT_SYS_ALARM
SYS_NI(alarm);
#endif
+COMPAT_SYS_NI(timer_create);
+COMPAT_SYS_NI(clock_adjtime);
+COMPAT_SYS_NI(timer_settime);
+COMPAT_SYS_NI(timer_gettime);
+COMPAT_SYS_NI(getitimer);
+COMPAT_SYS_NI(setitimer);
/*
* We preserve minimal support for CLOCK_REALTIME and CLOCK_MONOTONIC
case CLOCK_REALTIME:
case CLOCK_MONOTONIC:
case CLOCK_BOOTTIME:
- if (copy_from_user(&t, rqtp, sizeof (struct timespec)))
- return -EFAULT;
- t64 = timespec_to_timespec64(t);
- if (!timespec64_valid(&t64))
- return -EINVAL;
- return hrtimer_nanosleep(&t64, rmtp, flags & TIMER_ABSTIME ?
- HRTIMER_MODE_ABS : HRTIMER_MODE_REL,
- which_clock);
+ break;
default:
return -EINVAL;
}
+
+ if (copy_from_user(&t, rqtp, sizeof (struct timespec)))
+ return -EFAULT;
+ t64 = timespec_to_timespec64(t);
+ if (!timespec64_valid(&t64))
+ return -EINVAL;
+ if (flags & TIMER_ABSTIME)
+ rmtp = NULL;
+ current->restart_block.nanosleep.type = rmtp ? TT_NATIVE : TT_NONE;
+ current->restart_block.nanosleep.rmtp = rmtp;
+ return hrtimer_nanosleep(&t64, flags & TIMER_ABSTIME ?
+ HRTIMER_MODE_ABS : HRTIMER_MODE_REL,
+ which_clock);
}
#ifdef CONFIG_COMPAT
-long clock_nanosleep_restart(struct restart_block *restart_block)
+COMPAT_SYSCALL_DEFINE2(clock_settime, const clockid_t, which_clock,
+ struct compat_timespec __user *, tp)
+{
+ struct timespec64 new_tp64;
+ struct timespec new_tp;
+
+ if (which_clock != CLOCK_REALTIME)
+ return -EINVAL;
+ if (compat_get_timespec(&new_tp, tp))
+ return -EFAULT;
+
+ new_tp64 = timespec_to_timespec64(new_tp);
+ return do_sys_settimeofday64(&new_tp64, NULL);
+}
+
+COMPAT_SYSCALL_DEFINE2(clock_gettime, const clockid_t, which_clock,
+ struct compat_timespec __user *,tp)
{
- return hrtimer_nanosleep_restart(restart_block);
+ struct timespec64 kernel_tp64;
+ struct timespec kernel_tp;
+
+ switch (which_clock) {
+ case CLOCK_REALTIME: ktime_get_real_ts64(&kernel_tp64); break;
+ case CLOCK_MONOTONIC: ktime_get_ts64(&kernel_tp64); break;
+ case CLOCK_BOOTTIME: get_monotonic_boottime64(&kernel_tp64); break;
+ default: return -EINVAL;
+ }
+
+ kernel_tp = timespec64_to_timespec(kernel_tp64);
+ if (compat_put_timespec(&kernel_tp, tp))
+ return -EFAULT;
+ return 0;
+}
+
+COMPAT_SYSCALL_DEFINE2(clock_getres, const clockid_t, which_clock,
+ struct compat_timespec __user *, tp)
+{
+ struct timespec rtn_tp = {
+ .tv_sec = 0,
+ .tv_nsec = hrtimer_resolution,
+ };
+
+ switch (which_clock) {
+ case CLOCK_REALTIME:
+ case CLOCK_MONOTONIC:
+ case CLOCK_BOOTTIME:
+ if (compat_put_timespec(&rtn_tp, tp))
+ return -EFAULT;
+ return 0;
+ default:
+ return -EINVAL;
+ }
+}
+COMPAT_SYSCALL_DEFINE4(clock_nanosleep, clockid_t, which_clock, int, flags,
+ struct compat_timespec __user *, rqtp,
+ struct compat_timespec __user *, rmtp)
+{
+ struct timespec64 t64;
+ struct timespec t;
+
+ switch (which_clock) {
+ case CLOCK_REALTIME:
+ case CLOCK_MONOTONIC:
+ case CLOCK_BOOTTIME:
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ if (compat_get_timespec(&t, rqtp))
+ return -EFAULT;
+ t64 = timespec_to_timespec64(t);
+ if (!timespec64_valid(&t64))
+ return -EINVAL;
+ if (flags & TIMER_ABSTIME)
+ rmtp = NULL;
+ current->restart_block.nanosleep.type = rmtp ? TT_COMPAT : TT_NONE;
+ current->restart_block.nanosleep.compat_rmtp = rmtp;
+ return hrtimer_nanosleep(&t64, flags & TIMER_ABSTIME ?
+ HRTIMER_MODE_ABS : HRTIMER_MODE_REL,
+ which_clock);
}
#endif
#include <linux/workqueue.h>
#include <linux/export.h>
#include <linux/hashtable.h>
+#include <linux/compat.h>
#include "timekeeping.h"
+#include "posix-timers.h"
/*
* Management arrays for POSIX timers. Timers are now kept in static hash table
static DEFINE_HASHTABLE(posix_timers_hashtable, 9);
static DEFINE_SPINLOCK(hash_lock);
+static const struct k_clock * const posix_clocks[];
+static const struct k_clock *clockid_to_kclock(const clockid_t id);
+static const struct k_clock clock_realtime, clock_monotonic;
+
/*
* we assume that the new SIGEV_THREAD_ID shares no bits with the other
* SIGEV values. Here we put out an error if this assumption fails.
* have is CLOCK_REALTIME and its high res counter part, both of
* which we beg off on and pass to do_sys_settimeofday().
*/
-
-static struct k_clock posix_clocks[MAX_CLOCKS];
-
-/*
- * These ones are defined below.
- */
-static int common_nsleep(const clockid_t, int flags, struct timespec64 *t,
- struct timespec __user *rmtp);
-static int common_timer_create(struct k_itimer *new_timer);
-static void common_timer_get(struct k_itimer *, struct itimerspec64 *);
-static int common_timer_set(struct k_itimer *, int,
- struct itimerspec64 *, struct itimerspec64 *);
-static int common_timer_del(struct k_itimer *timer);
-
-static enum hrtimer_restart posix_timer_fn(struct hrtimer *data);
-
static struct k_itimer *__lock_timer(timer_t timer_id, unsigned long *flags);
#define lock_timer(tid, flags) \
*/
static __init int init_posix_timers(void)
{
- struct k_clock clock_realtime = {
- .clock_getres = posix_get_hrtimer_res,
- .clock_get = posix_clock_realtime_get,
- .clock_set = posix_clock_realtime_set,
- .clock_adj = posix_clock_realtime_adj,
- .nsleep = common_nsleep,
- .nsleep_restart = hrtimer_nanosleep_restart,
- .timer_create = common_timer_create,
- .timer_set = common_timer_set,
- .timer_get = common_timer_get,
- .timer_del = common_timer_del,
- };
- struct k_clock clock_monotonic = {
- .clock_getres = posix_get_hrtimer_res,
- .clock_get = posix_ktime_get_ts,
- .nsleep = common_nsleep,
- .nsleep_restart = hrtimer_nanosleep_restart,
- .timer_create = common_timer_create,
- .timer_set = common_timer_set,
- .timer_get = common_timer_get,
- .timer_del = common_timer_del,
- };
- struct k_clock clock_monotonic_raw = {
- .clock_getres = posix_get_hrtimer_res,
- .clock_get = posix_get_monotonic_raw,
- };
- struct k_clock clock_realtime_coarse = {
- .clock_getres = posix_get_coarse_res,
- .clock_get = posix_get_realtime_coarse,
- };
- struct k_clock clock_monotonic_coarse = {
- .clock_getres = posix_get_coarse_res,
- .clock_get = posix_get_monotonic_coarse,
- };
- struct k_clock clock_tai = {
- .clock_getres = posix_get_hrtimer_res,
- .clock_get = posix_get_tai,
- .nsleep = common_nsleep,
- .nsleep_restart = hrtimer_nanosleep_restart,
- .timer_create = common_timer_create,
- .timer_set = common_timer_set,
- .timer_get = common_timer_get,
- .timer_del = common_timer_del,
- };
- struct k_clock clock_boottime = {
- .clock_getres = posix_get_hrtimer_res,
- .clock_get = posix_get_boottime,
- .nsleep = common_nsleep,
- .nsleep_restart = hrtimer_nanosleep_restart,
- .timer_create = common_timer_create,
- .timer_set = common_timer_set,
- .timer_get = common_timer_get,
- .timer_del = common_timer_del,
- };
-
- posix_timers_register_clock(CLOCK_REALTIME, &clock_realtime);
- posix_timers_register_clock(CLOCK_MONOTONIC, &clock_monotonic);
- posix_timers_register_clock(CLOCK_MONOTONIC_RAW, &clock_monotonic_raw);
- posix_timers_register_clock(CLOCK_REALTIME_COARSE, &clock_realtime_coarse);
- posix_timers_register_clock(CLOCK_MONOTONIC_COARSE, &clock_monotonic_coarse);
- posix_timers_register_clock(CLOCK_BOOTTIME, &clock_boottime);
- posix_timers_register_clock(CLOCK_TAI, &clock_tai);
-
posix_timers_cache = kmem_cache_create("posix_timers_cache",
sizeof (struct k_itimer), 0, SLAB_PANIC,
NULL);
return 0;
}
-
__initcall(init_posix_timers);
-static void schedule_next_timer(struct k_itimer *timr)
+static void common_hrtimer_rearm(struct k_itimer *timr)
{
struct hrtimer *timer = &timr->it.real.timer;
- if (timr->it.real.interval == 0)
+ if (!timr->it_interval)
return;
timr->it_overrun += (unsigned int) hrtimer_forward(timer,
timer->base->get_time(),
- timr->it.real.interval);
-
- timr->it_overrun_last = timr->it_overrun;
- timr->it_overrun = -1;
- ++timr->it_requeue_pending;
+ timr->it_interval);
hrtimer_restart(timer);
}
* To protect against the timer going away while the interrupt is queued,
* we require that the it_requeue_pending flag be set.
*/
-void do_schedule_next_timer(struct siginfo *info)
+void posixtimer_rearm(struct siginfo *info)
{
struct k_itimer *timr;
unsigned long flags;
timr = lock_timer(info->si_tid, &flags);
+ if (!timr)
+ return;
- if (timr && timr->it_requeue_pending == info->si_sys_private) {
- if (timr->it_clock < 0)
- posix_cpu_timer_schedule(timr);
- else
- schedule_next_timer(timr);
+ if (timr->it_requeue_pending == info->si_sys_private) {
+ timr->kclock->timer_rearm(timr);
+
+ timr->it_active = 1;
+ timr->it_overrun_last = timr->it_overrun;
+ timr->it_overrun = -1;
+ ++timr->it_requeue_pending;
info->si_overrun += timr->it_overrun_last;
}
- if (timr)
- unlock_timer(timr, flags);
+ unlock_timer(timr, flags);
}
int posix_timer_event(struct k_itimer *timr, int si_private)
int shared, ret = -1;
/*
* FIXME: if ->sigq is queued we can race with
- * dequeue_signal()->do_schedule_next_timer().
+ * dequeue_signal()->posixtimer_rearm().
*
* If dequeue_signal() sees the "right" value of
- * si_sys_private it calls do_schedule_next_timer().
+ * si_sys_private it calls posixtimer_rearm().
* We re-queue ->sigq and drop ->it_lock().
- * do_schedule_next_timer() locks the timer
+ * posixtimer_rearm() locks the timer
* and re-schedules it while ->sigq is pending.
* Not really bad, but not that we want.
*/
/* If we failed to send the signal the timer stops. */
return ret > 0;
}
-EXPORT_SYMBOL_GPL(posix_timer_event);
/*
* This function gets called when a POSIX.1b interval timer expires. It
timr = container_of(timer, struct k_itimer, it.real.timer);
spin_lock_irqsave(&timr->it_lock, flags);
- if (timr->it.real.interval != 0)
+ timr->it_active = 0;
+ if (timr->it_interval != 0)
si_private = ++timr->it_requeue_pending;
if (posix_timer_event(timr, si_private)) {
* we will not get a call back to restart it AND
* it should be restarted.
*/
- if (timr->it.real.interval != 0) {
+ if (timr->it_interval != 0) {
ktime_t now = hrtimer_cb_get_time(timer);
/*
{
ktime_t kj = NSEC_PER_SEC / HZ;
- if (timr->it.real.interval < kj)
+ if (timr->it_interval < kj)
now = ktime_add(now, kj);
}
#endif
timr->it_overrun += (unsigned int)
hrtimer_forward(timer, now,
- timr->it.real.interval);
+ timr->it_interval);
ret = HRTIMER_RESTART;
++timr->it_requeue_pending;
+ timr->it_active = 1;
}
}
return task_pid(rtn);
}
-void posix_timers_register_clock(const clockid_t clock_id,
- struct k_clock *new_clock)
-{
- if ((unsigned) clock_id >= MAX_CLOCKS) {
- printk(KERN_WARNING "POSIX clock register failed for clock_id %d\n",
- clock_id);
- return;
- }
-
- if (!new_clock->clock_get) {
- printk(KERN_WARNING "POSIX clock id %d lacks clock_get()\n",
- clock_id);
- return;
- }
- if (!new_clock->clock_getres) {
- printk(KERN_WARNING "POSIX clock id %d lacks clock_getres()\n",
- clock_id);
- return;
- }
-
- posix_clocks[clock_id] = *new_clock;
-}
-EXPORT_SYMBOL_GPL(posix_timers_register_clock);
-
static struct k_itimer * alloc_posix_timer(void)
{
struct k_itimer *tmr;
call_rcu(&tmr->it.rcu, k_itimer_rcu_free);
}
-static struct k_clock *clockid_to_kclock(const clockid_t id)
-{
- if (id < 0)
- return (id & CLOCKFD_MASK) == CLOCKFD ?
- &clock_posix_dynamic : &clock_posix_cpu;
-
- if (id >= MAX_CLOCKS || !posix_clocks[id].clock_getres)
- return NULL;
- return &posix_clocks[id];
-}
-
static int common_timer_create(struct k_itimer *new_timer)
{
hrtimer_init(&new_timer->it.real.timer, new_timer->it_clock, 0);
}
/* Create a POSIX.1b interval timer. */
-
-SYSCALL_DEFINE3(timer_create, const clockid_t, which_clock,
- struct sigevent __user *, timer_event_spec,
- timer_t __user *, created_timer_id)
+static int do_timer_create(clockid_t which_clock, struct sigevent *event,
+ timer_t __user *created_timer_id)
{
- struct k_clock *kc = clockid_to_kclock(which_clock);
+ const struct k_clock *kc = clockid_to_kclock(which_clock);
struct k_itimer *new_timer;
int error, new_timer_id;
- sigevent_t event;
int it_id_set = IT_ID_NOT_SET;
if (!kc)
it_id_set = IT_ID_SET;
new_timer->it_id = (timer_t) new_timer_id;
new_timer->it_clock = which_clock;
+ new_timer->kclock = kc;
new_timer->it_overrun = -1;
- if (timer_event_spec) {
- if (copy_from_user(&event, timer_event_spec, sizeof (event))) {
- error = -EFAULT;
- goto out;
- }
+ if (event) {
rcu_read_lock();
- new_timer->it_pid = get_pid(good_sigevent(&event));
+ new_timer->it_pid = get_pid(good_sigevent(event));
rcu_read_unlock();
if (!new_timer->it_pid) {
error = -EINVAL;
goto out;
}
+ new_timer->it_sigev_notify = event->sigev_notify;
+ new_timer->sigq->info.si_signo = event->sigev_signo;
+ new_timer->sigq->info.si_value = event->sigev_value;
} else {
- memset(&event.sigev_value, 0, sizeof(event.sigev_value));
- event.sigev_notify = SIGEV_SIGNAL;
- event.sigev_signo = SIGALRM;
- event.sigev_value.sival_int = new_timer->it_id;
+ new_timer->it_sigev_notify = SIGEV_SIGNAL;
+ new_timer->sigq->info.si_signo = SIGALRM;
+ memset(&new_timer->sigq->info.si_value, 0, sizeof(sigval_t));
+ new_timer->sigq->info.si_value.sival_int = new_timer->it_id;
new_timer->it_pid = get_pid(task_tgid(current));
}
- new_timer->it_sigev_notify = event.sigev_notify;
- new_timer->sigq->info.si_signo = event.sigev_signo;
- new_timer->sigq->info.si_value = event.sigev_value;
new_timer->sigq->info.si_tid = new_timer->it_id;
new_timer->sigq->info.si_code = SI_TIMER;
return error;
}
+SYSCALL_DEFINE3(timer_create, const clockid_t, which_clock,
+ struct sigevent __user *, timer_event_spec,
+ timer_t __user *, created_timer_id)
+{
+ if (timer_event_spec) {
+ sigevent_t event;
+
+ if (copy_from_user(&event, timer_event_spec, sizeof (event)))
+ return -EFAULT;
+ return do_timer_create(which_clock, &event, created_timer_id);
+ }
+ return do_timer_create(which_clock, NULL, created_timer_id);
+}
+
+#ifdef CONFIG_COMPAT
+COMPAT_SYSCALL_DEFINE3(timer_create, clockid_t, which_clock,
+ struct compat_sigevent __user *, timer_event_spec,
+ timer_t __user *, created_timer_id)
+{
+ if (timer_event_spec) {
+ sigevent_t event;
+
+ if (get_compat_sigevent(&event, timer_event_spec))
+ return -EFAULT;
+ return do_timer_create(which_clock, &event, created_timer_id);
+ }
+ return do_timer_create(which_clock, NULL, created_timer_id);
+}
+#endif
+
/*
* Locking issues: We need to protect the result of the id look up until
* we get the timer locked down so it is not deleted under us. The
return NULL;
}
+static ktime_t common_hrtimer_remaining(struct k_itimer *timr, ktime_t now)
+{
+ struct hrtimer *timer = &timr->it.real.timer;
+
+ return __hrtimer_expires_remaining_adjusted(timer, now);
+}
+
+static int common_hrtimer_forward(struct k_itimer *timr, ktime_t now)
+{
+ struct hrtimer *timer = &timr->it.real.timer;
+
+ return (int)hrtimer_forward(timer, now, timr->it_interval);
+}
+
/*
* Get the time remaining on a POSIX.1b interval timer. This function
* is ALWAYS called with spin_lock_irq on the timer, thus it must not
* it is the same as a requeue pending timer WRT to what we should
* report.
*/
-static void
-common_timer_get(struct k_itimer *timr, struct itimerspec64 *cur_setting)
+void common_timer_get(struct k_itimer *timr, struct itimerspec64 *cur_setting)
{
+ const struct k_clock *kc = timr->kclock;
ktime_t now, remaining, iv;
- struct hrtimer *timer = &timr->it.real.timer;
+ struct timespec64 ts64;
+ bool sig_none;
- memset(cur_setting, 0, sizeof(*cur_setting));
-
- iv = timr->it.real.interval;
+ sig_none = (timr->it_sigev_notify & ~SIGEV_THREAD_ID) == SIGEV_NONE;
+ iv = timr->it_interval;
/* interval timer ? */
- if (iv)
+ if (iv) {
cur_setting->it_interval = ktime_to_timespec64(iv);
- else if (!hrtimer_active(timer) &&
- (timr->it_sigev_notify & ~SIGEV_THREAD_ID) != SIGEV_NONE)
- return;
+ } else if (!timr->it_active) {
+ /*
+ * SIGEV_NONE oneshot timers are never queued. Check them
+ * below.
+ */
+ if (!sig_none)
+ return;
+ }
- now = timer->base->get_time();
+ /*
+ * The timespec64 based conversion is suboptimal, but it's not
+ * worth to implement yet another callback.
+ */
+ kc->clock_get(timr->it_clock, &ts64);
+ now = timespec64_to_ktime(ts64);
/*
- * When a requeue is pending or this is a SIGEV_NONE
- * timer move the expiry time forward by intervals, so
- * expiry is > now.
+ * When a requeue is pending or this is a SIGEV_NONE timer move the
+ * expiry time forward by intervals, so expiry is > now.
*/
- if (iv && (timr->it_requeue_pending & REQUEUE_PENDING ||
- (timr->it_sigev_notify & ~SIGEV_THREAD_ID) == SIGEV_NONE))
- timr->it_overrun += (unsigned int) hrtimer_forward(timer, now, iv);
+ if (iv && (timr->it_requeue_pending & REQUEUE_PENDING || sig_none))
+ timr->it_overrun += kc->timer_forward(timr, now);
- remaining = __hrtimer_expires_remaining_adjusted(timer, now);
+ remaining = kc->timer_remaining(timr, now);
/* Return 0 only, when the timer is expired and not pending */
if (remaining <= 0) {
/*
* A single shot SIGEV_NONE timer must return 0, when
* it is expired !
*/
- if ((timr->it_sigev_notify & ~SIGEV_THREAD_ID) != SIGEV_NONE)
+ if (!sig_none)
cur_setting->it_value.tv_nsec = 1;
- } else
+ } else {
cur_setting->it_value = ktime_to_timespec64(remaining);
+ }
}
/* Get the time remaining on a POSIX.1b interval timer. */
-SYSCALL_DEFINE2(timer_gettime, timer_t, timer_id,
- struct itimerspec __user *, setting)
+static int do_timer_gettime(timer_t timer_id, struct itimerspec64 *setting)
{
- struct itimerspec64 cur_setting64;
- struct itimerspec cur_setting;
struct k_itimer *timr;
- struct k_clock *kc;
+ const struct k_clock *kc;
unsigned long flags;
int ret = 0;
if (!timr)
return -EINVAL;
- kc = clockid_to_kclock(timr->it_clock);
+ memset(setting, 0, sizeof(*setting));
+ kc = timr->kclock;
if (WARN_ON_ONCE(!kc || !kc->timer_get))
ret = -EINVAL;
else
- kc->timer_get(timr, &cur_setting64);
+ kc->timer_get(timr, setting);
unlock_timer(timr, flags);
+ return ret;
+}
- cur_setting = itimerspec64_to_itimerspec(&cur_setting64);
- if (!ret && copy_to_user(setting, &cur_setting, sizeof (cur_setting)))
- return -EFAULT;
+/* Get the time remaining on a POSIX.1b interval timer. */
+SYSCALL_DEFINE2(timer_gettime, timer_t, timer_id,
+ struct itimerspec __user *, setting)
+{
+ struct itimerspec64 cur_setting64;
+ int ret = do_timer_gettime(timer_id, &cur_setting64);
+ if (!ret) {
+ struct itimerspec cur_setting;
+ cur_setting = itimerspec64_to_itimerspec(&cur_setting64);
+ if (copy_to_user(setting, &cur_setting, sizeof (cur_setting)))
+ ret = -EFAULT;
+ }
+ return ret;
+}
+
+#ifdef CONFIG_COMPAT
+COMPAT_SYSCALL_DEFINE2(timer_gettime, timer_t, timer_id,
+ struct compat_itimerspec __user *, setting)
+{
+ struct itimerspec64 cur_setting64;
+
+ int ret = do_timer_gettime(timer_id, &cur_setting64);
+ if (!ret) {
+ struct itimerspec cur_setting;
+ cur_setting = itimerspec64_to_itimerspec(&cur_setting64);
+ if (put_compat_itimerspec(setting, &cur_setting))
+ ret = -EFAULT;
+ }
return ret;
}
+#endif
/*
* Get the number of overruns of a POSIX.1b interval timer. This is to
* accumulating overruns on the next timer. The overrun is frozen when
* the signal is delivered, either at the notify time (if the info block
* is not queued) or at the actual delivery time (as we are informed by
- * the call back to do_schedule_next_timer(). So all we need to do is
+ * the call back to posixtimer_rearm(). So all we need to do is
* to pick up the frozen overrun.
*/
SYSCALL_DEFINE1(timer_getoverrun, timer_t, timer_id)
return overrun;
}
-/* Set a POSIX.1b interval timer. */
-/* timr->it_lock is taken. */
-static int
-common_timer_set(struct k_itimer *timr, int flags,
- struct itimerspec64 *new_setting, struct itimerspec64 *old_setting)
+static void common_hrtimer_arm(struct k_itimer *timr, ktime_t expires,
+ bool absolute, bool sigev_none)
{
struct hrtimer *timer = &timr->it.real.timer;
enum hrtimer_mode mode;
+ mode = absolute ? HRTIMER_MODE_ABS : HRTIMER_MODE_REL;
+ /*
+ * Posix magic: Relative CLOCK_REALTIME timers are not affected by
+ * clock modifications, so they become CLOCK_MONOTONIC based under the
+ * hood. See hrtimer_init(). Update timr->kclock, so the generic
+ * functions which use timr->kclock->clock_get() work.
+ *
+ * Note: it_clock stays unmodified, because the next timer_set() might
+ * use ABSTIME, so it needs to switch back.
+ */
+ if (timr->it_clock == CLOCK_REALTIME)
+ timr->kclock = absolute ? &clock_realtime : &clock_monotonic;
+
+ hrtimer_init(&timr->it.real.timer, timr->it_clock, mode);
+ timr->it.real.timer.function = posix_timer_fn;
+
+ if (!absolute)
+ expires = ktime_add_safe(expires, timer->base->get_time());
+ hrtimer_set_expires(timer, expires);
+
+ if (!sigev_none)
+ hrtimer_start_expires(timer, HRTIMER_MODE_ABS);
+}
+
+static int common_hrtimer_try_to_cancel(struct k_itimer *timr)
+{
+ return hrtimer_try_to_cancel(&timr->it.real.timer);
+}
+
+/* Set a POSIX.1b interval timer. */
+int common_timer_set(struct k_itimer *timr, int flags,
+ struct itimerspec64 *new_setting,
+ struct itimerspec64 *old_setting)
+{
+ const struct k_clock *kc = timr->kclock;
+ bool sigev_none;
+ ktime_t expires;
+
if (old_setting)
common_timer_get(timr, old_setting);
- /* disable the timer */
- timr->it.real.interval = 0;
+ /* Prevent rearming by clearing the interval */
+ timr->it_interval = 0;
/*
- * careful here. If smp we could be in the "fire" routine which will
- * be spinning as we hold the lock. But this is ONLY an SMP issue.
+ * Careful here. On SMP systems the timer expiry function could be
+ * active and spinning on timr->it_lock.
*/
- if (hrtimer_try_to_cancel(timer) < 0)
+ if (kc->timer_try_to_cancel(timr) < 0)
return TIMER_RETRY;
- timr->it_requeue_pending = (timr->it_requeue_pending + 2) &
+ timr->it_active = 0;
+ timr->it_requeue_pending = (timr->it_requeue_pending + 2) &
~REQUEUE_PENDING;
timr->it_overrun_last = 0;
- /* switch off the timer when it_value is zero */
+ /* Switch off the timer when it_value is zero */
if (!new_setting->it_value.tv_sec && !new_setting->it_value.tv_nsec)
return 0;
- mode = flags & TIMER_ABSTIME ? HRTIMER_MODE_ABS : HRTIMER_MODE_REL;
- hrtimer_init(&timr->it.real.timer, timr->it_clock, mode);
- timr->it.real.timer.function = posix_timer_fn;
-
- hrtimer_set_expires(timer, timespec64_to_ktime(new_setting->it_value));
-
- /* Convert interval */
- timr->it.real.interval = timespec64_to_ktime(new_setting->it_interval);
-
- /* SIGEV_NONE timers are not queued ! See common_timer_get */
- if (((timr->it_sigev_notify & ~SIGEV_THREAD_ID) == SIGEV_NONE)) {
- /* Setup correct expiry time for relative timers */
- if (mode == HRTIMER_MODE_REL) {
- hrtimer_add_expires(timer, timer->base->get_time());
- }
- return 0;
- }
+ timr->it_interval = timespec64_to_ktime(new_setting->it_interval);
+ expires = timespec64_to_ktime(new_setting->it_value);
+ sigev_none = (timr->it_sigev_notify & ~SIGEV_THREAD_ID) == SIGEV_NONE;
- hrtimer_start_expires(timer, mode);
+ kc->timer_arm(timr, expires, flags & TIMER_ABSTIME, sigev_none);
+ timr->it_active = !sigev_none;
return 0;
}
-/* Set a POSIX.1b interval timer */
-SYSCALL_DEFINE4(timer_settime, timer_t, timer_id, int, flags,
- const struct itimerspec __user *, new_setting,
- struct itimerspec __user *, old_setting)
+static int do_timer_settime(timer_t timer_id, int flags,
+ struct itimerspec64 *new_spec64,
+ struct itimerspec64 *old_spec64)
{
- struct itimerspec64 new_spec64, old_spec64;
- struct itimerspec64 *rtn = old_setting ? &old_spec64 : NULL;
- struct itimerspec new_spec, old_spec;
+ const struct k_clock *kc;
struct k_itimer *timr;
unsigned long flag;
- struct k_clock *kc;
int error = 0;
- if (!new_setting)
+ if (!timespec64_valid(&new_spec64->it_interval) ||
+ !timespec64_valid(&new_spec64->it_value))
return -EINVAL;
- if (copy_from_user(&new_spec, new_setting, sizeof (new_spec)))
- return -EFAULT;
- new_spec64 = itimerspec_to_itimerspec64(&new_spec);
-
- if (!timespec64_valid(&new_spec64.it_interval) ||
- !timespec64_valid(&new_spec64.it_value))
- return -EINVAL;
+ if (old_spec64)
+ memset(old_spec64, 0, sizeof(*old_spec64));
retry:
timr = lock_timer(timer_id, &flag);
if (!timr)
return -EINVAL;
- kc = clockid_to_kclock(timr->it_clock);
+ kc = timr->kclock;
if (WARN_ON_ONCE(!kc || !kc->timer_set))
error = -EINVAL;
else
- error = kc->timer_set(timr, flags, &new_spec64, rtn);
+ error = kc->timer_set(timr, flags, new_spec64, old_spec64);
unlock_timer(timr, flag);
if (error == TIMER_RETRY) {
- rtn = NULL; // We already got the old time...
+ old_spec64 = NULL; // We already got the old time...
goto retry;
}
- old_spec = itimerspec64_to_itimerspec(&old_spec64);
- if (old_setting && !error &&
- copy_to_user(old_setting, &old_spec, sizeof (old_spec)))
- error = -EFAULT;
+ return error;
+}
+
+/* Set a POSIX.1b interval timer */
+SYSCALL_DEFINE4(timer_settime, timer_t, timer_id, int, flags,
+ const struct itimerspec __user *, new_setting,
+ struct itimerspec __user *, old_setting)
+{
+ struct itimerspec64 new_spec64, old_spec64;
+ struct itimerspec64 *rtn = old_setting ? &old_spec64 : NULL;
+ struct itimerspec new_spec;
+ int error = 0;
+
+ if (!new_setting)
+ return -EINVAL;
+
+ if (copy_from_user(&new_spec, new_setting, sizeof (new_spec)))
+ return -EFAULT;
+ new_spec64 = itimerspec_to_itimerspec64(&new_spec);
+
+ error = do_timer_settime(timer_id, flags, &new_spec64, rtn);
+ if (!error && old_setting) {
+ struct itimerspec old_spec;
+ old_spec = itimerspec64_to_itimerspec(&old_spec64);
+ if (copy_to_user(old_setting, &old_spec, sizeof (old_spec)))
+ error = -EFAULT;
+ }
+ return error;
+}
+
+#ifdef CONFIG_COMPAT
+COMPAT_SYSCALL_DEFINE4(timer_settime, timer_t, timer_id, int, flags,
+ struct compat_itimerspec __user *, new,
+ struct compat_itimerspec __user *, old)
+{
+ struct itimerspec64 new_spec64, old_spec64;
+ struct itimerspec64 *rtn = old ? &old_spec64 : NULL;
+ struct itimerspec new_spec;
+ int error = 0;
+
+ if (!new)
+ return -EINVAL;
+ if (get_compat_itimerspec(&new_spec, new))
+ return -EFAULT;
+ new_spec64 = itimerspec_to_itimerspec64(&new_spec);
+ error = do_timer_settime(timer_id, flags, &new_spec64, rtn);
+ if (!error && old) {
+ struct itimerspec old_spec;
+ old_spec = itimerspec64_to_itimerspec(&old_spec64);
+ if (put_compat_itimerspec(old, &old_spec))
+ error = -EFAULT;
+ }
return error;
}
+#endif
-static int common_timer_del(struct k_itimer *timer)
+int common_timer_del(struct k_itimer *timer)
{
- timer->it.real.interval = 0;
+ const struct k_clock *kc = timer->kclock;
- if (hrtimer_try_to_cancel(&timer->it.real.timer) < 0)
+ timer->it_interval = 0;
+ if (kc->timer_try_to_cancel(timer) < 0)
return TIMER_RETRY;
+ timer->it_active = 0;
return 0;
}
static inline int timer_delete_hook(struct k_itimer *timer)
{
- struct k_clock *kc = clockid_to_kclock(timer->it_clock);
+ const struct k_clock *kc = timer->kclock;
if (WARN_ON_ONCE(!kc || !kc->timer_del))
return -EINVAL;
SYSCALL_DEFINE2(clock_settime, const clockid_t, which_clock,
const struct timespec __user *, tp)
{
- struct k_clock *kc = clockid_to_kclock(which_clock);
+ const struct k_clock *kc = clockid_to_kclock(which_clock);
struct timespec64 new_tp64;
struct timespec new_tp;
SYSCALL_DEFINE2(clock_gettime, const clockid_t, which_clock,
struct timespec __user *,tp)
{
- struct k_clock *kc = clockid_to_kclock(which_clock);
+ const struct k_clock *kc = clockid_to_kclock(which_clock);
struct timespec64 kernel_tp64;
struct timespec kernel_tp;
int error;
SYSCALL_DEFINE2(clock_adjtime, const clockid_t, which_clock,
struct timex __user *, utx)
{
- struct k_clock *kc = clockid_to_kclock(which_clock);
+ const struct k_clock *kc = clockid_to_kclock(which_clock);
struct timex ktx;
int err;
SYSCALL_DEFINE2(clock_getres, const clockid_t, which_clock,
struct timespec __user *, tp)
{
- struct k_clock *kc = clockid_to_kclock(which_clock);
+ const struct k_clock *kc = clockid_to_kclock(which_clock);
struct timespec64 rtn_tp64;
struct timespec rtn_tp;
int error;
return error;
}
+#ifdef CONFIG_COMPAT
+
+COMPAT_SYSCALL_DEFINE2(clock_settime, clockid_t, which_clock,
+ struct compat_timespec __user *, tp)
+{
+ const struct k_clock *kc = clockid_to_kclock(which_clock);
+ struct timespec64 new_tp64;
+ struct timespec new_tp;
+
+ if (!kc || !kc->clock_set)
+ return -EINVAL;
+
+ if (compat_get_timespec(&new_tp, tp))
+ return -EFAULT;
+
+ new_tp64 = timespec_to_timespec64(new_tp);
+
+ return kc->clock_set(which_clock, &new_tp64);
+}
+
+COMPAT_SYSCALL_DEFINE2(clock_gettime, clockid_t, which_clock,
+ struct compat_timespec __user *, tp)
+{
+ const struct k_clock *kc = clockid_to_kclock(which_clock);
+ struct timespec64 kernel_tp64;
+ struct timespec kernel_tp;
+ int error;
+
+ if (!kc)
+ return -EINVAL;
+
+ error = kc->clock_get(which_clock, &kernel_tp64);
+ kernel_tp = timespec64_to_timespec(kernel_tp64);
+
+ if (!error && compat_put_timespec(&kernel_tp, tp))
+ error = -EFAULT;
+
+ return error;
+}
+
+COMPAT_SYSCALL_DEFINE2(clock_adjtime, clockid_t, which_clock,
+ struct compat_timex __user *, utp)
+{
+ const struct k_clock *kc = clockid_to_kclock(which_clock);
+ struct timex ktx;
+ int err;
+
+ if (!kc)
+ return -EINVAL;
+ if (!kc->clock_adj)
+ return -EOPNOTSUPP;
+
+ err = compat_get_timex(&ktx, utp);
+ if (err)
+ return err;
+
+ err = kc->clock_adj(which_clock, &ktx);
+
+ if (err >= 0)
+ err = compat_put_timex(utp, &ktx);
+
+ return err;
+}
+
+COMPAT_SYSCALL_DEFINE2(clock_getres, clockid_t, which_clock,
+ struct compat_timespec __user *, tp)
+{
+ const struct k_clock *kc = clockid_to_kclock(which_clock);
+ struct timespec64 rtn_tp64;
+ struct timespec rtn_tp;
+ int error;
+
+ if (!kc)
+ return -EINVAL;
+
+ error = kc->clock_getres(which_clock, &rtn_tp64);
+ rtn_tp = timespec64_to_timespec(rtn_tp64);
+
+ if (!error && tp && compat_put_timespec(&rtn_tp, tp))
+ error = -EFAULT;
+
+ return error;
+}
+#endif
+
/*
* nanosleep for monotonic and realtime clocks
*/
static int common_nsleep(const clockid_t which_clock, int flags,
- struct timespec64 *tsave, struct timespec __user *rmtp)
+ const struct timespec64 *rqtp)
{
- return hrtimer_nanosleep(tsave, rmtp, flags & TIMER_ABSTIME ?
+ return hrtimer_nanosleep(rqtp, flags & TIMER_ABSTIME ?
HRTIMER_MODE_ABS : HRTIMER_MODE_REL,
which_clock);
}
const struct timespec __user *, rqtp,
struct timespec __user *, rmtp)
{
- struct k_clock *kc = clockid_to_kclock(which_clock);
+ const struct k_clock *kc = clockid_to_kclock(which_clock);
struct timespec64 t64;
struct timespec t;
t64 = timespec_to_timespec64(t);
if (!timespec64_valid(&t64))
return -EINVAL;
+ if (flags & TIMER_ABSTIME)
+ rmtp = NULL;
+ current->restart_block.nanosleep.type = rmtp ? TT_NATIVE : TT_NONE;
+ current->restart_block.nanosleep.rmtp = rmtp;
- return kc->nsleep(which_clock, flags, &t64, rmtp);
+ return kc->nsleep(which_clock, flags, &t64);
}
-/*
- * This will restart clock_nanosleep. This is required only by
- * compat_clock_nanosleep_restart for now.
- */
-long clock_nanosleep_restart(struct restart_block *restart_block)
+#ifdef CONFIG_COMPAT
+COMPAT_SYSCALL_DEFINE4(clock_nanosleep, clockid_t, which_clock, int, flags,
+ struct compat_timespec __user *, rqtp,
+ struct compat_timespec __user *, rmtp)
{
- clockid_t which_clock = restart_block->nanosleep.clockid;
- struct k_clock *kc = clockid_to_kclock(which_clock);
+ const struct k_clock *kc = clockid_to_kclock(which_clock);
+ struct timespec64 t64;
+ struct timespec t;
- if (WARN_ON_ONCE(!kc || !kc->nsleep_restart))
+ if (!kc)
return -EINVAL;
+ if (!kc->nsleep)
+ return -ENANOSLEEP_NOTSUP;
- return kc->nsleep_restart(restart_block);
+ if (compat_get_timespec(&t, rqtp))
+ return -EFAULT;
+
+ t64 = timespec_to_timespec64(t);
+ if (!timespec64_valid(&t64))
+ return -EINVAL;
+ if (flags & TIMER_ABSTIME)
+ rmtp = NULL;
+ current->restart_block.nanosleep.type = rmtp ? TT_COMPAT : TT_NONE;
+ current->restart_block.nanosleep.compat_rmtp = rmtp;
+
+ return kc->nsleep(which_clock, flags, &t64);
+}
+#endif
+
+static const struct k_clock clock_realtime = {
+ .clock_getres = posix_get_hrtimer_res,
+ .clock_get = posix_clock_realtime_get,
+ .clock_set = posix_clock_realtime_set,
+ .clock_adj = posix_clock_realtime_adj,
+ .nsleep = common_nsleep,
+ .timer_create = common_timer_create,
+ .timer_set = common_timer_set,
+ .timer_get = common_timer_get,
+ .timer_del = common_timer_del,
+ .timer_rearm = common_hrtimer_rearm,
+ .timer_forward = common_hrtimer_forward,
+ .timer_remaining = common_hrtimer_remaining,
+ .timer_try_to_cancel = common_hrtimer_try_to_cancel,
+ .timer_arm = common_hrtimer_arm,
+};
+
+static const struct k_clock clock_monotonic = {
+ .clock_getres = posix_get_hrtimer_res,
+ .clock_get = posix_ktime_get_ts,
+ .nsleep = common_nsleep,
+ .timer_create = common_timer_create,
+ .timer_set = common_timer_set,
+ .timer_get = common_timer_get,
+ .timer_del = common_timer_del,
+ .timer_rearm = common_hrtimer_rearm,
+ .timer_forward = common_hrtimer_forward,
+ .timer_remaining = common_hrtimer_remaining,
+ .timer_try_to_cancel = common_hrtimer_try_to_cancel,
+ .timer_arm = common_hrtimer_arm,
+};
+
+static const struct k_clock clock_monotonic_raw = {
+ .clock_getres = posix_get_hrtimer_res,
+ .clock_get = posix_get_monotonic_raw,
+};
+
+static const struct k_clock clock_realtime_coarse = {
+ .clock_getres = posix_get_coarse_res,
+ .clock_get = posix_get_realtime_coarse,
+};
+
+static const struct k_clock clock_monotonic_coarse = {
+ .clock_getres = posix_get_coarse_res,
+ .clock_get = posix_get_monotonic_coarse,
+};
+
+static const struct k_clock clock_tai = {
+ .clock_getres = posix_get_hrtimer_res,
+ .clock_get = posix_get_tai,
+ .nsleep = common_nsleep,
+ .timer_create = common_timer_create,
+ .timer_set = common_timer_set,
+ .timer_get = common_timer_get,
+ .timer_del = common_timer_del,
+ .timer_rearm = common_hrtimer_rearm,
+ .timer_forward = common_hrtimer_forward,
+ .timer_remaining = common_hrtimer_remaining,
+ .timer_try_to_cancel = common_hrtimer_try_to_cancel,
+ .timer_arm = common_hrtimer_arm,
+};
+
+static const struct k_clock clock_boottime = {
+ .clock_getres = posix_get_hrtimer_res,
+ .clock_get = posix_get_boottime,
+ .nsleep = common_nsleep,
+ .timer_create = common_timer_create,
+ .timer_set = common_timer_set,
+ .timer_get = common_timer_get,
+ .timer_del = common_timer_del,
+ .timer_rearm = common_hrtimer_rearm,
+ .timer_forward = common_hrtimer_forward,
+ .timer_remaining = common_hrtimer_remaining,
+ .timer_try_to_cancel = common_hrtimer_try_to_cancel,
+ .timer_arm = common_hrtimer_arm,
+};
+
+static const struct k_clock * const posix_clocks[] = {
+ [CLOCK_REALTIME] = &clock_realtime,
+ [CLOCK_MONOTONIC] = &clock_monotonic,
+ [CLOCK_PROCESS_CPUTIME_ID] = &clock_process,
+ [CLOCK_THREAD_CPUTIME_ID] = &clock_thread,
+ [CLOCK_MONOTONIC_RAW] = &clock_monotonic_raw,
+ [CLOCK_REALTIME_COARSE] = &clock_realtime_coarse,
+ [CLOCK_MONOTONIC_COARSE] = &clock_monotonic_coarse,
+ [CLOCK_BOOTTIME] = &clock_boottime,
+ [CLOCK_REALTIME_ALARM] = &alarm_clock,
+ [CLOCK_BOOTTIME_ALARM] = &alarm_clock,
+ [CLOCK_TAI] = &clock_tai,
+};
+
+static const struct k_clock *clockid_to_kclock(const clockid_t id)
+{
+ if (id < 0)
+ return (id & CLOCKFD_MASK) == CLOCKFD ?
+ &clock_posix_dynamic : &clock_posix_cpu;
+
+ if (id >= ARRAY_SIZE(posix_clocks) || !posix_clocks[id])
+ return NULL;
+ return posix_clocks[id];
}
--- /dev/null
+#define TIMER_RETRY 1
+
+struct k_clock {
+ int (*clock_getres)(const clockid_t which_clock,
+ struct timespec64 *tp);
+ int (*clock_set)(const clockid_t which_clock,
+ const struct timespec64 *tp);
+ int (*clock_get)(const clockid_t which_clock,
+ struct timespec64 *tp);
+ int (*clock_adj)(const clockid_t which_clock, struct timex *tx);
+ int (*timer_create)(struct k_itimer *timer);
+ int (*nsleep)(const clockid_t which_clock, int flags,
+ const struct timespec64 *);
+ int (*timer_set)(struct k_itimer *timr, int flags,
+ struct itimerspec64 *new_setting,
+ struct itimerspec64 *old_setting);
+ int (*timer_del)(struct k_itimer *timr);
+ void (*timer_get)(struct k_itimer *timr,
+ struct itimerspec64 *cur_setting);
+ void (*timer_rearm)(struct k_itimer *timr);
+ int (*timer_forward)(struct k_itimer *timr, ktime_t now);
+ ktime_t (*timer_remaining)(struct k_itimer *timr, ktime_t now);
+ int (*timer_try_to_cancel)(struct k_itimer *timr);
+ void (*timer_arm)(struct k_itimer *timr, ktime_t expires,
+ bool absolute, bool sigev_none);
+};
+
+extern const struct k_clock clock_posix_cpu;
+extern const struct k_clock clock_posix_dynamic;
+extern const struct k_clock clock_process;
+extern const struct k_clock clock_thread;
+extern const struct k_clock alarm_clock;
+
+int posix_timer_event(struct k_itimer *timr, int si_private);
+
+void common_timer_get(struct k_itimer *timr, struct itimerspec64 *cur_setting);
+int common_timer_set(struct k_itimer *timr, int flags,
+ struct itimerspec64 *new_setting,
+ struct itimerspec64 *old_setting);
+int common_timer_del(struct k_itimer *timer);
touch_softlockup_watchdog_sched();
if (is_idle_task(current))
ts->idle_jiffies++;
+ /*
+ * In case the current tick fired too early past its expected
+ * expiration, make sure we don't bypass the next clock reprogramming
+ * to the same deadline.
+ */
+ ts->next_tick = 0;
}
#endif
update_process_times(user_mode(regs));
update_ts_time_stats(smp_processor_id(), ts, now, NULL);
ts->idle_active = 0;
- sched_clock_idle_wakeup_event(0);
+ sched_clock_idle_wakeup_event();
}
static ktime_t tick_nohz_start_idle(struct tick_sched *ts)
hrtimer_start_expires(&ts->sched_timer, HRTIMER_MODE_ABS_PINNED);
else
tick_program_event(hrtimer_get_expires(&ts->sched_timer), 1);
+
+ /*
+ * Reset to make sure next tick stop doesn't get fooled by past
+ * cached clock deadline.
+ */
+ ts->next_tick = 0;
}
static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts,
*/
delta = next_tick - basemono;
if (delta <= (u64)TICK_NSEC) {
- tick = 0;
-
/*
* Tell the timer code that the base is not idle, i.e. undo
* the effect of get_next_timer_interrupt():
* We've not stopped the tick yet, and there's a timer in the
* next period, so no point in stopping it either, bail.
*/
- if (!ts->tick_stopped)
- goto out;
-
- /*
- * If, OTOH, we did stop it, but there's a pending (expired)
- * timer reprogram the timer hardware to fire now.
- *
- * We will not restart the tick proper, just prod the timer
- * hardware into firing an interrupt to process the pending
- * timers. Just like tick_irq_exit() will not restart the tick
- * for 'normal' interrupts.
- *
- * Only once we exit the idle loop will we re-enable the tick,
- * see tick_nohz_idle_exit().
- */
- if (delta == 0) {
- tick_nohz_restart(ts, now);
+ if (!ts->tick_stopped) {
+ tick = 0;
goto out;
}
}
tick = expires;
/* Skip reprogram of event if its not changed */
- if (ts->tick_stopped && (expires == dev->next_event))
- goto out;
+ if (ts->tick_stopped && (expires == ts->next_tick)) {
+ /* Sanity check: make sure clockevent is actually programmed */
+ if (tick == KTIME_MAX || ts->next_tick == hrtimer_get_expires(&ts->sched_timer))
+ goto out;
+
+ WARN_ON_ONCE(1);
+ printk_once("basemono: %llu ts->next_tick: %llu dev->next_event: %llu timer->active: %d timer->expires: %llu\n",
+ basemono, ts->next_tick, dev->next_event,
+ hrtimer_active(&ts->sched_timer), hrtimer_get_expires(&ts->sched_timer));
+ }
/*
* nohz_stop_sched_tick can be called several times before
* the scheduler tick in nohz_restart_sched_tick.
*/
if (!ts->tick_stopped) {
- nohz_balance_enter_idle(cpu);
- calc_load_enter_idle();
+ calc_load_nohz_start();
cpu_load_update_nohz_start();
ts->last_tick = hrtimer_get_expires(&ts->sched_timer);
trace_tick_stop(1, TICK_DEP_MASK_NONE);
}
+ ts->next_tick = tick;
+
/*
* If the expiration time == KTIME_MAX, then we simply stop
* the tick timer.
goto out;
}
+ hrtimer_set_expires(&ts->sched_timer, tick);
+
if (ts->nohz_mode == NOHZ_MODE_HIGHRES)
- hrtimer_start(&ts->sched_timer, tick, HRTIMER_MODE_ABS_PINNED);
+ hrtimer_start_expires(&ts->sched_timer, HRTIMER_MODE_ABS_PINNED);
else
tick_program_event(tick, 1);
out:
- /* Update the estimated sleep length */
+ /*
+ * Update the estimated sleep length until the next timer
+ * (not only the tick).
+ */
ts->sleep_length = ktime_sub(dev->next_event, now);
return tick;
}
*/
timer_clear_idle();
- calc_load_exit_idle();
+ calc_load_nohz_stop();
touch_softlockup_watchdog_sched();
/*
* Cancel the scheduled timer and restore the tick
if (unlikely(!cpu_online(cpu))) {
if (cpu == tick_do_timer_cpu)
tick_do_timer_cpu = TICK_DO_TIMER_NONE;
+ /*
+ * Make sure the CPU doesn't get fooled by obsolete tick
+ * deadline if it comes back online later.
+ */
+ ts->next_tick = 0;
return false;
}
ts->idle_expires = expires;
}
- if (!was_stopped && ts->tick_stopped)
+ if (!was_stopped && ts->tick_stopped) {
ts->idle_jiffies = ts->last_jiffies;
+ nohz_balance_enter_idle(cpu);
+ }
}
}
*/
if (regs)
tick_sched_handle(ts, regs);
+ else
+ ts->next_tick = 0;
/* No need to reprogram if we are in idle or full dynticks mode */
if (unlikely(ts->tick_stopped))
* timer is modified for nohz sleeps. This is necessary
* to resume the tick timer operation in the timeline
* when the CPU returns from nohz sleep.
+ * @next_tick: Next tick to be fired when in dynticks mode.
* @tick_stopped: Indicator that the idle tick has been stopped
* @idle_jiffies: jiffies at the entry to idle for idle time accounting
* @idle_calls: Total number of idle calls
unsigned long check_clocks;
enum tick_nohz_mode nohz_mode;
ktime_t last_tick;
+ ktime_t next_tick;
int inidle;
int tick_stopped;
unsigned long idle_jiffies;
#include <linux/ptrace.h>
#include <linux/uaccess.h>
+#include <linux/compat.h>
#include <asm/unistd.h>
#include <generated/timeconst.h>
#endif /* __ARCH_WANT_SYS_TIME */
+#ifdef CONFIG_COMPAT
+#ifdef __ARCH_WANT_COMPAT_SYS_TIME
+
+/* compat_time_t is a 32 bit "long" and needs to get converted. */
+COMPAT_SYSCALL_DEFINE1(time, compat_time_t __user *, tloc)
+{
+ struct timeval tv;
+ compat_time_t i;
+
+ do_gettimeofday(&tv);
+ i = tv.tv_sec;
+
+ if (tloc) {
+ if (put_user(i,tloc))
+ return -EFAULT;
+ }
+ force_successful_syscall_return();
+ return i;
+}
+
+COMPAT_SYSCALL_DEFINE1(stime, compat_time_t __user *, tptr)
+{
+ struct timespec tv;
+ int err;
+
+ if (get_user(tv.tv_sec, tptr))
+ return -EFAULT;
+
+ tv.tv_nsec = 0;
+
+ err = security_settime(&tv, NULL);
+ if (err)
+ return err;
+
+ do_settimeofday(&tv);
+ return 0;
+}
+
+#endif /* __ARCH_WANT_COMPAT_SYS_TIME */
+#endif
+
SYSCALL_DEFINE2(gettimeofday, struct timeval __user *, tv,
struct timezone __user *, tz)
{
return do_sys_settimeofday64(tv ? &new_ts : NULL, tz ? &new_tz : NULL);
}
+#ifdef CONFIG_COMPAT
+COMPAT_SYSCALL_DEFINE2(gettimeofday, struct compat_timeval __user *, tv,
+ struct timezone __user *, tz)
+{
+ if (tv) {
+ struct timeval ktv;
+
+ do_gettimeofday(&ktv);
+ if (compat_put_timeval(&ktv, tv))
+ return -EFAULT;
+ }
+ if (tz) {
+ if (copy_to_user(tz, &sys_tz, sizeof(sys_tz)))
+ return -EFAULT;
+ }
+
+ return 0;
+}
+
+COMPAT_SYSCALL_DEFINE2(settimeofday, struct compat_timeval __user *, tv,
+ struct timezone __user *, tz)
+{
+ struct timespec64 new_ts;
+ struct timeval user_tv;
+ struct timezone new_tz;
+
+ if (tv) {
+ if (compat_get_timeval(&user_tv, tv))
+ return -EFAULT;
+ new_ts.tv_sec = user_tv.tv_sec;
+ new_ts.tv_nsec = user_tv.tv_usec * NSEC_PER_USEC;
+ }
+ if (tz) {
+ if (copy_from_user(&new_tz, tz, sizeof(*tz)))
+ return -EFAULT;
+ }
+
+ return do_sys_settimeofday64(tv ? &new_ts : NULL, tz ? &new_tz : NULL);
+}
+#endif
+
SYSCALL_DEFINE1(adjtimex, struct timex __user *, txc_p)
{
struct timex txc; /* Local copy of parameter */
* structure. But bear in mind that the structures
* may change
*/
- if(copy_from_user(&txc, txc_p, sizeof(struct timex)))
+ if (copy_from_user(&txc, txc_p, sizeof(struct timex)))
return -EFAULT;
ret = do_adjtimex(&txc);
return copy_to_user(txc_p, &txc, sizeof(struct timex)) ? -EFAULT : ret;
}
+#ifdef CONFIG_COMPAT
+
+COMPAT_SYSCALL_DEFINE1(adjtimex, struct compat_timex __user *, utp)
+{
+ struct timex txc;
+ int err, ret;
+
+ err = compat_get_timex(&txc, utp);
+ if (err)
+ return err;
+
+ ret = do_adjtimex(&txc);
+
+ err = compat_put_timex(utp, &txc);
+ if (err)
+ return err;
+
+ return ret;
+}
+#endif
+
/*
* Convert jiffies to milliseconds and back.
*
tk->tkr_mono.xtime_nsec -= (u64)NSEC_PER_SEC << tk->tkr_mono.shift;
tk->xtime_sec++;
}
+ while (tk->tkr_raw.xtime_nsec >= ((u64)NSEC_PER_SEC << tk->tkr_raw.shift)) {
+ tk->tkr_raw.xtime_nsec -= (u64)NSEC_PER_SEC << tk->tkr_raw.shift;
+ tk->raw_sec++;
+ }
}
static inline struct timespec64 tk_xtime(struct timekeeper *tk)
tk->offs_boot = ktime_add(tk->offs_boot, delta);
}
+/*
+ * tk_clock_read - atomic clocksource read() helper
+ *
+ * This helper is necessary to use in the read paths because, while the
+ * seqlock ensures we don't return a bad value while structures are updated,
+ * it doesn't protect from potential crashes. There is the possibility that
+ * the tkr's clocksource may change between the read reference, and the
+ * clock reference passed to the read function. This can cause crashes if
+ * the wrong clocksource is passed to the wrong read function.
+ * This isn't necessary to use when holding the timekeeper_lock or doing
+ * a read of the fast-timekeeper tkrs (which is protected by its own locking
+ * and update logic).
+ */
+static inline u64 tk_clock_read(struct tk_read_base *tkr)
+{
+ struct clocksource *clock = READ_ONCE(tkr->clock);
+
+ return clock->read(clock);
+}
+
#ifdef CONFIG_DEBUG_TIMEKEEPING
#define WARNING_FREQ (HZ*300) /* 5 minute rate-limiting */
*/
do {
seq = read_seqcount_begin(&tk_core.seq);
- now = tkr->read(tkr->clock);
+ now = tk_clock_read(tkr);
last = tkr->cycle_last;
mask = tkr->mask;
max = tkr->clock->max_cycles;
u64 cycle_now, delta;
/* read clocksource */
- cycle_now = tkr->read(tkr->clock);
+ cycle_now = tk_clock_read(tkr);
/* calculate the delta since the last update_wall_time */
delta = clocksource_delta(cycle_now, tkr->cycle_last, tkr->mask);
++tk->cs_was_changed_seq;
old_clock = tk->tkr_mono.clock;
tk->tkr_mono.clock = clock;
- tk->tkr_mono.read = clock->read;
tk->tkr_mono.mask = clock->mask;
- tk->tkr_mono.cycle_last = tk->tkr_mono.read(clock);
+ tk->tkr_mono.cycle_last = tk_clock_read(&tk->tkr_mono);
tk->tkr_raw.clock = clock;
- tk->tkr_raw.read = clock->read;
tk->tkr_raw.mask = clock->mask;
tk->tkr_raw.cycle_last = tk->tkr_mono.cycle_last;
/* Go back from cycles -> shifted ns */
tk->xtime_interval = interval * clock->mult;
tk->xtime_remainder = ntpinterval - tk->xtime_interval;
- tk->raw_interval = (interval * clock->mult) >> clock->shift;
+ tk->raw_interval = interval * clock->mult;
/* if changing clocks, convert xtime_nsec shift units */
if (old_clock) {
int shift_change = clock->shift - old_clock->shift;
- if (shift_change < 0)
+ if (shift_change < 0) {
tk->tkr_mono.xtime_nsec >>= -shift_change;
- else
+ tk->tkr_raw.xtime_nsec >>= -shift_change;
+ } else {
tk->tkr_mono.xtime_nsec <<= shift_change;
+ tk->tkr_raw.xtime_nsec <<= shift_change;
+ }
}
- tk->tkr_raw.xtime_nsec = 0;
tk->tkr_mono.shift = clock->shift;
tk->tkr_raw.shift = clock->shift;
now += timekeeping_delta_to_ns(tkr,
clocksource_delta(
- tkr->read(tkr->clock),
+ tk_clock_read(tkr),
tkr->cycle_last,
tkr->mask));
} while (read_seqcount_retry(&tkf->seq, seq));
return cycles_at_suspend;
}
+static struct clocksource dummy_clock = {
+ .read = dummy_clock_read,
+};
+
/**
* halt_fast_timekeeper - Prevent fast timekeeper from accessing clocksource.
* @tk: Timekeeper to snapshot.
struct tk_read_base *tkr = &tk->tkr_mono;
memcpy(&tkr_dummy, tkr, sizeof(tkr_dummy));
- cycles_at_suspend = tkr->read(tkr->clock);
- tkr_dummy.read = dummy_clock_read;
+ cycles_at_suspend = tk_clock_read(tkr);
+ tkr_dummy.clock = &dummy_clock;
update_fast_timekeeper(&tkr_dummy, &tk_fast_mono);
tkr = &tk->tkr_raw;
memcpy(&tkr_dummy, tkr, sizeof(tkr_dummy));
- tkr_dummy.read = dummy_clock_read;
+ tkr_dummy.clock = &dummy_clock;
update_fast_timekeeper(&tkr_dummy, &tk_fast_raw);
}
#ifdef CONFIG_GENERIC_TIME_VSYSCALL_OLD
+#warning Please contact your maintainers, as GENERIC_TIME_VSYSCALL_OLD compatibity will disappear soon.
static inline void update_vsyscall(struct timekeeper *tk)
{
nsec = (u32) tk->wall_to_monotonic.tv_nsec;
tk->tkr_mono.base = ns_to_ktime(seconds * NSEC_PER_SEC + nsec);
- /* Update the monotonic raw base */
- tk->tkr_raw.base = timespec64_to_ktime(tk->raw_time);
-
/*
* The sum of the nanoseconds portions of xtime and
* wall_to_monotonic can be greater/equal one second. Take
if (nsec >= NSEC_PER_SEC)
seconds++;
tk->ktime_sec = seconds;
+
+ /* Update the monotonic raw base */
+ seconds = tk->raw_sec;
+ nsec = (u32)(tk->tkr_raw.xtime_nsec >> tk->tkr_raw.shift);
+ tk->tkr_raw.base = ns_to_ktime(seconds * NSEC_PER_SEC + nsec);
}
/* must hold timekeeper_lock */
*/
static void timekeeping_forward_now(struct timekeeper *tk)
{
- struct clocksource *clock = tk->tkr_mono.clock;
u64 cycle_now, delta;
- u64 nsec;
- cycle_now = tk->tkr_mono.read(clock);
+ cycle_now = tk_clock_read(&tk->tkr_mono);
delta = clocksource_delta(cycle_now, tk->tkr_mono.cycle_last, tk->tkr_mono.mask);
tk->tkr_mono.cycle_last = cycle_now;
tk->tkr_raw.cycle_last = cycle_now;
/* If arch requires, add in get_arch_timeoffset() */
tk->tkr_mono.xtime_nsec += (u64)arch_gettimeoffset() << tk->tkr_mono.shift;
- tk_normalize_xtime(tk);
- nsec = clocksource_cyc2ns(delta, tk->tkr_raw.mult, tk->tkr_raw.shift);
- timespec64_add_ns(&tk->raw_time, nsec);
+ tk->tkr_raw.xtime_nsec += delta * tk->tkr_raw.mult;
+
+ /* If arch requires, add in get_arch_timeoffset() */
+ tk->tkr_raw.xtime_nsec += (u64)arch_gettimeoffset() << tk->tkr_raw.shift;
+
+ tk_normalize_xtime(tk);
}
/**
do {
seq = read_seqcount_begin(&tk_core.seq);
-
- now = tk->tkr_mono.read(tk->tkr_mono.clock);
+ now = tk_clock_read(&tk->tkr_mono);
systime_snapshot->cs_was_changed_seq = tk->cs_was_changed_seq;
systime_snapshot->clock_was_set_seq = tk->clock_was_set_seq;
base_real = ktime_add(tk->tkr_mono.base,
* Check whether the system counter value provided by the
* device driver is on the current timekeeping interval.
*/
- now = tk->tkr_mono.read(tk->tkr_mono.clock);
+ now = tk_clock_read(&tk->tkr_mono);
interval_start = tk->tkr_mono.cycle_last;
if (!cycle_between(interval_start, cycles, now)) {
clock_was_set_seq = tk->clock_was_set_seq;
void getrawmonotonic64(struct timespec64 *ts)
{
struct timekeeper *tk = &tk_core.timekeeper;
- struct timespec64 ts64;
unsigned long seq;
u64 nsecs;
do {
seq = read_seqcount_begin(&tk_core.seq);
+ ts->tv_sec = tk->raw_sec;
nsecs = timekeeping_get_ns(&tk->tkr_raw);
- ts64 = tk->raw_time;
} while (read_seqcount_retry(&tk_core.seq, seq));
- timespec64_add_ns(&ts64, nsecs);
- *ts = ts64;
+ ts->tv_nsec = 0;
+ timespec64_add_ns(ts, nsecs);
}
EXPORT_SYMBOL(getrawmonotonic64);
tk_setup_internals(tk, clock);
tk_set_xtime(tk, &now);
- tk->raw_time.tv_sec = 0;
- tk->raw_time.tv_nsec = 0;
+ tk->raw_sec = 0;
if (boot.tv_sec == 0 && boot.tv_nsec == 0)
boot = tk_xtime(tk);
* The less preferred source will only be tried if there is no better
* usable source. The rtc part is handled separately in rtc core code.
*/
- cycle_now = tk->tkr_mono.read(clock);
+ cycle_now = tk_clock_read(&tk->tkr_mono);
if ((clock->flags & CLOCK_SOURCE_SUSPEND_NONSTOP) &&
cycle_now > tk->tkr_mono.cycle_last) {
u64 nsec, cyc_delta;
u32 shift, unsigned int *clock_set)
{
u64 interval = tk->cycle_interval << shift;
- u64 raw_nsecs;
+ u64 snsec_per_sec;
/* If the offset is smaller than a shifted interval, do nothing */
if (offset < interval)
*clock_set |= accumulate_nsecs_to_secs(tk);
/* Accumulate raw time */
- raw_nsecs = (u64)tk->raw_interval << shift;
- raw_nsecs += tk->raw_time.tv_nsec;
- if (raw_nsecs >= NSEC_PER_SEC) {
- u64 raw_secs = raw_nsecs;
- raw_nsecs = do_div(raw_secs, NSEC_PER_SEC);
- tk->raw_time.tv_sec += raw_secs;
+ tk->tkr_raw.xtime_nsec += tk->raw_interval << shift;
+ snsec_per_sec = (u64)NSEC_PER_SEC << tk->tkr_raw.shift;
+ while (tk->tkr_raw.xtime_nsec >= snsec_per_sec) {
+ tk->tkr_raw.xtime_nsec -= snsec_per_sec;
+ tk->raw_sec++;
}
- tk->raw_time.tv_nsec = raw_nsecs;
/* Accumulate error between NTP and clock interval */
tk->ntp_error += tk->ntp_tick << shift;
#ifdef CONFIG_ARCH_USES_GETTIMEOFFSET
offset = real_tk->cycle_interval;
#else
- offset = clocksource_delta(tk->tkr_mono.read(tk->tkr_mono.clock),
+ offset = clocksource_delta(tk_clock_read(&tk->tkr_mono),
tk->tkr_mono.cycle_last, tk->tkr_mono.mask);
#endif
#endif
struct timer_base {
- spinlock_t lock;
+ raw_spinlock_t lock;
struct timer_list *running_timer;
unsigned long clk;
unsigned long next_expiry;
if (!(tf & TIMER_MIGRATING)) {
base = get_timer_base(tf);
- spin_lock_irqsave(&base->lock, *flags);
+ raw_spin_lock_irqsave(&base->lock, *flags);
if (timer->flags == tf)
return base;
- spin_unlock_irqrestore(&base->lock, *flags);
+ raw_spin_unlock_irqrestore(&base->lock, *flags);
}
cpu_relax();
}
/* See the comment in lock_timer_base() */
timer->flags |= TIMER_MIGRATING;
- spin_unlock(&base->lock);
+ raw_spin_unlock(&base->lock);
base = new_base;
- spin_lock(&base->lock);
+ raw_spin_lock(&base->lock);
WRITE_ONCE(timer->flags,
(timer->flags & ~TIMER_BASEMASK) | base->cpu);
}
}
out_unlock:
- spin_unlock_irqrestore(&base->lock, flags);
+ raw_spin_unlock_irqrestore(&base->lock, flags);
return ret;
}
if (base != new_base) {
timer->flags |= TIMER_MIGRATING;
- spin_unlock(&base->lock);
+ raw_spin_unlock(&base->lock);
base = new_base;
- spin_lock(&base->lock);
+ raw_spin_lock(&base->lock);
WRITE_ONCE(timer->flags,
(timer->flags & ~TIMER_BASEMASK) | cpu);
}
debug_activate(timer, timer->expires);
internal_add_timer(base, timer);
- spin_unlock_irqrestore(&base->lock, flags);
+ raw_spin_unlock_irqrestore(&base->lock, flags);
}
EXPORT_SYMBOL_GPL(add_timer_on);
if (timer_pending(timer)) {
base = lock_timer_base(timer, &flags);
ret = detach_if_pending(timer, base, true);
- spin_unlock_irqrestore(&base->lock, flags);
+ raw_spin_unlock_irqrestore(&base->lock, flags);
}
return ret;
/**
* try_to_del_timer_sync - Try to deactivate a timer
- * @timer: timer do del
+ * @timer: timer to delete
*
* This function tries to deactivate a timer. Upon successful (ret >= 0)
* exit the timer is not queued and the handler is not running on any CPU.
if (base->running_timer != timer)
ret = detach_if_pending(timer, base, true);
- spin_unlock_irqrestore(&base->lock, flags);
+ raw_spin_unlock_irqrestore(&base->lock, flags);
return ret;
}
data = timer->data;
if (timer->flags & TIMER_IRQSAFE) {
- spin_unlock(&base->lock);
+ raw_spin_unlock(&base->lock);
call_timer_fn(timer, fn, data);
- spin_lock(&base->lock);
+ raw_spin_lock(&base->lock);
} else {
- spin_unlock_irq(&base->lock);
+ raw_spin_unlock_irq(&base->lock);
call_timer_fn(timer, fn, data);
- spin_lock_irq(&base->lock);
+ raw_spin_lock_irq(&base->lock);
}
}
}
if (cpu_is_offline(smp_processor_id()))
return expires;
- spin_lock(&base->lock);
+ raw_spin_lock(&base->lock);
nextevt = __next_timer_interrupt(base);
is_max_delta = (nextevt == base->clk + NEXT_TIMER_MAX_DELTA);
base->next_expiry = nextevt;
if ((expires - basem) > TICK_NSEC)
base->is_idle = true;
}
- spin_unlock(&base->lock);
+ raw_spin_unlock(&base->lock);
return cmp_next_hrtimer_event(basem, expires);
}
if (!time_after_eq(jiffies, base->clk))
return;
- spin_lock_irq(&base->lock);
+ raw_spin_lock_irq(&base->lock);
while (time_after_eq(jiffies, base->clk)) {
expire_timers(base, heads + levels);
}
base->running_timer = NULL;
- spin_unlock_irq(&base->lock);
+ raw_spin_unlock_irq(&base->lock);
}
/*
* The caller is globally serialized and nobody else
* takes two locks at once, deadlock is not possible.
*/
- spin_lock_irq(&new_base->lock);
- spin_lock_nested(&old_base->lock, SINGLE_DEPTH_NESTING);
+ raw_spin_lock_irq(&new_base->lock);
+ raw_spin_lock_nested(&old_base->lock, SINGLE_DEPTH_NESTING);
BUG_ON(old_base->running_timer);
for (i = 0; i < WHEEL_SIZE; i++)
migrate_timer_list(new_base, old_base->vectors + i);
- spin_unlock(&old_base->lock);
- spin_unlock_irq(&new_base->lock);
+ raw_spin_unlock(&old_base->lock);
+ raw_spin_unlock_irq(&new_base->lock);
put_cpu_ptr(&timer_bases);
}
return 0;
for (i = 0; i < NR_BASES; i++) {
base = per_cpu_ptr(&timer_bases[i], cpu);
base->cpu = cpu;
- spin_lock_init(&base->lock);
+ raw_spin_lock_init(&base->lock);
base->clk = jiffies;
}
}
__blk_add_trace(bt, bio->bi_iter.bi_sector,
bio->bi_iter.bi_size, bio_op(bio), bio->bi_opf,
- BLK_TA_SPLIT, bio->bi_error, sizeof(rpdu),
+ BLK_TA_SPLIT, bio->bi_status, sizeof(rpdu),
&rpdu);
}
}
r.sector_from = cpu_to_be64(from);
__blk_add_trace(bt, bio->bi_iter.bi_sector, bio->bi_iter.bi_size,
- bio_op(bio), bio->bi_opf, BLK_TA_REMAP, bio->bi_error,
+ bio_op(bio), bio->bi_opf, BLK_TA_REMAP, bio->bi_status,
sizeof(r), &r);
}
command = strsep(&next, ":");
- if (WARN_ON_ONCE(!tr))
- return -EINVAL;
-
mutex_lock(&ftrace_cmd_mutex);
list_for_each_entry(p, &ftrace_commands, list) {
if (strcmp(p->name, command) == 0) {
char *number;
int ret;
+ if (!tr)
+ return -ENODEV;
+
/* hash funcs only work with set_ftrace_filter */
if (!enable)
return -EINVAL;
{
struct ftrace_probe_ops *ops;
+ if (!tr)
+ return -ENODEV;
+
/* we register both traceon and traceoff to this callback */
if (strcmp(cmd, "traceon") == 0)
ops = param ? &traceon_count_probe_ops : &traceon_probe_ops;
{
struct ftrace_probe_ops *ops;
+ if (!tr)
+ return -ENODEV;
+
ops = param ? &stacktrace_count_probe_ops : &stacktrace_probe_ops;
return ftrace_trace_probe_callback(tr, ops, hash, glob, cmd,
{
struct ftrace_probe_ops *ops;
+ if (!tr)
+ return -ENODEV;
+
ops = &dump_probe_ops;
/* Only dump once. */
{
struct ftrace_probe_ops *ops;
+ if (!tr)
+ return -ENODEV;
+
ops = &cpudump_probe_ops;
/* Only dump once. */
pr_info("Probe point is not specified.\n");
return -EINVAL;
}
- if (isdigit(argv[1][0])) {
- /* an address specified */
- ret = kstrtoul(&argv[1][0], 0, (unsigned long *)&addr);
- if (ret) {
- pr_info("Failed to parse address.\n");
- return ret;
- }
- } else {
+
+ /* try to parse an address. if that fails, try to read the
+ * input as a symbol. */
+ if (kstrtoul(argv[1], 0, (unsigned long *)&addr)) {
/* a symbol specified */
symbol = argv[1];
/* TODO: support .init module functions */
ret = traceprobe_split_symbol_offset(symbol, &offset);
if (ret) {
- pr_info("Failed to parse symbol.\n");
+ pr_info("Failed to parse either an address or a symbol.\n");
return ret;
}
if (offset && is_return &&
static int
stack_trace_filter_open(struct inode *inode, struct file *file)
{
- return ftrace_regex_open(&trace_ops, FTRACE_ITER_FILTER,
+ struct ftrace_ops *ops = inode->i_private;
+
+ return ftrace_regex_open(ops, FTRACE_ITER_FILTER,
inode, file);
}
NULL, &stack_trace_fops);
trace_create_file("stack_trace_filter", 0444, d_tracer,
- NULL, &stack_trace_filter_fops);
+ &trace_ops, &stack_trace_filter_fops);
if (stack_trace_filter_buf[0])
ftrace_set_early_filter(&trace_ops, stack_trace_filter_buf, 1);
EXPORT_SYMBOL_GPL(flush_work);
struct cwt_wait {
- wait_queue_t wait;
+ wait_queue_entry_t wait;
struct work_struct *work;
};
-static int cwt_wakefn(wait_queue_t *wait, unsigned mode, int sync, void *key)
+static int cwt_wakefn(wait_queue_entry_t *wait, unsigned mode, int sync, void *key)
{
struct cwt_wait *cwait = container_of(wait, struct cwt_wait, wait);
endchoice
+config CRC4
+ tristate "CRC4 functions"
+ help
+ This option is provided for the case where no in-kernel-tree
+ modules require CRC4 functions, but a module built outside
+ the kernel tree does. Such modules that use library CRC4
+ functions require M here.
+
config CRC7
tristate "CRC7 functions"
help
write to these files.
For detailed documentation on the debugfs API, see
- Documentation/DocBook/filesystems.
+ Documentation/filesystems/.
If unsure, say N.
depends on DEBUG_KERNEL && TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT
select DEBUG_SPINLOCK
select DEBUG_MUTEXES
+ select DEBUG_RT_MUTEXES if RT_MUTEXES
select LOCKDEP
help
This feature will check whether any held lock (spinlock, rwlock,
select LOCKDEP
select DEBUG_SPINLOCK
select DEBUG_MUTEXES
+ select DEBUG_RT_MUTEXES if RT_MUTEXES
select DEBUG_LOCK_ALLOC
select TRACE_IRQFLAGS
default n
select LOCKDEP
select DEBUG_SPINLOCK
select DEBUG_MUTEXES
+ select DEBUG_RT_MUTEXES if RT_MUTEXES
select DEBUG_LOCK_ALLOC
default n
help
If unsure, say N.
-menu "RCU Debugging"
-
-config PROVE_RCU
- def_bool PROVE_LOCKING
-
-config PROVE_RCU_REPEATEDLY
- bool "RCU debugging: don't disable PROVE_RCU on first splat"
- depends on PROVE_RCU
- default n
- help
- By itself, PROVE_RCU will disable checking upon issuing the
- first warning (or "splat"). This feature prevents such
- disabling, allowing multiple RCU-lockdep warnings to be printed
- on a single reboot.
-
- Say Y to allow multiple RCU-lockdep warnings per boot.
-
- Say N if you are unsure.
-
-config SPARSE_RCU_POINTER
- bool "RCU debugging: sparse-based checks for pointer usage"
- default n
- help
- This feature enables the __rcu sparse annotation for
- RCU-protected pointers. This annotation will cause sparse
- to flag any non-RCU used of annotated pointers. This can be
- helpful when debugging RCU usage. Please note that this feature
- is not intended to enforce code cleanliness; it is instead merely
- a debugging aid.
-
- Say Y to make sparse flag questionable use of RCU-protected pointers
-
- Say N if you are unsure.
-
-config TORTURE_TEST
- tristate
- default n
-
-config RCU_PERF_TEST
- tristate "performance tests for RCU"
- depends on DEBUG_KERNEL
- select TORTURE_TEST
- select SRCU
- select TASKS_RCU
- default n
- help
- This option provides a kernel module that runs performance
- tests on the RCU infrastructure. The kernel module may be built
- after the fact on the running kernel to be tested, if desired.
-
- Say Y here if you want RCU performance tests to be built into
- the kernel.
- Say M if you want the RCU performance tests to build as a module.
- Say N if you are unsure.
-
-config RCU_TORTURE_TEST
- tristate "torture tests for RCU"
- depends on DEBUG_KERNEL
- select TORTURE_TEST
- select SRCU
- select TASKS_RCU
- default n
- help
- This option provides a kernel module that runs torture tests
- on the RCU infrastructure. The kernel module may be built
- after the fact on the running kernel to be tested, if desired.
-
- Say Y here if you want RCU torture tests to be built into
- the kernel.
- Say M if you want the RCU torture tests to build as a module.
- Say N if you are unsure.
-
-config RCU_TORTURE_TEST_SLOW_PREINIT
- bool "Slow down RCU grace-period pre-initialization to expose races"
- depends on RCU_TORTURE_TEST
- help
- This option delays grace-period pre-initialization (the
- propagation of CPU-hotplug changes up the rcu_node combining
- tree) for a few jiffies between initializing each pair of
- consecutive rcu_node structures. This helps to expose races
- involving grace-period pre-initialization, in other words, it
- makes your kernel less stable. It can also greatly increase
- grace-period latency, especially on systems with large numbers
- of CPUs. This is useful when torture-testing RCU, but in
- almost no other circumstance.
-
- Say Y here if you want your system to crash and hang more often.
- Say N if you want a sane system.
-
-config RCU_TORTURE_TEST_SLOW_PREINIT_DELAY
- int "How much to slow down RCU grace-period pre-initialization"
- range 0 5
- default 3
- depends on RCU_TORTURE_TEST_SLOW_PREINIT
- help
- This option specifies the number of jiffies to wait between
- each rcu_node structure pre-initialization step.
-
-config RCU_TORTURE_TEST_SLOW_INIT
- bool "Slow down RCU grace-period initialization to expose races"
- depends on RCU_TORTURE_TEST
- help
- This option delays grace-period initialization for a few
- jiffies between initializing each pair of consecutive
- rcu_node structures. This helps to expose races involving
- grace-period initialization, in other words, it makes your
- kernel less stable. It can also greatly increase grace-period
- latency, especially on systems with large numbers of CPUs.
- This is useful when torture-testing RCU, but in almost no
- other circumstance.
-
- Say Y here if you want your system to crash and hang more often.
- Say N if you want a sane system.
-
-config RCU_TORTURE_TEST_SLOW_INIT_DELAY
- int "How much to slow down RCU grace-period initialization"
- range 0 5
- default 3
- depends on RCU_TORTURE_TEST_SLOW_INIT
- help
- This option specifies the number of jiffies to wait between
- each rcu_node structure initialization.
-
-config RCU_TORTURE_TEST_SLOW_CLEANUP
- bool "Slow down RCU grace-period cleanup to expose races"
- depends on RCU_TORTURE_TEST
- help
- This option delays grace-period cleanup for a few jiffies
- between cleaning up each pair of consecutive rcu_node
- structures. This helps to expose races involving grace-period
- cleanup, in other words, it makes your kernel less stable.
- It can also greatly increase grace-period latency, especially
- on systems with large numbers of CPUs. This is useful when
- torture-testing RCU, but in almost no other circumstance.
-
- Say Y here if you want your system to crash and hang more often.
- Say N if you want a sane system.
-
-config RCU_TORTURE_TEST_SLOW_CLEANUP_DELAY
- int "How much to slow down RCU grace-period cleanup"
- range 0 5
- default 3
- depends on RCU_TORTURE_TEST_SLOW_CLEANUP
- help
- This option specifies the number of jiffies to wait between
- each rcu_node structure cleanup operation.
-
-config RCU_CPU_STALL_TIMEOUT
- int "RCU CPU stall timeout in seconds"
- depends on RCU_STALL_COMMON
- range 3 300
- default 21
- help
- If a given RCU grace period extends more than the specified
- number of seconds, a CPU stall warning is printed. If the
- RCU grace period persists, additional CPU stall warnings are
- printed at more widely spaced intervals.
-
-config RCU_TRACE
- bool "Enable tracing for RCU"
- depends on DEBUG_KERNEL
- default y if TREE_RCU
- select TRACE_CLOCK
- help
- This option provides tracing in RCU which presents stats
- in debugfs for debugging RCU implementation. It also enables
- additional tracepoints for ftrace-style event tracing.
-
- Say Y here if you want to enable RCU tracing
- Say N if you are unsure.
-
-config RCU_EQS_DEBUG
- bool "Provide debugging asserts for adding NO_HZ support to an arch"
- depends on DEBUG_KERNEL
- help
- This option provides consistency checks in RCU's handling of
- NO_HZ. These checks have proven quite helpful in detecting
- bugs in arch-specific NO_HZ code.
-
- Say N here if you need ultimate kernel/user switch latencies
- Say Y if you are unsure
-
-endmenu # "RCU Debugging"
+source "kernel/rcu/Kconfig.debug"
config DEBUG_WQ_FORCE_RR_CPU
bool "Force round-robin CPU selection for unbound work items"
CONFIG_FRAME_POINTER to aid in producing more reliable stack
backtraces in the external debugger. Documentation of
kernel debugger is available at http://kgdb.sourceforge.net
- as well as in DocBook form in Documentation/DocBook/. If
+ as well as in Documentation/dev-tools/kgdb.rst. If
unsure, say N.
if KGDB
earlycpio.o seq_buf.o siphash.o \
nmi_backtrace.o nodemask.o win_minmax.o
-CFLAGS_radix-tree.o += -DCONFIG_SPARSE_RCU_POINTER
-CFLAGS_idr.o += -DCONFIG_SPARSE_RCU_POINTER
-
lib-$(CONFIG_MMU) += ioremap.o
lib-$(CONFIG_SMP) += cpumask.o
lib-$(CONFIG_DMA_NOOP_OPS) += dma-noop.o
obj-$(CONFIG_CRC_ITU_T) += crc-itu-t.o
obj-$(CONFIG_CRC32) += crc32.o
obj-$(CONFIG_CRC32_SELFTEST) += crc32test.o
+obj-$(CONFIG_CRC4) += crc4.o
obj-$(CONFIG_CRC7) += crc7.o
obj-$(CONFIG_LIBCRC32C) += libcrc32c.o
obj-$(CONFIG_CRC8) += crc8.o
* the values[M, M+1, ..., N] into the ints array in get_options.
*/
-static int get_range(char **str, int *pint)
+static int get_range(char **str, int *pint, int n)
{
int x, inc_counter, upper_range;
(*str)++;
upper_range = simple_strtol((*str), NULL, 0);
inc_counter = upper_range - *pint;
- for (x = *pint; x < upper_range; x++)
+ for (x = *pint; n && x < upper_range; x++, n--)
*pint++ = x;
return inc_counter;
}
break;
if (res == 3) {
int range_nums;
- range_nums = get_range((char **)&str, ints + i);
+ range_nums = get_range((char **)&str, ints + i, nints - i);
if (range_nums < 0)
break;
/*
}
EXPORT_SYMBOL(cpumask_any_but);
+/**
+ * cpumask_next_wrap - helper to implement for_each_cpu_wrap
+ * @n: the cpu prior to the place to search
+ * @mask: the cpumask pointer
+ * @start: the start point of the iteration
+ * @wrap: assume @n crossing @start terminates the iteration
+ *
+ * Returns >= nr_cpu_ids on completion
+ *
+ * Note: the @wrap argument is required for the start condition when
+ * we cannot assume @start is set in @mask.
+ */
+int cpumask_next_wrap(int n, const struct cpumask *mask, int start, bool wrap)
+{
+ int next;
+
+again:
+ next = cpumask_next(n, mask);
+
+ if (wrap && n < start && next >= start) {
+ return nr_cpumask_bits;
+
+ } else if (next >= nr_cpumask_bits) {
+ wrap = true;
+ n = -1;
+ goto again;
+ }
+
+ return next;
+}
+EXPORT_SYMBOL(cpumask_next_wrap);
+
/* These are not inline because of header tangles. */
#ifdef CONFIG_CPUMASK_OFFSTACK
/**
--- /dev/null
+/*
+ * crc4.c - simple crc-4 calculations.
+ *
+ * This source code is licensed under the GNU General Public License, Version
+ * 2. See the file COPYING for more details.
+ */
+
+#include <linux/crc4.h>
+#include <linux/module.h>
+
+static const uint8_t crc4_tab[] = {
+ 0x0, 0x7, 0xe, 0x9, 0xb, 0xc, 0x5, 0x2,
+ 0x1, 0x6, 0xf, 0x8, 0xa, 0xd, 0x4, 0x3,
+};
+
+/**
+ * crc4 - calculate the 4-bit crc of a value.
+ * @crc: starting crc4
+ * @x: value to checksum
+ * @bits: number of bits in @x to checksum
+ *
+ * Returns the crc4 value of @x, using polynomial 0b10111.
+ *
+ * The @x value is treated as left-aligned, and bits above @bits are ignored
+ * in the crc calculations.
+ */
+uint8_t crc4(uint8_t c, uint64_t x, int bits)
+{
+ int i;
+
+ /* mask off anything above the top bit */
+ x &= (1ull << bits) - 1;
+
+ /* Align to 4-bits */
+ bits = (bits + 3) & ~0x3;
+
+ /* Calculate crc4 over four-bit nibbles, starting at the MSbit */
+ for (i = bits - 4; i >= 0; i -= 4)
+ c = crc4_tab[c ^ ((x >> i) & 0xf)];
+
+ return c;
+}
+EXPORT_SYMBOL_GPL(crc4);
+
+MODULE_DESCRIPTION("CRC4 calculations");
+MODULE_LICENSE("GPL");
#include <linux/socket.h>
#include <linux/skbuff.h>
#include <linux/netlink.h>
+#include <linux/uuid.h>
+#include <linux/ctype.h>
#include <net/sock.h>
#include <net/net_namespace.h>
[KOBJ_OFFLINE] = "offline",
};
-/**
- * kobject_action_type - translate action string to numeric type
- *
- * @buf: buffer containing the action string, newline is ignored
- * @count: length of buffer
- * @type: pointer to the location to store the action type
- *
- * Returns 0 if the action string was recognized.
- */
-int kobject_action_type(const char *buf, size_t count,
- enum kobject_action *type)
+static int kobject_action_type(const char *buf, size_t count,
+ enum kobject_action *type,
+ const char **args)
{
enum kobject_action action;
+ size_t count_first;
+ const char *args_start;
int ret = -EINVAL;
if (count && (buf[count-1] == '\n' || buf[count-1] == '\0'))
if (!count)
goto out;
+ args_start = strnchr(buf, count, ' ');
+ if (args_start) {
+ count_first = args_start - buf;
+ args_start = args_start + 1;
+ } else
+ count_first = count;
+
for (action = 0; action < ARRAY_SIZE(kobject_actions); action++) {
- if (strncmp(kobject_actions[action], buf, count) != 0)
+ if (strncmp(kobject_actions[action], buf, count_first) != 0)
continue;
- if (kobject_actions[action][count] != '\0')
+ if (kobject_actions[action][count_first] != '\0')
continue;
+ if (args)
+ *args = args_start;
*type = action;
ret = 0;
break;
return ret;
}
+static const char *action_arg_word_end(const char *buf, const char *buf_end,
+ char delim)
+{
+ const char *next = buf;
+
+ while (next <= buf_end && *next != delim)
+ if (!isalnum(*next++))
+ return NULL;
+
+ if (next == buf)
+ return NULL;
+
+ return next;
+}
+
+static int kobject_action_args(const char *buf, size_t count,
+ struct kobj_uevent_env **ret_env)
+{
+ struct kobj_uevent_env *env = NULL;
+ const char *next, *buf_end, *key;
+ int key_len;
+ int r = -EINVAL;
+
+ if (count && (buf[count - 1] == '\n' || buf[count - 1] == '\0'))
+ count--;
+
+ if (!count)
+ return -EINVAL;
+
+ env = kzalloc(sizeof(*env), GFP_KERNEL);
+ if (!env)
+ return -ENOMEM;
+
+ /* first arg is UUID */
+ if (count < UUID_STRING_LEN || !uuid_is_valid(buf) ||
+ add_uevent_var(env, "SYNTH_UUID=%.*s", UUID_STRING_LEN, buf))
+ goto out;
+
+ /*
+ * the rest are custom environment variables in KEY=VALUE
+ * format with ' ' delimiter between each KEY=VALUE pair
+ */
+ next = buf + UUID_STRING_LEN;
+ buf_end = buf + count - 1;
+
+ while (next <= buf_end) {
+ if (*next != ' ')
+ goto out;
+
+ /* skip the ' ', key must follow */
+ key = ++next;
+ if (key > buf_end)
+ goto out;
+
+ buf = next;
+ next = action_arg_word_end(buf, buf_end, '=');
+ if (!next || next > buf_end || *next != '=')
+ goto out;
+ key_len = next - buf;
+
+ /* skip the '=', value must follow */
+ if (++next > buf_end)
+ goto out;
+
+ buf = next;
+ next = action_arg_word_end(buf, buf_end, ' ');
+ if (!next)
+ goto out;
+
+ if (add_uevent_var(env, "SYNTH_ARG_%.*s=%.*s",
+ key_len, key, (int) (next - buf), buf))
+ goto out;
+ }
+
+ r = 0;
+out:
+ if (r)
+ kfree(env);
+ else
+ *ret_env = env;
+ return r;
+}
+
+/**
+ * kobject_synth_uevent - send synthetic uevent with arguments
+ *
+ * @kobj: struct kobject for which synthetic uevent is to be generated
+ * @buf: buffer containing action type and action args, newline is ignored
+ * @count: length of buffer
+ *
+ * Returns 0 if kobject_synthetic_uevent() is completed with success or the
+ * corresponding error when it fails.
+ */
+int kobject_synth_uevent(struct kobject *kobj, const char *buf, size_t count)
+{
+ char *no_uuid_envp[] = { "SYNTH_UUID=0", NULL };
+ enum kobject_action action;
+ const char *action_args;
+ struct kobj_uevent_env *env;
+ const char *msg = NULL, *devpath;
+ int r;
+
+ r = kobject_action_type(buf, count, &action, &action_args);
+ if (r) {
+ msg = "unknown uevent action string\n";
+ goto out;
+ }
+
+ if (!action_args) {
+ r = kobject_uevent_env(kobj, action, no_uuid_envp);
+ goto out;
+ }
+
+ r = kobject_action_args(action_args,
+ count - (action_args - buf), &env);
+ if (r == -EINVAL) {
+ msg = "incorrect uevent action arguments\n";
+ goto out;
+ }
+
+ if (r)
+ goto out;
+
+ r = kobject_uevent_env(kobj, action, env->envp);
+ kfree(env);
+out:
+ if (r) {
+ devpath = kobject_get_path(kobj, GFP_KERNEL);
+ printk(KERN_WARNING "synth uevent: %s: %s",
+ devpath ?: "unknown device",
+ msg ?: "failed to send uevent");
+ kfree(devpath);
+ }
+ return r;
+}
+
#ifdef CONFIG_NET
static int kobj_bcast_filter(struct sock *dsk, struct sk_buff *skb, void *data)
{
--- /dev/null
+#undef LOCK
+#define LOCK RTL
+
+#undef UNLOCK
+#define UNLOCK RTU
+
+#undef RLOCK
+#undef WLOCK
+
+#undef INIT
+#define INIT RTI
#include <linux/interrupt.h>
#include <linux/debug_locks.h>
#include <linux/irqflags.h>
+#include <linux/rtmutex.h>
/*
* Change this to 1 if you want to see the failure printouts:
#define LOCKTYPE_MUTEX 0x4
#define LOCKTYPE_RWSEM 0x8
#define LOCKTYPE_WW 0x10
+#define LOCKTYPE_RTMUTEX 0x20
static struct ww_acquire_ctx t, t2;
static struct ww_mutex o, o2, o3;
static DECLARE_RWSEM(rwsem_C);
static DECLARE_RWSEM(rwsem_D);
+#ifdef CONFIG_RT_MUTEXES
+
+static DEFINE_RT_MUTEX(rtmutex_A);
+static DEFINE_RT_MUTEX(rtmutex_B);
+static DEFINE_RT_MUTEX(rtmutex_C);
+static DEFINE_RT_MUTEX(rtmutex_D);
+
+#endif
+
/*
* Locks that we initialize dynamically as well so that
* e.g. X1 and X2 becomes two instances of the same class,
static DECLARE_RWSEM(rwsem_Z1);
static DECLARE_RWSEM(rwsem_Z2);
+#ifdef CONFIG_RT_MUTEXES
+
+static DEFINE_RT_MUTEX(rtmutex_X1);
+static DEFINE_RT_MUTEX(rtmutex_X2);
+static DEFINE_RT_MUTEX(rtmutex_Y1);
+static DEFINE_RT_MUTEX(rtmutex_Y2);
+static DEFINE_RT_MUTEX(rtmutex_Z1);
+static DEFINE_RT_MUTEX(rtmutex_Z2);
+
+#endif
+
/*
* non-inlined runtime initializers, to let separate locks share
* the same lock-class:
static void init_shared_classes(void)
{
+#ifdef CONFIG_RT_MUTEXES
+ static struct lock_class_key rt_X, rt_Y, rt_Z;
+
+ __rt_mutex_init(&rtmutex_X1, __func__, &rt_X);
+ __rt_mutex_init(&rtmutex_X2, __func__, &rt_X);
+ __rt_mutex_init(&rtmutex_Y1, __func__, &rt_Y);
+ __rt_mutex_init(&rtmutex_Y2, __func__, &rt_Y);
+ __rt_mutex_init(&rtmutex_Z1, __func__, &rt_Z);
+ __rt_mutex_init(&rtmutex_Z2, __func__, &rt_Z);
+#endif
+
init_class_X(&lock_X1, &rwlock_X1, &mutex_X1, &rwsem_X1);
init_class_X(&lock_X2, &rwlock_X2, &mutex_X2, &rwsem_X2);
#define MU(x) mutex_unlock(&mutex_##x)
#define MI(x) mutex_init(&mutex_##x)
+#define RTL(x) rt_mutex_lock(&rtmutex_##x)
+#define RTU(x) rt_mutex_unlock(&rtmutex_##x)
+#define RTI(x) rt_mutex_init(&rtmutex_##x)
+
#define WSL(x) down_write(&rwsem_##x)
#define WSU(x) up_write(&rwsem_##x)
#include "locking-selftest-rsem.h"
GENERATE_TESTCASE(AA_rsem)
+#ifdef CONFIG_RT_MUTEXES
+#include "locking-selftest-rtmutex.h"
+GENERATE_TESTCASE(AA_rtmutex);
+#endif
+
#undef E
/*
#include "locking-selftest-rsem.h"
GENERATE_TESTCASE(ABBA_rsem)
+#ifdef CONFIG_RT_MUTEXES
+#include "locking-selftest-rtmutex.h"
+GENERATE_TESTCASE(ABBA_rtmutex);
+#endif
+
#undef E
/*
#include "locking-selftest-rsem.h"
GENERATE_TESTCASE(ABBCCA_rsem)
+#ifdef CONFIG_RT_MUTEXES
+#include "locking-selftest-rtmutex.h"
+GENERATE_TESTCASE(ABBCCA_rtmutex);
+#endif
+
#undef E
/*
#include "locking-selftest-rsem.h"
GENERATE_TESTCASE(ABCABC_rsem)
+#ifdef CONFIG_RT_MUTEXES
+#include "locking-selftest-rtmutex.h"
+GENERATE_TESTCASE(ABCABC_rtmutex);
+#endif
+
#undef E
/*
#include "locking-selftest-rsem.h"
GENERATE_TESTCASE(ABBCCDDA_rsem)
+#ifdef CONFIG_RT_MUTEXES
+#include "locking-selftest-rtmutex.h"
+GENERATE_TESTCASE(ABBCCDDA_rtmutex);
+#endif
+
#undef E
/*
#include "locking-selftest-rsem.h"
GENERATE_TESTCASE(ABCDBDDA_rsem)
+#ifdef CONFIG_RT_MUTEXES
+#include "locking-selftest-rtmutex.h"
+GENERATE_TESTCASE(ABCDBDDA_rtmutex);
+#endif
+
#undef E
/*
#include "locking-selftest-rsem.h"
GENERATE_TESTCASE(ABCDBCDA_rsem)
+#ifdef CONFIG_RT_MUTEXES
+#include "locking-selftest-rtmutex.h"
+GENERATE_TESTCASE(ABCDBCDA_rtmutex);
+#endif
+
#undef E
/*
#include "locking-selftest-rsem.h"
GENERATE_TESTCASE(double_unlock_rsem)
-#undef E
-
-/*
- * Bad unlock ordering:
- */
-#define E() \
- \
- LOCK(A); \
- LOCK(B); \
- UNLOCK(A); /* fail */ \
- UNLOCK(B);
-
-/*
- * 6 testcases:
- */
-#include "locking-selftest-spin.h"
-GENERATE_TESTCASE(bad_unlock_order_spin)
-#include "locking-selftest-wlock.h"
-GENERATE_TESTCASE(bad_unlock_order_wlock)
-#include "locking-selftest-rlock.h"
-GENERATE_TESTCASE(bad_unlock_order_rlock)
-#include "locking-selftest-mutex.h"
-GENERATE_TESTCASE(bad_unlock_order_mutex)
-#include "locking-selftest-wsem.h"
-GENERATE_TESTCASE(bad_unlock_order_wsem)
-#include "locking-selftest-rsem.h"
-GENERATE_TESTCASE(bad_unlock_order_rsem)
+#ifdef CONFIG_RT_MUTEXES
+#include "locking-selftest-rtmutex.h"
+GENERATE_TESTCASE(double_unlock_rtmutex);
+#endif
#undef E
#include "locking-selftest-rsem.h"
GENERATE_TESTCASE(init_held_rsem)
+#ifdef CONFIG_RT_MUTEXES
+#include "locking-selftest-rtmutex.h"
+GENERATE_TESTCASE(init_held_rtmutex);
+#endif
+
#undef E
/*
# define I_MUTEX(x) lockdep_reset_lock(&mutex_##x.dep_map)
# define I_RWSEM(x) lockdep_reset_lock(&rwsem_##x.dep_map)
# define I_WW(x) lockdep_reset_lock(&x.dep_map)
+#ifdef CONFIG_RT_MUTEXES
+# define I_RTMUTEX(x) lockdep_reset_lock(&rtmutex_##x.dep_map)
+#endif
#else
# define I_SPINLOCK(x)
# define I_RWLOCK(x)
# define I_WW(x)
#endif
+#ifndef I_RTMUTEX
+# define I_RTMUTEX(x)
+#endif
+
+#ifdef CONFIG_RT_MUTEXES
+#define I2_RTMUTEX(x) rt_mutex_init(&rtmutex_##x)
+#else
+#define I2_RTMUTEX(x)
+#endif
+
#define I1(x) \
do { \
I_SPINLOCK(x); \
I_RWLOCK(x); \
I_MUTEX(x); \
I_RWSEM(x); \
+ I_RTMUTEX(x); \
} while (0)
#define I2(x) \
rwlock_init(&rwlock_##x); \
mutex_init(&mutex_##x); \
init_rwsem(&rwsem_##x); \
+ I2_RTMUTEX(x); \
} while (0)
static void reset_locks(void)
reset_locks();
}
+#ifdef CONFIG_RT_MUTEXES
+#define dotest_rt(fn, e, m) dotest((fn), (e), (m))
+#else
+#define dotest_rt(fn, e, m)
+#endif
+
static inline void print_testname(const char *testname)
{
printk("%33s:", testname);
dotest(name##_mutex, FAILURE, LOCKTYPE_MUTEX); \
dotest(name##_wsem, FAILURE, LOCKTYPE_RWSEM); \
dotest(name##_rsem, FAILURE, LOCKTYPE_RWSEM); \
+ dotest_rt(name##_rtmutex, FAILURE, LOCKTYPE_RTMUTEX); \
pr_cont("\n");
#define DO_TESTCASE_6_SUCCESS(desc, name) \
dotest(name##_mutex, SUCCESS, LOCKTYPE_MUTEX); \
dotest(name##_wsem, SUCCESS, LOCKTYPE_RWSEM); \
dotest(name##_rsem, SUCCESS, LOCKTYPE_RWSEM); \
+ dotest_rt(name##_rtmutex, SUCCESS, LOCKTYPE_RTMUTEX); \
pr_cont("\n");
/*
dotest(name##_mutex, FAILURE, LOCKTYPE_MUTEX); \
dotest(name##_wsem, FAILURE, LOCKTYPE_RWSEM); \
dotest(name##_rsem, FAILURE, LOCKTYPE_RWSEM); \
+ dotest_rt(name##_rtmutex, FAILURE, LOCKTYPE_RTMUTEX); \
pr_cont("\n");
#define DO_TESTCASE_2I(desc, name, nr) \
DO_TESTCASE_6R("A-B-C-D-B-C-D-A deadlock", ABCDBCDA);
DO_TESTCASE_6("double unlock", double_unlock);
DO_TESTCASE_6("initialize held", init_held);
- DO_TESTCASE_6_SUCCESS("bad unlock order", bad_unlock_order);
printk(" --------------------------------------------------------------------------\n");
print_testname("recursive read-lock");
#include <linux/refcount.h>
#include <linux/bug.h>
+#ifdef CONFIG_REFCOUNT_FULL
+
/**
* refcount_add_not_zero - add a value to a refcount unless it is 0
* @i: the value to add to the refcount
WARN_ONCE(refcount_dec_and_test(r), "refcount_t: decrement hit 0; leaking memory.\n");
}
EXPORT_SYMBOL(refcount_dec);
+#endif /* CONFIG_REFCOUNT_FULL */
/**
* refcount_dec_if_one - decrement a refcount if it is 1
return sg_copy_buffer(sgl, nents, buf, buflen, skip, true);
}
EXPORT_SYMBOL(sg_pcopy_to_buffer);
+
+/**
+ * sg_zero_buffer - Zero-out a part of a SG list
+ * @sgl: The SG list
+ * @nents: Number of SG entries
+ * @buflen: The number of bytes to zero out
+ * @skip: Number of bytes to skip before zeroing
+ *
+ * Returns the number of bytes zeroed.
+ **/
+size_t sg_zero_buffer(struct scatterlist *sgl, unsigned int nents,
+ size_t buflen, off_t skip)
+{
+ unsigned int offset = 0;
+ struct sg_mapping_iter miter;
+ unsigned int sg_flags = SG_MITER_ATOMIC | SG_MITER_TO_SG;
+
+ sg_miter_start(&miter, sgl, nents, sg_flags);
+
+ if (!sg_miter_skip(&miter, skip))
+ return false;
+
+ while (offset < buflen && sg_miter_next(&miter)) {
+ unsigned int len;
+
+ len = min(miter.length, buflen - offset);
+ memset(miter.addr, 0, len);
+
+ offset += len;
+ }
+
+ sg_miter_stop(&miter);
+ return offset;
+}
+EXPORT_SYMBOL(sg_zero_buffer);
/*
* It is valid to assume CPU-locality during early bootup:
*/
- if (system_state != SYSTEM_RUNNING)
+ if (system_state < SYSTEM_SCHEDULING)
goto out;
/*
struct test_uuid_data {
const char *uuid;
- uuid_le le;
- uuid_be be;
+ guid_t le;
+ uuid_t be;
};
static const struct test_uuid_data test_uuid_test_data[] = {
{
.uuid = "c33f4995-3701-450e-9fbf-206a2e98e576",
- .le = UUID_LE(0xc33f4995, 0x3701, 0x450e, 0x9f, 0xbf, 0x20, 0x6a, 0x2e, 0x98, 0xe5, 0x76),
- .be = UUID_BE(0xc33f4995, 0x3701, 0x450e, 0x9f, 0xbf, 0x20, 0x6a, 0x2e, 0x98, 0xe5, 0x76),
+ .le = GUID_INIT(0xc33f4995, 0x3701, 0x450e, 0x9f, 0xbf, 0x20, 0x6a, 0x2e, 0x98, 0xe5, 0x76),
+ .be = UUID_INIT(0xc33f4995, 0x3701, 0x450e, 0x9f, 0xbf, 0x20, 0x6a, 0x2e, 0x98, 0xe5, 0x76),
},
{
.uuid = "64b4371c-77c1-48f9-8221-29f054fc023b",
- .le = UUID_LE(0x64b4371c, 0x77c1, 0x48f9, 0x82, 0x21, 0x29, 0xf0, 0x54, 0xfc, 0x02, 0x3b),
- .be = UUID_BE(0x64b4371c, 0x77c1, 0x48f9, 0x82, 0x21, 0x29, 0xf0, 0x54, 0xfc, 0x02, 0x3b),
+ .le = GUID_INIT(0x64b4371c, 0x77c1, 0x48f9, 0x82, 0x21, 0x29, 0xf0, 0x54, 0xfc, 0x02, 0x3b),
+ .be = UUID_INIT(0x64b4371c, 0x77c1, 0x48f9, 0x82, 0x21, 0x29, 0xf0, 0x54, 0xfc, 0x02, 0x3b),
},
{
.uuid = "0cb4ddff-a545-4401-9d06-688af53e7f84",
- .le = UUID_LE(0x0cb4ddff, 0xa545, 0x4401, 0x9d, 0x06, 0x68, 0x8a, 0xf5, 0x3e, 0x7f, 0x84),
- .be = UUID_BE(0x0cb4ddff, 0xa545, 0x4401, 0x9d, 0x06, 0x68, 0x8a, 0xf5, 0x3e, 0x7f, 0x84),
+ .le = GUID_INIT(0x0cb4ddff, 0xa545, 0x4401, 0x9d, 0x06, 0x68, 0x8a, 0xf5, 0x3e, 0x7f, 0x84),
+ .be = UUID_INIT(0x0cb4ddff, 0xa545, 0x4401, 0x9d, 0x06, 0x68, 0x8a, 0xf5, 0x3e, 0x7f, 0x84),
},
};
static void __init test_uuid_test(const struct test_uuid_data *data)
{
- uuid_le le;
- uuid_be be;
+ guid_t le;
+ uuid_t be;
char buf[48];
/* LE */
total_tests++;
- if (uuid_le_to_bin(data->uuid, &le))
+ if (guid_parse(data->uuid, &le))
test_uuid_failed("conversion", false, false, data->uuid, NULL);
total_tests++;
- if (uuid_le_cmp(data->le, le)) {
+ if (!guid_equal(&data->le, &le)) {
sprintf(buf, "%pUl", &le);
test_uuid_failed("cmp", false, false, data->uuid, buf);
}
/* BE */
total_tests++;
- if (uuid_be_to_bin(data->uuid, &be))
+ if (uuid_parse(data->uuid, &be))
test_uuid_failed("conversion", false, true, data->uuid, NULL);
total_tests++;
- if (uuid_be_cmp(data->be, be)) {
+ if (uuid_equal(&data->be, &be)) {
sprintf(buf, "%pUb", &be);
test_uuid_failed("cmp", false, true, data->uuid, buf);
}
static void __init test_uuid_wrong(const char *data)
{
- uuid_le le;
- uuid_be be;
+ guid_t le;
+ uuid_t be;
/* LE */
total_tests++;
- if (!uuid_le_to_bin(data, &le))
+ if (!guid_parse(data, &le))
test_uuid_failed("negative", true, false, data, NULL);
/* BE */
total_tests++;
- if (!uuid_be_to_bin(data, &be))
+ if (!uuid_parse(data, &be))
test_uuid_failed("negative", true, true, data, NULL);
}
#include <linux/uuid.h>
#include <linux/random.h>
-const u8 uuid_le_index[16] = {3,2,1,0,5,4,7,6,8,9,10,11,12,13,14,15};
-EXPORT_SYMBOL(uuid_le_index);
-const u8 uuid_be_index[16] = {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15};
-EXPORT_SYMBOL(uuid_be_index);
+const guid_t guid_null;
+EXPORT_SYMBOL(guid_null);
+const uuid_t uuid_null;
+EXPORT_SYMBOL(uuid_null);
+
+const u8 guid_index[16] = {3,2,1,0,5,4,7,6,8,9,10,11,12,13,14,15};
+const u8 uuid_index[16] = {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15};
/***************************************************************
* Random UUID interface
b[8] = (b[8] & 0x3F) | 0x80;
}
-void uuid_le_gen(uuid_le *lu)
+void guid_gen(guid_t *lu)
{
__uuid_gen_common(lu->b);
/* version 4 : random generation */
lu->b[7] = (lu->b[7] & 0x0F) | 0x40;
}
-EXPORT_SYMBOL_GPL(uuid_le_gen);
+EXPORT_SYMBOL_GPL(guid_gen);
-void uuid_be_gen(uuid_be *bu)
+void uuid_gen(uuid_t *bu)
{
__uuid_gen_common(bu->b);
/* version 4 : random generation */
bu->b[6] = (bu->b[6] & 0x0F) | 0x40;
}
-EXPORT_SYMBOL_GPL(uuid_be_gen);
+EXPORT_SYMBOL_GPL(uuid_gen);
/**
* uuid_is_valid - checks if UUID string valid
}
EXPORT_SYMBOL(uuid_is_valid);
-static int __uuid_to_bin(const char *uuid, __u8 b[16], const u8 ei[16])
+static int __uuid_parse(const char *uuid, __u8 b[16], const u8 ei[16])
{
static const u8 si[16] = {0,2,4,6,9,11,14,16,19,21,24,26,28,30,32,34};
unsigned int i;
return 0;
}
-int uuid_le_to_bin(const char *uuid, uuid_le *u)
+int guid_parse(const char *uuid, guid_t *u)
{
- return __uuid_to_bin(uuid, u->b, uuid_le_index);
+ return __uuid_parse(uuid, u->b, guid_index);
}
-EXPORT_SYMBOL(uuid_le_to_bin);
+EXPORT_SYMBOL(guid_parse);
-int uuid_be_to_bin(const char *uuid, uuid_be *u)
+int uuid_parse(const char *uuid, uuid_t *u)
{
- return __uuid_to_bin(uuid, u->b, uuid_be_index);
+ return __uuid_parse(uuid, u->b, uuid_index);
}
-EXPORT_SYMBOL(uuid_be_to_bin);
+EXPORT_SYMBOL(uuid_parse);
char uuid[UUID_STRING_LEN + 1];
char *p = uuid;
int i;
- const u8 *index = uuid_be_index;
+ const u8 *index = uuid_index;
bool uc = false;
switch (*(++fmt)) {
case 'L':
uc = true; /* fall-through */
case 'l':
- index = uuid_le_index;
+ index = guid_index;
break;
case 'B':
uc = true;
config HAVE_MEMBLOCK_PHYS_MAP
bool
-config HAVE_GENERIC_RCU_GUP
+config HAVE_GENERIC_GUP
bool
config ARCH_DISCARD_MEMBLOCK
int pool_id = CLEANCACHE_NO_BACKEND_SHARED;
if (cleancache_ops) {
- pool_id = cleancache_ops->init_shared_fs(sb->s_uuid, PAGE_SIZE);
+ pool_id = cleancache_ops->init_shared_fs(&sb->s_uuid, PAGE_SIZE);
if (pool_id < 0)
pool_id = CLEANCACHE_NO_POOL;
}
}
EXPORT_SYMBOL(filemap_flush);
+/**
+ * filemap_range_has_page - check if a page exists in range.
+ * @mapping: address space within which to check
+ * @start_byte: offset in bytes where the range starts
+ * @end_byte: offset in bytes where the range ends (inclusive)
+ *
+ * Find at least one page in the range supplied, usually used to check if
+ * direct writing in this range will trigger a writeback.
+ */
+bool filemap_range_has_page(struct address_space *mapping,
+ loff_t start_byte, loff_t end_byte)
+{
+ pgoff_t index = start_byte >> PAGE_SHIFT;
+ pgoff_t end = end_byte >> PAGE_SHIFT;
+ struct pagevec pvec;
+ bool ret;
+
+ if (end_byte < start_byte)
+ return false;
+
+ if (mapping->nrpages == 0)
+ return false;
+
+ pagevec_init(&pvec, 0);
+ if (!pagevec_lookup(&pvec, mapping, index, 1))
+ return false;
+ ret = (pvec.pages[0]->index <= end);
+ pagevec_release(&pvec);
+ return ret;
+}
+EXPORT_SYMBOL(filemap_range_has_page);
+
static int __filemap_fdatawait_range(struct address_space *mapping,
loff_t start_byte, loff_t end_byte)
{
struct wait_page_queue {
struct page *page;
int bit_nr;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
};
-static int wake_page_function(wait_queue_t *wait, unsigned mode, int sync, void *arg)
+static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, void *arg)
{
struct wait_page_key *key = arg;
struct wait_page_queue *wait_page
struct page *page, int bit_nr, int state, bool lock)
{
struct wait_page_queue wait_page;
- wait_queue_t *wait = &wait_page.wait;
+ wait_queue_entry_t *wait = &wait_page.wait;
int ret = 0;
init_wait(wait);
for (;;) {
spin_lock_irq(&q->lock);
- if (likely(list_empty(&wait->task_list))) {
+ if (likely(list_empty(&wait->entry))) {
if (lock)
- __add_wait_queue_tail_exclusive(q, wait);
+ __add_wait_queue_entry_tail_exclusive(q, wait);
else
__add_wait_queue(q, wait);
SetPageWaiters(page);
*
* Add an arbitrary @waiter to the wait queue for the nominated @page.
*/
-void add_page_wait_queue(struct page *page, wait_queue_t *waiter)
+void add_page_wait_queue(struct page *page, wait_queue_entry_t *waiter)
{
wait_queue_head_t *q = page_waitqueue(page);
unsigned long flags;
loff_t size;
size = i_size_read(inode);
- retval = filemap_write_and_wait_range(mapping, iocb->ki_pos,
- iocb->ki_pos + count - 1);
- if (retval < 0)
- goto out;
+ if (iocb->ki_flags & IOCB_NOWAIT) {
+ if (filemap_range_has_page(mapping, iocb->ki_pos,
+ iocb->ki_pos + count - 1))
+ return -EAGAIN;
+ } else {
+ retval = filemap_write_and_wait_range(mapping,
+ iocb->ki_pos,
+ iocb->ki_pos + count - 1);
+ if (retval < 0)
+ goto out;
+ }
file_accessed(file);
pos = iocb->ki_pos;
+ if ((iocb->ki_flags & IOCB_NOWAIT) && !(iocb->ki_flags & IOCB_DIRECT))
+ return -EINVAL;
+
if (limit != RLIM_INFINITY) {
if (iocb->ki_pos >= limit) {
send_sig(SIGXFSZ, current, 0);
write_len = iov_iter_count(from);
end = (pos + write_len - 1) >> PAGE_SHIFT;
- written = filemap_write_and_wait_range(mapping, pos, pos + write_len - 1);
- if (written)
- goto out;
+ if (iocb->ki_flags & IOCB_NOWAIT) {
+ /* If there are pages to writeback, return */
+ if (filemap_range_has_page(inode->i_mapping, pos,
+ pos + iov_iter_count(from)))
+ return -EAGAIN;
+ } else {
+ written = filemap_write_and_wait_range(mapping, pos,
+ pos + write_len - 1);
+ if (written)
+ goto out;
+ }
/*
* After a write we want buffered reads to be sure to go to disk to get
#endif /* CONFIG_ELF_CORE */
/*
- * Generic RCU Fast GUP
+ * Generic Fast GUP
*
* get_user_pages_fast attempts to pin user pages by walking the page
* tables directly and avoids taking locks. Thus the walker needs to be
* Before activating this code, please be aware that the following assumptions
* are currently made:
*
- * *) HAVE_RCU_TABLE_FREE is enabled, and tlb_remove_table is used to free
- * pages containing page tables.
+ * *) Either HAVE_RCU_TABLE_FREE is enabled, and tlb_remove_table() is used to
+ * free pages containing page tables or TLB flushing requires IPI broadcast.
*
* *) ptes can be read atomically by the architecture.
*
*
* This code is based heavily on the PowerPC implementation by Nick Piggin.
*/
-#ifdef CONFIG_HAVE_GENERIC_RCU_GUP
+#ifdef CONFIG_HAVE_GENERIC_GUP
#ifndef gup_get_pte
/*
return ret;
}
-#endif /* CONFIG_HAVE_GENERIC_RCU_GUP */
+#endif /* CONFIG_HAVE_GENERIC_GUP */
spin_unlock(ptl);
free_page_and_swap_cache(src_page);
}
- cond_resched();
}
}
*/
poll_table pt;
wait_queue_head_t *wqh;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
struct work_struct remove;
};
struct oom_wait_info {
struct mem_cgroup *memcg;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
};
-static int memcg_oom_wake_function(wait_queue_t *wait,
+static int memcg_oom_wake_function(wait_queue_entry_t *wait,
unsigned mode, int sync, void *arg)
{
struct mem_cgroup *wake_memcg = (struct mem_cgroup *)arg;
owait.wait.flags = 0;
owait.wait.func = memcg_oom_wake_function;
owait.wait.private = current;
- INIT_LIST_HEAD(&owait.wait.task_list);
+ INIT_LIST_HEAD(&owait.wait.entry);
prepare_to_wait(&memcg_oom_waitq, &owait.wait, TASK_KILLABLE);
mem_cgroup_mark_under_oom(memcg);
*
* Called with wqh->lock held and interrupts disabled.
*/
-static int memcg_event_wake(wait_queue_t *wait, unsigned mode,
+static int memcg_event_wake(wait_queue_entry_t *wait, unsigned mode,
int sync, void *key)
{
struct mem_cgroup_event *event =
{
void *element;
unsigned long flags;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
gfp_t gfp_temp;
VM_WARN_ON_ONCE(gfp_mask & __GFP_ZERO);
/* Check if current node has a suitable gap */
if (gap_start > high_limit)
return -ENOMEM;
- if (gap_end >= low_limit && gap_end - gap_start >= length)
+ if (gap_end >= low_limit &&
+ gap_end > gap_start && gap_end - gap_start >= length)
goto found;
/* Visit right subtree if it looks promising */
gap_end = vm_start_gap(vma);
if (gap_end < low_limit)
return -ENOMEM;
- if (gap_start <= high_limit && gap_end - gap_start >= length)
+ if (gap_start <= high_limit &&
+ gap_end > gap_start && gap_end - gap_start >= length)
goto found;
/* Visit left subtree if it looks promising */
if (!(vma->vm_flags & VM_GROWSUP))
return -EFAULT;
- /* Guard against wrapping around to address 0. */
+ /* Guard against exceeding limits of the address space. */
address &= PAGE_MASK;
- address += PAGE_SIZE;
- if (!address)
+ if (address >= TASK_SIZE)
return -ENOMEM;
+ address += PAGE_SIZE;
/* Enforce stack_guard_gap */
gap_addr = address + stack_guard_gap;
- if (gap_addr < address)
- return -ENOMEM;
+
+ /* Guard against overflow */
+ if (gap_addr < address || gap_addr > TASK_SIZE)
+ gap_addr = TASK_SIZE;
+
next = vma->vm_next;
if (next && next->vm_start < gap_addr) {
if (!(next->vm_flags & VM_GROWSUP))
{
struct page *page = bio->bi_io_vec[0].bv_page;
- if (bio->bi_error) {
+ if (bio->bi_status) {
SetPageError(page);
/*
* We failed to write the page out to swap-space.
{
struct page *page = bio->bi_io_vec[0].bv_page;
- if (bio->bi_error) {
+ if (bio->bi_status) {
SetPageError(page);
ClearPageUptodate(page);
pr_alert("Read-error on swap-device (%u:%u:%llu)\n",
void try_to_unmap_flush(void)
{
struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc;
- int cpu;
if (!tlb_ubc->flush_required)
return;
- cpu = get_cpu();
-
- if (cpumask_test_cpu(cpu, &tlb_ubc->cpumask)) {
- count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ALL);
- local_flush_tlb();
- trace_tlb_flush(TLB_LOCAL_SHOOTDOWN, TLB_FLUSH_ALL);
- }
-
- if (cpumask_any_but(&tlb_ubc->cpumask, cpu) < nr_cpu_ids)
- flush_tlb_others(&tlb_ubc->cpumask, NULL, 0, TLB_FLUSH_ALL);
- cpumask_clear(&tlb_ubc->cpumask);
+ arch_tlbbatch_flush(&tlb_ubc->arch);
tlb_ubc->flush_required = false;
tlb_ubc->writable = false;
- put_cpu();
}
/* Flush iff there are potentially writable TLB entries that can race with IO */
{
struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc;
- cpumask_or(&tlb_ubc->cpumask, &tlb_ubc->cpumask, mm_cpumask(mm));
+ arch_tlbbatch_add_mm(&tlb_ubc->arch, mm);
tlb_ubc->flush_required = true;
/*
#include <uapi/linux/memfd.h>
#include <linux/userfaultfd_k.h>
#include <linux/rmap.h>
+#include <linux/uuid.h>
#include <linux/uaccess.h>
#include <asm/pgtable.h>
* entry unconditionally - even if something else had already woken the
* target.
*/
-static int synchronous_wake_function(wait_queue_t *wait, unsigned mode, int sync, void *key)
+static int synchronous_wake_function(wait_queue_entry_t *wait, unsigned mode, int sync, void *key)
{
int ret = default_wake_function(wait, mode, sync, key);
- list_del_init(&wait->task_list);
+ list_del_init(&wait->entry);
return ret;
}
spin_lock(&inode->i_lock);
inode->i_private = NULL;
wake_up_all(&shmem_falloc_waitq);
- WARN_ON_ONCE(!list_empty(&shmem_falloc_waitq.task_list));
+ WARN_ON_ONCE(!list_empty(&shmem_falloc_waitq.head));
spin_unlock(&inode->i_lock);
error = 0;
goto out;
#ifdef CONFIG_TMPFS_POSIX_ACL
sb->s_flags |= MS_POSIXACL;
#endif
+ uuid_gen(&sb->s_uuid);
inode = shmem_get_inode(sb, NULL, S_IFDIR | sbinfo->mode, 0, VM_NORESERVE);
if (!inode)
return name;
}
+static void sysfs_slab_remove_workfn(struct work_struct *work)
+{
+ struct kmem_cache *s =
+ container_of(work, struct kmem_cache, kobj_remove_work);
+
+ if (!s->kobj.state_in_sysfs)
+ /*
+ * For a memcg cache, this may be called during
+ * deactivation and again on shutdown. Remove only once.
+ * A cache is never shut down before deactivation is
+ * complete, so no need to worry about synchronization.
+ */
+ return;
+
+#ifdef CONFIG_MEMCG
+ kset_unregister(s->memcg_kset);
+#endif
+ kobject_uevent(&s->kobj, KOBJ_REMOVE);
+ kobject_del(&s->kobj);
+ kobject_put(&s->kobj);
+}
+
static int sysfs_slab_add(struct kmem_cache *s)
{
int err;
struct kset *kset = cache_kset(s);
int unmergeable = slab_unmergeable(s);
+ INIT_WORK(&s->kobj_remove_work, sysfs_slab_remove_workfn);
+
if (!kset) {
kobject_init(&s->kobj, &slab_ktype);
return 0;
*/
return;
- if (!s->kobj.state_in_sysfs)
- /*
- * For a memcg cache, this may be called during
- * deactivation and again on shutdown. Remove only once.
- * A cache is never shut down before deactivation is
- * complete, so no need to worry about synchronization.
- */
- return;
-
-#ifdef CONFIG_MEMCG
- kset_unregister(s->memcg_kset);
-#endif
- kobject_uevent(&s->kobj, KOBJ_REMOVE);
- kobject_del(&s->kobj);
+ kobject_get(&s->kobj);
+ schedule_work(&s->kobj_remove_work);
}
void sysfs_slab_release(struct kmem_cache *s)
if (p4d_none(*p4d))
return NULL;
pud = pud_offset(p4d, addr);
- if (pud_none(*pud))
+
+ /*
+ * Don't dereference bad PUD or PMD (below) entries. This will also
+ * identify huge mappings, which we may encounter on architectures
+ * that define CONFIG_HAVE_ARCH_HUGE_VMAP=y. Such regions will be
+ * identified as vmalloc addresses by is_vmalloc_addr(), but are
+ * not [unambiguously] associated with a struct page, so there is
+ * no correct value to return for them.
+ */
+ WARN_ON_ONCE(pud_bad(*pud));
+ if (pud_none(*pud) || pud_bad(*pud))
return NULL;
pmd = pmd_offset(pud, addr);
- if (pmd_none(*pmd))
+ WARN_ON_ONCE(pmd_bad(*pmd));
+ if (pmd_none(*pmd) || pmd_bad(*pmd))
return NULL;
ptep = pte_offset_map(pmd, addr);
pgdat->kswapd = kthread_run(kswapd, pgdat, "kswapd%d", nid);
if (IS_ERR(pgdat->kswapd)) {
/* failure at boot is fatal */
- BUG_ON(system_state == SYSTEM_BOOTING);
+ BUG_ON(system_state < SYSTEM_RUNNING);
pr_err("Failed to start kswapd on node %d\n", nid);
ret = PTR_ERR(pgdat->kswapd);
pgdat->kswapd = NULL;
return 0;
out_free_newdev:
- free_netdev(new_dev);
+ if (new_dev->reg_state == NETREG_UNINITIALIZED)
+ free_netdev(new_dev);
return err;
}
struct p9_poll_wait {
struct p9_conn *conn;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
wait_queue_head_t *wait_addr;
};
clear_bit(Wworksched, &m->wsched);
}
-static int p9_pollwake(wait_queue_t *wait, unsigned int mode, int sync, void *key)
+static int p9_pollwake(wait_queue_entry_t *wait, unsigned int mode, int sync, void *key)
{
struct p9_poll_wait *pwait =
container_of(wait, struct p9_poll_wait, wait);
struct net_device *dev = s->dev;
struct sock *sk = s->sock->sk;
struct sk_buff *skb;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
BT_DBG("");
struct cmtp_session *session = arg;
struct sock *sk = session->sock->sk;
struct sk_buff *skb;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
BT_DBG("session %p", session);
static int hidp_session_thread(void *arg)
{
struct hidp_session *session = arg;
- wait_queue_t ctrl_wait, intr_wait;
+ wait_queue_entry_t ctrl_wait, intr_wait;
BT_DBG("session %p", session);
return sk->sk_type == SOCK_SEQPACKET || sk->sk_type == SOCK_STREAM;
}
-static int receiver_wake_function(wait_queue_t *wait, unsigned int mode, int sync,
+static int receiver_wake_function(wait_queue_entry_t *wait, unsigned int mode, int sync,
void *key)
{
unsigned long bits = (unsigned long)key;
*
* This function will lock the socket if a skb is returned, so
* the caller needs to unlock the socket in that case (usually by
- * calling skb_free_datagram). Returns NULL with *err set to
+ * calling skb_free_datagram). Returns NULL with @err set to
* -EAGAIN if no data was available or to some other value if an
* error was detected.
*
}
EXPORT_SYMBOL(gro_find_complete_by_type);
+static void napi_skb_free_stolen_head(struct sk_buff *skb)
+{
+ skb_dst_drop(skb);
+ secpath_reset(skb);
+ kmem_cache_free(skbuff_head_cache, skb);
+}
+
static gro_result_t napi_skb_finish(gro_result_t ret, struct sk_buff *skb)
{
switch (ret) {
break;
case GRO_MERGED_FREE:
- if (NAPI_GRO_CB(skb)->free == NAPI_GRO_FREE_STOLEN_HEAD) {
- skb_dst_drop(skb);
- secpath_reset(skb);
- kmem_cache_free(skbuff_head_cache, skb);
- } else {
+ if (NAPI_GRO_CB(skb)->free == NAPI_GRO_FREE_STOLEN_HEAD)
+ napi_skb_free_stolen_head(skb);
+ else
__kfree_skb(skb);
- }
break;
case GRO_HELD:
break;
case GRO_DROP:
- case GRO_MERGED_FREE:
napi_reuse_skb(napi, skb);
break;
+ case GRO_MERGED_FREE:
+ if (NAPI_GRO_CB(skb)->free == NAPI_GRO_FREE_STOLEN_HEAD)
+ napi_skb_free_stolen_head(skb);
+ else
+ napi_reuse_skb(napi, skb);
+ break;
+
case GRO_MERGED:
case GRO_CONSUMED:
break;
if (rc == BUSY_POLL_BUDGET)
__napi_schedule(napi);
local_bh_enable();
- if (local_softirq_pending())
- do_softirq();
}
void napi_busy_loop(unsigned int napi_id,
} else {
netdev_stats_to_stats64(storage, &dev->stats);
}
- storage->rx_dropped += atomic_long_read(&dev->rx_dropped);
- storage->tx_dropped += atomic_long_read(&dev->tx_dropped);
- storage->rx_nohandler += atomic_long_read(&dev->rx_nohandler);
+ storage->rx_dropped += (unsigned long)atomic_long_read(&dev->rx_dropped);
+ storage->tx_dropped += (unsigned long)atomic_long_read(&dev->tx_dropped);
+ storage->rx_nohandler += (unsigned long)atomic_long_read(&dev->rx_nohandler);
return storage;
}
EXPORT_SYMBOL(dev_get_stats);
if (cmd == SIOCGIFNAME)
return dev_ifname(net, (struct ifreq __user *)arg);
+ /*
+ * Take care of Wireless Extensions. Unfortunately struct iwreq
+ * isn't a proper subset of struct ifreq (it's 8 byte shorter)
+ * so we need to treat it specially, otherwise applications may
+ * fault if the struct they're passing happens to land at the
+ * end of a mapped page.
+ */
+ if (cmd >= SIOCIWFIRST && cmd <= SIOCIWLAST) {
+ struct iwreq iwr;
+
+ if (copy_from_user(&iwr, arg, sizeof(iwr)))
+ return -EFAULT;
+
+ return wext_handle_ioctl(net, &iwr, cmd, arg);
+ }
+
if (copy_from_user(&ifr, arg, sizeof(struct ifreq)))
return -EFAULT;
ret = -EFAULT;
return ret;
}
- /* Take care of Wireless Extensions */
- if (cmd >= SIOCIWFIRST && cmd <= SIOCIWLAST)
- return wext_handle_ioctl(net, &ifr, cmd, arg);
return -ENOTTY;
}
}
struct net *net = sock_net(skb->sk);
struct fib_rule_hdr *frh = nlmsg_data(nlh);
struct fib_rules_ops *ops = NULL;
- struct fib_rule *rule, *tmp;
+ struct fib_rule *rule, *r;
struct nlattr *tb[FRA_MAX+1];
struct fib_kuid_range range;
int err = -EINVAL;
/*
* Check if this rule is a target to any of them. If so,
+ * adjust to the next one with the same preference or
* disable them. As this operation is eventually very
- * expensive, it is only performed if goto rules have
- * actually been added.
+ * expensive, it is only performed if goto rules, except
+ * current if it is goto rule, have actually been added.
*/
if (ops->nr_goto_rules > 0) {
- list_for_each_entry(tmp, &ops->rules_list, list) {
- if (rtnl_dereference(tmp->ctarget) == rule) {
- RCU_INIT_POINTER(tmp->ctarget, NULL);
+ struct fib_rule *n;
+
+ n = list_next_entry(rule, list);
+ if (&n->list == &ops->rules_list || n->pref != rule->pref)
+ n = NULL;
+ list_for_each_entry(r, &ops->rules_list, list) {
+ if (rtnl_dereference(r->ctarget) != rule)
+ continue;
+ rcu_assign_pointer(r->ctarget, n);
+ if (!n)
ops->unresolved_rules++;
- }
}
}
+ nla_total_size(1) /* IFLA_LINKMODE */
+ nla_total_size(4) /* IFLA_CARRIER_CHANGES */
+ nla_total_size(4) /* IFLA_LINK_NETNSID */
+ + nla_total_size(4) /* IFLA_GROUP */
+ nla_total_size(ext_filter_mask
& RTEXT_FILTER_VF ? 4 : 0) /* IFLA_NUM_VF */
+ rtnl_vfinfo_size(dev, ext_filter_mask) /* IFLA_VFINFO_LIST */
[IFLA_LINK_NETNSID] = { .type = NLA_S32 },
[IFLA_PROTO_DOWN] = { .type = NLA_U8 },
[IFLA_XDP] = { .type = NLA_NESTED },
+ [IFLA_GROUP] = { .type = NLA_U32 },
};
static const struct nla_policy ifla_info_policy[IFLA_INFO_MAX+1] = {
* @sk: socket
*
* This version should be used for very small section, where process wont block
- * return false if fast path is taken
+ * return false if fast path is taken:
+ *
* sk_lock.slock locked, owned = 0, BH disabled
- * return true if slow path is taken
+ *
+ * return true if slow path is taken:
+ *
* sk_lock.slock unlocked, owned = 1, BH enabled
*/
bool lock_sock_fast(struct sock *sk)
call_rcu_bh(&rt->dst.rcu_head, dst_rcu_free);
}
-static inline void dnrt_drop(struct dn_route *rt)
-{
- dst_release(&rt->dst);
- call_rcu_bh(&rt->dst.rcu_head, dst_rcu_free);
-}
-
static void dn_dst_check_expire(unsigned long dummy)
{
int i;
}
*rtp = rt->dst.dn_next;
rt->dst.dn_next = NULL;
- dnrt_drop(rt);
+ dnrt_free(rt);
break;
}
spin_unlock_bh(&dn_rt_hash_table[i].lock);
dst_use(&rth->dst, now);
spin_unlock_bh(&dn_rt_hash_table[hash].lock);
- dnrt_drop(rt);
+ dst_free(&rt->dst);
*rp = rth;
return 0;
}
for(; rt; rt = next) {
next = rcu_dereference_raw(rt->dst.dn_next);
RCU_INIT_POINTER(rt->dst.dn_next, NULL);
- dst_free((struct dst_entry *)rt);
+ dnrt_free(rt);
}
nothing_to_declare:
if (dev_out->flags & IFF_LOOPBACK)
flags |= RTCF_LOCAL;
- rt = dst_alloc(&dn_dst_ops, dev_out, 1, DST_OBSOLETE_NONE, DST_HOST);
+ rt = dst_alloc(&dn_dst_ops, dev_out, 0, DST_OBSOLETE_NONE, DST_HOST);
if (rt == NULL)
goto e_nobufs;
pmc = kzalloc(sizeof(*pmc), GFP_KERNEL);
if (!pmc)
return;
+ spin_lock_init(&pmc->lock);
spin_lock_bh(&im->lock);
pmc->interface = im->interface;
in_dev_hold(in_dev);
csummode = CHECKSUM_PARTIAL;
cork->length += length;
- if ((((length + fragheaderlen) > mtu) || (skb && skb_is_gso(skb))) &&
+ if ((((length + (skb ? skb->len : fragheaderlen)) > mtu) ||
+ (skb && skb_is_gso(skb))) &&
(sk->sk_protocol == IPPROTO_UDP) &&
(rt->dst.dev->features & NETIF_F_UFO) && !dst_xfrm(&rt->dst) &&
(sk->sk_type == SOCK_DGRAM) && !sk->sk_no_check_tx) {
return 0;
drop:
+ if (tun_dst)
+ dst_release((struct dst_entry *)tun_dst);
kfree_skb(skb);
return 0;
}
tcp_init_send_head(sk);
memset(&tp->rx_opt, 0, sizeof(tp->rx_opt));
__sk_dst_reset(sk);
+ dst_release(sk->sk_rx_dst);
+ sk->sk_rx_dst = NULL;
tcp_saved_syn_free(tp);
/* Clean up fastopen related fields */
static void addrconf_mod_dad_work(struct inet6_ifaddr *ifp,
unsigned long delay)
{
- if (!delayed_work_pending(&ifp->dad_work))
- in6_ifa_hold(ifp);
- mod_delayed_work(addrconf_wq, &ifp->dad_work, delay);
+ in6_ifa_hold(ifp);
+ if (mod_delayed_work(addrconf_wq, &ifp->dad_work, delay))
+ in6_ifa_put(ifp);
}
static int snmp6_alloc_dev(struct inet6_dev *idev)
struct net_device *dev = netdev_notifier_info_to_dev(ptr);
struct netdev_notifier_changeupper_info *info;
struct inet6_dev *idev = __in6_dev_get(dev);
+ struct net *net = dev_net(dev);
int run_pending = 0;
int err;
case NETDEV_CHANGEMTU:
/* if MTU under IPV6_MIN_MTU stop IPv6 on this interface. */
if (dev->mtu < IPV6_MIN_MTU) {
- addrconf_ifdown(dev, 1);
+ addrconf_ifdown(dev, dev != net->loopback_dev);
break;
}
* IPV6_MIN_MTU stop IPv6 on this interface.
*/
if (dev->mtu < IPV6_MIN_MTU)
- addrconf_ifdown(dev, 1);
+ addrconf_ifdown(dev, dev != net->loopback_dev);
}
break;
*/
err = ip6_datagram_dst_update(sk, true);
- if (err)
+ if (err) {
+ /* Reset daddr and dport so that udp_v6_early_demux()
+ * fails to find this socket
+ */
+ memset(&sk->sk_v6_daddr, 0, sizeof(sk->sk_v6_daddr));
+ inet->inet_dport = 0;
goto out;
+ }
sk->sk_state = TCP_ESTABLISHED;
sk_set_txhash(sk);
#include <net/ipv6.h>
#include <linux/icmpv6.h>
+static __u16 esp6_nexthdr_esp_offset(struct ipv6hdr *ipv6_hdr, int nhlen)
+{
+ int off = sizeof(struct ipv6hdr);
+ struct ipv6_opt_hdr *exthdr;
+
+ if (likely(ipv6_hdr->nexthdr == NEXTHDR_ESP))
+ return offsetof(struct ipv6hdr, nexthdr);
+
+ while (off < nhlen) {
+ exthdr = (void *)ipv6_hdr + off;
+ if (exthdr->nexthdr == NEXTHDR_ESP)
+ return off;
+
+ off += ipv6_optlen(exthdr);
+ }
+
+ return 0;
+}
+
static struct sk_buff **esp6_gro_receive(struct sk_buff **head,
struct sk_buff *skb)
{
struct xfrm_state *x;
__be32 seq;
__be32 spi;
+ int nhoff;
int err;
skb_pull(skb, offset);
xo->flags |= XFRM_GRO;
+ nhoff = esp6_nexthdr_esp_offset(ipv6_hdr(skb), offset);
+ if (!nhoff)
+ goto out;
+
+ IP6CB(skb)->nhoff = nhoff;
XFRM_TUNNEL_SKB_CB(skb)->tunnel.ip6 = NULL;
XFRM_SPI_SKB_CB(skb)->family = AF_INET6;
XFRM_SPI_SKB_CB(skb)->daddroff = offsetof(struct ipv6hdr, daddr);
struct dst_entry *fib6_rule_lookup(struct net *net, struct flowi6 *fl6,
int flags, pol_lookup_t lookup)
{
- struct rt6_info *rt;
struct fib_lookup_arg arg = {
.lookup_ptr = lookup,
.flags = FIB_LOOKUP_NOREF,
fib_rules_lookup(net->ipv6.fib6_rules_ops,
flowi6_to_flowi(fl6), flags, &arg);
- rt = arg.result;
+ if (arg.result)
+ return arg.result;
- if (!rt) {
- dst_hold(&net->ipv6.ip6_null_entry->dst);
- return &net->ipv6.ip6_null_entry->dst;
- }
-
- if (rt->rt6i_flags & RTF_REJECT &&
- rt->dst.error == -EAGAIN) {
- ip6_rt_put(rt);
- rt = net->ipv6.ip6_null_entry;
- dst_hold(&rt->dst);
- }
-
- return &rt->dst;
+ dst_hold(&net->ipv6.ip6_null_entry->dst);
+ return &net->ipv6.ip6_null_entry->dst;
}
static int fib6_rule_action(struct fib_rule *rule, struct flowi *flp,
flp6->saddr = saddr;
}
err = rt->dst.error;
- goto out;
+ if (err != -EAGAIN)
+ goto out;
}
again:
ip6_rt_put(rt);
struct rt6_info *rt;
rt = lookup(net, net->ipv6.fib6_main_tbl, fl6, flags);
- if (rt->rt6i_flags & RTF_REJECT &&
- rt->dst.error == -EAGAIN) {
+ if (rt->dst.error == -EAGAIN) {
ip6_rt_put(rt);
rt = net->ipv6.ip6_null_entry;
dst_hold(&rt->dst);
*/
cork->length += length;
- if ((((length + fragheaderlen) > mtu) ||
+ if ((((length + (skb ? skb->len : headersize)) > mtu) ||
(skb && skb_is_gso(skb))) &&
(sk->sk_protocol == IPPROTO_UDP) &&
(rt->dst.dev->features & NETIF_F_UFO) && !dst_xfrm(&rt->dst) &&
return 0;
drop:
+ if (tun_dst)
+ dst_release((struct dst_entry *)tun_dst);
kfree_skb(skb);
return 0;
}
fl6.flowi6_proto = IPPROTO_IPIP;
fl6.daddr = key->u.ipv6.dst;
fl6.flowlabel = key->label;
- dsfield = ip6_tclass(key->label);
+ dsfield = key->tos;
} else {
if (!(t->parms.flags & IP6_TNL_F_IGN_ENCAP_LIMIT))
encap_limit = t->parms.encap_limit;
fl6.flowi6_proto = IPPROTO_IPV6;
fl6.daddr = key->u.ipv6.dst;
fl6.flowlabel = key->label;
- dsfield = ip6_tclass(key->label);
+ dsfield = key->tos;
} else {
offset = ip6_tnl_parse_tlv_enc_lim(skb, skb_network_header(skb));
/* ip6_tnl_parse_tlv_enc_lim() might have reallocated skb->head */
net->ipv6.ip6_blk_hole_entry->dst.dev = dev;
net->ipv6.ip6_blk_hole_entry->rt6i_idev = in6_dev_get(dev);
#endif
- } else if (event == NETDEV_UNREGISTER) {
+ } else if (event == NETDEV_UNREGISTER &&
+ dev->reg_state != NETREG_UNREGISTERED) {
+ /* NETDEV_UNREGISTER could be fired for multiple times by
+ * netdev_wait_allrefs(). Make sure we only call this once.
+ */
in6_dev_put(net->ipv6.ip6_null_entry->rt6i_idev);
#ifdef CONFIG_IPV6_MULTIPLE_TABLES
in6_dev_put(net->ipv6.ip6_prohibit_entry->rt6i_idev);
* we try harder to allocate.
*/
kp = (cmax <= 1 || capable(CAP_NET_ADMIN)) ?
- kcalloc(cmax, sizeof(*kp), GFP_KERNEL) :
+ kcalloc(cmax, sizeof(*kp), GFP_KERNEL | __GFP_NOWARN) :
NULL;
rcu_read_lock();
struct sock *sk;
udp_portaddr_for_each_entry_rcu(sk, &hslot2->head) {
- if (INET6_MATCH(sk, net, rmt_addr, loc_addr, ports, dif))
+ if (sk->sk_state == TCP_ESTABLISHED &&
+ INET6_MATCH(sk, net, rmt_addr, loc_addr, ports, dif))
return sk;
/* Only check first socket in chain */
break;
return 1;
#endif
- ipv6_hdr(skb)->payload_len = htons(skb->len);
__skb_push(skb, skb->data - skb_network_header(skb));
+ ipv6_hdr(skb)->payload_len = htons(skb->len - sizeof(struct ipv6hdr));
if (xo && (xo->flags & XFRM_GRO)) {
skb_mac_header_rebuild(skb);
self->settings.flow_control &= ~IRCOMM_RTS_CTS_IN;
}
tty_port_set_check_carrier(&self->port, ~cflag & CLOCAL);
-#if 0
- /*
- * Set up parity check flag
- */
-
- if (I_INPCK(self->tty))
- driver->read_status_mask |= LSR_FE | LSR_PE;
- if (I_BRKINT(driver->tty) || I_PARMRK(driver->tty))
- driver->read_status_mask |= LSR_BI;
-
- /*
- * Characters to ignore
- */
- driver->ignore_status_mask = 0;
- if (I_IGNPAR(driver->tty))
- driver->ignore_status_mask |= LSR_PE | LSR_FE;
-
- if (I_IGNBRK(self->tty)) {
- self->ignore_status_mask |= LSR_BI;
- /*
- * If we're ignore parity and break indicators, ignore
- * overruns too. (For real raw support).
- */
- if (I_IGNPAR(self->tty))
- self->ignore_status_mask |= LSR_OE;
- }
-#endif
+
self->settings.data_format = cval;
ircomm_param_request(self, IRCOMM_DATA_FORMAT, FALSE);
static int ircomm_tty_set_serial_info(struct ircomm_tty_cb *self,
struct serial_struct __user *new_info)
{
-#if 0
- struct serial_struct new_serial;
- struct ircomm_tty_cb old_state, *state;
-
- if (copy_from_user(&new_serial,new_info,sizeof(new_serial)))
- return -EFAULT;
-
-
- state = self
- old_state = *self;
-
- if (!capable(CAP_SYS_ADMIN)) {
- if ((new_serial.baud_base != state->settings.data_rate) ||
- (new_serial.close_delay != state->close_delay) ||
- ((new_serial.flags & ~ASYNC_USR_MASK) !=
- (self->flags & ~ASYNC_USR_MASK)))
- return -EPERM;
- state->flags = ((state->flags & ~ASYNC_USR_MASK) |
- (new_serial.flags & ASYNC_USR_MASK));
- self->flags = ((self->flags & ~ASYNC_USR_MASK) |
- (new_serial.flags & ASYNC_USR_MASK));
- /* self->custom_divisor = new_serial.custom_divisor; */
- goto check_and_exit;
- }
-
- /*
- * OK, past this point, all the error checking has been done.
- * At this point, we start making changes.....
- */
-
- if (self->settings.data_rate != new_serial.baud_base) {
- self->settings.data_rate = new_serial.baud_base;
- ircomm_param_request(self, IRCOMM_DATA_RATE, TRUE);
- }
-
- self->close_delay = new_serial.close_delay * HZ/100;
- self->closing_wait = new_serial.closing_wait * HZ/100;
- /* self->custom_divisor = new_serial.custom_divisor; */
-
- self->flags = ((self->flags & ~ASYNC_FLAGS) |
- (new_serial.flags & ASYNC_FLAGS));
- self->tty->low_latency = (self->flags & ASYNC_LOW_LATENCY) ? 1 : 0;
-
- check_and_exit:
-
- if (tty_port_initialized(self)) {
- if (((old_state.flags & ASYNC_SPD_MASK) !=
- (self->flags & ASYNC_SPD_MASK)) ||
- (old_driver.custom_divisor != driver->custom_divisor)) {
- if ((driver->flags & ASYNC_SPD_MASK) == ASYNC_SPD_HI)
- driver->tty->alt_speed = 57600;
- if ((driver->flags & ASYNC_SPD_MASK) == ASYNC_SPD_VHI)
- driver->tty->alt_speed = 115200;
- if ((driver->flags & ASYNC_SPD_MASK) == ASYNC_SPD_SHI)
- driver->tty->alt_speed = 230400;
- if ((driver->flags & ASYNC_SPD_MASK) == ASYNC_SPD_WARP)
- driver->tty->alt_speed = 460800;
- ircomm_tty_change_speed(driver);
- }
- }
-#endif
return 0;
}
case TIOCGICOUNT:
pr_debug("%s(), TIOCGICOUNT not impl!\n", __func__);
-#if 0
- save_flags(flags); cli();
- cnow = driver->icount;
- restore_flags(flags);
- p_cuser = (struct serial_icounter_struct __user *) arg;
- if (put_user(cnow.cts, &p_cuser->cts) ||
- put_user(cnow.dsr, &p_cuser->dsr) ||
- put_user(cnow.rng, &p_cuser->rng) ||
- put_user(cnow.dcd, &p_cuser->dcd) ||
- put_user(cnow.rx, &p_cuser->rx) ||
- put_user(cnow.tx, &p_cuser->tx) ||
- put_user(cnow.frame, &p_cuser->frame) ||
- put_user(cnow.overrun, &p_cuser->overrun) ||
- put_user(cnow.parity, &p_cuser->parity) ||
- put_user(cnow.brk, &p_cuser->brk) ||
- put_user(cnow.buf_overrun, &p_cuser->buf_overrun))
- return -EFAULT;
-#endif
return 0;
default:
ret = -ENOIOCTLCMD; /* ioctls which we must ignore */
goto out;
}
+ err = -ENOBUFS;
key = ext_hdrs[SADB_EXT_KEY_AUTH - 1];
if (sa->sadb_sa_auth) {
int keysize = 0;
if (key)
keysize = (key->sadb_key_bits + 7) / 8;
x->aalg = kmalloc(sizeof(*x->aalg) + keysize, GFP_KERNEL);
- if (!x->aalg)
+ if (!x->aalg) {
+ err = -ENOMEM;
goto out;
+ }
strcpy(x->aalg->alg_name, a->name);
x->aalg->alg_key_len = 0;
if (key) {
goto out;
}
x->calg = kmalloc(sizeof(*x->calg), GFP_KERNEL);
- if (!x->calg)
+ if (!x->calg) {
+ err = -ENOMEM;
goto out;
+ }
strcpy(x->calg->alg_name, a->name);
x->props.calgo = sa->sadb_sa_encrypt;
} else {
if (key)
keysize = (key->sadb_key_bits + 7) / 8;
x->ealg = kmalloc(sizeof(*x->ealg) + keysize, GFP_KERNEL);
- if (!x->ealg)
+ if (!x->ealg) {
+ err = -ENOMEM;
goto out;
+ }
strcpy(x->ealg->alg_name, a->name);
x->ealg->alg_key_len = 0;
if (key) {
struct xfrm_encap_tmpl *natt;
x->encap = kmalloc(sizeof(*x->encap), GFP_KERNEL);
- if (!x->encap)
+ if (!x->encap) {
+ err = -ENOMEM;
goto out;
+ }
natt = x->encap;
n_type = ext_hdrs[SADB_X_EXT_NAT_T_TYPE-1];
int err, err2;
err = xfrm_policy_flush(net, XFRM_POLICY_TYPE_MAIN, true);
+ if (!err)
+ xfrm_garbage_collect(net);
err2 = unicast_flush_resp(sk, hdr);
if (err || err2) {
if (err == -ESRCH) /* empty table - old silent behavior */
unsigned int *_toklen)
{
const __be32 *xdr = *_xdr;
- unsigned int toklen = *_toklen, n_parts, loop, tmp;
+ unsigned int toklen = *_toklen, n_parts, loop, tmp, paddedlen;
/* there must be at least one name, and at least #names+1 length
* words */
toklen -= 4;
if (tmp <= 0 || tmp > AFSTOKEN_STRING_MAX)
return -EINVAL;
- if (tmp > toklen)
+ paddedlen = (tmp + 3) & ~3;
+ if (paddedlen > toklen)
return -EINVAL;
princ->name_parts[loop] = kmalloc(tmp + 1, GFP_KERNEL);
if (!princ->name_parts[loop])
return -ENOMEM;
memcpy(princ->name_parts[loop], xdr, tmp);
princ->name_parts[loop][tmp] = 0;
- tmp = (tmp + 3) & ~3;
- toklen -= tmp;
- xdr += tmp >> 2;
+ toklen -= paddedlen;
+ xdr += paddedlen >> 2;
}
if (toklen < 4)
toklen -= 4;
if (tmp <= 0 || tmp > AFSTOKEN_K5_REALM_MAX)
return -EINVAL;
- if (tmp > toklen)
+ paddedlen = (tmp + 3) & ~3;
+ if (paddedlen > toklen)
return -EINVAL;
princ->realm = kmalloc(tmp + 1, GFP_KERNEL);
if (!princ->realm)
return -ENOMEM;
memcpy(princ->realm, xdr, tmp);
princ->realm[tmp] = 0;
- tmp = (tmp + 3) & ~3;
- toklen -= tmp;
- xdr += tmp >> 2;
+ toklen -= paddedlen;
+ xdr += paddedlen >> 2;
_debug("%s/...@%s", princ->name_parts[0], princ->realm);
unsigned int *_toklen)
{
const __be32 *xdr = *_xdr;
- unsigned int toklen = *_toklen, len;
+ unsigned int toklen = *_toklen, len, paddedlen;
/* there must be at least one tag and one length word */
if (toklen <= 8)
toklen -= 8;
if (len > max_data_size)
return -EINVAL;
+ paddedlen = (len + 3) & ~3;
+ if (paddedlen > toklen)
+ return -EINVAL;
td->data_len = len;
if (len > 0) {
td->data = kmemdup(xdr, len, GFP_KERNEL);
if (!td->data)
return -ENOMEM;
- len = (len + 3) & ~3;
- toklen -= len;
- xdr += len >> 2;
+ toklen -= paddedlen;
+ xdr += paddedlen >> 2;
}
_debug("tag %x len %x", td->tag, td->data_len);
const __be32 **_xdr, unsigned int *_toklen)
{
const __be32 *xdr = *_xdr;
- unsigned int toklen = *_toklen, len;
+ unsigned int toklen = *_toklen, len, paddedlen;
/* there must be at least one length word */
if (toklen <= 4)
toklen -= 4;
if (len > AFSTOKEN_K5_TIX_MAX)
return -EINVAL;
+ paddedlen = (len + 3) & ~3;
+ if (paddedlen > toklen)
+ return -EINVAL;
*_tktlen = len;
_debug("ticket len %u", len);
*_ticket = kmemdup(xdr, len, GFP_KERNEL);
if (!*_ticket)
return -ENOMEM;
- len = (len + 3) & ~3;
- toklen -= len;
- xdr += len >> 2;
+ toklen -= paddedlen;
+ xdr += paddedlen >> 2;
}
*_xdr = xdr;
{
const __be32 *xdr = prep->data, *token;
const char *cp;
- unsigned int len, tmp, loop, ntoken, toklen, sec_ix;
+ unsigned int len, paddedlen, loop, ntoken, toklen, sec_ix;
size_t datalen = prep->datalen;
int ret;
if (len < 1 || len > AFSTOKEN_CELL_MAX)
goto not_xdr;
datalen -= 4;
- tmp = (len + 3) & ~3;
- if (tmp > datalen)
+ paddedlen = (len + 3) & ~3;
+ if (paddedlen > datalen)
goto not_xdr;
cp = (const char *) xdr;
for (loop = 0; loop < len; loop++)
if (!isprint(cp[loop]))
goto not_xdr;
- if (len < tmp)
- for (; loop < tmp; loop++)
- if (cp[loop])
- goto not_xdr;
+ for (; loop < paddedlen; loop++)
+ if (cp[loop])
+ goto not_xdr;
_debug("cellname: [%u/%u] '%*.*s'",
- len, tmp, len, len, (const char *) xdr);
- datalen -= tmp;
- xdr += tmp >> 2;
+ len, paddedlen, len, len, (const char *) xdr);
+ datalen -= paddedlen;
+ xdr += paddedlen >> 2;
/* get the token count */
if (datalen < 12)
sec_ix = ntohl(*xdr);
datalen -= 4;
_debug("token: [%x/%zx] %x", toklen, datalen, sec_ix);
- if (toklen < 20 || toklen > datalen)
+ paddedlen = (toklen + 3) & ~3;
+ if (toklen < 20 || toklen > datalen || paddedlen > datalen)
goto not_xdr;
- datalen -= (toklen + 3) & ~3;
- xdr += (toklen + 3) >> 2;
+ datalen -= paddedlen;
+ xdr += paddedlen >> 2;
} while (--loop > 0);
return sch;
}
/* ops->init() failed, we call ->destroy() like qdisc_create_dflt() */
- ops->destroy(sch);
+ if (ops->destroy)
+ ops->destroy(sch);
err_out3:
dev_put(dev);
kfree((char *) sch - sch->padded);
if (sctp_sk(sk)->bind_hash)
sctp_put_port(sk);
+ sctp_sk(sk)->ep = NULL;
sock_put(sk);
}
static int sctp_sock_dump(struct sock *sk, void *p)
{
- struct sctp_endpoint *ep = sctp_sk(sk)->ep;
struct sctp_comm_param *commp = p;
struct sk_buff *skb = commp->skb;
struct netlink_callback *cb = commp->cb;
int err = 0;
lock_sock(sk);
- list_for_each_entry(assoc, &ep->asocs, asocs) {
+ if (!sctp_sk(sk)->ep)
+ goto release;
+ list_for_each_entry(assoc, &sctp_sk(sk)->ep->asocs, asocs) {
if (cb->args[4] < cb->args[1])
goto next;
if (err)
return err;
- sctp_transport_get_idx(net, &hti, pos);
- obj = sctp_transport_get_next(net, &hti);
- for (; obj && !IS_ERR(obj); obj = sctp_transport_get_next(net, &hti)) {
+ obj = sctp_transport_get_idx(net, &hti, pos + 1);
+ for (; !IS_ERR_OR_NULL(obj); obj = sctp_transport_get_next(net, &hti)) {
struct sctp_transport *transport = obj;
if (!sctp_transport_hold(transport))
* are still connected to it and there's no way to inform "a polling
* implementation" that it should let go of a certain wait queue
*
- * In order to propagate a wake up, a wait_queue_t of the client
+ * In order to propagate a wake up, a wait_queue_entry_t of the client
* socket is enqueued on the peer_wait queue of the server socket
* whose wake function does a wake_up on the ordinary client socket
* wait queue. This connection is established whenever a write (or
* was relayed.
*/
-static int unix_dgram_peer_wake_relay(wait_queue_t *q, unsigned mode, int flags,
+static int unix_dgram_peer_wake_relay(wait_queue_entry_t *q, unsigned mode, int flags,
void *key)
{
struct unix_sock *u;
* Main IOCTl dispatcher.
* Check the type of IOCTL and call the appropriate wrapper...
*/
-static int wireless_process_ioctl(struct net *net, struct ifreq *ifr,
+static int wireless_process_ioctl(struct net *net, struct iwreq *iwr,
unsigned int cmd,
struct iw_request_info *info,
wext_ioctl_func standard,
wext_ioctl_func private)
{
- struct iwreq *iwr = (struct iwreq *) ifr;
struct net_device *dev;
iw_handler handler;
* The copy_to/from_user() of ifr is also dealt with in there */
/* Make sure the device exist */
- if ((dev = __dev_get_by_name(net, ifr->ifr_name)) == NULL)
+ if ((dev = __dev_get_by_name(net, iwr->ifr_name)) == NULL)
return -ENODEV;
/* A bunch of special cases, then the generic case...
else if (private)
return private(dev, iwr, cmd, info, handler);
}
- /* Old driver API : call driver ioctl handler */
- if (dev->netdev_ops->ndo_do_ioctl)
- return dev->netdev_ops->ndo_do_ioctl(dev, ifr, cmd);
return -EOPNOTSUPP;
}
}
/* entry point from dev ioctl */
-static int wext_ioctl_dispatch(struct net *net, struct ifreq *ifr,
+static int wext_ioctl_dispatch(struct net *net, struct iwreq *iwr,
unsigned int cmd, struct iw_request_info *info,
wext_ioctl_func standard,
wext_ioctl_func private)
if (ret)
return ret;
- dev_load(net, ifr->ifr_name);
+ dev_load(net, iwr->ifr_name);
rtnl_lock();
- ret = wireless_process_ioctl(net, ifr, cmd, info, standard, private);
+ ret = wireless_process_ioctl(net, iwr, cmd, info, standard, private);
rtnl_unlock();
return ret;
}
-int wext_handle_ioctl(struct net *net, struct ifreq *ifr, unsigned int cmd,
+int wext_handle_ioctl(struct net *net, struct iwreq *iwr, unsigned int cmd,
void __user *arg)
{
struct iw_request_info info = { .cmd = cmd, .flags = 0 };
int ret;
- ret = wext_ioctl_dispatch(net, ifr, cmd, &info,
+ ret = wext_ioctl_dispatch(net, iwr, cmd, &info,
ioctl_standard_call,
ioctl_private_call);
if (ret >= 0 &&
IW_IS_GET(cmd) &&
- copy_to_user(arg, ifr, sizeof(struct iwreq)))
+ copy_to_user(arg, iwr, sizeof(struct iwreq)))
return -EFAULT;
return ret;
info.cmd = cmd;
info.flags = IW_REQUEST_FLAG_COMPAT;
- ret = wext_ioctl_dispatch(net, (struct ifreq *) &iwr, cmd, &info,
+ ret = wext_ioctl_dispatch(net, &iwr, cmd, &info,
compat_standard_call,
compat_private_call);
obj-$(CONFIG_XFRM) := xfrm_policy.o xfrm_state.o xfrm_hash.o \
xfrm_input.o xfrm_output.o \
- xfrm_sysctl.o xfrm_replay.o
-obj-$(CONFIG_XFRM_OFFLOAD) += xfrm_device.o
+ xfrm_sysctl.o xfrm_replay.o xfrm_device.o
obj-$(CONFIG_XFRM_STATISTICS) += xfrm_proc.o
obj-$(CONFIG_XFRM_ALGO) += xfrm_algo.o
obj-$(CONFIG_XFRM_USER) += xfrm_user.o
#include <net/xfrm.h>
#include <linux/notifier.h>
+#ifdef CONFIG_XFRM_OFFLOAD
int validate_xmit_xfrm(struct sk_buff *skb, netdev_features_t features)
{
int err;
return true;
}
EXPORT_SYMBOL_GPL(xfrm_dev_offload_ok);
+#endif
int xfrm_dev_register(struct net_device *dev)
{
err = -ESRCH;
out:
spin_unlock_bh(&net->xfrm.xfrm_policy_lock);
-
- if (cnt)
- xfrm_garbage_collect(net);
-
return err;
}
EXPORT_SYMBOL(xfrm_policy_flush);
return 0;
return err;
}
+ xfrm_garbage_collect(net);
c.data.type = type;
c.event = nlh->nlmsg_type;
unifdef
ihex2fw
recordmcount
-docproc
check-lc_ctype
sortextable
asn1_compiler
# pnmttologo: Convert pnm files to logo files
# conmakehash: Create chartable
# conmakehash: Create arrays for initializing the kernel console tables
-# docproc: Used in Documentation/DocBook
-# check-lc_ctype: Used in Documentation/DocBook
HOST_EXTRACFLAGS += -I$(srctree)/tools/include
always := $(hostprogs-y) $(hostprogs-m)
# The following hostprogs-y programs are only build on demand
-hostprogs-y += unifdef docproc check-lc_ctype
+hostprogs-y += unifdef
# These targets are used internally to avoid "is up to date" messages
-PHONY += build_unifdef build_docproc build_check-lc_ctype
+PHONY += build_unifdef
build_unifdef: $(obj)/unifdef
@:
-build_docproc: $(obj)/docproc
- @:
-build_check-lc_ctype: $(obj)/check-lc_ctype
- @:
subdir-$(CONFIG_MODVERSIONS) += genksyms
subdir-y += mod
include scripts/Kbuild.include
srcdir := $(srctree)/$(obj)
-subdirs := $(patsubst $(srcdir)/%/.,%,$(wildcard $(srcdir)/*/.))
+
+# When make is run under a fakechroot environment, the function
+# $(wildcard $(srcdir)/*/.) doesn't only return directories, but also regular
+# files. So, we are using a combination of sort/dir/wildcard which works
+# with fakechroot.
+subdirs := $(patsubst $(srcdir)/%/,%,\
+ $(filter-out $(srcdir)/,\
+ $(sort $(dir $(wildcard $(srcdir)/*/)))))
+
# caller may set destination dir (when installing to asm/)
_dst := $(if $(dst),$(dst),$(obj))
+++ /dev/null
-/*
- * Check that a specified locale works as LC_CTYPE. Used by the
- * DocBook build system to probe for C.UTF-8 support.
- */
-
-#include <locale.h>
-
-int main(void)
-{
- return !setlocale(LC_CTYPE, "");
-}
}
}
-# Check for expedited grace periods that interrupt non-idle non-nohz
-# online CPUs. These expedited can therefore degrade real-time response
-# if used carelessly, and should be avoided where not absolutely
-# needed. It is always OK to use synchronize_rcu_expedited() and
-# synchronize_sched_expedited() at boot time (before real-time applications
-# start) and in error situations where real-time response is compromised in
-# any case. Note that synchronize_srcu_expedited() does -not- interrupt
-# other CPUs, so don't warn on uses of synchronize_srcu_expedited().
-# Of course, nothing comes for free, and srcu_read_lock() and
-# srcu_read_unlock() do contain full memory barriers in payment for
-# synchronize_srcu_expedited() non-interruption properties.
- if ($line =~ /\b(synchronize_rcu_expedited|synchronize_sched_expedited)\(/) {
- WARN("EXPEDITED_RCU_GRACE_PERIOD",
- "expedited RCU grace periods should be avoided where they can degrade real-time response\n" . $herecurr);
-
- }
-
# check of hardware specific defines
if ($line =~ m@^.\s*\#\s*if.*\b(__i386__|__powerpc64__|__sun__|__s390x__)\b@ && $realfile !~ m@include/asm-@) {
CHK("ARCH_DEFINES",
+++ /dev/null
-/*
- * docproc is a simple preprocessor for the template files
- * used as placeholders for the kernel internal documentation.
- * docproc is used for documentation-frontend and
- * dependency-generator.
- * The two usages have in common that they require
- * some knowledge of the .tmpl syntax, therefore they
- * are kept together.
- *
- * documentation-frontend
- * Scans the template file and call kernel-doc for
- * all occurrences of ![EIF]file
- * Beforehand each referenced file is scanned for
- * any symbols that are exported via these macros:
- * EXPORT_SYMBOL(), EXPORT_SYMBOL_GPL(), &
- * EXPORT_SYMBOL_GPL_FUTURE()
- * This is used to create proper -function and
- * -nofunction arguments in calls to kernel-doc.
- * Usage: docproc doc file.tmpl
- *
- * dependency-generator:
- * Scans the template file and list all files
- * referenced in a format recognized by make.
- * Usage: docproc depend file.tmpl
- * Writes dependency information to stdout
- * in the following format:
- * file.tmpl src.c src2.c
- * The filenames are obtained from the following constructs:
- * !Efilename
- * !Ifilename
- * !Dfilename
- * !Ffilename
- * !Pfilename
- *
- */
-
-#define _GNU_SOURCE
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <ctype.h>
-#include <unistd.h>
-#include <limits.h>
-#include <errno.h>
-#include <getopt.h>
-#include <sys/types.h>
-#include <sys/wait.h>
-#include <time.h>
-
-/* exitstatus is used to keep track of any failing calls to kernel-doc,
- * but execution continues. */
-int exitstatus = 0;
-
-typedef void DFL(char *);
-DFL *defaultline;
-
-typedef void FILEONLY(char * file);
-FILEONLY *internalfunctions;
-FILEONLY *externalfunctions;
-FILEONLY *symbolsonly;
-FILEONLY *findall;
-
-typedef void FILELINE(char * file, char * line);
-FILELINE * singlefunctions;
-FILELINE * entity_system;
-FILELINE * docsection;
-
-#define MAXLINESZ 2048
-#define MAXFILES 250
-#define KERNELDOCPATH "scripts/"
-#define KERNELDOC "kernel-doc"
-#define DOCBOOK "-docbook"
-#define RST "-rst"
-#define LIST "-list"
-#define FUNCTION "-function"
-#define NOFUNCTION "-nofunction"
-#define NODOCSECTIONS "-no-doc-sections"
-#define SHOWNOTFOUND "-show-not-found"
-
-enum file_format {
- FORMAT_AUTO,
- FORMAT_DOCBOOK,
- FORMAT_RST,
-};
-
-static enum file_format file_format = FORMAT_AUTO;
-
-#define KERNELDOC_FORMAT (file_format == FORMAT_RST ? RST : DOCBOOK)
-
-static char *srctree, *kernsrctree;
-
-static char **all_list = NULL;
-static int all_list_len = 0;
-
-static void consume_symbol(const char *sym)
-{
- int i;
-
- for (i = 0; i < all_list_len; i++) {
- if (!all_list[i])
- continue;
- if (strcmp(sym, all_list[i]))
- continue;
- all_list[i] = NULL;
- break;
- }
-}
-
-static void usage (void)
-{
- fprintf(stderr, "Usage: docproc [{--docbook|--rst}] {doc|depend} file\n");
- fprintf(stderr, "Input is read from file.tmpl. Output is sent to stdout\n");
- fprintf(stderr, "doc: frontend when generating kernel documentation\n");
- fprintf(stderr, "depend: generate list of files referenced within file\n");
- fprintf(stderr, "Environment variable SRCTREE: absolute path to sources.\n");
- fprintf(stderr, " KBUILD_SRC: absolute path to kernel source tree.\n");
-}
-
-/*
- * Execute kernel-doc with parameters given in svec
- */
-static void exec_kernel_doc(char **svec)
-{
- pid_t pid;
- int ret;
- char real_filename[PATH_MAX + 1];
- /* Make sure output generated so far are flushed */
- fflush(stdout);
- switch (pid=fork()) {
- case -1:
- perror("fork");
- exit(1);
- case 0:
- memset(real_filename, 0, sizeof(real_filename));
- strncat(real_filename, kernsrctree, PATH_MAX);
- strncat(real_filename, "/" KERNELDOCPATH KERNELDOC,
- PATH_MAX - strlen(real_filename));
- execvp(real_filename, svec);
- fprintf(stderr, "exec ");
- perror(real_filename);
- exit(1);
- default:
- waitpid(pid, &ret ,0);
- }
- if (WIFEXITED(ret))
- exitstatus |= WEXITSTATUS(ret);
- else
- exitstatus = 0xff;
-}
-
-/* Types used to create list of all exported symbols in a number of files */
-struct symbols
-{
- char *name;
-};
-
-struct symfile
-{
- char *filename;
- struct symbols *symbollist;
- int symbolcnt;
-};
-
-struct symfile symfilelist[MAXFILES];
-int symfilecnt = 0;
-
-static void add_new_symbol(struct symfile *sym, char * symname)
-{
- sym->symbollist =
- realloc(sym->symbollist, (sym->symbolcnt + 1) * sizeof(char *));
- sym->symbollist[sym->symbolcnt++].name = strdup(symname);
-}
-
-/* Add a filename to the list */
-static struct symfile * add_new_file(char * filename)
-{
- symfilelist[symfilecnt++].filename = strdup(filename);
- return &symfilelist[symfilecnt - 1];
-}
-
-/* Check if file already are present in the list */
-static struct symfile * filename_exist(char * filename)
-{
- int i;
- for (i=0; i < symfilecnt; i++)
- if (strcmp(symfilelist[i].filename, filename) == 0)
- return &symfilelist[i];
- return NULL;
-}
-
-/*
- * List all files referenced within the template file.
- * Files are separated by tabs.
- */
-static void adddep(char * file) { printf("\t%s", file); }
-static void adddep2(char * file, char * line) { line = line; adddep(file); }
-static void noaction(char * line) { line = line; }
-static void noaction2(char * file, char * line) { file = file; line = line; }
-
-/* Echo the line without further action */
-static void printline(char * line) { printf("%s", line); }
-
-/*
- * Find all symbols in filename that are exported with EXPORT_SYMBOL &
- * EXPORT_SYMBOL_GPL (& EXPORT_SYMBOL_GPL_FUTURE implicitly).
- * All symbols located are stored in symfilelist.
- */
-static void find_export_symbols(char * filename)
-{
- FILE * fp;
- struct symfile *sym;
- char line[MAXLINESZ];
- if (filename_exist(filename) == NULL) {
- char real_filename[PATH_MAX + 1];
- memset(real_filename, 0, sizeof(real_filename));
- strncat(real_filename, srctree, PATH_MAX);
- strncat(real_filename, "/", PATH_MAX - strlen(real_filename));
- strncat(real_filename, filename,
- PATH_MAX - strlen(real_filename));
- sym = add_new_file(filename);
- fp = fopen(real_filename, "r");
- if (fp == NULL) {
- fprintf(stderr, "docproc: ");
- perror(real_filename);
- exit(1);
- }
- while (fgets(line, MAXLINESZ, fp)) {
- char *p;
- char *e;
- if (((p = strstr(line, "EXPORT_SYMBOL_GPL")) != NULL) ||
- ((p = strstr(line, "EXPORT_SYMBOL")) != NULL)) {
- /* Skip EXPORT_SYMBOL{_GPL} */
- while (isalnum(*p) || *p == '_')
- p++;
- /* Remove parentheses & additional whitespace */
- while (isspace(*p))
- p++;
- if (*p != '(')
- continue; /* Syntax error? */
- else
- p++;
- while (isspace(*p))
- p++;
- e = p;
- while (isalnum(*e) || *e == '_')
- e++;
- *e = '\0';
- add_new_symbol(sym, p);
- }
- }
- fclose(fp);
- }
-}
-
-/*
- * Document all external or internal functions in a file.
- * Call kernel-doc with following parameters:
- * kernel-doc [-docbook|-rst] -nofunction function_name1 filename
- * Function names are obtained from all the src files
- * by find_export_symbols.
- * intfunc uses -nofunction
- * extfunc uses -function
- */
-static void docfunctions(char * filename, char * type)
-{
- int i,j;
- int symcnt = 0;
- int idx = 0;
- char **vec;
-
- for (i=0; i <= symfilecnt; i++)
- symcnt += symfilelist[i].symbolcnt;
- vec = malloc((2 + 2 * symcnt + 3) * sizeof(char *));
- if (vec == NULL) {
- perror("docproc: ");
- exit(1);
- }
- vec[idx++] = KERNELDOC;
- vec[idx++] = KERNELDOC_FORMAT;
- vec[idx++] = NODOCSECTIONS;
- for (i=0; i < symfilecnt; i++) {
- struct symfile * sym = &symfilelist[i];
- for (j=0; j < sym->symbolcnt; j++) {
- vec[idx++] = type;
- consume_symbol(sym->symbollist[j].name);
- vec[idx++] = sym->symbollist[j].name;
- }
- }
- vec[idx++] = filename;
- vec[idx] = NULL;
- if (file_format == FORMAT_RST)
- printf(".. %s\n", filename);
- else
- printf("<!-- %s -->\n", filename);
- exec_kernel_doc(vec);
- fflush(stdout);
- free(vec);
-}
-static void intfunc(char * filename) { docfunctions(filename, NOFUNCTION); }
-static void extfunc(char * filename) { docfunctions(filename, FUNCTION); }
-
-/*
- * Document specific function(s) in a file.
- * Call kernel-doc with the following parameters:
- * kernel-doc -docbook -function function1 [-function function2]
- */
-static void singfunc(char * filename, char * line)
-{
- char *vec[200]; /* Enough for specific functions */
- int i, idx = 0;
- int startofsym = 1;
- vec[idx++] = KERNELDOC;
- vec[idx++] = KERNELDOC_FORMAT;
- vec[idx++] = SHOWNOTFOUND;
-
- /* Split line up in individual parameters preceded by FUNCTION */
- for (i=0; line[i]; i++) {
- if (isspace(line[i])) {
- line[i] = '\0';
- startofsym = 1;
- continue;
- }
- if (startofsym) {
- startofsym = 0;
- vec[idx++] = FUNCTION;
- vec[idx++] = &line[i];
- }
- }
- for (i = 0; i < idx; i++) {
- if (strcmp(vec[i], FUNCTION))
- continue;
- consume_symbol(vec[i + 1]);
- }
- vec[idx++] = filename;
- vec[idx] = NULL;
- exec_kernel_doc(vec);
-}
-
-/*
- * Insert specific documentation section from a file.
- * Call kernel-doc with the following parameters:
- * kernel-doc -docbook -function "doc section" filename
- */
-static void docsect(char *filename, char *line)
-{
- /* kerneldoc -docbook -show-not-found -function "section" file NULL */
- char *vec[7];
- char *s;
-
- for (s = line; *s; s++)
- if (*s == '\n')
- *s = '\0';
-
- if (asprintf(&s, "DOC: %s", line) < 0) {
- perror("asprintf");
- exit(1);
- }
- consume_symbol(s);
- free(s);
-
- vec[0] = KERNELDOC;
- vec[1] = KERNELDOC_FORMAT;
- vec[2] = SHOWNOTFOUND;
- vec[3] = FUNCTION;
- vec[4] = line;
- vec[5] = filename;
- vec[6] = NULL;
- exec_kernel_doc(vec);
-}
-
-static void find_all_symbols(char *filename)
-{
- char *vec[4]; /* kerneldoc -list file NULL */
- pid_t pid;
- int ret, i, count, start;
- char real_filename[PATH_MAX + 1];
- int pipefd[2];
- char *data, *str;
- size_t data_len = 0;
-
- vec[0] = KERNELDOC;
- vec[1] = LIST;
- vec[2] = filename;
- vec[3] = NULL;
-
- if (pipe(pipefd)) {
- perror("pipe");
- exit(1);
- }
-
- switch (pid=fork()) {
- case -1:
- perror("fork");
- exit(1);
- case 0:
- close(pipefd[0]);
- dup2(pipefd[1], 1);
- memset(real_filename, 0, sizeof(real_filename));
- strncat(real_filename, kernsrctree, PATH_MAX);
- strncat(real_filename, "/" KERNELDOCPATH KERNELDOC,
- PATH_MAX - strlen(real_filename));
- execvp(real_filename, vec);
- fprintf(stderr, "exec ");
- perror(real_filename);
- exit(1);
- default:
- close(pipefd[1]);
- data = malloc(4096);
- do {
- while ((ret = read(pipefd[0],
- data + data_len,
- 4096)) > 0) {
- data_len += ret;
- data = realloc(data, data_len + 4096);
- }
- } while (ret == -EAGAIN);
- if (ret != 0) {
- perror("read");
- exit(1);
- }
- waitpid(pid, &ret ,0);
- }
- if (WIFEXITED(ret))
- exitstatus |= WEXITSTATUS(ret);
- else
- exitstatus = 0xff;
-
- count = 0;
- /* poor man's strtok, but with counting */
- for (i = 0; i < data_len; i++) {
- if (data[i] == '\n') {
- count++;
- data[i] = '\0';
- }
- }
- start = all_list_len;
- all_list_len += count;
- all_list = realloc(all_list, sizeof(char *) * all_list_len);
- str = data;
- for (i = 0; i < data_len && start != all_list_len; i++) {
- if (data[i] == '\0') {
- all_list[start] = str;
- str = data + i + 1;
- start++;
- }
- }
-}
-
-/*
- * Terminate s at first space, if any. If there was a space, return pointer to
- * the character after that. Otherwise, return pointer to the terminating NUL.
- */
-static char *chomp(char *s)
-{
- while (*s && !isspace(*s))
- s++;
-
- if (*s)
- *s++ = '\0';
-
- return s;
-}
-
-/* Return pointer to directive content, or NULL if not a directive. */
-static char *is_directive(char *line)
-{
- if (file_format == FORMAT_DOCBOOK && line[0] == '!')
- return line + 1;
- else if (file_format == FORMAT_RST && !strncmp(line, ".. !", 4))
- return line + 4;
-
- return NULL;
-}
-
-/*
- * Parse file, calling action specific functions for:
- * 1) Lines containing !E
- * 2) Lines containing !I
- * 3) Lines containing !D
- * 4) Lines containing !F
- * 5) Lines containing !P
- * 6) Lines containing !C
- * 7) Default lines - lines not matching the above
- */
-static void parse_file(FILE *infile)
-{
- char line[MAXLINESZ];
- char *p, *s;
- while (fgets(line, MAXLINESZ, infile)) {
- p = is_directive(line);
- if (!p) {
- defaultline(line);
- continue;
- }
-
- switch (*p++) {
- case 'E':
- chomp(p);
- externalfunctions(p);
- break;
- case 'I':
- chomp(p);
- internalfunctions(p);
- break;
- case 'D':
- chomp(p);
- symbolsonly(p);
- break;
- case 'F':
- /* filename */
- s = chomp(p);
- /* function names */
- while (isspace(*s))
- s++;
- singlefunctions(p, s);
- break;
- case 'P':
- /* filename */
- s = chomp(p);
- /* DOC: section name */
- while (isspace(*s))
- s++;
- docsection(p, s);
- break;
- case 'C':
- chomp(p);
- if (findall)
- findall(p);
- break;
- default:
- defaultline(line);
- }
- }
- fflush(stdout);
-}
-
-/*
- * Is this a RestructuredText template? Answer the question by seeing if its
- * name ends in ".rst".
- */
-static int is_rst(const char *file)
-{
- char *dot = strrchr(file, '.');
-
- return dot && !strcmp(dot + 1, "rst");
-}
-
-enum opts {
- OPT_DOCBOOK,
- OPT_RST,
- OPT_HELP,
-};
-
-int main(int argc, char *argv[])
-{
- const char *subcommand, *filename;
- FILE * infile;
- int i;
-
- srctree = getenv("SRCTREE");
- if (!srctree)
- srctree = getcwd(NULL, 0);
- kernsrctree = getenv("KBUILD_SRC");
- if (!kernsrctree || !*kernsrctree)
- kernsrctree = srctree;
-
- for (;;) {
- int c;
- struct option opts[] = {
- { "docbook", no_argument, NULL, OPT_DOCBOOK },
- { "rst", no_argument, NULL, OPT_RST },
- { "help", no_argument, NULL, OPT_HELP },
- {}
- };
-
- c = getopt_long_only(argc, argv, "", opts, NULL);
- if (c == -1)
- break;
-
- switch (c) {
- case OPT_DOCBOOK:
- file_format = FORMAT_DOCBOOK;
- break;
- case OPT_RST:
- file_format = FORMAT_RST;
- break;
- case OPT_HELP:
- usage();
- return 0;
- default:
- case '?':
- usage();
- return 1;
- }
- }
-
- argc -= optind;
- argv += optind;
-
- if (argc != 2) {
- usage();
- exit(1);
- }
-
- subcommand = argv[0];
- filename = argv[1];
-
- if (file_format == FORMAT_AUTO)
- file_format = is_rst(filename) ? FORMAT_RST : FORMAT_DOCBOOK;
-
- /* Open file, exit on error */
- infile = fopen(filename, "r");
- if (infile == NULL) {
- fprintf(stderr, "docproc: ");
- perror(filename);
- exit(2);
- }
-
- if (strcmp("doc", subcommand) == 0) {
- if (file_format == FORMAT_RST) {
- time_t t = time(NULL);
- printf(".. generated from %s by docproc %s\n",
- filename, ctime(&t));
- }
-
- /* Need to do this in two passes.
- * First pass is used to collect all symbols exported
- * in the various files;
- * Second pass generate the documentation.
- * This is required because some functions are declared
- * and exported in different files :-((
- */
- /* Collect symbols */
- defaultline = noaction;
- internalfunctions = find_export_symbols;
- externalfunctions = find_export_symbols;
- symbolsonly = find_export_symbols;
- singlefunctions = noaction2;
- docsection = noaction2;
- findall = find_all_symbols;
- parse_file(infile);
-
- /* Rewind to start from beginning of file again */
- fseek(infile, 0, SEEK_SET);
- defaultline = printline;
- internalfunctions = intfunc;
- externalfunctions = extfunc;
- symbolsonly = printline;
- singlefunctions = singfunc;
- docsection = docsect;
- findall = NULL;
-
- parse_file(infile);
-
- for (i = 0; i < all_list_len; i++) {
- if (!all_list[i])
- continue;
- fprintf(stderr, "Warning: didn't use docs for %s\n",
- all_list[i]);
- }
- } else if (strcmp("depend", subcommand) == 0) {
- /* Create first part of dependency chain
- * file.tmpl */
- printf("%s\t", filename);
- defaultline = noaction;
- internalfunctions = adddep;
- externalfunctions = adddep;
- symbolsonly = adddep;
- singlefunctions = adddep2;
- docsection = adddep2;
- findall = adddep;
- parse_file(infile);
- printf("\n");
- } else {
- fprintf(stderr, "Unknown option: %s\n", subcommand);
- exit(1);
- }
- fclose(infile);
- fflush(stdout);
- return exitstatus;
-}
int yylex(void);
int yyparse(void);
-void error_with_pos(const char *, ...);
+void error_with_pos(const char *, ...) __attribute__ ((format(printf, 1, 2)));
/*----------------------------------------------------------------------*/
#define xmalloc(size) ({ void *__ptr = malloc(size); \
# Check that we have the required ncurses stuff installed for lxdialog (menuconfig)
PHONY += $(obj)/dochecklxdialog
-$(addprefix $(obj)/,$(lxdialog)): $(obj)/dochecklxdialog
+$(addprefix $(obj)/, mconf.o $(lxdialog)): $(obj)/dochecklxdialog
$(obj)/dochecklxdialog:
$(Q)$(CONFIG_SHELL) $(check-lxdialog) -check $(HOSTCC) $(HOST_EXTRACFLAGS) $(HOSTLOADLIBES_mconf)
static int items_num;
static int global_exit;
/* the currently selected button */
-const char *current_instructions = menu_instructions;
+static const char *current_instructions = menu_instructions;
static char *dialog_input_result;
static int dialog_input_result_len;
};
static const int function_keys_num = 9;
-struct function_keys function_keys[] = {
+static struct function_keys function_keys[] = {
{
.key_str = "F1",
.func = "Help",
index = (index + items_num) % items_num;
while (true) {
char *str = k_menu_items[index].str;
- if (strcasestr(str, match_str) != 0)
+ if (strcasestr(str, match_str) != NULL)
return index;
if (flag == FIND_NEXT_MATCH_UP ||
flag == MATCH_TINKER_PATTERN_UP)
static void conf(struct menu *menu)
{
- struct menu *submenu = 0;
+ struct menu *submenu = NULL;
const char *prompt = menu_get_prompt(menu);
struct symbol *sym;
int res;
static void conf_choice(struct menu *menu)
{
const char *prompt = _(menu_get_prompt(menu));
- struct menu *child = 0;
+ struct menu *child = NULL;
struct symbol *active;
int selected_index = 0;
int last_top_row = 0;
}
}
-void setup_windows(void)
+static void setup_windows(void)
{
int lines, columns;
mkattrn(FUNCTION_TEXT, A_REVERSE);
}
-void set_colors()
+void set_colors(void)
{
start_color();
use_default_colors();
int lines = 0;
if (!text)
- return 0;
+ return NULL;
for (i = 0; text[i] != '\0' && lines < line_no; i++)
if (text[i] == '\n')
$members =~ s/\s*CRYPTO_MINALIGN_ATTR//gos;
# replace DECLARE_BITMAP
$members =~ s/DECLARE_BITMAP\s*\(([^,)]+), ([^,)]+)\)/unsigned long $1\[BITS_TO_LONGS($2)\]/gos;
+ # replace DECLARE_HASHTABLE
+ $members =~ s/DECLARE_HASHTABLE\s*\(([^,)]+), ([^,)]+)\)/unsigned long $1\[1 << (($2) - 1)\]/gos;
create_parameterlist($members, ';', $file);
check_sections($file, $declaration_name, "struct", $sectcheck, $struct_actual, $nested);
+++ /dev/null
-#!/usr/bin/perl -w
-
-use strict;
-
-## Copyright (C) 2015 Intel Corporation ##
-# ##
-## This software falls under the GNU General Public License. ##
-## Please read the COPYING file for more information ##
-#
-#
-# This software reads a XML file and a list of valid interal
-# references to replace Docbook tags with links.
-#
-# The list of "valid internal references" must be one-per-line in the following format:
-# API-struct-foo
-# API-enum-bar
-# API-my-function
-#
-# The software walks over the XML file looking for xml tags representing possible references
-# to the Document. Each reference will be cross checked against the "Valid Internal Reference" list. If
-# the referece is found it replaces its content by a <link> tag.
-#
-# usage:
-# kernel-doc-xml-ref -db filename
-# xml filename > outputfile
-
-# read arguments
-if ($#ARGV != 2) {
- usage();
-}
-
-#Holds the database filename
-my $databasefile;
-my @database;
-
-#holds the inputfile
-my $inputfile;
-my $errors = 0;
-
-my %highlights = (
- "<function>(.*?)</function>",
- "\"<function>\" . convert_function(\$1, \$line) . \"</function>\"",
- "<structname>(.*?)</structname>",
- "\"<structname>\" . convert_struct(\$1) . \"</structname>\"",
- "<funcdef>(.*?)<function>(.*?)</function></funcdef>",
- "\"<funcdef>\" . convert_param(\$1) . \"<function>\$2</function></funcdef>\"",
- "<paramdef>(.*?)<parameter>(.*?)</parameter></paramdef>",
- "\"<paramdef>\" . convert_param(\$1) . \"<parameter>\$2</parameter></paramdef>\"");
-
-while($ARGV[0] =~ m/^-(.*)/) {
- my $cmd = shift @ARGV;
- if ($cmd eq "-db") {
- $databasefile = shift @ARGV
- } else {
- usage();
- }
-}
-$inputfile = shift @ARGV;
-
-sub open_database {
- open (my $handle, '<', $databasefile) or die "Cannot open $databasefile";
- chomp(my @lines = <$handle>);
- close $handle;
-
- @database = @lines;
-}
-
-sub process_file {
- open_database();
-
- my $dohighlight;
- foreach my $pattern (keys %highlights) {
- $dohighlight .= "\$line =~ s:$pattern:$highlights{$pattern}:eg;\n";
- }
-
- open(FILE, $inputfile) or die("Could not open $inputfile") or die ("Cannot open $inputfile");
- foreach my $line (<FILE>) {
- eval $dohighlight;
- print $line;
- }
-}
-
-sub trim($_)
-{
- my $str = $_[0];
- $str =~ s/^\s+|\s+$//g;
- return $str
-}
-
-sub has_key_defined($_)
-{
- if ( grep( /^$_[0]$/, @database)) {
- return 1;
- }
- return 0;
-}
-
-# Gets a <function> content and add it a hyperlink if possible.
-sub convert_function($_)
-{
- my $arg = $_[0];
- my $key = $_[0];
-
- my $line = $_[1];
-
- $key = trim($key);
-
- $key =~ s/[^A-Za-z0-9]/-/g;
- $key = "API-" . $key;
-
- # We shouldn't add links to <funcdef> prototype
- if (!has_key_defined($key) || $line =~ m/\s+<funcdef/i) {
- return $arg;
- }
-
- my $head = $arg;
- my $tail = "";
- if ($arg =~ /(.*?)( ?)$/) {
- $head = $1;
- $tail = $2;
- }
- return "<link linkend=\"$key\">$head</link>$tail";
-}
-
-# Converting a struct text to link
-sub convert_struct($_)
-{
- my $arg = $_[0];
- my $key = $_[0];
- $key =~ s/(struct )?(\w)/$2/g;
- $key =~ s/[^A-Za-z0-9]/-/g;
- $key = "API-struct-" . $key;
-
- if (!has_key_defined($key)) {
- return $arg;
- }
-
- my ($head, $tail) = split_pointer($arg);
- return "<link linkend=\"$key\">$head</link>$tail";
-}
-
-# Identify "object *" elements
-sub split_pointer($_)
-{
- my $arg = $_[0];
- if ($arg =~ /(.*?)( ?\* ?)/) {
- return ($1, $2);
- }
- return ($arg, "");
-}
-
-sub convert_param($_)
-{
- my $type = $_[0];
- my $keyname = convert_key_name($type);
-
- if (!has_key_defined($keyname)) {
- return $type;
- }
-
- my ($head, $tail) = split_pointer($type);
- return "<link linkend=\"$keyname\">$head</link>$tail";
-
-}
-
-# DocBook links are in the API-<TYPE>-<STRUCT-NAME> format
-# This method gets an element and returns a valid DocBook reference for it.
-sub convert_key_name($_)
-{
- #Pattern $2 is optional and might be uninitialized
- no warnings 'uninitialized';
-
- my $str = $_[0];
- $str =~ s/(const|static)? ?(struct)? ?([a-zA-Z0-9_]+) ?(\*|&)?/$2 $3/g ;
-
- # trim
- $str =~ s/^\s+|\s+$//g;
-
- # spaces and _ to -
- $str =~ s/[^A-Za-z0-9]/-/g;
-
- return "API-" . $str;
-}
-
-sub usage {
- print "Usage: $0 -db database filename\n";
- print " xml source file(s) > outputfile\n";
- exit 1;
-}
-
-# starting point
-process_file();
-
-if ($errors) {
- print STDERR "$errors errors\n";
-}
-
-exit($errors);
-Please see Documentation/security/SELinux.txt for information on
+Please see Documentation/admin-guide/LSM/SELinux.rst for information on
installing a dummy SELinux policy.
case "$i" in
*.[cS])
j=${i/\.[cS]/\.o}
+ j="${j#$tree}"
if [ -e $j ]; then
echo $i
fi
* @flags: flags controlling what type of accept tables are acceptable
*
* Unpack a dfa that has been serialized. To find information on the dfa
- * format look in Documentation/security/apparmor.txt
+ * format look in Documentation/admin-guide/LSM/apparmor.rst
* Assumes the dfa @blob stream has been aligned on a 8 byte boundary
*
* Returns: an unpacked dfa ready for matching or ERR_PTR on failure
* License.
*
* AppArmor uses a serialized binary format for loading policy. To find
- * policy format documentation look in Documentation/security/apparmor.txt
+ * policy format documentation see Documentation/admin-guide/LSM/apparmor.rst
* All policy is validated before it is used.
*/
hmac_misc.mode = inode->i_mode;
crypto_shash_update(desc, (const u8 *)&hmac_misc, sizeof(hmac_misc));
if (evm_hmac_attrs & EVM_ATTR_FSUUID)
- crypto_shash_update(desc, inode->i_sb->s_uuid,
+ crypto_shash_update(desc, &inode->i_sb->s_uuid.b[0],
sizeof(inode->i_sb->s_uuid));
crypto_shash_final(desc, digest);
}
enum ima_hooks func;
int mask;
unsigned long fsmagic;
- u8 fsuuid[16];
+ uuid_t fsuuid;
kuid_t uid;
kuid_t fowner;
bool (*uid_op)(kuid_t, kuid_t); /* Handlers for operators */
&& rule->fsmagic != inode->i_sb->s_magic)
return false;
if ((rule->flags & IMA_FSUUID) &&
- memcmp(rule->fsuuid, inode->i_sb->s_uuid, sizeof(rule->fsuuid)))
+ !uuid_equal(&rule->fsuuid, &inode->i_sb->s_uuid))
return false;
if ((rule->flags & IMA_UID) && !rule->uid_op(cred->uid, rule->uid))
return false;
case Opt_fsuuid:
ima_log_string(ab, "fsuuid", args[0].from);
- if (memchr_inv(entry->fsuuid, 0x00,
- sizeof(entry->fsuuid))) {
+ if (uuid_is_null(&entry->fsuuid)) {
result = -EINVAL;
break;
}
- result = blk_part_pack_uuid(args[0].from,
- entry->fsuuid);
+ result = uuid_parse(args[0].from, &entry->fsuuid);
if (!result)
entry->flags |= IMA_FSUUID;
break;
}
if (entry->flags & IMA_FSUUID) {
- seq_printf(m, "fsuuid=%pU", entry->fsuuid);
+ seq_printf(m, "fsuuid=%pU", &entry->fsuuid);
seq_puts(m, " ");
}
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, version 2 of the License.
*
- * See Documentation/security/keys-trusted-encrypted.txt
+ * See Documentation/security/keys/trusted-encrypted.rst
*/
#include <linux/uaccess.h>
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, version 2 of the License.
*
- * See Documentation/security/keys-trusted-encrypted.txt
+ * See Documentation/security/keys/trusted-encrypted.rst
*/
#include <linux/uaccess.h>
#define _INTERNAL_H
#include <linux/sched.h>
+#include <linux/wait_bit.h>
#include <linux/cred.h>
#include <linux/key-type.h>
#include <linux/task_work.h>
* as published by the Free Software Foundation; either version
* 2 of the License, or (at your option) any later version.
*
- * See Documentation/security/keys-request-key.txt
+ * See Documentation/security/keys/request-key.rst
*/
#include <linux/module.h>
* as published by the Free Software Foundation; either version
* 2 of the License, or (at your option) any later version.
*
- * See Documentation/security/keys-request-key.txt
+ * See Documentation/security/keys/request-key.rst
*/
#include <linux/module.h>
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, version 2 of the License.
*
- * See Documentation/security/keys-trusted-encrypted.txt
+ * See Documentation/security/keys/trusted-encrypted.rst
*/
#include <crypto/hash_info.h>
system-wide security settings beyond regular Linux discretionary
access controls. Currently available is ptrace scope restriction.
Like capabilities, this security module stacks with other LSMs.
- Further information can be found in Documentation/security/Yama.txt.
+ Further information can be found in
+ Documentation/admin-guide/LSM/Yama.rst.
If you are unsure how to answer this question, answer N.
struct snd_ctl_event ev;
struct snd_kctl_event *kev;
while (list_empty(&ctl->events)) {
- wait_queue_t wait;
+ wait_queue_entry_t wait;
if ((file->f_flags & O_NONBLOCK) != 0 || result > 0) {
err = -EAGAIN;
goto __end_lock;
int major = imajor(inode);
struct snd_hwdep *hw;
int err;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
if (major == snd_major) {
hw = snd_lookup_minor_data(iminor(inode),
*/
int snd_power_wait(struct snd_card *card, unsigned int power_state)
{
- wait_queue_t wait;
+ wait_queue_entry_t wait;
int result = 0;
/* fastpath */
ssize_t result = 0;
snd_pcm_state_t state;
long res;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
runtime = substream->runtime;
init_waitqueue_entry(&wait, current);
struct snd_pcm_oss_file *pcm_oss_file;
struct snd_pcm_oss_setup setup[2];
int nonblock;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
err = nonseekable_open(inode, file);
if (err < 0)
{
struct snd_pcm_runtime *runtime = substream->runtime;
int is_playback = substream->stream == SNDRV_PCM_STREAM_PLAYBACK;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
int err = 0;
snd_pcm_uframes_t avail = 0;
long wait_time, tout;
struct snd_pcm_substream *substream;
const struct snd_pcm_chmap_elem *map;
- if (snd_BUG_ON(!info->chmap))
+ if (!info->chmap)
return -EINVAL;
substream = snd_pcm_chmap_substream(info, idx);
if (!substream)
unsigned int __user *dst;
int c, count = 0;
- if (snd_BUG_ON(!info->chmap))
+ if (!info->chmap)
return -EINVAL;
if (size < 8)
return -ENOMEM;
struct snd_card *card;
struct snd_pcm_runtime *runtime;
struct snd_pcm_substream *s;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
int result = 0;
int nonblock = 0;
static int snd_pcm_open(struct file *file, struct snd_pcm *pcm, int stream)
{
int err;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
if (pcm == NULL) {
err = -ENODEV;
int err;
struct snd_rawmidi *rmidi;
struct snd_rawmidi_file *rawmidi_file = NULL;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
if ((file->f_flags & O_APPEND) && !(file->f_flags & O_NONBLOCK))
return -EINVAL; /* invalid combination */
while (count > 0) {
spin_lock_irq(&runtime->lock);
while (!snd_rawmidi_ready(substream)) {
- wait_queue_t wait;
+ wait_queue_entry_t wait;
if ((file->f_flags & O_NONBLOCK) != 0 || result > 0) {
spin_unlock_irq(&runtime->lock);
return result > 0 ? result : -EAGAIN;
while (count > 0) {
spin_lock_irq(&runtime->lock);
while (!snd_rawmidi_ready_append(substream, count)) {
- wait_queue_t wait;
+ wait_queue_entry_t wait;
if (file->f_flags & O_NONBLOCK) {
spin_unlock_irq(&runtime->lock);
return result > 0 ? result : -EAGAIN;
if (file->f_flags & O_DSYNC) {
spin_lock_irq(&runtime->lock);
while (runtime->avail != runtime->buffer_size) {
- wait_queue_t wait;
+ wait_queue_entry_t wait;
unsigned int last_avail = runtime->avail;
init_waitqueue_entry(&wait, current);
add_wait_queue(&runtime->sleep, &wait);
{
struct snd_seq_event_cell *cell;
unsigned long flags;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
if (snd_BUG_ON(!f))
return -EINVAL;
struct snd_seq_event_cell *cell;
unsigned long flags;
int err = -EAGAIN;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
if (pool == NULL)
return -EINVAL;
spin_lock_irq(&tu->qlock);
while ((long)count - result >= unit) {
while (!tu->qused) {
- wait_queue_t wait;
+ wait_queue_entry_t wait;
if ((file->f_flags & O_NONBLOCK) != 0 || result > 0) {
err = -EAGAIN;
cycle = increment_cycle_count(cycle, 1);
if (s->handle_packet(s, 0, cycle, i) < 0) {
s->packet_index = -1;
- amdtp_stream_pcm_abort(s);
+ if (in_interrupt())
+ amdtp_stream_pcm_abort(s);
+ WRITE_ONCE(s->pcm_buffer_pointer, SNDRV_PCM_POS_XRUN);
return;
}
}
/* Queueing error or detecting invalid payload. */
if (i < packets) {
s->packet_index = -1;
- amdtp_stream_pcm_abort(s);
+ if (in_interrupt())
+ amdtp_stream_pcm_abort(s);
+ WRITE_ONCE(s->pcm_buffer_pointer, SNDRV_PCM_POS_XRUN);
return;
}
/* For a PCM substream processing. */
struct snd_pcm_substream *pcm;
struct tasklet_struct period_tasklet;
- unsigned int pcm_buffer_pointer;
+ snd_pcm_uframes_t pcm_buffer_pointer;
unsigned int pcm_period_pointer;
/* To wait for first packet. */
int val, int port, unsigned long timeout)
{
- wait_queue_t wait;
+ wait_queue_entry_t wait;
init_waitqueue_entry(&wait, current);
spin_lock_irq(&dev->irq_lock);
#define list_for_each_codec(c, bus) \
list_for_each_entry(c, &(bus)->core.codec_list, core.list)
+#define list_for_each_codec_safe(c, n, bus) \
+ list_for_each_entry_safe(c, n, &(bus)->core.codec_list, core.list)
/* snd_hda_codec_read/write optional flags */
#define HDA_RW_NO_RESPONSE_FALLBACK (1 << 0)
/* configure each codec instance */
int azx_codec_configure(struct azx *chip)
{
- struct hda_codec *codec;
- list_for_each_codec(codec, &chip->bus) {
+ struct hda_codec *codec, *next;
+
+ /* use _safe version here since snd_hda_codec_configure() deregisters
+ * the device upon error and deletes itself from the bus list.
+ */
+ list_for_each_codec_safe(codec, next, &chip->bus) {
snd_hda_codec_configure(codec);
}
return 0;
spec->input_paths[i][nums]);
spec->input_paths[i][nums] =
spec->input_paths[i][n];
+ spec->input_paths[i][n] = 0;
}
}
nums++;
#define IS_KBL_LP(pci) ((pci)->vendor == 0x8086 && (pci)->device == 0x9d71)
#define IS_KBL_H(pci) ((pci)->vendor == 0x8086 && (pci)->device == 0xa2f0)
#define IS_BXT(pci) ((pci)->vendor == 0x8086 && (pci)->device == 0x5a98)
+#define IS_BXT_T(pci) ((pci)->vendor == 0x8086 && (pci)->device == 0x1a98)
#define IS_GLK(pci) ((pci)->vendor == 0x8086 && (pci)->device == 0x3198)
-#define IS_SKL_PLUS(pci) (IS_SKL(pci) || IS_SKL_LP(pci) || IS_BXT(pci)) || \
- IS_KBL(pci) || IS_KBL_LP(pci) || IS_KBL_H(pci) || \
- IS_GLK(pci)
+#define IS_CFL(pci) ((pci)->vendor == 0x8086 && (pci)->device == 0xa348)
+#define IS_SKL_PLUS(pci) (IS_SKL(pci) || IS_SKL_LP(pci) || IS_BXT(pci) || \
+ IS_BXT_T(pci) || IS_KBL(pci) || IS_KBL_LP(pci) || \
+ IS_KBL_H(pci) || IS_GLK(pci) || IS_CFL(pci))
static char *driver_short_names[] = {
[AZX_DRIVER_ICH] = "HDA Intel",
/* Kabylake-H */
{ PCI_DEVICE(0x8086, 0xa2f0),
.driver_data = AZX_DRIVER_PCH | AZX_DCAPS_INTEL_SKYLAKE },
+ /* Coffelake */
+ { PCI_DEVICE(0x8086, 0xa348),
+ .driver_data = AZX_DRIVER_PCH | AZX_DCAPS_INTEL_SKYLAKE},
/* Broxton-P(Apollolake) */
{ PCI_DEVICE(0x8086, 0x5a98),
.driver_data = AZX_DRIVER_PCH | AZX_DCAPS_INTEL_BROXTON },
struct mixart_msg resp;
u32 msg_frame = 0; /* set to 0, so it's no notification to wait for, but the answer */
int err;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
long timeout;
init_waitqueue_entry(&wait, current);
struct mixart_msg *request, u32 notif_event)
{
int err;
- wait_queue_t wait;
+ wait_queue_entry_t wait;
long timeout;
if (snd_BUG_ON(!notif_event))
static void snd_ymfpci_irq_wait(struct snd_ymfpci *chip)
{
- wait_queue_t wait;
+ wait_queue_entry_t wait;
int loops = 4;
while (loops-- > 0) {
#include "skl.h"
/* Unique identification for getting NHLT blobs */
-static u8 OSC_UUID[16] = {0x6E, 0x88, 0x9F, 0xA6, 0xEB, 0x6C, 0x94, 0x45,
- 0xA4, 0x1F, 0x7B, 0x5D, 0xCE, 0x24, 0xC5, 0x53};
+static guid_t osc_guid =
+ GUID_INIT(0xA69F886E, 0x6CEB, 0x4594,
+ 0xA4, 0x1F, 0x7B, 0x5D, 0xCE, 0x24, 0xC5, 0x53);
struct nhlt_acpi_table *skl_nhlt_init(struct device *dev)
{
return NULL;
}
- obj = acpi_evaluate_dsm(handle, OSC_UUID, 1, 1, NULL);
+ obj = acpi_evaluate_dsm(handle, &osc_guid, 1, 1, NULL);
if (obj && obj->type == ACPI_TYPE_BUFFER) {
nhlt_ptr = (struct nhlt_resource_desc *)obj->buffer.pointer;
nhlt_table = (struct nhlt_acpi_table *)
@echo ' kvm_stat - top-like utility for displaying kvm statistics'
@echo ' leds - LEDs tools'
@echo ' lguest - a minimal 32-bit x86 hypervisor'
+ @echo ' liblockdep - user-space wrapper for kernel locking-validator'
@echo ' net - misc networking tools'
@echo ' perf - Linux performance measurement and analysis tool'
@echo ' selftests - various kernel selftests'
kvm_stat: FORCE
$(call descend,kvm/$@)
-all: acpi cgroup cpupower gpio hv firewire lguest \
+all: acpi cgroup cpupower gpio hv firewire lguest liblockdep \
perf selftests turbostat usb \
virtio vm net x86_energy_perf_policy \
tmon freefall objtool kvm_stat
cgroup_install firewire_install gpio_install hv_install lguest_install perf_install usb_install virtio_install vm_install net_install objtool_install:
$(call descend,$(@:_install=),install)
+liblockdep_install:
+ $(call descend,lib/lockdep,install)
+
selftests_install:
$(call descend,testing/$(@:_install=),install)
$(call descend,kvm/$(@:_install=),install)
install: acpi_install cgroup_install cpupower_install gpio_install \
- hv_install firewire_install lguest_install \
+ hv_install firewire_install lguest_install liblockdep_install \
perf_install selftests_install turbostat_install usb_install \
virtio_install vm_install net_install x86_energy_perf_policy_install \
tmon_install freefall_install objtool_install kvm_stat_install
#include <fcntl.h>
#include <dirent.h>
#include <net/if.h>
+#include <limits.h>
#include <getopt.h>
/*
#define KVP_SCRIPTS_PATH "/usr/libexec/hypervkvpd/"
#endif
+#define KVP_NET_DIR "/sys/class/net/"
+
#define MAX_FILE_NAME 100
#define ENTRIES_PER_BLOCK 50
DIR *dir;
struct dirent *entry;
FILE *file;
- char *p, *q, *x;
+ char *p, *x;
char *if_name = NULL;
char buf[256];
- char *kvp_net_dir = "/sys/class/net/";
- char dev_id[256];
+ char dev_id[PATH_MAX];
- dir = opendir(kvp_net_dir);
+ dir = opendir(KVP_NET_DIR);
if (dir == NULL)
return NULL;
- snprintf(dev_id, sizeof(dev_id), "%s", kvp_net_dir);
- q = dev_id + strlen(kvp_net_dir);
-
while ((entry = readdir(dir)) != NULL) {
/*
* Set the state for the next pass.
*/
- *q = '\0';
- strcat(dev_id, entry->d_name);
- strcat(dev_id, "/device/device_id");
+ snprintf(dev_id, sizeof(dev_id), "%s%s/device/device_id",
+ KVP_NET_DIR, entry->d_name);
file = fopen(dev_id, "r");
if (file == NULL)
FILE *file;
char *p, *x;
char buf[256];
- char addr_file[256];
+ char addr_file[PATH_MAX];
unsigned int i;
char *mac_addr = NULL;
- snprintf(addr_file, sizeof(addr_file), "%s%s%s", "/sys/class/net/",
- if_name, "/address");
+ snprintf(addr_file, sizeof(addr_file), "%s%s%s", KVP_NET_DIR,
+ if_name, "/address");
file = fopen(addr_file, "r");
if (file == NULL)
DIR *dir;
struct dirent *entry;
FILE *file;
- char *p, *q, *x;
+ char *p, *x;
char *if_name = NULL;
char buf[256];
- char *kvp_net_dir = "/sys/class/net/";
- char dev_id[256];
+ char dev_id[PATH_MAX];
unsigned int i;
- dir = opendir(kvp_net_dir);
+ dir = opendir(KVP_NET_DIR);
if (dir == NULL)
return NULL;
- snprintf(dev_id, sizeof(dev_id), "%s", kvp_net_dir);
- q = dev_id + strlen(kvp_net_dir);
-
while ((entry = readdir(dir)) != NULL) {
/*
* Set the state for the next pass.
*/
- *q = '\0';
-
- strcat(dev_id, entry->d_name);
- strcat(dev_id, "/address");
+ snprintf(dev_id, sizeof(dev_id), "%s%s/address", KVP_NET_DIR,
+ entry->d_name);
file = fopen(dev_id, "r");
if (file == NULL)
static int kvp_set_ip_info(char *if_name, struct hv_kvp_ipaddr_value *new_val)
{
int error = 0;
- char if_file[128];
+ char if_file[PATH_MAX];
FILE *file;
- char cmd[512];
+ char cmd[PATH_MAX];
char *mac_addr;
/*
if (len != sizeof(struct hv_vss_msg)) {
syslog(LOG_ERR, "write failed; error: %d %s", errno,
strerror(errno));
- exit(EXIT_FAILURE);
+
+ if (op == VSS_OP_FREEZE)
+ vss_operate(VSS_OP_THAW);
}
}
CC = $(CROSS_COMPILE)gcc
-CFLAGS += -Wall -g -D_GNU_SOURCE
+CFLAGS += -Wall -g -D_GNU_SOURCE -D__EXPORTED_HEADERS__ -I../../include/uapi -I../../include
BINDIR=usr/bin
INSTALL_PROGRAM=install -m 755 -p
#include <stdint.h>
/* Made up value to limit allocation sizes */
-#define IIO_MAX_NAME_LENGTH 30
+#define IIO_MAX_NAME_LENGTH 64
#define FORMAT_SCAN_ELEMENTS_DIR "%s/scan_elements"
#define FORMAT_TYPE_FILE "%s_type"
--- /dev/null
+#ifndef __TOOLS_INCLUDE_LINUX_ASM_SECTIONS_H
+#define __TOOLS_INCLUDE_LINUX_ASM_SECTIONS_H
+
+#endif /* __TOOLS_INCLUDE_LINUX_ASM_SECTIONS_H */
return fls64(l);
}
+/**
+ * rol32 - rotate a 32-bit value left
+ * @word: value to rotate
+ * @shift: bits to roll
+ */
+static inline __u32 rol32(__u32 word, unsigned int shift)
+{
+ return (word << shift) | (word >> ((-shift) & 31));
+}
+
#endif
/* &a[0] degrades to a pointer: a different type from an array */
#define __must_be_array(a) BUILD_BUG_ON_ZERO(__same_type((a), &(a)[0]))
+
+#define noinline __attribute__((noinline))
+
+#define __packed __attribute__((packed))
+
+#define __noreturn __attribute__((noreturn))
+
+#define __aligned(x) __attribute__((aligned(x)))
+#define __printf(a, b) __attribute__((format(printf, a, b)))
+#define __scanf(a, b) __attribute__((format(scanf, a, b)))
# define __always_inline inline __attribute__((always_inline))
#endif
+#ifndef noinline
+#define noinline
+#endif
+
/* Are two types/vars the same type (ignoring qualifiers)? */
#ifndef __same_type
# define __same_type(a, b) __builtin_types_compatible_p(typeof(a), typeof(b))
# define __maybe_unused __attribute__((unused))
#endif
+#ifndef __used
+# define __used __attribute__((__unused__))
+#endif
+
#ifndef __packed
# define __packed __attribute__((__packed__))
#endif
# define unlikely(x) __builtin_expect(!!(x), 0)
#endif
+#ifndef __init
+# define __init
+#endif
+
+#ifndef noinline
+# define noinline
+#endif
+
#define uninitialized_var(x) x = *(&(x))
#define ACCESS_ONCE(x) (*(volatile typeof(x) *)&(x))
#include <stddef.h>
#include <linux/compiler.h>
+#include <asm/bug.h>
-#define DEBUG_LOCKS_WARN_ON(x) (x)
+#define DEBUG_LOCKS_WARN_ON(x) WARN_ON(x)
extern bool debug_locks;
extern bool debug_locks_silent;
--- /dev/null
+#ifndef _TOOLS_INCLUDE_LINUX_DELAY_H
+#define _TOOLS_INCLUDE_LINUX_DELAY_H
+
+#endif /* _TOOLS_INCLUDE_LINUX_DELAY_H */
return IS_ERR_VALUE((unsigned long)ptr);
}
+static inline bool __must_check IS_ERR_OR_NULL(__force const void *ptr)
+{
+ return unlikely(!ptr) || IS_ERR_VALUE((unsigned long)ptr);
+}
+
#endif /* _LINUX_ERR_H */
--- /dev/null
+#ifndef _TOOLS_INCLUDE_LINUX_FTRACE_H
+#define _TOOLS_INCLUDE_LINUX_FTRACE_H
+
+#endif /* _TOOLS_INCLUDE_LINUX_FTRACE_H */
--- /dev/null
+#ifndef _TOOLS_INCLUDE_LINUX_GFP_H
+#define _TOOLS_INCLUDE_LINUX_GFP_H
+
+#endif /* _TOOLS_INCLUDE_LINUX_GFP_H */
--- /dev/null
+#ifndef _TOOLS_INCLUDE_LINUX_INTERRUPT_H
+#define _TOOLS_INCLUDE_LINUX_INTERRUPT_H
+
+#endif /* _TOOLS_INCLUDE_LINUX_INTERRUPT_H */
#define raw_local_irq_disable() do { } while (0)
#define raw_local_irq_enable() do { } while (0)
#define raw_local_irq_save(flags) ((flags) = 0)
-#define raw_local_irq_restore(flags) do { } while (0)
+#define raw_local_irq_restore(flags) ((void)(flags))
#define raw_local_save_flags(flags) ((flags) = 0)
-#define raw_irqs_disabled_flags(flags) do { } while (0)
+#define raw_irqs_disabled_flags(flags) ((void)(flags))
#define raw_irqs_disabled() 0
#define raw_safe_halt()
#define local_irq_enable() do { } while (0)
#define local_irq_disable() do { } while (0)
#define local_irq_save(flags) ((flags) = 0)
-#define local_irq_restore(flags) do { } while (0)
+#define local_irq_restore(flags) ((void)(flags))
#define local_save_flags(flags) ((flags) = 0)
#define irqs_disabled() (1)
-#define irqs_disabled_flags(flags) (0)
+#define irqs_disabled_flags(flags) ((void)(flags), 0)
#define safe_halt() do { } while (0)
#define trace_lock_release(x, y)
--- /dev/null
+#ifndef _LINUX_JHASH_H
+#define _LINUX_JHASH_H
+
+/* jhash.h: Jenkins hash support.
+ *
+ * Copyright (C) 2006. Bob Jenkins (bob_jenkins@burtleburtle.net)
+ *
+ * http://burtleburtle.net/bob/hash/
+ *
+ * These are the credits from Bob's sources:
+ *
+ * lookup3.c, by Bob Jenkins, May 2006, Public Domain.
+ *
+ * These are functions for producing 32-bit hashes for hash table lookup.
+ * hashword(), hashlittle(), hashlittle2(), hashbig(), mix(), and final()
+ * are externally useful functions. Routines to test the hash are included
+ * if SELF_TEST is defined. You can use this free for any purpose. It's in
+ * the public domain. It has no warranty.
+ *
+ * Copyright (C) 2009-2010 Jozsef Kadlecsik (kadlec@blackhole.kfki.hu)
+ *
+ * I've modified Bob's hash to be useful in the Linux kernel, and
+ * any bugs present are my fault.
+ * Jozsef
+ */
+#include <linux/bitops.h>
+#include <linux/unaligned/packed_struct.h>
+
+/* Best hash sizes are of power of two */
+#define jhash_size(n) ((u32)1<<(n))
+/* Mask the hash value, i.e (value & jhash_mask(n)) instead of (value % n) */
+#define jhash_mask(n) (jhash_size(n)-1)
+
+/* __jhash_mix -- mix 3 32-bit values reversibly. */
+#define __jhash_mix(a, b, c) \
+{ \
+ a -= c; a ^= rol32(c, 4); c += b; \
+ b -= a; b ^= rol32(a, 6); a += c; \
+ c -= b; c ^= rol32(b, 8); b += a; \
+ a -= c; a ^= rol32(c, 16); c += b; \
+ b -= a; b ^= rol32(a, 19); a += c; \
+ c -= b; c ^= rol32(b, 4); b += a; \
+}
+
+/* __jhash_final - final mixing of 3 32-bit values (a,b,c) into c */
+#define __jhash_final(a, b, c) \
+{ \
+ c ^= b; c -= rol32(b, 14); \
+ a ^= c; a -= rol32(c, 11); \
+ b ^= a; b -= rol32(a, 25); \
+ c ^= b; c -= rol32(b, 16); \
+ a ^= c; a -= rol32(c, 4); \
+ b ^= a; b -= rol32(a, 14); \
+ c ^= b; c -= rol32(b, 24); \
+}
+
+/* An arbitrary initial parameter */
+#define JHASH_INITVAL 0xdeadbeef
+
+/* jhash - hash an arbitrary key
+ * @k: sequence of bytes as key
+ * @length: the length of the key
+ * @initval: the previous hash, or an arbitray value
+ *
+ * The generic version, hashes an arbitrary sequence of bytes.
+ * No alignment or length assumptions are made about the input key.
+ *
+ * Returns the hash value of the key. The result depends on endianness.
+ */
+static inline u32 jhash(const void *key, u32 length, u32 initval)
+{
+ u32 a, b, c;
+ const u8 *k = key;
+
+ /* Set up the internal state */
+ a = b = c = JHASH_INITVAL + length + initval;
+
+ /* All but the last block: affect some 32 bits of (a,b,c) */
+ while (length > 12) {
+ a += __get_unaligned_cpu32(k);
+ b += __get_unaligned_cpu32(k + 4);
+ c += __get_unaligned_cpu32(k + 8);
+ __jhash_mix(a, b, c);
+ length -= 12;
+ k += 12;
+ }
+ /* Last block: affect all 32 bits of (c) */
+ /* All the case statements fall through */
+ switch (length) {
+ case 12: c += (u32)k[11]<<24;
+ case 11: c += (u32)k[10]<<16;
+ case 10: c += (u32)k[9]<<8;
+ case 9: c += k[8];
+ case 8: b += (u32)k[7]<<24;
+ case 7: b += (u32)k[6]<<16;
+ case 6: b += (u32)k[5]<<8;
+ case 5: b += k[4];
+ case 4: a += (u32)k[3]<<24;
+ case 3: a += (u32)k[2]<<16;
+ case 2: a += (u32)k[1]<<8;
+ case 1: a += k[0];
+ __jhash_final(a, b, c);
+ case 0: /* Nothing left to add */
+ break;
+ }
+
+ return c;
+}
+
+/* jhash2 - hash an array of u32's
+ * @k: the key which must be an array of u32's
+ * @length: the number of u32's in the key
+ * @initval: the previous hash, or an arbitray value
+ *
+ * Returns the hash value of the key.
+ */
+static inline u32 jhash2(const u32 *k, u32 length, u32 initval)
+{
+ u32 a, b, c;
+
+ /* Set up the internal state */
+ a = b = c = JHASH_INITVAL + (length<<2) + initval;
+
+ /* Handle most of the key */
+ while (length > 3) {
+ a += k[0];
+ b += k[1];
+ c += k[2];
+ __jhash_mix(a, b, c);
+ length -= 3;
+ k += 3;
+ }
+
+ /* Handle the last 3 u32's: all the case statements fall through */
+ switch (length) {
+ case 3: c += k[2];
+ case 2: b += k[1];
+ case 1: a += k[0];
+ __jhash_final(a, b, c);
+ case 0: /* Nothing left to add */
+ break;
+ }
+
+ return c;
+}
+
+
+/* __jhash_nwords - hash exactly 3, 2 or 1 word(s) */
+static inline u32 __jhash_nwords(u32 a, u32 b, u32 c, u32 initval)
+{
+ a += initval;
+ b += initval;
+ c += initval;
+
+ __jhash_final(a, b, c);
+
+ return c;
+}
+
+static inline u32 jhash_3words(u32 a, u32 b, u32 c, u32 initval)
+{
+ return __jhash_nwords(a, b, c, initval + JHASH_INITVAL + (3 << 2));
+}
+
+static inline u32 jhash_2words(u32 a, u32 b, u32 initval)
+{
+ return __jhash_nwords(a, b, 0, initval + JHASH_INITVAL + (2 << 2));
+}
+
+static inline u32 jhash_1word(u32 a, u32 initval)
+{
+ return __jhash_nwords(a, 0, 0, initval + JHASH_INITVAL + (1 << 2));
+}
+
+#endif /* _LINUX_JHASH_H */
#include <linux/kernel.h>
#include <stdio.h>
+#include <unistd.h>
#define KSYM_NAME_LEN 128
name = backtrace_symbols((void **)&ip, 1);
- printf("%s\n", *name);
+ dprintf(STDOUT_FILENO, "%s\n", *name);
free(name);
}
#include <stddef.h>
#include <assert.h>
#include <linux/compiler.h>
+#include <endian.h>
+#include <byteswap.h>
#ifndef UINT_MAX
#define UINT_MAX (~0U)
(type *)((char *)__mptr - offsetof(type, member)); })
#endif
+#define BUILD_BUG_ON(condition) ((void)sizeof(char[1 - 2*!!(condition)]))
#define BUILD_BUG_ON_ZERO(e) (sizeof(struct { int:-!!(e); }))
#ifndef max
#endif
#endif
-/*
- * Both need more care to handle endianness
- * (Don't use bitmap_copy_le() for now)
- */
-#define cpu_to_le64(x) (x)
-#define cpu_to_le32(x) (x)
+#if __BYTE_ORDER == __BIG_ENDIAN
+#define cpu_to_le16 bswap_16
+#define cpu_to_le32 bswap_32
+#define cpu_to_le64 bswap_64
+#define le16_to_cpu bswap_16
+#define le32_to_cpu bswap_32
+#define le64_to_cpu bswap_64
+#define cpu_to_be16
+#define cpu_to_be32
+#define cpu_to_be64
+#define be16_to_cpu
+#define be32_to_cpu
+#define be64_to_cpu
+#else
+#define cpu_to_le16
+#define cpu_to_le32
+#define cpu_to_le64
+#define le16_to_cpu
+#define le32_to_cpu
+#define le64_to_cpu
+#define cpu_to_be16 bswap_16
+#define cpu_to_be32 bswap_32
+#define cpu_to_be64 bswap_64
+#define be16_to_cpu bswap_16
+#define be32_to_cpu bswap_32
+#define be64_to_cpu bswap_64
+#endif
int vscnprintf(char *buf, size_t size, const char *fmt, va_list args);
int scnprintf(char * buf, size_t size, const char * fmt, ...);
#define round_up(x, y) ((((x)-1) | __round_mask(x, y))+1)
#define round_down(x, y) ((x) & ~__round_mask(x, y))
+#define current_gfp_context(k) 0
+#define synchronize_sched()
+
#endif
--- /dev/null
+#ifndef _TOOLS_INCLUDE_LINUX_LINKAGE_H
+#define _TOOLS_INCLUDE_LINUX_LINKAGE_H
+
+#endif /* _TOOLS_INCLUDE_LINUX_LINKAGE_H */
#include <limits.h>
#include <linux/utsname.h>
#include <linux/compiler.h>
+#include <linux/export.h>
+#include <linux/kern_levels.h>
+#include <linux/err.h>
+#include <linux/rcu.h>
+#include <linux/list.h>
+#include <linux/hardirq.h>
+#include <unistd.h>
-#define MAX_LOCK_DEPTH 2000UL
+#define MAX_LOCK_DEPTH 63UL
#define asmlinkage
#define __visible
#define current (__curr())
-#define debug_locks_off() 1
+static inline int debug_locks_off(void)
+{
+ return 1;
+}
+
#define task_pid_nr(tsk) ((tsk)->pid)
#define KSYM_NAME_LEN 128
-#define printk printf
+#define printk(...) dprintf(STDOUT_FILENO, __VA_ARGS__)
+#define pr_err(format, ...) fprintf (stderr, format, ## __VA_ARGS__)
+#define pr_warn pr_err
#define list_del_rcu list_del
#define atomic_t unsigned long
#define atomic_inc(x) ((*(x))++)
-static struct new_utsname *init_utsname(void)
-{
- static struct new_utsname n = (struct new_utsname) {
- .release = "liblockdep",
- .version = LIBLOCKDEP_VERSION,
- };
-
- return &n;
-}
-
#define print_tainted() ""
#define static_obj(x) 1
#define debug_show_all_locks()
extern void debug_check_no_locks_held(void);
+static __used bool __is_kernel_percpu_address(unsigned long addr, void *can_addr)
+{
+ return false;
+}
+
#endif
#define module_param(name, type, perm)
+static inline bool __is_module_percpu_address(unsigned long addr, unsigned long *can_addr)
+{
+ return false;
+}
+
#endif
--- /dev/null
+#ifndef _TOOLS_INCLUDE_LINUX_MUTEX_H
+#define _TOOLS_INCLUDE_LINUX_MUTEX_H
+
+#endif /* _TOOLS_INCLUDE_LINUX_MUTEX_H */
--- /dev/null
+#ifndef _TOOLS_INCLUDE_LINUX_PROC_FS_H
+#define _TOOLS_INCLUDE_LINUX_PROC_FS_H
+
+#endif /* _TOOLS_INCLUDE_LINUX_PROC_FS_H */
return false;
}
+#define rcu_assign_pointer(p, v) ((p) = (v))
+#define RCU_INIT_POINTER(p, v) p=(v)
+
#endif
--- /dev/null
+#ifndef _TOOLS_PERF_LINUX_SCHED_CLOCK_H
+#define _TOOLS_PERF_LINUX_SCHED_CLOCK_H
+
+#endif /* _TOOLS_PERF_LINUX_SCHED_CLOCK_H */
--- /dev/null
+#ifndef _TOOLS_PERF_LINUX_SCHED_MM_H
+#define _TOOLS_PERF_LINUX_SCHED_MM_H
+
+#endif /* _TOOLS_PERF_LINUX_SCHED_MM_H */
--- /dev/null
+#ifndef _TOOLS_PERF_LINUX_SCHED_TASK_H
+#define _TOOLS_PERF_LINUX_SCHED_TASK_H
+
+#endif /* _TOOLS_PERF_LINUX_SCHED_TASK_H */
--- /dev/null
+#ifndef _TOOLS_INCLUDE_LINUX_SEQ_FILE_H
+#define _TOOLS_INCLUDE_LINUX_SEQ_FILE_H
+
+#endif /* _TOOLS_INCLUDE_LINUX_SEQ_FILE_H */
+#ifndef __LINUX_SPINLOCK_H_
+#define __LINUX_SPINLOCK_H_
+
+#include <pthread.h>
+#include <stdbool.h>
+
#define spinlock_t pthread_mutex_t
#define DEFINE_SPINLOCK(x) pthread_mutex_t x = PTHREAD_MUTEX_INITIALIZER;
#define spin_lock_irqsave(x, f) (void)f, pthread_mutex_lock(x)
#define spin_unlock_irqrestore(x, f) (void)f, pthread_mutex_unlock(x)
+
+#define arch_spinlock_t pthread_mutex_t
+#define __ARCH_SPIN_LOCK_UNLOCKED PTHREAD_MUTEX_INITIALIZER
+
+static inline void arch_spin_lock(arch_spinlock_t *mutex)
+{
+ pthread_mutex_lock(mutex);
+}
+
+static inline void arch_spin_unlock(arch_spinlock_t *mutex)
+{
+ pthread_mutex_unlock(mutex);
+}
+
+static inline bool arch_spin_is_locked(arch_spinlock_t *mutex)
+{
+ return true;
+}
+
+#endif
--- /dev/null
+#ifndef _LINUX_UNALIGNED_PACKED_STRUCT_H
+#define _LINUX_UNALIGNED_PACKED_STRUCT_H
+
+#include <linux/kernel.h>
+
+struct __una_u16 { u16 x; } __packed;
+struct __una_u32 { u32 x; } __packed;
+struct __una_u64 { u64 x; } __packed;
+
+static inline u16 __get_unaligned_cpu16(const void *p)
+{
+ const struct __una_u16 *ptr = (const struct __una_u16 *)p;
+ return ptr->x;
+}
+
+static inline u32 __get_unaligned_cpu32(const void *p)
+{
+ const struct __una_u32 *ptr = (const struct __una_u32 *)p;
+ return ptr->x;
+}
+
+static inline u64 __get_unaligned_cpu64(const void *p)
+{
+ const struct __una_u64 *ptr = (const struct __una_u64 *)p;
+ return ptr->x;
+}
+
+static inline void __put_unaligned_cpu16(u16 val, void *p)
+{
+ struct __una_u16 *ptr = (struct __una_u16 *)p;
+ ptr->x = val;
+}
+
+static inline void __put_unaligned_cpu32(u32 val, void *p)
+{
+ struct __una_u32 *ptr = (struct __una_u32 *)p;
+ ptr->x = val;
+}
+
+static inline void __put_unaligned_cpu64(u64 val, void *p)
+{
+ struct __una_u64 *ptr = (struct __una_u64 *)p;
+ ptr->x = val;
+}
+
+#endif /* _LINUX_UNALIGNED_PACKED_STRUCT_H */
--- /dev/null
+#ifndef _TOOLS_INCLUDE_TRACE_EVENTS_LOCK_H
+#define _TOOLS_INCLUDE_TRACE_EVENTS_LOCK_H
+
+#endif /* _TOOLS_INCLUDE_TRACE_EVENTS_LOCK_H */
return err;
}
+int filename__write_int(const char *filename, int value)
+{
+ int fd = open(filename, O_WRONLY), err = -1;
+ char buf[64];
+
+ if (fd < 0)
+ return err;
+
+ sprintf(buf, "%d", value);
+ if (write(fd, buf, sizeof(buf)) == sizeof(buf))
+ err = 0;
+
+ close(fd);
+ return err;
+}
+
int procfs__read_str(const char *entry, char **buf, size_t *sizep)
{
char path[PATH_MAX];
return filename__read_int(path, value);
}
+
+int sysfs__write_int(const char *entry, int value)
+{
+ char path[PATH_MAX];
+ const char *sysfs = sysfs__mountpoint();
+
+ if (!sysfs)
+ return -1;
+
+ if (snprintf(path, sizeof(path), "%s/%s", sysfs, entry) >= PATH_MAX)
+ return -1;
+
+ return filename__write_int(path, value);
+}
int filename__read_ull(const char *filename, unsigned long long *value);
int filename__read_str(const char *filename, char **buf, size_t *sizep);
+int filename__write_int(const char *filename, int value);
+
int procfs__read_str(const char *entry, char **buf, size_t *sizep);
int sysctl__read_int(const char *sysctl, int *value);
int sysfs__read_ull(const char *entry, unsigned long long *value);
int sysfs__read_str(const char *entry, char **buf, size_t *sizep);
int sysfs__read_bool(const char *entry, bool *value);
+
+int sysfs__write_int(const char *entry, int value);
#endif /* __API_FS__ */
# Set compile option CFLAGS if not set elsewhere
CFLAGS ?= -g -DCONFIG_LOCKDEP -DCONFIG_STACKTRACE -DCONFIG_PROVE_LOCKING -DBITS_PER_LONG=__WORDSIZE -DLIBLOCKDEP_VERSION='"$(LIBLOCKDEP_VERSION)"' -rdynamic -O0 -g
CFLAGS += -fPIC
+CFLAGS += -Wall
override CFLAGS += $(CONFIG_FLAGS) $(INCLUDES) $(PLUGIN_DIR_SQ)
do_compile_shared_library = \
($(print_shared_lib_compile) \
- $(CC) --shared $^ -o $@ -lpthread -ldl -Wl,-soname='"$@"';$(shell ln -sf $@ liblockdep.so))
+ $(CC) $(LDFLAGS) --shared $^ -o $@ -lpthread -ldl -Wl,-soname='$(@F)';$(shell ln -sf $(@F) $(@D)/liblockdep.so))
do_build_static_lib = \
($(print_static_lib_build) \
$(LIB_IN): force
$(Q)$(MAKE) $(build)=liblockdep
-liblockdep.so.$(LIBLOCKDEP_VERSION): $(LIB_IN)
+$(OUTPUT)liblockdep.so.$(LIBLOCKDEP_VERSION): $(LIB_IN)
$(Q)$(do_compile_shared_library)
-liblockdep.a: $(LIB_IN)
+$(OUTPUT)liblockdep.a: $(LIB_IN)
$(Q)$(do_build_static_lib)
tags: force
install: install_lib
clean:
- $(RM) *.o *~ $(TARGETS) *.a *liblockdep*.so* $(VERSION_FILES) .*.d .*.cmd
+ $(RM) $(OUTPUT)*.o *~ $(TARGETS) $(OUTPUT)*.a $(OUTPUT)*liblockdep*.so* $(VERSION_FILES) $(OUTPUT).*.d $(OUTPUT).*.cmd
$(RM) tags TAGS
PHONY += force
#include <linux/lockdep.h>
+#include <stdlib.h>
/* Trivial API wrappers, we don't (yet) have RCU in user-space: */
#define hlist_for_each_entry_rcu hlist_for_each_entry
#define hlist_add_head_rcu hlist_add_head
#define hlist_del_rcu hlist_del
+#define list_for_each_entry_rcu list_for_each_entry
+#define list_add_tail_rcu list_add_tail
+
+u32 prandom_u32(void)
+{
+ /* Used only by lock_pin_lock() which is dead code */
+ abort();
+}
+
+static struct new_utsname *init_utsname(void)
+{
+ static struct new_utsname n = (struct new_utsname) {
+ .release = "liblockdep",
+ .version = LIBLOCKDEP_VERSION,
+ };
+
+ return &n;
+}
#include "../../../kernel/locking/lockdep.c"
#include <dlfcn.h>
#include <stdlib.h>
#include <sysexits.h>
+#include <unistd.h>
#include "include/liblockdep/mutex.h"
#include "../../include/linux/rbtree.h"
#define LIBLOCKDEP_STATIC_ENTRIES 1024
#endif
-#define ARRAY_SIZE(arr) (sizeof(arr) / sizeof((arr)[0]))
-
static struct lock_lookup __locks[LIBLOCKDEP_STATIC_ENTRIES];
static int __locks_nr;
int idx = __locks_nr++;
if (idx >= ARRAY_SIZE(__locks)) {
- fprintf(stderr,
+ dprintf(STDERR_FILENO,
"LOCKDEP error: insufficient LIBLOCKDEP_STATIC_ENTRIES\n");
exit(EX_UNAVAILABLE);
}
-#include "../../../lib/rbtree.c"
+#include "../../lib/rbtree.c"
for i in `ls tests/*.c`; do
testname=$(basename "$i" .c)
- gcc -o tests/$testname -pthread -lpthread $i liblockdep.a -Iinclude -D__USE_LIBLOCKDEP &> /dev/null
+ gcc -o tests/$testname -pthread $i liblockdep.a -Iinclude -D__USE_LIBLOCKDEP &> /dev/null
echo -ne "$testname... "
- if [ $(timeout 1 ./tests/$testname | wc -l) -gt 0 ]; then
+ if [ $(timeout 1 ./tests/$testname 2>&1 | wc -l) -gt 0 ]; then
echo "PASSED!"
else
echo "FAILED!"
for i in `ls tests/*.c`; do
testname=$(basename "$i" .c)
- gcc -o tests/$testname -pthread -lpthread -Iinclude $i &> /dev/null
+ gcc -o tests/$testname -pthread -Iinclude $i &> /dev/null
echo -ne "(PRELOAD) $testname... "
- if [ $(timeout 1 ./lockdep ./tests/$testname | wc -l) -gt 0 ]; then
+ if [ $(timeout 1 ./lockdep ./tests/$testname 2>&1 | wc -l) -gt 0 ]; then
echo "PASSED!"
else
echo "FAILED!"
+++ /dev/null
-#ifndef __ASM_GENERIC_HASH_H
-#define __ASM_GENERIC_HASH_H
-
-/* Stub */
-
-#endif /* __ASM_GENERIC_HASH_H */
+++ /dev/null
-
-/* empty file */
-
+++ /dev/null
-
-/* empty file */
-
+++ /dev/null
-
-/* empty file */
-
+++ /dev/null
-#ifndef _LIBLOCKDEP_LINUX_COMPILER_H_
-#define _LIBLOCKDEP_LINUX_COMPILER_H_
-
-#define __used __attribute__((__unused__))
-#define unlikely
-#define READ_ONCE(x) (x)
-#define WRITE_ONCE(x, val) x=(val)
-#define RCU_INIT_POINTER(p, v) p=(v)
-
-#endif
+++ /dev/null
-
-/* empty file */
-
+++ /dev/null
-
-/* empty file */
-
+++ /dev/null
-
-/* empty file */
-
+++ /dev/null
-#include "../../../include/linux/hash.h"
+++ /dev/null
-
-/* empty file */
-
+++ /dev/null
-#ifndef _LIBLOCKDEP_LINUX_KERNEL_H_
-#define _LIBLOCKDEP_LINUX_KERNEL_H_
-
-#include <linux/export.h>
-#include <linux/types.h>
-#include <linux/rcu.h>
-#include <linux/hardirq.h>
-#include <linux/kern_levels.h>
-
-#ifndef container_of
-#define container_of(ptr, type, member) ({ \
- const typeof(((type *)0)->member) * __mptr = (ptr); \
- (type *)((char *)__mptr - offsetof(type, member)); })
-#endif
-
-#define max(x, y) ({ \
- typeof(x) _max1 = (x); \
- typeof(y) _max2 = (y); \
- (void) (&_max1 == &_max2); \
- _max1 > _max2 ? _max1 : _max2; })
-
-#define BUILD_BUG_ON(condition) ((void)sizeof(char[1 - 2*!!(condition)]))
-#define WARN_ON(x) (x)
-#define WARN_ON_ONCE(x) (x)
-#define likely(x) (x)
-#define WARN(x, y...) (x)
-#define uninitialized_var(x) x
-#define __init
-#define noinline
-#define list_add_tail_rcu list_add_tail
-#define list_for_each_entry_rcu list_for_each_entry
-#define barrier()
-#define synchronize_sched()
-
-#ifndef CALLER_ADDR0
-#define CALLER_ADDR0 ((unsigned long)__builtin_return_address(0))
-#endif
-
-#ifndef _RET_IP_
-#define _RET_IP_ CALLER_ADDR0
-#endif
-
-#ifndef _THIS_IP_
-#define _THIS_IP_ ({ __label__ __here; __here: (unsigned long)&&__here; })
-#endif
-
-#endif
+++ /dev/null
-
-/* empty file */
-
+++ /dev/null
-#include "../../../include/linux/list.h"
+++ /dev/null
-
-/* empty file */
-
+++ /dev/null
-#include "../../../include/linux/poison.h"
+++ /dev/null
-#ifndef _LIBLOCKDEP_LINUX_PREFETCH_H_
-#define _LIBLOCKDEP_LINUX_PREFETCH_H
-
-static inline void prefetch(void *a __attribute__((unused))) { }
-
-#endif
+++ /dev/null
-
-/* empty file */
-
+++ /dev/null
-#define __always_inline
-#include "../../../include/linux/rbtree_augmented.h"
+++ /dev/null
-
-/* empty file */
-
+++ /dev/null
-#ifndef _LIBLOCKDEP_SPINLOCK_H_
-#define _LIBLOCKDEP_SPINLOCK_H_
-
-#include <pthread.h>
-#include <stdbool.h>
-
-#define arch_spinlock_t pthread_mutex_t
-#define __ARCH_SPIN_LOCK_UNLOCKED PTHREAD_MUTEX_INITIALIZER
-
-static inline void arch_spin_lock(arch_spinlock_t *mutex)
-{
- pthread_mutex_lock(mutex);
-}
-
-static inline void arch_spin_unlock(arch_spinlock_t *mutex)
-{
- pthread_mutex_unlock(mutex);
-}
-
-static inline bool arch_spin_is_locked(arch_spinlock_t *mutex)
-{
- return true;
-}
-
-#endif
+++ /dev/null
-#ifndef _LIBLOCKDEP_LINUX_STRINGIFY_H_
-#define _LIBLOCKDEP_LINUX_STRINGIFY_H_
-
-#define __stringify_1(x...) #x
-#define __stringify(x...) __stringify_1(x)
-
-#endif
+++ /dev/null
-
-/* empty file */
-
objtool-y += arch/$(SRCARCH)/
objtool-y += builtin-check.o
+objtool-y += check.o
objtool-y += elf.o
objtool-y += special.o
objtool-y += objtool.o
c) Higher live patching compatibility rate
- (NOTE: This is not yet implemented)
-
- Currently with CONFIG_LIVEPATCH there's a basic live patching
- framework which is safe for roughly 85-90% of "security" fixes. But
- patches can't have complex features like function dependency or
- prototype changes, or data structure changes.
-
- There's a strong need to support patches which have the more complex
- features so that the patch compatibility rate for security fixes can
- eventually approach something resembling 100%. To achieve that, a
- "consistency model" is needed, which allows tasks to be safely
- transitioned from an unpatched state to a patched state.
-
- One of the key requirements of the currently proposed livepatch
- consistency model [*] is that it needs to walk the stack of each
- sleeping task to determine if it can be transitioned to the patched
- state. If objtool can ensure that stack traces are reliable, this
- consistency model can be used and the live patching compatibility
- rate can be improved significantly.
-
- [*] https://lkml.kernel.org/r/cover.1423499826.git.jpoimboe@redhat.com
+ Livepatch has an optional "consistency model", which is needed for
+ more complex patches. In order for the consistency model to work,
+ stack traces need to be reliable (or an unreliable condition needs to
+ be detectable). Objtool makes that possible.
+ For more details, see the livepatch documentation in the Linux kernel
+ source tree at Documentation/livepatch/livepatch.txt.
Rules
-----
return normally.
-Errors in .S files
-------------------
+Objtool warnings
+----------------
-If you're getting an error in a compiled .S file which you don't
-understand, first make sure that the affected code follows the above
-rules.
+For asm files, if you're getting an error which doesn't make sense,
+first make sure that the affected code follows the above rules.
+
+For C files, the common culprits are inline asm statements and calls to
+"noreturn" functions. See below for more details.
+
+Another possible cause for errors in C code is if the Makefile removes
+-fno-omit-frame-pointer or adds -fomit-frame-pointer to the gcc options.
Here are some examples of common warnings reported by objtool, what
they mean, and suggestions for how to fix them.
-1. asm_file.o: warning: objtool: func()+0x128: call without frame pointer save/setup
+1. file.o: warning: objtool: func()+0x128: call without frame pointer save/setup
The func() function made a function call without first saving and/or
- updating the frame pointer.
-
- If func() is indeed a callable function, add proper frame pointer
- logic using the FRAME_BEGIN and FRAME_END macros. Otherwise, remove
- its ELF function annotation by changing ENDPROC to END.
+ updating the frame pointer, and CONFIG_FRAME_POINTER is enabled.
- If you're getting this error in a .c file, see the "Errors in .c
- files" section.
+ If the error is for an asm file, and func() is indeed a callable
+ function, add proper frame pointer logic using the FRAME_BEGIN and
+ FRAME_END macros. Otherwise, if it's not a callable function, remove
+ its ELF function annotation by changing ENDPROC to END, and instead
+ use the manual CFI hint macros in asm/undwarf.h.
+ If it's a GCC-compiled .c file, the error may be because the function
+ uses an inline asm() statement which has a "call" instruction. An
+ asm() statement with a call instruction must declare the use of the
+ stack pointer in its output operand. For example, on x86_64:
-2. asm_file.o: warning: objtool: .text+0x53: return instruction outside of a callable function
-
- A return instruction was detected, but objtool couldn't find a way
- for a callable function to reach the instruction.
+ register void *__sp asm("rsp");
+ asm volatile("call func" : "+r" (__sp));
- If the return instruction is inside (or reachable from) a callable
- function, the function needs to be annotated with the ENTRY/ENDPROC
- macros.
+ Otherwise the stack frame may not get created before the call.
- If you _really_ need a return instruction outside of a function, and
- are 100% sure that it won't affect stack traces, you can tell
- objtool to ignore it. See the "Adding exceptions" section below.
+2. file.o: warning: objtool: .text+0x53: unreachable instruction
-3. asm_file.o: warning: objtool: func()+0x9: function has unreachable instruction
+ Objtool couldn't find a code path to reach the instruction.
- The instruction lives inside of a callable function, but there's no
- possible control flow path from the beginning of the function to the
- instruction.
+ If the error is for an asm file, and the instruction is inside (or
+ reachable from) a callable function, the function should be annotated
+ with the ENTRY/ENDPROC macros (ENDPROC is the important one).
+ Otherwise, the code should probably be annotated with the CFI hint
+ macros in asm/undwarf.h so objtool and the unwinder can know the
+ stack state associated with the code.
- If the instruction is actually needed, and it's actually in a
- callable function, ensure that its function is properly annotated
- with ENTRY/ENDPROC.
+ If you're 100% sure the code won't affect stack traces, or if you're
+ a just a bad person, you can tell objtool to ignore it. See the
+ "Adding exceptions" section below.
If it's not actually in a callable function (e.g. kernel entry code),
change ENDPROC to END.
-4. asm_file.o: warning: objtool: func(): can't find starting instruction
+4. file.o: warning: objtool: func(): can't find starting instruction
or
- asm_file.o: warning: objtool: func()+0x11dd: can't decode instruction
+ file.o: warning: objtool: func()+0x11dd: can't decode instruction
- Did you put data in a text section? If so, that can confuse
+ Does the file have data in a text section? If so, that can confuse
objtool's instruction decoder. Move the data to a more appropriate
section like .data or .rodata.
-5. asm_file.o: warning: objtool: func()+0x6: kernel entry/exit from callable instruction
-
- This is a kernel entry/exit instruction like sysenter or sysret.
- Such instructions aren't allowed in a callable function, and are most
- likely part of the kernel entry code.
+5. file.o: warning: objtool: func()+0x6: unsupported instruction in callable function
- If the instruction isn't actually in a callable function, change
- ENDPROC to END.
+ This is a kernel entry/exit instruction like sysenter or iret. Such
+ instructions aren't allowed in a callable function, and are most
+ likely part of the kernel entry code. They should usually not have
+ the callable function annotation (ENDPROC) and should always be
+ annotated with the CFI hint macros in asm/undwarf.h.
-6. asm_file.o: warning: objtool: func()+0x26: sibling call from callable instruction with changed frame pointer
+6. file.o: warning: objtool: func()+0x26: sibling call from callable instruction with modified stack frame
- This is a dynamic jump or a jump to an undefined symbol. Stacktool
+ This is a dynamic jump or a jump to an undefined symbol. Objtool
assumed it's a sibling call and detected that the frame pointer
wasn't first restored to its original state.
destination code to the local file.
If the instruction is not actually in a callable function (e.g.
- kernel entry code), change ENDPROC to END.
+ kernel entry code), change ENDPROC to END and annotate manually with
+ the CFI hint macros in asm/undwarf.h.
-7. asm_file: warning: objtool: func()+0x5c: frame pointer state mismatch
+7. file: warning: objtool: func()+0x5c: stack state mismatch
The instruction's frame pointer state is inconsistent, depending on
which execution path was taken to reach the instruction.
- Make sure the function pushes and sets up the frame pointer (for
- x86_64, this means rbp) at the beginning of the function and pops it
- at the end of the function. Also make sure that no other code in the
- function touches the frame pointer.
+ Make sure that, when CONFIG_FRAME_POINTER is enabled, the function
+ pushes and sets up the frame pointer (for x86_64, this means rbp) at
+ the beginning of the function and pops it at the end of the function.
+ Also make sure that no other code in the function touches the frame
+ pointer.
+ Another possibility is that the code has some asm or inline asm which
+ does some unusual things to the stack or the frame pointer. In such
+ cases it's probably appropriate to use the CFI hint macros in
+ asm/undwarf.h.
-Errors in .c files
-------------------
-1. c_file.o: warning: objtool: funcA() falls through to next function funcB()
+8. file.o: warning: objtool: funcA() falls through to next function funcB()
This means that funcA() doesn't end with a return instruction or an
unconditional jump, and that objtool has determined that the function
might be corrupt due to a gcc bug. For more details, see:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70646
-2. If you're getting any other objtool error in a compiled .c file, it
- may be because the file uses an asm() statement which has a "call"
- instruction. An asm() statement with a call instruction must declare
- the use of the stack pointer in its output operand. For example, on
- x86_64:
-
- register void *__sp asm("rsp");
- asm volatile("call func" : "+r" (__sp));
-
- Otherwise the stack frame may not get created before the call.
-
-3. Another possible cause for errors in C code is if the Makefile removes
- -fno-omit-frame-pointer or adds -fomit-frame-pointer to the gcc options.
-
-Also see the above section for .S file errors for more information what
-the individual error messages mean.
If the error doesn't seem to make sense, it could be a bug in objtool.
Feel free to ask the objtool maintainer for help.
all: $(OBJTOOL)
INCLUDES := -I$(srctree)/tools/include -I$(srctree)/tools/arch/$(HOSTARCH)/include/uapi
-CFLAGS += -Wall -Werror $(EXTRA_WARNINGS) -fomit-frame-pointer -O2 -g $(INCLUDES)
+CFLAGS += -Wall -Werror $(EXTRA_WARNINGS) -Wno-switch-default -Wno-switch-enum -fomit-frame-pointer -O2 -g $(INCLUDES)
LDFLAGS += -lelf $(LIBSUBCMD)
# Allow old libelf to be used:
#define _ARCH_H
#include <stdbool.h>
+#include <linux/list.h>
#include "elf.h"
+#include "cfi.h"
-#define INSN_FP_SAVE 1
-#define INSN_FP_SETUP 2
-#define INSN_FP_RESTORE 3
-#define INSN_JUMP_CONDITIONAL 4
-#define INSN_JUMP_UNCONDITIONAL 5
-#define INSN_JUMP_DYNAMIC 6
-#define INSN_CALL 7
-#define INSN_CALL_DYNAMIC 8
-#define INSN_RETURN 9
-#define INSN_CONTEXT_SWITCH 10
-#define INSN_NOP 11
-#define INSN_OTHER 12
+#define INSN_JUMP_CONDITIONAL 1
+#define INSN_JUMP_UNCONDITIONAL 2
+#define INSN_JUMP_DYNAMIC 3
+#define INSN_CALL 4
+#define INSN_CALL_DYNAMIC 5
+#define INSN_RETURN 6
+#define INSN_CONTEXT_SWITCH 7
+#define INSN_STACK 8
+#define INSN_NOP 9
+#define INSN_OTHER 10
#define INSN_LAST INSN_OTHER
+enum op_dest_type {
+ OP_DEST_REG,
+ OP_DEST_REG_INDIRECT,
+ OP_DEST_MEM,
+ OP_DEST_PUSH,
+ OP_DEST_LEAVE,
+};
+
+struct op_dest {
+ enum op_dest_type type;
+ unsigned char reg;
+ int offset;
+};
+
+enum op_src_type {
+ OP_SRC_REG,
+ OP_SRC_REG_INDIRECT,
+ OP_SRC_CONST,
+ OP_SRC_POP,
+ OP_SRC_ADD,
+ OP_SRC_AND,
+};
+
+struct op_src {
+ enum op_src_type type;
+ unsigned char reg;
+ int offset;
+};
+
+struct stack_op {
+ struct op_dest dest;
+ struct op_src src;
+};
+
+void arch_initial_func_cfi_state(struct cfi_state *state);
+
int arch_decode_instruction(struct elf *elf, struct section *sec,
unsigned long offset, unsigned int maxlen,
unsigned int *len, unsigned char *type,
- unsigned long *displacement);
+ unsigned long *immediate, struct stack_op *op);
+
+bool arch_callee_saved_reg(unsigned char reg);
#endif /* _ARCH_H */
#include "../../arch.h"
#include "../../warn.h"
+static unsigned char op_to_cfi_reg[][2] = {
+ {CFI_AX, CFI_R8},
+ {CFI_CX, CFI_R9},
+ {CFI_DX, CFI_R10},
+ {CFI_BX, CFI_R11},
+ {CFI_SP, CFI_R12},
+ {CFI_BP, CFI_R13},
+ {CFI_SI, CFI_R14},
+ {CFI_DI, CFI_R15},
+};
+
static int is_x86_64(struct elf *elf)
{
switch (elf->ehdr.e_machine) {
}
}
+bool arch_callee_saved_reg(unsigned char reg)
+{
+ switch (reg) {
+ case CFI_BP:
+ case CFI_BX:
+ case CFI_R12:
+ case CFI_R13:
+ case CFI_R14:
+ case CFI_R15:
+ return true;
+
+ case CFI_AX:
+ case CFI_CX:
+ case CFI_DX:
+ case CFI_SI:
+ case CFI_DI:
+ case CFI_SP:
+ case CFI_R8:
+ case CFI_R9:
+ case CFI_R10:
+ case CFI_R11:
+ case CFI_RA:
+ default:
+ return false;
+ }
+}
+
int arch_decode_instruction(struct elf *elf, struct section *sec,
unsigned long offset, unsigned int maxlen,
unsigned int *len, unsigned char *type,
- unsigned long *immediate)
+ unsigned long *immediate, struct stack_op *op)
{
struct insn insn;
- int x86_64;
- unsigned char op1, op2, ext;
+ int x86_64, sign;
+ unsigned char op1, op2, rex = 0, rex_b = 0, rex_r = 0, rex_w = 0,
+ modrm = 0, modrm_mod = 0, modrm_rm = 0, modrm_reg = 0,
+ sib = 0;
x86_64 = is_x86_64(elf);
if (x86_64 == -1)
return -1;
- insn_init(&insn, (void *)(sec->data + offset), maxlen, x86_64);
+ insn_init(&insn, sec->data->d_buf + offset, maxlen, x86_64);
insn_get_length(&insn);
- insn_get_opcode(&insn);
- insn_get_modrm(&insn);
- insn_get_immediate(&insn);
if (!insn_complete(&insn)) {
WARN_FUNC("can't decode instruction", sec, offset);
op1 = insn.opcode.bytes[0];
op2 = insn.opcode.bytes[1];
+ if (insn.rex_prefix.nbytes) {
+ rex = insn.rex_prefix.bytes[0];
+ rex_w = X86_REX_W(rex) >> 3;
+ rex_r = X86_REX_R(rex) >> 2;
+ rex_b = X86_REX_B(rex);
+ }
+
+ if (insn.modrm.nbytes) {
+ modrm = insn.modrm.bytes[0];
+ modrm_mod = X86_MODRM_MOD(modrm);
+ modrm_reg = X86_MODRM_REG(modrm);
+ modrm_rm = X86_MODRM_RM(modrm);
+ }
+
+ if (insn.sib.nbytes)
+ sib = insn.sib.bytes[0];
+
switch (op1) {
- case 0x55:
- if (!insn.rex_prefix.nbytes)
- /* push rbp */
- *type = INSN_FP_SAVE;
+
+ case 0x1:
+ case 0x29:
+ if (rex_w && !rex_b && modrm_mod == 3 && modrm_rm == 4) {
+
+ /* add/sub reg, %rsp */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_ADD;
+ op->src.reg = op_to_cfi_reg[modrm_reg][rex_r];
+ op->dest.type = OP_SRC_REG;
+ op->dest.reg = CFI_SP;
+ }
+ break;
+
+ case 0x50 ... 0x57:
+
+ /* push reg */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_REG;
+ op->src.reg = op_to_cfi_reg[op1 & 0x7][rex_b];
+ op->dest.type = OP_DEST_PUSH;
+
break;
- case 0x5d:
- if (!insn.rex_prefix.nbytes)
- /* pop rbp */
- *type = INSN_FP_RESTORE;
+ case 0x58 ... 0x5f:
+
+ /* pop reg */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_POP;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = op_to_cfi_reg[op1 & 0x7][rex_b];
+
+ break;
+
+ case 0x68:
+ case 0x6a:
+ /* push immediate */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_CONST;
+ op->dest.type = OP_DEST_PUSH;
break;
case 0x70 ... 0x7f:
*type = INSN_JUMP_CONDITIONAL;
break;
+ case 0x81:
+ case 0x83:
+ if (rex != 0x48)
+ break;
+
+ if (modrm == 0xe4) {
+ /* and imm, %rsp */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_AND;
+ op->src.reg = CFI_SP;
+ op->src.offset = insn.immediate.value;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = CFI_SP;
+ break;
+ }
+
+ if (modrm == 0xc4)
+ sign = 1;
+ else if (modrm == 0xec)
+ sign = -1;
+ else
+ break;
+
+ /* add/sub imm, %rsp */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_ADD;
+ op->src.reg = CFI_SP;
+ op->src.offset = insn.immediate.value * sign;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = CFI_SP;
+ break;
+
case 0x89:
- if (insn.rex_prefix.nbytes == 1 &&
- insn.rex_prefix.bytes[0] == 0x48 &&
- insn.modrm.nbytes && insn.modrm.bytes[0] == 0xe5)
- /* mov rsp, rbp */
- *type = INSN_FP_SETUP;
+ if (rex == 0x48 && modrm == 0xe5) {
+
+ /* mov %rsp, %rbp */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_REG;
+ op->src.reg = CFI_SP;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = CFI_BP;
+ break;
+ }
+ /* fallthrough */
+ case 0x88:
+ if (!rex_b &&
+ (modrm_mod == 1 || modrm_mod == 2) && modrm_rm == 5) {
+
+ /* mov reg, disp(%rbp) */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_REG;
+ op->src.reg = op_to_cfi_reg[modrm_reg][rex_r];
+ op->dest.type = OP_DEST_REG_INDIRECT;
+ op->dest.reg = CFI_BP;
+ op->dest.offset = insn.displacement.value;
+
+ } else if (rex_w && !rex_b && modrm_rm == 4 && sib == 0x24) {
+
+ /* mov reg, disp(%rsp) */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_REG;
+ op->src.reg = op_to_cfi_reg[modrm_reg][rex_r];
+ op->dest.type = OP_DEST_REG_INDIRECT;
+ op->dest.reg = CFI_SP;
+ op->dest.offset = insn.displacement.value;
+ }
+
+ break;
+
+ case 0x8b:
+ if (rex_w && !rex_b && modrm_mod == 1 && modrm_rm == 5) {
+
+ /* mov disp(%rbp), reg */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_REG_INDIRECT;
+ op->src.reg = CFI_BP;
+ op->src.offset = insn.displacement.value;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = op_to_cfi_reg[modrm_reg][rex_r];
+
+ } else if (rex_w && !rex_b && sib == 0x24 &&
+ modrm_mod != 3 && modrm_rm == 4) {
+
+ /* mov disp(%rsp), reg */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_REG_INDIRECT;
+ op->src.reg = CFI_SP;
+ op->src.offset = insn.displacement.value;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = op_to_cfi_reg[modrm_reg][rex_r];
+ }
+
break;
case 0x8d:
- if (insn.rex_prefix.nbytes &&
- insn.rex_prefix.bytes[0] == 0x48 &&
- insn.modrm.nbytes && insn.modrm.bytes[0] == 0x2c &&
- insn.sib.nbytes && insn.sib.bytes[0] == 0x24)
- /* lea %(rsp), %rbp */
- *type = INSN_FP_SETUP;
+ if (rex == 0x48 && modrm == 0x65) {
+
+ /* lea -disp(%rbp), %rsp */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_ADD;
+ op->src.reg = CFI_BP;
+ op->src.offset = insn.displacement.value;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = CFI_SP;
+ break;
+ }
+
+ if (rex == 0x4c && modrm == 0x54 && sib == 0x24 &&
+ insn.displacement.value == 8) {
+
+ /*
+ * lea 0x8(%rsp), %r10
+ *
+ * Here r10 is the "drap" pointer, used as a stack
+ * pointer helper when the stack gets realigned.
+ */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_ADD;
+ op->src.reg = CFI_SP;
+ op->src.offset = 8;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = CFI_R10;
+ break;
+ }
+
+ if (rex == 0x4c && modrm == 0x6c && sib == 0x24 &&
+ insn.displacement.value == 16) {
+
+ /*
+ * lea 0x10(%rsp), %r13
+ *
+ * Here r13 is the "drap" pointer, used as a stack
+ * pointer helper when the stack gets realigned.
+ */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_ADD;
+ op->src.reg = CFI_SP;
+ op->src.offset = 16;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = CFI_R13;
+ break;
+ }
+
+ if (rex == 0x49 && modrm == 0x62 &&
+ insn.displacement.value == -8) {
+
+ /*
+ * lea -0x8(%r10), %rsp
+ *
+ * Restoring rsp back to its original value after a
+ * stack realignment.
+ */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_ADD;
+ op->src.reg = CFI_R10;
+ op->src.offset = -8;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = CFI_SP;
+ break;
+ }
+
+ if (rex == 0x49 && modrm == 0x65 &&
+ insn.displacement.value == -16) {
+
+ /*
+ * lea -0x10(%r13), %rsp
+ *
+ * Restoring rsp back to its original value after a
+ * stack realignment.
+ */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_ADD;
+ op->src.reg = CFI_R13;
+ op->src.offset = -16;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = CFI_SP;
+ break;
+ }
+
+ break;
+
+ case 0x8f:
+ /* pop to mem */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_POP;
+ op->dest.type = OP_DEST_MEM;
break;
case 0x90:
*type = INSN_NOP;
break;
+ case 0x9c:
+ /* pushf */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_CONST;
+ op->dest.type = OP_DEST_PUSH;
+ break;
+
+ case 0x9d:
+ /* popf */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_POP;
+ op->dest.type = OP_DEST_MEM;
+ break;
+
case 0x0f:
+
if (op2 >= 0x80 && op2 <= 0x8f)
*type = INSN_JUMP_CONDITIONAL;
else if (op2 == 0x05 || op2 == 0x07 || op2 == 0x34 ||
op2 == 0x35)
+
/* sysenter, sysret */
*type = INSN_CONTEXT_SWITCH;
+
else if (op2 == 0x0d || op2 == 0x1f)
+
/* nopl/nopw */
*type = INSN_NOP;
- else if (op2 == 0x01 && insn.modrm.nbytes &&
- (insn.modrm.bytes[0] == 0xc2 ||
- insn.modrm.bytes[0] == 0xd8))
- /* vmlaunch, vmrun */
- *type = INSN_CONTEXT_SWITCH;
+
+ else if (op2 == 0xa0 || op2 == 0xa8) {
+
+ /* push fs/gs */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_CONST;
+ op->dest.type = OP_DEST_PUSH;
+
+ } else if (op2 == 0xa1 || op2 == 0xa9) {
+
+ /* pop fs/gs */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_POP;
+ op->dest.type = OP_DEST_MEM;
+ }
break;
- case 0xc9: /* leave */
- *type = INSN_FP_RESTORE;
+ case 0xc9:
+ /*
+ * leave
+ *
+ * equivalent to:
+ * mov bp, sp
+ * pop bp
+ */
+ *type = INSN_STACK;
+ op->dest.type = OP_DEST_LEAVE;
+
break;
- case 0xe3: /* jecxz/jrcxz */
+ case 0xe3:
+ /* jecxz/jrcxz */
*type = INSN_JUMP_CONDITIONAL;
break;
break;
case 0xff:
- ext = X86_MODRM_REG(insn.modrm.bytes[0]);
- if (ext == 2 || ext == 3)
+ if (modrm_reg == 2 || modrm_reg == 3)
+
*type = INSN_CALL_DYNAMIC;
- else if (ext == 4)
+
+ else if (modrm_reg == 4)
+
*type = INSN_JUMP_DYNAMIC;
- else if (ext == 5) /*jmpf */
+
+ else if (modrm_reg == 5)
+
+ /* jmpf */
*type = INSN_CONTEXT_SWITCH;
+ else if (modrm_reg == 6) {
+
+ /* push from mem */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_CONST;
+ op->dest.type = OP_DEST_PUSH;
+ }
+
break;
default:
return 0;
}
+
+void arch_initial_func_cfi_state(struct cfi_state *state)
+{
+ int i;
+
+ for (i = 0; i < CFI_NUM_REGS; i++) {
+ state->regs[i].base = CFI_UNDEFINED;
+ state->regs[i].offset = 0;
+ }
+
+ /* initial CFA (call frame address) */
+ state->cfa.base = CFI_SP;
+ state->cfa.offset = 8;
+
+ /* initial RA (return address) */
+ state->regs[16].base = CFI_CFA;
+ state->regs[16].offset = -8;
+}
1: fxstor | RDGSBASE Ry (F3),(11B)
2: vldmxcsr Md (v1) | WRFSBASE Ry (F3),(11B)
3: vstmxcsr Md (v1) | WRGSBASE Ry (F3),(11B)
-4: XSAVE
+4: XSAVE | ptwrite Ey (F3),(11B)
5: XRSTOR | lfence (11B)
6: XSAVEOPT | clwb (66) | mfence (11B)
7: clflush | clflushopt (66) | sfence (11B)
/*
- * Copyright (C) 2015 Josh Poimboeuf <jpoimboe@redhat.com>
+ * Copyright (C) 2015-2017 Josh Poimboeuf <jpoimboe@redhat.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* For more information, see tools/objtool/Documentation/stack-validation.txt.
*/
-#include <string.h>
-#include <stdlib.h>
#include <subcmd/parse-options.h>
-
#include "builtin.h"
-#include "elf.h"
-#include "special.h"
-#include "arch.h"
-#include "warn.h"
-
-#include <linux/hashtable.h>
-#include <linux/kernel.h>
-
-#define STATE_FP_SAVED 0x1
-#define STATE_FP_SETUP 0x2
-#define STATE_FENTRY 0x4
-
-struct instruction {
- struct list_head list;
- struct hlist_node hash;
- struct section *sec;
- unsigned long offset;
- unsigned int len, state;
- unsigned char type;
- unsigned long immediate;
- bool alt_group, visited, dead_end;
- struct symbol *call_dest;
- struct instruction *jump_dest;
- struct list_head alts;
- struct symbol *func;
-};
-
-struct alternative {
- struct list_head list;
- struct instruction *insn;
-};
-
-struct objtool_file {
- struct elf *elf;
- struct list_head insn_list;
- DECLARE_HASHTABLE(insn_hash, 16);
- struct section *rodata, *whitelist;
- bool ignore_unreachables, c_file;
-};
-
-const char *objname;
-static bool nofp;
-
-static struct instruction *find_insn(struct objtool_file *file,
- struct section *sec, unsigned long offset)
-{
- struct instruction *insn;
-
- hash_for_each_possible(file->insn_hash, insn, hash, offset)
- if (insn->sec == sec && insn->offset == offset)
- return insn;
-
- return NULL;
-}
-
-static struct instruction *next_insn_same_sec(struct objtool_file *file,
- struct instruction *insn)
-{
- struct instruction *next = list_next_entry(insn, list);
-
- if (&next->list == &file->insn_list || next->sec != insn->sec)
- return NULL;
-
- return next;
-}
-
-static bool gcov_enabled(struct objtool_file *file)
-{
- struct section *sec;
- struct symbol *sym;
-
- list_for_each_entry(sec, &file->elf->sections, list)
- list_for_each_entry(sym, &sec->symbol_list, list)
- if (!strncmp(sym->name, "__gcov_.", 8))
- return true;
-
- return false;
-}
-
-#define for_each_insn(file, insn) \
- list_for_each_entry(insn, &file->insn_list, list)
-
-#define func_for_each_insn(file, func, insn) \
- for (insn = find_insn(file, func->sec, func->offset); \
- insn && &insn->list != &file->insn_list && \
- insn->sec == func->sec && \
- insn->offset < func->offset + func->len; \
- insn = list_next_entry(insn, list))
-
-#define func_for_each_insn_continue_reverse(file, func, insn) \
- for (insn = list_prev_entry(insn, list); \
- &insn->list != &file->insn_list && \
- insn->sec == func->sec && insn->offset >= func->offset; \
- insn = list_prev_entry(insn, list))
-
-#define sec_for_each_insn_from(file, insn) \
- for (; insn; insn = next_insn_same_sec(file, insn))
-
-
-/*
- * Check if the function has been manually whitelisted with the
- * STACK_FRAME_NON_STANDARD macro, or if it should be automatically whitelisted
- * due to its use of a context switching instruction.
- */
-static bool ignore_func(struct objtool_file *file, struct symbol *func)
-{
- struct rela *rela;
- struct instruction *insn;
-
- /* check for STACK_FRAME_NON_STANDARD */
- if (file->whitelist && file->whitelist->rela)
- list_for_each_entry(rela, &file->whitelist->rela->rela_list, list) {
- if (rela->sym->type == STT_SECTION &&
- rela->sym->sec == func->sec &&
- rela->addend == func->offset)
- return true;
- if (rela->sym->type == STT_FUNC && rela->sym == func)
- return true;
- }
-
- /* check if it has a context switching instruction */
- func_for_each_insn(file, func, insn)
- if (insn->type == INSN_CONTEXT_SWITCH)
- return true;
-
- return false;
-}
-
-/*
- * This checks to see if the given function is a "noreturn" function.
- *
- * For global functions which are outside the scope of this object file, we
- * have to keep a manual list of them.
- *
- * For local functions, we have to detect them manually by simply looking for
- * the lack of a return instruction.
- *
- * Returns:
- * -1: error
- * 0: no dead end
- * 1: dead end
- */
-static int __dead_end_function(struct objtool_file *file, struct symbol *func,
- int recursion)
-{
- int i;
- struct instruction *insn;
- bool empty = true;
-
- /*
- * Unfortunately these have to be hard coded because the noreturn
- * attribute isn't provided in ELF data.
- */
- static const char * const global_noreturns[] = {
- "__stack_chk_fail",
- "panic",
- "do_exit",
- "do_task_dead",
- "__module_put_and_exit",
- "complete_and_exit",
- "kvm_spurious_fault",
- "__reiserfs_panic",
- "lbug_with_loc",
- "fortify_panic",
- };
-
- if (func->bind == STB_WEAK)
- return 0;
-
- if (func->bind == STB_GLOBAL)
- for (i = 0; i < ARRAY_SIZE(global_noreturns); i++)
- if (!strcmp(func->name, global_noreturns[i]))
- return 1;
-
- if (!func->sec)
- return 0;
-
- func_for_each_insn(file, func, insn) {
- empty = false;
-
- if (insn->type == INSN_RETURN)
- return 0;
- }
-
- if (empty)
- return 0;
-
- /*
- * A function can have a sibling call instead of a return. In that
- * case, the function's dead-end status depends on whether the target
- * of the sibling call returns.
- */
- func_for_each_insn(file, func, insn) {
- if (insn->sec != func->sec ||
- insn->offset >= func->offset + func->len)
- break;
-
- if (insn->type == INSN_JUMP_UNCONDITIONAL) {
- struct instruction *dest = insn->jump_dest;
- struct symbol *dest_func;
-
- if (!dest)
- /* sibling call to another file */
- return 0;
-
- if (dest->sec != func->sec ||
- dest->offset < func->offset ||
- dest->offset >= func->offset + func->len) {
- /* local sibling call */
- dest_func = find_symbol_by_offset(dest->sec,
- dest->offset);
- if (!dest_func)
- continue;
-
- if (recursion == 5) {
- WARN_FUNC("infinite recursion (objtool bug!)",
- dest->sec, dest->offset);
- return -1;
- }
-
- return __dead_end_function(file, dest_func,
- recursion + 1);
- }
- }
-
- if (insn->type == INSN_JUMP_DYNAMIC && list_empty(&insn->alts))
- /* sibling call */
- return 0;
- }
-
- return 1;
-}
-
-static int dead_end_function(struct objtool_file *file, struct symbol *func)
-{
- return __dead_end_function(file, func, 0);
-}
-
-/*
- * Call the arch-specific instruction decoder for all the instructions and add
- * them to the global instruction list.
- */
-static int decode_instructions(struct objtool_file *file)
-{
- struct section *sec;
- struct symbol *func;
- unsigned long offset;
- struct instruction *insn;
- int ret;
-
- list_for_each_entry(sec, &file->elf->sections, list) {
-
- if (!(sec->sh.sh_flags & SHF_EXECINSTR))
- continue;
-
- for (offset = 0; offset < sec->len; offset += insn->len) {
- insn = malloc(sizeof(*insn));
- memset(insn, 0, sizeof(*insn));
-
- INIT_LIST_HEAD(&insn->alts);
- insn->sec = sec;
- insn->offset = offset;
-
- ret = arch_decode_instruction(file->elf, sec, offset,
- sec->len - offset,
- &insn->len, &insn->type,
- &insn->immediate);
- if (ret)
- return ret;
-
- if (!insn->type || insn->type > INSN_LAST) {
- WARN_FUNC("invalid instruction type %d",
- insn->sec, insn->offset, insn->type);
- return -1;
- }
-
- hash_add(file->insn_hash, &insn->hash, insn->offset);
- list_add_tail(&insn->list, &file->insn_list);
- }
-
- list_for_each_entry(func, &sec->symbol_list, list) {
- if (func->type != STT_FUNC)
- continue;
-
- if (!find_insn(file, sec, func->offset)) {
- WARN("%s(): can't find starting instruction",
- func->name);
- return -1;
- }
-
- func_for_each_insn(file, func, insn)
- if (!insn->func)
- insn->func = func;
- }
- }
-
- return 0;
-}
-
-/*
- * Find all uses of the unreachable() macro, which are code path dead ends.
- */
-static int add_dead_ends(struct objtool_file *file)
-{
- struct section *sec;
- struct rela *rela;
- struct instruction *insn;
- bool found;
-
- sec = find_section_by_name(file->elf, ".rela.discard.unreachable");
- if (!sec)
- return 0;
-
- list_for_each_entry(rela, &sec->rela_list, list) {
- if (rela->sym->type != STT_SECTION) {
- WARN("unexpected relocation symbol type in %s", sec->name);
- return -1;
- }
- insn = find_insn(file, rela->sym->sec, rela->addend);
- if (insn)
- insn = list_prev_entry(insn, list);
- else if (rela->addend == rela->sym->sec->len) {
- found = false;
- list_for_each_entry_reverse(insn, &file->insn_list, list) {
- if (insn->sec == rela->sym->sec) {
- found = true;
- break;
- }
- }
-
- if (!found) {
- WARN("can't find unreachable insn at %s+0x%x",
- rela->sym->sec->name, rela->addend);
- return -1;
- }
- } else {
- WARN("can't find unreachable insn at %s+0x%x",
- rela->sym->sec->name, rela->addend);
- return -1;
- }
-
- insn->dead_end = true;
- }
-
- return 0;
-}
-
-/*
- * Warnings shouldn't be reported for ignored functions.
- */
-static void add_ignores(struct objtool_file *file)
-{
- struct instruction *insn;
- struct section *sec;
- struct symbol *func;
-
- list_for_each_entry(sec, &file->elf->sections, list) {
- list_for_each_entry(func, &sec->symbol_list, list) {
- if (func->type != STT_FUNC)
- continue;
-
- if (!ignore_func(file, func))
- continue;
-
- func_for_each_insn(file, func, insn)
- insn->visited = true;
- }
- }
-}
-
-/*
- * Find the destination instructions for all jumps.
- */
-static int add_jump_destinations(struct objtool_file *file)
-{
- struct instruction *insn;
- struct rela *rela;
- struct section *dest_sec;
- unsigned long dest_off;
-
- for_each_insn(file, insn) {
- if (insn->type != INSN_JUMP_CONDITIONAL &&
- insn->type != INSN_JUMP_UNCONDITIONAL)
- continue;
-
- /* skip ignores */
- if (insn->visited)
- continue;
-
- rela = find_rela_by_dest_range(insn->sec, insn->offset,
- insn->len);
- if (!rela) {
- dest_sec = insn->sec;
- dest_off = insn->offset + insn->len + insn->immediate;
- } else if (rela->sym->type == STT_SECTION) {
- dest_sec = rela->sym->sec;
- dest_off = rela->addend + 4;
- } else if (rela->sym->sec->idx) {
- dest_sec = rela->sym->sec;
- dest_off = rela->sym->sym.st_value + rela->addend + 4;
- } else {
- /* sibling call */
- insn->jump_dest = 0;
- continue;
- }
-
- insn->jump_dest = find_insn(file, dest_sec, dest_off);
- if (!insn->jump_dest) {
-
- /*
- * This is a special case where an alt instruction
- * jumps past the end of the section. These are
- * handled later in handle_group_alt().
- */
- if (!strcmp(insn->sec->name, ".altinstr_replacement"))
- continue;
-
- WARN_FUNC("can't find jump dest instruction at %s+0x%lx",
- insn->sec, insn->offset, dest_sec->name,
- dest_off);
- return -1;
- }
- }
-
- return 0;
-}
-
-/*
- * Find the destination instructions for all calls.
- */
-static int add_call_destinations(struct objtool_file *file)
-{
- struct instruction *insn;
- unsigned long dest_off;
- struct rela *rela;
-
- for_each_insn(file, insn) {
- if (insn->type != INSN_CALL)
- continue;
-
- rela = find_rela_by_dest_range(insn->sec, insn->offset,
- insn->len);
- if (!rela) {
- dest_off = insn->offset + insn->len + insn->immediate;
- insn->call_dest = find_symbol_by_offset(insn->sec,
- dest_off);
- if (!insn->call_dest) {
- WARN_FUNC("can't find call dest symbol at offset 0x%lx",
- insn->sec, insn->offset, dest_off);
- return -1;
- }
- } else if (rela->sym->type == STT_SECTION) {
- insn->call_dest = find_symbol_by_offset(rela->sym->sec,
- rela->addend+4);
- if (!insn->call_dest ||
- insn->call_dest->type != STT_FUNC) {
- WARN_FUNC("can't find call dest symbol at %s+0x%x",
- insn->sec, insn->offset,
- rela->sym->sec->name,
- rela->addend + 4);
- return -1;
- }
- } else
- insn->call_dest = rela->sym;
- }
-
- return 0;
-}
-
-/*
- * The .alternatives section requires some extra special care, over and above
- * what other special sections require:
- *
- * 1. Because alternatives are patched in-place, we need to insert a fake jump
- * instruction at the end so that validate_branch() skips all the original
- * replaced instructions when validating the new instruction path.
- *
- * 2. An added wrinkle is that the new instruction length might be zero. In
- * that case the old instructions are replaced with noops. We simulate that
- * by creating a fake jump as the only new instruction.
- *
- * 3. In some cases, the alternative section includes an instruction which
- * conditionally jumps to the _end_ of the entry. We have to modify these
- * jumps' destinations to point back to .text rather than the end of the
- * entry in .altinstr_replacement.
- *
- * 4. It has been requested that we don't validate the !POPCNT feature path
- * which is a "very very small percentage of machines".
- */
-static int handle_group_alt(struct objtool_file *file,
- struct special_alt *special_alt,
- struct instruction *orig_insn,
- struct instruction **new_insn)
-{
- struct instruction *last_orig_insn, *last_new_insn, *insn, *fake_jump;
- unsigned long dest_off;
-
- last_orig_insn = NULL;
- insn = orig_insn;
- sec_for_each_insn_from(file, insn) {
- if (insn->offset >= special_alt->orig_off + special_alt->orig_len)
- break;
-
- if (special_alt->skip_orig)
- insn->type = INSN_NOP;
-
- insn->alt_group = true;
- last_orig_insn = insn;
- }
-
- if (!next_insn_same_sec(file, last_orig_insn)) {
- WARN("%s: don't know how to handle alternatives at end of section",
- special_alt->orig_sec->name);
- return -1;
- }
-
- fake_jump = malloc(sizeof(*fake_jump));
- if (!fake_jump) {
- WARN("malloc failed");
- return -1;
- }
- memset(fake_jump, 0, sizeof(*fake_jump));
- INIT_LIST_HEAD(&fake_jump->alts);
- fake_jump->sec = special_alt->new_sec;
- fake_jump->offset = -1;
- fake_jump->type = INSN_JUMP_UNCONDITIONAL;
- fake_jump->jump_dest = list_next_entry(last_orig_insn, list);
-
- if (!special_alt->new_len) {
- *new_insn = fake_jump;
- return 0;
- }
-
- last_new_insn = NULL;
- insn = *new_insn;
- sec_for_each_insn_from(file, insn) {
- if (insn->offset >= special_alt->new_off + special_alt->new_len)
- break;
-
- last_new_insn = insn;
-
- if (insn->type != INSN_JUMP_CONDITIONAL &&
- insn->type != INSN_JUMP_UNCONDITIONAL)
- continue;
-
- if (!insn->immediate)
- continue;
-
- dest_off = insn->offset + insn->len + insn->immediate;
- if (dest_off == special_alt->new_off + special_alt->new_len)
- insn->jump_dest = fake_jump;
-
- if (!insn->jump_dest) {
- WARN_FUNC("can't find alternative jump destination",
- insn->sec, insn->offset);
- return -1;
- }
- }
-
- if (!last_new_insn) {
- WARN_FUNC("can't find last new alternative instruction",
- special_alt->new_sec, special_alt->new_off);
- return -1;
- }
-
- list_add(&fake_jump->list, &last_new_insn->list);
-
- return 0;
-}
-
-/*
- * A jump table entry can either convert a nop to a jump or a jump to a nop.
- * If the original instruction is a jump, make the alt entry an effective nop
- * by just skipping the original instruction.
- */
-static int handle_jump_alt(struct objtool_file *file,
- struct special_alt *special_alt,
- struct instruction *orig_insn,
- struct instruction **new_insn)
-{
- if (orig_insn->type == INSN_NOP)
- return 0;
-
- if (orig_insn->type != INSN_JUMP_UNCONDITIONAL) {
- WARN_FUNC("unsupported instruction at jump label",
- orig_insn->sec, orig_insn->offset);
- return -1;
- }
-
- *new_insn = list_next_entry(orig_insn, list);
- return 0;
-}
-
-/*
- * Read all the special sections which have alternate instructions which can be
- * patched in or redirected to at runtime. Each instruction having alternate
- * instruction(s) has them added to its insn->alts list, which will be
- * traversed in validate_branch().
- */
-static int add_special_section_alts(struct objtool_file *file)
-{
- struct list_head special_alts;
- struct instruction *orig_insn, *new_insn;
- struct special_alt *special_alt, *tmp;
- struct alternative *alt;
- int ret;
-
- ret = special_get_alts(file->elf, &special_alts);
- if (ret)
- return ret;
-
- list_for_each_entry_safe(special_alt, tmp, &special_alts, list) {
- alt = malloc(sizeof(*alt));
- if (!alt) {
- WARN("malloc failed");
- ret = -1;
- goto out;
- }
-
- orig_insn = find_insn(file, special_alt->orig_sec,
- special_alt->orig_off);
- if (!orig_insn) {
- WARN_FUNC("special: can't find orig instruction",
- special_alt->orig_sec, special_alt->orig_off);
- ret = -1;
- goto out;
- }
+#include "check.h"
- new_insn = NULL;
- if (!special_alt->group || special_alt->new_len) {
- new_insn = find_insn(file, special_alt->new_sec,
- special_alt->new_off);
- if (!new_insn) {
- WARN_FUNC("special: can't find new instruction",
- special_alt->new_sec,
- special_alt->new_off);
- ret = -1;
- goto out;
- }
- }
+bool nofp;
- if (special_alt->group) {
- ret = handle_group_alt(file, special_alt, orig_insn,
- &new_insn);
- if (ret)
- goto out;
- } else if (special_alt->jump_or_nop) {
- ret = handle_jump_alt(file, special_alt, orig_insn,
- &new_insn);
- if (ret)
- goto out;
- }
-
- alt->insn = new_insn;
- list_add_tail(&alt->list, &orig_insn->alts);
-
- list_del(&special_alt->list);
- free(special_alt);
- }
-
-out:
- return ret;
-}
-
-static int add_switch_table(struct objtool_file *file, struct symbol *func,
- struct instruction *insn, struct rela *table,
- struct rela *next_table)
-{
- struct rela *rela = table;
- struct instruction *alt_insn;
- struct alternative *alt;
-
- list_for_each_entry_from(rela, &file->rodata->rela->rela_list, list) {
- if (rela == next_table)
- break;
-
- if (rela->sym->sec != insn->sec ||
- rela->addend <= func->offset ||
- rela->addend >= func->offset + func->len)
- break;
-
- alt_insn = find_insn(file, insn->sec, rela->addend);
- if (!alt_insn) {
- WARN("%s: can't find instruction at %s+0x%x",
- file->rodata->rela->name, insn->sec->name,
- rela->addend);
- return -1;
- }
-
- alt = malloc(sizeof(*alt));
- if (!alt) {
- WARN("malloc failed");
- return -1;
- }
-
- alt->insn = alt_insn;
- list_add_tail(&alt->list, &insn->alts);
- }
-
- return 0;
-}
-
-/*
- * find_switch_table() - Given a dynamic jump, find the switch jump table in
- * .rodata associated with it.
- *
- * There are 3 basic patterns:
- *
- * 1. jmpq *[rodata addr](,%reg,8)
- *
- * This is the most common case by far. It jumps to an address in a simple
- * jump table which is stored in .rodata.
- *
- * 2. jmpq *[rodata addr](%rip)
- *
- * This is caused by a rare GCC quirk, currently only seen in three driver
- * functions in the kernel, only with certain obscure non-distro configs.
- *
- * As part of an optimization, GCC makes a copy of an existing switch jump
- * table, modifies it, and then hard-codes the jump (albeit with an indirect
- * jump) to use a single entry in the table. The rest of the jump table and
- * some of its jump targets remain as dead code.
- *
- * In such a case we can just crudely ignore all unreachable instruction
- * warnings for the entire object file. Ideally we would just ignore them
- * for the function, but that would require redesigning the code quite a
- * bit. And honestly that's just not worth doing: unreachable instruction
- * warnings are of questionable value anyway, and this is such a rare issue.
- *
- * 3. mov [rodata addr],%reg1
- * ... some instructions ...
- * jmpq *(%reg1,%reg2,8)
- *
- * This is a fairly uncommon pattern which is new for GCC 6. As of this
- * writing, there are 11 occurrences of it in the allmodconfig kernel.
- *
- * TODO: Once we have DWARF CFI and smarter instruction decoding logic,
- * ensure the same register is used in the mov and jump instructions.
- */
-static struct rela *find_switch_table(struct objtool_file *file,
- struct symbol *func,
- struct instruction *insn)
-{
- struct rela *text_rela, *rodata_rela;
- struct instruction *orig_insn = insn;
-
- text_rela = find_rela_by_dest_range(insn->sec, insn->offset, insn->len);
- if (text_rela && text_rela->sym == file->rodata->sym) {
- /* case 1 */
- rodata_rela = find_rela_by_dest(file->rodata,
- text_rela->addend);
- if (rodata_rela)
- return rodata_rela;
-
- /* case 2 */
- rodata_rela = find_rela_by_dest(file->rodata,
- text_rela->addend + 4);
- if (!rodata_rela)
- return NULL;
- file->ignore_unreachables = true;
- return rodata_rela;
- }
-
- /* case 3 */
- func_for_each_insn_continue_reverse(file, func, insn) {
- if (insn->type == INSN_JUMP_DYNAMIC)
- break;
-
- /* allow small jumps within the range */
- if (insn->type == INSN_JUMP_UNCONDITIONAL &&
- insn->jump_dest &&
- (insn->jump_dest->offset <= insn->offset ||
- insn->jump_dest->offset > orig_insn->offset))
- break;
-
- /* look for a relocation which references .rodata */
- text_rela = find_rela_by_dest_range(insn->sec, insn->offset,
- insn->len);
- if (!text_rela || text_rela->sym != file->rodata->sym)
- continue;
-
- /*
- * Make sure the .rodata address isn't associated with a
- * symbol. gcc jump tables are anonymous data.
- */
- if (find_symbol_containing(file->rodata, text_rela->addend))
- continue;
-
- return find_rela_by_dest(file->rodata, text_rela->addend);
- }
-
- return NULL;
-}
-
-static int add_func_switch_tables(struct objtool_file *file,
- struct symbol *func)
-{
- struct instruction *insn, *prev_jump = NULL;
- struct rela *rela, *prev_rela = NULL;
- int ret;
-
- func_for_each_insn(file, func, insn) {
- if (insn->type != INSN_JUMP_DYNAMIC)
- continue;
-
- rela = find_switch_table(file, func, insn);
- if (!rela)
- continue;
-
- /*
- * We found a switch table, but we don't know yet how big it
- * is. Don't add it until we reach the end of the function or
- * the beginning of another switch table in the same function.
- */
- if (prev_jump) {
- ret = add_switch_table(file, func, prev_jump, prev_rela,
- rela);
- if (ret)
- return ret;
- }
-
- prev_jump = insn;
- prev_rela = rela;
- }
-
- if (prev_jump) {
- ret = add_switch_table(file, func, prev_jump, prev_rela, NULL);
- if (ret)
- return ret;
- }
-
- return 0;
-}
-
-/*
- * For some switch statements, gcc generates a jump table in the .rodata
- * section which contains a list of addresses within the function to jump to.
- * This finds these jump tables and adds them to the insn->alts lists.
- */
-static int add_switch_table_alts(struct objtool_file *file)
-{
- struct section *sec;
- struct symbol *func;
- int ret;
-
- if (!file->rodata || !file->rodata->rela)
- return 0;
-
- list_for_each_entry(sec, &file->elf->sections, list) {
- list_for_each_entry(func, &sec->symbol_list, list) {
- if (func->type != STT_FUNC)
- continue;
-
- ret = add_func_switch_tables(file, func);
- if (ret)
- return ret;
- }
- }
-
- return 0;
-}
-
-static int decode_sections(struct objtool_file *file)
-{
- int ret;
-
- ret = decode_instructions(file);
- if (ret)
- return ret;
-
- ret = add_dead_ends(file);
- if (ret)
- return ret;
-
- add_ignores(file);
-
- ret = add_jump_destinations(file);
- if (ret)
- return ret;
-
- ret = add_call_destinations(file);
- if (ret)
- return ret;
-
- ret = add_special_section_alts(file);
- if (ret)
- return ret;
-
- ret = add_switch_table_alts(file);
- if (ret)
- return ret;
-
- return 0;
-}
-
-static bool is_fentry_call(struct instruction *insn)
-{
- if (insn->type == INSN_CALL &&
- insn->call_dest->type == STT_NOTYPE &&
- !strcmp(insn->call_dest->name, "__fentry__"))
- return true;
-
- return false;
-}
-
-static bool has_modified_stack_frame(struct instruction *insn)
-{
- return (insn->state & STATE_FP_SAVED) ||
- (insn->state & STATE_FP_SETUP);
-}
-
-static bool has_valid_stack_frame(struct instruction *insn)
-{
- return (insn->state & STATE_FP_SAVED) &&
- (insn->state & STATE_FP_SETUP);
-}
-
-static unsigned int frame_state(unsigned long state)
-{
- return (state & (STATE_FP_SAVED | STATE_FP_SETUP));
-}
-
-/*
- * Follow the branch starting at the given instruction, and recursively follow
- * any other branches (jumps). Meanwhile, track the frame pointer state at
- * each instruction and validate all the rules described in
- * tools/objtool/Documentation/stack-validation.txt.
- */
-static int validate_branch(struct objtool_file *file,
- struct instruction *first, unsigned char first_state)
-{
- struct alternative *alt;
- struct instruction *insn;
- struct section *sec;
- struct symbol *func = NULL;
- unsigned char state;
- int ret;
-
- insn = first;
- sec = insn->sec;
- state = first_state;
-
- if (insn->alt_group && list_empty(&insn->alts)) {
- WARN_FUNC("don't know how to handle branch to middle of alternative instruction group",
- sec, insn->offset);
- return 1;
- }
-
- while (1) {
- if (file->c_file && insn->func) {
- if (func && func != insn->func) {
- WARN("%s() falls through to next function %s()",
- func->name, insn->func->name);
- return 1;
- }
-
- func = insn->func;
- }
-
- if (insn->visited) {
- if (frame_state(insn->state) != frame_state(state)) {
- WARN_FUNC("frame pointer state mismatch",
- sec, insn->offset);
- return 1;
- }
-
- return 0;
- }
-
- insn->visited = true;
- insn->state = state;
-
- list_for_each_entry(alt, &insn->alts, list) {
- ret = validate_branch(file, alt->insn, state);
- if (ret)
- return 1;
- }
-
- switch (insn->type) {
-
- case INSN_FP_SAVE:
- if (!nofp) {
- if (state & STATE_FP_SAVED) {
- WARN_FUNC("duplicate frame pointer save",
- sec, insn->offset);
- return 1;
- }
- state |= STATE_FP_SAVED;
- }
- break;
-
- case INSN_FP_SETUP:
- if (!nofp) {
- if (state & STATE_FP_SETUP) {
- WARN_FUNC("duplicate frame pointer setup",
- sec, insn->offset);
- return 1;
- }
- state |= STATE_FP_SETUP;
- }
- break;
-
- case INSN_FP_RESTORE:
- if (!nofp) {
- if (has_valid_stack_frame(insn))
- state &= ~STATE_FP_SETUP;
-
- state &= ~STATE_FP_SAVED;
- }
- break;
-
- case INSN_RETURN:
- if (!nofp && has_modified_stack_frame(insn)) {
- WARN_FUNC("return without frame pointer restore",
- sec, insn->offset);
- return 1;
- }
- return 0;
-
- case INSN_CALL:
- if (is_fentry_call(insn)) {
- state |= STATE_FENTRY;
- break;
- }
-
- ret = dead_end_function(file, insn->call_dest);
- if (ret == 1)
- return 0;
- if (ret == -1)
- return 1;
-
- /* fallthrough */
- case INSN_CALL_DYNAMIC:
- if (!nofp && !has_valid_stack_frame(insn)) {
- WARN_FUNC("call without frame pointer save/setup",
- sec, insn->offset);
- return 1;
- }
- break;
-
- case INSN_JUMP_CONDITIONAL:
- case INSN_JUMP_UNCONDITIONAL:
- if (insn->jump_dest) {
- ret = validate_branch(file, insn->jump_dest,
- state);
- if (ret)
- return 1;
- } else if (has_modified_stack_frame(insn)) {
- WARN_FUNC("sibling call from callable instruction with changed frame pointer",
- sec, insn->offset);
- return 1;
- } /* else it's a sibling call */
-
- if (insn->type == INSN_JUMP_UNCONDITIONAL)
- return 0;
-
- break;
-
- case INSN_JUMP_DYNAMIC:
- if (list_empty(&insn->alts) &&
- has_modified_stack_frame(insn)) {
- WARN_FUNC("sibling call from callable instruction with changed frame pointer",
- sec, insn->offset);
- return 1;
- }
-
- return 0;
-
- default:
- break;
- }
-
- if (insn->dead_end)
- return 0;
-
- insn = next_insn_same_sec(file, insn);
- if (!insn) {
- WARN("%s: unexpected end of section", sec->name);
- return 1;
- }
- }
-
- return 0;
-}
-
-static bool is_kasan_insn(struct instruction *insn)
-{
- return (insn->type == INSN_CALL &&
- !strcmp(insn->call_dest->name, "__asan_handle_no_return"));
-}
-
-static bool is_ubsan_insn(struct instruction *insn)
-{
- return (insn->type == INSN_CALL &&
- !strcmp(insn->call_dest->name,
- "__ubsan_handle_builtin_unreachable"));
-}
-
-static bool ignore_unreachable_insn(struct symbol *func,
- struct instruction *insn)
-{
- int i;
-
- if (insn->type == INSN_NOP)
- return true;
-
- /*
- * Check if this (or a subsequent) instruction is related to
- * CONFIG_UBSAN or CONFIG_KASAN.
- *
- * End the search at 5 instructions to avoid going into the weeds.
- */
- for (i = 0; i < 5; i++) {
-
- if (is_kasan_insn(insn) || is_ubsan_insn(insn))
- return true;
-
- if (insn->type == INSN_JUMP_UNCONDITIONAL && insn->jump_dest) {
- insn = insn->jump_dest;
- continue;
- }
-
- if (insn->offset + insn->len >= func->offset + func->len)
- break;
- insn = list_next_entry(insn, list);
- }
-
- return false;
-}
-
-static int validate_functions(struct objtool_file *file)
-{
- struct section *sec;
- struct symbol *func;
- struct instruction *insn;
- int ret, warnings = 0;
-
- list_for_each_entry(sec, &file->elf->sections, list) {
- list_for_each_entry(func, &sec->symbol_list, list) {
- if (func->type != STT_FUNC)
- continue;
-
- insn = find_insn(file, sec, func->offset);
- if (!insn)
- continue;
-
- ret = validate_branch(file, insn, 0);
- warnings += ret;
- }
- }
-
- list_for_each_entry(sec, &file->elf->sections, list) {
- list_for_each_entry(func, &sec->symbol_list, list) {
- if (func->type != STT_FUNC)
- continue;
-
- func_for_each_insn(file, func, insn) {
- if (insn->visited)
- continue;
-
- insn->visited = true;
-
- if (file->ignore_unreachables || warnings ||
- ignore_unreachable_insn(func, insn))
- continue;
-
- /*
- * gcov produces a lot of unreachable
- * instructions. If we get an unreachable
- * warning and the file has gcov enabled, just
- * ignore it, and all other such warnings for
- * the file.
- */
- if (!file->ignore_unreachables &&
- gcov_enabled(file)) {
- file->ignore_unreachables = true;
- continue;
- }
-
- WARN_FUNC("function has unreachable instruction", insn->sec, insn->offset);
- warnings++;
- }
- }
- }
-
- return warnings;
-}
-
-static int validate_uncallable_instructions(struct objtool_file *file)
-{
- struct instruction *insn;
- int warnings = 0;
-
- for_each_insn(file, insn) {
- if (!insn->visited && insn->type == INSN_RETURN) {
- WARN_FUNC("return instruction outside of a callable function",
- insn->sec, insn->offset);
- warnings++;
- }
- }
-
- return warnings;
-}
-
-static void cleanup(struct objtool_file *file)
-{
- struct instruction *insn, *tmpinsn;
- struct alternative *alt, *tmpalt;
-
- list_for_each_entry_safe(insn, tmpinsn, &file->insn_list, list) {
- list_for_each_entry_safe(alt, tmpalt, &insn->alts, list) {
- list_del(&alt->list);
- free(alt);
- }
- list_del(&insn->list);
- hash_del(&insn->hash);
- free(insn);
- }
- elf_close(file->elf);
-}
-
-const char * const check_usage[] = {
+static const char * const check_usage[] = {
"objtool check [<options>] file.o",
NULL,
};
+const struct option check_options[] = {
+ OPT_BOOLEAN('f', "no-fp", &nofp, "Skip frame pointer validation"),
+ OPT_END(),
+};
+
int cmd_check(int argc, const char **argv)
{
- struct objtool_file file;
- int ret, warnings = 0;
-
- const struct option options[] = {
- OPT_BOOLEAN('f', "no-fp", &nofp, "Skip frame pointer validation"),
- OPT_END(),
- };
+ const char *objname;
- argc = parse_options(argc, argv, options, check_usage, 0);
+ argc = parse_options(argc, argv, check_options, check_usage, 0);
if (argc != 1)
- usage_with_options(check_usage, options);
+ usage_with_options(check_usage, check_options);
objname = argv[0];
- file.elf = elf_open(objname);
- if (!file.elf) {
- fprintf(stderr, "error reading elf file %s\n", objname);
- return 1;
- }
-
- INIT_LIST_HEAD(&file.insn_list);
- hash_init(file.insn_hash);
- file.whitelist = find_section_by_name(file.elf, ".discard.func_stack_frame_non_standard");
- file.rodata = find_section_by_name(file.elf, ".rodata");
- file.ignore_unreachables = false;
- file.c_file = find_section_by_name(file.elf, ".comment");
-
- ret = decode_sections(&file);
- if (ret < 0)
- goto out;
- warnings += ret;
-
- ret = validate_functions(&file);
- if (ret < 0)
- goto out;
- warnings += ret;
-
- ret = validate_uncallable_instructions(&file);
- if (ret < 0)
- goto out;
- warnings += ret;
-
-out:
- cleanup(&file);
-
- /* ignore warnings for now until we get all the code cleaned up */
- if (ret || warnings)
- return 0;
- return 0;
+ return check(objname, nofp);
}
--- /dev/null
+/*
+ * Copyright (C) 2015-2017 Josh Poimboeuf <jpoimboe@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef _OBJTOOL_CFI_H
+#define _OBJTOOL_CFI_H
+
+#define CFI_UNDEFINED -1
+#define CFI_CFA -2
+#define CFI_SP_INDIRECT -3
+#define CFI_BP_INDIRECT -4
+
+#define CFI_AX 0
+#define CFI_DX 1
+#define CFI_CX 2
+#define CFI_BX 3
+#define CFI_SI 4
+#define CFI_DI 5
+#define CFI_BP 6
+#define CFI_SP 7
+#define CFI_R8 8
+#define CFI_R9 9
+#define CFI_R10 10
+#define CFI_R11 11
+#define CFI_R12 12
+#define CFI_R13 13
+#define CFI_R14 14
+#define CFI_R15 15
+#define CFI_RA 16
+#define CFI_NUM_REGS 17
+
+struct cfi_reg {
+ int base;
+ int offset;
+};
+
+struct cfi_state {
+ struct cfi_reg cfa;
+ struct cfi_reg regs[CFI_NUM_REGS];
+};
+
+#endif /* _OBJTOOL_CFI_H */
--- /dev/null
+/*
+ * Copyright (C) 2015-2017 Josh Poimboeuf <jpoimboe@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <string.h>
+#include <stdlib.h>
+
+#include "check.h"
+#include "elf.h"
+#include "special.h"
+#include "arch.h"
+#include "warn.h"
+
+#include <linux/hashtable.h>
+#include <linux/kernel.h>
+
+struct alternative {
+ struct list_head list;
+ struct instruction *insn;
+};
+
+const char *objname;
+static bool nofp;
+struct cfi_state initial_func_cfi;
+
+static struct instruction *find_insn(struct objtool_file *file,
+ struct section *sec, unsigned long offset)
+{
+ struct instruction *insn;
+
+ hash_for_each_possible(file->insn_hash, insn, hash, offset)
+ if (insn->sec == sec && insn->offset == offset)
+ return insn;
+
+ return NULL;
+}
+
+static struct instruction *next_insn_same_sec(struct objtool_file *file,
+ struct instruction *insn)
+{
+ struct instruction *next = list_next_entry(insn, list);
+
+ if (!next || &next->list == &file->insn_list || next->sec != insn->sec)
+ return NULL;
+
+ return next;
+}
+
+static bool gcov_enabled(struct objtool_file *file)
+{
+ struct section *sec;
+ struct symbol *sym;
+
+ for_each_sec(file, sec)
+ list_for_each_entry(sym, &sec->symbol_list, list)
+ if (!strncmp(sym->name, "__gcov_.", 8))
+ return true;
+
+ return false;
+}
+
+#define func_for_each_insn(file, func, insn) \
+ for (insn = find_insn(file, func->sec, func->offset); \
+ insn && &insn->list != &file->insn_list && \
+ insn->sec == func->sec && \
+ insn->offset < func->offset + func->len; \
+ insn = list_next_entry(insn, list))
+
+#define func_for_each_insn_continue_reverse(file, func, insn) \
+ for (insn = list_prev_entry(insn, list); \
+ &insn->list != &file->insn_list && \
+ insn->sec == func->sec && insn->offset >= func->offset; \
+ insn = list_prev_entry(insn, list))
+
+#define sec_for_each_insn_from(file, insn) \
+ for (; insn; insn = next_insn_same_sec(file, insn))
+
+#define sec_for_each_insn_continue(file, insn) \
+ for (insn = next_insn_same_sec(file, insn); insn; \
+ insn = next_insn_same_sec(file, insn))
+
+/*
+ * Check if the function has been manually whitelisted with the
+ * STACK_FRAME_NON_STANDARD macro, or if it should be automatically whitelisted
+ * due to its use of a context switching instruction.
+ */
+static bool ignore_func(struct objtool_file *file, struct symbol *func)
+{
+ struct rela *rela;
+ struct instruction *insn;
+
+ /* check for STACK_FRAME_NON_STANDARD */
+ if (file->whitelist && file->whitelist->rela)
+ list_for_each_entry(rela, &file->whitelist->rela->rela_list, list) {
+ if (rela->sym->type == STT_SECTION &&
+ rela->sym->sec == func->sec &&
+ rela->addend == func->offset)
+ return true;
+ if (rela->sym->type == STT_FUNC && rela->sym == func)
+ return true;
+ }
+
+ /* check if it has a context switching instruction */
+ func_for_each_insn(file, func, insn)
+ if (insn->type == INSN_CONTEXT_SWITCH)
+ return true;
+
+ return false;
+}
+
+/*
+ * This checks to see if the given function is a "noreturn" function.
+ *
+ * For global functions which are outside the scope of this object file, we
+ * have to keep a manual list of them.
+ *
+ * For local functions, we have to detect them manually by simply looking for
+ * the lack of a return instruction.
+ *
+ * Returns:
+ * -1: error
+ * 0: no dead end
+ * 1: dead end
+ */
+static int __dead_end_function(struct objtool_file *file, struct symbol *func,
+ int recursion)
+{
+ int i;
+ struct instruction *insn;
+ bool empty = true;
+
+ /*
+ * Unfortunately these have to be hard coded because the noreturn
+ * attribute isn't provided in ELF data.
+ */
+ static const char * const global_noreturns[] = {
+ "__stack_chk_fail",
+ "panic",
+ "do_exit",
+ "do_task_dead",
+ "__module_put_and_exit",
+ "complete_and_exit",
+ "kvm_spurious_fault",
+ "__reiserfs_panic",
+ "lbug_with_loc",
+ "fortify_panic",
+ };
+
+ if (func->bind == STB_WEAK)
+ return 0;
+
+ if (func->bind == STB_GLOBAL)
+ for (i = 0; i < ARRAY_SIZE(global_noreturns); i++)
+ if (!strcmp(func->name, global_noreturns[i]))
+ return 1;
+
+ if (!func->sec)
+ return 0;
+
+ func_for_each_insn(file, func, insn) {
+ empty = false;
+
+ if (insn->type == INSN_RETURN)
+ return 0;
+ }
+
+ if (empty)
+ return 0;
+
+ /*
+ * A function can have a sibling call instead of a return. In that
+ * case, the function's dead-end status depends on whether the target
+ * of the sibling call returns.
+ */
+ func_for_each_insn(file, func, insn) {
+ if (insn->sec != func->sec ||
+ insn->offset >= func->offset + func->len)
+ break;
+
+ if (insn->type == INSN_JUMP_UNCONDITIONAL) {
+ struct instruction *dest = insn->jump_dest;
+ struct symbol *dest_func;
+
+ if (!dest)
+ /* sibling call to another file */
+ return 0;
+
+ if (dest->sec != func->sec ||
+ dest->offset < func->offset ||
+ dest->offset >= func->offset + func->len) {
+ /* local sibling call */
+ dest_func = find_symbol_by_offset(dest->sec,
+ dest->offset);
+ if (!dest_func)
+ continue;
+
+ if (recursion == 5) {
+ WARN_FUNC("infinite recursion (objtool bug!)",
+ dest->sec, dest->offset);
+ return -1;
+ }
+
+ return __dead_end_function(file, dest_func,
+ recursion + 1);
+ }
+ }
+
+ if (insn->type == INSN_JUMP_DYNAMIC && list_empty(&insn->alts))
+ /* sibling call */
+ return 0;
+ }
+
+ return 1;
+}
+
+static int dead_end_function(struct objtool_file *file, struct symbol *func)
+{
+ return __dead_end_function(file, func, 0);
+}
+
+static void clear_insn_state(struct insn_state *state)
+{
+ int i;
+
+ memset(state, 0, sizeof(*state));
+ state->cfa.base = CFI_UNDEFINED;
+ for (i = 0; i < CFI_NUM_REGS; i++)
+ state->regs[i].base = CFI_UNDEFINED;
+ state->drap_reg = CFI_UNDEFINED;
+}
+
+/*
+ * Call the arch-specific instruction decoder for all the instructions and add
+ * them to the global instruction list.
+ */
+static int decode_instructions(struct objtool_file *file)
+{
+ struct section *sec;
+ struct symbol *func;
+ unsigned long offset;
+ struct instruction *insn;
+ int ret;
+
+ for_each_sec(file, sec) {
+
+ if (!(sec->sh.sh_flags & SHF_EXECINSTR))
+ continue;
+
+ for (offset = 0; offset < sec->len; offset += insn->len) {
+ insn = malloc(sizeof(*insn));
+ if (!insn) {
+ WARN("malloc failed");
+ return -1;
+ }
+ memset(insn, 0, sizeof(*insn));
+ INIT_LIST_HEAD(&insn->alts);
+ clear_insn_state(&insn->state);
+
+ insn->sec = sec;
+ insn->offset = offset;
+
+ ret = arch_decode_instruction(file->elf, sec, offset,
+ sec->len - offset,
+ &insn->len, &insn->type,
+ &insn->immediate,
+ &insn->stack_op);
+ if (ret)
+ return ret;
+
+ if (!insn->type || insn->type > INSN_LAST) {
+ WARN_FUNC("invalid instruction type %d",
+ insn->sec, insn->offset, insn->type);
+ return -1;
+ }
+
+ hash_add(file->insn_hash, &insn->hash, insn->offset);
+ list_add_tail(&insn->list, &file->insn_list);
+ }
+
+ list_for_each_entry(func, &sec->symbol_list, list) {
+ if (func->type != STT_FUNC)
+ continue;
+
+ if (!find_insn(file, sec, func->offset)) {
+ WARN("%s(): can't find starting instruction",
+ func->name);
+ return -1;
+ }
+
+ func_for_each_insn(file, func, insn)
+ if (!insn->func)
+ insn->func = func;
+ }
+ }
+
+ return 0;
+}
+
+/*
+ * Find all uses of the unreachable() macro, which are code path dead ends.
+ */
+static int add_dead_ends(struct objtool_file *file)
+{
+ struct section *sec;
+ struct rela *rela;
+ struct instruction *insn;
+ bool found;
+
+ sec = find_section_by_name(file->elf, ".rela.discard.unreachable");
+ if (!sec)
+ return 0;
+
+ list_for_each_entry(rela, &sec->rela_list, list) {
+ if (rela->sym->type != STT_SECTION) {
+ WARN("unexpected relocation symbol type in %s", sec->name);
+ return -1;
+ }
+ insn = find_insn(file, rela->sym->sec, rela->addend);
+ if (insn)
+ insn = list_prev_entry(insn, list);
+ else if (rela->addend == rela->sym->sec->len) {
+ found = false;
+ list_for_each_entry_reverse(insn, &file->insn_list, list) {
+ if (insn->sec == rela->sym->sec) {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found) {
+ WARN("can't find unreachable insn at %s+0x%x",
+ rela->sym->sec->name, rela->addend);
+ return -1;
+ }
+ } else {
+ WARN("can't find unreachable insn at %s+0x%x",
+ rela->sym->sec->name, rela->addend);
+ return -1;
+ }
+
+ insn->dead_end = true;
+ }
+
+ return 0;
+}
+
+/*
+ * Warnings shouldn't be reported for ignored functions.
+ */
+static void add_ignores(struct objtool_file *file)
+{
+ struct instruction *insn;
+ struct section *sec;
+ struct symbol *func;
+
+ for_each_sec(file, sec) {
+ list_for_each_entry(func, &sec->symbol_list, list) {
+ if (func->type != STT_FUNC)
+ continue;
+
+ if (!ignore_func(file, func))
+ continue;
+
+ func_for_each_insn(file, func, insn)
+ insn->ignore = true;
+ }
+ }
+}
+
+/*
+ * Find the destination instructions for all jumps.
+ */
+static int add_jump_destinations(struct objtool_file *file)
+{
+ struct instruction *insn;
+ struct rela *rela;
+ struct section *dest_sec;
+ unsigned long dest_off;
+
+ for_each_insn(file, insn) {
+ if (insn->type != INSN_JUMP_CONDITIONAL &&
+ insn->type != INSN_JUMP_UNCONDITIONAL)
+ continue;
+
+ if (insn->ignore)
+ continue;
+
+ rela = find_rela_by_dest_range(insn->sec, insn->offset,
+ insn->len);
+ if (!rela) {
+ dest_sec = insn->sec;
+ dest_off = insn->offset + insn->len + insn->immediate;
+ } else if (rela->sym->type == STT_SECTION) {
+ dest_sec = rela->sym->sec;
+ dest_off = rela->addend + 4;
+ } else if (rela->sym->sec->idx) {
+ dest_sec = rela->sym->sec;
+ dest_off = rela->sym->sym.st_value + rela->addend + 4;
+ } else {
+ /* sibling call */
+ insn->jump_dest = 0;
+ continue;
+ }
+
+ insn->jump_dest = find_insn(file, dest_sec, dest_off);
+ if (!insn->jump_dest) {
+
+ /*
+ * This is a special case where an alt instruction
+ * jumps past the end of the section. These are
+ * handled later in handle_group_alt().
+ */
+ if (!strcmp(insn->sec->name, ".altinstr_replacement"))
+ continue;
+
+ WARN_FUNC("can't find jump dest instruction at %s+0x%lx",
+ insn->sec, insn->offset, dest_sec->name,
+ dest_off);
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+/*
+ * Find the destination instructions for all calls.
+ */
+static int add_call_destinations(struct objtool_file *file)
+{
+ struct instruction *insn;
+ unsigned long dest_off;
+ struct rela *rela;
+
+ for_each_insn(file, insn) {
+ if (insn->type != INSN_CALL)
+ continue;
+
+ rela = find_rela_by_dest_range(insn->sec, insn->offset,
+ insn->len);
+ if (!rela) {
+ dest_off = insn->offset + insn->len + insn->immediate;
+ insn->call_dest = find_symbol_by_offset(insn->sec,
+ dest_off);
+ if (!insn->call_dest) {
+ WARN_FUNC("can't find call dest symbol at offset 0x%lx",
+ insn->sec, insn->offset, dest_off);
+ return -1;
+ }
+ } else if (rela->sym->type == STT_SECTION) {
+ insn->call_dest = find_symbol_by_offset(rela->sym->sec,
+ rela->addend+4);
+ if (!insn->call_dest ||
+ insn->call_dest->type != STT_FUNC) {
+ WARN_FUNC("can't find call dest symbol at %s+0x%x",
+ insn->sec, insn->offset,
+ rela->sym->sec->name,
+ rela->addend + 4);
+ return -1;
+ }
+ } else
+ insn->call_dest = rela->sym;
+ }
+
+ return 0;
+}
+
+/*
+ * The .alternatives section requires some extra special care, over and above
+ * what other special sections require:
+ *
+ * 1. Because alternatives are patched in-place, we need to insert a fake jump
+ * instruction at the end so that validate_branch() skips all the original
+ * replaced instructions when validating the new instruction path.
+ *
+ * 2. An added wrinkle is that the new instruction length might be zero. In
+ * that case the old instructions are replaced with noops. We simulate that
+ * by creating a fake jump as the only new instruction.
+ *
+ * 3. In some cases, the alternative section includes an instruction which
+ * conditionally jumps to the _end_ of the entry. We have to modify these
+ * jumps' destinations to point back to .text rather than the end of the
+ * entry in .altinstr_replacement.
+ *
+ * 4. It has been requested that we don't validate the !POPCNT feature path
+ * which is a "very very small percentage of machines".
+ */
+static int handle_group_alt(struct objtool_file *file,
+ struct special_alt *special_alt,
+ struct instruction *orig_insn,
+ struct instruction **new_insn)
+{
+ struct instruction *last_orig_insn, *last_new_insn, *insn, *fake_jump;
+ unsigned long dest_off;
+
+ last_orig_insn = NULL;
+ insn = orig_insn;
+ sec_for_each_insn_from(file, insn) {
+ if (insn->offset >= special_alt->orig_off + special_alt->orig_len)
+ break;
+
+ if (special_alt->skip_orig)
+ insn->type = INSN_NOP;
+
+ insn->alt_group = true;
+ last_orig_insn = insn;
+ }
+
+ if (!next_insn_same_sec(file, last_orig_insn)) {
+ WARN("%s: don't know how to handle alternatives at end of section",
+ special_alt->orig_sec->name);
+ return -1;
+ }
+
+ fake_jump = malloc(sizeof(*fake_jump));
+ if (!fake_jump) {
+ WARN("malloc failed");
+ return -1;
+ }
+ memset(fake_jump, 0, sizeof(*fake_jump));
+ INIT_LIST_HEAD(&fake_jump->alts);
+ clear_insn_state(&fake_jump->state);
+
+ fake_jump->sec = special_alt->new_sec;
+ fake_jump->offset = -1;
+ fake_jump->type = INSN_JUMP_UNCONDITIONAL;
+ fake_jump->jump_dest = list_next_entry(last_orig_insn, list);
+ fake_jump->ignore = true;
+
+ if (!special_alt->new_len) {
+ *new_insn = fake_jump;
+ return 0;
+ }
+
+ last_new_insn = NULL;
+ insn = *new_insn;
+ sec_for_each_insn_from(file, insn) {
+ if (insn->offset >= special_alt->new_off + special_alt->new_len)
+ break;
+
+ last_new_insn = insn;
+
+ if (insn->type != INSN_JUMP_CONDITIONAL &&
+ insn->type != INSN_JUMP_UNCONDITIONAL)
+ continue;
+
+ if (!insn->immediate)
+ continue;
+
+ dest_off = insn->offset + insn->len + insn->immediate;
+ if (dest_off == special_alt->new_off + special_alt->new_len)
+ insn->jump_dest = fake_jump;
+
+ if (!insn->jump_dest) {
+ WARN_FUNC("can't find alternative jump destination",
+ insn->sec, insn->offset);
+ return -1;
+ }
+ }
+
+ if (!last_new_insn) {
+ WARN_FUNC("can't find last new alternative instruction",
+ special_alt->new_sec, special_alt->new_off);
+ return -1;
+ }
+
+ list_add(&fake_jump->list, &last_new_insn->list);
+
+ return 0;
+}
+
+/*
+ * A jump table entry can either convert a nop to a jump or a jump to a nop.
+ * If the original instruction is a jump, make the alt entry an effective nop
+ * by just skipping the original instruction.
+ */
+static int handle_jump_alt(struct objtool_file *file,
+ struct special_alt *special_alt,
+ struct instruction *orig_insn,
+ struct instruction **new_insn)
+{
+ if (orig_insn->type == INSN_NOP)
+ return 0;
+
+ if (orig_insn->type != INSN_JUMP_UNCONDITIONAL) {
+ WARN_FUNC("unsupported instruction at jump label",
+ orig_insn->sec, orig_insn->offset);
+ return -1;
+ }
+
+ *new_insn = list_next_entry(orig_insn, list);
+ return 0;
+}
+
+/*
+ * Read all the special sections which have alternate instructions which can be
+ * patched in or redirected to at runtime. Each instruction having alternate
+ * instruction(s) has them added to its insn->alts list, which will be
+ * traversed in validate_branch().
+ */
+static int add_special_section_alts(struct objtool_file *file)
+{
+ struct list_head special_alts;
+ struct instruction *orig_insn, *new_insn;
+ struct special_alt *special_alt, *tmp;
+ struct alternative *alt;
+ int ret;
+
+ ret = special_get_alts(file->elf, &special_alts);
+ if (ret)
+ return ret;
+
+ list_for_each_entry_safe(special_alt, tmp, &special_alts, list) {
+ alt = malloc(sizeof(*alt));
+ if (!alt) {
+ WARN("malloc failed");
+ ret = -1;
+ goto out;
+ }
+
+ orig_insn = find_insn(file, special_alt->orig_sec,
+ special_alt->orig_off);
+ if (!orig_insn) {
+ WARN_FUNC("special: can't find orig instruction",
+ special_alt->orig_sec, special_alt->orig_off);
+ ret = -1;
+ goto out;
+ }
+
+ new_insn = NULL;
+ if (!special_alt->group || special_alt->new_len) {
+ new_insn = find_insn(file, special_alt->new_sec,
+ special_alt->new_off);
+ if (!new_insn) {
+ WARN_FUNC("special: can't find new instruction",
+ special_alt->new_sec,
+ special_alt->new_off);
+ ret = -1;
+ goto out;
+ }
+ }
+
+ if (special_alt->group) {
+ ret = handle_group_alt(file, special_alt, orig_insn,
+ &new_insn);
+ if (ret)
+ goto out;
+ } else if (special_alt->jump_or_nop) {
+ ret = handle_jump_alt(file, special_alt, orig_insn,
+ &new_insn);
+ if (ret)
+ goto out;
+ }
+
+ alt->insn = new_insn;
+ list_add_tail(&alt->list, &orig_insn->alts);
+
+ list_del(&special_alt->list);
+ free(special_alt);
+ }
+
+out:
+ return ret;
+}
+
+static int add_switch_table(struct objtool_file *file, struct symbol *func,
+ struct instruction *insn, struct rela *table,
+ struct rela *next_table)
+{
+ struct rela *rela = table;
+ struct instruction *alt_insn;
+ struct alternative *alt;
+
+ list_for_each_entry_from(rela, &file->rodata->rela->rela_list, list) {
+ if (rela == next_table)
+ break;
+
+ if (rela->sym->sec != insn->sec ||
+ rela->addend <= func->offset ||
+ rela->addend >= func->offset + func->len)
+ break;
+
+ alt_insn = find_insn(file, insn->sec, rela->addend);
+ if (!alt_insn) {
+ WARN("%s: can't find instruction at %s+0x%x",
+ file->rodata->rela->name, insn->sec->name,
+ rela->addend);
+ return -1;
+ }
+
+ alt = malloc(sizeof(*alt));
+ if (!alt) {
+ WARN("malloc failed");
+ return -1;
+ }
+
+ alt->insn = alt_insn;
+ list_add_tail(&alt->list, &insn->alts);
+ }
+
+ return 0;
+}
+
+/*
+ * find_switch_table() - Given a dynamic jump, find the switch jump table in
+ * .rodata associated with it.
+ *
+ * There are 3 basic patterns:
+ *
+ * 1. jmpq *[rodata addr](,%reg,8)
+ *
+ * This is the most common case by far. It jumps to an address in a simple
+ * jump table which is stored in .rodata.
+ *
+ * 2. jmpq *[rodata addr](%rip)
+ *
+ * This is caused by a rare GCC quirk, currently only seen in three driver
+ * functions in the kernel, only with certain obscure non-distro configs.
+ *
+ * As part of an optimization, GCC makes a copy of an existing switch jump
+ * table, modifies it, and then hard-codes the jump (albeit with an indirect
+ * jump) to use a single entry in the table. The rest of the jump table and
+ * some of its jump targets remain as dead code.
+ *
+ * In such a case we can just crudely ignore all unreachable instruction
+ * warnings for the entire object file. Ideally we would just ignore them
+ * for the function, but that would require redesigning the code quite a
+ * bit. And honestly that's just not worth doing: unreachable instruction
+ * warnings are of questionable value anyway, and this is such a rare issue.
+ *
+ * 3. mov [rodata addr],%reg1
+ * ... some instructions ...
+ * jmpq *(%reg1,%reg2,8)
+ *
+ * This is a fairly uncommon pattern which is new for GCC 6. As of this
+ * writing, there are 11 occurrences of it in the allmodconfig kernel.
+ *
+ * TODO: Once we have DWARF CFI and smarter instruction decoding logic,
+ * ensure the same register is used in the mov and jump instructions.
+ */
+static struct rela *find_switch_table(struct objtool_file *file,
+ struct symbol *func,
+ struct instruction *insn)
+{
+ struct rela *text_rela, *rodata_rela;
+ struct instruction *orig_insn = insn;
+
+ text_rela = find_rela_by_dest_range(insn->sec, insn->offset, insn->len);
+ if (text_rela && text_rela->sym == file->rodata->sym) {
+ /* case 1 */
+ rodata_rela = find_rela_by_dest(file->rodata,
+ text_rela->addend);
+ if (rodata_rela)
+ return rodata_rela;
+
+ /* case 2 */
+ rodata_rela = find_rela_by_dest(file->rodata,
+ text_rela->addend + 4);
+ if (!rodata_rela)
+ return NULL;
+ file->ignore_unreachables = true;
+ return rodata_rela;
+ }
+
+ /* case 3 */
+ func_for_each_insn_continue_reverse(file, func, insn) {
+ if (insn->type == INSN_JUMP_DYNAMIC)
+ break;
+
+ /* allow small jumps within the range */
+ if (insn->type == INSN_JUMP_UNCONDITIONAL &&
+ insn->jump_dest &&
+ (insn->jump_dest->offset <= insn->offset ||
+ insn->jump_dest->offset > orig_insn->offset))
+ break;
+
+ /* look for a relocation which references .rodata */
+ text_rela = find_rela_by_dest_range(insn->sec, insn->offset,
+ insn->len);
+ if (!text_rela || text_rela->sym != file->rodata->sym)
+ continue;
+
+ /*
+ * Make sure the .rodata address isn't associated with a
+ * symbol. gcc jump tables are anonymous data.
+ */
+ if (find_symbol_containing(file->rodata, text_rela->addend))
+ continue;
+
+ return find_rela_by_dest(file->rodata, text_rela->addend);
+ }
+
+ return NULL;
+}
+
+static int add_func_switch_tables(struct objtool_file *file,
+ struct symbol *func)
+{
+ struct instruction *insn, *prev_jump = NULL;
+ struct rela *rela, *prev_rela = NULL;
+ int ret;
+
+ func_for_each_insn(file, func, insn) {
+ if (insn->type != INSN_JUMP_DYNAMIC)
+ continue;
+
+ rela = find_switch_table(file, func, insn);
+ if (!rela)
+ continue;
+
+ /*
+ * We found a switch table, but we don't know yet how big it
+ * is. Don't add it until we reach the end of the function or
+ * the beginning of another switch table in the same function.
+ */
+ if (prev_jump) {
+ ret = add_switch_table(file, func, prev_jump, prev_rela,
+ rela);
+ if (ret)
+ return ret;
+ }
+
+ prev_jump = insn;
+ prev_rela = rela;
+ }
+
+ if (prev_jump) {
+ ret = add_switch_table(file, func, prev_jump, prev_rela, NULL);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+/*
+ * For some switch statements, gcc generates a jump table in the .rodata
+ * section which contains a list of addresses within the function to jump to.
+ * This finds these jump tables and adds them to the insn->alts lists.
+ */
+static int add_switch_table_alts(struct objtool_file *file)
+{
+ struct section *sec;
+ struct symbol *func;
+ int ret;
+
+ if (!file->rodata || !file->rodata->rela)
+ return 0;
+
+ for_each_sec(file, sec) {
+ list_for_each_entry(func, &sec->symbol_list, list) {
+ if (func->type != STT_FUNC)
+ continue;
+
+ ret = add_func_switch_tables(file, func);
+ if (ret)
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static int decode_sections(struct objtool_file *file)
+{
+ int ret;
+
+ ret = decode_instructions(file);
+ if (ret)
+ return ret;
+
+ ret = add_dead_ends(file);
+ if (ret)
+ return ret;
+
+ add_ignores(file);
+
+ ret = add_jump_destinations(file);
+ if (ret)
+ return ret;
+
+ ret = add_call_destinations(file);
+ if (ret)
+ return ret;
+
+ ret = add_special_section_alts(file);
+ if (ret)
+ return ret;
+
+ ret = add_switch_table_alts(file);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static bool is_fentry_call(struct instruction *insn)
+{
+ if (insn->type == INSN_CALL &&
+ insn->call_dest->type == STT_NOTYPE &&
+ !strcmp(insn->call_dest->name, "__fentry__"))
+ return true;
+
+ return false;
+}
+
+static bool has_modified_stack_frame(struct insn_state *state)
+{
+ int i;
+
+ if (state->cfa.base != initial_func_cfi.cfa.base ||
+ state->cfa.offset != initial_func_cfi.cfa.offset ||
+ state->stack_size != initial_func_cfi.cfa.offset ||
+ state->drap)
+ return true;
+
+ for (i = 0; i < CFI_NUM_REGS; i++)
+ if (state->regs[i].base != initial_func_cfi.regs[i].base ||
+ state->regs[i].offset != initial_func_cfi.regs[i].offset)
+ return true;
+
+ return false;
+}
+
+static bool has_valid_stack_frame(struct insn_state *state)
+{
+ if (state->cfa.base == CFI_BP && state->regs[CFI_BP].base == CFI_CFA &&
+ state->regs[CFI_BP].offset == -16)
+ return true;
+
+ if (state->drap && state->regs[CFI_BP].base == CFI_BP)
+ return true;
+
+ return false;
+}
+
+static void save_reg(struct insn_state *state, unsigned char reg, int base,
+ int offset)
+{
+ if ((arch_callee_saved_reg(reg) ||
+ (state->drap && reg == state->drap_reg)) &&
+ state->regs[reg].base == CFI_UNDEFINED) {
+ state->regs[reg].base = base;
+ state->regs[reg].offset = offset;
+ }
+}
+
+static void restore_reg(struct insn_state *state, unsigned char reg)
+{
+ state->regs[reg].base = CFI_UNDEFINED;
+ state->regs[reg].offset = 0;
+}
+
+/*
+ * A note about DRAP stack alignment:
+ *
+ * GCC has the concept of a DRAP register, which is used to help keep track of
+ * the stack pointer when aligning the stack. r10 or r13 is used as the DRAP
+ * register. The typical DRAP pattern is:
+ *
+ * 4c 8d 54 24 08 lea 0x8(%rsp),%r10
+ * 48 83 e4 c0 and $0xffffffffffffffc0,%rsp
+ * 41 ff 72 f8 pushq -0x8(%r10)
+ * 55 push %rbp
+ * 48 89 e5 mov %rsp,%rbp
+ * (more pushes)
+ * 41 52 push %r10
+ * ...
+ * 41 5a pop %r10
+ * (more pops)
+ * 5d pop %rbp
+ * 49 8d 62 f8 lea -0x8(%r10),%rsp
+ * c3 retq
+ *
+ * There are some variations in the epilogues, like:
+ *
+ * 5b pop %rbx
+ * 41 5a pop %r10
+ * 41 5c pop %r12
+ * 41 5d pop %r13
+ * 41 5e pop %r14
+ * c9 leaveq
+ * 49 8d 62 f8 lea -0x8(%r10),%rsp
+ * c3 retq
+ *
+ * and:
+ *
+ * 4c 8b 55 e8 mov -0x18(%rbp),%r10
+ * 48 8b 5d e0 mov -0x20(%rbp),%rbx
+ * 4c 8b 65 f0 mov -0x10(%rbp),%r12
+ * 4c 8b 6d f8 mov -0x8(%rbp),%r13
+ * c9 leaveq
+ * 49 8d 62 f8 lea -0x8(%r10),%rsp
+ * c3 retq
+ *
+ * Sometimes r13 is used as the DRAP register, in which case it's saved and
+ * restored beforehand:
+ *
+ * 41 55 push %r13
+ * 4c 8d 6c 24 10 lea 0x10(%rsp),%r13
+ * 48 83 e4 f0 and $0xfffffffffffffff0,%rsp
+ * ...
+ * 49 8d 65 f0 lea -0x10(%r13),%rsp
+ * 41 5d pop %r13
+ * c3 retq
+ */
+static int update_insn_state(struct instruction *insn, struct insn_state *state)
+{
+ struct stack_op *op = &insn->stack_op;
+ struct cfi_reg *cfa = &state->cfa;
+ struct cfi_reg *regs = state->regs;
+
+ /* stack operations don't make sense with an undefined CFA */
+ if (cfa->base == CFI_UNDEFINED) {
+ if (insn->func) {
+ WARN_FUNC("undefined stack state", insn->sec, insn->offset);
+ return -1;
+ }
+ return 0;
+ }
+
+ switch (op->dest.type) {
+
+ case OP_DEST_REG:
+ switch (op->src.type) {
+
+ case OP_SRC_REG:
+ if (cfa->base == op->src.reg && cfa->base == CFI_SP &&
+ op->dest.reg == CFI_BP && regs[CFI_BP].base == CFI_CFA &&
+ regs[CFI_BP].offset == -cfa->offset) {
+
+ /* mov %rsp, %rbp */
+ cfa->base = op->dest.reg;
+ state->bp_scratch = false;
+ } else if (state->drap) {
+
+ /* drap: mov %rsp, %rbp */
+ regs[CFI_BP].base = CFI_BP;
+ regs[CFI_BP].offset = -state->stack_size;
+ state->bp_scratch = false;
+ } else if (!nofp) {
+
+ WARN_FUNC("unknown stack-related register move",
+ insn->sec, insn->offset);
+ return -1;
+ }
+
+ break;
+
+ case OP_SRC_ADD:
+ if (op->dest.reg == CFI_SP && op->src.reg == CFI_SP) {
+
+ /* add imm, %rsp */
+ state->stack_size -= op->src.offset;
+ if (cfa->base == CFI_SP)
+ cfa->offset -= op->src.offset;
+ break;
+ }
+
+ if (op->dest.reg == CFI_SP && op->src.reg == CFI_BP) {
+
+ /* lea disp(%rbp), %rsp */
+ state->stack_size = -(op->src.offset + regs[CFI_BP].offset);
+ break;
+ }
+
+ if (op->dest.reg != CFI_BP && op->src.reg == CFI_SP &&
+ cfa->base == CFI_SP) {
+
+ /* drap: lea disp(%rsp), %drap */
+ state->drap_reg = op->dest.reg;
+ break;
+ }
+
+ if (state->drap && op->dest.reg == CFI_SP &&
+ op->src.reg == state->drap_reg) {
+
+ /* drap: lea disp(%drap), %rsp */
+ cfa->base = CFI_SP;
+ cfa->offset = state->stack_size = -op->src.offset;
+ state->drap_reg = CFI_UNDEFINED;
+ state->drap = false;
+ break;
+ }
+
+ if (op->dest.reg == state->cfa.base) {
+ WARN_FUNC("unsupported stack register modification",
+ insn->sec, insn->offset);
+ return -1;
+ }
+
+ break;
+
+ case OP_SRC_AND:
+ if (op->dest.reg != CFI_SP ||
+ (state->drap_reg != CFI_UNDEFINED && cfa->base != CFI_SP) ||
+ (state->drap_reg == CFI_UNDEFINED && cfa->base != CFI_BP)) {
+ WARN_FUNC("unsupported stack pointer realignment",
+ insn->sec, insn->offset);
+ return -1;
+ }
+
+ if (state->drap_reg != CFI_UNDEFINED) {
+ /* drap: and imm, %rsp */
+ cfa->base = state->drap_reg;
+ cfa->offset = state->stack_size = 0;
+ state->drap = true;
+
+ }
+
+ /*
+ * Older versions of GCC (4.8ish) realign the stack
+ * without DRAP, with a frame pointer.
+ */
+
+ break;
+
+ case OP_SRC_POP:
+ if (!state->drap && op->dest.type == OP_DEST_REG &&
+ op->dest.reg == cfa->base) {
+
+ /* pop %rbp */
+ cfa->base = CFI_SP;
+ }
+
+ if (regs[op->dest.reg].offset == -state->stack_size) {
+
+ if (state->drap && cfa->base == CFI_BP_INDIRECT &&
+ op->dest.type == OP_DEST_REG &&
+ op->dest.reg == state->drap_reg) {
+
+ /* drap: pop %drap */
+ cfa->base = state->drap_reg;
+ cfa->offset = 0;
+ }
+
+ restore_reg(state, op->dest.reg);
+ }
+
+ state->stack_size -= 8;
+ if (cfa->base == CFI_SP)
+ cfa->offset -= 8;
+
+ break;
+
+ case OP_SRC_REG_INDIRECT:
+ if (state->drap && op->src.reg == CFI_BP &&
+ op->src.offset == regs[op->dest.reg].offset) {
+
+ /* drap: mov disp(%rbp), %reg */
+ if (op->dest.reg == state->drap_reg) {
+ cfa->base = state->drap_reg;
+ cfa->offset = 0;
+ }
+
+ restore_reg(state, op->dest.reg);
+
+ } else if (op->src.reg == cfa->base &&
+ op->src.offset == regs[op->dest.reg].offset + cfa->offset) {
+
+ /* mov disp(%rbp), %reg */
+ /* mov disp(%rsp), %reg */
+ restore_reg(state, op->dest.reg);
+ }
+
+ break;
+
+ default:
+ WARN_FUNC("unknown stack-related instruction",
+ insn->sec, insn->offset);
+ return -1;
+ }
+
+ break;
+
+ case OP_DEST_PUSH:
+ state->stack_size += 8;
+ if (cfa->base == CFI_SP)
+ cfa->offset += 8;
+
+ if (op->src.type != OP_SRC_REG)
+ break;
+
+ if (state->drap) {
+ if (op->src.reg == cfa->base && op->src.reg == state->drap_reg) {
+
+ /* drap: push %drap */
+ cfa->base = CFI_BP_INDIRECT;
+ cfa->offset = -state->stack_size;
+
+ /* save drap so we know when to undefine it */
+ save_reg(state, op->src.reg, CFI_CFA, -state->stack_size);
+
+ } else if (op->src.reg == CFI_BP && cfa->base == state->drap_reg) {
+
+ /* drap: push %rbp */
+ state->stack_size = 0;
+
+ } else if (regs[op->src.reg].base == CFI_UNDEFINED) {
+
+ /* drap: push %reg */
+ save_reg(state, op->src.reg, CFI_BP, -state->stack_size);
+ }
+
+ } else {
+
+ /* push %reg */
+ save_reg(state, op->src.reg, CFI_CFA, -state->stack_size);
+ }
+
+ /* detect when asm code uses rbp as a scratch register */
+ if (!nofp && insn->func && op->src.reg == CFI_BP &&
+ cfa->base != CFI_BP)
+ state->bp_scratch = true;
+ break;
+
+ case OP_DEST_REG_INDIRECT:
+
+ if (state->drap) {
+ if (op->src.reg == cfa->base && op->src.reg == state->drap_reg) {
+
+ /* drap: mov %drap, disp(%rbp) */
+ cfa->base = CFI_BP_INDIRECT;
+ cfa->offset = op->dest.offset;
+
+ /* save drap so we know when to undefine it */
+ save_reg(state, op->src.reg, CFI_CFA, op->dest.offset);
+ }
+
+ else if (regs[op->src.reg].base == CFI_UNDEFINED) {
+
+ /* drap: mov reg, disp(%rbp) */
+ save_reg(state, op->src.reg, CFI_BP, op->dest.offset);
+ }
+
+ } else if (op->dest.reg == cfa->base) {
+
+ /* mov reg, disp(%rbp) */
+ /* mov reg, disp(%rsp) */
+ save_reg(state, op->src.reg, CFI_CFA,
+ op->dest.offset - state->cfa.offset);
+ }
+
+ break;
+
+ case OP_DEST_LEAVE:
+ if ((!state->drap && cfa->base != CFI_BP) ||
+ (state->drap && cfa->base != state->drap_reg)) {
+ WARN_FUNC("leave instruction with modified stack frame",
+ insn->sec, insn->offset);
+ return -1;
+ }
+
+ /* leave (mov %rbp, %rsp; pop %rbp) */
+
+ state->stack_size = -state->regs[CFI_BP].offset - 8;
+ restore_reg(state, CFI_BP);
+
+ if (!state->drap) {
+ cfa->base = CFI_SP;
+ cfa->offset -= 8;
+ }
+
+ break;
+
+ case OP_DEST_MEM:
+ if (op->src.type != OP_SRC_POP) {
+ WARN_FUNC("unknown stack-related memory operation",
+ insn->sec, insn->offset);
+ return -1;
+ }
+
+ /* pop mem */
+ state->stack_size -= 8;
+ if (cfa->base == CFI_SP)
+ cfa->offset -= 8;
+
+ break;
+
+ default:
+ WARN_FUNC("unknown stack-related instruction",
+ insn->sec, insn->offset);
+ return -1;
+ }
+
+ return 0;
+}
+
+static bool insn_state_match(struct instruction *insn, struct insn_state *state)
+{
+ struct insn_state *state1 = &insn->state, *state2 = state;
+ int i;
+
+ if (memcmp(&state1->cfa, &state2->cfa, sizeof(state1->cfa))) {
+ WARN_FUNC("stack state mismatch: cfa1=%d%+d cfa2=%d%+d",
+ insn->sec, insn->offset,
+ state1->cfa.base, state1->cfa.offset,
+ state2->cfa.base, state2->cfa.offset);
+
+ } else if (memcmp(&state1->regs, &state2->regs, sizeof(state1->regs))) {
+ for (i = 0; i < CFI_NUM_REGS; i++) {
+ if (!memcmp(&state1->regs[i], &state2->regs[i],
+ sizeof(struct cfi_reg)))
+ continue;
+
+ WARN_FUNC("stack state mismatch: reg1[%d]=%d%+d reg2[%d]=%d%+d",
+ insn->sec, insn->offset,
+ i, state1->regs[i].base, state1->regs[i].offset,
+ i, state2->regs[i].base, state2->regs[i].offset);
+ break;
+ }
+
+ } else if (state1->drap != state2->drap ||
+ (state1->drap && state1->drap_reg != state2->drap_reg)) {
+ WARN_FUNC("stack state mismatch: drap1=%d(%d) drap2=%d(%d)",
+ insn->sec, insn->offset,
+ state1->drap, state1->drap_reg,
+ state2->drap, state2->drap_reg);
+
+ } else
+ return true;
+
+ return false;
+}
+
+/*
+ * Follow the branch starting at the given instruction, and recursively follow
+ * any other branches (jumps). Meanwhile, track the frame pointer state at
+ * each instruction and validate all the rules described in
+ * tools/objtool/Documentation/stack-validation.txt.
+ */
+static int validate_branch(struct objtool_file *file, struct instruction *first,
+ struct insn_state state)
+{
+ struct alternative *alt;
+ struct instruction *insn;
+ struct section *sec;
+ struct symbol *func = NULL;
+ int ret;
+
+ insn = first;
+ sec = insn->sec;
+
+ if (insn->alt_group && list_empty(&insn->alts)) {
+ WARN_FUNC("don't know how to handle branch to middle of alternative instruction group",
+ sec, insn->offset);
+ return -1;
+ }
+
+ while (1) {
+ if (file->c_file && insn->func) {
+ if (func && func != insn->func) {
+ WARN("%s() falls through to next function %s()",
+ func->name, insn->func->name);
+ return 1;
+ }
+ }
+
+ func = insn->func;
+
+ if (insn->visited) {
+ if (!!insn_state_match(insn, &state))
+ return 1;
+
+ return 0;
+ }
+
+ insn->state = state;
+
+ insn->visited = true;
+
+ list_for_each_entry(alt, &insn->alts, list) {
+ ret = validate_branch(file, alt->insn, state);
+ if (ret)
+ return 1;
+ }
+
+ switch (insn->type) {
+
+ case INSN_RETURN:
+ if (func && has_modified_stack_frame(&state)) {
+ WARN_FUNC("return with modified stack frame",
+ sec, insn->offset);
+ return 1;
+ }
+
+ if (state.bp_scratch) {
+ WARN("%s uses BP as a scratch register",
+ insn->func->name);
+ return 1;
+ }
+
+ return 0;
+
+ case INSN_CALL:
+ if (is_fentry_call(insn))
+ break;
+
+ ret = dead_end_function(file, insn->call_dest);
+ if (ret == 1)
+ return 0;
+ if (ret == -1)
+ return 1;
+
+ /* fallthrough */
+ case INSN_CALL_DYNAMIC:
+ if (!nofp && func && !has_valid_stack_frame(&state)) {
+ WARN_FUNC("call without frame pointer save/setup",
+ sec, insn->offset);
+ return 1;
+ }
+ break;
+
+ case INSN_JUMP_CONDITIONAL:
+ case INSN_JUMP_UNCONDITIONAL:
+ if (insn->jump_dest) {
+ ret = validate_branch(file, insn->jump_dest,
+ state);
+ if (ret)
+ return 1;
+ } else if (func && has_modified_stack_frame(&state)) {
+ WARN_FUNC("sibling call from callable instruction with modified stack frame",
+ sec, insn->offset);
+ return 1;
+ } /* else it's a sibling call */
+
+ if (insn->type == INSN_JUMP_UNCONDITIONAL)
+ return 0;
+
+ break;
+
+ case INSN_JUMP_DYNAMIC:
+ if (func && list_empty(&insn->alts) &&
+ has_modified_stack_frame(&state)) {
+ WARN_FUNC("sibling call from callable instruction with modified stack frame",
+ sec, insn->offset);
+ return 1;
+ }
+
+ return 0;
+
+ case INSN_STACK:
+ if (update_insn_state(insn, &state))
+ return -1;
+
+ break;
+
+ default:
+ break;
+ }
+
+ if (insn->dead_end)
+ return 0;
+
+ insn = next_insn_same_sec(file, insn);
+ if (!insn) {
+ WARN("%s: unexpected end of section", sec->name);
+ return 1;
+ }
+ }
+
+ return 0;
+}
+
+static bool is_kasan_insn(struct instruction *insn)
+{
+ return (insn->type == INSN_CALL &&
+ !strcmp(insn->call_dest->name, "__asan_handle_no_return"));
+}
+
+static bool is_ubsan_insn(struct instruction *insn)
+{
+ return (insn->type == INSN_CALL &&
+ !strcmp(insn->call_dest->name,
+ "__ubsan_handle_builtin_unreachable"));
+}
+
+static bool ignore_unreachable_insn(struct instruction *insn)
+{
+ int i;
+
+ if (insn->ignore || insn->type == INSN_NOP)
+ return true;
+
+ /*
+ * Ignore any unused exceptions. This can happen when a whitelisted
+ * function has an exception table entry.
+ */
+ if (!strcmp(insn->sec->name, ".fixup"))
+ return true;
+
+ /*
+ * Check if this (or a subsequent) instruction is related to
+ * CONFIG_UBSAN or CONFIG_KASAN.
+ *
+ * End the search at 5 instructions to avoid going into the weeds.
+ */
+ if (!insn->func)
+ return false;
+ for (i = 0; i < 5; i++) {
+
+ if (is_kasan_insn(insn) || is_ubsan_insn(insn))
+ return true;
+
+ if (insn->type == INSN_JUMP_UNCONDITIONAL && insn->jump_dest) {
+ insn = insn->jump_dest;
+ continue;
+ }
+
+ if (insn->offset + insn->len >= insn->func->offset + insn->func->len)
+ break;
+ insn = list_next_entry(insn, list);
+ }
+
+ return false;
+}
+
+static int validate_functions(struct objtool_file *file)
+{
+ struct section *sec;
+ struct symbol *func;
+ struct instruction *insn;
+ struct insn_state state;
+ int ret, warnings = 0;
+
+ clear_insn_state(&state);
+
+ state.cfa = initial_func_cfi.cfa;
+ memcpy(&state.regs, &initial_func_cfi.regs,
+ CFI_NUM_REGS * sizeof(struct cfi_reg));
+ state.stack_size = initial_func_cfi.cfa.offset;
+
+ for_each_sec(file, sec) {
+ list_for_each_entry(func, &sec->symbol_list, list) {
+ if (func->type != STT_FUNC)
+ continue;
+
+ insn = find_insn(file, sec, func->offset);
+ if (!insn || insn->ignore)
+ continue;
+
+ ret = validate_branch(file, insn, state);
+ warnings += ret;
+ }
+ }
+
+ return warnings;
+}
+
+static int validate_reachable_instructions(struct objtool_file *file)
+{
+ struct instruction *insn;
+
+ if (file->ignore_unreachables)
+ return 0;
+
+ for_each_insn(file, insn) {
+ if (insn->visited || ignore_unreachable_insn(insn))
+ continue;
+
+ /*
+ * gcov produces a lot of unreachable instructions. If we get
+ * an unreachable warning and the file has gcov enabled, just
+ * ignore it, and all other such warnings for the file. Do
+ * this here because this is an expensive function.
+ */
+ if (gcov_enabled(file))
+ return 0;
+
+ WARN_FUNC("unreachable instruction", insn->sec, insn->offset);
+ return 1;
+ }
+
+ return 0;
+}
+
+static void cleanup(struct objtool_file *file)
+{
+ struct instruction *insn, *tmpinsn;
+ struct alternative *alt, *tmpalt;
+
+ list_for_each_entry_safe(insn, tmpinsn, &file->insn_list, list) {
+ list_for_each_entry_safe(alt, tmpalt, &insn->alts, list) {
+ list_del(&alt->list);
+ free(alt);
+ }
+ list_del(&insn->list);
+ hash_del(&insn->hash);
+ free(insn);
+ }
+ elf_close(file->elf);
+}
+
+int check(const char *_objname, bool _nofp)
+{
+ struct objtool_file file;
+ int ret, warnings = 0;
+
+ objname = _objname;
+ nofp = _nofp;
+
+ file.elf = elf_open(objname);
+ if (!file.elf)
+ return 1;
+
+ INIT_LIST_HEAD(&file.insn_list);
+ hash_init(file.insn_hash);
+ file.whitelist = find_section_by_name(file.elf, ".discard.func_stack_frame_non_standard");
+ file.rodata = find_section_by_name(file.elf, ".rodata");
+ file.ignore_unreachables = false;
+ file.c_file = find_section_by_name(file.elf, ".comment");
+
+ arch_initial_func_cfi_state(&initial_func_cfi);
+
+ ret = decode_sections(&file);
+ if (ret < 0)
+ goto out;
+ warnings += ret;
+
+ if (list_empty(&file.insn_list))
+ goto out;
+
+ ret = validate_functions(&file);
+ if (ret < 0)
+ goto out;
+ warnings += ret;
+
+ if (!warnings) {
+ ret = validate_reachable_instructions(&file);
+ if (ret < 0)
+ goto out;
+ warnings += ret;
+ }
+
+out:
+ cleanup(&file);
+
+ /* ignore warnings for now until we get all the code cleaned up */
+ if (ret || warnings)
+ return 0;
+ return 0;
+}
--- /dev/null
+/*
+ * Copyright (C) 2017 Josh Poimboeuf <jpoimboe@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef _CHECK_H
+#define _CHECK_H
+
+#include <stdbool.h>
+#include "elf.h"
+#include "cfi.h"
+#include "arch.h"
+#include <linux/hashtable.h>
+
+struct insn_state {
+ struct cfi_reg cfa;
+ struct cfi_reg regs[CFI_NUM_REGS];
+ int stack_size;
+ bool bp_scratch;
+ bool drap;
+ int drap_reg;
+};
+
+struct instruction {
+ struct list_head list;
+ struct hlist_node hash;
+ struct section *sec;
+ unsigned long offset;
+ unsigned int len;
+ unsigned char type;
+ unsigned long immediate;
+ bool alt_group, visited, dead_end, ignore;
+ struct symbol *call_dest;
+ struct instruction *jump_dest;
+ struct list_head alts;
+ struct symbol *func;
+ struct stack_op stack_op;
+ struct insn_state state;
+};
+
+struct objtool_file {
+ struct elf *elf;
+ struct list_head insn_list;
+ DECLARE_HASHTABLE(insn_hash, 16);
+ struct section *rodata, *whitelist;
+ bool ignore_unreachables, c_file;
+};
+
+int check(const char *objname, bool nofp);
+
+#define for_each_insn(file, insn) \
+ list_for_each_entry(insn, &file->insn_list, list)
+
+#endif /* _CHECK_H */
#define ELF_C_READ_MMAP ELF_C_READ
#endif
+#define WARN_ELF(format, ...) \
+ WARN(format ": %s", ##__VA_ARGS__, elf_errmsg(-1))
+
struct section *find_section_by_name(struct elf *elf, const char *name)
{
struct section *sec;
int i;
if (elf_getshdrnum(elf->elf, §ions_nr)) {
- perror("elf_getshdrnum");
+ WARN_ELF("elf_getshdrnum");
return -1;
}
if (elf_getshdrstrndx(elf->elf, &shstrndx)) {
- perror("elf_getshdrstrndx");
+ WARN_ELF("elf_getshdrstrndx");
return -1;
}
s = elf_getscn(elf->elf, i);
if (!s) {
- perror("elf_getscn");
+ WARN_ELF("elf_getscn");
return -1;
}
sec->idx = elf_ndxscn(s);
if (!gelf_getshdr(s, &sec->sh)) {
- perror("gelf_getshdr");
+ WARN_ELF("gelf_getshdr");
return -1;
}
sec->name = elf_strptr(elf->elf, shstrndx, sec->sh.sh_name);
if (!sec->name) {
- perror("elf_strptr");
+ WARN_ELF("elf_strptr");
return -1;
}
- sec->elf_data = elf_getdata(s, NULL);
- if (!sec->elf_data) {
- perror("elf_getdata");
+ sec->data = elf_getdata(s, NULL);
+ if (!sec->data) {
+ WARN_ELF("elf_getdata");
return -1;
}
- if (sec->elf_data->d_off != 0 ||
- sec->elf_data->d_size != sec->sh.sh_size) {
+ if (sec->data->d_off != 0 ||
+ sec->data->d_size != sec->sh.sh_size) {
WARN("unexpected data attributes for %s", sec->name);
return -1;
}
- sec->data = (unsigned long)sec->elf_data->d_buf;
- sec->len = sec->elf_data->d_size;
+ sec->len = sec->data->d_size;
}
/* sanity check, one more call to elf_nextscn() should return NULL */
sym->idx = i;
- if (!gelf_getsym(symtab->elf_data, i, &sym->sym)) {
- perror("gelf_getsym");
+ if (!gelf_getsym(symtab->data, i, &sym->sym)) {
+ WARN_ELF("gelf_getsym");
goto err;
}
sym->name = elf_strptr(elf->elf, symtab->sh.sh_link,
sym->sym.st_name);
if (!sym->name) {
- perror("elf_strptr");
+ WARN_ELF("elf_strptr");
goto err;
}
}
memset(rela, 0, sizeof(*rela));
- if (!gelf_getrela(sec->elf_data, i, &rela->rela)) {
- perror("gelf_getrela");
+ if (!gelf_getrela(sec->data, i, &rela->rela)) {
+ WARN_ELF("gelf_getrela");
return -1;
}
INIT_LIST_HEAD(&elf->sections);
- elf->name = strdup(name);
- if (!elf->name) {
- perror("strdup");
- goto err;
- }
-
elf->fd = open(name, O_RDONLY);
if (elf->fd == -1) {
perror("open");
elf->elf = elf_begin(elf->fd, ELF_C_READ_MMAP, NULL);
if (!elf->elf) {
- perror("elf_begin");
+ WARN_ELF("elf_begin");
goto err;
}
if (!gelf_getehdr(elf->elf, &elf->ehdr)) {
- perror("gelf_getehdr");
+ WARN_ELF("gelf_getehdr");
goto err;
}
struct symbol *sym, *tmpsym;
struct rela *rela, *tmprela;
+ if (elf->elf)
+ elf_end(elf->elf);
+
+ if (elf->fd > 0)
+ close(elf->fd);
+
list_for_each_entry_safe(sec, tmpsec, &elf->sections, list) {
list_for_each_entry_safe(sym, tmpsym, &sec->symbol_list, list) {
list_del(&sym->list);
list_del(&sec->list);
free(sec);
}
- if (elf->name)
- free(elf->name);
- if (elf->fd > 0)
- close(elf->fd);
- if (elf->elf)
- elf_end(elf->elf);
+
free(elf);
}
DECLARE_HASHTABLE(rela_hash, 16);
struct section *base, *rela;
struct symbol *sym;
- Elf_Data *elf_data;
+ Elf_Data *data;
char *name;
int idx;
- unsigned long data;
unsigned int len;
};
struct symbol *find_containing_func(struct section *sec, unsigned long offset);
void elf_close(struct elf *elf);
-
+#define for_each_sec(file, sec) \
+ list_for_each_entry(sec, &file->elf->sections, list)
#endif /* _OBJTOOL_ELF_H */
alt->jump_or_nop = entry->jump_or_nop;
if (alt->group) {
- alt->orig_len = *(unsigned char *)(sec->data + offset +
+ alt->orig_len = *(unsigned char *)(sec->data->d_buf + offset +
entry->orig_len);
- alt->new_len = *(unsigned char *)(sec->data + offset +
+ alt->new_len = *(unsigned char *)(sec->data->d_buf + offset +
entry->new_len);
}
if (entry->feature) {
unsigned short feature;
- feature = *(unsigned short *)(sec->data + offset +
+ feature = *(unsigned short *)(sec->data->d_buf + offset +
entry->feature);
/*
#ifndef _WARN_H
#define _WARN_H
+#include <stdlib.h>
+#include <string.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include "elf.h"
+
extern const char *objname;
static inline char *offstr(struct section *sec, unsigned long offset)
free(_str); \
})
+#define WARN_ELF(format, ...) \
+ WARN(format ": %s", ##__VA_ARGS__, elf_errmsg(-1))
+
#endif /* _WARN_H */
script export-to-postgresql.py for more details, and to script
call-graph-from-postgresql.py for an example of using the database.
+There is also script intel-pt-events.py which provides an example of how to
+unpack the raw data for power events and PTWRITE.
+
As mentioned above, it is easy to capture too much data. One way to limit the
data captured is to use 'snapshot' mode which is explained further below.
Refer to 'new snapshot option' and 'Intel PT modes of operation' further below.
CYC packets are not requested by default.
+pt Specifies pass-through which enables the 'branch' config term.
+
+ The default config selects 'pt' if it is available, so a user will
+ never need to specify this term.
+
+branch Enable branch tracing. Branch tracing is enabled by default so to
+ disable branch tracing use 'branch=0'.
+
+ The default config selects 'branch' if it is available.
+
+ptw Enable PTWRITE packets which are produced when a ptwrite instruction
+ is executed.
+
+ Support for this feature is indicated by:
+
+ /sys/bus/event_source/devices/intel_pt/caps/ptwrite
+
+ which contains "1" if the feature is supported and
+ "0" otherwise.
+
+fup_on_ptw Enable a FUP packet to follow the PTWRITE packet. The FUP packet
+ provides the address of the ptwrite instruction. In the absence of
+ fup_on_ptw, the decoder will use the address of the previous branch
+ if branch tracing is enabled, otherwise the address will be zero.
+ Note that fup_on_ptw will work even when branch tracing is disabled.
+
+pwr_evt Enable power events. The power events provide information about
+ changes to the CPU C-state.
+
+ Support for this feature is indicated by:
+
+ /sys/bus/event_source/devices/intel_pt/caps/power_event_trace
+
+ which contains "1" if the feature is supported and
+ "0" otherwise.
+
new snapshot option
-------------------
which, in turn, is the same as
- --itrace=ibxe
+ --itrace=ibxwpe
The letters are:
i synthesize "instructions" events
b synthesize "branches" events
x synthesize "transactions" events
+ w synthesize "ptwrite" events
+ p synthesize "power" events
c synthesize branches events (calls only)
r synthesize branches events (returns only)
e synthesize tracing error events
'flags' field can be used in perf script to determine whether the event is a
tranasaction start, commit or abort.
-Error events are new. They show where the decoder lost the trace. Error events
+Note that "instructions", "branches" and "transactions" events depend on code
+flow packets which can be disabled by using the config term "branch=0". Refer
+to the config terms section above.
+
+"ptwrite" events record the payload of the ptwrite instruction and whether
+"fup_on_ptw" was used. "ptwrite" events depend on PTWRITE packets which are
+recorded only if the "ptw" config term was used. Refer to the config terms
+section above. perf script "synth" field displays "ptwrite" information like
+this: "ip: 0 payload: 0x123456789abcdef0" where "ip" is 1 if "fup_on_ptw" was
+used.
+
+"Power" events correspond to power event packets and CBR (core-to-bus ratio)
+packets. While CBR packets are always recorded when tracing is enabled, power
+event packets are recorded only if the "pwr_evt" config term was used. Refer to
+the config terms section above. The power events record information about
+C-state changes, whereas CBR is indicative of CPU frequency. perf script
+"event,synth" fields display information like this:
+ cbr: cbr: 22 freq: 2189 MHz (200%)
+ mwait: hints: 0x60 extensions: 0x1
+ pwre: hw: 0 cstate: 2 sub-cstate: 0
+ exstop: ip: 1
+ pwrx: deepest cstate: 2 last cstate: 2 wake reason: 0x4
+Where:
+ "cbr" includes the frequency and the percentage of maximum non-turbo
+ "mwait" shows mwait hints and extensions
+ "pwre" shows C-state transitions (to a C-state deeper than C0) and
+ whether initiated by hardware
+ "exstop" indicates execution stopped and whether the IP was recorded
+ exactly,
+ "pwrx" indicates return to C0
+For more details refer to the Intel 64 and IA-32 Architectures Software
+Developer Manuals.
+
+Error events show where the decoder lost the trace. Error events
are quite important. Users must know if what they are seeing is a complete
picture or not.
c synthesize branches events (calls only)
r synthesize branches events (returns only)
x synthesize transactions events
+ w synthesize ptwrite events
+ p synthesize power events
e synthesize error events
d create a debug log
g synthesize a call chain (use with i or x)
l synthesize last branch entries (use with i or x)
s skip initial number of events
- The default is all events i.e. the same as --itrace=ibxe
+ The default is all events i.e. the same as --itrace=ibxwpe
In addition, the period (default 100000) for instructions events
can be specified in units of:
Also the number of last branch entries (default 64, max. 1024) for
instructions or transactions events can be specified.
- It is also possible to skip events generated (instructions, branches, transactions)
- at the beginning. This is useful to ignore initialization code.
+ It is also possible to skip events generated (instructions, branches, transactions,
+ ptwrite, power) at the beginning. This is useful to ignore initialization code.
--itrace=i0nss1000000
Ranges of CPUs are specified with -: 0-2.
Default is to trace on all online CPUs.
+-T::
+--trace-funcs=::
+ Only trace functions given by the argument. Multiple functions
+ can be given by using this option more than once. The function
+ argument also can be a glob pattern. It will be passed to
+ 'set_ftrace_filter' in tracefs.
+
+-N::
+--notrace-funcs=::
+ Do not trace functions given by the argument. Like -T option,
+ this can be used more than once to specify multiple functions
+ (or glob patterns). It will be passed to 'set_ftrace_notrace'
+ in tracefs.
+
+-G::
+--graph-funcs=::
+ Set graph filter on the given function (or a glob pattern).
+ This is useful for the function_graph tracer only and enables
+ tracing for functions executed from the given function.
+ This can be used more than once to specify multiple functions.
+ It will be passed to 'set_graph_function' in tracefs.
+
+-g::
+--nograph-funcs=::
+ Set graph notrace filter on the given function (or a glob pattern).
+ Like -G option, this is useful for the function_graph tracer only
+ and disables tracing for function executed from the given function.
+ This can be used more than once to specify multiple functions.
+ It will be passed to 'set_graph_notrace' in tracefs.
+
+-D::
+--graph-depth=::
+ Set max depth for function graph tracer to follow
SEE ALSO
--------
--fields::
Comma separated list of fields to print. Options are:
comm, tid, pid, time, cpu, event, trace, ip, sym, dso, addr, symoff,
- srcline, period, iregs, brstack, brstacksym, flags, bpf-output, brstackinsn,
- callindent, insn, insnlen. Field list can be prepended with the type, trace, sw or hw,
+ srcline, period, iregs, brstack, brstacksym, flags, bpf-output, brstackinsn, brstackoff,
+ callindent, insn, insnlen, synth.
+ Field list can be prepended with the type, trace, sw or hw,
to indicate to which event type the field list applies.
e.g., -F sw:comm,tid,time,ip,sym and -F trace:time,cpu,trace
i.e., the specified fields apply to all event types if the type string
is not given.
+ In addition to overriding fields, it is also possible to add or remove
+ fields from the defaults. For example
+
+ -F -cpu,+insn
+
+ removes the cpu field and adds the insn field. Adding/removing fields
+ cannot be mixed with normal overriding.
+
The arguments are processed in the order received. A later usage can
reset a prior request. e.g.:
instruction bytes and the instruction length of the current
instruction.
+ The synth field is used by synthesized events which may be created when
+ Instruction Trace decoding.
+
Finally, a user may not set fields to none for all event types.
i.e., -F "" is not allowed.
is printed. This is the full execution path leading to the sample. This is only supported when the
sample was recorded with perf record -b or -j any.
+ The brstackoff field will print an offset into a specific dso/binary.
+
-k::
--vmlinux=<file>::
vmlinux pathname
--no-merge::
Do not merge results from same PMUs.
+--smi-cost::
+Measure SMI cost if msr/aperf/ and msr/smi/ events are supported.
+
+During the measurement, the /sys/device/cpu/freeze_on_smi will be set to
+freeze core counters on SMI.
+The aperf counter will not be effected by the setting.
+The cost of SMI can be measured by (aperf - unhalted core cycles).
+
+In practice, the percentages of SMI cycles is very useful for performance
+oriented analysis. --metric_only will be applied by default.
+The output is SMI cycles%, equals to (aperf - unhalted core cycles) / aperf
+
+Users who wants to get the actual value can apply --no-metric-only.
+
EXAMPLES
--------
# Disable it on all other architectures in case libdw unwind
# support is detected in system. Add supported architectures
# to the check.
-ifneq ($(SRCARCH),$(filter $(SRCARCH),x86 arm))
+ifneq ($(SRCARCH),$(filter $(SRCARCH),x86 arm powerpc))
NO_LIBDW_DWARF_UNWIND := 1
endif
#include <api/fs/fs.h>
#include <linux/bitops.h>
+#include <linux/compiler.h>
#include <linux/coresight-pmu.h>
#include <linux/kernel.h>
#include <linux/log2.h>
pr_debug2("%s snapshot size: %zu\n", CORESIGHT_ETM_PMU_NAME,
opts->auxtrace_snapshot_size);
- if (cs_etm_evsel) {
- /*
- * To obtain the auxtrace buffer file descriptor, the auxtrace
- * event must come first.
- */
- perf_evlist__to_front(evlist, cs_etm_evsel);
- /*
- * In the case of per-cpu mmaps, we need the CPU on the
- * AUX event.
- */
- if (!cpu_map__empty(cpus))
- perf_evsel__set_sample_bit(cs_etm_evsel, CPU);
- }
+ /*
+ * To obtain the auxtrace buffer file descriptor, the auxtrace
+ * event must come first.
+ */
+ perf_evlist__to_front(evlist, cs_etm_evsel);
+
+ /*
+ * In the case of per-cpu mmaps, we need the CPU on the
+ * AUX event.
+ */
+ if (!cpu_map__empty(cpus))
+ perf_evsel__set_sample_bit(cs_etm_evsel, CPU);
/* Add dummy event to keep tracking */
if (opts->full_auxtrace) {
}
-static __attribute__((format(printf, 2, 3)))
-int cs_device__print_file(const char *name, const char *fmt, ...)
+static int __printf(2, 3) cs_device__print_file(const char *name, const char *fmt, ...)
{
va_list args;
FILE *file;
libperf-$(CONFIG_DWARF) += dwarf-regs.o
libperf-$(CONFIG_DWARF) += skip-callchain-idx.o
+
libperf-$(CONFIG_LIBUNWIND) += unwind-libunwind.o
+libperf-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
--- /dev/null
+#include <elfutils/libdwfl.h>
+#include "../../util/unwind-libdw.h"
+#include "../../util/perf_regs.h"
+#include "../../util/event.h"
+
+/* See backends/ppc_initreg.c and backends/ppc_regs.c in elfutils. */
+static const int special_regs[3][2] = {
+ { 65, PERF_REG_POWERPC_LINK },
+ { 101, PERF_REG_POWERPC_XER },
+ { 109, PERF_REG_POWERPC_CTR },
+};
+
+bool libdw__arch_set_initial_registers(Dwfl_Thread *thread, void *arg)
+{
+ struct unwind_info *ui = arg;
+ struct regs_dump *user_regs = &ui->sample->user_regs;
+ Dwarf_Word dwarf_regs[32], dwarf_nip;
+ size_t i;
+
+#define REG(r) ({ \
+ Dwarf_Word val = 0; \
+ perf_reg_value(&val, user_regs, PERF_REG_POWERPC_##r); \
+ val; \
+})
+
+ dwarf_regs[0] = REG(R0);
+ dwarf_regs[1] = REG(R1);
+ dwarf_regs[2] = REG(R2);
+ dwarf_regs[3] = REG(R3);
+ dwarf_regs[4] = REG(R4);
+ dwarf_regs[5] = REG(R5);
+ dwarf_regs[6] = REG(R6);
+ dwarf_regs[7] = REG(R7);
+ dwarf_regs[8] = REG(R8);
+ dwarf_regs[9] = REG(R9);
+ dwarf_regs[10] = REG(R10);
+ dwarf_regs[11] = REG(R11);
+ dwarf_regs[12] = REG(R12);
+ dwarf_regs[13] = REG(R13);
+ dwarf_regs[14] = REG(R14);
+ dwarf_regs[15] = REG(R15);
+ dwarf_regs[16] = REG(R16);
+ dwarf_regs[17] = REG(R17);
+ dwarf_regs[18] = REG(R18);
+ dwarf_regs[19] = REG(R19);
+ dwarf_regs[20] = REG(R20);
+ dwarf_regs[21] = REG(R21);
+ dwarf_regs[22] = REG(R22);
+ dwarf_regs[23] = REG(R23);
+ dwarf_regs[24] = REG(R24);
+ dwarf_regs[25] = REG(R25);
+ dwarf_regs[26] = REG(R26);
+ dwarf_regs[27] = REG(R27);
+ dwarf_regs[28] = REG(R28);
+ dwarf_regs[29] = REG(R29);
+ dwarf_regs[30] = REG(R30);
+ dwarf_regs[31] = REG(R31);
+ if (!dwfl_thread_state_registers(thread, 0, 32, dwarf_regs))
+ return false;
+
+ dwarf_nip = REG(NIP);
+ dwfl_thread_state_register_pc(thread, dwarf_nip);
+ for (i = 0; i < ARRAY_SIZE(special_regs); i++) {
+ Dwarf_Word val = 0;
+ perf_reg_value(&val, user_regs, special_regs[i][1]);
+ if (!dwfl_thread_state_registers(thread,
+ special_regs[i][0], 1,
+ &val))
+ return false;
+ }
+
+ return true;
+}
"0f c7 1d 78 56 34 12 \txrstors 0x12345678",},
{{0x0f, 0xc7, 0x9c, 0xc8, 0x78, 0x56, 0x34, 0x12, }, 8, 0, "", "",
"0f c7 9c c8 78 56 34 12 \txrstors 0x12345678(%eax,%ecx,8)",},
+{{0xf3, 0x0f, 0xae, 0x20, }, 4, 0, "", "",
+"f3 0f ae 20 \tptwritel (%eax)",},
+{{0xf3, 0x0f, 0xae, 0x25, 0x78, 0x56, 0x34, 0x12, }, 8, 0, "", "",
+"f3 0f ae 25 78 56 34 12 \tptwritel 0x12345678",},
+{{0xf3, 0x0f, 0xae, 0xa4, 0xc8, 0x78, 0x56, 0x34, 0x12, }, 9, 0, "", "",
+"f3 0f ae a4 c8 78 56 34 12 \tptwritel 0x12345678(%eax,%ecx,8)",},
+{{0xf3, 0x0f, 0xae, 0x20, }, 4, 0, "", "",
+"f3 0f ae 20 \tptwritel (%eax)",},
+{{0xf3, 0x0f, 0xae, 0x25, 0x78, 0x56, 0x34, 0x12, }, 8, 0, "", "",
+"f3 0f ae 25 78 56 34 12 \tptwritel 0x12345678",},
+{{0xf3, 0x0f, 0xae, 0xa4, 0xc8, 0x78, 0x56, 0x34, 0x12, }, 9, 0, "", "",
+"f3 0f ae a4 c8 78 56 34 12 \tptwritel 0x12345678(%eax,%ecx,8)",},
"0f c7 9c c8 78 56 34 12 \txrstors 0x12345678(%rax,%rcx,8)",},
{{0x41, 0x0f, 0xc7, 0x9c, 0xc8, 0x78, 0x56, 0x34, 0x12, }, 9, 0, "", "",
"41 0f c7 9c c8 78 56 34 12 \txrstors 0x12345678(%r8,%rcx,8)",},
+{{0xf3, 0x0f, 0xae, 0x20, }, 4, 0, "", "",
+"f3 0f ae 20 \tptwritel (%rax)",},
+{{0xf3, 0x41, 0x0f, 0xae, 0x20, }, 5, 0, "", "",
+"f3 41 0f ae 20 \tptwritel (%r8)",},
+{{0xf3, 0x0f, 0xae, 0x24, 0x25, 0x78, 0x56, 0x34, 0x12, }, 9, 0, "", "",
+"f3 0f ae 24 25 78 56 34 12 \tptwritel 0x12345678",},
+{{0xf3, 0x0f, 0xae, 0xa4, 0xc8, 0x78, 0x56, 0x34, 0x12, }, 9, 0, "", "",
+"f3 0f ae a4 c8 78 56 34 12 \tptwritel 0x12345678(%rax,%rcx,8)",},
+{{0xf3, 0x41, 0x0f, 0xae, 0xa4, 0xc8, 0x78, 0x56, 0x34, 0x12, }, 10, 0, "", "",
+"f3 41 0f ae a4 c8 78 56 34 12 \tptwritel 0x12345678(%r8,%rcx,8)",},
+{{0xf3, 0x0f, 0xae, 0x20, }, 4, 0, "", "",
+"f3 0f ae 20 \tptwritel (%rax)",},
+{{0xf3, 0x41, 0x0f, 0xae, 0x20, }, 5, 0, "", "",
+"f3 41 0f ae 20 \tptwritel (%r8)",},
+{{0xf3, 0x0f, 0xae, 0x24, 0x25, 0x78, 0x56, 0x34, 0x12, }, 9, 0, "", "",
+"f3 0f ae 24 25 78 56 34 12 \tptwritel 0x12345678",},
+{{0xf3, 0x0f, 0xae, 0xa4, 0xc8, 0x78, 0x56, 0x34, 0x12, }, 9, 0, "", "",
+"f3 0f ae a4 c8 78 56 34 12 \tptwritel 0x12345678(%rax,%rcx,8)",},
+{{0xf3, 0x41, 0x0f, 0xae, 0xa4, 0xc8, 0x78, 0x56, 0x34, 0x12, }, 10, 0, "", "",
+"f3 41 0f ae a4 c8 78 56 34 12 \tptwritel 0x12345678(%r8,%rcx,8)",},
+{{0xf3, 0x48, 0x0f, 0xae, 0x20, }, 5, 0, "", "",
+"f3 48 0f ae 20 \tptwriteq (%rax)",},
+{{0xf3, 0x49, 0x0f, 0xae, 0x20, }, 5, 0, "", "",
+"f3 49 0f ae 20 \tptwriteq (%r8)",},
+{{0xf3, 0x48, 0x0f, 0xae, 0x24, 0x25, 0x78, 0x56, 0x34, 0x12, }, 10, 0, "", "",
+"f3 48 0f ae 24 25 78 56 34 12 \tptwriteq 0x12345678",},
+{{0xf3, 0x48, 0x0f, 0xae, 0xa4, 0xc8, 0x78, 0x56, 0x34, 0x12, }, 10, 0, "", "",
+"f3 48 0f ae a4 c8 78 56 34 12 \tptwriteq 0x12345678(%rax,%rcx,8)",},
+{{0xf3, 0x49, 0x0f, 0xae, 0xa4, 0xc8, 0x78, 0x56, 0x34, 0x12, }, 10, 0, "", "",
+"f3 49 0f ae a4 c8 78 56 34 12 \tptwriteq 0x12345678(%r8,%rcx,8)",},
asm volatile("xrstors 0x12345678(%rax,%rcx,8)");
asm volatile("xrstors 0x12345678(%r8,%rcx,8)");
+ /* ptwrite */
+
+ asm volatile("ptwrite (%rax)");
+ asm volatile("ptwrite (%r8)");
+ asm volatile("ptwrite (0x12345678)");
+ asm volatile("ptwrite 0x12345678(%rax,%rcx,8)");
+ asm volatile("ptwrite 0x12345678(%r8,%rcx,8)");
+
+ asm volatile("ptwritel (%rax)");
+ asm volatile("ptwritel (%r8)");
+ asm volatile("ptwritel (0x12345678)");
+ asm volatile("ptwritel 0x12345678(%rax,%rcx,8)");
+ asm volatile("ptwritel 0x12345678(%r8,%rcx,8)");
+
+ asm volatile("ptwriteq (%rax)");
+ asm volatile("ptwriteq (%r8)");
+ asm volatile("ptwriteq (0x12345678)");
+ asm volatile("ptwriteq 0x12345678(%rax,%rcx,8)");
+ asm volatile("ptwriteq 0x12345678(%r8,%rcx,8)");
+
#else /* #ifdef __x86_64__ */
/* bound r32, mem (same op code as EVEX prefix) */
asm volatile("xrstors (0x12345678)");
asm volatile("xrstors 0x12345678(%eax,%ecx,8)");
+ /* ptwrite */
+
+ asm volatile("ptwrite (%eax)");
+ asm volatile("ptwrite (0x12345678)");
+ asm volatile("ptwrite 0x12345678(%eax,%ecx,8)");
+
+ asm volatile("ptwritel (%eax)");
+ asm volatile("ptwritel (0x12345678)");
+ asm volatile("ptwritel 0x12345678(%eax,%ecx,8)");
+
#endif /* #ifndef __x86_64__ */
/* Following line is a marker for the awk script - do not change */
#define KiB_MASK(x) (KiB(x) - 1)
#define MiB_MASK(x) (MiB(x) - 1)
-#define INTEL_BTS_DFLT_SAMPLE_SIZE KiB(4)
-
-#define INTEL_BTS_MAX_SAMPLE_SIZE KiB(60)
-
struct intel_bts_snapshot_ref {
void *ref_buf;
size_t ref_offset;
#define KiB_MASK(x) (KiB(x) - 1)
#define MiB_MASK(x) (MiB(x) - 1)
-#define INTEL_PT_DEFAULT_SAMPLE_SIZE KiB(4)
-
-#define INTEL_PT_MAX_SAMPLE_SIZE KiB(60)
-
#define INTEL_PT_PSB_PERIOD_NEAR 256
struct intel_pt_snapshot_ref {
int psb_cyc, psb_periods, psb_period;
int pos = 0;
u64 config;
+ char c;
pos += scnprintf(buf + pos, sizeof(buf) - pos, "tsc");
}
}
+ if (perf_pmu__scan_file(intel_pt_pmu, "format/pt", "%c", &c) == 1 &&
+ perf_pmu__scan_file(intel_pt_pmu, "format/branch", "%c", &c) == 1)
+ pos += scnprintf(buf + pos, sizeof(buf) - pos, ",pt,branch");
+
pr_debug2("%s default config: %s\n", intel_pt_pmu->name, buf);
intel_pt_parse_terms(&intel_pt_pmu->format, buf, &config);
* kernel (KSM, zero page, etc.) cannot optimize away RAM
* accesses:
*/
-static inline u64 access_data(u64 *data __attribute__((unused)), u64 val)
+static inline u64 access_data(u64 *data, u64 val)
{
if (g->p.data_reads)
val += *data;
tok; tok = strtok_r(NULL, ", ", &tmp)) { \
ret = _fn(hpp_list, tok); \
if (ret == -EINVAL) { \
- error("Invalid --fields key: `%s'", tok); \
+ pr_err("Invalid --fields key: `%s'", tok); \
break; \
} else if (ret == -ESRCH) { \
- error("Unknown --fields key: `%s'", tok); \
+ pr_err("Unknown --fields key: `%s'", tok); \
break; \
} \
} \
int cmd_config(int argc, const char **argv)
{
- int i, ret = 0;
+ int i, ret = -1;
struct perf_config_set *set;
char *user_config = mkpath("%s/.perfconfig", getenv("HOME"));
const char *config_filename;
* because of reinitializing with options config file location.
*/
set = perf_config_set__new();
- if (!set) {
- ret = -1;
+ if (!set)
goto out_err;
- }
switch (actions) {
case ACTION_LIST:
pr_err("Error: takes no arguments\n");
parse_options_usage(config_usage, config_options, "l", 1);
} else {
- ret = show_config(set);
- if (ret < 0)
+ if (show_config(set) < 0) {
pr_err("Nothing configured, "
"please check your %s \n", config_filename);
+ goto out_err;
+ }
}
break;
default:
- if (argc) {
- for (i = 0; argv[i]; i++) {
- char *var, *value;
- char *arg = strdup(argv[i]);
-
- if (!arg) {
- pr_err("%s: strdup failed\n", __func__);
- ret = -1;
- break;
- }
+ if (!argc) {
+ usage_with_options(config_usage, config_options);
+ break;
+ }
- if (parse_config_arg(arg, &var, &value) < 0) {
- free(arg);
- ret = -1;
- break;
- }
+ for (i = 0; argv[i]; i++) {
+ char *var, *value;
+ char *arg = strdup(argv[i]);
+
+ if (!arg) {
+ pr_err("%s: strdup failed\n", __func__);
+ goto out_err;
+ }
- if (value == NULL)
- ret = show_spec_config(set, var);
- else
- ret = set_config(set, config_filename, var, value);
+ if (parse_config_arg(arg, &var, &value) < 0) {
free(arg);
+ goto out_err;
}
- } else
- usage_with_options(config_usage, config_options);
+
+ if (value == NULL) {
+ if (show_spec_config(set, var) < 0) {
+ pr_err("%s is not configured: %s\n",
+ var, config_filename);
+ free(arg);
+ goto out_err;
+ }
+ } else {
+ if (set_config(set, config_filename, var, value) < 0) {
+ pr_err("Failed to set '%s=%s' on %s\n",
+ var, value, config_filename);
+ free(arg);
+ goto out_err;
+ }
+ }
+ free(arg);
+ }
}
- perf_config_set__delete(set);
+ ret = 0;
out_err:
+ perf_config_set__delete(set);
return ret;
}
void *cb __maybe_unused)
{
if (!strcmp(var, "diff.order")) {
- sort_compute = perf_config_int(var, value);
+ int ret;
+ if (perf_config_int(&ret, var, value) < 0)
+ return -1;
+ sort_compute = ret;
return 0;
}
if (!strcmp(var, "diff.compute")) {
#define DEFAULT_TRACER "function_graph"
struct perf_ftrace {
- struct perf_evlist *evlist;
- struct target target;
- const char *tracer;
+ struct perf_evlist *evlist;
+ struct target target;
+ const char *tracer;
+ struct list_head filters;
+ struct list_head notrace;
+ struct list_head graph_funcs;
+ struct list_head nograph_funcs;
+ int graph_depth;
+};
+
+struct filter_entry {
+ struct list_head list;
+ char name[];
};
static bool done;
int fd, ret = -1;
ssize_t size = strlen(val);
int flags = O_WRONLY;
+ char errbuf[512];
file = get_tracing_file(name);
if (!file) {
fd = open(file, flags);
if (fd < 0) {
- pr_debug("cannot open tracing file: %s\n", name);
+ pr_debug("cannot open tracing file: %s: %s\n",
+ name, str_error_r(errno, errbuf, sizeof(errbuf)));
goto out;
}
if (write(fd, val, size) == size)
ret = 0;
else
- pr_debug("write '%s' to tracing/%s failed\n", val, name);
+ pr_debug("write '%s' to tracing/%s failed: %s\n",
+ val, name, str_error_r(errno, errbuf, sizeof(errbuf)));
close(fd);
out:
}
static int reset_tracing_cpu(void);
+static void reset_tracing_filters(void);
static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
{
if (reset_tracing_cpu() < 0)
return -1;
+ if (write_tracing_file("max_graph_depth", "0") < 0)
+ return -1;
+
+ reset_tracing_filters();
return 0;
}
return ret;
}
+static int __set_tracing_filter(const char *filter_file, struct list_head *funcs)
+{
+ struct filter_entry *pos;
+
+ list_for_each_entry(pos, funcs, list) {
+ if (append_tracing_file(filter_file, pos->name) < 0)
+ return -1;
+ }
+
+ return 0;
+}
+
+static int set_tracing_filters(struct perf_ftrace *ftrace)
+{
+ int ret;
+
+ ret = __set_tracing_filter("set_ftrace_filter", &ftrace->filters);
+ if (ret < 0)
+ return ret;
+
+ ret = __set_tracing_filter("set_ftrace_notrace", &ftrace->notrace);
+ if (ret < 0)
+ return ret;
+
+ ret = __set_tracing_filter("set_graph_function", &ftrace->graph_funcs);
+ if (ret < 0)
+ return ret;
+
+ /* old kernels do not have this filter */
+ __set_tracing_filter("set_graph_notrace", &ftrace->nograph_funcs);
+
+ return ret;
+}
+
+static void reset_tracing_filters(void)
+{
+ write_tracing_file("set_ftrace_filter", " ");
+ write_tracing_file("set_ftrace_notrace", " ");
+ write_tracing_file("set_graph_function", " ");
+ write_tracing_file("set_graph_notrace", " ");
+}
+
+static int set_tracing_depth(struct perf_ftrace *ftrace)
+{
+ char buf[16];
+
+ if (ftrace->graph_depth == 0)
+ return 0;
+
+ if (ftrace->graph_depth < 0) {
+ pr_err("invalid graph depth: %d\n", ftrace->graph_depth);
+ return -1;
+ }
+
+ snprintf(buf, sizeof(buf), "%d", ftrace->graph_depth);
+
+ if (write_tracing_file("max_graph_depth", buf) < 0)
+ return -1;
+
+ return 0;
+}
+
static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
{
char *trace_file;
goto out_reset;
}
+ if (set_tracing_filters(ftrace) < 0) {
+ pr_err("failed to set tracing filters\n");
+ goto out_reset;
+ }
+
+ if (set_tracing_depth(ftrace) < 0) {
+ pr_err("failed to set graph depth\n");
+ goto out_reset;
+ }
+
if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
goto out_reset;
}
+ setup_pager();
+
trace_file = get_tracing_file("trace_pipe");
if (!trace_file) {
pr_err("failed to open trace_pipe\n");
goto out_close_fd;
}
- setup_pager();
-
perf_evlist__start_workload(ftrace->evlist);
while (!done) {
return -1;
}
+static int parse_filter_func(const struct option *opt, const char *str,
+ int unset __maybe_unused)
+{
+ struct list_head *head = opt->value;
+ struct filter_entry *entry;
+
+ entry = malloc(sizeof(*entry) + strlen(str) + 1);
+ if (entry == NULL)
+ return -ENOMEM;
+
+ strcpy(entry->name, str);
+ list_add_tail(&entry->list, head);
+
+ return 0;
+}
+
+static void delete_filter_func(struct list_head *head)
+{
+ struct filter_entry *pos, *tmp;
+
+ list_for_each_entry_safe(pos, tmp, head, list) {
+ list_del(&pos->list);
+ free(pos);
+ }
+}
+
int cmd_ftrace(int argc, const char **argv)
{
int ret;
"system-wide collection from all CPUs"),
OPT_STRING('C', "cpu", &ftrace.target.cpu_list, "cpu",
"list of cpus to monitor"),
+ OPT_CALLBACK('T', "trace-funcs", &ftrace.filters, "func",
+ "trace given functions only", parse_filter_func),
+ OPT_CALLBACK('N', "notrace-funcs", &ftrace.notrace, "func",
+ "do not trace given functions", parse_filter_func),
+ OPT_CALLBACK('G', "graph-funcs", &ftrace.graph_funcs, "func",
+ "Set graph filter on given functions", parse_filter_func),
+ OPT_CALLBACK('g', "nograph-funcs", &ftrace.nograph_funcs, "func",
+ "Set nograph filter on given functions", parse_filter_func),
+ OPT_INTEGER('D', "graph-depth", &ftrace.graph_depth,
+ "Max depth for function graph tracer"),
OPT_END()
};
+ INIT_LIST_HEAD(&ftrace.filters);
+ INIT_LIST_HEAD(&ftrace.notrace);
+ INIT_LIST_HEAD(&ftrace.graph_funcs);
+ INIT_LIST_HEAD(&ftrace.nograph_funcs);
+
ret = perf_config(perf_ftrace_config, &ftrace);
if (ret < 0)
return -1;
target__strerror(&ftrace.target, ret, errbuf, 512);
pr_err("%s\n", errbuf);
- return -EINVAL;
+ goto out_delete_filters;
}
ftrace.evlist = perf_evlist__new();
- if (ftrace.evlist == NULL)
- return -ENOMEM;
+ if (ftrace.evlist == NULL) {
+ ret = -ENOMEM;
+ goto out_delete_filters;
+ }
ret = perf_evlist__create_maps(ftrace.evlist, &ftrace.target);
if (ret < 0)
out_delete_evlist:
perf_evlist__delete(ftrace.evlist);
+out_delete_filters:
+ delete_filter_func(&ftrace.filters);
+ delete_filter_func(&ftrace.notrace);
+ delete_filter_func(&ftrace.graph_funcs);
+ delete_filter_func(&ftrace.nograph_funcs);
+
return ret;
}
return ret;
}
-static void exec_woman_emacs(const char *path, const char *page)
+static void exec_failed(const char *cmd)
{
char sbuf[STRERR_BUFSIZE];
+ pr_warning("failed to exec '%s': %s", cmd, str_error_r(errno, sbuf, sizeof(sbuf)));
+}
+static void exec_woman_emacs(const char *path, const char *page)
+{
if (!check_emacsclient_version()) {
/* This works only with emacsclient version >= 22. */
char *man_page;
execlp(path, "emacsclient", "-e", man_page, NULL);
free(man_page);
}
- warning("failed to exec '%s': %s", path,
- str_error_r(errno, sbuf, sizeof(sbuf)));
+ exec_failed(path);
}
}
if (display && *display) {
char *man_page;
const char *filename = "kfmclient";
- char sbuf[STRERR_BUFSIZE];
/* It's simpler to launch konqueror using kfmclient. */
if (path) {
execlp(path, filename, "newTab", man_page, NULL);
free(man_page);
}
- warning("failed to exec '%s': %s", path,
- str_error_r(errno, sbuf, sizeof(sbuf)));
+ exec_failed(path);
}
}
static void exec_man_man(const char *path, const char *page)
{
- char sbuf[STRERR_BUFSIZE];
-
if (!path)
path = "man";
execlp(path, "man", page, NULL);
- warning("failed to exec '%s': %s", path,
- str_error_r(errno, sbuf, sizeof(sbuf)));
+ exec_failed(path);
}
static void exec_man_cmd(const char *cmd, const char *page)
{
- char sbuf[STRERR_BUFSIZE];
char *shell_cmd;
if (asprintf(&shell_cmd, "%s %s", cmd, page) > 0) {
execl("/bin/sh", "sh", "-c", shell_cmd, NULL);
free(shell_cmd);
}
- warning("failed to exec '%s': %s", cmd,
- str_error_r(errno, sbuf, sizeof(sbuf)));
+ exec_failed(cmd);
}
static void add_man_viewer(const char *name)
man_viewer_info_list = new;
}
+static void unsupported_man_viewer(const char *name, const char *var)
+{
+ pr_warning("'%s': path for unsupported man viewer.\n"
+ "Please consider using 'man.<tool>.%s' instead.", name, var);
+}
+
static int add_man_viewer_path(const char *name,
size_t len,
const char *value)
if (supported_man_viewer(name, len))
do_add_man_viewer_info(name, len, value);
else
- warning("'%s': path for unsupported man viewer.\n"
- "Please consider using 'man.<tool>.cmd' instead.",
- name);
+ unsupported_man_viewer(name, "cmd");
return 0;
}
const char *value)
{
if (supported_man_viewer(name, len))
- warning("'%s': cmd for supported man viewer.\n"
- "Please consider using 'man.<tool>.path' instead.",
- name);
+ unsupported_man_viewer(name, "path");
else
do_add_man_viewer_info(name, len, value);
const char *name = var + 4;
const char *subkey = strrchr(name, '.');
- if (!subkey)
- return error("Config with no key for man viewer: %s", name);
+ if (!subkey) {
+ pr_err("Config with no key for man viewer: %s", name);
+ return -1;
+ }
if (!strcmp(subkey, ".path")) {
if (!value)
return add_man_viewer_cmd(name, subkey - name, value);
}
- warning("'%s': unsupported man viewer sub key.", subkey);
+ pr_warning("'%s': unsupported man viewer sub key.", subkey);
return 0;
}
setenv("MANPATH", new_path, 1);
free(new_path);
} else {
- error("Unable to setup man path");
+ pr_err("Unable to setup man path");
}
}
else if (info)
exec_man_cmd(info, page);
else
- warning("'%s': unknown man viewer.", name);
+ pr_warning("'%s': unknown man viewer.", name);
}
static int show_man_page(const char *perf_cmd)
if (!tok)
break;
if (slab_sort_dimension__add(tok, sort_list) < 0) {
- error("Unknown slab --sort key: '%s'", tok);
+ pr_err("Unknown slab --sort key: '%s'", tok);
free(str);
return -1;
}
if (!tok)
break;
if (page_sort_dimension__add(tok, sort_list) < 0) {
- error("Unknown page --sort key: '%s'", tok);
+ pr_err("Unknown page --sort key: '%s'", tok);
free(str);
return -1;
}
}
if (perf_evlist__apply_filters(evlist, &pos)) {
- error("failed to set filter \"%s\" on event %s with %d (%s)\n",
+ pr_err("failed to set filter \"%s\" on event %s with %d (%s)\n",
pos->filter, perf_evsel__name(pos), errno,
str_error_r(errno, msg, sizeof(msg)));
rc = -1;
}
if (perf_evlist__apply_drv_configs(evlist, &pos, &err_term)) {
- error("failed to set config \"%s\" on event %s with %d (%s)\n",
+ pr_err("failed to set config \"%s\" on event %s with %d (%s)\n",
err_term->val.drv_cfg, perf_evsel__name(pos), errno,
str_error_r(errno, msg, sizeof(msg)));
rc = -1;
symbol_conf.cumulate_callchain = perf_config_bool(var, value);
return 0;
}
- if (!strcmp(var, "report.queue-size")) {
- rep->queue_size = perf_config_u64(var, value);
- return 0;
- }
+ if (!strcmp(var, "report.queue-size"))
+ return perf_config_u64(&rep->queue_size, var, value);
+
if (!strcmp(var, "report.sort_order")) {
default_sort_order = strdup(value);
return 0;
ui__error("failed to set cpu bitmap\n");
return ret;
}
+ session->itrace_synth_opts->cpu_bitmap = rep->cpu_bitmap;
}
if (rep->show_threads) {
if (thread__resolve_callchain(thread, cursor, evsel, sample,
NULL, NULL, sched->max_stack + 2) != 0) {
if (verbose > 0)
- error("Failed to resolve callchain. Skipping\n");
+ pr_err("Failed to resolve callchain. Skipping\n");
return;
}
PERF_OUTPUT_INSN = 1U << 21,
PERF_OUTPUT_INSNLEN = 1U << 22,
PERF_OUTPUT_BRSTACKINSN = 1U << 23,
+ PERF_OUTPUT_BRSTACKOFF = 1U << 24,
+ PERF_OUTPUT_SYNTH = 1U << 25,
};
struct output_option {
{.str = "insn", .field = PERF_OUTPUT_INSN},
{.str = "insnlen", .field = PERF_OUTPUT_INSNLEN},
{.str = "brstackinsn", .field = PERF_OUTPUT_BRSTACKINSN},
+ {.str = "brstackoff", .field = PERF_OUTPUT_BRSTACKOFF},
+ {.str = "synth", .field = PERF_OUTPUT_SYNTH},
+};
+
+enum {
+ OUTPUT_TYPE_SYNTH = PERF_TYPE_MAX,
+ OUTPUT_TYPE_MAX
};
/* default set to maintain compatibility with current format */
unsigned int print_ip_opts;
u64 fields;
u64 invalid_fields;
-} output[PERF_TYPE_MAX] = {
+} output[OUTPUT_TYPE_MAX] = {
[PERF_TYPE_HARDWARE] = {
.user_set = false,
.invalid_fields = PERF_OUTPUT_TRACE | PERF_OUTPUT_BPF_OUTPUT,
},
+
+ [OUTPUT_TYPE_SYNTH] = {
+ .user_set = false,
+
+ .fields = PERF_OUTPUT_COMM | PERF_OUTPUT_TID |
+ PERF_OUTPUT_CPU | PERF_OUTPUT_TIME |
+ PERF_OUTPUT_EVNAME | PERF_OUTPUT_IP |
+ PERF_OUTPUT_SYM | PERF_OUTPUT_DSO |
+ PERF_OUTPUT_SYNTH,
+
+ .invalid_fields = PERF_OUTPUT_TRACE | PERF_OUTPUT_BPF_OUTPUT,
+ },
};
+static inline int output_type(unsigned int type)
+{
+ switch (type) {
+ case PERF_TYPE_SYNTH:
+ return OUTPUT_TYPE_SYNTH;
+ default:
+ return type;
+ }
+}
+
+static inline unsigned int attr_type(unsigned int type)
+{
+ switch (type) {
+ case OUTPUT_TYPE_SYNTH:
+ return PERF_TYPE_SYNTH;
+ default:
+ return type;
+ }
+}
+
static bool output_set_by_user(void)
{
int j;
- for (j = 0; j < PERF_TYPE_MAX; ++j) {
+ for (j = 0; j < OUTPUT_TYPE_MAX; ++j) {
if (output[j].user_set)
return true;
}
return str;
}
-#define PRINT_FIELD(x) (output[attr->type].fields & PERF_OUTPUT_##x)
+#define PRINT_FIELD(x) (output[output_type(attr->type)].fields & PERF_OUTPUT_##x)
static int perf_evsel__do_check_stype(struct perf_evsel *evsel,
u64 sample_type, const char *sample_msg,
bool allow_user_set)
{
struct perf_event_attr *attr = &evsel->attr;
- int type = attr->type;
+ int type = output_type(attr->type);
const char *evname;
if (attr->sample_type & sample_type)
"selected.\n");
return -EINVAL;
}
- if (PRINT_FIELD(DSO) && !PRINT_FIELD(IP) && !PRINT_FIELD(ADDR)) {
- pr_err("Display of DSO requested but neither sample IP nor "
- "sample address\nis selected. Hence, no addresses to convert "
- "to DSO.\n");
+ if (PRINT_FIELD(DSO) && !PRINT_FIELD(IP) && !PRINT_FIELD(ADDR) &&
+ !PRINT_FIELD(BRSTACK) && !PRINT_FIELD(BRSTACKSYM) && !PRINT_FIELD(BRSTACKOFF)) {
+ pr_err("Display of DSO requested but no address to convert. Select\n"
+ "sample IP, sample address, brstack, brstacksym, or brstackoff.\n");
return -EINVAL;
}
if (PRINT_FIELD(SRCLINE) && !PRINT_FIELD(IP)) {
static void set_print_ip_opts(struct perf_event_attr *attr)
{
- unsigned int type = attr->type;
+ unsigned int type = output_type(attr->type);
output[type].print_ip_opts = 0;
if (PRINT_FIELD(IP))
unsigned int j;
struct perf_evsel *evsel;
- for (j = 0; j < PERF_TYPE_MAX; ++j) {
- evsel = perf_session__find_first_evtype(session, j);
+ for (j = 0; j < OUTPUT_TYPE_MAX; ++j) {
+ evsel = perf_session__find_first_evtype(session, attr_type(j));
/*
* even if fields is set to 0 (ie., show nothing) event must
* exist if user explicitly includes it on the command line
*/
- if (!evsel && output[j].user_set && !output[j].wildcard_set) {
+ if (!evsel && output[j].user_set && !output[j].wildcard_set &&
+ j != OUTPUT_TYPE_SYNTH) {
pr_err("%s events do not exist. "
- "Remove corresponding -f option to proceed.\n",
+ "Remove corresponding -F option to proceed.\n",
event_type(j));
return -1;
}
return br->flags.predicted ? 'P' : 'M';
}
-static void print_sample_brstack(struct perf_sample *sample)
+static void print_sample_brstack(struct perf_sample *sample,
+ struct thread *thread,
+ struct perf_event_attr *attr)
{
struct branch_stack *br = sample->branch_stack;
- u64 i;
+ struct addr_location alf, alt;
+ u64 i, from, to;
if (!(br && br->nr))
return;
for (i = 0; i < br->nr; i++) {
- printf(" 0x%"PRIx64"/0x%"PRIx64"/%c/%c/%c/%d ",
- br->entries[i].from,
- br->entries[i].to,
+ from = br->entries[i].from;
+ to = br->entries[i].to;
+
+ if (PRINT_FIELD(DSO)) {
+ memset(&alf, 0, sizeof(alf));
+ memset(&alt, 0, sizeof(alt));
+ thread__find_addr_map(thread, sample->cpumode, MAP__FUNCTION, from, &alf);
+ thread__find_addr_map(thread, sample->cpumode, MAP__FUNCTION, to, &alt);
+ }
+
+ printf("0x%"PRIx64, from);
+ if (PRINT_FIELD(DSO)) {
+ printf("(");
+ map__fprintf_dsoname(alf.map, stdout);
+ printf(")");
+ }
+
+ printf("/0x%"PRIx64, to);
+ if (PRINT_FIELD(DSO)) {
+ printf("(");
+ map__fprintf_dsoname(alt.map, stdout);
+ printf(")");
+ }
+
+ printf("/%c/%c/%c/%d ",
mispred_str( br->entries + i),
br->entries[i].flags.in_tx? 'X' : '-',
br->entries[i].flags.abort? 'A' : '-',
}
static void print_sample_brstacksym(struct perf_sample *sample,
- struct thread *thread)
+ struct thread *thread,
+ struct perf_event_attr *attr)
{
struct branch_stack *br = sample->branch_stack;
struct addr_location alf, alt;
alt.sym = map__find_symbol(alt.map, alt.addr);
symbol__fprintf_symname_offs(alf.sym, &alf, stdout);
+ if (PRINT_FIELD(DSO)) {
+ printf("(");
+ map__fprintf_dsoname(alf.map, stdout);
+ printf(")");
+ }
putchar('/');
symbol__fprintf_symname_offs(alt.sym, &alt, stdout);
+ if (PRINT_FIELD(DSO)) {
+ printf("(");
+ map__fprintf_dsoname(alt.map, stdout);
+ printf(")");
+ }
printf("/%c/%c/%c/%d ",
mispred_str( br->entries + i),
br->entries[i].flags.in_tx? 'X' : '-',
}
}
+static void print_sample_brstackoff(struct perf_sample *sample,
+ struct thread *thread,
+ struct perf_event_attr *attr)
+{
+ struct branch_stack *br = sample->branch_stack;
+ struct addr_location alf, alt;
+ u64 i, from, to;
+
+ if (!(br && br->nr))
+ return;
+
+ for (i = 0; i < br->nr; i++) {
+
+ memset(&alf, 0, sizeof(alf));
+ memset(&alt, 0, sizeof(alt));
+ from = br->entries[i].from;
+ to = br->entries[i].to;
+
+ thread__find_addr_map(thread, sample->cpumode, MAP__FUNCTION, from, &alf);
+ if (alf.map && !alf.map->dso->adjust_symbols)
+ from = map__map_ip(alf.map, from);
+
+ thread__find_addr_map(thread, sample->cpumode, MAP__FUNCTION, to, &alt);
+ if (alt.map && !alt.map->dso->adjust_symbols)
+ to = map__map_ip(alt.map, to);
+
+ printf("0x%"PRIx64, from);
+ if (PRINT_FIELD(DSO)) {
+ printf("(");
+ map__fprintf_dsoname(alf.map, stdout);
+ printf(")");
+ }
+ printf("/0x%"PRIx64, to);
+ if (PRINT_FIELD(DSO)) {
+ printf("(");
+ map__fprintf_dsoname(alt.map, stdout);
+ printf(")");
+ }
+ printf("/%c/%c/%c/%d ",
+ mispred_str(br->entries + i),
+ br->entries[i].flags.in_tx ? 'X' : '-',
+ br->entries[i].flags.abort ? 'A' : '-',
+ br->entries[i].flags.cycles);
+ }
+}
#define MAXBB 16384UL
static int grab_bb(u8 *buffer, u64 start, u64 end,
struct machine *machine)
{
struct perf_event_attr *attr = &evsel->attr;
+ unsigned int type = output_type(attr->type);
bool print_srcline_last = false;
if (PRINT_FIELD(CALLINDENT))
/* print branch_from information */
if (PRINT_FIELD(IP)) {
- unsigned int print_opts = output[attr->type].print_ip_opts;
+ unsigned int print_opts = output[type].print_ip_opts;
struct callchain_cursor *cursor = NULL;
if (symbol_conf.use_callchain && sample->callchain &&
/* print branch_to information */
if (PRINT_FIELD(ADDR) ||
((evsel->attr.sample_type & PERF_SAMPLE_ADDR) &&
- !output[attr->type].user_set)) {
+ !output[type].user_set)) {
printf(" => ");
print_sample_addr(sample, thread, attr);
}
(char *)(sample->raw_data));
}
+static void print_sample_spacing(int len, int spacing)
+{
+ if (len > 0 && len < spacing)
+ printf("%*s", spacing - len, "");
+}
+
+static void print_sample_pt_spacing(int len)
+{
+ print_sample_spacing(len, 34);
+}
+
+static void print_sample_synth_ptwrite(struct perf_sample *sample)
+{
+ struct perf_synth_intel_ptwrite *data = perf_sample__synth_ptr(sample);
+ int len;
+
+ if (perf_sample__bad_synth_size(sample, *data))
+ return;
+
+ len = printf(" IP: %u payload: %#" PRIx64 " ",
+ data->ip, le64_to_cpu(data->payload));
+ print_sample_pt_spacing(len);
+}
+
+static void print_sample_synth_mwait(struct perf_sample *sample)
+{
+ struct perf_synth_intel_mwait *data = perf_sample__synth_ptr(sample);
+ int len;
+
+ if (perf_sample__bad_synth_size(sample, *data))
+ return;
+
+ len = printf(" hints: %#x extensions: %#x ",
+ data->hints, data->extensions);
+ print_sample_pt_spacing(len);
+}
+
+static void print_sample_synth_pwre(struct perf_sample *sample)
+{
+ struct perf_synth_intel_pwre *data = perf_sample__synth_ptr(sample);
+ int len;
+
+ if (perf_sample__bad_synth_size(sample, *data))
+ return;
+
+ len = printf(" hw: %u cstate: %u sub-cstate: %u ",
+ data->hw, data->cstate, data->subcstate);
+ print_sample_pt_spacing(len);
+}
+
+static void print_sample_synth_exstop(struct perf_sample *sample)
+{
+ struct perf_synth_intel_exstop *data = perf_sample__synth_ptr(sample);
+ int len;
+
+ if (perf_sample__bad_synth_size(sample, *data))
+ return;
+
+ len = printf(" IP: %u ", data->ip);
+ print_sample_pt_spacing(len);
+}
+
+static void print_sample_synth_pwrx(struct perf_sample *sample)
+{
+ struct perf_synth_intel_pwrx *data = perf_sample__synth_ptr(sample);
+ int len;
+
+ if (perf_sample__bad_synth_size(sample, *data))
+ return;
+
+ len = printf(" deepest cstate: %u last cstate: %u wake reason: %#x ",
+ data->deepest_cstate, data->last_cstate,
+ data->wake_reason);
+ print_sample_pt_spacing(len);
+}
+
+static void print_sample_synth_cbr(struct perf_sample *sample)
+{
+ struct perf_synth_intel_cbr *data = perf_sample__synth_ptr(sample);
+ unsigned int percent, freq;
+ int len;
+
+ if (perf_sample__bad_synth_size(sample, *data))
+ return;
+
+ freq = (le32_to_cpu(data->freq) + 500) / 1000;
+ len = printf(" cbr: %2u freq: %4u MHz ", data->cbr, freq);
+ if (data->max_nonturbo) {
+ percent = (5 + (1000 * data->cbr) / data->max_nonturbo) / 10;
+ len += printf("(%3u%%) ", percent);
+ }
+ print_sample_pt_spacing(len);
+}
+
+static void print_sample_synth(struct perf_sample *sample,
+ struct perf_evsel *evsel)
+{
+ switch (evsel->attr.config) {
+ case PERF_SYNTH_INTEL_PTWRITE:
+ print_sample_synth_ptwrite(sample);
+ break;
+ case PERF_SYNTH_INTEL_MWAIT:
+ print_sample_synth_mwait(sample);
+ break;
+ case PERF_SYNTH_INTEL_PWRE:
+ print_sample_synth_pwre(sample);
+ break;
+ case PERF_SYNTH_INTEL_EXSTOP:
+ print_sample_synth_exstop(sample);
+ break;
+ case PERF_SYNTH_INTEL_PWRX:
+ print_sample_synth_pwrx(sample);
+ break;
+ case PERF_SYNTH_INTEL_CBR:
+ print_sample_synth_cbr(sample);
+ break;
+ default:
+ break;
+ }
+}
+
struct perf_script {
struct perf_tool tool;
struct perf_session *session;
{
struct thread *thread = al->thread;
struct perf_event_attr *attr = &evsel->attr;
+ unsigned int type = output_type(attr->type);
- if (output[attr->type].fields == 0)
+ if (output[type].fields == 0)
return;
print_sample_start(sample, thread, evsel);
if (PRINT_FIELD(TRACE))
event_format__print(evsel->tp_format, sample->cpu,
sample->raw_data, sample->raw_size);
+
+ if (attr->type == PERF_TYPE_SYNTH && PRINT_FIELD(SYNTH))
+ print_sample_synth(sample, evsel);
+
if (PRINT_FIELD(ADDR))
print_sample_addr(sample, thread, attr);
cursor = &callchain_cursor;
putchar(cursor ? '\n' : ' ');
- sample__fprintf_sym(sample, al, 0, output[attr->type].print_ip_opts, cursor, stdout);
+ sample__fprintf_sym(sample, al, 0, output[type].print_ip_opts, cursor, stdout);
}
if (PRINT_FIELD(IREGS))
print_sample_iregs(sample, attr);
if (PRINT_FIELD(BRSTACK))
- print_sample_brstack(sample);
+ print_sample_brstack(sample, thread, attr);
else if (PRINT_FIELD(BRSTACKSYM))
- print_sample_brstacksym(sample, thread);
+ print_sample_brstacksym(sample, thread, attr);
+ else if (PRINT_FIELD(BRSTACKOFF))
+ print_sample_brstackoff(sample, thread, attr);
if (perf_evsel__is_bpf_output(evsel) && PRINT_FIELD(BPF_OUTPUT))
print_sample_bpf_output(sample);
evlist = *pevlist;
evsel = perf_evlist__last(*pevlist);
- if (evsel->attr.type >= PERF_TYPE_MAX)
+ if (evsel->attr.type >= PERF_TYPE_MAX &&
+ evsel->attr.type != PERF_TYPE_SYNTH)
return 0;
evlist__for_each_entry(evlist, pos) {
int rc = 0;
char *str = strdup(arg);
int type = -1;
+ enum { DEFAULT, SET, ADD, REMOVE } change = DEFAULT;
if (!str)
return -ENOMEM;
type = PERF_TYPE_RAW;
else if (!strcmp(str, "break"))
type = PERF_TYPE_BREAKPOINT;
+ else if (!strcmp(str, "synth"))
+ type = OUTPUT_TYPE_SYNTH;
else {
fprintf(stderr, "Invalid event type in field string.\n");
rc = -EINVAL;
goto out;
}
+ /* Don't override defaults for +- */
+ if (strchr(str, '+') || strchr(str, '-'))
+ goto parse;
+
if (output_set_by_user())
pr_warning("Overriding previous field request for all events.\n");
- for (j = 0; j < PERF_TYPE_MAX; ++j) {
+ for (j = 0; j < OUTPUT_TYPE_MAX; ++j) {
output[j].fields = 0;
output[j].user_set = true;
output[j].wildcard_set = true;
}
}
+parse:
for (tok = strtok_r(tok, ",", &strtok_saveptr); tok; tok = strtok_r(NULL, ",", &strtok_saveptr)) {
+ if (*tok == '+') {
+ if (change == SET)
+ goto out_badmix;
+ change = ADD;
+ tok++;
+ } else if (*tok == '-') {
+ if (change == SET)
+ goto out_badmix;
+ change = REMOVE;
+ tok++;
+ } else {
+ if (change != SET && change != DEFAULT)
+ goto out_badmix;
+ change = SET;
+ }
+
for (i = 0; i < imax; ++i) {
if (strcmp(tok, all_output_options[i].str) == 0)
break;
}
if (i == imax && strcmp(tok, "flags") == 0) {
- print_flags = true;
+ print_flags = change == REMOVE ? false : true;
continue;
}
if (i == imax) {
/* add user option to all events types for
* which it is valid
*/
- for (j = 0; j < PERF_TYPE_MAX; ++j) {
+ for (j = 0; j < OUTPUT_TYPE_MAX; ++j) {
if (output[j].invalid_fields & all_output_options[i].field) {
pr_warning("\'%s\' not valid for %s events. Ignoring.\n",
all_output_options[i].str, event_type(j));
- } else
- output[j].fields |= all_output_options[i].field;
+ } else {
+ if (change == REMOVE)
+ output[j].fields &= ~all_output_options[i].field;
+ else
+ output[j].fields |= all_output_options[i].field;
+ }
}
} else {
if (output[type].invalid_fields & all_output_options[i].field) {
"Events will not be displayed.\n", event_type(type));
}
}
+ goto out;
+out_badmix:
+ fprintf(stderr, "Cannot mix +-field with overridden fields\n");
+ rc = -EINVAL;
out:
free(str);
return rc;
symbol__config_symfs),
OPT_CALLBACK('F', "fields", NULL, "str",
"comma separated output fields prepend with 'type:'. "
- "Valid types: hw,sw,trace,raw. "
+ "+field to add and -field to remove."
+ "Valid types: hw,sw,trace,raw,synth. "
"Fields: comm,tid,pid,time,cpu,event,trace,ip,sym,dso,"
"addr,symoff,period,iregs,brstack,brstacksym,flags,"
- "bpf-output,callindent,insn,insnlen,brstackinsn",
+ "bpf-output,callindent,insn,insnlen,brstackinsn,synth",
parse_output_fields),
OPT_BOOLEAN('a', "all-cpus", &system_wide,
"system-wide collection from all CPUs"),
err = perf_session__cpu_bitmap(session, cpu_list, cpu_bitmap);
if (err < 0)
goto out_delete;
+ itrace_synth_opts.cpu_bitmap = cpu_bitmap;
}
if (!no_callchain)
#define DEFAULT_SEPARATOR " "
#define CNTR_NOT_SUPPORTED "<not supported>"
#define CNTR_NOT_COUNTED "<not counted>"
+#define FREEZE_ON_SMI_PATH "devices/cpu/freeze_on_smi"
static void print_counters(struct timespec *ts, int argc, const char **argv);
NULL,
};
+static const char *smi_cost_attrs = {
+ "{"
+ "msr/aperf/,"
+ "msr/smi/,"
+ "cycles"
+ "}"
+};
+
static struct perf_evlist *evsel_list;
static struct target target = {
static int detailed_run = 0;
static bool transaction_run;
static bool topdown_run = false;
+static bool smi_cost = false;
+static bool smi_reset = false;
static bool big_num = true;
static int big_num_opt = -1;
static const char *csv_sep = NULL;
}
if (perf_evlist__apply_filters(evsel_list, &counter)) {
- error("failed to set filter \"%s\" on event %s with %d (%s)\n",
+ pr_err("failed to set filter \"%s\" on event %s with %d (%s)\n",
counter->filter, perf_evsel__name(counter), errno,
str_error_r(errno, msg, sizeof(msg)));
return -1;
}
if (perf_evlist__apply_drv_configs(evsel_list, &counter, &err_term)) {
- error("failed to set config \"%s\" on event %s with %d (%s)\n",
+ pr_err("failed to set config \"%s\" on event %s with %d (%s)\n",
err_term->val.drv_cfg, perf_evsel__name(counter), errno,
str_error_r(errno, msg, sizeof(msg)));
return -1;
"Only print computed metrics. No raw values", enable_metric_only),
OPT_BOOLEAN(0, "topdown", &topdown_run,
"measure topdown level 1 statistics"),
+ OPT_BOOLEAN(0, "smi-cost", &smi_cost,
+ "measure SMI cost"),
OPT_END()
};
return 0;
}
+ if (smi_cost) {
+ int smi;
+
+ if (sysfs__read_int(FREEZE_ON_SMI_PATH, &smi) < 0) {
+ fprintf(stderr, "freeze_on_smi is not supported.\n");
+ return -1;
+ }
+
+ if (!smi) {
+ if (sysfs__write_int(FREEZE_ON_SMI_PATH, 1) < 0) {
+ fprintf(stderr, "Failed to set freeze_on_smi.\n");
+ return -1;
+ }
+ smi_reset = true;
+ }
+
+ if (pmu_have_event("msr", "aperf") &&
+ pmu_have_event("msr", "smi")) {
+ if (!force_metric_only)
+ metric_only = true;
+ err = parse_events(evsel_list, smi_cost_attrs, NULL);
+ } else {
+ fprintf(stderr, "To measure SMI cost, it needs "
+ "msr/aperf/, msr/smi/ and cpu/cycles/ support\n");
+ return -1;
+ }
+ if (err) {
+ fprintf(stderr, "Cannot set up SMI cost events\n");
+ return -1;
+ }
+ return 0;
+ }
+
if (topdown_run) {
char *str = NULL;
bool warn = false;
perf_stat__exit_aggr_mode();
perf_evlist__free_stats(evsel_list);
out:
+ if (smi_cost && smi_reset)
+ sysfs__write_int(FREEZE_ON_SMI_PATH, 0);
+
perf_evlist__delete(evsel_list);
return status;
}
return err;
}
- err = symbol__disassemble(sym, map, NULL, 0);
+ err = symbol__disassemble(sym, map, NULL, 0, NULL);
if (err == 0) {
out_assign:
top->sym_filter_entry = he;
ret = perf_evlist__apply_drv_configs(evlist, &pos, &err_term);
if (ret) {
- error("failed to set config \"%s\" on event %s with %d (%s)\n",
+ pr_err("failed to set config \"%s\" on event %s with %d (%s)\n",
err_term->val.drv_cfg, perf_evsel__name(pos), errno,
str_error_r(errno, msg, sizeof(msg)));
goto out_delete;
FILE *fp = agent;
if (!fp) {
- warnx("jvmti: incalid fd in close_agent");
+ warnx("jvmti: invalid fd in close_agent");
return -1;
}
#include <stdint.h>
#include <jvmti.h>
-#define __unused __attribute__((unused))
-
#if defined(__cplusplus)
extern "C" {
#endif
+#include <linux/compiler.h>
#include <sys/types.h>
#include <stdio.h>
#include <string.h>
}
JNIEXPORT jint JNICALL
-Agent_OnLoad(JavaVM *jvm, char *options, void *reserved __unused)
+Agent_OnLoad(JavaVM *jvm, char *options, void *reserved __maybe_unused)
{
jvmtiEventCallbacks cb;
jvmtiCapabilities caps1;
}
JNIEXPORT void JNICALL
-Agent_OnUnload(JavaVM *jvm __unused)
+Agent_OnUnload(JavaVM *jvm __maybe_unused)
{
int ret;
#include "json.h"
#include "jevents.h"
-#ifndef __maybe_unused
-#define __maybe_unused __attribute__((unused))
-#endif
-
int verbose;
char *prog;
--- /dev/null
+#!/bin/bash
+
+#
+# print Intel PT Power Events and PTWRITE. The intel_pt PMU event needs
+# to be specified with appropriate config terms.
+#
+if ! echo "$@" | grep -q intel_pt ; then
+ echo "Options must include the Intel PT event e.g. -e intel_pt/pwr_evt,ptw/"
+ echo "and for power events it probably needs to be system wide i.e. -a option"
+ echo "For example: -a -e intel_pt/pwr_evt,branch=0/ sleep 1"
+ exit 1
+fi
+perf record $@
--- /dev/null
+#!/bin/bash
+# description: print Intel PT Power Events and PTWRITE
+perf script $@ -s "$PERF_EXEC_PATH"/scripts/python/intel-pt-events.py
\ No newline at end of file
--- /dev/null
+# intel-pt-events.py: Print Intel PT Power Events and PTWRITE
+# Copyright (c) 2017, Intel Corporation.
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms and conditions of the GNU General Public License,
+# version 2, as published by the Free Software Foundation.
+#
+# This program is distributed in the hope it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+# more details.
+
+import os
+import sys
+import struct
+
+sys.path.append(os.environ['PERF_EXEC_PATH'] + \
+ '/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
+
+# These perf imports are not used at present
+#from perf_trace_context import *
+#from Core import *
+
+def trace_begin():
+ print "Intel PT Power Events and PTWRITE"
+
+def trace_end():
+ print "End"
+
+def trace_unhandled(event_name, context, event_fields_dict):
+ print ' '.join(['%s=%s'%(k,str(v))for k,v in sorted(event_fields_dict.items())])
+
+def print_ptwrite(raw_buf):
+ data = struct.unpack_from("<IQ", raw_buf)
+ flags = data[0]
+ payload = data[1]
+ exact_ip = flags & 1
+ print "IP: %u payload: %#x" % (exact_ip, payload),
+
+def print_cbr(raw_buf):
+ data = struct.unpack_from("<BBBBII", raw_buf)
+ cbr = data[0]
+ f = (data[4] + 500) / 1000
+ p = ((cbr * 1000 / data[2]) + 5) / 10
+ print "%3u freq: %4u MHz (%3u%%)" % (cbr, f, p),
+
+def print_mwait(raw_buf):
+ data = struct.unpack_from("<IQ", raw_buf)
+ payload = data[1]
+ hints = payload & 0xff
+ extensions = (payload >> 32) & 0x3
+ print "hints: %#x extensions: %#x" % (hints, extensions),
+
+def print_pwre(raw_buf):
+ data = struct.unpack_from("<IQ", raw_buf)
+ payload = data[1]
+ hw = (payload >> 7) & 1
+ cstate = (payload >> 12) & 0xf
+ subcstate = (payload >> 8) & 0xf
+ print "hw: %u cstate: %u sub-cstate: %u" % (hw, cstate, subcstate),
+
+def print_exstop(raw_buf):
+ data = struct.unpack_from("<I", raw_buf)
+ flags = data[0]
+ exact_ip = flags & 1
+ print "IP: %u" % (exact_ip),
+
+def print_pwrx(raw_buf):
+ data = struct.unpack_from("<IQ", raw_buf)
+ payload = data[1]
+ deepest_cstate = payload & 0xf
+ last_cstate = (payload >> 4) & 0xf
+ wake_reason = (payload >> 8) & 0xf
+ print "deepest cstate: %u last cstate: %u wake reason: %#x" % (deepest_cstate, last_cstate, wake_reason),
+
+def print_common_start(comm, sample, name):
+ ts = sample["time"]
+ cpu = sample["cpu"]
+ pid = sample["pid"]
+ tid = sample["tid"]
+ print "%16s %5u/%-5u [%03u] %9u.%09u %7s:" % (comm, pid, tid, cpu, ts / 1000000000, ts %1000000000, name),
+
+def print_common_ip(sample, symbol, dso):
+ ip = sample["ip"]
+ print "%16x %s (%s)" % (ip, symbol, dso)
+
+def process_event(param_dict):
+ event_attr = param_dict["attr"]
+ sample = param_dict["sample"]
+ raw_buf = param_dict["raw_buf"]
+ comm = param_dict["comm"]
+ name = param_dict["ev_name"]
+
+ # Symbol and dso info are not always resolved
+ if (param_dict.has_key("dso")):
+ dso = param_dict["dso"]
+ else:
+ dso = "[unknown]"
+
+ if (param_dict.has_key("symbol")):
+ symbol = param_dict["symbol"]
+ else:
+ symbol = "[unknown]"
+
+ if name == "ptwrite":
+ print_common_start(comm, sample, name)
+ print_ptwrite(raw_buf)
+ print_common_ip(sample, symbol, dso)
+ elif name == "cbr":
+ print_common_start(comm, sample, name)
+ print_cbr(raw_buf)
+ print_common_ip(sample, symbol, dso)
+ elif name == "mwait":
+ print_common_start(comm, sample, name)
+ print_mwait(raw_buf)
+ print_common_ip(sample, symbol, dso)
+ elif name == "pwre":
+ print_common_start(comm, sample, name)
+ print_pwre(raw_buf)
+ print_common_ip(sample, symbol, dso)
+ elif name == "exstop":
+ print_common_start(comm, sample, name)
+ print_exstop(raw_buf)
+ print_common_ip(sample, symbol, dso)
+ elif name == "pwrx":
+ print_common_start(comm, sample, name)
+ print_pwrx(raw_buf)
+ print_common_ip(sample, symbol, dso)
* permissions. All the event text files are stored there.
*/
+#include <debug.h>
#include <errno.h>
#include <inttypes.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <unistd.h>
#include "../perf.h"
-#include "util.h"
#include <subcmd/exec-cmd.h>
#include "tests.h"
#define ENV "PERF_TEST_ATTR"
-extern int verbose;
-
static char *dir;
void test_attr__init(void)
{
int errno_saved = errno;
- if (store_event(attr, pid, cpu, fd, group_fd, flags))
- die("test attr FAILED");
+ if (store_event(attr, pid, cpu, fd, group_fd, flags)) {
+ pr_err("test attr FAILED");
+ exit(128);
+ }
errno = errno_saved;
}
def getMsg(self):
return '\'%s\' - %s' % (self.test.path, self.msg)
+class Notest(Exception):
+ def __init__(self, test, arch):
+ self.arch = arch
+ self.test = test
+ def getMsg(self):
+ return '[%s] \'%s\'' % (self.arch, self.test.path)
+
class Unsup(Exception):
def __init__(self, test):
self.test = test
# 'command' - perf command name
# 'args' - special command arguments
# 'ret' - expected command return value (0 by default)
+# 'arch' - architecture specific test (optional)
+# comma separated list, ! at the beginning
+# negates it.
#
# [eventX:base]
# - one or multiple instances in file
except:
self.ret = 0
+ try:
+ self.arch = parser.get('config', 'arch')
+ log.warning("test limitation '%s'" % self.arch)
+ except:
+ self.arch = ''
+
self.expect = {}
self.result = {}
log.debug(" loading expected events");
else:
return True
+ def skip_test(self, myarch):
+ # If architecture not set always run test
+ if self.arch == '':
+ # log.warning("test for arch %s is ok" % myarch)
+ return False
+
+ # Allow multiple values in assignment separated by ','
+ arch_list = self.arch.split(',')
+
+ # Handle negated list such as !s390x,ppc
+ if arch_list[0][0] == '!':
+ arch_list[0] = arch_list[0][1:]
+ log.warning("excluded architecture list %s" % arch_list)
+ for arch_item in arch_list:
+ # log.warning("test for %s arch is %s" % (arch_item, myarch))
+ if arch_item == myarch:
+ return True
+ return False
+
+ for arch_item in arch_list:
+ # log.warning("test for architecture '%s' current '%s'" % (arch_item, myarch))
+ if arch_item == myarch:
+ return False
+ return True
+
def load_events(self, path, events):
parser_event = ConfigParser.SafeConfigParser()
parser_event.read(path)
events[section] = e
def run_cmd(self, tempdir):
+ junk1, junk2, junk3, junk4, myarch = (os.uname())
+
+ if self.skip_test(myarch):
+ raise Notest(self, myarch)
+
cmd = "PERF_TEST_ATTR=%s %s %s -o %s/perf.data %s" % (tempdir,
self.perf, self.command, tempdir, self.args)
ret = os.WEXITSTATUS(os.system(cmd))
Test(f, options).run()
except Unsup, obj:
log.warning("unsupp %s" % obj.getMsg())
+ except Notest, obj:
+ log.warning("skipped %s" % obj.getMsg())
def setup_log(verbose):
global log
}
#endif
-__attribute__ ((noinline))
-static int test_function(void)
+static noinline int test_function(void)
{
__test_function(&the_var);
the_var++;
static int overflows;
-__attribute__ ((noinline))
-static int test_function(void)
+static noinline int test_function(void)
{
return time(NULL);
}
#include <uapi/linux/fs.h>
+/*
+ * If CONFIG_PROFILE_ALL_BRANCHES is selected,
+ * 'if' is redefined after include kernel header.
+ * Recover 'if' for BPF object code.
+ */
+#ifdef if
+# undef if
+#endif
+
#define FMODE_READ 0x1
#define FMODE_WRITE 0x2
return strcmp((const char *) symbol, funcs[idx]);
}
-__attribute__ ((noinline))
-static int unwind_thread(struct thread *thread)
+static noinline int unwind_thread(struct thread *thread)
{
struct perf_sample sample;
unsigned long cnt = 0;
static int global_unwind_retval = -INT_MAX;
-__attribute__ ((noinline))
-static int compare(void *p1, void *p2)
+static noinline int compare(void *p1, void *p2)
{
/* Any possible value should be 'thread' */
struct thread *thread = *(struct thread **)p1;
return p1 - p2;
}
-__attribute__ ((noinline))
-static int krava_3(struct thread *thread)
+static noinline int krava_3(struct thread *thread)
{
struct thread *array[2] = {thread, thread};
void *fp = &bsearch;
return global_unwind_retval;
}
-__attribute__ ((noinline))
-static int krava_2(struct thread *thread)
+static noinline int krava_2(struct thread *thread)
{
return krava_3(thread);
}
-__attribute__ ((noinline))
-static int krava_1(struct thread *thread)
+static noinline int krava_1(struct thread *thread)
{
return krava_2(thread);
}
return ret;
}
-static void debug_warn(const char *warn, va_list params)
-{
- char msg[1024];
-
- if (verbose <= 0)
- return;
-
- vsnprintf(msg, sizeof(msg), warn, params);
- fprintf(stderr, " Warning: %s\n", msg);
-}
-
int test__parse_events(int subtest __maybe_unused)
{
int ret1, ret2 = 0;
ret2 = ret1; \
} while (0)
- set_warning_routine(debug_warn);
-
TEST_EVENTS(test__events);
if (test_pmu())
.jump_arrows = true,
};
+struct arch;
+
struct annotate_browser {
struct ui_browser b;
struct rb_root entries;
struct rb_node *curr_hot;
struct disasm_line *selection;
struct disasm_line **offsets;
+ struct arch *arch;
int nr_events;
u64 start;
int nr_asm_entries;
int i, pcnt_width = annotate_browser__pcnt_width(ab);
double percent_max = 0.0;
char bf[256];
+ bool show_title = false;
for (i = 0; i < ab->nr_events; i++) {
if (bdl->samples[i].percent > percent_max)
percent_max = bdl->samples[i].percent;
}
+ if ((row == 0) && (dl->offset == -1 || percent_max == 0.0)) {
+ if (ab->have_cycles) {
+ if (dl->ipc == 0.0 && dl->cycles == 0)
+ show_title = true;
+ } else
+ show_title = true;
+ }
+
if (dl->offset != -1 && percent_max != 0.0) {
- if (percent_max != 0.0) {
- for (i = 0; i < ab->nr_events; i++) {
- ui_browser__set_percent_color(browser,
- bdl->samples[i].percent,
- current_entry);
- if (annotate_browser__opts.show_total_period) {
- ui_browser__printf(browser, "%6" PRIu64 " ",
- bdl->samples[i].nr);
- } else {
- ui_browser__printf(browser, "%6.2f ",
- bdl->samples[i].percent);
- }
+ for (i = 0; i < ab->nr_events; i++) {
+ ui_browser__set_percent_color(browser,
+ bdl->samples[i].percent,
+ current_entry);
+ if (annotate_browser__opts.show_total_period) {
+ ui_browser__printf(browser, "%6" PRIu64 " ",
+ bdl->samples[i].nr);
+ } else {
+ ui_browser__printf(browser, "%6.2f ",
+ bdl->samples[i].percent);
}
- } else {
- ui_browser__write_nstring(browser, " ", 7 * ab->nr_events);
}
} else {
ui_browser__set_percent_color(browser, 0, current_entry);
- ui_browser__write_nstring(browser, " ", 7 * ab->nr_events);
+
+ if (!show_title)
+ ui_browser__write_nstring(browser, " ", 7 * ab->nr_events);
+ else
+ ui_browser__printf(browser, "%*s", 7, "Percent");
}
if (ab->have_cycles) {
if (dl->ipc)
ui_browser__printf(browser, "%*.2f ", IPC_WIDTH - 1, dl->ipc);
- else
+ else if (!show_title)
ui_browser__write_nstring(browser, " ", IPC_WIDTH);
+ else
+ ui_browser__printf(browser, "%*s ", IPC_WIDTH - 1, "IPC");
+
if (dl->cycles)
ui_browser__printf(browser, "%*" PRIu64 " ",
CYCLES_WIDTH - 1, dl->cycles);
- else
+ else if (!show_title)
ui_browser__write_nstring(browser, " ", CYCLES_WIDTH);
+ else
+ ui_browser__printf(browser, "%*s ", CYCLES_WIDTH - 1, "Cycle");
}
SLsmg_write_char(' ');
(nr_pcnt - 1);
}
- err = symbol__disassemble(sym, map, perf_evsel__env_arch(evsel), sizeof_bdl);
+ err = symbol__disassemble(sym, map, perf_evsel__env_arch(evsel),
+ sizeof_bdl, &browser.arch);
if (err) {
char msg[BUFSIZ];
symbol__strerror_disassemble(sym, map, err, msg, sizeof(msg));
if (map->dso->annotate_warned)
return -1;
- err = symbol__disassemble(sym, map, perf_evsel__env_arch(evsel), 0);
+ err = symbol__disassemble(sym, map, perf_evsel__env_arch(evsel),
+ 0, NULL);
if (err) {
char msg[BUFSIZ];
symbol__strerror_disassemble(sym, map, err, msg, sizeof(msg));
return normalize_arch((char *)arch_name);
}
-int symbol__disassemble(struct symbol *sym, struct map *map, const char *arch_name, size_t privsize)
+int symbol__disassemble(struct symbol *sym, struct map *map,
+ const char *arch_name, size_t privsize,
+ struct arch **parch)
{
struct dso *dso = map->dso;
char command[PATH_MAX * 2];
if (arch == NULL)
return -ENOTSUP;
+ if (parch)
+ *parch = arch;
+
if (arch->init) {
err = arch->init(arch);
if (err) {
struct rb_root source_line = RB_ROOT;
u64 len;
- if (symbol__disassemble(sym, map, perf_evsel__env_arch(evsel), 0) < 0)
+ if (symbol__disassemble(sym, map, perf_evsel__env_arch(evsel),
+ 0, NULL) < 0)
return -1;
len = symbol__size(sym);
int symbol__alloc_hist(struct symbol *sym);
void symbol__annotate_zero_histograms(struct symbol *sym);
-int symbol__disassemble(struct symbol *sym, struct map *map, const char *arch_name, size_t privsize);
+int symbol__disassemble(struct symbol *sym, struct map *map,
+ const char *arch_name, size_t privsize,
+ struct arch **parch);
enum symbol_disassemble_errno {
SYMBOL_ANNOTATE_ERRNO__SUCCESS = 0,
return auxtrace_queues__add_buffer(queues, idx, buffer);
}
+static bool filter_cpu(struct perf_session *session, int cpu)
+{
+ unsigned long *cpu_bitmap = session->itrace_synth_opts->cpu_bitmap;
+
+ return cpu_bitmap && cpu != -1 && !test_bit(cpu, cpu_bitmap);
+}
+
int auxtrace_queues__add_event(struct auxtrace_queues *queues,
struct perf_session *session,
union perf_event *event, off_t data_offset,
unsigned int idx;
int err;
+ if (filter_cpu(session, event->auxtrace.cpu))
+ return 0;
+
buffer = zalloc(sizeof(struct auxtrace_buffer));
if (!buffer)
return -ENOMEM;
synth_opts->instructions = true;
synth_opts->branches = true;
synth_opts->transactions = true;
+ synth_opts->ptwrites = true;
+ synth_opts->pwr_events = true;
synth_opts->errors = true;
synth_opts->period_type = PERF_ITRACE_DEFAULT_PERIOD_TYPE;
synth_opts->period = PERF_ITRACE_DEFAULT_PERIOD;
case 'x':
synth_opts->transactions = true;
break;
+ case 'w':
+ synth_opts->ptwrites = true;
+ break;
+ case 'p':
+ synth_opts->pwr_events = true;
+ break;
case 'e':
synth_opts->errors = true;
break;
* @instructions: whether to synthesize 'instructions' events
* @branches: whether to synthesize 'branches' events
* @transactions: whether to synthesize events for transactions
+ * @ptwrites: whether to synthesize events for ptwrites
+ * @pwr_events: whether to synthesize power events
* @errors: whether to synthesize decoder error events
* @dont_decode: whether to skip decoding entirely
* @log: write a decoding log
* @period: 'instructions' events period
* @period_type: 'instructions' events period type
* @initial_skip: skip N events at the beginning.
+ * @cpu_bitmap: CPUs for which to synthesize events, or NULL for all
*/
struct itrace_synth_opts {
bool set;
bool instructions;
bool branches;
bool transactions;
+ bool ptwrites;
+ bool pwr_events;
bool errors;
bool dont_decode;
bool log;
unsigned long long period;
enum itrace_period_type period_type;
unsigned long initial_skip;
+ unsigned long *cpu_bitmap;
};
/**
#include <subcmd/pager.h>
#include "../ui/ui.h"
+#include <linux/compiler.h>
#include <linux/string.h>
#define CMD_EXEC_PATH "--exec-path"
return path[0] == '/';
}
-char *mkpath(const char *fmt, ...) __attribute__((format (printf, 1, 2)));
+char *mkpath(const char *fmt, ...) __printf(1, 2);
#endif /* __PERF_CACHE_H */
return 0;
}
-static void die_bad_config(const char *name)
+static void bad_config(const char *name)
{
if (config_file_name)
- die("bad config value for '%s' in %s", name, config_file_name);
- die("bad config value for '%s'", name);
+ pr_warning("bad config value for '%s' in %s, ignoring...\n", name, config_file_name);
+ else
+ pr_warning("bad config value for '%s', ignoring...\n", name);
}
-u64 perf_config_u64(const char *name, const char *value)
+int perf_config_u64(u64 *dest, const char *name, const char *value)
{
long long ret = 0;
- if (!perf_parse_llong(value, &ret))
- die_bad_config(name);
- return (u64) ret;
+ if (!perf_parse_llong(value, &ret)) {
+ bad_config(name);
+ return -1;
+ }
+
+ *dest = ret;
+ return 0;
}
-int perf_config_int(const char *name, const char *value)
+int perf_config_int(int *dest, const char *name, const char *value)
{
long ret = 0;
- if (!perf_parse_long(value, &ret))
- die_bad_config(name);
- return ret;
+ if (!perf_parse_long(value, &ret)) {
+ bad_config(name);
+ return -1;
+ }
+ *dest = ret;
+ return 0;
}
static int perf_config_bool_or_int(const char *name, const char *value, int *is_bool)
{
+ int ret;
+
*is_bool = 1;
if (!value)
return 1;
if (!strcasecmp(value, "false") || !strcasecmp(value, "no") || !strcasecmp(value, "off"))
return 0;
*is_bool = 0;
- return perf_config_int(name, value);
+ return perf_config_int(&ret, name, value) < 0 ? -1 : ret;
}
int perf_config_bool(const char *name, const char *value)
user_config = strdup(mkpath("%s/.perfconfig", home));
if (user_config == NULL) {
- warning("Not enough memory to process %s/.perfconfig, "
- "ignoring it.", home);
+ pr_warning("Not enough memory to process %s/.perfconfig, ignoring it.", home);
goto out;
}
ret = 0;
if (st.st_uid && (st.st_uid != geteuid())) {
- warning("File %s not owned by current user or root, "
- "ignoring it.", user_config);
+ pr_warning("File %s not owned by current user or root, ignoring it.", user_config);
goto out_free;
}
*/
int config_error_nonbool(const char *var)
{
- return error("Missing value for '%s'", var);
+ pr_err("Missing value for '%s'", var);
+ return -1;
}
void set_buildid_dir(const char *dir)
typedef int (*config_fn_t)(const char *, const char *, void *);
int perf_default_config(const char *, const char *, void *);
int perf_config(config_fn_t fn, void *);
-int perf_config_int(const char *, const char *);
-u64 perf_config_u64(const char *, const char *);
+int perf_config_int(int *dest, const char *, const char *);
+int perf_config_u64(u64 *dest, const char *, const char *);
int perf_config_bool(const char *, const char *);
int config_error_nonbool(const char *);
const char *perf_etc_perfconfig(void);
{
struct convert *c = cb;
- if (!strcmp(var, "convert.queue-size")) {
- c->queue_size = perf_config_u64(var, value);
- return 0;
- }
+ if (!strcmp(var, "convert.queue-size"))
+ return perf_config_u64(&c->queue_size, var, value);
return 0;
}
#include <stdbool.h>
#include <string.h>
+#include <linux/compiler.h>
#include "event.h"
#include "../ui/helpline.h"
#include "../ui/progress.h"
#define STRERR_BUFSIZE 128 /* For the buffer size of str_error_r */
-int dump_printf(const char *fmt, ...) __attribute__((format(printf, 1, 2)));
+int dump_printf(const char *fmt, ...) __printf(1, 2);
void trace_event(union perf_event *event);
-int ui__error(const char *format, ...) __attribute__((format(printf, 1, 2)));
-int ui__warning(const char *format, ...) __attribute__((format(printf, 1, 2)));
+int ui__error(const char *format, ...) __printf(1, 2);
+int ui__warning(const char *format, ...) __printf(1, 2);
void pr_stat(const char *fmt, ...);
-int eprintf(int level, int var, const char *fmt, ...) __attribute__((format(printf, 3, 4)));
-int eprintf_time(int level, int var, u64 t, const char *fmt, ...) __attribute__((format(printf, 4, 5)));
+int eprintf(int level, int var, const char *fmt, ...) __printf(3, 4);
+int eprintf_time(int level, int var, u64 t, const char *fmt, ...) __printf(4, 5);
int veprintf(int level, int var, const char *fmt, va_list args);
int perf_debug_option(const char *str);
PERF_AUXTRACE_ERROR_MAX
};
+/* Attribute type for custom synthesized events */
+#define PERF_TYPE_SYNTH (INT_MAX + 1U)
+
+/* Attribute config for custom synthesized events */
+enum perf_synth_id {
+ PERF_SYNTH_INTEL_PTWRITE,
+ PERF_SYNTH_INTEL_MWAIT,
+ PERF_SYNTH_INTEL_PWRE,
+ PERF_SYNTH_INTEL_EXSTOP,
+ PERF_SYNTH_INTEL_PWRX,
+ PERF_SYNTH_INTEL_CBR,
+};
+
+/*
+ * Raw data formats for synthesized events. Note that 4 bytes of padding are
+ * present to match the 'size' member of PERF_SAMPLE_RAW data which is always
+ * 8-byte aligned. That means we must dereference raw_data with an offset of 4.
+ * Refer perf_sample__synth_ptr() and perf_synth__raw_data(). It also means the
+ * structure sizes are 4 bytes bigger than the raw_size, refer
+ * perf_synth__raw_size().
+ */
+
+struct perf_synth_intel_ptwrite {
+ u32 padding;
+ union {
+ struct {
+ u32 ip : 1,
+ reserved : 31;
+ };
+ u32 flags;
+ };
+ u64 payload;
+};
+
+struct perf_synth_intel_mwait {
+ u32 padding;
+ u32 reserved;
+ union {
+ struct {
+ u64 hints : 8,
+ reserved1 : 24,
+ extensions : 2,
+ reserved2 : 30;
+ };
+ u64 payload;
+ };
+};
+
+struct perf_synth_intel_pwre {
+ u32 padding;
+ u32 reserved;
+ union {
+ struct {
+ u64 reserved1 : 7,
+ hw : 1,
+ subcstate : 4,
+ cstate : 4,
+ reserved2 : 48;
+ };
+ u64 payload;
+ };
+};
+
+struct perf_synth_intel_exstop {
+ u32 padding;
+ union {
+ struct {
+ u32 ip : 1,
+ reserved : 31;
+ };
+ u32 flags;
+ };
+};
+
+struct perf_synth_intel_pwrx {
+ u32 padding;
+ u32 reserved;
+ union {
+ struct {
+ u64 deepest_cstate : 4,
+ last_cstate : 4,
+ wake_reason : 4,
+ reserved1 : 52;
+ };
+ u64 payload;
+ };
+};
+
+struct perf_synth_intel_cbr {
+ u32 padding;
+ union {
+ struct {
+ u32 cbr : 8,
+ reserved1 : 8,
+ max_nonturbo : 8,
+ reserved2 : 8;
+ };
+ u32 flags;
+ };
+ u32 freq;
+ u32 reserved3;
+};
+
+/*
+ * raw_data is always 4 bytes from an 8-byte boundary, so subtract 4 to get
+ * 8-byte alignment.
+ */
+static inline void *perf_sample__synth_ptr(struct perf_sample *sample)
+{
+ return sample->raw_data - 4;
+}
+
+static inline void *perf_synth__raw_data(void *p)
+{
+ return p + 4;
+}
+
+#define perf_synth__raw_size(d) (sizeof(d) - 4)
+
+#define perf_sample__bad_synth_size(s, d) ((s)->raw_size < sizeof(d) - 4)
+
/*
* The kernel collects the number of events it couldn't send in a stretch and
* when possible sends this number in a PERF_RECORD_LOST event. The number of
#ifndef __PERF_EVLIST_H
#define __PERF_EVLIST_H 1
+#include <linux/compiler.h>
#include <linux/kernel.h>
#include <linux/refcount.h>
#include <linux/list.h>
refcount_t refcnt;
u64 prev;
struct auxtrace_mmap auxtrace_mmap;
- char event_copy[PERF_SAMPLE_MAX_SIZE] __attribute__((aligned(8)));
+ char event_copy[PERF_SAMPLE_MAX_SIZE] __aligned(8);
};
static inline size_t
#include <errno.h>
#include <inttypes.h>
#include <linux/bitops.h>
+#include <api/fs/fs.h>
#include <api/fs/tracing_path.h>
#include <traceevent/event-parse.h>
#include <linux/hw_breakpoint.h>
#include <linux/perf_event.h>
+#include <linux/compiler.h>
#include <linux/err.h>
#include <sys/ioctl.h>
#include <sys/resource.h>
+#include <sys/types.h>
+#include <dirent.h>
#include "asm/bug.h"
#include "callchain.h"
#include "cgroup.h"
}
static int __open_attr__fprintf(FILE *fp, const char *name, const char *val,
- void *priv __attribute__((unused)))
+ void *priv __maybe_unused)
{
return fprintf(fp, " %-32s %s\n", name, val);
}
return false;
}
+static bool find_process(const char *name)
+{
+ size_t len = strlen(name);
+ DIR *dir;
+ struct dirent *d;
+ int ret = -1;
+
+ dir = opendir(procfs__mountpoint());
+ if (!dir)
+ return false;
+
+ /* Walk through the directory. */
+ while (ret && (d = readdir(dir)) != NULL) {
+ char path[PATH_MAX];
+ char *data;
+ size_t size;
+
+ if ((d->d_type != DT_DIR) ||
+ !strcmp(".", d->d_name) ||
+ !strcmp("..", d->d_name))
+ continue;
+
+ scnprintf(path, sizeof(path), "%s/%s/comm",
+ procfs__mountpoint(), d->d_name);
+
+ if (filename__read_str(path, &data, &size))
+ continue;
+
+ ret = strncmp(name, data, len);
+ free(data);
+ }
+
+ closedir(dir);
+ return ret ? false : true;
+}
+
int perf_evsel__open_strerror(struct perf_evsel *evsel, struct target *target,
int err, char *msg, size_t size)
{
* @remark Copyright 2007 OProfile authors
* @author Philippe Elie
*/
+#include <linux/compiler.h>
#include <sys/types.h>
#include <stdio.h>
#include <getopt.h>
* and filesize, last entry is followed by en empty string.
*/
/* follow the first program statement */
-} __attribute__((packed));
+} __packed;
/* DWARF 2 spec talk only about one possible compilation unit header while
* binutils can handle two flavours of dwarf 2, 32 and 64 bits, this is not
uhalf version;
uword debug_abbrev_offset;
ubyte pointer_size;
-} __attribute__((packed));
+} __packed;
#define DW_LNS_num_opcode (DW_LNS_set_isa + 1)
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
+#include <linux/compiler.h>
#include <linux/list.h>
#include <linux/kernel.h>
#include <linux/bitops.h>
}
static int __desc_attr__fprintf(FILE *fp, const char *name, const char *val,
- void *priv __attribute__((unused)))
+ void *priv __maybe_unused)
{
return fprintf(fp, ", %s = %s", name, val);
}
void *cb __maybe_unused)
{
if (!strcmp(var, "help.autocorrect"))
- autocorrect = perf_config_int(var,value);
+ return perf_config_int(&autocorrect, var,value);
return 0;
}
fprintf(stdout, intel_bts_info_fmts[i], arr[i]);
}
-u64 intel_bts_auxtrace_info_priv[INTEL_BTS_AUXTRACE_PRIV_SIZE];
-
int intel_bts_process_auxtrace_info(union perf_event *event,
struct perf_session *session)
{
INTEL_PT_STATE_FUP_NO_TIP,
};
+static inline bool intel_pt_sample_time(enum intel_pt_pkt_state pkt_state)
+{
+ switch (pkt_state) {
+ case INTEL_PT_STATE_NO_PSB:
+ case INTEL_PT_STATE_NO_IP:
+ case INTEL_PT_STATE_ERR_RESYNC:
+ case INTEL_PT_STATE_IN_SYNC:
+ case INTEL_PT_STATE_TNT:
+ return true;
+ case INTEL_PT_STATE_TIP:
+ case INTEL_PT_STATE_TIP_PGD:
+ case INTEL_PT_STATE_FUP:
+ case INTEL_PT_STATE_FUP_NO_TIP:
+ return false;
+ default:
+ return true;
+ };
+}
+
#ifdef INTEL_PT_STRICT
#define INTEL_PT_STATE_ERR1 INTEL_PT_STATE_NO_PSB
#define INTEL_PT_STATE_ERR2 INTEL_PT_STATE_NO_PSB
const unsigned char *buf;
size_t len;
bool return_compression;
+ bool branch_enable;
bool mtc_insn;
bool pge;
bool have_tma;
bool have_cyc;
bool fixup_last_mtc;
+ bool have_last_ip;
uint64_t pos;
uint64_t last_ip;
uint64_t ip;
uint64_t timestamp;
uint64_t tsc_timestamp;
uint64_t ref_timestamp;
+ uint64_t sample_timestamp;
uint64_t ret_addr;
uint64_t ctc_timestamp;
uint64_t ctc_delta;
int pkt_len;
int last_packet_type;
unsigned int cbr;
+ unsigned int cbr_seen;
unsigned int max_non_turbo_ratio;
double max_non_turbo_ratio_fp;
double cbr_cyc_to_tsc;
bool continuous_period;
bool overflow;
bool set_fup_tx_flags;
+ bool set_fup_ptw;
+ bool set_fup_mwait;
+ bool set_fup_pwre;
+ bool set_fup_exstop;
unsigned int fup_tx_flags;
unsigned int tx_flags;
+ uint64_t fup_ptw_payload;
+ uint64_t fup_mwait_payload;
+ uint64_t fup_pwre_payload;
+ uint64_t cbr_payload;
uint64_t timestamp_insn_cnt;
+ uint64_t sample_insn_cnt;
uint64_t stuck_ip;
int no_progress;
int stuck_ip_prd;
decoder->pgd_ip = params->pgd_ip;
decoder->data = params->data;
decoder->return_compression = params->return_compression;
+ decoder->branch_enable = params->branch_enable;
decoder->period = params->period;
decoder->period_type = params->period_type;
static inline void intel_pt_set_last_ip(struct intel_pt_decoder *decoder)
{
decoder->last_ip = intel_pt_calc_ip(&decoder->packet, decoder->last_ip);
+ decoder->have_last_ip = true;
}
static inline void intel_pt_set_ip(struct intel_pt_decoder *decoder)
case INTEL_PT_PAD:
case INTEL_PT_VMCS:
case INTEL_PT_MNT:
+ case INTEL_PT_PTWRITE:
+ case INTEL_PT_PTWRITE_IP:
return 0;
case INTEL_PT_MTC:
break;
case INTEL_PT_TSC:
+ /*
+ * For now, do not support using TSC packets - refer
+ * intel_pt_calc_cyc_to_tsc().
+ */
+ if (data->from_mtc)
+ return 1;
timestamp = pkt_info->packet.payload |
(data->timestamp & (0xffULL << 56));
if (data->from_mtc && timestamp < data->timestamp &&
case INTEL_PT_TIP_PGD:
case INTEL_PT_TRACESTOP:
+ case INTEL_PT_EXSTOP:
+ case INTEL_PT_EXSTOP_IP:
+ case INTEL_PT_MWAIT:
+ case INTEL_PT_PWRE:
+ case INTEL_PT_PWRX:
case INTEL_PT_OVF:
case INTEL_PT_BAD: /* Does not happen */
default:
.cbr_cyc_to_tsc = 0,
};
+ /*
+ * For now, do not support using TSC packets for at least the reasons:
+ * 1) timing might have stopped
+ * 2) TSC packets within PSB+ can slip against CYC packets
+ */
+ if (!from_mtc)
+ return;
+
intel_pt_pkt_lookahead(decoder, intel_pt_calc_cyc_cb, &data);
}
decoder->tot_insn_cnt += insn_cnt;
decoder->timestamp_insn_cnt += insn_cnt;
+ decoder->sample_insn_cnt += insn_cnt;
decoder->period_insn_cnt += insn_cnt;
if (err) {
return err;
}
+static bool intel_pt_fup_event(struct intel_pt_decoder *decoder)
+{
+ bool ret = false;
+
+ if (decoder->set_fup_tx_flags) {
+ decoder->set_fup_tx_flags = false;
+ decoder->tx_flags = decoder->fup_tx_flags;
+ decoder->state.type = INTEL_PT_TRANSACTION;
+ decoder->state.from_ip = decoder->ip;
+ decoder->state.to_ip = 0;
+ decoder->state.flags = decoder->fup_tx_flags;
+ return true;
+ }
+ if (decoder->set_fup_ptw) {
+ decoder->set_fup_ptw = false;
+ decoder->state.type = INTEL_PT_PTW;
+ decoder->state.flags |= INTEL_PT_FUP_IP;
+ decoder->state.from_ip = decoder->ip;
+ decoder->state.to_ip = 0;
+ decoder->state.ptw_payload = decoder->fup_ptw_payload;
+ return true;
+ }
+ if (decoder->set_fup_mwait) {
+ decoder->set_fup_mwait = false;
+ decoder->state.type = INTEL_PT_MWAIT_OP;
+ decoder->state.from_ip = decoder->ip;
+ decoder->state.to_ip = 0;
+ decoder->state.mwait_payload = decoder->fup_mwait_payload;
+ ret = true;
+ }
+ if (decoder->set_fup_pwre) {
+ decoder->set_fup_pwre = false;
+ decoder->state.type |= INTEL_PT_PWR_ENTRY;
+ decoder->state.type &= ~INTEL_PT_BRANCH;
+ decoder->state.from_ip = decoder->ip;
+ decoder->state.to_ip = 0;
+ decoder->state.pwre_payload = decoder->fup_pwre_payload;
+ ret = true;
+ }
+ if (decoder->set_fup_exstop) {
+ decoder->set_fup_exstop = false;
+ decoder->state.type |= INTEL_PT_EX_STOP;
+ decoder->state.type &= ~INTEL_PT_BRANCH;
+ decoder->state.flags |= INTEL_PT_FUP_IP;
+ decoder->state.from_ip = decoder->ip;
+ decoder->state.to_ip = 0;
+ ret = true;
+ }
+ return ret;
+}
+
static int intel_pt_walk_fup(struct intel_pt_decoder *decoder)
{
struct intel_pt_insn intel_pt_insn;
if (err == INTEL_PT_RETURN)
return 0;
if (err == -EAGAIN) {
- if (decoder->set_fup_tx_flags) {
- decoder->set_fup_tx_flags = false;
- decoder->tx_flags = decoder->fup_tx_flags;
- decoder->state.type = INTEL_PT_TRANSACTION;
- decoder->state.from_ip = decoder->ip;
- decoder->state.to_ip = 0;
- decoder->state.flags = decoder->fup_tx_flags;
+ if (intel_pt_fup_event(decoder))
return 0;
- }
return err;
}
decoder->set_fup_tx_flags = false;
static void intel_pt_calc_cbr(struct intel_pt_decoder *decoder)
{
- unsigned int cbr = decoder->packet.payload;
+ unsigned int cbr = decoder->packet.payload & 0xff;
+
+ decoder->cbr_payload = decoder->packet.payload;
if (decoder->cbr == cbr)
return;
case INTEL_PT_TRACESTOP:
case INTEL_PT_BAD:
case INTEL_PT_PSB:
+ case INTEL_PT_PTWRITE:
+ case INTEL_PT_PTWRITE_IP:
+ case INTEL_PT_EXSTOP:
+ case INTEL_PT_EXSTOP_IP:
+ case INTEL_PT_MWAIT:
+ case INTEL_PT_PWRE:
+ case INTEL_PT_PWRX:
decoder->have_tma = false;
intel_pt_log("ERROR: Unexpected packet\n");
return -EAGAIN;
case INTEL_PT_FUP:
decoder->pge = true;
- intel_pt_set_last_ip(decoder);
+ if (decoder->packet.count)
+ intel_pt_set_last_ip(decoder);
break;
case INTEL_PT_MODE_TSX:
case INTEL_PT_MODE_TSX:
case INTEL_PT_BAD:
case INTEL_PT_PSBEND:
+ case INTEL_PT_PTWRITE:
+ case INTEL_PT_PTWRITE_IP:
+ case INTEL_PT_EXSTOP:
+ case INTEL_PT_EXSTOP_IP:
+ case INTEL_PT_MWAIT:
+ case INTEL_PT_PWRE:
+ case INTEL_PT_PWRX:
intel_pt_log("ERROR: Missing TIP after FUP\n");
decoder->pkt_state = INTEL_PT_STATE_ERR3;
return -ENOENT;
break;
}
intel_pt_set_last_ip(decoder);
+ if (!decoder->branch_enable) {
+ decoder->ip = decoder->last_ip;
+ if (intel_pt_fup_event(decoder))
+ return 0;
+ no_tip = false;
+ break;
+ }
+ if (decoder->set_fup_mwait)
+ no_tip = true;
err = intel_pt_walk_fup(decoder);
if (err != -EAGAIN) {
if (err)
break;
case INTEL_PT_PSB:
+ decoder->last_ip = 0;
+ decoder->have_last_ip = true;
intel_pt_clear_stack(&decoder->stack);
err = intel_pt_walk_psbend(decoder);
if (err == -EAGAIN)
case INTEL_PT_CBR:
intel_pt_calc_cbr(decoder);
+ if (!decoder->branch_enable &&
+ decoder->cbr != decoder->cbr_seen) {
+ decoder->cbr_seen = decoder->cbr;
+ decoder->state.type = INTEL_PT_CBR_CHG;
+ decoder->state.from_ip = decoder->ip;
+ decoder->state.to_ip = 0;
+ decoder->state.cbr_payload =
+ decoder->packet.payload;
+ return 0;
+ }
break;
case INTEL_PT_MODE_EXEC:
case INTEL_PT_PAD:
break;
+ case INTEL_PT_PTWRITE_IP:
+ decoder->fup_ptw_payload = decoder->packet.payload;
+ err = intel_pt_get_next_packet(decoder);
+ if (err)
+ return err;
+ if (decoder->packet.type == INTEL_PT_FUP) {
+ decoder->set_fup_ptw = true;
+ no_tip = true;
+ } else {
+ intel_pt_log_at("ERROR: Missing FUP after PTWRITE",
+ decoder->pos);
+ }
+ goto next;
+
+ case INTEL_PT_PTWRITE:
+ decoder->state.type = INTEL_PT_PTW;
+ decoder->state.from_ip = decoder->ip;
+ decoder->state.to_ip = 0;
+ decoder->state.ptw_payload = decoder->packet.payload;
+ return 0;
+
+ case INTEL_PT_MWAIT:
+ decoder->fup_mwait_payload = decoder->packet.payload;
+ decoder->set_fup_mwait = true;
+ break;
+
+ case INTEL_PT_PWRE:
+ if (decoder->set_fup_mwait) {
+ decoder->fup_pwre_payload =
+ decoder->packet.payload;
+ decoder->set_fup_pwre = true;
+ break;
+ }
+ decoder->state.type = INTEL_PT_PWR_ENTRY;
+ decoder->state.from_ip = decoder->ip;
+ decoder->state.to_ip = 0;
+ decoder->state.pwrx_payload = decoder->packet.payload;
+ return 0;
+
+ case INTEL_PT_EXSTOP_IP:
+ err = intel_pt_get_next_packet(decoder);
+ if (err)
+ return err;
+ if (decoder->packet.type == INTEL_PT_FUP) {
+ decoder->set_fup_exstop = true;
+ no_tip = true;
+ } else {
+ intel_pt_log_at("ERROR: Missing FUP after EXSTOP",
+ decoder->pos);
+ }
+ goto next;
+
+ case INTEL_PT_EXSTOP:
+ decoder->state.type = INTEL_PT_EX_STOP;
+ decoder->state.from_ip = decoder->ip;
+ decoder->state.to_ip = 0;
+ return 0;
+
+ case INTEL_PT_PWRX:
+ decoder->state.type = INTEL_PT_PWR_EXIT;
+ decoder->state.from_ip = decoder->ip;
+ decoder->state.to_ip = 0;
+ decoder->state.pwrx_payload = decoder->packet.payload;
+ return 0;
+
default:
return intel_pt_bug(decoder);
}
static inline bool intel_pt_have_ip(struct intel_pt_decoder *decoder)
{
- return decoder->last_ip || decoder->packet.count == 0 ||
- decoder->packet.count == 3 || decoder->packet.count == 6;
+ return decoder->packet.count &&
+ (decoder->have_last_ip || decoder->packet.count == 3 ||
+ decoder->packet.count == 6);
}
/* Walk PSB+ packets to get in sync. */
__fallthrough;
case INTEL_PT_TIP_PGE:
case INTEL_PT_TIP:
+ case INTEL_PT_PTWRITE:
+ case INTEL_PT_PTWRITE_IP:
+ case INTEL_PT_EXSTOP:
+ case INTEL_PT_EXSTOP_IP:
+ case INTEL_PT_MWAIT:
+ case INTEL_PT_PWRE:
+ case INTEL_PT_PWRX:
intel_pt_log("ERROR: Unexpected packet\n");
return -ENOENT;
break;
case INTEL_PT_FUP:
- if (decoder->overflow) {
- if (intel_pt_have_ip(decoder))
- intel_pt_set_ip(decoder);
- if (decoder->ip)
- return 0;
- }
- if (decoder->packet.count)
- intel_pt_set_last_ip(decoder);
+ if (intel_pt_have_ip(decoder))
+ intel_pt_set_ip(decoder);
+ if (decoder->ip)
+ return 0;
break;
case INTEL_PT_MTC:
break;
case INTEL_PT_PSB:
+ decoder->last_ip = 0;
+ decoder->have_last_ip = true;
+ intel_pt_clear_stack(&decoder->stack);
err = intel_pt_walk_psb(decoder);
if (err)
return err;
case INTEL_PT_VMCS:
case INTEL_PT_MNT:
case INTEL_PT_PAD:
+ case INTEL_PT_PTWRITE:
+ case INTEL_PT_PTWRITE_IP:
+ case INTEL_PT_EXSTOP:
+ case INTEL_PT_EXSTOP_IP:
+ case INTEL_PT_MWAIT:
+ case INTEL_PT_PWRE:
+ case INTEL_PT_PWRX:
default:
break;
}
{
int err;
+ decoder->set_fup_tx_flags = false;
+ decoder->set_fup_ptw = false;
+ decoder->set_fup_mwait = false;
+ decoder->set_fup_pwre = false;
+ decoder->set_fup_exstop = false;
+
+ if (!decoder->branch_enable) {
+ decoder->pkt_state = INTEL_PT_STATE_IN_SYNC;
+ decoder->overflow = false;
+ decoder->state.type = 0; /* Do not have a sample */
+ return 0;
+ }
+
intel_pt_log("Scanning for full IP\n");
err = intel_pt_walk_to_ip(decoder);
if (err)
decoder->pge = false;
decoder->continuous_period = false;
+ decoder->have_last_ip = false;
decoder->last_ip = 0;
decoder->ip = 0;
intel_pt_clear_stack(&decoder->stack);
if (err)
return err;
+ decoder->have_last_ip = true;
decoder->pkt_state = INTEL_PT_STATE_NO_IP;
err = intel_pt_walk_psb(decoder);
static uint64_t intel_pt_est_timestamp(struct intel_pt_decoder *decoder)
{
- uint64_t est = decoder->timestamp_insn_cnt << 1;
+ uint64_t est = decoder->sample_insn_cnt << 1;
if (!decoder->cbr || !decoder->max_non_turbo_ratio)
goto out;
est *= decoder->max_non_turbo_ratio;
est /= decoder->cbr;
out:
- return decoder->timestamp + est;
+ return decoder->sample_timestamp + est;
}
const struct intel_pt_state *intel_pt_decode(struct intel_pt_decoder *decoder)
err = intel_pt_sync(decoder);
break;
case INTEL_PT_STATE_NO_IP:
+ decoder->have_last_ip = false;
decoder->last_ip = 0;
- /* Fall through */
+ decoder->ip = 0;
+ __fallthrough;
case INTEL_PT_STATE_ERR_RESYNC:
err = intel_pt_sync_ip(decoder);
break;
}
} while (err == -ENOLINK);
- decoder->state.err = err ? intel_pt_ext_err(err) : 0;
- decoder->state.timestamp = decoder->timestamp;
+ if (err) {
+ decoder->state.err = intel_pt_ext_err(err);
+ decoder->state.from_ip = decoder->ip;
+ decoder->sample_timestamp = decoder->timestamp;
+ decoder->sample_insn_cnt = decoder->timestamp_insn_cnt;
+ } else {
+ decoder->state.err = 0;
+ if (decoder->cbr != decoder->cbr_seen && decoder->state.type) {
+ decoder->cbr_seen = decoder->cbr;
+ decoder->state.type |= INTEL_PT_CBR_CHG;
+ decoder->state.cbr_payload = decoder->cbr_payload;
+ }
+ if (intel_pt_sample_time(decoder->pkt_state)) {
+ decoder->sample_timestamp = decoder->timestamp;
+ decoder->sample_insn_cnt = decoder->timestamp_insn_cnt;
+ }
+ }
+
+ decoder->state.timestamp = decoder->sample_timestamp;
decoder->state.est_timestamp = intel_pt_est_timestamp(decoder);
decoder->state.cr3 = decoder->cr3;
decoder->state.tot_insn_cnt = decoder->tot_insn_cnt;
- if (err)
- decoder->state.from_ip = decoder->ip;
-
return &decoder->state;
}
#define INTEL_PT_IN_TX (1 << 0)
#define INTEL_PT_ABORT_TX (1 << 1)
#define INTEL_PT_ASYNC (1 << 2)
+#define INTEL_PT_FUP_IP (1 << 3)
enum intel_pt_sample_type {
INTEL_PT_BRANCH = 1 << 0,
INTEL_PT_INSTRUCTION = 1 << 1,
INTEL_PT_TRANSACTION = 1 << 2,
+ INTEL_PT_PTW = 1 << 3,
+ INTEL_PT_MWAIT_OP = 1 << 4,
+ INTEL_PT_PWR_ENTRY = 1 << 5,
+ INTEL_PT_EX_STOP = 1 << 6,
+ INTEL_PT_PWR_EXIT = 1 << 7,
+ INTEL_PT_CBR_CHG = 1 << 8,
};
enum intel_pt_period_type {
uint64_t timestamp;
uint64_t est_timestamp;
uint64_t trace_nr;
+ uint64_t ptw_payload;
+ uint64_t mwait_payload;
+ uint64_t pwre_payload;
+ uint64_t pwrx_payload;
+ uint64_t cbr_payload;
uint32_t flags;
enum intel_pt_insn_op insn_op;
int insn_len;
bool (*pgd_ip)(uint64_t ip, void *data);
void *data;
bool return_compression;
+ bool branch_enable;
uint64_t period;
enum intel_pt_period_type period_type;
unsigned max_non_turbo_ratio;
#ifndef INCLUDE__INTEL_PT_LOG_H__
#define INCLUDE__INTEL_PT_LOG_H__
+#include <linux/compiler.h>
#include <stdint.h>
#include <inttypes.h>
void __intel_pt_log_insn_no_data(struct intel_pt_insn *intel_pt_insn,
uint64_t ip);
-__attribute__((format(printf, 1, 2)))
-void __intel_pt_log(const char *fmt, ...);
+void __intel_pt_log(const char *fmt, ...) __printf(1, 2);
#define intel_pt_log(fmt, ...) \
do { \
[INTEL_PT_PIP] = "PIP",
[INTEL_PT_OVF] = "OVF",
[INTEL_PT_MNT] = "MNT",
+ [INTEL_PT_PTWRITE] = "PTWRITE",
+ [INTEL_PT_PTWRITE_IP] = "PTWRITE",
+ [INTEL_PT_EXSTOP] = "EXSTOP",
+ [INTEL_PT_EXSTOP_IP] = "EXSTOP",
+ [INTEL_PT_MWAIT] = "MWAIT",
+ [INTEL_PT_PWRE] = "PWRE",
+ [INTEL_PT_PWRX] = "PWRX",
};
const char *intel_pt_pkt_name(enum intel_pt_pkt_type type)
if (len < 4)
return INTEL_PT_NEED_MORE_BYTES;
packet->type = INTEL_PT_CBR;
- packet->payload = buf[2];
+ packet->payload = le16_to_cpu(*(uint16_t *)(buf + 2));
return 4;
}
}
}
+static int intel_pt_get_ptwrite(const unsigned char *buf, size_t len,
+ struct intel_pt_pkt *packet)
+{
+ packet->count = (buf[1] >> 5) & 0x3;
+ packet->type = buf[1] & BIT(7) ? INTEL_PT_PTWRITE_IP :
+ INTEL_PT_PTWRITE;
+
+ switch (packet->count) {
+ case 0:
+ if (len < 6)
+ return INTEL_PT_NEED_MORE_BYTES;
+ packet->payload = le32_to_cpu(*(uint32_t *)(buf + 2));
+ return 6;
+ case 1:
+ if (len < 10)
+ return INTEL_PT_NEED_MORE_BYTES;
+ packet->payload = le64_to_cpu(*(uint64_t *)(buf + 2));
+ return 10;
+ default:
+ return INTEL_PT_BAD_PACKET;
+ }
+}
+
+static int intel_pt_get_exstop(struct intel_pt_pkt *packet)
+{
+ packet->type = INTEL_PT_EXSTOP;
+ return 2;
+}
+
+static int intel_pt_get_exstop_ip(struct intel_pt_pkt *packet)
+{
+ packet->type = INTEL_PT_EXSTOP_IP;
+ return 2;
+}
+
+static int intel_pt_get_mwait(const unsigned char *buf, size_t len,
+ struct intel_pt_pkt *packet)
+{
+ if (len < 10)
+ return INTEL_PT_NEED_MORE_BYTES;
+ packet->type = INTEL_PT_MWAIT;
+ packet->payload = le64_to_cpu(*(uint64_t *)(buf + 2));
+ return 10;
+}
+
+static int intel_pt_get_pwre(const unsigned char *buf, size_t len,
+ struct intel_pt_pkt *packet)
+{
+ if (len < 4)
+ return INTEL_PT_NEED_MORE_BYTES;
+ packet->type = INTEL_PT_PWRE;
+ memcpy_le64(&packet->payload, buf + 2, 2);
+ return 4;
+}
+
+static int intel_pt_get_pwrx(const unsigned char *buf, size_t len,
+ struct intel_pt_pkt *packet)
+{
+ if (len < 7)
+ return INTEL_PT_NEED_MORE_BYTES;
+ packet->type = INTEL_PT_PWRX;
+ memcpy_le64(&packet->payload, buf + 2, 5);
+ return 7;
+}
+
static int intel_pt_get_ext(const unsigned char *buf, size_t len,
struct intel_pt_pkt *packet)
{
if (len < 2)
return INTEL_PT_NEED_MORE_BYTES;
+ if ((buf[1] & 0x1f) == 0x12)
+ return intel_pt_get_ptwrite(buf, len, packet);
+
switch (buf[1]) {
case 0xa3: /* Long TNT */
return intel_pt_get_long_tnt(buf, len, packet);
return intel_pt_get_tma(buf, len, packet);
case 0xC3: /* 3-byte header */
return intel_pt_get_3byte(buf, len, packet);
+ case 0x62: /* EXSTOP no IP */
+ return intel_pt_get_exstop(packet);
+ case 0xE2: /* EXSTOP with IP */
+ return intel_pt_get_exstop_ip(packet);
+ case 0xC2: /* MWAIT */
+ return intel_pt_get_mwait(buf, len, packet);
+ case 0x22: /* PWRE */
+ return intel_pt_get_pwre(buf, len, packet);
+ case 0xA2: /* PWRX */
+ return intel_pt_get_pwrx(buf, len, packet);
default:
return INTEL_PT_BAD_PACKET;
}
ret = snprintf(buf, buf_len, "%s 0x%llx (NR=%d)",
name, payload, nr);
return ret;
+ case INTEL_PT_PTWRITE:
+ return snprintf(buf, buf_len, "%s 0x%llx IP:0", name, payload);
+ case INTEL_PT_PTWRITE_IP:
+ return snprintf(buf, buf_len, "%s 0x%llx IP:1", name, payload);
+ case INTEL_PT_EXSTOP:
+ return snprintf(buf, buf_len, "%s IP:0", name);
+ case INTEL_PT_EXSTOP_IP:
+ return snprintf(buf, buf_len, "%s IP:1", name);
+ case INTEL_PT_MWAIT:
+ return snprintf(buf, buf_len, "%s 0x%llx Hints 0x%x Extensions 0x%x",
+ name, payload, (unsigned int)(payload & 0xff),
+ (unsigned int)((payload >> 32) & 0x3));
+ case INTEL_PT_PWRE:
+ return snprintf(buf, buf_len, "%s 0x%llx HW:%u CState:%u Sub-CState:%u",
+ name, payload, !!(payload & 0x80),
+ (unsigned int)((payload >> 12) & 0xf),
+ (unsigned int)((payload >> 8) & 0xf));
+ case INTEL_PT_PWRX:
+ return snprintf(buf, buf_len, "%s 0x%llx Last CState:%u Deepest CState:%u Wake Reason 0x%x",
+ name, payload,
+ (unsigned int)((payload >> 4) & 0xf),
+ (unsigned int)(payload & 0xf),
+ (unsigned int)((payload >> 8) & 0xf));
default:
break;
}
INTEL_PT_PIP,
INTEL_PT_OVF,
INTEL_PT_MNT,
+ INTEL_PT_PTWRITE,
+ INTEL_PT_PTWRITE_IP,
+ INTEL_PT_EXSTOP,
+ INTEL_PT_EXSTOP_IP,
+ INTEL_PT_MWAIT,
+ INTEL_PT_PWRE,
+ INTEL_PT_PWRX,
};
struct intel_pt_pkt {
1: fxstor | RDGSBASE Ry (F3),(11B)
2: vldmxcsr Md (v1) | WRFSBASE Ry (F3),(11B)
3: vstmxcsr Md (v1) | WRGSBASE Ry (F3),(11B)
-4: XSAVE
+4: XSAVE | ptwrite Ey (F3),(11B)
5: XRSTOR | lfence (11B)
6: XSAVEOPT | clwb (66) | mfence (11B)
7: clflush | clflushopt (66) | sfence (11B)
bool sample_instructions;
u64 instructions_sample_type;
- u64 instructions_sample_period;
u64 instructions_id;
bool sample_branches;
u64 transactions_sample_type;
u64 transactions_id;
+ bool sample_ptwrites;
+ u64 ptwrites_sample_type;
+ u64 ptwrites_id;
+
+ bool sample_pwr_events;
+ u64 pwr_events_sample_type;
+ u64 mwait_id;
+ u64 pwre_id;
+ u64 exstop_id;
+ u64 pwrx_id;
+ u64 cbr_id;
+
bool synth_needs_swap;
u64 tsc_bit;
u64 cyc_bit;
u64 noretcomp_bit;
unsigned max_non_turbo_ratio;
+ unsigned cbr2khz;
unsigned long num_events;
return true;
}
+static bool intel_pt_branch_enable(struct intel_pt *pt)
+{
+ struct perf_evsel *evsel;
+ u64 config;
+
+ evlist__for_each_entry(pt->session->evlist, evsel) {
+ if (intel_pt_get_config(pt, &evsel->attr, &config) &&
+ (config & 1) && !(config & 0x2000))
+ return false;
+ }
+ return true;
+}
+
static unsigned int intel_pt_mtc_period(struct intel_pt *pt)
{
struct perf_evsel *evsel;
params.walk_insn = intel_pt_walk_next_insn;
params.data = ptq;
params.return_compression = intel_pt_return_compression(pt);
+ params.branch_enable = intel_pt_branch_enable(pt);
params.max_non_turbo_ratio = pt->max_non_turbo_ratio;
params.mtc_period = intel_pt_mtc_period(pt);
params.tsc_ctc_ratio_n = pt->tsc_ctc_ratio_n;
bs->nr += 1;
}
+static inline bool intel_pt_skip_event(struct intel_pt *pt)
+{
+ return pt->synth_opts.initial_skip &&
+ pt->num_events++ < pt->synth_opts.initial_skip;
+}
+
+static void intel_pt_prep_b_sample(struct intel_pt *pt,
+ struct intel_pt_queue *ptq,
+ union perf_event *event,
+ struct perf_sample *sample)
+{
+ event->sample.header.type = PERF_RECORD_SAMPLE;
+ event->sample.header.misc = PERF_RECORD_MISC_USER;
+ event->sample.header.size = sizeof(struct perf_event_header);
+
+ if (!pt->timeless_decoding)
+ sample->time = tsc_to_perf_time(ptq->timestamp, &pt->tc);
+
+ sample->cpumode = PERF_RECORD_MISC_USER;
+ sample->ip = ptq->state->from_ip;
+ sample->pid = ptq->pid;
+ sample->tid = ptq->tid;
+ sample->addr = ptq->state->to_ip;
+ sample->period = 1;
+ sample->cpu = ptq->cpu;
+ sample->flags = ptq->flags;
+ sample->insn_len = ptq->insn_len;
+ memcpy(sample->insn, ptq->insn, INTEL_PT_INSN_BUF_SZ);
+}
+
static int intel_pt_inject_event(union perf_event *event,
struct perf_sample *sample, u64 type,
bool swapped)
return perf_event__synthesize_sample(event, type, 0, sample, swapped);
}
-static int intel_pt_synth_branch_sample(struct intel_pt_queue *ptq)
+static inline int intel_pt_opt_inject(struct intel_pt *pt,
+ union perf_event *event,
+ struct perf_sample *sample, u64 type)
+{
+ if (!pt->synth_opts.inject)
+ return 0;
+
+ return intel_pt_inject_event(event, sample, type, pt->synth_needs_swap);
+}
+
+static int intel_pt_deliver_synth_b_event(struct intel_pt *pt,
+ union perf_event *event,
+ struct perf_sample *sample, u64 type)
{
int ret;
+
+ ret = intel_pt_opt_inject(pt, event, sample, type);
+ if (ret)
+ return ret;
+
+ ret = perf_session__deliver_synth_event(pt->session, event, sample);
+ if (ret)
+ pr_err("Intel PT: failed to deliver event, error %d\n", ret);
+
+ return ret;
+}
+
+static int intel_pt_synth_branch_sample(struct intel_pt_queue *ptq)
+{
struct intel_pt *pt = ptq->pt;
union perf_event *event = ptq->event_buf;
struct perf_sample sample = { .ip = 0, };
if (pt->branches_filter && !(pt->branches_filter & ptq->flags))
return 0;
- if (pt->synth_opts.initial_skip &&
- pt->num_events++ < pt->synth_opts.initial_skip)
+ if (intel_pt_skip_event(pt))
return 0;
- event->sample.header.type = PERF_RECORD_SAMPLE;
- event->sample.header.misc = PERF_RECORD_MISC_USER;
- event->sample.header.size = sizeof(struct perf_event_header);
+ intel_pt_prep_b_sample(pt, ptq, event, &sample);
- if (!pt->timeless_decoding)
- sample.time = tsc_to_perf_time(ptq->timestamp, &pt->tc);
-
- sample.cpumode = PERF_RECORD_MISC_USER;
- sample.ip = ptq->state->from_ip;
- sample.pid = ptq->pid;
- sample.tid = ptq->tid;
- sample.addr = ptq->state->to_ip;
sample.id = ptq->pt->branches_id;
sample.stream_id = ptq->pt->branches_id;
- sample.period = 1;
- sample.cpu = ptq->cpu;
- sample.flags = ptq->flags;
- sample.insn_len = ptq->insn_len;
- memcpy(sample.insn, ptq->insn, INTEL_PT_INSN_BUF_SZ);
/*
* perf report cannot handle events without a branch stack when using
sample.branch_stack = (struct branch_stack *)&dummy_bs;
}
- if (pt->synth_opts.inject) {
- ret = intel_pt_inject_event(event, &sample,
- pt->branches_sample_type,
- pt->synth_needs_swap);
- if (ret)
- return ret;
+ return intel_pt_deliver_synth_b_event(pt, event, &sample,
+ pt->branches_sample_type);
+}
+
+static void intel_pt_prep_sample(struct intel_pt *pt,
+ struct intel_pt_queue *ptq,
+ union perf_event *event,
+ struct perf_sample *sample)
+{
+ intel_pt_prep_b_sample(pt, ptq, event, sample);
+
+ if (pt->synth_opts.callchain) {
+ thread_stack__sample(ptq->thread, ptq->chain,
+ pt->synth_opts.callchain_sz, sample->ip);
+ sample->callchain = ptq->chain;
}
- ret = perf_session__deliver_synth_event(pt->session, event, &sample);
- if (ret)
- pr_err("Intel Processor Trace: failed to deliver branch event, error %d\n",
- ret);
+ if (pt->synth_opts.last_branch) {
+ intel_pt_copy_last_branch_rb(ptq);
+ sample->branch_stack = ptq->last_branch;
+ }
+}
+
+static inline int intel_pt_deliver_synth_event(struct intel_pt *pt,
+ struct intel_pt_queue *ptq,
+ union perf_event *event,
+ struct perf_sample *sample,
+ u64 type)
+{
+ int ret;
+
+ ret = intel_pt_deliver_synth_b_event(pt, event, sample, type);
+
+ if (pt->synth_opts.last_branch)
+ intel_pt_reset_last_branch_rb(ptq);
return ret;
}
static int intel_pt_synth_instruction_sample(struct intel_pt_queue *ptq)
{
- int ret;
struct intel_pt *pt = ptq->pt;
union perf_event *event = ptq->event_buf;
struct perf_sample sample = { .ip = 0, };
- if (pt->synth_opts.initial_skip &&
- pt->num_events++ < pt->synth_opts.initial_skip)
+ if (intel_pt_skip_event(pt))
return 0;
- event->sample.header.type = PERF_RECORD_SAMPLE;
- event->sample.header.misc = PERF_RECORD_MISC_USER;
- event->sample.header.size = sizeof(struct perf_event_header);
-
- if (!pt->timeless_decoding)
- sample.time = tsc_to_perf_time(ptq->timestamp, &pt->tc);
+ intel_pt_prep_sample(pt, ptq, event, &sample);
- sample.cpumode = PERF_RECORD_MISC_USER;
- sample.ip = ptq->state->from_ip;
- sample.pid = ptq->pid;
- sample.tid = ptq->tid;
- sample.addr = ptq->state->to_ip;
sample.id = ptq->pt->instructions_id;
sample.stream_id = ptq->pt->instructions_id;
sample.period = ptq->state->tot_insn_cnt - ptq->last_insn_cnt;
- sample.cpu = ptq->cpu;
- sample.flags = ptq->flags;
- sample.insn_len = ptq->insn_len;
- memcpy(sample.insn, ptq->insn, INTEL_PT_INSN_BUF_SZ);
ptq->last_insn_cnt = ptq->state->tot_insn_cnt;
- if (pt->synth_opts.callchain) {
- thread_stack__sample(ptq->thread, ptq->chain,
- pt->synth_opts.callchain_sz, sample.ip);
- sample.callchain = ptq->chain;
- }
+ return intel_pt_deliver_synth_event(pt, ptq, event, &sample,
+ pt->instructions_sample_type);
+}
- if (pt->synth_opts.last_branch) {
- intel_pt_copy_last_branch_rb(ptq);
- sample.branch_stack = ptq->last_branch;
- }
+static int intel_pt_synth_transaction_sample(struct intel_pt_queue *ptq)
+{
+ struct intel_pt *pt = ptq->pt;
+ union perf_event *event = ptq->event_buf;
+ struct perf_sample sample = { .ip = 0, };
- if (pt->synth_opts.inject) {
- ret = intel_pt_inject_event(event, &sample,
- pt->instructions_sample_type,
- pt->synth_needs_swap);
- if (ret)
- return ret;
- }
+ if (intel_pt_skip_event(pt))
+ return 0;
- ret = perf_session__deliver_synth_event(pt->session, event, &sample);
- if (ret)
- pr_err("Intel Processor Trace: failed to deliver instruction event, error %d\n",
- ret);
+ intel_pt_prep_sample(pt, ptq, event, &sample);
- if (pt->synth_opts.last_branch)
- intel_pt_reset_last_branch_rb(ptq);
+ sample.id = ptq->pt->transactions_id;
+ sample.stream_id = ptq->pt->transactions_id;
- return ret;
+ return intel_pt_deliver_synth_event(pt, ptq, event, &sample,
+ pt->transactions_sample_type);
}
-static int intel_pt_synth_transaction_sample(struct intel_pt_queue *ptq)
+static void intel_pt_prep_p_sample(struct intel_pt *pt,
+ struct intel_pt_queue *ptq,
+ union perf_event *event,
+ struct perf_sample *sample)
+{
+ intel_pt_prep_sample(pt, ptq, event, sample);
+
+ /*
+ * Zero IP is used to mean "trace start" but that is not the case for
+ * power or PTWRITE events with no IP, so clear the flags.
+ */
+ if (!sample->ip)
+ sample->flags = 0;
+}
+
+static int intel_pt_synth_ptwrite_sample(struct intel_pt_queue *ptq)
{
- int ret;
struct intel_pt *pt = ptq->pt;
union perf_event *event = ptq->event_buf;
struct perf_sample sample = { .ip = 0, };
+ struct perf_synth_intel_ptwrite raw;
- if (pt->synth_opts.initial_skip &&
- pt->num_events++ < pt->synth_opts.initial_skip)
+ if (intel_pt_skip_event(pt))
return 0;
- event->sample.header.type = PERF_RECORD_SAMPLE;
- event->sample.header.misc = PERF_RECORD_MISC_USER;
- event->sample.header.size = sizeof(struct perf_event_header);
+ intel_pt_prep_p_sample(pt, ptq, event, &sample);
- if (!pt->timeless_decoding)
- sample.time = tsc_to_perf_time(ptq->timestamp, &pt->tc);
+ sample.id = ptq->pt->ptwrites_id;
+ sample.stream_id = ptq->pt->ptwrites_id;
- sample.cpumode = PERF_RECORD_MISC_USER;
- sample.ip = ptq->state->from_ip;
- sample.pid = ptq->pid;
- sample.tid = ptq->tid;
- sample.addr = ptq->state->to_ip;
- sample.id = ptq->pt->transactions_id;
- sample.stream_id = ptq->pt->transactions_id;
- sample.period = 1;
- sample.cpu = ptq->cpu;
- sample.flags = ptq->flags;
- sample.insn_len = ptq->insn_len;
- memcpy(sample.insn, ptq->insn, INTEL_PT_INSN_BUF_SZ);
+ raw.flags = 0;
+ raw.ip = !!(ptq->state->flags & INTEL_PT_FUP_IP);
+ raw.payload = cpu_to_le64(ptq->state->ptw_payload);
- if (pt->synth_opts.callchain) {
- thread_stack__sample(ptq->thread, ptq->chain,
- pt->synth_opts.callchain_sz, sample.ip);
- sample.callchain = ptq->chain;
- }
+ sample.raw_size = perf_synth__raw_size(raw);
+ sample.raw_data = perf_synth__raw_data(&raw);
- if (pt->synth_opts.last_branch) {
- intel_pt_copy_last_branch_rb(ptq);
- sample.branch_stack = ptq->last_branch;
- }
+ return intel_pt_deliver_synth_event(pt, ptq, event, &sample,
+ pt->ptwrites_sample_type);
+}
- if (pt->synth_opts.inject) {
- ret = intel_pt_inject_event(event, &sample,
- pt->transactions_sample_type,
- pt->synth_needs_swap);
- if (ret)
- return ret;
- }
+static int intel_pt_synth_cbr_sample(struct intel_pt_queue *ptq)
+{
+ struct intel_pt *pt = ptq->pt;
+ union perf_event *event = ptq->event_buf;
+ struct perf_sample sample = { .ip = 0, };
+ struct perf_synth_intel_cbr raw;
+ u32 flags;
- ret = perf_session__deliver_synth_event(pt->session, event, &sample);
- if (ret)
- pr_err("Intel Processor Trace: failed to deliver transaction event, error %d\n",
- ret);
+ if (intel_pt_skip_event(pt))
+ return 0;
- if (pt->synth_opts.last_branch)
- intel_pt_reset_last_branch_rb(ptq);
+ intel_pt_prep_p_sample(pt, ptq, event, &sample);
- return ret;
+ sample.id = ptq->pt->cbr_id;
+ sample.stream_id = ptq->pt->cbr_id;
+
+ flags = (u16)ptq->state->cbr_payload | (pt->max_non_turbo_ratio << 16);
+ raw.flags = cpu_to_le32(flags);
+ raw.freq = cpu_to_le32(raw.cbr * pt->cbr2khz);
+ raw.reserved3 = 0;
+
+ sample.raw_size = perf_synth__raw_size(raw);
+ sample.raw_data = perf_synth__raw_data(&raw);
+
+ return intel_pt_deliver_synth_event(pt, ptq, event, &sample,
+ pt->pwr_events_sample_type);
+}
+
+static int intel_pt_synth_mwait_sample(struct intel_pt_queue *ptq)
+{
+ struct intel_pt *pt = ptq->pt;
+ union perf_event *event = ptq->event_buf;
+ struct perf_sample sample = { .ip = 0, };
+ struct perf_synth_intel_mwait raw;
+
+ if (intel_pt_skip_event(pt))
+ return 0;
+
+ intel_pt_prep_p_sample(pt, ptq, event, &sample);
+
+ sample.id = ptq->pt->mwait_id;
+ sample.stream_id = ptq->pt->mwait_id;
+
+ raw.reserved = 0;
+ raw.payload = cpu_to_le64(ptq->state->mwait_payload);
+
+ sample.raw_size = perf_synth__raw_size(raw);
+ sample.raw_data = perf_synth__raw_data(&raw);
+
+ return intel_pt_deliver_synth_event(pt, ptq, event, &sample,
+ pt->pwr_events_sample_type);
+}
+
+static int intel_pt_synth_pwre_sample(struct intel_pt_queue *ptq)
+{
+ struct intel_pt *pt = ptq->pt;
+ union perf_event *event = ptq->event_buf;
+ struct perf_sample sample = { .ip = 0, };
+ struct perf_synth_intel_pwre raw;
+
+ if (intel_pt_skip_event(pt))
+ return 0;
+
+ intel_pt_prep_p_sample(pt, ptq, event, &sample);
+
+ sample.id = ptq->pt->pwre_id;
+ sample.stream_id = ptq->pt->pwre_id;
+
+ raw.reserved = 0;
+ raw.payload = cpu_to_le64(ptq->state->pwre_payload);
+
+ sample.raw_size = perf_synth__raw_size(raw);
+ sample.raw_data = perf_synth__raw_data(&raw);
+
+ return intel_pt_deliver_synth_event(pt, ptq, event, &sample,
+ pt->pwr_events_sample_type);
+}
+
+static int intel_pt_synth_exstop_sample(struct intel_pt_queue *ptq)
+{
+ struct intel_pt *pt = ptq->pt;
+ union perf_event *event = ptq->event_buf;
+ struct perf_sample sample = { .ip = 0, };
+ struct perf_synth_intel_exstop raw;
+
+ if (intel_pt_skip_event(pt))
+ return 0;
+
+ intel_pt_prep_p_sample(pt, ptq, event, &sample);
+
+ sample.id = ptq->pt->exstop_id;
+ sample.stream_id = ptq->pt->exstop_id;
+
+ raw.flags = 0;
+ raw.ip = !!(ptq->state->flags & INTEL_PT_FUP_IP);
+
+ sample.raw_size = perf_synth__raw_size(raw);
+ sample.raw_data = perf_synth__raw_data(&raw);
+
+ return intel_pt_deliver_synth_event(pt, ptq, event, &sample,
+ pt->pwr_events_sample_type);
+}
+
+static int intel_pt_synth_pwrx_sample(struct intel_pt_queue *ptq)
+{
+ struct intel_pt *pt = ptq->pt;
+ union perf_event *event = ptq->event_buf;
+ struct perf_sample sample = { .ip = 0, };
+ struct perf_synth_intel_pwrx raw;
+
+ if (intel_pt_skip_event(pt))
+ return 0;
+
+ intel_pt_prep_p_sample(pt, ptq, event, &sample);
+
+ sample.id = ptq->pt->pwrx_id;
+ sample.stream_id = ptq->pt->pwrx_id;
+
+ raw.reserved = 0;
+ raw.payload = cpu_to_le64(ptq->state->pwrx_payload);
+
+ sample.raw_size = perf_synth__raw_size(raw);
+ sample.raw_data = perf_synth__raw_data(&raw);
+
+ return intel_pt_deliver_synth_event(pt, ptq, event, &sample,
+ pt->pwr_events_sample_type);
}
static int intel_pt_synth_error(struct intel_pt *pt, int code, int cpu,
PERF_IP_FLAG_INTERRUPT | PERF_IP_FLAG_TX_ABORT));
}
+#define INTEL_PT_PWR_EVT (INTEL_PT_MWAIT_OP | INTEL_PT_PWR_ENTRY | \
+ INTEL_PT_EX_STOP | INTEL_PT_PWR_EXIT | \
+ INTEL_PT_CBR_CHG)
+
static int intel_pt_sample(struct intel_pt_queue *ptq)
{
const struct intel_pt_state *state = ptq->state;
ptq->have_sample = false;
- if (pt->sample_instructions &&
- (state->type & INTEL_PT_INSTRUCTION) &&
- (!pt->synth_opts.initial_skip ||
- pt->num_events++ >= pt->synth_opts.initial_skip)) {
+ if (pt->sample_pwr_events && (state->type & INTEL_PT_PWR_EVT)) {
+ if (state->type & INTEL_PT_CBR_CHG) {
+ err = intel_pt_synth_cbr_sample(ptq);
+ if (err)
+ return err;
+ }
+ if (state->type & INTEL_PT_MWAIT_OP) {
+ err = intel_pt_synth_mwait_sample(ptq);
+ if (err)
+ return err;
+ }
+ if (state->type & INTEL_PT_PWR_ENTRY) {
+ err = intel_pt_synth_pwre_sample(ptq);
+ if (err)
+ return err;
+ }
+ if (state->type & INTEL_PT_EX_STOP) {
+ err = intel_pt_synth_exstop_sample(ptq);
+ if (err)
+ return err;
+ }
+ if (state->type & INTEL_PT_PWR_EXIT) {
+ err = intel_pt_synth_pwrx_sample(ptq);
+ if (err)
+ return err;
+ }
+ }
+
+ if (pt->sample_instructions && (state->type & INTEL_PT_INSTRUCTION)) {
err = intel_pt_synth_instruction_sample(ptq);
if (err)
return err;
}
- if (pt->sample_transactions &&
- (state->type & INTEL_PT_TRANSACTION) &&
- (!pt->synth_opts.initial_skip ||
- pt->num_events++ >= pt->synth_opts.initial_skip)) {
+ if (pt->sample_transactions && (state->type & INTEL_PT_TRANSACTION)) {
err = intel_pt_synth_transaction_sample(ptq);
if (err)
return err;
}
+ if (pt->sample_ptwrites && (state->type & INTEL_PT_PTW)) {
+ err = intel_pt_synth_ptwrite_sample(ptq);
+ if (err)
+ return err;
+ }
+
if (!(state->type & INTEL_PT_BRANCH))
return 0;
NULL);
}
-static int intel_pt_synth_event(struct perf_session *session,
+static int intel_pt_synth_event(struct perf_session *session, const char *name,
struct perf_event_attr *attr, u64 id)
{
struct intel_pt_synth intel_pt_synth;
+ int err;
+
+ pr_debug("Synthesizing '%s' event with id %" PRIu64 " sample type %#" PRIx64 "\n",
+ name, id, (u64)attr->sample_type);
memset(&intel_pt_synth, 0, sizeof(struct intel_pt_synth));
intel_pt_synth.session = session;
- return perf_event__synthesize_attr(&intel_pt_synth.dummy_tool, attr, 1,
- &id, intel_pt_event_synth);
+ err = perf_event__synthesize_attr(&intel_pt_synth.dummy_tool, attr, 1,
+ &id, intel_pt_event_synth);
+ if (err)
+ pr_err("%s: failed to synthesize '%s' event type\n",
+ __func__, name);
+
+ return err;
}
-static int intel_pt_synth_events(struct intel_pt *pt,
- struct perf_session *session)
+static void intel_pt_set_event_name(struct perf_evlist *evlist, u64 id,
+ const char *name)
{
- struct perf_evlist *evlist = session->evlist;
struct perf_evsel *evsel;
- struct perf_event_attr attr;
- bool found = false;
- u64 id;
- int err;
evlist__for_each_entry(evlist, evsel) {
- if (evsel->attr.type == pt->pmu_type && evsel->ids) {
- found = true;
+ if (evsel->id && evsel->id[0] == id) {
+ if (evsel->name)
+ zfree(&evsel->name);
+ evsel->name = strdup(name);
break;
}
}
+}
- if (!found) {
+static struct perf_evsel *intel_pt_evsel(struct intel_pt *pt,
+ struct perf_evlist *evlist)
+{
+ struct perf_evsel *evsel;
+
+ evlist__for_each_entry(evlist, evsel) {
+ if (evsel->attr.type == pt->pmu_type && evsel->ids)
+ return evsel;
+ }
+
+ return NULL;
+}
+
+static int intel_pt_synth_events(struct intel_pt *pt,
+ struct perf_session *session)
+{
+ struct perf_evlist *evlist = session->evlist;
+ struct perf_evsel *evsel = intel_pt_evsel(pt, evlist);
+ struct perf_event_attr attr;
+ u64 id;
+ int err;
+
+ if (!evsel) {
pr_debug("There are no selected events with Intel Processor Trace data\n");
return 0;
}
if (!id)
id = 1;
+ if (pt->synth_opts.branches) {
+ attr.config = PERF_COUNT_HW_BRANCH_INSTRUCTIONS;
+ attr.sample_period = 1;
+ attr.sample_type |= PERF_SAMPLE_ADDR;
+ err = intel_pt_synth_event(session, "branches", &attr, id);
+ if (err)
+ return err;
+ pt->sample_branches = true;
+ pt->branches_sample_type = attr.sample_type;
+ pt->branches_id = id;
+ id += 1;
+ attr.sample_type &= ~(u64)PERF_SAMPLE_ADDR;
+ }
+
+ if (pt->synth_opts.callchain)
+ attr.sample_type |= PERF_SAMPLE_CALLCHAIN;
+ if (pt->synth_opts.last_branch)
+ attr.sample_type |= PERF_SAMPLE_BRANCH_STACK;
+
if (pt->synth_opts.instructions) {
attr.config = PERF_COUNT_HW_INSTRUCTIONS;
if (pt->synth_opts.period_type == PERF_ITRACE_PERIOD_NANOSECS)
intel_pt_ns_to_ticks(pt, pt->synth_opts.period);
else
attr.sample_period = pt->synth_opts.period;
- pt->instructions_sample_period = attr.sample_period;
- if (pt->synth_opts.callchain)
- attr.sample_type |= PERF_SAMPLE_CALLCHAIN;
- if (pt->synth_opts.last_branch)
- attr.sample_type |= PERF_SAMPLE_BRANCH_STACK;
- pr_debug("Synthesizing 'instructions' event with id %" PRIu64 " sample type %#" PRIx64 "\n",
- id, (u64)attr.sample_type);
- err = intel_pt_synth_event(session, &attr, id);
- if (err) {
- pr_err("%s: failed to synthesize 'instructions' event type\n",
- __func__);
+ err = intel_pt_synth_event(session, "instructions", &attr, id);
+ if (err)
return err;
- }
pt->sample_instructions = true;
pt->instructions_sample_type = attr.sample_type;
pt->instructions_id = id;
id += 1;
}
+ attr.sample_type &= ~(u64)PERF_SAMPLE_PERIOD;
+ attr.sample_period = 1;
+
if (pt->synth_opts.transactions) {
attr.config = PERF_COUNT_HW_INSTRUCTIONS;
- attr.sample_period = 1;
- if (pt->synth_opts.callchain)
- attr.sample_type |= PERF_SAMPLE_CALLCHAIN;
- if (pt->synth_opts.last_branch)
- attr.sample_type |= PERF_SAMPLE_BRANCH_STACK;
- pr_debug("Synthesizing 'transactions' event with id %" PRIu64 " sample type %#" PRIx64 "\n",
- id, (u64)attr.sample_type);
- err = intel_pt_synth_event(session, &attr, id);
- if (err) {
- pr_err("%s: failed to synthesize 'transactions' event type\n",
- __func__);
+ err = intel_pt_synth_event(session, "transactions", &attr, id);
+ if (err)
return err;
- }
pt->sample_transactions = true;
+ pt->transactions_sample_type = attr.sample_type;
pt->transactions_id = id;
+ intel_pt_set_event_name(evlist, id, "transactions");
id += 1;
- evlist__for_each_entry(evlist, evsel) {
- if (evsel->id && evsel->id[0] == pt->transactions_id) {
- if (evsel->name)
- zfree(&evsel->name);
- evsel->name = strdup("transactions");
- break;
- }
- }
}
- if (pt->synth_opts.branches) {
- attr.config = PERF_COUNT_HW_BRANCH_INSTRUCTIONS;
- attr.sample_period = 1;
- attr.sample_type |= PERF_SAMPLE_ADDR;
- attr.sample_type &= ~(u64)PERF_SAMPLE_CALLCHAIN;
- attr.sample_type &= ~(u64)PERF_SAMPLE_BRANCH_STACK;
- pr_debug("Synthesizing 'branches' event with id %" PRIu64 " sample type %#" PRIx64 "\n",
- id, (u64)attr.sample_type);
- err = intel_pt_synth_event(session, &attr, id);
- if (err) {
- pr_err("%s: failed to synthesize 'branches' event type\n",
- __func__);
+ attr.type = PERF_TYPE_SYNTH;
+ attr.sample_type |= PERF_SAMPLE_RAW;
+
+ if (pt->synth_opts.ptwrites) {
+ attr.config = PERF_SYNTH_INTEL_PTWRITE;
+ err = intel_pt_synth_event(session, "ptwrite", &attr, id);
+ if (err)
return err;
- }
- pt->sample_branches = true;
- pt->branches_sample_type = attr.sample_type;
- pt->branches_id = id;
+ pt->sample_ptwrites = true;
+ pt->ptwrites_sample_type = attr.sample_type;
+ pt->ptwrites_id = id;
+ intel_pt_set_event_name(evlist, id, "ptwrite");
+ id += 1;
+ }
+
+ if (pt->synth_opts.pwr_events) {
+ pt->sample_pwr_events = true;
+ pt->pwr_events_sample_type = attr.sample_type;
+
+ attr.config = PERF_SYNTH_INTEL_CBR;
+ err = intel_pt_synth_event(session, "cbr", &attr, id);
+ if (err)
+ return err;
+ pt->cbr_id = id;
+ intel_pt_set_event_name(evlist, id, "cbr");
+ id += 1;
+ }
+
+ if (pt->synth_opts.pwr_events && (evsel->attr.config & 0x10)) {
+ attr.config = PERF_SYNTH_INTEL_MWAIT;
+ err = intel_pt_synth_event(session, "mwait", &attr, id);
+ if (err)
+ return err;
+ pt->mwait_id = id;
+ intel_pt_set_event_name(evlist, id, "mwait");
+ id += 1;
+
+ attr.config = PERF_SYNTH_INTEL_PWRE;
+ err = intel_pt_synth_event(session, "pwre", &attr, id);
+ if (err)
+ return err;
+ pt->pwre_id = id;
+ intel_pt_set_event_name(evlist, id, "pwre");
+ id += 1;
+
+ attr.config = PERF_SYNTH_INTEL_EXSTOP;
+ err = intel_pt_synth_event(session, "exstop", &attr, id);
+ if (err)
+ return err;
+ pt->exstop_id = id;
+ intel_pt_set_event_name(evlist, id, "exstop");
+ id += 1;
+
+ attr.config = PERF_SYNTH_INTEL_PWRX;
+ err = intel_pt_synth_event(session, "pwrx", &attr, id);
+ if (err)
+ return err;
+ pt->pwrx_id = id;
+ intel_pt_set_event_name(evlist, id, "pwrx");
+ id += 1;
}
pt->synth_needs_swap = evsel->needs_swap;
intel_pt_log("TSC frequency %"PRIu64"\n", tsc_freq);
intel_pt_log("Maximum non-turbo ratio %u\n",
pt->max_non_turbo_ratio);
+ pt->cbr2khz = tsc_freq / pt->max_non_turbo_ratio / 1000;
}
if (pt->synth_opts.calls)
*/
map_groups__fixup_end(&machine->kmaps);
- if (machine__get_running_kernel_start(machine, &name, &addr)) {
- } else if (maps__set_kallsyms_ref_reloc_sym(machine->vmlinux_maps, name, addr)) {
- machine__destroy_kernel_maps(machine);
- return -1;
+ if (!machine__get_running_kernel_start(machine, &name, &addr)) {
+ if (name &&
+ maps__set_kallsyms_ref_reloc_sym(machine->vmlinux_maps, name, addr)) {
+ machine__destroy_kernel_maps(machine);
+ return -1;
+ }
}
return 0;
#define __PMU_H
#include <linux/bitmap.h>
+#include <linux/compiler.h>
#include <linux/perf_event.h>
#include <stdbool.h>
#include "evsel.h"
bool long_desc, bool details_flag);
bool pmu_have_event(const char *pname, const char *name);
-int perf_pmu__scan_file(struct perf_pmu *pmu, const char *name, const char *fmt,
- ...) __attribute__((format(scanf, 3, 4)));
+int perf_pmu__scan_file(struct perf_pmu *pmu, const char *name, const char *fmt, ...) __scanf(3, 4);
int perf_pmu__test(void);
struct map *map, unsigned long offs)
{
struct symbol *sym;
- u64 addr = tp->address + tp->offset - offs;
+ u64 addr = tp->address - offs;
sym = map__find_symbol(map, addr);
if (!sym)
#ifndef _PROBE_EVENT_H
#define _PROBE_EVENT_H
+#include <linux/compiler.h>
#include <stdbool.h>
#include "intlist.h"
struct symbol *sym);
/* If there is no space to write, returns -E2BIG. */
-int e_snprintf(char *str, size_t size, const char *format, ...)
- __attribute__((format(printf, 3, 4)));
+int e_snprintf(char *str, size_t size, const char *format, ...) __printf(3, 4);
/* Maximum index number of event-name postfix */
#define MAX_EVENT_INDEX 1024
#include <stdbool.h>
#include <errno.h>
#include <linux/bitmap.h>
+#include <linux/compiler.h>
#include <linux/time64.h>
#include "../../perf.h"
static struct tables tables_global;
-static void handler_call_die(const char *handler_name) NORETURN;
+static void handler_call_die(const char *handler_name) __noreturn;
static void handler_call_die(const char *handler_name)
{
PyErr_Print();
if (!(evsel->attr.sample_type & PERF_SAMPLE_CPU)) {
pr_err("File does not contain CPU events. "
- "Remove -c option to proceed.\n");
+ "Remove -C option to proceed.\n");
return -1;
}
}
ret = sort_dimension__add(list, tok, evlist, level);
if (ret == -EINVAL) {
if (!cacheline_size && !strncasecmp(tok, "dcacheline", strlen(tok)))
- error("The \"dcacheline\" --sort key needs to know the cacheline size and it couldn't be determined on this system");
+ pr_err("The \"dcacheline\" --sort key needs to know the cacheline size and it couldn't be determined on this system");
else
- error("Invalid --sort key: `%s'", tok);
+ pr_err("Invalid --sort key: `%s'", tok);
break;
} else if (ret == -ESRCH) {
- error("Unknown --sort key: `%s'", tok);
+ pr_err("Unknown --sort key: `%s'", tok);
break;
}
}
return 0;
if (sort_order[1] == '\0') {
- error("Invalid --sort key: `+'");
+ pr_err("Invalid --sort key: `+'");
return -EINVAL;
}
*/
if (asprintf(&new_sort_order, "%s,%s",
get_default_sort_order(evlist), sort_order + 1) < 0) {
- error("Not enough memory to set up --sort");
+ pr_err("Not enough memory to set up --sort");
return -ENOMEM;
}
str = strdup(sort_keys);
if (str == NULL) {
- error("Not enough memory to setup sort keys");
+ pr_err("Not enough memory to setup sort keys");
return -ENOMEM;
}
if (!is_strict_order(field_order)) {
str = setup_overhead(str);
if (str == NULL) {
- error("Not enough memory to setup overhead keys");
+ pr_err("Not enough memory to setup overhead keys");
return -ENOMEM;
}
}
tok; tok = strtok_r(NULL, ", ", &tmp)) {
ret = output_field_add(list, tok);
if (ret == -EINVAL) {
- error("Invalid --fields key: `%s'", tok);
+ pr_err("Invalid --fields key: `%s'", tok);
break;
} else if (ret == -ESRCH) {
- error("Unknown --fields key: `%s'", tok);
+ pr_err("Unknown --fields key: `%s'", tok);
break;
}
}
strp = str = strdup(field_order);
if (str == NULL) {
- error("Not enough memory to setup output fields");
+ pr_err("Not enough memory to setup output fields");
return -ENOMEM;
}
strp++;
if (!strlen(strp)) {
- error("Invalid --fields key: `+'");
+ pr_err("Invalid --fields key: `+'");
goto out;
}
static struct stats runtime_topdown_slots_retired[NUM_CTX][MAX_NR_CPUS];
static struct stats runtime_topdown_fetch_bubbles[NUM_CTX][MAX_NR_CPUS];
static struct stats runtime_topdown_recovery_bubbles[NUM_CTX][MAX_NR_CPUS];
+static struct stats runtime_smi_num_stats[NUM_CTX][MAX_NR_CPUS];
+static struct stats runtime_aperf_stats[NUM_CTX][MAX_NR_CPUS];
static struct rblist runtime_saved_values;
static bool have_frontend_stalled;
memset(runtime_topdown_slots_issued, 0, sizeof(runtime_topdown_slots_issued));
memset(runtime_topdown_fetch_bubbles, 0, sizeof(runtime_topdown_fetch_bubbles));
memset(runtime_topdown_recovery_bubbles, 0, sizeof(runtime_topdown_recovery_bubbles));
+ memset(runtime_smi_num_stats, 0, sizeof(runtime_smi_num_stats));
+ memset(runtime_aperf_stats, 0, sizeof(runtime_aperf_stats));
next = rb_first(&runtime_saved_values.entries);
while (next) {
update_stats(&runtime_dtlb_cache_stats[ctx][cpu], count[0]);
else if (perf_evsel__match(counter, HW_CACHE, HW_CACHE_ITLB))
update_stats(&runtime_itlb_cache_stats[ctx][cpu], count[0]);
+ else if (perf_stat_evsel__is(counter, SMI_NUM))
+ update_stats(&runtime_smi_num_stats[ctx][cpu], count[0]);
+ else if (perf_stat_evsel__is(counter, APERF))
+ update_stats(&runtime_aperf_stats[ctx][cpu], count[0]);
if (counter->collect_stat) {
struct saved_value *v = saved_value_lookup(counter, cpu, ctx,
return sanitize_val(1.0 - sum);
}
+static void print_smi_cost(int cpu, struct perf_evsel *evsel,
+ struct perf_stat_output_ctx *out)
+{
+ double smi_num, aperf, cycles, cost = 0.0;
+ int ctx = evsel_context(evsel);
+ const char *color = NULL;
+
+ smi_num = avg_stats(&runtime_smi_num_stats[ctx][cpu]);
+ aperf = avg_stats(&runtime_aperf_stats[ctx][cpu]);
+ cycles = avg_stats(&runtime_cycles_stats[ctx][cpu]);
+
+ if ((cycles == 0) || (aperf == 0))
+ return;
+
+ if (smi_num)
+ cost = (aperf - cycles) / aperf * 100.00;
+
+ if (cost > 10)
+ color = PERF_COLOR_RED;
+ out->print_metric(out->ctx, color, "%8.1f%%", "SMI cycles%", cost);
+ out->print_metric(out->ctx, NULL, "%4.0f", "SMI#", smi_num);
+}
+
void perf_stat__print_shadow_stats(struct perf_evsel *evsel,
double avg, int cpu,
struct perf_stat_output_ctx *out)
}
snprintf(unit_buf, sizeof(unit_buf), "%c/sec", unit);
print_metric(ctxp, NULL, "%8.3f", unit_buf, ratio);
+ } else if (perf_stat_evsel__is(evsel, SMI_NUM)) {
+ print_smi_cost(cpu, evsel, out);
} else {
print_metric(ctxp, NULL, NULL, NULL, 0);
}
ID(TOPDOWN_SLOTS_RETIRED, topdown-slots-retired),
ID(TOPDOWN_FETCH_BUBBLES, topdown-fetch-bubbles),
ID(TOPDOWN_RECOVERY_BUBBLES, topdown-recovery-bubbles),
+ ID(SMI_NUM, msr/smi/),
+ ID(APERF, msr/aperf/),
};
#undef ID
PERF_STAT_EVSEL_ID__TOPDOWN_SLOTS_RETIRED,
PERF_STAT_EVSEL_ID__TOPDOWN_FETCH_BUBBLES,
PERF_STAT_EVSEL_ID__TOPDOWN_RECOVERY_BUBBLES,
+ PERF_STAT_EVSEL_ID__SMI_NUM,
+ PERF_STAT_EVSEL_ID__APERF,
PERF_STAT_EVSEL_ID__MAX,
};
#include <stdarg.h>
#include <stddef.h>
#include <string.h>
+#include <linux/compiler.h>
#include <sys/types.h>
extern char strbuf_slopbuf[];
return strbuf_add(sb, s, strlen(s));
}
-__attribute__((format(printf,2,3)))
-int strbuf_addf(struct strbuf *sb, const char *fmt, ...);
+int strbuf_addf(struct strbuf *sb, const char *fmt, ...) __printf(2, 3);
/* XXX: if read fails, any partial read is undone */
ssize_t strbuf_read(struct strbuf *, int fd, ssize_t hint);
#include <errno.h>
#include "../perf.h"
-#include "util.h"
+#include "debug.h"
#include "trace-event.h"
#include "sane_ctype.h"
while (line) {
addr_str = strtok_r(line, ":", &fmt);
if (!addr_str) {
- warning("printk format with empty entry");
+ pr_warning("printk format with empty entry");
break;
}
addr = strtoull(addr_str, NULL, 16);
#include "util.h"
#include "debug.h"
-static void report(const char *prefix, const char *err, va_list params)
-{
- char msg[1024];
- vsnprintf(msg, sizeof(msg), err, params);
- fprintf(stderr, " %s%s\n", prefix, msg);
-}
-
-static NORETURN void usage_builtin(const char *err)
+static __noreturn void usage_builtin(const char *err)
{
fprintf(stderr, "\n Usage: %s\n", err);
exit(129);
}
-static NORETURN void die_builtin(const char *err, va_list params)
-{
- report(" Fatal: ", err, params);
- exit(128);
-}
-
-static void error_builtin(const char *err, va_list params)
-{
- report(" Error: ", err, params);
-}
-
-static void warn_builtin(const char *warn, va_list params)
-{
- report(" Warning: ", warn, params);
-}
-
/* If we are in a dlopen()ed .so write to a global variable would segfault
* (ugh), so keep things static. */
-static void (*usage_routine)(const char *err) NORETURN = usage_builtin;
-static void (*error_routine)(const char *err, va_list params) = error_builtin;
-static void (*warn_routine)(const char *err, va_list params) = warn_builtin;
-
-void set_warning_routine(void (*routine)(const char *err, va_list params))
-{
- warn_routine = routine;
-}
+static void (*usage_routine)(const char *err) __noreturn = usage_builtin;
void usage(const char *err)
{
usage_routine(err);
}
-
-void die(const char *err, ...)
-{
- va_list params;
-
- va_start(params, err);
- die_builtin(err, params);
- va_end(params);
-}
-
-int error(const char *err, ...)
-{
- va_list params;
-
- va_start(params, err);
- error_routine(err, params);
- va_end(params);
- return -1;
-}
-
-void warning(const char *warn, ...)
-{
- va_list params;
-
- va_start(params, warn);
- warn_routine(warn, params);
- va_end(params);
-}
return value;
}
-
-bool find_process(const char *name)
-{
- size_t len = strlen(name);
- DIR *dir;
- struct dirent *d;
- int ret = -1;
-
- dir = opendir(procfs__mountpoint());
- if (!dir)
- return false;
-
- /* Walk through the directory. */
- while (ret && (d = readdir(dir)) != NULL) {
- char path[PATH_MAX];
- char *data;
- size_t size;
-
- if ((d->d_type != DT_DIR) ||
- !strcmp(".", d->d_name) ||
- !strcmp("..", d->d_name))
- continue;
-
- scnprintf(path, sizeof(path), "%s/%s/comm",
- procfs__mountpoint(), d->d_name);
-
- if (filename__read_str(path, &data, &size))
- continue;
-
- ret = strncmp(name, data, len);
- free(data);
- }
-
- closedir(dir);
- return ret ? false : true;
-}
-
static int
fetch_ubuntu_kernel_version(unsigned int *puint)
{
size_t line_len = 0;
char *ptr, *line = NULL;
int version, patchlevel, sublevel, err;
- FILE *vsig = fopen("/proc/version_signature", "r");
+ FILE *vsig;
+
+ if (!puint)
+ return 0;
+ vsig = fopen("/proc/version_signature", "r");
if (!vsig) {
pr_debug("Open /proc/version_signature failed: %s\n",
strerror(errno));
goto errout;
}
- if (puint)
- *puint = (version << 16) + (patchlevel << 8) + sublevel;
+ *puint = (version << 16) + (patchlevel << 8) + sublevel;
err = 0;
errout:
free(line);
str[str_size - 1] = '\0';
}
+ if (!puint || int_ver_ready)
+ return 0;
+
err = sscanf(utsname.release, "%d.%d.%d",
&version, &patchlevel, &sublevel);
return -1;
}
- if (puint && !int_ver_ready)
- *puint = (version << 16) + (patchlevel << 8) + sublevel;
+ *puint = (version << 16) + (patchlevel << 8) + sublevel;
return 0;
}
#ifndef GIT_COMPAT_UTIL_H
#define GIT_COMPAT_UTIL_H
-#define _ALL_SOURCE 1
#define _BSD_SOURCE 1
/* glibc 2.20 deprecates _BSD_SOURCE in favour of _DEFAULT_SOURCE */
#define _DEFAULT_SOURCE 1
#include <stddef.h>
#include <stdlib.h>
#include <stdarg.h>
+#include <linux/compiler.h>
#include <linux/types.h>
-#ifdef __GNUC__
-#define NORETURN __attribute__((__noreturn__))
-#else
-#define NORETURN
-#ifndef __attribute__
-#define __attribute__(x)
-#endif
-#endif
-
/* General helper functions */
-void usage(const char *err) NORETURN;
-void die(const char *err, ...) NORETURN __attribute__((format (printf, 1, 2)));
-int error(const char *err, ...) __attribute__((format (printf, 1, 2)));
-void warning(const char *err, ...) __attribute__((format (printf, 1, 2)));
-
-void set_warning_routine(void (*routine)(const char *err, va_list params));
+void usage(const char *err) __noreturn;
+void die(const char *err, ...) __noreturn __printf(1, 2);
static inline void *zalloc(size_t size)
{
extern unsigned int page_size;
extern int cacheline_size;
-bool find_process(const char *name);
-
int fetch_kernel_version(unsigned int *puint,
char *str, size_t str_sz);
#define KVER_VERSION(x) (((x) >> 16) & 0xff)
{
acpi_status status = AE_OK;
sem_t *sem = (sem_t *) handle;
+ int ret_val;
#ifndef ACPI_USE_ALTERNATE_TIMEOUT
struct timespec time;
- int ret_val;
#endif
if (!sem) {
case ACPI_WAIT_FOREVER:
- if (sem_wait(sem)) {
+ while (((ret_val = sem_wait(sem)) == -1) && (errno == EINTR)) {
+ continue; /* Restart if interrupted */
+ }
+ if (ret_val != 0) {
status = (AE_TIME);
}
break;
while (((ret_val = sem_timedwait(sem, &time)) == -1)
&& (errno == EINTR)) {
- continue;
+ continue; /* Restart if interrupted */
+
}
if (ret_val != 0) {
unsigned res3:21;
unsigned en:1;
} bits;
+ struct {
+ unsigned fid:8;
+ unsigned did:6;
+ unsigned vid:8;
+ unsigned iddval:8;
+ unsigned idddiv:2;
+ unsigned res1:30;
+ unsigned en:1;
+ } fam17h_bits;
unsigned long long val;
};
if (family == 0x12)
t = pstate.val & 0xf;
+ else if (family == 0x17)
+ t = pstate.fam17h_bits.did;
else
t = pstate.bits.did;
static int get_cof(int family, union msr_pstate pstate)
{
int t;
- int fid, did;
+ int fid, did, cof;
did = get_did(family, pstate);
-
- t = 0x10;
- fid = pstate.bits.fid;
- if (family == 0x11)
- t = 0x8;
-
- return (100 * (fid + t)) >> did;
+ if (family == 0x17) {
+ fid = pstate.fam17h_bits.fid;
+ cof = 200 * fid / did;
+ } else {
+ t = 0x10;
+ fid = pstate.bits.fid;
+ if (family == 0x11)
+ t = 0x8;
+ cof = (100 * (fid + t)) >> did;
+ }
+ return cof;
}
/* Needs:
#define CPUPOWER_CAP_IS_SNB 0x00000020
#define CPUPOWER_CAP_INTEL_IDA 0x00000040
+#define CPUPOWER_AMD_CPBDIS 0x02000000
+
#define MAX_HW_PSTATES 10
struct cpupower_cpu_info {
#include "helpers/helpers.h"
+#define MSR_AMD_HWCR 0xc0010015
+
int cpufreq_has_boost_support(unsigned int cpu, int *support, int *active,
int *states)
{
struct cpupower_cpu_info cpu_info;
int ret;
+ unsigned long long val;
*support = *active = *states = 0;
if (cpupower_cpu_info.caps & CPUPOWER_CAP_AMD_CBP) {
*support = 1;
- amd_pci_get_num_boost_states(active, states);
- if (ret <= 0)
- return ret;
- *support = 1;
+
+ /* AMD Family 0x17 does not utilize PCI D18F4 like prior
+ * families and has no fixed discrete boost states but
+ * has Hardware determined variable increments instead.
+ */
+
+ if (cpu_info.family == 0x17) {
+ if (!read_msr(cpu, MSR_AMD_HWCR, &val)) {
+ if (!(val & CPUPOWER_AMD_CPBDIS))
+ *active = 1;
+ }
+ } else {
+ ret = amd_pci_get_num_boost_states(active, states);
+ if (ret)
+ return ret;
+ }
} else if (cpupower_cpu_info.caps & CPUPOWER_CAP_INTEL_IDA)
*support = *active = 1;
return 0;
unsigned int dump_only;
unsigned int do_snb_cstates;
unsigned int do_knl_cstates;
-unsigned int do_skl_residency;
unsigned int do_slm_cstates;
unsigned int use_c1_residency_msr;
unsigned int has_aperf;
unsigned int crystal_hz;
unsigned long long tsc_hz;
int base_cpu;
+int do_migrate;
double discover_bclk(unsigned int family, unsigned int model);
unsigned int has_hwp; /* IA32_PM_ENABLE, IA32_HWP_CAPABILITIES */
/* IA32_HWP_REQUEST, IA32_HWP_STATUS */
#define MAX_ADDED_COUNTERS 16
struct thread_data {
+ struct timeval tv_begin;
+ struct timeval tv_end;
unsigned long long tsc;
unsigned long long aperf;
unsigned long long mperf;
int cpu_migrate(int cpu)
{
+ if (!do_migrate)
+ return 0;
+
CPU_ZERO_S(cpu_affinity_setsize, cpu_affinity_set);
CPU_SET_S(cpu, cpu_affinity_setsize, cpu_affinity_set);
if (sched_setaffinity(0, cpu_affinity_setsize, cpu_affinity_set) == -1)
{ 0x0, "CPU" },
{ 0x0, "Mod%c6" },
{ 0x0, "sysfs" },
+ { 0x0, "Totl%C0" },
+ { 0x0, "Any%C0" },
+ { 0x0, "GFX%C0" },
+ { 0x0, "CPUGFX%" },
};
+
+
#define MAX_BIC (sizeof(bic) / sizeof(struct msr_counter))
#define BIC_Package (1ULL << 0)
#define BIC_Avg_MHz (1ULL << 1)
#define BIC_CPU (1ULL << 36)
#define BIC_Mod_c6 (1ULL << 37)
#define BIC_sysfs (1ULL << 38)
+#define BIC_Totl_c0 (1ULL << 39)
+#define BIC_Any_c0 (1ULL << 40)
+#define BIC_GFX_c0 (1ULL << 41)
+#define BIC_CPUGFX (1ULL << 42)
unsigned long long bic_enabled = 0xFFFFFFFFFFFFFFFFULL;
unsigned long long bic_present = BIC_sysfs;
struct msr_counter *mp;
int printed = 0;
+ if (debug)
+ outp += sprintf(outp, "usec %s", delim);
if (DO_BIC(BIC_Package))
outp += sprintf(outp, "%sPackage", (printed++ ? delim : ""));
if (DO_BIC(BIC_Core))
if (DO_BIC(BIC_GFXMHz))
outp += sprintf(outp, "%sGFXMHz", (printed++ ? delim : ""));
- if (do_skl_residency) {
+ if (DO_BIC(BIC_Totl_c0))
outp += sprintf(outp, "%sTotl%%C0", (printed++ ? delim : ""));
+ if (DO_BIC(BIC_Any_c0))
outp += sprintf(outp, "%sAny%%C0", (printed++ ? delim : ""));
+ if (DO_BIC(BIC_GFX_c0))
outp += sprintf(outp, "%sGFX%%C0", (printed++ ? delim : ""));
+ if (DO_BIC(BIC_CPUGFX))
outp += sprintf(outp, "%sCPUGFX%%", (printed++ ? delim : ""));
- }
if (DO_BIC(BIC_Pkgpc2))
outp += sprintf(outp, "%sPkg%%pc2", (printed++ ? delim : ""));
(cpu_subset && !CPU_ISSET_S(t->cpu_id, cpu_subset_size, cpu_subset)))
return 0;
+ if (debug) {
+ /* on each row, print how many usec each timestamp took to gather */
+ struct timeval tv;
+
+ timersub(&t->tv_end, &t->tv_begin, &tv);
+ outp += sprintf(outp, "%5ld\t", tv.tv_sec * 1000000 + tv.tv_usec);
+ }
+
interval_float = tv_delta.tv_sec + tv_delta.tv_usec/1000000.0;
tsc = t->tsc * tsc_tweak;
outp += sprintf(outp, "%s%d", (printed++ ? delim : ""), p->gfx_mhz);
/* Totl%C0, Any%C0 GFX%C0 CPUGFX% */
- if (do_skl_residency) {
+ if (DO_BIC(BIC_Totl_c0))
outp += sprintf(outp, "%s%.2f", (printed++ ? delim : ""), 100.0 * p->pkg_wtd_core_c0/tsc);
+ if (DO_BIC(BIC_Any_c0))
outp += sprintf(outp, "%s%.2f", (printed++ ? delim : ""), 100.0 * p->pkg_any_core_c0/tsc);
+ if (DO_BIC(BIC_GFX_c0))
outp += sprintf(outp, "%s%.2f", (printed++ ? delim : ""), 100.0 * p->pkg_any_gfxe_c0/tsc);
+ if (DO_BIC(BIC_CPUGFX))
outp += sprintf(outp, "%s%.2f", (printed++ ? delim : ""), 100.0 * p->pkg_both_core_gfxe_c0/tsc);
- }
if (DO_BIC(BIC_Pkgpc2))
outp += sprintf(outp, "%s%.2f", (printed++ ? delim : ""), 100.0 * p->pc2/tsc);
int i;
struct msr_counter *mp;
- if (do_skl_residency) {
+
+ if (DO_BIC(BIC_Totl_c0))
old->pkg_wtd_core_c0 = new->pkg_wtd_core_c0 - old->pkg_wtd_core_c0;
+ if (DO_BIC(BIC_Any_c0))
old->pkg_any_core_c0 = new->pkg_any_core_c0 - old->pkg_any_core_c0;
+ if (DO_BIC(BIC_GFX_c0))
old->pkg_any_gfxe_c0 = new->pkg_any_gfxe_c0 - old->pkg_any_gfxe_c0;
+ if (DO_BIC(BIC_CPUGFX))
old->pkg_both_core_gfxe_c0 = new->pkg_both_core_gfxe_c0 - old->pkg_both_core_gfxe_c0;
- }
+
old->pc2 = new->pc2 - old->pc2;
if (DO_BIC(BIC_Pkgpc3))
old->pc3 = new->pc3 - old->pc3;
if (!(t->flags & CPU_IS_FIRST_CORE_IN_PACKAGE))
return 0;
- if (do_skl_residency) {
+ if (DO_BIC(BIC_Totl_c0))
average.packages.pkg_wtd_core_c0 += p->pkg_wtd_core_c0;
+ if (DO_BIC(BIC_Any_c0))
average.packages.pkg_any_core_c0 += p->pkg_any_core_c0;
+ if (DO_BIC(BIC_GFX_c0))
average.packages.pkg_any_gfxe_c0 += p->pkg_any_gfxe_c0;
+ if (DO_BIC(BIC_CPUGFX))
average.packages.pkg_both_core_gfxe_c0 += p->pkg_both_core_gfxe_c0;
- }
average.packages.pc2 += p->pc2;
if (DO_BIC(BIC_Pkgpc3))
average.cores.c7 /= topo.num_cores;
average.cores.mc6_us /= topo.num_cores;
- if (do_skl_residency) {
+ if (DO_BIC(BIC_Totl_c0))
average.packages.pkg_wtd_core_c0 /= topo.num_packages;
+ if (DO_BIC(BIC_Any_c0))
average.packages.pkg_any_core_c0 /= topo.num_packages;
+ if (DO_BIC(BIC_GFX_c0))
average.packages.pkg_any_gfxe_c0 /= topo.num_packages;
+ if (DO_BIC(BIC_CPUGFX))
average.packages.pkg_both_core_gfxe_c0 /= topo.num_packages;
- }
average.packages.pc2 /= topo.num_packages;
if (DO_BIC(BIC_Pkgpc3))
struct msr_counter *mp;
int i;
+
+ gettimeofday(&t->tv_begin, (struct timezone *)NULL);
+
if (cpu_migrate(cpu)) {
fprintf(outf, "Could not migrate to CPU %d\n", cpu);
return -1;
/* collect core counters only for 1st thread in core */
if (!(t->flags & CPU_IS_FIRST_THREAD_IN_CORE))
- return 0;
+ goto done;
if (DO_BIC(BIC_CPU_c3) && !do_slm_cstates && !do_knl_cstates) {
if (get_msr(cpu, MSR_CORE_C3_RESIDENCY, &c->c3))
/* collect package counters only for 1st core in package */
if (!(t->flags & CPU_IS_FIRST_CORE_IN_PACKAGE))
- return 0;
+ goto done;
- if (do_skl_residency) {
+ if (DO_BIC(BIC_Totl_c0)) {
if (get_msr(cpu, MSR_PKG_WEIGHTED_CORE_C0_RES, &p->pkg_wtd_core_c0))
return -10;
+ }
+ if (DO_BIC(BIC_Any_c0)) {
if (get_msr(cpu, MSR_PKG_ANY_CORE_C0_RES, &p->pkg_any_core_c0))
return -11;
+ }
+ if (DO_BIC(BIC_GFX_c0)) {
if (get_msr(cpu, MSR_PKG_ANY_GFXE_C0_RES, &p->pkg_any_gfxe_c0))
return -12;
+ }
+ if (DO_BIC(BIC_CPUGFX)) {
if (get_msr(cpu, MSR_PKG_BOTH_CORE_GFXE_C0_RES, &p->pkg_both_core_gfxe_c0))
return -13;
}
if (get_mp(cpu, mp, &p->counter[i]))
return -10;
}
+done:
+ gettimeofday(&t->tv_end, (struct timezone *)NULL);
return 0;
}
{
unsigned long long msr;
+ if (!genuine_intel)
+ return;
+
if (!get_msr(base_cpu, MSR_IA32_MISC_ENABLE, &msr))
fprintf(outf, "cpu%d: MSR_IA32_MISC_ENABLE: 0x%08llx (%sTCC %sEIST %sMWAIT %sPREFETCH %sTURBO)\n",
base_cpu, msr,
BIC_PRESENT(BIC_Pkgpc10);
}
do_irtl_hsw = has_hsw_msrs(family, model);
- do_skl_residency = has_skl_msrs(family, model);
+ if (has_skl_msrs(family, model)) {
+ BIC_PRESENT(BIC_Totl_c0);
+ BIC_PRESENT(BIC_Any_c0);
+ BIC_PRESENT(BIC_GFX_c0);
+ BIC_PRESENT(BIC_CPUGFX);
+ }
do_slm_cstates = is_slm(family, model);
do_knl_cstates = is_knl(family, model);
}
void print_version() {
- fprintf(outf, "turbostat version 17.04.12"
+ fprintf(outf, "turbostat version 17.06.23"
" - Len Brown <lenb@kernel.org>\n");
}
{"hide", required_argument, 0, 'H'}, // meh, -h taken by --help
{"Joules", no_argument, 0, 'J'},
{"list", no_argument, 0, 'l'},
+ {"migrate", no_argument, 0, 'm'},
{"out", required_argument, 0, 'o'},
{"quiet", no_argument, 0, 'q'},
{"show", required_argument, 0, 's'},
progname = argv[0];
- while ((opt = getopt_long_only(argc, argv, "+C:c:Ddhi:JM:m:o:qST:v",
+ while ((opt = getopt_long_only(argc, argv, "+C:c:Ddhi:Jmo:qST:v",
long_options, &option_index)) != -1) {
switch (opt) {
case 'a':
list_header_only++;
quiet++;
break;
+ case 'm':
+ do_migrate = 1;
+ break;
case 'o':
outf = fopen_or_die(optarg, "w");
break;
-DESTDIR ?=
+CC = $(CROSS_COMPILE)gcc
+BUILD_OUTPUT := $(CURDIR)
+PREFIX := /usr
+DESTDIR :=
+
+ifeq ("$(origin O)", "command line")
+ BUILD_OUTPUT := $(O)
+endif
x86_energy_perf_policy : x86_energy_perf_policy.c
+CFLAGS += -Wall
+CFLAGS += -DMSRHEADER='"../../../../arch/x86/include/asm/msr-index.h"'
+
+%: %.c
+ @mkdir -p $(BUILD_OUTPUT)
+ $(CC) $(CFLAGS) $< -o $(BUILD_OUTPUT)/$@
+.PHONY : clean
clean :
- rm -f x86_energy_perf_policy
+ @rm -f $(BUILD_OUTPUT)/x86_energy_perf_policy
+
+install : x86_energy_perf_policy
+ install -d $(DESTDIR)$(PREFIX)/bin
+ install $(BUILD_OUTPUT)/x86_energy_perf_policy $(DESTDIR)$(PREFIX)/bin/x86_energy_perf_policy
+ install -d $(DESTDIR)$(PREFIX)/share/man/man8
+ install x86_energy_perf_policy.8 $(DESTDIR)$(PREFIX)/share/man/man8
-install :
- install x86_energy_perf_policy ${DESTDIR}/usr/bin/
- install x86_energy_perf_policy.8 ${DESTDIR}/usr/share/man/man8/
-.\" This page Copyright (C) 2010 Len Brown <len.brown@intel.com>
+.\" This page Copyright (C) 2010 - 2015 Len Brown <len.brown@intel.com>
.\" Distributed under the GPL, Copyleft 1994.
.TH X86_ENERGY_PERF_POLICY 8
.SH NAME
-x86_energy_perf_policy \- read or write MSR_IA32_ENERGY_PERF_BIAS
+x86_energy_perf_policy \- Manage Energy vs. Performance Policy via x86 Model Specific Registers
.SH SYNOPSIS
-.ft B
.B x86_energy_perf_policy
-.RB [ "\-c cpu" ]
-.RB [ "\-v" ]
-.RB "\-r"
+.RB "[ options ] [ scope ] [field \ value]"
.br
-.B x86_energy_perf_policy
-.RB [ "\-c cpu" ]
-.RB [ "\-v" ]
-.RB 'performance'
+.RB "scope: \-\-cpu\ cpu-list | \-\-pkg\ pkg-list"
.br
-.B x86_energy_perf_policy
-.RB [ "\-c cpu" ]
-.RB [ "\-v" ]
-.RB 'normal'
+.RB "cpu-list, pkg-list: # | #,# | #-# | all"
.br
-.B x86_energy_perf_policy
-.RB [ "\-c cpu" ]
-.RB [ "\-v" ]
-.RB 'powersave'
+.RB "field: \-\-all | \-\-epb | \-\-hwp-epp | \-\-hwp-min | \-\-hwp-max | \-\-hwp-desired"
.br
-.B x86_energy_perf_policy
-.RB [ "\-c cpu" ]
-.RB [ "\-v" ]
-.RB n
+.RB "other: (\-\-force | \-\-hwp-enable | \-\-turbo-enable) value)"
.br
+.RB "value: # | default | performance | balance-performance | balance-power | power"
.SH DESCRIPTION
\fBx86_energy_perf_policy\fP
-allows software to convey
-its policy for the relative importance of performance
-versus energy savings to the processor.
+displays and updates energy-performance policy settings specific to
+Intel Architecture Processors. Settings are accessed via Model Specific Register (MSR)
+updates, no matter if the Linux cpufreq sub-system is enabled or not.
-The processor uses this information in model-specific ways
-when it must select trade-offs between performance and
-energy efficiency.
+Policy in MSR_IA32_ENERGY_PERF_BIAS (EPB)
+may affect a wide range of hardware decisions,
+such as how aggressively the hardware enters and exits CPU idle states (C-states)
+and Processor Performance States (P-states).
+This policy hint does not replace explicit OS C-state and P-state selection.
+Rather, it tells the hardware how aggressively to implement those selections.
+Further, it allows the OS to influence energy/performance trade-offs where there
+is no software interface, such as in the opportunistic "turbo-mode" P-state range.
+Note that MSR_IA32_ENERGY_PERF_BIAS is defined per CPU,
+but some implementations
+share a single MSR among all CPUs in each processor package.
+On those systems, a write to EPB on one processor will
+be visible, and will have an effect, on all CPUs
+in the same processor package.
-This policy hint does not supersede Processor Performance states
-(P-states) or CPU Idle power states (C-states), but allows
-software to have influence where it would otherwise be unable
-to express a preference.
+Hardware P-States (HWP) are effectively an expansion of hardware
+P-state control from the opportunistic turbo-mode P-state range
+to include the entire range of available P-states.
+On Broadwell Xeon, the initial HWP implementation, EBP influenced HWP.
+That influence was removed in subsequent generations,
+where it was moved to the
+Energy_Performance_Preference (EPP) field in
+a pair of dedicated MSRs -- MSR_IA32_HWP_REQUEST and MSR_IA32_HWP_REQUEST_PKG.
-For example, this setting may tell the hardware how
-aggressively or conservatively to control frequency
-in the "turbo range" above the explicitly OS-controlled
-P-state frequency range. It may also tell the hardware
-how aggressively is should enter the OS requested C-states.
+EPP is the most commonly managed knob in HWP mode,
+but MSR_IA32_HWP_REQUEST also allows the user to specify
+minimum-frequency for Quality-of-Service,
+and maximum-frequency for power-capping.
+MSR_IA32_HWP_REQUEST is defined per-CPU.
-Support for this feature is indicated by CPUID.06H.ECX.bit3
-per the Intel Architectures Software Developer's Manual.
+MSR_IA32_HWP_REQUEST_PKG has the same capability as MSR_IA32_HWP_REQUEST,
+but it can simultaneously set the default policy for all CPUs within a package.
+A bit in per-CPU MSR_IA32_HWP_REQUEST indicates whether it is
+over-ruled-by or exempt-from MSR_IA32_HWP_REQUEST_PKG.
-.SS Options
-\fB-c\fP limits operation to a single CPU.
-The default is to operate on all CPUs.
-Note that MSR_IA32_ENERGY_PERF_BIAS is defined per
-logical processor, but that the initial implementations
-of the MSR were shared among all processors in each package.
-.PP
-\fB-v\fP increases verbosity. By default
-x86_energy_perf_policy is silent.
-.PP
-\fB-r\fP is for "read-only" mode - the unchanged state
-is read and displayed.
+MSR_HWP_CAPABILITIES shows the default values for the fields
+in MSR_IA32_HWP_REQUEST. It is displayed when no values
+are being written.
+
+.SS SCOPE OPTIONS
.PP
-.I performance
-Set a policy where performance is paramount.
-The processor will be unwilling to sacrifice any performance
-for the sake of energy saving. This is the hardware default.
+\fB-c, --cpu\fP Operate on the MSR_IA32_HWP_REQUEST for each CPU in a CPU-list.
+The CPU-list may be comma-separated CPU numbers, with dash for range
+or the string "all". Eg. '--cpu 1,4,6-8' or '--cpu all'.
+When --cpu is used, \fB--hwp-use-pkg\fP is available, which specifies whether the per-cpu
+MSR_IA32_HWP_REQUEST should be over-ruled by MSR_IA32_HWP_REQUEST_PKG (1),
+or exempt from MSR_IA32_HWP_REQUEST_PKG (0).
+
+\fB-p, --pkg\fP Operate on the MSR_IA32_HWP_REQUEST_PKG for each package in the package-list.
+The list is a string of individual package numbers separated
+by commas, and or ranges of package numbers separated by a dash,
+or the string "all".
+For example '--pkg 1,3' or '--pkg all'
+
+.SS VALUE OPTIONS
.PP
-.I normal
+.I normal | default
Set a policy with a normal balance between performance and energy efficiency.
The processor will tolerate minor performance compromise
for potentially significant energy savings.
-This reasonable default for most desktops and servers.
+This is a reasonable default for most desktops and servers.
+"default" is a synonym for "normal".
.PP
-.I powersave
+.I performance
+Set a policy for maximum performance,
+accepting no performance sacrifice for the benefit of energy efficiency.
+.PP
+.I balance-performance
+Set a policy with a high priority on performance,
+but allowing some performance loss to benefit energy efficiency.
+.PP
+.I balance-power
+Set a policy where the performance and power are balanced.
+This is the default.
+.PP
+.I power
Set a policy where the processor can accept
-a measurable performance hit to maximize energy efficiency.
+a measurable performance impact to maximize energy efficiency.
+
.PP
-.I n
-Set MSR_IA32_ENERGY_PERF_BIAS to the specified number.
-The range of valid numbers is 0-15, where 0 is maximum
-performance and 15 is maximum energy efficiency.
+The following table shows the mapping from the value strings above to actual MSR values.
+This mapping is defined in the Linux-kernel header, msr-index.h.
+.nf
+VALUE STRING EPB EPP
+performance 0 0
+balance-performance 4 128
+normal, default 6 128
+balance-power 8 192
+power 15 255
+.fi
+.PP
+For MSR_IA32_HWP_REQUEST performance fields
+(--hwp-min, --hwp-max, --hwp-desired), the value option
+is in units of 100 MHz, Eg. 12 signifies 1200 MHz.
+
+.SS FIELD OPTIONS
+\fB-a, --all value-string\fP Sets all EPB and EPP and HWP limit fields to the value associated with
+the value-string. In addition, enables turbo-mode and HWP-mode, if they were previous disabled.
+Thus "--all normal" will set a system without cpufreq into a well known configuration.
+.PP
+\fB-B, --epb\fP set EPB per-core or per-package.
+See value strings in the table above.
+.PP
+\fB-d, --debug\fP debug increases verbosity. By default
+x86_energy_perf_policy is silent for updates,
+and verbose for read-only mode.
+.PP
+\fB-P, --hwp-epp\fP set HWP.EPP per-core or per-package.
+See value strings in the table above.
+.PP
+\fB-m, --hwp-min\fP request HWP to not go below the specified core/bus ratio.
+The "default" is the value found in IA32_HWP_CAPABILITIES.min.
+.PP
+\fB-M, --hwp-max\fP request HWP not exceed a the specified core/bus ratio.
+The "default" is the value found in IA32_HWP_CAPABILITIES.max.
+.PP
+\fB-D, --hwp-desired\fP request HWP 'desired' frequency.
+The "normal" setting is 0, which
+corresponds to 'full autonomous' HWP control.
+Non-zero performance values request a specific performance
+level on this processor, specified in multiples of 100 MHz.
+.PP
+\fB-w, --hwp-window\fP specify integer number of microsec
+in the sliding window that HWP uses to maintain average frequency.
+This parameter is meaningful only when the "desired" field above is non-zero.
+Default is 0, allowing the HW to choose.
+.SH OTHER OPTIONS
+.PP
+\fB-f, --force\fP writes the specified values without bounds checking.
+.PP
+\fB-U, --hwp-use-pkg\fP (0 | 1), when used in conjunction with --cpu,
+indicates whether the per-CPU MSR_IA32_HWP_REQUEST should be overruled (1)
+or exempt (0) from per-Package MSR_IA32_HWP_REQUEST_PKG settings.
+The default is exempt.
+.PP
+\fB-H, --hwp-enable\fP enable HardWare-P-state (HWP) mode. Once enabled, system RESET is required to disable HWP mode.
+.PP
+\fB-t, --turbo-enable\fP enable (1) or disable (0) turbo mode.
+.PP
+\fB-v, --version\fP print version and exit.
+.PP
+If no request to change policy is made,
+the default behavior is to read
+and display the current system state,
+including the default capabilities.
+.SH WARNING
+.PP
+This utility writes directly to Model Specific Registers.
+There is no locking or coordination should this utility
+be used to modify HWP limit fields at the same time that
+intel_pstate's sysfs attributes access the same MSRs.
+.PP
+Note that --hwp-desired and --hwp-window are considered experimental.
+Future versions of Linux reserve the right to access these
+fields internally -- potentially conflicting with user-space access.
+.SH EXAMPLE
+.nf
+# sudo x86_energy_perf_policy
+cpu0: EPB 6
+cpu0: HWP_REQ: min 6 max 35 des 0 epp 128 window 0x0 (0*10^0us) use_pkg 0
+cpu0: HWP_CAP: low 1 eff 8 guar 27 high 35
+cpu1: EPB 6
+cpu1: HWP_REQ: min 6 max 35 des 0 epp 128 window 0x0 (0*10^0us) use_pkg 0
+cpu1: HWP_CAP: low 1 eff 8 guar 27 high 35
+cpu2: EPB 6
+cpu2: HWP_REQ: min 6 max 35 des 0 epp 128 window 0x0 (0*10^0us) use_pkg 0
+cpu2: HWP_CAP: low 1 eff 8 guar 27 high 35
+cpu3: EPB 6
+cpu3: HWP_REQ: min 6 max 35 des 0 epp 128 window 0x0 (0*10^0us) use_pkg 0
+cpu3: HWP_CAP: low 1 eff 8 guar 27 high 35
+.fi
.SH NOTES
-.B "x86_energy_perf_policy "
+.B "x86_energy_perf_policy"
runs only as root.
.SH FILES
.ta
.nf
/dev/cpu/*/msr
.fi
-
.SH "SEE ALSO"
+.nf
msr(4)
+Intel(R) 64 and IA-32 Architectures Software Developer's Manual
+.fi
.PP
.SH AUTHORS
.nf
-Written by Len Brown <len.brown@intel.com>
+Len Brown
* policy preference bias on recent X86 processors.
*/
/*
- * Copyright (c) 2010, Intel Corporation.
+ * Copyright (c) 2010 - 2017 Intel Corporation.
* Len Brown <len.brown@intel.com>
*
- * This program is free software; you can redistribute it and/or modify it
- * under the terms and conditions of the GNU General Public License,
- * version 2, as published by the Free Software Foundation.
- *
- * This program is distributed in the hope it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
- * more details.
- *
- * You should have received a copy of the GNU General Public License along with
- * this program; if not, write to the Free Software Foundation, Inc.,
- * 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA.
+ * This program is released under GPL v2
*/
+#define _GNU_SOURCE
+#include MSRHEADER
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
+#include <sched.h>
#include <sys/stat.h>
#include <sys/resource.h>
+#include <getopt.h>
+#include <err.h>
#include <fcntl.h>
#include <signal.h>
#include <sys/time.h>
+#include <limits.h>
#include <stdlib.h>
#include <string.h>
+#include <cpuid.h>
+#include <errno.h>
+
+#define OPTARG_NORMAL (INT_MAX - 1)
+#define OPTARG_POWER (INT_MAX - 2)
+#define OPTARG_BALANCE_POWER (INT_MAX - 3)
+#define OPTARG_BALANCE_PERFORMANCE (INT_MAX - 4)
+#define OPTARG_PERFORMANCE (INT_MAX - 5)
+
+struct msr_hwp_cap {
+ unsigned char highest;
+ unsigned char guaranteed;
+ unsigned char efficient;
+ unsigned char lowest;
+};
-unsigned int verbose; /* set with -v */
-unsigned int read_only; /* set with -r */
+struct msr_hwp_request {
+ unsigned char hwp_min;
+ unsigned char hwp_max;
+ unsigned char hwp_desired;
+ unsigned char hwp_epp;
+ unsigned int hwp_window;
+ unsigned char hwp_use_pkg;
+} req_update;
+
+unsigned int debug;
+unsigned int verbose;
+unsigned int force;
char *progname;
-unsigned long long new_bias;
-int cpu = -1;
+int base_cpu;
+unsigned char update_epb;
+unsigned long long new_epb;
+unsigned char turbo_is_enabled;
+unsigned char update_turbo;
+unsigned char turbo_update_value;
+unsigned char update_hwp_epp;
+unsigned char update_hwp_min;
+unsigned char update_hwp_max;
+unsigned char update_hwp_desired;
+unsigned char update_hwp_window;
+unsigned char update_hwp_use_pkg;
+unsigned char update_hwp_enable;
+#define hwp_update_enabled() (update_hwp_enable | update_hwp_epp | update_hwp_max | update_hwp_min | update_hwp_desired | update_hwp_window | update_hwp_use_pkg)
+int max_cpu_num;
+int max_pkg_num;
+#define MAX_PACKAGES 64
+unsigned int first_cpu_in_pkg[MAX_PACKAGES];
+unsigned long long pkg_present_set;
+unsigned long long pkg_selected_set;
+cpu_set_t *cpu_present_set;
+cpu_set_t *cpu_selected_set;
+int genuine_intel;
+
+size_t cpu_setsize;
+
+char *proc_stat = "/proc/stat";
+
+unsigned int has_epb; /* MSR_IA32_ENERGY_PERF_BIAS */
+unsigned int has_hwp; /* IA32_PM_ENABLE, IA32_HWP_CAPABILITIES */
+ /* IA32_HWP_REQUEST, IA32_HWP_STATUS */
+unsigned int has_hwp_notify; /* IA32_HWP_INTERRUPT */
+unsigned int has_hwp_activity_window; /* IA32_HWP_REQUEST[bits 41:32] */
+unsigned int has_hwp_epp; /* IA32_HWP_REQUEST[bits 31:24] */
+unsigned int has_hwp_request_pkg; /* IA32_HWP_REQUEST_PKG */
+
+unsigned int bdx_highest_ratio;
/*
- * Usage:
- *
- * -c cpu: limit action to a single CPU (default is all CPUs)
- * -v: verbose output (can invoke more than once)
- * -r: read-only, don't change any settings
- *
- * performance
- * Performance is paramount.
- * Unwilling to sacrifice any performance
- * for the sake of energy saving. (hardware default)
- *
- * normal
- * Can tolerate minor performance compromise
- * for potentially significant energy savings.
- * (reasonable default for most desktops and servers)
- *
- * powersave
- * Can tolerate significant performance hit
- * to maximize energy savings.
- *
- * n
- * a numerical value to write to the underlying MSR.
+ * maintain compatibility with original implementation, but don't document it:
*/
void usage(void)
{
- printf("%s: [-c cpu] [-v] "
- "(-r | 'performance' | 'normal' | 'powersave' | n)\n",
- progname);
+ fprintf(stderr, "%s [options] [scope][field value]\n", progname);
+ fprintf(stderr, "scope: --cpu cpu-list [--hwp-use-pkg #] | --pkg pkg-list\n");
+ fprintf(stderr, "field: --all | --epb | --hwp-epp | --hwp-min | --hwp-max | --hwp-desired\n");
+ fprintf(stderr, "other: --hwp-enable | --turbo-enable (0 | 1) | --help | --force\n");
+ fprintf(stderr,
+ "value: ( # | \"normal\" | \"performance\" | \"balance-performance\" | \"balance-power\"| \"power\")\n");
+ fprintf(stderr, "--hwp-window usec\n");
+
+ fprintf(stderr, "Specify only Energy Performance BIAS (legacy usage):\n");
+ fprintf(stderr, "%s: [-c cpu] [-v] (-r | policy-value )\n", progname);
+
exit(1);
}
-#define MSR_IA32_ENERGY_PERF_BIAS 0x000001b0
+/*
+ * If bdx_highest_ratio is set,
+ * then we must translate between MSR format and simple ratio
+ * used on the cmdline.
+ */
+int ratio_2_msr_perf(int ratio)
+{
+ int msr_perf;
+
+ if (!bdx_highest_ratio)
+ return ratio;
+
+ msr_perf = ratio * 255 / bdx_highest_ratio;
+
+ if (debug)
+ fprintf(stderr, "%d = ratio_to_msr_perf(%d)\n", msr_perf, ratio);
+
+ return msr_perf;
+}
+int msr_perf_2_ratio(int msr_perf)
+{
+ int ratio;
+ double d;
+
+ if (!bdx_highest_ratio)
+ return msr_perf;
+
+ d = (double)msr_perf * (double) bdx_highest_ratio / 255.0;
+ d = d + 0.5; /* round */
+ ratio = (int)d;
+
+ if (debug)
+ fprintf(stderr, "%d = msr_perf_ratio(%d) {%f}\n", ratio, msr_perf, d);
+
+ return ratio;
+}
+int parse_cmdline_epb(int i)
+{
+ if (!has_epb)
+ errx(1, "EPB not enabled on this platform");
+
+ update_epb = 1;
+
+ switch (i) {
+ case OPTARG_POWER:
+ return ENERGY_PERF_BIAS_POWERSAVE;
+ case OPTARG_BALANCE_POWER:
+ return ENERGY_PERF_BIAS_BALANCE_POWERSAVE;
+ case OPTARG_NORMAL:
+ return ENERGY_PERF_BIAS_NORMAL;
+ case OPTARG_BALANCE_PERFORMANCE:
+ return ENERGY_PERF_BIAS_BALANCE_PERFORMANCE;
+ case OPTARG_PERFORMANCE:
+ return ENERGY_PERF_BIAS_PERFORMANCE;
+ }
+ if (i < 0 || i > ENERGY_PERF_BIAS_POWERSAVE)
+ errx(1, "--epb must be from 0 to 15");
+ return i;
+}
+
+#define HWP_CAP_LOWEST 0
+#define HWP_CAP_HIGHEST 255
+
+/*
+ * "performance" changes hwp_min to cap.highest
+ * All others leave it at cap.lowest
+ */
+int parse_cmdline_hwp_min(int i)
+{
+ update_hwp_min = 1;
+
+ switch (i) {
+ case OPTARG_POWER:
+ case OPTARG_BALANCE_POWER:
+ case OPTARG_NORMAL:
+ case OPTARG_BALANCE_PERFORMANCE:
+ return HWP_CAP_LOWEST;
+ case OPTARG_PERFORMANCE:
+ return HWP_CAP_HIGHEST;
+ }
+ return i;
+}
+/*
+ * "power" changes hwp_max to cap.lowest
+ * All others leave it at cap.highest
+ */
+int parse_cmdline_hwp_max(int i)
+{
+ update_hwp_max = 1;
+
+ switch (i) {
+ case OPTARG_POWER:
+ return HWP_CAP_LOWEST;
+ case OPTARG_NORMAL:
+ case OPTARG_BALANCE_POWER:
+ case OPTARG_BALANCE_PERFORMANCE:
+ case OPTARG_PERFORMANCE:
+ return HWP_CAP_HIGHEST;
+ }
+ return i;
+}
+/*
+ * for --hwp-des, all strings leave it in autonomous mode
+ * If you want to change it, you need to explicitly pick a value
+ */
+int parse_cmdline_hwp_desired(int i)
+{
+ update_hwp_desired = 1;
+
+ switch (i) {
+ case OPTARG_POWER:
+ case OPTARG_BALANCE_POWER:
+ case OPTARG_BALANCE_PERFORMANCE:
+ case OPTARG_NORMAL:
+ case OPTARG_PERFORMANCE:
+ return 0; /* autonomous */
+ }
+ return i;
+}
+
+int parse_cmdline_hwp_window(int i)
+{
+ unsigned int exponent;
+
+ update_hwp_window = 1;
+
+ switch (i) {
+ case OPTARG_POWER:
+ case OPTARG_BALANCE_POWER:
+ case OPTARG_NORMAL:
+ case OPTARG_BALANCE_PERFORMANCE:
+ case OPTARG_PERFORMANCE:
+ return 0;
+ }
+ if (i < 0 || i > 1270000000) {
+ fprintf(stderr, "--hwp-window: 0 for auto; 1 - 1270000000 usec for window duration\n");
+ usage();
+ }
+ for (exponent = 0; ; ++exponent) {
+ if (debug)
+ printf("%d 10^%d\n", i, exponent);
+
+ if (i <= 127)
+ break;
+
+ i = i / 10;
+ }
+ if (debug)
+ fprintf(stderr, "%d*10^%d: 0x%x\n", i, exponent, (exponent << 7) | i);
+
+ return (exponent << 7) | i;
+}
+int parse_cmdline_hwp_epp(int i)
+{
+ update_hwp_epp = 1;
+
+ switch (i) {
+ case OPTARG_POWER:
+ return HWP_EPP_POWERSAVE;
+ case OPTARG_BALANCE_POWER:
+ return HWP_EPP_BALANCE_POWERSAVE;
+ case OPTARG_NORMAL:
+ case OPTARG_BALANCE_PERFORMANCE:
+ return HWP_EPP_BALANCE_PERFORMANCE;
+ case OPTARG_PERFORMANCE:
+ return HWP_EPP_PERFORMANCE;
+ }
+ if (i < 0 || i > 0xff) {
+ fprintf(stderr, "--hwp-epp must be from 0 to 0xff\n");
+ usage();
+ }
+ return i;
+}
+int parse_cmdline_turbo(int i)
+{
+ update_turbo = 1;
+
+ switch (i) {
+ case OPTARG_POWER:
+ return 0;
+ case OPTARG_NORMAL:
+ case OPTARG_BALANCE_POWER:
+ case OPTARG_BALANCE_PERFORMANCE:
+ case OPTARG_PERFORMANCE:
+ return 1;
+ }
+ if (i < 0 || i > 1) {
+ fprintf(stderr, "--turbo-enable: 1 to enable, 0 to disable\n");
+ usage();
+ }
+ return i;
+}
+
+int parse_optarg_string(char *s)
+{
+ int i;
+ char *endptr;
+
+ if (!strncmp(s, "default", 7))
+ return OPTARG_NORMAL;
+
+ if (!strncmp(s, "normal", 6))
+ return OPTARG_NORMAL;
+
+ if (!strncmp(s, "power", 9))
+ return OPTARG_POWER;
+
+ if (!strncmp(s, "balance-power", 17))
+ return OPTARG_BALANCE_POWER;
+
+ if (!strncmp(s, "balance-performance", 19))
+ return OPTARG_BALANCE_PERFORMANCE;
+
+ if (!strncmp(s, "performance", 11))
+ return OPTARG_PERFORMANCE;
+
+ i = strtol(s, &endptr, 0);
+ if (s == endptr) {
+ fprintf(stderr, "no digits in \"%s\"\n", s);
+ usage();
+ }
+ if (i == LONG_MIN || i == LONG_MAX)
+ errx(-1, "%s", s);
+
+ if (i > 0xFF)
+ errx(-1, "%d (0x%x) must be < 256", i, i);
+
+ if (i < 0)
+ errx(-1, "%d (0x%x) must be >= 0", i, i);
+ return i;
+}
+
+void parse_cmdline_all(char *s)
+{
+ force++;
+ update_hwp_enable = 1;
+ req_update.hwp_min = parse_cmdline_hwp_min(parse_optarg_string(s));
+ req_update.hwp_max = parse_cmdline_hwp_max(parse_optarg_string(s));
+ req_update.hwp_epp = parse_cmdline_hwp_epp(parse_optarg_string(s));
+ if (has_epb)
+ new_epb = parse_cmdline_epb(parse_optarg_string(s));
+ turbo_update_value = parse_cmdline_turbo(parse_optarg_string(s));
+ req_update.hwp_desired = parse_cmdline_hwp_desired(parse_optarg_string(s));
+ req_update.hwp_window = parse_cmdline_hwp_window(parse_optarg_string(s));
+}
+
+void validate_cpu_selected_set(void)
+{
+ int cpu;
+
+ if (CPU_COUNT_S(cpu_setsize, cpu_selected_set) == 0)
+ errx(0, "no CPUs requested");
+
+ for (cpu = 0; cpu <= max_cpu_num; ++cpu) {
+ if (CPU_ISSET_S(cpu, cpu_setsize, cpu_selected_set))
+ if (!CPU_ISSET_S(cpu, cpu_setsize, cpu_present_set))
+ errx(1, "Requested cpu% is not present", cpu);
+ }
+}
+
+void parse_cmdline_cpu(char *s)
+{
+ char *startp, *endp;
+ int cpu = 0;
+
+ if (pkg_selected_set) {
+ usage();
+ errx(1, "--cpu | --pkg");
+ }
+ cpu_selected_set = CPU_ALLOC((max_cpu_num + 1));
+ if (cpu_selected_set == NULL)
+ err(1, "cpu_selected_set");
+ CPU_ZERO_S(cpu_setsize, cpu_selected_set);
+
+ for (startp = s; startp && *startp;) {
+
+ if (*startp == ',') {
+ startp++;
+ continue;
+ }
+
+ if (*startp == '-') {
+ int end_cpu;
-#define BIAS_PERFORMANCE 0
-#define BIAS_BALANCE 6
-#define BIAS_POWERSAVE 15
+ startp++;
+ end_cpu = strtol(startp, &endp, 10);
+ if (startp == endp)
+ continue;
+
+ while (cpu <= end_cpu) {
+ if (cpu > max_cpu_num)
+ errx(1, "Requested cpu%d exceeds max cpu%d", cpu, max_cpu_num);
+ CPU_SET_S(cpu, cpu_setsize, cpu_selected_set);
+ cpu++;
+ }
+ startp = endp;
+ continue;
+ }
+
+ if (strncmp(startp, "all", 3) == 0) {
+ for (cpu = 0; cpu <= max_cpu_num; cpu += 1) {
+ if (CPU_ISSET_S(cpu, cpu_setsize, cpu_present_set))
+ CPU_SET_S(cpu, cpu_setsize, cpu_selected_set);
+ }
+ startp += 3;
+ if (*startp == 0)
+ break;
+ }
+ /* "--cpu even" is not documented */
+ if (strncmp(startp, "even", 4) == 0) {
+ for (cpu = 0; cpu <= max_cpu_num; cpu += 2) {
+ if (CPU_ISSET_S(cpu, cpu_setsize, cpu_present_set))
+ CPU_SET_S(cpu, cpu_setsize, cpu_selected_set);
+ }
+ startp += 4;
+ if (*startp == 0)
+ break;
+ }
+
+ /* "--cpu odd" is not documented */
+ if (strncmp(startp, "odd", 3) == 0) {
+ for (cpu = 1; cpu <= max_cpu_num; cpu += 2) {
+ if (CPU_ISSET_S(cpu, cpu_setsize, cpu_present_set))
+ CPU_SET_S(cpu, cpu_setsize, cpu_selected_set);
+ }
+ startp += 3;
+ if (*startp == 0)
+ break;
+ }
+
+ cpu = strtol(startp, &endp, 10);
+ if (startp == endp)
+ errx(1, "--cpu cpu-set: confused by '%s'", startp);
+ if (cpu > max_cpu_num)
+ errx(1, "Requested cpu%d exceeds max cpu%d", cpu, max_cpu_num);
+ CPU_SET_S(cpu, cpu_setsize, cpu_selected_set);
+ startp = endp;
+ }
+
+ validate_cpu_selected_set();
+
+}
+
+void parse_cmdline_pkg(char *s)
+{
+ char *startp, *endp;
+ int pkg = 0;
+
+ if (cpu_selected_set) {
+ usage();
+ errx(1, "--pkg | --cpu");
+ }
+ pkg_selected_set = 0;
+
+ for (startp = s; startp && *startp;) {
+
+ if (*startp == ',') {
+ startp++;
+ continue;
+ }
+
+ if (*startp == '-') {
+ int end_pkg;
+
+ startp++;
+ end_pkg = strtol(startp, &endp, 10);
+ if (startp == endp)
+ continue;
+
+ while (pkg <= end_pkg) {
+ if (pkg > max_pkg_num)
+ errx(1, "Requested pkg%d exceeds max pkg%d", pkg, max_pkg_num);
+ pkg_selected_set |= 1 << pkg;
+ pkg++;
+ }
+ startp = endp;
+ continue;
+ }
+
+ if (strncmp(startp, "all", 3) == 0) {
+ pkg_selected_set = pkg_present_set;
+ return;
+ }
+
+ pkg = strtol(startp, &endp, 10);
+ if (pkg > max_pkg_num)
+ errx(1, "Requested pkg%d Exceeds max pkg%d", pkg, max_pkg_num);
+ pkg_selected_set |= 1 << pkg;
+ startp = endp;
+ }
+}
+
+void for_packages(unsigned long long pkg_set, int (func)(int))
+{
+ int pkg_num;
+
+ for (pkg_num = 0; pkg_num <= max_pkg_num; ++pkg_num) {
+ if (pkg_set & (1UL << pkg_num))
+ func(pkg_num);
+ }
+}
+
+void print_version(void)
+{
+ printf("x86_energy_perf_policy 17.05.11 (C) Len Brown <len.brown@intel.com>\n");
+}
void cmdline(int argc, char **argv)
{
int opt;
+ int option_index = 0;
+
+ static struct option long_options[] = {
+ {"all", required_argument, 0, 'a'},
+ {"cpu", required_argument, 0, 'c'},
+ {"pkg", required_argument, 0, 'p'},
+ {"debug", no_argument, 0, 'd'},
+ {"hwp-desired", required_argument, 0, 'D'},
+ {"epb", required_argument, 0, 'B'},
+ {"force", no_argument, 0, 'f'},
+ {"hwp-enable", no_argument, 0, 'e'},
+ {"help", no_argument, 0, 'h'},
+ {"hwp-epp", required_argument, 0, 'P'},
+ {"hwp-min", required_argument, 0, 'm'},
+ {"hwp-max", required_argument, 0, 'M'},
+ {"read", no_argument, 0, 'r'},
+ {"turbo-enable", required_argument, 0, 't'},
+ {"hwp-use-pkg", required_argument, 0, 'u'},
+ {"version", no_argument, 0, 'v'},
+ {"hwp-window", required_argument, 0, 'w'},
+ {0, 0, 0, 0 }
+ };
progname = argv[0];
- while ((opt = getopt(argc, argv, "+rvc:")) != -1) {
+ while ((opt = getopt_long_only(argc, argv, "+a:c:dD:E:e:f:m:M:rt:u:vw",
+ long_options, &option_index)) != -1) {
switch (opt) {
+ case 'a':
+ parse_cmdline_all(optarg);
+ break;
+ case 'B':
+ new_epb = parse_cmdline_epb(parse_optarg_string(optarg));
+ break;
case 'c':
- cpu = atoi(optarg);
+ parse_cmdline_cpu(optarg);
+ break;
+ case 'e':
+ update_hwp_enable = 1;
+ break;
+ case 'h':
+ usage();
+ break;
+ case 'd':
+ debug++;
+ verbose++;
+ break;
+ case 'f':
+ force++;
+ break;
+ case 'D':
+ req_update.hwp_desired = parse_cmdline_hwp_desired(parse_optarg_string(optarg));
+ break;
+ case 'm':
+ req_update.hwp_min = parse_cmdline_hwp_min(parse_optarg_string(optarg));
+ break;
+ case 'M':
+ req_update.hwp_max = parse_cmdline_hwp_max(parse_optarg_string(optarg));
+ break;
+ case 'p':
+ parse_cmdline_pkg(optarg);
+ break;
+ case 'P':
+ req_update.hwp_epp = parse_cmdline_hwp_epp(parse_optarg_string(optarg));
break;
case 'r':
- read_only = 1;
+ /* v1 used -r to specify read-only mode, now the default */
+ break;
+ case 't':
+ turbo_update_value = parse_cmdline_turbo(parse_optarg_string(optarg));
+ break;
+ case 'u':
+ update_hwp_use_pkg++;
+ if (atoi(optarg) == 0)
+ req_update.hwp_use_pkg = 0;
+ else
+ req_update.hwp_use_pkg = 1;
break;
case 'v':
- verbose++;
+ print_version();
+ exit(0);
+ break;
+ case 'w':
+ req_update.hwp_window = parse_cmdline_hwp_window(parse_optarg_string(optarg));
break;
default:
usage();
}
}
- /* if -r, then should be no additional optind */
- if (read_only && (argc > optind))
- usage();
-
/*
- * if no -r , then must be one additional optind
+ * v1 allowed "performance"|"normal"|"power" with no policy specifier
+ * to update BIAS. Continue to support that, even though no longer documented.
*/
- if (!read_only) {
+ if (argc == optind + 1)
+ new_epb = parse_cmdline_epb(parse_optarg_string(argv[optind]));
- if (argc != optind + 1) {
- printf("must supply -r or policy param\n");
- usage();
- }
+ if (argc > optind + 1) {
+ fprintf(stderr, "stray parameter '%s'\n", argv[optind + 1]);
+ usage();
+ }
+}
- if (!strcmp("performance", argv[optind])) {
- new_bias = BIAS_PERFORMANCE;
- } else if (!strcmp("normal", argv[optind])) {
- new_bias = BIAS_BALANCE;
- } else if (!strcmp("powersave", argv[optind])) {
- new_bias = BIAS_POWERSAVE;
- } else {
- char *endptr;
-
- new_bias = strtoull(argv[optind], &endptr, 0);
- if (endptr == argv[optind] ||
- new_bias > BIAS_POWERSAVE) {
- fprintf(stderr, "invalid value: %s\n",
- argv[optind]);
- usage();
- }
- }
+
+int get_msr(int cpu, int offset, unsigned long long *msr)
+{
+ int retval;
+ char pathname[32];
+ int fd;
+
+ sprintf(pathname, "/dev/cpu/%d/msr", cpu);
+ fd = open(pathname, O_RDONLY);
+ if (fd < 0)
+ err(-1, "%s open failed, try chown or chmod +r /dev/cpu/*/msr, or run as root", pathname);
+
+ retval = pread(fd, msr, sizeof(*msr), offset);
+ if (retval != sizeof(*msr))
+ err(-1, "%s offset 0x%llx read failed", pathname, (unsigned long long)offset);
+
+ if (debug > 1)
+ fprintf(stderr, "get_msr(cpu%d, 0x%X, 0x%llX)\n", cpu, offset, *msr);
+
+ close(fd);
+ return 0;
+}
+
+int put_msr(int cpu, int offset, unsigned long long new_msr)
+{
+ char pathname[32];
+ int retval;
+ int fd;
+
+ sprintf(pathname, "/dev/cpu/%d/msr", cpu);
+ fd = open(pathname, O_RDWR);
+ if (fd < 0)
+ err(-1, "%s open failed, try chown or chmod +r /dev/cpu/*/msr, or run as root", pathname);
+
+ retval = pwrite(fd, &new_msr, sizeof(new_msr), offset);
+ if (retval != sizeof(new_msr))
+ err(-2, "pwrite(cpu%d, offset 0x%x, 0x%llx) = %d", cpu, offset, new_msr, retval);
+
+ close(fd);
+
+ if (debug > 1)
+ fprintf(stderr, "put_msr(cpu%d, 0x%X, 0x%llX)\n", cpu, offset, new_msr);
+
+ return 0;
+}
+
+void print_hwp_cap(int cpu, struct msr_hwp_cap *cap, char *str)
+{
+ if (cpu != -1)
+ printf("cpu%d: ", cpu);
+
+ printf("HWP_CAP: low %d eff %d guar %d high %d\n",
+ cap->lowest, cap->efficient, cap->guaranteed, cap->highest);
+}
+void read_hwp_cap(int cpu, struct msr_hwp_cap *cap, unsigned int msr_offset)
+{
+ unsigned long long msr;
+
+ get_msr(cpu, msr_offset, &msr);
+
+ cap->highest = msr_perf_2_ratio(HWP_HIGHEST_PERF(msr));
+ cap->guaranteed = msr_perf_2_ratio(HWP_GUARANTEED_PERF(msr));
+ cap->efficient = msr_perf_2_ratio(HWP_MOSTEFFICIENT_PERF(msr));
+ cap->lowest = msr_perf_2_ratio(HWP_LOWEST_PERF(msr));
+}
+
+void print_hwp_request(int cpu, struct msr_hwp_request *h, char *str)
+{
+ if (cpu != -1)
+ printf("cpu%d: ", cpu);
+
+ if (str)
+ printf("%s", str);
+
+ printf("HWP_REQ: min %d max %d des %d epp %d window 0x%x (%d*10^%dus) use_pkg %d\n",
+ h->hwp_min, h->hwp_max, h->hwp_desired, h->hwp_epp,
+ h->hwp_window, h->hwp_window & 0x7F, (h->hwp_window >> 7) & 0x7, h->hwp_use_pkg);
+}
+void print_hwp_request_pkg(int pkg, struct msr_hwp_request *h, char *str)
+{
+ printf("pkg%d: ", pkg);
+
+ if (str)
+ printf("%s", str);
+
+ printf("HWP_REQ_PKG: min %d max %d des %d epp %d window 0x%x (%d*10^%dus)\n",
+ h->hwp_min, h->hwp_max, h->hwp_desired, h->hwp_epp,
+ h->hwp_window, h->hwp_window & 0x7F, (h->hwp_window >> 7) & 0x7);
+}
+void read_hwp_request(int cpu, struct msr_hwp_request *hwp_req, unsigned int msr_offset)
+{
+ unsigned long long msr;
+
+ get_msr(cpu, msr_offset, &msr);
+
+ hwp_req->hwp_min = msr_perf_2_ratio((((msr) >> 0) & 0xff));
+ hwp_req->hwp_max = msr_perf_2_ratio((((msr) >> 8) & 0xff));
+ hwp_req->hwp_desired = msr_perf_2_ratio((((msr) >> 16) & 0xff));
+ hwp_req->hwp_epp = (((msr) >> 24) & 0xff);
+ hwp_req->hwp_window = (((msr) >> 32) & 0x3ff);
+ hwp_req->hwp_use_pkg = (((msr) >> 42) & 0x1);
+}
+
+void write_hwp_request(int cpu, struct msr_hwp_request *hwp_req, unsigned int msr_offset)
+{
+ unsigned long long msr = 0;
+
+ if (debug > 1)
+ printf("cpu%d: requesting min %d max %d des %d epp %d window 0x%0x use_pkg %d\n",
+ cpu, hwp_req->hwp_min, hwp_req->hwp_max,
+ hwp_req->hwp_desired, hwp_req->hwp_epp,
+ hwp_req->hwp_window, hwp_req->hwp_use_pkg);
+
+ msr |= HWP_MIN_PERF(ratio_2_msr_perf(hwp_req->hwp_min));
+ msr |= HWP_MAX_PERF(ratio_2_msr_perf(hwp_req->hwp_max));
+ msr |= HWP_DESIRED_PERF(ratio_2_msr_perf(hwp_req->hwp_desired));
+ msr |= HWP_ENERGY_PERF_PREFERENCE(hwp_req->hwp_epp);
+ msr |= HWP_ACTIVITY_WINDOW(hwp_req->hwp_window);
+ msr |= HWP_PACKAGE_CONTROL(hwp_req->hwp_use_pkg);
+
+ put_msr(cpu, msr_offset, msr);
+}
+
+int print_cpu_msrs(int cpu)
+{
+ unsigned long long msr;
+ struct msr_hwp_request req;
+ struct msr_hwp_cap cap;
+
+ if (has_epb) {
+ get_msr(cpu, MSR_IA32_ENERGY_PERF_BIAS, &msr);
+
+ printf("cpu%d: EPB %u\n", cpu, (unsigned int) msr);
}
+
+ if (!has_hwp)
+ return 0;
+
+ read_hwp_request(cpu, &req, MSR_HWP_REQUEST);
+ print_hwp_request(cpu, &req, "");
+
+ read_hwp_cap(cpu, &cap, MSR_HWP_CAPABILITIES);
+ print_hwp_cap(cpu, &cap, "");
+
+ return 0;
+}
+
+int print_pkg_msrs(int pkg)
+{
+ struct msr_hwp_request req;
+ unsigned long long msr;
+
+ if (!has_hwp)
+ return 0;
+
+ read_hwp_request(first_cpu_in_pkg[pkg], &req, MSR_HWP_REQUEST_PKG);
+ print_hwp_request_pkg(pkg, &req, "");
+
+ if (has_hwp_notify) {
+ get_msr(first_cpu_in_pkg[pkg], MSR_HWP_INTERRUPT, &msr);
+ fprintf(stderr,
+ "pkg%d: MSR_HWP_INTERRUPT: 0x%08llx (Excursion_Min-%sabled, Guaranteed_Perf_Change-%sabled)\n",
+ pkg, msr,
+ ((msr) & 0x2) ? "EN" : "Dis",
+ ((msr) & 0x1) ? "EN" : "Dis");
+ }
+ get_msr(first_cpu_in_pkg[pkg], MSR_HWP_STATUS, &msr);
+ fprintf(stderr,
+ "pkg%d: MSR_HWP_STATUS: 0x%08llx (%sExcursion_Min, %sGuaranteed_Perf_Change)\n",
+ pkg, msr,
+ ((msr) & 0x4) ? "" : "No-",
+ ((msr) & 0x1) ? "" : "No-");
+
+ return 0;
}
/*
- * validate_cpuid()
- * returns on success, quietly exits on failure (make verbose with -v)
+ * Assumption: All HWP systems have 100 MHz bus clock
*/
-void validate_cpuid(void)
+int ratio_2_sysfs_khz(int ratio)
{
- unsigned int eax, ebx, ecx, edx, max_level;
- unsigned int fms, family, model, stepping;
+ int bclk_khz = 100 * 1000; /* 100,000 KHz = 100 MHz */
- eax = ebx = ecx = edx = 0;
+ return ratio * bclk_khz;
+}
+/*
+ * If HWP is enabled and cpufreq sysfs attribtes are present,
+ * then update sysfs, so that it will not become
+ * stale when we write to MSRs.
+ * (intel_pstate's max_perf_pct and min_perf_pct will follow cpufreq,
+ * so we don't have to touch that.)
+ */
+void update_cpufreq_scaling_freq(int is_max, int cpu, unsigned int ratio)
+{
+ char pathname[64];
+ FILE *fp;
+ int retval;
+ int khz;
- asm("cpuid" : "=a" (max_level), "=b" (ebx), "=c" (ecx),
- "=d" (edx) : "a" (0));
+ sprintf(pathname, "/sys/devices/system/cpu/cpu%d/cpufreq/scaling_%s_freq",
+ cpu, is_max ? "max" : "min");
- if (ebx != 0x756e6547 || edx != 0x49656e69 || ecx != 0x6c65746e) {
- if (verbose)
- fprintf(stderr, "%.4s%.4s%.4s != GenuineIntel",
- (char *)&ebx, (char *)&edx, (char *)&ecx);
- exit(1);
+ fp = fopen(pathname, "w");
+ if (!fp) {
+ if (debug)
+ perror(pathname);
+ return;
}
- asm("cpuid" : "=a" (fms), "=c" (ecx), "=d" (edx) : "a" (1) : "ebx");
- family = (fms >> 8) & 0xf;
- model = (fms >> 4) & 0xf;
- stepping = fms & 0xf;
- if (family == 6 || family == 0xf)
- model += ((fms >> 16) & 0xf) << 4;
+ khz = ratio_2_sysfs_khz(ratio);
+ retval = fprintf(fp, "%d", khz);
+ if (retval < 0)
+ if (debug)
+ perror("fprintf");
+ if (debug)
+ printf("echo %d > %s\n", khz, pathname);
- if (verbose > 1)
- printf("CPUID %d levels family:model:stepping "
- "0x%x:%x:%x (%d:%d:%d)\n", max_level,
- family, model, stepping, family, model, stepping);
+ fclose(fp);
+}
- if (!(edx & (1 << 5))) {
- if (verbose)
- printf("CPUID: no MSR\n");
- exit(1);
+/*
+ * We update all sysfs before updating any MSRs because of
+ * bugs in cpufreq/intel_pstate where the sysfs writes
+ * for a CPU may change the min/max values on other CPUS.
+ */
+
+int update_sysfs(int cpu)
+{
+ if (!has_hwp)
+ return 0;
+
+ if (!hwp_update_enabled())
+ return 0;
+
+ if (access("/sys/devices/system/cpu/cpu0/cpufreq", F_OK))
+ return 0;
+
+ if (update_hwp_min)
+ update_cpufreq_scaling_freq(0, cpu, req_update.hwp_min);
+
+ if (update_hwp_max)
+ update_cpufreq_scaling_freq(1, cpu, req_update.hwp_max);
+
+ return 0;
+}
+
+int verify_hwp_req_self_consistency(int cpu, struct msr_hwp_request *req)
+{
+ /* fail if min > max requested */
+ if (req->hwp_min > req->hwp_max) {
+ errx(1, "cpu%d: requested hwp-min %d > hwp_max %d",
+ cpu, req->hwp_min, req->hwp_max);
}
- /*
- * Support for MSR_IA32_ENERGY_PERF_BIAS
- * is indicated by CPUID.06H.ECX.bit3
- */
- asm("cpuid" : "=a" (eax), "=b" (ebx), "=c" (ecx), "=d" (edx) : "a" (6));
- if (verbose)
- printf("CPUID.06H.ECX: 0x%x\n", ecx);
- if (!(ecx & (1 << 3))) {
- if (verbose)
- printf("CPUID: No MSR_IA32_ENERGY_PERF_BIAS\n");
- exit(1);
+ /* fail if desired > max requestd */
+ if (req->hwp_desired && (req->hwp_desired > req->hwp_max)) {
+ errx(1, "cpu%d: requested hwp-desired %d > hwp_max %d",
+ cpu, req->hwp_desired, req->hwp_max);
}
- return; /* success */
+ /* fail if desired < min requestd */
+ if (req->hwp_desired && (req->hwp_desired < req->hwp_min)) {
+ errx(1, "cpu%d: requested hwp-desired %d < requested hwp_min %d",
+ cpu, req->hwp_desired, req->hwp_min);
+ }
+
+ return 0;
}
-unsigned long long get_msr(int cpu, int offset)
+int check_hwp_request_v_hwp_capabilities(int cpu, struct msr_hwp_request *req, struct msr_hwp_cap *cap)
{
- unsigned long long msr;
- char msr_path[32];
- int retval;
- int fd;
+ if (update_hwp_max) {
+ if (req->hwp_max > cap->highest)
+ errx(1, "cpu%d: requested max %d > capabilities highest %d, use --force?",
+ cpu, req->hwp_max, cap->highest);
+ if (req->hwp_max < cap->lowest)
+ errx(1, "cpu%d: requested max %d < capabilities lowest %d, use --force?",
+ cpu, req->hwp_max, cap->lowest);
+ }
- sprintf(msr_path, "/dev/cpu/%d/msr", cpu);
- fd = open(msr_path, O_RDONLY);
- if (fd < 0) {
- printf("Try \"# modprobe msr\"\n");
- perror(msr_path);
- exit(1);
+ if (update_hwp_min) {
+ if (req->hwp_min > cap->highest)
+ errx(1, "cpu%d: requested min %d > capabilities highest %d, use --force?",
+ cpu, req->hwp_min, cap->highest);
+ if (req->hwp_min < cap->lowest)
+ errx(1, "cpu%d: requested min %d < capabilities lowest %d, use --force?",
+ cpu, req->hwp_min, cap->lowest);
}
- retval = pread(fd, &msr, sizeof msr, offset);
+ if (update_hwp_min && update_hwp_max && (req->hwp_min > req->hwp_max))
+ errx(1, "cpu%d: requested min %d > requested max %d",
+ cpu, req->hwp_min, req->hwp_max);
- if (retval != sizeof msr) {
- printf("pread cpu%d 0x%x = %d\n", cpu, offset, retval);
- exit(-2);
+ if (update_hwp_desired && req->hwp_desired) {
+ if (req->hwp_desired > req->hwp_max)
+ errx(1, "cpu%d: requested desired %d > requested max %d, use --force?",
+ cpu, req->hwp_desired, req->hwp_max);
+ if (req->hwp_desired < req->hwp_min)
+ errx(1, "cpu%d: requested desired %d < requested min %d, use --force?",
+ cpu, req->hwp_desired, req->hwp_min);
+ if (req->hwp_desired < cap->lowest)
+ errx(1, "cpu%d: requested desired %d < capabilities lowest %d, use --force?",
+ cpu, req->hwp_desired, cap->lowest);
+ if (req->hwp_desired > cap->highest)
+ errx(1, "cpu%d: requested desired %d > capabilities highest %d, use --force?",
+ cpu, req->hwp_desired, cap->highest);
}
- close(fd);
- return msr;
+
+ return 0;
}
-unsigned long long put_msr(int cpu, unsigned long long new_msr, int offset)
+int update_hwp_request(int cpu)
{
- unsigned long long old_msr;
- char msr_path[32];
- int retval;
- int fd;
+ struct msr_hwp_request req;
+ struct msr_hwp_cap cap;
+
+ int msr_offset = MSR_HWP_REQUEST;
+
+ read_hwp_request(cpu, &req, msr_offset);
+ if (debug)
+ print_hwp_request(cpu, &req, "old: ");
+
+ if (update_hwp_min)
+ req.hwp_min = req_update.hwp_min;
+
+ if (update_hwp_max)
+ req.hwp_max = req_update.hwp_max;
+
+ if (update_hwp_desired)
+ req.hwp_desired = req_update.hwp_desired;
+
+ if (update_hwp_window)
+ req.hwp_window = req_update.hwp_window;
+
+ if (update_hwp_epp)
+ req.hwp_epp = req_update.hwp_epp;
+
+ req.hwp_use_pkg = req_update.hwp_use_pkg;
+
+ read_hwp_cap(cpu, &cap, MSR_HWP_CAPABILITIES);
+ if (debug)
+ print_hwp_cap(cpu, &cap, "");
+
+ if (!force)
+ check_hwp_request_v_hwp_capabilities(cpu, &req, &cap);
+
+ verify_hwp_req_self_consistency(cpu, &req);
- sprintf(msr_path, "/dev/cpu/%d/msr", cpu);
- fd = open(msr_path, O_RDWR);
- if (fd < 0) {
- perror(msr_path);
- exit(1);
+ write_hwp_request(cpu, &req, msr_offset);
+
+ if (debug) {
+ read_hwp_request(cpu, &req, msr_offset);
+ print_hwp_request(cpu, &req, "new: ");
}
+ return 0;
+}
+int update_hwp_request_pkg(int pkg)
+{
+ struct msr_hwp_request req;
+ struct msr_hwp_cap cap;
+ int cpu = first_cpu_in_pkg[pkg];
+
+ int msr_offset = MSR_HWP_REQUEST_PKG;
+
+ read_hwp_request(cpu, &req, msr_offset);
+ if (debug)
+ print_hwp_request_pkg(pkg, &req, "old: ");
+
+ if (update_hwp_min)
+ req.hwp_min = req_update.hwp_min;
+
+ if (update_hwp_max)
+ req.hwp_max = req_update.hwp_max;
+
+ if (update_hwp_desired)
+ req.hwp_desired = req_update.hwp_desired;
+
+ if (update_hwp_window)
+ req.hwp_window = req_update.hwp_window;
+
+ if (update_hwp_epp)
+ req.hwp_epp = req_update.hwp_epp;
+
+ read_hwp_cap(cpu, &cap, MSR_HWP_CAPABILITIES);
+ if (debug)
+ print_hwp_cap(cpu, &cap, "");
+
+ if (!force)
+ check_hwp_request_v_hwp_capabilities(cpu, &req, &cap);
+
+ verify_hwp_req_self_consistency(cpu, &req);
+
+ write_hwp_request(cpu, &req, msr_offset);
- retval = pread(fd, &old_msr, sizeof old_msr, offset);
- if (retval != sizeof old_msr) {
- perror("pwrite");
- printf("pread cpu%d 0x%x = %d\n", cpu, offset, retval);
- exit(-2);
+ if (debug) {
+ read_hwp_request(cpu, &req, msr_offset);
+ print_hwp_request_pkg(pkg, &req, "new: ");
}
+ return 0;
+}
+
+int enable_hwp_on_cpu(int cpu)
+{
+ unsigned long long msr;
+
+ get_msr(cpu, MSR_PM_ENABLE, &msr);
+ put_msr(cpu, MSR_PM_ENABLE, 1);
+
+ if (verbose)
+ printf("cpu%d: MSR_PM_ENABLE old: %d new: %d\n", cpu, (unsigned int) msr, 1);
+
+ return 0;
+}
+
+int update_cpu_msrs(int cpu)
+{
+ unsigned long long msr;
+
- retval = pwrite(fd, &new_msr, sizeof new_msr, offset);
- if (retval != sizeof new_msr) {
- perror("pwrite");
- printf("pwrite cpu%d 0x%x = %d\n", cpu, offset, retval);
- exit(-2);
+ if (update_epb) {
+ get_msr(cpu, MSR_IA32_ENERGY_PERF_BIAS, &msr);
+ put_msr(cpu, MSR_IA32_ENERGY_PERF_BIAS, new_epb);
+
+ if (verbose)
+ printf("cpu%d: ENERGY_PERF_BIAS old: %d new: %d\n",
+ cpu, (unsigned int) msr, (unsigned int) new_epb);
}
- close(fd);
+ if (update_turbo) {
+ int turbo_is_present_and_disabled;
+
+ get_msr(cpu, MSR_IA32_MISC_ENABLE, &msr);
+
+ turbo_is_present_and_disabled = ((msr & MSR_IA32_MISC_ENABLE_TURBO_DISABLE) != 0);
+
+ if (turbo_update_value == 1) {
+ if (turbo_is_present_and_disabled) {
+ msr &= ~MSR_IA32_MISC_ENABLE_TURBO_DISABLE;
+ put_msr(cpu, MSR_IA32_MISC_ENABLE, msr);
+ if (verbose)
+ printf("cpu%d: turbo ENABLE\n", cpu);
+ }
+ } else {
+ /*
+ * if "turbo_is_enabled" were known to be describe this cpu
+ * then we could use it here to skip redundant disable requests.
+ * but cpu may be in a different package, so we always write.
+ */
+ msr |= MSR_IA32_MISC_ENABLE_TURBO_DISABLE;
+ put_msr(cpu, MSR_IA32_MISC_ENABLE, msr);
+ if (verbose)
+ printf("cpu%d: turbo DISABLE\n", cpu);
+ }
+ }
+
+ if (!has_hwp)
+ return 0;
+
+ if (!hwp_update_enabled())
+ return 0;
+
+ update_hwp_request(cpu);
+ return 0;
+}
+
+/*
+ * Open a file, and exit on failure
+ */
+FILE *fopen_or_die(const char *path, const char *mode)
+{
+ FILE *filep = fopen(path, "r");
- return old_msr;
+ if (!filep)
+ err(1, "%s: open failed", path);
+ return filep;
}
-void print_msr(int cpu)
+unsigned int get_pkg_num(int cpu)
{
- printf("cpu%d: 0x%016llx\n",
- cpu, get_msr(cpu, MSR_IA32_ENERGY_PERF_BIAS));
+ FILE *fp;
+ char pathname[128];
+ unsigned int pkg;
+ int retval;
+
+ sprintf(pathname, "/sys/devices/system/cpu/cpu%d/topology/physical_package_id", cpu);
+
+ fp = fopen_or_die(pathname, "r");
+ retval = fscanf(fp, "%d\n", &pkg);
+ if (retval != 1)
+ errx(1, "%s: failed to parse", pathname);
+ return pkg;
}
-void update_msr(int cpu)
+int set_max_cpu_pkg_num(int cpu)
{
- unsigned long long previous_msr;
+ unsigned int pkg;
- previous_msr = put_msr(cpu, new_bias, MSR_IA32_ENERGY_PERF_BIAS);
+ if (max_cpu_num < cpu)
+ max_cpu_num = cpu;
- if (verbose)
- printf("cpu%d msr0x%x 0x%016llx -> 0x%016llx\n",
- cpu, MSR_IA32_ENERGY_PERF_BIAS, previous_msr, new_bias);
+ pkg = get_pkg_num(cpu);
+
+ if (pkg >= MAX_PACKAGES)
+ errx(1, "cpu%d: %d >= MAX_PACKAGES (%d)", cpu, pkg, MAX_PACKAGES);
+
+ if (pkg > max_pkg_num)
+ max_pkg_num = pkg;
- return;
+ if ((pkg_present_set & (1ULL << pkg)) == 0) {
+ pkg_present_set |= (1ULL << pkg);
+ first_cpu_in_pkg[pkg] = cpu;
+ }
+
+ return 0;
+}
+int mark_cpu_present(int cpu)
+{
+ CPU_SET_S(cpu, cpu_setsize, cpu_present_set);
+ return 0;
}
-char *proc_stat = "/proc/stat";
/*
- * run func() on every cpu in /dev/cpu
+ * run func(cpu) on every cpu in /proc/stat
+ * return max_cpu number
*/
-void for_every_cpu(void (func)(int))
+int for_all_proc_cpus(int (func)(int))
{
FILE *fp;
+ int cpu_num;
int retval;
- fp = fopen(proc_stat, "r");
- if (fp == NULL) {
- perror(proc_stat);
- exit(1);
- }
+ fp = fopen_or_die(proc_stat, "r");
retval = fscanf(fp, "cpu %*d %*d %*d %*d %*d %*d %*d %*d %*d %*d\n");
- if (retval != 0) {
- perror("/proc/stat format");
- exit(1);
- }
+ if (retval != 0)
+ err(1, "%s: failed to parse format", proc_stat);
while (1) {
- int cpu;
-
- retval = fscanf(fp,
- "cpu%u %*d %*d %*d %*d %*d %*d %*d %*d %*d %*d\n",
- &cpu);
+ retval = fscanf(fp, "cpu%u %*d %*d %*d %*d %*d %*d %*d %*d %*d %*d\n", &cpu_num);
if (retval != 1)
break;
- func(cpu);
+ retval = func(cpu_num);
+ if (retval) {
+ fclose(fp);
+ return retval;
+ }
}
fclose(fp);
+ return 0;
+}
+
+void for_all_cpus_in_set(size_t set_size, cpu_set_t *cpu_set, int (func)(int))
+{
+ int cpu_num;
+
+ for (cpu_num = 0; cpu_num <= max_cpu_num; ++cpu_num)
+ if (CPU_ISSET_S(cpu_num, set_size, cpu_set))
+ func(cpu_num);
+}
+
+void init_data_structures(void)
+{
+ for_all_proc_cpus(set_max_cpu_pkg_num);
+
+ cpu_setsize = CPU_ALLOC_SIZE((max_cpu_num + 1));
+
+ cpu_present_set = CPU_ALLOC((max_cpu_num + 1));
+ if (cpu_present_set == NULL)
+ err(3, "CPU_ALLOC");
+ CPU_ZERO_S(cpu_setsize, cpu_present_set);
+ for_all_proc_cpus(mark_cpu_present);
+}
+
+/* clear has_hwp if it is not enable (or being enabled) */
+
+void verify_hwp_is_enabled(void)
+{
+ unsigned long long msr;
+
+ if (!has_hwp) /* set in early_cpuid() */
+ return;
+
+ /* MSR_PM_ENABLE[1] == 1 if HWP is enabled and MSRs visible */
+ get_msr(base_cpu, MSR_PM_ENABLE, &msr);
+ if ((msr & 1) == 0) {
+ fprintf(stderr, "HWP can be enabled using '--hwp-enable'\n");
+ has_hwp = 0;
+ return;
+ }
+}
+
+int req_update_bounds_check(void)
+{
+ if (!hwp_update_enabled())
+ return 0;
+
+ /* fail if min > max requested */
+ if ((update_hwp_max && update_hwp_min) &&
+ (req_update.hwp_min > req_update.hwp_max)) {
+ printf("hwp-min %d > hwp_max %d\n", req_update.hwp_min, req_update.hwp_max);
+ return -EINVAL;
+ }
+
+ /* fail if desired > max requestd */
+ if (req_update.hwp_desired && update_hwp_max &&
+ (req_update.hwp_desired > req_update.hwp_max)) {
+ printf("hwp-desired cannot be greater than hwp_max\n");
+ return -EINVAL;
+ }
+ /* fail if desired < min requestd */
+ if (req_update.hwp_desired && update_hwp_min &&
+ (req_update.hwp_desired < req_update.hwp_min)) {
+ printf("hwp-desired cannot be less than hwp_min\n");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+void set_base_cpu(void)
+{
+ base_cpu = sched_getcpu();
+ if (base_cpu < 0)
+ err(-ENODEV, "No valid cpus found");
+}
+
+
+void probe_dev_msr(void)
+{
+ struct stat sb;
+ char pathname[32];
+
+ sprintf(pathname, "/dev/cpu/%d/msr", base_cpu);
+ if (stat(pathname, &sb))
+ if (system("/sbin/modprobe msr > /dev/null 2>&1"))
+ err(-5, "no /dev/cpu/0/msr, Try \"# modprobe msr\" ");
+}
+/*
+ * early_cpuid()
+ * initialize turbo_is_enabled, has_hwp, has_epb
+ * before cmdline is parsed
+ */
+void early_cpuid(void)
+{
+ unsigned int eax, ebx, ecx, edx, max_level;
+ unsigned int fms, family, model;
+
+ __get_cpuid(0, &max_level, &ebx, &ecx, &edx);
+
+ if (max_level < 6)
+ errx(1, "Processor not supported\n");
+
+ __get_cpuid(1, &fms, &ebx, &ecx, &edx);
+ family = (fms >> 8) & 0xf;
+ model = (fms >> 4) & 0xf;
+ if (family == 6 || family == 0xf)
+ model += ((fms >> 16) & 0xf) << 4;
+
+ if (model == 0x4F) {
+ unsigned long long msr;
+
+ get_msr(base_cpu, MSR_TURBO_RATIO_LIMIT, &msr);
+
+ bdx_highest_ratio = msr & 0xFF;
+ }
+
+ __get_cpuid(0x6, &eax, &ebx, &ecx, &edx);
+ turbo_is_enabled = (eax >> 1) & 1;
+ has_hwp = (eax >> 7) & 1;
+ has_epb = (ecx >> 3) & 1;
+}
+
+/*
+ * parse_cpuid()
+ * set
+ * has_hwp, has_hwp_notify, has_hwp_activity_window, has_hwp_epp, has_hwp_request_pkg, has_epb
+ */
+void parse_cpuid(void)
+{
+ unsigned int eax, ebx, ecx, edx, max_level;
+ unsigned int fms, family, model, stepping;
+
+ eax = ebx = ecx = edx = 0;
+
+ __get_cpuid(0, &max_level, &ebx, &ecx, &edx);
+
+ if (ebx == 0x756e6547 && edx == 0x49656e69 && ecx == 0x6c65746e)
+ genuine_intel = 1;
+
+ if (debug)
+ fprintf(stderr, "CPUID(0): %.4s%.4s%.4s ",
+ (char *)&ebx, (char *)&edx, (char *)&ecx);
+
+ __get_cpuid(1, &fms, &ebx, &ecx, &edx);
+ family = (fms >> 8) & 0xf;
+ model = (fms >> 4) & 0xf;
+ stepping = fms & 0xf;
+ if (family == 6 || family == 0xf)
+ model += ((fms >> 16) & 0xf) << 4;
+
+ if (debug) {
+ fprintf(stderr, "%d CPUID levels; family:model:stepping 0x%x:%x:%x (%d:%d:%d)\n",
+ max_level, family, model, stepping, family, model, stepping);
+ fprintf(stderr, "CPUID(1): %s %s %s %s %s %s %s %s\n",
+ ecx & (1 << 0) ? "SSE3" : "-",
+ ecx & (1 << 3) ? "MONITOR" : "-",
+ ecx & (1 << 7) ? "EIST" : "-",
+ ecx & (1 << 8) ? "TM2" : "-",
+ edx & (1 << 4) ? "TSC" : "-",
+ edx & (1 << 5) ? "MSR" : "-",
+ edx & (1 << 22) ? "ACPI-TM" : "-",
+ edx & (1 << 29) ? "TM" : "-");
+ }
+
+ if (!(edx & (1 << 5)))
+ errx(1, "CPUID: no MSR");
+
+
+ __get_cpuid(0x6, &eax, &ebx, &ecx, &edx);
+ /* turbo_is_enabled already set */
+ /* has_hwp already set */
+ has_hwp_notify = eax & (1 << 8);
+ has_hwp_activity_window = eax & (1 << 9);
+ has_hwp_epp = eax & (1 << 10);
+ has_hwp_request_pkg = eax & (1 << 11);
+
+ if (!has_hwp_request_pkg && update_hwp_use_pkg)
+ errx(1, "--hwp-use-pkg is not available on this hardware");
+
+ /* has_epb already set */
+
+ if (debug)
+ fprintf(stderr,
+ "CPUID(6): %sTURBO, %sHWP, %sHWPnotify, %sHWPwindow, %sHWPepp, %sHWPpkg, %sEPB\n",
+ turbo_is_enabled ? "" : "No-",
+ has_hwp ? "" : "No-",
+ has_hwp_notify ? "" : "No-",
+ has_hwp_activity_window ? "" : "No-",
+ has_hwp_epp ? "" : "No-",
+ has_hwp_request_pkg ? "" : "No-",
+ has_epb ? "" : "No-");
+
+ return; /* success */
}
int main(int argc, char **argv)
{
+ set_base_cpu();
+ probe_dev_msr();
+ init_data_structures();
+
+ early_cpuid(); /* initial cpuid parse before cmdline */
+
cmdline(argc, argv);
- if (verbose > 1)
- printf("x86_energy_perf_policy Nov 24, 2010"
- " - Len Brown <lenb@kernel.org>\n");
- if (verbose > 1 && !read_only)
- printf("new_bias %lld\n", new_bias);
-
- validate_cpuid();
-
- if (cpu != -1) {
- if (read_only)
- print_msr(cpu);
- else
- update_msr(cpu);
- } else {
- if (read_only)
- for_every_cpu(print_msr);
- else
- for_every_cpu(update_msr);
+ if (debug)
+ print_version();
+
+ parse_cpuid();
+
+ /* If CPU-set and PKG-set are not initialized, default to all CPUs */
+ if ((cpu_selected_set == 0) && (pkg_selected_set == 0))
+ cpu_selected_set = cpu_present_set;
+
+ /*
+ * If HWP is being enabled, do it now, so that subsequent operations
+ * that access HWP registers can work.
+ */
+ if (update_hwp_enable)
+ for_all_cpus_in_set(cpu_setsize, cpu_selected_set, enable_hwp_on_cpu);
+
+ /* If HWP present, but disabled, warn and ignore from here forward */
+ verify_hwp_is_enabled();
+
+ if (req_update_bounds_check())
+ return -EINVAL;
+
+ /* display information only, no updates to settings */
+ if (!update_epb && !update_turbo && !hwp_update_enabled()) {
+ if (cpu_selected_set)
+ for_all_cpus_in_set(cpu_setsize, cpu_selected_set, print_cpu_msrs);
+
+ if (has_hwp_request_pkg) {
+ if (pkg_selected_set == 0)
+ pkg_selected_set = pkg_present_set;
+
+ for_packages(pkg_selected_set, print_pkg_msrs);
+ }
+
+ return 0;
}
+ /* update CPU set */
+ if (cpu_selected_set) {
+ for_all_cpus_in_set(cpu_setsize, cpu_selected_set, update_sysfs);
+ for_all_cpus_in_set(cpu_setsize, cpu_selected_set, update_cpu_msrs);
+ } else if (pkg_selected_set)
+ for_packages(pkg_selected_set, update_hwp_request_pkg);
+
return 0;
}
}
EXPORT_SYMBOL(__wrap_acpi_evaluate_object);
-union acpi_object * __wrap_acpi_evaluate_dsm(acpi_handle handle, const u8 *uuid,
+union acpi_object * __wrap_acpi_evaluate_dsm(acpi_handle handle, const guid_t *guid,
u64 rev, u64 func, union acpi_object *argv4)
{
union acpi_object *obj = ERR_PTR(-ENXIO);
rcu_read_lock();
ops = list_first_or_null_rcu(&iomap_head, typeof(*ops), list);
if (ops)
- obj = ops->evaluate_dsm(handle, uuid, rev, func, argv4);
+ obj = ops->evaluate_dsm(handle, guid, rev, func, argv4);
rcu_read_unlock();
if (IS_ERR(obj))
- return acpi_evaluate_dsm(handle, uuid, rev, func, argv4);
+ return acpi_evaluate_dsm(handle, guid, rev, func, argv4);
return obj;
}
EXPORT_SYMBOL(__wrap_acpi_evaluate_dsm);
union acpi_object *result;
static union acpi_object *nfit_test_evaluate_dsm(acpi_handle handle,
- const u8 *uuid, u64 rev, u64 func, union acpi_object *argv4)
+ const guid_t *guid, u64 rev, u64 func, union acpi_object *argv4)
{
if (handle != &nfit_ctl_handle)
return ERR_PTR(-ENXIO);
#ifndef __NFIT_TEST_H__
#define __NFIT_TEST_H__
#include <linux/list.h>
+#include <linux/uuid.h>
#include <linux/ioport.h>
#include <linux/spinlock_types.h>
typedef struct nfit_test_resource *(*nfit_test_lookup_fn)(resource_size_t);
typedef union acpi_object *(*nfit_test_evaluate_dsm_fn)(acpi_handle handle,
- const u8 *uuid, u64 rev, u64 func, union acpi_object *argv4);
+ const guid_t *guid, u64 rev, u64 func,
+ union acpi_object *argv4);
void __iomem *__wrap_ioremap_nocache(resource_size_t offset,
unsigned long size);
void __wrap_iounmap(volatile void __iomem *addr);
.result = REJECT,
.errstr = "invalid bpf_context access",
},
+ {
+ "leak pointer into ctx 1",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0,
+ offsetof(struct __sk_buff, cb[0])),
+ BPF_LD_MAP_FD(BPF_REG_2, 0),
+ BPF_STX_XADD(BPF_DW, BPF_REG_1, BPF_REG_2,
+ offsetof(struct __sk_buff, cb[0])),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map1 = { 2 },
+ .errstr_unpriv = "R2 leaks addr into mem",
+ .result_unpriv = REJECT,
+ .result = ACCEPT,
+ },
+ {
+ "leak pointer into ctx 2",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0,
+ offsetof(struct __sk_buff, cb[0])),
+ BPF_STX_XADD(BPF_DW, BPF_REG_1, BPF_REG_10,
+ offsetof(struct __sk_buff, cb[0])),
+ BPF_EXIT_INSN(),
+ },
+ .errstr_unpriv = "R10 leaks addr into mem",
+ .result_unpriv = REJECT,
+ .result = ACCEPT,
+ },
+ {
+ "leak pointer into ctx 3",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_LD_MAP_FD(BPF_REG_2, 0),
+ BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_2,
+ offsetof(struct __sk_buff, cb[0])),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map1 = { 1 },
+ .errstr_unpriv = "R2 leaks addr into ctx",
+ .result_unpriv = REJECT,
+ .result = ACCEPT,
+ },
+ {
+ "leak pointer into map val",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
+ BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3),
+ BPF_MOV64_IMM(BPF_REG_3, 0),
+ BPF_STX_MEM(BPF_DW, BPF_REG_0, BPF_REG_3, 0),
+ BPF_STX_XADD(BPF_DW, BPF_REG_0, BPF_REG_6, 0),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map1 = { 4 },
+ .errstr_unpriv = "R6 leaks addr into mem",
+ .result_unpriv = REJECT,
+ .result = ACCEPT,
+ },
{
"helper access to map: full range",
.insns = {
echo "Running remote perf test $WITH DMA"
write_file "" $REMOTE_PERF/run
echo -n " "
- read_file $LOCAL_PERF/run
+ read_file $REMOTE_PERF/run
echo " Passed"
_modprobe -r ntb_perf
cat $2 | sed -e 's/\(.*\)=n/# \1 is not set/' -e 's/^#CHECK#//' |
awk '
-BEGIN {
+{
print "if grep -q \"" $0 "\" < '"$T/.config"'";
print "then";
print "\t:";
trap 'rm -rf $T' 0
mkdir $T
-grep -v 'CONFIG_[A-Z]*_TORTURE_TEST' < ${config_template} > $T/config
+grep -v 'CONFIG_[A-Z]*_TORTURE_TEST=' < ${config_template} > $T/config
cat << ___EOF___ >> $T/config
CONFIG_INITRAMFS_SOURCE="$TORTURE_INITRD"
CONFIG_VIRTIO_PCI=y
then
git status >> $resdir/$ds/testid.txt
git rev-parse HEAD >> $resdir/$ds/testid.txt
- if ! git diff HEAD > $T/git-diff 2>&1
- then
- cp $T/git-diff $resdir/$ds
- fi
+ git diff HEAD >> $resdir/$ds/testid.txt
fi
___EOF___
awk < $T/cfgcpu.pack \
TREE09
SRCU-N
SRCU-P
+SRCU-t
+SRCU-u
TINY01
TINY02
TASKS01
--- /dev/null
+rcutorture.torture_type=srcud
CONFIG_PREEMPT_NONE=y
CONFIG_PREEMPT_VOLUNTARY=n
CONFIG_PREEMPT=n
-CONFIG_RCU_EXPERT=y
+#CHECK#CONFIG_RCU_EXPERT=n
CONFIG_SMP=y
CONFIG_NR_CPUS=8
CONFIG_HOTPLUG_CPU=y
+CONFIG_RCU_EXPERT=y
+CONFIG_RCU_FANOUT=2
+CONFIG_RCU_FANOUT_LEAF=2
CONFIG_PREEMPT_NONE=n
CONFIG_PREEMPT_VOLUNTARY=n
CONFIG_PREEMPT=y
-#CHECK#CONFIG_RCU_EXPERT=n
+CONFIG_DEBUG_LOCK_ALLOC=y
+CONFIG_PROVE_LOCKING=y
--- /dev/null
+CONFIG_SMP=n
+CONFIG_PREEMPT_NONE=y
+CONFIG_PREEMPT_VOLUNTARY=n
+CONFIG_PREEMPT=n
+#CHECK#CONFIG_TINY_SRCU=y
+CONFIG_RCU_TRACE=n
+CONFIG_DEBUG_LOCK_ALLOC=n
+CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
+CONFIG_DEBUG_ATOMIC_SLEEP=y
+#CHECK#CONFIG_PREEMPT_COUNT=y
--- /dev/null
+rcutorture.torture_type=srcu
--- /dev/null
+CONFIG_SMP=n
+CONFIG_PREEMPT_NONE=y
+CONFIG_PREEMPT_VOLUNTARY=n
+CONFIG_PREEMPT=n
+#CHECK#CONFIG_TINY_SRCU=y
+CONFIG_RCU_TRACE=n
+CONFIG_DEBUG_LOCK_ALLOC=n
+CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
+CONFIG_PREEMPT_COUNT=n
--- /dev/null
+rcutorture.torture_type=srcud
CONFIG_HZ_PERIODIC=y
CONFIG_NO_HZ_IDLE=n
CONFIG_NO_HZ_FULL=n
-CONFIG_RCU_TRACE=y
CONFIG_PROVE_LOCKING=y
-CONFIG_PROVE_RCU_REPEATEDLY=y
#CHECK#CONFIG_PROVE_RCU=y
CONFIG_DEBUG_LOCK_ALLOC=y
+CONFIG_DEBUG_OBJECTS=y
CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
-CONFIG_PREEMPT_COUNT=y
+CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_RCU_TRACE=y
CONFIG_HOTPLUG_CPU=y
CONFIG_MAXSMP=y
+CONFIG_CPUMASK_OFFSTACK=y
CONFIG_RCU_NOCB_CPU=y
-CONFIG_RCU_NOCB_CPU_ZERO=y
CONFIG_DEBUG_LOCK_ALLOC=n
CONFIG_RCU_BOOST=n
CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
CONFIG_RCU_EXPERT=y
-CONFIG_RCU_TORTURE_TEST_SLOW_CLEANUP=y
-CONFIG_RCU_TORTURE_TEST_SLOW_INIT=y
-CONFIG_RCU_TORTURE_TEST_SLOW_PREINIT=y
rcutorture.torture_type=rcu_bh maxcpus=8
+rcutree.gp_preinit_delay=3
+rcutree.gp_init_delay=3
+rcutree.gp_cleanup_delay=3
+rcu_nocbs=0
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_PROVE_LOCKING=n
CONFIG_RCU_BOOST=n
-CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
CONFIG_RCU_EXPERT=y
-CONFIG_RCU_TORTURE_TEST_SLOW_CLEANUP=y
-CONFIG_RCU_TORTURE_TEST_SLOW_INIT=y
-CONFIG_RCU_TORTURE_TEST_SLOW_PREINIT=y
+CONFIG_DEBUG_OBJECTS=y
CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
CONFIG_RCU_NOCB_CPU=n
CONFIG_DEBUG_LOCK_ALLOC=n
CONFIG_RCU_BOOST=y
-CONFIG_RCU_KTHREAD_PRIO=2
CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
CONFIG_RCU_EXPERT=y
-CONFIG_RCU_TORTURE_TEST_SLOW_CLEANUP=y
-CONFIG_RCU_TORTURE_TEST_SLOW_INIT=y
-CONFIG_RCU_TORTURE_TEST_SLOW_PREINIT=y
rcutorture.onoff_interval=1 rcutorture.onoff_holdoff=30
+rcutree.gp_preinit_delay=3
+rcutree.gp_init_delay=3
+rcutree.gp_cleanup_delay=3
+rcutree.kthread_prio=2
CONFIG_HIBERNATION=n
CONFIG_RCU_FANOUT=4
CONFIG_RCU_FANOUT_LEAF=3
-CONFIG_RCU_NOCB_CPU=n
CONFIG_DEBUG_LOCK_ALLOC=n
CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
CONFIG_RCU_EXPERT=y
-CONFIG_RCU_TORTURE_TEST_SLOW_CLEANUP=y
-CONFIG_RCU_TORTURE_TEST_SLOW_INIT=y
-CONFIG_RCU_TORTURE_TEST_SLOW_PREINIT=y
CONFIG_RCU_EQS_DEBUG=y
CONFIG_RCU_FANOUT=6
CONFIG_RCU_FANOUT_LEAF=6
CONFIG_RCU_NOCB_CPU=y
-CONFIG_RCU_NOCB_CPU_NONE=y
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_PROVE_LOCKING=y
#CHECK#CONFIG_PROVE_RCU=y
CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
CONFIG_RCU_EXPERT=y
-CONFIG_RCU_TORTURE_TEST_SLOW_CLEANUP=y
-CONFIG_RCU_TORTURE_TEST_SLOW_INIT=y
-CONFIG_RCU_TORTURE_TEST_SLOW_PREINIT=y
rcutorture.torture_type=sched
rcupdate.rcu_self_test_sched=1
+rcutree.gp_preinit_delay=3
+rcutree.gp_init_delay=3
+rcutree.gp_cleanup_delay=3
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_PROVE_LOCKING=y
#CHECK#CONFIG_PROVE_RCU=y
+CONFIG_DEBUG_OBJECTS=y
CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
CONFIG_RCU_EXPERT=y
-CONFIG_RCU_TORTURE_TEST_SLOW_CLEANUP=y
-CONFIG_RCU_TORTURE_TEST_SLOW_INIT=y
-CONFIG_RCU_TORTURE_TEST_SLOW_PREINIT=y
rcupdate.rcu_self_test_bh=1
rcupdate.rcu_self_test_sched=1
rcutree.rcu_fanout_exact=1
+rcutree.gp_preinit_delay=3
+rcutree.gp_init_delay=3
+rcutree.gp_cleanup_delay=3
CONFIG_SMP=y
CONFIG_NR_CPUS=16
-CONFIG_CPUMASK_OFFSTACK=y
CONFIG_PREEMPT_NONE=y
CONFIG_PREEMPT_VOLUNTARY=n
CONFIG_PREEMPT=n
CONFIG_NO_HZ_IDLE=n
CONFIG_NO_HZ_FULL=y
CONFIG_NO_HZ_FULL_ALL=n
-CONFIG_NO_HZ_FULL_SYSIDLE=y
CONFIG_RCU_FAST_NO_HZ=n
CONFIG_RCU_TRACE=y
CONFIG_HOTPLUG_CPU=y
CONFIG_RCU_FANOUT=2
CONFIG_RCU_FANOUT_LEAF=2
-CONFIG_RCU_NOCB_CPU=n
CONFIG_DEBUG_LOCK_ALLOC=n
CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
CONFIG_RCU_EXPERT=y
-CONFIG_RCU_TORTURE_TEST_SLOW_CLEANUP=y
-CONFIG_RCU_TORTURE_TEST_SLOW_INIT=y
-CONFIG_RCU_TORTURE_TEST_SLOW_PREINIT=y
CONFIG_RCU_FANOUT=3
CONFIG_RCU_FANOUT_LEAF=2
CONFIG_RCU_NOCB_CPU=y
-CONFIG_RCU_NOCB_CPU_ALL=y
CONFIG_DEBUG_LOCK_ALLOC=n
CONFIG_PROVE_LOCKING=n
CONFIG_RCU_BOOST=n
+++ /dev/null
-CONFIG_SMP=y
-CONFIG_NR_CPUS=16
-CONFIG_PREEMPT_NONE=n
-CONFIG_PREEMPT_VOLUNTARY=n
-CONFIG_PREEMPT=y
-#CHECK#CONFIG_PREEMPT_RCU=y
-CONFIG_HZ_PERIODIC=n
-CONFIG_NO_HZ_IDLE=y
-CONFIG_NO_HZ_FULL=n
-CONFIG_RCU_FAST_NO_HZ=n
-CONFIG_RCU_TRACE=y
-CONFIG_HOTPLUG_CPU=n
-CONFIG_SUSPEND=n
-CONFIG_HIBERNATION=n
-CONFIG_RCU_FANOUT=3
-CONFIG_RCU_FANOUT_LEAF=2
-CONFIG_RCU_NOCB_CPU=y
-CONFIG_RCU_NOCB_CPU_ALL=y
-CONFIG_DEBUG_LOCK_ALLOC=n
-CONFIG_RCU_BOOST=n
-CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
rcupdate.rcu_self_test=1
rcupdate.rcu_self_test_sched=1
rcutree.rcu_fanout_exact=1
+rcu_nocbs=0-7
-CONFIG_SMP=y
-CONFIG_NR_CPUS=8
-CONFIG_PREEMPT_NONE=n
+CONFIG_SMP=n
+CONFIG_PREEMPT_NONE=y
CONFIG_PREEMPT_VOLUNTARY=n
-CONFIG_PREEMPT=y
-#CHECK#CONFIG_PREEMPT_RCU=y
+CONFIG_PREEMPT=n
+#CHECK#CONFIG_TINY_RCU=y
CONFIG_HZ_PERIODIC=n
CONFIG_NO_HZ_IDLE=y
CONFIG_NO_HZ_FULL=n
CONFIG_RCU_FAST_NO_HZ=n
-CONFIG_RCU_TRACE=y
-CONFIG_HOTPLUG_CPU=n
-CONFIG_SUSPEND=n
-CONFIG_HIBERNATION=n
-CONFIG_RCU_FANOUT=3
-CONFIG_RCU_FANOUT_LEAF=3
CONFIG_RCU_NOCB_CPU=n
-CONFIG_DEBUG_LOCK_ALLOC=y
+CONFIG_DEBUG_LOCK_ALLOC=n
CONFIG_PROVE_LOCKING=n
CONFIG_RCU_BOOST=n
CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
+CONFIG_RCU_EXPERT=y
+CONFIG_RCU_TRACE=y
CONFIG_NO_HZ_IDLE=y
CONFIG_NO_HZ_FULL=n
CONFIG_RCU_FAST_NO_HZ=n
-CONFIG_RCU_TRACE=n
CONFIG_HOTPLUG_CPU=n
CONFIG_SUSPEND=n
CONFIG_HIBERNATION=n
CONFIG_NO_HZ_IDLE=y
CONFIG_NO_HZ_FULL=n
CONFIG_RCU_FAST_NO_HZ=n
-CONFIG_RCU_TRACE=n
CONFIG_HOTPLUG_CPU=n
CONFIG_SUSPEND=n
CONFIG_HIBERNATION=n
In common code tested by TREE_RCU test cases.
-CONFIG_NO_HZ_FULL_SYSIDLE
CONFIG_RCU_NOCB_CPU
Meaningless for TINY_RCU.
CONFIG_HOTPLUG_CPU -- Do half. (Every second.)
CONFIG_HZ_PERIODIC -- Do one.
CONFIG_NO_HZ_IDLE -- Do those not otherwise specified. (Groups of two.)
-CONFIG_NO_HZ_FULL -- Do two, one with CONFIG_NO_HZ_FULL_SYSIDLE.
-CONFIG_NO_HZ_FULL_SYSIDLE -- Do one.
+CONFIG_NO_HZ_FULL -- Do two, one with partial CPU enablement.
CONFIG_PREEMPT -- Do half. (First three and #8.)
CONFIG_PROVE_LOCKING -- Do several, covering CONFIG_DEBUG_LOCK_ALLOC=y and not.
CONFIG_PROVE_RCU -- Hardwired to CONFIG_PROVE_LOCKING.
-CONFIG_PROVE_RCU_REPEATEDLY -- Do one.
CONFIG_RCU_BOOST -- one of PREEMPT_RCU.
-CONFIG_RCU_KTHREAD_PRIO -- set to 2 for _BOOST testing.
CONFIG_RCU_FANOUT -- Cover hierarchy, but overlap with others.
CONFIG_RCU_FANOUT_LEAF -- Do one non-default.
-CONFIG_RCU_FAST_NO_HZ -- Do one, but not with CONFIG_RCU_NOCB_CPU_ALL.
-CONFIG_RCU_NOCB_CPU -- Do three, see below.
-CONFIG_RCU_NOCB_CPU_ALL -- Do one.
-CONFIG_RCU_NOCB_CPU_NONE -- Do one.
-CONFIG_RCU_NOCB_CPU_ZERO -- Do one.
+CONFIG_RCU_FAST_NO_HZ -- Do one, but not with all nohz_full CPUs.
+CONFIG_RCU_NOCB_CPU -- Do three, one with no rcu_nocbs CPUs, one with
+ rcu_nocbs=0, and one with all rcu_nocbs CPUs.
CONFIG_RCU_TRACE -- Do half.
CONFIG_SMP -- Need one !SMP for PREEMPT_RCU.
CONFIG_RCU_EXPERT=n -- Do a few, but these have to be vanilla configurations.
CONFIG_RCU_EQS_DEBUG -- Do at least one for CONFIG_NO_HZ_FULL and not.
-CONFIG_RCU_TORTURE_TEST_SLOW_CLEANUP -- Do for all but a couple TREE scenarios.
-CONFIG_RCU_TORTURE_TEST_SLOW_INIT -- Do for all but a couple TREE scenarios.
-CONFIG_RCU_TORTURE_TEST_SLOW_PREINIT -- Do for all but a couple TREE scenarios.
RCU-bh: Do one with PREEMPT and one with !PREEMPT.
RCU-sched: Do one with PREEMPT but not BOOST.
Used only to check CONFIG_RCU_FANOUT value, inspection suffices.
-CONFIG_NO_HZ_FULL_SYSIDLE_SMALL
-
- Defer until Frederic uses this.
-
CONFIG_PREEMPT_COUNT
CONFIG_PREEMPT_RCU
Always used in KVM testing.
-CONFIG_RCU_TORTURE_TEST_SLOW_PREINIT_DELAY
-CONFIG_RCU_TORTURE_TEST_SLOW_INIT_DELAY
-CONFIG_RCU_TORTURE_TEST_SLOW_CLEANUP_DELAY
-
- Inspection suffices, ignore.
-
CONFIG_PREEMPT_RCU
CONFIG_TREE_RCU
CONFIG_TINY_RCU
These are controlled by CONFIG_PREEMPT and/or CONFIG_SMP.
-CONFIG_SPARSE_RCU_POINTER
-
- Makes sense only for sparse runs, not for kernel builds.
-
CONFIG_SRCU
CONFIG_TASKS_RCU
Selected by CONFIG_RCU_TORTURE_TEST, so cannot disable.
-CONFIG_RCU_TRACE
-
- Implied by CONFIG_RCU_TRACE for Tree RCU.
-
boot parameters ignored: TBD
-#!/bin/awk -f
+#!/usr/bin/awk -f
# Modify SRCU for formal verification. The first argument should be srcu.h and
# the second should be srcu.c. Outputs modified srcu.h and srcu.c into the
BUILD_FLAGS = -DKTEST
CFLAGS += -O3 -Wl,-no-as-needed -Wall $(BUILD_FLAGS)
-LDFLAGS += -lrt -lpthread
+LDFLAGS += -lrt -lpthread -lm
# these are all "safe" tests that don't modify
# system time or require escalated privileges
inconsistency-check raw_skew threadtest rtctest
TEST_GEN_PROGS_EXTENDED = alarmtimer-suspend valid-adjtimex adjtick change_skew \
- skew_consistency clocksource-switch leap-a-day \
+ skew_consistency clocksource-switch freq-step leap-a-day \
leapcrash set-tai set-2038 set-tz
./change_skew
./skew_consistency
./clocksource-switch
+ ./freq-step
./leap-a-day -s -i 10
./leapcrash
./set-tz
--- /dev/null
+/*
+ * This test checks the response of the system clock to frequency
+ * steps made with adjtimex(). The frequency error and stability of
+ * the CLOCK_MONOTONIC clock relative to the CLOCK_MONOTONIC_RAW clock
+ * is measured in two intervals following the step. The test fails if
+ * values from the second interval exceed specified limits.
+ *
+ * Copyright (C) Miroslav Lichvar <mlichvar@redhat.com> 2017
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+
+#include <math.h>
+#include <stdio.h>
+#include <sys/timex.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "../kselftest.h"
+
+#define SAMPLES 100
+#define SAMPLE_READINGS 10
+#define MEAN_SAMPLE_INTERVAL 0.1
+#define STEP_INTERVAL 1.0
+#define MAX_PRECISION 100e-9
+#define MAX_FREQ_ERROR 10e-6
+#define MAX_STDDEV 1000e-9
+
+struct sample {
+ double offset;
+ double time;
+};
+
+static time_t mono_raw_base;
+static time_t mono_base;
+static long user_hz;
+static double precision;
+static double mono_freq_offset;
+
+static double diff_timespec(struct timespec *ts1, struct timespec *ts2)
+{
+ return ts1->tv_sec - ts2->tv_sec + (ts1->tv_nsec - ts2->tv_nsec) / 1e9;
+}
+
+static double get_sample(struct sample *sample)
+{
+ double delay, mindelay = 0.0;
+ struct timespec ts1, ts2, ts3;
+ int i;
+
+ for (i = 0; i < SAMPLE_READINGS; i++) {
+ clock_gettime(CLOCK_MONOTONIC_RAW, &ts1);
+ clock_gettime(CLOCK_MONOTONIC, &ts2);
+ clock_gettime(CLOCK_MONOTONIC_RAW, &ts3);
+
+ ts1.tv_sec -= mono_raw_base;
+ ts2.tv_sec -= mono_base;
+ ts3.tv_sec -= mono_raw_base;
+
+ delay = diff_timespec(&ts3, &ts1);
+ if (delay <= 1e-9) {
+ i--;
+ continue;
+ }
+
+ if (!i || delay < mindelay) {
+ sample->offset = diff_timespec(&ts2, &ts1);
+ sample->offset -= delay / 2.0;
+ sample->time = ts1.tv_sec + ts1.tv_nsec / 1e9;
+ mindelay = delay;
+ }
+ }
+
+ return mindelay;
+}
+
+static void reset_ntp_error(void)
+{
+ struct timex txc;
+
+ txc.modes = ADJ_SETOFFSET;
+ txc.time.tv_sec = 0;
+ txc.time.tv_usec = 0;
+
+ if (adjtimex(&txc) < 0) {
+ perror("[FAIL] adjtimex");
+ ksft_exit_fail();
+ }
+}
+
+static void set_frequency(double freq)
+{
+ struct timex txc;
+ int tick_offset;
+
+ tick_offset = 1e6 * freq / user_hz;
+
+ txc.modes = ADJ_TICK | ADJ_FREQUENCY;
+ txc.tick = 1000000 / user_hz + tick_offset;
+ txc.freq = (1e6 * freq - user_hz * tick_offset) * (1 << 16);
+
+ if (adjtimex(&txc) < 0) {
+ perror("[FAIL] adjtimex");
+ ksft_exit_fail();
+ }
+}
+
+static void regress(struct sample *samples, int n, double *intercept,
+ double *slope, double *r_stddev, double *r_max)
+{
+ double x, y, r, x_sum, y_sum, xy_sum, x2_sum, r2_sum;
+ int i;
+
+ x_sum = 0.0, y_sum = 0.0, xy_sum = 0.0, x2_sum = 0.0;
+
+ for (i = 0; i < n; i++) {
+ x = samples[i].time;
+ y = samples[i].offset;
+
+ x_sum += x;
+ y_sum += y;
+ xy_sum += x * y;
+ x2_sum += x * x;
+ }
+
+ *slope = (xy_sum - x_sum * y_sum / n) / (x2_sum - x_sum * x_sum / n);
+ *intercept = (y_sum - *slope * x_sum) / n;
+
+ *r_max = 0.0, r2_sum = 0.0;
+
+ for (i = 0; i < n; i++) {
+ x = samples[i].time;
+ y = samples[i].offset;
+ r = fabs(x * *slope + *intercept - y);
+ if (*r_max < r)
+ *r_max = r;
+ r2_sum += r * r;
+ }
+
+ *r_stddev = sqrt(r2_sum / n);
+}
+
+static int run_test(int calibration, double freq_base, double freq_step)
+{
+ struct sample samples[SAMPLES];
+ double intercept, slope, stddev1, max1, stddev2, max2;
+ double freq_error1, freq_error2;
+ int i;
+
+ set_frequency(freq_base);
+
+ for (i = 0; i < 10; i++)
+ usleep(1e6 * MEAN_SAMPLE_INTERVAL / 10);
+
+ reset_ntp_error();
+
+ set_frequency(freq_base + freq_step);
+
+ for (i = 0; i < 10; i++)
+ usleep(rand() % 2000000 * STEP_INTERVAL / 10);
+
+ set_frequency(freq_base);
+
+ for (i = 0; i < SAMPLES; i++) {
+ usleep(rand() % 2000000 * MEAN_SAMPLE_INTERVAL);
+ get_sample(&samples[i]);
+ }
+
+ if (calibration) {
+ regress(samples, SAMPLES, &intercept, &slope, &stddev1, &max1);
+ mono_freq_offset = slope;
+ printf("CLOCK_MONOTONIC_RAW frequency offset: %11.3f ppm\n",
+ 1e6 * mono_freq_offset);
+ return 0;
+ }
+
+ regress(samples, SAMPLES / 2, &intercept, &slope, &stddev1, &max1);
+ freq_error1 = slope * (1.0 - mono_freq_offset) - mono_freq_offset -
+ freq_base;
+
+ regress(samples + SAMPLES / 2, SAMPLES / 2, &intercept, &slope,
+ &stddev2, &max2);
+ freq_error2 = slope * (1.0 - mono_freq_offset) - mono_freq_offset -
+ freq_base;
+
+ printf("%6.0f %+10.3f %6.0f %7.0f %+10.3f %6.0f %7.0f\t",
+ 1e6 * freq_step,
+ 1e6 * freq_error1, 1e9 * stddev1, 1e9 * max1,
+ 1e6 * freq_error2, 1e9 * stddev2, 1e9 * max2);
+
+ if (fabs(freq_error2) > MAX_FREQ_ERROR || stddev2 > MAX_STDDEV) {
+ printf("[FAIL]\n");
+ return 1;
+ }
+
+ printf("[OK]\n");
+ return 0;
+}
+
+static void init_test(void)
+{
+ struct timespec ts;
+ struct sample sample;
+
+ if (clock_gettime(CLOCK_MONOTONIC_RAW, &ts)) {
+ perror("[FAIL] clock_gettime(CLOCK_MONOTONIC_RAW)");
+ ksft_exit_fail();
+ }
+
+ mono_raw_base = ts.tv_sec;
+
+ if (clock_gettime(CLOCK_MONOTONIC, &ts)) {
+ perror("[FAIL] clock_gettime(CLOCK_MONOTONIC)");
+ ksft_exit_fail();
+ }
+
+ mono_base = ts.tv_sec;
+
+ user_hz = sysconf(_SC_CLK_TCK);
+
+ precision = get_sample(&sample) / 2.0;
+ printf("CLOCK_MONOTONIC_RAW+CLOCK_MONOTONIC precision: %.0f ns\t\t",
+ 1e9 * precision);
+
+ if (precision > MAX_PRECISION) {
+ printf("[SKIP]\n");
+ ksft_exit_skip();
+ }
+
+ printf("[OK]\n");
+ srand(ts.tv_sec ^ ts.tv_nsec);
+
+ run_test(1, 0.0, 0.0);
+}
+
+int main(int argc, char **argv)
+{
+ double freq_base, freq_step;
+ int i, j, fails = 0;
+
+ init_test();
+
+ printf("Checking response to frequency step:\n");
+ printf(" Step 1st interval 2nd interval\n");
+ printf(" Freq Dev Max Freq Dev Max\n");
+
+ for (i = 2; i >= 0; i--) {
+ for (j = 0; j < 5; j++) {
+ freq_base = (rand() % (1 << 24) - (1 << 23)) / 65536e6;
+ freq_step = 10e-6 * (1 << (6 * i));
+ fails += run_test(0, freq_base, freq_step);
+ }
+ }
+
+ set_frequency(0.0);
+
+ if (fails)
+ ksft_exit_fail();
+
+ ksft_exit_pass();
+}
start_str = ctime(&t);
while (seconds == -1 || now - then < seconds) {
- inconsistent = 0;
+ inconsistent = -1;
/* Fill list */
for (i = 0; i < CALLS_PER_LOOP; i++)
inconsistent = i;
/* display inconsistency */
- if (inconsistent) {
+ if (inconsistent >= 0) {
unsigned long long delta;
printf("\%s\n", start_str);
/* pick defaults that works with all speeds, without short packets.
*
* Best per-frame data rates:
- * high speed, bulk 512 * 13 * 8 = 53248
- * interrupt 1024 * 3 * 8 = 24576
- * full speed, bulk/intr 64 * 19 = 1216
- * interrupt 64 * 1 = 64
- * low speed, interrupt 8 * 1 = 8
+ * super speed,bulk 1024 * 16 * 8 = 131072
+ * interrupt 1024 * 3 * 8 = 24576
+ * high speed, bulk 512 * 13 * 8 = 53248
+ * interrupt 1024 * 3 * 8 = 24576
+ * full speed, bulk/intr 64 * 19 = 1216
+ * interrupt 64 * 1 = 64
+ * low speed, interrupt 8 * 1 = 8
*/
param.iterations = 1000;
param.length = 1024;
- param.vary = 512;
+ param.vary = 1024;
param.sglen = 32;
/* for easy use when hotplugging */
"\t-c iterations default 1000\n"
"\t-s transfer length default 1024\n"
"\t-g sglen default 32\n"
- "\t-v vary default 512\n",
+ "\t-v vary default 1024\n",
argv[0]);
return 1;
}
#include <limits.h>
#include <netdb.h>
#include <libudev.h>
+#include <dirent.h>
#include "sysfs_utils.h"
#undef PROGNAME
return NULL;
}
-
-
static int parse_status(const char *value)
{
int ret = 0;
char *c;
-
- for (int i = 0; i < vhci_driver->nports; i++)
- memset(&vhci_driver->idev[i], 0, sizeof(vhci_driver->idev[i]));
-
-
/* skip a header line */
c = strchr(value, '\n');
if (!c)
int port, status, speed, devid;
unsigned long socket;
char lbusid[SYSFS_BUS_ID_SIZE];
+ struct usbip_imported_device *idev;
+ char hub[3];
- ret = sscanf(c, "%d %d %d %x %lx %31s\n",
- &port, &status, &speed,
+ ret = sscanf(c, "%2s %d %d %d %x %lx %31s\n",
+ hub, &port, &status, &speed,
&devid, &socket, lbusid);
if (ret < 5) {
BUG();
}
- dbg("port %d status %d speed %d devid %x",
- port, status, speed, devid);
+ dbg("hub %s port %d status %d speed %d devid %x",
+ hub, port, status, speed, devid);
dbg("socket %lx lbusid %s", socket, lbusid);
-
/* if a device is connected, look at it */
- {
- struct usbip_imported_device *idev = &vhci_driver->idev[port];
+ idev = &vhci_driver->idev[port];
+ memset(idev, 0, sizeof(*idev));
+
+ if (strncmp("hs", hub, 2) == 0)
+ idev->hub = HUB_SPEED_HIGH;
+ else /* strncmp("ss", hub, 2) == 0 */
+ idev->hub = HUB_SPEED_SUPER;
- idev->port = port;
- idev->status = status;
+ idev->port = port;
+ idev->status = status;
- idev->devid = devid;
+ idev->devid = devid;
- idev->busnum = (devid >> 16);
- idev->devnum = (devid & 0x0000ffff);
+ idev->busnum = (devid >> 16);
+ idev->devnum = (devid & 0x0000ffff);
- if (idev->status != VDEV_ST_NULL
- && idev->status != VDEV_ST_NOTASSIGNED) {
- idev = imported_device_init(idev, lbusid);
- if (!idev) {
- dbg("imported_device_init failed");
- return -1;
- }
+ if (idev->status != VDEV_ST_NULL
+ && idev->status != VDEV_ST_NOTASSIGNED) {
+ idev = imported_device_init(idev, lbusid);
+ if (!idev) {
+ dbg("imported_device_init failed");
+ return -1;
}
}
-
/* go to the next line */
c = strchr(c, '\n');
if (!c)
return 0;
}
+#define MAX_STATUS_NAME 16
+
static int refresh_imported_device_list(void)
{
const char *attr_status;
+ char status[MAX_STATUS_NAME+1] = "status";
+ int i, ret;
- attr_status = udev_device_get_sysattr_value(vhci_driver->hc_device,
- "status");
- if (!attr_status) {
- err("udev_device_get_sysattr_value failed");
- return -1;
+ for (i = 0; i < vhci_driver->ncontrollers; i++) {
+ if (i > 0)
+ snprintf(status, sizeof(status), "status.%d", i);
+
+ attr_status = udev_device_get_sysattr_value(vhci_driver->hc_device,
+ status);
+ if (!attr_status) {
+ err("udev_device_get_sysattr_value failed");
+ return -1;
+ }
+
+ dbg("controller %d", i);
+
+ ret = parse_status(attr_status);
+ if (ret != 0)
+ return ret;
}
- return parse_status(attr_status);
+ return 0;
}
static int get_nports(void)
return (int)strtoul(attr_nports, NULL, 10);
}
+static int vhci_hcd_filter(const struct dirent *dirent)
+{
+ return strcmp(dirent->d_name, "vhci_hcd") >= 0;
+}
+
+static int get_ncontrollers(void)
+{
+ struct dirent **namelist;
+ struct udev_device *platform;
+ int n;
+
+ platform = udev_device_get_parent(vhci_driver->hc_device);
+ if (platform == NULL)
+ return -1;
+
+ n = scandir(udev_device_get_syspath(platform), &namelist, vhci_hcd_filter, NULL);
+ if (n < 0)
+ err("scandir failed");
+ else {
+ for (int i = 0; i < n; i++)
+ free(namelist[i]);
+ free(namelist);
+ }
+
+ return n;
+}
+
/*
* Read the given port's record.
*
vhci_driver->hc_device =
udev_device_new_from_subsystem_sysname(udev_context,
USBIP_VHCI_BUS_TYPE,
- USBIP_VHCI_DRV_NAME);
+ USBIP_VHCI_DEVICE_NAME);
if (!vhci_driver->hc_device) {
err("udev_device_new_from_subsystem_sysname failed");
goto err;
}
vhci_driver->nports = get_nports();
-
dbg("available ports: %d", vhci_driver->nports);
+ if (vhci_driver->nports <= 0) {
+ err("no available ports");
+ goto err;
+ } else if (vhci_driver->nports > MAXNPORT) {
+ err("port number exceeds %d", MAXNPORT);
+ goto err;
+ }
+
+ vhci_driver->ncontrollers = get_ncontrollers();
+ dbg("available controllers: %d", vhci_driver->ncontrollers);
+
+ if (vhci_driver->ncontrollers <=0) {
+ err("no available usb controllers");
+ goto err;
+ }
+
if (refresh_imported_device_list())
goto err;
}
-int usbip_vhci_get_free_port(void)
+int usbip_vhci_get_free_port(uint32_t speed)
{
for (int i = 0; i < vhci_driver->nports; i++) {
+ if (speed == USB_SPEED_SUPER &&
+ vhci_driver->idev[i].hub != HUB_SPEED_SUPER)
+ continue;
+
if (vhci_driver->idev[i].status == VDEV_ST_NULL)
- return i;
+ return vhci_driver->idev[i].port;
}
return -1;
#include "usbip_common.h"
#define USBIP_VHCI_BUS_TYPE "platform"
+#define USBIP_VHCI_DEVICE_NAME "vhci_hcd.0"
#define MAXNPORT 128
+enum hub_speed {
+ HUB_SPEED_HIGH = 0,
+ HUB_SPEED_SUPER,
+};
+
struct usbip_imported_device {
+ enum hub_speed hub;
uint8_t port;
uint32_t status;
/* /sys/devices/platform/vhci_hcd */
struct udev_device *hc_device;
+ int ncontrollers;
int nports;
struct usbip_imported_device idev[MAXNPORT];
};
int usbip_vhci_refresh_device_list(void);
-int usbip_vhci_get_free_port(void);
+int usbip_vhci_get_free_port(uint32_t speed);
int usbip_vhci_attach_device2(uint8_t port, int sockfd, uint32_t devid,
uint32_t speed);
{
int rc;
int port;
+ uint32_t speed = udev->speed;
rc = usbip_vhci_driver_open();
if (rc < 0) {
return -1;
}
- port = usbip_vhci_get_free_port();
+ port = usbip_vhci_get_free_port(speed);
if (port < 0) {
err("no free port");
usbip_vhci_driver_close();
return -1;
}
+ dbg("got free port %d", port);
+
rc = usbip_vhci_attach_device(port, sockfd, udev->busnum,
udev->devnum, udev->speed);
if (rc < 0) {
* Called with wqh->lock held and interrupts disabled
*/
static int
-irqfd_wakeup(wait_queue_t *wait, unsigned mode, int sync, void *key)
+irqfd_wakeup(wait_queue_entry_t *wait, unsigned mode, int sync, void *key)
{
struct kvm_kernel_irqfd *irqfd =
container_of(wait, struct kvm_kernel_irqfd, wait);
projects / karo-tx-linux.git / commitdiff